2024-11-22

Title: RadPhi-3: Small Language Models for Radiology

• Authors: Mercy Ranjit, Shaury Srivastav, Tanuja Ganu
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13604
• Pdf URL: https://arxiv.org/pdf/2411.13604
• Copy Paste: [[2411.13604]] RadPhi-3: Small Language Models for Radiology(https://arxiv.org/abs/2411.13604)
• Keywords: generation
• Abstract: LLM based copilot assistants are useful in everyday tasks. There is a proliferation in the exploration of AI assistant use cases to support radiology workflows in a reliable manner. In this work, we present RadPhi-3, a Small Language Model instruction tuned from Phi-3-mini-4k-instruct with 3.8B parameters to assist with various tasks in radiology workflows. While impression summary generation has been the primary task which has been explored in prior works w.r.t radiology reports of Chest X-rays, we also explore other useful tasks like change summary generation comparing the current radiology report and its prior report, section extraction from radiology reports, tagging the reports with various pathologies and tubes, lines or devices present in them etc. In-addition, instruction tuning RadPhi-3 involved learning from a credible knowledge source used by radiologists, this http URL. RadPhi-3 can be used both to give reliable answers for radiology related queries as well as perform useful tasks related to radiology reports. RadPhi-3 achieves SOTA results on the RaLEs radiology report generation benchmark.
• 摘要：基于 LLM 的副驾驶助手在日常任务中非常有用。对 AI 助手用例的探索正在激增，以可靠的方式支持放射学工作流程。在这项工作中，我们提出了 RadPhi-3，这是一种小型语言模型指令，由 Phi-3-mini-4k-instruct 调整而来，具有 3.8B 个参数，可协助完成放射学工作流程中的各种任务。虽然印象摘要生成是先前关于胸部 X 光片放射学报告的工作中探索的主要任务，但我们还探索了其他有用的任务，例如比较当前放射学报告和其先前报告的变化摘要生成、从放射学报告中提取部分、用各种病理和管、线或设备标记报告等。此外，指令调整 RadPhi-3 涉及从放射科医生使用的可靠知识源（此 http URL）中学习。RadPhi-3 既可用于为放射学相关查询提供可靠的答案，也可用于执行与放射学报告相关的有用任务。 RadPhi-3 在 RaLEs 放射学报告生成基准上取得了 SOTA 结果。

Title: What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality

• Authors: Zihan Wang, Songlin Li, Lingyan Hao, Bowen Song, Xinyu Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13609
• Pdf URL: https://arxiv.org/pdf/2411.13609
• Copy Paste: [[2411.13609]] What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality(https://arxiv.org/abs/2411.13609)
• Keywords: generation
• Abstract: As video generation models advance rapidly, assessing the quality of generated videos has become increasingly critical. Existing metrics, such as Fréchet Video Distance (FVD), Inception Score (IS), and ClipSim, measure quality primarily in latent space rather than from a human visual perspective, often overlooking key aspects like appearance and motion consistency to physical laws. In this paper, we propose a novel metric, VAMP (Visual Appearance and Motion Plausibility), that evaluates both the visual appearance and physical plausibility of generated videos. VAMP is composed of two main components: an appearance score, which assesses color, shape, and texture consistency across frames, and a motion score, which evaluates the realism of object movements. We validate VAMP through two experiments: corrupted video evaluation and generated video evaluation. In the corrupted video evaluation, we introduce various types of corruptions into real videos and measure the correlation between corruption severity and VAMP scores. In the generated video evaluation, we use state-of-the-art models to generate videos from carefully designed prompts and compare VAMP's performance to human evaluators' rankings. Our results demonstrate that VAMP effectively captures both visual fidelity and temporal consistency, offering a more comprehensive evaluation of video quality than traditional methods.
• 摘要：随着视频生成模型的快速发展，评估生成视频的质量变得越来越重要。现有的指标，如 Fréchet 视频距离 (FVD)、初始分数 (IS) 和 ClipSim，主要在潜在空间而不是从人类视觉角度来衡量质量，往往忽略了外观和运动与物理定律的一致性等关键方面。在本文中，我们提出了一种新颖的指标 VAMP（视觉外观和运动可信度），用于评估生成视频的视觉外观和物理可信度。VAMP 由两个主要部分组成：外观分数，用于评估帧间颜色、形状和纹理的一致性，以及运动分数，用于评估物体运动的真实性。我们通过两个实验来验证 VAMP：损坏的视频评估和生成的视频评估。在损坏的视频评估中，我们将各种类型的损坏引入真实视频并测量损坏严重程度与 VAMP 分数之间的相关性。在生成的视频评估中，我们使用最先进的模型从精心设计的提示中生成视频，并将 VAMP 的性能与人类评估者的排名进行比较。我们的结果表明，VAMP 可以有效捕捉视觉保真度和时间一致性，比传统方法提供更全面的视频质量评估。

Title: Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning

• Authors: Tim Lenz, Peter Neidlinger, Marta Ligero, Georg Wölflein, Marko van Treeck, Jakob Nikolas Kather
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13623
• Pdf URL: https://arxiv.org/pdf/2411.13623
• Copy Paste: [[2411.13623]] Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning(https://arxiv.org/abs/2411.13623)
• Keywords: generation
• Abstract: Representation learning of pathology whole-slide images (WSIs) has primarily relied on weak supervision with Multiple Instance Learning (MIL). This approach leads to slide representations highly tailored to a specific clinical task. Self-supervised learning (SSL) has been successfully applied to train histopathology foundation models (FMs) for patch embedding generation. However, generating patient or slide level embeddings remains challenging. Existing approaches for slide representation learning extend the principles of SSL from patch level learning to entire slides by aligning different augmentations of the slide or by utilizing multimodal data. By integrating tile embeddings from multiple FMs, we propose a new single modality SSL method in feature space that generates useful slide representations. Our contrastive pretraining strategy, called COBRA, employs multiple FMs and an architecture based on Mamba-2. COBRA exceeds performance of state-of-the-art slide encoders on four different public CPTAC cohorts on average by at least +3.8% AUC, despite only being pretrained on 3048 WSIs from TCGA. Additionally, COBRA is readily compatible at inference time with previously unseen feature extractors.
• 摘要：病理全幻灯片图像 (WSI) 的表征学习主要依赖于多实例学习 (MIL) 的弱监督。这种方法可使幻灯片表征高度针对特定临床任务进行定制。自监督学习 (SSL) 已成功应用于训练组织病理学基础模型 (FM) 以生成斑块嵌入。但是，生成患者或幻灯片级别的嵌入仍然具有挑战性。现有的幻灯片表征学习方法通​​过对齐幻灯片的不同增强或利用多模态数据，将 SSL 的原理从斑块级别学习扩展到整个幻灯片。通过集成来自多个 FM 的图块嵌入，我们提出了一种新的特征空间单模态 SSL 方法，可生成有用的幻灯片表征。我们的对比预训练策略称为 COBRA，采用多个 FM 和基于 Mamba-2 的架构。尽管 COBRA 仅在 TCGA 的 3048 个 WSI 上进行了预训练，但在四个不同的公共 CPTAC 队列中，其性能平均比最先进的滑动编码器高出至少 +3.8% AUC。此外，COBRA 在推理时可与以前从未见过的特征提取器轻松兼容。

Title: ID-Patch: Robust ID Association for Group Photo Personalization

• Authors: Yimeng Zhang, Tiancheng Zhi, Jing Liu, Shen Sang, Liming Jiang, Qing Yan, Sijia Liu, Linjie Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13632
• Pdf URL: https://arxiv.org/pdf/2411.13632
• Copy Paste: [[2411.13632]] ID-Patch: Robust ID Association for Group Photo Personalization(https://arxiv.org/abs/2411.13632)
• Keywords: generation
• Abstract: The ability to synthesize personalized group photos and specify the positions of each identity offers immense creative potential. While such imagery can be visually appealing, it presents significant challenges for existing technologies. A persistent issue is identity (ID) leakage, where injected facial features interfere with one another, resulting in low face resemblance, incorrect positioning, and visual artifacts. Existing methods suffer from limitations such as the reliance on segmentation models, increased runtime, or a high probability of ID leakage. To address these challenges, we propose ID-Patch, a novel method that provides robust association between identities and 2D positions. Our approach generates an ID patch and ID embeddings from the same facial features: the ID patch is positioned on the conditional image for precise spatial control, while the ID embeddings integrate with text embeddings to ensure high resemblance. Experimental results demonstrate that ID-Patch surpasses baseline methods across metrics, such as face ID resemblance, ID-position association accuracy, and generation efficiency. Project Page is: this https URL
• 摘要：能够合成个性化的集体照并指定每个身份的位置提供了巨大的创造潜力。虽然这样的图像在视觉上很有吸引力，但它对现有技术提出了重大挑战。一个持续存在的问题是身份 (ID) 泄漏，其中注入的面部特征相互干扰，导致面部相似度低、定位不正确和视觉伪影。现有方法存在诸如依赖分割模型、运行时间增加或 ID 泄漏概率高等限制。为了应对这些挑战，我们提出了 ID-Patch，这是一种提供身份和 2D 位置之间稳健关联的新方法。我们的方法从相同的面部特征生成 ID 补丁和 ID 嵌入：ID 补丁定位在条件图像上以进行精确的空间控制，而 ID 嵌入与文本嵌入集成以确保高相似度。实验结果表明，ID-Patch 在面部 ID 相似度、ID 位置关联准确性和生成效率等指标方面超越了基线方法。项目页面是：这个 https URL

Title: Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU

• Authors: Haojia Sun, Yaqi Wang, Shuting Zhang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13691
• Pdf URL: https://arxiv.org/pdf/2411.13691
• Copy Paste: [[2411.13691]] Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU(https://arxiv.org/abs/2411.13691)
• Keywords: generation
• Abstract: We designed a Retrieval-Augmented Generation (RAG) system to provide large language models with relevant documents for answering domain-specific questions about Pittsburgh and Carnegie Mellon University (CMU). We extracted over 1,800 subpages using a greedy scraping strategy and employed a hybrid annotation process, combining manual and Mistral-generated question-answer pairs, achieving an inter-annotator agreement (IAA) score of 0.7625. Our RAG framework integrates BM25 and FAISS retrievers, enhanced with a reranker for improved document retrieval accuracy. Experimental results show that the RAG system significantly outperforms a non-RAG baseline, particularly in time-sensitive and complex queries, with an F1 score improvement from 5.45% to 42.21% and recall of 56.18%. This study demonstrates the potential of RAG systems in enhancing answer precision and relevance, while identifying areas for further optimization in document retrieval and model training.
• 摘要：我们设计了一个检索增强生成 (RAG) 系统，为大型语言模型提供相关文档，以回答有关匹兹堡和卡内基梅隆大学 (CMU) 的特定领域问题。我们使用贪婪抓取策略提取了 1,800 多个子页面，并采用了混合注释过程，结合了手动和 Mistral 生成的问答对，实现了注释者间一致性 (IAA) 得分 0.7625。我们的 RAG 框架集成了 BM25 和 FAISS 检索器，并通过重新排序器进行了增强，以提高文档检索准确性。实验结果表明，RAG 系统的表现明显优于非 RAG 基线，特别是在时间敏感和复杂查询方面，F1 得分从 5.45% 提高到 42.21%，召回率达到 56.18%。这项研究展示了 RAG 系统在提高答案准确率和相关性方面的潜力，同时确定了文档检索和模型训练中需要进一步优化的领域。

Title: Edge-Cloud Routing for Text-to-Image Model with Token-Level Multi-Metric Prediction

• Authors: Zewei Xin, Qinya Li, Chaoyue Niu, Fan Wu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13787
• Pdf URL: https://arxiv.org/pdf/2411.13787
• Copy Paste: [[2411.13787]] Edge-Cloud Routing for Text-to-Image Model with Token-Level Multi-Metric Prediction(https://arxiv.org/abs/2411.13787)
• Keywords: generation
• Abstract: Large text-to-image models demonstrate impressive generation capabilities; however, their substantial size necessitates expensive cloud servers for deployment. Conversely, light-weight models can be deployed on edge devices at lower cost but often with inferior generation quality for complex user prompts. To strike a balance between performance and cost, we propose a routing framework, called \texttt{RouteT2I}, which dynamically selects either the large cloud model or the light-weight edge model for each user prompt. Since generated image quality is challenging to measure directly, \texttt{RouteT2I} establishes multi-dimensional quality metrics, particularly, by evaluating the similarity between the generated images and both positive and negative texts that describe each specific quality metric. \texttt{RouteT2I} then predicts the expected quality of the generated images by identifying key tokens in the prompt and comparing their impact on the quality. \texttt{RouteT2I} further introduces the Pareto relative superiority to compare the multi-metric quality of the generated images. Based on this comparison and predefined cost constraints, \texttt{RouteT2I} allocates prompts to either the edge or the cloud. Evaluation reveals that \texttt{RouteT2I} significantly reduces the number of requesting large cloud model while maintaining high-quality image generation.
• 摘要：大型文本转图像模型展示了令人印象深刻的生成能力；然而，它们庞大的规模需要昂贵的云服务器来部署。相反，轻量级模型可以以较低的成本部署在边缘设备上，但对于复杂的用户提示，其生成质量通常较差。为了在性能和成本之间取得平衡，我们提出了一个名为 \texttt{RouteT2I} 的路由框架，它可以为每个用户提示动态选择大型云模型或轻量级边缘模型。由于生成的图像质量很难直接衡量，\texttt{RouteT2I} 建立了多维质量指标，特别是通过评估生成的图像与描述每个特定质量指标的正面和负面文本之间的相似性。\texttt{RouteT2I} 然后通过识别提示中的关键标记并比较它们对质量的影响来预测生成图像的预期质量。\texttt{RouteT2I} 进一步引入了帕累托相对优势来比较生成图像的多指标质量。根据此比较和预定义的成本约束，\texttt{RouteT2I} 将提示分配给边缘或云。评估表明，\texttt{RouteT2I} 显著减少了请求大型云模型的数量，同时保持了高质量的图像生成。

Title: GalaxyEdit: Large-Scale Image Editing Dataset with Enhanced Diffusion Adapter

• Authors: Aniruddha Bala, Rohan Jaiswal, Loay Rashid, Siddharth Roheda
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13794
• Pdf URL: https://arxiv.org/pdf/2411.13794
• Copy Paste: [[2411.13794]] GalaxyEdit: Large-Scale Image Editing Dataset with Enhanced Diffusion Adapter(https://arxiv.org/abs/2411.13794)
• Keywords: generation
• Abstract: Training of large-scale text-to-image and image-to-image models requires a huge amount of annotated data. While text-to-image datasets are abundant, data available for instruction-based image-to-image tasks like object addition and removal is limited. This is because of the several challenges associated with the data generation process, such as, significant human effort, limited automation, suboptimal end-to-end models, data diversity constraints and high expenses. We propose an automated data generation pipeline aimed at alleviating such limitations, and introduce GalaxyEdit - a large-scale image editing dataset for add and remove operations. We fine-tune the SD v1.5 model on our dataset and find that our model can successfully handle a broader range of objects and complex editing instructions, outperforming state-of-the-art methods in FID scores by 11.2\% and 26.1\% for add and remove tasks respectively. Furthermore, in light of on-device usage scenarios, we expand our research to include task-specific lightweight adapters leveraging the ControlNet-xs architecture. While ControlNet-xs excels in canny and depth guided generation, we propose to improve the communication between the control network and U-Net for more intricate add and remove tasks. We achieve this by enhancing ControlNet-xs with non-linear interaction layers based on Volterra filters. Our approach outperforms ControlNet-xs in both add/remove and canny-guided image generation tasks, highlighting the effectiveness of the proposed enhancement.
• 摘要：训练大规模文本到图像和图像到图像模型需要大量带注释的数据。虽然文本到图像数据集非常丰富，但可用于基于指令的图像到图像任务（如对象添加和删除）的数据却非常有限。这是因为数据生成过程存在一些挑战，例如，大量人力、有限的自动化、次优的端到端模型、数据多样性约束和高昂的费用。我们提出了一种旨在缓解这些限制的自动化数据生成管道，并引入了 GalaxyEdit - 一种用于添加和删除操作的大规模图像编辑数据集。我们在我们的数据集上微调了 SD v1.5 模型，发现我们的模型可以成功处理更广泛的对象和复杂的编辑指令，在添加和删除任务的 FID 得分上分别比最先进的方法高出 11.2% 和 26.1%。此外，鉴于设备上的使用场景，我们扩展了我们的研究范围，包括利用 ControlNet-xs 架构的特定于任务的轻量级适配器。虽然 ControlNet-xs 在精明和深度引导生成方面表现出色，但我们建议改进控制网络与 U-Net 之间的通信，以完成更复杂的添加和删除任务。我们通过基于 Volterra 滤波器的非线性交互层增强 ControlNet-xs 来实现这一点。我们的方法在添加/删除和精明引导图像生成任务中均优于 ControlNet-xs，凸显了所提出的增强功能的有效性。

Title: MagicDriveDiT: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control

• Authors: Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, Qiang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13807
• Pdf URL: https://arxiv.org/pdf/2411.13807
• Copy Paste: [[2411.13807]] MagicDriveDiT: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control(https://arxiv.org/abs/2411.13807)
• Keywords: generation
• Abstract: The rapid advancement of diffusion models has greatly improved video synthesis, especially in controllable video generation, which is essential for applications like autonomous driving. However, existing methods are limited by scalability and how control conditions are integrated, failing to meet the needs for high-resolution and long videos for autonomous driving applications. In this paper, we introduce MagicDriveDiT, a novel approach based on the DiT architecture, and tackle these challenges. Our method enhances scalability through flow matching and employs a progressive training strategy to manage complex scenarios. By incorporating spatial-temporal conditional encoding, MagicDriveDiT achieves precise control over spatial-temporal latents. Comprehensive experiments show its superior performance in generating realistic street scene videos with higher resolution and more frames. MagicDriveDiT significantly improves video generation quality and spatial-temporal controls, expanding its potential applications across various tasks in autonomous driving.
• 摘要：扩散模型的快速发展极大地改善了视频合成，尤其是可控视频生成，这对于自动驾驶等应用至关重要。然而，现有的方法受到可扩展性和控制条件集成方式的限制，无法满足自动驾驶应用对高分辨率和长视频的需求。在本文中，我们介绍了一种基于 DiT 架构的新方法 MagicDriveDiT，并解决了这些挑战。我们的方法通过流匹配增强了可扩展性，并采用渐进式训练策略来管理复杂场景。通过结合时空条件编码，MagicDriveDiT 实现了对时空潜伏信息的精确控制。全面的实验表明它在生成具有更高分辨率和更多帧的逼真街景视频方面具有卓越的性能。MagicDriveDiT 显著提高了视频生成质量和时空控制，扩大了其在自动驾驶各种任务中的潜在应用。

Title: Detecting Human Artifacts from Text-to-Image Models

• Authors: Kaihong Wang, Lingzhi Zhang, Jianming Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13842
• Pdf URL: https://arxiv.org/pdf/2411.13842
• Copy Paste: [[2411.13842]] Detecting Human Artifacts from Text-to-Image Models(https://arxiv.org/abs/2411.13842)
• Keywords: generation, generative
• Abstract: Despite recent advancements, text-to-image generation models often produce images containing artifacts, especially in human figures. These artifacts appear as poorly generated human bodies, including distorted, missing, or extra body parts, leading to visual inconsistencies with typical human anatomy and greatly impairing overall fidelity. In this study, we address this challenge by curating Human Artifact Dataset (HAD), the first large-scale dataset specifically designed to identify and localize human artifacts. HAD comprises over 37,000 images generated by several popular text-to-image models, annotated for human artifact localization. Using this dataset, we train the Human Artifact Detection Models (HADM), which can identify diverse artifact types across multiple generative domains and demonstrate strong generalization, even on images from unseen generators. Additionally, to further improve generators' perception of human structural coherence, we use the predictions from our HADM as feedback for diffusion model finetuning. Our experiments confirm a reduction in human artifacts in the resulting model. Furthermore, we showcase a novel application of our HADM in an iterative inpainting framework to correct human artifacts in arbitrary images directly, demonstrating its utility in improving image quality. Our dataset and detection models are available at: \url{this https URL}.
• 摘要：尽管最近取得了一些进展，但文本转图像生成模型通常会生成包含伪影的图像，尤其是在人体中。这些伪影看起来像是生成质量不佳的人体，包括扭曲、缺失或多余的身体部位，导致与典型人体解剖结构视觉不一致，并大大损害整体保真度。在本研究中，我们通过整理人体伪影数据集 (HAD) 来应对这一挑战，这是第一个专门用于识别和定位人体伪影的大型数据集。HAD 包含由几种流行的文本转图像模型生成的 37,000 多张图像，并带有注释以用于人体伪影定位。使用此数据集，我们训练人体伪影检测模型 (HADM)，该模型可以识别多个生成域中的各种伪影类型，并表现出强大的泛化能力，即使在来自看不见的生成器的图像上也是如此。此外，为了进一步提高生成器对人体结构一致性的感知，我们使用来自 HADM 的预测作为扩散模型微调的反馈。我们的实验证实了结果模型中人体伪影的减少。此外，我们展示了 HADM 在迭代修复框架中的新应用，可直接校正任意图像中的人体伪影，从而展示其在改善图像质量方面的实用性。我们的数据集和检测模型可在以下位置获得：\url{此 https URL}。

Title: Dealing with Synthetic Data Contamination in Online Continual Learning

• Authors: Maorong Wang, Nicolas Michel, Jiafeng Mao, Toshihiko Yamasaki
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13852
• Pdf URL: https://arxiv.org/pdf/2411.13852
• Copy Paste: [[2411.13852]] Dealing with Synthetic Data Contamination in Online Continual Learning(https://arxiv.org/abs/2411.13852)
• Keywords: generation
• Abstract: Image generation has shown remarkable results in generating high-fidelity realistic images, in particular with the advancement of diffusion-based models. However, the prevalence of AI-generated images may have side effects for the machine learning community that are not clearly identified. Meanwhile, the success of deep learning in computer vision is driven by the massive dataset collected on the Internet. The extensive quantity of synthetic data being added to the Internet would become an obstacle for future researchers to collect "clean" datasets without AI-generated content. Prior research has shown that using datasets contaminated by synthetic images may result in performance degradation when used for training. In this paper, we investigate the potential impact of contaminated datasets on Online Continual Learning (CL) research. We experimentally show that contaminated datasets might hinder the training of existing online CL methods. Also, we propose Entropy Selection with Real-synthetic similarity Maximization (ESRM), a method to alleviate the performance deterioration caused by synthetic images when training online CL models. Experiments show that our method can significantly alleviate performance deterioration, especially when the contamination is severe. For reproducibility, the source code of our work is available at this https URL.
• 摘要：图像生成在生成高保真逼真图像方面表现出色，尤其是随着基于扩散的模型的发展。然而，人工智能生成图像的流行可能会对机器学习社区产生尚未明确识别的副作用。同时，深度学习在计算机视觉领域的成功是由互联网上收集的海量数据集推动的。大量合成数据被添加到互联网上将成为未来研究人员收集没有人工智能生成内容的“干净”数据集的障碍。先前的研究表明，使用被合成图像污染的数据集进行训练可能会导致性能下降。在本文中，我们研究了受污染数据集对在线持续学习 (CL) 研究的潜在影响。我们通过实验表明，受污染的数据集可能会阻碍现有在线 CL 方法的训练。此外，我们提出了一种熵选择与真实合成相似性最大化 (ESRM) 的方法，用于缓解训练在线 CL 模型时合成图像造成的性能下降。实验表明，我们的方法可以显着缓解性能下降，尤其是在污染严重的情况下。为了可重复性，我们工作的源代码可在以下 https URL 上找到。

Title: Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel KAN Adapter for Enhanced Feature Adaptation

• Authors: Gayatri Deshmukh, Somsubhra De, Chirag Sehgal, Jishu Sen Gupta, Sparsh Mittal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13901
• Pdf URL: https://arxiv.org/pdf/2411.13901
• Copy Paste: [[2411.13901]] Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel KAN Adapter for Enhanced Feature Adaptation(https://arxiv.org/abs/2411.13901)
• Keywords: generation, generative
• Abstract: Specialized datasets that capture the fashion industry's rich language and styling elements can boost progress in AI-driven fashion design. We present FLORA (Fashion Language Outfit Representation for Apparel Generation), the first comprehensive dataset containing 4,330 curated pairs of fashion outfits and corresponding textual descriptions. Each description utilizes industry-specific terminology and jargon commonly used by professional fashion designers, providing precise and detailed insights into the outfits. Hence, the dataset captures the delicate features and subtle stylistic elements necessary to create high-fidelity fashion designs. We demonstrate that fine-tuning generative models on the FLORA dataset significantly enhances their capability to generate accurate and stylistically rich images from textual descriptions of fashion sketches. FLORA will catalyze the creation of advanced AI models capable of comprehending and producing subtle, stylistically rich fashion designs. It will also help fashion designers and end-users to bring their ideas to life. As a second orthogonal contribution, we introduce KAN Adapters, which leverage Kolmogorov-Arnold Networks (KAN) as adaptive modules. They serve as replacements for traditional MLP-based LoRA adapters. With learnable spline-based activations, KAN Adapters excel in modeling complex, non-linear relationships, achieving superior fidelity, faster convergence and semantic alignment. Extensive experiments and ablation studies on our proposed FLORA dataset validate the superiority of KAN Adapters over LoRA adapters. To foster further research and collaboration, we will open-source both the FLORA and our implementation code.
• 摘要：捕捉时尚行业丰富语言和造型元素的专业数据集可以促进人工智能驱动的时装设计的进步。我们推出了 FLORA（服装生成的时尚语言服装表示），这是第一个包含 4,330 对精选时尚服装和相应的文本描述的综合数据集。每项描述都使用了专业时装设计师常用的行业术语和行话，提供了对服装的精确而详细的见解。因此，该数据集捕捉到了创建高保真时装设计所必需的精细特征和微妙的风格元素。我们证明，在 FLORA 数据集上微调生成模型可显著增强它们从时装草图的文本描述中生成准确且风格丰富的图像的能力。FLORA 将催化创建能够理解和制作微妙、风格丰富的时装设计的先进人工智能模型。它还将帮助时装设计师和最终用户将他们的想法变成现实。作为第二个正交贡献，我们引入了 KAN 适配器，它利用 Kolmogorov-Arnold 网络 (KAN) 作为自适应模块。它们可替代传统的基于 MLP 的 LoRA 适配器。通过可学习的基于样条函数的激活，KAN 适配器在建模复杂的非线性关系方面表现出色，实现了卓越的保真度、更快的收敛和语义对齐。在我们提出的 FLORA 数据集上进行的大量实验和消融研究验证了 KAN 适配器相对于 LoRA 适配器的优越性。为了促进进一步的研究和合作，我们将开源 FLORA 和我们的实现代码。

Title: Learning to Cooperate with Humans using Generative Agents

• Authors: Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S. Du, Natasha Jaques
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2411.13934
• Pdf URL: https://arxiv.org/pdf/2411.13934
• Copy Paste: [[2411.13934]] Learning to Cooperate with Humans using Generative Agents(https://arxiv.org/abs/2411.13934)
• Keywords: generative
• Abstract: Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL). Current algorithms focus on training simulated human partner policies which are then used to train a Cooperator agent. The simulated human is produced either through behavior cloning over a dataset of human cooperation behavior, or by using MARL to create a population of simulated agents. However, these approaches often struggle to produce a Cooperator that can coordinate well with real humans, since the simulated humans fail to cover the diverse strategies and styles employed by people in the real world. We show \emph{learning a generative model of human partners} can effectively address this issue. Our model learns a latent variable representation of the human that can be regarded as encoding the human's unique strategy, intention, experience, or style. This generative model can be flexibly trained from any (human or neural policy) agent interaction data. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents. We evaluate our method -- \textbf{G}enerative \textbf{A}gent \textbf{M}odeling for \textbf{M}ulti-agent \textbf{A}daptation (GAMMA) -- on Overcooked, a challenging cooperative cooking game that has become a standard benchmark for zero-shot coordination. We conduct an evaluation with real human teammates, and the results show that GAMMA consistently improves performance, whether the generative model is trained on simulated populations or human datasets. Further, we propose a method for posterior sampling from the generative model that is biased towards the human data, enabling us to efficiently improve performance with only a small amount of expensive human interaction data.
• 摘要：训练能够与人类协调零次攻击的智能体是多智能体强化学习 (MARL) 的一项关键任务。当前的算法专注于训练模拟人类合作伙伴策略，然后将其用于训练合作者智能体。模拟人类是通过在人类合作行为数据集上进行行为克隆或使用 MARL 创建模拟智能体群体来生成的。然而，这些方法通常难以产生能够与真实人类很好地协调的合作者，因为模拟人类无法涵盖现实世界中人们采用的各种策略和风格。我们表明 \emph{学习人类合作伙伴的生成模型} 可以有效解决这个问题。我们的模型学习人类的潜在变量表示，可以将其视为对人类独特策略、意图、经验或风格的编码。该生成模型可以从任何（人类或神经策略）智能体交互数据中灵活地训练。通过从潜在空间采样，我们可以使用生成模型来生成不同的合作伙伴来训练合作者智能体。我们在 Overcooked 上评估了我们的方法 \textbf{G}enerative \textbf{A}gent \textbf{M}odeling for \textbf{M}ulti-agent \textbf{A}daptation (GAMMA)，这是一款具有挑战性的合作烹饪游戏，已成为零样本协调的标准基准。我们与真正的人类队友进行了评估，结果表明，无论生成模型是在模拟人群还是人类数据集上训练的，GAMMA 都能持续提高性能。此外，我们提出了一种从偏向人类数据的生成模型中进行后验采样的方法，使我们能够仅使用少量昂贵的人机交互数据来有效地提高性能。

Title: A Dataset for Evaluating Online Anomaly Detection Approaches for Discrete Multivariate Time Series

• Authors: Lucas Correia, Jan-Christoph Goos, Thomas Bäck, Anna V. Kononova
• Subjects: cs.LG, cs.AI, cs.CE, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.13951
• Pdf URL: https://arxiv.org/pdf/2411.13951
• Copy Paste: [[2411.13951]] A Dataset for Evaluating Online Anomaly Detection Approaches for Discrete Multivariate Time Series(https://arxiv.org/abs/2411.13951)
• Keywords: generation
• Abstract: Benchmarking anomaly detection approaches for multivariate time series is challenging due to the lack of high-quality datasets. Current publicly available datasets are too small, not diverse and feature trivial anomalies, which hinders measurable progress in this research area. We propose a solution: a diverse, extensive, and non-trivial dataset generated via state-of-the-art simulation tools that reflects realistic behaviour of an automotive powertrain, including its multivariate, dynamic and variable-state properties. To cater for both unsupervised and semi-supervised anomaly detection settings, as well as time series generation and forecasting, we make different versions of the dataset available, where training and test subsets are offered in contaminated and clean versions, depending on the task. We also provide baseline results from a small selection of approaches based on deterministic and variational autoencoders, as well as a non-parametric approach. As expected, the baseline experimentation shows that the approaches trained on the semi-supervised version of the dataset outperform their unsupervised counterparts, highlighting a need for approaches more robust to contaminated training data.
• 摘要：由于缺乏高质量的数据集，对多变量时间序列的异常检测方法进行基准测试具有挑战性。当前公开的数据集太小、不够多样化且具有琐碎的异常，这阻碍了该研究领域的可衡量进展。我们提出了一个解决方案：通过最先进的模拟工具生成一个多样化、广泛且非琐碎的数据集，该数据集反映了汽车动力系统的实际行为，包括其多变量、动态和可变状态属性。为了满足无监督和半监督异常检测设置以及时间序列生成和预测的需求，我们提供不同版本的数据集，其中训练和测试子集以受污染和干净的版本提供，具体取决于任务。我们还提供了基于确定性和变分自动编码器以及非参数方法的少数方法的基线结果。正如预期的那样，基线实验表明，在半监督版本的数据集上训练的方法优于无监督版本的方法，这凸显了需要对受污染的训练数据更具鲁棒性的方法。

Title: Zero-Shot Low-Light Image Enhancement via Joint Frequency Domain Priors Guided Diffusion

• Authors: Jinhong He, Shivakumara Palaiahnakote, Aoxiang Ning, Minglong Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13961
• Pdf URL: https://arxiv.org/pdf/2411.13961
• Copy Paste: [[2411.13961]] Zero-Shot Low-Light Image Enhancement via Joint Frequency Domain Priors Guided Diffusion(https://arxiv.org/abs/2411.13961)
• Keywords: generation
• Abstract: Due to the singularity of real-world paired datasets and the complexity of low-light environments, this leads to supervised methods lacking a degree of scene generalisation. Meanwhile, limited by poor lighting and content guidance, existing zero-shot methods cannot handle unknown severe degradation well. To address this problem, we will propose a new zero-shot low-light enhancement method to compensate for the lack of light and structural information in the diffusion sampling process by effectively combining the wavelet and Fourier frequency domains to construct rich a priori information. The key to the inspiration comes from the similarity between the wavelet and Fourier frequency domains: both light and structure information are closely related to specific frequency domain regions, respectively. Therefore, by transferring the diffusion process to the wavelet low-frequency domain and combining the wavelet and Fourier frequency domains by continuously decomposing them in the inverse process, the constructed rich illumination prior is utilised to guide the image generation enhancement process. Sufficient experiments show that the framework is robust and effective in various scenarios. The code will be available at: \href{this https URL}{this https URL}.
• 摘要：由于现实世界成对数据集的奇异性和弱光环境的复杂性，这导致监督方法缺乏一定程度的场景泛化。同时，受限于较差的光照和内容指导，现有的零样本方法无法很好地处理未知的严重退化。针对这一问题，我们将提出一种新的零样本弱光增强方法，通过有效地结合小波和傅里叶频域来构建丰富的先验信息，以弥补扩散采样过程中光照和结构信息的缺失。灵感的关键来自于小波和傅里叶频域之间的相似性：光照和结构信息分别与特定的频域区域紧密相关。因此，通过将扩散过程转移到小波低频域，并通过在逆过程中不断分解小波和傅里叶频域来结合它们，利用构建的丰富的光照先验来指导图像生成增强过程。充分的实验表明，该框架在各种场景下都是稳健有效的。代码将在此处提供：\href{this https URL}{this https URL}。

Title: Transforming Static Images Using Generative Models for Video Salient Object Detection

• Authors: Suhwan Cho, Minhyeok Lee, Jungho Lee, Sangyoun Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13975
• Pdf URL: https://arxiv.org/pdf/2411.13975
• Copy Paste: [[2411.13975]] Transforming Static Images Using Generative Models for Video Salient Object Detection(https://arxiv.org/abs/2411.13975)
• Keywords: generative
• Abstract: In many video processing tasks, leveraging large-scale image datasets is a common strategy, as image data is more abundant and facilitates comprehensive knowledge transfer. A typical approach for simulating video from static images involves applying spatial transformations, such as affine transformations and spline warping, to create sequences that mimic temporal progression. However, in tasks like video salient object detection, where both appearance and motion cues are critical, these basic image-to-video techniques fail to produce realistic optical flows that capture the independent motion properties of each object. In this study, we show that image-to-video diffusion models can generate realistic transformations of static images while understanding the contextual relationships between image components. This ability allows the model to generate plausible optical flows, preserving semantic integrity while reflecting the independent motion of scene elements. By augmenting individual images in this way, we create large-scale image-flow pairs that significantly enhance model training. Our approach achieves state-of-the-art performance across all public benchmark datasets, outperforming existing approaches.
• 摘要：在许多视频处理任务中，利用大规模图像数据集是一种常见策略，因为图像数据更丰富，有助于全面的知识转移。从静态图像模拟视频的典型方法涉及应用空间变换（例如仿射变换和样条扭曲）来创建模拟时间进程的序列。然而，在诸如视频显著物体检测之类的任务中，外观和运动线索都至关重要，这些基本的图像到视频技术无法产生捕捉每个物体独立运动属性的逼真光流。在本研究中，我们表明，图像到视频扩散模型可以生成静态图像的逼真变换，同时理解图像组件之间的上下文关系。这种能力使模型能够生成合理的光流，在反映场景元素独立运动的同时保持语义完整性。通过以这种方式增强单个图像，我们创建了大规模图像流对，从而显著增强了模型训练。我们的方法在所有公共基准数据集上都实现了最先进的性能，优于现有方法。

Title: On the Fairness, Diversity and Reliability of Text-to-Image Generative Models

• Authors: Jordan Vice, Naveed Akhtar, Richard Hartley, Ajmal Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13981
• Pdf URL: https://arxiv.org/pdf/2411.13981
• Copy Paste: [[2411.13981]] On the Fairness, Diversity and Reliability of Text-to-Image Generative Models(https://arxiv.org/abs/2411.13981)
• Keywords: generative
• Abstract: The widespread availability of multimodal generative models has sparked critical discussions on their fairness, reliability, and potential for misuse. While text-to-image models can produce high-fidelity, user-guided images, they also exhibit unpredictable behavior and vulnerabilities, which can be exploited to manipulate class or concept representations. To address this, we propose an evaluation framework designed to assess model reliability through their responses to globally- and locally-applied `semantic' perturbations in the embedding space, pinpointing inputs that trigger unreliable behavior. Our approach offers deeper insights into two essential aspects: (i) generative diversity, evaluating the breadth of visual representations for learned concepts, and (ii) generative fairness, examining how removing concepts from input prompts affects semantic guidance. Beyond these evaluations, our method lays the groundwork for detecting unreliable, bias-injected models and retrieval of bias provenance. We will release our code. Keywords: Fairness, Reliability, AI Ethics, Bias, Text-to-Image Models
• 摘要：多模态生成模型的广泛应用引发了对其公平性、可靠性和滥用可能性的批判性讨论。虽然文本到图像模型可以生成高保真、用户引导的图像，但它们也表现出不可预测的行为和漏洞，可以利用这些漏洞来操纵类或概念表示。为了解决这个问题，我们提出了一个评估框架，旨在通过模型对嵌入空间中全局和局部应用的“语义”扰动的响应来评估模型的可靠性，从而精确定位触发不可靠行为的输入。我们的方法提供了对两个基本方面的更深入的见解：（i）生成多样性，评估学习概念的视觉表示的广度，以及（ii）生成公平性，研究从输入提示中删除概念如何影响语义指导。除了这些评估之外，我们的方法还为检测不可靠的、注入偏见的模型和检索偏见来源奠定了基础。我们将发布我们的代码。关键词：公平性、可靠性、人工智能伦理、偏见、文本转图像模型

Title: Safety Without Semantic Disruptions: Editing-free Safe Image Generation via Context-preserving Dual Latent Reconstruction

• Authors: Jordan Vice, Naveed Akhtar, Richard Hartley, Ajmal Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13982
• Pdf URL: https://arxiv.org/pdf/2411.13982
• Copy Paste: [[2411.13982]] Safety Without Semantic Disruptions: Editing-free Safe Image Generation via Context-preserving Dual Latent Reconstruction(https://arxiv.org/abs/2411.13982)
• Keywords: generation, generative
• Abstract: Training multimodal generative models on large, uncurated datasets can result in users being exposed to harmful, unsafe and controversial or culturally-inappropriate outputs. While model editing has been proposed to remove or filter undesirable concepts in embedding and latent spaces, it can inadvertently damage learned manifolds, distorting concepts in close semantic proximity. We identify limitations in current model editing techniques, showing that even benign, proximal concepts may become misaligned. To address the need for safe content generation, we propose a modular, dynamic solution that leverages safety-context embeddings and a dual reconstruction process using tunable weighted summation in the latent space to generate safer images. Our method preserves global context without compromising the structural integrity of the learned manifolds. We achieve state-of-the-art results on safe image generation benchmarks, while offering controllable variation of model safety. We identify trade-offs between safety and censorship, which presents a necessary perspective in the development of ethical AI models. We will release our code. Keywords: Text-to-Image Models, Generative AI, Safety, Reliability, Model Editing
• 摘要：在大型、未经整理的数据集上训练多模态生成模型可能会导致用户接触到有害、不安全、有争议或文化上不适当的输出。虽然已经提出了模型编辑来删除或过滤嵌入和潜在空间中不良概念，但它可能会无意中损坏学习到的流形，扭曲语义接近的概念。我们发现了当前模型编辑技术的局限性，表明即使是良性的近端概念也可能出现错位。为了满足安全内容生成的需求，我们提出了一种模块化、动态的解决方案，该解决方案利用安全上下文嵌入和使用潜在空间中可调加权求和的双重重建过程来生成更安全的图像。我们的方法保留了全局上下文，而不会损害学习到的流形的结构完整性。我们在安全图像生成基准上取得了最先进的结果，同时提供了可控的模型安全性变化。我们确定了安全性和审查制度之间的权衡，这为道德 AI 模型的开发提供了必要的视角。我们将发布我们的代码。关键词：文本转图像模型、生成式人工智能、安全性、可靠性、模型编辑

Title: Generative Intervention Models for Causal Perturbation Modeling

• Authors: Nora Schneider, Lars Lorch, Niki Kilbertus, Bernhard Schölkopf, Andreas Krause
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.14003
• Pdf URL: https://arxiv.org/pdf/2411.14003
• Copy Paste: [[2411.14003]] Generative Intervention Models for Causal Perturbation Modeling(https://arxiv.org/abs/2411.14003)
• Keywords: generative
• Abstract: We consider the problem of predicting perturbation effects via causal models. In many applications, it is a priori unknown which mechanisms of a system are modified by an external perturbation, even though the features of the perturbation are available. For example, in genomics, some properties of a drug may be known, but not their causal effects on the regulatory pathways of cells. We propose a generative intervention model (GIM) that learns to map these perturbation features to distributions over atomic interventions in a jointly-estimated causal model. Contrary to prior approaches, this enables us to predict the distribution shifts of unseen perturbation features while gaining insights about their mechanistic effects in the underlying data-generating process. On synthetic data and scRNA-seq drug perturbation data, GIMs achieve robust out-of-distribution predictions on par with unstructured approaches, while effectively inferring the underlying perturbation mechanisms, often better than other causal inference methods.
• 摘要：我们考虑通过因果模型预测扰动效应的问题。在许多应用中，即使扰动的特征可用，也无法先验地知道系统的哪些机制会受到外部扰动的影响。例如，在基因组学中，药物的某些特性可能是已知的，但它们对细胞调节途径的因果影响却未知。我们提出了一种生成干预模型 (GIM)，该模型学习将这些扰动特征映射到联合估计的因果模型中原子干预的分布上。与之前的方法相反，这使我们能够预测看不见的扰动特征的分布变化，同时深入了解它们在底层数据生成过程中的机制效应。在合成数据和 scRNA-seq 药物扰动数据上，GIM 实现了与非结构化方法相当的稳健的分布外预测，同时有效地推断出底层的扰动机制，通常比其他因果推理方法更好。

Title: MMGenBench: Evaluating the Limits of LMMs from the Text-to-Image Generation Perspective

• Authors: Hailang Huang, Yong Wang, Zixuan Huang, Huaqiu Li, Tongwen Huang, Xiangxiang Chu, Richong Zhang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14062
• Pdf URL: https://arxiv.org/pdf/2411.14062
• Copy Paste: [[2411.14062]] MMGenBench: Evaluating the Limits of LMMs from the Text-to-Image Generation Perspective(https://arxiv.org/abs/2411.14062)
• Keywords: generation, generative
• Abstract: Large Multimodal Models (LMMs) have demonstrated remarkable capabilities. While existing benchmarks for evaluating LMMs mainly focus on image comprehension, few works evaluate them from the image generation perspective. To address this issue, we propose a straightforward automated evaluation pipeline. Specifically, this pipeline requires LMMs to generate an image-prompt from a given input image. Subsequently, it employs text-to-image generative models to create a new image based on these generated prompts. Finally, we evaluate the performance of LMMs by comparing the original image with the generated one. Furthermore, we introduce MMGenBench-Test, a comprehensive benchmark developed to evaluate LMMs across 13 distinct image patterns, and MMGenBench-Domain, targeting the performance evaluation of LMMs within the generative image domain. A thorough evaluation involving over 50 popular LMMs demonstrates the effectiveness and reliability in both the pipeline and benchmark. Our observations indicate that numerous LMMs excelling in existing benchmarks fail to adequately complete the basic tasks, related to image understanding and description. This finding highlights the substantial potential for performance improvement in current LMMs and suggests avenues for future model optimization. Concurrently, our pipeline facilitates the efficient assessment of LMMs performance across diverse domains by using solely image inputs.
• 摘要：大型多模态模型 (LMM) 已展现出卓越的能力。虽然现有的 LMM 评估基准主要侧重于图像理解，但很少有研究从图像生成的角度对其进行评估。为了解决这个问题，我们提出了一种简单的自动评估流程。具体来说，该流程要求 LMM 从给定的输入图像生成图像提示。随后，它采用文本到图像的生成模型根据这些生成的提示创建新图像。最后，我们通过将原始图像与生成的图像进行比较来评估 LMM 的性能。此外，我们引入了 MMGenBench-Test，这是一种全面的基准，旨在评估 13 种不同图像模式中的 LMM，以及 MMGenBench-Domain，旨在评估生成图像域中 LMM 的性能。对 50 多个流行 LMM 的全面评估证明了流程和基准的有效性和可靠性。我们的观察表明，许多在现有基准中表现出色的 LMM 未能充分完成与图像理解和描述相关的基本任务。这一发现凸显了当前 LMM 性能改进的巨大潜力，并为未来的模型优化指明了方向。同时，我们的流程仅使用图像输入即可高效评估不同领域的 LMM 性能。

Title: Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model

• Authors: Ege Kesim, Selahattin Serdar Helli
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14064
• Pdf URL: https://arxiv.org/pdf/2411.14064
• Copy Paste: [[2411.14064]] Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model(https://arxiv.org/abs/2411.14064)
• Keywords: generative
• Abstract: Parameter efficient finetuning (PEFT) methods are widely used in LLMs and generative models in computer vision. Especially one can use multiple of these during inference to change the behavior of the base model. In this paper we investigated whether multiple LoRA adapters trained on computer vision tasks can be merged together and used during inference without loss in performance. By achieving this, multitask models can be created just by merging different LoRAs. Merging these will reduce inference time and it will not require any additional retraining. We have trained adapters on six different tasks and evaluated their performance when they are merged together. For comparison we used a model with a frozen backbone and finetuned its head. Our results show that even with simple merging techniques creating a multitask model by merging adapters is achievable by slightly loosing performance in some cases. In our experiments we merged up to three adapters together. Depending on the task and the similarity of the data adapters were trained on, merges can outperform head finetuning. We have observed that LoRAs trained with dissimilar datasets tend to perform better compared to model trained on similar datasets.
• 摘要：参数高效微调 (PEFT) 方法广泛应用于计算机视觉中的 LLM 和生成模型。特别是在推理过程中，可以使用多个此类方法改变基础模型的行为。在本文中，我们研究了是否可以将经过计算机视觉任务训练的多个 LoRA 适配器合并在一起并在推理过程中使用而不会降低性能。通过实现这一点，只需合并不同的 LoRA 即可创建多任务模型。合并这些适配器将减少推理时间，并且不需要任何额外的再训练。我们已经在六个不同的任务上训练了适配器，并评估了它们合并在一起时的性能。为了进行比较，我们使用了一个主干冻结的模型并对其头部进行了微调。我们的结果表明，即使使用简单的合并技术，通过合并适配器创建多任务模型也可以实现，但在某些情况下性能会略有下降。在我们的实验中，我们最多合并了三个适配器。根据任务和训练适配器的数据的相似性，合并的效果可以优于头部微调。我们观察到，使用不同数据集训练的 LoRA 往往比在相似数据集上训练的模型表现更好。

Title: Point Cloud Resampling with Learnable Heat Diffusion

• Authors: Wenqiang Xu, Wenrui Dai, Duoduo Xue, Ziyang Zheng, Chenglin Li, Junni Zou, Hongkai Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14120
• Pdf URL: https://arxiv.org/pdf/2411.14120
• Copy Paste: [[2411.14120]] Point Cloud Resampling with Learnable Heat Diffusion(https://arxiv.org/abs/2411.14120)
• Keywords: generative
• Abstract: Generative diffusion models have shown empirical successes in point cloud resampling, generating a denser and more uniform distribution of points from sparse or noisy 3D point clouds by progressively refining noise into structure. However, existing diffusion models employ manually predefined schemes, which often fail to recover the underlying point cloud structure due to the rigid and disruptive nature of the geometric degradation. To address this issue, we propose a novel learnable heat diffusion framework for point cloud resampling, which directly parameterizes the marginal distribution for the forward process by learning the adaptive heat diffusion schedules and local filtering scales of the time-varying heat kernel, and consequently, generates an adaptive conditional prior for the reverse process. Unlike previous diffusion models with a fixed prior, the adaptive conditional prior selectively preserves geometric features of the point cloud by minimizing a refined variational lower bound, guiding the points to evolve towards the underlying surface during the reverse process. Extensive experimental results demonstrate that the proposed point cloud resampling achieves state-of-the-art performance in representative reconstruction tasks including point cloud denoising and upsampling.
• 摘要：生成式扩散模型在点云重采样方面已显示出经验上的成功，通过逐步将噪声细化为结构，从稀疏或嘈杂的 3D 点云中生成更密集、更均匀的点分布。然而，现有的扩散模型采用手动预定义的方案，由于几何退化的刚性和破坏性，这些方案通常无法恢复底层点云结构。为了解决这个问题，我们提出了一种用于点云重采样的新型可学习热扩散框架，它通过学习时变热核的自适应热扩散时间表和局部滤波尺度，直接参数化前向过程的边际分布，从而为反向过程生成自适应条件先验。与以前具有固定先验的扩散模型不同，自适应条件先验通过最小化细化的变分下限来选择性地保留点云的几何特征，从而引导点在反向过程中向底层表面演化。大量实验结果表明，所提出的点云重采样在点云去噪和上采样等代表性重建任务中实现了最先进的性能。

Title: RestorerID: Towards Tuning-Free Face Restoration with ID Preservation

• Authors: Jiacheng Ying, Mushui Liu, Zhe Wu, Runming Zhang, Zhu Yu, Siming Fu, Si-Yuan Cao, Chao Wu, Yunlong Yu, Hui-Liang Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14125
• Pdf URL: https://arxiv.org/pdf/2411.14125
• Copy Paste: [[2411.14125]] RestorerID: Towards Tuning-Free Face Restoration with ID Preservation(https://arxiv.org/abs/2411.14125)
• Keywords: restoration
• Abstract: Blind face restoration has made great progress in producing high-quality and lifelike images. Yet it remains challenging to preserve the ID information especially when the degradation is heavy. Current reference-guided face restoration approaches either require face alignment or personalized test-tuning, which are unfaithful or time-consuming. In this paper, we propose a tuning-free method named RestorerID that incorporates ID preservation during face restoration. RestorerID is a diffusion model-based method that restores low-quality images with varying levels of degradation by using a single reference image. To achieve this, we propose a unified framework to combine the ID injection with the base blind face restoration model. In addition, we design a novel Face ID Rebalancing Adapter (FIR-Adapter) to tackle the problems of content unconsistency and contours misalignment that are caused by information conflicts between the low-quality input and reference image. Furthermore, by employing an Adaptive ID-Scale Adjusting strategy, RestorerID can produce superior restored images across various levels of degradation. Experimental results on the Celeb-Ref dataset and real-world scenarios demonstrate that RestorerID effectively delivers high-quality face restoration with ID preservation, achieving a superior performance compared to the test-tuning approaches and other reference-guided ones. The code of RestorerID is available at \url{this https URL}.
• 摘要：盲人脸修复在生成高质量逼真图像方面取得了巨大进展。然而，保留 ID 信息仍然具有挑战性，尤其是在退化严重的情况下。当前的参考引导人脸修复方法要么需要人脸对齐，要么需要个性化测试调整，这些方法要么不真实，要么耗时。在本文中，我们提出了一种无需调整的方法 RestorerID，该方法在人脸修复过程中结合了 ID 保留。RestorerID 是一种基于扩散模型的方法，它使用单个参考图恢复具有不同退化程度的低质量图像。为了实现这一点，我们提出了一个统一的框架，将 ID 注入与基本盲人脸修复模型相结合。此外，我们设计了一种新颖的人脸 ID 再平衡适配器 (FIR-Adapter) 来解决由低质量输入和参考图像之间的信息冲突引起的内容不一致和轮廓错位的问题。此外，通过采用自适应 ID 尺度调整策略，RestorerID 可以在各种退化程度下生成出色的恢复图像。在 Celeb-Ref 数据集和真实场景中的实验结果表明，RestorerID 能够有效地提供高质量的人脸恢复，同时保留 ID，与测试调整方法和其他参考指导方法相比，其性能更佳。RestorerID 的代码可在 \url{此 https URL} 处获取。

Title: GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs

• Authors: Advik Raj Basani, Xiao Zhang
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14133
• Pdf URL: https://arxiv.org/pdf/2411.14133
• Copy Paste: [[2411.14133]] GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs(https://arxiv.org/abs/2411.14133)
• Keywords: generation, generative
• Abstract: Large Language Models (LLMs) have shown impressive proficiency across a range of natural language processing tasks yet remain vulnerable to adversarial prompts, known as jailbreak attacks, carefully designed to elicit harmful responses from LLMs. Traditional methods rely on manual heuristics, which suffer from limited generalizability. While being automatic, optimization-based attacks often produce unnatural jailbreak prompts that are easy to detect by safety filters or require high computational overhead due to discrete token optimization. Witnessing the limitations of existing jailbreak methods, we introduce Generative Adversarial Suffix Prompter (GASP), a novel framework that combines human-readable prompt generation with Latent Bayesian Optimization (LBO) to improve adversarial suffix creation in a fully black-box setting. GASP leverages LBO to craft adversarial suffixes by efficiently exploring continuous embedding spaces, gradually optimizing the model to improve attack efficacy while balancing prompt coherence through a targeted iterative refinement procedure. Our experiments show that GASP can generate natural jailbreak prompts, significantly improving attack success rates, reducing training times, and accelerating inference speed, thus making it an efficient and scalable solution for red-teaming LLMs.
• 摘要：大型语言模型 (LLM) 在一系列自然语言处理任务中表现出令人印象深刻的熟练程度，但仍然容易受到对抗性提示（称为越狱攻击）的攻击，这些提示经过精心设计，旨在从 LLM 中引发有害响应。传统方法依赖于手动启发式方法，其局限性在于通用性有限。基于优化的攻击虽然是自动的，但通常会产生不自然的越狱提示，这些提示很容易被安全过滤器检测到，或者由于离散令牌优化而需要高计算开销。鉴于现有越狱方法的局限性，我们引入了生成对抗性后缀提示器 (GASP)，这是一种新颖的框架，它将人类可读的提示生成与潜在贝叶斯优化 (LBO) 相结合，以在完全黑盒环境中改进对抗性后缀的创建。GASP 利用 LBO 来制作对抗性后缀，通过有效探索连续嵌入空间，逐步优化模型以提高攻击效率，同时通过有针对性的迭代细化程序平衡提示连贯性。我们的实验表明，GASP 可以生成自然的越狱提示，显著提高攻击成功率，减少训练时间，加快推理速度，从而使其成为红队 LLM 的高效且可扩展的解决方案。

Title: ComfyGI: Automatic Improvement of Image Generation Workflows

• Authors: Dominik Sobania, Martin Briesch, Franz Rothlauf
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2411.14193
• Pdf URL: https://arxiv.org/pdf/2411.14193
• Copy Paste: [[2411.14193]] ComfyGI: Automatic Improvement of Image Generation Workflows(https://arxiv.org/abs/2411.14193)
• Keywords: generation
• Abstract: Automatic image generation is no longer just of interest to researchers, but also to practitioners. However, current models are sensitive to the settings used and automatic optimization methods often require human involvement. To bridge this gap, we introduce ComfyGI, a novel approach to automatically improve workflows for image generation without the need for human intervention driven by techniques from genetic improvement. This enables image generation with significantly higher quality in terms of the alignment with the given description and the perceived aesthetics. On the performance side, we find that overall, the images generated with an optimized workflow are about 50% better compared to the initial workflow in terms of the median ImageReward score. These already good results are even surpassed in our human evaluation, as the participants preferred the images improved by ComfyGI in around 90% of the cases.
• 摘要：自动图像生成不再只是研究人员的兴趣所在，从业者也对此很感兴趣。然而，当前的模型对所使用的设置很敏感，而自动优化方法通常需要人工参与。为了弥补这一差距，我们引入了 ComfyGI，这是一种新颖的方法，可自动改进图像生成工作流程，而无需人工干预，由遗传改良技术驱动。这使得生成的图像在与给定描述的对齐和感知美感方面具有显著更高的质量。在性能方面，我们发现，总体而言，就 ImageReward 得分中位数而言，使用优化工作流程生成的图像比初始工作流程好约 50%。这些已经很好的结果甚至在我们的人工评估中得到了超越，因为参与者在约 90% 的情况下更喜欢通过 ComfyGI 改进的图像。

Title: Novel View Extrapolation with Video Diffusion Priors

• Authors: Kunhao Liu, Ling Shao, Shijian Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14208
• Pdf URL: https://arxiv.org/pdf/2411.14208
• Copy Paste: [[2411.14208]] Novel View Extrapolation with Video Diffusion Priors(https://arxiv.org/abs/2411.14208)
• Keywords: generative
• Abstract: The field of novel view synthesis has made significant strides thanks to the development of radiance field methods. However, most radiance field techniques are far better at novel view interpolation than novel view extrapolation where the synthesis novel views are far beyond the observed training views. We design ViewExtrapolator, a novel view synthesis approach that leverages the generative priors of Stable Video Diffusion (SVD) for realistic novel view extrapolation. By redesigning the SVD denoising process, ViewExtrapolator refines the artifact-prone views rendered by radiance fields, greatly enhancing the clarity and realism of the synthesized novel views. ViewExtrapolator is a generic novel view extrapolator that can work with different types of 3D rendering such as views rendered from point clouds when only a single view or monocular video is available. Additionally, ViewExtrapolator requires no fine-tuning of SVD, making it both data-efficient and computation-efficient. Extensive experiments demonstrate the superiority of ViewExtrapolator in novel view extrapolation. Project page: \url{this https URL}.
• 摘要：由于辐射场方法的发展，新视图合成领域取得了重大进展。然而，大多数辐射场技术在新视图插值方面远胜于新视图外推，因为新视图外推合成的新视图远远超出了观察到的训练视图。我们设计了 ViewExtrapolator，这是一种新视图合成方法，它利用稳定视频扩散 (SVD) 的生成先验来实现逼真的新视图外推。通过重新设计 SVD 去噪过程，ViewExtrapolator 改进了辐射场渲染的容易出现伪影的视图，大大提高了合成新视图的清晰度和真实感。ViewExtrapolator 是一种通用的新视图外推器，可以处理不同类型的 3D 渲染，例如当只有单个视图或单目视频可用时从点云渲染的视图。此外，ViewExtrapolator 不需要对 SVD 进行微调，使其既节省数据又节省计算。大量实验证明了 ViewExtrapolator 在新视图外推方面的优势。项目页面：\url{此 https URL}。

Title: Generative Outpainting To Enhance the Memorability of Short-Form Videos

• Authors: Alan Byju, Aman Sudhindra Ladwa, Lorin Sweeney, Alan F. Smeaton
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14213
• Pdf URL: https://arxiv.org/pdf/2411.14213
• Copy Paste: [[2411.14213]] Generative Outpainting To Enhance the Memorability of Short-Form Videos(https://arxiv.org/abs/2411.14213)
• Keywords: generative
• Abstract: With the expanding use of the short-form video format in advertising, social media, entertainment, education and more, there is a need for such media to both captivate and be remembered. Video memorability indicates to us how likely a video is to be remembered by a viewer who has no emotional or personal connection with its content. This paper presents the results of using generative outpainting to expand the screen size of a short-form video with a view to improving its memorability. Advances in machine learning and deep learning are compared and leveraged to understand how extending the borders of video screensizes can affect their memorability to viewers. Using quantitative evaluation we determine the best-performing model for outpainting and the impact of outpainting based on image saliency on video memorability scores
• 摘要：随着短视频格式在广告、社交媒体、娱乐、教育等领域的应用越来越广泛，这种媒体既需要吸引人，又需要被记住。视频记忆性向我们表明，一个与视频内容没有情感或个人联系的观众记住视频的可能性有多大。本文介绍了使用生成式覆盖绘画扩大短视频屏幕尺寸以提高其记忆性的结果。本文比较并利用机器学习和深度学习的进展来了解扩大视频屏幕尺寸的边界如何影响观众的记忆性。通过定量评估，我们确定了效果最佳的覆盖绘画模型，以及基于图像显著性的覆盖绘画对视频记忆性分数的影响

Title: Towards Context-Rich Automated Biodiversity Assessments: Deriving AI-Powered Insights from Camera Trap Data

• Authors: Paul Fergus, Carl Chalmers, Naomi Matthews, Stuart Nixon, Andre Burger, Oliver Hartley, Chris Sutherland, Xavier Lambin, Steven Longmore, Serge Wich
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14219
• Pdf URL: https://arxiv.org/pdf/2411.14219
• Copy Paste: [[2411.14219]] Towards Context-Rich Automated Biodiversity Assessments: Deriving AI-Powered Insights from Camera Trap Data(https://arxiv.org/abs/2411.14219)
• Keywords: generation
• Abstract: Camera traps offer enormous new opportunities in ecological studies, but current automated image analysis methods often lack the contextual richness needed to support impactful conservation outcomes. Here we present an integrated approach that combines deep learning-based vision and language models to improve ecological reporting using data from camera traps. We introduce a two-stage system: YOLOv10-X to localise and classify species (mammals and birds) within images, and a Phi-3.5-vision-instruct model to read YOLOv10-X binding box labels to identify species, overcoming its limitation with hard to classify objects in images. Additionally, Phi-3.5 detects broader variables, such as vegetation type, and time of day, providing rich ecological and environmental context to YOLO's species detection output. When combined, this output is processed by the model's natural language system to answer complex queries, and retrieval-augmented generation (RAG) is employed to enrich responses with external information, like species weight and IUCN status (information that cannot be obtained through direct visual analysis). This information is used to automatically generate structured reports, providing biodiversity stakeholders with deeper insights into, for example, species abundance, distribution, animal behaviour, and habitat selection. Our approach delivers contextually rich narratives that aid in wildlife management decisions. By providing contextually rich insights, our approach not only reduces manual effort but also supports timely decision-making in conservation, potentially shifting efforts from reactive to proactive management.
• 摘要：相机陷阱为生态研究提供了巨大的新机遇，但目前的自动图像分析方法往往缺乏支持有影响力的保护成果所需的丰富背景。在这里，我们提出了一种综合方法，结合基于深度学习的视觉和语言模型，使用来自相机陷阱的数据改进生态报告。我们引入了一个两阶段系统：YOLOv10-X 用于定位和分类图像中的物种（哺乳动物和鸟类），以及 Phi-3.5-vision-instruct 模型，用于读取 YOLOv10-X 绑定框标签以识别物种，从而克服了其难以对图像中的物体进行分类的局限性。此外，Phi-3.5 可以检测更广泛的变量，例如植被类型和一天中的时间，为 YOLO 的物种检测输出提供丰富的生态和环境背景。组合后，此输出由模型的自然语言系统处理以回答复杂的查询，并使用检索增强生成 (RAG) 来丰富响应，其中包含外部信息，例如物种重量和 IUCN 状态（无法通过直接视觉分析获得的信息）。这些信息用于自动生成结构化报告，为生物多样性利益相关者提供更深入的见解，例如物种丰富度、分布、动物行为和栖息地选择。我们的方法提供了丰富的背景叙述，有助于野生动物管理决策。通过提供丰富的背景见解，我们的方法不仅可以减少人工工作，还可以支持及时的保护决策，从而有可能将工作从被动管理转变为主动管理。

Title: BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI

• Authors: Natenaile Asmamaw Shiferaw, Simpenzwe Honore Leandre, Aman Sinha, Dillip Rout
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14254
• Pdf URL: https://arxiv.org/pdf/2411.14254
• Copy Paste: [[2411.14254]] BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI(https://arxiv.org/abs/2411.14254)
• Keywords: generation
• Abstract: Course Outcome (CO) and Program Outcome (PO)/Program-Specific Outcome (PSO) alignment is a crucial task for ensuring curriculum coherence and assessing educational effectiveness. The construction of a Course Articulation Matrix (CAM), which quantifies the relationship between COs and POs/PSOs, typically involves assigning numerical values (0, 1, 2, 3) to represent the degree of alignment. In this study, We experiment with four models from the BERT family: BERT Base, DistilBERT, ALBERT, and RoBERTa, and use multiclass classification to assess the alignment between CO and PO/PSO pairs. We first evaluate traditional machine learning classifiers, such as Decision Tree, Random Forest, and XGBoost, and then apply transfer learning to evaluate the performance of the pretrained BERT models. To enhance model interpretability, we apply Explainable AI technique, specifically Local Interpretable Model-agnostic Explanations (LIME), to provide transparency into the decision-making process. Our system achieves accuracy, precision, recall, and F1-score values of 98.66%, 98.67%, 98.66%, and 98.66%, respectively. This work demonstrates the potential of utilizing transfer learning with BERT-based models for the automated generation of CAMs, offering high performance and interpretability in educational outcome assessment.
• 摘要：课程成果 (CO) 和项目成果 (PO)/项目特定成果 (PSO) 的一致性对于确保课程连贯性和评估教育效果至关重要。课程衔接矩阵 (CAM) 的构建可以量化 CO 和 PO/PSO 之间的关系，通常涉及分配数值 (0、1、2、3) 来表示一致性程度。在本研究中，我们尝试了 BERT 系列中的四种模型：BERT Base、DistilBERT、ALBERT 和 RoBERTa，并使用多类分类来评估 CO 和 PO/PSO 对之间的一致性。我们首先评估传统的机器学习分类器，例如决策树、随机森林和 XGBoost，然后应用迁移学习来评估预训练的 BERT 模型的性能。为了增强模型的可解释性，我们应用可解释的 AI 技术，特别是局部可解释的模型不可知解释 (LIME)，为决策过程提供透明度。我们的系统分别实现了 98.66%、98.67%、98.66%、98.66% 和 98.66% 的准确率、精确率、召回率和 F1 值。这项工作展示了利用基于 BERT 的模型进行迁移学习以自动生成 CAM 的潜力，从而在教育成果评估中提供高性能和可解释性。

Title: Generating Realistic Adversarial Examples for Business Processes using Variational Autoencoders

• Authors: Alexander Stevens, Jari Peeperkorn, Johannes De Smedt, Jochen De Weerdt
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14263
• Pdf URL: https://arxiv.org/pdf/2411.14263
• Copy Paste: [[2411.14263]] Generating Realistic Adversarial Examples for Business Processes using Variational Autoencoders(https://arxiv.org/abs/2411.14263)
• Keywords: generation
• Abstract: In predictive process monitoring, predictive models are vulnerable to adversarial attacks, where input perturbations can lead to incorrect predictions. Unlike in computer vision, where these perturbations are designed to be imperceptible to the human eye, the generation of adversarial examples in predictive process monitoring poses unique challenges. Minor changes to the activity sequences can create improbable or even impossible scenarios to occur due to underlying constraints such as regulatory rules or process constraints. To address this, we focus on generating realistic adversarial examples tailored to the business process context, in contrast to the imperceptible, pixel-level changes commonly seen in computer vision adversarial attacks. This paper introduces two novel latent space attacks, which generate adversaries by adding noise to the latent space representation of the input data, rather than directly modifying the input attributes. These latent space methods are domain-agnostic and do not rely on process-specific knowledge, as we restrict the generation of adversarial examples to the learned class-specific data distributions by directly perturbing the latent space representation of the business process executions. We evaluate these two latent space methods with six other adversarial attacking methods on eleven real-life event logs and four predictive models. The first three attacking methods directly permute the activities of the historically observed business process executions. The fourth method constrains the adversarial examples to lie within the same data distribution as the original instances, by projecting the adversarial examples to the original data distribution.
• 摘要：在预测过程监控中，预测模型容易受到对抗性攻击，输入扰动可能导致错误预测。与计算机视觉中这些扰动被设计为人眼无法察觉的不同，预测过程监控中对抗性示例的生成带来了独特的挑战。由于监管规则或流程约束等潜在约束，活动序列的微小变化可能会导致不太可能甚至不可能发生的场景。为了解决这个问题，我们专注于生成针对业务流程环境量身定制的逼真的对抗性示例，而不是计算机视觉对抗性攻击中常见的难以察觉的像素级变化。本文介绍了两种新颖的潜在空间攻击，它们通过向输入数据的潜在空间表示添加噪声而不是直接修改输入属性来生成对手。这些潜在空间方法与领域无关，不依赖于特定于流程的知识，因为我们通过直接扰动业务流程执行的潜在空间表示，将对抗性示例的生成限制在学习到的特定于类的数据分布中。我们在 11 个真实事件日志和 4 个预测模型上评估了这两种潜在空间方法以及其他六种对抗性攻击方法。前三种攻击方法直接改变历史观察到的业务流程执行活动。第四种方法通过将对抗性示例投影到原始数据分布，将对抗性示例限制在与原始实例相同的数据分布中。

Title: StereoCrafter-Zero: Zero-Shot Stereo Video Generation with Noisy Restart

• Authors: Jian Shi, Qian Wang, Zhenyu Li, Peter Wonka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14295
• Pdf URL: https://arxiv.org/pdf/2411.14295
• Copy Paste: [[2411.14295]] StereoCrafter-Zero: Zero-Shot Stereo Video Generation with Noisy Restart(https://arxiv.org/abs/2411.14295)
• Keywords: generation
• Abstract: Generating high-quality stereo videos that mimic human binocular vision requires maintaining consistent depth perception and temporal coherence across frames. While diffusion models have advanced image and video synthesis, generating high-quality stereo videos remains challenging due to the difficulty of maintaining consistent temporal and spatial coherence between left and right views. We introduce \textit{StereoCrafter-Zero}, a novel framework for zero-shot stereo video generation that leverages video diffusion priors without the need for paired training data. Key innovations include a noisy restart strategy to initialize stereo-aware latents and an iterative refinement process that progressively harmonizes the latent space, addressing issues like temporal flickering and view inconsistencies. Comprehensive evaluations, including quantitative metrics and user studies, demonstrate that \textit{StereoCrafter-Zero} produces high-quality stereo videos with improved depth consistency and temporal smoothness, even when depth estimations are imperfect. Our framework is robust and adaptable across various diffusion models, setting a new benchmark for zero-shot stereo video generation and enabling more immersive visual experiences. Our code can be found in~\url{this https URL}.
• 摘要：生成模拟人类双目视觉的高质量立体视频需要保持一致的深度感知和跨帧的时间连贯性。虽然扩散模型具有先进的图像和视频合成，但由于难以在左视图和右视图之间保持一致的时间和空间连贯性，因此生成高质量立体视频仍然具有挑战性。我们引入了 \textit{StereoCrafter-Zero}，这是一种用于零镜头立体视频生成的新框架，它利用视频扩散先验，而无需配对训练数据。关键创新包括用于初始化立体感知潜伏的噪声重启策略和逐步协调潜伏空间的迭代细化过程，解决了时间闪烁和视图不一致等问题。综合评估（包括定量指标和用户研究）表明，即使在深度估计不完美的情况下，\textit{StereoCrafter-Zero} 也能生成具有改进的深度一致性和时间平滑度的高质量立体视频。我们的框架非常强大，可适应各种扩散模型，为零镜头立体视频生成树立了新标杆，并实现了更加身临其境的视觉体验。我们的代码可以在~\url{此 https URL} 中找到。

Title: Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation

• Authors: Yuanhao Cai, He Zhang, Kai Zhang, Yixun Liang, Mengwei Ren, Fujun Luan, Qing Liu, Soo Ye Kim, Jianming Zhang, Zhifei Zhang, Yuqian Zhou, Zhe Lin, Alan Yuille
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.14384
• Pdf URL: https://arxiv.org/pdf/2411.14384
• Copy Paste: [[2411.14384]] Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation(https://arxiv.org/abs/2411.14384)
• Keywords: generation
• Abstract: Existing feed-forward image-to-3D methods mainly rely on 2D multi-view diffusion models that cannot guarantee 3D consistency. These methods easily collapse when changing the prompt view direction and mainly handle object-centric prompt images. In this paper, we propose a novel single-stage 3D diffusion model, DiffusionGS, for object and scene generation from a single view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to enforce view consistency and allow the model to generate robustly given prompt views of any directions, beyond object-centric inputs. Plus, to improve the capability and generalization ability of DiffusionGS, we scale up 3D training data by developing a scene-object mixed training strategy. Experiments show that our method enjoys better generation quality (2.20 dB higher in PSNR and 23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA methods. The user study and text-to-3D applications also reveals the practical values of our method. Our Project page at this https URL shows the video and interactive generation results.
• 摘要：现有的前馈图像到 3D 方法主要依赖于无法保证 3D 一致性的 2D 多视图扩散模型。这些方法在更改提示视图方向时很容易崩溃，并且主要处理以对象为中心的提示图像。在本文中，我们提出了一种新颖的单阶段 3D 扩散模型 DiffusionGS，用于从单个视图生成对象和场景。DiffusionGS 在每个时间步直接输出 3D 高斯点云以增强视图一致性，并允许模型在以对象为中心的输入之外稳健地生成任何方向的给定提示视图。此外，为了提高 DiffusionGS 的能力和泛化能力，我们通过开发场景-对象混合训练策略来扩展 3D 训练数据。实验表明，与 SOTA 方法相比，我们的方法具有更好的生成质量（PSNR 高 2.20 dB，FID 低 23.25）和 5 倍以上的速度（在 A100 GPU 上约为 6 秒）。用户研究和文本到 3D 应用也揭示了我们方法的实用价值。我们的项目页面位于此 https URL，展示了视频和交互式生成结果。

Title: Stable Flow: Vital Layers for Training-Free Image Editing

• Authors: Omri Avrahami, Or Patashnik, Ohad Fried, Egor Nemchinov, Kfir Aberman, Dani Lischinski, Daniel Cohen-Or
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14430
• Pdf URL: https://arxiv.org/pdf/2411.14430
• Copy Paste: [[2411.14430]] Stable Flow: Vital Layers for Training-Free Image Editing(https://arxiv.org/abs/2411.14430)
• Keywords: generation
• Abstract: Diffusion models have revolutionized the field of content synthesis and editing. Recent models have replaced the traditional UNet architecture with the Diffusion Transformer (DiT), and employed flow-matching for improved training and sampling. However, they exhibit limited generation diversity. In this work, we leverage this limitation to perform consistent image edits via selective injection of attention features. The main challenge is that, unlike the UNet-based models, DiT lacks a coarse-to-fine synthesis structure, making it unclear in which layers to perform the injection. Therefore, we propose an automatic method to identify "vital layers" within DiT, crucial for image formation, and demonstrate how these layers facilitate a range of controlled stable edits, from non-rigid modifications to object addition, using the same mechanism. Next, to enable real-image editing, we introduce an improved image inversion method for flow models. Finally, we evaluate our approach through qualitative and quantitative comparisons, along with a user study, and demonstrate its effectiveness across multiple applications. The project page is available at this https URL
• 摘要：扩散模型彻底改变了内容合成和编辑领域。最近的模型已经用扩散变换器 (DiT) 取代了传统的 UNet 架构，并采用流匹配来改进训练和采样。然而，它们表现出有限的生成多样性。在这项工作中，我们利用这一限制通过选择性注入注意力特征来执行一致的图像编辑。主要的挑战是，与基于 UNet 的模型不同，DiT 缺乏从粗到细的合成结构，因此不清楚在哪些层执行注入。因此，我们提出了一种自动方法来识别 DiT 中对图像形成至关重要的“重要层”，并展示这些层如何使用相同的机制促进一系列受控的稳定编辑，从非刚性修改到对象添加。接下来，为了实现真实图像编辑，我们引入了一种改进的流模型图像反转方法。最后，我们通过定性和定量比较以及用户研究来评估我们的方法，并证明其在多个应用程序中的有效性。项目页面可在此 https URL 上找到

Title: Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

• Authors: Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14432
• Pdf URL: https://arxiv.org/pdf/2411.14432
• Copy Paste: [[2411.14432]] Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models(https://arxiv.org/abs/2411.14432)
• Keywords: generation
• Abstract: Large Language Models (LLMs) demonstrate enhanced capabilities and reliability by reasoning more, evolving from Chain-of-Thought prompting to product-level solutions like OpenAI o1. Despite various efforts to improve LLM reasoning, high-quality long-chain reasoning data and optimized training pipelines still remain inadequately explored in vision-language tasks. In this paper, we present Insight-V, an early effort to 1) scalably produce long and robust reasoning data for complex multi-modal tasks, and 2) an effective training pipeline to enhance the reasoning capabilities of multi-modal large language models (MLLMs). Specifically, to create long and structured reasoning data without human labor, we design a two-step pipeline with a progressive strategy to generate sufficiently long and diverse reasoning paths and a multi-granularity assessment method to ensure data quality. We observe that directly supervising MLLMs with such long and complex reasoning data will not yield ideal reasoning ability. To tackle this problem, we design a multi-agent system consisting of a reasoning agent dedicated to performing long-chain reasoning and a summary agent trained to judge and summarize reasoning results. We further incorporate an iterative DPO algorithm to enhance the reasoning agent's generation stability and quality. Based on the popular LLaVA-NeXT model and our stronger base MLLM, we demonstrate significant performance gains across challenging multi-modal benchmarks requiring visual reasoning. Benefiting from our multi-agent system, Insight-V can also easily maintain or improve performance on perception-focused multi-modal tasks.
• 摘要：大型语言模型 (LLM) 通过更多推理展示了增强的能力和可靠性，从思路链提示发展到像 OpenAI o1 这样的产品级解决方案。尽管人们为改进 LLM 推理做出了各种努力，但在视觉语言任务中，高质量的长链推理数据和优化的训练管道仍然没有得到充分探索。在本文中，我们介绍了 Insight-V，这是一项早期努力，旨在 1) 为复杂的多模态任务可扩展地生成长而鲁棒的推理数据，以及 2) 一种有效的训练管道，以增强多模态大型语言模型 (MLLM) 的推理能力。具体而言，为了在没有人工的情况下创建长而结构化的推理数据，我们设计了一个两步管道，采用渐进策略来生成足够长且多样化的推理路径，并采用多粒度评估方法来确保数据质量。我们观察到，直接用如此长而复杂的推理数据监督 MLLM 不会产生理想的推理能力。为了解决这个问题，我们设计了一个多智能体系统，该系统由一个专门用于执行长链推理的推理智能体和一个经过训练用于判断和总结推理结果的总结智能体组成。我们进一步结合了迭代 DPO 算法来提高推理智能体的生成稳定性和质量。基于流行的 LLaVA-NeXT 模型和我们更强大的基础 MLLM，我们在需要视觉推理的具有挑战性的多模态基准测试中展示了显著的性能提升。得益于我们的多智能体系统，Insight-V 还可以轻松保持或提高以感知为中心的多模态任务的性能。