2024-08-07

Title: RCDM: Enabling Robustness for Conditional Diffusion Model

• Authors: Weifeng Xu, Xiang Zhu, Xiaoyong Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02710
• Pdf URL: https://arxiv.org/pdf/2408.02710
• Copy Paste: [[2408.02710]] RCDM: Enabling Robustness for Conditional Diffusion Model(https://arxiv.org/abs/2408.02710)
• Keywords: diffusion
• Abstract: The conditional diffusion model (CDM) enhances the standard diffusion model by providing more control, improving the quality and relevance of the outputs, and making the model adaptable to a wider range of complex tasks. However, inaccurate conditional inputs in the inverse process of CDM can easily lead to generating fixed errors in the neural network, which diminishes the adaptability of a well-trained model. The existing methods like data augmentation, adversarial training, robust optimization can improve the robustness, while they often face challenges such as high computational complexity, limited applicability to unknown perturbations, and increased training difficulty. In this paper, we propose a lightweight solution, the Robust Conditional Diffusion Model (RCDM), based on control theory to dynamically reduce the impact of noise and significantly enhance the model's robustness. RCDM leverages the collaborative interaction between two neural networks, along with optimal control strategies derived from control theory, to optimize the weights of two networks during the sampling process. Unlike conventional techniques, RCDM establishes a mathematical relationship between fixed errors and the weights of the two neural networks without incurring additional computational overhead. Extensive experiments were conducted on MNIST and CIFAR-10 datasets, and the results demonstrate the effectiveness and adaptability of our proposed model.

Title: Privacy-Safe Iris Presentation Attack Detection

• Authors: Mahsa Mitcheff, Patrick Tinsley, Adam Czajka
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.02750
• Pdf URL: https://arxiv.org/pdf/2408.02750
• Copy Paste: [[2408.02750]] Privacy-Safe Iris Presentation Attack Detection(https://arxiv.org/abs/2408.02750)
• Keywords: generative
• Abstract: This paper proposes a framework for a privacy-safe iris presentation attack detection (PAD) method, designed solely with synthetically-generated, identity-leakage-free iris images. Once trained, the method is evaluated in a classical way using state-of-the-art iris PAD benchmarks. We designed two generative models for the synthesis of ISO/IEC 19794-6-compliant iris images. The first model synthesizes bona fide-looking samples. To avoid ``identity leakage,'' the generated samples that accidentally matched those used in the model's training were excluded. The second model synthesizes images of irises with textured contact lenses and is conditioned by a given contact lens brand to have better control over textured contact lens appearance when forming the training set. Our experiments demonstrate that models trained solely on synthetic data achieve a lower but still reasonable performance when compared to solutions trained with iris images collected from human subjects. This is the first-of-its-kind attempt to use solely synthetic data to train a fully-functional iris PAD solution, and despite the performance gap between regular and the proposed methods, this study demonstrates that with the increasing fidelity of generative models, creating such privacy-safe iris PAD methods may be possible. The source codes and generative models trained for this work are offered along with the paper.

Title: Diffusion Models as Data Mining Tools

• Authors: Ioannis Siglidis, Aleksander Holynski, Alexei A. Efros, Mathieu Aubry, Shiry Ginosar
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02752
• Pdf URL: https://arxiv.org/pdf/2408.02752
• Copy Paste: [[2408.02752]] Diffusion Models as Data Mining Tools(https://arxiv.org/abs/2408.02752)
• Keywords: diffusion, generative
• Abstract: This paper demonstrates how to use generative models trained for image synthesis as tools for visual data mining. Our insight is that since contemporary generative models learn an accurate representation of their training data, we can use them to summarize the data by mining for visual patterns. Concretely, we show that after finetuning conditional diffusion models to synthesize images from a specific dataset, we can use these models to define a typicality measure on that dataset. This measure assesses how typical visual elements are for different data labels, such as geographic location, time stamps, semantic labels, or even the presence of a disease. This analysis-by-synthesis approach to data mining has two key advantages. First, it scales much better than traditional correspondence-based approaches since it does not require explicitly comparing all pairs of visual elements. Second, while most previous works on visual data mining focus on a single dataset, our approach works on diverse datasets in terms of content and scale, including a historical car dataset, a historical face dataset, a large worldwide street-view dataset, and an even larger scene dataset. Furthermore, our approach allows for translating visual elements across class labels and analyzing consistent changes.

Title: Back-Projection Diffusion: Solving the Wideband Inverse Scattering Problem with Diffusion Models

• Authors: Borong Zhang, Martín Guerra, Qin Li, Leonardo Zepeda-Núñez
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2408.02866
• Pdf URL: https://arxiv.org/pdf/2408.02866
• Copy Paste: [[2408.02866]] Back-Projection Diffusion: Solving the Wideband Inverse Scattering Problem with Diffusion Models(https://arxiv.org/abs/2408.02866)
• Keywords: diffusion
• Abstract: We present \textit{Wideband back-projection diffusion}, an end-to-end probabilistic framework for approximating the posterior distribution induced by the inverse scattering map from wideband scattering data. This framework leverages conditional diffusion models coupled with the underlying physics of wave-propagation and symmetries in the problem, to produce highly accurate reconstructions. The framework introduces a factorization of the score function into a physics-based latent representation inspired by the filtered back-propagation formula and a conditional score function conditioned on this latent representation. These two steps are also constrained to obey symmetries in the formulation while being amenable to compression by imposing the rank structure found in the filtered back-projection formula. As a result, empirically, our framework is able to provide sharp reconstructions effortlessly, even recovering sub-Nyquist features in the multiple-scattering regime. It has low-sample and computational complexity, its number of parameters scales sub-linearly with the target resolution, and it has stable training dynamics.

Title: Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learning

• Authors: Dmitri Iourovitski, Sanat Sharma, Rakshak Talwar
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02871
• Pdf URL: https://arxiv.org/pdf/2408.02871
• Copy Paste: [[2408.02871]] Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learning(https://arxiv.org/abs/2408.02871)
• Keywords: in-context
• Abstract: As content generated by Large Language Model (LLM) has grown exponentially, the ability to accurately identify and fingerprint such text has become increasingly crucial. In this work, we introduce a novel black-box approach for fingerprinting LLMs, achieving an impressive 72% accuracy in identifying the correct family of models (Such as Llama, Mistral, Gemma, etc) among a lineup of LLMs. We present an evolutionary strategy that leverages the capabilities of one LLM to discover the most salient features for identifying other LLMs. Our method employs a unique "Hide and Seek" algorithm, where an Auditor LLM generates discriminative prompts, and a Detective LLM analyzes the responses to fingerprint the target models. This approach not only demonstrates the feasibility of LLM-driven model identification but also reveals insights into the semantic manifolds of different LLM families. By iteratively refining prompts through in-context learning, our system uncovers subtle distinctions between model outputs, providing a powerful tool for LLM analysis and verification. This research opens new avenues for understanding LLM behavior and has significant implications for model attribution, security, and the broader field of AI transparency.

Title: Diverse Generation while Maintaining Semantic Coordination: A Diffusion-Based Data Augmentation Method for Object Detection

• Authors: Sen Nie, Zhuo Wang, Xinxin Wang, Kun He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02891
• Pdf URL: https://arxiv.org/pdf/2408.02891
• Copy Paste: [[2408.02891]] Diverse Generation while Maintaining Semantic Coordination: A Diffusion-Based Data Augmentation Method for Object Detection(https://arxiv.org/abs/2408.02891)
• Keywords: diffusion
• Abstract: Recent studies emphasize the crucial role of data augmentation in enhancing the performance of object detection models. However,existing methodologies often struggle to effectively harmonize dataset diversity with semantic this http URL bridge this gap, we introduce an innovative augmentation technique leveraging pre-trained conditional diffusion models to mediate this balance. Our approach encompasses the development of a Category Affinity Matrix, meticulously designed to enhance dataset diversity, and a Surrounding Region Alignment strategy, which ensures the preservation of semantic coordination in the augmented images. Extensive experimental evaluations confirm the efficacy of our method in enriching dataset diversity while seamlessly maintaining semantic coordination. Our method yields substantial average improvements of +1.4AP, +0.9AP, and +3.4AP over existing alternatives on three distinct object detection models, respectively.

Title: MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

• Authors: Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, Yuyin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02900
• Pdf URL: https://arxiv.org/pdf/2408.02900
• Copy Paste: [[2408.02900]] MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine(https://arxiv.org/abs/2408.02900)
• Keywords: foundation model
• Abstract: This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities, with multigranular annotations for more than 65 diseases. These enriched annotations encompass both global textual information, such as disease/lesion type, modality, region-specific descriptions, and inter-regional relationships, as well as detailed local annotations for regions of interest (ROIs), including bounding boxes, segmentation masks. Unlike existing approach which is limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and texual annotations (in the form of image-ROI-description triplets) without the need for any paired text descriptions. Specifically, data from over 90 different sources have been collected, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions. We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmented generation with the identified ROIs as guidance, resulting in multigranular texual descriptions. Compared to existing datasets, MedTrinity-25M provides the most enriched annotations, supporting a comprehensive range of multimodal tasks such as captioning and report generation, as well as vision-centric tasks like classification and segmentation. Pretraining on MedTrinity-25M, our model achieves state-of-the-art performance on VQA-RAD and PathVQA, surpassing both multimodal large language models and other representative SoTA approaches. This dataset can also be utilized to support large-scale pre-training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain.

Title: Self-Supervised Learning for Multi-Channel Neural Transducer

• Authors: Atsushi Kojima
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.02945
• Pdf URL: https://arxiv.org/pdf/2408.02945
• Copy Paste: [[2408.02945]] Self-Supervised Learning for Multi-Channel Neural Transducer(https://arxiv.org/abs/2408.02945)
• Keywords: self-supervised
• Abstract: Self-supervised learning, such as with the wav2vec 2.0 framework significantly improves the accuracy of end-to-end automatic speech recognition (ASR). Wav2vec 2.0 has been applied to single-channel end-to-end ASR models. In this work, we explored a self-supervised learning method for a multi-channel end-to-end ASR model based on the wav2vec 2.0 framework. As the multi-channel end-to-end ASR model, we focused on a multi-channel neural transducer. In pre-training, we compared three different methods for feature quantization to train a multi-channel conformer audio encoder: joint quantization, feature-wise quantization and channel-wise quantization. In fine-tuning, we trained the multi-channel conformer-transducer. All experiments were conducted using the far-field in-house and CHiME-4 datasets. The results of the experiments showed that feature-wise quantization was the most effective among the methods. We observed a 66% relative reduction in character error rate compared with the model without any pre-training for the far-field in-house dataset.

Title: Data-Driven Stochastic Closure Modeling via Conditional Diffusion Model and Neural Operator

• Authors: Xinghao Dong, Chuanqi Chen, Jin-Long Wu
• Subjects: cs.LG, math.DS, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2408.02965
• Pdf URL: https://arxiv.org/pdf/2408.02965
• Copy Paste: [[2408.02965]] Data-Driven Stochastic Closure Modeling via Conditional Diffusion Model and Neural Operator(https://arxiv.org/abs/2408.02965)
• Keywords: diffusion, generative
• Abstract: Closure models are widely used in simulating complex multiscale dynamical systems such as turbulence and the earth system, for which direct numerical simulation that resolves all scales is often too expensive. For those systems without a clear scale separation, deterministic and local closure models often lack enough generalization capability, which limits their performance in many real-world applications. In this work, we propose a data-driven modeling framework for constructing stochastic and non-local closure models via conditional diffusion model and neural operator. Specifically, the Fourier neural operator is incorporated into a score-based diffusion model, which serves as a data-driven stochastic closure model for complex dynamical systems governed by partial differential equations (PDEs). We also demonstrate how accelerated sampling methods can improve the efficiency of the data-driven stochastic closure model. The results show that the proposed methodology provides a systematic approach via generative machine learning techniques to construct data-driven stochastic closure models for multiscale dynamical systems with continuous spatiotemporal fields.

Title: Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond

• Authors: Jichuan Zhang, Yali Li, Xin Liu, Shengjin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02983
• Pdf URL: https://arxiv.org/pdf/2408.02983
• Copy Paste: [[2408.02983]] Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond(https://arxiv.org/abs/2408.02983)
• Keywords: diffusion, self-supervised
• Abstract: Non-exemplar class-incremental learning (NECIL) is to resist catastrophic forgetting without saving old class samples. Prior methodologies generally employ simple rules to generate features for replaying, suffering from large distribution gap between replayed features and real ones. To address the aforementioned issue, we propose a simple, yet effective \textbf{Diff}usion-based \textbf{F}eature \textbf{R}eplay (\textbf{DiffFR}) method for NECIL. First, to alleviate the limited representational capacity caused by fixing the feature extractor, we employ Siamese-based self-supervised learning for initial generalizable features. Second, we devise diffusion models to generate class-representative features highly similar to real features, which provides an effective way for exemplar-free knowledge memorization. Third, we introduce prototype calibration to direct the diffusion model's focus towards learning the distribution shapes of features, rather than the entire distribution. Extensive experiments on public datasets demonstrate significant performance gains of our DiffFR, outperforming the state-of-the-art NECIL methods by 3.0\% in average. The code will be made publicly available soon.

Title: CKNN: Cleansed k-Nearest Neighbor for Unsupervised Video Anomaly Detection

• Authors: Jihun Yi, Sungroh Yoon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03014
• Pdf URL: https://arxiv.org/pdf/2408.03014
• Copy Paste: [[2408.03014]] CKNN: Cleansed k-Nearest Neighbor for Unsupervised Video Anomaly Detection(https://arxiv.org/abs/2408.03014)
• Keywords: anomaly
• Abstract: In this paper, we address the problem of unsupervised video anomaly detection (UVAD). The task aims to detect abnormal events in test video using unlabeled videos as training data. The presence of anomalies in the training data poses a significant challenge in this task, particularly because they form clusters in the feature space. We refer to this property as the "Anomaly Cluster" issue. The condensed nature of these anomalies makes it difficult to distinguish between normal and abnormal data in the training set. Consequently, training conventional anomaly detection techniques using an unlabeled dataset often leads to sub-optimal results. To tackle this difficulty, we propose a new method called Cleansed k-Nearest Neighbor (CKNN), which explicitly filters out the Anomaly Clusters by cleansing the training dataset. Following the k-nearest neighbor algorithm in the feature space provides powerful anomaly detection capability. Although the identified Anomaly Cluster issue presents a significant challenge to applying k-nearest neighbor in UVAD, our proposed cleansing scheme effectively addresses this problem. We evaluate the proposed method on various benchmark datasets and demonstrate that CKNN outperforms the previous state-of-the-art UVAD method by up to 8.5% (from 82.0 to 89.0) in terms of AUROC. Moreover, we emphasize that the performance of the proposed method is comparable to that of the state-of-the-art method trained using anomaly-free data.

Title: Enhancing Complex Causality Extraction via Improved Subtask Interaction and Knowledge Fusion

• Authors: Jinglong Gao, Chen Lu, Xiao Ding, Zhongyang Li, Ting Liu, Bing Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03079
• Pdf URL: https://arxiv.org/pdf/2408.03079
• Copy Paste: [[2408.03079]] Enhancing Complex Causality Extraction via Improved Subtask Interaction and Knowledge Fusion(https://arxiv.org/abs/2408.03079)
• Keywords: in-context
• Abstract: Event Causality Extraction (ECE) aims at extracting causal event pairs from texts. Despite ChatGPT's recent success, fine-tuning small models remains the best approach for the ECE task. However, existing fine-tuning based ECE methods cannot address all three key challenges in ECE simultaneously: 1) Complex Causality Extraction, where multiple causal-effect pairs occur within a single sentence; 2) Subtask~ Interaction, which involves modeling the mutual dependence between the two subtasks of ECE, i.e., extracting events and identifying the causal relationship between extracted events; and 3) Knowledge Fusion, which requires effectively fusing the knowledge in two modalities, i.e., the expressive pretrained language models and the structured knowledge graphs. In this paper, we propose a unified ECE framework (UniCE to address all three issues in ECE simultaneously. Specifically, we design a subtask interaction mechanism to enable mutual interaction between the two ECE subtasks. Besides, we design a knowledge fusion mechanism to fuse knowledge in the two modalities. Furthermore, we employ separate decoders for each subtask to facilitate complex causality extraction. Experiments on three benchmark datasets demonstrate that our method achieves state-of-the-art performance and outperforms ChatGPT with a margin of at least 30% F1-score. More importantly, our model can also be used to effectively improve the ECE performance of ChatGPT via in-context learning.

Title: Iterative CT Reconstruction via Latent Variable Optimization of Shallow Diffusion Models

• Authors: Sho Ozaki, Shizuo Kaji, Toshikazu Imae, Kanabu Nawa, Hideomi Yamashita, Keiichi Nakagawa
• Subjects: cs.CV, cs.LG, physics.med-ph
• Abstract URL: https://arxiv.org/abs/2408.03156
• Pdf URL: https://arxiv.org/pdf/2408.03156
• Copy Paste: [[2408.03156]] Iterative CT Reconstruction via Latent Variable Optimization of Shallow Diffusion Models(https://arxiv.org/abs/2408.03156)
• Keywords: diffusion, generative
• Abstract: Image generative AI has garnered significant attention in recent years. In particular, the diffusion model, a core component of recent generative AI, produces high-quality images with rich diversity. In this study, we propose a novel CT reconstruction method by combining the denoising diffusion probabilistic model with iterative CT reconstruction. In sharp contrast to previous studies, we optimize the fidelity loss of CT reconstruction with respect to the latent variable of the diffusion model, instead of the image and model parameters. To suppress anatomical structure changes produced by the diffusion model, we shallow the diffusion and reverse processes, and fix a set of added noises in the reverse process to make it deterministic during inference. We demonstrate the effectiveness of the proposed method through sparse view CT reconstruction of 1/10 view projection data. Despite the simplicity of the implementation, the proposed method shows the capability of reconstructing high-quality images while preserving the patient's anatomical structure, and outperforms existing methods including iterative reconstruction, iterative reconstruction with total variation, and the diffusion model alone in terms of quantitative indices such as SSIM and PSNR. We also explore further sparse view CT using 1/20 view projection data with the same trained diffusion model. As the number of iterations increases, image quality improvement comparable to that of 1/10 sparse view CT reconstruction is achieved. In principle, the proposed method can be widely applied not only to CT but also to other imaging modalities such as MRI, PET, and SPECT.

Title: An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion

• Authors: Xingguang Yan, Han-Hung Lee, Ziyu Wan, Angel X. Chang
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03178
• Pdf URL: https://arxiv.org/pdf/2408.03178
• Copy Paste: [[2408.03178]] An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion(https://arxiv.org/abs/2408.03178)
• Keywords: diffusion, generative
• Abstract: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

Title: IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts

• Authors: Ciara Rowles, Shimon Vainer, Dante De Nigris, Slava Elizarov, Konstantin Kutsy, Simon Donné
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03209
• Pdf URL: https://arxiv.org/pdf/2408.03209
• Copy Paste: [[2408.03209]] IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts(https://arxiv.org/abs/2408.03209)
• Keywords: diffusion, generative
• Abstract: Diffusion models continuously push the boundary of state-of-the-art image generation, but the process is hard to control with any nuance: practice proves that textual prompts are inadequate for accurately describing image style or fine structural details (such as faces). ControlNet and IPAdapter address this shortcoming by conditioning the generative process on imagery instead, but each individual instance is limited to modeling a single conditional posterior: for practical use-cases, where multiple different posteriors are desired within the same workflow, training and using multiple adapters is cumbersome. We propose IPAdapter-Instruct, which combines natural-image conditioning with ``Instruct'' prompts to swap between interpretations for the same conditioning image: style transfer, object extraction, both, or something else still? IPAdapterInstruct efficiently learns multiple tasks with minimal loss in quality compared to dedicated per-task models.

Title: Biomedical SAM 2: Segment Anything in Biomedical Images and Videos

• Authors: Zhiling Yan, Weixiang Sun, Rong Zhou, Zhengqing Yuan, Kai Zhang, Yiwei Li, Tianming Liu, Quanzheng Li, Xiang Li, Lifang He, Lichao Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03286
• Pdf URL: https://arxiv.org/pdf/2408.03286
• Copy Paste: [[2408.03286]] Biomedical SAM 2: Segment Anything in Biomedical Images and Videos(https://arxiv.org/abs/2408.03286)
• Keywords: foundation model
• Abstract: Medical image segmentation and video object segmentation are essential for diagnosing and analyzing diseases by identifying and measuring biological structures. Recent advances in natural domain have been driven by foundation models like the Segment Anything Model 2 (SAM 2). To explore the performance of SAM 2 in biomedical applications, we designed two evaluation pipelines for single-frame image segmentation and multi-frame video segmentation with varied prompt designs, revealing SAM 2's limitations in medical contexts. Consequently, we developed BioSAM 2, an enhanced foundation model optimized for biomedical data based on SAM 2. Our experiments show that BioSAM 2 not only surpasses the performance of existing state-of-the-art foundation models but also matches or even exceeds specialist models, demonstrating its efficacy and potential in the medical domain.

Title: TextIM: Part-aware Interactive Motion Synthesis from Text

• Authors: Siyuan Fan, Bo Du, Xiantao Cai, Bo Peng, Longling Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03302
• Pdf URL: https://arxiv.org/pdf/2408.03302
• Copy Paste: [[2408.03302]] TextIM: Part-aware Interactive Motion Synthesis from Text(https://arxiv.org/abs/2408.03302)
• Keywords: diffusion
• Abstract: In this work, we propose TextIM, a novel framework for synthesizing TEXT-driven human Interactive Motions, with a focus on the precise alignment of part-level semantics. Existing methods often overlook the critical roles of interactive body parts and fail to adequately capture and align part-level semantics, resulting in inaccuracies and even erroneous movement outcomes. To address these issues, TextIM utilizes a decoupled conditional diffusion framework to enhance the detailed alignment between interactive movements and corresponding semantic intents from textual descriptions. Our approach leverages large language models, functioning as a human brain, to identify interacting human body parts and to comprehend interaction semantics to generate complicated and subtle interactive motion. Guided by the refined movements of the interacting parts, TextIM further extends these movements into a coherent whole-body motion. We design a spatial coherence module to complement the entire body movements while maintaining consistency and harmony across body parts using a part graph convolutional network. For training and evaluation, we carefully selected and re-labeled interactive motions from HUMANML3D to develop a specialized dataset. Experimental results demonstrate that TextIM produces semantically accurate human interactive motions, significantly enhancing the realism and applicability of synthesized interactive motions in diverse scenarios, even including interactions with deformable and dynamically changing objects.

Title: MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation

• Authors: Xiaofeng Mao, Zhengkai Jiang, Qilin Wang, Chencan Fu, Jiangning Zhang, Jiafu Wu, Yabiao Wang, Chengjie Wang, Wei Li, Mingmin Chi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03312
• Pdf URL: https://arxiv.org/pdf/2408.03312
• Copy Paste: [[2408.03312]] MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation(https://arxiv.org/abs/2408.03312)
• Keywords: diffusion
• Abstract: Recent advancements in the field of Diffusion Transformers have substantially improved the generation of high-quality 2D images, 3D videos, and 3D shapes. However, the effectiveness of the Transformer architecture in the domain of co-speech gesture generation remains relatively unexplored, as prior methodologies have predominantly employed the Convolutional Neural Network (CNNs) or simple a few transformer layers. In an attempt to bridge this research gap, we introduce a novel Masked Diffusion Transformer for co-speech gesture generation, referred to as MDT-A2G, which directly implements the denoising process on gesture sequences. To enhance the contextual reasoning capability of temporally aligned speech-driven gestures, we incorporate a novel Masked Diffusion Transformer. This model employs a mask modeling scheme specifically designed to strengthen temporal relation learning among sequence gestures, thereby expediting the learning process and leading to coherent and realistic motions. Apart from audio, Our MDT-A2G model also integrates multi-modal information, encompassing text, emotion, and identity. Furthermore, we propose an efficient inference strategy that diminishes the denoising computation by leveraging previously calculated results, thereby achieving a speedup with negligible performance degradation. Experimental results demonstrate that MDT-A2G excels in gesture generation, boasting a learning speed that is over 6$\times$ faster than traditional diffusion transformers and an inference speed that is 5.7$\times$ than the standard diffusion model.