2023-12-20

diffusion

Title: Synthetic Shifts to Initial Seed Vector Exposes the Brittle Nature of Latent-Based Diffusion Models. (arXiv:2312.11473v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11473
• Code URL: null
• Copy Paste: [[2312.11473]] Synthetic Shifts to Initial Seed Vector Exposes the Brittle Nature of Latent-Based Diffusion Models(http://arxiv.org/abs/2312.11473)
• Summary:

  Recent advances in Conditional Diffusion Models have led to substantial capabilities in various domains. However, understanding the impact of variations in the initial seed vector remains an underexplored area of concern. Particularly, latent-based diffusion models display inconsistencies in image generation under standard conditions when initialized with suboptimal initial seed vectors. To understand the impact of the initial seed vector on generated samples, we propose a reliability evaluation framework that evaluates the generated samples of a diffusion model when the initial seed vector is subjected to various synthetic shifts. Our results indicate that slight manipulations to the initial seed vector of the state-of-the-art Stable Diffusion (Rombach et al., 2022) can lead to significant disturbances in the generated samples, consequently creating images without the effect of conditioning variables. In contrast, GLIDE (Nichol et al., 2022) stands out in generating reliable samples even when the initial seed vector is transformed. Thus, our study sheds light on the importance of the selection and the impact of the initial seed vector in the latent-based diffusion model.

Title: Customize-It-3D: High-Quality 3D Creation from A Single Image Using Subject-Specific Knowledge Prior. (arXiv:2312.11535v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11535
• Code URL: null
• Copy Paste: [[2312.11535]] Customize-It-3D: High-Quality 3D Creation from A Single Image Using Subject-Specific Knowledge Prior(http://arxiv.org/abs/2312.11535)
• Summary:

  In this paper, we present a novel two-stage approach that fully utilizes the information provided by the reference image to establish a customized knowledge prior for image-to-3D generation. While previous approaches primarily rely on a general diffusion prior, which struggles to yield consistent results with the reference image, we propose a subject-specific and multi-modal diffusion model. This model not only aids NeRF optimization by considering the shading mode for improved geometry but also enhances texture from the coarse results to achieve superior refinement. Both aspects contribute to faithfully aligning the 3D content with the subject. Extensive experiments showcase the superiority of our method, Customize-It-3D, outperforming previous works by a substantial margin. It produces faithful 360-degree reconstructions with impressive visual quality, making it well-suited for various applications, including text-to-3D creation.

Title: Diffusion-Based Particle-DETR for BEV Perception. (arXiv:2312.11578v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11578
• Code URL: null
• Copy Paste: [[2312.11578]] Diffusion-Based Particle-DETR for BEV Perception(http://arxiv.org/abs/2312.11578)
• Summary:

  The Bird-Eye-View (BEV) is one of the most widely-used scene representations for visual perception in Autonomous Vehicles (AVs) due to its well suited compatibility to downstream tasks. For the enhanced safety of AVs, modeling perception uncertainty in BEV is crucial. Recent diffusion-based methods offer a promising approach to uncertainty modeling for visual perception but fail to effectively detect small objects in the large coverage of the BEV. Such degradation of performance can be attributed primarily to the specific network architectures and the matching strategy used when training. Here, we address this problem by combining the diffusion paradigm with current state-of-the-art 3D object detectors in BEV. We analyze the unique challenges of this approach, which do not exist with deterministic detectors, and present a simple technique based on object query interpolation that allows the model to learn positional dependencies even in the presence of the diffusion noise. Based on this, we present a diffusion-based DETR model for object detection that bears similarities to particle methods. Abundant experimentation on the NuScenes dataset shows equal or better performance for our generative approach, compared to deterministic state-of-the-art methods. Our source code will be made publicly available.

Title: TIP: Text-Driven Image Processing with Semantic and Restoration Instructions. (arXiv:2312.11595v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11595
• Code URL: null
• Copy Paste: [[2312.11595]] TIP: Text-Driven Image Processing with Semantic and Restoration Instructions(http://arxiv.org/abs/2312.11595)
• Summary:

  Text-driven diffusion models have become increasingly popular for various image editing tasks, including inpainting, stylization, and object replacement. However, it still remains an open research problem to adopt this language-vision paradigm for more fine-level image processing tasks, such as denoising, super-resolution, deblurring, and compression artifact removal. In this paper, we develop TIP, a Text-driven Image Processing framework that leverages natural language as a user-friendly interface to control the image restoration process. We consider the capacity of text information in two dimensions. First, we use content-related prompts to enhance the semantic alignment, effectively alleviating identity ambiguity in the restoration outcomes. Second, our approach is the first framework that supports fine-level instruction through language-based quantitative specification of the restoration strength, without the need for explicit task-specific design. In addition, we introduce a novel fusion mechanism that augments the existing ControlNet architecture by learning to rescale the generative prior, thereby achieving better restoration fidelity. Our extensive experiments demonstrate the superior restoration performance of TIP compared to the state of the arts, alongside offering the flexibility of text-based control over the restoration effects.

Title: Unified framework for diffusion generative models in SO(3): applications in computer vision and astrophysics. (arXiv:2312.11707v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11707
• Code URL: null
• Copy Paste: [[2312.11707]] Unified framework for diffusion generative models in SO(3): applications in computer vision and astrophysics(http://arxiv.org/abs/2312.11707)
• Summary:

  Diffusion-based generative models represent the current state-of-the-art for image generation. However, standard diffusion models are based on Euclidean geometry and do not translate directly to manifold-valued data. In this work, we develop extensions of both score-based generative models (SGMs) and Denoising Diffusion Probabilistic Models (DDPMs) to the Lie group of 3D rotations, SO(3). SO(3) is of particular interest in many disciplines such as robotics, biochemistry and astronomy/cosmology science. Contrary to more general Riemannian manifolds, SO(3) admits a tractable solution to heat diffusion, and allows us to implement efficient training of diffusion models. We apply both SO(3) DDPMs and SGMs to synthetic densities on SO(3) and demonstrate state-of-the-art results. Additionally, we demonstrate the practicality of our model on pose estimation tasks and in predicting correlated galaxy orientations for astrophysics/cosmology.

Title: Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation. (arXiv:2312.11774v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11774
• Code URL: null
• Copy Paste: [[2312.11774]] Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation(http://arxiv.org/abs/2312.11774)
• Summary:

  By lifting the pre-trained 2D diffusion models into Neural Radiance Fields (NeRFs), text-to-3D generation methods have made great progress. Many state-of-the-art approaches usually apply score distillation sampling (SDS) to optimize the NeRF representations, which supervises the NeRF optimization with pre-trained text-conditioned 2D diffusion models such as Imagen. However, the supervision signal provided by such pre-trained diffusion models only depends on text prompts and does not constrain the multi-view consistency. To inject the cross-view consistency into diffusion priors, some recent works finetune the 2D diffusion model with multi-view data, but still lack fine-grained view coherence. To tackle this challenge, we incorporate multi-view image conditions into the supervision signal of NeRF optimization, which explicitly enforces fine-grained view consistency. With such stronger supervision, our proposed text-to-3D method effectively mitigates the generation of floaters (due to excessive densities) and completely empty spaces (due to insufficient densities). Our quantitative evaluations on the T$^3$Bench dataset demonstrate that our method achieves state-of-the-art performance over existing text-to-3D methods. We will make the code publicly available.

Title: IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognition. (arXiv:2312.11923v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11923
• Code URL: null
• Copy Paste: [[2312.11923]] IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognition(http://arxiv.org/abs/2312.11923)
• Summary:

  Nowadays, scene text recognition has attracted more and more attention due to its diverse applications. Most state-of-the-art methods adopt an encoder-decoder framework with the attention mechanism, autoregressively generating text from left to right. Despite the convincing performance, this sequential decoding strategy constrains inference speed. Conversely, non-autoregressive models provide faster, simultaneous predictions but often sacrifice accuracy. Although utilizing an explicit language model can improve performance, it burdens the computational load. Besides, separating linguistic knowledge from vision information may harm the final prediction. In this paper, we propose an alternative solution, using a parallel and iterative decoder that adopts an easy-first decoding strategy. Furthermore, we regard text recognition as an image-based conditional text generation task and utilize the discrete diffusion strategy, ensuring exhaustive exploration of bidirectional contextual information. Extensive experiments demonstrate that the proposed approach achieves superior results on the benchmark datasets, including both Chinese and English text images.

Title: Optimizing Diffusion Noise Can Serve As Universal Motion Priors. (arXiv:2312.11994v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11994
• Code URL: null
• Copy Paste: [[2312.11994]] Optimizing Diffusion Noise Can Serve As Universal Motion Priors(http://arxiv.org/abs/2312.11994)
• Summary:

  We propose Diffusion Noise Optimization (DNO), a new method that effectively leverages existing motion diffusion models as motion priors for a wide range of motion-related tasks. Instead of training a task-specific diffusion model for each new task, DNO operates by optimizing the diffusion latent noise of an existing pre-trained text-to-motion model. Given the corresponding latent noise of a human motion, it propagates the gradient from the target criteria defined on the motion space through the whole denoising process to update the diffusion latent noise. As a result, DNO supports any use cases where criteria can be defined as a function of motion. In particular, we show that, for motion editing and control, DNO outperforms existing methods in both achieving the objective and preserving the motion content. DNO accommodates a diverse range of editing modes, including changing trajectory, pose, joint locations, or avoiding newly added obstacles. In addition, DNO is effective in motion denoising and completion, producing smooth and realistic motion from noisy and partial inputs. DNO achieves these results at inference time without the need for model retraining, offering great versatility for any defined reward or loss function on the motion representation.

Title: Diffusing More Objects for Semi-Supervised Domain Adaptation with Less Labeling. (arXiv:2312.12000v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12000
• Code URL: null
• Copy Paste: [[2312.12000]] Diffusing More Objects for Semi-Supervised Domain Adaptation with Less Labeling(http://arxiv.org/abs/2312.12000)
• Summary:

  For object detection, it is possible to view the prediction of bounding boxes as a reverse diffusion process. Using a diffusion model, the random bounding boxes are iteratively refined in a denoising step, conditioned on the image. We propose a stochastic accumulator function that starts each run with random bounding boxes and combines the slightly different predictions. We empirically verify that this improves detection performance. The improved detections are leveraged on unlabelled images as weighted pseudo-labels for semi-supervised learning. We evaluate the method on a challenging out-of-domain test set. Our method brings significant improvements and is on par with human-selected pseudo-labels, while not requiring any human involvement.

Title: Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method. (arXiv:2312.12030v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12030
• Code URL: null
• Copy Paste: [[2312.12030]] Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method(http://arxiv.org/abs/2312.12030)
• Summary:

  Training-free guided sampling in diffusion models leverages off-the-shelf pre-trained networks, such as an aesthetic evaluation model, to guide the generation process. Current training-free guided sampling algorithms obtain the guidance energy function based on a one-step estimate of the clean image. However, since the off-the-shelf pre-trained networks are trained on clean images, the one-step estimation procedure of the clean image may be inaccurate, especially in the early stages of the generation process in diffusion models. This causes the guidance in the early time steps to be inaccurate. To overcome this problem, we propose Symplectic Adjoint Guidance (SAG), which calculates the gradient guidance in two inner stages. Firstly, SAG estimates the clean image via $n$ function calls, where $n$ serves as a flexible hyperparameter that can be tailored to meet specific image quality requirements. Secondly, SAG uses the symplectic adjoint method to obtain the gradients accurately and efficiently in terms of the memory requirements. Extensive experiments demonstrate that SAG generates images with higher qualities compared to the baselines in both guided image and video generation tasks.

Title: Learning a Diffusion Model Policy from Rewards via Q-Score Matching. (arXiv:2312.11752v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11752
• Code URL: null
• Copy Paste: [[2312.11752]] Learning a Diffusion Model Policy from Rewards via Q-Score Matching(http://arxiv.org/abs/2312.11752)
• Summary:

  Diffusion models have become a popular choice for representing actor policies in behavior cloning and offline reinforcement learning. This is due to their natural ability to optimize an expressive class of distributions over a continuous space. However, previous works fail to exploit the score-based structure of diffusion models, and instead utilize a simple behavior cloning term to train the actor, limiting their ability in the actor-critic setting. In this paper, we focus on off-policy reinforcement learning and propose a new method for learning a diffusion model policy that exploits the linked structure between the score of the policy and the action gradient of the Q-function. We denote this method Q-score matching and provide theoretical justification for this approach. We conduct experiments in simulated environments to demonstrate the effectiveness of our proposed method and compare to popular baselines.

self-supervised

Title: Self-supervised Learning for Enhancing Geometrical Modeling in 3D-Aware Generative Adversarial Network. (arXiv:2312.11856v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11856
• Code URL: null
• Copy Paste: [[2312.11856]] Self-supervised Learning for Enhancing Geometrical Modeling in 3D-Aware Generative Adversarial Network(http://arxiv.org/abs/2312.11856)
• Summary:

  3D-aware Generative Adversarial Networks (3D-GANs) currently exhibit artifacts in their 3D geometrical modeling, such as mesh imperfections and holes. These shortcomings are primarily attributed to the limited availability of annotated 3D data, leading to a constrained "valid latent area" for satisfactory modeling. To address this, we present a Self-Supervised Learning (SSL) technique tailored as an auxiliary loss for any 3D-GAN, designed to improve its 3D geometrical modeling capabilities. Our approach pioneers an inversion technique for 3D-GANs, integrating an encoder that performs adaptive spatially-varying range operations. Utilizing this inversion, we introduce the Cyclic Generative Constraint (CGC), aiming to densify the valid latent space. The CGC operates via augmented local latent vectors that maintain the same geometric form, and it imposes constraints on the cycle path outputs, specifically the generator-encoder-generator sequence. This SSL methodology seamlessly integrates with the inherent GAN loss, ensuring the integrity of pre-existing 3D-GAN architectures without necessitating alterations. We validate our approach with comprehensive experiments across various datasets and architectures, underscoring its efficacy. Our project website: https://3dgan-ssl.github.io

Title: DMT: Comprehensive Distillation with Multiple Self-supervised Teachers. (arXiv:2312.11938v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11938
• Code URL: null
• Copy Paste: [[2312.11938]] DMT: Comprehensive Distillation with Multiple Self-supervised Teachers(http://arxiv.org/abs/2312.11938)
• Summary:

  Numerous self-supervised learning paradigms, such as contrastive learning and masked image modeling, have been proposed to acquire powerful and general representations from unlabeled data. However, these models are commonly pretrained within their specific framework alone, failing to consider the complementary nature of visual representations. To tackle this issue, we introduce Comprehensive Distillation with Multiple Self-supervised Teachers (DMT) for pretrained model compression, which leverages the strengths of multiple off-the-shelf self-supervised models. Our experimental results on prominent benchmark datasets exhibit that the proposed method significantly surpasses state-of-the-art competitors while retaining favorable efficiency metrics. On classification tasks, our DMT framework utilizing three different self-supervised ViT-Base teachers enhances the performance of both small/tiny models and the base model itself. For dense tasks, DMT elevates the AP/mIoU of standard SSL models on MS-COCO and ADE20K datasets by 4.0%.

Title: Empowering Dual-Level Graph Self-Supervised Pretraining with Motif Discovery. (arXiv:2312.11927v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11927
• Code URL: https://github.com/rocccyan/dgpm
• Copy Paste: [[2312.11927]] Empowering Dual-Level Graph Self-Supervised Pretraining with Motif Discovery(http://arxiv.org/abs/2312.11927)
• Summary:

  While self-supervised graph pretraining techniques have shown promising results in various domains, their application still experiences challenges of limited topology learning, human knowledge dependency, and incompetent multi-level interactions. To address these issues, we propose a novel solution, Dual-level Graph self-supervised Pretraining with Motif discovery (DGPM), which introduces a unique dual-level pretraining structure that orchestrates node-level and subgraph-level pretext tasks. Unlike prior approaches, DGPM autonomously uncovers significant graph motifs through an edge pooling module, aligning learned motif similarities with graph kernel-based similarities. A cross-matching task enables sophisticated node-motif interactions and novel representation learning. Extensive experiments on 15 datasets validate DGPM's effectiveness and generalizability, outperforming state-of-the-art methods in unsupervised representation learning and transfer learning settings. The autonomously discovered motifs demonstrate the potential of DGPM to enhance robustness and interpretability.

Title: Time-Series Contrastive Learning against False Negatives and Class Imbalance. (arXiv:2312.11939v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11939
• Code URL: null
• Copy Paste: [[2312.11939]] Time-Series Contrastive Learning against False Negatives and Class Imbalance(http://arxiv.org/abs/2312.11939)
• Summary:

  As an exemplary self-supervised approach for representation learning, time-series contrastive learning has exhibited remarkable advancements in contemporary research. While recent contrastive learning strategies have focused on how to construct appropriate positives and negatives, in this study, we conduct theoretical analysis and find they have overlooked the fundamental issues: false negatives and class imbalance inherent in the InfoNCE loss-based framework. Therefore, we introduce a straightforward modification grounded in the SimCLR framework, universally adaptable to models engaged in the instance discrimination task. By constructing instance graphs to facilitate interactive learning among instances, we emulate supervised contrastive learning via the multiple-instances discrimination task, mitigating the harmful impact of false negatives. Moreover, leveraging the graph structure and few-labeled data, we perform semi-supervised consistency classification and enhance the representative ability of minority classes. We compared our method with the most popular time-series contrastive learning methods on four real-world time-series datasets and demonstrated our significant advantages in overall performance.

foundation model

Title: StarVector: Generating Scalable Vector Graphics Code from Images. (arXiv:2312.11556v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11556
• Code URL: null
• Copy Paste: [[2312.11556]] StarVector: Generating Scalable Vector Graphics Code from Images(http://arxiv.org/abs/2312.11556)
• Summary:

  Scalable Vector Graphics (SVGs) have become integral in modern image rendering applications due to their infinite scalability in resolution, versatile usability, and editing capabilities. SVGs are particularly popular in the fields of web development and graphic design. Existing approaches for SVG modeling using deep learning often struggle with generating complex SVGs and are restricted to simpler ones that require extensive processing and simplification. This paper introduces StarVector, a multimodal SVG generation model that effectively integrates Code Generation Large Language Models (CodeLLMs) and vision models. Our approach utilizes a CLIP image encoder to extract visual representations from pixel-based images, which are then transformed into visual tokens via an adapter module. These visual tokens are pre-pended to the SVG token embeddings, and the sequence is modeled by the StarCoder model using next-token prediction, effectively learning to align the visual and code tokens. This enables StarVector to generate unrestricted SVGs that accurately represent pixel images. To evaluate StarVector's performance, we present SVG-Bench, a comprehensive benchmark for evaluating SVG methods across multiple datasets and relevant metrics. Within this benchmark, we introduce novel datasets including SVG-Stack, a large-scale dataset of real-world SVG examples, and use it to pre-train StarVector as a large foundation model for SVGs. Our results demonstrate significant enhancements in visual quality and complexity handling over current methods, marking a notable advancement in SVG generation technology. Code and models: https://github.com/joanrod/star-vector

Title: RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation. (arXiv:2312.11829v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11829
• Code URL: null
• Copy Paste: [[2312.11829]] RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation(http://arxiv.org/abs/2312.11829)
• Summary:

  3D occupancy prediction is an emerging task that aims to estimate the occupancy states and semantics of 3D scenes using multi-view images. However, image-based scene perception encounters significant challenges in achieving accurate prediction due to the absence of geometric priors. In this paper, we address this issue by exploring cross-modal knowledge distillation in this task, i.e., we leverage a stronger multi-modal model to guide the visual model during training. In practice, we observe that directly applying features or logits alignment, proposed and widely used in bird's-eyeview (BEV) perception, does not yield satisfactory results. To overcome this problem, we introduce RadOcc, a Rendering assisted distillation paradigm for 3D Occupancy prediction. By employing differentiable volume rendering, we generate depth and semantic maps in perspective views and propose two novel consistency criteria between the rendered outputs of teacher and student models. Specifically, the depth consistency loss aligns the termination distributions of the rendered rays, while the semantic consistency loss mimics the intra-segment similarity guided by vision foundation models (VLMs). Experimental results on the nuScenes dataset demonstrate the effectiveness of our proposed method in improving various 3D occupancy prediction approaches, e.g., our proposed methodology enhances our baseline by 2.2% in the metric of mIoU and achieves 50% in Occ3D benchmark.

Title: 3D-LFM: Lifting Foundation Model. (arXiv:2312.11894v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11894
• Code URL: https://github.com/mosamdabhi/3dlfm
• Copy Paste: [[2312.11894]] 3D-LFM: Lifting Foundation Model(http://arxiv.org/abs/2312.11894)
• Summary:

  The lifting of 3D structure and camera from 2D landmarks is at the cornerstone of the entire discipline of computer vision. Traditional methods have been confined to specific rigid objects, such as those in Perspective-n-Point (PnP) problems, but deep learning has expanded our capability to reconstruct a wide range of object classes (e.g. C3PDO and PAUL) with resilience to noise, occlusions, and perspective distortions. All these techniques, however, have been limited by the fundamental need to establish correspondences across the 3D training data -- significantly limiting their utility to applications where one has an abundance of "in-correspondence" 3D data. Our approach harnesses the inherent permutation equivariance of transformers to manage varying number of points per 3D data instance, withstands occlusions, and generalizes to unseen categories. We demonstrate state of the art performance across 2D-3D lifting task benchmarks. Since our approach can be trained across such a broad class of structures we refer to it simply as a 3D Lifting Foundation Model (3D-LFM) -- the first of its kind.

Title: Big Learning Expectation Maximization. (arXiv:2312.11926v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11926
• Code URL: https://github.com/yulaicong/big-learning-expectation-maximization
• Copy Paste: [[2312.11926]] Big Learning Expectation Maximization(http://arxiv.org/abs/2312.11926)
• Summary:

  Mixture models serve as one fundamental tool with versatile applications. However, their training techniques, like the popular Expectation Maximization (EM) algorithm, are notoriously sensitive to parameter initialization and often suffer from bad local optima that could be arbitrarily worse than the optimal. To address the long-lasting bad-local-optima challenge, we draw inspiration from the recent ground-breaking foundation models and propose to leverage their underlying big learning principle to upgrade the EM. Specifically, we present the Big Learning EM (BigLearn-EM), an EM upgrade that simultaneously performs joint, marginal, and orthogonally transformed marginal matchings between data and model distributions. Through simulated experiments, we empirically show that the BigLearn-EM is capable of delivering the optimal with high probability; comparisons on benchmark clustering datasets further demonstrate its effectiveness and advantages over existing techniques. The code is available at https://github.com/YulaiCong/Big-Learning-Expectation-Maximization.

generative

Title: HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles. (arXiv:2312.11666v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11666
• Code URL: null
• Copy Paste: [[2312.11666]] HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles(http://arxiv.org/abs/2312.11666)
• Summary:

  We present HAAR, a new strand-based generative model for 3D human hairstyles. Specifically, based on textual inputs, HAAR produces 3D hairstyles that could be used as production-level assets in modern computer graphics engines. Current AI-based generative models take advantage of powerful 2D priors to reconstruct 3D content in the form of point clouds, meshes, or volumetric functions. However, by using the 2D priors, they are intrinsically limited to only recovering the visual parts. Highly occluded hair structures can not be reconstructed with those methods, and they only model the ''outer shell'', which is not ready to be used in physics-based rendering or simulation pipelines. In contrast, we propose a first text-guided generative method that uses 3D hair strands as an underlying representation. Leveraging 2D visual question-answering (VQA) systems, we automatically annotate synthetic hair models that are generated from a small set of artist-created hairstyles. This allows us to train a latent diffusion model that operates in a common hairstyle UV space. In qualitative and quantitative studies, we demonstrate the capabilities of the proposed model and compare it to existing hairstyle generation approaches.

Title: Large Language Models are Complex Table Parsers. (arXiv:2312.11521v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11521
• Code URL: null
• Copy Paste: [[2312.11521]] Large Language Models are Complex Table Parsers(http://arxiv.org/abs/2312.11521)
• Summary:

  With the Generative Pre-trained Transformer 3.5 (GPT-3.5) exhibiting remarkable reasoning and comprehension abilities in Natural Language Processing (NLP), most Question Answering (QA) research has primarily centered around general QA tasks based on GPT, neglecting the specific challenges posed by Complex Table QA. In this paper, we propose to incorporate GPT-3.5 to address such challenges, in which complex tables are reconstructed into tuples and specific prompt designs are employed for dialogues. Specifically, we encode each cell's hierarchical structure, position information, and content as a tuple. By enhancing the prompt template with an explanatory description of the meaning of each tuple and the logical reasoning process of the task, we effectively improve the hierarchical structure awareness capability of GPT-3.5 to better parse the complex tables. Extensive experiments and results on Complex Table QA datasets, i.e., the open-domain dataset HiTAB and the aviation domain dataset AIT-QA show that our approach significantly outperforms previous work on both datasets, leading to state-of-the-art (SOTA) performance.

Title: ToViLaG: Your Visual-Language Generative Model is Also An Evildoer. (arXiv:2312.11523v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11523
• Code URL: https://github.com/victorup/ToViLaG
• Copy Paste: [[2312.11523]] ToViLaG: Your Visual-Language Generative Model is Also An Evildoer(http://arxiv.org/abs/2312.11523)
• Summary:

  Warning: this paper includes model outputs showing offensive content. Recent large-scale Visual-Language Generative Models (VLGMs) have achieved unprecedented improvement in multimodal image/text generation. However, these models might also generate toxic content, e.g., offensive text and pornography images, raising significant ethical risks. Despite exhaustive studies on toxic degeneration of language models, this problem remains largely unexplored within the context of visual-language generation. This work delves into the propensity for toxicity generation and susceptibility to toxic data across various VLGMs. For this purpose, we built ToViLaG, a dataset comprising 32K co-toxic/mono-toxic text-image pairs and 1K innocuous but evocative text that tends to stimulate toxicity. Furthermore, we propose WInToRe, a novel toxicity metric tailored to visual-language generation, which theoretically reflects different aspects of toxicity considering both input and output. On such a basis, we benchmarked the toxicity of a diverse spectrum of VLGMs and discovered that some models do more evil than expected while some are more vulnerable to infection, underscoring the necessity of VLGMs detoxification. Therefore, we develop an innovative bottleneck-based detoxification method. Our method could reduce toxicity while maintaining comparable generation quality, providing a promising initial solution to this line of research.

Title: Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation. (arXiv:2312.11532v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11532
• Code URL: null
• Copy Paste: [[2312.11532]] Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation(http://arxiv.org/abs/2312.11532)
• Summary:

  This paper introduces a novel approach for topic modeling utilizing latent codebooks from Vector-Quantized Variational Auto-Encoder~(VQ-VAE), discretely encapsulating the rich information of the pre-trained embeddings such as the pre-trained language model. From the novel interpretation of the latent codebooks and embeddings as conceptual bag-of-words, we propose a new generative topic model called Topic-VQ-VAE~(TVQ-VAE) which inversely generates the original documents related to the respective latent codebook. The TVQ-VAE can visualize the topics with various generative distributions including the traditional BoW distribution and the autoregressive image generation. Our experimental results on document analysis and image generation demonstrate that TVQ-VAE effectively captures the topic context which reveals the underlying structures of the dataset and supports flexible forms of document generation. Official implementation of the proposed TVQ-VAE is available at https://github.com/clovaai/TVQ-VAE.

Title: COPD-FlowNet: Elevating Non-invasive COPD Diagnosis with CFD Simulations. (arXiv:2312.11561v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11561
• Code URL: null
• Copy Paste: [[2312.11561]] COPD-FlowNet: Elevating Non-invasive COPD Diagnosis with CFD Simulations(http://arxiv.org/abs/2312.11561)
• Summary:

  Chronic Obstructive Pulmonary Disorder (COPD) is a prevalent respiratory disease that significantly impacts the quality of life of affected individuals. This paper presents COPDFlowNet, a novel deep-learning framework that leverages a custom Generative Adversarial Network (GAN) to generate synthetic Computational Fluid Dynamics (CFD) velocity flow field images specific to the trachea of COPD patients. These synthetic images serve as a valuable resource for data augmentation and model training. Additionally, COPDFlowNet incorporates a custom Convolutional Neural Network (CNN) architecture to predict the location of the obstruction site.

Title: Time-Transformer: Integrating Local and Global Features for Better Time Series Generation. (arXiv:2312.11714v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11714
• Code URL: null
• Copy Paste: [[2312.11714]] Time-Transformer: Integrating Local and Global Features for Better Time Series Generation(http://arxiv.org/abs/2312.11714)
• Summary:

  Generating time series data is a promising approach to address data deficiency problems. However, it is also challenging due to the complex temporal properties of time series data, including local correlations as well as global dependencies. Most existing generative models have failed to effectively learn both the local and global properties of time series data. To address this open problem, we propose a novel time series generative model named 'Time-Transformer AAE', which consists of an adversarial autoencoder (AAE) and a newly designed architecture named 'Time-Transformer' within the decoder. The Time-Transformer first simultaneously learns local and global features in a layer-wise parallel design, combining the abilities of Temporal Convolutional Networks and Transformer in extracting local features and global dependencies respectively. Second, a bidirectional cross attention is proposed to provide complementary guidance across the two branches and achieve proper fusion between local and global features. Experimental results demonstrate that our model can outperform existing state-of-the-art models in 5 out of 6 datasets, specifically on those with data containing both global and local properties. Furthermore, we highlight our model's advantage on handling this kind of data via an artificial dataset. Finally, we show our model's ability to address a real-world problem: data augmentation to support learning with small datasets and imbalanced datasets.

Title: Robust Stochastic Graph Generator for Counterfactual Explanations. (arXiv:2312.11747v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11747
• Code URL: null
• Copy Paste: [[2312.11747]] Robust Stochastic Graph Generator for Counterfactual Explanations(http://arxiv.org/abs/2312.11747)
• Summary:

  Counterfactual Explanation (CE) techniques have garnered attention as a means to provide insights to the users engaging with AI systems. While extensively researched in domains such as medical imaging and autonomous vehicles, Graph Counterfactual Explanation (GCE) methods have been comparatively under-explored. GCEs generate a new graph similar to the original one, with a different outcome grounded on the underlying predictive model. Among these GCE techniques, those rooted in generative mechanisms have received relatively limited investigation despite demonstrating impressive accomplishments in other domains, such as artistic styles and natural language modelling. The preference for generative explainers stems from their capacity to generate counterfactual instances during inference, leveraging autonomously acquired perturbations of the input graph. Motivated by the rationales above, our study introduces RSGG-CE, a novel Robust Stochastic Graph Generator for Counterfactual Explanations able to produce counterfactual examples from the learned latent space considering a partially ordered generation sequence. Furthermore, we undertake quantitative and qualitative analyses to compare RSGG-CE's performance against SoA generative explainers, highlighting its increased ability to engendering plausible counterfactual candidates.

anomaly

Title: Anomaly detection for automated inspection of power line insulators. (arXiv:2312.11470v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11470
• Code URL: null
• Copy Paste: [[2312.11470]] Anomaly detection for automated inspection of power line insulators(http://arxiv.org/abs/2312.11470)
• Summary:

  Inspection of insulators is important to ensure reliable operation of the power system. Deep learning has recently been explored to automate the inspection process by leveraging aerial images captured by drones along with powerful object detection models. However, a purely object detection-based approach exhibits class imbalance-induced poor detection accuracy for faulty insulators, especially for incipient faults. In order to address this issue in a data-efficient manner, this article proposes a two-stage approach that leverages object detection in conjunction with anomaly detection to reliably detect faults in insulators. The article adopts an explainable deep neural network-based one-class classifier for anomaly detection, that reduces the reliance on plentifully available images of faulty insulators, that might be difficult to obtain in real-life applications. The anomaly detection model is trained with two datasets -- representing data abundant and data scarce scenarios -- in unsupervised and semi-supervised manner. The results suggest that including as few as six real anomalies in the training dataset significantly improves the performance of the model, and enables reliable detection of rarely occurring faults in insulators. An analysis of the explanations provided by the anomaly detection model reveals that the model is able to accurately identify faulty regions on the insulator disks, while also exhibiting some false predictions.

Title: Label-Free Multivariate Time Series Anomaly Detection. (arXiv:2312.11549v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11549
• Code URL: null
• Copy Paste: [[2312.11549]] Label-Free Multivariate Time Series Anomaly Detection(http://arxiv.org/abs/2312.11549)
• Summary:

  Anomaly detection in multivariate time series (MTS) has been widely studied in one-class classification (OCC) setting. The training samples in OCC are assumed to be normal, which is difficult to guarantee in practical situations. Such a case may degrade the performance of OCC-based anomaly detection methods which fit the training distribution as the normal distribution. In this paper, we propose MTGFlow, an unsupervised anomaly detection approach for MTS anomaly detection via dynamic Graph and entity-aware normalizing Flow. MTGFlow first estimates the density of the entire training samples and then identifies anomalous instances based on the density of the test samples within the fitted distribution. This relies on a widely accepted assumption that anomalous instances exhibit more sparse densities than normal ones, with no reliance on the clean training dataset. However, it is intractable to directly estimate the density due to complex dependencies among entities and their diverse inherent characteristics. To mitigate this, we utilize the graph structure learning model to learn interdependent and evolving relations among entities, which effectively captures complex and accurate distribution patterns of MTS. In addition, our approach incorporates the unique characteristics of individual entities by employing an entity-aware normalizing flow. This enables us to represent each entity as a parameterized normal distribution. Furthermore, considering that some entities present similar characteristics, we propose a cluster strategy that capitalizes on the commonalities of entities with similar characteristics, resulting in more precise and detailed density estimation. We refer to this cluster-aware extension as MTGFlow_cluster. Extensive experiments are conducted on six widely used benchmark datasets, in which MTGFlow and MTGFlow cluster demonstrate their superior detection performance.

Title: When Model Meets New Normals: Test-time Adaptation for Unsupervised Time-series Anomaly Detection. (arXiv:2312.11976v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11976
• Code URL: https://github.com/carrtesy/m2n2
• Copy Paste: [[2312.11976]] When Model Meets New Normals: Test-time Adaptation for Unsupervised Time-series Anomaly Detection(http://arxiv.org/abs/2312.11976)
• Summary:

  Time-series anomaly detection deals with the problem of detecting anomalous timesteps by learning normality from the sequence of observations. However, the concept of normality evolves over time, leading to a "new normal problem", where the distribution of normality can be changed due to the distribution shifts between training and test data. This paper highlights the prevalence of the new normal problem in unsupervised time-series anomaly detection studies. To tackle this issue, we propose a simple yet effective test-time adaptation strategy based on trend estimation and a self-supervised approach to learning new normalities during inference. Extensive experiments on real-world benchmarks demonstrate that incorporating the proposed strategy into the anomaly detector consistently improves the model's performance compared to the baselines, leading to robustness to the distribution shifts.

in-context