2024-04-11

Title: Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation

• Authors: Luca Barsellotti, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06542
• Pdf URL: https://arxiv.org/pdf/2404.06542
• Copy Paste: [[2404.06542]] Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation(https://arxiv.org/abs/2404.06542)
• Keywords: diffusion
• Abstract: Open-vocabulary semantic segmentation aims at segmenting arbitrary categories expressed in textual form. Previous works have trained over large amounts of image-caption pairs to enforce pixel-level multimodal alignments. However, captions provide global information about the semantics of a given image but lack direct localization of individual concepts. Further, training on large-scale datasets inevitably brings significant computational costs. In this paper, we propose FreeDA, a training-free diffusion-augmented method for open-vocabulary semantic segmentation, which leverages the ability of diffusion models to visually localize generated concepts and local-global similarities to match class-agnostic regions with semantic classes. Our approach involves an offline stage in which textual-visual reference embeddings are collected, starting from a large set of captions and leveraging visual and semantic contexts. At test time, these are queried to support the visual matching process, which is carried out by jointly considering class-agnostic regions and global semantic similarities. Extensive analyses demonstrate that FreeDA achieves state-of-the-art performance on five datasets, surpassing previous methods by more than 7.0 average points in terms of mIoU and without requiring any training.

Title: MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection

• Authors: Haoyang He, Yuhu Bai, Jiangning Zhang, Qingdong He, Hongxu Chen, Zhenye Gan, Chengjie Wang, Xiangtai Li, Guanzhong Tian, Lei Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06564
• Pdf URL: https://arxiv.org/pdf/2404.06564
• Copy Paste: [[2404.06564]] MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection(https://arxiv.org/abs/2404.06564)
• Keywords: anomaly
• Abstract: Recent advancements in anomaly detection have seen the efficacy of CNN- and transformer-based approaches. However, CNNs struggle with long-range dependencies, while transformers are burdened by quadratic computational complexity. Mamba-based models, with their superior long-range modeling and linear efficiency, have garnered substantial attention. This study pioneers the application of Mamba to multi-class unsupervised anomaly detection, presenting MambaAD, which consists of a pre-trained encoder and a Mamba decoder featuring Locality-Enhanced State Space (LSS) modules at multi-scales. The proposed LSS module, integrating parallel cascaded (Hybrid State Space) HSS blocks and multi-kernel convolutions operations, effectively captures both long-range and local information. The HSS block, utilizing (Hybrid Scanning) HS encoders, encodes feature maps into five scanning methods and eight directions, thereby strengthening global connections through the (State Space Model) SSM. The use of Hilbert scanning and eight directions significantly improves feature sequence modeling. Comprehensive experiments on six diverse anomaly detection datasets and seven metrics demonstrate SoTA performance, substantiating the method's effectiveness.

Title: FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations

• Authors: Jane Dwivedi-Yu, Raaz Dwivedi, Timo Schick
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06619
• Pdf URL: https://arxiv.org/pdf/2404.06619
• Copy Paste: [[2404.06619]] FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations(https://arxiv.org/abs/2404.06619)
• Keywords: generative
• Abstract: The accurate evaluation of differential treatment in language models to specific groups is critical to ensuring a positive and safe user experience. An ideal evaluation should have the properties of being robust, extendable to new groups or attributes, and being able to capture biases that appear in typical usage (rather than just extreme, rare cases). Relatedly, bias evaluation should surface not only egregious biases but also ones that are subtle and commonplace, such as a likelihood for talking about appearances with regard to women. We present FairPair, an evaluation framework for assessing differential treatment that occurs during ordinary usage. FairPair operates through counterfactual pairs, but crucially, the paired continuations are grounded in the same demographic group, which ensures equivalent comparison. Additionally, unlike prior work, our method factors in the inherent variability that comes from the generation process itself by measuring the sampling variability. We present an evaluation of several commonly used generative models and a qualitative analysis that indicates a preference for discussing family and hobbies with regard to women.

Title: Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?

• Authors: Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, Mohammad Hossein Rohban
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06644
• Pdf URL: https://arxiv.org/pdf/2404.06644
• Copy Paste: [[2404.06644]] Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?(https://arxiv.org/abs/2404.06644)
• Keywords: generative
• Abstract: Evaluating Large Language Models (LLMs) is challenging due to their generative nature, necessitating precise evaluation methodologies. Additionally, non-English LLM evaluation lags behind English, resulting in the absence or weakness of LLMs for many languages. In response to this necessity, we introduce Khayyam Challenge (also known as PersianMMLU), a meticulously curated collection comprising 20,192 four-choice questions sourced from 38 diverse tasks extracted from Persian examinations, spanning a wide spectrum of subjects, complexities, and ages. The primary objective of the Khayyam Challenge is to facilitate the rigorous evaluation of LLMs that support the Persian language. Distinctive features of the Khayyam Challenge are (i) its comprehensive coverage of various topics, including literary comprehension, mathematics, sciences, logic, intelligence testing, etc., aimed at assessing different facets of LLMs such as language comprehension, reasoning, and information retrieval across various educational stages, from lower primary school to upper secondary school (ii) its inclusion of rich metadata such as human response rates, difficulty levels, and descriptive answers (iii) its utilization of new data to avoid data contamination issues prevalent in existing frameworks (iv) its use of original, non-translated data tailored for Persian speakers, ensuring the framework is free from translation challenges and errors while encompassing cultural nuances (v) its inherent scalability for future data updates and evaluations without requiring special human effort. Previous works lacked an evaluation framework that combined all of these features into a single comprehensive benchmark. Furthermore, we evaluate a wide range of existing LLMs that support the Persian language, with statistical analyses and interpretations of their outputs.

Title: Efficient Denoising using Score Embedding in Score-based Diffusion Models

• Authors: Andrew S. Na, William Gao, Justin W.L. Wan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06661
• Pdf URL: https://arxiv.org/pdf/2404.06661
• Copy Paste: [[2404.06661]] Efficient Denoising using Score Embedding in Score-based Diffusion Models(https://arxiv.org/abs/2404.06661)
• Keywords: diffusion
• Abstract: It is well known that training a denoising score-based diffusion models requires tens of thousands of epochs and a substantial number of image data to train the model. In this paper, we propose to increase the efficiency in training score-based diffusion models. Our method allows us to decrease the number of epochs needed to train the diffusion model. We accomplish this by solving the log-density Fokker-Planck (FP) Equation numerically to compute the score \textit{before} training. The pre-computed score is embedded into the image to encourage faster training under slice Wasserstein distance. Consequently, it also allows us to decrease the number of images we need to train the neural network to learn an accurate score. We demonstrate through our numerical experiments the improved performance of our proposed method compared to standard score-based diffusion models. Our proposed method achieves a similar quality to the standard method meaningfully faster.

Title: Multi-modal Document Presentation Attack Detection With Forensics Trace Disentanglement

• Authors: Changsheng Chen, Yongyi Deng, Liangwei Lin, Zitong Yu, Zhimao Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06663
• Pdf URL: https://arxiv.org/pdf/2404.06663
• Copy Paste: [[2404.06663]] Multi-modal Document Presentation Attack Detection With Forensics Trace Disentanglement(https://arxiv.org/abs/2404.06663)
• Keywords: self-supervised
• Abstract: Document Presentation Attack Detection (DPAD) is an important measure in protecting the authenticity of a document image. However, recent DPAD methods demand additional resources, such as manual effort in collecting additional data or knowing the parameters of acquisition devices. This work proposes a DPAD method based on multi-modal disentangled traces (MMDT) without the above drawbacks. We first disentangle the recaptured traces by a self-supervised disentanglement and synthesis network to enhance the generalization capacity in document images with different contents and layouts. Then, unlike the existing DPAD approaches that rely only on data in the RGB domain, we propose to explicitly employ the disentangled recaptured traces as new modalities in the transformer backbone through adaptive multi-modal adapters to fuse RGB/trace features efficiently. Visualization of the disentangled traces confirms the effectiveness of the proposed method in different document contents. Extensive experiments on three benchmark datasets demonstrate the superiority of our MMDT method on representing forensic traces of recapturing distortion.

Title: Deep Generative Data Assimilation in Multimodal Setting

• Authors: Yongquan Qu, Juan Nathaniel, Shuolin Li, Pierre Gentine
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06665
• Pdf URL: https://arxiv.org/pdf/2404.06665
• Copy Paste: [[2404.06665]] Deep Generative Data Assimilation in Multimodal Setting(https://arxiv.org/abs/2404.06665)
• Keywords: diffusion, generative
• Abstract: Robust integration of physical knowledge and data is key to improve computational simulations, such as Earth system models. Data assimilation is crucial for achieving this goal because it provides a systematic framework to calibrate model outputs with observations, which can include remote sensing imagery and ground station measurements, with uncertainty quantification. Conventional methods, including Kalman filters and variational approaches, inherently rely on simplifying linear and Gaussian assumptions, and can be computationally expensive. Nevertheless, with the rapid adoption of data-driven methods in many areas of computational sciences, we see the potential of emulating traditional data assimilation with deep learning, especially generative models. In particular, the diffusion-based probabilistic framework has large overlaps with data assimilation principles: both allows for conditional generation of samples with a Bayesian inverse framework. These models have shown remarkable success in text-conditioned image generation or image-controlled video synthesis. Likewise, one can frame data assimilation as observation-conditioned state calibration. In this work, we propose SLAMS: Score-based Latent Assimilation in Multimodal Setting. Specifically, we assimilate in-situ weather station data and ex-situ satellite imagery to calibrate the vertical temperature profiles, globally. Through extensive ablation, we demonstrate that SLAMS is robust even in low-resolution, noisy, and sparse data settings. To our knowledge, our work is the first to apply deep generative framework for multimodal data assimilation using real-world datasets; an important step for building robust computational simulators, including the next-generation Earth system models. Our code is available at: https://github.com/yongquan-qu/SLAMS

Title: SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models

• Authors: Xinfeng Li, Yuchen Yang, Jiangyi Deng, Chen Yan, Yanjiao Chen, Xiaoyu Ji, Wenyuan Xu
• Subjects: cs.CV, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06666
• Pdf URL: https://arxiv.org/pdf/2404.06666
• Copy Paste: [[2404.06666]] SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models(https://arxiv.org/abs/2404.06666)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) models, such as Stable Diffusion, have exhibited remarkable performance in generating high-quality images from text descriptions in recent years. However, text-to-image models may be tricked into generating not-safe-for-work (NSFW) content, particularly in sexual scenarios. Existing countermeasures mostly focus on filtering inappropriate inputs and outputs, or suppressing improper text embeddings, which can block explicit NSFW-related content (e.g., naked or sexy) but may still be vulnerable to adversarial prompts inputs that appear innocent but are ill-intended. In this paper, we present SafeGen, a framework to mitigate unsafe content generation by text-to-image models in a text-agnostic manner. The key idea is to eliminate unsafe visual representations from the model regardless of the text input. In this way, the text-to-image model is resistant to adversarial prompts since unsafe visual representations are obstructed from within. Extensive experiments conducted on four datasets demonstrate SafeGen's effectiveness in mitigating unsafe content generation while preserving the high-fidelity of benign images. SafeGen outperforms eight state-of-the-art baseline methods and achieves 99.1% sexual content removal performance. Furthermore, our constructed benchmark of adversarial prompts provides a basis for future development and evaluation of anti-NSFW-generation methods.

Title: What's Mine becomes Yours: Defining, Annotating and Detecting Context-Dependent Paraphrases in News Interview Dialogs

• Authors: Anna Wegmann, Tijs van den Broek, Dong Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06670
• Pdf URL: https://arxiv.org/pdf/2404.06670
• Copy Paste: [[2404.06670]] What's Mine becomes Yours: Defining, Annotating and Detecting Context-Dependent Paraphrases in News Interview Dialogs(https://arxiv.org/abs/2404.06670)
• Keywords: in-context
• Abstract: Best practices for high conflict conversations like counseling or customer support almost always include recommendations to paraphrase the previous speaker. Although paraphrase classification has received widespread attention in NLP, paraphrases are usually considered independent from context, and common models and datasets are not applicable to dialog settings. In this work, we investigate paraphrases in dialog (e.g., Speaker 1: "That book is mine." becomes Speaker 2: "That book is yours."). We provide an operationalization of context-dependent paraphrases, and develop a training for crowd-workers to classify paraphrases in dialog. We introduce a dataset with utterance pairs from NPR and CNN news interviews annotated for context-dependent paraphrases. To enable analyses on label variation, the dataset contains 5,581 annotations on 600 utterance pairs. We present promising results with in-context learning and with token classification models for automatic paraphrase detection in dialog.

Title: Toward Cross-Layer Energy Optimizations in Machine Learning Systems

• Authors: Jae-Won Chung, Mosharaf Chowdhury
• Subjects: cs.LG, cs.AR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.06675
• Pdf URL: https://arxiv.org/pdf/2404.06675
• Copy Paste: [[2404.06675]] Toward Cross-Layer Energy Optimizations in Machine Learning Systems(https://arxiv.org/abs/2404.06675)
• Keywords: generative
• Abstract: The enormous energy consumption of machine learning (ML) and generative AI workloads shows no sign of waning, taking a toll on operating costs, power delivery, and environmental sustainability. Despite a long line of research on energy-efficient hardware, we found that software plays a critical role in ML energy optimization through two recent works: Zeus and Perseus. This is especially true for large language models (LLMs) because their model sizes and, therefore, energy demands are growing faster than hardware efficiency improvements. Therefore, we advocate for a cross-layer approach for energy optimizations in ML systems, where hardware provides architectural support that pushes energy-efficient software further, while software leverages and abstracts the hardware to develop techniques that bring hardware-agnostic energy-efficiency gains.

Title: Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncology

• Authors: Shashi Kant Gupta, Aditya Basu, Bradley Taylor, Anai Kothari, Hrituraj Singh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06680
• Pdf URL: https://arxiv.org/pdf/2404.06680
• Copy Paste: [[2404.06680]] Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncology(https://arxiv.org/abs/2404.06680)
• Keywords: generative
• Abstract: Retrieving information from EHR systems is essential for answering specific questions about patient journeys and improving the delivery of clinical care. Despite this fact, most EHR systems still rely on keyword-based searches. With the advent of generative large language models (LLMs), retrieving information can lead to better search and summarization capabilities. Such retrievers can also feed Retrieval-augmented generation (RAG) pipelines to answer any query. However, the task of retrieving information from EHR real-world clinical data contained within EHR systems in order to solve several downstream use cases is challenging due to the difficulty in creating query-document support pairs. We provide a blueprint for creating such datasets in an affordable manner using large language models. Our method results in a retriever that is 30-50 F-1 points better than propriety counterparts such as Ada and Mistral for oncology data elements. We further compare our model, called Onco-Retriever, against fine-tuned PubMedBERT model as well. We conduct an extensive manual evaluation on real-world EHR data along with latency analysis of the different models and provide a path forward for healthcare organizations to build domain-specific retrievers.

Title: How to Craft Backdoors with Unlabeled Data Alone?

• Authors: Yifei Wang, Wenhan Ma, Yisen Wang
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06694
• Pdf URL: https://arxiv.org/pdf/2404.06694
• Copy Paste: [[2404.06694]] How to Craft Backdoors with Unlabeled Data Alone?(https://arxiv.org/abs/2404.06694)
• Keywords: self-supervised, foundation model
• Abstract: Relying only on unlabeled data, Self-supervised learning (SSL) can learn rich features in an economical and scalable way. As the drive-horse for building foundation models, SSL has received a lot of attention recently with wide applications, which also raises security concerns where backdoor attack is a major type of threat: if the released dataset is maliciously poisoned, backdoored SSL models can behave badly when triggers are injected to test samples. The goal of this work is to investigate this potential risk. We notice that existing backdoors all require a considerable amount of \emph{labeled} data that may not be available for SSL. To circumvent this limitation, we explore a more restrictive setting called no-label backdoors, where we only have access to the unlabeled data alone, where the key challenge is how to select the proper poison set without using label information. We propose two strategies for poison selection: clustering-based selection using pseudolabels, and contrastive selection derived from the mutual information principle. Experiments on CIFAR-10 and ImageNet-100 show that both no-label backdoors are effective on many SSL methods and outperform random poisoning by a large margin. Code will be available at https://github.com/PKU-ML/nlb.

Title: Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting

• Authors: Hao Lu, Jiaqi Tang, Xinli Xu, Xu Cao, Yunpeng Zhang, Guoqing Wang, Dalong Du, Hao Chen, Yingcong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06700
• Pdf URL: https://arxiv.org/pdf/2404.06700
• Copy Paste: [[2404.06700]] Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting(https://arxiv.org/abs/2404.06700)
• Keywords: foundation model
• Abstract: The emergence of Multi-Camera 3D Object Detection (MC3D-Det), facilitated by bird's-eye view (BEV) representation, signifies a notable progression in 3D object detection. Scaling MC3D-Det training effectively accommodates varied camera parameters and urban landscapes, paving the way for the MC3D-Det foundation model. However, the multi-view fusion stage of the MC3D-Det method relies on the ill-posed monocular perception during training rather than surround refinement ability, leading to what we term "surround refinement degradation". To this end, our study presents a weak-to-strong eliciting framework aimed at enhancing surround refinement while maintaining robust monocular perception. Specifically, our framework employs weakly tuned experts trained on distinct subsets, and each is inherently biased toward specific camera configurations and scenarios. These biased experts can learn the perception of monocular degeneration, which can help the multi-view fusion stage to enhance surround refinement abilities. Moreover, a composite distillation strategy is proposed to integrate the universal knowledge of 2D foundation models and task-specific information. Finally, for MC3D-Det joint training, the elaborate dataset merge strategy is designed to solve the problem of inconsistent camera numbers and camera parameters. We set up a multiple dataset joint training benchmark for MC3D-Det and adequately evaluated existing methods. Further, we demonstrate the proposed framework brings a generalized and significant boost over multiple baselines. Our code is at \url{https://github.com/EnVision-Research/Scale-BEV}.

Title: Disguised Copyright Infringement of Latent Diffusion Model

• Authors: Yiwei Lu, Matthew Y.R. Yang, Zuoqiu Liu, Gautam Kamath, Yaoliang Yu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06737
• Pdf URL: https://arxiv.org/pdf/2404.06737
• Copy Paste: [[2404.06737]] Disguised Copyright Infringement of Latent Diffusion Model(https://arxiv.org/abs/2404.06737)
• Keywords: diffusion, generative
• Abstract: Copyright infringement may occur when a generative model produces samples substantially similar to some copyrighted data that it had access to during the training phase. The notion of access usually refers to including copyrighted samples directly in the training dataset, which one may inspect to identify an infringement. We argue that such visual auditing largely overlooks a concealed copyright infringement, where one constructs a disguise that looks drastically different from the copyrighted sample yet still induces the effect of training Latent Diffusion Models on it. Such disguises only require indirect access to the copyrighted material and cannot be visually distinguished, thus easily circumventing the current auditing tools. In this paper, we provide a better understanding of such disguised copyright infringement by uncovering the disguises generation algorithm, the revelation of the disguises, and importantly, how to detect them to augment the existing toolbox. Additionally, we introduce a broader notion of acknowledgment for comprehending such indirect access.

Title: MonoSelfRecon: Purely Self-Supervised Explicit Generalizable 3D Reconstruction of Indoor Scenes from Monocular RGB Views

• Authors: Runfa Li, Upal Mahbub, Vasudev Bhaskaran, Truong Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06753
• Pdf URL: https://arxiv.org/pdf/2404.06753
• Copy Paste: [[2404.06753]] MonoSelfRecon: Purely Self-Supervised Explicit Generalizable 3D Reconstruction of Indoor Scenes from Monocular RGB Views(https://arxiv.org/abs/2404.06753)
• Keywords: self-supervised
• Abstract: Current monocular 3D scene reconstruction (3DR) works are either fully-supervised, or not generalizable, or implicit in 3D representation. We propose a novel framework - MonoSelfRecon that for the first time achieves explicit 3D mesh reconstruction for generalizable indoor scenes with monocular RGB views by purely self-supervision on voxel-SDF (signed distance function). MonoSelfRecon follows an Autoencoder-based architecture, decodes voxel-SDF and a generalizable Neural Radiance Field (NeRF), which is used to guide voxel-SDF in self-supervision. We propose novel self-supervised losses, which not only support pure self-supervision, but can be used together with supervised signals to further boost supervised training. Our experiments show that "MonoSelfRecon" trained in pure self-supervision outperforms current best self-supervised indoor depth estimation models and is comparable to 3DR models trained in fully supervision with depth annotations. MonoSelfRecon is not restricted by specific model design, which can be used to any models with voxel-SDF for purely self-supervised manner.

Title: DiffusionDialog: A Diffusion Model for Diverse Dialog Generation with Latent Space

• Authors: Jianxiang Xiang, Zhenhua Liu, Haodong Liu, Yin Bai, Jia Cheng, Wenliang Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06760
• Pdf URL: https://arxiv.org/pdf/2404.06760
• Copy Paste: [[2404.06760]] DiffusionDialog: A Diffusion Model for Diverse Dialog Generation with Latent Space(https://arxiv.org/abs/2404.06760)
• Keywords: diffusion
• Abstract: In real-life conversations, the content is diverse, and there exists the one-to-many problem that requires diverse generation. Previous studies attempted to introduce discrete or Gaussian-based continuous latent variables to address the one-to-many problem, but the diversity is limited. Recently, diffusion models have made breakthroughs in computer vision, and some attempts have been made in natural language processing. In this paper, we propose DiffusionDialog, a novel approach to enhance the diversity of dialogue generation with the help of diffusion model. In our approach, we introduce continuous latent variables into the diffusion model. The problem of using latent variables in the dialog task is how to build both an effective prior of the latent space and an inferring process to obtain the proper latent given the context. By combining the encoder and latent-based diffusion model, we encode the response's latent representation in a continuous space as the prior, instead of fixed Gaussian distribution or simply discrete ones. We then infer the latent by denoising step by step with the diffusion model. The experimental results show that our model greatly enhances the diversity of dialog responses while maintaining coherence. Furthermore, in further analysis, we find that our diffusion model achieves high inference efficiency, which is the main challenge of applying diffusion models in natural language processing.

Title: Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior

• Authors: Fan Lu, Kwan-Yee Lin, Yan Xu, Hongsheng Li, Guang Chen, Changjun Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06780
• Pdf URL: https://arxiv.org/pdf/2404.06780
• Copy Paste: [[2404.06780]] Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior(https://arxiv.org/abs/2404.06780)
• Keywords: diffusion
• Abstract: Text-to-3D generation has achieved remarkable success via large-scale text-to-image diffusion models. Nevertheless, there is no paradigm for scaling up the methodology to urban scale. Urban scenes, characterized by numerous elements, intricate arrangement relationships, and vast scale, present a formidable barrier to the interpretability of ambiguous textual descriptions for effective model optimization. In this work, we surmount the limitations by introducing a compositional 3D layout representation into text-to-3D paradigm, serving as an additional prior. It comprises a set of semantic primitives with simple geometric structures and explicit arrangement relationships, complementing textual descriptions and enabling steerable generation. Upon this, we propose two modifications -- (1) We introduce Layout-Guided Variational Score Distillation to address model optimization inadequacies. It conditions the score distillation sampling process with geometric and semantic constraints of 3D layouts. (2) To handle the unbounded nature of urban scenes, we represent 3D scene with a Scalable Hash Grid structure, incrementally adapting to the growing scale of urban scenes. Extensive experiments substantiate the capability of our framework to scale text-to-3D generation to large-scale urban scenes that cover over 1000m driving distance for the first time. We also present various scene editing demonstrations, showing the powers of steerable urban scene generation. Website: https://urbanarchitect.github.io.

Title: Zero-shot Point Cloud Completion Via 2D Priors

• Authors: Tianxin Huang, Zhiwen Yan, Yuyang Zhao, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06814
• Pdf URL: https://arxiv.org/pdf/2404.06814
• Copy Paste: [[2404.06814]] Zero-shot Point Cloud Completion Via 2D Priors(https://arxiv.org/abs/2404.06814)
• Keywords: diffusion
• Abstract: 3D point cloud completion is designed to recover complete shapes from partially observed point clouds. Conventional completion methods typically depend on extensive point cloud data for training %, with their effectiveness often constrained to object categories similar to those seen during training. In contrast, we propose a zero-shot framework aimed at completing partially observed point clouds across any unseen categories. Leveraging point rendering via Gaussian Splatting, we develop techniques of Point Cloud Colorization and Zero-shot Fractal Completion that utilize 2D priors from pre-trained diffusion models to infer missing regions. Experimental results on both synthetic and real-world scanned point clouds demonstrate that our approach outperforms existing methods in completing a variety of objects without any requirement for specific training data.

Title: SplatPose & Detect: Pose-Agnostic 3D Anomaly Detection

• Authors: Mathis Kruse, Marco Rudolph, Dominik Woiwode, Bodo Rosenhahn
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06832
• Pdf URL: https://arxiv.org/pdf/2404.06832
• Copy Paste: [[2404.06832]] SplatPose & Detect: Pose-Agnostic 3D Anomaly Detection(https://arxiv.org/abs/2404.06832)
• Keywords: anomaly
• Abstract: Detecting anomalies in images has become a well-explored problem in both academia and industry. State-of-the-art algorithms are able to detect defects in increasingly difficult settings and data modalities. However, most current methods are not suited to address 3D objects captured from differing poses. While solutions using Neural Radiance Fields (NeRFs) have been proposed, they suffer from excessive computation requirements, which hinder real-world usability. For this reason, we propose the novel 3D Gaussian splatting-based framework SplatPose which, given multi-view images of a 3D object, accurately estimates the pose of unseen views in a differentiable manner, and detects anomalies in them. We achieve state-of-the-art results in both training and inference speed, and detection performance, even when using less training data than competing methods. We thoroughly evaluate our framework using the recently proposed Pose-agnostic Anomaly Detection benchmark and its multi-pose anomaly detection (MAD) data set.

Title: Tuning-Free Adaptive Style Incorporation for Structure-Consistent Text-Driven Style Transfer

• Authors: Yanqi Ge, Jiaqi Liu, Qingnan Fan, Xi Jiang, Ye Huang, Shuai Qin, Hong Gu, Wen Li, Lixin Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06835
• Pdf URL: https://arxiv.org/pdf/2404.06835
• Copy Paste: [[2404.06835]] Tuning-Free Adaptive Style Incorporation for Structure-Consistent Text-Driven Style Transfer(https://arxiv.org/abs/2404.06835)
• Keywords: diffusion
• Abstract: In this work, we target the task of text-driven style transfer in the context of text-to-image (T2I) diffusion models. The main challenge is consistent structure preservation while enabling effective style transfer effects. The past approaches in this field directly concatenate the content and style prompts for a prompt-level style injection, leading to unavoidable structure distortions. In this work, we propose a novel solution to the text-driven style transfer task, namely, Adaptive Style Incorporation~(ASI), to achieve fine-grained feature-level style incorporation. It consists of the Siamese Cross-Attention~(SiCA) to decouple the single-track cross-attention to a dual-track structure to obtain separate content and style features, and the Adaptive Content-Style Blending (AdaBlending) module to couple the content and style information from a structure-consistent manner. Experimentally, our method exhibits much better performance in both structure preservation and stylized effects.

Title: UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion

• Authors: Junsheng Zhou, Weiqi Zhang, Baorui Ma, Kanle Shi, Yu-Shen Liu, Zhizhong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06851
• Pdf URL: https://arxiv.org/pdf/2404.06851
• Copy Paste: [[2404.06851]] UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion(https://arxiv.org/abs/2404.06851)
• Keywords: diffusion
• Abstract: Diffusion models have shown remarkable results for image generation, editing and inpainting. Recent works explore diffusion models for 3D shape generation with neural implicit functions, i.e., signed distance function and occupancy function. However, they are limited to shapes with closed surfaces, which prevents them from generating diverse 3D real-world contents containing open surfaces. In this work, we present UDiFF, a 3D diffusion model for unsigned distance fields (UDFs) which is capable to generate textured 3D shapes with open surfaces from text conditions or unconditionally. Our key idea is to generate UDFs in spatial-frequency domain with an optimal wavelet transformation, which produces a compact representation space for UDF generation. Specifically, instead of selecting an appropriate wavelet transformation which requires expensive manual efforts and still leads to large information loss, we propose a data-driven approach to learn the optimal wavelet transformation for UDFs. We evaluate UDiFF to show our advantages by numerical and visual comparisons with the latest methods on widely used benchmarks. Page: https://weiqi-zhang.github.io/UDiFF.

Title: Fine color guidance in diffusion models and its application to image compression at extremely low bitrates

• Authors: Tom Bordin, Thomas Maugey
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06865
• Pdf URL: https://arxiv.org/pdf/2404.06865
• Copy Paste: [[2404.06865]] Fine color guidance in diffusion models and its application to image compression at extremely low bitrates(https://arxiv.org/abs/2404.06865)
• Keywords: diffusion
• Abstract: This study addresses the challenge of, without training or fine-tuning, controlling the global color aspect of images generated with a diffusion model. We rewrite the guidance equations to ensure that the outputs are closer to a known color map, and this without hindering the quality of the generation. Our method leads to new guidance equations. We show in the color guidance context that, the scaling of the guidance should not decrease but remains high throughout the diffusion process. In a second contribution, our guidance is applied in a compression framework, we combine both semantic and general color information on the image to decode the images at low cost. We show that our method is effective at improving fidelity and realism of compressed images at extremely low bit rates, when compared to other classical or more semantic oriented approaches.

Title: DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting

• Authors: Shijie Zhou, Zhiwen Fan, Dejia Xu, Haoran Chang, Pradyumna Chari, Tejas Bharadwaj, Suya You, Zhangyang Wang, Achuta Kadambi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06903
• Pdf URL: https://arxiv.org/pdf/2404.06903
• Copy Paste: [[2404.06903]] DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting(https://arxiv.org/abs/2404.06903)
• Keywords: diffusion, generative
• Abstract: The increasing demand for virtual reality applications has highlighted the significance of crafting immersive 3D assets. We present a text-to-3D 360$^{\circ}$ scene generation pipeline that facilitates the creation of comprehensive 360$^{\circ}$ scenes for in-the-wild environments in a matter of minutes. Our approach utilizes the generative power of a 2D diffusion model and prompt self-refinement to create a high-quality and globally coherent panoramic image. This image acts as a preliminary "flat" (2D) scene representation. Subsequently, it is lifted into 3D Gaussians, employing splatting techniques to enable real-time exploration. To produce consistent 3D geometry, our pipeline constructs a spatially coherent structure by aligning the 2D monocular depth into a globally optimized point cloud. This point cloud serves as the initial state for the centroids of 3D Gaussians. In order to address invisible issues inherent in single-view inputs, we impose semantic and geometric constraints on both synthesized and input camera views as regularizations. These guide the optimization of Gaussians, aiding in the reconstruction of unseen regions. In summary, our method offers a globally consistent 3D scene within a 360$^{\circ}$ perspective, providing an enhanced immersive experience over existing techniques. Project website at: this http URL

Title: FiP: a Fixed-Point Approach for Causal Generative Modeling

• Authors: Meyer Scetbon, Joel Jennings, Agrin Hilmkil, Cheng Zhang, Chao Ma
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.06969
• Pdf URL: https://arxiv.org/pdf/2404.06969
• Copy Paste: [[2404.06969]] FiP: a Fixed-Point Approach for Causal Generative Modeling(https://arxiv.org/abs/2404.06969)
• Keywords: generative
• Abstract: Modeling true world data-generating processes lies at the heart of empirical science. Structural Causal Models (SCMs) and their associated Directed Acyclic Graphs (DAGs) provide an increasingly popular answer to such problems by defining the causal generative process that transforms random noise into observations. However, learning them from observational data poses an ill-posed and NP-hard inverse problem in general. In this work, we propose a new and equivalent formalism that do not require DAGs to describe them, viewed as fixed-point problems on the causally ordered variables, and show three important cases where they can be uniquely recovered given the topological ordering (TO). To the best of our knowledge, we obtain the most general recovery results when the TO is known. Based on our theoretical findings, we design a two-stage causal generative model that first infers the causal order from observations in a zero-shot manner, thus by-passing the search, and then learns the generative fixed-point SCM on the ordered variables. To infer TOs from observations, we propose to amortize the learning of TOs on generated datasets by sequentially predicting the leaves of graphs seen during training. To learn fixed-point SCMs, we design a transformer-based architecture that exploits a new attention mechanism enabling the modeling of causal structures, and show that this parameterization is consistent with our formalism. Finally, we conduct an extensive evaluation of each method individually, and show that when combined, our model outperforms various baselines on generated out-of-distribution problems.

Title: Diffusion-based inpainting of incomplete Euclidean distance matrices of trajectories generated by a fractional Brownian motion

• Authors: Alexander Lobashev, Kirill Polovnikov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07029
• Pdf URL: https://arxiv.org/pdf/2404.07029
• Copy Paste: [[2404.07029]] Diffusion-based inpainting of incomplete Euclidean distance matrices of trajectories generated by a fractional Brownian motion(https://arxiv.org/abs/2404.07029)
• Keywords: diffusion, generative
• Abstract: Fractional Brownian trajectories (fBm) feature both randomness and strong scale-free correlations, challenging generative models to reproduce the intrinsic memory characterizing the underlying process. Here we test a diffusion probabilistic model on a specific dataset of corrupted images corresponding to incomplete Euclidean distance matrices of fBm at various memory exponents $H$. Our dataset implies uniqueness of the data imputation in the regime of low missing ratio, where the remaining partial graph is rigid, providing the ground truth for the inpainting. We find that the conditional diffusion generation stably reproduces the statistics of missing fBm-distributed distances for different values of $H$ exponent. Furthermore, while diffusion models have been recently shown to remember samples from the training database, we show that diffusion-based inpainting behaves qualitatively different from the database search with the increasing database size. Finally, we apply our fBm-trained diffusion model with $H=1/3$ for completion of chromosome distance matrices obtained in single-cell microscopy experiments, showing its superiority over the standard bioinformatics algorithms. Our source code is available on GitHub at https://github.com/alobashev/diffusion_fbm.

Title: Identification of Fine-grained Systematic Errors via Controlled Scene Generation

• Authors: Valentyn Boreiko, Matthias Hein, Jan Hendrik Metzen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07045
• Pdf URL: https://arxiv.org/pdf/2404.07045
• Copy Paste: [[2404.07045]] Identification of Fine-grained Systematic Errors via Controlled Scene Generation(https://arxiv.org/abs/2404.07045)
• Keywords: generative
• Abstract: Many safety-critical applications, especially in autonomous driving, require reliable object detectors. They can be very effectively assisted by a method to search for and identify potential failures and systematic errors before these detectors are deployed. Systematic errors are characterized by combinations of attributes such as object location, scale, orientation, and color, as well as the composition of their respective backgrounds. To identify them, one must rely on something other than real images from a test set because they do not account for very rare but possible combinations of attributes. To overcome this limitation, we propose a pipeline for generating realistic synthetic scenes with fine-grained control, allowing the creation of complex scenes with multiple objects. Our approach, BEV2EGO, allows for a realistic generation of the complete scene with road-contingent control that maps 2D bird's-eye view (BEV) scene configurations to a first-person view (EGO). In addition, we propose a benchmark for controlled scene generation to select the most appropriate generative outpainting model for BEV2EGO. We further use it to perform a systematic analysis of multiple state-of-the-art object detection models and discover differences between them.

Title: Implicit Multi-Spectral Transformer: An Lightweight and Effective Visible to Infrared Image Translation Model

• Authors: Yijia Chen, Pinghua Chen, Xiangxin Zhou, Yingtie Lei, Ziyang Zhou, Mingxian Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07072
• Pdf URL: https://arxiv.org/pdf/2404.07072
• Copy Paste: [[2404.07072]] Implicit Multi-Spectral Transformer: An Lightweight and Effective Visible to Infrared Image Translation Model(https://arxiv.org/abs/2404.07072)
• Keywords: generative
• Abstract: In the field of computer vision, visible light images often exhibit low contrast in low-light conditions, presenting a significant challenge. While infrared imagery provides a potential solution, its utilization entails high costs and practical limitations. Recent advancements in deep learning, particularly the deployment of Generative Adversarial Networks (GANs), have facilitated the transformation of visible light images to infrared images. However, these methods often experience unstable training phases and may produce suboptimal outputs. To address these issues, we propose a novel end-to-end Transformer-based model that efficiently converts visible light images into high-fidelity infrared images. Initially, the Texture Mapping Module and Color Perception Adapter collaborate to extract texture and color features from the visible light image. The Dynamic Fusion Aggregation Module subsequently integrates these features. Finally, the transformation into an infrared image is refined through the synergistic action of the Color Perception Adapter and the Enhanced Perception Attention mechanism. Comprehensive benchmarking experiments confirm that our model outperforms existing methods, producing infrared images of markedly superior quality, both qualitatively and quantitatively. Furthermore, the proposed model enables more effective downstream applications for infrared images than other methods.

Title: VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning

• Authors: Alexandros Xenos, Niki Maria Foteinopoulou, Ioanna Ntinou, Ioannis Patras, Georgios Tzimiropoulos
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.07078
• Pdf URL: https://arxiv.org/pdf/2404.07078
• Copy Paste: [[2404.07078]] VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning(https://arxiv.org/abs/2404.07078)
• Keywords: in-context
• Abstract: Recognising emotions in context involves identifying the apparent emotions of an individual, taking into account contextual cues from the surrounding scene. Previous approaches to this task have involved the design of explicit scene-encoding architectures or the incorporation of external scene-related information, such as captions. However, these methods often utilise limited contextual information or rely on intricate training pipelines. In this work, we leverage the groundbreaking capabilities of Vision-and-Large-Language Models (VLLMs) to enhance in-context emotion classification without introducing complexity to the training process in a two-stage approach. In the first stage, we propose prompting VLLMs to generate descriptions in natural language of the subject's apparent emotion relative to the visual context. In the second stage, the descriptions are used as contextual information and, along with the image input, are used to train a transformer-based architecture that fuses text and visual features before the final classification task. Our experimental results show that the text and image features have complementary information, and our fused architecture significantly outperforms the individual modalities without any complex training methods. We evaluate our approach on three different datasets, namely, EMOTIC, CAER-S, and BoLD, and achieve state-of-the-art or comparable accuracy across all datasets and metrics compared to much more complex approaches. The code will be made publicly available on github: https://github.com/NickyFot/EmoCommonSense.git

Title: LaTiM: Longitudinal representation learning in continuous-time models to predict disease progression

• Authors: Rachid Zeghlache, Pierre-Henri Conze, Mostafa El Habib Daho, Yihao Li, Hugo Le Boité, Ramin Tadayoni, Pascal Massin, Béatrice Cochener, Alireza Rezaei, Ikram Brahim, Gwenolé Quellec, Mathieu Lamard
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07091
• Pdf URL: https://arxiv.org/pdf/2404.07091
• Copy Paste: [[2404.07091]] LaTiM: Longitudinal representation learning in continuous-time models to predict disease progression(https://arxiv.org/abs/2404.07091)
• Keywords: self-supervised
• Abstract: This work proposes a novel framework for analyzing disease progression using time-aware neural ordinary differential equations (NODE). We introduce a "time-aware head" in a framework trained through self-supervised learning (SSL) to leverage temporal information in latent space for data augmentation. This approach effectively integrates NODEs with SSL, offering significant performance improvements compared to traditional methods that lack explicit temporal integration. We demonstrate the effectiveness of our strategy for diabetic retinopathy progression prediction using the OPHDIAT database. Compared to the baseline, all NODE architectures achieve statistically significant improvements in area under the ROC curve (AUC) and Kappa metrics, highlighting the efficacy of pre-training with SSL-inspired approaches. Additionally, our framework promotes stable training for NODEs, a commonly encountered challenge in time-aware modeling.

Title: What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation

• Authors: Aaditya K. Singh, Ted Moskovitz, Felix Hill, Stephanie C.Y. Chan, Andrew M. Saxe
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07129
• Pdf URL: https://arxiv.org/pdf/2404.07129
• Copy Paste: [[2404.07129]] What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation(https://arxiv.org/abs/2404.07129)
• Keywords: in-context
• Abstract: In-context learning is a powerful emergent ability in transformer models. Prior work in mechanistic interpretability has identified a circuit element that may be critical for in-context learning -- the induction head (IH), which performs a match-and-copy operation. During training of large transformers on natural language data, IHs emerge around the same time as a notable phase change in the loss. Despite the robust evidence for IHs and this interesting coincidence with the phase change, relatively little is known about the diversity and emergence dynamics of IHs. Why is there more than one IH, and how are they dependent on each other? Why do IHs appear all of a sudden, and what are the subcircuits that enable them to emerge? We answer these questions by studying IH emergence dynamics in a controlled setting by training on synthetic data. In doing so, we develop and share a novel optogenetics-inspired causal framework for modifying activations throughout training. Using this framework, we delineate the diverse and additive nature of IHs. By clamping subsets of activations throughout training, we then identify three underlying subcircuits that interact to drive IH formation, yielding the phase change. Furthermore, these subcircuits shed light on data-dependent properties of formation, such as phase change timing, already showing the promise of this more in-depth understanding of subcircuits that need to "go right" for an induction head.

Title: A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks

• Authors: Neel Mishra, Bamdev Mishra, Pratik Jawanpuria, Pawan Kumar
• Subjects: cs.LG, math.NA, math.OC
• Abstract URL: https://arxiv.org/abs/2404.07172
• Pdf URL: https://arxiv.org/pdf/2404.07172
• Copy Paste: [[2404.07172]] A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks(https://arxiv.org/abs/2404.07172)
• Keywords: generative
• Abstract: A novel first-order method is proposed for training generative adversarial networks (GANs). It modifies the Gauss-Newton method to approximate the min-max Hessian and uses the Sherman-Morrison inversion formula to calculate the inverse. The method corresponds to a fixed-point method that ensures necessary contraction. To evaluate its effectiveness, numerical experiments are conducted on various datasets commonly used in image generation tasks, such as MNIST, Fashion MNIST, CIFAR10, FFHQ, and LSUN. Our method is capable of generating high-fidelity images with greater diversity across multiple datasets. It also achieves the highest inception score for CIFAR10 among all compared methods, including state-of-the-art second-order methods. Additionally, its execution time is comparable to that of first-order min-max methods.

Title: Self-supervised Monocular Depth Estimation on Water Scenes via Specular Reflection Prior

• Authors: Zhengyang Lu, Ying Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07176
• Pdf URL: https://arxiv.org/pdf/2404.07176
• Copy Paste: [[2404.07176]] Self-supervised Monocular Depth Estimation on Water Scenes via Specular Reflection Prior(https://arxiv.org/abs/2404.07176)
• Keywords: self-supervised
• Abstract: Monocular depth estimation from a single image is an ill-posed problem for computer vision due to insufficient reliable cues as the prior knowledge. Besides the inter-frame supervision, namely stereo and adjacent frames, extensive prior information is available in the same frame. Reflections from specular surfaces, informative intra-frame priors, enable us to reformulate the ill-posed depth estimation task as a multi-view synthesis. This paper proposes the first self-supervision for deep-learning depth estimation on water scenes via intra-frame priors, known as reflection supervision and geometrical constraints. In the first stage, a water segmentation network is performed to separate the reflection components from the entire image. Next, we construct a self-supervised framework to predict the target appearance from reflections, perceived as other perspectives. The photometric re-projection error, incorporating SmoothL1 and a novel photometric adaptive SSIM, is formulated to optimize pose and depth estimation by aligning the transformed virtual depths and source ones. As a supplement, the water surface is determined from real and virtual camera positions, which complement the depth of the water area. Furthermore, to alleviate these laborious ground truth annotations, we introduce a large-scale water reflection scene (WRS) dataset rendered from Unreal Engine 4. Extensive experiments on the WRS dataset prove the feasibility of the proposed method compared to state-of-the-art depth estimation techniques.

Title: Move Anything with Layered Scene Diffusion

• Authors: Jiawei Ren, Mengmeng Xu, Jui-Chieh Wu, Ziwei Liu, Tao Xiang, Antoine Toisoul
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07178
• Pdf URL: https://arxiv.org/pdf/2404.07178
• Copy Paste: [[2404.07178]] Move Anything with Layered Scene Diffusion(https://arxiv.org/abs/2404.07178)
• Keywords: diffusion
• Abstract: Diffusion models generate images with an unprecedented level of quality, but how can we freely rearrange image layouts? Recent works generate controllable scenes via learning spatially disentangled latent codes, but these methods do not apply to diffusion models due to their fixed forward process. In this work, we propose SceneDiffusion to optimize a layered scene representation during the diffusion sampling process. Our key insight is that spatial disentanglement can be obtained by jointly denoising scene renderings at different spatial layouts. Our generated scenes support a wide range of spatial editing operations, including moving, resizing, cloning, and layer-wise appearance editing operations, including object restyling and replacing. Moreover, a scene can be generated conditioned on a reference image, thus enabling object moving for in-the-wild images. Notably, this approach is training-free, compatible with general text-to-image diffusion models, and responsive in less than a second.

Title: InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models

• Authors: Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07191
• Pdf URL: https://arxiv.org/pdf/2404.07191
• Copy Paste: [[2404.07191]] InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models(https://arxiv.org/abs/2404.07191)
• Keywords: diffusion, generative
• Abstract: We present InstantMesh, a feed-forward framework for instant 3D mesh generation from a single image, featuring state-of-the-art generation quality and significant training scalability. By synergizing the strengths of an off-the-shelf multiview diffusion model and a sparse-view reconstruction model based on the LRM architecture, InstantMesh is able to create diverse 3D assets within 10 seconds. To enhance the training efficiency and exploit more geometric supervisions, e.g, depths and normals, we integrate a differentiable iso-surface extraction module into our framework and directly optimize on the mesh representation. Experimental results on public datasets demonstrate that InstantMesh significantly outperforms other latest image-to-3D baselines, both qualitatively and quantitatively. We release all the code, weights, and demo of InstantMesh, with the intention that it can make substantial contributions to the community of 3D generative AI and empower both researchers and content creators.

Title: RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion

• Authors: Jaidev Shriram, Alex Trevithick, Lingjie Liu, Ravi Ramamoorthi
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07199
• Pdf URL: https://arxiv.org/pdf/2404.07199
• Copy Paste: [[2404.07199]] RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion(https://arxiv.org/abs/2404.07199)
• Keywords: diffusion
• Abstract: We introduce RealmDreamer, a technique for generation of general forward-facing 3D scenes from text descriptions. Our technique optimizes a 3D Gaussian Splatting representation to match complex text prompts. We initialize these splats by utilizing the state-of-the-art text-to-image generators, lifting their samples into 3D, and computing the occlusion volume. We then optimize this representation across multiple views as a 3D inpainting task with image-conditional diffusion models. To learn correct geometric structure, we incorporate a depth diffusion model by conditioning on the samples from the inpainting model, giving rich geometric structure. Finally, we finetune the model using sharpened samples from image generators. Notably, our technique does not require video or multi-view data and can synthesize a variety of high-quality 3D scenes in different styles, consisting of multiple objects. Its generality additionally allows 3D synthesis from a single image.

Title: GoodDrag: Towards Good Practices for Drag Editing with Diffusion Models

• Authors: Zewei Zhang, Huan Liu, Jun Chen, Xiangyu Xu
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.07206
• Pdf URL: https://arxiv.org/pdf/2404.07206
• Copy Paste: [[2404.07206]] GoodDrag: Towards Good Practices for Drag Editing with Diffusion Models(https://arxiv.org/abs/2404.07206)
• Keywords: diffusion
• Abstract: In this paper, we introduce GoodDrag, a novel approach to improve the stability and image quality of drag editing. Unlike existing methods that struggle with accumulated perturbations and often result in distortions, GoodDrag introduces an AlDD framework that alternates between drag and denoising operations within the diffusion process, effectively improving the fidelity of the result. We also propose an information-preserving motion supervision operation that maintains the original features of the starting point for precise manipulation and artifact reduction. In addition, we contribute to the benchmarking of drag editing by introducing a new dataset, Drag100, and developing dedicated quality assessment metrics, Dragging Accuracy Index and Gemini Score, utilizing Large Multimodal Models. Extensive experiments demonstrate that the proposed GoodDrag compares favorably against the state-of-the-art approaches both qualitatively and quantitatively. The project page is https://gooddrag.github.io.