2024-03-22

Title: Measuring Diversity in Co-creative Image Generation

• Authors: Francisco Ibarrola, Kazjon Grace
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.13826
• Pdf URL: https://arxiv.org/pdf/2403.13826
• Copy Paste: [[2403.13826]] Measuring Diversity in Co-creative Image Generation(https://arxiv.org/abs/2403.13826)
• Keywords: generative
• Abstract: Quality and diversity have been proposed as reasonable heuristics for assessing content generated by co-creative systems, but to date there has been little agreement around what constitutes the latter or how to measure it. Proposed approaches for assessing generative models in terms of diversity have limitations in that they compare the model's outputs to a ground truth that in the era of large pre-trained generative models might not be available, or entail an impractical number of computations. We propose an alternative based on entropy of neural network encodings for comparing diversity between sets of images that does not require ground-truth knowledge and is easy to compute. We also compare two pre-trained networks and show how the choice relates to the notion of diversity that we want to evaluate. We conclude with a discussion of the potential applications of these measures for ideation in interactive systems, model evaluation, and more broadly within computational creativity.

Title: Circuit Transformer: End-to-end Circuit Design by Predicting the Next Gate

• Authors: Xihan Li, Xing Li, Lei Chen, Xing Zhang, Mingxuan Yuan, Jun Wang
• Subjects: cs.LG, cs.AR, cs.LO
• Abstract URL: https://arxiv.org/abs/2403.13838
• Pdf URL: https://arxiv.org/pdf/2403.13838
• Copy Paste: [[2403.13838]] Circuit Transformer: End-to-end Circuit Design by Predicting the Next Gate(https://arxiv.org/abs/2403.13838)
• Keywords: generative
• Abstract: Language, a prominent human ability to express through sequential symbols, has been computationally mastered by recent advances of large language models (LLMs). By predicting the next word recurrently with huge neural models, LLMs have shown unprecedented capabilities in understanding and reasoning. Circuit, as the "language" of electronic design, specifies the functionality of an electronic device by cascade connections of logic gates. Then, can circuits also be mastered by a a sufficiently large "circuit model", which can conquer electronic design tasks by simply predicting the next logic gate? In this work, we take the first step to explore such possibilities. Two primary barriers impede the straightforward application of LLMs to circuits: their complex, non-sequential structure, and the intolerance of hallucination due to strict constraints (e.g., equivalence). For the first barrier, we encode a circuit as a memory-less, depth-first traversal trajectory, which allows Transformer-based neural models to better leverage its structural information, and predict the next gate on the trajectory as a circuit model. For the second barrier, we introduce an equivalence-preserving decoding process, which ensures that every token in the generated trajectory adheres to the specified equivalence constraints. Moreover, the circuit model can also be regarded as a stochastic policy to tackle optimization-oriented circuit design tasks. Experimentally, we trained a Transformer-based model of 88M parameters, named "Circuit Transformer", which demonstrates impressive performance in end-to-end logic synthesis. With Monte-Carlo tree search, Circuit Transformer significantly improves over resyn2 while retaining strict equivalence, showcasing the potential of generative AI in conquering electronic design challenges.

Title: Learning to better see the unseen: Broad-Deep Mixed Anti-Forgetting Framework for Incremental Zero-Shot Fault Diagnosis

• Authors: Jiancheng Zhao, Jiaqi Yue, Chunhui Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.13845
• Pdf URL: https://arxiv.org/pdf/2403.13845
• Copy Paste: [[2403.13845]] Learning to better see the unseen: Broad-Deep Mixed Anti-Forgetting Framework for Incremental Zero-Shot Fault Diagnosis(https://arxiv.org/abs/2403.13845)
• Keywords: generative
• Abstract: Zero-shot fault diagnosis (ZSFD) is capable of identifying unseen faults via predicting fault attributes labeled by human experts. We first recognize the demand of ZSFD to deal with continuous changes in industrial processes, i.e., the model's ability to adapt to new fault categories and attributes while avoiding forgetting the diagnosis ability learned previously. To overcome the issue that the existing ZSFD paradigm cannot learn from evolving streams of training data in industrial scenarios, the incremental ZSFD (IZSFD) paradigm is proposed for the first time, which incorporates category increment and attribute increment for both traditional ZSFD and generalized ZSFD paradigms. To achieve IZSFD, we present a broad-deep mixed anti-forgetting framework (BDMAFF) that aims to learn from new fault categories and attributes. To tackle the issue of forgetting, BDMAFF effectively accumulates previously acquired knowledge from two perspectives: features and attribute prototypes. The feature memory is established through a deep generative model that employs anti-forgetting training strategies, ensuring the generation quality of historical categories is supervised and maintained. The diagnosis model SEEs the UNSEEN faults with the help of generated samples from the generative model. The attribute prototype memory is established through a diagnosis model inspired by the broad learning system. Unlike traditional incremental learning algorithms, BDMAFF introduces a memory-driven iterative update strategy for the diagnosis model, which allows the model to learn new faults and attributes without requiring the storage of all historical training samples. The effectiveness of the proposed method is verified by a real hydraulic system and the Tennessee-Eastman benchmark process.

Title: Spatio-Temporal Fluid Dynamics Modeling via Physical-Awareness and Parameter Diffusion Guidance

• Authors: Hao Wu, Fan Xu, Yifan Duan, Ziwei Niu, Weiyan Wang, Gaofeng Lu, Kun Wang, Yuxuan Liang, Yang Wang
• Subjects: cs.LG, cs.AI, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2403.13850
• Pdf URL: https://arxiv.org/pdf/2403.13850
• Copy Paste: [[2403.13850]] Spatio-Temporal Fluid Dynamics Modeling via Physical-Awareness and Parameter Diffusion Guidance(https://arxiv.org/abs/2403.13850)
• Keywords: diffusion
• Abstract: This paper proposes a two-stage framework named ST-PAD for spatio-temporal fluid dynamics modeling in the field of earth sciences, aiming to achieve high-precision simulation and prediction of fluid dynamics through spatio-temporal physics awareness and parameter diffusion guidance. In the upstream stage, we design a vector quantization reconstruction module with temporal evolution characteristics, ensuring balanced and resilient parameter distribution by introducing general physical constraints. In the downstream stage, a diffusion probability network involving parameters is utilized to generate high-quality future states of fluids, while enhancing the model's generalization ability by perceiving parameters in various physical setups. Extensive experiments on multiple benchmark datasets have verified the effectiveness and robustness of the ST-PAD framework, which showcase that ST-PAD outperforms current mainstream models in fluid dynamics modeling and prediction, especially in effectively capturing local representations and maintaining significant advantages in OOD generations.

Title: Machine Learning-based Layer-wise Detection of Overheating Anomaly in LPBF using Photodiode Data

• Authors: Nazmul Hasan, Apurba Kumar Saha, Andrew Wessman, Mohammed Shafae
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2403.13861
• Pdf URL: https://arxiv.org/pdf/2403.13861
• Copy Paste: [[2403.13861]] Machine Learning-based Layer-wise Detection of Overheating Anomaly in LPBF using Photodiode Data(https://arxiv.org/abs/2403.13861)
• Keywords: anomaly
• Abstract: Overheating anomaly detection is essential for the quality and reliability of parts produced by laser powder bed fusion (LPBF) additive manufacturing (AM). In this research, we focus on the detection of overheating anomalies using photodiode sensor data. Photodiode sensors can collect high-frequency data from the melt pool, reflecting the process dynamics and thermal history. Hence, the proposed method offers a machine learning (ML) framework to utilize photodiode sensor data for layer-wise detection of overheating anomalies. In doing so, three sets of features are extracted from the raw photodiode data: MSMM (mean, standard deviation, median, maximum), MSQ (mean, standard deviation, quartiles), and MSD (mean, standard deviation, deciles). These three datasets are used to train several ML classifiers. Cost-sensitive learning is used to handle the class imbalance between the "anomalous" layers (affected by overheating) and "nominal" layers in the benchmark dataset. To boost detection accuracy, our proposed ML framework involves utilizing the majority voting ensemble (MVE) approach. The proposed method is demonstrated using a case study including an open benchmark dataset of photodiode measurements from an LPBF specimen with deliberate overheating anomalies at some layers. The results from the case study demonstrate that the MSD features yield the best performance for all classifiers, and the MVE classifier (with a mean F1-score of 0.8654) surpasses the individual ML classifiers. Moreover, our machine learning methodology achieves superior results (9.66% improvement in mean F1-score) in detecting layer-wise overheating anomalies, surpassing the existing methods in the literature that use the same benchmark dataset.

Title: DiffImpute: Tabular Data Imputation With Denoising Diffusion Probabilistic Model

• Authors: Yizhu Wen, Kai Yi, Jing Ke, Yiqing Shen
• Subjects: cs.LG, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2403.13863
• Pdf URL: https://arxiv.org/pdf/2403.13863
• Copy Paste: [[2403.13863]] DiffImpute: Tabular Data Imputation With Denoising Diffusion Probabilistic Model(https://arxiv.org/abs/2403.13863)
• Keywords: diffusion
• Abstract: Tabular data plays a crucial role in various domains but often suffers from missing values, thereby curtailing its potential utility. Traditional imputation techniques frequently yield suboptimal results and impose substantial computational burdens, leading to inaccuracies in subsequent modeling tasks. To address these challenges, we propose DiffImpute, a novel Denoising Diffusion Probabilistic Model (DDPM). Specifically, DiffImpute is trained on complete tabular datasets, ensuring that it can produce credible imputations for missing entries without undermining the authenticity of the existing data. Innovatively, it can be applied to various settings of Missing Completely At Random (MCAR) and Missing At Random (MAR). To effectively handle the tabular features in DDPM, we tailor four tabular denoising networks, spanning MLP, ResNet, Transformer, and U-Net. We also propose Harmonization to enhance coherence between observed and imputed data by infusing the data back and denoising them multiple times during the sampling stage. To enable efficient inference while maintaining imputation performance, we propose a refined non-Markovian sampling process that works along with Harmonization. Empirical evaluations on seven diverse datasets underscore the prowess of DiffImpute. Specifically, when paired with the Transformer as the denoising network, it consistently outperforms its competitors, boasting an average ranking of 1.7 and the most minimal standard deviation. In contrast, the next best method lags with a ranking of 2.8 and a standard deviation of 0.9. The code is available at https://github.com/Dendiiiii/DiffImpute.

Title: The Bid Picture: Auction-Inspired Multi-player Generative Adversarial Networks Training

• Authors: Joo Yong Shim, Jean Seong Bjorn Choe, Jong-Kook Kim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.13866
• Pdf URL: https://arxiv.org/pdf/2403.13866
• Copy Paste: [[2403.13866]] The Bid Picture: Auction-Inspired Multi-player Generative Adversarial Networks Training(https://arxiv.org/abs/2403.13866)
• Keywords: generative
• Abstract: This article proposes auction-inspired multi-player generative adversarial networks training, which mitigates the mode collapse problem of GANs. Mode collapse occurs when an over-fitted generator generates a limited range of samples, often concentrating on a small subset of the data distribution. Despite the restricted diversity of generated samples, the discriminator can still be deceived into distinguishing these samples as real samples from the actual distribution. In the absence of external standards, a model cannot recognize its failure during the training phase. We extend the two-player game of generative adversarial networks to the multi-player game. During the training, the values of each model are determined by the bids submitted by other players in an auction-like process.

Title: Enhancing Fingerprint Image Synthesis with GANs, Diffusion Models, and Style Transfer Techniques

• Authors: W. Tang, D. Figueroa, D. Liu, K. Johnsson, A. Sopasakis
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.13916
• Pdf URL: https://arxiv.org/pdf/2403.13916
• Copy Paste: [[2403.13916]] Enhancing Fingerprint Image Synthesis with GANs, Diffusion Models, and Style Transfer Techniques(https://arxiv.org/abs/2403.13916)
• Keywords: diffusion, generative
• Abstract: We present novel approaches involving generative adversarial networks and diffusion models in order to synthesize high quality, live and spoof fingerprint images while preserving features such as uniqueness and diversity. We generate live fingerprints from noise with a variety of methods, and we use image translation techniques to translate live fingerprint images to spoof. To generate different types of spoof images based on limited training data we incorporate style transfer techniques through a cycle autoencoder equipped with a Wasserstein metric along with Gradient Penalty (CycleWGAN-GP) in order to avoid mode collapse and instability. We find that when the spoof training data includes distinct spoof characteristics, it leads to improved live-to-spoof translation. We assess the diversity and realism of the generated live fingerprint images mainly through the Fr\'echet Inception Distance (FID) and the False Acceptance Rate (FAR). Our best diffusion model achieved a FID of 15.78. The comparable WGAN-GP model achieved slightly higher FID while performing better in the uniqueness assessment due to a slightly lower FAR when matched against the training data, indicating better creativity. Moreover, we give example images showing that a DDPM model clearly can generate realistic fingerprint images.

Title: ACDG-VTON: Accurate and Contained Diffusion Generation for Virtual Try-On

• Authors: Jeffrey Zhang, Kedan Li, Shao-Yu Chang, David Forsyth
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.13951
• Pdf URL: https://arxiv.org/pdf/2403.13951
• Copy Paste: [[2403.13951]] ACDG-VTON: Accurate and Contained Diffusion Generation for Virtual Try-On(https://arxiv.org/abs/2403.13951)
• Keywords: diffusion
• Abstract: Virtual Try-on (VTON) involves generating images of a person wearing selected garments. Diffusion-based methods, in particular, can create high-quality images, but they struggle to maintain the identities of the input garments. We identified this problem stems from the specifics in the training formulation for diffusion. To address this, we propose a unique training scheme that limits the scope in which diffusion is trained. We use a control image that perfectly aligns with the target image during training. In turn, this accurately preserves garment details during inference. We demonstrate our method not only effectively conserves garment details but also allows for layering, styling, and shoe try-on. Our method runs multi-garment try-on in a single inference cycle and can support high-quality zoomed-in generations without training in higher resolutions. Finally, we show our method surpasses prior methods in accuracy and quality.

Title: SeFFeC: Semantic Facial Feature Control for Fine-grained Face Editing

• Authors: Florian Strohm, Mihai Bâce, Markus Kaltenecker, Andreas Bulling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.13972
• Pdf URL: https://arxiv.org/pdf/2403.13972
• Copy Paste: [[2403.13972]] SeFFeC: Semantic Facial Feature Control for Fine-grained Face Editing(https://arxiv.org/abs/2403.13972)
• Keywords: generative
• Abstract: We propose Semantic Facial Feature Control (SeFFeC) - a novel method for fine-grained face shape editing. Our method enables the manipulation of human-understandable, semantic face features, such as nose length or mouth width, which are defined by different groups of facial landmarks. In contrast to existing methods, the use of facial landmarks enables precise measurement of the facial features, which then enables training SeFFeC without any manually annotated labels. SeFFeC consists of a transformer-based encoder network that takes a latent vector of a pre-trained generative model and a facial feature embedding as input, and learns to modify the latent vector to perform the desired face edit operation. To ensure that the desired feature measurement is changed towards the target value without altering uncorrelated features, we introduced a novel semantic face feature loss. Qualitative and quantitative results show that SeFFeC enables precise and fine-grained control of 23 facial features, some of which could not previously be controlled by other methods, without requiring manual annotations. Unlike existing methods, SeFFeC also provides deterministic control over the exact values of the facial features and more localised and disentangled face edits.

Title: Multi-Modal Hallucination Control by Visual Information Grounding

• Authors: Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, Stefano Soatto
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14003
• Pdf URL: https://arxiv.org/pdf/2403.14003
• Copy Paste: [[2403.14003]] Multi-Modal Hallucination Control by Visual Information Grounding(https://arxiv.org/abs/2403.14003)
• Keywords: generative
• Abstract: Generative Vision-Language Models (VLMs) are prone to generate plausible-sounding textual answers that, however, are not always grounded in the input image. We investigate this phenomenon, usually referred to as "hallucination" and show that it stems from an excessive reliance on the language prior. In particular, we show that as more tokens are generated, the reliance on the visual prompt decreases, and this behavior strongly correlates with the emergence of hallucinations. To reduce hallucinations, we introduce Multi-Modal Mutual-Information Decoding (M3ID), a new sampling method for prompt amplification. M3ID amplifies the influence of the reference image over the language prior, hence favoring the generation of tokens with higher mutual information with the visual prompt. M3ID can be applied to any pre-trained autoregressive VLM at inference time without necessitating further training and with minimal computational overhead. If training is an option, we show that M3ID can be paired with Direct Preference Optimization (DPO) to improve the model's reliance on the prompt image without requiring any labels. Our empirical findings show that our algorithms maintain the fluency and linguistic capabilities of pre-trained VLMs while reducing hallucinations by mitigating visually ungrounded answers. Specifically, for the LLaVA 13B model, M3ID and M3ID+DPO reduce the percentage of hallucinated objects in captioning tasks by 25% and 28%, respectively, and improve the accuracy on VQA benchmarks such as POPE by 21% and 24%.

Title: On Prompt Sensitivity of ChatGPT in Affective Computing

• Authors: Mostafa M. Amin, Björn W. Schuller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14006
• Pdf URL: https://arxiv.org/pdf/2403.14006
• Copy Paste: [[2403.14006]] On Prompt Sensitivity of ChatGPT in Affective Computing(https://arxiv.org/abs/2403.14006)
• Keywords: foundation model
• Abstract: Recent studies have demonstrated the emerging capabilities of foundation models like ChatGPT in several fields, including affective computing. However, accessing these emerging capabilities is facilitated through prompt engineering. Despite the existence of some prompting techniques, the field is still rapidly evolving and many prompting ideas still require investigation. In this work, we introduce a method to evaluate and investigate the sensitivity of the performance of foundation models based on different prompts or generation parameters. We perform our evaluation on ChatGPT within the scope of affective computing on three major problems, namely sentiment analysis, toxicity detection, and sarcasm detection. First, we carry out a sensitivity analysis on pivotal parameters in auto-regressive text generation, specifically the temperature parameter $T$ and the top-$p$ parameter in Nucleus sampling, dictating how conservative or creative the model should be during generation. Furthermore, we explore the efficacy of several prompting ideas, where we explore how giving different incentives or structures affect the performance. Our evaluation takes into consideration performance measures on the affective computing tasks, and the effectiveness of the model to follow the stated instructions, hence generating easy-to-parse responses to be smoothly used in downstream applications.

Title: Semantics from Space: Satellite-Guided Thermal Semantic Segmentation Annotation for Aerial Field Robots

• Authors: Connor Lee, Saraswati Soedarmadji, Matthew Anderson, Anthony J. Clark, Soon-Jo Chung
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.14056
• Pdf URL: https://arxiv.org/pdf/2403.14056
• Copy Paste: [[2403.14056]] Semantics from Space: Satellite-Guided Thermal Semantic Segmentation Annotation for Aerial Field Robots(https://arxiv.org/abs/2403.14056)
• Keywords: foundation model
• Abstract: We present a new method to automatically generate semantic segmentation annotations for thermal imagery captured from an aerial vehicle by utilizing satellite-derived data products alongside onboard global positioning and attitude estimates. This new capability overcomes the challenge of developing thermal semantic perception algorithms for field robots due to the lack of annotated thermal field datasets and the time and costs of manual annotation, enabling precise and rapid annotation of thermal data from field collection efforts at a massively-parallelizable scale. By incorporating a thermal-conditioned refinement step with visual foundation models, our approach can produce highly-precise semantic segmentation labels using low-resolution satellite land cover data for little-to-no cost. It achieves 98.5% of the performance from using costly high-resolution options and demonstrates between 70-160% improvement over popular zero-shot semantic segmentation methods based on large vision-language models currently used for generating annotations for RGB imagery. Code will be available at: https://github.com/connorlee77/aerial-auto-segment.

Title: DiffSTOCK: Probabilistic relational Stock Market Predictions using Diffusion Models

• Authors: Divyanshu Daiya, Monika Yadav, Harshit Singh Rao
• Subjects: cs.LG, cs.CE, q-fin.CP, q-fin.PM
• Abstract URL: https://arxiv.org/abs/2403.14063
• Pdf URL: https://arxiv.org/pdf/2403.14063
• Copy Paste: [[2403.14063]] DiffSTOCK: Probabilistic relational Stock Market Predictions using Diffusion Models(https://arxiv.org/abs/2403.14063)
• Keywords: diffusion
• Abstract: In this work, we propose an approach to generalize denoising diffusion probabilistic models for stock market predictions and portfolio management. Present works have demonstrated the efficacy of modeling interstock relations for market time-series forecasting and utilized Graph-based learning models for value prediction and portfolio management. Though convincing, these deterministic approaches still fall short of handling uncertainties i.e., due to the low signal-to-noise ratio of the financial data, it is quite challenging to learn effective deterministic models. Since the probabilistic methods have shown to effectively emulate higher uncertainties for time-series predictions. To this end, we showcase effective utilisation of Denoising Diffusion Probabilistic Models (DDPM), to develop an architecture for providing better market predictions conditioned on the historical financial indicators and inter-stock relations. Additionally, we also provide a novel deterministic architecture MaTCHS which uses Masked Relational Transformer(MRT) to exploit inter-stock relations along with historical stock features. We demonstrate that our model achieves SOTA performance for movement predication and Portfolio management.

Title: EventDance: Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition

• Authors: Xu Zheng, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14082
• Pdf URL: https://arxiv.org/pdf/2403.14082
• Copy Paste: [[2403.14082]] EventDance: Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition(https://arxiv.org/abs/2403.14082)
• Keywords: self-supervised
• Abstract: In this paper, we make the first attempt at achieving the cross-modal (i.e., image-to-events) adaptation for event-based object recognition without accessing any labeled source image data owning to privacy and commercial issues. Tackling this novel problem is non-trivial due to the novelty of event cameras and the distinct modality gap between images and events. In particular, as only the source model is available, a hurdle is how to extract the knowledge from the source model by only using the unlabeled target event data while achieving knowledge transfer. To this end, we propose a novel framework, dubbed EventDance for this unsupervised source-free cross-modal adaptation problem. Importantly, inspired by event-to-video reconstruction methods, we propose a reconstruction-based modality bridging (RMB) module, which reconstructs intensity frames from events in a self-supervised manner. This makes it possible to build up the surrogate images to extract the knowledge (i.e., labels) from the source model. We then propose a multi-representation knowledge adaptation (MKA) module that transfers the knowledge to target models learning events with multiple representation types for fully exploring the spatiotemporal information of events. The two modules connecting the source and target models are mutually updated so as to achieve the best performance. Experiments on three benchmark datasets with two adaption settings show that EventDance is on par with prior methods utilizing the source data.

Title: MaskSAM: Towards Auto-prompt SAM with Mask Classification for Medical Image Segmentation

• Authors: Bin Xie, Hao Tang, Bin Duan, Dawen Cai, Yan Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14103
• Pdf URL: https://arxiv.org/pdf/2403.14103
• Copy Paste: [[2403.14103]] MaskSAM: Towards Auto-prompt SAM with Mask Classification for Medical Image Segmentation(https://arxiv.org/abs/2403.14103)
• Keywords: foundation model
• Abstract: Segment Anything Model~(SAM), a prompt-driven foundation model for natural image segmentation, has demonstrated impressive zero-shot performance. However, SAM does not work when directly applied to medical image segmentation tasks, since SAM lacks the functionality to predict semantic labels for predicted masks and needs to provide extra prompts, such as points or boxes, to segment target regions. Meanwhile, there is a huge gap between 2D natural images and 3D medical images, so the performance of SAM is imperfect for medical image segmentation tasks. Following the above issues, we propose MaskSAM, a novel mask classification prompt-free SAM adaptation framework for medical image segmentation. We design a prompt generator combined with the image encoder in SAM to generate a set of auxiliary classifier tokens, auxiliary binary masks, and auxiliary bounding boxes. Each pair of auxiliary mask and box prompts, which can solve the requirements of extra prompts, is associated with class label predictions by the sum of the auxiliary classifier token and the learnable global classifier tokens in the mask decoder of SAM to solve the predictions of semantic labels. Meanwhile, we design a 3D depth-convolution adapter for image embeddings and a 3D depth-MLP adapter for prompt embeddings. We inject one of them into each transformer block in the image encoder and mask decoder to enable pre-trained 2D SAM models to extract 3D information and adapt to 3D medical images. Our method achieves state-of-the-art performance on AMOS2022, 90.52% Dice, which improved by 2.7% compared to nnUNet. Our method surpasses nnUNet by 1.7% on ACDC and 1.0% on Synapse datasets.

Title: External Knowledge Enhanced 3D Scene Generation from Sketch

• Authors: Zijie Wu, Mingtao Feng, Yaonan Wang, He Xie, Weisheng Dong, Bo Miao, Ajmal Mian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14121
• Pdf URL: https://arxiv.org/pdf/2403.14121
• Copy Paste: [[2403.14121]] External Knowledge Enhanced 3D Scene Generation from Sketch(https://arxiv.org/abs/2403.14121)
• Keywords: diffusion
• Abstract: Generating realistic 3D scenes is challenging due to the complexity of room layouts and object geometries.We propose a sketch based knowledge enhanced diffusion architecture (SEK) for generating customized, diverse, and plausible 3D scenes. SEK conditions the denoising process with a hand-drawn sketch of the target scene and cues from an object relationship knowledge base. We first construct an external knowledge base containing object relationships and then leverage knowledge enhanced graph reasoning to assist our model in understanding hand-drawn sketches. A scene is represented as a combination of 3D objects and their relationships, and then incrementally diffused to reach a Gaussian distribution.We propose a 3D denoising scene transformer that learns to reverse the diffusion process, conditioned by a hand-drawn sketch along with knowledge cues, to regressively generate the scene including the 3D object instances as well as their layout. Experiments on the 3D-FRONT dataset show that our model improves FID, CKL by 17.41%, 37.18% in 3D scene generation and FID, KID by 19.12%, 20.06% in 3D scene completion compared to the nearest competitor DiffuScene.

Title: 3D Object Detection from Point Cloud via Voting Step Diffusion

• Authors: Haoran Hou, Mingtao Feng, Zijie Wu, Weisheng Dong, Qing Zhu, Yaonan Wang, Ajmal Mian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14133
• Pdf URL: https://arxiv.org/pdf/2403.14133
• Copy Paste: [[2403.14133]] 3D Object Detection from Point Cloud via Voting Step Diffusion(https://arxiv.org/abs/2403.14133)
• Keywords: diffusion
• Abstract: 3D object detection is a fundamental task in scene understanding. Numerous research efforts have been dedicated to better incorporate Hough voting into the 3D object detection pipeline. However, due to the noisy, cluttered, and partial nature of real 3D scans, existing voting-based methods often receive votes from the partial surfaces of individual objects together with severe noises, leading to sub-optimal detection performance. In this work, we focus on the distributional properties of point clouds and formulate the voting process as generating new points in the high-density region of the distribution of object centers. To achieve this, we propose a new method to move random 3D points toward the high-density region of the distribution by estimating the score function of the distribution with a noise conditioned score network. Specifically, we first generate a set of object center proposals to coarsely identify the high-density region of the object center distribution. To estimate the score function, we perturb the generated object center proposals by adding normalized Gaussian noise, and then jointly estimate the score function of all perturbed distributions. Finally, we generate new votes by moving random 3D points to the high-density region of the object center distribution according to the estimated score function. Extensive experiments on two large scale indoor 3D scene datasets, SUN RGB-D and ScanNet V2, demonstrate the superiority of our proposed method. The code will be released at https://github.com/HHrEtvP/DiffVote.

Title: Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition

• Authors: Sihyun Yu, Weili Nie, De-An Huang, Boyi Li, Jinwoo Shin, Anima Anandkumar
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14148
• Pdf URL: https://arxiv.org/pdf/2403.14148
• Copy Paste: [[2403.14148]] Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition(https://arxiv.org/abs/2403.14148)
• Keywords: diffusion
• Abstract: Video diffusion models have recently made great progress in generation quality, but are still limited by the high memory and computational requirements. This is because current video diffusion models often attempt to process high-dimensional videos directly. To tackle this issue, we propose content-motion latent diffusion model (CMD), a novel efficient extension of pretrained image diffusion models for video generation. Specifically, we propose an autoencoder that succinctly encodes a video as a combination of a content frame (like an image) and a low-dimensional motion latent representation. The former represents the common content, and the latter represents the underlying motion in the video, respectively. We generate the content frame by fine-tuning a pretrained image diffusion model, and we generate the motion latent representation by training a new lightweight diffusion model. A key innovation here is the design of a compact latent space that can directly utilizes a pretrained image diffusion model, which has not been done in previous latent video diffusion models. This leads to considerably better quality generation and reduced computational costs. For instance, CMD can sample a video 7.7$\times$ faster than prior approaches by generating a video of 512$\times$1024 resolution and length 16 in 3.1 seconds. Moreover, CMD achieves an FVD score of 212.7 on WebVid-10M, 27.3% better than the previous state-of-the-art of 292.4.

Title: Deep Learning for Trajectory Data Management and Mining: A Survey and Beyond

• Authors: Wei Chen, Yuxuan Liang, Yuanshao Zhu, Yanchuan Chang, Kang Luo, Haomin Wen, Lei Li, Yanwei Yu, Qingsong Wen, Chao Chen, Kai Zheng, Yunjun Gao, Xiaofang Zhou, Yu Zheng
• Subjects: cs.LG, cs.AI, cs.CY, cs.DB
• Abstract URL: https://arxiv.org/abs/2403.14151
• Pdf URL: https://arxiv.org/pdf/2403.14151
• Copy Paste: [[2403.14151]] Deep Learning for Trajectory Data Management and Mining: A Survey and Beyond(https://arxiv.org/abs/2403.14151)
• Keywords: anomaly
• Abstract: Trajectory computing is a pivotal domain encompassing trajectory data management and mining, garnering widespread attention due to its crucial role in various practical applications such as location services, urban traffic, and public safety. Traditional methods, focusing on simplistic spatio-temporal features, face challenges of complex calculations, limited scalability, and inadequate adaptability to real-world complexities. In this paper, we present a comprehensive review of the development and recent advances in deep learning for trajectory computing (DL4Traj). We first define trajectory data and provide a brief overview of widely-used deep learning models. Systematically, we explore deep learning applications in trajectory management (pre-processing, storage, analysis, and visualization) and mining (trajectory-related forecasting, trajectory-related recommendation, trajectory classification, travel time estimation, anomaly detection, and mobility generation). Notably, we encapsulate recent advancements in Large Language Models (LLMs) that hold the potential to augment trajectory computing. Additionally, we summarize application scenarios, public datasets, and toolkits. Finally, we outline current challenges in DL4Traj research and propose future directions. Relevant papers and open-source resources have been collated and are continuously updated at: \href{https://github.com/yoshall/Awesome-Trajectory-Computing}{DL4Traj Repo}.

Title: Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization

• Authors: Yeji Song, Jimyeong Kim, Wonhark Park, Wonsik Shin, Wonjong Rhee, Nojun Kwak
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14155
• Pdf URL: https://arxiv.org/pdf/2403.14155
• Copy Paste: [[2403.14155]] Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization(https://arxiv.org/abs/2403.14155)
• Keywords: diffusion
• Abstract: In a surge of text-to-image (T2I) models and their customization methods that generate new images of a user-provided subject, current works focus on alleviating the costs incurred by a lengthy per-subject optimization. These zero-shot customization methods encode the image of a specified subject into a visual embedding which is then utilized alongside the textual embedding for diffusion guidance. The visual embedding incorporates intrinsic information about the subject, while the textual embedding provides a new, transient context. However, the existing methods often 1) are significantly affected by the input images, eg., generating images with the same pose, and 2) exhibit deterioration in the subject's identity. We first pin down the problem and show that redundant pose information in the visual embedding interferes with the textual embedding containing the desired pose information. To address this issue, we propose orthogonal visual embedding which effectively harmonizes with the given textual embedding. We also adopt the visual-only embedding and inject the subject's clear features utilizing a self-attention swap. Our results demonstrate the effectiveness and robustness of our method, which offers highly flexible zero-shot generation while effectively maintaining the subject's identity.

Title: Policy Mirror Descent with Lookahead

• Authors: Kimon Protopapas, Anas Barakat
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.14156
• Pdf URL: https://arxiv.org/pdf/2403.14156
• Copy Paste: [[2403.14156]] Policy Mirror Descent with Lookahead(https://arxiv.org/abs/2403.14156)
• Keywords: generative
• Abstract: Policy Mirror Descent (PMD) stands as a versatile algorithmic framework encompassing several seminal policy gradient algorithms such as natural policy gradient, with connections with state-of-the-art reinforcement learning (RL) algorithms such as TRPO and PPO. PMD can be seen as a soft Policy Iteration algorithm implementing regularized 1-step greedy policy improvement. However, 1-step greedy policies might not be the best choice and recent remarkable empirical successes in RL such as AlphaGo and AlphaZero have demonstrated that greedy approaches with respect to multiple steps outperform their 1-step counterpart. In this work, we propose a new class of PMD algorithms called $h$-PMD which incorporates multi-step greedy policy improvement with lookahead depth $h$ to the PMD update rule. To solve discounted infinite horizon Markov Decision Processes with discount factor $\gamma$, we show that $h$-PMD which generalizes the standard PMD enjoys a faster dimension-free $\gamma^h$-linear convergence rate, contingent on the computation of multi-step greedy policies. We propose an inexact version of $h$-PMD where lookahead action values are estimated. Under a generative model, we establish a sample complexity for $h$-PMD which improves over prior work. Finally, we extend our result to linear function approximation to scale to large state spaces. Under suitable assumptions, our sample complexity only involves dependence on the dimension of the feature map space instead of the state space size.

Title: Unsupervised Audio-Visual Segmentation with Modality Alignment

• Authors: Swapnil Bhosale, Haosen Yang, Diptesh Kanojia, Jiangkang Deng, Xiatian Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14203
• Pdf URL: https://arxiv.org/pdf/2403.14203
• Copy Paste: [[2403.14203]] Unsupervised Audio-Visual Segmentation with Modality Alignment(https://arxiv.org/abs/2403.14203)
• Keywords: foundation model
• Abstract: Audio-Visual Segmentation (AVS) aims to identify, at the pixel level, the object in a visual scene that produces a given sound. Current AVS methods rely on costly fine-grained annotations of mask-audio pairs, making them impractical for scalability. To address this, we introduce unsupervised AVS, eliminating the need for such expensive annotation. To tackle this more challenging problem, we propose an unsupervised learning method, named Modality Correspondence Alignment (MoCA), which seamlessly integrates off-the-shelf foundation models like DINO, SAM, and ImageBind. This approach leverages their knowledge complementarity and optimizes their joint usage for multi-modality association. Initially, we estimate positive and negative image pairs in the feature space. For pixel-level association, we introduce an audio-visual adapter and a novel pixel matching aggregation strategy within the image-level contrastive learning framework. This allows for a flexible connection between object appearance and audio signal at the pixel level, with tolerance to imaging variations such as translation and rotation. Extensive experiments on the AVSBench (single and multi-object splits) and AVSS datasets demonstrate that our MoCA outperforms strongly designed baseline methods and approaches supervised counterparts, particularly in complex scenarios with multiple auditory objects. Notably when comparing mIoU, MoCA achieves a substantial improvement over baselines in both the AVSBench (S4: +17.24%; MS3: +67.64%) and AVSS (+19.23%) audio-visual segmentation challenges.

Title: Toward Multi-class Anomaly Detection: Exploring Class-aware Unified Model against Inter-class Interference

• Authors: Xi Jiang, Ying Chen, Qiang Nie, Jianlin Liu, Yong Liu, Chengjie Wang, Feng Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14213
• Pdf URL: https://arxiv.org/pdf/2403.14213
• Copy Paste: [[2403.14213]] Toward Multi-class Anomaly Detection: Exploring Class-aware Unified Model against Inter-class Interference(https://arxiv.org/abs/2403.14213)
• Keywords: anomaly
• Abstract: In the context of high usability in single-class anomaly detection models, recent academic research has become concerned about the more complex multi-class anomaly detection. Although several papers have designed unified models for this task, they often overlook the utility of class labels, a potent tool for mitigating inter-class interference. To address this issue, we introduce a Multi-class Implicit Neural representation Transformer for unified Anomaly Detection (MINT-AD), which leverages the fine-grained category information in the training stage. By learning the multi-class distributions, the model generates class-aware query embeddings for the transformer decoder, mitigating inter-class interference within the reconstruction model. Utilizing such an implicit neural representation network, MINT-AD can project category and position information into a feature embedding space, further supervised by classification and prior probability loss functions. Experimental results on multiple datasets demonstrate that MINT-AD outperforms existing unified training models.

Title: SoftPatch: Unsupervised Anomaly Detection with Noisy Data

• Authors: Xi Jiang, Ying Chen, Qiang Nie, Yong Liu, Jianlin Liu, Bin-Bin Gao, Jun Liu, Chengjie Wang, Feng Zheng
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14233
• Pdf URL: https://arxiv.org/pdf/2403.14233
• Copy Paste: [[2403.14233]] SoftPatch: Unsupervised Anomaly Detection with Noisy Data(https://arxiv.org/abs/2403.14233)
• Keywords: anomaly
• Abstract: Although mainstream unsupervised anomaly detection (AD) algorithms perform well in academic datasets, their performance is limited in practical application due to the ideal experimental setting of clean training data. Training with noisy data is an inevitable problem in real-world anomaly detection but is seldom discussed. This paper considers label-level noise in image sensory anomaly detection for the first time. To solve this problem, we proposed a memory-based unsupervised AD method, SoftPatch, which efficiently denoises the data at the patch level. Noise discriminators are utilized to generate outlier scores for patch-level noise elimination before coreset construction. The scores are then stored in the memory bank to soften the anomaly detection boundary. Compared with existing methods, SoftPatch maintains a strong modeling ability of normal data and alleviates the overconfidence problem in coreset. Comprehensive experiments in various noise scenes demonstrate that SoftPatch outperforms the state-of-the-art AD methods on the MVTecAD and BTAD benchmarks and is comparable to those methods under the setting without noise.

Title: LLM-based Extraction of Contradictions from Patents

• Authors: Stefan Trapp, Joachim Warschat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14258
• Pdf URL: https://arxiv.org/pdf/2403.14258
• Copy Paste: [[2403.14258]] LLM-based Extraction of Contradictions from Patents(https://arxiv.org/abs/2403.14258)
• Keywords: generative
• Abstract: Already since the 1950s TRIZ shows that patents and the technical contradictions they solve are an important source of inspiration for the development of innovative products. However, TRIZ is a heuristic based on a historic patent analysis and does not make use of the ever-increasing number of latest technological solutions in current patents. Because of the huge number of patents, their length, and, last but not least, their complexity there is a need for modern patent retrieval and patent analysis to go beyond keyword-oriented methods. Recent advances in patent retrieval and analysis mainly focus on dense vectors based on neural AI Transformer language models like Google BERT. They are, for example, used for dense retrieval, question answering or summarization and key concept extraction. A research focus within the methods for patent summarization and key concept extraction are generic inventive concepts respectively TRIZ concepts like problems, solutions, advantage of invention, parameters, and contradictions. Succeeding rule-based approaches, finetuned BERT-like language models for sentence-wise classification represent the state-of-the-art of inventive concept extraction. While they work comparatively well for basic concepts like problems or solutions, contradictions - as a more complex abstraction - remain a challenge for these models. This paper goes one step further, as it presents a method to extract TRIZ contradictions from patent texts based on Prompt Engineering using a generative Large Language Model (LLM), namely OpenAI's GPT-4. Contradiction detection, sentence extraction, contradiction summarization, parameter extraction and assignment to the 39 abstract TRIZ engineering parameters are all performed in a single prompt using the LangChain framework. Our results show that "off-the-shelf" GPT-4 is a serious alternative to existing approaches.

Title: A Framework for Portrait Stylization with Skin-Tone Awareness and Nudity Identification

• Authors: Seungkwon Kim, Sangyeon Kim, Seung-Hun Nam
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14264
• Pdf URL: https://arxiv.org/pdf/2403.14264
• Copy Paste: [[2403.14264]] A Framework for Portrait Stylization with Skin-Tone Awareness and Nudity Identification(https://arxiv.org/abs/2403.14264)
• Keywords: diffusion
• Abstract: Portrait stylization is a challenging task involving the transformation of an input portrait image into a specific style while preserving its inherent characteristics. The recent introduction of Stable Diffusion (SD) has significantly improved the quality of outcomes in this field. However, a practical stylization framework that can effectively filter harmful input content and preserve the distinct characteristics of an input, such as skin-tone, while maintaining the quality of stylization remains lacking. These challenges have hindered the wide deployment of such a framework. To address these issues, this study proposes a portrait stylization framework that incorporates a nudity content identification module (NCIM) and a skin-tone-aware portrait stylization module (STAPSM). In experiments, NCIM showed good performance in enhancing explicit content filtering, and STAPSM accurately represented a diverse range of skin tones. Our proposed framework has been successfully deployed in practice, and it has effectively satisfied critical requirements of real-world applications.

Title: Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimation

• Authors: Francesco Di Felice, Alberto Remus, Stefano Gasperini, Benjamin Busam, Lionel Ott, Federico Tombari, Roland Siegwart, Carlo Alberto Avizzano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14279
• Pdf URL: https://arxiv.org/pdf/2403.14279
• Copy Paste: [[2403.14279]] Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimation(https://arxiv.org/abs/2403.14279)
• Keywords: diffusion
• Abstract: Estimating the pose of objects through vision is essential to make robotic platforms interact with the environment. Yet, it presents many challenges, often related to the lack of flexibility and generalizability of state-of-the-art solutions. Diffusion models are a cutting-edge neural architecture transforming 2D and 3D computer vision, outlining remarkable performances in zero-shot novel-view synthesis. Such a use case is particularly intriguing for reconstructing 3D objects. However, localizing objects in unstructured environments is rather unexplored. To this end, this work presents Zero123-6D to demonstrate the utility of Diffusion Model-based novel-view-synthesizers in enhancing RGB 6D pose estimation at category-level by integrating them with feature extraction techniques. The outlined method exploits such a novel view synthesizer to expand a sparse set of RGB-only reference views for the zero-shot 6D pose estimation task. Experiments are quantitatively analyzed on the CO3D dataset, showcasing increased performance over baselines, a substantial reduction in data requirements, and the removal of the necessity of depth information.

Title: Large Language Models for Blockchain Security: A Systematic Literature Review

• Authors: Zheyuan He, Zihao Li, Sen Yang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.14280
• Pdf URL: https://arxiv.org/pdf/2403.14280
• Copy Paste: [[2403.14280]] Large Language Models for Blockchain Security: A Systematic Literature Review(https://arxiv.org/abs/2403.14280)
• Keywords: anomaly
• Abstract: Large Language Models (LLMs) have emerged as powerful tools in various domains involving blockchain security (BS). Several recent studies are exploring LLMs applied to BS. However, there remains a gap in our understanding regarding the full scope of applications, impacts, and potential constraints of LLMs on blockchain security. To fill this gap, we conduct a literature review on LLM4BS. As the first review of LLM's application on blockchain security, our study aims to comprehensively analyze existing research and elucidate how LLMs contribute to enhancing the security of blockchain systems. Through a thorough examination of scholarly works, we delve into the integration of LLMs into various aspects of blockchain security. We explore the mechanisms through which LLMs can bolster blockchain security, including their applications in smart contract auditing, identity verification, anomaly detection, vulnerable repair, and so on. Furthermore, we critically assess the challenges and limitations associated with leveraging LLMs for blockchain security, considering factors such as scalability, privacy concerns, and adversarial attacks. Our review sheds light on the opportunities and potential risks inherent in this convergence, providing valuable insights for researchers, practitioners, and policymakers alike.

Title: Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models

• Authors: Pablo Marcos-Manchón, Roberto Alcover-Couso, Juan C. SanMiguel, Jose M. Martínez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14291
• Pdf URL: https://arxiv.org/pdf/2403.14291
• Copy Paste: [[2403.14291]] Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models(https://arxiv.org/abs/2403.14291)
• Keywords: diffusion
• Abstract: Diffusion models represent a new paradigm in text-to-image generation. Beyond generating high-quality images from text prompts, models such as Stable Diffusion have been successfully extended to the joint generation of semantic segmentation pseudo-masks. However, current extensions primarily rely on extracting attentions linked to prompt words used for image synthesis. This approach limits the generation of segmentation masks derived from word tokens not contained in the text prompt. In this work, we introduce Open-Vocabulary Attention Maps (OVAM)-a training-free method for text-to-image diffusion models that enables the generation of attention maps for any word. In addition, we propose a lightweight optimization process based on OVAM for finding tokens that generate accurate attention maps for an object class with a single annotation. We evaluate these tokens within existing state-of-the-art Stable Diffusion extensions. The best-performing model improves its mIoU from 52.1 to 86.6 for the synthetic images' pseudo-masks, demonstrating that our optimized tokens are an efficient way to improve the performance of existing methods without architectural changes or retraining.

Title: HySim: An Efficient Hybrid Similarity Measure for Patch Matching in Image Inpainting

• Authors: Saad Noufel, Nadir Maaroufi, Mehdi Najib, Mohamed Bakhouya
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14292
• Pdf URL: https://arxiv.org/pdf/2403.14292
• Copy Paste: [[2403.14292]] HySim: An Efficient Hybrid Similarity Measure for Patch Matching in Image Inpainting(https://arxiv.org/abs/2403.14292)
• Keywords: diffusion
• Abstract: Inpainting, for filling missing image regions, is a crucial task in various applications, such as medical imaging and remote sensing. Trending data-driven approaches efficiency, for image inpainting, often requires extensive data preprocessing. In this sense, there is still a need for model-driven approaches in case of application constrained with data availability and quality, especially for those related for time series forecasting using image inpainting techniques. This paper proposes an improved modeldriven approach relying on patch-based techniques. Our approach deviates from the standard Sum of Squared Differences (SSD) similarity measure by introducing a Hybrid Similarity (HySim), which combines both strengths of Chebychev and Minkowski distances. This hybridization enhances patch selection, leading to high-quality inpainting results with reduced mismatch errors. Experimental results proved the effectiveness of our approach against other model-driven techniques, such as diffusion or patch-based approaches, showcasing its effectiveness in achieving visually pleasing restorations.

Title: Exploring Task Unification in Graph Representation Learning via Generative Approach

• Authors: Yulan Hu, Sheng Ouyang, Zhirui Yang, Ge Chen, Junchen Wan, Xiao Wang, Yong Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14340
• Pdf URL: https://arxiv.org/pdf/2403.14340
• Copy Paste: [[2403.14340]] Exploring Task Unification in Graph Representation Learning via Generative Approach(https://arxiv.org/abs/2403.14340)
• Keywords: self-supervised, generative
• Abstract: Graphs are ubiquitous in real-world scenarios and encompass a diverse range of tasks, from node-, edge-, and graph-level tasks to transfer learning. However, designing specific tasks for each type of graph data is often costly and lacks generalizability. Recent endeavors under the "Pre-training + Fine-tuning" or "Pre-training + Prompt" paradigms aim to design a unified framework capable of generalizing across multiple graph tasks. Among these, graph autoencoders (GAEs), generative self-supervised models, have demonstrated their potential in effectively addressing various graph tasks. Nevertheless, these methods typically employ multi-stage training and require adaptive designs, which on one hand make it difficult to be seamlessly applied to diverse graph tasks and on the other hand overlook the negative impact caused by discrepancies in task objectives between the different stages. To address these challenges, we propose GA^2E, a unified adversarially masked autoencoder capable of addressing the above challenges seamlessly. Specifically, GA^2E proposes to use the subgraph as the meta-structure, which remains consistent across all graph tasks (ranging from node-, edge-, and graph-level to transfer learning) and all stages (both during training and inference). Further, GA^2E operates in a \textbf{"Generate then Discriminate"} manner. It leverages the masked GAE to reconstruct the input subgraph whilst treating it as a generator to compel the reconstructed graphs resemble the input subgraph. Furthermore, GA^2E introduces an auxiliary discriminator to discern the authenticity between the reconstructed (generated) subgraph and the input subgraph, thus ensuring the robustness of the graph representation through adversarial training mechanisms. We validate GA^2E's capabilities through extensive experiments on 21 datasets across four types of graph tasks.

Title: Enabling Visual Composition and Animation in Unsupervised Video Generation

• Authors: Aram Davtyan, Sepehr Sameni, Björn Ommer, Paolo Favaro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14368
• Pdf URL: https://arxiv.org/pdf/2403.14368
• Copy Paste: [[2403.14368]] Enabling Visual Composition and Animation in Unsupervised Video Generation(https://arxiv.org/abs/2403.14368)
• Keywords: self-supervised
• Abstract: In this work we propose a novel method for unsupervised controllable video generation. Once trained on a dataset of unannotated videos, at inference our model is capable of both composing scenes of predefined object parts and animating them in a plausible and controlled way. This is achieved by conditioning video generation on a randomly selected subset of local pre-trained self-supervised features during training. We call our model CAGE for visual Composition and Animation for video GEneration. We conduct a series of experiments to demonstrate capabilities of CAGE in various settings. Project website: https://araachie.github.io/cage.

Title: SyncTweedies: A General Generative Framework Based on Synchronized Diffusions

• Authors: Jaihoon Kim, Juil Koo, Kyeongmin Yeo, Minhyuk Sung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14370
• Pdf URL: https://arxiv.org/pdf/2403.14370
• Copy Paste: [[2403.14370]] SyncTweedies: A General Generative Framework Based on Synchronized Diffusions(https://arxiv.org/abs/2403.14370)
• Keywords: diffusion, generative
• Abstract: We introduce a general framework for generating diverse visual content, including ambiguous images, panorama images, mesh textures, and Gaussian splat textures, by synchronizing multiple diffusion processes. We present exhaustive investigation into all possible scenarios for synchronizing multiple diffusion processes through a canonical space and analyze their characteristics across applications. In doing so, we reveal a previously unexplored case: averaging the outputs of Tweedie's formula while conducting denoising in multiple instance spaces. This case also provides the best quality with the widest applicability to downstream tasks. We name this case SyncTweedies. In our experiments generating visual content aforementioned, we demonstrate the superior quality of generation by SyncTweedies compared to other synchronization methods, optimization-based and iterative-update-based methods.

Title: Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning

• Authors: Changtong Zan, Liang Ding, Li Shen, Yibing Zhen, Weifeng Liu, Dacheng Tao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14399
• Pdf URL: https://arxiv.org/pdf/2403.14399
• Copy Paste: [[2403.14399]] Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning(https://arxiv.org/abs/2403.14399)
• Keywords: in-context
• Abstract: Translation-tailored Large language models (LLMs) exhibit remarkable translation capabilities, even competing with supervised-trained commercial translation systems. However, off-target translation remains an unsolved problem, especially for low-resource languages, hindering us from developing accurate LLMs-based translation models. To mitigate the off-target translation problem and enhance the performance of LLMs on translation, recent works have either designed advanced prompting strategies to highlight the functionality of translation instructions or exploited the in-context learning ability of LLMs by feeding few-shot demonstrations. However, these methods essentially do not improve LLM's ability to follow translation instructions, especially the language direction information. In this work, we design a two-stage fine-tuning algorithm to improve the instruction-following ability (especially the translation direction) of LLMs. Specifically, we first tune LLMs with the maximum likelihood estimation loss on the translation dataset to elicit the basic translation capabilities. In the second stage, we construct instruction-conflicting samples by randomly replacing the translation directions with a wrong one within the instruction, and then introduce an extra unlikelihood loss to learn those samples. Experiments on IWSLT and WMT benchmarks upon the LLaMA model spanning 16 zero-shot directions show that, compared to the competitive baseline -- translation-finetuned LLama, our method could effectively reduce the off-target translation ratio (averagely -53.3\%), thus improving translation quality with average +5.7 SacreBLEU and +16.4 BLEURT. Analysis shows that our method could preserve the model's general task performance on AlpacaEval. Code and models will be released at \url{https://github.com/alphadl/LanguageAware_Tuning}.

Title: Physics-Informed Diffusion Models

• Authors: Jan-Hendrik Bastek, WaiChing Sun, Dennis M. Kochmann
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2403.14404
• Pdf URL: https://arxiv.org/pdf/2403.14404
• Copy Paste: [[2403.14404]] Physics-Informed Diffusion Models(https://arxiv.org/abs/2403.14404)
• Keywords: diffusion, generative
• Abstract: Generative models such as denoising diffusion models are quickly advancing their ability to approximate highly complex data distributions. They are also increasingly leveraged in scientific machine learning, where samples from the implied data distribution are expected to adhere to specific governing equations. We present a framework to inform denoising diffusion models on underlying constraints on such generated samples during model training. Our approach improves the alignment of the generated samples with the imposed constraints and significantly outperforms existing methods without affecting inference speed. Additionally, our findings suggest that incorporating such constraints during training provides a natural regularization against overfitting. Our framework is easy to implement and versatile in its applicability for imposing equality and inequality constraints as well as auxiliary optimization objectives.

Title: DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning

• Authors: Jonathan Lebensold, Maziar Sanjabi, Pietro Astolfi, Adriana Romero-Soriano, Kamalika Chaudhuri, Mike Rabbat, Chuan Guo
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14421
• Pdf URL: https://arxiv.org/pdf/2403.14421
• Copy Paste: [[2403.14421]] DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning(https://arxiv.org/abs/2403.14421)
• Keywords: diffusion, generative
• Abstract: Text-to-image diffusion models have been shown to suffer from sample-level memorization, possibly reproducing near-perfect replica of images that they are trained on, which may be undesirable. To remedy this issue, we develop the first differentially private (DP) retrieval-augmented generation algorithm that is capable of generating high-quality image samples while providing provable privacy guarantees. Specifically, we assume access to a text-to-image diffusion model trained on a small amount of public data, and design a DP retrieval mechanism to augment the text prompt with samples retrieved from a private retrieval dataset. Our \emph{differentially private retrieval-augmented diffusion model} (DP-RDM) requires no fine-tuning on the retrieval dataset to adapt to another domain, and can use state-of-the-art generative models to generate high-quality image samples while satisfying rigorous DP guarantees. For instance, when evaluated on MS-COCO, our DP-RDM can generate samples with a privacy budget of $\epsilon=10$, while providing a $3.5$ point improvement in FID compared to public-only retrieval for up to $10,000$ queries.

Title: Style-Extracting Diffusion Models for Semi-Supervised Histopathology Segmentation

• Authors: Mathias Öttl, Frauke Wilm, Jana Steenpass, Jingna Qiu, Matthias Rübner, Arndt Hartmann, Matthias Beckmann, Peter Fasching, Andreas Maier, Ramona Erber, Bernhard Kainz, Katharina Breininger
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14429
• Pdf URL: https://arxiv.org/pdf/2403.14429
• Copy Paste: [[2403.14429]] Style-Extracting Diffusion Models for Semi-Supervised Histopathology Segmentation(https://arxiv.org/abs/2403.14429)
• Keywords: diffusion
• Abstract: Deep learning-based image generation has seen significant advancements with diffusion models, notably improving the quality of generated images. Despite these developments, generating images with unseen characteristics beneficial for downstream tasks has received limited attention. To bridge this gap, we propose Style-Extracting Diffusion Models, featuring two conditioning mechanisms. Specifically, we utilize 1) a style conditioning mechanism which allows to inject style information of previously unseen images during image generation and 2) a content conditioning which can be targeted to a downstream task, e.g., layout for segmentation. We introduce a trainable style encoder to extract style information from images, and an aggregation block that merges style information from multiple style inputs. This architecture enables the generation of images with unseen styles in a zero-shot manner, by leveraging styles from unseen images, resulting in more diverse generations. In this work, we use the image layout as target condition and first show the capability of our method on a natural image dataset as a proof-of-concept. We further demonstrate its versatility in histopathology, where we combine prior knowledge about tissue composition and unannotated data to create diverse synthetic images with known layouts. This allows us to generate additional synthetic data to train a segmentation network in a semi-supervised fashion. We verify the added value of the generated images by showing improved segmentation results and lower performance variability between patients when synthetic images are included during segmentation training. Our code will be made publicly available at [LINK].

Title: gTBLS: Generating Tables from Text by Conditional Question Answering

• Authors: Anirudh Sundar, Christopher Richardson, Larry Heck
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14457
• Pdf URL: https://arxiv.org/pdf/2403.14457
• Copy Paste: [[2403.14457]] gTBLS: Generating Tables from Text by Conditional Question Answering(https://arxiv.org/abs/2403.14457)
• Keywords: generative
• Abstract: Distilling large, unstructured text into a structured, condensed form such as tables is an open research problem. One of the primary challenges in automatically generating tables is ensuring their syntactic validity. Prior approaches address this challenge by including additional parameters in the Transformer's attention mechanism to attend to specific rows and column headers. In contrast to this single-stage method, this paper presents a two-stage approach called Generative Tables (gTBLS). The first stage infers table structure (row and column headers) from the text. The second stage formulates questions using these headers and fine-tunes a causal language model to answer them. Furthermore, the gTBLS approach is amenable to the utilization of pre-trained Large Language Models in a zero-shot configuration, presenting a solution for table generation in situations where fine-tuning is not feasible. gTBLS improves prior approaches by up to 10% in BERTScore on the table construction task and up to 20% on the table content generation task of the E2E, WikiTableText, WikiBio, and RotoWire datasets.

Title: Multi-Level Explanations for Generative Language Models

• Authors: Lucas Monteiro Paes, Dennis Wei, Hyo Jin Do, Hendrik Strobelt, Ronny Luss, Amit Dhurandhar, Manish Nagireddy, Karthikeyan Natesan Ramamurthy, Prasanna Sattigeri, Werner Geyer, Soumya Ghosh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14459
• Pdf URL: https://arxiv.org/pdf/2403.14459
• Copy Paste: [[2403.14459]] Multi-Level Explanations for Generative Language Models(https://arxiv.org/abs/2403.14459)
• Keywords: generative
• Abstract: Perturbation-based explanation methods such as LIME and SHAP are commonly applied to text classification. This work focuses on their extension to generative language models. To address the challenges of text as output and long text inputs, we propose a general framework called MExGen that can be instantiated with different attribution algorithms. To handle text output, we introduce the notion of scalarizers for mapping text to real numbers and investigate multiple possibilities. To handle long inputs, we take a multi-level approach, proceeding from coarser levels of granularity to finer ones, and focus on algorithms with linear scaling in model queries. We conduct a systematic evaluation, both automated and human, of perturbation-based attribution methods for summarization and context-grounded question answering. The results show that our framework can provide more locally faithful explanations of generated outputs.

Title: ChatGPT Alternative Solutions: Large Language Models Survey

• Authors: Hanieh Alipour, Nick Pendar, Kohinoor Roy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14469
• Pdf URL: https://arxiv.org/pdf/2403.14469
• Copy Paste: [[2403.14469]] ChatGPT Alternative Solutions: Large Language Models Survey(https://arxiv.org/abs/2403.14469)
• Keywords: generative
• Abstract: In recent times, the grandeur of Large Language Models (LLMs) has not only shone in the realm of natural language processing but has also cast its brilliance across a vast array of applications. This remarkable display of LLM capabilities has ignited a surge in research contributions within this domain, spanning a diverse spectrum of topics. These contributions encompass advancements in neural network architecture, context length enhancements, model alignment, training datasets, benchmarking, efficiency improvements, and more. Recent years have witnessed a dynamic synergy between academia and industry, propelling the field of LLM research to new heights. A notable milestone in this journey is the introduction of ChatGPT, a powerful AI chatbot grounded in LLMs, which has garnered widespread societal attention. The evolving technology of LLMs has begun to reshape the landscape of the entire AI community, promising a revolutionary shift in the way we create and employ AI algorithms. Given this swift-paced technical evolution, our survey embarks on a journey to encapsulate the recent strides made in the world of LLMs. Through an exploration of the background, key discoveries, and prevailing methodologies, we offer an up-to-the-minute review of the literature. By examining multiple LLM models, our paper not only presents a comprehensive overview but also charts a course that identifies existing challenges and points toward potential future research trajectories. This survey furnishes a well-rounded perspective on the current state of generative AI, shedding light on opportunities for further exploration, enhancement, and innovation.

Title: MULDE: Multiscale Log-Density Estimation via Denoising Score Matching for Video Anomaly Detection

• Authors: Jakub Micorek, Horst Possegger, Dominik Narnhofer, Horst Bischof, Mateusz Kozinski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14497
• Pdf URL: https://arxiv.org/pdf/2403.14497
• Copy Paste: [[2403.14497]] MULDE: Multiscale Log-Density Estimation via Denoising Score Matching for Video Anomaly Detection(https://arxiv.org/abs/2403.14497)
• Keywords: anomaly
• Abstract: We propose a novel approach to video anomaly detection: we treat feature vectors extracted from videos as realizations of a random variable with a fixed distribution and model this distribution with a neural network. This lets us estimate the likelihood of test videos and detect video anomalies by thresholding the likelihood estimates. We train our video anomaly detector using a modification of denoising score matching, a method that injects training data with noise to facilitate modeling its distribution. To eliminate hyperparameter selection, we model the distribution of noisy video features across a range of noise levels and introduce a regularizer that tends to align the models for different levels of noise. At test time, we combine anomaly indications at multiple noise scales with a Gaussian mixture model. Running our video anomaly detector induces minimal delays as inference requires merely extracting the features and forward-propagating them through a shallow neural network and a Gaussian mixture model. Our experiments on five popular video anomaly detection benchmarks demonstrate state-of-the-art performance, both in the object-centric and in the frame-centric setup.

Title: Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference

• Authors: Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, Donglin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14520
• Pdf URL: https://arxiv.org/pdf/2403.14520
• Copy Paste: [[2403.14520]] Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference(https://arxiv.org/abs/2403.14520)
• Keywords: foundation model
• Abstract: In recent years, the application of multimodal large language models (MLLM) in various fields has achieved remarkable success. However, as the foundation model for many downstream tasks, current MLLMs are composed of the well-known Transformer network, which has a less efficient quadratic computation complexity. To improve the efficiency of such basic models, we propose Cobra, a linear computational complexity MLLM. Specifically, Cobra integrates the efficient Mamba language model into the visual modality. Moreover, we explore and study various modal fusion schemes to create an effective multi-modal Mamba. Extensive experiments demonstrate that (1) Cobra achieves extremely competitive performance with current computationally efficient state-of-the-art methods, \textit{e.g.}, LLaVA-Phi, TinyLLaVA, and MobileVLM v2, and has faster speed due to Cobra's linear sequential modeling. (2) Interestingly, the results of closed-set challenging prediction benchmarks show that Cobra performs well in overcoming visual illusions and spatial relationship judgments. (3) Notably, Cobra even achieves comparable performance to LLaVA with about 43% of the number of parameters. We will make all codes of Cobra open-source and hope that the proposed method can facilitate future research on complexity problems in MLLM. Our project page is available at: https://sites.google.com/view/cobravlm.

Title: Object-Centric Domain Randomization for 3D Shape Reconstruction in the Wild

• Authors: Junhyeong Cho, Kim Youwang, Hunmin Yang, Tae-Hyun Oh
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14539
• Pdf URL: https://arxiv.org/pdf/2403.14539
• Copy Paste: [[2403.14539]] Object-Centric Domain Randomization for 3D Shape Reconstruction in the Wild(https://arxiv.org/abs/2403.14539)
• Keywords: generative
• Abstract: One of the biggest challenges in single-view 3D shape reconstruction in the wild is the scarcity of <3D shape, 2D image>-paired data from real-world environments. Inspired by remarkable achievements via domain randomization, we propose ObjectDR which synthesizes such paired data via a random simulation of visual variations in object appearances and backgrounds. Our data synthesis framework exploits a conditional generative model (e.g., ControlNet) to generate images conforming to spatial conditions such as 2.5D sketches, which are obtainable through a rendering process of 3D shapes from object collections (e.g., Objaverse-XL). To simulate diverse variations while preserving object silhouettes embedded in spatial conditions, we also introduce a disentangled framework which leverages an initial object guidance. After synthesizing a wide range of data, we pre-train a model on them so that it learns to capture a domain-invariant geometry prior which is consistent across various domains. We validate its effectiveness by substantially improving 3D shape reconstruction models on a real-world benchmark. In a scale-up evaluation, our pre-training achieves 23.6% superior results compared with the pre-training on high-quality computer graphics renderings.

Title: Estimating Physical Information Consistency of Channel Data Augmentation for Remote Sensing Images

• Authors: Tom Burgert, Begüm Demir
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14547
• Pdf URL: https://arxiv.org/pdf/2403.14547
• Copy Paste: [[2403.14547]] Estimating Physical Information Consistency of Channel Data Augmentation for Remote Sensing Images(https://arxiv.org/abs/2403.14547)
• Keywords: self-supervised
• Abstract: The application of data augmentation for deep learning (DL) methods plays an important role in achieving state-of-the-art results in supervised, semi-supervised, and self-supervised image classification. In particular, channel transformations (e.g., solarize, grayscale, brightness adjustments) are integrated into data augmentation pipelines for remote sensing (RS) image classification tasks. However, contradicting beliefs exist about their proper applications to RS images. A common point of critique is that the application of channel augmentation techniques may lead to physically inconsistent spectral data (i.e., pixel signatures). To shed light on the open debate, we propose an approach to estimate whether a channel augmentation technique affects the physical information of RS images. To this end, the proposed approach estimates a score that measures the alignment of a pixel signature within a time series that can be naturally subject to deviations caused by factors such as acquisition conditions or phenological states of vegetation. We compare the scores associated with original and augmented pixel signatures to evaluate the physical consistency. Experimental results on a multi-label image classification task show that channel augmentations yielding a score that exceeds the expected deviation of original pixel signatures can not improve the performance of a baseline model trained without augmentation.

Title: DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video

• Authors: Narek Tumanyan, Assaf Singer, Shai Bagon, Tali Dekel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14548
• Pdf URL: https://arxiv.org/pdf/2403.14548
• Copy Paste: [[2403.14548]] DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video(https://arxiv.org/abs/2403.14548)
• Keywords: self-supervised
• Abstract: We present DINO-Tracker -- a new framework for long-term dense tracking in video. The pillar of our approach is combining test-time training on a single video, with the powerful localized semantic features learned by a pre-trained DINO-ViT model. Specifically, our framework simultaneously adopts DINO's features to fit to the motion observations of the test video, while training a tracker that directly leverages the refined features. The entire framework is trained end-to-end using a combination of self-supervised losses, and regularization that allows us to retain and benefit from DINO's semantic prior. Extensive evaluation demonstrates that our method achieves state-of-the-art results on known benchmarks. DINO-tracker significantly outperforms self-supervised methods and is competitive with state-of-the-art supervised trackers, while outperforming them in challenging cases of tracking under long-term occlusions.

Title: VXP: Voxel-Cross-Pixel Large-scale Image-LiDAR Place Recognition

• Authors: Yun-Jin Li, Mariia Gladkova, Yan Xia, Rui Wang, Daniel Cremers
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.14594
• Pdf URL: https://arxiv.org/pdf/2403.14594
• Copy Paste: [[2403.14594]] VXP: Voxel-Cross-Pixel Large-scale Image-LiDAR Place Recognition(https://arxiv.org/abs/2403.14594)
• Keywords: self-supervised
• Abstract: Recent works on the global place recognition treat the task as a retrieval problem, where an off-the-shelf global descriptor is commonly designed in image-based and LiDAR-based modalities. However, it is non-trivial to perform accurate image-LiDAR global place recognition since extracting consistent and robust global descriptors from different domains (2D images and 3D point clouds) is challenging. To address this issue, we propose a novel Voxel-Cross-Pixel (VXP) approach, which establishes voxel and pixel correspondences in a self-supervised manner and brings them into a shared feature space. Specifically, VXP is trained in a two-stage manner that first explicitly exploits local feature correspondences and enforces similarity of global descriptors. Extensive experiments on the three benchmarks (Oxford RobotCar, ViViD++ and KITTI) demonstrate our method surpasses the state-of-the-art cross-modal retrieval by a large margin.

Title: ReNoise: Real Image Inversion Through Iterative Noising

• Authors: Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Elor, Daniel Cohen-Or
• Subjects: cs.CV, cs.GR, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.14602
• Pdf URL: https://arxiv.org/pdf/2403.14602
• Copy Paste: [[2403.14602]] ReNoise: Real Image Inversion Through Iterative Noising(https://arxiv.org/abs/2403.14602)
• Keywords: diffusion
• Abstract: Recent advancements in text-guided diffusion models have unlocked powerful image manipulation capabilities. However, applying these methods to real images necessitates the inversion of the images into the domain of the pretrained diffusion model. Achieving faithful inversion remains a challenge, particularly for more recent models trained to generate images with a small number of denoising steps. In this work, we introduce an inversion method with a high quality-to-operation ratio, enhancing reconstruction accuracy without increasing the number of operations. Building on reversing the diffusion sampling process, our method employs an iterative renoising mechanism at each inversion sampling step. This mechanism refines the approximation of a predicted point along the forward diffusion trajectory, by iteratively applying the pretrained diffusion model, and averaging these predictions. We evaluate the performance of our ReNoise technique using various sampling algorithms and models, including recent accelerated diffusion models. Through comprehensive evaluations and comparisons, we show its effectiveness in terms of both accuracy and speed. Furthermore, we confirm that our method preserves editability by demonstrating text-driven image editing on real images.

Title: DreamReward: Text-to-3D Generation with Human Preference

• Authors: Junliang Ye, Fangfu Liu, Qixiu Li, Zhengyi Wang, Yikai Wang, Xinzhou Wang, Yueqi Duan, Jun Zhu
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14613
• Pdf URL: https://arxiv.org/pdf/2403.14613
• Copy Paste: [[2403.14613]] DreamReward: Text-to-3D Generation with Human Preference(https://arxiv.org/abs/2403.14613)
• Keywords: diffusion
• Abstract: 3D content creation from text prompts has shown remarkable success recently. However, current text-to-3D methods often generate 3D results that do not align well with human preferences. In this paper, we present a comprehensive framework, coined DreamReward, to learn and improve text-to-3D models from human preference feedback. To begin with, we collect 25k expert comparisons based on a systematic annotation pipeline including rating and ranking. Then, we build Reward3D -- the first general-purpose text-to-3D human preference reward model to effectively encode human preferences. Building upon the 3D reward model, we finally perform theoretical analysis and present the Reward3D Feedback Learning (DreamFL), a direct tuning algorithm to optimize the multi-view diffusion models with a redefined scorer. Grounded by theoretical proof and extensive experiment comparisons, our DreamReward successfully generates high-fidelity and 3D consistent results with significant boosts in prompt alignment with human intention. Our results demonstrate the great potential for learning from human feedback to improve text-to-3D models.

Title: Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning

• Authors: Hasindri Watawana, Kanchana Ranasinghe, Tariq Mahmood, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14616
• Pdf URL: https://arxiv.org/pdf/2403.14616
• Copy Paste: [[2403.14616]] Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning(https://arxiv.org/abs/2403.14616)
• Keywords: self-supervised
• Abstract: Self-supervised representation learning has been highly promising for histopathology image analysis with numerous approaches leveraging their patient-slide-patch hierarchy to learn better representations. In this paper, we explore how the combination of domain specific natural language information with such hierarchical visual representations can benefit rich representation learning for medical image tasks. Building on automated language description generation for features visible in histopathology images, we present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. We explore contrastive objectives and granular language description based text alignment at multiple hierarchies to inject language modality information into the visual representations. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets. Our framework also provides better interpretability with our language aligned representation space. Code is available at https://github.com/Hasindri/HLSS.

Title: Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion

• Authors: Xiang Fan, Anand Bhattad, Ranjay Krishna
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14617
• Pdf URL: https://arxiv.org/pdf/2403.14617
• Copy Paste: [[2403.14617]] Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion(https://arxiv.org/abs/2403.14617)
• Keywords: diffusion, generative
• Abstract: We introduce Videoshop, a training-free video editing algorithm for localized semantic edits. Videoshop allows users to use any editing software, including Photoshop and generative inpainting, to modify the first frame; it automatically propagates those changes, with semantic, spatial, and temporally consistent motion, to the remaining frames. Unlike existing methods that enable edits only through imprecise textual instructions, Videoshop allows users to add or remove objects, semantically change objects, insert stock photos into videos, etc. with fine-grained control over locations and appearance. We achieve this through image-based video editing by inverting latents with noise extrapolation, from which we generate videos conditioned on the edited image. Videoshop produces higher quality edits against 6 baselines on 2 editing benchmarks using 10 evaluation metrics.

Title: GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation

• Authors: Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, Gordon Wetzstein
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14621
• Pdf URL: https://arxiv.org/pdf/2403.14621
• Copy Paste: [[2403.14621]] GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation(https://arxiv.org/abs/2403.14621)
• Keywords: diffusion, generative
• Abstract: We introduce GRM, a large-scale reconstructor capable of recovering a 3D asset from sparse-view images in around 0.1s. GRM is a feed-forward transformer-based model that efficiently incorporates multi-view information to translate the input pixels into pixel-aligned Gaussians, which are unprojected to create a set of densely distributed 3D Gaussians representing a scene. Together, our transformer architecture and the use of 3D Gaussians unlock a scalable and efficient reconstruction framework. Extensive experimental results demonstrate the superiority of our method over alternatives regarding both reconstruction quality and efficiency. We also showcase the potential of GRM in generative tasks, i.e., text-to-3D and image-to-3D, by integrating it with existing multi-view diffusion models. Our project website is at: https://justimyhxu.github.io/projects/grm/.

Title: Simplified Diffusion Schrödinger Bridge

• Authors: Zhicong Tang, Tiankai Hang, Shuyang Gu, Dong Chen, Baining Guo
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14623
• Pdf URL: https://arxiv.org/pdf/2403.14623
• Copy Paste: [[2403.14623]] Simplified Diffusion Schrödinger Bridge(https://arxiv.org/abs/2403.14623)
• Keywords: diffusion, generative
• Abstract: This paper introduces a novel theoretical simplification of the Diffusion Schr\"odinger Bridge (DSB) that facilitates its unification with Score-based Generative Models (SGMs), addressing the limitations of DSB in complex data generation and enabling faster convergence and enhanced performance. By employing SGMs as an initial solution for DSB, our approach capitalizes on the strengths of both frameworks, ensuring a more efficient training process and improving the performance of SGM. We also propose a reparameterization technique that, despite theoretical approximations, practically improves the network's fitting capabilities. Our extensive experimental evaluations confirm the effectiveness of the simplified DSB, demonstrating its significant improvements. We believe the contributions of this work pave the way for advanced generative modeling. The code is available at https://github.com/tzco/Simplified-Diffusion-Schrodinger-Bridge.

Title: LiFT: A Surprisingly Simple Lightweight Feature Transform for Dense ViT Descriptors

• Authors: Saksham Suri, Matthew Walmer, Kamal Gupta, Abhinav Shrivastava
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14625
• Pdf URL: https://arxiv.org/pdf/2403.14625
• Copy Paste: [[2403.14625]] LiFT: A Surprisingly Simple Lightweight Feature Transform for Dense ViT Descriptors(https://arxiv.org/abs/2403.14625)
• Keywords: self-supervised
• Abstract: We present a simple self-supervised method to enhance the performance of ViT features for dense downstream tasks. Our Lightweight Feature Transform (LiFT) is a straightforward and compact postprocessing network that can be applied to enhance the features of any pre-trained ViT backbone. LiFT is fast and easy to train with a self-supervised objective, and it boosts the density of ViT features for minimal extra inference cost. Furthermore, we demonstrate that LiFT can be applied with approaches that use additional task-specific downstream modules, as we integrate LiFT with ViTDet for COCO detection and segmentation. Despite the simplicity of LiFT, we find that it is not simply learning a more complex version of bilinear interpolation. Instead, our LiFT training protocol leads to several desirable emergent properties that benefit ViT features in dense downstream tasks. This includes greater scale invariance for features, and better object boundary maps. By simply training LiFT for a few epochs, we show improved performance on keypoint correspondence, detection, segmentation, and object discovery tasks. Overall, LiFT provides an easy way to unlock the benefits of denser feature arrays for a fraction of the computational cost. For more details, refer to our project page at https://www.cs.umd.edu/~sakshams/LiFT/.