2024-04-04

Title: A Generative Deep Learning Approach for Crash Severity Modeling with Imbalanced Data

• Authors: Junlan Chen, Ziyuan Pu, Nan Zheng, Xiao Wen, Hongliang Ding, Xiucheng Guo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02187
• Pdf URL: https://arxiv.org/pdf/2404.02187
• Copy Paste: [[2404.02187]] A Generative Deep Learning Approach for Crash Severity Modeling with Imbalanced Data(https://arxiv.org/abs/2404.02187)
• Keywords: generative
• Abstract: Crash data is often greatly imbalanced, with the majority of crashes being non-fatal crashes, and only a small number being fatal crashes due to their rarity. Such data imbalance issue poses a challenge for crash severity modeling since it struggles to fit and interpret fatal crash outcomes with very limited samples. Usually, such data imbalance issues are addressed by data resampling methods, such as under-sampling and over-sampling techniques. However, most traditional and deep learning-based data resampling methods, such as synthetic minority oversampling technique (SMOTE) and generative Adversarial Networks (GAN) are designed dedicated to processing continuous variables. Though some resampling methods have improved to handle both continuous and discrete variables, they may have difficulties in dealing with the collapse issue associated with sparse discrete risk factors. Moreover, there is a lack of comprehensive studies that compare the performance of various resampling methods in crash severity modeling. To address the aforementioned issues, the current study proposes a crash data generation method based on the Conditional Tabular GAN. After data balancing, a crash severity model is employed to estimate the performance of classification and interpretation. A comparative study is conducted to assess classification accuracy and distribution consistency of the proposed generation method using a 4-year imbalanced crash dataset collected in Washington State, U.S. Additionally, Monte Carlo simulation is employed to estimate the performance of parameter and probability estimation in both two- and three-class imbalance scenarios. The results indicate that using synthetic data generated by CTGAN-RU for crash severity modeling outperforms using original data or synthetic data generated by other resampling methods.

Title: Emergent Abilities in Reduced-Scale Generative Language Models

• Authors: Sherin Muckatira, Vijeta Deshpande, Vladislav Lialin, Anna Rumshisky
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02204
• Pdf URL: https://arxiv.org/pdf/2404.02204
• Copy Paste: [[2404.02204]] Emergent Abilities in Reduced-Scale Generative Language Models(https://arxiv.org/abs/2404.02204)
• Keywords: generative, in-context
• Abstract: Large language models can solve new tasks without task-specific fine-tuning. This ability, also known as in-context learning (ICL), is considered an emergent ability and is primarily seen in large language models with billions of parameters. This study investigates if such emergent properties are strictly tied to model size or can be demonstrated by smaller models trained on reduced-scale data. To explore this, we simplify pre-training data and pre-train 36 causal language models with parameters varying from 1 million to 165 million parameters. We show that models trained on this simplified pre-training data demonstrate enhanced zero-shot capabilities across various tasks in simplified language, achieving performance comparable to that of pre-trained models six times larger on unrestricted language. This suggests that downscaling the language allows zero-shot learning capabilities to emerge in models with limited size. Additionally, we find that these smaller models pre-trained on simplified data demonstrate a power law relationship between the evaluation loss and the three scaling factors: compute, dataset size, and model size.

Title: Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better

• Authors: Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B. Blaschko, Sergey Yekhanin, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02241
• Pdf URL: https://arxiv.org/pdf/2404.02241
• Copy Paste: [[2404.02241]] Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better(https://arxiv.org/abs/2404.02241)
• Keywords: diffusion, generative
• Abstract: Diffusion Models (DM) and Consistency Models (CM) are two types of popular generative models with good generation quality on various tasks. When training DM and CM, intermediate weight checkpoints are not fully utilized and only the last converged checkpoint is used. In this work, we find that high-quality model weights often lie in a basin which cannot be reached by SGD but can be obtained by proper checkpoint averaging. Based on these observations, we propose LCSC, a simple but effective and efficient method to enhance the performance of DM and CM, by combining checkpoints along the training trajectory with coefficients deduced from evolutionary search. We demonstrate the value of LCSC through two use cases: $\textbf{(a) Reducing training cost.}$ With LCSC, we only need to train DM/CM with fewer number of iterations and/or lower batch sizes to obtain comparable sample quality with the fully trained model. For example, LCSC achieves considerable training speedups for CM (23$\times$ on CIFAR-10 and 15$\times$ on ImageNet-64). $\textbf{(b) Enhancing pre-trained models.}$ Assuming full training is already done, LCSC can further improve the generation quality or speed of the final converged models. For example, LCSC achieves better performance using 1 number of function evaluation (NFE) than the base model with 2 NFE on consistency distillation, and decreases the NFE of DM from 15 to 9 while maintaining the generation quality on CIFAR-10. Our code is available at https://github.com/imagination-research/LCSC.

Title: Heat Death of Generative Models in Closed-Loop Learning

• Authors: Matteo Marchi, Stefano Soatto, Pratik Chaudhari, Paulo Tabuada
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02325
• Pdf URL: https://arxiv.org/pdf/2404.02325
• Copy Paste: [[2404.02325]] Heat Death of Generative Models in Closed-Loop Learning(https://arxiv.org/abs/2404.02325)
• Keywords: diffusion, generative
• Abstract: Improvement and adoption of generative machine learning models is rapidly accelerating, as exemplified by the popularity of LLMs (Large Language Models) for text, and diffusion models for image generation.As generative models become widespread, data they generate is incorporated into shared content through the public web. This opens the question of what happens when data generated by a model is fed back to the model in subsequent training campaigns. This is a question about the stability of the training process, whether the distribution of publicly accessible content, which we refer to as "knowledge", remains stable or collapses. Small scale empirical experiments reported in the literature show that this closed-loop training process is prone to degenerating. Models may start producing gibberish data, or sample from only a small subset of the desired data distribution (a phenomenon referred to as mode collapse). So far there has been only limited theoretical understanding of this process, in part due to the complexity of the deep networks underlying these generative models. The aim of this paper is to provide insights into this process (that we refer to as "generative closed-loop learning") by studying the learning dynamics of generative models that are fed back their own produced content in addition to their original training dataset. The sampling of many of these models can be controlled via a "temperature" parameter. Using dynamical systems tools, we show that, unless a sufficient amount of external data is introduced at each iteration, any non-trivial temperature leads the model to asymptotically degenerate. In fact, either the generative distribution collapses to a small set of outputs, or becomes uniform over a large set of outputs.

Title: Semantic Augmentation in Images using Language

• Authors: Sahiti Yerramilli, Jayant Sravan Tamarapalli, Tanmay Girish Kulkarni, Jonathan Francis, Eric Nyberg
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02353
• Pdf URL: https://arxiv.org/pdf/2404.02353
• Copy Paste: [[2404.02353]] Semantic Augmentation in Images using Language(https://arxiv.org/abs/2404.02353)
• Keywords: diffusion
• Abstract: Deep Learning models are incredibly data-hungry and require very large labeled datasets for supervised learning. As a consequence, these models often suffer from overfitting, limiting their ability to generalize to real-world examples. Recent advancements in diffusion models have enabled the generation of photorealistic images based on textual inputs. Leveraging the substantial datasets used to train these diffusion models, we propose a technique to utilize generated images to augment existing datasets. This paper explores various strategies for effective data augmentation to improve the out-of-domain generalization capabilities of deep learning models.

Title: Enhancing Diffusion-based Point Cloud Generation with Smoothness Constraint

• Authors: Yukun Li, Liping Liu
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02396
• Pdf URL: https://arxiv.org/pdf/2404.02396
• Copy Paste: [[2404.02396]] Enhancing Diffusion-based Point Cloud Generation with Smoothness Constraint(https://arxiv.org/abs/2404.02396)
• Keywords: diffusion
• Abstract: Diffusion models have been popular for point cloud generation tasks. Existing works utilize the forward diffusion process to convert the original point distribution into a noise distribution and then learn the reverse diffusion process to recover the point distribution from the noise distribution. However, the reverse diffusion process can produce samples with non-smooth points on the surface because of the ignorance of the point cloud geometric properties. We propose alleviating the problem by incorporating the local smoothness constraint into the diffusion framework for point cloud generation. Experiments demonstrate the proposed model can generate realistic shapes and smoother point clouds, outperforming multiple state-of-the-art methods.

Title: Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data

• Authors: Parth Patwa, Simone Filice, Zhiyu Chen, Giuseppe Castellucci, Oleg Rokhlenko, Shervin Malmasi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02422
• Pdf URL: https://arxiv.org/pdf/2404.02422
• Copy Paste: [[2404.02422]] Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data(https://arxiv.org/abs/2404.02422)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) operating in 0-shot or few-shot settings achieve competitive results in Text Classification tasks. In-Context Learning (ICL) typically achieves better accuracy than the 0-shot setting, but it pays in terms of efficiency, due to the longer input prompt. In this paper, we propose a strategy to make LLMs as efficient as 0-shot text classifiers, while getting comparable or better accuracy than ICL. Our solution targets the low resource setting, i.e., when only 4 examples per class are available. Using a single LLM and few-shot real data we perform a sequence of generation, filtering and Parameter-Efficient Fine-Tuning steps to create a robust and efficient classifier. Experimental results show that our approach leads to competitive results on multiple text classification datasets.

Title: Designing a Photonic Physically Unclonable Function Having Resilience to Machine Learning Attacks

• Authors: Elena R. Henderson, Jessie M. Henderson, Hiva Shahoei, William V. Oxford, Eric C. Larson, Duncan L. MacFarlane, Mitchell A. Thornton
• Subjects: cs.CR, physics.optics
• Abstract URL: https://arxiv.org/abs/2404.02440
• Pdf URL: https://arxiv.org/pdf/2404.02440
• Copy Paste: [[2404.02440]] Designing a Photonic Physically Unclonable Function Having Resilience to Machine Learning Attacks(https://arxiv.org/abs/2404.02440)
• Keywords: generative
• Abstract: Physically unclonable functions (PUFs) are designed to act as device 'fingerprints.' Given an input challenge, the PUF circuit should produce an unpredictable response for use in situations such as root-of-trust applications and other hardware-level cybersecurity applications. PUFs are typically subcircuits present within integrated circuits (ICs), and while conventional IC PUFs are well-understood, several implementations have proven vulnerable to malicious exploits, including those perpetrated by machine learning (ML)-based attacks. Such attacks can be difficult to prevent because they are often designed to work even when relatively few challenge-response pairs are known in advance. Hence the need for both more resilient PUF designs and analysis of ML-attack susceptibility. Previous work has developed a PUF for photonic integrated circuits (PICs). A PIC PUF not only produces unpredictable responses given manufacturing-introduced tolerances, but is also less prone to electromagnetic radiation eavesdropping attacks than a purely electronic IC PUF. In this work, we analyze the resilience of the proposed photonic PUF when subjected to ML-based attacks. Specifically, we describe a computational PUF model for producing the large datasets required for training ML attacks; we analyze the quality of the model; and we discuss the modeled PUF's susceptibility to ML-based attacks. We find that the modeled PUF generates distributions that resemble uniform white noise, explaining the exhibited resilience to neural-network-based attacks designed to exploit latent relationships between challenges and responses. Preliminary analysis suggests that the PUF exhibits similar resilience to generative adversarial networks, and continued development will show whether more-sophisticated ML approaches better compromise the PUF and -- if so -- how design modifications might improve resilience.

Title: Masked Completion via Structured Diffusion with White-Box Transformers

• Authors: Druv Pai, Ziyang Wu, Sam Buchanan, Yaodong Yu, Yi Ma
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.02446
• Pdf URL: https://arxiv.org/pdf/2404.02446
• Copy Paste: [[2404.02446]] Masked Completion via Structured Diffusion with White-Box Transformers(https://arxiv.org/abs/2404.02446)
• Keywords: diffusion
• Abstract: Modern learning frameworks often train deep neural networks with massive amounts of unlabeled data to learn representations by solving simple pretext tasks, then use the representations as foundations for downstream tasks. These networks are empirically designed; as such, they are usually not interpretable, their representations are not structured, and their designs are potentially redundant. White-box deep networks, in which each layer explicitly identifies and transforms structures in the data, present a promising alternative. However, existing white-box architectures have only been shown to work at scale in supervised settings with labeled data, such as classification. In this work, we provide the first instantiation of the white-box design paradigm that can be applied to large-scale unsupervised representation learning. We do this by exploiting a fundamental connection between diffusion, compression, and (masked) completion, deriving a deep transformer-like masked autoencoder architecture, called CRATE-MAE, in which the role of each layer is mathematically fully interpretable: they transform the data distribution to and from a structured representation. Extensive empirical evaluations confirm our analytical insights. CRATE-MAE demonstrates highly promising performance on large-scale imagery datasets while using only ~30% of the parameters compared to the standard masked autoencoder with the same model configuration. The representations learned by CRATE-MAE have explicit structure and also contain semantic meaning. Code is available at https://github.com/Ma-Lab-Berkeley/CRATE .

Title: Task Agnostic Architecture for Algorithm Induction via Implicit Composition

• Authors: Sahil J. Sindhi, Ignas Budvytis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02450
• Pdf URL: https://arxiv.org/pdf/2404.02450
• Copy Paste: [[2404.02450]] Task Agnostic Architecture for Algorithm Induction via Implicit Composition(https://arxiv.org/abs/2404.02450)
• Keywords: generative, in-context
• Abstract: Different fields in applied machine learning such as computer vision, speech or natural language processing have been building domain-specialised solutions. Currently, we are witnessing an opposing trend towards developing more generalist architectures, driven by Large Language Models and multi-modal foundational models. These architectures are designed to tackle a variety of tasks, including those previously unseen and using inputs across multiple modalities. Taking this trend of generalization to the extreme suggests the possibility of a single deep network architecture capable of solving all tasks. This position paper aims to explore developing such a unified architecture and proposes a theoretical framework of how it could be constructed. Our proposal is based on the following assumptions. Firstly, tasks are solved by following a sequence of instructions, typically implemented in code for conventional computing hardware, which inherently operates sequentially. Second, recent Generative AI, especially Transformer-based models, demonstrate potential as an architecture capable of constructing algorithms for a wide range of domains. For example, GPT-4 shows exceptional capability at in-context learning of novel tasks which is hard to explain in any other way than the ability to compose novel solutions from fragments on previously learnt algorithms. Third, the observation that the main missing component in developing a truly generalised network is an efficient approach for self-consistent input of previously learnt sub-steps of an algorithm and their (implicit) composition during the network's internal forward pass. Our exploration delves into current capabilities and limitations of Transformer-based and other methods in efficient and correct algorithm composition and proposes a Transformer-like architecture as well as a discrete learning framework to overcome these limitations.

Title: Adaptive Cross-lingual Text Classification through In-Context One-Shot Demonstrations

• Authors: Emilio Villa-Cueva, A. Pastor López-Monroy, Fernando Sánchez-Vega, Thamar Solorio
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02452
• Pdf URL: https://arxiv.org/pdf/2404.02452
• Copy Paste: [[2404.02452]] Adaptive Cross-lingual Text Classification through In-Context One-Shot Demonstrations(https://arxiv.org/abs/2404.02452)
• Keywords: in-context
• Abstract: Zero-Shot Cross-lingual Transfer (ZS-XLT) utilizes a model trained in a source language to make predictions in another language, often with a performance loss. To alleviate this, additional improvements can be achieved through subsequent adaptation using examples in the target language. In this paper, we exploit In-Context Tuning (ICT) for One-Shot Cross-lingual transfer in the classification task by introducing In-Context Cross-lingual Transfer (IC-XLT). The novel concept involves training a model to learn from context examples and subsequently adapting it during inference to a target language by prepending a One-Shot context demonstration in that language. Our results show that IC-XLT successfully leverages target-language examples to improve the cross-lingual capabilities of the evaluated mT5 model, outperforming prompt-based models in the Zero and Few-shot scenarios adapted through fine-tuning. Moreover, we show that when source-language data is limited, the fine-tuning framework employed for IC-XLT performs comparably to prompt-based fine-tuning with significantly more training data in the source language.

Title: On the Efficiency and Robustness of Vibration-based Foundation Models for IoT Sensing: A Case Study

• Authors: Tomoyoshi Kimura, Jinyang Li, Tianshi Wang, Denizhan Kara, Yizhuo Chen, Yigong Hu, Ruijie Wang, Maggie Wigness, Shengzhong Liu, Mani Srivastava, Suhas Diggavi, Tarek Abdelzaher
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.02461
• Pdf URL: https://arxiv.org/pdf/2404.02461
• Copy Paste: [[2404.02461]] On the Efficiency and Robustness of Vibration-based Foundation Models for IoT Sensing: A Case Study(https://arxiv.org/abs/2404.02461)
• Keywords: self-supervised, foundation model
• Abstract: This paper demonstrates the potential of vibration-based Foundation Models (FMs), pre-trained with unlabeled sensing data, to improve the robustness of run-time inference in (a class of) IoT applications. A case study is presented featuring a vehicle classification application using acoustic and seismic sensing. The work is motivated by the success of foundation models in the areas of natural language processing and computer vision, leading to generalizations of the FM concept to other domains as well, where significant amounts of unlabeled data exist that can be used for self-supervised pre-training. One such domain is IoT applications. Foundation models for selected sensing modalities in the IoT domain can be pre-trained in an environment-agnostic fashion using available unlabeled sensor data and then fine-tuned to the deployment at hand using a small amount of labeled data. The paper shows that the pre-training/fine-tuning approach improves the robustness of downstream inference and facilitates adaptation to different environmental conditions. More specifically, we present a case study in a real-world setting to evaluate a simple (vibration-based) FM-like model, called FOCAL, demonstrating its superior robustness and adaptation, compared to conventional supervised deep neural networks (DNNs). We also demonstrate its superior convergence over supervised solutions. Our findings highlight the advantages of vibration-based FMs (and FM-inspired selfsupervised models in general) in terms of inference robustness, runtime efficiency, and model adaptation (via fine-tuning) in resource-limited IoT settings.

Title: A Unified Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability

• Authors: Jie Zhu, Jirong Zha, Ding Li, Leye Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02462
• Pdf URL: https://arxiv.org/pdf/2404.02462
• Copy Paste: [[2404.02462]] A Unified Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability(https://arxiv.org/abs/2404.02462)
• Keywords: self-supervised
• Abstract: Self-supervised learning shows promise in harnessing extensive unlabeled data, but it also confronts significant privacy concerns, especially in vision. In this paper, we aim to perform membership inference on visual self-supervised models in a more realistic setting: self-supervised training method and details are unknown for an adversary when attacking as he usually faces a black-box system in practice. In this setting, considering that self-supervised model could be trained by completely different self-supervised paradigms, e.g., masked image modeling and contrastive learning, with complex training details, we propose a unified membership inference method called PartCrop. It is motivated by the shared part-aware capability among models and stronger part response on the training data. Specifically, PartCrop crops parts of objects in an image to query responses with the image in representation space. We conduct extensive attacks on self-supervised models with different training protocols and structures using three widely used image datasets. The results verify the effectiveness and generalization of PartCrop. Moreover, to defend against PartCrop, we evaluate two common approaches, i.e., early stop and differential privacy, and propose a tailored method called shrinking crop scale range. The defense experiments indicate that all of them are effective. Our code is available at https://github.com/JiePKU/PartCrop

Title: uTeBC-NLP at SemEval-2024 Task 9: Can LLMs be Lateral Thinkers?

• Authors: Pouya Sadeghi, Amirhossein Abaskohi, Yadollah Yaghoobzadeh
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02474
• Pdf URL: https://arxiv.org/pdf/2404.02474
• Copy Paste: [[2404.02474]] uTeBC-NLP at SemEval-2024 Task 9: Can LLMs be Lateral Thinkers?(https://arxiv.org/abs/2404.02474)
• Keywords: in-context
• Abstract: Inspired by human cognition, Jiang et al.(2023c) create a benchmark for assessing LLMs' lateral thinking-thinking outside the box. Building upon this benchmark, we investigate how different prompting methods enhance LLMs' performance on this task to reveal their inherent power for outside-the-box thinking ability. Through participating in SemEval-2024, task 9, Sentence Puzzle sub-task, we explore prompt engineering methods: chain of thoughts (CoT) and direct prompting, enhancing with informative descriptions, and employing contextualizing prompts using a retrieval augmented generation (RAG) pipeline. Our experiments involve three LLMs including GPT-3.5, GPT-4, and Zephyr-7B-beta. We generate a dataset of thinking paths between riddles and options using GPT-4, validated by humans for quality. Findings indicate that compressed informative prompts enhance performance. Dynamic in-context learning enhances model performance significantly. Furthermore, fine-tuning Zephyr on our dataset enhances performance across other commonsense datasets, underscoring the value of innovative thinking.

Title: Dynamic Demonstration Retrieval and Cognitive Understanding for Emotional Support Conversation

• Authors: Zhe Xu, Daoyuan Chen, Jiayi Kuang, Zihao Yi, Yaliang Li, Ying Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02505
• Pdf URL: https://arxiv.org/pdf/2404.02505
• Copy Paste: [[2404.02505]] Dynamic Demonstration Retrieval and Cognitive Understanding for Emotional Support Conversation(https://arxiv.org/abs/2404.02505)
• Keywords: in-context
• Abstract: Emotional Support Conversation (ESC) systems are pivotal in providing empathetic interactions, aiding users through negative emotional states by understanding and addressing their unique experiences. In this paper, we tackle two key challenges in ESC: enhancing contextually relevant and empathetic response generation through dynamic demonstration retrieval, and advancing cognitive understanding to grasp implicit mental states comprehensively. We introduce Dynamic Demonstration Retrieval and Cognitive-Aspect Situation Understanding (\ourwork), a novel approach that synergizes these elements to improve the quality of support provided in ESCs. By leveraging in-context learning and persona information, we introduce an innovative retrieval mechanism that selects informative and personalized demonstration pairs. We also propose a cognitive understanding module that utilizes four cognitive relationships from the ATOMIC knowledge source to deepen situational awareness of help-seekers' mental states. Our supportive decoder integrates information from diverse knowledge sources, underpinning response generation that is both empathetic and cognitively aware. The effectiveness of \ourwork is demonstrated through extensive automatic and human evaluations, revealing substantial improvements over numerous state-of-the-art models, with up to 13.79\% enhancement in overall performance of ten metrics. Our codes are available for public access to facilitate further research and development.

Title: Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Languages

• Authors: Vandan Mujadia, Pruthwik Mishra, Arafat Ahsan, Dipti Misra Sharma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02512
• Pdf URL: https://arxiv.org/pdf/2404.02512
• Copy Paste: [[2404.02512]] Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Languages(https://arxiv.org/abs/2404.02512)
• Keywords: in-context
• Abstract: With the primary focus on evaluating the effectiveness of large language models for automatic reference-less translation assessment, this work presents our experiments on mimicking human direct assessment to evaluate the quality of translations in English and Indian languages. We constructed a translation evaluation task where we performed zero-shot learning, in-context example-driven learning, and fine-tuning of large language models to provide a score out of 100, where 100 represents a perfect translation and 1 represents a poor translation. We compared the performance of our trained systems with existing methods such as COMET, BERT-Scorer, and LABSE, and found that the LLM-based evaluator (LLaMA-2-13B) achieves a comparable or higher overall correlation with human judgments for the considered Indian language pairs.

Title: Severity Controlled Text-to-Image Generative Model Bias Manipulation

• Authors: Jordan Vice, Naveed Akhtar, Richard Hartley, Ajmal Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02530
• Pdf URL: https://arxiv.org/pdf/2404.02530
• Copy Paste: [[2404.02530]] Severity Controlled Text-to-Image Generative Model Bias Manipulation(https://arxiv.org/abs/2404.02530)
• Keywords: generative
• Abstract: Text-to-image (T2I) generative models are gaining wide popularity, especially in public domains. However, their intrinsic bias and potential malicious manipulations remain under-explored. Charting the susceptibility of T2I models to such manipulation, we first expose the new possibility of a dynamic and computationally efficient exploitation of model bias by targeting the embedded language models. By leveraging mathematical foundations of vector algebra, our technique enables a scalable and convenient control over the severity of output manipulation through model bias. As a by-product, this control also allows a form of precise prompt engineering to generate images which are generally implausible with regular text prompts. We also demonstrate a constructive application of our manipulation for balancing the frequency of generated classes - as in model debiasing. Our technique does not require training and is also framed as a backdoor attack with severity control using semantically-null text triggers in the prompts. With extensive analysis, we present interesting qualitative and quantitative results to expose potential manipulation possibilities for T2I models. Key-words: Text-to-Image Models, Generative Models, Backdoor Attacks, Prompt Engineering, Bias

Title: Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models

• Authors: Matteo Pennisi, Giovanni Bellitto, Simone Palazzo, Mubarak Shah, Concetto Spampinato
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02618
• Pdf URL: https://arxiv.org/pdf/2404.02618
• Copy Paste: [[2404.02618]] Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models(https://arxiv.org/abs/2404.02618)
• Keywords: diffusion
• Abstract: We present DiffExplainer, a novel framework that, leveraging language-vision models, enables multimodal global explainability. DiffExplainer employs diffusion models conditioned on optimized text prompts, synthesizing images that maximize class outputs and hidden features of a classifier, thus providing a visual tool for explaining decisions. Moreover, the analysis of generated visual descriptions allows for automatic identification of biases and spurious features, as opposed to traditional methods that often rely on manual intervention. The cross-modal transferability of language-vision models also enables the possibility to describe decisions in a more human-interpretable way, i.e., through text. We conduct comprehensive experiments, which include an extensive user study, demonstrating the effectiveness of DiffExplainer on 1) the generation of high-quality images explaining model decisions, surpassing existing activation maximization methods, and 2) the automated identification of biases and spurious features.

Title: Design2Cloth: 3D Cloth Generation from 2D Masks

• Authors: Jiali Zheng, Rolandos Alexandros Potamias, Stefanos Zafeiriou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02686
• Pdf URL: https://arxiv.org/pdf/2404.02686
• Copy Paste: [[2404.02686]] Design2Cloth: 3D Cloth Generation from 2D Masks(https://arxiv.org/abs/2404.02686)
• Keywords: generative
• Abstract: In recent years, there has been a significant shift in the field of digital avatar research, towards modeling, animating and reconstructing clothed human representations, as a key step towards creating realistic avatars. However, current 3D cloth generation methods are garment specific or trained completely on synthetic data, hence lacking fine details and realism. In this work, we make a step towards automatic realistic garment design and propose Design2Cloth, a high fidelity 3D generative model trained on a real world dataset from more than 2000 subject scans. To provide vital contribution to the fashion industry, we developed a user-friendly adversarial model capable of generating diverse and detailed clothes simply by drawing a 2D cloth mask. Under a series of both qualitative and quantitative experiments, we showcase that Design2Cloth outperforms current state-of-the-art cloth generative models by a large margin. In addition to the generative properties of our network, we showcase that the proposed method can be used to achieve high quality reconstructions from single in-the-wild images and 3D scans. Dataset, code and pre-trained model will become publicly available.

Title: Deep Privacy Funnel Model: From a Discriminative to a Generative Approach with an Application to Face Recognition

• Authors: Behrooz Razeghi, Parsa Rahimi, Sébastien Marcel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02696
• Pdf URL: https://arxiv.org/pdf/2404.02696
• Copy Paste: [[2404.02696]] Deep Privacy Funnel Model: From a Discriminative to a Generative Approach with an Application to Face Recognition(https://arxiv.org/abs/2404.02696)
• Keywords: diffusion, generative
• Abstract: In this study, we apply the information-theoretic Privacy Funnel (PF) model to the domain of face recognition, developing a novel method for privacy-preserving representation learning within an end-to-end training framework. Our approach addresses the trade-off between obfuscation and utility in data protection, quantified through logarithmic loss, also known as self-information loss. This research provides a foundational exploration into the integration of information-theoretic privacy principles with representation learning, focusing specifically on the face recognition systems. We particularly highlight the adaptability of our framework with recent advancements in face recognition networks, such as AdaFace and ArcFace. In addition, we introduce the Generative Privacy Funnel ($\mathsf{GenPF}$) model, a paradigm that extends beyond the traditional scope of the PF model, referred to as the Discriminative Privacy Funnel ($\mathsf{DisPF}$). This $\mathsf{GenPF}$ model brings new perspectives on data generation methods with estimation-theoretic and information-theoretic privacy guarantees. Complementing these developments, we also present the deep variational PF (DVPF) model. This model proposes a tractable variational bound for measuring information leakage, enhancing the understanding of privacy preservation challenges in deep representation learning. The DVPF model, associated with both $\mathsf{DisPF}$ and $\mathsf{GenPF}$ models, sheds light on connections with various generative models such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Diffusion models. Complementing our theoretical contributions, we release a reproducible PyTorch package, facilitating further exploration and application of these privacy-preserving methodologies in face recognition systems.

Title: Model-agnostic Origin Attribution of Generated Images with Few-shot Examples

• Authors: Fengyuan Liu, Haochen Luo, Yiming Li, Philip Torr, Jindong Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02697
• Pdf URL: https://arxiv.org/pdf/2404.02697
• Copy Paste: [[2404.02697]] Model-agnostic Origin Attribution of Generated Images with Few-shot Examples(https://arxiv.org/abs/2404.02697)
• Keywords: generative
• Abstract: Recent progress in visual generative models enables the generation of high-quality images. To prevent the misuse of generated images, it is important to identify the origin model that generates them. In this work, we study the origin attribution of generated images in a practical setting where only a few images generated by a source model are available and the source model cannot be accessed. The goal is to check if a given image is generated by the source model. We first formulate this problem as a few-shot one-class classification task. To solve the task, we propose OCC-CLIP, a CLIP-based framework for few-shot one-class classification, enabling the identification of an image's source model, even among multiple candidates. Extensive experiments corresponding to various generative models verify the effectiveness of our OCC-CLIP framework. Furthermore, an experiment based on the recently released DALL-E 3 API verifies the real-world applicability of our solution.

Title: Harnessing the Power of Large Vision Language Models for Synthetic Image Detection

• Authors: Mamadou Keita, Wassim Hamidouche, Hassen Bougueffa, Abdenour Hadid, Abdelmalik Taleb-Ahmed
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02726
• Pdf URL: https://arxiv.org/pdf/2404.02726
• Copy Paste: [[2404.02726]] Harnessing the Power of Large Vision Language Models for Synthetic Image Detection(https://arxiv.org/abs/2404.02726)
• Keywords: diffusion
• Abstract: In recent years, the emergence of models capable of generating images from text has attracted considerable interest, offering the possibility of creating realistic images from text descriptions. Yet these advances have also raised concerns about the potential misuse of these images, including the creation of misleading content such as fake news and propaganda. This study investigates the effectiveness of using advanced vision-language models (VLMs) for synthetic image identification. Specifically, the focus is on tuning state-of-the-art image captioning models for synthetic image detection. By harnessing the robust understanding capabilities of large VLMs, the aim is to distinguish authentic images from synthetic images produced by diffusion-based models. This study contributes to the advancement of synthetic image detection by exploiting the capabilities of visual language models such as BLIP-2 and ViTGPT2. By tailoring image captioning models, we address the challenges associated with the potential misuse of synthetic images in real-world applications. Results described in this paper highlight the promising role of VLMs in the field of synthetic image detection, outperforming conventional image-based detection techniques. Code and models can be found at https://github.com/Mamadou-Keita/VLM-DETECT.

Title: InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation

• Authors: Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, Anthony Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02733
• Pdf URL: https://arxiv.org/pdf/2404.02733
• Copy Paste: [[2404.02733]] InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation(https://arxiv.org/abs/2404.02733)
• Keywords: diffusion
• Abstract: Tuning-free diffusion-based models have demonstrated significant potential in the realm of image personalization and customization. However, despite this notable progress, current models continue to grapple with several complex challenges in producing style-consistent image generation. Firstly, the concept of style is inherently underdetermined, encompassing a multitude of elements such as color, material, atmosphere, design, and structure, among others. Secondly, inversion-based methods are prone to style degradation, often resulting in the loss of fine-grained details. Lastly, adapter-based approaches frequently require meticulous weight tuning for each reference image to achieve a balance between style intensity and text controllability. In this paper, we commence by examining several compelling yet frequently overlooked observations. We then proceed to introduce InstantStyle, a framework designed to address these issues through the implementation of two key strategies: 1) A straightforward mechanism that decouples style and content from reference images within the feature space, predicated on the assumption that features within the same space can be either added to or subtracted from one another. 2) The injection of reference image features exclusively into style-specific blocks, thereby preventing style leaks and eschewing the need for cumbersome weight tuning, which often characterizes more parameter-heavy designs.Our work demonstrates superior visual stylization outcomes, striking an optimal balance between the intensity of style and the controllability of textual elements. Our codes will be available at https://github.com/InstantStyle/InstantStyle.

Title: Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion Models

• Authors: Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, Jürgen Schmidhuber
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02747
• Pdf URL: https://arxiv.org/pdf/2404.02747
• Copy Paste: [[2404.02747]] Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion Models(https://arxiv.org/abs/2404.02747)
• Keywords: diffusion
• Abstract: This study explores the role of cross-attention during inference in text-conditional diffusion models. We find that cross-attention outputs converge to a fixed point after few inference steps. Accordingly, the time point of convergence naturally divides the entire inference process into two stages: an initial semantics-planning stage, during which, the model relies on cross-attention to plan text-oriented visual semantics, and a subsequent fidelity-improving stage, during which the model tries to generate images from previously planned semantics. Surprisingly, ignoring text conditions in the fidelity-improving stage not only reduces computation complexity, but also maintains model performance. This yields a simple and training-free method called TGATE for efficient generation, which caches the cross-attention output once it converges and keeps it fixed during the remaining inference steps. Our empirical study on the MS-COCO validation set confirms its effectiveness. The source code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.

Title: GenN2N: Generative NeRF2NeRF Translation

• Authors: Xiangyue Liu, Han Xue, Kunming Luo, Ping Tan, Li Yi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02788
• Pdf URL: https://arxiv.org/pdf/2404.02788
• Copy Paste: [[2404.02788]] GenN2N: Generative NeRF2NeRF Translation(https://arxiv.org/abs/2404.02788)
• Keywords: generative
• Abstract: We present GenN2N, a unified NeRF-to-NeRF translation framework for various NeRF translation tasks such as text-driven NeRF editing, colorization, super-resolution, inpainting, etc. Unlike previous methods designed for individual translation tasks with task-specific schemes, GenN2N achieves all these NeRF editing tasks by employing a plug-and-play image-to-image translator to perform editing in the 2D domain and lifting 2D edits into the 3D NeRF space. Since the 3D consistency of 2D edits may not be assured, we propose to model the distribution of the underlying 3D edits through a generative model that can cover all possible edited NeRFs. To model the distribution of 3D edited NeRFs from 2D edited images, we carefully design a VAE-GAN that encodes images while decoding NeRFs. The latent space is trained to align with a Gaussian distribution and the NeRFs are supervised through an adversarial loss on its renderings. To ensure the latent code does not depend on 2D viewpoints but truly reflects the 3D edits, we also regularize the latent code through a contrastive learning scheme. Extensive experiments on various editing tasks show GenN2N, as a universal framework, performs as well or better than task-specific specialists while possessing flexible generative power. More results on our project page: https://xiangyueliu.github.io/GenN2N/

Title: MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation

• Authors: Petru-Daniel Tudosiu, Yongxin Yang, Shifeng Zhang, Fei Chen, Steven McDonagh, Gerasimos Lampouras, Ignacio Iacobacci, Sarah Parisot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.02790
• Pdf URL: https://arxiv.org/pdf/2404.02790
• Copy Paste: [[2404.02790]] MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation(https://arxiv.org/abs/2404.02790)
• Keywords: generative
• Abstract: Text-to-image generation has achieved astonishing results, yet precise spatial controllability and prompt fidelity remain highly challenging. This limitation is typically addressed through cumbersome prompt engineering, scene layout conditioning, or image editing techniques which often require hand drawn masks. Nonetheless, pre-existing works struggle to take advantage of the natural instance-level compositionality of scenes due to the typically flat nature of rasterized RGB output images. Towards adressing this challenge, we introduce MuLAn: a novel dataset comprising over 44K MUlti-Layer ANnotations of RGB images as multilayer, instance-wise RGBA decompositions, and over 100K instance images. To build MuLAn, we developed a training free pipeline which decomposes a monocular RGB image into a stack of RGBA layers comprising of background and isolated instances. We achieve this through the use of pretrained general-purpose models, and by developing three modules: image decomposition for instance discovery and extraction, instance completion to reconstruct occluded areas, and image re-assembly. We use our pipeline to create MuLAn-COCO and MuLAn-LAION datasets, which contain a variety of image decompositions in terms of style, composition and complexity. With MuLAn, we provide the first photorealistic resource providing instance decomposition and occlusion information for high quality images, opening up new avenues for text-to-image generative AI research. With this, we aim to encourage the development of novel generation and editing technology, in particular layer-wise solutions. MuLAn data resources are available at https://MuLAn-dataset.github.io/.

Title: Generative-Contrastive Heterogeneous Graph Neural Network

• Authors: Yu Wang, Lei Sang, Yi Zhang, Yiwen Zhang
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.02810
• Pdf URL: https://arxiv.org/pdf/2404.02810
• Copy Paste: [[2404.02810]] Generative-Contrastive Heterogeneous Graph Neural Network(https://arxiv.org/abs/2404.02810)
• Keywords: self-supervised, generative
• Abstract: Heterogeneous Graphs (HGs) can effectively model complex relationships in the real world by multi-type nodes and edges. In recent years, inspired by self-supervised learning, contrastive Heterogeneous Graphs Neural Networks (HGNNs) have shown great potential by utilizing data augmentation and discriminators for downstream tasks. However, data augmentation is still limited due to the discrete and abstract nature of graphs. To tackle the above limitations, we propose a novel \textit{Generative-Contrastive Heterogeneous Graph Neural Network (GC-HGNN)}. Specifically, we first propose a heterogeneous graph generative learning enhanced contrastive paradigm. This paradigm includes: 1) A contrastive view augmentation strategy by using masked autoencoder. 2) Position-aware and semantics-aware positive sample sampling strategy for generate hard negative samples. 3) A hierarchical contrastive learning strategy for capturing local and global information. Furthermore, the hierarchical contrastive learning and sampling strategies aim to constitute an enhanced discriminator under the generative-contrastive perspective. Finally, we compare our model with seventeen baselines on eight real-world datasets. Our model outperforms the latest contrastive and generative baselines on node classification and link prediction tasks. To reproduce our work, we have open-sourced our code at https://github.com/xxx.

Title: Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison

• Authors: Maxime Bouthors, Josep Crego, Francois Yvon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.02835
• Pdf URL: https://arxiv.org/pdf/2404.02835
• Copy Paste: [[2404.02835]] Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison(https://arxiv.org/abs/2404.02835)
• Keywords: in-context
• Abstract: Retrieval-Augmented Neural Machine Translation (RAMT) architectures retrieve examples from memory to guide the generation process. While most works in this trend explore new ways to exploit the retrieved examples, the upstream retrieval step is mostly unexplored. In this paper, we study the effect of varying retrieval methods for several translation architectures, to better understand the interplay between these two processes. We conduct experiments in two language pairs in a multi-domain setting and consider several downstream architectures based on a standard autoregressive model, an edit-based model, and a large language model with in-context learning. Our experiments show that the choice of the retrieval technique impacts the translation scores, with variance across architectures. We also discuss the effects of increasing the number and diversity of examples, which are mostly positive across the board.

Title: End-To-End Self-tuning Self-supervised Time Series Anomaly Detection

• Authors: Boje Deforce, Meng-Chieh Lee, Bart Baesens, Estefanía Serral Asensio, Jaemin Yoo, Leman Akoglu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02865
• Pdf URL: https://arxiv.org/pdf/2404.02865
• Copy Paste: [[2404.02865]] End-To-End Self-tuning Self-supervised Time Series Anomaly Detection(https://arxiv.org/abs/2404.02865)
• Keywords: self-supervised, anomaly
• Abstract: Time series anomaly detection (TSAD) finds many applications such as monitoring environmental sensors, industry KPIs, patient biomarkers, etc. A two-fold challenge for TSAD is a versatile and unsupervised model that can detect various different types of time series anomalies (spikes, discontinuities, trend shifts, etc.) without any labeled data. Modern neural networks have outstanding ability in modeling complex time series. Self-supervised models in particular tackle unsupervised TSAD by transforming the input via various augmentations to create pseudo anomalies for training. However, their performance is sensitive to the choice of augmentation, which is hard to choose in practice, while there exists no effort in the literature on data augmentation tuning for TSAD without labels. Our work aims to fill this gap. We introduce TSAP for TSA "on autoPilot", which can (self-)tune augmentation hyperparameters end-to-end. It stands on two key components: a differentiable augmentation architecture and an unsupervised validation loss to effectively assess the alignment between augmentation type and anomaly type. Case studies show TSAP's ability to effectively select the (discrete) augmentation type and associated (continuous) hyperparameters. In turn, it outperforms established baselines, including SOTA self-supervised models, on diverse TSAD tasks exhibiting different anomaly types.

Title: On the Scalability of Diffusion-based Text-to-Image Generation

• Authors: Hao Li, Yang Zou, Ying Wang, Orchid Majumder, Yusheng Xie, R. Manmatha, Ashwin Swaminathan, Zhuowen Tu, Stefano Ermon, Stefano Soatto
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.02883
• Pdf URL: https://arxiv.org/pdf/2404.02883
• Copy Paste: [[2404.02883]] On the Scalability of Diffusion-based Text-to-Image Generation(https://arxiv.org/abs/2404.02883)
• Keywords: diffusion
• Abstract: Scaling up model and data size has been quite successful for the evolution of LLMs. However, the scaling law for the diffusion based text-to-image (T2I) models is not fully explored. It is also unclear how to efficiently scale the model for better performance at reduced cost. The different training settings and expensive training cost make a fair model comparison extremely difficult. In this work, we empirically study the scaling properties of diffusion based T2I models by performing extensive and rigours ablations on scaling both denoising backbones and training set, including training scaled UNet and Transformer variants ranging from 0.4B to 4B parameters on datasets upto 600M images. For model scaling, we find the location and amount of cross attention distinguishes the performance of existing UNet designs. And increasing the transformer blocks is more parameter-efficient for improving text-image alignment than increasing channel numbers. We then identify an efficient UNet variant, which is 45% smaller and 28% faster than SDXL's UNet. On the data scaling side, we show the quality and diversity of the training set matters more than simply dataset size. Increasing caption density and diversity improves text-image alignment performance and the learning efficiency. Finally, we provide scaling functions to predict the text-image alignment performance as functions of the scale of model size, compute and dataset size.

Title: MODNO: Multi Operator Learning With Distributed Neural Operators

• Authors: Zecheng Zhang
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2404.02892
• Pdf URL: https://arxiv.org/pdf/2404.02892
• Copy Paste: [[2404.02892]] MODNO: Multi Operator Learning With Distributed Neural Operators(https://arxiv.org/abs/2404.02892)
• Keywords: foundation model
• Abstract: The study of operator learning involves the utilization of neural networks to approximate operators. Traditionally, the focus has been on single-operator learning (SOL). However, recent advances have rapidly expanded this to include the approximation of multiple operators using foundation models equipped with millions or billions of trainable parameters, leading to the research of multi-operator learning (MOL). In this paper, we present a novel distributed training approach aimed at enabling a single neural operator with significantly fewer parameters to effectively tackle multi-operator learning challenges, all without incurring additional average costs. Our method is applicable to various Chen-Chen-type neural operators, such as Deep Operator Neural Networks (DON). The core idea is to independently learn the output basis functions for each operator using its dedicated data, while simultaneously centralizing the learning of the input function encoding shared by all operators using the entire dataset. Through a systematic study of five numerical examples, we compare the accuracy and cost of training a single neural operator for each operator independently versus training a MOL model using our proposed method. Our results demonstrate enhanced efficiency and satisfactory accuracy. Moreover, our approach illustrates that some operators with limited data can be more effectively constructed with the aid of data from analogous operators through MOL learning. This highlights another MOL's potential to bolster operator learning.

Title: MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignment

• Authors: Duygu Ceylan, Valentin Deschaintre, Thibault Groueix, Rosalie Martin, Chun-Hao Huang, Romain Rouffet, Vladimir Kim, Gaëtan Lassagne
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.02899
• Pdf URL: https://arxiv.org/pdf/2404.02899
• Copy Paste: [[2404.02899]] MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignment(https://arxiv.org/abs/2404.02899)
• Keywords: diffusion
• Abstract: We present MatAtlas, a method for consistent text-guided 3D model texturing. Following recent progress we leverage a large scale text-to-image generation model (e.g., Stable Diffusion) as a prior to texture a 3D model. We carefully design an RGB texturing pipeline that leverages a grid pattern diffusion, driven by depth and edges. By proposing a multi-step texture refinement process, we significantly improve the quality and 3D consistency of the texturing output. To further address the problem of baked-in lighting, we move beyond RGB colors and pursue assigning parametric materials to the assets. Given the high-quality initial RGB texture, we propose a novel material retrieval method capitalized on Large Language Models (LLM), enabling editabiliy and relightability. We evaluate our method on a wide variety of geometries and show that our method significantly outperform prior arts. We also analyze the role of each component through a detailed ablation study.

Title: LidarDM: Generative LiDAR Simulation in a Generated World

• Authors: Vlas Zyrianov, Henry Che, Zhijian Liu, Shenlong Wang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.02903
• Pdf URL: https://arxiv.org/pdf/2404.02903
• Copy Paste: [[2404.02903]] LidarDM: Generative LiDAR Simulation in a Generated World(https://arxiv.org/abs/2404.02903)
• Keywords: diffusion, generative
• Abstract: We present LidarDM, a novel LiDAR generative model capable of producing realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. LidarDM stands out with two unprecedented capabilities in LiDAR generative modeling: (i) LiDAR generation guided by driving scenarios, offering significant potential for autonomous driving simulations, and (ii) 4D LiDAR point cloud generation, enabling the creation of realistic and temporally coherent sequences. At the heart of our model is a novel integrated 4D world generation framework. Specifically, we employ latent diffusion models to generate the 3D scene, combine it with dynamic actors to form the underlying 4D world, and subsequently produce realistic sensory observations within this virtual environment. Our experiments indicate that our approach outperforms competing algorithms in realism, temporal coherency, and layout consistency. We additionally show that LidarDM can be used as a generative world model simulator for training and testing perception models.

Title: Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

• Authors: Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02905
• Pdf URL: https://arxiv.org/pdf/2404.02905
• Copy Paste: [[2404.02905]] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction(https://arxiv.org/abs/2404.02905)
• Keywords: diffusion
• Abstract: We present Visual AutoRegressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine "next-scale prediction" or "next-resolution prediction", diverging from the standard raster-scan "next-token prediction". This simple, intuitive methodology allows autoregressive (AR) transformers to learn visual distributions fast and generalize well: VAR, for the first time, makes AR models surpass diffusion transformers in image generation. On ImageNet 256x256 benchmark, VAR significantly improve AR baseline by improving Frechet inception distance (FID) from 18.65 to 1.80, inception score (IS) from 80.4 to 356.4, with around 20x faster inference speed. It is also empirically verified that VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions including image quality, inference speed, data efficiency, and scalability. Scaling up VAR models exhibits clear power-law scaling laws similar to those observed in LLMs, with linear correlation coefficients near -0.998 as solid evidence. VAR further showcases zero-shot generalization ability in downstream tasks including image in-painting, out-painting, and editing. These results suggest VAR has initially emulated the two important properties of LLMs: Scaling Laws and zero-shot task generalization. We have released all models and codes to promote the exploration of AR/VAR models for visual generation and unified learning.