2024-04-05

Title: An Unsupervised Adversarial Autoencoder for Cyber Attack Detection in Power Distribution Grids

• Authors: Mehdi Jabbari Zideh, Mohammad Reza Khalghani, Sarika Khushalani Solanki
• Subjects: cs.CR, cs.AI, cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2404.02923
• Pdf URL: https://arxiv.org/pdf/2404.02923
• Copy Paste: [[2404.02923]] An Unsupervised Adversarial Autoencoder for Cyber Attack Detection in Power Distribution Grids(https://arxiv.org/abs/2404.02923)
• Keywords: generative
• Abstract: Detection of cyber attacks in smart power distribution grids with unbalanced configurations poses challenges due to the inherent nonlinear nature of these uncertain and stochastic systems. It originates from the intermittent characteristics of the distributed energy resources (DERs) generation and load variations. Moreover, the unknown behavior of cyber attacks, especially false data injection attacks (FDIAs) in the distribution grids with complex temporal correlations and the limited amount of labeled data increases the vulnerability of the grids and imposes a high risk in the secure and reliable operation of the grids. To address these challenges, this paper proposes an unsupervised adversarial autoencoder (AAE) model to detect FDIAs in unbalanced power distribution grids integrated with DERs, i.e., PV systems and wind generation. The proposed method utilizes long short-term memory (LSTM) in the structure of the autoencoder to capture the temporal dependencies in the time-series measurements and leverages the power of generative adversarial networks (GANs) for better reconstruction of the input data. The advantage of the proposed data-driven model is that it can detect anomalous points for the system operation without reliance on abstract models or mathematical representations. To evaluate the efficacy of the approach, it is tested on IEEE 13-bus and 123-bus systems with historical meteorological data (wind speed, ambient temperature, and solar irradiance) as well as historical real-world load data under three types of data falsification functions. The comparison of the detection results of the proposed model with other unsupervised learning methods verifies its superior performance in detecting cyber attacks in unbalanced power distribution grids.

Title: Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models

• Authors: Jiachen Ma, Anda Cao, Zhiqing Xiao, Jie Zhang, Chao Ye, Junbo Zhao
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02928
• Pdf URL: https://arxiv.org/pdf/2404.02928
• Copy Paste: [[2404.02928]] Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models(https://arxiv.org/abs/2404.02928)
• Keywords: diffusion
• Abstract: The fast advance of the image generation community has attracted attention worldwide. The safety issue needs to be further scrutinized and studied. There have been a few works around this area mostly achieving a post-processing design, model-specific, or yielding suboptimal image quality generation. Despite that, in this article, we discover a black-box attack method that enjoys three merits. It enables (i)-attacks both directed and semantic-driven that theoretically and practically pose a hazard to this vast user community, (ii)-surprisingly surpasses the white-box attack in a black-box manner and (iii)-without requiring any post-processing effort. Core to our approach is inspired by the concept guidance intriguing property of Classifier-Free guidance (CFG) in T2I models, and we discover that conducting frustratingly simple guidance in the CLIP embedding space, coupled with the semantic loss and an additionally sensitive word list works very well. Moreover, our results expose and highlight the vulnerabilities in existing defense mechanisms.

Title: Using Large Language Models to Understand Telecom Standards

• Authors: Athanasios Karapantelakis, Mukesh Shakur, Alexandros Nikou, Farnaz Moradi, Christian Orlog, Fitsum Gaim, Henrik Holm, Doumitrou Daniil Nimara, Vincent Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02929
• Pdf URL: https://arxiv.org/pdf/2404.02929
• Copy Paste: [[2404.02929]] Using Large Language Models to Understand Telecom Standards(https://arxiv.org/abs/2404.02929)
• Keywords: generative
• Abstract: The Third Generation Partnership Project (3GPP) has successfully introduced standards for global mobility. However, the volume and complexity of these standards has increased over time, thus complicating access to relevant information for vendors and service providers. Use of Generative Artificial Intelligence (AI) and in particular Large Language Models (LLMs), may provide faster access to relevant information. In this paper, we evaluate the capability of state-of-art LLMs to be used as Question Answering (QA) assistants for 3GPP document reference. Our contribution is threefold. First, we provide a benchmark and measuring methods for evaluating performance of LLMs. Second, we do data preprocessing and fine-tuning for one of these LLMs and provide guidelines to increase accuracy of the responses that apply to all LLMs. Third, we provide a model of our own, TeleRoBERTa, that performs on-par with foundation LLMs but with an order of magnitude less number of parameters. Results show that LLMs can be used as a credible reference tool on telecom technical documents, and thus have potential for a number of different applications from troubleshooting and maintenance, to network operations and software product development.

Title: Explainable Traffic Flow Prediction with Large Language Models

• Authors: Xusen Guo, Qiming Zhang, Mingxing Peng, Meixin Zhua, Hao (Frank)Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.02937
• Pdf URL: https://arxiv.org/pdf/2404.02937
• Copy Paste: [[2404.02937]] Explainable Traffic Flow Prediction with Large Language Models(https://arxiv.org/abs/2404.02937)
• Keywords: foundation model
• Abstract: Traffic flow prediction provides essential future views in the intelligent transportation system. Explainable predictions offer valuable insights into the factors influencing traffic patterns, which help urban planners, traffic engineers, and policymakers make informed decisions about infrastructure development, traffic management strategies, and public transportation planning. Despite their widespread popularity and commendable accuracy, prediction methods grounded in deep learning frequently disappoint in terms of transparency and interpretability. Recently, the availability of large-scale spatio-temporal data and the development of large language models (LLMs) have opened up new opportunities for urban traffic prediction. With the popularity of LLMs, people witnessed the potential reasoning and generating ability of foundation models in various tasks. Considering text as input and output, LLMs have advantages in generating more intuitive and interpretable predictions. Hence, this work introduces TP-LLM, an explainable foundation-model-based method for traffic prediction, aiming at more direct and reasonable forecasting. TP-LLM presents a framework to unify multi-modality factors as language-based inputs, TP-LLM avoids complex spatial-temporal data programming and outperforms state-of-art baselines merely under fine-tuning foundation models. Also, TP-LLM can generate input-dependency explanations for more confident prediction and can be easily generalized to different city dynamics for zero-shot prediction with a similar framework. These findings demonstrate the potential of LLMs for explainable traffic prediction.

Title: Foundation Models for Structural Health Monitoring

• Authors: Luca Benfenati, Daniele Jahier Pagliari, Luca Zanatta, Yhorman Alexander Bedoya Velez, Andrea Acquaviva, Massimo Poncino, Enrico Macii, Luca Benini, Alessio Burrello
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2404.02944
• Pdf URL: https://arxiv.org/pdf/2404.02944
• Copy Paste: [[2404.02944]] Foundation Models for Structural Health Monitoring(https://arxiv.org/abs/2404.02944)
• Keywords: self-supervised, foundation model, anomaly
• Abstract: Structural Health Monitoring (SHM) is a critical task for ensuring the safety and reliability of civil infrastructures, typically realized on bridges and viaducts by means of vibration monitoring. In this paper, we propose for the first time the use of Transformer neural networks, with a Masked Auto-Encoder architecture, as Foundation Models for SHM. We demonstrate the ability of these models to learn generalizable representations from multiple large datasets through self-supervised pre-training, which, coupled with task-specific fine-tuning, allows them to outperform state-of-the-art traditional methods on diverse tasks, including Anomaly Detection (AD) and Traffic Load Estimation (TLE). We then extensively explore model size versus accuracy trade-offs and experiment with Knowledge Distillation (KD) to improve the performance of smaller Transformers, enabling their embedding directly into the SHM edge nodes. We showcase the effectiveness of our foundation models using data from three operational viaducts. For AD, we achieve a near-perfect 99.9% accuracy with a monitoring time span of just 15 windows. In contrast, a state-of-the-art method based on Principal Component Analysis (PCA) obtains its first good result (95.03% accuracy) only considering 120 windows. On two different TLE tasks, our models obtain state-of-the-art performance on multiple evaluation metrics (R$^2$ score, MAE% and MSE%). On the first benchmark, we achieve an R$^2$ score of 0.97 and 0.85 for light and heavy vehicle traffic, respectively, while the best previous approach stops at 0.91 and 0.84. On the second one, we achieve an R$^2$ score of 0.54 versus the 0.10 of the best existing method.

Title: Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections

• Authors: Gabriel Loaiza-Ganem, Brendan Leigh Ross, Rasa Hosseinzadeh, Anthony L. Caterini, Jesse C. Cresswell
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.02954
• Pdf URL: https://arxiv.org/pdf/2404.02954
• Copy Paste: [[2404.02954]] Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections(https://arxiv.org/abs/2404.02954)
• Keywords: diffusion, generative
• Abstract: In recent years there has been increased interest in understanding the interplay between deep generative models (DGMs) and the manifold hypothesis. Research in this area focuses on understanding the reasons why commonly-used DGMs succeed or fail at learning distributions supported on unknown low-dimensional manifolds, as well as developing new models explicitly designed to account for manifold-supported data. This manifold lens provides both clarity as to why some DGMs (e.g. diffusion models and some generative adversarial networks) empirically surpass others (e.g. likelihood-based models such as variational autoencoders, normalizing flows, or energy-based models) at sample generation, and guidance for devising more performant DGMs. We carry out the first survey of DGMs viewed through this lens, making two novel contributions along the way. First, we formally establish that numerical instability of high-dimensional likelihoods is unavoidable when modelling low-dimensional data. We then show that DGMs on learned representations of autoencoders can be interpreted as approximately minimizing Wasserstein distance: this result, which applies to latent diffusion models, helps justify their outstanding empirical results. The manifold lens provides a rich perspective from which to understand DGMs, which we aim to make more accessible and widespread.

Title: ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scale

• Authors: Jinbin Huang, Chen Chen, Aditi Mishra, Bum Chul Kwon, Zhicheng Liu, Chris Bryan
• Subjects: cs.CV, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.02990
• Pdf URL: https://arxiv.org/pdf/2404.02990
• Copy Paste: [[2404.02990]] ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scale(https://arxiv.org/abs/2404.02990)
• Keywords: diffusion, generative
• Abstract: Generative image models have emerged as a promising technology to produce realistic images. Despite potential benefits, concerns grow about its misuse, particularly in generating deceptive images that could raise significant ethical, legal, and societal issues. Consequently, there is growing demand to empower users to effectively discern and comprehend patterns of AI-generated images. To this end, we developed ASAP, an interactive visualization system that automatically extracts distinct patterns of AI-generated images and allows users to interactively explore them via various views. To uncover fake patterns, ASAP introduces a novel image encoder, adapted from CLIP, which transforms images into compact "distilled" representations, enriched with information for differentiating authentic and fake images. These representations generate gradients that propagate back to the attention maps of CLIP's transformer block. This process quantifies the relative importance of each pixel to image authenticity or fakeness, exposing key deceptive patterns. ASAP enables the at scale interactive analysis of these patterns through multiple, coordinated visualizations. This includes a representation overview with innovative cell glyphs to aid in the exploration and qualitative evaluation of fake patterns across a vast array of images, as well as a pattern view that displays authenticity-indicating patterns in images and quantifies their impact. ASAP supports the analysis of cutting-edge generative models with the latest architectures, including GAN-based models like proGAN and diffusion models like the latent diffusion model. We demonstrate ASAP's usefulness through two usage scenarios using multiple fake image detection benchmark datasets, revealing its ability to identify and understand hidden patterns in AI-generated images, especially in detecting fake human faces produced by diffusion-based techniques.

Title: Transfer learning applications for anomaly detection in wind turbines

• Authors: Cyriana M.A. Roelofs, Christian Gück, Stefan Faulstich
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03011
• Pdf URL: https://arxiv.org/pdf/2404.03011
• Copy Paste: [[2404.03011]] Transfer learning applications for anomaly detection in wind turbines(https://arxiv.org/abs/2404.03011)
• Keywords: anomaly
• Abstract: Anomaly detection in wind turbines typically involves using normal behaviour models to detect faults early. However, training autoencoder models for each turbine is time-consuming and resource intensive. Thus, transfer learning becomes essential for wind turbines with limited data or applications with limited computational resources. This study examines how cross-turbine transfer learning can be applied to autoencoder-based anomaly detection. Here, autoencoders are combined with constant thresholds for the reconstruction error to determine if input data contains an anomaly. The models are initially trained on one year's worth of data from one or more source wind turbines. They are then fine-tuned using smaller amounts of data from another turbine. Three methods for fine-tuning are investigated: adjusting the entire autoencoder, only the decoder, or only the threshold of the model. The performance of the transfer learning models is compared to baseline models that were trained on one year's worth of data from the target wind turbine. The results of the tests conducted in this study indicate that models trained on data of multiple wind turbines do not improve the anomaly detection capability compared to models trained on data of one source wind turbine. In addition, modifying the model's threshold can lead to comparable or even superior performance compared to the baseline, whereas fine-tuning the decoder or autoencoder further enhances the models' performance.

Title: Blessing or curse? A survey on the Impact of Generative AI on Fake News

• Authors: Alexander Loth, Martin Kappes, Marc-Oliver Pahl
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03021
• Pdf URL: https://arxiv.org/pdf/2404.03021
• Copy Paste: [[2404.03021]] Blessing or curse? A survey on the Impact of Generative AI on Fake News(https://arxiv.org/abs/2404.03021)
• Keywords: generative
• Abstract: Fake news significantly influence our society. They impact consumers, voters, and many other societal groups. While Fake News exist for a centuries, Generative AI brings fake news on a new level. It is now possible to automate the creation of masses of high-quality individually targeted Fake News. On the other end, Generative AI can also help detecting Fake News. Both fields are young but developing fast. This survey provides a comprehensive examination of the research and practical use of Generative AI for Fake News detection and creation in 2024. Following the Structured Literature Survey approach, the paper synthesizes current results in the following topic clusters 1) enabling technologies, 2) creation of Fake News, 3) case study social media as most relevant distribution channel, 4) detection of Fake News, and 5) deepfakes as upcoming technology. The article also identifies current challenges and open issues.

Title: An Incomplete Loop: Deductive, Inductive, and Abductive Learning in Large Language Models

• Authors: Emmy Liu, Graham Neubig, Jacob Andreas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03028
• Pdf URL: https://arxiv.org/pdf/2404.03028
• Copy Paste: [[2404.03028]] An Incomplete Loop: Deductive, Inductive, and Abductive Learning in Large Language Models(https://arxiv.org/abs/2404.03028)
• Keywords: in-context
• Abstract: Modern language models (LMs) can learn to perform new tasks in different ways: in instruction following, the target task is described explicitly in natural language; in few-shot prompting, the task is specified implicitly with a small number of examples; in instruction inference, LMs are presented with in-context examples and are then prompted to generate a natural language task description before making predictions. Each of these procedures may be thought of as invoking a different form of reasoning: instruction following involves deductive reasoning, few-shot prompting involves inductive reasoning, and instruction inference involves abductive reasoning. How do these different capabilities relate? Across four LMs (from the gpt and llama families) and two learning problems (involving arithmetic functions and machine translation) we find a strong dissociation between the different types of reasoning: LMs can sometimes learn effectively from few-shot prompts even when they are unable to explain their own prediction rules; conversely, they sometimes infer useful task descriptions while completely failing to learn from human-generated descriptions of the same task. Our results highlight the non-systematic nature of reasoning even in some of today's largest LMs, and underscore the fact that very different learning mechanisms may be invoked by seemingly similar prompting procedures.

Title: GPT-DETOX: An In-Context Learning-Based Paraphraser for Text Detoxification

• Authors: Ali Pesaranghader, Nikhil Verma, Manasa Bharadwaj
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03052
• Pdf URL: https://arxiv.org/pdf/2404.03052
• Copy Paste: [[2404.03052]] GPT-DETOX: An In-Context Learning-Based Paraphraser for Text Detoxification(https://arxiv.org/abs/2404.03052)
• Keywords: in-context
• Abstract: Harmful and offensive communication or content is detrimental to social bonding and the mental state of users on social media platforms. Text detoxification is a crucial task in natural language processing (NLP), where the goal is removing profanity and toxicity from text while preserving its content. Supervised and unsupervised learning are common approaches for designing text detoxification solutions. However, these methods necessitate fine-tuning, leading to computational overhead. In this paper, we propose GPT-DETOX as a framework for prompt-based in-context learning for text detoxification using GPT-3.5 Turbo. We utilize zero-shot and few-shot prompting techniques for detoxifying input sentences. To generate few-shot prompts, we propose two methods: word-matching example selection (WMES) and context-matching example selection (CMES). We additionally take into account ensemble in-context learning (EICL) where the ensemble is shaped by base prompts from zero-shot and all few-shot settings. We use ParaDetox and APPDIA as benchmark detoxification datasets. Our experimental results show that the zero-shot solution achieves promising performance, while our best few-shot setting outperforms the state-of-the-art models on ParaDetox and shows comparable results on APPDIA. Our EICL solutions obtain the greatest performance, adding at least 10% improvement, against both datasets.

Title: Mai Ho'omāuna i ka 'Ai: Language Models Improve Automatic Speech Recognition in Hawaiian

• Authors: Kaavya Chaparala, Guido Zarrella, Bruce Torres Fischer, Larry Kimura, Oiwi Parker Jones
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.03073
• Pdf URL: https://arxiv.org/pdf/2404.03073
• Copy Paste: [[2404.03073]] Mai Ho'omāuna i ka 'Ai: Language Models Improve Automatic Speech Recognition in Hawaiian(https://arxiv.org/abs/2404.03073)
• Keywords: foundation model
• Abstract: In this paper we address the challenge of improving Automatic Speech Recognition (ASR) for a low-resource language, Hawaiian, by incorporating large amounts of independent text data into an ASR foundation model, Whisper. To do this, we train an external language model (LM) on ~1.5M words of Hawaiian text. We then use the LM to rescore Whisper and compute word error rates (WERs) on a manually curated test set of labeled Hawaiian data. As a baseline, we use Whisper without an external LM. Experimental results reveal a small but significant improvement in WER when ASR outputs are rescored with a Hawaiian LM. The results support leveraging all available data in the development of ASR systems for underrepresented languages.

Title: SalFoM: Dynamic Saliency Prediction with Video Foundation Models

• Authors: Morteza Moradi, Mohammad Moradi, Francesco Rundo, Concetto Spampinato, Ali Borji, Simone Palazzo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03097
• Pdf URL: https://arxiv.org/pdf/2404.03097
• Copy Paste: [[2404.03097]] SalFoM: Dynamic Saliency Prediction with Video Foundation Models(https://arxiv.org/abs/2404.03097)
• Keywords: foundation model
• Abstract: Recent advancements in video saliency prediction (VSP) have shown promising performance compared to the human visual system, whose emulation is the primary goal of VSP. However, current state-of-the-art models employ spatio-temporal transformers trained on limited amounts of data, hindering generalizability adaptation to downstream tasks. The benefits of vision foundation models present a potential solution to improve the VSP process. However, adapting image foundation models to the video domain presents significant challenges in modeling scene dynamics and capturing temporal information. To address these challenges, and as the first initiative to design a VSP model based on video foundation models, we introduce SalFoM, a novel encoder-decoder video transformer architecture. Our model employs UnMasked Teacher (UMT) as feature extractor and presents a heterogeneous decoder which features a locality-aware spatio-temporal transformer and integrates local and global spatio-temporal information from various perspectives to produce the final saliency map. Our qualitative and quantitative experiments on the challenging VSP benchmark datasets of DHF1K, Hollywood-2 and UCF-Sports demonstrate the superiority of our proposed model in comparison with the state-of-the-art methods.

Title: Many-to-many Image Generation with Auto-regressive Diffusion Models

• Authors: Ying Shen, Yizhe Zhang, Shuangfei Zhai, Lifu Huang, Joshua M. Susskind, Jiatao Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03109
• Pdf URL: https://arxiv.org/pdf/2404.03109
• Copy Paste: [[2404.03109]] Many-to-many Image Generation with Auto-regressive Diffusion Models(https://arxiv.org/abs/2404.03109)
• Keywords: diffusion
• Abstract: Recent advancements in image generation have made significant progress, yet existing models present limitations in perceiving and generating an arbitrary number of interrelated images within a broad context. This limitation becomes increasingly critical as the demand for multi-image scenarios, such as multi-view images and visual narratives, grows with the expansion of multimedia platforms. This paper introduces a domain-general framework for many-to-many image generation, capable of producing interrelated image series from a given set of images, offering a scalable solution that obviates the need for task-specific solutions across different multi-image scenarios. To facilitate this, we present MIS, a novel large-scale multi-image dataset, containing 12M synthetic multi-image samples, each with 25 interconnected images. Utilizing Stable Diffusion with varied latent noises, our method produces a set of interconnected images from a single caption. Leveraging MIS, we learn M2M, an autoregressive model for many-to-many generation, where each image is modeled within a diffusion framework. Throughout training on the synthetic MIS, the model excels in capturing style and content from preceding images - synthetic or real - and generates novel images following the captured patterns. Furthermore, through task-specific fine-tuning, our model demonstrates its adaptability to various multi-image generation tasks, including Novel View Synthesis and Visual Procedure Generation.

Title: Diverse and Tailored Image Generation for Zero-shot Multi-label Classification

• Authors: Kaixin Zhang, Zhixiang Yuan, Tao Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03144
• Pdf URL: https://arxiv.org/pdf/2404.03144
• Copy Paste: [[2404.03144]] Diverse and Tailored Image Generation for Zero-shot Multi-label Classification(https://arxiv.org/abs/2404.03144)
• Keywords: diffusion
• Abstract: Recently, zero-shot multi-label classification has garnered considerable attention for its capacity to operate predictions on unseen labels without human annotations. Nevertheless, prevailing approaches often use seen classes as imperfect proxies for unseen ones, resulting in suboptimal performance. Drawing inspiration from the success of text-to-image generation models in producing realistic images, we propose an innovative solution: generating synthetic data to construct a training set explicitly tailored for proxyless training on unseen labels. Our approach introduces a novel image generation framework that produces multi-label synthetic images of unseen classes for classifier training. To enhance diversity in the generated images, we leverage a pre-trained large language model to generate diverse prompts. Employing a pre-trained multi-modal CLIP model as a discriminator, we assess whether the generated images accurately represent the target classes. This enables automatic filtering of inaccurately generated images, preserving classifier accuracy. To refine text prompts for more precise and effective multi-label object generation, we introduce a CLIP score-based discriminative loss to fine-tune the text encoder in the diffusion model. Additionally, to enhance visual features on the target task while maintaining the generalization of original features and mitigating catastrophic forgetting resulting from fine-tuning the entire visual encoder, we propose a feature fusion module inspired by transformer attention mechanisms. This module aids in capturing global dependencies between multiple objects more effectively. Extensive experimental results validate the effectiveness of our approach, demonstrating significant improvements over state-of-the-art methods.

Title: DreamWalk: Style Space Exploration using Diffusion Guidance

• Authors: Michelle Shu, Charles Herrmann, Richard Strong Bowen, Forrester Cole, Ramin Zabih
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03145
• Pdf URL: https://arxiv.org/pdf/2404.03145
• Copy Paste: [[2404.03145]] DreamWalk: Style Space Exploration using Diffusion Guidance(https://arxiv.org/abs/2404.03145)
• Keywords: diffusion
• Abstract: Text-conditioned diffusion models can generate impressive images, but fall short when it comes to fine-grained control. Unlike direct-editing tools like Photoshop, text conditioned models require the artist to perform "prompt engineering," constructing special text sentences to control the style or amount of a particular subject present in the output image. Our goal is to provide fine-grained control over the style and substance specified by the prompt, for example to adjust the intensity of styles in different regions of the image (Figure 1). Our approach is to decompose the text prompt into conceptual elements, and apply a separate guidance term for each element in a single diffusion process. We introduce guidance scale functions to control when in the diffusion process and \emph{where} in the image to intervene. Since the method is based solely on adjusting diffusion guidance, it does not require fine-tuning or manipulating the internal layers of the diffusion model's neural network, and can be used in conjunction with LoRA- or DreamBooth-trained models (Figure2). Project page: https://mshu1.github.io/dreamwalk.github.io/

Title: HandDiff: 3D Hand Pose Estimation with Diffusion on Image-Point Cloud

• Authors: Wencan Cheng, Hao Tang, Luc Van Gool, Jong Hwan Ko
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03159
• Pdf URL: https://arxiv.org/pdf/2404.03159
• Copy Paste: [[2404.03159]] HandDiff: 3D Hand Pose Estimation with Diffusion on Image-Point Cloud(https://arxiv.org/abs/2404.03159)
• Keywords: diffusion, generative
• Abstract: Extracting keypoint locations from input hand frames, known as 3D hand pose estimation, is a critical task in various human-computer interaction applications. Essentially, the 3D hand pose estimation can be regarded as a 3D point subset generative problem conditioned on input frames. Thanks to the recent significant progress on diffusion-based generative models, hand pose estimation can also benefit from the diffusion model to estimate keypoint locations with high quality. However, directly deploying the existing diffusion models to solve hand pose estimation is non-trivial, since they cannot achieve the complex permutation mapping and precise localization. Based on this motivation, this paper proposes HandDiff, a diffusion-based hand pose estimation model that iteratively denoises accurate hand pose conditioned on hand-shaped image-point clouds. In order to recover keypoint permutation and accurate location, we further introduce joint-wise condition and local detail condition. Experimental results demonstrate that the proposed HandDiff significantly outperforms the existing approaches on four challenging hand pose benchmark datasets. Codes and pre-trained models are publicly available at https://github.com/cwc1260/HandDiff.

Title: Adaptive Discrete Disparity Volume for Self-supervised Monocular Depth Estimation

• Authors: Jianwei Ren
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.03190
• Pdf URL: https://arxiv.org/pdf/2404.03190
• Copy Paste: [[2404.03190]] Adaptive Discrete Disparity Volume for Self-supervised Monocular Depth Estimation(https://arxiv.org/abs/2404.03190)
• Keywords: self-supervised
• Abstract: In self-supervised monocular depth estimation tasks, discrete disparity prediction has been proven to attain higher quality depth maps than common continuous methods. However, current discretization strategies often divide depth ranges of scenes into bins in a handcrafted and rigid manner, limiting model performance. In this paper, we propose a learnable module, Adaptive Discrete Disparity Volume (ADDV), which is capable of dynamically sensing depth distributions in different RGB images and generating adaptive bins for them. Without any extra supervision, this module can be integrated into existing CNN architectures, allowing networks to produce representative values for bins and a probability volume over them. Furthermore, we introduce novel training strategies - uniformizing and sharpening - through a loss term and temperature parameter, respectively, to provide regularizations under self-supervised conditions, preventing model degradation or collapse. Empirical results demonstrate that ADDV effectively processes global information, generating appropriate bins for various scenes and producing higher quality depth maps compared to handcrafted methods.

Title: Future-Proofing Class Incremental Learning

• Authors: Quentin Jodelet, Xin Liu, Yin Jun Phua, Tsuyoshi Murata
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03200
• Pdf URL: https://arxiv.org/pdf/2404.03200
• Copy Paste: [[2404.03200]] Future-Proofing Class Incremental Learning(https://arxiv.org/abs/2404.03200)
• Keywords: diffusion
• Abstract: Exemplar-Free Class Incremental Learning is a highly challenging setting where replay memory is unavailable. Methods relying on frozen feature extractors have drawn attention recently in this setting due to their impressive performances and lower computational costs. However, those methods are highly dependent on the data used to train the feature extractor and may struggle when an insufficient amount of classes are available during the first incremental step. To overcome this limitation, we propose to use a pre-trained text-to-image diffusion model in order to generate synthetic images of future classes and use them to train the feature extractor. Experiments on the standard benchmarks CIFAR100 and ImageNet-Subset demonstrate that our proposed method can be used to improve state-of-the-art methods for exemplar-free class incremental learning, especially in the most difficult settings where the first incremental step only contains few classes. Moreover, we show that using synthetic samples of future classes achieves higher performance than using real data from different classes, paving the way for better and less costly pre-training methods for incremental learning.

Title: HDR Imaging for Dynamic Scenes with Events

• Authors: Li Xiaopeng, Zeng Zhaoyuan, Fan Cien, Zhao Chen, Deng Lei, Yu Lei
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.03210
• Pdf URL: https://arxiv.org/pdf/2404.03210
• Copy Paste: [[2404.03210]] HDR Imaging for Dynamic Scenes with Events(https://arxiv.org/abs/2404.03210)
• Keywords: self-supervised
• Abstract: High dynamic range imaging (HDRI) for real-world dynamic scenes is challenging because moving objects may lead to hybrid degradation of low dynamic range and motion blur. Existing event-based approaches only focus on a separate task, while cascading HDRI and motion deblurring would lead to sub-optimal solutions, and unavailable ground-truth sharp HDR images aggravate the predicament. To address these challenges, we propose an Event-based HDRI framework within a Self-supervised learning paradigm, i.e., Self-EHDRI, which generalizes HDRI performance in real-world dynamic scenarios. Specifically, a self-supervised learning strategy is carried out by learning cross-domain conversions from blurry LDR images to sharp LDR images, which enables sharp HDR images to be accessible in the intermediate process even though ground-truth sharp HDR images are missing. Then, we formulate the event-based HDRI and motion deblurring model and conduct a unified network to recover the intermediate sharp HDR results, where both the high dynamic range and high temporal resolution of events are leveraged simultaneously for compensation. We construct large-scale synthetic and real-world datasets to evaluate the effectiveness of our method. Comprehensive experiments demonstrate that the proposed Self-EHDRI outperforms state-of-the-art approaches by a large margin. The codes, datasets, and results are available at https://lxp-whu.github.io/Self-EHDRI.

Title: iSeg: Interactive 3D Segmentation via Interactive Attention

• Authors: Itai Lang, Fei Xu, Dale Decatur, Sudarshan Babu, Rana Hanocka
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.03219
• Pdf URL: https://arxiv.org/pdf/2404.03219
• Copy Paste: [[2404.03219]] iSeg: Interactive 3D Segmentation via Interactive Attention(https://arxiv.org/abs/2404.03219)
• Keywords: foundation model
• Abstract: We present iSeg, a new interactive technique for segmenting 3D shapes. Previous works have focused mainly on leveraging pre-trained 2D foundation models for 3D segmentation based on text. However, text may be insufficient for accurately describing fine-grained spatial segmentations. Moreover, achieving a consistent 3D segmentation using a 2D model is challenging since occluded areas of the same semantic region may not be visible together from any 2D view. Thus, we design a segmentation method conditioned on fine user clicks, which operates entirely in 3D. Our system accepts user clicks directly on the shape's surface, indicating the inclusion or exclusion of regions from the desired shape partition. To accommodate various click settings, we propose a novel interactive attention module capable of processing different numbers and types of clicks, enabling the training of a single unified interactive segmentation model. We apply iSeg to a myriad of shapes from different domains, demonstrating its versatility and faithfulness to the user's specifications. Our project page is at https://threedle.github.io/iSeg/.

Title: Would Deep Generative Models Amplify Bias in Future Models?

• Authors: Tianwei Chen, Yusuke Hirota, Mayu Otani, Noa Garcia, Yuta Nakashima
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03242
• Pdf URL: https://arxiv.org/pdf/2404.03242
• Copy Paste: [[2404.03242]] Would Deep Generative Models Amplify Bias in Future Models?(https://arxiv.org/abs/2404.03242)
• Keywords: diffusion, generative
• Abstract: We investigate the impact of deep generative models on potential social biases in upcoming computer vision models. As the internet witnesses an increasing influx of AI-generated images, concerns arise regarding inherent biases that may accompany them, potentially leading to the dissemination of harmful content. This paper explores whether a detrimental feedback loop, resulting in bias amplification, would occur if generated images were used as the training data for future models. We conduct simulations by progressively substituting original images in COCO and CC3M datasets with images generated through Stable Diffusion. The modified datasets are used to train OpenCLIP and image captioning models, which we evaluate in terms of quality and bias. Contrary to expectations, our findings indicate that introducing generated images during training does not uniformly amplify bias. Instead, instances of bias mitigation across specific tasks are observed. We further explore the factors that may influence these phenomena, such as artifacts in image generation (e.g., blurry faces) or pre-existing biases in the original datasets.

Title: On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models

• Authors: Sean Farhat, Deming Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03263
• Pdf URL: https://arxiv.org/pdf/2404.03263
• Copy Paste: [[2404.03263]] On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models(https://arxiv.org/abs/2404.03263)
• Keywords: generative
• Abstract: In this paper, we propose that small models may not need to absorb the cost of pre-training to reap its benefits. Instead, they can capitalize on the astonishing results achieved by modern, enormous models to a surprising degree. We observe that, when distilled on a task from a pre-trained teacher model, a small model can achieve or surpass the performance it would achieve if it was pre-trained then finetuned on that task. To allow this phenomenon to be easily leveraged, we establish a connection reducing knowledge distillation to modern contrastive learning, opening two doors: (1) vastly different model architecture pairings can work for the distillation, and (2) most contrastive learning algorithms rooted in the theory of Noise Contrastive Estimation can be easily applied and used. We demonstrate this paradigm using pre-trained teacher models from open-source model hubs, Transformer and convolution based model combinations, and a novel distillation algorithm that massages the Alignment/Uniformity perspective of contrastive learning by Wang & Isola (2020) into a distillation objective. We choose this flavor of contrastive learning due to its low computational cost, an overarching theme of this work. We also observe that this phenomenon tends not to occur if the task is data-limited. However, this can be alleviated by leveraging yet another scale-inspired development: large, pre-trained generative models for dataset augmentation. Again, we use an open-source model, and our rudimentary prompts are sufficient to boost the small model`s performance. Thus, we highlight a training method for small models that is up to 94% faster than the standard pre-training paradigm without sacrificing performance. For practitioners discouraged from fully utilizing modern foundation datasets for their small models due to the prohibitive scale, we believe our work keeps that door open.

Title: SiloFuse: Cross-silo Synthetic Data Generation with Latent Tabular Diffusion Models

• Authors: Aditya Shankar, Hans Brouwer, Rihan Hai, Lydia Chen
• Subjects: cs.LG, cs.CR, cs.DB, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.03299
• Pdf URL: https://arxiv.org/pdf/2404.03299
• Copy Paste: [[2404.03299]] SiloFuse: Cross-silo Synthetic Data Generation with Latent Tabular Diffusion Models(https://arxiv.org/abs/2404.03299)
• Keywords: diffusion, generative
• Abstract: Synthetic tabular data is crucial for sharing and augmenting data across silos, especially for enterprises with proprietary data. However, existing synthesizers are designed for centrally stored data. Hence, they struggle with real-world scenarios where features are distributed across multiple silos, necessitating on-premise data storage. We introduce SiloFuse, a novel generative framework for high-quality synthesis from cross-silo tabular data. To ensure privacy, SiloFuse utilizes a distributed latent tabular diffusion architecture. Through autoencoders, latent representations are learned for each client's features, masking their actual values. We employ stacked distributed training to improve communication efficiency, reducing the number of rounds to a single step. Under SiloFuse, we prove the impossibility of data reconstruction for vertically partitioned synthesis and quantify privacy risks through three attacks using our benchmark framework. Experimental results on nine datasets showcase SiloFuse's competence against centralized diffusion-based synthesizers. Notably, SiloFuse achieves 43.8 and 29.8 higher percentage points over GANs in resemblance and utility. Experiments on communication show stacked training's fixed cost compared to the growing costs of end-to-end training as the number of training iterations increases. Additionally, SiloFuse proves robust to feature permutations and varying numbers of clients.

Title: Knowledge Distillation-Based Model Extraction Attack using Private Counterfactual Explanations

• Authors: Fatima Ezzeddine, Omran Ayoub, Silvia Giordano
• Subjects: cs.LG, cs.AI, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.03348
• Pdf URL: https://arxiv.org/pdf/2404.03348
• Copy Paste: [[2404.03348]] Knowledge Distillation-Based Model Extraction Attack using Private Counterfactual Explanations(https://arxiv.org/abs/2404.03348)
• Keywords: generative
• Abstract: In recent years, there has been a notable increase in the deployment of machine learning (ML) models as services (MLaaS) across diverse production software applications. In parallel, explainable AI (XAI) continues to evolve, addressing the necessity for transparency and trustworthiness in ML models. XAI techniques aim to enhance the transparency of ML models by providing insights, in terms of the model's explanations, into their decision-making process. Simultaneously, some MLaaS platforms now offer explanations alongside the ML prediction outputs. This setup has elevated concerns regarding vulnerabilities in MLaaS, particularly in relation to privacy leakage attacks such as model extraction attacks (MEA). This is due to the fact that explanations can unveil insights about the inner workings of the model which could be exploited by malicious users. In this work, we focus on investigating how model explanations, particularly Generative adversarial networks (GANs)-based counterfactual explanations (CFs), can be exploited for performing MEA within the MLaaS platform. We also delve into assessing the effectiveness of incorporating differential privacy (DP) as a mitigation strategy. To this end, we first propose a novel MEA methodology based on Knowledge Distillation (KD) to enhance the efficiency of extracting a substitute model of a target model exploiting CFs. Then, we advise an approach for training CF generators incorporating DP to generate private CFs. We conduct thorough experimental evaluations on real-world datasets and demonstrate that our proposed KD-based MEA can yield a high-fidelity substitute model with reduced queries with respect to baseline approaches. Furthermore, our findings reveal that the inclusion of a privacy layer impacts the performance of the explainer, the quality of CFs, and results in a reduction in the MEA performance.

Title: AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessment

• Authors: Chunyi Li, Tengchuan Kou, Yixuan Gao, Yuqin Cao, Wei Sun, Zicheng Zhang, Yingjie Zhou, Zhichao Zhang, Weixia Zhang, Haoning Wu, Xiaohong Liu, Xiongkuo Min, Guangtao Zhai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03407
• Pdf URL: https://arxiv.org/pdf/2404.03407
• Copy Paste: [[2404.03407]] AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessment(https://arxiv.org/abs/2404.03407)
• Keywords: generative
• Abstract: With the rapid advancements in AI-Generated Content (AIGC), AI-Generated Images (AIGIs) have been widely applied in entertainment, education, and social media. However, due to the significant variance in quality among different AIGIs, there is an urgent need for models that consistently match human subjective ratings. To address this issue, we organized a challenge towards AIGC quality assessment on NTIRE 2024 that extensively considers 15 popular generative models, utilizing dynamic hyper-parameters (including classifier-free guidance, iteration epochs, and output image resolution), and gather subjective scores that consider perceptual quality and text-to-image alignment altogether comprehensively involving 21 subjects. This approach culminates in the creation of the largest fine-grained AIGI subjective quality database to date with 20,000 AIGIs and 420,000 subjective ratings, known as AIGIQA-20K. Furthermore, we conduct benchmark experiments on this database to assess the correspondence between 16 mainstream AIGI quality models and human perception. We anticipate that this large-scale quality database will inspire robust quality indicators for AIGIs and propel the evolution of AIGC for vision. The database is released on https://www.modelscope.cn/datasets/lcysyzxdxc/AIGCQA-30K-Image.

Title: Edisum: Summarizing and Explaining Wikipedia Edits at Scale

• Authors: Marija Šakota, Isaac Johnson, Guosheng Feng, Robert West
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03428
• Pdf URL: https://arxiv.org/pdf/2404.03428
• Copy Paste: [[2404.03428]] Edisum: Summarizing and Explaining Wikipedia Edits at Scale(https://arxiv.org/abs/2404.03428)
• Keywords: generative
• Abstract: An edit summary is a succinct comment written by a Wikipedia editor explaining the nature of, and reasons for, an edit to a Wikipedia page. Edit summaries are crucial for maintaining the encyclopedia: they are the first thing seen by content moderators and help them decide whether to accept or reject an edit. Additionally, edit summaries constitute a valuable data source for researchers. Unfortunately, as we show, for many edits, summaries are either missing or incomplete. To overcome this problem and help editors write useful edit summaries, we propose a model for recommending edit summaries generated by a language model trained to produce good edit summaries given the representation of an edit diff. This is a challenging task for multiple reasons, including mixed-quality training data, the need to understand not only what was changed in the article but also why it was changed, and efficiency requirements imposed by the scale of Wikipedia. We address these challenges by curating a mix of human and synthetically generated training data and fine-tuning a generative language model sufficiently small to be used on Wikipedia at scale. Our model performs on par with human editors. Commercial large language models are able to solve this task better than human editors, but would be too expensive to run on Wikipedia at scale. More broadly, this paper showcases how language modeling technology can be used to support humans in maintaining one of the largest and most visible projects on the Web.

Title: Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructions

• Authors: Zhengyuan Liu, Stella Xin Yin, Carolyn Lee, Nancy F. Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03429
• Pdf URL: https://arxiv.org/pdf/2404.03429
• Copy Paste: [[2404.03429]] Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructions(https://arxiv.org/abs/2404.03429)
• Keywords: generative
• Abstract: Intelligent tutoring systems (ITSs) that imitate human tutors and aim to provide immediate and customized instructions or feedback to learners have shown their effectiveness in education. With the emergence of generative artificial intelligence, large language models (LLMs) further entitle the systems to complex and coherent conversational interactions. These systems would be of great help in language education as it involves developing skills in communication, which, however, drew relatively less attention. Additionally, due to the complicated cognitive development at younger ages, more endeavors are needed for practical uses. Scaffolding refers to a teaching technique where teachers provide support and guidance to students for learning and developing new concepts or skills. It is an effective way to support diverse learning needs, goals, processes, and outcomes. In this work, we investigate how pedagogical instructions facilitate the scaffolding in ITSs, by conducting a case study on guiding children to describe images for language learning. We construct different types of scaffolding tutoring systems grounded in four fundamental learning theories: knowledge construction, inquiry-based learning, dialogic teaching, and zone of proximal development. For qualitative and quantitative analyses, we build and refine a seven-dimension rubric to evaluate the scaffolding process. In our experiment on GPT-4V, we observe that LLMs demonstrate strong potential to follow pedagogical instructions and achieve self-paced learning in different student groups. Moreover, we extend our evaluation framework from a manual to an automated approach, paving the way to benchmark various conversational tutoring systems.

Title: Generative AI and Teachers -- For Us or Against Us? A Case Study

• Authors: Jenny Pettersson, Elias Hult, Tim Eriksson, Tosin Adewumi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03486
• Pdf URL: https://arxiv.org/pdf/2404.03486
• Copy Paste: [[2404.03486]] Generative AI and Teachers -- For Us or Against Us? A Case Study(https://arxiv.org/abs/2404.03486)
• Keywords: generative
• Abstract: We present insightful results of a survey on the adoption of generative artificial intelligence (GenAI) by university teachers in their teaching activities. The transformation of education by GenAI, particularly large language models (LLMs), has been presenting both opportunities and challenges, including cheating by students. We prepared the online survey according to best practices and the questions were created by the authors, who have pedagogy experience. The survey contained 12 questions and a pilot study was first conducted. The survey was then sent to all teachers in multiple departments across different campuses of the university of interest in Sweden: Lule{\aa} University of Technology. The survey was available in both Swedish and English. The results show that 35 teachers (more than half) use GenAI out of 67 respondents. Preparation is the teaching activity with the most frequency that GenAI is used for and ChatGPT is the most commonly used GenAI. 59% say it has impacted their teaching, however, 55% say there should be legislation around the use of GenAI, especially as inaccuracies and cheating are the biggest concerns.

Title: HAPNet: Toward Superior RGB-Thermal Scene Parsing via Hybrid, Asymmetric, and Progressive Heterogeneous Feature Fusion

• Authors: Jiahang Li, Peng Yun, Qijun Chen, Rui Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03527
• Pdf URL: https://arxiv.org/pdf/2404.03527
• Copy Paste: [[2404.03527]] HAPNet: Toward Superior RGB-Thermal Scene Parsing via Hybrid, Asymmetric, and Progressive Heterogeneous Feature Fusion(https://arxiv.org/abs/2404.03527)
• Keywords: foundation model
• Abstract: Data-fusion networks have shown significant promise for RGB-thermal scene parsing. However, the majority of existing studies have relied on symmetric duplex encoders for heterogeneous feature extraction and fusion, paying inadequate attention to the inherent differences between RGB and thermal modalities. Recent progress in vision foundation models (VFMs) trained through self-supervision on vast amounts of unlabeled data has proven their ability to extract informative, general-purpose features. However, this potential has yet to be fully leveraged in the domain. In this study, we take one step toward this new research area by exploring a feasible strategy to fully exploit VFM features for RGB-thermal scene parsing. Specifically, we delve deeper into the unique characteristics of RGB and thermal modalities, thereby designing a hybrid, asymmetric encoder that incorporates both a VFM and a convolutional neural network. This design allows for more effective extraction of complementary heterogeneous features, which are subsequently fused in a dual-path, progressive manner. Moreover, we introduce an auxiliary task to further enrich the local semantics of the fused features, thereby improving the overall performance of RGB-thermal scene parsing. Our proposed HAPNet, equipped with all these components, demonstrates superior performance compared to all other state-of-the-art RGB-thermal scene parsing networks, achieving top ranks across three widely used public RGB-thermal scene parsing datasets. We believe this new paradigm has opened up new opportunities for future developments in data-fusion scene parsing approaches.

Title: Evaluating Generative Language Models in Information Extraction as Subjective Question Correction

• Authors: Yuchen Fan, Yantao Liu, Zijun Yao, Jifan Yu, Lei Hou, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03532
• Pdf URL: https://arxiv.org/pdf/2404.03532
• Copy Paste: [[2404.03532]] Evaluating Generative Language Models in Information Extraction as Subjective Question Correction(https://arxiv.org/abs/2404.03532)
• Keywords: generative
• Abstract: Modern Large Language Models (LLMs) have showcased remarkable prowess in various tasks necessitating sophisticated cognitive behaviors. Nevertheless, a paradoxical performance discrepancy is observed, where these models underperform in seemingly elementary tasks like relation extraction and event extraction due to two issues in conventional evaluation. (1) The imprecision of existing evaluation metrics that struggle to effectively gauge semantic consistency between model outputs and ground truth, and (2) The inherent incompleteness of evaluation benchmarks, primarily due to restrictive human annotation schemas, resulting in underestimated LLM performances. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. This method innovatively utilizes LLMs, fine-tuned through subjective question correction data, to refine matching between model outputs and golden labels. Additionally, by incorporating a Natural Language Inference (NLI) model, SQC-Score enriches golden labels, addressing benchmark incompleteness by acknowledging correct yet previously omitted answers. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics. Utilizing SQC-Score, we conduct a comprehensive evaluation of the state-of-the-art LLMs and provide insights for future research for information extraction. Dataset and associated codes can be accessed at https://github.com/THU-KEG/SQC-Score.

Title: How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes

• Authors: Harmon Bhasin, Timothy Ossowski, Yiqiao Zhong, Junjie Hu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03558
• Pdf URL: https://arxiv.org/pdf/2404.03558
• Copy Paste: [[2404.03558]] How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes(https://arxiv.org/abs/2404.03558)
• Keywords: in-context
• Abstract: Large language models (LLM) have recently shown the extraordinary ability to perform unseen tasks based on few-shot examples provided as text, also known as in-context learning (ICL). While recent works have attempted to understand the mechanisms driving ICL, few have explored training strategies that incentivize these models to generalize to multiple tasks. Multi-task learning (MTL) for generalist models is a promising direction that offers transfer learning potential, enabling large parameterized models to be trained from simpler, related tasks. In this work, we investigate the combination of MTL with ICL to build models that efficiently learn tasks while being robust to out-of-distribution examples. We propose several effective curriculum learning strategies that allow ICL models to achieve higher data efficiency and more stable convergence. Our experiments reveal that ICL models can effectively learn difficult tasks by training on progressively harder tasks while mixing in prior tasks, denoted as mixed curriculum in this work. Our code and models are available at https://github.com/harmonbhasin/curriculum_learning_icl .

Title: PointInfinity: Resolution-Invariant Point Diffusion Models

• Authors: Zixuan Huang, Justin Johnson, Shoubhik Debnath, James M. Rehg, Chao-Yuan Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03566
• Pdf URL: https://arxiv.org/pdf/2404.03566
• Copy Paste: [[2404.03566]] PointInfinity: Resolution-Invariant Point Diffusion Models(https://arxiv.org/abs/2404.03566)
• Keywords: diffusion
• Abstract: We present PointInfinity, an efficient family of point cloud diffusion models. Our core idea is to use a transformer-based architecture with a fixed-size, resolution-invariant latent representation. This enables efficient training with low-resolution point clouds, while allowing high-resolution point clouds to be generated during inference. More importantly, we show that scaling the test-time resolution beyond the training resolution improves the fidelity of generated point clouds and surfaces. We analyze this phenomenon and draw a link to classifier-free guidance commonly used in diffusion models, demonstrating that both allow trading off fidelity and variability during inference. Experiments on CO3D show that PointInfinity can efficiently generate high-resolution point clouds (up to 131k points, 31 times more than Point-E) with state-of-the-art quality.

Title: Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithm

• Authors: Miao Lu, Han Zhong, Tong Zhang, Jose Blanchet
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.03578
• Pdf URL: https://arxiv.org/pdf/2404.03578
• Copy Paste: [[2404.03578]] Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithm(https://arxiv.org/abs/2404.03578)
• Keywords: generative
• Abstract: The sim-to-real gap, which represents the disparity between training and testing environments, poses a significant challenge in reinforcement learning (RL). A promising approach to addressing this challenge is distributionally robust RL, often framed as a robust Markov decision process (RMDP). In this framework, the objective is to find a robust policy that achieves good performance under the worst-case scenario among all environments within a pre-specified uncertainty set centered around the training environment. Unlike previous work, which relies on a generative model or a pre-collected offline dataset enjoying good coverage of the deployment environment, we tackle robust RL via interactive data collection, where the learner interacts with the training environment only and refines the policy through trial and error. In this robust RL paradigm, two main challenges emerge: managing distributional robustness while striking a balance between exploration and exploitation during data collection. Initially, we establish that sample-efficient learning without additional assumptions is unattainable owing to the curse of support shift; i.e., the potential disjointedness of the distributional supports between the training and testing environments. To circumvent such a hardness result, we introduce the vanishing minimal value assumption to RMDPs with a total-variation (TV) distance robust set, postulating that the minimal value of the optimal robust value function is zero. We prove that such an assumption effectively eliminates the support shift issue for RMDPs with a TV distance robust set, and present an algorithm with a provable sample complexity guarantee. Our work makes the initial step to uncovering the inherent difficulty of robust RL via interactive data collection and sufficient conditions for designing a sample-efficient algorithm accompanied by sharp sample complexity analysis.

Title: LCM-Lookahead for Encoder-based Text-to-Image Personalization

• Authors: Rinon Gal, Or Lichter, Elad Richardson, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.03620
• Pdf URL: https://arxiv.org/pdf/2404.03620
• Copy Paste: [[2404.03620]] LCM-Lookahead for Encoder-based Text-to-Image Personalization(https://arxiv.org/abs/2404.03620)
• Keywords: diffusion
• Abstract: Recent advancements in diffusion models have introduced fast sampling methods that can effectively produce high-quality images in just one or a few denoising steps. Interestingly, when these are distilled from existing diffusion models, they often maintain alignment with the original model, retaining similar outputs for similar prompts and seeds. These properties present opportunities to leverage fast sampling methods as a shortcut-mechanism, using them to create a preview of denoised outputs through which we can backpropagate image-space losses. In this work, we explore the potential of using such shortcut-mechanisms to guide the personalization of text-to-image models to specific facial identities. We focus on encoder-based personalization approaches, and demonstrate that by tuning them with a lookahead identity loss, we can achieve higher identity fidelity, without sacrificing layout diversity or prompt alignment. We further explore the use of attention sharing mechanisms and consistent data generation for the task of personalization, and find that encoder training can benefit from both.

Title: Reference-Based 3D-Aware Image Editing with Triplane

• Authors: Bahri Batuhan Bilecen, Yigit Yalin, Ning Yu, Aysegul Dundar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03632
• Pdf URL: https://arxiv.org/pdf/2404.03632
• Copy Paste: [[2404.03632]] Reference-Based 3D-Aware Image Editing with Triplane(https://arxiv.org/abs/2404.03632)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) have emerged as powerful tools not only for high-quality image generation but also for real image editing through manipulation of their interpretable latent spaces. Recent advancements in GANs include the development of 3D-aware models such as EG3D, characterized by efficient triplane-based architectures enabling the reconstruction of 3D geometry from single images. However, scant attention has been devoted to providing an integrated framework for high-quality reference-based 3D-aware image editing within this domain. This study addresses this gap by exploring and demonstrating the effectiveness of EG3D's triplane space for achieving advanced reference-based edits, presenting a unique perspective on 3D-aware image editing through our novel pipeline. Our approach integrates the encoding of triplane features, spatial disentanglement and automatic localization of features in the triplane domain, and fusion learning for desired image editing. Moreover, our framework demonstrates versatility across domains, extending its effectiveness to animal face edits and partial stylization of cartoon portraits. The method shows significant improvements over relevant 3D-aware latent editing and 2D reference-based editing methods, both qualitatively and quantitatively. Project page: https://three-bee.github.io/triplane_edit

Title: DiffBody: Human Body Restoration by Imagining with Generative Diffusion Prior

• Authors: Yiming Zhang, Zhe Wang, Xinjie Li, Yunchen Yuan, Chengsong Zhang, Xiao Sun, Zhihang Zhong, Jian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03642
• Pdf URL: https://arxiv.org/pdf/2404.03642
• Copy Paste: [[2404.03642]] DiffBody: Human Body Restoration by Imagining with Generative Diffusion Prior(https://arxiv.org/abs/2404.03642)
• Keywords: diffusion, generative
• Abstract: Human body restoration plays a vital role in various applications related to the human body. Despite recent advances in general image restoration using generative models, their performance in human body restoration remains mediocre, often resulting in foreground and background blending, over-smoothing surface textures, missing accessories, and distorted limbs. Addressing these challenges, we propose a novel approach by constructing a human body-aware diffusion model that leverages domain-specific knowledge to enhance performance. Specifically, we employ a pretrained body attention module to guide the diffusion model's focus on the foreground, addressing issues caused by blending between the subject and background. We also demonstrate the value of revisiting the language modality of the diffusion model in restoration tasks by seamlessly incorporating text prompt to improve the quality of surface texture and additional clothing and accessories details. Additionally, we introduce a diffusion sampler tailored for fine-grained human body parts, utilizing local semantic information to rectify limb distortions. Lastly, we collect a comprehensive dataset for benchmarking and advancing the field of human body restoration. Extensive experimental validation showcases the superiority of our approach, both quantitatively and qualitatively, over existing methods.

Title: The More You See in 2D, the More You Perceive in 3D

• Authors: Xinyang Han, Zelin Gao, Angjoo Kanazawa, Shubham Goel, Yossi Gandelsman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03652
• Pdf URL: https://arxiv.org/pdf/2404.03652
• Copy Paste: [[2404.03652]] The More You See in 2D, the More You Perceive in 3D(https://arxiv.org/abs/2404.03652)
• Keywords: diffusion
• Abstract: Humans can infer 3D structure from 2D images of an object based on past experience and improve their 3D understanding as they see more images. Inspired by this behavior, we introduce SAP3D, a system for 3D reconstruction and novel view synthesis from an arbitrary number of unposed images. Given a few unposed images of an object, we adapt a pre-trained view-conditioned diffusion model together with the camera poses of the images via test-time fine-tuning. The adapted diffusion model and the obtained camera poses are then utilized as instance-specific priors for 3D reconstruction and novel view synthesis. We show that as the number of input images increases, the performance of our approach improves, bridging the gap between optimization-based prior-less 3D reconstruction methods and single-image-to-3D diffusion-based methods. We demonstrate our system on real images as well as standard synthetic benchmarks. Our ablation studies confirm that this adaption behavior is key for more accurate 3D understanding.

Title: CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching

• Authors: Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, Hongsheng Li
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03653
• Pdf URL: https://arxiv.org/pdf/2404.03653
• Copy Paste: [[2404.03653]] CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching(https://arxiv.org/abs/2404.03653)
• Keywords: diffusion
• Abstract: Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. The root reason behind the misalignment has not been extensively investigated. We observe that the misalignment is caused by inadequate token attention activation. We further attribute this phenomenon to the diffusion model's insufficient condition utilization, which is caused by its training paradigm. To address the issue, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with an image-to-text concept matching mechanism. We leverage an image captioning model to measure image-to-text alignment and guide the diffusion model to revisit ignored tokens. A novel attribute concentration module is also proposed to address the attribute binding problem. Without any image or human preference data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL. Extensive experiments show that CoMat-SDXL significantly outperforms the baseline model SDXL in two text-to-image alignment benchmarks and achieves start-of-the-art performance.

Title: RaFE: Generative Radiance Fields Restoration

• Authors: Zhongkai Wu, Ziyu Wan, Jing Zhang, Jing Liao, Dong Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03654
• Pdf URL: https://arxiv.org/pdf/2404.03654
• Copy Paste: [[2404.03654]] RaFE: Generative Radiance Fields Restoration(https://arxiv.org/abs/2404.03654)
• Keywords: generative
• Abstract: NeRF (Neural Radiance Fields) has demonstrated tremendous potential in novel view synthesis and 3D reconstruction, but its performance is sensitive to input image quality, which struggles to achieve high-fidelity rendering when provided with low-quality sparse input viewpoints. Previous methods for NeRF restoration are tailored for specific degradation type, ignoring the generality of restoration. To overcome this limitation, we propose a generic radiance fields restoration pipeline, named RaFE, which applies to various types of degradations, such as low resolution, blurriness, noise, compression artifacts, or their combinations. Our approach leverages the success of off-the-shelf 2D restoration methods to recover the multi-view images individually. Instead of reconstructing a blurred NeRF by averaging inconsistencies, we introduce a novel approach using Generative Adversarial Networks (GANs) for NeRF generation to better accommodate the geometric and appearance inconsistencies present in the multi-view images. Specifically, we adopt a two-level tri-plane architecture, where the coarse level remains fixed to represent the low-quality NeRF, and a fine-level residual tri-plane to be added to the coarse level is modeled as a distribution with GAN to capture potential variations in restoration. We validate RaFE on both synthetic and real cases for various restoration tasks, demonstrating superior performance in both quantitative and qualitative evaluations, surpassing other 3D restoration methods specific to single task. Please see our project website https://zkaiwu.github.io/RaFE-Project/.

Title: MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation

• Authors: Hanzhe Hu, Zhizhuo Zhou, Varun Jampani, Shubham Tulsiani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03656
• Pdf URL: https://arxiv.org/pdf/2404.03656
• Copy Paste: [[2404.03656]] MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation(https://arxiv.org/abs/2404.03656)
• Keywords: diffusion, generative
• Abstract: We present MVD-Fusion: a method for single-view 3D inference via generative modeling of multi-view-consistent RGB-D images. While recent methods pursuing 3D inference advocate learning novel-view generative models, these generations are not 3D-consistent and require a distillation process to generate a 3D output. We instead cast the task of 3D inference as directly generating mutually-consistent multiple views and build on the insight that additionally inferring depth can provide a mechanism for enforcing this consistency. Specifically, we train a denoising diffusion model to generate multi-view RGB-D images given a single RGB input image and leverage the (intermediate noisy) depth estimates to obtain reprojection-based conditioning to maintain multi-view consistency. We train our model using large-scale synthetic dataset Obajverse as well as the real-world CO3D dataset comprising of generic camera viewpoints. We demonstrate that our approach can yield more accurate synthesis compared to recent state-of-the-art, including distillation-based 3D inference and prior multi-view generation methods. We also evaluate the geometry induced by our multi-view depth prediction and find that it yields a more accurate representation than other direct 3D inference approaches.