2024-03-07

Title: Knowledge-guided EEG Representation Learning

• Authors: Aditya Kommineni, Kleanthis Avramidis, Richard Leahy, Shrikanth Narayanan
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.03222
• Pdf URL: https://arxiv.org/pdf/2403.03222
• Copy Paste: [[2403.03222]] Knowledge-guided EEG Representation Learning(https://arxiv.org/abs/2403.03222)
• Keywords: self-supervised
• Abstract: Self-supervised learning has produced impressive results in multimedia domains of audio, vision and speech. This paradigm is equally, if not more, relevant for the domain of biosignals, owing to the scarcity of labelled data in such scenarios. The ability to leverage large-scale unlabelled data to learn robust representations could help improve the performance of numerous inference tasks on biosignals. Given the inherent domain differences between multimedia modalities and biosignals, the established objectives for self-supervised learning may not translate well to this domain. Hence, there is an unmet need to adapt these methods to biosignal analysis. In this work we propose a self-supervised model for EEG, which provides robust performance and remarkable parameter efficiency by using state space-based deep learning architecture. We also propose a novel knowledge-guided pre-training objective that accounts for the idiosyncrasies of the EEG signal. The results indicate improved embedding representation learning and downstream performance compared to prior works on exemplary tasks. Also, the proposed objective significantly reduces the amount of pre-training data required to obtain performance equivalent to prior works.

Title: DINOv2 based Self Supervised Learning For Few Shot Medical Image Segmentation

• Authors: Lev Ayzenberg, Raja Giryes, Hayit Greenspan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.03273
• Pdf URL: https://arxiv.org/pdf/2403.03273
• Copy Paste: [[2403.03273]] DINOv2 based Self Supervised Learning For Few Shot Medical Image Segmentation(https://arxiv.org/abs/2403.03273)
• Keywords: self-supervised
• Abstract: Deep learning models have emerged as the cornerstone of medical image segmentation, but their efficacy hinges on the availability of extensive manually labeled datasets and their adaptability to unforeseen categories remains a challenge. Few-shot segmentation (FSS) offers a promising solution by endowing models with the capacity to learn novel classes from limited labeled examples. A leading method for FSS is ALPNet, which compares features between the query image and the few available support segmented images. A key question about using ALPNet is how to design its features. In this work, we delve into the potential of using features from DINOv2, which is a foundational self-supervised learning model in computer vision. Leveraging the strengths of ALPNet and harnessing the feature extraction capabilities of DINOv2, we present a novel approach to few-shot segmentation that not only enhances performance but also paves the way for more robust and adaptable medical image analysis.

Title: Mad Libs Are All You Need: Augmenting Cross-Domain Document-Level Event Argument Data

• Authors: Joseph Gatto, Parker Seegmiller, Omar Sharif, Sarah M. Preum
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.03304
• Pdf URL: https://arxiv.org/pdf/2403.03304
• Copy Paste: [[2403.03304]] Mad Libs Are All You Need: Augmenting Cross-Domain Document-Level Event Argument Data(https://arxiv.org/abs/2403.03304)
• Keywords: generative
• Abstract: Document-Level Event Argument Extraction (DocEAE) is an extremely difficult information extraction problem -- with significant limitations in low-resource cross-domain settings. To address this problem, we introduce Mad Lib Aug (MLA), a novel generative DocEAE data augmentation framework. Our approach leverages the intuition that Mad Libs, which are categorically masked documents used as a part of a popular game, can be generated and solved by LLMs to produce data for DocEAE. Using MLA, we achieve a 2.6-point average improvement in overall F1 score. Moreover, this approach achieves a 3.9 and 5.2 point average increase in zero and few-shot event roles compared to augmentation-free baselines across all experiments. To better facilitate analysis of cross-domain DocEAE, we additionally introduce a new metric, Role-Depth F1 (RDF1), which uses statistical depth to identify roles in the target domain which are semantic outliers with respect to roles observed in the source domain. Our experiments show that MLA augmentation can boost RDF1 performance by an average of 5.85 points compared to non-augmented datasets.

Title: Japanese-English Sentence Translation Exercises Dataset for Automatic Grading

• Authors: Naoki Miura, Hiroaki Funayama, Seiya Kikuchi, Yuichiroh Matsubayashi, Yuya Iwase, Kentaro Inui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03396
• Pdf URL: https://arxiv.org/pdf/2403.03396
• Copy Paste: [[2403.03396]] Japanese-English Sentence Translation Exercises Dataset for Automatic Grading(https://arxiv.org/abs/2403.03396)
• Keywords: in-context
• Abstract: This paper proposes the task of automatic assessment of Sentence Translation Exercises (STEs), that have been used in the early stage of L2 language learning. We formalize the task as grading student responses for each rubric criterion pre-specified by the educators. We then create a dataset for STE between Japanese and English including 21 questions, along with a total of 3, 498 student responses (167 on average). The answer responses were collected from students and crowd workers. Using this dataset, we demonstrate the performance of baselines including finetuned BERT and GPT models with few-shot in-context learning. Experimental results show that the baseline model with finetuned BERT was able to classify correct responses with approximately 90% in F1, but only less than 80% for incorrect responses. Furthermore, the GPT models with few-shot learning show poorer results than finetuned BERT, indicating that our newly proposed task presents a challenging issue, even for the stateof-the-art large language models.

Title: Contrastive Learning of Person-independent Representations for Facial Action Unit Detection

• Authors: Yong Li, Shiguang Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03400
• Pdf URL: https://arxiv.org/pdf/2403.03400
• Copy Paste: [[2403.03400]] Contrastive Learning of Person-independent Representations for Facial Action Unit Detection(https://arxiv.org/abs/2403.03400)
• Keywords: self-supervised
• Abstract: Facial action unit (AU) detection, aiming to classify AU present in the facial image, has long suffered from insufficient AU annotations. In this paper, we aim to mitigate this data scarcity issue by learning AU representations from a large number of unlabelled facial videos in a contrastive learning paradigm. We formulate the self-supervised AU representation learning signals in two-fold: (1) AU representation should be frame-wisely discriminative within a short video clip; (2) Facial frames sampled from different identities but show analogous facial AUs should have consistent AU representations. As to achieve these goals, we propose to contrastively learn the AU representation within a video clip and devise a cross-identity reconstruction mechanism to learn the person-independent representations. Specially, we adopt a margin-based temporal contrastive learning paradigm to perceive the temporal AU coherence and evolution characteristics within a clip that consists of consecutive input facial frames. Moreover, the cross-identity reconstruction mechanism facilitates pushing the faces from different identities but show analogous AUs close in the latent embedding space. Experimental results on three public AU datasets demonstrate that the learned AU representation is discriminative for AU detection. Our method outperforms other contrastive learning methods and significantly closes the performance gap between the self-supervised and supervised AU detection approaches.

Title: Sculpting Molecules in 3D: A Flexible Substructure Aware Framework for Text-Oriented Molecular Optimization

• Authors: Kaiwei Zhang, Yange Lin, Guangcheng Wu, Yuxiang Ren, Xuecang Zhang, Bo wang, Xiaoyu Zhang, Weitao Du
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2403.03425
• Pdf URL: https://arxiv.org/pdf/2403.03425
• Copy Paste: [[2403.03425]] Sculpting Molecules in 3D: A Flexible Substructure Aware Framework for Text-Oriented Molecular Optimization(https://arxiv.org/abs/2403.03425)
• Keywords: diffusion
• Abstract: The integration of deep learning, particularly AI-Generated Content, with high-quality data derived from ab initio calculations has emerged as a promising avenue for transforming the landscape of scientific research. However, the challenge of designing molecular drugs or materials that incorporate multi-modality prior knowledge remains a critical and complex undertaking. Specifically, achieving a practical molecular design necessitates not only meeting the diversity requirements but also addressing structural and textural constraints with various symmetries outlined by domain experts. In this article, we present an innovative approach to tackle this inverse design problem by formulating it as a multi-modality guidance generation/optimization task. Our proposed solution involves a textural-structure alignment symmetric diffusion framework for the implementation of molecular generation/optimization tasks, namely 3DToMolo. 3DToMolo aims to harmonize diverse modalities, aligning them seamlessly to produce molecular structures adhere to specified symmetric structural and textural constraints by experts in the field. Experimental trials across three guidance generation settings have shown a superior hit generation performance compared to state-of-the-art methodologies. Moreover, 3DToMolo demonstrates the capability to generate novel molecules, incorporating specified target substructures, without the need for prior knowledge. This work not only holds general significance for the advancement of deep learning methodologies but also paves the way for a transformative shift in molecular design strategies. 3DToMolo creates opportunities for a more nuanced and effective exploration of the vast chemical space, opening new frontiers in the development of molecular entities with tailored properties and functionalities.

Title: Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing

• Authors: Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, Jun Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03431
• Pdf URL: https://arxiv.org/pdf/2403.03431
• Copy Paste: [[2403.03431]] Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing(https://arxiv.org/abs/2403.03431)
• Keywords: diffusion
• Abstract: Deep Text-to-Image Synthesis (TIS) models such as Stable Diffusion have recently gained significant popularity for creative Text-to-image generation. Yet, for domain-specific scenarios, tuning-free Text-guided Image Editing (TIE) is of greater importance for application developers, which modify objects or object properties in images by manipulating feature components in attention layers during the generation process. However, little is known about what semantic meanings these attention layers have learned and which parts of the attention maps contribute to the success of image editing. In this paper, we conduct an in-depth probing analysis and demonstrate that cross-attention maps in Stable Diffusion often contain object attribution information that can result in editing failures. In contrast, self-attention maps play a crucial role in preserving the geometric and shape details of the source image during the transformation to the target image. Our analysis offers valuable insights into understanding cross and self-attention maps in diffusion models. Moreover, based on our findings, we simplify popular image editing methods and propose a more straightforward yet more stable and efficient tuning-free procedure that only modifies self-attention maps of the specified attention layers during the denoising process. Experimental results show that our simplified method consistently surpasses the performance of popular approaches on multiple datasets.

Title: DLP-GAN: Learning to Draw Modern Chinese Landscape Photos with Generative Adversarial Network

• Authors: Xiangquan Gui, Binxuan Zhang, Li Li, Yi Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.03456
• Pdf URL: https://arxiv.org/pdf/2403.03456
• Copy Paste: [[2403.03456]] DLP-GAN: Learning to Draw Modern Chinese Landscape Photos with Generative Adversarial Network(https://arxiv.org/abs/2403.03456)
• Keywords: generative
• Abstract: Chinese landscape painting has a unique and artistic style, and its drawing technique is highly abstract in both the use of color and the realistic representation of objects. Previous methods focus on transferring from modern photos to ancient ink paintings. However, little attention has been paid to translating landscape paintings into modern photos. To solve such problems, in this paper, we (1) propose DLP-GAN (\textbf{D}raw Modern Chinese \textbf{L}andscape \textbf{P}hotos with \textbf{G}enerative \textbf{A}dversarial \textbf{N}etwork), an unsupervised cross-domain image translation framework with a novel asymmetric cycle mapping, and (2) introduce a generator based on a dense-fusion module to match different translation directions. Moreover, a dual-consistency loss is proposed to balance the realism and abstraction of model painting. In this way, our model can draw landscape photos and sketches in the modern sense. Finally, based on our collection of modern landscape and sketch datasets, we compare the images generated by our model with other benchmarks. Extensive experiments including user studies show that our model outperforms state-of-the-art methods.

Title: FLAME Diffuser: Grounded Wildfire Image Synthesis using Mask Guided Diffusion

• Authors: Hao Wang, Sayed Pedram Haeri Boroujeni, Xiwen Chen, Ashish Bastola, Huayu Li, Abolfazl Razi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03463
• Pdf URL: https://arxiv.org/pdf/2403.03463
• Copy Paste: [[2403.03463]] FLAME Diffuser: Grounded Wildfire Image Synthesis using Mask Guided Diffusion(https://arxiv.org/abs/2403.03463)
• Keywords: diffusion
• Abstract: The rise of machine learning in recent years has brought benefits to various research fields such as wide fire detection. Nevertheless, small object detection and rare object detection remain a challenge. To address this problem, we present a dataset automata that can generate ground truth paired datasets using diffusion models. Specifically, we introduce a mask-guided diffusion framework that can fusion the wildfire into the existing images while the flame position and size can be precisely controlled. In advance, to fill the gap that the dataset of wildfire images in specific scenarios is missing, we vary the background of synthesized images by controlling both the text prompt and input image. Furthermore, to solve the color tint problem or the well-known domain shift issue, we apply the CLIP model to filter the generated massive dataset to preserve quality. Thus, our proposed framework can generate a massive dataset of that images are high-quality and ground truth-paired, which well addresses the needs of the annotated datasets in specific tasks.

Title: NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging

• Authors: Takahiro Shirakawa, Seiichi Uchida
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03485
• Pdf URL: https://arxiv.org/pdf/2403.03485
• Copy Paste: [[2403.03485]] NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging(https://arxiv.org/abs/2403.03485)
• Keywords: diffusion
• Abstract: Layout-aware text-to-image generation is a task to generate multi-object images that reflect layout conditions in addition to text conditions. The current layout-aware text-to-image diffusion models still have several issues, including mismatches between the text and layout conditions and quality degradation of generated images. This paper proposes a novel layout-aware text-to-image diffusion model called NoiseCollage to tackle these issues. During the denoising process, NoiseCollage independently estimates noises for individual objects and then crops and merges them into a single noise. This operation helps avoid condition mismatches; in other words, it can put the right objects in the right places. Qualitative and quantitative evaluations show that NoiseCollage outperforms several state-of-the-art models. These successful results indicate that the crop-and-merge operation of noises is a reasonable strategy to control image generation. We also show that NoiseCollage can be integrated with ControlNet to use edges, sketches, and pose skeletons as additional conditions. Experimental results show that this integration boosts the layout accuracy of ControlNet. The code is available at https://github.com/univ-esuty/noisecollage.

Title: Unsupervised Multilingual Dense Retrieval via Generative Pseudo Labeling

• Authors: Chao-Wei Huang, Chen-An Li, Tsu-Yuan Hsu, Chen-Yu Hsu, Yun-Nung Chen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.03516
• Pdf URL: https://arxiv.org/pdf/2403.03516
• Copy Paste: [[2403.03516]] Unsupervised Multilingual Dense Retrieval via Generative Pseudo Labeling(https://arxiv.org/abs/2403.03516)
• Keywords: generative
• Abstract: Dense retrieval methods have demonstrated promising performance in multilingual information retrieval, where queries and documents can be in different languages. However, dense retrievers typically require a substantial amount of paired data, which poses even greater challenges in multilingual scenarios. This paper introduces UMR, an Unsupervised Multilingual dense Retriever trained without any paired data. Our approach leverages the sequence likelihood estimation capabilities of multilingual language models to acquire pseudo labels for training dense retrievers. We propose a two-stage framework which iteratively improves the performance of multilingual dense retrievers. Experimental results on two benchmark datasets show that UMR outperforms supervised baselines, showcasing the potential of training multilingual retrievers without paired data, thereby enhancing their practicality. Our source code, data, and models are publicly available at https://github.com/MiuLab/UMR

Title: DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training

• Authors: Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, Jun Zhu
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2403.03542
• Pdf URL: https://arxiv.org/pdf/2403.03542
• Copy Paste: [[2403.03542]] DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training(https://arxiv.org/abs/2403.03542)
• Keywords: foundation model
• Abstract: Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings. However, it is largely in its infancy due to the inherent complexity and diversity, such as long trajectories, multiple scales and varying dimensions of partial differential equations (PDEs) data. In this paper, we present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data and generalizes to various downstream tasks. Moreover, by designing a flexible and scalable model architecture based on Fourier attention, we can easily scale up the model for large-scale pre-training. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories. Extensive experiments show that we achieve SOTA on these benchmarks and validate the strong generalizability of our model to significantly enhance performance on diverse downstream PDE tasks like 3D data. Code is available at \url{https://github.com/thu-ml/DPOT}.

Title: Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem

• Authors: Yuhong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, Hui Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03558
• Pdf URL: https://arxiv.org/pdf/2403.03558
• Copy Paste: [[2403.03558]] Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem(https://arxiv.org/abs/2403.03558)
• Keywords: in-context
• Abstract: Large language models (LLMs) are highly effective in various natural language processing (NLP) tasks. However, they are susceptible to producing unreliable conjectures in ambiguous contexts called hallucination. This paper presents a new method for evaluating LLM hallucination in Question Answering (QA) based on the unanswerable math word problem (MWP). To support this approach, we innovatively develop a dataset called Unanswerable Math Word Problem (UMWP) which comprises 5200 questions across five categories. We developed an evaluation methodology combining text similarity and mathematical expression detection to determine whether LLM considers the question unanswerable. The results of extensive experiments conducted on 31 LLMs, including GPT-3, InstructGPT, LLaMA, and Claude, demonstrate that in-context learning and reinforcement learning with human feedback (RLHF) training significantly enhance the model's ability to avoid hallucination. We show that utilizing MWP is a reliable and effective approach to assess hallucination. Our code and data are available at https://github.com/Yuki-Asuuna/UMWP.

Title: Tackling Missing Values in Probabilistic Wind Power Forecasting: A Generative Approach

• Authors: Honglin Wen, Pierre Pinson, Jie Gu, Zhijian Jin
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.03631
• Pdf URL: https://arxiv.org/pdf/2403.03631
• Copy Paste: [[2403.03631]] Tackling Missing Values in Probabilistic Wind Power Forecasting: A Generative Approach(https://arxiv.org/abs/2403.03631)
• Keywords: generative
• Abstract: Machine learning techniques have been successfully used in probabilistic wind power forecasting. However, the issue of missing values within datasets due to sensor failure, for instance, has been overlooked for a long time. Although it is natural to consider addressing this issue by imputing missing values before model estimation and forecasting, we suggest treating missing values and forecasting targets indifferently and predicting all unknown values simultaneously based on observations. In this paper, we offer an efficient probabilistic forecasting approach by estimating the joint distribution of features and targets based on a generative model. It is free of preprocessing, and thus avoids introducing potential errors. Compared with the traditional "impute, then predict" pipeline, the proposed approach achieves better performance in terms of continuous ranked probability score.

Title: Portraying the Need for Temporal Data in Flood Detection via Sentinel-1

• Authors: Xavier Bou, Thibaud Ehret, Rafael Grompone von Gioi, Jeremy Anger
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.03671
• Pdf URL: https://arxiv.org/pdf/2403.03671
• Copy Paste: [[2403.03671]] Portraying the Need for Temporal Data in Flood Detection via Sentinel-1(https://arxiv.org/abs/2403.03671)
• Keywords: anomaly
• Abstract: Identifying flood affected areas in remote sensing data is a critical problem in earth observation to analyze flood impact and drive responses. While a number of methods have been proposed in the literature, there are two main limitations in available flood detection datasets: (1) a lack of region variability is commonly observed and/or (2) they require to distinguish permanent water bodies from flooded areas from a single image, which becomes an ill-posed setup. Consequently, we extend the globally diverse MMFlood dataset to multi-date by providing one year of Sentinel-1 observations around each flood event. To our surprise, we notice that the definition of flooded pixels in MMFlood is inconsistent when observing the entire image sequence. Hence, we re-frame the flood detection task as a temporal anomaly detection problem, where anomalous water bodies are segmented from a Sentinel-1 temporal sequence. From this definition, we provide a simple method inspired by the popular video change detector ViBe, results of which quantitatively align with the SAR image time series, providing a reasonable baseline for future works.

Title: Multimodal Transformer for Comics Text-Cloze

• Authors: Emanuele Vivoli, Joan Lafuente Baeza, Ernest Valveny Llobet, Dimosthenis Karatzas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03719
• Pdf URL: https://arxiv.org/pdf/2403.03719
• Copy Paste: [[2403.03719]] Multimodal Transformer for Comics Text-Cloze(https://arxiv.org/abs/2403.03719)
• Keywords: self-supervised, generative
• Abstract: This work explores a closure task in comics, a medium where visual and textual elements are intricately intertwined. Specifically, Text-cloze refers to the task of selecting the correct text to use in a comic panel, given its neighboring panels. Traditional methods based on recurrent neural networks have struggled with this task due to limited OCR accuracy and inherent model limitations. We introduce a novel Multimodal Large Language Model (Multimodal-LLM) architecture, specifically designed for Text-cloze, achieving a 10% improvement over existing state-of-the-art models in both its easy and hard variants. Central to our approach is a Domain-Adapted ResNet-50 based visual encoder, fine-tuned to the comics domain in a self-supervised manner using SimCLR. This encoder delivers comparable results to more complex models with just one-fifth of the parameters. Additionally, we release new OCR annotations for this dataset, enhancing model input quality and resulting in another 1% improvement. Finally, we extend the task to a generative format, establishing new baselines and expanding the research possibilities in the field of comics analysis.

Title: Diffusion on language model embeddings for protein sequence generation

• Authors: Viacheslav Meshchaninov, Pavel Strashnov, Andrey Shevtsov, Fedor Nikolaev, Nikita Ivanisenko, Olga Kardymon, Dmitry Vetrov
• Subjects: cs.LG, cs.AI, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2403.03726
• Pdf URL: https://arxiv.org/pdf/2403.03726
• Copy Paste: [[2403.03726]] Diffusion on language model embeddings for protein sequence generation(https://arxiv.org/abs/2403.03726)
• Keywords: diffusion
• Abstract: Protein design requires a deep understanding of the inherent complexities of the protein universe. While many efforts lean towards conditional generation or focus on specific families of proteins, the foundational task of unconditional generation remains underexplored and undervalued. Here, we explore this pivotal domain, introducing DiMA, a model that leverages continuous diffusion on embeddings derived from the protein language model, ESM-2, to generate amino acid sequences. DiMA surpasses leading solutions, including autoregressive transformer-based and discrete diffusion models, and we quantitatively illustrate the impact of the design choices that lead to its superior performance. We extensively evaluate the quality, diversity, distribution similarity, and biological relevance of the generated sequences using multiple metrics across various modalities. Our approach consistently produces novel, diverse protein sequences that accurately reflect the inherent structural and functional diversity of the protein space. This work advances the field of protein design and sets the stage for conditional models by providing a robust framework for scalable and high-quality protein sequence generation.

Title: Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training

• Authors: Paul Doucet, Benjamin Estermann, Till Aczel, Roger Wattenhofer
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03728
• Pdf URL: https://arxiv.org/pdf/2403.03728
• Copy Paste: [[2403.03728]] Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training(https://arxiv.org/abs/2403.03728)
• Keywords: self-supervised
• Abstract: This study addresses the integration of diversity-based and uncertainty-based sampling strategies in active learning, particularly within the context of self-supervised pre-trained models. We introduce a straightforward heuristic called TCM that mitigates the cold start problem while maintaining strong performance across various data levels. By initially applying TypiClust for diversity sampling and subsequently transitioning to uncertainty sampling with Margin, our approach effectively combines the strengths of both strategies. Our experiments demonstrate that TCM consistently outperforms existing methods across various datasets in both low and high data regimes.

Title: Unifying Generation and Compression: Ultra-low bitrate Image Coding Via Multi-stage Transformer

• Authors: Naifu Xue, Qi Mao, Zijian Wang, Yuan Zhang, Siwei Ma
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.03736
• Pdf URL: https://arxiv.org/pdf/2403.03736
• Copy Paste: [[2403.03736]] Unifying Generation and Compression: Ultra-low bitrate Image Coding Via Multi-stage Transformer(https://arxiv.org/abs/2403.03736)
• Keywords: generative
• Abstract: Recent progress in generative compression technology has significantly improved the perceptual quality of compressed data. However, these advancements primarily focus on producing high-frequency details, often overlooking the ability of generative models to capture the prior distribution of image content, thus impeding further bitrate reduction in extreme compression scenarios (<0.05 bpp). Motivated by the capabilities of predictive language models for lossless compression, this paper introduces a novel Unified Image Generation-Compression (UIGC) paradigm, merging the processes of generation and compression. A key feature of the UIGC framework is the adoption of vector-quantized (VQ) image models for tokenization, alongside a multi-stage transformer designed to exploit spatial contextual information for modeling the prior distribution. As such, the dual-purpose framework effectively utilizes the learned prior for entropy estimation and assists in the regeneration of lost tokens. Extensive experiments demonstrate the superiority of the proposed UIGC framework over existing codecs in perceptual quality and human perception, particularly in ultra-low bitrate scenarios (<=0.03 bpp), pioneering a new direction in generative compression.

Title: Self-supervised Photographic Image Layout Representation Learning

• Authors: Zhaoran Zhao, Peng Lu, Xujun Peng, Wenhao Guo
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.03740
• Pdf URL: https://arxiv.org/pdf/2403.03740
• Copy Paste: [[2403.03740]] Self-supervised Photographic Image Layout Representation Learning(https://arxiv.org/abs/2403.03740)
• Keywords: self-supervised
• Abstract: In the domain of image layout representation learning, the critical process of translating image layouts into succinct vector forms is increasingly significant across diverse applications, such as image retrieval, manipulation, and generation. Most approaches in this area heavily rely on costly labeled datasets and notably lack in adapting their modeling and learning methods to the specific nuances of photographic image layouts. This shortfall makes the learning process for photographic image layouts suboptimal. In our research, we directly address these challenges. We innovate by defining basic layout primitives that encapsulate various levels of layout information and by mapping these, along with their interconnections, onto a heterogeneous graph structure. This graph is meticulously engineered to capture the intricate layout information within the pixel domain explicitly. Advancing further, we introduce novel pretext tasks coupled with customized loss functions, strategically designed for effective self-supervised learning of these layout graphs. Building on this foundation, we develop an autoencoder-based network architecture skilled in compressing these heterogeneous layout graphs into precise, dimensionally-reduced layout representations. Additionally, we introduce the LODB dataset, which features a broader range of layout categories and richer semantics, serving as a comprehensive benchmark for evaluating the effectiveness of layout representation learning methods. Our extensive experimentation on this dataset demonstrates the superior performance of our approach in the realm of photographic image layout representation learning.

Title: German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset

• Authors: Laura Mascarell, Ribin Chalumattu, Annette Rios
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.03750
• Pdf URL: https://arxiv.org/pdf/2403.03750
• Copy Paste: [[2403.03750]] German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset(https://arxiv.org/abs/2403.03750)
• Keywords: in-context
• Abstract: The advent of Large Language Models (LLMs) has led to remarkable progress on a wide range of natural language processing tasks. Despite the advances, these large-sized models still suffer from hallucinating information in their output, which poses a major issue in automatic text summarization, as we must guarantee that the generated summary is consistent with the content of the source document. Previous research addresses the challenging task of detecting hallucinations in the output (i.e. inconsistency detection) in order to evaluate the faithfulness of the generated summaries. However, these works primarily focus on English and recent multilingual approaches lack German data. This work presents absinth, a manually annotated dataset for hallucination detection in German news summarization and explores the capabilities of novel open-source LLMs on this task in both fine-tuning and in-context learning settings. We open-source and release the absinth dataset to foster further research on hallucination detection in German.

Title: Feature Selection as Deep Sequential Generative Learning

• Authors: Wangyang Ying, Dongjie Wang, Haifeng Chen, Yanjie Fu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.03838
• Pdf URL: https://arxiv.org/pdf/2403.03838
• Copy Paste: [[2403.03838]] Feature Selection as Deep Sequential Generative Learning(https://arxiv.org/abs/2403.03838)
• Keywords: generative
• Abstract: Feature selection aims to identify the most pattern-discriminative feature subset. In prior literature, filter (e.g., backward elimination) and embedded (e.g., Lasso) methods have hyperparameters (e.g., top-K, score thresholding) and tie to specific models, thus, hard to generalize; wrapper methods search a feature subset in a huge discrete space and is computationally costly. To transform the way of feature selection, we regard a selected feature subset as a selection decision token sequence and reformulate feature selection as a deep sequential generative learning task that distills feature knowledge and generates decision sequences. Our method includes three steps: (1) We develop a deep variational transformer model over a joint of sequential reconstruction, variational, and performance evaluator losses. Our model can distill feature selection knowledge and learn a continuous embedding space to map feature selection decision sequences into embedding vectors associated with utility scores. (2) We leverage the trained feature subset utility evaluator as a gradient provider to guide the identification of the optimal feature subset embedding;(3) We decode the optimal feature subset embedding to autoregressively generate the best feature selection decision sequence with autostop. Extensive experimental results show this generative perspective is effective and generic, without large discrete search space and expert-specific hyperparameters.

Title: Accelerating Convergence of Score-Based Diffusion Models, Provably

• Authors: Gen Li, Yu Huang, Timofey Efimov, Yuting Wei, Yuejie Chi, Yuxin Chen
• Subjects: cs.LG, cs.AI, cs.IT, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.03852
• Pdf URL: https://arxiv.org/pdf/2403.03852
• Copy Paste: [[2403.03852]] Accelerating Convergence of Score-Based Diffusion Models, Provably(https://arxiv.org/abs/2403.03852)
• Keywords: diffusion, generative
• Abstract: Score-based diffusion models, while achieving remarkable empirical performance, often suffer from low sampling speed, due to extensive function evaluations needed during the sampling phase. Despite a flurry of recent activities towards speeding up diffusion generative modeling in practice, theoretical underpinnings for acceleration techniques remain severely limited. In this paper, we design novel training-free algorithms to accelerate popular deterministic (i.e., DDIM) and stochastic (i.e., DDPM) samplers. Our accelerated deterministic sampler converges at a rate $O(1/{T}^2)$ with $T$ the number of steps, improving upon the $O(1/T)$ rate for the DDIM sampler; and our accelerated stochastic sampler converges at a rate $O(1/T)$, outperforming the rate $O(1/\sqrt{T})$ for the DDPM sampler. The design of our algorithms leverages insights from higher-order approximation, and shares similar intuitions as popular high-order ODE solvers like the DPM-Solver-2. Our theory accommodates $\ell_2$-accurate score estimates, and does not require log-concavity or smoothness on the target distribution.

Title: Latent Dataset Distillation with Diffusion Models

• Authors: Brian B. Moser, Federico Raue, Sebastian Palacio, Stanislav Frolov, Andreas Dengel
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.03881
• Pdf URL: https://arxiv.org/pdf/2403.03881
• Copy Paste: [[2403.03881]] Latent Dataset Distillation with Diffusion Models(https://arxiv.org/abs/2403.03881)
• Keywords: diffusion
• Abstract: The efficacy of machine learning has traditionally relied on the availability of increasingly larger datasets. However, large datasets pose storage challenges and contain non-influential samples, which could be ignored during training without impacting the final accuracy of the model. In response to these limitations, the concept of distilling the information on a dataset into a condensed set of (synthetic) samples, namely a distilled dataset, emerged. One crucial aspect is the selected architecture (usually ConvNet) for linking the original and synthetic datasets. However, the final accuracy is lower if the employed model architecture differs from the model used during distillation. Another challenge is the generation of high-resolution images, e.g., 128x128 and higher. In this paper, we propose Latent Dataset Distillation with Diffusion Models (LD3M) that combine diffusion in latent space with dataset distillation to tackle both challenges. LD3M incorporates a novel diffusion process tailored for dataset distillation, which improves the gradient norms for learning synthetic images. By adjusting the number of diffusion steps, LD3M also offers a straightforward way of controlling the trade-off between speed and accuracy. We evaluate our approach in several ImageNet subsets and for high-resolution images (128x128 and 256x256). As a result, LD3M consistently outperforms state-of-the-art distillation techniques by up to 4.8 p.p. and 4.2 p.p. for 1 and 10 images per class, respectively.

Title: Extreme Precipitation Nowcasting using Transformer-based Generative Models

• Authors: Cristian Meo, Ankush Roy, Mircea Lică, Junzhe Yin, Zeineb Bou Che, Yanbo Wang, Ruben Imhoff, Remko Uijlenhoet, Justin Dauwels
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.03929
• Pdf URL: https://arxiv.org/pdf/2403.03929
• Copy Paste: [[2403.03929]] Extreme Precipitation Nowcasting using Transformer-based Generative Models(https://arxiv.org/abs/2403.03929)
• Keywords: generative
• Abstract: This paper presents an innovative approach to extreme precipitation nowcasting by employing Transformer-based generative models, namely NowcastingGPT with Extreme Value Loss (EVL) regularization. Leveraging a comprehensive dataset from the Royal Netherlands Meteorological Institute (KNMI), our study focuses on predicting short-term precipitation with high accuracy. We introduce a novel method for computing EVL without assuming fixed extreme representations, addressing the limitations of current models in capturing extreme weather events. We present both qualitative and quantitative analyses, demonstrating the superior performance of the proposed NowcastingGPT-EVL in generating accurate precipitation forecasts, especially when dealing with extreme precipitation events. The code is available at \url{https://github.com/Cmeo97/NowcastingGPT}.

Title: GUIDE: Guidance-based Incremental Learning with Diffusion Models

• Authors: Bartosz Cywiński, Kamil Deja, Tomasz Trzciński, Bartłomiej Twardowski, Łukasz Kuciński
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.03938
• Pdf URL: https://arxiv.org/pdf/2403.03938
• Copy Paste: [[2403.03938]] GUIDE: Guidance-based Incremental Learning with Diffusion Models(https://arxiv.org/abs/2403.03938)
• Keywords: diffusion, generative
• Abstract: We introduce GUIDE, a novel continual learning approach that directs diffusion models to rehearse samples at risk of being forgotten. Existing generative strategies combat catastrophic forgetting by randomly sampling rehearsal examples from a generative model. Such an approach contradicts buffer-based approaches where sampling strategy plays an important role. We propose to bridge this gap by integrating diffusion models with classifier guidance techniques to produce rehearsal examples specifically targeting information forgotten by a continuously trained model. This approach enables the generation of samples from preceding task distributions, which are more likely to be misclassified in the context of recently encountered classes. Our experimental results show that GUIDE significantly reduces catastrophic forgetting, outperforming conventional random sampling approaches and surpassing recent state-of-the-art methods in continual learning with generative replay.