2024-05-02

Title: Training a high-performance retinal foundation model with half-the-data and 400 times less compute

• Authors: Justin Engelmann, Miguel O. Bernabeu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00117
• Pdf URL: https://arxiv.org/pdf/2405.00117
• Copy Paste: [[2405.00117]] Training a high-performance retinal foundation model with half-the-data and 400 times less compute(https://arxiv.org/abs/2405.00117)
• Keywords: foundation model
• Abstract: Artificial Intelligence holds tremendous potential in medicine, but is traditionally limited by the lack of massive datasets to train models on. Foundation models, pre-trained models that can be adapted to downstream tasks with small datasets, could alleviate this problem. Researchers at Moorfields Eye Hospital (MEH) proposed RETFound-MEH, a foundation model for retinal imaging that was trained on 900,000 images, including private hospital data. Recently, data-efficient DERETFound was proposed that provides comparable performance while being trained on only 150,000 images that are all publicly available. However, both these models required very substantial resources to train initially and are resource-intensive in downstream use. We propose a novel Token Reconstruction objective that we use to train RETFound-Green, a retinal foundation model trained using only 75,000 publicly available images and 400 times less compute. We estimate the cost of training RETFound-MEH and DERETFound at $10,000 and $14,000, respectively, while RETFound-Green could be trained for less than $100, with equally reduced environmental impact. RETFound-Green is also far more efficient in downstream use: it can be downloaded 14 times faster, computes vector embeddings 2.7 times faster which then require 2.6 times less storage space. Despite this, RETFound-Green does not perform systematically worse. In fact, it performs best on 14 tasks, compared to six for DERETFound and two for RETFound-MEH. Our results suggest that RETFound-Green is a very efficient, high-performance retinal foundation model. We anticipate that our Token Reconstruction objective could be scaled up for even higher performance and be applied to other domains beyond retinal imaging.

Title: Re-visiting Skip-Gram Negative Sampling: Dimension Regularization for More Efficient Dissimilarity Preservation in Graph Embeddings

• Authors: David Liu, Arjun Seshadri, Tina Eliassi-Rad, Johan Ugander
• Subjects: cs.LG, cs.SI, stat.ML
• Abstract URL: https://arxiv.org/abs/2405.00172
• Pdf URL: https://arxiv.org/pdf/2405.00172
• Copy Paste: [[2405.00172]] Re-visiting Skip-Gram Negative Sampling: Dimension Regularization for More Efficient Dissimilarity Preservation in Graph Embeddings(https://arxiv.org/abs/2405.00172)
• Keywords: self-supervised
• Abstract: A wide range of graph embedding objectives decompose into two components: one that attracts the embeddings of nodes that are perceived as similar, and another that repels embeddings of nodes that are perceived as dissimilar. Because real-world graphs are sparse and the number of dissimilar pairs grows quadratically with the number of nodes, Skip-Gram Negative Sampling (SGNS) has emerged as a popular and efficient repulsion approach. SGNS repels each node from a sample of dissimilar nodes, as opposed to all dissimilar nodes. In this work, we show that node-wise repulsion is, in aggregate, an approximate re-centering of the node embedding dimensions. Such dimension operations are much more scalable than node operations. The dimension approach, in addition to being more efficient, yields a simpler geometric interpretation of the repulsion. Our result extends findings from the self-supervised learning literature to the skip-gram model, establishing a connection between skip-gram node contrast and dimension regularization. We show that in the limit of large graphs, under mild regularity conditions, the original node repulsion objective converges to optimization with dimension regularization. We use this observation to propose an algorithm augmentation framework that speeds up any existing algorithm, supervised or unsupervised, using SGNS. The framework prioritizes node attraction and replaces SGNS with dimension regularization. We instantiate this generic framework for LINE and node2vec and show that the augmented algorithms preserve downstream performance while dramatically increasing efficiency.

Title: Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly

• Authors: Hang Du, Sicheng Zhang, Binzhu Xie, Guoshun Nan, Jiayang Zhang, Junrui Xu, Hangyu Liu, Sicong Leng, Jiangming Liu, Hehe Fan, Dajiu Huang, Jing Feng, Linli Chen, Can Zhang, Xuhuan Li, Hao Zhang, Jianhang Chen, Qimei Cui, Xiaofeng Tao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00181
• Pdf URL: https://arxiv.org/pdf/2405.00181
• Copy Paste: [[2405.00181]] Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly(https://arxiv.org/abs/2405.00181)
• Keywords: anomaly
• Abstract: Video anomaly understanding (VAU) aims to automatically comprehend unusual occurrences in videos, thereby enabling various applications such as traffic surveillance and industrial manufacturing. While existing VAU benchmarks primarily concentrate on anomaly detection and localization, our focus is on more practicality, prompting us to raise the following crucial questions: "what anomaly occurred?", "why did it happen?", and "how severe is this abnormal event?". In pursuit of these answers, we present a comprehensive benchmark for Causation Understanding of Video Anomaly (CUVA). Specifically, each instance of the proposed benchmark involves three sets of human annotations to indicate the "what", "why" and "how" of an anomaly, including 1) anomaly type, start and end times, and event descriptions, 2) natural language explanations for the cause of an anomaly, and 3) free text reflecting the effect of the abnormality. In addition, we also introduce MMEval, a novel evaluation metric designed to better align with human preferences for CUVA, facilitating the measurement of existing LLMs in comprehending the underlying cause and corresponding effect of video anomalies. Finally, we propose a novel prompt-based method that can serve as a baseline approach for the challenging CUVA. We conduct extensive experiments to show the superiority of our evaluation metric and the prompt-based approach. Our code and dataset are available at https://github.com/fesvhtr/CUVA.

Title: Synthetic Image Verification in the Era of Generative AI: What Works and What Isn't There Yet

• Authors: Diangarti Tariang, Riccardo Corvi, Davide Cozzolino, Giovanni Poggi, Koki Nagano, Luisa Verdoliva
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00196
• Pdf URL: https://arxiv.org/pdf/2405.00196
• Copy Paste: [[2405.00196]] Synthetic Image Verification in the Era of Generative AI: What Works and What Isn't There Yet(https://arxiv.org/abs/2405.00196)
• Keywords: generative
• Abstract: In this work we present an overview of approaches for the detection and attribution of synthetic images and highlight their strengths and weaknesses. We also point out and discuss hot topics in this field and outline promising directions for future research.

Title: In-Context Learning with Long-Context Models: An In-Depth Exploration

• Authors: Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, Graham Neubig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.00200
• Pdf URL: https://arxiv.org/pdf/2405.00200
• Copy Paste: [[2405.00200]] In-Context Learning with Long-Context Models: An In-Depth Exploration(https://arxiv.org/abs/2405.00200)
• Keywords: in-context
• Abstract: As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets. We study the behavior of in-context learning (ICL) at this extreme scale on multiple datasets and models. We show that, for many datasets with large label spaces, performance continues to increase with hundreds or thousands of demonstrations. We contrast this with example retrieval and finetuning: example retrieval shows excellent performance at low context lengths but has diminished gains with more demonstrations; finetuning is more data hungry than ICL but can sometimes exceed long-context ICL performance with additional data. We use this ICL setting as a testbed to study several properties of both in-context learning and long-context models. We show that long-context ICL is less sensitive to random input shuffling than short-context ICL, that grouping of same-label examples can negatively impact performance, and that the performance boosts we see do not arise from cumulative gain from encoding many examples together. We conclude that although long-context ICL can be surprisingly effective, most of this gain comes from attending back to similar examples rather than task learning.

Title: Leveraging Active Subspaces to Capture Epistemic Model Uncertainty in Deep Generative Models for Molecular Design

• Authors: A N M Nafiz Abeer, Sanket Jantre, Nathan M Urban, Byung-Jun Yoon
• Subjects: cs.LG, q-bio.QM, stat.ML
• Abstract URL: https://arxiv.org/abs/2405.00202
• Pdf URL: https://arxiv.org/pdf/2405.00202
• Copy Paste: [[2405.00202]] Leveraging Active Subspaces to Capture Epistemic Model Uncertainty in Deep Generative Models for Molecular Design(https://arxiv.org/abs/2405.00202)
• Keywords: generative
• Abstract: Deep generative models have been accelerating the inverse design process in material and drug design. Unlike their counterpart property predictors in typical molecular design frameworks, generative molecular design models have seen fewer efforts on uncertainty quantification (UQ) due to computational challenges in Bayesian inference posed by their large number of parameters. In this work, we focus on the junction-tree variational autoencoder (JT-VAE), a popular model for generative molecular design, and address this issue by leveraging the low dimensional active subspace to capture the uncertainty in the model parameters. Specifically, we approximate the posterior distribution over the active subspace parameters to estimate the epistemic model uncertainty in an extremely high dimensional parameter space. The proposed UQ scheme does not require alteration of the model architecture, making it readily applicable to any pre-trained model. Our experiments demonstrate the efficacy of the AS-based UQ and its potential impact on molecular optimization by exploring the model diversity under epistemic uncertainty.

Title: A Primer on the Inner Workings of Transformer-based Language Models

• Authors: Javier Ferrando, Gabriele Sarti, Arianna Bisazza, Marta R. Costa-jussà
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.00208
• Pdf URL: https://arxiv.org/pdf/2405.00208
• Copy Paste: [[2405.00208]] A Primer on the Inner Workings of Transformer-based Language Models(https://arxiv.org/abs/2405.00208)
• Keywords: generative
• Abstract: The rapid progress of research aimed at interpreting the inner workings of advanced language models has highlighted a need for contextualizing the insights gained from years of work in this area. This primer provides a concise technical introduction to the current techniques used to interpret the inner workings of Transformer-based language models, focusing on the generative decoder-only architecture. We conclude by presenting a comprehensive overview of the known internal mechanisms implemented by these models, uncovering connections across popular approaches and active research directions in this area.

Title: Graphical Reasoning: LLM-based Semi-Open Relation Extraction

• Authors: Yicheng Tao, Yiqun Wang, Longju Bai
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00216
• Pdf URL: https://arxiv.org/pdf/2405.00216
• Copy Paste: [[2405.00216]] Graphical Reasoning: LLM-based Semi-Open Relation Extraction(https://arxiv.org/abs/2405.00216)
• Keywords: in-context
• Abstract: This paper presents a comprehensive exploration of relation extraction utilizing advanced language models, specifically Chain of Thought (CoT) and Graphical Reasoning (GRE) techniques. We demonstrate how leveraging in-context learning with GPT-3.5 can significantly enhance the extraction process, particularly through detailed example-based reasoning. Additionally, we introduce a novel graphical reasoning approach that dissects relation extraction into sequential sub-tasks, improving precision and adaptability in processing complex relational data. Our experiments, conducted on multiple datasets, including manually annotated data, show considerable improvements in performance metrics, underscoring the effectiveness of our methodologies.

Title: Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion

• Authors: David Geissbühler, Hatef Otroshi Shahreza, Sébastien Marcel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00228
• Pdf URL: https://arxiv.org/pdf/2405.00228
• Copy Paste: [[2405.00228]] Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion(https://arxiv.org/abs/2405.00228)
• Keywords: diffusion, generative
• Abstract: Face Recognition (FR) models are trained on large-scale datasets, which have privacy and ethical concerns. Lately, the use of synthetic data to complement or replace genuine data for the training of FR models has been proposed. While promising results have been obtained, it still remains unclear if generative models can yield diverse enough data for such tasks. In this work, we introduce a new method, inspired by the physical motion of soft particles subjected to stochastic Brownian forces, allowing us to sample identities distributions in a latent space under various constraints. With this in hands, we generate several face datasets and benchmark them by training FR models, showing that data generated with our method exceeds the performance of previously GAN-based datasets and achieves competitive performance with state-of-the-art diffusion-based synthetic datasets. We also show that this method can be used to mitigate leakage from the generator's training set and explore the ability of generative models to generate data beyond it.

Title: Semantically Consistent Video Inpainting with Conditional Diffusion Models

• Authors: Dylan Green, William Harvey, Saeid Naderiparizi, Matthew Niedoba, Yunpeng Liu, Xiaoxuan Liang, Jonathan Lavington, Ke Zhang, Vasileios Lioutas, Setareh Dabiri, Adam Scibior, Berend Zwartsenberg, Frank Wood
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00251
• Pdf URL: https://arxiv.org/pdf/2405.00251
• Copy Paste: [[2405.00251]] Semantically Consistent Video Inpainting with Conditional Diffusion Models(https://arxiv.org/abs/2405.00251)
• Keywords: diffusion, generative
• Abstract: Current state-of-the-art methods for video inpainting typically rely on optical flow or attention-based approaches to inpaint masked regions by propagating visual information across frames. While such approaches have led to significant progress on standard benchmarks, they struggle with tasks that require the synthesis of novel content that is not present in other frames. In this paper we reframe video inpainting as a conditional generative modeling problem and present a framework for solving such problems with conditional video diffusion models. We highlight the advantages of using a generative approach for this task, showing that our method is capable of generating diverse, high-quality inpaintings and synthesizing new content that is spatially, temporally, and semantically consistent with the provided context.

Title: ASAM: Boosting Segment Anything Model with Adversarial Tuning

• Authors: Bo Li, Haoke Xiao, Lv Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00256
• Pdf URL: https://arxiv.org/pdf/2405.00256
• Copy Paste: [[2405.00256]] ASAM: Boosting Segment Anything Model with Adversarial Tuning(https://arxiv.org/abs/2405.00256)
• Keywords: diffusion, foundation model
• Abstract: In the evolving landscape of computer vision, foundation models have emerged as pivotal tools, exhibiting exceptional adaptability to a myriad of tasks. Among these, the Segment Anything Model (SAM) by Meta AI has distinguished itself in image segmentation. However, SAM, like its counterparts, encounters limitations in specific niche applications, prompting a quest for enhancement strategies that do not compromise its inherent capabilities. This paper introduces ASAM, a novel methodology that amplifies SAM's performance through adversarial tuning. We harness the potential of natural adversarial examples, inspired by their successful implementation in natural language processing. By utilizing a stable diffusion model, we augment a subset (1%) of the SA-1B dataset, generating adversarial instances that are more representative of natural variations rather than conventional imperceptible perturbations. Our approach maintains the photorealism of adversarial examples and ensures alignment with original mask annotations, thereby preserving the integrity of the segmentation task. The fine-tuned ASAM demonstrates significant improvements across a diverse range of segmentation tasks without necessitating additional data or architectural modifications. The results of our extensive evaluations confirm that ASAM establishes new benchmarks in segmentation tasks, thereby contributing to the advancement of foundational models in computer vision. Our project page is in https://asam2024.github.io/.

Title: Social Life Simulation for Non-Cognitive Skills Learning

• Authors: Zihan Yan, Yaohong Xiang, Yun Huang
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2405.00273
• Pdf URL: https://arxiv.org/pdf/2405.00273
• Copy Paste: [[2405.00273]] Social Life Simulation for Non-Cognitive Skills Learning(https://arxiv.org/abs/2405.00273)
• Keywords: generative
• Abstract: Non-cognitive skills are crucial for personal and social life well-being, and such skill development can be supported by narrative-based (e.g., storytelling) technologies. While generative AI enables interactive and role-playing storytelling, little is known about how users engage with and perceive the use of AI in social life simulation for non-cognitive skills learning. To this end, we introduced SimuLife++, an interactive platform enabled by a large language model (LLM). The system allows users to act as protagonists, creating stories with one or multiple AI-based characters in diverse social scenarios. In particular, we expanded the Human-AI interaction to a Human-AI-AI collaboration by including a sage agent, who acts as a bystander to provide users with more insightful perspectives on their choices and conversations. Through a within-subject user study, we found that the inclusion of the sage agent significantly enhanced narrative immersion, according to the narrative transportation scale, leading to more messages, particularly in group chats. Participants' interactions with the sage agent were also associated with significantly higher scores in their perceived motivation, self-perceptions, and resilience and coping, indicating positive impacts on non-cognitive skills reflection. Participants' interview results further explained the sage agent's aid in decision-making, solving ethical dilemmas, and problem-solving; on the other hand, they suggested improvements in user control and balanced responses from multiple characters. We provide design implications on the application of generative AI in narrative solutions for non-cognitive skill development in broader social contexts.

Title: How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses

• Authors: Jionghao Lin, Eason Chen, Zeifei Han, Ashish Gurung, Danielle R. Thomas, Wei Tan, Ngoc Dang Nguyen, Kenneth R. Koedinger
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2405.00291
• Pdf URL: https://arxiv.org/pdf/2405.00291
• Copy Paste: [[2405.00291]] How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses(https://arxiv.org/abs/2405.00291)
• Keywords: generative
• Abstract: Automated explanatory feedback systems play a crucial role in facilitating learning for a large cohort of learners by offering feedback that incorporates explanations, significantly enhancing the learning process. However, delivering such explanatory feedback in real-time poses challenges, particularly when high classification accuracy for domain-specific, nuanced responses is essential. Our study leverages the capabilities of large language models, specifically Generative Pre-Trained Transformers (GPT), to explore a sequence labeling approach focused on identifying components of desired and less desired praise for providing explanatory feedback within a tutor training dataset. Our aim is to equip tutors with actionable, explanatory feedback during online training lessons. To investigate the potential of GPT models for providing the explanatory feedback, we employed two commonly-used approaches: prompting and fine-tuning. To quantify the quality of highlighted praise components identified by GPT models, we introduced a Modified Intersection over Union (M-IoU) score. Our findings demonstrate that: (1) the M-IoU score effectively correlates with human judgment in evaluating sequence quality; (2) using two-shot prompting on GPT-3.5 resulted in decent performance in recognizing effort-based (M-IoU of 0.46) and outcome-based praise (M-IoU of 0.68); and (3) our optimally fine-tuned GPT-3.5 model achieved M-IoU scores of 0.64 for effort-based praise and 0.84 for outcome-based praise, aligning with the satisfaction levels evaluated by human coders. Our results show promise for using GPT models to provide feedback that focuses on specific elements in their open-ended responses that are desirable or could use improvement.

Title: MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model

• Authors: Rajat Sahay, Andreas Savakis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00293
• Pdf URL: https://arxiv.org/pdf/2405.00293
• Copy Paste: [[2405.00293]] MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model(https://arxiv.org/abs/2405.00293)
• Keywords: foundation model
• Abstract: The emergence of foundation models, such as the Segment Anything Model (SAM), has sparked interest in Parameter-Efficient Fine-Tuning (PEFT) methods that tailor these large models to application domains outside their training data. However, different PEFT techniques modify the representation of a model differently, making it a non-trivial task to select the most appropriate method for the domain of interest. We propose a new framework, Mixture-of-PEFTs methods (MoPEFT), that is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.

Title: Streamlining Image Editing with Layered Diffusion Brushes

• Authors: Peyman Gholami, Robert Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00313
• Pdf URL: https://arxiv.org/pdf/2405.00313
• Copy Paste: [[2405.00313]] Streamlining Image Editing with Layered Diffusion Brushes(https://arxiv.org/abs/2405.00313)
• Keywords: diffusion
• Abstract: Denoising diffusion models have recently gained prominence as powerful tools for a variety of image generation and manipulation tasks. Building on this, we propose a novel tool for real-time editing of images that provides users with fine-grained region-targeted supervision in addition to existing prompt-based controls. Our novel editing technique, termed Layered Diffusion Brushes, leverages prompt-guided and region-targeted alteration of intermediate denoising steps, enabling precise modifications while maintaining the integrity and context of the input image. We provide an editor based on Layered Diffusion Brushes modifications, which incorporates well-known image editing concepts such as layer masks, visibility toggles, and independent manipulation of layers; regardless of their order. Our system renders a single edit on a 512x512 image within 140 ms using a high-end consumer GPU, enabling real-time feedback and rapid exploration of candidate edits. We validated our method and editing system through a user study involving both natural images (using inversion) and generated images, showcasing its usability and effectiveness compared to existing techniques such as InstructPix2Pix and Stable Diffusion Inpainting for refining images. Our approach demonstrates efficacy across a range of tasks, including object attribute adjustments, error correction, and sequential prompt-based object placement and manipulation, demonstrating its versatility and potential for enhancing creative workflows.

Title: Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis

• Authors: Huy H. Nguyen, Junichi Yamagishi, Isao Echizen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00355
• Pdf URL: https://arxiv.org/pdf/2405.00355
• Copy Paste: [[2405.00355]] Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis(https://arxiv.org/abs/2405.00355)
• Keywords: self-supervised
• Abstract: This paper investigates the effectiveness of self-supervised pre-trained transformers compared to supervised pre-trained transformers and conventional neural networks (ConvNets) for detecting various types of deepfakes. We focus on their potential for improved generalization, particularly when training data is limited. Despite the notable success of large vision-language models utilizing transformer architectures in various tasks, including zero-shot and few-shot learning, the deepfake detection community has still shown some reluctance to adopt pre-trained vision transformers (ViTs), especially large ones, as feature extractors. One concern is their perceived excessive capacity, which often demands extensive data, and the resulting suboptimal generalization when training or fine-tuning data is small or less diverse. This contrasts poorly with ConvNets, which have already established themselves as robust feature extractors. Additionally, training and optimizing transformers from scratch requires significant computational resources, making this accessible primarily to large companies and hindering broader investigation within the academic community. Recent advancements in using self-supervised learning (SSL) in transformers, such as DINO and its derivatives, have showcased significant adaptability across diverse vision tasks and possess explicit semantic segmentation capabilities. By leveraging DINO for deepfake detection with modest training data and implementing partial fine-tuning, we observe comparable adaptability to the task and the natural explainability of the detection result via the attention mechanism. Moreover, partial fine-tuning of transformers for deepfake detection offers a more resource-efficient alternative, requiring significantly fewer computational resources.

Title: Self-supervised Pre-training of Text Recognizers

• Authors: Martin Kišš, Michal Hradiš
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.00420
• Pdf URL: https://arxiv.org/pdf/2405.00420
• Copy Paste: [[2405.00420]] Self-supervised Pre-training of Text Recognizers(https://arxiv.org/abs/2405.00420)
• Keywords: self-supervised
• Abstract: In this paper, we investigate self-supervised pre-training methods for document text recognition. Nowadays, large unlabeled datasets can be collected for many research tasks, including text recognition, but it is costly to annotate them. Therefore, methods utilizing unlabeled data are researched. We study self-supervised pre-training methods based on masked label prediction using three different approaches -- Feature Quantization, VQ-VAE, and Post-Quantized AE. We also investigate joint-embedding approaches with VICReg and NT-Xent objectives, for which we propose an image shifting technique to prevent model collapse where it relies solely on positional encoding while completely ignoring the input image. We perform our experiments on historical handwritten (Bentham) and historical printed datasets mainly to investigate the benefits of the self-supervised pre-training techniques with different amounts of annotated target domain data. We use transfer learning as strong baselines. The evaluation shows that the self-supervised pre-training on data from the target domain is very effective, but it struggles to outperform transfer learning from closely related domains. This paper is one of the first researches exploring self-supervised pre-training in document text recognition, and we believe that it will become a cornerstone for future research in this area. We made our implementation of the investigated methods publicly available at https://github.com/DCGM/pero-pretraining.

Title: Detail-Enhancing Framework for Reference-Based Image Super-Resolution

• Authors: Zihan Wang, Ziliang Xiong, Hongying Tang, Xiaobing Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00431
• Pdf URL: https://arxiv.org/pdf/2405.00431
• Copy Paste: [[2405.00431]] Detail-Enhancing Framework for Reference-Based Image Super-Resolution(https://arxiv.org/abs/2405.00431)
• Keywords: diffusion
• Abstract: Recent years have witnessed the prosperity of reference-based image super-resolution (Ref-SR). By importing the high-resolution (HR) reference images into the single image super-resolution (SISR) approach, the ill-posed nature of this long-standing field has been alleviated with the assistance of texture transferred from reference images. Although the significant improvement in quantitative and qualitative results has verified the superiority of Ref-SR methods, the presence of misalignment before texture transfer indicates room for further performance improvement. Existing methods tend to neglect the significance of details in the context of comparison, therefore not fully leveraging the information contained within low-resolution (LR) images. In this paper, we propose a Detail-Enhancing Framework (DEF) for reference-based super-resolution, which introduces the diffusion model to generate and enhance the underlying detail in LR images. If corresponding parts are present in the reference image, our method can facilitate rigorous alignment. In cases where the reference image lacks corresponding parts, it ensures a fundamental improvement while avoiding the influence of the reference image. Extensive experiments demonstrate that our proposed method achieves superior visual results while maintaining comparable numerical outcomes.

Title: Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable

• Authors: Haozhe Liu, Wentian Zhang, Bing Li, Bernard Ghanem, Jürgen Schmidhuber
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2405.00466
• Pdf URL: https://arxiv.org/pdf/2405.00466
• Copy Paste: [[2405.00466]] Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable(https://arxiv.org/abs/2405.00466)
• Keywords: diffusion, generative
• Abstract: Foundational generative models should be traceable to protect their owners and facilitate safety regulation. To achieve this, traditional approaches embed identifiers based on supervisory trigger-response signals, which are commonly known as backdoor watermarks. They are prone to failure when the model is fine-tuned with nontrigger data. Our experiments show that this vulnerability is due to energetic changes in only a few 'busy' layers during fine-tuning. This yields a novel arbitrary-in-arbitrary-out (AIAO) strategy that makes watermarks resilient to fine-tuning-based removal. The trigger-response pairs of AIAO samples across various neural network depths can be used to construct watermarked subpaths, employing Monte Carlo sampling to achieve stable verification results. In addition, unlike the existing methods of designing a backdoor for the input/output space of diffusion models, in our method, we propose to embed the backdoor into the feature space of sampled subpaths, where a mask-controlled trigger function is proposed to preserve the generation performance and ensure the invisibility of the embedded backdoor. Our empirical studies on the MS-COCO, AFHQ, LSUN, CUB-200, and DreamBooth datasets confirm the robustness of AIAO; while the verification rates of other trigger-based methods fall from ~90% to ~70% after fine-tuning, those of our method remain consistently above 90%.

Title: In Anticipation of Perfect Deepfake: Identity-anchored Artifact-agnostic Detection under Rebalanced Deepfake Detection Protocol

• Authors: Wei-Han Wang, Chin-Yuan Yeh, Hsi-Wen Chen, De-Nian Yang, Ming-Syan Chen
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2405.00483
• Pdf URL: https://arxiv.org/pdf/2405.00483
• Copy Paste: [[2405.00483]] In Anticipation of Perfect Deepfake: Identity-anchored Artifact-agnostic Detection under Rebalanced Deepfake Detection Protocol(https://arxiv.org/abs/2405.00483)
• Keywords: generative
• Abstract: As deep generative models advance, we anticipate deepfakes achieving "perfection"-generating no discernible artifacts or noise. However, current deepfake detectors, intentionally or inadvertently, rely on such artifacts for detection, as they are exclusive to deepfakes and absent in genuine examples. To bridge this gap, we introduce the Rebalanced Deepfake Detection Protocol (RDDP) to stress-test detectors under balanced scenarios where genuine and forged examples bear similar artifacts. We offer two RDDP variants: RDDP-WHITEHAT uses white-hat deepfake algorithms to create 'self-deepfakes,' genuine portrait videos with the resemblance of the underlying identity, yet carry similar artifacts to deepfake videos; RDDP-SURROGATE employs surrogate functions (e.g., Gaussian noise) to process both genuine and forged examples, introducing equivalent noise, thereby sidestepping the need of deepfake algorithms. Towards detecting perfect deepfake videos that aligns with genuine ones, we present ID-Miner, a detector that identifies the puppeteer behind the disguise by focusing on motion over artifacts or appearances. As an identity-based detector, it authenticates videos by comparing them with reference footage. Equipped with the artifact-agnostic loss at frame-level and the identity-anchored loss at video-level, ID-Miner effectively singles out identity signals amidst distracting variations. Extensive experiments comparing ID-Miner with 12 baseline detectors under both conventional and RDDP evaluations with two deepfake datasets, along with additional qualitative studies, affirm the superiority of our method and the necessity for detectors designed to counter perfect deepfakes.

Title: Get Your Embedding Space in Order: Domain-Adaptive Regression for Forest Monitoring

• Authors: Sizhuo Li, Dimitri Gominski, Martin Brandt, Xiaoye Tong, Philippe Ciais
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00514
• Pdf URL: https://arxiv.org/pdf/2405.00514
• Copy Paste: [[2405.00514]] Get Your Embedding Space in Order: Domain-Adaptive Regression for Forest Monitoring(https://arxiv.org/abs/2405.00514)
• Keywords: diffusion
• Abstract: Image-level regression is an important task in Earth observation, where visual domain and label shifts are a core challenge hampering generalization. However, cross-domain regression with remote sensing data remains understudied due to the absence of suited datasets. We introduce a new dataset with aerial and satellite imagery in five countries with three forest-related regression tasks. To match real-world applicative interests, we compare methods through a restrictive setup where no prior on the target domain is available during training, and models are adapted with limited information during testing. Building on the assumption that ordered relationships generalize better, we propose manifold diffusion for regression as a strong baseline for transduction in low-data regimes. Our comparison highlights the comparative advantages of inductive and transductive methods in cross-domain regression.

Title: Discovering robust biomarkers of neurological disorders from functional MRI using graph neural networks: A Review

• Authors: Yi Hao Chan, Deepank Girish, Sukrit Gupta, Jing Xia, Chockalingam Kasi, Yinan He, Conghao Wang, Jagath C. Rajapakse
• Subjects: cs.LG, eess.SP, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2405.00577
• Pdf URL: https://arxiv.org/pdf/2405.00577
• Copy Paste: [[2405.00577]] Discovering robust biomarkers of neurological disorders from functional MRI using graph neural networks: A Review(https://arxiv.org/abs/2405.00577)
• Keywords: generative
• Abstract: Graph neural networks (GNN) have emerged as a popular tool for modelling functional magnetic resonance imaging (fMRI) datasets. Many recent studies have reported significant improvements in disorder classification performance via more sophisticated GNN designs and highlighted salient features that could be potential biomarkers of the disorder. In this review, we provide an overview of how GNN and model explainability techniques have been applied on fMRI datasets for disorder prediction tasks, with a particular emphasis on the robustness of biomarkers produced for neurodegenerative diseases and neuropsychiatric disorders. We found that while most studies have performant models, salient features highlighted in these studies vary greatly across studies on the same disorder and little has been done to evaluate their robustness. To address these issues, we suggest establishing new standards that are based on objective evaluation metrics to determine the robustness of these potential biomarkers. We further highlight gaps in the existing literature and put together a prediction-attribution-evaluation framework that could set the foundations for future research on improving the robustness of potential biomarkers discovered via GNNs.

Title: The Real, the Better: Aligning Large Language Models with Online Human Behaviors

• Authors: Guanying Jiang, Lingyong Yan, Haibo Shi, Dawei Yin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.00578
• Pdf URL: https://arxiv.org/pdf/2405.00578
• Copy Paste: [[2405.00578]] The Real, the Better: Aligning Large Language Models with Online Human Behaviors(https://arxiv.org/abs/2405.00578)
• Keywords: generative
• Abstract: Large language model alignment is widely used and studied to avoid LLM producing unhelpful and harmful responses. However, the lengthy training process and predefined preference bias hinder adaptation to online diverse human preferences. To this end, this paper proposes an alignment framework, called Reinforcement Learning with Human Behavior (RLHB), to align LLMs by directly leveraging real online human behaviors. By taking the generative adversarial framework, the generator is trained to respond following expected human behavior; while the discriminator tries to verify whether the triplets of query, response, and human behavior come from real online environments. Behavior modeling in natural-language form and the multi-model joint training mechanism enable an active and sustainable online alignment. Experimental results confirm the effectiveness of our proposed methods by both human and automatic evaluations.

Title: Lane Segmentation Refinement with Diffusion Models

• Authors: Antonio Ruiz, Andrew Melnik, Dong Wang, Helge Ritter
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00620
• Pdf URL: https://arxiv.org/pdf/2405.00620
• Copy Paste: [[2405.00620]] Lane Segmentation Refinement with Diffusion Models(https://arxiv.org/abs/2405.00620)
• Keywords: diffusion
• Abstract: The lane graph is a key component for building high-definition (HD) maps and crucial for downstream tasks such as autonomous driving or navigation planning. Previously, He et al. (2022) explored the extraction of the lane-level graph from aerial imagery utilizing a segmentation based approach. However, segmentation networks struggle to achieve perfect segmentation masks resulting in inaccurate lane graph extraction. We explore additional enhancements to refine this segmentation-based approach and extend it with a diffusion probabilistic model (DPM) component. This combination further improves the GEO F1 and TOPO F1 scores, which are crucial indicators of the quality of a lane graph, in the undirected graph in non-intersection areas. We conduct experiments on a publicly available dataset, demonstrating that our method outperforms the previous approach, particularly in enhancing the connectivity of such a graph, as measured by the TOPO F1 score. Moreover, we perform ablation studies on the individual components of our method to understand their contribution and evaluate their effectiveness.

Title: Deep Metric Learning-Based Out-of-Distribution Detection with Synthetic Outlier Exposure

• Authors: Assefa Seyoum Wahd
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.00631
• Pdf URL: https://arxiv.org/pdf/2405.00631
• Copy Paste: [[2405.00631]] Deep Metric Learning-Based Out-of-Distribution Detection with Synthetic Outlier Exposure(https://arxiv.org/abs/2405.00631)
• Keywords: diffusion
• Abstract: In this paper, we present a novel approach that combines deep metric learning and synthetic data generation using diffusion models for out-of-distribution (OOD) detection. One popular approach for OOD detection is outlier exposure, where models are trained using a mixture of in-distribution (ID) samples and ``seen" OOD samples. For the OOD samples, the model is trained to minimize the KL divergence between the output probability and the uniform distribution while correctly classifying the in-distribution (ID) data. In this paper, we propose a label-mixup approach to generate synthetic OOD data using Denoising Diffusion Probabilistic Models (DDPMs). Additionally, we explore recent advancements in metric learning to train our models. In the experiments, we found that metric learning-based loss functions perform better than the softmax. Furthermore, the baseline models (including softmax, and metric learning) show a significant improvement when trained with the generated OOD data. Our approach outperforms strong baselines in conventional OOD detection metrics.

Title: NLU-STR at SemEval-2024 Task 1: Generative-based Augmentation and Encoder-based Scoring for Semantic Textual Relatedness

• Authors: Sanad Malaysha, Mustafa Jarrar, Mohammed Khalilia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.00659
• Pdf URL: https://arxiv.org/pdf/2405.00659
• Copy Paste: [[2405.00659]] NLU-STR at SemEval-2024 Task 1: Generative-based Augmentation and Encoder-based Scoring for Semantic Textual Relatedness(https://arxiv.org/abs/2405.00659)
• Keywords: generative
• Abstract: Semantic textual relatedness is a broader concept of semantic similarity. It measures the extent to which two chunks of text convey similar meaning or topics, or share related concepts or contexts. This notion of relatedness can be applied in various applications, such as document clustering and summarizing. SemRel-2024, a shared task in SemEval-2024, aims at reducing the gap in the semantic relatedness task by providing datasets for fourteen languages and dialects including Arabic. This paper reports on our participation in Track A (Algerian and Moroccan dialects) and Track B (Modern Standard Arabic). A BERT-based model is augmented and fine-tuned for regression scoring in supervised track (A), while BERT-based cosine similarity is employed for unsupervised track (B). Our system ranked 1st in SemRel-2024 for MSA with a Spearman correlation score of 0.49. We ranked 5th for Moroccan and 12th for Algerian with scores of 0.83 and 0.53, respectively.

Title: RGB$\leftrightarrow$X: Image decomposition and synthesis using material- and lighting-aware diffusion models

• Authors: Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, Miloš Hašan
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2405.00666
• Pdf URL: https://arxiv.org/pdf/2405.00666
• Copy Paste: [[2405.00666]] RGB$\leftrightarrow$X: Image decomposition and synthesis using material- and lighting-aware diffusion models(https://arxiv.org/abs/2405.00666)
• Keywords: diffusion, generative
• Abstract: The three areas of realistic forward rendering, per-pixel inverse rendering, and generative image synthesis may seem like separate and unrelated sub-fields of graphics and vision. However, recent work has demonstrated improved estimation of per-pixel intrinsic channels (albedo, roughness, metallicity) based on a diffusion architecture; we call this the RGB$\rightarrow$X problem. We further show that the reverse problem of synthesizing realistic images given intrinsic channels, X$\rightarrow$RGB, can also be addressed in a diffusion framework. Focusing on the image domain of interior scenes, we introduce an improved diffusion model for RGB$\rightarrow$X, which also estimates lighting, as well as the first diffusion X$\rightarrow$RGB model capable of synthesizing realistic images from (full or partial) intrinsic channels. Our X$\rightarrow$RGB model explores a middle ground between traditional rendering and generative models: we can specify only certain appearance properties that should be followed, and give freedom to the model to hallucinate a plausible version of the rest. This flexibility makes it possible to use a mix of heterogeneous training datasets, which differ in the available channels. We use multiple existing datasets and extend them with our own synthetic and real data, resulting in a model capable of extracting scene properties better than previous work and of generating highly realistic images of interior scenes.