2024-04-24

Title: A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models

• Authors: Yefeng Yuan, Yuhong Liu, Liang Cheng
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14445
• Pdf URL: https://arxiv.org/pdf/2404.14445
• Copy Paste: [[2404.14445]] A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models(https://arxiv.org/abs/2404.14445)
• Keywords: generative
• Abstract: The rapid advancements in generative AI and large language models (LLMs) have opened up new avenues for producing synthetic data, particularly in the realm of structured tabular formats, such as product reviews. Despite the potential benefits, concerns regarding privacy leakage have surfaced, especially when personal information is utilized in the training datasets. In addition, there is an absence of a comprehensive evaluation framework capable of quantitatively measuring the quality of the generated synthetic data and their utility for downstream tasks. In response to this gap, we introduce SynEval, an open-source evaluation framework designed to assess the fidelity, utility, and privacy preservation of synthetically generated tabular data via a suite of diverse evaluation metrics. We validate the efficacy of our proposed framework - SynEval - by applying it to synthetic product review data generated by three state-of-the-art LLMs: ChatGPT, Claude, and Llama. Our experimental findings illuminate the trade-offs between various evaluation metrics in the context of synthetic data generation. Furthermore, SynEval stands as a critical instrument for researchers and practitioners engaged with synthetic tabular data,, empowering them to judiciously determine the suitability of the generated data for their specific applications, with an emphasis on upholding user privacy.

Title: Generative Subspace Adversarial Active Learning for Outlier Detection in Multiple Views of High-dimensional Data

• Authors: Jose Cribeiro-Ramallo, Vadim Arzamasov, Federico Matteucci, Denis Wambold, Klemens Böhm
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14451
• Pdf URL: https://arxiv.org/pdf/2404.14451
• Copy Paste: [[2404.14451]] Generative Subspace Adversarial Active Learning for Outlier Detection in Multiple Views of High-dimensional Data(https://arxiv.org/abs/2404.14451)
• Keywords: generative
• Abstract: Outlier detection in high-dimensional tabular data is an important task in data mining, essential for many downstream tasks and applications. Existing unsupervised outlier detection algorithms face one or more problems, including inlier assumption (IA), curse of dimensionality (CD), and multiple views (MV). To address these issues, we introduce Generative Subspace Adversarial Active Learning (GSAAL), a novel approach that uses a Generative Adversarial Network with multiple adversaries. These adversaries learn the marginal class probability functions over different data subspaces, while a single generator in the full space models the entire distribution of the inlier class. GSAAL is specifically designed to address the MV limitation while also handling the IA and CD, being the only method to do so. We provide a comprehensive mathematical formulation of MV, convergence guarantees for the discriminators, and scalability results for GSAAL. Our extensive experiments demonstrate the effectiveness and scalability of GSAAL, highlighting its superior performance compared to other popular OD methods, especially in MV scenarios.

Title: Reinforcement of Explainability of ChatGPT Prompts by Embedding Breast Cancer Self-Screening Rules into AI Responses

• Authors: Yousef Khan, Ahmed Abdeen Hamed
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14454
• Pdf URL: https://arxiv.org/pdf/2404.14454
• Copy Paste: [[2404.14454]] Reinforcement of Explainability of ChatGPT Prompts by Embedding Breast Cancer Self-Screening Rules into AI Responses(https://arxiv.org/abs/2404.14454)
• Keywords: generative
• Abstract: Addressing the global challenge of breast cancer, this research explores the fusion of generative AI, focusing on ChatGPT 3.5 turbo model, and the intricacies of breast cancer risk assessment. The research aims to evaluate ChatGPT's reasoning capabilities, emphasizing its potential to process rules and provide explanations for screening recommendations. The study seeks to bridge the technology gap between intelligent machines and clinicians by demonstrating ChatGPT's unique proficiency in natural language reasoning. The methodology employs a supervised prompt-engineering approach to enforce detailed explanations for ChatGPT's recommendations. Synthetic use cases, generated algorithmically, serve as the testing ground for the encoded rules, evaluating the model's processing prowess. Findings highlight ChatGPT's promising capacity in processing rules comparable to Expert System Shells, with a focus on natural language reasoning. The research introduces the concept of reinforcement explainability, showcasing its potential in elucidating outcomes and facilitating user-friendly interfaces for breast cancer risk assessment.

Title: A Neuro-Symbolic Explainer for Rare Events: A Case Study on Predictive Maintenance

• Authors: João Gama, Rita P. Ribeiro, Saulo Mastelini, Narjes Davarid, Bruno Veloso
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14455
• Pdf URL: https://arxiv.org/pdf/2404.14455
• Copy Paste: [[2404.14455]] A Neuro-Symbolic Explainer for Rare Events: A Case Study on Predictive Maintenance(https://arxiv.org/abs/2404.14455)
• Keywords: anomaly
• Abstract: Predictive Maintenance applications are increasingly complex, with interactions between many components. Black box models are popular approaches based on deep learning techniques due to their predictive accuracy. This paper proposes a neural-symbolic architecture that uses an online rule-learning algorithm to explain when the black box model predicts failures. The proposed system solves two problems in parallel: anomaly detection and explanation of the anomaly. For the first problem, we use an unsupervised state of the art autoencoder. For the second problem, we train a rule learning system that learns a mapping from the input features to the autoencoder reconstruction error. Both systems run online and in parallel. The autoencoder signals an alarm for the examples with a reconstruction error that exceeds a threshold. The causes of the signal alarm are hard for humans to understand because they result from a non linear combination of sensor data. The rule that triggers that example describes the relationship between the input features and the autoencoder reconstruction error. The rule explains the failure signal by indicating which sensors contribute to the alarm and allowing the identification of the component involved in the failure. The system can present global explanations for the black box model and local explanations for why the black box model predicts a failure. We evaluate the proposed system in a real-world case study of Metro do Porto and provide explanations that illustrate its benefits.

Title: Graph Coloring Using Heat Diffusion

• Authors: Vivek Chaudhary
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14457
• Pdf URL: https://arxiv.org/pdf/2404.14457
• Copy Paste: [[2404.14457]] Graph Coloring Using Heat Diffusion(https://arxiv.org/abs/2404.14457)
• Keywords: diffusion
• Abstract: Graph coloring is a problem with varied applications in industry and science such as scheduling, resource allocation, and circuit design. The purpose of this paper is to establish if a new gradient based iterative solver framework known as heat diffusion can solve the graph coloring problem. We propose a solution to the graph coloring problem using the heat diffusion framework. We compare the solutions against popular methods and establish the competitiveness of heat diffusion method for the graph coloring problem.

Title: Align Your Steps: Optimizing Sampling Schedules in Diffusion Models

• Authors: Amirmojtaba Sabour, Sanja Fidler, Karsten Kreis
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14507
• Pdf URL: https://arxiv.org/pdf/2404.14507
• Copy Paste: [[2404.14507]] Align Your Steps: Optimizing Sampling Schedules in Diffusion Models(https://arxiv.org/abs/2404.14507)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DMs) have established themselves as the state-of-the-art generative modeling approach in the visual domain and beyond. A crucial drawback of DMs is their slow sampling speed, relying on many sequential function evaluations through large neural networks. Sampling from DMs can be seen as solving a differential equation through a discretized set of noise levels known as the sampling schedule. While past works primarily focused on deriving efficient solvers, little attention has been given to finding optimal sampling schedules, and the entire literature relies on hand-crafted heuristics. In this work, for the first time, we propose a general and principled approach to optimizing the sampling schedules of DMs for high-quality outputs, called $\textit{Align Your Steps}$. We leverage methods from stochastic calculus and find optimal schedules specific to different solvers, trained DMs and datasets. We evaluate our novel approach on several image, video as well as 2D toy data synthesis benchmarks, using a variety of different samplers, and observe that our optimized schedules outperform previous hand-crafted schedules in almost all experiments. Our method demonstrates the untapped potential of sampling schedule optimization, especially in the few-step synthesis regime.

Title: UVMap-ID: A Controllable and Personalized UV Map Generative Model

• Authors: Weijie Wang, Jichao Zhang, Chang Liu, Xia Li, Xingqian Xu, Humphrey Shi, Nicu Sebe, Bruno Lepri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14568
• Pdf URL: https://arxiv.org/pdf/2404.14568
• Copy Paste: [[2404.14568]] UVMap-ID: A Controllable and Personalized UV Map Generative Model(https://arxiv.org/abs/2404.14568)
• Keywords: diffusion, generative
• Abstract: Recently, diffusion models have made significant strides in synthesizing realistic 2D human images based on provided text prompts. Building upon this, researchers have extended 2D text-to-image diffusion models into the 3D domain for generating human textures (UV Maps). However, some important problems about UV Map Generative models are still not solved, i.e., how to generate personalized texture maps for any given face image, and how to define and evaluate the quality of these generated texture maps. To solve the above problems, we introduce a novel method, UVMap-ID, which is a controllable and personalized UV Map generative model. Unlike traditional large-scale training methods in 2D, we propose to fine-tune a pre-trained text-to-image diffusion model which is integrated with a face fusion module for achieving ID-driven customized generation. To support the finetuning strategy, we introduce a small-scale attribute-balanced training dataset, including high-quality textures with labeled text and Face ID. Additionally, we introduce some metrics to evaluate the multiple aspects of the textures. Finally, both quantitative and qualitative analyses demonstrate the effectiveness of our method in controllable and personalized UV Map generation. Code is publicly available via https://github.com/twowwj/UVMap-ID.

Title: The Adversarial AI-Art: Understanding, Generation, Detection, and Benchmarking

• Authors: Yuying Li, Zeyan Liu, Junyi Zhao, Liangqin Ren, Fengjun Li, Jiebo Luo, Bo Luo
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.14581
• Pdf URL: https://arxiv.org/pdf/2404.14581
• Copy Paste: [[2404.14581]] The Adversarial AI-Art: Understanding, Generation, Detection, and Benchmarking(https://arxiv.org/abs/2404.14581)
• Keywords: generative
• Abstract: Generative AI models can produce high-quality images based on text prompts. The generated images often appear indistinguishable from images generated by conventional optical photography devices or created by human artists (i.e., real images). While the outstanding performance of such generative models is generally well received, security concerns arise. For instance, such image generators could be used to facilitate fraud or scam schemes, generate and spread misinformation, or produce fabricated artworks. In this paper, we present a systematic attempt at understanding and detecting AI-generated images (AI-art) in adversarial scenarios. First, we collect and share a dataset of real images and their corresponding artificial counterparts generated by four popular AI image generators. The dataset, named ARIA, contains over 140K images in five categories: artworks (painting), social media images, news photos, disaster scenes, and anime pictures. This dataset can be used as a foundation to support future research on adversarial AI-art. Next, we present a user study that employs the ARIA dataset to evaluate if real-world users can distinguish with or without reference images. In a benchmarking study, we further evaluate if state-of-the-art open-source and commercial AI image detectors can effectively identify the images in the ARIA dataset. Finally, we present a ResNet-50 classifier and evaluate its accuracy and transferability on the ARIA dataset.

Title: LaneCorrect: Self-supervised Lane Detection

• Authors: Ming Nie, Xinyue Cai, Hang Xu, Li Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14671
• Pdf URL: https://arxiv.org/pdf/2404.14671
• Copy Paste: [[2404.14671]] LaneCorrect: Self-supervised Lane Detection(https://arxiv.org/abs/2404.14671)
• Keywords: self-supervised
• Abstract: Lane detection has evolved highly functional autonomous driving system to understand driving scenes even under complex environments. In this paper, we work towards developing a generalized computer vision system able to detect lanes without using any annotation. We make the following contributions: (i) We illustrate how to perform unsupervised 3D lane segmentation by leveraging the distinctive intensity of lanes on the LiDAR point cloud frames, and then obtain the noisy lane labels in the 2D plane by projecting the 3D points; (ii) We propose a novel self-supervised training scheme, dubbed LaneCorrect, that automatically corrects the lane label by learning geometric consistency and instance awareness from the adversarial augmentations; (iii) With the self-supervised pre-trained model, we distill to train a student network for arbitrary target lane (e.g., TuSimple) detection without any human labels; (iv) We thoroughly evaluate our self-supervised method on four major lane detection benchmarks (including TuSimple, CULane, CurveLanes and LLAMAS) and demonstrate excellent performance compared with existing supervised counterpart, whilst showing more effective results on alleviating the domain gap, i.e., training on CULane and test on TuSimple.

Title: DreamPBR: Text-driven Generation of High-resolution SVBRDF with Multi-modal Guidance

• Authors: Linxuan Xin, Zheng Zhang, Jinfu Wei, Ge Li, Duan Gao
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.14676
• Pdf URL: https://arxiv.org/pdf/2404.14676
• Copy Paste: [[2404.14676]] DreamPBR: Text-driven Generation of High-resolution SVBRDF with Multi-modal Guidance(https://arxiv.org/abs/2404.14676)
• Keywords: diffusion, generative
• Abstract: Prior material creation methods had limitations in producing diverse results mainly because reconstruction-based methods relied on real-world measurements and generation-based methods were trained on relatively small material datasets. To address these challenges, we propose DreamPBR, a novel diffusion-based generative framework designed to create spatially-varying appearance properties guided by text and multi-modal controls, providing high controllability and diversity in material generation. Key to achieving diverse and high-quality PBR material generation lies in integrating the capabilities of recent large-scale vision-language models trained on billions of text-image pairs, along with material priors derived from hundreds of PBR material samples. We utilize a novel material Latent Diffusion Model (LDM) to establish the mapping between albedo maps and the corresponding latent space. The latent representation is then decoded into full SVBRDF parameter maps using a rendering-aware PBR decoder. Our method supports tileable generation through convolution with circular padding. Furthermore, we introduce a multi-modal guidance module, which includes pixel-aligned guidance, style image guidance, and 3D shape guidance, to enhance the control capabilities of the material LDM. We demonstrate the effectiveness of DreamPBR in material creation, showcasing its versatility and user-friendliness on a wide range of controllable generation and editing applications.

Title: Automated Multi-Language to English Machine Translation Using Generative Pre-Trained Transformers

• Authors: Elijah Pelofske, Vincent Urias, Lorie M. Liebrock
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14680
• Pdf URL: https://arxiv.org/pdf/2404.14680
• Copy Paste: [[2404.14680]] Automated Multi-Language to English Machine Translation Using Generative Pre-Trained Transformers(https://arxiv.org/abs/2404.14680)
• Keywords: generative
• Abstract: The task of accurate and efficient language translation is an extremely important information processing task. Machine learning enabled and automated translation that is accurate and fast is often a large topic of interest in the machine learning and data science communities. In this study, we examine using local Generative Pretrained Transformer (GPT) models to perform automated zero shot black-box, sentence wise, multi-natural-language translation into English text. We benchmark 16 different open-source GPT models, with no custom fine-tuning, from the Huggingface LLM repository for translating 50 different non-English languages into English using translated TED Talk transcripts as the reference dataset. These GPT model inference calls are performed strictly locally, on single A100 Nvidia GPUs. Benchmark metrics that are reported are language translation accuracy, using BLEU, GLEU, METEOR, and chrF text overlap measures, and wall-clock time for each sentence translation. The best overall performing GPT model for translating into English text for the BLEU metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.152$, for the GLEU metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.256$, for the chrF metric is Llama2-chat-AYT-13B with a mean score across all tested languages of $0.448$, and for the METEOR metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.438$.

Title: FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model

• Authors: Zezheng Song, Jiaxin Yuan, Haizhao Yang
• Subjects: cs.LG, cs.AI, cs.CE, math.DS, math.NA
• Abstract URL: https://arxiv.org/abs/2404.14688
• Pdf URL: https://arxiv.org/pdf/2404.14688
• Copy Paste: [[2404.14688]] FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model(https://arxiv.org/abs/2404.14688)
• Keywords: foundation model, generative, in-context
• Abstract: Human-designed algorithms have long been fundamental in solving a variety of scientific and engineering challenges. Recently, data-driven deep learning methods have also risen to prominence, offering innovative solutions across numerous scientific fields. While traditional algorithms excel in capturing the core aspects of specific problems, they often lack the flexibility needed for varying problem conditions due to the absence of specific data. Conversely, while data-driven approaches utilize vast datasets, they frequently fall short in domain-specific knowledge. To bridge these gaps, we introduce \textbf{FMint} (Foundation Model based on Initialization), a generative pre-trained model that synergizes the precision of human-designed algorithms with the adaptability of data-driven methods. This model is specifically engineered for high-accuracy simulation of dynamical systems. Starting from initial trajectories provided by conventional methods, FMint quickly delivers highly accurate solutions. It incorporates in-context learning and has been pre-trained on a diverse corpus of 500,000 dynamical systems, showcasing exceptional generalization across a broad spectrum of real-world applications. By effectively combining algorithmic rigor with data-driven flexibility, FMint sets the stage for the next generation of scientific foundation models, tackling complex problems with both efficiency and high accuracy.

Title: FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction

• Authors: Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Daoan Zhang, John Collomosse, Scott Cohen, Jiebo Luo
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14715
• Pdf URL: https://arxiv.org/pdf/2404.14715
• Copy Paste: [[2404.14715]] FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction(https://arxiv.org/abs/2404.14715)
• Keywords: in-context
• Abstract: Recent progress in large-scale pre-training has led to the development of advanced vision-language models (VLMs) with remarkable proficiency in comprehending and generating multimodal content. Despite the impressive ability to perform complex reasoning for VLMs, current models often struggle to effectively and precisely capture the compositional information on both the image and text sides. To address this, we propose FineMatch, a new aspect-based fine-grained text and image matching benchmark, focusing on text and image mismatch detection and correction. This benchmark introduces a novel task for boosting and evaluating the VLMs' compositionality for aspect-based fine-grained text and image matching. In this task, models are required to identify mismatched aspect phrases within a caption, determine the aspect's class, and propose corrections for an image-text pair that may contain between 0 and 3 mismatches. To evaluate the models' performance on this new task, we propose a new evaluation metric named ITM-IoU for which our experiments show a high correlation to human evaluation. In addition, we also provide a comprehensive experimental analysis of existing mainstream VLMs, including fully supervised learning and in-context learning settings. We have found that models trained on FineMatch demonstrate enhanced proficiency in detecting fine-grained text and image mismatches. Moreover, models (e.g., GPT-4V, Gemini Pro Vision) with strong abilities to perform multimodal in-context learning are not as skilled at fine-grained compositional image and text matching analysis. With FineMatch, we are able to build a system for text-to-image generation hallucination detection and correction.

Title: Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities

• Authors: Siyin Wang, Chao-Han Huck Yang, Ji Wu, Chao Zhang
• Subjects: cs.CL, cs.AI, cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.14716
• Pdf URL: https://arxiv.org/pdf/2404.14716
• Copy Paste: [[2404.14716]] Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities(https://arxiv.org/abs/2404.14716)
• Keywords: in-context
• Abstract: Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue history without any model parameter update. Despite such convenience, the performance of ICL heavily depends on the quality of the in-context examples presented, which makes the in-context example selection approach a critical choice. This paper proposes a novel Bayesian in-Context example Selection method (ByCS) for ICL. Extending the inference probability conditioned on in-context examples based on Bayes' theorem, ByCS focuses on the inverse inference conditioned on test input. Following the assumption that accurate inverse inference probability (likelihood) will result in accurate inference probability (posterior), in-context examples are selected based on their inverse inference results. Diverse and extensive cross-tasking and cross-modality experiments are performed with speech, text, and image examples. Experimental results show the efficacy and robustness of our ByCS method on various models, tasks and modalities.

Title: Skip the Benchmark: Generating System-Level High-Level Synthesis Data using Generative Machine Learning

• Authors: Yuchao Liao, Tosiron Adegbija, Roman Lysecky, Ravi Tandon
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2404.14754
• Pdf URL: https://arxiv.org/pdf/2404.14754
• Copy Paste: [[2404.14754]] Skip the Benchmark: Generating System-Level High-Level Synthesis Data using Generative Machine Learning(https://arxiv.org/abs/2404.14754)
• Keywords: generative
• Abstract: High-Level Synthesis (HLS) Design Space Exploration (DSE) is a widely accepted approach for efficiently exploring Pareto-optimal and optimal hardware solutions during the HLS process. Several HLS benchmarks and datasets are available for the research community to evaluate their methodologies. Unfortunately, these resources are limited and may not be sufficient for complex, multi-component system-level explorations. Generating new data using existing HLS benchmarks can be cumbersome, given the expertise and time required to effectively generate data for different HLS designs and directives. As a result, synthetic data has been used in prior work to evaluate system-level HLS DSE. However, the fidelity of the synthetic data to real data is often unclear, leading to uncertainty about the quality of system-level HLS DSE. This paper proposes a novel approach, called Vaegan, that employs generative machine learning to generate synthetic data that is robust enough to support complex system-level HLS DSE experiments that would be unattainable with only the currently available data. We explore and adapt a Variational Autoencoder (VAE) and Generative Adversarial Network (GAN) for this task and evaluate our approach using state-of-the-art datasets and metrics. We compare our approach to prior works and show that Vaegan effectively generates synthetic HLS data that closely mirrors the ground truth's distribution.

Title: Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusion

• Authors: Hongyu Chen, Yiqi Gao, Min Zhou, Peng Wang, Xubin Li, Tiezheng Ge, Bo Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14768
• Pdf URL: https://arxiv.org/pdf/2404.14768
• Copy Paste: [[2404.14768]] Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusion(https://arxiv.org/abs/2404.14768)
• Keywords: diffusion
• Abstract: Recently, integrating visual controls into text-to-image~(T2I) models, such as ControlNet method, has received significant attention for finer control capabilities. While various training-free methods make efforts to enhance prompt following in T2I models, the issue with visual control is still rarely studied, especially in the scenario that visual controls are misaligned with text prompts. In this paper, we address the challenge of ``Prompt Following With Visual Control" and propose a training-free approach named Mask-guided Prompt Following (MGPF). Object masks are introduced to distinct aligned and misaligned parts of visual controls and prompts. Meanwhile, a network, dubbed as Masked ControlNet, is designed to utilize these object masks for object generation in the misaligned visual control region. Further, to improve attribute matching, a simple yet efficient loss is designed to align the attention maps of attributes with object regions constrained by ControlNet and object masks. The efficacy and superiority of MGPF are validated through comprehensive quantitative and qualitative experiments.

Title: Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning

• Authors: Wenjin Hou, Shiming Chen, Shuhuang Chen, Ziming Hong, Yan Wang, Xuetao Feng, Salman Khan, Fahad Shahbaz Khan, Xinge You
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14808
• Pdf URL: https://arxiv.org/pdf/2404.14808
• Copy Paste: [[2404.14808]] Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning(https://arxiv.org/abs/2404.14808)
• Keywords: generative
• Abstract: Generative Zero-shot learning (ZSL) learns a generator to synthesize visual samples for unseen classes, which is an effective way to advance ZSL. However, existing generative methods rely on the conditions of Gaussian noise and the predefined semantic prototype, which limit the generator only optimized on specific seen classes rather than characterizing each visual instance, resulting in poor generalizations (\textit{e.g.}, overfitting to seen classes). To address this issue, we propose a novel Visual-Augmented Dynamic Semantic prototype method (termed VADS) to boost the generator to learn accurate semantic-visual mapping by fully exploiting the visual-augmented knowledge into semantic conditions. In detail, VADS consists of two modules: (1) Visual-aware Domain Knowledge Learning module (VDKL) learns the local bias and global prior of the visual features (referred to as domain visual knowledge), which replace pure Gaussian noise to provide richer prior noise information; (2) Vision-Oriented Semantic Updation module (VOSU) updates the semantic prototype according to the visual representations of the samples. Ultimately, we concatenate their output as a dynamic semantic prototype, which serves as the condition of the generator. Extensive experiments demonstrate that our VADS achieves superior CZSL and GZSL performances on three prominent datasets and outperforms other state-of-the-art methods with averaging increases by 6.4\%, 5.9\% and 4.2\% on SUN, CUB and AWA2, respectively.

Title: A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications

• Authors: Wenbo Shang, Xin Huang
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2404.14809
• Pdf URL: https://arxiv.org/pdf/2404.14809
• Copy Paste: [[2404.14809]] A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications(https://arxiv.org/abs/2404.14809)
• Keywords: generative
• Abstract: A graph is a fundamental data model to represent various entities and their complex relationships in society and nature, such as social networks, transportation networks, financial networks, and biomedical systems. Recently, large language models (LLMs) have showcased a strong generalization ability to handle various NLP and multi-mode tasks to answer users' arbitrary questions and specific-domain content generation. Compared with graph learning models, LLMs enjoy superior advantages in addressing the challenges of generalizing graph tasks by eliminating the need for training graph learning models and reducing the cost of manual annotation. In this survey, we conduct a comprehensive investigation of existing LLM studies on graph data, which summarizes the relevant graph analytics tasks solved by advanced LLM models and points out the existing remaining challenges and future directions. Specifically, we study the key problems of LLM-based generative graph analytics (LLM-GGA) with three categories: LLM-based graph query processing (LLM-GQP), LLM-based graph inference and learning (LLM-GIL), and graph-LLM-based applications. LLM-GQP focuses on an integration of graph analytics techniques and LLM prompts, including graph understanding and knowledge graph (KG) based augmented retrieval, while LLM-GIL focuses on learning and reasoning over graphs, including graph learning, graph-formed reasoning and graph representation. We summarize the useful prompts incorporated into LLM to handle different graph downstream tasks. Moreover, we give a summary of LLM model evaluation, benchmark datasets/tasks, and a deep pro and cons analysis of LLM models. We also explore open problems and future directions in this exciting interdisciplinary research area of LLMs and graph analytics.

Title: CoProNN: Concept-based Prototypical Nearest Neighbors for Explaining Vision Models

• Authors: Teodor Chiaburu, Frank Haußer, Felix Bießmann
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14830
• Pdf URL: https://arxiv.org/pdf/2404.14830
• Copy Paste: [[2404.14830]] CoProNN: Concept-based Prototypical Nearest Neighbors for Explaining Vision Models(https://arxiv.org/abs/2404.14830)
• Keywords: generative
• Abstract: Mounting evidence in explainability for artificial intelligence (XAI) research suggests that good explanations should be tailored to individual tasks and should relate to concepts relevant to the task. However, building task specific explanations is time consuming and requires domain expertise which can be difficult to integrate into generic XAI methods. A promising approach towards designing useful task specific explanations with domain experts is based on compositionality of semantic concepts. Here, we present a novel approach that enables domain experts to quickly create concept-based explanations for computer vision tasks intuitively via natural language. Leveraging recent progress in deep generative methods we propose to generate visual concept-based prototypes via text-to-image methods. These prototypes are then used to explain predictions of computer vision models via a simple k-Nearest-Neighbors routine. The modular design of CoProNN is simple to implement, it is straightforward to adapt to novel tasks and allows for replacing the classification and text-to-image models as more powerful models are released. The approach can be evaluated offline against the ground-truth of predefined prototypes that can be easily communicated also to domain experts as they are based on visual concepts. We show that our strategy competes very well with other concept-based XAI approaches on coarse grained image classification tasks and may even outperform those methods on more demanding fine grained tasks. We demonstrate the effectiveness of our method for human-machine collaboration settings in qualitative and quantitative user studies. All code and experimental data can be found in our GitHub $\href{https://github.com/TeodorChiaburu/beexplainable}{repository}$.

Title: Domain adaptive pose estimation via multi-level alignment

• Authors: Yugan Chen, Lin Zhao, Yalong Xu, Honglei Zu, Xiaoqi An, Guangyu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14885
• Pdf URL: https://arxiv.org/pdf/2404.14885
• Copy Paste: [[2404.14885]] Domain adaptive pose estimation via multi-level alignment(https://arxiv.org/abs/2404.14885)
• Keywords: self-supervised
• Abstract: Domain adaptive pose estimation aims to enable deep models trained on source domain (synthesized) datasets produce similar results on the target domain (real-world) datasets. The existing methods have made significant progress by conducting image-level or feature-level alignment. However, only aligning at a single level is not sufficient to fully bridge the domain gap and achieve excellent domain adaptive results. In this paper, we propose a multi-level domain adaptation aproach, which aligns different domains at the image, feature, and pose levels. Specifically, we first utilize image style transer to ensure that images from the source and target domains have a similar distribution. Subsequently, at the feature level, we employ adversarial training to make the features from the source and target domains preserve domain-invariant characeristics as much as possible. Finally, at the pose level, a self-supervised approach is utilized to enable the model to learn diverse knowledge, implicitly addressing the domain gap. Experimental results demonstrate that significant imrovement can be achieved by the proposed multi-level alignment method in pose estimation, which outperforms previous state-of-the-art in human pose by up to 2.4% and animal pose estimation by up to 3.1% for dogs and 1.4% for sheep.

Title: DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition

• Authors: Haozhe Cheng, Cheng Ju, Haicheng Wang, Jinxiang Liu, Mengting Chen, Qiang Hu, Xiaoyun Zhang, Yanfeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14890
• Pdf URL: https://arxiv.org/pdf/2404.14890
• Copy Paste: [[2404.14890]] DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition(https://arxiv.org/abs/2404.14890)
• Keywords: generative
• Abstract: As one of the fundamental video tasks in computer vision, Open-Vocabulary Action Recognition (OVAR) recently gains increasing attention, with the development of vision-language pre-trainings. To enable generalization of arbitrary classes, existing methods treat class labels as text descriptions, then formulate OVAR as evaluating embedding similarity between visual samples and textual classes. However, one crucial issue is completely ignored: the class descriptions given by users may be noisy, e.g., misspellings and typos, limiting the real-world practicality of vanilla OVAR. To fill the research gap, this paper pioneers to evaluate existing methods by simulating multi-level noises of various types, and reveals their poor robustness. To tackle the noisy OVAR task, we further propose one novel DENOISER framework, covering two parts: generation and discrimination. Concretely, the generative part denoises noisy class-text names via one decoding process, i.e., propose text candidates, then utilize inter-modal and intra-modal information to vote for the best. At the discriminative part, we use vanilla OVAR models to assign visual samples to class-text names, thus obtaining more semantics. For optimization, we alternately iterate between generative and discriminative parts for progressive refinements. The denoised text classes help OVAR models classify visual samples more accurately; in return, classified visual samples help better denoising. On three datasets, we carry out extensive experiments to show our superior robustness, and thorough ablations to dissect the effectiveness of each component.

Title: Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimation

• Authors: Hoang Chuong Nguyen, Tianyu Wang, Jose M. Alvarez, Miaomiao Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14908
• Pdf URL: https://arxiv.org/pdf/2404.14908
• Copy Paste: [[2404.14908]] Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimation(https://arxiv.org/abs/2404.14908)
• Keywords: self-supervised
• Abstract: This paper focuses on self-supervised monocular depth estimation in dynamic scenes trained on monocular videos. Existing methods jointly estimate pixel-wise depth and motion, relying mainly on an image reconstruction loss. Dynamic regions1 remain a critical challenge for these methods due to the inherent ambiguity in depth and motion estimation, resulting in inaccurate depth estimation. This paper proposes a self-supervised training framework exploiting pseudo depth labels for dynamic regions from training data. The key contribution of our framework is to decouple depth estimation for static and dynamic regions of images in the training data. We start with an unsupervised depth estimation approach, which provides reliable depth estimates for static regions and motion cues for dynamic regions and allows us to extract moving object information at the instance level. In the next stage, we use an object network to estimate the depth of those moving objects assuming rigid motions. Then, we propose a new scale alignment module to address the scale ambiguity between estimated depths for static and dynamic regions. We can then use the depth labels generated to train an end-to-end depth estimation network and improve its performance. Extensive experiments on the Cityscapes and KITTI datasets show that our self-training strategy consistently outperforms existing self/unsupervised depth estimation methods.

Title: Fin-Fed-OD: Federated Outlier Detection on Financial Tabular Data

• Authors: Dayananda Herurkar, Sebastian Palacio, Ahmed Anwar, Joern Hees, Andreas Dengel
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14933
• Pdf URL: https://arxiv.org/pdf/2404.14933
• Copy Paste: [[2404.14933]] Fin-Fed-OD: Federated Outlier Detection on Financial Tabular Data(https://arxiv.org/abs/2404.14933)
• Keywords: anomaly
• Abstract: Anomaly detection in real-world scenarios poses challenges due to dynamic and often unknown anomaly distributions, requiring robust methods that operate under an open-world assumption. This challenge is exacerbated in practical settings, where models are employed by private organizations, precluding data sharing due to privacy and competitive concerns. Despite potential benefits, the sharing of anomaly information across organizations is restricted. This paper addresses the question of enhancing outlier detection within individual organizations without compromising data confidentiality. We propose a novel method leveraging representation learning and federated learning techniques to improve the detection of unknown anomalies. Specifically, our approach utilizes latent representations obtained from client-owned autoencoders to refine the decision boundary of inliers. Notably, only model parameters are shared between organizations, preserving data privacy. The efficacy of our proposed method is evaluated on two standard financial tabular datasets and an image dataset for anomaly detection in a distributed setting. The results demonstrate a strong improvement in the classification of unknown outliers during the inference phase for each organization's model.

Title: Delayed Bottlenecking: Alleviating Forgetting in Pre-trained Graph Neural Networks

• Authors: Zhe Zhao, Pengkun Wang, Xu Wang, Haibin Wen, Xiaolong Xie, Zhengyang Zhou, Qingfu Zhang, Yang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14941
• Pdf URL: https://arxiv.org/pdf/2404.14941
• Copy Paste: [[2404.14941]] Delayed Bottlenecking: Alleviating Forgetting in Pre-trained Graph Neural Networks(https://arxiv.org/abs/2404.14941)
• Keywords: self-supervised
• Abstract: Pre-training GNNs to extract transferable knowledge and apply it to downstream tasks has become the de facto standard of graph representation learning. Recent works focused on designing self-supervised pre-training tasks to extract useful and universal transferable knowledge from large-scale unlabeled data. However, they have to face an inevitable question: traditional pre-training strategies that aim at extracting useful information about pre-training tasks, may not extract all useful information about the downstream task. In this paper, we reexamine the pre-training process within traditional pre-training and fine-tuning frameworks from the perspective of Information Bottleneck (IB) and confirm that the forgetting phenomenon in pre-training phase may cause detrimental effects on downstream tasks. Therefore, we propose a novel \underline{D}elayed \underline{B}ottlenecking \underline{P}re-training (DBP) framework which maintains as much as possible mutual information between latent representations and training data during pre-training phase by suppressing the compression operation and delays the compression operation to fine-tuning phase to make sure the compression can be guided with labeled fine-tuning data and downstream tasks. To achieve this, we design two information control objectives that can be directly optimized and further integrate them into the actual model design. Extensive experiments on both chemistry and biology domains demonstrate the effectiveness of DBP.

Title: Streamlining the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Model

• Authors: Ziqi Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14951
• Pdf URL: https://arxiv.org/pdf/2404.14951
• Copy Paste: [[2404.14951]] Streamlining the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Model(https://arxiv.org/abs/2404.14951)
• Keywords: generative
• Abstract: Learning-based image stitching techniques typically involve three distinct stages: registration, fusion, and rectangling. These stages are often performed sequentially, each trained independently, leading to potential cascading error propagation and complex parameter tuning challenges. In rethinking the mathematical modeling of the fusion and rectangling stages, we discovered that these processes can be effectively combined into a single, variety-intensity inpainting problem. Therefore, we propose the Simple and Robust Stitcher (SRStitcher), an efficient training-free image stitching method that merges the fusion and rectangling stages into a unified model. By employing the weighted mask and large-scale generative model, SRStitcher can solve the fusion and rectangling problems in a single inference, without additional training or fine-tuning of other models. Our method not only simplifies the stitching pipeline but also enhances fault tolerance towards misregistration errors. Extensive experiments demonstrate that SRStitcher outperforms state-of-the-art (SOTA) methods in both quantitative assessments and qualitative evaluations. The code is released at https://github.com/yayoyo66/SRStitcher

Title: $\texttt{MiniMol}$: A Parameter-Efficient Foundation Model for Molecular Learning

• Authors: Kerstin Kläser, Błażej Banaszewski, Samuel Maddrell-Mander, Callum McLean, Luis Müller, Ali Parviz, Shenyang Huang, Andrew Fitzgibbon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14986
• Pdf URL: https://arxiv.org/pdf/2404.14986
• Copy Paste: [[2404.14986]] $\texttt{MiniMol}$: A Parameter-Efficient Foundation Model for Molecular Learning(https://arxiv.org/abs/2404.14986)
• Keywords: foundation model
• Abstract: In biological tasks, data is rarely plentiful as it is generated from hard-to-gather measurements. Therefore, pre-training foundation models on large quantities of available data and then transfer to low-data downstream tasks is a promising direction. However, how to design effective foundation models for molecular learning remains an open question, with existing approaches typically focusing on models with large parameter capacities. In this work, we propose $\texttt{MiniMol}$, a foundational model for molecular learning with 10 million parameters. $\texttt{MiniMol}$ is pre-trained on a mix of roughly 3300 sparsely defined graph- and node-level tasks of both quantum and biological nature. The pre-training dataset includes approximately 6 million molecules and 500 million labels. To demonstrate the generalizability of $\texttt{MiniMol}$ across tasks, we evaluate it on downstream tasks from the Therapeutic Data Commons (TDC) ADMET group showing significant improvements over the prior state-of-the-art foundation model across 17 tasks. $\texttt{MiniMol}$ will be a public and open-sourced model for future research.

Title: Interpreting COVID Lateral Flow Tests' Results with Foundation Models

• Authors: Stuti Pandey, Josh Myers-Dean, Jarek Reynolds, Danna Gurari
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.14990
• Pdf URL: https://arxiv.org/pdf/2404.14990
• Copy Paste: [[2404.14990]] Interpreting COVID Lateral Flow Tests' Results with Foundation Models(https://arxiv.org/abs/2404.14990)
• Keywords: foundation model
• Abstract: Lateral flow tests (LFTs) enable rapid, low-cost testing for health conditions including Covid, pregnancy, HIV, and malaria. Automated readers of LFT results can yield many benefits including empowering blind people to independently learn about their health and accelerating data entry for large-scale monitoring (e.g., for pandemics such as Covid) by using only a single photograph per LFT test. Accordingly, we explore the abilities of modern foundation vision language models (VLMs) in interpreting such tests. To enable this analysis, we first create a new labeled dataset with hierarchical segmentations of each LFT test and its nested test result window. We call this dataset LFT-Grounding. Next, we benchmark eight modern VLMs in zero-shot settings for analyzing these images. We demonstrate that current VLMs frequently fail to correctly identify the type of LFT test, interpret the test results, locate the nested result window of the LFT tests, and recognize LFT tests when they partially obfuscated. To facilitate community-wide progress towards automated LFT reading, we publicly release our dataset at https://iamstuti.github.io/lft_grounding_foundation_models/.

Title: Comparison of Current Approaches to Lemmatization: A Case Study in Estonian

• Authors: Aleksei Dorkin, Kairit Sirts
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15003
• Pdf URL: https://arxiv.org/pdf/2404.15003
• Copy Paste: [[2404.15003]] Comparison of Current Approaches to Lemmatization: A Case Study in Estonian(https://arxiv.org/abs/2404.15003)
• Keywords: generative
• Abstract: This study evaluates three different lemmatization approaches to Estonian -- Generative character-level models, Pattern-based word-level classification models, and rule-based morphological analysis. According to our experiments, a significantly smaller Generative model consistently outperforms the Pattern-based classification model based on EstBERT. Additionally, we observe a relatively small overlap in errors made by all three models, indicating that an ensemble of different approaches could lead to improvements.

Title: OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving

• Authors: Guoqing Wang, Zhongdao Wang, Pin Tang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, Chao Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15014
• Pdf URL: https://arxiv.org/pdf/2404.15014
• Copy Paste: [[2404.15014]] OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving(https://arxiv.org/abs/2404.15014)
• Keywords: diffusion, generative
• Abstract: Existing solutions for 3D semantic occupancy prediction typically treat the task as a one-shot 3D voxel-wise segmentation perception problem. These discriminative methods focus on learning the mapping between the inputs and occupancy map in a single step, lacking the ability to gradually refine the occupancy map and the reasonable scene imaginative capacity to complete the local regions somewhere. In this paper, we introduce OccGen, a simple yet powerful generative perception model for the task of 3D semantic occupancy prediction. OccGen adopts a ''noise-to-occupancy'' generative paradigm, progressively inferring and refining the occupancy map by predicting and eliminating noise originating from a random Gaussian distribution. OccGen consists of two main components: a conditional encoder that is capable of processing multi-modal inputs, and a progressive refinement decoder that applies diffusion denoising using the multi-modal features as conditions. A key insight of this generative pipeline is that the diffusion denoising process is naturally able to model the coarse-to-fine refinement of the dense 3D occupancy map, therefore producing more detailed predictions. Extensive experiments on several occupancy benchmarks demonstrate the effectiveness of the proposed method compared to the state-of-the-art methods. For instance, OccGen relatively enhances the mIoU by 9.5%, 6.3%, and 13.3% on nuScenes-Occupancy dataset under the muli-modal, LiDAR-only, and camera-only settings, respectively. Moreover, as a generative perception model, OccGen exhibits desirable properties that discriminative models cannot achieve, such as providing uncertainty estimates alongside its multiple-step predictions.

Title: IPAD: Industrial Process Anomaly Detection Dataset

• Authors: Jinfan Liu, Yichao Yan, Junjie Li, Weiming Zhao, Pengzhi Chu, Xingdong Sheng, Yunhui Liu, Xiaokang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15033
• Pdf URL: https://arxiv.org/pdf/2404.15033
• Copy Paste: [[2404.15033]] IPAD: Industrial Process Anomaly Detection Dataset(https://arxiv.org/abs/2404.15033)
• Keywords: anomaly
• Abstract: Video anomaly detection (VAD) is a challenging task aiming to recognize anomalies in video frames, and existing large-scale VAD researches primarily focus on road traffic and human activity scenes. In industrial scenes, there are often a variety of unpredictable anomalies, and the VAD method can play a significant role in these scenarios. However, there is a lack of applicable datasets and methods specifically tailored for industrial production scenarios due to concerns regarding privacy and security. To bridge this gap, we propose a new dataset, IPAD, specifically designed for VAD in industrial scenarios. The industrial processes in our dataset are chosen through on-site factory research and discussions with engineers. This dataset covers 16 different industrial devices and contains over 6 hours of both synthetic and real-world video footage. Moreover, we annotate the key feature of the industrial process, ie, periodicity. Based on the proposed dataset, we introduce a period memory module and a sliding window inspection mechanism to effectively investigate the periodic information in a basic reconstruction model. Our framework leverages LoRA adapter to explore the effective migration of pretrained models, which are initially trained using synthetic data, into real-world scenarios. Our proposed dataset and method will fill the gap in the field of industrial video anomaly detection and drive the process of video understanding tasks as well as smart factory deployment.

Title: Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models

• Authors: Jingyao Xu, Yuetong Lu, Yandong Li, Siyang Lu, Dongdong Wang, Xiang Wei
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15081
• Pdf URL: https://arxiv.org/pdf/2404.15081
• Copy Paste: [[2404.15081]] Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models(https://arxiv.org/abs/2404.15081)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DMs) embark a new era of generative modeling and offer more opportunities for efficient generating high-quality and realistic data samples. However, their widespread use has also brought forth new challenges in model security, which motivates the creation of more effective adversarial attackers on DMs to understand its vulnerability. We propose CAAT, a simple but generic and efficient approach that does not require costly training to effectively fool latent diffusion models (LDMs). The approach is based on the observation that cross-attention layers exhibits higher sensitivity to gradient change, allowing for leveraging subtle perturbations on published images to significantly corrupt the generated images. We show that a subtle perturbation on an image can significantly impact the cross-attention layers, thus changing the mapping between text and image during the fine-tuning of customized diffusion models. Extensive experiments demonstrate that CAAT is compatible with diverse diffusion models and outperforms baseline attack methods in a more effective (more noise) and efficient (twice as fast as Anti-DreamBooth and Mist) manner.

Title: Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation

• Authors: Xun Wu, Shaohan Huang, Furu Wei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.15100
• Pdf URL: https://arxiv.org/pdf/2404.15100
• Copy Paste: [[2404.15100]] Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation(https://arxiv.org/abs/2404.15100)
• Keywords: generative
• Abstract: Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, enhancing the alignment between generated images and textual prompts. Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration. To address these challenges and promote the alignment of generative models through instruction tuning, we leverage multimodal large language models to create VisionPrefer, a high-quality and fine-grained preference dataset that captures multiple preference aspects. We aggregate feedback from AI annotators across four aspects: prompt-following, aesthetic, fidelity, and harmlessness to construct VisionPrefer. To validate the effectiveness of VisionPrefer, we train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators. Furthermore, we use two reinforcement learning methods to supervised fine-tune generative models to evaluate the performance of VisionPrefer, and extensive experimental results demonstrate that VisionPrefer significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic, and generalizes better than previous human-preference metrics across various image distributions. Moreover, VisionPrefer indicates that the integration of AI-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in vision generative models.

Title: MedDr: Diagnosis-Guided Bootstrapping for Large-Scale Medical Vision-Language Learning

• Authors: Sunan He, Yuxiang Nie, Zhixuan Chen, Zhiyuan Cai, Hongmei Wang, Shu Yang, Hao Chen
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.15127
• Pdf URL: https://arxiv.org/pdf/2404.15127
• Copy Paste: [[2404.15127]] MedDr: Diagnosis-Guided Bootstrapping for Large-Scale Medical Vision-Language Learning(https://arxiv.org/abs/2404.15127)
• Keywords: foundation model
• Abstract: The rapid advancement of large-scale vision-language models has showcased remarkable capabilities across various tasks. However, the lack of extensive and high-quality image-text data in medicine has greatly hindered the development of large-scale medical vision-language models. In this work, we present a diagnosis-guided bootstrapping strategy that exploits both image and label information to construct vision-language datasets. Based on the constructed dataset, we developed MedDr, a generalist foundation model for healthcare capable of handling diverse medical data modalities, including radiology, pathology, dermatology, retinography, and endoscopy. Moreover, during inference, we propose a simple but effective retrieval-augmented medical diagnosis strategy, which enhances the model's generalization ability. Extensive experiments on visual question answering, medical report generation, and medical image diagnosis demonstrate the superiority of our method.

Title: CutDiffusion: A Simple, Fast, Cheap, and Strong Diffusion Extrapolation Method

• Authors: Mingbao Lin, Zhihang Lin, Wengyi Zhan, Liujuan Cao, Rongrong Ji
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15141
• Pdf URL: https://arxiv.org/pdf/2404.15141
• Copy Paste: [[2404.15141]] CutDiffusion: A Simple, Fast, Cheap, and Strong Diffusion Extrapolation Method(https://arxiv.org/abs/2404.15141)
• Keywords: diffusion
• Abstract: Transforming large pre-trained low-resolution diffusion models to cater to higher-resolution demands, i.e., diffusion extrapolation, significantly improves diffusion adaptability. We propose tuning-free CutDiffusion, aimed at simplifying and accelerating the diffusion extrapolation process, making it more affordable and improving performance. CutDiffusion abides by the existing patch-wise extrapolation but cuts a standard patch diffusion process into an initial phase focused on comprehensive structure denoising and a subsequent phase dedicated to specific detail refinement. Comprehensive experiments highlight the numerous almighty advantages of CutDiffusion: (1) simple method construction that enables a concise higher-resolution diffusion process without third-party engagement; (2) fast inference speed achieved through a single-step higher-resolution diffusion process, and fewer inference patches required; (3) cheap GPU cost resulting from patch-wise inference and fewer patches during the comprehensive structure denoising; (4) strong generation performance, stemming from the emphasis on specific detail refinement.

Title: Do not think pink elephant!

• Authors: Kyomin Hwang, Suyoung Kim, JunHoo Lee, Nojun Kwak
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.15154
• Pdf URL: https://arxiv.org/pdf/2404.15154
• Copy Paste: [[2404.15154]] Do not think pink elephant!(https://arxiv.org/abs/2404.15154)
• Keywords: diffusion
• Abstract: Large Models (LMs) have heightened expectations for the potential of general AI as they are akin to human intelligence. This paper shows that recent large models such as Stable Diffusion and DALL-E3 also share the vulnerability of human intelligence, namely the "white bear phenomenon". We investigate the causes of the white bear phenomenon by analyzing their representation space. Based on this analysis, we propose a simple prompt-based attack method, which generates figures prohibited by the LM provider's policy. To counter these attacks, we introduce prompt-based defense strategies inspired by cognitive therapy techniques, successfully mitigating attacks by up to 48.22\%.

Title: Adaptive Collaboration Strategy for LLMs in Medical Decision Making

• Authors: Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, Hae Won Park
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15155
• Pdf URL: https://arxiv.org/pdf/2404.15155
• Copy Paste: [[2404.15155]] Adaptive Collaboration Strategy for LLMs in Medical Decision Making(https://arxiv.org/abs/2404.15155)
• Keywords: foundation model
• Abstract: Foundation models have become invaluable in advancing the medical field. Despite their promise, the strategic deployment of LLMs for effective utility in complex medical tasks remains an open question. Our novel framework, Medical Decision-making Agents (MDAgents) aims to address this gap by automatically assigning the effective collaboration structure for LLMs. Assigned solo or group collaboration structure is tailored to the complexity of the medical task at hand, emulating real-world medical decision making processes. We evaluate our framework and baseline methods with state-of-the-art LLMs across a suite of challenging medical benchmarks: MedQA, MedMCQA, PubMedQA, DDXPlus, PMC-VQA, Path-VQA, and MedVidQA, achieving the best performance in 5 out of 7 benchmarks that require an understanding of multi-modal medical reasoning. Ablation studies reveal that MDAgents excels in adapting the number of collaborating agents to optimize efficiency and accuracy, showcasing its robustness in diverse scenarios. We also explore the dynamics of group consensus, offering insights into how collaborative agents could behave in complex clinical team dynamics. Our code can be found at https://github.com/mitmedialab/MDAgents.

Title: Combating Missing Modalities in Egocentric Videos at Test Time

• Authors: Merey Ramazanova, Alejandro Pardo, Bernard Ghanem, Motasem Alfarra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15161
• Pdf URL: https://arxiv.org/pdf/2404.15161
• Copy Paste: [[2404.15161]] Combating Missing Modalities in Egocentric Videos at Test Time(https://arxiv.org/abs/2404.15161)
• Keywords: self-supervised
• Abstract: Understanding videos that contain multiple modalities is crucial, especially in egocentric videos, where combining various sensory inputs significantly improves tasks like action recognition and moment localization. However, real-world applications often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues. Current methods, while effective, often necessitate retraining the model entirely to handle missing modalities, making them computationally intensive, particularly with large training datasets. In this study, we propose a novel approach to address this issue at test time without requiring retraining. We frame the problem as a test-time adaptation task, where the model adjusts to the available unlabeled data at test time. Our method, MiDl~(Mutual information with self-Distillation), encourages the model to be insensitive to the specific modality source present during testing by minimizing the mutual information between the prediction and the available modality. Additionally, we incorporate self-distillation to maintain the model's original performance when both modalities are available. MiDl represents the first self-supervised, online solution for handling missing modalities exclusively at test time. Through experiments with various pretrained models and datasets, MiDl demonstrates substantial performance improvement without the need for retraining.

Title: Adaptive Mixed-Scale Feature Fusion Network for Blind AI-Generated Image Quality Assessment

• Authors: Tianwei Zhou, Songbai Tan, Wei Zhou, Yu Luo, Yuan-Gen Wang, Guanghui Yue
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.15163
• Pdf URL: https://arxiv.org/pdf/2404.15163
• Copy Paste: [[2404.15163]] Adaptive Mixed-Scale Feature Fusion Network for Blind AI-Generated Image Quality Assessment(https://arxiv.org/abs/2404.15163)
• Keywords: generative
• Abstract: With the increasing maturity of the text-to-image and image-to-image generative models, AI-generated images (AGIs) have shown great application potential in advertisement, entertainment, education, social media, etc. Although remarkable advancements have been achieved in generative models, very few efforts have been paid to design relevant quality assessment models. In this paper, we propose a novel blind image quality assessment (IQA) network, named AMFF-Net, for AGIs. AMFF-Net evaluates AGI quality from three dimensions, i.e., "visual quality", "authenticity", and "consistency". Specifically, inspired by the characteristics of the human visual system and motivated by the observation that "visual quality" and "authenticity" are characterized by both local and global aspects, AMFF-Net scales the image up and down and takes the scaled images and original-sized image as the inputs to obtain multi-scale features. After that, an Adaptive Feature Fusion (AFF) block is used to adaptively fuse the multi-scale features with learnable weights. In addition, considering the correlation between the image and prompt, AMFF-Net compares the semantic features from text encoder and image encoder to evaluate the text-to-image alignment. We carry out extensive experiments on three AGI quality assessment databases, and the experimental results show that our AMFF-Net obtains better performance than nine state-of-the-art blind IQA methods. The results of ablation experiments further demonstrate the effectiveness of the proposed multi-scale input strategy and AFF block.

Title: Pixels and Predictions: Potential of GPT-4V in Meteorological Imagery Analysis and Forecast Communication

• Authors: John R. Lawson, Montgomery L. Flora, Kevin H. Goebbert, Seth N. Lyman, Corey K. Potvin, David M. Schultz, Adam J. Stepanek, Joseph E. Trujillo-Falcón
• Subjects: cs.CL, cs.AI, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2404.15166
• Pdf URL: https://arxiv.org/pdf/2404.15166
• Copy Paste: [[2404.15166]] Pixels and Predictions: Potential of GPT-4V in Meteorological Imagery Analysis and Forecast Communication(https://arxiv.org/abs/2404.15166)
• Keywords: generative
• Abstract: Generative AI, such as OpenAI's GPT-4V large-language model, has rapidly entered mainstream discourse. Novel capabilities in image processing and natural-language communication may augment existing forecasting methods. Large language models further display potential to better communicate weather hazards in a style honed for diverse communities and different languages. This study evaluates GPT-4V's ability to interpret meteorological charts and communicate weather hazards appropriately to the user, despite challenges of hallucinations, where generative AI delivers coherent, confident, but incorrect responses. We assess GPT-4V's competence via its web interface ChatGPT in two tasks: (1) generating a severe-weather outlook from weather-chart analysis and conducting self-evaluation, revealing an outlook that corresponds well with a Storm Prediction Center human-issued forecast; and (2) producing hazard summaries in Spanish and English from weather charts. Responses in Spanish, however, resemble direct (not idiomatic) translations from English to Spanish, yielding poorly translated summaries that lose critical idiomatic precision required for optimal communication. Our findings advocate for cautious integration of tools like GPT-4V in meteorology, underscoring the necessity of human oversight and development of trustworthy, explainable AI.

Title: Lossless and Near-Lossless Compression for Foundation Models

• Authors: Moshik Hershcovitch, Leshem Choshen, Andrew Wood, Ilias Enmouri, Peter Chin, Swaminathan Sundararaman, Danny Harnik
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2404.15198
• Pdf URL: https://arxiv.org/pdf/2404.15198
• Copy Paste: [[2404.15198]] Lossless and Near-Lossless Compression for Foundation Models(https://arxiv.org/abs/2404.15198)
• Keywords: foundation model
• Abstract: With the growth of model sizes and scale of their deployment, their sheer size burdens the infrastructure requiring more network and more storage to accommodate these. While there is a vast literature about reducing model sizes, we investigate a more traditional type of compression -- one that compresses the model to a smaller form and is coupled with a decompression algorithm that returns it to its original size -- namely lossless compression. Somewhat surprisingly, we show that such lossless compression can gain significant network and storage reduction on popular models, at times reducing over $50\%$ of the model size. We investigate the source of model compressibility, introduce compression variants tailored for models and categorize models to compressibility groups. We also introduce a tunable lossy compression technique that can further reduce size even on the less compressible models with little to no effect on the model accuracy. We estimate that these methods could save over an ExaByte per month of network traffic downloaded from a large model hub like HuggingFace.

Title: Towards Large-Scale Training of Pathology Foundation Models

• Authors: kaiko.ai, Nanne Aben, Edwin D. de Jong, Ioannis Gatopoulos, Nicolas Känzig, Mikhail Karasikov, Axel Lagré, Roman Moser, Joost van Doorn, Fei Tang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.15217
• Pdf URL: https://arxiv.org/pdf/2404.15217
• Copy Paste: [[2404.15217]] Towards Large-Scale Training of Pathology Foundation Models(https://arxiv.org/abs/2404.15217)
• Keywords: self-supervised, foundation model
• Abstract: Driven by the recent advances in deep learning methods and, in particular, by the development of modern self-supervised learning algorithms, increased interest and efforts have been devoted to build foundation models (FMs) for medical images. In this work, we present our scalable training pipeline for large pathology imaging data, and a comprehensive analysis of various hyperparameter choices and training techniques for building pathology FMs. We release and make publicly available the first batch of our pathology FMs (https://github.com/kaiko-ai/towards_large_pathology_fms) trained on open-access TCGA whole slide images, a commonly used collection of pathology images. The experimental evaluation shows that our models reach state-of-the-art performance on various patch-level downstream tasks, ranging from breast cancer subtyping to colorectal nuclear segmentation. Finally, to unify the evaluation approaches used in the field and to simplify future comparisons of different FMs, we present an open-source framework (https://github.com/kaiko-ai/eva) designed for the consistent evaluation of pathology FMs across various downstream tasks.

Title: Massively Annotated Datasets for Assessment of Synthetic and Real Data in Face Recognition

• Authors: Pedro C. Neto, Rafael M. Mamede, Carolina Albuquerque, Tiago Gonçalves, Ana F. Sequeira
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15234
• Pdf URL: https://arxiv.org/pdf/2404.15234
• Copy Paste: [[2404.15234]] Massively Annotated Datasets for Assessment of Synthetic and Real Data in Face Recognition(https://arxiv.org/abs/2404.15234)
• Keywords: generative
• Abstract: Face recognition applications have grown in parallel with the size of datasets, complexity of deep learning models and computational power. However, while deep learning models evolve to become more capable and computational power keeps increasing, the datasets available are being retracted and removed from public access. Privacy and ethical concerns are relevant topics within these domains. Through generative artificial intelligence, researchers have put efforts into the development of completely synthetic datasets that can be used to train face recognition systems. Nonetheless, the recent advances have not been sufficient to achieve performance comparable to the state-of-the-art models trained on real data. To study the drift between the performance of models trained on real and synthetic datasets, we leverage a massive attribute classifier (MAC) to create annotations for four datasets: two real and two synthetic. From these annotations, we conduct studies on the distribution of each attribute within all four datasets. Additionally, we further inspect the differences between real and synthetic datasets on the attribute set. When comparing through the Kullback-Leibler divergence we have found differences between real and synthetic samples. Interestingly enough, we have verified that while real samples suffice to explain the synthetic distribution, the opposite could not be further from being true.

Title: From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation

• Authors: Zehuan Huang, Hongxing Fan, Lipeng Wang, Lu Sheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15267
• Pdf URL: https://arxiv.org/pdf/2404.15267
• Copy Paste: [[2404.15267]] From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation(https://arxiv.org/abs/2404.15267)
• Keywords: diffusion
• Abstract: Recent advancements in controllable human image generation have led to zero-shot generation using structural signals (e.g., pose, depth) or facial appearance. Yet, generating human images conditioned on multiple parts of human appearance remains challenging. Addressing this, we introduce Parts2Whole, a novel framework designed for generating customized portraits from multiple reference images, including pose images and various aspects of human appearance. To achieve this, we first develop a semantic-aware appearance encoder to retain details of different human parts, which processes each image based on its textual label to a series of multi-scale feature maps rather than one image token, preserving the image dimension. Second, our framework supports multi-image conditioned generation through a shared self-attention mechanism that operates across reference and target features during the diffusion process. We enhance the vanilla attention mechanism by incorporating mask information from the reference human images, allowing for the precise selection of any part. Extensive experiments demonstrate the superiority of our approach over existing alternatives, offering advanced capabilities for multi-part controllable human image customization. See our project page at https://huanngzh.github.io/Parts2Whole/.

Title: ID-Animator: Zero-Shot Identity-Preserving Human Video Generation

• Authors: Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, Man Zhou, Jie Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.15275
• Pdf URL: https://arxiv.org/pdf/2404.15275
• Copy Paste: [[2404.15275]] ID-Animator: Zero-Shot Identity-Preserving Human Video Generation(https://arxiv.org/abs/2404.15275)
• Keywords: diffusion
• Abstract: Generating high fidelity human video with specified identities has attracted significant attention in the content generation community. However, existing techniques struggle to strike a balance between training efficiency and identity preservation, either requiring tedious case-by-case finetuning or usually missing the identity details in video generation process. In this study, we present ID-Animator, a zero-shot human-video generation approach that can perform personalized video generation given single reference facial image without further training. ID-Animator inherits existing diffusion-based video generation backbones with a face adapter to encode the ID-relevant embeddings from learnable facial latent queries. To facilitate the extraction of identity information in video generation, we introduce an ID-oriented dataset construction pipeline, which incorporates decoupled human attribute and action captioning technique from a constructed facial image pool. Based on this pipeline, a random face reference training method is further devised to precisely capture the ID-relevant embeddings from reference images, thus improving the fidelity and generalization capacity of our model for ID-specific video generation. Extensive experiments demonstrate the superiority of ID-Animator to generate personalized human videos over previous models. Moreover, our method is highly compatible with popular pre-trained T2V models like animatediff and various community backbone models, showing high extendability in real-world applications for video generation where identity preservation is highly desired. Our codes and checkpoints will be released at https://github.com/ID-Animator/ID-Animator.