2024-08-16

Title: Impact of Inaccurate Contamination Ratio on Robust Unsupervised Anomaly Detection

• Authors: Jordan F. Masakuna, DJeff Kanda Nkashama, Arian Soltani, Marc Frappier, Pierre-Martin Tardif, Froduald Kabanza
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07718
• Pdf URL: https://arxiv.org/pdf/2408.07718
• Copy Paste: [[2408.07718]] Impact of Inaccurate Contamination Ratio on Robust Unsupervised Anomaly Detection(https://arxiv.org/abs/2408.07718)
• Keywords: anomaly
• Abstract: Training data sets intended for unsupervised anomaly detection, typically presumed to be anomaly-free, often contain anomalies (or contamination), a challenge that significantly undermines model performance. Most robust unsupervised anomaly detection models rely on contamination ratio information to tackle contamination. However, in reality, contamination ratio may be inaccurate. We investigate on the impact of inaccurate contamination ratio information in robust unsupervised anomaly detection. We verify whether they are resilient to misinformed contamination ratios. Our investigation on 6 benchmark data sets reveals that such models are not adversely affected by exposure to misinformation. In fact, they can exhibit improved performance when provided with such inaccurate contamination ratios.

Title: Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies

• Authors: Peiran Wang, Qiyu Li, Longxuan Yu, Ziyao Wang, Ang Li, Haojian Jin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07728
• Pdf URL: https://arxiv.org/pdf/2408.07728
• Copy Paste: [[2408.07728]] Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies(https://arxiv.org/abs/2408.07728)
• Keywords: diffusion
• Abstract: We present Moderator, a policy-based model management system that allows administrators to specify fine-grained content moderation policies and modify the weights of a text-to-image (TTI) model to make it significantly more challenging for users to produce images that violate the policies. In contrast to existing general-purpose model editing techniques, which unlearn concepts without considering the associated contexts, Moderator allows admins to specify what content should be moderated, under which context, how it should be moderated, and why moderation is necessary. Given a set of policies, Moderator first prompts the original model to generate images that need to be moderated, then uses these self-generated images to reverse fine-tune the model to compute task vectors for moderation and finally negates the original model with the task vectors to decrease its performance in generating moderated content. We evaluated Moderator with 14 participants to play the role of admins and found they could quickly learn and author policies to pass unit tests in approximately 2.29 policy iterations. Our experiment with 32 stable diffusion users suggested that Moderator can prevent 65% of users from generating moderated content under 15 attempts and require the remaining users an average of 8.3 times more attempts to generate undesired content.

Title: MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis

• Authors: Nimeesha Chan, Felix Parker, William Bennett, Tianyi Wu, Mung Yao Jia, James Fackler, Kimia Ghobadi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07773
• Pdf URL: https://arxiv.org/pdf/2408.07773
• Copy Paste: [[2408.07773]] MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis(https://arxiv.org/abs/2408.07773)
• Keywords: anomaly
• Abstract: The complexity and heterogeneity of data in many real-world applications pose significant challenges for traditional machine learning and signal processing techniques. For instance, in medicine, effective analysis of diverse physiological signals is crucial for patient monitoring and clinical decision-making and yet highly challenging. We introduce MedTsLLM, a general multimodal large language model (LLM) framework that effectively integrates time series data and rich contextual information in the form of text to analyze physiological signals, performing three tasks with clinical relevance: semantic segmentation, boundary detection, and anomaly detection in time series. These critical tasks enable deeper analysis of physiological signals and can provide actionable insights for clinicians. We utilize a reprogramming layer to align embeddings of time series patches with a pretrained LLM's embedding space and make effective use of raw time series, in conjunction with textual context. Given the multivariate nature of medical datasets, we develop methods to handle multiple covariates. We additionally tailor the text prompt to include patient-specific information. Our model outperforms state-of-the-art baselines, including deep learning models, other LLMs, and clinical methods across multiple medical domains, specifically electrocardiograms and respiratory waveforms. MedTsLLM presents a promising step towards harnessing the power of LLMs for medical time series analysis that can elevate data-driven tools for clinicians and improve patient outcomes.

Title: Cropper: Vision-Language Model for Image Cropping through In-Context Learning

• Authors: Seung Hyun Lee, Junjie Ke, Yinxiao Li, Junfeng He, Steven Hickson, Katie Datsenko, Sangpil Kim, Ming-Hsuan Yang, Irfan Essa, Feng Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07790
• Pdf URL: https://arxiv.org/pdf/2408.07790
• Copy Paste: [[2408.07790]] Cropper: Vision-Language Model for Image Cropping through In-Context Learning(https://arxiv.org/abs/2408.07790)
• Keywords: in-context
• Abstract: The goal of image cropping is to identify visually appealing crops within an image. Conventional methods rely on specialized architectures trained on specific datasets, which struggle to be adapted to new requirements. Recent breakthroughs in large vision-language models (VLMs) have enabled visual in-context learning without explicit training. However, effective strategies for vision downstream tasks with VLMs remain largely unclear and underexplored. In this paper, we propose an effective approach to leverage VLMs for better image cropping. First, we propose an efficient prompt retrieval mechanism for image cropping to automate the selection of in-context examples. Second, we introduce an iterative refinement strategy to iteratively enhance the predicted crops. The proposed framework, named Cropper, is applicable to a wide range of cropping tasks, including free-form cropping, subject-aware cropping, and aspect ratio-aware cropping. Extensive experiments and a user study demonstrate that Cropper significantly outperforms state-of-the-art methods across several benchmarks.

Title: SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognition

• Authors: Mohamed Osman, Daniel Z. Kaplan, Tamer Nadeem
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07851
• Pdf URL: https://arxiv.org/pdf/2408.07851
• Copy Paste: [[2408.07851]] SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognition(https://arxiv.org/abs/2408.07851)
• Keywords: self-supervised
• Abstract: Speech emotion recognition (SER) has made significant strides with the advent of powerful self-supervised learning (SSL) models. However, the generalization of these models to diverse languages and emotional expressions remains a challenge. We propose a large-scale benchmark to evaluate the robustness and adaptability of state-of-the-art SER models in both in-domain and out-of-domain settings. Our benchmark includes a diverse set of multilingual datasets, focusing on less commonly used corpora to assess generalization to new data. We employ logit adjustment to account for varying class distributions and establish a single dataset cluster for systematic evaluation. Surprisingly, we find that the Whisper model, primarily designed for automatic speech recognition, outperforms dedicated SSL models in cross-lingual SER. Our results highlight the need for more robust and generalizable SER models, and our benchmark serves as a valuable resource to drive future research in this direction.

Title: A Systematic Evaluation of Generated Time Series and Their Effects in Self-Supervised Pretraining

• Authors: Audrey Der, Chin-Chia Michael Yeh, Xin Dai, Huiyuan Chen, Yan Zheng, Yujie Fan, Zhongfang Zhuang, Vivian Lai, Junpeng Wang, Liang Wang, Wei Zhang, Eamonn Keogh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07869
• Pdf URL: https://arxiv.org/pdf/2408.07869
• Copy Paste: [[2408.07869]] A Systematic Evaluation of Generated Time Series and Their Effects in Self-Supervised Pretraining(https://arxiv.org/abs/2408.07869)
• Keywords: self-supervised
• Abstract: Self-supervised Pretrained Models (PTMs) have demonstrated remarkable performance in computer vision and natural language processing tasks. These successes have prompted researchers to design PTMs for time series data. In our experiments, most self-supervised time series PTMs were surpassed by simple supervised models. We hypothesize this undesired phenomenon may be caused by data scarcity. In response, we test six time series generation methods, use the generated data in pretraining in lieu of the real data, and examine the effects on classification performance. Our results indicate that replacing a real-data pretraining set with a greater volume of only generated samples produces noticeable improvement.

Title: Instruct Large Language Models to Generate Scientific Literature Survey Step by Step

• Authors: Yuxuan Lai, Yupeng Wu, Yidan Wang, Wenpeng Hu, Chen Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07884
• Pdf URL: https://arxiv.org/pdf/2408.07884
• Copy Paste: [[2408.07884]] Instruct Large Language Models to Generate Scientific Literature Survey Step by Step(https://arxiv.org/abs/2408.07884)
• Keywords: generative
• Abstract: Abstract. Automatically generating scientific literature surveys is a valuable task that can significantly enhance research efficiency. However, the diverse and complex nature of information within a literature survey poses substantial challenges for generative models. In this paper, we design a series of prompts to systematically leverage large language models (LLMs), enabling the creation of comprehensive literature surveys through a step-by-step approach. Specifically, we design prompts to guide LLMs to sequentially generate the title, abstract, hierarchical headings, and the main content of the literature survey. We argue that this design enables the generation of the headings from a high-level perspective. During the content generation process, this design effectively harnesses relevant information while minimizing costs by restricting the length of both input and output content in LLM queries. Our implementation with Qwen-long achieved third place in the NLPCC 2024 Scientific Literature Survey Generation evaluation task, with an overall score only 0.03% lower than the second-place team. Additionally, our soft heading recall is 95.84%, the second best among the submissions. Thanks to the efficient prompt design and the low cost of the Qwen-long API, our method reduces the expense for generating each literature survey to 0.1 RMB, enhancing the practical value of our method.

Title: MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL

• Authors: Wenxuan Xie, Gaochen Wu, Bowen Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07930
• Pdf URL: https://arxiv.org/pdf/2408.07930
• Copy Paste: [[2408.07930]] MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL(https://arxiv.org/abs/2408.07930)
• Keywords: generative, in-context
• Abstract: Recent In-Context Learning based methods have achieved remarkable success in Text-to-SQL task. However, there is still a large gap between the performance of these models and human performance on datasets with complex database schema and difficult questions, such as BIRD. Besides, existing work has neglected to supervise intermediate steps when solving questions iteratively with question decomposition methods, and the schema linking methods used in these works are very rudimentary. To address these issues, we propose MAG-SQL, a multi-agent generative approach with soft schema linking and iterative Sub-SQL refinement. In our framework, an entity-based method with tables' summary is used to select the columns in database, and a novel targets-conditions decomposition method is introduced to decompose those complex questions. Additionally, we build a iterative generating module which includes a Sub-SQL Generator and Sub-SQL Refiner, introducing external oversight for each step of generation. Through a series of ablation studies, the effectiveness of each agent in our framework has been demonstrated. When evaluated on the BIRD benchmark with GPT-4, MAG-SQL achieves an execution accuracy of 61.08\%, compared to the baseline accuracy of 46.35\% for vanilla GPT-4 and the baseline accuracy of 57.56\% for MAC-SQL. Besides, our approach makes similar progress on Spider.

Title: ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models

• Authors: Faris Hijazi (1), Somayah AlHarbi (1), Abdulaziz AlHussein (1), Harethah Abu Shairah (2), Reem AlZahrani (2), Hebah AlShamlan (1), Omar Knio (2), George Turkiyyah (2) ((1) THIQAH, (2) KAUST)
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.07983
• Pdf URL: https://arxiv.org/pdf/2408.07983
• Copy Paste: [[2408.07983]] ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models(https://arxiv.org/abs/2408.07983)
• Keywords: in-context
• Abstract: The rapid advancements in Large Language Models (LLMs) have led to significant improvements in various natural language processing tasks. However, the evaluation of LLMs' legal knowledge, particularly in non-English languages such as Arabic, remains under-explored. To address this gap, we introduce ArabLegalEval, a multitask benchmark dataset for assessing the Arabic legal knowledge of LLMs. Inspired by the MMLU and LegalBench datasets, ArabLegalEval consists of multiple tasks sourced from Saudi legal documents and synthesized questions. In this work, we aim to analyze the capabilities required to solve legal problems in Arabic and benchmark the performance of state-of-the-art LLMs. We explore the impact of in-context learning and investigate various evaluation methods. Additionally, we explore workflows for generating questions with automatic validation to enhance the dataset's quality. We benchmark multilingual and Arabic-centric LLMs, such as GPT-4 and Jais, respectively. We also share our methodology for creating the dataset and validation, which can be generalized to other domains. We hope to accelerate AI research in the Arabic Legal domain by releasing the ArabLegalEval dataset and code: this https URL

Title: Causal Discovery from Time-Series Data with Short-Term Invariance-Based Convolutional Neural Networks

• Authors: Rujia Shen, Boran Wang, Chao Zhao, Yi Guan, Jingchi Jiang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08023
• Pdf URL: https://arxiv.org/pdf/2408.08023
• Copy Paste: [[2408.08023]] Causal Discovery from Time-Series Data with Short-Term Invariance-Based Convolutional Neural Networks(https://arxiv.org/abs/2408.08023)
• Keywords: generative
• Abstract: Causal discovery from time-series data aims to capture both intra-slice (contemporaneous) and inter-slice (time-lagged) causality between variables within the temporal chain, which is crucial for various scientific disciplines. Compared to causal discovery from non-time-series data, causal discovery from time-series data necessitates more serialized samples with a larger amount of observed time steps. To address the challenges, we propose a novel gradient-based causal discovery approach STIC, which focuses on \textbf{S}hort-\textbf{T}erm \textbf{I}nvariance using \textbf{C}onvolutional neural networks to uncover the causal relationships from time-series data. Specifically, STIC leverages both the short-term time and mechanism invariance of causality within each window observation, which possesses the property of independence, to enhance sample efficiency. Furthermore, we construct two causal convolution kernels, which correspond to the short-term time and mechanism invariance respectively, to estimate the window causal graph. To demonstrate the necessity of convolutional neural networks for causal discovery from time-series data, we theoretically derive the equivalence between convolution and the underlying generative principle of time-series data under the assumption that the additive noise model is identifiable. Experimental evaluations conducted on both synthetic and FMRI benchmark datasets demonstrate that our STIC outperforms baselines significantly and achieves the state-of-the-art performance, particularly when the datasets contain a limited number of observed time steps. Code is available at \url{this https URL}.

Title: The Clever Hans Effect in Unsupervised Learning

• Authors: Jacob Kauffmann, Jonas Dippel, Lukas Ruff, Wojciech Samek, Klaus-Robert Müller, Grégoire Montavon
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.08041
• Pdf URL: https://arxiv.org/pdf/2408.08041
• Copy Paste: [[2408.08041]] The Clever Hans Effect in Unsupervised Learning(https://arxiv.org/abs/2408.08041)
• Keywords: foundation model
• Abstract: Unsupervised learning has become an essential building block of AI systems. The representations it produces, e.g. in foundation models, are critical to a wide variety of downstream applications. It is therefore important to carefully examine unsupervised models to ensure not only that they produce accurate predictions, but also that these predictions are not "right for the wrong reasons", the so-called Clever Hans (CH) effect. Using specially developed Explainable AI techniques, we show for the first time that CH effects are widespread in unsupervised learning. Our empirical findings are enriched by theoretical insights, which interestingly point to inductive biases in the unsupervised learning machine as a primary source of CH effects. Overall, our work sheds light on unexplored risks associated with practical applications of unsupervised learning and suggests ways to make unsupervised learning more robust.

Title: Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging

• Authors: Stefano Woerner, Christian F. Baumgartner
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08058
• Pdf URL: https://arxiv.org/pdf/2408.08058
• Copy Paste: [[2408.08058]] Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging(https://arxiv.org/abs/2408.08058)
• Keywords: foundation model
• Abstract: Data scarcity is a major limiting factor for applying modern machine learning techniques to clinical tasks. Although sufficient data exists for some well-studied medical tasks, there remains a long tail of clinically relevant tasks with poor data availability. Recently, numerous foundation models have demonstrated high suitability for few-shot learning (FSL) and zero-shot learning (ZSL), potentially making them more accessible to practitioners. However, it remains unclear which foundation model performs best on FSL medical image analysis tasks and what the optimal methods are for learning from limited data. We conducted a comprehensive benchmark study of ZSL and FSL using 16 pretrained foundation models on 19 diverse medical imaging datasets. Our results indicate that BiomedCLIP, a model pretrained exclusively on medical data, performs best on average for very small training set sizes, while very large CLIP models pretrained on LAION-2B perform best with slightly more training samples. However, simply fine-tuning a ResNet-18 pretrained on ImageNet performs similarly with more than five training examples per class. Our findings also highlight the need for further research on foundation models specifically tailored for medical applications and the collection of more datasets to train these models.

Title: MambaMIM: Pre-training Mamba with State Space Token-interpolation

• Authors: Fenghe Tang, Bingkun Nian, Yingtai Li, Jie Yang, Liu Wei, S. Kevin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08070
• Pdf URL: https://arxiv.org/pdf/2408.08070
• Copy Paste: [[2408.08070]] MambaMIM: Pre-training Mamba with State Space Token-interpolation(https://arxiv.org/abs/2408.08070)
• Keywords: self-supervised, generative
• Abstract: Generative self-supervised learning demonstrates outstanding representation learning capabilities in both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). However, there are currently no generative pre-training methods related to selective state space models (Mamba) that can handle long-range dependencies effectively. To address this challenge, we introduce a generative self-supervised learning method for Mamba (MambaMIM) based on Selective Structure State Space Sequence Token-interpolation (S6T), a general-purpose pre-training method for arbitrary Mamba architectures. Our method, MambaMIM, incorporates a bottom-up 3D hybrid masking strategy in the encoder to maintain masking consistency across different architectures. Additionally, S6T is employed to learn causal relationships between the masked sequence in the state space. MambaMIM can be used on any single or hybrid Mamba architectures to enhance the Mamba long-range representation capability. Extensive downstream experiments reveal the feasibility and advancement of using Mamba for pre-training medical image tasks. The code is available at: this https URL

Title: ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mamba

• Authors: Huiyu Zhai, Guang Jin, Xingxing Yang, Guosheng Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08087
• Pdf URL: https://arxiv.org/pdf/2408.08087
• Copy Paste: [[2408.08087]] ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mamba(https://arxiv.org/abs/2408.08087)
• Keywords: generative
• Abstract: Translating NIR to the visible spectrum is challenging due to cross-domain complexities. Current models struggle to balance a broad receptive field with computational efficiency, limiting practical use. Although the Selective Structured State Space Model, especially the improved version, Mamba, excels in generative tasks by capturing long-range dependencies with linear complexity, its default approach of converting 2D images into 1D sequences neglects local context. In this work, we propose a simple but effective backbone, dubbed ColorMamba, which first introduces Mamba into spectral translation tasks. To explore global long-range dependencies and local context for efficient spectral translation, we introduce learnable padding tokens to enhance the distinction of image boundaries and prevent potential confusion within the sequence model. Furthermore, local convolutional enhancement and agent attention are designed to improve the vanilla Mamba. Moreover, we exploit the HSV color to provide multi-scale guidance in the reconstruction process for more accurate spectral translation. Extensive experiments show that our ColorMamba achieves a 1.02 improvement in terms of PSNR compared with the state-of-the-art method. Our code is available at this https URL.

Title: When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding

• Authors: Pingping Zhang, Jinlong Li, Meng Wang, Nicu Sebe, Sam Kwong, Shiqi Wang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.08093
• Pdf URL: https://arxiv.org/pdf/2408.08093
• Copy Paste: [[2408.08093]] When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding(https://arxiv.org/abs/2408.08093)
• Keywords: generative
• Abstract: Existing codecs are designed to eliminate intrinsic redundancies to create a compact representation for compression. However, strong external priors from Multimodal Large Language Models (MLLMs) have not been explicitly explored in video compression. Herein, we introduce a unified paradigm for Cross-Modality Video Coding (CMVC), which is a pioneering approach to explore multimodality representation and video generative models in video coding. Specifically, on the encoder side, we disentangle a video into spatial content and motion components, which are subsequently transformed into distinct modalities to achieve very compact representation by leveraging MLLMs. During decoding, previously encoded components and video generation models are leveraged to create multiple encoding-decoding modes that optimize video reconstruction quality for specific decoding requirements, including Text-Text-to-Video (TT2V) mode to ensure high-quality semantic information and Image-Text-to-Video (IT2V) mode to achieve superb perceptual consistency. In addition, we propose an efficient frame interpolation model for IT2V mode via Low-Rank Adaption (LoRA) tuning to guarantee perceptual quality, which allows the generated motion cues to behave smoothly. Experiments on benchmarks indicate that TT2V achieves effective semantic reconstruction, while IT2V exhibits competitive perceptual consistency. These results highlight potential directions for future research in video coding.

Title: Not Every Image is Worth a Thousand Words: Quantifying Originality in Stable Diffusion

• Authors: Adi Haviv, Shahar Sarfaty, Uri Hacohen, Niva Elkin-Koren, Roi Livni, Amit H Bermano
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08184
• Pdf URL: https://arxiv.org/pdf/2408.08184
• Copy Paste: [[2408.08184]] Not Every Image is Worth a Thousand Words: Quantifying Originality in Stable Diffusion(https://arxiv.org/abs/2408.08184)
• Keywords: diffusion, generative
• Abstract: This work addresses the challenge of quantifying originality in text-to-image (T2I) generative diffusion models, with a focus on copyright originality. We begin by evaluating T2I models' ability to innovate and generalize through controlled experiments, revealing that stable diffusion models can effectively recreate unseen elements with sufficiently diverse training data. Then, our key insight is that concepts and combinations of image elements the model is familiar with, and saw more during training, are more concisly represented in the model's latent space. We hence propose a method that leverages textual inversion to measure the originality of an image based on the number of tokens required for its reconstruction by the model. Our approach is inspired by legal definitions of originality and aims to assess whether a model can produce original content without relying on specific prompts or having the training data of the model. We demonstrate our method using both a pre-trained stable diffusion model and a synthetic dataset, showing a correlation between the number of tokens and image originality. This work contributes to the understanding of originality in generative models and has implications for copyright infringement cases.

Title: Heavy Labels Out! Dataset Distillation with Label Space Lightening

• Authors: Ruonan Yu, Songhua Liu, Zigeng Chen, Jingwen Ye, Xinchao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08201
• Pdf URL: https://arxiv.org/pdf/2408.08201
• Copy Paste: [[2408.08201]] Heavy Labels Out! Dataset Distillation with Label Space Lightening(https://arxiv.org/abs/2408.08201)
• Keywords: foundation model
• Abstract: Dataset distillation or condensation aims to condense a large-scale training dataset into a much smaller synthetic one such that the training performance of distilled and original sets on neural networks are similar. Although the number of training samples can be reduced substantially, current state-of-the-art methods heavily rely on enormous soft labels to achieve satisfactory performance. As a result, the required storage can be comparable even to original datasets, especially for large-scale ones. To solve this problem, instead of storing these heavy labels, we propose a novel label-lightening framework termed HeLlO aiming at effective image-to-label projectors, with which synthetic labels can be directly generated online from synthetic images. Specifically, to construct such projectors, we leverage prior knowledge in open-source foundation models, e.g., CLIP, and introduce a LoRA-like fine-tuning strategy to mitigate the gap between pre-trained and target distributions, so that original models for soft-label generation can be distilled into a group of low-rank matrices. Moreover, an effective image optimization method is proposed to further mitigate the potential error between the original and distilled label generators. Extensive experiments demonstrate that with only about 0.003% of the original storage required for a complete set of soft labels, we achieve comparable performance to current state-of-the-art dataset distillation methods on large-scale datasets. Our code will be available.

Title: The Dawn of KAN in Image-to-Image (I2I) Translation: Integrating Kolmogorov-Arnold Networks with GANs for Unpaired I2I Translation

• Authors: Arpan Mahara, Naphtali D. Rishe, Liangdong Deng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08216
• Pdf URL: https://arxiv.org/pdf/2408.08216
• Copy Paste: [[2408.08216]] The Dawn of KAN in Image-to-Image (I2I) Translation: Integrating Kolmogorov-Arnold Networks with GANs for Unpaired I2I Translation(https://arxiv.org/abs/2408.08216)
• Keywords: generative
• Abstract: Image-to-Image translation in Generative Artificial Intelligence (Generative AI) has been a central focus of research, with applications spanning healthcare, remote sensing, physics, chemistry, photography, and more. Among the numerous methodologies, Generative Adversarial Networks (GANs) with contrastive learning have been particularly successful. This study aims to demonstrate that the Kolmogorov-Arnold Network (KAN) can effectively replace the Multi-layer Perceptron (MLP) method in generative AI, particularly in the subdomain of image-to-image translation, to achieve better generative quality. Our novel approach replaces the two-layer MLP with a two-layer KAN in the existing Contrastive Unpaired Image-to-Image Translation (CUT) model, developing the KAN-CUT model. This substitution favors the generation of more informative features in low-dimensional vector representations, which contrastive learning can utilize more effectively to produce high-quality images in the target domain. Extensive experiments, detailed in the results section, demonstrate the applicability of KAN in conjunction with contrastive learning and GANs in Generative AI, particularly for image-to-image translation. This work suggests that KAN could be a valuable component in the broader generative AI domain.

Title: Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding

• Authors: Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani, Aviv Regev, Sergey Levine, Masatoshi Uehara
• Subjects: cs.LG, cs.AI, q-bio.GN, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.08252
• Pdf URL: https://arxiv.org/pdf/2408.08252
• Copy Paste: [[2408.08252]] Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding(https://arxiv.org/abs/2408.08252)
• Keywords: diffusion, generative
• Abstract: Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. However, rather than merely generating designs that are natural, we often aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require ``differentiable'' proxy models (\textit{e.g.}, classifier guidance or DPS) or involve computationally expensive fine-tuning of diffusion models (\textit{e.g.}, classifier-free guidance, RL-based fine-tuning). In our work, we propose a new method to address these challenges. Our algorithm is an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future, into the standard inference procedure of pre-trained diffusion models. Notably, our approach avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us to (1) directly utilize non-differentiable features/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of our algorithm across several domains, including image generation, molecule generation, and DNA/RNA sequence generation. The code is available at \href{this https URL}{this https URL}.

Title: Snuffy: Efficient Whole Slide Image Classifier

• Authors: Hossein Jafarinia, Alireza Alipanah, Danial Hamdi, Saeed Razavi, Nahal Mirzaie, Mohammad Hossein Rohban
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.08258
• Pdf URL: https://arxiv.org/pdf/2408.08258
• Copy Paste: [[2408.08258]] Snuffy: Efficient Whole Slide Image Classifier(https://arxiv.org/abs/2408.08258)
• Keywords: self-supervised
• Abstract: Whole Slide Image (WSI) classification with multiple instance learning (MIL) in digital pathology faces significant computational challenges. Current methods mostly rely on extensive self-supervised learning (SSL) for satisfactory performance, requiring long training periods and considerable computational resources. At the same time, no pre-training affects performance due to domain shifts from natural images to WSIs. We introduce \textbf{\textit{Snuffy}} architecture, a novel MIL-pooling method based on sparse transformers that mitigates performance loss with limited pre-training and enables continual few-shot pre-training as a competitive option. Our sparsity pattern is tailored for pathology and is theoretically proven to be a universal approximator with the tightest probabilistic sharp bound on the number of layers for sparse transformers, to date. We demonstrate Snuffy's effectiveness on CAMELYON16 and TCGA Lung cancer datasets, achieving superior WSI and patch-level accuracies. The code is available on \url{this https URL}.

Title: mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis

• Authors: Dae-young Kim (1), Rebecca Hwa (2), Muhammad Mahbubur Rahman (1) ((1) Children's National Hospital, Washington, DC, (2) George Washington University, Washington, DC)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08261
• Pdf URL: https://arxiv.org/pdf/2408.08261
• Copy Paste: [[2408.08261]] mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis(https://arxiv.org/abs/2408.08261)
• Keywords: generative
• Abstract: This paper introduces mhGPT, a lightweight generative pre-trained transformer trained on mental health-related social media and PubMed articles. Fine-tuned for specific mental health tasks, mhGPT was evaluated under limited hardware constraints and compared with state-of-the-art models like MentaLLaMA and Gemma. Despite having only 1.98 billion parameters and using just 5% of the dataset, mhGPT outperformed larger models and matched the performance of models trained on significantly more data. The key contributions include integrating diverse mental health data, creating a custom tokenizer, and optimizing a smaller architecture for low-resource settings. This research could advance AI-driven mental health care, especially in areas with limited computing power.

Title: Understanding the Local Geometry of Generative Model Manifolds

• Authors: Ahmed Imtiaz Humayun, Ibtihel Amara, Candice Schumann, Golnoosh Farnadi, Negar Rostamzadeh, Mohammad Havaei
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08307
• Pdf URL: https://arxiv.org/pdf/2408.08307
• Copy Paste: [[2408.08307]] Understanding the Local Geometry of Generative Model Manifolds(https://arxiv.org/abs/2408.08307)
• Keywords: diffusion, foundation model, generative
• Abstract: Deep generative models learn continuous representations of complex data manifolds using a finite number of samples during training. For a pre-trained generative model, the common way to evaluate the quality of the manifold representation learned, is by computing global metrics like Fréchet Inception Distance using a large number of generated and real samples. However, generative model performance is not uniform across the learned manifold, e.g., for \textit{foundation models} like Stable Diffusion generation performance can vary significantly based on the conditioning or initial noise vector being denoised. In this paper we study the relationship between the \textit{local geometry of the learned manifold} and downstream generation. Based on the theory of continuous piecewise-linear (CPWL) generators, we use three geometric descriptors - scaling ($\psi$), rank ($\nu$), and complexity ($\delta$) - to characterize a pre-trained generative model manifold locally. We provide quantitative and qualitative evidence showing that for a given latent, the local descriptors are correlated with generation aesthetics, artifacts, uncertainty, and even memorization. Finally we demonstrate that training a \textit{reward model} on the local geometry can allow controlling the likelihood of a generated sample under the learned distribution.