2024-08-15

Title: Maximizing V-information for Pre-training Superior Foundation Models

• Authors: Wenxuan Yang, Weimin Tan, Hanyu Zhang, Bo Yan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07107
• Pdf URL: https://arxiv.org/pdf/2408.07107
• Copy Paste: [[2408.07107]] Maximizing V-information for Pre-training Superior Foundation Models(https://arxiv.org/abs/2408.07107)
• Keywords: foundation model
• Abstract: Pre-training foundation models on large-scale datasets demonstrates exceptional performance. However, recent research questions this traditional notion, exploring whether an increase in pre-training data always leads to enhanced model performance. To address this issue, data-effective learning approaches have been introduced. However, current methods in this area lack a clear standard for sample selection. Our experiments reveal that by maximizing V-information, sample selection can be framed as an optimization problem, enabling effective improvement in model performance even with fewer samples. Under this guidance, we develop an optimal data-effective learning method (OptiDEL) to maximize V-information. The OptiDEL method generates hard samples to achieve or even exceed the performance of models trained on the full dataset while using substantially less data. We compare the OptiDEL method with state-of-the-art approaches finding that OptiDEL consistently outperforms existing approaches across different datasets, with foundation models trained on only 5% of the pre-training data surpassing the performance of those trained on the full dataset.

Title: Generative Photomontage

• Authors: Sean J. Liu, Nupur Kumari, Ariel Shamir, Jun-Yan Zhu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.07116
• Pdf URL: https://arxiv.org/pdf/2408.07116
• Copy Paste: [[2408.07116]] Generative Photomontage(https://arxiv.org/abs/2408.07116)
• Keywords: diffusion, generative
• Abstract: Text-to-image models are powerful tools for image creation. However, the generation process is akin to a dice roll and makes it difficult to achieve a single image that captures everything a user wants. In this paper, we propose a framework for creating the desired image by compositing it from various parts of generated images, in essence forming a Generative Photomontage. Given a stack of images generated by ControlNet using the same input condition and different seeds, we let users select desired parts from the generated results using a brush stroke interface. We introduce a novel technique that takes in the user's brush strokes, segments the generated images using a graph-based optimization in diffusion feature space, and then composites the segmented regions via a new feature-space blending method. Our method faithfully preserves the user-selected regions while compositing them harmoniously. We demonstrate that our flexible framework can be used for many applications, including generating new appearance combinations, fixing incorrect shapes and artifacts, and improving prompt alignment. We show compelling results for each application and demonstrate that our method outperforms existing image blending methods and various baselines.

Title: Controlling the World by Sleight of Hand

• Authors: Sruthi Sudhakar, Ruoshi Liu, Basile Van Hoorick, Carl Vondrick, Richard Zemel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07147
• Pdf URL: https://arxiv.org/pdf/2408.07147
• Copy Paste: [[2408.07147]] Controlling the World by Sleight of Hand(https://arxiv.org/abs/2408.07147)
• Keywords: generative
• Abstract: Humans naturally build mental models of object interactions and dynamics, allowing them to imagine how their surroundings will change if they take a certain action. While generative models today have shown impressive results on generating/editing images unconditionally or conditioned on text, current methods do not provide the ability to perform object manipulation conditioned on actions, an important tool for world modeling and action planning. Therefore, we propose to learn an action-conditional generative models by learning from unlabeled videos of human hands interacting with objects. The vast quantity of such data on the internet allows for efficient scaling which can enable high-performing action-conditional models. Given an image, and the shape/location of a desired hand interaction, CosHand, synthesizes an image of a future after the interaction has occurred. Experiments show that the resulting model can predict the effects of hand-object interactions well, with strong generalization particularly to translation, stretching, and squeezing interactions of unseen objects in unseen environments. Further, CosHand can be sampled many times to predict multiple possible effects, modeling the uncertainty of forces in the interaction/environment. Finally, method generalizes to different embodiments, including non-human hands, i.e. robot hands, suggesting that generative video models can be powerful models for robotics.

Title: SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis

• Authors: Yuchen Mao, Hongwei Li, Wei Pang, Giorgos Papanastasiou, Guang Yang, Chengjia Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07196
• Pdf URL: https://arxiv.org/pdf/2408.07196
• Copy Paste: [[2408.07196]] SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis(https://arxiv.org/abs/2408.07196)
• Keywords: diffusion
• Abstract: The persistent challenge of medical image synthesis posed by the scarcity of annotated data and the need to synthesize `missing modalities' for multi-modal analysis, underscored the imperative development of effective synthesis methods. Recently, the combination of Low-Rank Adaptation (LoRA) with latent diffusion models (LDMs) has emerged as a viable approach for efficiently adapting pre-trained large language models, in the medical field. However, the direct application of LoRA assumes uniform ranking across all linear layers, overlooking the significance of different weight matrices, and leading to sub-optimal outcomes. Prior works on LoRA prioritize the reduction of trainable parameters, and there exists an opportunity to further tailor this adaptation process to the intricate demands of medical image synthesis. In response, we present SeLoRA, a Self-Expanding Low-Rank Adaptation Module, that dynamically expands its ranking across layers during training, strategically placing additional ranks on crucial layers, to allow the model to elevate synthesis quality where it matters most. The proposed method not only enables LDMs to fine-tune on medical data efficiently but also empowers the model to achieve improved image quality with minimal ranking. The code of our SeLoRA method is publicly available on https://anonymous.4open.science/r/SeLoRA-980D .

Title: A Review of Pseudo-Labeling for Computer Vision

• Authors: Patrick Kage, Jay C. Rothenberger, Pavlos Andreadis, Dimitrios I. Diochnos
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07221
• Pdf URL: https://arxiv.org/pdf/2408.07221
• Copy Paste: [[2408.07221]] A Review of Pseudo-Labeling for Computer Vision(https://arxiv.org/abs/2408.07221)
• Keywords: self-supervised
• Abstract: Deep neural models have achieved state of the art performance on a wide range of problems in computer science, especially in computer vision. However, deep neural networks often require large datasets of labeled samples to generalize effectively, and an important area of active research is semi-supervised learning, which attempts to instead utilize large quantities of (easily acquired) unlabeled samples. One family of methods in this space is pseudo-labeling, a class of algorithms that use model outputs to assign labels to unlabeled samples which are then used as labeled samples during training. Such assigned labels, called pseudo-labels, are most commonly associated with the field of semi-supervised learning. In this work we explore a broader interpretation of pseudo-labels within both self-supervised and unsupervised methods. By drawing the connection between these areas we identify new directions when advancements in one area would likely benefit others, such as curriculum learning and self-supervised regularization.

Title: GRIF-DM: Generation of Rich Impression Fonts using Diffusion Models

• Authors: Lei Kang, Fei Yang, Kai Wang, Mohamed Ali Souibgui, Lluis Gomez, Alicia Fornés, Ernest Valveny, Dimosthenis Karatzas
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07259
• Pdf URL: https://arxiv.org/pdf/2408.07259
• Copy Paste: [[2408.07259]] GRIF-DM: Generation of Rich Impression Fonts using Diffusion Models(https://arxiv.org/abs/2408.07259)
• Keywords: diffusion, generative
• Abstract: Fonts are integral to creative endeavors, design processes, and artistic productions. The appropriate selection of a font can significantly enhance artwork and endow advertisements with a higher level of expressivity. Despite the availability of numerous diverse font designs online, traditional retrieval-based methods for font selection are increasingly being supplanted by generation-based approaches. These newer methods offer enhanced flexibility, catering to specific user preferences and capturing unique stylistic impressions. However, current impression font techniques based on Generative Adversarial Networks (GANs) necessitate the utilization of multiple auxiliary losses to provide guidance during generation. Furthermore, these methods commonly employ weighted summation for the fusion of impression-related keywords. This leads to generic vectors with the addition of more impression keywords, ultimately lacking in detail generation capacity. In this paper, we introduce a diffusion-based method, termed \ourmethod, to generate fonts that vividly embody specific impressions, utilizing an input consisting of a single letter and a set of descriptive impression keywords. The core innovation of \ourmethod lies in the development of dual cross-attention modules, which process the characteristics of the letters and impression keywords independently but synergistically, ensuring effective integration of both types of information. Our experimental results, conducted on the MyFonts dataset, affirm that this method is capable of producing realistic, vibrant, and high-fidelity fonts that are closely aligned with user specifications. This confirms the potential of our approach to revolutionize font generation by accommodating a broad spectrum of user-driven design requirements. Our code is publicly available at \url{this https URL}.

Title: DDIM Redux: Mathematical Foundation and Some Extension

• Authors: Manhyung Han
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07285
• Pdf URL: https://arxiv.org/pdf/2408.07285
• Copy Paste: [[2408.07285]] DDIM Redux: Mathematical Foundation and Some Extension(https://arxiv.org/abs/2408.07285)
• Keywords: diffusion
• Abstract: This note provides a critical review of the mathematical concepts underlying the generalized diffusion denoising implicit model (gDDIM) and the exponential integrator (EI) scheme. We present enhanced mathematical results, including an exact expression for the reverse trajectory in the probability flow ODE and an exact expression for the covariance matrix in the gDDIM scheme. Furthermore, we offer an improved understanding of the EI scheme's efficiency in terms of the change of variables. The noising process in DDIM is analyzed from the perspective of non-equilibrium statistical physics. Additionally, we propose a new scheme for DDIM, called the principal-axis DDIM (paDDIM).

Title: LiPCoT: Linear Predictive Coding based Tokenizer for Self-supervised Learning of Time Series Data via Language Models

• Authors: Md Fahim Anjum
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.07292
• Pdf URL: https://arxiv.org/pdf/2408.07292
• Copy Paste: [[2408.07292]] LiPCoT: Linear Predictive Coding based Tokenizer for Self-supervised Learning of Time Series Data via Language Models(https://arxiv.org/abs/2408.07292)
• Keywords: self-supervised
• Abstract: Language models have achieved remarkable success in various natural language processing tasks. However, their application to time series data, a crucial component in many domains, remains limited. This paper proposes LiPCoT (Linear Predictive Coding based Tokenizer for time series), a novel tokenizer that encodes time series data into a sequence of tokens, enabling self-supervised learning of time series using existing Language model architectures such as BERT. Unlike traditional time series tokenizers that rely heavily on CNN encoder for time series feature generation, LiPCoT employs stochastic modeling through linear predictive coding to create a latent space for time series providing a compact yet rich representation of the inherent stochastic nature of the data. Furthermore, LiPCoT is computationally efficient and can effectively handle time series data with varying sampling rates and lengths, overcoming common limitations of existing time series tokenizers. In this proof-of-concept work, we present the effectiveness of LiPCoT in classifying Parkinson's disease (PD) using an EEG dataset from 46 participants. In particular, we utilize LiPCoT to encode EEG data into a small vocabulary of tokens and then use BERT for self-supervised learning and the downstream task of PD classification. We benchmark our approach against several state-of-the-art CNN-based deep learning architectures for PD detection. Our results reveal that BERT models utilizing self-supervised learning outperformed the best-performing existing method by 7.1% in precision, 2.3% in recall, 5.5% in accuracy, 4% in AUC, and 5% in F1-score highlighting the potential for self-supervised learning even on small datasets. Our work will inform future foundational models for time series, particularly for self-supervised learning.

Title: Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery

• Authors: Yue Yu, Ning Liu, Fei Lu, Tian Gao, Siavash Jafarzadeh, Stewart Silling
• Subjects: cs.LG, math.AP
• Abstract URL: https://arxiv.org/abs/2408.07307
• Pdf URL: https://arxiv.org/pdf/2408.07307
• Copy Paste: [[2408.07307]] Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery(https://arxiv.org/abs/2408.07307)
• Keywords: foundation model
• Abstract: Despite the recent popularity of attention-based neural architectures in core AI fields like natural language processing (NLP) and computer vision (CV), their potential in modeling complex physical systems remains under-explored. Learning problems in physical systems are often characterized as discovering operators that map between function spaces based on a few instances of function pairs. This task frequently presents a severely ill-posed PDE inverse problem. In this work, we propose a novel neural operator architecture based on the attention mechanism, which we coin Nonlocal Attention Operator (NAO), and explore its capability towards developing a foundation physical model. In particular, we show that the attention mechanism is equivalent to a double integral operator that enables nonlocal interactions among spatial tokens, with a data-dependent kernel characterizing the inverse mapping from data to the hidden parameter field of the underlying operator. As such, the attention mechanism extracts global prior information from training data generated by multiple systems, and suggests the exploratory space in the form of a nonlinear kernel map. Consequently, NAO can address ill-posedness and rank deficiency in inverse PDE problems by encoding regularization and achieving generalizability. We empirically demonstrate the advantages of NAO over baseline neural models in terms of generalizability to unseen data resolutions and system states. Our work not only suggests a novel neural operator architecture for learning interpretable foundation models of physical systems, but also offers a new perspective towards understanding the attention mechanism.

Title: KIND: Knowledge Integration and Diversion in Diffusion Models

• Authors: Yucheng Xie, Fu Feng, Jing Wang, Xin Geng, Yong Rui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07337
• Pdf URL: https://arxiv.org/pdf/2408.07337
• Copy Paste: [[2408.07337]] KIND: Knowledge Integration and Diversion in Diffusion Models(https://arxiv.org/abs/2408.07337)
• Keywords: diffusion
• Abstract: Pre-trained models have become the preferred backbone due to the expansion of model parameters, with techniques like Parameter-Efficient Fine-Tuning (PEFTs) typically fixing the parameters of these models. However, pre-trained models may not always be optimal, especially when there are discrepancies between training tasks and target tasks, potentially resulting in negative transfer. To address this, we introduce \textbf{KIND}, which performs \textbf{K}nowledge \textbf{IN}tegration and \textbf{D}iversion in diffusion models. KIND first integrates knowledge by decomposing parameter matrices of models using $U$, $\Sigma$, and $V$ matrices, formally inspired by singular value decomposition (SVD). Then it explicitly partitions the components of these matrices into \textbf{learngenes} and \textbf{tailors} to condense common and class-specific knowledge, respectively, through a class gate. In this way, KIND redefines traditional pre-training methods by adjusting training objectives from maximizing model performance on current tasks to condensing transferable common knowledge, leveraging the \textit{Learngene} framework. We conduct experiments on ImageNet-1K and compare KIND with PEFT and other learngene methods. Results indicate that KIND achieves state-of-the-art performance compared to other PEFT and learngene methods. Specifically, the images generated by KIND achieves more than 6.54 and 1.07 decrease in FID and sFID on DiT-L/2, utilizing only 45.4M trainable parameters and saving at least 35.4G FLOPs in computational cost.

Title: Gradient Alignment Improves Test-Time Adaptation for Medical Image Segmentation

• Authors: Ziyang Chen, Yiwen Ye, Yongsheng Pan, Yong Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07343
• Pdf URL: https://arxiv.org/pdf/2408.07343
• Copy Paste: [[2408.07343]] Gradient Alignment Improves Test-Time Adaptation for Medical Image Segmentation(https://arxiv.org/abs/2408.07343)
• Keywords: self-supervised
• Abstract: Although recent years have witnessed significant advancements in medical image segmentation, the pervasive issue of domain shift among medical images from diverse centres hinders the effective deployment of pre-trained models. Many Test-time Adaptation (TTA) methods have been proposed to address this issue by fine-tuning pre-trained models with test data during inference. These methods, however, often suffer from less-satisfactory optimization due to suboptimal optimization direction (dictated by the gradient) and fixed step-size (predicated on the learning rate). In this paper, we propose the Gradient alignment-based Test-time adaptation (GraTa) method to improve both the gradient direction and learning rate in the optimization procedure. Unlike conventional TTA methods, which primarily optimize the pseudo gradient derived from a self-supervised objective, our method incorporates an auxiliary gradient with the pseudo one to facilitate gradient alignment. Such gradient alignment enables the model to excavate the similarities between different gradients and correct the gradient direction to approximate the empirical gradient related to the current segmentation task. Additionally, we design a dynamic learning rate based on the cosine similarity between the pseudo and auxiliary gradients, thereby empowering the adaptive fine-tuning of pre-trained models on diverse test data. Extensive experiments establish the effectiveness of the proposed gradient alignment and dynamic learning rate and substantiate the superiority of our GraTa method over other state-of-the-art TTA methods on a benchmark medical image segmentation task. The code and weights of pre-trained source models will be available.

Title: Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics

• Authors: Peter Romero, Stephen Fitz, Teruo Nakatsuma
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.07377
• Pdf URL: https://arxiv.org/pdf/2408.07377
• Copy Paste: [[2408.07377]] Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics(https://arxiv.org/abs/2408.07377)
• Keywords: foundation model
• Abstract: Previous research on emergence in large language models shows these display apparent human-like abilities and psychological latent traits. However, results are partly contradicting in expression and magnitude of these latent traits, yet agree on the worrisome tendencies to score high on the Dark Triad of narcissism, psychopathy, and Machiavellianism, which, together with a track record of derailments, demands more rigorous research on safety of these models. We provided a state of the art language model with the same personality questionnaire in nine languages, and performed Bayesian analysis of Gaussian Mixture Model, finding evidence for a deeper-rooted issue. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models, and are increasingly integrated in human life. We subsequently discuss the shortcomings of modern psychometrics, abstract it, and provide a framework for its species-neutral, substrate-free formulation.

Title: Segment Using Just One Example

• Authors: Pratik Vora, Sudipan Saha
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.07393
• Pdf URL: https://arxiv.org/pdf/2408.07393
• Copy Paste: [[2408.07393]] Segment Using Just One Example(https://arxiv.org/abs/2408.07393)
• Keywords: foundation model
• Abstract: Semantic segmentation is an important topic in computer vision with many relevant application in Earth observation. While supervised methods exist, the constraints of limited annotated data has encouraged development of unsupervised approaches. However, existing unsupervised methods resemble clustering and cannot be directly mapped to explicit target classes. In this paper, we deal with single shot semantic segmentation, where one example for the target class is provided, which is used to segment the target class from query/test images. Our approach exploits recently popular Segment Anything (SAM), a promptable foundation model. We specifically design several techniques to automatically generate prompts from the only example/key image in such a way that the segmentation is successfully achieved on a stitch or concatenation of the example/key and query/test images. Proposed technique does not involve any training phase and just requires one example image to grasp the concept. Furthermore, no text-based prompt is required for the proposed method. We evaluated the proposed techniques on building and car classes.

Title: Sum-Product-Set Networks

• Authors: Milan Papež, Martin Rektoris, Tomáš Pevný, Václav Šmídl
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07394
• Pdf URL: https://arxiv.org/pdf/2408.07394
• Copy Paste: [[2408.07394]] Sum-Product-Set Networks(https://arxiv.org/abs/2408.07394)
• Keywords: generative
• Abstract: Daily internet communication relies heavily on tree-structured graphs, embodied by popular data formats such as XML and JSON. However, many recent generative (probabilistic) models utilize neural networks to learn a probability distribution over undirected cyclic graphs. This assumption of a generic graph structure brings various computational challenges, and, more importantly, the presence of non-linearities in neural networks does not permit tractable probabilistic inference. We address these problems by proposing sum-product-set networks, an extension of probabilistic circuits from unstructured tensor data to tree-structured graph data. To this end, we use random finite sets to reflect a variable number of nodes and edges in the graph and to allow for exact and efficient inference. We demonstrate that our tractable model performs comparably to various intractable models based on neural networks.

Title: LLMI3D: Empowering LLM with 3D Perception from a Single 2D Image

• Authors: Fan Yang, Sicheng Zhao, Yanhao Zhang, Haoxiang Chen, Hui Chen, Wenbo Tang, Haonan Lu, Pengfei Xu, Zhenyu Yang, Jungong Han, Guiguang Ding
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07422
• Pdf URL: https://arxiv.org/pdf/2408.07422
• Copy Paste: [[2408.07422]] LLMI3D: Empowering LLM with 3D Perception from a Single 2D Image(https://arxiv.org/abs/2408.07422)
• Keywords: generative
• Abstract: Recent advancements in autonomous driving, augmented reality, robotics, and embodied intelligence have necessitated 3D perception algorithms. However, current 3D perception methods, particularly small models, struggle with processing logical reasoning, question-answering, and handling open scenario categories. On the other hand, generative multimodal large language models (MLLMs) excel in general capacity but underperform in 3D tasks, due to weak spatial and local object perception, poor text-based geometric numerical output, and inability to handle camera focal variations. To address these challenges, we propose the following solutions: Spatial-Enhanced Local Feature Mining for better spatial feature extraction, 3D Query Token-Derived Info Decoding for precise geometric regression, and Geometry Projection-Based 3D Reasoning for handling camera focal length variations. We employ parameter-efficient fine-tuning for a pre-trained MLLM and develop LLMI3D, a powerful 3D perception MLLM. Additionally, we have constructed the IG3D dataset, which provides fine-grained descriptions and question-answer annotations. Extensive experiments demonstrate that our LLMI3D achieves state-of-the-art performance, significantly outperforming existing methods.

Title: BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning

• Authors: Asif Hanif, Fahad Shamshad, Muhammad Awais, Muzammal Naseer, Fahad Shahbaz Khan, Karthik Nandakumar, Salman Khan, Rao Muhammad Anwer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07440
• Pdf URL: https://arxiv.org/pdf/2408.07440
• Copy Paste: [[2408.07440]] BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning(https://arxiv.org/abs/2408.07440)
• Keywords: foundation model
• Abstract: Medical foundation models are gaining prominence in the medical community for their ability to derive general representations from extensive collections of medical image-text pairs. Recent research indicates that these models are susceptible to backdoor attacks, which allow them to classify clean images accurately but fail when specific triggers are introduced. However, traditional backdoor attacks necessitate a considerable amount of additional data to maliciously pre-train a model. This requirement is often impractical in medical imaging applications due to the usual scarcity of data. Inspired by the latest developments in learnable prompts, this work introduces a method to embed a backdoor into the medical foundation model during the prompt learning phase. By incorporating learnable prompts within the text encoder and introducing imperceptible learnable noise trigger to the input images, we exploit the full capabilities of the medical foundation models (Med-FM). Our method, BAPLe, requires only a minimal subset of data to adjust the noise trigger and the text prompts for downstream tasks, enabling the creation of an effective backdoor attack. Through extensive experiments with four medical foundation models, each pre-trained on different modalities and evaluated across six downstream datasets, we demonstrate the efficacy of our approach. BAPLe achieves a high backdoor success rate across all models and datasets, outperforming the baseline backdoor attack methods. Our work highlights the vulnerability of Med-FMs towards backdoor attacks and strives to promote the safe adoption of Med-FMs before their deployment in real-world applications. Code is available at this https URL.

Title: Domain-invariant Representation Learning via Segment Anything Model for Blood Cell Classification

• Authors: Yongcheng Li, Lingcong Cai, Ying Lu, Cheng Lin, Yupeng Zhang, Jingyan Jiang, Genan Dai, Bowen Zhang, Jingzhou Cao, Xiangzhong Zhang, Xiaomao Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07467
• Pdf URL: https://arxiv.org/pdf/2408.07467
• Copy Paste: [[2408.07467]] Domain-invariant Representation Learning via Segment Anything Model for Blood Cell Classification(https://arxiv.org/abs/2408.07467)
• Keywords: foundation model
• Abstract: Accurate classification of blood cells is of vital significance in the diagnosis of hematological disorders. However, in real-world scenarios, domain shifts caused by the variability in laboratory procedures and settings, result in a rapid deterioration of the model's generalization performance. To address this issue, we propose a novel framework of domain-invariant representation learning (DoRL) via segment anything model (SAM) for blood cell classification. The DoRL comprises two main components: a LoRA-based SAM (LoRA-SAM) and a cross-domain autoencoder (CAE). The advantage of DoRL is that it can extract domain-invariant representations from various blood cell datasets in an unsupervised manner. Specifically, we first leverage the large-scale foundation model of SAM, fine-tuned with LoRA, to learn general image embeddings and segment blood cells. Additionally, we introduce CAE to learn domain-invariant representations across different-domain datasets while mitigating images' artifacts. To validate the effectiveness of domain-invariant representations, we employ five widely used machine learning classifiers to construct blood cell classification models. Experimental results on two public blood cell datasets and a private real dataset demonstrate that our proposed DoRL achieves a new state-of-the-art cross-domain performance, surpassing existing methods by a significant margin. The source code can be available at the URL (this https URL).

Title: One Step Diffusion-based Super-Resolution with Time-Aware Distillation

• Authors: Xiao He, Huaao Tang, Zhijun Tu, Junchao Zhang, Kun Cheng, Hanting Chen, Yong Guo, Mingrui Zhu, Nannan Wang, Xinbo Gao, Jie Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07476
• Pdf URL: https://arxiv.org/pdf/2408.07476
• Copy Paste: [[2408.07476]] One Step Diffusion-based Super-Resolution with Time-Aware Distillation(https://arxiv.org/abs/2408.07476)
• Keywords: diffusion
• Abstract: Diffusion-based image super-resolution (SR) methods have shown promise in reconstructing high-resolution images with fine details from low-resolution counterparts. However, these approaches typically require tens or even hundreds of iterative samplings, resulting in significant latency. Recently, techniques have been devised to enhance the sampling efficiency of diffusion-based SR models via knowledge distillation. Nonetheless, when aligning the knowledge of student and teacher models, these solutions either solely rely on pixel-level loss constraints or neglect the fact that diffusion models prioritize varying levels of information at different time steps. To accomplish effective and efficient image super-resolution, we propose a time-aware diffusion distillation method, named TAD-SR. Specifically, we introduce a novel score distillation strategy to align the data distribution between the outputs of the student and teacher models after minor noise perturbation. This distillation strategy enables the student network to concentrate more on the high-frequency details. Furthermore, to mitigate performance limitations stemming from distillation, we integrate a latent adversarial loss and devise a time-aware discriminator that leverages diffusion priors to effectively distinguish between real images and generated images. Extensive experiments conducted on synthetic and real-world datasets demonstrate that the proposed method achieves comparable or even superior performance compared to both previous state-of-the-art (SOTA) methods and the teacher model in just one sampling step. Codes are available at this https URL.

Title: DeCo: Decoupled Human-Centered Diffusion Video Editing with Motion Consistency

• Authors: Xiaojing Zhong, Xinyi Huang, Xiaofeng Yang, Guosheng Lin, Qingyao Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07481
• Pdf URL: https://arxiv.org/pdf/2408.07481
• Copy Paste: [[2408.07481]] DeCo: Decoupled Human-Centered Diffusion Video Editing with Motion Consistency(https://arxiv.org/abs/2408.07481)
• Keywords: diffusion
• Abstract: Diffusion models usher a new era of video editing, flexibly manipulating the video contents with text prompts. Despite the widespread application demand in editing human-centered videos, these models face significant challenges in handling complex objects like humans. In this paper, we introduce DeCo, a novel video editing framework specifically designed to treat humans and the background as separate editable targets, ensuring global spatial-temporal consistency by maintaining the coherence of each individual component. Specifically, we propose a decoupled dynamic human representation that utilizes a parametric human body prior to generate tailored humans while preserving the consistent motions as the original video. In addition, we consider the background as a layered atlas to apply text-guided image editing approaches on it. To further enhance the geometry and texture of humans during the optimization, we extend the calculation of score distillation sampling into normal space and image space. Moreover, we tackle inconsistent lighting between the edited targets by leveraging a lighting-aware video harmonizer, a problem previously overlooked in decompose-edit-combine approaches. Extensive qualitative and numerical experiments demonstrate that DeCo outperforms prior video editing methods in human-centered videos, especially in longer videos.

Title: Attention-Guided Perturbation for Unsupervised Image Anomaly Detection

• Authors: Tingfeng Huang, Yuxuan Cheng, Jingbo Xia, Rui Yu, Yuxuan Cai, Jinhai Xiang, Xinwei He, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07490
• Pdf URL: https://arxiv.org/pdf/2408.07490
• Copy Paste: [[2408.07490]] Attention-Guided Perturbation for Unsupervised Image Anomaly Detection(https://arxiv.org/abs/2408.07490)
• Keywords: anomaly
• Abstract: Reconstruction-based methods have significantly advanced modern unsupervised anomaly detection. However, the strong capacity of neural networks often violates the underlying assumptions by reconstructing abnormal samples well. To alleviate this issue, we present a simple yet effective reconstruction framework named Attention-Guided Pertuation Network (AGPNet), which learns to add perturbation noise with an attention mask, for accurate unsupervised anomaly detection. Specifically, it consists of two branches, \ie, a plain reconstruction branch and an auxiliary attention-based perturbation branch. The reconstruction branch is simply a plain reconstruction network that learns to reconstruct normal samples, while the auxiliary branch aims to produce attention masks to guide the noise perturbation process for normal samples from easy to hard. By doing so, we are expecting to synthesize hard yet more informative anomalies for training, which enable the reconstruction branch to learn important inherent normal patterns both comprehensively and efficiently. Extensive experiments are conducted on three popular benchmarks covering MVTec-AD, VisA, and MVTec-3D, and show that our framework obtains leading anomaly detection performance under various setups including few-shot, one-class, and multi-class setups.

Title: Cross-Platform Video Person ReID: A New Benchmark Dataset and Adaptation Approach

• Authors: Shizhou Zhang, Wenlong Luo, De Cheng, Qingchun Yang, Lingyan Ran, Yinghui Xing, Yanning Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07500
• Pdf URL: https://arxiv.org/pdf/2408.07500
• Copy Paste: [[2408.07500]] Cross-Platform Video Person ReID: A New Benchmark Dataset and Adaptation Approach(https://arxiv.org/abs/2408.07500)
• Keywords: foundation model
• Abstract: In this paper, we construct a large-scale benchmark dataset for Ground-to-Aerial Video-based person Re-Identification, named G2A-VReID, which comprises 185,907 images and 5,576 tracklets, featuring 2,788 distinct identities. To our knowledge, this is the first dataset for video ReID under Ground-to-Aerial scenarios. G2A-VReID dataset has the following characteristics: 1) Drastic view changes; 2) Large number of annotated identities; 3) Rich outdoor scenarios; 4) Huge difference in resolution. Additionally, we propose a new benchmark approach for cross-platform ReID by transforming the cross-platform visual alignment problem into visual-semantic alignment through vision-language model (i.e., CLIP) and applying a parameter-efficient Video Set-Level-Adapter module to adapt image-based foundation model to video ReID tasks, termed VSLA-CLIP. Besides, to further reduce the great discrepancy across the platforms, we also devise the platform-bridge prompts for efficient visual feature alignment. Extensive experiments demonstrate the superiority of the proposed method on all existing video ReID datasets and our proposed G2A-VReID dataset.

Title: Large Language Models Know What Makes Exemplary Contexts

• Authors: Quanyu Long, Jianda Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07505
• Pdf URL: https://arxiv.org/pdf/2408.07505
• Copy Paste: [[2408.07505]] Large Language Models Know What Makes Exemplary Contexts(https://arxiv.org/abs/2408.07505)
• Keywords: in-context
• Abstract: In-context learning (ICL) has proven to be a significant capability with the advancement of Large Language models (LLMs). By instructing LLMs using few-shot demonstrative examples, ICL enables them to perform a wide range of tasks without needing to update millions of parameters. This paper presents a unified framework for LLMs that allows them to self-select influential in-context examples to compose their contexts; self-rank candidates with different demonstration compositions; self-optimize the demonstration selection and ordering through reinforcement learning. Specifically, our method designs a parameter-efficient retrieval head that generates the optimized demonstration after training with rewards from LLM's own preference. Experimental results validate the proposed method's effectiveness in enhancing ICL performance. Additionally, our approach effectively identifies and selects the most representative examples for the current task, and includes more diversity in retrieval.

Title: Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach

• Authors: Yarin Bar, Shalev Shaer, Yaniv Romano
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.07511
• Pdf URL: https://arxiv.org/pdf/2408.07511
• Copy Paste: [[2408.07511]] Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach(https://arxiv.org/abs/2408.07511)
• Keywords: self-supervised
• Abstract: We present a novel approach for test-time adaptation via online self-training, consisting of two components. First, we introduce a statistical framework that detects distribution shifts in the classifier's entropy values obtained on a stream of unlabeled samples. Second, we devise an online adaptation mechanism that utilizes the evidence of distribution shifts captured by the detection tool to dynamically update the classifier's parameters. The resulting adaptation process drives the distribution of test entropy values obtained from the self-trained classifier to match those of the source domain, building invariance to distribution shifts. This approach departs from the conventional self-training method, which focuses on minimizing the classifier's entropy. Our approach combines concepts in betting martingales and online learning to form a detection tool capable of quickly reacting to distribution shifts. We then reveal a tight relation between our adaptation scheme and optimal transport, which forms the basis of our novel self-supervised loss. Experimental results demonstrate that our approach improves test-time accuracy under distribution shifts while maintaining accuracy and calibration in their absence, outperforming leading entropy minimization methods across various scenarios.

Title: CNN-JEPA: Self-Supervised Pretraining Convolutional Neural Networks Using Joint Embedding Predictive Architecture

• Authors: András Kalapos, Bálint Gyires-Tóth
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07514
• Pdf URL: https://arxiv.org/pdf/2408.07514
• Copy Paste: [[2408.07514]] CNN-JEPA: Self-Supervised Pretraining Convolutional Neural Networks Using Joint Embedding Predictive Architecture(https://arxiv.org/abs/2408.07514)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) has become an important approach in pretraining large neural networks, enabling unprecedented scaling of model and dataset sizes. While recent advances like I-JEPA have shown promising results for Vision Transformers, adapting such methods to Convolutional Neural Networks (CNNs) presents unique challenges. In this paper, we introduce CNN-JEPA, a novel SSL method that successfully applies the joint embedding predictive architecture approach to CNNs. Our method incorporates a sparse CNN encoder to handle masked inputs, a fully convolutional predictor using depthwise separable convolutions, and an improved masking strategy. We demonstrate that CNN-JEPA outperforms I-JEPA with ViT architectures on ImageNet-100, achieving 73.3% linear top-1 accuracy with a standard ResNet-50 encoder. Compared to other CNN-based SSL methods, CNN-JEPA requires 17-35% less training time for the same number of epochs and approaches the linear and k-NN top-1 accuracies of BYOL, SimCLR, and VICReg. Our approach offers a simpler, more efficient alternative to existing SSL methods for CNNs, requiring minimal augmentations and no separate projector network.

Title: DIffSteISR: Harnessing Diffusion Prior for Superior Real-world Stereo Image Super-Resolution

• Authors: Yuanbo Zhou, Xinlin Zhang, Wei Deng, Tao Wang, Tao Tan, Qinquan Gao, Tong Tong
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.07516
• Pdf URL: https://arxiv.org/pdf/2408.07516
• Copy Paste: [[2408.07516]] DIffSteISR: Harnessing Diffusion Prior for Superior Real-world Stereo Image Super-Resolution(https://arxiv.org/abs/2408.07516)
• Keywords: diffusion
• Abstract: We introduce DiffSteISR, a pioneering framework for reconstructing real-world stereo images. DiffSteISR utilizes the powerful prior knowledge embedded in pre-trained text-to-image model to efficiently recover the lost texture details in low-resolution stereo images. Specifically, DiffSteISR implements a time-aware stereo cross attention with temperature adapter (TASCATA) to guide the diffusion process, ensuring that the generated left and right views exhibit high texture consistency thereby reducing disparity error between the super-resolved images and the ground truth (GT) images. Additionally, a stereo omni attention control network (SOA ControlNet) is proposed to enhance the consistency of super-resolved images with GT images in the pixel, perceptual, and distribution space. Finally, DiffSteISR incorporates a stereo semantic extractor (SSE) to capture unique viewpoint soft semantic information and shared hard tag semantic information, thereby effectively improving the semantic accuracy and consistency of the generated left and right images. Extensive experimental results demonstrate that DiffSteISR accurately reconstructs natural and precise textures from low-resolution stereo images while maintaining a high consistency of semantic and texture between the left and right views.

Title: Whitening Consistently Improves Self-Supervised Learning

• Authors: András Kalapos, Bálint Gyires-Tóth
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07519
• Pdf URL: https://arxiv.org/pdf/2408.07519
• Copy Paste: [[2408.07519]] Whitening Consistently Improves Self-Supervised Learning(https://arxiv.org/abs/2408.07519)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) has been shown to be a powerful approach for learning visual representations. In this study, we propose incorporating ZCA whitening as the final layer of the encoder in self-supervised learning to enhance the quality of learned features by normalizing and decorrelating them. Although whitening has been utilized in SSL in previous works, its potential to universally improve any SSL model has not been explored. We demonstrate that adding whitening as the last layer of SSL pretrained encoders is independent of the self-supervised learning method and encoder architecture, thus it improves performance for a wide range of SSL methods across multiple encoder architectures and datasets. Our experiments show that whitening is capable of improving linear and k-NN probing accuracy by 1-5%. Additionally, we propose metrics that allow for a comprehensive analysis of the learned features, provide insights into the quality of the representations and help identify collapse patterns.

Title: DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model

• Authors: Erez Yosef, Raja Giryes
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.07541
• Pdf URL: https://arxiv.org/pdf/2408.07541
• Copy Paste: [[2408.07541]] DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model(https://arxiv.org/abs/2408.07541)
• Keywords: diffusion
• Abstract: The flat lensless camera design reduces the camera size and weight significantly. In this design, the camera lens is replaced by another optical element that interferes with the incoming light. The image is recovered from the raw sensor measurements using a reconstruction algorithm. Yet, the quality of the reconstructed images is not satisfactory. To mitigate this, we propose utilizing a pre-trained diffusion model with a control network and a learned separable transformation for reconstruction. This allows us to build a prototype flat camera with high-quality imaging, presenting state-of-the-art results in both terms of quality and perceptuality. We demonstrate its ability to leverage also textual descriptions of the captured scene to further enhance reconstruction. Our reconstruction method which leverages the strong capabilities of a pre-trained diffusion model can be used in other imaging systems for improved reconstruction results.

Title: PolyCL: Contrastive Learning for Polymer Representation Learning via Explicit and Implicit Augmentations

• Authors: Jiajun Zhou, Yijie Yang, Austin M. Mroz, Kim E. Jelfs
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07556
• Pdf URL: https://arxiv.org/pdf/2408.07556
• Copy Paste: [[2408.07556]] PolyCL: Contrastive Learning for Polymer Representation Learning via Explicit and Implicit Augmentations(https://arxiv.org/abs/2408.07556)
• Keywords: self-supervised
• Abstract: Polymers play a crucial role in a wide array of applications due to their diverse and tunable properties. Establishing the relationship between polymer representations and their properties is crucial to the computational design and screening of potential polymers via machine learning. The quality of the representation significantly influences the effectiveness of these computational methods. Here, we present a self-supervised contrastive learning paradigm, PolyCL, for learning high-quality polymer representation without the need for labels. Our model combines explicit and implicit augmentation strategies for improved learning performance. The results demonstrate that our model achieves either better, or highly competitive, performances on transfer learning tasks as a feature extractor without an overcomplicated training strategy or hyperparameter optimisation. Further enhancing the efficacy of our model, we conducted extensive analyses on various augmentation combinations used in contrastive learning. This led to identifying the most effective combination to maximise PolyCL's performance.

Title: Disentangle and denoise: Tackling context misalignment for video moment retrieval

• Authors: Kaijing Ma, Han Fang, Xianghao Zang, Chao Ban, Lanxiang Zhou, Zhongjiang He, Yongxiang Li, Hao Sun, Zerun Feng, Xingsong Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07600
• Pdf URL: https://arxiv.org/pdf/2408.07600
• Copy Paste: [[2408.07600]] Disentangle and denoise: Tackling context misalignment for video moment retrieval(https://arxiv.org/abs/2408.07600)
• Keywords: in-context
• Abstract: Video Moment Retrieval, which aims to locate in-context video moments according to a natural language query, is an essential task for cross-modal grounding. Existing methods focus on enhancing the cross-modal interactions between all moments and the textual description for video understanding. However, constantly interacting with all locations is unreasonable because of uneven semantic distribution across the timeline and noisy visual backgrounds. This paper proposes a cross-modal Context Denoising Network (CDNet) for accurate moment retrieval by disentangling complex correlations and denoising irrelevant dynamics.Specifically, we propose a query-guided semantic disentanglement (QSD) to decouple video moments by estimating alignment levels according to the global and fine-grained correlation. A Context-aware Dynamic Denoisement (CDD) is proposed to enhance understanding of aligned spatial-temporal details by learning a group of query-relevant offsets. Extensive experiments on public benchmarks demonstrate that the proposed CDNet achieves state-of-the-art performances.

Title: Latent Anomaly Detection Through Density Matrices

• Authors: Joseph Gallego-Mejia, Oscar Bustos-Brinez, Fabio A. González
• Subjects: cs.LG, quant-ph, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.07623
• Pdf URL: https://arxiv.org/pdf/2408.07623
• Copy Paste: [[2408.07623]] Latent Anomaly Detection Through Density Matrices(https://arxiv.org/abs/2408.07623)
• Keywords: anomaly
• Abstract: This paper introduces a novel anomaly detection framework that combines the robust statistical principles of density-estimation-based anomaly detection methods with the representation-learning capabilities of deep learning models. The method originated from this framework is presented in two different versions: a shallow approach employing a density-estimation model based on adaptive Fourier features and density matrices, and a deep approach that integrates an autoencoder to learn a low-dimensional representation of the data. By estimating the density of new samples, both methods are able to find normality scores. The methods can be seamlessly integrated into an end-to-end architecture and optimized using gradient-based optimization techniques. To evaluate their performance, extensive experiments were conducted on various benchmark datasets. The results demonstrate that both versions of the method can achieve comparable or superior performance when compared to other state-of-the-art methods. Notably, the shallow approach performs better on datasets with fewer dimensions, while the autoencoder-based approach shows improved performance on datasets with higher dimensions.

Title: Detecting Near-Duplicate Face Images

• Authors: Sudipta Banerjee, Arun Ross
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07689
• Pdf URL: https://arxiv.org/pdf/2408.07689
• Copy Paste: [[2408.07689]] Detecting Near-Duplicate Face Images(https://arxiv.org/abs/2408.07689)
• Keywords: generative
• Abstract: Near-duplicate images are often generated when applying repeated photometric and geometric transformations that produce imperceptible variants of the original image. Consequently, a deluge of near-duplicates can be circulated online posing copyright infringement concerns. The concerns are more severe when biometric data is altered through such nuanced transformations. In this work, we address the challenge of near-duplicate detection in face images by, firstly, identifying the original image from a set of near-duplicates and, secondly, deducing the relationship between the original image and the near-duplicates. We construct a tree-like structure, called an Image Phylogeny Tree (IPT) using a graph-theoretic approach to estimate the relationship, i.e., determine the sequence in which they have been generated. We further extend our method to create an ensemble of IPTs known as Image Phylogeny Forests (IPFs). We rigorously evaluate our method to demonstrate robustness across other modalities, unseen transformations by latest generative models and IPT configurations, thereby significantly advancing the state-of-the-art performance by 42% on IPF reconstruction accuracy.