2024-07-26

Title: Learning from Memory: Non-Parametric Memory Augmented Self-Supervised Learning of Visual Features

• Authors: Thalles Silva, Helio Pedrini, Adín Ramírez Rivera
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17486
• Pdf URL: https://arxiv.org/pdf/2407.17486
• Copy Paste: [[2407.17486]] Learning from Memory: Non-Parametric Memory Augmented Self-Supervised Learning of Visual Features(https://arxiv.org/abs/2407.17486)
• Keywords: self-supervised
• Abstract: This paper introduces a novel approach to improving the training stability of self-supervised learning (SSL) methods by leveraging a non-parametric memory of seen concepts. The proposed method involves augmenting a neural network with a memory component to stochastically compare current image views with previously encountered concepts. Additionally, we introduce stochastic memory blocks to regularize training and enforce consistency between image views. We extensively benchmark our method on many vision tasks, such as linear probing, transfer learning, low-shot classification, and image retrieval on many datasets. The experimental results consolidate the effectiveness of the proposed approach in achieving stable SSL training without additional regularizers while learning highly transferable representations and requiring less computing time and resources.

Title: Robust Adaptation of Foundation Models with Black-Box Visual Prompting

• Authors: Changdae Oh, Gyeongdeok Seo, Geunyoung Jung, Zhi-Qi Cheng, Hosik Choi, Jiyoung Jung, Kyungwoo Song
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17491
• Pdf URL: https://arxiv.org/pdf/2407.17491
• Copy Paste: [[2407.17491]] Robust Adaptation of Foundation Models with Black-Box Visual Prompting(https://arxiv.org/abs/2407.17491)
• Keywords: foundation model
• Abstract: With the surge of large-scale pre-trained models (PTMs), adapting these models to numerous downstream tasks becomes a crucial problem. Consequently, parameter-efficient transfer learning (PETL) of large models has grasped huge attention. While PETL methods show impressive performance, they commonly rely on two optimistic assumptions: 1) the entire parameters of a PTM are available, and 2) a sufficiently large memory capacity is equipped for caching all the intermediate activations to compute gradients. However, in most real-world applications, PTMs are served as black-box APIs or proprietary software without explicit parameter accessibility. Besides, it is hard to meet a large memory requirement for modern PTMs. This work proposes black-box visual prompting (BlackVIP), which efficiently adapts the PTMs without knowledge about model architectures and parameters. BlackVIP has two components; 1) Coordinator and 2) simultaneous perturbation stochastic approximation with gradient correction (SPSA-GC). The Coordinator designs input-dependent visual prompts, which allow the target PTM to adapt in the wild. SPSA-GC efficiently estimates the gradient of PTM to update the Coordinator. Besides, we propose a variant, BlackVIP-SE, which significantly reduces the runtime and computational cost of BlackVIP. Extensive experiments on 19 datasets demonstrate that BlackVIPs enable robust adaptation to diverse domains and tasks with minimal memory requirements. We further provide theoretical analysis on the generalization of visual prompting methods by presenting their connection to the certified robustness of randomized smoothing.

Title: ReDiFine: Reusable Diffusion Finetuning for Mitigating Degradation in the Chain of Diffusion

• Authors: Youngseok Yoon, Dainong Hu, Iain Weissburg, Yao Qin, Haewon Jeong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17493
• Pdf URL: https://arxiv.org/pdf/2407.17493
• Copy Paste: [[2407.17493]] ReDiFine: Reusable Diffusion Finetuning for Mitigating Degradation in the Chain of Diffusion(https://arxiv.org/abs/2407.17493)
• Keywords: diffusion, generative
• Abstract: Diffusion models have achieved tremendous improvements in generative modeling for images, enabling high-quality generation that is indistinguishable by humans from real images. The qualities of images have reached a threshold at which we can reuse synthetic images for training machine learning models again. This attracts the area as it can relieve the high cost of data collection and fundamentally solve many problems in data-limited areas. In this paper, we focus on a practical scenario in which pretrained text-to-image diffusion models are iteratively finetuned using a set of synthetic images, which we call the Chain of Diffusion. Finetuned models generate images that are used for the next iteration of finetuning. We first demonstrate how these iterative processes result in severe degradation in image qualities. Thorough investigations reveal the most impactful factor for the degradation, and we propose finetuning and generation strategies that can effectively resolve the degradation. Our method, Reusable Diffusion Finetuning (ReDiFine), combines condition drop finetuning and CFG scheduling to maintain the qualities of generated images throughout iterations. ReDiFine works effectively for multiple datasets and models without further hyperparameter search, making synthetic images reusable to finetune future generative models.

Title: Generative artificial intelligence in dentistry: Current approaches and future challenges

• Authors: Fabián Villena, Claudia Véliz, Rosario García-Huidobro, Sebastián Aguayo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17532
• Pdf URL: https://arxiv.org/pdf/2407.17532
• Copy Paste: [[2407.17532]] Generative artificial intelligence in dentistry: Current approaches and future challenges(https://arxiv.org/abs/2407.17532)
• Keywords: generative
• Abstract: Artificial intelligence (AI) has become a commodity for people because of the advent of generative AI (GenAI) models that bridge the usability gap of AI by providing a natural language interface to interact with complex models. These GenAI models range from text generation - such as two-way chat systems - to the generation of image or video from textual descriptions input by a user. These advancements in AI have impacted Dentistry in multiple aspects. In dental education, the student now has the opportunity to solve a plethora of questions by only prompting a GenAI model and have the answer in a matter of seconds. GenAI models can help us deliver better patient healthcare by helping practitioners gather knowledge quickly and efficiently. Finally, GenAI can also be used in dental research, where the applications range from new drug discovery to assistance in academic writing. In this review, we first define GenAI models and describe their multiple generation modalities; then, we explain and discuss their current and potential applications in Dentistry; and finally, we describe the challenges these new technologies impose in our area.

Title: Diffusion Models for Multi-Task Generative Modeling

• Authors: Changyou Chen, Han Ding, Bunyamin Sisman, Yi Xu, Ouye Xie, Benjamin Z. Yao, Son Dinh Tran, Belinda Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17571
• Pdf URL: https://arxiv.org/pdf/2407.17571
• Copy Paste: [[2407.17571]] Diffusion Models for Multi-Task Generative Modeling(https://arxiv.org/abs/2407.17571)
• Keywords: diffusion, generative
• Abstract: Diffusion-based generative modeling has been achieving state-of-the-art results on various generation tasks. Most diffusion models, however, are limited to a single-generation modeling. Can we generalize diffusion models with the ability of multi-modal generative training for more generalizable modeling? In this paper, we propose a principled way to define a diffusion model by constructing a unified multi-modal diffusion model in a common diffusion space. We define the forward diffusion process to be driven by an information aggregation from multiple types of task-data, e.g., images for a generation task and labels for a classification task. In the reverse process, we enforce information sharing by parameterizing a shared backbone denoising network with additional modality-specific decoder heads. Such a structure can simultaneously learn to generate different types of multi-modal data with a multi-task loss, which is derived from a new multi-modal variational lower bound that generalizes the standard diffusion model. We propose several multimodal generation settings to verify our framework, including image transition, masked-image training, joint image-label and joint image-representation generative modeling. Extensive experimental results on ImageNet indicate the effectiveness of our framework for various multi-modal generative modeling, which we believe is an important research direction worthy of more future explorations.

Title: PEEKABOO: Hiding parts of an image for unsupervised object localization

• Authors: Hasib Zunair, A. Ben Hamza
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17628
• Pdf URL: https://arxiv.org/pdf/2407.17628
• Copy Paste: [[2407.17628]] PEEKABOO: Hiding parts of an image for unsupervised object localization(https://arxiv.org/abs/2407.17628)
• Keywords: self-supervised
• Abstract: Localizing objects in an unsupervised manner poses significant challenges due to the absence of key visual information such as the appearance, type and number of objects, as well as the lack of labeled object classes typically available in supervised settings. While recent approaches to unsupervised object localization have demonstrated significant progress by leveraging self-supervised visual representations, they often require computationally intensive training processes, resulting in high resource demands in terms of computation, learnable parameters, and data. They also lack explicit modeling of visual context, potentially limiting their accuracy in object localization. To tackle these challenges, we propose a single-stage learning framework, dubbed PEEKABOO, for unsupervised object localization by learning context-based representations at both the pixel- and shape-level of the localized objects through image masking. The key idea is to selectively hide parts of an image and leverage the remaining image information to infer the location of objects without explicit supervision. The experimental results, both quantitative and qualitative, across various benchmark datasets, demonstrate the simplicity, effectiveness and competitive performance of our approach compared to state-of-the-art methods in both single object discovery and unsupervised salient object detection tasks. Code and pre-trained models are available at: this https URL

Title: Generative Learning for Simulation of US Army Vehicle Faults

• Authors: Patrick Kuiper, Sirui Lin, Jose Blanchet, Vahid Tarokh
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.17654
• Pdf URL: https://arxiv.org/pdf/2407.17654
• Copy Paste: [[2407.17654]] Generative Learning for Simulation of US Army Vehicle Faults(https://arxiv.org/abs/2407.17654)
• Keywords: generative
• Abstract: We develop a novel generative model to simulate vehicle health and forecast faults, conditioned on practical operational considerations. The model, trained on data from the US Army's Predictive Logistics program, aims to support predictive maintenance. It forecasts faults far enough in advance to execute a maintenance intervention before a breakdown occurs. The model incorporates real-world factors that affect vehicle health. It also allows us to understand the vehicle's condition by analyzing operating data, and characterizing each vehicle into discrete states. Importantly, the model predicts the time to first fault with high accuracy. We compare its performance to other models and demonstrate its successful training.

Title: Explaining the Model, Protecting Your Data: Revealing and Mitigating the Data Privacy Risks of Post-Hoc Model Explanations via Membership Inference

• Authors: Catherine Huang, Martin Pawelczyk, Himabindu Lakkaraju
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.17663
• Pdf URL: https://arxiv.org/pdf/2407.17663
• Copy Paste: [[2407.17663]] Explaining the Model, Protecting Your Data: Revealing and Mitigating the Data Privacy Risks of Post-Hoc Model Explanations via Membership Inference(https://arxiv.org/abs/2407.17663)
• Keywords: foundation model
• Abstract: Predictive machine learning models are becoming increasingly deployed in high-stakes contexts involving sensitive personal data; in these contexts, there is a trade-off between model explainability and data privacy. In this work, we push the boundaries of this trade-off: with a focus on foundation models for image classification fine-tuning, we reveal unforeseen privacy risks of post-hoc model explanations and subsequently offer mitigation strategies for such risks. First, we construct VAR-LRT and L1/L2-LRT, two new membership inference attacks based on feature attribution explanations that are significantly more successful than existing explanation-leveraging attacks, particularly in the low false-positive rate regime that allows an adversary to identify specific training set members with confidence. Second, we find empirically that optimized differentially private fine-tuning substantially diminishes the success of the aforementioned attacks, while maintaining high model accuracy. We carry out a systematic empirical investigation of our 2 new attacks with 5 vision transformer architectures, 5 benchmark datasets, 4 state-of-the-art post-hoc explanation methods, and 4 privacy strength settings.

Title: Unsqueeze [CLS] Bottleneck to Learn Rich Representations

• Authors: Qing Su, Shihao Ji
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17671
• Pdf URL: https://arxiv.org/pdf/2407.17671
• Copy Paste: [[2407.17671]] Unsqueeze [CLS] Bottleneck to Learn Rich Representations(https://arxiv.org/abs/2407.17671)
• Keywords: self-supervised
• Abstract: Distillation-based self-supervised learning typically leads to more compressed representations due to its radical clustering process and the implementation of a sharper target distribution. To overcome this limitation and preserve more information from input, we introduce UDI, conceptualized as Unsqueezed Distillation-based self-supervised learning (SSL). UDI enriches the learned representation by encouraging multimodal prediction distilled from a consolidated profile of local predictions that are derived via stratified sampling. Our evaluations show that UDI not only promotes semantically meaningful representations at instance level, delivering superior or competitive results to state-of-the-art SSL methods in image classification, but also effectively preserves the nuisance of input, which yields significant improvement in dense prediction tasks, including object detection and segmentation. Additionally, UDI performs competitively in low-shot image classification, improving the scalability of joint-embedding pipelines. Various visualizations and ablation studies are presented to further elucidate the mechanisms behind UDI. Our source code is available at this https URL.

Title: Synthetic High-resolution Cryo-EM Density Maps with Generative Adversarial Networks

• Authors: Chenwei Zhang, Anne Condon, Khanh Dao Duc
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2407.17674
• Pdf URL: https://arxiv.org/pdf/2407.17674
• Copy Paste: [[2407.17674]] Synthetic High-resolution Cryo-EM Density Maps with Generative Adversarial Networks(https://arxiv.org/abs/2407.17674)
• Keywords: generative
• Abstract: Generating synthetic cryogenic electron microscopy (cryo-EM) 3D density maps from molecular structures has potential important applications in structural biology. Yet existing simulation-based methods cannot mimic all the complex features present in experimental maps, such as secondary structure elements. As an alternative, we propose struc2mapGAN, a novel data-driven method that employs a generative adversarial network (GAN) to produce high-resolution experimental-like density maps from molecular structures. More specifically, struc2mapGAN uses a U-Net++ architecture as the generator, with an additional L1 loss term and further processing of raw experimental maps to enhance learning efficiency. While struc2mapGAN can promptly generate maps after training, we demonstrate that it outperforms existing simulation-based methods for a wide array of tested maps and across various evaluation metrics. Our code is available at this https URL.

Title: Transformers on Markov Data: Constant Depth Suffices

• Authors: Nived Rajaraman, Marco Bondaschi, Kannan Ramchandran, Michael Gastpar, Ashok Vardhan Makkuva
• Subjects: cs.LG, cs.CL, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.17686
• Pdf URL: https://arxiv.org/pdf/2407.17686
• Copy Paste: [[2407.17686]] Transformers on Markov Data: Constant Depth Suffices(https://arxiv.org/abs/2407.17686)
• Keywords: generative, in-context
• Abstract: Attention-based transformers have been remarkably successful at modeling generative processes across various domains and modalities. In this paper, we study the behavior of transformers on data drawn from \kth Markov processes, where the conditional distribution of the next symbol in a sequence depends on the previous $k$ symbols observed. We observe a surprising phenomenon empirically which contradicts previous findings: when trained for sufficiently long, a transformer with a fixed depth and $1$ head per layer is able to achieve low test loss on sequences drawn from \kth Markov sources, even as $k$ grows. Furthermore, this low test loss is achieved by the transformer's ability to represent and learn the in-context conditional empirical distribution. On the theoretical side, our main result is that a transformer with a single head and three layers can represent the in-context conditional empirical distribution for \kth Markov sources, concurring with our empirical observations. Along the way, we prove that \textit{attention-only} transformers with $O(\log_2(k))$ layers can represent the in-context conditional empirical distribution by composing induction heads to track the previous $k$ symbols in the sequence. These results provide more insight into our current understanding of the mechanisms by which transformers learn to capture context, by understanding their behavior on Markov sources.

Title: ALMRR: Anomaly Localization Mamba on Industrial Textured Surface with Feature Reconstruction and Refinement

• Authors: Shichen Qu, Xian Tao, Zhen Qu, Xinyi Gong, Zhengtao Zhang, Mukesh Prasad
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17705
• Pdf URL: https://arxiv.org/pdf/2407.17705
• Copy Paste: [[2407.17705]] ALMRR: Anomaly Localization Mamba on Industrial Textured Surface with Feature Reconstruction and Refinement(https://arxiv.org/abs/2407.17705)
• Keywords: anomaly
• Abstract: Unsupervised anomaly localization on industrial textured images has achieved remarkable results through reconstruction-based methods, yet existing approaches based on image reconstruction and feature reconstruc-tion each have their own shortcomings. Firstly, image-based methods tend to reconstruct both normal and anomalous regions well, which lead to over-generalization. Feature-based methods contain a large amount of distin-guishable semantic information, however, its feature structure is redundant and lacks anomalous information, which leads to significant reconstruction errors. In this paper, we propose an Anomaly Localization method based on Mamba with Feature Reconstruction and Refinement(ALMRR) which re-constructs semantic features based on Mamba and then refines them through a feature refinement module. To equip the model with prior knowledge of anomalies, we enhance it by adding artificially simulated anomalies to the original images. Unlike image reconstruction or repair, the features of synthesized defects are repaired along with those of normal areas. Finally, the aligned features containing rich semantic information are fed in-to the refinement module to obtain the anomaly map. Extensive experiments have been conducted on the MVTec-AD-Textured dataset and other real-world industrial dataset, which has demonstrated superior performance com-pared to state-of-the-art (SOTA) methods.

Title: Multi-modal Data Binding for Survival Analysis Modeling with Incomplete Data and Annotations

• Authors: Linhao Qu, Dan Huang, Shaoting Zhang, Xiaosong Wang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17726
• Pdf URL: https://arxiv.org/pdf/2407.17726
• Copy Paste: [[2407.17726]] Multi-modal Data Binding for Survival Analysis Modeling with Incomplete Data and Annotations(https://arxiv.org/abs/2407.17726)
• Keywords: foundation model
• Abstract: Survival analysis stands as a pivotal process in cancer treatment research, crucial for predicting patient survival rates accurately. Recent advancements in data collection techniques have paved the way for enhancing survival predictions by integrating information from multiple modalities. However, real-world scenarios often present challenges with incomplete data, particularly when dealing with censored survival labels. Prior works have addressed missing modalities but have overlooked incomplete labels, which can introduce bias and limit model efficacy. To bridge this gap, we introduce a novel framework that simultaneously handles incomplete data across modalities and censored survival labels. Our approach employs advanced foundation models to encode individual modalities and align them into a universal representation space for seamless fusion. By generating pseudo labels and incorporating uncertainty, we significantly enhance predictive accuracy. The proposed method demonstrates outstanding prediction accuracy in two survival analysis tasks on both employed datasets. This innovative approach overcomes limitations associated with disparate modalities and improves the feasibility of comprehensive survival analysis using multiple large foundation models.

Title: Enhancing Eye Disease Diagnosis with Deep Learning and Synthetic Data Augmentation

• Authors: Saideep Kilaru, Kothamasu Jayachandra, Tanishka Yagneshwar, Suchi Kumari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17755
• Pdf URL: https://arxiv.org/pdf/2407.17755
• Copy Paste: [[2407.17755]] Enhancing Eye Disease Diagnosis with Deep Learning and Synthetic Data Augmentation(https://arxiv.org/abs/2407.17755)
• Keywords: generative
• Abstract: In recent years, the focus is on improving the diagnosis of diabetic retinopathy (DR) using machine learning and deep learning technologies. Researchers have explored various approaches, including the use of high-definition medical imaging, AI-driven algorithms such as convolutional neural networks (CNNs) and generative adversarial networks (GANs). Among all the available tools, CNNs have emerged as a preferred tool due to their superior classification accuracy and efficiency. Although the accuracy of CNNs is comparatively better but it can be improved by introducing some hybrid models by combining various machine learning and deep learning models. Therefore, in this paper, an ensemble learning technique is proposed for early detection and management of DR with higher accuracy. The proposed model is tested on the APTOS dataset and it is showing supremacy on the validation accuracy ($99\%)$ in comparison to the previous models. Hence, the model can be helpful for early detection and treatment of the DR, thereby enhancing the overall quality of care for affected individuals.

Title: Mpox Detection Advanced: Rapid Epidemic Response Through Synthetic Data

• Authors: Yudara Kularathne, Prathapa Janitha, Sithira Ambepitiya, Prarththanan Sothyrajah, Thanveer Ahamed, Dinuka Wijesundara
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17762
• Pdf URL: https://arxiv.org/pdf/2407.17762
• Copy Paste: [[2407.17762]] Mpox Detection Advanced: Rapid Epidemic Response Through Synthetic Data(https://arxiv.org/abs/2407.17762)
• Keywords: diffusion
• Abstract: Rapid development of disease detection models using computer vision is crucial in responding to medical emergencies, such as epidemics or bioterrorism events. Traditional data collection methods are often too slow in these scenarios, requiring innovative approaches for quick, reliable model generation from minimal data. Our study introduces a novel approach by constructing a comprehensive computer vision model to detect Mpox lesions using only synthetic data. Initially, these models generated a diverse set of synthetic images representing Mpox lesions on various body parts (face, back, chest, leg, neck, arm) across different skin tones as defined by the Fitzpatrick scale (fair, brown, dark skin). Subsequently, we trained and tested a vision model with this synthetic dataset to evaluate the diffusion models' efficacy in producing high-quality training data and its impact on the vision model's medical image recognition performance. The results were promising; the vision model achieved a 97% accuracy rate, with 96% precision and recall for Mpox cases, and similarly high metrics for normal and other skin disorder cases, demonstrating its ability to correctly identify true positives and minimize false positives. The model achieved an F1-Score of 96% for Mpox cases and 98% for normal and other skin disorders, reflecting a balanced precision-recall relationship, thus ensuring reliability and robustness in its predictions. Our proposed SynthVision methodology indicates the potential to develop accurate computer vision models with minimal data input for future medical emergencies.

Title: Unified Lexical Representation for Interpretable Visual-Language Alignment

• Authors: Yifan Li, Yikai Wang, Yanwei Fu, Dongyu Ru, Zheng Zhang, Tong He
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17827
• Pdf URL: https://arxiv.org/pdf/2407.17827
• Copy Paste: [[2407.17827]] Unified Lexical Representation for Interpretable Visual-Language Alignment(https://arxiv.org/abs/2407.17827)
• Keywords: generative, in-context
• Abstract: Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work. Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores. On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words. However, lexical representations is difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively. In this paper, we introduce LexVLA, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design. We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability. To avoid the false discovery, we propose an overuse penalty to refrain the lexical representation from falsely frequently activating meaningless words. We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on modest multi-modal dataset and avoid intricate training configurations. On cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g., 1.1B data, including CC-12M). We conduct extensive experiments to analyze LexVLA.

Title: On the Opportunities of (Re)-Exploring Atmospheric Science by Foundation Models: A Case Study

• Authors: Lujia Zhang, Hanzhe Cui, Yurong Song, Chenyue Li, Binhang Yuan, Mengqian Lu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17842
• Pdf URL: https://arxiv.org/pdf/2407.17842
• Copy Paste: [[2407.17842]] On the Opportunities of (Re)-Exploring Atmospheric Science by Foundation Models: A Case Study(https://arxiv.org/abs/2407.17842)
• Keywords: foundation model
• Abstract: Most state-of-the-art AI applications in atmospheric science are based on classic deep learning approaches. However, such approaches cannot automatically integrate multiple complicated procedures to construct an intelligent agent, since each functionality is enabled by a separate model learned from independent climate datasets. The emergence of foundation models, especially multimodal foundation models, with their ability to process heterogeneous input data and execute complex tasks, offers a substantial opportunity to overcome this challenge. In this report, we want to explore a central question - how the state-of-the-art foundation model, i.e., GPT-4o, performs various atmospheric scientific tasks. Toward this end, we conduct a case study by categorizing the tasks into four main classes, including climate data processing, physical diagnosis, forecast and prediction, and adaptation and mitigation. For each task, we comprehensively evaluate the GPT-4o's performance along with a concrete discussion. We hope that this report may shed new light on future AI applications and research in atmospheric science.

Title: DragText: Rethinking Text Embedding in Point-based Image Editing

• Authors: Gayoon Choi, Taejin Jeong, Sujung Hong, Jaehoon Joo, Seong Jae Hwang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17843
• Pdf URL: https://arxiv.org/pdf/2407.17843
• Copy Paste: [[2407.17843]] DragText: Rethinking Text Embedding in Point-based Image Editing(https://arxiv.org/abs/2407.17843)
• Keywords: diffusion
• Abstract: Point-based image editing enables accurate and flexible control through content dragging. However, the role of text embedding in the editing process has not been thoroughly investigated. A significant aspect that remains unexplored is the interaction between text and image embeddings. In this study, we show that during the progressive editing of an input image in a diffusion model, the text embedding remains constant. As the image embedding increasingly diverges from its initial state, the discrepancy between the image and text embeddings presents a significant challenge. Moreover, we found that the text prompt significantly influences the dragging process, particularly in maintaining content integrity and achieving the desired manipulation. To utilize these insights, we propose DragText, which optimizes text embedding in conjunction with the dragging process to pair with the modified image embedding. Simultaneously, we regularize the text optimization process to preserve the integrity of the original text prompt. Our approach can be seamlessly integrated with existing diffusion-based drag methods with only a few lines of code.

Title: FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editing

• Authors: Gwanhyeong Koo, Sunjae Yoon, Ji Woo Hong, Chang D. Yoo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17850
• Pdf URL: https://arxiv.org/pdf/2407.17850
• Copy Paste: [[2407.17850]] FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editing(https://arxiv.org/abs/2407.17850)
• Keywords: diffusion
• Abstract: Current image editing methods primarily utilize DDIM Inversion, employing a two-branch diffusion approach to preserve the attributes and layout of the original image. However, these methods encounter challenges with non-rigid edits, which involve altering the image's layout or structure. Our comprehensive analysis reveals that the high-frequency components of DDIM latent, crucial for retaining the original image's key features and layout, significantly contribute to these limitations. Addressing this, we introduce FlexiEdit, which enhances fidelity to input text prompts by refining DDIM latent, by reducing high-frequency components in targeted editing areas. FlexiEdit comprises two key components: (1) Latent Refinement, which modifies DDIM latent to better accommodate layout adjustments, and (2) Edit Fidelity Enhancement via Re-inversion, aimed at ensuring the edits more accurately reflect the input text prompts. Our approach represents notable progress in image editing, particularly in performing complex non-rigid edits, showcasing its enhanced capability through comparative experiments.

Title: Is the Digital Forensics and Incident Response Pipeline Ready for Text-Based Threats in LLM Era?

• Authors: Avanti Bhandarkar, Ronald Wilson, Anushka Swarup, Mengdi Zhu, Damon Woodard
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17870
• Pdf URL: https://arxiv.org/pdf/2407.17870
• Copy Paste: [[2407.17870]] Is the Digital Forensics and Incident Response Pipeline Ready for Text-Based Threats in LLM Era?(https://arxiv.org/abs/2407.17870)
• Keywords: generative
• Abstract: In the era of generative AI, the widespread adoption of Neural Text Generators (NTGs) presents new cybersecurity challenges, particularly within the realms of Digital Forensics and Incident Response (DFIR). These challenges primarily involve the detection and attribution of sources behind advanced attacks like spearphishing and disinformation campaigns. As NTGs evolve, the task of distinguishing between human and NTG-authored texts becomes critically complex. This paper rigorously evaluates the DFIR pipeline tailored for text-based security systems, specifically focusing on the challenges of detecting and attributing authorship of NTG-authored texts. By introducing a novel human-NTG co-authorship text attack, termed CS-ACT, our study uncovers significant vulnerabilities in traditional DFIR methodologies, highlighting discrepancies between ideal scenarios and real-world conditions. Utilizing 14 diverse datasets and 43 unique NTGs, up to the latest GPT-4, our research identifies substantial vulnerabilities in the forensic profiling phase, particularly in attributing authorship to NTGs. Our comprehensive evaluation points to factors such as model sophistication and the lack of distinctive style within NTGs as significant contributors for these vulnerabilities. Our findings underscore the necessity for more sophisticated and adaptable strategies, such as incorporating adversarial learning, stylizing NTGs, and implementing hierarchical attribution through the mapping of NTG lineages to enhance source attribution. This sets the stage for future research and the development of more resilient text-based security systems.

Title: DAM: Towards A Foundation Model for Time Series Forecasting

• Authors: Luke Darlow, Qiwen Deng, Ahmed Hassan, Martin Asenov, Rajkarn Singh, Artjom Joosen, Adam Barker, Amos Storkey
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17880
• Pdf URL: https://arxiv.org/pdf/2407.17880
• Copy Paste: [[2407.17880]] DAM: Towards A Foundation Model for Time Series Forecasting(https://arxiv.org/abs/2407.17880)
• Keywords: foundation model
• Abstract: It is challenging to scale time series forecasting models such that they forecast accurately for multiple distinct domains and datasets, all with potentially different underlying collection procedures (e.g., sample resolution), patterns (e.g., periodicity), and prediction requirements (e.g., reconstruction vs. forecasting). We call this general task universal forecasting. Existing methods usually assume that input data is regularly sampled, and they forecast to pre-determined horizons, resulting in failure to generalise outside of the scope of their training. We propose the DAM - a neural model that takes randomly sampled histories and outputs an adjustable basis composition as a continuous function of time for forecasting to non-fixed horizons. It involves three key components: (1) a flexible approach for using randomly sampled histories from a long-tail distribution, that enables an efficient global perspective of the underlying temporal dynamics while retaining focus on the recent history; (2) a transformer backbone that is trained on these actively sampled histories to produce, as representational output, (3) the basis coefficients of a continuous function of time. We show that a single univariate DAM, trained on 25 time series datasets, either outperformed or closely matched existing SoTA models at multivariate long-term forecasting across 18 datasets, including 8 held-out for zero-shot transfer, even though these models were trained to specialise for each dataset-horizon combination. This single DAM excels at zero-shot transfer and very-long-term forecasting, performs well at imputation, is interpretable via basis function composition and attention, can be tuned for different inference-cost requirements, is robust to missing and irregularly sampled data {by design}.

Title: Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Vision

• Authors: Tim J.M. Jaspers, Ronald L.P.D. de Jong, Yasmina Al Khalil, Tijn Zeelenberg, Carolus H.J. Kusters, Yiping Li, Romy C. van Jaarsveld, Franciscus H.A. Bakker, Jelle P. Ruurda, Willem M. Brinkman, Peter H.N. De With, Fons van der Sommen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17904
• Pdf URL: https://arxiv.org/pdf/2407.17904
• Copy Paste: [[2407.17904]] Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Vision(https://arxiv.org/abs/2407.17904)
• Keywords: self-supervised
• Abstract: Over the past decade, computer vision applications in minimally invasive surgery have rapidly increased. Despite this growth, the impact of surgical computer vision remains limited compared to other medical fields like pathology and radiology, primarily due to the scarcity of representative annotated data. Whereas transfer learning from large annotated datasets such as ImageNet has been conventionally the norm to achieve high-performing models, recent advancements in self-supervised learning (SSL) have demonstrated superior performance. In medical image analysis, in-domain SSL pretraining has already been shown to outperform ImageNet-based initialization. Although unlabeled data in the field of surgical computer vision is abundant, the diversity within this data is limited. This study investigates the role of dataset diversity in SSL for surgical computer vision, comparing procedure-specific datasets against a more heterogeneous general surgical dataset across three different downstream surgical applications. The obtained results show that using solely procedure-specific data can lead to substantial improvements of 13.8%, 9.5%, and 36.8% compared to ImageNet pretraining. However, extending this data with more heterogeneous surgical data further increases performance by an additional 5.0%, 5.2%, and 2.5%, suggesting that increasing diversity within SSL data is beneficial for model performance. The code and pretrained model weights are made publicly available at this https URL.

Title: Amortized Posterior Sampling with Diffusion Prior Distillation

• Authors: Abbas Mammadov, Hyungjin Chung, Jong Chul Ye
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17907
• Pdf URL: https://arxiv.org/pdf/2407.17907
• Copy Paste: [[2407.17907]] Amortized Posterior Sampling with Diffusion Prior Distillation(https://arxiv.org/abs/2407.17907)
• Keywords: diffusion
• Abstract: We propose a variational inference approach to sample from the posterior distribution for solving inverse problems. From a pre-trained diffusion model, our approach trains a conditional flow model to minimize the divergence between the proposal variational distribution and the posterior distribution implicitly defined through the diffusion model. Once trained, the flow model is capable of sampling from the posterior distribution with a single NFE, amortized with respect to the measurement. The proposed method paves a new path for distilling a diffusion prior for efficient posterior sampling. We show that our method is applicable to standard signals in Euclidean space, as well as signals on manifold.

Title: Separating Novel Features for Logical Anomaly Detection: A Straightforward yet Effective Approach

• Authors: Kangil Lee, Geonuk Kim
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17909
• Pdf URL: https://arxiv.org/pdf/2407.17909
• Copy Paste: [[2407.17909]] Separating Novel Features for Logical Anomaly Detection: A Straightforward yet Effective Approach(https://arxiv.org/abs/2407.17909)
• Keywords: anomaly
• Abstract: Vision-based inspection algorithms have significantly contributed to quality control in industrial settings, particularly in addressing structural defects like dent and contamination which are prevalent in mass production. Extensive research efforts have led to the development of related benchmarks such as MVTec AD (Bergmann et al., 2019). However, in industrial settings, there can be instances of logical defects, where acceptable items are found in unsuitable locations or product pairs do not match as expected. Recent methods tackling logical defects effectively employ knowledge distillation to generate difference maps. Knowledge distillation (KD) is used to learn normal data distribution in unsupervised manner. Despite their effectiveness, these methods often overlook the potential false negatives. Excessive similarity between the teacher network and student network can hinder the generation of a suitable difference map for logical anomaly detection. This technical report provides insights on handling potential false negatives by utilizing a simple constraint in KD-based logical anomaly detection methods. We select EfficientAD as a state-of-the-art baseline and apply a margin-based constraint to its unsupervised learning scheme. Applying this constraint, we can improve the AUROC for MVTec LOCO AD by 1.3 %.

Title: Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models

• Authors: Anna Bavaresco, Marianne de Heer Kloots, Sandro Pezzelle, Raquel Fernández
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17914
• Pdf URL: https://arxiv.org/pdf/2407.17914
• Copy Paste: [[2407.17914]] Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models(https://arxiv.org/abs/2407.17914)
• Keywords: generative
• Abstract: Representations from deep neural networks (DNNs) have proven remarkably predictive of neural activity involved in both visual and linguistic processing. Despite these successes, most studies to date concern unimodal DNNs, encoding either visual or textual input but not both. Yet, there is growing evidence that human meaning representations integrate linguistic and sensory-motor information. Here we investigate whether the integration of multimodal information operated by current vision-and-language DNN models (VLMs) leads to representations that are more aligned with human brain activity than those obtained by language-only and vision-only DNNs. We focus on fMRI responses recorded while participants read concept words in the context of either a full sentence or an accompanying picture. Our results reveal that VLM representations correlate more strongly than language- and vision-only DNNs with activations in brain areas functionally related to language processing. A comparison between different types of visuo-linguistic architectures shows that recent generative VLMs tend to be less brain-aligned than previous architectures with lower performance on downstream applications. Moreover, through an additional analysis comparing brain vs. behavioural alignment across multiple VLMs, we show that -- with one remarkable exception -- representations that strongly align with behavioural judgments do not correlate highly with brain responses. This indicates that brain similarity does not go hand in hand with behavioural similarity, and vice versa.

Title: Guided Latent Slot Diffusion for Object-Centric Learning

• Authors: Krishnakant Singh, Simone Schaub-Meyer, Stefan Roth
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17929
• Pdf URL: https://arxiv.org/pdf/2407.17929
• Copy Paste: [[2407.17929]] Guided Latent Slot Diffusion for Object-Centric Learning(https://arxiv.org/abs/2407.17929)
• Keywords: diffusion
• Abstract: Slot attention aims to decompose an input image into a set of meaningful object files (slots). These latent object representations enable various downstream tasks. Yet, these slots often bind to object parts, not objects themselves, especially for real-world datasets. To address this, we introduce Guided Latent Slot Diffusion - GLASS, an object-centric model that uses generated captions as a guiding signal to better align slots with objects. Our key insight is to learn the slot-attention module in the space of generated images. This allows us to repurpose the pre-trained diffusion decoder model, which reconstructs the images from the slots, as a semantic mask generator based on the generated captions. GLASS learns an object-level representation suitable for multiple tasks simultaneously, e.g., segmentation, image generation, and property prediction, outperforming previous methods. For object discovery, GLASS achieves approx. a +35% and +10% relative improvement for mIoU over the previous state-of-the-art (SOTA) method on the VOC and COCO datasets, respectively, and establishes a new SOTA FID score for conditional image generation amongst slot-attention-based methods. For the segmentation task, GLASS surpasses SOTA weakly-supervised and language-based segmentation models, which were specifically designed for the task.

Title: BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation

• Authors: Xiang Zhang, Bingxin Ke, Hayko Riemenschneider, Nando Metzger, Anton Obukhov, Markus Gross, Konrad Schindler, Christopher Schroers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17952
• Pdf URL: https://arxiv.org/pdf/2407.17952
• Copy Paste: [[2407.17952]] BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation(https://arxiv.org/abs/2407.17952)
• Keywords: diffusion
• Abstract: By training over large-scale datasets, zero-shot monocular depth estimation (MDE) methods show robust performance in the wild but often suffer from insufficiently precise details. Although recent diffusion-based MDE approaches exhibit appealing detail extraction ability, they still struggle in geometrically challenging scenes due to the difficulty of gaining robust geometric priors from diverse datasets. To leverage the complementary merits of both worlds, we propose BetterDepth to efficiently achieve geometrically correct affine-invariant MDE performance while capturing fine-grained details. Specifically, BetterDepth is a conditional diffusion-based refiner that takes the prediction from pre-trained MDE models as depth conditioning, in which the global depth context is well-captured, and iteratively refines details based on the input image. For the training of such a refiner, we propose global pre-alignment and local patch masking methods to ensure the faithfulness of BetterDepth to depth conditioning while learning to capture fine-grained scene details. By efficient training on small-scale synthetic datasets, BetterDepth achieves state-of-the-art zero-shot MDE performance on diverse public datasets and in-the-wild scenes. Moreover, BetterDepth can improve the performance of other MDE models in a plug-and-play manner without additional re-training.

Title: Relating the Seemingly Unrelated: Principled Understanding of Generalization for Generative Models in Arithmetic Reasoning Tasks

• Authors: Xingcheng Xu, Zibo Zhao, Haipeng Zhang, Yanqing Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17963
• Pdf URL: https://arxiv.org/pdf/2407.17963
• Copy Paste: [[2407.17963]] Relating the Seemingly Unrelated: Principled Understanding of Generalization for Generative Models in Arithmetic Reasoning Tasks(https://arxiv.org/abs/2407.17963)
• Keywords: generative
• Abstract: Large language models (LLMs) have demonstrated impressive versatility across numerous tasks, yet their generalization capabilities remain poorly understood. To investigate these behaviors, arithmetic tasks serve as important venues. In previous studies, seemingly unrelated mysteries still exist -- (1) models with appropriate positional embeddings can correctly perform longer unseen arithmetic operations such as addition, but their effectiveness varies in more complex tasks like multiplication; (2) models perform well for longer unseen cases in modular addition under specific moduli (e.g., modulo 100) but struggle under very close moduli (e.g., modulo 101), regardless of the positional encoding used. We believe previous studies have been treating the symptoms rather than addressing the root cause -- they have paid excessive attention to improving model components, while overlooking the differences in task properties that may be the real drivers. This is confirmed by our unified theoretical framework for different arithmetic scenarios. For example, unlike multiplication, the digital addition task has the property of translation invariance which naturally aligns with the relative positional encoding, and this combination leads to successful generalization of addition to unseen longer domains. The discrepancy in operations modulo 100 and 101 arises from the base. Modulo 100, unlike 101, is compatible with the decimal system (base 10), such that unseen information in digits beyond the units digit and the tens digit is actually not needed for the task. Extensive experiments with GPT-like models validate our theoretical predictions. These findings deepen our understanding of the generalization mechanisms, and facilitate more data-efficient model training and objective-oriented AI alignment.

Title: Self-Supervision Improves Diffusion Models for Tabular Data Imputation

• Authors: Yixin Liu, Thalaiyasingam Ajanthan, Hisham Husain, Vu Nguyen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18013
• Pdf URL: https://arxiv.org/pdf/2407.18013
• Copy Paste: [[2407.18013]] Self-Supervision Improves Diffusion Models for Tabular Data Imputation(https://arxiv.org/abs/2407.18013)
• Keywords: diffusion, self-supervised
• Abstract: The ubiquity of missing data has sparked considerable attention and focus on tabular data imputation methods. Diffusion models, recognized as the cutting-edge technique for data generation, demonstrate significant potential in tabular data imputation tasks. However, in pursuit of diversity, vanilla diffusion models often exhibit sensitivity to initialized noises, which hinders the models from generating stable and accurate imputation results. Additionally, the sparsity inherent in tabular data poses challenges for diffusion models in accurately modeling the data manifold, impacting the robustness of these models for data imputation. To tackle these challenges, this paper introduces an advanced diffusion model named Self-supervised imputation Diffusion Model (SimpDM for brevity), specifically tailored for tabular data imputation tasks. To mitigate sensitivity to noise, we introduce a self-supervised alignment mechanism that aims to regularize the model, ensuring consistent and stable imputation predictions. Furthermore, we introduce a carefully devised state-dependent data augmentation strategy within SimpDM, enhancing the robustness of the diffusion model when dealing with limited data. Extensive experiments demonstrate that SimpDM matches or outperforms state-of-the-art imputation methods across various scenarios.

Title: ECG Arrhythmia Detection Using Disease-specific Attention-based Deep Learning Model

• Authors: Linpeng Jin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18033
• Pdf URL: https://arxiv.org/pdf/2407.18033
• Copy Paste: [[2407.18033]] ECG Arrhythmia Detection Using Disease-specific Attention-based Deep Learning Model(https://arxiv.org/abs/2407.18033)
• Keywords: self-supervised
• Abstract: The electrocardiogram (ECG) is one of the most commonly-used tools to diagnose cardiovascular disease in clinical practice. Although deep learning models have achieved very impressive success in the field of automatic ECG analysis, they often lack model interpretability that is significantly important in the healthcare applications. To this end, many schemes such as general-purpose attention mechanism, Grad-CAM technique and ECG knowledge graph were proposed to be integrated with deep learning models. However, they either result in decreased classification performance or do not consist with the one in cardiologists' mind when interpreting ECG. In this study, we propose a novel disease-specific attention-based deep learning model (DANet) for arrhythmia detection from short ECG recordings. The novel idea is to introduce a soft-coding or hard-coding waveform enhanced module into existing deep neural networks, which amends original ECG signals with the guidance of the rule for diagnosis of a given disease type before being fed into the classification module. For the soft-coding DANet, we also develop a learning framework combining self-supervised pre-training with two-stage supervised training. To verify the effectiveness of our proposed DANet, we applied it to the problem of atrial premature contraction detection and the experimental results shows that it demonstrates superior performance compared to the benchmark model. Moreover, it also provides the waveform regions that deserve special attention in the model's decision-making process, allowing it to be a medical diagnostic assistant for physicians.

Title: AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wild

• Authors: Junho Park, Kyeongbo Kong, Suk-Ju Kang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18034
• Pdf URL: https://arxiv.org/pdf/2407.18034
• Copy Paste: [[2407.18034]] AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wild(https://arxiv.org/abs/2407.18034)
• Keywords: diffusion
• Abstract: Recently, there has been a significant amount of research conducted on 3D hand reconstruction to use various forms of human-computer interaction. However, 3D hand reconstruction in the wild is challenging due to extreme lack of in-the-wild 3D hand datasets. Especially, when hands are in complex pose such as interacting hands, the problems like appearance similarity, self-handed occclusion and depth ambiguity make it more difficult. To overcome these issues, we propose AttentionHand, a novel method for text-driven controllable hand image generation. Since AttentionHand can generate various and numerous in-the-wild hand images well-aligned with 3D hand label, we can acquire a new 3D hand dataset, and can relieve the domain gap between indoor and outdoor scenes. Our method needs easy-to-use four modalities (i.e, an RGB image, a hand mesh image from 3D label, a bounding box, and a text prompt). These modalities are embedded into the latent space by the encoding phase. Then, through the text attention stage, hand-related tokens from the given text prompt are attended to highlight hand-related regions of the latent embedding. After the highlighted embedding is fed to the visual attention stage, hand-related regions in the embedding are attended by conditioning global and local hand mesh images with the diffusion-based pipeline. In the decoding phase, the final feature is decoded to new hand images, which are well-aligned with the given hand mesh image and text prompt. As a result, AttentionHand achieved state-of-the-art among text-to-hand image generation models, and the performance of 3D hand mesh reconstruction was improved by additionally training with hand images generated by AttentionHand.

Title: Difficulty Estimation and Simplification of French Text Using LLMs

• Authors: Henri Jamet, Yash Raj Shrestha, Michalis Vlachos
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18061
• Pdf URL: https://arxiv.org/pdf/2407.18061
• Copy Paste: [[2407.18061]] Difficulty Estimation and Simplification of French Text Using LLMs(https://arxiv.org/abs/2407.18061)
• Keywords: generative
• Abstract: We leverage generative large language models for language learning applications, focusing on estimating the difficulty of foreign language texts and simplifying them to lower difficulty levels. We frame both tasks as prediction problems and develop a difficulty classification model using labeled examples, transfer learning, and large language models, demonstrating superior accuracy compared to previous approaches. For simplification, we evaluate the trade-off between simplification quality and meaning preservation, comparing zero-shot and fine-tuned performances of large language models. We show that meaningful text simplifications can be obtained with limited fine-tuning. Our experiments are conducted on French texts, but our methods are language-agnostic and directly applicable to other foreign languages.

Title: PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization

• Authors: Christopher Clarke, Yuzhao Heng, Lingjia Tang, Jason Mars
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18078
• Pdf URL: https://arxiv.org/pdf/2407.18078
• Copy Paste: [[2407.18078]] PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization(https://arxiv.org/abs/2407.18078)
• Keywords: foundation model
• Abstract: The recent emergence of Large Language Models (LLMs) has heralded a new era of human-AI interaction. These sophisticated models, exemplified by Chat-GPT and its successors, have exhibited remarkable capabilities in language understanding. However, as these LLMs have undergone exponential growth, a crucial dimension that remains understudied is the personalization of these models. Large foundation models such as GPT-3 etc. focus on creating a universal model that serves a broad range of tasks and users. This approach emphasizes the model's generalization capabilities, treating users as a collective rather than as distinct individuals. While practical for many common applications, this one-size-fits-all approach often fails to address the rich tapestry of human diversity and individual needs. To explore this issue we introduce the PEFT-U Benchmark: a new dataset for building and evaluating NLP models for user personalization. \datasetname{} consists of a series of user-centered tasks containing diverse and individualized expressions where the preferences of users can potentially differ for the same input. Using PEFT-U, we explore the challenge of efficiently personalizing LLMs to accommodate user-specific preferences in the context of diverse user-centered tasks.

Title: Self-supervised pre-training with diffusion model for few-shot landmark detection in x-ray images

• Authors: Roberto Di Via, Francesca Odone, Vito Paolo Pastore
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18125
• Pdf URL: https://arxiv.org/pdf/2407.18125
• Copy Paste: [[2407.18125]] Self-supervised pre-training with diffusion model for few-shot landmark detection in x-ray images(https://arxiv.org/abs/2407.18125)
• Keywords: diffusion, self-supervised
• Abstract: In the last few years, deep neural networks have been extensively applied in the medical domain for different tasks, ranging from image classification and segmentation to landmark detection. However, the application of these technologies in the medical domain is often hindered by data scarcity, both in terms of available annotations and images. This study introduces a new self-supervised pre-training protocol based on diffusion models for landmark detection in x-ray images. Our results show that the proposed self-supervised framework can provide accurate landmark detection with a minimal number of available annotated training images (up to 50), outperforming ImageNet supervised pre-training and state-of-the-art self-supervised pre-trainings for three popular x-ray benchmark datasets. To our knowledge, this is the first exploration of diffusion models for self-supervised learning in landmark detection, which may offer a valuable pre-training approach in few-shot regimes, for mitigating data scarcity.

Title: $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs

• Authors: Vlad Sobal, Mark Ibrahim, Randall Balestriero, Vivien Cabannes, Diane Bouchacourt, Pietro Astolfi, Kyunghyun Cho, Yann LeCun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18134
• Pdf URL: https://arxiv.org/pdf/2407.18134
• Copy Paste: [[2407.18134]] $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs(https://arxiv.org/abs/2407.18134)
• Keywords: self-supervised, foundation model
• Abstract: Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss - an objective matching related samples - underlies methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying a similarity graph to indicate how samples should relate in the embedding space. This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities \textit{across} samples are ignored. Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive, to train vision models based on similarities in class or text caption descriptions. Our study spans three scales: ImageNet-1k with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision-language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by $0.6\%$ on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8\%$ on ImageNet and $18.1\%$ on ImageNet Real when training with CC3M. Finally, our objective seems to encourage the model to learn representations that separate objects from their attributes and backgrounds, with gains of $3.3$-$5.6$\% over CLIP on ImageNet9. We hope the proposed solution takes a small step towards developing richer learning objectives for understanding sample relations in foundation models.

Title: Recursive Introspection: Teaching Language Model Agents How to Self-Improve

• Authors: Yuxiao Qu, Tianjun Zhang, Naman Garg, Aviral Kumar
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18219
• Pdf URL: https://arxiv.org/pdf/2407.18219
• Copy Paste: [[2407.18219]] Recursive Introspection: Teaching Language Model Agents How to Self-Improve(https://arxiv.org/abs/2407.18219)
• Keywords: foundation model
• Abstract: A central piece in enabling intelligent agentic behavior in foundation models is to make them capable of introspecting upon their behavior, reasoning, and correcting their mistakes as more computation or interaction is available. Even the strongest proprietary large language models (LLMs) do not quite exhibit the ability of continually improving their responses sequentially, even in scenarios where they are explicitly told that they are making a mistake. In this paper, we develop RISE: Recursive IntroSpEction, an approach for fine-tuning LLMs to introduce this capability, despite prior work hypothesizing that this capability may not be possible to attain. Our approach prescribes an iterative fine-tuning procedure, which attempts to teach the model how to alter its response after having executed previously unsuccessful attempts to solve a hard test-time problem, with optionally additional environment feedback. RISE poses fine-tuning for a single-turn prompt as solving a multi-turn Markov decision process (MDP), where the initial state is the prompt. Inspired by principles in online imitation learning and reinforcement learning, we propose strategies for multi-turn data collection and training so as to imbue an LLM with the capability to recursively detect and correct its previous mistakes in subsequent iterations. Our experiments show that RISE enables Llama2, Llama3, and Mistral models to improve themselves with more turns on math reasoning tasks, outperforming several single-turn strategies given an equal amount of inference-time computation. We also find that RISE scales well, often attaining larger benefits with more capable models. Our analysis shows that RISE makes meaningful improvements to responses to arrive at the correct solution for challenging prompts, without disrupting one-turn abilities as a result of expressing more complex distributions.

Title: LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

• Authors: Zhengbo Wang, Jian Liang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18242
• Pdf URL: https://arxiv.org/pdf/2407.18242
• Copy Paste: [[2407.18242]] LoRA-Pro: Are Low-Rank Adapters Properly Optimized?(https://arxiv.org/abs/2407.18242)
• Keywords: foundation model
• Abstract: Low-Rank Adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning foundation models by re-parameterizing the original matrix into the product of two low-rank matrices. Despite its efficiency, LoRA often yields inferior performance compared to full fine-tuning. In this paper, we propose LoRA-Pro to bridge this performance gap. Firstly, we delve into the optimization processes in LoRA and full fine-tuning. We reveal that while LoRA employs low-rank approximation, it neglects to approximate the optimization process of full fine-tuning. To address this, we introduce a novel concept called the "equivalent gradient." This virtual gradient makes the optimization process on the re-parameterized matrix equivalent to LoRA, which can be used to quantify the differences between LoRA and full fine-tuning. The equivalent gradient is derived from the gradients of matrices $A$ and $B$. To narrow the performance gap, our approach minimizes the differences between the equivalent gradient and the gradient obtained from full fine-tuning during the optimization process. By solving this objective, we derive optimal closed-form solutions for updating matrices $A$ and $B$. Our method constrains the optimization process, shrinking the performance gap between LoRA and full fine-tuning. Extensive experiments on natural language processing tasks validate the effectiveness of our method.

Title: VGGHeads: A Large-Scale Synthetic Dataset for 3D Human Heads

• Authors: Orest Kupyn, Eugene Khvedchenia, Christian Rupprecht
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18245
• Pdf URL: https://arxiv.org/pdf/2407.18245
• Copy Paste: [[2407.18245]] VGGHeads: A Large-Scale Synthetic Dataset for 3D Human Heads(https://arxiv.org/abs/2407.18245)
• Keywords: diffusion
• Abstract: Human head detection, keypoint estimation, and 3D head model fitting are important tasks with many applications. However, traditional real-world datasets often suffer from bias, privacy, and ethical concerns, and they have been recorded in laboratory environments, which makes it difficult for trained models to generalize. Here, we introduce VGGHeads -- a large scale synthetic dataset generated with diffusion models for human head detection and 3D mesh estimation. Our dataset comprises over 1 million high-resolution images, each annotated with detailed 3D head meshes, facial landmarks, and bounding boxes. Using this dataset we introduce a new model architecture capable of simultaneous heads detection and head meshes reconstruction from a single image in a single step. Through extensive experimental evaluations, we demonstrate that models trained on our synthetic data achieve strong performance on real images. Furthermore, the versatility of our dataset makes it applicable across a broad spectrum of tasks, offering a general and comprehensive representation of human heads. Additionally, we provide detailed information about the synthetic data generation pipeline, enabling it to be re-used for other tasks and domains.

Title: RegionDrag: Fast Region-Based Image Editing with Diffusion Models

• Authors: Jingyi Lu, Xinghui Li, Kai Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18247
• Pdf URL: https://arxiv.org/pdf/2407.18247
• Copy Paste: [[2407.18247]] RegionDrag: Fast Region-Based Image Editing with Diffusion Models(https://arxiv.org/abs/2407.18247)
• Keywords: diffusion
• Abstract: Point-drag-based image editing methods, like DragDiffusion, have attracted significant attention. However, point-drag-based approaches suffer from computational overhead and misinterpretation of user intentions due to the sparsity of point-based editing instructions. In this paper, we propose a region-based copy-and-paste dragging method, RegionDrag, to overcome these limitations. RegionDrag allows users to express their editing instructions in the form of handle and target regions, enabling more precise control and alleviating ambiguity. In addition, region-based operations complete editing in one iteration and are much faster than point-drag-based methods. We also incorporate the attention-swapping technique for enhanced stability during editing. To validate our approach, we extend existing point-drag-based datasets with region-based dragging instructions. Experimental results demonstrate that RegionDrag outperforms existing point-drag-based approaches in terms of speed, accuracy, and alignment with user intentions. Remarkably, RegionDrag completes the edit on an image with a resolution of 512x512 in less than 2 seconds, which is more than 100x faster than DragDiffusion, while achieving better performance. Project page: this https URL.

Title: Trajectory-aligned Space-time Tokens for Few-shot Action Recognition

• Authors: Pulkit Kumar, Namitha Padmanabhan, Luke Luo, Sai Saketh Rambhatla, Abhinav Shrivastava
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18249
• Pdf URL: https://arxiv.org/pdf/2407.18249
• Copy Paste: [[2407.18249]] Trajectory-aligned Space-time Tokens for Few-shot Action Recognition(https://arxiv.org/abs/2407.18249)
• Keywords: self-supervised
• Abstract: We propose a simple yet effective approach for few-shot action recognition, emphasizing the disentanglement of motion and appearance representations. By harnessing recent progress in tracking, specifically point trajectories and self-supervised representation learning, we build trajectory-aligned tokens (TATs) that capture motion and appearance information. This approach significantly reduces the data requirements while retaining essential information. To process these representations, we use a Masked Space-time Transformer that effectively learns to aggregate information to facilitate few-shot action recognition. We demonstrate state-of-the-art results on few-shot action recognition across multiple datasets. Our project page is available at this https URL