2024-04-12

Title: Goal-guided Generative Prompt Injection Attack on Large Language Models

• Authors: Chong Zhang, Mingyu Jin, Qinkai Yu, Chengzhi Liu, Haochen Xue, Xiaobo Jin
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07234
• Pdf URL: https://arxiv.org/pdf/2404.07234
• Copy Paste: [[2404.07234]] Goal-guided Generative Prompt Injection Attack on Large Language Models(https://arxiv.org/abs/2404.07234)
• Keywords: generative
• Abstract: Current large language models (LLMs) provide a strong foundation for large-scale user-oriented natural language tasks. A large number of users can easily inject adversarial text or instructions through the user interface, thus causing LLMs model security challenges. Although there is currently a large amount of research on prompt injection attacks, most of these black-box attacks use heuristic strategies. It is unclear how these heuristic strategies relate to the success rate of attacks and thus effectively improve model robustness. To solve this problem, we redefine the goal of the attack: to maximize the KL divergence between the conditional probabilities of the clean text and the adversarial text. Furthermore, we prove that maximizing the KL divergence is equivalent to maximizing the Mahalanobis distance between the embedded representation $x$ and $x'$ of the clean text and the adversarial text when the conditional probability is a Gaussian distribution and gives a quantitative relationship on $x$ and $x'$. Then we designed a simple and effective goal-guided generative prompt injection strategy (G2PIA) to find an injection text that satisfies specific constraints to achieve the optimal attack effect approximately. It is particularly noteworthy that our attack method is a query-free black-box attack method with low computational cost. Experimental results on seven LLM models and four datasets show the effectiveness of our attack method.

Title: Generative Resident Separation and Multi-label Classification for Multi-person Activity Recognition

• Authors: Xi Chen (LIG), Julien Cumin, Fano Ramparany, Dominique Vaufreydaz (LIG)
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.07245
• Pdf URL: https://arxiv.org/pdf/2404.07245
• Copy Paste: [[2404.07245]] Generative Resident Separation and Multi-label Classification for Multi-person Activity Recognition(https://arxiv.org/abs/2404.07245)
• Keywords: generative
• Abstract: This paper presents two models to address the problem of multi-person activity recognition using ambient sensors in a home. The first model, Seq2Res, uses a sequence generation approach to separate sensor events from different residents. The second model, BiGRU+Q2L, uses a Query2Label multi-label classifier to predict multiple activities simultaneously. Performances of these models are compared to a state-of-the-art model in different experimental scenarios, using a state-of-the-art dataset of two residents in a home instrumented with ambient sensors. These results lead to a discussion on the advantages and drawbacks of resident separation and multi-label classification for multi-person activity recognition.

Title: Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers

• Authors: Jinyang Liu, Wondmgezahu Teshome, Sandesh Ghimire, Mario Sznaier, Octavia Camps
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07292
• Pdf URL: https://arxiv.org/pdf/2404.07292
• Copy Paste: [[2404.07292]] Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers(https://arxiv.org/abs/2404.07292)
• Keywords: diffusion
• Abstract: Solving image and video jigsaw puzzles poses the challenging task of rearranging image fragments or video frames from unordered sequences to restore meaningful images and video sequences. Existing approaches often hinge on discriminative models tasked with predicting either the absolute positions of puzzle elements or the permutation actions applied to the original data. Unfortunately, these methods face limitations in effectively solving puzzles with a large number of elements. In this paper, we propose JPDVT, an innovative approach that harnesses diffusion transformers to address this challenge. Specifically, we generate positional information for image patches or video frames, conditioned on their underlying visual content. This information is then employed to accurately assemble the puzzle pieces in their correct positions, even in scenarios involving missing pieces. Our method achieves state-of-the-art performance on several datasets.

Title: PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scores

• Authors: Lucas Goncalves, Prashant Mathur, Chandrashekhar Lavania, Metehan Cekic, Marcello Federico, Kyu J. Han
• Subjects: cs.CV, cs.MM, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.07336
• Pdf URL: https://arxiv.org/pdf/2404.07336
• Copy Paste: [[2404.07336]] PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scores(https://arxiv.org/abs/2404.07336)
• Keywords: generative
• Abstract: Recent advancements in audio-visual generative modeling have been propelled by progress in deep learning and the availability of data-rich benchmarks. However, the growth is not attributed solely to models and benchmarks. Universally accepted evaluation metrics also play an important role in advancing the field. While there are many metrics available to evaluate audio and visual content separately, there is a lack of metrics that offer a quantitative and interpretable measure of audio-visual synchronization for videos "in the wild". To address this gap, we first created a large scale human annotated dataset (100+ hrs) representing nine types of synchronization errors in audio-visual content and how human perceive them. We then developed a PEAVS (Perceptual Evaluation of Audio-Visual Synchrony) score, a novel automatic metric with a 5-point scale that evaluates the quality of audio-visual synchronization. We validate PEAVS using a newly generated dataset, achieving a Pearson correlation of 0.79 at the set level and 0.54 at the clip level when compared to human labels. In our experiments, we observe a relative gain 50% over a natural extension of Fr\'echet based metrics for Audio-Visual synchrony, confirming PEAVS efficacy in objectively modeling subjective perceptions of audio-visual synchronization for videos "in the wild".

Title: GANsemble for Small and Imbalanced Data Sets: A Baseline for Synthetic Microplastics Data

• Authors: Daniel Platnick, Sourena Khanzadeh, Alireza Sadeghian, Richard Anthony Valenzano
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07356
• Pdf URL: https://arxiv.org/pdf/2404.07356
• Copy Paste: [[2404.07356]] GANsemble for Small and Imbalanced Data Sets: A Baseline for Synthetic Microplastics Data(https://arxiv.org/abs/2404.07356)
• Keywords: generative
• Abstract: Microplastic particle ingestion or inhalation by humans is a problem of growing concern. Unfortunately, current research methods that use machine learning to understand their potential harms are obstructed by a lack of available data. Deep learning techniques in particular are challenged by such domains where only small or imbalanced data sets are available. Overcoming this challenge often involves oversampling underrepresented classes or augmenting the existing data to improve model performance. This paper proposes GANsemble: a two-module framework connecting data augmentation with conditional generative adversarial networks (cGANs) to generate class-conditioned synthetic data. First, the data chooser module automates augmentation strategy selection by searching for the best data augmentation strategy. Next, the cGAN module uses this strategy to train a cGAN for generating enhanced synthetic data. We experiment with the GANsemble framework on a small and imbalanced microplastics data set. A Microplastic-cGAN (MPcGAN) algorithm is introduced, and baselines for synthetic microplastics (SYMP) data are established in terms of Frechet Inception Distance (FID) and Inception Scores (IS). We also provide a synthetic microplastics filter (SYMP-Filter) algorithm to increase the quality of generated SYMP. Additionally, we show the best amount of oversampling with augmentation to fix class imbalance in small microplastics data sets. To our knowledge, this study is the first application of generative AI to synthetically create microplastics data.

Title: Gradient Networks

• Authors: Shreyas Chaudhari, Srinivasa Pranav, José M. F. Moura
• Subjects: cs.LG, cs.NE, eess.SP, math.OC
• Abstract URL: https://arxiv.org/abs/2404.07361
• Pdf URL: https://arxiv.org/pdf/2404.07361
• Copy Paste: [[2404.07361]] Gradient Networks(https://arxiv.org/abs/2404.07361)
• Keywords: generative
• Abstract: Directly parameterizing and learning gradients of functions has widespread significance, with specific applications in optimization, generative modeling, and optimal transport. This paper introduces gradient networks (GradNets): novel neural network architectures that parameterize gradients of various function classes. GradNets exhibit specialized architectural constraints that ensure correspondence to gradient functions. We provide a comprehensive GradNet design framework that includes methods for transforming GradNets into monotone gradient networks (mGradNets), which are guaranteed to represent gradients of convex functions. We establish the approximation capabilities of the proposed GradNet and mGradNet. Our results demonstrate that these networks universally approximate the gradients of (convex) functions. Furthermore, these networks can be customized to correspond to specific spaces of (monotone) gradient functions, including gradients of transformed sums of (convex) ridge functions. Our analysis leads to two distinct GradNet architectures, GradNet-C and GradNet-M, and we describe the corresponding monotone versions, mGradNet-C and mGradNet-M. Our empirical results show that these architectures offer efficient parameterizations and outperform popular methods in gradient field learning tasks.

Title: Differentially Private GANs for Generating Synthetic Indoor Location Data

• Authors: Vahideh Moghtadaiee, Mina Alishahi, Milad Rabiei
• Subjects: cs.CR, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.07366
• Pdf URL: https://arxiv.org/pdf/2404.07366
• Copy Paste: [[2404.07366]] Differentially Private GANs for Generating Synthetic Indoor Location Data(https://arxiv.org/abs/2404.07366)
• Keywords: generative
• Abstract: The advent of location-based services has led to the widespread adoption of indoor localization systems, which enable location tracking of individuals within enclosed spaces such as buildings. While these systems provide numerous benefits such as improved security and personalized services, they also raise concerns regarding privacy violations. As such, there is a growing need for privacy-preserving solutions that can protect users' sensitive location information while still enabling the functionality of indoor localization systems. In recent years, Differentially Private Generative Adversarial Networks (DPGANs) have emerged as a powerful methodology that aims to protect the privacy of individual data points while generating realistic synthetic data similar to original data. DPGANs combine the power of generative adversarial networks (GANs) with the privacy-preserving technique of differential privacy (DP). In this paper, we introduce an indoor localization framework employing DPGANs in order to generate privacy-preserving indoor location data. We evaluate the performance of our framework on a real-world indoor localization dataset and demonstrate its effectiveness in preserving privacy while maintaining the accuracy of the localization system.

Title: LLMs in Biomedicine: A study on clinical Named Entity Recognition

• Authors: Masoud Monajatipoor, Jiaxin Yang, Joel Stremmel, Melika Emami, Fazlolah Mohaghegh, Mozhdeh Rouhsedaghat, Kai-Wei Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07376
• Pdf URL: https://arxiv.org/pdf/2404.07376
• Copy Paste: [[2404.07376]] LLMs in Biomedicine: A study on clinical Named Entity Recognition(https://arxiv.org/abs/2404.07376)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) demonstrate remarkable versatility in various NLP tasks but encounter distinct challenges in biomedicine due to medical language complexities and data scarcity. This paper investigates the application of LLMs in the medical domain by exploring strategies to enhance their performance for the Named-Entity Recognition (NER) task. Specifically, our study reveals the importance of meticulously designed prompts in biomedicine. Strategic selection of in-context examples yields a notable improvement, showcasing ~15-20\% increase in F1 score across all benchmark datasets for few-shot clinical NER. Additionally, our findings suggest that integrating external resources through prompting strategies can bridge the gap between general-purpose LLM proficiency and the specialized demands of medical NER. Leveraging a medical knowledge base, our proposed method inspired by Retrieval-Augmented Generation (RAG) can boost the F1 score of LLMs for zero-shot clinical NER. We will release the code upon publication.

Title: Deep Generative Sampling in the Dual Divergence Space: A Data-efficient & Interpretative Approach for Generative AI

• Authors: Sahil Garg, Anderson Schneider, Anant Raj, Kashif Rasul, Yuriy Nevmyvaka, Sneihil Gopal, Amit Dhurandhar, Guillermo Cecchi, Irina Rish
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV, cs.IT
• Abstract URL: https://arxiv.org/abs/2404.07377
• Pdf URL: https://arxiv.org/pdf/2404.07377
• Copy Paste: [[2404.07377]] Deep Generative Sampling in the Dual Divergence Space: A Data-efficient & Interpretative Approach for Generative AI(https://arxiv.org/abs/2404.07377)
• Keywords: generative
• Abstract: Building on the remarkable achievements in generative sampling of natural images, we propose an innovative challenge, potentially overly ambitious, which involves generating samples of entire multivariate time series that resemble images. However, the statistical challenge lies in the small sample size, sometimes consisting of a few hundred subjects. This issue is especially problematic for deep generative models that follow the conventional approach of generating samples from a canonical distribution and then decoding or denoising them to match the true data distribution. In contrast, our method is grounded in information theory and aims to implicitly characterize the distribution of images, particularly the (global and local) dependency structure between pixels. We achieve this by empirically estimating its KL-divergence in the dual form with respect to the respective marginal distribution. This enables us to perform generative sampling directly in the optimized 1-D dual divergence space. Specifically, in the dual space, training samples representing the data distribution are embedded in the form of various clusters between two end points. In theory, any sample embedded between those two end points is in-distribution w.r.t. the data distribution. Our key idea for generating novel samples of images is to interpolate between the clusters via a walk as per gradients of the dual function w.r.t. the data dimensions. In addition to the data efficiency gained from direct sampling, we propose an algorithm that offers a significant reduction in sample complexity for estimating the divergence of the data distribution with respect to the marginal distribution. We provide strong theoretical guarantees along with an extensive empirical evaluation using many real-world datasets from diverse domains, establishing the superiority of our approach w.r.t. state-of-the-art deep learning methods.

Title: Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Models

• Authors: Yasi Zhang, Peiyu Yu, Ying Nian Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07389
• Pdf URL: https://arxiv.org/pdf/2404.07389
• Copy Paste: [[2404.07389]] Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Models(https://arxiv.org/abs/2404.07389)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models have shown great success in generating high-quality text-guided images. Yet, these models may still fail to semantically align generated images with the provided text prompts, leading to problems like incorrect attribute binding and/or catastrophic object neglect. Given the pervasive object-oriented structure underlying text prompts, we introduce a novel object-conditioned Energy-Based Attention Map Alignment (EBAMA) method to address the aforementioned problems. We show that an object-centric attribute binding loss naturally emerges by approximately maximizing the log-likelihood of a $z$-parameterized energy-based model with the help of the negative sampling technique. We further propose an object-centric intensity regularizer to prevent excessive shifts of objects attention towards their attributes. Extensive qualitative and quantitative experiments, including human evaluation, on several challenging benchmarks demonstrate the superior performance of our method over previous strong counterparts. With better aligned attention maps, our approach shows great promise in further enhancing the text-controlled image editing ability of diffusion models.

Title: JetMoE: Reaching Llama2 Performance with 0.1M Dollars

• Authors: Yikang Shen, Zhen Guo, Tianle Cai, Zengyi Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07413
• Pdf URL: https://arxiv.org/pdf/2404.07413
• Copy Paste: [[2404.07413]] JetMoE: Reaching Llama2 Performance with 0.1M Dollars(https://arxiv.org/abs/2404.07413)
• Keywords: foundation model
• Abstract: Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the Llama2-13B-Chat model. These results suggest that LLM training can be much more cost-effective than generally thought. JetMoE-8B is based on an efficient Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B to have 8B parameters while only activating 2B for each input token, reducing inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B is highly open and academia-friendly, using only public datasets and training code. All training parameters and data mixtures have been detailed in this report to facilitate future efforts in the development of open foundation models. This transparency aims to encourage collaboration and further advancements in the field of accessible and efficient LLMs. The model weights are publicly available at https://github.com/myshell-ai/JetMoE.

Title: CopilotCAD: Empowering Radiologists with Report Completion Models and Quantitative Evidence from Medical Image Foundation Models

• Authors: Sheng Wang, Tianming Du, Katherine Fischer, Gregory E Tasian, Justin Ziemba, Joanie M Garratt, Hersh Sagreiya, Yong Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07424
• Pdf URL: https://arxiv.org/pdf/2404.07424
• Copy Paste: [[2404.07424]] CopilotCAD: Empowering Radiologists with Report Completion Models and Quantitative Evidence from Medical Image Foundation Models(https://arxiv.org/abs/2404.07424)
• Keywords: foundation model
• Abstract: Computer-aided diagnosis systems hold great promise to aid radiologists and clinicians in radiological clinical practice and enhance diagnostic accuracy and efficiency. However, the conventional systems primarily focus on delivering diagnostic results through text report generation or medical image classification, positioning them as standalone decision-makers rather than helpers and ignoring radiologists' expertise. This study introduces an innovative paradigm to create an assistive co-pilot system for empowering radiologists by leveraging Large Language Models (LLMs) and medical image analysis tools. Specifically, we develop a collaborative framework to integrate LLMs and quantitative medical image analysis results generated by foundation models with radiologists in the loop, achieving efficient and safe generation of radiology reports and effective utilization of computational power of AI and the expertise of medical professionals. This approach empowers radiologists to generate more precise and detailed diagnostic reports, enhancing patient outcomes while reducing the burnout of clinicians. Our methodology underscores the potential of AI as a supportive tool in medical diagnostics, promoting a harmonious integration of technology and human expertise to advance the field of radiology.

Title: Encoding Urban Ecologies: Automated Building Archetype Generation through Self-Supervised Learning for Energy Modeling

• Authors: Xinwei Zhuang, Zixun Huang, Wentao Zeng, Luisa Caldas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07435
• Pdf URL: https://arxiv.org/pdf/2404.07435
• Copy Paste: [[2404.07435]] Encoding Urban Ecologies: Automated Building Archetype Generation through Self-Supervised Learning for Energy Modeling(https://arxiv.org/abs/2404.07435)
• Keywords: self-supervised
• Abstract: As the global population and urbanization expand, the building sector has emerged as the predominant energy consumer and carbon emission contributor. The need for innovative Urban Building Energy Modeling grows, yet existing building archetypes often fail to capture the unique attributes of local buildings and the nuanced distinctions between different cities, jeopardizing the precision of energy modeling. This paper presents an alternative tool employing self-supervised learning to distill complex geometric data into representative, locale-specific archetypes. This study attempts to foster a new paradigm of interaction with built environments, incorporating local parameters to conduct bespoke energy simulations at the community level. The catered archetypes can augment the precision and applicability of energy consumption modeling at different scales across diverse building inventories. This tool provides a potential solution that encourages the exploration of emerging local ecologies. By integrating building envelope characteristics and cultural granularity into the building archetype generation process, we seek a future where architecture and urban design are intricately interwoven with the energy sector in shaping our built environments.

Title: Privacy preserving layer partitioning for Deep Neural Network models

• Authors: Kishore Rajasekar, Randolph Loh, Kar Wai Fok, Vrizlynn L. L. Thing
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07437
• Pdf URL: https://arxiv.org/pdf/2404.07437
• Copy Paste: [[2404.07437]] Privacy preserving layer partitioning for Deep Neural Network models(https://arxiv.org/abs/2404.07437)
• Keywords: generative
• Abstract: MLaaS (Machine Learning as a Service) has become popular in the cloud computing domain, allowing users to leverage cloud resources for running private inference of ML models on their data. However, ensuring user input privacy and secure inference execution is essential. One of the approaches to protect data privacy and integrity is to use Trusted Execution Environments (TEEs) by enabling execution of programs in secure hardware enclave. Using TEEs can introduce significant performance overhead due to the additional layers of encryption, decryption, security and integrity checks. This can lead to slower inference times compared to running on unprotected hardware. In our work, we enhance the runtime performance of ML models by introducing layer partitioning technique and offloading computations to GPU. The technique comprises two distinct partitions: one executed within the TEE, and the other carried out using a GPU accelerator. Layer partitioning exposes intermediate feature maps in the clear which can lead to reconstruction attacks to recover the input. We conduct experiments to demonstrate the effectiveness of our approach in protecting against input reconstruction attacks developed using trained conditional Generative Adversarial Network(c-GAN). The evaluation is performed on widely used models such as VGG-16, ResNet-50, and EfficientNetB0, using two datasets: ImageNet for Image classification and TON IoT dataset for cybersecurity attack detection.

Title: Transferable and Principled Efficiency for Open-Vocabulary Segmentation

• Authors: Jingxuan Xu, Wuyang Chen, Yao Zhao, Yunchao Wei
• Subjects: cs.CV, cs.CL, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.07448
• Pdf URL: https://arxiv.org/pdf/2404.07448
• Copy Paste: [[2404.07448]] Transferable and Principled Efficiency for Open-Vocabulary Segmentation(https://arxiv.org/abs/2404.07448)
• Keywords: foundation model
• Abstract: Recent success of pre-trained foundation vision-language models makes Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance, this approach introduces heavy computational overheads for two challenges: 1) large model sizes of the backbone; 2) expensive costs during the fine-tuning. These challenges hinder this OVS strategy from being widely applicable and affordable in real-world scenarios. Although traditional methods such as model compression and efficient fine-tuning can address these challenges, they often rely on heuristics. This means that their solutions cannot be easily transferred and necessitate re-training on different models, which comes at a cost. In the context of efficient OVS, we target achieving performance that is comparable to or even better than prior OVS works based on large vision-language foundation models, by utilizing smaller models that incur lower training costs. The core strategy is to make our efficiency principled and thus seamlessly transferable from one OVS framework to others without further customization. Comprehensive experiments on diverse OVS benchmarks demonstrate our superior trade-off between segmentation accuracy and computation costs over previous works. Our code is available on https://github.com/Xujxyang/OpenTrans

Title: Enhancing Network Intrusion Detection Performance using Generative Adversarial Networks

• Authors: Xinxing Zhao, Kar Wai Fok, Vrizlynn L. L. Thing
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07464
• Pdf URL: https://arxiv.org/pdf/2404.07464
• Copy Paste: [[2404.07464]] Enhancing Network Intrusion Detection Performance using Generative Adversarial Networks(https://arxiv.org/abs/2404.07464)
• Keywords: generative
• Abstract: Network intrusion detection systems (NIDS) play a pivotal role in safeguarding critical digital infrastructures against cyber threats. Machine learning-based detection models applied in NIDS are prevalent today. However, the effectiveness of these machine learning-based models is often limited by the evolving and sophisticated nature of intrusion techniques as well as the lack of diverse and updated training samples. In this research, a novel approach for enhancing the performance of an NIDS through the integration of Generative Adversarial Networks (GANs) is proposed. By harnessing the power of GANs in generating synthetic network traffic data that closely mimics real-world network behavior, we address a key challenge associated with NIDS training datasets, which is the data scarcity. Three distinct GAN models (Vanilla GAN, Wasserstein GAN and Conditional Tabular GAN) are implemented in this work to generate authentic network traffic patterns specifically tailored to represent the anomalous activity. We demonstrate how this synthetic data resampling technique can significantly improve the performance of the NIDS model for detecting such activity. By conducting comprehensive experiments using the CIC-IDS2017 benchmark dataset, augmented with GAN-generated data, we offer empirical evidence that shows the effectiveness of our proposed approach. Our findings show that the integration of GANs into NIDS can lead to enhancements in intrusion detection performance for attacks with limited training data, making it a promising avenue for bolstering the cybersecurity posture of organizations in an increasingly interconnected and vulnerable digital landscape.

Title: Laissez-Faire Harms: Algorithmic Biases in Generative Language Models

• Authors: Evan Shieh, Faye-Marie Vassel, Cassidy Sugimoto, Thema Monroe-White
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07475
• Pdf URL: https://arxiv.org/pdf/2404.07475
• Copy Paste: [[2404.07475]] Laissez-Faire Harms: Algorithmic Biases in Generative Language Models(https://arxiv.org/abs/2404.07475)
• Keywords: generative
• Abstract: The rapid deployment of generative language models (LMs) has raised concerns about social biases affecting the well-being of diverse consumers. The extant literature on generative LMs has primarily examined bias via explicit identity prompting. However, prior research on bias in earlier language-based technology platforms, including search engines, has shown that discrimination can occur even when identity terms are not specified explicitly. Studies of bias in LM responses to open-ended prompts (where identity classifications are left unspecified) are lacking and have not yet been grounded in end-consumer harms. Here, we advance studies of generative LM bias by considering a broader set of natural use cases via open-ended prompting. In this "laissez-faire" setting, we find that synthetically generated texts from five of the most pervasive LMs (ChatGPT3.5, ChatGPT4, Claude2.0, Llama2, and PaLM2) perpetuate harms of omission, subordination, and stereotyping for minoritized individuals with intersectional race, gender, and/or sexual orientation identities (AI/AN, Asian, Black, Latine, MENA, NH/PI, Female, Non-binary, Queer). We find widespread evidence of bias to an extent that such individuals are hundreds to thousands of times more likely to encounter LM-generated outputs that portray their identities in a subordinated manner compared to representative or empowering portrayals. We also document a prevalence of stereotypes (e.g. perpetual foreigner) in LM-generated outputs that are known to trigger psychological harms that disproportionately affect minoritized individuals. These include stereotype threat, which leads to impaired cognitive performance and increased negative self-perception. Our findings highlight the urgent need to protect consumers from discriminatory harms caused by language models and invest in critical AI education programs tailored towards empowering diverse consumers.

Title: Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange

• Authors: Yanhao Wu, Tong Zhang, Wei Ke, Congpei Qiu, Sabine Susstrunk, Mathieu Salzmann
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07504
• Pdf URL: https://arxiv.org/pdf/2404.07504
• Copy Paste: [[2404.07504]] Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange(https://arxiv.org/abs/2404.07504)
• Keywords: self-supervised
• Abstract: In the realm of point cloud scene understanding, particularly in indoor scenes, objects are arranged following human habits, resulting in objects of certain semantics being closely positioned and displaying notable inter-object correlations. This can create a tendency for neural networks to exploit these strong dependencies, bypassing the individual object patterns. To address this challenge, we introduce a novel self-supervised learning (SSL) strategy. Our approach leverages both object patterns and contextual cues to produce robust features. It begins with the formulation of an object-exchanging strategy, where pairs of objects with comparable sizes are exchanged across different scenes, effectively disentangling the strong contextual dependencies. Subsequently, we introduce a context-aware feature learning strategy, which encodes object patterns without relying on their specific context by aggregating object features across various scenes. Our extensive experiments demonstrate the superiority of our method over existing SSL techniques, further showing its better robustness to environmental changes. Moreover, we showcase the applicability of our approach by transferring pre-trained models to diverse point cloud datasets.

Title: Remembering Transformer for Continual Learning

• Authors: Yuwei Sun, Jun Sakuma, Ryota Kanai
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07518
• Pdf URL: https://arxiv.org/pdf/2404.07518
• Copy Paste: [[2404.07518]] Remembering Transformer for Continual Learning(https://arxiv.org/abs/2404.07518)
• Keywords: generative
• Abstract: Neural networks encounter the challenge of Catastrophic Forgetting (CF) in continual learning, where new task knowledge interferes with previously learned knowledge. We propose Remembering Transformer, inspired by the brain's Complementary Learning Systems (CLS), to tackle this issue. Remembering Transformer employs a mixture-of-adapters and a generative model-based routing mechanism to alleviate CF by dynamically routing task data to relevant adapters. Our approach demonstrated a new SOTA performance in various vision continual learning tasks and great parameter efficiency.

Title: From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

• Authors: Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, Mihai Surdeanu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07544
• Pdf URL: https://arxiv.org/pdf/2404.07544
• Copy Paste: [[2404.07544]] From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples(https://arxiv.org/abs/2404.07544)
• Keywords: in-context
• Abstract: We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g., GPT-4, Claude 3) are able to perform regression tasks with a performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, on the challenging Friedman #2 regression dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM, Random Forest, KNN, or Gradient Boosting. We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining a sub-linear regret.

Title: Decomposing Label Space, Format and Discrimination: Rethinking How LLMs Respond and Solve Tasks via In-Context Learning

• Authors: Quanyu Long, Yin Wu, Wenya Wang, Sinno Jialin Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07546
• Pdf URL: https://arxiv.org/pdf/2404.07546
• Copy Paste: [[2404.07546]] Decomposing Label Space, Format and Discrimination: Rethinking How LLMs Respond and Solve Tasks via In-Context Learning(https://arxiv.org/abs/2404.07546)
• Keywords: in-context
• Abstract: In-context Learning (ICL) has emerged as a powerful capability alongside the development of scaled-up large language models (LLMs). By instructing LLMs using few-shot demonstrative examples, ICL enables them to perform a wide range of tasks without updating millions of parameters. However, the precise contributions of demonstrations towards improving end-task performance have not been thoroughly investigated in recent analytical studies. In this paper, we empirically decompose the overall performance of ICL into three dimensions, label space, format, and discrimination, and we evaluate four general-purpose LLMs across a diverse range of tasks. Counter-intuitively, we find that the demonstrations have a marginal impact on provoking discriminative knowledge of language models. However, ICL exhibits significant efficacy in regulating the label space and format which helps LLMs to respond in desired label words. We then demonstrate this ability functions similar to detailed instructions for LLMs to follow. We additionally provide an in-depth analysis of the mechanism of retrieval helping with ICL and find that retrieving the most semantically similar examples notably boosts model's discriminative capability.

Title: CAT: Contrastive Adapter Training for Personalized Image Generation

• Authors: Jae Wan Park, Sang Hyun Park, Jun Young Koh, Junha Lee, Min Song
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07554
• Pdf URL: https://arxiv.org/pdf/2404.07554
• Copy Paste: [[2404.07554]] CAT: Contrastive Adapter Training for Personalized Image Generation(https://arxiv.org/abs/2404.07554)
• Keywords: diffusion
• Abstract: The emergence of various adapters, including Low-Rank Adaptation (LoRA) applied from the field of natural language processing, has allowed diffusion models to personalize image generation at a low cost. However, due to the various challenges including limited datasets and shortage of regularization and computation resources, adapter training often results in unsatisfactory outcomes, leading to the corruption of the backbone model's prior knowledge. One of the well known phenomena is the loss of diversity in object generation, especially within the same class which leads to generating almost identical objects with minor variations. This poses challenges in generation capabilities. To solve this issue, we present Contrastive Adapter Training (CAT), a simple yet effective strategy to enhance adapter training through the application of CAT loss. Our approach facilitates the preservation of the base model's original knowledge when the model initiates adapters. Furthermore, we introduce the Knowledge Preservation Score (KPS) to evaluate CAT's ability to keep the former information. We qualitatively and quantitatively compare CAT's improvement. Finally, we mention the possibility of CAT in the aspects of multi-concept adapter and optimization.

Title: ObjBlur: A Curriculum Learning Approach With Progressive Object-Level Blurring for Improved Layout-to-Image Generation

• Authors: Stanislav Frolov, Brian B. Moser, Sebastian Palacio, Andreas Dengel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07564
• Pdf URL: https://arxiv.org/pdf/2404.07564
• Copy Paste: [[2404.07564]] ObjBlur: A Curriculum Learning Approach With Progressive Object-Level Blurring for Improved Layout-to-Image Generation(https://arxiv.org/abs/2404.07564)
• Keywords: diffusion, generative
• Abstract: We present ObjBlur, a novel curriculum learning approach to improve layout-to-image generation models, where the task is to produce realistic images from layouts composed of boxes and labels. Our method is based on progressive object-level blurring, which effectively stabilizes training and enhances the quality of generated images. This curriculum learning strategy systematically applies varying degrees of blurring to individual objects or the background during training, starting from strong blurring to progressively cleaner images. Our findings reveal that this approach yields significant performance improvements, stabilized training, smoother convergence, and reduced variance between multiple runs. Moreover, our technique demonstrates its versatility by being compatible with generative adversarial networks and diffusion models, underlining its applicability across various generative modeling paradigms. With ObjBlur, we reach new state-of-the-art results on the complex COCO and Visual Genome datasets.

Title: Generating Comprehensive Lithium Battery Charging Data with Generative AI

• Authors: Lidang Jiang, Changyan Hu, Sibei Ji, Hang Zhao, Junxiong Chen, Ge He
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.07577
• Pdf URL: https://arxiv.org/pdf/2404.07577
• Copy Paste: [[2404.07577]] Generating Comprehensive Lithium Battery Charging Data with Generative AI(https://arxiv.org/abs/2404.07577)
• Keywords: generative
• Abstract: In optimizing performance and extending the lifespan of lithium batteries, accurate state prediction is pivotal. Traditional regression and classification methods have achieved some success in battery state prediction. However, the efficacy of these data-driven approaches heavily relies on the availability and quality of public datasets. Additionally, generating electrochemical data predominantly through battery experiments is a lengthy and costly process, making it challenging to acquire high-quality electrochemical data. This difficulty, coupled with data incompleteness, significantly impacts prediction accuracy. Addressing these challenges, this study introduces the End of Life (EOL) and Equivalent Cycle Life (ECL) as conditions for generative AI models. By integrating an embedding layer into the CVAE model, we developed the Refined Conditional Variational Autoencoder (RCVAE). Through preprocessing data into a quasi-video format, our study achieves an integrated synthesis of electrochemical data, including voltage, current, temperature, and charging capacity, which is then processed by the RCVAE model. Coupled with customized training and inference algorithms, this model can generate specific electrochemical data for EOL and ECL under supervised conditions. This method provides users with a comprehensive electrochemical dataset, pioneering a new research domain for the artificial synthesis of lithium battery data. Furthermore, based on the detailed synthetic data, various battery state indicators can be calculated, offering new perspectives and possibilities for lithium battery performance prediction.

Title: Implicit and Explicit Language Guidance for Diffusion-based Visual Perception

• Authors: Hefeng Wang, Jiale Cao, Jin Xie, Aiping Yang, Yanwei Pang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07600
• Pdf URL: https://arxiv.org/pdf/2404.07600
• Copy Paste: [[2404.07600]] Implicit and Explicit Language Guidance for Diffusion-based Visual Perception(https://arxiv.org/abs/2404.07600)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models have shown powerful ability on conditional image synthesis. With large-scale vision-language pre-training, diffusion models are able to generate high-quality images with rich texture and reasonable structure under different text prompts. However, it is an open problem to adapt the pre-trained diffusion model for visual perception. In this paper, we propose an implicit and explicit language guidance framework for diffusion-based perception, named IEDP. Our IEDP comprises of an implicit language guidance branch and an explicit language guidance branch. The implicit branch employs frozen CLIP image encoder to directly generate implicit text embeddings that are fed to diffusion model, without using explicit text prompts. The explicit branch utilizes the ground-truth labels of corresponding images as text prompts to condition feature extraction of diffusion model. During training, we jointly train diffusion model by sharing the model weights of these two branches. As a result, implicit and explicit branches can jointly guide feature learning. During inference, we only employ implicit branch for final prediction, which does not require any ground-truth labels. Experiments are performed on two typical perception tasks, including semantic segmentation and depth estimation. Our IEDP achieves promising performance on both tasks. For semantic segmentation, our IEDP has the mIoU score of 55.9% on AD20K validation set, which outperforms the baseline method VPD by 2.2%. For depth estimation, our IEDP outperforms the baseline method VPD with a relative gain of 10.2%.

Title: GLID: Pre-training a Generalist Encoder-Decoder Vision Model

• Authors: Jihao Liu, Jinliang Zheng, Yu Liu, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07603
• Pdf URL: https://arxiv.org/pdf/2404.07603
• Copy Paste: [[2404.07603]] GLID: Pre-training a Generalist Encoder-Decoder Vision Model(https://arxiv.org/abs/2404.07603)
• Keywords: self-supervised
• Abstract: This paper proposes a GeneraLIst encoder-Decoder (GLID) pre-training method for better handling various downstream computer vision tasks. While self-supervised pre-training approaches, e.g., Masked Autoencoder, have shown success in transfer learning, task-specific sub-architectures are still required to be appended for different downstream tasks, which cannot enjoy the benefits of large-scale pre-training. GLID overcomes this challenge by allowing the pre-trained generalist encoder-decoder to be fine-tuned on various vision tasks with minimal task-specific architecture modifications. In the GLID training scheme, pre-training pretext task and other downstream tasks are modeled as "query-to-answer" problems, including the pre-training pretext task and other downstream tasks. We pre-train a task-agnostic encoder-decoder with query-mask pairs. During fine-tuning, GLID maintains the pre-trained encoder-decoder and queries, only replacing the topmost linear transformation layer with task-specific linear heads. This minimizes the pretrain-finetune architecture inconsistency and enables the pre-trained model to better adapt to downstream tasks. GLID achieves competitive performance on various vision tasks, including object detection, image segmentation, pose estimation, and depth estimation, outperforming or matching specialist models such as Mask2Former, DETR, ViTPose, and BinsFormer.

Title: Contrastive-Based Deep Embeddings for Label Noise-Resilient Histopathology Image Classification

• Authors: Lucas Dedieu, Nicolas Nerrienet, Adrien Nivaggioli, Clara Simmat, Marceau Clavel, Arnaud Gauthier, Stéphane Sockeel, Rémy Peyret
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07605
• Pdf URL: https://arxiv.org/pdf/2404.07605
• Copy Paste: [[2404.07605]] Contrastive-Based Deep Embeddings for Label Noise-Resilient Histopathology Image Classification(https://arxiv.org/abs/2404.07605)
• Keywords: self-supervised, foundation model
• Abstract: Recent advancements in deep learning have proven highly effective in medical image classification, notably within histopathology. However, noisy labels represent a critical challenge in histopathology image classification, where accurate annotations are vital for training robust deep learning models. Indeed, deep neural networks can easily overfit label noise, leading to severe degradations in model performance. While numerous public pathology foundation models have emerged recently, none have evaluated their resilience to label noise. Through thorough empirical analyses across multiple datasets, we exhibit the label noise resilience property of embeddings extracted from foundation models trained in a self-supervised contrastive manner. We demonstrate that training with such embeddings substantially enhances label noise robustness when compared to non-contrastive-based ones as well as commonly used noise-resilient methods. Our results unequivocally underline the superiority of contrastive learning in effectively mitigating the label noise challenge. Code is publicly available at https://github.com/LucasDedieu/NoiseResilientHistopathology.

Title: NoticIA: A Clickbait Article Summarization Dataset in Spanish

• Authors: Iker García-Ferrero, Begoña Altuna
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07611
• Pdf URL: https://arxiv.org/pdf/2404.07611
• Copy Paste: [[2404.07611]] NoticIA: A Clickbait Article Summarization Dataset in Spanish(https://arxiv.org/abs/2404.07611)
• Keywords: generative
• Abstract: We present NoticIA, a dataset consisting of 850 Spanish news articles featuring prominent clickbait headlines, each paired with high-quality, single-sentence generative summarizations written by humans. This task demands advanced text understanding and summarization abilities, challenging the models' capacity to infer and connect diverse pieces of information to meet the user's informational needs generated by the clickbait headline. We evaluate the Spanish text comprehension capabilities of a wide range of state-of-the-art large language models. Additionally, we use the dataset to train ClickbaitFighter, a task-specific model that achieves near-human performance in this task.

Title: Multi-Image Visual Question Answering for Unsupervised Anomaly Detection

• Authors: Jun Li, Cosmin I. Bercea, Philip Müller, Lina Felsner, Suhwan Kim, Daniel Rueckert, Benedikt Wiestler, Julia A. Schnabel
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07622
• Pdf URL: https://arxiv.org/pdf/2404.07622
• Copy Paste: [[2404.07622]] Multi-Image Visual Question Answering for Unsupervised Anomaly Detection(https://arxiv.org/abs/2404.07622)
• Keywords: anomaly
• Abstract: Unsupervised anomaly detection enables the identification of potential pathological areas by juxtaposing original images with their pseudo-healthy reconstructions generated by models trained exclusively on normal images. However, the clinical interpretation of resultant anomaly maps presents a challenge due to a lack of detailed, understandable explanations. Recent advancements in language models have shown the capability of mimicking human-like understanding and providing detailed descriptions. This raises an interesting question: \textit{How can language models be employed to make the anomaly maps more explainable?} To the best of our knowledge, we are the first to leverage a language model for unsupervised anomaly detection, for which we construct a dataset with different questions and answers. Additionally, we present a novel multi-image visual question answering framework tailored for anomaly detection, incorporating diverse feature fusion strategies to enhance visual knowledge extraction. Our experiments reveal that the framework, augmented by our new Knowledge Q-Former module, adeptly answers questions on the anomaly detection dataset. Besides, integrating anomaly maps as inputs distinctly aids in improving the detection of unseen pathologies.

Title: rollama: An R package for using generative large language models through Ollama

• Authors: Johannes B. Gruber, Maximilian Weber
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07654
• Pdf URL: https://arxiv.org/pdf/2404.07654
• Copy Paste: [[2404.07654]] rollama: An R package for using generative large language models through Ollama(https://arxiv.org/abs/2404.07654)
• Keywords: generative
• Abstract: rollama is an R package that wraps the Ollama API, which allows you to run different Generative Large Language Models (GLLM) locally. The package and learning material focus on making it easy to use Ollama for annotating textual or imagine data with open-source models as well as use these models for document embedding. But users can use or extend rollama to do essentially anything else that is possible through OpenAI's API, yet more private, reproducible and for free.

Title: Finding Dino: A plug-and-play framework for unsupervised detection of out-of-distribution objects using prototypes

• Authors: Poulami Sinhamahapatra, Franziska Schwaiger, Shirsha Bose, Huiyu Wang, Karsten Roscher, Stephan Guennemann
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07664
• Pdf URL: https://arxiv.org/pdf/2404.07664
• Copy Paste: [[2404.07664]] Finding Dino: A plug-and-play framework for unsupervised detection of out-of-distribution objects using prototypes(https://arxiv.org/abs/2404.07664)
• Keywords: self-supervised, anomaly
• Abstract: Detecting and localising unknown or Out-of-distribution (OOD) objects in any scene can be a challenging task in vision. Particularly, in safety-critical cases involving autonomous systems like automated vehicles or trains. Supervised anomaly segmentation or open-world object detection models depend on training on exhaustively annotated datasets for every domain and still struggle in distinguishing between background and OOD objects. In this work, we present a plug-and-play generalised framework - PRototype-based zero-shot OOD detection Without Labels (PROWL). It is an inference-based method that does not require training on the domain dataset and relies on extracting relevant features from self-supervised pre-trained models. PROWL can be easily adapted to detect OOD objects in any operational design domain by specifying a list of known classes from this domain. PROWL, as an unsupervised method, outperforms other supervised methods trained without auxiliary OOD data on the RoadAnomaly and RoadObstacle datasets provided in SegmentMeIfYouCan (SMIYC) benchmark. We also demonstrate its suitability for other domains such as rail and maritime scenes.

Title: Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models

• Authors: Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, Jaakko Lehtinen
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.07724
• Pdf URL: https://arxiv.org/pdf/2404.07724
• Copy Paste: [[2404.07724]] Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models(https://arxiv.org/abs/2404.07724)
• Keywords: diffusion
• Abstract: Guidance is a crucial technique for extracting the best performance out of image-generating diffusion models. Traditionally, a constant guidance weight has been applied throughout the sampling chain of an image. We show that guidance is clearly harmful toward the beginning of the chain (high noise levels), largely unnecessary toward the end (low noise levels), and only beneficial in the middle. We thus restrict it to a specific range of noise levels, improving both the inference speed and result quality. This limited guidance interval improves the record FID in ImageNet-512 significantly, from 1.81 to 1.40. We show that it is quantitatively and qualitatively beneficial across different sampler parameters, network architectures, and datasets, including the large-scale setting of Stable Diffusion XL. We thus suggest exposing the guidance interval as a hyperparameter in all diffusion models that use guidance.

Title: 3D-CSAD: Untrained 3D Anomaly Detection for Complex Manufacturing Surfaces

• Authors: Xuanming Cao, Chengyu Tao, Juan Du
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07748
• Pdf URL: https://arxiv.org/pdf/2404.07748
• Copy Paste: [[2404.07748]] 3D-CSAD: Untrained 3D Anomaly Detection for Complex Manufacturing Surfaces(https://arxiv.org/abs/2404.07748)
• Keywords: anomaly
• Abstract: The surface quality inspection of manufacturing parts based on 3D point cloud data has attracted increasing attention in recent years. The reason is that the 3D point cloud can capture the entire surface of manufacturing parts, unlike the previous practices that focus on some key product characteristics. However, achieving accurate 3D anomaly detection is challenging, due to the complex surfaces of manufacturing parts and the difficulty of collecting sufficient anomaly samples. To address these challenges, we propose a novel untrained anomaly detection method based on 3D point cloud data for complex manufacturing parts, which can achieve accurate anomaly detection in a single sample without training data. In the proposed framework, we transform an input sample into two sets of profiles along different directions. Based on one set of the profiles, a novel segmentation module is devised to segment the complex surface into multiple basic and simple components. In each component, another set of profiles, which have the nature of similar shapes, can be modeled as a low-rank matrix. Thus, accurate 3D anomaly detection can be achieved by using Robust Principal Component Analysis (RPCA) on these low-rank matrices. Extensive numerical experiments on different types of parts show that our method achieves promising results compared with the benchmark methods.

Title: Generating Synthetic Satellite Imagery With Deep-Learning Text-to-Image Models -- Technical Challenges and Implications for Monitoring and Verification

• Authors: Tuong Vy Nguyen, Alexander Glaser, Felix Biessmann
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07754
• Pdf URL: https://arxiv.org/pdf/2404.07754
• Copy Paste: [[2404.07754]] Generating Synthetic Satellite Imagery With Deep-Learning Text-to-Image Models -- Technical Challenges and Implications for Monitoring and Verification(https://arxiv.org/abs/2404.07754)
• Keywords: diffusion
• Abstract: Novel deep-learning (DL) architectures have reached a level where they can generate digital media, including photorealistic images, that are difficult to distinguish from real data. These technologies have already been used to generate training data for Machine Learning (ML) models, and large text-to-image models like DALL-E 2, Imagen, and Stable Diffusion are achieving remarkable results in realistic high-resolution image generation. Given these developments, issues of data authentication in monitoring and verification deserve a careful and systematic analysis: How realistic are synthetic images? How easily can they be generated? How useful are they for ML researchers, and what is their potential for Open Science? In this work, we use novel DL models to explore how synthetic satellite images can be created using conditioning mechanisms. We investigate the challenges of synthetic satellite image generation and evaluate the results based on authenticity and state-of-the-art metrics. Furthermore, we investigate how synthetic data can alleviate the lack of data in the context of ML methods for remote-sensing. Finally we discuss implications of synthetic satellite imagery in the context of monitoring and verification.

Title: Joint Conditional Diffusion Model for Image Restoration with Mixed Degradations

• Authors: Yufeng Yue, Meng Yu, Luojie Yang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07770
• Pdf URL: https://arxiv.org/pdf/2404.07770
• Copy Paste: [[2404.07770]] Joint Conditional Diffusion Model for Image Restoration with Mixed Degradations(https://arxiv.org/abs/2404.07770)
• Keywords: diffusion, generative
• Abstract: Image restoration is rather challenging in adverse weather conditions, especially when multiple degradations occur simultaneously. Blind image decomposition was proposed to tackle this issue, however, its effectiveness heavily relies on the accurate estimation of each component. Although diffusion-based models exhibit strong generative abilities in image restoration tasks, they may generate irrelevant contents when the degraded images are severely corrupted. To address these issues, we leverage physical constraints to guide the whole restoration process, where a mixed degradation model based on atmosphere scattering model is constructed. Then we formulate our Joint Conditional Diffusion Model (JCDM) by incorporating the degraded image and degradation mask to provide precise guidance. To achieve better color and detail recovery results, we further integrate a refinement network to reconstruct the restored image, where Uncertainty Estimation Block (UEB) is employed to enhance the features. Extensive experiments performed on both multi-weather and weather-specific datasets demonstrate the superiority of our method over state-of-the-art competing methods.

Title: An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization

• Authors: Minshuo Chen, Song Mei, Jianqing Fan, Mengdi Wang
• Subjects: cs.LG, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.07771
• Pdf URL: https://arxiv.org/pdf/2404.07771
• Copy Paste: [[2404.07771]] An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization(https://arxiv.org/abs/2404.07771)
• Keywords: diffusion, generative
• Abstract: Diffusion models, a powerful and universal generative AI technology, have achieved tremendous success in computer vision, audio, reinforcement learning, and computational biology. In these applications, diffusion models provide flexible high-dimensional data modeling, and act as a sampler for generating new samples under active guidance towards task-desired properties. Despite the significant empirical success, theory of diffusion models is very limited, potentially slowing down principled methodological innovations for further harnessing and improving diffusion models. In this paper, we review emerging applications of diffusion models, understanding their sample generation under various controls. Next, we overview the existing theories of diffusion models, covering their statistical properties and sampling capabilities. We adopt a progressive routine, beginning with unconditional diffusion models and connecting to conditional counterparts. Further, we review a new avenue in high-dimensional structured optimization through conditional diffusion models, where searching for solutions is reformulated as a conditional sampling problem and solved by diffusion models. Lastly, we discuss future directions about diffusion models. The purpose of this paper is to provide a well-rounded theoretical exposure for stimulating forward-looking theories and methods of diffusion models.

Title: ConsistencyDet: Robust Object Detector with Denoising Paradigm of Consistency Model

• Authors: Lifan Jiang, Zhihui Wang, Changmiao Wang, Ming Li, Jiaxu Leng, Xindong Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07773
• Pdf URL: https://arxiv.org/pdf/2404.07773
• Copy Paste: [[2404.07773]] ConsistencyDet: Robust Object Detector with Denoising Paradigm of Consistency Model(https://arxiv.org/abs/2404.07773)
• Keywords: diffusion, generative
• Abstract: Object detection, a quintessential task in the realm of perceptual computing, can be tackled using a generative methodology. In the present study, we introduce a novel framework designed to articulate object detection as a denoising diffusion process, which operates on perturbed bounding boxes of annotated entities. This framework, termed ConsistencyDet, leverages an innovative denoising concept known as the Consistency Model. The hallmark of this model is its self-consistency feature, which empowers the model to map distorted information from any temporal stage back to its pristine state, thereby realizing a ``one-step denoising'' mechanism. Such an attribute markedly elevates the operational efficiency of the model, setting it apart from the conventional Diffusion Model. Throughout the training phase, ConsistencyDet initiates the diffusion sequence with noise-infused boxes derived from the ground-truth annotations and conditions the model to perform the denoising task. Subsequently, in the inference stage, the model employs a denoising sampling strategy that commences with bounding boxes randomly sampled from a normal distribution. Through iterative refinement, the model transforms an assortment of arbitrarily generated boxes into the definitive detections. Comprehensive evaluations employing standard benchmarks, such as MS-COCO and LVIS, corroborate that ConsistencyDet surpasses other leading-edge detectors in performance metrics.

Title: Discourse-Aware In-Context Learning for Temporal Expression Normalization

• Authors: Akash Kumar Gautam, Lukas Lange, Jannik Strötgen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07775
• Pdf URL: https://arxiv.org/pdf/2404.07775
• Copy Paste: [[2404.07775]] Discourse-Aware In-Context Learning for Temporal Expression Normalization(https://arxiv.org/abs/2404.07775)
• Keywords: in-context
• Abstract: Temporal expression (TE) normalization is a well-studied problem. However, the predominately used rule-based systems are highly restricted to specific settings, and upcoming machine learning approaches suffer from a lack of labeled data. In this work, we explore the feasibility of proprietary and open-source large language models (LLMs) for TE normalization using in-context learning to inject task, document, and example information into the model. We explore various sample selection strategies to retrieve the most relevant set of examples. By using a window-based prompt design approach, we can perform TE normalization across sentences, while leveraging the LLM knowledge without training the model. Our experiments show competitive results to models designed for this task. In particular, our method achieves large performance improvements for non-standard settings by dynamically including relevant examples during inference.

Title: PRAM: Place Recognition Anywhere Model for Efficient Visual Localization

• Authors: Fei Xue, Ignas Budvytis, Roberto Cipolla
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.07785
• Pdf URL: https://arxiv.org/pdf/2404.07785
• Copy Paste: [[2404.07785]] PRAM: Place Recognition Anywhere Model for Efficient Visual Localization(https://arxiv.org/abs/2404.07785)
• Keywords: self-supervised
• Abstract: Humans localize themselves efficiently in known environments by first recognizing landmarks defined on certain objects and their spatial relationships, and then verifying the location by aligning detailed structures of recognized objects with those in the memory. Inspired by this, we propose the place recognition anywhere model (PRAM) to perform visual localization as efficiently as humans do. PRAM consists of two main components - recognition and registration. In detail, first of all, a self-supervised map-centric landmark definition strategy is adopted, making places in either indoor or outdoor scenes act as unique landmarks. Then, sparse keypoints extracted from images, are utilized as the input to a transformer-based deep neural network for landmark recognition; these keypoints enable PRAM to recognize hundreds of landmarks with high time and memory efficiency. Keypoints along with recognized landmark labels are further used for registration between query images and the 3D landmark map. Different from previous hierarchical methods, PRAM discards global and local descriptors, and reduces over 90% storage. Since PRAM utilizes recognition and landmark-wise verification to replace global reference search and exhaustive matching respectively, it runs 2.4 times faster than prior state-of-the-art approaches. Moreover, PRAM opens new directions for visual localization including multi-modality localization, map-centric feature learning, and hierarchical scene coordinate regression.

Title: Streamlined Photoacoustic Image Processing with Foundation Models: A Training-Free Solution

• Authors: Handi Deng, Yucheng Zhou, Jiaxuan Xiang, Liujie Gu, Yan Luo, Hai Feng, Mingyuan Liu, Cheng Ma
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07833
• Pdf URL: https://arxiv.org/pdf/2404.07833
• Copy Paste: [[2404.07833]] Streamlined Photoacoustic Image Processing with Foundation Models: A Training-Free Solution(https://arxiv.org/abs/2404.07833)
• Keywords: foundation model
• Abstract: Foundation models have rapidly evolved and have achieved significant accomplishments in computer vision tasks. Specifically, the prompt mechanism conveniently allows users to integrate image prior information into the model, making it possible to apply models without any training. Therefore, we propose a method based on foundation models and zero training to solve the tasks of photoacoustic (PA) image segmentation. We employed the segment anything model (SAM) by setting simple prompts and integrating the model's outputs with prior knowledge of the imaged objects to accomplish various tasks, including: (1) removing the skin signal in three-dimensional PA image rendering; (2) dual speed-of-sound reconstruction, and (3) segmentation of finger blood vessels. Through these demonstrations, we have concluded that deep learning can be directly applied in PA imaging without the requirement for network design and training. This potentially allows for a hands-on, convenient approach to achieving efficient and accurate segmentation of PA images. This letter serves as a comprehensive tutorial, facilitating the mastery of the technique through the provision of code and sample datasets.

Title: On Training Data Influence of GPT Models

• Authors: Qingyi Liu, Yekun Chai, Shuohuan Wang, Yu Sun, Keze Wang, Hua Wu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07840
• Pdf URL: https://arxiv.org/pdf/2404.07840
• Copy Paste: [[2404.07840]] On Training Data Influence of GPT Models(https://arxiv.org/abs/2404.07840)
• Keywords: generative
• Abstract: Amidst the rapid advancements in generative language models, the investigation of how training data shapes the performance of GPT models is still emerging. This paper presents GPTfluence, a novel approach that leverages a featurized simulation to assess the impact of training examples on the training dynamics of GPT models. Our approach not only traces the influence of individual training instances on performance trajectories, such as loss and other key metrics, on targeted test points but also enables a comprehensive comparison with existing methods across various training scenarios in GPT models, ranging from 14 million to 2.8 billion parameters, across a range of downstream tasks. Contrary to earlier methods that struggle with generalization to new data, GPTfluence introduces a parameterized simulation of training dynamics, demonstrating robust generalization capabilities to unseen training data. This adaptability is evident across both fine-tuning and instruction-tuning scenarios, spanning tasks in natural language understanding and generation. We will make our code and data publicly available.

Title: TBSN: Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising

• Authors: Junyi Li, Zhilu Zhang, Wangmeng Zuo
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.07846
• Pdf URL: https://arxiv.org/pdf/2404.07846
• Copy Paste: [[2404.07846]] TBSN: Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising(https://arxiv.org/abs/2404.07846)
• Keywords: self-supervised
• Abstract: Blind-spot networks (BSN) have been prevalent network architectures in self-supervised image denoising (SSID). Existing BSNs are mostly conducted with convolution layers. Although transformers offer potential solutions to the limitations of convolutions and have demonstrated success in various image restoration tasks, their attention mechanisms may violate the blind-spot requirement, thus restricting their applicability in SSID. In this paper, we present a transformer-based blind-spot network (TBSN) by analyzing and redesigning the transformer operators that meet the blind-spot requirement. Specifically, TBSN follows the architectural principles of dilated BSNs, and incorporates spatial as well as channel self-attention layers to enhance the network capability. For spatial self-attention, an elaborate mask is applied to the attention matrix to restrict its receptive field, thus mimicking the dilated convolution. For channel self-attention, we observe that it may leak the blind-spot information when the channel number is greater than spatial size in the deep layers of multi-scale architectures. To eliminate this effect, we divide the channel into several groups and perform channel attention separately. Furthermore, we introduce a knowledge distillation strategy that distills TBSN into smaller denoisers to improve computational efficiency while maintaining performance. Extensive experiments on real-world image denoising datasets show that TBSN largely extends the receptive field and exhibits favorable performance against state-of-the-art SSID methods. The code and pre-trained models will be publicly available at https://github.com/nagejacob/TBSN.

Title: Context-aware Video Anomaly Detection in Long-Term Datasets

• Authors: Zhengye Yang, Richard Radke
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07887
• Pdf URL: https://arxiv.org/pdf/2404.07887
• Copy Paste: [[2404.07887]] Context-aware Video Anomaly Detection in Long-Term Datasets(https://arxiv.org/abs/2404.07887)
• Keywords: anomaly
• Abstract: Video anomaly detection research is generally evaluated on short, isolated benchmark videos only a few minutes long. However, in real-world environments, security cameras observe the same scene for months or years at a time, and the notion of anomalous behavior critically depends on context, such as the time of day, day of week, or schedule of events. Here, we propose a context-aware video anomaly detection algorithm, Trinity, specifically targeted to these scenarios. Trinity is especially well-suited to crowded scenes in which individuals cannot be easily tracked, and anomalies are due to speed, direction, or absence of group motion. Trinity is a contrastive learning framework that aims to learn alignments between context, appearance, and motion, and uses alignment quality to classify videos as normal or anomalous. We evaluate our algorithm on both conventional benchmarks and a public webcam-based dataset we collected that spans more than three months of activity.

Title: Anomaly Detection in Power Grids via Context-Agnostic Learning

• Authors: SangWoo Park, Amritanshu Pandey
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2404.07898
• Pdf URL: https://arxiv.org/pdf/2404.07898
• Copy Paste: [[2404.07898]] Anomaly Detection in Power Grids via Context-Agnostic Learning(https://arxiv.org/abs/2404.07898)
• Keywords: anomaly
• Abstract: An important tool grid operators use to safeguard against failures, whether naturally occurring or malicious, involves detecting anomalies in the power system SCADA data. In this paper, we aim to solve a real-time anomaly detection problem. Given time-series measurement values coming from a fixed set of sensors on the grid, can we identify anomalies in the network topology or measurement data? Existing methods, primarily optimization-based, mostly use only a single snapshot of the measurement values and do not scale well with the network size. Recent data-driven ML techniques have shown promise by using a combination of current and historical data for anomaly detection but generally do not consider physical attributes like the impact of topology or load/generation changes on sensor measurements and thus cannot accommodate regular context-variability in the historical data. To address this gap, we propose a novel context-aware anomaly detection algorithm, GridCAL, that considers the effect of regular topology and load/generation changes. This algorithm converts the real-time power flow measurements to context-agnostic values, which allows us to analyze measurement coming from different grid contexts in an aggregate fashion, enabling us to derive a unified statistical model that becomes the basis of anomaly detection. Through numerical simulations on networks up to 2383 nodes, we show that our approach is accurate, outperforming state-of-the-art approaches, and is computationally efficient.

Title: AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs

• Authors: Zeyi Liao, Huan Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07921
• Pdf URL: https://arxiv.org/pdf/2404.07921
• Copy Paste: [[2404.07921]] AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs(https://arxiv.org/abs/2404.07921)
• Keywords: generative
• Abstract: As large language models (LLMs) become increasingly prevalent and integrated into autonomous systems, ensuring their safety is imperative. Despite significant strides toward safety alignment, recent work GCG~\citep{zou2023universal} proposes a discrete token optimization algorithm and selects the single suffix with the lowest loss to successfully jailbreak aligned LLMs. In this work, we first discuss the drawbacks of solely picking the suffix with the lowest loss during GCG optimization for jailbreaking and uncover the missed successful suffixes during the intermediate steps. Moreover, we utilize those successful suffixes as training data to learn a generative model, named AmpleGCG, which captures the distribution of adversarial suffixes given a harmful query and enables the rapid generation of hundreds of suffixes for any harmful queries in seconds. AmpleGCG achieves near 100\% attack success rate (ASR) on two aligned LLMs (Llama-2-7B-chat and Vicuna-7B), surpassing two strongest attack baselines. More interestingly, AmpleGCG also transfers seamlessly to attack different models, including closed-source LLMs, achieving a 99\% ASR on the latest GPT-3.5. To summarize, our work amplifies the impact of GCG by training a generative model of adversarial suffixes that is universal to any harmful queries and transferable from attacking open-source LLMs to closed-source LLMs. In addition, it can generate 200 adversarial suffixes for one harmful query in only 4 seconds, rendering it more challenging to defend.

Title: Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation

• Authors: Keonhee Han, Dominik Muhle, Felix Wimbauer, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07933
• Pdf URL: https://arxiv.org/pdf/2404.07933
• Copy Paste: [[2404.07933]] Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation(https://arxiv.org/abs/2404.07933)
• Keywords: self-supervised
• Abstract: Inferring scene geometry from images via Structure from Motion is a long-standing and fundamental problem in computer vision. While classical approaches and, more recently, depth map predictions only focus on the visible parts of a scene, the task of scene completion aims to reason about geometry even in occluded regions. With the popularity of neural radiance fields (NeRFs), implicit representations also became popular for scene completion by predicting so-called density fields. Unlike explicit approaches. e.g. voxel-based methods, density fields also allow for accurate depth prediction and novel-view synthesis via image-based rendering. In this work, we propose to fuse the scene reconstruction from multiple images and distill this knowledge into a more accurate single-view scene reconstruction. To this end, we propose Multi-View Behind the Scenes (MVBTS) to fuse density fields from multiple posed images, trained fully self-supervised only from image data. Using knowledge distillation, we use MVBTS to train a single-view scene completion network via direct supervision called KDBTS. It achieves state-of-the-art performance on occupancy prediction, especially in occluded regions.

Title: Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon

• Authors: Tianshuo Xu, Peng Mi, Ruilin Wang, Yingcong Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07946
• Pdf URL: https://arxiv.org/pdf/2404.07946
• Copy Paste: [[2404.07946]] Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon(https://arxiv.org/abs/2404.07946)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DMs) are a powerful generative framework that have attracted significant attention in recent years. However, the high computational cost of training DMs limits their practical applications. In this paper, we start with a consistency phenomenon of DMs: we observe that DMs with different initializations or even different architectures can produce very similar outputs given the same noise inputs, which is rare in other generative models. We attribute this phenomenon to two factors: (1) the learning difficulty of DMs is lower when the noise-prediction diffusion model approaches the upper bound of the timestep (the input becomes pure noise), where the structural information of the output is usually generated; and (2) the loss landscape of DMs is highly smooth, which implies that the model tends to converge to similar local minima and exhibit similar behavior patterns. This finding not only reveals the stability of DMs, but also inspires us to devise two strategies to accelerate the training of DMs. First, we propose a curriculum learning based timestep schedule, which leverages the noise rate as an explicit indicator of the learning difficulty and gradually reduces the training frequency of easier timesteps, thus improving the training efficiency. Second, we propose a momentum decay strategy, which reduces the momentum coefficient during the optimization process, as the large momentum may hinder the convergence speed and cause oscillations due to the smoothness of the loss landscape. We demonstrate the effectiveness of our proposed strategies on various models and show that they can significantly reduce the training time and improve the quality of the generated images.

Title: Taming Stable Diffusion for Text to 360° Panorama Image Generation

• Authors: Cheng Zhang, Qianyi Wu, Camilo Cruz Gambardella, Xiaoshui Huang, Dinh Phung, Wanli Ouyang, Jianfei Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07949
• Pdf URL: https://arxiv.org/pdf/2404.07949
• Copy Paste: [[2404.07949]] Taming Stable Diffusion for Text to 360° Panorama Image Generation(https://arxiv.org/abs/2404.07949)
• Keywords: diffusion, generative
• Abstract: Generative models, e.g., Stable Diffusion, have enabled the creation of photorealistic images from text prompts. Yet, the generation of 360-degree panorama images from text remains a challenge, particularly due to the dearth of paired text-panorama data and the domain gap between panorama and perspective images. In this paper, we introduce a novel dual-branch diffusion model named PanFusion to generate a 360-degree image from a text prompt. We leverage the stable diffusion model as one branch to provide prior knowledge in natural image generation and register it to another panorama branch for holistic image generation. We propose a unique cross-attention mechanism with projection awareness to minimize distortion during the collaborative denoising process. Our experiments validate that PanFusion surpasses existing methods and, thanks to its dual-branch structure, can integrate additional constraints like room layout for customized panorama outputs. Code is available at https://chengzhag.github.io/publication/panfusion.

Title: Triple Component Matrix Factorization: Untangling Global, Local, and Noisy Components

• Authors: Naichen Shi, Salar Fattahi, Raed Al Kontar
• Subjects: cs.LG, math.ST
• Abstract URL: https://arxiv.org/abs/2404.07955
• Pdf URL: https://arxiv.org/pdf/2404.07955
• Copy Paste: [[2404.07955]] Triple Component Matrix Factorization: Untangling Global, Local, and Noisy Components(https://arxiv.org/abs/2404.07955)
• Keywords: anomaly
• Abstract: In this work, we study the problem of common and unique feature extraction from noisy data. When we have N observation matrices from N different and associated sources corrupted by sparse and potentially gross noise, can we recover the common and unique components from these noisy observations? This is a challenging task as the number of parameters to estimate is approximately thrice the number of observations. Despite the difficulty, we propose an intuitive alternating minimization algorithm called triple component matrix factorization (TCMF) to recover the three components exactly. TCMF is distinguished from existing works in literature thanks to two salient features. First, TCMF is a principled method to separate the three components given noisy observations provably. Second, the bulk of the computation in TCMF can be distributed. On the technical side, we formulate the problem as a constrained nonconvex nonsmooth optimization problem. Despite the intricate nature of the problem, we provide a Taylor series characterization of its solution by solving the corresponding Karush-Kuhn-Tucker conditions. Using this characterization, we can show that the alternating minimization algorithm makes significant progress at each iteration and converges into the ground truth at a linear rate. Numerical experiments in video segmentation and anomaly detection highlight the superior feature extraction abilities of TCMF.

Title: Self-supervised Dataset Distillation: A Good Compression Is All You Need

• Authors: Muxin Zhou, Zeyuan Yin, Shitong Shao, Zhiqiang Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07976
• Pdf URL: https://arxiv.org/pdf/2404.07976
• Copy Paste: [[2404.07976]] Self-supervised Dataset Distillation: A Good Compression Is All You Need(https://arxiv.org/abs/2404.07976)
• Keywords: self-supervised
• Abstract: Dataset distillation aims to compress information from a large-scale original dataset to a new compact dataset while striving to preserve the utmost degree of the original data informational essence. Previous studies have predominantly concentrated on aligning the intermediate statistics between the original and distilled data, such as weight trajectory, features, gradient, BatchNorm, etc. In this work, we consider addressing this task through the new lens of model informativeness in the compression stage on the original dataset pretraining. We observe that with the prior state-of-the-art SRe$^2$L, as model sizes increase, it becomes increasingly challenging for supervised pretrained models to recover learned information during data synthesis, as the channel-wise mean and variance inside the model are flatting and less informative. We further notice that larger variances in BN statistics from self-supervised models enable larger loss signals to update the recovered data by gradients, enjoying more informativeness during synthesis. Building on this observation, we introduce SC-DD, a simple yet effective Self-supervised Compression framework for Dataset Distillation that facilitates diverse information compression and recovery compared to traditional supervised learning schemes, further reaps the potential of large pretrained models with enhanced capabilities. Extensive experiments are conducted on CIFAR-100, Tiny-ImageNet and ImageNet-1K datasets to demonstrate the superiority of our proposed approach. The proposed SC-DD outperforms all previous state-of-the-art supervised dataset distillation methods when employing larger models, such as SRe$^2$L, MTT, TESLA, DC, CAFE, etc., by large margins under the same recovery and post-training budgets. Code is available at https://github.com/VILA-Lab/SRe2L/tree/main/SCDD/.

Title: LLoCO: Learning Long Contexts Offline

• Authors: Sijun Tan, Xiuyu Li, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E. Gonzalez, Raluca Ada Popa
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07979
• Pdf URL: https://arxiv.org/pdf/2404.07979
• Copy Paste: [[2404.07979]] LLoCO: Learning Long Contexts Offline(https://arxiv.org/abs/2404.07979)
• Keywords: in-context
• Abstract: Processing long contexts remains a challenge for large language models (LLMs) due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation. We propose a novel approach to address this problem by learning contexts offline through context compression and in-domain parameter-efficient finetuning. Our method enables an LLM to create a concise representation of the original context and efficiently retrieve relevant information to answer questions accurately. We introduce LLoCO, a technique that combines context compression, retrieval, and parameter-efficient finetuning using LoRA. Our approach extends the effective context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We evaluate our approach on several long-context question-answering datasets, demonstrating that LLoCO significantly outperforms in-context learning while using $30\times$ fewer tokens during inference. LLoCO achieves up to $7.62\times$ speed-up and substantially reduces the cost of long document question answering, making it a promising solution for efficient long context processing. Our code is publicly available at https://github.com/jeffreysijuntan/lloco.

Title: View Selection for 3D Captioning via Diffusion Ranking

• Authors: Tiange Luo, Justin Johnson, Honglak Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07984
• Pdf URL: https://arxiv.org/pdf/2404.07984
• Copy Paste: [[2404.07984]] View Selection for 3D Captioning via Diffusion Ranking(https://arxiv.org/abs/2404.07984)
• Keywords: diffusion
• Abstract: Scalable annotation approaches are crucial for constructing extensive 3D-text datasets, facilitating a broader range of applications. However, existing methods sometimes lead to the generation of hallucinated captions, compromising caption quality. This paper explores the issue of hallucination in 3D object captioning, with a focus on Cap3D method, which renders 3D objects into 2D views for captioning using pre-trained models. We pinpoint a major challenge: certain rendered views of 3D objects are atypical, deviating from the training data of standard image captioning models and causing hallucinations. To tackle this, we present DiffuRank, a method that leverages a pre-trained text-to-3D model to assess the alignment between 3D objects and their 2D rendered views, where the view with high alignment closely represent the object's characteristics. By ranking all rendered views and feeding the top-ranked ones into GPT4-Vision, we enhance the accuracy and detail of captions, enabling the correction of 200k captions in the Cap3D dataset and extending it to 1 million captions across Objaverse and Objaverse-XL datasets. Additionally, we showcase the adaptability of DiffuRank by applying it to pre-trained text-to-image models for a Visual Question Answering task, where it outperforms the CLIP model.

Title: ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback

• Authors: Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, Chen Chen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07987
• Pdf URL: https://arxiv.org/pdf/2404.07987
• Copy Paste: [[2404.07987]] ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback(https://arxiv.org/abs/2404.07987)
• Keywords: diffusion
• Abstract: To enhance the controllability of text-to-image diffusion models, existing efforts like ControlNet incorporated image-based conditional controls. In this paper, we reveal that existing methods still face significant challenges in generating images that align with the image conditional controls. To this end, we propose ControlNet++, a novel approach that improves controllable generation by explicitly optimizing pixel-level cycle consistency between generated images and conditional controls. Specifically, for an input conditional control, we use a pre-trained discriminative reward model to extract the corresponding condition of the generated images, and then optimize the consistency loss between the input conditional control and extracted condition. A straightforward implementation would be generating images from random noises and then calculating the consistency loss, but such an approach requires storing gradients for multiple sampling timesteps, leading to considerable time and memory costs. To address this, we introduce an efficient reward strategy that deliberately disturbs the input images by adding noise, and then uses the single-step denoised images for reward fine-tuning. This avoids the extensive costs associated with image sampling, allowing for more efficient reward fine-tuning. Extensive experiments show that ControlNet++ significantly improves controllability under various conditional controls. For example, it achieves improvements over ControlNet by 7.9% mIoU, 13.4% SSIM, and 7.6% RMSE, respectively, for segmentation mask, line-art edge, and depth conditions.

Title: Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding

• Authors: Yiwen Tang, Jiaming Liu, Dong Wang, Zhigang Wang, Shanghang Zhang, Bin Zhao, Xuelong Li
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.07989
• Pdf URL: https://arxiv.org/pdf/2404.07989
• Copy Paste: [[2404.07989]] Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding(https://arxiv.org/abs/2404.07989)
• Keywords: foundation model
• Abstract: Large foundation models have recently emerged as a prominent focus of interest, attaining superior performance in widespread scenarios. Due to the scarcity of 3D data, many efforts have been made to adapt pre-trained transformers from vision to 3D domains. However, such 2D-to-3D approaches are still limited, due to the potential loss of spatial geometries and high computation cost. More importantly, their frameworks are mainly designed for 2D models, lacking a general any-to-3D paradigm. In this paper, we introduce Any2Point, a parameter-efficient method to empower any-modality large models (vision, language, audio) for 3D understanding. Given a frozen transformer from any source modality, we propose a 3D-to-any (1D or 2D) virtual projection strategy that correlates the input 3D points to the original 1D or 2D positions within the source modality. This mechanism enables us to assign each 3D token with a positional encoding paired with the pre-trained model, which avoids 3D geometry loss caused by the true projection and better motivates the transformer for 3D learning with 1D/2D positional priors. Then, within each transformer block, we insert an any-to-3D guided adapter module for parameter-efficient fine-tuning. The adapter incorporates prior spatial knowledge from the source modality to guide the local feature aggregation of 3D tokens, compelling the semantic adaption of any-modality transformers. We conduct extensive experiments to showcase the effectiveness and efficiency of our method. Code and models are released at https://github.com/Ivan-Tang-3D/Any2Point.

Title: OpenBias: Open-set Bias Detection in Text-to-Image Generative Models

• Authors: Moreno D'Incà, Elia Peruzzo, Massimiliano Mancini, Dejia Xu, Vidit Goel, Xingqian Xu, Zhangyang Wang, Humphrey Shi, Nicu Sebe
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07990
• Pdf URL: https://arxiv.org/pdf/2404.07990
• Copy Paste: [[2404.07990]] OpenBias: Open-set Bias Detection in Text-to-Image Generative Models(https://arxiv.org/abs/2404.07990)
• Keywords: diffusion, generative
• Abstract: Text-to-image generative models are becoming increasingly popular and accessible to the general public. As these models see large-scale deployments, it is necessary to deeply investigate their safety and fairness to not disseminate and perpetuate any kind of biases. However, existing works focus on detecting closed sets of biases defined a priori, limiting the studies to well-known concepts. In this paper, we tackle the challenge of open-set bias detection in text-to-image generative models presenting OpenBias, a new pipeline that identifies and quantifies the severity of biases agnostically, without access to any precompiled set. OpenBias has three stages. In the first phase, we leverage a Large Language Model (LLM) to propose biases given a set of captions. Secondly, the target generative model produces images using the same set of captions. Lastly, a Vision Question Answering model recognizes the presence and extent of the previously proposed biases. We study the behavior of Stable Diffusion 1.5, 2, and XL emphasizing new biases, never investigated before. Via quantitative experiments, we demonstrate that OpenBias agrees with current closed-set bias detection methods and human judgement.