2024-04-12

Title: Uncertainty-guided annotation enhances segmentation with the human-in-the-loop

• Authors: Nadieh Khalili, Joey Spronck, Francesco Ciompi, Jeroen van der Laak, Geert Litjens
• Subjects: cs.CV, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.07208
• Pdf URL: https://arxiv.org/pdf/2404.07208
• Copy Paste: [[2404.07208]] Uncertainty-guided annotation enhances segmentation with the human-in-the-loop(https://arxiv.org/abs/2404.07208)
• Keywords: segmentation
• Abstract: Deep learning algorithms, often critiqued for their 'black box' nature, traditionally fall short in providing the necessary transparency for trusted clinical use. This challenge is particularly evident when such models are deployed in local hospitals, encountering out-of-domain distributions due to varying imaging techniques and patient-specific pathologies. Yet, this limitation offers a unique avenue for continual learning. The Uncertainty-Guided Annotation (UGA) framework introduces a human-in-the-loop approach, enabling AI to convey its uncertainties to clinicians, effectively acting as an automated quality control mechanism. UGA eases this interaction by quantifying uncertainty at the pixel level, thereby revealing the model's limitations and opening the door for clinician-guided corrections. We evaluated UGA on the Camelyon dataset for lymph node metastasis segmentation which revealed that UGA improved the Dice coefficient (DC), from 0.66 to 0.76 by adding 5 patches, and further to 0.84 with 10 patches. To foster broader application and community contribution, we have made our code accessible at

Title: Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions

• Authors: Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, Aman CHadha
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07214
• Pdf URL: https://arxiv.org/pdf/2404.07214
• Copy Paste: [[2404.07214]] Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions(https://arxiv.org/abs/2404.07214)
• Keywords: large language model
• Abstract: The advent of Large Language Models (LLMs) has significantly reshaped the trajectory of the AI revolution. Nevertheless, these LLMs exhibit a notable limitation, as they are primarily adept at processing textual information. To address this constraint, researchers have endeavored to integrate visual capabilities with LLMs, resulting in the emergence of Vision-Language Models (VLMs). These advanced models are instrumental in tackling more intricate tasks such as image captioning and visual question answering. In our comprehensive survey paper, we delve into the key advancements within the realm of VLMs. Our classification organizes VLMs into three distinct categories: models dedicated to vision-language understanding, models that process multimodal inputs to generate unimodal (textual) outputs and models that both accept and produce multimodal inputs and outputs.This classification is based on their respective capabilities and functionalities in processing and generating various modalities of data.We meticulously dissect each model, offering an extensive analysis of its foundational architecture, training data sources, as well as its strengths and limitations wherever possible, providing readers with a comprehensive understanding of its essential components. We also analyzed the performance of VLMs in various benchmark datasets. By doing so, we aim to offer a nuanced understanding of the diverse landscape of VLMs. Additionally, we underscore potential avenues for future research in this dynamic domain, anticipating further breakthroughs and advancements.

Title: Goal-guided Generative Prompt Injection Attack on Large Language Models

• Authors: Chong Zhang, Mingyu Jin, Qinkai Yu, Chengzhi Liu, Haochen Xue, Xiaobo Jin
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07234
• Pdf URL: https://arxiv.org/pdf/2404.07234
• Copy Paste: [[2404.07234]] Goal-guided Generative Prompt Injection Attack on Large Language Models(https://arxiv.org/abs/2404.07234)
• Keywords: security, attack, robust, generative, large language model
• Abstract: Current large language models (LLMs) provide a strong foundation for large-scale user-oriented natural language tasks. A large number of users can easily inject adversarial text or instructions through the user interface, thus causing LLMs model security challenges. Although there is currently a large amount of research on prompt injection attacks, most of these black-box attacks use heuristic strategies. It is unclear how these heuristic strategies relate to the success rate of attacks and thus effectively improve model robustness. To solve this problem, we redefine the goal of the attack: to maximize the KL divergence between the conditional probabilities of the clean text and the adversarial text. Furthermore, we prove that maximizing the KL divergence is equivalent to maximizing the Mahalanobis distance between the embedded representation $x$ and $x'$ of the clean text and the adversarial text when the conditional probability is a Gaussian distribution and gives a quantitative relationship on $x$ and $x'$. Then we designed a simple and effective goal-guided generative prompt injection strategy (G2PIA) to find an injection text that satisfies specific constraints to achieve the optimal attack effect approximately. It is particularly noteworthy that our attack method is a query-free black-box attack method with low computational cost. Experimental results on seven LLM models and four datasets show the effectiveness of our attack method.

Title: Lightweight Deep Learning for Resource-Constrained Environments: A Survey

• Authors: Hou-I Liu, Marco Galindo, Hongxia Xie, Lai-Kuan Wong, Hong-Han Shuai, Yung-Yui Li, Wen-Huang Cheng
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07236
• Pdf URL: https://arxiv.org/pdf/2404.07236
• Copy Paste: [[2404.07236]] Lightweight Deep Learning for Resource-Constrained Environments: A Survey(https://arxiv.org/abs/2404.07236)
• Keywords: large language model
• Abstract: Over the past decade, the dominance of deep learning has prevailed across various domains of artificial intelligence, including natural language processing, computer vision, and biomedical signal processing. While there have been remarkable improvements in model accuracy, deploying these models on lightweight devices, such as mobile phones and microcontrollers, is constrained by limited resources. In this survey, we provide comprehensive design guidance tailored for these devices, detailing the meticulous design of lightweight models, compression methods, and hardware acceleration strategies. The principal goal of this work is to explore methods and concepts for getting around hardware constraints without compromising the model's accuracy. Additionally, we explore two notable paths for lightweight deep learning in the future: deployment techniques for TinyML and Large Language Models. Although these paths undoubtedly have potential, they also present significant challenges, encouraging research into unexplored areas.

Title: Sandwich attack: Multi-language Mixture Adaptive Attack on LLMs

• Authors: Bibek Upadhayay, Vahid Behzadan
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07242
• Pdf URL: https://arxiv.org/pdf/2404.07242
• Copy Paste: [[2404.07242]] Sandwich attack: Multi-language Mixture Adaptive Attack on LLMs(https://arxiv.org/abs/2404.07242)
• Keywords: secure, privacy, attack, robust, large language model
• Abstract: Large Language Models (LLMs) are increasingly being developed and applied, but their widespread use faces challenges. These include aligning LLMs' responses with human values to prevent harmful outputs, which is addressed through safety training methods. Even so, bad actors and malicious users have succeeded in attempts to manipulate the LLMs to generate misaligned responses for harmful questions such as methods to create a bomb in school labs, recipes for harmful drugs, and ways to evade privacy rights. Another challenge is the multilingual capabilities of LLMs, which enable the model to understand and respond in multiple languages. Consequently, attackers exploit the unbalanced pre-training datasets of LLMs in different languages and the comparatively lower model performance in low-resource languages than high-resource ones. As a result, attackers use a low-resource languages to intentionally manipulate the model to create harmful responses. Many of the similar attack vectors have been patched by model providers, making the LLMs more robust against language-based manipulation. In this paper, we introduce a new black-box attack vector called the \emph{Sandwich attack}: a multi-language mixture attack, which manipulates state-of-the-art LLMs into generating harmful and misaligned responses. Our experiments with five different models, namely Google's Bard, Gemini Pro, LLaMA-2-70-B-Chat, GPT-3.5-Turbo, GPT-4, and Claude-3-OPUS, show that this attack vector can be used by adversaries to generate harmful responses and elicit misaligned responses from these models. By detailing both the mechanism and impact of the Sandwich attack, this paper aims to guide future research and development towards more secure and resilient LLMs, ensuring they serve the public good while minimizing potential for misuse.

Title: Generative Resident Separation and Multi-label Classification for Multi-person Activity Recognition

• Authors: Xi Chen (LIG), Julien Cumin, Fano Ramparany, Dominique Vaufreydaz (LIG)
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.07245
• Pdf URL: https://arxiv.org/pdf/2404.07245
• Copy Paste: [[2404.07245]] Generative Resident Separation and Multi-label Classification for Multi-person Activity Recognition(https://arxiv.org/abs/2404.07245)
• Keywords: generative
• Abstract: This paper presents two models to address the problem of multi-person activity recognition using ambient sensors in a home. The first model, Seq2Res, uses a sequence generation approach to separate sensor events from different residents. The second model, BiGRU+Q2L, uses a Query2Label multi-label classifier to predict multiple activities simultaneously. Performances of these models are compared to a state-of-the-art model in different experimental scenarios, using a state-of-the-art dataset of two residents in a home instrumented with ambient sensors. These results lead to a discussion on the advantages and drawbacks of resident separation and multi-label classification for multi-person activity recognition.

Title: Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers

• Authors: Jinyang Liu, Wondmgezahu Teshome, Sandesh Ghimire, Mario Sznaier, Octavia Camps
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07292
• Pdf URL: https://arxiv.org/pdf/2404.07292
• Copy Paste: [[2404.07292]] Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers(https://arxiv.org/abs/2404.07292)
• Keywords: diffusion, transformer
• Abstract: Solving image and video jigsaw puzzles poses the challenging task of rearranging image fragments or video frames from unordered sequences to restore meaningful images and video sequences. Existing approaches often hinge on discriminative models tasked with predicting either the absolute positions of puzzle elements or the permutation actions applied to the original data. Unfortunately, these methods face limitations in effectively solving puzzles with a large number of elements. In this paper, we propose JPDVT, an innovative approach that harnesses diffusion transformers to address this challenge. Specifically, we generate positional information for image patches or video frames, conditioned on their underlying visual content. This information is then employed to accurately assemble the puzzle pieces in their correct positions, even in scenarios involving missing pieces. Our method achieves state-of-the-art performance on several datasets.

Title: We're Calling an Intervention: Taking a Closer Look at Language Model Adaptation to Different Types of Linguistic Variation

• Authors: Aarohi Srivastava, David Chiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07304
• Pdf URL: https://arxiv.org/pdf/2404.07304
• Copy Paste: [[2404.07304]] We're Calling an Intervention: Taking a Closer Look at Language Model Adaptation to Different Types of Linguistic Variation(https://arxiv.org/abs/2404.07304)
• Keywords: robust
• Abstract: We present a suite of interventions and experiments that allow us to understand language model adaptation to text with linguistic variation (e.g., nonstandard or dialectal text). Our interventions address several features of linguistic variation, resulting in character, subword, and word-level changes. Applying our interventions during language model adaptation with varying size and nature of training data, we gain important insights into what makes linguistic variation particularly difficult for language models to deal with. For instance, on text with character-level variation, performance improves with even a few training examples but approaches a plateau, suggesting that more data is not the solution. In contrast, on text with variation involving new words or meanings, far more data is needed, but it leads to a massive breakthrough in performance. Our findings inform future work on dialectal NLP and making language models more robust to linguistic variation overall. We make the code for our interventions, which can be applied to any English text data, publicly available.

Title: AI-Guided Defect Detection Techniques to Model Single Crystal Diamond Growth

• Authors: Rohan Reddy Mekala, Elias Garratt, Matthias Muehle, Arjun Srinivasan, Adam Porter, Mikael Lindvall
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07306
• Pdf URL: https://arxiv.org/pdf/2404.07306
• Copy Paste: [[2404.07306]] AI-Guided Defect Detection Techniques to Model Single Crystal Diamond Growth(https://arxiv.org/abs/2404.07306)
• Keywords: segmentation
• Abstract: From a process development perspective, diamond growth via chemical vapor deposition has made significant strides. However, challenges persist in achieving high quality and large-area material production. These difficulties include controlling conditions to maintain uniform growth rates for the entire growth surface. As growth progresses, various factors or defect states emerge, altering the uniform conditions. These changes affect the growth rate and result in the formation of crystalline defects at the microscale. However, there is a distinct lack of methods to identify these defect states and their geometry using images taken during the growth process. This paper details seminal work on defect segmentation pipeline using in-situ optical images to identify features that indicate defective states that are visible at the macroscale. Using a semantic segmentation approach as applied in our previous work, these defect states and corresponding derivative features are isolated and classified by their pixel masks. Using an annotation focused human-in-the-loop software architecture to produce training datasets, with modules for selective data labeling using active learning, data augmentations, and model-assisted labeling, our approach achieves effective annotation accuracy and drastically reduces the time and cost of labeling by orders of magnitude. On the model development front, we found that deep learning-based algorithms are the most efficient. They can accurately learn complex representations from feature-rich datasets. Our best-performing model, based on the YOLOV3 and DeeplabV3plus architectures, achieved excellent accuracy for specific features of interest. Specifically, it reached 93.35% accuracy for center defects, 92.83% for polycrystalline defects, and 91.98% for edge defects.

Title: PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scores

• Authors: Lucas Goncalves, Prashant Mathur, Chandrashekhar Lavania, Metehan Cekic, Marcello Federico, Kyu J. Han
• Subjects: cs.CV, cs.MM, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.07336
• Pdf URL: https://arxiv.org/pdf/2404.07336
• Copy Paste: [[2404.07336]] PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scores(https://arxiv.org/abs/2404.07336)
• Keywords: generative
• Abstract: Recent advancements in audio-visual generative modeling have been propelled by progress in deep learning and the availability of data-rich benchmarks. However, the growth is not attributed solely to models and benchmarks. Universally accepted evaluation metrics also play an important role in advancing the field. While there are many metrics available to evaluate audio and visual content separately, there is a lack of metrics that offer a quantitative and interpretable measure of audio-visual synchronization for videos "in the wild". To address this gap, we first created a large scale human annotated dataset (100+ hrs) representing nine types of synchronization errors in audio-visual content and how human perceive them. We then developed a PEAVS (Perceptual Evaluation of Audio-Visual Synchrony) score, a novel automatic metric with a 5-point scale that evaluates the quality of audio-visual synchronization. We validate PEAVS using a newly generated dataset, achieving a Pearson correlation of 0.79 at the set level and 0.54 at the clip level when compared to human labels. In our experiments, we observe a relative gain 50% over a natural extension of Fr\'echet based metrics for Audio-Visual synchrony, confirming PEAVS efficacy in objectively modeling subjective perceptions of audio-visual synchronization for videos "in the wild".

Title: Indoor Location Fingerprinting Privacy: A Comprehensive Survey

• Authors: Amir Fathalizadeh, Vahideh Moghtadaiee, Mina Alishahi
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.07345
• Pdf URL: https://arxiv.org/pdf/2404.07345
• Copy Paste: [[2404.07345]] Indoor Location Fingerprinting Privacy: A Comprehensive Survey(https://arxiv.org/abs/2404.07345)
• Keywords: privacy, attack, federate
• Abstract: The pervasive integration of Indoor Positioning Systems (IPS) arises from the limitations of Global Navigation Satellite Systems (GNSS) in indoor environments, leading to the widespread adoption of Location-Based Services (LBS). Specifically, indoor location fingerprinting employs diverse signal fingerprints from user devices, enabling precise location identification by Location Service Providers (LSP). Despite its broad applications across various domains, indoor location fingerprinting introduces a notable privacy risk, as both LSP and potential adversaries inherently have access to this sensitive information, compromising users' privacy. Consequently, concerns regarding privacy vulnerabilities in this context necessitate a focused exploration of privacy-preserving mechanisms. In response to these concerns, this survey presents a comprehensive review of Privacy-Preserving Mechanisms in Indoor Location Fingerprinting (ILFPPM) based on cryptographic, anonymization, differential privacy (DP), and federated learning (FL) techniques. We also propose a distinctive and novel grouping of privacy vulnerabilities, adversary and attack models, and available evaluation metrics specific to indoor location fingerprinting systems. Given the identified limitations and research gaps in this survey, we highlight numerous prospective opportunities for future investigation, aiming to motivate researchers interested in advancing this field. This survey serves as a valuable reference for researchers and provides a clear overview for those beyond this specific research domain.

Title: A Transformer-Based Model for the Prediction of Human Gaze Behavior on Videos

• Authors: Suleyman Ozdel, Yao Rong, Berat Mert Albaba, Yen-Ling Kuo, Xi Wang
• Subjects: cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07351
• Pdf URL: https://arxiv.org/pdf/2404.07351
• Copy Paste: [[2404.07351]] A Transformer-Based Model for the Prediction of Human Gaze Behavior on Videos(https://arxiv.org/abs/2404.07351)
• Keywords: transformer
• Abstract: Eye-tracking applications that utilize the human gaze in video understanding tasks have become increasingly important. To effectively automate the process of video analysis based on eye-tracking data, it is important to accurately replicate human gaze behavior. However, this task presents significant challenges due to the inherent complexity and ambiguity of human gaze patterns. In this work, we introduce a novel method for simulating human gaze behavior. Our approach uses a transformer-based reinforcement learning algorithm to train an agent that acts as a human observer, with the primary role of watching videos and simulating human gaze behavior. We employed an eye-tracking dataset gathered from videos generated by the VirtualHome simulator, with a primary focus on activity recognition. Our experimental results demonstrate the effectiveness of our gaze prediction method by highlighting its capability to replicate human gaze behavior and its applicability for downstream tasks where real human-gaze is used as input.

Title: GANsemble for Small and Imbalanced Data Sets: A Baseline for Synthetic Microplastics Data

• Authors: Daniel Platnick, Sourena Khanzadeh, Alireza Sadeghian, Richard Anthony Valenzano
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07356
• Pdf URL: https://arxiv.org/pdf/2404.07356
• Copy Paste: [[2404.07356]] GANsemble for Small and Imbalanced Data Sets: A Baseline for Synthetic Microplastics Data(https://arxiv.org/abs/2404.07356)
• Keywords: generative
• Abstract: Microplastic particle ingestion or inhalation by humans is a problem of growing concern. Unfortunately, current research methods that use machine learning to understand their potential harms are obstructed by a lack of available data. Deep learning techniques in particular are challenged by such domains where only small or imbalanced data sets are available. Overcoming this challenge often involves oversampling underrepresented classes or augmenting the existing data to improve model performance. This paper proposes GANsemble: a two-module framework connecting data augmentation with conditional generative adversarial networks (cGANs) to generate class-conditioned synthetic data. First, the data chooser module automates augmentation strategy selection by searching for the best data augmentation strategy. Next, the cGAN module uses this strategy to train a cGAN for generating enhanced synthetic data. We experiment with the GANsemble framework on a small and imbalanced microplastics data set. A Microplastic-cGAN (MPcGAN) algorithm is introduced, and baselines for synthetic microplastics (SYMP) data are established in terms of Frechet Inception Distance (FID) and Inception Scores (IS). We also provide a synthetic microplastics filter (SYMP-Filter) algorithm to increase the quality of generated SYMP. Additionally, we show the best amount of oversampling with augmentation to fix class imbalance in small microplastics data sets. To our knowledge, this study is the first application of generative AI to synthetically create microplastics data.

Title: Gradient Networks

• Authors: Shreyas Chaudhari, Srinivasa Pranav, José M. F. Moura
• Subjects: cs.LG, cs.NE, eess.SP, math.OC
• Abstract URL: https://arxiv.org/abs/2404.07361
• Pdf URL: https://arxiv.org/pdf/2404.07361
• Copy Paste: [[2404.07361]] Gradient Networks(https://arxiv.org/abs/2404.07361)
• Keywords: generative
• Abstract: Directly parameterizing and learning gradients of functions has widespread significance, with specific applications in optimization, generative modeling, and optimal transport. This paper introduces gradient networks (GradNets): novel neural network architectures that parameterize gradients of various function classes. GradNets exhibit specialized architectural constraints that ensure correspondence to gradient functions. We provide a comprehensive GradNet design framework that includes methods for transforming GradNets into monotone gradient networks (mGradNets), which are guaranteed to represent gradients of convex functions. We establish the approximation capabilities of the proposed GradNet and mGradNet. Our results demonstrate that these networks universally approximate the gradients of (convex) functions. Furthermore, these networks can be customized to correspond to specific spaces of (monotone) gradient functions, including gradients of transformed sums of (convex) ridge functions. Our analysis leads to two distinct GradNet architectures, GradNet-C and GradNet-M, and we describe the corresponding monotone versions, mGradNet-C and mGradNet-M. Our empirical results show that these architectures offer efficient parameterizations and outperform popular methods in gradient field learning tasks.

Title: Differentially Private GANs for Generating Synthetic Indoor Location Data

• Authors: Vahideh Moghtadaiee, Mina Alishahi, Milad Rabiei
• Subjects: cs.CR, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.07366
• Pdf URL: https://arxiv.org/pdf/2404.07366
• Copy Paste: [[2404.07366]] Differentially Private GANs for Generating Synthetic Indoor Location Data(https://arxiv.org/abs/2404.07366)
• Keywords: security, privacy, protect, generative
• Abstract: The advent of location-based services has led to the widespread adoption of indoor localization systems, which enable location tracking of individuals within enclosed spaces such as buildings. While these systems provide numerous benefits such as improved security and personalized services, they also raise concerns regarding privacy violations. As such, there is a growing need for privacy-preserving solutions that can protect users' sensitive location information while still enabling the functionality of indoor localization systems. In recent years, Differentially Private Generative Adversarial Networks (DPGANs) have emerged as a powerful methodology that aims to protect the privacy of individual data points while generating realistic synthetic data similar to original data. DPGANs combine the power of generative adversarial networks (GANs) with the privacy-preserving technique of differential privacy (DP). In this paper, we introduce an indoor localization framework employing DPGANs in order to generate privacy-preserving indoor location data. We evaluate the performance of our framework on a real-world indoor localization dataset and demonstrate its effectiveness in preserving privacy while maintaining the accuracy of the localization system.

Title: LLMs in Biomedicine: A study on clinical Named Entity Recognition

• Authors: Masoud Monajatipoor, Jiaxin Yang, Joel Stremmel, Melika Emami, Fazlolah Mohaghegh, Mozhdeh Rouhsedaghat, Kai-Wei Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07376
• Pdf URL: https://arxiv.org/pdf/2404.07376
• Copy Paste: [[2404.07376]] LLMs in Biomedicine: A study on clinical Named Entity Recognition(https://arxiv.org/abs/2404.07376)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) demonstrate remarkable versatility in various NLP tasks but encounter distinct challenges in biomedicine due to medical language complexities and data scarcity. This paper investigates the application of LLMs in the medical domain by exploring strategies to enhance their performance for the Named-Entity Recognition (NER) task. Specifically, our study reveals the importance of meticulously designed prompts in biomedicine. Strategic selection of in-context examples yields a notable improvement, showcasing ~15-20\% increase in F1 score across all benchmark datasets for few-shot clinical NER. Additionally, our findings suggest that integrating external resources through prompting strategies can bridge the gap between general-purpose LLM proficiency and the specialized demands of medical NER. Leveraging a medical knowledge base, our proposed method inspired by Retrieval-Augmented Generation (RAG) can boost the F1 score of LLMs for zero-shot clinical NER. We will release the code upon publication.

Title: Deep Generative Sampling in the Dual Divergence Space: A Data-efficient & Interpretative Approach for Generative AI

• Authors: Sahil Garg, Anderson Schneider, Anant Raj, Kashif Rasul, Yuriy Nevmyvaka, Sneihil Gopal, Amit Dhurandhar, Guillermo Cecchi, Irina Rish
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV, cs.IT
• Abstract URL: https://arxiv.org/abs/2404.07377
• Pdf URL: https://arxiv.org/pdf/2404.07377
• Copy Paste: [[2404.07377]] Deep Generative Sampling in the Dual Divergence Space: A Data-efficient & Interpretative Approach for Generative AI(https://arxiv.org/abs/2404.07377)
• Keywords: generative
• Abstract: Building on the remarkable achievements in generative sampling of natural images, we propose an innovative challenge, potentially overly ambitious, which involves generating samples of entire multivariate time series that resemble images. However, the statistical challenge lies in the small sample size, sometimes consisting of a few hundred subjects. This issue is especially problematic for deep generative models that follow the conventional approach of generating samples from a canonical distribution and then decoding or denoising them to match the true data distribution. In contrast, our method is grounded in information theory and aims to implicitly characterize the distribution of images, particularly the (global and local) dependency structure between pixels. We achieve this by empirically estimating its KL-divergence in the dual form with respect to the respective marginal distribution. This enables us to perform generative sampling directly in the optimized 1-D dual divergence space. Specifically, in the dual space, training samples representing the data distribution are embedded in the form of various clusters between two end points. In theory, any sample embedded between those two end points is in-distribution w.r.t. the data distribution. Our key idea for generating novel samples of images is to interpolate between the clusters via a walk as per gradients of the dual function w.r.t. the data dimensions. In addition to the data efficiency gained from direct sampling, we propose an algorithm that offers a significant reduction in sample complexity for estimating the divergence of the data distribution with respect to the marginal distribution. We provide strong theoretical guarantees along with an extensive empirical evaluation using many real-world datasets from diverse domains, establishing the superiority of our approach w.r.t. state-of-the-art deep learning methods.

Title: Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Models

• Authors: Yasi Zhang, Peiyu Yu, Ying Nian Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07389
• Pdf URL: https://arxiv.org/pdf/2404.07389
• Copy Paste: [[2404.07389]] Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Models(https://arxiv.org/abs/2404.07389)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models have shown great success in generating high-quality text-guided images. Yet, these models may still fail to semantically align generated images with the provided text prompts, leading to problems like incorrect attribute binding and/or catastrophic object neglect. Given the pervasive object-oriented structure underlying text prompts, we introduce a novel object-conditioned Energy-Based Attention Map Alignment (EBAMA) method to address the aforementioned problems. We show that an object-centric attribute binding loss naturally emerges by approximately maximizing the log-likelihood of a $z$-parameterized energy-based model with the help of the negative sampling technique. We further propose an object-centric intensity regularizer to prevent excessive shifts of objects attention towards their attributes. Extensive qualitative and quantitative experiments, including human evaluation, on several challenging benchmarks demonstrate the superior performance of our method over previous strong counterparts. With better aligned attention maps, our approach shows great promise in further enhancing the text-controlled image editing ability of diffusion models.

Title: Post-hurricane building damage assessment using street-view imagery and structured data: A multi-modal deep learning approach

• Authors: Zhuoqun Xue, Xiaojian Zhang, David O. Prevatt, Jennifer Bridge, Susu Xu, Xilei Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07399
• Pdf URL: https://arxiv.org/pdf/2404.07399
• Copy Paste: [[2404.07399]] Post-hurricane building damage assessment using street-view imagery and structured data: A multi-modal deep learning approach(https://arxiv.org/abs/2404.07399)
• Keywords: transformer
• Abstract: Accurately assessing building damage is critical for disaster response and recovery. However, many existing models for detecting building damage have poor prediction accuracy due to their limited capabilities of identifying detailed, comprehensive structural and/or non-structural damage from the street-view image. Additionally, these models mainly rely on the imagery data for damage classification, failing to account for other critical information, such as wind speed, building characteristics, evacuation zones, and distance of the building to the hurricane track. To address these limitations, in this study, we propose a novel multi-modal (i.e., imagery and structured data) approach for post-hurricane building damage classification, named the Multi-Modal Swin Transformer (MMST). We empirically train and evaluate the proposed MMST using data collected from the 2022 Hurricane Ian in Florida, USA. Results show that MMST outperforms all selected state-of-the-art benchmark models and can achieve an accuracy of 92.67%, which are 7.71% improvement in accuracy compared to Visual Geometry Group 16 (VGG-16). In addition to the street-view imagery data, building value, building age, and wind speed are the most important predictors for damage level classification. The proposed MMST can be deployed to assist in rapid damage assessment and guide reconnaissance efforts in future hurricanes.

Title: Simplifying Two-Stage Detectors for On-Device Inference in Remote Sensing

• Authors: Jaemin Kang, Hoeseok Yang, Hyungshin Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07405
• Pdf URL: https://arxiv.org/pdf/2404.07405
• Copy Paste: [[2404.07405]] Simplifying Two-Stage Detectors for On-Device Inference in Remote Sensing(https://arxiv.org/abs/2404.07405)
• Keywords: extraction
• Abstract: Deep learning has been successfully applied to object detection from remotely sensed images. Images are typically processed on the ground rather than on-board due to the computation power of the ground system. Such offloaded processing causes delays in acquiring target mission information, which hinders its application to real-time use cases. For on-device object detection, researches have been conducted on designing efficient detectors or model compression to reduce inference latency. However, highly accurate two-stage detectors still need further exploitation for acceleration. In this paper, we propose a model simplification method for two-stage object detectors. Instead of constructing a general feature pyramid, we utilize only one feature extraction in the two-stage detector. To compensate for the accuracy drop, we apply a high pass filter to the RPN's score map. Our approach is applicable to any two-stage detector using a feature pyramid network. In the experiments with state-of-the-art two-stage detectors such as ReDet, Oriented-RCNN, and LSKNet, our method reduced computation costs upto 61.2% with the accuracy loss within 2.1% on the DOTAv1.5 dataset. Source code will be released.

Title: Improving Shift Invariance in Convolutional Neural Networks with Translation Invariant Polyphase Sampling

• Authors: Sourajit Saha, Tejas Gokhale
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07410
• Pdf URL: https://arxiv.org/pdf/2404.07410
• Copy Paste: [[2404.07410]] Improving Shift Invariance in Convolutional Neural Networks with Translation Invariant Polyphase Sampling(https://arxiv.org/abs/2404.07410)
• Keywords: robust, segmentation
• Abstract: Downsampling operators break the shift invariance of convolutional neural networks (CNNs) and this affects the robustness of features learned by CNNs when dealing with even small pixel-level shift. Through a large-scale correlation analysis framework, we study shift invariance of CNNs by inspecting existing downsampling operators in terms of their maximum-sampling bias (MSB), and find that MSB is negatively correlated with shift invariance. Based on this crucial insight, we propose a learnable pooling operator called Translation Invariant Polyphase Sampling (TIPS) and two regularizations on the intermediate feature maps of TIPS to reduce MSB and learn translation-invariant representations. TIPS can be integrated into any CNN and can be trained end-to-end with marginal computational overhead. Our experiments demonstrate that TIPS results in consistent performance gains in terms of accuracy, shift consistency, and shift fidelity on multiple benchmarks for image classification and semantic segmentation compared to previous methods and also leads to improvements in adversarial and distributional robustness. TIPS results in the lowest MSB compared to all previous methods, thus explaining our strong empirical results.

Title: JetMoE: Reaching Llama2 Performance with 0.1M Dollars

• Authors: Yikang Shen, Zhen Guo, Tianle Cai, Zengyi Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07413
• Pdf URL: https://arxiv.org/pdf/2404.07413
• Copy Paste: [[2404.07413]] JetMoE: Reaching Llama2 Performance with 0.1M Dollars(https://arxiv.org/abs/2404.07413)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the Llama2-13B-Chat model. These results suggest that LLM training can be much more cost-effective than generally thought. JetMoE-8B is based on an efficient Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B to have 8B parameters while only activating 2B for each input token, reducing inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B is highly open and academia-friendly, using only public datasets and training code. All training parameters and data mixtures have been detailed in this report to facilitate future efforts in the development of open foundation models. This transparency aims to encourage collaboration and further advancements in the field of accessible and efficient LLMs. The model weights are publicly available at https://github.com/myshell-ai/JetMoE.

Title: CopilotCAD: Empowering Radiologists with Report Completion Models and Quantitative Evidence from Medical Image Foundation Models

• Authors: Sheng Wang, Tianming Du, Katherine Fischer, Gregory E Tasian, Justin Ziemba, Joanie M Garratt, Hersh Sagreiya, Yong Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07424
• Pdf URL: https://arxiv.org/pdf/2404.07424
• Copy Paste: [[2404.07424]] CopilotCAD: Empowering Radiologists with Report Completion Models and Quantitative Evidence from Medical Image Foundation Models(https://arxiv.org/abs/2404.07424)
• Keywords: large language model
• Abstract: Computer-aided diagnosis systems hold great promise to aid radiologists and clinicians in radiological clinical practice and enhance diagnostic accuracy and efficiency. However, the conventional systems primarily focus on delivering diagnostic results through text report generation or medical image classification, positioning them as standalone decision-makers rather than helpers and ignoring radiologists' expertise. This study introduces an innovative paradigm to create an assistive co-pilot system for empowering radiologists by leveraging Large Language Models (LLMs) and medical image analysis tools. Specifically, we develop a collaborative framework to integrate LLMs and quantitative medical image analysis results generated by foundation models with radiologists in the loop, achieving efficient and safe generation of radiology reports and effective utilization of computational power of AI and the expertise of medical professionals. This approach empowers radiologists to generate more precise and detailed diagnostic reports, enhancing patient outcomes while reducing the burnout of clinicians. Our methodology underscores the potential of AI as a supportive tool in medical diagnostics, promoting a harmonious integration of technology and human expertise to advance the field of radiology.

Title: RTL Interconnect Obfuscation By Polymorphic Switch Boxes For Secure Hardware Generation

• Authors: Haimanti Chakraborty, Ranga Vemuri
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07426
• Pdf URL: https://arxiv.org/pdf/2404.07426
• Copy Paste: [[2404.07426]] RTL Interconnect Obfuscation By Polymorphic Switch Boxes For Secure Hardware Generation(https://arxiv.org/abs/2404.07426)
• Keywords: secure, security, protect, attack, robust
• Abstract: Logic Obfuscation is a well renowned design-for-trust solution to protect an Integrated Circuit (IC) from unauthorized use and illegal overproduction by including key-gates to lock the design. This is particularly necessary for ICs manufactured at untrusted third-party foundries getting exposed to security threats. In the past, several logic obfuscation methodologies have been proposed that are vulnerable to attacks such as the Boolean Satisfiability Attack. Many of these techniques are implemented at the gate level that may involve expensive re-synthesis cycles. In this paper, we present an interconnect obfuscation scheme at the Register-Transfer Level (RTL) using Switch Boxes (SBs) constructed of Polymorphic Transistors. A polymorphic SB can be designed using the same transistor count as its Complementary-Metal-Oxide-Semiconductor based counterpart, thereby no increased area in comparison, but serving as an advantage in having more key-bit combinations for an attacker to correctly identify and unlock each polymorphic SB. Security-aware high-level synthesis algorithms have also been presented to increase RTL interconnects to Functional Units impacting multiple outputs such that when a polymorphic SB is strategically inserted, those outputs would be corrupted upon incorrect key-bit identification. Finally, we run the SMT (Satisfiability Modulo Theories)-based RTL Logic Attack on the obfuscated design to examine its robustness.

Title: Data-Driven Portfolio Management for Motion Pictures Industry: A New Data-Driven Optimization Methodology Using a Large Language Model as the Expert

• Authors: Mohammad Alipour-Vaezi, Kwok-Leung Tsui
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2404.07434
• Pdf URL: https://arxiv.org/pdf/2404.07434
• Copy Paste: [[2404.07434]] Data-Driven Portfolio Management for Motion Pictures Industry: A New Data-Driven Optimization Methodology Using a Large Language Model as the Expert(https://arxiv.org/abs/2404.07434)
• Keywords: large language model
• Abstract: Portfolio management is one of the unresponded problems of the Motion Pictures Industry (MPI). To design an optimal portfolio for an MPI distributor, it is essential to predict the box office of each project. Moreover, for an accurate box office prediction, it is critical to consider the effect of the celebrities involved in each MPI project, which was impossible with any precedent expert-based method. Additionally, the asymmetric characteristic of MPI data decreases the performance of any predictive algorithm. In this paper, firstly, the fame score of the celebrities is determined using a large language model. Then, to tackle the asymmetric character of MPI's data, projects are classified. Furthermore, the box office prediction takes place for each class of projects. Finally, using a hybrid multi-attribute decision-making technique, the preferability of each project for the distributor is calculated, and benefiting from a bi-objective optimization model, the optimal portfolio is designed.

Title: Privacy preserving layer partitioning for Deep Neural Network models

• Authors: Kishore Rajasekar, Randolph Loh, Kar Wai Fok, Vrizlynn L. L. Thing
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07437
• Pdf URL: https://arxiv.org/pdf/2404.07437
• Copy Paste: [[2404.07437]] Privacy preserving layer partitioning for Deep Neural Network models(https://arxiv.org/abs/2404.07437)
• Keywords: secure, security, privacy, protect, attack, generative
• Abstract: MLaaS (Machine Learning as a Service) has become popular in the cloud computing domain, allowing users to leverage cloud resources for running private inference of ML models on their data. However, ensuring user input privacy and secure inference execution is essential. One of the approaches to protect data privacy and integrity is to use Trusted Execution Environments (TEEs) by enabling execution of programs in secure hardware enclave. Using TEEs can introduce significant performance overhead due to the additional layers of encryption, decryption, security and integrity checks. This can lead to slower inference times compared to running on unprotected hardware. In our work, we enhance the runtime performance of ML models by introducing layer partitioning technique and offloading computations to GPU. The technique comprises two distinct partitions: one executed within the TEE, and the other carried out using a GPU accelerator. Layer partitioning exposes intermediate feature maps in the clear which can lead to reconstruction attacks to recover the input. We conduct experiments to demonstrate the effectiveness of our approach in protecting against input reconstruction attacks developed using trained conditional Generative Adversarial Network(c-GAN). The evaluation is performed on widely used models such as VGG-16, ResNet-50, and EfficientNetB0, using two datasets: ImageNet for Image classification and TON IoT dataset for cybersecurity attack detection.

Title: Multi-view Aggregation Network for Dichotomous Image Segmentation

• Authors: Qian Yu, Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07445
• Pdf URL: https://arxiv.org/pdf/2404.07445
• Copy Paste: [[2404.07445]] Multi-view Aggregation Network for Dichotomous Image Segmentation(https://arxiv.org/abs/2404.07445)
• Keywords: segmentation
• Abstract: Dichotomous Image Segmentation (DIS) has recently emerged towards high-precision object segmentation from high-resolution natural images. When designing an effective DIS model, the main challenge is how to balance the semantic dispersion of high-resolution targets in the small receptive field and the loss of high-precision details in the large receptive field. Existing methods rely on tedious multiple encoder-decoder streams and stages to gradually complete the global localization and local refinement. Human visual system captures regions of interest by observing them from multiple views. Inspired by it, we model DIS as a multi-view object perception problem and provide a parsimonious multi-view aggregation network (MVANet), which unifies the feature fusion of the distant view and close-up view into a single stream with one encoder-decoder structure. With the help of the proposed multi-view complementary localization and refinement modules, our approach established long-range, profound visual interactions across multiple views, allowing the features of the detailed close-up view to focus on highly slender structures.Experiments on the popular DIS-5K dataset show that our MVANet significantly outperforms state-of-the-art methods in both accuracy and speed. The source code and datasets will be publicly available at \href{https://github.com/qianyu-dlut/MVANet}{MVANet}.

Title: Transferable and Principled Efficiency for Open-Vocabulary Segmentation

• Authors: Jingxuan Xu, Wuyang Chen, Yao Zhao, Yunchao Wei
• Subjects: cs.CV, cs.CL, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.07448
• Pdf URL: https://arxiv.org/pdf/2404.07448
• Copy Paste: [[2404.07448]] Transferable and Principled Efficiency for Open-Vocabulary Segmentation(https://arxiv.org/abs/2404.07448)
• Keywords: segmentation
• Abstract: Recent success of pre-trained foundation vision-language models makes Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance, this approach introduces heavy computational overheads for two challenges: 1) large model sizes of the backbone; 2) expensive costs during the fine-tuning. These challenges hinder this OVS strategy from being widely applicable and affordable in real-world scenarios. Although traditional methods such as model compression and efficient fine-tuning can address these challenges, they often rely on heuristics. This means that their solutions cannot be easily transferred and necessitate re-training on different models, which comes at a cost. In the context of efficient OVS, we target achieving performance that is comparable to or even better than prior OVS works based on large vision-language foundation models, by utilizing smaller models that incur lower training costs. The core strategy is to make our efficiency principled and thus seamlessly transferable from one OVS framework to others without further customization. Comprehensive experiments on diverse OVS benchmarks demonstrate our superior trade-off between segmentation accuracy and computation costs over previous works. Our code is available on https://github.com/Xujxyang/OpenTrans

Title: Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs

• Authors: Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed, Michael S. Ryoo, Tsung-Yu Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07449
• Pdf URL: https://arxiv.org/pdf/2404.07449
• Copy Paste: [[2404.07449]] Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs(https://arxiv.org/abs/2404.07449)
• Keywords: large language model
• Abstract: Integration of Large Language Models (LLMs) into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks, particularly for visual question answering (VQA). However, existing V-LLMs (e.g. BLIP-2, LLaVA) demonstrate weak spatial reasoning and localization awareness. Despite generating highly descriptive and elaborate textual answers, these models fail at simple tasks like distinguishing a left vs right location. In this work, we explore how image-space coordinate based instruction fine-tuning objectives could inject spatial awareness into V-LLMs. We discover optimal coordinate representations, data-efficient instruction fine-tuning objectives, and pseudo-data generation strategies that lead to improved spatial awareness in V-LLMs. Additionally, our resulting model improves VQA across image and video domains, reduces undesired hallucination, and generates better contextual object descriptions. Experiments across 5 vision-language tasks involving 14 different datasets establish the clear performance improvements achieved by our proposed framework.

Title: "Confidently Nonsensical?'': A Critical Survey on the Perspectives and Challenges of 'Hallucinations' in NLP

• Authors: Pranav Narayanan Venkit, Tatiana Chakravorti, Vipul Gupta, Heidi Biggs, Mukund Srinath, Koustava Goswami, Sarah Rajtmajer, Shomir Wilson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07461
• Pdf URL: https://arxiv.org/pdf/2404.07461
• Copy Paste: [[2404.07461]] "Confidently Nonsensical?'': A Critical Survey on the Perspectives and Challenges of 'Hallucinations' in NLP(https://arxiv.org/abs/2404.07461)
• Keywords: large language model
• Abstract: We investigate how hallucination in large language models (LLM) is characterized in peer-reviewed literature using a critical examination of 103 publications across NLP research. Through a comprehensive review of sociological and technological literature, we identify a lack of agreement with the term `hallucination.' Additionally, we conduct a survey with 171 practitioners from the field of NLP and AI to capture varying perspectives on hallucination. Our analysis underscores the necessity for explicit definitions and frameworks outlining hallucination within NLP, highlighting potential challenges, and our survey inputs provide a thematic understanding of the influence and ramifications of hallucination in society.

Title: Enhancing Network Intrusion Detection Performance using Generative Adversarial Networks

• Authors: Xinxing Zhao, Kar Wai Fok, Vrizlynn L. L. Thing
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07464
• Pdf URL: https://arxiv.org/pdf/2404.07464
• Copy Paste: [[2404.07464]] Enhancing Network Intrusion Detection Performance using Generative Adversarial Networks(https://arxiv.org/abs/2404.07464)
• Keywords: security, attack, generative
• Abstract: Network intrusion detection systems (NIDS) play a pivotal role in safeguarding critical digital infrastructures against cyber threats. Machine learning-based detection models applied in NIDS are prevalent today. However, the effectiveness of these machine learning-based models is often limited by the evolving and sophisticated nature of intrusion techniques as well as the lack of diverse and updated training samples. In this research, a novel approach for enhancing the performance of an NIDS through the integration of Generative Adversarial Networks (GANs) is proposed. By harnessing the power of GANs in generating synthetic network traffic data that closely mimics real-world network behavior, we address a key challenge associated with NIDS training datasets, which is the data scarcity. Three distinct GAN models (Vanilla GAN, Wasserstein GAN and Conditional Tabular GAN) are implemented in this work to generate authentic network traffic patterns specifically tailored to represent the anomalous activity. We demonstrate how this synthetic data resampling technique can significantly improve the performance of the NIDS model for detecting such activity. By conducting comprehensive experiments using the CIC-IDS2017 benchmark dataset, augmented with GAN-generated data, we offer empirical evidence that shows the effectiveness of our proposed approach. Our findings show that the integration of GANs into NIDS can lead to enhancements in intrusion detection performance for attacks with limited training data, making it a promising avenue for bolstering the cybersecurity posture of organizations in an increasingly interconnected and vulnerable digital landscape.

Title: Scalable Language Model with Generalized Continual Learning

• Authors: Bohao Peng, Zhuotao Tian, Shu Liu, Mingchang Yang, Jiaya Jia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07470
• Pdf URL: https://arxiv.org/pdf/2404.07470
• Copy Paste: [[2404.07470]] Scalable Language Model with Generalized Continual Learning(https://arxiv.org/abs/2404.07470)
• Keywords: large language model
• Abstract: Continual learning has gained increasing importance as it facilitates the acquisition and refinement of scalable knowledge and skills in language models. However, existing methods typically encounter strict limitations and challenges in real-world scenarios, such as reliance on experience replay, optimization constraints, and inference task-ID. In this study, we introduce the Scalable Language Model (SLM) to overcome these limitations within a more challenging and generalized setting, representing a significant advancement toward practical applications for continual learning. Specifically, we propose the Joint Adaptive Re-Parameterization (JARe), integrated with Dynamic Task-related Knowledge Retrieval (DTKR), to enable adaptive adjustment of language models based on specific downstream tasks. This approach leverages the task distribution within the vector space, aiming to achieve a smooth and effortless continual learning process. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting. Moreover, while prior research primarily focused on a single task type such as classification, our study goes beyond, with the large language model, i.e., LLaMA-2, to explore the effects across diverse domains and task types, such that a single language model can be decently scaled to broader applications.

Title: Laissez-Faire Harms: Algorithmic Biases in Generative Language Models

• Authors: Evan Shieh, Faye-Marie Vassel, Cassidy Sugimoto, Thema Monroe-White
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07475
• Pdf URL: https://arxiv.org/pdf/2404.07475
• Copy Paste: [[2404.07475]] Laissez-Faire Harms: Algorithmic Biases in Generative Language Models(https://arxiv.org/abs/2404.07475)
• Keywords: protect, fair, generative
• Abstract: The rapid deployment of generative language models (LMs) has raised concerns about social biases affecting the well-being of diverse consumers. The extant literature on generative LMs has primarily examined bias via explicit identity prompting. However, prior research on bias in earlier language-based technology platforms, including search engines, has shown that discrimination can occur even when identity terms are not specified explicitly. Studies of bias in LM responses to open-ended prompts (where identity classifications are left unspecified) are lacking and have not yet been grounded in end-consumer harms. Here, we advance studies of generative LM bias by considering a broader set of natural use cases via open-ended prompting. In this "laissez-faire" setting, we find that synthetically generated texts from five of the most pervasive LMs (ChatGPT3.5, ChatGPT4, Claude2.0, Llama2, and PaLM2) perpetuate harms of omission, subordination, and stereotyping for minoritized individuals with intersectional race, gender, and/or sexual orientation identities (AI/AN, Asian, Black, Latine, MENA, NH/PI, Female, Non-binary, Queer). We find widespread evidence of bias to an extent that such individuals are hundreds to thousands of times more likely to encounter LM-generated outputs that portray their identities in a subordinated manner compared to representative or empowering portrayals. We also document a prevalence of stereotypes (e.g. perpetual foreigner) in LM-generated outputs that are known to trigger psychological harms that disproportionately affect minoritized individuals. These include stereotype threat, which leads to impaired cognitive performance and increased negative self-perception. Our findings highlight the urgent need to protect consumers from discriminatory harms caused by language models and invest in critical AI education programs tailored towards empowering diverse consumers.

Title: PillarTrack: Redesigning Pillar-based Transformer Network for Single Object Tracking on Point Clouds

• Authors: Weisheng Xu, Sifan Zhou, Zhihang Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07495
• Pdf URL: https://arxiv.org/pdf/2404.07495
• Copy Paste: [[2404.07495]] PillarTrack: Redesigning Pillar-based Transformer Network for Single Object Tracking on Point Clouds(https://arxiv.org/abs/2404.07495)
• Keywords: transformer
• Abstract: LiDAR-based 3D single object tracking (3D SOT) is a critical issue in robotics and autonomous driving. It aims to obtain accurate 3D BBox from the search area based on similarity or motion. However, existing 3D SOT methods usually follow the point-based pipeline, where the sampling operation inevitably leads to redundant or lost information, resulting in unexpected performance. To address these issues, we propose PillarTrack, a pillar-based 3D single object tracking framework. Firstly, we transform sparse point clouds into dense pillars to preserve the local and global geometrics. Secondly, we introduce a Pyramid-type Encoding Pillar Feature Encoder (PE-PFE) design to help the feature representation of each pillar. Thirdly, we present an efficient Transformer-based backbone from the perspective of modality differences. Finally, we construct our PillarTrack tracker based above designs. Extensive experiments on the KITTI and nuScenes dataset demonstrate the superiority of our proposed method. Notably, our method achieves state-of-the-art performance on the KITTI and nuScenes dataset and enables real-time tracking speed. We hope our work could encourage the community to rethink existing 3D SOT tracker designs.We will open source our code to the research community in https://github.com/StiphyJay/PillarTrack.

Title: Interactive Prompt Debugging with Sequence Salience

• Authors: Ian Tenney, Ryan Mullins, Bin Du, Shree Pandya, Minsuk Kahng, Lucas Dixon
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07498
• Pdf URL: https://arxiv.org/pdf/2404.07498
• Copy Paste: [[2404.07498]] Interactive Prompt Debugging with Sequence Salience(https://arxiv.org/abs/2404.07498)
• Keywords: interpretability
• Abstract: We present Sequence Salience, a visual tool for interactive prompt debugging with input salience methods. Sequence Salience builds on widely used salience methods for text classification and single-token prediction, and extends this to a system tailored for debugging complex LLM prompts. Our system is well-suited for long texts, and expands on previous work by 1) providing controllable aggregation of token-level salience to the word, sentence, or paragraph level, making salience over long inputs tractable; and 2) supporting rapid iteration where practitioners can act on salience results, refine prompts, and run salience on the new output. We include case studies showing how Sequence Salience can help practitioners work with several complex prompting strategies, including few-shot, chain-of-thought, and constitutional principles. Sequence Salience is built on the Learning Interpretability Tool, an open-source platform for ML model visualizations, and code, notebooks, and tutorials are available at this http URL

Title: Leveraging Data Augmentation for Process Information Extraction

• Authors: Julian Neuberger, Leonie Doll, Benedict Engelmann, Lars Ackermann, Stefan Jablonski
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07501
• Pdf URL: https://arxiv.org/pdf/2404.07501
• Copy Paste: [[2404.07501]] Leveraging Data Augmentation for Process Information Extraction(https://arxiv.org/abs/2404.07501)
• Keywords: extraction
• Abstract: Business Process Modeling projects often require formal process models as a central component. High costs associated with the creation of such formal process models motivated many different fields of research aimed at automated generation of process models from readily available data. These include process mining on event logs, and generating business process models from natural language texts. Research in the latter field is regularly faced with the problem of limited data availability, hindering both evaluation and development of new techniques, especially learning-based ones. To overcome this data scarcity issue, in this paper we investigate the application of data augmentation for natural language text data. Data augmentation methods are well established in machine learning for creating new, synthetic data without human assistance. We find that many of these methods are applicable to the task of business process information extraction, improving the accuracy of extraction. Our study shows, that data augmentation is an important component in enabling machine learning methods for the task of business process model generation from natural language text, where currently mostly rule-based systems are still state of the art. Simple data augmentation techniques improved the $F_1$ score of mention extraction by 2.9 percentage points, and the $F_1$ of relation extraction by $4.5$. To better understand how data augmentation alters human annotated texts, we analyze the resulting text, visualizing and discussing the properties of augmented textual data. We make all code and experiments results publicly available.

Title: Generating Counterfactual Explanations Using Cardinality Constraints

• Authors: Rubén Ruiz-Torrubiano
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07502
• Pdf URL: https://arxiv.org/pdf/2404.07502
• Copy Paste: [[2404.07502]] Generating Counterfactual Explanations Using Cardinality Constraints(https://arxiv.org/abs/2404.07502)
• Keywords: robust, fair
• Abstract: Providing explanations about how machine learning algorithms work and/or make particular predictions is one of the main tools that can be used to improve their trusworthiness, fairness and robustness. Among the most intuitive type of explanations are counterfactuals, which are examples that differ from a given point only in the prediction target and some set of features, presenting which features need to be changed in the original example to flip the prediction for that example. However, such counterfactuals can have many different features than the original example, making their interpretation difficult. In this paper, we propose to explicitly add a cardinality constraint to counterfactual generation limiting how many features can be different from the original example, thus providing more interpretable and easily understantable counterfactuals.

Title: Best Practices and Lessons Learned on Synthetic Data for Language Models

• Authors: Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, Andrew M. Dai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07503
• Pdf URL: https://arxiv.org/pdf/2404.07503
• Copy Paste: [[2404.07503]] Best Practices and Lessons Learned on Synthetic Data for Language Models(https://arxiv.org/abs/2404.07503)
• Keywords: privacy
• Abstract: The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for responsible use of synthetic data to build more powerful, inclusive, and trustworthy language models.

Title: Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange

• Authors: Yanhao Wu, Tong Zhang, Wei Ke, Congpei Qiu, Sabine Susstrunk, Mathieu Salzmann
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07504
• Pdf URL: https://arxiv.org/pdf/2404.07504
• Copy Paste: [[2404.07504]] Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange(https://arxiv.org/abs/2404.07504)
• Keywords: robust
• Abstract: In the realm of point cloud scene understanding, particularly in indoor scenes, objects are arranged following human habits, resulting in objects of certain semantics being closely positioned and displaying notable inter-object correlations. This can create a tendency for neural networks to exploit these strong dependencies, bypassing the individual object patterns. To address this challenge, we introduce a novel self-supervised learning (SSL) strategy. Our approach leverages both object patterns and contextual cues to produce robust features. It begins with the formulation of an object-exchanging strategy, where pairs of objects with comparable sizes are exchanged across different scenes, effectively disentangling the strong contextual dependencies. Subsequently, we introduce a context-aware feature learning strategy, which encodes object patterns without relying on their specific context by aggregating object features across various scenes. Our extensive experiments demonstrate the superiority of our method over existing SSL techniques, further showing its better robustness to environmental changes. Moreover, we showcase the applicability of our approach by transferring pre-trained models to diverse point cloud datasets.

Title: Remembering Transformer for Continual Learning

• Authors: Yuwei Sun, Jun Sakuma, Ryota Kanai
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07518
• Pdf URL: https://arxiv.org/pdf/2404.07518
• Copy Paste: [[2404.07518]] Remembering Transformer for Continual Learning(https://arxiv.org/abs/2404.07518)
• Keywords: transformer, generative
• Abstract: Neural networks encounter the challenge of Catastrophic Forgetting (CF) in continual learning, where new task knowledge interferes with previously learned knowledge. We propose Remembering Transformer, inspired by the brain's Complementary Learning Systems (CLS), to tackle this issue. Remembering Transformer employs a mixture-of-adapters and a generative model-based routing mechanism to alleviate CF by dynamically routing task data to relevant adapters. Our approach demonstrated a new SOTA performance in various vision continual learning tasks and great parameter efficiency.

Title: Security Modelling for Cyber-Physical Systems: A Systematic Literature Review

• Authors: Shaofei Huang, Christopher M. Poskitt, Lwin Khin Shar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07527
• Pdf URL: https://arxiv.org/pdf/2404.07527
• Copy Paste: [[2404.07527]] Security Modelling for Cyber-Physical Systems: A Systematic Literature Review(https://arxiv.org/abs/2404.07527)
• Keywords: security, attack
• Abstract: Cyber-physical systems (CPS) are at the intersection of digital technology and engineering domains, rendering them high-value targets of sophisticated and well-funded cybersecurity threat actors. Prominent cybersecurity attacks on CPS have brought attention to the vulnerability of these systems, and the soft underbelly of critical infrastructure reliant on CPS. Security modelling for CPS is an important mechanism to systematically identify and assess vulnerabilities, threats, and risks throughout system lifecycles, and to ultimately ensure system resilience, safety, and reliability. This literature review delves into state-of-the-art research in CPS security modelling, encompassing both threat and attack modelling. While these terms are sometimes used interchangeably, they are different concepts. This article elaborates on the differences between threat and attack modelling, examining their implications for CPS security. A systematic search yielded 428 articles, from which 15 were selected and categorised into three clusters: those focused on threat modelling methods, attack modelling methods, and literature reviews. Specifically, we sought to examine what security modelling methods exist today, and how they address real-world cybersecurity threats and CPS-specific attacker capabilities throughout the lifecycle of CPS, which typically span longer durations compared to traditional IT systems. This article also highlights several limitations in existing research, wherein security models adopt simplistic approaches that do not adequately consider the dynamic, multi-layer, multi-path, and multi-agent characteristics of real-world cyber-physical attacks.

Title: Bayesian Federated Model Compression for Communication and Computation Efficiency

• Authors: Chengyu Xia, Danny H. K. Tsang, Vincent K. N. Lau
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.07532
• Pdf URL: https://arxiv.org/pdf/2404.07532
• Copy Paste: [[2404.07532]] Bayesian Federated Model Compression for Communication and Computation Efficiency(https://arxiv.org/abs/2404.07532)
• Keywords: federate
• Abstract: In this paper, we investigate Bayesian model compression in federated learning (FL) to construct sparse models that can achieve both communication and computation efficiencies. We propose a decentralized Turbo variational Bayesian inference (D-Turbo-VBI) FL framework where we firstly propose a hierarchical sparse prior to promote a clustered sparse structure in the weight matrix. Then, by carefully integrating message passing and VBI with a decentralized turbo framework, we propose the D-Turbo-VBI algorithm which can (i) reduce both upstream and downstream communication overhead during federated training, and (ii) reduce the computational complexity during local inference. Additionally, we establish the convergence property for thr proposed D-Turbo-VBI algorithm. Simulation results show the significant gain of our proposed algorithm over the baselines in reducing communication overhead during federated training and computational complexity of final model.

Title: IITP-VDLand: A Comprehensive Dataset on Decentraland Parcels

• Authors: Ankit K. Bhagat, Dipika Jha, Raju Halder, Rajendra N. Paramanik, Chandra M. Kumar
• Subjects: cs.LG, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2404.07533
• Pdf URL: https://arxiv.org/pdf/2404.07533
• Copy Paste: [[2404.07533]] IITP-VDLand: A Comprehensive Dataset on Decentraland Parcels(https://arxiv.org/abs/2404.07533)
• Keywords: robust
• Abstract: This paper presents IITP-VDLand, a comprehensive dataset of Decentraland parcels sourced from diverse platforms. Unlike existing datasets which have limited attributes and records, IITP-VDLand offers a rich array of attributes, encompassing parcel characteristics, trading history, past activities, transactions, and social media interactions. Alongside, we introduce a key attribute in the dataset, namely Rarity score, which measures the uniqueness of each parcel within the virtual world. Addressing the significant challenge posed by the dispersed nature of this data across various sources, we employ a systematic approach, utilizing both available APIs and custom scripts, to gather it. Subsequently, we meticulously curate and organize the information into four distinct segments: (1) Characteristics Data-Fragment, (2) OpenSea Trading History Data-Fragment, (3) Ethereum Activity Transactions Data-Fragment, and (4) Social Media Data-Fragment. We envisage that this dataset would serve as a robust resource for training machine- and deep-learning models specifically designed to address real-world challenges within the domain of Decentraland parcels. The performance benchmarking of more than 20 state-of-the-art price prediction models on our dataset yields promising results, achieving a maximum R2 score of 0.8251 and an accuracy of 74.23% in case of Extra Trees Regressor and Classifier. The key findings reveal that the ensemble models performs better than both deep learning and linear models for our dataset. We observe a significant impact of coordinates, geographical proximity, rarity score, and few other economic indicators on the prediction of parcel prices.

Title: From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

• Authors: Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, Mihai Surdeanu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07544
• Pdf URL: https://arxiv.org/pdf/2404.07544
• Copy Paste: [[2404.07544]] From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples(https://arxiv.org/abs/2404.07544)
• Keywords: large language model
• Abstract: We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g., GPT-4, Claude 3) are able to perform regression tasks with a performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, on the challenging Friedman #2 regression dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM, Random Forest, KNN, or Gradient Boosting. We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining a sub-linear regret.

Title: Decomposing Label Space, Format and Discrimination: Rethinking How LLMs Respond and Solve Tasks via In-Context Learning

• Authors: Quanyu Long, Yin Wu, Wenya Wang, Sinno Jialin Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07546
• Pdf URL: https://arxiv.org/pdf/2404.07546
• Copy Paste: [[2404.07546]] Decomposing Label Space, Format and Discrimination: Rethinking How LLMs Respond and Solve Tasks via In-Context Learning(https://arxiv.org/abs/2404.07546)
• Keywords: large language model
• Abstract: In-context Learning (ICL) has emerged as a powerful capability alongside the development of scaled-up large language models (LLMs). By instructing LLMs using few-shot demonstrative examples, ICL enables them to perform a wide range of tasks without updating millions of parameters. However, the precise contributions of demonstrations towards improving end-task performance have not been thoroughly investigated in recent analytical studies. In this paper, we empirically decompose the overall performance of ICL into three dimensions, label space, format, and discrimination, and we evaluate four general-purpose LLMs across a diverse range of tasks. Counter-intuitively, we find that the demonstrations have a marginal impact on provoking discriminative knowledge of language models. However, ICL exhibits significant efficacy in regulating the label space and format which helps LLMs to respond in desired label words. We then demonstrate this ability functions similar to detailed instructions for LLMs to follow. We additionally provide an in-depth analysis of the mechanism of retrieval helping with ICL and find that retrieving the most semantically similar examples notably boosts model's discriminative capability.

Title: Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective

• Authors: Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07549
• Pdf URL: https://arxiv.org/pdf/2404.07549
• Copy Paste: [[2404.07549]] Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective(https://arxiv.org/abs/2404.07549)
• Keywords: robust, large language model
• Abstract: Code generation aims to understand the problem description and generate corresponding code snippets, where existing works generally decompose such complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants. While these studies have achieved some success, their effectiveness is highly dependent on the capabilities of advanced Large Language Models (LLMs) such as GPT-4, particularly in terms of API calls, which significantly limits their practical applicability. Consequently, how to enhance the code generation capabilities of small and medium-scale code LLMs without significantly increasing training costs is an appealing challenge. In this paper, we suggest that code comments are the natural logic pivot between natural language and code language and propose using comments to boost the code generation ability of code LLMs. Concretely, we propose MANGO (comMents As Natural loGic pivOts), including a comment contrastive training strategy and a corresponding logical comment decoding strategy. Experiments are performed on HumanEval and MBPP, utilizing StarCoder and WizardCoder as backbone models, and encompassing model parameter sizes between 3B and 7B. The results indicate that MANGO significantly improves the code pass rate based on the strong baselines. Meanwhile, the robustness of the logical comment decoding strategy is notably higher than the Chain-of-thoughts prompting. The code is publicly available at \url{https://github.com/pppa2019/Mango}.

Title: CAT: Contrastive Adapter Training for Personalized Image Generation

• Authors: Jae Wan Park, Sang Hyun Park, Jun Young Koh, Junha Lee, Min Song
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07554
• Pdf URL: https://arxiv.org/pdf/2404.07554
• Copy Paste: [[2404.07554]] CAT: Contrastive Adapter Training for Personalized Image Generation(https://arxiv.org/abs/2404.07554)
• Keywords: diffusion
• Abstract: The emergence of various adapters, including Low-Rank Adaptation (LoRA) applied from the field of natural language processing, has allowed diffusion models to personalize image generation at a low cost. However, due to the various challenges including limited datasets and shortage of regularization and computation resources, adapter training often results in unsatisfactory outcomes, leading to the corruption of the backbone model's prior knowledge. One of the well known phenomena is the loss of diversity in object generation, especially within the same class which leads to generating almost identical objects with minor variations. This poses challenges in generation capabilities. To solve this issue, we present Contrastive Adapter Training (CAT), a simple yet effective strategy to enhance adapter training through the application of CAT loss. Our approach facilitates the preservation of the base model's original knowledge when the model initiates adapters. Furthermore, we introduce the Knowledge Preservation Score (KPS) to evaluate CAT's ability to keep the former information. We qualitatively and quantitatively compare CAT's improvement. Finally, we mention the possibility of CAT in the aspects of multi-concept adapter and optimization.

Title: Towards Secure and Reliable Heterogeneous Real-time Telemetry Communication in Autonomous UAV Swarms

• Authors: Pavlo Mykytyn, Marcin Brzozowski, Zoya Dyka, Peter Langendörfer
• Subjects: cs.CR, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.07557
• Pdf URL: https://arxiv.org/pdf/2404.07557
• Copy Paste: [[2404.07557]] Towards Secure and Reliable Heterogeneous Real-time Telemetry Communication in Autonomous UAV Swarms(https://arxiv.org/abs/2404.07557)
• Keywords: secure, security
• Abstract: In the era of cutting-edge autonomous systems, Unmanned Aerial Vehicles (UAVs) are becoming an essential part of the solutions for numerous complex challenges. This paper evaluates UAV peer-to-peer telemetry communication, highlighting its security vulnerabilities and explores a transition to a het-erogeneous multi-hop mesh all-to-all communication architecture to increase inter-swarm connectivity and reliability. Additionally, we suggest a symmetric key agreement and data encryption mechanism implementation for inter - swarm communication, to ensure data integrity and confidentiality without compromising performance.

Title: Differentially Private Reinforcement Learning with Self-Play

• Authors: Dan Qiao, Yu-Xiang Wang
• Subjects: cs.LG, cs.AI, cs.CR, cs.MA, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.07559
• Pdf URL: https://arxiv.org/pdf/2404.07559
• Copy Paste: [[2404.07559]] Differentially Private Reinforcement Learning with Self-Play(https://arxiv.org/abs/2404.07559)
• Keywords: privacy, protect
• Abstract: We study the problem of multi-agent reinforcement learning (multi-agent RL) with differential privacy (DP) constraints. This is well-motivated by various real-world applications involving sensitive data, where it is critical to protect users' private information. We first extend the definitions of Joint DP (JDP) and Local DP (LDP) to two-player zero-sum episodic Markov Games, where both definitions ensure trajectory-wise privacy protection. Then we design a provably efficient algorithm based on optimistic Nash value iteration and privatization of Bernstein-type bonuses. The algorithm is able to satisfy JDP and LDP requirements when instantiated with appropriate privacy mechanisms. Furthermore, for both notions of DP, our regret bound generalizes the best known result under the single-agent RL case, while our regret could also reduce to the best known result for multi-agent RL without privacy constraints. To the best of our knowledge, these are the first line of results towards understanding trajectory-wise privacy protection in multi-agent RL.

Title: ObjBlur: A Curriculum Learning Approach With Progressive Object-Level Blurring for Improved Layout-to-Image Generation

• Authors: Stanislav Frolov, Brian B. Moser, Sebastian Palacio, Andreas Dengel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07564
• Pdf URL: https://arxiv.org/pdf/2404.07564
• Copy Paste: [[2404.07564]] ObjBlur: A Curriculum Learning Approach With Progressive Object-Level Blurring for Improved Layout-to-Image Generation(https://arxiv.org/abs/2404.07564)
• Keywords: diffusion, generative
• Abstract: We present ObjBlur, a novel curriculum learning approach to improve layout-to-image generation models, where the task is to produce realistic images from layouts composed of boxes and labels. Our method is based on progressive object-level blurring, which effectively stabilizes training and enhances the quality of generated images. This curriculum learning strategy systematically applies varying degrees of blurring to individual objects or the background during training, starting from strong blurring to progressively cleaner images. Our findings reveal that this approach yields significant performance improvements, stabilized training, smoother convergence, and reduced variance between multiple runs. Moreover, our technique demonstrates its versatility by being compatible with generative adversarial networks and diffusion models, underlining its applicability across various generative modeling paradigms. With ObjBlur, we reach new state-of-the-art results on the complex COCO and Visual Genome datasets.

Title: Fragile Model Watermark for integrity protection: leveraging boundary volatility and sensitive sample-pairing

• Authors: ZhenZhe Gao, Zhenjun Tang, Zhaoxia Yin, Baoyuan Wu, Yue Lu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07572
• Pdf URL: https://arxiv.org/pdf/2404.07572
• Copy Paste: [[2404.07572]] Fragile Model Watermark for integrity protection: leveraging boundary volatility and sensitive sample-pairing(https://arxiv.org/abs/2404.07572)
• Keywords: protect, attack, watermark
• Abstract: Neural networks have increasingly influenced people's lives. Ensuring the faithful deployment of neural networks as designed by their model owners is crucial, as they may be susceptible to various malicious or unintentional modifications, such as backdooring and poisoning attacks. Fragile model watermarks aim to prevent unexpected tampering that could lead DNN models to make incorrect decisions. They ensure the detection of any tampering with the model as sensitively as possible.However, prior watermarking methods suffered from inefficient sample generation and insufficient sensitivity, limiting their practical applicability. Our approach employs a sample-pairing technique, placing the model boundaries between pairs of samples, while simultaneously maximizing logits. This ensures that the model's decision results of sensitive samples change as much as possible and the Top-1 labels easily alter regardless of the direction it moves.

Title: Generating Comprehensive Lithium Battery Charging Data with Generative AI

• Authors: Lidang Jiang, Changyan Hu, Sibei Ji, Hang Zhao, Junxiong Chen, Ge He
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.07577
• Pdf URL: https://arxiv.org/pdf/2404.07577
• Copy Paste: [[2404.07577]] Generating Comprehensive Lithium Battery Charging Data with Generative AI(https://arxiv.org/abs/2404.07577)
• Keywords: generative
• Abstract: In optimizing performance and extending the lifespan of lithium batteries, accurate state prediction is pivotal. Traditional regression and classification methods have achieved some success in battery state prediction. However, the efficacy of these data-driven approaches heavily relies on the availability and quality of public datasets. Additionally, generating electrochemical data predominantly through battery experiments is a lengthy and costly process, making it challenging to acquire high-quality electrochemical data. This difficulty, coupled with data incompleteness, significantly impacts prediction accuracy. Addressing these challenges, this study introduces the End of Life (EOL) and Equivalent Cycle Life (ECL) as conditions for generative AI models. By integrating an embedding layer into the CVAE model, we developed the Refined Conditional Variational Autoencoder (RCVAE). Through preprocessing data into a quasi-video format, our study achieves an integrated synthesis of electrochemical data, including voltage, current, temperature, and charging capacity, which is then processed by the RCVAE model. Coupled with customized training and inference algorithms, this model can generate specific electrochemical data for EOL and ECL under supervised conditions. This method provides users with a comprehensive electrochemical dataset, pioneering a new research domain for the artificial synthesis of lithium battery data. Furthermore, based on the detailed synthetic data, various battery state indicators can be calculated, offering new perspectives and possibilities for lithium battery performance prediction.

Title: Multi-rater Prompting for Ambiguous Medical Image Segmentation

• Authors: Jinhong Wang, Yi Cheng, Jintai Chen, Hongxia Xu, Danny Chen, Jian Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07580
• Pdf URL: https://arxiv.org/pdf/2404.07580
• Copy Paste: [[2404.07580]] Multi-rater Prompting for Ambiguous Medical Image Segmentation(https://arxiv.org/abs/2404.07580)
• Keywords: segmentation
• Abstract: Multi-rater annotations commonly occur when medical images are independently annotated by multiple experts (raters). In this paper, we tackle two challenges arisen in multi-rater annotations for medical image segmentation (called ambiguous medical image segmentation): (1) How to train a deep learning model when a group of raters produces a set of diverse but plausible annotations, and (2) how to fine-tune the model efficiently when computation resources are not available for re-training the entire model on a different dataset domain. We propose a multi-rater prompt-based approach to address these two challenges altogether. Specifically, we introduce a series of rater-aware prompts that can be plugged into the U-Net model for uncertainty estimation to handle multi-annotation cases. During the prompt-based fine-tuning process, only 0.3% of learnable parameters are required to be updated comparing to training the entire model. Further, in order to integrate expert consensus and disagreement, we explore different multi-rater incorporation strategies and design a mix-training strategy for comprehensive insight learning. Extensive experiments verify the effectiveness of our new approach for ambiguous medical image segmentation on two public datasets while alleviating the heavy burden of model re-training.

Title: UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs

• Authors: Chaoqun He, Renjie Luo, Shengding Hu, Yuanqian Zhao, Jie Zhou, Hanghao Wu, Jiajie Zhang, Xu Han, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07584
• Pdf URL: https://arxiv.org/pdf/2404.07584
• Copy Paste: [[2404.07584]] UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs(https://arxiv.org/abs/2404.07584)
• Keywords: large language model
• Abstract: Evaluation is pivotal for honing Large Language Models (LLMs), pinpointing their capabilities and guiding enhancements. The rapid development of LLMs calls for a lightweight and easy-to-use framework for swift evaluation deployment. However, due to the various implementation details to consider, developing a comprehensive evaluation platform is never easy. Existing platforms are often complex and poorly modularized, hindering seamless incorporation into researcher's workflows. This paper introduces UltraEval, a user-friendly evaluation framework characterized by lightweight, comprehensiveness, modularity, and efficiency. We identify and reimplement three core components of model evaluation (models, data, and metrics). The resulting composability allows for the free combination of different models, tasks, prompts, and metrics within a unified evaluation workflow. Additionally, UltraEval supports diverse models owing to a unified HTTP service and provides sufficient inference acceleration. UltraEval is now available for researchers publicly~\footnote{Website is at \url{https://github.com/OpenBMB/UltraEval}}.

Title: Weakly-Supervised Learning via Multi-Lateral Decoder Branching for Guidewire Segmentation in Robot-Assisted Cardiovascular Catheterization

• Authors: Olatunji Mumini Omisore, Toluwanimi Akinyemi, Anh Nguyen, Lei Wang
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.07594
• Pdf URL: https://arxiv.org/pdf/2404.07594
• Copy Paste: [[2404.07594]] Weakly-Supervised Learning via Multi-Lateral Decoder Branching for Guidewire Segmentation in Robot-Assisted Cardiovascular Catheterization(https://arxiv.org/abs/2404.07594)
• Keywords: segmentation
• Abstract: Although robot-assisted cardiovascular catheterization is commonly performed for intervention of cardiovascular diseases, more studies are needed to support the procedure with automated tool segmentation. This can aid surgeons on tool tracking and visualization during intervention. Learning-based segmentation has recently offered state-of-the-art segmentation performances however, generating ground-truth signals for fully-supervised methods is labor-intensive and time consuming for the interventionists. In this study, a weakly-supervised learning method with multi-lateral pseudo labeling is proposed for tool segmentation in cardiac angiograms. The method includes a modified U-Net model with one encoder and multiple lateral-branched decoders that produce pseudo labels as supervision signals under different perturbation. The pseudo labels are self-generated through a mixed loss function and shared consistency in the decoders. We trained the model end-to-end with weakly-annotated data obtained during robotic cardiac catheterization. Experiments with the proposed model shows weakly annotated data has closer performance to when fully annotated data is used. Compared to three existing weakly-supervised methods, our approach yielded higher segmentation performance across three different cardiac angiogram data. With ablation study, we showed consistent performance under different parameters. Thus, we offer a less expensive method for real-time tool segmentation and tracking during robot-assisted cardiac catheterization.

Title: Implicit and Explicit Language Guidance for Diffusion-based Visual Perception

• Authors: Hefeng Wang, Jiale Cao, Jin Xie, Aiping Yang, Yanwei Pang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07600
• Pdf URL: https://arxiv.org/pdf/2404.07600
• Copy Paste: [[2404.07600]] Implicit and Explicit Language Guidance for Diffusion-based Visual Perception(https://arxiv.org/abs/2404.07600)
• Keywords: extraction, diffusion, segmentation
• Abstract: Text-to-image diffusion models have shown powerful ability on conditional image synthesis. With large-scale vision-language pre-training, diffusion models are able to generate high-quality images with rich texture and reasonable structure under different text prompts. However, it is an open problem to adapt the pre-trained diffusion model for visual perception. In this paper, we propose an implicit and explicit language guidance framework for diffusion-based perception, named IEDP. Our IEDP comprises of an implicit language guidance branch and an explicit language guidance branch. The implicit branch employs frozen CLIP image encoder to directly generate implicit text embeddings that are fed to diffusion model, without using explicit text prompts. The explicit branch utilizes the ground-truth labels of corresponding images as text prompts to condition feature extraction of diffusion model. During training, we jointly train diffusion model by sharing the model weights of these two branches. As a result, implicit and explicit branches can jointly guide feature learning. During inference, we only employ implicit branch for final prediction, which does not require any ground-truth labels. Experiments are performed on two typical perception tasks, including semantic segmentation and depth estimation. Our IEDP achieves promising performance on both tasks. For semantic segmentation, our IEDP has the mIoU score of 55.9% on AD20K validation set, which outperforms the baseline method VPD by 2.2%. For depth estimation, our IEDP outperforms the baseline method VPD with a relative gain of 10.2%.

Title: Attention based End to end network for Offline Writer Identification on Word level data

• Authors: Vineet Kumar, Suresh Sundaram
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07602
• Pdf URL: https://arxiv.org/pdf/2404.07602
• Copy Paste: [[2404.07602]] Attention based End to end network for Offline Writer Identification on Word level data(https://arxiv.org/abs/2404.07602)
• Keywords: robust
• Abstract: Writer identification due to its widespread application in various fields has gained popularity over the years. In scenarios where optimum handwriting samples are available, whether they be in the form of a single line, a sentence, or an entire page, writer identification algorithms have demonstrated noteworthy levels of accuracy. However, in scenarios where only a limited number of handwritten samples are available, particularly in the form of word images, there is a significant scope for improvement. In this paper, we propose a writer identification system based on an attention-driven Convolutional Neural Network (CNN). The system is trained utilizing image segments, known as fragments, extracted from word images, employing a pyramid-based strategy. This methodology enables the system to capture a comprehensive representation of the data, encompassing both fine-grained details and coarse features across various levels of abstraction. These extracted fragments serve as the training data for the convolutional network, enabling it to learn a more robust representation compared to traditional convolution-based networks trained on word images. Additionally, the paper explores the integration of an attention mechanism to enhance the representational power of the learned features. The efficacy of the proposed algorithm is evaluated on three benchmark databases, demonstrating its proficiency in writer identification tasks, particularly in scenarios with limited access to handwriting data.

Title: GLID: Pre-training a Generalist Encoder-Decoder Vision Model

• Authors: Jihao Liu, Jinliang Zheng, Yu Liu, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07603
• Pdf URL: https://arxiv.org/pdf/2404.07603
• Copy Paste: [[2404.07603]] GLID: Pre-training a Generalist Encoder-Decoder Vision Model(https://arxiv.org/abs/2404.07603)
• Keywords: segmentation
• Abstract: This paper proposes a GeneraLIst encoder-Decoder (GLID) pre-training method for better handling various downstream computer vision tasks. While self-supervised pre-training approaches, e.g., Masked Autoencoder, have shown success in transfer learning, task-specific sub-architectures are still required to be appended for different downstream tasks, which cannot enjoy the benefits of large-scale pre-training. GLID overcomes this challenge by allowing the pre-trained generalist encoder-decoder to be fine-tuned on various vision tasks with minimal task-specific architecture modifications. In the GLID training scheme, pre-training pretext task and other downstream tasks are modeled as "query-to-answer" problems, including the pre-training pretext task and other downstream tasks. We pre-train a task-agnostic encoder-decoder with query-mask pairs. During fine-tuning, GLID maintains the pre-trained encoder-decoder and queries, only replacing the topmost linear transformation layer with task-specific linear heads. This minimizes the pretrain-finetune architecture inconsistency and enables the pre-trained model to better adapt to downstream tasks. GLID achieves competitive performance on various vision tasks, including object detection, image segmentation, pose estimation, and depth estimation, outperforming or matching specialist models such as Mask2Former, DETR, ViTPose, and BinsFormer.

Title: Contrastive-Based Deep Embeddings for Label Noise-Resilient Histopathology Image Classification

• Authors: Lucas Dedieu, Nicolas Nerrienet, Adrien Nivaggioli, Clara Simmat, Marceau Clavel, Arnaud Gauthier, Stéphane Sockeel, Rémy Peyret
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07605
• Pdf URL: https://arxiv.org/pdf/2404.07605
• Copy Paste: [[2404.07605]] Contrastive-Based Deep Embeddings for Label Noise-Resilient Histopathology Image Classification(https://arxiv.org/abs/2404.07605)
• Keywords: robust
• Abstract: Recent advancements in deep learning have proven highly effective in medical image classification, notably within histopathology. However, noisy labels represent a critical challenge in histopathology image classification, where accurate annotations are vital for training robust deep learning models. Indeed, deep neural networks can easily overfit label noise, leading to severe degradations in model performance. While numerous public pathology foundation models have emerged recently, none have evaluated their resilience to label noise. Through thorough empirical analyses across multiple datasets, we exhibit the label noise resilience property of embeddings extracted from foundation models trained in a self-supervised contrastive manner. We demonstrate that training with such embeddings substantially enhances label noise robustness when compared to non-contrastive-based ones as well as commonly used noise-resilient methods. Our results unequivocally underline the superiority of contrastive learning in effectively mitigating the label noise challenge. Code is publicly available at https://github.com/LucasDedieu/NoiseResilientHistopathology.

Title: NoticIA: A Clickbait Article Summarization Dataset in Spanish

• Authors: Iker García-Ferrero, Begoña Altuna
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07611
• Pdf URL: https://arxiv.org/pdf/2404.07611
• Copy Paste: [[2404.07611]] NoticIA: A Clickbait Article Summarization Dataset in Spanish(https://arxiv.org/abs/2404.07611)
• Keywords: generative, large language model
• Abstract: We present NoticIA, a dataset consisting of 850 Spanish news articles featuring prominent clickbait headlines, each paired with high-quality, single-sentence generative summarizations written by humans. This task demands advanced text understanding and summarization abilities, challenging the models' capacity to infer and connect diverse pieces of information to meet the user's informational needs generated by the clickbait headline. We evaluate the Spanish text comprehension capabilities of a wide range of state-of-the-art large language models. Additionally, we use the dataset to train ClickbaitFighter, a task-specific model that achieves near-human performance in this task.

Title: Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

• Authors: Iker García-Ferrero, Rodrigo Agerri, Aitziber Atutxa Salazar, Elena Cabrio, Iker de la Iglesia, Alberto Lavelli, Bernardo Magnini, Benjamin Molinet, Johana Ramirez-Romero, German Rigau, Jose Maria Villa-Gonzalez, Serena Villata, Andrea Zaninello
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07613
• Pdf URL: https://arxiv.org/pdf/2404.07613
• Copy Paste: [[2404.07613]] Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain(https://arxiv.org/abs/2404.07613)
• Keywords: large language model
• Abstract: Research on language technology for the development of medical applications is currently a hot topic in Natural Language Understanding and Generation. Thus, a number of large language models (LLMs) have recently been adapted to the medical domain, so that they can be used as a tool for mediating in human-AI interaction. While these LLMs display competitive performance on automated medical texts benchmarks, they have been pre-trained and evaluated with a focus on a single language (English mostly). This is particularly true of text-to-text models, which typically require large amounts of domain-specific pre-training data, often not easily accessible for many languages. In this paper, we address these shortcomings by compiling, to the best of our knowledge, the largest multilingual corpus for the medical domain in four languages, namely English, French, Italian and Spanish. This new corpus has been used to train Medical mT5, the first open-source text-to-text multilingual model for the medical domain. Additionally, we present two new evaluation benchmarks for all four languages with the aim of facilitating multilingual research in this domain. A comprehensive evaluation shows that Medical mT5 outperforms both encoders and similarly sized text-to-text models for the Spanish, French, and Italian benchmarks, while being competitive with current state-of-the-art LLMs in English.

Title: Audio Dialogues: Dialogues dataset for audio and music understanding

• Authors: Arushi Goel, Zhifeng Kong, Rafael Valle, Bryan Catanzaro
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.07616
• Pdf URL: https://arxiv.org/pdf/2404.07616
• Copy Paste: [[2404.07616]] Audio Dialogues: Dialogues dataset for audio and music understanding(https://arxiv.org/abs/2404.07616)
• Keywords: large language model
• Abstract: Existing datasets for audio understanding primarily focus on single-turn interactions (i.e. audio captioning, audio question answering) for describing audio in natural language, thus limiting understanding audio via interactive dialogue. To address this gap, we introduce Audio Dialogues: a multi-turn dialogue dataset containing 163.8k samples for general audio sounds and music. In addition to dialogues, Audio Dialogues also has question-answer pairs to understand and compare multiple input audios together. Audio Dialogues leverages a prompting-based approach and caption annotations from existing datasets to generate multi-turn dialogues using a Large Language Model (LLM). We evaluate existing audio-augmented large language models on our proposed dataset to demonstrate the complexity and applicability of Audio Dialogues. Our code for generating the dataset will be made publicly available. Detailed prompts and generated dialogues can be found on the demo website https://audiodialogues.github.io/.

Title: Multi-Image Visual Question Answering for Unsupervised Anomaly Detection

• Authors: Jun Li, Cosmin I. Bercea, Philip Müller, Lina Felsner, Suhwan Kim, Daniel Rueckert, Benedikt Wiestler, Julia A. Schnabel
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07622
• Pdf URL: https://arxiv.org/pdf/2404.07622
• Copy Paste: [[2404.07622]] Multi-Image Visual Question Answering for Unsupervised Anomaly Detection(https://arxiv.org/abs/2404.07622)
• Keywords: extraction
• Abstract: Unsupervised anomaly detection enables the identification of potential pathological areas by juxtaposing original images with their pseudo-healthy reconstructions generated by models trained exclusively on normal images. However, the clinical interpretation of resultant anomaly maps presents a challenge due to a lack of detailed, understandable explanations. Recent advancements in language models have shown the capability of mimicking human-like understanding and providing detailed descriptions. This raises an interesting question: \textit{How can language models be employed to make the anomaly maps more explainable?} To the best of our knowledge, we are the first to leverage a language model for unsupervised anomaly detection, for which we construct a dataset with different questions and answers. Additionally, we present a novel multi-image visual question answering framework tailored for anomaly detection, incorporating diverse feature fusion strategies to enhance visual knowledge extraction. Our experiments reveal that the framework, augmented by our new Knowledge Q-Former module, adeptly answers questions on the anomaly detection dataset. Besides, integrating anomaly maps as inputs distinctly aids in improving the detection of unseen pathologies.

Title: Homography Guided Temporal Fusion for Road Line and Marking Segmentation

• Authors: Shan Wang, Chuong Nguyen, Jiawei Liu, Kaihao Zhang, Wenhan Luo, Yanhao Zhang, Sundaram Muthu, Fahira Afzal Maken, Hongdong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07626
• Pdf URL: https://arxiv.org/pdf/2404.07626
• Copy Paste: [[2404.07626]] Homography Guided Temporal Fusion for Road Line and Marking Segmentation(https://arxiv.org/abs/2404.07626)
• Keywords: segmentation
• Abstract: Reliable segmentation of road lines and markings is critical to autonomous driving. Our work is motivated by the observations that road lines and markings are (1) frequently occluded in the presence of moving vehicles, shadow, and glare and (2) highly structured with low intra-class shape variance and overall high appearance consistency. To solve these issues, we propose a Homography Guided Fusion (HomoFusion) module to exploit temporally-adjacent video frames for complementary cues facilitating the correct classification of the partially occluded road lines or markings. To reduce computational complexity, a novel surface normal estimator is proposed to establish spatial correspondences between the sampled frames, allowing the HomoFusion module to perform a pixel-to-pixel attention mechanism in updating the representation of the occluded road lines or markings. Experiments on ApolloScape, a large-scale lane mark segmentation dataset, and ApolloScape Night with artificial simulated night-time road conditions, demonstrate that our method outperforms other existing SOTA lane mark segmentation models with less than 9\% of their parameters and computational complexity. We show that exploiting available camera intrinsic data and ground plane assumption for cross-frame correspondence can lead to a light-weight network with significantly improved performances in speed and accuracy. We also prove the versatility of our HomoFusion approach by applying it to the problem of water puddle segmentation and achieving SOTA performance.

Title: Simba: Mamba augmented U-ShiftGCN for Skeletal Action Recognition in Videos

• Authors: Soumyabrata Chaudhuri, Saumik Bhattacharya
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07645
• Pdf URL: https://arxiv.org/pdf/2404.07645
• Copy Paste: [[2404.07645]] Simba: Mamba augmented U-ShiftGCN for Skeletal Action Recognition in Videos(https://arxiv.org/abs/2404.07645)
• Keywords: transformer
• Abstract: Skeleton Action Recognition (SAR) involves identifying human actions using skeletal joint coordinates and their interconnections. While plain Transformers have been attempted for this task, they still fall short compared to the current leading methods, which are rooted in Graph Convolutional Networks (GCNs) due to the absence of structural priors. Recently, a novel selective state space model, Mamba, has surfaced as a compelling alternative to the attention mechanism in Transformers, offering efficient modeling of long sequences. In this work, to the utmost extent of our awareness, we present the first SAR framework incorporating Mamba. Each fundamental block of our model adopts a novel U-ShiftGCN architecture with Mamba as its core component. The encoder segment of the U-ShiftGCN is devised to extract spatial features from the skeletal data using downsampling vanilla Shift S-GCN blocks. These spatial features then undergo intermediate temporal modeling facilitated by the Mamba block before progressing to the encoder section, which comprises vanilla upsampling Shift S-GCN blocks. Additionally, a Shift T-GCN (ShiftTCN) temporal modeling unit is employed before the exit of each fundamental block to refine temporal representations. This particular integration of downsampling spatial, intermediate temporal, upsampling spatial, and ultimate temporal subunits yields promising results for skeleton action recognition. We dub the resulting model \textbf{Simba}, which attains state-of-the-art performance across three well-known benchmark skeleton action recognition datasets: NTU RGB+D, NTU RGB+D 120, and Northwestern-UCLA. Interestingly, U-ShiftGCN (Simba without Intermediate Mamba Block) by itself is capable of performing reasonably well and surpasses our baseline.

Title: rollama: An R package for using generative large language models through Ollama

• Authors: Johannes B. Gruber, Maximilian Weber
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07654
• Pdf URL: https://arxiv.org/pdf/2404.07654
• Copy Paste: [[2404.07654]] rollama: An R package for using generative large language models through Ollama(https://arxiv.org/abs/2404.07654)
• Keywords: generative, large language model
• Abstract: rollama is an R package that wraps the Ollama API, which allows you to run different Generative Large Language Models (GLLM) locally. The package and learning material focus on making it easy to use Ollama for annotating textual or imagine data with open-source models as well as use these models for document embedding. But users can use or extend rollama to do essentially anything else that is possible through OpenAI's API, yet more private, reproducible and for free.

Title: Finding Dino: A plug-and-play framework for unsupervised detection of out-of-distribution objects using prototypes

• Authors: Poulami Sinhamahapatra, Franziska Schwaiger, Shirsha Bose, Huiyu Wang, Karsten Roscher, Stephan Guennemann
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07664
• Pdf URL: https://arxiv.org/pdf/2404.07664
• Copy Paste: [[2404.07664]] Finding Dino: A plug-and-play framework for unsupervised detection of out-of-distribution objects using prototypes(https://arxiv.org/abs/2404.07664)
• Keywords: segmentation
• Abstract: Detecting and localising unknown or Out-of-distribution (OOD) objects in any scene can be a challenging task in vision. Particularly, in safety-critical cases involving autonomous systems like automated vehicles or trains. Supervised anomaly segmentation or open-world object detection models depend on training on exhaustively annotated datasets for every domain and still struggle in distinguishing between background and OOD objects. In this work, we present a plug-and-play generalised framework - PRototype-based zero-shot OOD detection Without Labels (PROWL). It is an inference-based method that does not require training on the domain dataset and relies on extracting relevant features from self-supervised pre-trained models. PROWL can be easily adapted to detect OOD objects in any operational design domain by specifying a list of known classes from this domain. PROWL, as an unsupervised method, outperforms other supervised methods trained without auxiliary OOD data on the RoadAnomaly and RoadObstacle datasets provided in SegmentMeIfYouCan (SMIYC) benchmark. We also demonstrate its suitability for other domains such as rail and maritime scenes.

Title: Dealing with Subject Similarity in Differential Morphing Attack Detection

• Authors: Nicolò Di Domenico, Guido Borghi, Annalisa Franco, Davide Maltoni
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07667
• Pdf URL: https://arxiv.org/pdf/2404.07667
• Copy Paste: [[2404.07667]] Dealing with Subject Similarity in Differential Morphing Attack Detection(https://arxiv.org/abs/2404.07667)
• Keywords: security, attack, robust
• Abstract: The advent of morphing attacks has posed significant security concerns for automated Face Recognition systems, raising the pressing need for robust and effective Morphing Attack Detection (MAD) methods able to effectively address this issue. In this paper, we focus on Differential MAD (D-MAD), where a trusted live capture, usually representing the criminal, is compared with the document image to classify it as morphed or bona fide. We show these approaches based on identity features are effective when the morphed image and the live one are sufficiently diverse; unfortunately, the effectiveness is significantly reduced when the same approaches are applied to look-alike subjects or in all those cases when the similarity between the two compared images is high (e.g. comparison between the morphed image and the accomplice). Therefore, in this paper, we propose ACIdA, a modular D-MAD system, consisting of a module for the attempt type classification, and two modules for the identity and artifacts analysis on input images. Successfully addressing this task would allow broadening the D-MAD applications including, for instance, the document enrollment stage, which currently relies entirely on human evaluation, thus limiting the possibility of releasing ID documents with manipulated images, as well as the automated gates to detect both accomplices and criminals. An extensive cross-dataset experimental evaluation conducted on the introduced scenario shows that ACIdA achieves state-of-the-art results, outperforming literature competitors, while maintaining good performance in traditional D-MAD benchmarks.

Title: Deep learning-driven pulmonary arteries and veins segmentation reveals demography-associated pulmonary vasculature anatomy

• Authors: Yuetan Chu, Gongning Luo, Longxi Zhou, Shaodong Cao, Guolin Ma, Xianglin Meng, Juexiao Zhou, Changchun Yang, Dexuan Xie, Ricardo Henao, Xigang Xiao, Lianming Wu, Zhaowen Qiu, Xin Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07671
• Pdf URL: https://arxiv.org/pdf/2404.07671
• Copy Paste: [[2404.07671]] Deep learning-driven pulmonary arteries and veins segmentation reveals demography-associated pulmonary vasculature anatomy(https://arxiv.org/abs/2404.07671)
• Keywords: segmentation
• Abstract: Pulmonary artery-vein segmentation is crucial for diagnosing pulmonary diseases and surgical planning, and is traditionally achieved by Computed Tomography Pulmonary Angiography (CTPA). However, concerns regarding adverse health effects from contrast agents used in CTPA have constrained its clinical utility. In contrast, identifying arteries and veins using non-contrast CT, a conventional and low-cost clinical examination routine, has long been considered impossible. Here we propose a High-abundant Pulmonary Artery-vein Segmentation (HiPaS) framework achieving accurate artery-vein segmentation on both non-contrast CT and CTPA across various spatial resolutions. HiPaS first performs spatial normalization on raw CT scans via a super-resolution module, and then iteratively achieves segmentation results at different branch levels by utilizing the low-level vessel segmentation as a prior for high-level vessel segmentation. We trained and validated HiPaS on our established multi-centric dataset comprising 1,073 CT volumes with meticulous manual annotation. Both quantitative experiments and clinical evaluation demonstrated the superior performance of HiPaS, achieving a dice score of 91.8% and a sensitivity of 98.0%. Further experiments demonstrated the non-inferiority of HiPaS segmentation on non-contrast CT compared to segmentation on CTPA. Employing HiPaS, we have conducted an anatomical study of pulmonary vasculature on 10,613 participants in China (five sites), discovering a new association between pulmonary vessel abundance and sex and age: vessel abundance is significantly higher in females than in males, and slightly decreases with age, under the controlling of lung volumes (p < 0.0001). HiPaS realizing accurate artery-vein segmentation delineates a promising avenue for clinical diagnosis and understanding pulmonary physiology in a non-invasive manner.

Title: Opportunistic Sensor-Based Multi-Factor Authentication in and for the Internet of Things

• Authors: Marc Saideh, Jean-Paul Jamont, Laurent Vercouter
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07675
• Pdf URL: https://arxiv.org/pdf/2404.07675
• Copy Paste: [[2404.07675]] Opportunistic Sensor-Based Multi-Factor Authentication in and for the Internet of Things(https://arxiv.org/abs/2404.07675)
• Keywords: secure, security
• Abstract: Communication between connected objects often requires secure and reliable authentication mechanisms. These mechanisms are essential for verifying the identities of objects and preventing unauthorized access. The IoT offers several advantages and opportunities that are not necessarily found in other domains. For instance, IoT sensors collect real-time data about their environment and other objects which contain valuable information that, if used, can reinforce authentication. In this paper, we propose a novel idea for building opportunistic sensor-based authentication factors between IoT objects by leveraging the sensors already present in the systems where they interact. We claim that sensors can be utilized to build factors that reinforce object-to-object authentication mechanisms. Through the integration of these opportunistic sensor-based authentication factors into multi-factor authentication mechanisms, authentication in IoT can achieve a higher level of security. We provide illustrative experiments on two types of vehicles : mobile robots and cars.

Title: ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs

• Authors: Lei Sun, Zhengwei Tao, Youdi Li, Hiroshi Arakawa
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07677
• Pdf URL: https://arxiv.org/pdf/2404.07677
• Copy Paste: [[2404.07677]] ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs(https://arxiv.org/abs/2404.07677)
• Keywords: large language model
• Abstract: The integration of Large Language Models (LLMs) and knowledge graphs (KGs) has achieved remarkable success in various natural language processing tasks. However, existing methodologies that integrate LLMs and KGs often navigate the task-solving process solely based on the LLM's analysis of the question, overlooking the rich cognitive potential inherent in the vast knowledge encapsulated in KGs. To address this, we introduce Observation-Driven Agent (ODA), a novel AI agent framework tailored for tasks involving KGs. ODA incorporates KG reasoning abilities via global observation that enhances reasoning capabilities through a cyclical paradigm of observation, action, and reflection. Confronting the exponential explosion of knowledge during observation, we innovatively design a recursive observation mechanism. Subsequently, we integrate the observed knowledge into the action and reflection modules. Through extensive experiments, ODA demonstrates state-of-the-art performance on several datasets, notably achieving accuracy improvements of 12.87% and 8.9%.

Title: Depth Estimation using Weighted-loss and Transfer Learning

• Authors: Muhammad Adeel Hafeez, Michael G. Madden, Ganesh Sistu, Ihsan Ullah
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07686
• Pdf URL: https://arxiv.org/pdf/2404.07686
• Copy Paste: [[2404.07686]] Depth Estimation using Weighted-loss and Transfer Learning(https://arxiv.org/abs/2404.07686)
• Keywords: robust
• Abstract: Depth estimation from 2D images is a common computer vision task that has applications in many fields including autonomous vehicles, scene understanding and robotics. The accuracy of a supervised depth estimation method mainly relies on the chosen loss function, the model architecture, quality of data and performance metrics. In this study, we propose a simplified and adaptable approach to improve depth estimation accuracy using transfer learning and an optimized loss function. The optimized loss function is a combination of weighted losses to which enhance robustness and generalization: Mean Absolute Error (MAE), Edge Loss and Structural Similarity Index (SSIM). We use a grid search and a random search method to find optimized weights for the losses, which leads to an improved model. We explore multiple encoder-decoder-based models including DenseNet121, DenseNet169, DenseNet201, and EfficientNet for the supervised depth estimation model on NYU Depth Dataset v2. We observe that the EfficientNet model, pre-trained on ImageNet for classification when used as an encoder, with a simple upsampling decoder, gives the best results in terms of RSME, REL and log10: 0.386, 0.113 and 0.049, respectively. We also perform a qualitative analysis which illustrates that our model produces depth maps that closely resemble ground truth, even in cases where the ground truth is flawed. The results indicate significant improvements in accuracy and robustness, with EfficientNet being the most successful architecture.

Title: Chaos in Motion: Unveiling Robustness in Remote Heart Rate Measurement through Brain-Inspired Skin Tracking

• Authors: Jie Wang, Jing Lian, Minjie Ma, Junqiang Lei, Chunbiao Li, Bin Li, Jizhao Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07687
• Pdf URL: https://arxiv.org/pdf/2404.07687
• Copy Paste: [[2404.07687]] Chaos in Motion: Unveiling Robustness in Remote Heart Rate Measurement through Brain-Inspired Skin Tracking(https://arxiv.org/abs/2404.07687)
• Keywords: privacy, protect, robust, extraction
• Abstract: Heart rate is an important physiological indicator of human health status. Existing remote heart rate measurement methods typically involve facial detection followed by signal extraction from the region of interest (ROI). These SOTA methods have three serious problems: (a) inaccuracies even failures in detection caused by environmental influences or subject movement; (b) failures for special patients such as infants and burn victims; (c) privacy leakage issues resulting from collecting face video. To address these issues, we regard the remote heart rate measurement as the process of analyzing the spatiotemporal characteristics of the optical flow signal in the video. We apply chaos theory to computer vision tasks for the first time, thus designing a brain-inspired framework. Firstly, using an artificial primary visual cortex model to extract the skin in the videos, and then calculate heart rate by time-frequency analysis on all pixels. Our method achieves Robust Skin Tracking for Heart Rate measurement, called HR-RST. The experimental results show that HR-RST overcomes the difficulty of environmental influences and effectively tracks the subject movement. Moreover, the method could extend to other body parts. Consequently, the method can be applied to special patients and effectively protect individual privacy, offering an innovative solution.

Title: ViM-UNet: Vision Mamba for Biomedical Segmentation

• Authors: Anwai Archit, Constantin Pape
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07705
• Pdf URL: https://arxiv.org/pdf/2404.07705
• Copy Paste: [[2404.07705]] ViM-UNet: Vision Mamba for Biomedical Segmentation(https://arxiv.org/abs/2404.07705)
• Keywords: transformer, segmentation
• Abstract: CNNs, most notably the UNet, are the default architecture for biomedical segmentation. Transformer-based approaches, such as UNETR, have been proposed to replace them, benefiting from a global field of view, but suffering from larger runtimes and higher parameter counts. The recent Vision Mamba architecture offers a compelling alternative to transformers, also providing a global field of view, but at higher efficiency. Here, we introduce ViM-UNet, a novel segmentation architecture based on it and compare it to UNet and UNETR for two challenging microscopy instance segmentation tasks. We find that it performs similarly or better than UNet, depending on the task, and outperforms UNETR while being more efficient. Our code is open source and documented at https://github.com/constantinpape/torch-em/blob/main/vimunet.md.

Title: OpenTrench3D: A Photogrammetric 3D Point Cloud Dataset for Semantic Segmentation of Underground Utilities

• Authors: Lasse H. Hansen, Simon B. Jensen, Mark P. Philipsen, Andreas Møgelmose, Lars Bodum, Thomas B. Moeslund
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07711
• Pdf URL: https://arxiv.org/pdf/2404.07711
• Copy Paste: [[2404.07711]] OpenTrench3D: A Photogrammetric 3D Point Cloud Dataset for Semantic Segmentation of Underground Utilities(https://arxiv.org/abs/2404.07711)
• Keywords: segmentation
• Abstract: Identifying and classifying underground utilities is an important task for efficient and effective urban planning and infrastructure maintenance. We present OpenTrench3D, a novel and comprehensive 3D Semantic Segmentation point cloud dataset, designed to advance research and development in underground utility surveying and mapping. OpenTrench3D covers a completely novel domain for public 3D point cloud datasets and is unique in its focus, scope, and cost-effective capturing method. The dataset consists of 310 point clouds collected across 7 distinct areas. These include 5 water utility areas and 2 district heating utility areas. The inclusion of different geographical areas and main utilities (water and district heating utilities) makes OpenTrench3D particularly valuable for inter-domain transfer learning experiments. We provide benchmark results for the dataset using three state-of-the-art semantic segmentation models, PointNeXt, PointVector and PointMetaBase. Benchmarks are conducted by training on data from water areas, fine-tuning on district heating area 1 and evaluating on district heating area 2. The dataset is publicly available. With OpenTrench3D, we seek to foster innovation and progress in the field of 3D semantic segmentation in applications related to detection and documentation of underground utilities as well as in transfer learning methods in general.

Title: Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning

• Authors: Shiming Chen, Wenjin Hou, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07713
• Pdf URL: https://arxiv.org/pdf/2404.07713
• Copy Paste: [[2404.07713]] Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning(https://arxiv.org/abs/2404.07713)
• Keywords: transformer
• Abstract: Zero-shot learning (ZSL) recognizes the unseen classes by conducting visual-semantic interactions to transfer semantic knowledge from seen classes to unseen ones, supported by semantic information (e.g., attributes). However, existing ZSL methods simply extract visual features using a pre-trained network backbone (i.e., CNN or ViT), which fail to learn matched visual-semantic correspondences for representing semantic-related visual features as lacking of the guidance of semantic information, resulting in undesirable visual-semantic interactions. To tackle this issue, we propose a progressive semantic-guided vision transformer for zero-shot learning (dubbed ZSLViT). ZSLViT mainly considers two properties in the whole network: i) discover the semantic-related visual representations explicitly, and ii) discard the semantic-unrelated visual information. Specifically, we first introduce semantic-embedded token learning to improve the visual-semantic correspondences via semantic enhancement and discover the semantic-related visual tokens explicitly with semantic-guided token attention. Then, we fuse low semantic-visual correspondence visual tokens to discard the semantic-unrelated visual information for visual enhancement. These two operations are integrated into various encoders to progressively learn semantic-related visual representations for accurate visual-semantic interactions in ZSL. The extensive experiments show that our ZSLViT achieves significant performance gains on three popular benchmark datasets, i.e., CUB, SUN, and AWA2.

Title: Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models

• Authors: Andreas Säuberli, Simon Clematide
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07720
• Pdf URL: https://arxiv.org/pdf/2404.07720
• Copy Paste: [[2404.07720]] Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models(https://arxiv.org/abs/2404.07720)
• Keywords: large language model
• Abstract: Reading comprehension tests are used in a variety of applications, reaching from education to assessing the comprehensibility of simplified texts. However, creating such tests manually and ensuring their quality is difficult and time-consuming. In this paper, we explore how large language models (LLMs) can be used to generate and evaluate multiple-choice reading comprehension items. To this end, we compiled a dataset of German reading comprehension items and developed a new protocol for human and automatic evaluation, including a metric we call text informativity, which is based on guessability and answerability. We then used this protocol and the dataset to evaluate the quality of items generated by Llama 2 and GPT-4. Our results suggest that both models are capable of generating items of acceptable quality in a zero-shot setting, but GPT-4 clearly outperforms Llama 2. We also show that LLMs can be used for automatic evaluation by eliciting item reponses from them. In this scenario, evaluation results with GPT-4 were the most similar to human annotators. Overall, zero-shot generation with LLMs is a promising approach for generating and evaluating reading comprehension test items, in particular for languages without large amounts of available data.

Title: Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models

• Authors: Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, Jaakko Lehtinen
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.07724
• Pdf URL: https://arxiv.org/pdf/2404.07724
• Copy Paste: [[2404.07724]] Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models(https://arxiv.org/abs/2404.07724)
• Keywords: diffusion
• Abstract: Guidance is a crucial technique for extracting the best performance out of image-generating diffusion models. Traditionally, a constant guidance weight has been applied throughout the sampling chain of an image. We show that guidance is clearly harmful toward the beginning of the chain (high noise levels), largely unnecessary toward the end (low noise levels), and only beneficial in the middle. We thus restrict it to a specific range of noise levels, improving both the inference speed and result quality. This limited guidance interval improves the record FID in ImageNet-512 significantly, from 1.81 to 1.40. We show that it is quantitatively and qualitatively beneficial across different sampler parameters, network architectures, and datasets, including the large-scale setting of Stable Diffusion XL. We thus suggest exposing the guidance interval as a hyperparameter in all diffusion models that use guidance.

Title: Realistic Continual Learning Approach using Pre-trained Models

• Authors: Nadia Nasri, Carlos Gutiérrez-Álvarez, Sergio Lafuente-Arroyo, Saturnino Maldonado-Bascón, Roberto J. López-Sastre
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07729
• Pdf URL: https://arxiv.org/pdf/2404.07729
• Copy Paste: [[2404.07729]] Realistic Continual Learning Approach using Pre-trained Models(https://arxiv.org/abs/2404.07729)
• Keywords: robust
• Abstract: Continual learning (CL) is crucial for evaluating adaptability in learning solutions to retain knowledge. Our research addresses the challenge of catastrophic forgetting, where models lose proficiency in previously learned tasks as they acquire new ones. While numerous solutions have been proposed, existing experimental setups often rely on idealized class-incremental learning scenarios. We introduce Realistic Continual Learning (RealCL), a novel CL paradigm where class distributions across tasks are random, departing from structured setups. We also present CLARE (Continual Learning Approach with pRE-trained models for RealCL scenarios), a pre-trained model-based solution designed to integrate new knowledge while preserving past learning. Our contributions include pioneering RealCL as a generalization of traditional CL setups, proposing CLARE as an adaptable approach for RealCL tasks, and conducting extensive experiments demonstrating its effectiveness across various RealCL scenarios. Notably, CLARE outperforms existing models on RealCL benchmarks, highlighting its versatility and robustness in unpredictable learning environments.

Title: ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models

• Authors: Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, Sung Ju Hwang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07738
• Pdf URL: https://arxiv.org/pdf/2404.07738
• Copy Paste: [[2404.07738]] ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models(https://arxiv.org/abs/2404.07738)
• Keywords: large language model
• Abstract: Scientific Research, vital for improving human life, is hindered by its inherent complexity, slow pace, and the need for specialized experts. To enhance its productivity, we propose a ResearchAgent, a large language model-powered research idea writing agent, which automatically generates problems, methods, and experiment designs while iteratively refining them based on scientific literature. Specifically, starting with a core paper as the primary focus to generate ideas, our ResearchAgent is augmented not only with relevant publications through connecting information over an academic graph but also entities retrieved from an entity-centric knowledge store based on their underlying concepts, mined and shared across numerous papers. In addition, mirroring the human approach to iteratively improving ideas with peer discussions, we leverage multiple ReviewingAgents that provide reviews and feedback iteratively. Further, they are instantiated with human preference-aligned large language models whose criteria for evaluation are derived from actual human judgments. We experimentally validate our ResearchAgent on scientific publications across multiple disciplines, showcasing its effectiveness in generating novel, clear, and valid research ideas based on human and model-based evaluation results.

Title: Exploiting Object-based and Segmentation-based Semantic Features for Deep Learning-based Indoor Scene Classification

• Authors: Ricardo Pereira, Luís Garrote, Tiago Barros, Ana Lopes, Urbano J. Nunes
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07739
• Pdf URL: https://arxiv.org/pdf/2404.07739
• Copy Paste: [[2404.07739]] Exploiting Object-based and Segmentation-based Semantic Features for Deep Learning-based Indoor Scene Classification(https://arxiv.org/abs/2404.07739)
• Keywords: segmentation
• Abstract: Indoor scenes are usually characterized by scattered objects and their relationships, which turns the indoor scene classification task into a challenging computer vision task. Despite the significant performance boost in classification tasks achieved in recent years, provided by the use of deep-learning-based methods, limitations such as inter-category ambiguity and intra-category variation have been holding back their performance. To overcome such issues, gathering semantic information has been shown to be a promising source of information towards a more complete and discriminative feature representation of indoor scenes. Therefore, the work described in this paper uses both semantic information, obtained from object detection, and semantic segmentation techniques. While object detection techniques provide the 2D location of objects allowing to obtain spatial distributions between objects, semantic segmentation techniques provide pixel-level information that allows to obtain, at a pixel-level, a spatial distribution and shape-related features of the segmentation categories. Hence, a novel approach that uses a semantic segmentation mask to provide Hu-moments-based segmentation categories' shape characterization, designated by Segmentation-based Hu-Moments Features (SHMFs), is proposed. Moreover, a three-main-branch network, designated by GOS$^2$F$^2$App, that exploits deep-learning-based global features, object-based features, and semantic segmentation-based features is also proposed. GOS$^2$F$^2$App was evaluated in two indoor scene benchmark datasets: SUN RGB-D and NYU Depth V2, where, to the best of our knowledge, state-of-the-art results were achieved on both datasets, which present evidences of the effectiveness of the proposed approach.

Title: 3D-CSAD: Untrained 3D Anomaly Detection for Complex Manufacturing Surfaces

• Authors: Xuanming Cao, Chengyu Tao, Juan Du
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07748
• Pdf URL: https://arxiv.org/pdf/2404.07748
• Copy Paste: [[2404.07748]] 3D-CSAD: Untrained 3D Anomaly Detection for Complex Manufacturing Surfaces(https://arxiv.org/abs/2404.07748)
• Keywords: robust, segmentation
• Abstract: The surface quality inspection of manufacturing parts based on 3D point cloud data has attracted increasing attention in recent years. The reason is that the 3D point cloud can capture the entire surface of manufacturing parts, unlike the previous practices that focus on some key product characteristics. However, achieving accurate 3D anomaly detection is challenging, due to the complex surfaces of manufacturing parts and the difficulty of collecting sufficient anomaly samples. To address these challenges, we propose a novel untrained anomaly detection method based on 3D point cloud data for complex manufacturing parts, which can achieve accurate anomaly detection in a single sample without training data. In the proposed framework, we transform an input sample into two sets of profiles along different directions. Based on one set of the profiles, a novel segmentation module is devised to segment the complex surface into multiple basic and simple components. In each component, another set of profiles, which have the nature of similar shapes, can be modeled as a low-rank matrix. Thus, accurate 3D anomaly detection can be achieved by using Robust Principal Component Analysis (RPCA) on these low-rank matrices. Extensive numerical experiments on different types of parts show that our method achieves promising results compared with the benchmark methods.

Title: Generating Synthetic Satellite Imagery With Deep-Learning Text-to-Image Models -- Technical Challenges and Implications for Monitoring and Verification

• Authors: Tuong Vy Nguyen, Alexander Glaser, Felix Biessmann
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07754
• Pdf URL: https://arxiv.org/pdf/2404.07754
• Copy Paste: [[2404.07754]] Generating Synthetic Satellite Imagery With Deep-Learning Text-to-Image Models -- Technical Challenges and Implications for Monitoring and Verification(https://arxiv.org/abs/2404.07754)
• Keywords: diffusion
• Abstract: Novel deep-learning (DL) architectures have reached a level where they can generate digital media, including photorealistic images, that are difficult to distinguish from real data. These technologies have already been used to generate training data for Machine Learning (ML) models, and large text-to-image models like DALL-E 2, Imagen, and Stable Diffusion are achieving remarkable results in realistic high-resolution image generation. Given these developments, issues of data authentication in monitoring and verification deserve a careful and systematic analysis: How realistic are synthetic images? How easily can they be generated? How useful are they for ML researchers, and what is their potential for Open Science? In this work, we use novel DL models to explore how synthetic satellite images can be created using conditioning mechanisms. We investigate the challenges of synthetic satellite image generation and evaluate the results based on authenticity and state-of-the-art metrics. Furthermore, we investigate how synthetic data can alleviate the lack of data in the context of ML methods for remote-sensing. Finally we discuss implications of synthetic satellite imagery in the context of monitoring and verification.

Title: AnnoCTR: A Dataset for Detecting and Linking Entities, Tactics, and Techniques in Cyber Threat Reports

• Authors: Lukas Lange, Marc Müller, Ghazaleh Haratinezhad Torbati, Dragan Milchevski, Patrick Grau, Subhash Pujari, Annemarie Friedrich
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07765
• Pdf URL: https://arxiv.org/pdf/2404.07765
• Copy Paste: [[2404.07765]] AnnoCTR: A Dataset for Detecting and Linking Entities, Tactics, and Techniques in Cyber Threat Reports(https://arxiv.org/abs/2404.07765)
• Keywords: security, attack
• Abstract: Monitoring the threat landscape to be aware of actual or potential attacks is of utmost importance to cybersecurity professionals. Information about cyber threats is typically distributed using natural language reports. Natural language processing can help with managing this large amount of unstructured information, yet to date, the topic has received little attention. With this paper, we present AnnoCTR, a new CC-BY-SA-licensed dataset of cyber threat reports. The reports have been annotated by a domain expert with named entities, temporal expressions, and cybersecurity-specific concepts including implicitly mentioned techniques and tactics. Entities and concepts are linked to Wikipedia and the MITRE ATT&CK knowledge base, the most widely-used taxonomy for classifying types of attacks. Prior datasets linking to MITRE ATT&CK either provide a single label per document or annotate sentences out-of-context; our dataset annotates entire documents in a much finer-grained way. In an experimental study, we model the annotations of our dataset using state-of-the-art neural models. In our few-shot scenario, we find that for identifying the MITRE ATT&CK concepts that are mentioned explicitly or implicitly in a text, concept descriptions from MITRE ATT&CK are an effective source for training data augmentation.

Title: RMAFF-PSN: A Residual Multi-Scale Attention Feature Fusion Photometric Stereo Network

• Authors: Kai Luo, Yakun Ju, Lin Qi, Kaixuan Wang, Junyu Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07766
• Pdf URL: https://arxiv.org/pdf/2404.07766
• Copy Paste: [[2404.07766]] RMAFF-PSN: A Residual Multi-Scale Attention Feature Fusion Photometric Stereo Network(https://arxiv.org/abs/2404.07766)
• Keywords: extraction
• Abstract: Predicting accurate normal maps of objects from two-dimensional images in regions of complex structure and spatial material variations is challenging using photometric stereo methods due to the influence of surface reflection properties caused by variations in object geometry and surface materials. To address this issue, we propose a photometric stereo network called a RMAFF-PSN that uses residual multiscale attentional feature fusion to handle the ``difficult'' regions of the object. Unlike previous approaches that only use stacked convolutional layers to extract deep features from the input image, our method integrates feature information from different resolution stages and scales of the image. This approach preserves more physical information, such as texture and geometry of the object in complex regions, through shallow-deep stage feature extraction, double branching enhancement, and attention optimization. To test the network structure under real-world conditions, we propose a new real dataset called Simple PS data, which contains multiple objects with varying structures and materials. Experimental results on a publicly available benchmark dataset demonstrate that our method outperforms most existing calibrated photometric stereo methods for the same number of input images, especially in the case of highly non-convex object structures. Our method also obtains good results under sparse lighting conditions.

Title: Joint Conditional Diffusion Model for Image Restoration with Mixed Degradations

• Authors: Yufeng Yue, Meng Yu, Luojie Yang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07770
• Pdf URL: https://arxiv.org/pdf/2404.07770
• Copy Paste: [[2404.07770]] Joint Conditional Diffusion Model for Image Restoration with Mixed Degradations(https://arxiv.org/abs/2404.07770)
• Keywords: diffusion, generative
• Abstract: Image restoration is rather challenging in adverse weather conditions, especially when multiple degradations occur simultaneously. Blind image decomposition was proposed to tackle this issue, however, its effectiveness heavily relies on the accurate estimation of each component. Although diffusion-based models exhibit strong generative abilities in image restoration tasks, they may generate irrelevant contents when the degraded images are severely corrupted. To address these issues, we leverage physical constraints to guide the whole restoration process, where a mixed degradation model based on atmosphere scattering model is constructed. Then we formulate our Joint Conditional Diffusion Model (JCDM) by incorporating the degraded image and degradation mask to provide precise guidance. To achieve better color and detail recovery results, we further integrate a refinement network to reconstruct the restored image, where Uncertainty Estimation Block (UEB) is employed to enhance the features. Extensive experiments performed on both multi-weather and weather-specific datasets demonstrate the superiority of our method over state-of-the-art competing methods.

Title: An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization

• Authors: Minshuo Chen, Song Mei, Jianqing Fan, Mengdi Wang
• Subjects: cs.LG, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.07771
• Pdf URL: https://arxiv.org/pdf/2404.07771
• Copy Paste: [[2404.07771]] An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization(https://arxiv.org/abs/2404.07771)
• Keywords: diffusion, generative
• Abstract: Diffusion models, a powerful and universal generative AI technology, have achieved tremendous success in computer vision, audio, reinforcement learning, and computational biology. In these applications, diffusion models provide flexible high-dimensional data modeling, and act as a sampler for generating new samples under active guidance towards task-desired properties. Despite the significant empirical success, theory of diffusion models is very limited, potentially slowing down principled methodological innovations for further harnessing and improving diffusion models. In this paper, we review emerging applications of diffusion models, understanding their sample generation under various controls. Next, we overview the existing theories of diffusion models, covering their statistical properties and sampling capabilities. We adopt a progressive routine, beginning with unconditional diffusion models and connecting to conditional counterparts. Further, we review a new avenue in high-dimensional structured optimization through conditional diffusion models, where searching for solutions is reformulated as a conditional sampling problem and solved by diffusion models. Lastly, we discuss future directions about diffusion models. The purpose of this paper is to provide a well-rounded theoretical exposure for stimulating forward-looking theories and methods of diffusion models.

Title: ConsistencyDet: Robust Object Detector with Denoising Paradigm of Consistency Model

• Authors: Lifan Jiang, Zhihui Wang, Changmiao Wang, Ming Li, Jiaxu Leng, Xindong Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07773
• Pdf URL: https://arxiv.org/pdf/2404.07773
• Copy Paste: [[2404.07773]] ConsistencyDet: Robust Object Detector with Denoising Paradigm of Consistency Model(https://arxiv.org/abs/2404.07773)
• Keywords: robust, diffusion, generative
• Abstract: Object detection, a quintessential task in the realm of perceptual computing, can be tackled using a generative methodology. In the present study, we introduce a novel framework designed to articulate object detection as a denoising diffusion process, which operates on perturbed bounding boxes of annotated entities. This framework, termed ConsistencyDet, leverages an innovative denoising concept known as the Consistency Model. The hallmark of this model is its self-consistency feature, which empowers the model to map distorted information from any temporal stage back to its pristine state, thereby realizing a ``one-step denoising'' mechanism. Such an attribute markedly elevates the operational efficiency of the model, setting it apart from the conventional Diffusion Model. Throughout the training phase, ConsistencyDet initiates the diffusion sequence with noise-infused boxes derived from the ground-truth annotations and conditions the model to perform the denoising task. Subsequently, in the inference stage, the model employs a denoising sampling strategy that commences with bounding boxes randomly sampled from a normal distribution. Through iterative refinement, the model transforms an assortment of arbitrarily generated boxes into the definitive detections. Comprehensive evaluations employing standard benchmarks, such as MS-COCO and LVIS, corroborate that ConsistencyDet surpasses other leading-edge detectors in performance metrics.

Title: Discourse-Aware In-Context Learning for Temporal Expression Normalization

• Authors: Akash Kumar Gautam, Lukas Lange, Jannik Strötgen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07775
• Pdf URL: https://arxiv.org/pdf/2404.07775
• Copy Paste: [[2404.07775]] Discourse-Aware In-Context Learning for Temporal Expression Normalization(https://arxiv.org/abs/2404.07775)
• Keywords: large language model
• Abstract: Temporal expression (TE) normalization is a well-studied problem. However, the predominately used rule-based systems are highly restricted to specific settings, and upcoming machine learning approaches suffer from a lack of labeled data. In this work, we explore the feasibility of proprietary and open-source large language models (LLMs) for TE normalization using in-context learning to inject task, document, and example information into the model. We explore various sample selection strategies to retrieve the most relevant set of examples. By using a window-based prompt design approach, we can perform TE normalization across sentences, while leveraging the LLM knowledge without training the model. Our experiments show competitive results to models designed for this task. In particular, our method achieves large performance improvements for non-standard settings by dynamically including relevant examples during inference.

Title: PRAM: Place Recognition Anywhere Model for Efficient Visual Localization

• Authors: Fei Xue, Ignas Budvytis, Roberto Cipolla
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.07785
• Pdf URL: https://arxiv.org/pdf/2404.07785
• Copy Paste: [[2404.07785]] PRAM: Place Recognition Anywhere Model for Efficient Visual Localization(https://arxiv.org/abs/2404.07785)
• Keywords: transformer
• Abstract: Humans localize themselves efficiently in known environments by first recognizing landmarks defined on certain objects and their spatial relationships, and then verifying the location by aligning detailed structures of recognized objects with those in the memory. Inspired by this, we propose the place recognition anywhere model (PRAM) to perform visual localization as efficiently as humans do. PRAM consists of two main components - recognition and registration. In detail, first of all, a self-supervised map-centric landmark definition strategy is adopted, making places in either indoor or outdoor scenes act as unique landmarks. Then, sparse keypoints extracted from images, are utilized as the input to a transformer-based deep neural network for landmark recognition; these keypoints enable PRAM to recognize hundreds of landmarks with high time and memory efficiency. Keypoints along with recognized landmark labels are further used for registration between query images and the 3D landmark map. Different from previous hierarchical methods, PRAM discards global and local descriptors, and reduces over 90% storage. Since PRAM utilizes recognition and landmark-wise verification to replace global reference search and exhaustive matching respectively, it runs 2.4 times faster than prior state-of-the-art approaches. Moreover, PRAM opens new directions for visual localization including multi-modality localization, map-centric feature learning, and hierarchical scene coordinate regression.

Title: VIFNet: An End-to-end Visible-Infrared Fusion Network for Image Dehazing

• Authors: Meng Yu, Te Cui, Haoyang Lu, Yufeng Yue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07790
• Pdf URL: https://arxiv.org/pdf/2404.07790
• Copy Paste: [[2404.07790]] VIFNet: An End-to-end Visible-Infrared Fusion Network for Image Dehazing(https://arxiv.org/abs/2404.07790)
• Keywords: robust, extraction
• Abstract: Image dehazing poses significant challenges in environmental perception. Recent research mainly focus on deep learning-based methods with single modality, while they may result in severe information loss especially in dense-haze scenarios. The infrared image exhibits robustness to the haze, however, existing methods have primarily treated the infrared modality as auxiliary information, failing to fully explore its rich information in dehazing. To address this challenge, the key insight of this study is to design a visible-infrared fusion network for image dehazing. In particular, we propose a multi-scale Deep Structure Feature Extraction (DSFE) module, which incorporates the Channel-Pixel Attention Block (CPAB) to restore more spatial and marginal information within the deep structural features. Additionally, we introduce an inconsistency weighted fusion strategy to merge the two modalities by leveraging the more reliable information. To validate this, we construct a visible-infrared multimodal dataset called AirSim-VID based on the AirSim simulation platform. Extensive experiments performed on challenging real and simulated image datasets demonstrate that VIFNet can outperform many state-of-the-art competing methods. The code and dataset are available at https://github.com/mengyu212/VIFNet_dehazing.

Title: Nostra Domina at EvaLatin 2024: Improving Latin Polarity Detection through Data Augmentation

• Authors: Stephen Bothwell, Abigail Swenor, David Chiang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07792
• Pdf URL: https://arxiv.org/pdf/2404.07792
• Copy Paste: [[2404.07792]] Nostra Domina at EvaLatin 2024: Improving Latin Polarity Detection through Data Augmentation(https://arxiv.org/abs/2404.07792)
• Keywords: large language model
• Abstract: This paper describes submissions from the team Nostra Domina to the EvaLatin 2024 shared task of emotion polarity detection. Given the low-resource environment of Latin and the complexity of sentiment in rhetorical genres like poetry, we augmented the available data through automatic polarity annotation. We present two methods for doing so on the basis of the $k$-means algorithm, and we employ a variety of Latin large language models (LLMs) in a neural architecture to better capture the underlying contextual sentiment representations. Our best approach achieved the second highest macro-averaged Macro-$F_1$ score on the shared task's test set.

Title: DGMamba: Domain Generalization via Generalized State Space Model

• Authors: Shaocong Long, Qianyu Zhou, Xiangtai Li, Xuequan Lu, Chenhao Ying, Yuan Luo, Lizhuang Ma, Shuicheng Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07794
• Pdf URL: https://arxiv.org/pdf/2404.07794
• Copy Paste: [[2404.07794]] DGMamba: Domain Generalization via Generalized State Space Model(https://arxiv.org/abs/2404.07794)
• Keywords: transformer
• Abstract: Domain generalization~(DG) aims at solving distribution shift problems in various scenes. Existing approaches are based on Convolution Neural Networks (CNNs) or Vision Transformers (ViTs), which suffer from limited receptive fields or quadratic complexities issues. Mamba, as an emerging state space model (SSM), possesses superior linear complexity and global receptive fields. Despite this, it can hardly be applied to DG to address distribution shifts, due to the hidden state issues and inappropriate scan mechanisms. In this paper, we propose a novel framework for DG, named DGMamba, that excels in strong generalizability toward unseen domains and meanwhile has the advantages of global receptive fields, and efficient linear complexity. Our DGMamba compromises two core components: Hidden State Suppressing~(HSS) and Semantic-aware Patch refining~(SPR). In particular, HSS is introduced to mitigate the influence of hidden states associated with domain-specific features during output prediction. SPR strives to encourage the model to concentrate more on objects rather than context, consisting of two designs: Prior-Free Scanning~(PFS), and Domain Context Interchange~(DCI). Concretely, PFS aims to shuffle the non-semantic patches within images, creating more flexible and effective sequences from images, and DCI is designed to regularize Mamba with the combination of mismatched non-semantic and semantic information by fusing patches among domains. Extensive experiments on four commonly used DG benchmarks demonstrate that the proposed DGMamba achieves remarkably superior results to state-of-the-art models. The code will be made publicly available.

Title: Voice-Assisted Real-Time Traffic Sign Recognition System Using Convolutional Neural Network

• Authors: Mayura Manawadu, Udaya Wijenayake
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07807
• Pdf URL: https://arxiv.org/pdf/2404.07807
• Copy Paste: [[2404.07807]] Voice-Assisted Real-Time Traffic Sign Recognition System Using Convolutional Neural Network(https://arxiv.org/abs/2404.07807)
• Keywords: robust
• Abstract: Traffic signs are important in communicating information to drivers. Thus, comprehension of traffic signs is essential for road safety and ignorance may result in road accidents. Traffic sign detection has been a research spotlight over the past few decades. Real-time and accurate detections are the preliminaries of robust traffic sign detection system which is yet to be achieved. This study presents a voice-assisted real-time traffic sign recognition system which is capable of assisting drivers. This system functions under two subsystems. Initially, the detection and recognition of the traffic signs are carried out using a trained Convolutional Neural Network (CNN). After recognizing the specific traffic sign, it is narrated to the driver as a voice message using a text-to-speech engine. An efficient CNN model for a benchmark dataset is developed for real-time detection and recognition using Deep Learning techniques. The advantage of this system is that even if the driver misses a traffic sign, or does not look at the traffic sign, or is unable to comprehend the sign, the system detects it and narrates it to the driver. A system of this type is also important in the development of autonomous vehicles.

Title: Post-Hoc Reversal: Are We Selecting Models Prematurely?

• Authors: Rishabh Ranjan, Saurabh Garg, Mrigank Raman, Carlos Guestrin, Zachary Chase Lipton
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.07815
• Pdf URL: https://arxiv.org/pdf/2404.07815
• Copy Paste: [[2404.07815]] Post-Hoc Reversal: Are We Selecting Models Prematurely?(https://arxiv.org/abs/2404.07815)
• Keywords: robust
• Abstract: Trained models are often composed with post-hoc transforms such as temperature scaling (TS), ensembling and stochastic weight averaging (SWA) to improve performance, robustness, uncertainty estimation, etc. However, such transforms are typically applied only after the base models have already been finalized by standard means. In this paper, we challenge this practice with an extensive empirical study. In particular, we demonstrate a phenomenon that we call post-hoc reversal, where performance trends are reversed after applying these post-hoc transforms. This phenomenon is especially prominent in high-noise settings. For example, while base models overfit badly early in training, both conventional ensembling and SWA favor base models trained for more epochs. Post-hoc reversal can also suppress the appearance of double descent and mitigate mismatches between test loss and test error seen in base models. Based on our findings, we propose post-hoc selection, a simple technique whereby post-hoc metrics inform model development decisions such as early stopping, checkpointing, and broader hyperparameter choices. Our experimental analyses span real-world vision, language, tabular and graph datasets from domains like satellite imaging, language modeling, census prediction and social network analysis. On an LLM instruction tuning dataset, post-hoc selection results in > 1.5x MMLU improvement compared to naive selection. Code is available at https://github.com/rishabh-ranjan/post-hoc-reversal.

Title: Sparse Laneformer

• Authors: Ji Liu, Zifeng Zhang, Mingjie Lu, Hongyang Wei, Dong Li, Yile Xie, Jinzhang Peng, Lu Tian, Ashish Sirasao, Emad Barsoum
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07821
• Pdf URL: https://arxiv.org/pdf/2404.07821
• Copy Paste: [[2404.07821]] Sparse Laneformer(https://arxiv.org/abs/2404.07821)
• Keywords: transformer
• Abstract: Lane detection is a fundamental task in autonomous driving, and has achieved great progress as deep learning emerges. Previous anchor-based methods often design dense anchors, which highly depend on the training dataset and remain fixed during inference. We analyze that dense anchors are not necessary for lane detection, and propose a transformer-based lane detection framework based on a sparse anchor mechanism. To this end, we generate sparse anchors with position-aware lane queries and angle queries instead of traditional explicit anchors. We adopt Horizontal Perceptual Attention (HPA) to aggregate the lane features along the horizontal direction, and adopt Lane-Angle Cross Attention (LACA) to perform interactions between lane queries and angle queries. We also propose Lane Perceptual Attention (LPA) based on deformable cross attention to further refine the lane predictions. Our method, named Sparse Laneformer, is easy-to-implement and end-to-end trainable. Extensive experiments demonstrate that Sparse Laneformer performs favorably against the state-of-the-art methods, e.g., surpassing Laneformer by 3.0% F1 score and O2SFormer by 0.7% F1 score with fewer MACs on CULane with the same ResNet-34 backbone.

Title: Protected QR Code-based Anti-counterfeit System for Pharmaceutical Manufacturing

• Authors: Nitol Saha, Md Masruk Aulia, Md. Mostafizur Rahman, Mohammed Shafiul Alam Khan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07831
• Pdf URL: https://arxiv.org/pdf/2404.07831
• Copy Paste: [[2404.07831]] Protected QR Code-based Anti-counterfeit System for Pharmaceutical Manufacturing(https://arxiv.org/abs/2404.07831)
• Keywords: secure, protect
• Abstract: The pharmaceutical manufacturing faces critical challenges due to the global threat of counterfeit drugs. This paper proposes a new approach of protected QR codes to secure unique product information for safeguarding the pharmaceutical supply chain. The proposed solution integrates secure QR code generation and encrypted data transmission to establish a comprehensive anti-counterfeit ecosystem. The protected QR codes encapsulate product information that cannot be identified using traditional QR code scanners which protect the information against replication and tampering. The system is developed with scalability in mind, which can be easily implemented without introducing any additional modification in the traditional supply chain.

Title: Streamlined Photoacoustic Image Processing with Foundation Models: A Training-Free Solution

• Authors: Handi Deng, Yucheng Zhou, Jiaxuan Xiang, Liujie Gu, Yan Luo, Hai Feng, Mingyuan Liu, Cheng Ma
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07833
• Pdf URL: https://arxiv.org/pdf/2404.07833
• Copy Paste: [[2404.07833]] Streamlined Photoacoustic Image Processing with Foundation Models: A Training-Free Solution(https://arxiv.org/abs/2404.07833)
• Keywords: segmentation
• Abstract: Foundation models have rapidly evolved and have achieved significant accomplishments in computer vision tasks. Specifically, the prompt mechanism conveniently allows users to integrate image prior information into the model, making it possible to apply models without any training. Therefore, we propose a method based on foundation models and zero training to solve the tasks of photoacoustic (PA) image segmentation. We employed the segment anything model (SAM) by setting simple prompts and integrating the model's outputs with prior knowledge of the imaged objects to accomplish various tasks, including: (1) removing the skin signal in three-dimensional PA image rendering; (2) dual speed-of-sound reconstruction, and (3) segmentation of finger blood vessels. Through these demonstrations, we have concluded that deep learning can be directly applied in PA imaging without the requirement for network design and training. This potentially allows for a hands-on, convenient approach to achieving efficient and accurate segmentation of PA images. This letter serves as a comprehensive tutorial, facilitating the mastery of the technique through the provision of code and sample datasets.

Title: Question Generation in Knowledge-Driven Dialog: Explainability and Evaluation

• Authors: Juliette Faille, Quentin Brabant, Gwenole Lecorve, Lina M. Rojas-Barahona, Claire Gardent
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07836
• Pdf URL: https://arxiv.org/pdf/2404.07836
• Copy Paste: [[2404.07836]] Question Generation in Knowledge-Driven Dialog: Explainability and Evaluation(https://arxiv.org/abs/2404.07836)
• Keywords: explainability
• Abstract: We explore question generation in the context of knowledge-grounded dialogs focusing on explainability and evaluation. Inspired by previous work on planning-based summarisation, we present a model which instead of directly generating a question, sequentially predicts first a fact then a question. We evaluate our approach on 37k test dialogs adapted from the KGConv dataset and we show that, although more demanding in terms of inference, our approach performs on par with a standard model which solely generates a question while allowing for a detailed referenceless evaluation of the model behaviour in terms of relevance, factuality and pronominalisation.

Title: RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

• Authors: Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Sertan Girgin, Olivier Bachem, Alek Andreev, Kathleen Kenealy, Thomas Mesnard, Cassidy Hardin, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Armand Joulin, Noah Fiedel, Evan Senter, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, David Budden, Arnaud Doucet, Sharad Vikram, Adam Paszke, Trevor Gale, Sebastian Borgeaud, Charlie Chen, Andy Brock, Antonia Paterson, Jenny Brennan, Meg Risdal, Raj Gundluru, Nesh Devanathan, Paul Mooney, Nilay Chauhan, Phil Culliton, Luiz GUStavo Martins, Elisa Bandy, David Huntsperger, et al. (12 additional authors not shown)
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07839
• Pdf URL: https://arxiv.org/pdf/2404.07839
• Copy Paste: [[2404.07839]] RecurrentGemma: Moving Past Transformers for Efficient Open Language Models(https://arxiv.org/abs/2404.07839)
• Keywords: transformer
• Abstract: We introduce RecurrentGemma, an open language model which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide a pre-trained model with 2B non-embedding parameters, and an instruction tuned variant. Both models achieve comparable performance to Gemma-2B despite being trained on fewer tokens.

Title: On Training Data Influence of GPT Models

• Authors: Qingyi Liu, Yekun Chai, Shuohuan Wang, Yu Sun, Keze Wang, Hua Wu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07840
• Pdf URL: https://arxiv.org/pdf/2404.07840
• Copy Paste: [[2404.07840]] On Training Data Influence of GPT Models(https://arxiv.org/abs/2404.07840)
• Keywords: robust, generative
• Abstract: Amidst the rapid advancements in generative language models, the investigation of how training data shapes the performance of GPT models is still emerging. This paper presents GPTfluence, a novel approach that leverages a featurized simulation to assess the impact of training examples on the training dynamics of GPT models. Our approach not only traces the influence of individual training instances on performance trajectories, such as loss and other key metrics, on targeted test points but also enables a comprehensive comparison with existing methods across various training scenarios in GPT models, ranging from 14 million to 2.8 billion parameters, across a range of downstream tasks. Contrary to earlier methods that struggle with generalization to new data, GPTfluence introduces a parameterized simulation of training dynamics, demonstrating robust generalization capabilities to unseen training data. This adaptability is evident across both fine-tuning and instruction-tuning scenarios, spanning tasks in natural language understanding and generation. We will make our code and data publicly available.

Title: TBSN: Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising

• Authors: Junyi Li, Zhilu Zhang, Wangmeng Zuo
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.07846
• Pdf URL: https://arxiv.org/pdf/2404.07846
• Copy Paste: [[2404.07846]] TBSN: Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising(https://arxiv.org/abs/2404.07846)
• Keywords: transformer
• Abstract: Blind-spot networks (BSN) have been prevalent network architectures in self-supervised image denoising (SSID). Existing BSNs are mostly conducted with convolution layers. Although transformers offer potential solutions to the limitations of convolutions and have demonstrated success in various image restoration tasks, their attention mechanisms may violate the blind-spot requirement, thus restricting their applicability in SSID. In this paper, we present a transformer-based blind-spot network (TBSN) by analyzing and redesigning the transformer operators that meet the blind-spot requirement. Specifically, TBSN follows the architectural principles of dilated BSNs, and incorporates spatial as well as channel self-attention layers to enhance the network capability. For spatial self-attention, an elaborate mask is applied to the attention matrix to restrict its receptive field, thus mimicking the dilated convolution. For channel self-attention, we observe that it may leak the blind-spot information when the channel number is greater than spatial size in the deep layers of multi-scale architectures. To eliminate this effect, we divide the channel into several groups and perform channel attention separately. Furthermore, we introduce a knowledge distillation strategy that distills TBSN into smaller denoisers to improve computational efficiency while maintaining performance. Extensive experiments on real-world image denoising datasets show that TBSN largely extends the receptive field and exhibits favorable performance against state-of-the-art SSID methods. The code and pre-trained models will be publicly available at https://github.com/nagejacob/TBSN.

Title: Fuss-Free Network: A Simplified and Efficient Neural Network for Crowd Counting

• Authors: Lei Chen, Xingen Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07847
• Pdf URL: https://arxiv.org/pdf/2404.07847
• Copy Paste: [[2404.07847]] Fuss-Free Network: A Simplified and Efficient Neural Network for Crowd Counting(https://arxiv.org/abs/2404.07847)
• Keywords: robust
• Abstract: In the field of crowd-counting research, many recent deep learning based methods have demonstrated robust capabilities for accurately estimating crowd sizes. However, the enhancement in their performance often arises from an increase in the complexity of the model structure. This paper introduces the Fuss-Free Network (FFNet), a crowd counting deep learning model that is characterized by its simplicity and efficiency in terms of its structure. The model comprises only a backbone of a neural network and a multi-scale feature fusion structure.The multi-scale feature fusion structure is a simple architecture consisting of three branches, each only equipped with a focus transition module, and combines the features from these branches through the concatenation operation.Our proposed crowd counting model is trained and evaluated on four widely used public datasets, and it achieves accuracy that is comparable to that of existing complex models.The experimental results further indicate that excellent performance in crowd counting tasks can also be achieved by utilizing a simple, low-parameter, and computationally efficient neural network structure.

Title: Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations

• Authors: Dayeon Ki, Marine Carpuat
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07851
• Pdf URL: https://arxiv.org/pdf/2404.07851
• Copy Paste: [[2404.07851]] Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations(https://arxiv.org/abs/2404.07851)
• Keywords: large language model
• Abstract: Machine Translation (MT) remains one of the last NLP tasks where large language models (LLMs) have not yet replaced dedicated supervised systems. This work exploits the complementary strengths of LLMs and supervised MT by guiding LLMs to automatically post-edit MT with external feedback on its quality, derived from Multidimensional Quality Metric (MQM) annotations. Working with LLaMA-2 models, we consider prompting strategies varying the nature of feedback provided and then fine-tune the LLM to improve its ability to exploit the provided guidance. Through experiments on Chinese-English, English-German, and English-Russian MQM data, we demonstrate that prompting LLMs to post-edit MT improves TER, BLEU and COMET scores, although the benefits of fine-grained feedback are not clear. Fine-tuning helps integrate fine-grained feedback more effectively and further improves translation quality based on both automatic and human evaluation.

Title: Backdoor Contrastive Learning via Bi-level Trigger Optimization

• Authors: Weiyu Sun, Xinyu Zhang, Hao Lu, Yingcong Chen, Ting Wang, Jinghui Chen, Lu Lin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07863
• Pdf URL: https://arxiv.org/pdf/2404.07863
• Copy Paste: [[2404.07863]] Backdoor Contrastive Learning via Bi-level Trigger Optimization(https://arxiv.org/abs/2404.07863)
• Keywords: defense, attack
• Abstract: Contrastive Learning (CL) has attracted enormous attention due to its remarkable capability in unsupervised representation learning. However, recent works have revealed the vulnerability of CL to backdoor attacks: the feature extractor could be misled to embed backdoored data close to an attack target class, thus fooling the downstream predictor to misclassify it as the target. Existing attacks usually adopt a fixed trigger pattern and poison the training set with trigger-injected data, hoping for the feature extractor to learn the association between trigger and target class. However, we find that such fixed trigger design fails to effectively associate trigger-injected data with target class in the embedding space due to special CL mechanisms, leading to a limited attack success rate (ASR). This phenomenon motivates us to find a better backdoor trigger design tailored for CL framework. In this paper, we propose a bi-level optimization approach to achieve this goal, where the inner optimization simulates the CL dynamics of a surrogate victim, and the outer optimization enforces the backdoor trigger to stay close to the target throughout the surrogate CL procedure. Extensive experiments show that our attack can achieve a higher attack success rate (e.g., $99\%$ ASR on ImageNet-100) with a very low poisoning rate ($1\%$). Besides, our attack can effectively evade existing state-of-the-art defenses. Code is available at: https://github.com/SWY666/SSL-backdoor-BLTO.

Title: LeapFrog: The Rowhammer Instruction Skip Attack

• Authors: Andrew Adiletta, Caner Tol, Berk Sunar
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2404.07878
• Pdf URL: https://arxiv.org/pdf/2404.07878
• Copy Paste: [[2404.07878]] LeapFrog: The Rowhammer Instruction Skip Attack(https://arxiv.org/abs/2404.07878)
• Keywords: security, defense, attack, robust
• Abstract: Since its inception, Rowhammer exploits have rapidly evolved into increasingly sophisticated threats not only compromising data integrity but also the control flow integrity of victim processes. Nevertheless, it remains a challenge for an attacker to identify vulnerable targets (i.e., Rowhammer gadgets), understand the outcome of the attempted fault, and formulate an attack that yields useful results. In this paper, we present a new type of Rowhammer gadget, called a LeapFrog gadget, which, when present in the victim code, allows an adversary to subvert code execution to bypass a critical piece of code (e.g., authentication check logic, encryption rounds, padding in security protocols). The Leapfrog gadget manifests when the victim code stores the Program Counter (PC) value in the user or kernel stack (e.g., a return address during a function call) which, when tampered with, re-positions the return address to a location that bypasses a security-critical code pattern. This research also presents a systematic process to identify Leapfrog gadgets. This methodology enables the automated detection of susceptible targets and the determination of optimal attack parameters. We first showcase this new attack vector through a practical demonstration on a TLS handshake client/server scenario, successfully inducing an instruction skip in a client application. We then demonstrate the attack on real-world code found in the wild, implementing an attack on OpenSSL. Our findings extend the impact of Rowhammer attacks on control flow and contribute to the development of more robust defenses against these increasingly sophisticated threats.

Title: Context-aware Video Anomaly Detection in Long-Term Datasets

• Authors: Zhengye Yang, Richard Radke
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07887
• Pdf URL: https://arxiv.org/pdf/2404.07887
• Copy Paste: [[2404.07887]] Context-aware Video Anomaly Detection in Long-Term Datasets(https://arxiv.org/abs/2404.07887)
• Keywords: security
• Abstract: Video anomaly detection research is generally evaluated on short, isolated benchmark videos only a few minutes long. However, in real-world environments, security cameras observe the same scene for months or years at a time, and the notion of anomalous behavior critically depends on context, such as the time of day, day of week, or schedule of events. Here, we propose a context-aware video anomaly detection algorithm, Trinity, specifically targeted to these scenarios. Trinity is especially well-suited to crowded scenes in which individuals cannot be easily tracked, and anomalies are due to speed, direction, or absence of group motion. Trinity is a contrastive learning framework that aims to learn alignments between context, appearance, and motion, and uses alignment quality to classify videos as normal or anomalous. We evaluate our algorithm on both conventional benchmarks and a public webcam-based dataset we collected that spans more than three months of activity.

Title: A Measurement of Genuine Tor Traces for Realistic Website Fingerprinting

• Authors: Rob Jansen, Ryan Wails, Aaron Johnson
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07892
• Pdf URL: https://arxiv.org/pdf/2404.07892
• Copy Paste: [[2404.07892]] A Measurement of Genuine Tor Traces for Realistic Website Fingerprinting(https://arxiv.org/abs/2404.07892)
• Keywords: privacy, defense, attack
• Abstract: Website fingerprinting (WF) is a dangerous attack on web privacy because it enables an adversary to predict the website a user is visiting, despite the use of encryption, VPNs, or anonymizing networks such as Tor. Previous WF work almost exclusively uses synthetic datasets to evaluate the performance and estimate the feasibility of WF attacks despite evidence that synthetic data misrepresents the real world. In this paper we present GTT23, the first WF dataset of genuine Tor traces, which we obtain through a large-scale measurement of the Tor network. GTT23 represents real Tor user behavior better than any existing WF dataset, is larger than any existing WF dataset by at least an order of magnitude, and will help ground the future study of realistic WF attacks and defenses. In a detailed evaluation, we survey 25 WF datasets published over the last 15 years and compare their characteristics to those of GTT23. We discover common deficiencies of synthetic datasets that make them inferior to GTT23 for drawing meaningful conclusions about the effectiveness of WF attacks directed at real Tor users. We have made GTT23 available to promote reproducible research and to help inspire new directions for future work.

Title: High-Dimension Human Value Representation in Large Language Models

• Authors: Samuel Cahyawijaya, Delong Chen, Yejin Bang, Leila Khalatbari, Bryan Wilie, Ziwei Ji, Etsuko Ishii, Pascale Fung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07900
• Pdf URL: https://arxiv.org/pdf/2404.07900
• Copy Paste: [[2404.07900]] High-Dimension Human Value Representation in Large Language Models(https://arxiv.org/abs/2404.07900)
• Keywords: large language model
• Abstract: The widespread application of Large Language Models (LLMs) across various tasks and fields has necessitated the alignment of these models with human values and preferences. Given various approaches of human value alignment, ranging from Reinforcement Learning with Human Feedback (RLHF), to constitutional learning, etc. there is an urgent need to understand the scope and nature of human values injected into these models before their release. There is also a need for model alignment without a costly large scale human annotation effort. We propose UniVaR, a high-dimensional representation of human value distributions in LLMs, orthogonal to model architecture and training data. Trained from the value-relevant output of eight multilingual LLMs and tested on the output from four multilingual LLMs, namely LlaMA2, ChatGPT, JAIS and Yi, we show that UniVaR is a powerful tool to compare the distribution of human values embedded in different LLMs with different langauge sources. Through UniVaR, we explore how different LLMs prioritize various values in different languages and cultures, shedding light on the complex interplay between human values and language modeling.

Title: HGRN2: Gated Linear RNNs with State Expansion

• Authors: Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, Yiran Zhong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07904
• Pdf URL: https://arxiv.org/pdf/2404.07904
• Copy Paste: [[2404.07904]] HGRN2: Gated Linear RNNs with State Expansion(https://arxiv.org/abs/2404.07904)
• Keywords: transformer
• Abstract: Hierarchically gated linear RNN (HGRN,Qin et al. 2023) has demonstrated competitive training speed and performance in language modeling, while offering efficient inference. However, the recurrent state size of HGRN remains relatively small, which limits its expressiveness.To address this issue, inspired by linear attention, we introduce a simple outer-product-based state expansion mechanism so that the recurrent state size can be significantly enlarged without introducing any additional parameters. The linear attention form also allows for hardware-efficient training.Our extensive experiments verify the advantage of HGRN2 over HGRN1 in language modeling, image classification, and Long Range Arena.Our largest 3B HGRN2 model slightly outperforms Mamba and LLaMa Architecture Transformer for language modeling in a controlled experiment setting; and performs competitively with many open-source 3B models in downstream evaluation while using much fewer total training tokens.

Title: AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs

• Authors: Zeyi Liao, Huan Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07921
• Pdf URL: https://arxiv.org/pdf/2404.07921
• Copy Paste: [[2404.07921]] AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs(https://arxiv.org/abs/2404.07921)
• Keywords: attack, generative, large language model
• Abstract: As large language models (LLMs) become increasingly prevalent and integrated into autonomous systems, ensuring their safety is imperative. Despite significant strides toward safety alignment, recent work GCG~\citep{zou2023universal} proposes a discrete token optimization algorithm and selects the single suffix with the lowest loss to successfully jailbreak aligned LLMs. In this work, we first discuss the drawbacks of solely picking the suffix with the lowest loss during GCG optimization for jailbreaking and uncover the missed successful suffixes during the intermediate steps. Moreover, we utilize those successful suffixes as training data to learn a generative model, named AmpleGCG, which captures the distribution of adversarial suffixes given a harmful query and enables the rapid generation of hundreds of suffixes for any harmful queries in seconds. AmpleGCG achieves near 100\% attack success rate (ASR) on two aligned LLMs (Llama-2-7B-chat and Vicuna-7B), surpassing two strongest attack baselines. More interestingly, AmpleGCG also transfers seamlessly to attack different models, including closed-source LLMs, achieving a 99\% ASR on the latest GPT-3.5. To summarize, our work amplifies the impact of GCG by training a generative model of adversarial suffixes that is universal to any harmful queries and transferable from attacking open-source LLMs to closed-source LLMs. In addition, it can generate 200 adversarial suffixes for one harmful query in only 4 seconds, rendering it more challenging to defend.

Title: LaVy: Vietnamese Multimodal Large Language Model

• Authors: Chi Tran, Huong Le Thanh
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07922
• Pdf URL: https://arxiv.org/pdf/2404.07922
• Copy Paste: [[2404.07922]] LaVy: Vietnamese Multimodal Large Language Model(https://arxiv.org/abs/2404.07922)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) and Multimodal Large language models (MLLMs) have taken the world by storm with impressive abilities in complex reasoning and linguistic comprehension. Meanwhile there are plethora of works related to Vietnamese Large Language Models, the lack of high-quality resources in multimodality limits the progress of Vietnamese MLLMs. In this paper, we pioneer in address this by introducing LaVy, a state-of-the-art Vietnamese MLLM, and we also introduce LaVy-Bench benchmark designated for evaluating MLLMs's understanding on Vietnamese visual language tasks. All code and model weights are public at https://github.com/baochi0212/LaVy

Title: A Parsimonious Setup for Streamflow Forecasting using CNN-LSTM

• Authors: Sudan Pokharel, Tirthankar Roy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07924
• Pdf URL: https://arxiv.org/pdf/2404.07924
• Copy Paste: [[2404.07924]] A Parsimonious Setup for Streamflow Forecasting using CNN-LSTM(https://arxiv.org/abs/2404.07924)
• Keywords: robust
• Abstract: Significant strides have been made in advancing streamflow predictions, notably with the introduction of cutting-edge machine-learning models. Predominantly, Long Short-Term Memories (LSTMs) and Convolution Neural Networks (CNNs) have been widely employed in this domain. While LSTMs are applicable in both rainfall-runoff and time series settings, CNN-LSTMs have primarily been utilized in rainfall-runoff scenarios. In this study, we extend the application of CNN-LSTMs to time series settings, leveraging lagged streamflow data in conjunction with precipitation and temperature data to predict streamflow. Our results show a substantial improvement in predictive performance in 21 out of 32 HUC8 basins in Nebraska, showcasing noteworthy increases in the Kling-Gupta Efficiency (KGE) values. These results highlight the effectiveness of CNN-LSTMs in time series settings, particularly for spatiotemporal hydrological modeling, for more accurate and robust streamflow predictions.

Title: FusionMamba: Efficient Image Fusion with State Space Model

• Authors: Siran Peng, Xiangyu Zhu, Haoyu Deng, Zhen Lei, Liang-Jian Deng
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.07932
• Pdf URL: https://arxiv.org/pdf/2404.07932
• Copy Paste: [[2404.07932]] FusionMamba: Efficient Image Fusion with State Space Model(https://arxiv.org/abs/2404.07932)
• Keywords: transformer
• Abstract: Image fusion aims to generate a high-resolution multi/hyper-spectral image by combining a high-resolution image with limited spectral information and a low-resolution image with abundant spectral data. Current deep learning (DL)-based methods for image fusion primarily rely on CNNs or Transformers to extract features and merge different types of data. While CNNs are efficient, their receptive fields are limited, restricting their capacity to capture global context. Conversely, Transformers excel at learning global information but are hindered by their quadratic complexity. Fortunately, recent advancements in the State Space Model (SSM), particularly Mamba, offer a promising solution to this issue by enabling global awareness with linear complexity. However, there have been few attempts to explore the potential of SSM in information fusion, which is a crucial ability in domains like image fusion. Therefore, we propose FusionMamba, an innovative method for efficient image fusion. Our contributions mainly focus on two aspects. Firstly, recognizing that images from different sources possess distinct properties, we incorporate Mamba blocks into two U-shaped networks, presenting a novel architecture that extracts spatial and spectral features in an efficient, independent, and hierarchical manner. Secondly, to effectively combine spatial and spectral information, we extend the Mamba block to accommodate dual inputs. This expansion leads to the creation of a new module called the FusionMamba block, which outperforms existing fusion techniques such as concatenation and cross-attention. To validate FusionMamba's effectiveness, we conduct a series of experiments on five datasets related to three image fusion tasks. The quantitative and qualitative evaluation results demonstrate that our method achieves state-of-the-art (SOTA) performance, underscoring the superiority of FusionMamba.

Title: Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon

• Authors: Tianshuo Xu, Peng Mi, Ruilin Wang, Yingcong Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07946
• Pdf URL: https://arxiv.org/pdf/2404.07946
• Copy Paste: [[2404.07946]] Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon(https://arxiv.org/abs/2404.07946)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DMs) are a powerful generative framework that have attracted significant attention in recent years. However, the high computational cost of training DMs limits their practical applications. In this paper, we start with a consistency phenomenon of DMs: we observe that DMs with different initializations or even different architectures can produce very similar outputs given the same noise inputs, which is rare in other generative models. We attribute this phenomenon to two factors: (1) the learning difficulty of DMs is lower when the noise-prediction diffusion model approaches the upper bound of the timestep (the input becomes pure noise), where the structural information of the output is usually generated; and (2) the loss landscape of DMs is highly smooth, which implies that the model tends to converge to similar local minima and exhibit similar behavior patterns. This finding not only reveals the stability of DMs, but also inspires us to devise two strategies to accelerate the training of DMs. First, we propose a curriculum learning based timestep schedule, which leverages the noise rate as an explicit indicator of the learning difficulty and gradually reduces the training frequency of easier timesteps, thus improving the training efficiency. Second, we propose a momentum decay strategy, which reduces the momentum coefficient during the optimization process, as the large momentum may hinder the convergence speed and cause oscillations due to the smoothness of the loss landscape. We demonstrate the effectiveness of our proposed strategies on various models and show that they can significantly reduce the training time and improve the quality of the generated images.

Title: Taming Stable Diffusion for Text to 360° Panorama Image Generation

• Authors: Cheng Zhang, Qianyi Wu, Camilo Cruz Gambardella, Xiaoshui Huang, Dinh Phung, Wanli Ouyang, Jianfei Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07949
• Pdf URL: https://arxiv.org/pdf/2404.07949
• Copy Paste: [[2404.07949]] Taming Stable Diffusion for Text to 360° Panorama Image Generation(https://arxiv.org/abs/2404.07949)
• Keywords: diffusion, generative
• Abstract: Generative models, e.g., Stable Diffusion, have enabled the creation of photorealistic images from text prompts. Yet, the generation of 360-degree panorama images from text remains a challenge, particularly due to the dearth of paired text-panorama data and the domain gap between panorama and perspective images. In this paper, we introduce a novel dual-branch diffusion model named PanFusion to generate a 360-degree image from a text prompt. We leverage the stable diffusion model as one branch to provide prior knowledge in natural image generation and register it to another panorama branch for holistic image generation. We propose a unique cross-attention mechanism with projection awareness to minimize distortion during the collaborative denoising process. Our experiments validate that PanFusion surpasses existing methods and, thanks to its dual-branch structure, can integrate additional constraints like room layout for customized panorama outputs. Code is available at https://chengzhag.github.io/publication/panfusion.

Title: Reinforcement Learning with Generalizable Gaussian Splatting

• Authors: Jiaxu Wang, Qiang Zhang, Jingkai Sun, Jiahang Cao, Yecheng Shao, Renjing Xu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07950
• Pdf URL: https://arxiv.org/pdf/2404.07950
• Copy Paste: [[2404.07950]] Reinforcement Learning with Generalizable Gaussian Splatting(https://arxiv.org/abs/2404.07950)
• Keywords: interpretability
• Abstract: An excellent representation is crucial for reinforcement learning (RL) performance, especially in vision-based reinforcement learning tasks. The quality of the environment representation directly influences the achievement of the learning task. Previous vision-based RL typically uses explicit or implicit ways to represent environments, such as images, points, voxels, and neural radiance fields. However, these representations contain several drawbacks. They cannot either describe complex local geometries or generalize well to unseen scenes, or require precise foreground masks. Moreover, these implicit neural representations are akin to a ``black box", significantly hindering interpretability. 3D Gaussian Splatting (3DGS), with its explicit scene representation and differentiable rendering nature, is considered a revolutionary change for reconstruction and representation methods. In this paper, we propose a novel Generalizable Gaussian Splatting framework to be the representation of RL tasks, called GSRL. Through validation in the RoboMimic environment, our method achieves better results than other baselines in multiple tasks, improving the performance by 10%, 44%, and 15% compared with baselines on the hardest task. This work is the first attempt to leverage generalizable 3DGS as a representation for RL.

Title: Deep learning-based auto-segmentation of paraganglioma for growth monitoring

• Authors: E.M.C. Sijben, J.C. Jansen, M. de Ridder, P.A.N. Bosman, T. Alderliesten
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07952
• Pdf URL: https://arxiv.org/pdf/2404.07952
• Copy Paste: [[2404.07952]] Deep learning-based auto-segmentation of paraganglioma for growth monitoring(https://arxiv.org/abs/2404.07952)
• Keywords: segmentation
• Abstract: Volume measurement of a paraganglioma (a rare neuroendocrine tumor that typically forms along major blood vessels and nerve pathways in the head and neck region) is crucial for monitoring and modeling tumor growth in the long term. However, in clinical practice, using available tools to do these measurements is time-consuming and suffers from tumor-shape assumptions and observer-to-observer variation. Growth modeling could play a significant role in solving a decades-old dilemma (stemming from uncertainty regarding how the tumor will develop over time). By giving paraganglioma patients treatment, severe symptoms can be prevented. However, treating patients who do not actually need it, comes at the cost of unnecessary possible side effects and complications. Improved measurement techniques could enable growth model studies with a large amount of tumor volume data, possibly giving valuable insights into how these tumors develop over time. Therefore, we propose an automated tumor volume measurement method based on a deep learning segmentation model using no-new-UNnet (nnUNet). We assess the performance of the model based on visual inspection by a senior otorhinolaryngologist and several quantitative metrics by comparing model outputs with manual delineations, including a comparison with variation in manual delineation by multiple observers. Our findings indicate that the automatic method performs (at least) equal to manual delineation. Finally, using the created model, and a linking procedure that we propose to track the tumor over time, we show how additional volume measurements affect the fit of known growth functions.

Title: Triple Component Matrix Factorization: Untangling Global, Local, and Noisy Components

• Authors: Naichen Shi, Salar Fattahi, Raed Al Kontar
• Subjects: cs.LG, math.ST
• Abstract URL: https://arxiv.org/abs/2404.07955
• Pdf URL: https://arxiv.org/pdf/2404.07955
• Copy Paste: [[2404.07955]] Triple Component Matrix Factorization: Untangling Global, Local, and Noisy Components(https://arxiv.org/abs/2404.07955)
• Keywords: extraction, segmentation
• Abstract: In this work, we study the problem of common and unique feature extraction from noisy data. When we have N observation matrices from N different and associated sources corrupted by sparse and potentially gross noise, can we recover the common and unique components from these noisy observations? This is a challenging task as the number of parameters to estimate is approximately thrice the number of observations. Despite the difficulty, we propose an intuitive alternating minimization algorithm called triple component matrix factorization (TCMF) to recover the three components exactly. TCMF is distinguished from existing works in literature thanks to two salient features. First, TCMF is a principled method to separate the three components given noisy observations provably. Second, the bulk of the computation in TCMF can be distributed. On the technical side, we formulate the problem as a constrained nonconvex nonsmooth optimization problem. Despite the intricate nature of the problem, we provide a Taylor series characterization of its solution by solving the corresponding Karush-Kuhn-Tucker conditions. Using this characterization, we can show that the alternating minimization algorithm makes significant progress at each iteration and converges into the ground truth at a linear rate. Numerical experiments in video segmentation and anomaly detection highlight the superior feature extraction abilities of TCMF.

Title: Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models

• Authors: Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, Yinfei Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07973
• Pdf URL: https://arxiv.org/pdf/2404.07973
• Copy Paste: [[2404.07973]] Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models(https://arxiv.org/abs/2404.07973)
• Keywords: large language model
• Abstract: While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks. In this work, we unveil Ferret-v2, a significant upgrade to Ferret, with three key designs. (1) Any resolution grounding and referring: A flexible approach that effortlessly handles higher image resolution, improving the model's ability to process and understand images in greater detail. (2) Multi-granularity visual encoding: By integrating the additional DINOv2 encoder, the model learns better and diverse underlying contexts for global and fine-grained visual information. (3) A three-stage training paradigm: Besides image-caption alignment, an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Experiments show that Ferret-v2 provides substantial improvements over Ferret and other state-of-the-art methods, thanks to its high-resolution scaling and fine-grained visual processing.

Title: Gaga: Group Any Gaussians via 3D-aware Memory Bank

• Authors: Weijie Lyu, Xueting Li, Abhijit Kundu, Yi-Hsuan Tsai, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07977
• Pdf URL: https://arxiv.org/pdf/2404.07977
• Copy Paste: [[2404.07977]] Gaga: Group Any Gaussians via 3D-aware Memory Bank(https://arxiv.org/abs/2404.07977)
• Keywords: robust, segmentation
• Abstract: We introduce Gaga, a framework that reconstructs and segments open-world 3D scenes by leveraging inconsistent 2D masks predicted by zero-shot segmentation models. Contrasted to prior 3D scene segmentation approaches that heavily rely on video object tracking, Gaga utilizes spatial information and effectively associates object masks across diverse camera poses. By eliminating the assumption of continuous view changes in training images, Gaga demonstrates robustness to variations in camera poses, particularly beneficial for sparsely sampled images, ensuring precise mask label consistency. Furthermore, Gaga accommodates 2D segmentation masks from diverse sources and demonstrates robust performance with different open-world zero-shot segmentation models, enhancing its versatility. Extensive qualitative and quantitative evaluations demonstrate that Gaga performs favorably against state-of-the-art methods, emphasizing its potential for real-world applications such as scene understanding and manipulation.

Title: LLoCO: Learning Long Contexts Offline

• Authors: Sijun Tan, Xiuyu Li, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E. Gonzalez, Raluca Ada Popa
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07979
• Pdf URL: https://arxiv.org/pdf/2404.07979
• Copy Paste: [[2404.07979]] LLoCO: Learning Long Contexts Offline(https://arxiv.org/abs/2404.07979)
• Keywords: large language model
• Abstract: Processing long contexts remains a challenge for large language models (LLMs) due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation. We propose a novel approach to address this problem by learning contexts offline through context compression and in-domain parameter-efficient finetuning. Our method enables an LLM to create a concise representation of the original context and efficiently retrieve relevant information to answer questions accurately. We introduce LLoCO, a technique that combines context compression, retrieval, and parameter-efficient finetuning using LoRA. Our approach extends the effective context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We evaluate our approach on several long-context question-answering datasets, demonstrating that LLoCO significantly outperforms in-context learning while using $30\times$ fewer tokens during inference. LLoCO achieves up to $7.62\times$ speed-up and substantially reduces the cost of long document question answering, making it a promising solution for efficient long context processing. Our code is publicly available at https://github.com/jeffreysijuntan/lloco.

Title: View Selection for 3D Captioning via Diffusion Ranking

• Authors: Tiange Luo, Justin Johnson, Honglak Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07984
• Pdf URL: https://arxiv.org/pdf/2404.07984
• Copy Paste: [[2404.07984]] View Selection for 3D Captioning via Diffusion Ranking(https://arxiv.org/abs/2404.07984)
• Keywords: diffusion
• Abstract: Scalable annotation approaches are crucial for constructing extensive 3D-text datasets, facilitating a broader range of applications. However, existing methods sometimes lead to the generation of hallucinated captions, compromising caption quality. This paper explores the issue of hallucination in 3D object captioning, with a focus on Cap3D method, which renders 3D objects into 2D views for captioning using pre-trained models. We pinpoint a major challenge: certain rendered views of 3D objects are atypical, deviating from the training data of standard image captioning models and causing hallucinations. To tackle this, we present DiffuRank, a method that leverages a pre-trained text-to-3D model to assess the alignment between 3D objects and their 2D rendered views, where the view with high alignment closely represent the object's characteristics. By ranking all rendered views and feeding the top-ranked ones into GPT4-Vision, we enhance the accuracy and detail of captions, enabling the correction of 200k captions in the Cap3D dataset and extending it to 1 million captions across Objaverse and Objaverse-XL datasets. Additionally, we showcase the adaptability of DiffuRank by applying it to pre-trained text-to-image models for a Visual Question Answering task, where it outperforms the CLIP model.

Title: ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback

• Authors: Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, Chen Chen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07987
• Pdf URL: https://arxiv.org/pdf/2404.07987
• Copy Paste: [[2404.07987]] ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback(https://arxiv.org/abs/2404.07987)
• Keywords: diffusion, segmentation
• Abstract: To enhance the controllability of text-to-image diffusion models, existing efforts like ControlNet incorporated image-based conditional controls. In this paper, we reveal that existing methods still face significant challenges in generating images that align with the image conditional controls. To this end, we propose ControlNet++, a novel approach that improves controllable generation by explicitly optimizing pixel-level cycle consistency between generated images and conditional controls. Specifically, for an input conditional control, we use a pre-trained discriminative reward model to extract the corresponding condition of the generated images, and then optimize the consistency loss between the input conditional control and extracted condition. A straightforward implementation would be generating images from random noises and then calculating the consistency loss, but such an approach requires storing gradients for multiple sampling timesteps, leading to considerable time and memory costs. To address this, we introduce an efficient reward strategy that deliberately disturbs the input images by adding noise, and then uses the single-step denoised images for reward fine-tuning. This avoids the extensive costs associated with image sampling, allowing for more efficient reward fine-tuning. Extensive experiments show that ControlNet++ significantly improves controllability under various conditional controls. For example, it achieves improvements over ControlNet by 7.9% mIoU, 13.4% SSIM, and 7.6% RMSE, respectively, for segmentation mask, line-art edge, and depth conditions.

Title: Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding

• Authors: Yiwen Tang, Jiaming Liu, Dong Wang, Zhigang Wang, Shanghang Zhang, Bin Zhao, Xuelong Li
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.07989
• Pdf URL: https://arxiv.org/pdf/2404.07989
• Copy Paste: [[2404.07989]] Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding(https://arxiv.org/abs/2404.07989)
• Keywords: transformer
• Abstract: Large foundation models have recently emerged as a prominent focus of interest, attaining superior performance in widespread scenarios. Due to the scarcity of 3D data, many efforts have been made to adapt pre-trained transformers from vision to 3D domains. However, such 2D-to-3D approaches are still limited, due to the potential loss of spatial geometries and high computation cost. More importantly, their frameworks are mainly designed for 2D models, lacking a general any-to-3D paradigm. In this paper, we introduce Any2Point, a parameter-efficient method to empower any-modality large models (vision, language, audio) for 3D understanding. Given a frozen transformer from any source modality, we propose a 3D-to-any (1D or 2D) virtual projection strategy that correlates the input 3D points to the original 1D or 2D positions within the source modality. This mechanism enables us to assign each 3D token with a positional encoding paired with the pre-trained model, which avoids 3D geometry loss caused by the true projection and better motivates the transformer for 3D learning with 1D/2D positional priors. Then, within each transformer block, we insert an any-to-3D guided adapter module for parameter-efficient fine-tuning. The adapter incorporates prior spatial knowledge from the source modality to guide the local feature aggregation of 3D tokens, compelling the semantic adaption of any-modality transformers. We conduct extensive experiments to showcase the effectiveness and efficiency of our method. Code and models are released at https://github.com/Ivan-Tang-3D/Any2Point.

Title: OpenBias: Open-set Bias Detection in Text-to-Image Generative Models

• Authors: Moreno D'Incà, Elia Peruzzo, Massimiliano Mancini, Dejia Xu, Vidit Goel, Xingqian Xu, Zhangyang Wang, Humphrey Shi, Nicu Sebe
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07990
• Pdf URL: https://arxiv.org/pdf/2404.07990
• Copy Paste: [[2404.07990]] OpenBias: Open-set Bias Detection in Text-to-Image Generative Models(https://arxiv.org/abs/2404.07990)
• Keywords: fair, diffusion, generative, large language model
• Abstract: Text-to-image generative models are becoming increasingly popular and accessible to the general public. As these models see large-scale deployments, it is necessary to deeply investigate their safety and fairness to not disseminate and perpetuate any kind of biases. However, existing works focus on detecting closed sets of biases defined a priori, limiting the studies to well-known concepts. In this paper, we tackle the challenge of open-set bias detection in text-to-image generative models presenting OpenBias, a new pipeline that identifies and quantifies the severity of biases agnostically, without access to any precompiled set. OpenBias has three stages. In the first phase, we leverage a Large Language Model (LLM) to propose biases given a set of captions. Secondly, the target generative model produces images using the same set of captions. Lastly, a Vision Question Answering model recognizes the presence and extent of the previously proposed biases. We study the behavior of Stable Diffusion 1.5, 2, and XL emphasizing new biases, never investigated before. Via quantitative experiments, we demonstrate that OpenBias agrees with current closed-set bias detection methods and human judgement.