2024-07-26

Title: Real-Time Automated donning and doffing detection of PPE based on Yolov4-tiny

• Authors: Anusha Verma, Ghazal Ghajari, K M Tawsik Jawad, Dr. Hugh P. Salehi, Dr. Fathi Amsaad
• Subjects: cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.17471
• Pdf URL: https://arxiv.org/pdf/2407.17471
• Copy Paste: [[2407.17471]] Real-Time Automated donning and doffing detection of PPE based on Yolov4-tiny(https://arxiv.org/abs/2407.17471)
• Keywords: protect
• Abstract: Maintaining patient safety and the safety of healthcare workers (HCWs) in hospitals and clinics highly depends on following the proper protocol for donning and taking off personal protective equipment (PPE). HCWs can benefit from a feedback system during the putting on and removal process because the process is cognitively demanding and errors are common. Centers for Disease Control and Prevention (CDC) provided guidelines for correct PPE use which should be followed. A real time object detection along with a unique sequencing algorithms are used to identify and determine the donning and doffing process in real time. The purpose of this technical research is two-fold: The user gets real time alert to the step they missed in the sequence if they don't follow the proper procedure during donning or doffing. Secondly, the use of tiny machine learning (yolov4-tiny) in embedded system architecture makes it feasible and cost-effective to deploy in different healthcare settings.

Title: The #Somos600M Project: Generating NLP resources that represent the diversity of the languages from LATAM, the Caribbean, and Spain

• Authors: María Grandury
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17479
• Pdf URL: https://arxiv.org/pdf/2407.17479
• Copy Paste: [[2407.17479]] The #Somos600M Project: Generating NLP resources that represent the diversity of the languages from LATAM, the Caribbean, and Spain(https://arxiv.org/abs/2407.17479)
• Keywords: large language model
• Abstract: We are 600 million Spanish speakers. We launched the #Somos600M Project because the diversity of the languages from LATAM, the Caribbean and Spain needs to be represented in Artificial Intelligence (AI) systems. Despite being the 7.5% of the world population, there is no open dataset to instruction-tune large language models (LLMs), nor a leaderboard to evaluate and compare them. In this paper, we present how we have created as an international open-source community the first versions of the instruction and evaluation datasets, indispensable resources for the advancement of Natural Language Processing (NLP) in our languages.

Title: Universal Approximation Theory: The basic theory for deep learning-based computer vision models

• Authors: Wei Wang, Qing Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17480
• Pdf URL: https://arxiv.org/pdf/2407.17480
• Copy Paste: [[2407.17480]] Universal Approximation Theory: The basic theory for deep learning-based computer vision models(https://arxiv.org/abs/2407.17480)
• Keywords: robust, transformer
• Abstract: Computer vision (CV) is one of the most crucial fields in artificial intelligence. In recent years, a variety of deep learning models based on convolutional neural networks (CNNs) and Transformers have been designed to tackle diverse problems in CV. These algorithms have found practical applications in areas such as robotics and facial recognition. Despite the increasing power of current CV models, several fundamental questions remain unresolved: Why do CNNs require deep layers? What ensures the generalization ability of CNNs? Why do residual-based networks outperform fully convolutional networks like VGG? What is the fundamental difference between residual-based CNNs and Transformer-based networks? Why can CNNs utilize LoRA and pruning techniques? The root cause of these questions lies in the lack of a robust theoretical foundation for deep learning models in CV. To address these critical issues and techniques, we employ the Universal Approximation Theorem (UAT) to provide a theoretical basis for convolution- and Transformer-based models in CV. By doing so, we aim to elucidate these questions from a theoretical perspective.

Title: Robust Adaptation of Foundation Models with Black-Box Visual Prompting

• Authors: Changdae Oh, Gyeongdeok Seo, Geunyoung Jung, Zhi-Qi Cheng, Hosik Choi, Jiyoung Jung, Kyungwoo Song
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17491
• Pdf URL: https://arxiv.org/pdf/2407.17491
• Copy Paste: [[2407.17491]] Robust Adaptation of Foundation Models with Black-Box Visual Prompting(https://arxiv.org/abs/2407.17491)
• Keywords: robust
• Abstract: With the surge of large-scale pre-trained models (PTMs), adapting these models to numerous downstream tasks becomes a crucial problem. Consequently, parameter-efficient transfer learning (PETL) of large models has grasped huge attention. While PETL methods show impressive performance, they commonly rely on two optimistic assumptions: 1) the entire parameters of a PTM are available, and 2) a sufficiently large memory capacity is equipped for caching all the intermediate activations to compute gradients. However, in most real-world applications, PTMs are served as black-box APIs or proprietary software without explicit parameter accessibility. Besides, it is hard to meet a large memory requirement for modern PTMs. This work proposes black-box visual prompting (BlackVIP), which efficiently adapts the PTMs without knowledge about model architectures and parameters. BlackVIP has two components; 1) Coordinator and 2) simultaneous perturbation stochastic approximation with gradient correction (SPSA-GC). The Coordinator designs input-dependent visual prompts, which allow the target PTM to adapt in the wild. SPSA-GC efficiently estimates the gradient of PTM to update the Coordinator. Besides, we propose a variant, BlackVIP-SE, which significantly reduces the runtime and computational cost of BlackVIP. Extensive experiments on 19 datasets demonstrate that BlackVIPs enable robust adaptation to diverse domains and tasks with minimal memory requirements. We further provide theoretical analysis on the generalization of visual prompting methods by presenting their connection to the certified robustness of randomized smoothing.

Title: ReDiFine: Reusable Diffusion Finetuning for Mitigating Degradation in the Chain of Diffusion

• Authors: Youngseok Yoon, Dainong Hu, Iain Weissburg, Yao Qin, Haewon Jeong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17493
• Pdf URL: https://arxiv.org/pdf/2407.17493
• Copy Paste: [[2407.17493]] ReDiFine: Reusable Diffusion Finetuning for Mitigating Degradation in the Chain of Diffusion(https://arxiv.org/abs/2407.17493)
• Keywords: diffusion, generative
• Abstract: Diffusion models have achieved tremendous improvements in generative modeling for images, enabling high-quality generation that is indistinguishable by humans from real images. The qualities of images have reached a threshold at which we can reuse synthetic images for training machine learning models again. This attracts the area as it can relieve the high cost of data collection and fundamentally solve many problems in data-limited areas. In this paper, we focus on a practical scenario in which pretrained text-to-image diffusion models are iteratively finetuned using a set of synthetic images, which we call the Chain of Diffusion. Finetuned models generate images that are used for the next iteration of finetuning. We first demonstrate how these iterative processes result in severe degradation in image qualities. Thorough investigations reveal the most impactful factor for the degradation, and we propose finetuning and generation strategies that can effectively resolve the degradation. Our method, Reusable Diffusion Finetuning (ReDiFine), combines condition drop finetuning and CFG scheduling to maintain the qualities of generated images throughout iterations. ReDiFine works effectively for multiple datasets and models without further hyperparameter search, making synthetic images reusable to finetune future generative models.

Title: Generative artificial intelligence in dentistry: Current approaches and future challenges

• Authors: Fabián Villena, Claudia Véliz, Rosario García-Huidobro, Sebastián Aguayo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17532
• Pdf URL: https://arxiv.org/pdf/2407.17532
• Copy Paste: [[2407.17532]] Generative artificial intelligence in dentistry: Current approaches and future challenges(https://arxiv.org/abs/2407.17532)
• Keywords: generative
• Abstract: Artificial intelligence (AI) has become a commodity for people because of the advent of generative AI (GenAI) models that bridge the usability gap of AI by providing a natural language interface to interact with complex models. These GenAI models range from text generation - such as two-way chat systems - to the generation of image or video from textual descriptions input by a user. These advancements in AI have impacted Dentistry in multiple aspects. In dental education, the student now has the opportunity to solve a plethora of questions by only prompting a GenAI model and have the answer in a matter of seconds. GenAI models can help us deliver better patient healthcare by helping practitioners gather knowledge quickly and efficiently. Finally, GenAI can also be used in dental research, where the applications range from new drug discovery to assistance in academic writing. In this review, we first define GenAI models and describe their multiple generation modalities; then, we explain and discuss their current and potential applications in Dentistry; and finally, we describe the challenges these new technologies impose in our area.

Title: SFPrompt: Communication-Efficient Split Federated Fine-Tuning for Large Pre-Trained Models over Resource-Limited Devices

• Authors: Linxiao Cao, Yifei Zhu, Wei Gong
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2407.17533
• Pdf URL: https://arxiv.org/pdf/2407.17533
• Copy Paste: [[2407.17533]] SFPrompt: Communication-Efficient Split Federated Fine-Tuning for Large Pre-Trained Models over Resource-Limited Devices(https://arxiv.org/abs/2407.17533)
• Keywords: privacy, federate
• Abstract: Large pre-trained models have exhibited remarkable achievements across various domains. The substantial training costs associated with these models have led to wide studies of fine-tuning for effectively harnessing their capabilities in solving downstream tasks. Yet, conventional fine-tuning approaches become infeasible when the model lacks access to downstream data due to privacy concerns. Naively integrating fine-tuning approaches with the emerging federated learning frameworks incurs substantial communication overhead and exerts high demand on local computing resources, making it impractical for common resource-limited devices. In this paper, we introduce SFPrompt, an innovative privacy-preserving fine-tuning method tailored for the federated setting where direct uploading of raw data is prohibited and local devices are resource-constrained to run a complete pre-trained model. In essence, SFPrompt judiciously combines split learning with federated learning to handle these challenges. Specifically, the pre-trained model is first partitioned into client and server components, thereby streamlining the client-side model and substantially alleviating computational demands on local resources. SFPrompt then introduces soft prompts into the federated model to enhance the fine-tuning performance. To further reduce communication costs, a novel dataset pruning algorithm and a local-loss update strategy are devised during the fine-tuning process. Extensive experiments demonstrate that SFPrompt delivers competitive performance as the federated full fine-tuning approach while consuming a mere 0.46% of local computing resources and incurring 53% less communication cost.

Title: Dataset Distribution Impacts Model Fairness: Single vs. Multi-Task Learning

• Authors: Ralf Raumanns, Gerard Schouten, Josien P. W. Pluim, Veronika Cheplygina
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.17543
• Pdf URL: https://arxiv.org/pdf/2407.17543
• Copy Paste: [[2407.17543]] Dataset Distribution Impacts Model Fairness: Single vs. Multi-Task Learning(https://arxiv.org/abs/2407.17543)
• Keywords: fair
• Abstract: The influence of bias in datasets on the fairness of model predictions is a topic of ongoing research in various fields. We evaluate the performance of skin lesion classification using ResNet-based CNNs, focusing on patient sex variations in training data and three different learning strategies. We present a linear programming method for generating datasets with varying patient sex and class labels, taking into account the correlations between these variables. We evaluated the model performance using three different learning strategies: a single-task model, a reinforcing multi-task model, and an adversarial learning scheme. Our observations include: 1) sex-specific training data yields better results, 2) single-task models exhibit sex bias, 3) the reinforcement approach does not remove sex bias, 4) the adversarial model eliminates sex bias in cases involving only female patients, and 5) datasets that include male patients enhance model performance for the male subgroup, even when female patients are the majority. To generalise these findings, in future research, we will examine more demographic attributes, like age, and other possibly confounding factors, such as skin colour and artefacts in the skin lesions. We make all data and models available on GitHub.

Title: Exploring Domain Robust Lightweight Reward Models based on Router Mechanism

• Authors: Hyuk Namgoong, Jeesu Jung, Sangkeun Jung, Yoonhyung Roh
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17546
• Pdf URL: https://arxiv.org/pdf/2407.17546
• Copy Paste: [[2407.17546]] Exploring Domain Robust Lightweight Reward Models based on Router Mechanism(https://arxiv.org/abs/2407.17546)
• Keywords: robust, large language model
• Abstract: Recent advancements in large language models have heavily relied on the large reward model from reinforcement learning from human feedback for fine-tuning. However, the use of a single reward model across various domains may not always be optimal, often requiring retraining from scratch when new domain data is introduced. To address these challenges, we explore the utilization of small language models operating in a domain-specific manner based on router mechanisms. Our three approaches are: 1) utilize mixture of experts to form a single reward model by modularizing an internal router and experts, 2) employing external router to select the appropriate reward model from multiple domain-specific models, and 3) the framework reduces parameter size by loading reward models and router adapters onto a single small language model using adapters. Experimental validation underscores the effectiveness of our approach, demonstrating performance comparable to baseline methods while also reducing the total parameter size.

Title: Diffusion Models for Multi-Task Generative Modeling

• Authors: Changyou Chen, Han Ding, Bunyamin Sisman, Yi Xu, Ouye Xie, Benjamin Z. Yao, Son Dinh Tran, Belinda Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17571
• Pdf URL: https://arxiv.org/pdf/2407.17571
• Copy Paste: [[2407.17571]] Diffusion Models for Multi-Task Generative Modeling(https://arxiv.org/abs/2407.17571)
• Keywords: diffusion, generative
• Abstract: Diffusion-based generative modeling has been achieving state-of-the-art results on various generation tasks. Most diffusion models, however, are limited to a single-generation modeling. Can we generalize diffusion models with the ability of multi-modal generative training for more generalizable modeling? In this paper, we propose a principled way to define a diffusion model by constructing a unified multi-modal diffusion model in a common diffusion space. We define the forward diffusion process to be driven by an information aggregation from multiple types of task-data, e.g., images for a generation task and labels for a classification task. In the reverse process, we enforce information sharing by parameterizing a shared backbone denoising network with additional modality-specific decoder heads. Such a structure can simultaneously learn to generate different types of multi-modal data with a multi-task loss, which is derived from a new multi-modal variational lower bound that generalizes the standard diffusion model. We propose several multimodal generation settings to verify our framework, including image transition, masked-image training, joint image-label and joint image-representation generative modeling. Extensive experimental results on ImageNet indicate the effectiveness of our framework for various multi-modal generative modeling, which we believe is an important research direction worthy of more future explorations.

Title: S-E Pipeline: A Vision Transformer (ViT) based Resilient Classification Pipeline for Medical Imaging Against Adversarial Attacks

• Authors: Neha A S, Vivek Chaturvedi, Muhammad Shafique
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17587
• Pdf URL: https://arxiv.org/pdf/2407.17587
• Copy Paste: [[2407.17587]] S-E Pipeline: A Vision Transformer (ViT) based Resilient Classification Pipeline for Medical Imaging Against Adversarial Attacks(https://arxiv.org/abs/2407.17587)
• Keywords: attack, robust, transformer, segmentation
• Abstract: Vision Transformer (ViT) is becoming widely popular in automating accurate disease diagnosis in medical imaging owing to its robust self-attention mechanism. However, ViTs remain vulnerable to adversarial attacks that may thwart the diagnosis process by leading it to intentional misclassification of critical disease. In this paper, we propose a novel image classification pipeline, namely, S-E Pipeline, that performs multiple pre-processing steps that allow ViT to be trained on critical features so as to reduce the impact of input perturbations by adversaries. Our method uses a combination of segmentation and image enhancement techniques such as Contrast Limited Adaptive Histogram Equalization (CLAHE), Unsharp Masking (UM), and High-Frequency Emphasis filtering (HFE) as preprocessing steps to identify critical features that remain intact even after adversarial perturbations. The experimental study demonstrates that our novel pipeline helps in reducing the effect of adversarial attacks by 72.22% for the ViT-b32 model and 86.58% for the ViT-l32 model. Furthermore, we have shown an end-to-end deployment of our proposed method on the NVIDIA Jetson Orin Nano board to demonstrate its practical use case in modern hand-held devices that are usually resource-constrained.

Title: Coupling Speech Encoders with Downstream Text Models

• Authors: Ciprian Chelba, Johan Schalkwyk
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.17605
• Pdf URL: https://arxiv.org/pdf/2407.17605
• Copy Paste: [[2407.17605]] Coupling Speech Encoders with Downstream Text Models(https://arxiv.org/abs/2407.17605)
• Keywords: large language model
• Abstract: We present a modular approach to building cascade speech translation (AST) models that guarantees that the resulting model performs no worse than the 1-best cascade baseline while preserving state-of-the-art speech recognition (ASR) and text translation (MT) performance for a given task. Our novel contribution is the use of an ``exporter'' layer that is trained under L2-loss to ensure a strong match between ASR embeddings and the MT token embeddings for the 1-best sequence. The ``exporter'' output embeddings are fed directly to the MT model in lieu of 1-best token embeddings, thus guaranteeing that the resulting model performs no worse than the 1-best cascade baseline, while allowing back-propagation gradient to flow from the MT model into the ASR components. The matched-embeddings cascade architecture provide a significant improvement over its 1-best counterpart in scenarios where incremental training of the MT model is not an option and yet we seek to improve quality by leveraging (speech, transcription, translated transcription) data provided with the AST task. The gain disappears when the MT model is incrementally trained on the parallel text data available with the AST task. The approach holds promise for other scenarios that seek to couple ASR encoders and immutable text models, such at large language models (LLM).

Title: Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks

• Authors: Spyros Rigas, Michalis Papachristou, Theofilos Papadopoulos, Fotios Anagnostopoulos, Georgios Alexandridis
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2407.17611
• Pdf URL: https://arxiv.org/pdf/2407.17611
• Copy Paste: [[2407.17611]] Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks(https://arxiv.org/abs/2407.17611)
• Keywords: robust, interpretability
• Abstract: Physics-Informed Neural Networks (PINNs) have emerged as a robust framework for solving Partial Differential Equations (PDEs) by approximating their solutions via neural networks and imposing physics-based constraints on the loss function. Traditionally, Multilayer Perceptrons (MLPs) are the neural network of choice, and significant progress has been made in optimizing their training. Recently, Kolmogorov-Arnold Networks (KANs) were introduced as a viable alternative, with the potential of offering better interpretability and efficiency while requiring fewer parameters. In this paper, we present a fast JAX-based implementation of grid-dependent Physics-Informed Kolmogorov-Arnold Networks (PIKANs) for solving PDEs. We propose an adaptive training scheme for PIKANs, incorporating known MLP-based PINN techniques, introducing an adaptive state transition scheme to avoid loss function peaks between grid updates, and proposing a methodology for designing PIKANs with alternative basis functions. Through comparative experiments we demonstrate that these adaptive features significantly enhance training efficiency and solution accuracy. Our results illustrate the effectiveness of PIKANs in improving performance for PDE solutions, highlighting their potential as a superior alternative in scientific and engineering applications.

Title: Towards Neural Network based Cognitive Models of Dynamic Decision-Making by Humans

• Authors: Changyu Chen, Shashank Reddy Chirra, Maria José Ferreira, Cleotilde Gonzalez, Arunesh Sinha, Pradeep Varakantham
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.17622
• Pdf URL: https://arxiv.org/pdf/2407.17622
• Copy Paste: [[2407.17622]] Towards Neural Network based Cognitive Models of Dynamic Decision-Making by Humans(https://arxiv.org/abs/2407.17622)
• Keywords: security, attack, large language model
• Abstract: Modelling human cognitive processes in dynamic decision-making tasks has been an endeavor in AI for a long time. Some initial works have attempted to utilize neural networks (and large language models) but often assume one common model for all humans and aim to emulate human behavior in aggregate. However, behavior of each human is distinct, heterogeneous and relies on specific past experiences in specific tasks. To that end, we build on a well known model of cognition, namely Instance Based Learning (IBL), that posits that decisions are made based on similar situations encountered in the past. We propose two new attention based neural network models to model human decision-making in dynamic settings. We experiment with two distinct datasets gathered from human subject experiment data, one focusing on detection of phishing email by humans and another where humans act as attackers in a cybersecurity setting and decide on an attack option. We conduct extensive experiments with our two neural network models, IBL, and GPT3.5, and demonstrate that one of our neural network models achieves the best performance in representing human decision-making. We find an interesting trend that all models predict a human's decision better if that human is better at the task. We also explore explanation of human decisions based on what our model considers important in prediction. Overall, our work yields promising results for further use of neural networks in cognitive modelling of human decision making. Our code is available at this https URL.

Title: Revising the Problem of Partial Labels from the Perspective of CNNs' Robustness

• Authors: Xin Zhang, Yuqi Song, Wyatt McCurdy, Xiaofeng Wang, Fei Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17630
• Pdf URL: https://arxiv.org/pdf/2407.17630
• Copy Paste: [[2407.17630]] Revising the Problem of Partial Labels from the Perspective of CNNs' Robustness(https://arxiv.org/abs/2407.17630)
• Keywords: attack, robust
• Abstract: Convolutional neural networks (CNNs) have gained increasing popularity and versatility in recent decades, finding applications in diverse domains. These remarkable achievements are greatly attributed to the support of extensive datasets with precise labels. However, annotating image datasets is intricate and complex, particularly in the case of multi-label datasets. Hence, the concept of partial-label setting has been proposed to reduce annotation costs, and numerous corresponding solutions have been introduced. The evaluation methods for these existing solutions have been primarily based on accuracy. That is, their performance is assessed by their predictive accuracy on the test set. However, we insist that such an evaluation is insufficient and one-sided. On one hand, since the quality of the test set has not been evaluated, the assessment results are unreliable. On the other hand, the partial-label problem may also be raised by undergoing adversarial attacks. Therefore, incorporating robustness into the evaluation system is crucial. For this purpose, we first propose two attack models to generate multiple partial-label datasets with varying degrees of label missing rates. Subsequently, we introduce a lightweight partial-label solution using pseudo-labeling techniques and a designed loss function. Then, we employ D-Score to analyze both the proposed and existing methods to determine whether they can enhance robustness while improving accuracy. Extensive experimental results demonstrate that while certain methods may improve accuracy, the enhancement in robustness is not significant, and in some cases, it even diminishes.

Title: IgnitionInnovators at "Discharge Me!": Chain-of-Thought Instruction Finetuning Large Language Models for Discharge Summaries

• Authors: An Quang Tang, Xiuzhen Zhang, Minh Ngoc Dinh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17636
• Pdf URL: https://arxiv.org/pdf/2407.17636
• Copy Paste: [[2407.17636]] IgnitionInnovators at "Discharge Me!": Chain-of-Thought Instruction Finetuning Large Language Models for Discharge Summaries(https://arxiv.org/abs/2407.17636)
• Keywords: large language model
• Abstract: This paper presents our proposed approach to the Discharge Me! shared task, collocated with the 23th Workshop on Biomedical Natural Language Processing (BioNLP). In this work, we develop an LLM-based framework for solving the Discharge Summary Documentation (DSD) task, i.e., generating the two critical target sections `Brief Hospital Course' and `Discharge Instructions' in the discharge summary. By streamlining the recent instruction-finetuning process on LLMs, we explore several prompting strategies for optimally adapting LLMs to specific generation task of DSD. Experimental results show that providing a clear output structure, complimented by a set of comprehensive Chain-of-Thoughts (CoT) questions, effectively improves the model's reasoning capability, and thereby, enhancing the structural correctness and faithfulness of clinical information in the generated text. Source code is available at: this https URL

Title: SMA-Hyper: Spatiotemporal Multi-View Fusion Hypergraph Learning for Traffic Accident Prediction

• Authors: Xiaowei Gao, James Haworth, Ilya Ilyankou, Xianghui Zhang, Tao Cheng, Stephen Law, Huanfa Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17642
• Pdf URL: https://arxiv.org/pdf/2407.17642
• Copy Paste: [[2407.17642]] SMA-Hyper: Spatiotemporal Multi-View Fusion Hypergraph Learning for Traffic Accident Prediction(https://arxiv.org/abs/2407.17642)
• Keywords: interpretability
• Abstract: Predicting traffic accidents is the key to sustainable city management, which requires effective address of the dynamic and complex spatiotemporal characteristics of cities. Current data-driven models often struggle with data sparsity and typically overlook the integration of diverse urban data sources and the high-order dependencies within them. Additionally, they frequently rely on predefined topologies or weights, limiting their adaptability in spatiotemporal predictions. To address these issues, we introduce the Spatiotemporal Multiview Adaptive HyperGraph Learning (SMA-Hyper) model, a dynamic deep learning framework designed for traffic accident prediction. Building on previous research, this innovative model incorporates dual adaptive spatiotemporal graph learning mechanisms that enable high-order cross-regional learning through hypergraphs and dynamic adaptation to evolving urban data. It also utilises contrastive learning to enhance global and local data representations in sparse datasets and employs an advance attention mechanism to fuse multiple views of accident data and urban functional features, thereby enriching the contextual understanding of risk factors. Extensive testing on the London traffic accident dataset demonstrates that the SMA-Hyper model significantly outperforms baseline models across various temporal horizons and multistep outputs, affirming the effectiveness of its multiview fusion and adaptive learning strategies. The interpretability of the results further underscores its potential to improve urban traffic management and safety by leveraging complex spatiotemporal urban data, offering a scalable framework adaptable to diverse urban environments.

Title: Hopfield Networks for Asset Allocation

• Authors: Carlo Nicolini, Monisha Gopalan, Jacopo Staiano, Bruno Lepri
• Subjects: cs.LG, q-fin.CP, q-fin.PM
• Abstract URL: https://arxiv.org/abs/2407.17645
• Pdf URL: https://arxiv.org/pdf/2407.17645
• Copy Paste: [[2407.17645]] Hopfield Networks for Asset Allocation(https://arxiv.org/abs/2407.17645)
• Keywords: robust, transformer
• Abstract: We present the first application of modern Hopfield networks to the problem of portfolio optimization. We performed an extensive study based on combinatorial purged cross-validation over several datasets and compared our results to both traditional and deep-learning-based methods for portfolio selection. Compared to state-of-the-art deep-learning methods such as Long-Short Term Memory networks and Transformers, we find that the proposed approach performs on par or better, while providing faster training times and better stability. Our results show that Modern Hopfield Networks represent a promising approach to portfolio optimization, allowing for an efficient, scalable, and robust solution for asset allocation, risk management, and dynamic rebalancing.

Title: Generative Learning for Simulation of US Army Vehicle Faults

• Authors: Patrick Kuiper, Sirui Lin, Jose Blanchet, Vahid Tarokh
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.17654
• Pdf URL: https://arxiv.org/pdf/2407.17654
• Copy Paste: [[2407.17654]] Generative Learning for Simulation of US Army Vehicle Faults(https://arxiv.org/abs/2407.17654)
• Keywords: generative
• Abstract: We develop a novel generative model to simulate vehicle health and forecast faults, conditioned on practical operational considerations. The model, trained on data from the US Army's Predictive Logistics program, aims to support predictive maintenance. It forecasts faults far enough in advance to execute a maintenance intervention before a breakdown occurs. The model incorporates real-world factors that affect vehicle health. It also allows us to understand the vehicle's condition by analyzing operating data, and characterizing each vehicle into discrete states. Importantly, the model predicts the time to first fault with high accuracy. We compare its performance to other models and demonstrate its successful training.

Title: Explaining the Model, Protecting Your Data: Revealing and Mitigating the Data Privacy Risks of Post-Hoc Model Explanations via Membership Inference

• Authors: Catherine Huang, Martin Pawelczyk, Himabindu Lakkaraju
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.17663
• Pdf URL: https://arxiv.org/pdf/2407.17663
• Copy Paste: [[2407.17663]] Explaining the Model, Protecting Your Data: Revealing and Mitigating the Data Privacy Risks of Post-Hoc Model Explanations via Membership Inference(https://arxiv.org/abs/2407.17663)
• Keywords: privacy, protect, attack, membership infer, explainability, transformer
• Abstract: Predictive machine learning models are becoming increasingly deployed in high-stakes contexts involving sensitive personal data; in these contexts, there is a trade-off between model explainability and data privacy. In this work, we push the boundaries of this trade-off: with a focus on foundation models for image classification fine-tuning, we reveal unforeseen privacy risks of post-hoc model explanations and subsequently offer mitigation strategies for such risks. First, we construct VAR-LRT and L1/L2-LRT, two new membership inference attacks based on feature attribution explanations that are significantly more successful than existing explanation-leveraging attacks, particularly in the low false-positive rate regime that allows an adversary to identify specific training set members with confidence. Second, we find empirically that optimized differentially private fine-tuning substantially diminishes the success of the aforementioned attacks, while maintaining high model accuracy. We carry out a systematic empirical investigation of our 2 new attacks with 5 vision transformer architectures, 5 benchmark datasets, 4 state-of-the-art post-hoc explanation methods, and 4 privacy strength settings.

Title: SDLNet: Statistical Deep Learning Network for Co-Occurring Object Detection and Identification

• Authors: Binay Kumar Singh, Niels Da Vitoria Lobo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17664
• Pdf URL: https://arxiv.org/pdf/2407.17664
• Copy Paste: [[2407.17664]] SDLNet: Statistical Deep Learning Network for Co-Occurring Object Detection and Identification(https://arxiv.org/abs/2407.17664)
• Keywords: security
• Abstract: With the growing advances in deep learning based technologies the detection and identification of co-occurring objects is a challenging task which has many applications in areas such as, security and surveillance. In this paper, we propose a novel framework called SDLNet- Statistical analysis with Deep Learning Network that identifies co-occurring objects in conjunction with base objects in multilabel object categories. The pipeline of proposed work is implemented in two stages: in the first stage of SDLNet we deal with multilabel detectors for discovering labels, and in the second stage we perform co-occurrence matrix analysis. In co-occurrence matrix analysis, we learn co-occurrence statistics by setting base classes and frequently occurring classes, following this we build association rules and generate frequent patterns. The crucial part of SDLNet is recognizing base classes and making consideration for co-occurring classes. Finally, the generated co-occurrence matrix based on frequent patterns will show base classes and their corresponding co-occurring classes. SDLNet is evaluated on two publicly available datasets: Pascal VOC and MS-COCO. The experimental results on these benchmark datasets are reported in Sec 4.

Title: Unsqueeze [CLS] Bottleneck to Learn Rich Representations

• Authors: Qing Su, Shihao Ji
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17671
• Pdf URL: https://arxiv.org/pdf/2407.17671
• Copy Paste: [[2407.17671]] Unsqueeze [CLS] Bottleneck to Learn Rich Representations(https://arxiv.org/abs/2407.17671)
• Keywords: segmentation
• Abstract: Distillation-based self-supervised learning typically leads to more compressed representations due to its radical clustering process and the implementation of a sharper target distribution. To overcome this limitation and preserve more information from input, we introduce UDI, conceptualized as Unsqueezed Distillation-based self-supervised learning (SSL). UDI enriches the learned representation by encouraging multimodal prediction distilled from a consolidated profile of local predictions that are derived via stratified sampling. Our evaluations show that UDI not only promotes semantically meaningful representations at instance level, delivering superior or competitive results to state-of-the-art SSL methods in image classification, but also effectively preserves the nuisance of input, which yields significant improvement in dense prediction tasks, including object detection and segmentation. Additionally, UDI performs competitively in low-shot image classification, improving the scalability of joint-embedding pipelines. Various visualizations and ablation studies are presented to further elucidate the mechanisms behind UDI. Our source code is available at this https URL.

Title: Spiking Neural Networks in Vertical Federated Learning: Performance Trade-offs

• Authors: Maryam Abbasihafshejani, Anindya Maiti, Murtuza Jadliwala
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17672
• Pdf URL: https://arxiv.org/pdf/2407.17672
• Copy Paste: [[2407.17672]] Spiking Neural Networks in Vertical Federated Learning: Performance Trade-offs(https://arxiv.org/abs/2407.17672)
• Keywords: privacy, federate
• Abstract: Federated machine learning enables model training across multiple clients while maintaining data privacy. Vertical Federated Learning (VFL) specifically deals with instances where the clients have different feature sets of the same samples. As federated learning models aim to improve efficiency and adaptability, innovative neural network architectures like Spiking Neural Networks (SNNs) are being leveraged to enable fast and accurate processing at the edge. SNNs, known for their efficiency over Artificial Neural Networks (ANNs), have not been analyzed for their applicability in VFL, thus far. In this paper, we investigate the benefits and trade-offs of using SNN models in a vertical federated learning setting. We implement two different federated learning architectures -- with model splitting and without model splitting -- that have different privacy and performance implications. We evaluate the setup using CIFAR-10 and CIFAR-100 benchmark datasets along with SNN implementations of VGG9 and ResNET classification models. Comparative evaluations demonstrate that the accuracy of SNN models is comparable to that of traditional ANNs for VFL applications, albeit significantly more energy efficient.

Title: Synthetic High-resolution Cryo-EM Density Maps with Generative Adversarial Networks

• Authors: Chenwei Zhang, Anne Condon, Khanh Dao Duc
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2407.17674
• Pdf URL: https://arxiv.org/pdf/2407.17674
• Copy Paste: [[2407.17674]] Synthetic High-resolution Cryo-EM Density Maps with Generative Adversarial Networks(https://arxiv.org/abs/2407.17674)
• Keywords: generative
• Abstract: Generating synthetic cryogenic electron microscopy (cryo-EM) 3D density maps from molecular structures has potential important applications in structural biology. Yet existing simulation-based methods cannot mimic all the complex features present in experimental maps, such as secondary structure elements. As an alternative, we propose struc2mapGAN, a novel data-driven method that employs a generative adversarial network (GAN) to produce high-resolution experimental-like density maps from molecular structures. More specifically, struc2mapGAN uses a U-Net++ architecture as the generator, with an additional L1 loss term and further processing of raw experimental maps to enhance learning efficiency. While struc2mapGAN can promptly generate maps after training, we demonstrate that it outperforms existing simulation-based methods for a wide array of tested maps and across various evaluation metrics. Our code is available at this https URL.

Title: Semi-Compressed CRYSTALS-Kyber

• Authors: Shuiyin Liu, Amin Sakzad
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.17684
• Pdf URL: https://arxiv.org/pdf/2407.17684
• Copy Paste: [[2407.17684]] Semi-Compressed CRYSTALS-Kyber(https://arxiv.org/abs/2407.17684)
• Keywords: security
• Abstract: In this paper, we investigate the communication overhead of the Kyber, which has recently been standardized by the National Institute of Standards and Technology (NIST). Given the same decryption failure rate (DFR) and security argument, we show it is feasible to reduce the communication overhead of the Kyber by 54%. The improvement is based on two technologies: ciphertext quantization and plaintext encoding. First, we prove that the Lloyd-Max quantization is optimal to minimize the decryption decoding noise. The original Kyber compression function is not optimal. Second, we propose an encoding scheme, which combines Pulse-Amplitude Modulation (PAM), Gray mapping, and a binary error correcting code. An explicit expression for the DFR is derived. The minimum possible communication overhead is also derived. Finally, we demonstrate that with the Lloyd-Max quantization, 8-PAM, Gray mapping, and a shortened binary BCH(768,638,13) code, the proposed scheme encapsulates 638 bits (e.g., 2.5 AES keys) in a single ciphertext.

Title: Transformers on Markov Data: Constant Depth Suffices

• Authors: Nived Rajaraman, Marco Bondaschi, Kannan Ramchandran, Michael Gastpar, Ashok Vardhan Makkuva
• Subjects: cs.LG, cs.CL, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.17686
• Pdf URL: https://arxiv.org/pdf/2407.17686
• Copy Paste: [[2407.17686]] Transformers on Markov Data: Constant Depth Suffices(https://arxiv.org/abs/2407.17686)
• Keywords: transformer, generative
• Abstract: Attention-based transformers have been remarkably successful at modeling generative processes across various domains and modalities. In this paper, we study the behavior of transformers on data drawn from \kth Markov processes, where the conditional distribution of the next symbol in a sequence depends on the previous $k$ symbols observed. We observe a surprising phenomenon empirically which contradicts previous findings: when trained for sufficiently long, a transformer with a fixed depth and $1$ head per layer is able to achieve low test loss on sequences drawn from \kth Markov sources, even as $k$ grows. Furthermore, this low test loss is achieved by the transformer's ability to represent and learn the in-context conditional empirical distribution. On the theoretical side, our main result is that a transformer with a single head and three layers can represent the in-context conditional empirical distribution for \kth Markov sources, concurring with our empirical observations. Along the way, we prove that \textit{attention-only} transformers with $O(\log_2(k))$ layers can represent the in-context conditional empirical distribution by composing induction heads to track the previous $k$ symbols in the sequence. These results provide more insight into our current understanding of the mechanisms by which transformers learn to capture context, by understanding their behavior on Markov sources.

Title: Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification

• Authors: Lynnette Hui Xian Ng, Iain Cruickshank, Roy Ka-Wei Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17688
• Pdf URL: https://arxiv.org/pdf/2407.17688
• Copy Paste: [[2407.17688]] Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification(https://arxiv.org/abs/2407.17688)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in executing tasks based on natural language queries. However, these models, trained on curated datasets, inherently embody biases ranging from racial to national and gender biases. It remains uncertain whether these biases impact the performance of LLMs for certain tasks. In this study, we investigate the political biases of LLMs within the stance classification task, specifically examining whether these models exhibit a tendency to more accurately classify politically-charged stances. Utilizing three datasets, seven LLMs, and four distinct prompting schemes, we analyze the performance of LLMs on politically oriented statements and targets. Our findings reveal a statistically significant difference in the performance of LLMs across various politically oriented stance classification tasks. Furthermore, we observe that this difference primarily manifests at the dataset level, with models and prompting schemes showing statistically similar performances across different stance classification datasets. Lastly, we observe that when there is greater ambiguity in the target the statement is directed towards, LLMs have poorer stance classification accuracy.

Title: SAM-MIL: A Spatial Contextual Aware Multiple Instance Learning Approach for Whole Slide Image Classification

• Authors: Heng Fang, Sheng Huang, Wenhao Tang, Luwen Huangfu, Bo Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17689
• Pdf URL: https://arxiv.org/pdf/2407.17689
• Copy Paste: [[2407.17689]] SAM-MIL: A Spatial Contextual Aware Multiple Instance Learning Approach for Whole Slide Image Classification(https://arxiv.org/abs/2407.17689)
• Keywords: extraction, segmentation
• Abstract: Multiple Instance Learning (MIL) represents the predominant framework in Whole Slide Image (WSI) classification, covering aspects such as sub-typing, diagnosis, and beyond. Current MIL models predominantly rely on instance-level features derived from pretrained models such as ResNet. These models segment each WSI into independent patches and extract features from these local patches, leading to a significant loss of global spatial context and restricting the model's focus to merely local features. To address this issue, we propose a novel MIL framework, named SAM-MIL, that emphasizes spatial contextual awareness and explicitly incorporates spatial context by extracting comprehensive, image-level information. The Segment Anything Model (SAM) represents a pioneering visual segmentation foundational model that can capture segmentation features without the need for additional fine-tuning, rendering it an outstanding tool for extracting spatial context directly from raw WSIs. Our approach includes the design of group feature extraction based on spatial context and a SAM-Guided Group Masking strategy to mitigate class imbalance issues. We implement a dynamic mask ratio for different segmentation categories and supplement these with representative group features of categories. Moreover, SAM-MIL divides instances to generate additional pseudo-bags, thereby augmenting the training set, and introduces consistency of spatial context across pseudo-bags to further enhance the model's performance. Experimental results on the CAMELYON-16 and TCGA Lung Cancer datasets demonstrate that our proposed SAM-MIL model outperforms existing mainstream methods in WSIs classification. Our open-source implementation code is is available at this https URL.

Title: SOK: Blockchain for Provenance

• Authors: Asma Jodeiri Akbarfam, Hoda Maleki
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2407.17699
• Pdf URL: https://arxiv.org/pdf/2407.17699
• Copy Paste: [[2407.17699]] SOK: Blockchain for Provenance(https://arxiv.org/abs/2407.17699)
• Keywords: secure
• Abstract: Provenance, which traces data from its creation to manipulation, is crucial for ensuring data integrity, reliability, and trustworthiness. It is valuable for single-user applications, collaboration within organizations, and across organizations. Blockchain technology has become a popular choice for implementing provenance due to its distributed, transparent, and immutable nature. Numerous studies on blockchain designs are specifically dedicated to provenance, and specialize in this area. Our goal is to provide a new perspective in blockchain based provenance field by identifying the challenges faced and suggesting future research directions. In this paper, we categorize the problem statement into three main research questions to investigate key issues comprehensively and propose a new outlook on the use of blockchains. The first focuses on challenges in non-collaborative, single-source environments, the second examines implications in collaborative environments and different domains such as supply chain, scientific collaboration and digital forensic, and the last one analyzes communication and data exchange challenges between organizations using different blockchains. The interconnected nature of these research questions ensures a thorough exploration of provenance requirements, leading to more effective and secure systems. After analyzing the requirements of provenance in different environments, we provide future design considerations for provenance-based blockchains, including blockchain type, query mechanisms, provenance capture methods, and domain-specific considerations. We also discuss future work and possible extensions in this field.

Title: Revisiting Machine Unlearning with Dimensional Alignment

• Authors: Seonguk Seo, Dongwan Kim, Bohyung Han
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17710
• Pdf URL: https://arxiv.org/pdf/2407.17710
• Copy Paste: [[2407.17710]] Revisiting Machine Unlearning with Dimensional Alignment(https://arxiv.org/abs/2407.17710)
• Keywords: privacy, robust
• Abstract: Machine unlearning, an emerging research topic focusing on compliance with data privacy regulations, enables trained models to remove the information learned from specific data. While many existing methods indirectly address this issue by intentionally injecting incorrect supervisions, they can drastically and unpredictably alter the decision boundaries and feature spaces, leading to training instability and undesired side effects. To fundamentally approach this task, we first analyze the changes in latent feature spaces between original and retrained models, and observe that the feature representations of samples not involved in training are closely aligned with the feature manifolds of previously seen samples in training. Based on these findings, we introduce a novel evaluation metric for machine unlearning, coined dimensional alignment, which measures the alignment between the eigenspaces of the forget and retain set samples. We employ this metric as a regularizer loss to build a robust and stable unlearning framework, which is further enhanced by integrating a self-distillation loss and an alternating training scheme. Our framework effectively eliminates information from the forget set and preserves knowledge from the retain set. Lastly, we identify critical flaws in established evaluation metrics for machine unlearning, and introduce new evaluation tools that more accurately reflect the fundamental goals of machine unlearning.

Title: Are Large Language Models Possible to Conduct Cognitive Behavioral Therapy?

• Authors: Hao Shen, Zihan Li, Minqiang Yang, Minghui Ni, Yongfeng Tao, Zhengyang Yu, Weihao Zheng, Chen Xu, Bin Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17730
• Pdf URL: https://arxiv.org/pdf/2407.17730
• Copy Paste: [[2407.17730]] Are Large Language Models Possible to Conduct Cognitive Behavioral Therapy?(https://arxiv.org/abs/2407.17730)
• Keywords: large language model
• Abstract: In contemporary society, the issue of psychological health has become increasingly prominent, characterized by the diversification, complexity, and universality of mental disorders. Cognitive Behavioral Therapy (CBT), currently the most influential and clinically effective psychological treatment method with no side effects, has limited coverage and poor quality in most countries. In recent years, researches on the recognition and intervention of emotional disorders using large language models (LLMs) have been validated, providing new possibilities for psychological assistance therapy. However, are LLMs truly possible to conduct cognitive behavioral therapy? Many concerns have been raised by mental health experts regarding the use of LLMs for therapy. Seeking to answer this question, we collected real CBT corpus from online video websites, designed and conducted a targeted automatic evaluation framework involving the evaluation of emotion tendency of generated text, structured dialogue pattern and proactive inquiry ability. For emotion tendency, we calculate the emotion tendency score of the CBT dialogue text generated by each model. For structured dialogue pattern, we use a diverse range of automatic evaluation metrics to compare speaking style, the ability to maintain consistency of topic and the use of technology in CBT between different models . As for inquiring to guide the patient, we utilize PQA (Proactive Questioning Ability) metric. We also evaluated the CBT ability of the LLM after integrating a CBT knowledge base to explore the help of introducing additional knowledge to enhance the model's CBT counseling ability. Four LLM variants with excellent performance on natural language processing are evaluated, and the experimental result shows the great potential of LLMs in psychological counseling realm, especially after combining with other technological means.

Title: Enhancing Fine-grained Object Detection in Aerial Images via Orthogonal Mapping

• Authors: Haoran Zhu, Yifan Zhou, Chang Xu, Ruixiang Zhang, Wen Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17738
• Pdf URL: https://arxiv.org/pdf/2407.17738
• Copy Paste: [[2407.17738]] Enhancing Fine-grained Object Detection in Aerial Images via Orthogonal Mapping(https://arxiv.org/abs/2407.17738)
• Keywords: fair
• Abstract: Fine-Grained Object Detection (FGOD) is a critical task in high-resolution aerial image analysis. This letter introduces Orthogonal Mapping (OM), a simple yet effective method aimed at addressing the challenge of semantic confusion inherent in FGOD. OM introduces orthogonal constraints in the feature space by decoupling features from the last layer of the classification branch with a class-wise orthogonal vector basis. This effectively mitigates semantic confusion and enhances classification accuracy. Moreover, OM can be seamlessly integrated into mainstream object detectors. Extensive experiments conducted on three FGOD datasets (FAIR1M, ShipRSImageNet, and MAR20) demonstrate the effectiveness and superiority of the proposed approach. Notably, with just one line of code, OM achieves a 4.08% improvement in mean Average Precision (mAP) over FCOS on the ShipRSImageNet dataset. Codes are released at this https URL.

Title: DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations

• Authors: Guogang Zhu, Xuefeng Liu, Jianwei Niu, Shaojie Tang, Xinghao Wu, Jiayuan Zhang
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2407.17754
• Pdf URL: https://arxiv.org/pdf/2407.17754
• Copy Paste: [[2407.17754]] DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations(https://arxiv.org/abs/2407.17754)
• Keywords: federate
• Abstract: In personalized federated learning (PFL), it is widely recognized that achieving both high model generalization and effective personalization poses a significant challenge due to their conflicting nature. As a result, existing PFL methods can only manage a trade-off between these two objectives. This raises an interesting question: Is it feasible to develop a model capable of achieving both objectives simultaneously? Our paper presents an affirmative answer, and the key lies in the observation that deep models inherently exhibit hierarchical architectures, which produce representations with various levels of generalization and personalization at different stages. A straightforward approach stemming from this observation is to select multiple representations from these layers and combine them to concurrently achieve generalization and personalization. However, the number of candidate representations is commonly huge, which makes this method infeasible due to high computational this http URL address this problem, we propose DualFed, a new method that can directly yield dual representations correspond to generalization and personalization respectively, thereby simplifying the optimization task. Specifically, DualFed inserts a personalized projection network between the encoder and classifier. The pre-projection representations are able to capture generalized information shareable across clients, and the post-projection representations are effective to capture task-specific information on local clients. This design minimizes the mutual interference between generalization and personalization, thereby achieving a win-win situation. Extensive experiments show that DualFed can outperform other FL methods. Code is available at this https URL.

Title: Enhancing Eye Disease Diagnosis with Deep Learning and Synthetic Data Augmentation

• Authors: Saideep Kilaru, Kothamasu Jayachandra, Tanishka Yagneshwar, Suchi Kumari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17755
• Pdf URL: https://arxiv.org/pdf/2407.17755
• Copy Paste: [[2407.17755]] Enhancing Eye Disease Diagnosis with Deep Learning and Synthetic Data Augmentation(https://arxiv.org/abs/2407.17755)
• Keywords: generative
• Abstract: In recent years, the focus is on improving the diagnosis of diabetic retinopathy (DR) using machine learning and deep learning technologies. Researchers have explored various approaches, including the use of high-definition medical imaging, AI-driven algorithms such as convolutional neural networks (CNNs) and generative adversarial networks (GANs). Among all the available tools, CNNs have emerged as a preferred tool due to their superior classification accuracy and efficiency. Although the accuracy of CNNs is comparatively better but it can be improved by introducing some hybrid models by combining various machine learning and deep learning models. Therefore, in this paper, an ensemble learning technique is proposed for early detection and management of DR with higher accuracy. The proposed model is tested on the APTOS dataset and it is showing supremacy on the validation accuracy ($99\%)$ in comparison to the previous models. Hence, the model can be helpful for early detection and treatment of the DR, thereby enhancing the overall quality of care for affected individuals.

Title: CRASH: Crash Recognition and Anticipation System Harnessing with Context-Aware and Temporal Focus Attentions

• Authors: Haicheng Liao, Haoyu Sun, Huanming Shen, Chengyue Wang, Kahou Tam, Chunlin Tian, Li Li, Chengzhong Xu, Zhenning Li
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.17757
• Pdf URL: https://arxiv.org/pdf/2407.17757
• Copy Paste: [[2407.17757]] CRASH: Crash Recognition and Anticipation System Harnessing with Context-Aware and Temporal Focus Attentions(https://arxiv.org/abs/2407.17757)
• Keywords: robust
• Abstract: Accurately and promptly predicting accidents among surrounding traffic agents from camera footage is crucial for the safety of autonomous vehicles (AVs). This task presents substantial challenges stemming from the unpredictable nature of traffic accidents, their long-tail distribution, the intricacies of traffic scene dynamics, and the inherently constrained field of vision of onboard cameras. To address these challenges, this study introduces a novel accident anticipation framework for AVs, termed CRASH. It seamlessly integrates five components: object detector, feature extractor, object-aware module, context-aware module, and multi-layer fusion. Specifically, we develop the object-aware module to prioritize high-risk objects in complex and ambiguous environments by calculating the spatial-temporal relationships between traffic agents. In parallel, the context-aware is also devised to extend global visual information from the temporal to the frequency domain using the Fast Fourier Transform (FFT) and capture fine-grained visual features of potential objects and broader context cues within traffic scenes. To capture a wider range of visual cues, we further propose a multi-layer fusion that dynamically computes the temporal dependencies between different scenes and iteratively updates the correlations between different visual features for accurate and timely accident prediction. Evaluated on real-world datasets--Dashcam Accident Dataset (DAD), Car Crash Dataset (CCD), and AnAn Accident Detection (A3D) datasets--our model surpasses existing top baselines in critical evaluation metrics like Average Precision (AP) and mean Time-To-Accident (mTTA). Importantly, its robustness and adaptability are particularly evident in challenging driving scenarios with missing or limited training data, demonstrating significant potential for application in real-world autonomous driving systems.

Title: Towards the Blockchain Massive Adoption with Permissionless Storage

• Authors: Jia Kan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.17761
• Pdf URL: https://arxiv.org/pdf/2407.17761
• Copy Paste: [[2407.17761]] Towards the Blockchain Massive Adoption with Permissionless Storage(https://arxiv.org/abs/2407.17761)
• Keywords: security, attack
• Abstract: Blockchain technology emerged with the advent of Bitcoin and rapidly developed over the past few decades, becoming widely accepted and known by the public. However, in the past decades, the massive adoption of blockchain technology has yet to come. Rather than the scalability issue, the blockchain application is challenged by its expensive usage cost. However, the high cost of blockchain usage is deeply connected with the blockchain consensus and security mechanism. The permissionless blockchain must maintain its high cost for security against the 51% Attack. Chain users indirectly cover the cost as coins are appointed for blockchain usage fees. This conflict prevents the massive adoption of blockchain. Thus, blockchain must be improved to solve those problems: 1. The cost of blockchain usage should be low enough. 2. The blockchain should remain decentralized. 3. The scalability of blockchain must meet the demand. In my thesis, new approaches are applied to solve the issues above. The key contribution is the discovery of the useful PoW. It extends the Nakamoto PoW with another usage of file data encoding during the same Nakamoto Consensus computation to prove honest data preservation. Based on this theory, a permissionless storage network is proposed as the new security engine for the blockchain. It bridges the high blockchain security cost to the storage users with real demands who are willing to pay for the storage resource. On the other hand, the chain users can benefit from the low transaction fee. Meanwhile, we also provide a scalability solution to shard the blockchain. It enables high TPS and keeps decentralization. The solutions in this thesis provide the answers to all the dependencies of the massive adoption.

Title: Mpox Detection Advanced: Rapid Epidemic Response Through Synthetic Data

• Authors: Yudara Kularathne, Prathapa Janitha, Sithira Ambepitiya, Prarththanan Sothyrajah, Thanveer Ahamed, Dinuka Wijesundara
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17762
• Pdf URL: https://arxiv.org/pdf/2407.17762
• Copy Paste: [[2407.17762]] Mpox Detection Advanced: Rapid Epidemic Response Through Synthetic Data(https://arxiv.org/abs/2407.17762)
• Keywords: robust, fair, diffusion
• Abstract: Rapid development of disease detection models using computer vision is crucial in responding to medical emergencies, such as epidemics or bioterrorism events. Traditional data collection methods are often too slow in these scenarios, requiring innovative approaches for quick, reliable model generation from minimal data. Our study introduces a novel approach by constructing a comprehensive computer vision model to detect Mpox lesions using only synthetic data. Initially, these models generated a diverse set of synthetic images representing Mpox lesions on various body parts (face, back, chest, leg, neck, arm) across different skin tones as defined by the Fitzpatrick scale (fair, brown, dark skin). Subsequently, we trained and tested a vision model with this synthetic dataset to evaluate the diffusion models' efficacy in producing high-quality training data and its impact on the vision model's medical image recognition performance. The results were promising; the vision model achieved a 97% accuracy rate, with 96% precision and recall for Mpox cases, and similarly high metrics for normal and other skin disorder cases, demonstrating its ability to correctly identify true positives and minimize false positives. The model achieved an F1-Score of 96% for Mpox cases and 98% for normal and other skin disorders, reflecting a balanced precision-recall relationship, thus ensuring reliability and robustness in its predictions. Our proposed SynthVision methodology indicates the potential to develop accurate computer vision models with minimal data input for future medical emergencies.

Title: How Lightweight Can A Vision Transformer Be

• Authors: Jen Hong Tan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17783
• Pdf URL: https://arxiv.org/pdf/2407.17783
• Copy Paste: [[2407.17783]] How Lightweight Can A Vision Transformer Be(https://arxiv.org/abs/2407.17783)
• Keywords: transformer
• Abstract: In this paper, we explore a strategy that uses Mixture-of-Experts (MoE) to streamline, rather than augment, vision transformers. Each expert in an MoE layer is a SwiGLU feedforward network, where V and W2 are shared across the layer. No complex attention or convolutional mechanisms are employed. Depth-wise scaling is applied to progressively reduce the size of the hidden layer and the number of experts is increased in stages. Grouped query attention is used. We studied the proposed approach with and without pre-training on small datasets and investigated whether transfer learning works at this scale. We found that the architecture is competitive even at a size of 0.67M parameters.

Title: Topology-Preserving Downsampling of Binary Images

• Authors: Chia-Chia Chen, Chi-Han Peng
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2407.17786
• Pdf URL: https://arxiv.org/pdf/2407.17786
• Copy Paste: [[2407.17786]] Topology-Preserving Downsampling of Binary Images(https://arxiv.org/abs/2407.17786)
• Keywords: segmentation
• Abstract: We present a novel discrete optimization-based approach to generate downsampled versions of binary images that are guaranteed to have the same topology as the original, measured by the zeroth and first Betti numbers of the black regions, while having good similarity to the original image as measured by IoU and Dice scores. To our best knowledge, all existing binary image downsampling methods do not have such topology-preserving guarantees. We also implemented a baseline morphological operation (dilation)-based approach that always generates topologically correct results. However, we found the similarity scores to be much worse. We demonstrate several applications of our approach. First, generating smaller versions of medical image segmentation masks for easier human inspection. Second, improving the efficiency of binary image operations, including persistent homology computation and shortest path computation, by substituting the original images with smaller ones. In particular, the latter is a novel application that is made feasible only by the full topology-preservation guarantee of our method.

Title: PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation

• Authors: Junjie Huang, Quanyan Zhu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.17788
• Pdf URL: https://arxiv.org/pdf/2407.17788
• Copy Paste: [[2407.17788]] PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediation(https://arxiv.org/abs/2407.17788)
• Keywords: security, defense, attack, large language model
• Abstract: Recent advances in Large Language Models (LLMs) have shown significant potential in enhancing cybersecurity defenses against sophisticated threats. LLM-based penetration testing is an essential step in automating system security evaluations by identifying vulnerabilities. Remediation, the subsequent crucial step, addresses these discovered vulnerabilities. Since details about vulnerabilities, exploitation methods, and software versions offer crucial insights into system weaknesses, integrating penetration testing with vulnerability remediation into a cohesive system has become both intuitive and necessary. This paper introduces PenHeal, a two-stage LLM-based framework designed to autonomously identify and mitigate security vulnerabilities. The framework integrates two LLM-enabled components: the Pentest Module, which detects multiple vulnerabilities within a system, and the Remediation Module, which recommends optimal remediation strategies. The integration is facilitated through Counterfactual Prompting and an Instructor module that guides the LLMs using external knowledge to explore multiple potential attack paths effectively. Our experimental results demonstrate that PenHeal not only automates the identification and remediation of vulnerabilities but also significantly improves vulnerability coverage by 31%, increases the effectiveness of remediation strategies by 32%, and reduces the associated costs by 46% compared to baseline models. These outcomes highlight the transformative potential of LLMs in reshaping cybersecurity practices, offering an innovative solution to defend against cyber threats.

Title: Harnessing Temporal Causality for Advanced Temporal Action Detection

• Authors: Shuming Liu, Lin Sui, Chen-Lin Zhang, Fangzhou Mu, Chen Zhao, Bernard Ghanem
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17792
• Pdf URL: https://arxiv.org/pdf/2407.17792
• Copy Paste: [[2407.17792]] Harnessing Temporal Causality for Advanced Temporal Action Detection(https://arxiv.org/abs/2407.17792)
• Keywords: transformer
• Abstract: As a fundamental task in long-form video understanding, temporal action detection (TAD) aims to capture inherent temporal relations in untrimmed videos and identify candidate actions with precise boundaries. Over the years, various networks, including convolutions, graphs, and transformers, have been explored for effective temporal modeling for TAD. However, these modules typically treat past and future information equally, overlooking the crucial fact that changes in action boundaries are essentially causal events. Inspired by this insight, we propose leveraging the temporal causality of actions to enhance TAD representation by restricting the model's access to only past or future context. We introduce CausalTAD, which combines causal attention and causal Mamba to achieve state-of-the-art performance on multiple benchmarks. Notably, with CausalTAD, we ranked 1st in the Action Recognition, Action Detection, and Audio-Based Interaction Detection tracks at the EPIC-Kitchens Challenge 2024, as well as 1st in the Moment Queries track at the Ego4D Challenge 2024. Our code is available at this https URL.

Title: Enhancing Diversity in Multi-objective Feature Selection

• Authors: Sevil Zanjani Miyandoab, Shahryar Rahnamayan, Azam Asilian Bidgoli, Sevda Ebrahimi, Masoud Makrehchi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17795
• Pdf URL: https://arxiv.org/pdf/2407.17795
• Copy Paste: [[2407.17795]] Enhancing Diversity in Multi-objective Feature Selection(https://arxiv.org/abs/2407.17795)
• Keywords: interpretability
• Abstract: Feature selection plays a pivotal role in the data preprocessing and model-building pipeline, significantly enhancing model performance, interpretability, and resource efficiency across diverse domains. In population-based optimization methods, the generation of diverse individuals holds utmost importance for adequately exploring the problem landscape, particularly in highly multi-modal multi-objective optimization problems. Our study reveals that, in line with findings from several prior research papers, commonly employed crossover and mutation operations lack the capability to generate high-quality diverse individuals and tend to become confined to limited areas around various local optima. This paper introduces an augmentation to the diversity of the population in the well-established multi-objective scheme of the genetic algorithm, NSGA-II. This enhancement is achieved through two key components: the genuine initialization method and the substitution of the worst individuals with new randomly generated individuals as a re-initialization approach in each generation. The proposed multi-objective feature selection method undergoes testing on twelve real-world classification problems, with the number of features ranging from 2,400 to nearly 50,000. The results demonstrate that replacing the last front of the population with an equivalent number of new random individuals generated using the genuine initialization method and featuring a limited number of features substantially improves the population's quality and, consequently, enhances the performance of the multi-objective algorithm.

Title: A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models

• Authors: Haonan Zheng, Xinyang Deng, Wen Jiang, Wenrui Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17797
• Pdf URL: https://arxiv.org/pdf/2407.17797
• Copy Paste: [[2407.17797]] A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models(https://arxiv.org/abs/2407.17797)
• Keywords: security, attack, robust
• Abstract: With Vision-Language Pre-training (VLP) models demonstrating powerful multimodal interaction capabilities, the application scenarios of neural networks are no longer confined to unimodal domains but have expanded to more complex multimodal V+L downstream tasks. The security vulnerabilities of unimodal models have been extensively examined, whereas those of VLP models remain challenging. We note that in CV models, the understanding of images comes from annotated information, while VLP models are designed to learn image representations directly from raw text. Motivated by this discrepancy, we developed the Feature Guidance Attack (FGA), a novel method that uses text representations to direct the perturbation of clean images, resulting in the generation of adversarial images. FGA is orthogonal to many advanced attack strategies in the unimodal domain, facilitating the direct application of rich research findings from the unimodal to the multimodal scenario. By appropriately introducing text attack into FGA, we construct Feature Guidance with Text Attack (FGA-T). Through the interaction of attacking two modalities, FGA-T achieves superior attack effects against VLP models. Moreover, incorporating data augmentation and momentum mechanisms significantly improves the black-box transferability of FGA-T. Our method demonstrates stable and effective attack capabilities across various datasets, downstream tasks, and both black-box and white-box settings, offering a unified baseline for exploring the robustness of VLP models.

Title: Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning

• Authors: Vedanshu, MM Tripathi, Bhavnesh Jaint
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17813
• Pdf URL: https://arxiv.org/pdf/2407.17813
• Copy Paste: [[2407.17813]] Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning(https://arxiv.org/abs/2407.17813)
• Keywords: robust, large language model
• Abstract: The integration of large language models (LLMs) with vision-language (VL) tasks has been a transformative development in the realm of artificial intelligence, highlighting the potential of LLMs as a versatile general-purpose chatbot. However, the current trend in this evolution focuses on the integration of vision and language to create models that can operate in more diverse and real-world contexts. We present a novel approach, termed Bottleneck Adapter, specifically crafted for enhancing the multimodal functionalities of these complex models, enabling joint optimization of the entire multimodal LLM framework through a process known as Multimodal Model Tuning (MMT). Our approach utilizes lightweight adapters to connect the image encoder and LLM without the need for large, complex neural networks. Unlike the conventional modular training schemes, our approach adopts an end-to-end optimization regime, which, when combined with the adapters, facilitates the joint optimization using a significantly smaller parameter set. Our method exhibits robust performance with 90.12\% accuracy, outperforming both human-level performance (88.4\%) and LaVIN-7B (89.41\%).

Title: Demystifying Verbatim Memorization in Large Language Models

• Authors: Jing Huang, Diyi Yang, Christopher Potts
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17817
• Pdf URL: https://arxiv.org/pdf/2407.17817
• Copy Paste: [[2407.17817]] Demystifying Verbatim Memorization in Large Language Models(https://arxiv.org/abs/2407.17817)
• Keywords: privacy, large language model
• Abstract: Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications. Much prior work has studied such verbatim memorization using observational data. To complement such work, we develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to verbatim memorize sequences, even for out-of-distribution sequences; (3) the generation of memorized sequences is triggered by distributed model states that encode high-level features and makes important use of general language modeling capabilities. Guided by these insights, we develop stress tests to evaluate unlearning methods and find they often fail to remove the verbatim memorized information, while also degrading the LM. Overall, these findings challenge the hypothesis that verbatim memorization stems from specific model weights or mechanisms. Rather, verbatim memorization is intertwined with the LM's general capabilities and thus will be very difficult to isolate and suppress without degrading model quality.

Title: Advanced deep-reinforcement-learning methods for flow control: group-invariant and positional-encoding networks improve learning speed and quality

• Authors: Joogoo Jeon, Jean Rabault, Joel Vasanth, Francisco Alcántara-Ávila, Shilaj Baral, Ricardo Vinuesa
• Subjects: cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2407.17822
• Pdf URL: https://arxiv.org/pdf/2407.17822
• Copy Paste: [[2407.17822]] Advanced deep-reinforcement-learning methods for flow control: group-invariant and positional-encoding networks improve learning speed and quality(https://arxiv.org/abs/2407.17822)
• Keywords: transformer
• Abstract: Flow control is key to maximize energy efficiency in a wide range of applications. However, traditional flow-control methods face significant challenges in addressing non-linear systems and high-dimensional data, limiting their application in realistic energy systems. This study advances deep-reinforcement-learning (DRL) methods for flow control, particularly focusing on integrating group-invariant networks and positional encoding into DRL architectures. Our methods leverage multi-agent reinforcement learning (MARL) to exploit policy invariance in space, in combination with group-invariant networks to ensure local symmetry invariance. Additionally, a positional encoding inspired by the transformer architecture is incorporated to provide location information to the agents, mitigating action constraints from strict invariance. The proposed methods are verified using a case study of Rayleigh-Bénard convection, where the goal is to minimize the Nusselt number Nu. The group-invariant neural networks (GI-NNs) show faster convergence compared to the base MARL, achieving better average policy performance. The GI-NNs not only cut DRL training time in half but also notably enhance learning reproducibility. Positional encoding further enhances these results, effectively reducing the minimum Nu and stabilizing convergence. Interestingly, group invariant networks specialize in improving learning speed and positional encoding specializes in improving learning quality. These results demonstrate that choosing a suitable feature-representation method according to the purpose as well as the characteristics of each control problem is essential. We believe that the results of this study will not only inspire novel DRL methods with invariant and unique representations, but also provide useful insights for industrial applications.

Title: Unified Lexical Representation for Interpretable Visual-Language Alignment

• Authors: Yifan Li, Yikai Wang, Yanwei Fu, Dongyu Ru, Zheng Zhang, Tong He
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17827
• Pdf URL: https://arxiv.org/pdf/2407.17827
• Copy Paste: [[2407.17827]] Unified Lexical Representation for Interpretable Visual-Language Alignment(https://arxiv.org/abs/2407.17827)
• Keywords: generative
• Abstract: Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work. Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores. On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words. However, lexical representations is difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively. In this paper, we introduce LexVLA, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design. We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability. To avoid the false discovery, we propose an overuse penalty to refrain the lexical representation from falsely frequently activating meaningless words. We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on modest multi-modal dataset and avoid intricate training configurations. On cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g., 1.1B data, including CC-12M). We conduct extensive experiments to analyze LexVLA.

Title: Image Segmentation via Divisive Normalization: dealing with environmental diversity

• Authors: Pablo Hernández-Cámara, Jorge Vila-Tomás, Paula Dauden-Oliver, Nuria Alabau-Bosque, Valero Laparra, Jesús Malo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17829
• Pdf URL: https://arxiv.org/pdf/2407.17829
• Copy Paste: [[2407.17829]] Image Segmentation via Divisive Normalization: dealing with environmental diversity(https://arxiv.org/abs/2407.17829)
• Keywords: segmentation
• Abstract: Autonomous driving is a challenging scenario for image segmentation due to the presence of uncontrolled environmental conditions and the eventually catastrophic consequences of failures. Previous work suggested that a biologically motivated computation, the so-called Divisive Normalization, could be useful to deal with image variability, but its effects have not been systematically studied over different data sources and environmental factors. Here we put segmentation U-nets augmented with Divisive Normalization to work far from training conditions to find where this adaptation is more critical. We categorize the scenes according to their radiance level and dynamic range (day/night), and according to their achromatic/chromatic contrasts. We also consider video game (synthetic) images to broaden the range of environments. We check the performance in the extreme percentiles of such categorization. Then, we push the limits further by artificially modifying the images in perceptually/environmentally relevant dimensions: luminance, contrasts and spectral radiance. Results show that neural networks with Divisive Normalization get better results in all the scenarios and their performance remains more stable with regard to the considered environmental factors and nature of the source. Finally, we explain the improvements in segmentation performance in two ways: (1) by quantifying the invariance of the responses that incorporate Divisive Normalization, and (2) by illustrating the adaptive nonlinearity of the different layers that depends on the local activity.

Title: UMono: Physical Model Informed Hybrid CNN-Transformer Framework for Underwater Monocular Depth Estimation

• Authors: Jian Wang, Jing Wang, Shenghui Rong, Bo He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17838
• Pdf URL: https://arxiv.org/pdf/2407.17838
• Copy Paste: [[2407.17838]] UMono: Physical Model Informed Hybrid CNN-Transformer Framework for Underwater Monocular Depth Estimation(https://arxiv.org/abs/2407.17838)
• Keywords: transformer
• Abstract: Underwater monocular depth estimation serves as the foundation for tasks such as 3D reconstruction of underwater scenes. However, due to the influence of light and medium, the underwater environment undergoes a distinctive imaging process, which presents challenges in accurately estimating depth from a single image. The existing methods fail to consider the unique characteristics of underwater environments, leading to inadequate estimation results and limited generalization performance. Furthermore, underwater depth estimation requires extracting and fusing both local and global features, which is not fully explored in existing methods. In this paper, an end-to-end learning framework for underwater monocular depth estimation called UMono is presented, which incorporates underwater image formation model characteristics into network architecture, and effectively utilize both local and global features of underwater image. Experimental results demonstrate that the proposed method is effective for underwater monocular depth estimation and outperforms the existing methods in both quantitative and qualitative analyses.

Title: DragText: Rethinking Text Embedding in Point-based Image Editing

• Authors: Gayoon Choi, Taejin Jeong, Sujung Hong, Jaehoon Joo, Seong Jae Hwang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17843
• Pdf URL: https://arxiv.org/pdf/2407.17843
• Copy Paste: [[2407.17843]] DragText: Rethinking Text Embedding in Point-based Image Editing(https://arxiv.org/abs/2407.17843)
• Keywords: diffusion
• Abstract: Point-based image editing enables accurate and flexible control through content dragging. However, the role of text embedding in the editing process has not been thoroughly investigated. A significant aspect that remains unexplored is the interaction between text and image embeddings. In this study, we show that during the progressive editing of an input image in a diffusion model, the text embedding remains constant. As the image embedding increasingly diverges from its initial state, the discrepancy between the image and text embeddings presents a significant challenge. Moreover, we found that the text prompt significantly influences the dragging process, particularly in maintaining content integrity and achieving the desired manipulation. To utilize these insights, we propose DragText, which optimizes text embedding in conjunction with the dragging process to pair with the modified image embedding. Simultaneously, we regularize the text optimization process to preserve the integrity of the original text prompt. Our approach can be seamlessly integrated with existing diffusion-based drag methods with only a few lines of code.

Title: FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editing

• Authors: Gwanhyeong Koo, Sunjae Yoon, Ji Woo Hong, Chang D. Yoo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17850
• Pdf URL: https://arxiv.org/pdf/2407.17850
• Copy Paste: [[2407.17850]] FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editing(https://arxiv.org/abs/2407.17850)
• Keywords: diffusion
• Abstract: Current image editing methods primarily utilize DDIM Inversion, employing a two-branch diffusion approach to preserve the attributes and layout of the original image. However, these methods encounter challenges with non-rigid edits, which involve altering the image's layout or structure. Our comprehensive analysis reveals that the high-frequency components of DDIM latent, crucial for retaining the original image's key features and layout, significantly contribute to these limitations. Addressing this, we introduce FlexiEdit, which enhances fidelity to input text prompts by refining DDIM latent, by reducing high-frequency components in targeted editing areas. FlexiEdit comprises two key components: (1) Latent Refinement, which modifies DDIM latent to better accommodate layout adjustments, and (2) Edit Fidelity Enhancement via Re-inversion, aimed at ensuring the edits more accurately reflect the input text prompts. Our approach represents notable progress in image editing, particularly in performing complex non-rigid edits, showcasing its enhanced capability through comparative experiments.

Title: MDS-ED: Multimodal Decision Support in the Emergency Department -- a Benchmark Dataset for Diagnoses and Deterioration Prediction in Emergency Medicine

• Authors: Juan Miguel Lopez Alcaraz, Nils Strodthoff
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2407.17856
• Pdf URL: https://arxiv.org/pdf/2407.17856
• Copy Paste: [[2407.17856]] MDS-ED: Multimodal Decision Support in the Emergency Department -- a Benchmark Dataset for Diagnoses and Deterioration Prediction in Emergency Medicine(https://arxiv.org/abs/2407.17856)
• Keywords: robust, biometric
• Abstract: Background: Benchmarking medical decision support algorithms often struggles due to limited access to datasets, narrow prediction tasks, and restricted input modalities. These limitations affect their clinical relevance and performance in high-stakes areas like emergency care, complicating replication, validation, and improvement of benchmarks. Methods: We introduce a dataset based on MIMIC-IV, benchmarking protocol, and initial results for evaluating multimodal decision support in the emergency department (ED). We use diverse data modalities from the first 1.5 hours of patient arrival, including demographics, biometrics, vital signs, lab values, and electrocardiogram waveforms. We analyze 1443 clinical labels across two contexts: predicting diagnoses with ICD-10 codes and forecasting patient deterioration. Results: Our multimodal diagnostic model achieves an AUROC score over 0.8 in a statistically significant manner for 357 out of 1428 conditions, including cardiac issues like myocardial infarction and non-cardiac conditions such as renal disease and diabetes. The deterioration model scores above 0.8 in a statistically significant manner for 13 out of 15 targets, including critical events like cardiac arrest and mechanical ventilation, ICU admission as well as short- and long-term mortality. Incorporating raw waveform data significantly improves model performance, which represents one of the first robust demonstrations of this effect. Conclusions: This study highlights the uniqueness of our dataset, which encompasses a wide range of clinical tasks and utilizes a comprehensive set of features collected early during the emergency after arriving at the ED. The strong performance, as evidenced by high AUROC scores across diagnostic and deterioration targets, underscores the potential of our approach to revolutionize decision-making in acute and emergency medicine.

Title: factgenie: A Framework for Span-based Evaluation of Generated Texts

• Authors: Zdeněk Kasner, Ondřej Plátek, Patrícia Schmidtová, Simone Balloccu, Ondřej Dušek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17863
• Pdf URL: https://arxiv.org/pdf/2407.17863
• Copy Paste: [[2407.17863]] factgenie: A Framework for Span-based Evaluation of Generated Texts(https://arxiv.org/abs/2407.17863)
• Keywords: large language model
• Abstract: We present factgenie: a framework for annotating and visualizing word spans in textual model outputs. Annotations can capture various span-based phenomena such as semantic inaccuracies or irrelevant text. With factgenie, the annotations can be collected both from human crowdworkers and large language models. Our framework consists of a web interface for data visualization and gathering text annotations, powered by an easily extensible codebase.

Title: Is the Digital Forensics and Incident Response Pipeline Ready for Text-Based Threats in LLM Era?

• Authors: Avanti Bhandarkar, Ronald Wilson, Anushka Swarup, Mengdi Zhu, Damon Woodard
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17870
• Pdf URL: https://arxiv.org/pdf/2407.17870
• Copy Paste: [[2407.17870]] Is the Digital Forensics and Incident Response Pipeline Ready for Text-Based Threats in LLM Era?(https://arxiv.org/abs/2407.17870)
• Keywords: security, attack, generative
• Abstract: In the era of generative AI, the widespread adoption of Neural Text Generators (NTGs) presents new cybersecurity challenges, particularly within the realms of Digital Forensics and Incident Response (DFIR). These challenges primarily involve the detection and attribution of sources behind advanced attacks like spearphishing and disinformation campaigns. As NTGs evolve, the task of distinguishing between human and NTG-authored texts becomes critically complex. This paper rigorously evaluates the DFIR pipeline tailored for text-based security systems, specifically focusing on the challenges of detecting and attributing authorship of NTG-authored texts. By introducing a novel human-NTG co-authorship text attack, termed CS-ACT, our study uncovers significant vulnerabilities in traditional DFIR methodologies, highlighting discrepancies between ideal scenarios and real-world conditions. Utilizing 14 diverse datasets and 43 unique NTGs, up to the latest GPT-4, our research identifies substantial vulnerabilities in the forensic profiling phase, particularly in attributing authorship to NTGs. Our comprehensive evaluation points to factors such as model sophistication and the lack of distinctive style within NTGs as significant contributors for these vulnerabilities. Our findings underscore the necessity for more sophisticated and adaptable strategies, such as incorporating adversarial learning, stylizing NTGs, and implementing hierarchical attribution through the mapping of NTG lineages to enhance source attribution. This sets the stage for future research and the development of more resilient text-based security systems.

Title: Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions

• Authors: Jiwon Suh, Injae Na, Woohwan Jung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17874
• Pdf URL: https://arxiv.org/pdf/2407.17874
• Copy Paste: [[2407.17874]] Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions(https://arxiv.org/abs/2407.17874)
• Keywords: large language model
• Abstract: End-to-end automatic speech recognition (E2E ASR) systems have significantly improved speech recognition through training on extensive datasets. Despite these advancements, they still struggle to accurately recognize domain specific words, such as proper nouns and technical terminologies. To address this problem, we propose a method to utilize the state-of-the-art Whisper without modifying its architecture, preserving its generalization performance while enabling it to leverage descriptions effectively. Moreover, we propose two additional training techniques to improve the domain specific ASR: decoder fine-tuning, and context perturbation. We also propose a method to use a Large Language Model (LLM) to generate descriptions with simple metadata, when descriptions are unavailable. Our experiments demonstrate that proposed methods notably enhance domain-specific ASR accuracy on real-life datasets, with LLM-generated descriptions outperforming human-crafted ones in effectiveness.

Title: Advancing 3D Point Cloud Understanding through Deep Transfer Learning: A Comprehensive Survey

• Authors: Shahab Saquib Sohail, Yassine Himeur, Hamza Kheddar, Abbes Amira, Fodil Fadli, Shadi Atalla, Abigail Copiaco, Wathiq Mansoor
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17877
• Pdf URL: https://arxiv.org/pdf/2407.17877
• Copy Paste: [[2407.17877]] Advancing 3D Point Cloud Understanding through Deep Transfer Learning: A Comprehensive Survey(https://arxiv.org/abs/2407.17877)
• Keywords: segmentation
• Abstract: The 3D point cloud (3DPC) has significantly evolved and benefited from the advance of deep learning (DL). However, the latter faces various issues, including the lack of data or annotated data, the existence of a significant gap between training data and test data, and the requirement for high computational resources. To that end, deep transfer learning (DTL), which decreases dependency and costs by utilizing knowledge gained from a source data/task in training a target data/task, has been widely investigated. Numerous DTL frameworks have been suggested for aligning point clouds obtained from several scans of the same scene. Additionally, DA, which is a subset of DTL, has been modified to enhance the point cloud data's quality by dealing with noise and missing points. Ultimately, fine-tuning and DA approaches have demonstrated their effectiveness in addressing the distinct difficulties inherent in point cloud data. This paper presents the first review shedding light on this aspect. it provides a comprehensive overview of the latest techniques for understanding 3DPC using DTL and domain adaptation (DA). Accordingly, DTL's background is first presented along with the datasets and evaluation metrics. A well-defined taxonomy is introduced, and detailed comparisons are presented, considering different aspects such as different knowledge transfer strategies, and performance. The paper covers various applications, such as 3DPC object detection, semantic labeling, segmentation, classification, registration, downsampling/upsampling, and denoising. Furthermore, the article discusses the advantages and limitations of the presented frameworks, identifies open challenges, and suggests potential research directions.

Title: DAM: Towards A Foundation Model for Time Series Forecasting

• Authors: Luke Darlow, Qiwen Deng, Ahmed Hassan, Martin Asenov, Rajkarn Singh, Artjom Joosen, Adam Barker, Amos Storkey
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17880
• Pdf URL: https://arxiv.org/pdf/2407.17880
• Copy Paste: [[2407.17880]] DAM: Towards A Foundation Model for Time Series Forecasting(https://arxiv.org/abs/2407.17880)
• Keywords: robust, transformer
• Abstract: It is challenging to scale time series forecasting models such that they forecast accurately for multiple distinct domains and datasets, all with potentially different underlying collection procedures (e.g., sample resolution), patterns (e.g., periodicity), and prediction requirements (e.g., reconstruction vs. forecasting). We call this general task universal forecasting. Existing methods usually assume that input data is regularly sampled, and they forecast to pre-determined horizons, resulting in failure to generalise outside of the scope of their training. We propose the DAM - a neural model that takes randomly sampled histories and outputs an adjustable basis composition as a continuous function of time for forecasting to non-fixed horizons. It involves three key components: (1) a flexible approach for using randomly sampled histories from a long-tail distribution, that enables an efficient global perspective of the underlying temporal dynamics while retaining focus on the recent history; (2) a transformer backbone that is trained on these actively sampled histories to produce, as representational output, (3) the basis coefficients of a continuous function of time. We show that a single univariate DAM, trained on 25 time series datasets, either outperformed or closely matched existing SoTA models at multivariate long-term forecasting across 18 datasets, including 8 held-out for zero-shot transfer, even though these models were trained to specialise for each dataset-horizon combination. This single DAM excels at zero-shot transfer and very-long-term forecasting, performs well at imputation, is interpretable via basis function composition and attention, can be tuned for different inference-cost requirements, is robust to missing and irregularly sampled data {by design}.

Title: The Power of Combining Data and Knowledge: GPT-4o is an Effective Interpreter of Machine Learning Models in Predicting Lymph Node Metastasis of Lung Cancer

• Authors: Danqing Hu, Bing Liu, Xiaofeng Zhu, Nan Wu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17900
• Pdf URL: https://arxiv.org/pdf/2407.17900
• Copy Paste: [[2407.17900]] The Power of Combining Data and Knowledge: GPT-4o is an Effective Interpreter of Machine Learning Models in Predicting Lymph Node Metastasis of Lung Cancer(https://arxiv.org/abs/2407.17900)
• Keywords: large language model
• Abstract: Lymph node metastasis (LNM) is a crucial factor in determining the initial treatment for patients with lung cancer, yet accurate preoperative diagnosis of LNM remains challenging. Recently, large language models (LLMs) have garnered significant attention due to their remarkable text generation capabilities. Leveraging the extensive medical knowledge learned from vast corpora, LLMs can estimate probabilities for clinical problems, though their performance has historically been inferior to data-driven machine learning models. In this paper, we propose a novel ensemble method that combines the medical knowledge acquired by LLMs with the latent patterns identified by machine learning models to enhance LNM prediction performance. Initially, we developed machine learning models using patient data. We then designed a prompt template to integrate the patient data with the predicted probability from the machine learning model. Subsequently, we instructed GPT-4o, the most advanced LLM developed by OpenAI, to estimate the likelihood of LNM based on patient data and then adjust the estimate using the machine learning output. Finally, we collected three outputs from the GPT-4o using the same prompt and ensembled these results as the final prediction. Using the proposed method, our models achieved an AUC value of 0.765 and an AP value of 0.415 for LNM prediction, significantly improving predictive performance compared to baseline machine learning models. The experimental results indicate that GPT-4o can effectively leverage its medical knowledge and the probabilities predicted by machine learning models to achieve more accurate LNM predictions. These findings demonstrate that LLMs can perform well in clinical risk prediction tasks, offering a new paradigm for integrating medical knowledge and patient data in clinical predictions.

Title: StreamMOS: Streaming Moving Object Segmentation with Multi-View Perception and Dual-Span Memory

• Authors: Zhiheng Li, Yubo Cui, Jiexi Zhong, Zheng Fang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.17905
• Pdf URL: https://arxiv.org/pdf/2407.17905
• Copy Paste: [[2407.17905]] StreamMOS: Streaming Moving Object Segmentation with Multi-View Perception and Dual-Span Memory(https://arxiv.org/abs/2407.17905)
• Keywords: segmentation
• Abstract: Moving object segmentation based on LiDAR is a crucial and challenging task for autonomous driving and mobile robotics. Most approaches explore spatio-temporal information from LiDAR sequences to predict moving objects in the current frame. However, they often focus on transferring temporal cues in a single inference and regard every prediction as independent of others. This may cause inconsistent segmentation results for the same object in different frames. To overcome this issue, we propose a streaming network with a memory mechanism, called StreamMOS, to build the association of features and predictions among multiple inferences. Specifically, we utilize a short-term memory to convey historical features, which can be regarded as spatial prior of moving objects and adopted to enhance current inference by temporal fusion. Meanwhile, we build a long-term memory to store previous predictions and exploit them to refine the present forecast at voxel and instance levels through voting. Besides, we present multi-view encoder with cascade projection and asymmetric convolution to extract motion feature of objects in different representations. Extensive experiments validate that our algorithm gets competitive performance on SemanticKITTI and Sipailou Campus datasets. Code will be released at this https URL.

Title: Hierarchical Object Detection and Recognition Framework for Practical Plant Disease Diagnosis

• Authors: Kohei Iwano, Shogo Shibuya, Satoshi Kagiwada, Hitoshi Iyatomi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17906
• Pdf URL: https://arxiv.org/pdf/2407.17906
• Copy Paste: [[2407.17906]] Hierarchical Object Detection and Recognition Framework for Practical Plant Disease Diagnosis(https://arxiv.org/abs/2407.17906)
• Keywords: robust
• Abstract: Recently, object detection methods (OD; e.g., YOLO-based models) have been widely utilized in plant disease diagnosis. These methods demonstrate robustness to distance variations and excel at detecting small lesions compared to classification methods (CL; e.g., CNN models). However, there are issues such as low diagnostic performance for hard-to-detect diseases and high labeling costs. Additionally, since healthy cases cannot be explicitly trained, there is a risk of false positives. We propose the Hierarchical object detection and recognition framework (HODRF), a sophisticated and highly integrated two-stage system that combines the strengths of both OD and CL for plant disease diagnosis. In the first stage, HODRF uses OD to identify regions of interest (ROIs) without specifying the disease. In the second stage, CL diagnoses diseases surrounding the ROIs. HODRF offers several advantages: (1) Since OD detects only one type of ROI, HODRF can detect diseases with limited training images by leveraging its ability to identify other lesions. (2) While OD over-detects healthy cases, HODRF significantly reduces these errors by using CL in the second stage. (3) CL's accuracy improves in HODRF as it identifies diagnostic targets given as ROIs, making it less vulnerable to size changes. (4) HODRF benefits from CL's lower annotation costs, allowing it to learn from a larger number of images. We implemented HODRF using YOLOv7 for OD and EfficientNetV2 for CL and evaluated its performance on a large-scale dataset (4 crops, 20 diseased and healthy classes, 281K images). HODRF outperformed YOLOv7 alone by 5.8 to 21.5 points on healthy data and 0.6 to 7.5 points on macro F1 scores, and it improved macro F1 by 1.1 to 7.2 points over EfficientNetV2.

Title: Amortized Posterior Sampling with Diffusion Prior Distillation

• Authors: Abbas Mammadov, Hyungjin Chung, Jong Chul Ye
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17907
• Pdf URL: https://arxiv.org/pdf/2407.17907
• Copy Paste: [[2407.17907]] Amortized Posterior Sampling with Diffusion Prior Distillation(https://arxiv.org/abs/2407.17907)
• Keywords: diffusion
• Abstract: We propose a variational inference approach to sample from the posterior distribution for solving inverse problems. From a pre-trained diffusion model, our approach trains a conditional flow model to minimize the divergence between the proposal variational distribution and the posterior distribution implicitly defined through the diffusion model. Once trained, the flow model is capable of sampling from the posterior distribution with a single NFE, amortized with respect to the measurement. The proposed method paves a new path for distilling a diffusion prior for efficient posterior sampling. We show that our method is applicable to standard signals in Euclidean space, as well as signals on manifold.

Title: Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models

• Authors: Anna Bavaresco, Marianne de Heer Kloots, Sandro Pezzelle, Raquel Fernández
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17914
• Pdf URL: https://arxiv.org/pdf/2407.17914
• Copy Paste: [[2407.17914]] Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models(https://arxiv.org/abs/2407.17914)
• Keywords: generative
• Abstract: Representations from deep neural networks (DNNs) have proven remarkably predictive of neural activity involved in both visual and linguistic processing. Despite these successes, most studies to date concern unimodal DNNs, encoding either visual or textual input but not both. Yet, there is growing evidence that human meaning representations integrate linguistic and sensory-motor information. Here we investigate whether the integration of multimodal information operated by current vision-and-language DNN models (VLMs) leads to representations that are more aligned with human brain activity than those obtained by language-only and vision-only DNNs. We focus on fMRI responses recorded while participants read concept words in the context of either a full sentence or an accompanying picture. Our results reveal that VLM representations correlate more strongly than language- and vision-only DNNs with activations in brain areas functionally related to language processing. A comparison between different types of visuo-linguistic architectures shows that recent generative VLMs tend to be less brain-aligned than previous architectures with lower performance on downstream applications. Moreover, through an additional analysis comparing brain vs. behavioural alignment across multiple VLMs, we show that -- with one remarkable exception -- representations that strongly align with behavioural judgments do not correlate highly with brain responses. This indicates that brain similarity does not go hand in hand with behavioural similarity, and vice versa.

Title: The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models

• Authors: Zihui Wu, Haichang Gao, Jianping He, Ping Wang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17915
• Pdf URL: https://arxiv.org/pdf/2407.17915
• Copy Paste: [[2407.17915]] The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models(https://arxiv.org/abs/2407.17915)
• Keywords: security, attack, large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities, but their power comes with significant security considerations. While extensive research has been conducted on the safety of LLMs in chat mode, the security implications of their function calling feature have been largely overlooked. This paper uncovers a critical vulnerability in the function calling process of LLMs, introducing a novel "jailbreak function" attack method that exploits alignment discrepancies, user coercion, and the absence of rigorous safety filters. Our empirical study, conducted on six state-of-the-art LLMs including GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro, reveals an alarming average success rate of over 90\% for this attack. We provide a comprehensive analysis of why function calls are susceptible to such attacks and propose defensive strategies, including the use of defensive prompts. Our findings highlight the urgent need for enhanced security measures in the function calling capabilities of LLMs, contributing to the field of AI safety by identifying a previously unexplored risk, designing an effective attack method, and suggesting practical defensive measures. Our code is available at this https URL.

Title: Guided Latent Slot Diffusion for Object-Centric Learning

• Authors: Krishnakant Singh, Simone Schaub-Meyer, Stefan Roth
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17929
• Pdf URL: https://arxiv.org/pdf/2407.17929
• Copy Paste: [[2407.17929]] Guided Latent Slot Diffusion for Object-Centric Learning(https://arxiv.org/abs/2407.17929)
• Keywords: diffusion, segmentation
• Abstract: Slot attention aims to decompose an input image into a set of meaningful object files (slots). These latent object representations enable various downstream tasks. Yet, these slots often bind to object parts, not objects themselves, especially for real-world datasets. To address this, we introduce Guided Latent Slot Diffusion - GLASS, an object-centric model that uses generated captions as a guiding signal to better align slots with objects. Our key insight is to learn the slot-attention module in the space of generated images. This allows us to repurpose the pre-trained diffusion decoder model, which reconstructs the images from the slots, as a semantic mask generator based on the generated captions. GLASS learns an object-level representation suitable for multiple tasks simultaneously, e.g., segmentation, image generation, and property prediction, outperforming previous methods. For object discovery, GLASS achieves approx. a +35% and +10% relative improvement for mIoU over the previous state-of-the-art (SOTA) method on the VOC and COCO datasets, respectively, and establishes a new SOTA FID score for conditional image generation amongst slot-attention-based methods. For the segmentation task, GLASS surpasses SOTA weakly-supervised and language-based segmentation models, which were specifically designed for the task.

Title: Segmentation by registration-enabled SAM prompt engineering using five reference images

• Authors: Yaxi Chen, Aleksandra Ivanova, Shaheer U. Saeed, Rikin Hargunani, Jie Huang, Chaozong Liu, Yipeng Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17933
• Pdf URL: https://arxiv.org/pdf/2407.17933
• Copy Paste: [[2407.17933]] Segmentation by registration-enabled SAM prompt engineering using five reference images(https://arxiv.org/abs/2407.17933)
• Keywords: fair, segmentation
• Abstract: The recently proposed Segment Anything Model (SAM) is a general tool for image segmentation, but it requires additional adaptation and careful fine-tuning for medical image segmentation, especially for small, irregularly-shaped, and boundary-ambiguous anatomical structures such as the knee cartilage that is of interest in this work. Repaired cartilage, after certain surgical procedures, exhibits imaging patterns unseen to pre-training, posing further challenges for using models like SAM with or without general-purpose fine-tuning. To address this, we propose a novel registration-based prompt engineering framework for medical image segmentation using SAM. This approach utilises established image registration algorithms to align the new image (to-be-segmented) and a small number of reference images, without requiring segmentation labels. The spatial transformations generated by registration align either the new image or pre-defined point-based prompts, before using them as input to SAM. This strategy, requiring as few as five reference images with defined point prompts, effectively prompts SAM for inference on new images, without needing any segmentation labels. Evaluation of MR images from patients who received cartilage stem cell therapy yielded Dice scores of 0.89, 0.87, 0.53, and 0.52 for segmenting femur, tibia, femoral- and tibial cartilages, respectively. This outperforms atlas-based label fusion and is comparable to supervised nnUNet, an upper-bound fair baseline in this application, both of which require full segmentation labels for reference samples. The codes are available at: this https URL

Title: BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation

• Authors: Xiang Zhang, Bingxin Ke, Hayko Riemenschneider, Nando Metzger, Anton Obukhov, Markus Gross, Konrad Schindler, Christopher Schroers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17952
• Pdf URL: https://arxiv.org/pdf/2407.17952
• Copy Paste: [[2407.17952]] BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation(https://arxiv.org/abs/2407.17952)
• Keywords: robust, extraction, diffusion
• Abstract: By training over large-scale datasets, zero-shot monocular depth estimation (MDE) methods show robust performance in the wild but often suffer from insufficiently precise details. Although recent diffusion-based MDE approaches exhibit appealing detail extraction ability, they still struggle in geometrically challenging scenes due to the difficulty of gaining robust geometric priors from diverse datasets. To leverage the complementary merits of both worlds, we propose BetterDepth to efficiently achieve geometrically correct affine-invariant MDE performance while capturing fine-grained details. Specifically, BetterDepth is a conditional diffusion-based refiner that takes the prediction from pre-trained MDE models as depth conditioning, in which the global depth context is well-captured, and iteratively refines details based on the input image. For the training of such a refiner, we propose global pre-alignment and local patch masking methods to ensure the faithfulness of BetterDepth to depth conditioning while learning to capture fine-grained scene details. By efficient training on small-scale synthetic datasets, BetterDepth achieves state-of-the-art zero-shot MDE performance on diverse public datasets and in-the-wild scenes. Moreover, BetterDepth can improve the performance of other MDE models in a plug-and-play manner without additional re-training.

Title: Relating the Seemingly Unrelated: Principled Understanding of Generalization for Generative Models in Arithmetic Reasoning Tasks

• Authors: Xingcheng Xu, Zibo Zhao, Haipeng Zhang, Yanqing Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17963
• Pdf URL: https://arxiv.org/pdf/2407.17963
• Copy Paste: [[2407.17963]] Relating the Seemingly Unrelated: Principled Understanding of Generalization for Generative Models in Arithmetic Reasoning Tasks(https://arxiv.org/abs/2407.17963)
• Keywords: generative, large language model
• Abstract: Large language models (LLMs) have demonstrated impressive versatility across numerous tasks, yet their generalization capabilities remain poorly understood. To investigate these behaviors, arithmetic tasks serve as important venues. In previous studies, seemingly unrelated mysteries still exist -- (1) models with appropriate positional embeddings can correctly perform longer unseen arithmetic operations such as addition, but their effectiveness varies in more complex tasks like multiplication; (2) models perform well for longer unseen cases in modular addition under specific moduli (e.g., modulo 100) but struggle under very close moduli (e.g., modulo 101), regardless of the positional encoding used. We believe previous studies have been treating the symptoms rather than addressing the root cause -- they have paid excessive attention to improving model components, while overlooking the differences in task properties that may be the real drivers. This is confirmed by our unified theoretical framework for different arithmetic scenarios. For example, unlike multiplication, the digital addition task has the property of translation invariance which naturally aligns with the relative positional encoding, and this combination leads to successful generalization of addition to unseen longer domains. The discrepancy in operations modulo 100 and 101 arises from the base. Modulo 100, unlike 101, is compatible with the decimal system (base 10), such that unseen information in digits beyond the units digit and the tens digit is actually not needed for the task. Extensive experiments with GPT-like models validate our theoretical predictions. These findings deepen our understanding of the generalization mechanisms, and facilitate more data-efficient model training and objective-oriented AI alignment.

Title: Lightweight Industrial Cohorted Federated Learning for Heterogeneous Assets

• Authors: Madapu Amarlingam, Abhishek Wani, Adarsh NL
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2407.17999
• Pdf URL: https://arxiv.org/pdf/2407.17999
• Copy Paste: [[2407.17999]] Lightweight Industrial Cohorted Federated Learning for Heterogeneous Assets(https://arxiv.org/abs/2407.17999)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) is the most widely adopted collaborative learning approach for training decentralized Machine Learning (ML) models by exchanging learning between clients without sharing the data and compromising privacy. However, since great data similarity or homogeneity is taken for granted in all FL tasks, FL is still not specifically designed for the industrial setting. Rarely this is the case in industrial data because there are differences in machine type, firmware version, operational conditions, environmental factors, and hence, data distribution. Albeit its popularity, it has been observed that FL performance degrades if the clients have heterogeneous data distributions. Therefore, we propose a Lightweight Industrial Cohorted FL (LICFL) algorithm that uses model parameters for cohorting without any additional on-edge (clientlevel) computations and communications than standard FL and mitigates the shortcomings from data heterogeneity in industrial applications. Our approach enhances client-level model performance by allowing them to collaborate with similar clients and train more specialized or personalized models. Also, we propose an adaptive aggregation algorithm that extends the LICFL to Adaptive LICFL (ALICFL) for further improving the global model performance and speeding up the convergence. Through numerical experiments on real-time data, we demonstrate the efficacy of the proposed algorithms and compare the performance with existing approaches.

Title: Investigation to answer three key questions concerning plant pest identification and development of a practical identification framework

• Authors: Ryosuke Wayama, Yuki Sasaki, Satoshi Kagiwada, Nobusuke Iwasaki, Hitoshi Iyatomi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18000
• Pdf URL: https://arxiv.org/pdf/2407.18000
• Copy Paste: [[2407.18000]] Investigation to answer three key questions concerning plant pest identification and development of a practical identification framework(https://arxiv.org/abs/2407.18000)
• Keywords: robust, extraction
• Abstract: The development of practical and robust automated diagnostic systems for identifying plant pests is crucial for efficient agricultural production. In this paper, we first investigate three key research questions (RQs) that have not been addressed thus far in the field of image-based plant pest identification. Based on the knowledge gained, we then develop an accurate, robust, and fast plant pest identification framework using 334K images comprising 78 combinations of four plant portions (the leaf front, leaf back, fruit, and flower of cucumber, tomato, strawberry, and eggplant) and 20 pest species captured at 27 farms. The results reveal the following. (1) For an appropriate evaluation of the model, the test data should not include images of the field from which the training images were collected, or other considerations to increase the diversity of the test set should be taken into account. (2) Pre-extraction of ROIs, such as leaves and fruits, helps to improve identification accuracy. (3) Integration of closely related species using the same control methods and cross-crop training methods for the same pests, are effective. Our two-stage plant pest identification framework, enabling ROI detection and convolutional neural network (CNN)-based identification, achieved a highly practical performance of 91.0% and 88.5% in mean accuracy and macro F1 score, respectively, for 12,223 instances of test data of 21 classes collected from unseen fields, where 25 classes of images from 318,971 samples were used for training; the average identification time was 476 ms/image.

Title: Network Inversion of Convolutional Neural Nets

• Authors: Pirzada Suhail, Amit Sethi
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18002
• Pdf URL: https://arxiv.org/pdf/2407.18002
• Copy Paste: [[2407.18002]] Network Inversion of Convolutional Neural Nets(https://arxiv.org/abs/2407.18002)
• Keywords: interpretability, explainability
• Abstract: Neural networks have emerged as powerful tools across various applications, yet their decision-making process often remains opaque, leading to them being perceived as "black boxes." This opacity raises concerns about their interpretability and reliability, especially in safety-critical scenarios. Network inversion techniques offer a solution by allowing us to peek inside these black boxes, revealing the features and patterns learned by the networks behind their decision-making processes and thereby provide valuable insights into how neural networks arrive at their conclusions, making them more interpretable and trustworthy. This paper presents a simple yet effective approach to network inversion using a carefully conditioned generator that learns the data distribution in the input space of the trained neural network, enabling the reconstruction of inputs that would most likely lead to the desired outputs. To capture the diversity in the input space for a given output, instead of simply revealing the conditioning labels to the generator, we hideously encode the conditioning label information into vectors, further exemplified by heavy dropout in the generation process and minimisation of cosine similarity between the features corresponding to the generated images. The paper concludes with immediate applications of Network Inversion including in interpretability, explainability and generation of adversarial samples.

Title: Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

• Authors: Shi Luohe, Zhang Hongyi, Yao Yao, Li Zuchao, Zhao Hai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18003
• Pdf URL: https://arxiv.org/pdf/2407.18003
• Copy Paste: [[2407.18003]] Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption(https://arxiv.org/abs/2407.18003)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs), epitomized by ChatGPT' s release in late 2022, have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture' s struggle with handling long texts. KV-Cache has emerged as a pivotal solution to this issue, converting the time complexity of token generation from quadratic to linear, albeit with increased GPU memory overhead proportional to conversation length. With the development of the LLM community and academia, various KV-Cache compression methods have been proposed. In this review, we dissect the various properties of KV-Cache and elaborate on various methods currently used to optimize the KV-Cache space usage of LLMs. These methods span the pre-training phase, deployment phase, and inference phase, and we summarize the commonalities and differences among these methods. Additionally, we list some metrics for evaluating the long-text capabilities of large language models, from both efficiency and capability perspectives. Our review thus sheds light on the evolving landscape of LLM optimization, offering insights into future advancements in this dynamic field.

Title: Self-Supervision Improves Diffusion Models for Tabular Data Imputation

• Authors: Yixin Liu, Thalaiyasingam Ajanthan, Hisham Husain, Vu Nguyen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18013
• Pdf URL: https://arxiv.org/pdf/2407.18013
• Copy Paste: [[2407.18013]] Self-Supervision Improves Diffusion Models for Tabular Data Imputation(https://arxiv.org/abs/2407.18013)
• Keywords: robust, diffusion
• Abstract: The ubiquity of missing data has sparked considerable attention and focus on tabular data imputation methods. Diffusion models, recognized as the cutting-edge technique for data generation, demonstrate significant potential in tabular data imputation tasks. However, in pursuit of diversity, vanilla diffusion models often exhibit sensitivity to initialized noises, which hinders the models from generating stable and accurate imputation results. Additionally, the sparsity inherent in tabular data poses challenges for diffusion models in accurately modeling the data manifold, impacting the robustness of these models for data imputation. To tackle these challenges, this paper introduces an advanced diffusion model named Self-supervised imputation Diffusion Model (SimpDM for brevity), specifically tailored for tabular data imputation tasks. To mitigate sensitivity to noise, we introduce a self-supervised alignment mechanism that aims to regularize the model, ensuring consistent and stable imputation predictions. Furthermore, we introduce a carefully devised state-dependent data augmentation strategy within SimpDM, enhancing the robustness of the diffusion model when dealing with limited data. Extensive experiments demonstrate that SimpDM matches or outperforms state-of-the-art imputation methods across various scenarios.

Title: ECG Arrhythmia Detection Using Disease-specific Attention-based Deep Learning Model

• Authors: Linpeng Jin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18033
• Pdf URL: https://arxiv.org/pdf/2407.18033
• Copy Paste: [[2407.18033]] ECG Arrhythmia Detection Using Disease-specific Attention-based Deep Learning Model(https://arxiv.org/abs/2407.18033)
• Keywords: interpretability
• Abstract: The electrocardiogram (ECG) is one of the most commonly-used tools to diagnose cardiovascular disease in clinical practice. Although deep learning models have achieved very impressive success in the field of automatic ECG analysis, they often lack model interpretability that is significantly important in the healthcare applications. To this end, many schemes such as general-purpose attention mechanism, Grad-CAM technique and ECG knowledge graph were proposed to be integrated with deep learning models. However, they either result in decreased classification performance or do not consist with the one in cardiologists' mind when interpreting ECG. In this study, we propose a novel disease-specific attention-based deep learning model (DANet) for arrhythmia detection from short ECG recordings. The novel idea is to introduce a soft-coding or hard-coding waveform enhanced module into existing deep neural networks, which amends original ECG signals with the guidance of the rule for diagnosis of a given disease type before being fed into the classification module. For the soft-coding DANet, we also develop a learning framework combining self-supervised pre-training with two-stage supervised training. To verify the effectiveness of our proposed DANet, we applied it to the problem of atrial premature contraction detection and the experimental results shows that it demonstrates superior performance compared to the benchmark model. Moreover, it also provides the waveform regions that deserve special attention in the model's decision-making process, allowing it to be a medical diagnostic assistant for physicians.

Title: AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wild

• Authors: Junho Park, Kyeongbo Kong, Suk-Ju Kang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18034
• Pdf URL: https://arxiv.org/pdf/2407.18034
• Copy Paste: [[2407.18034]] AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wild(https://arxiv.org/abs/2407.18034)
• Keywords: diffusion
• Abstract: Recently, there has been a significant amount of research conducted on 3D hand reconstruction to use various forms of human-computer interaction. However, 3D hand reconstruction in the wild is challenging due to extreme lack of in-the-wild 3D hand datasets. Especially, when hands are in complex pose such as interacting hands, the problems like appearance similarity, self-handed occclusion and depth ambiguity make it more difficult. To overcome these issues, we propose AttentionHand, a novel method for text-driven controllable hand image generation. Since AttentionHand can generate various and numerous in-the-wild hand images well-aligned with 3D hand label, we can acquire a new 3D hand dataset, and can relieve the domain gap between indoor and outdoor scenes. Our method needs easy-to-use four modalities (i.e, an RGB image, a hand mesh image from 3D label, a bounding box, and a text prompt). These modalities are embedded into the latent space by the encoding phase. Then, through the text attention stage, hand-related tokens from the given text prompt are attended to highlight hand-related regions of the latent embedding. After the highlighted embedding is fed to the visual attention stage, hand-related regions in the embedding are attended by conditioning global and local hand mesh images with the diffusion-based pipeline. In the decoding phase, the final feature is decoded to new hand images, which are well-aligned with the given hand mesh image and text prompt. As a result, AttentionHand achieved state-of-the-art among text-to-hand image generation models, and the performance of 3D hand mesh reconstruction was improved by additionally training with hand images generated by AttentionHand.

Title: RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models

• Authors: Haoyu Chen, Wenbo Li, Jinjin Gu, Jingjing Ren, Sixiang Chen, Tian Ye, Renjing Pei, Kaiwen Zhou, Fenglong Song, Lei Zhu
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18035
• Pdf URL: https://arxiv.org/pdf/2407.18035
• Copy Paste: [[2407.18035]] RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models(https://arxiv.org/abs/2407.18035)
• Keywords: large language model
• Abstract: Natural images captured by mobile devices often suffer from multiple types of degradation, such as noise, blur, and low light. Traditional image restoration methods require manual selection of specific tasks, algorithms, and execution sequences, which is time-consuming and may yield suboptimal results. All-in-one models, though capable of handling multiple tasks, typically support only a limited range and often produce overly smooth, low-fidelity outcomes due to their broad data distribution fitting. To address these challenges, we first define a new pipeline for restoring images with multiple degradations, and then introduce RestoreAgent, an intelligent image restoration system leveraging multimodal large language models. RestoreAgent autonomously assesses the type and extent of degradation in input images and performs restoration through (1) determining the appropriate restoration tasks, (2) optimizing the task sequence, (3) selecting the most suitable models, and (4) executing the restoration. Experimental results demonstrate the superior performance of RestoreAgent in handling complex degradation, surpassing human experts. Furthermore, the system modular design facilitates the fast integration of new tasks and models, enhancing its flexibility and scalability for various applications.

Title: TiCoSS: Tightening the Coupling between Semantic Segmentation and Stereo Matching within A Joint Learning Framework

• Authors: Guanfeng Tang, Zhiyuan Wu, Rui Fan
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.18038
• Pdf URL: https://arxiv.org/pdf/2407.18038
• Copy Paste: [[2407.18038]] TiCoSS: Tightening the Coupling between Semantic Segmentation and Stereo Matching within A Joint Learning Framework(https://arxiv.org/abs/2407.18038)
• Keywords: segmentation
• Abstract: Semantic segmentation and stereo matching, respectively analogous to the ventral and dorsal streams in our human brain, are two key components of autonomous driving perception systems. Addressing these two tasks with separate networks is no longer the mainstream direction in developing computer vision algorithms, particularly with the recent advances in large vision models and embodied artificial intelligence. The trend is shifting towards combining them within a joint learning framework, especially emphasizing feature sharing between the two tasks. The major contributions of this study lie in comprehensively tightening the coupling between semantic segmentation and stereo matching. Specifically, this study introduces three novelties: (1) a tightly coupled, gated feature fusion strategy, (2) a hierarchical deep supervision strategy, and (3) a coupling tightening loss function. The combined use of these technical contributions results in TiCoSS, a state-of-the-art joint learning framework that simultaneously tackles semantic segmentation and stereo matching. Through extensive experiments on the KITTI and vKITTI2 datasets, along with qualitative and quantitative analyses, we validate the effectiveness of our developed strategies and loss function, and demonstrate its superior performance compared to prior arts, with a notable increase in mIoU by over 9%. Our source code will be publicly available at mias.group/TiCoSS upon publication.

Title: Peak-Controlled Logits Poisoning Attack in Federated Distillation

• Authors: Yuhan Tang, Aoxu Zhang, Zhiyuan Wu, Bo Gao, Tian Wen, Yuwei Wang, Sheng Sun
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18039
• Pdf URL: https://arxiv.org/pdf/2407.18039
• Copy Paste: [[2407.18039]] Peak-Controlled Logits Poisoning Attack in Federated Distillation(https://arxiv.org/abs/2407.18039)
• Keywords: attack, steal, federate
• Abstract: Federated Distillation (FD) offers an innovative approach to distributed machine learning, leveraging knowledge distillation for efficient and flexible cross-device knowledge transfer without necessitating the upload of extensive model parameters to a central server. While FD has gained popularity, its vulnerability to poisoning attacks remains underexplored. To address this gap, we previously introduced FDLA (Federated Distillation Logits Attack), a method that manipulates logits communication to mislead and degrade the performance of client models. However, the impact of FDLA on participants with different identities and the effects of malicious modifications at various stages of knowledge transfer remain unexplored. To this end, we present PCFDLA (Peak-Controlled Federated Distillation Logits Attack), an advanced and more stealthy logits poisoning attack method for FD. PCFDLA enhances the effectiveness of FDLA by carefully controlling the peak values of logits to create highly misleading yet inconspicuous modifications. Furthermore, we introduce a novel metric for better evaluating attack efficacy, demonstrating that PCFDLA maintains stealth while being significantly more disruptive to victim models compared to its predecessors. Experimental results across various datasets confirm the superior impact of PCFDLA on model accuracy, solidifying its potential threat in federated distillation systems.

Title: The Geometry of Queries: Query-Based Innovations in Retrieval-Augmented Generation

• Authors: Eric Yang, Jonathan Amar, Jong Ha Lee, Bhawesh Kumar, Yugang Jia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18044
• Pdf URL: https://arxiv.org/pdf/2407.18044
• Copy Paste: [[2407.18044]] The Geometry of Queries: Query-Based Innovations in Retrieval-Augmented Generation(https://arxiv.org/abs/2407.18044)
• Keywords: robust, large language model
• Abstract: Digital health chatbots powered by Large Language Models (LLMs) have the potential to significantly improve personal health management for chronic conditions by providing accessible and on-demand health coaching and question-answering. However, these chatbots risk providing unverified and inaccurate information because LLMs generate responses based on patterns learned from diverse internet data. Retrieval Augmented Generation (RAG) can help mitigate hallucinations and inaccuracies in LLM responses by grounding it on reliable content. However, efficiently and accurately retrieving most relevant set of content for real-time user questions remains a challenge. In this work, we introduce Query-Based Retrieval Augmented Generation (QB-RAG), a novel approach that pre-computes a database of potential queries from a content base using LLMs. For an incoming patient question, QB-RAG efficiently matches it against this pre-generated query database using vector search, improving alignment between user questions and the content. We establish a theoretical foundation for QB-RAG and provide a comparative analysis of existing retrieval enhancement techniques for RAG systems. Finally, our empirical evaluation demonstrates that QB-RAG significantly improves the accuracy of healthcare question answering, paving the way for robust and trustworthy LLM applications in digital health.

Title: Cross-Vendor Reproducibility of Radiomics-based Machine Learning Models for Computer-aided Diagnosis

• Authors: Jatin Chaudhary, Ivan Jambor, Hannu Aronen, Otto Ettala, Jani Saunavaara, Peter Boström, Jukka Heikkonen, Rajeev Kanth, Harri Merisaari
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18060
• Pdf URL: https://arxiv.org/pdf/2407.18060
• Copy Paste: [[2407.18060]] Cross-Vendor Reproducibility of Radiomics-based Machine Learning Models for Computer-aided Diagnosis(https://arxiv.org/abs/2407.18060)
• Keywords: robust
• Abstract: Background: The reproducibility of machine-learning models in prostate cancer detection across different MRI vendors remains a significant challenge. Methods: This study investigates Support Vector Machines (SVM) and Random Forest (RF) models trained on radiomic features extracted from T2-weighted MRI images using Pyradiomics and MRCradiomics libraries. Feature selection was performed using the maximum relevance minimum redundancy (MRMR) technique. We aimed to enhance clinical decision support through multimodal learning and feature fusion. Results: Our SVM model, utilizing combined features from Pyradiomics and MRCradiomics, achieved an AUC of 0.74 on the Multi-Improd dataset (Siemens scanner) but decreased to 0.60 on the Philips test set. The RF model showed similar trends, with notable robustness for models using Pyradiomics features alone (AUC of 0.78 on Philips). Conclusions: These findings demonstrate the potential of multimodal feature integration to improve the robustness and generalizability of machine-learning models for clinical decision support in prostate cancer detection. This study marks a significant step towards developing reliable AI-driven diagnostic tools that maintain efficacy across various imaging platforms.

Title: Difficulty Estimation and Simplification of French Text Using LLMs

• Authors: Henri Jamet, Yash Raj Shrestha, Michalis Vlachos
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18061
• Pdf URL: https://arxiv.org/pdf/2407.18061
• Copy Paste: [[2407.18061]] Difficulty Estimation and Simplification of French Text Using LLMs(https://arxiv.org/abs/2407.18061)
• Keywords: generative, large language model
• Abstract: We leverage generative large language models for language learning applications, focusing on estimating the difficulty of foreign language texts and simplifying them to lower difficulty levels. We frame both tasks as prediction problems and develop a difficulty classification model using labeled examples, transfer learning, and large language models, demonstrating superior accuracy compared to previous approaches. For simplification, we evaluate the trade-off between simplification quality and meaning preservation, comparing zero-shot and fine-tuned performances of large language models. We show that meaningful text simplifications can be obtained with limited fine-tuning. Our experiments are conducted on French texts, but our methods are language-agnostic and directly applicable to other foreign languages.

Title: HVM-1: Large-scale video models pretrained with nearly 5000 hours of human-like video data

• Authors: A. Emin Orhan
• Subjects: cs.CV, cs.LG, cs.NE, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2407.18067
• Pdf URL: https://arxiv.org/pdf/2407.18067
• Copy Paste: [[2407.18067]] HVM-1: Large-scale video models pretrained with nearly 5000 hours of human-like video data(https://arxiv.org/abs/2407.18067)
• Keywords: robust
• Abstract: We introduce Human-like Video Models (HVM-1), large-scale video models pretrained with nearly 5000 hours of curated human-like video data (mostly egocentric, temporally extended, continuous video recordings), using the spatiotemporal masked autoencoder (ST-MAE) algorithm. We release two 633M parameter models trained at spatial resolutions of 224x224 and 448x448 pixels. We evaluate the performance of these models in downstream few-shot video and image recognition tasks and compare them against a model pretrained with 1330 hours of short action-oriented video clips from YouTube (Kinetics-700). HVM-1 models perform competitively against the Kinetics-700 pretrained model in downstream evaluations despite substantial qualitative differences between the spatiotemporal characteristics of the corresponding pretraining datasets. HVM-1 models also learn more accurate and more robust object representations compared to models pretrained with the image-based MAE algorithm on the same data, demonstrating the potential benefits of learning to predict temporal regularities in natural videos for learning better object representations.

Title: PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization

• Authors: Christopher Clarke, Yuzhao Heng, Lingjia Tang, Jason Mars
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18078
• Pdf URL: https://arxiv.org/pdf/2407.18078
• Copy Paste: [[2407.18078]] PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization(https://arxiv.org/abs/2407.18078)
• Keywords: large language model
• Abstract: The recent emergence of Large Language Models (LLMs) has heralded a new era of human-AI interaction. These sophisticated models, exemplified by Chat-GPT and its successors, have exhibited remarkable capabilities in language understanding. However, as these LLMs have undergone exponential growth, a crucial dimension that remains understudied is the personalization of these models. Large foundation models such as GPT-3 etc. focus on creating a universal model that serves a broad range of tasks and users. This approach emphasizes the model's generalization capabilities, treating users as a collective rather than as distinct individuals. While practical for many common applications, this one-size-fits-all approach often fails to address the rich tapestry of human diversity and individual needs. To explore this issue we introduce the PEFT-U Benchmark: a new dataset for building and evaluating NLP models for user personalization. \datasetname{} consists of a series of user-centered tasks containing diverse and individualized expressions where the preferences of users can potentially differ for the same input. Using PEFT-U, we explore the challenge of efficiently personalizing LLMs to accommodate user-specific preferences in the context of diverse user-centered tasks.

Title: Privacy Threats and Countermeasures in Federated Learning for Internet of Things: A Systematic Review

• Authors: Adel ElZemity, Budi Arief
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18096
• Pdf URL: https://arxiv.org/pdf/2407.18096
• Copy Paste: [[2407.18096]] Privacy Threats and Countermeasures in Federated Learning for Internet of Things: A Systematic Review(https://arxiv.org/abs/2407.18096)
• Keywords: secure, security, privacy, protect, attack, robust, steal, federate
• Abstract: Federated Learning (FL) in the Internet of Things (IoT) environments can enhance machine learning by utilising decentralised data, but at the same time, it might introduce significant privacy and security concerns due to the constrained nature of IoT devices. This represents a research challenge that we aim to address in this paper. We systematically analysed recent literature to identify privacy threats in FL within IoT environments, and evaluate the defensive measures that can be employed to mitigate these threats. Using a Systematic Literature Review (SLR) approach, we searched five publication databases (Scopus, IEEE Xplore, Wiley, ACM, and Science Direct), collating relevant papers published between 2017 and April 2024, a period which spans from the introduction of FL until now. Guided by the PRISMA protocol, we selected 49 papers to focus our systematic review on. We analysed these papers, paying special attention to the privacy threats and defensive measures -- specifically within the context of IoT -- using inclusion and exclusion criteria tailored to highlight recent advances and critical insights. We identified various privacy threats, including inference attacks, poisoning attacks, and eavesdropping, along with defensive measures such as Differential Privacy and Secure Multi-Party Computation. These defences were evaluated for their effectiveness in protecting privacy without compromising the functional integrity of FL in IoT settings. Our review underscores the necessity for robust and efficient privacy-preserving strategies tailored for IoT environments. Notably, there is a need for strategies against replay, evasion, and model stealing attacks. Exploring lightweight defensive measures and emerging technologies such as blockchain may help improve the privacy of FL in IoT, leading to the creation of FL models that can operate under variable network conditions.

Title: SSTD: Stripe-Like Space Target Detection using Single-Point Supervision

• Authors: Zijian Zhu, Ali Zia, Xuesong Li, Bingbing Dan, Yuebo Ma, Enhai Liu, Rujin Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18097
• Pdf URL: https://arxiv.org/pdf/2407.18097
• Copy Paste: [[2407.18097]] SSTD: Stripe-Like Space Target Detection using Single-Point Supervision(https://arxiv.org/abs/2407.18097)
• Keywords: segmentation
• Abstract: Stripe-like space target detection (SSTD) plays a key role in enhancing space situational awareness and assessing spacecraft behaviour. This domain faces three challenges: the lack of publicly available datasets, interference from stray light and stars, and the variability of stripe-like targets, which complicates pixel-level annotation. In response, we introduces `AstroStripeSet', a pioneering dataset designed for SSTD, aiming to bridge the gap in academic resources and advance research in SSTD. Furthermore, we propose a novel pseudo-label evolution teacher-student framework with single-point supervision. This framework starts with generating initial pseudo-labels using the zero-shot capabilities of the Segment Anything Model (SAM) in a single-point setting, and refines these labels iteratively. In our framework, the fine-tuned StripeSAM serves as the teacher and the newly developed StripeNet as the student, consistently improving segmentation performance by improving the quality of pseudo-labels. We also introduce `GeoDice', a new loss function customized for the linear characteristics of stripe-like targets. Extensive experiments show that the performance of our approach matches fully supervised methods on all evaluation metrics, establishing a new state-of-the-art (SOTA) benchmark. Our dataset and code will be made publicly available.

Title: DINOv2 Rocks Geological Image Analysis: Classification, Segmentation, and Interpretability

• Authors: Florent Brondolo, Samuel Beaussant
• Subjects: cs.CV, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2407.18100
• Pdf URL: https://arxiv.org/pdf/2407.18100
• Copy Paste: [[2407.18100]] DINOv2 Rocks Geological Image Analysis: Classification, Segmentation, and Interpretability(https://arxiv.org/abs/2407.18100)
• Keywords: interpretability, segmentation
• Abstract: This study investigates the interpretability, classification, and segmentation of CT-scan images of rock samples, with a particular focus on the application of DINOv2 within Geosciences. We compared various segmentation techniques to evaluate their efficacy, efficiency, and adaptability in geological image analysis. The methods assessed include the Otsu thresholding method, clustering techniques (K-means and fuzzy C-means), a supervised machine learning approach (Random Forest), and deep learning methods (UNet and DINOv2). We tested these methods using ten binary sandstone datasets and three multi-class calcite datasets. To begin, we provide a thorough interpretability analysis of DINOv2's features in the geoscientific context, discussing its suitability and inherent ability to process CT-scanned rock data. In terms of classification, the out-of-the-box DINOv2 demonstrates an impressive capability to perfectly classify rock images, even when the CT scans are out of its original training set. Regarding segmentation, thresholding and unsupervised methods, while fast, perform poorly despite image preprocessing, whereas supervised methods show better results. We underscore the computational demands of deep learning but highlight its minimal intervention, superior generalization, and performance without additional image preprocessing. Additionally, we observe a lack of correlation between a network's depth or the number of parameters and its performance. Our results show that a LoRA fine-tuned DINOv2 excels in out-of-distribution segmentation and significantly outperforms other methods in multi-class segmentation. By systematically comparing these methods, we identify the most efficient strategy for meticulous and laborious segmentation tasks. DINOv2 proves advantageous, achieving segmentations that could be described as "better than ground-truth" against relatively small training sets.

Title: Unsupervised Training of Neural Cellular Automata on Edge Devices

• Authors: John Kalkhof, Amin Ranem, Anirban Mukhopadhyay
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18114
• Pdf URL: https://arxiv.org/pdf/2407.18114
• Copy Paste: [[2407.18114]] Unsupervised Training of Neural Cellular Automata on Edge Devices(https://arxiv.org/abs/2407.18114)
• Keywords: robust, segmentation
• Abstract: The disparity in access to machine learning tools for medical imaging across different regions significantly limits the potential for universal healthcare innovation, particularly in remote areas. Our research addresses this issue by implementing Neural Cellular Automata (NCA) training directly on smartphones for accessible X-ray lung segmentation. We confirm the practicality and feasibility of deploying and training these advanced models on five Android devices, improving medical diagnostics accessibility and bridging the tech divide to extend machine learning benefits in medical imaging to low- and middle-income countries (LMICs). We further enhance this approach with an unsupervised adaptation method using the novel Variance-Weighted Segmentation Loss (VWSL), which efficiently learns from unlabeled data by minimizing the variance from multiple NCA predictions. This strategy notably improves model adaptability and performance across diverse medical imaging contexts without the need for extensive computational resources or labeled datasets, effectively lowering the participation threshold. Our methodology, tested on three multisite X-ray datasets -- Padchest, ChestX-ray8, and MIMIC-III -- demonstrates improvements in segmentation Dice accuracy by 0.7 to 2.8%, compared to the classic Med-NCA. Additionally, in extreme cases where no digital copy is available and images must be captured by a phone from an X-ray lightbox or monitor, VWSL enhances Dice accuracy by 5-20%, demonstrating the method's robustness even with suboptimal image sources.

Title: Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification

• Authors: Vivi Nastase, Paola Merlo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18119
• Pdf URL: https://arxiv.org/pdf/2407.18119
• Copy Paste: [[2407.18119]] Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification(https://arxiv.org/abs/2407.18119)
• Keywords: transformer
• Abstract: Analyses of transformer-based models have shown that they encode a variety of linguistic information from their textual input. While these analyses have shed a light on the relation between linguistic information on one side, and internal architecture and parameters on the other, a question remains unanswered: how is this linguistic information reflected in sentence embeddings? Using datasets consisting of sentences with known structure, we test to what degree information about chunks (in particular noun, verb or prepositional phrases), such as grammatical number, or semantic role, can be localized in sentence embeddings. Our results show that such information is not distributed over the entire sentence embedding, but rather it is encoded in specific regions. Understanding how the information from an input text is compressed into sentence embeddings helps understand current transformer models and help build future explainable neural models.

Title: Self-supervised pre-training with diffusion model for few-shot landmark detection in x-ray images

• Authors: Roberto Di Via, Francesca Odone, Vito Paolo Pastore
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18125
• Pdf URL: https://arxiv.org/pdf/2407.18125
• Copy Paste: [[2407.18125]] Self-supervised pre-training with diffusion model for few-shot landmark detection in x-ray images(https://arxiv.org/abs/2407.18125)
• Keywords: diffusion, segmentation
• Abstract: In the last few years, deep neural networks have been extensively applied in the medical domain for different tasks, ranging from image classification and segmentation to landmark detection. However, the application of these technologies in the medical domain is often hindered by data scarcity, both in terms of available annotations and images. This study introduces a new self-supervised pre-training protocol based on diffusion models for landmark detection in x-ray images. Our results show that the proposed self-supervised framework can provide accurate landmark detection with a minimal number of available annotated training images (up to 50), outperforming ImageNet supervised pre-training and state-of-the-art self-supervised pre-trainings for three popular x-ray benchmark datasets. To our knowledge, this is the first exploration of diffusion models for self-supervised learning in landmark detection, which may offer a valuable pre-training approach in few-shot regimes, for mitigating data scarcity.

Title: Estimating Earthquake Magnitude in Sentinel-1 Imagery via Ranking

• Authors: Daniele Rege Cambrin, Isaac Corley, Paolo Garza, Peyman Najafirad
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.18128
• Pdf URL: https://arxiv.org/pdf/2407.18128
• Copy Paste: [[2407.18128]] Estimating Earthquake Magnitude in Sentinel-1 Imagery via Ranking(https://arxiv.org/abs/2407.18128)
• Keywords: transformer
• Abstract: Earthquakes are commonly estimated using physical seismic stations, however, due to the installation requirements and costs of these stations, global coverage quickly becomes impractical. An efficient and lower-cost alternative is to develop machine learning models to globally monitor earth observation data to pinpoint regions impacted by these natural disasters. However, due to the small amount of historically recorded earthquakes, this becomes a low-data regime problem requiring algorithmic improvements to achieve peak performance when learning to regress earthquake magnitude. In this paper, we propose to pose the estimation of earthquake magnitudes as a metric-learning problem, training models to not only estimate earthquake magnitude from Sentinel-1 satellite imagery but to additionally rank pairwise samples. Our experiments show at max a 30%+ improvement in MAE over prior regression-only based methods, particularly transformer-based architectures.

Title: Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic

• Authors: Fakhraddin Alwajih, Gagan Bhatia, Muhammad Abdul-Mageed
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18129
• Pdf URL: https://arxiv.org/pdf/2407.18129
• Copy Paste: [[2407.18129]] Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic(https://arxiv.org/abs/2407.18129)
• Keywords: robust, large language model
• Abstract: Recent advancements have significantly enhanced the capabilities of Multimodal Large Language Models (MLLMs) in generating and understanding image-to-text content. Despite these successes, progress is predominantly limited to English due to the scarcity of high quality multimodal resources in other languages. This limitation impedes the development of competitive models in languages such as Arabic. To alleviate this situation, we introduce an efficient Arabic multimodal assistant, dubbed Dallah, that utilizes an advanced language model based on LLaMA-2 to facilitate multimodal interactions. Dallah demonstrates state-of-the-art performance in Arabic MLLMs. Through fine-tuning six Arabic dialects, Dallah showcases its capability to handle complex dialectal interactions incorporating both textual and visual elements. The model excels in two benchmark tests: one evaluating its performance on Modern Standard Arabic (MSA) and another specifically designed to assess dialectal responses. Beyond its robust performance in multimodal interaction tasks, Dallah has the potential to pave the way for further development of dialect-aware Arabic MLLMs.

Title: Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perception

• Authors: Julia Hindel, Daniele Cattaneo, Abhinav Valada
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.18145
• Pdf URL: https://arxiv.org/pdf/2407.18145
• Copy Paste: [[2407.18145]] Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perception(https://arxiv.org/abs/2407.18145)
• Keywords: robust, segmentation
• Abstract: Semantic segmentation models are typically trained on a fixed set of classes, limiting their applicability in open-world scenarios. Class-incremental semantic segmentation aims to update models with emerging new classes while preventing catastrophic forgetting of previously learned ones. However, existing methods impose strict rigidity on old classes, reducing their effectiveness in learning new incremental classes. In this work, we propose Taxonomy-Oriented Poincaré-regularized Incremental-Class Segmentation (TOPICS) that learns feature embeddings in hyperbolic space following explicit taxonomy-tree structures. This supervision provides plasticity for old classes, updating ancestors based on new classes while integrating new classes at fitting positions. Additionally, we maintain implicit class relational constraints on the geometric basis of the Poincaré ball. This ensures that the latent space can continuously adapt to new constraints while maintaining a robust structure to combat catastrophic forgetting. We also establish eight realistic incremental learning protocols for autonomous driving scenarios, where novel classes can originate from known classes or the background. Extensive evaluations of TOPICS on the Cityscapes and Mapillary Vistas 2.0 benchmarks demonstrate that it achieves state-of-the-art performance. We make the code and trained models publicly available at this http URL.

Title: Enhanced Privacy Bound for Shuffle Model with Personalized Privacy

• Authors: Yixuan Liu, Yuhan Liu, Li Xiong, Yujie Gu, Hong Chen
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2407.18157
• Pdf URL: https://arxiv.org/pdf/2407.18157
• Copy Paste: [[2407.18157]] Enhanced Privacy Bound for Shuffle Model with Personalized Privacy(https://arxiv.org/abs/2407.18157)
• Keywords: privacy
• Abstract: The shuffle model of Differential Privacy (DP) is an enhanced privacy protocol which introduces an intermediate trusted server between local users and a central data curator. It significantly amplifies the central DP guarantee by anonymizing and shuffling the local randomized data. Yet, deriving a tight privacy bound is challenging due to its complicated randomization protocol. While most existing work are focused on unified local privacy settings, this work focuses on deriving the central privacy bound for a more practical setting where personalized local privacy is required by each user. To bound the privacy after shuffling, we first need to capture the probability of each user generating clones of the neighboring data points. Second, we need to quantify the indistinguishability between two distributions of the number of clones on neighboring datasets. Existing works either inaccurately capture the probability, or underestimate the indistinguishability between neighboring datasets. Motivated by this, we develop a more precise analysis, which yields a general and tighter bound for arbitrary DP mechanisms. Firstly, we derive the clone-generating probability by hypothesis testing %from a randomizer-specific perspective, which leads to a more accurate characterization of the probability. Secondly, we analyze the indistinguishability in the context of $f$-DP, where the convexity of the distributions is leveraged to achieve a tighter privacy bound. Theoretical and numerical results demonstrate that our bound remarkably outperforms the existing results in the literature.

Title: RIDA: A Robust Attack Framework on Incomplete Graphs

• Authors: Jianke Yu, Hanchen Wang, Chen Chen, Xiaoyang Wang, Wenjie Zhang, Ying Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18170
• Pdf URL: https://arxiv.org/pdf/2407.18170
• Copy Paste: [[2407.18170]] RIDA: A Robust Attack Framework on Incomplete Graphs(https://arxiv.org/abs/2407.18170)
• Keywords: attack, robust
• Abstract: Graph Neural Networks (GNNs) are vital in data science but are increasingly susceptible to adversarial attacks. To help researchers develop more robust GNN models, it's essential to focus on designing strong attack models as foundational benchmarks and guiding references. Among adversarial attacks, gray-box poisoning attacks are noteworthy due to their effectiveness and fewer constraints. These attacks exploit GNNs' need for retraining on updated data, thereby impacting their performance by perturbing these datasets. However, current research overlooks the real-world scenario of incomplete this http URL address this gap, we introduce the Robust Incomplete Deep Attack Framework (RIDA). It is the first algorithm for robust gray-box poisoning attacks on incomplete graphs. The approach innovatively aggregates distant vertex information and ensures powerful data utilization.Extensive tests against 9 SOTA baselines on 3 real-world datasets demonstrate RIDA's superiority in handling incompleteness and high attack performance on the incomplete graph.

Title: Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers

• Authors: Zhengang Li, Alec Lu, Yanyue Xie, Zhenglun Kong, Mengshu Sun, Hao Tang, Zhong Jia Xue, Peiyan Dong, Caiwen Ding, Yanzhi Wang, Xue Lin, Zhenman Fang
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18175
• Pdf URL: https://arxiv.org/pdf/2407.18175
• Copy Paste: [[2407.18175]] Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers(https://arxiv.org/abs/2407.18175)
• Keywords: transformer
• Abstract: Vision transformers (ViTs) have demonstrated their superior accuracy for computer vision tasks compared to convolutional neural networks (CNNs). However, ViT models are often computation-intensive for efficient deployment on resource-limited edge devices. This work proposes Quasar-ViT, a hardware-oriented quantization-aware architecture search framework for ViTs, to design efficient ViT models for hardware implementation while preserving the accuracy. First, Quasar-ViT trains a supernet using our row-wise flexible mixed-precision quantization scheme, mixed-precision weight entanglement, and supernet layer scaling techniques. Then, it applies an efficient hardware-oriented search algorithm, integrated with hardware latency and resource modeling, to determine a series of optimal subnets from supernet under different inference latency targets. Finally, we propose a series of model-adaptive designs on the FPGA platform to support the architecture search and mitigate the gap between the theoretical computation reduction and the practical inference speedup. Our searched models achieve 101.5, 159.6, and 251.6 frames-per-second (FPS) inference speed on the AMD/Xilinx ZCU102 FPGA with 80.4%, 78.6%, and 74.9% top-1 accuracy, respectively, for the ImageNet dataset, consistently outperforming prior works.

Title: Gene Regulatory Network Inference from Pre-trained Single-Cell Transcriptomics Transformer with Joint Graph Learning

• Authors: Sindhura Kommu, Yizhi Wang, Yue Wang, Xuan Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18181
• Pdf URL: https://arxiv.org/pdf/2407.18181
• Copy Paste: [[2407.18181]] Gene Regulatory Network Inference from Pre-trained Single-Cell Transcriptomics Transformer with Joint Graph Learning(https://arxiv.org/abs/2407.18181)
• Keywords: transformer
• Abstract: Inferring gene regulatory networks (GRNs) from single-cell RNA sequencing (scRNA-seq) data is a complex challenge that requires capturing the intricate relationships between genes and their regulatory interactions. In this study, we tackle this challenge by leveraging the single-cell BERT-based pre-trained transformer model (scBERT), trained on extensive unlabeled scRNA-seq data, to augment structured biological knowledge from existing GRNs. We introduce a novel joint graph learning approach that combines the rich contextual representations learned by pre-trained single-cell language models with the structured knowledge encoded in GRNs using graph neural networks (GNNs). By integrating these two modalities, our approach effectively reasons over boththe gene expression level constraints provided by the scRNA-seq data and the structured biological knowledge inherent in GRNs. We evaluate our method on human cell benchmark datasets from the BEELINE study with cell type-specific ground truth networks. The results demonstrate superior performance over current state-of-the-art baselines, offering a deeper understanding of cellular regulatory mechanisms.

Title: AsEP: Benchmarking Deep Learning Methods for Antibody-specific Epitope Prediction

• Authors: Chunan Liu, Lilian Denzler, Yihong Chen, Andrew Martin, Brooks Paige
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18184
• Pdf URL: https://arxiv.org/pdf/2407.18184
• Copy Paste: [[2407.18184]] AsEP: Benchmarking Deep Learning Methods for Antibody-specific Epitope Prediction(https://arxiv.org/abs/2407.18184)
• Keywords: interpretability
• Abstract: Epitope identification is vital for antibody design yet challenging due to the inherent variability in antibodies. While many deep learning methods have been developed for general protein binding site prediction tasks, whether they work for epitope prediction remains an understudied research question. The challenge is also heightened by the lack of a consistent evaluation pipeline with sufficient dataset size and epitope diversity. We introduce a filtered antibody-antigen complex structure dataset, AsEP (Antibody-specific Epitope Prediction). AsEP is the largest of its kind and provides clustered epitope groups, allowing the community to develop and test novel epitope prediction methods. AsEP comes with an easy-to-use interface in Python and pre-built graph representations of each antibody-antigen complex while also supporting customizable embedding methods. Based on this new dataset, we benchmarked various representative general protein-binding site prediction methods and find that their performances are not satisfactory as expected for epitope prediction. We thus propose a new method, WALLE, that leverages both protein language models and graph neural networks. WALLE demonstrate about 5X performance gain over existing methods. Our empirical findings evidence that epitope prediction benefits from combining sequential embeddings provided by language models and geometrical information from graph representations, providing a guideline for future method design. In addition, we reformulate the task as bipartite link prediction, allowing easy model performance attribution and interpretability. We open-source our data and code at this https URL.

Title: Exploring Scaling Trends in LLM Robustness

• Authors: Nikolhaus Howe, Michał Zajac, Ian McKenzie, Oskar Hollinsworth, Tom Tseng, Pierre-Luc Bacon, Adam Gleave
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.18213
• Pdf URL: https://arxiv.org/pdf/2407.18213
• Copy Paste: [[2407.18213]] Exploring Scaling Trends in LLM Robustness(https://arxiv.org/abs/2407.18213)
• Keywords: defense, robust, large language model
• Abstract: Language model capabilities predictably improve from scaling a model's size and training data. Motivated by this, increasingly large language models have been trained, yielding an array of impressive capabilities. Yet these models are vulnerable to adversarial prompts, such as "jailbreaks" that hijack models to perform undesired behaviors, posing a significant risk of misuse. Prior work indicates that computer vision models become more robust with model and data scaling, raising the question: does language model robustness also improve with scale? We study this question empirically, finding that larger models respond substantially better to adversarial training, but there is little to no benefit from model scale in the absence of explicit defenses.

Title: Recursive Introspection: Teaching Language Model Agents How to Self-Improve

• Authors: Yuxiao Qu, Tianjun Zhang, Naman Garg, Aviral Kumar
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18219
• Pdf URL: https://arxiv.org/pdf/2407.18219
• Copy Paste: [[2407.18219]] Recursive Introspection: Teaching Language Model Agents How to Self-Improve(https://arxiv.org/abs/2407.18219)
• Keywords: large language model
• Abstract: A central piece in enabling intelligent agentic behavior in foundation models is to make them capable of introspecting upon their behavior, reasoning, and correcting their mistakes as more computation or interaction is available. Even the strongest proprietary large language models (LLMs) do not quite exhibit the ability of continually improving their responses sequentially, even in scenarios where they are explicitly told that they are making a mistake. In this paper, we develop RISE: Recursive IntroSpEction, an approach for fine-tuning LLMs to introduce this capability, despite prior work hypothesizing that this capability may not be possible to attain. Our approach prescribes an iterative fine-tuning procedure, which attempts to teach the model how to alter its response after having executed previously unsuccessful attempts to solve a hard test-time problem, with optionally additional environment feedback. RISE poses fine-tuning for a single-turn prompt as solving a multi-turn Markov decision process (MDP), where the initial state is the prompt. Inspired by principles in online imitation learning and reinforcement learning, we propose strategies for multi-turn data collection and training so as to imbue an LLM with the capability to recursively detect and correct its previous mistakes in subsequent iterations. Our experiments show that RISE enables Llama2, Llama3, and Mistral models to improve themselves with more turns on math reasoning tasks, outperforming several single-turn strategies given an equal amount of inference-time computation. We also find that RISE scales well, often attaining larger benefits with more capable models. Our analysis shows that RISE makes meaningful improvements to responses to arrive at the correct solution for challenging prompts, without disrupting one-turn abilities as a result of expressing more complex distributions.

Title: Automated Ensemble Multimodal Machine Learning for Healthcare

• Authors: Fergus Imrie, Stefan Denner, Lucas S. Brunschwig, Klaus Maier-Hein, Mihaela van der Schaar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18227
• Pdf URL: https://arxiv.org/pdf/2407.18227
• Copy Paste: [[2407.18227]] Automated Ensemble Multimodal Machine Learning for Healthcare(https://arxiv.org/abs/2407.18227)
• Keywords: transformer
• Abstract: The application of machine learning in medicine and healthcare has led to the creation of numerous diagnostic and prognostic models. However, despite their success, current approaches generally issue predictions using data from a single modality. This stands in stark contrast with clinician decision-making which employs diverse information from multiple sources. While several multimodal machine learning approaches exist, significant challenges in developing multimodal systems remain that are hindering clinical adoption. In this paper, we introduce a multimodal framework, AutoPrognosis-M, that enables the integration of structured clinical (tabular) data and medical imaging using automated machine learning. AutoPrognosis-M incorporates 17 imaging models, including convolutional neural networks and vision transformers, and three distinct multimodal fusion strategies. In an illustrative application using a multimodal skin lesion dataset, we highlight the importance of multimodal machine learning and the power of combining multiple fusion strategies using ensemble learning. We have open-sourced our framework as a tool for the community and hope it will accelerate the uptake of multimodal machine learning in healthcare and spur further innovation.

Title: LION: Linear Group RNN for 3D Object Detection in Point Clouds

• Authors: Zhe Liu, Jinghua Hou, Xinyu Wang, Xiaoqing Ye, Jingdong Wang, Hengshuang Zhao, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18232
• Pdf URL: https://arxiv.org/pdf/2407.18232
• Copy Paste: [[2407.18232]] LION: Linear Group RNN for 3D Object Detection in Point Clouds(https://arxiv.org/abs/2407.18232)
• Keywords: transformer
• Abstract: The benefit of transformers in large-scale 3D point cloud perception tasks, such as 3D object detection, is limited by their quadratic computation cost when modeling long-range relationships. In contrast, linear RNNs have low computational complexity and are suitable for long-range modeling. Toward this goal, we propose a simple and effective window-based framework built on LInear grOup RNN (i.e., perform linear RNN for grouped features) for accurate 3D object detection, called LION. The key property is to allow sufficient feature interaction in a much larger group than transformer-based methods. However, effectively applying linear group RNN to 3D object detection in highly sparse point clouds is not trivial due to its limitation in handling spatial modeling. To tackle this problem, we simply introduce a 3D spatial feature descriptor and integrate it into the linear group RNN operators to enhance their spatial features rather than blindly increasing the number of scanning orders for voxel features. To further address the challenge in highly sparse point clouds, we propose a 3D voxel generation strategy to densify foreground features thanks to linear group RNN as a natural property of auto-regressive models. Extensive experiments verify the effectiveness of the proposed components and the generalization of our LION on different linear group RNN operators including Mamba, RWKV, and RetNet. Furthermore, it is worth mentioning that our LION-Mamba achieves state-of-the-art on Waymo, nuScenes, Argoverse V2, and ONCE dataset. Last but not least, our method supports kinds of advanced linear RNN operators (e.g., RetNet, RWKV, Mamba, xLSTM and TTT) on small but popular KITTI dataset for a quick experience with our linear RNN-based framework.

Title: BIV-Priv-Seg: Locating Private Content in Images Taken by People With Visual Impairments

• Authors: Yu-Yun Tseng, Tanusree Sharma, Lotus Zhang, Abigale Stangl, Leah Findlater, Yang Wang, Danna Gurari Yu-Yun Tseng, Tanusree Sharma, Lotus Zhang, Abigale Stangl, Leah Findlater, Yang Wang, Danna Gurari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18243
• Pdf URL: https://arxiv.org/pdf/2407.18243
• Copy Paste: [[2407.18243]] BIV-Priv-Seg: Locating Private Content in Images Taken by People With Visual Impairments(https://arxiv.org/abs/2407.18243)
• Keywords: privacy, segmentation
• Abstract: Individuals who are blind or have low vision (BLV) are at a heightened risk of sharing private information if they share photographs they have taken. To facilitate developing technologies that can help preserve privacy, we introduce BIV-Priv-Seg, the first localization dataset originating from people with visual impairments that shows private content. It contains 1,028 images with segmentation annotations for 16 private object categories. We first characterize BIV-Priv-Seg and then evaluate modern models' performance for locating private content in the dataset. We find modern models struggle most with locating private objects that are not salient, small, and lack text as well as recognizing when private content is absent from an image. We facilitate future extensions by sharing our new dataset with the evaluation server at this https URL.

Title: RefMask3D: Language-Guided Transformer for 3D Referring Segmentation

• Authors: Shuting He, Henghui Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18244
• Pdf URL: https://arxiv.org/pdf/2407.18244
• Copy Paste: [[2407.18244]] RefMask3D: Language-Guided Transformer for 3D Referring Segmentation(https://arxiv.org/abs/2407.18244)
• Keywords: transformer, segmentation
• Abstract: 3D referring segmentation is an emerging and challenging vision-language task that aims to segment the object described by a natural language expression in a point cloud scene. The key challenge behind this task is vision-language feature fusion and alignment. In this work, we propose RefMask3D to explore the comprehensive multi-modal feature interaction and understanding. First, we propose a Geometry-Enhanced Group-Word Attention to integrate language with geometrically coherent sub-clouds through cross-modal group-word attention, which effectively addresses the challenges posed by the sparse and irregular nature of point clouds. Then, we introduce a Linguistic Primitives Construction to produce semantic primitives representing distinct semantic attributes, which greatly enhance the vision-language understanding at the decoding stage. Furthermore, we introduce an Object Cluster Module that analyzes the interrelationships among linguistic primitives to consolidate their insights and pinpoint common characteristics, helping to capture holistic information and enhance the precision of target identification. The proposed RefMask3D achieves new state-of-the-art performance on 3D referring segmentation, 3D visual grounding, and also 2D referring image segmentation. Especially, RefMask3D outperforms previous state-of-the-art method by a large margin of 3.16% mIoU} on the challenging ScanRefer dataset. Code is available at this https URL.

Title: VGGHeads: A Large-Scale Synthetic Dataset for 3D Human Heads

• Authors: Orest Kupyn, Eugene Khvedchenia, Christian Rupprecht
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18245
• Pdf URL: https://arxiv.org/pdf/2407.18245
• Copy Paste: [[2407.18245]] VGGHeads: A Large-Scale Synthetic Dataset for 3D Human Heads(https://arxiv.org/abs/2407.18245)
• Keywords: privacy, diffusion
• Abstract: Human head detection, keypoint estimation, and 3D head model fitting are important tasks with many applications. However, traditional real-world datasets often suffer from bias, privacy, and ethical concerns, and they have been recorded in laboratory environments, which makes it difficult for trained models to generalize. Here, we introduce VGGHeads -- a large scale synthetic dataset generated with diffusion models for human head detection and 3D mesh estimation. Our dataset comprises over 1 million high-resolution images, each annotated with detailed 3D head meshes, facial landmarks, and bounding boxes. Using this dataset we introduce a new model architecture capable of simultaneous heads detection and head meshes reconstruction from a single image in a single step. Through extensive experimental evaluations, we demonstrate that models trained on our synthetic data achieve strong performance on real images. Furthermore, the versatility of our dataset makes it applicable across a broad spectrum of tasks, offering a general and comprehensive representation of human heads. Additionally, we provide detailed information about the synthetic data generation pipeline, enabling it to be re-used for other tasks and domains.

Title: RegionDrag: Fast Region-Based Image Editing with Diffusion Models

• Authors: Jingyi Lu, Xinghui Li, Kai Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18247
• Pdf URL: https://arxiv.org/pdf/2407.18247
• Copy Paste: [[2407.18247]] RegionDrag: Fast Region-Based Image Editing with Diffusion Models(https://arxiv.org/abs/2407.18247)
• Keywords: diffusion
• Abstract: Point-drag-based image editing methods, like DragDiffusion, have attracted significant attention. However, point-drag-based approaches suffer from computational overhead and misinterpretation of user intentions due to the sparsity of point-based editing instructions. In this paper, we propose a region-based copy-and-paste dragging method, RegionDrag, to overcome these limitations. RegionDrag allows users to express their editing instructions in the form of handle and target regions, enabling more precise control and alleviating ambiguity. In addition, region-based operations complete editing in one iteration and are much faster than point-drag-based methods. We also incorporate the attention-swapping technique for enhanced stability during editing. To validate our approach, we extend existing point-drag-based datasets with region-based dragging instructions. Experimental results demonstrate that RegionDrag outperforms existing point-drag-based approaches in terms of speed, accuracy, and alignment with user intentions. Remarkably, RegionDrag completes the edit on an image with a resolution of 512x512 in less than 2 seconds, which is more than 100x faster than DragDiffusion, while achieving better performance. Project page: this https URL.

Title: Trajectory-aligned Space-time Tokens for Few-shot Action Recognition

• Authors: Pulkit Kumar, Namitha Padmanabhan, Luke Luo, Sai Saketh Rambhatla, Abhinav Shrivastava
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18249
• Pdf URL: https://arxiv.org/pdf/2407.18249
• Copy Paste: [[2407.18249]] Trajectory-aligned Space-time Tokens for Few-shot Action Recognition(https://arxiv.org/abs/2407.18249)
• Keywords: transformer
• Abstract: We propose a simple yet effective approach for few-shot action recognition, emphasizing the disentanglement of motion and appearance representations. By harnessing recent progress in tracking, specifically point trajectories and self-supervised representation learning, we build trajectory-aligned tokens (TATs) that capture motion and appearance information. This approach significantly reduces the data requirements while retaining essential information. To process these representations, we use a Masked Space-time Transformer that effectively learns to aggregate information to facilitate few-shot action recognition. We demonstrate state-of-the-art results on few-shot action recognition across multiple datasets. Our project page is available at this https URL

Title: Sparse vs Contiguous Adversarial Pixel Perturbations in Multimodal Models: An Empirical Analysis

• Authors: Cristian-Alexandru Botocan, Raphael Meier, Ljiljana Dolamic
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18251
• Pdf URL: https://arxiv.org/pdf/2407.18251
• Copy Paste: [[2407.18251]] Sparse vs Contiguous Adversarial Pixel Perturbations in Multimodal Models: An Empirical Analysis(https://arxiv.org/abs/2407.18251)
• Keywords: attack, robust
• Abstract: Assessing the robustness of multimodal models against adversarial examples is an important aspect for the safety of its users. We craft L0-norm perturbation attacks on the preprocessed input images. We launch them in a black-box setup against four multimodal models and two unimodal DNNs, considering both targeted and untargeted misclassification. Our attacks target less than 0.04% of perturbed image area and integrate different spatial positioning of perturbed pixels: sparse positioning and pixels arranged in different contiguous shapes (row, column, diagonal, and patch). To the best of our knowledge, we are the first to assess the robustness of three state-of-the-art multimodal models (ALIGN, AltCLIP, GroupViT) against different sparse and contiguous pixel distribution perturbations. The obtained results indicate that unimodal DNNs are more robust than multimodal models. Furthermore, models using CNN-based Image Encoder are more vulnerable than models with ViT - for untargeted attacks, we obtain a 99% success rate by perturbing less than 0.02% of the image area.