2024-08-26

Title: Towards Non-invasive and Personalized Management of Breast Cancer Patients from Multiparametric MRI via A Large Mixture-of-Modality-Experts Model

• Authors: Luyang Luo, Mingxiang Wu, Mei Li, Yi Xin, Qiong Wang, Varut Vardhanabhuti, Winnie CW Chu, Zhenhui Li, Juan Zhou, Pranav Rajpurkar, Hao Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12606
• Pdf URL: https://arxiv.org/pdf/2408.12606
• Copy Paste: [[2408.12606]] Towards Non-invasive and Personalized Management of Breast Cancer Patients from Multiparametric MRI via A Large Mixture-of-Modality-Experts Model(https://arxiv.org/abs/2408.12606)
• Keywords: robust
• Abstract: Breast magnetic resonance imaging (MRI) is the imaging technique with the highest sensitivity for detecting breast cancer and is routinely used for women at high risk. Despite the comprehensive multiparametric protocol of breast MRI, existing artificial intelligence-based studies predominantly rely on single sequences and have limited validation. Here we report a large mixture-of-modality-experts model (MOME) that integrates multiparametric MRI information within a unified structure, offering a noninvasive method for personalized breast cancer management. We have curated the largest multiparametric breast MRI dataset, involving 5,205 patients from three hospitals in the north, southeast, and southwest of China, for the development and extensive evaluation of our model. MOME demonstrated accurate and robust identification of breast cancer. It achieved comparable performance for malignancy recognition to that of four senior radiologists and significantly outperformed a junior radiologist, with 0.913 AUROC, 0.948 AUPRC, 0.905 F1 score, and 0.723 MCC. Our findings suggest that MOME could reduce the need for biopsies in BI-RADS 4 patients with a ratio of 7.3%, classify triple-negative breast cancer with an AUROC of 0.709, and predict pathological complete response to neoadjuvant chemotherapy with an AUROC of 0.694. The model further supports scalable and interpretable inference, adapting to missing modalities and providing decision explanations by highlighting lesions and measuring modality contributions. MOME exemplifies a discriminative, robust, scalable, and interpretable multimodal model, paving the way for noninvasive, personalized management of breast cancer patients based on multiparametric breast imaging data.

Title: Semantic Communication based on Large Language Model for Underwater Image Transmission

• Authors: Weilong Chen, Wenxuan Xu, Haoran Chen, Xinran Zhang, Zhijin Qin, Yanru Zhang, Zhu Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12616
• Pdf URL: https://arxiv.org/pdf/2408.12616
• Copy Paste: [[2408.12616]] Semantic Communication based on Large Language Model for Underwater Image Transmission(https://arxiv.org/abs/2408.12616)
• Keywords: robust, large language model
• Abstract: Underwater communication is essential for environmental monitoring, marine biology research, and underwater exploration. Traditional underwater communication faces limitations like low bandwidth, high latency, and susceptibility to noise, while semantic communication (SC) offers a promising solution by focusing on the exchange of semantics rather than symbols or bits. However, SC encounters challenges in underwater environments, including information loss and difficulties in accurately identifying and transmitting critical information that aligns with the diverse requirements of underwater applications. To address these challenges, we propose a novel Semantic Communication (SC) framework based on Large Language Models (LLMs). Our framework leverages visual LLMs to perform semantic compression and prioritization of underwater image data according to the query from users. By identifying and encoding key semantic elements within the images, the system selectively transmits high-priority information while applying higher compression rates to less critical regions. On the receiver side, an LLM-based recovery mechanism, along with Global Vision ControlNet and Key Region ControlNet networks, aids in reconstructing the images, thereby enhancing communication efficiency and robustness. Our framework reduces the overall data size to 0.8\% of the original. Experimental results demonstrate that our method significantly outperforms existing approaches, ensuring high-quality, semantically accurate image reconstruction.

Title: Data-Free Class Incremental Gesture Recognition via Synthetic Feature Sampling

• Authors: Zhenyu Lu, Hao Tang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12629
• Pdf URL: https://arxiv.org/pdf/2408.12629
• Copy Paste: [[2408.12629]] Data-Free Class Incremental Gesture Recognition via Synthetic Feature Sampling(https://arxiv.org/abs/2408.12629)
• Keywords: data-free
• Abstract: Data-Free Class Incremental Learning (DFCIL) aims to enable models to continuously learn new classes while retraining knowledge of old classes, even when the training data for old classes is unavailable. Although explored primarily with image datasets by researchers, this study focuses on investigating DFCIL for skeleton-based gesture classification due to its significant real-world implications, particularly considering the growing prevalence of VR/AR headsets where gestures serve as the primary means of control and interaction. In this work, we made an intriguing observation: skeleton models trained with base classes(even very limited) demonstrate strong generalization capabilities to unseen classes without requiring additional training. Building on this insight, we developed Synthetic Feature Replay (SFR) that can sample synthetic features from class prototypes to replay for old classes and augment for new classes (under a few-shot setting). Our proposed method showcases significant advancements over the state-of-the-art, achieving up to 15% enhancements in mean accuracy across all steps and largely mitigating the accuracy imbalance between base classes and new classes.

Title: AI-driven Transformer Model for Fault Prediction in Non-Linear Dynamic Automotive System

• Authors: Priyanka Kumar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12638
• Pdf URL: https://arxiv.org/pdf/2408.12638
• Copy Paste: [[2408.12638]] AI-driven Transformer Model for Fault Prediction in Non-Linear Dynamic Automotive System(https://arxiv.org/abs/2408.12638)
• Keywords: robust, transformer
• Abstract: Fault detection in automotive engine systems is one of the most promising research areas. Several works have been done in the field of model-based fault diagnosis. Many researchers have discovered more advanced statistical methods and algorithms for better fault detection on any automotive dynamic engine system. The gas turbines/diesel engines produce highly complex and huge data which are highly non-linear. So, researchers should come up with an automated system that is more resilient and robust enough to handle this huge, complex data in highly non-linear dynamic automotive systems. Here, I present an AI-based fault classification and prediction model in the diesel engine that can be applied to any highly non-linear dynamic automotive system. The main contribution of this paper is the AI-based Transformer fault classification and prediction model in the diesel engine concerning the worldwide harmonic light vehicle test procedure (WLTP) driving cycle. This model used 27 input dimensions, 64 hidden dimensions with 2 layers, and 9 heads to create a classifier with 12 output heads (one for fault-free data and 11 different fault types). This model was trained on the UTSA Arc High-Performance Compute (HPC) cluster with 5 NVIDIA V100 GPUs, 40-core CPUs, and 384GB RAM and achieved 70.01 % accuracy on a held test set.

Title: Fairness-Aware Streaming Feature Selection with Causal Graphs

• Authors: Leizhen Zhang, Lusi Li, Di Wu, Sheng Chen, Yi He
• Subjects: cs.LG, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.12665
• Pdf URL: https://arxiv.org/pdf/2408.12665
• Copy Paste: [[2408.12665]] Fairness-Aware Streaming Feature Selection with Causal Graphs(https://arxiv.org/abs/2408.12665)
• Keywords: protect, fair
• Abstract: Its crux lies in the optimization of a tradeoff between accuracy and fairness of resultant models on the selected feature subset. The technical challenge of our setting is twofold: 1) streaming feature inputs, such that an informative feature may become obsolete or redundant for prediction if its information has been covered by other similar features that arrived prior to it, and 2) non-associational feature correlation, such that bias may be leaked from those seemingly admissible, non-protected features. To overcome this, we propose Streaming Feature Selection with Causal Fairness (SFCF) that builds two causal graphs egocentric to prediction label and protected feature, respectively, striving to model the complex correlation structure among streaming features, labels, and protected information. As such, bias can be eradicated from predictive modeling by removing those features being causally correlated with the protected feature yet independent to the labels. We theorize that the originally redundant features for prediction can later become admissible, when the learning accuracy is compromised by the large number of removed features (non-protected but can be used to reconstruct bias information). We benchmark SFCF\ on five datasets widely used in streaming feature research, and the results substantiate its performance superiority over six rival models in terms of efficiency and sparsity of feature selection and equalized odds of the resultant predictive models.

Title: Benchmarking Counterfactual Interpretability in Deep Learning Models for Time Series Classification

• Authors: Ziwen Kan, Shahbaz Rezaei, Xin liu
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.12666
• Pdf URL: https://arxiv.org/pdf/2408.12666
• Copy Paste: [[2408.12666]] Benchmarking Counterfactual Interpretability in Deep Learning Models for Time Series Classification(https://arxiv.org/abs/2408.12666)
• Keywords: interpretability
• Abstract: The popularity of deep learning methods in the time series domain boosts interest in interpretability studies, including counterfactual (CF) methods. CF methods identify minimal changes in instances to alter the model predictions. Despite extensive research, no existing work benchmarks CF methods in the time series domain. Additionally, the results reported in the literature are inconclusive due to the limited number of datasets and inadequate metrics. In this work, we redesign quantitative metrics to accurately capture desirable characteristics in CFs. We specifically redesign the metrics for sparsity and plausibility and introduce a new metric for consistency. Combined with validity, generation time, and proximity, we form a comprehensive metric set. We systematically benchmark 6 different CF methods on 20 univariate datasets and 10 multivariate datasets with 3 different classifiers. Results indicate that the performance of CF methods varies across metrics and among different models. Finally, we provide case studies and a guideline for practical usage.

Title: Leveraging Information Consistency in Frequency and Spatial Domain for Adversarial Attacks

• Authors: Zhibo Jin, Jiayu Zhang, Zhiyu Zhu, Xinyi Wang, Yiyun Huang, Huaming Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12670
• Pdf URL: https://arxiv.org/pdf/2408.12670
• Copy Paste: [[2408.12670]] Leveraging Information Consistency in Frequency and Spatial Domain for Adversarial Attacks(https://arxiv.org/abs/2408.12670)
• Keywords: attack
• Abstract: Adversarial examples are a key method to exploit deep neural networks. Using gradient information, such examples can be generated in an efficient way without altering the victim model. Recent frequency domain transformation has further enhanced the transferability of such adversarial examples, such as spectrum simulation attack. In this work, we investigate the effectiveness of frequency domain-based attacks, aligning with similar findings in the spatial domain. Furthermore, such consistency between the frequency and spatial domains provides insights into how gradient-based adversarial attacks induce perturbations across different domains, which is yet to be explored. Hence, we propose a simple, effective, and scalable gradient-based adversarial attack algorithm leveraging the information consistency in both frequency and spatial domains. We evaluate the algorithm for its effectiveness against different models. Extensive experiments demonstrate that our algorithm achieves state-of-the-art results compared to other gradient-based algorithms. Our code is available at: this https URL.

Title: MultiMed: Massively Multimodal and Multitask Medical Understanding

• Authors: Shentong Mo, Paul Pu Liang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.12682
• Pdf URL: https://arxiv.org/pdf/2408.12682
• Copy Paste: [[2408.12682]] MultiMed: Massively Multimodal and Multitask Medical Understanding(https://arxiv.org/abs/2408.12682)
• Keywords: robust
• Abstract: Biomedical data is inherently multimodal, consisting of electronic health records, medical imaging, digital pathology, genome sequencing, wearable sensors, and more. The application of artificial intelligence tools to these multifaceted sensing technologies has the potential to revolutionize the prognosis, diagnosis, and management of human health and disease. However, current approaches to biomedical AI typically only train and evaluate with one or a small set of medical modalities and tasks. This limitation hampers the development of comprehensive tools that can leverage the rich interconnected information across many heterogeneous biomedical sensors. To address this challenge, we present MultiMed, a benchmark designed to evaluate and enable large-scale learning across a wide spectrum of medical modalities and tasks. MultiMed consists of 2.56 million samples across ten medical modalities such as medical reports, pathology, genomics, and protein data, and is structured into eleven challenging tasks, including disease prognosis, protein structure prediction, and medical question answering. Using MultiMed, we conduct comprehensive experiments benchmarking state-of-the-art unimodal, multimodal, and multitask models. Our analysis highlights the advantages of training large-scale medical models across many related modalities and tasks. Moreover, MultiMed enables studies of generalization across related medical concepts, robustness to real-world noisy data and distribution shifts, and novel modality combinations to improve prediction performance. MultiMed will be publicly available and regularly updated and welcomes inputs from the community.

Title: Late Breaking Results: On the One-Key Premise of Logic Locking

• Authors: Yinghua Hu, Hari Cherupalli, Mike Borza, Deepak Sherlekar
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.12690
• Pdf URL: https://arxiv.org/pdf/2408.12690
• Copy Paste: [[2408.12690]] Late Breaking Results: On the One-Key Premise of Logic Locking(https://arxiv.org/abs/2408.12690)
• Keywords: secure, protect, attack
• Abstract: The evaluation of logic locking methods has long been predicated on an implicit assumption that only the correct key can unveil the true functionality of a protected circuit. Consequently, a locking technique is deemed secure if it resists a good array of attacks aimed at finding this correct key. This paper challenges this one-key premise by introducing a more efficient attack methodology, focused not on identifying that one correct key, but on finding multiple, potentially incorrect keys that can collectively produce correct functionality from the protected circuit. The tasks of finding these keys can be parallelized, which is well suited for multi-core computing environments. Empirical results show our attack achieves a runtime reduction of up to 99.6% compared to the conventional attack that tries to find a single correct key.

Title: Revisiting Cross-Domain Problem for LiDAR-based 3D Object Detection

• Authors: Ruixiao Zhang, Juheon Lee, Xiaohao Cai, Adam Prugel-Bennett
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12708
• Pdf URL: https://arxiv.org/pdf/2408.12708
• Copy Paste: [[2408.12708]] Revisiting Cross-Domain Problem for LiDAR-based 3D Object Detection(https://arxiv.org/abs/2408.12708)
• Keywords: transformer
• Abstract: Deep learning models such as convolutional neural networks and transformers have been widely applied to solve 3D object detection problems in the domain of autonomous driving. While existing models have achieved outstanding performance on most open benchmarks, the generalization ability of these deep networks is still in doubt. To adapt models to other domains including different cities, countries, and weather, retraining with the target domain data is currently necessary, which hinders the wide application of autonomous driving. In this paper, we deeply analyze the cross-domain performance of the state-of-the-art models. We observe that most models will overfit the training domains and it is challenging to adapt them to other domains directly. Existing domain adaptation methods for 3D object detection problems are actually shifting the models' knowledge domain instead of improving their generalization ability. We then propose additional evaluation metrics -- the side-view and front-view AP -- to better analyze the core issues of the methods' heavy drops in accuracy levels. By using the proposed metrics and further evaluating the cross-domain performance in each dimension, we conclude that the overfitting problem happens more obviously on the front-view surface and the width dimension which usually faces the sensor and has more 3D points surrounding it. Meanwhile, our experiments indicate that the density of the point cloud data also significantly influences the models' cross-domain performance.

Title: Macro-Queries: An Exploration into Guided Chart Generation from High Level Prompts

• Authors: Christopher J. Lee, Giorgio Tran, Roderick Tabalba, Jason Leigh, Ryan Longman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12726
• Pdf URL: https://arxiv.org/pdf/2408.12726
• Copy Paste: [[2408.12726]] Macro-Queries: An Exploration into Guided Chart Generation from High Level Prompts(https://arxiv.org/abs/2408.12726)
• Keywords: large language model
• Abstract: This paper explores the intersection of data visualization and Large Language Models (LLMs). Driven by the need to make a broader range of data visualization types accessible for novice users, we present a guided LLM-based pipeline designed to transform data, guided by high-level user questions (referred to as macro-queries), into a diverse set of useful visualizations. This approach leverages various prompting techniques, fine-tuning inspired by Abela's Chart Taxonomy, and integrated SQL tool usage.

Title: BankTweak: Adversarial Attack against Multi-Object Trackers by Manipulating Feature Banks

• Authors: Woojin Shin, Donghwa Kang, Daejin Choi, Brent Kang, Jinkyu Lee, Hyeongboo Baek
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12727
• Pdf URL: https://arxiv.org/pdf/2408.12727
• Copy Paste: [[2408.12727]] BankTweak: Adversarial Attack against Multi-Object Trackers by Manipulating Feature Banks(https://arxiv.org/abs/2408.12727)
• Keywords: attack, robust, transformer
• Abstract: Multi-object tracking (MOT) aims to construct moving trajectories for objects, and modern multi-object trackers mainly utilize the tracking-by-detection methodology. Initial approaches to MOT attacks primarily aimed to degrade the detection quality of the frames under attack, thereby reducing accuracy only in those specific frames, highlighting a lack of \textit{efficiency}. To improve efficiency, recent advancements manipulate object positions to cause persistent identity (ID) switches during the association phase, even after the attack ends within a few frames. However, these position-manipulating attacks have inherent limitations, as they can be easily counteracted by adjusting distance-related parameters in the association phase, revealing a lack of \textit{robustness}. In this paper, we present \textsf{BankTweak}, a novel adversarial attack designed for MOT trackers, which features efficiency and robustness. \textsf{BankTweak} focuses on the feature extractor in the association phase and reveals vulnerability in the Hungarian matching method used by feature-based MOT systems. Exploiting the vulnerability, \textsf{BankTweak} induces persistent ID switches (addressing \textit{efficiency}) even after the attack ends by strategically injecting altered features into the feature banks without modifying object positions (addressing \textit{robustness}). To demonstrate the applicability, we apply \textsf{BankTweak} to three multi-object trackers (DeepSORT, StrongSORT, and MOTDT) with one-stage, two-stage, anchor-free, and transformer detectors. Extensive experiments on the MOT17 and MOT20 datasets show that our method substantially surpasses existing attacks, exposing the vulnerability of the tracking-by-detection framework to \textsf{BankTweak}.

Title: Segment Anything Model for Grain Characterization in Hard Drive Design

• Authors: Kai Nichols, Matthew Hauwiller, Nicholas Propes, Shaowei Wu, Stephanie Hernandez, Mike Kautzky
• Subjects: cs.CV, cond-mat.mtrl-sci, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12732
• Pdf URL: https://arxiv.org/pdf/2408.12732
• Copy Paste: [[2408.12732]] Segment Anything Model for Grain Characterization in Hard Drive Design(https://arxiv.org/abs/2408.12732)
• Keywords: extraction, segmentation
• Abstract: Development of new materials in hard drive designs requires characterization of nanoscale materials through grain segmentation. The high-throughput quickly changing research environment makes zero-shot generalization an incredibly desirable feature. For this reason, we explore the application of Meta's Segment Anything Model (SAM) to this problem. We first analyze the out-of-the-box use of SAM. Then we discuss opportunities and strategies for improvement under the assumption of minimal labeled data availability. Out-of-the-box SAM shows promising accuracy at property distribution extraction. We are able to identify four potential areas for improvement and show preliminary gains in two of the four areas.

Title: A Formal, Symbolic Analysis of the Matrix Cryptographic Protocol Suite

• Authors: Jacob Ginesin, Cristina Nita-Rotaru
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.12743
• Pdf URL: https://arxiv.org/pdf/2408.12743
• Copy Paste: [[2408.12743]] A Formal, Symbolic Analysis of the Matrix Cryptographic Protocol Suite(https://arxiv.org/abs/2408.12743)
• Keywords: secure, security, attack, federate
• Abstract: Secure instant group messaging applications such as WhatsApp, Facebook Messenger, Matrix, and the Signal Application have become ubiquitous in today's internet, cumulatively serving billions of users. Unlike WhatsApp, for example, Matrix can be deployed in a federated manner, allowing users to choose which server manages their chats. To account for this difference in architecture, Matrix employs two novel cryptographic protocols: Olm, which secures pairwise communications, and Megolm, which relies on Olm and secures group communications. Olm and Megolm are similar to and share security goals with Signal and Sender Keys, which are widely deployed in practice to secure group communications. While Olm, Megolm, and Sender Keys have been manually analyzed in the computational model, no symbolic analysis nor mechanized proofs of correctness exist. Using mechanized proofs and computer-aided analysis is important for cryptographic protocols, as hand-written proofs and analysis are error-prone and often carry subtle mistakes. Using Verifpal, we construct formal models of Olm and Megolm, as well as their composition. We prove various properties of interest about Olm and Megolm, including authentication, confidentiality, forward secrecy, and post-compromise security. We also mechanize known limitations, previously discovered attacks, and trivial attacker wins from the specifications and previous literature. Finally, we model Sender Keys and the composition of Signal with Sender Keys in order to draw a comparison with Olm, Megolm, and their composition. From our analysis we conclude the composition of Olm and Megolm has comparable security to the composition of Signal and Sender Keys if Olm pre-keys are signed, and provably worse post-compromise security if Olm pre-keys are not signed.

Title: CatFree3D: Category-agnostic 3D Object Detection with Diffusion

• Authors: Wenjing Bian, Zirui Wang, Andrea Vedaldi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12747
• Pdf URL: https://arxiv.org/pdf/2408.12747
• Copy Paste: [[2408.12747]] CatFree3D: Category-agnostic 3D Object Detection with Diffusion(https://arxiv.org/abs/2408.12747)
• Keywords: diffusion
• Abstract: Image-based 3D object detection is widely employed in applications such as autonomous vehicles and robotics, yet current systems struggle with generalisation due to complex problem setup and limited training data. We introduce a novel pipeline that decouples 3D detection from 2D detection and depth prediction, using a diffusion-based approach to improve accuracy and support category-agnostic detection. Additionally, we introduce the Normalised Hungarian Distance (NHD) metric for an accurate evaluation of 3D detection results, addressing the limitations of traditional IoU and GIoU metrics. Experimental results demonstrate that our method achieves state-of-the-art accuracy and strong generalisation across various object categories and datasets.

Title: SLM Meets LLM: Balancing Latency, Interpretability and Consistency in Hallucination Detection

• Authors: Mengya Hu, Rui Xu, Deren Lei, Yaxi Li, Mingyu Wang, Emily Ching, Eslam Kamal, Alex Deng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12748
• Pdf URL: https://arxiv.org/pdf/2408.12748
• Copy Paste: [[2408.12748]] SLM Meets LLM: Balancing Latency, Interpretability and Consistency in Hallucination Detection(https://arxiv.org/abs/2408.12748)
• Keywords: interpretability, large language model
• Abstract: Large language models (LLMs) are highly capable but face latency challenges in real-time applications, such as conducting online hallucination detection. To overcome this issue, we propose a novel framework that leverages a small language model (SLM) classifier for initial detection, followed by a LLM as constrained reasoner to generate detailed explanations for detected hallucinated content. This study optimizes the real-time interpretable hallucination detection by introducing effective prompting techniques that align LLM-generated explanations with SLM decisions. Empirical experiment results demonstrate its effectiveness, thereby enhancing the overall user experience.

Title: Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models

• Authors: Jean Park, Kuk Jin Jang, Basam Alasaly, Sriharsha Mopidevi, Andrew Zolensky, Eric Eaton, Insup Lee, Kevin Johnson
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12763
• Pdf URL: https://arxiv.org/pdf/2408.12763
• Copy Paste: [[2408.12763]] Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models(https://arxiv.org/abs/2408.12763)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) can simultaneously process visual, textual, and auditory data, capturing insights that complement human analysis. However, existing video question-answering (VidQA) benchmarks and datasets often exhibit a bias toward a single modality, despite the goal of requiring advanced reasoning skills that integrate diverse modalities to answer the queries. In this work, we introduce the modality importance score (MIS) to identify such bias. It is designed to assess which modality embeds the necessary information to answer the question. Additionally, we propose an innovative method using state-of-the-art MLLMs to estimate the modality importance, which can serve as a proxy for human judgments of modality perception. With this MIS, we demonstrate the presence of unimodal bias and the scarcity of genuinely multimodal questions in existing datasets. We further validate the modality importance score with multiple ablation studies to evaluate the performance of MLLMs on permuted feature sets. Our results indicate that current models do not effectively integrate information due to modality imbalance in existing datasets. Our proposed MLLM-derived MIS can guide the curation of modality-balanced datasets that advance multimodal learning and enhance MLLMs' capabilities to understand and utilize synergistic relations across modalities.

Title: Enhancing Vehicle Environmental Awareness via Federated Learning and Automatic Labeling

• Authors: Chih-Yu Lin, Jin-Wei Liang
• Subjects: cs.CV, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.12769
• Pdf URL: https://arxiv.org/pdf/2408.12769
• Copy Paste: [[2408.12769]] Enhancing Vehicle Environmental Awareness via Federated Learning and Automatic Labeling(https://arxiv.org/abs/2408.12769)
• Keywords: privacy, federate
• Abstract: Vehicle environmental awareness is a crucial issue in improving road safety. Through a variety of sensors and vehicle-to-vehicle communication, vehicles can collect a wealth of data. However, to make these data useful, sensor data must be integrated effectively. This paper focuses on the integration of image data and vehicle-to-vehicle communication data. More specifically, our goal is to identify the locations of vehicles sending messages within images, a challenge termed the vehicle identification problem. In this paper, we employ a supervised learning model to tackle the vehicle identification problem. However, we face two practical issues: first, drivers are typically unwilling to share privacy-sensitive image data, and second, drivers usually do not engage in data labeling. To address these challenges, this paper introduces a comprehensive solution to the vehicle identification problem, which leverages federated learning and automatic labeling techniques in combination with the aforementioned supervised learning model. We have validated the feasibility of our proposed approach through experiments.

Title: Symmetric masking strategy enhances the performance of Masked Image Modeling

• Authors: Khanh-Binh Nguyen, Chae Jung Park
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12772
• Pdf URL: https://arxiv.org/pdf/2408.12772
• Copy Paste: [[2408.12772]] Symmetric masking strategy enhances the performance of Masked Image Modeling(https://arxiv.org/abs/2408.12772)
• Keywords: transformer, segmentation
• Abstract: Masked Image Modeling (MIM) is a technique in self-supervised learning that focuses on acquiring detailed visual representations from unlabeled images by estimating the missing pixels in randomly masked sections. It has proven to be a powerful tool for the preliminary training of Vision Transformers (ViTs), yielding impressive results across various tasks. Nevertheless, most MIM methods heavily depend on the random masking strategy to formulate the pretext task. This strategy necessitates numerous trials to ascertain the optimal dropping ratio, which can be resource-intensive, requiring the model to be pre-trained for anywhere between 800 to 1600 epochs. Furthermore, this approach may not be suitable for all datasets. In this work, we propose a new masking strategy that effectively helps the model capture global and local features. Based on this masking strategy, SymMIM, our proposed training pipeline for MIM is introduced. SymMIM achieves a new SOTA accuracy of 85.9\% on ImageNet using ViT-Large and surpasses previous SOTA across downstream tasks such as image classification, semantic segmentation, object detection, instance segmentation tasks, and so on.

Title: Semi-Supervised Variational Adversarial Active Learning via Learning to Rank and Agreement-Based Pseudo Labeling

• Authors: Zongyao Lyu, William J. Beksi
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12774
• Pdf URL: https://arxiv.org/pdf/2408.12774
• Copy Paste: [[2408.12774]] Semi-Supervised Variational Adversarial Active Learning via Learning to Rank and Agreement-Based Pseudo Labeling(https://arxiv.org/abs/2408.12774)
• Keywords: segmentation
• Abstract: Active learning aims to alleviate the amount of labor involved in data labeling by automating the selection of unlabeled samples via an acquisition function. For example, variational adversarial active learning (VAAL) leverages an adversarial network to discriminate unlabeled samples from labeled ones using latent space information. However, VAAL has the following shortcomings: (i) it does not exploit target task information, and (ii) unlabeled data is only used for sample selection rather than model training. To address these limitations, we introduce novel techniques that significantly improve the use of abundant unlabeled data during training and take into account the task information. Concretely, we propose an improved pseudo-labeling algorithm that leverages information from all unlabeled data in a semi-supervised manner, thus allowing a model to explore a richer data space. In addition, we develop a ranking-based loss prediction module that converts predicted relative ranking information into a differentiable ranking loss. This loss can be embedded as a rank variable into the latent space of a variational autoencoder and then trained with a discriminator in an adversarial fashion for sample selection. We demonstrate the superior performance of our approach over the state of the art on various image classification and segmentation benchmark datasets.

Title: Investigating LLM Applications in E-Commerce

• Authors: Chester Palen-Michel, Ruixiang Wang, Yipeng Zhang, David Yu, Canran Xu, Zhe Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12779
• Pdf URL: https://arxiv.org/pdf/2408.12779
• Copy Paste: [[2408.12779]] Investigating LLM Applications in E-Commerce(https://arxiv.org/abs/2408.12779)
• Keywords: large language model
• Abstract: The emergence of Large Language Models (LLMs) has revolutionized natural language processing in various applications especially in e-commerce. One crucial step before the application of such LLMs in these fields is to understand and compare the performance in different use cases in such tasks. This paper explored the efficacy of LLMs in the e-commerce domain, focusing on instruction-tuning an open source LLM model with public e-commerce datasets of varying sizes and comparing the performance with the conventional models prevalent in industrial applications. We conducted a comprehensive comparison between LLMs and traditional pre-trained language models across specific tasks intrinsic to the e-commerce domain, namely classification, generation, summarization, and named entity recognition (NER). Furthermore, we examined the effectiveness of the current niche industrial application of very large LLM, using in-context learning, in e-commerce specific tasks. Our findings indicate that few-shot inference with very large LLMs often does not outperform fine-tuning smaller pre-trained models, underscoring the importance of task-specific model optimization.Additionally, we investigated different training methodologies such as single-task training, mixed-task training, and LoRA merging both within domain/tasks and between different tasks. Through rigorous experimentation and analysis, this paper offers valuable insights into the potential effectiveness of LLMs to advance natural language processing capabilities within the e-commerce industry.

Title: Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation

• Authors: Vivek Iyer, Bhavitvya Malik, Pavel Stepachev, Pinzhen Chen, Barry Haddow, Alexandra Birch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12780
• Pdf URL: https://arxiv.org/pdf/2408.12780
• Copy Paste: [[2408.12780]] Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation(https://arxiv.org/abs/2408.12780)
• Keywords: large language model
• Abstract: Despite the recent popularity of Large Language Models (LLMs) in Machine Translation (MT), their performance in low-resource translation still lags significantly behind Neural Machine Translation (NMT) models. In this paper, we explore what it would take to adapt LLMs for low-resource settings. In particular, we re-examine the role of two factors: a) the importance and application of parallel data, and b) diversity in Supervised Fine-Tuning (SFT). Recently, parallel data has been shown to be less important for MT using LLMs than in previous MT research. Similarly, diversity during SFT has been shown to promote significant transfer in LLMs across languages and tasks. However, for low-resource LLM-MT, we show that the opposite is true for both of these considerations: a) parallel data is critical during both pretraining and SFT, and b) diversity tends to cause interference, not transfer. Our experiments, conducted with 3 LLMs across 2 low-resourced language groups - indigenous American and North-East Indian - reveal consistent patterns in both cases, underscoring the generalizability of our findings. We believe these insights will be valuable for scaling to massively multilingual LLM-MT models that can effectively serve lower-resource languages.

Title: LLM-PBE: Assessing Data Privacy in Large Language Models

• Authors: Qinbin Li, Junyuan Hong, Chulin Xie, Jeffrey Tan, Rachel Xin, Junyi Hou, Xavier Yin, Zhun Wang, Dan Hendrycks, Zhangyang Wang, Bo Li, Bingsheng He, Dawn Song
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12787
• Pdf URL: https://arxiv.org/pdf/2408.12787
• Copy Paste: [[2408.12787]] LLM-PBE: Assessing Data Privacy in Large Language Models(https://arxiv.org/abs/2408.12787)
• Keywords: privacy, defense, attack, large language model
• Abstract: Large Language Models (LLMs) have become integral to numerous domains, significantly advancing applications in data management, mining, and analysis. Their profound capabilities in processing and interpreting complex language data, however, bring to light pressing concerns regarding data privacy, especially the risk of unintentional training data leakage. Despite the critical nature of this issue, there has been no existing literature to offer a comprehensive assessment of data privacy risks in LLMs. Addressing this gap, our paper introduces LLM-PBE, a toolkit crafted specifically for the systematic evaluation of data privacy risks in LLMs. LLM-PBE is designed to analyze privacy across the entire lifecycle of LLMs, incorporating diverse attack and defense strategies, and handling various data types and metrics. Through detailed experimentation with multiple LLMs, LLM-PBE facilitates an in-depth exploration of data privacy concerns, shedding light on influential factors such as model size, data characteristics, and evolving temporal dimensions. This study not only enriches the understanding of privacy issues in LLMs but also serves as a vital resource for future research in the field. Aimed at enhancing the breadth of knowledge in this area, the findings, resources, and our full technical report are made available at this https URL, providing an open platform for academic and practical advancements in LLM privacy assessment.

Title: Context-Aware Temporal Embedding of Objects in Video Data

• Authors: Ahnaf Farhan, M. Shahriar Hossain
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12789
• Pdf URL: https://arxiv.org/pdf/2408.12789
• Copy Paste: [[2408.12789]] Context-Aware Temporal Embedding of Objects in Video Data(https://arxiv.org/abs/2408.12789)
• Keywords: large language model
• Abstract: In video analysis, understanding the temporal context is crucial for recognizing object interactions, event patterns, and contextual changes over time. The proposed model leverages adjacency and semantic similarities between objects from neighboring video frames to construct context-aware temporal object embeddings. Unlike traditional methods that rely solely on visual appearance, our temporal embedding model considers the contextual relationships between objects, creating a meaningful embedding space where temporally connected object's vectors are positioned in proximity. Empirical studies demonstrate that our context-aware temporal embeddings can be used in conjunction with conventional visual embeddings to enhance the effectiveness of downstream applications. Moreover, the embeddings can be used to narrate a video using a Large Language Model (LLM). This paper describes the intricate details of the proposed objective function to generate context-aware temporal object embeddings for video data and showcases the potential applications of the generated embeddings in video analysis and object classification tasks.

Title: Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture

• Authors: Chenqi Kong, Anwei Luo, Peijun Bao, Haoliang Li, Renjie Wan, Zengwei Zheng, Anderson Rocha, Alex C. Kot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12791
• Pdf URL: https://arxiv.org/pdf/2408.12791
• Copy Paste: [[2408.12791]] Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture(https://arxiv.org/abs/2408.12791)
• Keywords: security, extraction, transformer
• Abstract: Open-set face forgery detection poses significant security threats and presents substantial challenges for existing detection models. These detectors primarily have two limitations: they cannot generalize across unknown forgery domains and inefficiently adapt to new data. To address these issues, we introduce an approach that is both general and parameter-efficient for face forgery detection. It builds on the assumption that different forgery source domains exhibit distinct style statistics. Previous methods typically require fully fine-tuning pre-trained networks, consuming substantial time and computational resources. In turn, we design a forgery-style mixture formulation that augments the diversity of forgery source domains, enhancing the model's generalizability across unseen domains. Drawing on recent advancements in vision transformers (ViT) for face forgery detection, we develop a parameter-efficient ViT-based detection model that includes lightweight forgery feature extraction modules and enables the model to extract global and local forgery clues simultaneously. We only optimize the inserted lightweight modules during training, maintaining the original ViT structure with its pre-trained ImageNet weights. This training strategy effectively preserves the informative pre-trained knowledge while flexibly adapting the model to the task of Deepfake detection. Extensive experimental results demonstrate that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters, representing an important step toward open-set Deepfake detection in the wild.

Title: La-SoftMoE CLIP for Unified Physical-Digital Face Attack Detection

• Authors: Hang Zou, Chenxi Du, Hui Zhang, Yuan Zhang, Ajian Liu, Jun Wan, Zhen Lei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12793
• Pdf URL: https://arxiv.org/pdf/2408.12793
• Copy Paste: [[2408.12793]] La-SoftMoE CLIP for Unified Physical-Digital Face Attack Detection(https://arxiv.org/abs/2408.12793)
• Keywords: security, attack
• Abstract: Facial recognition systems are susceptible to both physical and digital attacks, posing significant security risks. Traditional approaches often treat these two attack types separately due to their distinct characteristics. Thus, when being combined attacked, almost all methods could not deal. Some studies attempt to combine the sparse data from both types of attacks into a single dataset and try to find a common feature space, which is often impractical due to the space is difficult to be found or even non-existent. To overcome these challenges, we propose a novel approach that uses the sparse model to handle sparse data, utilizing different parameter groups to process distinct regions of the sparse feature space. Specifically, we employ the Mixture of Experts (MoE) framework in our model, expert parameters are matched to tokens with varying weights during training and adaptively activated during testing. However, the traditional MoE struggles with the complex and irregular classification boundaries of this problem. Thus, we introduce a flexible self-adapting weighting mechanism, enabling the model to better fit and adapt. In this paper, we proposed La-SoftMoE CLIP, which allows for more flexible adaptation to the Unified Attack Detection (UAD) task, significantly enhancing the model's capability to handle diversity attacks. Experiment results demonstrate that our proposed method has SOTA performance.

Title: Less for More: Enhancing Preference Learning in Generative Language Models with Automated Self-Curation of Training Corpora

• Authors: JoonHo Lee, JuYoun Son, Juree Seok, Wooseok Jang, Yeong-Dae Kwon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12799
• Pdf URL: https://arxiv.org/pdf/2408.12799
• Copy Paste: [[2408.12799]] Less for More: Enhancing Preference Learning in Generative Language Models with Automated Self-Curation of Training Corpora(https://arxiv.org/abs/2408.12799)
• Keywords: generative
• Abstract: Ambiguity in language presents challenges in developing more enhanced language models, particularly in preference learning, where variability among annotators results in inconsistently annotated datasets used for model alignment. To address this issue, we introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on these datasets. Our method enhances preference learning by automatically detecting and removing ambiguous annotations within the dataset. The proposed approach is validated through extensive experiments, demonstrating a marked improvement in performance across various instruction-following tasks. Our work provides a straightforward and reliable method to overcome annotation inconsistencies, serving as an initial step towards the development of more advanced preference learning techniques.

Title: Robust Predictions with Ambiguous Time Delays: A Bootstrap Strategy

• Authors: Jiajie Wang, Zhiyuan Jerry Lin, Wen Chen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.12801
• Pdf URL: https://arxiv.org/pdf/2408.12801
• Copy Paste: [[2408.12801]] Robust Predictions with Ambiguous Time Delays: A Bootstrap Strategy(https://arxiv.org/abs/2408.12801)
• Keywords: robust
• Abstract: In contemporary data-driven environments, the generation and processing of multivariate time series data is an omnipresent challenge, often complicated by time delays between different time series. These delays, originating from a multitude of sources like varying data transmission dynamics, sensor interferences, and environmental changes, introduce significant complexities. Traditional Time Delay Estimation methods, which typically assume a fixed constant time delay, may not fully capture these variabilities, compromising the precision of predictive models in diverse settings. To address this issue, we introduce the Time Series Model Bootstrap (TSMB), a versatile framework designed to handle potentially varying or even nondeterministic time delays in time series modeling. Contrary to traditional approaches that hinge on the assumption of a single, consistent time delay, TSMB adopts a nonparametric stance, acknowledging and incorporating time delay uncertainties. TSMB significantly bolsters the performance of models that are trained and make predictions using this framework, making it highly suitable for a wide range of dynamic and interconnected data environments.

Title: Is Generative AI the Next Tactical Cyber Weapon For Threat Actors? Unforeseen Implications of AI Generated Cyber Attacks

• Authors: Yusuf Usman, Aadesh Upadhyay, Prashnna Gyawali, Robin Chataut
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12806
• Pdf URL: https://arxiv.org/pdf/2408.12806
• Copy Paste: [[2408.12806]] Is Generative AI the Next Tactical Cyber Weapon For Threat Actors? Unforeseen Implications of AI Generated Cyber Attacks(https://arxiv.org/abs/2408.12806)
• Keywords: security, privacy, protect, defense, attack, robust, generative, large language model
• Abstract: In an era where digital threats are increasingly sophisticated, the intersection of Artificial Intelligence and cybersecurity presents both promising defenses and potent dangers. This paper delves into the escalating threat posed by the misuse of AI, specifically through the use of Large Language Models (LLMs). This study details various techniques like the switch method and character play method, which can be exploited by cybercriminals to generate and automate cyber attacks. Through a series of controlled experiments, the paper demonstrates how these models can be manipulated to bypass ethical and privacy safeguards to effectively generate cyber attacks such as social engineering, malicious code, payload generation, and spyware. By testing these AI generated attacks on live systems, the study assesses their effectiveness and the vulnerabilities they exploit, offering a practical perspective on the risks AI poses to critical infrastructure. We also introduce Occupy AI, a customized, finetuned LLM specifically engineered to automate and execute cyberattacks. This specialized AI driven tool is adept at crafting steps and generating executable code for a variety of cyber threats, including phishing, malware injection, and system exploitation. The results underscore the urgency for ethical AI practices, robust cybersecurity measures, and regulatory oversight to mitigate AI related threats. This paper aims to elevate awareness within the cybersecurity community about the evolving digital threat landscape, advocating for proactive defense strategies and responsible AI development to protect against emerging cyber threats.

Title: VALE: A Multimodal Visual and Language Explanation Framework for Image Classifiers using eXplainable AI and Language Models

• Authors: Purushothaman Natarajan, Athira Nambiar
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12808
• Pdf URL: https://arxiv.org/pdf/2408.12808
• Copy Paste: [[2408.12808]] VALE: A Multimodal Visual and Language Explanation Framework for Image Classifiers using eXplainable AI and Language Models(https://arxiv.org/abs/2408.12808)
• Keywords: interpretability, segmentation
• Abstract: Deep Neural Networks (DNNs) have revolutionized various fields by enabling task automation and reducing human error. However, their internal workings and decision-making processes remain obscure due to their black box nature. Consequently, the lack of interpretability limits the application of these models in high-risk scenarios. To address this issue, the emerging field of eXplainable Artificial Intelligence (XAI) aims to explain and interpret the inner workings of DNNs. Despite advancements, XAI faces challenges such as the semantic gap between machine and human understanding, the trade-off between interpretability and performance, and the need for context-specific explanations. To overcome these limitations, we propose a novel multimodal framework named VALE Visual and Language Explanation. VALE integrates explainable AI techniques with advanced language models to provide comprehensive explanations. This framework utilizes visual explanations from XAI tools, an advanced zero-shot image segmentation model, and a visual language model to generate corresponding textual explanations. By combining visual and textual explanations, VALE bridges the semantic gap between machine outputs and human interpretation, delivering results that are more comprehensible to users. In this paper, we conduct a pilot study of the VALE framework for image classification tasks. Specifically, Shapley Additive Explanations (SHAP) are used to identify the most influential regions in classified images. The object of interest is then extracted using the Segment Anything Model (SAM), and explanations are generated using state-of-the-art pre-trained Vision-Language Models (VLMs). Extensive experimental studies are performed on two datasets: the ImageNet dataset and a custom underwater SONAR image dataset, demonstrating VALEs real-world applicability in underwater image classification.

Title: From Few to More: Scribble-based Medical Image Segmentation via Masked Context Modeling and Continuous Pseudo Labels

• Authors: Zhisong Wang, Yiwen Ye, Ziyang Chen, Minglei Shu, Yong Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12814
• Pdf URL: https://arxiv.org/pdf/2408.12814
• Copy Paste: [[2408.12814]] From Few to More: Scribble-based Medical Image Segmentation via Masked Context Modeling and Continuous Pseudo Labels(https://arxiv.org/abs/2408.12814)
• Keywords: segmentation
• Abstract: Scribble-based weakly supervised segmentation techniques offer comparable performance to fully supervised methods while significantly reducing annotation costs, making them an appealing alternative. Existing methods often rely on auxiliary tasks to enforce semantic consistency and use hard pseudo labels for supervision. However, these methods often overlook the unique requirements of models trained with sparse annotations. Since the model must predict pixel-wise segmentation maps with limited annotations, the ability to handle varying levels of annotation richness is critical. In this paper, we adopt the principle of `from few to more' and propose MaCo, a weakly supervised framework designed for medical image segmentation. MaCo employs masked context modeling (MCM) and continuous pseudo labels (CPL). MCM uses an attention-based masking strategy to disrupt the input image, compelling the model's predictions to remain consistent with those of the original image. CPL converts scribble annotations into continuous pixel-wise labels by applying an exponential decay function to distance maps, resulting in continuous maps that represent the confidence of each pixel belonging to a specific category, rather than using hard pseudo labels. We evaluate MaCo against other weakly supervised methods using three public datasets. The results indicate that MaCo outperforms competing methods across all datasets, setting a new record in weakly supervised medical image segmentation.

Title: Staircase Cascaded Fusion of Lightweight Local Pattern Recognition and Long-Range Dependencies for Structural Crack Segmentation

• Authors: Hui Liu, Chen Jia, Fan Shi, Xu Cheng, Mianzhao Wang, Shengyong Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12815
• Pdf URL: https://arxiv.org/pdf/2408.12815
• Copy Paste: [[2408.12815]] Staircase Cascaded Fusion of Lightweight Local Pattern Recognition and Long-Range Dependencies for Structural Crack Segmentation(https://arxiv.org/abs/2408.12815)
• Keywords: segmentation
• Abstract: Detecting cracks with pixel-level precision for key structures is a significant challenge, as existing methods struggle to effectively integrate local textures and pixel dependencies of cracks. Furthermore, these methods often possess numerous parameters and substantial computational requirements, complicating deployment on edge devices. In this paper, we propose a staircase cascaded fusion crack segmentation network (CrackSCF) that generates high-quality crack segmentation maps using minimal computational resources. We constructed a staircase cascaded fusion module that effectively captures local patterns of cracks and long-range dependencies of pixels, and it can suppress background noise well. To reduce the computational resources required by the model, we introduced a lightweight convolution block, which replaces all convolution operations in the network, significantly reducing the required computation and parameters without affecting the network's performance. To evaluate our method, we created a challenging benchmark dataset called TUT and conducted experiments on this dataset and five other public datasets. The experimental results indicate that our method offers significant advantages over existing methods, especially in handling background noise interference and detailed crack segmentation. The F1 and mIoU scores on the TUT dataset are 0.8382 and 0.8473, respectively, achieving state-of-the-art (SOTA) performance while requiring the least computational resources. The code and dataset is available at this https URL.

Title: Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagery

• Authors: Zhenyuan Yang, Xuhui Lin, Qinyi He, Ziye Huang, Zhengliang Liu, Hanqi Jiang, Peng Shu, Zihao Wu, Yiwei Li, Stephen Law, Gengchen Mai, Tianming Liu, Tao Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12821
• Pdf URL: https://arxiv.org/pdf/2408.12821
• Copy Paste: [[2408.12821]] Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagery(https://arxiv.org/abs/2408.12821)
• Keywords: large language model
• Abstract: The emergence of Large Language Models (LLMs) and multimodal foundation models (FMs) has generated heightened interest in their applications that integrate vision and language. This paper investigates the capabilities of ChatGPT-4V and Gemini Pro for Street View Imagery, Built Environment, and Interior by evaluating their performance across various tasks. The assessments include street furniture identification, pedestrian and car counts, and road width measurement in Street View Imagery; building function classification, building age analysis, building height analysis, and building structure classification in the Built Environment; and interior room classification, interior design style analysis, interior furniture counts, and interior length measurement in Interior. The results reveal proficiency in length measurement, style analysis, question answering, and basic image understanding, but highlight limitations in detailed recognition and counting tasks. While zero-shot learning shows potential, performance varies depending on the problem domains and image complexities. This study provides new insights into the strengths and weaknesses of multimodal foundation models for practical challenges in Street View Imagery, Built Environment, and Interior. Overall, the findings demonstrate foundational multimodal intelligence, emphasizing the potential of FMs to drive forward interdisciplinary applications at the intersection of computer vision and language.

Title: LIMP: Large Language Model Enhanced Intent-aware Mobility Prediction

• Authors: Songwei Li, Jie Feng, Jiawei Chi, Xinyuan Hu, Xiaomeng Zhao, Fengli Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12832
• Pdf URL: https://arxiv.org/pdf/2408.12832
• Copy Paste: [[2408.12832]] LIMP: Large Language Model Enhanced Intent-aware Mobility Prediction(https://arxiv.org/abs/2408.12832)
• Keywords: interpretability, transformer, large language model
• Abstract: Human mobility prediction is essential for applications like urban planning and transportation management, yet it remains challenging due to the complex, often implicit, intentions behind human behavior. Existing models predominantly focus on spatiotemporal patterns, paying less attention to the underlying intentions that govern movements. Recent advancements in large language models (LLMs) offer a promising alternative research angle for integrating commonsense reasoning into mobility prediction. However, it is a non-trivial problem because LLMs are not natively built for mobility intention inference, and they also face scalability issues and integration difficulties with spatiotemporal models. To address these challenges, we propose a novel LIMP (LLMs for Intent-ware Mobility Prediction) framework. Specifically, LIMP introduces an "Analyze-Abstract-Infer" (A2I) agentic workflow to unleash LLM's commonsense reasoning power for mobility intention inference. Besides, we design an efficient fine-tuning scheme to transfer reasoning power from commercial LLM to smaller-scale, open-source language model, ensuring LIMP's scalability to millions of mobility records. Moreover, we propose a transformer-based intention-aware mobility prediction model to effectively harness the intention inference ability of LLM. Evaluated on two real-world datasets, LIMP significantly outperforms baseline models, demonstrating improved accuracy in next-location prediction and effective intention inference. The interpretability of intention-aware mobility prediction highlights our LIMP framework's potential for real-world applications. Codes and data can be found in this https URL .

Title: S3Simulator: A benchmarking Side Scan Sonar Simulator dataset for Underwater Image Analysis

• Authors: Kamal Basha S, Athira Nambiar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12833
• Pdf URL: https://arxiv.org/pdf/2408.12833
• Copy Paste: [[2408.12833]] S3Simulator: A benchmarking Side Scan Sonar Simulator dataset for Underwater Image Analysis(https://arxiv.org/abs/2408.12833)
• Keywords: segmentation
• Abstract: Acoustic sonar imaging systems are widely used for underwater surveillance in both civilian and military sectors. However, acquiring high-quality sonar datasets for training Artificial Intelligence (AI) models confronts challenges such as limited data availability, financial constraints, and data confidentiality. To overcome these challenges, we propose a novel benchmark dataset of Simulated Side-Scan Sonar images, which we term as 'S3Simulator dataset'. Our dataset creation utilizes advanced simulation techniques to accurately replicate underwater conditions and produce diverse synthetic sonar imaging. In particular, the cutting-edge AI segmentation tool i.e. Segment Anything Model (SAM) is leveraged for optimally isolating and segmenting the object images, such as ships and planes, from real scenes. Further, advanced Computer-Aided Design tools i.e. SelfCAD and simulation software such as Gazebo are employed to create the 3D model and to optimally visualize within realistic environments, respectively. Further, a range of computational imaging techniques are employed to improve the quality of the data, enabling the AI models for the analysis of the sonar images. Extensive analyses are carried out on S3simulator as well as real sonar datasets to validate the performance of AI models for underwater object classification. Our experimental results highlight that the S3Simulator dataset will be a promising benchmark dataset for research on underwater image analysis. this https URL.

Title: CLLMFS: A Contrastive Learning enhanced Large Language Model Framework for Few-Shot Named Entity Recognition

• Authors: Yafeng Zhang, Zilan Yu, Yuang Huang, Jing Tang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12834
• Pdf URL: https://arxiv.org/pdf/2408.12834
• Copy Paste: [[2408.12834]] CLLMFS: A Contrastive Learning enhanced Large Language Model Framework for Few-Shot Named Entity Recognition(https://arxiv.org/abs/2408.12834)
• Keywords: robust, large language model
• Abstract: Few-shot Named Entity Recognition (NER), the task of identifying named entities with only a limited amount of labeled data, has gained increasing significance in natural language processing. While existing methodologies have shown some effectiveness, such as enriching label semantics through various prompting modes or employing metric learning techniques, their performance exhibits limited robustness across diverse domains due to the lack of rich knowledge in their pre-trained models. To address this issue, we propose CLLMFS, a Contrastive Learning enhanced Large Language Model (LLM) Framework for Few-Shot Named Entity Recognition, achieving promising results with limited training data. Considering the impact of LLM's internal representations on downstream tasks, CLLMFS integrates Low-Rank Adaptation (LoRA) and contrastive learning mechanisms specifically tailored for few-shot NER. By enhancing the model's internal representations, CLLMFS effectively improves both entity boundary awareness ability and entity recognition accuracy. Our method has achieved state-of-the-art performance improvements on F1-score ranging from 2.58\% to 97.74\% over existing best-performing methods across several recognized benchmarks. Furthermore, through cross-domain NER experiments conducted on multiple datasets, we have further validated the robust generalization capability of our method. Our code will be released in the near future.

Title: Underwater SONAR Image Classification and Analysis using LIME-based Explainable Artificial Intelligence

• Authors: Purushothaman Natarajan, Athira Nambiar
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12837
• Pdf URL: https://arxiv.org/pdf/2408.12837
• Copy Paste: [[2408.12837]] Underwater SONAR Image Classification and Analysis using LIME-based Explainable Artificial Intelligence(https://arxiv.org/abs/2408.12837)
• Keywords: security, interpretability, explainability
• Abstract: Deep learning techniques have revolutionized image classification by mimicking human cognition and automating complex decision-making processes. However, the deployment of AI systems in the wild, especially in high-security domains such as defence, is curbed by the lack of explainability of the model. To this end, eXplainable AI (XAI) is an emerging area of research that is intended to explore the unexplained hidden black box nature of deep neural networks. This paper explores the application of the eXplainable Artificial Intelligence (XAI) tool to interpret the underwater image classification results, one of the first works in the domain to the best of our knowledge. Our study delves into the realm of SONAR image classification using a custom dataset derived from diverse sources, including the Seabed Objects KLSG dataset, the camera SONAR dataset, the mine SONAR images dataset, and the SCTD dataset. An extensive analysis of transfer learning techniques for image classification using benchmark Convolutional Neural Network (CNN) architectures such as VGG16, ResNet50, InceptionV3, DenseNet121, etc. is carried out. On top of this classification model, a post-hoc XAI technique, viz. Local Interpretable Model-Agnostic Explanations (LIME) are incorporated to provide transparent justifications for the model's decisions by perturbing input data locally to see how predictions change. Furthermore, Submodular Picks LIME (SP-LIME) a version of LIME particular to images, that perturbs the image based on the submodular picks is also extensively studied. To this end, two submodular optimization algorithms i.e. Quickshift and Simple Linear Iterative Clustering (SLIC) are leveraged towards submodular picks. The extensive analysis of XAI techniques highlights interpretability of the results in a more human-compliant way, thus boosting our confidence and reliability.

Title: HGNAS: Hardware-Aware Graph Neural Architecture Search for Edge Devices

• Authors: Ao Zhou, Jianlei Yang, Yingjie Qi, Tong Qiao, Yumeng Shi, Cenlin Duan, Weisheng Zhao, Chunming Hu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12840
• Pdf URL: https://arxiv.org/pdf/2408.12840
• Copy Paste: [[2408.12840]] HGNAS: Hardware-Aware Graph Neural Architecture Search for Edge Devices(https://arxiv.org/abs/2408.12840)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) are becoming increasingly popular for graph-based learning tasks such as point cloud processing due to their state-of-the-art (SOTA) performance. Nevertheless, the research community has primarily focused on improving model expressiveness, lacking consideration of how to design efficient GNN models for edge scenarios with real-time requirements and limited resources. Examining existing GNN models reveals varied execution across platforms and frequent Out-Of-Memory (OOM) problems, highlighting the need for hardware-aware GNN design. To address this challenge, this work proposes a novel hardware-aware graph neural architecture search framework tailored for resource constraint edge devices, namely HGNAS. To achieve hardware awareness, HGNAS integrates an efficient GNN hardware performance predictor that evaluates the latency and peak memory usage of GNNs in milliseconds. Meanwhile, we study GNN memory usage during inference and offer a peak memory estimation method, enhancing the robustness of architecture evaluations when combined with predictor outcomes. Furthermore, HGNAS constructs a fine-grained design space to enable the exploration of extreme performance architectures by decoupling the GNN paradigm. In addition, the multi-stage hierarchical search strategy is leveraged to facilitate the navigation of huge candidates, which can reduce the single search time to a few GPU hours. To the best of our knowledge, HGNAS is the first automated GNN design framework for edge devices, and also the first work to achieve hardware awareness of GNNs across different platforms. Extensive experiments across various applications and edge devices have proven the superiority of HGNAS. It can achieve up to a 10.6x speedup and an 82.5% peak memory reduction with negligible accuracy loss compared to DGCNN on ModelNet40.

Title: Differentially Private Spatiotemporal Trajectory Synthesis with Retained Data Utility

• Authors: Yuqing Ge, Yunsheng Wang, Nana Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.12842
• Pdf URL: https://arxiv.org/pdf/2408.12842
• Copy Paste: [[2408.12842]] Differentially Private Spatiotemporal Trajectory Synthesis with Retained Data Utility(https://arxiv.org/abs/2408.12842)
• Keywords: privacy, protect
• Abstract: Spatiotemporal trajectories collected from GPS-enabled devices are of vital importance to many applications, such as urban planning and traffic analysis. Due to the privacy leakage concerns, many privacy-preserving trajectory publishing methods have been proposed. However, most of them could not strike a good balance between privacy protection and good data utility. In this paper, we propose DP-STTS, a differentially private spatiotemporal trajectory synthesizer with high data utility, which employs a model composed of a start spatiotemporal cube distribution and a 1-order Markov process. Specially, DP-STTS firstly discretizes the raw spatiotemporal trajectories into neighboring cubes, such that the model size is limited and the model's tolerance for noise could be enhanced. Then, a Markov process is utilized for the next location point picking. After adding noise under differential privacy (DP) to the model, synthetic trajectories that preserve essential spatial and temporal characteristics of the real trajectories are generated from the noisy model. Experiments on one real-life dataset demonstrate that DP-STTS provides good data utility. Our code is available at this https URL.

Title: Online Fair Division with Contextual Bandits

• Authors: Arun Verma, Indrajit Saha, Makoto Yokoo, Bryan Kian Hsiang Low
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.12845
• Pdf URL: https://arxiv.org/pdf/2408.12845
• Copy Paste: [[2408.12845]] Online Fair Division with Contextual Bandits(https://arxiv.org/abs/2408.12845)
• Keywords: fair
• Abstract: This paper considers a novel online fair division problem involving multiple agents in which a learner observes an indivisible item that has to be irrevocably allocated to one of the agents while satisfying a fairness and efficiency constraint. Existing algorithms assume a small number of items with a sufficiently large number of copies, which ensures a good utility estimation for all item-agent pairs. However, such an assumption may not hold in many real-life applications, e.g., an online platform that has a large number of users (items) who only use the platform's service providers (agents) a few times (a few copies of items), which makes it difficult to estimate the utility for all item-agent pairs. To overcome this challenge, we model the online fair division problem using contextual bandits, assuming the utility is an unknown function of the item-agent features. We then propose algorithms for online fair division with sub-linear regret guarantees. Our experimental results also verify the different performance aspects of the proposed algorithms.

Title: Obfuscated Memory Malware Detection

• Authors: Sharmila S P, Aruna Tiwari, Narendra S Chaudhari
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12866
• Pdf URL: https://arxiv.org/pdf/2408.12866
• Copy Paste: [[2408.12866]] Obfuscated Memory Malware Detection(https://arxiv.org/abs/2408.12866)
• Keywords: security, privacy, attack
• Abstract: Providing security for information is highly critical in the current era with devices enabled with smart technology, where assuming a day without the internet is highly impossible. Fast internet at a cheaper price, not only made communication easy for legitimate users but also for cybercriminals to induce attacks in various dimensions to breach privacy and security. Cybercriminals gain illegal access and breach the privacy of users to harm them in multiple ways. Malware is one such tool used by hackers to execute their malicious intent. Development in AI technology is utilized by malware developers to cause social harm. In this work, we intend to show how Artificial Intelligence and Machine learning can be used to detect and mitigate these cyber-attacks induced by malware in specific obfuscated malware. We conducted experiments with memory feature engineering on memory analysis of malware samples. Binary classification can identify whether a given sample is malware or not, but identifying the type of malware will only guide what next step to be taken for that malware, to stop it from proceeding with its further action. Hence, we propose a multi-class classification model to detect the three types of obfuscated malware with an accuracy of 89.07% using the Classic Random Forest algorithm. To the best of our knowledge, there is very little amount of work done in classifying multiple obfuscated malware by a single model. We also compared our model with a few state-of-the-art models and found it comparatively better.

Title: Semantic Alignment for Multimodal Large Language Models

• Authors: Tao Wu, Mengze Li, Jingyuan Chen, Wei Ji, Wang Lin, Jinyang Gao, Kun Kuang, Zhou Zhao, Fei Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12867
• Pdf URL: https://arxiv.org/pdf/2408.12867
• Copy Paste: [[2408.12867]] Semantic Alignment for Multimodal Large Language Models(https://arxiv.org/abs/2408.12867)
• Keywords: extraction, large language model
• Abstract: Research on Multi-modal Large Language Models (MLLMs) towards the multi-image cross-modal instruction has received increasing attention and made significant progress, particularly in scenarios involving closely resembling images (e.g., change captioning). Existing MLLMs typically follow a two-step process in their pipelines: first, extracting visual tokens independently for each input image, and then aligning these visual tokens from different images with the Large Language Model (LLM) in its textual feature space. However, the independent extraction of visual tokens for each image may result in different semantics being prioritized for different images in the first step, leading to a lack of preservation of linking information among images for subsequent LLM analysis. This issue becomes more serious in scenarios where significant variations exist among the images (e.g., visual storytelling). To address this challenge, we introduce Semantic Alignment for Multi-modal large language models (SAM). By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis and align the semantics of different images before feeding them into LLM. As the test bed, we propose a large-scale dataset named MmLINK consisting of 69K samples. Different from most existing datasets for MLLMs fine-tuning, our MmLINK dataset comprises multi-modal instructions with significantly diverse images. Extensive experiments on the group captioning task and the storytelling task prove the effectiveness of our SAM model, surpassing the state-of-the-art methods by a large margin (+37% for group captioning and +22% for storytelling on CIDEr score). Project page: this https URL.

Title: Disentangling, Amplifying, and Debiasing: Learning Disentangled Representations for Fair Graph Neural Networks

• Authors: Yeon-Chang Lee, Hojung Shin, Sang-Wook Kim
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2408.12875
• Pdf URL: https://arxiv.org/pdf/2408.12875
• Copy Paste: [[2408.12875]] Disentangling, Amplifying, and Debiasing: Learning Disentangled Representations for Fair Graph Neural Networks(https://arxiv.org/abs/2408.12875)
• Keywords: fair
• Abstract: Graph Neural Networks (GNNs) have become essential tools for graph representation learning in various domains, such as social media and healthcare. However, they often suffer from fairness issues due to inherent biases in node attributes and graph structure, leading to unfair predictions. To address these challenges, we propose a novel GNN framework, DAB-GNN, that Disentangles, Amplifies, and deBiases attribute, structure, and potential biases in the GNN mechanism. DAB-GNN employs a disentanglement and amplification module that isolates and amplifies each type of bias through specialized disentanglers, followed by a debiasing module that minimizes the distance between subgroup distributions to ensure fairness. Extensive experiments on five datasets demonstrate that DAB-GNN significantly outperforms ten state-of-the-art competitors in terms of achieving an optimal balance between accuracy and fairness.

Title: Unleashing the Potential of SAM2 for Biomedical Images and Videos: A Survey

• Authors: Yichi Zhang, Zhenrong Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12889
• Pdf URL: https://arxiv.org/pdf/2408.12889
• Copy Paste: [[2408.12889]] Unleashing the Potential of SAM2 for Biomedical Images and Videos: A Survey(https://arxiv.org/abs/2408.12889)
• Keywords: segmentation
• Abstract: The unprecedented developments in segmentation foundational models have become a dominant force in the field of computer vision, introducing a multitude of previously unexplored capabilities in a wide range of natural images and videos. Specifically, the Segment Anything Model (SAM) signifies a noteworthy expansion of the prompt-driven paradigm into the domain of image segmentation. The recent introduction of SAM2 effectively extends the original SAM to a streaming fashion and demonstrates strong performance in video segmentation. However, due to the substantial distinctions between natural and medical images, the effectiveness of these models on biomedical images and videos is still under exploration. This paper presents an overview of recent efforts in applying and adapting SAM2 to biomedical images and videos. The findings indicate that while SAM2 shows promise in reducing annotation burdens and enabling zero-shot segmentation, its performance varies across different datasets and tasks. Addressing the domain gap between natural and medical images through adaptation and fine-tuning is essential to fully unleash SAM2's potential in clinical applications. To support ongoing research endeavors, we maintain an active repository that contains up-to-date SAM & SAM2-related papers and projects at this https URL.

Title: SecDOAR: A Software Reference Architecture for Security Data Orchestration, Analysis and Reporting

• Authors: Muhammad Aufeef Chauhana, Muhammad Ali Babara, Fethi Rabhi
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.12904
• Pdf URL: https://arxiv.org/pdf/2408.12904
• Copy Paste: [[2408.12904]] SecDOAR: A Software Reference Architecture for Security Data Orchestration, Analysis and Reporting(https://arxiv.org/abs/2408.12904)
• Keywords: security
• Abstract: A Software Reference Architecture (SRA) is a useful tool for standardising existing architectures in a specific domain and facilitating concrete architecture design, development and evaluation by instantiating SRA and using SRA as a benchmark for the development of new systems. In this paper, we have presented an SRA for Security Data Orchestration, Analysis and Reporting (SecDOAR) to provide standardisation of security data platforms that can facilitate the integration of security orchestration, analysis and reporting tools for security data. The SecDOAR SRA has been designed by leveraging existing scientific literature and security data standards. We have documented SecDOAR SRA in terms of design methodology, meta-models to relate to different concepts in the security data architecture, and details on different elements and components of the SRA. We have evaluated SecDOAR SRA for its effectiveness and completeness by comparing it with existing commercial solutions. We have demonstrated the feasibility of the proposed SecDOAR SRA by instantiating it as a prototype platform to support security orchestration, analysis and reporting for a selected set of tools. The proposed SecDOAR SRA consists of meta-models for security data, security events and security data management processes as well as security metrics and corresponding measurement schemes, a security data integration model, and a description of SecDOAR SRA components. The proposed SecDOAR SRA can be used by researchers and practitioners as a structured approach for designing and implementing cybersecurity monitoring, analysis and reporting systems in various domains.

Title: ParGo: Bridging Vision-Language with Partial and Global Views

• Authors: An-Lan Wang, Bin Shan, Wei Shi, Kun-Yu Lin, Xiang Fei, Guozhi Tang, Lei Liao, Jingqun Tang, Can Huang, Wei-Shi Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12928
• Pdf URL: https://arxiv.org/pdf/2408.12928
• Copy Paste: [[2408.12928]] ParGo: Bridging Vision-Language with Partial and Global Views(https://arxiv.org/abs/2408.12928)
• Keywords: large language model
• Abstract: This work presents ParGo, a novel Partial-Global projector designed to connect the vision and language modalities for Multimodal Large Language Models (MLLMs). Unlike previous works that rely on global attention-based projectors, our ParGo bridges the representation gap between the separately pre-trained vision encoders and the LLMs by integrating global and partial views, which alleviates the overemphasis on prominent regions. To facilitate the effective training of ParGo, we collect a large-scale detail-captioned image-text dataset named ParGoCap-1M-PT, consisting of 1 million images paired with high-quality captions. Extensive experiments on several MLLM benchmarks demonstrate the effectiveness of our ParGo, highlighting its superiority in aligning vision and language modalities. Compared to conventional Q-Former projector, our ParGo achieves an improvement of 259.96 in MME benchmark. Furthermore, our experiments reveal that ParGo significantly outperforms other projectors, particularly in tasks that emphasize detail perception ability.

Title: Animal Identification with Independent Foreground and Background Modeling

• Authors: Lukas Picek, Lukas Neumann, Jiri Matas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12930
• Pdf URL: https://arxiv.org/pdf/2408.12930
• Copy Paste: [[2408.12930]] Animal Identification with Independent Foreground and Background Modeling(https://arxiv.org/abs/2408.12930)
• Keywords: robust
• Abstract: We propose a method that robustly exploits background and foreground in visual identification of individual animals. Experiments show that their automatic separation, made easy with methods like Segment Anything, together with independent foreground and background-related modeling, improves results. The two predictions are combined in a principled way, thanks to novel Per-Instance Temperature Scaling that helps the classifier to deal with appearance ambiguities in training and to produce calibrated outputs in the inference phase. For identity prediction from the background, we propose novel spatial and temporal models. On two problems, the relative error w.r.t. the baseline was reduced by 22.3% and 8.8%, respectively. For cases where objects appear in new locations, an example of background drift, accuracy doubles.

Title: Smooth InfoMax -- Towards easier Post-Hoc interpretability

• Authors: Fabian Denoodt, Bart de Boer, José Oramas
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12936
• Pdf URL: https://arxiv.org/pdf/2408.12936
• Copy Paste: [[2408.12936]] Smooth InfoMax -- Towards easier Post-Hoc interpretability(https://arxiv.org/abs/2408.12936)
• Keywords: interpretability
• Abstract: We introduce Smooth InfoMax (SIM), a novel method for self-supervised representation learning that incorporates an interpretability constraint into the learned representations at various depths of the neural network. SIM's architecture is split up into probabilistic modules, each locally optimized using the InfoNCE bound. Inspired by VAEs, the representations from these modules are designed to be samples from Gaussian distributions and are further constrained to be close to the standard normal distribution. This results in a smooth and predictable space, enabling traversal of the latent space through a decoder for easier post-hoc analysis of the learned representations. We evaluate SIM's performance on sequential speech data, showing that it performs competitively with its less interpretable counterpart, Greedy InfoMax (GIM). Moreover, we provide insights into SIM's internal representations, demonstrating that the contained information is less entangled throughout the representation and more concentrated in a smaller subset of the dimensions. This further highlights the improved interpretability of SIM.

Title: Causal-Guided Active Learning for Debiasing Large Language Models

• Authors: Zhouhao Sun, Li Du, Xiao Ding, Yixuan Ma, Kaitao Qiu, Ting Liu, Bing Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12942
• Pdf URL: https://arxiv.org/pdf/2408.12942
• Copy Paste: [[2408.12942]] Causal-Guided Active Learning for Debiasing Large Language Models(https://arxiv.org/abs/2408.12942)
• Keywords: generative, large language model
• Abstract: Although achieving promising performance, recent analyses show that current generative large language models (LLMs) may still capture dataset biases and utilize them for generation, leading to poor generalizability and harmfulness of LLMs. However, due to the diversity of dataset biases and the over-optimization problem, previous prior-knowledge-based debiasing methods and fine-tuning-based debiasing methods may not be suitable for current LLMs. To address this issue, we explore combining active learning with the causal mechanisms and propose a casual-guided active learning (CAL) framework, which utilizes LLMs itself to automatically and autonomously identify informative biased samples and induce the bias patterns. Then a cost-effective and efficient in-context learning based method is employed to prevent LLMs from utilizing dataset biases during generation. Experimental results show that CAL can effectively recognize typical biased instances and induce various bias patterns for debiasing LLMs.

Title: Find the Assembly Mistakes: Error Segmentation for Industrial Applications

• Authors: Dan Lehman, Tim J. Schoonbeek, Shao-Hsuan Hung, Jacek Kustra, Peter H.N. de With, Fons van der Sommen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12945
• Pdf URL: https://arxiv.org/pdf/2408.12945
• Copy Paste: [[2408.12945]] Find the Assembly Mistakes: Error Segmentation for Industrial Applications(https://arxiv.org/abs/2408.12945)
• Keywords: segmentation
• Abstract: Recognizing errors in assembly and maintenance procedures is valuable for industrial applications, since it can increase worker efficiency and prevent unplanned down-time. Although assembly state recognition is gaining attention, none of the current works investigate assembly error localization. Therefore, we propose StateDiffNet, which localizes assembly errors based on detecting the differences between a (correct) intended assembly state and a test image from a similar viewpoint. StateDiffNet is trained on synthetically generated image pairs, providing full control over the type of meaningful change that should be detected. The proposed approach is the first to correctly localize assembly errors taken from real ego-centric video data for both states and error types that are never presented during training. Furthermore, the deployment of change detection to this industrial application provides valuable insights and considerations into the mechanisms of state-of-the-art change detection algorithms. The code and data generation pipeline are publicly available at: this https URL.

Title: State-of-the-Art Fails in the Art of Damage Detection

• Authors: Daniela Ivanova, Marco Aversa, Paul Henderson, John Williamson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12953
• Pdf URL: https://arxiv.org/pdf/2408.12953
• Copy Paste: [[2408.12953]] State-of-the-Art Fails in the Art of Damage Detection(https://arxiv.org/abs/2408.12953)
• Keywords: diffusion, transformer, segmentation
• Abstract: Accurately detecting and classifying damage in analogue media such as paintings, photographs, textiles, mosaics, and frescoes is essential for cultural heritage preservation. While machine learning models excel in correcting global degradation if the damage operator is known a priori, we show that they fail to predict where the damage is even after supervised training; thus, reliable damage detection remains a challenge. We introduce DamBench, a dataset for damage detection in diverse analogue media, with over 11,000 annotations covering 15 damage types across various subjects and media. We evaluate CNN, Transformer, and text-guided diffusion segmentation models, revealing their limitations in generalising across media types.

Title: Image Segmentation in Foundation Model Era: A Survey

• Authors: Tianfei Zhou, Fei Zhang, Boyu Chang, Wenguan Wang, Ye Yuan, Ender Konukoglu, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12957
• Pdf URL: https://arxiv.org/pdf/2408.12957
• Copy Paste: [[2408.12957]] Image Segmentation in Foundation Model Era: A Survey(https://arxiv.org/abs/2408.12957)
• Keywords: diffusion, segmentation
• Abstract: Image segmentation is a long-standing challenge in computer vision, studied continuously over several decades, as evidenced by seminal algorithms such as N-Cut, FCN, and MaskFormer. With the advent of foundation models (FMs), contemporary segmentation methodologies have embarked on a new epoch by either adapting FMs (e.g., CLIP, Stable Diffusion, DINO) for image segmentation or developing dedicated segmentation foundation models (e.g., SAM). These approaches not only deliver superior segmentation performance, but also herald newfound segmentation capabilities previously unseen in deep learning context. However, current research in image segmentation lacks a detailed analysis of distinct characteristics, challenges, and solutions associated with these advancements. This survey seeks to fill this gap by providing a thorough review of cutting-edge research centered around FM-driven image segmentation. We investigate two basic lines of research -- generic image segmentation (i.e., semantic segmentation, instance segmentation, panoptic segmentation), and promptable image segmentation (i.e., interactive segmentation, referring segmentation, few-shot segmentation) -- by delineating their respective task settings, background concepts, and key challenges. Furthermore, we provide insights into the emergence of segmentation knowledge from FMs like CLIP, Stable Diffusion, and DINO. An exhaustive overview of over 300 segmentation approaches is provided to encapsulate the breadth of current research efforts. Subsequently, we engage in a discussion of open issues and potential avenues for future research. We envisage that this fresh, comprehensive, and systematic survey catalyzes the evolution of advanced image segmentation systems.

Title: Multimodal Contrastive In-Context Learning

• Authors: Yosuke Miyanishi, Minh Le Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12959
• Pdf URL: https://arxiv.org/pdf/2408.12959
• Copy Paste: [[2408.12959]] Multimodal Contrastive In-Context Learning(https://arxiv.org/abs/2408.12959)
• Keywords: robust, large language model
• Abstract: The rapid growth of Large Language Models (LLMs) usage has highlighted the importance of gradient-free in-context learning (ICL). However, interpreting their inner workings remains challenging. This paper introduces a novel multimodal contrastive in-context learning framework to enhance our understanding of ICL in LLMs. First, we present a contrastive learning-based interpretation of ICL in real-world settings, marking the distance of the key-value representation as the differentiator in ICL. Second, we develop an analytical framework to address biases in multimodal input formatting for real-world datasets. We demonstrate the effectiveness of ICL examples where baseline performance is poor, even when they are represented in unseen formats. Lastly, we propose an on-the-fly approach for ICL (Anchored-by-Text ICL) that demonstrates effectiveness in detecting hateful memes, a task where typical ICL struggles due to resource limitations. Extensive experiments on multimodal datasets reveal that our approach significantly improves ICL performance across various scenarios, such as challenging tasks and resource-constrained environments. Moreover, it provides valuable insights into the mechanisms of in-context learning in LLMs. Our findings have important implications for developing more interpretable, efficient, and robust multimodal AI systems, especially in challenging tasks and resource-constrained environments.

Title: Open Llama2 Model for the Lithuanian Language

• Authors: Artūras Nakvosas, Povilas Daniušis, Vytas Mulevičius
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12963
• Pdf URL: https://arxiv.org/pdf/2408.12963
• Copy Paste: [[2408.12963]] Open Llama2 Model for the Lithuanian Language(https://arxiv.org/abs/2408.12963)
• Keywords: large language model
• Abstract: In this paper, we propose and describe the first open Llama2 large language models (LLMs) for the Lithuanian language, including an accompanying question/answer (Q/A) dataset and translations of popular LLM benchmarks. We provide a brief review of open regional LLMs and detailed information on the proposed LLMs and their training process. We also conduct an empirical evaluation, comparing the perplexities of the proposed LLMs with those of other modern open LLMs. In addition, benchmarking the proposed LLMs against language understanding tasks reveals that high-quality pretraining datasets may be essential for achieving models that perform efficiently on these benchmarks. The full realisations of the described LLMs are available in the accompanying open repository~\url{this https URL}.

Title: Accuracy Improvement of Cell Image Segmentation Using Feedback Former

• Authors: Hinako Mitsuoka, Kazuhiro Hotta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.12974
• Pdf URL: https://arxiv.org/pdf/2408.12974
• Copy Paste: [[2408.12974]] Accuracy Improvement of Cell Image Segmentation Using Feedback Former(https://arxiv.org/abs/2408.12974)
• Keywords: transformer, segmentation
• Abstract: Semantic segmentation of microscopy cell images by deep learning is a significant technique. We considered that the Transformers, which have recently outperformed CNNs in image recognition, could also be improved and developed for cell image segmentation. Transformers tend to focus more on contextual information than on detailed information. This tendency leads to a lack of detailed information for segmentation. Therefore, to supplement or reinforce the missing detailed information, we hypothesized that feedback processing in the human visual cortex should be effective. Our proposed Feedback Former is a novel architecture for semantic segmentation, in which Transformers is used as an encoder and has a feedback processing mechanism. Feature maps with detailed information are fed back to the lower layers from near the output of the model to compensate for the lack of detailed information which is the weakness of Transformers and improve the segmentation accuracy. By experiments on three cell image datasets, we confirmed that our method surpasses methods without feedback, demonstrating its superior accuracy in cell image segmentation. Our method achieved higher segmentation accuracy while consuming less computational cost than conventional feedback approaches. Moreover, our method offered superior precision without simply increasing the model size of Transformer encoder, demonstrating higher accuracy with lower computational cost.

Title: Energy-Efficient Spiking Recurrent Neural Network for Gesture Recognition on Embedded GPUs

• Authors: Marzieh Hassanshahi Varposhti, Mahyar Shahsavari, Marcel van Gerven
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12978
• Pdf URL: https://arxiv.org/pdf/2408.12978
• Copy Paste: [[2408.12978]] Energy-Efficient Spiking Recurrent Neural Network for Gesture Recognition on Embedded GPUs(https://arxiv.org/abs/2408.12978)
• Keywords: robust
• Abstract: Implementing AI algorithms on event-based embedded devices enables real-time processing of data, minimizes latency, and enhances power efficiency in edge computing. This research explores the deployment of a spiking recurrent neural network (SRNN) with liquid time constant neurons for gesture recognition. We focus on the energy efficiency and computational efficacy of NVIDIA Jetson Nano embedded GPU platforms. The embedded GPU showcases a 14-fold increase in power efficiency relative to a conventional GPU, making a compelling argument for its use in energy-constrained applications. The study's empirical findings also highlight that batch processing significantly boosts frame rates across various batch sizes while maintaining accuracy levels well above the baseline. These insights validate the SRNN with liquid time constant neurons as a robust model for interpreting temporal-spatial data in gesture recognition, striking a critical balance between processing speed and power frugality.

Title: MedDec: A Dataset for Extracting Medical Decisions from Discharge Summaries

• Authors: Mohamed Elgaar, Jiali Cheng, Nidhi Vakil, Hadi Amiri, Leo Anthony Celi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12980
• Pdf URL: https://arxiv.org/pdf/2408.12980
• Copy Paste: [[2408.12980]] MedDec: A Dataset for Extracting Medical Decisions from Discharge Summaries(https://arxiv.org/abs/2408.12980)
• Keywords: extraction
• Abstract: Medical decisions directly impact individuals' health and well-being. Extracting decision spans from clinical notes plays a crucial role in understanding medical decision-making processes. In this paper, we develop a new dataset called "MedDec", which contains clinical notes of eleven different phenotypes (diseases) annotated by ten types of medical decisions. We introduce the task of medical decision extraction, aiming to jointly extract and classify different types of medical decisions within clinical notes. We provide a comprehensive analysis of the dataset, develop a span detection model as a baseline for this task, evaluate recent span detection approaches, and employ a few metrics to measure the complexity of data samples. Our findings shed light on the complexities inherent in clinical decision extraction and enable future work in this area of research. The dataset and code are available through this https URL.

Title: Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection

• Authors: Niklas Risse, Marcel Böhme
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12986
• Pdf URL: https://arxiv.org/pdf/2408.12986
• Copy Paste: [[2408.12986]] Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection(https://arxiv.org/abs/2408.12986)
• Keywords: security
• Abstract: According to our survey of the machine learning for vulnerability detection (ML4VD) literature published in the top Software Engineering conferences, every paper in the past 5 years defines ML4VD as a binary classification problem: Given a function, does it contain a security flaw? In this paper, we ask whether this decision can really be made without further context and study both vulnerable and non-vulnerable functions in the most popular ML4VD datasets. A function is vulnerable if it was involved in a patch of an actual security flaw and confirmed to cause the vulnerability. It is non-vulnerable otherwise. We find that in almost all cases this decision cannot be made without further context. Vulnerable functions are often vulnerable only because a corresponding vulnerability-inducing calling context exists while non-vulnerable functions would often be vulnerable if a corresponding context existed. But why do ML4VD techniques perform so well even though there is demonstrably not enough information in these samples? Spurious correlations: We find that high accuracy can be achieved even when only word counts are available. This shows that these datasets can be exploited to achieve high accuracy without actually detecting any security vulnerabilities. We conclude that the current problem statement of ML4VD is ill-defined and call into question the internal validity of this growing body of work. Constructively, we call for more effective benchmarking methodologies to evaluate the true capabilities of ML4VD, propose alternative problem statements, and examine broader implications for the evaluation of machine learning and programming analysis research.

Title: RIFF: Inducing Rules for Fraud Detection from Decision Trees

• Authors: João Lucas Martins, João Bravo, Ana Sofia Gomes, Carlos Soares, Pedro Bizarro
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12989
• Pdf URL: https://arxiv.org/pdf/2408.12989
• Copy Paste: [[2408.12989]] RIFF: Inducing Rules for Fraud Detection from Decision Trees(https://arxiv.org/abs/2408.12989)
• Keywords: interpretability
• Abstract: Financial fraud is the cause of multi-billion dollar losses annually. Traditionally, fraud detection systems rely on rules due to their transparency and interpretability, key features in domains where decisions need to be explained. However, rule systems require significant input from domain experts to create and tune, an issue that rule induction algorithms attempt to mitigate by inferring rules directly from data. We explore the application of these algorithms to fraud detection, where rule systems are constrained to have a low false positive rate (FPR) or alert rate, by proposing RIFF, a rule induction algorithm that distills a low FPR rule set directly from decision trees. Our experiments show that the induced rules are often able to maintain or improve performance of the original models for low FPR tasks, while substantially reducing their complexity and outperforming rules hand-tuned by experts.

Title: Measuring Variable Importance in Individual Treatment Effect Estimation with High Dimensional Data

• Authors: Joseph Paillard, Vitaliy Kolodyazhniy, Bertrand Thirion, Denis A. Engemann
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13002
• Pdf URL: https://arxiv.org/pdf/2408.13002
• Copy Paste: [[2408.13002]] Measuring Variable Importance in Individual Treatment Effect Estimation with High Dimensional Data(https://arxiv.org/abs/2408.13002)
• Keywords: interpretability
• Abstract: Causal machine learning (ML) promises to provide powerful tools for estimating individual treatment effects. Although causal ML methods are now well established, they still face the significant challenge of interpretability, which is crucial for medical applications. In this work, we propose a new algorithm based on the Conditional Permutation Importance (CPI) method for statistically rigorous variable importance assessment in the context of Conditional Average Treatment Effect (CATE) estimation. Our method termed PermuCATE is agnostic to both the meta-learner and the ML model used. Through theoretical analysis and empirical studies, we show that this approach provides a reliable measure of variable importance and exhibits lower variance compared to the standard Leave-One-Covariate-Out (LOCO) method. We illustrate how this property leads to increased statistical power, which is crucial for the application of explainable ML in small sample sizes or high-dimensional settings. We empirically demonstrate the benefits of our approach in various simulation scenarios, including previously proposed benchmarks as well as more complex settings with high-dimensional and correlated variables that require advanced CATE estimators.

Title: EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolation

• Authors: Cong Wang, Jiaxi Gu, Panwen Hu, Haoyu Zhao, Yuanfan Guo, Jianhua Han, Hang Xu, Xiaodan Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13005
• Pdf URL: https://arxiv.org/pdf/2408.13005
• Copy Paste: [[2408.13005]] EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolation(https://arxiv.org/abs/2408.13005)
• Keywords: diffusion
• Abstract: Following the advancements in text-guided image generation technology exemplified by Stable Diffusion, video generation is gaining increased attention in the academic community. However, relying solely on text guidance for video generation has serious limitations, as videos contain much richer content than images, especially in terms of motion. This information can hardly be adequately described with plain text. Fortunately, in computer vision, various visual representations can serve as additional control signals to guide generation. With the help of these signals, video generation can be controlled in finer detail, allowing for greater flexibility for different applications. Integrating various controls, however, is nontrivial. In this paper, we propose a universal framework called EasyControl. By propagating and injecting condition features through condition adapters, our method enables users to control video generation with a single condition map. With our framework, various conditions including raw pixels, depth, HED, etc., can be integrated into different Unet-based pre-trained video diffusion models at a low practical cost. We conduct comprehensive experiments on public datasets, and both quantitative and qualitative results indicate that our method outperforms state-of-the-art methods. EasyControl significantly improves various evaluation metrics across multiple validation datasets compared to previous works. Specifically, for the sketch-to-video generation task, EasyControl achieves an improvement of 152.0 on FVD and 19.9 on IS, respectively, in UCF101 compared with VideoComposer. For fidelity, our model demonstrates powerful image retention ability, resulting in high FVD and IS in UCF101 and MSR-VTT compared to other image-to-video models.

Title: Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates

• Authors: Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, Mei Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13006
• Pdf URL: https://arxiv.org/pdf/2408.13006
• Copy Paste: [[2408.13006]] Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates(https://arxiv.org/abs/2408.13006)
• Keywords: robust, interpretability, explainability, large language model
• Abstract: Alignment approaches such as RLHF and DPO are actively investigated to align large language models (LLMs) with human preferences. Commercial large language models (LLMs) like GPT-4 have been recently employed to evaluate and compare different LLM alignment approaches. These models act as surrogates for human evaluators due to their promising abilities to approximate human preferences with remarkably faster feedback and lower costs. This methodology is referred to as LLM-as-a-judge. However, concerns regarding its reliability have emerged, attributed to LLM judges' biases and inconsistent decision-making. Previous research has sought to develop robust evaluation frameworks for assessing the reliability of LLM judges and their alignment with human preferences. However, the employed evaluation metrics often lack adequate explainability and fail to address the internal inconsistency of LLMs. Additionally, existing studies inadequately explore the impact of various prompt templates when applying LLM-as-a-judge methods, which leads to potentially inconsistent comparisons between different alignment algorithms. In this work, we systematically evaluate LLM judges on alignment tasks (e.g. summarization) by defining evaluation metrics with improved theoretical interpretability and disentangling reliability metrics with LLM internal inconsistency. We develop a framework to evaluate, compare, and visualize the reliability and alignment of LLM judges to provide informative observations that help choose LLM judges for alignment tasks. Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.

Title: A Web-Based Solution for Federated Learning with LLM-Based Automation

• Authors: Chamith Mawela, Chaouki Ben Issaid, Mehdi Bennis
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2408.13010
• Pdf URL: https://arxiv.org/pdf/2408.13010
• Copy Paste: [[2408.13010]] A Web-Based Solution for Federated Learning with LLM-Based Automation(https://arxiv.org/abs/2408.13010)
• Keywords: federate
• Abstract: Federated Learning (FL) offers a promising approach for collaborative machine learning across distributed devices. However, its adoption is hindered by the complexity of building reliable communication architectures and the need for expertise in both machine learning and network programming. This paper presents a comprehensive solution that simplifies the orchestration of FL tasks while integrating intent-based automation. We develop a user-friendly web application supporting the federated averaging (FedAvg) algorithm, enabling users to configure parameters through an intuitive interface. The backend solution efficiently manages communication between the parameter server and edge nodes. We also implement model compression and scheduling algorithms to optimize FL performance. Furthermore, we explore intent-based automation in FL using a fine-tuned Language Model (LLM) trained on a tailored dataset, allowing users to conduct FL tasks using high-level prompts. We observe that the LLM-based automated solution achieves comparable test accuracy to the standard web-based solution while reducing transferred bytes by up to 64% and CPU time by up to 46% for FL tasks. Also, we leverage the neural architecture search (NAS) and hyperparameter optimization (HPO) using LLM to improve the performance. We observe that by using this approach test accuracy can be improved by 10-20% for the carried out FL tasks.

Title: Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding

• Authors: Xianqiang Gao, Pingrui Zhang, Delin Qu, Dong Wang, Zhigang Wang, Yan Ding, Bin Zhao, Xuelong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13024
• Pdf URL: https://arxiv.org/pdf/2408.13024
• Copy Paste: [[2408.13024]] Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding(https://arxiv.org/abs/2408.13024)
• Keywords: extraction
• Abstract: 3D Object Affordance Grounding aims to predict the functional regions on a 3D object and has laid the foundation for a wide range of applications in robotics. Recent advances tackle this problem via learning a mapping between 3D regions and a single human-object interaction image. However, the geometric structure of the 3D object and the object in the human-object interaction image are not always consistent, leading to poor generalization. To address this issue, we propose to learn generalizable invariant affordance knowledge from multiple human-object interaction images within the same affordance category. Specifically, we introduce the \textbf{M}ulti-\textbf{I}mage Guided Invariant-\textbf{F}eature-Aware 3D \textbf{A}ffordance \textbf{G}rounding (\textbf{MIFAG}) framework. It grounds 3D object affordance regions by identifying common interaction patterns across multiple human-object interaction images. First, the Invariant Affordance Knowledge Extraction Module (\textbf{IAM}) utilizes an iterative updating strategy to gradually extract aligned affordance knowledge from multiple images and integrate it into an affordance dictionary. Then, the Affordance Dictionary Adaptive Fusion Module (\textbf{ADM}) learns comprehensive point cloud representations that consider all affordance candidates in multiple images. Besides, the Multi-Image and Point Affordance (\textbf{MIPA}) benchmark is constructed and our method outperforms existing state-of-the-art methods on various experimental comparisons. Project page: \url{this https URL}

Title: In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting

• Authors: Haowei Du, Dongyan Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13028
• Pdf URL: https://arxiv.org/pdf/2408.13028
• Copy Paste: [[2408.13028]] In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting(https://arxiv.org/abs/2408.13028)
• Keywords: large language model
• Abstract: In-context learning (ICL) of large language models (LLMs) has attracted increasing attention in the community where LLMs make predictions only based on instructions augmented with a few examples. Existing example selection methods for ICL utilize sparse or dense retrievers and derive effective performance. However, these methods do not utilize direct feedback of LLM to train the retriever and the examples selected can not necessarily improve the analogy ability of LLM. To tackle this, we propose our policy-based reinforcement learning framework for example selection (RLS), which consists of a language model (LM) selector and an LLM generator. The LM selector encodes the candidate examples into dense representations and selects the top-k examples into the demonstration for LLM. The outputs of LLM are adopted to compute the reward and policy gradient to optimize the LM selector. We conduct experiments on different datasets and significantly outperform existing example selection methods. Moreover, our approach shows advantages over supervised finetuning (SFT) models in few shot setting. Further experiments show the balance of abundance and the similarity with the test case of examples is important for ICL performance of LLM.

Title: Indoor scene recognition from images under visual corruptions

• Authors: Willams de Lima Costa, Raul Ismayilov, Nicola Strisciuglio, Estefania Talavera Martinez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13029
• Pdf URL: https://arxiv.org/pdf/2408.13029
• Copy Paste: [[2408.13029]] Indoor scene recognition from images under visual corruptions(https://arxiv.org/abs/2408.13029)
• Keywords: robust
• Abstract: The classification of indoor scenes is a critical component in various applications, such as intelligent robotics for assistive living. While deep learning has significantly advanced this field, models often suffer from reduced performance due to image corruption. This paper presents an innovative approach to indoor scene recognition that leverages multimodal data fusion, integrating caption-based semantic features with visual data to enhance both accuracy and robustness against corruption. We examine two multimodal networks that synergize visual features from CNN models with semantic captions via a Graph Convolutional Network (GCN). Our study shows that this fusion markedly improves model performance, with notable gains in Top-1 accuracy when evaluated against a corrupted subset of the Places365 dataset. Moreover, while standalone visual models displayed high accuracy on uncorrupted images, their performance deteriorated significantly with increased corruption severity. Conversely, the multimodal models demonstrated improved accuracy in clean conditions and substantial robustness to a range of image corruptions. These results highlight the efficacy of incorporating high-level contextual information through captions, suggesting a promising direction for enhancing the resilience of classification systems.

Title: VFM-Det: Towards High-Performance Vehicle Detection via Large Foundation Models

• Authors: Wentao Wu, Fanghua Hong, Xiao Wang, Chenglong Li, Jin Tang
• Subjects: cs.CV, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2408.13031
• Pdf URL: https://arxiv.org/pdf/2408.13031
• Copy Paste: [[2408.13031]] VFM-Det: Towards High-Performance Vehicle Detection via Large Foundation Models(https://arxiv.org/abs/2408.13031)
• Keywords: large language model
• Abstract: Existing vehicle detectors are usually obtained by training a typical detector (e.g., YOLO, RCNN, DETR series) on vehicle images based on a pre-trained backbone (e.g., ResNet, ViT). Some researchers also exploit and enhance the detection performance using pre-trained large foundation models. However, we think these detectors may only get sub-optimal results because the large models they use are not specifically designed for vehicles. In addition, their results heavily rely on visual features, and seldom of they consider the alignment between the vehicle's semantic information and visual representations. In this work, we propose a new vehicle detection paradigm based on a pre-trained foundation vehicle model (VehicleMAE) and a large language model (T5), termed VFM-Det. It follows the region proposal-based detection framework and the features of each proposal can be enhanced using VehicleMAE. More importantly, we propose a new VAtt2Vec module that predicts the vehicle semantic attributes of these proposals and transforms them into feature vectors to enhance the vision features via contrastive learning. Extensive experiments on three vehicle detection benchmark datasets thoroughly proved the effectiveness of our vehicle detector. Specifically, our model improves the baseline approach by $+5.1\%$, $+6.2\%$ on the $AP_{0.5}$, $AP_{0.75}$ metrics, respectively, on the Cityscapes dataset.The source code of this work will be released at this https URL.

Title: S4D: Streaming 4D Real-World Reconstruction with Gaussians and 3D Control Points

• Authors: Bing He, Yunuo Chen, Guo Lu, Li Song, Wenjun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13036
• Pdf URL: https://arxiv.org/pdf/2408.13036
• Copy Paste: [[2408.13036]] S4D: Streaming 4D Real-World Reconstruction with Gaussians and 3D Control Points(https://arxiv.org/abs/2408.13036)
• Keywords: robust, segmentation
• Abstract: Recently, the dynamic scene reconstruction using Gaussians has garnered increased interest. Mainstream approaches typically employ a global deformation field to warp a 3D scene in the canonical space. However, the inherently low-frequency nature of implicit neural fields often leads to ineffective representations of complex motions. Moreover, their structural rigidity can hinder adaptation to scenes with varying resolutions and durations. To overcome these challenges, we introduce a novel approach utilizing discrete 3D control points. This method models local rays physically and establishes a motion-decoupling coordinate system, which effectively merges traditional graphics with learnable pipelines for a robust and efficient local 6-degrees-of-freedom (6-DoF) motion representation. Additionally, we have developed a generalized framework that incorporates our control points with Gaussians. Starting from an initial 3D reconstruction, our workflow decomposes the streaming 4D real-world reconstruction into four independent submodules: 3D segmentation, 3D control points generation, object-wise motion manipulation, and residual compensation. Our experiments demonstrate that this method outperforms existing state-of-the-art 4D Gaussian Splatting techniques on both the Neu3DV and CMU-Panoptic datasets. Our approach also significantly accelerates training, with the optimization of our 3D control points achievable within just 2 seconds per frame on a single NVIDIA 4070 GPU.

Title: Improving the Classification Effect of Clinical Images of Diseases for Multi-Source Privacy Protection

• Authors: Tian Bowen, Xu Zhengyang, Yin Zhihao, Wang Jingying, Yue Yutao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13038
• Pdf URL: https://arxiv.org/pdf/2408.13038
• Copy Paste: [[2408.13038]] Improving the Classification Effect of Clinical Images of Diseases for Multi-Source Privacy Protection(https://arxiv.org/abs/2408.13038)
• Keywords: privacy, protect, federate
• Abstract: Privacy data protection in the medical field poses challenges to data sharing, limiting the ability to integrate data across hospitals for training high-precision auxiliary diagnostic models. Traditional centralized training methods are difficult to apply due to violations of privacy protection principles. Federated learning, as a distributed machine learning framework, helps address this issue, but it requires multiple hospitals to participate in training simultaneously, which is hard to achieve in practice. To address these challenges, we propose a medical privacy data training framework based on data vectors. This framework allows each hospital to fine-tune pre-trained models on private data, calculate data vectors (representing the optimization direction of model parameters in the solution space), and sum them up to generate synthetic weights that integrate model information from multiple hospitals. This approach enhances model performance without exchanging private data or requiring synchronous training. Experimental results demonstrate that this method effectively utilizes dispersed private data resources while protecting patient privacy. The auxiliary diagnostic model trained using this approach significantly outperforms models trained independently by a single hospital, providing a new perspective for resolving the conflict between medical data privacy protection and model training and advancing the development of medical intelligence.

Title: G3FA: Geometry-guided GAN for Face Animation

• Authors: Alireza Javanmardi, Alain Pagani, Didier Stricker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13049
• Pdf URL: https://arxiv.org/pdf/2408.13049
• Copy Paste: [[2408.13049]] G3FA: Geometry-guided GAN for Face Animation(https://arxiv.org/abs/2408.13049)
• Keywords: generative
• Abstract: Animating human face images aims to synthesize a desired source identity in a natural-looking way mimicking a driving video's facial movements. In this context, Generative Adversarial Networks have demonstrated remarkable potential in real-time face reenactment using a single source image, yet are constrained by limited geometry consistency compared to graphic-based approaches. In this paper, we introduce Geometry-guided GAN for Face Animation (G3FA) to tackle this limitation. Our novel approach empowers the face animation model to incorporate 3D information using only 2D images, improving the image generation capabilities of the talking head synthesis model. We integrate inverse rendering techniques to extract 3D facial geometry properties, improving the feedback loop to the generator through a weighted average ensemble of discriminators. In our face reenactment model, we leverage 2D motion warping to capture motion dynamics along with orthogonal ray sampling and volume rendering techniques to produce the ultimate visual output. To evaluate the performance of our G3FA, we conducted comprehensive experiments using various evaluation protocols on VoxCeleb2 and TalkingHead benchmarks to demonstrate the effectiveness of our proposed framework compared to the state-of-the-art real-time face animation methods.

Title: Atlas Gaussians Diffusion for 3D Generation with Infinite Number of Points

• Authors: Haitao Yang, Yuan Dong, Hanwen Jiang, Dejia Xu, Georgios Pavlakos, Qixing Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13055
• Pdf URL: https://arxiv.org/pdf/2408.13055
• Copy Paste: [[2408.13055]] Atlas Gaussians Diffusion for 3D Generation with Infinite Number of Points(https://arxiv.org/abs/2408.13055)
• Keywords: diffusion, transformer
• Abstract: Using the latent diffusion model has proven effective in developing novel 3D generation techniques. To harness the latent diffusion model, a key challenge is designing a high-fidelity and efficient representation that links the latent space and the 3D space. In this paper, we introduce Atlas Gaussians, a novel representation for feed-forward native 3D generation. Atlas Gaussians represent a shape as the union of local patches, and each patch can decode 3D Gaussians. We parameterize a patch as a sequence of feature vectors and design a learnable function to decode 3D Gaussians from the feature vectors. In this process, we incorporate UV-based sampling, enabling the generation of a sufficiently large, and theoretically infinite, number of 3D Gaussian points. The large amount of 3D Gaussians enables high-quality details of generation results. Moreover, due to local awareness of the representation, the transformer-based decoding procedure operates on a patch level, ensuring efficiency. We train a variational autoencoder to learn the Atlas Gaussians representation, and then apply a latent diffusion model on its latent space for learning 3D Generation. Experiments show that our approach outperforms the prior arts of feed-forward native 3D generation.

Title: IntelliCare: Improving Healthcare Analysis with Variance-Controlled Patient-Level Knowledge from Large Language Models

• Authors: Zhihao Yu, Yujie Jin, Yongxin Xu, Xu Chu, Yasha Wang, Junfeng Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13073
• Pdf URL: https://arxiv.org/pdf/2408.13073
• Copy Paste: [[2408.13073]] IntelliCare: Improving Healthcare Analysis with Variance-Controlled Patient-Level Knowledge from Large Language Models(https://arxiv.org/abs/2408.13073)
• Keywords: large language model
• Abstract: While pioneering deep learning methods have made great strides in analyzing electronic health record (EHR) data, they often struggle to fully capture the semantics of diverse medical codes from limited data. The integration of external knowledge from Large Language Models (LLMs) presents a promising avenue for improving healthcare predictions. However, LLM analyses may exhibit significant variance due to ambiguity problems and inconsistency issues, hindering their effective utilization. To address these challenges, we propose IntelliCare, a novel framework that leverages LLMs to provide high-quality patient-level external knowledge and enhance existing EHR models. Concretely, IntelliCare identifies patient cohorts and employs task-relevant statistical information to augment LLM understanding and generation, effectively mitigating the ambiguity problem. Additionally, it refines LLM-derived knowledge through a hybrid approach, generating multiple analyses and calibrating them using both the EHR model and perplexity measures. Experimental evaluations on three clinical prediction tasks across two large-scale EHR datasets demonstrate that IntelliCare delivers significant performance improvements to existing methods, highlighting its potential in advancing personalized healthcare predictions and decision support systems.

Title: Multivariate Time-Series Anomaly Detection based on Enhancing Graph Attention Networks with Topological Analysis

• Authors: Zhe Liu, Xiang Huang, Jingyun Zhang, Zhifeng Hao, Li Sun, Hao Peng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13082
• Pdf URL: https://arxiv.org/pdf/2408.13082
• Copy Paste: [[2408.13082]] Multivariate Time-Series Anomaly Detection based on Enhancing Graph Attention Networks with Topological Analysis(https://arxiv.org/abs/2408.13082)
• Keywords: robust, extraction, transformer
• Abstract: Unsupervised anomaly detection in time series is essential in industrial applications, as it significantly reduces the need for manual intervention. Multivariate time series pose a complex challenge due to their feature and temporal dimensions. Traditional methods use Graph Neural Networks (GNNs) or Transformers to analyze spatial while RNNs to model temporal dependencies. These methods focus narrowly on one dimension or engage in coarse-grained feature extraction, which can be inadequate for large datasets characterized by intricate relationships and dynamic changes. This paper introduces a novel temporal model built on an enhanced Graph Attention Network (GAT) for multivariate time series anomaly detection called TopoGDN. Our model analyzes both time and feature dimensions from a fine-grained perspective. First, we introduce a multi-scale temporal convolution module to extract detailed temporal features. Additionally, we present an augmented GAT to manage complex inter-feature dependencies, which incorporates graph topology into node features across multiple scales, a versatile, plug-and-play enhancement that significantly boosts the performance of GAT. Our experimental results confirm that our approach surpasses the baseline models on four datasets, demonstrating its potential for widespread application in fields requiring robust anomaly detection. The code is available at this https URL.

Title: Map-Free Visual Relocalization Enhanced by Instance Knowledge and Depth Knowledge

• Authors: Mingyu Xiao, Runze Chen, Haiyong Luo, Fang Zhao, Juan Wang, Xuepeng Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13085
• Pdf URL: https://arxiv.org/pdf/2408.13085
• Copy Paste: [[2408.13085]] Map-Free Visual Relocalization Enhanced by Instance Knowledge and Depth Knowledge(https://arxiv.org/abs/2408.13085)
• Keywords: robust
• Abstract: Map-free relocalization technology is crucial for applications in autonomous navigation and augmented reality, but relying on pre-built maps is often impractical. It faces significant challenges due to limitations in matching methods and the inherent lack of scale in monocular images. These issues lead to substantial rotational and metric errors and even localization failures in real-world scenarios. Large matching errors significantly impact the overall relocalization process, affecting both rotational and translational accuracy. Due to the inherent limitations of the camera itself, recovering the metric scale from a single image is crucial, as this significantly impacts the translation error. To address these challenges, we propose a map-free relocalization method enhanced by instance knowledge and depth knowledge. By leveraging instance-based matching information to improve global matching results, our method significantly reduces the possibility of mismatching across different objects. The robustness of instance knowledge across the scene helps the feature point matching model focus on relevant regions and enhance matching accuracy. Additionally, we use estimated metric depth from a single image to reduce metric errors and improve scale recovery accuracy. By integrating methods dedicated to mitigating large translational and rotational errors, our approach demonstrates superior performance in map-free relocalization techniques.

Title: Analysis of child development facts and myths using text mining techniques and classification models

• Authors: Mehedi Tajrian, Azizur Rahman, Muhammad Ashad Kabir, Md Rafiqul Islam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13091
• Pdf URL: https://arxiv.org/pdf/2408.13091
• Copy Paste: [[2408.13091]] Analysis of child development facts and myths using text mining techniques and classification models(https://arxiv.org/abs/2408.13091)
• Keywords: robust, extraction
• Abstract: The rapid dissemination of misinformation on the internet complicates the decision-making process for individuals seeking reliable information, particularly parents researching child development topics. This misinformation can lead to adverse consequences, such as inappropriate treatment of children based on myths. While previous research has utilized text-mining techniques to predict child abuse cases, there has been a gap in the analysis of child development myths and facts. This study addresses this gap by applying text mining techniques and classification models to distinguish between myths and facts about child development, leveraging newly gathered data from publicly available websites. The research methodology involved several stages. First, text mining techniques were employed to pre-process the data, ensuring enhanced accuracy. Subsequently, the structured data was analysed using six robust Machine Learning (ML) classifiers and one Deep Learning (DL) model, with two feature extraction techniques applied to assess their performance across three different training-testing splits. To ensure the reliability of the results, cross-validation was performed using both k-fold and leave-one-out methods. Among the classification models tested, Logistic Regression (LR) demonstrated the highest accuracy, achieving a 90% accuracy with the Bag-of-Words (BoW) feature extraction technique. LR stands out for its exceptional speed and efficiency, maintaining low testing time per statement (0.97 microseconds). These findings suggest that LR, when combined with BoW, is effective in accurately classifying child development information, thus providing a valuable tool for combating misinformation and assisting parents in making informed decisions.

Title: Diffusion-based Episodes Augmentation for Offline Multi-Agent Reinforcement Learning

• Authors: Jihwan Oh, Sungnyun Kim, Gahee Kim, Sunghwan Kim, Se-Young Yun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13092
• Pdf URL: https://arxiv.org/pdf/2408.13092
• Copy Paste: [[2408.13092]] Diffusion-based Episodes Augmentation for Offline Multi-Agent Reinforcement Learning(https://arxiv.org/abs/2408.13092)
• Keywords: robust, diffusion
• Abstract: Offline multi-agent reinforcement learning (MARL) is increasingly recognized as crucial for effectively deploying RL algorithms in environments where real-time interaction is impractical, risky, or costly. In the offline setting, learning from a static dataset of past interactions allows for the development of robust and safe policies without the need for live data collection, which can be fraught with challenges. Building on this foundational importance, we present EAQ, Episodes Augmentation guided by Q-total loss, a novel approach for offline MARL framework utilizing diffusion models. EAQ integrates the Q-total function directly into the diffusion model as a guidance to maximize the global returns in an episode, eliminating the need for separate training. Our focus primarily lies on cooperative scenarios, where agents are required to act collectively towards achieving a shared goal-essentially, maximizing global returns. Consequently, we demonstrate that our episodes augmentation in a collaborative manner significantly boosts offline MARL algorithm compared to the original dataset, improving the normalized return by +17.3% and +12.9% for medium and poor behavioral policies in SMAC simulator, respectively.

Title: Functional Tensor Decompositions for Physics-Informed Neural Networks

• Authors: Sai Karthikeya Vemuri, Tim Büchner, Julia Niebling, Joachim Denzler
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13101
• Pdf URL: https://arxiv.org/pdf/2408.13101
• Copy Paste: [[2408.13101]] Functional Tensor Decompositions for Physics-Informed Neural Networks(https://arxiv.org/abs/2408.13101)
• Keywords: robust
• Abstract: Physics-Informed Neural Networks (PINNs) have shown continuous and increasing promise in approximating partial differential equations (PDEs), although they remain constrained by the curse of dimensionality. In this paper, we propose a generalized PINN version of the classical variable separable method. To do this, we first show that, using the universal approximation theorem, a multivariate function can be approximated by the outer product of neural networks, whose inputs are separated variables. We leverage tensor decomposition forms to separate the variables in a PINN setting. By employing Canonic Polyadic (CP), Tensor-Train (TT), and Tucker decomposition forms within the PINN framework, we create robust architectures for learning multivariate functions from separate neural networks connected by outer products. Our methodology significantly enhances the performance of PINNs, as evidenced by improved results on complex high-dimensional PDEs, including the 3d Helmholtz and 5d Poisson equations, among others. This research underscores the potential of tensor decomposition-based variably separated PINNs to surpass the state-of-the-art, offering a compelling solution to the dimensionality challenge in PDE approximation.

Title: Dynamic Label Adversarial Training for Deep Learning Robustness Against Adversarial Attacks

• Authors: Zhenyu Liu, Haoran Duan, Huizhi Liang, Yang Long, Vaclav Snasel, Guiseppe Nicosia, Rajiv Ranjan, Varun Ojha
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13102
• Pdf URL: https://arxiv.org/pdf/2408.13102
• Copy Paste: [[2408.13102]] Dynamic Label Adversarial Training for Deep Learning Robustness Against Adversarial Attacks(https://arxiv.org/abs/2408.13102)
• Keywords: defense, attack, robust
• Abstract: Adversarial training is one of the most effective methods for enhancing model robustness. Recent approaches incorporate adversarial distillation in adversarial training architectures. However, we notice two scenarios of defense methods that limit their performance: (1) Previous methods primarily use static ground truth for adversarial training, but this often causes robust overfitting; (2) The loss functions are either Mean Squared Error or KL-divergence leading to a sub-optimal performance on clean accuracy. To solve those problems, we propose a dynamic label adversarial training (DYNAT) algorithm that enables the target model to gradually and dynamically gain robustness from the guide model's decisions. Additionally, we found that a budgeted dimension of inner optimization for the target model may contribute to the trade-off between clean accuracy and robust accuracy. Therefore, we propose a novel inner optimization method to be incorporated into the adversarial training. This will enable the target model to adaptively search for adversarial examples based on dynamic labels from the guiding model, contributing to the robustness of the target model. Extensive experiments validate the superior performance of our approach.

Title: CathAction: A Benchmark for Endovascular Intervention Understanding

• Authors: Baoru Huang, Tuan Vo, Chayun Kongtongvattana, Giulio Dagnino, Dennis Kundrat, Wenqiang Chi, Mohamed Abdelaziz, Trevor Kwok, Tudor Jianu, Tuong Do, Hieu Le, Minh Nguyen, Hoan Nguyen, Erman Tjiputra, Quang Tran, Jianyang Xie, Yanda Meng, Binod Bhattarai, Zhaorui Tan, Hongbin Liu, Hong Seng Gan, Wei Wang, Xi Yang, Qiufeng Wang, Jionglong Su, Kaizhu Huang, Angelos Stefanidis, Min Guo, Bo Du, Rong Tao, Minh Vu, Guoyan Zheng, Yalin Zheng, Francisco Vasconcelos, Danail Stoyanov, Daniel Elson, Ferdinando Rodriguez y Baena, Anh Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13126
• Pdf URL: https://arxiv.org/pdf/2408.13126
• Copy Paste: [[2408.13126]] CathAction: A Benchmark for Endovascular Intervention Understanding(https://arxiv.org/abs/2408.13126)
• Keywords: segmentation
• Abstract: Real-time visual feedback from catheterization analysis is crucial for enhancing surgical safety and efficiency during endovascular interventions. However, existing datasets are often limited to specific tasks, small scale, and lack the comprehensive annotations necessary for broader endovascular intervention understanding. To tackle these limitations, we introduce CathAction, a large-scale dataset for catheterization understanding. Our CathAction dataset encompasses approximately 500,000 annotated frames for catheterization action understanding and collision detection, and 25,000 ground truth masks for catheter and guidewire segmentation. For each task, we benchmark recent related works in the field. We further discuss the challenges of endovascular intentions compared to traditional computer vision tasks and point out open research questions. We hope that CathAction will facilitate the development of endovascular intervention understanding methods that can be applied to real-world applications. The dataset is available at this https URL.

Title: DeTPP: Leveraging Object Detection for Robust Long-Horizon Event Prediction

• Authors: Ivan Karpukhin, Andrey Savchenko
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13131
• Pdf URL: https://arxiv.org/pdf/2408.13131
• Copy Paste: [[2408.13131]] DeTPP: Leveraging Object Detection for Robust Long-Horizon Event Prediction(https://arxiv.org/abs/2408.13131)
• Keywords: robust
• Abstract: Forecasting future events over extended periods, known as long-horizon prediction, is a fundamental task in various domains, including retail, finance, healthcare, and social networks. Traditional methods, such as Marked Temporal Point Processes (MTPP), typically use autoregressive models to predict multiple future events. However, these models frequently encounter issues such as converging to constant or repetitive outputs, which significantly limits their effectiveness and applicability. To overcome these limitations, we propose DeTPP (Detection-based Temporal Point Processes), a novel approach inspired by object detection methods from computer vision. DeTPP utilizes a novel matching-based loss function that selectively focuses on reliably predictable events, enhancing both training robustness and inference diversity. Our method sets a new state-of-the-art in long-horizon event prediction, significantly outperforming existing MTPP and next-K approaches. The implementation of DeTPP is publicly available on GitHub.

Title: Deep Learning at the Intersection: Certified Robustness as a Tool for 3D Vision

• Authors: Gabriel Pérez S, Juan C. Pérez, Motasem Alfarra, Jesús Zarzar, Sara Rojas, Bernard Ghanem, Pablo Arbeláez
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13135
• Pdf URL: https://arxiv.org/pdf/2408.13135
• Copy Paste: [[2408.13135]] Deep Learning at the Intersection: Certified Robustness as a Tool for 3D Vision(https://arxiv.org/abs/2408.13135)
• Keywords: robust
• Abstract: This paper presents preliminary work on a novel connection between certified robustness in machine learning and the modeling of 3D objects. We highlight an intriguing link between the Maximal Certified Radius (MCR) of a classifier representing a space's occupancy and the space's Signed Distance Function (SDF). Leveraging this relationship, we propose to use the certification method of randomized smoothing (RS) to compute SDFs. Since RS' high computational cost prevents its practical usage as a way to compute SDFs, we propose an algorithm to efficiently run RS in low-dimensional applications, such as 3D space, by expressing RS' fundamental operations as Gaussian smoothing on pre-computed voxel grids. Our approach offers an innovative and practical tool to compute SDFs, validated through proof-of-concept experiments in novel view synthesis. This paper bridges two previously disparate areas of machine learning, opening new avenues for further exploration and potential cross-domain advancements.

Title: Tamgram: A Frontend for Large-scale Protocol Modeling in Tamarin

• Authors: Di Long Li, Jim de Groot, Alwen Tiu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.13138
• Pdf URL: https://arxiv.org/pdf/2408.13138
• Copy Paste: [[2408.13138]] Tamgram: A Frontend for Large-scale Protocol Modeling in Tamarin(https://arxiv.org/abs/2408.13138)
• Keywords: security
• Abstract: Automated security protocol verifiers such as ProVerif and Tamarin have been increasingly applied to verify large scale complex real-world protocols. While their ability to automate difficult reasoning processes required to handle protocols at that scale is impressive, there remains a gap in the modeling languages used. In particular, providing support for writing and maintaining large protocol specifications. This work attempts to fill this gap by introducing a high-level protocol modeling language, called Tamgram, with a formal semantics that can be translated to the multiset rewriting semantics of Tamarin. Tamgram supports writing native Tamarin code directly, but also allows for easier structuring of large specifications through various high-level constructs, in particular those needed to manipulate states in protocols. We prove the soundness and the completeness of Tamgram with respect to the trace semantics of Tamarin, discuss different translation strategies, and identify an optimal strategy that yields performance comparable to manually coded Tamarin specifications. Finally we show the practicality of Tamgram with a set of small case studies and one large scale case study.

Title: Verification of Geometric Robustness of Neural Networks via Piecewise Linear Approximation and Lipschitz Optimisation

• Authors: Ben Batten, Yang Zheng, Alessandro De Palma, Panagiotis Kouvaros, Alessio Lomuscio
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13140
• Pdf URL: https://arxiv.org/pdf/2408.13140
• Copy Paste: [[2408.13140]] Verification of Geometric Robustness of Neural Networks via Piecewise Linear Approximation and Lipschitz Optimisation(https://arxiv.org/abs/2408.13140)
• Keywords: robust
• Abstract: We address the problem of verifying neural networks against geometric transformations of the input image, including rotation, scaling, shearing, and translation. The proposed method computes provably sound piecewise linear constraints for the pixel values by using sampling and linear approximations in combination with branch-and-bound Lipschitz optimisation. A feature of the method is that it obtains tighter over-approximations of the perturbation region than the present state-of-the-art. We report results from experiments on a comprehensive set of benchmarks. We show that our proposed implementation resolves more verification cases than present approaches while being more computationally efficient.

Title: Focus on Neighbors and Know the Whole: Towards Consistent Dense Multiview Text-to-Image Generator for 3D Creation

• Authors: Bonan Li, Zicheng Zhang, Xingyi Yang, Xinchao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13149
• Pdf URL: https://arxiv.org/pdf/2408.13149
• Copy Paste: [[2408.13149]] Focus on Neighbors and Know the Whole: Towards Consistent Dense Multiview Text-to-Image Generator for 3D Creation(https://arxiv.org/abs/2408.13149)
• Keywords: robust
• Abstract: Generating dense multiview images from text prompts is crucial for creating high-fidelity 3D assets. Nevertheless, existing methods struggle with space-view correspondences, resulting in sparse and low-quality outputs. In this paper, we introduce CoSER, a novel consistent dense Multiview Text-to-Image Generator for Text-to-3D, achieving both efficiency and quality by meticulously learning neighbor-view coherence and further alleviating ambiguity through the swift traversal of all views. For achieving neighbor-view consistency, each viewpoint densely interacts with adjacent viewpoints to perceive the global spatial structure, and aggregates information along motion paths explicitly defined by physical principles to refine details. To further enhance cross-view consistency and alleviate content drift, CoSER rapidly scan all views in spiral bidirectional manner to aware holistic information and then scores each point based on semantic material. Subsequently, we conduct weighted down-sampling along the spatial dimension based on scores, thereby facilitating prominent information fusion across all views with lightweight computation. Technically, the core module is built by integrating the attention mechanism with a selective state space model, exploiting the robust learning capabilities of the former and the low overhead of the latter. Extensive evaluation shows that CoSER is capable of producing dense, high-fidelity, content-consistent multiview images that can be flexibly integrated into various 3D generation models.

Title: Long-Term Pre-training for Temporal Action Detection with Transformers

• Authors: Jihwan Kim, Miso Lee, Jae-Pil Heo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13152
• Pdf URL: https://arxiv.org/pdf/2408.13152
• Copy Paste: [[2408.13152]] Long-Term Pre-training for Temporal Action Detection with Transformers(https://arxiv.org/abs/2408.13152)
• Keywords: transformer
• Abstract: Temporal action detection (TAD) is challenging, yet fundamental for real-world video applications. Recently, DETR-based models for TAD have been prevailing thanks to their unique benefits. However, transformers demand a huge dataset, and unfortunately data scarcity in TAD causes a severe degeneration. In this paper, we identify two crucial problems from data scarcity: attention collapse and imbalanced performance. To this end, we propose a new pre-training strategy, Long-Term Pre-training (LTP), tailored for transformers. LTP has two main components: 1) class-wise synthesis, 2) long-term pretext tasks. Firstly, we synthesize long-form video features by merging video snippets of a target class and non-target classes. They are analogous to untrimmed data used in TAD, despite being created from trimmed data. In addition, we devise two types of long-term pretext tasks to learn long-term dependency. They impose long-term conditions such as finding second-to-fourth or short-duration actions. Our extensive experiments show state-of-the-art performances in DETR-based methods on ActivityNet-v1.3 and THUMOS14 by a large margin. Moreover, we demonstrate that LTP significantly relieves the data scarcity issues in TAD.

Title: Interpretable breast cancer classification using CNNs on mammographic images

• Authors: Ann-Kristin Balve, Peter Hendrix
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13154
• Pdf URL: https://arxiv.org/pdf/2408.13154
• Copy Paste: [[2408.13154]] Interpretable breast cancer classification using CNNs on mammographic images(https://arxiv.org/abs/2408.13154)
• Keywords: interpretability
• Abstract: Deep learning models have achieved promising results in breast cancer classification, yet their 'black-box' nature raises interpretability concerns. This research addresses the crucial need to gain insights into the decision-making process of convolutional neural networks (CNNs) for mammogram classification, specifically focusing on the underlying reasons for the CNN's predictions of breast cancer. For CNNs trained on the Mammographic Image Analysis Society (MIAS) dataset, we compared the post-hoc interpretability techniques LIME, Grad-CAM, and Kernel SHAP in terms of explanatory depth and computational efficiency. The results of this analysis indicate that Grad-CAM, in particular, provides comprehensive insights into the behavior of the CNN, revealing distinctive patterns in normal, benign, and malignant breast tissue. We discuss the implications of the current findings for the use of machine learning models and interpretation techniques in clinical practice.

Title: Causal machine learning for sustainable agroecosystems

• Authors: Vasileios Sitokonstantinou, Emiliano Díaz Salas Porras, Jordi Cerdà Bautista, Maria Piles, Ioannis Athanasiadis, Hannah Kerner, Giulia Martini, Lily-belle Sweet, Ilias Tsoumas, Jakob Zscheischler, Gustau Camps-Valls
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.13155
• Pdf URL: https://arxiv.org/pdf/2408.13155
• Copy Paste: [[2408.13155]] Causal machine learning for sustainable agroecosystems(https://arxiv.org/abs/2408.13155)
• Keywords: security, robust
• Abstract: In a changing climate, sustainable agriculture is essential for food security and environmental health. However, it is challenging to understand the complex interactions among its biophysical, social, and economic components. Predictive machine learning (ML), with its capacity to learn from data, is leveraged in sustainable agriculture for applications like yield prediction and weather forecasting. Nevertheless, it cannot explain causal mechanisms and remains descriptive rather than prescriptive. To address this gap, we propose causal ML, which merges ML's data processing with causality's ability to reason about change. This facilitates quantifying intervention impacts for evidence-based decision-making and enhances predictive model robustness. We showcase causal ML through eight diverse applications that benefit stakeholders across the agri-food chain, including farmers, policymakers, and researchers.

Title: KonvLiNA: Integrating Kolmogorov-Arnold Network with Linear Nystr\"om Attention for feature fusion in Crop Field Detection

• Authors: Haruna Yunusa, Qin Shiyin, Adamu Lawan, Abdulrahman Hamman Adama Chukkol
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13160
• Pdf URL: https://arxiv.org/pdf/2408.13160
• Copy Paste: [[2408.13160]] KonvLiNA: Integrating Kolmogorov-Arnold Network with Linear Nystr\"om Attention for feature fusion in Crop Field Detection(https://arxiv.org/abs/2408.13160)
• Keywords: extraction
• Abstract: Crop field detection is a critical component of precision agriculture, essential for optimizing resource allocation and enhancing agricultural productivity. This study introduces KonvLiNA, a novel framework that integrates Convolutional Kolmogorov-Arnold Networks (cKAN) with Nyström attention mechanisms for effective crop field detection. Leveraging KAN adaptive activation functions and the efficiency of Nyström attention in handling largescale data, KonvLiNA significantly enhances feature extraction, enabling the model to capture intricate patterns in complex agricultural environments. Experimental results on rice crop dataset demonstrate KonvLiNA superiority over state-of-the-art methods, achieving a 0.415 AP and 0.459 AR with the Swin-L backbone, outperforming traditional YOLOv8 by significant margins. Additionally, evaluation on the COCO dataset showcases competitive performance across small, medium, and large objects, highlighting KonvLiNA efficacy in diverse agricultural settings. This work highlights the potential of hybrid KAN and attention mechanisms for advancing precision agriculture through improved crop field detection and management.

Title: Towards Weaknesses and Attack Patterns Prediction for IoT Devices

• Authors: Carlos A. Rivera A., Arash Shaghaghi, Gustavo Batista, Salil S. Kanhere
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.13172
• Pdf URL: https://arxiv.org/pdf/2408.13172
• Copy Paste: [[2408.13172]] Towards Weaknesses and Attack Patterns Prediction for IoT Devices(https://arxiv.org/abs/2408.13172)
• Keywords: security, attack
• Abstract: As the adoption of Internet of Things (IoT) devices continues to rise in enterprise environments, the need for effective and efficient security measures becomes increasingly critical. This paper presents a cost-efficient platform to facilitate the pre-deployment security checks of IoT devices by predicting potential weaknesses and associated attack patterns. The platform employs a Bidirectional Long Short-Term Memory (Bi-LSTM) network to analyse device-related textual data and predict weaknesses. At the same time, a Gradient Boosting Machine (GBM) model predicts likely attack patterns that could exploit these weaknesses. When evaluated on a dataset curated from the National Vulnerability Database (NVD) and publicly accessible IoT data sources, the system demonstrates high accuracy and reliability. The dataset created for this solution is publicly accessible.

Title: Can LLM be a Good Path Planner based on Prompt Engineering? Mitigating the Hallucination for Path Planning

• Authors: Hourui Deng, Hongjie Zhang, Jie Ou, Chaosheng Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13184
• Pdf URL: https://arxiv.org/pdf/2408.13184
• Copy Paste: [[2408.13184]] Can LLM be a Good Path Planner based on Prompt Engineering? Mitigating the Hallucination for Path Planning(https://arxiv.org/abs/2408.13184)
• Keywords: large language model
• Abstract: Spatial reasoning in Large Language Models (LLMs) is the foundation for embodied intelligence. However, even in simple maze environments, LLMs still encounter challenges in long-term path-planning, primarily influenced by their spatial hallucination and context inconsistency hallucination by long-term reasoning. To address this challenge, this study proposes an innovative model, Spatial-to-Relational Transformation and Curriculum Q-Learning (S2RCQL). To address the spatial hallucination of LLMs, we propose the Spatial-to-Relational approach, which transforms spatial prompts into entity relations and paths representing entity relation chains. This approach fully taps the potential of LLMs in terms of sequential thinking. As a result, we design a path-planning algorithm based on Q-learning to mitigate the context inconsistency hallucination, which enhances the reasoning ability of LLMs. Using the Q-value of state-action as auxiliary information for prompts, we correct the hallucinations of LLMs, thereby guiding LLMs to learn the optimal path. Finally, we propose a reverse curriculum learning technique based on LLMs to further mitigate the context inconsistency hallucination. LLMs can rapidly accumulate successful experiences by reducing task difficulty and leveraging them to tackle more complex tasks. We performed comprehensive experiments based on Baidu's self-developed LLM: ERNIE-Bot 4.0. The results showed that our S2RCQL achieved a 23%--40% improvement in both success and optimality rates compared with advanced prompt engineering.

Title: IFH: a Diffusion Framework for Flexible Design of Graph Generative Models

• Authors: Samuel Cognolato, Alessandro Sperduti, Luciano Serafini
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.13194
• Pdf URL: https://arxiv.org/pdf/2408.13194
• Copy Paste: [[2408.13194]] IFH: a Diffusion Framework for Flexible Design of Graph Generative Models(https://arxiv.org/abs/2408.13194)
• Keywords: diffusion, generative
• Abstract: Graph generative models can be classified into two prominent families: one-shot models, which generate a graph in one go, and sequential models, which generate a graph by successive additions of nodes and edges. Ideally, between these two extreme models lies a continuous range of models that adopt different levels of sequentiality. This paper proposes a graph generative model, called Insert-Fill-Halt (IFH), that supports the specification of a sequentiality degree. IFH is based upon the theory of Denoising Diffusion Probabilistic Models (DDPM), designing a node removal process that gradually destroys a graph. An insertion process learns to reverse this removal process by inserting arcs and nodes according to the specified sequentiality degree. We evaluate the performance of IFH in terms of quality, run time, and memory, depending on different sequentiality degrees. We also show that using DiGress, a diffusion-based one-shot model, as a generative step in IFH leads to improvement to the model itself, and is competitive with the current state-of-the-art.

Title: Instruct-DeBERTa: A Hybrid Approach for Aspect-based Sentiment Analysis on Textual Reviews

• Authors: Dineth Jayakody, A V A Malkith, Koshila Isuranda, Vishal Thenuwara, Nisansa de Silva, Sachintha Rajith Ponnamperuma, G G N Sandamali, K L K Sudheera
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13202
• Pdf URL: https://arxiv.org/pdf/2408.13202
• Copy Paste: [[2408.13202]] Instruct-DeBERTa: A Hybrid Approach for Aspect-based Sentiment Analysis on Textual Reviews(https://arxiv.org/abs/2408.13202)
• Keywords: robust, extraction, transformer
• Abstract: Aspect-based Sentiment Analysis (ABSA) is a critical task in Natural Language Processing (NLP) that focuses on extracting sentiments related to specific aspects within a text, offering deep insights into customer opinions. Traditional sentiment analysis methods, while useful for determining overall sentiment, often miss the implicit opinions about particular product or service features. This paper presents a comprehensive review of the evolution of ABSA methodologies, from lexicon-based approaches to machine learning and deep learning techniques. We emphasize the recent advancements in Transformer-based models, particularly Bidirectional Encoder Representations from Transformers (BERT) and its variants, which have set new benchmarks in ABSA tasks. We focused on finetuning Llama and Mistral models, building hybrid models using the SetFit framework, and developing our own model by exploiting the strengths of state-of-the-art (SOTA) Transformer-based models for aspect term extraction (ATE) and aspect sentiment classification (ASC). Our hybrid model Instruct - DeBERTa uses SOTA InstructABSA for aspect extraction and DeBERTa-V3-baseabsa-V1 for aspect sentiment classification. We utilize datasets from different domains to evaluate our model's performance. Our experiments indicate that the proposed hybrid model significantly improves the accuracy and reliability of sentiment analysis across all experimented domains. As per our findings, our hybrid model Instruct - DeBERTa is the best-performing model for the joint task of ATE and ASC for both SemEval restaurant 2014 and SemEval laptop 2014 datasets separately. By addressing the limitations of existing methodologies, our approach provides a robust solution for understanding detailed consumer feedback, thus offering valuable insights for businesses aiming to enhance customer satisfaction and product development.

Title: Protecting against simultaneous data poisoning attacks

• Authors: Neel Alex, Shoaib Ahmed Siddiqui, Amartya Sanyal, David Krueger
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13221
• Pdf URL: https://arxiv.org/pdf/2408.13221
• Copy Paste: [[2408.13221]] Protecting against simultaneous data poisoning attacks(https://arxiv.org/abs/2408.13221)
• Keywords: protect, defense, attack
• Abstract: Current backdoor defense methods are evaluated against a single attack at a time. This is unrealistic, as powerful machine learning systems are trained on large datasets scraped from the internet, which may be attacked multiple times by one or more attackers. We demonstrate that simultaneously executed data poisoning attacks can effectively install multiple backdoors in a single model without substantially degrading clean accuracy. Furthermore, we show that existing backdoor defense methods do not effectively prevent attacks in this setting. Finally, we leverage insights into the nature of backdoor attacks to develop a new defense, BaDLoss, that is effective in the multi-attack setting. With minimal clean accuracy degradation, BaDLoss attains an average attack success rate in the multi-attack setting of 7.98% in CIFAR-10 and 10.29% in GTSRB, compared to the average of other defenses at 64.48% and 84.28% respectively.

Title: D&M: Enriching E-commerce Videos with Sound Effects by Key Moment Detection and SFX Matching

• Authors: Jingyu Liu, Minquan Wang, Ye Ma, Bo Wang, Aozhu Chen, Quan Chen, Peng Jiang, Xirong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13226
• Pdf URL: https://arxiv.org/pdf/2408.13226
• Copy Paste: [[2408.13226]] D&M: Enriching E-commerce Videos with Sound Effects by Key Moment Detection and SFX Matching(https://arxiv.org/abs/2408.13226)
• Keywords: fair
• Abstract: Videos showcasing specific products are increasingly important for E-commerce. Key moments naturally exist as the first appearance of a specific product, presentation of its distinctive features, the presence of a buying link, etc. Adding proper sound effects (SFX) to these key moments, or video decoration with SFX (VDSFX), is crucial for enhancing the user engaging experience. Previous studies about adding SFX to videos perform video to SFX matching at a holistic level, lacking the ability of adding SFX to a specific moment. Meanwhile, previous studies on video highlight detection or video moment retrieval consider only moment localization, leaving moment to SFX matching untouched. By contrast, we propose in this paper D&M, a unified method that accomplishes key moment detection and moment to SFX matching simultaneously. Moreover, for the new VDSFX task we build a large-scale dataset SFX-Moment from an E-commerce platform. For a fair comparison, we build competitive baselines by extending a number of current video moment detection methods to the new task. Extensive experiments on SFX-Moment show the superior performance of the proposed method over the baselines. Code and data will be released.

Title: Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time

• Authors: Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, Yufa Zhou
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13233
• Pdf URL: https://arxiv.org/pdf/2408.13233
• Copy Paste: [[2408.13233]] Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time(https://arxiv.org/abs/2408.13233)
• Keywords: transformer, large language model
• Abstract: The quadratic computational complexity in the self-attention mechanism of popular transformer architectures poses significant challenges for training and inference, particularly in terms of efficiency and memory requirements. Towards addressing these challenges, this paper introduces a novel fast computation method for gradient calculation in multi-layer transformer models. Our approach enables the computation of gradients for the entire multi-layer transformer model in almost linear time $n^{1+o(1)}$, where $n$ is the input sequence length. This breakthrough significantly reduces the computational bottleneck associated with the traditional quadratic time complexity. Our theory holds for any loss function and maintains a bounded approximation error across the entire model. Furthermore, our analysis can hold when the multi-layer transformer model contains many practical sub-modules, such as residual connection, casual mask, and multi-head attention. By improving the efficiency of gradient computation in large language models, we hope that our work will facilitate the more effective training and deployment of long-context language models based on our theoretical results.

Title: CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities

• Authors: Tao Wu, Yong Zhang, Xintao Wang, Xianpan Zhou, Guangcong Zheng, Zhongang Qi, Ying Shan, Xi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13239
• Pdf URL: https://arxiv.org/pdf/2408.13239
• Copy Paste: [[2408.13239]] CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities(https://arxiv.org/abs/2408.13239)
• Keywords: diffusion
• Abstract: Customized video generation aims to generate high-quality videos guided by text prompts and subject's reference images. However, since it is only trained on static images, the fine-tuning process of subject learning disrupts abilities of video diffusion models (VDMs) to combine concepts and generate motions. To restore these abilities, some methods use additional video similar to the prompt to fine-tune or guide the model. This requires frequent changes of guiding videos and even re-tuning of the model when generating different motions, which is very inconvenient for users. In this paper, we propose CustomCrafter, a novel framework that preserves the model's motion generation and conceptual combination abilities without additional video and fine-tuning to recovery. For preserving conceptual combination ability, we design a plug-and-play module to update few parameters in VDMs, enhancing the model's ability to capture the appearance details and the ability of concept combinations for new subjects. For motion generation, we observed that VDMs tend to restore the motion of video in the early stage of denoising, while focusing on the recovery of subject details in the later stage. Therefore, we propose Dynamic Weighted Video Sampling Strategy. Using the pluggability of our subject learning modules, we reduce the impact of this module on motion generation in the early stage of denoising, preserving the ability to generate motion of VDMs. In the later stage of denoising, we restore this module to repair the appearance details of the specified subject, thereby ensuring the fidelity of the subject's appearance. Experimental results show that our method has a significant improvement compared to previous methods.

Title: MCTR: Multi Camera Tracking Transformer

• Authors: Alexandru Niculescu-Mizil, Deep Patel, Iain Melvin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13243
• Pdf URL: https://arxiv.org/pdf/2408.13243
• Copy Paste: [[2408.13243]] MCTR: Multi Camera Tracking Transformer(https://arxiv.org/abs/2408.13243)
• Keywords: transformer
• Abstract: Multi-camera tracking plays a pivotal role in various real-world applications. While end-to-end methods have gained significant interest in single-camera tracking, multi-camera tracking remains predominantly reliant on heuristic techniques. In response to this gap, this paper introduces Multi-Camera Tracking tRansformer (MCTR), a novel end-to-end approach tailored for multi-object detection and tracking across multiple cameras with overlapping fields of view. MCTR leverages end-to-end detectors like DEtector TRansformer (DETR) to produce detections and detection embeddings independently for each camera view. The framework maintains set of track embeddings that encaplusate global information about the tracked objects, and updates them at every frame by integrating the local information from the view-specific detection embeddings. The track embeddings are probabilistically associated with detections in every camera view and frame to generate consistent object tracks. The soft probabilistic association facilitates the design of differentiable losses that enable end-to-end training of the entire system. To validate our approach, we conduct experiments on MMPTrack and AI City Challenge, two recently introduced large-scale multi-camera multi-object tracking datasets.

Title: Data Exposure from LLM Apps: An In-depth Investigation of OpenAI's GPTs

• Authors: Evin Jaff, Yuhao Wu, Ning Zhang, Umar Iqbal
• Subjects: cs.CR, cs.AI, cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13247
• Pdf URL: https://arxiv.org/pdf/2408.13247
• Copy Paste: [[2408.13247]] Data Exposure from LLM Apps: An In-depth Investigation of OpenAI's GPTs(https://arxiv.org/abs/2408.13247)
• Keywords: privacy
• Abstract: LLM app ecosystems are quickly maturing and supporting a wide range of use cases, which requires them to collect excessive user data. Given that the LLM apps are developed by third-parties and that anecdotal evidence suggests LLM platforms currently do not strictly enforce their policies, user data shared with arbitrary third-parties poses a significant privacy risk. In this paper we aim to bring transparency in data practices of LLM apps. As a case study, we study OpenAI's GPT app ecosystem. We develop an LLM-based framework to conduct the static analysis of natural language-based source code of GPTs and their Actions (external services) to characterize their data collection practices. Our findings indicate that Actions collect expansive data about users, including sensitive information prohibited by OpenAI, such as passwords. We find that some Actions, including related to advertising and analytics, are embedded in multiple GPTs, which allow them to track user activities across GPTs. Additionally, co-occurrence of Actions exposes as much as 9.5x more data to them, than it is exposed to individual Actions. Lastly, we develop an LLM-based privacy policy analysis framework to automatically check the consistency of data collection by Actions with disclosures in their privacy policies. Our measurements indicate that the disclosures for most of the collected data types are omitted in privacy policies, with only 5.8% of Actions clearly disclosing their data collection practices.

Title: Foundational Model for Electron Micrograph Analysis: Instruction-Tuning Small-Scale Language-and-Vision Assistant for Enterprise Adoption

• Authors: Sakhinana Sagar Srinivas, Chidaksh Ravuru, Geethan Sannidhi, Venkataramana Runkana
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13248
• Pdf URL: https://arxiv.org/pdf/2408.13248
• Copy Paste: [[2408.13248]] Foundational Model for Electron Micrograph Analysis: Instruction-Tuning Small-Scale Language-and-Vision Assistant for Enterprise Adoption(https://arxiv.org/abs/2408.13248)
• Keywords: privacy
• Abstract: Semiconductor imaging and analysis are critical yet understudied in deep learning, limiting our ability for precise control and optimization in semiconductor manufacturing. We introduce a small-scale multimodal framework for analyzing semiconductor electron microscopy images (MAEMI) through vision-language instruction tuning. We generate a customized instruction-following dataset using large multimodal models on microscopic image analysis. We perform knowledge transfer from larger to smaller models through knowledge distillation, resulting in improved accuracy of smaller models on visual question answering (VQA) tasks. This approach eliminates the need for expensive, human expert-annotated datasets for microscopic image analysis tasks. Enterprises can further finetune MAEMI on their intellectual data, enhancing privacy and performance on low-cost consumer hardware. Our experiments show that MAEMI outperforms traditional methods, adapts to data distribution shifts, and supports high-throughput screening.

Title: Re-evaluation of Face Anti-spoofing Algorithm in Post COVID-19 Era Using Mask Based Occlusion Attack

• Authors: Vaibhav Sundharam, Abhijit Sarkar, A. Lynn Abbott
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13251
• Pdf URL: https://arxiv.org/pdf/2408.13251
• Copy Paste: [[2408.13251]] Re-evaluation of Face Anti-spoofing Algorithm in Post COVID-19 Era Using Mask Based Occlusion Attack(https://arxiv.org/abs/2408.13251)
• Keywords: attack, robust, biometric
• Abstract: Face anti-spoofing algorithms play a pivotal role in the robust deployment of face recognition systems against presentation attacks. Conventionally, full facial images are required by such systems to correctly authenticate individuals, but the widespread requirement of masks due to the current COVID-19 pandemic has introduced new challenges for these biometric authentication systems. Hence, in this work, we investigate the performance of presentation attack detection (PAD) algorithms under synthetic facial occlusions using masks and glasses. We have used five variants of masks to cover the lower part of the face with varying coverage areas (low-coverage, medium-coverage, high-coverage, round coverage), and 3D cues. We have also used different variants of glasses that cover the upper part of the face. We systematically tested the performance of four PAD algorithms under these occlusion attacks using a benchmark dataset. We have specifically looked at four different baseline PAD algorithms that focus on, texture, image quality, frame difference/motion, and abstract features through a convolutional neural network (CNN). Additionally we have introduced a new hybrid model that uses CNN and local binary pattern textures. Our experiment shows that adding the occlusions significantly degrades the performance of all of the PAD algorithms. Our results show the vulnerability of face anti-spoofing algorithms with occlusions, which could be in the usage of such algorithms in the post-pandemic era.

Title: LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation

• Authors: Shuai Yang, Jing Tan, Mengchen Zhang, Tong Wu, Yixuan Li, Gordon Wetzstein, Ziwei Liu, Dahua Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13252
• Pdf URL: https://arxiv.org/pdf/2408.13252
• Copy Paste: [[2408.13252]] LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation(https://arxiv.org/abs/2408.13252)
• Keywords: diffusion
• Abstract: 3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.

Title: Domain-specific long text classification from sparse relevant information

• Authors: Célia D'Cruz, Jean-Marc Bereder, Frédéric Precioso, Michel Riveill
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13253
• Pdf URL: https://arxiv.org/pdf/2408.13253
• Copy Paste: [[2408.13253]] Domain-specific long text classification from sparse relevant information(https://arxiv.org/abs/2408.13253)
• Keywords: large language model
• Abstract: Large Language Models have undoubtedly revolutionized the Natural Language Processing field, the current trend being to promote one-model-for-all tasks (sentiment analysis, translation, etc.). However, the statistical mechanisms at work in the larger language models struggle to exploit the relevant information when it is very sparse, when it is a weak signal. This is the case, for example, for the classification of long domain-specific documents, when the relevance relies on a single relevant word or on very few relevant words from technical jargon. In the medical domain, it is essential to determine whether a given report contains critical information about a patient's condition. This critical information is often based on one or few specific isolated terms. In this paper, we propose a hierarchical model which exploits a short list of potential target terms to retrieve candidate sentences and represent them into the contextualized embedding of the target term(s) they contain. A pooling of the term(s) embedding(s) entails the document representation to be classified. We evaluate our model on one public medical document benchmark in English and on one private French medical dataset. We show that our narrower hierarchical model is better than larger language models for retrieving relevant long documents in a domain-specific context.

Title: Ensemble Modeling of Multiple Physical Indicators to Dynamically Phenotype Autism Spectrum Disorder

• Authors: Marie Huynh (1), Aaron Kline (1), Saimourya Surabhi (1), Kaitlyn Dunlap (1), Onur Cezmi Mutlu (1), Mohammadmahdi Honarmand (1), Parnian Azizian (1), Peter Washington (2), Dennis P. Wall (1) ((1) Stanford University, (2) University of Hawaii at Manoa)
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13255
• Pdf URL: https://arxiv.org/pdf/2408.13255
• Copy Paste: [[2408.13255]] Ensemble Modeling of Multiple Physical Indicators to Dynamically Phenotype Autism Spectrum Disorder(https://arxiv.org/abs/2408.13255)
• Keywords: robust
• Abstract: Early detection of autism, a neurodevelopmental disorder marked by social communication challenges, is crucial for timely intervention. Recent advancements have utilized naturalistic home videos captured via the mobile application GuessWhat. Through interactive games played between children and their guardians, GuessWhat has amassed over 3,000 structured videos from 382 children, both diagnosed with and without Autism Spectrum Disorder (ASD). This collection provides a robust dataset for training computer vision models to detect ASD-related phenotypic markers, including variations in emotional expression, eye contact, and head movements. We have developed a protocol to curate high-quality videos from this dataset, forming a comprehensive training set. Utilizing this set, we trained individual LSTM-based models using eye gaze, head positions, and facial landmarks as input features, achieving test AUCs of 86%, 67%, and 78%, respectively. To boost diagnostic accuracy, we applied late fusion techniques to create ensemble models, improving the overall AUC to 90%. This approach also yielded more equitable results across different genders and age groups. Our methodology offers a significant step forward in the early detection of ASD by potentially reducing the reliance on subjective assessments and making early identification more accessibly and equitable.

Title: MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

• Authors: Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin, Tieniu Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.13257
• Pdf URL: https://arxiv.org/pdf/2408.13257
• Copy Paste: [[2408.13257]] MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?(https://arxiv.org/abs/2408.13257)
• Keywords: large language model
• Abstract: Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has recently garnered widespread attention in the research community. However, we observe that existing benchmarks present several common barriers that make it difficult to measure the significant challenges that models face in the real world, including: 1) small data scale leads to a large performance variance; 2) reliance on model-based annotations results in restricted data quality; 3) insufficient task difficulty, especially caused by the limited image resolution. To tackle these issues, we introduce MME-RealWorld. Specifically, we collect more than $300$K images from public datasets and the Internet, filtering $13,366$ high-quality images for annotation. This involves the efforts of professional $25$ annotators and $7$ experts in MLLMs, contributing to $29,429$ question-answer pairs that cover $43$ subtasks across $5$ real-world scenarios, extremely challenging even for humans. As far as we know, MME-RealWorld is the largest manually annotated benchmark to date, featuring the highest resolution and a targeted focus on real-world applications. We further conduct a thorough evaluation involving $28$ prominent MLLMs, such as GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the most advanced models struggle with our benchmarks, where none of them reach $60\%$ accuracy. The challenges of perceiving high-resolution images and understanding complex real-world scenarios remain urgent issues to be addressed. The data and evaluation code are released at this https URL .