2024-08-08

Title: Explainable AI-based Intrusion Detection System for Industry 5.0: An Overview of the Literature, associated Challenges, the existing Solutions, and Potential Research Directions

• Authors: Naseem Khan, Kashif Ahmad, Aref Al Tamimi, Mohammed M. Alani, Amine Bermak, Issa Khalil
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03335
• Pdf URL: https://arxiv.org/pdf/2408.03335
• Copy Paste: [[2408.03335]] Explainable AI-based Intrusion Detection System for Industry 5.0: An Overview of the Literature, associated Challenges, the existing Solutions, and Potential Research Directions(https://arxiv.org/abs/2408.03335)
• Keywords: security, defense, interpretability, explainability
• Abstract: Industry 5.0, which focuses on human and Artificial Intelligence (AI) collaboration for performing different tasks in manufacturing, involves a higher number of robots, Internet of Things (IoTs) devices and interconnections, Augmented/Virtual Reality (AR), and other smart devices. The huge involvement of these devices and interconnection in various critical areas, such as economy, health, education and defense systems, poses several types of potential security flaws. AI itself has been proven a very effective and powerful tool in different areas of cybersecurity, such as intrusion detection, malware detection, and phishing detection, among others. Just as in many application areas, cybersecurity professionals were reluctant to accept black-box ML solutions for cybersecurity applications. This reluctance pushed forward the adoption of eXplainable Artificial Intelligence (XAI) as a tool that helps explain how decisions are made in ML-based systems. In this survey, we present a comprehensive study of different XAI-based intrusion detection systems for industry 5.0, and we also examine the impact of explainability and interpretability on Cybersecurity practices through the lens of Adversarial XIDS (Adv-XIDS) approaches. Furthermore, we analyze the possible opportunities and challenges in XAI cybersecurity systems for industry 5.0 that elicit future research toward XAI-based solutions to be adopted by high-stakes industry 5.0 applications. We believe this rigorous analysis will establish a foundational framework for subsequent research endeavors within the specified domain.

Title: Adversarial Domain Adaptation for Cross-user Activity Recognition Using Diffusion-based Noise-centred Learning

• Authors: Xiaozhou Ye, Kevin I-Kai Wang
• Subjects: cs.LG, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.03353
• Pdf URL: https://arxiv.org/pdf/2408.03353
• Copy Paste: [[2408.03353]] Adversarial Domain Adaptation for Cross-user Activity Recognition Using Diffusion-based Noise-centred Learning(https://arxiv.org/abs/2408.03353)
• Keywords: robust, diffusion, generative
• Abstract: Human Activity Recognition (HAR) plays a crucial role in various applications such as human-computer interaction and healthcare monitoring. However, challenges persist in HAR models due to the data distribution differences between training and real-world data distributions, particularly evident in cross-user scenarios. This paper introduces a novel framework, termed Diffusion-based Noise-centered Adversarial Learning Domain Adaptation (Diff-Noise-Adv-DA), designed to address these challenges by leveraging generative diffusion modeling and adversarial learning techniques. Traditional HAR models often struggle with the diversity of user behaviors and sensor data distributions. Diff-Noise-Adv-DA innovatively integrates the inherent noise within diffusion models, harnessing its latent information to enhance domain adaptation. Specifically, the framework transforms noise into a critical carrier of activity and domain class information, facilitating robust classification across different user domains. Experimental evaluations demonstrate the effectiveness of Diff-Noise-Adv-DA in improving HAR model performance across different users, surpassing traditional domain adaptation methods. The framework not only mitigates distribution mismatches but also enhances data quality through noise-based denoising techniques.

Title: The Use of Large Language Models (LLM) for Cyber Threat Intelligence (CTI) in Cybercrime Forums

• Authors: Vanessa Clairoux-Trepanier, Isa-May Beauchamp, Estelle Ruellan, Masarah Paquet-Clouston, Serge-Olivier Paquette, Eric Clay
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03354
• Pdf URL: https://arxiv.org/pdf/2408.03354
• Copy Paste: [[2408.03354]] The Use of Large Language Models (LLM) for Cyber Threat Intelligence (CTI) in Cybercrime Forums(https://arxiv.org/abs/2408.03354)
• Keywords: large language model
• Abstract: Large language models (LLMs) can be used to analyze cyber threat intelligence (CTI) data from cybercrime forums, which contain extensive information and key discussions about emerging cyber threats. However, to date, the level of accuracy and efficiency of LLMs for such critical tasks has yet to be thoroughly evaluated. Hence, this study assesses the accuracy of an LLM system built on the OpenAI GPT-3.5-turbo model [7] to extract CTI information. To do so, a random sample of 500 daily conversations from three cybercrime forums, XSS, this http URL, and RAMP, was extracted, and the LLM system was instructed to summarize the conversations and code 10 key CTI variables, such as whether a large organization and/or a critical infrastructure is being targeted. Then, two coders reviewed each conversation and evaluated whether the information extracted by the LLM was accurate. The LLM system performed strikingly well, with an average accuracy score of 98%. Various ways to enhance the model were uncovered, such as the need to help the LLM distinguish between stories and past events, as well as being careful with verb tenses in prompts. Nevertheless, the results of this study highlight the efficiency and relevance of using LLMs for cyber threat intelligence.

Title: FastEdit: Fast Text-Guided Single-Image Editing via Semantic-Aware Diffusion Fine-Tuning

• Authors: Zhi Chen, Zecheng Zhao, Yadan Luo, Zi Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03355
• Pdf URL: https://arxiv.org/pdf/2408.03355
• Copy Paste: [[2408.03355]] FastEdit: Fast Text-Guided Single-Image Editing via Semantic-Aware Diffusion Fine-Tuning(https://arxiv.org/abs/2408.03355)
• Keywords: diffusion, generative
• Abstract: Conventional Text-guided single-image editing approaches require a two-step process, including fine-tuning the target text embedding for over 1K iterations and the generative model for another 1.5K iterations. Although it ensures that the resulting image closely aligns with both the input image and the target text, this process often requires 7 minutes per image, posing a challenge for practical application due to its time-intensive nature. To address this bottleneck, we introduce FastEdit, a fast text-guided single-image editing method with semantic-aware diffusion fine-tuning, dramatically accelerating the editing process to only 17 seconds. FastEdit streamlines the generative model's fine-tuning phase, reducing it from 1.5K to a mere 50 iterations. For diffusion fine-tuning, we adopt certain time step values based on the semantic discrepancy between the input image and target text. Furthermore, FastEdit circumvents the initial fine-tuning step by utilizing an image-to-image model that conditions on the feature space, rather than the text embedding space. It can effectively align the target text prompt and input image within the same feature space and save substantial processing time. Additionally, we apply the parameter-efficient fine-tuning technique LoRA to U-net. With LoRA, FastEdit minimizes the model's trainable parameters to only 0.37\% of the original size. At the same time, we can achieve comparable editing outcomes with significantly reduced computational overhead. We conduct extensive experiments to validate the editing performance of our approach and show promising editing capabilities, including content addition, style transfer, background replacement, and posture manipulation, etc.

Title: MLC-GCN: Multi-Level Generated Connectome Based GCN for AD Analysis

• Authors: Wenqi Zhu, Yinghua Fu, Ze Wang (for the Alzheimer's Disease Neuroimaging Initiative)
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03358
• Pdf URL: https://arxiv.org/pdf/2408.03358
• Copy Paste: [[2408.03358]] MLC-GCN: Multi-Level Generated Connectome Based GCN for AD Analysis(https://arxiv.org/abs/2408.03358)
• Keywords: extraction, explainability
• Abstract: Alzheimer's Disease (AD) is a currently incurable neurodegeneartive disease. Accurately detecting AD, especially in the early stage, represents a high research priority. AD is characterized by progressive cognitive impairments that are related to alterations in brain functional connectivity (FC). Based on this association, many studies have been published over the decades using FC and machine learning to differentiate AD from healthy aging. The most recent development in this detection method highlights the use of graph neural network (GNN) as the brain functionality analysis. In this paper, we proposed a stack of spatio-temporal feature extraction and graph generation based AD classification model using resting state fMRI. The proposed multi-level generated connectome (MLC) based graph convolutional network (GCN) (MLC-GCN) contains a multi-graph generation block and a GCN prediction block. The multi-graph generation block consists of a hierarchy of spatio-temporal feature extraction layers for extracting spatio-temporal rsfMRI features at different depths and building the corresponding connectomes. The GCN prediction block takes the learned multi-level connectomes to build and optimize GCNs at each level and concatenates the learned graphical features as the final predicting features for AD classification. Through independent cohort validations, MLC-GCN shows better performance for differentiating MCI, AD, and normal aging than state-of-art GCN and rsfMRI based AD classifiers. The proposed MLC-GCN also showed high explainability in terms of learning clinically reasonable connectome node and connectivity features from two independent datasets. While we only tested MLC-GCN on AD, the basic rsfMRI-based multi-level learned GCN based outcome prediction strategy is valid for other diseases or clinical outcomes.

Title: LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification

• Authors: Zhen Qin, Junru Wu, Jiaming Shen, Tianqi Liu, Xuanhui Wang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03359
• Pdf URL: https://arxiv.org/pdf/2408.03359
• Copy Paste: [[2408.03359]] LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification(https://arxiv.org/abs/2408.03359)
• Keywords: large language model
• Abstract: We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Unlike conventional methods, which concatenate all demonstration examples with the test instance and prompt LLMs to produce the pointwise prediction, our framework uses the LLM as a preference machine that makes a relative comparative decision between the test instance and each demonstration. A self-supervised method is then introduced to aggregate these binary comparisons into the final ordinal decision. LAMPO addresses several limitations inherent in previous methods, including context length constraints, ordering biases, and challenges associated with absolute point-wise estimation. Extensive experiments on seven public datasets demonstrate LAMPO's remarkably competitive performance across a diverse spectrum of applications (e.g., movie review analysis and hate speech detection). Notably, in certain applications, the improvement can be substantial, exceeding 20% in an absolute term. Moreover, we believe LAMPO represents an interesting addition to the non-parametric application layered on top of LLMs, as it supports black-box LLMs without necessitating the outputting of LLM's internal states (e.g., embeddings), as seen in previous approaches.

Title: Prioritize Alignment in Dataset Distillation

• Authors: Zekai Li, Ziyao Guo, Wangbo Zhao, Tianle Zhang, Zhi-Qi Cheng, Samir Khaki, Kaipeng Zhang, Ahmad Sajed, Konstantinos N Plataniotis, Kai Wang, Yang You
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03360
• Pdf URL: https://arxiv.org/pdf/2408.03360
• Copy Paste: [[2408.03360]] Prioritize Alignment in Dataset Distillation(https://arxiv.org/abs/2408.03360)
• Keywords: extraction
• Abstract: Dataset Distillation aims to compress a large dataset into a significantly more compact, synthetic one without compromising the performance of the trained models. To achieve this, existing methods use the agent model to extract information from the target dataset and embed it into the distilled dataset. Consequently, the quality of extracted and embedded information determines the quality of the distilled dataset. In this work, we find that existing methods introduce misaligned information in both information extraction and embedding stages. To alleviate this, we propose Prioritize Alignment in Dataset Distillation (PAD), which aligns information from the following two perspectives. 1) We prune the target dataset according to the compressing ratio to filter the information that can be extracted by the agent model. 2) We use only deep layers of the agent model to perform the distillation to avoid excessively introducing low-level information. This simple strategy effectively filters out misaligned information and brings non-trivial improvement for mainstream matching-based distillation algorithms. Furthermore, built on trajectory matching, \textbf{PAD} achieves remarkable improvements on various benchmarks, achieving state-of-the-art performance.

Title: A Non-negative VAE:the Generalized Gamma Belief Network

• Authors: Zhibin Duan, Tiansheng Wen, Muyao Wang, Bo Chen, Mingyuan Zhou
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03388
• Pdf URL: https://arxiv.org/pdf/2408.03388
• Copy Paste: [[2408.03388]] A Non-negative VAE:the Generalized Gamma Belief Network(https://arxiv.org/abs/2408.03388)
• Keywords: generative
• Abstract: The gamma belief network (GBN), often regarded as a deep topic model, has demonstrated its potential for uncovering multi-layer interpretable latent representations in text data. Its notable capability to acquire interpretable latent factors is partially attributed to sparse and non-negative gamma-distributed latent variables. However, the existing GBN and its variations are constrained by the linear generative model, thereby limiting their expressiveness and applicability. To address this limitation, we introduce the generalized gamma belief network (Generalized GBN) in this paper, which extends the original linear generative model to a more expressive non-linear generative model. Since the parameters of the Generalized GBN no longer possess an analytic conditional posterior, we further propose an upward-downward Weibull inference network to approximate the posterior distribution of the latent variables. The parameters of both the generative model and the inference network are jointly trained within the variational inference framework. Finally, we conduct comprehensive experiments on both expressivity and disentangled representation learning tasks to evaluate the performance of the Generalized GBN against state-of-the-art Gaussian variational autoencoders serving as baselines.

Title: RHiOTS: A Framework for Evaluating Hierarchical Time Series Forecasting Algorithms

• Authors: Luis Roque, Carlos Soares, Luís Torgo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03399
• Pdf URL: https://arxiv.org/pdf/2408.03399
• Copy Paste: [[2408.03399]] RHiOTS: A Framework for Evaluating Hierarchical Time Series Forecasting Algorithms(https://arxiv.org/abs/2408.03399)
• Keywords: robust
• Abstract: We introduce the Robustness of Hierarchically Organized Time Series (RHiOTS) framework, designed to assess the robustness of hierarchical time series forecasting models and algorithms on real-world datasets. Hierarchical time series, where lower-level forecasts must sum to upper-level ones, are prevalent in various contexts, such as retail sales across countries. Current empirical evaluations of forecasting methods are often limited to a small set of benchmark datasets, offering a narrow view of algorithm behavior. RHiOTS addresses this gap by systematically altering existing datasets and modifying the characteristics of individual series and their interrelations. It uses a set of parameterizable transformations to simulate those changes in the data distribution. Additionally, RHiOTS incorporates an innovative visualization component, turning complex, multidimensional robustness evaluation results into intuitive, easily interpretable visuals. This approach allows an in-depth analysis of algorithm and model behavior under diverse conditions. We illustrate the use of RHiOTS by analyzing the predictive performance of several algorithms. Our findings show that traditional statistical methods are more robust than state-of-the-art deep learning algorithms, except when the transformation effect is highly disruptive. Furthermore, we found no significant differences in the robustness of the algorithms when applying specific reconciliation methods, such as MinT. RHiOTS provides researchers with a comprehensive tool for understanding the nuanced behavior of forecasting algorithms, offering a more reliable basis for selecting the most appropriate method for a given problem.

Title: Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey

• Authors: Vu Tuan Truong, Luan Ba Dang, Long Bao Le
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03400
• Pdf URL: https://arxiv.org/pdf/2408.03400
• Copy Paste: [[2408.03400]] Attacks and Defenses for Generative Diffusion Models: A Comprehensive Survey(https://arxiv.org/abs/2408.03400)
• Keywords: security, defense, attack, membership infer, diffusion, generative
• Abstract: Diffusion models (DMs) have achieved state-of-the-art performance on various generative tasks such as image synthesis, text-to-image, and text-guided image-to-image generation. However, the more powerful the DMs, the more harmful they potentially are. Recent studies have shown that DMs are prone to a wide range of attacks, including adversarial attacks, membership inference, backdoor injection, and various multi-modal threats. Since numerous pre-trained DMs are published widely on the Internet, potential threats from these attacks are especially detrimental to the society, making DM-related security a worth investigating topic. Therefore, in this paper, we conduct a comprehensive survey on the security aspect of DMs, focusing on various attack and defense methods for DMs. First, we present crucial knowledge of DMs with five main types of DMs, including denoising diffusion probabilistic models, denoising diffusion implicit models, noise conditioned score networks, stochastic differential equations, and multi-modal conditional DMs. We further survey a variety of recent studies investigating different types of attacks that exploit the vulnerabilities of DMs. Then, we thoroughly review potential countermeasures to mitigate each of the presented threats. Finally, we discuss open challenges of DM-related security and envision certain research directions for this topic.

Title: ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning

• Authors: Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Thien Huu Nguyen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.03402
• Pdf URL: https://arxiv.org/pdf/2408.03402
• Copy Paste: [[2408.03402]] ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning(https://arxiv.org/abs/2408.03402)
• Keywords: generative, large language model
• Abstract: Large Language Models (LLMs) excel in various natural language processing tasks, but leveraging them for dense passage embedding remains challenging. This is due to their causal attention mechanism and the misalignment between their pre-training objectives and the text ranking tasks. Despite some recent efforts to address these issues, existing frameworks for LLM-based text embeddings have been limited by their support for only a limited range of LLM architectures and fine-tuning strategies, limiting their practical application and versatility. In this work, we introduce the Unified framework for Large Language Model Embedding (ULLME), a flexible, plug-and-play implementation that enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. We also propose Generation-augmented Representation Learning (GRL), a novel fine-tuning method to boost LLMs for text embedding tasks. GRL enforces consistency between representation-based and generation-based relevance scores, leveraging LLMs' powerful generative abilities for learning passage embeddings. To showcase our framework's flexibility and effectiveness, we release three pre-trained models from ULLME with different backbone architectures, ranging from 1.5B to 8B parameters, all of which demonstrate strong performance on the Massive Text Embedding Benchmark. Our framework is publicly available at: this https URL. A demo video for ULLME can also be found at this https URL.

Title: Set2Seq Transformer: Learning Permutation Aware Set Representations of Artistic Sequences

• Authors: Athanasios Efthymiou, Stevan Rudinac, Monika Kackovic, Nachoem Wijnberg, Marcel Worring
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03404
• Pdf URL: https://arxiv.org/pdf/2408.03404
• Copy Paste: [[2408.03404]] Set2Seq Transformer: Learning Permutation Aware Set Representations of Artistic Sequences(https://arxiv.org/abs/2408.03404)
• Keywords: transformer
• Abstract: We propose Set2Seq Transformer, a novel sequential multiple instance architecture, that learns to rank permutation aware set representations of sequences. First, we illustrate that learning temporal position-aware representations of discrete timesteps can greatly improve static visual multiple instance learning methods that do not regard temporality and concentrate almost exclusively on visual content analysis. We further demonstrate the significant advantages of end-to-end sequential multiple instance learning, integrating visual content and temporal information in a multimodal manner. As application we focus on fine art analysis related tasks. To that end, we show that our Set2Seq Transformer can leverage visual set and temporal position-aware representations for modelling visual artists' oeuvres for predicting artistic success. Finally, through extensive quantitative and qualitative evaluation using a novel dataset, WikiArt-Seq2Rank, and a visual learning-to-rank downstream task, we show that our Set2Seq Transformer captures essential temporal information improving the performance of strong static and sequential multiple instance learning methods for predicting artistic success.

Title: A TVD neural network closure and application to turbulent combustion

• Authors: Seung Won Suh, Jonathan F MacArt, Luke N Olson, Jonathan B Freund
• Subjects: cs.LG, cs.CE, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2408.03413
• Pdf URL: https://arxiv.org/pdf/2408.03413
• Copy Paste: [[2408.03413]] A TVD neural network closure and application to turbulent combustion(https://arxiv.org/abs/2408.03413)
• Keywords: diffusion
• Abstract: Trained neural networks (NN) have attractive features for closing governing equations, but in the absence of additional constraints, they can stray from physical reality. A NN formulation is introduced to preclude spurious oscillations that violate solution boundedness or positivity. It is embedded in the discretized equations as a machine learning closure and strictly constrained, inspired by total variation diminishing (TVD) methods for hyperbolic conservation laws. The constraint is exactly enforced during gradient-descent training by rescaling the NN parameters, which maps them onto an explicit feasible set. Demonstrations show that the constrained NN closure model usefully recovers linear and nonlinear hyperbolic phenomena and anti-diffusion while enforcing the non-oscillatory property. Finally, the model is applied to subgrid-scale (SGS) modeling of a turbulent reacting flow, for which it suppresses spurious oscillations in scalar fields that otherwise violate the solution boundedness. It outperforms a simple penalization of oscillations in the loss function.

Title: Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers

• Authors: Marcus Buckmann, Edward Hill
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.03414
• Pdf URL: https://arxiv.org/pdf/2408.03414
• Copy Paste: [[2408.03414]] Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers(https://arxiv.org/abs/2408.03414)
• Keywords: privacy, explainability, generative
• Abstract: For simple classification tasks, we show that users can benefit from the advantages of using small, local, generative language models instead of large commercial models without a trade-off in performance or introducing extra labelling costs. These advantages, including those around privacy, availability, cost, and explainability, are important both in commercial applications and in the broader democratisation of AI. Through experiments on 17 sentence classification tasks (2-4 classes), we show that penalised logistic regression on the embeddings from a small LLM equals (and usually betters) the performance of a large LLM in the "tens-of-shot" regime. This requires no more labelled instances than are needed to validate the performance of the large LLM. Finally, we extract stable and sensible explanations for classification decisions.

Title: Sequential Conditional Transport on Probabilistic Graphs for Interpretable Counterfactual Fairness

• Authors: Agathe Fernandes Machado, Arthur Charpentier, Ewen Gallic
• Subjects: cs.LG, stat.ME
• Abstract URL: https://arxiv.org/abs/2408.03425
• Pdf URL: https://arxiv.org/pdf/2408.03425
• Copy Paste: [[2408.03425]] Sequential Conditional Transport on Probabilistic Graphs for Interpretable Counterfactual Fairness(https://arxiv.org/abs/2408.03425)
• Keywords: fair
• Abstract: In this paper, we link two existing approaches to derive counterfactuals: adaptations based on a causal graph, as suggested in Plečko and Meinshausen (2020) and optimal transport, as in De Lara et al. (2024). We extend "Knothe's rearrangement" Bonnotte (2013) and "triangular transport" Zech and Marzouk (2022a) to probabilistic graphical models, and use this counterfactual approach, referred to as sequential transport, to discuss individual fairness. After establishing the theoretical foundations of the proposed method, we demonstrate its application through numerical experiments on both synthetic and real datasets.

Title: Dissecting the Infrastructure Used in Web-based Cryptojacking: A Measurement Perspective

• Authors: Ayodeji Adeniran, Kieran Human, David Mohaisen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.03426
• Pdf URL: https://arxiv.org/pdf/2408.03426
• Copy Paste: [[2408.03426]] Dissecting the Infrastructure Used in Web-based Cryptojacking: A Measurement Perspective(https://arxiv.org/abs/2408.03426)
• Keywords: attack
• Abstract: This paper conducts a comprehensive examination of the infrastructure supporting cryptojacking operations. The analysis elucidates the methodologies, frameworks, and technologies malicious entities employ to misuse computational resources for unauthorized cryptocurrency mining. The investigation focuses on identifying websites serving as platforms for cryptojacking activities. A dataset of 887 websites, previously identified as cryptojacking sites, was compiled and analyzed to categorize the attacks and malicious activities observed. The study further delves into the DNS IP addresses, registrars, and name servers associated with hosting these websites to understand their structure and components. Various malware and illicit activities linked to these sites were identified, indicating the presence of unauthorized cryptocurrency mining via compromised sites. The findings highlight the vulnerability of website infrastructures to cryptojacking.

Title: Hybrid diffusion models: combining supervised and generative pretraining for label-efficient fine-tuning of segmentation models

• Authors: Bruno Sauvalle, Mathieu Salzmann
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03433
• Pdf URL: https://arxiv.org/pdf/2408.03433
• Copy Paste: [[2408.03433]] Hybrid diffusion models: combining supervised and generative pretraining for label-efficient fine-tuning of segmentation models(https://arxiv.org/abs/2408.03433)
• Keywords: diffusion, generative, segmentation
• Abstract: We are considering in this paper the task of label-efficient fine-tuning of segmentation models: We assume that a large labeled dataset is available and allows to train an accurate segmentation model in one domain, and that we have to adapt this model on a related domain where only a few samples are available. We observe that this adaptation can be done using two distinct methods: The first method, supervised pretraining, is simply to take the model trained on the first domain using classical supervised learning, and fine-tune it on the second domain with the available labeled samples. The second method is to perform self-supervised pretraining on the first domain using a generic pretext task in order to get high-quality representations which can then be used to train a model on the second domain in a label-efficient way. We propose in this paper to fuse these two approaches by introducing a new pretext task, which is to perform simultaneously image denoising and mask prediction on the first domain. We motivate this choice by showing that in the same way that an image denoiser conditioned on the noise level can be considered as a generative model for the unlabeled image distribution using the theory of diffusion models, a model trained using this new pretext task can be considered as a generative model for the joint distribution of images and segmentation masks under the assumption that the mapping from images to segmentation masks is deterministic. We then empirically show on several datasets that fine-tuning a model pretrained using this approach leads to better results than fine-tuning a similar model trained using either supervised or unsupervised pretraining only.

Title: Simple Perturbations Subvert Ethereum Phishing Transactions Detection: An Empirical Analysis

• Authors: Ahod Alghureid, David Mohaisen
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03441
• Pdf URL: https://arxiv.org/pdf/2408.03441
• Copy Paste: [[2408.03441]] Simple Perturbations Subvert Ethereum Phishing Transactions Detection: An Empirical Analysis(https://arxiv.org/abs/2408.03441)
• Keywords: attack, robust
• Abstract: This paper explores the vulnerability of machine learning models, specifically Random Forest, Decision Tree, and K-Nearest Neighbors, to very simple single-feature adversarial attacks in the context of Ethereum fraudulent transaction detection. Through comprehensive experimentation, we investigate the impact of various adversarial attack strategies on model performance metrics, such as accuracy, precision, recall, and F1-score. Our findings, highlighting how prone those techniques are to simple attacks, are alarming, and the inconsistency in the attacks' effect on different algorithms promises ways for attack mitigation. We examine the effectiveness of different mitigation strategies, including adversarial training and enhanced feature selection, in enhancing model robustness.

Title: Probabilistic Surrogate Model for Accelerating the Design of Electric Vehicle Battery Enclosures for Crash Performance

• Authors: Shadab Anwar Shaikh, Harish Cherukuri, Kranthi Balusu, Ram Devanathan, Ayoub Soulami
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2408.03450
• Pdf URL: https://arxiv.org/pdf/2408.03450
• Copy Paste: [[2408.03450]] Probabilistic Surrogate Model for Accelerating the Design of Electric Vehicle Battery Enclosures for Crash Performance(https://arxiv.org/abs/2408.03450)
• Keywords: robust
• Abstract: This paper presents a probabilistic surrogate model for the accelerated design of electric vehicle battery enclosures with a focus on crash performance. The study integrates high-throughput finite element simulations and Gaussian Process Regression to develop a surrogate model that predicts crash parameters with high accuracy while providing uncertainty estimates. The model was trained using data generated from thermoforming and crash simulations over a range of material and process parameters. Validation against new simulation data demonstrated the model's predictive accuracy with mean absolute percentage errors within 8.08% for all output variables. Additionally, a Monte Carlo uncertainty propagation study revealed the impact of input variability on outputs. The results highlight the efficacy of the Gaussian Process Regression model in capturing complex relationships within the dataset, offering a robust and efficient tool for the design optimization of composite battery enclosures.

Title: On the Generalization of Preference Learning with DPO

• Authors: Shawn Im, Yixuan Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03459
• Pdf URL: https://arxiv.org/pdf/2408.03459
• Copy Paste: [[2408.03459]] On the Generalization of Preference Learning with DPO(https://arxiv.org/abs/2408.03459)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities but often struggle to align with human preferences, leading to harmful or undesirable outputs. Preference learning, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become a crucial component for ensuring that LLMs align with human values. Despite the widespread adoption in real-world systems, a thorough theoretical understanding of the generalization guarantees for these models remain lacking. This paper bridges that gap by introducing a new theoretical framework to analyze the generalization guarantees of models trained with direct preference optimization (DPO). While existing generalization theory often focuses on overparameterized models achieving near-optimal loss or models independent of the training process, our framework rigorously assesses how well models generalize after a finite number of gradient steps, reflecting real-world LLM training practices. By analyzing the reward margin associated with each sample and its trajectory throughout training, we can effectively bound the generalization error. We derive learning guarantees showing that, under specific conditions, models trained with DPO can correctly discern preferred responses on unseen data with high probability. These insights are empirically validated on contemporary LLMs, underscoring the practical relevance of our theoretical findings.

Title: AI Foundation Models in Remote Sensing: A Survey

• Authors: Siqi Lu, Junlin Guo, James R Zimmer-Dauphinee, Jordan M Nieusma, Xiao Wang, Parker VanValkenburgh, Steven A Wernke, Yuankai Huo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03464
• Pdf URL: https://arxiv.org/pdf/2408.03464
• Copy Paste: [[2408.03464]] AI Foundation Models in Remote Sensing: A Survey(https://arxiv.org/abs/2408.03464)
• Keywords: robust
• Abstract: Artificial Intelligence (AI) technologies have profoundly transformed the field of remote sensing, revolutionizing data collection, processing, and analysis. Traditionally reliant on manual interpretation and task-specific models, remote sensing has been significantly enhanced by the advent of foundation models--large-scale, pre-trained AI models capable of performing a wide array of tasks with unprecedented accuracy and efficiency. This paper provides a comprehensive survey of foundation models in the remote sensing domain, covering models released between June 2021 and June 2024. We categorize these models based on their applications in computer vision and domain-specific tasks, offering insights into their architectures, pre-training datasets, and methodologies. Through detailed performance comparisons, we highlight emerging trends and the significant advancements achieved by these foundation models. Additionally, we discuss the technical challenges, practical implications, and future research directions, addressing the need for high-quality data, computational resources, and improved model generalization. Our research also finds that pre-training methods, particularly self-supervised learning techniques like contrastive learning and masked autoencoders, significantly enhance the performance and robustness of foundation models in remote sensing tasks such as scene classification, object detection, and other applications. This survey aims to serve as a resource for researchers and practitioners by providing a panorama of advances and promising pathways for continued development and application of foundation models in remote sensing.

Title: Can LLMs Serve As Time Series Anomaly Detectors?

• Authors: Manqing Dong, Hao Huang, Longbing Cao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03475
• Pdf URL: https://arxiv.org/pdf/2408.03475
• Copy Paste: [[2408.03475]] Can LLMs Serve As Time Series Anomaly Detectors?(https://arxiv.org/abs/2408.03475)
• Keywords: large language model
• Abstract: An emerging topic in large language models (LLMs) is their application to time series forecasting, characterizing mainstream and patternable characteristics of time series. A relevant but rarely explored and more challenging question is whether LLMs can detect and explain time series anomalies, a critical task across various real-world applications. In this paper, we investigate the capabilities of LLMs, specifically GPT-4 and LLaMA3, in detecting and explaining anomalies in time series. Our studies reveal that: 1) LLMs cannot be directly used for time series anomaly detection. 2) By designing prompt strategies such as in-context learning and chain-of-thought prompting, GPT-4 can detect time series anomalies with results competitive to baseline methods. 3) We propose a synthesized dataset to automatically generate time series anomalies with corresponding explanations. By applying instruction fine-tuning on this dataset, LLaMA3 demonstrates improved performance in time series anomaly detection tasks. In summary, our exploration shows the promising potential of LLMs as time series anomaly detectors.

Title: Effect of Kernel Size on CNN-Vision-Transformer-Based Gaze Prediction Using Electroencephalography Data

• Authors: Chuhui Qiu, Bugao Liang, Matthew L Key
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03478
• Pdf URL: https://arxiv.org/pdf/2408.03478
• Copy Paste: [[2408.03478]] Effect of Kernel Size on CNN-Vision-Transformer-Based Gaze Prediction Using Electroencephalography Data(https://arxiv.org/abs/2408.03478)
• Keywords: transformer
• Abstract: In this paper, we present an algorithm of gaze prediction from Electroencephalography (EEG) data. EEG-based gaze prediction is a new research topic that can serve as an alternative to traditional video-based eye-tracking. Compared to the existing state-of-the-art (SOTA) method, we improved the root mean-squared-error of EEG-based gaze prediction to 53.06 millimeters, while reducing the training time to less than 33% of its original duration. Our source code can be found at this https URL

Title: Advancing EEG-Based Gaze Prediction Using Depthwise Separable Convolution and Enhanced Pre-Processing

• Authors: Matthew L Key, Tural Mehtiyev, Xiaodong Qu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03480
• Pdf URL: https://arxiv.org/pdf/2408.03480
• Copy Paste: [[2408.03480]] Advancing EEG-Based Gaze Prediction Using Depthwise Separable Convolution and Enhanced Pre-Processing(https://arxiv.org/abs/2408.03480)
• Keywords: transformer
• Abstract: In the field of EEG-based gaze prediction, the application of deep learning to interpret complex neural data poses significant challenges. This study evaluates the effectiveness of pre-processing techniques and the effect of additional depthwise separable convolution on EEG vision transformers (ViTs) in a pretrained model architecture. We introduce a novel method, the EEG Deeper Clustered Vision Transformer (EEG-DCViT), which combines depthwise separable convolutional neural networks (CNNs) with vision transformers, enriched by a pre-processing strategy involving data clustering. The new approach demonstrates superior performance, establishing a new benchmark with a Root Mean Square Error (RMSE) of 51.6 mm. This achievement underscores the impact of pre-processing and model refinement in enhancing EEG-based applications.

Title: Beyond App Markets: Demystifying Underground Mobile App Distribution Via Telegram

• Authors: Yanhui Guo, Dong Wang, Liu Wang, Yongsheng Fang, Chao Wang, Minghui Yang, Tianming Liu, Haoyu Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.03482
• Pdf URL: https://arxiv.org/pdf/2408.03482
• Copy Paste: [[2408.03482]] Beyond App Markets: Demystifying Underground Mobile App Distribution Via Telegram(https://arxiv.org/abs/2408.03482)
• Keywords: security, protect, robust
• Abstract: The thriving mobile app ecosystem encompasses a wide range of functionalities. However, within this ecosystem, a subset of apps provides illicit services such as gambling and pornography to pursue economic gains, collectively referred to as "underground economy apps". While previous studies have examined these apps' characteristics and identification methods, investigations into their distribution via platforms beyond app markets (like Telegram) remain scarce, which has emerged as a crucial channel for underground activities and cybercrime due to the robust encryption and user anonymity. This study provides the first comprehensive exploration of the underground mobile app ecosystem on Telegram. Overcoming the complexities of the Telegram environment, we build a novel dataset and analyze the prevalence, promotional strategies, and characteristics of these apps. Our findings reveal that these apps reach approximately 1% of Telegram's user base, primarily catering to gambling and pornography services. We uncover sophisticated promotional strategies involving complex networks of apps, websites, users, and channels, and identify significant gaps in Telegram's content moderation capabilities. Our analysis also exposes the misuse of iOS features for app distribution and the prevalence of malicious behaviors in these apps. This research not only enhances our understanding of the underground app ecosystem but also provides valuable insights for developing effective regulatory measures and protecting users from potential risks associated with these covert operations. Our findings provide implications for platform regulators, app market operators, law enforcement agencies, and cybersecurity professionals in combating the proliferation of underground apps on encrypted messaging platforms.

Title: Simultaneous and Meshfree Topology Optimization with Physics-informed Gaussian Processes

• Authors: Amin Yousefpour, Shirin Hosseinmardi, Carlos Mora, Ramin Bostanabad
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03490
• Pdf URL: https://arxiv.org/pdf/2408.03490
• Copy Paste: [[2408.03490]] Simultaneous and Meshfree Topology Optimization with Physics-informed Gaussian Processes(https://arxiv.org/abs/2408.03490)
• Keywords: robust
• Abstract: Topology optimization (TO) provides a principled mathematical approach for optimizing the performance of a structure by designing its material spatial distribution in a pre-defined domain and subject to a set of constraints. The majority of existing TO approaches leverage numerical solvers for design evaluations during the optimization and hence have a nested nature and rely on discretizing the design variables. Contrary to these approaches, herein we develop a new class of TO methods based on the framework of Gaussian processes (GPs) whose mean functions are parameterized via deep neural networks. Specifically, we place GP priors on all design and state variables to represent them via parameterized continuous functions. These GPs share a deep neural network as their mean function but have as many independent kernels as there are state and design variables. We estimate all the parameters of our model in a single for loop that optimizes a penalized version of the performance metric where the penalty terms correspond to the state equations and design constraints. Attractive features of our approach include $(1)$ having a built-in continuation nature since the performance metric is optimized at the same time that the state equations are solved, and $(2)$ being discretization-invariant and accommodating complex domains and topologies. To test our method against conventional TO approaches implemented in commercial software, we evaluate it on four problems involving the minimization of dissipated power in Stokes flow. The results indicate that our approach does not need filtering techniques, has consistent computational costs, and is highly robust against random initializations and problem setup.

Title: Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation

• Authors: Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, Minlan Yu
• Subjects: cs.CL, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.03505
• Pdf URL: https://arxiv.org/pdf/2408.03505
• Copy Paste: [[2408.03505]] Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation(https://arxiv.org/abs/2408.03505)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) have extended the success of large language models (LLMs) to multiple data types, such as image, text and audio, achieving significant performance in various domains, including multimodal translation, visual question answering and content generation. Nonetheless, existing systems are inefficient to train MLLMs due to substantial GPU bubbles caused by the heterogeneous modality models and complex data dependencies in 3D parallelism. This paper proposes Optimus, a distributed MLLM training system that reduces end-to-end MLLM training time. Optimus is based on our principled analysis that scheduling the encoder computation within the LLM bubbles can reduce bubbles in MLLM training. To make scheduling encoder computation possible for all GPUs, Optimus searches the separate parallel plans for encoder and LLM, and adopts a bubble scheduling algorithm to enable exploiting LLM bubbles without breaking the original data dependencies in the MLLM model architecture. We further decompose encoder layer computation into a series of kernels, and analyze the common bubble pattern of 3D parallelism to carefully optimize the sub-millisecond bubble scheduling, minimizing the overall training time. Our experiments in a production cluster show that Optimus accelerates MLLM training by 20.5%-21.3% with ViT-22B and GPT-175B model over 3072 GPUs compared to baselines.

Title: MoExtend: Tuning New Experts for Modality and Task Extension

• Authors: Shanshan Zhong, Shanghua Gao, Zhongzhan Huang, Wushao Wen, Marinka Zitnik, Pan Zhou
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03511
• Pdf URL: https://arxiv.org/pdf/2408.03511
• Copy Paste: [[2408.03511]] MoExtend: Tuning New Experts for Modality and Task Extension(https://arxiv.org/abs/2408.03511)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel in various tasks but are primarily trained on text data, limiting their application scope. Expanding LLM capabilities to include vision-language understanding is vital, yet training them on multimodal data from scratch is challenging and costly. Existing instruction tuning methods, e.g., LLAVA, often connects a pretrained CLIP vision encoder and LLMs via fully fine-tuning LLMs to bridge the modality gap. However, full fine-tuning is plagued by catastrophic forgetting, i.e., forgetting previous knowledge, and high training costs particularly in the era of increasing tasks and modalities. To solve this issue, we introduce MoExtend, an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. MoExtend seamlessly integrates new experts into pre-trained MoE models, endowing them with novel knowledge without the need to tune pretrained models such as MoE and vision encoders. This approach enables rapid adaptation and extension to new modal data or tasks, effectively addressing the challenge of accommodating new modalities within LLMs. Furthermore, MoExtend avoids tuning pretrained models, thus mitigating the risk of catastrophic forgetting. Experimental results demonstrate the efficacy and efficiency of MoExtend in enhancing the multimodal capabilities of LLMs, contributing to advancements in multimodal AI research. Code: this https URL.

Title: Leveraging LLMs for Enhanced Open-Vocabulary 3D Scene Understanding in Autonomous Driving

• Authors: Amirhosein Chahe, Lifeng Zhou
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.03516
• Pdf URL: https://arxiv.org/pdf/2408.03516
• Copy Paste: [[2408.03516]] Leveraging LLMs for Enhanced Open-Vocabulary 3D Scene Understanding in Autonomous Driving(https://arxiv.org/abs/2408.03516)
• Keywords: large language model, segmentation
• Abstract: This paper introduces a novel method for open-vocabulary 3D scene understanding in autonomous driving by combining Language Embedded 3D Gaussians with Large Language Models (LLMs) for enhanced inference. We propose utilizing LLMs to generate contextually relevant canonical phrases for segmentation and scene interpretation. Our method leverages the contextual and semantic capabilities of LLMs to produce a set of canonical phrases, which are then compared with the language features embedded in the 3D Gaussians. This LLM-guided approach significantly improves zero-shot scene understanding and detection of objects of interest, even in the most challenging or unfamiliar environments. Experimental results on the WayveScenes101 dataset demonstrate that our approach surpasses state-of-the-art methods in terms of accuracy and flexibility for open-vocabulary object detection and segmentation. This work represents a significant advancement towards more intelligent, context-aware autonomous driving systems, effectively bridging 3D scene representation with high-level semantic understanding.

Title: SwinShadow: Shifted Window for Ambiguous Adjacent Shadow Detection

• Authors: Yonghui Wang, Shaokai Liu, Li Li, Wengang Zhou, Houqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03521
• Pdf URL: https://arxiv.org/pdf/2408.03521
• Copy Paste: [[2408.03521]] SwinShadow: Shifted Window for Ambiguous Adjacent Shadow Detection(https://arxiv.org/abs/2408.03521)
• Keywords: transformer
• Abstract: Shadow detection is a fundamental and challenging task in many computer vision applications. Intuitively, most shadows come from the occlusion of light by the object itself, resulting in the object and its shadow being contiguous (referred to as the adjacent shadow in this paper). In this case, when the color of the object is similar to that of the shadow, existing methods struggle to achieve accurate detection. To address this problem, we present SwinShadow, a transformer-based architecture that fully utilizes the powerful shifted window mechanism for detecting adjacent shadows. The mechanism operates in two steps. Initially, it applies local self-attention within a single window, enabling the network to focus on local details. Subsequently, it shifts the attention windows to facilitate inter-window attention, enabling the capture of a broader range of adjacent information. These combined steps significantly improve the network's capacity to distinguish shadows from nearby objects. And the whole process can be divided into three parts: encoder, decoder, and feature integration. During encoding, we adopt Swin Transformer to acquire hierarchical features. Then during decoding, for shallow layers, we propose a deep supervision (DS) module to suppress the false positives and boost the representation capability of shadow features for subsequent processing, while for deep layers, we leverage a double attention (DA) module to integrate local and shifted window in one stage to achieve a larger receptive field and enhance the continuity of information. Ultimately, a new multi-level aggregation (MLA) mechanism is applied to fuse the decoded features for mask prediction. Extensive experiments on three shadow detection benchmark datasets, SBU, UCF, and ISTD, demonstrate that our network achieves good performance in terms of balance error rate (BER).

Title: EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora

• Authors: Faisal Qarah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03524
• Pdf URL: https://arxiv.org/pdf/2408.03524
• Copy Paste: [[2408.03524]] EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora(https://arxiv.org/abs/2408.03524)
• Keywords: large language model
• Abstract: This study presents EgyBERT, an Arabic language model pretrained on 10.4 GB of Egyptian dialectal texts. We evaluated EgyBERT's performance by comparing it with five other multidialect Arabic language models across 10 evaluation datasets. EgyBERT achieved the highest average F1-score of 84.25% and an accuracy of 87.33%, significantly outperforming all other comparative models, with MARBERTv2 as the second best model achieving an F1-score 83.68% and an accuracy 87.19%. Additionally, we introduce two novel Egyptian dialectal corpora: the Egyptian Tweets Corpus (ETC), containing over 34.33 million tweets (24.89 million sentences) amounting to 2.5 GB of text, and the Egyptian Forums Corpus (EFC), comprising over 44.42 million sentences (7.9 GB of text) collected from various Egyptian online forums. Both corpora are used in pretraining the new model, and they are the largest Egyptian dialectal corpora to date reported in the literature. Furthermore, this is the first study to evaluate the performance of various language models on Egyptian dialect datasets, revealing significant differences in performance that highlight the need for more dialect-specific models. The results confirm the effectiveness of EgyBERT model in processing and analyzing Arabic text expressed in Egyptian dialect, surpassing other language models included in the study. EgyBERT model is publicly available on \url{this https URL}.

Title: PoseMamba: Monocular 3D Human Pose Estimation with Bidirectional Global-Local Spatio-Temporal State Space Model

• Authors: Yunlong Huang, Junshuo Liu, Ke Xian, Robert Caiming Qiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03540
• Pdf URL: https://arxiv.org/pdf/2408.03540
• Copy Paste: [[2408.03540]] PoseMamba: Monocular 3D Human Pose Estimation with Bidirectional Global-Local Spatio-Temporal State Space Model(https://arxiv.org/abs/2408.03540)
• Keywords: transformer
• Abstract: Transformers have significantly advanced the field of 3D human pose estimation (HPE). However, existing transformer-based methods primarily use self-attention mechanisms for spatio-temporal modeling, leading to a quadratic complexity, unidirectional modeling of spatio-temporal relationships, and insufficient learning of spatial-temporal correlations. Recently, the Mamba architecture, utilizing the state space model (SSM), has exhibited superior long-range modeling capabilities in a variety of vision tasks with linear complexity. In this paper, we propose PoseMamba, a novel purely SSM-based approach with linear complexity for 3D human pose estimation in monocular video. Specifically, we propose a bidirectional global-local spatio-temporal SSM block that comprehensively models human joint relations within individual frames as well as temporal correlations across frames. Within this bidirectional global-local spatio-temporal SSM block, we introduce a reordering strategy to enhance the local modeling capability of the SSM. This strategy provides a more logical geometric scanning order and integrates it with the global SSM, resulting in a combined global-local spatial scan. We have quantitatively and qualitatively evaluated our approach using two benchmark datasets: Human3.6M and MPI-INF-3DHP. Extensive experiments demonstrate that PoseMamba achieves state-of-the-art performance on both datasets while maintaining a smaller model size and reducing computational costs. The code and models will be released.

Title: EXAONE 3.0 7.8B Instruction Tuned Language Model

• Authors: LG AI Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Stanley Jungkyu Choi, Yemuk Choi, Seokhee Hong, Yeonjung Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Yountae Jung, Euisoon Kim, Hyosang Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Youchul Kim, Edward Hwayoung Lee, Haeju Lee, Honglak Lee, Jinsik Lee, Kyungmin Lee, Moontae Lee, Seungjun Lee, Woohyung Lim, Sangha Park, Sooyoun Park, Yongmin Park, Boseong Seo, Sihoon Yang, Heuiyeen Yeen, Kyungjae Yoo, Hyeongu Yun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03541
• Pdf URL: https://arxiv.org/pdf/2408.03541
• Copy Paste: [[2408.03541]] EXAONE 3.0 7.8B Instruction Tuned Language Model(https://arxiv.org/abs/2408.03541)
• Keywords: large language model
• Abstract: We introduce EXAONE 3.0 instruction-tuned language model, the first open model in the family of Large Language Models (LLMs) developed by LG AI Research. Among different model sizes, we publicly release the 7.8B instruction-tuned model to promote open research and innovations. Through extensive evaluations across a wide range of public and in-house benchmarks, EXAONE 3.0 demonstrates highly competitive real-world performance with instruction-following capability against other state-of-the-art open models of similar size. Our comparative analysis shows that EXAONE 3.0 excels particularly in Korean, while achieving compelling performance across general tasks and complex reasoning. With its strong real-world effectiveness and bilingual proficiency, we hope that EXAONE keeps contributing to advancements in Expert AI. Our EXAONE 3.0 instruction-tuned model is available at this https URL

Title: Automatic identification of the area covered by acorn trees in the dehesa (pastureland) Extremadura of Spain

• Authors: Ojeda-Magaña Benjamin, Ruelas Ruben, Quintanilla-Dominguez Joel, Gomez-Barba Leopoldo, Lopez de Herrera Juan, Robledo-Hernandez Jose, Tarquis Ana
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03542
• Pdf URL: https://arxiv.org/pdf/2408.03542
• Copy Paste: [[2408.03542]] Automatic identification of the area covered by acorn trees in the dehesa (pastureland) Extremadura of Spain(https://arxiv.org/abs/2408.03542)
• Keywords: segmentation
• Abstract: The acorn is the fruit of the oak and is an important crop in the Spanish dehesa extremeña, especially for the value it provides in the Iberian pig food to obtain the "acorn" certification. For this reason, we want to maximise the production of Iberian pigs with the appropriate weight. Hence the need to know the area covered by the crowns of the acorn trees, to determine the covered wooded area (CWA, from the Spanish Superficie Arbolada Cubierta SAC) and thereby estimate the number of Iberian pigs that can be released per hectare, as indicated by the royal decree 4/2014. In this work, we propose the automatic estimation of the CWA, through aerial digital images (orthophotos) of the pastureland of Extremadura, and with this, to offer the possibility of determining the number of Iberian pigs to be released in a specific plot of land. Among the main issues for automatic detection are, first, the correct identification of acorn trees, secondly, correctly discriminating the shades of the acorn trees and, finally, detect the arbuscles (young acorn trees not yet productive, or shrubs that are not oaks). These difficulties represent a real challenge, both for the automatic segmentation process and for manual segmentation. In this work, the proposed method for automatic segmentation is based on the clustering algorithm proposed by Gustafson-Kessel (GK) but the modified version of Babuska (GK-B) and on the use of real orthophotos. The obtained results are promising both in their comparison with the real images and when compared with the images segmented by hand. The whole set of orthophotos used in this work correspond to an approximate area of 142 hectares, and the results are of great interest to producers of certified "acorn" pork.

Title: Unlocking the Non-Native Language Context Limitation: Native Language Prompting Facilitates Knowledge Elicitation

• Authors: Baixuan Li, Yunlong Fan, Zhiqiang Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03544
• Pdf URL: https://arxiv.org/pdf/2408.03544
• Copy Paste: [[2408.03544]] Unlocking the Non-Native Language Context Limitation: Native Language Prompting Facilitates Knowledge Elicitation(https://arxiv.org/abs/2408.03544)
• Keywords: large language model
• Abstract: Multilingual large language models (MLLMs) struggle to answer questions posed in non-dominant languages, even though they have already acquired the relevant knowledge from their dominant language corpus. In contrast, human multilinguals can overcome this issue by invoking the relatively rich knowledge acquired from native language texts through Positive Native Language Transfer (PNLT). Inspired by this, we analogize the dominant language of MLLMs to the native language of human multilinguals, and propose Native Language Prompting (NatLan) to simulate the PNLT observed in human multilinguals. It explicitly creates native language contexts for MLLMs to facilitate the elicitation of the rich native language knowledge during question-answering, unlocking the limitations imposed by non-native language contexts on the effective application of knowledge. By employing multi-MLLM collaboration, NatLan reduces the workload on each MLLM in simulating PNLT and refines semantic transfer. On the C-Eval benchmark, NatLan provides up to a 10.1% average accuracy improvement and up to a 5.0% increase in the hard-level subset across five MLLMs, surpassing all top-notch related methods. Our code is available at this https URL.

Title: VPOcc: Exploiting Vanishing Point for Monocular 3D Semantic Occupancy Prediction

• Authors: Junsu Kim, Junhee Lee, Ukcheol Shin, Jean Oh, Kyungdon Joo
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.03551
• Pdf URL: https://arxiv.org/pdf/2408.03551
• Copy Paste: [[2408.03551]] VPOcc: Exploiting Vanishing Point for Monocular 3D Semantic Occupancy Prediction(https://arxiv.org/abs/2408.03551)
• Keywords: extraction
• Abstract: Monocular 3D semantic occupancy prediction is becoming important in robot vision due to the compactness of using a single RGB camera. However, existing methods often do not adequately account for camera perspective geometry, resulting in information imbalance along the depth range of the image. To address this issue, we propose a vanishing point (VP) guided monocular 3D semantic occupancy prediction framework named VPOcc. Our framework consists of three novel modules utilizing VP. First, in the VPZoomer module, we initially utilize VP in feature extraction to achieve information balanced feature extraction across the scene by generating a zoom-in image based on VP. Second, we perform perspective geometry-aware feature aggregation by sampling points towards VP using a VP-guided cross-attention (VPCA) module. Finally, we create an information-balanced feature volume by effectively fusing original and zoom-in voxel feature volumes with a balanced feature volume fusion (BVFV) module. Experiments demonstrate that our method achieves state-of-the-art performance for both IoU and mIoU on SemanticKITTI and SSCBench-KITTI360. These results are obtained by effectively addressing the information imbalance in images through the utilization of VP. Our code will be available at this http URL.

Title: Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection

• Authors: Subaru Kimura, Ryota Tanaka, Shumpei Miyawaki, Jun Suzuki, Keisuke Sakaguchi
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03554
• Pdf URL: https://arxiv.org/pdf/2408.03554
• Copy Paste: [[2408.03554]] Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection(https://arxiv.org/abs/2408.03554)
• Keywords: security, attack
• Abstract: We explore visual prompt injection (VPI) that maliciously exploits the ability of large vision-language models (LVLMs) to follow instructions drawn onto the input image. We propose a new VPI method, "goal hijacking via visual prompt injection" (GHVPI), that swaps the execution task of LVLMs from an original task to an alternative task designated by an attacker. The quantitative analysis indicates that GPT-4V is vulnerable to the GHVPI and demonstrates a notable attack success rate of 15.8%, which is an unignorable security risk. Our analysis also shows that successful GHVPI requires high character recognition capability and instruction-following ability in LVLMs.

Title: D2Styler: Advancing Arbitrary Style Transfer with Discrete Diffusion Methods

• Authors: Onkar Susladkar, Gayatri Deshmukh, Sparsh Mittal, Parth Shastri
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03558
• Pdf URL: https://arxiv.org/pdf/2408.03558
• Copy Paste: [[2408.03558]] D2Styler: Advancing Arbitrary Style Transfer with Discrete Diffusion Methods(https://arxiv.org/abs/2408.03558)
• Keywords: diffusion
• Abstract: In image processing, one of the most challenging tasks is to render an image's semantic meaning using a variety of artistic approaches. Existing techniques for arbitrary style transfer (AST) frequently experience mode-collapse, over-stylization, or under-stylization due to a disparity between the style and content images. We propose a novel framework called D$^2$Styler (Discrete Diffusion Styler) that leverages the discrete representational capability of VQ-GANs and the advantages of discrete diffusion, including stable training and avoidance of mode collapse. Our method uses Adaptive Instance Normalization (AdaIN) features as a context guide for the reverse diffusion process. This makes it easy to move features from the style image to the content image without bias. The proposed method substantially enhances the visual quality of style-transferred images, allowing the combination of content and style in a visually appealing manner. We take style images from the WikiArt dataset and content images from the COCO dataset. Experimental results demonstrate that D$^2$Styler produces high-quality style-transferred images and outperforms twelve existing methods on nearly all the metrics. The qualitative results and ablation studies provide further insights into the efficacy of our technique. The code is available at this https URL.

Title: In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Models

• Authors: Ayrton San Joaquin, Bin Wang, Zhengyuan Liu, Nicholas Asher, Brian Lim, Philippe Muller, Nancy Chen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.03560
• Pdf URL: https://arxiv.org/pdf/2408.03560
• Copy Paste: [[2408.03560]] In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Models(https://arxiv.org/abs/2408.03560)
• Keywords: large language model
• Abstract: Despite advancements, fine-tuning Large Language Models (LLMs) remains costly due to the extensive parameter count and substantial data requirements for model generalization. Accessibility to computing resources remains a barrier for the open-source community. To address this challenge, we propose the In2Core algorithm, which selects a coreset by analyzing the correlation between training and evaluation samples with a trained model. Notably, we assess the model's internal gradients to estimate this relationship, aiming to rank the contribution of each training point. To enhance efficiency, we propose an optimization to compute influence functions with a reduced number of layers while achieving similar accuracy. By applying our algorithm to instruction fine-tuning data of LLMs, we can achieve similar performance with just 50% of the training data. Meantime, using influence functions to analyze model coverage to certain testing samples could provide a reliable and interpretable signal on the training set's coverage of those test points.

Title: MPC-Minimized Secure LLM Inference

• Authors: Deevashwer Rathee, Dacheng Li, Ion Stoica, Hao Zhang, Raluca Popa
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03561
• Pdf URL: https://arxiv.org/pdf/2408.03561
• Copy Paste: [[2408.03561]] MPC-Minimized Secure LLM Inference(https://arxiv.org/abs/2408.03561)
• Keywords: secure, security, privacy, large language model
• Abstract: Many inference services based on large language models (LLMs) pose a privacy concern, either revealing user prompts to the service or the proprietary weights to the user. Secure inference offers a solution to this problem through secure multi-party computation (MPC), however, it is still impractical for modern LLM workload due to the large overhead imposed by MPC. To address this overhead, we propose Marill, a framework that adapts LLM fine-tuning to minimize MPC usage during secure inference. Marill introduces high-level architectural changes during fine-tuning that significantly reduce the number of expensive operations needed within MPC during inference, by removing some and relocating others outside MPC without compromising security. As a result, Marill-generated models are more efficient across all secure inference protocols and our approach complements MPC-friendly approximations for such operations. Compared to standard fine-tuning, Marill results in 3.6-11.3x better runtime and 2.4-6.9x better communication during secure inference across various MPC settings, while typically preserving over 90% performance across downstream tasks.

Title: A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Case

• Authors: Sonia Meyer, Shreya Singh, Bertha Tam, Christopher Ton, Angel Ren
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03562
• Pdf URL: https://arxiv.org/pdf/2408.03562
• Copy Paste: [[2408.03562]] A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Case(https://arxiv.org/abs/2408.03562)
• Keywords: large language model
• Abstract: This research compares large language model (LLM) fine-tuning methods, including Quantized Low Rank Adapter (QLoRA), Retrieval Augmented fine-tuning (RAFT), and Reinforcement Learning from Human Feedback (RLHF), and additionally compared LLM evaluation methods including End to End (E2E) benchmark method of "Golden Answers", traditional natural language processing (NLP) metrics, RAG Assessment (Ragas), OpenAI GPT-4 evaluation metrics, and human evaluation, using the travel chatbot use case. The travel dataset was sourced from the the Reddit API by requesting posts from travel-related subreddits to get travel-related conversation prompts and personalized travel experiences, and augmented for each fine-tuning method. We used two pretrained LLMs utilized for fine-tuning research: LLaMa 2 7B, and Mistral 7B. QLoRA and RAFT are applied to the two pretrained models. The inferences from these models are extensively evaluated against the aforementioned metrics. The best model according to human evaluation and some GPT-4 metrics was Mistral RAFT, so this underwent a Reinforcement Learning from Human Feedback (RLHF) training pipeline, and ultimately was evaluated as the best model. Our main findings are that: 1) quantitative and Ragas metrics do not align with human evaluation, 2) Open AI GPT-4 evaluation most aligns with human evaluation, 3) it is essential to keep humans in the loop for evaluation because, 4) traditional NLP metrics insufficient, 5) Mistral generally outperformed LLaMa, 6) RAFT outperforms QLoRA, but still needs postprocessing, 7) RLHF improves model performance significantly. Next steps include improving data quality, increasing data quantity, exploring RAG methods, and focusing data collection on a specific city, which would improve data quality by narrowing the focus, while creating a useful product.

Title: A comparative study of generative adversarial networks for image recognition algorithms based on deep learning and traditional methods

• Authors: Yihao Zhong, Yijing Wei, Yingbin Liang, Xiqing Liu, Rongwei Ji, Yiru Cang
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.03568
• Pdf URL: https://arxiv.org/pdf/2408.03568
• Copy Paste: [[2408.03568]] A comparative study of generative adversarial networks for image recognition algorithms based on deep learning and traditional methods(https://arxiv.org/abs/2408.03568)
• Keywords: extraction, generative
• Abstract: In this paper, an image recognition algorithm based on the combination of deep learning and generative adversarial network (GAN) is studied, and compared with traditional image recognition methods. The purpose of this study is to evaluate the advantages and application prospects of deep learning technology, especially GAN, in the field of image recognition. Firstly, this paper reviews the basic principles and techniques of traditional image recognition methods, including the classical algorithms based on feature extraction such as SIFT, HOG and their combination with support vector machine (SVM), random forest, and other classifiers. Then, the working principle, network structure, and unique advantages of GAN in image generation and recognition are introduced. In order to verify the effectiveness of GAN in image recognition, a series of experiments are designed and carried out using multiple public image data sets for training and testing. The experimental results show that compared with traditional methods, GAN has excellent performance in processing complex images, recognition accuracy, and anti-noise ability. Specifically, Gans are better able to capture high-dimensional features and details of images, significantly improving recognition performance. In addition, Gans shows unique advantages in dealing with image noise, partial missing information, and generating high-quality images.

Title: 2D-OOB: Attributing Data Contribution through Joint Valuation Framework

• Authors: Yifan Sun, Jingyan Shen, Yongchan Kwon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03572
• Pdf URL: https://arxiv.org/pdf/2408.03572
• Copy Paste: [[2408.03572]] 2D-OOB: Attributing Data Contribution through Joint Valuation Framework(https://arxiv.org/abs/2408.03572)
• Keywords: attack, interpretability
• Abstract: Data valuation has emerged as a powerful framework to quantify the contribution of each datum to the training of a particular machine learning model. However, it is crucial to recognize that the quality of various cells within a single data point can vary greatly in practice. For example, even in the case of an abnormal data point, not all cells are necessarily noisy. The single scalar valuation assigned by existing methods blurs the distinction between noisy and clean cells of a data point, thereby compromising the interpretability of the valuation. In this paper, we propose 2D-OOB, an out-of-bag estimation framework for jointly determining helpful (or detrimental) samples, as well as the particular cells that drive them. Our comprehensive experiments demonstrate that 2D-OOB achieves state-of-the-art performance across multiple use cases, while being exponentially faster. 2D-OOB excels in detecting and rectifying fine-grained outliers at the cell level, as well as localizing backdoor triggers in data poisoning attacks.

Title: Unraveling Privacy Threat Modeling Complexity: Conceptual Privacy Analysis Layers

• Authors: Kim Wuyts, Avi Douglen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.03578
• Pdf URL: https://arxiv.org/pdf/2408.03578
• Copy Paste: [[2408.03578]] Unraveling Privacy Threat Modeling Complexity: Conceptual Privacy Analysis Layers(https://arxiv.org/abs/2408.03578)
• Keywords: privacy
• Abstract: Analyzing privacy threats in software products is an essential part of software development to ensure systems are privacy-respecting; yet it is still a far from trivial activity. While there have been many advancements in the past decade, they tend to focus on describing 'what' the threats are. What isn't entirely clear yet is 'how' to actually find these threats. Privacy is a complex domain. We propose to use four conceptual layers (feature, ecosystem, business context, and environment) to capture this privacy complexity. These layers can be used as a frame to structure and specify the privacy analysis support in a more tangible and actionable way, thereby improving applicability of the analysis process.

Title: Hierarchical Neural Constructive Solver for Real-world TSP Scenarios

• Authors: Yong Liang Goh, Zhiguang Cao, Yining Ma, Yanfei Dong, Mohammed Haroon Dupty, Wee Sun Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03585
• Pdf URL: https://arxiv.org/pdf/2408.03585
• Copy Paste: [[2408.03585]] Hierarchical Neural Constructive Solver for Real-world TSP Scenarios(https://arxiv.org/abs/2408.03585)
• Keywords: robust, transformer
• Abstract: Existing neural constructive solvers for routing problems have predominantly employed transformer architectures, conceptualizing the route construction as a set-to-sequence learning task. However, their efficacy has primarily been demonstrated on entirely random problem instances that inadequately capture real-world scenarios. In this paper, we introduce realistic Traveling Salesman Problem (TSP) scenarios relevant to industrial settings and derive the following insights: (1) The optimal next node (or city) to visit often lies within proximity to the current node, suggesting the potential benefits of biasing choices based on current locations. (2) Effectively solving the TSP requires robust tracking of unvisited nodes and warrants succinct grouping strategies. Building upon these insights, we propose integrating a learnable choice layer inspired by Hypernetworks to prioritize choices based on the current location, and a learnable approximate clustering algorithm inspired by the Expectation-Maximization algorithm to facilitate grouping the unvisited cities. Together, these two contributions form a hierarchical approach towards solving the realistic TSP by considering both immediate local neighbourhoods and learning an intermediate set of node representations. Our hierarchical approach yields superior performance compared to both classical and recent transformer models, showcasing the efficacy of the key designs.

Title: EnJa: Ensemble Jailbreak on Large Language Models

• Authors: Jiahao Zhang, Zilong Wang, Ruofan Wang, Xingjun Ma, Yu-Gang Jiang
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03603
• Pdf URL: https://arxiv.org/pdf/2408.03603
• Copy Paste: [[2408.03603]] EnJa: Ensemble Jailbreak on Large Language Models(https://arxiv.org/abs/2408.03603)
• Keywords: protect, attack, large language model
• Abstract: As Large Language Models (LLMs) are increasingly being deployed in safety-critical applications, their vulnerability to potential jailbreaks -- malicious prompts that can disable the safety mechanism of LLMs -- has attracted growing research attention. While alignment methods have been proposed to protect LLMs from jailbreaks, many have found that aligned LLMs can still be jailbroken by carefully crafted malicious prompts, producing content that violates policy regulations. Existing jailbreak attacks on LLMs can be categorized into prompt-level methods which make up stories/logic to circumvent safety alignment and token-level attack methods which leverage gradient methods to find adversarial tokens. In this work, we introduce the concept of Ensemble Jailbreak and explore methods that can integrate prompt-level and token-level jailbreak into a more powerful hybrid jailbreak attack. Specifically, we propose a novel EnJa attack to hide harmful instructions using prompt-level jailbreak, boost the attack success rate using a gradient-based attack, and connect the two types of jailbreak attacks via a template-based connector. We evaluate the effectiveness of EnJa on several aligned models and show that it achieves a state-of-the-art attack success rate with fewer queries and is much stronger than any individual jailbreak.

Title: InPer: Whole-Process Domain Generalization via Causal Intervention and Perturbation

• Authors: Luyao Tang, Yuxuan Yuan, Chaoqi Chen, Xinghao Ding, Yue Huang
• Subjects: cs.LG, cs.CV, stat.ME
• Abstract URL: https://arxiv.org/abs/2408.03608
• Pdf URL: https://arxiv.org/pdf/2408.03608
• Copy Paste: [[2408.03608]] InPer: Whole-Process Domain Generalization via Causal Intervention and Perturbation(https://arxiv.org/abs/2408.03608)
• Keywords: robust
• Abstract: Despite the considerable advancements achieved by deep neural networks, their performance tends to degenerate when the test environment diverges from the training ones. Domain generalization (DG) solves this issue by learning representations independent of domain-related information, thus facilitating extrapolation to unseen environments. Existing approaches typically focus on formulating tailored training objectives to extract shared features from the source data. However, the disjointed training and testing procedures may compromise robustness, particularly in the face of unforeseen variations during deployment. In this paper, we propose a novel and holistic framework based on causality, named InPer, designed to enhance model generalization by incorporating causal intervention during training and causal perturbation during testing. Specifically, during the training phase, we employ entropy-based causal intervention (EnIn) to refine the selection of causal variables. To identify samples with anti-interference causal variables from the target domain, we propose a novel metric, homeostatic score, through causal perturbation (HoPer) to construct a prototype classifier in test time. Experimental results across multiple cross-domain tasks confirm the efficacy of InPer.

Title: JARViS: Detecting Actions in Video Using Unified Actor-Scene Context Relation Modeling

• Authors: Seok Hwan Lee, Taein Son, Soo Won Seo, Jisong Kim, Jun Won Choi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03612
• Pdf URL: https://arxiv.org/pdf/2408.03612
• Copy Paste: [[2408.03612]] JARViS: Detecting Actions in Video Using Unified Actor-Scene Context Relation Modeling(https://arxiv.org/abs/2408.03612)
• Keywords: transformer
• Abstract: Video action detection (VAD) is a formidable vision task that involves the localization and classification of actions within the spatial and temporal dimensions of a video clip. Among the myriad VAD architectures, two-stage VAD methods utilize a pre-trained person detector to extract the region of interest features, subsequently employing these features for action detection. However, the performance of two-stage VAD methods has been limited as they depend solely on localized actor features to infer action semantics. In this study, we propose a new two-stage VAD framework called Joint Actor-scene context Relation modeling based on Visual Semantics (JARViS), which effectively consolidates cross-modal action semantics distributed globally across spatial and temporal dimensions using Transformer attention. JARViS employs a person detector to produce densely sampled actor features from a keyframe. Concurrently, it uses a video backbone to create spatio-temporal scene features from a video clip. Finally, the fine-grained interactions between actors and scenes are modeled through a Unified Action-Scene Context Transformer to directly output the final set of actions in parallel. Our experimental results demonstrate that JARViS outperforms existing methods by significant margins and achieves state-of-the-art performance on three popular VAD datasets, including AVA, UCF101-24, and JHMDB51-21.

Title: A Logical Fallacy-Informed Framework for Argument Generation

• Authors: Luca Mouchel, Debjit Paul, Shaobo Cui, Robert West, Antoine Bosselut, Boi Faltings
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03618
• Pdf URL: https://arxiv.org/pdf/2408.03618
• Copy Paste: [[2408.03618]] A Logical Fallacy-Informed Framework for Argument Generation(https://arxiv.org/abs/2408.03618)
• Keywords: large language model
• Abstract: Despite the remarkable performance of Large Language Models (LLMs), they still struggle with generating logically sound arguments, resulting in potential risks such as spreading misinformation. An important factor contributing to LLMs' suboptimal performance in generating coherent arguments is their oversight of logical fallacies. To address this issue, we introduce FIPO, a fallacy-informed framework that leverages preference optimization methods to steer LLMs toward logically sound arguments. FIPO includes a classification loss, to capture the fine-grained information on fallacy categories. Our results on argumentation datasets show that our method reduces the fallacy errors by up to 17.5%. Furthermore, our human evaluation results indicate that the quality of the generated arguments by our method significantly outperforms the fine-tuned baselines, as well as prior preference optimization methods, such as DPO. These findings highlight the importance of ensuring models are aware of logical fallacies for effective argument generation.

Title: Making Robust Generalizers Less Rigid with Soft Ascent-Descent

• Authors: Matthew J. Holland, Toma Hamada
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03619
• Pdf URL: https://arxiv.org/pdf/2408.03619
• Copy Paste: [[2408.03619]] Making Robust Generalizers Less Rigid with Soft Ascent-Descent(https://arxiv.org/abs/2408.03619)
• Keywords: robust
• Abstract: While the traditional formulation of machine learning tasks is in terms of performance on average, in practice we are often interested in how well a trained model performs on rare or difficult data points at test time. To achieve more robust and balanced generalization, methods applying sharpness-aware minimization to a subset of worst-case examples have proven successful for image classification tasks, but only using deep neural networks in a scenario where the most difficult points are also the least common. In this work, we show how such a strategy can dramatically break down under more diverse models, and as a more robust alternative, instead of typical sharpness we propose and evaluate a training criterion which penalizes poor loss concentration, which can be easily combined with loss transformations such as CVaR or DRO that control tail emphasis.

Title: Improving the quality of Persian clinical text with a novel spelling correction system

• Authors: Seyed Mohammad Sadegh Dashti, Seyedeh Fatemeh Dashti
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03622
• Pdf URL: https://arxiv.org/pdf/2408.03622
• Copy Paste: [[2408.03622]] Improving the quality of Persian clinical text with a novel spelling correction system(https://arxiv.org/abs/2408.03622)
• Keywords: robust
• Abstract: Background: The accuracy of spelling in Electronic Health Records (EHRs) is a critical factor for efficient clinical care, research, and ensuring patient safety. The Persian language, with its abundant vocabulary and complex characteristics, poses unique challenges for real-word error correction. This research aimed to develop an innovative approach for detecting and correcting spelling errors in Persian clinical text. Methods: Our strategy employs a state-of-the-art pre-trained model that has been meticulously fine-tuned specifically for the task of spelling correction in the Persian clinical domain. This model is complemented by an innovative orthographic similarity matching algorithm, PERTO, which uses visual similarity of characters for ranking correction candidates. Results: The evaluation of our approach demonstrated its robustness and precision in detecting and rectifying word errors in Persian clinical text. In terms of non-word error correction, our model achieved an F1-Score of 90.0% when the PERTO algorithm was employed. For real-word error detection, our model demonstrated its highest performance, achieving an F1-Score of 90.6%. Furthermore, the model reached its highest F1-Score of 91.5% for real-word error correction when the PERTO algorithm was employed. Conclusions: Despite certain limitations, our method represents a substantial advancement in the field of spelling error detection and correction for Persian clinical text. By effectively addressing the unique challenges posed by the Persian language, our approach paves the way for more accurate and efficient clinical documentation, contributing to improved patient care and safety. Future research could explore its use in other areas of the Persian medical domain, enhancing its impact and utility.

Title: AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Merging

• Authors: Senkang Hu, Zhengru Fang, Zihan Fang, Yiqin Deng, Xianhao Chen, Yuguang Fang, Sam Kwong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03624
• Pdf URL: https://arxiv.org/pdf/2408.03624
• Copy Paste: [[2408.03624]] AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Merging(https://arxiv.org/abs/2408.03624)
• Keywords: large language model
• Abstract: Ramp merging is one of the bottlenecks in traffic systems, which commonly cause traffic congestion, accidents, and severe carbon emissions. In order to address this essential issue and enhance the safety and efficiency of connected and autonomous vehicles (CAVs) at multi-lane merging zones, we propose a novel collaborative decision-making framework, named AgentsCoMerge, to leverage large language models (LLMs). Specifically, we first design a scene observation and understanding module to allow an agent to capture the traffic environment. Then we propose a hierarchical planning module to enable the agent to make decisions and plan trajectories based on the observation and the agent's own state. In addition, in order to facilitate collaboration among multiple agents, we introduce a communication module to enable the surrounding agents to exchange necessary information and coordinate their actions. Finally, we develop a reinforcement reflection guided training paradigm to further enhance the decision-making capability of the framework. Extensive experiments are conducted to evaluate the performance of our proposed method, demonstrating its superior efficiency and effectiveness for multi-agent collaborative decision-making under various ramp merging scenarios.

Title: PAGED: A Benchmark for Procedural Graphs Extraction from Documents

• Authors: Weihong Du, Wenrui Liao, Hongru Liang, Wenqiang Lei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03630
• Pdf URL: https://arxiv.org/pdf/2408.03630
• Copy Paste: [[2408.03630]] PAGED: A Benchmark for Procedural Graphs Extraction from Documents(https://arxiv.org/abs/2408.03630)
• Keywords: extraction, large language model
• Abstract: Automatic extraction of procedural graphs from documents creates a low-cost way for users to easily understand a complex procedure by skimming visual graphs. Despite the progress in recent studies, it remains unanswered: whether the existing studies have well solved this task (Q1) and whether the emerging large language models (LLMs) can bring new opportunities to this task (Q2). To this end, we propose a new benchmark PAGED, equipped with a large high-quality dataset and standard evaluations. It investigates five state-of-the-art baselines, revealing that they fail to extract optimal procedural graphs well because of their heavy reliance on hand-written rules and limited available data. We further involve three advanced LLMs in PAGED and enhance them with a novel self-refine strategy. The results point out the advantages of LLMs in identifying textual elements and their gaps in building logical structures. We hope PAGED can serve as a major landmark for automatic procedural graph extraction and the investigations in PAGED can offer insights into the research on logic reasoning among non-sequential elements.

Title: TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimization

• Authors: Kien T. Pham, Jingye Chen, Qifeng Chen
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.03637
• Pdf URL: https://arxiv.org/pdf/2408.03637
• Copy Paste: [[2408.03637]] TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimization(https://arxiv.org/abs/2408.03637)
• Keywords: robust, diffusion, generative
• Abstract: We present TALE, a novel training-free framework harnessing the generative capabilities of text-to-image diffusion models to address the cross-domain image composition task that focuses on flawlessly incorporating user-specified objects into a designated visual contexts regardless of domain disparity. Previous methods often involve either training auxiliary networks or finetuning diffusion models on customized datasets, which are expensive and may undermine the robust textual and visual priors of pre-trained diffusion models. Some recent works attempt to break the barrier by proposing training-free workarounds that rely on manipulating attention maps to tame the denoising process implicitly. However, composing via attention maps does not necessarily yield desired compositional outcomes. These approaches could only retain some semantic information and usually fall short in preserving identity characteristics of input objects or exhibit limited background-object style adaptation in generated images. In contrast, TALE is a novel method that operates directly on latent space to provide explicit and effective guidance for the composition process to resolve these problems. Specifically, we equip TALE with two mechanisms dubbed Adaptive Latent Manipulation and Energy-guided Latent Optimization. The former formulates noisy latents conducive to initiating and steering the composition process by directly leveraging background and foreground latents at corresponding timesteps, and the latter exploits designated energy functions to further optimize intermediate latents conforming to specific conditions that complement the former to generate desired final results. Our experiments demonstrate that TALE surpasses prior baselines and attains state-of-the-art performance in image-guided composition across various photorealistic and artistic domains.

Title: Consumer Transactions Simulation through Generative Adversarial Networks

• Authors: Sergiy Tkachuk, Szymon Łukasik, Anna Wróblewska
• Subjects: cs.LG, cs.IR, q-fin.CP
• Abstract URL: https://arxiv.org/abs/2408.03655
• Pdf URL: https://arxiv.org/pdf/2408.03655
• Copy Paste: [[2408.03655]] Consumer Transactions Simulation through Generative Adversarial Networks(https://arxiv.org/abs/2408.03655)
• Keywords: generative
• Abstract: In the rapidly evolving domain of large-scale retail data systems, envisioning and simulating future consumer transactions has become a crucial area of interest. It offers significant potential to fortify demand forecasting and fine-tune inventory management. This paper presents an innovative application of Generative Adversarial Networks (GANs) to generate synthetic retail transaction data, specifically focusing on a novel system architecture that combines consumer behavior modeling with stock-keeping unit (SKU) availability constraints to address real-world assortment optimization challenges. We diverge from conventional methodologies by integrating SKU data into our GAN architecture and using more sophisticated embedding methods (e.g., hyper-graphs). This design choice enables our system to generate not only simulated consumer purchase behaviors but also reflects the dynamic interplay between consumer behavior and SKU availability -- an aspect often overlooked, among others, because of data scarcity in legacy retail simulation models. Our GAN model generates transactions under stock constraints, pioneering a resourceful experimental system with practical implications for real-world retail operation and strategy. Preliminary results demonstrate enhanced realism in simulated transactions measured by comparing generated items with real ones using methods employed earlier in related studies. This underscores the potential for more accurate predictive modeling.

Title: Designing Extremely Memory-Efficient CNNs for On-device Vision Tasks

• Authors: Jaewook Lee, Yoel Park, Seulki Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03663
• Pdf URL: https://arxiv.org/pdf/2408.03663
• Copy Paste: [[2408.03663]] Designing Extremely Memory-Efficient CNNs for On-device Vision Tasks(https://arxiv.org/abs/2408.03663)
• Keywords: segmentation
• Abstract: In this paper, we introduce a memory-efficient CNN (convolutional neural network), which enables resource-constrained low-end embedded and IoT devices to perform on-device vision tasks, such as image classification and object detection, using extremely low memory, i.e., only 63 KB on ImageNet classification. Based on the bottleneck block of MobileNet, we propose three design principles that significantly curtail the peak memory usage of a CNN so that it can fit the limited KB memory of the low-end device. First, 'input segmentation' divides an input image into a set of patches, including the central patch overlapped with the others, reducing the size (and memory requirement) of a large input image. Second, 'patch tunneling' builds independent tunnel-like paths consisting of multiple bottleneck blocks per patch, penetrating through the entire model from an input patch to the last layer of the network, maintaining lightweight memory usage throughout the whole network. Lastly, 'bottleneck reordering' rearranges the execution order of convolution operations inside the bottleneck block such that the memory usage remains constant regardless of the size of the convolution output channels. The experiment result shows that the proposed network classifies ImageNet with extremely low memory (i.e., 63 KB) while achieving competitive top-1 accuracy (i.e., 61.58\%). To the best of our knowledge, the memory usage of the proposed network is far smaller than state-of-the-art memory-efficient networks, i.e., up to 89x and 3.1x smaller than MobileNet (i.e., 5.6 MB) and MCUNet (i.e., 196 KB), respectively.

Title: AI-Driven approach for sustainable extraction of earth's subsurface renewable energy while minimizing seismic activity

• Authors: Diego Gutierrez-Oribio, Alexandros Stathas, Ioannis Stefanou
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.03664
• Pdf URL: https://arxiv.org/pdf/2408.03664
• Copy Paste: [[2408.03664]] AI-Driven approach for sustainable extraction of earth's subsurface renewable energy while minimizing seismic activity(https://arxiv.org/abs/2408.03664)
• Keywords: robust, extraction
• Abstract: Deep Geothermal Energy, Carbon Capture and Storage, and Hydrogen Storage hold considerable promise for meeting the energy sector's large-scale requirements and reducing CO$_2$ emissions. However, the injection of fluids into the Earth's crust, essential for these activities, can induce or trigger earthquakes. In this paper, we highlight a new approach based on Reinforcement Learning for the control of human-induced seismicity in the highly complex environment of an underground reservoir. This complex system poses significant challenges in the control design due to parameter uncertainties and unmodeled dynamics. We show that the reinforcement learning algorithm can interact efficiently with a robust controller, by choosing the controller parameters in real-time, reducing human-induced seismicity and allowing the consideration of further production objectives, \textit{e.g.}, minimal control power. Simulations are presented for a simplified underground reservoir under various energy demand scenarios, demonstrating the reliability and effectiveness of the proposed control-reinforcement learning approach.

Title: NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time

• Authors: Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, Hua Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03675
• Pdf URL: https://arxiv.org/pdf/2408.03675
• Copy Paste: [[2408.03675]] NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time(https://arxiv.org/abs/2408.03675)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have ignited an innovative surge of AI applications, marking a new era of exciting possibilities equipped with extended context windows. However, hosting these models is cost-prohibitive mainly due to the extensive memory consumption of KV Cache involving long-context modeling. Despite several works proposing to evict unnecessary tokens from the KV Cache, most of them rely on the biased local statistics of accumulated attention scores and report performance using unconvincing metric like perplexity on inadequate short-text evaluation. In this paper, we propose NACL, a general framework for long-context KV cache eviction that achieves more optimal and efficient eviction in a single operation during the encoding phase. Due to NACL's efficiency, we combine more accurate attention score statistics in PROXY TOKENS EVICTION with the diversified random eviction strategy of RANDOM EVICTION, aiming to alleviate the issue of attention bias and enhance the robustness in maintaining pivotal tokens for long-context modeling tasks. Notably, our method significantly improves the performance on short- and long-text tasks by 80% and 76% respectively, reducing KV Cache by up to 50% with over 95% performance maintenance. The code is available at https: //github.com/PaddlePaddle/Research/ tree/master/NLP/ACL2024-NACL.

Title: L4DR: LiDAR-4DRadar Fusion for Weather-Robust 3D Object Detection

• Authors: Xun Huang, Ziyu Xu, Hai Wu, Jinlong Wang, Qiming Xia, Yan Xia, Jonathan Li, Kyle Gao, Chenglu Wen, Cheng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03677
• Pdf URL: https://arxiv.org/pdf/2408.03677
• Copy Paste: [[2408.03677]] L4DR: LiDAR-4DRadar Fusion for Weather-Robust 3D Object Detection(https://arxiv.org/abs/2408.03677)
• Keywords: robust, extraction
• Abstract: LiDAR-based vision systems are integral for 3D object detection, which is crucial for autonomous navigation. However, they suffer from performance degradation in adverse weather conditions due to the quality deterioration of LiDAR point clouds. Fusing LiDAR with the weather-robust 4D radar sensor is expected to solve this problem. However, the fusion of LiDAR and 4D radar is challenging because they differ significantly in terms of data quality and the degree of degradation in adverse weather. To address these issues, we introduce L4DR, a weather-robust 3D object detection method that effectively achieves LiDAR and 4D Radar fusion. Our L4DR includes Multi-Modal Encoding (MME) and Foreground-Aware Denoising (FAD) technique to reconcile sensor gaps, which is the first exploration of the complementarity of early fusion between LiDAR and 4D radar. Additionally, we design an Inter-Modal and Intra-Modal ({IM}2 ) parallel feature extraction backbone coupled with a Multi-Scale Gated Fusion (MSGF) module to counteract the varying degrees of sensor degradation under adverse weather conditions. Experimental evaluation on a VoD dataset with simulated fog proves that L4DR is more adaptable to changing weather conditions. It delivers a significant performance increase under different fog levels, improving the 3D mAP by up to 18.17% over the traditional LiDAR-only approach. Moreover, the results on the K-Radar dataset validate the consistent performance improvement of L4DR in real-world adverse weather conditions.

Title: Generative Design of Periodic Orbits in the Restricted Three-Body Problem

• Authors: Alvaro Francisco Gil, Walther Litteri, Victor Rodriguez-Fernandez, David Camacho, Massimiliano Vasile
• Subjects: cs.LG, astro-ph.EP, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03691
• Pdf URL: https://arxiv.org/pdf/2408.03691
• Copy Paste: [[2408.03691]] Generative Design of Periodic Orbits in the Restricted Three-Body Problem(https://arxiv.org/abs/2408.03691)
• Keywords: generative
• Abstract: The Three-Body Problem has fascinated scientists for centuries and it has been crucial in the design of modern space missions. Recent developments in Generative Artificial Intelligence hold transformative promise for addressing this longstanding problem. This work investigates the use of Variational Autoencoder (VAE) and its internal representation to generate periodic orbits. We utilize a comprehensive dataset of periodic orbits in the Circular Restricted Three-Body Problem (CR3BP) to train deep-learning architectures that capture key orbital characteristics, and we set up physical evaluation metrics for the generated trajectories. Through this investigation, we seek to enhance the understanding of how Generative AI can improve space mission planning and astrodynamics research, leading to novel, data-driven approaches in the field.

Title: Openstory++: A Large-scale Dataset and Benchmark for Instance-aware Open-domain Visual Storytelling

• Authors: Zilyu Ye, Jinxiu Liu, Ruotian Peng, Jinjin Cao, Zhiyang Chen, Yiyang Zhang, Ziwei Xuan, Mingyuan Zhou, Xiaoqian Shen, Mohamed Elhoseiny, Qi Liu, Guo-Jun Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03695
• Pdf URL: https://arxiv.org/pdf/2408.03695
• Copy Paste: [[2408.03695]] Openstory++: A Large-scale Dataset and Benchmark for Instance-aware Open-domain Visual Storytelling(https://arxiv.org/abs/2408.03695)
• Keywords: extraction, large language model
• Abstract: Recent image generation models excel at creating high-quality images from brief captions. However, they fail to maintain consistency of multiple instances across images when encountering lengthy contexts. This inconsistency is largely due to in existing training datasets the absence of granular instance feature labeling in existing training datasets. To tackle these issues, we introduce Openstory++, a large-scale dataset combining additional instance-level annotations with both images and text. Furthermore, we develop a training methodology that emphasizes entity-centric image-text generation, ensuring that the models learn to effectively interweave visual and textual information. Specifically, Openstory++ streamlines the process of keyframe extraction from open-domain videos, employing vision-language models to generate captions that are then polished by a large language model for narrative continuity. It surpasses previous datasets by offering a more expansive open-domain resource, which incorporates automated captioning, high-resolution imagery tailored for instance count, and extensive frame sequences for temporal consistency. Additionally, we present Cohere-Bench, a pioneering benchmark framework for evaluating the image generation tasks when long multimodal context is provided, including the ability to keep the background, style, instances in the given context coherent. Compared to existing benchmarks, our work fills critical gaps in multi-modal generation, propelling the development of models that can adeptly generate and interpret complex narratives in open-domain environments. Experiments conducted within Cohere-Bench confirm the superiority of Openstory++ in nurturing high-quality visual storytelling models, enhancing their ability to address open-domain generation tasks. More details can be found at this https URL

Title: CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications

• Authors: Tianfang Zhang, Lei Li, Yang Zhou, Wentao Liu, Chen Qian, Xiangyang Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03703
• Pdf URL: https://arxiv.org/pdf/2408.03703
• Copy Paste: [[2408.03703]] CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications(https://arxiv.org/abs/2408.03703)
• Keywords: transformer, segmentation
• Abstract: Vision Transformers (ViTs) mark a revolutionary advance in neural networks with their token mixer's powerful global context capability. However, the pairwise token affinity and complex matrix operations limit its deployment on resource-constrained scenarios and real-time applications, such as mobile devices, although considerable efforts have been made in previous works. In this paper, we introduce CAS-ViT: Convolutional Additive Self-attention Vision Transformers, to achieve a balance between efficiency and performance in mobile applications. Firstly, we argue that the capability of token mixers to obtain global contextual information hinges on multiple information interactions, such as spatial and channel domains. Subsequently, we construct a novel additive similarity function following this paradigm and present an efficient implementation named Convolutional Additive Token Mixer (CATM). This simplification leads to a significant reduction in computational overhead. We evaluate CAS-ViT across a variety of vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Our experiments, conducted on GPUs, ONNX, and iPhones, demonstrate that CAS-ViT achieves a competitive performance when compared to other state-of-the-art backbones, establishing it as a viable option for efficient mobile vision applications. Our code and model are available at: \url{this https URL}

Title: BioDeepHash: Mapping Biometrics into a Stable Code

• Authors: Baogang Song, Dongdong Zhao, Jiang Yan, Huanhuan Li, Hao Jiang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.03704
• Pdf URL: https://arxiv.org/pdf/2408.03704
• Copy Paste: [[2408.03704]] BioDeepHash: Mapping Biometrics into a Stable Code(https://arxiv.org/abs/2408.03704)
• Keywords: security, protect, attack, biometric
• Abstract: With the wide application of biometrics, more and more attention has been paid to the security of biometric templates. However most of existing biometric template protection (BTP) methods have some security problems, e.g. the problem that protected templates leak part of the original biometric data (exists in Cancelable Biometrics (CB)), the use of error-correcting codes (ECC) leads to decodable attack, statistical attack (exists in Biometric Cryptosystems (BCS)), the inability to achieve revocability (exists in methods using Neural Network (NN) to learn pre-defined templates), the inability to use cryptographic hash to guarantee strong security (exists in CB and methods using NN to learn latent templates). In this paper, we propose a framework called BioDeepHash based on deep hashing and cryptographic hashing to address the above four problems, where different biometric data of the same user are mapped to a stable code using deep hashing instead of predefined binary codes thus avoiding the use of ECC. An application-specific binary string is employed to achieve revocability. Then cryptographic hashing is used to get the final protected template to ensure strong security. Ultimately our framework achieves not storing any data that would leak part of the original biometric data. We also conduct extensive experiments on facial and iris datasets. Our method achieves an improvement of 10.12$\%$ on the average Genuine Acceptance Rate (GAR) for iris data and 3.12$\%$ for facial data compared to existing methods. In addition, BioDeepHash achieves extremely low False Acceptance Rate (FAR), i.e. 0$\%$ FAR on the iris dataset and the highest FAR on the facial dataset is only 0.0002$\%$.

Title: Local Topology Measures of Contextual Language Model Latent Spaces With Applications to Dialogue Term Extraction

• Authors: Benjamin Matthias Ruppik, Michael Heck, Carel van Niekerk, Renato Vukovic, Hsien-chin Lin, Shutong Feng, Marcus Zibrowius, Milica Gašić
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03706
• Pdf URL: https://arxiv.org/pdf/2408.03706
• Copy Paste: [[2408.03706]] Local Topology Measures of Contextual Language Model Latent Spaces With Applications to Dialogue Term Extraction(https://arxiv.org/abs/2408.03706)
• Keywords: extraction
• Abstract: A common approach for sequence tagging tasks based on contextual word representations is to train a machine learning classifier directly on these embedding vectors. This approach has two shortcomings. First, such methods consider single input sequences in isolation and are unable to put an individual embedding vector in relation to vectors outside the current local context of use. Second, the high performance of these models relies on fine-tuning the embedding model in conjunction with the classifier, which may not always be feasible due to the size or inaccessibility of the underlying feature-generation model. It is thus desirable, given a collection of embedding vectors of a corpus, i.e., a datastore, to find features of each vector that describe its relation to other, similar vectors in the datastore. With this in mind, we introduce complexity measures of the local topology of the latent space of a contextual language model with respect to a given datastore. The effectiveness of our features is demonstrated through their application to dialogue term extraction. Our work continues a line of research that explores the manifold hypothesis for word embeddings, demonstrating that local structure in the space carved out by word embeddings can be exploited to infer semantic properties.

Title: Centralized Defense: Logging and Mitigation of Kubernetes Misconfigurations with Open Source Tools

• Authors: Eoghan Russell, Kapal Dev
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.03714
• Pdf URL: https://arxiv.org/pdf/2408.03714
• Copy Paste: [[2408.03714]] Centralized Defense: Logging and Mitigation of Kubernetes Misconfigurations with Open Source Tools(https://arxiv.org/abs/2408.03714)
• Keywords: security, defense
• Abstract: Kubernetes, an open-source platform for automating the deployment, scaling, and management of containerized applications, is widely used for its efficiency and scalability. However, its complexity and extensive configuration options often lead to security vulnerabilities if not managed properly. This paper presents a detailed analysis of misconfigurations in Kubernetes environments and their significant impact on system reliability and security. A centralized logging solution was developed to detect such misconfigurations, detailing the integration process with a Kubernetes cluster and the implementation of role-based access control. Utilizing a combination of open-source tools, the solution systematically identifies misconfigurations and aggregates diagnostic data into a central repository. The effectiveness of the solution was evaluated using specific metrics, such as the total cycle time for running the central logging solution against the individual open source tools.

Title: A Convex-optimization-based Layer-wise Post-training Pruner for Large Language Models

• Authors: Pengxiang Zhao, Hanyu Hu, Ping Li, Yi Zheng, Zhefeng Wang, Xiaoming Yuan
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2408.03728
• Pdf URL: https://arxiv.org/pdf/2408.03728
• Copy Paste: [[2408.03728]] A Convex-optimization-based Layer-wise Post-training Pruner for Large Language Models(https://arxiv.org/abs/2408.03728)
• Keywords: large language model
• Abstract: Pruning is a critical strategy for compressing trained large language models (LLMs), aiming at substantial memory conservation and computational acceleration without compromising performance. However, existing pruning methods often necessitate inefficient retraining for billion-scale LLMs or rely on heuristic methods such as the optimal brain surgeon framework, which degrade performance. In this paper, we introduce FISTAPruner, the first post-training pruner based on convex optimization models and algorithms. Specifically, we propose a convex optimization model incorporating $\ell_1$ norm to induce sparsity and utilize the FISTA solver for optimization. FISTAPruner incorporates an intra-layer cumulative error correction mechanism and supports parallel pruning. We comprehensively evaluate FISTAPruner on models such as OPT, LLaMA, LLaMA-2, and LLaMA-3 with 125M to 70B parameters under unstructured and 2:4 semi-structured sparsity, demonstrating superior performance over existing state-of-the-art methods across various language benchmarks.

Title: Question Rephrasing for Quantifying Uncertainty in Large Language Models: Applications in Molecular Chemistry Tasks

• Authors: Zizhang Chen, Pengyu Hong, Sandeep Madireddy
• Subjects: cs.CL, cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2408.03732
• Pdf URL: https://arxiv.org/pdf/2408.03732
• Copy Paste: [[2408.03732]] Question Rephrasing for Quantifying Uncertainty in Large Language Models: Applications in Molecular Chemistry Tasks(https://arxiv.org/abs/2408.03732)
• Keywords: large language model
• Abstract: Uncertainty quantification enables users to assess the reliability of responses generated by large language models (LLMs). We present a novel Question Rephrasing technique to evaluate the input uncertainty of LLMs, which refers to the uncertainty arising from equivalent variations of the inputs provided to LLMs. This technique is integrated with sampling methods that measure the output uncertainty of LLMs, thereby offering a more comprehensive uncertainty assessment. We validated our approach on property prediction and reaction prediction for molecular chemistry tasks.

Title: Soft-Hard Attention U-Net Model and Benchmark Dataset for Multiscale Image Shadow Removal

• Authors: Eirini Cholopoulou, Dimitrios E. Diamantis, Dimitra-Christina C. Koutsiou, Dimitris K. Iakovidis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03734
• Pdf URL: https://arxiv.org/pdf/2408.03734
• Copy Paste: [[2408.03734]] Soft-Hard Attention U-Net Model and Benchmark Dataset for Multiscale Image Shadow Removal(https://arxiv.org/abs/2408.03734)
• Keywords: privacy, extraction
• Abstract: Effective shadow removal is pivotal in enhancing the visual quality of images in various applications, ranging from computer vision to digital photography. During the last decades physics and machine learning -based methodologies have been proposed; however, most of them have limited capacity in capturing complex shadow patterns due to restrictive model assumptions, neglecting the fact that shadows usually appear at different scales. Also, current datasets used for benchmarking shadow removal are composed of a limited number of images with simple scenes containing mainly uniform shadows cast by single objects, whereas only a few of them include both manual shadow annotations and paired shadow-free images. Aiming to address all these limitations in the context of natural scene imaging, including urban environments with complex scenes, the contribution of this study is twofold: a) it proposes a novel deep learning architecture, named Soft-Hard Attention U-net (SHAU), focusing on multiscale shadow removal; b) it provides a novel synthetic dataset, named Multiscale Shadow Removal Dataset (MSRD), containing complex shadow patterns of multiple scales, aiming to serve as a privacy-preserving dataset for a more comprehensive benchmarking of future shadow removal methodologies. Key architectural components of SHAU are the soft and hard attention modules, which along with multiscale feature extraction blocks enable effective shadow removal of different scales and intensities. The results demonstrate the effectiveness of SHAU over the relevant state-of-the-art shadow removal methods across various benchmark datasets, improving the Peak Signal-to-Noise Ratio and Root Mean Square Error for the shadow area by 25.1% and 61.3%, respectively.

Title: Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation

• Authors: Jingjing Xie, Yuxin Zhang, Mingbao Lin, Liujuan Cao, Rongrong Ji
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03735
• Pdf URL: https://arxiv.org/pdf/2408.03735
• Copy Paste: [[2408.03735]] Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation(https://arxiv.org/abs/2408.03735)
• Keywords: large language model
• Abstract: This paper presents the first study to explore the potential of parameter quantization for multimodal large language models to alleviate the significant resource constraint encountered during vision-language instruction tuning. We introduce a Quantization-aware Scale LeArning method based on multimodal Warmup, termed QSLAW. This method is grounded in two key innovations: (1) The learning of group-wise scale factors for quantized LLM weights to mitigate the quantization error arising from activation outliers and achieve more effective vision-language instruction tuning; (2) The implementation of a multimodal warmup that progressively integrates linguistic and multimodal training samples, thereby preventing overfitting of the quantized model to multimodal data while ensuring stable adaptation of multimodal large language models to downstream vision-language tasks. Extensive experiments demonstrate that models quantized by QSLAW perform on par with, or even surpass, their full-precision counterparts, while facilitating up to 1.4 times reduction in VL tuning time and GPU consumption. Our code is released at this https URL.

Title: Intuitionistic Fuzzy Cognitive Maps for Interpretable Image Classification

• Authors: Georgia Sovatzidi, Michael D. Vasilakakis, Dimitris K. Iakovidis
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03745
• Pdf URL: https://arxiv.org/pdf/2408.03745
• Copy Paste: [[2408.03745]] Intuitionistic Fuzzy Cognitive Maps for Interpretable Image Classification(https://arxiv.org/abs/2408.03745)
• Keywords: extraction, interpretability
• Abstract: The interpretability of machine learning models is critical, as users may be reluctant to rely on their inferences. Intuitionistic FCMs (iFCMs) have been proposed as an extension of FCMs offering a natural mechanism to assess the quality of their output through the estimation of hesitancy, a concept resembling to human hesitation in decision making. To address the challenge of interpretable image classification, this paper introduces a novel framework, named Interpretable Intuitionistic FCM (I2FCM) which is domain-independent, simple to implement, and can be applied on Convolutional Neural Network (CNN) models, rendering them interpretable. To the best of our knowledge this is the first time iFCMs are applied for image classification. Further novel contributions include: a feature extraction process focusing on the most informative image regions; a learning algorithm for data-driven determination of the intuitionistic fuzzy interconnections of the iFCM; an inherently interpretable classification approach based on image contents. In the context of image classification, hesitancy is considered as a degree of inconfidence with which an image is categorized to a class. The constructed iFCM model distinguishes the most representative image semantics and analyses them utilizing cause-and-effect relations. The effectiveness of the introduced framework is evaluated on publicly available datasets, and the experimental results confirm that it can provide enhanced classification performance, while providing interpretable inferences.

Title: Flexible Bayesian Last Layer Models Using Implicit Priors and Diffusion Posterior Sampling

• Authors: Jian Xu, Zhiqi Lin, Shigui Li, Min Chen, Junmei Yang, Delu Zeng, John Paisley
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.03746
• Pdf URL: https://arxiv.org/pdf/2408.03746
• Copy Paste: [[2408.03746]] Flexible Bayesian Last Layer Models Using Implicit Priors and Diffusion Posterior Sampling(https://arxiv.org/abs/2408.03746)
• Keywords: diffusion
• Abstract: Bayesian Last Layer (BLL) models focus solely on uncertainty in the output layer of neural networks, demonstrating comparable performance to more complex Bayesian models. However, the use of Gaussian priors for last layer weights in Bayesian Last Layer (BLL) models limits their expressive capacity when faced with non-Gaussian, outlier-rich, or high-dimensional datasets. To address this shortfall, we introduce a novel approach that combines diffusion techniques and implicit priors for variational learning of Bayesian last layer weights. This method leverages implicit distributions for modeling weight priors in BLL, coupled with diffusion samplers for approximating true posterior predictions, thereby establishing a comprehensive Bayesian prior and posterior estimation strategy. By delivering an explicit and computationally efficient variational lower bound, our method aims to augment the expressive abilities of BLL models, enhancing model accuracy, calibration, and out-of-distribution detection proficiency. Through detailed exploration and experimental validation, We showcase the method's potential for improving predictive accuracy and uncertainty quantification while ensuring computational efficiency.

Title: Data Generation Scheme for Thermal Modality with Edge-Guided Adversarial Conditional Diffusion Model

• Authors: Guoqing Zhu, Honghu Pan, Qiang Wang, Chao Tian, Chao Yang, Zhenyu He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03748
• Pdf URL: https://arxiv.org/pdf/2408.03748
• Copy Paste: [[2408.03748]] Data Generation Scheme for Thermal Modality with Edge-Guided Adversarial Conditional Diffusion Model(https://arxiv.org/abs/2408.03748)
• Keywords: diffusion
• Abstract: In challenging low light and adverse weather conditions,thermal vision algorithms,especially object detection,have exhibited remarkable potential,contrasting with the frequent struggles encountered by visible vision algorithms. Nevertheless,the efficacy of thermal vision algorithms driven by deep learning models remains constrained by the paucity of available training data samples. To this end,this paper introduces a novel approach termed the edge guided conditional diffusion model. This framework aims to produce meticulously aligned pseudo thermal images at the pixel level,leveraging edge information extracted from visible images. By utilizing edges as contextual cues from the visible domain,the diffusion model achieves meticulous control over the delineation of objects within the generated images. To alleviate the impacts of those visible-specific edge information that should not appear in the thermal domain,a two-stage modality adversarial training strategy is proposed to filter them out from the generated images by differentiating the visible and thermal modality. Extensive experiments on LLVIP demonstrate ECDM s superiority over existing state-of-the-art approaches in terms of image generation quality.

Title: MTDSense: AI-Based Fingerprinting of Moving Target Defense Techniques in Software-Defined Networking

• Authors: Tina Moghaddam, Guowei Yang, Chandra Thapa, Seyit Camtepe, Dan Dongseong Kim
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.03758
• Pdf URL: https://arxiv.org/pdf/2408.03758
• Copy Paste: [[2408.03758]] MTDSense: AI-Based Fingerprinting of Moving Target Defense Techniques in Software-Defined Networking(https://arxiv.org/abs/2408.03758)
• Keywords: security, defense, attack
• Abstract: Moving target defenses (MTD) are proactive security techniques that enhance network security by confusing the attacker and limiting their attack window. MTDs have been shown to have significant benefits when evaluated against traditional network attacks, most of which are automated and untargeted. However, little has been done to address an attacker who is aware the network uses an MTD. In this work, we propose a novel approach named MTDSense, which can determine when the MTD has been triggered using the footprints the MTD operation leaves in the network traffic. MTDSense uses unsupervised clustering to identify traffic following an MTD trigger and extract the MTD interval. An attacker can use this information to maximize their attack window and tailor their attacks, which has been shown to significantly reduce the effectiveness of MTD. Through analyzing the attacker's approach, we propose and evaluate two new MTD update algorithms that aim to reduce the information leaked into the network by the MTD. We present an extensive experimental evaluation by creating, to our knowledge, the first dataset of the operation of an IP-shuffling MTD in a software-defined network. Our work reveals that despite previous results showing the effectiveness of MTD as a defense, traditional implementations of MTD are highly susceptible to a targeted attacker.

Title: MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video

• Authors: Xiaoqing Guo, Qianhui Men, J. Alison Noble
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03761
• Pdf URL: https://arxiv.org/pdf/2408.03761
• Copy Paste: [[2408.03761]] MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video(https://arxiv.org/abs/2408.03761)
• Keywords: biometric, large language model, segmentation
• Abstract: We present the first automated multimodal summary generation system, MMSummary, for medical imaging video, particularly with a focus on fetal ultrasound analysis. Imitating the examination process performed by a human sonographer, MMSummary is designed as a three-stage pipeline, progressing from keyframe detection to keyframe captioning and finally anatomy segmentation and measurement. In the keyframe detection stage, an innovative automated workflow is proposed to progressively select a concise set of keyframes, preserving sufficient video information without redundancy. Subsequently, we adapt a large language model to generate meaningful captions for fetal ultrasound keyframes in the keyframe captioning stage. If a keyframe is captioned as fetal biometry, the segmentation and measurement stage estimates biometric parameters by segmenting the region of interest according to the textual prior. The MMSummary system provides comprehensive summaries for fetal ultrasound examinations and based on reported experiments is estimated to reduce scanning time by approximately 31.5%, thereby suggesting the potential to enhance clinical workflow efficiency.

Title: Methodological Explainability Evaluation of an Interpretable Deep Learning Model for Post-Hepatectomy Liver Failure Prediction Incorporating Counterfactual Explanations and Layerwise Relevance Propagation: A Prospective In Silico Trial

• Authors: Xian Zhong, Zohaib Salahuddin, Yi Chen, Henry C Woodruff, Haiyi Long, Jianyun Peng, Nuwan Udawatte, Roberto Casale, Ayoub Mokhtari, Xiaoer Zhang, Jiayao Huang, Qingyu Wu, Li Tan, Lili Chen, Dongming Li, Xiaoyan Xie, Manxia Lin, Philippe Lambin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03771
• Pdf URL: https://arxiv.org/pdf/2408.03771
• Copy Paste: [[2408.03771]] Methodological Explainability Evaluation of an Interpretable Deep Learning Model for Post-Hepatectomy Liver Failure Prediction Incorporating Counterfactual Explanations and Layerwise Relevance Propagation: A Prospective In Silico Trial(https://arxiv.org/abs/2408.03771)
• Keywords: explainability
• Abstract: Artificial intelligence (AI)-based decision support systems have demonstrated value in predicting post-hepatectomy liver failure (PHLF) in hepatocellular carcinoma (HCC). However, they often lack transparency, and the impact of model explanations on clinicians' decisions has not been thoroughly evaluated. Building on prior research, we developed a variational autoencoder-multilayer perceptron (VAE-MLP) model for preoperative PHLF prediction. This model integrated counterfactuals and layerwise relevance propagation (LRP) to provide insights into its decision-making mechanism. Additionally, we proposed a methodological framework for evaluating the explainability of AI systems. This framework includes qualitative and quantitative assessments of explanations against recognized biomarkers, usability evaluations, and an in silico clinical trial. Our evaluations demonstrated that the model's explanation correlated with established biomarkers and exhibited high usability at both the case and system levels. Furthermore, results from the three-track in silico clinical trial showed that clinicians' prediction accuracy and confidence increased when AI explanations were provided.

Title: Generative Language Models with Retrieval Augmented Generation for Automated Short Answer Scoring

• Authors: Zifan Wang, Christopher Ormerod
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.03811
• Pdf URL: https://arxiv.org/pdf/2408.03811
• Copy Paste: [[2408.03811]] Generative Language Models with Retrieval Augmented Generation for Automated Short Answer Scoring(https://arxiv.org/abs/2408.03811)
• Keywords: transformer, generative
• Abstract: Automated Short Answer Scoring (ASAS) is a critical component in educational assessment. While traditional ASAS systems relied on rule-based algorithms or complex deep learning methods, recent advancements in Generative Language Models (GLMs) offer new opportunities for improvement. This study explores the application of GLMs to ASAS, leveraging their off-the-shelf capabilities and performance in various domains. We propose a novel pipeline that combines vector databases, transformer-based encoders, and GLMs to enhance short answer scoring accuracy. Our approach stores training responses in a vector database, retrieves semantically similar responses during inference, and employs a GLM to analyze these responses and determine appropriate scores. We further optimize the system through fine-tuned retrieval processes and prompt engineering. Evaluation on the SemEval 2013 dataset demonstrates a significant improvement on the SCIENTSBANK 3-way and 2-way tasks compared to existing methods, highlighting the potential of GLMs in advancing ASAS technology.

Title: Early Prediction of Causes (not Effects) in Healthcare by Long-Term Clinical Time Series Forecasting

• Authors: Michael Staniek, Marius Fracarolli, Michael Hagmann, Stefan Riezler
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03816
• Pdf URL: https://arxiv.org/pdf/2408.03816
• Copy Paste: [[2408.03816]] Early Prediction of Causes (not Effects) in Healthcare by Long-Term Clinical Time Series Forecasting(https://arxiv.org/abs/2408.03816)
• Keywords: transformer
• Abstract: Machine learning for early syndrome diagnosis aims to solve the intricate task of predicting a ground truth label that most often is the outcome (effect) of a medical consensus definition applied to observed clinical measurements (causes), given clinical measurements observed several hours before. Instead of focusing on the prediction of the future effect, we propose to directly predict the causes via time series forecasting (TSF) of clinical variables and determine the effect by applying the gold standard consensus definition to the forecasted values. This method has the invaluable advantage of being straightforwardly interpretable to clinical practitioners, and because model training does not rely on a particular label anymore, the forecasted data can be used to predict any consensus-based label. We exemplify our method by means of long-term TSF with Transformer models, with a focus on accurate prediction of sparse clinical variables involved in the SOFA-based Sepsis-3 definition and the new Simplified Acute Physiology Score (SAPS-II) definition. Our experiments are conducted on two datasets and show that contrary to recent proposals which advocate set function encoders for time series and direct multi-step decoders, best results are achieved by a combination of standard dense encoders with iterative multi-step decoders. The key for success of iterative multi-step decoding can be attributed to its ability to capture cross-variate dependencies and to a student forcing training strategy that teaches the model to rely on its own previous time step predictions for the next time step prediction.

Title: Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning

• Authors: Simret Araya Gebreegziabher, Kuangshi Ai, Zheng Zhang, Elena L. Glassman, Toby Jia-Jun Li
• Subjects: cs.LG, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.03819
• Pdf URL: https://arxiv.org/pdf/2408.03819
• Copy Paste: [[2408.03819]] Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning(https://arxiv.org/abs/2408.03819)
• Keywords: large language model
• Abstract: Active Learning (AL) allows models to learn interactively from user feedback. This paper introduces a counterfactual data augmentation approach to AL, particularly addressing the selection of datapoints for user querying, a pivotal concern in enhancing data efficiency. Our approach is inspired by Variation Theory, a theory of human concept learning that emphasizes the essential features of a concept by focusing on what stays the same and what changes. Instead of just querying with existing datapoints, our approach synthesizes artificial datapoints that highlight potential key similarities and differences among labels using a neuro-symbolic pipeline combining large language models (LLMs) and rule-based models. Through an experiment in the example domain of text classification, we show that our approach achieves significantly higher performance when there are fewer annotated data. As the annotated training data gets larger the impact of the generated data starts to diminish showing its capability to address the cold start problem in AL. This research sheds light on integrating theories of human learning into the optimization of AL.

Title: Target Prompting for Information Extraction with Vision Language Model

• Authors: Dipankar Medhi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03834
• Pdf URL: https://arxiv.org/pdf/2408.03834
• Copy Paste: [[2408.03834]] Target Prompting for Information Extraction with Vision Language Model(https://arxiv.org/abs/2408.03834)
• Keywords: extraction, large language model
• Abstract: The recent trend in the Large Vision and Language model has brought a new change in how information extraction systems are built. VLMs have set a new benchmark with their State-of-the-art techniques in understanding documents and building question-answering systems across various industries. They are significantly better at generating text from document images and providing accurate answers to questions. However, there are still some challenges in effectively utilizing these models to build a precise conversational system. General prompting techniques used with large language models are often not suitable for these specially designed vision language models. The output generated by such generic input prompts is ordinary and may contain information gaps when compared with the actual content of the document. To obtain more accurate and specific answers, a well-targeted prompt is required by the vision language model, along with the document image. In this paper, a technique is discussed called Target prompting, which focuses on explicitly targeting parts of document images and generating related answers from those specific regions only. The paper also covers the evaluation of response for each prompting technique using different user queries and input prompts.

Title: WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models

• Authors: Prannaya Gupta, Le Qi Yau, Hao Han Low, I-Shiang Lee, Hugo Maximus Lim, Yu Xin Teoh, Jia Hng Koh, Dar Win Liew, Rishabh Bhardwaj, Rajat Bhardwaj, Soujanya Poria
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03837
• Pdf URL: https://arxiv.org/pdf/2408.03837
• Copy Paste: [[2408.03837]] WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models(https://arxiv.org/abs/2408.03837)
• Keywords: large language model
• Abstract: WalledEval is a comprehensive AI safety testing toolkit designed to evaluate large language models (LLMs). It accommodates a diverse range of models, including both open-weight and API-based ones, and features over 35 safety benchmarks covering areas such as multilingual safety, exaggerated safety, and prompt injections. The framework supports both LLM and judge benchmarking, and incorporates custom mutators to test safety against various text-style mutations such as future tense and paraphrasing. Additionally, WalledEval introduces WalledGuard, a new, small and performant content moderation tool, and SGXSTest, a benchmark for assessing exaggerated safety in cultural contexts. We make WalledEval publicly available at this https URL.

Title: Bi-Level Spatial and Channel-aware Transformer for Learned Image Compression

• Authors: Hamidreza Soltani, Erfan Ghasemi
• Subjects: cs.CV, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03842
• Pdf URL: https://arxiv.org/pdf/2408.03842
• Copy Paste: [[2408.03842]] Bi-Level Spatial and Channel-aware Transformer for Learned Image Compression(https://arxiv.org/abs/2408.03842)
• Keywords: extraction, transformer
• Abstract: Recent advancements in learned image compression (LIC) methods have demonstrated superior performance over traditional hand-crafted codecs. These learning-based methods often employ convolutional neural networks (CNNs) or Transformer-based architectures. However, these nonlinear approaches frequently overlook the frequency characteristics of images, which limits their compression efficiency. To address this issue, we propose a novel Transformer-based image compression method that enhances the transformation stage by considering frequency components within the feature map. Our method integrates a novel Hybrid Spatial-Channel Attention Transformer Block (HSCATB), where a spatial-based branch independently handles high and low frequencies at the attention layer, and a Channel-aware Self-Attention (CaSA) module captures information across channels, significantly improving compression performance. Additionally, we introduce a Mixed Local-Global Feed Forward Network (MLGFFN) within the Transformer block to enhance the extraction of diverse and rich information, which is crucial for effective compression. These innovations collectively improve the transformation's ability to project data into a more decorrelated latent space, thereby boosting overall compression efficiency. Experimental results demonstrate that our framework surpasses state-of-the-art LIC methods in rate-distortion performance.

Title: Why transformers are obviously good models of language

• Authors: Felix Hill
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03855
• Pdf URL: https://arxiv.org/pdf/2408.03855
• Copy Paste: [[2408.03855]] Why transformers are obviously good models of language(https://arxiv.org/abs/2408.03855)
• Keywords: transformer
• Abstract: Nobody knows how language works, but many theories abound. Transformers are a class of neural networks that process language automatically with more success than alternatives, both those based on neural computations and those that rely on other (e.g. more symbolic) mechanisms. Here, I highlight direct connections between the transformer architecture and certain theoretical perspectives on language. The empirical success of transformers relative to alternative models provides circumstantial evidence that the linguistic approaches that transformers embody should be, at least, evaluated with greater scrutiny by the linguistics community and, at best, considered to be the currently best available theories.

Title: PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training

• Authors: Haoran Xu, Ziqian Liu, Rong Fu, Zhongling Su, Zerui Wang, Zheng Cai, Zhilin Pei, Xingcheng Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03865
• Pdf URL: https://arxiv.org/pdf/2408.03865
• Copy Paste: [[2408.03865]] PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training(https://arxiv.org/abs/2408.03865)
• Keywords: transformer, generative, large language model
• Abstract: With the evolution of large language models, traditional Transformer models become computationally demanding for lengthy sequences due to the quadratic growth in computation with respect to the sequence length. Mamba, emerging as a groundbreaking architecture in the field of generative AI, demonstrates remarkable proficiency in handling elongated sequences with reduced computational and memory complexity. Nevertheless, the existing training framework of Mamba presents inefficiency with variable-length sequence inputs. Either single-sequence training results in low GPU utilization, or batched processing of variable-length sequences to a maximum length incurs considerable memory and computational overhead. To address this problem, we analyze the performance of bottleneck operators in Mamba under diverse tensor shapes and proposed PackMamba, a high-throughput Mamba that efficiently handles variable-length sequences. Diving deep into state-space models (SSMs), we modify the parallel operators to avoid passing information between individual sequences while maintaining high performance. Experimental results on an NVIDIA A100 GPU demonstrate throughput exceeding the baseline single-sequence processing scheme: 3.06x speedup on the 1.4B model and 2.62x on the 2.8B model.

Title: Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition

• Authors: Shu Yang, Luyang Luo, Qiong Wang, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03867
• Pdf URL: https://arxiv.org/pdf/2408.03867
• Copy Paste: [[2408.03867]] Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition(https://arxiv.org/abs/2408.03867)
• Keywords: extraction, transformer
• Abstract: Existing state-of-the-art methods for surgical phase recognition either rely on the extraction of spatial-temporal features at a short-range temporal resolution or adopt the sequential extraction of the spatial and temporal features across the entire temporal resolution. However, these methods have limitations in modeling spatial-temporal dependency and addressing spatial-temporal redundancy: 1) These methods fail to effectively model spatial-temporal dependency, due to the lack of long-range information or joint spatial-temporal modeling. 2) These methods utilize dense spatial features across the entire temporal resolution, resulting in significant spatial-temporal redundancy. In this paper, we propose the Surgical Transformer (Surgformer) to address the issues of spatial-temporal modeling and redundancy in an end-to-end manner, which employs divided spatial-temporal attention and takes a limited set of sparse frames as input. Moreover, we propose a novel Hierarchical Temporal Attention (HTA) to capture both global and local information within varied temporal resolutions from a target frame-centric perspective. Distinct from conventional temporal attention that primarily emphasizes dense long-range similarity, HTA not only captures long-term information but also considers local latent consistency among informative frames. HTA then employs pyramid feature aggregation to effectively utilize temporal information across diverse temporal resolutions, thereby enhancing the overall temporal representation. Extensive experiments on two challenging benchmark datasets verify that our proposed Surgformer performs favorably against the state-of-the-art methods. The code is released at this https URL.

Title: Inter-Series Transformer: Attending to Products in Time Series Forecasting

• Authors: Rares Cristian, Pavithra Harsha, Clemente Ocejo, Georgia Perakis, Brian Quanz, Ioannis Spantidakis, Hamza Zerhouni
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03872
• Pdf URL: https://arxiv.org/pdf/2408.03872
• Copy Paste: [[2408.03872]] Inter-Series Transformer: Attending to Products in Time Series Forecasting(https://arxiv.org/abs/2408.03872)
• Keywords: transformer
• Abstract: Time series forecasting is an important task in many fields ranging from supply chain management to weather forecasting. Recently, Transformer neural network architectures have shown promising results in forecasting on common time series benchmark datasets. However, application to supply chain demand forecasting, which can have challenging characteristics such as sparsity and cross-series effects, has been limited. In this work, we explore the application of Transformer-based models to supply chain demand forecasting. In particular, we develop a new Transformer-based forecasting approach using a shared, multi-task per-time series network with an initial component applying attention across time series, to capture interactions and help address sparsity. We provide a case study applying our approach to successfully improve demand prediction for a medical device manufacturing company. To further validate our approach, we also apply it to public demand forecasting datasets as well and demonstrate competitive to superior performance compared to a variety of baseline and state-of-the-art forecast methods across the private and public datasets.

Title: Global-Local Progressive Integration Network for Blind Image Quality Assessment

• Authors: Xiaoqi Wang, Yun Zhang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.03885
• Pdf URL: https://arxiv.org/pdf/2408.03885
• Copy Paste: [[2408.03885]] Global-Local Progressive Integration Network for Blind Image Quality Assessment(https://arxiv.org/abs/2408.03885)
• Keywords: extraction, transformer
• Abstract: Vision transformers (ViTs) excel in computer vision for modeling long-term dependencies, yet face two key challenges for image quality assessment (IQA): discarding fine details during patch embedding, and requiring extensive training data due to lack of inductive biases. In this study, we propose a Global-Local progressive INTegration network for IQA, called GlintIQA, to address these issues through three key components: 1) Hybrid feature extraction combines ViT-based global feature extractor (VGFE) and convolutional neural networks (CNNs)-based local feature extractor (CLFE) to capture global coarse-grained features and local fine-grained features, respectively. The incorporation of CNNs mitigates the patch-level information loss and inductive bias constraints inherent to ViT architectures. 2) Progressive feature integration leverages diverse kernel sizes in embedding to spatially align coarse- and fine-grained features, and progressively aggregate these features by interactively stacking channel-wise attention and spatial enhancement modules to build effective quality-aware representations. 3) Content similarity-based labeling approach is proposed that automatically assigns quality labels to images with diverse content based on subjective quality scores. This addresses the scarcity of labeled training data in synthetic datasets and bolsters model generalization. The experimental results demonstrate the efficacy of our approach, yielding 5.04% average SROCC gains on cross-authentic dataset evaluations. Moreover, our model and its counterpart pre-trained on the proposed dataset respectively exhibited 5.40% and 13.23% improvements on across-synthetic datasets evaluation. The codes and proposed dataset will be released at this https URL.

Title: Dual-Modeling Decouple Distillation for Unsupervised Anomaly Detection

• Authors: Xinyue Liu, Jianyuan Wang, Biao Leng, Shuo Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03888
• Pdf URL: https://arxiv.org/pdf/2408.03888
• Copy Paste: [[2408.03888]] Dual-Modeling Decouple Distillation for Unsupervised Anomaly Detection(https://arxiv.org/abs/2408.03888)
• Keywords: segmentation
• Abstract: Knowledge distillation based on student-teacher network is one of the mainstream solution paradigms for the challenging unsupervised Anomaly Detection task, utilizing the difference in representation capabilities of the teacher and student networks to implement anomaly localization. However, over-generalization of the student network to the teacher network may lead to negligible differences in representation capabilities of anomaly, thus affecting the detection effectiveness. Existing methods address the possible over-generalization by using differentiated students and teachers from the structural perspective or explicitly expanding distilled information from the content perspective, which inevitably result in an increased likelihood of underfitting of the student network and poor anomaly detection capabilities in anomaly center or edge. In this paper, we propose Dual-Modeling Decouple Distillation (DMDD) for the unsupervised anomaly detection. In DMDD, a Decouple Student-Teacher Network is proposed to decouple the initial student features into normality and abnormality features. We further introduce Dual-Modeling Distillation based on normal-anomaly image pairs, fitting normality features of anomalous image and the teacher features of the corresponding normal image, widening the distance between abnormality features and the teacher features in anomalous regions. Synthesizing these two distillation ideas, we achieve anomaly detection which focuses on both edge and center of anomaly. Finally, a Multi-perception Segmentation Network is proposed to achieve focused anomaly map fusion based on multiple attention. Experimental results on MVTec AD show that DMDD surpasses SOTA localization performance of previous knowledge distillation-based methods, reaching 98.85% on pixel-level AUC and 96.13% on PRO.

Title: Simplifying Scholarly Abstracts for Accessible Digital Libraries

• Authors: Haining Wang, Jason Clark
• Subjects: cs.CL, cs.AI, cs.CY, cs.DL
• Abstract URL: https://arxiv.org/abs/2408.03899
• Pdf URL: https://arxiv.org/pdf/2408.03899
• Copy Paste: [[2408.03899]] Simplifying Scholarly Abstracts for Accessible Digital Libraries(https://arxiv.org/abs/2408.03899)
• Keywords: privacy
• Abstract: Standing at the forefront of knowledge dissemination, digital libraries curate vast collections of scientific literature. However, these scholarly writings are often laden with jargon and tailored for domain experts rather than the general public. As librarians, we strive to offer services to a diverse audience, including those with lower reading levels. To extend our services beyond mere access, we propose fine-tuning a language model to rewrite scholarly abstracts into more comprehensible versions, thereby making scholarly literature more accessible when requested. We began by introducing a corpus specifically designed for training models to simplify scholarly abstracts. This corpus consists of over three thousand pairs of abstracts and significance statements from diverse disciplines. We then fine-tuned four language models using this corpus. The outputs from the models were subsequently examined both quantitatively for accessibility and semantic coherence, and qualitatively for language quality, faithfulness, and completeness. Our findings show that the resulting models can improve readability by over three grade levels, while maintaining fidelity to the original content. Although commercial state-of-the-art models still hold an edge, our models are much more compact, can be deployed locally in an affordable manner, and alleviate the privacy concerns associated with using commercial models. We envision this work as a step toward more inclusive and accessible libraries, improving our services for young readers and those without a college degree.

Title: Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models

• Authors: Shachi H Kumar, Saurav Sahay, Sahisnu Mazumder, Eda Okur, Ramesh Manuvinakurike, Nicole Beckage, Hsuan Su, Hung-yi Lee, Lama Nachman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03907
• Pdf URL: https://arxiv.org/pdf/2408.03907
• Copy Paste: [[2408.03907]] Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models(https://arxiv.org/abs/2408.03907)
• Keywords: attack, large language model
• Abstract: Large Language Models (LLMs) have excelled at language understanding and generating human-level text. However, even with supervised training and human alignment, these LLMs are susceptible to adversarial attacks where malicious users can prompt the model to generate undesirable text. LLMs also inherently encode potential biases that can cause various harmful effects during interactions. Bias evaluation metrics lack standards as well as consensus and existing methods often rely on human-generated templates and annotations which are expensive and labor intensive. In this work, we train models to automatically create adversarial prompts to elicit biased responses from target LLMs. We present LLM- based bias evaluation metrics and also analyze several existing automatic evaluation methods and metrics. We analyze the various nuances of model responses, identify the strengths and weaknesses of model families, and assess where evaluation methods fall short. We compare these metrics to human evaluation and validate that the LLM-as-a-Judge metric aligns with human judgement on bias in response generation.

Title: LaFA: Latent Feature Attacks on Non-negative Matrix Factorization

• Authors: Minh Vu, Ben Nebgen, Erik Skau, Geigh Zollicoffer, Juan Castorena, Kim Rasmussen, Boian Alexandrov, Manish Bhattarai
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.03909
• Pdf URL: https://arxiv.org/pdf/2408.03909
• Copy Paste: [[2408.03909]] LaFA: Latent Feature Attacks on Non-negative Matrix Factorization(https://arxiv.org/abs/2408.03909)
• Keywords: attack
• Abstract: As Machine Learning (ML) applications rapidly grow, concerns about adversarial attacks compromising their reliability have gained significant attention. One unsupervised ML method known for its resilience to such attacks is Non-negative Matrix Factorization (NMF), an algorithm that decomposes input data into lower-dimensional latent features. However, the introduction of powerful computational tools such as Pytorch enables the computation of gradients of the latent features with respect to the original data, raising concerns about NMF's reliability. Interestingly, naively deriving the adversarial loss for NMF as in the case of ML would result in the reconstruction loss, which can be shown theoretically to be an ineffective attacking objective. In this work, we introduce a novel class of attacks in NMF termed Latent Feature Attacks (LaFA), which aim to manipulate the latent features produced by the NMF process. Our method utilizes the Feature Error (FE) loss directly on the latent features. By employing FE loss, we generate perturbations in the original data that significantly affect the extracted latent features, revealing vulnerabilities akin to those found in other ML techniques. To handle large peak-memory overhead from gradient back-propagation in FE attacks, we develop a method based on implicit differentiation which enables their scaling to larger datasets. We validate NMF vulnerabilities and FE attacks effectiveness through extensive experiments on synthetic and real-world data.

Title: AdapMTL: Adaptive Pruning Framework for Multitask Learning Model

• Authors: Mingcan Xiang, Steven Jiaxun Tang, Qizheng Yang, Hui Guan, Tongping Liu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03913
• Pdf URL: https://arxiv.org/pdf/2408.03913
• Copy Paste: [[2408.03913]] AdapMTL: Adaptive Pruning Framework for Multitask Learning Model(https://arxiv.org/abs/2408.03913)
• Keywords: robust
• Abstract: In the domain of multimedia and multimodal processing, the efficient handling of diverse data streams such as images, video, and sensor data is paramount. Model compression and multitask learning (MTL) are crucial in this field, offering the potential to address the resource-intensive demands of processing and interpreting multiple forms of media simultaneously. However, effectively compressing a multitask model presents significant challenges due to the complexities of balancing sparsity allocation and accuracy performance across multiple tasks. To tackle these challenges, we propose AdapMTL, an adaptive pruning framework for MTL models. AdapMTL leverages multiple learnable soft thresholds independently assigned to the shared backbone and the task-specific heads to capture the nuances in different components' sensitivity to pruning. During training, it co-optimizes the soft thresholds and MTL model weights to automatically determine the suitable sparsity level at each component to achieve both high task accuracy and high overall sparsity. It further incorporates an adaptive weighting mechanism that dynamically adjusts the importance of task-specific losses based on each task's robustness to pruning. We demonstrate the effectiveness of AdapMTL through comprehensive experiments on popular multitask datasets, namely NYU-v2 and Tiny-Taskonomy, with different architectures, showcasing superior performance compared to state-of-the-art pruning methods.

Title: Hard to Explain: On the Computational Hardness of In-Distribution Model Interpretation

• Authors: Guy Amir, Shahaf Bassan, Guy Katz
• Subjects: cs.LG, cs.CC, cs.LO
• Abstract URL: https://arxiv.org/abs/2408.03915
• Pdf URL: https://arxiv.org/pdf/2408.03915
• Copy Paste: [[2408.03915]] Hard to Explain: On the Computational Hardness of In-Distribution Model Interpretation(https://arxiv.org/abs/2408.03915)
• Keywords: interpretability
• Abstract: The ability to interpret Machine Learning (ML) models is becoming increasingly essential. However, despite significant progress in the field, there remains a lack of rigorous characterization regarding the innate interpretability of different models. In an attempt to bridge this gap, recent work has demonstrated that it is possible to formally assess interpretability by studying the computational complexity of explaining the decisions of various models. In this setting, if explanations for a particular model can be obtained efficiently, the model is considered interpretable (since it can be explained ``easily''). However, if generating explanations over an ML model is computationally intractable, it is considered uninterpretable. Prior research identified two key factors that influence the complexity of interpreting an ML model: (i) the type of the model (e.g., neural networks, decision trees, etc.); and (ii) the form of explanation (e.g., contrastive explanations, Shapley values, etc.). In this work, we claim that a third, important factor must also be considered for this analysis -- the underlying distribution over which the explanation is obtained. Considering the underlying distribution is key in avoiding explanations that are socially misaligned, i.e., convey information that is biased and unhelpful to users. We demonstrate the significant influence of the underlying distribution on the resulting overall interpretation complexity, in two settings: (i) prediction models paired with an external out-of-distribution (OOD) detector; and (ii) prediction models designed to inherently generate socially aligned explanations. Our findings prove that the expressiveness of the distribution can significantly influence the overall complexity of interpretation, and identify essential prerequisites that a model must possess to generate socially aligned explanations.

Title: Fast Sprite Decomposition from Animated Graphics

• Authors: Tomoyuki Suzuki, Kotaro Kikuchi, Kota Yamaguchi
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.03923
• Pdf URL: https://arxiv.org/pdf/2408.03923
• Copy Paste: [[2408.03923]] Fast Sprite Decomposition from Animated Graphics(https://arxiv.org/abs/2408.03923)
• Keywords: segmentation
• Abstract: This paper presents an approach to decomposing animated graphics into sprites, a set of basic elements or layers. Our approach builds on the optimization of sprite parameters to fit the raster video. For efficiency, we assume static textures for sprites to reduce the search space while preventing artifacts using a texture prior model. To further speed up the optimization, we introduce the initialization of the sprite parameters utilizing a pre-trained video object segmentation model and user input of single frame annotations. For our study, we construct the Crello Animation dataset from an online design service and define quantitative metrics to measure the quality of the extracted sprites. Experiments show that our method significantly outperforms baselines for similar decomposition tasks in terms of the quality/efficiency tradeoff.

Title: SLIM-RAFT: A Novel Fine-Tuning Approach to Improve Cross-Linguistic Performance for Mercosur Common Nomenclature

• Authors: Vinícius Di Oliveira, Yuri Façanha Bezerra, Li Weigang, Pedro Carvalho Brom, Victor Rafael R. Celestino
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03936
• Pdf URL: https://arxiv.org/pdf/2408.03936
• Copy Paste: [[2408.03936]] SLIM-RAFT: A Novel Fine-Tuning Approach to Improve Cross-Linguistic Performance for Mercosur Common Nomenclature(https://arxiv.org/abs/2408.03936)
• Keywords: large language model
• Abstract: Natural language processing (NLP) has seen significant advancements with the advent of large language models (LLMs). However, substantial improvements are still needed for languages other than English, especially for specific domains like the applications of Mercosur Common Nomenclature (NCM), a Brazilian Harmonized System (HS). To address this gap, this study uses TeenyTineLLaMA, a foundational Portuguese LLM, as an LLM source to implement the NCM application processing. Additionally, a simplified Retrieval-Augmented Fine-Tuning (RAFT) technique, termed SLIM-RAFT, is proposed for task-specific fine-tuning of LLMs. This approach retains the chain-of-thought (CoT) methodology for prompt development in a more concise and streamlined manner, utilizing brief and focused documents for training. The proposed model demonstrates an efficient and cost-effective alternative for fine-tuning smaller LLMs, significantly outperforming TeenyTineLLaMA and ChatGPT-4 in the same task. Although the research focuses on NCM applications, the methodology can be easily adapted for HS applications worldwide.

Title: How Well Can Vision Language Models See Image Details?

• Authors: Chenhui Gou, Abdulwahab Felemban, Faizan Farooq Khan, Deyao Zhu, Jianfei Cai, Hamid Rezatofighi, Mohamed Elhoseiny
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03940
• Pdf URL: https://arxiv.org/pdf/2408.03940
• Copy Paste: [[2408.03940]] How Well Can Vision Language Models See Image Details?(https://arxiv.org/abs/2408.03940)
• Keywords: large language model, segmentation
• Abstract: Large Language Model-based Vision-Language Models (LLM-based VLMs) have demonstrated impressive results in various vision-language understanding tasks. However, how well these VLMs can see image detail beyond the semantic level remains unclear. In our study, we introduce a pixel value prediction task (PVP) to explore "How Well Can Vision Language Models See Image Details?" and to assist VLMs in perceiving more details. Typically, these models comprise a frozen CLIP visual encoder, a large language model, and a connecting module. After fine-tuning VLMs on the PVP task, we find: 1) existing VLMs struggle to predict precise pixel values by only fine-tuning the connection module and LLM; and 2) prediction precision is significantly improved when the vision encoder is also adapted. Additionally, our research reveals that incorporating pixel value prediction as one of the VLM pre-training tasks and vision encoder adaptation markedly boosts VLM performance on downstream image-language understanding tasks requiring detailed image perception, such as referring image segmentation (with an average +10.19 cIoU improvement) and video game decision making (with average score improvements of +80.34 and +70.54 on two games, respectively).