2024-04-22

Title: A Big Data Analytics System for Predicting Suicidal Ideation in Real-Time Based on Social Media Streaming Data

• Authors: Mohamed A. Allayla, Serkan Ayvaz
• Subjects: cs.LG, cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.12394
• Pdf URL: https://arxiv.org/pdf/2404.12394
• Copy Paste: [[2404.12394]] A Big Data Analytics System for Predicting Suicidal Ideation in Real-Time Based on Social Media Streaming Data(https://arxiv.org/abs/2404.12394)
• Keywords: extraction
• Abstract: Online social media platforms have recently become integral to our society and daily routines. Every day, users worldwide spend a couple of hours on such platforms, expressing their sentiments and emotional state and contacting each other. Analyzing such huge amounts of data from these platforms can provide a clear insight into public sentiments and help detect their mental status. The early identification of these health condition risks may assist in preventing or reducing the number of suicide ideation and potentially saving people's lives. The traditional techniques have become ineffective in processing such streams and large-scale datasets. Therefore, the paper proposed a new methodology based on a big data architecture to predict suicidal ideation from social media content. The proposed approach provides a practical analysis of social media data in two phases: batch processing and real-time streaming prediction. The batch dataset was collected from the Reddit forum and used for model building and training, while streaming big data was extracted using Twitter streaming API and used for real-time prediction. After the raw data was preprocessed, the extracted features were fed to multiple Apache Spark ML classifiers: NB, LR, LinearSVC, DT, RF, and MLP. We conducted various experiments using various feature-extraction techniques with different testing scenarios. The experimental results of the batch processing phase showed that the features extracted of (Unigram + Bigram) + CV-IDF with MLP classifier provided high performance for classifying suicidal ideation, with an accuracy of 93.47%, and then applied for real-time streaming prediction phase.

Title: Optimized Dynamic Mode Decomposition for Reconstruction and Forecasting of Atmospheric Chemistry Data

• Authors: Meghana Velegar, Christoph Keller, J. Nathan Kutz
• Subjects: cs.LG, math.DS, physics.ao-ph, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.12396
• Pdf URL: https://arxiv.org/pdf/2404.12396
• Copy Paste: [[2404.12396]] Optimized Dynamic Mode Decomposition for Reconstruction and Forecasting of Atmospheric Chemistry Data(https://arxiv.org/abs/2404.12396)
• Keywords: interpretability
• Abstract: We introduce the optimized dynamic mode decomposition algorithm for constructing an adaptive and computationally efficient reduced order model and forecasting tool for global atmospheric chemistry dynamics. By exploiting a low-dimensional set of global spatio-temporal modes, interpretable characterizations of the underlying spatial and temporal scales can be computed. Forecasting is also achieved with a linear model that uses a linear superposition of the dominant spatio-temporal features. The DMD method is demonstrated on three months of global chemistry dynamics data, showing its significant performance in computational speed and interpretability. We show that the presented decomposition method successfully extracts known major features of atmospheric chemistry, such as summertime surface pollution and biomass burning activities. Moreover, the DMD algorithm allows for rapid reconstruction of the underlying linear model, which can then easily accommodate non-stationary data and changes in the dynamics.

Title: Efflex: Efficient and Flexible Pipeline for Spatio-Temporal Trajectory Graph Modeling and Representation Learning

• Authors: Ming Cheng, Ziyi Zhou, Bowen Zhang, Ziyu Wang, Jiaqi Gan, Ziang Ren, Weiqi Feng, Yi Lyu, Hefan Zhang, Xingjian Diao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12400
• Pdf URL: https://arxiv.org/pdf/2404.12400
• Copy Paste: [[2404.12400]] Efflex: Efficient and Flexible Pipeline for Spatio-Temporal Trajectory Graph Modeling and Representation Learning(https://arxiv.org/abs/2404.12400)
• Keywords: extraction
• Abstract: In the landscape of spatio-temporal data analytics, effective trajectory representation learning is paramount. To bridge the gap of learning accurate representations with efficient and flexible mechanisms, we introduce Efflex, a comprehensive pipeline for transformative graph modeling and representation learning of the large-volume spatio-temporal trajectories. Efflex pioneers the incorporation of a multi-scale k-nearest neighbors (KNN) algorithm with feature fusion for graph construction, marking a leap in dimensionality reduction techniques by preserving essential data features. Moreover, the groundbreaking graph construction mechanism and the high-performance lightweight GCN increase embedding extraction speed by up to 36 times faster. We further offer Efflex in two versions, Efflex-L for scenarios demanding high accuracy, and Efflex-B for environments requiring swift data processing. Comprehensive experimentation with the Porto and Geolife datasets validates our approach, positioning Efflex as the state-of-the-art in the domain. Such enhancements in speed and accuracy highlight the versatility of Efflex, underscoring its wide-ranging potential for deployment in time-sensitive and computationally constrained applications.

Title: Sup3r: A Semi-Supervised Algorithm for increasing Sparsity, Stability, and Separability in Hierarchy Of Time-Surfaces architectures

• Authors: Marco Rasetto, Himanshu Akolkar
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.12402
• Pdf URL: https://arxiv.org/pdf/2404.12402
• Copy Paste: [[2404.12402]] Sup3r: A Semi-Supervised Algorithm for increasing Sparsity, Stability, and Separability in Hierarchy Of Time-Surfaces architectures(https://arxiv.org/abs/2404.12402)
• Keywords: extraction
• Abstract: The Hierarchy Of Time-Surfaces (HOTS) algorithm, a neuromorphic approach for feature extraction from event data, presents promising capabilities but faces challenges in accuracy and compatibility with neuromorphic hardware. In this paper, we introduce Sup3r, a Semi-Supervised algorithm aimed at addressing these challenges. Sup3r enhances sparsity, stability, and separability in the HOTS networks. It enables end-to-end online training of HOTS networks replacing external classifiers, by leveraging semi-supervised learning. Sup3r learns class-informative patterns, mitigates confounding features, and reduces the number of processed events. Moreover, Sup3r facilitates continual and incremental learning, allowing adaptation to data distribution shifts and learning new tasks without forgetting. Preliminary results on N-MNIST demonstrate that Sup3r achieves comparable accuracy to similarly sized Artificial Neural Networks trained with back-propagation. This work showcases the potential of Sup3r to advance the capabilities of HOTS networks, offering a promising avenue for neuromorphic algorithms in real-world applications.

Title: Group-wise Prompting for Synthetic Tabular Data Generation using Large Language Models

• Authors: Jinhee Kim, Taesung Kim, Jaegul Choo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12404
• Pdf URL: https://arxiv.org/pdf/2404.12404
• Copy Paste: [[2404.12404]] Group-wise Prompting for Synthetic Tabular Data Generation using Large Language Models(https://arxiv.org/abs/2404.12404)
• Keywords: large language model
• Abstract: Generating realistic synthetic tabular data presents a critical challenge in machine learning. This study introduces a simple yet effective method employing Large Language Models (LLMs) tailored to generate synthetic data, specifically addressing data imbalance problems. We propose a novel group-wise prompting method in CSV-style formatting that leverages the in-context learning capabilities of LLMs to produce data that closely adheres to the specified requirements and characteristics of the target dataset. Moreover, our proposed random word replacement strategy significantly improves the handling of monotonous categorical values, enhancing the accuracy and representativeness of the synthetic data. The effectiveness of our method is extensively validated across eight real-world public datasets, achieving state-of-the-art performance in downstream classification and regression tasks while maintaining inter-feature correlations and improving token efficiency over existing approaches. This advancement significantly contributes to addressing the key challenges of machine learning applications, particularly in the context of tabular data generation and handling class imbalance. The source code for our work is available at: https://github.com/seharanul17/synthetic-tabular-LLM

Title: Adaptive Catalyst Discovery Using Multicriteria Bayesian Optimization with Representation Learning

• Authors: Jie Chen, Pengfei Ou, Yuxin Chang, Hengrui Zhang, Xiao-Yan Li, Edward H. Sargent, Wei Chen
• Subjects: cs.LG, cs.CE, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2404.12445
• Pdf URL: https://arxiv.org/pdf/2404.12445
• Copy Paste: [[2404.12445]] Adaptive Catalyst Discovery Using Multicriteria Bayesian Optimization with Representation Learning(https://arxiv.org/abs/2404.12445)
• Keywords: extraction
• Abstract: High-performance catalysts are crucial for sustainable energy conversion and human health. However, the discovery of catalysts faces challenges due to the absence of efficient approaches to navigating vast and high-dimensional structure and composition spaces. In this study, we propose a high-throughput computational catalyst screening approach integrating density functional theory (DFT) and Bayesian Optimization (BO). Within the BO framework, we propose an uncertainty-aware atomistic machine learning model, UPNet, which enables automated representation learning directly from high-dimensional catalyst structures and achieves principled uncertainty quantification. Utilizing a constrained expected improvement acquisition function, our BO framework simultaneously considers multiple evaluation criteria. Using the proposed methods, we explore catalyst discovery for the CO2 reduction reaction. The results demonstrate that our approach achieves high prediction accuracy, facilitates interpretable feature extraction, and enables multicriteria design optimization, leading to significant reduction of computing power and time (10x reduction of required DFT calculations) in high-performance catalyst discovery.

Title: Enhancing AI Diagnostics: Autonomous Lesion Masking via Semi-Supervised Deep Learning

• Authors: Ting-Ruen Wei, Michele Hell, Dang Bich Thuy Le, Aren Vierra, Ran Pang, Mahesh Patel, Young Kang, Yuling Yan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12450
• Pdf URL: https://arxiv.org/pdf/2404.12450
• Copy Paste: [[2404.12450]] Enhancing AI Diagnostics: Autonomous Lesion Masking via Semi-Supervised Deep Learning(https://arxiv.org/abs/2404.12450)
• Keywords: explainability
• Abstract: This study presents an unsupervised domain adaptation method aimed at autonomously generating image masks outlining regions of interest (ROIs) for differentiating breast lesions in breast ultrasound (US) imaging. Our semi-supervised learning approach utilizes a primitive model trained on a small public breast US dataset with true annotations. This model is then iteratively refined for the domain adaptation task, generating pseudo-masks for our private, unannotated breast US dataset. The dataset, twice the size of the public one, exhibits considerable variability in image acquisition perspectives and demographic representation, posing a domain-shift challenge. Unlike typical domain adversarial training, we employ downstream classification outcomes as a benchmark to guide the updating of pseudo-masks in subsequent iterations. We found the classification precision to be highly correlated with the completeness of the generated ROIs, which promotes the explainability of the deep learning classification model. Preliminary findings demonstrate the efficacy and reliability of this approach in streamlining the ROI annotation process, thereby enhancing the classification and localization of breast lesions for more precise and interpretable diagnoses.

Title: NORMAD: A Benchmark for Measuring the Cultural Adaptability of Large Language Models

• Authors: Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, Maarten Sap
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12464
• Pdf URL: https://arxiv.org/pdf/2404.12464
• Copy Paste: [[2404.12464]] NORMAD: A Benchmark for Measuring the Cultural Adaptability of Large Language Models(https://arxiv.org/abs/2404.12464)
• Keywords: large language model
• Abstract: The integration of Large Language Models (LLMs) into various global cultures fundamentally presents a cultural challenge: LLMs must navigate interactions, respect social norms, and avoid transgressing cultural boundaries. However, it is still unclear if LLMs can adapt their outputs to diverse cultural norms. Our study focuses on this aspect. We introduce NormAd, a novel dataset, which includes 2.6k stories that represent social and cultural norms from 75 countries, to assess the ability of LLMs to adapt to different granular levels of socio-cultural contexts such as the country of origin, its associated cultural values, and prevalent social norms. Our study reveals that LLMs struggle with cultural reasoning across all contextual granularities, showing stronger adaptability to English-centric cultures over those from the Global South. Even with explicit social norms, the top-performing model, Mistral-7b-Instruct, achieves only 81.8\% accuracy, lagging behind the 95.6\% achieved by humans. Evaluation on NormAd further reveals that LLMs struggle to adapt to stories involving gift-giving across cultures. Due to inherent agreement or sycophancy biases, LLMs find it considerably easier to assess the social acceptability of stories that adhere to cultural norms than those that deviate from them. Our benchmark measures the cultural adaptability (or lack thereof) of LLMs, emphasizing the potential to make these technologies more equitable and useful for global audiences.

Title: Toward a Quantum Information System Cybersecurity Taxonomy and Testbed: Exploiting a Unique Opportunity for Early Impact

• Authors: Benjamin Blakely, Joaquin Chung, Alec Poczatek, Ryan Syed, Raj Kettimuthu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12465
• Pdf URL: https://arxiv.org/pdf/2404.12465
• Copy Paste: [[2404.12465]] Toward a Quantum Information System Cybersecurity Taxonomy and Testbed: Exploiting a Unique Opportunity for Early Impact(https://arxiv.org/abs/2404.12465)
• Keywords: security
• Abstract: Any human-designed system can potentially be exploited in ways that its designers did not envision, and information systems or networks using quantum components do not escape this reality. We are presented with a unique but quickly waning opportunity to bring cybersecurity concerns to the forefront for quantum information systems before they become widely deployed. The resources and knowledge required to do so, however, may not be common in the cybersecurity community. Yet, a nexus exist. Cybersecurity starts with risk, and there are good taxonomies for security vulnerabilities and impacts in classical systems. In this paper, we propose a preliminary taxonomy for quantum cybersecurity vulnerabilities that accounts for the latest advances in quantum information systems, and must evolve to incorporate well-established cybersecurity principles and methodologies. We envision a testbed environment designed and instrumented with the specific purpose of enabling a broad collaborative community of cybersecurity and quantum information system experts to conduct experimental evaluation of software and hardware security including both physical and virtual quantum components. Furthermore, we envision that such a resource may be available as a user facility to the open science research community.

Title: Towards Multi-modal Transformers in Federated Learning

• Authors: Guangyu Sun, Matias Mendieta, Aritra Dutta, Xin Li, Chen Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12467
• Pdf URL: https://arxiv.org/pdf/2404.12467
• Copy Paste: [[2404.12467]] Towards Multi-modal Transformers in Federated Learning(https://arxiv.org/abs/2404.12467)
• Keywords: privacy, federate, transformer
• Abstract: Multi-modal transformers mark significant progress in different domains, but siloed high-quality data hinders their further improvement. To remedy this, federated learning (FL) has emerged as a promising privacy-preserving paradigm for training models without direct access to the raw data held by different clients. Despite its potential, a considerable research direction regarding the unpaired uni-modal clients and the transformer architecture in FL remains unexplored. To fill this gap, this paper explores a transfer multi-modal federated learning (MFL) scenario within the vision-language domain, where clients possess data of various modalities distributed across different datasets. We systematically evaluate the performance of existing methods when a transformer architecture is utilized and introduce a novel framework called Federated modality complementary and collaboration (FedCola) by addressing the in-modality and cross-modality gaps among clients. Through extensive experiments across various FL settings, FedCola demonstrates superior performance over previous approaches, offering new perspectives on future federated training of multi-modal transformers.

Title: Explainable Deep Learning Models for Dynamic and Online Malware Classification

• Authors: Quincy Card, Daniel Simpson, Kshitiz Aryal, Maanak Gupta, Sheikh Rabiul Islam
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12473
• Pdf URL: https://arxiv.org/pdf/2404.12473
• Copy Paste: [[2404.12473]] Explainable Deep Learning Models for Dynamic and Online Malware Classification(https://arxiv.org/abs/2404.12473)
• Keywords: attack, explainability
• Abstract: In recent years, there has been a significant surge in malware attacks, necessitating more advanced preventive measures and remedial strategies. While several successful AI-based malware classification approaches exist categorized into static, dynamic, or online analysis, most successful AI models lack easily interpretable decisions and explanations for their processes. Our paper aims to delve into explainable malware classification across various execution environments (such as dynamic and online), thoroughly analyzing their respective strengths, weaknesses, and commonalities. To evaluate our approach, we train Feed Forward Neural Networks (FFNN) and Convolutional Neural Networks (CNN) to classify malware based on features obtained from dynamic and online analysis environments. The feature attribution for malware classification is performed by explainability tools, SHAP, LIME and Permutation Importance. We perform a detailed evaluation of the calculated global and local explanations from the experiments, discuss limitations and, ultimately, offer recommendations for achieving a balanced approach.

Title: Advancing Applications of Satellite Photogrammetry: Novel Approaches for Built-up Area Modeling and Natural Environment Monitoring using Stereo/Multi-view Satellite Image-derived 3D Data

• Authors: Shengxi Gui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12487
• Pdf URL: https://arxiv.org/pdf/2404.12487
• Copy Paste: [[2404.12487]] Advancing Applications of Satellite Photogrammetry: Novel Approaches for Built-up Area Modeling and Natural Environment Monitoring using Stereo/Multi-view Satellite Image-derived 3D Data(https://arxiv.org/abs/2404.12487)
• Keywords: segmentation
• Abstract: With the development of remote sensing technology in recent decades, spaceborne sensors with sub-meter and meter spatial resolution (Worldview and PlanetScope) have achieved a considerable image quality to generate 3D geospatial data via a stereo matching pipeline. These achievements have significantly increased the data accessibility in 3D, necessitating adapting these 3D geospatial data to analyze human and natural environments. This dissertation explores several novel approaches based on stereo and multi-view satellite image-derived 3D geospatial data, to deal with remote sensing application issues for built-up area modeling and natural environment monitoring, including building model 3D reconstruction, glacier dynamics tracking, and lake algae monitoring. Specifically, the dissertation introduces four parts of novel approaches that deal with the spatial and temporal challenges with satellite-derived 3D data. The first study advances LoD-2 building modeling from satellite-derived Orthophoto and DSMs with a novel approach employing a model-driven workflow that generates building rectangular 3D geometry models. Secondly, we further enhanced our building reconstruction framework for dense urban areas and non-rectangular purposes, we implemented deep learning for unit-level segmentation and introduced a gradient-based circle reconstruction for circular buildings to develop a polygon composition technique for advanced building LoD2 reconstruction. Our third study utilizes high-spatiotemporal resolution PlanetScope satellite imagery for glacier tracking at 3D level in mid-latitude regions. Finally, we proposed a term as "Algal Behavior Function" to refine the quantification of chlorophyll-a concentrations from satellite imagery in water quality monitoring, addressing algae fluctuations and timing discrepancies between satellite observations and field measurements, thus enhancing the precision of underwater algae volume estimates. Overall, this dissertation demonstrates the extensive potential of satellite photogrammetry applications in addressing urban and environmental challenges. It further showcases innovative analytical methodologies that enhance the applicability of adapting stereo and multi-view very high-resolution satellite-derived 3D data. (See full abstract in the document)

Title: Global Counterfactual Directions

• Authors: Bartlomiej Sobieski, Przemysław Biecek
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12488
• Pdf URL: https://arxiv.org/pdf/2404.12488
• Copy Paste: [[2404.12488]] Global Counterfactual Directions(https://arxiv.org/abs/2404.12488)
• Keywords: diffusion
• Abstract: Despite increasing progress in development of methods for generating visual counterfactual explanations, especially with the recent rise of Denoising Diffusion Probabilistic Models, previous works consider them as an entirely local technique. In this work, we take the first step at globalizing them. Specifically, we discover that the latent space of Diffusion Autoencoders encodes the inference process of a given classifier in the form of global directions. We propose a novel proxy-based approach that discovers two types of these directions with the use of only single image in an entirely black-box manner. Precisely, g-directions allow for flipping the decision of a given classifier on an entire dataset of images, while h-directions further increase the diversity of explanations. We refer to them in general as Global Counterfactual Directions (GCDs). Moreover, we show that GCDs can be naturally combined with Latent Integrated Gradients resulting in a new black-box attribution method, while simultaneously enhancing the understanding of counterfactual explanations. We validate our approach on existing benchmarks and show that it generalizes to real-world use-cases.

Title: GraphER: A Structure-aware Text-to-Graph Model for Entity and Relation Extraction

• Authors: Urchade Zaratiana, Nadi Tomeh, Niama El Khbir, Pierre Holat, Thierry Charnois
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12491
• Pdf URL: https://arxiv.org/pdf/2404.12491
• Copy Paste: [[2404.12491]] GraphER: A Structure-aware Text-to-Graph Model for Entity and Relation Extraction(https://arxiv.org/abs/2404.12491)
• Keywords: extraction
• Abstract: Information extraction (IE) is an important task in Natural Language Processing (NLP), involving the extraction of named entities and their relationships from unstructured text. In this paper, we propose a novel approach to this task by formulating it as graph structure learning (GSL). By formulating IE as GSL, we enhance the model's ability to dynamically refine and optimize the graph structure during the extraction process. This formulation allows for better interaction and structure-informed decisions for entity and relation prediction, in contrast to previous models that have separate or untied predictions for these tasks. When compared against state-of-the-art baselines on joint entity and relation extraction benchmarks, our model, GraphER, achieves competitive results.

Title: EnriCo: Enriched Representation and Globally Constrained Inference for Entity and Relation Extraction

• Authors: Urchade Zaratiana, Nadi Tomeh, Yann Dauxais, Pierre Holat, Thierry Charnois
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12493
• Pdf URL: https://arxiv.org/pdf/2404.12493
• Copy Paste: [[2404.12493]] EnriCo: Enriched Representation and Globally Constrained Inference for Entity and Relation Extraction(https://arxiv.org/abs/2404.12493)
• Keywords: extraction
• Abstract: Joint entity and relation extraction plays a pivotal role in various applications, notably in the construction of knowledge graphs. Despite recent progress, existing approaches often fall short in two key aspects: richness of representation and coherence in output structure. These models often rely on handcrafted heuristics for computing entity and relation representations, potentially leading to loss of crucial information. Furthermore, they disregard task and/or dataset-specific constraints, resulting in output structures that lack coherence. In our work, we introduce EnriCo, which mitigates these shortcomings. Firstly, to foster rich and expressive representation, our model leverage attention mechanisms that allow both entities and relations to dynamically determine the pertinent information required for accurate extraction. Secondly, we introduce a series of decoding algorithms designed to infer the highest scoring solutions while adhering to task and dataset-specific constraints, thus promoting structured and coherent outputs. Our model demonstrates competitive performance compared to baselines when evaluated on Joint IE datasets.

Title: BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models

• Authors: Yu Feng, Ben Zhou, Weidong Lin, Dan Roth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12494
• Pdf URL: https://arxiv.org/pdf/2404.12494
• Copy Paste: [[2404.12494]] BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models(https://arxiv.org/abs/2404.12494)
• Keywords: large language model
• Abstract: Large language models primarily rely on inductive reasoning for decision making. This results in unreliable decisions when applied to real-world tasks that often present incomplete contexts and conditions. Thus, accurate probability estimation and appropriate interpretations are required to enhance decision-making reliability. In this paper, we propose a Bayesian inference framework called BIRD for large language models. BIRD provides controllable and interpretable probability estimation for model decisions, based on abductive factors, LLM entailment, as well as learnable deductive Bayesian modeling. Experiments show that BIRD produces probability estimations that align with human judgments over 65% of the time using open-sourced Llama models, outperforming the state-of-the-art GPT-4 by 35%. We also show that BIRD can be directly used for trustworthy decision making on many real-world applications.

Title: Generalizing Machine Learning Evaluation through the Integration of Shannon Entropy and Rough Set Theory

• Authors: Olga Cherednichenko, Dmytro Chernyshov, Dmytro Sytnikov, Polina Sytnikova
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12511
• Pdf URL: https://arxiv.org/pdf/2404.12511
• Copy Paste: [[2404.12511]] Generalizing Machine Learning Evaluation through the Integration of Shannon Entropy and Rough Set Theory(https://arxiv.org/abs/2404.12511)
• Keywords: robust, interpretability
• Abstract: This research paper delves into the innovative integration of Shannon entropy and rough set theory, presenting a novel approach to generalize the evaluation approach in machine learning. The conventional application of entropy, primarily focused on information uncertainty, is extended through its combination with rough set theory to offer a deeper insight into data's intrinsic structure and the interpretability of machine learning models. We introduce a comprehensive framework that synergizes the granularity of rough set theory with the uncertainty quantification of Shannon entropy, applied across a spectrum of machine learning algorithms. Our methodology is rigorously tested on various datasets, showcasing its capability to not only assess predictive performance but also to illuminate the underlying data complexity and model robustness. The results underscore the utility of this integrated approach in enhancing the evaluation landscape of machine learning, offering a multi-faceted perspective that balances accuracy with a profound understanding of data attributes and model dynamics. This paper contributes a groundbreaking perspective to machine learning evaluation, proposing a method that encapsulates a holistic view of model performance, thereby facilitating more informed decision-making in model selection and application.

Title: Proteus: Preserving Model Confidentiality during Graph Optimizations

• Authors: Yubo Gao, Maryam Haghifam, Christina Giannoula, Renbo Tu, Gennady Pekhimenko, Nandita Vijaykumar
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12512
• Pdf URL: https://arxiv.org/pdf/2404.12512
• Copy Paste: [[2404.12512]] Proteus: Preserving Model Confidentiality during Graph Optimizations(https://arxiv.org/abs/2404.12512)
• Keywords: protect, attack, steal
• Abstract: Deep learning (DL) models have revolutionized numerous domains, yet optimizing them for computational efficiency remains a challenging endeavor. Development of new DL models typically involves two parties: the model developers and performance optimizers. The collaboration between the parties often necessitates the model developers exposing the model architecture and computational graph to the optimizers. However, this exposure is undesirable since the model architecture is an important intellectual property, and its innovations require significant investments and expertise. During the exchange, the model is also vulnerable to adversarial attacks via model stealing. This paper presents Proteus, a novel mechanism that enables model optimization by an independent party while preserving the confidentiality of the model architecture. Proteus obfuscates the protected model by partitioning its computational graph into subgraphs and concealing each subgraph within a large pool of generated realistic subgraphs that cannot be easily distinguished from the original. We evaluate Proteus on a range of DNNs, demonstrating its efficacy in preserving confidentiality without compromising performance optimization opportunities. Proteus effectively hides the model as one alternative among up to $10^{32}$ possible model architectures, and is resilient against attacks with a learning-based adversary. We also demonstrate that heuristic based and manual approaches are ineffective in identifying the protected model. To our knowledge, Proteus is the first work that tackles the challenge of model confidentiality during performance optimization. Proteus will be open-sourced for direct use and experimentation, with easy integration with compilers such as ONNXRuntime.

Title: DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects

• Authors: Dominik Bauer, Zhenjia Xu, Shuran Song
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.12524
• Pdf URL: https://arxiv.org/pdf/2404.12524
• Copy Paste: [[2404.12524]] DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects(https://arxiv.org/abs/2404.12524)
• Keywords: transformer
• Abstract: Manipulation of elastoplastic objects like dough often involves topological changes such as splitting and merging. The ability to accurately predict these topological changes that a specific action might incur is critical for planning interactions with elastoplastic objects. We present DoughNet, a Transformer-based architecture for handling these challenges, consisting of two components. First, a denoising autoencoder represents deformable objects of varying topology as sets of latent codes. Second, a visual predictive model performs autoregressive set prediction to determine long-horizon geometrical deformation and topological changes purely in latent space. Given a partial initial state and desired manipulation trajectories, it infers all resulting object geometries and topologies at each step. DoughNet thereby allows to plan robotic manipulation; selecting a suited tool, its pose and opening width to recreate robot- or human-made goals. Our experiments in simulated and real environments show that DoughNet is able to significantly outperform related approaches that consider deformation only as geometrical change.

Title: HalluciBot: Is There No Such Thing as a Bad Question?

• Authors: William Watson, Nicole Cho
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12535
• Pdf URL: https://arxiv.org/pdf/2404.12535
• Copy Paste: [[2404.12535]] HalluciBot: Is There No Such Thing as a Bad Question?(https://arxiv.org/abs/2404.12535)
• Keywords: large language model
• Abstract: Hallucination continues to be one of the most critical challenges in the institutional adoption journey of Large Language Models (LLMs). In this context, an overwhelming number of studies have focused on analyzing the post-generation phase - refining outputs via feedback, analyzing logit output values, or deriving clues via the outputs' artifacts. We propose HalluciBot, a model that predicts the probability of hallucination $\textbf{before generation}$, for any query imposed to an LLM. In essence, HalluciBot does not invoke any generation during inference. To derive empirical evidence for HalluciBot, we employ a Multi-Agent Monte Carlo Simulation using a Query Perturbator to craft $n$ variations per query at train time. The construction of our Query Perturbator is motivated by our introduction of a new definition of hallucination - $\textit{truthful hallucination}$. Our training methodology generated 2,219,022 estimates for a training corpus of 369,837 queries, spanning 13 diverse datasets and 3 question-answering scenarios. HalluciBot predicts both binary and multi-class probabilities of hallucination, enabling a means to judge the query's quality with regards to its propensity to hallucinate. Therefore, HalluciBot paves the way to revise or cancel a query before generation and the ensuing computational waste. Moreover, it provides a lucid means to measure user accountability for hallucinatory queries.

Title: TrACT: A Training Dynamics Aware Contrastive Learning Framework for Long-tail Trajectory Prediction

• Authors: Junrui Zhang, Mozhgan Pourkeshavarz, Amir Rasouli
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12538
• Pdf URL: https://arxiv.org/pdf/2404.12538
• Copy Paste: [[2404.12538]] TrACT: A Training Dynamics Aware Contrastive Learning Framework for Long-tail Trajectory Prediction(https://arxiv.org/abs/2404.12538)
• Keywords: robust
• Abstract: As a safety critical task, autonomous driving requires accurate predictions of road users' future trajectories for safe motion planning, particularly under challenging conditions. Yet, many recent deep learning methods suffer from a degraded performance on the challenging scenarios, mainly because these scenarios appear less frequently in the training data. To address such a long-tail issue, existing methods force challenging scenarios closer together in the feature space during training to trigger information sharing among them for more robust learning. These methods, however, primarily rely on the motion patterns to characterize scenarios, omitting more informative contextual information, such as interactions and scene layout. We argue that exploiting such information not only improves prediction accuracy but also scene compliance of the generated trajectories. In this paper, we propose to incorporate richer training dynamics information into a prototypical contrastive learning framework. More specifically, we propose a two-stage process. First, we generate rich contextual features using a baseline encoder-decoder framework. These features are split into clusters based on the model's output errors, using the training dynamics information, and a prototype is computed within each cluster. Second, we retrain the model using the prototypes in a contrastive learning framework. We conduct empirical evaluations of our approach using two large-scale naturalistic datasets and show that our method achieves state-of-the-art performance by improving accuracy and scene compliance on the long-tail samples. Furthermore, we perform experiments on a subset of the clusters to highlight the additional benefit of our approach in reducing training bias.

Title: GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I Diffusion Models

• Authors: Sai Sree Harsha, Ambareesh Revanur, Dhwanit Agarwal, Shradha Agrawal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12541
• Pdf URL: https://arxiv.org/pdf/2404.12541
• Copy Paste: [[2404.12541]] GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I Diffusion Models(https://arxiv.org/abs/2404.12541)
• Keywords: diffusion
• Abstract: Video editing methods based on diffusion models that rely solely on a text prompt for the edit are hindered by the limited expressive power of text prompts. Thus, incorporating a reference target image as a visual guide becomes desirable for precise control over edit. Also, most existing methods struggle to accurately edit a video when the shape and size of the object in the target image differ from the source object. To address these challenges, we propose "GenVideo" for editing videos leveraging target-image aware T2I models. Our approach handles edits with target objects of varying shapes and sizes while maintaining the temporal consistency of the edit using our novel target and shape aware InvEdit masks. Further, we propose a novel target-image aware latent noise correction strategy during inference to improve the temporal consistency of the edits. Experimental analyses indicate that GenVideo can effectively handle edits with objects of varying shapes, where existing approaches fail.

Title: Dubo-SQL: Diverse Retrieval-Augmented Generation and Fine Tuning for Text-to-SQL

• Authors: Dayton G. Thorpe, Andrew J. Duberstein, Ian A. Kinsey
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2404.12560
• Pdf URL: https://arxiv.org/pdf/2404.12560
• Copy Paste: [[2404.12560]] Dubo-SQL: Diverse Retrieval-Augmented Generation and Fine Tuning for Text-to-SQL(https://arxiv.org/abs/2404.12560)
• Keywords: large language model
• Abstract: The current state-of-the-art (SOTA) for automated text-to-SQL still falls well short of expert human performance as measured by execution accuracy (EX) on the BIRD-SQL benchmark. The most accurate methods are also slow and expensive. To advance the SOTA for text-to-SQL while reducing cost and improving speed, we explore the combination of low-cost fine tuning, novel methods for diverse retrieval-augmented generation (RAG) and new input and output formats that help large language models (LLMs) achieve higher EX. We introduce two new methods, Dubo-SQL v1 and v2. Dubo-SQL v1 sets a new record for EX on the holdout test set of BIRD-SQL. Dubo-SQL v2 achieves even higher performance on the BIRD-SQL dev set. Dubo-SQL v1 relies on LLMs from OpenAI, but uses the low-cost GPT-3.5 Turbo while exceeding the performance of the next-best model using OpenAI, which instead uses the more expensive GPT-4. Dubo-SQL v1 exceeds the performance of the next-best model using GPT-3.5 by over 20%. Dubo-SQL v2 uses GPT-4 Turbo and RAG in place of fine tuning to push EX higher.

Title: Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation

• Authors: Lasal Jayawardena, Prasan Yapa
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12596
• Pdf URL: https://arxiv.org/pdf/2404.12596
• Copy Paste: [[2404.12596]] Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation(https://arxiv.org/abs/2404.12596)
• Keywords: large language model
• Abstract: Over the past year, the field of Natural Language Generation (NLG) has experienced an exponential surge, largely due to the introduction of Large Language Models (LLMs). These models have exhibited the most effective performance in a range of domains within the Natural Language Processing and Generation domains. However, their application in domain-specific tasks, such as paraphrasing, presents significant challenges. The extensive number of parameters makes them difficult to operate on commercial hardware, and they require substantial time for inference, leading to high costs in a production setting. In this study, we tackle these obstacles by employing LLMs to develop three distinct models for the paraphrasing field, applying a method referred to as sequence-level knowledge distillation. These distilled models are capable of maintaining the quality of paraphrases generated by the LLM. They demonstrate faster inference times and the ability to generate diverse paraphrases of comparable quality. A notable characteristic of these models is their ability to exhibit syntactic diversity while also preserving lexical diversity, features previously uncommon due to existing data quality issues in datasets and not typically observed in neural-based approaches. Human evaluation of our models shows that there is only a 4% drop in performance compared to the LLM teacher model used in the distillation process, despite being 1000 times smaller. This research provides a significant contribution to the NLG field, offering a more efficient and cost-effective solution for paraphrasing tasks.

Title: Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty

• Authors: Yanwei Jia
• Subjects: cs.LG, eess.SY, q-fin.CP, q-fin.PM
• Abstract URL: https://arxiv.org/abs/2404.12598
• Pdf URL: https://arxiv.org/pdf/2404.12598
• Copy Paste: [[2404.12598]] Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty(https://arxiv.org/abs/2404.12598)
• Keywords: robust, diffusion
• Abstract: This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. The risk-sensitive objective arises either as the agent's risk attitude or as a distributionally robust approach against the model uncertainty. Owing to the martingale perspective in Jia and Zhou (2023) the risk-sensitive RL problem is shown to be equivalent to ensuring the martingale property of a process involving both the value function and the q-function, augmented by an additional penalty term: the quadratic variation of the value process, capturing the variability of the value-to-go along the trajectory. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process. Additionally, I highlight that the conventional policy gradient representation is inadequate for risk-sensitive problems due to the nonlinear nature of quadratic variation; however, q-learning offers a solution and extends to infinite horizon settings. Finally, I prove the convergence of the proposed algorithm for Merton's investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure. I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem.

Title: A visualization method for data domain changes in CNN networks and the optimization method for selecting thresholds in classification tasks

• Authors: Minzhe Huang, Changwei Nie, Weihong Zhong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12602
• Pdf URL: https://arxiv.org/pdf/2404.12602
• Copy Paste: [[2404.12602]] A visualization method for data domain changes in CNN networks and the optimization method for selecting thresholds in classification tasks(https://arxiv.org/abs/2404.12602)
• Keywords: secure, security, attack, robust
• Abstract: In recent years, Face Anti-Spoofing (FAS) has played a crucial role in preserving the security of face recognition technology. With the rise of counterfeit face generation techniques, the challenge posed by digitally edited faces to face anti-spoofing is escalating. Existing FAS technologies primarily focus on intercepting physically forged faces and lack a robust solution for cross-domain FAS challenges. Moreover, determining an appropriate threshold to achieve optimal deployment results remains an issue for intra-domain FAS. To address these issues, we propose a visualization method that intuitively reflects the training outcomes of models by visualizing the prediction results on datasets. Additionally, we demonstrate that employing data augmentation techniques, such as downsampling and Gaussian blur, can effectively enhance performance on cross-domain tasks. Building upon our data visualization approach, we also introduce a methodology for setting threshold values based on the distribution of the training dataset. Ultimately, our methods secured us second place in both the Unified Physical-Digital Face Attack Detection competition and the Snapshot Spectral Imaging Face Anti-spoofing contest. The training code is available at https://github.com/SeaRecluse/CVPRW2024.

Title: ELEV-VISION-SAM: Integrated Vision Language and Foundation Model for Automated Estimation of Building Lowest Floor Elevation

• Authors: Yu-Hsuan Ho, Longxiang Li, Ali Mostafavi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12606
• Pdf URL: https://arxiv.org/pdf/2404.12606
• Copy Paste: [[2404.12606]] ELEV-VISION-SAM: Integrated Vision Language and Foundation Model for Automated Estimation of Building Lowest Floor Elevation(https://arxiv.org/abs/2404.12606)
• Keywords: segmentation
• Abstract: Street view imagery, aided by advancements in image quality and accessibility, has emerged as a valuable resource for urban analytics research. Recent studies have explored its potential for estimating lowest floor elevation (LFE), offering a scalable alternative to traditional on-site measurements, crucial for assessing properties' flood risk and damage extent. While existing methods rely on object detection, the introduction of image segmentation has broadened street view images' utility for LFE estimation, although challenges still remain in segmentation quality and capability to distinguish front doors from other doors. To address these challenges in LFE estimation, this study integrates the Segment Anything model, a segmentation foundation model, with vision language models to conduct text-prompt image segmentation on street view images for LFE estimation. By evaluating various vision language models, integration methods, and text prompts, we identify the most suitable model for street view image analytics and LFE estimation tasks, thereby improving the availability of the current LFE estimation model based on image segmentation from 33% to 56% of properties. Remarkably, our proposed method significantly enhances the availability of LFE estimation to almost all properties in which the front door is visible in the street view image. Also the findings present the first baseline and comparison of various vision models of street view image-based LFE estimation. The model and findings not only contribute to advancing street view image segmentation for urban analytics but also provide a novel approach for image segmentation tasks for other civil engineering and infrastructure analytics tasks.

Title: Rethinking Clothes Changing Person ReID: Conflicts, Synthesis, and Optimization

• Authors: Junjie Li, Guanshuo Wang, Fufu Yu, Yichao Yan, Qiong Jia, Shouhong Ding, Xingdong Sheng, Yunhui Liu, Xiaokang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12611
• Pdf URL: https://arxiv.org/pdf/2404.12611
• Copy Paste: [[2404.12611]] Rethinking Clothes Changing Person ReID: Conflicts, Synthesis, and Optimization(https://arxiv.org/abs/2404.12611)
• Keywords: diffusion
• Abstract: Clothes-changing person re-identification (CC-ReID) aims to retrieve images of the same person wearing different outfits. Mainstream researches focus on designing advanced model structures and strategies to capture identity information independent of clothing. However, the same-clothes discrimination as the standard ReID learning objective in CC-ReID is persistently ignored in previous researches. In this study, we dive into the relationship between standard and clothes-changing~(CC) learning objectives, and bring the inner conflicts between these two objectives to the fore. We try to magnify the proportion of CC training pairs by supplementing high-fidelity clothes-varying synthesis, produced by our proposed Clothes-Changing Diffusion model. By incorporating the synthetic images into CC-ReID model training, we observe a significant improvement under CC protocol. However, such improvement sacrifices the performance under the standard protocol, caused by the inner conflict between standard and CC. For conflict mitigation, we decouple these objectives and re-formulate CC-ReID learning as a multi-objective optimization (MOO) problem. By effectively regularizing the gradient curvature across multiple objectives and introducing preference restrictions, our MOO solution surpasses the single-task training paradigm. Our framework is model-agnostic, and demonstrates superior performance under both CC and standard ReID protocols.

Title: SA-Attack: Speed-adaptive stealthy adversarial attack on trajectory prediction

• Authors: Huilin Yin, Jiaxiang Li, Pengju Zhen, Jun Yan
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12612
• Pdf URL: https://arxiv.org/pdf/2404.12612
• Copy Paste: [[2404.12612]] SA-Attack: Speed-adaptive stealthy adversarial attack on trajectory prediction(https://arxiv.org/abs/2404.12612)
• Keywords: attack, steal
• Abstract: Trajectory prediction is critical for the safe planning and navigation of automated vehicles. The trajectory prediction models based on the neural networks are vulnerable to adversarial attacks. Previous attack methods have achieved high attack success rates but overlook the adaptability to realistic scenarios and the concealment of the deceits. To address this problem, we propose a speed-adaptive stealthy adversarial attack method named SA-Attack. This method searches the sensitive region of trajectory prediction models and generates the adversarial trajectories by using the vehicle-following method and incorporating information about forthcoming trajectories. Our method has the ability to adapt to different speed scenarios by reconstructing the trajectory from scratch. Fusing future trajectory trends and curvature constraints can guarantee the smoothness of adversarial trajectories, further ensuring the stealthiness of attacks. The empirical study on the datasets of nuScenes and Apolloscape demonstrates the attack performance of our proposed method. Finally, we also demonstrate the adaptability and stealthiness of SA-Attack for different speed scenarios. Our code is available at the repository: https://github.com/eclipse-bot/SA-Attack.

Title: End-to-End Verifiable Decentralized Federated Learning

• Authors: Chaehyeon Lee, Jonathan Heiss, Stefan Tai, James Won-Ki Hong
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.12623
• Pdf URL: https://arxiv.org/pdf/2404.12623
• Copy Paste: [[2404.12623]] End-to-End Verifiable Decentralized Federated Learning(https://arxiv.org/abs/2404.12623)
• Keywords: federate
• Abstract: Verifiable decentralized federated learning (FL) systems combining blockchains and zero-knowledge proofs (ZKP) make the computational integrity of local learning and global aggregation verifiable across workers. However, they are not end-to-end: data can still be corrupted prior to the learning. In this paper, we propose a verifiable decentralized FL system for end-to-end integrity and authenticity of data and computation extending verifiability to the data source. Addressing an inherent conflict of confidentiality and transparency, we introduce a two-step proving and verification (2PV) method that we apply to central system procedures: a registration workflow that enables non-disclosing verification of device certificates and a learning workflow that extends existing blockchain and ZKP-based FL systems through non-disclosing data authenticity proofs. Our evaluation on a prototypical implementation demonstrates the technical feasibility with only marginal overheads to state-of-the-art solutions.

Title: SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers

• Authors: Vandad Davoodnia, Saeed Ghorbani, Alexandre Messier, Ali Etemad
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12625
• Pdf URL: https://arxiv.org/pdf/2404.12625
• Copy Paste: [[2404.12625]] SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers(https://arxiv.org/abs/2404.12625)
• Keywords: robust, transformer
• Abstract: We introduce SkelFormer, a novel markerless motion capture pipeline for multi-view human pose and shape estimation. Our method first uses off-the-shelf 2D keypoint estimators, pre-trained on large-scale in-the-wild data, to obtain 3D joint positions. Next, we design a regression-based inverse-kinematic skeletal transformer that maps the joint positions to pose and shape representations from heavily noisy observations. This module integrates prior knowledge about pose space and infers the full pose state at runtime. Separating the 3D keypoint detection and inverse-kinematic problems, along with the expressive representations learned by our skeletal transformer, enhance the generalization of our method to unseen noisy data. We evaluate our method on three public datasets in both in-distribution and out-of-distribution settings using three datasets, and observe strong performance with respect to prior works. Moreover, ablation experiments demonstrate the impact of each of the modules of our architecture. Finally, we study the performance of our method in dealing with noise and heavy occlusions and find considerable robustness with respect to other solutions.

Title: Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatment

• Authors: Danqing Ma, Meng Wang, Ao Xiang, Zongqing Qi, Qin Yang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12634
• Pdf URL: https://arxiv.org/pdf/2404.12634
• Copy Paste: [[2404.12634]] Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatment(https://arxiv.org/abs/2404.12634)
• Keywords: transformer
• Abstract: This study proposes a multi-modal fusion framework Multitrans based on the Transformer architecture and self-attention mechanism. This architecture combines the study of non-contrast computed tomography (NCCT) images and discharge diagnosis reports of patients undergoing stroke treatment, using a variety of methods based on Transformer architecture approach to predicting functional outcomes of stroke treatment. The results show that the performance of single-modal text classification is significantly better than single-modal image classification, but the effect of multi-modal combination is better than any single modality. Although the Transformer model only performs worse on imaging data, when combined with clinical meta-diagnostic information, both can learn better complementary information and make good contributions to accurately predicting stroke treatment effects..

Title: AED-PADA:Improving Generalizability of Adversarial Example Detection via Principal Adversarial Domain Adaptation

• Authors: Heqi Peng, Yunhong Wang, Ruijie Yang, Beichen Li, Rui Wang, Yuanfang Guo
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12635
• Pdf URL: https://arxiv.org/pdf/2404.12635
• Copy Paste: [[2404.12635]] AED-PADA:Improving Generalizability of Adversarial Example Detection via Principal Adversarial Domain Adaptation(https://arxiv.org/abs/2404.12635)
• Keywords: defense, attack
• Abstract: Adversarial example detection, which can be conveniently applied in many scenarios, is important in the area of adversarial defense. Unfortunately, existing detection methods suffer from poor generalization performance, because their training process usually relies on the examples generated from a single known adversarial attack and there exists a large discrepancy between the training and unseen testing adversarial examples. To address this issue, we propose a novel method, named Adversarial Example Detection via Principal Adversarial Domain Adaptation (AED-PADA). Specifically, our approach identifies the Principal Adversarial Domains (PADs), i.e., a combination of features of the adversarial examples from different attacks, which possesses large coverage of the entire adversarial feature space. Then, we pioneer to exploit multi-source domain adaptation in adversarial example detection with PADs as source domains. Experiments demonstrate the superior generalization ability of our proposed AED-PADA. Note that this superiority is particularly achieved in challenging scenarios characterized by employing the minimal magnitude constraint for the perturbations.

Title: SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysis

• Authors: Hongzhi Qi, Hanfei Liu, Jianqiang Li, Qing Zhao, Wei Zhai, Dan Luo, Tian Yu He, Shuo Liu, Bing Xiang Yang, Guanghui Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12659
• Pdf URL: https://arxiv.org/pdf/2404.12659
• Copy Paste: [[2404.12659]] SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysis(https://arxiv.org/abs/2404.12659)
• Keywords: large language model
• Abstract: In the social media, users frequently express personal emotions, a subset of which may indicate potential suicidal tendencies. The implicit and varied forms of expression in internet language complicate accurate and rapid identification of suicidal intent on social media, thus creating challenges for timely intervention efforts. The development of deep learning models for suicide risk detection is a promising solution, but there is a notable lack of relevant datasets, especially in the Chinese context. To address this gap, this study presents a Chinese social media dataset designed for fine-grained suicide risk classification, focusing on indicators such as expressions of suicide intent, methods of suicide, and urgency of timing. Seven pre-trained models were evaluated in two tasks: high and low suicide risk, and fine-grained suicide risk classification on a level of 0 to 10. In our experiments, deep learning models show good performance in distinguishing between high and low suicide risk, with the best model achieving an F1 score of 88.39%. However, the results for fine-grained suicide risk classification were still unsatisfactory, with an weighted F1 score of 50.89%. To address the issues of data imbalance and limited dataset size, we investigated both traditional and advanced, large language model based data augmentation techniques, demonstrating that data augmentation can enhance model performance by up to 4.65% points in F1-score. Notably, the Chinese MentalBERT model, which was pre-trained on psychological domain data, shows superior performance in both tasks. This study provides valuable insights for automatic identification of suicidal individuals, facilitating timely psychological intervention on social media platforms. The source code and data are publicly available.

Title: Detecting Out-Of-Distribution Earth Observation Images with Diffusion Models

• Authors: Georges Le Bellier (CEDRIC - VERTIGO, CNAM), Nicolas Audebert (CEDRIC - VERTIGO, CNAM, IGN)
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12667
• Pdf URL: https://arxiv.org/pdf/2404.12667
• Copy Paste: [[2404.12667]] Detecting Out-Of-Distribution Earth Observation Images with Diffusion Models(https://arxiv.org/abs/2404.12667)
• Keywords: diffusion, generative
• Abstract: Earth Observation imagery can capture rare and unusual events, such as disasters and major landscape changes, whose visual appearance contrasts with the usual observations. Deep models trained on common remote sensing data will output drastically different features for these out-of-distribution samples, compared to those closer to their training dataset. Detecting them could therefore help anticipate changes in the observations, either geographical or environmental. In this work, we show that the reconstruction error of diffusion models can effectively serve as unsupervised out-of-distribution detectors for remote sensing images, using them as a plausibility score. Moreover, we introduce ODEED, a novel reconstruction-based scorer using the probability-flow ODE of diffusion models. We validate it experimentally on SpaceNet 8 with various scenarios, such as classical OOD detection with geographical shift and near-OOD setups: pre/post-flood and non-flooded/flooded image recognition. We show that our ODEED scorer significantly outperforms other diffusion-based and discriminative baselines on the more challenging near-OOD scenarios of flood image detection, where OOD images are close to the distribution tail. We aim to pave the way towards better use of generative models for anomaly detection in remote sensing.

Title: Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model

• Authors: Jihao Dong, Renjie Pan, Hua Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12678
• Pdf URL: https://arxiv.org/pdf/2404.12678
• Copy Paste: [[2404.12678]] Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model(https://arxiv.org/abs/2404.12678)
• Keywords: transformer
• Abstract: Human-Object Interaction (HOI) detection aims to localize human-object pairs and comprehend their interactions. Recently, two-stage transformer-based methods have demonstrated competitive performance. However, these methods frequently focus on object appearance features and ignore global contextual information. Besides, vision-language model CLIP which effectively aligns visual and text embeddings has shown great potential in zero-shot HOI detection. Based on the former facts, We introduce a novel HOI detector named ISA-HOI, which extensively leverages knowledge from CLIP, aligning interactive semantics between visual and textual features. We first extract global context of image and local features of object to Improve interaction Features in images (IF). On the other hand, we propose a Verb Semantic Improvement (VSI) module to enhance textual features of verb labels via cross-modal fusion. Ultimately, our method achieves competitive results on the HICO-DET and V-COCO benchmarks with much fewer training epochs, and outperforms the state-of-the-art under zero-shot settings.

Title: MLSD-GAN -- Generating Strong High Quality Face Morphing Attacks using Latent Semantic Disentanglement

• Authors: Aravinda Reddy PN, Raghavendra Ramachandra, Krothapalli Sreenivasa Rao, Pabitra Mitra
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12679
• Pdf URL: https://arxiv.org/pdf/2404.12679
• Copy Paste: [[2404.12679]] MLSD-GAN -- Generating Strong High Quality Face Morphing Attacks using Latent Semantic Disentanglement(https://arxiv.org/abs/2404.12679)
• Keywords: attack, biometric, generative
• Abstract: Face-morphing attacks are a growing concern for biometric researchers, as they can be used to fool face recognition systems (FRS). These attacks can be generated at the image level (supervised) or representation level (unsupervised). Previous unsupervised morphing attacks have relied on generative adversarial networks (GANs). More recently, researchers have used linear interpolation of StyleGAN-encoded images to generate morphing attacks. In this paper, we propose a new method for generating high-quality morphing attacks using StyleGAN disentanglement. Our approach, called MLSD-GAN, spherically interpolates the disentangled latents to produce realistic and diverse morphing attacks. We evaluate the vulnerability of MLSD-GAN on two deep-learning-based FRS techniques. The results show that MLSD-GAN poses a significant threat to FRS, as it can generate morphing attacks that are highly effective at fooling these systems.

Title: VoxAtnNet: A 3D Point Clouds Convolutional Neural Network for Generalizable Face Presentation Attack Detection

• Authors: Raghavendra Ramachandra, Narayan Vetrekar, Sushma Venkatesh, Savita Nageshker, Jag Mohan Singh, R. S. Gad
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12680
• Pdf URL: https://arxiv.org/pdf/2404.12680
• Copy Paste: [[2404.12680]] VoxAtnNet: A 3D Point Clouds Convolutional Neural Network for Generalizable Face Presentation Attack Detection(https://arxiv.org/abs/2404.12680)
• Keywords: attack, biometric
• Abstract: Facial biometrics are an essential components of smartphones to ensure reliable and trustworthy authentication. However, face biometric systems are vulnerable to Presentation Attacks (PAs), and the availability of more sophisticated presentation attack instruments such as 3D silicone face masks will allow attackers to deceive face recognition systems easily. In this work, we propose a novel Presentation Attack Detection (PAD) algorithm based on 3D point clouds captured using the frontal camera of a smartphone to detect presentation attacks. The proposed PAD algorithm, VoxAtnNet, processes 3D point clouds to obtain voxelization to preserve the spatial structure. Then, the voxelized 3D samples were trained using the novel convolutional attention network to detect PAs on the smartphone. Extensive experiments were carried out on the newly constructed 3D face point cloud dataset comprising bona fide and two different 3D PAIs (3D silicone face mask and wrap photo mask), resulting in 3480 samples. The performance of the proposed method was compared with existing methods to benchmark the detection performance using three different evaluation protocols. The experimental results demonstrate the improved performance of the proposed method in detecting both known and unknown face presentation attacks.

Title: ESC: Evolutionary Stitched Camera Calibration in the Wild

• Authors: Grzegorz Rypeść, Grzegorz Kurzejamski
• Subjects: cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.12694
• Pdf URL: https://arxiv.org/pdf/2404.12694
• Copy Paste: [[2404.12694]] ESC: Evolutionary Stitched Camera Calibration in the Wild(https://arxiv.org/abs/2404.12694)
• Keywords: segmentation
• Abstract: This work introduces a novel end-to-end approach for estimating extrinsic parameters of cameras in multi-camera setups on real-life sports fields. We identify the source of significant calibration errors in multi-camera environments and address the limitations of existing calibration methods, particularly the disparity between theoretical models and actual sports field characteristics. We propose the Evolutionary Stitched Camera calibration (ESC) algorithm to bridge this gap. It consists of image segmentation followed by evolutionary optimization of a novel loss function, providing a unified and accurate multi-camera calibration solution with high visual fidelity. The outcome allows the creation of virtual stitched views from multiple video sources, being as important for practical applications as numerical accuracy. We demonstrate the superior performance of our approach compared to state-of-the-art methods across diverse real-life football fields with varying physical characteristics.

Title: Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations

• Authors: Xiao Zhang, Gosse Bouma, Johan Bos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12698
• Pdf URL: https://arxiv.org/pdf/2404.12698
• Copy Paste: [[2404.12698]] Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations(https://arxiv.org/abs/2404.12698)
• Keywords: interpretability
• Abstract: Current open-domain neural semantics parsers show impressive performance. However, closer inspection of the symbolic meaning representations they produce reveals significant weaknesses: sometimes they tend to merely copy character sequences from the source text to form symbolic concepts, defaulting to the most frequent word sense based in the training distribution. By leveraging the hierarchical structure of a lexical ontology, we introduce a novel compositional symbolic representation for concepts based on their position in the taxonomical hierarchy. This representation provides richer semantic information and enhances interpretability. We introduce a neural "taxonomical" semantic parser to utilize this new representation system of predicates, and compare it with a standard neural semantic parser trained on the traditional meaning representation format, employing a novel challenge set and evaluation metric for evaluation. Our experimental findings demonstrate that the taxonomical model, trained on much richer and complex meaning representations, is slightly subordinate in performance to the traditional model using the standard metrics for evaluation, but outperforms it when dealing with out-of-vocabulary concepts. This finding is encouraging for research in computational semantics that aims to combine data-driven distributional meanings with knowledge-based symbolic representations.

Title: SOPHON: Non-Fine-Tunable Learning to Restrain Task Transferability For Pre-trained Models

• Authors: Jiangyi Deng (1), Shengyuan Pang (1), Yanjiao Chen (1), Liangming Xia (1), Yijie Bai (1), Haiqin Weng (2), Wenyuan Xu (1) ((1) Zhejiang University, (2) Ant Group)
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12699
• Pdf URL: https://arxiv.org/pdf/2404.12699
• Copy Paste: [[2404.12699]] SOPHON: Non-Fine-Tunable Learning to Restrain Task Transferability For Pre-trained Models(https://arxiv.org/abs/2404.12699)
• Keywords: privacy, protect, robust
• Abstract: Instead of building deep learning models from scratch, developers are more and more relying on adapting pre-trained models to their customized tasks. However, powerful pre-trained models may be misused for unethical or illegal tasks, e.g., privacy inference and unsafe content generation. In this paper, we introduce a pioneering learning paradigm, non-fine-tunable learning, which prevents the pre-trained model from being fine-tuned to indecent tasks while preserving its performance on the original task. To fulfill this goal, we propose SOPHON, a protection framework that reinforces a given pre-trained model to be resistant to being fine-tuned in pre-defined restricted domains. Nonetheless, this is challenging due to a diversity of complicated fine-tuning strategies that may be adopted by adversaries. Inspired by model-agnostic meta-learning, we overcome this difficulty by designing sophisticated fine-tuning simulation and fine-tuning evaluation algorithms. In addition, we carefully design the optimization process to entrap the pre-trained model within a hard-to-escape local optimum regarding restricted domains. We have conducted extensive experiments on two deep learning modes (classification and generation), seven restricted domains, and six model architectures to verify the effectiveness of SOPHON. Experiment results verify that fine-tuning SOPHON-protected models incurs an overhead comparable to or even greater than training from scratch. Furthermore, we confirm the robustness of SOPHON to three fine-tuning methods, five optimizers, various learning rates and batch sizes. SOPHON may help boost further investigations into safe and responsible AI.

Title: Modeling Multi-Granularity Context Information Flow for Pavement Crack Detection

• Authors: Junbiao Pang, Baocheng Xiong, Jiaqi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12702
• Pdf URL: https://arxiv.org/pdf/2404.12702
• Copy Paste: [[2404.12702]] Modeling Multi-Granularity Context Information Flow for Pavement Crack Detection(https://arxiv.org/abs/2404.12702)
• Keywords: extraction
• Abstract: Crack detection has become an indispensable, interesting yet challenging task in the computer vision community. Specially, pavement cracks have a highly complex spatial structure, a low contrasting background and a weak spatial continuity, posing a significant challenge to an effective crack detection method. In this paper, we address these problems from a view that utilizes contexts of the cracks and propose an end-to-end deep learning method to model the context information flow. To precisely localize crack from an image, it is critical to effectively extract and aggregate multi-granularity context, including the fine-grained local context around the cracks (in spatial-level) and the coarse-grained semantics (in segment-level). Concretely, in Convolutional Neural Network (CNN), low-level features extracted by the shallow layers represent the local information, while the deep layers extract the semantic features. Additionally, a second main insight in this work is that the semantic context should be an guidance to local context feature. By the above insights, the proposed method we first apply the dilated convolution as the backbone feature extractor to model local context, then we build a context guidance module to leverage semantic context to guide local feature extraction at multiple stages. To handle label alignment between stages, we apply the Multiple Instance Learning (MIL) strategy to align the high-level feature to the low-level ones in the stage-wise context flow. In addition, compared with these public crack datasets, to our best knowledge, we release the largest, most complex and most challenging Bitumen Pavement Crack (BPC) dataset. The experimental results on the three crack datasets demonstrate that the proposed method performs well and outperforms the current state-of-the-art methods.

Title: FedMeS: Personalized Federated Continual Learning Leveraging Local Memory

• Authors: Jin Xie, Chenqing Zhu, Songze Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12710
• Pdf URL: https://arxiv.org/pdf/2404.12710
• Copy Paste: [[2404.12710]] FedMeS: Personalized Federated Continual Learning Leveraging Local Memory(https://arxiv.org/abs/2404.12710)
• Keywords: federate
• Abstract: We focus on the problem of Personalized Federated Continual Learning (PFCL): a group of distributed clients, each with a sequence of local tasks on arbitrary data distributions, collaborate through a central server to train a personalized model at each client, with the model expected to achieve good performance on all local tasks. We propose a novel PFCL framework called Federated Memory Strengthening FedMeS to address the challenges of client drift and catastrophic forgetting. In FedMeS, each client stores samples from previous tasks using a small amount of local memory, and leverages this information to both 1) calibrate gradient updates in training process; and 2) perform KNN-based Gaussian inference to facilitate personalization. FedMeS is designed to be task-oblivious, such that the same inference process is applied to samples from all tasks to achieve good performance. FedMeS is analyzed theoretically and evaluated experimentally. It is shown to outperform all baselines in average accuracy and forgetting rate, over various combinations of datasets, task distributions, and client numbers.

Title: Dynamic Temperature Knowledge Distillation

• Authors: Yukang Wei, Yu Bai
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12711
• Pdf URL: https://arxiv.org/pdf/2404.12711
• Copy Paste: [[2404.12711]] Dynamic Temperature Knowledge Distillation(https://arxiv.org/abs/2404.12711)
• Keywords: robust
• Abstract: Temperature plays a pivotal role in moderating label softness in the realm of knowledge distillation (KD). Traditional approaches often employ a static temperature throughout the KD process, which fails to address the nuanced complexities of samples with varying levels of difficulty and overlooks the distinct capabilities of different teacher-student pairings. This leads to a less-than-ideal transfer of knowledge. To improve the process of knowledge propagation, we proposed Dynamic Temperature Knowledge Distillation (DTKD) which introduces a dynamic, cooperative temperature control for both teacher and student models simultaneously within each training iterafion. In particular, we proposed "\textbf{sharpness}" as a metric to quantify the smoothness of a model's output distribution. By minimizing the sharpness difference between the teacher and the student, we can derive sample-specific temperatures for them respectively. Extensive experiments on CIFAR-100 and ImageNet-2012 demonstrate that DTKD performs comparably to leading KD techniques, with added robustness in Target Class KD and None-target Class KD scenarios.The code is available at https://github.com/JinYu1998/DTKD.

Title: Enabling Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration

• Authors: Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Bing Qin, Ting Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12715
• Pdf URL: https://arxiv.org/pdf/2404.12715
• Copy Paste: [[2404.12715]] Enabling Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration(https://arxiv.org/abs/2404.12715)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown complementary strengths in various tasks and instances, motivating the research of ensembling LLMs to push the frontier leveraging the wisdom of the crowd. Existing work achieves this objective via training the extra reward model or fusion model to select or fuse all candidate answers. However, these methods pose a great challenge to the generalizability of the trained models. Besides, existing methods use the textual responses as communication media, ignoring the rich information in the inner representations of neural networks. Therefore, we propose a training-free ensemble framework DEEPEN, averaging the probability distributions outputted by different LLMs. A key challenge in this paradigm is the vocabulary discrepancy between heterogeneous LLMs, which hinders the operation of probability distribution averaging. To address this challenge, DEEPEN maps the probability distribution of each model from the probability space to a universe relative space based on the relative representation theory, and performs aggregation. Then, the result of aggregation is mapped back to the probability space of one LLM via a search-based inverse transformation to determine the generated token. We conduct experiments on the ensemble of various LLMs of 6B to 70B. Experimental results show that DEEPEN achieves consistent improvements across six popular benchmarks involving subject examination, reasoning and knowledge-QA, proving the effectiveness of our approach.

Title: Improving Prediction Accuracy of Semantic Segmentation Methods Using Convolutional Autoencoder Based Pre-processing Layers

• Authors: Hisashi Shimodaira
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12718
• Pdf URL: https://arxiv.org/pdf/2404.12718
• Copy Paste: [[2404.12718]] Improving Prediction Accuracy of Semantic Segmentation Methods Using Convolutional Autoencoder Based Pre-processing Layers(https://arxiv.org/abs/2404.12718)
• Keywords: segmentation
• Abstract: In this paper, we propose a method to improve prediction accuracy of semantic segmentation methods as follows: (1) construct a neural network that has pre-processing layers based on a convolutional autoencoder ahead of a semantic segmentation network, and (2) train the entire network initialized by the weights of the pre-trained autoencoder. We applied this method to the fully convolutional network (FCN) and experimentally compared its prediction accuracy on the cityscapes dataset. The Mean IoU of the proposed target model with the He normal initialization is 18.7% higher than that of FCN with the He normal initialization. In addition, those of the modified models of the target model are significantly higher than that of FCN with the He normal initialization. The accuracy and loss curves during the training showed that these are resulting from the improvement of the generalization ability. All of these results provide strong evidence that the proposed method is significantly effective in improving the prediction accuracy of FCN. The proposed method has the following features: it is comparatively simple, whereas the effect on improving the generalization ability and prediction accuracy of FCN is significant; the increase in the number of parameters by using it is very small, and that in the computation time is substantially large. In principle, the proposed method can be applied to other semantic segmentation methods. For semantic segmentation, at present, there is no effective way to improve the prediction accuracy of existing methods. None have published a method which is the same as or similar to our method and none have used such a method in practice. Therefore, we believe that our method is useful in practice and worthy of being widely known and used.

Title: Generalized Few-Shot Meets Remote Sensing: Discovering Novel Classes in Land Cover Mapping via Hybrid Semantic Segmentation Framework

• Authors: Zhuohong Li, Fangxiao Lu, Jiaqi Zou, Lei Hu, Hongyan Zhang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12721
• Pdf URL: https://arxiv.org/pdf/2404.12721
• Copy Paste: [[2404.12721]] Generalized Few-Shot Meets Remote Sensing: Discovering Novel Classes in Land Cover Mapping via Hybrid Semantic Segmentation Framework(https://arxiv.org/abs/2404.12721)
• Keywords: segmentation
• Abstract: Land-cover mapping is one of the vital applications in Earth observation, aiming at classifying each pixel's land-cover type of remote-sensing images. As natural and human activities change the landscape, the land-cover map needs to be rapidly updated. However, discovering newly appeared land-cover types in existing classification systems is still a non-trivial task hindered by various scales of complex land objects and insufficient labeled data over a wide-span geographic area. In this paper, we propose a generalized few-shot segmentation-based framework, named SegLand, to update novel classes in high-resolution land-cover mapping. Specifically, the proposed framework is designed in three parts: (a) Data pre-processing: the base training set and the few-shot support sets of novel classes are analyzed and augmented; (b) Hybrid segmentation structure; Multiple base learners and a modified Projection onto Orthogonal Prototypes (POP) network are combined to enhance the base-class recognition and to dig novel classes from insufficient labels data; (c) Ultimate fusion: the semantic segmentation results of the base learners and POP network are reasonably fused. The proposed framework has won first place in the leaderboard of the OpenEarthMap Land Cover Mapping Few-Shot Challenge. Experiments demonstrate the superiority of the framework for automatically updating novel land-cover classes with limited labeled data.

Title: Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works

• Authors: Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, Deqing Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12726
• Pdf URL: https://arxiv.org/pdf/2404.12726
• Copy Paste: [[2404.12726]] Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works(https://arxiv.org/abs/2404.12726)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated impressive performance and spurred numerous AI applications, in which role-playing agents (RPAs) are particularly popular, especially for fictional characters. The prerequisite for these RPAs lies in the capability of LLMs to understand characters from fictional works. Previous efforts have evaluated this capability via basic classification tasks or characteristic imitation, failing to capture the nuanced character understanding with LLMs. In this paper, we propose evaluating LLMs' character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development. Specifically, we construct the CroSS dataset from literature experts and assess the generated profiles by comparing ground truth references and their applicability in downstream tasks. Our experiments, which cover various summarization methods and LLMs, have yielded promising results. These results strongly validate the character understanding capability of LLMs. We believe our constructed resource will promote further research in this field. Resources are available at https://github.com/Joanna0123/character_profiling.

Title: Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?

• Authors: Chengwei Qin, Wenhan Xia, Tan Wang, Fangkai Jiao, Yuchen Hu, Bosheng Ding, Ruirui Chen, Shafiq Joty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12728
• Pdf URL: https://arxiv.org/pdf/2404.12728
• Copy Paste: [[2404.12728]] Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?(https://arxiv.org/abs/2404.12728)
• Keywords: large language model
• Abstract: Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences. One key finding in psychology is that compared with irrelevant past experiences, recalling relevant ones can help humans better handle new tasks. Coincidentally, the NLP community has also recently found that self-generating relevant examples in the context can help large language models (LLMs) better solve a given problem than hand-crafted prompts. However, it is yet not clear whether relevance is the key factor eliciting such capability, i.e., can LLMs benefit more from self-generated relevant examples than irrelevant ones? In this work, we systematically explore whether LLMs can truly perform analogical reasoning on a diverse set of reasoning tasks. With extensive experiments and analysis, we show that self-generated random examples can surprisingly achieve comparable or even better performance, e.g., 4% performance boost on GSM8K with random biological examples. We find that the accuracy of self-generated examples is the key factor and subsequently design two improved methods with significantly reduced inference costs. Overall, we aim to advance a deeper understanding of LLM analogical reasoning and hope this work stimulates further research in the design of self-generated contexts.

Title: PATE-TripleGAN: Privacy-Preserving Image Synthesis with Gaussian Differential Privacy

• Authors: Zepeng Jiang, Weiwei Ni, Yifan Zhang
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12730
• Pdf URL: https://arxiv.org/pdf/2404.12730
• Copy Paste: [[2404.12730]] PATE-TripleGAN: Privacy-Preserving Image Synthesis with Gaussian Differential Privacy(https://arxiv.org/abs/2404.12730)
• Keywords: privacy, protect, generative
• Abstract: Conditional Generative Adversarial Networks (CGANs) exhibit significant potential in supervised learning model training by virtue of their ability to generate realistic labeled images. However, numerous studies have indicated the privacy leakage risk in CGANs models. The solution DPCGAN, incorporating the differential privacy framework, faces challenges such as heavy reliance on labeled data for model training and potential disruptions to original gradient information due to excessive gradient clipping, making it difficult to ensure model accuracy. To address these challenges, we present a privacy-preserving training framework called PATE-TripleGAN. This framework incorporates a classifier to pre-classify unlabeled data, establishing a three-party min-max game to reduce dependence on labeled data. Furthermore, we present a hybrid gradient desensitization algorithm based on the Private Aggregation of Teacher Ensembles (PATE) framework and Differential Private Stochastic Gradient Descent (DPSGD) method. This algorithm allows the model to retain gradient information more effectively while ensuring privacy protection, thereby enhancing the model's utility. Privacy analysis and extensive experiments affirm that the PATE-TripleGAN model can generate a higher quality labeled image dataset while ensuring the privacy of the training data.

Title: DLoRA-TrOCR: Mixed Text Mode Optical Character Recognition Based On Transformer

• Authors: Da Chang, Yu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12734
• Pdf URL: https://arxiv.org/pdf/2404.12734
• Copy Paste: [[2404.12734]] DLoRA-TrOCR: Mixed Text Mode Optical Character Recognition Based On Transformer(https://arxiv.org/abs/2404.12734)
• Keywords: robust, transformer
• Abstract: With the continuous development of OCR technology and the expansion of application fields, text recognition in complex scenes has become a key challenge. Factors such as multiple fonts, mixed scenes and complex layouts seriously affect the recognition accuracy of traditional OCR models. Although OCR models based on deep learning have performed well in specific fields or similar data sets in recent years, the generalization ability and robustness of the model are still a big challenge when facing complex environments with multiple scenes. Furthermore, training an OCR model from scratch or fine-tuning all parameters is very demanding on computing resources and inference time, which limits the flexibility of its application. This study focuses on a fundamental aspect of mixed text recognition in response to the challenges mentioned above, which involves effectively fine-tuning the pre-trained basic OCR model to demonstrate exceptional performance across various downstream tasks. To this end, we propose a parameter-efficient hybrid text recognition method based on pre-trained OCR Transformer, namely DLoRA-TrOCR. This method embeds DoRA into the image encoder and LoRA into the internal structure of the text decoder, enabling efficient parameter fine-tuning for downstream tasks. Experimental results show that compared to similar parameter adjustment methods, our model DLoRA-TrOCR has the smallest number of parameters and performs better. It can achieve state-of-the-art performance on complex scene data sets involving simultaneous recognition of mixed handwritten, printed and street view texts.

Title: Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches

• Authors: Pablo Biedma, Xiaoyuan Yi, Linus Huang, Maosong Sun, Xing Xie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12744
• Pdf URL: https://arxiv.org/pdf/2404.12744
• Copy Paste: [[2404.12744]] Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches(https://arxiv.org/abs/2404.12744)
• Keywords: generative, large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have revolutionized the AI field but also pose potential safety and ethical risks. Deciphering LLMs' embedded values becomes crucial for assessing and mitigating their risks. Despite extensive investigation into LLMs' values, previous studies heavily rely on human-oriented value systems in social sciences. Then, a natural question arises: Do LLMs possess unique values beyond those of humans? Delving into it, this work proposes a novel framework, ValueLex, to reconstruct LLMs' unique value system from scratch, leveraging psychological methodologies from human personality/value research. Based on Lexical Hypothesis, ValueLex introduces a generative approach to elicit diverse values from 30+ LLMs, synthesizing a taxonomy that culminates in a comprehensive value framework via factor analysis and semantic clustering. We identify three core value dimensions, Competence, Character, and Integrity, each with specific subdimensions, revealing that LLMs possess a structured, albeit non-human, value system. Based on this system, we further develop tailored projective tests to evaluate and analyze the value inclinations of LLMs across different model sizes, training methods, and data sources. Our framework fosters an interdisciplinary paradigm of understanding LLMs, paving the way for future AI alignment and regulation.

Title: AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation

• Authors: Wenhao Huang, Chenghao Peng, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Liqian Wen, Zulong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12753
• Pdf URL: https://arxiv.org/pdf/2404.12753
• Copy Paste: [[2404.12753]] AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation(https://arxiv.org/abs/2404.12753)
• Keywords: generative, large language model
• Abstract: Web automation is a significant technique that accomplishes complicated web tasks by automating common web actions, enhancing operational efficiency, and reducing the need for manual intervention. Traditional methods, such as wrappers, suffer from limited adaptability and scalability when faced with a new website. On the other hand, generative agents empowered by large language models (LLMs) exhibit poor performance and reusability in open-world scenarios. In this work, we introduce a crawler generation task for vertical information web pages and the paradigm of combining LLMs with crawlers, which helps crawlers handle diverse and changing web environments more efficiently. We propose AutoCrawler, a two-stage framework that leverages the hierarchical structure of HTML for progressive understanding. Through top-down and step-back operations, AutoCrawler can learn from erroneous actions and continuously prune HTML for better action generation. We conduct comprehensive experiments with multiple LLMs and demonstrate the effectiveness of our framework. Resources of this paper can be found at \url{https://github.com/EZ-hwh/AutoCrawler}

Title: decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points

• Authors: Yi Guo, Fanliu Kong, Xiaoyang Li, Hui Li, Wei Chen, Xiaogang Tian, Jinping Cai, Yang Zhang, Shouda Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12759
• Pdf URL: https://arxiv.org/pdf/2404.12759
• Copy Paste: [[2404.12759]] decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points(https://arxiv.org/abs/2404.12759)
• Keywords: robust
• Abstract: Quantization emerges as one of the most promising compression technologies for deploying efficient large models for various real time application in recent years. Considering that the storage and IO of weights take up the vast majority of the overhead inside a large model, weight only quantization can lead to large gains. However, existing quantization schemes suffer from significant accuracy degradation at very low bits, or require some additional computational overhead when deployed, making it difficult to be applied to large-scale applications in industry. In this paper, we propose decoupleQ, achieving a substantial increase in model accuracy, especially at very low bits. decoupleQ abandons the traditional heuristic quantization paradigm and decouples the model parameters into integer and floating-point parts, thus transforming the quantization problem into a traditional mathematical optimization problem with constraints, which is then solved alternatively by off-the-shelf optimization methods. Quantization via decoupleQ is linear and uniform, making it hardware-friendlier than non-uniform counterpart, and enabling the idea to be migrated to high-bit quantization to enhance its robustness. Our method has achieved well on-line accuracy near fp16/bf16 on the 2-bit quantization of large speech models in ByteDance. The code is available at https://github.com/bytedance/decoupleQ

Title: Camera Agnostic Two-Head Network for Ego-Lane Inference

• Authors: Chaehyeon Song, Sungho Yoon, Minhyeok Heo, Ayoung Kim, Sujung Kim
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.12770
• Pdf URL: https://arxiv.org/pdf/2404.12770
• Copy Paste: [[2404.12770]] Camera Agnostic Two-Head Network for Ego-Lane Inference(https://arxiv.org/abs/2404.12770)
• Keywords: robust
• Abstract: Vision-based ego-lane inference using High-Definition (HD) maps is essential in autonomous driving and advanced driver assistance systems. The traditional approach necessitates well-calibrated cameras, which confines variation of camera configuration, as the algorithm relies on intrinsic and extrinsic calibration. In this paper, we propose a learning-based ego-lane inference by directly estimating the ego-lane index from a single image. To enhance robust performance, our model incorporates the two-head structure inferring ego-lane in two perspectives simultaneously. Furthermore, we utilize an attention mechanism guided by vanishing point-and-line to adapt to changes in viewpoint without requiring accurate calibration. The high adaptability of our model was validated in diverse environments, devices, and camera mounting points and orientations.

Title: Defending against Data Poisoning Attacks in Federated Learning via User Elimination

• Authors: Nick Galanis
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12778
• Pdf URL: https://arxiv.org/pdf/2404.12778
• Copy Paste: [[2404.12778]] Defending against Data Poisoning Attacks in Federated Learning via User Elimination(https://arxiv.org/abs/2404.12778)
• Keywords: privacy, attack, federate
• Abstract: In the evolving landscape of Federated Learning (FL), a new type of attacks concerns the research community, namely Data Poisoning Attacks, which threaten the model integrity by maliciously altering training data. This paper introduces a novel defensive framework focused on the strategic elimination of adversarial users within a federated model. We detect those anomalies in the aggregation phase of the Federated Algorithm, by integrating metadata gathered by the local training instances with Differential Privacy techniques, to ensure that no data leakage is possible. To our knowledge, this is the first proposal in the field of FL that leverages metadata other than the model's gradients in order to ensure honesty in the reported local models. Our extensive experiments demonstrate the efficacy of our methods, significantly mitigating the risk of data poisoning while maintaining user privacy and model performance. Our findings suggest that this new user elimination approach serves us with a great balance between privacy and utility, thus contributing to the arsenal of arguments in favor of the safe adoption of FL in safe domains, both in academic setting and in the industry.

Title: Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting

• Authors: Fengyi Fu, Shancheng Fang, Weidong Chen, Zhendong Mao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12782
• Pdf URL: https://arxiv.org/pdf/2404.12782
• Copy Paste: [[2404.12782]] Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting(https://arxiv.org/abs/2404.12782)
• Keywords: transformer
• Abstract: Automatic live video commenting is with increasing attention due to its significance in narration generation, topic explanation, etc. However, the diverse sentiment consideration of the generated comments is missing from the current methods. Sentimental factors are critical in interactive commenting, and lack of research so far. Thus, in this paper, we propose a Sentiment-oriented Transformer-based Variational Autoencoder (So-TVAE) network which consists of a sentiment-oriented diversity encoder module and a batch attention module, to achieve diverse video commenting with multiple sentiments and multiple semantics. Specifically, our sentiment-oriented diversity encoder elegantly combines VAE and random mask mechanism to achieve semantic diversity under sentiment guidance, which is then fused with cross-modal features to generate live video comments. Furthermore, a batch attention module is also proposed in this paper to alleviate the problem of missing sentimental samples, caused by the data imbalance, which is common in live videos as the popularity of videos varies. Extensive experiments on Livebot and VideoIC datasets demonstrate that the proposed So-TVAE outperforms the state-of-the-art methods in terms of the quality and diversity of generated comments. Related code is available at https://github.com/fufy1024/So-TVAE.

Title: A Proactive Decoy Selection Scheme for Cyber Deception using MITRE ATT&CK

• Authors: Marco Zambianco, Claudio Facchinetti, Domenico Siracusa
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12783
• Pdf URL: https://arxiv.org/pdf/2404.12783
• Copy Paste: [[2404.12783]] A Proactive Decoy Selection Scheme for Cyber Deception using MITRE ATT&CK(https://arxiv.org/abs/2404.12783)
• Keywords: defense, attack, steal
• Abstract: Cyber deception allows compensating the late response of defenders countermeasures to the ever evolving tactics, techniques, and procedures (TTPs) of attackers. This proactive defense strategy employs decoys resembling legitimate system components to lure stealthy attackers within the defender environment, slowing and/or denying the accomplishment of their goals. In this regard, the selection of decoys that can expose the techniques used by malicious users plays a central role to incentivize their engagement. However, this is a difficult task to achieve in practice, since it requires an accurate and realistic modeling of the attacker capabilities and his possible targets. In this work, we tackle this challenge and we design a decoy selection scheme that is supported by an adversarial modeling based on empirical observation of real-world attackers. We take advantage of a domain-specific threat modelling language using MITRE ATT&CK framework as source of attacker TTPs targeting enterprise systems. In detail, we extract the information about the execution preconditions of each technique as well as its possible effects on the environment to generate attack graphs modeling the adversary capabilities. Based on this, we formulate a graph partition problem that minimizes the number of decoys detecting a corresponding number of techniques employed in various attack paths directed to specific targets. We compare our optimization-based decoy selection approach against several benchmark schemes that ignore the preconditions between the various attack steps. Results reveal that the proposed scheme provides the highest interception rate of attack paths using the lowest amount of decoys.

Title: Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation

• Authors: Myrna C. Silva, Mahtab Dahaghin, Matteo Toso, Alessio Del Bue
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12784
• Pdf URL: https://arxiv.org/pdf/2404.12784
• Copy Paste: [[2404.12784]] Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation(https://arxiv.org/abs/2404.12784)
• Keywords: segmentation
• Abstract: We introduce Contrastive Gaussian Clustering, a novel approach capable of provide segmentation masks from any viewpoint and of enabling 3D segmentation of the scene. Recent works in novel-view synthesis have shown how to model the appearance of a scene via a cloud of 3D Gaussians, and how to generate accurate images from a given viewpoint by projecting on it the Gaussians before $\alpha$ blending their color. Following this example, we train a model to include also a segmentation feature vector for each Gaussian. These can then be used for 3D scene segmentation, by clustering Gaussians according to their feature vectors; and to generate 2D segmentation masks, by projecting the Gaussians on a plane and $\alpha$ blending over their segmentation features. Using a combination of contrastive learning and spatial regularization, our method can be trained on inconsistent 2D segmentation masks, and still learn to generate segmentation masks consistent across all views. Moreover, the resulting model is extremely accurate, improving the IoU accuracy of the predicted masks by $+8\%$ over the state of the art. Code and trained models will be released soon.

Title: REXEL: An End-to-end Model for Document-Level Relation Extraction and Entity Linking

• Authors: Nacime Bouziani, Shubhi Tyagi, Joseph Fisher, Jens Lehmann, Andrea Pierleoni
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12788
• Pdf URL: https://arxiv.org/pdf/2404.12788
• Copy Paste: [[2404.12788]] REXEL: An End-to-end Model for Document-Level Relation Extraction and Entity Linking(https://arxiv.org/abs/2404.12788)
• Keywords: extraction
• Abstract: Extracting structured information from unstructured text is critical for many downstream NLP applications and is traditionally achieved by closed information extraction (cIE). However, existing approaches for cIE suffer from two limitations: (i) they are often pipelines which makes them prone to error propagation, and/or (ii) they are restricted to sentence level which prevents them from capturing long-range dependencies and results in expensive inference time. We address these limitations by proposing REXEL, a highly efficient and accurate model for the joint task of document level cIE (DocIE). REXEL performs mention detection, entity typing, entity disambiguation, coreference resolution and document-level relation classification in a single forward pass to yield facts fully linked to a reference knowledge graph. It is on average 11 times faster than competitive existing approaches in a similar setting and performs competitively both when optimised for any of the individual subtasks and a variety of combinations of different joint tasks, surpassing the baselines by an average of more than 6 F1 points. The combination of speed and accuracy makes REXEL an accurate cost-efficient system for extracting structured information at web-scale. We also release an extension of the DocRED dataset to enable benchmarking of future work on DocIE, which is available at https://github.com/amazon-science/e2e-docie.

Title: MambaMOS: LiDAR-based 3D Moving Object Segmentation with Motion-aware State Space Model

• Authors: Kang Zeng, Hao Shi, Jiacheng Lin, Siyu Li, Jintao Cheng, Kaiwei Wang, Zhiyong Li, Kailun Yang
• Subjects: cs.CV, cs.MM, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.12794
• Pdf URL: https://arxiv.org/pdf/2404.12794
• Copy Paste: [[2404.12794]] MambaMOS: LiDAR-based 3D Moving Object Segmentation with Motion-aware State Space Model(https://arxiv.org/abs/2404.12794)
• Keywords: segmentation
• Abstract: LiDAR-based Moving Object Segmentation (MOS) aims to locate and segment moving objects in point clouds of the current scan using motion information from previous scans. Despite the promising results achieved by previous MOS methods, several key issues, such as the weak coupling of temporal and spatial information, still need further study. In this paper, we propose a novel LiDAR-based 3D Moving Object Segmentation with Motion-aware State Space Model, termed MambaMOS. Firstly, we develop a novel embedding module, the Time Clue Bootstrapping Embedding (TCBE), to enhance the coupling of temporal and spatial information in point clouds and alleviate the issue of overlooked temporal clues. Secondly, we introduce the Motion-aware State Space Model (MSSM) to endow the model with the capacity to understand the temporal correlations of the same object across different time steps. Specifically, MSSM emphasizes the motion states of the same object at different time steps through two distinct temporal modeling and correlation steps. We utilize an improved state space model to represent these motion differences, significantly modeling the motion states. Finally, extensive experiments on the SemanticKITTI-MOS and KITTI-Road benchmarks demonstrate that the proposed MambaMOS achieves state-of-the-art performance. The source code of this work will be made publicly available at https://github.com/Terminal-K/MambaMOS.

Title: A Point-Based Approach to Efficient LiDAR Multi-Task Perception

• Authors: Christopher Lang, Alexander Braun, Lars Schillingmann, Abhinav Valada
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12798
• Pdf URL: https://arxiv.org/pdf/2404.12798
• Copy Paste: [[2404.12798]] A Point-Based Approach to Efficient LiDAR Multi-Task Perception(https://arxiv.org/abs/2404.12798)
• Keywords: transformer, segmentation
• Abstract: Multi-task networks can potentially improve performance and computational efficiency compared to single-task networks, facilitating online deployment. However, current multi-task architectures in point cloud perception combine multiple task-specific point cloud representations, each requiring a separate feature encoder and making the network structures bulky and slow. We propose PAttFormer, an efficient multi-task architecture for joint semantic segmentation and object detection in point clouds that only relies on a point-based representation. The network builds on transformer-based feature encoders using neighborhood attention and grid-pooling and a query-based detection decoder using a novel 3D deformable-attention detection head design. Unlike other LiDAR-based multi-task architectures, our proposed PAttFormer does not require separate feature encoders for multiple task-specific point cloud representations, resulting in a network that is 3x smaller and 1.4x faster while achieving competitive performance on the nuScenes and KITTI benchmarks for autonomous driving perception. Our extensive evaluations show substantial gains from multi-task learning, improving LiDAR semantic segmentation by +1.7% in mIou and 3D object detection by +1.7% in mAP on the nuScenes benchmark compared to the single-task models.

Title: TextSquare: Scaling up Text-Centric Visual Instruction Tuning

• Authors: Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng, Yang Li, Siqi Wang, Lei Liao, Wei Shi, Yuliang Liu, Hao Liu, Yuan Xie, Xiang Bai, Can Huang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12803
• Pdf URL: https://arxiv.org/pdf/2404.12803
• Copy Paste: [[2404.12803]] TextSquare: Scaling up Text-Centric Visual Instruction Tuning(https://arxiv.org/abs/2404.12803)
• Keywords: large language model
• Abstract: Text-centric visual question answering (VQA) has made great strides with the development of Multimodal Large Language Models (MLLMs), yet open-source models still fall short of leading models like GPT4V and Gemini, partly due to a lack of extensive, high-quality instruction tuning data. To this end, we introduce a new approach for creating a massive, high-quality instruction-tuning dataset, Square-10M, which is generated using closed-source MLLMs. The data construction process, termed Square, consists of four steps: Self-Questioning, Answering, Reasoning, and Evaluation. Our experiments with Square-10M led to three key findings: 1) Our model, TextSquare, considerably surpasses open-source previous state-of-the-art Text-centric MLLMs and sets a new standard on OCRBench(62.2%). It even outperforms top-tier models like GPT4V and Gemini in 6 of 10 text-centric benchmarks. 2) Additionally, we demonstrate the critical role of VQA reasoning data in offering comprehensive contextual insights for specific questions. This not only improves accuracy but also significantly mitigates hallucinations. Specifically, TextSquare scores an average of 75.1% across four general VQA and hallucination evaluation datasets, outperforming previous state-of-the-art models. 3) Notably, the phenomenon observed in scaling text-centric VQA datasets reveals a vivid pattern: the exponential increase of instruction tuning data volume is directly proportional to the improvement in model performance, thereby validating the necessity of the dataset scale and the high quality of Square-10M.

Title: Linearly-evolved Transformer for Pan-sharpening

• Authors: Junming Hou, Zihan Cao, Naishan Zheng, Xuan Li, Xiaoyu Chen, Xinyang Liu, Xiaofeng Cong, Man Zhou, Danfeng Hong
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.12804
• Pdf URL: https://arxiv.org/pdf/2404.12804
• Copy Paste: [[2404.12804]] Linearly-evolved Transformer for Pan-sharpening(https://arxiv.org/abs/2404.12804)
• Keywords: transformer
• Abstract: Vision transformer family has dominated the satellite pan-sharpening field driven by the global-wise spatial information modeling mechanism from the core self-attention ingredient. The standard modeling rules within these promising pan-sharpening methods are to roughly stack the transformer variants in a cascaded manner. Despite the remarkable advancement, their success may be at the huge cost of model parameters and FLOPs, thus preventing its application over low-resource satellites.To address this challenge between favorable performance and expensive computation, we tailor an efficient linearly-evolved transformer variant and employ it to construct a lightweight pan-sharpening framework. In detail, we deepen into the popular cascaded transformer modeling with cutting-edge methods and develop the alternative 1-order linearly-evolved transformer variant with the 1-dimensional linear convolution chain to achieve the same function. In this way, our proposed method is capable of benefiting the cascaded modeling rule while achieving favorable performance in the efficient manner. Extensive experiments over multiple satellite datasets suggest that our proposed method achieves competitive performance against other state-of-the-art with fewer computational resources. Further, the consistently favorable performance has been verified over the hyper-spectral image fusion task. Our main focus is to provide an alternative global modeling framework with an efficient structure. The code will be publicly available.

Title: Systematic Evaluation of Forensic Data Acquisition using Smartphone Local Backup

• Authors: Julian Geus, Jenny Ottmann, Felix Freiling
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12808
• Pdf URL: https://arxiv.org/pdf/2404.12808
• Copy Paste: [[2404.12808]] Systematic Evaluation of Forensic Data Acquisition using Smartphone Local Backup(https://arxiv.org/abs/2404.12808)
• Keywords: security
• Abstract: Due to the increasing security standards of modern smartphones, forensic data acquisition from such devices is a growing challenge. One rather generic way to access data on smartphones in practice is to use the local backup mechanism offered by the mobile operating systems. We study the suitability of such mechanisms for forensic data acquisition by performing a thorough evaluation of iOS's and Android's local backup mechanisms on two mobile devices. Based on a systematic and generic evaluation procedure comparing the contents of local backup to the original storage, we show that in our exemplary practical evaluations, in most cases (but not all) local backup actually yields a correct copy of the original data from storage. Our study also highlights corner cases, such as database files with pending changes, that need to be considered when assessing the integrity and authenticity of evidence acquired through local backup.

Title: Enhancing Counterfactual Explanation Search with Diffusion Distance and Directional Coherence

• Authors: Marharyta Domnich, Raul Vicente
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12810
• Pdf URL: https://arxiv.org/pdf/2404.12810
• Copy Paste: [[2404.12810]] Enhancing Counterfactual Explanation Search with Diffusion Distance and Directional Coherence(https://arxiv.org/abs/2404.12810)
• Keywords: diffusion
• Abstract: A pressing issue in the adoption of AI models is the increasing demand for more human-centric explanations of their predictions. To advance towards more human-centric explanations, understanding how humans produce and select explanations has been beneficial. In this work, inspired by insights of human cognition we propose and test the incorporation of two novel biases to enhance the search for effective counterfactual explanations. Central to our methodology is the application of diffusion distance, which emphasizes data connectivity and actionability in the search for feasible counterfactual explanations. In particular, diffusion distance effectively weights more those points that are more interconnected by numerous short-length paths. This approach brings closely connected points nearer to each other, identifying a feasible path between them. We also introduce a directional coherence term that allows the expression of a preference for the alignment between the joint and marginal directional changes in feature space to reach a counterfactual. This term enables the generation of counterfactual explanations that align with a set of marginal predictions based on expectations of how the outcome of the model varies by changing one feature at a time. We evaluate our method, named Coherent Directional Counterfactual Explainer (CoDiCE), and the impact of the two novel biases against existing methods such as DiCE, FACE, Prototypes, and Growing Spheres. Through a series of ablation experiments on both synthetic and real datasets with continuous and mixed-type features, we demonstrate the effectiveness of our method.

Title: Generative Modelling with High-Order Langevin Dynamics

• Authors: Ziqiang Shi, Rujie Liu
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12814
• Pdf URL: https://arxiv.org/pdf/2404.12814
• Copy Paste: [[2404.12814]] Generative Modelling with High-Order Langevin Dynamics(https://arxiv.org/abs/2404.12814)
• Keywords: diffusion, generative
• Abstract: Diffusion generative modelling (DGM) based on stochastic differential equations (SDEs) with score matching has achieved unprecedented results in data generation. In this paper, we propose a novel fast high-quality generative modelling method based on high-order Langevin dynamics (HOLD) with score matching. This motive is proved by third-order Langevin dynamics. By augmenting the previous SDEs, e.g. variance exploding or variance preserving SDEs for single-data variable processes, HOLD can simultaneously model position, velocity, and acceleration, thereby improving the quality and speed of the data generation at the same time. HOLD is composed of one Ornstein-Uhlenbeck process and two Hamiltonians, which reduce the mixing time by two orders of magnitude. Empirical experiments for unconditional image generation on the public data set CIFAR-10 and CelebA-HQ show that the effect is significant in both Frechet inception distance (FID) and negative log-likelihood, and achieves the state-of-the-art FID of 1.85 on CIFAR-10.

Title: LiMe: a Latin Corpus of Late Medieval Criminal Sentences

• Authors: Alessandra Bassani, Beatrice Del Bo, Alfio Ferrara, Marta Mangini, Sergio Picascia, Ambra Stefanello
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12829
• Pdf URL: https://arxiv.org/pdf/2404.12829
• Copy Paste: [[2404.12829]] LiMe: a Latin Corpus of Late Medieval Criminal Sentences(https://arxiv.org/abs/2404.12829)
• Keywords: large language model
• Abstract: The Latin language has received attention from the computational linguistics research community, which has built, over the years, several valuable resources, ranging from detailed annotated corpora to sophisticated tools for linguistic analysis. With the recent advent of large language models, researchers have also started developing models capable of generating vector representations of Latin texts. The performances of such models remain behind the ones for modern languages, given the disparity in available data. In this paper, we present the LiMe dataset, a corpus of 325 documents extracted from a series of medieval manuscripts called Libri sententiarum potestatis Mediolani, and thoroughly annotated by experts, in order to be employed for masked language model, as well as supervised natural language processing tasks.

Title: COIN: Counterfactual inpainting for weakly supervised semantic segmentation for medical images

• Authors: Dmytro Shvetsov, Joonas Ariva, Marharyta Domnich, Raul Vicente, Dmytro Fishman
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12832
• Pdf URL: https://arxiv.org/pdf/2404.12832
• Copy Paste: [[2404.12832]] COIN: Counterfactual inpainting for weakly supervised semantic segmentation for medical images(https://arxiv.org/abs/2404.12832)
• Keywords: generative, segmentation
• Abstract: Deep learning is dramatically transforming the field of medical imaging and radiology, enabling the identification of pathologies in medical images, including computed tomography (CT) and X-ray scans. However, the performance of deep learning models, particularly in segmentation tasks, is often limited by the need for extensive annotated datasets. To address this challenge, the capabilities of weakly supervised semantic segmentation are explored through the lens of Explainable AI and the generation of counterfactual explanations. The scope of this research is development of a novel counterfactual inpainting approach (COIN) that flips the predicted classification label from abnormal to normal by using a generative model. For instance, if the classifier deems an input medical image X as abnormal, indicating the presence of a pathology, the generative model aims to inpaint the abnormal region, thus reversing the classifier's original prediction label. The approach enables us to produce precise segmentations for pathologies without depending on pre-existing segmentation masks. Crucially, image-level labels are utilized, which are substantially easier to acquire than creating detailed segmentation masks. The effectiveness of the method is demonstrated by segmenting synthetic targets and actual kidney tumors from CT images acquired from Tartu University Hospital in Estonia. The findings indicate that COIN greatly surpasses established attribution methods, such as RISE, ScoreCAM, and LayerCAM, as well as an alternative counterfactual explanation method introduced by Singla et al. This evidence suggests that COIN is a promising approach for semantic segmentation of tumors in CT images, and presents a step forward in making deep learning applications more accessible and effective in healthcare, where annotated data is scarce.

Title: Towards a decentralized data privacy protocol for self-sovereignty in the digital world

• Authors: Rodrigo Falcão, Arghavan Hosseinzadeh
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.12837
• Pdf URL: https://arxiv.org/pdf/2404.12837
• Copy Paste: [[2404.12837]] Towards a decentralized data privacy protocol for self-sovereignty in the digital world(https://arxiv.org/abs/2404.12837)
• Keywords: privacy
• Abstract: A typical user interacts with many digital services nowadays, providing these services with their data. As of now, the management of privacy preferences is service-centric: Users must manage their privacy preferences according to the rules of each service provider, meaning that every provider offers its unique mechanisms for users to control their privacy settings. However, managing privacy preferences holistically (i.e., across multiple digital services) is just impractical. In this vision paper, we propose a paradigm shift towards an enriched user-centric approach for cross-service privacy preferences management: the realization of a decentralized data privacy protocol.

Title: ECOR: Explainable CLIP for Object Recognition

• Authors: Ali Rasekh, Sepehr Kazemi Ranjbar, Milad Heidari, Wolfgang Nejdl
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12839
• Pdf URL: https://arxiv.org/pdf/2404.12839
• Copy Paste: [[2404.12839]] ECOR: Explainable CLIP for Object Recognition(https://arxiv.org/abs/2404.12839)
• Keywords: explainability
• Abstract: Large Vision Language Models (VLMs), such as CLIP, have significantly contributed to various computer vision tasks, including object recognition and object detection. Their open vocabulary feature enhances their value. However, their black-box nature and lack of explainability in predictions make them less trustworthy in critical domains. Recently, some work has been done to force VLMs to provide reasonable rationales for object recognition, but this often comes at the expense of classification accuracy. In this paper, we first propose a mathematical definition of explainability in the object recognition task based on the joint probability distribution of categories and rationales, then leverage this definition to fine-tune CLIP in an explainable manner. Through evaluations of different datasets, our method demonstrates state-of-the-art performance in explainable classification. Notably, it excels in zero-shot settings, showcasing its adaptability. This advancement improves explainable object recognition, enhancing trust across diverse applications. The code will be made available online upon publication.

Title: Explainable Deepfake Video Detection using Convolutional Neural Network and CapsuleNet

• Authors: Gazi Hasin Ishrak, Zalish Mahmud, MD. Zami Al Zunaed Farabe, Tahera Khanom Tinni, Tanzim Reza, Mohammad Zavid Parvez
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.12841
• Pdf URL: https://arxiv.org/pdf/2404.12841
• Copy Paste: [[2404.12841]] Explainable Deepfake Video Detection using Convolutional Neural Network and CapsuleNet(https://arxiv.org/abs/2404.12841)
• Keywords: security, generative
• Abstract: Deepfake technology, derived from deep learning, seamlessly inserts individuals into digital media, irrespective of their actual participation. Its foundation lies in machine learning and Artificial Intelligence (AI). Initially, deepfakes served research, industry, and entertainment. While the concept has existed for decades, recent advancements render deepfakes nearly indistinguishable from reality. Accessibility has soared, empowering even novices to create convincing deepfakes. However, this accessibility raises security concerns.The primary deepfake creation algorithm, GAN (Generative Adversarial Network), employs machine learning to craft realistic images or videos. Our objective is to utilize CNN (Convolutional Neural Network) and CapsuleNet with LSTM to differentiate between deepfake-generated frames and originals. Furthermore, we aim to elucidate our model's decision-making process through Explainable AI, fostering transparent human-AI relationships and offering practical examples for real-life scenarios.

Title: Towards Logically Consistent Language Models via Probabilistic Reasoning

• Authors: Diego Calanzone, Stefano Teso, Antonio Vergari
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12843
• Pdf URL: https://arxiv.org/pdf/2404.12843
• Copy Paste: [[2404.12843]] Towards Logically Consistent Language Models via Probabilistic Reasoning(https://arxiv.org/abs/2404.12843)
• Keywords: large language model
• Abstract: Large language models (LLMs) are a promising venue for natural language understanding and generation tasks. However, current LLMs are far from reliable: they are prone to generate non-factual information and, more crucially, to contradict themselves when prompted to reason about beliefs of the world. These problems are currently addressed with large scale fine-tuning or by delegating consistent reasoning to external tools. In this work, we strive for a middle ground and introduce a training objective based on principled probabilistic reasoning that teaches a LLM to be consistent with external knowledge in the form of a set of facts and rules. Fine-tuning with our loss on a limited set of facts enables our LLMs to be more logically consistent than previous baselines and allows them to extrapolate to unseen but semantically similar factual knowledge more systematically.

Title: KoReA-SFL: Knowledge Replay-based Split Federated Learning Against Catastrophic Forgetting

• Authors: Zeke Xia, Ming Hu, Dengke Yan, Ruixuan Liu, Anran Li, Xiaofei Xie, Mingsong Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12846
• Pdf URL: https://arxiv.org/pdf/2404.12846
• Copy Paste: [[2404.12846]] KoReA-SFL: Knowledge Replay-based Split Federated Learning Against Catastrophic Forgetting(https://arxiv.org/abs/2404.12846)
• Keywords: federate
• Abstract: Although Split Federated Learning (SFL) is good at enabling knowledge sharing among resource-constrained clients, it suffers from the problem of low training accuracy due to the neglect of data heterogeneity and catastrophic forgetting. To address this issue, we propose a novel SFL approach named KoReA-SFL, which adopts a multi-model aggregation mechanism to alleviate gradient divergence caused by heterogeneous data and a knowledge replay strategy to deal with catastrophic forgetting. Specifically, in KoReA-SFL cloud servers (i.e., fed server and main server) maintain multiple branch model portions rather than a global portion for local training and an aggregated master-model portion for knowledge sharing among branch portions. To avoid catastrophic forgetting, the main server of KoReA-SFL selects multiple assistant devices for knowledge replay according to the training data distribution of each server-side branch-model portion. Experimental results obtained from non-IID and IID scenarios demonstrate that KoReA-SFL significantly outperforms conventional SFL methods (by up to 23.25\% test accuracy improvement).

Title: CaBaFL: Asynchronous Federated Learning via Hierarchical Cache and Feature Balance

• Authors: Zeke Xia, Ming Hu, Dengke Yan, Xiaofei Xie, Tianlin Li, Anran Li, Junlong Zhou, Mingsong Chen
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.12850
• Pdf URL: https://arxiv.org/pdf/2404.12850
• Copy Paste: [[2404.12850]] CaBaFL: Asynchronous Federated Learning via Hierarchical Cache and Feature Balance(https://arxiv.org/abs/2404.12850)
• Keywords: federate
• Abstract: Federated Learning (FL) as a promising distributed machine learning paradigm has been widely adopted in Artificial Intelligence of Things (AIoT) applications. However, the efficiency and inference capability of FL is seriously limited due to the presence of stragglers and data imbalance across massive AIoT devices, respectively. To address the above challenges, we present a novel asynchronous FL approach named CaBaFL, which includes a hierarchical Cache-based aggregation mechanism and a feature Balance-guided device selection strategy. CaBaFL maintains multiple intermediate models simultaneously for local training. The hierarchical cache-based aggregation mechanism enables each intermediate model to be trained on multiple devices to align the training time and mitigate the straggler issue. In specific, each intermediate model is stored in a low-level cache for local training and when it is trained by sufficient local devices, it will be stored in a high-level cache for aggregation. To address the problem of imbalanced data, the feature balance-guided device selection strategy in CaBaFL adopts the activation distribution as a metric, which enables each intermediate model to be trained across devices with totally balanced data distributions before aggregation. Experimental results show that compared with the state-of-the-art FL methods, CaBaFL achieves up to 9.26X training acceleration and 19.71\% accuracy improvements.

Title: LSP Framework: A Compensatory Model for Defeating Trigger Reverse Engineering via Label Smoothing Poisoning

• Authors: Beichen Li, Yuanfang Guo, Heqi Peng, Yangxi Li, Yunhong Wang
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12852
• Pdf URL: https://arxiv.org/pdf/2404.12852
• Copy Paste: [[2404.12852]] LSP Framework: A Compensatory Model for Defeating Trigger Reverse Engineering via Label Smoothing Poisoning(https://arxiv.org/abs/2404.12852)
• Keywords: defense, attack
• Abstract: Deep neural networks are vulnerable to backdoor attacks. Among the existing backdoor defense methods, trigger reverse engineering based approaches, which reconstruct the backdoor triggers via optimizations, are the most versatile and effective ones compared to other types of methods. In this paper, we summarize and construct a generic paradigm for the typical trigger reverse engineering process. Based on this paradigm, we propose a new perspective to defeat trigger reverse engineering by manipulating the classification confidence of backdoor samples. To determine the specific modifications of classification confidence, we propose a compensatory model to compute the lower bound of the modification. With proper modifications, the backdoor attack can easily bypass the trigger reverse engineering based methods. To achieve this objective, we propose a Label Smoothing Poisoning (LSP) framework, which leverages label smoothing to specifically manipulate the classification confidences of backdoor samples. Extensive experiments demonstrate that the proposed work can defeat the state-of-the-art trigger reverse engineering based methods, and possess good compatibility with a variety of existing backdoor attacks.

Title: Migrating Software Systems towards Post-Quantum-Cryptography -- A Systematic Literature Review

• Authors: Christian Näther, Daniel Herzinger, Stefan-Lukas Gazdag, Jan-Philipp Steghöfer, Simon Daum, Daniel Loebenberger
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12854
• Pdf URL: https://arxiv.org/pdf/2404.12854
• Copy Paste: [[2404.12854]] Migrating Software Systems towards Post-Quantum-Cryptography -- A Systematic Literature Review(https://arxiv.org/abs/2404.12854)
• Keywords: security
• Abstract: Networks such as the Internet are essential for our connected world. Quantum computing poses a threat to this heterogeneous infrastructure since it threatens fundamental security mechanisms. Therefore, a migration to post-quantum-cryptography (PQC) is necessary for networks and their components. At the moment, there is little knowledge on how such migrations should be structured and implemented in practice. Our systematic literature review addresses migration approaches for IP networks towards PQC. It surveys papers about the migration process and exemplary real-world software system migrations. On the process side, we found that terminology, migration steps, and roles are not defined precisely or consistently across the literature. Still, we identified four major phases and appropriate substeps which we matched with also emerging archetypes of roles. In terms of real-world migrations, we see that reports used several different PQC implementations and hybrid solutions for migrations of systems belonging to a wide range of system types. Across all papers we noticed three major challenges for adopters: missing experience of PQC and a high realization effort, concerns about the security of the upcoming system, and finally, high complexity. Our findings indicate that recent standardization efforts already push quantum-safe networking forward. However, the literature is still not in consensus about definitions and best practices. Implementations are mostly experimental and not necessarily practical, leading to an overall chaotic situation. To better grasp this fast moving field of (applied) research, our systematic literature review provides a comprehensive overview of its current state and serves as a starting point for delving into the matter of PQC migration.

Title: Ransomware Detection and Classification Using Random Forest: A Case Study with the UGRansome2024 Dataset

• Authors: Peace Azugo, Hein Venter, Mike Wa Nkongolo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12855
• Pdf URL: https://arxiv.org/pdf/2404.12855
• Copy Paste: [[2404.12855]] Ransomware Detection and Classification Using Random Forest: A Case Study with the UGRansome2024 Dataset(https://arxiv.org/abs/2404.12855)
• Keywords: security, protect
• Abstract: Cybersecurity faces challenges in identifying and mitigating ransomware, which is important for protecting critical infrastructures. The absence of datasets for distinguishing normal versus abnormal network behaviour hinders the development of proactive detection strategies against ransomware. An obstacle in proactive prevention methods is the absence of comprehensive datasets for contrasting normal versus abnormal network behaviours. The dataset enabling such contrasts would significantly expedite threat anomaly mitigation. In this study, we introduce UGRansome2024, an optimised dataset for ransomware detection in network traffic. This dataset is derived from the UGRansome data using an intuitionistic feature engineering approach that considers only relevant patterns in network behaviour analysis. The study presents an analysis of ransomware detection using the UGRansome2024 dataset and the Random Forest algorithm. Through encoding and feature relevance determination, the Random Forest achieved a classification accuracy of 96% and effectively identified unusual ransomware transactions. Findings indicate that certain ransomware variants, such as those utilising Encrypt Decrypt Algorithms (EDA) and Globe ransomware, have the highest financial impact. These insights have significant implications for real-world cybersecurity practices, highlighting the importance of machine learning in ransomware detection and mitigation. Further research is recommended to expand datasets, explore alternative detection methods, and address limitations in current approaches.

Title: Foundation Model assisted Weakly Supervised LiDAR Semantic Segmentation

• Authors: Yilong Chen, Zongyi Xu, xiaoshui Huang, Ruicheng Zhang, Xinqi Jiang, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12861
• Pdf URL: https://arxiv.org/pdf/2404.12861
• Copy Paste: [[2404.12861]] Foundation Model assisted Weakly Supervised LiDAR Semantic Segmentation(https://arxiv.org/abs/2404.12861)
• Keywords: segmentation
• Abstract: Current point cloud semantic segmentation has achieved great advances when given sufficient labels. However, the dense annotation of LiDAR point clouds remains prohibitively expensive and time-consuming, unable to keep up with the continuously growing volume of data. In this paper, we propose annotating images with scattered points, followed by utilizing SAM (a Foundation model) to generate semantic segmentation labels for the images. Finally, by mapping the segmentation labels of the images to the LiDAR space using the intrinsic and extrinsic parameters of the camera and LiDAR, we obtain labels for point cloud semantic segmentation, and release Scatter-KITTI and Scatter-nuScenes, which are the first works to utilize image segmentation-based SAM for weakly supervised point cloud semantic segmentation. Furthermore, to mitigate the influence of erroneous pseudo labels obtained from sparse annotations on point cloud features, we propose a multi-modal weakly supervised network for LiDAR semantic segmentation, called MM-ScatterNet. This network combines features from both point cloud and image modalities, enhancing the representation learning of point clouds by introducing consistency constraints between multi-modal features and point cloud features. On the SemanticKITTI dataset, we achieve 66\% of fully supervised performance using only 0.02% of annotated data, and on the NuScenes dataset, we achieve 95% of fully supervised performance using only 0.1% labeled points.

Title: How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?

• Authors: Yang Luo, Zangwei Zheng, Zirui Zhu, Yang You
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12866
• Pdf URL: https://arxiv.org/pdf/2404.12866
• Copy Paste: [[2404.12866]] How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?(https://arxiv.org/abs/2404.12866)
• Keywords: large language model
• Abstract: The increase in parameter size of multimodal large language models (MLLMs) introduces significant capabilities, particularly in-context learning, where MLLMs enhance task performance without updating pre-trained parameters. This effectiveness, however, hinges on the appropriate selection of in-context examples, a process that is currently biased towards visual data, overlooking textual information. Furthermore, the area of supervised retrievers for MLLMs, crucial for optimal in-context example selection, continues to be uninvestigated. Our study offers an in-depth evaluation of the impact of textual information on the unsupervised selection of in-context examples in multimodal contexts, uncovering a notable sensitivity of retriever performance to the employed modalities. Responding to this, we introduce a novel supervised MLLM-retriever MSIER that employs a neural network to select examples that enhance multimodal in-context learning efficiency. This approach is validated through extensive testing across three distinct tasks, demonstrating the method's effectiveness. Additionally, we investigate the influence of modalities on our supervised retrieval method's training and pinpoint factors contributing to our model's success. This exploration paves the way for future advancements, highlighting the potential for refined in-context learning in MLLMs through the strategic use of multimodal data.

Title: FipTR: A Simple yet Effective Transformer Framework for Future Instance Prediction in Autonomous Driving

• Authors: Xingtai Gui, Tengteng Huang, Haonan Shao, Haotian Yao, Chi Zhang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.12867
• Pdf URL: https://arxiv.org/pdf/2404.12867
• Copy Paste: [[2404.12867]] FipTR: A Simple yet Effective Transformer Framework for Future Instance Prediction in Autonomous Driving(https://arxiv.org/abs/2404.12867)
• Keywords: transformer, segmentation
• Abstract: The future instance prediction from a Bird's Eye View(BEV) perspective is a vital component in autonomous driving, which involves future instance segmentation and instance motion prediction. Existing methods usually rely on a redundant and complex pipeline which requires multiple auxiliary outputs and post-processing procedures. Moreover, estimated errors on each of the auxiliary predictions will lead to degradation of the prediction performance. In this paper, we propose a simple yet effective fully end-to-end framework named Future Instance Prediction Transformer(FipTR), which views the task as BEV instance segmentation and prediction for future frames. We propose to adopt instance queries representing specific traffic participants to directly estimate the corresponding future occupied masks, and thus get rid of complex post-processing procedures. Besides, we devise a flow-aware BEV predictor for future BEV feature prediction composed of a flow-aware deformable attention that takes backward flow guiding the offset sampling. A novel future instance matching strategy is also proposed to further improve the temporal coherence. Extensive experiments demonstrate the superiority of FipTR and its effectiveness under different temporal BEV encoders.

Title: A Large-scale Medical Visual Task Adaptation Benchmark

• Authors: Shentong Mo, Xufang Luo, Yansen Wang, Dongsheng Li
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12876
• Pdf URL: https://arxiv.org/pdf/2404.12876
• Copy Paste: [[2404.12876]] A Large-scale Medical Visual Task Adaptation Benchmark(https://arxiv.org/abs/2404.12876)
• Keywords: transformer
• Abstract: Visual task adaptation has been demonstrated to be effective in adapting pre-trained Vision Transformers (ViTs) to general downstream visual tasks using specialized learnable layers or tokens. However, there is yet a large-scale benchmark to fully explore the effect of visual task adaptation on the realistic and important medical domain, particularly across diverse medical visual modalities, such as color images, X-ray, and CT. To close this gap, we present Med-VTAB, a large-scale Medical Visual Task Adaptation Benchmark consisting of 1.68 million medical images for diverse organs, modalities, and adaptation approaches. Based on Med-VTAB, we explore the scaling law of medical prompt tuning concerning tunable parameters and the generalizability of medical visual adaptation using non-medical/medical pre-train weights. Besides, we study the impact of patient ID out-of-distribution on medical visual adaptation, which is a real and challenging scenario. Furthermore, results from Med-VTAB indicate that a single pre-trained model falls short in medical task adaptation. Therefore, we introduce GMoE-Adapter, a novel method that combines medical and general pre-training weights through a gated mixture-of-experts adapter, achieving state-of-the-art results in medical visual task adaptation.

Title: Unlocking Multi-View Insights in Knowledge-Dense Retrieval-Augmented Generation

• Authors: Guanhua Chen, Wenhan Yu, Lei Sha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12879
• Pdf URL: https://arxiv.org/pdf/2404.12879
• Copy Paste: [[2404.12879]] Unlocking Multi-View Insights in Knowledge-Dense Retrieval-Augmented Generation(https://arxiv.org/abs/2404.12879)
• Keywords: interpretability, large language model
• Abstract: While Retrieval-Augmented Generation (RAG) plays a crucial role in the application of Large Language Models (LLMs), existing retrieval methods in knowledge-dense domains like law and medicine still suffer from a lack of multi-perspective views, which are essential for improving interpretability and reliability. Previous research on multi-view retrieval often focused solely on different semantic forms of queries, neglecting the expression of specific domain knowledge perspectives. This paper introduces a novel multi-view RAG framework, MVRAG, tailored for knowledge-dense domains that utilizes intention-aware query rewriting from multiple domain viewpoints to enhance retrieval precision, thereby improving the effectiveness of the final inference. Experiments conducted on legal and medical case retrieval demonstrate significant improvements in recall and precision rates with our framework. Our multi-perspective retrieval approach unleashes the potential of multi-view information enhancing RAG tasks, accelerating the further application of LLMs in knowledge-intensive fields.

Title: MCM: Multi-condition Motion Synthesis Framework

• Authors: Zeyu Ling, Bo Han, Yongkang Wongkan, Han Lin, Mohan Kankanhalli, Weidong Geng
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12886
• Pdf URL: https://arxiv.org/pdf/2404.12886
• Copy Paste: [[2404.12886]] MCM: Multi-condition Motion Synthesis Framework(https://arxiv.org/abs/2404.12886)
• Keywords: diffusion, transformer
• Abstract: Conditional human motion synthesis (HMS) aims to generate human motion sequences that conform to specific conditions. Text and audio represent the two predominant modalities employed as HMS control conditions. While existing research has primarily focused on single conditions, the multi-condition human motion synthesis remains underexplored. In this study, we propose a multi-condition HMS framework, termed MCM, based on a dual-branch structure composed of a main branch and a control branch. This framework effectively extends the applicability of the diffusion model, which is initially predicated solely on textual conditions, to auditory conditions. This extension encompasses both music-to-dance and co-speech HMS while preserving the intrinsic quality of motion and the capabilities for semantic association inherent in the original model. Furthermore, we propose the implementation of a Transformer-based diffusion model, designated as MWNet, as the main branch. This model adeptly apprehends the spatial intricacies and inter-joint correlations inherent in motion sequences, facilitated by the integration of multi-wise self-attention modules. Extensive experiments show that our method achieves competitive results in single-condition and multi-condition HMS tasks.

Title: The Power of Words: Generating PowerShell Attacks from Natural Language

• Authors: Pietro Liguori, Christian Marescalco, Roberto Natella, Vittorio Orbinato, Luciano Pianese
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.12893
• Pdf URL: https://arxiv.org/pdf/2404.12893
• Copy Paste: [[2404.12893]] The Power of Words: Generating PowerShell Attacks from Natural Language(https://arxiv.org/abs/2404.12893)
• Keywords: security, attack
• Abstract: As the Windows OS stands out as one of the most targeted systems, the PowerShell language has become a key tool for malicious actors and cybersecurity professionals (e.g., for penetration testing). This work explores an uncharted domain in AI code generation by automatically generating offensive PowerShell code from natural language descriptions using Neural Machine Translation (NMT). For training and evaluation purposes, we propose two novel datasets with PowerShell code samples, one with manually curated descriptions in natural language and another code-only dataset for reinforcing the training. We present an extensive evaluation of state-of-the-art NMT models and analyze the generated code both statically and dynamically. Results indicate that tuning NMT using our dataset is effective at generating offensive PowerShell code. Comparative analysis against the most widely used LLM service ChatGPT reveals the specialized strengths of our fine-tuned models.

Title: Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning

• Authors: Ahmed Elshabrawy, Yongix Huang, Iryna Gurevych, Alham Fikri Aji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12897
• Pdf URL: https://arxiv.org/pdf/2404.12897
• Copy Paste: [[2404.12897]] Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning(https://arxiv.org/abs/2404.12897)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) exhibit remarkable capabilities in zero-shot and few-shot scenarios, they often require computationally prohibitive sizes. Conversely, smaller Masked Language Models (MLMs) like BERT and RoBERTa achieve state-of-the-art results through fine-tuning but struggle with extending to few-shot and zero-shot settings due to their architectural constraints. Hence, we propose Statement-Tuning, a technique that models discriminative tasks as a set of finite statements and trains an Encoder model to discriminate between the potential statements to determine the label. We do Statement-Tuning on multiple tasks to enable cross-task generalization. Experimental results demonstrate that Statement Tuning achieves competitive performance compared to state-of-the-art LLMs with significantly fewer parameters. Moreover, the study investigates the impact of several design choices on few-shot and zero-shot generalization, revealing that Statement Tuning can achieve sufficient performance with modest training data and benefits from task and statement diversity for unseen task generalizability.

Title: Training-and-prompt-free General Painterly Harmonization Using Image-wise Attention Sharing

• Authors: Teng-Fang Hsiao, Bo-Kai Ruan, Hong-Han Shuai
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.12900
• Pdf URL: https://arxiv.org/pdf/2404.12900
• Copy Paste: [[2404.12900]] Training-and-prompt-free General Painterly Harmonization Using Image-wise Attention Sharing(https://arxiv.org/abs/2404.12900)
• Keywords: diffusion
• Abstract: Painterly Image Harmonization aims at seamlessly blending disparate visual elements within a single coherent image. However, previous approaches often encounter significant limitations due to training data constraints, the need for time-consuming fine-tuning, or reliance on additional prompts. To surmount these hurdles, we design a Training-and-prompt-Free General Painterly Harmonization method using image-wise attention sharing (TF-GPH), which integrates a novel "share-attention module". This module redefines the traditional self-attention mechanism by allowing for comprehensive image-wise attention, facilitating the use of a state-of-the-art pretrained latent diffusion model without the typical training data limitations. Additionally, we further introduce "similarity reweighting" mechanism enhances performance by effectively harnessing cross-image information, surpassing the capabilities of fine-tuning or prompt-based approaches. At last, we recognize the deficiencies in existing benchmarks and propose the "General Painterly Harmonization Benchmark", which employs range-based evaluation metrics to more accurately reflect real-world application. Extensive experiments demonstrate the superior efficacy of our method across various benchmarks. The code and web demo are available at https://github.com/BlueDyee/TF-GPH.

Title: Robust CLIP-Based Detector for Exposing Diffusion Model-Generated Images

• Authors: Santosh, Li Lin, Irene Amerini, Xin Wang, Shu Hu
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.12908
• Pdf URL: https://arxiv.org/pdf/2404.12908
• Copy Paste: [[2404.12908]] Robust CLIP-Based Detector for Exposing Diffusion Model-Generated Images(https://arxiv.org/abs/2404.12908)
• Keywords: robust, diffusion
• Abstract: Diffusion models (DMs) have revolutionized image generation, producing high-quality images with applications spanning various fields. However, their ability to create hyper-realistic images poses significant challenges in distinguishing between real and synthetic content, raising concerns about digital authenticity and potential misuse in creating deepfakes. This work introduces a robust detection framework that integrates image and text features extracted by CLIP model with a Multilayer Perceptron (MLP) classifier. We propose a novel loss that can improve the detector's robustness and handle imbalanced datasets. Additionally, we flatten the loss landscape during the model training to improve the detector's generalization capabilities. The effectiveness of our method, which outperforms traditional detection techniques, is demonstrated through extensive experiments, underscoring its potential to set a new state-of-the-art approach in DM-generated image detection. The code is available at https://github.com/Purdue-M2/Robust_DM_Generated_Image_Detection.

Title: Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Models

• Authors: Zhenyang Ni, Rui Ye, Yuxi Wei, Zhen Xiang, Yanfeng Wang, Siheng Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12916
• Pdf URL: https://arxiv.org/pdf/2404.12916
• Copy Paste: [[2404.12916]] Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Models(https://arxiv.org/abs/2404.12916)
• Keywords: security, protect, defense, attack, robust, steal
• Abstract: Vision-Large-Language-models(VLMs) have great application prospects in autonomous driving. Despite the ability of VLMs to comprehend and make decisions in complex scenarios, their integration into safety-critical autonomous driving systems poses serious security risks. In this paper, we propose BadVLMDriver, the first backdoor attack against VLMs for autonomous driving that can be launched in practice using physical objects. Unlike existing backdoor attacks against VLMs that rely on digital modifications, BadVLMDriver uses common physical items, such as a red balloon, to induce unsafe actions like sudden acceleration, highlighting a significant real-world threat to autonomous vehicle safety. To execute BadVLMDriver, we develop an automated pipeline utilizing natural language instructions to generate backdoor training samples with embedded malicious behaviors. This approach allows for flexible trigger and behavior selection, enhancing the stealth and practicality of the attack in diverse scenarios. We conduct extensive experiments to evaluate BadVLMDriver for two representative VLMs, five different trigger objects, and two types of malicious backdoor behaviors. BadVLMDriver achieves a 92% attack success rate in inducing a sudden acceleration when coming across a pedestrian holding a red balloon. Thus, BadVLMDriver not only demonstrates a critical security risk but also emphasizes the urgent need for developing robust defense mechanisms to protect against such vulnerabilities in autonomous driving technologies.

Title: Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models

• Authors: Konstantinos Vilouras, Pedro Sanchez, Alison Q. O'Neil, Sotirios A. Tsaftaris
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12920
• Pdf URL: https://arxiv.org/pdf/2404.12920
• Copy Paste: [[2404.12920]] Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models(https://arxiv.org/abs/2404.12920)
• Keywords: diffusion, generative
• Abstract: Localizing the exact pathological regions in a given medical scan is an important imaging problem that requires a large amount of bounding box ground truth annotations to be accurately solved. However, there exist alternative, potentially weaker, forms of supervision, such as accompanying free-text reports, which are readily available. The task of performing localization with textual guidance is commonly referred to as phrase grounding. In this work, we use a publicly available Foundation Model, namely the Latent Diffusion Model, to solve this challenging task. This choice is supported by the fact that the Latent Diffusion Model, despite being generative in nature, contains mechanisms (cross-attention) that implicitly align visual and textual features, thus leading to intermediate representations that are suitable for the task at hand. In addition, we aim to perform this task in a zero-shot manner, i.e., without any further training on target data, meaning that the model's weights remain frozen. To this end, we devise strategies to select features and also refine them via post-processing without extra learnable parameters. We compare our proposed method with state-of-the-art approaches which explicitly enforce image-text alignment in a joint embedding space via contrastive learning. Results on a popular chest X-ray benchmark indicate that our method is competitive wih SOTA on different types of pathology, and even outperforms them on average in terms of two metrics (mean IoU and AUC-ROC). Source code will be released upon acceptance.

Title: A Hybrid Generative and Discriminative PointNet on Unordered Point Sets

• Authors: Yang Ye, Shihao Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12925
• Pdf URL: https://arxiv.org/pdf/2404.12925
• Copy Paste: [[2404.12925]] A Hybrid Generative and Discriminative PointNet on Unordered Point Sets(https://arxiv.org/abs/2404.12925)
• Keywords: generative
• Abstract: As point cloud provides a natural and flexible representation usable in myriad applications (e.g., robotics and self-driving cars), the ability to synthesize point clouds for analysis becomes crucial. Recently, Xie et al. propose a generative model for unordered point sets in the form of an energy-based model (EBM). Despite the model achieving an impressive performance for point cloud generation, one separate model needs to be trained for each category to capture the complex point set distributions. Besides, their method is unable to classify point clouds directly and requires additional fine-tuning for classification. One interesting question is: Can we train a single network for a hybrid generative and discriminative model of point clouds? A similar question has recently been answered in the affirmative for images, introducing the framework of Joint Energy-based Model (JEM), which achieves high performance in image classification and generation simultaneously. This paper proposes GDPNet, the first hybrid Generative and Discriminative PointNet that extends JEM for point cloud classification and generation. Our GDPNet retains strong discriminative power of modern PointNet classifiers, while generating point cloud samples rivaling state-of-the-art generative approaches.

Title: Purposer: Putting Human Motion Generation in Context

• Authors: Nicolas Ugrinovic, Thomas Lucas, Fabien Baradel, Philippe Weinzaepfel, Gregory Rogez, Francesc Moreno-Noguer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12942
• Pdf URL: https://arxiv.org/pdf/2404.12942
• Copy Paste: [[2404.12942]] Purposer: Putting Human Motion Generation in Context(https://arxiv.org/abs/2404.12942)
• Keywords: generative
• Abstract: We present a novel method to generate human motion to populate 3D indoor scenes. It can be controlled with various combinations of conditioning signals such as a path in a scene, target poses, past motions, and scenes represented as 3D point clouds. State-of-the-art methods are either models specialized to one single setting, require vast amounts of high-quality and diverse training data, or are unconditional models that do not integrate scene or other contextual information. As a consequence, they have limited applicability and rely on costly training data. To address these limitations, we propose a new method ,dubbed Purposer, based on neural discrete representation learning. Our model is capable of exploiting, in a flexible manner, different types of information already present in open access large-scale datasets such as AMASS. First, we encode unconditional human motion into a discrete latent space. Second, an autoregressive generative model, conditioned with key contextual information, either with prompting or additive tokens, and trained for next-step prediction in this space, synthesizes sequences of latent indices. We further design a novel conditioning block to handle future conditioning information in such a causal model by using a network with two branches to compute separate stacks of features. In this manner, Purposer can generate realistic motion sequences in diverse test scenes. Through exhaustive evaluation, we demonstrate that our multi-contextual solution outperforms existing specialized approaches for specific contextual information, both in terms of quality and diversity. Our model is trained with short sequences, but a byproduct of being able to use various conditioning signals is that at test time different combinations can be used to chain short sequences together and generate long motions within a context scene.

Title: Next Generation Loss Function for Image Classification

• Authors: Shakhnaz Akhmedova, Nils Körber (Center for Artificial Intelligence in Public Health Research, Robert Koch Institute, Berlin, Germany)
• Subjects: cs.CV, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.12948
• Pdf URL: https://arxiv.org/pdf/2404.12948
• Copy Paste: [[2404.12948]] Next Generation Loss Function for Image Classification(https://arxiv.org/abs/2404.12948)
• Keywords: segmentation
• Abstract: Neural networks are trained by minimizing a loss function that defines the discrepancy between the predicted model output and the target value. The selection of the loss function is crucial to achieve task-specific behaviour and highly influences the capability of the model. A variety of loss functions have been proposed for a wide range of tasks affecting training and model performance. For classification tasks, the cross entropy is the de-facto standard and usually the first choice. Here, we try to experimentally challenge the well-known loss functions, including cross entropy (CE) loss, by utilizing the genetic programming (GP) approach, a population-based evolutionary algorithm. GP constructs loss functions from a set of operators and leaf nodes and these functions are repeatedly recombined and mutated to find an optimal structure. Experiments were carried out on different small-sized datasets CIFAR-10, CIFAR-100 and Fashion-MNIST using an Inception model. The 5 best functions found were evaluated for different model architectures on a set of standard datasets ranging from 2 to 102 classes and very different sizes. One function, denoted as Next Generation Loss (NGL), clearly stood out showing same or better performance for all tested datasets compared to CE. To evaluate the NGL function on a large-scale dataset, we tested its performance on the Imagenet-1k dataset where it showed improved top-1 accuracy compared to models trained with identical settings and other losses. Finally, the NGL was trained on a segmentation downstream task for Pascal VOC 2012 and COCO-Stuff164k datasets improving the underlying model performance.

Title: Towards Reliable Latent Knowledge Estimation in LLMs: In-Context Learning vs. Prompting Based Factual Knowledge Extraction

• Authors: Qinyuan Wu, Mohammad Aflah Khan, Soumi Das, Vedant Nanda, Bishwamittra Ghosh, Camila Kolling, Till Speicher, Laurent Bindschaedler, Krishna P. Gummadi, Evimaria Terzi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12957
• Pdf URL: https://arxiv.org/pdf/2404.12957
• Copy Paste: [[2404.12957]] Towards Reliable Latent Knowledge Estimation in LLMs: In-Context Learning vs. Prompting Based Factual Knowledge Extraction(https://arxiv.org/abs/2404.12957)
• Keywords: extraction, large language model
• Abstract: We propose an approach for estimating the latent knowledge embedded inside large language models (LLMs). We leverage the in-context learning (ICL) abilities of LLMs to estimate the extent to which an LLM knows the facts stored in a knowledge base. Our knowledge estimator avoids reliability concerns with previous prompting-based methods, is both conceptually simpler and easier to apply, and we demonstrate that it can surface more of the latent knowledge embedded in LLMs. We also investigate how different design choices affect the performance of ICL-based knowledge estimation. Using the proposed estimator, we perform a large-scale evaluation of the factual knowledge of a variety of open source LLMs, like OPT, Pythia, Llama(2), Mistral, Gemma, etc. over a large set of relations and facts from the Wikidata knowledge base. We observe differences in the factual knowledge between different model families and models of different sizes, that some relations are consistently better known than others but that models differ in the precise facts they know, and differences in the knowledge of base models and their finetuned counterparts.

Title: Eyes Can Deceive: Benchmarking Counterfactual Reasoning Abilities of Multi-modal Large Language Models

• Authors: Yian Li, Wentao Tian, Yang Jiao, Jingjing Chen, Yu-Gang Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12966
• Pdf URL: https://arxiv.org/pdf/2404.12966
• Copy Paste: [[2404.12966]] Eyes Can Deceive: Benchmarking Counterfactual Reasoning Abilities of Multi-modal Large Language Models(https://arxiv.org/abs/2404.12966)
• Keywords: large language model
• Abstract: Counterfactual reasoning, as a crucial manifestation of human intelligence, refers to making presuppositions based on established facts and extrapolating potential outcomes. Existing multimodal large language models (MLLMs) have exhibited impressive cognitive and reasoning capabilities, which have been examined across a wide range of Visual Question Answering (VQA) benchmarks. Nevertheless, how will existing MLLMs perform when faced with counterfactual questions? To answer this question, we first curate a novel \textbf{C}ounter\textbf{F}actual \textbf{M}ulti\textbf{M}odal reasoning benchmark, abbreviated as \textbf{CFMM}, to systematically assess the counterfactual reasoning capabilities of MLLMs. Our CFMM comprises six challenging tasks, each including hundreds of carefully human-labeled counterfactual questions, to evaluate MLLM's counterfactual reasoning capabilities across diverse aspects. Through experiments, interestingly, we find that existing MLLMs prefer to believe what they see, but ignore the counterfactual presuppositions presented in the question, thereby leading to inaccurate responses. Furthermore, we evaluate a wide range of prevalent MLLMs on our proposed CFMM. The significant gap between their performance on our CFMM and that on several VQA benchmarks indicates that there is still considerable room for improvement in existing MLLMs toward approaching human-level intelligence. On the other hand, through boosting MLLMs performances on our CFMM in the future, potential avenues toward developing MLLMs with advanced intelligence can be explored.

Title: RedactBuster: Entity Type Recognition from Redacted Documents

• Authors: Mirco Beltrame, Mauro Conti, Pierpaolo Guglielmin, Francesco Marchiori, Gabriele Orazi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.12991
• Pdf URL: https://arxiv.org/pdf/2404.12991
• Copy Paste: [[2404.12991]] RedactBuster: Entity Type Recognition from Redacted Documents(https://arxiv.org/abs/2404.12991)
• Keywords: privacy, protect, attack, robust, transformer
• Abstract: The widespread exchange of digital documents in various domains has resulted in abundant private information being shared. This proliferation necessitates redaction techniques to protect sensitive content and user privacy. While numerous redaction methods exist, their effectiveness varies, with some proving more robust than others. As such, the literature proposes several deanonymization techniques, raising awareness of potential privacy threats. However, while none of these methods are successful against the most effective redaction techniques, these attacks only focus on the anonymized tokens and ignore the sentence context. In this paper, we propose RedactBuster, the first deanonymization model using sentence context to perform Named Entity Recognition on reacted text. Our methodology leverages fine-tuned state-of-the-art Transformers and Deep Learning models to determine the anonymized entity types in a document. We test RedactBuster against the most effective redaction technique and evaluate it using the publicly available Text Anonymization Benchmark (TAB). Our results show accuracy values up to 0.985 regardless of the document nature or entity type. In raising awareness of this privacy issue, we propose a countermeasure we call character evasion that helps strengthen the secrecy of sensitive information. Furthermore, we make our model and testbed open-source to aid researchers and practitioners in evaluating the resilience of novel redaction techniques and enhancing document privacy.

Title: Goal Exploration via Adaptive Skill Distribution for Goal-Conditioned Reinforcement Learning

• Authors: Lisheng Wu, Ke Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12999
• Pdf URL: https://arxiv.org/pdf/2404.12999
• Copy Paste: [[2404.12999]] Goal Exploration via Adaptive Skill Distribution for Goal-Conditioned Reinforcement Learning(https://arxiv.org/abs/2404.12999)
• Keywords: robust
• Abstract: Exploration efficiency poses a significant challenge in goal-conditioned reinforcement learning (GCRL) tasks, particularly those with long horizons and sparse rewards. A primary limitation to exploration efficiency is the agent's inability to leverage environmental structural patterns. In this study, we introduce a novel framework, GEASD, designed to capture these patterns through an adaptive skill distribution during the learning process. This distribution optimizes the local entropy of achieved goals within a contextual horizon, enhancing goal-spreading behaviors and facilitating deep exploration in states containing familiar structural patterns. Our experiments reveal marked improvements in exploration efficiency using the adaptive skill distribution compared to a uniform skill distribution. Additionally, the learned skill distribution demonstrates robust generalization capabilities, achieving substantial exploration progress in unseen tasks containing similar local structures.

Title: Towards Robust Ferrous Scrap Material Classification with Deep Learning and Conformal Prediction

• Authors: Paulo Henrique dos Santos, Valéria de Carvalho Santos, Eduardo José da Silva Luz
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13002
• Pdf URL: https://arxiv.org/pdf/2404.13002
• Copy Paste: [[2404.13002]] Towards Robust Ferrous Scrap Material Classification with Deep Learning and Conformal Prediction(https://arxiv.org/abs/2404.13002)
• Keywords: robust, explainability, transformer
• Abstract: In the steel production domain, recycling ferrous scrap is essential for environmental and economic sustainability, as it reduces both energy consumption and greenhouse gas emissions. However, the classification of scrap materials poses a significant challenge, requiring advancements in automation technology. Additionally, building trust among human operators is a major obstacle. Traditional approaches often fail to quantify uncertainty and lack clarity in model decision-making, which complicates acceptance. In this article, we describe how conformal prediction can be employed to quantify uncertainty and add robustness in scrap classification. We have adapted the Split Conformal Prediction technique to seamlessly integrate with state-of-the-art computer vision models, such as the Vision Transformer (ViT), Swin Transformer, and ResNet-50, while also incorporating Explainable Artificial Intelligence (XAI) methods. We evaluate the approach using a comprehensive dataset of 8147 images spanning nine ferrous scrap classes. The application of the Split Conformal Prediction method allowed for the quantification of each model's uncertainties, which enhanced the understanding of predictions and increased the reliability of the results. Specifically, the Swin Transformer model demonstrated more reliable outcomes than the others, as evidenced by its smaller average size of prediction sets and achieving an average classification accuracy exceeding 95%. Furthermore, the Score-CAM method proved highly effective in clarifying visual features, significantly enhancing the explainability of the classification decisions.

Title: Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models

• Authors: Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, Xiaojuan Qi
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13013
• Pdf URL: https://arxiv.org/pdf/2404.13013
• Copy Paste: [[2404.13013]] Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models(https://arxiv.org/abs/2404.13013)
• Keywords: large language model
• Abstract: We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded and fine-grained visual perception ability. Beyond holistic image understanding, Groma is adept at region-level tasks such as region captioning and visual grounding. Such capabilities are built upon a localized visual tokenization mechanism, where an image input is decomposed into regions of interest and subsequently encoded into region tokens. By integrating region tokens into user instructions and model responses, we seamlessly enable Groma to understand user-specified region inputs and ground its textual output to images. Besides, to enhance the grounded chat ability of Groma, we curate a visually grounded instruction dataset by leveraging the powerful GPT-4V and visual prompting techniques. Compared with MLLMs that rely on the language model or external module for localization, Groma consistently demonstrates superior performances in standard referring and grounding benchmarks, highlighting the advantages of embedding localization into image tokenization. Project page: https://groma-mllm.github.io/.

Title: Sample Design Engineering: An Empirical Study of What Makes Good Downstream Fine-Tuning Samples for LLMs

• Authors: Biyang Guo, He Wang, Wenyilin Xiao, Hong Chen, Zhuxin Lee, Songqiao Han, Hailiang Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13033
• Pdf URL: https://arxiv.org/pdf/2404.13033
• Copy Paste: [[2404.13033]] Sample Design Engineering: An Empirical Study of What Makes Good Downstream Fine-Tuning Samples for LLMs(https://arxiv.org/abs/2404.13033)
• Keywords: extraction, large language model
• Abstract: In the burgeoning field of Large Language Models (LLMs) like ChatGPT and LLaMA, Prompt Engineering (PE) is renowned for boosting zero-shot or in-context learning (ICL) through prompt modifications. Yet, the realm of the sample design for downstream fine-tuning, crucial for task-specific LLM adaptation, is largely unexplored. This paper introduces Sample Design Engineering (SDE), a methodical approach to enhancing LLMs' post-tuning performance by refining input, output, and reasoning designs. We conduct a series of in-domain (ID) and out-of-domain (OOD) experiments to assess the impact of various design options on LLMs' downstream performance, revealing several intriguing patterns that hold consistently across different LLMs. Based on these insights, we propose an integrated SDE strategy, combining the most effective options, and validate its consistent superiority over heuristic sample designs in complex downstream tasks like multi-aspect sentiment analysis, event extraction, and nested entity recognition. Additionally, analyses of LLMs' inherent prompt/output perplexity, zero-shot, and ICL abilities illustrate that good PE strategies may not always translate to good SDE strategies. Code available at https://github.com/beyondguo/LLM-Tuning.

Title: Analysis of Classifier-Free Guidance Weight Schedulers

• Authors: Xi Wang, Nicolas Dufour, Nefeli Andreou, Marie-Paule Cani, Victoria Fernandez Abrevaya, David Picard, Vicky Kalogeiton
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13040
• Pdf URL: https://arxiv.org/pdf/2404.13040
• Copy Paste: [[2404.13040]] Analysis of Classifier-Free Guidance Weight Schedulers(https://arxiv.org/abs/2404.13040)
• Keywords: diffusion
• Abstract: Classifier-Free Guidance (CFG) enhances the quality and condition adherence of text-to-image diffusion models. It operates by combining the conditional and unconditional predictions using a fixed weight. However, recent works vary the weights throughout the diffusion process, reporting superior results but without providing any rationale or analysis. By conducting comprehensive experiments, this paper provides insights into CFG weight schedulers. Our findings suggest that simple, monotonically increasing weight schedulers consistently lead to improved performances, requiring merely a single line of code. In addition, more complex parametrized schedulers can be optimized for further improvement, but do not generalize across different models and tasks.

Title: Data Alignment for Zero-Shot Concept Generation in Dermatology AI

• Authors: Soham Gadgil, Mahtab Bigverdi
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13043
• Pdf URL: https://arxiv.org/pdf/2404.13043
• Copy Paste: [[2404.13043]] Data Alignment for Zero-Shot Concept Generation in Dermatology AI(https://arxiv.org/abs/2404.13043)
• Keywords: large language model
• Abstract: AI in dermatology is evolving at a rapid pace but the major limitation to training trustworthy classifiers is the scarcity of data with ground-truth concept level labels, which are meta-labels semantically meaningful to humans. Foundation models like CLIP providing zero-shot capabilities can help alleviate this challenge by leveraging vast amounts of image-caption pairs available on the internet. CLIP can be fine-tuned using domain specific image-caption pairs to improve classification performance. However, CLIP's pre-training data is not well-aligned with the medical jargon that clinicians use to perform diagnoses. The development of large language models (LLMs) in recent years has led to the possibility of leveraging the expressive nature of these models to generate rich text. Our goal is to use these models to generate caption text that aligns well with both the clinical lexicon and with the natural human language used in CLIP's pre-training data. Starting with captions used for images in PubMed articles, we extend them by passing the raw captions through an LLM fine-tuned on the field's several textbooks. We find that using captions generated by an expressive fine-tuned LLM like GPT-3.5 improves downstream zero-shot concept classification performance.

Title: Unified Scene Representation and Reconstruction for 3D Large Language Models

• Authors: Tao Chu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Qiong Liu, Jiaqi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13044
• Pdf URL: https://arxiv.org/pdf/2404.13044
• Copy Paste: [[2404.13044]] Unified Scene Representation and Reconstruction for 3D Large Language Models(https://arxiv.org/abs/2404.13044)
• Keywords: large language model
• Abstract: Enabling Large Language Models (LLMs) to interact with 3D environments is challenging. Existing approaches extract point clouds either from ground truth (GT) geometry or 3D scenes reconstructed by auxiliary models. Text-image aligned 2D features from CLIP are then lifted to point clouds, which serve as inputs for LLMs. However, this solution lacks the establishment of 3D point-to-point connections, leading to a deficiency of spatial structure information. Concurrently, the absence of integration and unification between the geometric and semantic representations of the scene culminates in a diminished level of 3D scene understanding. In this paper, we demonstrate the importance of having a unified scene representation and reconstruction framework, which is essential for LLMs in 3D scenes. Specifically, we introduce Uni3DR^2 extracts 3D geometric and semantic aware representation features via the frozen pre-trained 2D foundation models (e.g., CLIP and SAM) and a multi-scale aggregate 3D decoder. Our learned 3D representations not only contribute to the reconstruction process but also provide valuable knowledge for LLMs. Experimental results validate that our Uni3DR^2 yields convincing gains over the baseline on the 3D reconstruction dataset ScanNet (increasing F-Score by +1.8\%). When applied to LLMs, our Uni3DR^2-LLM exhibits superior performance over the baseline on the 3D vision-language understanding dataset ScanQA (increasing BLEU-1 by +4.0\% and +4.2\% on the val set and test set, respectively). Furthermore, it outperforms the state-of-the-art method that uses additional GT point clouds on both ScanQA and 3DMV-VQA.

Title: MoVA: Adapting Mixture of Vision Experts to Multimodal Context

• Authors: Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, Yu Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13046
• Pdf URL: https://arxiv.org/pdf/2404.13046
• Copy Paste: [[2404.13046]] MoVA: Adapting Mixture of Vision Experts to Multimodal Context(https://arxiv.org/abs/2404.13046)
• Keywords: large language model
• Abstract: As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e.g., the CLIP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts. This benefits from the powerful model function understanding ability of the large language model (LLM) equipped with expert-routing low-rank adaptation (LoRA). In the fine-grained stage, we elaborately conduct the mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge from various experts. This coarse-to-fine paradigm effectively leverages representations from experts based on multimodal context and model expertise, further enhancing the generalization ability. We conduct extensive experiments to evaluate the effectiveness of the proposed approach. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of challenging multimodal benchmarks. Codes and models will be available at https://github.com/TempleX98/MoVA.