2024-05-21

Title: Depth-aware Panoptic Segmentation

• Authors: Tuan Nguyen, Max Mehltretter, Franz Rottensteiner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Depth-aware Panoptic Segmentation(https://arxiv.org/abs/)
• Keywords: segmentation
• Abstract: Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by 2.2% in terms of panoptic quality.

Title: Global License Plate Dataset

• Authors: Siddharth Agrawal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Global License Plate Dataset(https://arxiv.org/abs/)
• Keywords: segmentation
• Abstract: In the pursuit of advancing the state-of-the-art (SOTA) in road safety, traffic monitoring, surveillance, and logistics automation, we introduce the Global License Plate Dataset (GLPD). The dataset consists of over 5 million images, including diverse samples captured from 74 countries with meticulous annotations, including license plate characters, license plate segmentation masks, license plate corner vertices, as well as vehicle make, colour, and model. We also include annotated data on more classes, such as pedestrians, vehicles, roads, etc. We include a statistical analysis of the dataset, and provide baseline efficient and accurate models. The GLPD aims to be the primary benchmark dataset for model development and finetuning for license plate recognition.

Title: Block Selective Reprogramming for On-device Training of Vision Transformers

• Authors: Sreetama Sarkar, Souvik Kundu, Kai Zheng, Peter A. Beerel
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Block Selective Reprogramming for On-device Training of Vision Transformers(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: The ubiquity of vision transformers (ViTs) for various edge applications, including personalized learning, has created the demand for on-device fine-tuning. However, training with the limited memory and computation power of edge devices remains a significant challenge. In particular, the memory required for training is much higher than that needed for inference, primarily due to the need to store activations across all layers in order to compute the gradients needed for weight updates. Previous works have explored reducing this memory requirement via frozen-weight training as well storing the activations in a compressed format. However, these methods are deemed inefficient due to their inability to provide training or inference speedup. In this paper, we first investigate the limitations of existing on-device training methods aimed at reducing memory and compute requirements. We then present block selective reprogramming (BSR) in which we fine-tune only a fraction of total blocks of a pre-trained model and selectively drop tokens based on self-attention scores of the frozen layers. To show the efficacy of BSR, we present extensive evaluations on ViT-B and DeiT-S with five different datasets. Compared to the existing alternatives, our approach simultaneously reduces training memory by up to 1.4x and compute cost by up to 2x while maintaining similar accuracy. We also showcase results for Mixture-of-Expert (MoE) models, demonstrating the effectiveness of our approach in multitask learning scenarios.

Title: Multimodal CLIP Inference for Meta-Few-Shot Image Classification

• Authors: Constance Ferragu, Philomene Chagniot, Vincent Coyette
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Multimodal CLIP Inference for Meta-Few-Shot Image Classification(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: In recent literature, few-shot classification has predominantly been defined by the N-way k-shot meta-learning problem. Models designed for this purpose are usually trained to excel on standard benchmarks following a restricted setup, excluding the use of external data. Given the recent advancements in large language and vision models, a question naturally arises: can these models directly perform well on meta-few-shot learning benchmarks? Multimodal foundation models like CLIP, which learn a joint (image, text) embedding, are of particular interest. Indeed, multimodal training has proven to enhance model robustness, especially regarding ambiguities, a limitation frequently observed in the few-shot setup. This study demonstrates that combining modalities from CLIP's text and image encoders outperforms state-of-the-art meta-few-shot learners on widely adopted benchmarks, all without additional training. Our results confirm the potential and robustness of multimodal foundation models like CLIP and serve as a baseline for existing and future approaches leveraging such models.

Title: Untargeted Adversarial Attack on Knowledge Graph Embeddings

• Authors: Tianzhe Zhao, Jiaoyan Chen, Yanchi Ru, Qika Lin, Yuxia Geng, Jun Liu
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Untargeted Adversarial Attack on Knowledge Graph Embeddings(https://arxiv.org/abs/)
• Keywords: attack, robust
• Abstract: Knowledge graph embedding (KGE) methods have achieved great success in handling various knowledge graph (KG) downstream tasks. However, KGE methods may learn biased representations on low-quality KGs that are prevalent in the real world. Some recent studies propose adversarial attacks to investigate the vulnerabilities of KGE methods, but their attackers are target-oriented with the KGE method and the target triples to predict are given in advance, which lacks practicability. In this work, we explore untargeted attacks with the aim of reducing the global performances of KGE methods over a set of unknown test triples and conducting systematic analyses on KGE robustness. Considering logic rules can effectively summarize the global structure of a KG, we develop rule-based attack strategies to enhance the attack efficiency. In particular,we consider adversarial deletion which learns rules, applying the rules to score triple importance and delete important triples, and adversarial addition which corrupts the learned rules and applies them for negative triples as perturbations. Extensive experiments on two datasets over three representative classes of KGE methods demonstrate the effectiveness of our proposed untargeted attacks in diminishing the link prediction results. And we also find that different KGE methods exhibit different robustness to untargeted attacks. For example, the robustness of methods engaged with graph neural networks and logic rules depends on the density of the graph. But rule-based methods like NCRL are easily affected by adversarial addition attacks to capture negative rules

Title: Private Data Leakage in Federated Human Activity Recognition for Wearable Healthcare Devices

• Authors: Kongyang Chen, Dongping Zhang, Bing Mi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Private Data Leakage in Federated Human Activity Recognition for Wearable Healthcare Devices(https://arxiv.org/abs/)
• Keywords: security, privacy, attack, robust, membership infer, federate
• Abstract: Wearable wristband or watch can be utilized for health monitoring, such as determining the user's activity status based on behavior and providing reasonable exercise recommendations. Obviously, the individual data perception and local computing capabilities of a single wearable device are limited, making it difficult to train a robust user behavior recognition model. Typically, joint modeling requires the collaboration of multiple wearable devices. An appropriate research approach is Federated Human Activity Recognition (HAR), which can train a global model without uploading users' local exercise data. Nevertheless, recent studies indicate that federated learning still faces serious data security and privacy issues. To the best of our knowledge, there is no existing research on membership information leakage in Federated HAR. Therefore, our study aims to investigate the joint modeling process of multiple wearable devices for user behavior recognition, with a focus on analyzing the privacy leakage issues of wearable data. In our system, we consider a federated learning architecture consisting of $N$ wearable device users and a parameter server. The parameter server distributes the initial model to each user, who independently perceives their motion sensor data, conducts local model training, and uploads it to the server. The server aggregates these local models until convergence. In the federated learning architecture, the server may be curious and seek to obtain privacy information about relevant users from the model parameters. Hence, we consider membership inference attacks based on malicious servers, which exploit differences in model generalization across different client data. Through experimentation deployed on five publicly available HAR datasets, we demonstrate that the accuracy of malicious server membership inference reaches 92\%.

Title: Benchmark Early and Red Team Often: A Framework for Assessing and Managing Dual-Use Hazards of AI Foundation Models

• Authors: Anthony M. Barrett, Krystal Jackson, Evan R. Murphy, Nada Madkour, Jessica Newman
• Subjects: cs.CR, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Benchmark Early and Red Team Often: A Framework for Assessing and Managing Dual-Use Hazards of AI Foundation Models(https://arxiv.org/abs/)
• Keywords: security, attack
• Abstract: A concern about cutting-edge or "frontier" AI foundation models is that an adversary may use the models for preparing chemical, biological, radiological, nuclear, (CBRN), cyber, or other attacks. At least two methods can identify foundation models with potential dual-use capability; each has advantages and disadvantages: A. Open benchmarks (based on openly available questions and answers), which are low-cost but accuracy-limited by the need to omit security-sensitive details; and B. Closed red team evaluations (based on private evaluation by CBRN and cyber experts), which are higher-cost but can achieve higher accuracy by incorporating sensitive details. We propose a research and risk-management approach using a combination of methods including both open benchmarks and closed red team evaluations, in a way that leverages advantages of both methods. We recommend that one or more groups of researchers with sufficient resources and access to a range of near-frontier and frontier foundation models run a set of foundation models through dual-use capability evaluation benchmarks and red team evaluations, then analyze the resulting sets of models' scores on benchmark and red team evaluations to see how correlated those are. If, as we expect, there is substantial correlation between the dual-use potential benchmark scores and the red team evaluation scores, then implications include the following: The open benchmarks should be used frequently during foundation model development as a quick, low-cost measure of a model's dual-use potential; and if a particular model gets a high score on the dual-use potential benchmark, then more in-depth red team assessments of that model's dual-use capability should be performed. We also discuss limitations and mitigations for our approach, e.g., if model developers try to game benchmarks by including a version of benchmark test data in a model's training data.

Title: Flow Score Distillation for Diverse Text-to-3D Generation

• Authors: Runjie Yan, Kailu Wu, Kaisheng Ma
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Flow Score Distillation for Diverse Text-to-3D Generation(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: Recent advancements in Text-to-3D generation have yielded remarkable progress, particularly through methods that rely on Score Distillation Sampling (SDS). While SDS exhibits the capability to create impressive 3D assets, it is hindered by its inherent maximum-likelihood-seeking essence, resulting in limited diversity in generation outcomes. In this paper, we discover that the Denoise Diffusion Implicit Models (DDIM) generation process (\ie PF-ODE) can be succinctly expressed using an analogue of SDS loss. One step further, one can see SDS as a generalized DDIM generation process. Following this insight, we show that the noise sampling strategy in the noise addition stage significantly restricts the diversity of generation results. To address this limitation, we present an innovative noise sampling approach and introduce a novel text-to-3D method called Flow Score Distillation (FSD). Our validation experiments across various text-to-image Diffusion Models demonstrate that FSD substantially enhances generation diversity without compromising quality.

Title: Learnable Privacy Neurons Localization in Language Models

• Authors: Ruizhe Chen, Tianxiang Hu, Yang Feng, Zuozhu Liu
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Learnable Privacy Neurons Localization in Language Models(https://arxiv.org/abs/)
• Keywords: privacy, large language model
• Abstract: Concerns regarding Large Language Models (LLMs) to memorize and disclose private information, particularly Personally Identifiable Information (PII), become prominent within the community. Many efforts have been made to mitigate the privacy risks. However, the mechanism through which LLMs memorize PII remains poorly understood. To bridge this gap, we introduce a pioneering method for pinpointing PII-sensitive neurons (privacy neurons) within LLMs. Our method employs learnable binary weight masks to localize specific neurons that account for the memorization of PII in LLMs through adversarial training. Our investigations discover that PII is memorized by a small subset of neurons across all layers, which shows the property of PII specificity. Furthermore, we propose to validate the potential in PII risk mitigation by deactivating the localized privacy neurons. Both quantitative and qualitative experiments demonstrate the effectiveness of our neuron localization algorithm.

Title: "What do you want from theory alone?" Experimenting with Tight Auditing of Differentially Private Synthetic Data Generation

• Authors: Meenatchi Sundaram Muthu Selva Annamalai, Georgi Ganev, Emiliano De Cristofaro
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] "What do you want from theory alone?" Experimenting with Tight Auditing of Differentially Private Synthetic Data Generation(https://arxiv.org/abs/)
• Keywords: privacy, attack, membership infer
• Abstract: Differentially private synthetic data generation (DP-SDG) algorithms are used to release datasets that are structurally and statistically similar to sensitive data while providing formal bounds on the information they leak. However, bugs in algorithms and implementations may cause the actual information leakage to be higher. This prompts the need to verify whether the theoretical guarantees of state-of-the-art DP-SDG implementations also hold in practice. We do so via a rigorous auditing process: we compute the information leakage via an adversary playing a distinguishing game and running membership inference attacks (MIAs). If the leakage observed empirically is higher than the theoretical bounds, we identify a DP violation; if it is non-negligibly lower, the audit is loose. We audit six DP-SDG implementations using different datasets and threat models and find that black-box MIAs commonly used against DP-SDGs are severely limited in power, yielding remarkably loose empirical privacy estimates. We then consider MIAs in stronger threat models, i.e., passive and active white-box, using both existing and newly proposed attacks. Overall, we find that, currently, we do not only need white-box MIAs but also worst-case datasets to tightly estimate the privacy leakage from DP-SDGs. Finally, we show that our automated auditing procedure finds both known DP violations (in 4 out of the 6 implementations) as well as a new one in the DPWGAN implementation that was successfully submitted to the NIST DP Synthetic Data Challenge. The source code needed to reproduce our experiments is available from this https URL.

Title: Physics-incorporated Graph Neural Network for Multivariate Time Series Imputation

• Authors: Guojun Liang, Prayag Tiwari, Slawomir Nowaczyk, Stefan Byttner
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Physics-incorporated Graph Neural Network for Multivariate Time Series Imputation(https://arxiv.org/abs/)
• Keywords: explainability
• Abstract: Exploring the missing values is an essential but challenging issue due to the complex latent spatio-temporal correlation and dynamic nature of time series. Owing to the outstanding performance in dealing with structure learning potentials, Graph Neural Networks (GNNs) and Recurrent Neural Networks (RNNs) are often used to capture such complex spatio-temporal features in multivariate time series. However, these data-driven models often fail to capture the essential spatio-temporal relationships when significant signal corruption occurs. Additionally, calculating the high-order neighbor nodes in these models is of high computational complexity. To address these problems, we propose a novel higher-order spatio-temporal physics-incorporated GNN (HSPGNN). Firstly, the dynamic Laplacian matrix can be obtained by the spatial attention mechanism. Then, the generic inhomogeneous partial differential equation (PDE) of physical dynamic systems is used to construct the dynamic higher-order spatio-temporal GNN to obtain the missing time series values. Moreover, we estimate the missing impact by Normalizing Flows (NF) to evaluate the importance of each node in the graph for better explainability. Experimental results on four benchmark datasets demonstrate the effectiveness of HSPGNN and the superior performance when combining various order neighbor nodes. Also, graph-like optical flow, dynamic graphs, and missing impact can be obtained naturally by HSPGNN, which provides better dynamic analysis and explanation than traditional data-driven models. Our code is available at this https URL.

Title: Large Language Models for Tuning Evolution Strategies

• Authors: Oliver Kramer
• Subjects: cs.LG, cs.CL, cs.NE
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Large Language Models for Tuning Evolution Strategies(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit world knowledge and inference capabilities, making them powerful tools for various applications. This paper proposes a feedback loop mechanism that leverages these capabilities to tune Evolution Strategies (ES) parameters effectively. The mechanism involves a structured process of providing programming instructions, executing the corresponding code, and conducting thorough analysis. This process is specifically designed for the optimization of ES parameters. The method operates through an iterative cycle, ensuring continuous refinement of the ES parameters. First, LLMs process the instructions to generate or modify the code. The code is then executed, and the results are meticulously logged. Subsequent analysis of these results provides insights that drive further improvements. An experiment on tuning the learning rates of ES using the LLaMA3 model demonstrate the feasibility of this approach. This research illustrates how LLMs can be harnessed to improve ES algorithms' performance and suggests broader applications for similar feedback loop mechanisms in various domains.

Title: Large Language Models in Wireless Application Design: In-Context Learning-enhanced Automatic Network Intrusion Detection

• Authors: Han Zhang, Akram Bin Sediq, Ali Afana, Melike Erol-Kantarci
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Large Language Models in Wireless Application Design: In-Context Learning-enhanced Automatic Network Intrusion Detection(https://arxiv.org/abs/)
• Keywords: attack, transformer, generative, large language model
• Abstract: Large language models (LLMs), especially generative pre-trained transformers (GPTs), have recently demonstrated outstanding ability in information comprehension and problem-solving. This has motivated many studies in applying LLMs to wireless communication networks. In this paper, we propose a pre-trained LLM-empowered framework to perform fully automatic network intrusion detection. Three in-context learning methods are designed and compared to enhance the performance of LLMs. With experiments on a real network intrusion detection dataset, in-context learning proves to be highly beneficial in improving the task processing performance in a way that no further training or fine-tuning of LLMs is required. We show that for GPT-4, testing accuracy and F1-Score can be improved by 90%. Moreover, pre-trained LLMs demonstrate big potential in performing wireless communication-related tasks. Specifically, the proposed framework can reach an accuracy and F1-Score of over 95% on different types of attacks with GPT-4 using only 10 in-context learning examples.

Title: Generative modeling of Sparse Approximate Inverse Preconditioners

• Authors: Mou Li, He Wang, Peter K. Jimack
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Generative modeling of Sparse Approximate Inverse Preconditioners(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: We present a new deep learning paradigm for the generation of sparse approximate inverse (SPAI) preconditioners for matrix systems arising from the mesh-based discretization of elliptic differential operators. Our approach is based upon the observation that matrices generated in this manner are not arbitrary, but inherit properties from differential operators that they discretize. Consequently, we seek to represent a learnable distribution of high-performance preconditioners from a low-dimensional subspace through a carefully-designed autoencoder, which is able to generate SPAI preconditioners for these systems. The concept has been implemented on a variety of finite element discretizations of second- and fourth-order elliptic partial differential equations with highly promising results.

Title: Photorealistic 3D Urban Scene Reconstruction and Point Cloud Extraction using Google Earth Imagery and Gaussian Splatting

• Authors: Kyle Gao, Dening Lu, Hongjie He, Linlin Xu, Jonathan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Photorealistic 3D Urban Scene Reconstruction and Point Cloud Extraction using Google Earth Imagery and Gaussian Splatting(https://arxiv.org/abs/)
• Keywords: extraction
• Abstract: 3D urban scene reconstruction and modelling is a crucial research area in remote sensing with numerous applications in academia, commerce, industry, and administration. Recent advancements in view synthesis models have facilitated photorealistic 3D reconstruction solely from 2D images. Leveraging Google Earth imagery, we construct a 3D Gaussian Splatting model of the Waterloo region centered on the University of Waterloo and are able to achieve view-synthesis results far exceeding previous 3D view-synthesis results based on neural radiance fields which we demonstrate in our benchmark. Additionally, we retrieved the 3D geometry of the scene using the 3D point cloud extracted from the 3D Gaussian Splatting model which we benchmarked against our Multi- View-Stereo dense reconstruction of the scene, thereby reconstructing both the 3D geometry and photorealistic lighting of the large-scale urban scene through 3D Gaussian Splatting

Title: Generative Artificial Intelligence: A Systematic Review and Applications

• Authors: Sandeep Singh Sengar, Affan Bin Hasan, Sanjay Kumar, Fiona Carroll
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Generative Artificial Intelligence: A Systematic Review and Applications(https://arxiv.org/abs/)
• Keywords: generative, large language model
• Abstract: In recent years, the study of artificial intelligence (AI) has undergone a paradigm shift. This has been propelled by the groundbreaking capabilities of generative models both in supervised and unsupervised learning scenarios. Generative AI has shown state-of-the-art performance in solving perplexing real-world conundrums in fields such as image translation, medical diagnostics, textual imagery fusion, natural language processing, and beyond. This paper documents the systematic review and analysis of recent advancements and techniques in Generative AI with a detailed discussion of their applications including application-specific models. Indeed, the major impact that generative AI has made to date, has been in language generation with the development of large language models, in the field of image translation and several other interdisciplinary applications of generative AI. Moreover, the primary contribution of this paper lies in its coherent synthesis of the latest advancements in these areas, seamlessly weaving together contemporary breakthroughs in the field. Particularly, how it shares an exploration of the future trajectory for generative AI. In conclusion, the paper ends with a discussion of Responsible AI principles, and the necessary ethical considerations for the sustainability and growth of these generative models.

Title: Safety in Graph Machine Learning: Threats and Safeguards

• Authors: Song Wang, Yushun Dong, Binchi Zhang, Zihan Chen, Xingbo Fu, Yinhan He, Cong Shen, Chuxu Zhang, Nitesh V. Chawla, Jundong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Safety in Graph Machine Learning: Threats and Safeguards(https://arxiv.org/abs/)
• Keywords: protect, attack
• Abstract: Graph Machine Learning (Graph ML) has witnessed substantial advancements in recent years. With their remarkable ability to process graph-structured data, Graph ML techniques have been extensively utilized across diverse applications, including critical domains like finance, healthcare, and transportation. Despite their societal benefits, recent research highlights significant safety concerns associated with the widespread use of Graph ML models. Lacking safety-focused designs, these models can produce unreliable predictions, demonstrate poor generalizability, and compromise data confidentiality. In high-stakes scenarios such as financial fraud detection, these vulnerabilities could jeopardize both individuals and society at large. Therefore, it is imperative to prioritize the development of safety-oriented Graph ML models to mitigate these risks and enhance public confidence in their applications. In this survey paper, we explore three critical aspects vital for enhancing safety in Graph ML: reliability, generalizability, and confidentiality. We categorize and analyze threats to each aspect under three headings: model threats, data threats, and attack threats. This novel taxonomy guides our review of effective strategies to protect against these threats. Our systematic review lays a groundwork for future research aimed at developing practical, safety-centered Graph ML models. Furthermore, we highlight the significance of safe Graph ML practices and suggest promising avenues for further investigation in this crucial area.

Title: DeFiTail: DeFi Protocol Inspection through Cross-Contract Execution Analysis

• Authors: Wenkai Li, Xiaoqi Li, Yuqing Zhang, Zongwei Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] DeFiTail: DeFi Protocol Inspection through Cross-Contract Execution Analysis(https://arxiv.org/abs/)
• Keywords: attack
• Abstract: Decentralized finance (DeFi) protocols are crypto projects developed on the blockchain to manage digital assets. Attacks on DeFi have been frequent and have resulted in losses exceeding \$77 billion. However, detection methods for malicious DeFi events are still lacking. In this paper, we propose DeFiTail, the first framework that utilizes deep learning to detect access control and flash loan exploits that may occur on DeFi. Since the DeFi protocol events involve invocations with multi-account transactions, which requires execution path unification with different contracts. Moreover, to mitigate the impact of mistakes in Control Flow Graph (CFG) connections, we validate the data path by employing the symbolic execution stack. Furthermore, we feed the data paths through our model to achieve the inspection of DeFi protocols. Experimental results indicate that DeFiTail achieves the highest accuracy, with 98.39% in access control and 97.43% in flash loan exploits. DeFiTail also demonstrates an enhanced capability to detect malicious contracts, identifying 86.67% accuracy from the CVE dataset.

Title: CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl

• Authors: Ilya Ilyankou, James Haworth, Stefano Cavazzi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: The Common Crawl (CC) corpus is the largest open web crawl dataset containing 9.5+ petabytes of data captured since 2008. The dataset is instrumental in training large language models, and as such it has been studied for (un)desirable content, and distilled for smaller, domain-specific datasets. However, to our knowledge, no research has been dedicated to using CC as a source of annotated geospatial data. In this paper, we introduce an efficient pipeline to extract annotated user-generated tracks from GPX files found in CC, and the resulting multimodal dataset with 1,416 pairings of human-written descriptions and MultiLineString vector data. The dataset can be used to study people's outdoor activity patterns, the way people talk about their outdoor experiences, and for developing trajectory generation or track annotation models.

Title: From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoT

• Authors: Jace Grandinetti, Rafe McBeth
• Subjects: cs.CL, physics.med-ph
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoT(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable progress, yet their application in specialized fields, such as medical physics, remains challenging due to the need for domain-specific knowledge. This study introduces ARCoT (Adaptable Retrieval-based Chain of Thought), a framework designed to enhance the domain-specific accuracy of LLMs without requiring fine-tuning or extensive retraining. ARCoT integrates a retrieval mechanism to access relevant domain-specific information and employs step-back and chain-of-thought prompting techniques to guide the LLM's reasoning process, ensuring more accurate and context-aware responses. Benchmarking on a medical physics multiple-choice exam, our model outperformed standard LLMs and reported average human performance, demonstrating improvements of up to 68% and achieving a high score of 90%. This method reduces hallucinations and increases domain-specific performance. The versatility and model-agnostic nature of ARCoT make it easily adaptable to various domains, showcasing its significant potential for enhancing the accuracy and reliability of LLMs in specialized fields.

Title: Prompt Exploration with Prompt Regression

• Authors: Michael Feffer, Ronald Xu, Yuekai Sun, Mikhail Yurochkin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Prompt Exploration with Prompt Regression(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: In the advent of democratized usage of large language models (LLMs), there is a growing desire to systematize LLM prompt creation and selection processes beyond iterative trial-and-error. Prior works majorly focus on searching the space of prompts without accounting for relations between prompt variations. Here we propose a framework, Prompt Exploration with Prompt Regression (PEPR), to predict the effect of prompt combinations given results for individual prompt elements as well as a simple method to select an effective prompt for a given use-case. We evaluate our approach with open-source LLMs of different sizes on several different tasks.

Title: Enhancing Watermarked Language Models to Identify Users

• Authors: Aloni Cohen, Alexander Hoover, Gabe Schoenbach
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Enhancing Watermarked Language Models to Identify Users(https://arxiv.org/abs/)
• Keywords: robust, watermark
• Abstract: A zero-bit watermarked language model produces text that is indistinguishable from that of the underlying model, but which can be detected as machine-generated using a secret key. But merely detecting AI-generated spam, say, as watermarked may not prevent future abuses. If we could additionally trace the text to a spammer's API token, we could then cut off their access to the model. We introduce multi-user watermarks, which allow tracing model-generated text to individuals or to groups of colluding users. We construct multi-user watermarking schemes from undetectable zero-bit watermarking schemes. Importantly, our schemes provide both zero-bit and multi-user assurances at the same time: detecting shorter snippets as well as the original scheme and tracing longer excerpts to individuals. Along the way, we give a generic construction of a watermarking scheme that embeds long messages into generated text. Ours are the first black-box reductions between watermarking schemes for language models. A major challenge for black-box reductions is the lack of a unified abstraction for robustness -- that marked text is detectable after edits. Existing works give incomparable robustness guarantees, based on bespoke requirements on the language model's outputs and the users' edits. We introduce a new abstraction -- AEB-robustness -- to overcome this challenge. AEB-robustness provides that the watermark is detectable whenever the edited text "approximates enough blocks" of model-generated output. Specifying the robustness condition amounts to defining approximates, enough, and blocks. Using our new abstraction, we relate the robustness properties of our constructions to that of the underlying zero-bit scheme. Whereas prior works only guarantee robustness for a single text generated in response to a single prompt, our schemes are robust against adaptive prompting, a stronger adversarial model.

Title: AdaWaveNet: Adaptive Wavelet Network for Time Series Analysis

• Authors: Han Yu, Peikun Guo, Akane Sano
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] AdaWaveNet: Adaptive Wavelet Network for Time Series Analysis(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Time series data analysis is a critical component in various domains such as finance, healthcare, and meteorology. Despite the progress in deep learning for time series analysis, there remains a challenge in addressing the non-stationary nature of time series data. Traditional models, which are built on the assumption of constant statistical properties over time, often struggle to capture the temporal dynamics in realistic time series, resulting in bias and error in time series analysis. This paper introduces the Adaptive Wavelet Network (AdaWaveNet), a novel approach that employs Adaptive Wavelet Transformation for multi-scale analysis of non-stationary time series data. AdaWaveNet designed a lifting scheme-based wavelet decomposition and construction mechanism for adaptive and learnable wavelet transforms, which offers enhanced flexibility and robustness in analysis. We conduct extensive experiments on 10 datasets across 3 different tasks, including forecasting, imputation, and a newly established super-resolution task. The evaluations demonstrate the effectiveness of AdaWaveNet over existing methods in all three tasks, which illustrates its potential in various real-world applications.

Title: Flexible Motion In-betweening with Diffusion Models

• Authors: Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, Michiel van de Panne
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Flexible Motion In-betweening with Diffusion Models(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: Motion in-betweening, a fundamental task in character animation, consists of generating motion sequences that plausibly interpolate user-provided keyframe constraints. It has long been recognized as a labor-intensive and challenging process. We investigate the potential of diffusion models in generating diverse human motions guided by keyframes. Unlike previous inbetweening methods, we propose a simple unified model capable of generating precise and diverse motions that conform to a flexible range of user-specified spatial constraints, as well as text conditioning. To this end, we propose Conditional Motion Diffusion In-betweening (CondMDI) which allows for arbitrary dense-or-sparse keyframe placement and partial keyframe constraints while generating high-quality motions that are diverse and coherent with the given keyframes. We evaluate the performance of CondMDI on the text-conditioned HumanML3D dataset and demonstrate the versatility and efficacy of diffusion models for keyframe in-betweening. We further explore the use of guidance and imputation-based approaches for inference-time keyframing and compare CondMDI against these methods.

Title: MotionGS : Compact Gaussian Splatting SLAM by Motion Filter

• Authors: Xinli Guo, Peng Han, Weidong Zhang, Hongtian Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MotionGS : Compact Gaussian Splatting SLAM by Motion Filter(https://arxiv.org/abs/)
• Keywords: extraction
• Abstract: With their high-fidelity scene representation capability, the attention of SLAM field is deeply attracted by the Neural Radiation Field (NeRF) and 3D Gaussian Splatting (3DGS). Recently, there has been a Surge in NeRF-based SLAM, while 3DGS-based SLAM is sparse. A novel 3DGS-based SLAM approach with a fusion of deep visual feature, dual keyframe selection and 3DGS is presented in this paper. Compared with the existing methods, the proposed selectively tracking is achieved by feature extraction and motion filter on each frame. The joint optimization of pose and 3D Gaussian runs through the entire mapping process. Additionally, the coarse-to-fine pose estimation and compact Gaussian scene representation are implemented by dual keyfeature selection and novel loss functions. Experimental results demonstrate that the proposed algorithm not only outperforms the existing methods in tracking and mapping, but also has less memory usage.

Title: AquaLoRA: Toward White-box Protection for Customized Stable Diffusion Models via Watermark LoRA

• Authors: Weitao Feng, Wenbo Zhou, Jiyan He, Jie Zhang, Tianyi Wei, Guanlin Li, Tianwei Zhang, Weiming Zhang, Nenghai Yu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] AquaLoRA: Toward White-box Protection for Customized Stable Diffusion Models via Watermark LoRA(https://arxiv.org/abs/)
• Keywords: protect, watermark, diffusion
• Abstract: Diffusion models have achieved remarkable success in generating high-quality images. Recently, the open-source models represented by Stable Diffusion (SD) are thriving and are accessible for customization, giving rise to a vibrant community of creators and enthusiasts. However, the widespread availability of customized SD models has led to copyright concerns, like unauthorized model distribution and unconsented commercial use. To address it, recent works aim to let SD models output watermarked content for post-hoc forensics. Unfortunately, none of them can achieve the challenging white-box protection, wherein the malicious user can easily remove or replace the watermarking module to fail the subsequent verification. For this, we propose \texttt{\method} as the first implementation under this scenario. Briefly, we merge watermark information into the U-Net of Stable Diffusion Models via a watermark Low-Rank Adaptation (LoRA) module in a two-stage manner. For watermark LoRA module, we devise a scaling matrix to achieve flexible message updates without retraining. To guarantee fidelity, we design Prior Preserving Fine-Tuning (PPFT) to ensure watermark learning with minimal impacts on model distribution, validated by proofs. Finally, we conduct extensive experiments and ablation studies to verify our design.

Title: Enhancing Automata Learning with Statistical Machine Learning: A Network Security Case Study

• Authors: Negin Ayoughi, Shiva Nejati, Mehrdad Sabetzadeh, Patricio Saavedra
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Enhancing Automata Learning with Statistical Machine Learning: A Network Security Case Study(https://arxiv.org/abs/)
• Keywords: security
• Abstract: Intrusion detection systems are crucial for network security. Verification of these systems is complicated by various factors, including the heterogeneity of network platforms and the continuously changing landscape of cyber threats. In this paper, we use automata learning to derive state machines from network-traffic data with the objective of supporting behavioural verification of intrusion detection systems. The most innovative aspect of our work is addressing the inability to directly apply existing automata learning techniques to network-traffic data due to the numeric nature of such data. Specifically, we use interpretable machine learning (ML) to partition numeric ranges into intervals that strongly correlate with a system's decisions regarding intrusion detection. These intervals are subsequently used to abstract numeric ranges before automata learning. We apply our ML-enhanced automata learning approach to a commercial network intrusion detection system developed by our industry partner, RabbitRun Technologies. Our approach results in an average 67.5% reduction in the number of states and transitions of the learned state machines, while achieving an average 28% improvement in accuracy compared to using expertise-based numeric data abstraction. Furthermore, the resulting state machines help practitioners in verifying system-level security requirements and exploring previously unknown system behaviours through model checking and temporal query checking. We make our implementation and experimental data available online.

Title: Multi-scale Information Sharing and Selection Network with Boundary Attention for Polyp Segmentation

• Authors: Xiaolu Kang, Zhuoqi Ma, Kang Liu, Yunan Li, Qiguang Miao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Multi-scale Information Sharing and Selection Network with Boundary Attention for Polyp Segmentation(https://arxiv.org/abs/)
• Keywords: segmentation
• Abstract: Polyp segmentation for colonoscopy images is of vital importance in clinical practice. It can provide valuable information for colorectal cancer diagnosis and surgery. While existing methods have achieved relatively good performance, polyp segmentation still faces the following challenges: (1) Varying lighting conditions in colonoscopy and differences in polyp locations, sizes, and morphologies. (2) The indistinct boundary between polyps and surrounding tissue. To address these challenges, we propose a Multi-scale information sharing and selection network (MISNet) for polyp segmentation task. We design a Selectively Shared Fusion Module (SSFM) to enforce information sharing and active selection between low-level and high-level features, thereby enhancing model's ability to capture comprehensive information. We then design a Parallel Attention Module (PAM) to enhance model's attention to boundaries, and a Balancing Weight Module (BWM) to facilitate the continuous refinement of boundary segmentation in the bottom-up process. Experiments on five polyp segmentation datasets demonstrate that MISNet successfully improved the accuracy and clarity of segmentation result, outperforming state-of-the-art methods.

Title: Revisiting the Robust Generalization of Adversarial Prompt Tuning

• Authors: Fan Yang, Mingxuan Xia, Sangzhou Xia, Chicheng Ma, Hui Hui
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Revisiting the Robust Generalization of Adversarial Prompt Tuning(https://arxiv.org/abs/)
• Keywords: defense, attack, robust
• Abstract: Understanding the vulnerability of large-scale pre-trained vision-language models like CLIP against adversarial attacks is key to ensuring zero-shot generalization capacity on various downstream tasks. State-of-the-art defense mechanisms generally adopt prompt learning strategies for adversarial fine-tuning to improve the adversarial robustness of the pre-trained model while keeping the efficiency of adapting to downstream tasks. Such a setup leads to the problem of over-fitting which impedes further improvement of the model's generalization capacity on both clean and adversarial examples. In this work, we propose an adaptive Consistency-guided Adversarial Prompt Tuning (i.e., CAPT) framework that utilizes multi-modal prompt learning to enhance the alignment of image and text features for adversarial examples and leverage the strong generalization of pre-trained CLIP to guide the model-enhancing its robust generalization on adversarial examples while maintaining its accuracy on clean ones. We also design a novel adaptive consistency objective function to balance the consistency of adversarial inputs and clean inputs between the fine-tuning model and the pre-trained model. We conduct extensive experiments across 14 datasets and 4 data sparsity schemes (from 1-shot to full training data settings) to show the superiority of CAPT over other state-of-the-art adaption methods. CAPT demonstrated excellent performance in terms of the in-distribution performance and the generalization under input distribution shift and across datasets.

Title: Towards Modular LLMs by Building and Reusing a Library of LoRAs

• Authors: Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Matheus Pereira, Lucas Caccia, Alessandro Sordoni
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Towards Modular LLMs by Building and Reusing a Library of LoRAs(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: The growing number of parameter-efficient adaptations of a base large language model (LLM) calls for studying whether we can reuse such trained adapters to improve performance for new tasks. We study how to best build a library of adapters given multi-task data and devise techniques for both zero-shot and supervised task generalization through routing in such library. We benchmark existing approaches to build this library and introduce model-based clustering, MBC, a method that groups tasks based on the similarity of their adapter parameters, indirectly optimizing for transfer across the multi-task dataset. To re-use the library, we present a novel zero-shot routing mechanism, Arrow, which enables dynamic selection of the most relevant adapters for new inputs without the need for retraining. We experiment with several LLMs, such as Phi-2 and Mistral, on a wide array of held-out tasks, verifying that MBC-based adapters and Arrow routing lead to superior generalization to new tasks. We make steps towards creating modular, adaptable LLMs that can match or outperform traditional joint training.

Title: Dusk Till Dawn: Self-supervised Nighttime Stereo Depth Estimation using Visual Foundation Models

• Authors: Madhu Vankadari, Samuel Hodgson, Sangyun Shin, Kaichen Zhou Andrew Markham, Niki Trigoni
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Dusk Till Dawn: Self-supervised Nighttime Stereo Depth Estimation using Visual Foundation Models(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Self-supervised depth estimation algorithms rely heavily on frame-warping relationships, exhibiting substantial performance degradation when applied in challenging circumstances, such as low-visibility and nighttime scenarios with varying illumination conditions. Addressing this challenge, we introduce an algorithm designed to achieve accurate self-supervised stereo depth estimation focusing on nighttime conditions. Specifically, we use pretrained visual foundation models to extract generalised features across challenging scenes and present an efficient method for matching and integrating these features from stereo frames. Moreover, to prevent pixels violating photometric consistency assumption from negatively affecting the depth predictions, we propose a novel masking approach designed to filter out such pixels. Lastly, addressing weaknesses in the evaluation of current depth estimation algorithms, we present novel evaluation metrics. Our experiments, conducted on challenging datasets including Oxford RobotCar and Multi-Spectral Stereo, demonstrate the robust improvements realized by our approach. Code is available at: this https URL

Title: LG AI Research & KAIST at EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs

• Authors: Yongrae Jo, Seongyun Lee, Minju Seo, Sung Ju Hwang, Moontae Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] LG AI Research & KAIST at EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Text-to-SQL models are pivotal for making Electronic Health Records (EHRs) accessible to healthcare professionals without SQL knowledge. With the advancements in large language models, these systems have become more adept at translating complex questions into SQL queries. Nonetheless, the critical need for reliability in healthcare necessitates these models to accurately identify unanswerable questions or uncertain predictions, preventing misinformation. To address this problem, we present a self-training strategy using pseudo-labeled unanswerable questions to enhance the reliability of text-to-SQL models for EHRs. This approach includes a two-stage training process followed by a filtering method based on the token entropy and query execution. Our methodology's effectiveness is validated by our top performance in the EHRSQL 2024 shared task, showcasing the potential to improve healthcare decision-making through more reliable text-to-SQL systems.

Title: Automated Multi-level Preference for MLLMs

• Authors: Mengxi Zhang, Kang Rong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Automated Multi-level Preference for MLLMs(https://arxiv.org/abs/)
• Keywords: robust, large language model
• Abstract: Current multimodal Large Language Models (MLLMs) suffer from ``hallucination'', occasionally generating responses that are not grounded in the input images. To tackle this challenge, one promising path is to utilize reinforcement learning from human feedback (RLHF), which steers MLLMs towards learning superior responses while avoiding inferior ones. We rethink the common practice of using binary preferences (\emph{i.e.}, superior, inferior), and find that adopting multi-level preferences (\emph{e.g.}, superior, medium, inferior) is better for two benefits: 1) It narrows the gap between adjacent levels, thereby encouraging MLLMs to discern subtle differences. 2) It further integrates cross-level comparisons (beyond adjacent-level comparisons), thus providing a broader range of comparisons with hallucination examples. To verify our viewpoint, we present the Automated Multi-level Preference (\textbf{AMP}) framework for MLLMs. To facilitate this framework, we first develop an automated dataset generation pipeline that provides high-quality multi-level preference datasets without any human annotators. Furthermore, we design the Multi-level Direct Preference Optimization (MDPO) algorithm to robustly conduct complex multi-level preference learning. Additionally, we propose a new hallucination benchmark, MRHal-Bench. Extensive experiments across public hallucination and general benchmarks, as well as our MRHal-Bench, demonstrate the effectiveness of our proposed method.

Title: Automating PTSD Diagnostics in Clinical Interviews: Leveraging Large Language Models for Trauma Assessments

• Authors: Sichang Tu, Abigail Powers, Natalie Merrill, Negar Fani, Sierra Carter, Stephen Doogan, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Automating PTSD Diagnostics in Clinical Interviews: Leveraging Large Language Models for Trauma Assessments(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: The shortage of clinical workforce presents significant challenges in mental healthcare, limiting access to formal diagnostics and services. We aim to tackle this shortage by integrating a customized large language model (LLM) into the workflow, thus promoting equity in mental healthcare for the general population. Although LLMs have showcased their capability in clinical decision-making, their adaptation to severe conditions like Post-traumatic Stress Disorder (PTSD) remains largely unexplored. Therefore, we collect 411 clinician-administered diagnostic interviews and devise a novel approach to obtain high-quality data. Moreover, we build a comprehensive framework to automate PTSD diagnostic assessments based on interview contents by leveraging two state-of-the-art LLMs, GPT-4 and Llama-2, with potential for broader clinical diagnoses. Our results illustrate strong promise for LLMs, tested on our dataset, to aid clinicians in diagnostic validation. To the best of our knowledge, this is the first AI system that fully automates assessments for mental illness based on clinician-administered interviews.

Title: GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition

• Authors: Mallika Garg, Debashis Ghosh, Pyari Mohan Pradhan
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Transformer model have achieved state-of-the-art results in many applications like NLP, classification, etc. But their exploration in gesture recognition task is still limited. So, we propose a novel GestFormer architecture for dynamic hand gesture recognition. The motivation behind this design is to propose a resource efficient transformer model, since transformers are computationally expensive and very complex. So, we propose to use a pooling based token mixer named PoolFormer, since it uses only pooling layer which is a non-parametric layer instead of quadratic attention. The proposed model also leverages the space-invariant features of the wavelet transform and also the multiscale features are selected using multi-scale pooling. Further, a gated mechanism helps to focus on fine details of the gesture with the contextual information. This enhances the performance of the proposed model compared to the traditional transformer with fewer parameters, when evaluated on dynamic hand gesture datasets, NVidia Dynamic Hand Gesture and Briareo datasets. To prove the efficacy of the proposed model, we have experimented on single as well multimodal inputs such as infrared, normals, depth, optical flow and color images. We have also compared the proposed GestFormer in terms of resource efficiency and number of operations. The source code is available at this https URL.

Title: Wind Power Prediction across Different Locations using Deep Domain Adaptive Learning

• Authors: Md Saiful Islam Sajol, Md Shazid Islam, A S M Jahid Hasan, Md Saydur Rahman, Jubair Yusuf
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Wind Power Prediction across Different Locations using Deep Domain Adaptive Learning(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Accurate prediction of wind power is essential for the grid integration of this intermittent renewable source and aiding grid planners in forecasting available wind capacity. Spatial differences lead to discrepancies in climatological data distributions between two geographically dispersed regions, consequently making the prediction task more difficult. Thus, a prediction model that learns from the data of a particular climatic region can suffer from being less robust. A deep neural network (DNN) based domain adaptive approach is proposed to counter this drawback. Effective weather features from a large set of weather parameters are selected using a random forest approach. A pre-trained model from the source domain is utilized to perform the prediction task, assuming no source data is available during target domain prediction. The weights of only the last few layers of the DNN model are updated throughout the task, keeping the rest of the network unchanged, making the model faster compared to the traditional approaches. The proposed approach demonstrates higher accuracy ranging from 6.14% to even 28.44% compared to the traditional non-adaptive method.

Title: ReasonPix2Pix: Instruction Reasoning Dataset for Advanced Image Editing

• Authors: Ying Jin, Pengyang Ling, Xiaoyi Dong, Pan Zhang, Jiaqi Wang, Dahua Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] ReasonPix2Pix: Instruction Reasoning Dataset for Advanced Image Editing(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: Instruction-based image editing focuses on equipping a generative model with the capacity to adhere to human-written instructions for editing images. Current approaches typically comprehend explicit and specific instructions. However, they often exhibit a deficiency in executing active reasoning capacities required to comprehend instructions that are implicit or insufficiently defined. To enhance active reasoning capabilities and impart intelligence to the editing model, we introduce ReasonPix2Pix, a comprehensive reasoning-attentive instruction editing dataset. The dataset is characterized by 1) reasoning instruction, 2) more realistic images from fine-grained categories, and 3) increased variances between input and edited images. When fine-tuned with our dataset under supervised conditions, the model demonstrates superior performance in instructional editing tasks, independent of whether the tasks require reasoning or not. The code, model, and dataset will be publicly available.

Title: Trustworthy Actionable Perturbations

• Authors: Jesse Friedbaum, Sudarshan Adiga, Ravi Tandon
• Subjects: cs.LG, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Trustworthy Actionable Perturbations(https://arxiv.org/abs/)
• Keywords: attack
• Abstract: Counterfactuals, or modified inputs that lead to a different outcome, are an important tool for understanding the logic used by machine learning classifiers and how to change an undesirable classification. Even if a counterfactual changes a classifier's decision, however, it may not affect the true underlying class probabilities, i.e. the counterfactual may act like an adversarial attack and ``fool'' the classifier. We propose a new framework for creating modified inputs that change the true underlying probabilities in a beneficial way which we call Trustworthy Actionable Perturbations (TAP). This includes a novel verification procedure to ensure that TAP change the true class probabilities instead of acting adversarially. Our framework also includes new cost, reward, and goal definitions that are better suited to effectuating change in the real world. We present PAC-learnability results for our verification procedure and theoretically analyze our new method for measuring reward. We also develop a methodology for creating TAP and compare our results to those achieved by previous counterfactual methods.

Title: Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Dueling

• Authors: Yuwei Cheng, Fan Yao, Xuefeng Liu, Haifeng Xu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Dueling(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: This paper studies Learning from Imperfect Human Feedback (LIHF), motivated by humans' potential irrationality or imperfect perception of true preference. We revisit the classic dueling bandit problem as a model of learning from comparative human feedback, and enrich it by casting the imperfection in human feedback as agnostic corruption to user utilities. We start by identifying the fundamental limits of LIHF and prove a regret lower bound of $\Omega(\max\{T^{1/2},C\})$, even when the total corruption $C$ is known and when the corruption decays gracefully over time (i.e., user feedback becomes increasingly more accurate). We then turn to design robust algorithms applicable in real-world scenarios with arbitrary corruption and unknown $C$. Our key finding is that gradient-based algorithms enjoy a smooth efficiency-robustness tradeoff under corruption by varying their learning rates. Specifically, under general concave user utility, Dueling Bandit Gradient Descent (DBGD) of Yue and Joachims (2009) can be tuned to achieve regret $O(T^{1-\alpha} + T^{ \alpha} C)$ for any given parameter $\alpha \in (0, \frac{1}{4}]$. Additionally, this result enables us to pin down the regret lower bound of the standard DBGD (the $\alpha=1/4$ case) as $\Omega(T^{3/4})$ for the first time, to the best of our knowledge. For strongly concave user utility we show a better tradeoff: there is an algorithm that achieves $O(T^{\alpha} + T^{\frac{1}{2}(1-\alpha)}C)$ for any given $\alpha \in [\frac{1}{2},1)$. Our theoretical insights are corroborated by extensive experiments on real-world recommendation data.

Title: Fuse & Calibrate: A bi-directional Vision-Language Guided Framework for Referring Image Segmentation

• Authors: Yichen Yan, Xingjian He, Sihan Chen, Shichen Lu, Jing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Fuse & Calibrate: A bi-directional Vision-Language Guided Framework for Referring Image Segmentation(https://arxiv.org/abs/)
• Keywords: segmentation
• Abstract: Referring Image Segmentation (RIS) aims to segment an object described in natural language from an image, with the main challenge being a text-to-pixel correlation. Previous methods typically rely on single-modality features, such as vision or language features, to guide the multi-modal fusion process. However, this approach limits the interaction between vision and language, leading to a lack of fine-grained correlation between the language description and pixel-level details during the decoding process. In this paper, we introduce FCNet, a framework that employs a bi-directional guided fusion approach where both vision and language play guiding roles. Specifically, we use a vision-guided approach to conduct initial multi-modal fusion, obtaining multi-modal features that focus on key vision information. We then propose a language-guided calibration module to further calibrate these multi-modal features, ensuring they understand the context of the input sentence. This bi-directional vision-language guided approach produces higher-quality multi-modal features sent to the decoder, facilitating adaptive propagation of fine-grained semantic information from textual features to visual features. Experiments on RefCOCO, RefCOCO+, and G-Ref datasets with various backbones consistently show our approach outperforming state-of-the-art methods.

Title: Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses

• Authors: Thanh Nguyen, Tung M. Luu, Tri Ton, Chang D. Yoo
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses(https://arxiv.org/abs/)
• Keywords: defense, attack, robust
• Abstract: Offline reinforcement learning (RL) addresses the challenge of expensive and high-risk data exploration inherent in RL by pre-training policies on vast amounts of offline data, enabling direct deployment or fine-tuning in real-world environments. However, this training paradigm can compromise policy robustness, leading to degraded performance in practical conditions due to observation perturbations or intentional attacks. While adversarial attacks and defenses have been extensively studied in deep learning, their application in offline RL is limited. This paper proposes a framework to enhance the robustness of offline RL models by leveraging advanced adversarial attacks and defenses. The framework attacks the actor and critic components by perturbing observations during training and using adversarial defenses as regularization to enhance the learned policy. Four attacks and two defenses are introduced and evaluated on the D4RL benchmark. The results show the vulnerability of both the actor and critic to attacks and the effectiveness of the defenses in improving policy robustness. This framework holds promise for enhancing the reliability of offline RL models in practical scenarios.

Title: MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing

• Authors: Siddhant Agarwal, Shivam Sharma, Preslav Nakov, Tanmoy Chakraborty
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Memes have evolved as a prevalent medium for diverse communication, ranging from humour to propaganda. With the rising popularity of image-focused content, there is a growing need to explore its potential harm from different aspects. Previous studies have analyzed memes in closed settings - detecting harm, applying semantic labels, and offering natural language explanations. To extend this research, we introduce MemeMQA, a multimodal question-answering framework aiming to solicit accurate responses to structured questions while providing coherent explanations. We curate MemeMQACorpus, a new dataset featuring 1,880 questions related to 1,122 memes with corresponding answer-explanation pairs. We further propose ARSENAL, a novel two-stage multimodal framework that leverages the reasoning capabilities of LLMs to address MemeMQA. We benchmark MemeMQA using competitive baselines and demonstrate its superiority - ~18% enhanced answer prediction accuracy and distinct text generation lead across various metrics measuring lexical and semantic alignment over the best baseline. We analyze ARSENAL's robustness through diversification of question-set, confounder-based evaluation regarding MemeMQA's generalizability, and modality-specific assessment, enhancing our understanding of meme interpretation in the multimodal communication landscape.

Title: Transformer based neural networks for emotion recognition in conversations

• Authors: Claudiu Creanga, Liviu P. Dinu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Transformer based neural networks for emotion recognition in conversations(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: This paper outlines the approach of the ISDS-NLP team in the SemEval 2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF). For Subtask 1 we obtained a weighted F1 score of 0.43 and placed 12 in the leaderboard. We investigate two distinct approaches: Masked Language Modeling (MLM) and Causal Language Modeling (CLM). For MLM, we employ pre-trained BERT-like models in a multilingual setting, fine-tuning them with a classifier to predict emotions. Experiments with varying input lengths, classifier architectures, and fine-tuning strategies demonstrate the effectiveness of this approach. Additionally, we utilize Mistral 7B Instruct V0.2, a state-of-the-art model, applying zero-shot and few-shot prompting techniques. Our findings indicate that while Mistral shows promise, MLMs currently outperform them in sentence-level emotion classification.

Title: The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback

• Authors: Ruitao Chen, Liwei Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Reinforcement learning from human feedback (RLHF) has contributed to performance improvements in large language models. To tackle its reliance on substantial amounts of human-labeled data, a successful approach is multi-task representation learning, which involves learning a high-quality, low-dimensional representation from a wide range of source tasks. In this paper, we formulate RLHF as the contextual dueling bandit problem and assume a common linear representation. We demonstrate that the sample complexity of source tasks in multi-task RLHF can be reduced by considering task relevance and allocating different sample sizes to source tasks with varying task relevance. We further propose an algorithm to estimate task relevance by a small number of additional data and then learn a policy. We prove that to achieve $\varepsilon-$optimal, the sample complexity of the source tasks can be significantly reduced compared to uniform sampling. Additionally, the sample complexity of the target task is only linear in the dimension of the latent space, thanks to representation learning.

Title: BadActs: A Universal Backdoor Defense in the Activation Space

• Authors: Biao Yi, Sishuo Chen, Yiming Li, Tong Li, Baolei Zhang, Zheli Liu
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] BadActs: A Universal Backdoor Defense in the Activation Space(https://arxiv.org/abs/)
• Keywords: security, defense, attack
• Abstract: Backdoor attacks pose an increasingly severe security threat to Deep Neural Networks (DNNs) during their development stage. In response, backdoor sample purification has emerged as a promising defense mechanism, aiming to eliminate backdoor triggers while preserving the integrity of the clean content in the samples. However, existing approaches have been predominantly focused on the word space, which are ineffective against feature-space triggers and significantly impair performance on clean data. To address this, we introduce a universal backdoor defense that purifies backdoor samples in the activation space by drawing abnormal activations towards optimized minimum clean activation distribution intervals. The advantages of our approach are twofold: (1) By operating in the activation space, our method captures from surface-level information like words to higher-level semantic concepts such as syntax, thus counteracting diverse triggers; (2) the fine-grained continuous nature of the activation space allows for more precise preservation of clean content while removing triggers. Furthermore, we propose a detection module based on statistical information of abnormal activations, to achieve a better trade-off between clean accuracy and defending performance.

Title: TriLoRA: Integrating SVD for Advanced Style Personalization in Text-to-Image Generation

• Authors: Chengcheng Feng, Mu He, Qiuyu Tian, Haojie Yin, Xiaofang Zhao, Hongwei Tang, Xingqiang Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] TriLoRA: Integrating SVD for Advanced Style Personalization in Text-to-Image Generation(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: As deep learning technology continues to advance, image generation models, especially models like Stable Diffusion, are finding increasingly widespread application in visual arts creation. However, these models often face challenges such as overfitting, lack of stability in generated results, and difficulties in accurately capturing the features desired by creators during the fine-tuning process. In response to these challenges, we propose an innovative method that integrates Singular Value Decomposition (SVD) into the Low-Rank Adaptation (LoRA) parameter update strategy, aimed at enhancing the fine-tuning efficiency and output quality of image generation models. By incorporating SVD within the LoRA framework, our method not only effectively reduces the risk of overfitting but also enhances the stability of model outputs, and captures subtle, creator-desired feature adjustments more accurately. We evaluated our method on multiple datasets, and the results show that, compared to traditional fine-tuning methods, our approach significantly improves the model's generalization ability and creative flexibility while maintaining the quality of generation. Moreover, this method maintains LoRA's excellent performance under resource-constrained conditions, allowing for significant improvements in image generation quality without sacrificing the original efficiency and resource advantages.

Title: SimAD: A Simple Dissimilarity-based Approach for Time Series Anomaly Detection

• Authors: Zhijie Zhong, Zhiwen Yu, Xing Xi, Yue Xu, Jiahui Chen, Kaixiang Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] SimAD: A Simple Dissimilarity-based Approach for Time Series Anomaly Detection(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Despite the prevalence of reconstruction-based deep learning methods, time series anomaly detection remains challenging. Existing approaches often struggle with limited temporal contexts, inadequate representation of normal patterns, and flawed evaluation metrics, hindering their effectiveness in identifying aberrant behavior. To address these issues, we introduce $\textbf{SimAD}$, a $\textbf{Sim}$ple dissimilarity-based approach for time series $\textbf{A}$nomaly $\textbf{D}$etection. SimAD incorporates an advanced feature extractor adept at processing extended temporal windows, utilizes the EmbedPatch encoder to integrate normal behavioral patterns comprehensively, and introduces an innovative ContrastFusion module designed to accentuate distributional divergences between normal and abnormal data, thereby enhancing the robustness of anomaly discrimination. Additionally, we propose two robust evaluation metrics, UAff and NAff, addressing the limitations of existing metrics and demonstrating their reliability through theoretical and experimental analyses. Experiments across $\textbf{seven}$ diverse time series datasets demonstrate SimAD's superior performance compared to state-of-the-art methods, achieving relative improvements of $\textbf{19.85%}$ on F1, $\textbf{4.44%}$ on Aff-F1, $\textbf{77.79%}$ on NAff-F1, and $\textbf{9.69%}$ on AUC on six multivariate datasets. Code and pre-trained models are available at this https URL.

Title: Testing the Performance of Face Recognition for People with Down Syndrome

• Authors: Christian Rathgeb, Mathias Ibsen, Denise Hartmann, Simon Hradetzky, Berglind Ólafsdóttir
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Testing the Performance of Face Recognition for People with Down Syndrome(https://arxiv.org/abs/)
• Keywords: biometric, fair
• Abstract: The fairness of biometric systems, in particular facial recognition, is often analysed for larger demographic groups, e.g. female vs. male or black vs. white. In contrast to this, minority groups are commonly ignored. This paper investigates the performance of facial recognition algorithms on individuals with Down syndrome, a common chromosomal abnormality that affects approximately one in 1,000 births per year. To do so, a database of 98 individuals with Down syndrome, each represented by at least five facial images, is semi-automatically collected from YouTube. Subsequently, two facial image quality assessment algorithms and five recognition algorithms are evaluated on the newly collected database and on the public facial image databases CelebA and FRGCv2. The results show that the quality scores of facial images for individuals with Down syndrome are comparable to those of individuals without Down syndrome captured under similar conditions. Furthermore, it is observed that face recognition performance decreases significantly for individuals with Down syndrome, which is largely attributed to the increased likelihood of false matches.

Title: Dynamic Quantum Key Distribution for Microgrids with Distributed Error Correction

• Authors: Suman Rath, Neel Kanth Kundu, Subham Sahoo
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Dynamic Quantum Key Distribution for Microgrids with Distributed Error Correction(https://arxiv.org/abs/)
• Keywords: secure, attack, robust
• Abstract: Quantum key distribution (QKD) has often been hailed as a reliable technology for secure communication in cyber-physical microgrids. Even though unauthorized key measurements are not possible in QKD, attempts to read them can disturb quantum states leading to mutations in the transmitted value. Further, inaccurate quantum keys can lead to erroneous decryption producing garbage values, destabilizing microgrid operation. QKD can also be vulnerable to node-level manipulations incorporating attack values into measurements before they are encrypted at the communication layer. To address these issues, this paper proposes a secure QKD protocol that can identify errors in keys and/or nodal measurements by observing violations in control dynamics. Additionally, the protocol uses a dynamic adjacency matrix-based formulation strategy enabling the affected nodes to reconstruct a trustworthy signal and replace it with the attacked signal in a multi-hop manner. This enables microgrids to perform nominal operations in the presence of adversaries who try to eavesdrop on the system causing an increase in the quantum bit error rate (QBER). We provide several case studies to showcase the robustness of the proposed strategy against eavesdroppers and node manipulations. The results demonstrate that it can resist unwanted observation and attack vectors that manipulate signals before encryption.

Title: Few-Shot API Attack Anomaly Detection in a Classification-by-Retrieval Framework

• Authors: Udi Aharon, Ran Dubin, Amit Dvir, Chen Hajaj
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Few-Shot API Attack Anomaly Detection in a Classification-by-Retrieval Framework(https://arxiv.org/abs/)
• Keywords: security, attack
• Abstract: Application Programming Interface (API) attacks refer to the unauthorized or malicious use of APIs, which are often exploited to gain access to sensitive data or manipulate online systems for illicit purposes. Identifying actors that deceitfully utilize an API poses a demanding problem. Although there have been notable advancements and contributions in the field of API security, there still remains a significant challenge when dealing with attackers who use novel approaches that don't match the well-known payloads commonly seen in attacks. Also, attackers may exploit standard functionalities in unconventional manners and with objectives surpassing their intended boundaries. This means API security needs to be more sophisticated and dynamic than ever, with advanced computational intelligence methods, such as machine learning models that can quickly identify and respond to anomalous behavior. In response to these challenges, we propose a novel few-shot anomaly detection framework, named FT-ANN. This framework is composed of two parts: First, we train a dedicated generic language model for API based on FastText embedding. Next, we use Approximate Nearest Neighbor search in a classification-by-retrieval approach. Our framework enables the development of a lightweight model that can be trained with minimal examples per class or even a model capable of classifying multiple classes. The results show that our framework effectively improves API attack detection accuracy compared to various baselines.

Title: Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory Score Matching

• Authors: Xingyu Miao, Haoran Duan, Varun Ojha, Jun Song, Tejal Shah, Yang Long, Rajiv Ranjan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory Score Matching(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: In this work, we propose a novel Trajectory Score Matching (TSM) method that aims to solve the pseudo ground truth inconsistency problem caused by the accumulated error in Interval Score Matching (ISM) when using the Denoising Diffusion Implicit Models (DDIM) inversion process. Unlike ISM which adopts the inversion process of DDIM to calculate on a single path, our TSM method leverages the inversion process of DDIM to generate two paths from the same starting point for calculation. Since both paths start from the same starting point, TSM can reduce the accumulated error compared to ISM, thus alleviating the problem of pseudo ground truth inconsistency. TSM enhances the stability and consistency of the model's generated paths during the distillation process. We demonstrate this experimentally and further show that ISM is a special case of TSM. Furthermore, to optimize the current multi-stage optimization process from high-resolution text to 3D generation, we adopt Stable Diffusion XL for guidance. In response to the issues of abnormal replication and splitting caused by unstable gradients during the 3D Gaussian splatting process when using Stable Diffusion XL, we propose a pixel-by-pixel gradient clipping method. Extensive experiments show that our model significantly surpasses the state-of-the-art models in terms of visual quality and performance. Code: \url{this https URL}.

Title: WisPerMed at "Discharge Me!": Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC-IV

• Authors: Hendrik Damm, Tabea M. G. Pakull, Bahadır Eryılmaz, Helmut Becker, Ahmad Idrissi-Yaghir, Henning Schäfer, Sergej Schultenkämper, Christoph M. Friedrich
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] WisPerMed at "Discharge Me!": Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC-IV(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: This study aims to leverage state of the art language models to automate generating the "Brief Hospital Course" and "Discharge Instructions" sections of Discharge Summaries from the MIMIC-IV dataset, reducing clinicians' administrative workload. We investigate how automation can improve documentation accuracy, alleviate clinician burnout, and enhance operational efficacy in healthcare facilities. This research was conducted within our participation in the Shared Task Discharge Me! at BioNLP @ ACL 2024. Various strategies were employed, including few-shot learning, instruction tuning, and Dynamic Expert Selection (DES), to develop models capable of generating the required text sections. Notably, utilizing an additional clinical domain-specific dataset demonstrated substantial potential to enhance clinical language processing. The DES method, which optimizes the selection of text outputs from multiple predictions, proved to be especially effective. It achieved the highest overall score of 0.332 in the competition, surpassing single-model outputs. This finding suggests that advanced deep learning methods in combination with DES can effectively automate parts of electronic health record documentation. These advancements could enhance patient care by freeing clinician time for patient interactions. The integration of text selection strategies represents a promising avenue for further research.

Title: Few-Shot API Attack Detection: Overcoming Data Scarcity with GAN-Inspired Learning

• Authors: Udi Aharon, Revital Marbel, Ran Dubin, Amit Dvir, Chen Hajaj
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Few-Shot API Attack Detection: Overcoming Data Scarcity with GAN-Inspired Learning(https://arxiv.org/abs/)
• Keywords: security, attack, robust, transformer, generative
• Abstract: Web applications and APIs face constant threats from malicious actors seeking to exploit vulnerabilities for illicit gains. These threats necessitate robust anomaly detection systems capable of identifying malicious API traffic efficiently despite limited and diverse datasets. This paper proposes a novel few-shot detection approach motivated by Natural Language Processing (NLP) and advanced Generative Adversarial Network (GAN)-inspired techniques. Leveraging state-of-the-art Transformer architectures, particularly RoBERTa, our method enhances the contextual understanding of API requests, leading to improved anomaly detection compared to traditional methods. We showcase the technique's versatility by demonstrating its effectiveness with both Out-of-Distribution (OOD) and Transformer-based binary classification methods on two distinct datasets: CSIC 2010 and ATRDF 2023. Our evaluations reveal consistently enhanced or, at worst, equivalent detection rates across various metrics in most vectors, highlighting the promise of our approach for improving API security.

Title: EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models

• Authors: Yu Huang, Liang Guo, Wanqian Guo, Zhe Tao, Yang Lv, Zhihao Sun, Dongfang Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models(https://arxiv.org/abs/)
• Keywords: robust, large language model
• Abstract: In the field of environmental science, it is crucial to have robust evaluation metrics for large language models to ensure their efficacy and accuracy. We propose EnviroExam, a comprehensive evaluation method designed to assess the knowledge of large language models in the field of environmental science. EnviroExam is based on the curricula of top international universities, covering undergraduate, master's, and doctoral courses, and includes 936 questions across 42 core courses. By conducting 0-shot and 5-shot tests on 31 open-source large language models, EnviroExam reveals the performance differences among these models in the domain of environmental science and provides detailed evaluation standards. The results show that 61.3% of the models passed the 5-shot tests, while 48.39% passed the 0-shot tests. By introducing the coefficient of variation as an indicator, we evaluate the performance of mainstream open-source large language models in environmental science from multiple perspectives, providing effective criteria for selecting and fine-tuning language models in this field. Future research will involve constructing more domain-specific test sets using specialized environmental science textbooks to further enhance the accuracy and specificity of the evaluation.

Title: HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos

• Authors: Qifeng Chen, Rengan Xie, Kai Huang, Qi Wang, Wenting Zheng, Rong Li, Yuchi Huo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: Recently, implicit neural representation has been widely used to generate animatable human avatars. However, the materials and geometry of those representations are coupled in the neural network and hard to edit, which hinders their application in traditional graphics engines. We present a framework for acquiring human avatars that are attached with high-resolution physically-based material textures and triangular mesh from monocular video. Our method introduces a novel information fusion strategy to combine the information from the monocular video and synthesize virtual multi-view images to tackle the sparsity of the input view. We reconstruct humans as deformable neural implicit surfaces and extract triangle mesh in a well-behaved pose as the initial mesh of the next stage. In addition, we introduce an approach to correct the bias for the boundary and size of the coarse mesh extracted. Finally, we adapt prior knowledge of the latent diffusion model at super-resolution in multi-view to distill the decomposed texture. Experiments show that our approach outperforms previous representations in terms of high fidelity, and this explicit result supports deployment on common renderers.

Title: Joint Analysis of Single-Cell Data across Cohorts with Missing Modalities

• Authors: Marianne Arriola, Weishen Pan, Manqi Zhou, Qiannan Zhang, Chang Su, Fei Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Joint Analysis of Single-Cell Data across Cohorts with Missing Modalities(https://arxiv.org/abs/)
• Keywords: robust, generative
• Abstract: Joint analysis of multi-omic single-cell data across cohorts has significantly enhanced the comprehensive analysis of cellular processes. However, most of the existing approaches for this purpose require access to samples with complete modality availability, which is impractical in many real-world scenarios. In this paper, we propose (Single-Cell Cross-Cohort Cross-Category) integration, a novel framework that learns unified cell representations under domain shift without requiring full-modality reference samples. Our generative approach learns rich cross-modal and cross-domain relationships that enable imputation of these missing modalities. Through experiments on real-world multi-omic datasets, we demonstrate that offers a robust solution to single-cell tasks such as cell type clustering, cell type classification, and feature imputation.

Title: MBIAS: Mitigating Bias in Large Language Models While Retaining Context

• Authors: Shaina Raza, Ananya Raval, Veronica Chatrath
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MBIAS: Mitigating Bias in Large Language Models While Retaining Context(https://arxiv.org/abs/)
• Keywords: robust, large language model
• Abstract: In addressing the critical need for safety in Large Language Models (LLMs), it is crucial to ensure that the outputs are not only safe but also retain their contextual accuracy. Many existing LLMs are safe fine-tuned either with safety demonstrations, or rely only on adversarial testing. While able to get safe outputs, they often risk losing contextual meaning as they mitigate bias and toxicity. In response, we present MBIAS, a LLM framework instruction fine-tuned on a custom dataset specifically designed for safety interventions. MBIAS aims to address the significant issues of bias and toxicity in LLMs generations that typically manifest as underrepresentation or negative portrayals across various demographics, including inappropriate linguistic mentions and biased content in social media. We experiment on MBIAS for safety interventions using various configurations, and demonstrate more than a 30\% reduction in overall bias and toxicity while successfully retaining key information. Additionally, a demographic analysis on an out-of-distribution test set confirms the robustness of our approach, with reductions in bias and toxicity exceeding 90\% across various demographics. The dataset and instruction fine-tuned MBIAS are made available to the research community at this https URL.

Title: A Dual Power Grid Cascading Failure Model for the Vulnerability Analysis

• Authors: Tianxin Zhou, Xiang Li, Haibing Lu
• Subjects: cs.LG, cs.ET
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A Dual Power Grid Cascading Failure Model for the Vulnerability Analysis(https://arxiv.org/abs/)
• Keywords: attack, transformer
• Abstract: Considering the attacks against the power grid, one of the most effective approaches could be the attack to the transmission lines that leads to large cascading failures. Hence, the problem of locating the most critical or vulnerable transmission lines for a Power Grid Cascading Failure (PGCF) has drawn much attention from the research society. There exists many deterministic solutions and stochastic approximation algorithms aiming to analyze the power grid vulnerability. However, it has been challenging to reveal the correlations between the transmission lines to identify the critical ones. In this paper, we propose a novel approach of learning such correlations via attention mechanism inspired by the Transformer based models that were initially designated to learn the correlation of words in sentences. Multiple modifications and adjustments are proposed to support the attention mechanism producing an informative correlation matrix, the Attention Matrix. With the Attention Ranking algorithm, we are able to identify the most critical lines. The proposed Dual PGCF model provide a novel and effective analysis to improve the power grid resilience against cascading failure, which is proved by extensive experiment results.

Title: Securing 3rd Party App Integration in Docker-based Cloud Software Ecosystems

• Authors: Christian Binkowski, Stefan Appel, Andreas Aßmuth
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Securing 3rd Party App Integration in Docker-based Cloud Software Ecosystems(https://arxiv.org/abs/)
• Keywords: secure, security, attack
• Abstract: Open software ecosystems are beneficial for customers; they benefit from 3rd party services and applications, e.g. analysis of data using apps, developed and deployed by other companies or open-source communities. One significant advantage of this approach is that other customers may benefit from these newly developed applications as well. Especially software ecosystems utilizing container technologies are prone to certain risks. Docker, in particular, is more vulnerable to attacks than hypervisor based virtualisation as it directly operates on the host system. Docker is a popular representative of containerisation technology which offers a lightweight architecture in order to facilitate the set-up and creation of such software ecosystems. Popular Infrastructure as a Service cloud service providers, like Amazon Web Services or Microsoft Azure, jump on the containerisation bandwagon and provide interfaces for provisioning and managing containers. Companies can benefit from that change of technology and create software ecosystems more efficiently. In this paper, we present a new concept for significant security improvements for cloud-based software ecosystems using Docker for 3rd party app integration. Based on the security features of Docker we describe a secure integration of applications in the cloud environment securely. Our approach considers the whole software lifecycle and includes sandbox testing of potentially dangerous 3rd party apps before these became available to the customers.

Title: Sampling Strategies for Mitigating Bias in Face Synthesis Methods

• Authors: Emmanouil Maragkoudakis, Symeon Papadopoulos, Iraklis Varlamis, Christos Diou
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Sampling Strategies for Mitigating Bias in Face Synthesis Methods(https://arxiv.org/abs/)
• Keywords: protect, generative
• Abstract: Synthetically generated images can be used to create media content or to complement datasets for training image analysis models. Several methods have recently been proposed for the synthesis of high-fidelity face images; however, the potential biases introduced by such methods have not been sufficiently addressed. This paper examines the bias introduced by the widely popular StyleGAN2 generative model trained on the Flickr Faces HQ dataset and proposes two sampling strategies to balance the representation of selected attributes in the generated face images. We focus on two protected attributes, gender and age, and reveal that biases arise in the distribution of randomly sampled images against very young and very old age groups, as well as against female faces. These biases are also assessed for different image quality levels based on the GIQA score. To mitigate bias, we propose two alternative methods for sampling on selected lines or spheres of the latent space to increase the number of generated samples from the under-represented classes. The experimental results show a decrease in bias against underrepresented groups and a more uniform distribution of the protected features at different levels of image quality.

Title: On the Trajectory Regularity of ODE-based Diffusion Sampling

• Authors: Defang Chen, Zhenyu Zhou, Can Wang, Chunhua Shen, Siwei Lyu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] On the Trajectory Regularity of ODE-based Diffusion Sampling(https://arxiv.org/abs/)
• Keywords: diffusion, generative
• Abstract: Diffusion-based generative models use stochastic differential equations (SDEs) and their equivalent ordinary differential equations (ODEs) to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models. We characterize an implicit denoising trajectory and discuss its vital role in forming the coupled sampling trajectory with a strong shape regularity, regardless of the generated content. We also describe a dynamic programming-based scheme to make the time schedule in sampling better fit the underlying trajectory structure. This simple strategy requires minimal modification to any given ODE-based numerical solvers and incurs negligible computational cost, while delivering superior performance in image generation, especially in $5\sim 10$ function evaluations.

Title: Detecting Complex Multi-step Attacks with Explainable Graph Neural Network

• Authors: Wei Liu, Peng Gao, Haotian Zhang, Ke Li, Weiyong Yang, Xingshen Wei, Shuji Wu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Detecting Complex Multi-step Attacks with Explainable Graph Neural Network(https://arxiv.org/abs/)
• Keywords: attack, explainability
• Abstract: Complex multi-step attacks have caused significant damage to numerous critical infrastructures. To detect such attacks, graph neural network based methods have shown promising results by modeling the system's events as a graph. However, existing methods still face several challenges when deployed in practice. First, there is a lack of sufficient real attack data especially considering the large volume of normal data. Second, the modeling of event graphs is challenging due to their dynamic and heterogeneous nature. Third, the lack of explanation in learning models undermines the trustworthiness of such methods in production environments. To address the above challenges, in this paper, we propose an attack detection method, Trace2Vec. The approach first designs an erosion function to augment rare attack samples, and integrates them into the event graphs. Next, it models the event graphs via a continuous-time dynamic heterogeneous graph neural network. Finally, it employs the Monte Carlo tree search algorithm to identify events with greater contributions to the attack, thus enhancing the explainability of the detection result. We have implemented a prototype for Trace2Vec, and the experimental evaluations demonstrate its superior detection and explanation performance compared to existing methods.

Title: UPAM: Unified Prompt Attack in Text-to-Image Generation Models Against Both Textual Filters and Visual Checkers

• Authors: Duo Peng, Qiuhong Ke, Jun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] UPAM: Unified Prompt Attack in Text-to-Image Generation Models Against Both Textual Filters and Visual Checkers(https://arxiv.org/abs/)
• Keywords: security, defense, attack, robust, steal
• Abstract: Text-to-Image (T2I) models have raised security concerns due to their potential to generate inappropriate or harmful images. In this paper, we propose UPAM, a novel framework that investigates the robustness of T2I models from the attack perspective. Unlike most existing attack methods that focus on deceiving textual defenses, UPAM aims to deceive both textual and visual defenses in T2I models. UPAM enables gradient-based optimization, offering greater effectiveness and efficiency than previous methods. Given that T2I models might not return results due to defense mechanisms, we introduce a Sphere-Probing Learning (SPL) scheme to support gradient optimization even when no results are returned. Additionally, we devise a Semantic-Enhancing Learning (SEL) scheme to finetune UPAM for generating target-aligned images. Our framework also ensures attack stealthiness. Extensive experiments demonstrate UPAM's effectiveness and efficiency.

Title: A Unified Approach Towards Active Learning and Out-of-Distribution Detection

• Authors: Sebastian Schmidt, Leonard Schenk, Leo Schwinn, Stephan Günnemann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A Unified Approach Towards Active Learning and Out-of-Distribution Detection(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: When applying deep learning models in open-world scenarios, active learning (AL) strategies are crucial for identifying label candidates from a nearly infinite amount of unlabeled data. In this context, robust out-of-distribution (OOD) detection mechanisms are essential for handling data outside the target distribution of the application. However, current works investigate both problems separately. In this work, we introduce SISOM as the first unified solution for both AL and OOD detection. By leveraging feature space distance metrics SISOM combines the strengths of the currently independent tasks to solve both effectively. We conduct extensive experiments showing the problems arising when migrating between both tasks. In these evaluations SISOM underlined its effectiveness by achieving first place in two of the widely used OpenOOD benchmarks and second place in the remaining one. In AL, SISOM outperforms others and delivers top-1 performance in three benchmarks

Title: A Secure and Privacy-Friendly Logging Scheme

• Authors: Andreas Aßmuth, Robert Duncan, Simon Liebl, Matthias Söllner
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A Secure and Privacy-Friendly Logging Scheme(https://arxiv.org/abs/)
• Keywords: secure, security, privacy, protect, attack, robust
• Abstract: Finding a robust security mechanism for audit trail logging has long been a poorly satisfied goal. There are many reasons for this. The most significant of these is that the audit trail is a highly sought after goal of attackers to ensure that they do not get caught. Thus they have an incredibly strong incentive to prevent companies from succeeding in this worthy aim. Regulation, such as the European Union General Data Protection Regulation, has brought a strong incentive for companies to achieve success in this area due to the punitive level of fines that can now be levied in the event of a successful breach by an attacker. We seek to resolve this issue through the use of an encrypted audit trail process that saves encrypted records to a true immutable database, which can ensure audit trail records are permanently retained in encrypted form, with no possibility of the records being compromised. This ensures compliance with the General Data Protection Regulation can be achieved.

Title: Improved Content Understanding With Effective Use of Multi-task Contrastive Learning

• Authors: Akanksha Bindal, Sudarshan Ramanujam, Dave Golland, TJ Hazen, Tina Jiang, Fengyu Zhang, Peng Yan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Improved Content Understanding With Effective Use of Multi-task Contrastive Learning(https://arxiv.org/abs/)
• Keywords: robust, transformer
• Abstract: In enhancing LinkedIn core content recommendation models, a significant challenge lies in improving their semantic understanding capabilities. This paper addresses the problem by leveraging multi-task learning, a method that has shown promise in various domains. We fine-tune a pre-trained, transformer-based LLM using multi-task contrastive learning with data from a diverse set of semantic labeling tasks. We observe positive transfer, leading to superior performance across all tasks when compared to training independently on each. Our model outperforms the baseline on zero shot learning and offers improved multilingual support, highlighting its potential for broader application. The specialized content embeddings produced by our model outperform generalized embeddings offered by OpenAI on Linkedin dataset and tasks. This work provides a robust foundation for vertical teams across LinkedIn to customize and fine-tune the LLM to their specific applications. Our work offers insights and best practices for the field to build on.

Title: City-Scale Multi-Camera Vehicle Tracking System with Improved Self-Supervised Camera Link Model

• Authors: Yuqiang Lin, Sam Lockyer, Adrian Evans, Markus Zarbock, Nic Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] City-Scale Multi-Camera Vehicle Tracking System with Improved Self-Supervised Camera Link Model(https://arxiv.org/abs/)
• Keywords: extraction
• Abstract: Multi-Target Multi-Camera Tracking (MTMCT) has broad applications and forms the basis for numerous future city-wide systems (e.g. traffic management, crash detection, etc.). However, the challenge of matching vehicle trajectories across different cameras based solely on feature extraction poses significant difficulties. This article introduces an innovative multi-camera vehicle tracking system that utilizes a self-supervised camera link model. In contrast to related works that rely on manual spatial-temporal annotations, our model automatically extracts crucial multi-camera relationships for vehicle matching. The camera link is established through a pre-matching process that evaluates feature similarities, pair numbers, and time variance for high-quality tracks. This process calculates the probability of spatial linkage for all camera combinations, selecting the highest scoring pairs to create camera links. Our approach significantly improves deployment times by eliminating the need for human annotation, offering substantial improvements in efficiency and cost-effectiveness when it comes to real-world application. This pairing process supports cross camera matching by setting spatial-temporal constraints, reducing the searching space for potential vehicle matches. According to our experimental results, the proposed method achieves a new state-of-the-art among automatic camera-link based methods in CityFlow V2 benchmarks with 61.07% IDF1 Score.

Title: Cloud Security and Security Challenges Revisited

• Authors: Fabian Süß, Marco Freimuth, Andreas Aßmuth, George R. S. Weir, Bob Duncan
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Cloud Security and Security Challenges Revisited(https://arxiv.org/abs/)
• Keywords: security, attack
• Abstract: In recent years, Cloud Computing has transformed local businesses and created new business models on the Internet- and Cloud services are still flourishing. But after the emphatic hype in the early years, a more realistic perception of Cloud services has emerged. One reason for this surely is that today, Cloud Computing is considered as an established and well-accepted technology and no longer as a technical novelty. But the second reason for this assessment might also be numerous security issues that Cloud Computing in general or specific Cloud services have experienced since then. In this paper, we revisit attacks on Cloud services and Cloud-related attack vectors that have been published in recent years. We then consider successful or proposed solutions to cope with these challenges. Based on these findings, we apply a security metric in order to rank all these Cloud-related security challenges concerning their severity. This should assist security professionals to prioritize their efforts toward addressing these issues.

Title: NTTSuite: Number Theoretic Transform Benchmarks for Accelerating Encrypted Computation

• Authors: Juran Ding, Yuanzhe Liu, Lingbin Sun, Brandon Reagen
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] NTTSuite: Number Theoretic Transform Benchmarks for Accelerating Encrypted Computation(https://arxiv.org/abs/)
• Keywords: security, privacy, protect
• Abstract: Privacy concerns have thrust privacy-preserving computation into the spotlight. Homomorphic encryption (HE) is a cryptographic system that enables computation to occur directly on encrypted data, providing users with strong privacy (and security) guarantees while using the same services they enjoy today unprotected. While promising, HE has seen little adoption due to extremely high computational overheads, rendering it impractical. Homomorphic encryption (HE) is a cryptographic system that enables computation to occur directly on encrypted data. In this paper we develop a benchmark suite, named NTTSuite, to enable researchers to better address these overheads by studying the primary source of HE's slowdown: the number theoretic transform (NTT). NTTSuite constitutes seven unique NTT algorithms with support for CPUs (C++), GPUs (CUDA), and custom hardware (Catapult HLS).In addition, we propose optimizations to improve the performance of NTT running on FPGAs. We find our implementation outperforms the state-of-the-art by 30%.

Title: Large Language Models Lack Understanding of Character Composition of Words

• Authors: Andrew Shin, Kunitake Kaneko
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Large Language Models Lack Understanding of Character Composition of Words(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable performances on a wide range of natural language tasks. Yet, LLMs' successes have been largely restricted to tasks concerning words, sentences, or documents, and it remains questionable how much they understand the minimal units of text, namely characters. In this paper, we examine contemporary LLMs regarding their ability to understand character composition of words, and show that most of them fail to reliably carry out even the simple tasks that can be handled by humans with perfection. We analyze their behaviors with comparison to token level performances, and discuss the potential directions for future research.

Title: Security of Cloud Services with Low-Performance Devices in Critical Infrastructures

• Authors: Michael Molle, Ulrich Raithel, Dirk Kraemer, Norbert Graß, Matthias Söllner, Andreas Aßmuth
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Security of Cloud Services with Low-Performance Devices in Critical Infrastructures(https://arxiv.org/abs/)
• Keywords: security
• Abstract: As part of the Internet of Things (IoT) and Industry 4.0 Cloud services are increasingly interacting with low-performance devices that are used in automation. This results in security issues that will be presented in this paper. Particular attention is paid to so-called critical infrastructures. The authors intend to work on the addressed security challenges as part of a funded research project, using electrical actuators and battery storages as specific applications. The core ideas of this research project are also presented in this paper.

Title: MapCoder: Multi-Agent Code Generation for Competitive Problem Solving

• Authors: Md. Ashraful Islam, Mohammed Eunus Ali, Md Rizwan Parvez
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MapCoder: Multi-Agent Code Generation for Competitive Problem Solving(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Code synthesis, which requires a deep understanding of complex natural language problem descriptions, generation of code instructions for complex algorithms and data structures, and the successful execution of comprehensive unit tests, presents a significant challenge. While large language models (LLMs) demonstrate impressive proficiency in natural language processing, their performance in code generation tasks remains limited. In this paper, we introduce a new approach to code generation tasks leveraging multi-agent prompting that uniquely replicates the full cycle of program synthesis as observed in human developers. Our framework, MapCoder, consists of four LLM agents specifically designed to emulate the stages of this cycle: recalling relevant examples, planning, code generation, and debugging. After conducting thorough experiments, with multiple LLM ablations and analyses across eight challenging competitive problem-solving and program synthesis benchmarks, MapCoder showcases remarkable code generation capabilities, achieving new state-of-the-art results (pass@1) on HumanEval (93.9%), MBPP (83.1%), APPS (22.0%), CodeContests (28.5%), and xCodeEval (45.3%). Moreover, our method consistently delivers superior performance across various programming languages and varying problem difficulties. We open-source our framework at this https URL.

Title: Can Public LLMs be used for Self-Diagnosis of Medical Conditions ?

• Authors: Nikil Sharan Prabahar Balasubramanian, Sagnik Dakshit
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Can Public LLMs be used for Self-Diagnosis of Medical Conditions ?(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: The advancements in the development of Large Language Models have evolved as a transformative paradigm in conversational tasks which has led to its integration in the critical domain of healthcare. With LLMs becoming widely popular and their public access through open-source models, there is a need to investigate their potential and limitations. One such critical task where LLMs are applied but require a deeper understanding is that of self-diagnosis of medical conditions in the interest of public health. The widespread integration of Gemini with Google search, GPT-4.0 with Bing search, has led to shift in trend of self-diagnosis from search engine LLMs. In this paper, we prepare a prompt engineered dataset of 10000 samples and test the performance on the general task of self-diagnosis. We compare the performance of GPT-4.0 and Gemini model on the task of self-diagnosis and record accuracies of 63.07% and 6.01% respectively. We also discuss the challenges, limitations, and potential of both Gemini and GPT-4.0 for the task of self-diagnosis to facilitate future research and towards the broader impact of general public knowledge. Furthermore, we demonstrate the potential and improvement in performance for the task of self-diagnosis using Retrieval Augmented Generation.

Title: Discrete-state Continuous-time Diffusion for Graph Generation

• Authors: Zhe Xu, Ruizhong Qiu, Yuzhong Chen, Huiyuan Chen, Xiran Fan, Menghai Pan, Zhichen Zeng, Mahashweta Das, Hanghang Tong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Discrete-state Continuous-time Diffusion for Graph Generation(https://arxiv.org/abs/)
• Keywords: diffusion, generative
• Abstract: Graph is a prevalent discrete data structure, whose generation has wide applications such as drug discovery and circuit design. Diffusion generative models, as an emerging research focus, have been applied to graph generation tasks. Overall, according to the space of states and time steps, diffusion generative models can be categorized into discrete-/continuous-state discrete-/continuous-time fashions. In this paper, we formulate the graph diffusion generation in a discrete-state continuous-time setting, which has never been studied in previous graph diffusion models. The rationale of such a formulation is to preserve the discrete nature of graph-structured data and meanwhile provide flexible sampling trade-offs between sample quality and efficiency. Analysis shows that our training objective is closely related to generation quality, and our proposed generation framework enjoys ideal invariant/equivariant properties concerning the permutation of node ordering. Our proposed model shows competitive empirical performance against state-of-the-art graph generation solutions on various benchmarks and, at the same time, can flexibly trade off the generation quality and efficiency in the sampling phase.

Title: Large Language Models are Biased Reinforcement Learners

• Authors: William M. Hayes, Nicolas Yax, Stefano Palminteri
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Large Language Models are Biased Reinforcement Learners(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: In-context learning enables large language models (LLMs) to perform a variety of tasks, including learning to make reward-maximizing choices in simple bandit tasks. Given their potential use as (autonomous) decision-making agents, it is important to understand how these models perform such reinforcement learning (RL) tasks and the extent to which they are susceptible to biases. Motivated by the fact that, in humans, it has been widely documented that the value of an outcome depends on how it compares to other local outcomes, the present study focuses on whether similar value encoding biases apply to how LLMs encode rewarding outcomes. Results from experiments with multiple bandit tasks and models show that LLMs exhibit behavioral signatures of a relative value bias. Adding explicit outcome comparisons to the prompt produces opposing effects on performance, enhancing maximization in trained choice sets but impairing generalization to new choice sets. Computational cognitive modeling reveals that LLM behavior is well-described by a simple RL algorithm that incorporates relative values at the outcome encoding stage. Lastly, we present preliminary evidence that the observed biases are not limited to fine-tuned LLMs, and that relative value processing is detectable in the final hidden layer activations of a raw, pretrained model. These findings have important implications for the use of LLMs in decision-making applications.

Title: MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

• Authors: Jianbo Dai, Jianqiao Lu, Yunlong Feng, Rongju Ruan, Ming Cheng, Haochen Tan, Zhijiang Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have greatly improved code generation, specifically at the function level. For instance, GPT-4 has achieved an 88.4% pass rate on HumanEval. However, this draws into question the adequacy of existing benchmarks in thoroughly assessing function-level code generation capabilities. Our study analyzed two common benchmarks, HumanEval and MBPP, and found that these might not thoroughly evaluate LLMs' code generation capacities due to limitations in quality, difficulty, and granularity. To resolve this, we introduce the Mostly Hard Python Problems (MHPP) dataset, consisting of 140 unique human-curated problems. By focusing on the combination of natural language and code reasoning, MHPP gauges LLMs' abilities to comprehend specifications and restrictions, engage in multi-step reasoning, and apply coding knowledge effectively. Initial evaluations of 22 LLMs using MHPP showed many high-performing models on HumanEval failed to achieve similar success on MHPP. Moreover, MHPP highlighted various previously undiscovered limitations within various LLMs, leading us to believe that it could pave the way for a better understanding of LLMs' capabilities and limitations. Dataset and code are available at this https URL.

Title: Review of deep learning models for crypto price prediction: implementation and evaluation

• Authors: Jingyang Wu, Xinyi Zhang, Fangyixuan Huang, Haochen Zhou, Rohtiash Chandra
• Subjects: cs.LG, q-fin.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Review of deep learning models for crypto price prediction: implementation and evaluation(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: There has been much interest in accurate cryptocurrency price forecast models by investors and researchers. Deep Learning models are prominent machine learning techniques that have transformed various fields and have shown potential for finance and economics. Although various deep learning models have been explored for cryptocurrency price forecasting, it is not clear which models are suitable due to high market volatility. In this study, we review the literature about deep learning for cryptocurrency price forecasting and evaluate novel deep learning models for cryptocurrency stock price prediction. Our deep learning models include variants of long short-term memory (LSTM) recurrent neural networks, variants of convolutional neural networks (CNNs), and the Transformer model. We evaluate univariate and multivariate approaches for multi-step ahead predicting of cryptocurrencies close-price. Our results show that the univariate LSTM model variants perform best for cryptocurrency predictions. We also carry out volatility analysis on the four cryptocurrencies which reveals significant fluctuations in their prices throughout the COVID-19 pandemic. Additionally, we investigate the prediction accuracy of two scenarios identified by different training sets for the models. First, we use the pre-COVID-19 datasets to model cryptocurrency close-price forecasting during the early period of COVID-19. Secondly, we utilise data from the COVID-19 period to predict prices for 2023 to 2024.

Title: On Robust Reinforcement Learning with Lipschitz-Bounded Policy Networks

• Authors: Nicholas H. Barbara, Ruigang Wang, Ian R. Manchester
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] On Robust Reinforcement Learning with Lipschitz-Bounded Policy Networks(https://arxiv.org/abs/)
• Keywords: attack, robust
• Abstract: This paper presents a study of robust policy networks in deep reinforcement learning. We investigate the benefits of policy parameterizations that naturally satisfy constraints on their Lipschitz bound, analyzing their empirical performance and robustness on two representative problems: pendulum swing-up and Atari Pong. We illustrate that policy networks with small Lipschitz bounds are significantly more robust to disturbances, random noise, and targeted adversarial attacks than unconstrained policies composed of vanilla multi-layer perceptrons or convolutional neural networks. Moreover, we find that choosing a policy parameterization with a non-conservative Lipschitz bound and an expressive, nonlinear layer architecture gives the user much finer control over the performance-robustness trade-off than existing state-of-the-art methods based on spectral normalization.

Title: A GAN-Based Data Poisoning Attack Against Federated Learning Systems and Its Countermeasure

• Authors: Wei Sun, Bo Gao, Ke Xiong, Yuwei Wang, Pingyi Fan, Khaled Ben Letaief
• Subjects: cs.CR, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A GAN-Based Data Poisoning Attack Against Federated Learning Systems and Its Countermeasure(https://arxiv.org/abs/)
• Keywords: security, privacy, defense, attack, steal, federate, generative
• Abstract: As a distributed machine learning paradigm, federated learning (FL) is collaboratively carried out on privately owned datasets but without direct data access. Although the original intention is to allay data privacy concerns, "available but not visible" data in FL potentially brings new security threats, particularly poisoning attacks that target such "not visible" local data. Initial attempts have been made to conduct data poisoning attacks against FL systems, but cannot be fully successful due to their high chance of causing statistical anomalies. To unleash the potential for truly "invisible" attacks and build a more deterrent threat model, in this paper, a new data poisoning attack model named VagueGAN is proposed, which can generate seemingly legitimate but noisy poisoned data by untraditionally taking advantage of generative adversarial network (GAN) variants. Capable of manipulating the quality of poisoned data on demand, VagueGAN enables to trade-off attack effectiveness and stealthiness. Furthermore, a cost-effective countermeasure named Model Consistency-Based Defense (MCD) is proposed to identify GAN-poisoned data or models after finding out the consistency of GAN outputs. Extensive experiments on multiple datasets indicate that our attack method is generally much more stealthy as well as more effective in degrading FL performance with low complexity. Our defense method is also shown to be more competent in identifying GAN-poisoned data or models. The source codes are publicly available at \href{this https URL}{this https URL}.

Title: Unifying 3D Vision-Language Understanding via Promptable Queries

• Authors: Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, Qing Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Unifying 3D Vision-Language Understanding via Promptable Queries(https://arxiv.org/abs/)
• Keywords: segmentation
• Abstract: A unified model for 3D vision-language (3D-VL) understanding is expected to take various scene representations and perform a wide range of tasks in a 3D scene. However, a considerable gap exists between existing methods and such a unified model, due to the independent application of representation and insufficient exploration of 3D multi-task training. In this paper, we introduce PQ3D, a unified model capable of using Promptable Queries to tackle a wide range of 3D-VL tasks, from low-level instance segmentation to high-level reasoning and planning. This is achieved through three key innovations: (1) unifying various 3D scene representations (i.e., voxels, point clouds, multi-view images) into a shared 3D coordinate space by segment-level grouping, (2) an attention-based query decoder for task-specific information retrieval guided by prompts, and (3) universal output heads for different tasks to support multi-task training. Tested across ten diverse 3D-VL datasets, PQ3D demonstrates impressive performance on these tasks, setting new records on most benchmarks. Particularly, PQ3D improves the state-of-the-art on ScanNet200 by 1.8% (AP), ScanRefer by 5.4% (acc@0.5), Multi3DRefer by 11.7% (F1@0.5), and Scan2Cap by 13.4% (CIDEr@0.5). Moreover, PQ3D supports flexible inference with individual or combined forms of available 3D representations, e.g., solely voxel input.

Title: MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning

• Authors: Sanchit Sinha, Yuguang Yue, Victor Soto, Mayank Kulkarni, Jianhua Lu, Aidong Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning(https://arxiv.org/abs/)
• Keywords: robust, large language model
• Abstract: Adapting large language models (LLMs) to unseen tasks with in-context training samples without fine-tuning remains an important research problem. To learn a robust LLM that adapts well to unseen tasks, multiple meta-training approaches have been proposed such as MetaICL and MetaICT, which involve meta-training pre-trained LLMs on a wide variety of diverse tasks. These meta-training approaches essentially perform in-context multi-task fine-tuning and evaluate on a disjointed test set of tasks. Even though they achieve impressive performance, their goal is never to compute a truly general set of parameters. In this paper, we propose MAML-en-LLM, a novel method for meta-training LLMs, which can learn truly generalizable parameters that not only perform well on disjointed tasks but also adapts to unseen tasks. We see an average increase of 2% on unseen domains in the performance while a massive 4% improvement on adaptation performance. Furthermore, we demonstrate that MAML-en-LLM outperforms baselines in settings with limited amount of training data on both seen and unseen domains by an average of 2%. Finally, we discuss the effects of type of tasks, optimizers and task complexity, an avenue barely explored in meta-training literature. Exhaustive experiments across 7 task settings along with two data settings demonstrate that models trained with MAML-en-LLM outperform SOTA meta-training approaches.

Title: NetMamba: Efficient Network Traffic Classification via Pre-training Unidirectional Mamba

• Authors: Tongze Wang, Xiaohui Xie, Wenduo Wang, Chuyi Wang, Youjian Zhao, Yong Cui
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] NetMamba: Efficient Network Traffic Classification via Pre-training Unidirectional Mamba(https://arxiv.org/abs/)
• Keywords: security, transformer
• Abstract: Network traffic classification is a crucial research area aiming to enhance service quality, streamline network management, and bolster cybersecurity. To address the growing complexity of transmission encryption techniques, various machine learning and deep learning methods have been proposed. However, existing approaches encounter two main challenges. Firstly, they struggle with model inefficiency due to the quadratic complexity of the widely used Transformer architecture. Secondly, they suffer from unreliable traffic representation because of discarding important byte information while retaining unwanted biases. To address these challenges, we propose NetMamba, an efficient linear-time state space model equipped with a comprehensive traffic representation scheme. We replace the Transformer with our specially selected and improved Mamba architecture for the networking field to address efficiency issues. In addition, we design a scheme for traffic representation, which is used to extract valid information from massive traffic while removing biased information. Evaluation experiments on six public datasets encompassing three main classification tasks showcase NetMamba's superior classification performance compared to state-of-the-art baselines. It achieves up to 4.83\% higher accuracy and 4.64\% higher f1 score on encrypted traffic classification tasks. Additionally, NetMamba demonstrates excellent efficiency, improving inference speed by 2.24 times while maintaining comparably low memory usage. Furthermore, NetMamba exhibits superior few-shot learning abilities, achieving better classification performance with fewer labeled data. To the best of our knowledge, NetMamba is the first model to tailor the Mamba architecture for networking.

Title: Comparisons Are All You Need for Optimizing Smooth Functions

• Authors: Chenyi Zhang, Tongyang Li
• Subjects: cs.LG, cs.DS, math.OC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Comparisons Are All You Need for Optimizing Smooth Functions(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: When optimizing machine learning models, there are various scenarios where gradient computations are challenging or even infeasible. Furthermore, in reinforcement learning (RL), preference-based RL that only compares between options has wide applications, including reinforcement learning with human feedback in large language models. In this paper, we systematically study optimization of a smooth function $f\colon\mathbb{R}^n\to\mathbb{R}$ only assuming an oracle that compares function values at two points and tells which is larger. When $f$ is convex, we give two algorithms using $\tilde{O}(n/\epsilon)$ and $\tilde{O}(n^{2})$ comparison queries to find an $\epsilon$-optimal solution, respectively. When $f$ is nonconvex, our algorithm uses $\tilde{O}(n/\epsilon^2)$ comparison queries to find an $\epsilon$-approximate stationary point. All these results match the best-known zeroth-order algorithms with function evaluation queries in $n$ dependence, thus suggest that \emph{comparisons are all you need for optimizing smooth functions using derivative-free methods}. In addition, we also give an algorithm for escaping saddle points and reaching an $\epsilon$-second order stationary point of a nonconvex $f$, using $\tilde{O}(n^{1.5}/\epsilon^{2.5})$ comparison queries.

Title: Biometrics-Based Authenticated Key Exchange with Multi-Factor Fuzzy Extractor

• Authors: Hong Yen Tran, Jiankun Hu, Wen Hu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Biometrics-Based Authenticated Key Exchange with Multi-Factor Fuzzy Extractor(https://arxiv.org/abs/)
• Keywords: secure, attack, biometric
• Abstract: Existing fuzzy extractors and similar methods provide an effective way for extracting a secret key from a user's biometric data, but are susceptible to impersonation attack: once a valid biometric sample is captured, the scheme is no longer secure. We propose a novel multi-factor fuzzy extractor that integrates both a user's secret (e.g., a password) and a user's biometrics in the generation and reconstruction process of a cryptographic key. We then employ this multi-factor fuzzy extractor to construct personal identity credentials which can be used in a new multi-factor authenticated key exchange protocol that possesses multiple important features. First, the protocol provides mutual authentication. Second, the user and service provider can authenticate each other without the involvement of the identity authority. Third, the protocol can prevent user impersonation from a compromised identity authority. Finally, even when both a biometric sample and the secret are captured, the user can re-register to create a new credential using a new secret (reusable/reissued identity credentials). Most existing works on multi-factor authenticated key exchange only have a subset of these features. We formally prove that the proposed protocol is semantically secure. Our experiments carried out on the finger vein dataset SDUMLA achieved a low equal error rate (EER) of 0.04%, a reasonable averaged computation time of 0.93 seconds for the user and service provider to authenticate and establish a shared session key, and a small communication overhead of only 448 bytes.

Title: Effective In-Context Example Selection through Data Compression

• Authors: Zhongxiang Sun, Kepu Zhang, Haoyu Wang, Xiao Zhang, Jun Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Effective In-Context Example Selection through Data Compression(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: In-context learning has been extensively validated in large language models. However, the mechanism and selection strategy for in-context example selection, which is a crucial ingredient in this approach, lacks systematic and in-depth research. In this paper, we propose a data compression approach to the selection of in-context examples. We introduce a two-stage method that can effectively choose relevant examples and retain sufficient information about the training dataset within the in-context examples. Our method shows a significant improvement of an average of 5.90% across five different real-world datasets using four language models.

Title: VCformer: Variable Correlation Transformer with Inherent Lagged Correlation for Multivariate Time Series Forecasting

• Authors: Yingnan Yang, Qingling Zhu, Jianyong Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] VCformer: Variable Correlation Transformer with Inherent Lagged Correlation for Multivariate Time Series Forecasting(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Multivariate time series (MTS) forecasting has been extensively applied across diverse domains, such as weather prediction and energy consumption. However, current studies still rely on the vanilla point-wise self-attention mechanism to capture cross-variable dependencies, which is inadequate in extracting the intricate cross-correlation implied between variables. To fill this gap, we propose Variable Correlation Transformer (VCformer), which utilizes Variable Correlation Attention (VCA) module to mine the correlations among variables. Specifically, based on the stochastic process theory, VCA calculates and integrates the cross-correlation scores corresponding to different lags between queries and keys, thereby enhancing its ability to uncover multivariate relationships. Additionally, inspired by Koopman dynamics theory, we also develop Koopman Temporal Detector (KTD) to better address the non-stationarity in time series. The two key components enable VCformer to extract both multivariate correlations and temporal dependencies. Our extensive experiments on eight real-world datasets demonstrate the effectiveness of VCformer, achieving top-tier performance compared to other state-of-the-art baseline models. Code is available at this repository: this https URL.

Title: FIFO-Diffusion: Generating Infinite Videos from Text without Training

• Authors: Jihwan Kim, Junoh Kang, Jinyoung Choi, Bohyung Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] FIFO-Diffusion: Generating Infinite Videos from Text without Training(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: We propose a novel inference technique based on a pretrained diffusion model for text-conditional video generation. Our approach, called FIFO-Diffusion, is conceptually capable of generating infinitely long videos without training. This is achieved by iteratively performing diagonal denoising, which concurrently processes a series of consecutive frames with increasing noise levels in a queue; our method dequeues a fully denoised frame at the head while enqueuing a new random noise frame at the tail. However, diagonal denoising is a double-edged sword as the frames near the tail can take advantage of cleaner ones by forward reference but such a strategy induces the discrepancy between training and inference. Hence, we introduce latent partitioning to reduce the training-inference gap and lookahead denoising to leverage the benefit of forward referencing. We have demonstrated the promising results and effectiveness of the proposed methods on existing text-to-video generation baselines.

Title: NubbleDrop: A Simple Way to Improve Matching Strategy for Prompted One-Shot Segmentation

• Authors: Zhiyu Xu, Qingliang Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] NubbleDrop: A Simple Way to Improve Matching Strategy for Prompted One-Shot Segmentation(https://arxiv.org/abs/)
• Keywords: robust, segmentation
• Abstract: Driven by large data trained segmentation models, such as SAM , research in one-shot segmentation has experienced significant advancements. Recent contributions like PerSAM and MATCHER , presented at ICLR 2024, utilize a similar approach by leveraging SAM with one or a few reference images to generate high quality segmentation masks for target images. Specifically, they utilize raw encoded features to compute cosine similarity between patches within reference and target images along the channel dimension, effectively generating prompt points or boxes for the target images a technique referred to as the matching strategy. However, relying solely on raw features might introduce biases and lack robustness for such a complex task. To address this concern, we delve into the issues of feature interaction and uneven distribution inherent in raw feature based matching. In this paper, we propose a simple and training-free method to enhance the validity and robustness of the matching strategy at no additional computational cost (NubbleDrop). The core concept involves randomly dropping feature channels (setting them to zero) during the matching process, thereby preventing models from being influenced by channels containing deceptive information. This technique mimics discarding pathological nubbles, and it can be seamlessly applied to other similarity computing scenarios. We conduct a comprehensive set of experiments, considering a wide range of factors, to demonstrate the effectiveness and validity of our proposed method. Our results showcase the significant improvements achieved through this simmple and straightforward approach.

Title: BOSC: A Backdoor-based Framework for Open Set Synthetic Image Attribution

• Authors: Jun Wang, Benedetta Tondi, Mauro Barni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] BOSC: A Backdoor-based Framework for Open Set Synthetic Image Attribution(https://arxiv.org/abs/)
• Keywords: attack, robust, generative
• Abstract: Synthetic image attribution addresses the problem of tracing back the origin of images produced by generative models. Extensive efforts have been made to explore unique representations of generative models and use them to attribute a synthetic image to the model that produced it. Most of the methods classify the models or the architectures among those in a closed set without considering the possibility that the system is fed with samples produced by unknown architectures. With the continuous progress of AI technology, new generative architectures continuously appear, thus driving the attention of researchers towards the development of tools capable of working in open-set scenarios. In this paper, we propose a framework for open set attribution of synthetic images, named BOSC (Backdoor-based Open Set Classification), that relies on the concept of backdoor attacks to design a classifier with rejection option. BOSC works by purposely injecting class-specific triggers inside a portion of the images in the training set to induce the network to establish a matching between class features and trigger features. The behavior of the trained model with respect to triggered samples is then exploited at test time to perform sample rejection using an ad-hoc score. Experiments show that the proposed method has good performance, always surpassing the state-of-the-art. Robustness against image processing is also very good. Although we designed our method for the task of synthetic image attribution, the proposed framework is a general one and can be used for other image forensic applications.

Title: Automated Coastline Extraction Using Edge Detection Algorithms

• Authors: Conor O'Sullivan, Seamus Coveney, Xavier Monteys, Soumyabrata Dev
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Automated Coastline Extraction Using Edge Detection Algorithms(https://arxiv.org/abs/)
• Keywords: extraction
• Abstract: We analyse the effectiveness of edge detection algorithms for the purpose of automatically extracting coastlines from satellite images. Four algorithms - Canny, Sobel, Scharr and Prewitt are compared visually and using metrics. With an average SSIM of 0.8, Canny detected edges that were closest to the reference edges. However, the algorithm had difficulty distinguishing noisy edges, e.g. due to development, from coastline edges. In addition, histogram equalization and Gaussian blur were shown to improve the effectiveness of the edge detection algorithms by up to 1.5 and 1.6 times respectively.

Title: DEMO: A Statistical Perspective for Efficient Image-Text Matching

• Authors: Fan Zhang, Xian-Sheng Hua, Chong Chen, Xiao Luo
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] DEMO: A Statistical Perspective for Efficient Image-Text Matching(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Image-text matching has been a long-standing problem, which seeks to connect vision and language through semantic understanding. Due to the capability to manage large-scale raw data, unsupervised hashing-based approaches have gained prominence recently. They typically construct a semantic similarity structure using the natural distance, which subsequently provides guidance to the model optimization process. However, the similarity structure could be biased at the boundaries of semantic distributions, causing error accumulation during sequential optimization. To tackle this, we introduce a novel hashing approach termed Distribution-based Structure Mining with Consistency Learning (DEMO) for efficient image-text matching. From a statistical view, DEMO characterizes each image using multiple augmented views, which are considered as samples drawn from its intrinsic semantic distribution. Then, we employ a non-parametric distribution divergence to ensure a robust and precise similarity structure. In addition, we introduce collaborative consistency learning which not only preserves the similarity structure in the Hamming space but also encourages consistency between retrieval distribution from different directions in a self-supervised manner. Through extensive experiments on three benchmark image-text matching datasets, we demonstrate that DEMO achieves superior performance compared with many state-of-the-art methods.

Title: Towards in-situ Psychological Profiling of Cybercriminals Using Dynamically Generated Deception Environments

• Authors: Jacob Quibell
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Towards in-situ Psychological Profiling of Cybercriminals Using Dynamically Generated Deception Environments(https://arxiv.org/abs/)
• Keywords: security, attack
• Abstract: Cybercrime is estimated to cost the global economy almost \$10 trillion annually and with businesses and governments reporting an ever-increasing number of successful cyber-attacks there is a growing demand to rethink the strategy towards cyber security. The traditional, perimeter security approach to cyber defence has so far proved inadequate to combat the growing threat of cybercrime. Cyber deception offers a promising alternative by creating a dynamic defence environment. Deceptive techniques aim to mislead attackers, diverting them from critical assets whilst simultaneously gathering cyber threat intelligence on the threat actor. This article presents a proof-of-concept (POC) cyber deception system that has been developed to capture the profile of an attacker in-situ, during a simulated cyber-attack in real time. By dynamically and autonomously generating deception material based on the observed attacker behaviour and analysing how the attacker interacts with the deception material, the system outputs a prediction on the attacker's motive. The article also explores how this POC can be expanded to infer other features of the attacker's profile such as psychological characteristics. By dynamically and autonomously generating deception material based on observed attacker behaviour and analysing how the attacker interacts with the deception material, the system outputs a prediciton on the attacker's motive. The article also explores how this POC can be expanded to infer other features of the attacker's profile such as psychological characteristics.

Title: Interpreting a Semantic Segmentation Model for Coastline Detection

• Authors: Conor O'Sullivan, Seamus Coveney, Xavier Monteys, Soumyabrata Dev
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Interpreting a Semantic Segmentation Model for Coastline Detection(https://arxiv.org/abs/)
• Keywords: extraction, segmentation
• Abstract: We interpret a deep-learning semantic segmentation model used to classify coastline satellite images into land and water. This is to build trust in the model and gain new insight into the process of coastal water body extraction. Specifically, we seek to understand which spectral bands are important for predicting segmentation masks. This is done using a permutation importance approach. Results show that the NIR is the most important spectral band. Permuting this band lead to a decrease in accuracy of 38.12 percentage points. This is followed by Water Vapour, SWIR 1, and Blue bands with 2.58, 0.78 and 0.19 respectively. Water Vapour is not typically used in water indices and these results suggest it may be useful for water body extraction. Permuting, the Coastal Aerosol, Green, Red, RE1, RE2, RE3, RE4, and SWIR 2 bands did not decrease accuracy. This suggests they could be excluded from future model builds reducing complexity and computational requirements.

Title: Online Action Representation using Change Detection and Symbolic Programming

• Authors: Vishnu S Nair, Sneha Sree, Jayaraj Joseph, Mohanasankar Sivaprakasam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Online Action Representation using Change Detection and Symbolic Programming(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: This paper addresses the critical need for online action representation, which is essential for various applications like rehabilitation, surveillance, etc. The task can be defined as representation of actions as soon as they happen in a streaming video without access to video frames in the future. Most of the existing methods use predefined window sizes for video segments, which is a restrictive assumption on the dynamics. The proposed method employs a change detection algorithm to automatically segment action sequences, which form meaningful sub-actions and subsequently fit symbolic generative motion programs to the clipped segments. We determine the start time and end time of segments using change detection followed by a piece-wise linear fit algorithm on joint angle and bone length sequences. Domain-specific symbolic primitives are fit to pose keypoint trajectories of those extracted segments in order to obtain a higher level semantic representation. Since this representation is part-based, it is complementary to the compositional nature of human actions, i.e., a complex activity can be broken down into elementary sub-actions. We show the effectiveness of this representation in the downstream task of class agnostic repetition detection. We propose a repetition counting algorithm based on consecutive similarity matching of primitives, which can do online repetition counting. We also compare the results with a similar but offline repetition counting algorithm. The results of the experiments demonstrate that, despite operating online, the proposed method performs better or on par with the existing method.

Title: Diffusion-Based Hierarchical Image Steganography

• Authors: Youmin Xu, Xuanyu Zhang, Jiwen Yu, Chong Mou, Xiandong Meng, Jian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Diffusion-Based Hierarchical Image Steganography(https://arxiv.org/abs/)
• Keywords: secure, security, privacy, protect, robust, diffusion
• Abstract: This paper introduces Hierarchical Image Steganography, a novel method that enhances the security and capacity of embedding multiple images into a single container using diffusion models. HIS assigns varying levels of robustness to images based on their importance, ensuring enhanced protection against manipulation. It adaptively exploits the robustness of the Diffusion Model alongside the reversibility of the Flow Model. The integration of Embed-Flow and Enhance-Flow improves embedding efficiency and image recovery quality, respectively, setting HIS apart from conventional multi-image steganography techniques. This innovative structure can autonomously generate a container image, thereby securely and efficiently concealing multiple images and text. Rigorous subjective and objective evaluations underscore our advantage in analytical resistance, robustness, and capacity, illustrating its expansive applicability in content safeguarding and privacy fortification.

Title: Simple-Sampling and Hard-Mixup with Prototypes to Rebalance Contrastive Learning for Text Classification

• Authors: Mengyu Li, Yonghao Liu, Fausto Giunchiglia, Xiaoyue Feng, Renchu Guan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Simple-Sampling and Hard-Mixup with Prototypes to Rebalance Contrastive Learning for Text Classification(https://arxiv.org/abs/)
• Keywords: robust, large language model
• Abstract: Text classification is a crucial and fundamental task in natural language processing. Compared with the previous learning paradigm of pre-training and fine-tuning by cross entropy loss, the recently proposed supervised contrastive learning approach has received tremendous attention due to its powerful feature learning capability and robustness. Although several studies have incorporated this technique for text classification, some limitations remain. First, many text datasets are imbalanced, and the learning mechanism of supervised contrastive learning is sensitive to data imbalance, which may harm the model performance. Moreover, these models leverage separate classification branch with cross entropy and supervised contrastive learning branch without explicit mutual guidance. To this end, we propose a novel model named SharpReCL for imbalanced text classification tasks. First, we obtain the prototype vector of each class in the balanced classification branch to act as a representation of each class. Then, by further explicitly leveraging the prototype vectors, we construct a proper and sufficient target sample set with the same size for each class to perform the supervised contrastive learning procedure. The empirical results show the effectiveness of our model, which even outperforms popular large language models across several datasets.

Title: Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors

• Authors: Chun-Yin Huang, Kartik Srinivas, Xin Zhang, Xiaoxiao Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors(https://arxiv.org/abs/)
• Keywords: privacy, federate
• Abstract: Conventional Federated Learning (FL) involves collaborative training of a global model while maintaining user data privacy. One of its branches, decentralized FL, is a serverless network that allows clients to own and optimize different local models separately, which results in saving management and communication resources. Despite the promising advancements in decentralized FL, it may reduce model generalizability due to lacking a global model. In this scenario, managing data and model heterogeneity among clients becomes a crucial problem, which poses a unique challenge that must be overcome: How can every client's local model learn generalizable representation in a decentralized manner? To address this challenge, we propose a novel Decentralized FL technique by introducing Synthetic Anchors, dubbed as DeSA. Based on the theory of domain adaptation and Knowledge Distillation (KD), we theoretically and empirically show that synthesizing global anchors based on raw data distribution facilitates mutual knowledge transfer. We further design two effective regularization terms for local training: 1) REG loss that regularizes the distribution of the client's latent embedding with the anchors and 2) KD loss that enables clients to learn from others. Through extensive experiments on diverse client data distributions, we showcase the effectiveness of DeSA in enhancing both inter- and intra-domain accuracy of each client.

Title: RobMOT: Robust 3D Multi-Object Tracking by Observational Noise and State Estimation Drift Mitigation on LiDAR PointCloud

• Authors: Mohamed Nagy, Naoufel Werghi, Bilal Hassan, Jorge Dias, Majid Khonji
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] RobMOT: Robust 3D Multi-Object Tracking by Observational Noise and State Estimation Drift Mitigation on LiDAR PointCloud(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: This work addresses the inherited limitations in the current state-of-the-art 3D multi-object tracking (MOT) methods that follow the tracking-by-detection paradigm, notably trajectory estimation drift for long-occluded objects in LiDAR point cloud streams acquired by autonomous cars. In addition, the absence of adequate track legitimacy verification results in ghost track accumulation. To tackle these issues, we introduce a two-fold innovation. Firstly, we propose refinement in Kalman filter that enhances trajectory drift noise mitigation, resulting in more robust state estimation for occluded objects. Secondly, we propose a novel online track validity mechanism to distinguish between legitimate and ghost tracks combined with a multi-stage observational gating process for incoming observations. This mechanism substantially reduces ghost tracks by up to 80\% and improves HOTA by 7\%. Accordingly, we propose an online 3D MOT framework, RobMOT, that demonstrates superior performance over the top-performing state-of-the-art methods, including deep learning approaches, across various detectors with up to 3.28\% margin in MOTA and 2.36\% in HOTA. RobMOT excels under challenging conditions, such as prolonged occlusions and the tracking of distant objects, with up to 59\% enhancement in processing latency.

Title: From Fourier to Neural ODEs: Flow matching for modeling complex systems

• Authors: Xin Li, Jingdong Zhang, Qunxi Zhu, Chengli Zhao, Xue Zhang, Xiaojun Duan, Wei Lin
• Subjects: cs.LG, physics.ed-ph
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] From Fourier to Neural ODEs: Flow matching for modeling complex systems(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Modeling complex systems using standard neural ordinary differential equations (NODEs) often faces some essential challenges, including high computational costs and susceptibility to local optima. To address these challenges, we propose a simulation-free framework, called Fourier NODEs (FNODEs), that effectively trains NODEs by directly matching the target vector field based on Fourier analysis. Specifically, we employ the Fourier analysis to estimate temporal and potential high-order spatial gradients from noisy observational data. We then incorporate the estimated spatial gradients as additional inputs to a neural network. Furthermore, we utilize the estimated temporal gradient as the optimization objective for the output of the neural network. Later, the trained neural network generates more data points through an ODE solver without participating in the computational graph, facilitating more accurate estimations of gradients based on Fourier analysis. These two steps form a positive feedback loop, enabling accurate dynamics modeling in our framework. Consequently, our approach outperforms state-of-the-art methods in terms of training time, dynamics prediction, and robustness. Finally, we demonstrate the superior performance of our framework using a number of representative complex systems.

Title: An Invisible Backdoor Attack Based On Semantic Feature

• Authors: Yangming Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] An Invisible Backdoor Attack Based On Semantic Feature(https://arxiv.org/abs/)
• Keywords: defense, attack, robust, steal
• Abstract: Backdoor attacks have severely threatened deep neural network (DNN) models in the past several years. These attacks can occur in almost every stage of the deep learning pipeline. Although the attacked model behaves normally on benign samples, it makes wrong predictions for samples containing triggers. However, most existing attacks use visible patterns (e.g., a patch or image transformations) as triggers, which are vulnerable to human inspection. In this paper, we propose a novel backdoor attack, making imperceptible changes. Concretely, our attack first utilizes the pre-trained victim model to extract low-level and high-level semantic features from clean images and generates trigger pattern associated with high-level features based on channel attention. Then, the encoder model generates poisoned images based on the trigger and extracted low-level semantic features without causing noticeable feature loss. We evaluate our attack on three prominent image classification DNN across three standard datasets. The results demonstrate that our attack achieves high attack success rates while maintaining robustness against backdoor defenses. Furthermore, we conduct extensive image similarity experiments to emphasize the stealthiness of our attack strategy.

Title: Uncertainty-Aware PPG-2-ECG for Enhanced Cardiovascular Diagnosis using Diffusion Models

• Authors: Omer Belhasin, Idan Kligvasser, George Leifman, Regev Cohen, Erin Rainaldi, Li-Fang Cheng, Nishant Verma, Paul Varghese, Ehud Rivlin, Michael Elad
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Uncertainty-Aware PPG-2-ECG for Enhanced Cardiovascular Diagnosis using Diffusion Models(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: Analyzing the cardiovascular system condition via Electrocardiography (ECG) is a common and highly effective approach, and it has been practiced and perfected over many decades. ECG sensing is non-invasive and relatively easy to acquire, and yet it is still cumbersome for holter monitoring tests that may span over hours and even days. A possible alternative in this context is Photoplethysmography (PPG): An optically-based signal that measures blood volume fluctuations, as typically sensed by conventional ``wearable devices''. While PPG presents clear advantages in acquisition, convenience, and cost-effectiveness, ECG provides more comprehensive information, allowing for a more precise detection of heart conditions. This implies that a conversion from PPG to ECG, as recently discussed in the literature, inherently involves an unavoidable level of uncertainty. In this paper we introduce a novel methodology for addressing the PPG-2-ECG conversion, and offer an enhanced classification of cardiovascular conditions using the given PPG, all while taking into account the uncertainties arising from the conversion process. We provide a mathematical justification for our proposed computational approach, and present empirical studies demonstrating its superior performance compared to state-of-the-art baseline methods.

Title: Quantile Activation: departing from single point estimation for better generalization across distortions

• Authors: Aditya Challa, Sravan Danda, Laurent Najman, Snehanshu Saha
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Quantile Activation: departing from single point estimation for better generalization across distortions(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: A classifier is, in its essence, a function which takes an input and returns the class of the input and implicitly assumes an underlying distribution. We argue in this article that one has to move away from this basic tenet to obtain generalisation across distributions. Specifically, the class of the sample should depend on the points from its context distribution for better generalisation across distributions. How does one achieve this? The key idea is to adapt the outputs of each neuron of the network to its context distribution. We propose quantile activation, QACT, which, in simple terms, outputs the relative quantile of the sample in its context distribution, instead of the actual values in traditional networks. The scope of this article is to validate the proposed activation across several experimental settings, and compare it with conventional techniques. For this, we use the datasets developed to test robustness against distortions CIFAR10C, CIFAR100C, MNISTC, TinyImagenetC, and show that we achieve a significantly higher generalisation across distortions than the conventional classifiers, across different architectures. Although this paper is only a proof of concept, we surprisingly find that this approach outperforms DINOv2(small) at large distortions, even though DINOv2 is trained with a far bigger network on a considerably larger dataset.

Title: SEEP: Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks

• Authors: Xuanli He, Qiongkai Xu, Jun Wang, Benjamin I. P. Rubinstein, Trevor Cohn
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] SEEP: Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks(https://arxiv.org/abs/)
• Keywords: protect, defense, attack
• Abstract: Modern NLP models are often trained on public datasets drawn from diverse sources, rendering them vulnerable to data poisoning attacks. These attacks can manipulate the model's behavior in ways engineered by the attacker. One such tactic involves the implantation of backdoors, achieved by poisoning specific training instances with a textual trigger and a target class label. Several strategies have been proposed to mitigate the risks associated with backdoor attacks by identifying and removing suspected poisoned examples. However, we observe that these strategies fail to offer effective protection against several advanced backdoor attacks. To remedy this deficiency, we propose a novel defensive mechanism that first exploits training dynamics to identify poisoned samples with high precision, followed by a label propagation step to improve recall and thus remove the majority of poisoned instances. Compared with recent advanced defense methods, our method considerably reduces the success rates of several backdoor attacks while maintaining high classification accuracy on clean test sets.

Title: A Multi-Perspective Analysis of Memorization in Large Language Models

• Authors: Bowen Chen, Namgi Han, Yusuke Miyao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A Multi-Perspective Analysis of Memorization in Large Language Models(https://arxiv.org/abs/)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs), trained on massive corpora with billions of parameters, show unprecedented performance in various fields. Though surprised by their excellent performances, researchers also noticed some special behaviors of those LLMs. One of those behaviors is memorization, in which LLMs can generate the same content used to train them. Though previous research has discussed memorization, the memorization of LLMs still lacks explanation, especially the cause of memorization and the dynamics of generating them. In this research, we comprehensively discussed memorization from various perspectives and extended the discussion scope to not only just the memorized content but also less and unmemorized content. Through various studies, we found that: (1) Through experiments, we revealed the relation of memorization between model size, continuation size, and context size. Further, we showed how unmemorized sentences transition to memorized sentences. (2) Through embedding analysis, we showed the distribution and decoding dynamics across model size in embedding space for sentences with different memorization scores. The n-gram statistics analysis presents d (3) An analysis over n-gram and entropy decoding dynamics discovered a boundary effect when the model starts to generate memorized sentences or unmemorized sentences. (4)We trained a Transformer model to predict the memorization of different models, showing that it is possible to predict memorizations by context.

Title: Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applications

• Authors: Subhankar Maity, Aniket Deroy, Sudeshna Sarkar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applications(https://arxiv.org/abs/)
• Keywords: generative, large language model
• Abstract: In the era of generative artificial intelligence (AI), the fusion of large language models (LLMs) offers unprecedented opportunities for innovation in the field of modern education. We embark on an exploration of prompted LLMs within the context of educational and assessment applications to uncover their potential. Through a series of carefully crafted research questions, we investigate the effectiveness of prompt-based techniques in generating open-ended questions from school-level textbooks, assess their efficiency in generating open-ended questions from undergraduate-level technical textbooks, and explore the feasibility of employing a chain-of-thought inspired multi-stage prompting approach for language-agnostic multiple-choice question (MCQ) generation. Additionally, we evaluate the ability of prompted LLMs for language learning, exemplified through a case study in the low-resource Indian language Bengali, to explain Bengali grammatical errors. We also evaluate the potential of prompted LLMs to assess human resource (HR) spoken interview transcripts. By juxtaposing the capabilities of LLMs with those of human experts across various educational tasks and domains, our aim is to shed light on the potential and limitations of LLMs in reshaping educational practices.

Title: Securing Health Data on the Blockchain: A Differential Privacy and Federated Learning Framework

• Authors: Daniel Commey, Sena Hounsinou, Garth V. Crosby
• Subjects: cs.CR, cs.CY, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Securing Health Data on the Blockchain: A Differential Privacy and Federated Learning Framework(https://arxiv.org/abs/)
• Keywords: secure, privacy, protect, attack, federate
• Abstract: This study proposes a framework to enhance privacy in Blockchain-based Internet of Things (BIoT) systems used in the healthcare sector. The framework addresses the challenge of leveraging health data for analytics while protecting patient privacy. To achieve this, the study integrates Differential Privacy (DP) with Federated Learning (FL) to protect sensitive health data collected by IoT nodes. The proposed framework utilizes dynamic personalization and adaptive noise distribution strategies to balance privacy and data utility. Additionally, blockchain technology ensures secure and transparent aggregation and storage of model updates. Experimental results on the SVHN dataset demonstrate that the proposed framework achieves strong privacy guarantees against various attack scenarios while maintaining high accuracy in health analytics tasks. For 15 rounds of federated learning with an epsilon value of 8.0, the model obtains an accuracy of 64.50%. The blockchain integration, utilizing Ethereum, Ganache, this http URL, and IPFS, exhibits an average transaction latency of around 6 seconds and consistent gas consumption across rounds, validating the practicality and feasibility of the proposed approach.

Title: SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization

• Authors: Jialong Guo, Xinghao Chen, Yehui Tang, Yunhe Wang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Transformers have become foundational architectures for both natural language and computer vision tasks. However, the high computational cost makes it quite challenging to deploy on resource-constraint devices. This paper investigates the computational bottleneck modules of efficient transformer, i.e., normalization layers and attention modules. LayerNorm is commonly used in transformer architectures but is not computational friendly due to statistic calculation during inference. However, replacing LayerNorm with more efficient BatchNorm in transformer often leads to inferior performance and collapse in training. To address this problem, we propose a novel method named PRepBN to progressively replace LayerNorm with re-parameterized BatchNorm in training. Moreover, we propose a simplified linear attention (SLA) module that is simple yet effective to achieve strong performance. Extensive experiments on image classification as well as object detection demonstrate the effectiveness of our proposed method. For example, our SLAB-Swin obtains $83.6\%$ top-1 accuracy on ImageNet-1K with $16.2$ms latency, which is $2.4$ms less than that of Flatten-Swin with $0.1\%$ higher accuracy. We also evaluated our method for language modeling task and obtain comparable performance and lower latency.Codes are publicly available at this https URL and this https URL.

Title: How to integrate cloud service, data analytic and machine learning technique to reduce cyber risks associated with the modern cloud based infrastructure

• Authors: Upakar Bhatta
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] How to integrate cloud service, data analytic and machine learning technique to reduce cyber risks associated with the modern cloud based infrastructure(https://arxiv.org/abs/)
• Keywords: security
• Abstract: The combination of cloud technology, machine learning, and data visualization techniques allows hybrid enterprise networks to hold massive volumes of data and provide employees and customers easy access to these cloud data. These massive collections of complex data sets are facing security challenges. While cloud platforms are more vulnerable to security threats and traditional security technologies are unable to cope with the rapid data explosion in cloud platforms, machine learning powered security solutions and data visualization techniques are playing instrumental roles in detecting security threat, data breaches, and automatic finding software vulnerabilities. The purpose of this paper is to present some of the widely used cloud services, machine learning techniques and data visualization approach and demonstrate how to integrate cloud service, data analytic and machine learning techniques that can be used to detect and reduce cyber risks associated with the modern cloud based infrastructure. In this paper I applied the machine learning supervised classifier to design a model based on well-known UNSW-NB15 dataset to predict the network behavior metrics and demonstrated how data analytics techniques can be integrated to visualize network traffics.

Title: Switched Flow Matching: Eliminating Singularities via Switching ODEs

• Authors: Qunxi Zhu, Wei Lin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Switched Flow Matching: Eliminating Singularities via Switching ODEs(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: Continuous-time generative models, such as Flow Matching (FM), construct probability paths to transport between one distribution and another through the simulation-free learning of the neural ordinary differential equations (ODEs). During inference, however, the learned model often requires multiple neural network evaluations to accurately integrate the flow, resulting in a slow sampling speed. We attribute the reason to the inherent (joint) heterogeneity of source and/or target distributions, namely the singularity problem, which poses challenges for training the neural ODEs effectively. To address this issue, we propose a more general framework, termed Switched FM (SFM), that eliminates singularities via switching ODEs, as opposed to using a uniform ODE in FM. Importantly, we theoretically show that FM cannot transport between two simple distributions due to the existence and uniqueness of initial value problems of ODEs, while these limitations can be well tackled by SFM. From an orthogonal perspective, our framework can seamlessly integrate with the existing advanced techniques, such as minibatch optimal transport, to further enhance the straightness of the flow, yielding a more efficient sampling process with reduced costs. We demonstrate the effectiveness of the newly proposed SFM through several numerical examples.

Title: Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts

• Authors: Baolong Bi, Shenghua Liu, Lingrui Mei, Yiwei Wang, Pengliang Ji, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts(https://arxiv.org/abs/)
• Keywords: interpretability, large language model
• Abstract: The knowledge within large language models (LLMs) may become outdated quickly. While in-context editing (ICE) is currently the most effective method for knowledge editing (KE), it is constrained by the black-box modeling of LLMs and thus lacks interpretability. Our work aims to elucidate the superior performance of ICE on the KE by analyzing the impacts of in-context new knowledge on token-wise distributions. We observe that despite a significant boost in logits of the new knowledge, the performance of is still hindered by stubborn knowledge. Stubborn knowledge refers to as facts that have gained excessive confidence during pretraining, making it hard to edit effectively. To address this issue and further enhance the performance of ICE, we propose a novel approach termed $\textbf{De}$coding by $\textbf{C}$ontrasting $\textbf{K}$nowledge (DeCK). DeCK derives the distribution of the next token by contrasting the logits obtained from the newly edited knowledge guided by ICE with those from the unedited parametric knowledge. Our experiments consistently demonstrate that DeCK enhances the confidence of LLMs in edited facts. For instance, it improves the performance of LLaMA3-8B-instruct on MQuAKE by up to 219%, demonstrating its capability to strengthen ICE in the editing of stubborn knowledge. Our work paves the way to develop the both effective and accountable KE methods for LLMs. (The source code is available at: $\href{this https URL}{\text{this https URL.}}$ )

Title: Nickel and Diming Your GAN: A Dual-Method Approach to Enhancing GAN Efficiency via Knowledge Distillation

• Authors: Sangyeop Yeo, Yoojin Jang, Jaejun Yoo
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Nickel and Diming Your GAN: A Dual-Method Approach to Enhancing GAN Efficiency via Knowledge Distillation(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: In this paper, we address the challenge of compressing generative adversarial networks (GANs) for deployment in resource-constrained environments by proposing two novel methodologies: Distribution Matching for Efficient compression (DiME) and Network Interactive Compression via Knowledge Exchange and Learning (NICKEL). DiME employs foundation models as embedding kernels for efficient distribution matching, leveraging maximum mean discrepancy to facilitate effective knowledge distillation. Simultaneously, NICKEL employs an interactive compression method that enhances the communication between the student generator and discriminator, achieving a balanced and stable compression process. Our comprehensive evaluation on the StyleGAN2 architecture with the FFHQ dataset shows the effectiveness of our approach, with NICKEL & DiME achieving FID scores of 10.45 and 15.93 at compression rates of 95.73% and 98.92%, respectively. Remarkably, our methods sustain generative quality even at an extreme compression rate of 99.69%, surpassing the previous state-of-the-art performance by a large margin. These findings not only demonstrate our methodologies' capacity to significantly lower GANs' computational demands but also pave the way for deploying high-quality GAN models in settings with limited resources. Our code will be released soon.

Title: Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention

• Authors: Peng Li, Yuan Liu, Xiaoxiao Long, Feihu Zhang, Cheng Lin, Mengfei Li, Xingqun Qi, Shanghang Zhang, Wenhan Luo, Ping Tan, Wenping Wang, Qifeng Liu, Yike Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: In this paper, we introduce Era3D, a novel multiview diffusion method that generates high-resolution multiview images from a single-view image. Despite significant advancements in multiview generation, existing methods still suffer from camera prior mismatch, inefficacy, and low resolution, resulting in poor-quality multiview images. Specifically, these methods assume that the input images should comply with a predefined camera type, e.g. a perspective camera with a fixed focal length, leading to distorted shapes when the assumption fails. Moreover, the full-image or dense multiview attention they employ leads to an exponential explosion of computational complexity as image resolution increases, resulting in prohibitively expensive training costs. To bridge the gap between assumption and reality, Era3D first proposes a diffusion-based camera prediction module to estimate the focal length and elevation of the input image, which allows our method to generate images without shape distortions. Furthermore, a simple but efficient attention layer, named row-wise attention, is used to enforce epipolar priors in the multiview diffusion, facilitating efficient cross-view information fusion. Consequently, compared with state-of-the-art methods, Era3D generates high-quality multiview images with up to a 512*512 resolution while reducing computation complexity by 12x times. Comprehensive experiments demonstrate that Era3D can reconstruct high-quality and detailed 3D meshes from diverse single-view input images, significantly outperforming baseline multiview diffusion methods.

Title: Novel Interpretable and Robust Web-based AI Platform for Phishing Email Detection

• Authors: Abdulla Al-Subaiey, Mohammed Al-Thani, Naser Abdullah Alam, Kaniz Fatema Antora, Amith Khandakar, SM Ashfaq Uz Zaman
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Novel Interpretable and Robust Web-based AI Platform for Phishing Email Detection(https://arxiv.org/abs/)
• Keywords: security, robust
• Abstract: Phishing emails continue to pose a significant threat, causing financial losses and security breaches. This study addresses limitations in existing research, such as reliance on proprietary datasets and lack of real-world application, by proposing a high-performance machine learning model for email classification. Utilizing a comprehensive and largest available public dataset, the model achieves a f1 score of 0.99 and is designed for deployment within relevant applications. Additionally, Explainable AI (XAI) is integrated to enhance user trust. This research offers a practical and highly accurate solution, contributing to the fight against phishing by empowering users with a real-time web-based application for phishing email detection.

Title: Computer Vision in the Food Industry: Accurate, Real-time, and Automatic Food Recognition with Pretrained MobileNetV2

• Authors: Shayan Rokhva, Babak Teimourpour, Amir Hossein Soltani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Computer Vision in the Food Industry: Accurate, Real-time, and Automatic Food Recognition with Pretrained MobileNetV2(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: In contemporary society, the application of artificial intelligence for automatic food recognition offers substantial potential for nutrition tracking, reducing food waste, and enhancing productivity in food production and consumption scenarios. Modern technologies such as Computer Vision and Deep Learning are highly beneficial, enabling machines to learn automatically, thereby facilitating automatic visual recognition. Despite some research in this field, the challenge of achieving accurate automatic food recognition quickly remains a significant research gap. Some models have been developed and implemented, but maintaining high performance swiftly, with low computational cost and low access to expensive hardware accelerators, still needs further exploration and research. This study employs the pretrained MobileNetV2 model, which is efficient and fast, for food recognition on the public Food11 dataset, comprising 16643 images. It also utilizes various techniques such as dataset understanding, transfer learning, data augmentation, regularization, dynamic learning rate, hyperparameter tuning, and consideration of images in different sizes to enhance performance and robustness. These techniques aid in choosing appropriate metrics, achieving better performance, avoiding overfitting and accuracy fluctuations, speeding up the model, and increasing the generalization of findings, making the study and its results applicable to practical applications. Despite employing a light model with a simpler structure and fewer trainable parameters compared to some deep and dense models in the deep learning area, it achieved commendable accuracy in a short time. This underscores the potential for practical implementation, which is the main intention of this study.

Title: Searching Realistic-Looking Adversarial Objects For Autonomous Driving Systems

• Authors: Shengxiang Sun, Shenzhe Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Searching Realistic-Looking Adversarial Objects For Autonomous Driving Systems(https://arxiv.org/abs/)
• Keywords: attack, robust
• Abstract: Numerous studies on adversarial attacks targeting self-driving policies fail to incorporate realistic-looking adversarial objects, limiting real-world applicability. Building upon prior research that facilitated the transition of adversarial objects from simulations to practical applications, this paper discusses a modified gradient-based texture optimization method to discover realistic-looking adversarial objects. While retaining the core architecture and techniques of the prior research, the proposed addition involves an entity termed the 'Judge'. This agent assesses the texture of a rendered object, assigning a probability score reflecting its realism. This score is integrated into the loss function to encourage the NeRF object renderer to concurrently learn realistic and adversarial textures. The paper analyzes four strategies for developing a robust 'Judge': 1) Leveraging cutting-edge vision-language models. 2) Fine-tuning open-sourced vision-language models. 3) Pretraining neurosymbolic systems. 4) Utilizing traditional image processing techniques. Our findings indicate that strategies 1) and 4) yield less reliable outcomes, pointing towards strategies 2) or 3) as more promising directions for future research.

Title: Geometry-Aware Instrumental Variable Regression

• Authors: Heiner Kremer, Bernhard Schölkopf
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Geometry-Aware Instrumental Variable Regression(https://arxiv.org/abs/)
• Keywords: attack, robust
• Abstract: Instrumental variable (IV) regression can be approached through its formulation in terms of conditional moment restrictions (CMR). Building on variants of the generalized method of moments, most CMR estimators are implicitly based on approximating the population data distribution via reweightings of the empirical sample. While for large sample sizes, in the independent identically distributed (IID) setting, reweightings can provide sufficient flexibility, they might fail to capture the relevant information in presence of corrupted data or data prone to adversarial attacks. To address these shortcomings, we propose the Sinkhorn Method of Moments, an optimal transport-based IV estimator that takes into account the geometry of the data manifold through data-derivative information. We provide a simple plug-and-play implementation of our method that performs on par with related estimators in standard settings but improves robustness against data corruption and adversarial attacks.

Title: Zero-Shot Stance Detection using Contextual Data Generation with LLMs

• Authors: Ghazaleh Mahmoudi, Babak Behkamkia, Sauleh Eetemadi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Zero-Shot Stance Detection using Contextual Data Generation with LLMs(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Stance detection, the classification of attitudes expressed in a text towards a specific topic, is vital for applications like fake news detection and opinion mining. However, the scarcity of labeled data remains a challenge for this task. To address this problem, we propose Dynamic Model Adaptation with Contextual Data Generation (DyMoAdapt) that combines Few-Shot Learning and Large Language Models. In this approach, we aim to fine-tune an existing model at test time. We achieve this by generating new topic-specific data using GPT-3. This method could enhance performance by allowing the adaptation of the model to new topics. However, the results did not increase as we expected. Furthermore, we introduce the Multi Generated Topic VAST (MGT-VAST) dataset, which extends VAST using GPT-3. In this dataset, each context is associated with multiple topics, allowing the model to understand the relationship between contexts and various potential topics

Title: Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology

• Authors: Andrew H. Song, Richard J. Chen, Tong Ding, Drew F.K. Williamson, Guillaume Jaume, Faisal Mahmood
• Subjects: cs.CV, cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology(https://arxiv.org/abs/)
• Keywords: interpretability
• Abstract: Representation learning of pathology whole-slide images (WSIs) has been has primarily relied on weak supervision with Multiple Instance Learning (MIL). However, the slide representations resulting from this approach are highly tailored to specific clinical tasks, which limits their expressivity and generalization, particularly in scenarios with limited data. Instead, we hypothesize that morphological redundancy in tissue can be leveraged to build a task-agnostic slide representation in an unsupervised fashion. To this end, we introduce PANTHER, a prototype-based approach rooted in the Gaussian mixture model that summarizes the set of WSI patches into a much smaller set of morphological prototypes. Specifically, each patch is assumed to have been generated from a mixture distribution, where each mixture component represents a morphological exemplar. Utilizing the estimated mixture parameters, we then construct a compact slide representation that can be readily used for a wide range of downstream tasks. By performing an extensive evaluation of PANTHER on subtyping and survival tasks using 13 datasets, we show that 1) PANTHER outperforms or is on par with supervised MIL baselines and 2) the analysis of morphological prototypes brings new qualitative and quantitative insights into model interpretability.

Title: Movie Revenue Prediction using Machine Learning Models

• Authors: Vikranth Udandarao, Pratyush Gupta
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Movie Revenue Prediction using Machine Learning Models(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: In the contemporary film industry, accurately predicting a movie's earnings is paramount for maximizing profitability. This project aims to develop a machine learning model for predicting movie earnings based on input features like the movie name, the MPAA rating of the movie, the genre of the movie, the year of release of the movie, the IMDb Rating, the votes by the watchers, the director, the writer and the leading cast, the country of production of the movie, the budget of the movie, the production company and the runtime of the movie. Through a structured methodology involving data collection, preprocessing, analysis, model selection, evaluation, and improvement, a robust predictive model is constructed. Linear Regression, Decision Trees, Random Forest Regression, Bagging, XGBoosting and Gradient Boosting have been trained and tested. Model improvement strategies include hyperparameter tuning and cross-validation. The resulting model offers promising accuracy and generalization, facilitating informed decision-making in the film industry to maximize profits.

Title: Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study

• Authors: Hadeel Saadany, Ashraf Tantawy, Constantin Orasan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like other state-of-the-art AI generative systems, are prone to errors that are deemed machine hallucinations. The problem with NMT hallucinations is that they are remarkably \textit{fluent} hallucinations. Since they are trained to produce grammatically correct utterances, NMT systems are capable of producing mistranslations that are too fluent to be recognised by both users of the MT tool, as well as by automatic quality metrics that are used to gauge their performance. In this paper, we introduce an authentic dataset of machine translation critical errors to point to the ethical and safety issues involved in the common use of MT. The dataset comprises mistranslations of Arabic mental health postings manually annotated with critical error types. We also show how the commonly used quality metrics do not penalise critical errors and highlight this as a critical issue that merits further attention from researchers.

Title: FADet: A Multi-sensor 3D Object Detection Network based on Local Featured Attention

• Authors: Ziang Guo, Zakhar Yagudin, Selamawit Asfaw, Artem Lykov, Dzmitry Tsetserukou
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] FADet: A Multi-sensor 3D Object Detection Network based on Local Featured Attention(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Camera, LiDAR and radar are common perception sensors for autonomous driving tasks. Robust prediction of 3D object detection is optimally based on the fusion of these sensors. To exploit their abilities wisely remains a challenge because each of these sensors has its own characteristics. In this paper, we propose FADet, a multi-sensor 3D detection network, which specifically studies the characteristics of different sensors based on our local featured attention modules. For camera images, we propose dual-attention-based sub-module. For LiDAR point clouds, triple-attention-based sub-module is utilized while mixed-attention-based sub-module is applied for features of radar points. With local featured attention sub-modules, our FADet has effective detection results in long-tail and complex scenes from camera, LiDAR and radar input. On NuScenes validation dataset, FADet achieves state-of-the-art performance on LiDAR-camera object detection tasks with 71.8% NDS and 69.0% mAP, at the same time, on radar-camera object detection tasks with 51.7% NDS and 40.3% mAP. Code will be released at this https URL.

Title: ColorFoil: Investigating Color Blindness in Large Vision and Language Models

• Authors: Ahnaf Mozib Samin, M. Firoz Ahmed, Md. Mushtaq Shahriyar Rafee
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] ColorFoil: Investigating Color Blindness in Large Vision and Language Models(https://arxiv.org/abs/)
• Keywords: robust, transformer
• Abstract: With the utilization of Transformer architecture, large Vision and Language (V&L) models have shown promising performance in even zero-shot settings. Several studies, however, indicate a lack of robustness of the models when dealing with complex linguistics and visual attributes. In this work, we introduce a novel V&L benchmark - ColorFoil, by creating color-related foils to assess the models' perception ability to detect colors like red, white, green, etc. We evaluate seven state-of-the-art V&L models including CLIP, ViLT, GroupViT, and BridgeTower, etc. in a zero-shot setting and present intriguing findings from the V&L models. The experimental evaluation indicates that ViLT and BridgeTower demonstrate much better color perception capabilities compared to CLIP and its variants and GroupViT. Moreover, CLIP-based models and GroupViT struggle to distinguish colors that are visually distinct to humans with normal color perception ability.

Title: InterAct: Capture and Modelling of Realistic, Expressive and Interactive Activities between Two Persons in Daily Scenarios

• Authors: Yinghao Huang, Leo Ho, Dafei Qin, Mingyi Shi, Taku Komura
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] InterAct: Capture and Modelling of Realistic, Expressive and Interactive Activities between Two Persons in Daily Scenarios(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: We address the problem of accurate capture and expressive modelling of interactive behaviors happening between two persons in daily scenarios. Different from previous works which either only consider one person or focus on conversational gestures, we propose to simultaneously model the activities of two persons, and target objective-driven, dynamic, and coherent interactions which often span long duration. To this end, we capture a new dataset dubbed InterAct, which is composed of 241 motion sequences where two persons perform a realistic scenario over the whole sequence. The audios, body motions, and facial expressions of both persons are all captured in our dataset. We also demonstrate the first diffusion model based approach that directly estimates the interactive motions between two persons from their audios alone. All the data and code will be available for research purposes upon acceptance of the paper.

Title: Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks

• Authors: Taiyuan Mei, Yun Zi, Xiaohan Cheng, Zijun Gao, Qi Wang, Haowei Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: The internal structure and operation mechanism of large-scale language models are analyzed theoretically, especially how Transformer and its derivative architectures can restrict computing efficiency while capturing long-term dependencies. Further, we dig deep into the efficiency bottleneck of the training phase, and evaluate in detail the contribution of adaptive optimization algorithms (such as AdamW), massively parallel computing techniques, and mixed precision training strategies to accelerate convergence and reduce memory footprint. By analyzing the mathematical principles and implementation details of these algorithms, we reveal how they effectively improve training efficiency in practice. In terms of model deployment and inference optimization, this paper systematically reviews the latest advances in model compression techniques, focusing on strategies such as quantification, pruning, and knowledge distillation. By comparing the theoretical frameworks of these techniques and their effects in different application scenarios, we demonstrate their ability to significantly reduce model size and inference delay while maintaining model prediction accuracy. In addition, this paper critically examines the limitations of current efficiency optimization methods, such as the increased risk of overfitting, the control of performance loss after compression, and the problem of algorithm generality, and proposes some prospects for future research. In conclusion, this study provides a comprehensive theoretical framework for understanding the efficiency optimization of large-scale language models.

Title: Adaptive Batch Normalization Networks for Adversarial Robustness

• Authors: Shao-Yuan Lo, Vishal M. Patel
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Adaptive Batch Normalization Networks for Adversarial Robustness(https://arxiv.org/abs/)
• Keywords: defense, attack, robust
• Abstract: Deep networks are vulnerable to adversarial examples. Adversarial Training (AT) has been a standard foundation of modern adversarial defense approaches due to its remarkable effectiveness. However, AT is extremely time-consuming, refraining it from wide deployment in practical applications. In this paper, we aim at a non-AT defense: How to design a defense method that gets rid of AT but is still robust against strong adversarial attacks? To answer this question, we resort to adaptive Batch Normalization (BN), inspired by the recent advances in test-time domain adaptation. We propose a novel defense accordingly, referred to as the Adaptive Batch Normalization Network (ABNN). ABNN employs a pre-trained substitute model to generate clean BN statistics and sends them to the target model. The target model is exclusively trained on clean data and learns to align the substitute model's BN statistics. Experimental results show that ABNN consistently improves adversarial robustness against both digital and physically realizable attacks on both image and video datasets. Furthermore, ABNN can achieve higher clean data performance and significantly lower training time complexity compared to AT-based approaches.

Title: Decentralized Privacy Preservation for Critical Connections in Graphs

• Authors: Conggai Li, Wei Ni, Ming Ding, Youyang Qu, Jianjun Chen, David Smith, Wenjie Zhang, Thierry Rakotoarivelo
• Subjects: cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Decentralized Privacy Preservation for Critical Connections in Graphs(https://arxiv.org/abs/)
• Keywords: privacy, protect
• Abstract: Many real-world interconnections among entities can be characterized as graphs. Collecting local graph information with balanced privacy and data utility has garnered notable interest recently. This paper delves into the problem of identifying and protecting critical information of entity connections for individual participants in a graph based on cohesive subgraph searches. This problem has not been addressed in the literature. To address the problem, we propose to extract the critical connections of a queried vertex using a fortress-like cohesive subgraph model known as $p$-cohesion. A user's connections within a fortress are obfuscated when being released, to protect critical information about the user. Novel merit and penalty score functions are designed to measure each participant's critical connections in the minimal $p$-cohesion, facilitating effective identification of the connections. We further propose to preserve the privacy of a vertex enquired by only protecting its critical connections when responding to queries raised by data collectors. We prove that, under the decentralized differential privacy (DDP) mechanism, one's response satisfies $(\varepsilon, \delta)$-DDP when its critical connections are protected while the rest remains unperturbed. The effectiveness of our proposed method is demonstrated through extensive experiments on real-life graph datasets.

Title: Token-wise Influential Training Data Retrieval for Large Language Models

• Authors: Huawei Lin, Jikai Long, Zhaozhuo Xu, Weijie Zhao
• Subjects: cs.CL, cs.AI, cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Token-wise Influential Training Data Retrieval for Large Language Models(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Given a Large Language Model (LLM) generation, how can we identify which training data led to this generation? In this paper, we proposed RapidIn, a scalable framework adapting to LLMs for estimating the influence of each training data. The proposed framework consists of two stages: caching and retrieval. First, we compress the gradient vectors by over 200,000x, allowing them to be cached on disk or in GPU/CPU memory. Then, given a generation, RapidIn efficiently traverses the cached gradients to estimate the influence within minutes, achieving over a 6,326x speedup. Moreover, RapidIn supports multi-GPU parallelization to substantially accelerate caching and retrieval. Our empirical result confirms the efficiency and effectiveness of RapidIn.

Title: Quality assurance of organs-at-risk delineation in radiotherapy

• Authors: Yihao Zhao, Cuiyun Yuan, Ying Liang, Yang Li, Chunxia Li, Man Zhao, Jun Hu, Wei Liu, Chenbin Liu
• Subjects: cs.CV, physics.med-ph
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Quality assurance of organs-at-risk delineation in radiotherapy(https://arxiv.org/abs/)
• Keywords: segmentation
• Abstract: The delineation of tumor target and organs-at-risk is critical in the radiotherapy treatment planning. Automatic segmentation can be used to reduce the physician workload and improve the consistency. However, the quality assurance of the automatic segmentation is still an unmet need in clinical practice. The patient data used in our study was a standardized dataset from AAPM Thoracic Auto-Segmentation Challenge. The OARs included were left and right lungs, heart, esophagus, and spinal cord. Two groups of OARs were generated, the benchmark dataset manually contoured by experienced physicians and the test dataset automatically created using a software AccuContour. A resnet-152 network was performed as feature extractor, and one-class support vector classifier was used to determine the high or low quality. We evaluate the model performance with balanced accuracy, F-score, sensitivity, specificity and the area under the receiving operator characteristic curve. We randomly generated contour errors to assess the generalization of our method, explored the detection limit, and evaluated the correlations between detection limit and various metrics such as volume, Dice similarity coefficient, Hausdorff distance, and mean surface distance. The proposed one-class classifier outperformed in metrics such as balanced accuracy, AUC, and others. The proposed method showed significant improvement over binary classifiers in handling various types of errors. Our proposed model, which introduces residual network and attention mechanism in the one-class classification framework, was able to detect the various types of OAR contour errors with high accuracy. The proposed method can significantly reduce the burden of physician review for contour delineation.

Title: A General Theory for Compositional Generalization

• Authors: Jingwen Fu, Zhizheng Zhang, Yan Lu, Nanning Zheng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A General Theory for Compositional Generalization(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: Compositional Generalization (CG) embodies the ability to comprehend novel combinations of familiar concepts, representing a significant cognitive leap in human intellectual advancement. Despite its critical importance, the deep neural network (DNN) faces challenges in addressing the compositional generalization problem, prompting considerable research interest. However, existing theories often rely on task-specific assumptions, constraining the comprehensive understanding of CG. This study aims to explore compositional generalization from a task-agnostic perspective, offering a complementary viewpoint to task-specific analyses. The primary challenge is to define CG without overly restricting its scope, a feat achieved by identifying its fundamental characteristics and basing the definition on them. Using this definition, we seek to answer the question "what does the ultimate solution to CG look like?" through the following theoretical findings: 1) the first No Free Lunch theorem in CG, indicating the absence of general solutions; 2) a novel generalization bound applicable to any CG problem, specifying the conditions for an effective CG solution; and 3) the introduction of the generative effect to enhance understanding of CG problems and their solutions. This paper's significance lies in providing a general theory for CG problems, which, when combined with prior theorems under task-specific scenarios, can lead to a comprehensive understanding of CG.

Title: DLAFormer: An End-to-End Transformer For Document Layout Analysis

• Authors: Jiawei Wang, Kai Hu, Qiang Huo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] DLAFormer: An End-to-End Transformer For Document Layout Analysis(https://arxiv.org/abs/)
• Keywords: extraction, transformer
• Abstract: Document layout analysis (DLA) is crucial for understanding the physical layout and logical structure of documents, serving information retrieval, document summarization, knowledge extraction, etc. However, previous studies have typically used separate models to address individual sub-tasks within DLA, including table/figure detection, text region detection, logical role classification, and reading order prediction. In this work, we propose an end-to-end transformer-based approach for document layout analysis, called DLAFormer, which integrates all these sub-tasks into a single model. To achieve this, we treat various DLA sub-tasks (such as text region detection, logical role classification, and reading order prediction) as relation prediction problems and consolidate these relation prediction labels into a unified label space, allowing a unified relation prediction module to handle multiple tasks concurrently. Additionally, we introduce a novel set of type-wise queries to enhance the physical meaning of content queries in DETR. Moreover, we adopt a coarse-to-fine strategy to accurately identify graphical page objects. Experimental results demonstrate that our proposed DLAFormer outperforms previous approaches that employ multi-branch or multi-stage architectures for multiple tasks on two document layout analysis benchmarks, DocLayNet and Comp-HRDoc.

Title: Fed-Credit: Robust Federated Learning with Credibility Management

• Authors: Jiayan Chen, Zhirong Qian, Tianhui Meng, Xitong Gao, Tian Wang, Weijia Jia
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Fed-Credit: Robust Federated Learning with Credibility Management(https://arxiv.org/abs/)
• Keywords: security, privacy, attack, robust, federate
• Abstract: Aiming at privacy preservation, Federated Learning (FL) is an emerging machine learning approach enabling model training on decentralized devices or data sources. The learning mechanism of FL relies on aggregating parameter updates from individual clients. However, this process may pose a potential security risk due to the presence of malicious devices. Existing solutions are either costly due to the use of compute-intensive technology, or restrictive for reasons of strong assumptions such as the prior knowledge of the number of attackers and how they attack. Few methods consider both privacy constraints and uncertain attack scenarios. In this paper, we propose a robust FL approach based on the credibility management scheme, called Fed-Credit. Unlike previous studies, our approach does not require prior knowledge of the nodes and the data distribution. It maintains and employs a credibility set, which weighs the historical clients' contributions based on the similarity between the local models and global model, to adjust the global model update. The subtlety of Fed-Credit is that the time decay and attitudinal value factor are incorporated into the dynamic adjustment of the reputation weights and it boasts a computational complexity of O(n) (n is the number of the clients). We conducted extensive experiments on the MNIST and CIFAR-10 datasets under 5 types of attacks. The results exhibit superior accuracy and resilience against adversarial attacks, all while maintaining comparatively low computational complexity. Among these, on the Non-IID CIFAR-10 dataset, our algorithm exhibited performance enhancements of 19.5% and 14.5%, respectively, in comparison to the state-of-the-art algorithm when dealing with two types of data poisoning attacks.

Title: Uncertainty of interpretability in Landslide Susceptibility Mapping: A Comparative Analysis of Statistical, Machine Learning, and Deep Learning Models

• Authors: Cheng Chen, Lei Fan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Uncertainty of interpretability in Landslide Susceptibility Mapping: A Comparative Analysis of Statistical, Machine Learning, and Deep Learning Models(https://arxiv.org/abs/)
• Keywords: interpretability
• Abstract: Landslide susceptibility mapping (LSM) is crucial for identifying high-risk areas and informing prevention strategies. This study investigates the interpretability of statistical, machine learning (ML), and deep learning (DL) models in predicting landslide susceptibility. This is achieved by incorporating various relevant interpretation methods and two types of input factors: a comprehensive set of 19 contributing factors that are statistically relevant to landslides, as well as a dedicated set of 9 triggering factors directly associated with triggering landslides. Given that model performance is a crucial metric in LSM, our investigations into interpretability naturally involve assessing and comparing LSM accuracy across different models considered. In our investigation, the convolutional neural network model achieved the highest accuracy (0.8447 with 19 factors; 0.8048 with 9 factors), while Extreme Gradient Boosting and Support Vector Machine also demonstrated strong predictive capabilities, outperforming conventional statistical models. These findings indicate that DL and sophisticated ML algorithms can effectively capture the complex relationships between input factors and landslide occurrence. However, the interpretability of predictions varied among different models, particularly when using the broader set of 19 contributing factors. Explanation methods like SHAP, LIME, and DeepLIFT also led to variations in interpretation results. Using a comprehensive set of 19 contributing factors improved prediction accuracy but introduced complexities and inconsistency in model interpretations. Focusing on a dedicated set of 9 triggering factors sacrificed some predictive power but enhanced interpretability, as evidenced by more consistent key factors identified across various models and alignment with the findings of field investigation reports....

Title: DATR: Unsupervised Domain Adaptive Detection Transformer with Dataset-Level Adaptation and Prototypical Alignment

• Authors: Jianhong Han, Liang Chen, Yupei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] DATR: Unsupervised Domain Adaptive Detection Transformer with Dataset-Level Adaptation and Prototypical Alignment(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Object detectors frequently encounter significant performance degradation when confronted with domain gaps between collected data (source domain) and data from real-world applications (target domain). To address this task, numerous unsupervised domain adaptive detectors have been proposed, leveraging carefully designed feature alignment techniques. However, these techniques primarily align instance-level features in a class-agnostic manner, overlooking the differences between extracted features from different categories, which results in only limited improvement. Furthermore, the scope of current alignment modules is often restricted to a limited batch of images, failing to learn the entire dataset-level cues, thereby severely constraining the detector's generalization ability to the target domain. To this end, we introduce a strong DETR-based detector named Domain Adaptive detection TRansformer (DATR) for unsupervised domain adaptation of object detection. Firstly, we propose the Class-wise Prototypes Alignment (CPA) module, which effectively aligns cross-domain features in a class-aware manner by bridging the gap between object detection task and domain adaptation task. Then, the designed Dataset-level Alignment Scheme (DAS) explicitly guides the detector to achieve global representation and enhance inter-class distinguishability of instance-level features across the entire dataset, which spans both domains, by leveraging contrastive learning. Moreover, DATR incorporates a mean-teacher based self-training framework, utilizing pseudo-labels generated by the teacher model to further mitigate domain bias. Extensive experimental results demonstrate superior performance and generalization capabilities of our proposed DATR in multiple domain adaptation scenarios. Code is released at this https URL.

Title: Reward-Punishment Reinforcement Learning with Maximum Entropy

• Authors: Jiexin Wang, Eiji Uchibe
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Reward-Punishment Reinforcement Learning with Maximum Entropy(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: We introduce the ``soft Deep MaxPain'' (softDMP) algorithm, which integrates the optimization of long-term policy entropy into reward-punishment reinforcement learning objectives. Our motivation is to facilitate a smoother variation of operators utilized in the updating of action values beyond traditional ``max'' and ``min'' operators, where the goal is enhancing sample efficiency and robustness. We also address two unresolved issues from the previous Deep MaxPain method. Firstly, we investigate how the negated (``flipped'') pain-seeking sub-policy, derived from the punishment action value, collaborates with the ``min'' operator to effectively learn the punishment module and how softDMP's smooth learning operator provides insights into the ``flipping'' trick. Secondly, we tackle the challenge of data collection for learning the punishment module to mitigate inconsistencies arising from the involvement of the ``flipped'' sub-policy (pain-avoidance sub-policy) in the unified behavior policy. We empirically explore the first issue in two discrete Markov Decision Process (MDP) environments, elucidating the crucial advancements of the DMP approach and the necessity for soft treatments on the hard operators. For the second issue, we propose a probabilistic classifier based on the ratio of the pain-seeking sub-policy to the sum of the pain-seeking and goal-reaching sub-policies. This classifier assigns roll-outs to separate replay buffers for updating reward and punishment action-value functions, respectively. Our framework demonstrates superior performance in Turtlebot 3's maze navigation tasks under the ROS Gazebo simulation.

Title: ViViD: Video Virtual Try-on using Diffusion Models

• Authors: Zixun Fang, Wei Zhai, Aimin Su, Hongliang Song, Kai Zhu, Mao Wang, Yu Chen, Zhiheng Liu, Yang Cao, Zheng-Jun Zha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] ViViD: Video Virtual Try-on using Diffusion Models(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: Video virtual try-on aims to transfer a clothing item onto the video of a target person. Directly applying the technique of image-based try-on to the video domain in a frame-wise manner will cause temporal-inconsistent outcomes while previous video-based try-on solutions can only generate low visual quality and blurring results. In this work, we present ViViD, a novel framework employing powerful diffusion models to tackle the task of video virtual try-on. Specifically, we design the Garment Encoder to extract fine-grained clothing semantic features, guiding the model to capture garment details and inject them into the target video through the proposed attention feature fusion mechanism. To ensure spatial-temporal consistency, we introduce a lightweight Pose Encoder to encode pose signals, enabling the model to learn the interactions between clothing and human posture and insert hierarchical Temporal Modules into the text-to-image stable diffusion model for more coherent and lifelike video synthesis. Furthermore, we collect a new dataset, which is the largest, with the most diverse types of garments and the highest resolution for the task of video virtual try-on to date. Extensive experiments demonstrate that our approach is able to yield satisfactory video try-on results. The dataset, codes, and weights will be publicly available. Project page: this https URL.

Title: (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts

• Authors: Minghao Wu, Yulin Yuan, Gholamreza Haffari, Longyue Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Recent advancements in machine translation (MT) have significantly enhanced translation quality across various domains. However, the translation of literary texts remains a formidable challenge due to their complex language, figurative expressions, and cultural nuances. In this work, we introduce a novel multi-agent framework based on large language models (LLMs) for literary translation, implemented as a company called TransAgents, which mirrors traditional translation publication process by leveraging the collective capabilities of multiple agents, to address the intricate demands of translating literary works. To evaluate the effectiveness of our system, we propose two innovative evaluation strategies: Monolingual Human Preference (MHP) and Bilingual LLM Preference (BLP). MHP assesses translations from the perspective of monolingual readers of the target language, while BLP uses advanced LLMs to compare translations directly with the original texts. Empirical findings indicate that despite lower d-BLEU scores, translations from TransAgents are preferred by both human evaluators and LLMs over human-written references, particularly in genres requiring domain-specific knowledge. We also highlight the strengths and limitations of TransAgents through case studies and suggests directions for future research.

Title: FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learning

• Authors: Liuzhi Zhou, Yu He, Kun Zhai, Xiang Liu, Sen Liu, Xingjun Ma, Guangnan Ye, Yu-Gang Jiang, Hongfeng Chai
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learning(https://arxiv.org/abs/)
• Keywords: privacy, robust, federate
• Abstract: Federated learning (FL) has emerged as a prominent approach for collaborative training of machine learning models across distributed clients while preserving data privacy. However, the quest to balance acceleration and stability becomes a significant challenge in FL, especially on the client-side. In this paper, we introduce FedCAda, an innovative federated client adaptive algorithm designed to tackle this challenge. FedCAda leverages the Adam algorithm to adjust the correction process of the first moment estimate $m$ and the second moment estimate $v$ on the client-side and aggregate adaptive algorithm parameters on the server-side, aiming to accelerate convergence speed and communication efficiency while ensuring stability and performance. Additionally, we investigate several algorithms incorporating different adjustment functions. This comparative analysis revealed that due to the limited information contained within client models from other clients during the initial stages of federated learning, more substantial constraints need to be imposed on the parameters of the adaptive algorithm. As federated learning progresses and clients gather more global information, FedCAda gradually diminishes the impact on adaptive parameters. These findings provide insights for enhancing the robustness and efficiency of algorithmic improvements. Through extensive experiments on computer vision (CV) and natural language processing (NLP) datasets, we demonstrate that FedCAda outperforms the state-of-the-art methods in terms of adaptability, convergence, stability, and overall performance. This work contributes to adaptive algorithms for federated learning, encouraging further exploration.

Title: Climatic & Anthropogenic Hazards to the Nasca World Heritage: Application of Remote Sensing, AI, and Flood Modelling

• Authors: Masato Sakai, Marcus Freitag, Akihisa Sakurai, Conrad M Albrecht, Hendrik F Hamann
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Climatic & Anthropogenic Hazards to the Nasca World Heritage: Application of Remote Sensing, AI, and Flood Modelling(https://arxiv.org/abs/)
• Keywords: protect
• Abstract: Preservation of the Nasca geoglyphs at the UNESCO World Heritage Site in Peru is urgent as natural and human impact accelerates. More frequent weather extremes such as flashfloods threaten Nasca artifacts. We demonstrate that runoff models based on (sub-)meter scale, LiDAR-derived digital elevation data can highlight AI-detected geoglyphs that are in danger of erosion. We recommend measures of mitigation to protect the famous "lizard", "tree", and "hand" geoglyphs located close by, or even cut by the Pan-American Highway.

Title: Federated Learning with Incomplete Sensing Modalities

• Authors: Adiba Orzikulova, Jaehyun Kwak, Jaemin Shin, Sung-Ju Lee
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Federated Learning with Incomplete Sensing Modalities(https://arxiv.org/abs/)
• Keywords: privacy, robust, federate
• Abstract: Many mobile sensing applications utilize data from various modalities, including motion and physiological sensors in mobile and wearable devices. Federated Learning (FL) is particularly suitable for these applications thanks to its privacy-preserving feature. However, challenges such as limited battery life, poor network conditions, and sensor malfunctions can restrict the use of all available modalities for local model training. Additionally, existing multimodal FL systems also struggle with scalability and efficiency as the number of modality sources increases. To address these issues, we introduce FLISM, a framework designed to enable multimodal FL with incomplete modalities. FLISM leverages simulation technique to learn robust representations that can handle missing modalities and transfers model knowledge across clients with varying set of modalities. The evaluation results using three real-world datasets and simulations demonstrate FLISM's effective balance between model performance and system efficiency. It shows an average improvement of .067 in F1-score, while also reducing communication (2.69x faster) and computational (2.28x more efficient) overheads compared to existing methods addressing incomplete modalities. Moreover, in simulated scenarios involving tasks with a larger number of modalities, FLISM achieves a significant speedup of 3.23x~85.10x in communication and 3.73x~32.29x in computational efficiency.

Title: Adversarially Diversified Rehearsal Memory (ADRM): Mitigating Memory Overfitting Challenge in Continual Learning

• Authors: Hikmat Khan, Ghulam Rasool, Nidhal Carla Bouaynaya
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Adversarially Diversified Rehearsal Memory (ADRM): Mitigating Memory Overfitting Challenge in Continual Learning(https://arxiv.org/abs/)
• Keywords: attack, robust
• Abstract: Continual learning focuses on learning non-stationary data distribution without forgetting previous knowledge. Rehearsal-based approaches are commonly used to combat catastrophic forgetting. However, these approaches suffer from a problem called "rehearsal memory overfitting, " where the model becomes too specialized on limited memory samples and loses its ability to generalize effectively. As a result, the effectiveness of the rehearsal memory progressively decays, ultimately resulting in catastrophic forgetting of the learned tasks. We introduce the Adversarially Diversified Rehearsal Memory (ADRM) to address the memory overfitting challenge. This novel method is designed to enrich memory sample diversity and bolster resistance against natural and adversarial noise disruptions. ADRM employs the FGSM attacks to introduce adversarially modified memory samples, achieving two primary objectives: enhancing memory diversity and fostering a robust response to continual feature drifts in memory samples. Our contributions are as follows: Firstly, ADRM addresses overfitting in rehearsal memory by employing FGSM to diversify and increase the complexity of the memory buffer. Secondly, we demonstrate that ADRM mitigates memory overfitting and significantly improves the robustness of CL models, which is crucial for safety-critical applications. Finally, our detailed analysis of features and visualization demonstrates that ADRM mitigates feature drifts in CL memory samples, significantly reducing catastrophic forgetting and resulting in a more resilient CL model. Additionally, our in-depth t-SNE visualizations of feature distribution and the quantification of the feature similarity further enrich our understanding of feature representation in existing CL approaches. Our code is publically available at this https URL.

Title: Improving the Explain-Any-Concept by Introducing Nonlinearity to the Trainable Surrogate Model

• Authors: Mounes Zaval, Sedat Ozer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Improving the Explain-Any-Concept by Introducing Nonlinearity to the Trainable Surrogate Model(https://arxiv.org/abs/)
• Keywords: segmentation
• Abstract: In the evolving field of Explainable AI (XAI), interpreting the decisions of deep neural networks (DNNs) in computer vision tasks is an important process. While pixel-based XAI methods focus on identifying significant pixels, existing concept-based XAI methods use pre-defined or human-annotated concepts. The recently proposed Segment Anything Model (SAM) achieved a significant step forward to prepare automatic concept sets via comprehensive instance segmentation. Building upon this, the Explain Any Concept (EAC) model emerged as a flexible method for explaining DNN decisions. EAC model is based on using a surrogate model which has one trainable linear layer to simulate the target model. In this paper, by introducing an additional nonlinear layer to the original surrogate model, we show that we can improve the performance of the EAC model. We compare our proposed approach to the original EAC model and report improvements obtained on both ImageNet and MS COCO datasets.

Title: EPPS: Advanced Polyp Segmentation via Edge Information Injection and Selective Feature Decoupling

• Authors: Mengqi Lei, Xin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] EPPS: Advanced Polyp Segmentation via Edge Information Injection and Selective Feature Decoupling(https://arxiv.org/abs/)
• Keywords: segmentation
• Abstract: Accurate segmentation of polyps in colonoscopy images is essential for early-stage diagnosis and management of colorectal cancer. Despite advancements in deep learning for polyp segmentation, enduring limitations persist. The edges of polyps are typically ambiguous, making them difficult to discern from the background, and the model performance is often compromised by the influence of irrelevant or unimportant features. To alleviate these challenges, we propose a novel model named Edge-Prioritized Polyp Segmentation (EPPS). Specifically, we incorporate an Edge Mapping Engine (EME) aimed at accurately extracting the edges of polyps. Subsequently, an Edge Information Injector (EII) is devised to augment the mask prediction by injecting the captured edge information into Decoder blocks. Furthermore, we introduce a component called Selective Feature Decoupler (SFD) to suppress the influence of noise and extraneous features on the model. Extensive experiments on 3 widely used polyp segmentation benchmarks demonstrate the superior performance of our method compared with other state-of-the-art approaches.

Title: Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Models

• Authors: Xiyu Wang, Yufei Wang, Satoshi Tsutsui, Weisi Lin, Bihan Wen, Alex C. Kot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Models(https://arxiv.org/abs/)
• Keywords: diffusion, generative
• Abstract: Diffusion-based models for story visualization have shown promise in generating content-coherent images for storytelling tasks. However, how to effectively integrate new characters into existing narratives while maintaining character consistency remains an open problem, particularly with limited data. Two major limitations hinder the progress: (1) the absence of a suitable benchmark due to potential character leakage and inconsistent text labeling, and (2) the challenge of distinguishing between new and old characters, leading to ambiguous results. To address these challenges, we introduce the NewEpisode benchmark, comprising refined datasets designed to evaluate generative models' adaptability in generating new stories with fresh characters using just a single example story. The refined dataset involves refined text prompts and eliminates character leakage. Additionally, to mitigate the character confusion of generated results, we propose EpicEvo, a method that customizes a diffusion-based visual story generation model with a single story featuring the new characters seamlessly integrating them into established character dynamics. EpicEvo introduces a novel adversarial character alignment module to align the generated images progressively in the diffusive process, with exemplar images of new characters, while applying knowledge distillation to prevent forgetting of characters and background details. Our evaluation quantitatively demonstrates that EpicEvo outperforms existing baselines on the NewEpisode benchmark, and qualitative studies confirm its superior customization of visual story generation in diffusion models. In summary, EpicEvo provides an effective way to incorporate new characters using only one example story, unlocking new possibilities for applications such as serialized cartoons.

Title: SEMv3: A Fast and Robust Approach to Table Separation Line Detection

• Authors: Chunxia Qin, Zhenrong Zhang, Pengfei Hu, Chenyu Liu, Jiefeng Ma, Jun Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] SEMv3: A Fast and Robust Approach to Table Separation Line Detection(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Table structure recognition (TSR) aims to parse the inherent structure of a table from its input image. The `"split-and-merge" paradigm is a pivotal approach to parse table structure, where the table separation line detection is crucial. However, challenges such as wireless and deformed tables make it demanding. In this paper, we adhere to the "split-and-merge" paradigm and propose SEMv3 (SEM: Split, Embed and Merge), a method that is both fast and robust for detecting table separation lines. During the split stage, we introduce a Keypoint Offset Regression (KOR) module, which effectively detects table separation lines by directly regressing the offset of each line relative to its keypoint proposals. Moreover, in the merge stage, we define a series of merge actions to efficiently describe the table structure based on table grids. Extensive ablation studies demonstrate that our proposed KOR module can detect table separation lines quickly and accurately. Furthermore, on public datasets (e.g. WTW, ICDAR-2019 cTDaR Historical and iFLYTAB), SEMv3 achieves state-of-the-art (SOTA) performance. The code is available at this https URL.

Title: xFinder: Robust and Pinpoint Answer Extraction for Large Language Models

• Authors: Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li, Feiyu Xiong, Bo Tang, Ding Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] xFinder: Robust and Pinpoint Answer Extraction for Large Language Models(https://arxiv.org/abs/)
• Keywords: robust, extraction, fair, large language model
• Abstract: The continuous advancement of large language models (LLMs) has brought increasing attention to the critical issue of developing fair and reliable methods for evaluating their performance. Particularly, the emergence of subjective or non-subjective cheating phenomena, such as test set leakage and prompt format overfitting, poses significant challenges to the reliable evaluation of LLMs. Since evaluation frameworks often utilize Regular Expression (RegEx) for answer extraction, some models may adjust their responses to comply with specific formats that are easily extractable by RegEx. Nevertheless, the key answer extraction module based on RegEx frequently suffers from extraction errors. This paper conducts a comprehensive analysis of the entire LLM evaluation chain, demonstrating that optimizing the key answer extraction module can improve extraction accuracy, reduce LLMs' reliance on specific answer formats, and enhance the reliability of LLM evaluation. To address these issues, we propose xFinder, a model specifically designed for key answer extraction. As part of this process, we create a specialized dataset, the Key Answer Finder (KAF) dataset, to ensure effective model training and evaluation. Through generalization testing and evaluation in real-world scenarios, the results demonstrate that the smallest xFinder model with only 500 million parameters achieves an average answer extraction accuracy of 93.42%. In contrast, RegEx accuracy in the best evaluation framework is 74.38%. xFinder exhibits stronger robustness and higher accuracy compared to existing evaluation frameworks. All resources for xFinder are available at \url{this https URL}.

Title: A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus

• Authors: Eduard Poesina, Cornelia Caragea, Radu Tudor Ionescu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Natural language inference (NLI), the task of recognizing the entailment relationship in sentence pairs, is an actively studied topic serving as a proxy for natural language understanding. Despite the relevance of the task in building conversational agents and improving text classification, machine translation and other NLP tasks, to the best of our knowledge, there is no publicly available NLI corpus for the Romanian language. To this end, we introduce the first Romanian NLI corpus (RoNLI) comprising 58K training sentence pairs, which are obtained via distant supervision, and 6K validation and test sentence pairs, which are manually annotated with the correct labels. We conduct experiments with multiple machine learning methods based on distant learning, ranging from shallow models based on word embeddings to transformer-based neural networks, to establish a set of competitive baselines. Furthermore, we improve on the best model by employing a new curriculum learning strategy based on data cartography. Our dataset and code to reproduce the baselines are available this https URL.

Title: Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs

• Authors: Siyu Lou, Yuntian Chen, Xiaodan Liang, Liang Lin, Quanshi Zhang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: In this study, we propose an axiomatic system to define and quantify the precise memorization and in-context reasoning effects used by the large language model (LLM) for language generation. These effects are formulated as non-linear interactions between tokens/words encoded by the LLM. Specifically, the axiomatic system enables us to categorize the memorization effects into foundational memorization effects and chaotic memorization effects, and further classify in-context reasoning effects into enhanced inference patterns, eliminated inference patterns, and reversed inference patterns. Besides, the decomposed effects satisfy the sparsity property and the universal matching property, which mathematically guarantee that the LLM's confidence score can be faithfully decomposed into the memorization effects and in-context reasoning effects. Experiments show that the clear disentanglement of memorization effects and in-context reasoning effects enables a straightforward examination of detailed inference patterns encoded by LLMs.

Title: Out-of-Distribution Detection with a Single Unconditional Diffusion Model

• Authors: Alvin Heng, Alexandre H. Thiery, Harold Soh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Out-of-Distribution Detection with a Single Unconditional Diffusion Model(https://arxiv.org/abs/)
• Keywords: diffusion, generative
• Abstract: Out-of-distribution (OOD) detection is a critical task in machine learning that seeks to identify abnormal samples. Traditionally, unsupervised methods utilize a deep generative model for OOD detection. However, such approaches necessitate a different model when evaluating abnormality against a new distribution. With the emergence of foundational generative models, this paper explores whether a single generalist model can also perform OOD detection across diverse tasks. To that end, we introduce our method, Diffusion Paths, (DiffPath) in this work. DiffPath proposes to utilize a single diffusion model originally trained to perform unconditional generation for OOD detection. Specifically, we introduce a novel technique of measuring the rate-of-change and curvature of the diffusion paths connecting samples to the standard normal. Extensive experiments show that with a single model, DiffPath outperforms prior work on a variety of OOD tasks involving different distributions. Our code is publicly available at this https URL.

Title: Vertical Federated Learning Hybrid Local Pre-training

• Authors: Wenguo Li, Xinling Guo, Xu Jiao, Tiancheng Huang, Xiaoran Yan, Yao Yang
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Vertical Federated Learning Hybrid Local Pre-training(https://arxiv.org/abs/)
• Keywords: secure, federate
• Abstract: Vertical Federated Learning (VFL), which has a broad range of real-world applications, has received much attention in both academia and industry. Enterprises aspire to exploit more valuable features of the same users from diverse departments to boost their model prediction skills. VFL addresses this demand and concurrently secures individual parties from exposing their raw data. However, conventional VFL encounters a bottleneck as it only leverages aligned samples, whose size shrinks with more parties involved, resulting in data scarcity and the waste of unaligned data. To address this problem, we propose a novel VFL Hybrid Local Pre-training (VFLHLP) approach. VFLHLP first pre-trains local networks on the local data of participating parties. Then it utilizes these pre-trained networks to adjust the sub-model for the labeled party or enhance representation learning for other parties during downstream federated learning on aligned data, boosting the performance of federated models.

Title: Post-Quantum Security: Origin, Fundamentals, and Adoption

• Authors: Johanna Barzen, Frank Leymann
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Post-Quantum Security: Origin, Fundamentals, and Adoption(https://arxiv.org/abs/)
• Keywords: secure, security, attack
• Abstract: Nowadays, predominant asymmetric cryptographic schemes are considered to be secure because discrete logarithms are believed to be hard to be computed. The algorithm of Shor can effectively compute discrete logarithms, i.e. it can brake such asymmetric schemes. But the algorithm of Shor is a quantum algorithm and at the time this algorithm has been invented, quantum computers that may successfully execute this algorithm seemed to be far out in the future. The latter has changed: quantum computers that are powerful enough are likely to be available in a couple of years. In this article, we first describe the relation between discrete logarithms and two well-known asymmetric security schemes, RSA and Elliptic Curve Cryptography. Next, we present the foundations of lattice-based cryptography which is the bases of schemes that are considered to be safe against attacks by quantum algorithms (as well as by classical algorithms). Then we describe two such quantum-safe algorithms (Kyber and Dilithium) in more detail. Finally, we give a very brief and selective overview of a few actions currently taken by governments and industry as well as standardization in this area. The article especially strives towards being self-contained: the required mathematical foundations to understand post-quantum cryptography are provided and examples are given.

Title: Unveiling and Manipulating Prompt Influence in Large Language Models

• Authors: Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Junlang Qian, Kezhi Mao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Unveiling and Manipulating Prompt Influence in Large Language Models(https://arxiv.org/abs/)
• Keywords: robust, large language model
• Abstract: Prompts play a crucial role in guiding the responses of Large Language Models (LLMs). However, the intricate role of individual tokens in prompts, known as input saliency, in shaping the responses remains largely underexplored. Existing saliency methods either misalign with LLM generation objectives or rely heavily on linearity assumptions, leading to potential inaccuracies. To address this, we propose Token Distribution Dynamics (TDD), a \textcolor{black}{simple yet effective} approach to unveil and manipulate the role of prompts in generating LLM outputs. TDD leverages the robust interpreting capabilities of the language model head (LM head) to assess input saliency. It projects input tokens into the embedding space and then estimates their significance based on distribution dynamics over the vocabulary. We introduce three TDD variants: forward, backward, and bidirectional, each offering unique insights into token relevance. Extensive experiments reveal that the TDD surpasses state-of-the-art baselines with a big margin in elucidating the causal relationships between prompts and LLM outputs. Beyond mere interpretation, we apply TDD to two prompt manipulation tasks for controlled text generation: zero-shot toxic language suppression and sentiment steering. Empirical results underscore TDD's proficiency in identifying both toxic and sentimental cues in prompts, subsequently mitigating toxicity or modulating sentiment in the generated content.

Title: CReMa: Crisis Response through Computational Identification and Matching of Cross-Lingual Requests and Offers Shared on Social Media

• Authors: Rabindra Lamsal, Maria Rodriguez Read, Shanika Karunasekera, Muhammad Imran
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] CReMa: Crisis Response through Computational Identification and Matching of Cross-Lingual Requests and Offers Shared on Social Media(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: During times of crisis, social media platforms play a vital role in facilitating communication and coordinating resources. Amidst chaos and uncertainty, communities often rely on these platforms to share urgent pleas for help, extend support, and organize relief efforts. However, the sheer volume of conversations during such periods, which can escalate to unprecedented levels, necessitates the automated identification and matching of requests and offers to streamline relief operations. This study addresses the challenge of efficiently identifying and matching assistance requests and offers on social media platforms during emergencies. We propose CReMa (Crisis Response Matcher), a systematic approach that integrates textual, temporal, and spatial features for multi-lingual request-offer matching. By leveraging CrisisTransformers, a set of pre-trained models specific to crises, and a cross-lingual embedding space, our methodology enhances the identification and matching tasks while outperforming strong baselines such as RoBERTa, MPNet, and BERTweet, in classification tasks, and Universal Sentence Encoder, Sentence Transformers in crisis embeddings generation tasks. We introduce a novel multi-lingual dataset that simulates scenarios of help-seeking and offering assistance on social media across the 16 most commonly used languages in Australia. We conduct comprehensive cross-lingual experiments across these 16 languages, also while examining trade-offs between multiple vector search strategies and accuracy. Additionally, we analyze a million-scale geotagged global dataset to comprehend patterns in relation to seeking help and offering assistance on social media. Overall, these contributions advance the field of crisis informatics and provide benchmarks for future research in the area.

Title: A comprehensive overview of deep learning techniques for 3D point cloud classification and semantic segmentation

• Authors: Sushmita Sarker, Prithul Sarker, Gunner Stone, Ryan Gorman, Alireza Tavakkoli, George Bebis, Javad Sattarvand
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A comprehensive overview of deep learning techniques for 3D point cloud classification and semantic segmentation(https://arxiv.org/abs/)
• Keywords: segmentation
• Abstract: Point cloud analysis has a wide range of applications in many areas such as computer vision, robotic manipulation, and autonomous driving. While deep learning has achieved remarkable success on image-based tasks, there are many unique challenges faced by deep neural networks in processing massive, unordered, irregular and noisy 3D points. To stimulate future research, this paper analyzes recent progress in deep learning methods employed for point cloud processing and presents challenges and potential directions to advance this field. It serves as a comprehensive review on two major tasks in 3D point cloud processing-- namely, 3D shape classification and semantic segmentation.

Title: A Constraint-Enforcing Reward for Adversarial Attacks on Text Classifiers

• Authors: Tom Roth, Inigo Jauregi Unanue, Alsharif Abuadbba, Massimo Piccardi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A Constraint-Enforcing Reward for Adversarial Attacks on Text Classifiers(https://arxiv.org/abs/)
• Keywords: attack
• Abstract: Text classifiers are vulnerable to adversarial examples -- correctly-classified examples that are deliberately transformed to be misclassified while satisfying acceptability constraints. The conventional approach to finding adversarial examples is to define and solve a combinatorial optimisation problem over a space of allowable transformations. While effective, this approach is slow and limited by the choice of transformations. An alternate approach is to directly generate adversarial examples by fine-tuning a pre-trained language model, as is commonly done for other text-to-text tasks. This approach promises to be much quicker and more expressive, but is relatively unexplored. For this reason, in this work we train an encoder-decoder paraphrase model to generate a diverse range of adversarial examples. For training, we adopt a reinforcement learning algorithm and propose a constraint-enforcing reward that promotes the generation of valid adversarial examples. Experimental results over two text classification datasets show that our model has achieved a higher success rate than the original paraphrase model, and overall has proved more effective than other competitive attacks. Finally, we show how key design choices impact the generated examples and discuss the strengths and weaknesses of the proposed approach.

Title: Diff-BGM: A Diffusion Model for Video Background Music Generation

• Authors: Sizhe Li, Yiming Qin, Minghang Zheng, Xin Jin, Yang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Diff-BGM: A Diffusion Model for Video Background Music Generation(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: When editing a video, a piece of attractive background music is indispensable. However, video background music generation tasks face several challenges, for example, the lack of suitable training datasets, and the difficulties in flexibly controlling the music generation process and sequentially aligning the video and music. In this work, we first propose a high-quality music-video dataset BGM909 with detailed annotation and shot detection to provide multi-modal information about the video and music. We then present evaluation metrics to assess music quality, including music diversity and alignment between music and video with retrieval precision metrics. Finally, we propose the Diff-BGM framework to automatically generate the background music for a given video, which uses different signals to control different aspects of the music during the generation process, i.e., uses dynamic video features to control music rhythm and semantic features to control the melody and atmosphere. We propose to align the video and music sequentially by introducing a segment-aware cross-attention layer. Experiments verify the effectiveness of our proposed method. The code and models are available at this https URL.

Title: PT43D: A Probabilistic Transformer for Generating 3D Shapes from Single Highly-Ambiguous RGB Images

• Authors: Yiheng Xiong, Angela Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] PT43D: A Probabilistic Transformer for Generating 3D Shapes from Single Highly-Ambiguous RGB Images(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Generating 3D shapes from single RGB images is essential in various applications such as robotics. Current approaches typically target images containing clear and complete visual descriptions of the object, without considering common realistic cases where observations of objects that are largely occluded or truncated. We thus propose a transformer-based autoregressive model to generate the probabilistic distribution of 3D shapes conditioned on an RGB image containing potentially highly ambiguous observations of the object. To handle realistic scenarios such as occlusion or field-of-view truncation, we create simulated image-to-shape training pairs that enable improved fine-tuning for real-world scenarios. We then adopt cross-attention to effectively identify the most relevant region of interest from the input image for shape generation. This enables inference of sampled shapes with reasonable diversity and strong alignment with the input image. We train and test our model on our synthetic data then fine-tune and test it on real-world data. Experiments demonstrate that our model outperforms state of the art in both scenarios

Title: Information Leakage from Embedding in Large Language Models

• Authors: Zhipeng Wang, Anda Cheng, Yinggui Wang, Lei Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Information Leakage from Embedding in Large Language Models(https://arxiv.org/abs/)
• Keywords: security, privacy, defense, attack, transformer, large language model
• Abstract: The widespread adoption of large language models (LLMs) has raised concerns regarding data privacy. This study aims to investigate the potential for privacy invasion through input reconstruction attacks, in which a malicious model provider could potentially recover user inputs from embeddings. We first propose two base methods to reconstruct original texts from a model's hidden states. We find that these two methods are effective in attacking the embeddings from shallow layers, but their effectiveness decreases when attacking embeddings from deeper layers. To address this issue, we then present Embed Parrot, a Transformer-based method, to reconstruct input from embeddings in deep layers. Our analysis reveals that Embed Parrot effectively reconstructs original inputs from the hidden states of ChatGLM-6B and Llama2-7B, showcasing stable performance across various token lengths and data distributions. To mitigate the risk of privacy breaches, we introduce a defense mechanism to deter exploitation of the embedding reconstruction process. Our findings emphasize the importance of safeguarding user privacy in distributed learning systems and contribute valuable insights to enhance the security protocols within such environments.

Title: Data Contamination Calibration for Black-box LLMs

• Authors: Wentao Ye, Jiaqi Hu, Liyao Li, Haobo Wang, Gang Chen, Junbo Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Data Contamination Calibration for Black-box LLMs(https://arxiv.org/abs/)
• Keywords: attack, membership infer, large language model
• Abstract: The rapid advancements of Large Language Models (LLMs) tightly associate with the expansion of the training data size. However, the unchecked ultra-large-scale training sets introduce a series of potential risks like data contamination, i.e. the benchmark data is used for training. In this work, we propose a holistic method named Polarized Augment Calibration (PAC) along with a new to-be-released dataset to detect the contaminated data and diminish the contamination effect. PAC extends the popular MIA (Membership Inference Attack) -- from machine learning community -- by forming a more global target at detecting training data to Clarify invisible training data. As a pioneering work, PAC is very much plug-and-play that can be integrated with most (if not all) current white- and black-box LLMs. By extensive experiments, PAC outperforms existing methods by at least 4.5%, towards data contamination detection on more 4 dataset formats, with more than 10 base LLMs. Besides, our application in real-world scenarios highlights the prominent presence of contamination and related issues.

Title: Biomedical Entity Linking for Dutch: Fine-tuning a Self-alignment BERT Model on an Automatically Generated Wikipedia Corpus

• Authors: Fons Hartendorp, Tom Seinen, Erik van Mulligen, Suzan Verberne
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Biomedical Entity Linking for Dutch: Fine-tuning a Self-alignment BERT Model on an Automatically Generated Wikipedia Corpus(https://arxiv.org/abs/)
• Keywords: extraction
• Abstract: Biomedical entity linking, a main component in automatic information extraction from health-related texts, plays a pivotal role in connecting textual entities (such as diseases, drugs and body parts mentioned by patients) to their corresponding concepts in a structured biomedical knowledge base. The task remains challenging despite recent developments in natural language processing. This paper presents the first evaluated biomedical entity linking model for the Dutch language. We use this http URL as base model and perform second-phase pretraining through self-alignment on a Dutch biomedical ontology extracted from the UMLS and Dutch SNOMED. We derive a corpus from Wikipedia of ontology-linked Dutch biomedical entities in context and fine-tune our model on this dataset. We evaluate our model on the Dutch portion of the Mantra GSC-corpus and achieve 54.7% classification accuracy and 69.8% 1-distance accuracy. We then perform a case study on a collection of unlabeled, patient-support forum data and show that our model is hampered by the limited quality of the preceding entity recognition step. Manual evaluation of small sample indicates that of the correctly extracted entities, around 65% is linked to the correct concept in the ontology. Our results indicate that biomedical entity linking in a language other than English remains challenging, but our Dutch model can be used to for high-level analysis of patient-generated text.

Title: WisPerMed at BioLaySumm: Adapting Autoregressive Large Language Models for Lay Summarization of Scientific Articles

• Authors: Tabea M. G. Pakull, Hendrik Damm, Ahmad Idrissi-Yaghir, Henning Schäfer, Peter A. Horn, Christoph M. Friedrich
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] WisPerMed at BioLaySumm: Adapting Autoregressive Large Language Models for Lay Summarization of Scientific Articles(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: This paper details the efforts of the WisPerMed team in the BioLaySumm2024 Shared Task on automatic lay summarization in the biomedical domain, aimed at making scientific publications accessible to non-specialists. Large language models (LLMs), specifically the BioMistral and Llama3 models, were fine-tuned and employed to create lay summaries from complex scientific texts. The summarization performance was enhanced through various approaches, including instruction tuning, few-shot learning, and prompt variations tailored to incorporate specific context information. The experiments demonstrated that fine-tuning generally led to the best performance across most evaluated metrics. Few-shot learning notably improved the models' ability to generate relevant and factually accurate texts, particularly when using a well-crafted prompt. Additionally, a Dynamic Expert Selection (DES) mechanism to optimize the selection of text outputs based on readability and factuality metrics was developed. Out of 54 participants, the WisPerMed team reached the 4th place, measured by readability, factuality, and relevance. Determined by the overall score, our approach improved upon the baseline by approx. 5.5 percentage points and was only approx 1.5 percentage points behind the first place.

Title: Distinguished In Uniform: Self Attention Vs. Virtual Nodes

• Authors: Eran Rosenbluth, Jan Tönshoff, Martin Ritzert, Berke Kisin, Martin Grohe
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Distinguished In Uniform: Self Attention Vs. Virtual Nodes(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Graph Transformers (GTs) such as SAN and GPS are graph processing models that combine Message-Passing GNNs (MPGNNs) with global Self-Attention. They were shown to be universal function approximators, with two reservations: 1. The initial node features must be augmented with certain positional encodings. 2. The approximation is non-uniform: Graphs of different sizes may require a different approximating network. We first clarify that this form of universality is not unique to GTs: Using the same positional encodings, also pure MPGNNs and even 2-layer MLPs are non-uniform universal approximators. We then consider uniform expressivity: The target function is to be approximated by a single network for graphs of all sizes. There, we compare GTs to the more efficient MPGNN + Virtual Node architecture. The essential difference between the two model definitions is in their global computation method -- Self-Attention Vs Virtual Node. We prove that none of the models is a uniform-universal approximator, before proving our main result: Neither model's uniform expressivity subsumes the other's. We demonstrate the theory with experiments on synthetic data. We further augment our study with real-world datasets, observing mixed results which indicate no clear ranking in practice as well.

Title: Exploring Commonalities in Explanation Frameworks: A Multi-Domain Survey Analysis

• Authors: Eduard Barbu, Marharytha Domnich, Raul Vicente, Nikos Sakkas, André Morim
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Exploring Commonalities in Explanation Frameworks: A Multi-Domain Survey Analysis(https://arxiv.org/abs/)
• Keywords: interpretability, explainability
• Abstract: This study presents insights gathered from surveys and discussions with specialists in three domains, aiming to find essential elements for a universal explanation framework that could be applied to these and other similar use cases. The insights are incorporated into a software tool that utilizes GP algorithms, known for their interpretability. The applications analyzed include a medical scenario (involving predictive ML), a retail use case (involving prescriptive ML), and an energy use case (also involving predictive ML). We interviewed professionals from each sector, transcribing their conversations for further analysis. Additionally, experts and non-experts in these fields filled out questionnaires designed to probe various dimensions of explanatory methods. The findings indicate a universal preference for sacrificing a degree of accuracy in favor of greater explainability. Additionally, we highlight the significance of feature importance and counterfactual explanations as critical components of such a framework. Our questionnaires are publicly available to facilitate the dissemination of knowledge in the field of XAI.

Title: Multiple-Choice Questions are Efficient and Robust LLM Evaluators

• Authors: Ziyin Zhang, Lizhen Xu, Zhaokun Jiang, Hongkun Hao, Rui Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Multiple-Choice Questions are Efficient and Robust LLM Evaluators(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: We present GSM-MC and MATH-MC, two multiple-choice (MC) datasets constructed by collecting answers and incorrect predictions on GSM8K and MATH from over 50 open-source models. Through extensive experiments, we show that LLMs' performance on the MC versions of these two popular benchmarks is strongly correlated with their performance on the original versions, and is quite robust to distractor choices and option orders, while the evaluation time is reduced by a factor of up to 30. Following a similar procedure, we also introduce PythonIO, a new program output prediction MC dataset constructed from two other popular LLM evaluation benchmarks HumanEval and MBPP. Our data and code are available at this https URL.

Title: Conditional Shift-Robust Conformal Prediction for Graph Neural Network

• Authors: S. Akansha
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Conditional Shift-Robust Conformal Prediction for Graph Neural Network(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have emerged as potent tools for predicting outcomes in graph-structured data. Despite their efficacy, a significant drawback of GNNs lies in their limited ability to provide robust uncertainty estimates, posing challenges to their reliability in contexts where errors carry significant consequences. Moreover, GNNs typically excel in in-distribution settings, assuming that training and test data follow identical distributions: a condition often unmet in real-world graph data scenarios. In this article, we leverage conformal prediction, a widely recognized statistical technique for quantifying uncertainty by transforming predictive model outputs into prediction sets, to address uncertainty quantification in GNN predictions amidst conditional shift \footnote{Representing the change in conditional probability distribution $P(label |input)$ from source domain to target domain.} in graph-based semi-supervised learning (SSL). Additionally, we propose a novel loss function aimed at refining model predictions by minimizing conditional shift in latent stages. Termed Conditional Shift Robust (CondSR) conformal prediction for GNNs, our approach CondSR is model-agnostic and adaptable to various classification models. We validate the effectiveness of our method on standard graph benchmark datasets, integrating it with state-of-the-art GNNs in node classification tasks. The code implementation is publicly available for further exploration and experimentation.

Title: Data Augmentation for Text-based Person Retrieval Using Large Language Models

• Authors: Zheng Li, Lijia Si, Caili Guo, Yang Yang, Qiushi Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Data Augmentation for Text-based Person Retrieval Using Large Language Models(https://arxiv.org/abs/)
• Keywords: privacy, protect, large language model
• Abstract: Text-based Person Retrieval (TPR) aims to retrieve person images that match the description given a text query. The performance improvement of the TPR model relies on high-quality data for supervised training. However, it is difficult to construct a large-scale, high-quality TPR dataset due to expensive annotation and privacy protection. Recently, Large Language Models (LLMs) have approached or even surpassed human performance on many NLP tasks, creating the possibility to expand high-quality TPR datasets. This paper proposes an LLM-based Data Augmentation (LLM-DA) method for TPR. LLM-DA uses LLMs to rewrite the text in the current TPR dataset, achieving high-quality expansion of the dataset concisely and efficiently. These rewritten texts are able to increase the diversity of vocabulary and sentence structure while retaining the original key concepts and semantic information. In order to alleviate the hallucinations of LLMs, LLM-DA introduces a Text Faithfulness Filter (TFF) to filter out unfaithful rewritten text. To balance the contributions of original text and augmented text, a Balanced Sampling Strategy (BSS) is proposed to control the proportion of original text and augmented text used for training. LLM-DA is a plug-and-play method that can be easily integrated into various TPR models. Comprehensive experiments on three TPR benchmarks show that LLM-DA can improve the retrieval performance of current TPR models.

Title: GuidedRec: Guiding Ill-Posed Unsupervised Volumetric Recovery

• Authors: Alexandre Cafaro, Amaury Leroy, Guillaume Beldjoudi, Pauline Maury, Charlotte Robert, Eric Deutsch, Vincent Grégoire, Vincent Lepetit, Nikos Paragios
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] GuidedRec: Guiding Ill-Posed Unsupervised Volumetric Recovery(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: We introduce a novel unsupervised approach to reconstructing a 3D volume from only two planar projections that exploits a previous\-ly-captured 3D volume of the patient. Such volume is readily available in many important medical procedures and previous methods already used such a volume. Earlier methods that work by deforming this volume to match the projections typically fail when the number of projections is very low as the alignment becomes underconstrained. We show how to use a generative model of the volume structures to constrain the deformation and obtain a correct estimate. Moreover, our method is not bounded to a specific sensor calibration and can be applied to new calibrations without retraining. We evaluate our approach on a challenging dataset and show it outperforms state-of-the-art methods. As a result, our method could be used in treatment scenarios such as surgery and radiotherapy while drastically reducing patient radiation exposure.

Title: Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space

• Authors: Qianmei Liu, Yufei Kuang, Jie Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Deep reinforcement learning (DRL) algorithms can suffer from modeling errors between the simulation and the real world. Many studies use adversarial learning to generate perturbation during training process to model the discrepancy and improve the robustness of DRL. However, most of these approaches use a fixed parameter to control the intensity of the adversarial perturbation, which can lead to a trade-off between average performance and robustness. In fact, finding the optimal parameter of the perturbation is challenging, as excessive perturbations may destabilize training and compromise agent performance, while insufficient perturbations may not impart enough information to enhance robustness. To keep the training stable while improving robustness, we propose a simple but effective method, namely, Adaptive Adversarial Perturbation (A2P), which can dynamically select appropriate adversarial perturbations for each sample. Specifically, we propose an adaptive adversarial coefficient framework to adjust the effect of the adversarial perturbation during training. By designing a metric for the current intensity of the perturbation, our method can calculate the suitable perturbation levels based on the current relative performance. The appealing feature of our method is that it is simple to deploy in real-world applications and does not require accessing the simulator in advance. The experiments in MuJoCo show that our method can improve the training stability and learn a robust policy when migrated to different test environments. The code is available at this https URL.

Title: A review on the use of large language models as virtual tutors

• Authors: Silvia García-Méndez, Francisco de Arriba-Pérez, María del Carmen Somoza-López
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A review on the use of large language models as virtual tutors(https://arxiv.org/abs/)
• Keywords: transformer, generative, large language model
• Abstract: Transformer architectures contribute to managing long-term dependencies for Natural Language Processing, representing one of the most recent changes in the field. These architectures are the basis of the innovative, cutting-edge Large Language Models (LLMs) that have produced a huge buzz in several fields and industrial sectors, among the ones education stands out. Accordingly, these generative Artificial Intelligence-based solutions have directed the change in techniques and the evolution in educational methods and contents, along with network infrastructure, towards high-quality learning. Given the popularity of LLMs, this review seeks to provide a comprehensive overview of those solutions designed specifically to generate and evaluate educational materials and which involve students and teachers in their design or experimental plan. To the best of our knowledge, this is the first review of educational applications (e.g., student assessment) of LLMs. As expected, the most common role of these systems is as virtual tutors for automatic question generation. Moreover, the most popular models are GTP-3 and BERT. However, due to the continuous launch of new generative models, new works are expected to be published shortly.

Title: MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering

• Authors: Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, Yanjie Wang, Yuliang Liu, Hao Liu, Xiang Bai, Can Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Text-Centric Visual Question Answering (TEC-VQA) in its proper format not only facilitates human-machine interaction in text-centric visual environments but also serves as a de facto gold proxy to evaluate AI models in the domain of text-centric scene understanding. However, most TEC-VQA benchmarks have focused on high-resource languages like English and Chinese. Despite pioneering works to expand multilingual QA pairs in non-text-centric VQA datasets using translation engines, the translation-based protocol encounters a substantial ``Visual-textual misalignment'' problem when applied to TEC-VQA. Specifically, it prioritizes the text in question-answer pairs while disregarding the visual text present in images. Furthermore, it does not adequately tackle challenges related to nuanced meaning, contextual distortion, language bias, and question-type diversity. In this work, we address the task of multilingual TEC-VQA and provide a benchmark with high-quality human expert annotations in 9 diverse languages, called MTVQA. To our knowledge, MTVQA is the first multilingual TEC-VQA benchmark to provide human expert annotations for text-centric scenarios. Further, by evaluating several state-of-the-art Multimodal Large Language Models (MLLMs), including GPT-4V, on our MTVQA dataset, it is evident that there is still room for performance improvement, underscoring the value of our dataset. We hope this dataset will provide researchers with fresh perspectives and inspiration within the community. The MTVQA dataset will be available at this https URL.

Title: GGAvatar: Geometric Adjustment of Gaussian Head Avatar

• Authors: Xinyang Li, Jiaxin Wang, Yixin Xuan, Gongxin Yao, Yu Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] GGAvatar: Geometric Adjustment of Gaussian Head Avatar(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: We propose GGAvatar, a novel 3D avatar representation designed to robustly model dynamic head avatars with complex identities and deformations. GGAvatar employs a coarse-to-fine structure, featuring two core modules: Neutral Gaussian Initialization Module and Geometry Morph Adjuster. Neutral Gaussian Initialization Module pairs Gaussian primitives with deformable triangular meshes, employing an adaptive density control strategy to model the geometric structure of the target subject with neutral expressions. Geometry Morph Adjuster introduces deformation bases for each Gaussian in global space, creating fine-grained low-dimensional representations of deformation behaviors to address the Linear Blend Skinning formula's limitations effectively. Extensive experiments show that GGAvatar can produce high-fidelity renderings, outperforming state-of-the-art methods in visual quality and quantitative metrics.

Title: Mamba-in-Mamba: Centralized Mamba-Cross-Scan in Tokenized Mamba Model for Hyperspectral Image Classification

• Authors: Weilian Zhou, Sei-Ichiro Kamata, Haipeng Wang, Man-Sing Wong, Huiying (Cynthia)Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Mamba-in-Mamba: Centralized Mamba-Cross-Scan in Tokenized Mamba Model for Hyperspectral Image Classification(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Hyperspectral image (HSI) classification is pivotal in the remote sensing (RS) field, particularly with the advancement of deep learning techniques. Sequential models, adapted from the natural language processing (NLP) field such as Recurrent Neural Networks (RNNs) and Transformers, have been tailored to this task, offering a unique viewpoint. However, several challenges persist 1) RNNs struggle with centric feature aggregation and are sensitive to interfering pixels, 2) Transformers require significant computational resources and often underperform with limited HSI training samples, and 3) Current scanning methods for converting images into sequence-data are simplistic and inefficient. In response, this study introduces the innovative Mamba-in-Mamba (MiM) architecture for HSI classification, the first attempt of deploying State Space Model (SSM) in this task. The MiM model includes 1) A novel centralized Mamba-Cross-Scan (MCS) mechanism for transforming images into sequence-data, 2) A Tokenized Mamba (T-Mamba) encoder that incorporates a Gaussian Decay Mask (GDM), a Semantic Token Learner (STL), and a Semantic Token Fuser (STF) for enhanced feature generation and concentration, and 3) A Weighted MCS Fusion (WMF) module coupled with a Multi-Scale Loss Design to improve decoding efficiency. Experimental results from three public HSI datasets with fixed and disjoint training-testing samples demonstrate that our method outperforms existing baselines and state-of-the-art approaches, highlighting its efficacy and potential in HSI applications.

Title: Continuous Sign Language Recognition with Adapted Conformer via Unsupervised Pretraining

• Authors: Neena Aloysius, Geetha M, Prema Nedungadi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Continuous Sign Language Recognition with Adapted Conformer via Unsupervised Pretraining(https://arxiv.org/abs/)
• Keywords: extraction, transformer
• Abstract: Conventional Deep Learning frameworks for continuous sign language recognition (CSLR) are comprised of a single or multi-modal feature extractor, a sequence-learning module, and a decoder for outputting the glosses. The sequence learning module is a crucial part wherein transformers have demonstrated their efficacy in the sequence-to-sequence tasks. Analyzing the research progress in the field of Natural Language Processing and Speech Recognition, a rapid introduction of various transformer variants is observed. However, in the realm of sign language, experimentation in the sequence learning component is limited. In this work, the state-of-the-art Conformer model for Speech Recognition is adapted for CSLR and the proposed model is termed ConSignformer. This marks the first instance of employing Conformer for a vision-based task. ConSignformer has bimodal pipeline of CNN as feature extractor and Conformer for sequence learning. For improved context learning we also introduce Cross-Modal Relative Attention (CMRA). By incorporating CMRA into the model, it becomes more adept at learning and utilizing complex relationships within the data. To further enhance the Conformer model, unsupervised pretraining called Regressional Feature Extraction is conducted on a curated sign language dataset. The pretrained Conformer is then fine-tuned for the downstream recognition task. The experimental results confirm the effectiveness of the adopted pretraining strategy and demonstrate how CMRA contributes to the recognition process. Remarkably, leveraging a Conformer-based backbone, our model achieves state-of-the-art performance on the benchmark datasets: PHOENIX-2014 and PHOENIX-2014T.

Title: Can AI Relate: Testing Large Language Model Response for Mental Health Support

• Authors: Saadia Gabriel, Isha Puri, Xuhai Xu, Matteo Malgaroli, Marzyeh Ghassemi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Can AI Relate: Testing Large Language Model Response for Mental Health Support(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Large language models (LLMs) are already being piloted for clinical use in hospital systems like NYU Langone, Dana-Farber and the NHS. A proposed deployment use case is psychotherapy, where a LLM-powered chatbot can treat a patient undergoing a mental health crisis. Deployment of LLMs for mental health response could hypothetically broaden access to psychotherapy and provide new possibilities for personalizing care. However, recent high-profile failures, like damaging dieting advice offered by the Tessa chatbot to patients with eating disorders, have led to doubt about their reliability in high-stakes and safety-critical settings. In this work, we develop an evaluation framework for determining whether LLM response is a viable and ethical path forward for the automation of mental health treatment. Using human evaluation with trained clinicians and automatic quality-of-care metrics grounded in psychology research, we compare the responses provided by peer-to-peer responders to those provided by a state-of-the-art LLM. We show that LLMs like GPT-4 use implicit and explicit cues to infer patient demographics like race. We then show that there are statistically significant discrepancies between patient subgroups: Responses to Black posters consistently have lower empathy than for any other demographic group (2%-13% lower than the control group). Promisingly, we do find that the manner in which responses are generated significantly impacts the quality of the response. We conclude by proposing safety guidelines for the potential deployment of LLMs for mental health response.

Title: Adaptive Extraction Network for Multivariate Long Sequence Time-Series Forecasting

• Authors: Dandan Zhang, Yun Wang
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Adaptive Extraction Network for Multivariate Long Sequence Time-Series Forecasting(https://arxiv.org/abs/)
• Keywords: extraction
• Abstract: Models employing CNN architecture have made significant progress in multivariate long sequence time-series forecasting (MLSTF), particularly in modeling local time series characteristics. However, during the MLSTF process, extracting the global time series patterns and understanding the correlations among different variables are highly significant. To address this challenge, we introduce multi-resolution convolution and deformable convolution operations. By enlarging the receptive field using convolution kernels with different dilation factors to capture temporal correlation information across different resolutions, and adaptively adjusting the sampling positions through additional offset vectors, we enhance the network's ability to capture correlated features between variables. Building upon this, we propose ATVCNet, an adaptive temporal-variable convolutional network designed to effectively model the local/global temporal dependencies and inter-variable dependencies of multivariate time series. Specifically, extracting and fusing time series features at different resolutions, captures both local contextual information and global patterns in the time series. The designed inter-variable feature adaptive extraction module captures the correlation among different variables in the time series. We evaluated the performance of ATVCNet across eight real-world datasets. The results indicate that ATVCNet achieved a performance improvement of approximately 63.4% over state-of-the-art MLSTF models.

Title: Attribute-Based Authentication in Secure Group Messaging for Distributed Environments

• Authors: David Soler (1), Carlos Dafonte (1), Manuel Fernández-Veiga (2), Ana Fernández Vilas (2), Francisco J. Nóvoa (1) ((1) CITIC, Universidade da Coruňa, A Coruňa, Spain, (2) atlanTTic, Universidade de Vigo, Vigo, Spain)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Attribute-Based Authentication in Secure Group Messaging for Distributed Environments(https://arxiv.org/abs/)
• Keywords: secure, security, privacy, attack
• Abstract: Messaging Layer security (MLS) and its underlying Continuous Group Key Agreement (CGKA) protocol allows a group of users to share a cryptographic secret in a dynamic manner, such that the secret is modified in member insertions and deletions. Although this flexibility makes MLS ideal for implementations in distributed environments, a number of issues need to be overcome. Particularly, the use of digital certificates for authentication in a group goes against the group members' privacy. In this work we provide an alternative method of authentication in which the solicitors, instead of revealing their identity, only need to prove possession of certain attributes, dynamically defined by the group, to become a member. Instead of digital certificates, we employ Attribute-Based Credentials accompanied with Selective Disclosure in order to reveal the minimum required amount of information and to prevent attackers from linking the activity of a user through multiple groups. We formally define a CGKA variant named Attribute-Authenticated Continuous Group Key Agreement (AA-CGKA) and provide security proofs for its properties of Requirement Integrity, Unforgeability and Unlinkability. We also provide guidelines for an integration of our construction in MLS.

Title: The Incoherency Risk in the EU's New Cyber Security Policies

• Authors: Jukka Ruohonen
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] The Incoherency Risk in the EU's New Cyber Security Policies(https://arxiv.org/abs/)
• Keywords: security
• Abstract: The European Union (EU) has been pursuing new cyber security policies in recent years. This paper presents a short evaluation of four such policies. The focus is on potential incoherency, meaning a lack of integration, divergence between the member states, institutional dysfunction, and other related problems that should be at least partially avoidable by sound policy-making. According to the evaluation, the four policies have substantially increased the complexity of the EU's cyber security framework. In addition, there are potential problems with trust, divergence between industry sectors and different technologies, bureaucratic conflicts, and technical issues, among other things. With these insights, the paper not only contributes to the study of EU policies but also advances the understanding of cyber security policies in general.

Title: Energy-Efficient Federated Edge Learning with Streaming Data: A Lyapunov Optimization Approach

• Authors: Chung-Hsuan Hu, Zheng Chen, Erik G. Larsson
• Subjects: cs.LG, cs.DC, cs.IT, eess.SP
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Energy-Efficient Federated Edge Learning with Streaming Data: A Lyapunov Optimization Approach(https://arxiv.org/abs/)
• Keywords: federate
• Abstract: Federated learning (FL) has received significant attention in recent years for its advantages in efficient training of machine learning models across distributed clients without disclosing user-sensitive data. Specifically, in federated edge learning (FEEL) systems, the time-varying nature of wireless channels introduces inevitable system dynamics in the communication process, thereby affecting training latency and energy consumption. In this work, we further consider a streaming data scenario where new training data samples are randomly generated over time at edge devices. Our goal is to develop a dynamic scheduling and resource allocation algorithm to address the inherent randomness in data arrivals and resource availability under long-term energy constraints. To achieve this, we formulate a stochastic network optimization problem and use the Lyapunov drift-plus-penalty framework to obtain a dynamic resource management design. Our proposed algorithm makes adaptive decisions on device scheduling, computational capacity adjustment, and allocation of bandwidth and transmit power in every round. We provide convergence analysis for the considered setting with heterogeneous data and time-varying objective functions, which supports the rationale behind our proposed scheduling design. The effectiveness of our scheme is verified through simulation results, demonstrating improved learning performance and energy efficiency as compared to baseline schemes.

Title: NPLMV-PS: Neural Point-Light Multi-View Photometric Stereo

• Authors: Fotios Logothetis, Ignas Budvytis, Roberto Cipolla
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] NPLMV-PS: Neural Point-Light Multi-View Photometric Stereo(https://arxiv.org/abs/)
• Keywords: robust, segmentation
• Abstract: In this work we present a novel multi-view photometric stereo (PS) method. Like many works in 3D reconstruction we are leveraging neural shape representations and learnt renderers. However, our work differs from the state-of-the-art multi-view PS methods such as PS-NeRF or SuperNormal we explicity leverage per-pixel intensity renderings rather than relying mainly on estimated normals. We model point light attenuation and explicitly raytrace cast shadows in order to best approximate each points incoming radiance. This is used as input to a fully neural material renderer that uses minimal prior assumptions and it is jointly optimised with the surface. Finally, estimated normal and segmentation maps can also incorporated in order to maximise the surface accuracy. Our method is among the first to outperform the classical approach of DiLiGenT-MV and achieves average 0.2mm Chamfer distance for objects imaged at approx 1.5m distance away with approximate 400x400 resolution. Moreover, we show robustness to poor normals in low light count scenario, achieving 0.27mm Chamfer distance when pixel rendering is used instead of estimated normals.

Title: STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents

• Authors: Yue Chen, Chen Huang, Yang Deng, Wenqiang Lei, Dingnan Jin, Jia Liu, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Equipping a conversational search engine with strategies regarding when to ask clarification questions is becoming increasingly important across various domains. Attributing to the context understanding capability of LLMs and their access to domain-specific sources of knowledge, LLM-based clarification strategies feature rapid transfer to various domains in a post-hoc manner. However, they still struggle to deliver promising performance on unseen domains, struggling to achieve effective domain transferability. We take the first step to investigate this issue and existing methods tend to produce one-size-fits-all strategies across diverse domains, limiting their search effectiveness. In response, we introduce a novel method, called Style, to achieve effective domain transferability. Our experimental results indicate that Style bears strong domain transferability, resulting in an average search performance improvement of ~10% on four unseen domains.

Title: CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models

• Authors: Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Large language models (LLMs) are increasingly used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown, ultimately risking user trust and satisfaction. To this end, we introduce CLAMBER, a benchmark for evaluating LLMs using a well-organized taxonomy. Building upon the taxonomy, we construct ~12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries, even enhanced by chain-of-thought (CoT) and few-shot prompting. These techniques may result in overconfidence in LLMs and yield only marginal enhancements in identifying ambiguity. Furthermore, current LLMs fall short in generating high-quality clarifying questions due to a lack of conflict resolution and inaccurate utilization of inherent knowledge. In this paper, CLAMBER presents a guidance and promotes further research on proactive and trustworthy LLMs. Our dataset is available at this https URL

Title: Gaussian Head & Shoulders: High Fidelity Neural Upper Body Avatars with Anchor Gaussian Guided Texture Warping

• Authors: Tianhao Wu, Jing Yang, Zhilin Guo, Jingyi Wan, Fangcheng Zhong, Cengiz Oztireli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Gaussian Head & Shoulders: High Fidelity Neural Upper Body Avatars with Anchor Gaussian Guided Texture Warping(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: By equipping the most recent 3D Gaussian Splatting representation with head 3D morphable models (3DMM), existing methods manage to create head avatars with high fidelity. However, most existing methods only reconstruct a head without the body, substantially limiting their application scenarios. We found that naively applying Gaussians to model the clothed chest and shoulders tends to result in blurry reconstruction and noisy floaters under novel poses. This is because of the fundamental limitation of Gaussians and point clouds -- each Gaussian or point can only have a single directional radiance without spatial variance, therefore an unnecessarily large number of them is required to represent complicated spatially varying texture, even for simple geometry. In contrast, we propose to model the body part with a neural texture that consists of coarse and pose-dependent fine colors. To properly render the body texture for each view and pose without accurate geometry nor UV mapping, we optimize another sparse set of Gaussians as anchors that constrain the neural warping field that maps image plane coordinates to the texture space. We demonstrate that Gaussian Head & Shoulders can fit the high-frequency details on the clothed upper body with high fidelity and potentially improve the accuracy and fidelity of the head region. We evaluate our method with casual phone-captured and internet videos and show our method archives superior reconstruction quality and robustness in both self and cross reenactment tasks. To fully utilize the efficient rendering speed of Gaussian splatting, we additionally propose an accelerated inference method of our trained model without Multi-Layer Perceptron (MLP) queries and reach a stable rendering speed of around 130 FPS for any subjects.

Title: GAN-GRID: A Novel Generative Attack on Smart Grid Stability Prediction

• Authors: Emad Efatinasab, Alessandro Brighente, Mirco Rampazzo, Nahal Azadi, Mauro Conti
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] GAN-GRID: A Novel Generative Attack on Smart Grid Stability Prediction(https://arxiv.org/abs/)
• Keywords: security, protect, attack, generative
• Abstract: The smart grid represents a pivotal innovation in modernizing the electricity sector, offering an intelligent, digitalized energy network capable of optimizing energy delivery from source to consumer. It hence represents the backbone of the energy sector of a nation. Due to its central role, the availability of the smart grid is paramount and is hence necessary to have in-depth control of its operations and safety. To this aim, researchers developed multiple solutions to assess the smart grid's stability and guarantee that it operates in a safe state. Artificial intelligence and Machine learning algorithms have proven to be effective measures to accurately predict the smart grid's stability. Despite the presence of known adversarial attacks and potential solutions, currently, there exists no standardized measure to protect smart grids against this threat, leaving them open to new adversarial attacks. In this paper, we propose GAN-GRID a novel adversarial attack targeting the stability prediction system of a smart grid tailored to real-world constraints. Our findings reveal that an adversary armed solely with the stability model's output, devoid of data or model knowledge, can craft data classified as stable with an Attack Success Rate (ASR) of 0.99. Also by manipulating authentic data and sensor values, the attacker can amplify grid issues, potentially undetected due to a compromised stability prediction system. These results underscore the imperative of fortifying smart grid security mechanisms against adversarial manipulation to uphold system stability and reliability.

Title: Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?

• Authors: Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, Li Shen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Transformer-based trajectory optimization methods have demonstrated exceptional performance in offline Reinforcement Learning (offline RL), yet it poses challenges due to substantial parameter size and limited scalability, which is particularly critical in sequential decision-making scenarios where resources are constrained such as in robots and drones with limited computational power. Mamba, a promising new linear-time sequence model, offers performance on par with transformers while delivering substantially fewer parameters on long sequences. As it remains unclear whether Mamba is compatible with trajectory optimization, this work aims to conduct comprehensive experiments to explore the potential of Decision Mamba in offline RL (dubbed DeMa) from the aspect of data structures and network architectures with the following insights: (1) Long sequences impose a significant computational burden without contributing to performance improvements due to the fact that DeMa's focus on sequences diminishes approximately exponentially. Consequently, we introduce a Transformer-like DeMa as opposed to an RNN-like DeMa. (2) For the components of DeMa, we identify that the hidden attention mechanism is key to its success, which can also work well with other residual structures and does not require position embedding. Extensive evaluations from eight Atari games demonstrate that our specially designed DeMa is compatible with trajectory optimization and surpasses previous state-of-the-art methods, outdoing Decision Transformer (DT) by 80\% with 30\% fewer parameters, and exceeds DT in MuJoCo with only a quarter of the parameters.

Title: PATE: Proximity-Aware Time series anomaly Evaluation

• Authors: Ramin Ghorbani, Marcel J.T. Reinders, David M.J. Tax
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] PATE: Proximity-Aware Time series anomaly Evaluation(https://arxiv.org/abs/)
• Keywords: fair
• Abstract: Evaluating anomaly detection algorithms in time series data is critical as inaccuracies can lead to flawed decision-making in various domains where real-time analytics and data-driven strategies are essential. Traditional performance metrics assume iid data and fail to capture the complex temporal dynamics and specific characteristics of time series anomalies, such as early and delayed detections. We introduce Proximity-Aware Time series anomaly Evaluation (PATE), a novel evaluation metric that incorporates the temporal relationship between prediction and anomaly intervals. PATE uses proximity-based weighting considering buffer zones around anomaly intervals, enabling a more detailed and informed assessment of a detection. Using these weights, PATE computes a weighted version of the area under the Precision and Recall curve. Our experiments with synthetic and real-world datasets show the superiority of PATE in providing more sensible and accurate evaluations than other evaluation metrics. We also tested several state-of-the-art anomaly detectors across various benchmark datasets using the PATE evaluation scheme. The results show that a common metric like Point-Adjusted F1 Score fails to characterize the detection performances well, and that PATE is able to provide a more fair model comparison. By introducing PATE, we redefine the understanding of model efficacy that steers future studies toward developing more effective and accurate detection models.

Title: DOP: Diagnostic-Oriented Prompting for Large Language Models in Mathematical Correction

• Authors: Hao Chen, Biaojie Zeng, Xin Lin, Liang He, Aimin Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] DOP: Diagnostic-Oriented Prompting for Large Language Models in Mathematical Correction(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Math world problems correction(MWPC) is a novel task dedicated to rectifying reasoning errors in the process of solving mathematical problems. In this paper, leveraging the advancements in large language models (LLMs), we address two key objectives:(1) Distinguishing between mathematical reasoning and error correction; (2) Exploring strategies to enhance the error correction capabilities of LLMs in mathematics to solve MWPC task. We noticed that, in real-time education,assisting students in recognizing their mistakes is more crucial than simply providing correct answers. However, current research tends to prioritize obtaining accurate solutions to math problems rather than correcting potentially incorrect ones. Therefore, we modify the research paradigm, demonstrating that improving mathematical reasoning abilities does not equate to mastery in error correction. Meanwhile, we propose a novel method called diagnostic-oriented promping(DOP) aimed at facilitating LLMs to excel in error correction. In experiments, DOP has shown outstanding performance, highlighting its significant impact. We argue that in mathematical education, the demand for outstanding correctors surpasses that for proficient reasoners. Codes and data are available on this https URL.

Title: Deciding branching hyperproperties for real time systems

• Authors: Nabarun Deka, Minjian Zhang, Rohit Chadha, Mahesh Viswanathan
• Subjects: cs.CR, cs.LO
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Deciding branching hyperproperties for real time systems(https://arxiv.org/abs/)
• Keywords: security, attack
• Abstract: Security properties of real-time systems often involve reasoning about hyper-properties, as opposed to properties of single executions or trees of executions. These hyper-properties need to additionally be expressive enough to reason about real-time constraints. Examples of such properties include information flow, side channel attacks and service-level agreements. In this paper we study computational problems related to a branching-time, hyper-property extension of metric temporal logic (MTL) that we call HCMTL*. We consider both the interval-based and point-based semantics of this logic. The verification problem that we consider is to determine if a given HCMTL* formula $\varphi$ is true in a system represented by a timed automaton. We show that this problem is undecidable. We then show that the verification problem is decidable if we consider executions upto a fixed time horizon $T$. Our decidability result relies on reducing the verification problem to the truth of an MSO formula over reals with a bounded time interval.

Title: Sheet Music Transformer ++: End-to-End Full-Page Optical Music Recognition for Pianoform Sheet Music

• Authors: Antonio Ríos-Vila, Jorge Calvo-Zaragoza, David Rizo, Thierry Paquet
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Sheet Music Transformer ++: End-to-End Full-Page Optical Music Recognition for Pianoform Sheet Music(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: Optical Music Recognition is a field that has progressed significantly, bringing accurate systems that transcribe effectively music scores into digital formats. Despite this, there are still several limitations that hinder OMR from achieving its full potential. Specifically, state of the art OMR still depends on multi-stage pipelines for performing full-page transcription, as well as it has only been demonstrated in monophonic cases, leaving behind very relevant engravings. In this work, we present the Sheet Music Transformer++, an end-to-end model that is able to transcribe full-page polyphonic music scores without the need of a previous Layout Analysis step. This is done thanks to an extensive curriculum learning-based pretraining with synthetic data generation. We conduct several experiments on a full-page extension of a public polyphonic transcription dataset. The experimental outcomes confirm that the model is competent at transcribing full-page pianoform scores, marking a noteworthy milestone in end-to-end OMR transcription.

Title: Imp: Highly Capable Large Multimodal Models for Mobile Devices

• Authors: Zhenwei Shao, Zhou Yu, Jun Yu, Xuecheng Ouyang, Lihao Zheng, Zhenbiao Gai, Mingyang Wang, Jiajun Ding
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Imp: Highly Capable Large Multimodal Models for Mobile Devices(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: By harnessing the capabilities of large language models (LLMs), recent large multimodal models (LMMs) have shown remarkable versatility in open-world multimodal understanding. Nevertheless, they are usually parameter-heavy and computation-intensive, thus hindering their applicability in resource-constrained scenarios. To this end, several lightweight LMMs have been proposed successively to maximize the capabilities under constrained scale (e.g., 3B). Despite the encouraging results achieved by these methods, most of them only focus on one or two aspects of the design space, and the key design choices that influence model capability have not yet been thoroughly investigated. In this paper, we conduct a systematic study for lightweight LMMs from the aspects of model architecture, training strategy, and training data. Based on our findings, we obtain Imp -- a family of highly capable LMMs at the 2B-4B scales. Notably, our Imp-3B model steadily outperforms all the existing lightweight LMMs of similar size, and even surpasses the state-of-the-art LMMs at the 13B scale. With low-bit quantization and resolution reduction techniques, our Imp model can be deployed on a Qualcomm Snapdragon 8Gen3 mobile chip with a high inference speed of about 13 tokens/s.

Title: MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

• Authors: Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters. To achieve it, we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA. We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.

Title: Bangladeshi Native Vehicle Detection in Wild

• Authors: Bipin Saha, Md. Johirul Islam, Shaikh Khaled Mostaque, Aditya Bhowmik, Tapodhir Karmakar Taton, Md. Nakib Hayat Chowdhury, Mamun Bin Ibne Reaz
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Bangladeshi Native Vehicle Detection in Wild(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: The success of autonomous navigation relies on robust and precise vehicle recognition, hindered by the scarcity of region-specific vehicle detection datasets, impeding the development of context-aware systems. To advance terrestrial object detection research, this paper proposes a native vehicle detection dataset for the most commonly appeared vehicle classes in Bangladesh. 17 distinct vehicle classes have been taken into account, with fully annotated 81542 instances of 17326 images. Each image width is set to at least 1280px. The dataset's average vehicle bounding box-to-image ratio is 4.7036. This Bangladesh Native Vehicle Dataset (BNVD) has accounted for several geographical, illumination, variety of vehicle sizes, and orientations to be more robust on surprised scenarios. In the context of examining the BNVD dataset, this work provides a thorough assessment with four successive You Only Look Once (YOLO) models, namely YOLO v5, v6, v7, and v8. These dataset's effectiveness is methodically evaluated and contrasted with other vehicle datasets already in use. The BNVD dataset exhibits mean average precision(mAP) at 50% intersection over union (IoU) is 0.848 corresponding precision and recall values of 0.841 and 0.774. The research findings indicate a mAP of 0.643 at an IoU range of 0.5 to 0.95. The experiments show that the BNVD dataset serves as a reliable representation of vehicle distribution and presents considerable complexities.

Title: Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging

• Authors: Xiaobo Liang, Haoke Zhang, Helan hu, Juntao Li, Jun Xu, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: The rapid advancement of large language models has given rise to a plethora of applications across a myriad of real-world tasks, mainly centered on aligning with human intent. However, the complexities inherent in human intent necessitate a dependence on labor-intensive and time-consuming human evaluation. To alleviate this constraint, we delve into the paradigm of employing open-source large language models as evaluators, aligning with the prevailing trend of utilizing GPT-4. Particularly, we present a step-by-step evaluation framework: \textbf{Fennec}, capable of \textbf{F}ine-grained \textbf{E}valuatio\textbf{N} and correctio\textbf{N} \textbf{E}xtended through bran\textbf{C}hing and bridging. Specifically, the branching operation dissects the evaluation task into various dimensions and granularities, thereby alleviating the challenges associated with evaluation. Concurrently, the bridging operation amalgamates diverse training datasets, augmenting the variety of evaluation tasks. In experimental trials, our 7B model consistently outperforms open-source larger-scale evaluation models across various widely adopted benchmarks in terms of both \textit{Agreement} and \textit{Consistency}, closely approaching the capabilities of GPT-4. We employ the fine-grained correction capabilities induced by the evaluation model to refine multiple model responses, and the results show that the refinement elevates the quality of responses, leading to an improvement of 1-2 points on the MT-Bench. Our code is available at Github\footnote{\url{this https URL}}.

Title: CT-Eval: Benchmarking Chinese Text-to-Table Performance in Large Language Models

• Authors: Haoxiang Shi, Jiaan Wang, Jiarong Xu, Cen Wang, Tetsuya Sakai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] CT-Eval: Benchmarking Chinese Text-to-Table Performance in Large Language Models(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Text-to-Table aims to generate structured tables to convey the key information from unstructured documents. Existing text-to-table datasets are typically oriented English, limiting the research in non-English languages. Meanwhile, the emergence of large language models (LLMs) has shown great success as general task solvers in multi-lingual settings (e.g., ChatGPT), theoretically enabling text-to-table in other languages. In this paper, we propose a Chinese text-to-table dataset, CT-Eval, to benchmark LLMs on this task. Our preliminary analysis of English text-to-table datasets highlights two key factors for dataset construction: data diversity and data hallucination. Inspired by this, the CT-Eval dataset selects a popular Chinese multidisciplinary online encyclopedia as the source and covers 28 domains to ensure data diversity. To minimize data hallucination, we first train an LLM to judge and filter out the task samples with hallucination, then employ human annotators to clean the hallucinations in the validation and testing sets. After this process, CT-Eval contains 88.6K task samples. Using CT-Eval, we evaluate the performance of open-source and closed-source LLMs. Our results reveal that zero-shot LLMs (including GPT-4) still have a significant performance gap compared with human judgment. Furthermore, after fine-tuning, open-source LLMs can significantly improve their text-to-table ability, outperforming GPT-4 by a large margin. In short, CT-Eval not only helps researchers evaluate and quickly understand the Chinese text-to-table ability of existing LLMs but also serves as a valuable resource to significantly improve the text-to-table performance of LLMs.

Title: Enhancing Explainable AI: A Hybrid Approach Combining GradCAM and LRP for CNN Interpretability

• Authors: Vaibhav Dhore, Achintya Bhat, Viraj Nerlekar, Kashyap Chavhan, Aniket Umare
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Enhancing Explainable AI: A Hybrid Approach Combining GradCAM and LRP for CNN Interpretability(https://arxiv.org/abs/)
• Keywords: robust, interpretability
• Abstract: We present a new technique that explains the output of a CNN-based model using a combination of GradCAM and LRP methods. Both of these methods produce visual explanations by highlighting input regions that are important for predictions. In the new method, the explanation produced by GradCAM is first processed to remove noises. The processed output is then multiplied elementwise with the output of LRP. Finally, a Gaussian blur is applied on the product. We compared the proposed method with GradCAM and LRP on the metrics of Faithfulness, Robustness, Complexity, Localisation and Randomisation. It was observed that this method performs better on Complexity than both GradCAM and LRP and is better than atleast one of them in the other metrics.

Title: Establishing Trust in the Beyond-5G Core Network using Trusted Execution Environments

• Authors: Marinos Vomvas, Norbert Ludant, Guevara Noubir
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Establishing Trust in the Beyond-5G Core Network using Trusted Execution Environments(https://arxiv.org/abs/)
• Keywords: security, attack
• Abstract: The fifth generation (5G) of cellular networks starts a paradigm shift from the traditional monolithic system design to a Service Based Architecture, that fits modern performance requirements and scales efficiently to new services. This paradigm will be the foundation of future cellular core networks beyond 5G. The new architecture splits network functionalities into smaller logical entities that can be disaggregated logically, physically, and geographically. This affords interoperability between the mobile network operators and commercial software and hardware vendors or cloud providers. By making use of commodity services and products, this system construct inherits the vulnerabilities in those underlying technologies, thereby increasing its attack surface and requiring a rigorous security analysis. In this work, we review the security implications introduced in B5G networks, and the security mechanisms that are supported by the 5G standard. We emphasize on the support of Zero Trust Architecture in 5G and its relevance in decentralized deployments. We revisit the definition of trust in modern enterprise network operations and identify important Zero Trust properties that are weakened by the nature of cloud deployments. To that end, we propose a vertical extension of Zero Trust, namely, Zero Trust Execution, to model untrusted execution environments, and we provide an analysis on how to establish trust in Beyond-5G network architectures using Trusted Execution Environments. Our analysis shows how our model architecture handles the increased attack surface and reinforces the Zero Trust Architecture principles in the 5G Core, without any changes to the 5G standard. Finally, we provide experimental results over a 5G testbed using Open5GS and UERANSIM that demonstrate minimal performance overhead, and a monetary cost evaluation.

Title: Brewer-Nash Scrutinised: Mechanised Checking of Policies featuring Write Revocation

• Authors: Alfredo Capozucca, Maximiliano Cristiá, Ross Horne, Ricardo Katz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Brewer-Nash Scrutinised: Mechanised Checking of Policies featuring Write Revocation(https://arxiv.org/abs/)
• Keywords: security
• Abstract: This paper revisits the Brewer-Nash security policy model inspired by ethical Chinese Wall policies. We draw attention to the fact that write access can be revoked in the Brewer-Nash model. The semantics of write access were underspecified originally, leading to multiple interpretations for which we provide a modern operational semantics. We go on to modernise the analysis of information flow in the Brewer-Nash model, by adopting a more precise definition adapted from Kessler. For our modernised reformulation, we provide full mechanised coverage for all theorems proposed by Brewer & Nash. Most theorems are established automatically using the tool {log} with the exception of a theorem regarding information flow, which combines a lemma in {log} with a theorem mechanised in Coq. Having covered all theorems originally posed by Brewer-Nash, achieving modern precision and mechanisation, we propose this work as a step towards a methodology for automated checking of more complex security policy models.

Title: Automated Hardware Logic Obfuscation Framework Using GPT

• Authors: Banafsheh Saber Latibari, Sujan Ghimire, Muhtasim Alam Chowdhury, Najmeh Nazari, Kevin Immanuel Gubbi, Houman Homayoun, Avesta Sasan, Soheil Salehi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Automated Hardware Logic Obfuscation Framework Using GPT(https://arxiv.org/abs/)
• Keywords: security, attack, transformer, generative
• Abstract: Obfuscation stands as a promising solution for safeguarding hardware intellectual property (IP) against a spectrum of threats including reverse engineering, IP piracy, and tampering. In this paper, we introduce Obfus-chat, a novel framework leveraging Generative Pre-trained Transformer (GPT) models to automate the obfuscation process. The proposed framework accepts hardware design netlists and key sizes as inputs, and autonomously generates obfuscated code tailored to enhance security. To evaluate the effectiveness of our approach, we employ the Trust-Hub Obfuscation Benchmark for comparative analysis. We employed SAT attacks to assess the security of the design, along with functional verification procedures to ensure that the obfuscated design remains consistent with the original. Our results demonstrate the efficacy and efficiency of the proposed framework in fortifying hardware IP against potential threats, thus providing a valuable contribution to the field of hardware security.

Title: Hierarchical Neural Operator Transformer with Learnable Frequency-aware Loss Prior for Arbitrary-scale Super-resolution

• Authors: Xihaier Luo, Xiaoning Qian, Byung-Jun Yoon
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Hierarchical Neural Operator Transformer with Learnable Frequency-aware Loss Prior for Arbitrary-scale Super-resolution(https://arxiv.org/abs/)
• Keywords: transformer
• Abstract: In this work, we present an arbitrary-scale super-resolution (SR) method to enhance the resolution of scientific data, which often involves complex challenges such as continuity, multi-scale physics, and the intricacies of high-frequency signals. Grounded in operator learning, the proposed method is resolution-invariant. The core of our model is a hierarchical neural operator that leverages a Galerkin-type self-attention mechanism, enabling efficient learning of mappings between function spaces. Sinc filters are used to facilitate the information transfer across different levels in the hierarchy, thereby ensuring representation equivalence in the proposed neural operator. Additionally, we introduce a learnable prior structure that is derived from the spectral resizing of the input data. This loss prior is model-agnostic and is designed to dynamically adjust the weighting of pixel contributions, thereby balancing gradients effectively across the model. We conduct extensive experiments on diverse datasets from different domains and demonstrate consistent improvements compared to strong baselines, which consist of various state-of-the-art SR methods.

Title: Modeling citation worthiness by using attention-based bidirectional long short-term memory networks and interpretable models

• Authors: Tong Zeng, Daniel E. Acuna
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Modeling citation worthiness by using attention-based bidirectional long short-term memory networks and interpretable models(https://arxiv.org/abs/)
• Keywords: robust
• Abstract: Scientist learn early on how to cite scientific sources to support their claims. Sometimes, however, scientists have challenges determining where a citation should be situated -- or, even worse, fail to cite a source altogether. Automatically detecting sentences that need a citation (i.e., citation worthiness) could solve both of these issues, leading to more robust and well-constructed scientific arguments. Previous researchers have applied machine learning to this task but have used small datasets and models that do not take advantage of recent algorithmic developments such as attention mechanisms in deep learning. We hypothesize that we can develop significantly accurate deep learning architectures that learn from large supervised datasets constructed from open access publications. In this work, we propose a Bidirectional Long Short-Term Memory (BiLSTM) network with attention mechanism and contextual information to detect sentences that need citations. We also produce a new, large dataset (PMOA-CITE) based on PubMed Open Access Subset, which is orders of magnitude larger than previous datasets. Our experiments show that our architecture achieves state of the art performance on the standard ACL-ARC dataset ($F_{1}=0.507$) and exhibits high performance ($F_{1}=0.856$) on the new PMOA-CITE. Moreover, we show that it can transfer learning across these datasets. We further use interpretable models to illuminate how specific language is used to promote and inhibit citations. We discover that sections and surrounding sentences are crucial for our improved predictions. We further examined purported mispredictions of the model, and uncovered systematic human mistakes in citation behavior and source data. This opens the door for our model to check documents during pre-submission and pre-archival procedures. We make this new dataset, the code, and a web-based tool available to the community.

Title: MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

• Authors: Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark(https://arxiv.org/abs/)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have showcased significant improvements in mathematics. However, traditional math benchmarks like GSM8k offer a unidimensional perspective, falling short in providing a holistic assessment of the LLMs' math capabilities. To address this gap, we introduce MathBench, a new benchmark that rigorously assesses the mathematical capabilities of large language models. MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. The benchmark progresses through five distinct stages, from basic arithmetic to college mathematics, and is structured to evaluate models at various depths of knowledge. Each stage includes theoretical questions and application problems, allowing us to measure a model's mathematical proficiency and its ability to apply concepts in practical scenarios. MathBench aims to enhance the evaluation of LLMs' mathematical abilities, providing a nuanced view of their knowledge understanding levels and problem solving skills in a bilingual context. The project is released at this https URL .

Title: Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices

• Authors: Nathaniel Cohen, Vladimir Kulikov, Matan Kleiner, Inbar Huberman-Spiegelglas, Tomer Michaeli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) diffusion models achieve state-of-the-art results in image synthesis and editing. However, leveraging such pretrained models for video editing is considered a major challenge. Many existing works attempt to enforce temporal consistency in the edited video through explicit correspondence mechanisms, either in pixel space or between deep features. These methods, however, struggle with strong nonrigid motion. In this paper, we introduce a fundamentally different approach, which is based on the observation that spatiotemporal slices of natural videos exhibit similar characteristics to natural images. Thus, the same T2I diffusion model that is normally used only as a prior on video frames, can also serve as a strong prior for enhancing temporal consistency by applying it on spatiotemporal slices. Based on this observation, we present Slicedit, a method for text-based video editing that utilizes a pretrained T2I diffusion model to process both spatial and spatiotemporal slices. Our method generates videos that retain the structure and motion of the original video while adhering to the target text. Through extensive experiments, we demonstrate Slicedit's ability to edit a wide range of real-world videos, confirming its clear advantages compared to existing competing methods. Webpage: this https URL

Title: Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learning

• Authors: Guanglin Zhou, Zhongyi Han, Shiming Chen, Biwei Huang, Liming Zhu, Salman Khan, Xin Gao, Lina Yao
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learning(https://arxiv.org/abs/)
• Keywords: robust, large language model
• Abstract: Recent studies indicate that large multimodal models (LMMs) are highly robust against natural distribution shifts, often surpassing previous baselines. Despite this, domain-specific adaptation is still necessary, particularly in specialized areas like healthcare. Due to the impracticality of fine-tuning LMMs given their vast parameter space, this work investigates in-context learning (ICL) as an effective alternative for enhancing LMMs' adaptability. We find that the success of ICL heavily relies on the choice of demonstration, mirroring challenges seen in large language models but introducing unique complexities for LMMs facing distribution shifts. Our study addresses this by evaluating an unsupervised ICL method, TopKNearestPR, which selects in-context examples through a nearest example search based on feature similarity. We uncover that its effectiveness is limited by the deficiencies of pre-trained vision encoders under distribution shift scenarios. To address these challenges, we propose InvariantSelectPR, a novel method leveraging Class-conditioned Contrastive Invariance (CCI) for more robust demonstration selection. Specifically, CCI enhances pre-trained vision encoders by improving their discriminative capabilities across different classes and ensuring invariance to domain-specific variations. This enhancement allows the encoders to effectively identify and retrieve the most informative examples, which are then used to guide LMMs in adapting to new query samples under varying distributions. Our experiments show that InvariantSelectPR substantially improves the adaptability of LMMs, achieving significant performance gains on benchmark datasets, with a 34.2%$\uparrow$ accuracy increase in 7-shot on Camelyon17 and 16.9%$\uparrow$ increase in 7-shot on HAM10000 compared to the baseline zero-shot performance.

Title: Images that Sound: Composing Images and Sounds on a Single Canvas

• Authors: Ziyang Chen, Daniel Geng, Andrew Owens
• Subjects: cs.CV, cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Images that Sound: Composing Images and Sounds on a Single Canvas(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: Spectrograms are 2D representations of sound that look very different from the images found in our visual world. And natural images, when played as spectrograms, make unnatural sounds. In this paper, we show that it is possible to synthesize spectrograms that simultaneously look like natural images and sound like natural audio. We call these spectrograms images that sound. Our approach is simple and zero-shot, and it leverages pre-trained text-to-image and text-to-spectrogram diffusion models that operate in a shared latent space. During the reverse process, we denoise noisy latents with both the audio and image diffusion models in parallel, resulting in a sample that is likely under both models. Through quantitative evaluations and perceptual studies, we find that our method successfully generates spectrograms that align with a desired audio prompt while also taking the visual appearance of a desired image prompt. Please see our project page for video results: this https URL