2024-05-01

Title: Dynamic Model Switching for Improved Accuracy in Machine Learning

• Authors: Syed Tahir Abbas Hasani
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18932
• Pdf URL: https://arxiv.org/pdf/2404.18932
• Copy Paste: [[2404.18932]] Dynamic Model Switching for Improved Accuracy in Machine Learning(https://arxiv.org/abs/2404.18932)
• Keywords: robust
• Abstract: In the dynamic landscape of machine learning, where datasets vary widely in size and complexity, selecting the most effective model poses a significant challenge. Rather than fixating on a single model, our research propels the field forward with a novel emphasis on dynamic model switching. This paradigm shift allows us to harness the inherent strengths of different models based on the evolving size of the dataset. Consider the scenario where CatBoost demonstrates exceptional efficacy in handling smaller datasets, providing nuanced insights and accurate predictions. However, as datasets grow in size and intricacy, XGBoost, with its scalability and robustness, becomes the preferred choice. Our approach introduces an adaptive ensemble that intuitively transitions between CatBoost and XGBoost. This seamless switching is not arbitrary; instead, it's guided by a user-defined accuracy threshold, ensuring a meticulous balance between model sophistication and data requirements. The user sets a benchmark, say 80% accuracy, prompting the system to dynamically shift to the new model only if it guarantees improved performance. This dynamic model-switching mechanism aligns with the evolving nature of data in real-world scenarios. It offers practitioners a flexible and efficient solution, catering to diverse dataset sizes and optimising predictive accuracy at every juncture. Our research, therefore, stands at the forefront of innovation, redefining how machine learning models adapt and excel in the face of varying dataset dynamics.

Title: Learning Low-Rank Feature for Thorax Disease Classification

• Authors: Rajeev Goel, Utkarsh Nath, Yancheng Wang, Alvin C. Silva, Teresa Wu, Yingzhen Yang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18933
• Pdf URL: https://arxiv.org/pdf/2404.18933
• Copy Paste: [[2404.18933]] Learning Low-Rank Feature for Thorax Disease Classification(https://arxiv.org/abs/2404.18933)
• Keywords: extraction, transformer
• Abstract: Deep neural networks, including Convolutional Neural Networks (CNNs) and Visual Transformers (ViT), have achieved stunning success in medical image domain. We study thorax disease classification in this paper. Effective extraction of features for the disease areas is crucial for disease classification on radiographic images. While various neural architectures and training techniques, such as self-supervised learning with contrastive/restorative learning, have been employed for disease classification on radiographic images, there are no principled methods which can effectively reduce the adverse effect of noise and background, or non-disease areas, on the radiographic images for disease classification. To address this challenge, we propose a novel Low-Rank Feature Learning (LRFL) method in this paper, which is universally applicable to the training of all neural networks. The LRFL method is both empirically motivated by the low frequency property observed on all the medical datasets in this paper, and theoretically motivated by our sharp generalization bound for neural networks with low-rank features. In the empirical study, using a neural network such as a ViT or a CNN pre-trained on unlabeled chest X-rays by Masked Autoencoders (MAE), our novel LRFL method is applied on the pre-trained neural network and demonstrate better classification results in terms of both multiclass area under the receiver operating curve (mAUC) and classification accuracy.

Title: The Visual Experience Dataset: Over 200 Recorded Hours of Integrated Eye Movement, Odometry, and Egocentric Video

• Authors: Michelle R. Greene, Benjamin J. Balas, Mark D. Lescroart, Paul R. MacNeilage, Jennifer A. Hart, Kamran Binaee, Peter A. Hausamann, Ronald Mezile, Bharath Shankar, Christian B. Sinnott, Kaylie Capurro, Savannah Halow, Hunter Howe, Mariam Josyula, Annie Li, Abraham Mieses, Amina Mohamed, Ilya Nudnou, Ezra Parkhill, Peter Riley, Brett Schmidt, Matthew W. Shinkle, Wentao Si, Brian Szekely, Joaquin M. Torres, Eliana Weissmann
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.18934
• Pdf URL: https://arxiv.org/pdf/2404.18934
• Copy Paste: [[2404.18934]] The Visual Experience Dataset: Over 200 Recorded Hours of Integrated Eye Movement, Odometry, and Egocentric Video(https://arxiv.org/abs/2404.18934)
• Keywords: privacy
• Abstract: We introduce the Visual Experience Dataset (VEDB), a compilation of over 240 hours of egocentric video combined with gaze- and head-tracking data that offers an unprecedented view of the visual world as experienced by human observers. The dataset consists of 717 sessions, recorded by 58 observers ranging from 6-49 years old. This paper outlines the data collection, processing, and labeling protocols undertaken to ensure a representative sample and discusses the potential sources of error or bias within the dataset. The VEDB's potential applications are vast, including improving gaze tracking methodologies, assessing spatiotemporal image statistics, and refining deep neural networks for scene and activity recognition. The VEDB is accessible through established open science platforms and is intended to be a living dataset with plans for expansion and community contributions. It is released with an emphasis on ethical considerations, such as participant privacy and the mitigation of potential biases. By providing a dataset grounded in real-world experiences and accompanied by extensive metadata and supporting code, the authors invite the research community to utilize and contribute to the VEDB, facilitating a richer understanding of visual perception and behavior in naturalistic settings.

Title: Sub-Adjacent Transformer: Improving Time Series Anomaly Detection with Reconstruction Error from Sub-Adjacent Neighborhoods

• Authors: Wenzhen Yue, Xianghua Ying, Ruohao Guo, DongDong Chen, Ji Shi, Bowei Xing, Yuqing Zhu, Taiyan Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18948
• Pdf URL: https://arxiv.org/pdf/2404.18948
• Copy Paste: [[2404.18948]] Sub-Adjacent Transformer: Improving Time Series Anomaly Detection with Reconstruction Error from Sub-Adjacent Neighborhoods(https://arxiv.org/abs/2404.18948)
• Keywords: transformer
• Abstract: In this paper, we present the Sub-Adjacent Transformer with a novel attention mechanism for unsupervised time series anomaly detection. Unlike previous approaches that rely on all the points within some neighborhood for time point reconstruction, our method restricts the attention to regions not immediately adjacent to the target points, termed sub-adjacent neighborhoods. Our key observation is that owing to the rarity of anomalies, they typically exhibit more pronounced differences from their sub-adjacent neighborhoods than from their immediate vicinities. By focusing the attention on the sub-adjacent areas, we make the reconstruction of anomalies more challenging, thereby enhancing their detectability. Technically, our approach concentrates attention on the non-diagonal areas of the attention matrix by enlarging the corresponding elements in the training stage. To facilitate the implementation of the desired attention matrix pattern, we adopt linear attention because of its flexibility and adaptability. Moreover, a learnable mapping function is proposed to improve the performance of linear attention. Empirically, the Sub-Adjacent Transformer achieves state-of-the-art performance across six real-world anomaly detection benchmarks, covering diverse fields such as server monitoring, space exploration, and water treatment.

Title: An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

• Authors: Yuan Wang, Huazhu Fu, Renuga Kanagavelu, Qingsong Wei, Yong Liu, Rick Siow Mong Goh
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.18962
• Pdf URL: https://arxiv.org/pdf/2404.18962
• Copy Paste: [[2404.18962]] An Aggregation-Free Federated Learning for Tackling Data Heterogeneity(https://arxiv.org/abs/2404.18962)
• Keywords: federate
• Abstract: The performance of Federated Learning (FL) hinges on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. This process can cause client drift, especially with significant cross-client data heterogeneity, impacting model performance and convergence of the FL algorithm. To address these challenges, we introduce FedAF, a novel aggregation-free FL algorithm. In this framework, clients collaboratively learn condensed data by leveraging peer knowledge, the server subsequently trains the global model using the condensed data and soft labels received from the clients. FedAF inherently avoids the issue of client drift, enhances the quality of condensed data amid notable data heterogeneity, and improves the global model performance. Extensive numerical studies on several popular benchmark datasets show FedAF surpasses various state-of-the-art FL algorithms in handling label-skew and feature-skew data heterogeneity, leading to superior global model accuracy and faster convergence.

Title: Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking

• Authors: Zacharias Chrysidis, Stefanos-Iordanis Papadopoulos, Symeon Papadopoulos, Panagiotis C. Petrantonakis
• Subjects: cs.CL, cs.CY, cs.IR, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.18971
• Pdf URL: https://arxiv.org/pdf/2404.18971
• Copy Paste: [[2404.18971]] Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking(https://arxiv.org/abs/2404.18971)
• Keywords: robust
• Abstract: Automated fact-checking (AFC) is garnering increasing attention by researchers aiming to help fact-checkers combat the increasing spread of misinformation online. While many existing AFC methods incorporate external information from the Web to help examine the veracity of claims, they often overlook the importance of verifying the source and quality of collected "evidence". One overlooked challenge involves the reliance on "leaked evidence", information gathered directly from fact-checking websites and used to train AFC systems, resulting in an unrealistic setting for early misinformation detection. Similarly, the inclusion of information from unreliable sources can undermine the effectiveness of AFC systems. To address these challenges, we present a comprehensive approach to evidence verification and filtering. We create the "CREDible, Unreliable or LEaked" (CREDULE) dataset, which consists of 91,632 articles classified as Credible, Unreliable and Fact checked (Leaked). Additionally, we introduce the EVidence VERification Network (EVVER-Net), trained on CREDULE to detect leaked and unreliable evidence in both short and long texts. EVVER-Net can be used to filter evidence collected from the Web, thus enhancing the robustness of end-to-end AFC systems. We experiment with various language models and show that EVVER-Net can demonstrate impressive performance of up to 91.5% and 94.4% accuracy, while leveraging domain credibility scores along with short or long texts, respectively. Finally, we assess the evidence provided by widely-used fact-checking datasets including LIAR-PLUS, MOCHEG, FACTIFY, NewsCLIPpings+ and VERITE, some of which exhibit concerning rates of leaked and unreliable evidence.

Title: M3H: Multimodal Multitask Machine Learning for Healthcare

• Authors: Dimitris Bertsimas, Yu Ma
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.18975
• Pdf URL: https://arxiv.org/pdf/2404.18975
• Copy Paste: [[2404.18975]] M3H: Multimodal Multitask Machine Learning for Healthcare(https://arxiv.org/abs/2404.18975)
• Keywords: robust, explainability
• Abstract: Recent breakthroughs in AI are poised to fundamentally enhance our study and understanding of healthcare. The development of an integrated many-to-many framework that leverages multiple data modality inputs for the analytical modeling of multiple medical tasks, is critical for a unified understanding of modern medicine. In this work, we introduce M3H, an explainable Multimodal Multitask Machine Learning for Healthcare framework that consolidates learning from diverse multimodal inputs across a broad spectrum of medical task categories and machine learning problem classes. The modular design of the framework ensures its generalizable data processing, task definition, and rapid model prototyping, applicable to both clinical and operational healthcare settings. We evaluate the M3H framework by validating models trained from four modalities (tabular, time-series, language, and vision) on 41 medical tasks across 4 machine learning problem classes. Our results demonstrate that M3H consistently produces multitask models that outperform canonical single-task models (by 1.1- 37.2%) across 37 disease diagnoses from 16 medical departments, three hospital operation forecasts, and one patient phenotyping task: spanning ML problem classes of supervised binary classification, multiclass classification, regression, and clustering. Additionally, the framework introduces a novel attention mechanism to balance self-exploitation (focus on learning source task), and cross-exploration (encourage learning from other tasks). Furthermore, M3H provides explainability insights on how joint learning of additional tasks impacts the learning of source task using a proposed TIM score, shedding light into the dynamics of task interdependencies. Its adaptable architecture facilitates the customization and integration, establishing it as a robust and scalable candidate solution for future AI-driven healthcare systems.

Title: Foundations of Multisensory Artificial Intelligence

• Authors: Paul Pu Liang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.18976
• Pdf URL: https://arxiv.org/pdf/2404.18976
• Copy Paste: [[2404.18976]] Foundations of Multisensory Artificial Intelligence(https://arxiv.org/abs/2404.18976)
• Keywords: transformer, large language model
• Abstract: Building multisensory AI systems that learn from multiple sensory inputs such as text, speech, video, real-world sensors, wearable devices, and medical data holds great promise for impact in many scientific areas with practical benefits, such as in supporting human health and well-being, enabling multimedia content processing, and enhancing real-world autonomous agents. By synthesizing a range of theoretical frameworks and application domains, this thesis aims to advance the machine learning foundations of multisensory AI. In the first part, we present a theoretical framework formalizing how modalities interact with each other to give rise to new information for a task. These interactions are the basic building blocks in all multimodal problems, and their quantification enables users to understand their multimodal datasets, design principled approaches to learn these interactions, and analyze whether their model has succeeded in learning. In the second part, we study the design of practical multimodal foundation models that generalize over many modalities and tasks, which presents a step toward grounding large language models to real-world sensory modalities. We introduce MultiBench, a unified large-scale benchmark across a wide range of modalities, tasks, and research areas, followed by the cross-modal attention and multimodal transformer architectures that now underpin many of today's multimodal foundation models. Scaling these architectures on MultiBench enables the creation of general-purpose multisensory AI systems, and we discuss our collaborative efforts in applying these models for real-world impact in affective computing, mental health, cancer prognosis, and robotics. Finally, we conclude this thesis by discussing how future work can leverage these ideas toward more general, interactive, and safe multisensory AI.

Title: Computational Job Market Analysis with Natural Language Processing

• Authors: Mike Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.18977
• Pdf URL: https://arxiv.org/pdf/2404.18977
• Copy Paste: [[2404.18977]] Computational Job Market Analysis with Natural Language Processing(https://arxiv.org/abs/2404.18977)
• Keywords: extraction
• Abstract: [Abridged Abstract] Recent technological advances underscore labor market dynamics, yielding significant consequences for employment prospects and increasing job vacancy data across platforms and languages. Aggregating such data holds potential for valuable insights into labor market demands, new skills emergence, and facilitating job matching for various stakeholders. However, despite prevalent insights in the private sector, transparent language technology systems and data for this domain are lacking. This thesis investigates Natural Language Processing (NLP) technology for extracting relevant information from job descriptions, identifying challenges including scarcity of training data, lack of standardized annotation guidelines, and shortage of effective extraction methods from job ads. We frame the problem, obtaining annotated data, and introducing extraction methodologies. Our contributions include job description datasets, a de-identification dataset, and a novel active learning algorithm for efficient model training. We propose skill extraction using weak supervision, a taxonomy-aware pre-training methodology adapting multilingual language models to the job market domain, and a retrieval-augmented model leveraging multiple skill extraction datasets to enhance overall performance. Finally, we ground extracted information within a designated taxonomy.

Title: Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs

• Authors: Bahar Radmehr, Adish Singla, Tanja Käser
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.18978
• Pdf URL: https://arxiv.org/pdf/2404.18978
• Copy Paste: [[2404.18978]] Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs(https://arxiv.org/abs/2404.18978)
• Keywords: large language model
• Abstract: There has been a growing interest in developing learner models to enhance learning and teaching experiences in educational environments. However, existing works have primarily focused on structured environments relying on meticulously crafted representations of tasks, thereby limiting the agent's ability to generalize skills across tasks. In this paper, we aim to enhance the generalization capabilities of agents in open-ended text-based learning environments by integrating Reinforcement Learning (RL) with Large Language Models (LLMs). We investigate three types of agents: (i) RL-based agents that utilize natural language for state and action representations to find the best interaction strategy, (ii) LLM-based agents that leverage the model's general knowledge and reasoning through prompting, and (iii) hybrid LLM-assisted RL agents that combine these two strategies to improve agents' performance and generalization. To support the development and evaluation of these agents, we introduce PharmaSimText, a novel benchmark derived from the PharmaSim virtual pharmacy environment designed for practicing diagnostic conversations. Our results show that RL-based agents excel in task completion but lack in asking quality diagnostic questions. In contrast, LLM-based agents perform better in asking diagnostic questions but fall short of completing the task. Finally, hybrid LLM-assisted RL agents enable us to overcome these limitations, highlighting the potential of combining RL and LLMs to develop high-performing agents for open-ended learning environments.

Title: Machine Unlearning for Document Classification

• Authors: Lei Kang, Mohamed Ali Souibgui, Fei Yang, Lluis Gomez, Ernest Valveny, Dimosthenis Karatzas
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19031
• Pdf URL: https://arxiv.org/pdf/2404.19031
• Copy Paste: [[2404.19031]] Machine Unlearning for Document Classification(https://arxiv.org/abs/2404.19031)
• Keywords: privacy
• Abstract: Document understanding models have recently demonstrated remarkable performance by leveraging extensive collections of user documents. However, since documents often contain large amounts of personal data, their usage can pose a threat to user privacy and weaken the bonds of trust between humans and AI services. In response to these concerns, legislation advocating ``the right to be forgotten" has recently been proposed, allowing users to request the removal of private information from computer systems and neural network models. A novel approach, known as machine unlearning, has emerged to make AI models forget about a particular class of data. In our research, we explore machine unlearning for document classification problems, representing, to the best of our knowledge, the first investigation into this area. Specifically, we consider a realistic scenario where a remote server houses a well-trained model and possesses only a small portion of training data. This setup is designed for efficient forgetting manipulation. This work represents a pioneering step towards the development of machine unlearning methods aimed at addressing privacy concerns in document analysis applications. Our code is publicly available at \url{https://github.com/leitro/MachineUnlearning-DocClassification}.

Title: Embedded Representation Learning Network for Animating Styled Video Portrait

• Authors: Tianyong Wang, Xiangyu Liang, Wangguandong Zheng, Dan Niu, Haifeng Xia, Siyu Xia
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19038
• Pdf URL: https://arxiv.org/pdf/2404.19038
• Copy Paste: [[2404.19038]] Embedded Representation Learning Network for Animating Styled Video Portrait(https://arxiv.org/abs/2404.19038)
• Keywords: generative
• Abstract: The talking head generation recently attracted considerable attention due to its widespread application prospects, especially for digital avatars and 3D animation design. Inspired by this practical demand, several works explored Neural Radiance Fields (NeRF) to synthesize the talking heads. However, these methods based on NeRF face two challenges: (1) Difficulty in generating style-controllable talking heads. (2) Displacement artifacts around the neck in rendered images. To overcome these two challenges, we propose a novel generative paradigm \textit{Embedded Representation Learning Network} (ERLNet) with two learning stages. First, the \textit{ audio-driven FLAME} (ADF) module is constructed to produce facial expression and head pose sequences synchronized with content audio and style video. Second, given the sequence deduced by the ADF, one novel \textit{dual-branch fusion NeRF} (DBF-NeRF) explores these contents to render the final images. Extensive empirical studies demonstrate that the collaboration of these two stages effectively facilitates our method to render a more realistic talking head than the existing algorithms.

Title: Improving Interpretability of Deep Active Learning for Flood Inundation Mapping Through Class Ambiguity Indices Using Multi-spectral Satellite Imagery

• Authors: Hyunho Lee, Wenwen Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19043
• Pdf URL: https://arxiv.org/pdf/2404.19043
• Copy Paste: [[2404.19043]] Improving Interpretability of Deep Active Learning for Flood Inundation Mapping Through Class Ambiguity Indices Using Multi-spectral Satellite Imagery(https://arxiv.org/abs/2404.19043)
• Keywords: interpretability
• Abstract: Flood inundation mapping is a critical task for responding to the increasing risk of flooding linked to global warming. Significant advancements of deep learning in recent years have triggered its extensive applications, including flood inundation mapping. To cope with the time-consuming and labor-intensive data labeling process in supervised learning, deep active learning strategies are one of the feasible approaches. However, there remains limited exploration into the interpretability of how deep active learning strategies operate, with a specific focus on flood inundation mapping in the field of remote sensing. In this study, we introduce a novel framework of Interpretable Deep Active Learning for Flood inundation Mapping (IDAL-FIM), specifically in terms of class ambiguity of multi-spectral satellite images. In the experiments, we utilize Sen1Floods11 dataset, and adopt U-Net with MC-dropout. In addition, we employ five acquisition functions, which are the random, K-means, BALD, entropy, and margin acquisition functions. Based on the experimental results, we demonstrate that two proposed class ambiguity indices are effective variables to interpret the deep active learning by establishing statistically significant correlation with the predictive uncertainty of the deep learning model at the tile level. Then, we illustrate the behaviors of deep active learning through visualizing two-dimensional density plots and providing interpretations regarding the operation of deep active learning, in flood inundation mapping.

Title: A Framework for Real-time Safeguarding the Text Generation of Large Language

• Authors: Ximing Dong, Dayi Lin, Shaowei Wang, Ahmed E. Hassan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19048
• Pdf URL: https://arxiv.org/pdf/2404.19048
• Copy Paste: [[2404.19048]] A Framework for Real-time Safeguarding the Text Generation of Large Language(https://arxiv.org/abs/2404.19048)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have significantly advanced natural language processing (NLP) tasks but also pose ethical and societal risks due to their propensity to generate harmful content. To address this, various approaches have been developed to safeguard LLMs from producing unsafe content. However, existing methods have limitations, including the need for training specific control models and proactive intervention during text generation, that lead to quality degradation and increased computational overhead. To mitigate those limitations, we propose LLMSafeGuard, a lightweight framework to safeguard LLM text generation in real-time. LLMSafeGuard integrates an external validator into the beam search algorithm during decoding, rejecting candidates that violate safety constraints while allowing valid ones to proceed. We introduce a similarity based validation approach, simplifying constraint introduction and eliminating the need for control model training. Additionally, LLMSafeGuard employs a context-wise timing selection strategy, intervening LLMs only when necessary. We evaluate LLMSafe-Guard on two tasks, detoxification and copyright safeguarding, and demonstrate its superior performance over SOTA baselines. For instance, LLMSafeGuard reduces the average toxic score of. LLM output by 29.7% compared to the best baseline meanwhile preserving similar linguistic quality as natural output in detoxification task. Similarly, in the copyright task, LLMSafeGuard decreases the Longest Common Subsequence (LCS) by 56.2% compared to baselines. Moreover, our context-wise timing selection strategy reduces inference time by at least 24% meanwhile maintaining comparable effectiveness as validating each time step. LLMSafeGuard also offers tunable parameters to balance its effectiveness and efficiency.

Title: SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications

• Authors: Liang Xu, Lei Zhu, Yaotong Wu, Hang Xue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19063
• Pdf URL: https://arxiv.org/pdf/2404.19063
• Copy Paste: [[2404.19063]] SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications(https://arxiv.org/abs/2404.19063)
• Keywords: secure, large language model
• Abstract: The SuperCLUE-Fin (SC-Fin) benchmark is a pioneering evaluation framework tailored for Chinese-native financial large language models (FLMs). It assesses FLMs across six financial application domains and twenty-five specialized tasks, encompassing theoretical knowledge and practical applications such as compliance, risk management, and investment analysis. Using multi-turn, open-ended conversations that mimic real-life scenarios, SC-Fin measures models on a range of criteria, including accurate financial understanding, logical reasoning, clarity, computational efficiency, business acumen, risk perception, and compliance with Chinese regulations. In a rigorous evaluation involving over a thousand questions, SC-Fin identifies a performance hierarchy where domestic models like GLM-4 and MoonShot-v1-128k outperform others with an A-grade, highlighting the potential for further development in transforming theoretical knowledge into pragmatic financial solutions. This benchmark serves as a critical tool for refining FLMs in the Chinese context, directing improvements in financial knowledge databases, standardizing financial interpretations, and promoting models that prioritize compliance, risk management, and secure practices. We create a contextually relevant and comprehensive benchmark that drives the development of AI in the Chinese financial sector. SC-Fin facilitates the advancement and responsible deployment of FLMs, offering valuable insights for enhancing model performance and usability for both individual and institutional users in the Chinese market..~\footnote{Our benchmark can be found at \url{https://www.CLUEbenchmarks.com}}.

Title: Zero Knowledge Proof for Multiple Sequence Alignment

• Authors: Worasait Suwannik
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19064
• Pdf URL: https://arxiv.org/pdf/2404.19064
• Copy Paste: [[2404.19064]] Zero Knowledge Proof for Multiple Sequence Alignment(https://arxiv.org/abs/2404.19064)
• Keywords: protect
• Abstract: Multiple sequence alignment (MSA) is a fundamental algorithm in bioinformatics. In a situation when the alignment might need to be protected while revealing the other information such the input sequences and the alignment score, zero knowledge proof can be used. In this paper, a validator checks the consistency between the input sequence and the alignment, and between the alignment and the alignment score. The validator is written in Circom language which will be compile into a circuit. Using a zero knowledge prove system called zkSNARK, a cryptographic proof is generates for the circuit and its input. This proof demonstrates that all inputs are consistent without revealing the actual alignment.

Title: Revolutionizing Traffic Sign Recognition: Unveiling the Potential of Vision Transformers

• Authors: Susano Mingwin, Yulong Shisu, Yongshuai Wanwag, Sunshin Huing
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19066
• Pdf URL: https://arxiv.org/pdf/2404.19066
• Copy Paste: [[2404.19066]] Revolutionizing Traffic Sign Recognition: Unveiling the Potential of Vision Transformers(https://arxiv.org/abs/2404.19066)
• Keywords: extraction, transformer
• Abstract: This research introduces an innovative method for Traffic Sign Recognition (TSR) by leveraging deep learning techniques, with a particular emphasis on Vision Transformers. TSR holds a vital role in advancing driver assistance systems and autonomous vehicles. Traditional TSR approaches, reliant on manual feature extraction, have proven to be labor-intensive and costly. Moreover, methods based on shape and color have inherent limitations, including susceptibility to various factors and changes in lighting conditions. This study explores three variants of Vision Transformers (PVT, TNT, LNL) and six convolutional neural networks (AlexNet, ResNet, VGG16, MobileNet, EfficientNet, GoogleNet) as baseline models. To address the shortcomings of traditional methods, a novel pyramid EATFormer backbone is proposed, amalgamating Evolutionary Algorithms (EAs) with the Transformer architecture. The introduced EA-based Transformer block captures multi-scale, interactive, and individual information through its components: Feed-Forward Network, Global and Local Interaction, and Multi-Scale Region Aggregation modules. Furthermore, a Modulated Deformable MSA module is introduced to dynamically model irregular locations. Experimental evaluations on the GTSRB and BelgiumTS datasets demonstrate the efficacy of the proposed approach in enhancing both prediction speed and accuracy. This study concludes that Vision Transformers hold significant promise in traffic sign classification and contributes a fresh algorithmic framework for TSR. These findings set the stage for the development of precise and dependable TSR algorithms, benefiting driver assistance systems and autonomous vehicles.

Title: In-Context Symbolic Regression: Leveraging Language Models for Function Discovery

• Authors: Matteo Merler, Nicola Dainese, Katsiaryna Haitsiukevich
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.19094
• Pdf URL: https://arxiv.org/pdf/2404.19094
• Copy Paste: [[2404.19094]] In-Context Symbolic Regression: Leveraging Language Models for Function Discovery(https://arxiv.org/abs/2404.19094)
• Keywords: transformer, large language model
• Abstract: Symbolic Regression (SR) is a task which aims to extract the mathematical expression underlying a set of empirical observations. Transformer-based methods trained on SR datasets detain the current state-of-the-art in this task, while the application of Large Language Models (LLMs) to SR remains unexplored. This work investigates the integration of pre-trained LLMs into the SR pipeline, utilizing an approach that iteratively refines a functional form based on the prediction error it achieves on the observation set, until it reaches convergence. Our method leverages LLMs to propose an initial set of possible functions based on the observations, exploiting their strong pre-training prior. These functions are then iteratively refined by the model itself and by an external optimizer for their coefficients. The process is repeated until the results are satisfactory. We then analyze Vision-Language Models in this context, exploring the inclusion of plots as visual inputs to aid the optimization process. Our findings reveal that LLMs are able to successfully recover good symbolic equations that fit the given data, outperforming SR baselines based on Genetic Programming, with the addition of images in the input showing promising results for the most complex benchmarks.

Title: Real-Time Convolutional Neural Network-Based Star Detection and Centroiding Method for CubeSat Star Tracker

• Authors: Hongrui Zhao, Michael F. Lembeck, Adrian Zhuang, Riya Shah, Jesse Wei
• Subjects: cs.CV, astro-ph.IM, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.19108
• Pdf URL: https://arxiv.org/pdf/2404.19108
• Copy Paste: [[2404.19108]] Real-Time Convolutional Neural Network-Based Star Detection and Centroiding Method for CubeSat Star Tracker(https://arxiv.org/abs/2404.19108)
• Keywords: segmentation
• Abstract: Star trackers are one of the most accurate celestial sensors used for absolute attitude determination. The devices detect stars in captured images and accurately compute their projected centroids on an imaging focal plane with subpixel precision. Traditional algorithms for star detection and centroiding often rely on threshold adjustments for star pixel detection and pixel brightness weighting for centroid computation. However, challenges like high sensor noise and stray light can compromise algorithm performance. This article introduces a Convolutional Neural Network (CNN)-based approach for star detection and centroiding, tailored to address the issues posed by noisy star tracker images in the presence of stray light and other artifacts. Trained using simulated star images overlayed with real sensor noise and stray light, the CNN produces both a binary segmentation map distinguishing star pixels from the background and a distance map indicating each pixel's proximity to the nearest star centroid. Leveraging this distance information alongside pixel coordinates transforms centroid calculations into a set of trilateration problems solvable via the least squares method. Our method employs efficient UNet variants for the underlying CNN architectures, and the variants' performances are evaluated. Comprehensive testing has been undertaken with synthetic image evaluations, hardware-in-the-loop assessments, and night sky tests. The tests consistently demonstrated that our method outperforms several existing algorithms in centroiding accuracy and exhibits superior resilience to high sensor noise and stray light interference. An additional benefit of our algorithms is that they can be executed in real-time on low-power edge AI processors.

Title: Source-Free Domain Adaptation of Weakly-Supervised Object Localization Models for Histology

• Authors: Alexis Guichemerre, Soufiane Belharbi, Tsiry Mayet, Shakeeb Murtaza, Pourya Shamsolmoali, Luke McCaffrey, Eric Granger
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.19113
• Pdf URL: https://arxiv.org/pdf/2404.19113
• Copy Paste: [[2404.19113]] Source-Free Domain Adaptation of Weakly-Supervised Object Localization Models for Histology(https://arxiv.org/abs/2404.19113)
• Keywords: privacy
• Abstract: Given the emergence of deep learning, digital pathology has gained popularity for cancer diagnosis based on histology images. Deep weakly supervised object localization (WSOL) models can be trained to classify histology images according to cancer grade and identify regions of interest (ROIs) for interpretation, using inexpensive global image-class annotations. A WSOL model initially trained on some labeled source image data can be adapted using unlabeled target data in cases of significant domain shifts caused by variations in staining, scanners, and cancer type. In this paper, we focus on source-free (unsupervised) domain adaptation (SFDA), a challenging problem where a pre-trained source model is adapted to a new target domain without using any source domain data for privacy and efficiency reasons. SFDA of WSOL models raises several challenges in histology, most notably because they are not intended to adapt for both classification and localization tasks. In this paper, 4 state-of-the-art SFDA methods, each one representative of a main SFDA family, are compared for WSOL in terms of classification and localization accuracy. They are the SFDA-Distribution Estimation, Source HypOthesis Transfer, Cross-Domain Contrastive Learning, and Adaptively Domain Statistics Alignment. Experimental results on the challenging Glas (smaller, breast cancer) and Camelyon16 (larger, colon cancer) histology datasets indicate that these SFDA methods typically perform poorly for localization after adaptation when optimized for classification.

Title: Enhancing IoT Security: A Novel Feature Engineering Approach for ML-Based Intrusion Detection Systems

• Authors: Afsaneh Mahanipour, Hana Khamfroush
• Subjects: cs.CR, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.19114
• Pdf URL: https://arxiv.org/pdf/2404.19114
• Copy Paste: [[2404.19114]] Enhancing IoT Security: A Novel Feature Engineering Approach for ML-Based Intrusion Detection Systems(https://arxiv.org/abs/2404.19114)
• Keywords: security, attack
• Abstract: The integration of Internet of Things (IoT) applications in our daily lives has led to a surge in data traffic, posing significant security challenges. IoT applications using cloud and edge computing are at higher risk of cyberattacks because of the expanded attack surface from distributed edge and cloud services, the vulnerability of IoT devices, and challenges in managing security across interconnected systems leading to oversights. This led to the rise of ML-based solutions for intrusion detection systems (IDSs), which have proven effective in enhancing network security and defending against diverse threats. However, ML-based IDS in IoT systems encounters challenges, particularly from noisy, redundant, and irrelevant features in varied IoT datasets, potentially impacting its performance. Therefore, reducing such features becomes crucial to enhance system performance and minimize computational costs. This paper focuses on improving the effectiveness of ML-based IDS at the edge level by introducing a novel method to find a balanced trade-off between cost and accuracy through the creation of informative features in a two-tier edge-user IoT environment. A hybrid Binary Quantum-inspired Artificial Bee Colony and Genetic Programming algorithm is utilized for this purpose. Three IoT intrusion detection datasets, namely NSL-KDD, UNSW-NB15, and BoT-IoT, are used for the evaluation of the proposed approach.

Title: Accelerating Production LLMs with Combined Token/Embedding Speculators

• Authors: Davis Wertheimer, Joshua Rosenkranz, Thomas Parnell, Sahil Suneja, Pavithra Ranganathan, Raghu Ganti, Mudhakar Srivatsa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19124
• Pdf URL: https://arxiv.org/pdf/2404.19124
• Copy Paste: [[2404.19124]] Accelerating Production LLMs with Combined Token/Embedding Speculators(https://arxiv.org/abs/2404.19124)
• Keywords: large language model
• Abstract: This technical report describes the design and training of novel speculative decoding draft models, for accelerating the inference speeds of large language models in a production environment. By conditioning draft predictions on both context vectors and sampled tokens, we can train our speculators to efficiently predict high-quality n-grams, which the base model then accepts or rejects. This allows us to effectively predict multiple tokens per inference forward pass, accelerating wall-clock inference speeds of highly optimized base model implementations by a factor of 2-3x. We explore these initial results and describe next steps for further improvements.

Title: RTF: Region-based Table Filling Method for Relational Triple Extraction

• Authors: Ning An, Lei Hei, Yong Jiang, Weiping Meng, Jingjing Hu, Boran Huang, Feiliang Ren
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19154
• Pdf URL: https://arxiv.org/pdf/2404.19154
• Copy Paste: [[2404.19154]] RTF: Region-based Table Filling Method for Relational Triple Extraction(https://arxiv.org/abs/2404.19154)
• Keywords: extraction
• Abstract: Relational triple extraction is crucial work for the automatic construction of knowledge graphs. Existing methods only construct shallow representations from a token or token pair-level. However, previous works ignore local spatial dependencies of relational triples, resulting in a weakness of entity pair boundary detection. To tackle this problem, we propose a novel Region-based Table Filling method (RTF). We devise a novel region-based tagging scheme and bi-directional decoding strategy, which regard each relational triple as a region on the relation-specific table, and identifies triples by determining two endpoints of each region. We also introduce convolution to construct region-level table representations from a spatial perspective which makes triples easier to be captured. In addition, we share partial tagging scores among different relations to improve learning efficiency of relation classifier. Experimental results show that our method achieves state-of-the-art with better generalization capability on three variants of two widely used benchmark datasets.

Title: What Drives Performance in Multilingual Language Models?

• Authors: Sina Bagheri Nezhad, Ameeta Agrawal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19159
• Pdf URL: https://arxiv.org/pdf/2404.19159
• Copy Paste: [[2404.19159]] What Drives Performance in Multilingual Language Models?(https://arxiv.org/abs/2404.19159)
• Keywords: large language model
• Abstract: This study investigates the factors influencing the performance of multilingual large language models (MLLMs) across diverse languages. We study 6 MLLMs, including masked language models, autoregressive models, and instruction-tuned LLMs, on the SIB-200 dataset, a topic classification dataset encompassing 204 languages. Our analysis considers three scenarios: ALL languages, SEEN languages (present in the model's pretraining data), and UNSEEN languages (not present or documented in the model's pretraining data in any meaningful way). We examine the impact of factors such as pretraining data size, general resource availability, language family, and script type on model performance. Decision tree analysis reveals that pretraining data size is the most influential factor for SEEN languages. However, interestingly, script type and language family are crucial for UNSEEN languages, highlighting the importance of cross-lingual transfer learning. Notably, model size and architecture do not significantly alter the most important features identified. Our findings provide valuable insights into the strengths and limitations of current MLLMs and hope to guide the development of more effective and equitable multilingual NLP systems.

Title: PEVA-Net: Prompt-Enhanced View Aggregation Network for Zero/Few-Shot Multi-View 3D Shape Recognition

• Authors: Dongyun Lin, Yi Cheng, Shangbo Mao, Aiyuan Guo, Yiqun Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19168
• Pdf URL: https://arxiv.org/pdf/2404.19168
• Copy Paste: [[2404.19168]] PEVA-Net: Prompt-Enhanced View Aggregation Network for Zero/Few-Shot Multi-View 3D Shape Recognition(https://arxiv.org/abs/2404.19168)
• Keywords: transformer
• Abstract: Large vision-language models have impressively promote the performance of 2D visual recognition under zero/few-shot scenarios. In this paper, we focus on exploiting the large vision-language model, i.e., CLIP, to address zero/few-shot 3D shape recognition based on multi-view representations. The key challenge for both tasks is to generate a discriminative descriptor of the 3D shape represented by multiple view images under the scenarios of either without explicit training (zero-shot 3D shape recognition) or training with a limited number of data (few-shot 3D shape recognition). We analyze that both tasks are relevant and can be considered simultaneously. Specifically, leveraging the descriptor which is effective for zero-shot inference to guide the tuning of the aggregated descriptor under the few-shot training can significantly improve the few-shot learning efficacy. Hence, we propose Prompt-Enhanced View Aggregation Network (PEVA-Net) to simultaneously address zero/few-shot 3D shape recognition. Under the zero-shot scenario, we propose to leverage the prompts built up from candidate categories to enhance the aggregation process of multiple view-associated visual features. The resulting aggregated feature serves for effective zero-shot recognition of the 3D shapes. Under the few-shot scenario, we first exploit a transformer encoder to aggregate the view-associated visual features into a global descriptor. To tune the encoder, together with the main classification loss, we propose a self-distillation scheme via a feature distillation loss by treating the zero-shot descriptor as the guidance signal for the few-shot descriptor. This scheme can significantly enhance the few-shot learning efficacy.

Title: XFeat: Accelerated Features for Lightweight Image Matching

• Authors: Guilherme Potje, Felipe Cadar, Andre Araujo, Renato Martins, Erickson R. Nascimento
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19174
• Pdf URL: https://arxiv.org/pdf/2404.19174
• Copy Paste: [[2404.19174]] XFeat: Accelerated Features for Lightweight Image Matching(https://arxiv.org/abs/2404.19174)
• Keywords: robust
• Abstract: We introduce a lightweight and accurate architecture for resource-efficient visual correspondence. Our method, dubbed XFeat (Accelerated Features), revisits fundamental design choices in convolutional neural networks for detecting, extracting, and matching local features. Our new model satisfies a critical need for fast and robust algorithms suitable to resource-limited devices. In particular, accurate image matching requires sufficiently large image resolutions - for this reason, we keep the resolution as large as possible while limiting the number of channels in the network. Besides, our model is designed to offer the choice of matching at the sparse or semi-dense levels, each of which may be more suitable for different downstream applications, such as visual navigation and augmented reality. Our model is the first to offer semi-dense matching efficiently, leveraging a novel match refinement module that relies on coarse local descriptors. XFeat is versatile and hardware-independent, surpassing current deep learning-based local features in speed (up to 5x faster) with comparable or better accuracy, proven in pose estimation and visual localization. We showcase it running in real-time on an inexpensive laptop CPU without specialized hardware optimizations. Code and weights are available at www.verlab.dcc.ufmg.br/descriptors/xfeat_cvpr24.

Title: Game-MUG: Multimodal Oriented Game Situation Understanding and Commentary Generation Dataset

• Authors: Zhihao Zhang, Feiqi Cao, Yingbin Mo, Yiran Zhang, Josiah Poon, Caren Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19175
• Pdf URL: https://arxiv.org/pdf/2404.19175
• Copy Paste: [[2404.19175]] Game-MUG: Multimodal Oriented Game Situation Understanding and Commentary Generation Dataset(https://arxiv.org/abs/2404.19175)
• Keywords: robust
• Abstract: The dynamic nature of esports makes the situation relatively complicated for average viewers. Esports broadcasting involves game expert casters, but the caster-dependent game commentary is not enough to fully understand the game situation. It will be richer by including diverse multimodal esports information, including audiences' talks/emotions, game audio, and game match event information. This paper introduces GAME-MUG, a new multimodal game situation understanding and audience-engaged commentary generation dataset and its strong baseline. Our dataset is collected from 2020-2022 LOL game live streams from YouTube and Twitch, and includes multimodal esports game information, including text, audio, and time-series event logs, for detecting the game situation. In addition, we also propose a new audience conversation augmented commentary dataset by covering the game situation and audience conversation understanding, and introducing a robust joint multimodal dual learning model as a baseline. We examine the model's game situation/event understanding ability and commentary generation capability to show the effectiveness of the multimodal aspects coverage and the joint integration learning approach.

Title: Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics

• Authors: James A. Michaelov, Catherine Arnett, Benjamin K. Bergen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19178
• Pdf URL: https://arxiv.org/pdf/2404.19178
• Copy Paste: [[2404.19178]] Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics(https://arxiv.org/abs/2404.19178)
• Keywords: transformer
• Abstract: Transformers have supplanted Recurrent Neural Networks as the dominant architecture for both natural language processing tasks and, despite criticisms of cognitive implausibility, for modelling the effect of predictability on online human language comprehension. However, two recently developed recurrent neural network architectures, RWKV and Mamba, appear to perform natural language tasks comparably to or better than transformers of equivalent scale. In this paper, we show that contemporary recurrent models are now also able to match - and in some cases, exceed - performance of comparably sized transformers at modeling online human language comprehension. This suggests that transformer language models are not uniquely suited to this task, and opens up new directions for debates about the extent to which architectural features of language models make them better or worse models of human language comprehension.

Title: Mix of Experts Language Model for Named Entity Recognition

• Authors: Xinwei Chen, Kun Li, Tianyou Song, Jiangjian Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19192
• Pdf URL: https://arxiv.org/pdf/2404.19192
• Copy Paste: [[2404.19192]] Mix of Experts Language Model for Named Entity Recognition(https://arxiv.org/abs/2404.19192)
• Keywords: robust, fair
• Abstract: Named Entity Recognition (NER) is an essential steppingstone in the field of natural language processing. Although promising performance has been achieved by various distantly supervised models, we argue that distant supervision inevitably introduces incomplete and noisy annotations, which may mislead the model training process. To address this issue, we propose a robust NER model named BOND-MoE based on Mixture of Experts (MoE). Instead of relying on a single model for NER prediction, multiple models are trained and ensembled under the Expectation-Maximization (EM) framework, so that noisy supervision can be dramatically alleviated. In addition, we introduce a fair assignment module to balance the document-model assignment process. Extensive experiments on real-world datasets show that the proposed method achieves state-of-the-art performance compared with other distantly supervised NER.

Title: TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains

• Authors: Yoonsik Kim, Moonbin Yim, Ka Yeon Song
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19205
• Pdf URL: https://arxiv.org/pdf/2404.19205
• Copy Paste: [[2404.19205]] TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains(https://arxiv.org/abs/2404.19205)
• Keywords: large language model
• Abstract: In this paper, we establish a benchmark for table visual question answering, referred to as the TableVQA-Bench, derived from pre-existing table question-answering (QA) and table structure recognition datasets. It is important to note that existing datasets have not incorporated images or QA pairs, which are two crucial components of TableVQA. As such, the primary objective of this paper is to obtain these necessary components. Specifically, images are sourced either through the application of a \textit{stylesheet} or by employing the proposed table rendering system. QA pairs are generated by exploiting the large language model (LLM) where the input is a text-formatted table. Ultimately, the completed TableVQA-Bench comprises 1,500 QA pairs. We comprehensively compare the performance of various multi-modal large language models (MLLMs) on TableVQA-Bench. GPT-4V achieves the highest accuracy among commercial and open-sourced MLLMs from our experiments. Moreover, we discover that the number of vision queries plays a significant role in TableVQA performance. To further analyze the capabilities of MLLMs in comparison to their LLM backbones, we investigate by presenting image-formatted tables to MLLMs and text-formatted tables to LLMs, respectively. Our findings suggest that processing visual inputs is more challenging than text inputs, as evidenced by the lower performance of MLLMs, despite generally requiring higher computational costs than LLMs. The proposed TableVQA-Bench and evaluation codes are available at \href{https://github.com/naver-ai/tablevqabench}{https://github.com/naver-ai/tablevqabench}.

Title: Transcrib3D: 3D Referring Expression Resolution through Large Language Models

• Authors: Jiading Fang, Xiangshan Tan, Shengjie Lin, Igor Vasiljevic, Vitor Guizilini, Hongyuan Mei, Rares Ambrus, Gregory Shakhnarovich, Matthew R Walter
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19221
• Pdf URL: https://arxiv.org/pdf/2404.19221
• Copy Paste: [[2404.19221]] Transcrib3D: 3D Referring Expression Resolution through Large Language Models(https://arxiv.org/abs/2404.19221)
• Keywords: large language model
• Abstract: If robots are to work effectively alongside people, they must be able to interpret natural language references to objects in their 3D environment. Understanding 3D referring expressions is challenging -- it requires the ability to both parse the 3D structure of the scene and correctly ground free-form language in the presence of distraction and clutter. We introduce Transcrib3D, an approach that brings together 3D detection methods and the emergent reasoning capabilities of large language models (LLMs). Transcrib3D uses text as the unifying medium, which allows us to sidestep the need to learn shared representations connecting multi-modal inputs, which would require massive amounts of annotated 3D data. As a demonstration of its effectiveness, Transcrib3D achieves state-of-the-art results on 3D reference resolution benchmarks, with a great leap in performance from previous multi-modality baselines. To improve upon zero-shot performance and facilitate local deployment on edge computers and robots, we propose self-correction for fine-tuning that trains smaller models, resulting in performance close to that of large models. We show that our method enables a real robot to perform pick-and-place tasks given queries that contain challenging referring expressions. Project site is at https://ripl.github.io/Transcrib3D.

Title: Espresso: Robust Concept Filtering in Text-to-Image Models

• Authors: Anudeep Das, Vasisht Duddu, Rui Zhang, N. Asokan
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19227
• Pdf URL: https://arxiv.org/pdf/2404.19227
• Copy Paste: [[2404.19227]] Espresso: Robust Concept Filtering in Text-to-Image Models(https://arxiv.org/abs/2404.19227)
• Keywords: robust, diffusion
• Abstract: Diffusion-based text-to-image (T2I) models generate high-fidelity images for given textual prompts. They are trained on large datasets scraped from the Internet, potentially containing unacceptable concepts (e.g., copyright infringing or unsafe). Retraining T2I models after filtering out unacceptable concepts in the training data is inefficient and degrades utility. Hence, there is a need for concept removal techniques (CRTs) which are effective in removing unacceptable concepts, utility-preserving on acceptable concepts, and robust against evasion with adversarial prompts. None of the prior filtering and fine-tuning CRTs satisfy all these requirements simultaneously. We introduce Espresso, the first robust concept filter based on Contrastive Language-Image Pre-Training (CLIP). It identifies unacceptable concepts by projecting the generated image's embedding onto the vector connecting unacceptable and acceptable concepts in the joint text-image embedding space. This ensures robustness by restricting the adversary to adding noise only along this vector, in the direction of the acceptable concept. Further fine-tuning Espresso to separate embeddings of acceptable and unacceptable concepts, while preserving their pairing with image embeddings, ensures both effectiveness and utility. We evaluate Espresso on eleven concepts to show that it is effective (~5% CLIP accuracy on unacceptable concepts), utility-preserving (~93% normalized CLIP score on acceptable concepts), and robust (~4% CLIP accuracy on adversarial prompts for unacceptable concepts). Finally, we present theoretical bounds for the certified robustness of Espresso against adversarial prompts, and an empirical analysis.

Title: GRAMMAR: Grounded and Modular Evaluation of Domain-Specific Retrieval-Augmented Language Models

• Authors: Xinzhe Li, Ming Liu, Shang Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19232
• Pdf URL: https://arxiv.org/pdf/2404.19232
• Copy Paste: [[2404.19232]] GRAMMAR: Grounded and Modular Evaluation of Domain-Specific Retrieval-Augmented Language Models(https://arxiv.org/abs/2404.19232)
• Keywords: robust
• Abstract: Retrieval-augmented Generation (RAG) systems have been actively studied and deployed across various industries to query on domain-specific knowledge base. However, evaluating these systems presents unique challenges due to the scarcity of domain-specific queries and corresponding ground truths, as well as a lack of systematic approaches to diagnosing the cause of failure cases -- whether they stem from knowledge deficits or issues related to system robustness. To address these challenges, we introduce GRAMMAR (GRounded And Modular Methodology for Assessment of RAG), an evaluation framework comprising two key elements: 1) a data generation process that leverages relational databases and LLMs to efficiently produce scalable query-answer pairs. This method facilitates the separation of query logic from linguistic variations for enhanced debugging capabilities; and 2) an evaluation framework that differentiates knowledge gaps from robustness and enables the identification of defective modules. Our empirical results underscore the limitations of current reference-free evaluation approaches and the reliability of GRAMMAR to accurately identify model vulnerabilities.

Title: HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning

• Authors: Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, Chengzhong Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19245
• Pdf URL: https://arxiv.org/pdf/2404.19245
• Copy Paste: [[2404.19245]] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning(https://arxiv.org/abs/2404.19245)
• Keywords: large language model
• Abstract: Adapting Large Language Models (LLMs) to new tasks through fine-tuning has been made more efficient by the introduction of Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced in complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through a series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA, a LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases. \href{https://github.com/Clin0212/HydraLoRA}{Code}.

Title: Improved AutoEncoder with LSTM module and KL divergence

• Authors: Wei Huang, Bingyang Zhang, Kaituo Zhang, Hua Gao, Rongchun Wan
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19247
• Pdf URL: https://arxiv.org/pdf/2404.19247
• Copy Paste: [[2404.19247]] Improved AutoEncoder with LSTM module and KL divergence(https://arxiv.org/abs/2404.19247)
• Keywords: robust
• Abstract: The task of anomaly detection is to separate anomalous data from normal data in the dataset. Models such as deep convolutional autoencoder (CAE) network and deep supporting vector data description (SVDD) model have been universally employed and have demonstrated significant success in detecting anomalies. However, the over-reconstruction ability of CAE network for anomalous data can easily lead to high false negative rate in detecting anomalous data. On the other hand, the deep SVDD model has the drawback of feature collapse, which leads to a decrease of detection accuracy for anomalies. To address these problems, we propose the Improved AutoEncoder with LSTM module and Kullback-Leibler divergence (IAE-LSTM-KL) model in this paper. An LSTM network is added after the encoder to memorize feature representations of normal data. In the meanwhile, the phenomenon of feature collapse can also be mitigated by penalizing the featured input to SVDD module via KL divergence. The efficacy of the IAE-LSTM-KL model is validated through experiments on both synthetic and real-world datasets. Experimental results show that IAE-LSTM-KL model yields higher detection accuracy for anomalies. In addition, it is also found that the IAE-LSTM-KL model demonstrates enhanced robustness to contaminated outliers in the dataset.

Title: Suvach -- Generated Hindi QA benchmark

• Authors: Vaishak Narayanan, Prabin Raj KP, Saifudheen Nouphal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19254
• Pdf URL: https://arxiv.org/pdf/2404.19254
• Copy Paste: [[2404.19254]] Suvach -- Generated Hindi QA benchmark(https://arxiv.org/abs/2404.19254)
• Keywords: large language model
• Abstract: Current evaluation benchmarks for question answering (QA) in Indic languages often rely on machine translation of existing English datasets. This approach suffers from bias and inaccuracies inherent in machine translation, leading to datasets that may not reflect the true capabilities of EQA models for Indic languages. This paper proposes a new benchmark specifically designed for evaluating Hindi EQA models and discusses the methodology to do the same for any task. This method leverages large language models (LLMs) to generate a high-quality dataset in an extractive setting, ensuring its relevance for the target language. We believe this new resource will foster advancements in Hindi NLP research by providing a more accurate and reliable evaluation tool.

Title: DELINE8K: A Synthetic Data Pipeline for the Semantic Segmentation of Historical Documents

• Authors: Taylor Archibald, Tony Martinez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19259
• Pdf URL: https://arxiv.org/pdf/2404.19259
• Copy Paste: [[2404.19259]] DELINE8K: A Synthetic Data Pipeline for the Semantic Segmentation of Historical Documents(https://arxiv.org/abs/2404.19259)
• Keywords: segmentation
• Abstract: Document semantic segmentation is a promising avenue that can facilitate document analysis tasks, including optical character recognition (OCR), form classification, and document editing. Although several synthetic datasets have been developed to distinguish handwriting from printed text, they fall short in class variety and document diversity. We demonstrate the limitations of training on existing datasets when solving the National Archives Form Semantic Segmentation dataset (NAFSS), a dataset which we introduce. To address these limitations, we propose the most comprehensive document semantic segmentation synthesis pipeline to date, incorporating preprinted text, handwriting, and document backgrounds from over 10 sources to create the Document Element Layer INtegration Ensemble 8K, or DELINE8K dataset. Our customized dataset exhibits superior performance on the NAFSS benchmark, demonstrating it as a promising tool in further research. The DELINE8K dataset is available at https://github.com/Tahlor/deline8k.

Title: Aspect and Opinion Term Extraction Using Graph Attention Network

• Authors: Abir Chakraborty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19260
• Pdf URL: https://arxiv.org/pdf/2404.19260
• Copy Paste: [[2404.19260]] Aspect and Opinion Term Extraction Using Graph Attention Network(https://arxiv.org/abs/2404.19260)
• Keywords: extraction, transformer
• Abstract: In this work we investigate the capability of Graph Attention Network for extracting aspect and opinion terms. Aspect and opinion term extraction is posed as a token-level classification task akin to named entity recognition. We use the dependency tree of the input query as additional feature in a Graph Attention Network along with the token and part-of-speech features. We show that the dependency structure is a powerful feature that in the presence of a CRF layer substantially improves the performance and generates the best result on the commonly used datasets from SemEval 2014, 2015 and 2016. We experiment with additional layers like BiLSTM and Transformer in addition to the CRF layer. We also show that our approach works well in the presence of multiple aspects or sentiments in the same query and it is not necessary to modify the dependency tree based on a single aspect as was the original application for sentiment classification.

Title: High dimensional analysis reveals conservative sharpening and a stochastic edge of stability

• Authors: Atish Agarwala, Jeffrey Pennington
• Subjects: cs.LG, math.OC, math.ST, physics.data-an
• Abstract URL: https://arxiv.org/abs/2404.19261
• Pdf URL: https://arxiv.org/pdf/2404.19261
• Copy Paste: [[2404.19261]] High dimensional analysis reveals conservative sharpening and a stochastic edge of stability(https://arxiv.org/abs/2404.19261)
• Keywords: robust
• Abstract: Recent empirical and theoretical work has shown that the dynamics of the large eigenvalues of the training loss Hessian have some remarkably robust features across models and datasets in the full batch regime. There is often an early period of progressive sharpening where the large eigenvalues increase, followed by stabilization at a predictable value known as the edge of stability. Previous work showed that in the stochastic setting, the eigenvalues increase more slowly - a phenomenon we call conservative sharpening. We provide a theoretical analysis of a simple high-dimensional model which shows the origin of this slowdown. We also show that there is an alternative stochastic edge of stability which arises at small batch size that is sensitive to the trace of the Neural Tangent Kernel rather than the large Hessian eigenvalues. We conduct an experimental study which highlights the qualitative differences from the full batch phenomenology, and suggests that controlling the stochastic edge of stability can help optimization.

Title: Mapping New Realities: Ground Truth Image Creation with Pix2Pix Image-to-Image Translation

• Authors: Zhenglin Li, Bo Guan, Yuanzhou Wei, Yiming Zhou, Jingyu Zhang, Jinxin Xu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.19265
• Pdf URL: https://arxiv.org/pdf/2404.19265
• Copy Paste: [[2404.19265]] Mapping New Realities: Ground Truth Image Creation with Pix2Pix Image-to-Image Translation(https://arxiv.org/abs/2404.19265)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) have significantly advanced image processing, with Pix2Pix being a notable framework for image-to-image translation. This paper explores a novel application of Pix2Pix to transform abstract map images into realistic ground truth images, addressing the scarcity of such images crucial for domains like urban planning and autonomous vehicle training. We detail the Pix2Pix model's utilization for generating high-fidelity datasets, supported by a dataset of paired map and aerial images, and enhanced by a tailored training regimen. The results demonstrate the model's capability to accurately render complex urban features, establishing its efficacy and potential for broad real-world applications.

Title: C2FDrone: Coarse-to-Fine Drone-to-Drone Detection using Vision Transformer Networks

• Authors: Sairam VC Rebbapragada, Pranoy Panda, Vineeth N Balasubramanian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19276
• Pdf URL: https://arxiv.org/pdf/2404.19276
• Copy Paste: [[2404.19276]] C2FDrone: Coarse-to-Fine Drone-to-Drone Detection using Vision Transformer Networks(https://arxiv.org/abs/2404.19276)
• Keywords: transformer
• Abstract: A vision-based drone-to-drone detection system is crucial for various applications like collision avoidance, countering hostile drones, and search-and-rescue operations. However, detecting drones presents unique challenges, including small object sizes, distortion, occlusion, and real-time processing requirements. Current methods integrating multi-scale feature fusion and temporal information have limitations in handling extreme blur and minuscule objects. To address this, we propose a novel coarse-to-fine detection strategy based on vision transformers. We evaluate our approach on three challenging drone-to-drone detection datasets, achieving F1 score enhancements of 7%, 3%, and 1% on the FL-Drones, AOT, and NPS-Drones datasets, respectively. Additionally, we demonstrate real-time processing capabilities by deploying our model on an edge-computing device. Our code will be made publicly available.

Title: Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model

• Authors: Wentao Lei, Li Liu, Jun Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19277
• Pdf URL: https://arxiv.org/pdf/2404.19277
• Copy Paste: [[2404.19277]] Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model(https://arxiv.org/abs/2404.19277)
• Keywords: diffusion
• Abstract: Cued Speech (CS) is an advanced visual phonetic encoding system that integrates lip reading with hand codings, enabling people with hearing impairments to communicate efficiently. CS video generation aims to produce specific lip and gesture movements of CS from audio or text inputs. The main challenge is that given limited CS data, we strive to simultaneously generate fine-grained hand and finger movements, as well as lip movements, meanwhile the two kinds of movements need to be asynchronously aligned. Existing CS generation methods are fragile and prone to poor performance due to template-based statistical models and careful hand-crafted pre-processing to fit the models. Therefore, we propose a novel Gloss-prompted Diffusion-based CS Gesture generation framework (called GlossDiff). Specifically, to integrate additional linguistic rules knowledge into the model. we first introduce a bridging instruction called \textbf{Gloss}, which is an automatically generated descriptive text to establish a direct and more delicate semantic connection between spoken language and CS gestures. Moreover, we first suggest rhythm is an important paralinguistic feature for CS to improve the communication efficacy. Therefore, we propose a novel Audio-driven Rhythmic Module (ARM) to learn rhythm that matches audio speech. Moreover, in this work, we design, record, and publish the first Chinese CS dataset with four CS cuers. Extensive experiments demonstrate that our method quantitatively and qualitatively outperforms current state-of-the-art (SOTA) methods. We release the code and data at https://glossdiff.github.io/.

Title: Quater-GCN: Enhancing 3D Human Pose Estimation with Orientation and Semi-supervised Training

• Authors: Xingyu Song, Zhan Li, Shi Chen, Kazuyuki Demachi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19279
• Pdf URL: https://arxiv.org/pdf/2404.19279
• Copy Paste: [[2404.19279]] Quater-GCN: Enhancing 3D Human Pose Estimation with Orientation and Semi-supervised Training(https://arxiv.org/abs/2404.19279)
• Keywords: security
• Abstract: 3D human pose estimation is a vital task in computer vision, involving the prediction of human joint positions from images or videos to reconstruct a skeleton of a human in three-dimensional space. This technology is pivotal in various fields, including animation, security, human-computer interaction, and automotive safety, where it promotes both technological progress and enhanced human well-being. The advent of deep learning significantly advances the performance of 3D pose estimation by incorporating temporal information for predicting the spatial positions of human joints. However, traditional methods often fall short as they primarily focus on the spatial coordinates of joints and overlook the orientation and rotation of the connecting bones, which are crucial for a comprehensive understanding of human pose in 3D space. To address these limitations, we introduce Quater-GCN (Q-GCN), a directed graph convolutional network tailored to enhance pose estimation by orientation. Q-GCN excels by not only capturing the spatial dependencies among node joints through their coordinates but also integrating the dynamic context of bone rotations in 2D space. This approach enables a more sophisticated representation of human poses by also regressing the orientation of each bone in 3D space, moving beyond mere coordinate prediction. Furthermore, we complement our model with a semi-supervised training strategy that leverages unlabeled data, addressing the challenge of limited orientation ground truth data. Through comprehensive evaluations, Q-GCN has demonstrated outstanding performance against current state-of-the-art methods.

Title: Soft Prompt Generation for Domain Generalization

• Authors: Shuanghao Bai, Yuedi Zhang, Wanqi Zhou, Zhirong Luan, Badong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19286
• Pdf URL: https://arxiv.org/pdf/2404.19286
• Copy Paste: [[2404.19286]] Soft Prompt Generation for Domain Generalization(https://arxiv.org/abs/2404.19286)
• Keywords: generative
• Abstract: Large pre-trained vision language models (VLMs) have shown impressive zero-shot ability on downstream tasks with manually designed prompt, which are not optimal for specific domains. To further adapt VLMs to downstream tasks, soft prompt is proposed to replace manually designed prompt, which acts as a learning vector that undergoes fine-tuning based on specific domain data. Prior prompt learning methods primarily learn a fixed prompt and residuled prompt from training samples. However, the learned prompts lack diversity and ignore information about unseen domains, potentially compromising the transferability of the prompts. In this paper, we reframe the prompt learning framework from a generative perspective and propose a simple yet efficient method for the Domain Generalization (DG) task, namely \textbf{S}oft \textbf{P}rompt \textbf{G}eneration (SPG). To the best of our knowledge, we are the first to introduce the generative model into prompt learning in VLMs and explore its potential for producing soft prompts by relying solely on the generative model, ensuring the diversity of prompts. Specifically, SPG consists of a two-stage training phase and an inference phase. During the training phase, we introduce soft prompt labels for each domain, aiming to incorporate the generative model domain knowledge. During the inference phase, the generator of the generative model is employed to obtain instance-specific soft prompts for the unseen target domain. Extensive experiments on five domain generalization benchmarks of three DG tasks demonstrate that our proposed SPG achieves state-of-the-art performance. The code will be available soon.

Title: Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective

• Authors: Wanqi Zhou, Shuanghao Bai, Qibin Zhao, Badong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19287
• Pdf URL: https://arxiv.org/pdf/2404.19287
• Copy Paste: [[2404.19287]] Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective(https://arxiv.org/abs/2404.19287)
• Keywords: security, attack, robust
• Abstract: Pretrained vision-language models (VLMs) like CLIP have shown impressive generalization performance across various downstream tasks, yet they remain vulnerable to adversarial attacks. While prior research has primarily concentrated on improving the adversarial robustness of image encoders to guard against attacks on images, the exploration of text-based and multimodal attacks has largely been overlooked. In this work, we initiate the first known and comprehensive effort to study adapting vision-language models for adversarial robustness under the multimodal attack. Firstly, we introduce a multimodal attack strategy and investigate the impact of different attacks. We then propose a multimodal contrastive adversarial training loss, aligning the clean and adversarial text embeddings with the adversarial and clean visual features, to enhance the adversarial robustness of both image and text encoders of CLIP. Extensive experiments on 15 datasets across two tasks demonstrate that our method significantly improves the adversarial robustness of CLIP. Interestingly, we find that the model fine-tuned against multimodal adversarial attacks exhibits greater robustness than its counterpart fine-tuned solely against image-based attacks, even in the context of image attacks, which may open up new possibilities for enhancing the security of VLMs.

Title: Robust Pedestrian Detection via Constructing Versatile Pedestrian Knowledge Bank

• Authors: Sungjune Park, Hyunjun Kim, Yong Man Ro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19299
• Pdf URL: https://arxiv.org/pdf/2404.19299
• Copy Paste: [[2404.19299]] Robust Pedestrian Detection via Constructing Versatile Pedestrian Knowledge Bank(https://arxiv.org/abs/2404.19299)
• Keywords: robust
• Abstract: Pedestrian detection is a crucial field of computer vision research which can be adopted in various real-world applications (e.g., self-driving systems). However, despite noticeable evolution of pedestrian detection, pedestrian representations learned within a detection framework are usually limited to particular scene data in which they were trained. Therefore, in this paper, we propose a novel approach to construct versatile pedestrian knowledge bank containing representative pedestrian knowledge which can be applicable to various detection frameworks and adopted in diverse scenes. We extract generalized pedestrian knowledge from a large-scale pretrained model, and we curate them by quantizing most representative features and guiding them to be distinguishable from background scenes. Finally, we construct versatile pedestrian knowledge bank which is composed of such representations, and then we leverage it to complement and enhance pedestrian features within a pedestrian detection framework. Through comprehensive experiments, we validate the effectiveness of our method, demonstrating its versatility and outperforming state-of-the-art detection performances.

Title: A Light-weight Transformer-based Self-supervised Matching Network for Heterogeneous Images

• Authors: Wang Zhang, Tingting Li, Yuntian Zhang, Gensheng Pei, Xiruo Jiang, Yazhou Yao
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.19311
• Pdf URL: https://arxiv.org/pdf/2404.19311
• Copy Paste: [[2404.19311]] A Light-weight Transformer-based Self-supervised Matching Network for Heterogeneous Images(https://arxiv.org/abs/2404.19311)
• Keywords: robust, transformer
• Abstract: Matching visible and near-infrared (NIR) images remains a significant challenge in remote sensing image fusion. The nonlinear radiometric differences between heterogeneous remote sensing images make the image matching task even more difficult. Deep learning has gained substantial attention in computer vision tasks in recent years. However, many methods rely on supervised learning and necessitate large amounts of annotated data. Nevertheless, annotated data is frequently limited in the field of remote sensing image matching. To address this challenge, this paper proposes a novel keypoint descriptor approach that obtains robust feature descriptors via a self-supervised matching network. A light-weight transformer network, termed as LTFormer, is designed to generate deep-level feature descriptors. Furthermore, we implement an innovative triplet loss function, LT Loss, to enhance the matching performance further. Our approach outperforms conventional hand-crafted local feature descriptors and proves equally competitive compared to state-of-the-art deep learning-based methods, even amidst the shortage of annotated data.

Title: Modeling Orthographic Variation in Occitan's Dialects

• Authors: Zachary William Hopton (Language and Space Lab, University of Zurich), Noëmi Aepli (Department of Computational Linguistics, University of Zurich)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19315
• Pdf URL: https://arxiv.org/pdf/2404.19315
• Copy Paste: [[2404.19315]] Modeling Orthographic Variation in Occitan's Dialects(https://arxiv.org/abs/2404.19315)
• Keywords: robust
• Abstract: Effectively normalizing textual data poses a considerable challenge, especially for low-resource languages lacking standardized writing systems. In this study, we fine-tuned a multilingual model with data from several Occitan dialects and conducted a series of experiments to assess the model's representations of these dialects. For evaluation purposes, we compiled a parallel lexicon encompassing four Occitan dialects. Intrinsic evaluations of the model's embeddings revealed that surface similarity between the dialects strengthened representations. When the model was further fine-tuned for part-of-speech tagging and Universal Dependency parsing, its performance was robust to dialectical variation, even when trained solely on part-of-speech data from a single dialect. Our findings suggest that large multilingual models minimize the need for spelling normalization during pre-processing.

Title: QLSC: A Query Latent Semantic Calibrator for Robust Extractive Question Answering

• Authors: Sheng Ouyang, Jianzong Wang, Yong Zhang, Zhitao Li, Ziqi Liang, Xulong Zhang, Ning Cheng, Jing Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19316
• Pdf URL: https://arxiv.org/pdf/2404.19316
• Copy Paste: [[2404.19316]] QLSC: A Query Latent Semantic Calibrator for Robust Extractive Question Answering(https://arxiv.org/abs/2404.19316)
• Keywords: robust
• Abstract: Extractive Question Answering (EQA) in Machine Reading Comprehension (MRC) often faces the challenge of dealing with semantically identical but format-variant inputs. Our work introduces a novel approach, called the ``Query Latent Semantic Calibrator (QLSC)'', designed as an auxiliary module for existing MRC models. We propose a unique scaling strategy to capture latent semantic center features of queries. These features are then seamlessly integrated into traditional query and passage embeddings using an attention mechanism. By deepening the comprehension of the semantic queries-passage relationship, our approach diminishes sensitivity to variations in text format and boosts the model's capability in pinpointing accurate answers. Experimental results on robust Question-Answer datasets confirm that our approach effectively handles format-variant but semantically identical queries, highlighting the effectiveness and adaptability of our proposed method.

Title: Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget

• Authors: Minh Duc Bui, Fabian David Schmidt, Goran Glavaš, Katharina von der Wense
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19319
• Pdf URL: https://arxiv.org/pdf/2404.19319
• Copy Paste: [[2404.19319]] Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget(https://arxiv.org/abs/2404.19319)
• Keywords: fair
• Abstract: Compared to standard language model (LM) pretraining (i.e., from scratch), Knowledge Distillation (KD) entails an additional forward pass through a teacher model that is typically substantially larger than the target student model. As such, KD in LM pretraining materially slows down throughput of pretraining instances vis-a-vis pretraining from scratch. Scaling laws of LM pretraining suggest that smaller models can close the gap to larger counterparts if trained on more data (i.e., processing more tokens)-and under a fixed computation budget, smaller models are able be process more data than larger models. We thus hypothesize that KD might, in fact, be suboptimal to pretraining from scratch for obtaining smaller LMs, when appropriately accounting for the compute budget. To test this, we compare pretraining from scratch against several KD strategies for masked language modeling (MLM) in a fair experimental setup, with respect to amount of computation as well as pretraining data. Downstream results on GLUE, however, do not confirm our hypothesis: while pretraining from scratch performs comparably to ordinary KD under a fixed computation budget, more sophisticated KD strategies, namely TinyBERT (Jiao et al., 2020) and MiniLM (Wang et al., 2023), outperform it by a notable margin. We further find that KD yields larger gains over pretraining from scratch when the data must be repeated under the fixed computation budget.

Title: LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation

• Authors: Lingyi Hong, Zhongying Liu, Wenchao Chen, Chenzhi Tan, Yuang Feng, Xinyu Zhou, Pinxue Guo, Jinglun Li, Zhaoyu Chen, Shuyong Gao, Wei Zhang, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19326
• Pdf URL: https://arxiv.org/pdf/2404.19326
• Copy Paste: [[2404.19326]] LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation(https://arxiv.org/abs/2404.19326)
• Keywords: segmentation
• Abstract: Video object segmentation (VOS) aims to distinguish and track target objects in a video. Despite the excellent performance achieved by off-the-shell VOS models, existing VOS benchmarks mainly focus on short-term videos lasting about 5 seconds, where objects remain visible most of the time. However, these benchmarks poorly represent practical applications, and the absence of long-term datasets restricts further investigation of VOS in realistic scenarios. Thus, we propose a novel benchmark named LVOS, comprising 720 videos with 296,401 frames and 407,945 high-quality annotations. Videos in LVOS last 1.14 minutes on average, approximately 5 times longer than videos in existing datasets. Each video includes various attributes, especially challenges deriving from the wild, such as long-term reappearing and cross-temporal similar objects. Compared to previous benchmarks, our LVOS better reflects VOS models' performance in real scenarios. Based on LVOS, we evaluate 20 existing VOS models under 4 different settings and conduct a comprehensive analysis. On LVOS, these models suffer a large performance drop, highlighting the challenge of achieving precise tracking and segmentation in real-world scenarios. Attribute-based analysis indicates that key factor to accuracy decline is the increased video length, emphasizing LVOS's crucial role. We hope our LVOS can advance development of VOS in real scenes. Data and code are available at https://lingyihongfd.github.io/lvos.github.io/.

Title: End-to-end information extraction in handwritten documents: Understanding Paris marriage records from 1880 to 1940

• Authors: Thomas Constum, Lucas Preel, Théo Larcher, Pierrick Tranouez, Thierry Paquet, Sandra Brée
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19329
• Pdf URL: https://arxiv.org/pdf/2404.19329
• Copy Paste: [[2404.19329]] End-to-end information extraction in handwritten documents: Understanding Paris marriage records from 1880 to 1940(https://arxiv.org/abs/2404.19329)
• Keywords: extraction, segmentation
• Abstract: The EXO-POPP project aims to establish a comprehensive database comprising 300,000 marriage records from Paris and its suburbs, spanning the years 1880 to 1940, which are preserved in over 130,000 scans of double pages. Each marriage record may encompass up to 118 distinct types of information that require extraction from plain text. In this paper, we introduce the M-POPP dataset, a subset of the M-POPP database with annotations for full-page text recognition and information extraction in both handwritten and printed documents, and which is now publicly available. We present a fully end-to-end architecture adapted from the DAN, designed to perform both handwritten text recognition and information extraction directly from page images without the need for explicit segmentation. We showcase the information extraction capabilities of this architecture by achieving a new state of the art for full-page Information Extraction on Esposalles and we use this architecture as a baseline for the M-POPP dataset. We also assess and compare how different encoding strategies for named entities in the text affect the performance of jointly recognizing handwritten text and extracting information, from full pages.

Title: StablePT: Towards Stable Prompting for Few-shot Learning via Input Separation

• Authors: Xiaoming Liu, Chen Liu, Zhaohan Zhang, Chengzhengxu Li, Longtian Wang, Yu Lan, Chao Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19335
• Pdf URL: https://arxiv.org/pdf/2404.19335
• Copy Paste: [[2404.19335]] StablePT: Towards Stable Prompting for Few-shot Learning via Input Separation(https://arxiv.org/abs/2404.19335)
• Keywords: robust, large language model
• Abstract: Large language models have shown their ability to become effective few-shot learners with prompting, revoluting the paradigm of learning with data scarcity. However, this approach largely depends on the quality of prompt initialization, and always exhibits large variability among different runs. Such property makes prompt tuning highly unreliable and vulnerable to poorly constructed prompts, which limits its extension to more real-world applications. To tackle this issue, we propose to treat the hard prompt and soft prompt as separate inputs to mitigate noise brought by the prompt initialization. Furthermore, we optimize soft prompts with contrastive learning for utilizing class-aware information in the training process to maintain model performance. Experimental results demonstrate that \sysname outperforms state-of-the-art methods by 7.20% in accuracy and reduces the standard deviation by 2.02 on average. Furthermore, extensive experiments underscore its robustness and stability across 7 datasets covering various tasks.

Title: Reliable or Deceptive? Investigating Gated Features for Smooth Visual Explanations in CNNs

• Authors: Soham Mitra, Atri Sukul, Swalpa Kumar Roy, Pravendra Singh, Vinay Verma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19341
• Pdf URL: https://arxiv.org/pdf/2404.19341
• Copy Paste: [[2404.19341]] Reliable or Deceptive? Investigating Gated Features for Smooth Visual Explanations in CNNs(https://arxiv.org/abs/2404.19341)
• Keywords: fair, interpretability, explainability
• Abstract: Deep learning models have achieved remarkable success across diverse domains. However, the intricate nature of these models often impedes a clear understanding of their decision-making processes. This is where Explainable AI (XAI) becomes indispensable, offering intuitive explanations for model decisions. In this work, we propose a simple yet highly effective approach, ScoreCAM++, which introduces modifications to enhance the promising ScoreCAM method for visual explainability. Our proposed approach involves altering the normalization function within the activation layer utilized in ScoreCAM, resulting in significantly improved results compared to previous efforts. Additionally, we apply an activation function to the upsampled activation layers to enhance interpretability. This improvement is achieved by selectively gating lower-priority values within the activation layer. Through extensive experiments and qualitative comparisons, we demonstrate that ScoreCAM++ consistently achieves notably superior performance and fairness in interpreting the decision-making process compared to both ScoreCAM and previous methods.

Title: Evaluating Lexicon Incorporation for Depression Symptom Estimation

• Authors: Kirill Milintsevich, Gaël Dias, Kairit Sirts
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19359
• Pdf URL: https://arxiv.org/pdf/2404.19359
• Copy Paste: [[2404.19359]] Evaluating Lexicon Incorporation for Depression Symptom Estimation(https://arxiv.org/abs/2404.19359)
• Keywords: transformer
• Abstract: This paper explores the impact of incorporating sentiment, emotion, and domain-specific lexicons into a transformer-based model for depression symptom estimation. Lexicon information is added by marking the words in the input transcripts of patient-therapist conversations as well as in social media posts. Overall results show that the introduction of external knowledge within pre-trained language models can be beneficial for prediction performance, while different lexicons show distinct behaviours depending on the targeted task. Additionally, new state-of-the-art results are obtained for the estimation of depression level over patient-therapist interviews.

Title: Large Language Model Informed Patent Image Retrieval

• Authors: Hao-Cheng Lo, Jung-Mei Chu, Jieh Hsiang, Chun-Chieh Cho
• Subjects: cs.CV, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.19360
• Pdf URL: https://arxiv.org/pdf/2404.19360
• Copy Paste: [[2404.19360]] Large Language Model Informed Patent Image Retrieval(https://arxiv.org/abs/2404.19360)
• Keywords: large language model
• Abstract: In patent prosecution, image-based retrieval systems for identifying similarities between current patent images and prior art are pivotal to ensure the novelty and non-obviousness of patent applications. Despite their growing popularity in recent years, existing attempts, while effective at recognizing images within the same patent, fail to deliver practical value due to their limited generalizability in retrieving relevant prior art. Moreover, this task inherently involves the challenges posed by the abstract visual features of patent images, the skewed distribution of image classifications, and the semantic information of image descriptions. Therefore, we propose a language-informed, distribution-aware multimodal approach to patent image feature learning, which enriches the semantic understanding of patent image by integrating Large Language Models and improves the performance of underrepresented classes with our proposed distribution-aware contrastive losses. Extensive experiments on DeepPatent2 dataset show that our proposed method achieves state-of-the-art or comparable performance in image-based patent retrieval with mAP +53.3%, Recall@10 +41.8%, and MRR@10 +51.9%. Furthermore, through an in-depth user analysis, we explore our model in aiding patent professionals in their image retrieval efforts, highlighting the model's real-world applicability and effectiveness.

Title: Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models

• Authors: Yunhao Zhang, Shaonan Wang, Xinyi Dong, Jiajun Yu, Chengqing Zong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19364
• Pdf URL: https://arxiv.org/pdf/2404.19364
• Copy Paste: [[2404.19364]] Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models(https://arxiv.org/abs/2404.19364)
• Keywords: robust
• Abstract: Neural language models, particularly large-scale ones, have been consistently proven to be most effective in predicting brain neural activity across a range of studies. However, previous research overlooked the comparison of these models with psychologically plausible ones. Moreover, evaluations were reliant on limited, single-modality, and English cognitive datasets. To address these questions, we conducted an analysis comparing encoding performance of various neural language models and psychologically plausible models. Our study utilized extensive multi-modal cognitive datasets, examining bilingual word and discourse levels. Surprisingly, our findings revealed that psychologically plausible models outperformed neural language models across diverse contexts, encompassing different modalities such as fMRI and eye-tracking, and spanning languages from English to Chinese. Among psychologically plausible models, the one incorporating embodied information emerged as particularly exceptional. This model demonstrated superior performance at both word and discourse levels, exhibiting robust prediction of brain activation across numerous regions in both English and Chinese.

Title: Evaluating Telugu Proficiency in Large Language Models_ A Comparative Analysis of ChatGPT and Gemini

• Authors: Katikela Sreeharsha Kishore, Rahimanuddin Shaik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19369
• Pdf URL: https://arxiv.org/pdf/2404.19369
• Copy Paste: [[2404.19369]] Evaluating Telugu Proficiency in Large Language Models_ A Comparative Analysis of ChatGPT and Gemini(https://arxiv.org/abs/2404.19369)
• Keywords: large language model
• Abstract: The growing prominence of large language models (LLMs) necessitates the exploration of their capabilities beyond English. This research investigates the Telugu language proficiency of ChatGPT and Gemini, two leading LLMs. Through a designed set of 20 questions encompassing greetings, grammar, vocabulary, common phrases, task completion, and situational reasoning, the study delves into their strengths and weaknesses in handling Telugu. The analysis aims to identify the LLM that demonstrates a deeper understanding of Telugu grammatical structures, possesses a broader vocabulary, and exhibits superior performance in tasks like writing and reasoning. By comparing their ability to comprehend and use everyday Telugu expressions, the research sheds light on their suitability for real-world language interaction. Furthermore, the evaluation of adaptability and reasoning capabilities provides insights into how each LLM leverages Telugu to respond to dynamic situations. This comparative analysis contributes to the ongoing discussion on multilingual capabilities in AI and paves the way for future research in developing LLMs that can seamlessly integrate with Telugu-speaking communities.

Title: Probing Unlearned Diffusion Models: A Transferable Adversarial Attack Perspective

• Authors: Xiaoxuan Han, Songlin Yang, Wei Wang, Yang Li, Jing Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19382
• Pdf URL: https://arxiv.org/pdf/2404.19382
• Copy Paste: [[2404.19382]] Probing Unlearned Diffusion Models: A Transferable Adversarial Attack Perspective(https://arxiv.org/abs/2404.19382)
• Keywords: privacy, attack, robust, diffusion, generative
• Abstract: Advanced text-to-image diffusion models raise safety concerns regarding identity privacy violation, copyright infringement, and Not Safe For Work content generation. Towards this, unlearning methods have been developed to erase these involved concepts from diffusion models. However, these unlearning methods only shift the text-to-image mapping and preserve the visual content within the generative space of diffusion models, leaving a fatal flaw for restoring these erased concepts. This erasure trustworthiness problem needs probe, but previous methods are sub-optimal from two perspectives: (1) Lack of transferability: Some methods operate within a white-box setting, requiring access to the unlearned model. And the learned adversarial input often fails to transfer to other unlearned models for concept restoration; (2) Limited attack: The prompt-level methods struggle to restore narrow concepts from unlearned models, such as celebrity identity. Therefore, this paper aims to leverage the transferability of the adversarial attack to probe the unlearning robustness under a black-box setting. This challenging scenario assumes that the unlearning method is unknown and the unlearned model is inaccessible for optimization, requiring the attack to be capable of transferring across different unlearned models. Specifically, we employ an adversarial search strategy to search for the adversarial embedding which can transfer across different unlearned models. This strategy adopts the original Stable Diffusion model as a surrogate model to iteratively erase and search for embeddings, enabling it to find the embedding that can restore the target concept for different unlearning methods. Extensive experiments demonstrate the transferability of the searched adversarial embedding across several state-of-the-art unlearning methods and its effectiveness for different levels of concepts.

Title: Cross-Block Fine-Grained Semantic Cascade for Skeleton-Based Sports Action Recognition

• Authors: Zhendong Liu, Haifeng Xia, Tong Guo, Libo Sun, Ming Shao, Siyu Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19383
• Pdf URL: https://arxiv.org/pdf/2404.19383
• Copy Paste: [[2404.19383]] Cross-Block Fine-Grained Semantic Cascade for Skeleton-Based Sports Action Recognition(https://arxiv.org/abs/2404.19383)
• Keywords: security
• Abstract: Human action video recognition has recently attracted more attention in applications such as video security and sports posture correction. Popular solutions, including graph convolutional networks (GCNs) that model the human skeleton as a spatiotemporal graph, have proven very effective. GCNs-based methods with stacked blocks usually utilize top-layer semantics for classification/annotation purposes. Although the global features learned through the procedure are suitable for the general classification, they have difficulty capturing fine-grained action change across adjacent frames -- decisive factors in sports actions. In this paper, we propose a novel ``Cross-block Fine-grained Semantic Cascade (CFSC)'' module to overcome this challenge. In summary, the proposed CFSC progressively integrates shallow visual knowledge into high-level blocks to allow networks to focus on action details. In particular, the CFSC module utilizes the GCN feature maps produced at different levels, as well as aggregated features from proceeding levels to consolidate fine-grained features. In addition, a dedicated temporal convolution is applied at each level to learn short-term temporal features, which will be carried over from shallow to deep layers to maximize the leverage of low-level details. This cross-block feature aggregation methodology, capable of mitigating the loss of fine-grained information, has resulted in improved performance. Last, FD-7, a new action recognition dataset for fencing sports, was collected and will be made publicly available. Experimental results and empirical analysis on public benchmarks (FSD-10) and self-collected (FD-7) demonstrate the advantage of our CFSC module on learning discriminative patterns for action classification over others.

Title: CLIP-Mamba: CLIP Pretrained Mamba Models with OOD and Hessian Evaluation

• Authors: Weiquan Huang, Yifei Shen, Yifan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19394
• Pdf URL: https://arxiv.org/pdf/2404.19394
• Copy Paste: [[2404.19394]] CLIP-Mamba: CLIP Pretrained Mamba Models with OOD and Hessian Evaluation(https://arxiv.org/abs/2404.19394)
• Keywords: transformer
• Abstract: State space models and Mamba-based models have been increasingly applied across various domains, achieving state-of-the-art performance. This technical report introduces the first attempt to train a transferable Mamba model utilizing contrastive language-image pretraining (CLIP). We have trained Mamba models of varying sizes and undertaken comprehensive evaluations of these models on 26 zero-shot classification datasets and 16 out-of-distribution (OOD) datasets. Our findings reveal that a Mamba model with 67 million parameters is on par with a 307 million-parameter Vision Transformer (ViT) model in zero-shot classification tasks, highlighting the parameter efficiency of Mamba models. In tests of OOD generalization, Mamba-based models exhibit exceptional performance in conditions of OOD image contrast or when subjected to high-pass filtering. However, a Hessian analysis indicates that Mamba models feature a sharper and more non-convex landscape compared to ViT-based models, making them more challenging to train. The source code is available at https://github.com/raytrun/mamba-clip.

Title: UniFS: Universal Few-shot Instance Perception with Point Representations

• Authors: Sheng Jin, Ruijie Yao, Lumin Xu, Wentao Liu, Chen Qian, Ji Wu, Ping Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19401
• Pdf URL: https://arxiv.org/pdf/2404.19401
• Copy Paste: [[2404.19401]] UniFS: Universal Few-shot Instance Perception with Point Representations(https://arxiv.org/abs/2404.19401)
• Keywords: segmentation
• Abstract: Instance perception tasks (object detection, instance segmentation, pose estimation, counting) play a key role in industrial applications of visual models. As supervised learning methods suffer from high labeling cost, few-shot learning methods which effectively learn from a limited number of labeled examples are desired. Existing few-shot learning methods primarily focus on a restricted set of tasks, presumably due to the challenges involved in designing a generic model capable of representing diverse tasks in a unified manner. In this paper, we propose UniFS, a universal few-shot instance perception model that unifies a wide range of instance perception tasks by reformulating them into a dynamic point representation learning framework. Additionally, we propose Structure-Aware Point Learning (SAPL) to exploit the higher-order structural relationship among points to further enhance representation learning. Our approach makes minimal assumptions about the tasks, yet it achieves competitive results compared to highly specialized and well optimized specialist models. Codes will be released soon.

Title: Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning

• Authors: Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, Olivier Pietquin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19409
• Pdf URL: https://arxiv.org/pdf/2404.19409
• Copy Paste: [[2404.19409]] Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning(https://arxiv.org/abs/2404.19409)
• Keywords: large language model
• Abstract: While Reinforcement Learning (RL) has been proven essential for tuning large language models (LLMs), it can lead to reward over-optimization (ROO). Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. Additionally, KL regularization focuses solely on regularizing the language policy, neglecting a potential source of regularization: the reward function itself. Inspired by demonstration-guided RL, we here introduce the Reward Calibration from Demonstration (RCfD), which leverages human demonstrations and a reward model to recalibrate the reward objective. Formally, given a prompt, the RCfD objective minimizes the distance between the demonstrations' and LLM's rewards rather than directly maximizing the reward function. This objective shift avoids incentivizing the LLM to exploit the reward model and promotes more natural and diverse language generation. We show the effectiveness of RCfD on three language tasks, which achieves comparable performance to carefully tuned baselines while mitigating ROO.

Title: Physical Backdoor: Towards Temperature-based Backdoor Attacks in the Physical World

• Authors: Wen Yin, Jian Lou, Pan Zhou, Yulai Xie, Dan Feng, Yuhua Sun, Tailai Zhang, Lichao Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19417
• Pdf URL: https://arxiv.org/pdf/2404.19417
• Copy Paste: [[2404.19417]] Physical Backdoor: Towards Temperature-based Backdoor Attacks in the Physical World(https://arxiv.org/abs/2404.19417)
• Keywords: security, attack
• Abstract: Backdoor attacks have been well-studied in visible light object detection (VLOD) in recent years. However, VLOD can not effectively work in dark and temperature-sensitive scenarios. Instead, thermal infrared object detection (TIOD) is the most accessible and practical in such environments. In this paper, our team is the first to investigate the security vulnerabilities associated with TIOD in the context of backdoor attacks, spanning both the digital and physical realms. We introduce two novel types of backdoor attacks on TIOD, each offering unique capabilities: Object-affecting Attack and Range-affecting Attack. We conduct a comprehensive analysis of key factors influencing trigger design, which include temperature, size, material, and concealment. These factors, especially temperature, significantly impact the efficacy of backdoor attacks on TIOD. A thorough understanding of these factors will serve as a foundation for designing physical triggers and temperature controlling experiments. Our study includes extensive experiments conducted in both digital and physical environments. In the digital realm, we evaluate our approach using benchmark datasets for TIOD, achieving an Attack Success Rate (ASR) of up to 98.21%. In the physical realm, we test our approach in two real-world settings: a traffic intersection and a parking lot, using a thermal infrared camera. Here, we attain an ASR of up to 98.38%.

Title: Energy Cyber Attacks to Smart Healthcare Devices: A Testbed

• Authors: Zainab Alwaisi, Simone Soderi, Rocco De Nicola
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19418
• Pdf URL: https://arxiv.org/pdf/2404.19418
• Copy Paste: [[2404.19418]] Energy Cyber Attacks to Smart Healthcare Devices: A Testbed(https://arxiv.org/abs/2404.19418)
• Keywords: security, defense, attack
• Abstract: The Internet of Things (IoT) has garnered significant interest in both research and industry due to its profound impact on human life. The rapid expansion of IoT technology has ushered in smart healthcare, smart devices, smart cities, and smart grids. However, the security of IoT devices, particularly in healthcare, has become a major concern, with recent attacks revealing serious vulnerabilities. In IoT networks, where connected devices are susceptible to resource-constraint attacks, such as energy consumption attacks, security is paramount. This paper explores the impact of Distributed Denial of Service (DDoS) and Fake Access Points (F-APs) attacks on WiFi-enabled smart healthcare devices. Specifically, it investigates how these attacks can disrupt service on victim devices and Access Points (APs), focusing on device connectivity and energy consumption during attacks. Key findings include identifying the attack rates of DDoS attacks that disrupt services and quantifying the energy consumption impact of Energy Consumption Distributed Denial of Service (EC-DDoS) and F-APs attacks on smart healthcare devices. The study highlights communication protocols, attack rates, payload sizes, and port states of victim devices as critical factors influencing energy consumption. These insights provide a comprehensive understanding of IoT device vulnerabilities in smart healthcare environments and lay the groundwork for future defense strategies.

Title: Let's Focus: Focused Backdoor Attack against Federated Transfer Learning

• Authors: Marco Arazzi, Stefanos Koffas, Antonino Nocera, Stjepan Picek
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19420
• Pdf URL: https://arxiv.org/pdf/2404.19420
• Copy Paste: [[2404.19420]] Let's Focus: Focused Backdoor Attack against Federated Transfer Learning(https://arxiv.org/abs/2404.19420)
• Keywords: defense, attack, federate
• Abstract: Federated Transfer Learning (FTL) is the most general variation of Federated Learning. According to this distributed paradigm, a feature learning pre-step is commonly carried out by only one party, typically the server, on publicly shared data. After that, the Federated Learning phase takes place to train a classifier collaboratively using the learned feature extractor. Each involved client contributes by locally training only the classification layers on a private training set. The peculiarity of an FTL scenario makes it hard to understand whether poisoning attacks can be developed to craft an effective backdoor. State-of-the-art attack strategies assume the possibility of shifting the model attention toward relevant features introduced by a forged trigger injected in the input data by some untrusted clients. Of course, this is not feasible in FTL, as the learned features are fixed once the server performs the pre-training step. Consequently, in this paper, we investigate this intriguing Federated Learning scenario to identify and exploit a vulnerability obtained by combining eXplainable AI (XAI) and dataset distillation. In particular, the proposed attack can be carried out by one of the clients during the Federated Learning phase of FTL by identifying the optimal local for the trigger through XAI and encapsulating compressed information of the backdoor class. Due to its behavior, we refer to our approach as a focused backdoor approach (FB-FTL for short) and test its performance by explicitly referencing an image classification scenario. With an average 80% attack success rate, obtained results show the effectiveness of our attack also against existing defenses for Federated Learning.

Title: Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationships

• Authors: D. Panas, S. Seth, V. Belle
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19432
• Pdf URL: https://arxiv.org/pdf/2404.19432
• Copy Paste: [[2404.19432]] Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationships(https://arxiv.org/abs/2404.19432)
• Keywords: large language model
• Abstract: Two major areas of interest in the era of Large Language Models regard questions of what do LLMs know, and if and how they may be able to reason (or rather, approximately reason). Since to date these lines of work progressed largely in parallel (with notable exceptions), we are interested in investigating the intersection: probing for reasoning about the implicitly-held knowledge. Suspecting the performance to be lacking in this area, we use a very simple set-up of comparisons between cardinalities associated with elements of various subjects (e.g. the number of legs a bird has versus the number of wheels on a tricycle). We empirically demonstrate that although LLMs make steady progress in knowledge acquisition and (pseudo)reasoning with each new GPT release, their capabilities are limited to statistical inference only. It is difficult to argue that pure statistical learning can cope with the combinatorial explosion inherent in many commonsense reasoning tasks, especially once arithmetical notions are involved. Further, we argue that bigger is not always better and chasing purely statistical improvements is flawed at the core, since it only exacerbates the dangerous conflation of the production of correct answers with genuine reasoning ability.

Title: Detection of Energy Consumption Cyber Attacks on Smart Devices

• Authors: Zainab Alwaisi, Simone Soderi, Rocco De Nicola
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19434
• Pdf URL: https://arxiv.org/pdf/2404.19434
• Copy Paste: [[2404.19434]] Detection of Energy Consumption Cyber Attacks on Smart Devices(https://arxiv.org/abs/2404.19434)
• Keywords: security, attack
• Abstract: With the rapid development of Internet of Things (IoT) technology, intelligent systems are increasingly integrating into everyday life and people's homes. However, the proliferation of these technologies raises concerns about the security of smart home devices. These devices often face resource constraints and may connect to unreliable networks, posing risks to the data they handle. Securing IoT technology is crucial due to the sensitive data involved. Preventing energy attacks and ensuring the security of IoT infrastructure are key challenges in modern smart homes. Monitoring energy consumption can be an effective approach to detecting abnormal behavior and IoT cyberattacks. Lightweight algorithms are necessary to accommodate the resource limitations of IoT devices. This paper presents a lightweight technique for detecting energy consumption attacks on smart home devices by analyzing received packets. The proposed algorithm considers TCP, UDP, and MQTT protocols, as well as device statuses (Idle, active, under attack). It accounts for resource constraints and promptly alerts administrators upon detecting an attack. The proposed approach effectively identifies energy consumption attacks by measuring packet reception rates for different protocols.

Title: Which Nigerian-Pidgin does Generative AI speak?: Issues about Representativeness and Bias for Multilingual and Low Resource Languages

• Authors: David Ifeoluwa Adelani, A. Seza Doğruöz, Iyanuoluwa Shode, Anuoluwapo Aremu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19442
• Pdf URL: https://arxiv.org/pdf/2404.19442
• Copy Paste: [[2404.19442]] Which Nigerian-Pidgin does Generative AI speak?: Issues about Representativeness and Bias for Multilingual and Low Resource Languages(https://arxiv.org/abs/2404.19442)
• Keywords: generative
• Abstract: Naija is the Nigerian-Pidgin spoken by approx. 120M speakers in Nigeria and it is a mixed language (e.g., English, Portuguese and Indigenous languages). Although it has mainly been a spoken language until recently, there are currently two written genres (BBC and Wikipedia) in Naija. Through statistical analyses and Machine Translation experiments, we prove that these two genres do not represent each other (i.e., there are linguistic differences in word order and vocabulary) and Generative AI operates only based on Naija written in the BBC genre. In other words, Naija written in Wikipedia genre is not represented in Generative AI.

Title: AnomalyXFusion: Multi-modal Anomaly Synthesis with Diffusion

• Authors: Jie Hu, Yawen Huang, Yilin Lu, Guoyang Xie, Guannan Jiang, Yefeng Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19444
• Pdf URL: https://arxiv.org/pdf/2404.19444
• Copy Paste: [[2404.19444]] AnomalyXFusion: Multi-modal Anomaly Synthesis with Diffusion(https://arxiv.org/abs/2404.19444)
• Keywords: diffusion
• Abstract: Anomaly synthesis is one of the effective methods to augment abnormal samples for training. However, current anomaly synthesis methods predominantly rely on texture information as input, which limits the fidelity of synthesized abnormal samples. Because texture information is insufficient to correctly depict the pattern of anomalies, especially for logical anomalies. To surmount this obstacle, we present the AnomalyXFusion framework, designed to harness multi-modality information to enhance the quality of synthesized abnormal samples. The AnomalyXFusion framework comprises two distinct yet synergistic modules: the Multi-modal In-Fusion (MIF) module and the Dynamic Dif-Fusion (DDF) module. The MIF module refines modality alignment by aggregating and integrating various modality features into a unified embedding space, termed X-embedding, which includes image, text, and mask features. Concurrently, the DDF module facilitates controlled generation through an adaptive adjustment of X-embedding conditioned on the diffusion steps. In addition, to reveal the multi-modality representational power of AnomalyXFusion, we propose a new dataset, called MVTec Caption. More precisely, MVTec Caption extends 2.2k accurate image-mask-text annotations for the MVTec AD and LOCO datasets. Comprehensive evaluations demonstrate the effectiveness of AnomalyXFusion, especially regarding the fidelity and diversity for logical anomalies. Project page: http:github.com/hujiecpp/MVTec-Caption

Title: AttackBench: Evaluating Gradient-based Attacks for Adversarial Examples

• Authors: Antonio Emanuele Cinà, Jérôme Rony, Maura Pintor, Luca Demetrio, Ambra Demontis, Battista Biggio, Ismail Ben Ayed, Fabio Roli
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19460
• Pdf URL: https://arxiv.org/pdf/2404.19460
• Copy Paste: [[2404.19460]] AttackBench: Evaluating Gradient-based Attacks for Adversarial Examples(https://arxiv.org/abs/2404.19460)
• Keywords: attack, fair
• Abstract: Adversarial examples are typically optimized with gradient-based attacks. While novel attacks are continuously proposed, each is shown to outperform its predecessors using different experimental setups, hyperparameter settings, and number of forward and backward calls to the target models. This provides overly-optimistic and even biased evaluations that may unfairly favor one particular attack over the others. In this work, we aim to overcome these limitations by proposing AttackBench, i.e., the first evaluation framework that enables a fair comparison among different attacks. To this end, we first propose a categorization of gradient-based attacks, identifying their main components and differences. We then introduce our framework, which evaluates their effectiveness and efficiency. We measure these characteristics by (i) defining an optimality metric that quantifies how close an attack is to the optimal solution, and (ii) limiting the number of forward and backward queries to the model, such that all attacks are compared within a given maximum query budget. Our extensive experimental analysis compares more than 100 attack implementations with a total of over 800 different configurations against CIFAR-10 and ImageNet models, highlighting that only very few attacks outperform all the competing approaches. Within this analysis, we shed light on several implementation issues that prevent many attacks from finding better solutions or running at all. We release AttackBench as a publicly available benchmark, aiming to continuously update it to include and evaluate novel gradient-based attacks for optimizing adversarial examples.

Title: TwinDiffusion: Enhancing Coherence and Efficiency in Panoramic Image Generation with Diffusion Models

• Authors: Teng Zhou, Yongchuan Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19475
• Pdf URL: https://arxiv.org/pdf/2404.19475
• Copy Paste: [[2404.19475]] TwinDiffusion: Enhancing Coherence and Efficiency in Panoramic Image Generation with Diffusion Models(https://arxiv.org/abs/2404.19475)
• Keywords: diffusion
• Abstract: Diffusion models have emerged as effective tools for generating diverse and high-quality content. However, their capability in high-resolution image generation, particularly for panoramic images, still faces challenges such as visible seams and incoherent transitions. In this paper, we propose TwinDiffusion, an optimized framework designed to address these challenges through two key innovations: Crop Fusion for quality enhancement and Cross Sampling for efficiency optimization. We introduce a training-free optimizing stage to refine the similarity of the adjacent image areas, as well as an interleaving sampling strategy to yield dynamic patches during the cropping process. A comprehensive evaluation is conducted to compare TwinDiffusion with the existing methods, considering factors including coherence, fidelity, compatibility, and efficiency. The results demonstrate the superior performance of our approach in generating seamless and coherent panoramas, setting a new standard in quality and efficiency for panoramic image generation.

Title: Mitigating and Analysis of Memory Usage Attack in IoE System

• Authors: Zainab Alwaisi, Simone Soderi, Rocco De Nicola
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19480
• Pdf URL: https://arxiv.org/pdf/2404.19480
• Copy Paste: [[2404.19480]] Mitigating and Analysis of Memory Usage Attack in IoE System(https://arxiv.org/abs/2404.19480)
• Keywords: security, privacy, protect, attack
• Abstract: Internet of Everything (IoE) is a newly emerging trend, especially in homes. Marketing forces toward smart homes are also accelerating the spread of IoE devices in households. An obvious risk of the rapid adoption of these smart devices is that many lack controls for protecting the privacy and security of end users from attacks designed to disrupt lives and incur financial losses. Today the smart home is a system for managing the basic life support processes of both small systems, e.g., commercial, office premises, apartments, cottages, and largely automated complexes, e.g., commercial and industrial complexes. One of the critical tasks to be solved by the concept of a modern smart home is the problem of preventing the usage of IoE resources. Recently, there has been a rapid increase in attacks on consumer IoE devices. Memory corruption vulnerabilities constitute a significant class of vulnerabilities in software security through which attackers can gain control of an entire system. Numerous memory corruption vulnerabilities have been found in IoE firmware already deployed in the consumer market. This paper aims to analyze and explain the resource usage attack and create a low-cost simulation environment to aid in the dynamic analysis of the attack. Further, we perform controlled resource usage attacks while measuring resource consumption on resource-constrained victims' IoE devices, such as CPU and memory utilization. We also build a lightweight algorithm to detect memory usage attacks in the IoE environment. The result shows high efficiency in detecting and mitigating memory usage attacks by detecting when the intruder starts and stops the attack.

Title: FactCheck Editor: Multilingual Text Editor with End-to-End fact-checking

• Authors: Vinay Setty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19482
• Pdf URL: https://arxiv.org/pdf/2404.19482
• Copy Paste: [[2404.19482]] FactCheck Editor: Multilingual Text Editor with End-to-End fact-checking(https://arxiv.org/abs/2404.19482)
• Keywords: transformer
• Abstract: We introduce 'FactCheck Editor', an advanced text editor designed to automate fact-checking and correct factual inaccuracies. Given the widespread issue of misinformation, often a result of unintentional mistakes by content creators, our tool aims to address this challenge. It supports over 90 languages and utilizes transformer models to assist humans in the labor-intensive process of fact verification. This demonstration showcases a complete workflow that detects text claims in need of verification, generates relevant search engine queries, and retrieves appropriate documents from the web. It employs Natural Language Inference (NLI) to predict the veracity of claims and uses LLMs to summarize the evidence and suggest textual revisions to correct any errors in the text. Additionally, the effectiveness of models used in claim detection and veracity assessment is evaluated across multiple languages.

Title: More Compute Is What You Need

• Authors: Zhen Guo
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19484
• Pdf URL: https://arxiv.org/pdf/2404.19484
• Copy Paste: [[2404.19484]] More Compute Is What You Need(https://arxiv.org/abs/2404.19484)
• Keywords: transformer, large language model
• Abstract: Large language model pre-training has become increasingly expensive, with most practitioners relying on scaling laws to allocate compute budgets for model size and training tokens, commonly referred to as Compute-Optimal or Chinchilla Optimal. In this paper, we hypothesize a new scaling law that suggests model performance depends mostly on the amount of compute spent for transformer-based models, independent of the specific allocation to model size and dataset size. Using this unified scaling law, we predict that (a) for inference efficiency, training should prioritize smaller model sizes and larger training datasets, and (b) assuming the exhaustion of available web datasets, scaling the model size might be the only way to further improve model performance.

Title: Safe Training with Sensitive In-domain Data: Leveraging Data Fragmentation To Mitigate Linkage Attacks

• Authors: Mariia Ignashina, Julia Ive
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.19486
• Pdf URL: https://arxiv.org/pdf/2404.19486
• Copy Paste: [[2404.19486]] Safe Training with Sensitive In-domain Data: Leveraging Data Fragmentation To Mitigate Linkage Attacks(https://arxiv.org/abs/2404.19486)
• Keywords: protect, attack
• Abstract: Current text generation models are trained using real data which can potentially contain sensitive information, such as confidential patient information and the like. Under certain conditions output of the training data which they have memorised can be triggered, exposing sensitive data. To mitigate against this risk we propose a safer alternative which sees fragmented data in the form of domain-specific short phrases randomly grouped together shared instead of full texts. Thus, text fragments that could re-identify an individual cannot be reproduced by the model in one sequence, giving significant protection against linkage attacks. We fine-tune several state-of-the-art LLMs using meaningful syntactic chunks to explore their utility. In particular, we fine-tune BERT-based models to predict two cardiovascular diagnoses. Our results demonstrate the capacity of LLMs to benefit from the pre-trained knowledge and deliver classification results when fine-tuned with fragmented data comparable to fine-tuning with full training data.

Title: Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom

• Authors: Shisen Yue, Siyuan Song, Xinyuan Cheng, Hai Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19509
• Pdf URL: https://arxiv.org/pdf/2404.19509
• Copy Paste: [[2404.19509]] Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom(https://arxiv.org/abs/2404.19509)
• Keywords: large language model
• Abstract: Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom $\textit{My Own Swordsman}$. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT-3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs' performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at https://github.com/sjtu-compling/llm-pragmatics.

Title: Generating Robust Counterfactual Witnesses for Graph Neural Networks

• Authors: Dazhuo Qiu, Mengying Wang, Arijit Khan, Yinghui Wu
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2404.19519
• Pdf URL: https://arxiv.org/pdf/2404.19519
• Copy Paste: [[2404.19519]] Generating Robust Counterfactual Witnesses for Graph Neural Networks(https://arxiv.org/abs/2404.19519)
• Keywords: robust
• Abstract: This paper introduces a new class of explanation structures, called robust counterfactual witnesses (RCWs), to provide robust, both counterfactual and factual explanations for graph neural networks. Given a graph neural network M, a robust counterfactual witness refers to the fraction of a graph G that are counterfactual and factual explanation of the results of M over G, but also remains so for any "disturbed" G by flipping up to k of its node pairs. We establish the hardness results, from tractable results to co-NP-hardness, for verifying and generating robust counterfactual witnesses. We study such structures for GNN-based node classification, and present efficient algorithms to verify and generate RCWs. We also provide a parallel algorithm to verify and generate RCWs for large graphs with scalability guarantees. We experimentally verify our explanation generation process for benchmark datasets, and showcase their applications.

Title: MicroDreamer: Zero-shot 3D Generation in $\sim$20 Seconds by Score-based Iterative Reconstruction

• Authors: Luxi Chen, Zhengyi Wang, Chongxuan Li, Tingting Gao, Hang Su, Jun Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19525
• Pdf URL: https://arxiv.org/pdf/2404.19525
• Copy Paste: [[2404.19525]] MicroDreamer: Zero-shot 3D Generation in $\sim$20 Seconds by Score-based Iterative Reconstruction(https://arxiv.org/abs/2404.19525)
• Keywords: diffusion
• Abstract: Optimization-based approaches, such as score distillation sampling (SDS), show promise in zero-shot 3D generation but suffer from low efficiency, primarily due to the high number of function evaluations (NFEs) required for each sample. In this paper, we introduce score-based iterative reconstruction (SIR), an efficient and general algorithm for 3D generation with a multi-view score-based diffusion model. Given the images produced by the diffusion model, SIR reduces NFEs by repeatedly optimizing 3D parameters, unlike the single optimization in SDS, mimicking the 3D reconstruction process. With other improvements including optimization in the pixel space, we present an efficient approach called MicroDreamer that generally applies to various 3D representations and 3D generation tasks. In particular, retaining a comparable performance, MicroDreamer is 5-20 times faster than SDS in generating neural radiance field and takes about 20 seconds to generate meshes from 3D Gaussian splitting on a single A100 GPU, halving the time of the fastest zero-shot baseline, DreamGaussian. Our code is available at https://github.com/ML-GSAI/MicroDreamer.

Title: MoST: Multi-modality Scene Tokenization for Motion Prediction

• Authors: Norman Mu, Jingwei Ji, Zhenpei Yang, Nate Harada, Haotian Tang, Kan Chen, Charles R. Qi, Runzhou Ge, Kratarth Goel, Zoey Yang, Scott Ettinger, Rami Al-Rfou, Dragomir Anguelov, Yin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19531
• Pdf URL: https://arxiv.org/pdf/2404.19531
• Copy Paste: [[2404.19531]] MoST: Multi-modality Scene Tokenization for Motion Prediction(https://arxiv.org/abs/2404.19531)
• Keywords: interpretability, transformer
• Abstract: Many existing motion prediction approaches rely on symbolic perception outputs to generate agent trajectories, such as bounding boxes, road graph information and traffic lights. This symbolic representation is a high-level abstraction of the real world, which may render the motion prediction model vulnerable to perception errors (e.g., failures in detecting open-vocabulary obstacles) while missing salient information from the scene context (e.g., poor road conditions). An alternative paradigm is end-to-end learning from raw sensors. However, this approach suffers from the lack of interpretability and requires significantly more training resources. In this work, we propose tokenizing the visual world into a compact set of scene elements and then leveraging pre-trained image foundation models and LiDAR neural networks to encode all the scene elements in an open-vocabulary manner. The image foundation model enables our scene tokens to encode the general knowledge of the open world while the LiDAR neural network encodes geometry information. Our proposed representation can efficiently encode the multi-frame multi-modality observations with a few hundred tokens and is compatible with most transformer-based architectures. To evaluate our method, we have augmented Waymo Open Motion Dataset with camera embeddings. Experiments over Waymo Open Motion Dataset show that our approach leads to significant performance improvements over the state-of-the-art.

Title: RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing

• Authors: Yucheng Hu, Yuxing Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19543
• Pdf URL: https://arxiv.org/pdf/2404.19543
• Copy Paste: [[2404.19543]] RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing(https://arxiv.org/abs/2404.19543)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have catalyzed significant advancements in Natural Language Processing (NLP), yet they encounter challenges such as hallucination and the need for domain-specific knowledge. To mitigate these, recent methodologies have integrated information retrieved from external resources with LLMs, substantially enhancing their performance across NLP tasks. This survey paper addresses the absence of a comprehensive overview on Retrieval-Augmented Language Models (RALMs), both Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Understanding (RAU), providing an in-depth examination of their paradigm, evolution, taxonomy, and applications. The paper discusses the essential components of RALMs, including Retrievers, Language Models, and Augmentations, and how their interactions lead to diverse model structures and applications. RALMs demonstrate utility in a spectrum of tasks, from translation and dialogue systems to knowledge-intensive applications. The survey includes several evaluation methods of RALMs, emphasizing the importance of robustness, accuracy, and relevance in their assessment. It also acknowledges the limitations of RALMs, particularly in retrieval quality and computational efficiency, offering directions for future research. In conclusion, this survey aims to offer a structured insight into RALMs, their potential, and the avenues for their future development in NLP. The paper is supplemented with a Github Repository containing the surveyed works and resources for further study: https://github.com/2471023025/RALM_Survey.

Title: Causal Perception Inspired Representation Learning for Trustworthy Image Quality Assessment

• Authors: Lei Wang, Desen Yuan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.19567
• Pdf URL: https://arxiv.org/pdf/2404.19567
• Copy Paste: [[2404.19567]] Causal Perception Inspired Representation Learning for Trustworthy Image Quality Assessment(https://arxiv.org/abs/2404.19567)
• Keywords: defense, attack, robust
• Abstract: Despite great success in modeling visual perception, deep neural network based image quality assessment (IQA) still remains unreliable in real-world applications due to its vulnerability to adversarial perturbations and the inexplicit black-box structure. In this paper, we propose to build a trustworthy IQA model via Causal Perception inspired Representation Learning (CPRL), and a score reflection attack method for IQA model. More specifically, we assume that each image is composed of Causal Perception Representation (CPR) and non-causal perception representation (N-CPR). CPR serves as the causation of the subjective quality label, which is invariant to the imperceptible adversarial perturbations. Inversely, N-CPR presents spurious associations with the subjective quality label, which may significantly change with the adversarial perturbations. To extract the CPR from each input image, we develop a soft ranking based channel-wise activation function to mediate the causally sufficient (beneficial for high prediction accuracy) and necessary (beneficial for high robustness) deep features, and based on intervention employ minimax game to optimize. Experiments on four benchmark databases show that the proposed CPRL method outperforms many state-of-the-art adversarial defense methods and provides explicit model interpretation.

Title: Leveraging Label Information for Stealthy Data Stealing in Vertical Federated Learning

• Authors: Duanyi Yao, Songze Li, Xueluan Gong, Sizai Hou, Gaoning Pan
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19582
• Pdf URL: https://arxiv.org/pdf/2404.19582
• Copy Paste: [[2404.19582]] Leveraging Label Information for Stealthy Data Stealing in Vertical Federated Learning(https://arxiv.org/abs/2404.19582)
• Keywords: defense, attack, robust, steal, federate
• Abstract: We develop DMAVFL, a novel attack strategy that evades current detection mechanisms. The key idea is to integrate a discriminator with auxiliary classifier that takes a full advantage of the label information (which was completely ignored in previous attacks): on one hand, label information helps to better characterize embeddings of samples from distinct classes, yielding an improved reconstruction performance; on the other hand, computing malicious gradients with label information better mimics the honest training, making the malicious gradients indistinguishable from the honest ones, and the attack much more stealthy. Our comprehensive experiments demonstrate that DMAVFL significantly outperforms existing attacks, and successfully circumvents SOTA defenses for malicious attacks. Additional ablation studies and evaluations on other defenses further underscore the robustness and effectiveness of DMAVFL.

Title: AI techniques for near real-time monitoring of contaminants in coastal waters on board future Phisat-2 mission

• Authors: Francesca Razzano, Pietro Di Stasio, Francesco Mauro, Gabriele Meoni, Marco Esposito, Gilda Schirinzi, Silvia L. Ullo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19586
• Pdf URL: https://arxiv.org/pdf/2404.19586
• Copy Paste: [[2404.19586]] AI techniques for near real-time monitoring of contaminants in coastal waters on board future Phisat-2 mission(https://arxiv.org/abs/2404.19586)
• Keywords: protect
• Abstract: Differently from conventional procedures, the proposed solution advocates for a groundbreaking paradigm in water quality monitoring through the integration of satellite Remote Sensing (RS) data, Artificial Intelligence (AI) techniques, and onboard processing. The objective is to offer nearly real-time detection of contaminants in coastal waters addressing a significant gap in the existing literature. Moreover, the expected outcomes include substantial advancements in environmental monitoring, public health protection, and resource conservation. The specific focus of our study is on the estimation of Turbidity and pH parameters, for their implications on human and aquatic health. Nevertheless, the designed framework can be extended to include other parameters of interest in the water environment and beyond. Originating from our participation in the European Space Agency (ESA) OrbitalAI Challenge, this article describes the distinctive opportunities and issues for the contaminants monitoring on the Phisat-2 mission. The specific characteristics of this mission, with the tools made available, will be presented, with the methodology proposed by the authors for the onboard monitoring of water contaminants in near real-time. Preliminary promising results are discussed and in progress and future work introduced.

Title: Transferring Troubles: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning

• Authors: Xuanli He, Jun Wang, Qiongkai Xu, Pasquale Minervini, Pontus Stenetorp, Benjamin I. P. Rubinstein, Trevor Cohn
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19597
• Pdf URL: https://arxiv.org/pdf/2404.19597
• Copy Paste: [[2404.19597]] Transferring Troubles: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning(https://arxiv.org/abs/2404.19597)
• Keywords: security, attack, large language model
• Abstract: The implications of backdoor attacks on English-centric large language models (LLMs) have been widely examined - such attacks can be achieved by embedding malicious behaviors during training and activated under specific conditions that trigger malicious outputs. However, the impact of backdoor attacks on multilingual models remains under-explored. Our research focuses on cross-lingual backdoor attacks against multilingual LLMs, particularly investigating how poisoning the instruction-tuning data in one or two languages can affect the outputs in languages whose instruction-tuning data was not poisoned. Despite its simplicity, our empirical analysis reveals that our method exhibits remarkable efficacy in models like mT5, BLOOM, and GPT-3.5-turbo, with high attack success rates, surpassing 95% in several languages across various scenarios. Alarmingly, our findings also indicate that larger models show increased susceptibility to transferable cross-lingual backdoor attacks, which also applies to LLMs predominantly pre-trained on English data, such as Llama2, Llama3, and Gemma. Moreover, our experiments show that triggers can still work even after paraphrasing, and the backdoor mechanism proves highly effective in cross-lingual response settings across 25 languages, achieving an average attack success rate of 50%. Our study aims to highlight the vulnerabilities and significant security risks present in current multilingual LLMs, underscoring the emergent need for targeted security measures.

Title: Seeing Through the Clouds: Cloud Gap Imputation with Prithvi Foundation Model

• Authors: Denys Godwin, Hanxi Li, Michael Cecil, Hamed Alemohammad
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.19609
• Pdf URL: https://arxiv.org/pdf/2404.19609
• Copy Paste: [[2404.19609]] Seeing Through the Clouds: Cloud Gap Imputation with Prithvi Foundation Model(https://arxiv.org/abs/2404.19609)
• Keywords: transformer, generative
• Abstract: Filling cloudy pixels in multispectral satellite imagery is essential for accurate data analysis and downstream applications, especially for tasks which require time series data. To address this issue, we compare the performance of a foundational Vision Transformer (ViT) model with a baseline Conditional Generative Adversarial Network (CGAN) model for missing value imputation in time series of multispectral satellite imagery. We randomly mask time series of satellite images using real-world cloud masks and train each model to reconstruct the missing pixels. The ViT model is fine-tuned from a pretrained model, while the CGAN is trained from scratch. Using quantitative evaluation metrics such as structural similarity index and mean absolute error as well as qualitative visual analysis, we assess imputation accuracy and contextual preservation.

Title: Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction

• Authors: Jared D. Willard, Peter Harrington, Shashank Subramanian, Ankur Mahesh, Travis A. O'Brien, William D. Collins
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.19630
• Pdf URL: https://arxiv.org/pdf/2404.19630
• Copy Paste: [[2404.19630]] Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction(https://arxiv.org/abs/2404.19630)
• Keywords: transformer
• Abstract: The rapid rise of deep learning (DL) in numerical weather prediction (NWP) has led to a proliferation of models which forecast atmospheric variables with comparable or superior skill than traditional physics-based NWP. However, among these leading DL models, there is a wide variance in both the training settings and architecture used. Further, the lack of thorough ablation studies makes it hard to discern which components are most critical to success. In this work, we show that it is possible to attain high forecast skill even with relatively off-the-shelf architectures, simple training procedures, and moderate compute budgets. Specifically, we train a minimally modified SwinV2 transformer on ERA5 data, and find that it attains superior forecast skill when compared against IFS. We present some ablations on key aspects of the training pipeline, exploring different loss functions, model sizes and depths, and multi-step fine-tuning to investigate their effect. We also examine the model performance with metrics beyond the typical ACC and RMSE, and investigate how the performance scales with model size.

Title: On Training a Neural Network to Explain Binaries

• Authors: Alexander Interrante-Grant, Andy Davis, Heather Preslier, Tim Leek
• Subjects: cs.LG, cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.19631
• Pdf URL: https://arxiv.org/pdf/2404.19631
• Copy Paste: [[2404.19631]] On Training a Neural Network to Explain Binaries(https://arxiv.org/abs/2404.19631)
• Keywords: generative, large language model
• Abstract: In this work, we begin to investigate the possibility of training a deep neural network on the task of binary code understanding. Specifically, the network would take, as input, features derived directly from binaries and output English descriptions of functionality to aid a reverse engineer in investigating the capabilities of a piece of closed-source software, be it malicious or benign. Given recent success in applying large language models (generative AI) to the task of source code summarization, this seems a promising direction. However, in our initial survey of the available datasets, we found nothing of sufficiently high quality and volume to train these complex models. Instead, we build our own dataset derived from a capture of Stack Overflow containing 1.1M entries. A major result of our work is a novel dataset evaluation method using the correlation between two distances on sample pairs: one distance in the embedding space of inputs and the other in the embedding space of outputs. Intuitively, if two samples have inputs close in the input embedding space, their outputs should also be close in the output embedding space. We found this Embedding Distance Correlation (EDC) test to be highly diagnostic, indicating that our collected dataset and several existing open-source datasets are of low quality as the distances are not well correlated. We proceed to explore the general applicability of EDC, applying it to a number of qualitatively known good datasets and a number of synthetically known bad ones and found it to be a reliable indicator of dataset value.

Title: Attacking Bayes: On the Adversarial Robustness of Bayesian Neural Networks

• Authors: Yunzhen Feng, Tim G. J. Rudner, Nikolaos Tsilivis, Julia Kempe
• Subjects: cs.LG, cs.AI, cs.CV, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.19640
• Pdf URL: https://arxiv.org/pdf/2404.19640
• Copy Paste: [[2404.19640]] Attacking Bayes: On the Adversarial Robustness of Bayesian Neural Networks(https://arxiv.org/abs/2404.19640)
• Keywords: attack, robust
• Abstract: Adversarial examples have been shown to cause neural networks to fail on a wide range of vision and language tasks, but recent work has claimed that Bayesian neural networks (BNNs) are inherently robust to adversarial perturbations. In this work, we examine this claim. To study the adversarial robustness of BNNs, we investigate whether it is possible to successfully break state-of-the-art BNN inference methods and prediction pipelines using even relatively unsophisticated attacks for three tasks: (1) label prediction under the posterior predictive mean, (2) adversarial example detection with Bayesian predictive uncertainty, and (3) semantic shift detection. We find that BNNs trained with state-of-the-art approximate inference methods, and even BNNs trained with Hamiltonian Monte Carlo, are highly susceptible to adversarial attacks. We also identify various conceptual and experimental errors in previous works that claimed inherent adversarial robustness of BNNs and conclusively demonstrate that BNNs and uncertainty-aware Bayesian prediction pipelines are not inherently robust against adversarial attacks.

Title: Landmark Alternating Diffusion

• Authors: Sing-Yuan Yeh, Hau-Tieng Wu, Ronen Talmon, Mao-Pei Tsui
• Subjects: cs.LG, math.ST, physics.data-an, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.19649
• Pdf URL: https://arxiv.org/pdf/2404.19649
• Copy Paste: [[2404.19649]] Landmark Alternating Diffusion(https://arxiv.org/abs/2404.19649)
• Keywords: robust, diffusion
• Abstract: Alternating Diffusion (AD) is a commonly applied diffusion-based sensor fusion algorithm. While it has been successfully applied to various problems, its computational burden remains a limitation. Inspired by the landmark diffusion idea considered in the Robust and Scalable Embedding via Landmark Diffusion (ROSELAND), we propose a variation of AD, called Landmark AD (LAD), which captures the essence of AD while offering superior computational efficiency. We provide a series of theoretical analyses of LAD under the manifold setup and apply it to the automatic sleep stage annotation problem with two electroencephalogram channels to demonstrate its application.

Title: Provably Robust Conformal Prediction with Improved Efficiency

• Authors: Ge Yan, Yaniv Romano, Tsui-Wei Weng
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19651
• Pdf URL: https://arxiv.org/pdf/2404.19651
• Copy Paste: [[2404.19651]] Provably Robust Conformal Prediction with Improved Efficiency(https://arxiv.org/abs/2404.19651)
• Keywords: robust
• Abstract: Conformal prediction is a powerful tool to generate uncertainty sets with guaranteed coverage using any predictive model, under the assumption that the training and test data are i.i.d.. Recently, it has been shown that adversarial examples are able to manipulate conformal methods to construct prediction sets with invalid coverage rates, as the i.i.d. assumption is violated. To address this issue, a recent work, Randomized Smoothed Conformal Prediction (RSCP), was first proposed to certify the robustness of conformal prediction methods to adversarial noise. However, RSCP has two major limitations: (i) its robustness guarantee is flawed when used in practice and (ii) it tends to produce large uncertainty sets. To address these limitations, we first propose a novel framework called RSCP+ to provide provable robustness guarantee in evaluation, which fixes the issues in the original RSCP method. Next, we propose two novel methods, Post-Training Transformation (PTT) and Robust Conformal Training (RCT), to effectively reduce prediction set size with little computation overhead. Experimental results in CIFAR10, CIFAR100, and ImageNet suggest the baseline method only yields trivial predictions including full label set, while our methods could boost the efficiency by up to $4.36\times$, $5.46\times$, and $16.9\times$ respectively and provide practical robustness guarantee. Our codes are available at https://github.com/Trustworthy-ML-Lab/Provably-Robust-Conformal-Prediction.

Title: VimTS: A Unified Video and Image Text Spotter for Enhancing the Cross-domain Generalization

• Authors: Yuliang Liu, Mingxin Huang, Hao Yan, Linger Deng, Weijia Wu, Hao Lu, Chunhua Shen, Lianwen Jin, Xiang Bai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19652
• Pdf URL: https://arxiv.org/pdf/2404.19652
• Copy Paste: [[2404.19652]] VimTS: A Unified Video and Image Text Spotter for Enhancing the Cross-domain Generalization(https://arxiv.org/abs/2404.19652)
• Keywords: extraction
• Abstract: Text spotting, a task involving the extraction of textual information from image or video sequences, faces challenges in cross-domain adaption, such as image-to-image and image-to-video generalization. In this paper, we introduce a new method, termed VimTS, which enhances the generalization ability of the model by achieving better synergy among different tasks. Typically, we propose a Prompt Queries Generation Module and a Tasks-aware Adapter to effectively convert the original single-task model into a multi-task model suitable for both image and video scenarios with minimal additional parameters. The Prompt Queries Generation Module facilitates explicit interaction between different tasks, while the Tasks-aware Adapter helps the model dynamically learn suitable features for each task. Additionally, to further enable the model to learn temporal information at a lower cost, we propose a synthetic video text dataset (VTD-368k) by leveraging the Content Deformation Fields (CoDeF) algorithm. Notably, our method outperforms the state-of-the-art method by an average of 2.6% in six cross-domain benchmarks such as TT-to-IC15, CTW1500-to-TT, and TT-to-CTW1500. For video-level cross-domain adaption, our method even surpasses the previous end-to-end video spotting method in ICDAR2015 video and DSText v2 by an average of 5.5% on the MOTA metric, using only image-level data. We further demonstrate that existing Large Multimodal Models exhibit limitations in generating cross-domain scene text spotting, in contrast to our VimTS model which requires significantly fewer parameters and data. The code and datasets will be made available at the https://VimTextSpotter.github.io.

Title: Masked Multi-Query Slot Attention for Unsupervised Object Discovery

• Authors: Rishav Pramanik, José-Fabian Villa-Vásquez, Marco Pedersoli
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.19654
• Pdf URL: https://arxiv.org/pdf/2404.19654
• Copy Paste: [[2404.19654]] Masked Multi-Query Slot Attention for Unsupervised Object Discovery(https://arxiv.org/abs/2404.19654)
• Keywords: segmentation
• Abstract: Unsupervised object discovery is becoming an essential line of research for tackling recognition problems that require decomposing an image into entities, such as semantic segmentation and object detection. Recently, object-centric methods that leverage self-supervision have gained popularity, due to their simplicity and adaptability to different settings and conditions. However, those methods do not exploit effective techniques already employed in modern self-supervised approaches. In this work, we consider an object-centric approach in which DINO ViT features are reconstructed via a set of queried representations called slots. Based on that, we propose a masking scheme on input features that selectively disregards the background regions, inducing our model to focus more on salient objects during the reconstruction phase. Moreover, we extend the slot attention to a multi-query approach, allowing the model to learn multiple sets of slots, producing more stable masks. During training, these multiple sets of slots are learned independently while, at test time, these sets are merged through Hungarian matching to obtain the final slots. Our experimental results and ablations on the PASCAL-VOC 2012 dataset show the importance of each component and highlight how their combination consistently improves object localization. Our source code is available at: https://github.com/rishavpramanik/maskedmultiqueryslot

Title: Decoder Decomposition for the Analysis of the Latent Space of Nonlinear Autoencoders With Wind-Tunnel Experimental Data

• Authors: Yaxin Mo, Tullio Traverso, Luca Magri
• Subjects: cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2404.19660
• Pdf URL: https://arxiv.org/pdf/2404.19660
• Copy Paste: [[2404.19660]] Decoder Decomposition for the Analysis of the Latent Space of Nonlinear Autoencoders With Wind-Tunnel Experimental Data(https://arxiv.org/abs/2404.19660)
• Keywords: interpretability
• Abstract: Turbulent flows are chaotic and multi-scale dynamical systems, which have large numbers of degrees of freedom. Turbulent flows, however, can be modelled with a smaller number of degrees of freedom when using the appropriate coordinate system, which is the goal of dimensionality reduction via nonlinear autoencoders. Autoencoders are expressive tools, but they are difficult to interpret. The goal of this paper is to propose a method to aid the interpretability of autoencoders. This is the decoder decomposition. First, we propose the decoder decomposition, which is a post-processing method to connect the latent variables to the coherent structures of flows. Second, we apply the decoder decomposition to analyse the latent space of synthetic data of a two-dimensional unsteady wake past a cylinder. We find that the dimension of latent space has a significant impact on the interpretability of autoencoders. We identify the physical and spurious latent variables. Third, we apply the decoder decomposition to the latent space of wind-tunnel experimental data of a three-dimensional turbulent wake past a bluff body. We show that the reconstruction error is a function of both the latent space dimension and the decoder size, which are correlated. Finally, we apply the decoder decomposition to rank and select latent variables based on the coherent structures that they represent. This is useful to filter unwanted or spurious latent variables, or to pinpoint specific coherent structures of interest. The ability to rank and select latent variables will help users design and interpret nonlinear autoencoders.

Title: A Comprehensive Analysis of Pegasus Spyware and Its Implications for Digital Privacy and Security

• Authors: Karwan Kareem
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19677
• Pdf URL: https://arxiv.org/pdf/2404.19677
• Copy Paste: [[2404.19677]] A Comprehensive Analysis of Pegasus Spyware and Its Implications for Digital Privacy and Security(https://arxiv.org/abs/2404.19677)
• Keywords: security, privacy, protect
• Abstract: This paper comprehensively analyzes the Pegasus spyware and its implications for digital privacy and security. The Israeli cyber intelligence company NSO Group's Pegasus has gained recognition as a potent surveillance tool capable of hacking into smartphones and extracting data without the user's knowledge [49], [50]. The research emphasizes the technical aspects of this spyware, its deployment methods, and the controversies surrounding its use. The research also emphasizes the growing worries surrounding digital privacy and security as a result of the prevalent use of advanced spyware. By delving into legal, ethical, and policy issues, the objective of this study is to deliver a holistic understanding of the challenges posed by Pegasus and similar spyware tools. Through a comprehensive examination of the subject, the paper presents potential solutions to mitigate the threats and protect users from invasive surveillance techniques.

Title: Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners

• Authors: Chun Feng, Joy Hsu, Weiyu Liu, Jiajun Wu
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.19696
• Pdf URL: https://arxiv.org/pdf/2404.19696
• Copy Paste: [[2404.19696]] Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners(https://arxiv.org/abs/2404.19696)
• Keywords: large language model
• Abstract: 3D visual grounding is a challenging task that often requires direct and dense supervision, notably the semantic label for each object in the scene. In this paper, we instead study the naturally supervised setting that learns from only 3D scene and QA pairs, where prior works underperform. We propose the Language-Regularized Concept Learner (LARC), which uses constraints from language as regularization to significantly improve the accuracy of neuro-symbolic concept learners in the naturally supervised setting. Our approach is based on two core insights: the first is that language constraints (e.g., a word's relation to another) can serve as effective regularization for structured representations in neuro-symbolic models; the second is that we can query large language models to distill such constraints from language properties. We show that LARC improves performance of prior works in naturally supervised 3D visual grounding, and demonstrates a wide range of 3D visual reasoning capabilities-from zero-shot composition, to data efficiency and transferability. Our method represents a promising step towards regularizing structured visual reasoning frameworks with language-based priors, for learning in settings without dense supervision.

Title: GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting

• Authors: Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, Zexiang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19702
• Pdf URL: https://arxiv.org/pdf/2404.19702
• Copy Paste: [[2404.19702]] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting(https://arxiv.org/abs/2404.19702)
• Keywords: transformer
• Abstract: We propose GS-LRM, a scalable large reconstruction model that can predict high-quality 3D Gaussian primitives from 2-4 posed sparse images in 0.23 seconds on single A100 GPU. Our model features a very simple transformer-based architecture; we patchify input posed images, pass the concatenated multi-view image tokens through a sequence of transformer blocks, and decode final per-pixel Gaussian parameters directly from these tokens for differentiable rendering. In contrast to previous LRMs that can only reconstruct objects, by predicting per-pixel Gaussians, GS-LRM naturally handles scenes with large variations in scale and complexity. We show that our model can work on both object and scene captures by training it on Objaverse and RealEstate10K respectively. In both scenarios, the models outperform state-of-the-art baselines by a wide margin. We also demonstrate applications of our model in downstream 3D generation tasks. Our project webpage is available at: https://sai-bi.github.io/project/gs-lrm/ .

Title: When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively

• Authors: Tiziano Labruna, Jon Ander Campos, Gorka Azkune
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.19705
• Pdf URL: https://arxiv.org/pdf/2404.19705
• Copy Paste: [[2404.19705]] When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively(https://arxiv.org/abs/2404.19705)
• Keywords: large language model
• Abstract: In this paper, we demonstrate how Large Language Models (LLMs) can effectively learn to use an off-the-shelf information retrieval (IR) system specifically when additional context is required to answer a given question. Given the performance of IR systems, the optimal strategy for question answering does not always entail external information retrieval; rather, it often involves leveraging the parametric memory of the LLM itself. Prior research has identified this phenomenon in the PopQA dataset, wherein the most popular questions are effectively addressed using the LLM's parametric memory, while less popular ones require IR system usage. Following this, we propose a tailored training approach for LLMs, leveraging existing open-domain question answering datasets. Here, LLMs are trained to generate a special token, , when they do not know the answer to a question. Our evaluation of the Adaptive Retrieval LLM (Adapt-LLM) on the PopQA dataset showcases improvements over the same LLM under three configurations: (i) retrieving information for all the questions, (ii) using always the parametric memory of the LLM, and (iii) using a popularity threshold to decide when to use a retriever. Through our analysis, we demonstrate that Adapt-LLM is able to generate the token when it determines that it does not know how to answer a question, indicating the need for IR, while it achieves notably high accuracy levels when it chooses to rely only on its parametric memory.

Title: Harmonic LLMs are Trustworthy

• Authors: Nicholas S. Kersting, Mohammad Rahman, Suchismitha Vedala, Yang Wang
• Subjects: cs.LG, cs.AI, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.19708
• Pdf URL: https://arxiv.org/pdf/2404.19708
• Copy Paste: [[2404.19708]] Harmonic LLMs are Trustworthy(https://arxiv.org/abs/2404.19708)
• Keywords: robust, explainability
• Abstract: We introduce an intuitive method to test the robustness (stability and explainability) of any black-box LLM in real-time, based upon the local deviation from harmoniticity, denoted as $\gamma$. To the best of our knowledge this is the first completely model-agnostic and unsupervised method of measuring the robustness of any given response from an LLM, based upon the model itself conforming to a purely mathematical standard. We conduct human annotation experiments to show the positive correlation of $\gamma$ with false or misleading answers, and demonstrate that following the gradient of $\gamma$ in stochastic gradient ascent efficiently exposes adversarial prompts. Measuring $\gamma$ across thousands of queries in popular LLMs (GPT-4, ChatGPT, Claude-2.1, Mixtral-8x7B, Smaug-72B, Llama2-7B, and MPT-7B) allows us to estimate the liklihood of wrong or hallucinatory answers automatically and quantitatively rank the reliability of these models in various objective domains (Web QA, TruthfulQA, and Programming QA). Across all models and domains tested, human ratings confirm that $\gamma \to 0$ indicates trustworthiness, and the low-$\gamma$ leaders among these models are GPT-4, ChatGPT, and Smaug-72B.

Title: Automated Generation of High-Quality Medical Simulation Scenarios Through Integration of Semi-Structured Data and Large Language Models

• Authors: Scott Sumpter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19713
• Pdf URL: https://arxiv.org/pdf/2404.19713
• Copy Paste: [[2404.19713]] Automated Generation of High-Quality Medical Simulation Scenarios Through Integration of Semi-Structured Data and Large Language Models(https://arxiv.org/abs/2404.19713)
• Keywords: large language model
• Abstract: This study introduces a transformative framework for medical education by integrating semi-structured data with Large Language Models (LLMs), primarily OpenAIs ChatGPT3.5, to automate the creation of medical simulation scenarios. Traditionally, developing these scenarios was a time-intensive process with limited flexibility to meet diverse educational needs. The proposed approach utilizes AI to efficiently generate detailed, clinically relevant scenarios that are tailored to specific educational objectives. This innovation has significantly reduced the time and resources required for scenario development, allowing for a broader variety of simulations. Preliminary feedback from educators and learners has shown enhanced engagement and improved knowledge acquisition, confirming the effectiveness of this AI-enhanced methodology in simulation-based learning. The integration of structured data with LLMs not only streamlines the creation process but also offers a scalable, dynamic solution that could revolutionize medical training, highlighting the critical role of AI in advancing educational outcomes and patient care standards.

Title: Assessing LLMs in Malicious Code Deobfuscation of Real-world Malware Campaigns

• Authors: Constantinos Patsakis, Fran Casino, Nikolaos Lykousas
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.19715
• Pdf URL: https://arxiv.org/pdf/2404.19715
• Copy Paste: [[2404.19715]] Assessing LLMs in Malicious Code Deobfuscation of Real-world Malware Campaigns(https://arxiv.org/abs/2404.19715)
• Keywords: security, attack, large language model
• Abstract: The integration of large language models (LLMs) into various pipelines is increasingly widespread, effectively automating many manual tasks and often surpassing human capabilities. Cybersecurity researchers and practitioners have recognised this potential. Thus, they are actively exploring its applications, given the vast volume of heterogeneous data that requires processing to identify anomalies, potential bypasses, attacks, and fraudulent incidents. On top of this, LLMs' advanced capabilities in generating functional code, comprehending code context, and summarising its operations can also be leveraged for reverse engineering and malware deobfuscation. To this end, we delve into the deobfuscation capabilities of state-of-the-art LLMs. Beyond merely discussing a hypothetical scenario, we evaluate four LLMs with real-world malicious scripts used in the notorious Emotet malware campaign. Our results indicate that while not absolutely accurate yet, some LLMs can efficiently deobfuscate such payloads. Thus, fine-tuning LLMs for this task can be a viable potential for future AI-powered threat intelligence pipelines in the fight against obfuscated malware.

Title: Fairness Without Demographics in Human-Centered Federated Learning

• Authors: Roy Shaily, Sharma Harshit, Salekin Asif
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.19725
• Pdf URL: https://arxiv.org/pdf/2404.19725
• Copy Paste: [[2404.19725]] Fairness Without Demographics in Human-Centered Federated Learning(https://arxiv.org/abs/2404.19725)
• Keywords: privacy, federate, fair
• Abstract: Federated learning (FL) enables collaborative model training while preserving data privacy, making it suitable for decentralized human-centered AI applications. However, a significant research gap remains in ensuring fairness in these systems. Current fairness strategies in FL require knowledge of bias-creating/sensitive attributes, clashing with FL's privacy principles. Moreover, in human-centered datasets, sensitive attributes may remain latent. To tackle these challenges, we present a novel bias mitigation approach inspired by "Fairness without Demographics" in machine learning. The presented approach achieves fairness without needing knowledge of sensitive attributes by minimizing the top eigenvalue of the Hessian matrix during training, ensuring equitable loss landscapes across FL participants. Notably, we introduce a novel FL aggregation scheme that promotes participating models based on error rates and loss landscape curvature attributes, fostering fairness across the FL system. This work represents the first approach to attaining "Fairness without Demographics" in human-centered FL. Through comprehensive evaluation, our approach demonstrates effectiveness in balancing fairness and efficacy across various real-world applications, FL setups, and scenarios involving single and multiple bias-inducing factors, representing a significant advancement in human-centered FL.

Title: Better & Faster Large Language Models via Multi-token Prediction

• Authors: Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.19737
• Pdf URL: https://arxiv.org/pdf/2404.19737
• Copy Paste: [[2404.19737]] Better & Faster Large Language Models via Multi-token Prediction(https://arxiv.org/abs/2404.19737)
• Keywords: generative, large language model
• Abstract: Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. Considering multi-token prediction as an auxiliary training task, we measure improved downstream capabilities with no overhead in training time for both code and natural language models. The method is increasingly useful for larger model sizes, and keeps its appeal when training for multiple epochs. Gains are especially pronounced on generative benchmarks like coding, where our models consistently outperform strong baselines by several percentage points. Our 13B parameter models solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models. Experiments on small algorithmic tasks demonstrate that multi-token prediction is favorable for the development of induction heads and algorithmic reasoning capabilities. As an additional benefit, models trained with 4-token prediction are up to 3 times faster at inference, even with large batch sizes.

Title: PrivComp-KG : Leveraging Knowledge Graph and Large Language Models for Privacy Policy Compliance Verification

• Authors: Leon Garza, Lavanya Elluri, Anantaa Kotal, Aritran Piplai, Deepti Gupta, Anupam Joshi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.19744
• Pdf URL: https://arxiv.org/pdf/2404.19744
• Copy Paste: [[2404.19744]] PrivComp-KG : Leveraging Knowledge Graph and Large Language Models for Privacy Policy Compliance Verification(https://arxiv.org/abs/2404.19744)
• Keywords: security, privacy, protect, large language model
• Abstract: Data protection and privacy is becoming increasingly crucial in the digital era. Numerous companies depend on third-party vendors and service providers to carry out critical functions within their operations, encompassing tasks such as data handling and storage. However, this reliance introduces potential vulnerabilities, as these vendors' security measures and practices may not always align with the standards expected by regulatory bodies. Businesses are required, often under the penalty of law, to ensure compliance with the evolving regulatory rules. Interpreting and implementing these regulations pose challenges due to their complexity. Regulatory documents are extensive, demanding significant effort for interpretation, while vendor-drafted privacy policies often lack the detail required for full legal compliance, leading to ambiguity. To ensure a concise interpretation of the regulatory requirements and compliance of organizational privacy policy with said regulations, we propose a Large Language Model (LLM) and Semantic Web based approach for privacy compliance. In this paper, we develop the novel Privacy Policy Compliance Verification Knowledge Graph, PrivComp-KG. It is designed to efficiently store and retrieve comprehensive information concerning privacy policies, regulatory frameworks, and domain-specific knowledge pertaining to the legal landscape of privacy. Using Retrieval Augmented Generation, we identify the relevant sections in a privacy policy with corresponding regulatory rules. This information about individual privacy policies is populated into the PrivComp-KG. Combining this with the domain context and rules, the PrivComp-KG can be queried to check for compliance with privacy policies by each vendor against relevant policy regulations. We demonstrate the relevance of the PrivComp-KG, by verifying compliance of privacy policy documents for various organizations.

Title: Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation

• Authors: Yunhao Ge, Xiaohui Zeng, Jacob Samuel Huffman, Tsung-Yi Lin, Ming-Yu Liu, Yin Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19752
• Pdf URL: https://arxiv.org/pdf/2404.19752
• Copy Paste: [[2404.19752]] Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation(https://arxiv.org/abs/2404.19752)
• Keywords: large language model
• Abstract: Existing automatic captioning methods for visual content face challenges such as lack of detail, content hallucination, and poor instruction following. In this work, we propose VisualFactChecker (VFC), a flexible training-free pipeline that generates high-fidelity and detailed captions for both 2D images and 3D objects. VFC consists of three steps: 1) proposal, where image-to-text captioning models propose multiple initial captions; 2) verification, where a large language model (LLM) utilizes tools such as object detection and VQA models to fact-check proposed captions; 3) captioning, where an LLM generates the final caption by summarizing caption proposals and the fact check verification results. In this step, VFC can flexibly generate captions in various styles following complex instructions. We conduct comprehensive captioning evaluations using four metrics: 1) CLIP-Score for image-text similarity; 2) CLIP-Image-Score for measuring the image-image similarity between the original and the reconstructed image generated by a text-to-image model using the caption. 3) human study on Amazon Mechanical Turk; 4) GPT-4V for fine-grained evaluation. Evaluation results show that VFC outperforms state-of-the-art open-sourced captioning methods for 2D images on the COCO dataset and 3D assets on the Objaverse dataset. Our study demonstrates that by combining open-source models into a pipeline, we can attain captioning capability comparable to proprietary models such as GPT-4V, despite being over 10x smaller in model size.

Title: KAN: Kolmogorov-Arnold Networks

• Authors: Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, Max Tegmark
• Subjects: cs.LG, cond-mat.dis-nn, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.19756
• Pdf URL: https://arxiv.org/pdf/2404.19756
• Copy Paste: [[2404.19756]] KAN: Kolmogorov-Arnold Networks(https://arxiv.org/abs/2404.19756)
• Keywords: interpretability
• Abstract: Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability. For accuracy, much smaller KANs can achieve comparable or better accuracy than much larger MLPs in data fitting and PDE solving. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful collaborators helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs, opening opportunities for further improving today's deep learning models which rely heavily on MLPs.

Title: Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting

• Authors: Paul Engstler, Andrea Vedaldi, Iro Laina, Christian Rupprecht
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19758
• Pdf URL: https://arxiv.org/pdf/2404.19758
• Copy Paste: [[2404.19758]] Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting(https://arxiv.org/abs/2404.19758)
• Keywords: diffusion, generative
• Abstract: 3D scene generation has quickly become a challenging new research direction, fueled by consistent improvements of 2D generative diffusion models. Most prior work in this area generates scenes by iteratively stitching newly generated frames with existing geometry. These works often depend on pre-trained monocular depth estimators to lift the generated images into 3D, fusing them with the existing scene representation. These approaches are then often evaluated via a text metric, measuring the similarity between the generated images and a given text prompt. In this work, we make two fundamental contributions to the field of 3D scene generation. First, we note that lifting images to 3D with a monocular depth estimation model is suboptimal as it ignores the geometry of the existing scene. We thus introduce a novel depth completion model, trained via teacher distillation and self-training to learn the 3D fusion process, resulting in improved geometric coherence of the scene. Second, we introduce a new benchmarking scheme for scene generation methods that is based on ground truth geometry, and thus measures the quality of the structure of the scene.

Title: MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model

• Authors: Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, Yansong Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.19759
• Pdf URL: https://arxiv.org/pdf/2404.19759
• Copy Paste: [[2404.19759]] MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model(https://arxiv.org/abs/2404.19759)
• Keywords: diffusion
• Abstract: This work introduces MotionLCM, extending controllable motion generation to a real-time level. Existing methods for spatial control in text-conditioned motion generation suffer from significant runtime inefficiency. To address this issue, we first propose the motion latent consistency model (MotionLCM) for motion generation, building upon the latent diffusion model (MLD). By employing one-step (or few-step) inference, we further improve the runtime efficiency of the motion latent diffusion model for motion generation. To ensure effective controllability, we incorporate a motion ControlNet within the latent space of MotionLCM and enable explicit control signals (e.g., pelvis trajectory) in the vanilla motion space to control the generation process directly, similar to controlling other latent-free diffusion models for motion generation. By employing these techniques, our approach can generate human motions with text and control signals in real-time. Experimental results demonstrate the remarkable generation and controlling capabilities of MotionLCM while maintaining real-time runtime efficiency.