2024-04-18

Title: Phishing Website Detection Using a Combined Model of ANN and LSTM

• Authors: Muhammad Shoaib Farooq, Hina jabbar
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.10780
• Pdf URL: https://arxiv.org/pdf/2404.10780
• Copy Paste: [[2404.10780]] Phishing Website Detection Using a Combined Model of ANN and LSTM(https://arxiv.org/abs/2404.10780)
• Keywords: security, attack, steal
• Abstract: In this digital era, our lives highly depend on the internet and worldwide technology. Wide usage of technology and platforms of communication makes our lives better and easier. But on the other side it carries out some security issues and cruel activities, phishing is one activity of these cruel activities. It is a type of cybercrime, which has the purpose of stealing the personal information of the computer user, and enterprises, which carry out fake websites that are the copy of the original websites. The attackers used personal information like account IDs, passwords, and usernames for the purpose of some fraudulent activities against the user of the computer. To overcome this problem researchers focused on the machine learning and deep learning approaches. In our study, we are going to use machine learning and deep learning models to identify the fake web pages on the secondary dataset.

Title: Authenticity in Authorship: The Writer's Integrity Framework for Verifying Human-Generated Text

• Authors: Sanad Aburass, Maha Abu Rumman
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.10781
• Pdf URL: https://arxiv.org/pdf/2404.10781
• Copy Paste: [[2404.10781]] Authenticity in Authorship: The Writer's Integrity Framework for Verifying Human-Generated Text(https://arxiv.org/abs/2404.10781)
• Keywords: privacy
• Abstract: The "Writer's Integrity" framework introduces a paradigm shift in maintaining the sanctity of human-generated text in the realms of academia, research, and publishing. This innovative system circumvents the shortcomings of current AI detection tools by monitoring the writing process, rather than the product, capturing the distinct behavioral footprint of human authorship. Here, we offer a comprehensive examination of the framework, its development, and empirical results. We highlight its potential in revolutionizing the validation of human intellectual work, emphasizing its role in upholding academic integrity and intellectual property rights in the face of sophisticated AI models capable of emulating human-like text. This paper also discusses the implementation considerations, addressing potential user concerns regarding ease of use and privacy, and outlines a business model for tech companies to monetize the framework effectively. Through licensing, partnerships, and subscriptions, companies can cater to universities, publishers, and independent writers, ensuring the preservation of original thought and effort in written content. This framework is open source and available here, https://github.com/sanadv/Integrity.github.io

Title: Quantifying AI Vulnerabilities: A Synthesis of Complexity, Dynamical Systems, and Game Theory

• Authors: B Kereopa-Yorke
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.10782
• Pdf URL: https://arxiv.org/pdf/2404.10782
• Copy Paste: [[2404.10782]] Quantifying AI Vulnerabilities: A Synthesis of Complexity, Dynamical Systems, and Game Theory(https://arxiv.org/abs/2404.10782)
• Keywords: secure, security, robust
• Abstract: The rapid integration of Artificial Intelligence (AI) systems across critical domains necessitates robust security evaluation frameworks. We propose a novel approach that introduces three metrics: System Complexity Index (SCI), Lyapunov Exponent for AI Stability (LEAIS), and Nash Equilibrium Robustness (NER). SCI quantifies the inherent complexity of an AI system, LEAIS captures its stability and sensitivity to perturbations, and NER evaluates its strategic robustness against adversarial manipulation. Through comparative analysis, we demonstrate the advantages of our framework over existing techniques. We discuss the theoretical and practical implications, potential applications, limitations, and future research directions. Our work contributes to the development of secure and trustworthy AI technologies by providing a holistic, theoretically grounded approach to AI security evaluation. As AI continues to advance, prioritising and advancing AI security through interdisciplinary collaboration is crucial to ensure its responsible deployment for the benefit of society.

Title: The Path To Autonomous Cyber Defense

• Authors: Sean Oesch, Phillipe Austria, Amul Chaulagain, Brian Weber, Cory Watson, Matthew Dixson, Amir Sadovnik
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.10788
• Pdf URL: https://arxiv.org/pdf/2404.10788
• Copy Paste: [[2404.10788]] The Path To Autonomous Cyber Defense(https://arxiv.org/abs/2404.10788)
• Keywords: defense, attack
• Abstract: Defenders are overwhelmed by the number and scale of attacks against their networks.This problem will only be exacerbated as attackers leverage artificial intelligence to automate their workflows. We propose a path to autonomous cyber agents able to augment defenders by automating critical steps in the cyber defense life cycle.

Title: PASA: Attack Agnostic Unsupervised Adversarial Detection using Prediction & Attribution Sensitivity Analysis

• Authors: Dipkamal Bhusal, Md Tanvirul Alam, Monish K. Veerabhadran, Michael Clifford, Sara Rampazzi, Nidhi Rastogi
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.10789
• Pdf URL: https://arxiv.org/pdf/2404.10789
• Copy Paste: [[2404.10789]] PASA: Attack Agnostic Unsupervised Adversarial Detection using Prediction & Attribution Sensitivity Analysis(https://arxiv.org/abs/2404.10789)
• Keywords: defense, attack
• Abstract: Deep neural networks for classification are vulnerable to adversarial attacks, where small perturbations to input samples lead to incorrect predictions. This susceptibility, combined with the black-box nature of such networks, limits their adoption in critical applications like autonomous driving. Feature-attribution-based explanation methods provide relevance of input features for model predictions on input samples, thus explaining model decisions. However, we observe that both model predictions and feature attributions for input samples are sensitive to noise. We develop a practical method for this characteristic of model prediction and feature attribution to detect adversarial samples. Our method, PASA, requires the computation of two test statistics using model prediction and feature attribution and can reliably detect adversarial samples using thresholds learned from benign samples. We validate our lightweight approach by evaluating the performance of PASA on varying strengths of FGSM, PGD, BIM, and CW attacks on multiple image and non-image datasets. On average, we outperform state-of-the-art statistical unsupervised adversarial detectors on CIFAR-10 and ImageNet by 14\% and 35\% ROC-AUC scores, respectively. Moreover, our approach demonstrates competitive performance even when an adversary is aware of the defense mechanism.

Title: Multimodal Attack Detection for Action Recognition Models

• Authors: Furkan Mumcu, Yasin Yilmaz
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.10790
• Pdf URL: https://arxiv.org/pdf/2404.10790
• Copy Paste: [[2404.10790]] Multimodal Attack Detection for Action Recognition Models(https://arxiv.org/abs/2404.10790)
• Keywords: security, defense, attack, robust
• Abstract: Adversarial machine learning attacks on video action recognition models is a growing research area and many effective attacks were introduced in recent years. These attacks show that action recognition models can be breached in many ways. Hence using these models in practice raises significant security concerns. However, there are very few works which focus on defending against or detecting attacks. In this work, we propose a novel universal detection method which is compatible with any action recognition model. In our extensive experiments, we show that our method consistently detects various attacks against different target models with high true positive rates while satisfying very low false positive rates. Tested against four state-of-the-art attacks targeting four action recognition models, the proposed detector achieves an average AUC of 0.911 over 16 test cases while the best performance achieved by the existing detectors is 0.645 average AUC. This 41.2% improvement is enabled by the robustness of the proposed detector to varying attack methods and target models. The lowest AUC achieved by our detector across the 16 test cases is 0.837 while the competing detector's performance drops as low as 0.211. We also show that the proposed detector is robust to varying attack strengths. In addition, we analyze our method's real-time performance with different hardware setups to demonstrate its potential as a practical defense mechanism.

Title: Reconfigurable Edge Hardware for Intelligent IDS: Systematic Approach

• Authors: Wadid Foudhaili, Anouar Nechi, Celine Thermann, Mohammad Al Johmani, Rainer Buchty, Mladen Berekovic, Saleh Mulhem
• Subjects: cs.CR, cs.AR, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.10792
• Pdf URL: https://arxiv.org/pdf/2404.10792
• Copy Paste: [[2404.10792]] Reconfigurable Edge Hardware for Intelligent IDS: Systematic Approach(https://arxiv.org/abs/2404.10792)
• Keywords: security
• Abstract: Intrusion detection systems (IDS) are crucial security measures nowadays to enforce network security. Their task is to detect anomalies in network communication and identify, if not thwart, possibly malicious behavior. Recently, machine learning has been deployed to construct intelligent IDS. This approach, however, is quite challenging particularly in distributed, highly dynamic, yet resource-constrained systems like Edge setups. In this paper, we tackle this issue from multiple angles by analyzing the concept of intelligent IDS (I-IDS) while addressing the specific requirements of Edge devices with a special focus on reconfigurability. Then, we introduce a systematic approach to constructing the I-IDS on reconfigurable Edge hardware. For this, we implemented our proposed IDS on state-of-the-art Field Programmable Gate Arrays (FPGAs) technology as (1) a purely FPGA-based dataflow processor (DFP) and (2) a co-designed approach featuring RISC-V soft-core as FPGA-based soft-core processor (SCP). We complete our paper with a comparison of the state of the art (SoA) in this domain. The results show that DFP and SCP are both suitable for Edge applications from hardware resource and energy efficiency perspectives. Our proposed DFP solution clearly outperforms the SoA and demonstrates that required high performance can be achieved without prohibitively high hardware costs. This makes our proposed DFP suitable for Edge-based high-speed applications like modern communication technology.

Title: Black-box Adversarial Transferability: An Empirical Study in Cybersecurity Perspective

• Authors: Khushnaseeb Roshan, Aasim Zafar
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.10796
• Pdf URL: https://arxiv.org/pdf/2404.10796
• Copy Paste: [[2404.10796]] Black-box Adversarial Transferability: An Empirical Study in Cybersecurity Perspective(https://arxiv.org/abs/2404.10796)
• Keywords: security, attack, robust
• Abstract: The rapid advancement of artificial intelligence within the realm of cybersecurity raises significant security concerns. The vulnerability of deep learning models in adversarial attacks is one of the major issues. In adversarial machine learning, malicious users try to fool the deep learning model by inserting adversarial perturbation inputs into the model during its training or testing phase. Subsequently, it reduces the model confidence score and results in incorrect classifications. The novel key contribution of the research is to empirically test the black-box adversarial transferability phenomena in cyber attack detection systems. It indicates that the adversarial perturbation input generated through the surrogate model has a similar impact on the target model in producing the incorrect classification. To empirically validate this phenomenon, surrogate and target models are used. The adversarial perturbation inputs are generated based on the surrogate-model for which the hacker has complete information. Based on these adversarial perturbation inputs, both surrogate and target models are evaluated during the inference phase. We have done extensive experimentation over the CICDDoS-2019 dataset, and the results are classified in terms of various performance metrics like accuracy, precision, recall, and f1-score. The findings indicate that any deep learning model is highly susceptible to adversarial attacks, even if the attacker does not have access to the internal details of the target model. The results also indicate that white-box adversarial attacks have a severe impact compared to black-box adversarial attacks. There is a need to investigate and explore adversarial defence techniques to increase the robustness of the deep learning models against adversarial attacks.

Title: Fewer Truncations Improve Language Modeling

• Authors: Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, Stefano Soatto
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.10830
• Pdf URL: https://arxiv.org/pdf/2404.10830
• Copy Paste: [[2404.10830]] Fewer Truncations Improve Language Modeling(https://arxiv.org/abs/2404.10830)
• Keywords: large language model
• Abstract: In large language model training, input documents are typically concatenated together and then split into sequences of equal length to avoid padding tokens. Despite its efficiency, the concatenation approach compromises data integrity -- it inevitably breaks many documents into incomplete pieces, leading to excessive truncations that hinder the model from learning to compose logically coherent and factually consistent content that is grounded on the complete context. To address the issue, we propose Best-fit Packing, a scalable and efficient method that packs documents into training sequences through length-aware combinatorial optimization. Our method completely eliminates unnecessary truncations while retaining the same training efficiency as concatenation. Empirical results from both text and code pre-training show that our method achieves superior performance (e.g., relatively +4.7% on reading comprehension; +16.8% in context following; and +9.2% on program synthesis), and reduces closed-domain hallucination effectively by up to 58.3%.

Title: Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning

• Authors: Zhengyang Liang, Meiyu Liang, Wei Huang, Yawen Li, Zhe Xue
• Subjects: cs.CV, cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.10838
• Pdf URL: https://arxiv.org/pdf/2404.10838
• Copy Paste: [[2404.10838]] Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning(https://arxiv.org/abs/2404.10838)
• Keywords: extraction
• Abstract: In recent years, pre-trained multimodal large models have attracted widespread attention due to their outstanding performance in various multimodal applications. Nonetheless, the extensive computational resources and vast datasets required for their training present significant hurdles for deployment in environments with limited computational resources. To address this challenge, we propose a novel dynamic self-adaptive multiscale distillation from pre-trained multimodal large model for efficient cross-modal representation learning for the first time. Unlike existing distillation methods, our strategy employs a multiscale perspective, enabling the extraction structural knowledge across from the pre-trained multimodal large model. Ensuring that the student model inherits a comprehensive and nuanced understanding of the teacher knowledge. To optimize each distillation loss in a balanced and efficient manner, we propose a dynamic self-adaptive distillation loss balancer, a novel component eliminating the need for manual loss weight adjustments and dynamically balances each loss item during the distillation process. Our methodology streamlines pre-trained multimodal large models using only their output features and original image-level information, requiring minimal computational resources. This efficient approach is suited for various applications and allows the deployment of advanced multimodal technologies even in resource-limited settings. Extensive experiments has demonstrated that our method maintains high performance while significantly reducing model complexity and training costs. Moreover, our distilled student model utilizes only image-level information to achieve state-of-the-art performance on cross-modal retrieval tasks, surpassing previous methods that relied on region-level information.

Title: Gasformer: A Transformer-based Architecture for Segmenting Methane Emissions from Livestock in Optical Gas Imaging

• Authors: Toqi Tahamid Sarker, Mohamed G Embaby, Khaled R Ahmed, Amer AbuGhazaleh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.10841
• Pdf URL: https://arxiv.org/pdf/2404.10841
• Copy Paste: [[2404.10841]] Gasformer: A Transformer-based Architecture for Segmenting Methane Emissions from Livestock in Optical Gas Imaging(https://arxiv.org/abs/2404.10841)
• Keywords: transformer, segmentation
• Abstract: Methane emissions from livestock, particularly cattle, significantly contribute to climate change. Effective methane emission mitigation strategies are crucial as the global population and demand for livestock products increase. We introduce Gasformer, a novel semantic segmentation architecture for detecting low-flow rate methane emissions from livestock, and controlled release experiments using optical gas imaging. We present two unique datasets captured with a FLIR GF77 OGI camera. Gasformer leverages a Mix Vision Transformer encoder and a Light-Ham decoder to generate multi-scale features and refine segmentation maps. Gasformer outperforms other state-of-the-art models on both datasets, demonstrating its effectiveness in detecting and segmenting methane plumes in controlled and real-world scenarios. On the livestock dataset, Gasformer achieves mIoU of 88.56%, surpassing other state-of-the-art models. Materials are available at: github.com/toqitahamid/Gasformer.

Title: A LayoutLMv3-Based Model for Enhanced Relation Extraction in Visually-Rich Documents

• Authors: Wiam Adnan, Joel Tang, Yassine Bel Khayat Zouggari, Seif Edinne Laatiri, Laurent Lam, Fabien Caspani
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.10848
• Pdf URL: https://arxiv.org/pdf/2404.10848
• Copy Paste: [[2404.10848]] A LayoutLMv3-Based Model for Enhanced Relation Extraction in Visually-Rich Documents(https://arxiv.org/abs/2404.10848)
• Keywords: extraction
• Abstract: Document Understanding is an evolving field in Natural Language Processing (NLP). In particular, visual and spatial features are essential in addition to the raw text itself and hence, several multimodal models were developed in the field of Visual Document Understanding (VDU). However, while research is mainly focused on Key Information Extraction (KIE), Relation Extraction (RE) between identified entities is still under-studied. For instance, RE is crucial to regroup entities or obtain a comprehensive hierarchy of data in a document. In this paper, we present a model that, initialized from LayoutLMv3, can match or outperform the current state-of-the-art results in RE applied to Visually-Rich Documents (VRD) on FUNSD and CORD datasets, without any specific pre-training and with fewer parameters. We also report an extensive ablation study performed on FUNSD, highlighting the great impact of certain features and modelization choices on the performances.

Title: UruDendro, a public dataset of cross-section images of Pinus taeda

• Authors: Henry Marichal, Diego Passarella, Christine Lucas, Ludmila Profumo, Verónica Casaravilla, María Noel Rocha Galli, Serrana Ambite, Gregory Randall
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2404.10856
• Pdf URL: https://arxiv.org/pdf/2404.10856
• Copy Paste: [[2404.10856]] UruDendro, a public dataset of cross-section images of Pinus taeda(https://arxiv.org/abs/2404.10856)
• Keywords: robust
• Abstract: The automatic detection of tree-ring boundaries and other anatomical features using image analysis has progressed substantially over the past decade with advances in machine learning and imagery technology, as well as increasing demands from the dendrochronology community. This paper presents a publicly available database of 64 scanned images of transverse sections of commercially grown Pinus taeda trees from northern Uruguay, ranging from 17 to 24 years old. The collection contains several challenging features for automatic ring detection, including illumination and surface preparation variation, fungal infection (blue stains), knot formation, missing cortex or interruptions in outer rings, and radial cracking. This dataset can be used to develop and test automatic tree ring detection algorithms. This paper presents to the dendrochronology community one such method, Cross-Section Tree-Ring Detection (CS-TRD), which identifies and marks complete annual rings in cross-sections for tree species presenting a clear definition between early and latewood. We compare the CS-TRD performance against the ground truth manual delineation of all rings over the UruDendro dataset. The CS-TRD software identified rings with an average F-score of 89% and RMSE error of 5.27px for the entire database in less than 20 seconds per image. Finally, we propose a robust measure of the ring growth using the \emph{equivalent radius} of a circle having the same area enclosed by the detected tree ring. Overall, this study contributes to the dendrochronologist's toolbox of fast and low-cost methods to automatically detect rings in conifer species, particularly for measuring diameter growth rates and stem transverse area using entire cross-sections.

Title: Forcing Diffuse Distributions out of Language Models

• Authors: Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, Zico Kolter, Daphne Ippolito
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.10859
• Pdf URL: https://arxiv.org/pdf/2404.10859
• Copy Paste: [[2404.10859]] Forcing Diffuse Distributions out of Language Models(https://arxiv.org/abs/2404.10859)
• Keywords: large language model
• Abstract: Despite being trained specifically to follow user instructions, today's language models perform poorly when instructed to produce random outputs. For example, when prompted to pick a number uniformly between one and ten Llama-2-13B-chat disproportionately favors the number five, and when tasked with picking a first name at random, Mistral-7B-Instruct chooses Avery 40 times more often than we would expect based on the U.S. population. When these language models are used for real-world tasks where diversity of outputs is crucial, such as language model assisted dataset construction, their inability to produce diffuse distributions over valid choices is a major hurdle. In this work, we propose a fine-tuning method that encourages language models to output distributions that are diffuse over valid outcomes. The methods we introduce generalize across a variety of tasks and distributions and make large language models practical for synthetic dataset generation with little human intervention.

Title: Vocabulary-free Image Classification and Semantic Segmentation

• Authors: Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, Elisa Ricci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.10864
• Pdf URL: https://arxiv.org/pdf/2404.10864
• Copy Paste: [[2404.10864]] Vocabulary-free Image Classification and Semantic Segmentation(https://arxiv.org/abs/2404.10864)
• Keywords: segmentation
• Abstract: Large vision-language models revolutionized image classification and semantic segmentation paradigms. However, they typically assume a pre-defined set of categories, or vocabulary, at test time for composing textual prompts. This assumption is impractical in scenarios with unknown or evolving semantic context. Here, we address this issue and introduce the Vocabulary-free Image Classification (VIC) task, which aims to assign a class from an unconstrained language-induced semantic space to an input image without needing a known vocabulary. VIC is challenging due to the vastness of the semantic space, which contains millions of concepts, including fine-grained categories. To address VIC, we propose Category Search from External Databases (CaSED), a training-free method that leverages a pre-trained vision-language model and an external database. CaSED first extracts the set of candidate categories from the most semantically similar captions in the database and then assigns the image to the best-matching candidate category according to the same vision-language model. Furthermore, we demonstrate that CaSED can be applied locally to generate a coarse segmentation mask that classifies image regions, introducing the task of Vocabulary-free Semantic Segmentation. CaSED and its variants outperform other more complex vision-language models, on classification and semantic segmentation benchmarks, while using much fewer parameters.

Title: HumMUSS: Human Motion Understanding using State Space Models

• Authors: Arnab Kumar Mondal, Stefano Alletto, Denis Tome
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.10880
• Pdf URL: https://arxiv.org/pdf/2404.10880
• Copy Paste: [[2404.10880]] HumMUSS: Human Motion Understanding using State Space Models(https://arxiv.org/abs/2404.10880)
• Keywords: transformer
• Abstract: Understanding human motion from video is essential for a range of applications, including pose estimation, mesh recovery and action recognition. While state-of-the-art methods predominantly rely on transformer-based architectures, these approaches have limitations in practical scenarios. Transformers are slower when sequentially predicting on a continuous stream of frames in real-time, and do not generalize to new frame rates. In light of these constraints, we propose a novel attention-free spatiotemporal model for human motion understanding building upon recent advancements in state space models. Our model not only matches the performance of transformer-based models in various motion understanding tasks but also brings added benefits like adaptability to different video frame rates and enhanced training speed when working with longer sequence of keypoints. Moreover, the proposed model supports both offline and real-time applications. For real-time sequential prediction, our model is both memory efficient and several times faster than transformer-based approaches while maintaining their high accuracy.

Title: Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning

• Authors: Moghis Fereidouni, A.B. Siddique
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.10887
• Pdf URL: https://arxiv.org/pdf/2404.10887
• Copy Paste: [[2404.10887]] Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning(https://arxiv.org/abs/2404.10887)
• Keywords: transformer
• Abstract: Traditional search systems focus on query formulation for effective results but face challenges in scenarios such as product searches where crucial product details (e.g., size, color) remain concealed until users visit specific product pages. This highlights the need for intelligent web navigation agents capable of formulating queries and navigating web pages according to users' high-level intents. In response to this need, this work introduces a Grounded Language Agent for Intelligent Web Interactions, called GLAINTEL. Drawing upon advancements in language modeling and reinforcement learning, GLAINTEL investigates the efficacy of transformer-based models in enhancing the search capabilities of interactive web environments. Given the dynamic action space for each state in web navigation, GLAINTEL employs the Flan-T5 architecture and incorporates language modeling and value estimation heads. This work focuses on training smaller language models as agents across various scenarios, systematically evaluating the impact of human demonstrations on the training process. Specifically, we investigate scenarios where no human demonstrations are available and subsequently assess the effective utilization of such demonstrations. We also explore unsupervised domain adaptation for situations where demonstrations are confined to a specific domain. Experimental evaluations across diverse setups demonstrate the effectiveness of training agents in unsupervised settings, outperforming in-context learning-based approaches that employ larger models with up to 540 billion parameters. Surprisingly, behavioral cloning-based methods that straightforwardly use human demonstrations do not outperform unsupervised learning-based methods. Additionally, combining human demonstrations with Reinforcement Learning-based training yields results comparable to models utilizing GPT-4.

Title: From a Lossless (~1.5:1) Compression Algorithm for Llama2 7B Weights to Variable Precision, Variable Range, Compressed Numeric Data Types for CNNs and LLMs

• Authors: Vincenzo Liguori
• Subjects: cs.CV, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2404.10896
• Pdf URL: https://arxiv.org/pdf/2404.10896
• Copy Paste: [[2404.10896]] From a Lossless (~1.5:1) Compression Algorithm for Llama2 7B Weights to Variable Precision, Variable Range, Compressed Numeric Data Types for CNNs and LLMs(https://arxiv.org/abs/2404.10896)
• Keywords: large language model
• Abstract: This paper starts with a simple lossless ~1.5:1 compression algorithm for the weights of the Large Language Model (LLM) Llama2 7B [1] that can be implemented in ~200 LUTs in AMD FPGAs, processing over 800 million bfloat16 numbers per second. This framework is then extended to variable precision, variable range, compressed numerical data types that are a user defined super set of both floats and posits [2]. The paper then discusses a simple hardware implementation of such format based on ANS (Asymmetrical Numeral Systems) [3] that acts as a bridge between this flexible data format and a computational engine while, at the same time, achieving bandwidth reduction. An example of a token factory using weight compression and sharing is also given.

Title: Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training

• Authors: Pavel Denisov, Ngoc Thang Vu
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.10922
• Pdf URL: https://arxiv.org/pdf/2404.10922
• Copy Paste: [[2404.10922]] Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training(https://arxiv.org/abs/2404.10922)
• Keywords: robust, large language model
• Abstract: Recent advancements in language modeling have led to the emergence of Large Language Models (LLMs) capable of various natural language processing tasks. Despite their success in text-based tasks, applying LLMs to the speech domain remains limited and challenging. This paper presents BLOOMZMMS, a novel model that integrates a multilingual LLM with a multilingual speech encoder, aiming to harness the capabilities of LLMs for speech recognition and beyond. Utilizing a multi-instructional training approach, we demonstrate the transferability of linguistic knowledge from the text to the speech modality. Our experiments, conducted on 1900 hours of transcribed data from 139 languages, establish that a multilingual speech representation can be effectively learned and aligned with a multilingual LLM. While this learned representation initially shows limitations in task generalization, we address this issue by generating synthetic targets in a multi-instructional style. Our zero-shot evaluation results confirm the robustness of our approach across multiple tasks, including speech translation and multilingual spoken language understanding, thereby opening new avenues for applying LLMs in the speech domain.

Title: A Concise Tiling Strategy for Preserving Spatial Context in Earth Observation Imagery

• Authors: Ellianna Abrahams, Tasha Snow, Matthew R. Siegfried, Fernando Pérez
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.10927
• Pdf URL: https://arxiv.org/pdf/2404.10927
• Copy Paste: [[2404.10927]] A Concise Tiling Strategy for Preserving Spatial Context in Earth Observation Imagery(https://arxiv.org/abs/2404.10927)
• Keywords: segmentation
• Abstract: We propose a new tiling strategy, Flip-n-Slide, which has been developed for specific use with large Earth observation satellite images when the location of objects-of-interest (OoI) is unknown and spatial context can be necessary for class disambiguation. Flip-n-Slide is a concise and minimalistic approach that allows OoI to be represented at multiple tile positions and orientations. This strategy introduces multiple views of spatio-contextual information, without introducing redundancies into the training set. By maintaining distinct transformation permutations for each tile overlap, we enhance the generalizability of the training set without misrepresenting the true data distribution. Our experiments validate the effectiveness of Flip-n-Slide in the task of semantic segmentation, a necessary data product in geophysical studies. We find that Flip-n-Slide outperforms the previous state-of-the-art augmentation routines for tiled data in all evaluation metrics. For underrepresented classes, Flip-n-Slide increases precision by as much as 15.8%.

Title: Shears: Unstructured Sparsity with Neural Low-rank Adapter Search

• Authors: J. Pablo Muñoz, Jinjie Yuan, Nilesh Jain
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.10934
• Pdf URL: https://arxiv.org/pdf/2404.10934
• Copy Paste: [[2404.10934]] Shears: Unstructured Sparsity with Neural Low-rank Adapter Search(https://arxiv.org/abs/2404.10934)
• Keywords: large language model
• Abstract: Recently, several approaches successfully demonstrated that weight-sharing Neural Architecture Search (NAS) can effectively explore a search space of elastic low-rank adapters (LoRA), allowing the parameter-efficient fine-tuning (PEFT) and compression of large language models. In this paper, we introduce a novel approach called Shears, demonstrating how the integration of cost-effective sparsity and a proposed Neural Low-rank adapter Search (NLS) algorithm can further improve the efficiency of PEFT approaches. Results demonstrate the benefits of Shears compared to other methods, reaching high sparsity levels while improving or with little drop in accuracy, utilizing a single GPU for a pair of hours.

Title: Neuromorphic Vision-based Motion Segmentation with Graph Transformer Neural Network

• Authors: Yusra Alkendi, Rana Azzam, Sajid Javed, Lakmal Seneviratne, Yahya Zweiri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.10940
• Pdf URL: https://arxiv.org/pdf/2404.10940
• Copy Paste: [[2404.10940]] Neuromorphic Vision-based Motion Segmentation with Graph Transformer Neural Network(https://arxiv.org/abs/2404.10940)
• Keywords: transformer, segmentation
• Abstract: Moving object segmentation is critical to interpret scene dynamics for robotic navigation systems in challenging environments. Neuromorphic vision sensors are tailored for motion perception due to their asynchronous nature, high temporal resolution, and reduced power consumption. However, their unconventional output requires novel perception paradigms to leverage their spatially sparse and temporally dense nature. In this work, we propose a novel event-based motion segmentation algorithm using a Graph Transformer Neural Network, dubbed GTNN. Our proposed algorithm processes event streams as 3D graphs by a series of nonlinear transformations to unveil local and global spatiotemporal correlations between events. Based on these correlations, events belonging to moving objects are segmented from the background without prior knowledge of the dynamic scene geometry. The algorithm is trained on publicly available datasets including MOD, EV-IMO, and \textcolor{black}{EV-IMO2} using the proposed training scheme to facilitate efficient training on extensive datasets. Moreover, we introduce the Dynamic Object Mask-aware Event Labeling (DOMEL) approach for generating approximate ground-truth labels for event-based motion segmentation datasets. We use DOMEL to label our own recorded Event dataset for Motion Segmentation (EMS-DOMEL), which we release to the public for further research and benchmarking. Rigorous experiments are conducted on several unseen publicly-available datasets where the results revealed that GTNN outperforms state-of-the-art methods in the presence of dynamic background variations, motion patterns, and multiple dynamic objects with varying sizes and velocities. GTNN achieves significant performance gains with an average increase of 9.4% and 4.5% in terms of motion segmentation accuracy (IoU%) and detection rate (DR%), respectively.

Title: What Hides behind Unfairness? Exploring Dynamics Fairness in Reinforcement Learning

• Authors: Zhihong Deng, Jing Jiang, Guodong Long, Chengqi Zhang
• Subjects: cs.LG, cs.AI, cs.CY, stat.ME
• Abstract URL: https://arxiv.org/abs/2404.10942
• Pdf URL: https://arxiv.org/pdf/2404.10942
• Copy Paste: [[2404.10942]] What Hides behind Unfairness? Exploring Dynamics Fairness in Reinforcement Learning(https://arxiv.org/abs/2404.10942)
• Keywords: fair
• Abstract: In sequential decision-making problems involving sensitive attributes like race and gender, reinforcement learning (RL) agents must carefully consider long-term fairness while maximizing returns. Recent works have proposed many different types of fairness notions, but how unfairness arises in RL problems remains unclear. In this paper, we address this gap in the literature by investigating the sources of inequality through a causal lens. We first analyse the causal relationships governing the data generation process and decompose the effect of sensitive attributes on long-term well-being into distinct components. We then introduce a novel notion called dynamics fairness, which explicitly captures the inequality stemming from environmental dynamics, distinguishing it from those induced by decision-making or inherited from the past. This notion requires evaluating the expected changes in the next state and the reward induced by changing the value of the sensitive attribute while holding everything else constant. To quantitatively evaluate this counterfactual concept, we derive identification formulas that allow us to obtain reliable estimations from data. Extensive experiments demonstrate the effectiveness of the proposed techniques in explaining, detecting, and reducing inequality in reinforcement learning.

Title: Personalized Federated Learning via Stacking

• Authors: Emilio Cantu-Cervini
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.10957
• Pdf URL: https://arxiv.org/pdf/2404.10957
• Copy Paste: [[2404.10957]] Personalized Federated Learning via Stacking(https://arxiv.org/abs/2404.10957)
• Keywords: privacy, federate
• Abstract: Traditional Federated Learning (FL) methods typically train a single global model collaboratively without exchanging raw data. In contrast, Personalized Federated Learning (PFL) techniques aim to create multiple models that are better tailored to individual clients' data. We present a novel personalization approach based on stacked generalization where clients directly send each other privacy-preserving models to be used as base models to train a meta-model on private data. Our approach is flexible, accommodating various privacy-preserving techniques and model types, and can be applied in horizontal, hybrid, and vertically partitioned federations. Additionally, it offers a natural mechanism for assessing each client's contribution to the federation. Through comprehensive evaluations across diverse simulated data heterogeneity scenarios, we showcase the effectiveness of our method.

Title: Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations

• Authors: Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, Mark Ibrahim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.10960
• Pdf URL: https://arxiv.org/pdf/2404.10960
• Copy Paste: [[2404.10960]] Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations(https://arxiv.org/abs/2404.10960)
• Keywords: large language model
• Abstract: A major barrier towards the practical deployment of large language models (LLMs) is their lack of reliability. Three situations where this is particularly apparent are correctness, hallucinations when given unanswerable questions, and safety. In all three cases, models should ideally abstain from responding, much like humans, whose ability to understand uncertainty makes us refrain from answering questions we don't know. Inspired by analogous approaches in classification, this study explores the feasibility and efficacy of abstaining while uncertain in the context of LLMs within the domain of question-answering. We investigate two kinds of uncertainties, statistical uncertainty metrics and a distinct verbalized measure, termed as In-Dialogue Uncertainty (InDU). Using these uncertainty measures combined with models with and without Reinforcement Learning with Human Feedback (RLHF), we show that in all three situations, abstention based on the right kind of uncertainty measure can boost the reliability of LLMs. By sacrificing only a few highly uncertain samples we can improve correctness by 2% to 8%, avoid 50% hallucinations via correctly identifying unanswerable questions and increase safety by 70% up to 99% with almost no additional computational overhead.

Title: Domain-Specific Block Selection and Paired-View Pseudo-Labeling for Online Test-Time Adaptation

• Authors: Yeonguk Yu, Sungho Shin, Seunghyeok Back, Minhwan Ko, Sangjun Noh, Kyoobin Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.10966
• Pdf URL: https://arxiv.org/pdf/2404.10966
• Copy Paste: [[2404.10966]] Domain-Specific Block Selection and Paired-View Pseudo-Labeling for Online Test-Time Adaptation(https://arxiv.org/abs/2404.10966)
• Keywords: extraction
• Abstract: Test-time adaptation (TTA) aims to adapt a pre-trained model to a new test domain without access to source data after deployment. Existing approaches typically rely on self-training with pseudo-labels since ground-truth cannot be obtained from test data. Although the quality of pseudo labels is important for stable and accurate long-term adaptation, it has not been previously addressed. In this work, we propose DPLOT, a simple yet effective TTA framework that consists of two components: (1) domain-specific block selection and (2) pseudo-label generation using paired-view images. Specifically, we select blocks that involve domain-specific feature extraction and train these blocks by entropy minimization. After blocks are adjusted for current test domain, we generate pseudo-labels by averaging given test images and corresponding flipped counterparts. By simply using flip augmentation, we prevent a decrease in the quality of the pseudo-labels, which can be caused by the domain gap resulting from strong augmentation. Our experimental results demonstrate that DPLOT outperforms previous TTA methods in CIFAR10-C, CIFAR100-C, and ImageNet-C benchmarks, reducing error by up to 5.4%, 9.1%, and 2.9%, respectively. Also, we provide an extensive analysis to demonstrate effectiveness of our framework. Code is available at https://github.com/gist-ailab/domain-specific-block-selection-and-paired-view-pseudo-labeling-for-online-TTA.

Title: Leveraging 3D LiDAR Sensors to Enable Enhanced Urban Safety and Public Health: Pedestrian Monitoring and Abnormal Activity Detection

• Authors: Nawfal Guefrachi, Jian Shi, Hakim Ghazzai, Ahmad Alsharoa
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.10978
• Pdf URL: https://arxiv.org/pdf/2404.10978
• Copy Paste: [[2404.10978]] Leveraging 3D LiDAR Sensors to Enable Enhanced Urban Safety and Public Health: Pedestrian Monitoring and Abnormal Activity Detection(https://arxiv.org/abs/2404.10978)
• Keywords: robust
• Abstract: The integration of Light Detection and Ranging (LiDAR) and Internet of Things (IoT) technologies offers transformative opportunities for public health informatics in urban safety and pedestrian well-being. This paper proposes a novel framework utilizing these technologies for enhanced 3D object detection and activity classification in urban traffic scenarios. By employing elevated LiDAR, we obtain detailed 3D point cloud data, enabling precise pedestrian activity monitoring. To overcome urban data scarcity, we create a specialized dataset through simulated traffic environments in Blender, facilitating targeted model training. Our approach employs a modified Point Voxel-Region-based Convolutional Neural Network (PV-RCNN) for robust 3D detection and PointNet for classifying pedestrian activities, significantly benefiting urban traffic management and public health by offering insights into pedestrian behavior and promoting safer urban environments. Our dual-model approach not only enhances urban traffic management but also contributes significantly to public health by providing insights into pedestrian behavior and promoting safer urban environment.

Title: Graph Continual Learning with Debiased Lossless Memory Replay

• Authors: Chaoxi Niu, Guansong Pang, Ling Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.10984
• Pdf URL: https://arxiv.org/pdf/2404.10984
• Copy Paste: [[2404.10984]] Graph Continual Learning with Debiased Lossless Memory Replay(https://arxiv.org/abs/2404.10984)
• Keywords: privacy
• Abstract: Real-life graph data often expands continually, rendering the learning of graph neural networks (GNNs) on static graph data impractical. Graph continual learning (GCL) tackles this problem by continually adapting GNNs to the expanded graph of the current task while maintaining the performance over the graph of previous tasks. Memory replay-based methods, which aim to replay data of previous tasks when learning new tasks, have been explored as one principled approach to mitigate the forgetting of the knowledge learned from the previous tasks. In this paper we extend this methodology with a novel framework, called Debiased Lossless Memory replay (DeLoMe). Unlike existing methods that sample nodes/edges of previous graphs to construct the memory, DeLoMe learns small lossless synthetic node representations as the memory. The learned memory can not only preserve the graph data privacy but also capture the holistic graph information, for which the sampling-based methods are not viable. Further, prior methods suffer from bias toward the current task due to the data imbalance between the classes in the memory data and the current data. A debiased GCL loss function is devised in DeLoMe to effectively alleviate this bias. Extensive experiments on four graph datasets show the effectiveness of DeLoMe under both class- and task-incremental learning settings.

Title: FairSSD: Understanding Bias in Synthetic Speech Detectors

• Authors: Amit Kumar Singh Yadav, Kratika Bhagtani, Davide Salvi, Paolo Bestagini, Edward J.Delp
• Subjects: cs.CV, cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.10989
• Pdf URL: https://arxiv.org/pdf/2404.10989
• Copy Paste: [[2404.10989]] FairSSD: Understanding Bias in Synthetic Speech Detectors(https://arxiv.org/abs/2404.10989)
• Keywords: robust, fair
• Abstract: Methods that can generate synthetic speech which is perceptually indistinguishable from speech recorded by a human speaker, are easily available. Several incidents report misuse of synthetic speech generated from these methods to commit fraud. To counter such misuse, many methods have been proposed to detect synthetic speech. Some of these detectors are more interpretable, can generalize to detect synthetic speech in the wild and are robust to noise. However, limited work has been done on understanding bias in these detectors. In this work, we examine bias in existing synthetic speech detectors to determine if they will unfairly target a particular gender, age and accent group. We also inspect whether these detectors will have a higher misclassification rate for bona fide speech from speech-impaired speakers w.r.t fluent speakers. Extensive experiments on 6 existing synthetic speech detectors using more than 0.9 million speech signals demonstrate that most detectors are gender, age and accent biased, and future work is needed to ensure fairness. To support future research, we release our evaluation dataset, models used in our study and source code at https://gitlab.com/viper-purdue/fairssd.

Title: AKGNet: Attribute Knowledge-Guided Unsupervised Lung-Infected Area Segmentation

• Authors: Qing En, Yuhong Guo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11008
• Pdf URL: https://arxiv.org/pdf/2404.11008
• Copy Paste: [[2404.11008]] AKGNet: Attribute Knowledge-Guided Unsupervised Lung-Infected Area Segmentation(https://arxiv.org/abs/2404.11008)
• Keywords: segmentation
• Abstract: Lung-infected area segmentation is crucial for assessing the severity of lung diseases. However, existing image-text multi-modal methods typically rely on labour-intensive annotations for model training, posing challenges regarding time and expertise. To address this issue, we propose a novel attribute knowledge-guided framework for unsupervised lung-infected area segmentation (AKGNet), which achieves segmentation solely based on image-text data without any mask annotation. AKGNet facilitates text attribute knowledge learning, attribute-image cross-attention fusion, and high-confidence-based pseudo-label exploration simultaneously. It can learn statistical information and capture spatial correlations between image and text attributes in the embedding space, iteratively refining the mask to enhance segmentation. Specifically, we introduce a text attribute knowledge learning module by extracting attribute knowledge and incorporating it into feature representations, enabling the model to learn statistical information and adapt to different attributes. Moreover, we devise an attribute-image cross-attention module by calculating the correlation between attributes and images in the embedding space to capture spatial dependency information, thus selectively focusing on relevant regions while filtering irrelevant areas. Finally, a self-training mask improvement process is employed by generating pseudo-labels using high-confidence predictions to iteratively enhance the mask and segmentation. Experimental results on a benchmark medical image dataset demonstrate the superior performance of our method compared to state-of-the-art segmentation techniques in unsupervised scenarios.

Title: FedFa: A Fully Asynchronous Training Paradigm for Federated Learning

• Authors: Haotian Xu, Zhaorui Zhang, Sheng Di, Benben Liu, Alharthi Khalid, Jiannong Cao
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.11015
• Pdf URL: https://arxiv.org/pdf/2404.11015
• Copy Paste: [[2404.11015]] FedFa: A Fully Asynchronous Training Paradigm for Federated Learning(https://arxiv.org/abs/2404.11015)
• Keywords: privacy, federate
• Abstract: Federated learning has been identified as an efficient decentralized training paradigm for scaling the machine learning model training on a large number of devices while guaranteeing the data privacy of the trainers. FedAvg has become a foundational parameter update strategy for federated learning, which has been promising to eliminate the effect of the heterogeneous data across clients and guarantee convergence. However, the synchronization parameter update barriers for each communication round during the training significant time on waiting, slowing down the training procedure. Therefore, recent state-of-the-art solutions propose using semi-asynchronous approaches to mitigate the waiting time cost with guaranteed convergence. Nevertheless, emerging semi-asynchronous approaches are unable to eliminate the waiting time completely. We propose a full asynchronous training paradigm, called FedFa, which can guarantee model convergence and eliminate the waiting time completely for federated learning by using a few buffered results on the server for parameter updating. Further, we provide theoretical proof of the convergence rate for our proposed FedFa. Extensive experimental results indicate our approach effectively improves the training performance of federated learning by up to 6x and 4x speedup compared to the state-of-the-art synchronous and semi-asynchronous strategies while retaining high accuracy in both IID and Non-IID scenarios.

Title: MaeFuse: Transferring Omni Features with Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training

• Authors: Jiayang Li, Junjun Jiang, Pengwei Liang, Jiayi Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11016
• Pdf URL: https://arxiv.org/pdf/2404.11016
• Copy Paste: [[2404.11016]] MaeFuse: Transferring Omni Features with Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training(https://arxiv.org/abs/2404.11016)
• Keywords: extraction
• Abstract: In this research, we introduce MaeFuse, a novel autoencoder model designed for infrared and visible image fusion (IVIF). The existing approaches for image fusion often rely on training combined with downstream tasks to obtain high-level visual information, which is effective in emphasizing target objects and delivering impressive results in visual quality and task-specific applications. MaeFuse, however, deviates from the norm. Instead of being driven by downstream tasks, our model utilizes a pretrained encoder from Masked Autoencoders (MAE), which facilities the omni features extraction for low-level reconstruction and high-level vision tasks, to obtain perception friendly features with a low cost. In order to eliminate the domain gap of different modal features and the block effect caused by the MAE encoder, we further develop a guided training strategy. This strategy is meticulously crafted to ensure that the fusion layer seamlessly adjusts to the feature space of the encoder, gradually enhancing the fusion effect. It facilitates the comprehensive integration of feature vectors from both infrared and visible modalities, preserving the rich details inherent in each. MaeFuse not only introduces a novel perspective in the realm of fusion techniques but also stands out with impressive performance across various public datasets.

Title: Many-Shot In-Context Learning

• Authors: Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, Hugo Larochelle
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11018
• Pdf URL: https://arxiv.org/pdf/2404.11018
• Copy Paste: [[2404.11018]] Many-Shot In-Context Learning(https://arxiv.org/abs/2404.11018)
• Keywords: generative, large language model
• Abstract: Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, without any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases and can learn high-dimensional functions with numerical inputs. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.

Title: CORE: Data Augmentation for Link Prediction via Information Bottleneck

• Authors: Kaiwen Dong, Zhichun Guo, Nitesh V. Chawla
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.11032
• Pdf URL: https://arxiv.org/pdf/2404.11032
• Copy Paste: [[2404.11032]] CORE: Data Augmentation for Link Prediction via Information Bottleneck(https://arxiv.org/abs/2404.11032)
• Keywords: robust
• Abstract: Link prediction (LP) is a fundamental task in graph representation learning, with numerous applications in diverse domains. However, the generalizability of LP models is often compromised due to the presence of noisy or spurious information in graphs and the inherent incompleteness of graph data. To address these challenges, we draw inspiration from the Information Bottleneck principle and propose a novel data augmentation method, COmplete and REduce (CORE) to learn compact and predictive augmentations for LP models. In particular, CORE aims to recover missing edges in graphs while simultaneously removing noise from the graph structures, thereby enhancing the model's robustness and performance. Extensive experiments on multiple benchmark datasets demonstrate the applicability and superiority of CORE over state-of-the-art methods, showcasing its potential as a leading approach for robust LP in graph representation learning.

Title: Offset Unlearning for Large Language Models

• Authors: James Y. Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, Muhao Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11045
• Pdf URL: https://arxiv.org/pdf/2404.11045
• Copy Paste: [[2404.11045]] Offset Unlearning for Large Language Models(https://arxiv.org/abs/2404.11045)
• Keywords: protect, large language model
• Abstract: Despite the strong capabilities of Large Language Models (LLMs) to acquire knowledge from their training corpora, the memorization of sensitive information in the corpora such as copyrighted, harmful, and private content has led to ethical and legal concerns. In response to these challenges, unlearning has emerged as a potential remedy for LLMs affected by problematic training data. However, previous unlearning techniques are either not applicable to black-box LLMs due to required access to model internal weights, or violate data protection principles by retaining sensitive data for inference-time correction. We propose $\delta$-unlearning, an offset unlearning framework for black-box LLMs. Instead of tuning the black-box LLM itself, $\delta$-unlearning learns the logit offset needed for unlearning by contrasting the logits from a pair of smaller models. Experiments demonstrate that $\delta$-unlearning can effectively unlearn target data while maintaining similar or even stronger performance on general out-of-forget-scope tasks. $\delta$-unlearning also effectively incorporates different unlearning algorithms, making our approach a versatile solution to adapting various existing unlearning algorithms to black-box LLMs.

Title: Stepwise Alignment for Constrained Language Model Policy Optimization

• Authors: Akifumi Wachi, Thien Q Tran, Rei Sato, Takumi Tanabe, Yohei Akimoto
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11049
• Pdf URL: https://arxiv.org/pdf/2404.11049
• Copy Paste: [[2404.11049]] Stepwise Alignment for Constrained Language Model Policy Optimization(https://arxiv.org/abs/2404.11049)
• Keywords: large language model
• Abstract: Safety and trustworthiness are indispensable requirements for applying AI systems based on large language models (LLMs) in real-world applications. This paper formulates a human value alignment as a language model policy optimization problem to maximize reward under a safety constraint and then proposes an algorithm called Stepwise Alignment for Constrained Policy Optimization (SACPO). A key idea behind SACPO, supported by theory, is that the optimal policy incorporating both reward and safety can be directly obtained from a reward-aligned policy. Based on this key idea, SACPO aligns the LLMs with each metric step-wise while leveraging simple yet powerful alignment algorithms such as direct preference optimization (DPO). SACPO provides many benefits such as simplicity, stability, computational efficiency, and flexibility regarding algorithms and dataset selection. Under mild assumption, our theoretical analysis provides the upper bounds regarding near-optimality and safety constraint violation. Our experimental results show that SACPO can fine-tune Alpaca-7B better than the state-of-the-art method in terms of both helpfulness and harmlessness

Title: WPS-Dataset: A benchmark for wood plate segmentation in bark removal processing

• Authors: Rijun Wang, Guanghao Zhang, Fulong Liang, Bo Wang, Xiangwei Mou, Yesheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11051
• Pdf URL: https://arxiv.org/pdf/2404.11051
• Copy Paste: [[2404.11051]] WPS-Dataset: A benchmark for wood plate segmentation in bark removal processing(https://arxiv.org/abs/2404.11051)
• Keywords: segmentation
• Abstract: Using deep learning methods is a promising approach to improving bark removal efficiency and enhancing the quality of wood products. However, the lack of publicly available datasets for wood plate segmentation in bark removal processing poses challenges for researchers in this field. To address this issue, a benchmark for wood plate segmentation in bark removal processing named WPS-dataset is proposed in this study, which consists of 4863 images. We designed an image acquisition device and assembled it on a bark removal equipment to capture images in real industrial settings. We evaluated the WPS-dataset using six typical segmentation models. The models effectively learn and understand the WPS-dataset characteristics during training, resulting in high performance and accuracy in wood plate segmentation tasks. We believe that our dataset can lay a solid foundation for future research in bark removal processing and contribute to advancements in this field.

Title: Supervised Contrastive Vision Transformer for Breast Histopathological Image Classification

• Authors: Mohammad Shiri, Jiangwen Sun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11052
• Pdf URL: https://arxiv.org/pdf/2404.11052
• Copy Paste: [[2404.11052]] Supervised Contrastive Vision Transformer for Breast Histopathological Image Classification(https://arxiv.org/abs/2404.11052)
• Keywords: transformer
• Abstract: Invasive ductal carcinoma (IDC) is the most prevalent form of breast cancer. Breast tissue histopathological examination is critical in diagnosing and classifying breast cancer. Although existing methods have shown promising results, there is still room for improvement in the classification accuracy and generalization of IDC using histopathology images. We present a novel approach, Supervised Contrastive Vision Transformer (SupCon-ViT), for improving the classification of invasive ductal carcinoma in terms of accuracy and generalization by leveraging the inherent strengths and advantages of both transfer learning, i.e., pre-trained vision transformer, and supervised contrastive learning. Our results on a benchmark breast cancer dataset demonstrate that SupCon-Vit achieves state-of-the-art performance in IDC classification, with an F1-score of 0.8188, precision of 0.7692, and specificity of 0.8971, outperforming existing methods. In addition, the proposed model demonstrates resilience in scenarios with minimal labeled data, making it highly efficient in real-world clinical settings where labelled data is limited. Our findings suggest that supervised contrastive learning in conjunction with pre-trained vision transformers appears to be a viable strategy for an accurate classification of IDC, thus paving the way for a more efficient and reliable diagnosis of breast cancer through histopathological image analysis.

Title: Multilateral Temporal-view Pyramid Transformer for Video Inpainting Detection

• Authors: Ying Zhang, Bo Peng, Jiaran Zhou, Huiyu Zhou, Junyu Dong, Yuezun Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11054
• Pdf URL: https://arxiv.org/pdf/2404.11054
• Copy Paste: [[2404.11054]] Multilateral Temporal-view Pyramid Transformer for Video Inpainting Detection(https://arxiv.org/abs/2404.11054)
• Keywords: transformer
• Abstract: The task of video inpainting detection is to expose the pixel-level inpainted regions within a video sequence. Existing methods usually focus on leveraging spatial and temporal inconsistencies. However, these methods typically employ fixed operations to combine spatial and temporal clues, limiting their applicability in different scenarios. In this paper, we introduce a novel Multilateral Temporal-view Pyramid Transformer ({\em MumPy}) that collaborates spatial-temporal clues flexibly. Our method utilizes a newly designed multilateral temporal-view encoder to extract various collaborations of spatial-temporal clues and introduces a deformable window-based temporal-view interaction module to enhance the diversity of these collaborations. Subsequently, we develop a multi-pyramid decoder to aggregate the various types of features and generate detection maps. By adjusting the contribution strength of spatial and temporal clues, our method can effectively identify inpainted regions. We validate our method on existing datasets and also introduce a new challenging and large-scale Video Inpainting dataset based on the YouTube-VOS dataset, which employs several more recent inpainting methods. The results demonstrate the superiority of our method in both in-domain and cross-domain evaluation scenarios.

Title: LMEraser: Large Model Unlearning through Adaptive Prompt Tuning

• Authors: Jie Xu, Zihan Wu, Cong Wang, Xiaohua Jia
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.11056
• Pdf URL: https://arxiv.org/pdf/2404.11056
• Copy Paste: [[2404.11056]] LMEraser: Large Model Unlearning through Adaptive Prompt Tuning(https://arxiv.org/abs/2404.11056)
• Keywords: privacy, protect
• Abstract: To address the growing demand for privacy protection in machine learning, we propose a novel and efficient machine unlearning approach for \textbf{L}arge \textbf{M}odels, called \textbf{LM}Eraser. Existing unlearning research suffers from entangled training data and complex model architectures, incurring extremely high computational costs for large models. LMEraser takes a divide-and-conquer strategy with a prompt tuning architecture to isolate data influence. The training dataset is partitioned into public and private datasets. Public data are used to train the backbone of the model. Private data are adaptively clustered based on their diversity, and each cluster is used to optimize a prompt separately. This adaptive prompt tuning mechanism reduces unlearning costs and maintains model performance. Experiments demonstrate that LMEraser achieves a $100$-fold reduction in unlearning costs without compromising accuracy compared to prior work. Our code is available at: \url{https://github.com/lmeraser/lmeraser}.

Title: Sky-GVIO: an enhanced GNSS/INS/Vision navigation with FCN-based sky-segmentation in urban canyon

• Authors: Jingrong Wang, Bo Xu, Ronghe Jin, Shoujian Zhang, Kefu Gao, Jingnan Liu
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.11070
• Pdf URL: https://arxiv.org/pdf/2404.11070
• Copy Paste: [[2404.11070]] Sky-GVIO: an enhanced GNSS/INS/Vision navigation with FCN-based sky-segmentation in urban canyon(https://arxiv.org/abs/2404.11070)
• Keywords: segmentation
• Abstract: Accurate, continuous, and reliable positioning is a critical component of achieving autonomous driving. However, in complex urban canyon environments, the vulnerability of a stand-alone sensor and non-line-of-sight (NLOS) caused by high buildings, trees, and elevated structures seriously affect positioning results. To address these challenges, a sky-view images segmentation algorithm based on Fully Convolutional Network (FCN) is proposed for GNSS NLOS detection. Building upon this, a novel NLOS detection and mitigation algorithm (named S-NDM) is extended to the tightly coupled Global Navigation Satellite Systems (GNSS), Inertial Measurement Units (IMU), and visual feature system which is called Sky-GVIO, with the aim of achieving continuous and accurate positioning in urban canyon environments. Furthermore, the system harmonizes Single Point Positioning (SPP) with Real-Time Kinematic (RTK) methodologies to bolster its operational versatility and resilience. In urban canyon environments, the positioning performance of S-NDM algorithm proposed in this paper is evaluated under different tightly coupled SPP-related and RTK-related models. The results exhibit that Sky-GVIO system achieves meter-level accuracy under SPP mode and sub-decimeter precision with RTK, surpassing the performance of GNSS/INS/Vision frameworks devoid of S-NDM. Additionally, the sky-view image dataset, inclusive of training and evaluation subsets, has been made publicly accessible for scholarly exploration at https://github.com/whuwangjr/sky-view-images .

Title: ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models

• Authors: Trong-Hieu Nguyen, Anh-Cuong Le, Viet-Cuong Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11086
• Pdf URL: https://arxiv.org/pdf/2404.11086
• Copy Paste: [[2404.11086]] ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models(https://arxiv.org/abs/2404.11086)
• Keywords: large language model
• Abstract: The rapid advancement of large language models (LLMs) necessitates the development of new benchmarks to accurately assess their capabilities. To address this need for Vietnamese, this work aims to introduce ViLLM-Eval, the comprehensive evaluation suite designed to measure the advanced knowledge and reasoning abilities of foundation models within a Vietnamese context. ViLLM-Eval consists of multiple-choice questions and predict next word tasks spanning various difficulty levels and diverse disciplines, ranging from humanities to science and engineering. A thorough evaluation of the most advanced LLMs on ViLLM-Eval revealed that even the best performing models have significant room for improvement in understanding and responding to Vietnamese language tasks. ViLLM-Eval is believed to be instrumental in identifying key strengths and weaknesses of foundation models, ultimately promoting their development and enhancing their performance for Vietnamese users.

Title: Inductive-Deductive Strategy Reuse for Multi-Turn Instructional Dialogues

• Authors: Jiao Ou, Jiayu Wu, Che Liu, Fuzheng Zhang, Di Zhang, Kun Gai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11095
• Pdf URL: https://arxiv.org/pdf/2404.11095
• Copy Paste: [[2404.11095]] Inductive-Deductive Strategy Reuse for Multi-Turn Instructional Dialogues(https://arxiv.org/abs/2404.11095)
• Keywords: large language model
• Abstract: Aligning large language models (LLMs) with human expectations requires high-quality instructional dialogues, which can be achieved by raising diverse, in-depth, and insightful instructions that deepen interactions. Existing methods target instructions from real instruction dialogues as a learning goal and fine-tune a user simulator for posing instructions. However, the user simulator struggles to implicitly model complex dialogue flows and pose high-quality instructions. In this paper, we take inspiration from the cognitive abilities inherent in human learning and propose the explicit modeling of complex dialogue flows through instructional strategy reuse. Specifically, we first induce high-level strategies from various real instruction dialogues. These strategies are applied to new dialogue scenarios deductively, where the instructional strategies facilitate high-quality instructions. Experimental results show that our method can generate diverse, in-depth, and insightful instructions for a given dialogue history. The constructed multi-turn instructional dialogues can outperform competitive baselines on the downstream chat model.

Title: LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models

• Authors: Dingkun Zhang, Sijia Li, Chen Chen, Qingsong Xie, Haonan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11098
• Pdf URL: https://arxiv.org/pdf/2404.11098
• Copy Paste: [[2404.11098]] LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models(https://arxiv.org/abs/2404.11098)
• Keywords: diffusion
• Abstract: In the era of AIGC, the demand for low-budget or even on-device applications of diffusion models emerged. In terms of compressing the Stable Diffusion models (SDMs), several approaches have been proposed, and most of them leveraged the handcrafted layer removal methods to obtain smaller U-Nets, along with knowledge distillation to recover the network performance. However, such a handcrafting manner of layer removal is inefficient and lacks scalability and generalization, and the feature distillation employed in the retraining phase faces an imbalance issue that a few numerically significant feature loss terms dominate over others throughout the retraining process. To this end, we proposed the layer pruning and normalized distillation for compressing diffusion models (LAPTOP-Diff). We, 1) introduced the layer pruning method to compress SDM's U-Net automatically and proposed an effective one-shot pruning criterion whose one-shot performance is guaranteed by its good additivity property, surpassing other layer pruning and handcrafted layer removal methods, 2) proposed the normalized feature distillation for retraining, alleviated the imbalance issue. Using the proposed LAPTOP-Diff, we compressed the U-Nets of SDXL and SDM-v1.5 for the most advanced performance, achieving a minimal 4.0% decline in PickScore at a pruning ratio of 50% while the comparative methods' minimal PickScore decline is 8.2%. We will release our code.

Title: KernJC: Automated Vulnerable Environment Generation for Linux Kernel Vulnerabilities

• Authors: Bonan Ruan, Jiahao Liu, Chuqi Zhang, Zhenkai Liang
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.11107
• Pdf URL: https://arxiv.org/pdf/2404.11107
• Copy Paste: [[2404.11107]] KernJC: Automated Vulnerable Environment Generation for Linux Kernel Vulnerabilities(https://arxiv.org/abs/2404.11107)
• Keywords: security
• Abstract: Linux kernel vulnerability reproduction is a critical task in system security. To reproduce a kernel vulnerability, the vulnerable environment and the Proof of Concept (PoC) program are needed. Most existing research focuses on the generation of PoC, while the construction of environment is overlooked. However, establishing an effective vulnerable environment to trigger a vulnerability is challenging. Firstly, it is hard to guarantee that the selected kernel version for reproduction is vulnerable, as the vulnerability version claims in online databases can occasionally be spurious. Secondly, many vulnerabilities can not be reproduced in kernels built with default configurations. Intricate non-default kernel configurations must be set to include and trigger a kernel vulnerability, but less information is available on how to recognize these configurations. To solve these challenges, we propose a patch-based approach to identify real vulnerable kernel versions and a graph-based approach to identify necessary configs for activating a specific vulnerability. We implement these approaches in a tool, KernJC, automating the generation of vulnerable environments for kernel vulnerabilities. To evaluate the efficacy of KernJC, we build a dataset containing 66 representative real-world vulnerabilities with PoCs from kernel vulnerability research in the past five years. The evaluation shows that KernJC builds vulnerable environments for all these vulnerabilities, 48.5% of which require non-default configs, and 4 have incorrect version claims in the National Vulnerability Database (NVD). Furthermore, we conduct large-scale spurious version detection on kernel vulnerabilities and identify 128 vulnerabilities which have spurious version claims in NVD. To foster future research, we release KernJC with the dataset in the community.

Title: Consistency Training by Synthetic Question Generation for Conversational Question Answering

• Authors: Hamed Hematian Hemati, Hamid Beigy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11109
• Pdf URL: https://arxiv.org/pdf/2404.11109
• Copy Paste: [[2404.11109]] Consistency Training by Synthetic Question Generation for Conversational Question Answering(https://arxiv.org/abs/2404.11109)
• Keywords: robust
• Abstract: Efficiently modeling historical information is a critical component in addressing user queries within a conversational question-answering (QA) context, as historical context plays a vital role in clarifying the user's questions. However, irrelevant history induces noise in the reasoning process, especially for those questions with a considerable historical context. In our novel model-agnostic approach, referred to as CoTaH (Consistency-Trained augmented History), we augment the historical information with synthetic questions and subsequently employ consistency training to train a model that utilizes both real and augmented historical data to implicitly make the reasoning robust to irrelevant history. To the best of our knowledge, this is the first instance of research using question generation as a form of data augmentation to model conversational QA settings. By citing a common modeling error prevalent in previous research, we introduce a new baseline model and compare our model's performance against it, demonstrating an improvement in results, particularly when dealing with questions that include a substantial amount of historical context. The source code can be found on our GitHub page.

Title: CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation

• Authors: Lianyu Hu, Wei Feng, Liqing Gao, Zekang Liu, Liang Wan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11111
• Pdf URL: https://arxiv.org/pdf/2404.11111
• Copy Paste: [[2404.11111]] CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation(https://arxiv.org/abs/2404.11111)
• Keywords: extraction
• Abstract: In sign language, the conveyance of human body trajectories predominantly relies upon the coordinated movements of hands and facial expressions across successive frames. Despite the recent advancements of sign language understanding methods, they often solely focus on individual frames, inevitably overlooking the inter-frame correlations that are essential for effectively modeling human body trajectories. To address this limitation, this paper introduces a spatial-temporal correlation network, denoted as CorrNet+, which explicitly identifies body trajectories across multiple frames. In specific, CorrNet+ employs a correlation module and an identification module to build human body trajectories. Afterwards, a temporal attention module is followed to adaptively evaluate the contributions of different frames. The resultant features offer a holistic perspective on human body movements, facilitating a deeper understanding of sign language. As a unified model, CorrNet+ achieves new state-of-the-art performance on two extensive sign language understanding tasks, including continuous sign language recognition (CSLR) and sign language translation (SLT). Especially, CorrNet+ surpasses previous methods equipped with resource-intensive pose-estimation networks or pre-extracted heatmaps for hand and facial feature extraction. Compared with CorrNet, CorrNet+ achieves a significant performance boost across all benchmarks while halving the computational overhead. A comprehensive comparison with previous spatial-temporal reasoning methods verifies the superiority of CorrNet+. Code is available at https://github.com/hulianyuyy/CorrNet_Plus.

Title: TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing

• Authors: Sherry X. Chen, Yaron Vaxman, Elad Ben Baruch, David Asulin, Aviad Moreshet, Kuo-Chin Lien, Misha Sra, Pradeep Sen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11120
• Pdf URL: https://arxiv.org/pdf/2404.11120
• Copy Paste: [[2404.11120]] TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing(https://arxiv.org/abs/2404.11120)
• Keywords: robust, diffusion
• Abstract: Despite many attempts to leverage pre-trained text-to-image models (T2I) like Stable Diffusion (SD) for controllable image editing, producing good predictable results remains a challenge. Previous approaches have focused on either fine-tuning pre-trained T2I models on specific datasets to generate certain kinds of images (e.g., with a specific object or person), or on optimizing the weights, text prompts, and/or learning features for each input image in an attempt to coax the image generator to produce the desired result. However, these approaches all have shortcomings and fail to produce good results in a predictable and controllable manner. To address this problem, we present TiNO-Edit, an SD-based method that focuses on optimizing the noise patterns and diffusion timesteps during editing, something previously unexplored in the literature. With this simple change, we are able to generate results that both better align with the original images and reflect the desired result. Furthermore, we propose a set of new loss functions that operate in the latent domain of SD, greatly speeding up the optimization when compared to prior approaches, which operate in the pixel domain. Our method can be easily applied to variations of SD including Textual Inversion and DreamBooth that encode new concepts and incorporate them into the edited results. We present a host of image-editing capabilities enabled by our approach. Our code is publicly available at https://github.com/SherryXTChen/TiNO-Edit.

Title: TransLinkGuard: Safeguarding Transformer Models Against Model Stealing in Edge Deployment

• Authors: Qinfeng Li, Zhiqiang Shen, Zhenghan Qin, Yangfan Xie, Xuhong Zhang, Tianyu Du, Jianwei Yin
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11121
• Pdf URL: https://arxiv.org/pdf/2404.11121
• Copy Paste: [[2404.11121]] TransLinkGuard: Safeguarding Transformer Models Against Model Stealing in Edge Deployment(https://arxiv.org/abs/2404.11121)
• Keywords: secure, security, privacy, protect, defense, attack, steal, transformer, large language model
• Abstract: Proprietary large language models (LLMs) have been widely applied in various scenarios. Additionally, deploying LLMs on edge devices is trending for efficiency and privacy reasons. However, edge deployment of proprietary LLMs introduces new security challenges: edge-deployed models are exposed as white-box accessible to users, enabling adversaries to conduct effective model stealing (MS) attacks. Unfortunately, existing defense mechanisms fail to provide effective protection. Specifically, we identify four critical protection properties that existing methods fail to simultaneously satisfy: (1) maintaining protection after a model is physically copied; (2) authorizing model access at request level; (3) safeguarding runtime reverse engineering; (4) achieving high security with negligible runtime overhead. To address the above issues, we propose TransLinkGuard, a plug-and-play model protection approach against model stealing on edge devices. The core part of TransLinkGuard is a lightweight authorization module residing in a secure environment, e.g., TEE. The authorization module can freshly authorize each request based on its input. Extensive experiments show that TransLinkGuard achieves the same security protection as the black-box security guarantees with negligible overhead.

Title: What's under the hood: Investigating Automatic Metrics on Meeting Summarization

• Authors: Frederic Kirstein, Jan Philip Wahle, Terry Ruas, Bela Gipp
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11124
• Pdf URL: https://arxiv.org/pdf/2404.11124
• Copy Paste: [[2404.11124]] What's under the hood: Investigating Automatic Metrics on Meeting Summarization(https://arxiv.org/abs/2404.11124)
• Keywords: transformer
• Abstract: Meeting summarization has become a critical task considering the increase in online interactions. While new techniques are introduced regularly, their evaluation uses metrics not designed to capture meeting-specific errors, undermining effective evaluation. This paper investigates what the frequently used automatic metrics capture and which errors they mask by correlating automatic metric scores with human evaluations across a broad error taxonomy. We commence with a comprehensive literature review on English meeting summarization to define key challenges like speaker dynamics and contextual turn-taking and error types such as missing information and linguistic inaccuracy, concepts previously loosely defined in the field. We examine the relationship between characteristic challenges and errors by using annotated transcripts and summaries from Transformer-based sequence-to-sequence and autoregressive models from the general summary QMSum dataset. Through experimental validation, we find that different model architectures respond variably to challenges in meeting transcripts, resulting in different pronounced links between challenges and errors. Current default-used metrics struggle to capture observable errors, showing weak to mid-correlations, while a third of the correlations show trends of error masking. Only a subset reacts accurately to specific errors, while most correlations show either unresponsiveness or failure to reflect the error's impact on summary quality.

Title: Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationales

• Authors: Minghe Gao, Shuang Chen, Liang Pang, Yuan Yao, Jisheng Dang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Yueting Zhuang, Tat-Seng Chua
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11129
• Pdf URL: https://arxiv.org/pdf/2404.11129
• Copy Paste: [[2404.11129]] Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationales(https://arxiv.org/abs/2404.11129)
• Keywords: large language model
• Abstract: The remarkable performance of Multimodal Large Language Models (MLLMs) has unequivocally demonstrated their proficient understanding capabilities in handling a wide array of visual tasks. Nevertheless, the opaque nature of their black-box reasoning processes persists as an enigma, rendering them uninterpretable and struggling with hallucination. Their ability to execute intricate compositional reasoning tasks is also constrained, culminating in a stagnation of learning progression for these models. In this work, we introduce Fact, a novel paradigm designed to generate multimodal rationales that are faithful, concise, and transferable for teaching MLLMs. This paradigm utilizes verifiable visual programming to generate executable code guaranteeing faithfulness and precision. Subsequently, through a series of operations including pruning, merging, and bridging, the rationale enhances its conciseness. Furthermore, we filter rationales that can be transferred to end-to-end paradigms from programming paradigms to guarantee transferability. Empirical evidence from experiments demonstrates the superiority of our method across models of varying parameter sizes, significantly enhancing their compositional reasoning and generalization ability. Our approach also reduces hallucinations owing to its high correlation between images and text.

Title: GeoReF: Geometric Alignment Across Shape Variation for Category-level Object Pose Refinement

• Authors: Linfang Zheng, Tze Ho Elden Tse, Chen Wang, Yinghan Sun, Hua Chen, Ales Leonardis, Wei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11139
• Pdf URL: https://arxiv.org/pdf/2404.11139
• Copy Paste: [[2404.11139]] GeoReF: Geometric Alignment Across Shape Variation for Category-level Object Pose Refinement(https://arxiv.org/abs/2404.11139)
• Keywords: robust, extraction
• Abstract: Object pose refinement is essential for robust object pose estimation. Previous work has made significant progress towards instance-level object pose refinement. Yet, category-level pose refinement is a more challenging problem due to large shape variations within a category and the discrepancies between the target object and the shape prior. To address these challenges, we introduce a novel architecture for category-level object pose refinement. Our approach integrates an HS-layer and learnable affine transformations, which aims to enhance the extraction and alignment of geometric information. Additionally, we introduce a cross-cloud transformation mechanism that efficiently merges diverse data sources. Finally, we push the limits of our model by incorporating the shape prior information for translation and size error prediction. We conducted extensive experiments to demonstrate the effectiveness of the proposed framework. Through extensive quantitative experiments, we demonstrate significant improvement over the baseline method by a large margin across all metrics.

Title: LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory

• Authors: Zicheng Liu, Li Wang, Siyuan Li, Zedong Wang, Haitao Lin, Stan Z. Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11163
• Pdf URL: https://arxiv.org/pdf/2404.11163
• Copy Paste: [[2404.11163]] LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory(https://arxiv.org/abs/2404.11163)
• Keywords: transformer
• Abstract: Transformer models have been successful in various sequence processing tasks, but the self-attention mechanism's computational cost limits its practicality for long sequences. Although there are existing attention variants that improve computational efficiency, they have a limited ability to abstract global information effectively based on their hand-crafted mixing strategies. On the other hand, state-space models (SSMs) are tailored for long sequences but cannot capture complicated local information. Therefore, the combination of them as a unified token mixer is a trend in recent long-sequence models. However, the linearized attention degrades performance significantly even when equipped with SSMs. To address the issue, we propose a new method called LongVQ. LongVQ uses the vector quantization (VQ) technique to compress the global abstraction as a length-fixed codebook, enabling the linear-time computation of the attention matrix. This technique effectively maintains dynamic global and local patterns, which helps to complement the lack of long-range dependency issues. Our experiments on the Long Range Arena benchmark, autoregressive language modeling, and image and speech classification demonstrate the effectiveness of LongVQ. Our model achieves significant improvements over other sequence models, including variants of Transformers, Convolutions, and recent State Space Models.

Title: Personalized Heart Disease Detection via ECG Digital Twin Generation

• Authors: Yaojun Hu, Jintai Chen, Lianting Hu, Dantong Li, Jiahuan Yan, Haochao Ying, Huiying Liang, Jian Wu
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.11171
• Pdf URL: https://arxiv.org/pdf/2404.11171
• Copy Paste: [[2404.11171]] Personalized Heart Disease Detection via ECG Digital Twin Generation(https://arxiv.org/abs/2404.11171)
• Keywords: privacy, protect, robust
• Abstract: Heart diseases rank among the leading causes of global mortality, demonstrating a crucial need for early diagnosis and intervention. Most traditional electrocardiogram (ECG) based automated diagnosis methods are trained at population level, neglecting the customization of personalized ECGs to enhance individual healthcare management. A potential solution to address this limitation is to employ digital twins to simulate symptoms of diseases in real patients. In this paper, we present an innovative prospective learning approach for personalized heart disease detection, which generates digital twins of healthy individuals' anomalous ECGs and enhances the model sensitivity to the personalized symptoms. In our approach, a vector quantized feature separator is proposed to locate and isolate the disease symptom and normal segments in ECG signals with ECG report guidance. Thus, the ECG digital twins can simulate specific heart diseases used to train a personalized heart disease detection model. Experiments demonstrate that our approach not only excels in generating high-fidelity ECG signals but also improves personalized heart disease detection. Moreover, our approach ensures robust privacy protection, safeguarding patient data in model development.

Title: Deep Neural Networks via Complex Network Theory: a Perspective

• Authors: Emanuele La Malfa, Gabriele La Malfa, Giuseppe Nicosia, Vito Latora
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11172
• Pdf URL: https://arxiv.org/pdf/2404.11172
• Copy Paste: [[2404.11172]] Deep Neural Networks via Complex Network Theory: a Perspective(https://arxiv.org/abs/2404.11172)
• Keywords: interpretability
• Abstract: Deep Neural Networks (DNNs) can be represented as graphs whose links and vertices iteratively process data and solve tasks sub-optimally. Complex Network Theory (CNT), merging statistical physics with graph theory, provides a method for interpreting neural networks by analysing their weights and neuron structures. However, classic works adapt CNT metrics that only permit a topological analysis as they do not account for the effect of the input data. In addition, CNT metrics have been applied to a limited range of architectures, mainly including Fully Connected neural networks. In this work, we extend the existing CNT metrics with measures that sample from the DNNs' training distribution, shifting from a purely topological analysis to one that connects with the interpretability of deep learning. For the novel metrics, in addition to the existing ones, we provide a mathematical formalisation for Fully Connected, AutoEncoder, Convolutional and Recurrent neural networks, of which we vary the activation functions and the number of hidden layers. We show that these metrics differentiate DNNs based on the architecture, the number of hidden layers, and the activation function. Our contribution provides a method rooted in physics for interpreting DNNs that offers insights beyond the traditional input-output relationship and the CNT topological analysis.

Title: KI-GAN: Knowledge-Informed Generative Adversarial Networks for Enhanced Multi-Vehicle Trajectory Forecasting at Signalized Intersections

• Authors: Chuheng Wei, Guoyuan Wu, Matthew J. Barth, Amr Abdelraouf, Rohit Gupta, Kyungtae Han
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.11181
• Pdf URL: https://arxiv.org/pdf/2404.11181
• Copy Paste: [[2404.11181]] KI-GAN: Knowledge-Informed Generative Adversarial Networks for Enhanced Multi-Vehicle Trajectory Forecasting at Signalized Intersections(https://arxiv.org/abs/2404.11181)
• Keywords: generative
• Abstract: Reliable prediction of vehicle trajectories at signalized intersections is crucial to urban traffic management and autonomous driving systems. However, it presents unique challenges, due to the complex roadway layout at intersections, involvement of traffic signal controls, and interactions among different types of road users. To address these issues, we present in this paper a novel model called Knowledge-Informed Generative Adversarial Network (KI-GAN), which integrates both traffic signal information and multi-vehicle interactions to predict vehicle trajectories accurately. Additionally, we propose a specialized attention pooling method that accounts for vehicle orientation and proximity at intersections. Based on the SinD dataset, our KI-GAN model is able to achieve an Average Displacement Error (ADE) of 0.05 and a Final Displacement Error (FDE) of 0.12 for a 6-second observation and 6-second prediction cycle. When the prediction window is extended to 9 seconds, the ADE and FDE values are further reduced to 0.11 and 0.26, respectively. These results demonstrate the effectiveness of the proposed KI-GAN model in vehicle trajectory prediction under complex scenarios at signalized intersections, which represents a significant advancement in the target field.

Title: FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document

• Authors: Joonho Yang, Seunghyun Yoon, Byeongjeong Kim, Hwanhee Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11184
• Pdf URL: https://arxiv.org/pdf/2404.11184
• Copy Paste: [[2404.11184]] FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document(https://arxiv.org/abs/2404.11184)
• Keywords: interpretability
• Abstract: Through the advent of pre-trained language models, there have been notable advancements in abstractive summarization systems. Simultaneously, a considerable number of novel methods for evaluating factual consistency in abstractive summarization systems has been developed. But these evaluation approaches incorporate substantial limitations, especially on refinement and interpretability. In this work, we propose highly effective and interpretable factual inconsistency detection method metric Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document for abstractive summarization systems that is based on fine-grained atomic facts decomposition. Moreover, we align atomic facts decomposed from the summary with the source document through adaptive granularity expansion. These atomic facts represent a more fine-grained unit of information, facilitating detailed understanding and interpretability of the summary's factual inconsistency. Experimental results demonstrate that our proposed factual consistency checking system significantly outperforms existing systems. We release the code at https://github.com/plm3332/FIZZ.

Title: The Writing is on the Wall: Analyzing the Boom of Inscriptions and its Impact on Rollup Performance and Cost Efficiency

• Authors: Krzysztof Gogol, Johnnatan Messias, Maria Ines Silva, Benjamin Livshits
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.11189
• Pdf URL: https://arxiv.org/pdf/2404.11189
• Copy Paste: [[2404.11189]] The Writing is on the Wall: Analyzing the Boom of Inscriptions and its Impact on Rollup Performance and Cost Efficiency(https://arxiv.org/abs/2404.11189)
• Keywords: security
• Abstract: Late 2023 witnessed significant user activity on EVM chains, resulting in a surge in transaction activity and putting many rollups into the first live test. While some rollups performed well, some others experienced downtime during this period, affecting transaction finality time and gas fees. To address the lack of empirical research on rollups, we perform the first study during a heightened activity during the late 2023 transaction boom, as attributed to inscriptions - a novel technique that enables NFT and ERC-20 token creation on Bitcoin and other blockchains. We observe that minting inscription-based meme tokens on zkSync Era allows for trading at a fraction of the costs, compared to the Bitcoin or Ethereum networks. We also found that the increased transaction activity, over 99% attributed to the minting of new inscription tokens, positively affected other users of zkSync Era, resulting in lowered gas fees. Unlike L1 blockchains, ZK rollups may experience lower gas fees with increased transaction volume. Lastly, the introduction of blobs - a form of temporary data storage - decreased the gas costs of Ethereum rollups, but also raised a number of questions about the security of inscription-based tokens.

Title: Prompt-tuning for Clickbait Detection via Text Summarization

• Authors: Haoxiang Deng, Yi Zhu, Ye Wang, Jipeng Qiang, Yunhao Yuan, Yun Li, Runmei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11206
• Pdf URL: https://arxiv.org/pdf/2404.11206
• Copy Paste: [[2404.11206]] Prompt-tuning for Clickbait Detection via Text Summarization(https://arxiv.org/abs/2404.11206)
• Keywords: attack
• Abstract: Clickbaits are surprising social posts or deceptive news headlines that attempt to lure users for more clicks, which have posted at unprecedented rates for more profit or commercial revenue. The spread of clickbait has significant negative impacts on the users, which brings users misleading or even click-jacking attacks. Different from fake news, the crucial problem in clickbait detection is determining whether the headline matches the corresponding content. Most existing methods compute the semantic similarity between the headlines and contents for detecting clickbait. However, due to significant differences in length and semantic features between headlines and contents, directly calculating semantic similarity is often difficult to summarize the relationship between them. To address this problem, we propose a prompt-tuning method for clickbait detection via text summarization in this paper, text summarization is introduced to summarize the contents, and clickbait detection is performed based on the similarity between the generated summary and the contents. Specifically, we first introduce a two-stage text summarization model to produce high-quality news summaries based on pre-trained language models, and then both the headlines and new generated summaries are incorporated as the inputs for prompt-tuning. Additionally, a variety of strategies are conducted to incorporate external knowledge for improving the performance of clickbait detection. The extensive experiments on well-known clickbait detection datasets demonstrate that our method achieved state-of-the-art performance.

Title: Exploring the Transferability of Visual Prompting for Multimodal Large Language Models

• Authors: Yichi Zhang, Yinpeng Dong, Siyuan Zhang, Tianzan Min, Hang Su, Jun Zhu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11207
• Pdf URL: https://arxiv.org/pdf/2404.11207
• Copy Paste: [[2404.11207]] Exploring the Transferability of Visual Prompting for Multimodal Large Language Models(https://arxiv.org/abs/2404.11207)
• Keywords: large language model
• Abstract: Although Multimodal Large Language Models (MLLMs) have demonstrated promising versatile capabilities, their performance is still inferior to specialized models on downstream tasks, which makes adaptation necessary to enhance their utility. However, fine-tuning methods require independent training for every model, leading to huge computation and memory overheads. In this paper, we propose a novel setting where we aim to improve the performance of diverse MLLMs with a group of shared parameters optimized for a downstream task. To achieve this, we propose Transferable Visual Prompting (TVP), a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model. We introduce two strategies to address the issue of cross-model feature corruption of existing visual prompting methods and enhance the transferability of the learned prompts, including 1) Feature Consistency Alignment: which imposes constraints to the prompted feature changes to maintain task-agnostic knowledge; 2) Task Semantics Enrichment: which encourages the prompted images to contain richer task-specific semantics with language guidance. We validate the effectiveness of TVP through extensive experiments with 6 modern MLLMs on a wide variety of tasks ranging from object recognition and counting to multimodal reasoning and hallucination correction.

Title: Position Engineering: Boosting Large Language Models through Positional Information Manipulation

• Authors: Zhiyuan He, Huiqiang Jiang, Zilong Wang, Yuqing Yang, Luna Qiu, Lili Qiu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11216
• Pdf URL: https://arxiv.org/pdf/2404.11216
• Copy Paste: [[2404.11216]] Position Engineering: Boosting Large Language Models through Positional Information Manipulation(https://arxiv.org/abs/2404.11216)
• Keywords: large language model
• Abstract: The performance of large language models (LLMs) is significantly influenced by the quality of the prompts provided. In response, researchers have developed enormous prompt engineering strategies aimed at modifying the prompt text to enhance task performance. In this paper, we introduce a novel technique termed position engineering, which offers a more efficient way to guide large language models. Unlike prompt engineering, which requires substantial effort to modify the text provided to LLMs, position engineering merely involves altering the positional information in the prompt without modifying the text itself. We have evaluated position engineering in two widely-used LLM scenarios: retrieval-augmented generation (RAG) and in-context learning (ICL). Our findings show that position engineering substantially improves upon the baseline in both cases. Position engineering thus represents a promising new strategy for exploiting the capabilities of large language models.

Title: In-Context Learning State Vector with Inner and Momentum Optimization

• Authors: Dongfang Li, Zhenyu Liu, Xinshuo Hu, Zetian Sun, Baotian Hu, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11225
• Pdf URL: https://arxiv.org/pdf/2404.11225
• Copy Paste: [[2404.11225]] In-Context Learning State Vector with Inner and Momentum Optimization(https://arxiv.org/abs/2404.11225)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) have exhibited an impressive ability to perform In-Context Learning (ICL) from only a few examples. Recent works have indicated that the functions learned by ICL can be represented through compressed vectors derived from the transformer. However, the working mechanisms and optimization of these vectors are yet to be thoroughly explored. In this paper, we address this gap by presenting a comprehensive analysis of these compressed vectors, drawing parallels to the parameters trained with gradient descent, and introduce the concept of state vector. Inspired by the works on model soup and momentum-based gradient descent, we propose inner and momentum optimization methods that are applied to refine the state vector progressively as test-time adaptation. Moreover, we simulate state vector aggregation in the multiple example setting, where demonstrations comprising numerous examples are usually too lengthy for regular ICL, and further propose a divide-and-conquer aggregation method to address this challenge. We conduct extensive experiments using Llama-2 and GPT-J in both zero-shot setting and few-shot setting. The experimental results show that our optimization method effectively enhances the state vector and achieves the state-of-the-art performance on diverse tasks. Code is available at https://github.com/HITsz-TMG/ICL-State-Vector

Title: ONOT: a High-Quality ICAO-compliant Synthetic Mugshot Dataset

• Authors: Nicolò Di Domenico, Guido Borghi, Annalisa Franco, Davide Maltoni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11236
• Pdf URL: https://arxiv.org/pdf/2404.11236
• Copy Paste: [[2404.11236]] ONOT: a High-Quality ICAO-compliant Synthetic Mugshot Dataset(https://arxiv.org/abs/2404.11236)
• Keywords: privacy, attack, generative
• Abstract: Nowadays, state-of-the-art AI-based generative models represent a viable solution to overcome privacy issues and biases in the collection of datasets containing personal information, such as faces. Following this intuition, in this paper we introduce ONOT, a synthetic dataset specifically focused on the generation of high-quality faces in adherence to the requirements of the ISO/IEC 39794-5 standards that, following the guidelines of the International Civil Aviation Organization (ICAO), defines the interchange formats of face images in electronic Machine-Readable Travel Documents (eMRTD). The strictly controlled and varied mugshot images included in ONOT are useful in research fields related to the analysis of face images in eMRTD, such as Morphing Attack Detection and Face Quality Assessment. The dataset is publicly released, in combination with the generation procedure details in order to improve the reproducibility and enable future extensions.

Title: Optical Image-to-Image Translation Using Denoising Diffusion Models: Heterogeneous Change Detection as a Use Case

• Authors: João Gabriel Vinholi, Marco Chini, Anis Amziane, Renato Machado, Danilo Silva, Patrick Matgen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11243
• Pdf URL: https://arxiv.org/pdf/2404.11243
• Copy Paste: [[2404.11243]] Optical Image-to-Image Translation Using Denoising Diffusion Models: Heterogeneous Change Detection as a Use Case(https://arxiv.org/abs/2404.11243)
• Keywords: diffusion
• Abstract: We introduce an innovative deep learning-based method that uses a denoising diffusion-based model to translate low-resolution images to high-resolution ones from different optical sensors while preserving the contents and avoiding undesired artifacts. The proposed method is trained and tested on a large and diverse data set of paired Sentinel-II and Planet Dove images. We show that it can solve serious image generation issues observed when the popular classifier-free guided Denoising Diffusion Implicit Model (DDIM) framework is used in the task of Image-to-Image Translation of multi-sensor optical remote sensing images and that it can generate large images with highly consistent patches, both in colors and in features. Moreover, we demonstrate how our method improves heterogeneous change detection results in two urban areas: Beirut, Lebanon, and Austin, USA. Our contributions are: i) a new training and testing algorithm based on denoising diffusion models for optical image translation; ii) a comprehensive image quality evaluation and ablation study; iii) a comparison with the classifier-free guided DDIM framework; and iv) change detection experiments on heterogeneous data.

Title: A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene

• Authors: Wenbo Zhang, Yifan Zhang, Jianfeng Lin, Binqiang Huang, Jinlu Zhang, Wenhao Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11249
• Pdf URL: https://arxiv.org/pdf/2404.11249
• Copy Paste: [[2404.11249]] A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene(https://arxiv.org/abs/2404.11249)
• Keywords: robust
• Abstract: Pre-trained vision-language (V-L) models such as CLIP have shown excellent performance in many downstream cross-modal tasks. However, most of them are only applicable to the English context. Subsequent research has focused on this problem and proposed improved models, such as CN-CLIP and AltCLIP, to facilitate their applicability to Chinese and even other languages. Nevertheless, these models suffer from high latency and a large memory footprint in inference, which limits their further deployment on resource-constrained edge devices. In this work, we propose a conceptually simple yet effective multilingual CLIP Compression framework and train a lightweight multilingual vision-language model, called DC-CLIP, for both Chinese and English context. In this framework, we collect high-quality Chinese and English text-image pairs and design two training stages, including multilingual vision-language feature distillation and alignment. During the first stage, lightweight image/text student models are designed to learn robust visual/multilingual textual feature representation ability from corresponding teacher models, respectively. Subsequently, the multilingual vision-language alignment stage enables effective alignment of visual and multilingual textual features to further improve the model's multilingual performance. Comprehensive experiments in zero-shot image classification, conducted based on the ELEVATER benchmark, showcase that DC-CLIP achieves superior performance in the English context and competitive performance in the Chinese context, even with less training data, when compared to existing models of similar parameter magnitude. The evaluation demonstrates the effectiveness of our designed training mechanism.

Title: Sampling-based Pseudo-Likelihood for Membership Inference Attacks

• Authors: Masahiro Kaneko, Youmi Ma, Yuki Wata, Naoaki Okazaki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11262
• Pdf URL: https://arxiv.org/pdf/2404.11262
• Copy Paste: [[2404.11262]] Sampling-based Pseudo-Likelihood for Membership Inference Attacks(https://arxiv.org/abs/2404.11262)
• Keywords: attack, membership infer, large language model
• Abstract: Large Language Models (LLMs) are trained on large-scale web data, which makes it difficult to grasp the contribution of each text. This poses the risk of leaking inappropriate data such as benchmarks, personal information, and copyrighted texts in the training data. Membership Inference Attacks (MIA), which determine whether a given text is included in the model's training data, have been attracting attention. Previous studies of MIAs revealed that likelihood-based classification is effective for detecting leaks in LLMs. However, the existing methods cannot be applied to some proprietary models like ChatGPT or Claude 3 because the likelihood is unavailable to the user. In this study, we propose a Sampling-based Pseudo-Likelihood (\textbf{SPL}) method for MIA (\textbf{SaMIA}) that calculates SPL using only the text generated by an LLM to detect leaks. The SaMIA treats the target text as the reference text and multiple outputs from the LLM as text samples, calculates the degree of $n$-gram match as SPL, and determines the membership of the text in the training data. Even without likelihoods, SaMIA performed on par with existing likelihood-based methods.

Title: The Victim and The Beneficiary: Exploiting a Poisoned Model to Train a Clean Model on Poisoned Data

• Authors: Zixuan Zhu, Rui Wang, Cong Zou, Lihua Jing
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11265
• Pdf URL: https://arxiv.org/pdf/2404.11265
• Copy Paste: [[2404.11265]] The Victim and The Beneficiary: Exploiting a Poisoned Model to Train a Clean Model on Poisoned Data(https://arxiv.org/abs/2404.11265)
• Keywords: security, defense, attack, robust
• Abstract: Recently, backdoor attacks have posed a serious security threat to the training process of deep neural networks (DNNs). The attacked model behaves normally on benign samples but outputs a specific result when the trigger is present. However, compared with the rocketing progress of backdoor attacks, existing defenses are difficult to deal with these threats effectively or require benign samples to work, which may be unavailable in real scenarios. In this paper, we find that the poisoned samples and benign samples can be distinguished with prediction entropy. This inspires us to propose a novel dual-network training framework: The Victim and The Beneficiary (V&B), which exploits a poisoned model to train a clean model without extra benign samples. Firstly, we sacrifice the Victim network to be a powerful poisoned sample detector by training on suspicious samples. Secondly, we train the Beneficiary network on the credible samples selected by the Victim to inhibit backdoor injection. Thirdly, a semi-supervised suppression strategy is adopted for erasing potential backdoors and improving model performance. Furthermore, to better inhibit missed poisoned samples, we propose a strong data augmentation method, AttentionMix, which works well with our proposed V&B framework. Extensive experiments on two widely used datasets against 6 state-of-the-art attacks demonstrate that our framework is effective in preventing backdoor injection and robust to various attacks while maintaining the performance on benign samples. Our code is available at https://github.com/Zixuan-Zhu/VaB.

Title: Criteria for Uncertainty-based Corner Cases Detection in Instance Segmentation

• Authors: Florian Heidecker, Ahmad El-Khateeb, Maarten Bieshaar, Bernhard Sick
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11266
• Pdf URL: https://arxiv.org/pdf/2404.11266
• Copy Paste: [[2404.11266]] Criteria for Uncertainty-based Corner Cases Detection in Instance Segmentation(https://arxiv.org/abs/2404.11266)
• Keywords: segmentation
• Abstract: The operating environment of a highly automated vehicle is subject to change, e.g., weather, illumination, or the scenario containing different objects and other participants in which the highly automated vehicle has to navigate its passengers safely. These situations must be considered when developing and validating highly automated driving functions. This already poses a problem for training and evaluating deep learning models because without the costly labeling of thousands of recordings, not knowing whether the data contains relevant, interesting data for further model training, it is a guess under which conditions and situations the model performs poorly. For this purpose, we present corner case criteria based on the predictive uncertainty. With our corner case criteria, we are able to detect uncertainty-based corner cases of an object instance segmentation model without relying on ground truth (GT) data. We evaluated each corner case criterion using the COCO and the NuImages dataset to analyze the potential of our approach. We also provide a corner case decision function that allows us to distinguish each object into True Positive (TP), localization and/or classification corner case, or False Positive (FP). We also present our first results of an iterative training cycle that outperforms the baseline and where the data added to the training dataset is selected based on the corner case decision function.

Title: DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series

• Authors: Zahra Zamanzadeh Darban, Geoffrey I. Webb, Mahsa Salehi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11269
• Pdf URL: https://arxiv.org/pdf/2404.11269
• Copy Paste: [[2404.11269]] DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series(https://arxiv.org/abs/2404.11269)
• Keywords: robust, extraction
• Abstract: Time series anomaly detection (TAD) faces a significant challenge due to the scarcity of labelled data, which hinders the development of accurate detection models. Unsupervised domain adaptation (UDA) addresses this challenge by leveraging a labelled dataset from a related domain to detect anomalies in a target dataset. Existing domain adaptation techniques assume that the number of anomalous classes does not change between the source and target domains. In this paper, we propose a novel Domain Adaptation Contrastive learning for Anomaly Detection in multivariate time series (DACAD) model to address this issue by combining UDA and contrastive representation learning. DACAD's approach includes an anomaly injection mechanism that introduces various types of synthetic anomalies, enhancing the model's ability to generalise across unseen anomalous classes in different domains. This method significantly broadens the model's adaptability and robustness. Additionally, we propose a supervised contrastive loss for the source domain and a self-supervised contrastive triplet loss for the target domain, improving comprehensive feature representation learning and extraction of domain-invariant features. Finally, an effective Centre-based Entropy Classifier (CEC) is proposed specifically for anomaly detection, facilitating accurate learning of normal boundaries in the source domain. Our extensive evaluation across multiple real-world datasets against leading models in time series anomaly detection and UDA underscores DACAD's effectiveness. The results validate DACAD's superiority in transferring knowledge across domains and its potential to mitigate the challenge of limited labelled data in time series anomaly detection.

Title: SoK: Decentralized Finance (DeFi) -- Fundamentals, Taxonomy and Risks

• Authors: Krzysztof Gogol, Christian Killer, Malte Schlosser, Thomas Bocek, Burkhard Stiller, Claudio Tessone
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.11281
• Pdf URL: https://arxiv.org/pdf/2404.11281
• Copy Paste: [[2404.11281]] SoK: Decentralized Finance (DeFi) -- Fundamentals, Taxonomy and Risks(https://arxiv.org/abs/2404.11281)
• Keywords: security
• Abstract: Decentralized Finance (DeFi) refers to financial services that are not necessarily related to crypto-currencies. By employing blockchain for security and integrity, DeFi creates new possibilities that attract retail and institution users, including central banks. Given its novel applications and sophisticated designs, the distinction between DeFi services and understanding the risk involved is often complex. This work systematically presents the major categories of DeFi protocols that cover over 90\% of total value locked (TVL) in DeFi. It establishes a structured methodology to differentiate between DeFi protocols based on their design and architecture. Every DeFi protocol is classified into one of three groups: liquidity pools, pegged and synthetic tokens, and aggregator protocols, followed by risk analysis. In particular, we classify stablecoins, liquid staking tokens, and bridged (wrapped) assets as pegged tokens resembling similar risks. The full risk exposure of DeFi users is derived not only from the DeFi protocol design but also from how it is used and with which tokens.

Title: Amplifying Main Memory-Based Timing Covert and Side Channels using Processing-in-Memory Operations

• Authors: Konstantinos Kanellopoulos, F. Nisa Bostanci, Ataberk Olgun, A. Giray Yaglikci, Ismail Emir Yuksel, Nika Mansouri Ghiasi, Zulal Bingol, Mohammad Sadrosadati, Onur Mutlu
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2404.11284
• Pdf URL: https://arxiv.org/pdf/2404.11284
• Copy Paste: [[2404.11284]] Amplifying Main Memory-Based Timing Covert and Side Channels using Processing-in-Memory Operations(https://arxiv.org/abs/2404.11284)
• Keywords: secure, security, attack, robust
• Abstract: The adoption of processing-in-memory (PiM) architectures has been gaining momentum because they provide high performance and low energy consumption by alleviating the data movement bottleneck. Yet, the security of such architectures has not been thoroughly explored. The adoption of PiM solutions provides a new way to directly access main memory, which can be potentially exploited by malicious user applications. We show that this new way to access main memory opens opportunities for high-throughput timing attack vectors that are hard-to-mitigate without significant performance overhead. We introduce IMPACT, a set of high-throughput main memory-based timing attacks that leverage characteristics of PiM architectures to establish covert and side channels. IMPACT enables high-throughput communication and private information leakage. To achieve this, IMPACT (i) eliminates expensive cache bypassing steps required by processor-centric main memory and cache-based timing attacks and (ii) leverages the intrinsic parallelism of PiM operations. First, we showcase two covert-channel attack variants that run on the host CPU and leverage PiM architectures to gain direct and fast access to main memory and establish high-throughput communication covert channels. Second, we showcase a side-channel attack on a DNA sequence analysis application that leaks the private characteristics of a user's sample genome by leveraging PiM operations. Our results demonstrate that (i) our covert channels achieve up to 14.16 Mb/s communication throughput, which is 6.38x faster than the state-of-the-art main memory-based covert channels, and (ii) our side-channel attack allows the attacker to determine the properties of a sample genome at a throughput of 7.5 Mb/s with 96% accuracy. We discuss and evaluate several countermeasures for IMPACT to enable secure and robust PiM architectures.

Title: A Preference-driven Paradigm for Enhanced Translation with Large Language Models

• Authors: Dawei Zhu, Sony Trenous, Xiaoyu Shen, Dietrich Klakow, Bill Byrne, Eva Hasler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11288
• Pdf URL: https://arxiv.org/pdf/2404.11288
• Copy Paste: [[2404.11288]] A Preference-driven Paradigm for Enhanced Translation with Large Language Models(https://arxiv.org/abs/2404.11288)
• Keywords: large language model
• Abstract: Recent research has shown that large language models (LLMs) can achieve remarkable translation performance through supervised fine-tuning (SFT) using only a small amount of parallel data. However, SFT simply instructs the model to imitate the reference translations at the token level, making it vulnerable to the noise present in the references. Hence, the assistance from SFT often reaches a plateau once the LLMs have achieved a certain level of translation capability, and further increasing the size of parallel data does not provide additional benefits. To overcome this plateau associated with imitation-based SFT, we propose a preference-based approach built upon the Plackett-Luce model. The objective is to steer LLMs towards a more nuanced understanding of translation preferences from a holistic view, while also being more resilient in the absence of gold translations. We further build a dataset named MAPLE to verify the effectiveness of our approach, which includes multiple translations of varying quality for each source sentence. Extensive experiments demonstrate the superiority of our approach in "breaking the plateau" across diverse LLMs and test settings. Our in-depth analysis underscores the pivotal role of diverse translations and accurate preference scores in the success of our approach.

Title: Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption

• Authors: Buzhen Huang, Chen Li, Chongyang Xu, Liang Pan, Yangang Wang, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11291
• Pdf URL: https://arxiv.org/pdf/2404.11291
• Copy Paste: [[2404.11291]] Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption(https://arxiv.org/abs/2404.11291)
• Keywords: diffusion
• Abstract: Existing multi-person human reconstruction approaches mainly focus on recovering accurate poses or avoiding penetration, but overlook the modeling of close interactions. In this work, we tackle the task of reconstructing closely interactive humans from a monocular video. The main challenge of this task comes from insufficient visual information caused by depth ambiguity and severe inter-person occlusion. In view of this, we propose to leverage knowledge from proxemic behavior and physics to compensate the lack of visual information. This is based on the observation that human interaction has specific patterns following the social proxemics. Specifically, we first design a latent representation based on Vector Quantised-Variational AutoEncoder (VQ-VAE) to model human interaction. A proxemics and physics guided diffusion model is then introduced to denoise the initial distribution. We design the diffusion model as dual branch with each branch representing one individual such that the interaction can be modeled via cross attention. With the learned priors of VQ-VAE and physical constraint as the additional information, our proposed approach is capable of estimating accurate poses that are also proxemics and physics plausible. Experimental results on Hi4D, 3DPW, and CHI3D demonstrate that our method outperforms existing approaches. The code is available at \url{https://github.com/boycehbz/HumanInteraction}.

Title: Learning from Unlabelled Data with Transformers: Domain Adaptation for Semantic Segmentation of High Resolution Aerial Images

• Authors: Nikolaos Dionelis, Francesco Pro, Luca Maiano, Irene Amerini, Bertrand Le Saux
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11299
• Pdf URL: https://arxiv.org/pdf/2404.11299
• Copy Paste: [[2404.11299]] Learning from Unlabelled Data with Transformers: Domain Adaptation for Semantic Segmentation of High Resolution Aerial Images(https://arxiv.org/abs/2404.11299)
• Keywords: transformer, segmentation
• Abstract: Data from satellites or aerial vehicles are most of the times unlabelled. Annotating such data accurately is difficult, requires expertise, and is costly in terms of time. Even if Earth Observation (EO) data were correctly labelled, labels might change over time. Learning from unlabelled data within a semi-supervised learning framework for segmentation of aerial images is challenging. In this paper, we develop a new model for semantic segmentation of unlabelled images, the Non-annotated Earth Observation Semantic Segmentation (NEOS) model. NEOS performs domain adaptation as the target domain does not have ground truth semantic segmentation masks. The distribution inconsistencies between the target and source domains are due to differences in acquisition scenes, environment conditions, sensors, and times. Our model aligns the learned representations of the different domains to make them coincide. The evaluation results show that NEOS is successful and outperforms other models for semantic segmentation of unlabelled data.

Title: A Semantic Segmentation-guided Approach for Ground-to-Aerial Image Matching

• Authors: Francesco Pro, Nikolaos Dionelis, Luca Maiano, Bertrand Le Saux, Irene Amerini
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11302
• Pdf URL: https://arxiv.org/pdf/2404.11302
• Copy Paste: [[2404.11302]] A Semantic Segmentation-guided Approach for Ground-to-Aerial Image Matching(https://arxiv.org/abs/2404.11302)
• Keywords: segmentation
• Abstract: Nowadays the accurate geo-localization of ground-view images has an important role across domains as diverse as journalism, forensics analysis, transports, and Earth Observation. This work addresses the problem of matching a query ground-view image with the corresponding satellite image without GPS data. This is done by comparing the features from a ground-view image and a satellite one, innovatively leveraging the corresponding latter's segmentation mask through a three-stream Siamese-like network. The proposed method, Semantic Align Net (SAN), focuses on limited Field-of-View (FoV) and ground panorama images (images with a FoV of 360{\deg}). The novelty lies in the fusion of satellite images in combination with their semantic segmentation masks, aimed at ensuring that the model can extract useful features and focus on the significant parts of the images. This work shows how SAN through semantic analysis of images improves the performance on the unlabelled CVUSA dataset for all the tested FoVs.

Title: Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives

• Authors: Zhangchi Feng, Richong Zhang, Zhijie Nie
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11317
• Pdf URL: https://arxiv.org/pdf/2404.11317
• Copy Paste: [[2404.11317]] Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives(https://arxiv.org/abs/2404.11317)
• Keywords: large language model
• Abstract: The Composed Image Retrieval (CIR) task aims to retrieve target images using a composed query consisting of a reference image and a modified text. Advanced methods often utilize contrastive learning as the optimization objective, which benefits from adequate positive and negative examples. However, the triplet for CIR incurs high manual annotation costs, resulting in limited positive examples. Furthermore, existing methods commonly use in-batch negative sampling, which reduces the negative number available for the model. To address the problem of lack of positives, we propose a data generation method by leveraging a multi-modal large language model to construct triplets for CIR. To introduce more negatives during fine-tuning, we design a two-stage fine-tuning framework for CIR, whose second stage introduces plenty of static representations of negatives to optimize the representation space rapidly. The above two improvements can be effectively stacked and designed to be plug-and-play, easily applied to existing CIR models without changing their original architectures. Extensive experiments and ablation analysis demonstrate that our method effectively scales positives and negatives and achieves state-of-the-art results on both FashionIQ and CIRR datasets. In addition, our methods also perform well in zero-shot composed image retrieval, providing a new CIR solution for the low-resources scenario.

Title: Leveraging Fine-Grained Information and Noise Decoupling for Remote Sensing Change Detection

• Authors: Qiangang Du, Jinlong Peng, Changan Wang, Xu Chen, Qingdong He, Wenbing Zhu, Mingmin Chi, Yabiao Wang, Chengjie Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11318
• Pdf URL: https://arxiv.org/pdf/2404.11318
• Copy Paste: [[2404.11318]] Leveraging Fine-Grained Information and Noise Decoupling for Remote Sensing Change Detection(https://arxiv.org/abs/2404.11318)
• Keywords: robust
• Abstract: Change detection aims to identify remote sense object changes by analyzing data between bitemporal image pairs. Due to the large temporal and spatial span of data collection in change detection image pairs, there are often a significant amount of task-specific and task-agnostic noise. Previous effort has focused excessively on denoising, with this goes a great deal of loss of fine-grained information. In this paper, we revisit the importance of fine-grained features in change detection and propose a series of operations for fine-grained information compensation and noise decoupling (FINO). First, the context is utilized to compensate for the fine-grained information in the feature space. Next, a shape-aware and a brightness-aware module are designed to improve the capacity for representation learning. The shape-aware module guides the backbone for more precise shape estimation, guiding the backbone network in extracting object shape features. The brightness-aware module learns a overall brightness estimation to improve the model's robustness to task-agnostic noise. Finally, a task-specific noise decoupling structure is designed as a way to improve the model's ability to separate noise interference from feature similarity. With these training schemes, our proposed method achieves new state-of-the-art (SOTA) results in multiple change detection benchmarks. The code will be made available.

Title: On Learning Parities with Dependent Noise

• Authors: Noah Golowich, Ankur Moitra, Dhruv Rohatgi
• Subjects: cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2404.11325
• Pdf URL: https://arxiv.org/pdf/2404.11325
• Copy Paste: [[2404.11325]] On Learning Parities with Dependent Noise(https://arxiv.org/abs/2404.11325)
• Keywords: robust
• Abstract: In this expository note we show that the learning parities with noise (LPN) assumption is robust to weak dependencies in the noise distribution of small batches of samples. This provides a partial converse to the linearization technique of [AG11]. The material in this note is drawn from a recent work by the authors [GMR24], where the robustness guarantee was a key component in a cryptographic separation between reinforcement learning and supervised learning.

Title: LLMs for Cyber Security: New Opportunities

• Authors: Dinil Mon Divakaran, Sai Teja Peddinti
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.11338
• Pdf URL: https://arxiv.org/pdf/2404.11338
• Copy Paste: [[2404.11338]] LLMs for Cyber Security: New Opportunities(https://arxiv.org/abs/2404.11338)
• Keywords: security, large language model
• Abstract: Large language models (LLMs) are a class of powerful and versatile models that are beneficial to many industries. With the emergence of LLMs, we take a fresh look at cyber security, specifically exploring and summarizing the potential of LLMs in addressing challenging problems in the security and safety domains.

Title: Detector Collapse: Backdooring Object Detection to Catastrophic Overload or Blindness

• Authors: Hangtao Zhang, Shengshan Hu, Yichen Wang, Leo Yu Zhang, Ziqi Zhou, Xianlong Wang, Yanjun Zhang, Chao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11357
• Pdf URL: https://arxiv.org/pdf/2404.11357
• Copy Paste: [[2404.11357]] Detector Collapse: Backdooring Object Detection to Catastrophic Overload or Blindness(https://arxiv.org/abs/2404.11357)
• Keywords: attack
• Abstract: Object detection tasks, crucial in safety-critical systems like autonomous driving, focus on pinpointing object locations. These detectors are known to be susceptible to backdoor attacks. However, existing backdoor techniques have primarily been adapted from classification tasks, overlooking deeper vulnerabilities specific to object detection. This paper is dedicated to bridging this gap by introducing Detector Collapse} (DC), a brand-new backdoor attack paradigm tailored for object detection. DC is designed to instantly incapacitate detectors (i.e., severely impairing detector's performance and culminating in a denial-of-service). To this end, we develop two innovative attack schemes: Sponge for triggering widespread misidentifications and Blinding for rendering objects invisible. Remarkably, we introduce a novel poisoning strategy exploiting natural objects, enabling DC to act as a practical backdoor in real-world environments. Our experiments on different detectors across several benchmarks show a significant improvement ($\sim$10\%-60\% absolute and $\sim$2-7$\times$ relative) in attack efficacy over state-of-the-art attacks.

Title: S3PHER: Secure and Searchable System for Patient-driven HEalth data shaRing

• Authors: Ivan Costa, Ivone Amorim, Eva Maia, Pedro Barbosa, Isabel Praca
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.11372
• Pdf URL: https://arxiv.org/pdf/2404.11372
• Copy Paste: [[2404.11372]] S3PHER: Secure and Searchable System for Patient-driven HEalth data shaRing(https://arxiv.org/abs/2404.11372)
• Keywords: secure, security, privacy
• Abstract: Healthcare data contains some of the most sensitive information about an individual, yet sharing this data with healthcare practitioners can significantly enhance patient care and support research efforts. However, current systems for sharing health data between patients and caregivers do not fully address the critical security requirements of privacy, confidentiality, and consent management. Furthermore, compliance with regulatory laws such as GDPR and HIPAA is often deficient, largely because patients typically are asked to provide general consent for healthcare entities to access their data. Recognizing the limitations of existing systems, we present S3PHER, a novel approach to sharing health data that provides patients with control over who accesses their data, what data is accessed, and when. Our system ensures end to end privacy by integrating a Proxy ReEncryption Scheme with a Searchable Encryption Scheme, utilizing Homomorphic Encryption to enable healthcare practitioners to privately search and access patients' documents. The practicality and benefits of S3PHER are further validated through end to end deployment and use case analyses, with tests on real datasets demonstrating promising execution times.

Title: Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motion

• Authors: Xinghan Wang, Zixi Kang, Yadong Mu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.11375
• Pdf URL: https://arxiv.org/pdf/2404.11375
• Copy Paste: [[2404.11375]] Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motion(https://arxiv.org/abs/2404.11375)
• Keywords: transformer
• Abstract: Human motion understanding is a fundamental task with diverse practical applications, facilitated by the availability of large-scale motion capture datasets. Recent studies focus on text-motion tasks, such as text-based motion generation, editing and question answering. In this study, we introduce the novel task of text-based human motion grounding (THMG), aimed at precisely localizing temporal segments corresponding to given textual descriptions within untrimmed motion sequences. Capturing global temporal information is crucial for the THMG task. However, transformer-based models that rely on global temporal self-attention face challenges when handling long untrimmed sequences due to the quadratic computational cost. We address these challenges by proposing Text-controlled Motion Mamba (TM-Mamba), a unified model that integrates temporal global context, language query control, and spatial graph topology with only linear memory cost. The core of the model is a text-controlled selection mechanism which dynamically incorporates global temporal information based on text query. The model is further enhanced to be topology-aware through the integration of relational embeddings. For evaluation, we introduce BABEL-Grounding, the first text-motion dataset that provides detailed textual descriptions of human actions along with their corresponding temporal segments. Extensive evaluations demonstrate the effectiveness of TM-Mamba on BABEL-Grounding.

Title: Exploring Key Point Analysis with Pairwise Generation and Graph Partitioning

• Authors: Xiao Li, Yong Jiang, Shen Huang, Pengjun Xie, Gong Cheng, Fei Huang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11384
• Pdf URL: https://arxiv.org/pdf/2404.11384
• Copy Paste: [[2404.11384]] Exploring Key Point Analysis with Pairwise Generation and Graph Partitioning(https://arxiv.org/abs/2404.11384)
• Keywords: generative
• Abstract: Key Point Analysis (KPA), the summarization of multiple arguments into a concise collection of key points, continues to be a significant and unresolved issue within the field of argument mining. Existing models adapt a two-stage pipeline of clustering arguments or generating key points for argument clusters. This approach rely on semantic similarity instead of measuring the existence of shared key points among arguments. Additionally, it only models the intra-cluster relationship among arguments, disregarding the inter-cluster relationship between arguments that do not share key points. To address these limitations, we propose a novel approach for KPA with pairwise generation and graph partitioning. Our objective is to train a generative model that can simultaneously provide a score indicating the presence of shared key point between a pair of arguments and generate the shared key point. Subsequently, to map generated redundant key points to a concise set of key points, we proceed to construct an arguments graph by considering the arguments as vertices, the generated key points as edges, and the scores as edge weights. We then propose a graph partitioning algorithm to partition all arguments sharing the same key points to the same subgraph. Notably, our experimental findings demonstrate that our proposed model surpasses previous models when evaluated on both the ArgKP and QAM datasets.

Title: Enhancing Data Privacy In Wireless Sensor Networks: Investigating Techniques And Protocols To Protect Privacy Of Data Transmitted Over Wireless Sensor Networks In Critical Applications Of Healthcare And National Security

• Authors: Akinsola Ahmed, Ejiofor Oluomachi, Akinde Abdullah, Njoku Tochukwu
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.11388
• Pdf URL: https://arxiv.org/pdf/2404.11388
• Copy Paste: [[2404.11388]] Enhancing Data Privacy In Wireless Sensor Networks: Investigating Techniques And Protocols To Protect Privacy Of Data Transmitted Over Wireless Sensor Networks In Critical Applications Of Healthcare And National Security(https://arxiv.org/abs/2404.11388)
• Keywords: security, privacy, protect, diffusion
• Abstract: The article discusses the emergence of Wireless Sensor Networks (WSNs) as a groundbreaking technology in data processing and communication. It outlines how WSNs, composed of dispersed autonomous sensors, are utilized to monitor physical and environmental factors, transmitting data wirelessly for analysis. The article explores various applications of WSNs in healthcare, national security, emergency response, and infrastructure monitoring, highlighting their roles in enhancing patient care, public health surveillance, border security, disaster management, and military operations. Additionally, it examines the foundational concepts of data privacy in WSNs, focusing on encryption techniques, authentication mechanisms, anonymization techniques, and access control mechanisms. The article also addresses vulnerabilities, threats, and challenges related to data privacy in healthcare and national security contexts, emphasizing regulatory compliance, ethical considerations, and socio-economic factors. Furthermore, it introduces the Diffusion of Innovation Theory as a framework for understanding the adoption of privacy-enhancing technologies in WSNs. Finally, the article reviews empirical studies demonstrating the efficacy of security solutions in preserving data privacy in WSNs, offering insights into advancements in safeguarding sensitive information.

Title: SERENE: A Collusion Resilient Replication-based Verification Framework

• Authors: Amir Esmaeili, Abderrahmen Mtibaa
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.11410
• Pdf URL: https://arxiv.org/pdf/2404.11410
• Copy Paste: [[2404.11410]] SERENE: A Collusion Resilient Replication-based Verification Framework(https://arxiv.org/abs/2404.11410)
• Keywords: security
• Abstract: The rapid advancement of autonomous driving technology is accompanied by substantial challenges, particularly the reliance on remote task execution without ensuring a reliable and accurate returned results. This reliance on external compute servers, which may be malicious or rogue, represents a major security threat. While researchers have been exploring verifiable computing, and replication-based task verification as a simple, fast, and dependable method to assess the correctness of results. However, colluding malicious workers can easily defeat this method. Existing collusion detection and mitigation solutions often require the use of a trusted third party server or verified tasks which may be hard to guarantee, or solutions that assume the presence of a minority of colluding servers. We propose SERENE a collusion resilient replication-based verification framework that detects, and mitigates colluding workers. Unlike state-of-the-art solutions, SERENE uses a lightweight detection algorithm that detects collusion based on a single verification task. Mitigation requires a two stage process to group the workers and identifying colluding from honest workers. We implement and compare SERENE's performance to Staab et. al, resulting in an average of 50\% and 60\% accuracy improvement in detection and mitigation accuracy respectively.

Title: Neural Shrödinger Bridge Matching for Pansharpening

• Authors: Zihan Cao, Xiao Wu, Liang-Jian Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11416
• Pdf URL: https://arxiv.org/pdf/2404.11416
• Copy Paste: [[2404.11416]] Neural Shrödinger Bridge Matching for Pansharpening(https://arxiv.org/abs/2404.11416)
• Keywords: diffusion
• Abstract: Recent diffusion probabilistic models (DPM) in the field of pansharpening have been gradually gaining attention and have achieved state-of-the-art (SOTA) performance. In this paper, we identify shortcomings in directly applying DPMs to the task of pansharpening as an inverse problem: 1) initiating sampling directly from Gaussian noise neglects the low-resolution multispectral image (LRMS) as a prior; 2) low sampling efficiency often necessitates a higher number of sampling steps. We first reformulate pansharpening into the stochastic differential equation (SDE) form of an inverse problem. Building upon this, we propose a Schr\"odinger bridge matching method that addresses both issues. We design an efficient deep neural network architecture tailored for the proposed SB matching. In comparison to the well-established DL-regressive-based framework and the recent DPM framework, our method demonstrates SOTA performance with fewer sampling steps. Moreover, we discuss the relationship between SB matching and other methods based on SDEs and ordinary differential equations (ODEs), as well as its connection with optimal transport. Code will be available.

Title: SLAIM: Robust Dense Neural SLAM for Online Tracking and Mapping

• Authors: Vincent Cartillier, Grant Schindler, Irfan Essa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11419
• Pdf URL: https://arxiv.org/pdf/2404.11419
• Copy Paste: [[2404.11419]] SLAIM: Robust Dense Neural SLAM for Online Tracking and Mapping(https://arxiv.org/abs/2404.11419)
• Keywords: robust
• Abstract: We present SLAIM - Simultaneous Localization and Implicit Mapping. We propose a novel coarse-to-fine tracking model tailored for Neural Radiance Field SLAM (NeRF-SLAM) to achieve state-of-the-art tracking performance. Notably, existing NeRF-SLAM systems consistently exhibit inferior tracking performance compared to traditional SLAM algorithms. NeRF-SLAM methods solve camera tracking via image alignment and photometric bundle-adjustment. Such optimization processes are difficult to optimize due to the narrow basin of attraction of the optimization loss in image space (local minima) and the lack of initial correspondences. We mitigate these limitations by implementing a Gaussian pyramid filter on top of NeRF, facilitating a coarse-to-fine tracking optimization strategy. Furthermore, NeRF systems encounter challenges in converging to the right geometry with limited input views. While prior approaches use a Signed-Distance Function (SDF)-based NeRF and directly supervise SDF values by approximating ground truth SDF through depth measurements, this often results in suboptimal geometry. In contrast, our method employs a volume density representation and introduces a novel KL regularizer on the ray termination distribution, constraining scene geometry to consist of empty space and opaque surfaces. Our solution implements both local and global bundle-adjustment to produce a robust (coarse-to-fine) and accurate (KL regularizer) SLAM solution. We conduct experiments on multiple datasets (ScanNet, TUM, Replica) showing state-of-the-art results in tracking and in reconstruction accuracy.

Title: Short-term wind speed forecasting model based on an attention-gated recurrent neural network and error correction strategy

• Authors: Haojian Huang
• Subjects: cs.LG, cs.AI, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2404.11422
• Pdf URL: https://arxiv.org/pdf/2404.11422
• Copy Paste: [[2404.11422]] Short-term wind speed forecasting model based on an attention-gated recurrent neural network and error correction strategy(https://arxiv.org/abs/2404.11422)
• Keywords: security
• Abstract: The accurate wind speed series forecast is very pivotal to security of grid dispatching and the application of wind power. Nevertheless, on account of their nonlinear and non-stationary nature, their short-term forecast is extremely challenging. Therefore, this dissertation raises one short-term wind speed forecast pattern on the foundation of attention with an improved gated recurrent neural network (AtGRU) and a tactic of error correction. That model uses the AtGRU model as the preliminary predictor and the GRU model as the error corrector. At the beginning, SSA (singular spectrum analysis) is employed in previous wind speed series for lessening the noise. Subsequently, historical wind speed series is going to be used for the predictor training. During this process, the prediction can have certain errors. The sequence of these errors processed by variational modal decomposition (VMD) is used to train the corrector of error. The eventual forecast consequence is just the sum of predictor forecast and error corrector. The proposed SSA-AtGRU-VMD-GRU model outperforms the compared models in three case studies on Woodburn, St. Thomas, and Santa Cruz. It is indicated that the model evidently enhances the correction of the wind speed forecast.

Title: CarcassFormer: An End-to-end Transformer-based Framework for Simultaneous Localization, Segmentation and Classification of Poultry Carcass Defect

• Authors: Minh Tran, Sang Truong, Arthur F. A. Fernandes, Michael T. Kidd, Ngan Le
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11429
• Pdf URL: https://arxiv.org/pdf/2404.11429
• Copy Paste: [[2404.11429]] CarcassFormer: An End-to-end Transformer-based Framework for Simultaneous Localization, Segmentation and Classification of Poultry Carcass Defect(https://arxiv.org/abs/2404.11429)
• Keywords: transformer, segmentation
• Abstract: In the food industry, assessing the quality of poultry carcasses during processing is a crucial step. This study proposes an effective approach for automating the assessment of carcass quality without requiring skilled labor or inspector involvement. The proposed system is based on machine learning (ML) and computer vision (CV) techniques, enabling automated defect detection and carcass quality assessment. To this end, an end-to-end framework called CarcassFormer is introduced. It is built upon a Transformer-based architecture designed to effectively extract visual representations while simultaneously detecting, segmenting, and classifying poultry carcass defects. Our proposed framework is capable of analyzing imperfections resulting from production and transport welfare issues, as well as processing plant stunner, scalder, picker, and other equipment malfunctions. To benchmark the framework, a dataset of 7,321 images was initially acquired, which contained both single and multiple carcasses per image. In this study, the performance of the CarcassFormer system is compared with other state-of-the-art (SOTA) approaches for both classification, detection, and segmentation tasks. Through extensive quantitative experiments, our framework consistently outperforms existing methods, demonstrating remarkable improvements across various evaluation metrics such as AP, AP@50, and AP@75. Furthermore, the qualitative results highlight the strengths of CarcassFormer in capturing fine details, including feathers, and accurately localizing and segmenting carcasses with high precision. To facilitate further research and collaboration, the pre-trained model and source code of CarcassFormer is available for research purposes at: \url{https://github.com/UARK-AICV/CarcassFormer}.

Title: Open-Ended Wargames with Large Language Models

• Authors: Daniel P. Hogan, Andrea Brennen
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.11446
• Pdf URL: https://arxiv.org/pdf/2404.11446
• Copy Paste: [[2404.11446]] Open-Ended Wargames with Large Language Models(https://arxiv.org/abs/2404.11446)
• Keywords: large language model
• Abstract: Wargames are a powerful tool for understanding and rehearsing real-world decision making. Automated play of wargames using artificial intelligence (AI) enables possibilities beyond those of human-conducted games, such as playing the game many times over to see a range of possible outcomes. There are two categories of wargames: quantitative games, with discrete types of moves, and qualitative games, which revolve around open-ended responses. Historically, automation efforts have focused on quantitative games, but large language models (LLMs) make it possible to automate qualitative wargames. We introduce "Snow Globe," an LLM-powered multi-agent system for playing qualitative wargames. With Snow Globe, every stage of a text-based qualitative wargame from scenario preparation to post-game analysis can be optionally carried out by AI, humans, or a combination thereof. We describe its software architecture conceptually and release an open-source implementation alongside this publication. As case studies, we simulate a tabletop exercise about an AI incident response and a political wargame about a geopolitical crisis. We discuss potential applications of the approach and how it fits into the broader wargaming ecosystem.

Title: AI-Enhanced Cognitive Behavioral Therapy: Deep Learning and Large Language Models for Extracting Cognitive Pathways from Social Media Texts

• Authors: Meng Jiang, Yi Jing Yu, Qing Zhao, Jianqiang Li, Changwei Song, Hongzhi Qi, Wei Zhai, Dan Luo, Xiaoqin Wang, Guanghui Fu, Bing Xiang Yang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11449
• Pdf URL: https://arxiv.org/pdf/2404.11449
• Copy Paste: [[2404.11449]] AI-Enhanced Cognitive Behavioral Therapy: Deep Learning and Large Language Models for Extracting Cognitive Pathways from Social Media Texts(https://arxiv.org/abs/2404.11449)
• Keywords: large language model
• Abstract: Cognitive Behavioral Therapy (CBT) is an effective technique for addressing the irrational thoughts stemming from mental illnesses, but it necessitates precise identification of cognitive pathways to be successfully implemented in patient care. In current society, individuals frequently express negative emotions on social media on specific topics, often exhibiting cognitive distortions, including suicidal behaviors in extreme cases. Yet, there is a notable absence of methodologies for analyzing cognitive pathways that could aid psychotherapists in conducting effective interventions online. In this study, we gathered data from social media and established the task of extracting cognitive pathways, annotating the data based on a cognitive theoretical framework. We initially categorized the task of extracting cognitive pathways as a hierarchical text classification with four main categories and nineteen subcategories. Following this, we structured a text summarization task to help psychotherapists quickly grasp the essential information. Our experiments evaluate the performance of deep learning and large language models (LLMs) on these tasks. The results demonstrate that our deep learning method achieved a micro-F1 score of 62.34% in the hierarchical text classification task. Meanwhile, in the text summarization task, GPT-4 attained a Rouge-1 score of 54.92 and a Rouge-2 score of 30.86, surpassing the experimental deep learning model's performance. However, it may suffer from an issue of hallucination. We have made all models and codes publicly available to support further research in this field.

Title: Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent

• Authors: Wei Chen, Zhiyuan Li
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11459
• Pdf URL: https://arxiv.org/pdf/2404.11459
• Copy Paste: [[2404.11459]] Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent(https://arxiv.org/abs/2404.11459)
• Keywords: large language model
• Abstract: A multimodal AI agent is characterized by its ability to process and learn from various types of data, including natural language, visual, and audio inputs, to inform its actions. Despite advancements in large language models that incorporate visual data, such as GPT-4V, effectively translating image-based data into actionable outcomes for AI agents continues to be challenging. In this paper, we introduce a multimodal model that incorporates the concept of functional token specifically designed for AI agent applications. To ensure compatibility with edge devices, our model is optimized to a compact size of less than 1B parameters. Like GPT-4, our model can process both English and Chinese. We demonstrate that this model is capable of operating efficiently on a wide range of edge devices, including as constrained as a Raspberry Pi.

Title: A Federated Learning Approach to Privacy Preserving Offensive Language Identification

• Authors: Marcos Zampieri, Damith Premasiri, Tharindu Ranasinghe
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11470
• Pdf URL: https://arxiv.org/pdf/2404.11470
• Copy Paste: [[2404.11470]] A Federated Learning Approach to Privacy Preserving Offensive Language Identification(https://arxiv.org/abs/2404.11470)
• Keywords: privacy, federate
• Abstract: The spread of various forms of offensive speech online is an important concern in social media. While platforms have been investing heavily in ways of coping with this problem, the question of privacy remains largely unaddressed. Models trained to detect offensive language on social media are trained and/or fine-tuned using large amounts of data often stored in centralized servers. Since most social media data originates from end users, we propose a privacy preserving decentralized architecture for identifying offensive language online by introducing Federated Learning (FL) in the context of offensive language identification. FL is a decentralized architecture that allows multiple models to be trained locally without the need for data sharing hence preserving users' privacy. We propose a model fusion approach to perform FL. We trained multiple deep learning models on four publicly available English benchmark datasets (AHSD, HASOC, HateXplain, OLID) and evaluated their performance in detail. We also present initial cross-lingual experiments in English and Spanish. We show that the proposed model fusion approach outperforms baselines in all the datasets while preserving privacy.

Title: Assessing The Effectiveness Of Current Cybersecurity Regulations And Policies In The US

• Authors: Ejiofor Oluomachi, Akinsola Ahmed, Wahab Ahmed, Edozie Samson
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.11473
• Pdf URL: https://arxiv.org/pdf/2404.11473
• Copy Paste: [[2404.11473]] Assessing The Effectiveness Of Current Cybersecurity Regulations And Policies In The US(https://arxiv.org/abs/2404.11473)
• Keywords: security
• Abstract: This article assesses the effectiveness of current cybersecurity regulations and policies in the United States amidst the escalating frequency and sophistication of cyber threats. The focus is on the comprehensive framework established by the U.S. government, with a spotlight on the National Institute of Standards and Technology (NIST) Cybersecurity Framework and key regulations such as HIPAA, GLBA, FISMA, CISA, CCPA, and the DOD Cybersecurity Maturity Model Certification. The study evaluates the impact of these regulations on different sectors and analyzes trends in cybercrime data from 2000 to 2022. The findings highlight the challenges, successes, and the need for continuous adaptation in the face of evolving cyber threats

Title: Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt

• Authors: Zhanjie Zhang, Quanwei Zhang, Huaizhong Lin, Wei Xing, Juncheng Mo, Shuaicheng Huang, Jinheng Xie, Guangyuan Li, Junsheng Luan, Lei Zhao, Dalong Zhang, Lixia Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11474
• Pdf URL: https://arxiv.org/pdf/2404.11474
• Copy Paste: [[2404.11474]] Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt(https://arxiv.org/abs/2404.11474)
• Keywords: diffusion, generative
• Abstract: Artistic style transfer aims to transfer the learned artistic style onto an arbitrary content image, generating artistic stylized images. Existing generative adversarial network-based methods fail to generate highly realistic stylized images and always introduce obvious artifacts and disharmonious patterns. Recently, large-scale pre-trained diffusion models opened up a new way for generating highly realistic artistic stylized images. However, diffusion model-based methods generally fail to preserve the content structure of input content images well, introducing some undesired content structure and style patterns. To address the above problems, we propose a novel pre-trained diffusion-based artistic style transfer method, called LSAST, which can generate highly realistic artistic stylized images while preserving the content structure of input content images well, without bringing obvious artifacts and disharmonious style patterns. Specifically, we introduce a Step-aware and Layer-aware Prompt Space, a set of learnable prompts, which can learn the style information from the collection of artworks and dynamically adjusts the input images' content structure and style pattern. To train our prompt space, we propose a novel inversion method, called Step-ware and Layer-aware Prompt Inversion, which allows the prompt space to learn the style information of the artworks collection. In addition, we inject a pre-trained conditional branch of ControlNet into our LSAST, which further improved our framework's ability to maintain content structure. Extensive experiments demonstrate that our proposed method can generate more highly realistic artistic stylized images than the state-of-the-art artistic style transfer methods.

Title: arcjetCV: an open-source software to analyze material ablation

• Authors: Alexandre Quintart, Magnus Haw, Federico Semeraro
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11492
• Pdf URL: https://arxiv.org/pdf/2404.11492
• Copy Paste: [[2404.11492]] arcjetCV: an open-source software to analyze material ablation(https://arxiv.org/abs/2404.11492)
• Keywords: extraction, segmentation
• Abstract: arcjetCV is an open-source Python software designed to automate time-resolved measurements of heatshield material recession and recession rates from arcjet test video footage. This new automated and accessible capability greatly exceeds previous manual extraction methods, enabling rapid and detailed characterization of material recession for any sample with a profile video. arcjetCV automates the video segmentation process using machine learning models, including a one-dimensional (1D) Convolutional Neural Network (CNN) to infer the time-window of interest, a two-dimensional (2D) CNN for image and edge segmentation, and a Local Outlier Factor (LOF) for outlier filtering. A graphical user interface (GUI) simplifies the user experience and an application programming interface (API) allows users to call the core functions from scripts, enabling video batch processing. arcjetCV's capability to measure time-resolved recession in turn enables characterization of non-linear processes (shrinkage, swelling, melt flows, etc.), contributing to higher fidelity validation and improved modeling of heatshield material performance. The source code associated with this article can be found at https://github.com/magnus-haw/arcjetCV.

Title: A Data-Driven Representation for Sign Language Production

• Authors: Harry Walsh, Abolfazl Ravanshad, Mariam Rahmani, Richard Bowden
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11499
• Pdf URL: https://arxiv.org/pdf/2404.11499
• Copy Paste: [[2404.11499]] A Data-Driven Representation for Sign Language Production(https://arxiv.org/abs/2404.11499)
• Keywords: transformer
• Abstract: Phonetic representations are used when recording spoken languages, but no equivalent exists for recording signed languages. As a result, linguists have proposed several annotation systems that operate on the gloss or sub-unit level; however, these resources are notably irregular and scarce. Sign Language Production (SLP) aims to automatically translate spoken language sentences into continuous sequences of sign language. However, current state-of-the-art approaches rely on scarce linguistic resources to work. This has limited progress in the field. This paper introduces an innovative solution by transforming the continuous pose generation problem into a discrete sequence generation problem. Thus, overcoming the need for costly annotation. Although, if available, we leverage the additional information to enhance our approach. By applying Vector Quantisation (VQ) to sign language data, we first learn a codebook of short motions that can be combined to create a natural sequence of sign. Where each token in the codebook can be thought of as the lexicon of our representation. Then using a transformer we perform a translation from spoken language text to a sequence of codebook tokens. Each token can be directly mapped to a sequence of poses allowing the translation to be performed by a single network. Furthermore, we present a sign stitching method to effectively join tokens together. We evaluate on the RWTH-PHOENIX-Weather-2014T (PHOENIX14T) and the more challenging Meine DGS Annotated (mDGS) datasets. An extensive evaluation shows our approach outperforms previous methods, increasing the BLEU-1 back translation score by up to 72%.

Title: Paraphrase and Solve: Exploring and Exploiting the Impact of Surface Form on Mathematical Reasoning in Large Language Models

• Authors: Yue Zhou, Yada Zhu, Diego Antognini, Yoon Kim, Yang Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11500
• Pdf URL: https://arxiv.org/pdf/2404.11500
• Copy Paste: [[2404.11500]] Paraphrase and Solve: Exploring and Exploiting the Impact of Surface Form on Mathematical Reasoning in Large Language Models(https://arxiv.org/abs/2404.11500)
• Keywords: robust, large language model
• Abstract: This paper studies the relationship between the surface form of a mathematical problem and its solvability by large language models. We find that subtle alterations in the surface form can significantly impact the answer distribution and the solve rate, exposing the language model's lack of robustness and sensitivity to the surface form in reasoning through complex problems. To improve mathematical reasoning performance, we propose Self-Consistency-over-Paraphrases (SCoP), which diversifies reasoning paths from specific surface forms of the problem. We evaluate our approach on four mathematics reasoning benchmarks over three large language models and show that SCoP improves mathematical reasoning performance over vanilla self-consistency, particularly for problems initially deemed unsolvable. Finally, we provide additional experiments and discussion regarding problem difficulty and surface forms, including cross-model difficulty agreement and paraphrasing transferability, and Variance of Variations (VOV) for language model evaluation.

Title: Towards Coarse-to-Fine Evaluation of Inference Efficiency for Large Language Models

• Authors: Yushuo Chen, Tianyi Tang, Erge Xiang, Linjiang Li, Wayne Xin Zhao, Jing Wang, Yunpeng Chai, Ji-Rong Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11502
• Pdf URL: https://arxiv.org/pdf/2404.11502
• Copy Paste: [[2404.11502]] Towards Coarse-to-Fine Evaluation of Inference Efficiency for Large Language Models(https://arxiv.org/abs/2404.11502)
• Keywords: transformer, large language model
• Abstract: In real world, large language models (LLMs) can serve as the assistant to help users accomplish their jobs, and also support the development of advanced applications. For the wide application of LLMs, the inference efficiency is an essential concern, which has been widely studied in existing work, and numerous optimization algorithms and code libraries have been proposed to improve it. Nonetheless, users still find it challenging to compare the effectiveness of all the above methods and understand the underlying mechanisms. In this work, we perform a detailed coarse-to-fine analysis of the inference performance of various code libraries. To evaluate the overall effectiveness, we examine four usage scenarios within two practical applications. We further provide both theoretical and empirical fine-grained analyses of each module in the Transformer architecture. Our experiments yield comprehensive results that are invaluable for researchers to evaluate code libraries and improve inference strategies.

Title: JointViT: Modeling Oxygen Saturation Levels with Joint Supervision on Long-Tailed OCTA

• Authors: Zeyu Zhang, Xuyin Qi, Mingxi Chen, Guangxi Li, Ryan Pham, Ayub Zuhair, Ella Berry, Zhibin Liao, Owen Siggs, Robert Mclaughlin, Jamie Craig, Minh-Son To
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.11525
• Pdf URL: https://arxiv.org/pdf/2404.11525
• Copy Paste: [[2404.11525]] JointViT: Modeling Oxygen Saturation Levels with Joint Supervision on Long-Tailed OCTA(https://arxiv.org/abs/2404.11525)
• Keywords: transformer
• Abstract: The oxygen saturation level in the blood (SaO2) is crucial for health, particularly in relation to sleep-related breathing disorders. However, continuous monitoring of SaO2 is time-consuming and highly variable depending on patients' conditions. Recently, optical coherence tomography angiography (OCTA) has shown promising development in rapidly and effectively screening eye-related lesions, offering the potential for diagnosing sleep-related disorders. To bridge this gap, our paper presents three key contributions. Firstly, we propose JointViT, a novel model based on the Vision Transformer architecture, incorporating a joint loss function for supervision. Secondly, we introduce a balancing augmentation technique during data preprocessing to improve the model's performance, particularly on the long-tail distribution within the OCTA dataset. Lastly, through comprehensive experiments on the OCTA dataset, our proposed method significantly outperforms other state-of-the-art methods, achieving improvements of up to 12.28% in overall accuracy. This advancement lays the groundwork for the future utilization of OCTA in diagnosing sleep-related disorders. See project website https://steve-zeyu-zhang.github.io/JointViT

Title: Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization

• Authors: Costas Mavromatis, Petros Karypis, George Karypis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11531
• Pdf URL: https://arxiv.org/pdf/2404.11531
• Copy Paste: [[2404.11531]] Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization(https://arxiv.org/abs/2404.11531)
• Keywords: large language model
• Abstract: Fusing knowledge from multiple Large Language Models (LLMs) can combine their diverse strengths to achieve improved performance on a given task. However, current fusion approaches either rely on learning-based fusers that do not generalize to new LLMs, or do not take into account how well each LLM understands the input. In this work, we study LLM fusion at test-time, which enables leveraging knowledge from arbitrary user-specified LLMs during inference. We introduce Pack of LLMs (PackLLM), an effective method for test-time fusion that leverages each LLM's expertise, given an input prompt. PackLLM performs model fusion by solving an optimization problem for determining each LLM's importance, so that perplexity over the input prompt is minimized. First, our simple PackLLM-sim variant validates that perplexity is a good indicator for measuring each LLM's expertise. Second, our PackLLM-opt variant approximately solves the perplexity minimization problem via a greedy algorithm. The derived importance weights are used to combine the LLMs during inference. We conduct experiments with over 100 total LLMs on a diverse set of tasks. Experimental results show that (i) perplexity is a reliable measure for LLM fusion, (ii) PackLLM outperforms test-time fusion baselines by 1.89% accuracy points, and (iii) PackLLM can leverage new LLMs to improve performance over learning-based fusion approaches by 3.92-11.94% accuracy points.

Title: Decomposing and Editing Predictions by Modeling Model Computation

• Authors: Harshay Shah, Andrew Ilyas, Aleksander Madry
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.11534
• Pdf URL: https://arxiv.org/pdf/2404.11534
• Copy Paste: [[2404.11534]] Decomposing and Editing Predictions by Modeling Model Computation(https://arxiv.org/abs/2404.11534)
• Keywords: attack, robust
• Abstract: How does the internal computation of a machine learning model transform inputs into predictions? In this paper, we introduce a task called component modeling that aims to address this question. The goal of component modeling is to decompose an ML model's prediction in terms of its components -- simple functions (e.g., convolution filters, attention heads) that are the "building blocks" of model computation. We focus on a special case of this task, component attribution, where the goal is to estimate the counterfactual impact of individual components on a given prediction. We then present COAR, a scalable algorithm for estimating component attributions; we demonstrate its effectiveness across models, datasets, and modalities. Finally, we show that component attributions estimated with COAR directly enable model editing across five tasks, namely: fixing model errors, ``forgetting'' specific classes, boosting subpopulation robustness, localizing backdoor attacks, and improving robustness to typographic attacks. We provide code for COAR at https://github.com/MadryLab/modelcomponents .

Title: FedPFT: Federated Proxy Fine-Tuning of Foundation Models

• Authors: Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan, Chenglu Wen, Ruisheng Zhang, Cheng Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11536
• Pdf URL: https://arxiv.org/pdf/2404.11536
• Copy Paste: [[2404.11536]] FedPFT: Federated Proxy Fine-Tuning of Foundation Models(https://arxiv.org/abs/2404.11536)
• Keywords: privacy, protect, federate
• Abstract: Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges a promising strategy for protecting data privacy and valuable FMs. Existing methods fine-tune FM by allocating sub-FM to clients in FL, however, leading to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients. In this paper, we propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation in downstream tasks through FL by two key modules. First, the sub-FM construction module employs a layer-wise compression approach, facilitating comprehensive FM fine-tuning across all layers by emphasizing those crucial neurons. Second, the sub-FM alignment module conducts a two-step distillations-layer-level and neuron-level-before and during FL fine-tuning respectively, to reduce error of gradient by accurately aligning sub-FM with FM under theoretical guarantees. Experimental results on seven commonly used datasets (i.e., four text and three vision) demonstrate the superiority of FedPFT.

Title: SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening

• Authors: Yu Zhong, Xiao Wu, Liang-Jian Deng, Zihan Cao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.11537
• Pdf URL: https://arxiv.org/pdf/2404.11537
• Copy Paste: [[2404.11537]] SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening(https://arxiv.org/abs/2404.11537)
• Keywords: diffusion
• Abstract: Pansharpening is a significant image fusion technique that merges the spatial content and spectral characteristics of remote sensing images to generate high-resolution multispectral images. Recently, denoising diffusion probabilistic models have been gradually applied to visual tasks, enhancing controllable image generation through low-rank adaptation (LoRA). In this paper, we introduce a spatial-spectral integrated diffusion model for the remote sensing pansharpening task, called SSDiff, which considers the pansharpening process as the fusion process of spatial and spectral components from the perspective of subspace decomposition. Specifically, SSDiff utilizes spatial and spectral branches to learn spatial details and spectral features separately, then employs a designed alternating projection fusion module (APFM) to accomplish the fusion. Furthermore, we propose a frequency modulation inter-branch module (FMIM) to modulate the frequency distribution between branches. The two components of SSDiff can perform favorably against the APFM when utilizing a LoRA-like branch-wise alternative fine-tuning method. It refines SSDiff to capture component-discriminating features more sufficiently. Finally, extensive experiments on four commonly used datasets, i.e., WorldView-3, WorldView-2, GaoFen-2, and QuickBird, demonstrate the superiority of SSDiff both visually and quantitatively. The code will be made open source after possible acceptance.

Title: GenFighter: A Generative and Evolutive Textual Attack Removal

• Authors: Md Athikul Islam, Edoardo Serra, Sushil Jajodia
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11538
• Pdf URL: https://arxiv.org/pdf/2404.11538
• Copy Paste: [[2404.11538]] GenFighter: A Generative and Evolutive Textual Attack Removal(https://arxiv.org/abs/2404.11538)
• Keywords: defense, attack, robust, transformer, generative
• Abstract: Adversarial attacks pose significant challenges to deep neural networks (DNNs) such as Transformer models in natural language processing (NLP). This paper introduces a novel defense strategy, called GenFighter, which enhances adversarial robustness by learning and reasoning on the training classification distribution. GenFighter identifies potentially malicious instances deviating from the distribution, transforms them into semantically equivalent instances aligned with the training data, and employs ensemble techniques for a unified and robust response. By conducting extensive experiments, we show that GenFighter outperforms state-of-the-art defenses in accuracy under attack and attack success rate metrics. Additionally, it requires a high number of queries per attack, making the attack more challenging in real scenarios. The ablation study shows that our approach integrates transfer learning, a generative/evolutive procedure, and an ensemble method, providing an effective defense against NLP adversarial attacks.

Title: Evaluating Span Extraction in Generative Paradigm: A Reflection on Aspect-Based Sentiment Analysis

• Authors: Soyoung Yang, Won Ik Cho
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11539
• Pdf URL: https://arxiv.org/pdf/2404.11539
• Copy Paste: [[2404.11539]] Evaluating Span Extraction in Generative Paradigm: A Reflection on Aspect-Based Sentiment Analysis(https://arxiv.org/abs/2404.11539)
• Keywords: extraction, generative
• Abstract: In the era of rapid evolution of generative language models within the realm of natural language processing, there is an imperative call to revisit and reformulate evaluation methodologies, especially in the domain of aspect-based sentiment analysis (ABSA). This paper addresses the emerging challenges introduced by the generative paradigm, which has moderately blurred traditional boundaries between understanding and generation tasks. Building upon prevailing practices in the field, we analyze the advantages and shortcomings associated with the prevalent ABSA evaluation paradigms. Through an in-depth examination, supplemented by illustrative examples, we highlight the intricacies involved in aligning generative outputs with other evaluative metrics, specifically those derived from other tasks, including question answering. While we steer clear of advocating for a singular and definitive metric, our contribution lies in paving the path for a comprehensive guideline tailored for ABSA evaluations in this generative paradigm. In this position paper, we aim to provide practitioners with profound reflections, offering insights and directions that can aid in navigating this evolving landscape, ensuring evaluations that are both accurate and reflective of generative capabilities.

Title: Quantifying Multilingual Performance of Large Language Models Across Languages

• Authors: Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ninghao Liu, Mengnan Du
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11553
• Pdf URL: https://arxiv.org/pdf/2404.11553
• Copy Paste: [[2404.11553]] Quantifying Multilingual Performance of Large Language Models Across Languages(https://arxiv.org/abs/2404.11553)
• Keywords: large language model
• Abstract: The training process of Large Language Models (LLMs) requires extensive text corpus. However, these data are often unevenly distributed in different languages. As a result, LLMs perform well on common languages, such as English, German, and French, but perform poorly on low-resource languages. However, currently there is no work to quantitatively measure the performance of LLMs in low-resource languages. To fill this gap, we proposed the Language Ranker that aims to benchmark and rank different languages according to the performance of LLMs on those languages. We employ the LLM's performance on the English corpus as a baseline to compare the performances of different languages and English. We have the following three findings: 1. The performance rankings of different LLMs in all languages are roughly the same. 2. LLMs with different sizes have the same partial order of performance. 3. There is a strong correlation between LlaMa2's performance in different languages and the proportion of the pre-training corpus. These findings illustrate that the Language Ranker can be used as an indicator to measure the language performance of LLMs.

Title: Predicting Long-horizon Futures by Conditioning on Geometry and Time

• Authors: Tarasha Khurana, Deva Ramanan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11554
• Pdf URL: https://arxiv.org/pdf/2404.11554
• Copy Paste: [[2404.11554]] Predicting Long-horizon Futures by Conditioning on Geometry and Time(https://arxiv.org/abs/2404.11554)
• Keywords: diffusion
• Abstract: Our work explores the task of generating future sensor observations conditioned on the past. We are motivated by `predictive coding' concepts from neuroscience as well as robotic applications such as self-driving vehicles. Predictive video modeling is challenging because the future may be multi-modal and learning at scale remains computationally expensive for video processing. To address both challenges, our key insight is to leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We repurpose image models for video prediction by conditioning on new frame timestamps. Such models can be trained with videos of both static and dynamic scenes. To allow them to be trained with modestly-sized datasets, we introduce invariances by factoring out illumination and texture by forcing the model to predict (pseudo) depth, readily obtained for in-the-wild videos via off-the-shelf monocular depth networks. In fact, we show that simply modifying networks to predict grayscale pixels already improves the accuracy of video prediction. Given the extra controllability with timestamp conditioning, we propose sampling schedules that work better than the traditional autoregressive and hierarchical sampling strategies. Motivated by probabilistic metrics from the object forecasting literature, we create a benchmark for video prediction on a diverse set of videos spanning indoor and outdoor scenes and a large vocabulary of objects. Our experiments illustrate the effectiveness of learning to condition on timestamps, and show the importance of predicting the future with invariant modalities.

Title: MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation

• Authors: Kuan-Chieh (Jackson)Wang, Daniil Ostashev, Yuwei Fang, Sergey Tulyakov, Kfir Aberman
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.11565
• Pdf URL: https://arxiv.org/pdf/2404.11565
• Copy Paste: [[2404.11565]] MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation(https://arxiv.org/abs/2404.11565)
• Keywords: diffusion, large language model
• Abstract: We introduce a new architecture for personalization of text-to-image diffusion models, coined Mixture-of-Attention (MoA). Inspired by the Mixture-of-Experts mechanism utilized in large language models (LLMs), MoA distributes the generation workload between two attention pathways: a personalized branch and a non-personalized prior branch. MoA is designed to retain the original model's prior by fixing its attention layers in the prior branch, while minimally intervening in the generation process with the personalized branch that learns to embed subjects in the layout and context generated by the prior branch. A novel routing mechanism manages the distribution of pixels in each layer across these branches to optimize the blend of personalized and generic content creation. Once trained, MoA facilitates the creation of high-quality, personalized images featuring multiple subjects with compositions and interactions as diverse as those generated by the original model. Crucially, MoA enhances the distinction between the model's pre-existing capability and the newly augmented personalized intervention, thereby offering a more disentangled subject-context control that was previously unattainable. Project page: https://snap-research.github.io/mixture-of-attention

Title: On the Scalability of GNNs for Molecular Graphs

• Authors: Maciej Sypetkowski, Frederik Wenkel, Farimah Poursafaei, Nia Dickson, Karush Suri, Philip Fradkin, Dominique Beaini
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11568
• Pdf URL: https://arxiv.org/pdf/2404.11568
• Copy Paste: [[2404.11568]] On the Scalability of GNNs for Molecular Graphs(https://arxiv.org/abs/2404.11568)
• Keywords: transformer
• Abstract: Scaling deep learning models has been at the heart of recent revolutions in language modelling and image generation. Practitioners have observed a strong relationship between model size, dataset size, and performance. However, structure-based architectures such as Graph Neural Networks (GNNs) are yet to show the benefits of scale mainly due to the lower efficiency of sparse operations, large data requirements, and lack of clarity about the effectiveness of various architectures. We address this drawback of GNNs by studying their scaling behavior. Specifically, we analyze message-passing networks, graph Transformers, and hybrid architectures on the largest public collection of 2D molecular graphs. For the first time, we observe that GNNs benefit tremendously from the increasing scale of depth, width, number of molecules, number of labels, and the diversity in the pretraining datasets, resulting in a 30.25% improvement when scaling to 1 billion parameters and 28.98% improvement when increasing size of dataset to eightfold. We further demonstrate strong finetuning scaling behavior on 38 tasks, outclassing previous large models. We hope that our work paves the way for an era where foundational GNNs drive pharmaceutical drug discovery.

Title: Towards Reliable Empirical Machine Unlearning Evaluation: A Game-Theoretic View

• Authors: Yiwen Tu, Pingbang Hu, Jiaqi Ma
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11577
• Pdf URL: https://arxiv.org/pdf/2404.11577
• Copy Paste: [[2404.11577]] Towards Reliable Empirical Machine Unlearning Evaluation: A Game-Theoretic View(https://arxiv.org/abs/2404.11577)
• Keywords: protect, attack, membership infer
• Abstract: Machine unlearning is the process of updating machine learning models to remove the information of specific training data samples, in order to comply with data protection regulations that allow individuals to request the removal of their personal data. Despite the recent development of numerous unlearning algorithms, reliable evaluation of these algorithms remains an open research question. In this work, we focus on membership inference attack (MIA) based evaluation, one of the most common approaches for evaluating unlearning algorithms, and address various pitfalls of existing evaluation metrics that lack reliability. Specifically, we propose a game-theoretic framework that formalizes the evaluation process as a game between unlearning algorithms and MIA adversaries, measuring the data removal efficacy of unlearning algorithms by the capability of the MIA adversaries. Through careful design of the game, we demonstrate that the natural evaluation metric induced from the game enjoys provable guarantees that the existing evaluation metrics fail to satisfy. Furthermore, we propose a practical and efficient algorithm to estimate the evaluation metric induced from the game, and demonstrate its effectiveness through both theoretical analysis and empirical experiments. This work presents a novel and reliable approach to empirically evaluating unlearning algorithms, paving the way for the development of more effective unlearning techniques.

Title: Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding

• Authors: Zezhong Fan, Xiaohan Li, Chenhao Fang, Topojoy Biswas, Kaushiki Nag, Jianpeng Xu, Kannan Achan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11589
• Pdf URL: https://arxiv.org/pdf/2404.11589
• Copy Paste: [[2404.11589]] Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding(https://arxiv.org/abs/2404.11589)
• Keywords: diffusion, generative
• Abstract: The rapid evolution of text-to-image diffusion models has opened the door of generative AI, enabling the translation of textual descriptions into visually compelling images with remarkable quality. However, a persistent challenge within this domain is the optimization of prompts to effectively convey abstract concepts into concrete objects. For example, text encoders can hardly express "peace", while can easily illustrate olive branches and white doves. This paper introduces a novel approach named Prompt Optimizer for Abstract Concepts (POAC) specifically designed to enhance the performance of text-to-image diffusion models in interpreting and generating images from abstract concepts. We propose a Prompt Language Model (PLM), which is initialized from a pre-trained language model, and then fine-tuned with a curated dataset of abstract concept prompts. The dataset is created with GPT-4 to extend the abstract concept to a scene and concrete objects. Our framework employs a Reinforcement Learning (RL)-based optimization strategy, focusing on the alignment between the generated images by a stable diffusion model and optimized prompts. Through extensive experiments, we demonstrate that our proposed POAC significantly improves the accuracy and aesthetic quality of generated images, particularly in the description of abstract concepts and alignment with optimized prompts. We also present a comprehensive analysis of our model's performance across diffusion models under different settings, showcasing its versatility and effectiveness in enhancing abstract concept representation.

Title: A Subspace-Constrained Tyler's Estimator and its Applications to Structure from Motion

• Authors: Feng Yu, Teng Zhang, Gilad Lerman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11590
• Pdf URL: https://arxiv.org/pdf/2404.11590
• Copy Paste: [[2404.11590]] A Subspace-Constrained Tyler's Estimator and its Applications to Structure from Motion(https://arxiv.org/abs/2404.11590)
• Keywords: robust
• Abstract: We present the subspace-constrained Tyler's estimator (STE) designed for recovering a low-dimensional subspace within a dataset that may be highly corrupted with outliers. STE is a fusion of the Tyler's M-estimator (TME) and a variant of the fast median subspace. Our theoretical analysis suggests that, under a common inlier-outlier model, STE can effectively recover the underlying subspace, even when it contains a smaller fraction of inliers relative to other methods in the field of robust subspace recovery. We apply STE in the context of Structure from Motion (SfM) in two ways: for robust estimation of the fundamental matrix and for the removal of outlying cameras, enhancing the robustness of the SfM pipeline. Numerical experiments confirm the state-of-the-art performance of our method in these applications. This research makes significant contributions to the field of robust subspace recovery, particularly in the context of computer vision and 3D reconstruction.

Title: IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination

• Authors: Xi Chen (1), Sida Peng (1), Dongchen Yang (1), Yuan Liu (2), Bowen Pan (3), Chengfei Lv (3), Xiaowei Zhou (1) ((1) Zhejiang University, (2) The University of Hong Kong, (3) Tao Technology Department, Alibaba Group)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11593
• Pdf URL: https://arxiv.org/pdf/2404.11593
• Copy Paste: [[2404.11593]] IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination(https://arxiv.org/abs/2404.11593)
• Keywords: diffusion, generative
• Abstract: This paper aims to recover object materials from posed images captured under an unknown static lighting condition. Recent methods solve this task by optimizing material parameters through differentiable physically based rendering. However, due to the coupling between object geometry, materials, and environment lighting, there is inherent ambiguity during the inverse rendering process, preventing previous methods from obtaining accurate results. To overcome this ill-posed problem, our key idea is to learn the material prior with a generative model for regularizing the optimization process. We observe that the general rendering equation can be split into diffuse and specular shading terms, and thus formulate the material prior as diffusion models of albedo and specular. Thanks to this design, our model can be trained using the existing abundant 3D object data, and naturally acts as a versatile tool to resolve the ambiguity when recovering material representations from RGB images. In addition, we develop a coarse-to-fine training strategy that leverages estimated materials to guide diffusion models to satisfy multi-view consistent constraints, leading to more stable and accurate results. Extensive experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery. The code will be available at https://zju3dv.github.io/IntrinsicAnything.

Title: InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

• Authors: Zhiheng Liu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Jie Xiao, Kai Zhu, Nan Xue, Yu Liu, Yujun Shen, Yang Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11613
• Pdf URL: https://arxiv.org/pdf/2404.11613
• Copy Paste: [[2404.11613]] InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior(https://arxiv.org/abs/2404.11613)
• Keywords: diffusion
• Abstract: 3D Gaussians have recently emerged as an efficient representation for novel view synthesis. This work studies its editability with a particular focus on the inpainting task, which aims to supplement an incomplete set of 3D Gaussians with additional points for visually harmonious rendering. Compared to 2D inpainting, the crux of inpainting 3D Gaussians is to figure out the rendering-relevant properties of the introduced points, whose optimization largely benefits from their initial 3D positions. To this end, we propose to guide the point initialization with an image-conditioned depth completion model, which learns to directly restore the depth map based on the observed image. Such a design allows our model to fill in depth values at an aligned scale with the original depth, and also to harness strong generalizability from largescale diffusion prior. Thanks to the more accurate depth completion, our approach, dubbed InFusion, surpasses existing alternatives with sufficiently better fidelity and efficiency under various complex scenarios. We further demonstrate the effectiveness of InFusion with several practical applications, such as inpainting with user-specific texture or with novel object insertion.

Title: Factorized Diffusion: Perceptual Illusions by Noise Decomposition

• Authors: Daniel Geng, Inbum Park, Andrew Owens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11615
• Pdf URL: https://arxiv.org/pdf/2404.11615
• Copy Paste: [[2404.11615]] Factorized Diffusion: Perceptual Illusions by Noise Decomposition(https://arxiv.org/abs/2404.11615)
• Keywords: diffusion
• Abstract: Given a factorization of an image into a sum of linear components, we present a zero-shot method to control each individual component through diffusion model sampling. For example, we can decompose an image into low and high spatial frequencies and condition these components on different text prompts. This produces hybrid images, which change appearance depending on viewing distance. By decomposing an image into three frequency subbands, we can generate hybrid images with three prompts. We also use a decomposition into grayscale and color components to produce images whose appearance changes when they are viewed in grayscale, a phenomena that naturally occurs under dim lighting. And we explore a decomposition by a motion blur kernel, which produces images that change appearance under motion blurring. Our method works by denoising with a composite noise estimate, built from the components of noise estimates conditioned on different prompts. We also show that for certain decompositions, our method recovers prior approaches to compositional generation and spatial control. Finally, we show that we can extend our approach to generate hybrid images from real images. We do this by holding one component fixed and generating the remaining components, effectively solving an inverse problem.