2024-03-12

Title: Privacy Amplification for the Gaussian Mechanism via Bounded Support

• Authors: Shengyuan Hu, Saeed Mahloujifar, Virginia Smith, Kamalika Chaudhuri, Chuan Guo
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05598
• Pdf URL: https://arxiv.org/pdf/2403.05598
• Copy Paste: [[2403.05598]] Privacy Amplification for the Gaussian Mechanism via Bounded Support(https://arxiv.org/abs/2403.05598)
• Keywords: privacy
• Abstract: Data-dependent privacy accounting frameworks such as per-instance differential privacy (pDP) and Fisher information loss (FIL) confer fine-grained privacy guarantees for individuals in a fixed training dataset. These guarantees can be desirable compared to vanilla DP in real world settings as they tightly upper-bound the privacy leakage for a $\textit{specific}$ individual in an $\textit{actual}$ dataset, rather than considering worst-case datasets. While these frameworks are beginning to gain popularity, to date, there is a lack of private mechanisms that can fully leverage advantages of data-dependent accounting. To bridge this gap, we propose simple modifications of the Gaussian mechanism with bounded support, showing that they amplify privacy guarantees under data-dependent accounting. Experiments on model training with DP-SGD show that using bounded support Gaussian mechanisms can provide a reduction of the pDP bound $\epsilon$ by as much as 30% without negative effects on model utility.

Title: Select High-Level Features: Efficient Experts from a Hierarchical Classification Network

• Authors: André Kelm, Niels Hannemann, Bruno Heberle, Lucas Schmidt, Tim Rolff, Christian Wilms, Ehsan Yaghoubi, Simone Frintrop
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05601
• Pdf URL: https://arxiv.org/pdf/2403.05601
• Copy Paste: [[2403.05601]] Select High-Level Features: Efficient Experts from a Hierarchical Classification Network(https://arxiv.org/abs/2403.05601)
• Keywords: extraction
• Abstract: This study introduces a novel expert generation method that dynamically reduces task and computational complexity without compromising predictive performance. It is based on a new hierarchical classification network topology that combines sequential processing of generic low-level features with parallelism and nesting of high-level features. This structure allows for the innovative extraction technique: the ability to select only high-level features of task-relevant categories. In certain cases, it is possible to skip almost all unneeded high-level features, which can significantly reduce the inference cost and is highly beneficial in resource-constrained conditions. We believe this method paves the way for future network designs that are lightweight and adaptable, making them suitable for a wide range of applications, from compact edge devices to large-scale clouds. In terms of dynamic inference our methodology can achieve an exclusion of up to 88.7\,\% of parameters and 73.4\,\% fewer giga-multiply accumulate (GMAC) operations, analysis against comparative baselines showing an average reduction of 47.6\,\% in parameters and 5.8\,\% in GMACs across the cases we evaluated.

Title: Unfamiliar Finetuning Examples Control How Language Models Hallucinate

• Authors: Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, Sergey Levine
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05612
• Pdf URL: https://arxiv.org/pdf/2403.05612
• Copy Paste: [[2403.05612]] Unfamiliar Finetuning Examples Control How Language Models Hallucinate(https://arxiv.org/abs/2403.05612)
• Keywords: large language model
• Abstract: Large language models (LLMs) have a tendency to generate plausible-sounding yet factually incorrect responses, especially when queried on unfamiliar concepts. In this work, we explore the underlying mechanisms that govern how finetuned LLMs hallucinate. Our investigation reveals an interesting pattern: as inputs become more unfamiliar, LLM outputs tend to default towards a ``hedged'' prediction, whose form is determined by how the unfamiliar examples in the finetuning data are supervised. Thus, by strategically modifying these examples' supervision, we can control LLM predictions for unfamiliar inputs (e.g., teach them to say ``I don't know''). Based on these principles, we develop an RL approach that more reliably mitigates hallucinations for long-form generation tasks, by tackling the challenges presented by reward model hallucinations. We validate our findings with a series of controlled experiments in multiple-choice QA on MMLU, as well as long-form biography and book/movie plot generation tasks.

Title: Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification

• Authors: Zhijian Li, Stefan Larson, Kevin Leach
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05640
• Pdf URL: https://arxiv.org/pdf/2403.05640
• Copy Paste: [[2403.05640]] Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification(https://arxiv.org/abs/2403.05640)
• Keywords: robust
• Abstract: Intent classifiers must be able to distinguish when a user's utterance does not belong to any supported intent to avoid producing incorrect and unrelated system responses. Although out-of-scope (OOS) detection for intent classifiers has been studied, previous work has not yet studied changes in classifier performance against hard-negative out-of-scope utterances (i.e., inputs that share common features with in-scope data, but are actually out-of-scope). We present an automated technique to generate hard-negative OOS data using ChatGPT. We use our technique to build five new hard-negative OOS datasets, and evaluate each against three benchmark intent classifiers. We show that classifiers struggle to correctly identify hard-negative OOS utterances more than general OOS utterances. Finally, we show that incorporating hard-negative OOS data for training improves model robustness when detecting hard-negative OOS data and general OOS data. Our technique, datasets, and evaluation address an important void in the field, offering a straightforward and inexpensive way to collect hard-negative OOS data and improve intent classifiers' robustness.

Title: Feature CAM: Interpretable AI in Image Classification

• Authors: Frincy Clement, Ji Yang, Irene Cheng
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.05658
• Pdf URL: https://arxiv.org/pdf/2403.05658
• Copy Paste: [[2403.05658]] Feature CAM: Interpretable AI in Image Classification(https://arxiv.org/abs/2403.05658)
• Keywords: security, interpretability
• Abstract: Deep Neural Networks have often been called the black box because of the complex, deep architecture and non-transparency presented by the inner layers. There is a lack of trust to use Artificial Intelligence in critical and high-precision fields such as security, finance, health, and manufacturing industries. A lot of focused work has been done to provide interpretable models, intending to deliver meaningful insights into the thoughts and behavior of neural networks. In our research, we compare the state-of-the-art methods in the Activation-based methods (ABM) for interpreting predictions of CNN models, specifically in the application of Image Classification. We then extend the same for eight CNN-based architectures to compare the differences in visualization and thus interpretability. We introduced a novel technique Feature CAM, which falls in the perturbation-activation combination, to create fine-grained, class-discriminative visualizations. The resulting saliency maps from our experiments proved to be 3-4 times better human interpretable than the state-of-the-art in ABM. At the same time it reserves machine interpretability, which is the average confidence scores in classification.

Title: Audio-Synchronized Visual Animation

• Authors: Lin Zhang, Shentong Mo, Yijing Zhang, Pedro Morgado
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05659
• Pdf URL: https://arxiv.org/pdf/2403.05659
• Copy Paste: [[2403.05659]] Audio-Synchronized Visual Animation(https://arxiv.org/abs/2403.05659)
• Keywords: diffusion
• Abstract: Current visual generation methods can produce high quality videos guided by texts. However, effectively controlling object dynamics remains a challenge. This work explores audio as a cue to generate temporally synchronized image animations. We introduce Audio Synchronized Visual Animation (ASVA), a task animating a static image to demonstrate motion dynamics, temporally guided by audio clips across multiple classes. To this end, we present AVSync15, a dataset curated from VGGSound with videos featuring synchronized audio visual events across 15 categories. We also present a diffusion model, AVSyncD, capable of generating dynamic animations guided by audios. Extensive evaluations validate AVSync15 as a reliable benchmark for synchronized generation and demonstrate our models superior performance. We further explore AVSyncDs potential in a variety of audio synchronized generation tasks, from generating full videos without a base image to controlling object motions with various sounds. We hope our established benchmark can open new avenues for controllable visual generation. More videos on project webpage https://lzhangbj.github.io/projects/asva/asva.html.

Title: A Formal Analysis of SCTP: Attack Synthesis and Patch Verification

• Authors: Jacob Ginesin, Max von Hippel, Evan Defloor, Cristina Nita-Rotaru, Michael Tüxen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.05663
• Pdf URL: https://arxiv.org/pdf/2403.05663
• Copy Paste: [[2403.05663]] A Formal Analysis of SCTP: Attack Synthesis and Patch Verification(https://arxiv.org/abs/2403.05663)
• Keywords: secure, security, attack
• Abstract: SCTP is a transport protocol offering features such as multi-homing, multi-streaming, and message-oriented delivery. Its two main implementations were subjected to conformance tests using the PacketDrill tool. Conformance testing is not exhaustive and a recent vulnerability (CVE-2021-3772) showed SCTP is not immune to attacks. Changes addressing the vulnerability were implemented, but the question remains whether other flaws might persist in the protocol design. We study the security of the SCTP design, taking a rigorous approach rooted in formal methods. We create a formal Promela model of SCTP, and define 10 properties capturing the essential protocol functionality based on its RFC specification and consultation with the lead RFC author. Then we show using the Spin model checker that our model satisfies these properties. We define 4 attacker models - Off-Path, where the attacker is an outsider that can spoof the port and IP of a peer; Evil-Server, where the attacker is a malicious peer; Replay, where an attacker can capture and replay, but not modify, packets; and On-Path, where the attacker controls the channel between peers. We modify an attack synthesis tool designed for transport protocols, Korg, to support our SCTP model and four attacker models. We synthesize 14 unique attacks using the attacker models - including the CVE vulnerability in the Off-Path attacker model, 4 attacks in the Evil-Server attacker model, an opportunistic ABORT attack in the Replay attacker model, and eight connection manipulation attacks in the On-Path attacker model. We show that the proposed patch eliminates the vulnerability and does not introduce new ones according to our model and protocol properties. Finally, we identify and analyze an ambiguity in the RFC, which we show can be interpreted insecurely. We propose an erratum and show that it eliminates the ambiguity.

Title: PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design

• Authors: Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, Tim Kraska
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05676
• Pdf URL: https://arxiv.org/pdf/2403.05676
• Copy Paste: [[2403.05676]] PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design(https://arxiv.org/abs/2403.05676)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) can enhance the generation quality of large language models (LLMs) by incorporating external token databases. However, retrievals from large databases can constitute a substantial portion of the overall generation time, particularly when retrievals are periodically performed to align the retrieved content with the latest states of generation. In this paper, we introduce PipeRAG, a novel algorithm-system co-design approach to reduce generation latency and enhance generation quality. PipeRAG integrates (1) pipeline parallelism to enable concurrent retrieval and generation processes, (2) flexible retrieval intervals to maximize the efficiency of pipeline parallelism, and (3) a performance model to automatically balance retrieval quality and latency based on the generation states and underlying hardware. Our evaluation shows that, by combining the three aforementioned methods, PipeRAG achieves up to 2.6$\times$ speedup in end-to-end generation latency while improving generation quality. These promising results showcase the effectiveness of co-designing algorithms with underlying systems, paving the way for the adoption of PipeRAG in future RAG systems.

Title: DP-TabICL: In-Context Learning with Differentially Private Tabular Data

• Authors: Alycia N. Carey, Karuna Bhaila, Kennedy Edemacu, Xintao Wu
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05681
• Pdf URL: https://arxiv.org/pdf/2403.05681
• Copy Paste: [[2403.05681]] DP-TabICL: In-Context Learning with Differentially Private Tabular Data(https://arxiv.org/abs/2403.05681)
• Keywords: privacy, protect, large language model
• Abstract: In-context learning (ICL) enables large language models (LLMs) to adapt to new tasks by conditioning on demonstrations of question-answer pairs and it has been shown to have comparable performance to costly model retraining and fine-tuning. Recently, ICL has been extended to allow tabular data to be used as demonstration examples by serializing individual records into natural language formats. However, it has been shown that LLMs can leak information contained in prompts, and since tabular data often contain sensitive information, understanding how to protect the underlying tabular data used in ICL is a critical area of research. This work serves as an initial investigation into how to use differential privacy (DP) -- the long-established gold standard for data privacy and anonymization -- to protect tabular data used in ICL. Specifically, we investigate the application of DP mechanisms for private tabular ICL via data privatization prior to serialization and prompting. We formulate two private ICL frameworks with provable privacy guarantees in both the local (LDP-TabICL) and global (GDP-TabICL) DP scenarios via injecting noise into individual records or group statistics, respectively. We evaluate our DP-based frameworks on eight real-world tabular datasets and across multiple ICL and DP settings. Our evaluations show that DP-based ICL can protect the privacy of the underlying tabular data while achieving comparable performance to non-LLM baselines, especially under high privacy regimes.

Title: Micro-Fracture Detection in Photovoltaic Cells with Hardware-Constrained Devices and Computer Vision

• Authors: Booy Vitas Faassen, Jorge Serrano, Paul D. Rosero-Montalvo
• Subjects: cs.CV, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2403.05694
• Pdf URL: https://arxiv.org/pdf/2403.05694
• Copy Paste: [[2403.05694]] Micro-Fracture Detection in Photovoltaic Cells with Hardware-Constrained Devices and Computer Vision(https://arxiv.org/abs/2403.05694)
• Keywords: robust
• Abstract: Solar energy is rapidly becoming a robust renewable energy source to conventional finite resources such as fossil fuels. It is harvested using interconnected photovoltaic panels, typically built with crystalline silicon cells, i.e. semiconducting materials that convert effectively the solar radiation into electricity. However, crystalline silicon is fragile and vulnerable to cracking over time or in predictive maintenance tasks, which can lead to electric isolation of parts of the solar cell and even failure, thus affecting the panel performance and reducing electricity generation. This work aims to developing a system for detecting cell cracks in solar panels to anticipate and alaert of a potential failure of the photovoltaic system by using computer vision techniques. Three scenarios are defined where these techniques will bring value. In scenario A, images are taken manually and the system detecting failures in the solar cells is not subject to any computationa constraints. In scenario B, an Edge device is placed near the solar farm, able to make inferences. Finally, in scenario C, a small microcontroller is placed in a drone flying over the solar farm and making inferences about the solar cells' states. Three different architectures are found the most suitable solutions, one for each scenario, namely the InceptionV3 model, an EfficientNetB0 model shrunk into full integer quantization, and a customized CNN architechture built with VGG16 blocks.

Title: SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes

• Authors: Mukul Bhutani, Kevin Robinson, Vinodkumar Prabhakaran, Shachi Dave, Sunipa Dev
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05696
• Pdf URL: https://arxiv.org/pdf/2403.05696
• Copy Paste: [[2403.05696]] SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes(https://arxiv.org/abs/2403.05696)
• Keywords: fair, generative
• Abstract: While generative multilingual models are rapidly being deployed, their safety and fairness evaluations are largely limited to resources collected in English. This is especially problematic for evaluations targeting inherently socio-cultural phenomena such as stereotyping, where it is important to build multi-lingual resources that reflect the stereotypes prevalent in respective language communities. However, gathering these resources, at scale, in varied languages and regions pose a significant challenge as it requires broad socio-cultural knowledge and can also be prohibitively expensive. To overcome this critical gap, we employ a recently introduced approach that couples LLM generations for scale with culturally situated validations for reliability, and build SeeGULL Multilingual, a global-scale multilingual dataset of social stereotypes, containing over 25K stereotypes, spanning 20 languages, with human annotations across 23 regions, and demonstrate its utility in identifying gaps in model evaluations. Content warning: Stereotypes shared in this paper can be offensive.

Title: Not just Birds and Cars: Generic, Scalable and Explainable Models for Professional Visual Recognition

• Authors: Junde Wu, Jiayuan Zhu, Min Xu, Yueming Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05703
• Pdf URL: https://arxiv.org/pdf/2403.05703
• Copy Paste: [[2403.05703]] Not just Birds and Cars: Generic, Scalable and Explainable Models for Professional Visual Recognition(https://arxiv.org/abs/2403.05703)
• Keywords: explainability, segmentation
• Abstract: Some visual recognition tasks are more challenging then the general ones as they require professional categories of images. The previous efforts, like fine-grained vision classification, primarily introduced models tailored to specific tasks, like identifying bird species or car brands with limited scalability and generalizability. This paper aims to design a scalable and explainable model to solve Professional Visual Recognition tasks from a generic standpoint. We introduce a biologically-inspired structure named Pro-NeXt and reveal that Pro-NeXt exhibits substantial generalizability across diverse professional fields such as fashion, medicine, and art-areas previously considered disparate. Our basic-sized Pro-NeXt-B surpasses all preceding task-specific models across 12 distinct datasets within 5 diverse domains. Furthermore, we find its good scaling property that scaling up Pro-NeXt in depth and width with increasing GFlops can consistently enhances its accuracy. Beyond scalability and adaptability, the intermediate features of Pro-NeXt achieve reliable object detection and segmentation performance without extra training, highlighting its solid explainability. We will release the code to foster further research in this area.

Title: $\mathtt{tsGT}$: Stochastic Time Series Modeling With Transformer

• Authors: Łukasz Kuciński, Witold Drzewakowski, Mateusz Olko, Piotr Kozakowski, Łukasz Maziarka, Marta Emilia Nowakowska, Łukasz Kaiser, Piotr Miłoś
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05713
• Pdf URL: https://arxiv.org/pdf/2403.05713
• Copy Paste: [[2403.05713]] $\mathtt{tsGT}$: Stochastic Time Series Modeling With Transformer(https://arxiv.org/abs/2403.05713)
• Keywords: transformer
• Abstract: Time series methods are of fundamental importance in virtually any field of science that deals with temporally structured data. Recently, there has been a surge of deterministic transformer models with time series-specific architectural biases. In this paper, we go in a different direction by introducing $\mathtt{tsGT}$, a stochastic time series model built on a general-purpose transformer architecture. We focus on using a well-known and theoretically justified rolling window backtesting and evaluation protocol. We show that $\mathtt{tsGT}$ outperforms the state-of-the-art models on MAD and RMSE, and surpasses its stochastic peers on QL and CRPS, on four commonly used datasets. We complement these results with a detailed analysis of $\mathtt{tsGT}$'s ability to model the data distribution and predict marginal quantile values.

Title: A Benchmark of Domain-Adapted Large Language Models for Generating Brief Hospital Course Summaries

• Authors: Asad Aali, Dave Van Veen, Yamin Ishraq Arefeen, Jason Hom, Christian Bluethgen, Eduardo Pontes Reis, Sergios Gatidis, Namuun Clifford, Joseph Daws, Arash S. Tehrani, Jangwon Kim, Akshay S. Chaudhari
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05720
• Pdf URL: https://arxiv.org/pdf/2403.05720
• Copy Paste: [[2403.05720]] A Benchmark of Domain-Adapted Large Language Models for Generating Brief Hospital Course Summaries(https://arxiv.org/abs/2403.05720)
• Keywords: large language model
• Abstract: Brief hospital course (BHC) summaries are common clinical documents generated by summarizing clinical notes. While large language models (LLMs) depict remarkable capabilities in automating real-world tasks, their capabilities for healthcare applications such as BHC synthesis have not been shown. To enable the adaptation of LLMs for BHC synthesis, we introduce a novel benchmark consisting of a pre-processed dataset extracted from MIMIC-IV notes, encapsulating clinical note, and brief hospital course (BHC) pairs. We assess the performance of two general-purpose LLMs and three healthcare-adapted LLMs to improve BHC synthesis from clinical notes. Using clinical notes as input for generating BHCs, we apply prompting-based (using in-context learning) and fine-tuning-based adaptation strategies to three open-source LLMs (Clinical-T5-Large, Llama2-13B, FLAN-UL2) and two proprietary LLMs (GPT-3.5, GPT-4). We quantitatively evaluate the performance of these LLMs across varying context-length inputs using conventional natural language similarity metrics. We further perform a qualitative study where five diverse clinicians blindly compare clinician-written BHCs and two LLM-generated BHCs for 30 samples across metrics of comprehensiveness, conciseness, factual correctness, and fluency. Overall, we present a new benchmark and pre-processed dataset for using LLMs in BHC synthesis from clinical notes. We observe high-quality summarization performance for both in-context proprietary and fine-tuned open-source LLMs using both quantitative metrics and a qualitative clinical reader study. We propose our work as a benchmark to motivate future works to adapt and assess the performance of LLMs in BHC synthesis.

Title: Inception Attacks: Immersive Hijacking in Virtual Reality Systems

• Authors: Zhuolin Yang, Cathy Yuanchen Li, Arman Bhalla, Ben Y. Zhao, Haitao Zheng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.05721
• Pdf URL: https://arxiv.org/pdf/2403.05721
• Copy Paste: [[2403.05721]] Inception Attacks: Immersive Hijacking in Virtual Reality Systems(https://arxiv.org/abs/2403.05721)
• Keywords: security, defense, attack, generative
• Abstract: Recent advances in virtual reality (VR) system provide fully immersive interactions that connect users with online resources, applications, and each other. Yet these immersive interfaces can make it easier for users to fall prey to a new type of security attacks. We introduce the inception attack, where an attacker controls and manipulates a user's interaction with their VR environment and applications, by trapping them inside a malicious VR application that masquerades as the full VR system. Once trapped in an "inception VR layer", all of the user's interactions with remote servers, network applications, and other VR users can be recorded or modified without their knowledge. This enables traditional attacks (recording passwords and modifying user actions in flight), as well as VR interaction attacks, where (with generative AI tools) two VR users interacting can experience two dramatically different conversations. In this paper, we introduce inception attacks and their design, and describe our implementation that works on all Meta Quest VR headsets. Our implementation of inception attacks includes a cloned version of the Meta Quest browser that can modify data as it's displayed to the user, and alter user input en route to the server (e.g. modify amount of $ transferred in a banking session). Our implementation also includes a cloned VRChat app, where an attacker can eavesdrop and modify live audio between two VR users. We then conduct a study on users with a range of VR experiences, execute the inception attack during their session, and debrief them about their experiences. Only 37% of users noticed the momentary visual "glitch" when the inception attack began, and all but 1 user attributed it to imperfections in the VR platform. Finally, we consider and discuss efficacy and tradeoffs for a wide range of potential inception defenses.

Title: Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text

• Authors: Sara Abdali, Richard Anarfi, CJ Barberan, Jia He
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05750
• Pdf URL: https://arxiv.org/pdf/2403.05750
• Copy Paste: [[2403.05750]] Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text(https://arxiv.org/abs/2403.05750)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have revolutionized the field of Natural Language Generation (NLG) by demonstrating an impressive ability to generate human-like text. However, their widespread usage introduces challenges that necessitate thoughtful examination, ethical scrutiny, and responsible practices. In this study, we delve into these challenges, explore existing strategies for mitigating them, with a particular emphasis on identifying AI-generated text as the ultimate solution. Additionally, we assess the feasibility of detection from a theoretical perspective and propose novel research directions to address the current limitations in this domain.

Title: MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process

• Authors: Xinyao Fan, Yueying Wu, Chang Xu, Yuhao Huang, Weiqing Liu, Jiang Bian
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05751
• Pdf URL: https://arxiv.org/pdf/2403.05751
• Copy Paste: [[2403.05751]] MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process(https://arxiv.org/abs/2403.05751)
• Keywords: diffusion, generative
• Abstract: Recently, diffusion probabilistic models have attracted attention in generative time series forecasting due to their remarkable capacity to generate high-fidelity samples. However, the effective utilization of their strong modeling ability in the probabilistic time series forecasting task remains an open question, partially due to the challenge of instability arising from their stochastic nature. To address this challenge, we introduce a novel Multi-Granularity Time Series Diffusion (MG-TSD) model, which achieves state-of-the-art predictive performance by leveraging the inherent granularity levels within the data as given targets at intermediate diffusion steps to guide the learning process of diffusion models. The way to construct the targets is motivated by the observation that the forward process of the diffusion model, which sequentially corrupts the data distribution to a standard normal distribution, intuitively aligns with the process of smoothing fine-grained data into a coarse-grained representation, both of which result in a gradual loss of fine distribution features. In the study, we derive a novel multi-granularity guidance diffusion loss function and propose a concise implementation method to effectively utilize coarse-grained data across various granularity levels. More importantly, our approach does not rely on additional external data, making it versatile and applicable across various domains. Extensive experiments conducted on real-world datasets demonstrate that our MG-TSD model outperforms existing time series prediction methods.

Title: Task-Oriented GNNs Training on Large Knowledge Graphs for Accurate and Efficient Modeling

• Authors: Hussein Abdallah, Waleed Afandi, Panos Kalnis, Essam Mansour
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05752
• Pdf URL: https://arxiv.org/pdf/2403.05752
• Copy Paste: [[2403.05752]] Task-Oriented GNNs Training on Large Knowledge Graphs for Accurate and Efficient Modeling(https://arxiv.org/abs/2403.05752)
• Keywords: extraction
• Abstract: A Knowledge Graph (KG) is a heterogeneous graph encompassing a diverse range of node and edge types. Heterogeneous Graph Neural Networks (HGNNs) are popular for training machine learning tasks like node classification and link prediction on KGs. However, HGNN methods exhibit excessive complexity influenced by the KG's size, density, and the number of node and edge types. AI practitioners handcraft a subgraph of a KG G relevant to a specific task. We refer to this subgraph as a task-oriented subgraph (TOSG), which contains a subset of task-related node and edge types in G. Training the task using TOSG instead of G alleviates the excessive computation required for a large KG. Crafting the TOSG demands a deep understanding of the KG's structure and the task's objectives. Hence, it is challenging and time-consuming. This paper proposes KG-TOSA, an approach to automate the TOSG extraction for task-oriented HGNN training on a large KG. In KG-TOSA, we define a generic graph pattern that captures the KG's local and global structure relevant to a specific task. We explore different techniques to extract subgraphs matching our graph pattern: namely (i) two techniques sampling around targeted nodes using biased random walk or influence scores, and (ii) a SPARQL-based extraction method leveraging RDF engines' built-in indices. Hence, it achieves negligible preprocessing overhead compared to the sampling techniques. We develop a benchmark of real KGs of large sizes and various tasks for node classification and link prediction. Our experiments show that KG-TOSA helps state-of-the-art HGNN methods reduce training time and memory usage by up to 70% while improving the model performance, e.g., accuracy and inference time.

Title: Hybrid Quantum-inspired Resnet and Densenet for Pattern Recognition with Completeness Analysis

• Authors: Andi Chen, Hua-Lei Yin, Zeng-Bing Chen, Shengjun Wu
• Subjects: cs.LG, cs.ET
• Abstract URL: https://arxiv.org/abs/2403.05754
• Pdf URL: https://arxiv.org/pdf/2403.05754
• Copy Paste: [[2403.05754]] Hybrid Quantum-inspired Resnet and Densenet for Pattern Recognition with Completeness Analysis(https://arxiv.org/abs/2403.05754)
• Keywords: attack
• Abstract: With the contemporary digital technology approaching, deep neural networks are emerging as the foundational algorithm of the artificial intelligence boom. Whereas, the evolving social demands have been emphasizing the necessity of novel methodologies to substitute traditional neural networks. Concurrently, the advent of the post-Moore era has spurred the development of quantum-inspired neural networks with outstanding potentials at certain circumstances. Nonetheless, a definitive evaluating system with detailed metrics is tremendously vital and indispensable owing to the vague indicators in comparison between the novel and traditional deep learning models at present. Hence, to improve and evaluate the performances of the novel neural networks more comprehensively in complex and unpredictable environments, we propose two hybrid quantum-inspired neural networks which are rooted in residual and dense connections respectively for pattern recognitions with completeness representation theory for model assessment. Comparative analyses against pure classical models with detailed frameworks reveal that our hybrid models with lower parameter complexity not only match the generalization power of pure classical models, but also outperform them notably in resistance to parameter attacks with various asymmetric noises. Moreover, our hybrid models indicate unique superiority to prevent gradient explosion problems through theoretical argumentation. Eventually, We elaborate on the application scenarios where our hybrid models are applicable and efficient, which paves the way for their industrialization and commercialization.

Title: Extending Activation Steering to Broad Skills and Multiple Behaviours

• Authors: Teun van der Weij, Massimo Poesio, Nandi Schoots
• Subjects: cs.LG, cs.AI, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.05767
• Pdf URL: https://arxiv.org/pdf/2403.05767
• Copy Paste: [[2403.05767]] Extending Activation Steering to Broad Skills and Multiple Behaviours(https://arxiv.org/abs/2403.05767)
• Keywords: large language model
• Abstract: Current large language models have dangerous capabilities, which are likely to become more problematic in the future. Activation steering techniques can be used to reduce risks from these capabilities. In this paper, we investigate the efficacy of activation steering for broad skills and multiple behaviours. First, by comparing the effects of reducing performance on general coding ability and Python-specific ability, we find that steering broader skills is competitive to steering narrower skills. Second, we steer models to become more or less myopic and wealth-seeking, among other behaviours. In our experiments, combining steering vectors for multiple different behaviours into one steering vector is largely unsuccessful. On the other hand, injecting individual steering vectors at different places in a model simultaneously is promising.

Title: Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning

• Authors: Bingqian Lin, Yanxin Long, Yi Zhu, Fengda Zhu, Xiaodan Liang, Qixiang Ye, Liang Lin
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.05770
• Pdf URL: https://arxiv.org/pdf/2403.05770
• Copy Paste: [[2403.05770]] Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning(https://arxiv.org/abs/2403.05770)
• Keywords: robust
• Abstract: Vision-and-language navigation (VLN) asks an agent to follow a given language instruction to navigate through a real 3D environment. Despite significant advances, conventional VLN agents are trained typically under disturbance-free environments and may easily fail in real-world scenarios, since they are unaware of how to deal with various possible disturbances, such as sudden obstacles or human interruptions, which widely exist and may usually cause an unexpected route deviation. In this paper, we present a model-agnostic training paradigm, called Progressive Perturbation-aware Contrastive Learning (PROPER) to enhance the generalization ability of existing VLN agents, by requiring them to learn towards deviation-robust navigation. Specifically, a simple yet effective path perturbation scheme is introduced to implement the route deviation, with which the agent is required to still navigate successfully following the original instruction. Since directly enforcing the agent to learn perturbed trajectories may lead to inefficient training, a progressively perturbed trajectory augmentation strategy is designed, where the agent can self-adaptively learn to navigate under perturbation with the improvement of its navigation performance for each specific trajectory. For encouraging the agent to well capture the difference brought by perturbation, a perturbation-aware contrastive learning mechanism is further developed by contrasting perturbation-free trajectory encodings and perturbation-based counterparts. Extensive experiments on R2R show that PROPER can benefit multiple VLN baselines in perturbation-free scenarios. We further collect the perturbed path data to construct an introspection subset based on the R2R, called Path-Perturbed R2R (PP-R2R). The results on PP-R2R show unsatisfying robustness of popular VLN agents and the capability of PROPER in improving the navigation robustness.

Title: Unveiling Ancient Maya Settlements Using Aerial LiDAR Image Segmentation

• Authors: Jincheng Zhang, William Ringle, Andrew R. Willis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05773
• Pdf URL: https://arxiv.org/pdf/2403.05773
• Copy Paste: [[2403.05773]] Unveiling Ancient Maya Settlements Using Aerial LiDAR Image Segmentation(https://arxiv.org/abs/2403.05773)
• Keywords: segmentation
• Abstract: Manual identification of archaeological features in LiDAR imagery is labor-intensive, costly, and requires archaeological expertise. This paper shows how recent advancements in deep learning (DL) present efficient solutions for accurately segmenting archaeological structures in aerial LiDAR images using the YOLOv8 neural network. The proposed approach uses novel pre-processing of the raw LiDAR data and dataset augmentation methods to produce trained YOLOv8 networks to improve accuracy, precision, and recall for the segmentation of two important Maya structure types: annular structures and platforms. The results show an IoU performance of 0.842 for platforms and 0.809 for annular structures which outperform existing approaches. Further, analysis via domain experts considers the topological consistency of segmented regions and performance vs. area providing important insights. The approach automates time-consuming LiDAR image labeling which significantly accelerates accurate analysis of historical landscapes.

Title: Spatial Clustering Approach for Vessel Path Identification

• Authors: Mohamed Abuella, M. Amine Atoui, Slawomir Nowaczyk, Simon Johansson, Ethan Faghan
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2403.05778
• Pdf URL: https://arxiv.org/pdf/2403.05778
• Copy Paste: [[2403.05778]] Spatial Clustering Approach for Vessel Path Identification(https://arxiv.org/abs/2403.05778)
• Keywords: segmentation
• Abstract: This paper addresses the challenge of identifying the paths for vessels with operating routes of repetitive paths, partially repetitive paths, and new paths. We propose a spatial clustering approach for labeling the vessel paths by using only position information. We develop a path clustering framework employing two methods: a distance-based path modeling and a likelihood estimation method. The former enhances the accuracy of path clustering through the integration of unsupervised machine learning techniques, while the latter focuses on likelihood-based path modeling and introduces segmentation for a more detailed analysis. The result findings highlight the superior performance and efficiency of the developed approach, as both methods for clustering vessel paths into five classes achieve a perfect F1-score. The approach aims to offer valuable insights for route planning, ultimately contributing to improving safety and efficiency in maritime transportation.

Title: ItD: Large Language Models Can Teach Themselves Induction through Deduction

• Authors: Wangtao Sun, Haotian Xu, Xuanqing Yu, Pei Chen, Shizhu He, Jun Zhao, Kang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05789
• Pdf URL: https://arxiv.org/pdf/2403.05789
• Copy Paste: [[2403.05789]] ItD: Large Language Models Can Teach Themselves Induction through Deduction(https://arxiv.org/abs/2403.05789)
• Keywords: large language model
• Abstract: Although Large Language Models (LLMs) are showing impressive performance on a wide range of Natural Language Processing tasks, researchers have found that they still have limited ability to conduct induction. Recent works mainly adopt ``post processes'' paradigms to improve the performance of LLMs on induction (e.g., the hypothesis search & refinement methods), but their performance is still constrained by the inherent inductive capability of the LLMs. In this paper, we propose a novel framework, Induction through Deduction (ItD), to enable the LLMs to teach themselves induction through deduction. The ItD framework is composed of two main components: a Deductive Data Generation module to generate induction data and a Naive Bayesian Induction module to optimize the fine-tuning and decoding of LLMs. Our empirical results showcase the effectiveness of ItD on two induction benchmarks, achieving relative performance improvement of 36% and 10% compared with previous state-of-the-art, respectively. Our ablation study verifies the effectiveness of two key modules of ItD. We also verify the effectiveness of ItD across different LLMs and deductors. The data and code of this paper can be found at https://anonymous.4open.science/r/ItD-E844.

Title: Privacy-Preserving Diffusion Model Using Homomorphic Encryption

• Authors: Yaojian Chen, Qiben Yan
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05794
• Pdf URL: https://arxiv.org/pdf/2403.05794
• Copy Paste: [[2403.05794]] Privacy-Preserving Diffusion Model Using Homomorphic Encryption(https://arxiv.org/abs/2403.05794)
• Keywords: privacy, protect, diffusion, generative
• Abstract: In this paper, we introduce a privacy-preserving stable diffusion framework leveraging homomorphic encryption, called HE-Diffusion, which primarily focuses on protecting the denoising phase of the diffusion process. HE-Diffusion is a tailored encryption framework specifically designed to align with the unique architecture of stable diffusion, ensuring both privacy and functionality. To address the inherent computational challenges, we propose a novel min-distortion method that enables efficient partial image encryption, significantly reducing the overhead without compromising the model's output quality. Furthermore, we adopt a sparse tensor representation to expedite computational operations, enhancing the overall efficiency of the privacy-preserving diffusion process. We successfully implement HE-based privacy-preserving stable diffusion inference. The experimental results show that HE-Diffusion achieves 500 times speedup compared with the baseline method, and reduces time cost of the homomorphically encrypted inference to the minute level. Both the performance and accuracy of the HE-Diffusion are on par with the plaintext counterpart. Our approach marks a significant step towards integrating advanced cryptographic techniques with state-of-the-art generative models, paving the way for privacy-preserving and efficient image generation in critical applications.

Title: ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes

• Authors: Zhichao Yang, Avijit Mitra, Sunjae Kwon, Hong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05795
• Pdf URL: https://arxiv.org/pdf/2403.05795
• Copy Paste: [[2403.05795]] ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes(https://arxiv.org/abs/2403.05795)
• Keywords: extraction, generative
• Abstract: The advancement of natural language processing (NLP) systems in healthcare hinges on language model ability to interpret the intricate information contained within clinical notes. This process often requires integrating information from various time points in a patient's medical history. However, most earlier clinical language models were pretrained with a context length limited to roughly one clinical document. In this study, We introduce ClinicalMamba, a specialized version of the Mamba language model, pretrained on a vast corpus of longitudinal clinical notes to address the unique linguistic characteristics and information processing needs of the medical domain. ClinicalMamba, with 130 million and 2.8 billion parameters, demonstrates a superior performance in modeling clinical language across extended text lengths compared to Mamba and clinical Llama. With few-shot learning, ClinicalMamba achieves notable benchmarks in speed and accuracy, outperforming existing clinical language models and general domain large models like GPT-4 in longitudinal clinical notes information extraction tasks.

Title: $\textbf{S}^2$IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting

• Authors: Zijie Pan, Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, Dongjin Song
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05798
• Pdf URL: https://arxiv.org/pdf/2403.05798
• Copy Paste: [[2403.05798]] $\textbf{S}^2$IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting(https://arxiv.org/abs/2403.05798)
• Keywords: large language model
• Abstract: Recently, there has been a growing interest in leveraging pre-trained large language models (LLMs) for various time series applications. However, the semantic space of LLMs, established through the pre-training, is still underexplored and may help yield more distinctive and informative representations to facilitate time series forecasting. To this end, we propose Semantic Space Informed Prompt learning with LLM ($S^2$IP-LLM) to align the pre-trained semantic space with time series embeddings space and perform time series forecasting based on learned prompts from the joint space. We first design a tokenization module tailored for cross-modality alignment, which explicitly concatenates patches of decomposed time series components to create embeddings that effectively encode the temporal dynamics. Next, we leverage the pre-trained word token embeddings to derive semantic anchors and align selected anchors with time series embeddings by maximizing the cosine similarity in the joint space. This way, $S^2$IP-LLM can retrieve relevant semantic anchors as prompts to provide strong indicators (context) for time series that exhibit different temporal dynamics. With thorough empirical studies on multiple benchmark datasets, we demonstrate that the proposed $S^2$IP-LLM can achieve superior forecasting performance over state-of-the-art baselines. Furthermore, our ablation studies and visualizations verify the necessity of prompt learning informed by semantic space.

Title: A self-supervised CNN for image watermark removal

• Authors: Chunwei Tian, Menghua Zheng, Tiancai Jiao, Wangmeng Zuo, Yanning Zhang, Chia-Wen Lin
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.05807
• Pdf URL: https://arxiv.org/pdf/2403.05807
• Copy Paste: [[2403.05807]] A self-supervised CNN for image watermark removal(https://arxiv.org/abs/2403.05807)
• Keywords: robust, watermark
• Abstract: Popular convolutional neural networks mainly use paired images in a supervised way for image watermark removal. However, watermarked images do not have reference images in the real world, which results in poor robustness of image watermark removal techniques. In this paper, we propose a self-supervised convolutional neural network (CNN) in image watermark removal (SWCNN). SWCNN uses a self-supervised way to construct reference watermarked images rather than given paired training samples, according to watermark distribution. A heterogeneous U-Net architecture is used to extract more complementary structural information via simple components for image watermark removal. Taking into account texture information, a mixed loss is exploited to improve visual effects of image watermark removal. Besides, a watermark dataset is conducted. Experimental results show that the proposed SWCNN is superior to popular CNNs in image watermark removal.

Title: Adaptive Multi-modal Fusion of Spatially Variant Kernel Refinement with Diffusion Model for Blind Image Super-Resolution

• Authors: Junxiong Lin, Yan Wang, Zeng Tao, Boyang Wang, Qing Zhao, Haorang Wang, Xuan Tong, Xinji Mai, Yuxuan Lin, Wei Song, Jiawen Yu, Shaoqi Yan, Wenqiang Zhang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.05808
• Pdf URL: https://arxiv.org/pdf/2403.05808
• Copy Paste: [[2403.05808]] Adaptive Multi-modal Fusion of Spatially Variant Kernel Refinement with Diffusion Model for Blind Image Super-Resolution(https://arxiv.org/abs/2403.05808)
• Keywords: diffusion
• Abstract: Pre-trained diffusion models utilized for image generation encapsulate a substantial reservoir of a priori knowledge pertaining to intricate textures. Harnessing the potential of leveraging this a priori knowledge in the context of image super-resolution presents a compelling avenue. Nonetheless, prevailing diffusion-based methodologies presently overlook the constraints imposed by degradation information on the diffusion process. Furthermore, these methods fail to consider the spatial variability inherent in the estimated blur kernel, stemming from factors such as motion jitter and out-of-focus elements in open-environment scenarios. This oversight results in a notable deviation of the image super-resolution effect from fundamental realities. To address these concerns, we introduce a framework known as Adaptive Multi-modal Fusion of \textbf{S}patially Variant Kernel Refinement with Diffusion Model for Blind Image \textbf{S}uper-\textbf{R}esolution (SSR). Within the SSR framework, we propose a Spatially Variant Kernel Refinement (SVKR) module. SVKR estimates a Depth-Informed Kernel, which takes the depth information into account and is spatially variant. Additionally, SVKR enhance the accuracy of depth information acquired from LR images, allowing for mutual enhancement between the depth map and blur kernel estimates. Finally, we introduce the Adaptive Multi-Modal Fusion (AMF) module to align the information from three modalities: low-resolution images, depth maps, and blur kernels. This alignment can constrain the diffusion model to generate more authentic SR results. Quantitative and qualitative experiments affirm the superiority of our approach, while ablation experiments corroborate the effectiveness of the modules we have proposed.

Title: Algorithmic progress in language models

• Authors: Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, Jaime Sevilla
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05812
• Pdf URL: https://arxiv.org/pdf/2403.05812
• Copy Paste: [[2403.05812]] Algorithmic progress in language models(https://arxiv.org/abs/2403.05812)
• Keywords: transformer
• Abstract: We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms.

Title: MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphs

• Authors: Yerin Hwang, Yongil Kim, Yunah Jang, Jeesoo Bang, Hyunkyung Bae, Kyomin Jung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05814
• Pdf URL: https://arxiv.org/pdf/2403.05814
• Copy Paste: [[2403.05814]] MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphs(https://arxiv.org/abs/2403.05814)
• Keywords: large language model
• Abstract: Despite advancements in on-topic dialogue systems, effectively managing topic shifts within dialogues remains a persistent challenge, largely attributed to the limited availability of training datasets. To address this issue, we propose Multi-Passage to Dialogue (MP2D), a data generation framework that automatically creates conversational question-answering datasets with natural topic transitions. By leveraging the relationships between entities in a knowledge graph, MP2D maps the flow of topics within a dialogue, effectively mirroring the dynamics of human conversation. It retrieves relevant passages corresponding to the topics and transforms them into dialogues through the passage-to-dialogue method. Through quantitative and qualitative experiments, we demonstrate MP2D's efficacy in generating dialogue with natural topic shifts. Furthermore, this study introduces a novel benchmark for topic shift dialogues, TS-WikiDialog. Utilizing the dataset, we demonstrate that even Large Language Models (LLMs) struggle to handle topic shifts in dialogue effectively, and we showcase the performance improvements of models trained on datasets generated by MP2D across diverse topic shift dialogue tasks.

Title: SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detection

• Authors: Gang Zhang, Junnan Chen, Guohuan Gao, Jianmin Li, Si Liu, Xiaolin Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05817
• Pdf URL: https://arxiv.org/pdf/2403.05817
• Copy Paste: [[2403.05817]] SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detection(https://arxiv.org/abs/2403.05817)
• Keywords: diffusion
• Abstract: LiDAR-based 3D object detection plays an essential role in autonomous driving. Existing high-performing 3D object detectors usually build dense feature maps in the backbone network and prediction head. However, the computational costs introduced by the dense feature maps grow quadratically as the perception range increases, making these models hard to scale up to long-range detection. Some recent works have attempted to construct fully sparse detectors to solve this issue; nevertheless, the resulting models either rely on a complex multi-stage pipeline or exhibit inferior performance. In this work, we propose SAFDNet, a straightforward yet highly effective architecture, tailored for fully sparse 3D object detection. In SAFDNet, an adaptive feature diffusion strategy is designed to address the center feature missing problem. We conducted extensive experiments on Waymo Open, nuScenes, and Argoverse2 datasets. SAFDNet performed slightly better than the previous SOTA on the first two datasets but much better on the last dataset, which features long-range detection, verifying the efficacy of SAFDNet in scenarios where long-range detection is required. Notably, on Argoverse2, SAFDNet surpassed the previous best hybrid detector HEDNet by 2.6% mAP while being 2.1x faster, and yielded 2.1% mAP gains over the previous best sparse detector FSDv2 while being 1.3x faster. The code will be available at https://github.com/zhanggang001/HEDNet.

Title: PR-NET: Leveraging Pathway Refined Network Structures for Prostate Cancer Patient Condition Prediction

• Authors: R. Li, J. Liu, X.L. Deng, X. Liu, J.C. Guo, W.Y. Wu, L. Yang
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2403.05818
• Pdf URL: https://arxiv.org/pdf/2403.05818
• Copy Paste: [[2403.05818]] PR-NET: Leveraging Pathway Refined Network Structures for Prostate Cancer Patient Condition Prediction(https://arxiv.org/abs/2403.05818)
• Keywords: robust, interpretability
• Abstract: Motivation: The diagnosis and monitoring of Castrate Resistant Prostate Cancer (CRPC) are crucial for cancer patients, but the current models (such as P-NET) have limitations in terms of parameter count, generalization, and cost. Results: To address the above issues, we develop a more accurate and efficient Prostate Cancer patient condition prediction model, named PR-NET. By compressing and optimizing the network structure of P-NET, the model complexity is reduced while maintaining high accuracy and interpretability. The PR-NET demonstrated superior performance in predicting prostate cancer patient outcomes, outshining P-NET and six other traditional models with a significant margin. In our rigorous evaluation, PR-NET not only achieved impressive average AUC and Recall scores of 0.94 and 0.83, respectively, on known data but also maintained robust generalizability on five unknown datasets with a higher average AUC of 0.73 and Recall of 0.72, compared to P-NET's 0.68 and 0.5. PR-NET's efficiency was evidenced by its shorter average training and inference times, and its gene-level analysis revealed 46 key genes, demonstrating its enhanced predictive power and efficiency in identifying critical biomarkers for prostate cancer. Future research can further expand its application domains and optimize the model's performance and reliability.

Title: Optimizing LLM Queries in Relational Workloads

• Authors: Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E. Gonzalez, Ion Stoica, Matei Zaharia
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2403.05821
• Pdf URL: https://arxiv.org/pdf/2403.05821
• Copy Paste: [[2403.05821]] Optimizing LLM Queries in Relational Workloads(https://arxiv.org/abs/2403.05821)
• Keywords: extraction, large language model
• Abstract: Analytical database providers (e.g., Redshift, Databricks, BigQuery) have rapidly added support for invoking Large Language Models (LLMs) through native user-defined functions (UDFs) to help users perform natural language tasks, such as classification, entity extraction, and translation, inside analytical workloads. For instance, an analyst might want to extract customer sentiments on millions of product reviews. However, LLM inference is highly expensive in both computational and economic terms: for example, an NVIDIA L4 GPU running Llama2-7B can only process 6 KB of text per second. In this paper, we explore how to optimize LLM inference for analytical workloads that invoke LLMs within relational queries. We show that relational queries present novel opportunities for accelerating LLM inference, including reordering rows to maximize key-value (KV) cache reuse within the LLM inference engine, reordering columns within a row to further increase cache reuse, and deduplicating redundant inference requests. We implement these optimizations in Apache Spark, with vLLM as the model serving backend and achieve up to 4.4x improvement in end-to-end latency on a benchmark of diverse LLM-based queries on real datasets. To the best of our knowledge, this is the first work to explicitly address the problem of optimizing LLM invocations within SQL queries.

Title: TrafficGPT: Breaking the Token Barrier for Efficient Long Traffic Analysis and Generation

• Authors: Jian Qu, Xiaobo Ma, Jianfeng Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05822
• Pdf URL: https://arxiv.org/pdf/2403.05822
• Copy Paste: [[2403.05822]] TrafficGPT: Breaking the Token Barrier for Efficient Long Traffic Analysis and Generation(https://arxiv.org/abs/2403.05822)
• Keywords: security, robust, generative
• Abstract: Over the years, network traffic analysis and generation have advanced significantly. From traditional statistical methods, the field has progressed to sophisticated deep learning techniques. This progress has improved the ability to detect complex patterns and security threats, as well as to test and optimize network performance. However, obstacles persist, such as the dependence on labeled data for analysis and the difficulty of generating traffic samples that follow realistic patterns. Pre-trained deep neural networks have emerged as powerful tools to resolve these issues, offering improved performance by learning robust data representations from large unlabeled datasets. Despite their benefits, existing pre-trained models face challenges like token length limitation, which restricts their usefulness in comprehensive traffic analysis and realistic traffic generation. To address these challenges, we introduce TrafficGPT, a deep learning model that can tackle complex challenges related to long flow classification and generation tasks. This model uses generative pre-training with the linear attention mechanism, which allows for a substantially increased capacity of up to 12,032 tokens from the previous limit of only 512 tokens. TrafficGPT demonstrates superior performance in classification tasks, reaching state-of-the-art levels. In generation tasks, it closely resembles real traffic flows, with low JS divergence and an F1 score close to 0.5 (representing a random guess) in discriminating generated data. These advancements hold promise for future applications in both traffic flow classification and generation tasks.

Title: Long-term Frame-Event Visual Tracking: Benchmark Dataset and Baseline

• Authors: Xiao Wang, Ju Huang, Shiao Wang, Chuanming Tang, Bo Jiang, Yonghong Tian, Jin Tang, Bin Luo
• Subjects: cs.CV, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2403.05839
• Pdf URL: https://arxiv.org/pdf/2403.05839
• Copy Paste: [[2403.05839]] Long-term Frame-Event Visual Tracking: Benchmark Dataset and Baseline(https://arxiv.org/abs/2403.05839)
• Keywords: transformer
• Abstract: Current event-/frame-event based trackers undergo evaluation on short-term tracking datasets, however, the tracking of real-world scenarios involves long-term tracking, and the performance of existing tracking algorithms in these scenarios remains unclear. In this paper, we first propose a new long-term and large-scale frame-event single object tracking dataset, termed FELT. It contains 742 videos and 1,594,474 RGB frames and event stream pairs and has become the largest frame-event tracking dataset to date. We re-train and evaluate 15 baseline trackers on our dataset for future works to compare. More importantly, we find that the RGB frames and event streams are naturally incomplete due to the influence of challenging factors and spatially sparse event flow. In response to this, we propose a novel associative memory Transformer network as a unified backbone by introducing modern Hopfield layers into multi-head self-attention blocks to fuse both RGB and event data. Extensive experiments on both FELT and RGB-T tracking dataset LasHeR fully validated the effectiveness of our model. The dataset and source code can be found at \url{https://github.com/Event-AHU/FELT_SOT_Benchmark}.

Title: Hufu: A Modality-Agnositc Watermarking System for Pre-Trained Transformers via Permutation Equivariance

• Authors: Hengyuan Xu, Liyao Xiang, Xingjun Ma, Borui Yang, Baochun Li
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05842
• Pdf URL: https://arxiv.org/pdf/2403.05842
• Copy Paste: [[2403.05842]] Hufu: A Modality-Agnositc Watermarking System for Pre-Trained Transformers via Permutation Equivariance(https://arxiv.org/abs/2403.05842)
• Keywords: protect, robust, extraction, watermark, transformer
• Abstract: With the blossom of deep learning models and services, it has become an imperative concern to safeguard the valuable model parameters from being stolen. Watermarking is considered an important tool for ownership verification. However, current watermarking schemes are customized for different models and tasks, hard to be integrated as an integrated intellectual protection service. We propose Hufu, a modality-agnostic watermarking system for pre-trained Transformer-based models, relying on the permutation equivariance property of Transformers. Hufu embeds watermark by fine-tuning the pre-trained model on a set of data samples specifically permuted, and the embedded model essentially contains two sets of weights -- one for normal use and the other for watermark extraction which is triggered on permuted inputs. The permutation equivariance ensures minimal interference between these two sets of model weights and thus high fidelity on downstream tasks. Since our method only depends on the model itself, it is naturally modality-agnostic, task-independent, and trigger-sample-free. Extensive experiments on the state-of-the-art vision Transformers, BERT, and GPT2 have demonstrated Hufu's superiority in meeting watermarking requirements including effectiveness, efficiency, fidelity, and robustness, showing its great potential to be deployed as a uniform ownership verification service for various Transformers.

Title: Reverse That Number! Decoding Order Matters in Arithmetic Learning

• Authors: Daniel Zhang-Li, Nianyi Lin, Jifan Yu, Zheyuan Zhang, Zijun Yao, Xiaokang Zhang, Lei Hou, Jing Zhang, Juanzi Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05845
• Pdf URL: https://arxiv.org/pdf/2403.05845
• Copy Paste: [[2403.05845]] Reverse That Number! Decoding Order Matters in Arithmetic Learning(https://arxiv.org/abs/2403.05845)
• Keywords: large language model
• Abstract: Recent advancements in pretraining have demonstrated that modern Large Language Models (LLMs) possess the capability to effectively learn arithmetic operations. However, despite acknowledging the significance of digit order in arithmetic computation, current methodologies predominantly rely on sequential, step-by-step approaches for teaching LLMs arithmetic, resulting in a conclusion where obtaining better performance involves fine-grained step-by-step. Diverging from this conventional path, our work introduces a novel strategy that not only reevaluates the digit order by prioritizing output from the least significant digit but also incorporates a step-by-step methodology to substantially reduce complexity. We have developed and applied this method in a comprehensive set of experiments. Compared to the previous state-of-the-art (SOTA) method, our findings reveal an overall improvement of in accuracy while requiring only a third of the tokens typically used during training. For the purpose of facilitating replication and further research, we have made our code and dataset publicly available at \url{https://anonymous.4open.science/r/RAIT-9FB7/}.

Title: Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines

• Authors: Michael Toker, Hadas Orgad, Mor Ventura, Dana Arad, Yonatan Belinkov
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05846
• Pdf URL: https://arxiv.org/pdf/2403.05846
• Copy Paste: [[2403.05846]] Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines(https://arxiv.org/abs/2403.05846)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models (T2I) use a latent representation of a text prompt to guide the image generation process. However, the process by which the encoder produces the text representation is unknown. We propose the Diffusion Lens, a method for analyzing the text encoder of T2I models by generating images from its intermediate representations. Using the Diffusion Lens, we perform an extensive analysis of two recent T2I models. Exploring compound prompts, we find that complex scenes describing multiple objects are composed progressively and more slowly compared to simple scenes; Exploring knowledge retrieval, we find that representation of uncommon concepts requires further computation compared to common concepts, and that knowledge retrieval is gradual across layers. Overall, our findings provide valuable insights into the text encoder component in T2I pipelines.

Title: MirrorAttack: Backdoor Attack on 3D Point Cloud with a Distorting Mirror

• Authors: Yuhao Bian, Shengjing Tian, Xiuping Liu
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05847
• Pdf URL: https://arxiv.org/pdf/2403.05847
• Copy Paste: [[2403.05847]] MirrorAttack: Backdoor Attack on 3D Point Cloud with a Distorting Mirror(https://arxiv.org/abs/2403.05847)
• Keywords: security, defense, attack, robust, steal
• Abstract: The widespread deployment of Deep Neural Networks (DNNs) for 3D point cloud processing starkly contrasts with their susceptibility to security breaches, notably backdoor attacks. These attacks hijack DNNs during training, embedding triggers in the data that, once activated, cause the network to make predetermined errors while maintaining normal performance on unaltered data. This vulnerability poses significant risks, especially given the insufficient research on robust defense mechanisms for 3D point cloud networks against such sophisticated threats. Existing attacks either struggle to resist basic point cloud pre-processing methods, or rely on delicate manual design. Exploring simple, effective, imperceptible, and difficult-to-defend triggers in 3D point clouds is still challenging.To address these challenges, we introduce MirrorAttack, a novel effective 3D backdoor attack method, which implants the trigger by simply reconstructing a clean point cloud with an auto-encoder. The data-driven nature of the MirrorAttack obviates the need for complex manual design. Minimizing the reconstruction loss automatically improves imperceptibility. Simultaneously, the reconstruction network endows the trigger with pronounced nonlinearity and sample specificity, rendering traditional preprocessing techniques ineffective in eliminating it. A trigger smoothing module based on spherical harmonic transformation is also attached to regulate the intensity of the attack.Both quantitive and qualitative results verify the effectiveness of our method. We achieve state-of-the-art ASR on different types of victim models with the intervention of defensive techniques. Moreover, the minimal perturbation introduced by our trigger, as assessed by various metrics, attests to the method's stealth, ensuring its imperceptibility.

Title: tLaSDI: Thermodynamics-informed latent space dynamics identification

• Authors: Jun Sur Richard Park, Siu Wun Cheung, Youngsoo Choi, Yeonjong Shin
• Subjects: cs.LG, math.DS
• Abstract URL: https://arxiv.org/abs/2403.05848
• Pdf URL: https://arxiv.org/pdf/2403.05848
• Copy Paste: [[2403.05848]] tLaSDI: Thermodynamics-informed latent space dynamics identification(https://arxiv.org/abs/2403.05848)
• Keywords: robust
• Abstract: We propose a data-driven latent space dynamics identification method (tLaSDI) that embeds the first and second principles of thermodynamics. The latent variables are learned through an autoencoder as a nonlinear dimension reduction model. The dynamics of the latent variables are constructed by a neural network-based model that preserves certain structures to respect the thermodynamic laws through the GENERIC formalism. An abstract error estimate of the approximation is established, which provides a new loss formulation involving the Jacobian computation of autoencoder. Both the autoencoder and the latent dynamics are trained to minimize the new loss. Numerical examples are presented to demonstrate the performance of tLaSDI, which exhibits robust generalization ability, even in extrapolation. In addition, an intriguing correlation is empirically observed between the entropy production rates in the latent space and the behaviors of the full-state solution.

Title: SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking

• Authors: Hanzheng Wang, Wei Li, Xiang-Gen Xia, Qian Du, Jing Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05852
• Pdf URL: https://arxiv.org/pdf/2403.05852
• Copy Paste: [[2403.05852]] SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking(https://arxiv.org/abs/2403.05852)
• Keywords: robust, extraction
• Abstract: Hyperspectral video (HSV) offers valuable spatial, spectral, and temporal information simultaneously, making it highly suitable for handling challenges such as background clutter and visual similarity in object tracking. However, existing methods primarily focus on band regrouping and rely on RGB trackers for feature extraction, resulting in limited exploration of spectral information and difficulties in achieving complementary representations of object features. In this paper, a spatial-spectral fusion network with spectral angle awareness (SST-Net) is proposed for hyperspectral (HS) object tracking. Firstly, to address the issue of insufficient spectral feature extraction in existing networks, a spatial-spectral feature backbone ($S^2$FB) is designed. With the spatial and spectral extraction branch, a joint representation of texture and spectrum is obtained. Secondly, a spectral attention fusion module (SAFM) is presented to capture the intra- and inter-modality correlation to obtain the fused features from the HS and RGB modalities. It can incorporate the visual information into the HS spectral context to form a robust representation. Thirdly, to ensure a more accurate response of the tracker to the object position, a spectral angle awareness module (SAAM) investigates the region-level spectral similarity between the template and search images during the prediction stage. Furthermore, we develop a novel spectral angle awareness loss (SAAL) to offer guidance for the SAAM based on similar regions. Finally, to obtain the robust tracking results, a weighted prediction method is considered to combine the HS and RGB predicted motions of objects to leverage the strengths of each modality. Extensive experiments on the HOTC dataset demonstrate the effectiveness of the proposed SSF-Net, compared with state-of-the-art trackers.

Title: LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Content

• Authors: Qihao Zhao, Yalun Dai, Hao Li, Wei Hu, Fan Zhang, Jun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05854
• Pdf URL: https://arxiv.org/pdf/2403.05854
• Copy Paste: [[2403.05854]] LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Content(https://arxiv.org/abs/2403.05854)
• Keywords: generative, large language model
• Abstract: Long-tail recognition is challenging because it requires the model to learn good representations from tail categories and address imbalances across all categories. In this paper, we propose a novel generative and fine-tuning framework, LTGC, to handle long-tail recognition via leveraging generated content. Firstly, inspired by the rich implicit knowledge in large-scale models (e.g., large language models, LLMs), LTGC leverages the power of these models to parse and reason over the original tail data to produce diverse tail-class content. We then propose several novel designs for LTGC to ensure the quality of the generated data and to efficiently fine-tune the model using both the generated and original data. The visualization demonstrates the effectiveness of the generation module in LTGC, which produces accurate and diverse tail data. Additionally, the experimental results demonstrate that our LTGC outperforms existing state-of-the-art methods on popular long-tailed benchmarks.

Title: PAPER-HILT: Personalized and Adaptive Privacy-Aware Early-Exit for Reinforcement Learning in Human-in-the-Loop Systems

• Authors: Mojtaba Taherisadr, Salma Elmalaki
• Subjects: cs.LG, cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.05864
• Pdf URL: https://arxiv.org/pdf/2403.05864
• Copy Paste: [[2403.05864]] PAPER-HILT: Personalized and Adaptive Privacy-Aware Early-Exit for Reinforcement Learning in Human-in-the-Loop Systems(https://arxiv.org/abs/2403.05864)
• Keywords: privacy, protect
• Abstract: Reinforcement Learning (RL) has increasingly become a preferred method over traditional rule-based systems in diverse human-in-the-loop (HITL) applications due to its adaptability to the dynamic nature of human interactions. However, integrating RL in such settings raises significant privacy concerns, as it might inadvertently expose sensitive user information. Addressing this, our paper focuses on developing PAPER-HILT, an innovative, adaptive RL strategy through exploiting an early-exit approach designed explicitly for privacy preservation in HITL environments. This approach dynamically adjusts the tradeoff between privacy protection and system utility, tailoring its operation to individual behavioral patterns and preferences. We mainly highlight the challenge of dealing with the variable and evolving nature of human behavior, which renders static privacy models ineffective. PAPER-HILT's effectiveness is evaluated through its application in two distinct contexts: Smart Home environments and Virtual Reality (VR) Smart Classrooms. The empirical results demonstrate PAPER-HILT's capability to provide a personalized equilibrium between user privacy and application utility, adapting effectively to individual user needs and preferences. On average for both experiments, utility (performance) drops by 24%, and privacy (state prediction) improves by 31%.

Title: SPAFormer: Sequential 3D Part Assembly with Transformers

• Authors: Boshen Xu, Sipeng Zheng, Qin Jin
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.05874
• Pdf URL: https://arxiv.org/pdf/2403.05874
• Copy Paste: [[2403.05874]] SPAFormer: Sequential 3D Part Assembly with Transformers(https://arxiv.org/abs/2403.05874)
• Keywords: transformer
• Abstract: We introduce SPAFormer, an innovative model designed to overcome the combinatorial explosion challenge in the 3D Part Assembly (3D-PA) task. This task requires accurate prediction of each part's pose and shape in sequential steps, and as the number of parts increases, the possible assembly combinations increase exponentially, leading to a combinatorial explosion that severely hinders the efficacy of 3D-PA. SPAFormer addresses this problem by leveraging weak constraints from assembly sequences, effectively reducing the solution space's complexity. Since assembly part sequences convey construction rules similar to sentences being structured through words, our model explores both parallel and autoregressive generation. It further enhances assembly through knowledge enhancement strategies that utilize the attributes of parts and their sequence information, enabling it to capture the inherent assembly pattern and relationships among sequentially ordered parts. We also construct a more challenging benchmark named PartNet-Assembly covering 21 varied categories to more comprehensively validate the effectiveness of SPAFormer. Extensive experiments demonstrate the superior generalization capabilities of SPAFormer, particularly with multi-tasking and in scenarios requiring long-horizon assembly. Codes and model weights will be released at \url{https://github.com/xuboshen/SPAFormer}.

Title: KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques

• Authors: Rui Yang, Haoran Liu, Qingcheng Zeng, Yu He Ke, Wanxin Li, Lechao Cheng, Qingyu Chen, James Caverlee, Yutaka Matsuo, Irene Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05881
• Pdf URL: https://arxiv.org/pdf/2403.05881
• Copy Paste: [[2403.05881]] KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques(https://arxiv.org/abs/2403.05881)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have significantly advanced healthcare innovation on generation capabilities. However, their application in real clinical settings is challenging due to potential deviations from medical facts and inherent biases. In this work, we develop an augmented LLM framework, KG-Rank, which leverages a medical knowledge graph (KG) with ranking and re-ranking techniques, aiming to improve free-text question-answering (QA) in the medical domain. Specifically, upon receiving a question, we initially retrieve triplets from a medical KG to gather factual information. Subsequently, we innovatively apply ranking methods to refine the ordering of these triplets, aiming to yield more precise answers. To the best of our knowledge, KG-Rank is the first application of ranking models combined with KG in medical QA specifically for generating long answers. Evaluation of four selected medical QA datasets shows that KG-Rank achieves an improvement of over 18% in the ROUGE-L score. Moreover, we extend KG-Rank to open domains, where it realizes a 14% improvement in ROUGE-L, showing the effectiveness and potential of KG-Rank.

Title: Towards Efficient Replay in Federated Incremental Learning

• Authors: Yichen Li, Qunwei Li, Haozhao Wang, Ruixuan Li, Wenliang Zhong, Guannan Zhang
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.05890
• Pdf URL: https://arxiv.org/pdf/2403.05890
• Copy Paste: [[2403.05890]] Towards Efficient Replay in Federated Incremental Learning(https://arxiv.org/abs/2403.05890)
• Keywords: federate
• Abstract: In Federated Learning (FL), the data in each client is typically assumed fixed or static. However, data often comes in an incremental manner in real-world applications, where the data domain may increase dynamically. In this work, we study catastrophic forgetting with data heterogeneity in Federated Incremental Learning (FIL) scenarios where edge clients may lack enough storage space to retain full data. We propose to employ a simple, generic framework for FIL named Re-Fed, which can coordinate each client to cache important samples for replay. More specifically, when a new task arrives, each client first caches selected previous samples based on their global and local importance. Then, the client trains the local model with both the cached samples and the samples from the new task. Theoretically, we analyze the ability of Re-Fed to discover important samples for replay thus alleviating the catastrophic forgetting problem. Moreover, we empirically show that Re-Fed achieves competitive performance compared to state-of-the-art methods.

Title: RealNet: A Feature Selection Network with Realistic Synthetic Anomaly for Anomaly Detection

• Authors: Ximiao Zhang, Min Xu, Xiuzhuang Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.05897
• Pdf URL: https://arxiv.org/pdf/2403.05897
• Copy Paste: [[2403.05897]] RealNet: A Feature Selection Network with Realistic Synthetic Anomaly for Anomaly Detection(https://arxiv.org/abs/2403.05897)
• Keywords: diffusion
• Abstract: Self-supervised feature reconstruction methods have shown promising advances in industrial image anomaly detection and localization. Despite this progress, these methods still face challenges in synthesizing realistic and diverse anomaly samples, as well as addressing the feature redundancy and pre-training bias of pre-trained feature. In this work, we introduce RealNet, a feature reconstruction network with realistic synthetic anomaly and adaptive feature selection. It is incorporated with three key innovations: First, we propose Strength-controllable Diffusion Anomaly Synthesis (SDAS), a diffusion process-based synthesis strategy capable of generating samples with varying anomaly strengths that mimic the distribution of real anomalous samples. Second, we develop Anomaly-aware Features Selection (AFS), a method for selecting representative and discriminative pre-trained feature subsets to improve anomaly detection performance while controlling computational costs. Third, we introduce Reconstruction Residuals Selection (RRS), a strategy that adaptively selects discriminative residuals for comprehensive identification of anomalous regions across multiple levels of granularity. We assess RealNet on four benchmark datasets, and our results demonstrate significant improvements in both Image AUROC and Pixel AUROC compared to the current state-o-the-art methods. The code, data, and models are available at https://github.com/cnulab/RealNet.

Title: SEMRes-DDPM: Residual Network Based Diffusion Modelling Applied to Imbalanced Data

• Authors: Ming Zheng, Yang Yang, Zhi-Hang Zhao, Shan-Chao Gan, Yang Chen, Si-Kai Ni, Yang Lu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05918
• Pdf URL: https://arxiv.org/pdf/2403.05918
• Copy Paste: [[2403.05918]] SEMRes-DDPM: Residual Network Based Diffusion Modelling Applied to Imbalanced Data(https://arxiv.org/abs/2403.05918)
• Keywords: diffusion, generative
• Abstract: In the field of data mining and machine learning, commonly used classification models cannot effectively learn in unbalanced data. In order to balance the data distribution before model training,oversamplingmethods are often used to generate data for a small number of classes to solve the problem of classifying unbalanced data. Most of the classical oversampling methods are based on theSMOTE technique, which only focuses on the local information of the data, and therefore the generated data may have the problem of not being realistic enough. In the current oversampling methods based on generative networks, the methods based on GANs can capture the true distribution of data, but there is the problem of pattern collapse and training instability in training; in the oversampling methods based on denoising diffusion probability models, the neural network of the inverse diffusion process using the U-Net is not applicable to tabular data, and although the MLP can be used to replace the U-Net, the problem exists due to the simplicity of the structure and the poor effect of removing noise. problem of poor noise removal. In order to overcome the above problems, we propose a novel oversampling method SEMRes-DDPM.In the SEMRes?DDPM backward diffusion process, a new neural network structure SEMST-ResNet is used, which is suitable for tabular data and has good noise removal effect, and it can generate tabular data with higher quality. Experiments show that the SEMResNet network removes noise better than MLP; SEMRes?DDPM generates data distributions that are closer to the real data distributions than TabDDPM with CWGAN-GP; on 20 real unbalanced tabular datasets with 9 classification models, SEMRes-DDPM improves the quality of the generated tabular data in terms of three evaluation metrics (F1, G-mean, AUC) with better classification performance than other SOTA oversampling methods.

Title: High Throughput Phenotyping of Physician Notes with Large Language and Hybrid NLP Models

• Authors: Syed I. Munzir, Daniel B. Hier, Michael D. Carrithers
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.05920
• Pdf URL: https://arxiv.org/pdf/2403.05920
• Copy Paste: [[2403.05920]] High Throughput Phenotyping of Physician Notes with Large Language and Hybrid NLP Models(https://arxiv.org/abs/2403.05920)
• Keywords: large language model
• Abstract: Deep phenotyping is the detailed description of patient signs and symptoms using concepts from an ontology. The deep phenotyping of the numerous physician notes in electronic health records requires high throughput methods. Over the past thirty years, progress toward making high throughput phenotyping feasible. In this study, we demonstrate that a large language model and a hybrid NLP model (combining word vectors with a machine learning classifier) can perform high throughput phenotyping on physician notes with high accuracy. Large language models will likely emerge as the preferred method for high throughput deep phenotyping of physician notes.

Title: Thread Detection and Response Generation using Transformers with Prompt Optimisation

• Authors: Kevin Joshua T, Arnav Agarwal, Shriya Sanjay, Yash Sarda, John Sahaya Rani Alex, Saurav Gupta, Sushant Kumar, Vishwanath Kamath
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05931
• Pdf URL: https://arxiv.org/pdf/2403.05931
• Copy Paste: [[2403.05931]] Thread Detection and Response Generation using Transformers with Prompt Optimisation(https://arxiv.org/abs/2403.05931)
• Keywords: transformer, large language model
• Abstract: Conversational systems are crucial for human-computer interaction, managing complex dialogues by identifying threads and prioritising responses. This is especially vital in multi-party conversations, where precise identification of threads and strategic response prioritisation ensure efficient dialogue management. To address these challenges an end-to-end model that identifies threads and prioritises their response generation based on the importance was developed, involving a systematic decomposition of the problem into discrete components - thread detection, prioritisation, and performance optimisation which was meticulously analysed and optimised. These refined components seamlessly integrate into a unified framework, in conversational systems. Llama2 7b is used due to its high level of generalisation but the system can be updated with any open source Large Language Model(LLM). The computational capabilities of the Llama2 model was augmented by using fine tuning methods and strategic prompting techniques to optimise the model's performance, reducing computational time and increasing the accuracy of the model. The model achieves up to 10x speed improvement, while generating more coherent results compared to existing models.

Title: General surgery vision transformer: A video pre-trained foundation model for general surgery

• Authors: Samuel Schmidgall, Ji Woong Kim, Jeffery Jopling, Axel Krieger
• Subjects: cs.CV, cs.LG, q-bio.TO
• Abstract URL: https://arxiv.org/abs/2403.05949
• Pdf URL: https://arxiv.org/pdf/2403.05949
• Copy Paste: [[2403.05949]] General surgery vision transformer: A video pre-trained foundation model for general surgery(https://arxiv.org/abs/2403.05949)
• Keywords: transformer
• Abstract: The absence of openly accessible data and specialized foundation models is a major barrier for computational research in surgery. Toward this, (i) we open-source the largest dataset of general surgery videos to-date, consisting of 680 hours of surgical videos, including data from robotic and laparoscopic techniques across 28 procedures; (ii) we propose a technique for video pre-training a general surgery vision transformer (GSViT) on surgical videos based on forward video prediction that can run in real-time for surgical applications, toward which we open-source the code and weights of GSViT; (iii) we also release code and weights for procedure-specific fine-tuned versions of GSViT across 10 procedures; (iv) we demonstrate the performance of GSViT on the Cholec80 phase annotation task, displaying improved performance over state-of-the-art single frame predictors.

Title: Robust Emotion Recognition in Context Debiasing

• Authors: Dingkang Yang, Kun Yang, Mingcheng Li, Shunli Wang, Shuaibing Wang, Lihua Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05963
• Pdf URL: https://arxiv.org/pdf/2403.05963
• Copy Paste: [[2403.05963]] Robust Emotion Recognition in Context Debiasing(https://arxiv.org/abs/2403.05963)
• Keywords: robust
• Abstract: Context-aware emotion recognition (CAER) has recently boosted the practical applications of affective computing techniques in unconstrained environments. Mainstream CAER methods invariably extract ensemble representations from diverse contexts and subject-centred characteristics to perceive the target person's emotional state. Despite advancements, the biggest challenge remains due to context bias interference. The harmful bias forces the models to rely on spurious correlations between background contexts and emotion labels in likelihood estimation, causing severe performance bottlenecks and confounding valuable context priors. In this paper, we propose a counterfactual emotion inference (CLEF) framework to address the above issue. Specifically, we first formulate a generalized causal graph to decouple the causal relationships among the variables in CAER. Following the causal graph, CLEF introduces a non-invasive context branch to capture the adverse direct effect caused by the context bias. During the inference, we eliminate the direct context effect from the total causal effect by comparing factual and counterfactual outcomes, resulting in bias mitigation and robust prediction. As a model-agnostic framework, CLEF can be readily integrated into existing methods, bringing consistent performance gains.

Title: Can Generative Models Improve Self-Supervised Representation Learning?

• Authors: Arash Afkanpour, Vahid Reza Khazaie, Sana Ayromlou, Fereshteh Forghani
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05966
• Pdf URL: https://arxiv.org/pdf/2403.05966
• Copy Paste: [[2403.05966]] Can Generative Models Improve Self-Supervised Representation Learning?(https://arxiv.org/abs/2403.05966)
• Keywords: robust, generative
• Abstract: The rapid advancement in self-supervised learning (SSL) has highlighted its potential to leverage unlabeled data for learning powerful visual representations. However, existing SSL approaches, particularly those employing different views of the same image, often rely on a limited set of predefined data augmentations. This constrains the diversity and quality of transformations, which leads to sub-optimal representations. In this paper, we introduce a novel framework that enriches the SSL paradigm by utilizing generative models to produce semantically consistent image augmentations. By directly conditioning generative models on a source image representation, our method enables the generation of diverse augmentations while maintaining the semantics of the source image, thus offering a richer set of data for self-supervised learning. Our experimental results demonstrate that our framework significantly enhances the quality of learned visual representations. This research demonstrates that incorporating generative models into the SSL workflow opens new avenues for exploring the potential of unlabeled visual data. This development paves the way for more robust and versatile representation learning techniques.

Title: Calibrating Large Language Models Using Their Generations Only

• Authors: Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, Seong Joon Oh
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.05973
• Pdf URL: https://arxiv.org/pdf/2403.05973
• Copy Paste: [[2403.05973]] Calibrating Large Language Models Using Their Generations Only(https://arxiv.org/abs/2403.05973)
• Keywords: large language model
• Abstract: As large language models (LLMs) are increasingly deployed in user-facing applications, building trust and maintaining safety by accurately quantifying a model's confidence in its prediction becomes even more important. However, finding effective ways to calibrate LLMs - especially when the only interface to the models is their generated text - remains a challenge. We propose APRICOT (auxiliary prediction of confidence targets): A method to set confidence targets and train an additional model that predicts an LLM's confidence based on its textual input and output alone. This approach has several advantages: It is conceptually simple, does not require access to the target model beyond its output, does not interfere with the language generation, and has a multitude of potential usages, for instance by verbalizing the predicted confidence or adjusting the given answer based on the confidence. We show how our approach performs competitively in terms of calibration error for white-box and black-box LLMs on closed-book question-answering to detect incorrect LLM answers.

Title: Measuring Bias in a Ranked List using Term-based Representations

• Authors: Amin Abolghasemi, Leif Azzopardi, Arian Askari, Maarten de Rijke, Suzan Verberne
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.05975
• Pdf URL: https://arxiv.org/pdf/2403.05975
• Copy Paste: [[2403.05975]] Measuring Bias in a Ranked List using Term-based Representations(https://arxiv.org/abs/2403.05975)
• Keywords: fair
• Abstract: In most recent studies, gender bias in document ranking is evaluated with the NFaiRR metric, which measures bias in a ranked list based on an aggregation over the unbiasedness scores of each ranked document. This perspective in measuring the bias of a ranked list has a key limitation: individual documents of a ranked list might be biased while the ranked list as a whole balances the groups' representations. To address this issue, we propose a novel metric called TExFAIR (term exposure-based fairness), which is based on two new extensions to a generic fairness evaluation framework, attention-weighted ranking fairness (AWRF). TExFAIR assesses fairness based on the term-based representation of groups in a ranked list: (i) an explicit definition of associating documents to groups based on probabilistic term-level associations, and (ii) a rank-biased discounting factor (RBDF) for counting non-representative documents towards the measurement of the fairness of a ranked list. We assess TExFAIR on the task of measuring gender bias in passage ranking, and study the relationship between TExFAIR and NFaiRR. Our experiments show that there is no strong correlation between TExFAIR and NFaiRR, which indicates that TExFAIR measures a different dimension of fairness than NFaiRR. With TExFAIR, we extend the AWRF framework to allow for the evaluation of fairness in settings with term-based representations of groups in documents in a ranked list.

Title: Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations

• Authors: Swapnaja Achintalwar, Adriana Alvarado Garcia, Ateret Anaby-Tavor, Ioana Baldini, Sara E. Berger, Bishwaranjan Bhattacharjee, Djallel Bouneffouf, Subhajit Chaudhury, Pin-Yu Chen, Lamogha Chiazor, Elizabeth M. Daly, Rogério Abreu de Paula, Pierre Dognin, Eitan Farchi, Soumya Ghosh, Michael Hind, Raya Horesh, George Kour, Ja Young Lee, Erik Miehling, Keerthiram Murugesan, Manish Nagireddy, Inkit Padhi, David Piorkowski, Ambrish Rawat, Orna Raz, Prasanna Sattigeri, Hendrik Strobelt, Sarathkrishna Swaminathan, Christoph Tillmann, Aashka Trivedi, Kush R. Varshney, Dennis Wei, Shalisha Witherspooon, Marcel Zalmanovici
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06009
• Pdf URL: https://arxiv.org/pdf/2403.06009
• Copy Paste: [[2403.06009]] Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations(https://arxiv.org/abs/2403.06009)
• Keywords: large language model
• Abstract: Large language models (LLMs) are susceptible to a variety of risks, from non-faithful output to biased and toxic generations. Due to several limiting factors surrounding LLMs (training cost, API access, data availability, etc.), it may not always be feasible to impose direct safety constraints on a deployed model. Therefore, an efficient and reliable alternative is required. To this end, we present our ongoing efforts to create and deploy a library of detectors: compact and easy-to-build classification models that provide labels for various harms. In addition to the detectors themselves, we discuss a wide range of uses for these detector models - from acting as guardrails to enabling effective AI governance. We also deep dive into inherent challenges in their development and discuss future work aimed at making the detectors more reliable and broadening their scope.

Title: Are Classification Robustness and Explanation Robustness Really Strongly Correlated? An Analysis Through Input Loss Landscape

• Authors: Tiejin Chen, Wenwang Huang, Linsey Pang, Dongsheng Luo, Hua Wei
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06013
• Pdf URL: https://arxiv.org/pdf/2403.06013
• Copy Paste: [[2403.06013]] Are Classification Robustness and Explanation Robustness Really Strongly Correlated? An Analysis Through Input Loss Landscape(https://arxiv.org/abs/2403.06013)
• Keywords: robust
• Abstract: This paper delves into the critical area of deep learning robustness, challenging the conventional belief that classification robustness and explanation robustness in image classification systems are inherently correlated. Through a novel evaluation approach leveraging clustering for efficient assessment of explanation robustness, we demonstrate that enhancing explanation robustness does not necessarily flatten the input loss landscape with respect to explanation loss - contrary to flattened loss landscapes indicating better classification robustness. To deeply investigate this contradiction, a groundbreaking training method designed to adjust the loss landscape with respect to explanation loss is proposed. Through the new training method, we uncover that although such adjustments can impact the robustness of explanations, they do not have an influence on the robustness of classification. These findings not only challenge the prevailing assumption of a strong correlation between the two forms of robustness but also pave new pathways for understanding relationship between loss landscape and explanation loss.

Title: Hard-label based Small Query Black-box Adversarial Attack

• Authors: Jeonghwan Park, Paul Miller, Niall McLaughlin
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06014
• Pdf URL: https://arxiv.org/pdf/2403.06014
• Copy Paste: [[2403.06014]] Hard-label based Small Query Black-box Adversarial Attack(https://arxiv.org/abs/2403.06014)
• Keywords: attack
• Abstract: We consider the hard label based black box adversarial attack setting which solely observes predicted classes from the target model. Most of the attack methods in this setting suffer from impractical number of queries required to achieve a successful attack. One approach to tackle this drawback is utilising the adversarial transferability between white box surrogate models and black box target model. However, the majority of the methods adopting this approach are soft label based to take the full advantage of zeroth order optimisation. Unlike mainstream methods, we propose a new practical setting of hard label based attack with an optimisation process guided by a pretrained surrogate model. Experiments show the proposed method significantly improves the query efficiency of the hard label based black-box attack across various target model architectures. We find the proposed method achieves approximately 5 times higher attack success rate compared to the benchmarks, especially at the small query budgets as 100 and 250.

Title: Addressing Shortcomings in Fair Graph Learning Datasets: Towards a New Benchmark

• Authors: Xiaowei Qian, Zhimeng Guo, Jialiang Li, Haitao Mao, Bingheng Li, Suhang Wang, Yao Ma
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.06017
• Pdf URL: https://arxiv.org/pdf/2403.06017
• Copy Paste: [[2403.06017]] Addressing Shortcomings in Fair Graph Learning Datasets: Towards a New Benchmark(https://arxiv.org/abs/2403.06017)
• Keywords: fair
• Abstract: Fair graph learning plays a pivotal role in numerous practical applications. Recently, many fair graph learning methods have been proposed; however, their evaluation often relies on poorly constructed semi-synthetic datasets or substandard real-world datasets. In such cases, even a basic Multilayer Perceptron (MLP) can outperform Graph Neural Networks (GNNs) in both utility and fairness. In this work, we illustrate that many datasets fail to provide meaningful information in the edges, which may challenge the necessity of using graph structures in these problems. To address these issues, we develop and introduce a collection of synthetic, semi-synthetic, and real-world datasets that fulfill a broad spectrum of requirements. These datasets are thoughtfully designed to include relevant graph structures and bias information crucial for the fair evaluation of models. The proposed synthetic and semi-synthetic datasets offer the flexibility to create data with controllable bias parameters, thereby enabling the generation of desired datasets with user-defined bias values with ease. Moreover, we conduct systematic evaluations of these proposed datasets and establish a unified evaluation approach for fair graph learning models. Our extensive experimental results with fair graph learning methods across our datasets demonstrate their effectiveness in benchmarking the performance of these methods. Our datasets and the code for reproducing our experiments are available at https://github.com/XweiQ/Benchmark-GraphFairness.

Title: Few-Shot Cross-Lingual Transfer for Prompting Large Language Models in Low-Resource Languages

• Authors: Christopher Toukmaji
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06018
• Pdf URL: https://arxiv.org/pdf/2403.06018
• Copy Paste: [[2403.06018]] Few-Shot Cross-Lingual Transfer for Prompting Large Language Models in Low-Resource Languages(https://arxiv.org/abs/2403.06018)
• Keywords: large language model
• Abstract: Large pre-trained language models (PLMs) are at the forefront of advances in Natural Language Processing. One widespread use case of PLMs is "prompting" - or in-context learning - where a user provides a description of a task and some completed examples of the task to a PLM as context before prompting the PLM to perform the task on a new example. Only the largest, most capable PLMs are able to perform in-context learning effectively, and these models are typically trained with a predominantly English corpus, leaving all other languages behind. The data limitations in most languages preclude the training of language-specific PLMs capable of prompting. Albeit the surge in work of prompting settings, it is still unclear how PLMs should be adapted cross-lingually specifically for prompting. We evaluate the possible methods to adapt LLaMa, a 7B parameter open-source PLM mainly trained in English, for prompting in low-resource languages, namely for Kinyarwanda, Hausa, and Luganda. We consider three methods: few-shot prompting (prompt), language-adaptive fine-tuning (LAFT), and neural machine translation (translate), and evaluate on abstractive summarization, multi-class topic classification, and named-entity recognition. Although LAFT carries the greatest compute cost and intuitively should lead to the best results, our experiments exhibit that LAFT is only occasionally the optimal choice for adapting PLMs for prompting. Rather, the translate and prompt settings are a compute-efficient and cost-effective method of few-shot prompting for the selected low-resource languages. We find that the results are task and language dependent but find that the prompting method is the best on average across all tasks and languages. Results show that the prompt setting performs better than both translating and LAFT with statistical significance for all shots when aggregated across all tasks and languages.

Title: Multi-conditioned Graph Diffusion for Neural Architecture Search

• Authors: Rohan Asthana, Joschua Conrad, Youssef Dawoud, Maurits Ortmanns, Vasileios Belagiannis
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06020
• Pdf URL: https://arxiv.org/pdf/2403.06020
• Copy Paste: [[2403.06020]] Multi-conditioned Graph Diffusion for Neural Architecture Search(https://arxiv.org/abs/2403.06020)
• Keywords: diffusion
• Abstract: Neural architecture search automates the design of neural network architectures usually by exploring a large and thus complex architecture search space. To advance the architecture search, we present a graph diffusion-based NAS approach that uses discrete conditional graph diffusion processes to generate high-performing neural network architectures. We then propose a multi-conditioned classifier-free guidance approach applied to graph diffusion networks to jointly impose constraints such as high accuracy and low hardware latency. Unlike the related work, our method is completely differentiable and requires only a single model training. In our evaluations, we show promising results on six standard benchmarks, yielding novel and unique architectures at a fast speed, i.e. less than 0.2 seconds per architecture. Furthermore, we demonstrate the generalisability and efficiency of our method through experiments on ImageNet dataset.

Title: CarbonNet: How Computer Vision Plays a Role in Climate Change? Application: Learning Geomechanics from Subsurface Geometry of CCS to Mitigate Global Warming

• Authors: Wei Chen, Yunan Li, Yuan Tian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06025
• Pdf URL: https://arxiv.org/pdf/2403.06025
• Copy Paste: [[2403.06025]] CarbonNet: How Computer Vision Plays a Role in Climate Change? Application: Learning Geomechanics from Subsurface Geometry of CCS to Mitigate Global Warming(https://arxiv.org/abs/2403.06025)
• Keywords: transformer
• Abstract: We introduce a new approach using computer vision to predict the land surface displacement from subsurface geometry images for Carbon Capture and Sequestration (CCS). CCS has been proved to be a key component for a carbon neutral society. However, scientists see there are challenges along the way including the high computational cost due to the large model scale and limitations to generalize a pre-trained model with complex physics. We tackle those challenges by training models directly from the subsurface geometry images. The goal is to understand the respons of land surface displacement due to carbon injection and utilize our trained models to inform decision making in CCS projects. We implement multiple models (CNN, ResNet, and ResNetUNet) for static mechanics problem, which is a image prediction problem. Next, we use the LSTM and transformer for transient mechanics scenario, which is a video prediction problem. It shows ResNetUNet outperforms the others thanks to its architecture in static mechanics problem, and LSTM shows comparable performance to transformer in transient problem. This report proceeds by outlining our dataset in detail followed by model descriptions in method section. Result and discussion state the key learning, observations, and conclusion with future work rounds out the paper.

Title: FairTargetSim: An Interactive Simulator for Understanding and Explaining the Fairness Effects of Target Variable Definition

• Authors: Dalia Gala, Milo Phillips-Brown, Naman Goel, Carinal Prunkl, Laura Alvarez Jubete, medb corcoran, Ray Eitel-Porter
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.06031
• Pdf URL: https://arxiv.org/pdf/2403.06031
• Copy Paste: [[2403.06031]] FairTargetSim: An Interactive Simulator for Understanding and Explaining the Fairness Effects of Target Variable Definition(https://arxiv.org/abs/2403.06031)
• Keywords: fair
• Abstract: Machine learning requires defining one's target variable for predictions or decisions, a process that can have profound implications on fairness: biases are often encoded in target variable definition itself, before any data collection or training. We present an interactive simulator, FairTargetSim (FTS), that illustrates how target variable definition impacts fairness. FTS is a valuable tool for algorithm developers, researchers, and non-technical stakeholders. FTS uses a case study of algorithmic hiring, using real-world data and user-defined target variables. FTS is open-source and available at: this http URL The video accompanying this paper is here: this http URL

Title: Target-constrained Bidirectional Planning for Generation of Target-oriented Proactive Dialogue

• Authors: Jian Wang, Dongding Lin, Wenjie Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06063
• Pdf URL: https://arxiv.org/pdf/2403.06063
• Copy Paste: [[2403.06063]] Target-constrained Bidirectional Planning for Generation of Target-oriented Proactive Dialogue(https://arxiv.org/abs/2403.06063)
• Keywords: transformer
• Abstract: Target-oriented proactive dialogue systems aim to lead conversations from a dialogue context toward a pre-determined target, such as making recommendations on designated items or introducing new specific topics. To this end, it is critical for such dialogue systems to plan reasonable actions to drive the conversation proactively, and meanwhile, to plan appropriate topics to move the conversation forward to the target topic smoothly. In this work, we mainly focus on effective dialogue planning for target-oriented dialogue generation. Inspired by decision-making theories in cognitive science, we propose a novel target-constrained bidirectional planning (TRIP) approach, which plans an appropriate dialogue path by looking ahead and looking back. By formulating the planning as a generation task, our TRIP bidirectionally generates a dialogue path consisting of a sequence of pairs using two Transformer decoders. They are expected to supervise each other and converge on consistent actions and topics by minimizing the decision gap and contrastive generation of targets. Moreover, we propose a target-constrained decoding algorithm with a bidirectional agreement to better control the planning process. Subsequently, we adopt the planned dialogue paths to guide dialogue generation in a pipeline manner, where we explore two variants: prompt-based generation and plan-controlled generation. Extensive experiments are conducted on two challenging dialogue datasets, which are re-purposed for exploring target-oriented dialogue. Our automatic and human evaluations demonstrate that the proposed methods significantly outperform various baseline models.

Title: Federated Learning: Attacks, Defenses, Opportunities, and Challenges

• Authors: Ghazaleh Shirvani, Saeid Ghasemshirazi, Behzad Beigzadeh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06067
• Pdf URL: https://arxiv.org/pdf/2403.06067
• Copy Paste: [[2403.06067]] Federated Learning: Attacks, Defenses, Opportunities, and Challenges(https://arxiv.org/abs/2403.06067)
• Keywords: secure, security, privacy, defense, attack, robust, federate
• Abstract: Using dispersed data and training, federated learning (FL) moves AI capabilities to edge devices or does tasks locally. Many consider FL the start of a new era in AI, yet it is still immature. FL has not garnered the community's trust since its security and privacy implications are controversial. FL's security and privacy concerns must be discovered, analyzed, and recorded before widespread usage and adoption. A solid comprehension of risk variables allows an FL practitioner to construct a secure environment and provide researchers with a clear perspective of potential study fields, making FL the best solution in situations where security and privacy are primary issues. This research aims to deliver a complete overview of FL's security and privacy features to help bridge the gap between current federated AI and broad adoption in the future. In this paper, we present a comprehensive overview of the attack surface to investigate FL's existing challenges and defense measures to evaluate its robustness and reliability. According to our study, security concerns regarding FL are more frequent than privacy issues. Communication bottlenecks, poisoning, and backdoor attacks represent FL's privacy's most significant security threats. In the final part, we detail future research that will assist FL in adapting to real-world settings.

Title: Reframe Anything: LLM Agent for Open World Video Reframing

• Authors: Jiawang Cao, Yongliang Wu, Weiheng Chi, Wenbo Zhu, Ziyue Su, Jay Wu
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.06070
• Pdf URL: https://arxiv.org/pdf/2403.06070
• Copy Paste: [[2403.06070]] Reframe Anything: LLM Agent for Open World Video Reframing(https://arxiv.org/abs/2403.06070)
• Keywords: large language model
• Abstract: The proliferation of mobile devices and social media has revolutionized content dissemination, with short-form video becoming increasingly prevalent. This shift has introduced the challenge of video reframing to fit various screen aspect ratios, a process that highlights the most compelling parts of a video. Traditionally, video reframing is a manual, time-consuming task requiring professional expertise, which incurs high production costs. A potential solution is to adopt some machine learning models, such as video salient object detection, to automate the process. However, these methods often lack generalizability due to their reliance on specific training data. The advent of powerful large language models (LLMs) open new avenues for AI capabilities. Building on this, we introduce Reframe Any Video Agent (RAVA), a LLM-based agent that leverages visual foundation models and human instructions to restructure visual content for video reframing. RAVA operates in three stages: perception, where it interprets user instructions and video content; planning, where it determines aspect ratios and reframing strategies; and execution, where it invokes the editing tools to produce the final video. Our experiments validate the effectiveness of RAVA in video salient object detection and real-world reframing tasks, demonstrating its potential as a tool for AI-powered video editing.

Title: Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing

• Authors: Liyang He, Zhenya Huang, Jiayu Liu, Enhong Chen, Fei Wang, Jing Sha, Shijin Wang
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.06071
• Pdf URL: https://arxiv.org/pdf/2403.06071
• Copy Paste: [[2403.06071]] Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing(https://arxiv.org/abs/2403.06071)
• Keywords: robust
• Abstract: Unsupervised semantic hashing has emerged as an indispensable technique for fast image search, which aims to convert images into binary hash codes without relying on labels. Recent advancements in the field demonstrate that employing large-scale backbones (e.g., ViT) in unsupervised semantic hashing models can yield substantial improvements. However, the inference delay has become increasingly difficult to overlook. Knowledge distillation provides a means for practical model compression to alleviate this delay. Nevertheless, the prevailing knowledge distillation approaches are not explicitly designed for semantic hashing. They ignore the unique search paradigm of semantic hashing, the inherent necessities of the distillation process, and the property of hash codes. In this paper, we propose an innovative Bit-mask Robust Contrastive knowledge Distillation (BRCD) method, specifically devised for the distillation of semantic hashing models. To ensure the effectiveness of two kinds of search paradigms in the context of semantic hashing, BRCD first aligns the semantic spaces between the teacher and student models through a contrastive knowledge distillation objective. Additionally, to eliminate noisy augmentations and ensure robust optimization, a cluster-based method within the knowledge distillation process is introduced. Furthermore, through a bit-level analysis, we uncover the presence of redundancy bits resulting from the bit independence property. To mitigate these effects, we introduce a bit mask mechanism in our knowledge distillation objective. Finally, extensive experiments not only showcase the noteworthy performance of our BRCD method in comparison to other knowledge distillation methods but also substantiate the generality of our methods across diverse semantic hashing models and backbones. The code for BRCD is available at https://github.com/hly1998/BRCD.

Title: Generalization of Graph Neural Networks through the Lens of Homomorphism

• Authors: Shouheng Li, Dongwoo Kim, Qing Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06079
• Pdf URL: https://arxiv.org/pdf/2403.06079
• Copy Paste: [[2403.06079]] Generalization of Graph Neural Networks through the Lens of Homomorphism(https://arxiv.org/abs/2403.06079)
• Keywords: robust
• Abstract: Despite the celebrated popularity of Graph Neural Networks (GNNs) across numerous applications, the ability of GNNs to generalize remains less explored. In this work, we propose to study the generalization of GNNs through a novel perspective - analyzing the entropy of graph homomorphism. By linking graph homomorphism with information-theoretic measures, we derive generalization bounds for both graph and node classifications. These bounds are capable of capturing subtleties inherent in various graph structures, including but not limited to paths, cycles and cliques. This enables a data-dependent generalization analysis with robust theoretical guarantees. To shed light on the generality of of our proposed bounds, we present a unifying framework that can characterize a broad spectrum of GNN models through the lens of graph homomorphism. We validate the practical applicability of our theoretical findings by showing the alignment between the proposed bounds and the empirically observed generalization gaps over both real-world and synthetic datasets.

Title: FrameQuant: Flexible Low-Bit Quantization for Transformers

• Authors: Harshavardhan Adepu, Zhanpeng Zeng, Li Zhang, Vikas Singh
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06082
• Pdf URL: https://arxiv.org/pdf/2403.06082
• Copy Paste: [[2403.06082]] FrameQuant: Flexible Low-Bit Quantization for Transformers(https://arxiv.org/abs/2403.06082)
• Keywords: robust, transformer
• Abstract: Transformers are the backbone of powerful foundation models for many Vision and Natural Language Processing tasks. But their compute and memory/storage footprint is large, and so, serving such models is expensive often requiring high-end hardware. To mitigate this difficulty, Post-Training Quantization seeks to modify a pre-trained model and quantize it to eight bits or lower, significantly boosting compute/memory/latency efficiency. Such models have been successfully quantized to four bits with some performance loss. In this work, we outline a simple scheme to quantize Transformer-based models to just two bits (plus some overhead) with only a small drop in accuracy. Key to our formulation is a concept borrowed from Harmonic analysis called Fusion Frames. Our main finding is that the quantization must take place not in the original weight space, but instead in the Fusion Frame representations. If quantization is interpreted as the addition of noise, our casting of the problem allows invoking an extensive body of known consistent recovery and noise robustness guarantees. Further, if desired, de-noising filters are known in closed form. We show empirically, via a variety of experiments, that (almost) two-bit quantization for Transformer models promises sizable efficiency gains.

Title: Towards In-Vehicle Multi-Task Facial Attribute Recognition: Investigating Synthetic Data and Vision Foundation Models

• Authors: Esmaeil Seraj, Walter Talamonti
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.06088
• Pdf URL: https://arxiv.org/pdf/2403.06088
• Copy Paste: [[2403.06088]] Towards In-Vehicle Multi-Task Facial Attribute Recognition: Investigating Synthetic Data and Vision Foundation Models(https://arxiv.org/abs/2403.06088)
• Keywords: robust, transformer
• Abstract: In the burgeoning field of intelligent transportation systems, enhancing vehicle-driver interaction through facial attribute recognition, such as facial expression, eye gaze, age, etc., is of paramount importance for safety, personalization, and overall user experience. However, the scarcity of comprehensive large-scale, real-world datasets poses a significant challenge for training robust multi-task models. Existing literature often overlooks the potential of synthetic datasets and the comparative efficacy of state-of-the-art vision foundation models in such constrained settings. This paper addresses these gaps by investigating the utility of synthetic datasets for training complex multi-task models that recognize facial attributes of passengers of a vehicle, such as gaze plane, age, and facial expression. Utilizing transfer learning techniques with both pre-trained Vision Transformer (ViT) and Residual Network (ResNet) models, we explore various training and adaptation methods to optimize performance, particularly when data availability is limited. We provide extensive post-evaluation analysis, investigating the effects of synthetic data distributions on model performance in in-distribution data and out-of-distribution inference. Our study unveils counter-intuitive findings, notably the superior performance of ResNet over ViTs in our specific multi-task context, which is attributed to the mismatch in model complexity relative to task complexity. Our results highlight the challenges and opportunities for enhancing the use of synthetic data and vision foundation models in practical applications.

Title: Diffusion Models Trained with Large Data Are Transferable Visual Models

• Authors: Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, Chunhua Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06090
• Pdf URL: https://arxiv.org/pdf/2403.06090
• Copy Paste: [[2403.06090]] Diffusion Models Trained with Large Data Are Transferable Visual Models(https://arxiv.org/abs/2403.06090)
• Keywords: robust, diffusion, transformer, segmentation
• Abstract: We show that, simply initializing image understanding models using a pre-trained UNet (or transformer) of diffusion models, it is possible to achieve remarkable transferable performance on fundamental vision perception tasks using a moderate amount of target data (even synthetic data only), including monocular depth, surface normal, image segmentation, matting, human pose estimation, among virtually many others. Previous works have adapted diffusion models for various perception tasks, often reformulating these tasks as generation processes to align with the diffusion process. In sharp contrast, we demonstrate that fine-tuning these models with minimal adjustments can be a more effective alternative, offering the advantages of being embarrassingly simple and significantly faster. As the backbone network of Stable Diffusion models is trained on giant datasets comprising billions of images, we observe very robust generalization capabilities of the diffusion backbone. Experimental results showcase the remarkable transferability of the backbone of diffusion models across diverse tasks and real-world datasets.

Title: Enhancing 3D Object Detection with 2D Detection-Guided Query Anchors

• Authors: Haoxuanye Ji, Pengpeng Liang, Erkang Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06093
• Pdf URL: https://arxiv.org/pdf/2403.06093
• Copy Paste: [[2403.06093]] Enhancing 3D Object Detection with 2D Detection-Guided Query Anchors(https://arxiv.org/abs/2403.06093)
• Keywords: extraction
• Abstract: Multi-camera-based 3D object detection has made notable progress in the past several years. However, we observe that there are cases (e.g. faraway regions) in which popular 2D object detectors are more reliable than state-of-the-art 3D detectors. In this paper, to improve the performance of query-based 3D object detectors, we present a novel query generating approach termed QAF2D, which infers 3D query anchors from 2D detection results. A 2D bounding box of an object in an image is lifted to a set of 3D anchors by associating each sampled point within the box with depth, yaw angle, and size candidates. Then, the validity of each 3D anchor is verified by comparing its projection in the image with its corresponding 2D box, and only valid anchors are kept and used to construct queries. The class information of the 2D bounding box associated with each query is also utilized to match the predicted boxes with ground truth for the set-based loss. The image feature extraction backbone is shared between the 3D detector and 2D detector by adding a small number of prompt parameters. We integrate QAF2D into three popular query-based 3D object detectors and carry out comprehensive evaluations on the nuScenes dataset. The largest improvement that QAF2D can bring about on the nuScenes validation subset is $2.3\%$ NDS and $2.7\%$ mAP. Code is available at https://github.com/nullmax-vision/QAF2D.

Title: SecureRights: A Blockchain-Powered Trusted DRM Framework for Robust Protection and Asserting Digital Rights

• Authors: Tiroshan Madushanka, Dhammika S. Kumara, Atheesh A. Rathnaweera
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06094
• Pdf URL: https://arxiv.org/pdf/2403.06094
• Copy Paste: [[2403.06094]] SecureRights: A Blockchain-Powered Trusted DRM Framework for Robust Protection and Asserting Digital Rights(https://arxiv.org/abs/2403.06094)
• Keywords: secure, protect, attack, robust, watermark
• Abstract: In the dynamic realm of digital content, safeguarding intellectual property rights poses critical challenges. This paper presents "SecureRights," an innovative Blockchain-based Trusted Digital Rights Management (DRM) framework. It strengthens the defence against unauthorized use and streamlines the claim of digital rights. Utilizing blockchain, digital watermarking, perceptual hashing, Quick Response (QR) codes, and the Interplanetary File System (IPFS), SecureRights securely stores watermark information on the blockchain with timestamp authentication. Incorporating perceptual hashing generates robust hash tokens based on image structure. The addition of QR codes enhances the watermarking, offering a comprehensive solution for resilient intellectual property rights protection. Rigorous evaluations affirm SecureRights' resilience against various attacks, establishing its efficacy in safeguarding digital content and simplifying rightful ownership assertion.

Title: Can LLM Substitute Human Labeling? A Case Study of Fine-grained Chinese Address Entity Recognition Dataset for UAV Delivery

• Authors: Yuxuan Yao, Sichun Luo, Haohan Zhao, Guanzhi Deng, Linqi Song
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.06097
• Pdf URL: https://arxiv.org/pdf/2403.06097
• Copy Paste: [[2403.06097]] Can LLM Substitute Human Labeling? A Case Study of Fine-grained Chinese Address Entity Recognition Dataset for UAV Delivery(https://arxiv.org/abs/2403.06097)
• Keywords: privacy
• Abstract: We present CNER-UAV, a fine-grained \textbf{C}hinese \textbf{N}ame \textbf{E}ntity \textbf{R}ecognition dataset specifically designed for the task of address resolution in \textbf{U}nmanned \textbf{A}erial \textbf{V}ehicle delivery systems. The dataset encompasses a diverse range of five categories, enabling comprehensive training and evaluation of NER models. To construct this dataset, we sourced the data from a real-world UAV delivery system and conducted a rigorous data cleaning and desensitization process to ensure privacy and data integrity. The resulting dataset, consisting of around 12,000 annotated samples, underwent human experts and \textbf{L}arge \textbf{L}anguage \textbf{M}odel annotation. We evaluated classical NER models on our dataset and provided in-depth analysis. The dataset and models are publicly available at \url{https://github.com/zhhvvv/CNER-UAV}.

Title: VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models

• Authors: Wenhao Wang, Yi Yang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06098
• Pdf URL: https://arxiv.org/pdf/2403.06098
• Copy Paste: [[2403.06098]] VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models(https://arxiv.org/abs/2403.06098)
• Keywords: diffusion
• Abstract: The arrival of Sora marks a new era for text-to-video diffusion models, bringing significant advancements in video generation and potential applications. However, Sora, as well as other text-to-video diffusion models, highly relies on the prompts, and there is no publicly available dataset featuring a study of text-to-video prompts. In this paper, we introduce VidProM, the first large-scale dataset comprising 1.67 million unique text-to-video prompts from real users. Additionally, the dataset includes 6.69 million videos generated by four state-of-the-art diffusion models and some related data. We initially demonstrate the curation of this large-scale dataset, which is a time-consuming and costly process. Subsequently, we show how the proposed VidProM differs from DiffusionDB, a large-scale prompt-gallery dataset for image generation. Based on the analysis of these prompts, we identify the necessity for a new prompt dataset specifically designed for text-to-video generation and gain insights into the preferences of real users when creating videos. Our large-scale and diverse dataset also inspires many exciting new research areas. For instance, to develop better, more efficient, and safer text-to-video diffusion models, we suggest exploring text-to-video prompt engineering, efficient video generation, and video copy detection for diffusion models. We make the collected dataset VidProM publicly available at GitHub and Hugging Face under the CC-BY- NC 4.0 License.

Title: Coherent Temporal Synthesis for Incremental Action Segmentation

• Authors: Guodong Ding, Hans Golong, Angela Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06102
• Pdf URL: https://arxiv.org/pdf/2403.06102
• Copy Paste: [[2403.06102]] Coherent Temporal Synthesis for Incremental Action Segmentation(https://arxiv.org/abs/2403.06102)
• Keywords: generative, segmentation
• Abstract: Data replay is a successful incremental learning technique for images. It prevents catastrophic forgetting by keeping a reservoir of previous data, original or synthesized, to ensure the model retains past knowledge while adapting to novel concepts. However, its application in the video domain is rudimentary, as it simply stores frame exemplars for action recognition. This paper presents the first exploration of video data replay techniques for incremental action segmentation, focusing on action temporal modeling. We propose a Temporally Coherent Action (TCA) model, which represents actions using a generative model instead of storing individual frames. The integration of a conditioning variable that captures temporal coherence allows our model to understand the evolution of action features over time. Therefore, action segments generated by TCA for replay are diverse and temporally coherent. In a 10-task incremental setup on the Breakfast dataset, our approach achieves significant increases in accuracy for up to 22% compared to the baselines.

Title: Universal Debiased Editing for Fair Medical Image Classification

• Authors: Ruinan Jin, Wenlong Deng, Minghui Chen, Xiaoxiao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06104
• Pdf URL: https://arxiv.org/pdf/2403.06104
• Copy Paste: [[2403.06104]] Universal Debiased Editing for Fair Medical Image Classification(https://arxiv.org/abs/2403.06104)
• Keywords: fair
• Abstract: In the era of Foundation Models' (FMs) rising prominence in AI, our study addresses the challenge of biases in medical images while using FM API, particularly spurious correlations between pixels and sensitive attributes. Traditional methods for bias mitigation face limitations due to the restricted access to web-hosted FMs and difficulties in addressing the underlying bias encoded within the FM API. We propose an U(niversal) D(ebiased) E(diting) strategy, termed UDE, which generates UDE noise to mask such spurious correlation. UDE is capable of mitigating bias both within the FM API embedding and the images themselves. Furthermore, UDE is suitable for both white-box and black-box FM APIs, where we introduced G(reedy) (Z)eroth-O(rder) (GeZO) optimization for it when the gradient is inaccessible in black-box APIs. Our whole pipeline enables fairness-aware image editing that can be applied across various medical contexts without requiring direct model manipulation or significant computational resources. Our empirical results demonstrate the method's effectiveness in maintaining fairness and utility across different patient groups and diseases. In the era of AI-driven medicine, this work contributes to making healthcare diagnostics more equitable, showcasing a practical solution for bias mitigation in pre-trained image FMs.

Title: Textureless Object Recognition: An Edge-based Approach

• Authors: Frincy Clement, Kirtan Shah, Dhara Pancholi, Gabriel Lugo Bustillo, Dr. Irene Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06107
• Pdf URL: https://arxiv.org/pdf/2403.06107
• Copy Paste: [[2403.06107]] Textureless Object Recognition: An Edge-based Approach(https://arxiv.org/abs/2403.06107)
• Keywords: robust
• Abstract: Textureless object recognition has become a significant task in Computer Vision with the advent of Robotics and its applications in manufacturing sector. It has been challenging to obtain good accuracy in real time because of its lack of discriminative features and reflectance properties which makes the techniques for textured object recognition insufficient for textureless objects. A lot of work has been done in the last 20 years, especially in the recent 5 years after the TLess and other textureless dataset were introduced. In this project, by applying image processing techniques we created a robust augmented dataset from initial imbalanced smaller dataset. We extracted edge features, feature combinations and RGB images enhanced with feature/feature combinations to create 15 datasets, each with a size of ~340,000. We then trained four classifiers on these 15 datasets to arrive at a conclusion as to which dataset performs the best overall and whether edge features are important for textureless objects. Based on our experiments and analysis, RGB images enhanced with combination of 3 edge features performed the best compared to all others. Model performance on dataset with HED edges performed comparatively better than other edge detectors like Canny or Prewitt.

Title: Large Language Models on Fine-grained Emotion Detection Dataset with Data Augmentation and Transfer Learning

• Authors: Kaipeng Wang, Zhi Jing, Yongye Su, Yikun Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06108
• Pdf URL: https://arxiv.org/pdf/2403.06108
• Copy Paste: [[2403.06108]] Large Language Models on Fine-grained Emotion Detection Dataset with Data Augmentation and Transfer Learning(https://arxiv.org/abs/2403.06108)
• Keywords: large language model
• Abstract: This paper delves into enhancing the classification performance on the GoEmotions dataset, a large, manually annotated dataset for emotion detection in text. The primary goal of this paper is to address the challenges of detecting subtle emotions in text, a complex issue in Natural Language Processing (NLP) with significant practical applications. The findings offer valuable insights into addressing the challenges of emotion detection in text and suggest directions for future research, including the potential for a survey paper that synthesizes methods and performances across various datasets in this domain.

Title: FMPAF: How Do Fed Chairs Affect the Financial Market? A Fine-grained Monetary Policy Analysis Framework on Their Language

• Authors: Yayue Deng, Mohan Xu, Yao Tang
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2403.06115
• Pdf URL: https://arxiv.org/pdf/2403.06115
• Copy Paste: [[2403.06115]] FMPAF: How Do Fed Chairs Affect the Financial Market? A Fine-grained Monetary Policy Analysis Framework on Their Language(https://arxiv.org/abs/2403.06115)
• Keywords: large language model
• Abstract: The effectiveness of central bank communication is a crucial aspect of monetary policy transmission. While recent research has examined the influence of policy communication by the chairs of the Federal Reserve on various financial variables, much of the literature relies on rule-based or dictionary-based methods in parsing the language of the chairs, leaving nuanced information about policy stance contained in nonverbal emotion out of the analysis. In the current study, we propose the Fine-Grained Monetary Policy Analysis Framework (FMPAF), a novel approach that integrates large language models (LLMs) with regression analysis to provide a comprehensive analysis of the impact of the press-conference communications of chairs of the Federal Reserve on financial markets. We conduct extensive comparisons of model performance under different levels of granularity, modalities, and communication scenarios. Based on our preferred specification, a one-unit increase in the sentiment score is associated with an increase of the price of S\&P 500 Exchange-Traded Fund by approximately 500 basis points, a 15-basis-point decrease in the policy interest rate, while not leading to a significant response in exchange rates.

Title: CLEAR: Cross-Transformers with Pre-trained Language Model is All you need for Person Attribute Recognition and Retrieval

• Authors: Doanh C. Bui, Thinh V. Le, Hung Ba Ngo, Tae Jong Choi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06119
• Pdf URL: https://arxiv.org/pdf/2403.06119
• Copy Paste: [[2403.06119]] CLEAR: Cross-Transformers with Pre-trained Language Model is All you need for Person Attribute Recognition and Retrieval(https://arxiv.org/abs/2403.06119)
• Keywords: robust, transformer
• Abstract: Person attribute recognition and attribute-based retrieval are two core human-centric tasks. In the recognition task, the challenge is specifying attributes depending on a person's appearance, while the retrieval task involves searching for matching persons based on attribute queries. There is a significant relationship between recognition and retrieval tasks. In this study, we demonstrate that if there is a sufficiently robust network to solve person attribute recognition, it can be adapted to facilitate better performance for the retrieval task. Another issue that needs addressing in the retrieval task is the modality gap between attribute queries and persons' images. Therefore, in this paper, we present CLEAR, a unified network designed to address both tasks. We introduce a robust cross-transformers network to handle person attribute recognition. Additionally, leveraging a pre-trained language model, we construct pseudo-descriptions for attribute queries and introduce an effective training strategy to train only a few additional parameters for adapters, facilitating the handling of the retrieval task. Finally, the unified CLEAR model is evaluated on five benchmarks: PETA, PA100K, Market-1501, RAPv2, and UPAR-2024. Without bells and whistles, CLEAR achieves state-of-the-art performance or competitive results for both tasks, significantly outperforming other competitors in terms of person retrieval performance on the widely-used Market-1501 dataset.

Title: Style Blind Domain Generalized Semantic Segmentation via Covariance Alignment and Semantic Consistence Contrastive Learning

• Authors: Woo-Jin Ahn, Geun-Yeong Yang, Hyun-Duck Choi, Myo-Taeg Lim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06122
• Pdf URL: https://arxiv.org/pdf/2403.06122
• Copy Paste: [[2403.06122]] Style Blind Domain Generalized Semantic Segmentation via Covariance Alignment and Semantic Consistence Contrastive Learning(https://arxiv.org/abs/2403.06122)
• Keywords: robust, segmentation
• Abstract: Deep learning models for semantic segmentation often experience performance degradation when deployed to unseen target domains unidentified during the training phase. This is mainly due to variations in image texture (\ie style) from different data sources. To tackle this challenge, existing domain generalized semantic segmentation (DGSS) methods attempt to remove style variations from the feature. However, these approaches struggle with the entanglement of style and content, which may lead to the unintentional removal of crucial content information, causing performance degradation. This study addresses this limitation by proposing BlindNet, a novel DGSS approach that blinds the style without external modules or datasets. The main idea behind our proposed approach is to alleviate the effect of style in the encoder whilst facilitating robust segmentation in the decoder. To achieve this, BlindNet comprises two key components: covariance alignment and semantic consistency contrastive learning. Specifically, the covariance alignment trains the encoder to uniformly recognize various styles and preserve the content information of the feature, rather than removing the style-sensitive factor. Meanwhile, semantic consistency contrastive learning enables the decoder to construct discriminative class embedding space and disentangles features that are vulnerable to misclassification. Through extensive experiments, our approach outperforms existing DGSS methods, exhibiting robustness and superior performance for semantic segmentation on unseen target domains.

Title: PSS-BA: LiDAR Bundle Adjustment with Progressive Spatial Smoothing

• Authors: Jianping Li, Thien-Minh Nguyen, Shenghai Yuan, Lihua Xie
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.06124
• Pdf URL: https://arxiv.org/pdf/2403.06124
• Copy Paste: [[2403.06124]] PSS-BA: LiDAR Bundle Adjustment with Progressive Spatial Smoothing(https://arxiv.org/abs/2403.06124)
• Keywords: robust
• Abstract: Accurate and consistent construction of point clouds from LiDAR scanning data is fundamental for 3D modeling applications. Current solutions, such as multiview point cloud registration and LiDAR bundle adjustment, predominantly depend on the local plane assumption, which may be inadequate in complex environments lacking of planar geometries or substantial initial pose errors. To mitigate this problem, this paper presents a LiDAR bundle adjustment with progressive spatial smoothing, which is suitable for complex environments and exhibits improved convergence capabilities. The proposed method consists of a spatial smoothing module and a pose adjustment module, which combines the benefits of local consistency and global accuracy. With the spatial smoothing module, we can obtain robust and rich surface constraints employing smoothing kernels across various scales. Then the pose adjustment module corrects all poses utilizing the novel surface constraints. Ultimately, the proposed method simultaneously achieves fine poses and parametric surfaces that can be directly employed for high-quality point cloud reconstruction. The effectiveness and robustness of our proposed approach have been validated on both simulation and real-world datasets. The experimental results demonstrate that the proposed method outperforms the existing methods and achieves better accuracy in complex environments with low planar structures.

Title: ClickVOS: Click Video Object Segmentation

• Authors: Pinxue Guo, Lingyi Hong, Xinyu Zhou, Shuyong Gao, Wanyun Li, Jinglun Li, Zhaoyu Chen, Xiaoqiang Li, Wei Zhang, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06130
• Pdf URL: https://arxiv.org/pdf/2403.06130
• Copy Paste: [[2403.06130]] ClickVOS: Click Video Object Segmentation(https://arxiv.org/abs/2403.06130)
• Keywords: segmentation
• Abstract: Video Object Segmentation (VOS) task aims to segment objects in videos. However, previous settings either require time-consuming manual masks of target objects at the first frame during inference or lack the flexibility to specify arbitrary objects of interest. To address these limitations, we propose the setting named Click Video Object Segmentation (ClickVOS) which segments objects of interest across the whole video according to a single click per object in the first frame. And we provide the extended datasets DAVIS-P and YouTubeVOSP that with point annotations to support this task. ClickVOS is of significant practical applications and research implications due to its only 1-2 seconds interaction time for indicating an object, comparing annotating the mask of an object needs several minutes. However, ClickVOS also presents increased challenges. To address this task, we propose an end-to-end baseline approach named called Attention Before Segmentation (ABS), motivated by the attention process of humans. ABS utilizes the given point in the first frame to perceive the target object through a concise yet effective segmentation attention. Although the initial object mask is possibly inaccurate, in our ABS, as the video goes on, the initially imprecise object mask can self-heal instead of deteriorating due to error accumulation, which is attributed to our designed improvement memory that continuously records stable global object memory and updates detailed dense memory. In addition, we conduct various baseline explorations utilizing off-the-shelf algorithms from related fields, which could provide insights for the further exploration of ClickVOS. The experimental results demonstrate the superiority of the proposed ABS approach. Extended datasets and codes will be available at https://github.com/PinxueGuo/ClickVOS.

Title: FedPIT: Towards Privacy-preserving and Few-shot Federated Instruction Tuning

• Authors: Zhuo Zhang, Jingyuan Zhang, Jintao Huang, Lizhen Qu, Hongzhi Zhang, Zenglin Xu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06131
• Pdf URL: https://arxiv.org/pdf/2403.06131
• Copy Paste: [[2403.06131]] FedPIT: Towards Privacy-preserving and Few-shot Federated Instruction Tuning(https://arxiv.org/abs/2403.06131)
• Keywords: privacy, attack, robust, extraction, federate, large language model
• Abstract: Instruction tuning has proven essential for enhancing the performance of large language models (LLMs) in generating human-aligned responses. However, collecting diverse, high-quality instruction data for tuning poses challenges, particularly in privacy-sensitive domains. Federated instruction tuning (FedIT) has emerged as a solution, leveraging federated learning from multiple data owners while preserving privacy. Yet, it faces challenges due to limited instruction data and vulnerabilities to training data extraction attacks. To address these issues, we propose a novel federated algorithm, FedPIT, which utilizes LLMs' in-context learning capability to self-generate task-specific synthetic data for training autonomously. Our method employs parameter-isolated training to maintain global parameters trained on synthetic data and local parameters trained on augmented local data, effectively thwarting data extraction attacks. Extensive experiments on real-world medical data demonstrate the effectiveness of FedPIT in improving federated few-shot performance while preserving privacy and robustness against data heterogeneity.

Title: MACE: Mass Concept Erasure in Diffusion Models

• Authors: Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, Adams Wai-Kin Kong
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06135
• Pdf URL: https://arxiv.org/pdf/2403.06135
• Copy Paste: [[2403.06135]] MACE: Mass Concept Erasure in Diffusion Models(https://arxiv.org/abs/2403.06135)
• Keywords: diffusion
• Abstract: The rapid expansion of large-scale text-to-image diffusion models has raised growing concerns regarding their potential misuse in creating harmful or misleading content. In this paper, we introduce MACE, a finetuning framework for the task of mass concept erasure. This task aims to prevent models from generating images that embody unwanted concepts when prompted. Existing concept erasure methods are typically restricted to handling fewer than five concepts simultaneously and struggle to find a balance between erasing concept synonyms (generality) and maintaining unrelated concepts (specificity). In contrast, MACE differs by successfully scaling the erasure scope up to 100 concepts and by achieving an effective balance between generality and specificity. This is achieved by leveraging closed-form cross-attention refinement along with LoRA finetuning, collectively eliminating the information of undesirable concepts. Furthermore, MACE integrates multiple LoRAs without mutual interference. We conduct extensive evaluations of MACE against prior methods across four different tasks: object erasure, celebrity erasure, explicit content erasure, and artistic style erasure. Our results reveal that MACE surpasses prior methods in all evaluated tasks. Code is available at https://github.com/Shilin-LU/MACE.

Title: Bayesian Random Semantic Data Augmentation for Medical Image Classification

• Authors: Yaoyao Zhu, Xiuding Cai, Xueyao Wang, Yu Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06138
• Pdf URL: https://arxiv.org/pdf/2403.06138
• Copy Paste: [[2403.06138]] Bayesian Random Semantic Data Augmentation for Medical Image Classification(https://arxiv.org/abs/2403.06138)
• Keywords: robust, generative
• Abstract: Data augmentation is a critical regularization technique for deep neural networks, particularly in medical image classification. Popular data augmentation approaches include image transformation-based methods, generative data augmentation, and automatic data augmentation. However, these approaches encounter notable limitations: image transformation-based and automated data augmentation techniques cannot implement semantic transformations, leading to a constrained variety of augmented samples, and generative data augmentation methods are computationally expensive. In response to these challenges, we proposed Bayesian Random Semantic Data Augmentation (BRSDA), a novel, efficient, and plug-and-play semantic data augmentation method. BRSDA is motivated by a simple translation in the feature space along specific directions that can effectuate semantic transformations. When given a feature, we define its augmentable semantic magnitude as a random variable and estimate its distribution using variational Bayesian, then sample semantic magnitude and add to the randomly selected semantic direction to achieve semantic data augmentation. We demonstrate the effectiveness of BRSDA on five 2D and six 3D medical image datasets covering nine modalities. We also test BRSDA with mainstream neural network architectures, showcasing its robustness. Furthermore, combining BRSDA with other leading data augmentation methods achieves superior performance. Code is available online at \url{https://github.com/YaoyaoZhu19/BRSDA}.

Title: Fine-grainedly Synthesize Streaming Data Based On Large Language Models With Graph Structure Understanding For Data Sparsity

• Authors: Xin Zhang, Linhai Zhang, Deyu Zhou, Guoqiang Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06139
• Pdf URL: https://arxiv.org/pdf/2403.06139
• Copy Paste: [[2403.06139]] Fine-grainedly Synthesize Streaming Data Based On Large Language Models With Graph Structure Understanding For Data Sparsity(https://arxiv.org/abs/2403.06139)
• Keywords: extraction, large language model
• Abstract: Due to the sparsity of user data, sentiment analysis on user reviews in e-commerce platforms often suffers from poor performance, especially when faced with extremely sparse user data or long-tail labels. Recently, the emergence of LLMs has introduced new solutions to such problems by leveraging graph structures to generate supplementary user profiles. However, previous approaches have not fully utilized the graph understanding capabilities of LLMs and have struggled to adapt to complex streaming data environments. In this work, we propose a fine-grained streaming data synthesis framework that categorizes sparse users into three categories: Mid-tail, Long-tail, and Extreme. Specifically, we design LLMs to comprehensively understand three key graph elements in streaming data, including Local-global Graph Understanding, Second-Order Relationship Extraction, and Product Attribute Understanding, which enables the generation of high-quality synthetic data to effectively address sparsity across different categories. Experimental results on three real datasets demonstrate significant performance improvements, with synthesized data contributing to MSE reductions of 45.85%, 3.16%, and 62.21%, respectively.

Title: Fluent: Round-efficient Secure Aggregation for Private Federated Learning

• Authors: Xincheng Li, Jianting Ning, Geong Sen Poh, Leo Yu Zhang, Xinchun Yin, Tianwei Zhang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06143
• Pdf URL: https://arxiv.org/pdf/2403.06143
• Copy Paste: [[2403.06143]] Fluent: Round-efficient Secure Aggregation for Private Federated Learning(https://arxiv.org/abs/2403.06143)
• Keywords: secure, privacy, attack, federate
• Abstract: Federated learning (FL) facilitates collaborative training of machine learning models among a large number of clients while safeguarding the privacy of their local datasets. However, FL remains susceptible to vulnerabilities such as privacy inference and inversion attacks. Single-server secure aggregation schemes were proposed to address these threats. Nonetheless, they encounter practical constraints due to their round and communication complexities. This work introduces Fluent, a round and communication-efficient secure aggregation scheme for private FL. Fluent has several improvements compared to state-of-the-art solutions like Bell et al. (CCS 2020) and Ma et al. (SP 2023): (1) it eliminates frequent handshakes and secret sharing operations by efficiently reusing the shares across multiple training iterations without leaking any private information; (2) it accomplishes both the consistency check and gradient unmasking in one logical step, thereby reducing another round of communication. With these innovations, Fluent achieves the fewest communication rounds (i.e., two in the collection phase) in the malicious server setting, in contrast to at least three rounds in existing schemes. This significantly minimizes the latency for geographically distributed clients; (3) Fluent also introduces Fluent-Dynamic with a participant selection algorithm and an alternative secret sharing scheme. This can facilitate dynamic client joining and enhance the system flexibility and scalability. We implemented Fluent and compared it with existing solutions. Experimental results show that Fluent improves the computational cost by at least 75% and communication overhead by at least 25% for normal clients. Fluent also reduces the communication overhead for the server at the expense of a marginal increase in computational cost.

Title: All-in-one platform for AI R&D in medical imaging, encompassing data collection, selection, annotation, and pre-processing

• Authors: Changhee Han, Kyohei Shibano, Wataru Ozaki, Keishiro Osaki, Takafumi Haraguchi, Daisuke Hirahara, Shumon Kimura, Yasuyuki Kobayashi, Gento Mogi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06145
• Pdf URL: https://arxiv.org/pdf/2403.06145
• Copy Paste: [[2403.06145]] All-in-one platform for AI R&D in medical imaging, encompassing data collection, selection, annotation, and pre-processing(https://arxiv.org/abs/2403.06145)
• Keywords: security, generative
• Abstract: Deep Learning is advancing medical imaging Research and Development (R&D), leading to the frequent clinical use of Artificial Intelligence/Machine Learning (AI/ML)-based medical devices. However, to advance AI R&D, two challenges arise: 1) significant data imbalance, with most data from Europe/America and under 10% from Asia, despite its 60% global population share; and 2) hefty time and investment needed to curate proprietary datasets for commercial use. In response, we established the first commercial medical imaging platform, encompassing steps like: 1) data collection, 2) data selection, 3) annotation, and 4) pre-processing. Moreover, we focus on harnessing under-represented data from Japan and broader Asia, including Computed Tomography, Magnetic Resonance Imaging, and Whole Slide Imaging scans. Using the collected data, we are preparing/providing ready-to-use datasets for medical AI R&D by 1) offering these datasets to AI firms, biopharma, and medical device makers and 2) using them as training/test data to develop tailored AI solutions for such entities. We also aim to merge Blockchain for data security and plan to synthesize rare disease data via generative AI. DataHub Website: https://medical-datahub.ai/

Title: Can Large Language Models Automatically Score Proficiency of Written Essays?

• Authors: Watheq Mansour, Salam Albatarni, Sohaila Eltanbouly, Tamer Elsayed
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06149
• Pdf URL: https://arxiv.org/pdf/2403.06149
• Copy Paste: [[2403.06149]] Can Large Language Models Automatically Score Proficiency of Written Essays?(https://arxiv.org/abs/2403.06149)
• Keywords: transformer, large language model
• Abstract: Although several methods were proposed to address the problem of automated essay scoring (AES) in the last 50 years, there is still much to desire in terms of effectiveness. Large Language Models (LLMs) are transformer-based models that demonstrate extraordinary capabilities on various tasks. In this paper, we test the ability of LLMs, given their powerful linguistic knowledge, to analyze and effectively score written essays. We experimented with two popular LLMs, namely ChatGPT and Llama. We aim to check if these models can do this task and, if so, how their performance is positioned among the state-of-the-art (SOTA) models across two levels, holistically and per individual writing trait. We utilized prompt-engineering tactics in designing four different prompts to bring their maximum potential to this task. Our experiments conducted on the ASAP dataset revealed several interesting observations. First, choosing the right prompt depends highly on the model and nature of the task. Second, the two LLMs exhibited comparable average performance in AES, with a slight advantage for ChatGPT. Finally, despite the performance gap between the two LLMs and SOTA models in terms of predictions, they provide feedback to enhance the quality of the essays, which can potentially help both teachers and students.

Title: GlanceVAD: Exploring Glance Supervision for Label-efficient Video Anomaly Detection

• Authors: Huaxin Zhang, Xiang Wang, Xiaohao Xu, Xiaonan Huang, Chuchu Han, Yuehuan Wang, Changxin Gao, Shanjun Zhang, Nong Sang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06154
• Pdf URL: https://arxiv.org/pdf/2403.06154
• Copy Paste: [[2403.06154]] GlanceVAD: Exploring Glance Supervision for Label-efficient Video Anomaly Detection(https://arxiv.org/abs/2403.06154)
• Keywords: robust
• Abstract: In recent years, video anomaly detection has been extensively investigated in both unsupervised and weakly supervised settings to alleviate costly temporal labeling. Despite significant progress, these methods still suffer from unsatisfactory results such as numerous false alarms, primarily due to the absence of precise temporal anomaly annotation. In this paper, we present a novel labeling paradigm, termed "glance annotation", to achieve a better balance between anomaly detection accuracy and annotation cost. Specifically, glance annotation is a random frame within each abnormal event, which can be easily accessed and is cost-effective. To assess its effectiveness, we manually annotate the glance annotations for two standard video anomaly detection datasets: UCF-Crime and XD-Violence. Additionally, we propose a customized GlanceVAD method, that leverages gaussian kernels as the basic unit to compose the temporal anomaly distribution, enabling the learning of diverse and robust anomaly representations from the glance annotations. Through comprehensive analysis and experiments, we verify that the proposed labeling paradigm can achieve an excellent trade-off between annotation cost and model performance. Extensive experimental results also demonstrate the effectiveness of our GlanceVAD approach, which significantly outperforms existing advanced unsupervised and weakly supervised methods. Code and annotations will be publicly available at https://github.com/pipixin321/GlanceVAD.

Title: Platypose: Calibrated Zero-Shot Multi-Hypothesis 3D Human Motion Estimation

• Authors: Paweł A. Pierzchlewicz, Caio da Silva, R. James Cotton, Fabian H. Sinz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06164
• Pdf URL: https://arxiv.org/pdf/2403.06164
• Copy Paste: [[2403.06164]] Platypose: Calibrated Zero-Shot Multi-Hypothesis 3D Human Motion Estimation(https://arxiv.org/abs/2403.06164)
• Keywords: diffusion
• Abstract: Single camera 3D pose estimation is an ill-defined problem due to inherent ambiguities from depth, occlusion or keypoint noise. Multi-hypothesis pose estimation accounts for this uncertainty by providing multiple 3D poses consistent with the 2D measurements. Current research has predominantly concentrated on generating multiple hypotheses for single frame static pose estimation. In this study we focus on the new task of multi-hypothesis motion estimation. Motion estimation is not simply pose estimation applied to multiple frames, which would ignore temporal correlation across frames. Instead, it requires distributions which are capable of generating temporally consistent samples, which is significantly more challenging. To this end, we introduce Platypose, a framework that uses a diffusion model pretrained on 3D human motion sequences for zero-shot 3D pose sequence estimation. Platypose outperforms baseline methods on multiple hypotheses for motion estimation. Additionally, Platypose also achieves state-of-the-art calibration and competitive joint error when tested on static poses from Human3.6M, MPI-INF-3DHP and 3DPW. Finally, because it is zero-shot, our method generalizes flexibly to different settings such as multi-camera inference.

Title: DiffuMatting: Synthesizing Arbitrary Objects with Matting-level Annotation

• Authors: Xiaobin Hu, Xu Peng, Donghao Luo, Xiaozhong Ji, Jinlong Peng, Zhengkai Jiang, Jiangning Zhang, Taisong Jin, Chengjie Wang, Rongrong Ji
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06168
• Pdf URL: https://arxiv.org/pdf/2403.06168
• Copy Paste: [[2403.06168]] DiffuMatting: Synthesizing Arbitrary Objects with Matting-level Annotation(https://arxiv.org/abs/2403.06168)
• Keywords: diffusion
• Abstract: Due to the difficulty and labor-consuming nature of getting highly accurate or matting annotations, there only exists a limited amount of highly accurate labels available to the public. To tackle this challenge, we propose a DiffuMatting which inherits the strong Everything generation ability of diffusion and endows the power of "matting anything". Our DiffuMatting can 1). act as an anything matting factory with high accurate annotations 2). be well-compatible with community LoRAs or various conditional control approaches to achieve the community-friendly art design and controllable generation. Specifically, inspired by green-screen-matting, we aim to teach the diffusion model to paint on a fixed green screen canvas. To this end, a large-scale greenscreen dataset (Green100K) is collected as a training dataset for DiffuMatting. Secondly, a green background control loss is proposed to keep the drawing board as a pure green color to distinguish the foreground and background. To ensure the synthesized object has more edge details, a detailed-enhancement of transition boundary loss is proposed as a guideline to generate objects with more complicated edge structures. Aiming to simultaneously generate the object and its matting annotation, we build a matting head to make a green color removal in the latent space of the VAE decoder. Our DiffuMatting shows several potential applications (e.g., matting-data generator, community-friendly art design and controllable generation). As a matting-data generator, DiffuMatting synthesizes general object and portrait matting sets, effectively reducing the relative MSE error by 15.4% in General Object Matting and 11.4% in Portrait Matting tasks.

Title: An Improved Analysis of Langevin Algorithms with Prior Diffusion for Non-Log-Concave Sampling

• Authors: Xunpeng Huang, Hanze Dong, Difan Zou, Tong Zhang
• Subjects: cs.LG, math.OC, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.06183
• Pdf URL: https://arxiv.org/pdf/2403.06183
• Copy Paste: [[2403.06183]] An Improved Analysis of Langevin Algorithms with Prior Diffusion for Non-Log-Concave Sampling(https://arxiv.org/abs/2403.06183)
• Keywords: diffusion
• Abstract: Understanding the dimension dependency of computational complexity in high-dimensional sampling problem is a fundamental problem, both from a practical and theoretical perspective. Compared with samplers with unbiased stationary distribution, e.g., Metropolis-adjusted Langevin algorithm (MALA), biased samplers, e.g., Underdamped Langevin Dynamics (ULD), perform better in low-accuracy cases just because a lower dimension dependency in their complexities. Along this line, Freund et al. (2022) suggest that the modified Langevin algorithm with prior diffusion is able to converge dimension independently for strongly log-concave target distributions. Nonetheless, it remains open whether such property establishes for more general cases. In this paper, we investigate the prior diffusion technique for the target distributions satisfying log-Sobolev inequality (LSI), which covers a much broader class of distributions compared to the strongly log-concave ones. In particular, we prove that the modified Langevin algorithm can also obtain the dimension-independent convergence of KL divergence with different step size schedules. The core of our proof technique is a novel construction of an interpolating SDE, which significantly helps to conduct a more accurate characterization of the discrete updates of the overdamped Langevin dynamics. Our theoretical analysis demonstrates the benefits of prior diffusion for a broader class of target distributions and provides new insights into developing faster sampling algorithms.

Title: Harmonious Group Choreography with Trajectory-Controllable Diffusion

• Authors: Yuqin Dai, Wanlu Zhu, Ronghui Li, Zeping Ren, Xiangzheng Zhou, Xiu Li, Jun Li, Jian Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06189
• Pdf URL: https://arxiv.org/pdf/2403.06189
• Copy Paste: [[2403.06189]] Harmonious Group Choreography with Trajectory-Controllable Diffusion(https://arxiv.org/abs/2403.06189)
• Keywords: diffusion
• Abstract: Creating group choreography from music has gained attention in cultural entertainment and virtual reality, aiming to coordinate visually cohesive and diverse group movements. Despite increasing interest, recent works face challenges in achieving aesthetically appealing choreography, primarily for two key issues: multi-dancer collision and single-dancer foot slide. To address these issues, we propose a Trajectory-Controllable Diffusion (TCDiff), a novel approach that harnesses non-overlapping trajectories to facilitate coherent dance movements. Specifically, to tackle dancer collisions, we introduce a Dance-Beat Navigator capable of generating trajectories for multiple dancers based on the music, complemented by a Distance-Consistency loss to maintain appropriate spacing among trajectories within a reasonable threshold. To mitigate foot sliding, we present a Footwork Adaptor that utilizes trajectory displacement from adjacent frames to enable flexible footwork, coupled with a Relative Forward-Kinematic loss to adjust the positioning of individual dancers' root nodes and joints. Extensive experiments demonstrate that our method achieves state-of-the-art results.

Title: On depth prediction for autonomous driving using self-supervised learning

• Authors: Houssem Boulahbal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06194
• Pdf URL: https://arxiv.org/pdf/2403.06194
• Copy Paste: [[2403.06194]] On depth prediction for autonomous driving using self-supervised learning(https://arxiv.org/abs/2403.06194)
• Keywords: transformer, generative
• Abstract: Perception of the environment is a critical component for enabling autonomous driving. It provides the vehicle with the ability to comprehend its surroundings and make informed decisions. Depth prediction plays a pivotal role in this process, as it helps the understanding of the geometry and motion of the environment. This thesis focuses on the challenge of depth prediction using monocular self-supervised learning techniques. The problem is approached from a broader perspective first, exploring conditional generative adversarial networks (cGANs) as a potential technique to achieve better generalization was performed. In doing so, a fundamental contribution to the conditional GANs, the acontrario cGAN was proposed. The second contribution entails a single image-to-depth self-supervised method, proposing a solution for the rigid-scene assumption using a novel transformer-based method that outputs a pose for each dynamic object. The third significant aspect involves the introduction of a video-to-depth map forecasting approach. This method serves as an extension of self-supervised techniques to predict future depths. This involves the creation of a novel transformer model capable of predicting the future depth of a given scene. Moreover, the various limitations of the aforementioned methods were addressed and a video-to-video depth maps model was proposed. This model leverages the spatio-temporal consistency of the input and output sequence to predict a more accurate depth sequence output. These methods have significant applications in autonomous driving (AD) and advanced driver assistance systems (ADAS).

Title: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models

• Authors: Minjie Zhu, Yichen Zhu, Xin Liu, Ning Liu, Zhiyuan Xu, Chaomin Shen, Yaxin Peng, Zhicai Ou, Feifei Feng, Jian Tang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06199
• Pdf URL: https://arxiv.org/pdf/2403.06199
• Copy Paste: [[2403.06199]] A Comprehensive Overhaul of Multimodal Assistant with Small Language Models(https://arxiv.org/abs/2403.06199)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have showcased impressive skills in tasks related to visual understanding and reasoning. Yet, their widespread application faces obstacles due to the high computational demands during both the training and inference phases, restricting their use to a limited audience within the research and user communities. In this paper, we investigate the design aspects of Multimodal Small Language Models (MSLMs) and propose an efficient multimodal assistant named Mipha, which is designed to create synergy among various aspects: visual representation, language models, and optimization strategies. We show that without increasing the volume of training data, our Mipha-3B outperforms the state-of-the-art large MLLMs, especially LLaVA-1.5-13B, on multiple benchmarks. Through detailed discussion, we provide insights and guidelines for developing strong MSLMs that rival the capabilities of MLLMs. Our code is available at https://github.com/zhuyiche/Mipha.

Title: Are You Being Tracked? Discover the Power of Zero-Shot Trajectory Tracing with LLMs!

• Authors: Huanqi Yang, Sijie Ji, Rucheng Wu, Weitao Xu
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06201
• Pdf URL: https://arxiv.org/pdf/2403.06201
• Copy Paste: [[2403.06201]] Are You Being Tracked? Discover the Power of Zero-Shot Trajectory Tracing with LLMs!(https://arxiv.org/abs/2403.06201)
• Keywords: large language model
• Abstract: There is a burgeoning discussion around the capabilities of Large Language Models (LLMs) in acting as fundamental components that can be seamlessly incorporated into Artificial Intelligence of Things (AIoT) to interpret complex trajectories. This study introduces LLMTrack, a model that illustrates how LLMs can be leveraged for Zero-Shot Trajectory Recognition by employing a novel single-prompt technique that combines role-play and think step-by-step methodologies with unprocessed Inertial Measurement Unit (IMU) data. We evaluate the model using real-world datasets designed to challenge it with distinct trajectories characterized by indoor and outdoor scenarios. In both test scenarios, LLMTrack not only meets but exceeds the performance benchmarks set by traditional machine learning approaches and even contemporary state-of-the-art deep learning models, all without the requirement of training on specialized datasets. The results of our research suggest that, with strategically designed prompts, LLMs can tap into their extensive knowledge base and are well-equipped to analyze raw sensor data with remarkable effectiveness.

Title: $V_kD:$ Improving Knowledge Distillation using Orthogonal Projections

• Authors: Roy Miles, Ismail Elezi, Jiankang Deng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06213
• Pdf URL: https://arxiv.org/pdf/2403.06213
• Copy Paste: [[2403.06213]] $V_kD:$ Improving Knowledge Distillation using Orthogonal Projections(https://arxiv.org/abs/2403.06213)
• Keywords: transformer
• Abstract: Knowledge distillation is an effective method for training small and efficient deep learning models. However, the efficacy of a single method can degenerate when transferring to other tasks, modalities, or even other architectures. To address this limitation, we propose a novel constrained feature distillation method. This method is derived from a small set of core principles, which results in two emerging components: an orthogonal projection and a task-specific normalisation. Equipped with both of these components, our transformer models can outperform all previous methods on ImageNet and reach up to a 4.4% relative improvement over the previous state-of-the-art methods. To further demonstrate the generality of our method, we apply it to object detection and image generation, whereby we obtain consistent and substantial performance improvements over state-of-the-art. Code and models are publicly available: https://github.com/roymiles/vkd

Title: MoST: Motion Style Transformer between Diverse Action Contents

• Authors: Boeun Kim, Jungho Kim, Hyung Jin Chang, Jin Young Choi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06225
• Pdf URL: https://arxiv.org/pdf/2403.06225
• Copy Paste: [[2403.06225]] MoST: Motion Style Transformer between Diverse Action Contents(https://arxiv.org/abs/2403.06225)
• Keywords: transformer
• Abstract: While existing motion style transfer methods are effective between two motions with identical content, their performance significantly diminishes when transferring style between motions with different contents. This challenge lies in the lack of clear separation between content and style of a motion. To tackle this challenge, we propose a novel motion style transformer that effectively disentangles style from content and generates a plausible motion with transferred style from a source motion. Our distinctive approach to achieving the goal of disentanglement is twofold: (1) a new architecture for motion style transformer with 'part-attentive style modulator across body parts' and 'Siamese encoders that encode style and content features separately'; (2) style disentanglement loss. Our method outperforms existing methods and demonstrates exceptionally high quality, particularly in motion pairs with different contents, without the need for heuristic post-processing. Codes are available at https://github.com/Boeun-Kim/MoST.

Title: LinearAPT: An Adaptive Algorithm for the Fixed-Budget Thresholding Linear Bandit Problem

• Authors: Yun-Ang Wu, Yun-Da Tsai, Shou-De Lin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.06230
• Pdf URL: https://arxiv.org/pdf/2403.06230
• Copy Paste: [[2403.06230]] LinearAPT: An Adaptive Algorithm for the Fixed-Budget Thresholding Linear Bandit Problem(https://arxiv.org/abs/2403.06230)
• Keywords: robust
• Abstract: In this study, we delve into the Thresholding Linear Bandit (TLB) problem, a nuanced domain within stochastic Multi-Armed Bandit (MAB) problems, focusing on maximizing decision accuracy against a linearly defined threshold under resource constraints. We present LinearAPT, a novel algorithm designed for the fixed budget setting of TLB, providing an efficient solution to optimize sequential decision-making. This algorithm not only offers a theoretical upper bound for estimated loss but also showcases robust performance on both synthetic and real-world datasets. Our contributions highlight the adaptability, simplicity, and computational efficiency of LinearAPT, making it a valuable addition to the toolkit for addressing complex sequential decision-making challenges.

Title: Finding Visual Saliency in Continuous Spike Stream

• Authors: Lin Zhu, Xianzhang Chen, Xiao Wang, Hua Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06233
• Pdf URL: https://arxiv.org/pdf/2403.06233
• Copy Paste: [[2403.06233]] Finding Visual Saliency in Continuous Spike Stream(https://arxiv.org/abs/2403.06233)
• Keywords: extraction, transformer, segmentation
• Abstract: As a bio-inspired vision sensor, the spike camera emulates the operational principles of the fovea, a compact retinal region, by employing spike discharges to encode the accumulation of per-pixel luminance intensity. Leveraging its high temporal resolution and bio-inspired neuromorphic design, the spike camera holds significant promise for advancing computer vision applications. Saliency detection mimics the behavior of human beings and captures the most salient region from the scenes. In this paper, we investigate the visual saliency in the continuous spike stream for the first time. To effectively process the binary spike stream, we propose a Recurrent Spiking Transformer (RST) framework, which is based on a full spiking neural network. Our framework enables the extraction of spatio-temporal features from the continuous spatio-temporal spike stream while maintaining low power consumption. To facilitate the training and validation of our proposed model, we build a comprehensive real-world spike-based visual saliency dataset, enriched with numerous light conditions. Extensive experiments demonstrate the superior performance of our Recurrent Spiking Transformer framework in comparison to other spike neural network-based methods. Our framework exhibits a substantial margin of improvement in capturing and highlighting visual saliency in the spike stream, which not only provides a new perspective for spike-based saliency segmentation but also shows a new paradigm for full SNN-based transformer models. The code and dataset are available at \url{https://github.com/BIT-Vision/SVS}.

Title: Cooperative Classification and Rationalization for Graph Generalization

• Authors: Linan Yue, Qi Liu, Ye Liu, Weibo Gao, Fangzhou Yao, Wenfeng Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06239
• Pdf URL: https://arxiv.org/pdf/2403.06239
• Copy Paste: [[2403.06239]] Cooperative Classification and Rationalization for Graph Generalization(https://arxiv.org/abs/2403.06239)
• Keywords: robust, generative
• Abstract: Graph Neural Networks (GNNs) have achieved impressive results in graph classification tasks, but they struggle to generalize effectively when faced with out-of-distribution (OOD) data. Several approaches have been proposed to address this problem. Among them, one solution is to diversify training distributions in vanilla classification by modifying the data environment, yet accessing the environment information is complex. Besides, another promising approach involves rationalization, extracting invariant rationales for predictions. However, extracting rationales is difficult due to limited learning signals, resulting in less accurate rationales and diminished predictions. To address these challenges, in this paper, we propose a Cooperative Classification and Rationalization (C2R) method, consisting of the classification and the rationalization module. Specifically, we first assume that multiple environments are available in the classification module. Then, we introduce diverse training distributions using an environment-conditional generative network, enabling robust graph representations. Meanwhile, the rationalization module employs a separator to identify relevant rationale subgraphs while the remaining non-rationale subgraphs are de-correlated with labels. Next, we align graph representations from the classification module with rationale subgraph representations using the knowledge distillation methods, enhancing the learning signal for rationales. Finally, we infer multiple environments by gathering non-rationale representations and incorporate them into the classification module for cooperative learning. Extensive experimental results on both benchmarks and synthetic datasets demonstrate the effectiveness of C2R. Code is available at https://github.com/yuelinan/Codes-of-C2R.

Title: Text-Guided Variational Image Generation for Industrial Anomaly Detection and Segmentation

• Authors: Mingyu Lee, Jongwon Choi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06247
• Pdf URL: https://arxiv.org/pdf/2403.06247
• Copy Paste: [[2403.06247]] Text-Guided Variational Image Generation for Industrial Anomaly Detection and Segmentation(https://arxiv.org/abs/2403.06247)
• Keywords: segmentation
• Abstract: We propose a text-guided variational image generation method to address the challenge of getting clean data for anomaly detection in industrial manufacturing. Our method utilizes text information about the target object, learned from extensive text library documents, to generate non-defective data images resembling the input image. The proposed framework ensures that the generated non-defective images align with anticipated distributions derived from textual and image-based knowledge, ensuring stability and generality. Experimental results demonstrate the effectiveness of our approach, surpassing previous methods even with limited non-defective data. Our approach is validated through generalization tests across four baseline models and three distinct datasets. We present an additional analysis to enhance the effectiveness of anomaly detection models by utilizing the generated images.

Title: Editing Conceptual Knowledge for Large Language Models

• Authors: Xiaohan Wang, Shengyu Mao, Ningyu Zhang, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.DB, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06259
• Pdf URL: https://arxiv.org/pdf/2403.06259
• Copy Paste: [[2403.06259]] Editing Conceptual Knowledge for Large Language Models(https://arxiv.org/abs/2403.06259)
• Keywords: large language model
• Abstract: Recently, there has been a growing interest in knowledge editing for Large Language Models (LLMs). Current approaches and evaluations merely explore the instance-level editing, while whether LLMs possess the capability to modify concepts remains unclear. This paper pioneers the investigation of editing conceptual knowledge for LLMs, by constructing a novel benchmark dataset ConceptEdit and establishing a suite of new metrics for evaluation. The experimental results reveal that, although existing editing methods can efficiently modify concept-level definition to some extent, they also have the potential to distort the related instantial knowledge in LLMs, leading to poor performance. We anticipate this can inspire further progress in better understanding LLMs. Our project homepage is available at https://zjunlp.github.io/project/ConceptEdit.

Title: SCORE: Self-supervised Correspondence Fine-tuning for Improved Content Representations

• Authors: Amit Meghanani, Thomas Hain
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.06260
• Pdf URL: https://arxiv.org/pdf/2403.06260
• Copy Paste: [[2403.06260]] SCORE: Self-supervised Correspondence Fine-tuning for Improved Content Representations(https://arxiv.org/abs/2403.06260)
• Keywords: robust
• Abstract: There is a growing interest in cost-effective self-supervised fine-tuning (SSFT) of self-supervised learning (SSL)-based speech models to obtain task-specific representations. These task-specific representations are used for robust performance on various downstream tasks by fine-tuning on the labelled data. This work presents a cost-effective SSFT method named Self-supervised Correspondence (SCORE) fine-tuning to adapt the SSL speech representations for content-related tasks. The proposed method uses a correspondence training strategy, aiming to learn similar representations from perturbed speech and original speech. Commonly used data augmentation techniques for content-related tasks (ASR) are applied to obtain perturbed speech. SCORE fine-tuned HuBERT outperforms the vanilla HuBERT on SUPERB benchmark with only a few hours of fine-tuning (< 5 hrs) on a single GPU for automatic speech recognition, phoneme recognition, and query-by-example tasks, with relative improvements of 1.09%, 3.58%, and 12.65%, respectively. SCORE provides competitive results with the recently proposed SSFT method SPIN, using only 1/3 of the processed speech compared to SPIN.

Title: ABC-Channel: An Advanced Blockchain-based Covert Channel

• Authors: Xiaobo Ma, Pengyu Pan, Jianfeng Li, Wei Wang, Weizhi Meng, Xiaohong Guan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06261
• Pdf URL: https://arxiv.org/pdf/2403.06261
• Copy Paste: [[2403.06261]] ABC-Channel: An Advanced Blockchain-based Covert Channel(https://arxiv.org/abs/2403.06261)
• Keywords: secure, robust
• Abstract: Establishing efficient and robust covert channels is crucial for secure communication within insecure network environments. With its inherent benefits of decentralization and anonymization, blockchain has gained considerable attention in developing covert channels. To guarantee a highly secure covert channel, channel negotiation should be contactless before the communication, carrier transaction features must be indistinguishable from normal transactions during the communication, and communication identities must be untraceable after the communication. Such a full-lifecycle covert channel is indispensable to defend against a versatile adversary who intercepts two communicating parties comprehensively (e.g., on-chain and off-chain). Unfortunately, it has not been thoroughly investigated in the literature. We make the first effort to achieve a full-lifecycle covert channel, a novel blockchain-based covert channel named ABC-Channel. We tackle a series of challenges, such as off-chain contact dependency, increased masquerading difficulties as growing transaction volume, and time-evolving, communicable yet untraceable identities, to achieve contactless channel negotiation, indistinguishable transaction features, and untraceable communication identities, respectively. We develop a working prototype to validate ABC-Channel and conduct extensive tests on the Bitcoin testnet. The experimental results demonstrate that ABC-Channel achieves substantially secure covert capabilities. In comparison to existing methods, it also exhibits state-of-the-art transmission efficiency.

Title: Physics-Guided Abnormal Trajectory Gap Detection

• Authors: Arun Sharma, Shashi Shekhar
• Subjects: cs.CV, cs.AI, cs.CG, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06268
• Pdf URL: https://arxiv.org/pdf/2403.06268
• Copy Paste: [[2403.06268]] Physics-Guided Abnormal Trajectory Gap Detection(https://arxiv.org/abs/2403.06268)
• Keywords: security
• Abstract: Given trajectories with gaps (i.e., missing data), we investigate algorithms to identify abnormal gaps in trajectories which occur when a given moving object did not report its location, but other moving objects in the same geographic region periodically did. The problem is important due to its societal applications, such as improving maritime safety and regulatory enforcement for global security concerns such as illegal fishing, illegal oil transfers, and trans-shipments. The problem is challenging due to the difficulty of bounding the possible locations of the moving object during a trajectory gap, and the very high computational cost of detecting gaps in such a large volume of location data. The current literature on anomalous trajectory detection assumes linear interpolation within gaps, which may not be able to detect abnormal gaps since objects within a given region may have traveled away from their shortest path. In preliminary work, we introduced an abnormal gap measure that uses a classical space-time prism model to bound an object's possible movement during the trajectory gap and provided a scalable memoized gap detection algorithm (Memo-AGD). In this paper, we propose a Space Time-Aware Gap Detection (STAGD) approach to leverage space-time indexing and merging of trajectory gaps. We also incorporate a Dynamic Region Merge-based (DRM) approach to efficiently compute gap abnormality scores. We provide theoretical proofs that both algorithms are correct and complete and also provide analysis of asymptotic time complexity. Experimental results on synthetic and real-world maritime trajectory data show that the proposed approach substantially improves computation time over the baseline technique.

Title: FastVideoEdit: Leveraging Consistency Models for Efficient Text-to-Video Editing

• Authors: Youyuan Zhang, Xuan Ju, James J. Clark
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06269
• Pdf URL: https://arxiv.org/pdf/2403.06269
• Copy Paste: [[2403.06269]] FastVideoEdit: Leveraging Consistency Models for Efficient Text-to-Video Editing(https://arxiv.org/abs/2403.06269)
• Keywords: extraction, diffusion
• Abstract: Diffusion models have demonstrated remarkable capabilities in text-to-image and text-to-video generation, opening up possibilities for video editing based on textual input. However, the computational cost associated with sequential sampling in diffusion models poses challenges for efficient video editing. Existing approaches relying on image generation models for video editing suffer from time-consuming one-shot fine-tuning, additional condition extraction, or DDIM inversion, making real-time applications impractical. In this work, we propose FastVideoEdit, an efficient zero-shot video editing approach inspired by Consistency Models (CMs). By leveraging the self-consistency property of CMs, we eliminate the need for time-consuming inversion or additional condition extraction, reducing editing time. Our method enables direct mapping from source video to target video with strong preservation ability utilizing a special variance schedule. This results in improved speed advantages, as fewer sampling steps can be used while maintaining comparable generation quality. Experimental results validate the state-of-the-art performance and speed advantages of FastVideoEdit across evaluation metrics encompassing editing speed, temporal consistency, and text-video alignment.

Title: Refinement of MMIO Models for Improving the Coverage of Firmware Fuzzing

• Authors: Wei-Lun Huang, Kang G. Shin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06281
• Pdf URL: https://arxiv.org/pdf/2403.06281
• Copy Paste: [[2403.06281]] Refinement of MMIO Models for Improving the Coverage of Firmware Fuzzing(https://arxiv.org/abs/2403.06281)
• Keywords: security, privacy
• Abstract: Embedded systems (ESes) are now ubiquitous, collecting sensitive user data and helping the users make safety-critical decisions. Their vulnerability may thus pose a grave threat to the security and privacy of billions of ES users. Grey-box fuzzing is widely used for testing ES firmware. It usually runs the firmware in a fully emulated environment for efficient testing. In such a setting, the fuzzer cannot access peripheral hardware and hence must model the firmware's interactions with peripherals to achieve decent code coverage. The state-of-the-art (SOTA) firmware fuzzers focus on modeling the memory-mapped I/O (MMIO) of peripherals. We find that SOTA MMIO models for firmware fuzzing do not describe the MMIO reads well for retrieving a data chunk, leaving ample room for improvement of code coverage. Thus, we propose ES-Fuzz that boosts the code coverage by refining the MMIO models in use. ES-Fuzz uses a given firmware fuzzer to generate stateless and fixed MMIO models besides test cases after testing an ES firmware. ES-Fuzz then instruments a given test harness, runs it with the highest-coverage test case, and gets the execution trace. The trace guides ES-Fuzz to build stateful and adaptable MMIO models. The given fuzzer thereafter tests the firmware with the newly-built models. The alternation between the fuzzer and ES-Fuzz iteratively enhances the coverage of fuzz-testing. We have implemented ES-Fuzz upon Fuzzware and evaluated it with 21 popular ES firmware. ES-Fuzz boosts Fuzzware's coverage by up to $160\%$ in some of these firmware without lowering the coverage in the others much.

Title: Understanding and Mitigating Human-Labelling Errors in Supervised Contrastive Learning

• Authors: Zijun Long, Lipeng Zhuang, George Killick, Richard McCreadie, Gerardo Aragon Camarasa, Paul Henderson
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06289
• Pdf URL: https://arxiv.org/pdf/2403.06289
• Copy Paste: [[2403.06289]] Understanding and Mitigating Human-Labelling Errors in Supervised Contrastive Learning(https://arxiv.org/abs/2403.06289)
• Keywords: robust
• Abstract: Human-annotated vision datasets inevitably contain a fraction of human mislabelled examples. While the detrimental effects of such mislabelling on supervised learning are well-researched, their influence on Supervised Contrastive Learning (SCL) remains largely unexplored. In this paper, we show that human-labelling errors not only differ significantly from synthetic label errors, but also pose unique challenges in SCL, different to those in traditional supervised learning methods. Specifically, our results indicate they adversely impact the learning process in the ~99% of cases when they occur as false positive samples. Existing noise-mitigating methods primarily focus on synthetic label errors and tackle the unrealistic setting of very high synthetic noise rates (40-80%), but they often underperform on common image datasets due to overfitting. To address this issue, we introduce a novel SCL objective with robustness to human-labelling errors, SCL-RHE. SCL-RHE is designed to mitigate the effects of real-world mislabelled examples, typically characterized by much lower noise rates (<5%). We demonstrate that SCL-RHE consistently outperforms state-of-the-art representation learning and noise-mitigating methods across various vision benchmarks, by offering improved resilience against human-labelling errors.

Title: Transformer based Multitask Learning for Image Captioning and Object Detection

• Authors: Debolena Basak, P.K. Srijith, Maunendra Sankar Desarkar
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06292
• Pdf URL: https://arxiv.org/pdf/2403.06292
• Copy Paste: [[2403.06292]] Transformer based Multitask Learning for Image Captioning and Object Detection(https://arxiv.org/abs/2403.06292)
• Keywords: transformer
• Abstract: In several real-world scenarios like autonomous navigation and mobility, to obtain a better visual understanding of the surroundings, image captioning and object detection play a crucial role. This work introduces a novel multitask learning framework that combines image captioning and object detection into a joint model. We propose TICOD, Transformer-based Image Captioning and Object detection model for jointly training both tasks by combining the losses obtained from image captioning and object detection networks. By leveraging joint training, the model benefits from the complementary information shared between the two tasks, leading to improved performance for image captioning. Our approach utilizes a transformer-based architecture that enables end-to-end network integration for image captioning and object detection and performs both tasks jointly. We evaluate the effectiveness of our approach through comprehensive experiments on the MS-COCO dataset. Our model outperforms the baselines from image captioning literature by achieving a 3.65% improvement in BERTScore.

Title: Analysis of Total Variation Minimization for Clustered Federated Learning

• Authors: A. Jung
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06298
• Pdf URL: https://arxiv.org/pdf/2403.06298
• Copy Paste: [[2403.06298]] Analysis of Total Variation Minimization for Clustered Federated Learning(https://arxiv.org/abs/2403.06298)
• Keywords: robust, federate
• Abstract: A key challenge in federated learning applications is the statistical heterogeneity of local datasets. Clustered federated learning addresses this challenge by identifying clusters of local datasets that are approximately homogeneous. One recent approach to clustered federated learning is generalized total variation minimization (GTVMin). This approach requires a similarity graph which can be obtained by domain expertise or in a data-driven fashion via graph learning techniques. Under a widely applicable clustering assumption, we derive an upper bound the deviation between GTVMin solutions and their cluster-wise averages. This bound provides valuable insights into the effectiveness and robustness of GTVMin in addressing statistical heterogeneity within federated learning environments.

Title: LIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognition

• Authors: Xiaomeng Zhu, Robert Frank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06301
• Pdf URL: https://arxiv.org/pdf/2403.06301
• Copy Paste: [[2403.06301]] LIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognition(https://arxiv.org/abs/2403.06301)
• Keywords: large language model
• Abstract: Discourse Entity (DE) recognition is the task of identifying novel and known entities introduced within a text. While previous work has found that large language models have basic, if imperfect, DE recognition abilities (Schuster and Linzen, 2022), it remains largely unassessed which of the fundamental semantic properties that govern the introduction and subsequent reference to DEs they have knowledge of. We propose the Linguistically-Informed Evaluation for Discourse Entity Recognition (LIEDER) dataset that allows for a detailed examination of language models' knowledge of four crucial semantic properties: existence, uniqueness, plurality, and novelty. We find evidence that state-of-the-art large language models exhibit sensitivity to all of these properties except novelty, which demonstrates that they have yet to reach human-level language understanding abilities.

Title: An End-to-End Deep Learning Generative Framework for Refinable Shape Matching and Generation

• Authors: Soodeh Kalaie, Andy Bulpitt, Alejandro F. Frangi, Ali Gooya
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06317
• Pdf URL: https://arxiv.org/pdf/2403.06317
• Copy Paste: [[2403.06317]] An End-to-End Deep Learning Generative Framework for Refinable Shape Matching and Generation(https://arxiv.org/abs/2403.06317)
• Keywords: generative
• Abstract: Generative modelling for shapes is a prerequisite for In-Silico Clinical Trials (ISCTs), which aim to cost-effectively validate medical device interventions using synthetic anatomical shapes, often represented as 3D surface meshes. However, constructing AI models to generate shapes closely resembling the real mesh samples is challenging due to variable vertex counts, connectivities, and the lack of dense vertex-wise correspondences across the training data. Employing graph representations for meshes, we develop a novel unsupervised geometric deep-learning model to establish refinable shape correspondences in a latent space, construct a population-derived atlas and generate realistic synthetic shapes. We additionally extend our proposed base model to a joint shape generative-clustering multi-atlas framework to incorporate further variability and preserve more details in the generated shapes. Experimental results using liver and left-ventricular models demonstrate the approach's applicability to computational medicine, highlighting its suitability for ISCTs through a comparative analysis.

Title: Fake or Compromised? Making Sense of Malicious Clients in Federated Learning

• Authors: Hamid Mozaffari, Sunav Choudhary, Amir Houmansadr
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06319
• Pdf URL: https://arxiv.org/pdf/2403.06319
• Copy Paste: [[2403.06319]] Fake or Compromised? Making Sense of Malicious Clients in Federated Learning(https://arxiv.org/abs/2403.06319)
• Keywords: security, attack, robust, federate, generative
• Abstract: Federated learning (FL) is a distributed machine learning paradigm that enables training models on decentralized data. The field of FL security against poisoning attacks is plagued with confusion due to the proliferation of research that makes different assumptions about the capabilities of adversaries and the adversary models they operate under. Our work aims to clarify this confusion by presenting a comprehensive analysis of the various poisoning attacks and defensive aggregation rules (AGRs) proposed in the literature, and connecting them under a common framework. To connect existing adversary models, we present a hybrid adversary model, which lies in the middle of the spectrum of adversaries, where the adversary compromises a few clients, trains a generative (e.g., DDPM) model with their compromised samples, and generates new synthetic data to solve an optimization for a stronger (e.g., cheaper, more practical) attack against different robust aggregation rules. By presenting the spectrum of FL adversaries, we aim to provide practitioners and researchers with a clear understanding of the different types of threats they need to consider when designing FL systems, and identify areas where further research is needed.

Title: Leveraging Computer Vision in the Intensive Care Unit (ICU) for Examining Visitation and Mobility

• Authors: Scott Siegel, Jiaqing Zhang, Sabyasachi Bandyopadhyay, Subhash Nerella, Brandon Silva, Tezcan Baslanti, Azra Bihorac, Parisa Rashidi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06322
• Pdf URL: https://arxiv.org/pdf/2403.06322
• Copy Paste: [[2403.06322]] Leveraging Computer Vision in the Intensive Care Unit (ICU) for Examining Visitation and Mobility(https://arxiv.org/abs/2403.06322)
• Keywords: defense
• Abstract: Despite the importance of closely monitoring patients in the Intensive Care Unit (ICU), many aspects are still assessed in a limited manner due to the time constraints imposed on healthcare providers. For example, although excessive visitations during rest hours can potentially exacerbate the risk of circadian rhythm disruption and delirium, it is not captured in the ICU. Likewise, while mobility can be an important indicator of recovery or deterioration in ICU patients, it is only captured sporadically or not captured at all. In the past few years, the computer vision field has found application in many domains by reducing the human burden. Using computer vision systems in the ICU can also potentially enable non-existing assessments or enhance the frequency and accuracy of existing assessments while reducing the staff workload. In this study, we leverage a state-of-the-art noninvasive computer vision system based on depth imaging to characterize ICU visitations and patients' mobility. We then examine the relationship between visitation and several patient outcomes, such as pain, acuity, and delirium. We found an association between deteriorating patient acuity and the incidence of delirium with increased visitations. In contrast, self-reported pain, reported using the Defense and Veteran Pain Rating Scale (DVPRS), was correlated with decreased visitations. Our findings highlight the feasibility and potential of using noninvasive autonomous systems to monitor ICU patients.

Title: Transferable Reinforcement Learning via Generalized Occupancy Models

• Authors: Chuning Zhu, Xinqi Wang, Tyler Han, Simon S. Du, Abhishek Gupta
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06328
• Pdf URL: https://arxiv.org/pdf/2403.06328
• Copy Paste: [[2403.06328]] Transferable Reinforcement Learning via Generalized Occupancy Models(https://arxiv.org/abs/2403.06328)
• Keywords: diffusion
• Abstract: Intelligent agents must be generalists - showing the ability to quickly adapt and generalize to varying tasks. Within the framework of reinforcement learning (RL), model-based RL algorithms learn a task-agnostic dynamics model of the world, in principle allowing them to generalize to arbitrary rewards. However, one-step models naturally suffer from compounding errors, making them ineffective for problems with long horizons and large state spaces. In this work, we propose a novel class of models - generalized occupancy models (GOMs) - that retain the generality of model-based RL while avoiding compounding error. The key idea behind GOMs is to model the distribution of all possible long-term outcomes from a given state under the coverage of a stationary dataset, along with a policy that realizes a particular outcome from the given state. These models can then quickly be used to select the optimal action for arbitrary new tasks, without having to redo policy optimization. By directly modeling long-term outcomes, GOMs avoid compounding error while retaining generality across arbitrary reward functions. We provide a practical instantiation of GOMs using diffusion models and show its efficacy as a new class of transferable models, both theoretically and empirically across a variety of simulated robotics problems. Videos and code at https://weirdlabuw.github.io/gom/.

Title: Practically adaptable CPABE based Health-Records sharing framework

• Authors: Raza Imam, Faisal Anwer
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06347
• Pdf URL: https://arxiv.org/pdf/2403.06347
• Copy Paste: [[2403.06347]] Practically adaptable CPABE based Health-Records sharing framework(https://arxiv.org/abs/2403.06347)
• Keywords: privacy
• Abstract: With recent elevated adaptation of cloud services in almost every major public sector, the health sector emerges as a vulnerable segment, particularly in data exchange of sensitive Health records, as determining the retention, exchange, and efficient use of patient records without jeopardizing patient privacy, particularly on mobile-applications remains an area to expand. In the existing scenarios of cloud-mobile services, several vulnerabilities can be found including trapping of data within a single cloud-service-provider and loss of resource control being the significant ones. In this study, we have suggested a CPABE and OAuth2.0 based framework for efficient access-control and authorization respectively to improve the practicality of EHR sharing across a single client-application. In addition to solving issues like practicality, data entrapment, and resource control loss, the suggested framework also aims to provide two significant functionalities simultaneously, the specific operation of client application itself, and straightforward access of data to institutions, governments, and organizations seeking delicate EHRs. Our implementation of the suggested framework along with its analytical comparison signifies its potential in terms of efficient performance and minimal latency as this study would have a considerable impact on the recent literature as it intends to bridge the pragmatic deficit in CPABE-based EHR services.

Title: Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videos

• Authors: Mi Luo, Zihui Xue, Alex Dimakis, Kristen Grauman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06351
• Pdf URL: https://arxiv.org/pdf/2403.06351
• Copy Paste: [[2403.06351]] Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videos(https://arxiv.org/abs/2403.06351)
• Keywords: diffusion, generative
• Abstract: We investigate exocentric-to-egocentric cross-view translation, which aims to generate a first-person (egocentric) view of an actor based on a video recording that captures the actor from a third-person (exocentric) perspective. To this end, we propose a generative framework called Exo2Ego that decouples the translation process into two stages: high-level structure transformation, which explicitly encourages cross-view correspondence between exocentric and egocentric views, and a diffusion-based pixel-level hallucination, which incorporates a hand layout prior to enhance the fidelity of the generated egocentric view. To pave the way for future advancements in this field, we curate a comprehensive exo-to-ego cross-view translation benchmark. It consists of a diverse collection of synchronized ego-exo tabletop activity video pairs sourced from three public datasets: H2O, Aria Pilot, and Assembly101. The experimental results validate that Exo2Ego delivers photorealistic video results with clear hand manipulation details and outperforms several baselines in terms of both synthesis quality and generalization ability to new actions.

Title: Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages

• Authors: Michael Andersland
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06354
• Pdf URL: https://arxiv.org/pdf/2403.06354
• Copy Paste: [[2403.06354]] Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages(https://arxiv.org/abs/2403.06354)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) like GPT-4 and LLaMA have shown incredible proficiency at natural language processing tasks and have even begun to excel at tasks across other modalities such as vision and audio. Despite their success, LLMs often struggle to perform well on low-resource languages because there is so little training data available. This shortcoming is especially prevalent with open source models. In this work, we explore training LLaMA-2 to speak Amharic, a language which is spoken by over 50 million people world wide, but has orders of magnitude less data available than languages like English. We employ methods previously used for training LLMs on other languages with data scarcity, and use open source translation models to perform data augmentation and grow our dataset from millions of tokens to billions. We further enhance the capabilities of our model by connecting an image encoder and training on a translated visual instruction tuning dataset in the same manner as LLaVA, resulting in a multimodal Amharic LLM that can understand images along with text. We introduce an Amharic version of a popular benchmarking dataset to evaluate our work. Our models and dataset are open sourced and available on GitHub.

Title: See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI

• Authors: Yulong Liu, Yongqiang Ma, Guibo Zhu, Haodong Jing, Nanning Zheng
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.06361
• Pdf URL: https://arxiv.org/pdf/2403.06361
• Copy Paste: [[2403.06361]] See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI(https://arxiv.org/abs/2403.06361)
• Keywords: robust
• Abstract: Deciphering visual content from functional Magnetic Resonance Imaging (fMRI) helps illuminate the human vision system. However, the scarcity of fMRI data and noise hamper brain decoding model performance. Previous approaches primarily employ subject-specific models, sensitive to training sample size. In this paper, we explore a straightforward but overlooked solution to address data scarcity. We propose shallow subject-specific adapters to map cross-subject fMRI data into unified representations. Subsequently, a shared deeper decoding model decodes cross-subject features into the target feature space. During training, we leverage both visual and textual supervision for multi-modal brain decoding. Our model integrates a high-level perception decoding pipeline and a pixel-wise reconstruction pipeline guided by high-level perceptions, simulating bottom-up and top-down processes in neuroscience. Empirical experiments demonstrate robust neural representation learning across subjects for both pipelines. Moreover, merging high-level and low-level information improves both low-level and high-level reconstruction metrics. Additionally, we successfully transfer learned general knowledge to new subjects by training new adapters with limited training data. Compared to previous state-of-the-art methods, notably pre-training-based methods (Mind-Vis and fMRI-PTE), our approach achieves comparable or superior results across diverse tasks, showing promise as an alternative method for cross-subject fMRI data pre-training. Our code and pre-trained weights will be publicly released at https://github.com/YulongBonjour/See_Through_Their_Minds.

Title: Say Anything with Any Style

• Authors: Shuai Tan, Bin Ji, Yu Ding, Ye Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06363
• Pdf URL: https://arxiv.org/pdf/2403.06363
• Copy Paste: [[2403.06363]] Say Anything with Any Style(https://arxiv.org/abs/2403.06363)
• Keywords: robust, extraction, generative
• Abstract: Generating stylized talking head with diverse head motions is crucial for achieving natural-looking videos but still remains challenging. Previous works either adopt a regressive method to capture the speaking style, resulting in a coarse style that is averaged across all training data, or employ a universal network to synthesize videos with different styles which causes suboptimal performance. To address these, we propose a novel dynamic-weight method, namely Say Anything withAny Style (SAAS), which queries the discrete style representation via a generative model with a learned style codebook. Specifically, we develop a multi-task VQ-VAE that incorporates three closely related tasks to learn a style codebook as a prior for style extraction. This discrete prior, along with the generative model, enhances the precision and robustness when extracting the speaking styles of the given style clips. By utilizing the extracted style, a residual architecture comprising a canonical branch and style-specific branch is employed to predict the mouth shapes conditioned on any driving audio while transferring the speaking style from the source to any desired one. To adapt to different speaking styles, we steer clear of employing a universal network by exploring an elaborate HyperStyle to produce the style-specific weights offset for the style branch. Furthermore, we construct a pose generator and a pose codebook to store the quantized pose representation, allowing us to sample diverse head motions aligned with the audio and the extracted style. Experiments demonstrate that our approach surpasses state-of-theart methods in terms of both lip-synchronization and stylized expression. Besides, we extend our SAAS to video-driven style editing field and achieve satisfactory performance.

Title: Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style

• Authors: Shuai Tan, Bin Ji, Ye Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06365
• Pdf URL: https://arxiv.org/pdf/2403.06365
• Copy Paste: [[2403.06365]] Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style(https://arxiv.org/abs/2403.06365)
• Keywords: diffusion
• Abstract: Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audiovisual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.

Title: Eliminating Warping Shakes for Unsupervised Online Video Stitching

• Authors: Lang Nie, Chunyu Lin, Kang Liao, Yun Zhang, Shuaicheng Liu, Yao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06378
• Pdf URL: https://arxiv.org/pdf/2403.06378
• Copy Paste: [[2403.06378]] Eliminating Warping Shakes for Unsupervised Online Video Stitching(https://arxiv.org/abs/2403.06378)
• Keywords: robust
• Abstract: In this paper, we retarget video stitching to an emerging issue, named warping shake, when extending image stitching to video stitching. It unveils the temporal instability of warped content in non-overlapping regions, despite image stitching having endeavored to preserve the natural structures. Therefore, in most cases, even if the input videos to be stitched are stable, the stitched video will inevitably cause undesired warping shakes and affect the visual experience. To eliminate the shakes, we propose StabStitch to simultaneously realize video stitching and video stabilization in a unified unsupervised learning framework. Starting from the camera paths in video stabilization, we first derive the expression of stitching trajectories in video stitching by elaborately integrating spatial and temporal warps. Then a warp smoothing model is presented to optimize them with a comprehensive consideration regarding content alignment, trajectory smoothness, spatial consistency, and online collaboration. To establish an evaluation benchmark and train the learning framework, we build a video stitching dataset with a rich diversity in camera motions and scenes. Compared with existing stitching solutions, StabStitch exhibits significant superiority in scene robustness and inference speed in addition to stitching and stabilization performance, contributing to a robust and real-time online video stitching system. The code and dataset will be available at https://github.com/nie-lang/StabStitch.

Title: Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models

• Authors: Yang Zhang, Teoh Tze Tzun, Lim Wei Hern, Tiviatis Sim, Kenji Kawaguchi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06381
• Pdf URL: https://arxiv.org/pdf/2403.06381
• Copy Paste: [[2403.06381]] Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models(https://arxiv.org/abs/2403.06381)
• Keywords: diffusion
• Abstract: Recent advancements in diffusion models have notably improved the perceptual quality of generated images in text-to-image synthesis tasks. However, diffusion models often struggle to produce images that accurately reflect the intended semantics of the associated text prompts. We examine cross-attention layers in diffusion models and observe a propensity for these layers to disproportionately focus on certain tokens during the generation process, thereby undermining semantic fidelity. To address the issue of dominant attention, we introduce attention regulation, a computation-efficient on-the-fly optimization approach at inference time to align attention maps with the input text prompt. Notably, our method requires no additional training or fine-tuning and serves as a plug-in module on a model. Hence, the generation capacity of the original model is fully preserved. We compare our approach with alternative approaches across various datasets, evaluation metrics, and diffusion models. Experiment results show that our method consistently outperforms other baselines, yielding images that more faithfully reflect the desired concepts with reduced computation overhead. Code is available at https://github.com/YaNgZhAnG-V5/attention_regulation.

Title: A Zero Trust Framework for Realization and Defense Against Generative AI Attacks in Power Grid

• Authors: Md. Shirajum Munir, Sravanthi Proddatoori, Manjushree Muralidhara, Walid Saad, Zhu Han, Sachin Shetty
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06388
• Pdf URL: https://arxiv.org/pdf/2403.06388
• Copy Paste: [[2403.06388]] A Zero Trust Framework for Realization and Defense Against Generative AI Attacks in Power Grid(https://arxiv.org/abs/2403.06388)
• Keywords: protect, defense, attack, generative
• Abstract: Understanding the potential of generative AI (GenAI)-based attacks on the power grid is a fundamental challenge that must be addressed in order to protect the power grid by realizing and validating risk in new attack vectors. In this paper, a novel zero trust framework for a power grid supply chain (PGSC) is proposed. This framework facilitates early detection of potential GenAI-driven attack vectors (e.g., replay and protocol-type attacks), assessment of tail risk-based stability measures, and mitigation of such threats. First, a new zero trust system model of PGSC is designed and formulated as a zero-trust problem that seeks to guarantee for a stable PGSC by realizing and defending against GenAI-driven cyber attacks. Second, in which a domain-specific generative adversarial networks (GAN)-based attack generation mechanism is developed to create a new vulnerability cyberspace for further understanding that threat. Third, tail-based risk realization metrics are developed and implemented for quantifying the extreme risk of a potential attack while leveraging a trust measurement approach for continuous validation. Fourth, an ensemble learning-based bootstrap aggregation scheme is devised to detect the attacks that are generating synthetic identities with convincing user and distributed energy resources device profiles. Experimental results show the efficacy of the proposed zero trust framework that achieves an accuracy of 95.7% on attack vector generation, a risk measure of 9.61% for a 95% stable PGSC, and a 99% confidence in defense against GenAI-driven attack.

Title: Towards Robust Out-of-Distribution Generalization Bounds via Sharpness

• Authors: Yingtian Zou, Kenji Kawaguchi, Yingnan Liu, Jiashuo Liu, Mong-Li Lee, Wynne Hsu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06392
• Pdf URL: https://arxiv.org/pdf/2403.06392
• Copy Paste: [[2403.06392]] Towards Robust Out-of-Distribution Generalization Bounds via Sharpness(https://arxiv.org/abs/2403.06392)
• Keywords: robust
• Abstract: Generalizing to out-of-distribution (OOD) data or unseen domain, termed OOD generalization, still lacks appropriate theoretical guarantees. Canonical OOD bounds focus on different distance measurements between source and target domains but fail to consider the optimization property of the learned model. As empirically shown in recent work, the sharpness of learned minima influences OOD generalization. To bridge this gap between optimization and OOD generalization, we study the effect of sharpness on how a model tolerates data change in domain shift which is usually captured by "robustness" in generalization. In this paper, we give a rigorous connection between sharpness and robustness, which gives better OOD guarantees for robust algorithms. It also provides a theoretical backing for "flat minima leads to better OOD generalization". Overall, we propose a sharpness-based OOD generalization bound by taking robustness into consideration, resulting in a tighter bound than non-robust guarantees. Our findings are supported by the experiments on a ridge regression model, as well as the experiments on deep learning classification tasks.

Title: FSViewFusion: Few-Shots View Generation of Novel Objects

• Authors: Rukhshanda Hussain, Hui Xian Grace Lim, Borchun Chen, Mubarak Shah, Ser Nam Lim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06394
• Pdf URL: https://arxiv.org/pdf/2403.06394
• Copy Paste: [[2403.06394]] FSViewFusion: Few-Shots View Generation of Novel Objects(https://arxiv.org/abs/2403.06394)
• Keywords: diffusion
• Abstract: Novel view synthesis has observed tremendous developments since the arrival of NeRFs. However, Nerf models overfit on a single scene, lacking generalization to out of distribution objects. Recently, diffusion models have exhibited remarkable performance on introducing generalization in view synthesis. Inspired by these advancements, we explore the capabilities of a pretrained stable diffusion model for view synthesis without explicit 3D priors. Specifically, we base our method on a personalized text to image model, Dreambooth, given its strong ability to adapt to specific novel objects with a few shots. Our research reveals two interesting findings. First, we observe that Dreambooth can learn the high level concept of a view, compared to arguably more complex strategies which involve finetuning diffusions on large amounts of multi-view data. Second, we establish that the concept of a view can be disentangled and transferred to a novel object irrespective of the original object's identify from which the views are learnt. Motivated by this, we introduce a learning strategy, FSViewFusion, which inherits a specific view through only one image sample of a single scene, and transfers the knowledge to a novel object, learnt from few shots, using low rank adapters. Through extensive experiments we demonstrate that our method, albeit simple, is efficient in generating reliable view samples for in the wild images. Code and models will be released.

Title: DivCon: Divide and Conquer for Progressive Text-to-Image Generation

• Authors: Yuhao Jia, Wenhan Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06400
• Pdf URL: https://arxiv.org/pdf/2403.06400
• Copy Paste: [[2403.06400]] DivCon: Divide and Conquer for Progressive Text-to-Image Generation(https://arxiv.org/abs/2403.06400)
• Keywords: diffusion, large language model
• Abstract: Diffusion-driven text-to-image (T2I) generation has achieved remarkable advancements. To further improve T2I models' capability in numerical and spatial reasoning, the layout is employed as an intermedium to bridge large language models and layout-based diffusion models. However, these methods still struggle with generating images from textural prompts with multiple objects and complicated spatial relationships. To tackle this challenge, we introduce a divide-and-conquer approach which decouples the T2I generation task into simple subtasks. Our approach divides the layout prediction stage into numerical \& spatial reasoning and bounding box prediction. Then, the layout-to-image generation stage is conducted in an iterative manner to reconstruct objects from easy ones to difficult ones. We conduct experiments on the HRS and NSR-1K benchmarks and our approach outperforms previous state-of-the-art models with notable margins. In addition, visual results demonstrate that our approach significantly improves the controllability and consistency in generating multiple objects from complex textural prompts.

Title: Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation

• Authors: Peng Zhang, Ting Wu, Jinsheng Sun, Weiqing Li, Zhiyong Su
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06401
• Pdf URL: https://arxiv.org/pdf/2403.06401
• Copy Paste: [[2403.06401]] Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation(https://arxiv.org/abs/2403.06401)
• Keywords: segmentation
• Abstract: Existing interactive point cloud segmentation approaches primarily focus on the object segmentation, which aim to determine which points belong to the object of interest guided by user interactions. This paper concentrates on an unexplored yet meaningful task, i.e., interactive point cloud semantic segmentation, which assigns high-quality semantic labels to all points in a scene with user corrective clicks. Concretely, we presents the first interactive framework for point cloud semantic segmentation, named InterPCSeg, which seamlessly integrates with off-the-shelf semantic segmentation networks without offline re-training, enabling it to run in an on-the-fly manner. To achieve online refinement, we treat user interactions as sparse training examples during the test-time. To address the instability caused by the sparse supervision, we design a stabilization energy to regulate the test-time training process. For objective and reproducible evaluation, we develop an interaction simulation scheme tailored for the interactive point cloud semantic segmentation task. We evaluate our framework on the S3DIS and ScanNet datasets with off-the-shelf segmentation networks, incorporating interactions from both the proposed interaction simulator and real users. Quantitative and qualitative experimental results demonstrate the efficacy of our framework in refining the semantic segmentation results with user interactions. The source code will be publicly available.

Title: 'One size doesn't fit all': Learning how many Examples to use for In-Context Learning for Improved Text Classification

• Authors: Manish Chandra, Debasis Ganguly, Yiwen Li, Iadh Ounis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06402
• Pdf URL: https://arxiv.org/pdf/2403.06402
• Copy Paste: [[2403.06402]] 'One size doesn't fit all': Learning how many Examples to use for In-Context Learning for Improved Text Classification(https://arxiv.org/abs/2403.06402)
• Keywords: generative
• Abstract: Predictive models in natural language processing (NLP) have evolved from training models from scratch to fine-tuning pre-trained models with labelled data. An extreme form of this fine-tuning involves in-context learning (ICL), where the output of a pre-trained generative model (frozen decoder parameters) is controlled only with variations in the input strings (called instructions or prompts). An important component of ICL is the use of a small number of labelled data instances as examples in the prompt. While existing work uses a static number of examples during inference for each data instance, in this paper we propose a novel methodology of dynamically adapting the number of examples as per the data. This is analogous to the use of a variable-sized neighborhood in k-nearest neighbors (k-NN) classifier. In our proposed workflow of adaptive ICL (AICL), the number of demonstrations to employ during the inference on a particular data instance is predicted by the Softmax posteriors of a classifier. The parameters of this classifier are fitted on the optimal number of examples in ICL required to correctly infer the label of each instance in the training set with the hypothesis that a test instance that is similar to a training instance should use the same (or a closely matching) number of few-shot examples. Our experiments show that our AICL method results in improvement in text classification task on several standard datasets.

Title: PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via Foundation Models

• Authors: Qingdong He, Jinlong Peng, Zhengkai Jiang, Xiaobin Hu, Jiangning Zhang, Qiang Nie, Yabiao Wang, Chengjie Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06403
• Pdf URL: https://arxiv.org/pdf/2403.06403
• Copy Paste: [[2403.06403]] PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via Foundation Models(https://arxiv.org/abs/2403.06403)
• Keywords: segmentation
• Abstract: Recent success of vision foundation models have shown promising performance for the 2D perception tasks. However, it is difficult to train a 3D foundation network directly due to the limited dataset and it remains under explored whether existing foundation models can be lifted to 3D space seamlessly. In this paper, we present PointSeg, a novel training-free paradigm that leverages off-the-shelf vision foundation models to address 3D scene perception tasks. PointSeg can segment anything in 3D scene by acquiring accurate 3D prompts to align their corresponding pixels across frames. Concretely, we design a two-branch prompts learning structure to construct the 3D point-box prompts pairs, combining with the bidirectional matching strategy for accurate point and proposal prompts generation. Then, we perform the iterative post-refinement adaptively when cooperated with different vision foundation models. Moreover, we design a affinity-aware merging algorithm to improve the final ensemble masks. PointSeg demonstrates impressive segmentation performance across various datasets, all without training. Specifically, our approach significantly surpasses the state-of-the-art specialist model by 13.4$\%$, 11.3$\%$, and 12$\%$ mAP on ScanNet, ScanNet++, and KITTI-360 datasets, respectively. On top of that, PointSeg can incorporate with various segmentation models and even surpasses the supervised methods.

Title: Comparison of No-Reference Image Quality Models via MAP Estimation in Diffusion Latents

• Authors: Weixia Zhang, Dingquan Li, Guangtao Zhai, Xiaokang Yang, Kede Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06406
• Pdf URL: https://arxiv.org/pdf/2403.06406
• Copy Paste: [[2403.06406]] Comparison of No-Reference Image Quality Models via MAP Estimation in Diffusion Latents(https://arxiv.org/abs/2403.06406)
• Keywords: diffusion
• Abstract: Contemporary no-reference image quality assessment (NR-IQA) models can effectively quantify the perceived image quality, with high correlations between model predictions and human perceptual scores on fixed test sets. However, little progress has been made in comparing NR-IQA models from a perceptual optimization perspective. Here, for the first time, we demonstrate that NR-IQA models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement. This is achieved by taking the gradients in differentiable and bijective diffusion latents rather than in the raw pixel domain. Different NR-IQA models are likely to induce different enhanced images, which are ultimately subject to psychophysical testing. This leads to a new computational method for comparing NR-IQA models within the analysis-by-synthesis framework. Compared to conventional correlation-based metrics, our method provides complementary insights into the relative strengths and weaknesses of the competing NR-IQA models in the context of perceptual optimization.

Title: Can LLMs' Tuning Methods Work in Medical Multimodal Domain?

• Authors: Jiawei Chen, Yue Jiang, Dingkang Yang, Mingcheng Li, Jinjie Wei, Ziyun Qian, Lihua Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06407
• Pdf URL: https://arxiv.org/pdf/2403.06407
• Copy Paste: [[2403.06407]] Can LLMs' Tuning Methods Work in Medical Multimodal Domain?(https://arxiv.org/abs/2403.06407)
• Keywords: large language model
• Abstract: While large language models (LLMs) excel in world knowledge understanding, adapting them to specific subfields requires precise adjustments. Due to the model's vast scale, traditional global fine-tuning methods for large models can be computationally expensive and impact generalization. To address this challenge, a range of innovative Parameters-Efficient Fine-Tuning (PEFT) methods have emerged and achieved remarkable success in both LLMs and Large Vision-Language Models (LVLMs). In the medical domain, fine-tuning a medical Vision-Language Pretrained (VLP) model is essential for adapting it to specific tasks. Can the fine-tuning methods for large models be transferred to the medical field to enhance transfer learning efficiency? In this paper, we delve into the fine-tuning methods of LLMs and conduct extensive experiments to investigate the impact of fine-tuning methods for large models on existing multimodal models in the medical domain from the training data level and the model structure level. We show the different impacts of fine-tuning methods for large models on medical VLMs and develop the most efficient ways to fine-tune medical VLP models. We hope this research can guide medical domain researchers in optimizing VLMs' training costs, fostering the broader application of VLMs in healthcare fields. Code and dataset will be released upon acceptance.

Title: What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation

• Authors: Zhuocheng Gong, Jiahao Liu, Jingang Wang, Xunliang Cai, Dongyan Zhao, Rui Yan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06408
• Pdf URL: https://arxiv.org/pdf/2403.06408
• Copy Paste: [[2403.06408]] What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation(https://arxiv.org/abs/2403.06408)
• Keywords: robust, large language model
• Abstract: Quantization has emerged as a promising technique for improving the memory and computational efficiency of large language models (LLMs). Though the trade-off between performance and efficiency is well-known, there is still much to be learned about the relationship between quantization and LLM performance. To shed light on this relationship, we propose a new perspective on quantization, viewing it as perturbations added to the weights and activations of LLMs. We call this approach "the lens of perturbation". Using this lens, we conduct experiments with various artificial perturbations to explore their impact on LLM performance. Our findings reveal several connections between the properties of perturbations and LLM performance, providing insights into the failure cases of uniform quantization and suggesting potential solutions to improve the robustness of LLM quantization. To demonstrate the significance of our findings, we implement a simple non-uniform quantization approach based on our insights. Our experiments show that this approach achieves minimal performance degradation on both 4-bit weight quantization and 8-bit quantization for weights and activations. These results validate the correctness of our approach and highlight its potential to improve the efficiency of LLMs without sacrificing performance.

Title: CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean

• Authors: Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, Alice Oh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06412
• Pdf URL: https://arxiv.org/pdf/2403.06412
• Copy Paste: [[2403.06412]] CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean(https://arxiv.org/abs/2403.06412)
• Keywords: large language model
• Abstract: Despite the rapid development of large language models (LLMs) for the Korean language, there remains an obvious lack of benchmark datasets that test the requisite Korean cultural and linguistic knowledge. Because many existing Korean benchmark datasets are derived from the English counterparts through translation, they often overlook the different cultural contexts. For the few benchmark datasets that are sourced from Korean data capturing cultural knowledge, only narrow tasks such as bias and hate speech detection are offered. To address this gap, we introduce a benchmark of Cultural and Linguistic Intelligence in Korean (CLIcK), a dataset comprising 1,995 QA pairs. CLIcK sources its data from official Korean exams and textbooks, partitioning the questions into eleven categories under the two main categories of language and culture. For each instance in CLIcK, we provide fine-grained annotation of which cultural and linguistic knowledge is required to answer the question correctly. Using CLIcK, we test 13 language models to assess their performance. Our evaluation uncovers insights into their performances across the categories, as well as the diverse factors affecting their comprehension. CLIcK offers the first large-scale comprehensive Korean-centric analysis of LLMs' proficiency in Korean culture and language.

Title: Evolving Knowledge Distillation with Large Language Models and Active Learning

• Authors: Chengyuan Liu, Yangyang Kang, Fubang Zhao, Kun Kuang, Zhuoren Jiang, Changlong Sun, Fei Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06414
• Pdf URL: https://arxiv.org/pdf/2403.06414
• Copy Paste: [[2403.06414]] Evolving Knowledge Distillation with Large Language Models and Active Learning(https://arxiv.org/abs/2403.06414)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across various NLP tasks. However, their computational costs are prohibitively high. To address this issue, previous research has attempted to distill the knowledge of LLMs into smaller models by generating annotated data. Nonetheless, these works have mainly focused on the direct use of LLMs for text generation and labeling, without fully exploring their potential to comprehend the target task and acquire valuable knowledge. In this paper, we propose EvoKD: Evolving Knowledge Distillation, which leverages the concept of active learning to interactively enhance the process of data generation using large language models, simultaneously improving the task capabilities of small domain model (student model). Different from previous work, we actively analyze the student model's weaknesses, and then synthesize labeled samples based on the analysis. In addition, we provide iterative feedback to the LLMs regarding the student model's performance to continuously construct diversified and challenging samples. Experiments and analysis on different NLP tasks, namely, text classification and named entity recognition show the effectiveness of EvoKD.

Title: Causal Multi-Label Feature Selection in Federated Setting

• Authors: Yukun Song, Dayuan Cao, Jiali Miao, Shuai Yang, Kui Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06419
• Pdf URL: https://arxiv.org/pdf/2403.06419
• Copy Paste: [[2403.06419]] Causal Multi-Label Feature Selection in Federated Setting(https://arxiv.org/abs/2403.06419)
• Keywords: privacy, federate
• Abstract: Multi-label feature selection serves as an effective mean for dealing with high-dimensional multi-label data. To achieve satisfactory performance, existing methods for multi-label feature selection often require the centralization of substantial data from multiple sources. However, in Federated setting, centralizing data from all sources and merging them into a single dataset is not feasible. To tackle this issue, in this paper, we study a challenging problem of causal multi-label feature selection in federated setting and propose a Federated Causal Multi-label Feature Selection (FedCMFS) algorithm with three novel subroutines. Specifically, FedCMFS first uses the FedCFL subroutine that considers the correlations among label-label, label-feature, and feature-feature to learn the relevant features (candidate parents and children) of each class label while preserving data privacy without centralizing data. Second, FedCMFS employs the FedCFR subroutine to selectively recover the missed true relevant features. Finally, FedCMFS utilizes the FedCFC subroutine to remove false relevant features. The extensive experiments on 8 datasets have shown that FedCMFS is effect for causal multi-label feature selection in federated setting.

Title: A Comparative Study of Perceptual Quality Metrics for Audio-driven Talking Head Videos

• Authors: Weixia Zhang, Chengguang Zhu, Jingnan Gao, Yichao Yan, Guangtao Zhai, Xiaokang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06421
• Pdf URL: https://arxiv.org/pdf/2403.06421
• Copy Paste: [[2403.06421]] A Comparative Study of Perceptual Quality Metrics for Audio-driven Talking Head Videos(https://arxiv.org/abs/2403.06421)
• Keywords: generative
• Abstract: The rapid advancement of Artificial Intelligence Generated Content (AIGC) technology has propelled audio-driven talking head generation, gaining considerable research attention for practical applications. However, performance evaluation research lags behind the development of talking head generation techniques. Existing literature relies on heuristic quantitative metrics without human validation, hindering accurate progress assessment. To address this gap, we collect talking head videos generated from four generative methods and conduct controlled psychophysical experiments on visual quality, lip-audio synchronization, and head movement naturalness. Our experiments validate consistency between model predictions and human annotations, identifying metrics that align better with human opinions than widely-used measures. We believe our work will facilitate performance evaluation and model development, providing insights into AIGC in a broader context. Code and data will be made available at https://github.com/zwx8981/ADTH-QA.

Title: A Differential Geometric View and Explainability of GNN on Evolving Graphs

• Authors: Yazheng Liu, Xi Zhang, Sihong Xie
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06425
• Pdf URL: https://arxiv.org/pdf/2403.06425
• Copy Paste: [[2403.06425]] A Differential Geometric View and Explainability of GNN on Evolving Graphs(https://arxiv.org/abs/2403.06425)
• Keywords: explainability
• Abstract: Graphs are ubiquitous in social networks and biochemistry, where Graph Neural Networks (GNN) are the state-of-the-art models for prediction. Graphs can be evolving and it is vital to formally model and understand how a trained GNN responds to graph evolution. We propose a smooth parameterization of the GNN predicted distributions using axiomatic attribution, where the distributions are on a low-dimensional manifold within a high-dimensional embedding space. We exploit the differential geometric viewpoint to model distributional evolution as smooth curves on the manifold. We reparameterize families of curves on the manifold and design a convex optimization problem to find a unique curve that concisely approximates the distributional evolution for human interpretation. Extensive experiments on node classification, link prediction, and graph classification tasks with evolving graphs demonstrate the better sparsity, faithfulness, and intuitiveness of the proposed method over the state-of-the-art methods.

Title: Intra-Section Code Cave Injection for Adversarial Evasion Attacks on Windows PE Malware File

• Authors: Kshitiz Aryal, Maanak Gupta, Mahmoud Abdelsalam, Moustafa Saleh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06428
• Pdf URL: https://arxiv.org/pdf/2403.06428
• Copy Paste: [[2403.06428]] Intra-Section Code Cave Injection for Adversarial Evasion Attacks on Windows PE Malware File(https://arxiv.org/abs/2403.06428)
• Keywords: attack
• Abstract: Windows malware is predominantly available in cyberspace and is a prime target for deliberate adversarial evasion attacks. Although researchers have investigated the adversarial malware attack problem, a multitude of important questions remain unanswered, including (a) Are the existing techniques to inject adversarial perturbations in Windows Portable Executable (PE) malware files effective enough for evasion purposes?; (b) Does the attack process preserve the original behavior of malware?; (c) Are there unexplored approaches/locations that can be used to carry out adversarial evasion attacks on Windows PE malware?; and (d) What are the optimal locations and sizes of adversarial perturbations required to evade an ML-based malware detector without significant structural change in the PE file? To answer some of these questions, this work proposes a novel approach that injects a code cave within the section (i.e., intra-section) of Windows PE malware files to make space for adversarial perturbations. In addition, a code loader is also injected inside the PE file, which reverts adversarial malware to its original form during the execution, preserving the malware's functionality and executability. To understand the effectiveness of our approach, we injected adversarial perturbations inside the .text, .data and .rdata sections, generated using the gradient descent and Fast Gradient Sign Method (FGSM), to target the two popular CNN-based malware detectors, MalConv and MalConv2. Our experiments yielded notable results, achieving a 92.31% evasion rate with gradient descent and 96.26% with FGSM against MalConv, compared to the 16.17% evasion rate for append attacks. Similarly, when targeting MalConv2, our approach achieved a remarkable maximum evasion rate of 97.93% with gradient descent and 94.34% with FGSM, significantly surpassing the 4.01% evasion rate observed with append attacks.

Title: AS-FIBA: Adaptive Selective Frequency-Injection for Backdoor Attack on Deep Face Restoration

• Authors: Zhenbo Song, Wenhao Gao, Kaihao Zhang, Wenhan Luo, Zhaoxin Fan, Jianfeng Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06430
• Pdf URL: https://arxiv.org/pdf/2403.06430
• Copy Paste: [[2403.06430]] AS-FIBA: Adaptive Selective Frequency-Injection for Backdoor Attack on Deep Face Restoration(https://arxiv.org/abs/2403.06430)
• Keywords: attack
• Abstract: Deep learning-based face restoration models, increasingly prevalent in smart devices, have become targets for sophisticated backdoor attacks. These attacks, through subtle trigger injection into input face images, can lead to unexpected restoration outcomes. Unlike conventional methods focused on classification tasks, our approach introduces a unique degradation objective tailored for attacking restoration models. Moreover, we propose the Adaptive Selective Frequency Injection Backdoor Attack (AS-FIBA) framework, employing a neural network for input-specific trigger generation in the frequency domain, seamlessly blending triggers with benign images. This results in imperceptible yet effective attacks, guiding restoration predictions towards subtly degraded outputs rather than conspicuous targets. Extensive experiments demonstrate the efficacy of the degradation objective on state-of-the-art face restoration models. Additionally, it is notable that AS-FIBA can insert effective backdoors that are more imperceptible than existing backdoor attack methods, including WaNet, ISSBA, and FIBA.

Title: Joint-Embedding Masked Autoencoder for Self-supervised Learning of Dynamic Functional Connectivity from the Human Brain

• Authors: Jungwon Choi, Hyungi Lee, Byung-Hoon Kim, Juho Lee
• Subjects: cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2403.06432
• Pdf URL: https://arxiv.org/pdf/2403.06432
• Copy Paste: [[2403.06432]] Joint-Embedding Masked Autoencoder for Self-supervised Learning of Dynamic Functional Connectivity from the Human Brain(https://arxiv.org/abs/2403.06432)
• Keywords: robust, generative
• Abstract: Graph Neural Networks (GNNs) have shown promise in learning dynamic functional connectivity for distinguishing phenotypes from human brain networks. However, obtaining extensive labeled clinical data for training is often resource-intensive, making practical application difficult. Leveraging unlabeled data thus becomes crucial for representation learning in a label-scarce setting. Although generative self-supervised learning techniques, especially masked autoencoders, have shown promising results in representation learning in various domains, their application to dynamic graphs for dynamic functional connectivity remains underexplored, facing challenges in capturing high-level semantic representations. Here, we introduce the Spatio-Temporal Joint Embedding Masked Autoencoder (ST-JEMA), drawing inspiration from the Joint Embedding Predictive Architecture (JEPA) in computer vision. ST-JEMA employs a JEPA-inspired strategy for reconstructing dynamic graphs, which enables the learning of higher-level semantic representations considering temporal perspectives, addressing the challenges in fMRI data representation learning. Utilizing the large-scale UK Biobank dataset for self-supervised learning, ST-JEMA shows exceptional representation learning performance on dynamic functional connectivity demonstrating superiority over previous methods in predicting phenotypes and psychiatric diagnoses across eight benchmark fMRI datasets even with limited samples and effectiveness of temporal reconstruction on missing data scenarios. These findings highlight the potential of our approach as a robust representation learning method for leveraging label-scarce fMRI data.

Title: Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models

• Authors: Weihang Su, Changyue Wang, Qingyao Ai, Yiran HU, Zhijing Wu, Yujia Zhou, Yiqun Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06448
• Pdf URL: https://arxiv.org/pdf/2403.06448
• Copy Paste: [[2403.06448]] Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models(https://arxiv.org/abs/2403.06448)
• Keywords: large language model
• Abstract: Hallucinations in large language models (LLMs) refer to the phenomenon of LLMs producing responses that are coherent yet factually inaccurate. This issue undermines the effectiveness of LLMs in practical applications, necessitating research into detecting and mitigating hallucinations of LLMs. Previous studies have mainly concentrated on post-processing techniques for hallucination detection, which tend to be computationally intensive and limited in effectiveness due to their separation from the LLM's inference process. To overcome these limitations, we introduce MIND, an unsupervised training framework that leverages the internal states of LLMs for real-time hallucination detection without requiring manual annotations. Additionally, we present HELM, a new benchmark for evaluating hallucination detection across multiple LLMs, featuring diverse LLM outputs and the internal states of LLMs during their inference process. Our experiments demonstrate that MIND outperforms existing state-of-the-art methods in hallucination detection.

Title: Text2QR: Harmonizing Aesthetic Customization and Scanning Robustness for Text-Guided QR Code Generation

• Authors: Guangyang Wu, Xiaohong Liu, Jun Jia, Xuehao Cui, Guangtao Zhai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06452
• Pdf URL: https://arxiv.org/pdf/2403.06452
• Copy Paste: [[2403.06452]] Text2QR: Harmonizing Aesthetic Customization and Scanning Robustness for Text-Guided QR Code Generation(https://arxiv.org/abs/2403.06452)
• Keywords: robust, diffusion
• Abstract: In the digital era, QR codes serve as a linchpin connecting virtual and physical realms. Their pervasive integration across various applications highlights the demand for aesthetically pleasing codes without compromised scannability. However, prevailing methods grapple with the intrinsic challenge of balancing customization and scannability. Notably, stable-diffusion models have ushered in an epoch of high-quality, customizable content generation. This paper introduces Text2QR, a pioneering approach leveraging these advancements to address a fundamental challenge: concurrently achieving user-defined aesthetics and scanning robustness. To ensure stable generation of aesthetic QR codes, we introduce the QR Aesthetic Blueprint (QAB) module, generating a blueprint image exerting control over the entire generation process. Subsequently, the Scannability Enhancing Latent Refinement (SELR) process refines the output iteratively in the latent space, enhancing scanning robustness. This approach harnesses the potent generation capabilities of stable-diffusion models, navigating the trade-off between image aesthetics and QR code scannability. Our experiments demonstrate the seamless fusion of visual appeal with the practical utility of aesthetic QR codes, markedly outperforming prior methods. Codes are available at \url{https://github.com/mulns/Text2QR}

Title: Ensemble Quadratic Assignment Network for Graph Matching

• Authors: Haoru Tan, Chuang Wang, Sitong Wu, Xu-Yao Zhang, Fei Yin, Cheng-Lin Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06457
• Pdf URL: https://arxiv.org/pdf/2403.06457
• Copy Paste: [[2403.06457]] Ensemble Quadratic Assignment Network for Graph Matching(https://arxiv.org/abs/2403.06457)
• Keywords: robust
• Abstract: Graph matching is a commonly used technique in computer vision and pattern recognition. Recent data-driven approaches have improved the graph matching accuracy remarkably, whereas some traditional algorithm-based methods are more robust to feature noises, outlier nodes, and global transformation (e.g.~rotation). In this paper, we propose a graph neural network (GNN) based approach to combine the advantages of data-driven and traditional methods. In the GNN framework, we transform traditional graph-matching solvers as single-channel GNNs on the association graph and extend the single-channel architecture to the multi-channel network. The proposed model can be seen as an ensemble method that fuses multiple algorithms at every iteration. Instead of averaging the estimates at the end of the ensemble, in our approach, the independent iterations of the ensembled algorithms exchange their information after each iteration via a 1x1 channel-wise convolution layer. Experiments show that our model improves the performance of traditional algorithms significantly. In addition, we propose a random sampling strategy to reduce the computational complexity and GPU memory usage, so the model applies to matching graphs with thousands of nodes. We evaluate the performance of our method on three tasks: geometric graph matching, semantic feature matching, and few-shot 3D shape classification. The proposed model performs comparably or outperforms the best existing GNN-based methods.

Title: Reliable Spatial-Temporal Voxels For Multi-Modal Test-Time Adaptation

• Authors: Haozhi Cao, Yuecong Xu, Jianfei Yang, Pengyu Yin, Xingyu Ji, Shenghai Yuan, Lihua Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06461
• Pdf URL: https://arxiv.org/pdf/2403.06461
• Copy Paste: [[2403.06461]] Reliable Spatial-Temporal Voxels For Multi-Modal Test-Time Adaptation(https://arxiv.org/abs/2403.06461)
• Keywords: segmentation
• Abstract: Multi-modal test-time adaptation (MM-TTA) is proposed to adapt models to an unlabeled target domain by leveraging the complementary multi-modal inputs in an online manner. Previous MM-TTA methods rely on predictions of cross-modal information in each input frame, while they ignore the fact that predictions of geometric neighborhoods within consecutive frames are highly correlated, leading to unstable predictions across time. To fulfill this gap, we propose ReLiable Spatial-temporal Voxels (Latte), an MM-TTA method that leverages reliable cross-modal spatial-temporal correspondences for multi-modal 3D segmentation. Motivated by the fact that reliable predictions should be consistent with their spatial-temporal correspondences, Latte aggregates consecutive frames in a slide window manner and constructs ST voxel to capture temporally local prediction consistency for each modality. After filtering out ST voxels with high ST entropy, Latte conducts cross-modal learning for each point and pixel by attending to those with reliable and consistent predictions among both spatial and temporal neighborhoods. Experimental results show that Latte achieves state-of-the-art performance on three different MM-TTA benchmarks compared to previous MM-TTA or TTA methods.

Title: Towards the Uncharted: Density-Descending Feature Perturbation for Semi-supervised Semantic Segmentation

• Authors: Xiaoyang Wang, Huihui Bai, Limin Yu, Yao Zhao, Jimin Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06462
• Pdf URL: https://arxiv.org/pdf/2403.06462
• Copy Paste: [[2403.06462]] Towards the Uncharted: Density-Descending Feature Perturbation for Semi-supervised Semantic Segmentation(https://arxiv.org/abs/2403.06462)
• Keywords: segmentation
• Abstract: Semi-supervised semantic segmentation allows model to mine effective supervision from unlabeled data to complement label-guided training. Recent research has primarily focused on consistency regularization techniques, exploring perturbation-invariant training at both the image and feature levels. In this work, we proposed a novel feature-level consistency learning framework named Density-Descending Feature Perturbation (DDFP). Inspired by the low-density separation assumption in semi-supervised learning, our key insight is that feature density can shed a light on the most promising direction for the segmentation classifier to explore, which is the regions with lower density. We propose to shift features with confident predictions towards lower-density regions by perturbation injection. The perturbed features are then supervised by the predictions on the original features, thereby compelling the classifier to explore less dense regions to effectively regularize the decision boundary. Central to our method is the estimation of feature density. To this end, we introduce a lightweight density estimator based on normalizing flow, allowing for efficient capture of the feature density distribution in an online manner. By extracting gradients from the density estimator, we can determine the direction towards less dense regions for each feature. The proposed DDFP outperforms other designs on feature-level perturbations and shows state of the art performances on both Pascal VOC and Cityscapes dataset under various partition protocols. The project is available at https://github.com/Gavinwxy/DDFP.

Title: Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategy

• Authors: Jiuming Liu, Ruiji Yu, Yian Wang, Yu Zheng, Tianchen Deng, Weicai Ye, Hesheng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06467
• Pdf URL: https://arxiv.org/pdf/2403.06467
• Copy Paste: [[2403.06467]] Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategy(https://arxiv.org/abs/2403.06467)
• Keywords: transformer, segmentation
• Abstract: Recently, state space model (SSM) has gained great attention due to its promising performance, linear complexity, and long sequence modeling ability in both language and image domains. However, it is non-trivial to extend SSM to the point cloud field, because of the causality requirement of SSM and the disorder and irregularity nature of point clouds. In this paper, we propose a novel SSM-based point cloud processing backbone, named Point Mamba, with a causality-aware ordering mechanism. To construct the causal dependency relationship, we design an octree-based ordering strategy on raw irregular points, globally sorting points in a z-order sequence and also retaining their spatial proximity. Our method achieves state-of-the-art performance compared with transformer-based counterparts, with 93.4% accuracy and 75.7 mIOU respectively on the ModelNet40 classification dataset and ScanNet semantic segmentation dataset. Furthermore, our Point Mamba has linear complexity, which is more efficient than transformer-based methods. Our method demonstrates the great potential that SSM can serve as a generic backbone in point cloud understanding. Codes are released at https://github.com/IRMVLab/Point-Mamba.

Title: 3D-aware Image Generation and Editing with Multi-modal Conditions

• Authors: Bo Li, Yi-ke Li, Zhi-fen He, Bin Liu, Yun-Kun Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06470
• Pdf URL: https://arxiv.org/pdf/2403.06470
• Copy Paste: [[2403.06470]] 3D-aware Image Generation and Editing with Multi-modal Conditions(https://arxiv.org/abs/2403.06470)
• Keywords: generative
• Abstract: 3D-consistent image generation from a single 2D semantic label is an important and challenging research topic in computer graphics and computer vision. Although some related works have made great progress in this field, most of the existing methods suffer from poor disentanglement performance of shape and appearance, and lack multi-modal control. In this paper, we propose a novel end-to-end 3D-aware image generation and editing model incorporating multiple types of conditional inputs, including pure noise, text and reference image. On the one hand, we dive into the latent space of 3D Generative Adversarial Networks (GANs) and propose a novel disentanglement strategy to separate appearance features from shape features during the generation process. On the other hand, we propose a unified framework for flexible image generation and editing tasks with multi-modal conditions. Our method can generate diverse images with distinct noises, edit the attribute through a text description and conduct style transfer by giving a reference RGB image. Extensive experiments demonstrate that the proposed method outperforms alternative approaches both qualitatively and quantitatively on image generation and editing.

Title: Toward Robust Canine Cardiac Diagnosis: Deep Prototype Alignment Network-Based Few-Shot Segmentation in Veterinary Medicine

• Authors: Jun-Young Oh, In-Gyu Lee, Tae-Eui Kam, Ji-Hoon Jeong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06471
• Pdf URL: https://arxiv.org/pdf/2403.06471
• Copy Paste: [[2403.06471]] Toward Robust Canine Cardiac Diagnosis: Deep Prototype Alignment Network-Based Few-Shot Segmentation in Veterinary Medicine(https://arxiv.org/abs/2403.06471)
• Keywords: robust, segmentation
• Abstract: In the cutting-edge domain of medical artificial intelligence (AI), remarkable advances have been achieved in areas such as diagnosis, prediction, and therapeutic interventions. Despite these advances, the technology for image segmentation faces the significant barrier of having to produce extensively annotated datasets. To address this challenge, few-shot segmentation (FSS) has been recognized as one of the innovative solutions. Although most of the FSS research has focused on human health care, its application in veterinary medicine, particularly for pet care, remains largely limited. This study has focused on accurate segmentation of the heart and left atrial enlargement on canine chest radiographs using the proposed deep prototype alignment network (DPANet). The PANet architecture is adopted as the backbone model, and experiments are conducted using various encoders based on VGG-19, ResNet-18, and ResNet-50 to extract features. Experimental results demonstrate that the proposed DPANet achieves the highest performance. In the 2way-1shot scenario, it achieves the highest intersection over union (IoU) value of 0.6966, and in the 2way-5shot scenario, it achieves the highest IoU value of 0.797. The DPANet not only signifies a performance improvement, but also shows an improved training speed in the 2way-5shot scenario. These results highlight our model's exceptional capability as a trailblazing solution for segmenting the heart and left atrial enlargement in veterinary applications through FSS, setting a new benchmark in veterinary AI research, and demonstrating its superior potential to veterinary medicine advances.

Title: Ada-Tracker: Soft Tissue Tracking via Inter-Frame and Adaptive-Template Matching

• Authors: Jiaxin Guo, Jiangliu Wang, Zhaoshuo Li, Tongyu Jia, Qi Dou, Yun-Hui Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06479
• Pdf URL: https://arxiv.org/pdf/2403.06479
• Copy Paste: [[2403.06479]] Ada-Tracker: Soft Tissue Tracking via Inter-Frame and Adaptive-Template Matching(https://arxiv.org/abs/2403.06479)
• Keywords: robust
• Abstract: Soft tissue tracking is crucial for computer-assisted interventions. Existing approaches mainly rely on extracting discriminative features from the template and videos to recover corresponding matches. However, it is difficult to adopt these techniques in surgical scenes, where tissues are changing in shape and appearance throughout the surgery. To address this problem, we exploit optical flow to naturally capture the pixel-wise tissue deformations and adaptively correct the tracked template. Specifically, we first implement an inter-frame matching mechanism to extract a coarse region of interest based on optical flow from consecutive frames. To accommodate appearance change and alleviate drift, we then propose an adaptive-template matching method, which updates the tracked template based on the reliability of the estimates. Our approach, Ada-Tracker, enjoys both short-term dynamics modeling by capturing local deformations and long-term dynamics modeling by introducing global temporal compensation. We evaluate our approach on the public SurgT benchmark, which is generated from Hamlyn, SCARED, and Kidney boundary datasets. The experimental results show that Ada-Tracker achieves superior accuracy and performs more robustly against prior works. Code is available at https://github.com/wrld/Ada-Tracker.

Title: Multilingual Turn-taking Prediction Using Voice Activity Projection

• Authors: Koji Inoue, Bing'er Jiang, Erik Ekstedt, Tatsuya Kawahara, Gabriel Skantze
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.06487
• Pdf URL: https://arxiv.org/pdf/2403.06487
• Copy Paste: [[2403.06487]] Multilingual Turn-taking Prediction Using Voice Activity Projection(https://arxiv.org/abs/2403.06487)
• Keywords: transformer
• Abstract: This paper investigates the application of voice activity projection (VAP), a predictive turn-taking model for spoken dialogue, on multilingual data, encompassing English, Mandarin, and Japanese. The VAP model continuously predicts the upcoming voice activities of participants in dyadic dialogue, leveraging a cross-attention Transformer to capture the dynamic interplay between participants. The results show that a monolingual VAP model trained on one language does not make good predictions when applied to other languages. However, a multilingual model, trained on all three languages, demonstrates predictive performance on par with monolingual models across all languages. Further analyses show that the multilingual model has learned to discern the language of the input signal. We also analyze the sensitivity to pitch, a prosodic cue that is thought to be important for turn-taking. Finally, we compare two different audio encoders, contrastive predictive coding (CPC) pre-trained on English, with a recent model based on multilingual wav2vec 2.0 (MMS).

Title: Query-guided Prototype Evolution Network for Few-Shot Segmentation

• Authors: Runmin Cong, Hang Xiong, Jinpeng Chen, Wei Zhang, Qingming Huang, Yao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06488
• Pdf URL: https://arxiv.org/pdf/2403.06488
• Copy Paste: [[2403.06488]] Query-guided Prototype Evolution Network for Few-Shot Segmentation(https://arxiv.org/abs/2403.06488)
• Keywords: segmentation
• Abstract: Previous Few-Shot Segmentation (FSS) approaches exclusively utilize support features for prototype generation, neglecting the specific requirements of the query. To address this, we present the Query-guided Prototype Evolution Network (QPENet), a new method that integrates query features into the generation process of foreground and background prototypes, thereby yielding customized prototypes attuned to specific queries. The evolution of the foreground prototype is accomplished through a \textit{support-query-support} iterative process involving two new modules: Pseudo-prototype Generation (PPG) and Dual Prototype Evolution (DPE). The PPG module employs support features to create an initial prototype for the preliminary segmentation of the query image, resulting in a pseudo-prototype reflecting the unique needs of the current query. Subsequently, the DPE module performs reverse segmentation on support images using this pseudo-prototype, leading to the generation of evolved prototypes, which can be considered as custom solutions. As for the background prototype, the evolution begins with a global background prototype that represents the generalized features of all training images. We also design a Global Background Cleansing (GBC) module to eliminate potential adverse components mirroring the characteristics of the current foreground class. Experimental results on the PASCAL-$5^i$ and COCO-$20^i$ datasets attest to the substantial enhancements achieved by QPENet over prevailing state-of-the-art techniques, underscoring the validity of our ideas.

Title: QuantTune: Optimizing Model Quantization with Adaptive Outlier-Driven Fine Tuning

• Authors: Jiun-Man Chen, Yu-Hsuan Chao, Yu-Jie Wang, Ming-Der Shieh, Chih-Chung Hsu, Wei-Fen Lin
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.06497
• Pdf URL: https://arxiv.org/pdf/2403.06497
• Copy Paste: [[2403.06497]] QuantTune: Optimizing Model Quantization with Adaptive Outlier-Driven Fine Tuning(https://arxiv.org/abs/2403.06497)
• Keywords: transformer
• Abstract: Transformer-based models have gained widespread popularity in both the computer vision (CV) and natural language processing (NLP) fields. However, significant challenges arise during post-training linear quantization, leading to noticeable reductions in inference accuracy. Our study focuses on uncovering the underlying causes of these accuracy drops and proposing a quantization-friendly fine-tuning method, \textbf{QuantTune}. Firstly, our analysis revealed that, on average, 65\% of quantization errors result from the precision loss incurred by the dynamic range amplification effect of outliers across the target Transformer-based models. Secondly, \textbf{QuantTune} adjusts weights based on the deviation of outlier activations and effectively constrains the dynamic ranges of the problematic activations. As a result, it successfully mitigates the negative impact of outliers on the inference accuracy of quantized models. Lastly, \textbf{QuantTune} can be seamlessly integrated into the back-propagation pass in the fine-tuning process without requiring extra complexity in inference software and hardware design. Our approach showcases significant improvements in post-training quantization across a range of Transformer-based models, including ViT, Bert-base, and OPT. QuantTune reduces accuracy drops by 12.09\% at 8-bit quantization and 33.8\% at 7-bit compared to top calibration methods, outperforming state-of-the-art solutions by over 18.84\% across ViT models.

Title: 3D Semantic Segmentation-Driven Representations for 3D Object Detection

• Authors: Hayeon O, Kunsoo Huh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06501
• Pdf URL: https://arxiv.org/pdf/2403.06501
• Copy Paste: [[2403.06501]] 3D Semantic Segmentation-Driven Representations for 3D Object Detection(https://arxiv.org/abs/2403.06501)
• Keywords: segmentation
• Abstract: In autonomous driving, 3D detection provides more precise information to downstream tasks, including path planning and motion estimation, compared to 2D detection. Therefore, the need for 3D detection research has emerged. However, although single and multi-view images and depth maps obtained from the camera were used, detection accuracy was relatively low compared to other modality-based detectors due to the lack of geometric information. The proposed multi-modal 3D object detection combines semantic features obtained from images and geometric features obtained from point clouds, but there are difficulties in defining unified representation to fuse data existing in different domains and synchronization between them. In this paper, we propose SeSame : point-wise semantic feature as a new presentation to ensure sufficient semantic information of the existing LiDAR-only based 3D detection. Experiments show that our approach outperforms previous state-of-the-art at different levels of difficulty in car and performance improvement on the KITTI object detection benchmark. Our code is available at https://github.com/HAMA-DL-dev/SeSame

Title: Skeleton Supervised Airway Segmentation

• Authors: Mingyue Zhao, Han Li, Li Fan, Shiyuan Liu, Xiaolan Qiu, S.Kevin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06510
• Pdf URL: https://arxiv.org/pdf/2403.06510
• Copy Paste: [[2403.06510]] Skeleton Supervised Airway Segmentation(https://arxiv.org/abs/2403.06510)
• Keywords: extraction, segmentation
• Abstract: Fully-supervised airway segmentation has accomplished significant triumphs over the years in aiding pre-operative diagnosis and intra-operative navigation. However, full voxel-level annotation constitutes a labor-intensive and time-consuming task, often plagued by issues such as missing branches, branch annotation discontinuity, or erroneous edge delineation. label-efficient solutions for airway extraction are rarely explored yet primarily demanding in medical practice. To this end, we introduce a novel skeleton-level annotation (SkA) tailored to the airway, which simplifies the annotation workflow while enhancing annotation consistency and accuracy, preserving the complete topology. Furthermore, we propose a skeleton-supervised learning framework to achieve accurate airway segmentation. Firstly, a dual-stream buffer inference is introduced to realize initial label propagation from SkA, avoiding the collapse of direct learning from SkA. Then, we construct a geometry-aware dual-path propagation framework (GDP) to further promote complementary propagation learning, composed of hard geometry-aware propagation learning and soft geometry-aware propagation guidance. Experiments reveal that our proposed framework outperforms the competing methods with SKA, which amounts to only 1.96% airways, and achieves comparable performance with the baseline model that is fully supervised with 100% airways, demonstrating its significant potential in achieving label-efficient segmentation for other tubular structures, such as vessels.

Title: Asset-driven Threat Modeling for AI-based Systems

• Authors: Jan von der Assen, Jamo Sharif, Chao Feng, Gérôme Bovet, Burkhard Stiller
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.06512
• Pdf URL: https://arxiv.org/pdf/2403.06512
• Copy Paste: [[2403.06512]] Asset-driven Threat Modeling for AI-based Systems(https://arxiv.org/abs/2403.06512)
• Keywords: secure
• Abstract: Threat modeling is a popular method to securely develop systems by achieving awareness of potential areas of future damage caused by adversaries. The benefit of threat modeling lies in its ability to indicate areas of concern, paving the way to consider mitigation during the design stage. However, threat modeling for systems relying on Artificial Intelligence is still not well explored. While conventional threat modeling methods and tools did not address AI-related threats, research on this amalgamation still lacks solutions capable of guiding and automating the process, as well as providing evidence that the methods hold up in practice. To evaluate that the work at hand is able to guide and automatically identify AI-related threats during the architecture definition stage, several experts were tasked to create a threat model of an AI system designed in the healthcare domain. The usability of the solution was well-perceived, and the results indicate that it is effective for threat identification.

Title: Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learning

• Authors: Woojung Han, Chanyoung Kim, Dayun Ju, Yumin Shim, Seong Jae Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06516
• Pdf URL: https://arxiv.org/pdf/2403.06516
• Copy Paste: [[2403.06516]] Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learning(https://arxiv.org/abs/2403.06516)
• Keywords: diffusion
• Abstract: Recent advances in text-conditioned image generation diffusion models have begun paving the way for new opportunities in modern medical domain, in particular, generating Chest X-rays (CXRs) from diagnostic reports. Nonetheless, to further drive the diffusion models to generate CXRs that faithfully reflect the complexity and diversity of real data, it has become evident that a nontrivial learning approach is needed. In light of this, we propose CXRL, a framework motivated by the potential of reinforcement learning (RL). Specifically, we integrate a policy gradient RL approach with well-designed multiple distinctive CXR-domain specific reward models. This approach guides the diffusion denoising trajectory, achieving precise CXR posture and pathological details. Here, considering the complex medical image environment, we present "RL with Comparative Feedback" (RLCF) for the reward mechanism, a human-like comparative evaluation that is known to be more effective and reliable in complex scenarios compared to direct evaluation. Our CXRL framework includes jointly optimizing learnable adaptive condition embeddings (ACE) and the image generator, enabling the model to produce more accurate and higher perceptual CXR quality. Our extensive evaluation of the MIMIC-CXR-JPG dataset demonstrates the effectiveness of our RL-based tuning approach. Consequently, our CXRL generates pathologically realistic CXRs, establishing a new standard for generating CXRs with high fidelity to real-world clinical scenarios.

Title: Active Generation for Image Classification

• Authors: Tao Huang, Jiaqi Liu, Shan You, Chang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06517
• Pdf URL: https://arxiv.org/pdf/2403.06517
• Copy Paste: [[2403.06517]] Active Generation for Image Classification(https://arxiv.org/abs/2403.06517)
• Keywords: diffusion, generative
• Abstract: Recently, the growing capabilities of deep generative models have underscored their potential in enhancing image classification accuracy. However, existing methods often demand the generation of a disproportionately large number of images compared to the original dataset, while having only marginal improvements in accuracy. This computationally expensive and time-consuming process hampers the practicality of such approaches. In this paper, we propose to address the efficiency of image generation by focusing on the specific needs and characteristics of the model. With a central tenet of active learning, our method, named ActGen, takes a training-aware approach to image generation. It aims to create images akin to the challenging or misclassified samples encountered by the current model and incorporates these generated images into the training set to augment model performance. ActGen introduces an attentive image guidance technique, using real images as guides during the denoising process of a diffusion model. The model's attention on class prompt is leveraged to ensure the preservation of similar foreground object while diversifying the background. Furthermore, we introduce a gradient-based generation guidance method, which employs two losses to generate more challenging samples and prevent the generated images from being too similar to previously generated ones. Experimental results on the CIFAR and ImageNet datasets demonstrate that our method achieves better performance with a significantly reduced number of generated images.

Title: Adaptive Federated Learning Over the Air

• Authors: Chenhao Wang, Zihan Chen, Nikolaos Pappas, Howard H. Yang, Tony Q. S. Quek, H. Vincent Poor
• Subjects: cs.LG, cs.IT, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.06528
• Pdf URL: https://arxiv.org/pdf/2403.06528
• Copy Paste: [[2403.06528]] Adaptive Federated Learning Over the Air(https://arxiv.org/abs/2403.06528)
• Keywords: robust, federate
• Abstract: We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. This approach capitalizes on the inherent superposition property of wireless channels, facilitating fast and scalable parameter aggregation. Meanwhile, it enhances the robustness of the model training process by dynamically adjusting the stepsize in accordance with the global gradient update. We derive the convergence rate of the training algorithms, encompassing the effects of channel fading and interference, for a broad spectrum of nonconvex loss functions. Our analysis shows that the AdaGrad-based algorithm converges to a stationary point at the rate of $\mathcal{O}( \ln{(T)} /{ T^{ 1 - \frac{1}{\alpha} } } )$, where $\alpha$ represents the tail index of the electromagnetic interference. This result indicates that the level of heavy-tailedness in interference distribution plays a crucial role in the training efficiency: the heavier the tail, the slower the algorithm converges. In contrast, an Adam-like algorithm converges at the $\mathcal{O}( 1/T )$ rate, demonstrating its advantage in expediting the model training process. We conduct extensive experiments that corroborate our theoretical findings and affirm the practical efficacy of our proposed federated adaptive gradient methods.

Title: Confidence-Aware RGB-D Face Recognition via Virtual Depth Synthesis

• Authors: Zijian Chen, Mei Wang, Weihong Deng, Hongzhi Shi, Dongchao Wen, Yingjie Zhang, Xingchen Cui, Jian Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06529
• Pdf URL: https://arxiv.org/pdf/2403.06529
• Copy Paste: [[2403.06529]] Confidence-Aware RGB-D Face Recognition via Virtual Depth Synthesis(https://arxiv.org/abs/2403.06529)
• Keywords: robust
• Abstract: 2D face recognition encounters challenges in unconstrained environments due to varying illumination, occlusion, and pose. Recent studies focus on RGB-D face recognition to improve robustness by incorporating depth information. However, collecting sufficient paired RGB-D training data is expensive and time-consuming, hindering wide deployment. In this work, we first construct a diverse depth dataset generated by 3D Morphable Models for depth model pre-training. Then, we propose a domain-independent pre-training framework that utilizes readily available pre-trained RGB and depth models to separately perform face recognition without needing additional paired data for retraining. To seamlessly integrate the two distinct networks and harness the complementary benefits of RGB and depth information for improved accuracy, we propose an innovative Adaptive Confidence Weighting (ACW). This mechanism is designed to learn confidence estimates for each modality to achieve modality fusion at the score level. Our method is simple and lightweight, only requiring ACW training beyond the backbone models. Experiments on multiple public RGB-D face recognition benchmarks demonstrate state-of-the-art performance surpassing previous methods based on depth estimation and feature fusion, validating the efficacy of our approach.

Title: Multi-Scale Implicit Transformer with Re-parameterize for Arbitrary-Scale Super-Resolution

• Authors: Jinchen Zhu, Mingjian Zhang, Ling Zheng, Shizhuang Weng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06536
• Pdf URL: https://arxiv.org/pdf/2403.06536
• Copy Paste: [[2403.06536]] Multi-Scale Implicit Transformer with Re-parameterize for Arbitrary-Scale Super-Resolution(https://arxiv.org/abs/2403.06536)
• Keywords: extraction, transformer
• Abstract: Recently, the methods based on implicit neural representations have shown excellent capabilities for arbitrary-scale super-resolution (ASSR). Although these methods represent the features of an image by generating latent codes, these latent codes are difficult to adapt for different magnification factors of super-resolution, which seriously affects their performance. Addressing this, we design Multi-Scale Implicit Transformer (MSIT), consisting of an Multi-scale Neural Operator (MSNO) and Multi-Scale Self-Attention (MSSA). Among them, MSNO obtains multi-scale latent codes through feature enhancement, multi-scale characteristics extraction, and multi-scale characteristics merging. MSSA further enhances the multi-scale characteristics of latent codes, resulting in better performance. Furthermore, to improve the performance of network, we propose the Re-Interaction Module (RIM) combined with the cumulative training strategy to improve the diversity of learned information for the network. We have systematically introduced multi-scale characteristics for the first time in ASSR, extensive experiments are performed to validate the effectiveness of MSIT, and our method achieves state-of-the-art performance in arbitrary super-resolution tasks.

Title: OMH: Structured Sparsity via Optimally Matched Hierarchy for Unsupervised Semantic Segmentation

• Authors: Baran Ozaydin, Tong Zhang, Deblina Bhattacharjee, Sabine Süsstrunk, Mathieu Salzmann
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06546
• Pdf URL: https://arxiv.org/pdf/2403.06546
• Copy Paste: [[2403.06546]] OMH: Structured Sparsity via Optimally Matched Hierarchy for Unsupervised Semantic Segmentation(https://arxiv.org/abs/2403.06546)
• Keywords: segmentation
• Abstract: Unsupervised Semantic Segmentation (USS) involves segmenting images without relying on predefined labels, aiming to alleviate the burden of extensive human labeling. Existing methods utilize features generated by self-supervised models and specific priors for clustering. However, their clustering objectives are not involved in the optimization of the features during training. Additionally, due to the lack of clear class definitions in USS, the resulting segments may not align well with the clustering objective. In this paper, we introduce a novel approach called Optimally Matched Hierarchy (OMH) to simultaneously address the above issues. The core of our method lies in imposing structured sparsity on the feature space, which allows the features to encode information with different levels of granularity. The structure of this sparsity stems from our hierarchy (OMH). To achieve this, we learn a soft but sparse hierarchy among parallel clusters through Optimal Transport. Our OMH yields better unsupervised segmentation performance compared to existing USS methods. Our extensive experiments demonstrate the benefits of OMH when utilizing our differentiable paradigm. We will make our code publicly available.

Title: Unraveling the Mystery of Scaling Laws: Part I

• Authors: Hui Su, Zhi Tian, Xiaoyu Shen, Xunliang Cai
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06563
• Pdf URL: https://arxiv.org/pdf/2403.06563
• Copy Paste: [[2403.06563]] Unraveling the Mystery of Scaling Laws: Part I(https://arxiv.org/abs/2403.06563)
• Keywords: large language model
• Abstract: Scaling law principles indicate a power-law correlation between loss and variables such as model size, dataset size, and computational resources utilized during training. These principles play a vital role in optimizing various aspects of model pre-training, ultimately contributing to the success of large language models such as GPT-4, Llama and Gemini. However, the original scaling law paper by OpenAI did not disclose the complete details necessary to derive the precise scaling law formulas, and their conclusions are only based on models containing up to 1.5 billion parameters. Though some subsequent works attempt to unveil these details and scale to larger models, they often neglect the training dependency of important factors such as the learning rate, context length and batch size, leading to their failure to establish a reliable formula for predicting the test loss trajectory. In this technical report, we confirm that the scaling law formulations proposed in the original OpenAI paper remain valid when scaling the model size up to 33 billion, but the constant coefficients in these formulas vary significantly with the experiment setup. We meticulously identify influential factors and provide transparent, step-by-step instructions to estimate all constant terms in scaling-law formulas by training on models with only 1M~60M parameters. Using these estimated formulas, we showcase the capability to accurately predict various attributes for models with up to 33B parameters before their training, including (1) the minimum possible test loss; (2) the minimum required training steps and processed tokens to achieve a specific loss; (3) the critical batch size with an optimal time/computation trade-off at any loss value; and (4) the complete test loss trajectory with arbitrary batch size.

Title: Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications

• Authors: Can Cui (MULTISPEECH), Imran Ahamad Sheikh, Mostafa Sadeghi (MULTISPEECH), Emmanuel Vincent (MULTISPEECH)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06570
• Pdf URL: https://arxiv.org/pdf/2403.06570
• Copy Paste: [[2403.06570]] Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications(https://arxiv.org/abs/2403.06570)
• Keywords: extraction
• Abstract: Past studies on end-to-end meeting transcription have focused on model architecture and have mostly been evaluated on simulated meeting data. We present a novel study aiming to optimize the use of a Speaker-Attributed ASR (SA-ASR) system in real-life scenarios, such as the AMI meeting corpus, for improved speaker assignment of speech segments. First, we propose a pipeline tailored to real-life applications involving Voice Activity Detection (VAD), Speaker Diarization (SD), and SA-ASR. Second, we advocate using VAD output segments to fine-tune the SA-ASR model, considering that it is also applied to VAD segments during test, and show that this results in a relative reduction of Speaker Error Rate (SER) up to 28%. Finally, we explore strategies to enhance the extraction of the speaker embedding templates used as inputs by the SA-ASR system. We show that extracting them from SD output rather than annotated speaker segments results in a relative SER reduction up to 20%.

Title: AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models

• Authors: Yuting Wei, Yuanxing Xu, Xinru Wei, Simin Yang, Yangfu Zhu, Yuqing Li, Di Liu, Bin Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06574
• Pdf URL: https://arxiv.org/pdf/2403.06574
• Copy Paste: [[2403.06574]] AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models(https://arxiv.org/abs/2403.06574)
• Keywords: large language model
• Abstract: Given the importance of ancient Chinese in capturing the essence of rich historical and cultural heritage, the rapid advancements in Large Language Models (LLMs) necessitate benchmarks that can effectively evaluate their understanding of ancient contexts. To meet this need, we present AC-EVAL, an innovative benchmark designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of ancient Chinese. AC-EVAL is structured across three levels of difficulty reflecting different facets of language comprehension: general historical knowledge, short text understanding, and long text comprehension. The benchmark comprises 13 tasks, spanning historical facts, geography, social customs, art, philosophy, classical poetry and prose, providing a comprehensive assessment framework. Our extensive evaluation of top-performing LLMs, tailored for both English and Chinese, reveals a substantial potential for enhancing ancient text comprehension. By highlighting the strengths and weaknesses of LLMs, AC-EVAL aims to promote their development and application forward in the realms of ancient Chinese language education and scholarly research. The AC-EVAL data and evaluation code are available at https://github.com/yuting-wei/AC-EVAL.

Title: FFAD: A Novel Metric for Assessing Generated Time Series Data Utilizing Fourier Transform and Auto-encoder

• Authors: Yang Chen, Dustin J. Kempton, Rafal A. Angryk
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06576
• Pdf URL: https://arxiv.org/pdf/2403.06576
• Copy Paste: [[2403.06576]] FFAD: A Novel Metric for Assessing Generated Time Series Data Utilizing Fourier Transform and Auto-encoder(https://arxiv.org/abs/2403.06576)
• Keywords: generative
• Abstract: The success of deep learning-based generative models in producing realistic images, videos, and audios has led to a crucial consideration: how to effectively assess the quality of synthetic samples. While the Fr\'{e}chet Inception Distance (FID) serves as the standard metric for evaluating generative models in image synthesis, a comparable metric for time series data is notably absent. This gap in assessment capabilities stems from the absence of a widely accepted feature vector extractor pre-trained on benchmark time series datasets. In addressing these challenges related to assessing the quality of time series, particularly in the context of Fr\'echet Distance, this work proposes a novel solution leveraging the Fourier transform and Auto-encoder, termed the Fr\'{e}chet Fourier-transform Auto-encoder Distance (FFAD). Through our experimental results, we showcase the potential of FFAD for effectively distinguishing samples from different classes. This novel metric emerges as a fundamental tool for the evaluation of generative time series data, contributing to the ongoing efforts of enhancing assessment methodologies in the realm of deep learning-based generative models.

Title: Transformer-based Fusion of 2D-pose and Spatio-temporal Embeddings for Distracted Driver Action Recognition

• Authors: Erkut Akdag, Zeqi Zhu, Egor Bondarev, Peter H. N. De With
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06577
• Pdf URL: https://arxiv.org/pdf/2403.06577
• Copy Paste: [[2403.06577]] Transformer-based Fusion of 2D-pose and Spatio-temporal Embeddings for Distracted Driver Action Recognition(https://arxiv.org/abs/2403.06577)
• Keywords: robust, transformer
• Abstract: Classification and localization of driving actions over time is important for advanced driver-assistance systems and naturalistic driving studies. Temporal localization is challenging because it requires robustness, reliability, and accuracy. In this study, we aim to improve the temporal localization and classification accuracy performance by adapting video action recognition and 2D human-pose estimation networks to one model. Therefore, we design a transformer-based fusion architecture to effectively combine 2D-pose features and spatio-temporal features. The model uses 2D-pose features as the positional embedding of the transformer architecture and spatio-temporal features as the main input to the encoder of the transformer. The proposed solution is generic and independent of the camera numbers and positions, giving frame-based class probabilities as output. Finally, the post-processing step combines information from different camera views to obtain final predictions and eliminate false positives. The model performs well on the A2 test set of the 2023 NVIDIA AI City Challenge for naturalistic driving action recognition, achieving the overlap score of the organizer-defined distracted driver behaviour metric of 0.5079.

Title: DNNShield: Embedding Identifiers for Deep Neural Network Ownership Verification

• Authors: Jasper Stang, Torsten Krauß, Alexandra Dmitrienko
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06581
• Pdf URL: https://arxiv.org/pdf/2403.06581
• Copy Paste: [[2403.06581]] DNNShield: Embedding Identifiers for Deep Neural Network Ownership Verification(https://arxiv.org/abs/2403.06581)
• Keywords: secure, security, protect, attack
• Abstract: The surge in popularity of machine learning (ML) has driven significant investments in training Deep Neural Networks (DNNs). However, these models that require resource-intensive training are vulnerable to theft and unauthorized use. This paper addresses this challenge by introducing DNNShield, a novel approach for DNN protection that integrates seamlessly before training. DNNShield embeds unique identifiers within the model architecture using specialized protection layers. These layers enable secure training and deployment while offering high resilience against various attacks, including fine-tuning, pruning, and adaptive adversarial attacks. Notably, our approach achieves this security with minimal performance and computational overhead (less than 5\% runtime increase). We validate the effectiveness and efficiency of DNNShield through extensive evaluations across three datasets and four model architectures. This practical solution empowers developers to protect their DNNs and intellectual property rights.

Title: ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models

• Authors: Luca Arrotta, Claudio Bettini, Gabriele Civitarese, Michele Fiori
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06586
• Pdf URL: https://arxiv.org/pdf/2403.06586
• Copy Paste: [[2403.06586]] ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models(https://arxiv.org/abs/2403.06586)
• Keywords: large language model
• Abstract: Context-aware Human Activity Recognition (HAR) is a hot research area in mobile computing, and the most effective solutions in the literature are based on supervised deep learning models. However, the actual deployment of these systems is limited by the scarcity of labeled data that is required for training. Neuro-Symbolic AI (NeSy) provides an interesting research direction to mitigate this issue, by infusing common-sense knowledge about human activities and the contexts in which they can be performed into HAR deep learning classifiers. Existing NeSy methods for context-aware HAR rely on knowledge encoded in logic-based models (e.g., ontologies) whose design, implementation, and maintenance to capture new activities and contexts require significant human engineering efforts, technical knowledge, and domain expertise. Recent works show that pre-trained Large Language Models (LLMs) effectively encode common-sense knowledge about human activities. In this work, we propose ContextGPT: a novel prompt engineering approach to retrieve from LLMs common-sense knowledge about the relationship between human activities and the context in which they are performed. Unlike ontologies, ContextGPT requires limited human effort and expertise. An extensive evaluation carried out on two public datasets shows how a NeSy model obtained by infusing common-sense knowledge from ContextGPT is effective in data scarcity scenarios, leading to similar (and sometimes better) recognition rates than logic-based approaches with a fraction of the effort.

Title: Academically intelligent LLMs are not necessarily socially intelligent

• Authors: Ruoxi Xu, Hongyu Lin, Xianpei Han, Le Sun, Yingfei Sun
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.06591
• Pdf URL: https://arxiv.org/pdf/2403.06591
• Copy Paste: [[2403.06591]] Academically intelligent LLMs are not necessarily socially intelligent(https://arxiv.org/abs/2403.06591)
• Keywords: large language model
• Abstract: The academic intelligence of large language models (LLMs) has made remarkable progress in recent times, but their social intelligence performance remains unclear. Inspired by established human social intelligence frameworks, particularly Daniel Goleman's social intelligence theory, we have developed a standardized social intelligence test based on real-world social scenarios to comprehensively assess the social intelligence of LLMs, termed as the Situational Evaluation of Social Intelligence (SESI). We conducted an extensive evaluation with 13 recent popular and state-of-art LLM agents on SESI. The results indicate the social intelligence of LLMs still has significant room for improvement, with superficially friendliness as a primary reason for errors. Moreover, there exists a relatively low correlation between the social intelligence and academic intelligence exhibited by LLMs, suggesting that social intelligence is distinct from academic intelligence for LLMs. Additionally, while it is observed that LLMs can't ``understand'' what social intelligence is, their social intelligence, similar to that of humans, is influenced by social factors.

Title: Towards more accurate and useful data anonymity vulnerability measures

• Authors: Paul Francis, David Wagner
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06595
• Pdf URL: https://arxiv.org/pdf/2403.06595
• Copy Paste: [[2403.06595]] Towards more accurate and useful data anonymity vulnerability measures(https://arxiv.org/abs/2403.06595)
• Keywords: privacy, protect, attack, membership infer
• Abstract: The purpose of anonymizing structured data is to protect the privacy of individuals in the data while retaining the statistical properties of the data. There is a large body of work that examines anonymization vulnerabilities. Focusing on strong anonymization mechanisms, this paper examines a number of prominent attack papers and finds several problems, all of which lead to overstating risk. First, some papers fail to establish a correct statistical inference baseline (or any at all), leading to incorrect measures. Notably, the reconstruction attack from the US Census Bureau that led to a redesign of its disclosure method made this mistake. We propose the non-member framework, an improved method for how to compute a more accurate inference baseline, and give examples of its operation. Second, some papers don't use a realistic membership base rate, leading to incorrect precision measures if precision is reported. Third, some papers unnecessarily report measures in such a way that it is difficult or impossible to assess risk. Virtually the entire literature on membership inference attacks, dozens of papers, make one or both of these errors. We propose that membership inference papers report precision/recall values using a representative range of base rates.

Title: BEV2PR: BEV-Enhanced Visual Place Recognition with Structural Cues

• Authors: Fudong Ge, Yiwei Zhang, Shuhan Shen, Yue Wang, Weiming Hu, Jin Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06600
• Pdf URL: https://arxiv.org/pdf/2403.06600
• Copy Paste: [[2403.06600]] BEV2PR: BEV-Enhanced Visual Place Recognition with Structural Cues(https://arxiv.org/abs/2403.06600)
• Keywords: segmentation
• Abstract: In this paper, we propose a new image-based visual place recognition (VPR) framework by exploiting the structural cues in bird's-eye view (BEV) from a single monocular camera. The motivation arises from two key observations about VPR: 1) For the methods based on both camera and LiDAR sensors, the integration of LiDAR in robotic systems has led to increased expenses, while the alignment of data between different sensors is also a major challenge. 2) Other image-/camera-based methods, involving integrating RGB images and their derived variants (e.g., pseudo depth images, pseudo 3D point clouds), exhibit several limitations, such as the failure to effectively exploit the explicit spatial relationships between different objects. To tackle the above issues, we design a new BEV-enhanced VPR framework, nemely BEV2PR, which can generate a composite descriptor with both visual cues and spatial awareness solely based on a single camera. For the visual cues, any popular aggregation module for RGB global features can be integrated into our framework. The key points lie in: 1) We use BEV segmentation features as an explicit source of structural knowledge in constructing global features. 2) The lower layers of the pre-trained backbone from BEV map generation are shared for visual and structural streams in VPR, facilitating the learning of fine-grained local features in the visual stream. 3) The complementary visual features and structural features can jointly enhance VPR performance. Our BEV2PR framework enables consistent performance improvements over several popular camera-based VPR aggregation modules when integrating them. The experiments on our collected VPR-NuScenes dataset demonstrate an absolute gain of 2.47% on Recall@1 for the strong Conv-AP baseline to achieve the best performance in our setting, and notably, a 18.06% gain on the hard set.

Title: Cross-domain and Cross-dimension Learning for Image-to-Graph Transformers

• Authors: Alexander H. Berger, Laurin Lux, Suprosanna Shit, Ivan Ezhov, Georgios Kaissis, Martin J. Menten, Daniel Rueckert, Johannes C. Paetzold
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06601
• Pdf URL: https://arxiv.org/pdf/2403.06601
• Copy Paste: [[2403.06601]] Cross-domain and Cross-dimension Learning for Image-to-Graph Transformers(https://arxiv.org/abs/2403.06601)
• Keywords: extraction, transformer
• Abstract: Direct image-to-graph transformation is a challenging task that solves object detection and relationship prediction in a single model. Due to the complexity of this task, large training datasets are rare in many domains, which makes the training of large networks challenging. This data sparsity necessitates the establishment of pre-training strategies akin to the state-of-the-art in computer vision. In this work, we introduce a set of methods enabling cross-domain and cross-dimension transfer learning for image-to-graph transformers. We propose (1) a regularized edge sampling loss for sampling the optimal number of object relationships (edges) across domains, (2) a domain adaptation framework for image-to-graph transformers that aligns features from different domains, and (3) a simple projection function that allows us to pretrain 3D transformers on 2D input data. We demonstrate our method's utility in cross-domain and cross-dimension experiments, where we pretrain our models on 2D satellite images before applying them to vastly different target domains in 2D and 3D. Our method consistently outperforms a series of baselines on challenging benchmarks, such as retinal or whole-brain vessel graph extraction.

Title: Distributionally Generative Augmentation for Fair Facial Attribute Classification

• Authors: Fengda Zhang, Qianpei He, Kun Kuang, Jiashuo Liu, Long Chen, Chao Wu, Jun Xiao, Hanwang Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06606
• Pdf URL: https://arxiv.org/pdf/2403.06606
• Copy Paste: [[2403.06606]] Distributionally Generative Augmentation for Fair Facial Attribute Classification(https://arxiv.org/abs/2403.06606)
• Keywords: fair, interpretability, generative
• Abstract: Facial Attribute Classification (FAC) holds substantial promise in widespread applications. However, FAC models trained by traditional methodologies can be unfair by exhibiting accuracy inconsistencies across varied data subpopulations. This unfairness is largely attributed to bias in data, where some spurious attributes (e.g., Male) statistically correlate with the target attribute (e.g., Smiling). Most of existing fairness-aware methods rely on the labels of spurious attributes, which may be unavailable in practice. This work proposes a novel, generation-based two-stage framework to train a fair FAC model on biased data without additional annotation. Initially, we identify the potential spurious attributes based on generative models. Notably, it enhances interpretability by explicitly showing the spurious attributes in image space. Following this, for each image, we first edit the spurious attributes with a random degree sampled from a uniform distribution, while keeping target attribute unchanged. Then we train a fair FAC model by fostering model invariance to these augmentation. Extensive experiments on three common datasets demonstrate the effectiveness of our method in promoting fairness in FAC without compromising accuracy. Codes are in https://github.com/heqianpei/DiGA.

Title: Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds

• Authors: Jiageng WU, Xian Wu, Jie Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06609
• Pdf URL: https://arxiv.org/pdf/2403.06609
• Copy Paste: [[2403.06609]] Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds(https://arxiv.org/abs/2403.06609)
• Keywords: large language model
• Abstract: Clinical reasoning refers to the cognitive process that physicians employ in evaluating and managing patients. This process typically involves suggesting necessary examinations, diagnosing patients' diseases, and deciding on appropriate therapies, etc. Accurate clinical reasoning requires extensive medical knowledge and rich clinical experience, setting a high bar for physicians. This is particularly challenging in developing countries due to the overwhelming number of patients and limited physician resources, contributing significantly to global health inequity and necessitating automated clinical reasoning approaches. Recently, the emergence of large language models (LLMs) such as ChatGPT and GPT-4 have demonstrated their potential in clinical reasoning. However, these LLMs are prone to hallucination problems, and the reasoning process of LLMs may not align with the clinical decision path of physicians. In this study, we introduce a novel framework, In-Context Padding (ICP), designed to enhance LLMs with medical knowledge. Specifically, we infer critical clinical reasoning elements (referred to as knowledge seeds) and use these as anchors to guide the generation process of LLMs. Experiments on two clinical question datasets demonstrate that ICP significantly improves the clinical reasoning ability of LLMs.

Title: Real is not True: Backdoor Attacks Against Deepfake Detection

• Authors: Hong Sun, Ziqiang Li, Lei Liu, Bin Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06610
• Pdf URL: https://arxiv.org/pdf/2403.06610
• Copy Paste: [[2403.06610]] Real is not True: Backdoor Attacks Against Deepfake Detection(https://arxiv.org/abs/2403.06610)
• Keywords: attack
• Abstract: The proliferation of malicious deepfake applications has ignited substantial public apprehension, casting a shadow of doubt upon the integrity of digital media. Despite the development of proficient deepfake detection mechanisms, they persistently demonstrate pronounced vulnerability to an array of attacks. It is noteworthy that the pre-existing repertoire of attacks predominantly comprises adversarial example attack, predominantly manifesting during the testing phase. In the present study, we introduce a pioneering paradigm denominated as Bad-Deepfake, which represents a novel foray into the realm of backdoor attacks levied against deepfake detectors. Our approach hinges upon the strategic manipulation of a delimited subset of the training data, enabling us to wield disproportionate influence over the operational characteristics of a trained model. This manipulation leverages inherent frailties inherent to deepfake detectors, affording us the capacity to engineer triggers and judiciously select the most efficacious samples for the construction of the poisoned set. Through the synergistic amalgamation of these sophisticated techniques, we achieve an remarkable performance-a 100% attack success rate (ASR) against extensively employed deepfake detectors.

Title: MedKP: Medical Dialogue with Knowledge Enhancement and Clinical Pathway Encoding

• Authors: Jiageng Wu, Xian Wu, Yefeng Zheng, Jie Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06611
• Pdf URL: https://arxiv.org/pdf/2403.06611
• Copy Paste: [[2403.06611]] MedKP: Medical Dialogue with Knowledge Enhancement and Clinical Pathway Encoding(https://arxiv.org/abs/2403.06611)
• Keywords: large language model
• Abstract: With appropriate data selection and training techniques, Large Language Models (LLMs) have demonstrated exceptional success in various medical examinations and multiple-choice questions. However, the application of LLMs in medical dialogue generation-a task more closely aligned with actual medical practice-has been less explored. This gap is attributed to the insufficient medical knowledge of LLMs, which leads to inaccuracies and hallucinated information in the generated medical responses. In this work, we introduce the Medical dialogue with Knowledge enhancement and clinical Pathway encoding (MedKP) framework, which integrates an external knowledge enhancement module through a medical knowledge graph and an internal clinical pathway encoding via medical entities and physician actions. Evaluated with comprehensive metrics, our experiments on two large-scale, real-world online medical consultation datasets (MedDG and KaMed) demonstrate that MedKP surpasses multiple baselines and mitigates the incidence of hallucinations, achieving a new state-of-the-art. Extensive ablation studies further reveal the effectiveness of each component of MedKP. This enhancement advances the development of reliable, automated medical consultation responses using LLMs, thereby broadening the potential accessibility of precise and real-time medical assistance.

Title: Density-Guided Label Smoothing for Temporal Localization of Driving Actions

• Authors: Tunc Alkanat, Erkut Akdag, Egor Bondarev, Peter H. N. De With
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06616
• Pdf URL: https://arxiv.org/pdf/2403.06616
• Copy Paste: [[2403.06616]] Density-Guided Label Smoothing for Temporal Localization of Driving Actions(https://arxiv.org/abs/2403.06616)
• Keywords: robust
• Abstract: Temporal localization of driving actions plays a crucial role in advanced driver-assistance systems and naturalistic driving studies. However, this is a challenging task due to strict requirements for robustness, reliability and accurate localization. In this work, we focus on improving the overall performance by efficiently utilizing video action recognition networks and adapting these to the problem of action localization. To this end, we first develop a density-guided label smoothing technique based on label probability distributions to facilitate better learning from boundary video-segments that typically include multiple labels. Second, we design a post-processing step to efficiently fuse information from video-segments and multiple camera views into scene-level predictions, which facilitates elimination of false positives. Our methodology yields a competitive performance on the A2 test set of the naturalistic driving action recognition track of the 2022 NVIDIA AI City Challenge with an F1 score of 0.271.

Title: Forest Inspection Dataset for Aerial Semantic Segmentation and Depth Estimation

• Authors: Bianca-Cerasela-Zelia Blaga, Sergiu Nedevschi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06621
• Pdf URL: https://arxiv.org/pdf/2403.06621
• Copy Paste: [[2403.06621]] Forest Inspection Dataset for Aerial Semantic Segmentation and Depth Estimation(https://arxiv.org/abs/2403.06621)
• Keywords: segmentation
• Abstract: Humans use UAVs to monitor changes in forest environments since they are lightweight and provide a large variety of surveillance data. However, their information does not present enough details for understanding the scene which is needed to assess the degree of deforestation. Deep learning algorithms must be trained on large amounts of data to output accurate interpretations, but ground truth recordings of annotated forest imagery are not available. To solve this problem, we introduce a new large aerial dataset for forest inspection which contains both real-world and virtual recordings of natural environments, with densely annotated semantic segmentation labels and depth maps, taken in different illumination conditions, at various altitudes and recording angles. We test the performance of two multi-scale neural networks for solving the semantic segmentation task (HRNet and PointFlow network), studying the impact of the various acquisition conditions and the capabilities of transfer learning from virtual to real data. Our results showcase that the best results are obtained when the training is done on a dataset containing a large variety of scenarios, rather than separating the data into specific categories. We also develop a framework to assess the deforestation degree of an area.

Title: Self-Sovereign Identity for Electric Vehicle Charging

• Authors: Adrian Kailus, Dustin Kern, Christoph Krauß
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06632
• Pdf URL: https://arxiv.org/pdf/2403.06632
• Copy Paste: [[2403.06632]] Self-Sovereign Identity for Electric Vehicle Charging(https://arxiv.org/abs/2403.06632)
• Keywords: security, privacy
• Abstract: Electric Vehicles (EVs) are more and more charged at public Charge Points (CPs) using Plug-and-Charge (PnC) protocols such as the ISO 15118 standard which eliminates user interaction for authentication and authorization. Currently, this requires a rather complex Public Key Infrastructure (PKI) and enables driver tracking via the included unique identifiers. In this paper, we propose an approach for using Self-Sovereign Identities (SSIs) as trusted credentials for EV charging authentication and authorization which overcomes the privacy problems and the issues of a complex centralized PKI. Our implementation shows the feasibility of our approach with ISO 15118. The security and privacy of the proposed approach is shown in a formal analysis using the Tamarin prover.

Title: Stealing Part of a Production Language Model

• Authors: Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick, Florian Tramèr
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06634
• Pdf URL: https://arxiv.org/pdf/2403.06634
• Copy Paste: [[2403.06634]] Stealing Part of a Production Language Model(https://arxiv.org/abs/2403.06634)
• Keywords: defense, attack, steal, transformer
• Abstract: We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \$20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under \$2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

Title: Elephants Never Forget: Testing Language Models for Memorization of Tabular Data

• Authors: Sebastian Bordt, Harsha Nori, Rich Caruana
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06644
• Pdf URL: https://arxiv.org/pdf/2403.06644
• Copy Paste: [[2403.06644]] Elephants Never Forget: Testing Language Models for Memorization of Tabular Data(https://arxiv.org/abs/2403.06644)
• Keywords: large language model
• Abstract: While many have shown how Large Language Models (LLMs) can be applied to a diverse set of tasks, the critical issues of data contamination and memorization are often glossed over. In this work, we address this concern for tabular data. Starting with simple qualitative tests for whether an LLM knows the names and values of features, we introduce a variety of different techniques to assess the degrees of contamination, including statistical tests for conditional distribution modeling and four tests that identify memorization. Our investigation reveals that LLMs are pre-trained on many popular tabular datasets. This exposure can lead to invalid performance evaluation on downstream tasks because the LLMs have, in effect, been fit to the test set. Interestingly, we also identify a regime where the language model reproduces important statistics of the data, but fails to reproduce the dataset verbatim. On these datasets, although seen during training, good performance on downstream tasks might not be due to overfitting. Our findings underscore the need for ensuring data integrity in machine learning tasks with LLMs. To facilitate future research, we release an open-source tool that can perform various tests for memorization \url{https://github.com/interpretml/LLM-Tabular-Memorization-Checker}.

Title: Towards Zero-Shot Interpretable Human Recognition: A 2D-3D Registration Framework

• Authors: Henrique Jesus, Hugo Proença
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06658
• Pdf URL: https://arxiv.org/pdf/2403.06658
• Copy Paste: [[2403.06658]] Towards Zero-Shot Interpretable Human Recognition: A 2D-3D Registration Framework(https://arxiv.org/abs/2403.06658)
• Keywords: biometric, interpretability, explainability, generative
• Abstract: Large vision models based in deep learning architectures have been consistently advancing the state-of-the-art in biometric recognition. However, three weaknesses are commonly reported for such kind of approaches: 1) their extreme demands in terms of learning data; 2) the difficulties in generalising between different domains; and 3) the lack of interpretability/explainability, with biometrics being of particular interest, as it is important to provide evidence able to be used for forensics/legal purposes (e.g., in courts). To the best of our knowledge, this paper describes the first recognition framework/strategy that aims at addressing the three weaknesses simultaneously. At first, it relies exclusively in synthetic samples for learning purposes. Instead of requiring a large amount and variety of samples for each subject, the idea is to exclusively enroll a 3D point cloud per identity. Then, using generative strategies, we synthesize a very large (potentially infinite) number of samples, containing all the desired covariates (poses, clothing, distances, perspectives, lighting, occlusions,...). Upon the synthesizing method used, it is possible to adapt precisely to different kind of domains, which accounts for generalization purposes. Such data are then used to learn a model that performs local registration between image pairs, establishing positive correspondences between body parts that are the key, not only to recognition (according to cardinality and distribution), but also to provide an interpretable description of the response (e.g.: "both samples are from the same person, as they have similar facial shape, hair color and legs thickness").

Title: epsilon-Mesh Attack: A Surface-based Adversarial Point Cloud Attack for Facial Expression Recognition

• Authors: Batuhan Cengiz, Mert Gulsen, Yusuf H. Sahin, Gozde Unal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06661
• Pdf URL: https://arxiv.org/pdf/2403.06661
• Copy Paste: [[2403.06661]] epsilon-Mesh Attack: A Surface-based Adversarial Point Cloud Attack for Facial Expression Recognition(https://arxiv.org/abs/2403.06661)
• Keywords: attack, robust
• Abstract: Point clouds and meshes are widely used 3D data structures for many computer vision applications. While the meshes represent the surfaces of an object, point cloud represents sampled points from the surface which is also the output of modern sensors such as LiDAR and RGB-D cameras. Due to the wide application area of point clouds and the recent advancements in deep neural networks, studies focusing on robust classification of the 3D point cloud data emerged. To evaluate the robustness of deep classifier networks, a common method is to use adversarial attacks where the gradient direction is followed to change the input slightly. The previous studies on adversarial attacks are generally evaluated on point clouds of daily objects. However, considering 3D faces, these adversarial attacks tend to affect the person's facial structure more than the desired amount and cause malformation. Specifically for facial expressions, even a small adversarial attack can have a significant effect on the face structure. In this paper, we suggest an adversarial attack called $\epsilon$-Mesh Attack, which operates on point cloud data via limiting perturbations to be on the mesh surface. We also parameterize our attack by $\epsilon$ to scale the perturbation mesh. Our surface-based attack has tighter perturbation bounds compared to $L_2$ and $L_\infty$ norm bounded attacks that operate on unit-ball. Even though our method has additional constraints, our experiments on CoMA, Bosphorus and FaceWarehouse datasets show that $\epsilon$-Mesh Attack (Perpendicular) successfully confuses trained DGCNN and PointNet models $99.72\%$ and $97.06\%$ of the time, with indistinguishable facial deformations. The code is available at https://github.com/batuceng/e-mesh-attack.

Title: PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor

• Authors: Jaewon Jung, Hongsun Jang, Jaeyong Song, Jinho Lee
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06668
• Pdf URL: https://arxiv.org/pdf/2403.06668
• Copy Paste: [[2403.06668]] PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor(https://arxiv.org/abs/2403.06668)
• Keywords: security, attack, robust
• Abstract: Adversarial robustness of the neural network is a significant concern when it is applied to security-critical domains. In this situation, adversarial distillation is a promising option which aims to distill the robustness of the teacher network to improve the robustness of a small student network. Previous works pretrain the teacher network to make it robust to the adversarial examples aimed at itself. However, the adversarial examples are dependent on the parameters of the target network. The fixed teacher network inevitably degrades its robustness against the unseen transferred adversarial examples which targets the parameters of the student network in the adversarial distillation process. We propose PeerAiD to make a peer network learn the adversarial examples of the student network instead of adversarial examples aimed at itself. PeerAiD is an adversarial distillation that trains the peer network and the student network simultaneously in order to make the peer network specialized for defending the student network. We observe that such peer networks surpass the robustness of pretrained robust teacher network against student-attacked adversarial samples. With this peer network and adversarial distillation, PeerAiD achieves significantly higher robustness of the student network with AutoAttack (AA) accuracy up to 1.66%p and improves the natural accuracy of the student network up to 4.72%p with ResNet-18 and TinyImageNet dataset.

Title: CEAT: Continual Expansion and Absorption Transformer for Non-Exemplar Class-Incremental Learnin

• Authors: Xinyuan Gao, Songlin Dong, Yuhang He, Xing Wei, Yihong Gong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06670
• Pdf URL: https://arxiv.org/pdf/2403.06670
• Copy Paste: [[2403.06670]] CEAT: Continual Expansion and Absorption Transformer for Non-Exemplar Class-Incremental Learnin(https://arxiv.org/abs/2403.06670)
• Keywords: privacy, protect, transformer
• Abstract: In real-world applications, dynamic scenarios require the models to possess the capability to learn new tasks continuously without forgetting the old knowledge. Experience-Replay methods store a subset of the old images for joint training. In the scenario of more strict privacy protection, storing the old images becomes infeasible, which leads to a more severe plasticity-stability dilemma and classifier bias. To meet the above challenges, we propose a new architecture, named continual expansion and absorption transformer~(CEAT). The model can learn the novel knowledge by extending the expanded-fusion layers in parallel with the frozen previous parameters. After the task ends, we losslessly absorb the extended parameters into the backbone to ensure that the number of parameters remains constant. To improve the learning ability of the model, we designed a novel prototype contrastive loss to reduce the overlap between old and new classes in the feature space. Besides, to address the classifier bias towards the new classes, we propose a novel approach to generate the pseudo-features to correct the classifier. We experiment with our methods on three standard Non-Exemplar Class-Incremental Learning~(NECIL) benchmarks. Extensive experiments demonstrate that our model gets a significant improvement compared with the previous works and achieves 5.38%, 5.20%, and 4.92% improvement on CIFAR-100, TinyImageNet, and ImageNet-Subset.

Title: Poisoning Programs by Un-Repairing Code: Security Concerns of AI-generated Code

• Authors: Cristina Improta
• Subjects: cs.CR, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.06675
• Pdf URL: https://arxiv.org/pdf/2403.06675
• Copy Paste: [[2403.06675]] Poisoning Programs by Un-Repairing Code: Security Concerns of AI-generated Code(https://arxiv.org/abs/2403.06675)
• Keywords: security, attack, large language model
• Abstract: AI-based code generators have gained a fundamental role in assisting developers in writing software starting from natural language (NL). However, since these large language models are trained on massive volumes of data collected from unreliable online sources (e.g., GitHub, Hugging Face), AI models become an easy target for data poisoning attacks, in which an attacker corrupts the training data by injecting a small amount of poison into it, i.e., astutely crafted malicious samples. In this position paper, we address the security of AI code generators by identifying a novel data poisoning attack that results in the generation of vulnerable code. Next, we devise an extensive evaluation of how these attacks impact state-of-the-art models for code generation. Lastly, we discuss potential solutions to overcome this threat.

Title: CAM Back Again: Large Kernel CNNs from a Weakly Supervised Object Localization Perspective

• Authors: Shunsuke Yasuki, Masato Taki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06676
• Pdf URL: https://arxiv.org/pdf/2403.06676
• Copy Paste: [[2403.06676]] CAM Back Again: Large Kernel CNNs from a Weakly Supervised Object Localization Perspective(https://arxiv.org/abs/2403.06676)
• Keywords: robust, transformer
• Abstract: Recently, convolutional neural networks (CNNs) with large size kernels have attracted much attention in the computer vision field, following the success of the Vision Transformers. Large kernel CNNs have been reported to perform well in downstream vision tasks as well as in classification performance. The reason for the high-performance of large kernel CNNs in downstream tasks has been attributed to the large effective receptive field (ERF) produced by large size kernels, but this view has not been fully tested. We therefore revisit the performance of large kernel CNNs in downstream task, focusing on the weakly supervised object localization (WSOL) task. WSOL, a difficult downstream task that is not fully supervised, provides a new angle to explore the capabilities of the large kernel CNNs. Our study compares the modern large kernel CNNs ConvNeXt, RepLKNet, and SLaK to test the validity of the naive expectation that ERF size is important for improving downstream task performance. Our analysis of the factors contributing to high performance provides a different perspective, in which the main factor is feature map improvement. Furthermore, we find that modern CNNs are robust to the CAM problems of local regions of objects being activated, which has long been discussed in WSOL. CAM is the most classic WSOL method, but because of the above-mentioned problems, it is often used as a baseline method for comparison. However, experiments on the CUB-200-2011 dataset show that simply combining a large kernel CNN, CAM, and simple data augmentation methods can achieve performance (90.99% MaxBoxAcc) comparable to the latest WSOL method, which is CNN-based and requires special training or complex post-processing. The code is available at https://github.com/snskysk/CAM-Back-Again.

Title: Trustworthy Partial Label Learning with Out-of-distribution Detection

• Authors: Jintao Huang, Yiu-Ming Cheung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06681
• Pdf URL: https://arxiv.org/pdf/2403.06681
• Copy Paste: [[2403.06681]] Trustworthy Partial Label Learning with Out-of-distribution Detection(https://arxiv.org/abs/2403.06681)
• Keywords: robust
• Abstract: Partial Label Learning (PLL) grapples with learning from ambiguously labelled data, and it has been successfully applied in fields such as image recognition. Nevertheless, traditional PLL methods rely on the closed-world assumption, which can be limiting in open-world scenarios and negatively impact model performance and generalization. To tackle these challenges, our study introduces a novel method called PLL-OOD, which is the first to incorporate Out-of-Distribution (OOD) detection into the PLL framework. PLL-OOD significantly enhances model adaptability and accuracy by merging self-supervised learning with partial label loss and pioneering the Partial-Energy (PE) score for OOD detection. This approach improves data feature representation and effectively disambiguates candidate labels, using a dynamic label confidence matrix to refine predictions. The PE score, adjusted by label confidence, precisely identifies OOD instances, optimizing model training towards in-distribution data. This innovative method markedly boosts PLL model robustness and performance in open-world settings. To validate our approach, we conducted a comprehensive comparative experiment combining the existing state-of-the-art PLL model with multiple OOD scores on the CIFAR-10 and CIFAR-100 datasets with various OOD datasets. The results demonstrate that the proposed PLL-OOD framework is highly effective and effectiveness outperforms existing models, showcasing its superiority and effectiveness.

Title: Advancing Graph Neural Networks with HL-HGAT: A Hodge-Laplacian and Attention Mechanism Approach for Heterogeneous Graph-Structured Data

• Authors: Jinghan Huang, Qiufeng Chen, Yijun Bian, Pengli Zhu, Nanguang Chen, Moo K. Chung, Anqi Qiu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06687
• Pdf URL: https://arxiv.org/pdf/2403.06687
• Copy Paste: [[2403.06687]] Advancing Graph Neural Networks with HL-HGAT: A Hodge-Laplacian and Attention Mechanism Approach for Heterogeneous Graph-Structured Data(https://arxiv.org/abs/2403.06687)
• Keywords: transformer
• Abstract: Graph neural networks (GNNs) have proven effective in capturing relationships among nodes in a graph. This study introduces a novel perspective by considering a graph as a simplicial complex, encompassing nodes, edges, triangles, and $k$-simplices, enabling the definition of graph-structured data on any $k$-simplices. Our contribution is the Hodge-Laplacian heterogeneous graph attention network (HL-HGAT), designed to learn heterogeneous signal representations across $k$-simplices. The HL-HGAT incorporates three key components: HL convolutional filters (HL-filters), simplicial projection (SP), and simplicial attention pooling (SAP) operators, applied to $k$-simplices. HL-filters leverage the unique topology of $k$-simplices encoded by the Hodge-Laplacian (HL) operator, operating within the spectral domain of the $k$-th HL operator. To address computation challenges, we introduce a polynomial approximation for HL-filters, exhibiting spatial localization properties. Additionally, we propose a pooling operator to coarsen $k$-simplices, combining features through simplicial attention mechanisms of self-attention and cross-attention via transformers and SP operators, capturing topological interconnections across multiple dimensions of simplices. The HL-HGAT is comprehensively evaluated across diverse graph applications, including NP-hard problems, graph multi-label and classification challenges, and graph regression tasks in logistics, computer vision, biology, chemistry, and neuroscience. The results demonstrate the model's efficacy and versatility in handling a wide range of graph-based scenarios.

Title: PCLD: Point Cloud Layerwise Diffusion for Adversarial Purification

• Authors: Mert Gulsen, Batuhan Cengiz, Yusuf H. Sahin, Gozde Unal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06698
• Pdf URL: https://arxiv.org/pdf/2403.06698
• Copy Paste: [[2403.06698]] PCLD: Point Cloud Layerwise Diffusion for Adversarial Purification(https://arxiv.org/abs/2403.06698)
• Keywords: defense, attack, robust, diffusion
• Abstract: Point clouds are extensively employed in a variety of real-world applications such as robotics, autonomous driving and augmented reality. Despite the recent success of point cloud neural networks, especially for safety-critical tasks, it is essential to also ensure the robustness of the model. A typical way to assess a model's robustness is through adversarial attacks, where test-time examples are generated based on gradients to deceive the model. While many different defense mechanisms are studied in 2D, studies on 3D point clouds have been relatively limited in the academic field. Inspired from PointDP, which denoises the network inputs by diffusion, we propose Point Cloud Layerwise Diffusion (PCLD), a layerwise diffusion based 3D point cloud defense strategy. Unlike PointDP, we propagated the diffusion denoising after each layer to incrementally enhance the results. We apply our defense method to different types of commonly used point cloud models and adversarial attacks to evaluate its robustness. Our experiments demonstrate that the proposed defense method achieved results that are comparable to or surpass those of existing methodologies, establishing robustness through a novel technique. Code is available at https://github.com/batuceng/diffusion-layer-robustness-pc.

Title: Unprotected 4G/5G Control Procedures at Low Layers Considered Dangerous

• Authors: Norbert Ludant, Marinos Vomvas, Guevara Noubir
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06717
• Pdf URL: https://arxiv.org/pdf/2403.06717
• Copy Paste: [[2403.06717]] Unprotected 4G/5G Control Procedures at Low Layers Considered Dangerous(https://arxiv.org/abs/2403.06717)
• Keywords: security, protect, attack, steal
• Abstract: Over the years, several security vulnerabilities in the 3GPP cellular systems have been demonstrated in the literature. Most studies focus on higher layers of the cellular radio stack, such as the RRC and NAS, which are cryptographically protected. However, lower layers of the stack, such as PHY and MAC, are not as thoroughly studied, even though they are neither encrypted nor integrity protected. Furthermore, the latest releases of 5G significantly increased the number of low-layer control messages and procedures. The complexity of the cellular standards and the high degree of cross-layer operations, makes reasoning about security non-trivial, and requires a systematic analysis. We study the control procedures carried by each physical channel, and find that current cellular systems are susceptible to several new passive attacks due to information leakage, and active attacks by injecting MAC and PHY messages. For instance, we find that beamforming information leakage enables fingerprinting-based localization and tracking of users. We identify active attacks that reduce the users' throughput by disabling RF front ends at the UE, disrupt user communications by tricking other connected UEs into acting as jammers, or stealthily disconnect an active user. We evaluate our attacks against COTS UEs in various scenarios and demonstrate their practicality by measuring current operators' configurations across three countries. Our results show that an attacker can, among other things, localize users with an accuracy of 20 meters 96% of the time, track users' moving paths with a probability of 90%, reduce throughput by more than 95% within 2 seconds (by spoofing a 39 bits DCI), and disconnect users.

Title: Large Model driven Radiology Report Generation with Clinical Quality Reinforcement Learning

• Authors: Zijian Zhou, Miaojing Shi, Meng Wei, Oluwatosin Alabi, Zijie Yue, Tom Vercauteren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06728
• Pdf URL: https://arxiv.org/pdf/2403.06728
• Copy Paste: [[2403.06728]] Large Model driven Radiology Report Generation with Clinical Quality Reinforcement Learning(https://arxiv.org/abs/2403.06728)
• Keywords: large language model
• Abstract: Radiology report generation (RRG) has attracted significant attention due to its potential to reduce the workload of radiologists. Current RRG approaches are still unsatisfactory against clinical standards. This paper introduces a novel RRG method, \textbf{LM-RRG}, that integrates large models (LMs) with clinical quality reinforcement learning to generate accurate and comprehensive chest X-ray radiology reports. Our method first designs a large language model driven feature extractor to analyze and interpret different regions of the chest X-ray image, emphasizing specific regions with medical significance. Next, based on the large model's decoder, we develop a multimodal report generator that leverages multimodal prompts from visual features and textual instruction to produce the radiology report in an auto-regressive way. Finally, to better reflect the clinical significant and insignificant errors that radiologists would normally assign in the report, we introduce a novel clinical quality reinforcement learning strategy. It utilizes the radiology report clinical quality (RadCliQ) metric as a reward function in the learning process. Extensive experiments on the MIMIC-CXR and IU-Xray datasets demonstrate the superiority of our method over the state of the art.

Title: Enhancing Image Caption Generation Using Reinforcement Learning with Human Feedback

• Authors: Adarsh N L, Arun P V, Aravindh N L
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06735
• Pdf URL: https://arxiv.org/pdf/2403.06735
• Copy Paste: [[2403.06735]] Enhancing Image Caption Generation Using Reinforcement Learning with Human Feedback(https://arxiv.org/abs/2403.06735)
• Keywords: generative
• Abstract: Research on generative models to produce human-aligned / human-preferred outputs has seen significant recent contributions. Between text and image-generative models, we narrowed our focus to text-based generative models, particularly to produce captions for images that align with human preferences. In this research, we explored a potential method to amplify the performance of the Deep Neural Network Model to generate captions that are preferred by humans. This was achieved by integrating Supervised Learning and Reinforcement Learning with Human Feedback (RLHF) using the Flickr8k dataset. Also, a novel loss function that is capable of optimizing the model based on human feedback is introduced. In this paper, we provide a concise sketch of our approach and results, hoping to contribute to the ongoing advances in the field of human-aligned generative AI models.

Title: V3D: Video Diffusion Models are Effective 3D Generators

• Authors: Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, Huaping Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06738
• Pdf URL: https://arxiv.org/pdf/2403.06738
• Copy Paste: [[2403.06738]] V3D: Video Diffusion Models are Effective 3D Generators(https://arxiv.org/abs/2403.06738)
• Keywords: diffusion
• Abstract: Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency. Our code is available at https://github.com/heheyas/V3D

Title: Distribution-Aware Data Expansion with Diffusion Models

• Authors: Haowei Zhu, Ling Yang, Jun-Hai Yong, Wentao Zhang, Bin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06741
• Pdf URL: https://arxiv.org/pdf/2403.06741
• Copy Paste: [[2403.06741]] Distribution-Aware Data Expansion with Diffusion Models(https://arxiv.org/abs/2403.06741)
• Keywords: diffusion
• Abstract: The scale and quality of a dataset significantly impact the performance of deep models. However, acquiring large-scale annotated datasets is both a costly and time-consuming endeavor. To address this challenge, dataset expansion technologies aim to automatically augment datasets, unlocking the full potential of deep models. Current data expansion methods encompass image transformation-based and synthesis-based methods. The transformation-based methods introduce only local variations, resulting in poor diversity. While image synthesis-based methods can create entirely new content, significantly enhancing informativeness. However, existing synthesis methods carry the risk of distribution deviations, potentially degrading model performance with out-of-distribution samples. In this paper, we propose DistDiff, an effective data expansion framework based on the distribution-aware diffusion model. DistDiff constructs hierarchical prototypes to approximate the real data distribution, optimizing latent data points within diffusion models with hierarchical energy guidance. We demonstrate its ability to generate distribution-consistent samples, achieving substantial improvements in data expansion tasks. Specifically, without additional training, DistDiff achieves a 30.7% improvement in accuracy across six image datasets compared to the model trained on original datasets and a 9.8% improvement compared to the state-of-the-art diffusion-based method. Our code is available at https://github.com/haoweiz23/DistDiff

Title: ACT-MNMT Auto-Constriction Turning for Multilingual Neural Machine Translation

• Authors: Shaojie Dai, Xin Liu, Ping Luo, Yue Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06745
• Pdf URL: https://arxiv.org/pdf/2403.06745
• Copy Paste: [[2403.06745]] ACT-MNMT Auto-Constriction Turning for Multilingual Neural Machine Translation(https://arxiv.org/abs/2403.06745)
• Keywords: large language model
• Abstract: Large language model (LLM) has achieved promising performance in multilingual machine translation tasks through zero/few-shot prompts or prompt-tuning. However, due to the mixture of multilingual data during the pre-training of LLM, the LLM-based translation models face the off-target issue in both prompt-based methods, including a series of phenomena, namely instruction misunderstanding, translation with wrong language and over-generation. For this issue, this paper introduces an \textbf{\underline{A}}uto-\textbf{\underline{C}}onstriction \textbf{\underline{T}}urning mechanism for \textbf{\underline{M}}ultilingual \textbf{\underline{N}}eural \textbf{\underline{M}}achine \textbf{\underline{T}}ranslation (\model), which is a novel supervised fine-tuning mechanism and orthogonal to the traditional prompt-based methods. In this method, \model automatically constructs a constrained template in the target side by adding trigger tokens ahead of the ground truth. Furthermore, trigger tokens can be arranged and combined freely to represent different task semantics, and they can be iteratively updated to maximize the label likelihood. Experiments are performed on WMT test sets with multiple metrics, and the experimental results demonstrate that \model achieves substantially improved performance across multiple translation directions and reduce the off-target phenomena in the translation.

Title: ALaRM: Align Language Models via Hierarchical Rewards Modeling

• Authors: Yuhang Lai, Siyuan Wang, Shujun Liu, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06754
• Pdf URL: https://arxiv.org/pdf/2403.06754
• Copy Paste: [[2403.06754]] ALaRM: Align Language Models via Hierarchical Rewards Modeling(https://arxiv.org/abs/2403.06754)
• Keywords: large language model
• Abstract: We introduce ALaRM, the first framework modeling hierarchical rewards in reinforcement learning from human feedback (RLHF), which is designed to enhance the alignment of large language models (LLMs) with human preferences. The framework addresses the limitations of current alignment approaches, which often struggle with the inconsistency and sparsity of human supervision signals, by integrating holistic rewards with aspect-specific rewards. This integration enables more precise and consistent guidance of language models towards desired outcomes, particularly in complex and open text generation tasks. By employing a methodology that filters and combines multiple rewards based on their consistency, the framework provides a reliable mechanism for improving model alignment. We validate our approach through applications in long-form question answering and machine translation tasks, employing gpt-3.5-turbo for pairwise comparisons, and demonstrate improvements over existing baselines. Our work underscores the effectiveness of hierarchical rewards modeling in refining LLM training processes for better human preference alignment. We release our code at https://ALaRM-fdu.github.io.

Title: Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation

• Authors: Theodore Barfoot, Luis Garcia-Peraza-Herrera, Ben Glocker, Tom Vercauteren
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06759
• Pdf URL: https://arxiv.org/pdf/2403.06759
• Copy Paste: [[2403.06759]] Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation(https://arxiv.org/abs/2403.06759)
• Keywords: segmentation
• Abstract: Deep neural networks for medical image segmentation often produce overconfident results misaligned with empirical observations. Such miscalibration, challenges their clinical translation. We propose to use marginal L1 average calibration error (mL1-ACE) as a novel auxiliary loss function to improve pixel-wise calibration without compromising segmentation quality. We show that this loss, despite using hard binning, is directly differentiable, bypassing the need for approximate but differentiable surrogate or soft binning approaches. Our work also introduces the concept of dataset reliability histograms which generalises standard reliability diagrams for refined visual assessment of calibration in semantic segmentation aggregated at the dataset level. Using mL1-ACE, we reduce average and maximum calibration error by 45% and 55% respectively, maintaining a Dice score of 87% on the BraTS 2021 dataset. We share our code here: https://github.com/cai4cai/ACE-DLIRIS

Title: ConspEmoLLM: Conspiracy Theory Detection Using an Emotion-Based Large Language Model

• Authors: Zhiwei Liu, Boyang Liu, Paul Thompson, Kailai Yang, Raghav Jain, Sophia Ananiadou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06765
• Pdf URL: https://arxiv.org/pdf/2403.06765
• Copy Paste: [[2403.06765]] ConspEmoLLM: Conspiracy Theory Detection Using an Emotion-Based Large Language Model(https://arxiv.org/abs/2403.06765)
• Keywords: large language model
• Abstract: The internet has brought both benefits and harms to society. A prime example of the latter is misinformation, including conspiracy theories, which flood the web. Recent advances in natural language processing, particularly the emergence of large language models (LLMs), have improved the prospects of accurate misinformation detection. However, most LLM-based approaches to conspiracy theory detection focus only on binary classification and fail to account for the important relationship between misinformation and affective features (i.e., sentiment and emotions). Driven by a comprehensive analysis of conspiracy text that reveals its distinctive affective features, we propose ConspEmoLLM, the first open-source LLM that integrates affective information and is able to perform diverse tasks relating to conspiracy theories. These tasks include not only conspiracy theory detection, but also classification of theory type and detection of related discussion (e.g., opinions towards theories). ConspEmoLLM is fine-tuned based on an emotion-oriented LLM using our novel ConDID dataset, which includes five tasks to support LLM instruction tuning and evaluation. We demonstrate that when applied to these tasks, ConspEmoLLM largely outperforms several open-source general domain LLMs and ChatGPT, as well as an LLM that has been fine-tuned using ConDID, but which does not use affective features. This project will be released on https://github.com/lzw108/ConspEmoLLM/.

Title: Strength Lies in Differences! Towards Effective Non-collaborative Dialogues via Tailored Strategy Planning

• Authors: Tong Zhang, Chen Huang, Yang Deng, Hongru Liang, Jia Liu, Zujie Wen, Wenqiang Lei, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06769
• Pdf URL: https://arxiv.org/pdf/2403.06769
• Copy Paste: [[2403.06769]] Strength Lies in Differences! Towards Effective Non-collaborative Dialogues via Tailored Strategy Planning(https://arxiv.org/abs/2403.06769)
• Keywords: secure
• Abstract: We investigate non-collaborative dialogue agents that must engage in tailored strategic planning for diverse users to secure a favorable agreement. This poses challenges for existing dialogue agents due to two main reasons: their inability to integrate user-specific characteristics into their strategic planning and their training paradigm's failure to produce strategic planners that can generalize to diverse users. To address these challenges, we propose TRIP to enhance the capability in tailored strategic planning, incorporating a user-aware strategic planning module and a population-based training paradigm. Through experiments on benchmark non-collaborative dialogue tasks, we demonstrate the effectiveness of TRIP in catering to diverse users.

Title: Boosting Image Restoration via Priors from Pre-trained Models

• Authors: Xiaogang Xu, Shu Kong, Tao Hu, Zhe Liu, Hujun Bao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06793
• Pdf URL: https://arxiv.org/pdf/2403.06793
• Copy Paste: [[2403.06793]] Boosting Image Restoration via Priors from Pre-trained Models(https://arxiv.org/abs/2403.06793)
• Keywords: diffusion
• Abstract: Pre-trained models with large-scale training data, such as CLIP and Stable Diffusion, have demonstrated remarkable performance in various high-level computer vision tasks such as image understanding and generation from language descriptions. Yet, their potential for low-level tasks such as image restoration remains relatively unexplored. In this paper, we explore such models to enhance image restoration. As off-the-shelf features (OSF) from pre-trained models do not directly serve image restoration, we propose to learn an additional lightweight module called Pre-Train-Guided Refinement Module (PTG-RM) to refine restoration results of a target restoration network with OSF. PTG-RM consists of two components, Pre-Train-Guided Spatial-Varying Enhancement (PTG-SVE), and Pre-Train-Guided Channel-Spatial Attention (PTG-CSA). PTG-SVE enables optimal short- and long-range neural operations, while PTG-CSA enhances spatial-channel attention for restoration-related learning. Extensive experiments demonstrate that PTG-RM, with its compact size ($<$1M parameters), effectively enhances restoration performance of various models across different tasks, including low-light enhancement, deraining, deblurring, and denoising.

Title: Leveraging Internal Representations of Model for Magnetic Image Classification

• Authors: Adarsh N L, Arun P V, Alok Porwal, Malcolm Aranha
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06797
• Pdf URL: https://arxiv.org/pdf/2403.06797
• Copy Paste: [[2403.06797]] Leveraging Internal Representations of Model for Magnetic Image Classification(https://arxiv.org/abs/2403.06797)
• Keywords: security, privacy
• Abstract: Data generated by edge devices has the potential to train intelligent autonomous systems across various domains. Despite the emergence of diverse machine learning approaches addressing privacy concerns and utilizing distributed data, security issues persist due to the sensitive storage of data shards in disparate locations. This paper introduces a potentially groundbreaking paradigm for machine learning model training, specifically designed for scenarios with only a single magnetic image and its corresponding label image available. We harness the capabilities of Deep Learning to generate concise yet informative samples, aiming to overcome data scarcity. Through the utilization of deep learning's internal representations, our objective is to efficiently address data scarcity issues and produce meaningful results. This methodology presents a promising avenue for training machine learning models with minimal data.

Title: Data-Independent Operator: A Training-Free Artifact Representation Extractor for Generalizable Deepfake Detection

• Authors: Chuangchuang Tan, Ping Liu, RenShuai Tao, Huan Liu, Yao Zhao, Baoyuan Wu, Yunchao Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06803
• Pdf URL: https://arxiv.org/pdf/2403.06803
• Copy Paste: [[2403.06803]] Data-Independent Operator: A Training-Free Artifact Representation Extractor for Generalizable Deepfake Detection(https://arxiv.org/abs/2403.06803)
• Keywords: generative
• Abstract: Recently, the proliferation of increasingly realistic synthetic images generated by various generative adversarial networks has increased the risk of misuse. Consequently, there is a pressing need to develop a generalizable detector for accurately recognizing fake images. The conventional methods rely on generating diverse training sources or large pretrained models. In this work, we show that, on the contrary, the small and training-free filter is sufficient to capture more general artifact representations. Due to its unbias towards both the training and test sources, we define it as Data-Independent Operator (DIO) to achieve appealing improvements on unseen sources. In our framework, handcrafted filters and the randomly-initialized convolutional layer can be used as the training-free artifact representations extractor with excellent results. With the data-independent operator of a popular classifier, such as Resnet50, one could already reach a new state-of-the-art without bells and whistles. We evaluate the effectiveness of the DIO on 33 generation models, even DALLE and Midjourney. Our detector achieves a remarkable improvement of $13.3\%$, establishing a new state-of-the-art performance. The DIO and its extension can serve as strong baselines for future methods. The code is available at \url{https://github.com/chuangchuangtan/Data-Independent-Operator}.

Title: Multistep Consistency Models

• Authors: Jonathan Heek, Emiel Hoogeboom, Tim Salimans
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.06807
• Pdf URL: https://arxiv.org/pdf/2403.06807
• Copy Paste: [[2403.06807]] Multistep Consistency Models(https://arxiv.org/abs/2403.06807)
• Keywords: diffusion
• Abstract: Diffusion models are relatively easy to train but require many steps to generate samples. Consistency models are far more difficult to train, but generate samples in a single step. In this paper we propose Multistep Consistency Models: A unification between Consistency Models (Song et al., 2023) and TRACT (Berthelot et al., 2023) that can interpolate between a consistency model and a diffusion model: a trade-off between sampling speed and sampling quality. Specifically, a 1-step consistency model is a conventional consistency model whereas we show that a $\infty$-step consistency model is a diffusion model. Multistep Consistency Models work really well in practice. By increasing the sample budget from a single step to 2-8 steps, we can train models more easily that generate higher quality samples, while retaining much of the sampling speed benefits. Notable results are 1.4 FID on Imagenet 64 in 8 step and 2.1 FID on Imagenet128 in 8 steps with consistency distillation. We also show that our method scales to a text-to-image diffusion model, generating samples that are very close to the quality of the original model.

Title: Deep Learning Approaches for Human Action Recognition in Video Data

• Authors: Yufei Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06810
• Pdf URL: https://arxiv.org/pdf/2403.06810
• Copy Paste: [[2403.06810]] Deep Learning Approaches for Human Action Recognition in Video Data(https://arxiv.org/abs/2403.06810)
• Keywords: robust
• Abstract: Human action recognition in videos is a critical task with significant implications for numerous applications, including surveillance, sports analytics, and healthcare. The challenge lies in creating models that are both precise in their recognition capabilities and efficient enough for practical use. This study conducts an in-depth analysis of various deep learning models to address this challenge. Utilizing a subset of the UCF101 Videos dataset, we focus on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Two-Stream ConvNets. The research reveals that while CNNs effectively capture spatial features and RNNs encode temporal sequences, Two-Stream ConvNets exhibit superior performance by integrating spatial and temporal dimensions. These insights are distilled from the evaluation metrics of accuracy, precision, recall, and F1-score. The results of this study underscore the potential of composite models in achieving robust human action recognition and suggest avenues for future research in optimizing these models for real-world deployment.

Title: Monotone Individual Fairness

• Authors: Yahav Bechavod
• Subjects: cs.LG, cs.CY, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.06812
• Pdf URL: https://arxiv.org/pdf/2403.06812
• Copy Paste: [[2403.06812]] Monotone Individual Fairness(https://arxiv.org/abs/2403.06812)
• Keywords: fair
• Abstract: We revisit the problem of online learning with individual fairness, where an online learner strives to maximize predictive accuracy while ensuring that similar individuals are treated similarly. We first extend the frameworks of Gillen et al. (2018); Bechavod et al. (2020), which rely on feedback from human auditors regarding fairness violations, as we consider auditing schemes that are capable of aggregating feedback from any number of auditors, using a rich class we term monotone aggregation functions. We then prove a characterization for such auditing schemes, practically reducing the analysis of auditing for individual fairness by multiple auditors to that of auditing by (instance-specific) single auditors. Using our generalized framework, we present an oracle-efficient algorithm achieving an upper bound frontier of $(\mathcal{O}(T^{1/2+2b}),\mathcal{O}(T^{3/4-b}))$ respectively for regret, number of fairness violations, for $0\leq b \leq 1/4$. We then study an online classification setting where label feedback is available for positively-predicted individuals only, and present an oracle-efficient algorithm achieving an upper bound frontier of $(\mathcal{O}(T^{2/3+2b}),\mathcal{O}(T^{5/6-b}))$ for regret, number of fairness violations, for $0\leq b \leq 1/6$. In both settings, our algorithms improve on the best known bounds for oracle-efficient algorithms. Furthermore, our algorithms offer significant improvements in computational efficiency, greatly reducing the number of required calls to an (offline) optimization oracle per round, to $\tilde{\mathcal{O}}(\alpha^{-2})$ in the full information setting, and $\tilde{\mathcal{O}}(\alpha^{-2} + k^2T^{1/3})$ in the partial information setting, where $\alpha$ is the sensitivity for reporting fairness violations, and $k$ is the number of individuals in a round.

Title: In-context Exploration-Exploitation for Reinforcement Learning

• Authors: Zhenwen Dai, Federico Tomasi, Sina Ghiassian
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.06826
• Pdf URL: https://arxiv.org/pdf/2403.06826
• Copy Paste: [[2403.06826]] In-context Exploration-Exploitation for Reinforcement Learning(https://arxiv.org/abs/2403.06826)
• Keywords: transformer
• Abstract: In-context learning is a promising approach for online policy learning of offline reinforcement learning (RL) methods, which can be achieved at inference time without gradient optimization. However, this method is hindered by significant computational costs resulting from the gathering of large training trajectory sets and the need to train large Transformer models. We address this challenge by introducing an In-context Exploration-Exploitation (ICEE) algorithm, designed to optimize the efficiency of in-context policy learning. Unlike existing models, ICEE performs an exploration-exploitation trade-off at inference time within a Transformer model, without the need for explicit Bayesian inference. Consequently, ICEE can solve Bayesian optimization problems as efficiently as Gaussian process biased methods do, but in significantly less time. Through experiments in grid world environments, we demonstrate that ICEE can learn to solve new RL tasks using only tens of episodes, marking a substantial improvement over the hundreds of episodes needed by the previous in-context learning method.

Title: HDRTransDC: High Dynamic Range Image Reconstruction with Transformer Deformation Convolution

• Authors: Shuaikang Shang, Xuejing Kang, Anlong Ming
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06831
• Pdf URL: https://arxiv.org/pdf/2403.06831
• Copy Paste: [[2403.06831]] HDRTransDC: High Dynamic Range Image Reconstruction with Transformer Deformation Convolution(https://arxiv.org/abs/2403.06831)
• Keywords: transformer
• Abstract: High Dynamic Range (HDR) imaging aims to generate an artifact-free HDR image with realistic details by fusing multi-exposure Low Dynamic Range (LDR) images. Caused by large motion and severe under-/over-exposure among input LDR images, HDR imaging suffers from ghosting artifacts and fusion distortions. To address these critical issues, we propose an HDR Transformer Deformation Convolution (HDRTransDC) network to generate high-quality HDR images, which consists of the Transformer Deformable Convolution Alignment Module (TDCAM) and the Dynamic Weight Fusion Block (DWFB). To solve the ghosting artifacts, the proposed TDCAM extracts long-distance content similar to the reference feature in the entire non-reference features, which can accurately remove misalignment and fill the content occluded by moving objects. For the purpose of eliminating fusion distortions, we propose DWFB to spatially adaptively select useful information across frames to effectively fuse multi-exposed features. Extensive experiments show that our method quantitatively and qualitatively achieves state-of-the-art performance.

Title: The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework

• Authors: Zhuo Chen, Yin Fang, Yichi Zhang, Lingbing Guo, Jiaoyan Chen, Huajun Chen, Wen Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06832
• Pdf URL: https://arxiv.org/pdf/2403.06832
• Copy Paste: [[2403.06832]] The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework(https://arxiv.org/abs/2403.06832)
• Keywords: robust, transformer, large language model
• Abstract: The advancement of Multi-modal Pre-training highlights the necessity for a robust Multi-Modal Knowledge Graph (MMKG) representation learning framework. This framework is crucial for integrating structured knowledge into multi-modal Large Language Models (LLMs) at scale, aiming to alleviate issues like knowledge misconceptions and multi-modal hallucinations. In this work, to evaluate models' ability to accurately embed entities within MMKGs, we focus on two widely researched tasks: Multi-modal Knowledge Graph Completion (MKGC) and Multi-modal Entity Alignment (MMEA). Building on this foundation, we propose a novel SNAG method that utilizes a Transformer-based architecture equipped with modality-level noise masking for the robust integration of multi-modal entity features in KGs. By incorporating specific training objectives for both MKGC and MMEA, our approach achieves SOTA performance across a total of ten datasets (three for MKGC and seven for MEMA), demonstrating its robustness and versatility. Besides, SNAG can not only function as a standalone model but also enhance other existing methods, providing stable performance improvements. Our code and data are available at: https://github.com/zjukg/SNAG.

Title: Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?

• Authors: Egor Zverev, Sahar Abdelnabi, Mario Fritz, Christoph H. Lampert
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06833
• Pdf URL: https://arxiv.org/pdf/2403.06833
• Copy Paste: [[2403.06833]] Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?(https://arxiv.org/abs/2403.06833)
• Keywords: large language model
• Abstract: Instruction-tuned Large Language Models (LLMs) have achieved breakthrough results, opening countless new possibilities for many practical applications. However, LLMs lack elementary safety features that are established norms in other areas of computer science, such as the separation between instructions and data, causing them to malfunction or rendering them vulnerable to manipulation and interference by third parties e.g., via indirect prompt/command injection. Even worse, so far, there is not even an established definition of what precisely such a separation would mean and how its violation could be tested. In this work, we aim to close this gap. We introduce a formal measure to quantify the phenomenon of instruction-data separation as well as an empirical variant of the measure that can be computed from a model`s black-box outputs. We also introduce a new dataset, SEP (Should it be Executed or Processed?), which allows estimating the measure, and we report results on several state-of-the-art open-source and closed LLMs. Finally, we quantitatively demonstrate that all evaluated LLMs fail to achieve a high amount of separation, according to our measure. The source code and SEP dataset are openly accessible at https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed.

Title: Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting

• Authors: Wenting Chen, Pengyu Wang, Hui Ren, Lichao Sun, Quanzheng Li, Yixuan Yuan, Xiang Li
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06835
• Pdf URL: https://arxiv.org/pdf/2403.06835
• Copy Paste: [[2403.06835]] Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting(https://arxiv.org/abs/2403.06835)
• Keywords: privacy, generative
• Abstract: Data scarcity and privacy concerns limit the availability of high-quality medical images for public use, which can be mitigated through medical image synthesis. However, current medical image synthesis methods often struggle to accurately capture the complexity of detailed anatomical structures and pathological conditions. To address these challenges, we propose a novel medical image synthesis model that leverages fine-grained image-text alignment and anatomy-pathology prompts to generate highly detailed and accurate synthetic medical images. Our method integrates advanced natural language processing techniques with image generative modeling, enabling precise alignment between descriptive text prompts and the synthesized images' anatomical and pathological details. The proposed approach consists of two key components: an anatomy-pathology prompting module and a fine-grained alignment-based synthesis module. The anatomy-pathology prompting module automatically generates descriptive prompts for high-quality medical images. To further synthesize high-quality medical images from the generated prompts, the fine-grained alignment-based synthesis module pre-defines a visual codebook for the radiology dataset and performs fine-grained alignment between the codebook and generated prompts to obtain key patches as visual clues, facilitating accurate image synthesis. We validate the superiority of our method through experiments on public chest X-ray datasets and demonstrate that our synthetic images preserve accurate semantic information, making them valuable for various medical applications.

Title: Stochastic Cortical Self-Reconstruction

• Authors: Christian Wachinger, Dennis Hedderich, Fabian Bongratz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06837
• Pdf URL: https://arxiv.org/pdf/2403.06837
• Copy Paste: [[2403.06837]] Stochastic Cortical Self-Reconstruction(https://arxiv.org/abs/2403.06837)
• Keywords: generative
• Abstract: Magnetic resonance imaging (MRI) is critical for diagnosing neurodegenerative diseases, yet accurately assessing mild cortical atrophy remains a challenge due to its subtlety. Automated cortex reconstruction, paired with healthy reference ranges, aids in pinpointing pathological atrophy, yet their generalization is limited by biases from image acquisition and processing. We introduce the concept of stochastic cortical self-reconstruction (SCSR) that creates a subject-specific healthy reference by taking MRI-derived thicknesses as input and, therefore, implicitly accounting for potential confounders. SCSR randomly corrupts parts of the cortex and self-reconstructs them from the remaining information. Trained exclusively on healthy individuals, repeated self-reconstruction generates a stochastic reference cortex for assessing deviations from the norm. We present three implementations of this concept: XGBoost applied on parcels, and two autoencoders on vertex level -- one based on a multilayer perceptron and the other using a spherical U-Net. These models were trained on healthy subjects from the UK Biobank and subsequently evaluated across four public Alzheimer's datasets. Finally, we deploy the model on clinical in-house data, where deviation maps' high spatial resolution aids in discriminating between four types of dementia.

Title: RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback

• Authors: Yanming Liu, Xinyue Peng, Xuhong Zhang, Weihao Liu, Jianwei Yin, Jiannan Cao, Tianyu Du
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06840
• Pdf URL: https://arxiv.org/pdf/2403.06840
• Copy Paste: [[2403.06840]] RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback(https://arxiv.org/abs/2403.06840)
• Keywords: large language model
• Abstract: Large language models (LLMs) demonstrate exceptional performance in numerous tasks but still heavily rely on knowledge stored in their parameters. Moreover, updating this knowledge incurs high training costs. Retrieval-augmented generation (RAG) methods address this issue by integrating external knowledge. The model can answer questions it couldn't previously by retrieving knowledge relevant to the query. This approach improves performance in certain scenarios for specific tasks. However, if irrelevant texts are retrieved, it may impair model performance. In this paper, we propose Retrieval Augmented Iterative Self-Feedback (RA-ISF), a framework that iteratively decomposes tasks and processes them in three submodules to enhance the model's problem-solving capabilities. Experiments show that our method outperforms existing benchmarks, performing well on models like GPT3.5, Llama2, significantly enhancing factual reasoning capabilities and reducing hallucinations.

Title: DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation

• Authors: Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, Xingang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06845
• Pdf URL: https://arxiv.org/pdf/2403.06845
• Copy Paste: [[2403.06845]] DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation(https://arxiv.org/abs/2403.06845)
• Keywords: large language model
• Abstract: World models have demonstrated superiority in autonomous driving, particularly in the generation of multi-view driving videos. However, significant challenges still exist in generating customized driving videos. In this paper, we propose DriveDreamer-2, which builds upon the framework of DriveDreamer and incorporates a Large Language Model (LLM) to generate user-defined driving videos. Specifically, an LLM interface is initially incorporated to convert a user's query into agent trajectories. Subsequently, a HDMap, adhering to traffic regulations, is generated based on the trajectories. Ultimately, we propose the Unified Multi-View Model to enhance temporal and spatial coherence in the generated driving videos. DriveDreamer-2 is the first world model to generate customized driving videos, it can generate uncommon driving videos (e.g., vehicles abruptly cut in) in a user-friendly manner. Besides, experimental results demonstrate that the generated videos enhance the training of driving perception methods (e.g., 3D detection and tracking). Furthermore, video generation quality of DriveDreamer-2 surpasses other state-of-the-art methods, showcasing FID and FVD scores of 11.2 and 55.7, representing relative improvements of 30% and 50%.

Title: Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification

• Authors: Joar Skalse, Alessandro Abate
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06854
• Pdf URL: https://arxiv.org/pdf/2403.06854
• Copy Paste: [[2403.06854]] Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification(https://arxiv.org/abs/2403.06854)
• Keywords: robust
• Abstract: Inverse reinforcement learning (IRL) aims to infer an agent's preferences (represented as a reward function $R$) from their behaviour (represented as a policy $\pi$). To do this, we need a behavioural model of how $\pi$ relates to $R$. In the current literature, the most common behavioural models are optimality, Boltzmann-rationality, and causal entropy maximisation. However, the true relationship between a human's preferences and their behaviour is much more complex than any of these behavioural models. This means that the behavioural models are misspecified, which raises the concern that they may lead to systematic errors if applied to real data. In this paper, we analyse how sensitive the IRL problem is to misspecification of the behavioural model. Specifically, we provide necessary and sufficient conditions that completely characterise how the observed data may differ from the assumed behavioural model without incurring an error above a given threshold. In addition to this, we also characterise the conditions under which a behavioural model is robust to small perturbations of the observed policy, and we analyse how robust many behavioural models are to misspecification of their parameter values (such as e.g.\ the discount rate). Our analysis suggests that the IRL problem is highly sensitive to misspecification, in the sense that very mild misspecification can lead to very large errors in the inferred reward function.

Title: Development of a Reliable and Accessible Caregiving Language Model (CaLM)

• Authors: Bambang Parmanto, Bayu Aryoyudanta, Wilbert Soekinto, I Made Agus Setiawan, Yuhan Wang, Haomin Hu, Andi Saptono, Yong K. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06857
• Pdf URL: https://arxiv.org/pdf/2403.06857
• Copy Paste: [[2403.06857]] Development of a Reliable and Accessible Caregiving Language Model (CaLM)(https://arxiv.org/abs/2403.06857)
• Keywords: large language model
• Abstract: Unlike professional caregivers, family caregivers often assume this role without formal preparation or training. Because of this, there is an urgent need to enhance the capacity of family caregivers to provide quality care. Large language models can potentially be used as a foundation technology for supporting caregivers as educational tools or as adjunct to care. This study aimed to develop a reliable Caregiving Language Model (CaLM) by using FMs and a caregiving knowledge base, develop an accessible CaLM using a small FM that requires fewer computing resources, and evaluate the performance of the model compared to a large FM. We developed CaLM using the Retrieval Augmented Generation (RAG) framework combined with FM fine-tuning for improving the quality of FM answers by grounding the model on a caregiving knowledge base. We used two small FMs as candidates for the FM of CaLM (LLaMA-2 and Falcon with 7B parameters) and larger FM GPT-3.5 as a benchmark. We developed the caregiving knowledge base by gathering various types of documents from the Internet. In this study, we focused on caregivers of individuals with Alzheimer's Disease Related Dementias. We evaluated the models' performance using the benchmark metrics commonly used in evaluating language models and their reliability to provide accurate references with the answers. The RAG framework improved the performance of all FMs used in this study across all measures. As expected, the large FM performed better than small FMs across all metrics. The most interesting result is that small fine-tuned FMs with RAG performed significantly better than GPT 3.5 across all metrics. The fine-tuned LLaMA-2 small FM performed better than GPT 3.5 (even with RAG) in returning references with the answers. The study shows that reliable and accessible CaLM can be developed by using small FMs with a knowledge base specific to the caregiving domain.

Title: A Geospatial Approach to Predicting Desert Locust Breeding Grounds in Africa

• Authors: Ibrahim Salihu Yusuf, Mukhtar Opeyemi Yusuf, Kobby Panford-Quainoo, Arnu Pretorius
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06860
• Pdf URL: https://arxiv.org/pdf/2403.06860
• Copy Paste: [[2403.06860]] A Geospatial Approach to Predicting Desert Locust Breeding Grounds in Africa(https://arxiv.org/abs/2403.06860)
• Keywords: security
• Abstract: Desert locust swarms present a major threat to agriculture and food security. Addressing this challenge, our study develops an operationally-ready model for predicting locust breeding grounds, which has the potential to enhance early warning systems and targeted control measures. We curated a dataset from the United Nations Food and Agriculture Organization's (UN-FAO) locust observation records and analyzed it using two types of spatio-temporal input features: remotely-sensed environmental and climate data as well as multi-spectral earth observation images. Our approach employed custom deep learning models (three-dimensional and LSTM-based recurrent convolutional networks), along with the geospatial foundational model Prithvi recently released by Jakubik et al., 2023. These models notably outperformed existing baselines, with the Prithvi-based model, fine-tuned on multi-spectral images from NASA's Harmonized Landsat and Sentinel-2 (HLS) dataset, achieving the highest accuracy, F1 and ROC-AUC scores (83.03%, 81.53% and 87.69%, respectively). A significant finding from our research is that multi-spectral earth observation images alone are sufficient for effective locust breeding ground prediction without the need to explicitly incorporate climatic or environmental features.

Title: QUASAR: QUality and Aesthetics Scoring with Advanced Representations

• Authors: Sergey Kastryulin (1 and 3), Denis Prokopenko (2), Artem Babenko (3), Dmitry V. Dylov (1 and 4) ((1) Skolkovo Institute of Science and Technology, (2) King's Colledge London, (3) Yandex, (4) AIRI)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06866
• Pdf URL: https://arxiv.org/pdf/2403.06866
• Copy Paste: [[2403.06866]] QUASAR: QUality and Aesthetics Scoring with Advanced Representations(https://arxiv.org/abs/2403.06866)
• Keywords: robust
• Abstract: This paper introduces a new data-driven, non-parametric method for image quality and aesthetics assessment, surpassing existing approaches and requiring no prompt engineering or fine-tuning. We eliminate the need for expressive textual embeddings by proposing efficient image anchors in the data. Through extensive evaluations of 7 state-of-the-art self-supervised models, our method demonstrates superior performance and robustness across various datasets and benchmarks. Notably, it achieves high agreement with human assessments even with limited data and shows high robustness to the nature of data and their pre-processing pipeline. Our contributions offer a streamlined solution for assessment of images while providing insights into the perception of visual information.

Title: On the Generalization Ability of Unsupervised Pretraining

• Authors: Yuyang Deng, Junyuan Hong, Jiayu Zhou, Mehrdad Mahdavi
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.06871
• Pdf URL: https://arxiv.org/pdf/2403.06871
• Copy Paste: [[2403.06871]] On the Generalization Ability of Unsupervised Pretraining(https://arxiv.org/abs/2403.06871)
• Keywords: transformer
• Abstract: Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. However, a rigorous understanding of how the representation function learned on an unlabeled dataset affects the generalization of the fine-tuned model is lacking. Existing theoretical research does not adequately account for the heterogeneity of the distribution and tasks in pre-training and fine-tuning stage. To bridge this gap, this paper introduces a novel theoretical framework that illuminates the critical factor influencing the transferability of knowledge acquired during unsupervised pre-training to the subsequent fine-tuning phase, ultimately affecting the generalization capabilities of the fine-tuned model on downstream tasks. We apply our theoretical framework to analyze generalization bound of two distinct scenarios: Context Encoder pre-training with deep neural networks and Masked Autoencoder pre-training with deep transformers, followed by fine-tuning on a binary classification task. Finally, inspired by our findings, we propose a novel regularization method during pre-training to further enhances the generalization of fine-tuned model. Overall, our results contribute to a better understanding of unsupervised pre-training and fine-tuning paradigm, and can shed light on the design of more effective pre-training algorithms.

Title: Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documents

• Authors: Nishchal Prasad, Mohand Boughanem, Taoufiq Dkaki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06872
• Pdf URL: https://arxiv.org/pdf/2403.06872
• Copy Paste: [[2403.06872]] Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documents(https://arxiv.org/abs/2403.06872)
• Keywords: transformer, large language model
• Abstract: Legal judgment prediction suffers from the problem of long case documents exceeding tens of thousands of words, in general, and having a non-uniform structure. Predicting judgments from such documents becomes a challenging task, more so on documents with no structural annotation. We explore the classification of these large legal documents and their lack of structural information with a deep-learning-based hierarchical framework which we call MESc; "Multi-stage Encoder-based Supervised with-clustering"; for judgment prediction. Specifically, we divide a document into parts to extract their embeddings from the last four layers of a custom fine-tuned Large Language Model, and try to approximate their structure through unsupervised clustering. Which we use in another set of transformer encoder layers to learn the inter-chunk representations. We analyze the adaptability of Large Language Models (LLMs) with multi-billion parameters (GPT-Neo, and GPT-J) with the hierarchical framework of MESc and compare them with their standalone performance on legal texts. We also study their intra-domain(legal) transfer learning capability and the impact of combining embeddings from their last layers in MESc. We test these methods and their effectiveness with extensive experiments and ablation studies on legal documents from India, the European Union, and the United States with the ILDC dataset and a subset of the LexGLUE dataset. Our approach achieves a minimum total performance gain of approximately 2 points over previous state-of-the-art methods.

Title: Real-time Transformer-based Open-Vocabulary Detection with Efficient Fusion Head

• Authors: Tiancheng Zhao, Peng Liu, Xuan He, Lu Zhang, Kyusong Lee
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06892
• Pdf URL: https://arxiv.org/pdf/2403.06892
• Copy Paste: [[2403.06892]] Real-time Transformer-based Open-Vocabulary Detection with Efficient Fusion Head(https://arxiv.org/abs/2403.06892)
• Keywords: transformer
• Abstract: End-to-end transformer-based detectors (DETRs) have shown exceptional performance in both closed-set and open-vocabulary object detection (OVD) tasks through the integration of language modalities. However, their demanding computational requirements have hindered their practical application in real-time object detection (OD) scenarios. In this paper, we scrutinize the limitations of two leading models in the OVDEval benchmark, OmDet and Grounding-DINO, and introduce OmDet-Turbo. This novel transformer-based real-time OVD model features an innovative Efficient Fusion Head (EFH) module designed to alleviate the bottlenecks observed in OmDet and Grounding-DINO. Notably, OmDet-Turbo-Base achieves a 100.2 frames per second (FPS) with TensorRT and language cache techniques applied. Notably, in zero-shot scenarios on COCO and LVIS datasets, OmDet-Turbo achieves performance levels nearly on par with current state-of-the-art supervised models. Furthermore, it establishes new state-of-the-art benchmarks on ODinW and OVDEval, boasting an AP of 30.1 and an NMS-AP of 26.86, respectively. The practicality of OmDet-Turbo in industrial applications is underscored by its exceptional performance on benchmark datasets and superior inference speed, positioning it as a compelling choice for real-time object detection tasks. Code: \url{https://github.com/om-ai-lab/OmDet}

Title: GRITv2: Efficient and Light-weight Social Relation Recognition

• Authors: N K Sagar Reddy, Neeraj Kasera, Avinash Thakur
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06895
• Pdf URL: https://arxiv.org/pdf/2403.06895
• Copy Paste: [[2403.06895]] GRITv2: Efficient and Light-weight Social Relation Recognition(https://arxiv.org/abs/2403.06895)
• Keywords: transformer
• Abstract: Our research focuses on the analysis and improvement of the Graph-based Relation Inference Transformer (GRIT), which serves as an important benchmark in the field. We conduct a comprehensive ablation study using the PISC-fine dataset, to find and explore improvement in efficiency and performance of GRITv2. Our research has provided a new state-of-the-art relation recognition model on the PISC relation dataset. We introduce several features in the GRIT model and analyse our new benchmarks in two versions: GRITv2-L (large) and GRITv2-S (small). Our proposed GRITv2-L surpasses existing methods on relation recognition and the GRITv2-S is within 2% performance gap of GRITv2-L, which has only 0.0625x the model size and parameters of GRITv2-L. Furthermore, we also address the need for model compression, an area crucial for deploying efficient models on resource-constrained platforms. By applying quantization techniques, we efficiently reduced the GRITv2-S size to 22MB and deployed it on the flagship OnePlus 12 mobile which still surpasses the PISC-fine benchmarks in performance, highlighting the practical viability and improved efficiency of our model on mobile devices.

Title: Deep adaptative spectral zoom for improved remote heart rate estimation

• Authors: Joaquim Comas, Adria Ruiz, Federico Sukno
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06902
• Pdf URL: https://arxiv.org/pdf/2403.06902
• Copy Paste: [[2403.06902]] Deep adaptative spectral zoom for improved remote heart rate estimation(https://arxiv.org/abs/2403.06902)
• Keywords: robust
• Abstract: Recent advances in remote heart rate measurement, motivated by data-driven approaches, have notably enhanced accuracy. However, these improvements primarily focus on recovering the rPPG signal, overlooking the implicit challenges of estimating the heart rate (HR) from the derived signal. While many methods employ the Fast Fourier Transform (FFT) for HR estimation, the performance of the FFT is inherently affected by a limited frequency resolution. In contrast, the Chirp-Z Transform (CZT), a generalization form of FFT, can refine the spectrum to the narrow-band range of interest for heart rate, providing improved frequential resolution and, consequently, more accurate estimation. This paper presents the advantages of employing the CZT for remote HR estimation and introduces a novel data-driven adaptive CZT estimator. The objective of our proposed model is to tailor the CZT to match the characteristics of each specific dataset sensor, facilitating a more optimal and accurate estimation of HR from the rPPG signal without compromising generalization across diverse datasets. This is achieved through a Sparse Matrix Optimization (SMO). We validate the effectiveness of our model through exhaustive evaluations on three publicly available datasets UCLA-rPPG, PURE, and UBFC-rPPG employing both intra- and cross-database performance metrics. The results reveal outstanding heart rate estimation capabilities, establishing the proposed approach as a robust and versatile estimator for any rPPG method.

Title: Towards Incident Response Orchestration and Automation for the Advanced Metering Infrastructure

• Authors: Alexios Lekidis, Vasileios Mavroeidis, Konstantinos Fysarakis
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.06907
• Pdf URL: https://arxiv.org/pdf/2403.06907
• Copy Paste: [[2403.06907]] Towards Incident Response Orchestration and Automation for the Advanced Metering Infrastructure(https://arxiv.org/abs/2403.06907)
• Keywords: security, attack
• Abstract: The threat landscape of industrial infrastructures has expanded exponentially over the last few years. Such infrastructures include services such as the smart meter data exchange that should have real-time availability. Smart meters constitute the main component of the Advanced Metering Infrastructure, and their measurements are also used as historical data for forecasting the energy demand to avoid load peaks that could lead to blackouts within specific areas. Hence, a comprehensive Incident Response plan must be in place to ensure high service availability in case of cyber-attacks or operational errors. Currently, utility operators execute such plans mostly manually, requiring extensive time, effort, and domain expertise, and they are prone to human errors. In this paper, we present a method to provide an orchestrated and highly automated Incident Response plan targeting specific use cases and attack scenarios in the energy sector, including steps for preparedness, detection and analysis, containment, eradication, recovery, and post-incident activity through the use of playbooks. In particular, we use the OASIS Collaborative Automated Course of Action Operations (CACAO) standard to define highly automatable workflows in support of cyber security operations for the Advanced Metering Infrastructure. The proposed method is validated through an Advanced Metering Infrastructure testbed where the most prominent cyber-attacks are emulated, and playbooks are instantiated to ensure rapid response for the containment and eradication of the threat, business continuity on the smart meter data exchange service, and compliance with incident reporting requirements.

Title: MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning

• Authors: Yichuan Li, Xiyao Ma, Sixing Lu, Kyumin Lee, Xiaohu Liu, Chenlei Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.06914
• Pdf URL: https://arxiv.org/pdf/2403.06914
• Copy Paste: [[2403.06914]] MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning(https://arxiv.org/abs/2403.06914)
• Keywords: large language model
• Abstract: Large Language models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities, where a LLM makes predictions for a given test input together with a few input-output pairs (demonstrations). Nevertheless, the inclusion of demonstrations leads to a quadratic increase in the computational overhead of the self-attention mechanism. Existing solutions attempt to distill lengthy demonstrations into compact vectors. However, they often require task-specific retraining or compromise LLM's in-context learning performance. To mitigate these challenges, we present Meta dEmonstratioN Distillation (MEND), where a language model learns to distill any lengthy demonstrations into vectors without retraining for a new downstream task. We exploit the knowledge distillation to enhance alignment between MEND and LLM, achieving both efficiency and effectiveness simultaneously. MEND is endowed with the meta-knowledge of distilling demonstrations through a two-stage training process, which includes meta-distillation pretraining and fine-tuning. Comprehensive evaluations across seven diverse ICL task partitions using decoder-only (GPT-2) and encoder-decoder (T5) attest to MEND's prowess. It not only matches but often outperforms the Vanilla ICL as well as other state-of-the-art distillation models, while significantly reducing the computational demands. This innovation promises enhanced scalability and efficiency for the practical deployment of large language models

Title: Simplicity Bias of Transformers to Learn Low Sensitivity Functions

• Authors: Bhavya Vasudeva, Deqing Fu, Tianyi Zhou, Elliott Kau, Youqi Huang, Vatsal Sharan
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.06925
• Pdf URL: https://arxiv.org/pdf/2403.06925
• Copy Paste: [[2403.06925]] Simplicity Bias of Transformers to Learn Low Sensitivity Functions(https://arxiv.org/abs/2403.06925)
• Keywords: robust, transformer
• Abstract: Transformers achieve state-of-the-art accuracy and robustness across many tasks, but an understanding of the inductive biases that they have and how those biases are different from other neural network architectures remains elusive. Various neural network architectures such as fully connected networks have been found to have a simplicity bias towards simple functions of the data; one version of this simplicity bias is a spectral bias to learn simple functions in the Fourier space. In this work, we identify the notion of sensitivity of the model to random changes in the input as a notion of simplicity bias which provides a unified metric to explain the simplicity and spectral bias of transformers across different data modalities. We show that transformers have lower sensitivity than alternative architectures, such as LSTMs, MLPs and CNNs, across both vision and language tasks. We also show that low-sensitivity bias correlates with improved robustness; furthermore, it can also be used as an efficient intervention to further improve the robustness of transformers.

Title: ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysis

• Authors: Yanming Liu, Xinyue Peng, Tianyu Du, Jianwei Yin, Weihao Liu, Xuhong Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06932
• Pdf URL: https://arxiv.org/pdf/2403.06932
• Copy Paste: [[2403.06932]] ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysis(https://arxiv.org/abs/2403.06932)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved commendable accomplishments in various natural language processing tasks. However, LLMs still encounter significant challenges when dealing with complex scenarios involving multiple entities. These challenges arise from the presence of implicit relationships that demand multi-step reasoning. In this paper, we propose a novel approach ERA-CoT, which aids LLMs in understanding context by capturing relationships between entities and supports the reasoning of diverse tasks through Chain-of-Thoughts (CoT). Experimental results show that ERA-CoT demonstrates the superior performance of our proposed method compared to current CoT prompting methods, achieving a significant improvement of an average of 5.1\% on GPT3.5 compared to previous SOTA baselines. Our analysis indicates that ERA-CoT increases the LLM's understanding of entity relationships, significantly improves the accuracy of question answering, and enhances the reasoning ability of LLMs.

Title: Naming, Describing, and Quantifying Visual Objects in Humans and LLMs

• Authors: Alberto Testoni, Juell Sprott, Sandro Pezzelle
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06935
• Pdf URL: https://arxiv.org/pdf/2403.06935
• Copy Paste: [[2403.06935]] Naming, Describing, and Quantifying Visual Objects in Humans and LLMs(https://arxiv.org/abs/2403.06935)
• Keywords: large language model
• Abstract: While human speakers use a variety of different expressions when describing the same object in an image, giving rise to a distribution of plausible labels driven by pragmatic constraints, the extent to which current Vision \& Language Large Language Models (VLLMs) can mimic this crucial feature of language use is an open question. This applies to common, everyday objects, but it is particularly interesting for uncommon or novel objects for which a category label may be lacking or fuzzy. Furthermore, humans show clear production preferences for highly context-sensitive expressions, such as the quantifiers `few' or `most'. In our work, we evaluate VLLMs (FROMAGe, BLIP-2, LLaVA) on three categories (nouns, attributes, and quantifiers) where humans show great subjective variability concerning the distribution over plausible labels, using datasets and resources mostly under-explored in previous work. Our results reveal mixed evidence on the ability of VLLMs to capture human naming preferences, with all models failing in tasks that require high-level reasoning such as assigning quantifiers.

Title: Advancing Generalizable Remote Physiological Measurement through the Integration of Explicit and Implicit Prior Knowledge

• Authors: Yuting Zhang, Hao Lu, Xin Liu, Yingcong Chen, Kaishun Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06947
• Pdf URL: https://arxiv.org/pdf/2403.06947
• Copy Paste: [[2403.06947]] Advancing Generalizable Remote Physiological Measurement through the Integration of Explicit and Implicit Prior Knowledge(https://arxiv.org/abs/2403.06947)
• Keywords: security
• Abstract: Remote photoplethysmography (rPPG) is a promising technology that captures physiological signals from face videos, with potential applications in medical health, emotional computing, and biosecurity recognition. The demand for rPPG tasks has expanded from demonstrating good performance on intra-dataset testing to cross-dataset testing (i.e., domain generalization). However, most existing methods have overlooked the prior knowledge of rPPG, resulting in poor generalization ability. In this paper, we propose a novel framework that simultaneously utilizes explicit and implicit prior knowledge in the rPPG task. Specifically, we systematically analyze the causes of noise sources (e.g., different camera, lighting, skin types, and movement) across different domains and incorporate these prior knowledge into the network. Additionally, we leverage a two-branch network to disentangle the physiological feature distribution from noises through implicit label correlation. Our extensive experiments demonstrate that the proposed method not only outperforms state-of-the-art methods on RGB cross-dataset evaluation but also generalizes well from RGB datasets to NIR datasets. The code is available at https://github.com/keke-nice/Greip.

Title: DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations

• Authors: Tianhao Qi, Shancheng Fang, Yanze Wu, Hongtao Xie, Jiawei Liu, Lang Chen, Qian He, Yongdong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06951
• Pdf URL: https://arxiv.org/pdf/2403.06951
• Copy Paste: [[2403.06951]] DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations(https://arxiv.org/abs/2403.06951)
• Keywords: diffusion
• Abstract: The diffusion-based text-to-image model harbors immense potential in transferring reference style. However, current encoder-based approaches significantly impair the text controllability of text-to-image models while transferring styles. In this paper, we introduce \textit{DEADiff} to address this issue using the following two strategies: 1) a mechanism to decouple the style and semantics of reference images. The decoupled feature representations are first extracted by Q-Formers which are instructed by different text descriptions. Then they are injected into mutually exclusive subsets of cross-attention layers for better disentanglement. 2) A non-reconstructive learning method. The Q-Formers are trained using paired images rather than the identical target, in which the reference image and the ground-truth image are with the same style or semantics. We show that DEADiff attains the best visual stylization results and optimal balance between the text controllability inherent in the text-to-image model and style similarity to the reference image, as demonstrated both quantitatively and qualitatively. Our project page is~\href{https://tianhao-qi.github.io/DEADiff/}{https://tianhao-qi.github.io/DEADiff/}.

Title: SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data

• Authors: Jialu Li, Jaemin Cho, Yi-Lin Sung, Jaehong Yoon, Mohit Bansal
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06952
• Pdf URL: https://arxiv.org/pdf/2403.06952
• Copy Paste: [[2403.06952]] SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data(https://arxiv.org/abs/2403.06952)
• Keywords: diffusion
• Abstract: Recent text-to-image (T2I) generation models have demonstrated impressive capabilities in creating images from text descriptions. However, these T2I generation models often fall short of generating images that precisely match the details of the text inputs, such as incorrect spatial relationship or missing objects. In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging. First, SELMA leverages an LLM's in-context learning capability to generate multiple datasets of text prompts that can teach different skills, and then generates the images with a T2I model based on the prompts. Next, SELMA adapts the T2I model to the new skills by learning multiple single-skill LoRA (low-rank adaptation) experts followed by expert merging. Our independent expert fine-tuning specializes multiple models for different skills, and expert merging helps build a joint multi-skill T2I model that can generate faithful images given diverse text prompts, while mitigating the knowledge conflict from different datasets. We empirically demonstrate that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-art T2I diffusion models on multiple benchmarks (+2.1% on TIFA and +6.9% on DSG), human preference metrics (PickScore, ImageReward, and HPS), as well as human evaluation. Moreover, fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data. Lastly, we show that fine-tuning with images from a weaker T2I model can help improve the generation quality of a stronger T2I model, suggesting promising weak-to-strong generalization in T2I models.

Title: Explainable Transformer Prototypes for Medical Diagnoses

• Authors: Ugur Demir, Debesh Jha, Zheyuan Zhang, Elif Keles, Bradley Allen, Aggelos K. Katsaggelos, Ulas Bagci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06961
• Pdf URL: https://arxiv.org/pdf/2403.06961
• Copy Paste: [[2403.06961]] Explainable Transformer Prototypes for Medical Diagnoses(https://arxiv.org/abs/2403.06961)
• Keywords: explainability, transformer
• Abstract: Deployments of artificial intelligence in medical diagnostics mandate not just accuracy and efficacy but also trust, emphasizing the need for explainability in machine decisions. The recent trend in automated medical image diagnostics leans towards the deployment of Transformer-based architectures, credited to their impressive capabilities. Since the self-attention feature of transformers contributes towards identifying crucial regions during the classification process, they enhance the trustability of the methods. However, the complex intricacies of these attention mechanisms may fall short of effectively pinpointing the regions of interest directly influencing AI decisions. Our research endeavors to innovate a unique attention block that underscores the correlation between 'regions' rather than 'pixels'. To address this challenge, we introduce an innovative system grounded in prototype learning, featuring an advanced self-attention mechanism that goes beyond conventional ad-hoc visual explanation techniques by offering comprehensible visual insights. A combined quantitative and qualitative methodological approach was used to demonstrate the effectiveness of the proposed method on the large-scale NIH chest X-ray dataset. Experimental results showed that our proposed method offers a promising direction for explainability, which can lead to the development of more trustable systems, which can facilitate easier and rapid adoption of such technology into routine clinics. The code is available at www.github.com/NUBagcilab/r2r_proto.

Title: The pitfalls of next-token prediction

• Authors: Gregor Bachmann, Vaishnavh Nagarajan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06963
• Pdf URL: https://arxiv.org/pdf/2403.06963
• Copy Paste: [[2403.06963]] The pitfalls of next-token prediction(https://arxiv.org/abs/2403.06963)
• Keywords: transformer
• Abstract: Can a mere next-token predictor faithfully model human intelligence? We crystallize this intuitive concern, which is fragmented in the literature. As a starting point, we argue that the two often-conflated phases of next-token prediction -- autoregressive inference and teacher-forced training -- must be treated distinctly. The popular criticism that errors can compound during autoregressive inference, crucially assumes that teacher-forcing has learned an accurate next-token predictor. This assumption sidesteps a more deep-rooted problem we expose: in certain classes of tasks, teacher-forcing can simply fail to learn an accurate next-token predictor in the first place. We describe a general mechanism of how teacher-forcing can fail, and design a minimal planning task where both the Transformer and the Mamba architecture empirically fail in that manner -- remarkably, despite the task being straightforward to learn. We provide preliminary evidence that this failure can be resolved when training to predict multiple tokens in advance. We hope this finding can ground future debates and inspire explorations beyond the next-token prediction paradigm. We make our code available under https://github.com/gregorbachmann/Next-Token-Failures

Title: Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena

• Authors: Leonie Weissweiler, Abdullatif Köksal, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06965
• Pdf URL: https://arxiv.org/pdf/2403.06965
• Copy Paste: [[2403.06965]] Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena(https://arxiv.org/abs/2403.06965)
• Keywords: large language model
• Abstract: Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (CxG). For example, the caused-motion construction (CMC, ``She sneezed the foam off her cappuccino'') demonstrates that constructions must carry meaning, otherwise the fact that ``sneeze'' in this context causes movement cannot be explained. We form the hypothesis that this remains challenging even for state-of-the-art Large Language Models (LLMs), for which we devise a test based on substituting the verb with a prototypical motion verb. To be able to perform this test at statistically significant scale, in the absence of adequate CxG corpora, we develop a novel pipeline of NLP-assisted collection of linguistically annotated text. We show how dependency parsing and GPT-3.5 can be used to significantly reduce annotation cost and thus enable the annotation of rare phenomena at scale. We then evaluate GPT, Gemini, Llama2 and Mistral models for their understanding of the CMC using the newly collected corpus. We find that all models struggle with understanding the motion component that the CMC adds to a sentence.

Title: MRL Parsing Without Tears: The Case of Hebrew

• Authors: Shaltiel Shmidman, Avi Shmidman, Moshe Koppel, Reut Tsarfaty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.06970
• Pdf URL: https://arxiv.org/pdf/2403.06970
• Copy Paste: [[2403.06970]] MRL Parsing Without Tears: The Case of Hebrew(https://arxiv.org/abs/2403.06970)
• Keywords: extraction, segmentation
• Abstract: Syntactic parsing remains a critical tool for relation extraction and information extraction, especially in resource-scarce languages where LLMs are lacking. Yet in morphologically rich languages (MRLs), where parsers need to identify multiple lexical units in each token, existing systems suffer in latency and setup complexity. Some use a pipeline to peel away the layers: first segmentation, then morphology tagging, and then syntax parsing; however, errors in earlier layers are then propagated forward. Others use a joint architecture to evaluate all permutations at once; while this improves accuracy, it is notoriously slow. In contrast, and taking Hebrew as a test case, we present a new "flipped pipeline": decisions are made directly on the whole-token units by expert classifiers, each one dedicated to one specific task. The classifiers are independent of one another, and only at the end do we synthesize their predictions. This blazingly fast approach sets a new SOTA in Hebrew POS tagging and dependency parsing, while also reaching near-SOTA performance on other Hebrew NLP tasks. Because our architecture does not rely on any language-specific resources, it can serve as a model to develop similar parsers for other MRLs.

Title: Bayesian Diffusion Models for 3D Shape Reconstruction

• Authors: Haiyang Xu, Yu Lei, Zeyuan Chen, Xiang Zhang, Yue Zhao, Yilin Wang, Zhuowen Tu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.06973
• Pdf URL: https://arxiv.org/pdf/2403.06973
• Copy Paste: [[2403.06973]] Bayesian Diffusion Models for 3D Shape Reconstruction(https://arxiv.org/abs/2403.06973)
• Keywords: diffusion
• Abstract: We present Bayesian Diffusion Models (BDM), a prediction algorithm that performs effective Bayesian inference by tightly coupling the top-down (prior) information with the bottom-up (data-driven) procedure via joint diffusion processes. We show the effectiveness of BDM on the 3D shape reconstruction task. Compared to prototypical deep learning data-driven approaches trained on paired (supervised) data-labels (e.g. image-point clouds) datasets, our BDM brings in rich prior information from standalone labels (e.g. point clouds) to improve the bottom-up 3D reconstruction. As opposed to the standard Bayesian frameworks where explicit prior and likelihood are required for the inference, BDM performs seamless information fusion via coupled diffusion processes with learned gradient computation networks. The specialty of our BDM lies in its capability to engage the active and effective information exchange and fusion of the top-down and bottom-up processes where each itself is a diffusion process. We demonstrate state-of-the-art results on both synthetic and real-world benchmarks for 3D shape reconstruction.

Title: BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion

• Authors: Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, Qiang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06976
• Pdf URL: https://arxiv.org/pdf/2403.06976
• Copy Paste: [[2403.06976]] BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion(https://arxiv.org/abs/2403.06976)
• Keywords: diffusion, segmentation
• Abstract: Image inpainting, the process of restoring corrupted images, has seen significant advancements with the advent of diffusion models (DMs). Despite these advancements, current DM adaptations for inpainting, which involve modifications to the sampling strategy or the development of inpainting-specific DMs, frequently suffer from semantic inconsistencies and reduced image quality. Addressing these challenges, our work introduces a novel paradigm: the division of masked image features and noisy latent into separate branches. This division dramatically diminishes the model's learning load, facilitating a nuanced incorporation of essential masked image information in a hierarchical fashion. Herein, we present BrushNet, a novel plug-and-play dual-branch model engineered to embed pixel-level masked image features into any pre-trained DM, guaranteeing coherent and enhanced image inpainting outcomes. Additionally, we introduce BrushData and BrushBench to facilitate segmentation-based inpainting training and performance assessment. Our extensive experimental analysis demonstrates BrushNet's superior performance over existing models across seven key metrics, including image quality, mask region preservation, and textual coherence.

Title: VideoMamba: State Space Model for Efficient Video Understanding

• Authors: Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, Yu Qiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06977
• Pdf URL: https://arxiv.org/pdf/2403.06977
• Copy Paste: [[2403.06977]] VideoMamba: State Space Model for Efficient Video Understanding(https://arxiv.org/abs/2403.06977)
• Keywords: robust, transformer
• Abstract: Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain. The proposed VideoMamba overcomes the limitations of existing 3D convolution neural networks and video transformers. Its linear-complexity operator enables efficient long-term modeling, which is crucial for high-resolution long video understanding. Extensive evaluations reveal VideoMamba's four core abilities: (1) Scalability in the visual domain without extensive dataset pretraining, thanks to a novel self-distillation technique; (2) Sensitivity for recognizing short-term actions even with fine-grained motion differences; (3) Superiority in long-term video understanding, showcasing significant advancements over traditional feature-based models; and (4) Compatibility with other modalities, demonstrating robustness in multi-modal contexts. Through these distinct advantages, VideoMamba sets a new benchmark for video understanding, offering a scalable and efficient solution for comprehensive video understanding. All the code and models are available at https://github.com/OpenGVLab/VideoMamba.

Title: Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling

• Authors: Wele Gedara Chaminda Bandara, Vishal M. Patel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.06978
• Pdf URL: https://arxiv.org/pdf/2403.06978
• Copy Paste: [[2403.06978]] Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling(https://arxiv.org/abs/2403.06978)
• Keywords: robust, transformer
• Abstract: In this paper, we introduce Attention Prompt Tuning (APT) - a computationally efficient variant of prompt tuning for video-based applications such as action recognition. Prompt tuning approaches involve injecting a set of learnable prompts along with data tokens during fine-tuning while keeping the backbone frozen. This approach greatly reduces the number of learnable parameters compared to full tuning. For image-based downstream tasks, normally a couple of learnable prompts achieve results close to those of full tuning. However, videos, which contain more complex spatiotemporal information, require hundreds of tunable prompts to achieve reasonably good results. This reduces the parameter efficiency observed in images and significantly increases latency and the number of floating-point operations (FLOPs) during inference. To tackle these issues, we directly inject the prompts into the keys and values of the non-local attention mechanism within the transformer block. Additionally, we introduce a novel prompt reparameterization technique to make APT more robust against hyperparameter selection. The proposed APT approach greatly reduces the number of FLOPs and latency while achieving a significant performance boost over the existing parameter-efficient tuning methods on UCF101, HMDB51, and SSv2 datasets for action recognition. The code and pre-trained models are available at https://github.com/wgcban/apt