2024-04-23

Title: FlowMind: Automatic Workflow Generation with LLMs

• Authors: Zhen Zeng, William Watson, Nicole Cho, Saba Rahimi, Shayleen Reynolds, Tucker Balch, Manuela Veloso
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13050
• Pdf URL: https://arxiv.org/pdf/2404.13050
• Copy Paste: [[2404.13050]] FlowMind: Automatic Workflow Generation with LLMs(https://arxiv.org/abs/2404.13050)
• Keywords: transformer, generative, large language model
• Abstract: The rapidly evolving field of Robotic Process Automation (RPA) has made significant strides in automating repetitive processes, yet its effectiveness diminishes in scenarios requiring spontaneous or unpredictable tasks demanded by users. This paper introduces a novel approach, FlowMind, leveraging the capabilities of Large Language Models (LLMs) such as Generative Pretrained Transformer (GPT), to address this limitation and create an automatic workflow generation system. In FlowMind, we propose a generic prompt recipe for a lecture that helps ground LLM reasoning with reliable Application Programming Interfaces (APIs). With this, FlowMind not only mitigates the common issue of hallucinations in LLMs, but also eliminates direct interaction between LLMs and proprietary data or code, thus ensuring the integrity and confidentiality of information - a cornerstone in financial services. FlowMind further simplifies user interaction by presenting high-level descriptions of auto-generated workflows, enabling users to inspect and provide feedback effectively. We also introduce NCEN-QA, a new dataset in finance for benchmarking question-answering tasks from N-CEN reports on funds. We used NCEN-QA to evaluate the performance of workflows generated by FlowMind against baseline and ablation variants of FlowMind. We demonstrate the success of FlowMind, the importance of each component in the proposed lecture recipe, and the effectiveness of user interaction and feedback in FlowMind.

Title: Leveraging Large Language Model as Simulated Patients for Clinical Education

• Authors: Yaneng Li, Cheng Zeng, Jialun Zhong, Ruoyu Zhang, Minhao Zhang, Lei Zou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13066
• Pdf URL: https://arxiv.org/pdf/2404.13066
• Copy Paste: [[2404.13066]] Leveraging Large Language Model as Simulated Patients for Clinical Education(https://arxiv.org/abs/2404.13066)
• Keywords: large language model
• Abstract: Simulated Patients (SPs) play a crucial role in clinical medical education by providing realistic scenarios for student practice. However, the high cost of training and hiring qualified SPs, along with the heavy workload and potential risks they face in consistently portraying actual patients, limit students' access to this type of clinical training. Consequently, the integration of computer program-based simulated patients has emerged as a valuable educational tool in recent years. With the rapid development of Large Language Models (LLMs), their exceptional capabilities in conversational artificial intelligence and role-playing have been demonstrated, making them a feasible option for implementing Virtual Simulated Patient (VSP). In this paper, we present an integrated model-agnostic framework called CureFun that harnesses the potential of LLMs in clinical medical education. This framework facilitates natural conversations between students and simulated patients, evaluates their dialogue, and provides suggestions to enhance students' clinical inquiry skills. Through comprehensive evaluations, our approach demonstrates more authentic and professional SP-scenario dialogue flows compared to other LLM-based chatbots, thus proving its proficiency in simulating patients. Additionally, leveraging CureFun's evaluation ability, we assess several medical LLMs and discuss the possibilities and limitations of using LLMs as virtual doctors from the perspective of their diagnostic abilities.

Title: Towards Efficient Resume Understanding: A Multi-Granularity Multi-Modal Pre-Training Approach

• Authors: Feihu Jiang, Chuan Qin, Jingshuai Zhang, Kaichun Yao, Xi Chen, Dazhong Shen, Chen Zhu, Hengshu Zhu, Hui Xiong
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13067
• Pdf URL: https://arxiv.org/pdf/2404.13067
• Copy Paste: [[2404.13067]] Towards Efficient Resume Understanding: A Multi-Granularity Multi-Modal Pre-Training Approach(https://arxiv.org/abs/2404.13067)
• Keywords: transformer
• Abstract: In the contemporary era of widespread online recruitment, resume understanding has been widely acknowledged as a fundamental and crucial task, which aims to extract structured information from resume documents automatically. Compared to the traditional rule-based approaches, the utilization of recently proposed pre-trained document understanding models can greatly enhance the effectiveness of resume understanding. The present approaches have, however, disregarded the hierarchical relations within the structured information presented in resumes, and have difficulty parsing resumes in an efficient manner. To this end, in this paper, we propose a novel model, namely ERU, to achieve efficient resume understanding. Specifically, we first introduce a layout-aware multi-modal fusion transformer for encoding the segments in the resume with integrated textual, visual, and layout information. Then, we design three self-supervised tasks to pre-train this module via a large number of unlabeled resumes. Next, we fine-tune the model with a multi-granularity sequence labeling task to extract structured information from resumes. Finally, extensive experiments on a real-world dataset clearly demonstrate the effectiveness of ERU.

Title: Evidence from counterfactual tasks supports emergent analogical reasoning in large language models

• Authors: Taylor Webb, Keith J. Holyoak, Hongjing Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13070
• Pdf URL: https://arxiv.org/pdf/2404.13070
• Copy Paste: [[2404.13070]] Evidence from counterfactual tasks supports emergent analogical reasoning in large language models(https://arxiv.org/abs/2404.13070)
• Keywords: large language model
• Abstract: We recently reported evidence that large language models are capable of solving a wide range of text-based analogy problems in a zero-shot manner, indicating the presence of an emergent capacity for analogical reasoning. Two recent commentaries have challenged these results, citing evidence from so-called `counterfactual' tasks in which the standard sequence of the alphabet is arbitrarily permuted so as to decrease similarity with materials that may have been present in the language model's training data. Here, we reply to these critiques, clarifying some misunderstandings about the test materials used in our original work, and presenting evidence that language models are also capable of generalizing to these new counterfactual task variants.

Title: Modeling Emotions and Ethics with Large Language Models

• Authors: Edward Y. Chang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13071
• Pdf URL: https://arxiv.org/pdf/2404.13071
• Copy Paste: [[2404.13071]] Modeling Emotions and Ethics with Large Language Models(https://arxiv.org/abs/2404.13071)
• Keywords: large language model
• Abstract: This paper explores the integration of human-like emotions and ethical considerations into Large Language Models (LLMs). We first model eight fundamental human emotions, presented as opposing pairs, and employ collaborative LLMs to reinterpret and express these emotions across a spectrum of intensity. Our focus extends to embedding a latent ethical dimension within LLMs, guided by a novel self-supervised learning algorithm with human feedback (SSHF). This approach enables LLMs to perform self-evaluations and adjustments concerning ethical guidelines, enhancing their capability to generate content that is not only emotionally resonant but also ethically aligned. The methodologies and case studies presented herein illustrate the potential of LLMs to transcend mere text and image generation, venturing into the realms of empathetic interaction and principled decision-making, thereby setting a new precedent in the development of emotionally aware and ethically conscious AI systems.

Title: Towards Compositionally Generalizable Semantic Parsing in Large Language Models: A Survey

• Authors: Amogh Mannekote
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13074
• Pdf URL: https://arxiv.org/pdf/2404.13074
• Copy Paste: [[2404.13074]] Towards Compositionally Generalizable Semantic Parsing in Large Language Models: A Survey(https://arxiv.org/abs/2404.13074)
• Keywords: large language model
• Abstract: Compositional generalization is the ability of a model to generalize to complex, previously unseen types of combinations of entities from just having seen the primitives. This type of generalization is particularly relevant to the semantic parsing community for applications such as task-oriented dialogue, text-to-SQL parsing, and information retrieval, as they can harbor infinite complexity. Despite the success of large language models (LLMs) in a wide range of NLP tasks, unlocking perfect compositional generalization still remains one of the few last unsolved frontiers. The past few years has seen a surge of interest in works that explore the limitations of, methods to improve, and evaluation metrics for compositional generalization capabilities of LLMs for semantic parsing tasks. In this work, we present a literature survey geared at synthesizing recent advances in analysis, methods, and evaluation schemes to offer a starting point for both practitioners and researchers in this area.

Title: LLM Evaluators Recognize and Favor Their Own Generations

• Authors: Arjun Panickssery, Samuel R. Bowman, Shi Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13076
• Pdf URL: https://arxiv.org/pdf/2404.13076
• Copy Paste: [[2404.13076]] LLM Evaluators Recognize and Favor Their Own Generations(https://arxiv.org/abs/2404.13076)
• Keywords: large language model
• Abstract: Self-evaluation using large language models (LLMs) has proven valuable not only in benchmarking but also methods like reward modeling, constitutional AI, and self-refinement. But new biases are introduced due to the same LLM acting as both the evaluator and the evaluatee. One such bias is self-preference, where an LLM evaluator scores its own outputs higher than others' while human annotators consider them of equal quality. But do LLMs actually recognize their own outputs when they give those texts higher scores, or is it just a coincidence? In this paper, we investigate if self-recognition capability contributes to self-preference. We discover that, out of the box, LLMs such as GPT-4 and Llama 2 have non-trivial accuracy at distinguishing themselves from other LLMs and humans. By fine-tuning LLMs, we discover a linear correlation between self-recognition capability and the strength of self-preference bias; using controlled experiments, we show that the causal explanation resists straightforward confounders. We discuss how self-recognition can interfere with unbiased evaluations and AI safety more generally.

Title: Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning

• Authors: Yilin Gao, Sai Kumar Arava, Yancheng Li, James W. Snyder Jr
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13077
• Pdf URL: https://arxiv.org/pdf/2404.13077
• Copy Paste: [[2404.13077]] Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning(https://arxiv.org/abs/2404.13077)
• Keywords: large language model
• Abstract: Artificial intelligence (AI) is widely deployed to solve problems related to marketing attribution and budget optimization. However, AI models can be quite complex, and it can be difficult to understand model workings and insights without extensive implementation teams. In principle, recently developed large language models (LLMs), like GPT-4, can be deployed to provide marketing insights, reducing the time and effort required to make critical decisions. In practice, there are substantial challenges that need to be overcome to reliably use such models. We focus on domain-specific question-answering, SQL generation needed for data retrieval, and tabular analysis and show how a combination of semantic search, prompt engineering, and fine-tuning can be applied to dramatically improve the ability of LLMs to execute these tasks accurately. We compare both proprietary models, like GPT-4, and open-source models, like Llama-2-70b, as well as various embedding methods. These models are tested on sample use cases specific to marketing mix modeling and attribution.

Title: Empowering Interdisciplinary Research with BERT-Based Models: An Approach Through SciBERT-CNN with Topic Modeling

• Authors: Darya Likhareva, Hamsini Sankaran, Sivakumar Thiyagarajan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13078
• Pdf URL: https://arxiv.org/pdf/2404.13078
• Copy Paste: [[2404.13078]] Empowering Interdisciplinary Research with BERT-Based Models: An Approach Through SciBERT-CNN with Topic Modeling(https://arxiv.org/abs/2404.13078)
• Keywords: extraction
• Abstract: Researchers must stay current in their fields by regularly reviewing academic literature, a task complicated by the daily publication of thousands of papers. Traditional multi-label text classification methods often ignore semantic relationships and fail to address the inherent class imbalances. This paper introduces a novel approach using the SciBERT model and CNNs to systematically categorize academic abstracts from the Elsevier OA CC-BY corpus. We use a multi-segment input strategy that processes abstracts, body text, titles, and keywords obtained via BERT topic modeling through SciBERT. Here, the [CLS] token embeddings capture the contextual representation of each segment, concatenated and processed through a CNN. The CNN uses convolution and pooling to enhance feature extraction and reduce dimensionality, optimizing the data for classification. Additionally, we incorporate class weights based on label frequency to address the class imbalance, significantly improving the classification F1 score and enhancing text classification systems and literature review efficiency.

Title: Relational Graph Convolutional Networks for Sentiment Analysis

• Authors: Asal Khosravi, Zahed Rahmati, Ali Vefghi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13079
• Pdf URL: https://arxiv.org/pdf/2404.13079
• Copy Paste: [[2404.13079]] Relational Graph Convolutional Networks for Sentiment Analysis(https://arxiv.org/abs/2404.13079)
• Keywords: interpretability
• Abstract: With the growth of textual data across online platforms, sentiment analysis has become crucial for extracting insights from user-generated content. While traditional approaches and deep learning models have shown promise, they cannot often capture complex relationships between entities. In this paper, we propose leveraging Relational Graph Convolutional Networks (RGCNs) for sentiment analysis, which offer interpretability and flexibility by capturing dependencies between data points represented as nodes in a graph. We demonstrate the effectiveness of our approach by using pre-trained language models such as BERT and RoBERTa with RGCN architecture on product reviews from Amazon and Digikala datasets and evaluating the results. Our experiments highlight the effectiveness of RGCNs in capturing relational information for sentiment analysis tasks.

Title: SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs

• Authors: Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, Jinwoo Shin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13081
• Pdf URL: https://arxiv.org/pdf/2404.13081
• Copy Paste: [[2404.13081]] SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs(https://arxiv.org/abs/2404.13081)
• Keywords: large language model
• Abstract: Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising way to improve QA with LLMs, the existing methods often require additional fine-tuning which becomes infeasible with recent LLMs. Augmenting retrieved passages via prompting has the potential to address this limitation, but this direction has been limitedly explored. To this end, we design a simple yet effective framework to enhance open-domain QA (ODQA) with LLMs, based on the summarized retrieval (SuRe). SuRe helps LLMs predict more accurate answers for a given question, which are well-supported by the summarized retrieval that could be viewed as an explicit rationale extracted from the retrieved passages. Specifically, SuRe first constructs summaries of the retrieved passages for each of the multiple answer candidates. Then, SuRe confirms the most plausible answer from the candidate set by evaluating the validity and ranking of the generated summaries. Experimental results on diverse ODQA benchmarks demonstrate the superiority of SuRe, with improvements of up to 4.6% in exact match (EM) and 4.0% in F1 score over standard prompting approaches. SuRe also can be integrated with a broad range of retrieval methods and LLMs. Finally, the generated summaries from SuRe show additional advantages to measure the importance of retrieved passages and serve as more preferred rationales by models and humans.

Title: TREACLE: Thrifty Reasoning via Context-Aware LLM and Prompt Selection

• Authors: Xuechen Zhang, Zijian Huang, Ege Onur Taga, Carlee Joe-Wong, Samet Oymak, Jiasi Chen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13082
• Pdf URL: https://arxiv.org/pdf/2404.13082
• Copy Paste: [[2404.13082]] TREACLE: Thrifty Reasoning via Context-Aware LLM and Prompt Selection(https://arxiv.org/abs/2404.13082)
• Keywords: large language model
• Abstract: Recent successes in natural language processing have led to the proliferation of large language models (LLMs) by multiple providers. Each LLM offering has different inference accuracy, monetary cost, and latency, and their accuracy further depends on the exact wording of the question (i.e., the specific prompt). At the same time, users often have a limit on monetary budget and latency to answer all their questions, and they do not know which LLMs to choose for each question to meet their accuracy and long-term budget requirements. To navigate this rich design space, we propose TREACLE (Thrifty Reasoning via Context-Aware LLM and Prompt Selection), a reinforcement learning policy that jointly selects the model and prompting scheme while respecting the user's monetary cost and latency constraints. TREACLE uses the problem context, including question text embeddings (reflecting the type or difficulty of a query) and the response history (reflecting the consistency of previous responses) to make smart decisions. Our evaluations on standard reasoning datasets (GSM8K, CSQA, and LLC ) with various LLMs and prompts show that TREACLE enables cost savings of up to 85% compared to baselines while maintaining high accuracy. Importantly, it provides the user with the ability to gracefully trade off accuracy for cost.

Title: Demystifying Legalese: An Automated Approach for Summarizing and Analyzing Overlaps in Privacy Policies and Terms of Service

• Authors: Shikha Soneji, Mitchell Hoesing, Sujay Koujalgi, Jonathan Dodge
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13087
• Pdf URL: https://arxiv.org/pdf/2404.13087
• Copy Paste: [[2404.13087]] Demystifying Legalese: An Automated Approach for Summarizing and Analyzing Overlaps in Privacy Policies and Terms of Service(https://arxiv.org/abs/2404.13087)
• Keywords: privacy, transformer
• Abstract: The complexities of legalese in terms and policy documents can bind individuals to contracts they do not fully comprehend, potentially leading to uninformed data sharing. Our work seeks to alleviate this issue by developing language models that provide automated, accessible summaries and scores for such documents, aiming to enhance user understanding and facilitate informed decisions. We compared transformer-based and conventional models during training on our dataset, and RoBERTa performed better overall with a remarkable 0.74 F1-score. Leveraging our best-performing model, RoBERTa, we highlighted redundancies and potential guideline violations by identifying overlaps in GDPR-required documents, underscoring the necessity for stricter GDPR compliance.

Title: Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks

• Authors: Avinash Anand, Mohit Gupta, Kritarth Prasad, Navya Singla, Sanjana Sanjeev, Jatin Kumar, Adarsh Raj Shivam, Rajiv Ratn Shah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13099
• Pdf URL: https://arxiv.org/pdf/2404.13099
• Copy Paste: [[2404.13099]] Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks(https://arxiv.org/abs/2404.13099)
• Keywords: robust, large language model
• Abstract: The rapid progress in the field of natural language processing (NLP) systems and the expansion of large language models (LLMs) have opened up numerous opportunities in the field of education and instructional methods. These advancements offer the potential for tailored learning experiences and immediate feedback, all delivered through accessible and cost-effective services. One notable application area for this technological advancement is in the realm of solving mathematical problems. Mathematical problem-solving not only requires the ability to decipher complex problem statements but also the skill to perform precise arithmetic calculations at each step of the problem-solving process. However, the evaluation of the arithmetic capabilities of large language models remains an area that has received relatively little attention. In response, we introduce an extensive mathematics dataset called "MathQuest" sourced from the 11th and 12th standard Mathematics NCERT textbooks. This dataset encompasses mathematical challenges of varying complexity and covers a wide range of mathematical concepts. Utilizing this dataset, we conduct fine-tuning experiments with three prominent LLMs: LLaMA-2, WizardMath, and MAmmoTH. These fine-tuned models serve as benchmarks for evaluating their performance on our dataset. Our experiments reveal that among the three models, MAmmoTH-13B emerges as the most proficient, achieving the highest level of competence in solving the presented mathematical problems. Consequently, MAmmoTH-13B establishes itself as a robust and dependable benchmark for addressing NCERT mathematics problems.

Title: Multi Class Depression Detection Through Tweets using Artificial Intelligence

• Authors: Muhammad Osama Nusrat, Waseem Shahzad, Saad Ahmed Jamal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13104
• Pdf URL: https://arxiv.org/pdf/2404.13104
• Copy Paste: [[2404.13104]] Multi Class Depression Detection Through Tweets using Artificial Intelligence(https://arxiv.org/abs/2404.13104)
• Keywords: extraction, transformer
• Abstract: Depression is a significant issue nowadays. As per the World Health Organization (WHO), in 2023, over 280 million individuals are grappling with depression. This is a huge number; if not taken seriously, these numbers will increase rapidly. About 4.89 billion individuals are social media users. People express their feelings and emotions on platforms like Twitter, Facebook, Reddit, Instagram, etc. These platforms contain valuable information which can be used for research purposes. Considerable research has been conducted across various social media platforms. However, certain limitations persist in these endeavors. Particularly, previous studies were only focused on detecting depression and the intensity of depression in tweets. Also, there existed inaccuracies in dataset labeling. In this research work, five types of depression (Bipolar, major, psychotic, atypical, and postpartum) were predicted using tweets from the Twitter database based on lexicon labeling. Explainable AI was used to provide reasoning by highlighting the parts of tweets that represent type of depression. Bidirectional Encoder Representations from Transformers (BERT) was used for feature extraction and training. Machine learning and deep learning methodologies were used to train the model. The BERT model presented the most promising results, achieving an overall accuracy of 0.96.

Title: Towards Robust Real-Time Hardware-based Mobile Malware Detection using Multiple Instance Learning Formulation

• Authors: Harshit Kumar, Sudarshan Sharma, Biswadeep Chakraborty, Saibal Mukhopadhyay
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13125
• Pdf URL: https://arxiv.org/pdf/2404.13125
• Copy Paste: [[2404.13125]] Towards Robust Real-Time Hardware-based Mobile Malware Detection using Multiple Instance Learning Formulation(https://arxiv.org/abs/2404.13125)
• Keywords: robust
• Abstract: This study introduces RT-HMD, a Hardware-based Malware Detector (HMD) for mobile devices, that refines malware representation in segmented time-series through a Multiple Instance Learning (MIL) approach. We address the mislabeling issue in real-time HMDs, where benign segments in malware time-series incorrectly inherit malware labels, leading to increased false positives. Utilizing the proposed Malicious Discriminative Score within the MIL framework, RT-HMD effectively identifies localized malware behaviors, thereby improving the predictive accuracy. Empirical analysis, using a hardware telemetry dataset collected from a mobile platform across 723 benign and 1033 malware samples, shows a 5% precision boost while maintaining recall, outperforming baselines affected by mislabeled benign segments.

Title: On-board classification of underwater images using hybrid classical-quantum CNN based method

• Authors: Sreeraj Rajan Warrier, D Sri Harshavardhan Reddy, Sriya Bada, Rohith Achampeta, Sebastian Uppapalli, Jayasri Dontabhaktuni
• Subjects: cs.CV, quant-ph
• Abstract URL: https://arxiv.org/abs/2404.13130
• Pdf URL: https://arxiv.org/pdf/2404.13130
• Copy Paste: [[2404.13130]] On-board classification of underwater images using hybrid classical-quantum CNN based method(https://arxiv.org/abs/2404.13130)
• Keywords: extraction
• Abstract: Underwater images taken from autonomous underwater vehicles (AUV's) often suffer from low light, high turbidity, poor contrast, motion-blur and excessive light scattering and hence require image enhancement techniques for object recognition. Machine learning methods are being increasingly used for object recognition under such adverse conditions. These enhanced object recognition methods of images taken from AUV's has potential applications in underwater pipeline and optical fibre surveillance, ocean bed resource extraction, ocean floor mapping, underwater species exploration, etc. While the classical machine learning methods are very efficient in terms of accuracy, they require large datasets and high computational time for image classification. In the current work, we use quantum-classical hybrid machine learning methods for real-time under-water object recognition on-board an AUV for the first time. We use real-time motion-blurred and low-light images taken from an on-board camera of AUV built in-house and apply existing hybrid machine learning methods for object recognition. Our hybrid methods consist of quantum encoding and flattening of classical images using quantum circuits and sending them to classical neural networks for image classification. The results of hybrid methods carried out using Pennylane based quantum simulators both on GPU and using pre-trained models on an on-board NVIDIA GPU chipset are compared with results from corresponding classical machine learning methods. We observe that the hybrid quantum machine learning methods show an efficiency greater than 65\% and reduction in run-time by one-thirds and require 50\% smaller dataset sizes for training the models compared to classical machine learning methods. We hope that our work opens up further possibilities in quantum enhanced real-time computer vision in autonomous vehicles.

Title: Explainable AI for Fair Sepsis Mortality Predictive Model

• Authors: Chia-Hsuan Chang, Xiaoyang Wang, Christopher C. Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13139
• Pdf URL: https://arxiv.org/pdf/2404.13139
• Copy Paste: [[2404.13139]] Explainable AI for Fair Sepsis Mortality Predictive Model(https://arxiv.org/abs/2404.13139)
• Keywords: fair, explainability
• Abstract: Artificial intelligence supports healthcare professionals with predictive modeling, greatly transforming clinical decision-making. This study addresses the crucial need for fairness and explainability in AI applications within healthcare to ensure equitable outcomes across diverse patient demographics. By focusing on the predictive modeling of sepsis-related mortality, we propose a method that learns a performance-optimized predictive model and then employs the transfer learning process to produce a model with better fairness. Our method also introduces a novel permutation-based feature importance algorithm aiming at elucidating the contribution of each feature in enhancing fairness on predictions. Unlike existing explainability methods concentrating on explaining feature contribution to predictive performance, our proposed method uniquely bridges the gap in understanding how each feature contributes to fairness. This advancement is pivotal, given sepsis's significant mortality rate and its role in one-third of hospital deaths. Our method not only aids in identifying and mitigating biases within the predictive model but also fosters trust among healthcare stakeholders by improving the transparency and fairness of model predictions, thereby contributing to more equitable and trustworthy healthcare delivery.

Title: DeepFake-O-Meter v2.0: An Open Platform for DeepFake Detection

• Authors: Shuwei Hou, Yan Ju, Chengzhe Sun, Shan Jia, Lipeng Ke, Riky Zhou, Anita Nikolich, Siwei Lyu
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13146
• Pdf URL: https://arxiv.org/pdf/2404.13146
• Copy Paste: [[2404.13146]] DeepFake-O-Meter v2.0: An Open Platform for DeepFake Detection(https://arxiv.org/abs/2404.13146)
• Keywords: secure, security, privacy
• Abstract: Deepfakes, as AI-generated media, have increasingly threatened media integrity and personal privacy with realistic yet fake digital content. In this work, we introduce an open-source and user-friendly online platform, DeepFake-O-Meter v2.0, that integrates state-of-the-art methods for detecting Deepfake images, videos, and audio. Built upon DeepFake-O-Meter v1.0, we have made significant upgrades and improvements in platform architecture design, including user interaction, detector integration, job balancing, and security management. The platform aims to offer everyday users a convenient service for analyzing DeepFake media using multiple state-of-the-art detection algorithms. It ensures secure and private delivery of the analysis results. Furthermore, it serves as an evaluation and benchmarking platform for researchers in digital media forensics to compare the performance of multiple algorithms on the same input. We have also conducted detailed usage analysis based on the collected data to gain deeper insights into our platform's statistics. This involves analyzing two-month trends in user activity and evaluating the processing efficiency of each detector.

Title: BACS: Background Aware Continual Semantic Segmentation

• Authors: Mostafa ElAraby, Ali Harakeh, Liam Paull
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13148
• Pdf URL: https://arxiv.org/pdf/2404.13148
• Copy Paste: [[2404.13148]] BACS: Background Aware Continual Semantic Segmentation(https://arxiv.org/abs/2404.13148)
• Keywords: transformer, segmentation
• Abstract: Semantic segmentation plays a crucial role in enabling comprehensive scene understanding for robotic systems. However, generating annotations is challenging, requiring labels for every pixel in an image. In scenarios like autonomous driving, there's a need to progressively incorporate new classes as the operating environment of the deployed agent becomes more complex. For enhanced annotation efficiency, ideally, only pixels belonging to new classes would be annotated. This approach is known as Continual Semantic Segmentation (CSS). Besides the common problem of classical catastrophic forgetting in the continual learning setting, CSS suffers from the inherent ambiguity of the background, a phenomenon we refer to as the "background shift'', since pixels labeled as background could correspond to future classes (forward background shift) or previous classes (backward background shift). As a result, continual learning approaches tend to fail. This paper proposes a Backward Background Shift Detector (BACS) to detect previously observed classes based on their distance in the latent space from the foreground centroids of previous steps. Moreover, we propose a modified version of the cross-entropy loss function, incorporating the BACS detector to down-weight background pixels associated with formerly observed classes. To combat catastrophic forgetting, we employ masked feature distillation alongside dark experience replay. Additionally, our approach includes a transformer decoder capable of adjusting to new classes without necessitating an additional classification head. We validate BACS's superior performance over existing state-of-the-art methods on standard CSS benchmarks.

Title: Beyond Self-Consistency: Ensemble Reasoning Boosts Consistency and Accuracy of LLMs in Cancer Staging

• Authors: Chia-Hsuan Chang, Mary M. Lucas, Yeawon Lee, Christopher C. Yang, Grace Lu-Yao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13149
• Pdf URL: https://arxiv.org/pdf/2404.13149
• Copy Paste: [[2404.13149]] Beyond Self-Consistency: Ensemble Reasoning Boosts Consistency and Accuracy of LLMs in Cancer Staging(https://arxiv.org/abs/2404.13149)
• Keywords: large language model
• Abstract: Advances in large language models (LLMs) have encouraged their adoption in the healthcare domain where vital clinical information is often contained in unstructured notes. Cancer staging status is available in clinical reports, but it requires natural language processing to extract the status from the unstructured text. With the advance in clinical-oriented LLMs, it is promising to extract such status without extensive efforts in training the algorithms. Prompting approaches of the pre-trained LLMs that elicit a model's reasoning process, such as chain-of-thought, may help to improve the trustworthiness of the generated responses. Using self-consistency further improves model performance, but often results in inconsistent generations across the multiple reasoning paths. In this study, we propose an ensemble reasoning approach with the aim of improving the consistency of the model generations. Using an open access clinical large language model to determine the pathologic cancer stage from real-world pathology reports, we show that the ensemble reasoning approach is able to improve both the consistency and performance of the LLM in determining cancer stage, thereby demonstrating the potential to use these models in clinical or other domains where reliability and trustworthiness are critical.

Title: CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models

• Authors: Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, Joshua Saxe
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13161
• Pdf URL: https://arxiv.org/pdf/2404.13161
• Copy Paste: [[2404.13161]] CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models(https://arxiv.org/abs/2404.13161)
• Keywords: security, attack, large language model
• Abstract: Large language models (LLMs) introduce new security risks, but there are few comprehensive evaluation suites to measure and reduce these risks. We present BenchmarkName, a novel benchmark to quantify LLM security risks and capabilities. We introduce two new areas for testing: prompt injection and code interpreter abuse. We evaluated multiple state-of-the-art (SOTA) LLMs, including GPT-4, Mistral, Meta Llama 3 70B-Instruct, and Code Llama. Our results show that conditioning away risk of attack remains an unsolved problem; for example, all tested models showed between 26% and 41% successful prompt injection tests. We further introduce the safety-utility tradeoff: conditioning an LLM to reject unsafe prompts can cause the LLM to falsely reject answering benign prompts, which lowers utility. We propose quantifying this tradeoff using False Refusal Rate (FRR). As an illustration, we introduce a novel test set to quantify FRR for cyberattack helpfulness risk. We find many LLMs able to successfully comply with "borderline" benign requests while still rejecting most unsafe requests. Finally, we quantify the utility of LLMs for automating a core cybersecurity task, that of exploiting software vulnerabilities. This is important because the offensive capabilities of LLMs are of intense interest; we quantify this by creating novel test sets for four representative problems. We find that models with coding capabilities perform better than those without, but that further work is needed for LLMs to become proficient at exploit generation. Our code is open source and can be used to evaluate other LLMs.

Title: Heterogeneous Subgraph Transformer for Fake News Detection

• Authors: Yuchen Zhang, Xiaoxiao Ma, Jia Wu, Jian Yang, Hao Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13192
• Pdf URL: https://arxiv.org/pdf/2404.13192
• Copy Paste: [[2404.13192]] Heterogeneous Subgraph Transformer for Fake News Detection(https://arxiv.org/abs/2404.13192)
• Keywords: transformer
• Abstract: Fake news is pervasive on social media, inflicting substantial harm on public discourse and societal well-being. We investigate the explicit structural information and textual features of news pieces by constructing a heterogeneous graph concerning the relations among news topics, entities, and content. Through our study, we reveal that fake news can be effectively detected in terms of the atypical heterogeneous subgraphs centered on them, which encapsulate the essential semantics and intricate relations between news elements. However, suffering from the heterogeneity, exploring such heterogeneous subgraphs remains an open problem. To bridge the gap, this work proposes a heterogeneous subgraph transformer (HeteroSGT) to exploit subgraphs in our constructed heterogeneous graph. In HeteroSGT, we first employ a pre-trained language model to derive both word-level and sentence-level semantics. Then the random walk with restart (RWR) is applied to extract subgraphs centered on each news, which are further fed to our proposed subgraph Transformer to quantify the authenticity. Extensive experiments on five real-world datasets demonstrate the superior performance of HeteroSGT over five baselines. Further case and ablation studies validate our motivation and demonstrate that performance improvement stems from our specially designed components.

Title: Privacy-Preserving Debiasing using Data Augmentation and Machine Unlearning

• Authors: Zhixin Pan, Emma Andrews, Laura Chang, Prabhat Mishra
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13194
• Pdf URL: https://arxiv.org/pdf/2404.13194
• Copy Paste: [[2404.13194]] Privacy-Preserving Debiasing using Data Augmentation and Machine Unlearning(https://arxiv.org/abs/2404.13194)
• Keywords: privacy, protect, defense, attack, robust, membership infer, fair, diffusion
• Abstract: Data augmentation is widely used to mitigate data bias in the training dataset. However, data augmentation exposes machine learning models to privacy attacks, such as membership inference attacks. In this paper, we propose an effective combination of data augmentation and machine unlearning, which can reduce data bias while providing a provable defense against known attacks. Specifically, we maintain the fairness of the trained model with diffusion-based data augmentation, and then utilize multi-shard unlearning to remove identifying information of original data from the ML model for protection against privacy attacks. Experimental evaluation across diverse datasets demonstrates that our approach can achieve significant improvements in bias reduction as well as robustness against state-of-the-art privacy attacks.

Title: The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

• Authors: Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, Alex Beutel
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13208
• Pdf URL: https://arxiv.org/pdf/2404.13208
• Copy Paste: [[2404.13208]] The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions(https://arxiv.org/abs/2404.13208)
• Keywords: attack, robust
• Abstract: Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. We then propose a data generation method to demonstrate this hierarchical instruction following behavior, which teaches LLMs to selectively ignore lower-privileged instructions. We apply this method to GPT-3.5, showing that it drastically increases robustness -- even for attack types not seen during training -- while imposing minimal degradations on standard capabilities.

Title: Security and Privacy Product Inclusion

• Authors: Dave Kleidermacher, Emmanuel Arriaga, Eric Wang, Sebastian Porst, Roger Piqueras Jover
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13220
• Pdf URL: https://arxiv.org/pdf/2404.13220
• Copy Paste: [[2404.13220]] Security and Privacy Product Inclusion(https://arxiv.org/abs/2404.13220)
• Keywords: security, privacy, fair
• Abstract: In this paper, we explore the challenges of ensuring security and privacy for users from diverse demographic backgrounds. We propose a threat modeling approach to identify potential risks and countermeasures for product inclusion in security and privacy. We discuss various factors that can affect a user's ability to achieve a high level of security and privacy, including low-income demographics, poor connectivity, shared device usage, ML fairness, etc. We present results from a global security and privacy user experience survey and discuss the implications for product developers. Our work highlights the need for a more inclusive approach to security and privacy and provides a framework for researchers and practitioners to consider when designing products and services for a diverse range of users.

Title: TrialDura: Hierarchical Attention Transformer for Interpretable Clinical Trial Duration Prediction

• Authors: Ling Yue, Jonathan Li, Md Zabirul Islam, Bolun Xia, Tianfan Fu, Jintai Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13235
• Pdf URL: https://arxiv.org/pdf/2404.13235
• Copy Paste: [[2404.13235]] TrialDura: Hierarchical Attention Transformer for Interpretable Clinical Trial Duration Prediction(https://arxiv.org/abs/2404.13235)
• Keywords: transformer
• Abstract: The clinical trial process, also known as drug development, is an indispensable step toward the development of new treatments. The major objective of interventional clinical trials is to assess the safety and effectiveness of drug-based treatment in treating certain diseases in the human body. However, clinical trials are lengthy, labor-intensive, and costly. The duration of a clinical trial is a crucial factor that influences overall expenses. Therefore, effective management of the timeline of a clinical trial is essential for controlling the budget and maximizing the economic viability of the research. To address this issue, We propose TrialDura, a machine learning-based method that estimates the duration of clinical trials using multimodal data, including disease names, drug molecules, trial phases, and eligibility criteria. Then, we encode them into Bio-BERT embeddings specifically tuned for biomedical contexts to provide a deeper and more relevant semantic understanding of clinical trial data. Finally, the model's hierarchical attention mechanism connects all of the embeddings to capture their interactions and predict clinical trial duration. Our proposed model demonstrated superior performance with a mean absolute error (MAE) of 1.04 years and a root mean square error (RMSE) of 1.39 years compared to the other models, indicating more accurate clinical trial duration prediction. Publicly available code can be found at https://anonymous.4open.science/r/TrialDura-F196

Title: PAFedFV: Personalized and Asynchronous Federated Learning for Finger Vein Recognition

• Authors: Hengyu Mu, Jian Guo, Chong Han, Lijuan Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13237
• Pdf URL: https://arxiv.org/pdf/2404.13237
• Copy Paste: [[2404.13237]] PAFedFV: Personalized and Asynchronous Federated Learning for Finger Vein Recognition(https://arxiv.org/abs/2404.13237)
• Keywords: privacy, protect, robust, biometric, federate
• Abstract: With the increasing emphasis on user privacy protection, biometric recognition based on federated learning have become the latest research hotspot. However, traditional federated learning methods cannot be directly applied to finger vein recognition, due to heterogeneity of data and open-set verification. Therefore, only a few application cases have been proposed. And these methods still have two drawbacks. (1) Uniform model results in poor performance in some clients, as the finger vein data is highly heterogeneous and non-Independently Identically Distributed (non-IID). (2) On individual client, a large amount of time is underutilized, such as the time to wait for returning model from server. To address those problems, this paper proposes a Personalized and Asynchronous Federated Learning for Finger Vein Recognition (PAFedFV) framework. PAFedFV designs personalized model aggregation method to solve the heterogeneity among non-IID data. Meanwhile, it employs an asynchronized training module for clients to utilize their waiting time. Finally, extensive experiments on six finger vein datasets are conducted. Base on these experiment results, the impact of non-IID finger vein data on performance of federated learning are analyzed, and the superiority of PAFedFV in accuracy and robustness are demonstrated.

Title: Personalized Wireless Federated Learning for Large Language Models

• Authors: Feibo Jiang, Li Dong, Siwei Tu, Yubo Peng, Kezhi Wang, Kun Yang, Cunhua Pan, Dusit Niyato
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13238
• Pdf URL: https://arxiv.org/pdf/2404.13238
• Copy Paste: [[2404.13238]] Personalized Wireless Federated Learning for Large Language Models(https://arxiv.org/abs/2404.13238)
• Keywords: security, privacy, protect, federate, large language model
• Abstract: Large Language Models (LLMs) have revolutionized natural language processing tasks. However, their deployment in wireless networks still face challenges, i.e., a lack of privacy and security protection mechanisms. Federated Learning (FL) has emerged as a promising approach to address these challenges. Yet, it suffers from issues including inefficient handling with big and heterogeneous data, resource-intensive training, and high communication overhead. To tackle these issues, we first compare different learning stages and their features of LLMs in wireless networks. Next, we introduce two personalized wireless federated fine-tuning methods with low communication overhead, i.e., (1) Personalized Federated Instruction Tuning (PFIT), which employs reinforcement learning to fine-tune local LLMs with diverse reward models to achieve personalization; (2) Personalized Federated Task Tuning (PFTT), which can leverage global adapters and local Low-Rank Adaptations (LoRA) to collaboratively fine-tune local LLMs, where the local LoRAs can be applied to achieve personalization without aggregation. Finally, we perform simulations to demonstrate the effectiveness of the proposed two methods and comprehensively discuss open issues.

Title: Beyond Pixel-Wise Supervision for Medical Image Segmentation: From Traditional Models to Foundation Models

• Authors: Yuyan Shi, Jialu Ma, Jin Yang, Shasha Wang, Yichi Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13239
• Pdf URL: https://arxiv.org/pdf/2404.13239
• Copy Paste: [[2404.13239]] Beyond Pixel-Wise Supervision for Medical Image Segmentation: From Traditional Models to Foundation Models(https://arxiv.org/abs/2404.13239)
• Keywords: segmentation
• Abstract: Medical image segmentation plays an important role in many image-guided clinical approaches. However, existing segmentation algorithms mostly rely on the availability of fully annotated images with pixel-wise annotations for training, which can be both labor-intensive and expertise-demanding, especially in the medical imaging domain where only experts can provide reliable and accurate annotations. To alleviate this challenge, there has been a growing focus on developing segmentation methods that can train deep models with weak annotations, such as image-level, bounding boxes, scribbles, and points. The emergence of vision foundation models, notably the Segment Anything Model (SAM), has introduced innovative capabilities for segmentation tasks using weak annotations for promptable segmentation enabled by large-scale pre-training. Adopting foundation models together with traditional learning methods has increasingly gained recent interest research community and shown potential for real-world applications. In this paper, we present a comprehensive survey of recent progress on annotation-efficient learning for medical image segmentation utilizing weak annotations before and in the era of foundation models. Furthermore, we analyze and discuss several challenges of existing approaches, which we believe will provide valuable guidance for shaping the trajectory of foundational models to further advance the field of medical image segmentation.

Title: Intelligent Agents for Auction-based Federated Learning: A Survey

• Authors: Xiaoli Tang, Han Yu, Xiaoxiao Li, Sarit Kraus
• Subjects: cs.LG, cs.GT
• Abstract URL: https://arxiv.org/abs/2404.13244
• Pdf URL: https://arxiv.org/pdf/2404.13244
• Copy Paste: [[2404.13244]] Intelligent Agents for Auction-based Federated Learning: A Survey(https://arxiv.org/abs/2404.13244)
• Keywords: federate, fair
• Abstract: Auction-based federated learning (AFL) is an important emerging category of FL incentive mechanism design, due to its ability to fairly and efficiently motivate high-quality data owners to join data consumers' (i.e., servers') FL training tasks. To enhance the efficiency in AFL decision support for stakeholders (i.e., data consumers, data owners, and the auctioneer), intelligent agent-based techniques have emerged. However, due to the highly interdisciplinary nature of this field and the lack of a comprehensive survey providing an accessible perspective, it is a challenge for researchers to enter and contribute to this field. This paper bridges this important gap by providing a first-of-its-kind survey on the Intelligent Agents for AFL (IA-AFL) literature. We propose a unique multi-tiered taxonomy that organises existing IA-AFL works according to 1) the stakeholders served, 2) the auction mechanism adopted, and 3) the goals of the agents, to provide readers with a multi-perspective view into this field. In addition, we analyse the limitations of existing approaches, summarise the commonly adopted performance evaluation metrics, and discuss promising future directions leading towards effective and efficient stakeholder-oriented decision support in IA-AFL ecosystems.

Title: 3D-Convolution Guided Spectral-Spatial Transformer for Hyperspectral Image Classification

• Authors: Shyam Varahagiri, Aryaman Sinha, Shiv Ram Dubey, Satish Kumar Singh
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.13252
• Pdf URL: https://arxiv.org/pdf/2404.13252
• Copy Paste: [[2404.13252]] 3D-Convolution Guided Spectral-Spatial Transformer for Hyperspectral Image Classification(https://arxiv.org/abs/2404.13252)
• Keywords: transformer
• Abstract: In recent years, Vision Transformers (ViTs) have shown promising classification performance over Convolutional Neural Networks (CNNs) due to their self-attention mechanism. Many researchers have incorporated ViTs for Hyperspectral Image (HSI) classification. HSIs are characterised by narrow contiguous spectral bands, providing rich spectral data. Although ViTs excel with sequential data, they cannot extract spectral-spatial information like CNNs. Furthermore, to have high classification performance, there should be a strong interaction between the HSI token and the class (CLS) token. To solve these issues, we propose a 3D-Convolution guided Spectral-Spatial Transformer (3D-ConvSST) for HSI classification that utilizes a 3D-Convolution Guided Residual Module (CGRM) in-between encoders to "fuse" the local spatial and spectral information and to enhance the feature propagation. Furthermore, we forego the class token and instead apply Global Average Pooling, which effectively encodes more discriminative and pertinent high-level features for classification. Extensive experiments have been conducted on three public HSI datasets to show the superiority of the proposed model over state-of-the-art traditional, convolutional, and Transformer models. The code is available at https://github.com/ShyamVarahagiri/3D-ConvSST.

Title: ST-SSMs: Spatial-Temporal Selective State of Space Model for Traffic Forecasting

• Authors: Zhiqi Shao, Michael G.H. Bell, Ze Wang, D. Glenn Geers, Haoning Xi, Junbin Gao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13257
• Pdf URL: https://arxiv.org/pdf/2404.13257
• Copy Paste: [[2404.13257]] ST-SSMs: Spatial-Temporal Selective State of Space Model for Traffic Forecasting(https://arxiv.org/abs/2404.13257)
• Keywords: transformer
• Abstract: Accurate and efficient traffic prediction is crucial for planning, management, and control of intelligent transportation systems. Most state-of-the-art methods for traffic prediction effectively predict both long-term and short-term by employing spatio-temporal neural networks as prediction models, together with transformers to learn global information on prediction objects (e.g., traffic states of road segments). However, these methods often have a high computational cost to obtain good performance. This paper introduces an innovative approach to traffic flow prediction, the Spatial-Temporal Selective State Space Model (ST-SSMs), featuring the novel ST-Mamba block, which can achieve good prediction accuracy with less computational cost. A comparative analysis highlights the ST-Mamba layer's efficiency, revealing its equivalence to three attention layers, yet with markedly reduced processing time. Through rigorous testing on diverse real-world datasets, the ST-SSMs model demonstrates exceptional improvements in prediction accuracy and computational simplicity, setting new benchmarks in the domain of traffic flow forecasting

Title: FilterPrompt: Guiding Image Transfer in Diffusion Models

• Authors: Xi Wang, Yichen Peng, Heng Fang, Haoran Xie, Xi Yang, Chuntao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13263
• Pdf URL: https://arxiv.org/pdf/2404.13263
• Copy Paste: [[2404.13263]] FilterPrompt: Guiding Image Transfer in Diffusion Models(https://arxiv.org/abs/2404.13263)
• Keywords: diffusion, generative
• Abstract: In controllable generation tasks, flexibly manipulating the generated images to attain a desired appearance or structure based on a single input image cue remains a critical and longstanding challenge. Achieving this requires the effective decoupling of key attributes within the input image data, aiming to get representations accurately. Previous research has predominantly concentrated on disentangling image attributes within feature space. However, the complex distribution present in real-world data often makes the application of such decoupling algorithms to other datasets challenging. Moreover, the granularity of control over feature encoding frequently fails to meet specific task requirements. Upon scrutinizing the characteristics of various generative models, we have observed that the input sensitivity and dynamic evolution properties of the diffusion model can be effectively fused with the explicit decomposition operation in pixel space. This integration enables the image processing operations performed in pixel space for a specific feature distribution of the input image, and can achieve the desired control effect in the generated results. Therefore, we propose FilterPrompt, an approach to enhance the model control effect. It can be universally applied to any diffusion model, allowing users to adjust the representation of specific image features in accordance with task requirements, thereby facilitating more precise and controllable generation outcomes. In particular, our designed experiments demonstrate that the FilterPrompt optimizes feature correlation, mitigates content conflicts during the generation process, and enhances the model's control capability.

Title: Multi-Cell Decoder and Mutual Learning for Table Structure and Character Recognition

• Authors: Takaya Kawakatsu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13268
• Pdf URL: https://arxiv.org/pdf/2404.13268
• Copy Paste: [[2404.13268]] Multi-Cell Decoder and Mutual Learning for Table Structure and Character Recognition(https://arxiv.org/abs/2404.13268)
• Keywords: large language model
• Abstract: Extracting table contents from documents such as scientific papers and financial reports and converting them into a format that can be processed by large language models is an important task in knowledge information processing. End-to-end approaches, which recognize not only table structure but also cell contents, achieved performance comparable to state-of-the-art models using external character recognition systems, and have potential for further improvements. In addition, these models can now recognize long tables with hundreds of cells by introducing local attention. However, the models recognize table structure in one direction from the header to the footer, and cell content recognition is performed independently for each cell, so there is no opportunity to retrieve useful information from the neighbor cells. In this paper, we propose a multi-cell content decoder and bidirectional mutual learning mechanism to improve the end-to-end approach. The effectiveness is demonstrated on two large datasets, and the experimental results show comparable performance to state-of-the-art models, even for long tables with large numbers of cells.

Title: StrideNET: Swin Transformer for Terrain Recognition with Dynamic Roughness Extraction

• Authors: Maitreya Shelare, Neha Shigvan, Atharva Satam, Poonam Sonar
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13270
• Pdf URL: https://arxiv.org/pdf/2404.13270
• Copy Paste: [[2404.13270]] StrideNET: Swin Transformer for Terrain Recognition with Dynamic Roughness Extraction(https://arxiv.org/abs/2404.13270)
• Keywords: extraction, transformer
• Abstract: Advancements in deep learning are revolutionizing the classification of remote-sensing images. Transformer-based architectures, utilizing self-attention mechanisms, have emerged as alternatives to conventional convolution methods, enabling the capture of long-range dependencies along with global relationships in the image. Motivated by these advancements, this paper presents StrideNET, a novel dual-branch architecture designed for terrain recognition and implicit properties estimation. The terrain recognition branch utilizes the Swin Transformer, leveraging its hierarchical representation and low computational cost to efficiently capture both local and global features. The terrain properties branch focuses on the extraction of surface properties such as roughness and slipperiness using a statistical texture analysis method. By computing surface terrain properties, an enhanced environmental perception can be obtained. The StrideNET model is trained on a dataset comprising four target terrain classes: Grassy, Marshy, Sandy, and Rocky. StrideNET attains competitive performance compared to contemporary methods. The implications of this work extend to various applications, including environmental monitoring, land use and land cover (LULC) classification, disaster response, precision agriculture, and much more.

Title: Multi-feature Reconstruction Network using Crossed-mask Restoration for Unsupervised Anomaly Detection

• Authors: Junpu Wang, Guili Xu, Chunlei Li, Guangshuai Gao, Yuehua Cheng
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13273
• Pdf URL: https://arxiv.org/pdf/2404.13273
• Copy Paste: [[2404.13273]] Multi-feature Reconstruction Network using Crossed-mask Restoration for Unsupervised Anomaly Detection(https://arxiv.org/abs/2404.13273)
• Keywords: transformer
• Abstract: Unsupervised anomaly detection using only normal samples is of great significance for quality inspection in industrial manufacturing. Although existing reconstruction-based methods have achieved promising results, they still face two problems: poor distinguishable information in image reconstruction and well abnormal regeneration caused by model over-generalization ability. To overcome the above issues, we convert the image reconstruction into a combination of parallel feature restorations and propose a multi-feature reconstruction network, MFRNet, using crossed-mask restoration in this paper. Specifically, a multi-scale feature aggregator is first developed to generate more discriminative hierarchical representations of the input images from a pre-trained model. Subsequently, a crossed-mask generator is adopted to randomly cover the extracted feature map, followed by a restoration network based on the transformer structure for high-quality repair of the missing regions. Finally, a hybrid loss is equipped to guide model training and anomaly estimation, which gives consideration to both the pixel and structural similarity. Extensive experiments show that our method is highly competitive with or significantly outperforms other state-of-the-arts on four public available datasets and one self-made dataset.

Title: Federated Transfer Learning with Task Personalization for Condition Monitoring in Ultrasonic Metal Welding

• Authors: Ahmadreza Eslaminia, Yuquan Meng, Klara Nahrstedt, Chenhui Shao
• Subjects: cs.LG, cs.AI, cs.DC, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.13278
• Pdf URL: https://arxiv.org/pdf/2404.13278
• Copy Paste: [[2404.13278]] Federated Transfer Learning with Task Personalization for Condition Monitoring in Ultrasonic Metal Welding(https://arxiv.org/abs/2404.13278)
• Keywords: privacy, federate
• Abstract: Ultrasonic metal welding (UMW) is a key joining technology with widespread industrial applications. Condition monitoring (CM) capabilities are critically needed in UMW applications because process anomalies significantly deteriorate the joining quality. Recently, machine learning models emerged as a promising tool for CM in many manufacturing applications due to their ability to learn complex patterns. Yet, the successful deployment of these models requires substantial training data that may be expensive and time-consuming to collect. Additionally, many existing machine learning models lack generalizability and cannot be directly applied to new process configurations (i.e., domains). Such issues may be potentially alleviated by pooling data across manufacturers, but data sharing raises critical data privacy concerns. To address these challenges, this paper presents a Federated Transfer Learning with Task Personalization (FTL-TP) framework that provides domain generalization capabilities in distributed learning while ensuring data privacy. By effectively learning a unified representation from feature space, FTL-TP can adapt CM models for clients working on similar tasks, thereby enhancing their overall adaptability and performance jointly. To demonstrate the effectiveness of FTL-TP, we investigate two distinct UMW CM tasks, tool condition monitoring and workpiece surface condition classification. Compared with state-of-the-art FL algorithms, FTL-TP achieves a 5.35%--8.08% improvement of accuracy in CM in new target domains. FTL-TP is also shown to perform excellently in challenging scenarios involving unbalanced data distributions and limited client fractions. Furthermore, by implementing the FTL-TP method on an edge-cloud architecture, we show that this method is both viable and efficient in practice. The FTL-TP framework is readily extensible to various other manufacturing applications.

Title: Backdoor Attacks and Defenses on Semantic-Symbol Reconstruction in Semantic Communications

• Authors: Yuan Zhou, Rose Qingyang Hu, Yi Qian
• Subjects: cs.CR, eess.IV, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.13279
• Pdf URL: https://arxiv.org/pdf/2404.13279
• Copy Paste: [[2404.13279]] Backdoor Attacks and Defenses on Semantic-Symbol Reconstruction in Semantic Communications(https://arxiv.org/abs/2404.13279)
• Keywords: protect, defense, attack
• Abstract: Semantic communication is of crucial importance for the next-generation wireless communication networks. The existing works have developed semantic communication frameworks based on deep learning. However, systems powered by deep learning are vulnerable to threats such as backdoor attacks and adversarial attacks. This paper delves into backdoor attacks targeting deep learning-enabled semantic communication systems. Since current works on backdoor attacks are not tailored for semantic communication scenarios, a new backdoor attack paradigm on semantic symbols (BASS) is introduced, based on which the corresponding defense measures are designed. Specifically, a training framework is proposed to prevent BASS. Additionally, reverse engineering-based and pruning-based defense strategies are designed to protect against backdoor attacks in semantic communication. Simulation results demonstrate the effectiveness of both the proposed attack paradigm and the defense strategies.

Title: Wills Aligner: A Robust Multi-Subject Brain Representation Learner

• Authors: Guangyin Bao, Zixuan Gong, Qi Zhang, Jialei Zhou, Wei Fan, Kun Yi, Usman Naseem, Liang Hu, Duoqian Miao
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.13282
• Pdf URL: https://arxiv.org/pdf/2404.13282
• Copy Paste: [[2404.13282]] Wills Aligner: A Robust Multi-Subject Brain Representation Learner(https://arxiv.org/abs/2404.13282)
• Keywords: robust
• Abstract: Decoding visual information from human brain activity has seen remarkable advancements in recent research. However, due to the significant variability in cortical parcellation and cognition patterns across subjects, current approaches personalized deep models for each subject, constraining the practicality of this technology in real-world contexts. To tackle the challenges, we introduce Wills Aligner, a robust multi-subject brain representation learner. Our Wills Aligner initially aligns different subjects' brains at the anatomical level. Subsequently, it incorporates a mixture of brain experts to learn individual cognition patterns. Additionally, it decouples the multi-subject learning task into a two-stage training, propelling the deep model and its plugin network to learn inter-subject commonality knowledge and various cognition patterns, respectively. Wills Aligner enables us to overcome anatomical differences and to efficiently leverage a single model for multi-subject brain representation learning. We meticulously evaluate the performance of our approach across coarse-grained and fine-grained visual decoding tasks. The experimental results demonstrate that our Wills Aligner achieves state-of-the-art performance.

Title: Double Mixture: Towards Continual Event Detection from Speech

• Authors: Jingqi Kang, Tongtong Wu, Jinming Zhao, Guitao Wang, Yinwei Wei, Hao Yang, Guilin Qi, Yuan-Fang Li, Gholamreza Haffari
• Subjects: cs.CL, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.13289
• Pdf URL: https://arxiv.org/pdf/2404.13289
• Copy Paste: [[2404.13289]] Double Mixture: Towards Continual Event Detection from Speech(https://arxiv.org/abs/2404.13289)
• Keywords: robust
• Abstract: Speech event detection is crucial for multimedia retrieval, involving the tagging of both semantic and acoustic events. Traditional ASR systems often overlook the interplay between these events, focusing solely on content, even though the interpretation of dialogue can vary with environmental context. This paper tackles two primary challenges in speech event detection: the continual integration of new events without forgetting previous ones, and the disentanglement of semantic from acoustic events. We introduce a new task, continual event detection from speech, for which we also provide two benchmark datasets. To address the challenges of catastrophic forgetting and effective disentanglement, we propose a novel method, 'Double Mixture.' This method merges speech expertise with robust memory mechanisms to enhance adaptability and prevent forgetting. Our comprehensive experiments show that this task presents significant challenges that are not effectively addressed by current state-of-the-art methods in either computer vision or natural language processing. Our approach achieves the lowest rates of forgetting and the highest levels of generalization, proving robust across various continual learning sequences. Our code and data are available at https://anonymous.4open.science/status/Continual-SpeechED-6461.

Title: PCQA: A Strong Baseline for AIGC Quality Assessment Based on Prompt Condition

• Authors: Xi Fang, Weigang Wang, Xiaoxin Lv, Jun Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13299
• Pdf URL: https://arxiv.org/pdf/2404.13299
• Copy Paste: [[2404.13299]] PCQA: A Strong Baseline for AIGC Quality Assessment Based on Prompt Condition(https://arxiv.org/abs/2404.13299)
• Keywords: diffusion, large language model
• Abstract: The development of Large Language Models (LLM) and Diffusion Models brings the boom of Artificial Intelligence Generated Content (AIGC). It is essential to build an effective quality assessment framework to provide a quantifiable evaluation of different images or videos based on the AIGC technologies. The content generated by AIGC methods is driven by the crafted prompts. Therefore, it is intuitive that the prompts can also serve as the foundation of the AIGC quality assessment. This study proposes an effective AIGC quality assessment (QA) framework. First, we propose a hybrid prompt encoding method based on a dual-source CLIP (Contrastive Language-Image Pre-Training) text encoder to understand and respond to the prompt conditions. Second, we propose an ensemble-based feature mixer module to effectively blend the adapted prompt and vision features. The empirical study practices in two datasets: AIGIQA-20K (AI-Generated Image Quality Assessment database) and T2VQA-DB (Text-to-Video Quality Assessment DataBase), which validates the effectiveness of our proposed method: Prompt Condition Quality Assessment (PCQA). Our proposed simple and feasible framework may promote research development in the multimodal generation field.

Title: EHRFL: Federated Learning Framework for Heterogeneous EHRs and Precision-guided Selection of Participating Clients

• Authors: Jiyoun Kim, Junu Kim, Kyunghoon Hur, Edward Choi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13318
• Pdf URL: https://arxiv.org/pdf/2404.13318
• Copy Paste: [[2404.13318]] EHRFL: Federated Learning Framework for Heterogeneous EHRs and Precision-guided Selection of Participating Clients(https://arxiv.org/abs/2404.13318)
• Keywords: federate
• Abstract: In this study, we provide solutions to two practical yet overlooked scenarios in federated learning for electronic health records (EHRs): firstly, we introduce EHRFL, a framework that facilitates federated learning across healthcare institutions with distinct medical coding systems and database schemas using text-based linearization of EHRs. Secondly, we focus on a scenario where a single healthcare institution initiates federated learning to build a model tailored for itself, in which the number of clients must be optimized in order to reduce expenses incurred by the host. For selecting participating clients, we present a novel precision-based method, leveraging data latents to identify suitable participants for the institution. Our empirical results show that EHRFL effectively enables federated learning across hospitals with different EHR systems. Furthermore, our results demonstrate the efficacy of our precision-based method in selecting reduced number of participating clients without compromising model performance, resulting in lower operational costs when constructing institution-specific models. We believe this work lays a foundation for the broader adoption of federated learning on EHRs.

Title: Pixel is a Barrier: Diffusion Models Are More Adversarially Robust Than We Think

• Authors: Haotian Xue, Yongxin Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13320
• Pdf URL: https://arxiv.org/pdf/2404.13320
• Copy Paste: [[2404.13320]] Pixel is a Barrier: Diffusion Models Are More Adversarially Robust Than We Think(https://arxiv.org/abs/2404.13320)
• Keywords: protect, attack, robust, diffusion
• Abstract: Adversarial examples for diffusion models are widely used as solutions for safety concerns. By adding adversarial perturbations to personal images, attackers can not edit or imitate them easily. However, it is essential to note that all these protections target the latent diffusion model (LDMs), the adversarial examples for diffusion models in the pixel space (PDMs) are largely overlooked. This may mislead us to think that the diffusion models are vulnerable to adversarial attacks like most deep models. In this paper, we show novel findings that: even though gradient-based white-box attacks can be used to attack the LDMs, they fail to attack PDMs. This finding is supported by extensive experiments of almost a wide range of attacking methods on various PDMs and LDMs with different model structures, which means diffusion models are indeed much more robust against adversarial attacks. We also find that PDMs can be used as an off-the-shelf purifier to effectively remove the adversarial patterns that were generated on LDMs to protect the images, which means that most protection methods nowadays, to some extent, cannot protect our images from malicious attacks. We hope that our insights will inspire the community to rethink the adversarial samples for diffusion models as protection methods and move forward to more effective protection. Codes are available in https://github.com/xavihart/PDM-Pure.

Title: MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalities

• Authors: Kunxi Li, Tianyu Zhan, Shengyu Zhang, Kun Kuang, Jiwei Li, Zhou Zhao, Fei Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13322
• Pdf URL: https://arxiv.org/pdf/2404.13322
• Copy Paste: [[2404.13322]] MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalities(https://arxiv.org/abs/2404.13322)
• Keywords: extraction
• Abstract: In this study, we focus on heterogeneous knowledge transfer across entirely different model architectures, tasks, and modalities. Existing knowledge transfer methods (e.g., backbone sharing, knowledge distillation) often hinge on shared elements within model structures or task-specific features/labels, limiting transfers to complex model types or tasks. To overcome these challenges, we present MergeNet, which learns to bridge the gap of parameter spaces of heterogeneous models, facilitating the direct interaction, extraction, and application of knowledge within these parameter spaces. The core mechanism of MergeNet lies in the parameter adapter, which operates by querying the source model's low-rank parameters and adeptly learning to identify and map parameters into the target model. MergeNet is learned alongside both models, allowing our framework to dynamically transfer and adapt knowledge relevant to the current stage, including the training trajectory knowledge of the source model. Extensive experiments on heterogeneous knowledge transfer demonstrate significant improvements in challenging settings, where representative approaches may falter or prove less applicable.

Title: Collaborative Visual Place Recognition through Federated Learning

• Authors: Mattia Dutto, Gabriele Berton, Debora Caldarola, Eros Fanì, Gabriele Trivigno, Carlo Masone
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13324
• Pdf URL: https://arxiv.org/pdf/2404.13324
• Copy Paste: [[2404.13324]] Collaborative Visual Place Recognition through Federated Learning(https://arxiv.org/abs/2404.13324)
• Keywords: federate
• Abstract: Visual Place Recognition (VPR) aims to estimate the location of an image by treating it as a retrieval problem. VPR uses a database of geo-tagged images and leverages deep neural networks to extract a global representation, called descriptor, from each image. While the training data for VPR models often originates from diverse, geographically scattered sources (geo-tagged images), the training process itself is typically assumed to be centralized. This research revisits the task of VPR through the lens of Federated Learning (FL), addressing several key challenges associated with this adaptation. VPR data inherently lacks well-defined classes, and models are typically trained using contrastive learning, which necessitates a data mining step on a centralized database. Additionally, client devices in federated systems can be highly heterogeneous in terms of their processing capabilities. The proposed FedVPR framework not only presents a novel approach for VPR but also introduces a new, challenging, and realistic task for FL research, paving the way to other image retrieval tasks in FL.

Title: Comparative Analysis on Snowmelt-Driven Streamflow Forecasting Using Machine Learning Techniques

• Authors: Ukesh Thapa, Bipun Man Pati, Samit Thapa, Dhiraj Pyakurel, Anup Shrestha
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13327
• Pdf URL: https://arxiv.org/pdf/2404.13327
• Copy Paste: [[2404.13327]] Comparative Analysis on Snowmelt-Driven Streamflow Forecasting Using Machine Learning Techniques(https://arxiv.org/abs/2404.13327)
• Keywords: transformer
• Abstract: The rapid advancement of machine learning techniques has led to their widespread application in various domains including water resources. However, snowmelt modeling remains an area that has not been extensively explored. In this study, we propose a state-of-the-art (SOTA) deep learning sequential model, leveraging the Temporal Convolutional Network (TCN), for snowmelt-driven discharge modeling in the Himalayan basin of the Hindu Kush Himalayan Region. To evaluate the performance of our proposed model, we conducted a comparative analysis with other popular models including Support Vector Regression (SVR), Long Short Term Memory (LSTM), and Transformer. Furthermore, Nested cross-validation (CV) is used with five outer folds and three inner folds, and hyper-parameter tuning is performed on the inner folds. To evaluate the performance of the model mean absolute error (MAE), root mean square error (RMSE), R square ($R^{2}$), Kling-Gupta Efficiency (KGE), and Nash-Sutcliffe Efficiency (NSE) are computed for each outer fold. The average metrics revealed that TCN outperformed the other models, with an average MAE of 0.011, RMSE of 0.023, $R^{2}$ of 0.991, KGE of 0.992, and NSE of 0.991. The findings of this study demonstrate the effectiveness of the deep learning model as compared to traditional machine learning approaches for snowmelt-driven streamflow forecasting. Moreover, the superior performance of TCN highlights its potential as a promising deep learning model for similar hydrological applications.

Title: Fuzzychain: An Equitable Consensus Mechanism for Blockchain Networks

• Authors: Bruno Ramos-Cruz, Javier Andreu-Pérez, Francisco J. Quesada, Luis Martínez
• Subjects: cs.CR, cs.ET, cs.LO
• Abstract URL: https://arxiv.org/abs/2404.13337
• Pdf URL: https://arxiv.org/pdf/2404.13337
• Copy Paste: [[2404.13337]] Fuzzychain: An Equitable Consensus Mechanism for Blockchain Networks(https://arxiv.org/abs/2404.13337)
• Keywords: secure, fair
• Abstract: Blockchain technology has become a trusted method for establishing secure and transparent transactions through a distributed, encrypted network. The operation of blockchain is governed by consensus algorithms, among which Proof of Stake (PoS) is popular yet has its drawbacks, notably the potential for centralising power in nodes with larger stakes or higher rewards. Fuzzychain, our proposed solution, introduces the use of fuzzy sets to define stake semantics, promoting decentralised and distributed processing control. This system selects validators based on their degree of membership to the stake fuzzy sets rather than just the size of their stakes. As a pioneer proposal in applying fuzzy sets to blockchain, Fuzzychain aims to rectify PoS's limitations. Our results indicate that Fuzzychain not only matches PoS in functionality but also ensures a fairer distribution of stakes among validators, leading to more inclusive validator selection and a better-distributed network.

Title: UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions

• Authors: Ana-Cristina Rogoz, Radu Tudor Ionescu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13343
• Pdf URL: https://arxiv.org/pdf/2404.13343
• Copy Paste: [[2404.13343]] UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions(https://arxiv.org/abs/2404.13343)
• Keywords: transformer, large language model
• Abstract: This work explores a novel data augmentation method based on Large Language Models (LLMs) for predicting item difficulty and response time of retired USMLE Multiple-Choice Questions (MCQs) in the BEA 2024 Shared Task. Our approach is based on augmenting the dataset with answers from zero-shot LLMs (Falcon, Meditron, Mistral) and employing transformer-based models based on six alternative feature combinations. The results suggest that predicting the difficulty of questions is more challenging. Notably, our top performing methods consistently include the question text, and benefit from the variability of LLM answers, highlighting the potential of LLMs for improving automated assessment in medical licensing exams. We make our code available https://github.com/ana-rogoz/BEA-2024.

Title: Generating Daylight-driven Architectural Design via Diffusion Models

• Authors: Pengzhi Li, Baijuan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13353
• Pdf URL: https://arxiv.org/pdf/2404.13353
• Copy Paste: [[2404.13353]] Generating Daylight-driven Architectural Design via Diffusion Models(https://arxiv.org/abs/2404.13353)
• Keywords: diffusion
• Abstract: In recent years, the rapid development of large-scale models has made new possibilities for interdisciplinary fields such as architecture. In this paper, we present a novel daylight-driven AI-aided architectural design method. Firstly, we formulate a method for generating massing models, producing architectural massing models using random parameters quickly. Subsequently, we integrate a daylight-driven facade design strategy, accurately determining window layouts and applying them to the massing models. Finally, we seamlessly combine a large-scale language model with a text-to-image model, enhancing the efficiency of generating visual architectural design renderings. Experimental results demonstrate that our approach supports architects' creative inspirations and pioneers novel avenues for architectural design development. Project page: https://zrealli.github.io/DDADesign/.

Title: Semantically Corrected Amharic Automatic Speech Recognition

• Authors: Samuael Adnew, Paul Pu Liang
• Subjects: cs.CL, cs.AI, cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.13362
• Pdf URL: https://arxiv.org/pdf/2404.13362
• Copy Paste: [[2404.13362]] Semantically Corrected Amharic Automatic Speech Recognition(https://arxiv.org/abs/2404.13362)
• Keywords: transformer
• Abstract: Automatic Speech Recognition (ASR) can play a crucial role in enhancing the accessibility of spoken languages worldwide. In this paper, we build a set of ASR tools for Amharic, a language spoken by more than 50 million people primarily in eastern Africa. Amharic is written in the Ge'ez script, a sequence of graphemes with spacings denoting word boundaries. This makes computational processing of Amharic challenging since the location of spacings can significantly impact the meaning of formed sentences. We find that existing benchmarks for Amharic ASR do not account for these spacings and only measure individual grapheme error rates, leading to significantly inflated measurements of in-the-wild performance. In this paper, we first release corrected transcriptions of existing Amharic ASR test datasets, enabling the community to accurately evaluate progress. Furthermore, we introduce a post-processing approach using a transformer encoder-decoder architecture to organize raw ASR outputs into a grammatically complete and semantically meaningful Amharic sentence. Through experiments on the corrected test dataset, our model enhances the semantic correctness of Amharic speech recognition systems, achieving a Character Error Rate (CER) of 5.5\% and a Word Error Rate (WER) of 23.3\%.

Title: MahaSQuAD: Bridging Linguistic Divides in Marathi Question-Answering

• Authors: Ruturaj Ghatage, Aditya Kulkarni, Rajlaxmi Patil, Sharvi Endait, Raviraj Joshi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13364
• Pdf URL: https://arxiv.org/pdf/2404.13364
• Copy Paste: [[2404.13364]] MahaSQuAD: Bridging Linguistic Divides in Marathi Question-Answering(https://arxiv.org/abs/2404.13364)
• Keywords: robust
• Abstract: Question-answering systems have revolutionized information retrieval, but linguistic and cultural boundaries limit their widespread accessibility. This research endeavors to bridge the gap of the absence of efficient QnA datasets in low-resource languages by translating the English Question Answering Dataset (SQuAD) using a robust data curation approach. We introduce MahaSQuAD, the first-ever full SQuAD dataset for the Indic language Marathi, consisting of 118,516 training, 11,873 validation, and 11,803 test samples. We also present a gold test set of manually verified 500 examples. Challenges in maintaining context and handling linguistic nuances are addressed, ensuring accurate translations. Moreover, as a QnA dataset cannot be simply converted into any low-resource language using translation, we need a robust method to map the answer translation to its span in the translated passage. Hence, to address this challenge, we also present a generic approach for translating SQuAD into any low-resource language. Thus, we offer a scalable approach to bridge linguistic and cultural gaps present in low-resource languages, in the realm of question-answering systems. The datasets and models are shared publicly at https://github.com/l3cube-pune/MarathiNLP .

Title: DNA: Differentially private Neural Augmentation for contact tracing

• Authors: Rob Romijnders, Christos Louizos, Yuki M. Asano, Max Welling
• Subjects: cs.LG, cs.CR, cs.MA, q-bio.PE
• Abstract URL: https://arxiv.org/abs/2404.13381
• Pdf URL: https://arxiv.org/pdf/2404.13381
• Copy Paste: [[2404.13381]] DNA: Differentially private Neural Augmentation for contact tracing(https://arxiv.org/abs/2404.13381)
• Keywords: privacy
• Abstract: The COVID19 pandemic had enormous economic and societal consequences. Contact tracing is an effective way to reduce infection rates by detecting potential virus carriers early. However, this was not generally adopted in the recent pandemic, and privacy concerns are cited as the most important reason. We substantially improve the privacy guarantees of the current state of the art in decentralized contact tracing. Whereas previous work was based on statistical inference only, we augment the inference with a learned neural network and ensure that this neural augmentation satisfies differential privacy. In a simulator for COVID19, even at epsilon=1 per message, this can significantly improve the detection of potentially infected individuals and, as a result of targeted testing, reduce infection rates. This work marks an important first step in integrating deep learning into contact tracing while maintaining essential privacy guarantees.

Title: Explanation based Bias Decoupling Regularization for Natural Language Inference

• Authors: Jianxiang Zang, Hui Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13390
• Pdf URL: https://arxiv.org/pdf/2404.13390
• Copy Paste: [[2404.13390]] Explanation based Bias Decoupling Regularization for Natural Language Inference(https://arxiv.org/abs/2404.13390)
• Keywords: robust, transformer
• Abstract: The robustness of Transformer-based Natural Language Inference encoders is frequently compromised as they tend to rely more on dataset biases than on the intended task-relevant features. Recent studies have attempted to mitigate this by reducing the weight of biased samples during the training process. However, these debiasing methods primarily focus on identifying which samples are biased without explicitly determining the biased components within each case. This limitation restricts those methods' capability in out-of-distribution inference. To address this issue, we aim to train models to adopt the logic humans use in explaining causality. We propose a simple, comprehensive, and interpretable method: Explanation based Bias Decoupling Regularization (EBD-Reg). EBD-Reg employs human explanations as criteria, guiding the encoder to establish a tripartite parallel supervision of Distinguishing, Decoupling and Aligning. This method enables encoders to identify and focus on keywords that represent the task-relevant features during inference, while discarding the residual elements acting as biases. Empirical evidence underscores that EBD-Reg effectively guides various Transformer-based encoders to decouple biases through a human-centric lens, significantly surpassing other methods in terms of out-of-distribution inference capabilities.

Title: Retrieval-Augmented Generation-based Relation Extraction

• Authors: Sefika Efeoglu, Adrian Paschke
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13397
• Pdf URL: https://arxiv.org/pdf/2404.13397
• Copy Paste: [[2404.13397]] Retrieval-Augmented Generation-based Relation Extraction(https://arxiv.org/abs/2404.13397)
• Keywords: extraction, large language model
• Abstract: Information Extraction (IE) is a transformative process that converts unstructured text data into a structured format by employing entity and relation extraction (RE) methodologies. The identification of the relation between a pair of entities plays a crucial role within this framework. Despite the existence of various techniques for relation extraction, their efficacy heavily relies on access to labeled data and substantial computational resources. In addressing these challenges, Large Language Models (LLMs) emerge as promising solutions; however, they might return hallucinating responses due to their own training data. To overcome these limitations, Retrieved-Augmented Generation-based Relation Extraction (RAG4RE) in this work is proposed, offering a pathway to enhance the performance of relation extraction tasks. This work evaluated the effectiveness of our RAG4RE approach utilizing different LLMs. Through the utilization of established benchmarks, such as TACRED, TACREV, Re-TACRED, and SemEval RE datasets, our aim is to comprehensively evaluate the efficacy of our RAG4RE approach. In particularly, we leverage prominent LLMs including Flan T5, Llama2, and Mistral in our investigation. The results of our study demonstrate that our RAG4RE approach surpasses performance of traditional RE approaches based solely on LLMs, particularly evident in the TACRED dataset and its variations. Furthermore, our approach exhibits remarkable performance compared to previous RE methodologies across both TACRED and TACREV datasets, underscoring its efficacy and potential for advancing RE tasks in natural language processing.

Title: Intrusion Detection at Scale with the Assistance of a Command-line Language Model

• Authors: Jiongliang Lin, Yiwen Guo, Hao Chen
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13402
• Pdf URL: https://arxiv.org/pdf/2404.13402
• Copy Paste: [[2404.13402]] Intrusion Detection at Scale with the Assistance of a Command-line Language Model(https://arxiv.org/abs/2404.13402)
• Keywords: security, attack, large language model
• Abstract: Intrusion detection is a long standing and crucial problem in security. A system capable of detecting intrusions automatically is on great demand in enterprise security solutions. Existing solutions rely heavily on hand-crafted rules designed by security operators, which suffer from high false negative rates and poor generalization ability to new, zero-day attacks at scale. AI and machine learning offer promising solutions to address the issues, by inspecting abnormal user behaviors intelligently and automatically from data. However, existing learning-based intrusion detection systems in the literature are mostly designed for small data, and they lack the ability to leverage the power of big data in cloud environments. In this paper, we target at this problem and introduce an intrusion detection system which incorporates large-scale pre-training, so as to train a large language model based on tens of millions of command lines for AI-based intrusion detection. Experiments performed on 30 million training samples and 10 million test samples verify the effectiveness of our solution.

Title: A Framework for Managing Multifaceted Privacy Leakage While Optimizing Utility in Continuous LBS Interactions

• Authors: Anis Bkakria, Reda Yaich
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.13407
• Pdf URL: https://arxiv.org/pdf/2404.13407
• Copy Paste: [[2404.13407]] A Framework for Managing Multifaceted Privacy Leakage While Optimizing Utility in Continuous LBS Interactions(https://arxiv.org/abs/2404.13407)
• Keywords: privacy, protect
• Abstract: Privacy in Location-Based Services (LBS) has become a paramount concern with the ubiquity of mobile devices and the increasing integration of location data into various applications. In this paper, we present several novel contributions aimed at advancing the understanding and management of privacy leakage in LBS. Our contributions provides a more comprehensive framework for analyzing privacy concerns across different facets of location-based interactions. Specifically, we introduce $(\epsilon, \delta)$-location privacy, $(\epsilon, \delta, \theta)$-trajectory privacy, and $(\epsilon, \delta, \theta)$-POI privacy, which offer refined mechanisms for quantifying privacy risks associated with location, trajectory, and points of interest when continuously interacting with LBS. Furthermore, we establish fundamental connections between these privacy notions, facilitating a holistic approach to privacy preservation in LBS. Additionally, we present a lower bound analysis to evaluate the utility of the proposed privacy-preserving mechanisms, offering insights into the trade-offs between privacy protection and data utility. Finally, we instantiate our framework with the Plannar Isotopic Mechanism to demonstrate its practical applicability while ensuring optimal utility and quantifying privacy leakages across various dimensions. The conducted evaluations provide a comprehensive insight into the efficacy of our framework in capturing privacy loss on location, trajectory, and Points of Interest (POI) while facilitating quantification of the ensured accuracy.

Title: AMMUNet: Multi-Scale Attention Map Merging for Remote Sensing Image Segmentation

• Authors: Yang Yang, Shunyi Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13408
• Pdf URL: https://arxiv.org/pdf/2404.13408
• Copy Paste: [[2404.13408]] AMMUNet: Multi-Scale Attention Map Merging for Remote Sensing Image Segmentation(https://arxiv.org/abs/2404.13408)
• Keywords: segmentation
• Abstract: The advancement of deep learning has driven notable progress in remote sensing semantic segmentation. Attention mechanisms, while enabling global modeling and utilizing contextual information, face challenges of high computational costs and require window-based operations that weaken capturing long-range dependencies, hindering their effectiveness for remote sensing image processing. In this letter, we propose AMMUNet, a UNet-based framework that employs multi-scale attention map merging, comprising two key innovations: the granular multi-head self-attention (GMSA) module and the attention map merging mechanism (AMMM). GMSA efficiently acquires global information while substantially mitigating computational costs in contrast to global multi-head self-attention mechanism. This is accomplished through the strategic utilization of dimension correspondence to align granularity and the reduction of relative position bias parameters, thereby optimizing computational efficiency. The proposed AMMM effectively combines multi-scale attention maps into a unified representation using a fixed mask template, enabling the modeling of global attention mechanism. Experimental evaluations highlight the superior performance of our approach, achieving remarkable mean intersection over union (mIoU) scores of 75.48\% on the challenging Vaihingen dataset and an exceptional 77.90\% on the Potsdam dataset, demonstrating the superiority of our method in precise remote sensing semantic segmentation. Codes are available at https://github.com/interpretty/AMMUNet.

Title: MultiConfederated Learning: Inclusive Non-IID Data handling with Decentralized Federated Learning

• Authors: Michael Duchesne, Kaiwen Zhang, Chamseddine Talhi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13421
• Pdf URL: https://arxiv.org/pdf/2404.13421
• Copy Paste: [[2404.13421]] MultiConfederated Learning: Inclusive Non-IID Data handling with Decentralized Federated Learning(https://arxiv.org/abs/2404.13421)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) has emerged as a prominent privacy-preserving technique for enabling use cases like confidential clinical machine learning. FL operates by aggregating models trained by remote devices which owns the data. Thus, FL enables the training of powerful global models using crowd-sourced data from a large number of learners, without compromising their privacy. However, the aggregating server is a single point of failure when generating the global model. Moreover, the performance of the model suffers when the data is not independent and identically distributed (non-IID data) on all remote devices. This leads to vastly different models being aggregated, which can reduce the performance by as much as 50% in certain scenarios. In this paper, we seek to address the aforementioned issues while retaining the benefits of FL. We propose MultiConfederated Learning: a decentralized FL framework which is designed to handle non-IID data. Unlike traditional FL, MultiConfederated Learning will maintain multiple models in parallel (instead of a single global model) to help with convergence when the data is non-IID. With the help of transfer learning, learners can converge to fewer models. In order to increase adaptability, learners are allowed to choose which updates to aggregate from their peers.

Title: AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models

• Authors: Yuheng Ji, Yue Liu, Zhicheng Zhang, Zhao Zhang, Yuting Zhao, Gang Zhou, Xingwei Zhang, Xinwang Liu, Xiaolong Zheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13425
• Pdf URL: https://arxiv.org/pdf/2404.13425
• Copy Paste: [[2404.13425]] AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models(https://arxiv.org/abs/2404.13425)
• Keywords: security, robust
• Abstract: Vision-Language Models (VLMs) are a significant technique for Artificial General Intelligence (AGI). With the fast growth of AGI, the security problem become one of the most important challenges for VLMs. In this paper, through extensive experiments, we demonstrate the vulnerability of the conventional adaptation methods for VLMs, which may bring significant security risks. In addition, as the size of the VLMs increases, performing conventional adversarial adaptation techniques on VLMs results in high computational costs. To solve these problems, we propose a parameter-efficient \underline{Adv}ersarial adaptation method named \underline{AdvLoRA} by \underline{Lo}w-\underline{R}ank \underline{A}daptation. At first, we investigate and reveal the intrinsic low-rank property during the adversarial adaptation for VLMs. Different from LoRA, we improve the efficiency and robustness of adversarial adaptation by designing a novel reparameterizing method based on parameter clustering and parameter alignment. In addition, an adaptive parameter update strategy is proposed to further improve the robustness. By these settings, our proposed AdvLoRA alleviates the model security and high resource waste problems. Extensive experiments demonstrate the effectiveness and efficiency of the AdvLoRA.

Title: Nested-TNT: Hierarchical Vision Transformers with Multi-Scale Feature Processing

• Authors: Yuang Liu, Zhiheng Qiu, Xiaokai Qin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13434
• Pdf URL: https://arxiv.org/pdf/2404.13434
• Copy Paste: [[2404.13434]] Nested-TNT: Hierarchical Vision Transformers with Multi-Scale Feature Processing(https://arxiv.org/abs/2404.13434)
• Keywords: transformer
• Abstract: Transformer has been applied in the field of computer vision due to its excellent performance in natural language processing, surpassing traditional convolutional neural networks and achieving new state-of-the-art. ViT divides an image into several local patches, known as "visual sentences". However, the information contained in the image is vast and complex, and focusing only on the features at the "visual sentence" level is not enough. The features between local patches should also be taken into consideration. In order to achieve further improvement, the TNT model is proposed, whose algorithm further divides the image into smaller patches, namely "visual words," achieving more accurate results. The core of Transformer is the Multi-Head Attention mechanism, and traditional attention mechanisms ignore interactions across different attention heads. In order to reduce redundancy and improve utilization, we introduce the nested algorithm and apply the Nested-TNT to image classification tasks. The experiment confirms that the proposed model has achieved better classification performance over ViT and TNT, exceeding 2.25%, 1.1% on dataset CIFAR10 and 2.78%, 0.25% on dataset FLOWERS102 respectively.

Title: Fine-Grained Named Entities for Corona News

• Authors: Sefika Efeoglu, Adrian Paschke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13439
• Pdf URL: https://arxiv.org/pdf/2404.13439
• Copy Paste: [[2404.13439]] Fine-Grained Named Entities for Corona News(https://arxiv.org/abs/2404.13439)
• Keywords: extraction
• Abstract: Information resources such as newspapers have produced unstructured text data in various languages related to the corona outbreak since December 2019. Analyzing these unstructured texts is time-consuming without representing them in a structured format; therefore, representing them in a structured format is crucial. An information extraction pipeline with essential tasks -- named entity tagging and relation extraction -- to accomplish this goal might be applied to these texts. This study proposes a data annotation pipeline to generate training data from corona news articles, including generic and domain-specific entities. Named entity recognition models are trained on this annotated corpus and then evaluated on test sentences manually annotated by domain experts evaluating the performance of a trained model. The code base and demonstration are available at https://github.com/sefeoglu/coronanews-ner.git.

Title: FisheyeDetNet: Object Detection on Fisheye Surround View Camera Systems for Automated Driving

• Authors: Ganesh Sistu, Senthil Yogamani
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.13443
• Pdf URL: https://arxiv.org/pdf/2404.13443
• Copy Paste: [[2404.13443]] FisheyeDetNet: Object Detection on Fisheye Surround View Camera Systems for Automated Driving(https://arxiv.org/abs/2404.13443)
• Keywords: segmentation
• Abstract: Object detection is a mature problem in autonomous driving with pedestrian detection being one of the first deployed algorithms. It has been comprehensively studied in the literature. However, object detection is relatively less explored for fisheye cameras used for surround-view near field sensing. The standard bounding box representation fails in fisheye cameras due to heavy radial distortion, particularly in the periphery. To mitigate this, we explore extending the standard object detection output representation of bounding box. We design rotated bounding boxes, ellipse, generic polygon as polar arc/angle representations and define an instance segmentation mIOU metric to analyze these representations. The proposed model FisheyeDetNet with polygon outperforms others and achieves a mAP score of 49.5 % on Valeo fisheye surround-view dataset for automated driving applications. This dataset has 60K images captured from 4 surround-view cameras across Europe, North America and Asia. To the best of our knowledge, this is the first detailed study on object detection on fisheye cameras for autonomous driving scenarios.

Title: SiNC+: Adaptive Camera-Based Vitals with Unsupervised Learning of Periodic Signals

• Authors: Jeremy Speth, Nathan Vance, Patrick Flynn, Adam Czajka
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13449
• Pdf URL: https://arxiv.org/pdf/2404.13449
• Copy Paste: [[2404.13449]] SiNC+: Adaptive Camera-Based Vitals with Unsupervised Learning of Periodic Signals(https://arxiv.org/abs/2404.13449)
• Keywords: robust
• Abstract: Subtle periodic signals, such as blood volume pulse and respiration, can be extracted from RGB video, enabling noncontact health monitoring at low cost. Advancements in remote pulse estimation -- or remote photoplethysmography (rPPG) -- are currently driven by deep learning solutions. However, modern approaches are trained and evaluated on benchmark datasets with ground truth from contact-PPG sensors. We present the first non-contrastive unsupervised learning framework for signal regression to mitigate the need for labelled video data. With minimal assumptions of periodicity and finite bandwidth, our approach discovers the blood volume pulse directly from unlabelled videos. We find that encouraging sparse power spectra within normal physiological bandlimits and variance over batches of power spectra is sufficient for learning visual features of periodic signals. We perform the first experiments utilizing unlabelled video data not specifically created for rPPG to train robust pulse rate estimators. Given the limited inductive biases, we successfully applied the same approach to camera-based respiration by changing the bandlimits of the target signal. This shows that the approach is general enough for unsupervised learning of bandlimited quasi-periodic signals from different domains. Furthermore, we show that the framework is effective for finetuning models on unlabelled video from a single subject, allowing for personalized and adaptive signal regressors.

Title: Do "English" Named Entity Recognizers Work Well on Global Englishes?

• Authors: Alexander Shan, John Bauer, Riley Carlson, Christopher Manning
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13465
• Pdf URL: https://arxiv.org/pdf/2404.13465
• Copy Paste: [[2404.13465]] Do "English" Named Entity Recognizers Work Well on Global Englishes?(https://arxiv.org/abs/2404.13465)
• Keywords: transformer
• Abstract: The vast majority of the popular English named entity recognition (NER) datasets contain American or British English data, despite the existence of many global varieties of English. As such, it is unclear whether they generalize for analyzing use of English globally. To test this, we build a newswire dataset, the Worldwide English NER Dataset, to analyze NER model performance on low-resource English variants from around the world. We test widely used NER toolkits and transformer models, including models using the pre-trained contextual models RoBERTa and ELECTRA, on three datasets: a commonly used British English newswire dataset, CoNLL 2003, a more American focused dataset OntoNotes, and our global dataset. All models trained on the CoNLL or OntoNotes datasets experienced significant performance drops-over 10 F1 in some cases-when tested on the Worldwide English dataset. Upon examination of region-specific errors, we observe the greatest performance drops for Oceania and Africa, while Asia and the Middle East had comparatively strong performance. Lastly, we find that a combined model trained on the Worldwide dataset and either CoNLL or OntoNotes lost only 1-2 F1 on both test sets.

Title: Leveraging Adversarial Detection to Enable Scalable and Low Overhead RowHammer Mitigations

• Authors: Oğuzhan Canpolat, A. Giray Yağlıkçı, Ataberk Olgun, İsmail Emir Yüksel, Yahya Can Tuğrul, Konstantinos Kanellopoulos, Oğuz Ergin, Onur Mutlu
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2404.13477
• Pdf URL: https://arxiv.org/pdf/2404.13477
• Copy Paste: [[2404.13477]] Leveraging Adversarial Detection to Enable Scalable and Low Overhead RowHammer Mitigations(https://arxiv.org/abs/2404.13477)
• Keywords: security, privacy
• Abstract: RowHammer is a prime example of read disturbance in DRAM where repeatedly accessing (hammering) a row of DRAM cells (DRAM row) induces bitflips in other physically nearby DRAM rows. RowHammer solutions perform preventive actions (e.g., refresh neighbor rows of the hammered row) that mitigate such bitflips to preserve memory isolation, a fundamental building block of security and privacy in modern computing systems. However, preventive actions induce non-negligible memory request latency and system performance overheads as they interfere with memory requests in the memory controller. As shrinking technology node size over DRAM chip generations exacerbates RowHammer, the overheads of RowHammer solutions become prohibitively large. As a result, a malicious program can effectively hog the memory system and deny service to benign applications by causing many RowHammer preventive actions. In this work, we tackle the performance overheads of RowHammer solutions by tracking the generators of memory accesses that trigger RowHammer solutions. To this end, we propose BreakHammer. BreakHammer cooperates with existing RowHammer solutions to identify hardware threads that trigger preventive actions. To do so, BreakHammer estimates the RowHammer likelihood of a thread, based on how frequently it triggers RowHammer preventive actions. BreakHammer limits the number of on-the-fly requests a thread can inject into the memory system based on the thread's RowHammer likelihood. By doing so, BreakHammer significantly reduces the number of performed counter-measures, improves the system performance by an average (maximum) of 48.7% (105.5%), and reduces the maximum slowdown induced on a benign application by 14.6% with near-zero area overhead (e.g., 0.0002% of a highend processor's chip area).

Title: Authentic Emotion Mapping: Benchmarking Facial Expressions in Real News

• Authors: Qixuan Zhang, Zhifeng Wang, Yang Liu, Zhenyue Qin, Kaihao Zhang, Sabrina Caldwell, Tom Gedeon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13493
• Pdf URL: https://arxiv.org/pdf/2404.13493
• Copy Paste: [[2404.13493]] Authentic Emotion Mapping: Benchmarking Facial Expressions in Real News(https://arxiv.org/abs/2404.13493)
• Keywords: transformer
• Abstract: In this paper, we present a novel benchmark for Emotion Recognition using facial landmarks extracted from realistic news videos. Traditional methods relying on RGB images are resource-intensive, whereas our approach with Facial Landmark Emotion Recognition (FLER) offers a simplified yet effective alternative. By leveraging Graph Neural Networks (GNNs) to analyze the geometric and spatial relationships of facial landmarks, our method enhances the understanding and accuracy of emotion recognition. We discuss the advancements and challenges in deep learning techniques for emotion recognition, particularly focusing on Graph Neural Networks (GNNs) and Transformers. Our experimental results demonstrate the viability and potential of our dataset as a benchmark, setting a new direction for future research in emotion recognition technologies. The codes and models are at: https://github.com/wangzhifengharrison/benchmark_real_news

Title: Generalized Regression with Conditional GANs

• Authors: Deddy Jobson, Eddy Hudson
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.13500
• Pdf URL: https://arxiv.org/pdf/2404.13500
• Copy Paste: [[2404.13500]] Generalized Regression with Conditional GANs(https://arxiv.org/abs/2404.13500)
• Keywords: generative
• Abstract: Regression is typically treated as a curve-fitting process where the goal is to fit a prediction function to data. With the help of conditional generative adversarial networks, we propose to solve this age-old problem in a different way; we aim to learn a prediction function whose outputs, when paired with the corresponding inputs, are indistinguishable from feature-label pairs in the training dataset. We show that this approach to regression makes fewer assumptions on the distribution of the data we are fitting to and, therefore, has better representation capabilities. We draw parallels with generalized linear models in statistics and show how our proposal serves as an extension of them to neural networks. We demonstrate the superiority of this new approach to standard regression with experiments on multiple synthetic and publicly available real-world datasets, finding encouraging results, especially with real-world heavy-tailed regression datasets. To make our work more reproducible, we release our source code. Link to repository: https://anonymous.4open.science/r/regressGAN-7B71/

Title: Dynamic in Static: Hybrid Visual Correspondence for Self-Supervised Video Object Segmentation

• Authors: Gensheng Pei, Yazhou Yao, Jianbo Jiao, Wenguan Wang, Liqiang Nie, Jinhui Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13505
• Pdf URL: https://arxiv.org/pdf/2404.13505
• Copy Paste: [[2404.13505]] Dynamic in Static: Hybrid Visual Correspondence for Self-Supervised Video Object Segmentation(https://arxiv.org/abs/2404.13505)
• Keywords: segmentation
• Abstract: Conventional video object segmentation (VOS) methods usually necessitate a substantial volume of pixel-level annotated video data for fully supervised learning. In this paper, we present HVC, a \textbf{h}ybrid static-dynamic \textbf{v}isual \textbf{c}orrespondence framework for self-supervised VOS. HVC extracts pseudo-dynamic signals from static images, enabling an efficient and scalable VOS model. Our approach utilizes a minimalist fully-convolutional architecture to capture static-dynamic visual correspondence in image-cropped views. To achieve this objective, we present a unified self-supervised approach to learn visual representations of static-dynamic feature similarity. Firstly, we establish static correspondence by utilizing a priori coordinate information between cropped views to guide the formation of consistent static feature representations. Subsequently, we devise a concise convolutional layer to capture the forward / backward pseudo-dynamic signals between two views, serving as cues for dynamic representations. Finally, we propose a hybrid visual correspondence loss to learn joint static and dynamic consistency representations. Our approach, without bells and whistles, necessitates only one training session using static image data, significantly reducing memory consumption ($\sim$16GB) and training time ($\sim$\textbf{2h}). Moreover, HVC achieves state-of-the-art performance in several self-supervised VOS benchmarks and additional video label propagation tasks.

Title: FedTrans: Efficient Federated Learning Over Heterogeneous Clients via Model Transformation

• Authors: Yuxuan Zhu, Jiachen Liu, Mosharaf Chowdhury, Fan Lai
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.13515
• Pdf URL: https://arxiv.org/pdf/2404.13515
• Copy Paste: [[2404.13515]] FedTrans: Efficient Federated Learning Over Heterogeneous Clients via Model Transformation(https://arxiv.org/abs/2404.13515)
• Keywords: federate
• Abstract: Federated learning (FL) aims to train machine learning (ML) models across potentially millions of edge client devices. Yet, training and customizing models for FL clients is notoriously challenging due to the heterogeneity of client data, device capabilities, and the massive scale of clients, making individualized model exploration prohibitively expensive. State-of-the-art FL solutions personalize a globally trained model or concurrently train multiple models, but they often incur suboptimal model accuracy and huge training costs. In this paper, we introduce FedTrans, a multi-model FL training framework that automatically produces and trains high-accuracy, hardware-compatible models for individual clients at scale. FedTrans begins with a basic global model, identifies accuracy bottlenecks in model architectures during training, and then employs model transformation to derive new models for heterogeneous clients on the fly. It judiciously assigns models to individual clients while performing soft aggregation on multi-model updates to minimize total training costs. Our evaluations using realistic settings show that FedTrans improves individual client model accuracy by 14% - 72% while slashing training costs by 1.6X - 20X over state-of-the-art solutions.

Title: Reliable Model Watermarking: Defending Against Theft without Compromising on Evasion

• Authors: Hongyu Zhu, Sichu Liang, Wentao Hu, Fangqi Li, Ju Jia, Shilin Wang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13518
• Pdf URL: https://arxiv.org/pdf/2404.13518
• Copy Paste: [[2404.13518]] Reliable Model Watermarking: Defending Against Theft without Compromising on Evasion(https://arxiv.org/abs/2404.13518)
• Keywords: protect, attack, robust, extraction, watermark, diffusion
• Abstract: With the rise of Machine Learning as a Service (MLaaS) platforms,safeguarding the intellectual property of deep learning models is becoming paramount. Among various protective measures, trigger set watermarking has emerged as a flexible and effective strategy for preventing unauthorized model distribution. However, this paper identifies an inherent flaw in the current paradigm of trigger set watermarking: evasion adversaries can readily exploit the shortcuts created by models memorizing watermark samples that deviate from the main task distribution, significantly impairing their generalization in adversarial settings. To counteract this, we leverage diffusion models to synthesize unrestricted adversarial examples as trigger sets. By learning the model to accurately recognize them, unique watermark behaviors are promoted through knowledge injection rather than error memorization, thus avoiding exploitable shortcuts. Furthermore, we uncover that the resistance of current trigger set watermarking against removal attacks primarily relies on significantly damaging the decision boundaries during embedding, intertwining unremovability with adverse impacts. By optimizing the knowledge transfer properties of protected models, our approach conveys watermark behaviors to extraction surrogates without aggressively decision boundary perturbation. Experimental results on CIFAR-10/100 and Imagenette datasets demonstrate the effectiveness of our method, showing not only improved robustness against evasion adversaries but also superior resistance to watermark removal attacks compared to state-of-the-art solutions.

Title: SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile

• Authors: Wei Niu, Md Musfiqur Rahman Sanim, Zhihao Shu, Jiexiong Guan, Xipeng Shen, Miao Yin, Gagan Agrawal, Bin Ren
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.13528
• Pdf URL: https://arxiv.org/pdf/2404.13528
• Copy Paste: [[2404.13528]] SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile(https://arxiv.org/abs/2404.13528)
• Keywords: diffusion, transformer
• Abstract: This work is motivated by recent developments in Deep Neural Networks, particularly the Transformer architectures underlying applications such as ChatGPT, and the need for performing inference on mobile devices. Focusing on emerging transformers (specifically the ones with computationally efficient Swin-like architectures) and large models (e.g., Stable Diffusion and LLMs) based on transformers, we observe that layout transformations between the computational operators cause a significant slowdown in these applications. This paper presents SmartMem, a comprehensive framework for eliminating most layout transformations, with the idea that multiple operators can use the same tensor layout through careful choice of layout and implementation of operations. Our approach is based on classifying the operators into four groups, and considering combinations of producer-consumer edges between the operators. We develop a set of methods for searching such layouts. Another component of our work is developing efficient memory layouts for 2.5 dimensional memory commonly seen in mobile devices. Our experimental results show that SmartMem outperforms 5 state-of-the-art DNN execution frameworks on mobile devices across 18 varied neural networks, including CNNs, Transformers with both local and global attention, as well as LLMs. In particular, compared to DNNFusion, SmartMem achieves an average speedup of 2.8$\times$, and outperforms TVM and MNN with speedups of 6.9$\times$ and 7.9$\times$, respectively, on average.

Title: Motion-aware Latent Diffusion Models for Video Frame Interpolation

• Authors: Zhilin Huang, Yijie Yu, Ling Yang, Chujun Qin, Bing Zheng, Xiawu Zheng, Zikun Zhou, Yaowei Wang, Wenming Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13534
• Pdf URL: https://arxiv.org/pdf/2404.13534
• Copy Paste: [[2404.13534]] Motion-aware Latent Diffusion Models for Video Frame Interpolation(https://arxiv.org/abs/2404.13534)
• Keywords: diffusion
• Abstract: With the advancement of AIGC, video frame interpolation (VFI) has become a crucial component in existing video generation frameworks, attracting widespread research interest. For the VFI task, the motion estimation between neighboring frames plays a crucial role in avoiding motion ambiguity. However, existing VFI methods always struggle to accurately predict the motion information between consecutive frames, and this imprecise estimation leads to blurred and visually incoherent interpolated frames. In this paper, we propose a novel diffusion framework, motion-aware latent diffusion models (MADiff), which is specifically designed for the VFI task. By incorporating motion priors between the conditional neighboring frames with the target interpolated frame predicted throughout the diffusion sampling procedure, MADiff progressively refines the intermediate outcomes, culminating in generating both visually smooth and realistic results. Extensive experiments conducted on benchmark datasets demonstrate that our method achieves state-of-the-art performance significantly outperforming existing approaches, especially under challenging scenarios involving dynamic textures with complex motion.

Title: DesTest: A Decentralised Testing Architecture for Improving Data Accuracy of Blockchain Oracle

• Authors: Xueying Zeng, Youquan Xian, Chunpei Li, Zhengdong Hu, Peng Liu
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.13535
• Pdf URL: https://arxiv.org/pdf/2404.13535
• Copy Paste: [[2404.13535]] DesTest: A Decentralised Testing Architecture for Improving Data Accuracy of Blockchain Oracle(https://arxiv.org/abs/2404.13535)
• Keywords: secure, attack
• Abstract: Blockchain technology ensures secure and trustworthy data flow between multiple participants on the chain, but interoperability of on-chain and off-chain data has always been a difficult problem that needs to be solved. To solve the problem that blockchain systems cannot access off-chain data, oracle is introduced. however, existing research mainly focuses on the consistency and integrity of data, but ignores the problem that oracle nodes may be externally attacked or provide false data for selfish motives, resulting in the unresolved problem of data accuracy. In this paper, we introduce a new decentralized testing architecture (DesTest) that aims to improve data accuracy. A blockchain oracle random secret testing mechanism is first proposed to enhance the monitoring and verification of nodes by introducing a dynamic anonymized question-verification committee. Based on this, a comprehensive evaluation incentive mechanism is designed to incentivize honest work performance by evaluating nodes based on their reputation scores. The simulation results show that we successfully reduced the discrete entropy value of the acquired data and the real value of the data by 61.4%.

Title: Delving into Post-Quantum TLS Performance: Faster ML-KEM in TLS 1.3 Implementation and Assessment

• Authors: Jieyu Zheng, Haoliang Zhu, Yifan Dong, Zhenyu Song, Zhenhao Zhang, Yafang Yang, Yunlei Zhao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.13544
• Pdf URL: https://arxiv.org/pdf/2404.13544
• Copy Paste: [[2404.13544]] Delving into Post-Quantum TLS Performance: Faster ML-KEM in TLS 1.3 Implementation and Assessment(https://arxiv.org/abs/2404.13544)
• Keywords: secure, security
• Abstract: TLS is extensively utilized for secure data transmission over networks. However, with the advent of quantum computers, the security of TLS based on traditional public-key cryptography is under threat. To counter quantum threats, it is imperative to integrate post-quantum algorithms into TLS. Most PQ-TLS research focuses on integration and evaluation, but few studies address the improvement of PQ-TLS performance by optimizing PQC implementation. For the TLS protocol, handshake performance is crucial, and for post-quantum TLS (PQ-TLS) the performance of post-quantum key encapsulation mechanisms (KEMs) directly impacts handshake performance. In this work, we explore the impact of post-quantum KEMs on PQ-TLS performance. We explore how to improve ML-KEM performance using the latest Intel's Advanced Vector Extensions instruction set AVX-512. We detail a spectrum of techniques devised to parallelize polynomial multiplication, modular reduction, and other computationally intensive modules within ML-KEM. Our optimized ML-KEM implementation achieves up to 1.64x speedup compared to the latest AVX2 implementation. Furthermore, we introduce a novel batch key generation method for ML-KEM that can seamlessly integrate into the TLS protocols. The batch method accelerates the key generation procedure by 3.5x to 4.9x. We integrate the optimized AVX-512 implementation of ML-KEM into TLS 1.3, and assess handshake performance under both PQ-only and hybrid modes. The assessment demonstrates that our faster ML-KEM implementation results in a higher number of TLS 1.3 handshakes per second under both modes. Additionally, we revisit two IND-1-CCA KEM constructions discussed in Eurocrypt22 and Asiacrypt23. Besides, we implement them based on ML-KEM and integrate the one of better performance into TLS 1.3 with benchmarks.

Title: Pointsoup: High-Performance and Extremely Low-Decoding-Latency Learned Geometry Codec for Large-Scale Point Cloud Scenes

• Authors: Kang You, Kai Liu, Li Yu, Pan Gao, Dandan Ding
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.13550
• Pdf URL: https://arxiv.org/pdf/2404.13550
• Copy Paste: [[2404.13550]] Pointsoup: High-Performance and Extremely Low-Decoding-Latency Learned Geometry Codec for Large-Scale Point Cloud Scenes(https://arxiv.org/abs/2404.13550)
• Keywords: fair
• Abstract: Despite considerable progress being achieved in point cloud geometry compression, there still remains a challenge in effectively compressing large-scale scenes with sparse surfaces. Another key challenge lies in reducing decoding latency, a crucial requirement in real-world application. In this paper, we propose Pointsoup, an efficient learning-based geometry codec that attains high-performance and extremely low-decoding-latency simultaneously. Inspired by conventional Trisoup codec, a point model-based strategy is devised to characterize local surfaces. Specifically, skin features are embedded from local windows via an attention-based encoder, and dilated windows are introduced as cross-scale priors to infer the distribution of quantized features in parallel. During decoding, features undergo fast refinement, followed by a folding-based point generator that reconstructs point coordinates with fairly fast speed. Experiments show that Pointsoup achieves state-of-the-art performance on multiple benchmarks with significantly lower decoding complexity, i.e., up to 90$\sim$160$\times$ faster than the G-PCCv23 Trisoup decoder on a comparatively low-end platform (e.g., one RTX 2080Ti). Furthermore, it offers variable-rate control with a single neural model (2.9MB), which is attractive for industrial practitioners.

Title: Cell Phone Image-Based Persian Rice Detection and Classification Using Deep Learning Techniques

• Authors: Mahmood Saeedi kelishami, Amin Saeidi Kelishami, Sajjad Saeedi Kelishami
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13555
• Pdf URL: https://arxiv.org/pdf/2404.13555
• Copy Paste: [[2404.13555]] Cell Phone Image-Based Persian Rice Detection and Classification Using Deep Learning Techniques(https://arxiv.org/abs/2404.13555)
• Keywords: segmentation
• Abstract: This study introduces an innovative approach to classifying various types of Persian rice using image-based deep learning techniques, highlighting the practical application of everyday technology in food categorization. Recognizing the diversity of Persian rice and its culinary significance, we leveraged the capabilities of convolutional neural networks (CNNs), specifically by fine-tuning a ResNet model for accurate identification of different rice varieties and employing a U-Net architecture for precise segmentation of rice grains in bulk images. This dual-methodology framework allows for both individual grain classification and comprehensive analysis of bulk rice samples, addressing two crucial aspects of rice quality assessment. Utilizing images captured with consumer-grade cell phones reflects a realistic scenario in which individuals can leverage this technology for assistance with grocery shopping and meal preparation. The dataset, comprising various rice types photographed under natural conditions without professional lighting or equipment, presents a challenging yet practical classification problem. Our findings demonstrate the feasibility of using non-professional images for food classification and the potential of deep learning models, like ResNet and U-Net, to adapt to the nuances of everyday objects and textures. This study contributes to the field by providing insights into the applicability of image-based deep learning in daily life, specifically for enhancing consumer experiences and knowledge in food selection. Furthermore, it opens avenues for extending this approach to other food categories and practical applications, emphasizing the role of accessible technology in bridging the gap between sophisticated computational methods and everyday tasks.

Title: LASER: Tuning-Free LLM-Driven Attention Control for Efficient Text-conditioned Image-to-Animation

• Authors: Haoyu Zheng, Wenqiao Zhang, Yaoke Wang, Hao Zhou, Jiang Liu, Juncheng Li, Zheqi Lv, Siliang Tang, Yueting Zhuang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13558
• Pdf URL: https://arxiv.org/pdf/2404.13558
• Copy Paste: [[2404.13558]] LASER: Tuning-Free LLM-Driven Attention Control for Efficient Text-conditioned Image-to-Animation(https://arxiv.org/abs/2404.13558)
• Keywords: large language model
• Abstract: Revolutionary advancements in text-to-image models have unlocked new dimensions for sophisticated content creation, e.g., text-conditioned image editing, allowing us to edit the diverse images that convey highly complex visual concepts according to the textual guidance. Despite being promising, existing methods focus on texture- or non-rigid-based visual manipulation, which struggles to produce the fine-grained animation of smooth text-conditioned image morphing without fine-tuning, i.e., due to their highly unstructured latent space. In this paper, we introduce a tuning-free LLM-driven attention control framework, encapsulated by the progressive process of LLM planning, prompt-Aware editing, StablE animation geneRation, abbreviated as LASER. LASER employs a large language model (LLM) to refine coarse descriptions into detailed prompts, guiding pre-trained text-to-image models for subsequent image generation. We manipulate the model's spatial features and self-attention mechanisms to maintain animation integrity and enable seamless morphing directly from text prompts, eliminating the need for additional fine-tuning or annotations. Our meticulous control over spatial features and self-attention ensures structural consistency in the images. This paper presents a novel framework integrating LLMs with text-to-image models to create high-quality animations from a single text input. We also propose a Text-conditioned Image-to-Animation Benchmark to validate the effectiveness and efficacy of LASER. Extensive experiments demonstrate that LASER produces impressive, consistent, and efficient results in animation generation, positioning it as a powerful tool for advanced digital content creation.

Title: Masked Latent Transformer with the Random Masking Ratio to Advance the Diagnosis of Dental Fluorosis

• Authors: Yun Wu, Hao Xu, Maohua Gu, Zhongchuan Jiang, Jun Xu, Youliang Tian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13564
• Pdf URL: https://arxiv.org/pdf/2404.13564
• Copy Paste: [[2404.13564]] Masked Latent Transformer with the Random Masking Ratio to Advance the Diagnosis of Dental Fluorosis(https://arxiv.org/abs/2404.13564)
• Keywords: transformer
• Abstract: Dental fluorosis is a chronic disease caused by long-term overconsumption of fluoride, which leads to changes in the appearance of tooth enamel. It is an important basis for early non-invasive diagnosis of endemic fluorosis. However, even dental professionals may not be able to accurately distinguish dental fluorosis and its severity based on tooth images. Currently, there is still a gap in research on applying deep learning to diagnosing dental fluorosis. Therefore, we construct the first open-source dental fluorosis image dataset (DFID), laying the foundation for deep learning research in this field. To advance the diagnosis of dental fluorosis, we propose a pioneering deep learning model called masked latent transformer with the random masking ratio (MLTrMR). MLTrMR introduces a mask latent modeling scheme based on Vision Transformer to enhance contextual learning of dental fluorosis lesion characteristics. Consisting of a latent embedder, encoder, and decoder, MLTrMR employs the latent embedder to extract latent tokens from the original image, whereas the encoder and decoder comprising the latent transformer (LT) block are used to process unmasked tokens and predict masked tokens, respectively. To mitigate the lack of inductive bias in Vision Transformer, which may result in performance degradation, the LT block introduces latent tokens to enhance the learning capacity of latent lesion features. Furthermore, we design an auxiliary loss function to constrain the parameter update direction of the model. MLTrMR achieves 80.19% accuracy, 75.79% F1, and 81.28% quadratic weighted kappa on DFID, making it state-of-the-art (SOTA).

Title: Exploring Diverse Methods in Visual Question Answering

• Authors: Panfeng Li, Qikai Yang, Xieming Geng, Wenjing Zhou, Zhicheng Ding, Yi Nian
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13565
• Pdf URL: https://arxiv.org/pdf/2404.13565
• Copy Paste: [[2404.13565]] Exploring Diverse Methods in Visual Question Answering(https://arxiv.org/abs/2404.13565)
• Keywords: generative
• Abstract: This study explores innovative methods for improving Visual Question Answering (VQA) using Generative Adversarial Networks (GANs), autoencoders, and attention mechanisms. Leveraging a balanced VQA dataset, we investigate three distinct strategies. Firstly, GAN-based approaches aim to generate answer embeddings conditioned on image and question inputs, showing potential but struggling with more complex tasks. Secondly, autoencoder-based techniques focus on learning optimal embeddings for questions and images, achieving comparable results with GAN due to better ability on complex questions. Lastly, attention mechanisms, incorporating Multimodal Compact Bilinear pooling (MCB), address language priors and attention modeling, albeit with a complexity-performance trade-off. This study underscores the challenges and opportunities in VQA and suggests avenues for future research, including alternative GAN formulations and attentional mechanisms.

Title: Test-Time Training on Graphs with Large Language Models (LLMs)

• Authors: Jiaxin Zhang, Yiqi Wang, Xihong Yang, Siwei Wang, Yu Feng, Yu Shi, Ruicaho Ren, En Zhu, Xinwang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13571
• Pdf URL: https://arxiv.org/pdf/2404.13571
• Copy Paste: [[2404.13571]] Test-Time Training on Graphs with Large Language Models (LLMs)(https://arxiv.org/abs/2404.13571)
• Keywords: large language model
• Abstract: Graph Neural Networks have demonstrated great success in various fields of multimedia. However, the distribution shift between the training and test data challenges the effectiveness of GNNs. To mitigate this challenge, Test-Time Training (TTT) has been proposed as a promising approach. Traditional TTT methods require a demanding unsupervised training strategy to capture the information from test to benefit the main task. Inspired by the great annotation ability of Large Language Models (LLMs) on Text-Attributed Graphs (TAGs), we propose to enhance the test-time training on graphs with LLMs as annotators. In this paper, we design a novel Test-Time Training pipeline, LLMTTT, which conducts the test-time adaptation under the annotations by LLMs on a carefully-selected node set. Specifically, LLMTTT introduces a hybrid active node selection strategy that considers not only node diversity and representativeness, but also prediction signals from the pre-trained model. Given annotations from LLMs, a two-stage training strategy is designed to tailor the test-time model with the limited and noisy labels. A theoretical analysis ensures the validity of our method and extensive experiments demonstrate that the proposed LLMTTT can achieve a significant performance improvement compared to existing Out-of-Distribution (OOD) generalization methods.

Title: Exploring AIGC Video Quality: A Focus on Visual Harmony, Video-Text Consistency and Domain Distribution Gap

• Authors: Bowen Qu, Xiaoyu Liang, Shangkun Sun, Wei Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13573
• Pdf URL: https://arxiv.org/pdf/2404.13573
• Copy Paste: [[2404.13573]] Exploring AIGC Video Quality: A Focus on Visual Harmony, Video-Text Consistency and Domain Distribution Gap(https://arxiv.org/abs/2404.13573)
• Keywords: generative
• Abstract: The recent advancements in Text-to-Video Artificial Intelligence Generated Content (AIGC) have been remarkable. Compared with traditional videos, the assessment of AIGC videos encounters various challenges: visual inconsistency that defy common sense, discrepancies between content and the textual prompt, and distribution gap between various generative models, etc. Target at these challenges, in this work, we categorize the assessment of AIGC video quality into three dimensions: visual harmony, video-text consistency, and domain distribution gap. For each dimension, we design specific modules to provide a comprehensive quality assessment of AIGC videos. Furthermore, our research identifies significant variations in visual quality, fluidity, and style among videos generated by different text-to-video models. Predicting the source generative model can make the AIGC video features more discriminative, which enhances the quality assessment performance. The proposed method was used in the third-place winner of the NTIRE 2024 Quality Assessment for AI-Generated Content - Track 2 Video, demonstrating its effectiveness.

Title: FedMPQ: Secure and Communication-Efficient Federated Learning with Multi-codebook Product Quantization

• Authors: Xu Yang, Jiapeng Zhang, Qifeng Zhang, Zhuo Tang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13575
• Pdf URL: https://arxiv.org/pdf/2404.13575
• Copy Paste: [[2404.13575]] FedMPQ: Secure and Communication-Efficient Federated Learning with Multi-codebook Product Quantization(https://arxiv.org/abs/2404.13575)
• Keywords: secure, attack, robust, federate
• Abstract: In federated learning, particularly in cross-device scenarios, secure aggregation has recently gained popularity as it effectively defends against inference attacks by malicious aggregators. However, secure aggregation often requires additional communication overhead and can impede the convergence rate of the global model, which is particularly challenging in wireless network environments with extremely limited bandwidth. Therefore, achieving efficient communication compression under the premise of secure aggregation presents a highly challenging and valuable problem. In this work, we propose a novel uplink communication compression method for federated learning, named FedMPQ, which is based on multi shared codebook product quantization.Specifically, we utilize updates from the previous round to generate sufficiently robust codebooks. Secure aggregation is then achieved through trusted execution environments (TEE) or a trusted third party (TTP).In contrast to previous works, our approach exhibits greater robustness in scenarios where data is not independently and identically distributed (non-IID) and there is a lack of sufficient public data. The experiments conducted on the LEAF dataset demonstrate that our proposed method achieves 99% of the baseline's final accuracy, while reducing uplink communications by 90-95%

Title: I2CANSAY:Inter-Class Analogical Augmentation and Intra-Class Significance Analysis for Non-Exemplar Online Task-Free Continual Learning

• Authors: Songlin Dong, Yingjie Chen, Yuhang He, Yuhan Jin, Alex C. Kot, Yihong Gong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13576
• Pdf URL: https://arxiv.org/pdf/2404.13576
• Copy Paste: [[2404.13576]] I2CANSAY:Inter-Class Analogical Augmentation and Intra-Class Significance Analysis for Non-Exemplar Online Task-Free Continual Learning(https://arxiv.org/abs/2404.13576)
• Keywords: privacy
• Abstract: Online task-free continual learning (OTFCL) is a more challenging variant of continual learning which emphasizes the gradual shift of task boundaries and learns in an online mode. Existing methods rely on a memory buffer composed of old samples to prevent forgetting. However,the use of memory buffers not only raises privacy concerns but also hinders the efficient learning of new samples. To address this problem, we propose a novel framework called I2CANSAY that gets rid of the dependence on memory buffers and efficiently learns the knowledge of new data from one-shot samples. Concretely, our framework comprises two main modules. Firstly, the Inter-Class Analogical Augmentation (ICAN) module generates diverse pseudo-features for old classes based on the inter-class analogy of feature distributions for different new classes, serving as a substitute for the memory buffer. Secondly, the Intra-Class Significance Analysis (ISAY) module analyzes the significance of attributes for each class via its distribution standard deviation, and generates the importance vector as a correction bias for the linear classifier, thereby enhancing the capability of learning from new samples. We run our experiments on four popular image classification datasets: CoRe50, CIFAR-10, CIFAR-100, and CUB-200, our approach outperforms the prior state-of-the-art by a large margin.

Title: Rethink Arbitrary Style Transfer with Transformer and Contrastive Learning

• Authors: Zhanjie Zhang, Jiakai Sun, Guangyuan Li, Lei Zhao, Quanwei Zhang, Zehua Lan, Haolin Yin, Wei Xing, Huaizhong Lin, Zhiwen Zuo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13584
• Pdf URL: https://arxiv.org/pdf/2404.13584
• Copy Paste: [[2404.13584]] Rethink Arbitrary Style Transfer with Transformer and Contrastive Learning(https://arxiv.org/abs/2404.13584)
• Keywords: transformer
• Abstract: Arbitrary style transfer holds widespread attention in research and boasts numerous practical applications. The existing methods, which either employ cross-attention to incorporate deep style attributes into content attributes or use adaptive normalization to adjust content features, fail to generate high-quality stylized images. In this paper, we introduce an innovative technique to improve the quality of stylized images. Firstly, we propose Style Consistency Instance Normalization (SCIN), a method to refine the alignment between content and style features. In addition, we have developed an Instance-based Contrastive Learning (ICL) approach designed to understand the relationships among various styles, thereby enhancing the quality of the resulting stylized images. Recognizing that VGG networks are more adept at extracting classification features and need to be better suited for capturing style features, we have also introduced the Perception Encoder (PE) to capture style features. Extensive experiments demonstrate that our proposed method generates high-quality stylized images and effectively prevents artifacts compared with the existing state-of-the-art methods.

Title: MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning

• Authors: Yifan Jiang, Jiarui Zhang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma, Filip Ilievski, Jay Pujara
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13591
• Pdf URL: https://arxiv.org/pdf/2404.13591
• Copy Paste: [[2404.13591]] MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning(https://arxiv.org/abs/2404.13591)
• Keywords: large language model
• Abstract: While multi-modal large language models (MLLMs) have shown significant progress on many popular visual reasoning benchmarks, whether they possess abstract visual reasoning abilities remains an open question. Similar to the Sudoku puzzles, abstract visual reasoning (AVR) problems require finding high-level patterns (e.g., repetition constraints) that control the input shapes (e.g., digits) in a specific task configuration (e.g., matrix). However, existing AVR benchmarks only considered a limited set of patterns (addition, conjunction), input shapes (rectangle, square), and task configurations (3 by 3 matrices). To evaluate MLLMs' reasoning abilities comprehensively, we introduce MARVEL, a multidimensional AVR benchmark with 770 puzzles composed of six core knowledge patterns, geometric and abstract shapes, and five different task configurations. To inspect whether the model accuracy is grounded in perception and reasoning, MARVEL complements the general AVR question with perception questions in a hierarchical evaluation framework. We conduct comprehensive experiments on MARVEL with nine representative MLLMs in zero-shot and few-shot settings. Our experiments reveal that all models show near-random performance on the AVR question, with significant performance gaps (40%) compared to humans across all patterns and task configurations. Further analysis of perception questions reveals that MLLMs struggle to comprehend the visual features (near-random performance) and even count the panels in the puzzle ( <45%), hindering their ability for abstract reasoning. We release our entire code and dataset.

Title: Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers

• Authors: Georgios Pantazopoulos, Alessandro Suglia, Oliver Lemon, Arash Eshghi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13594
• Pdf URL: https://arxiv.org/pdf/2404.13594
• Copy Paste: [[2404.13594]] Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers(https://arxiv.org/abs/2404.13594)
• Keywords: large language model
• Abstract: An effective method for combining frozen large language models (LLM) and visual encoders involves a resampler module that creates a `visual prompt' which is provided to the LLM, along with the textual prompt. While this approach has enabled impressive performance across many coarse-grained tasks like image captioning and visual question answering, more fine-grained tasks that require spatial understanding have not been thoroughly examined. In this paper, we use \textit{diagnostic classifiers} to measure the extent to which the visual prompt produced by the resampler encodes spatial information. Our results show that this information is largely absent from the resampler output when kept frozen during training of the classifiers. However, when the resampler and classifier are trained jointly, we observe a significant performance boost. This shows that the compression achieved by the resamplers can in principle encode the requisite spatial information, but that more object-aware objectives are needed at the pretraining stage to facilitate this capability

Title: "A good pun is its own reword": Can Large Language Models Understand Puns?

• Authors: Zhijun Xu, Siyu Yuan, Lingjie Chen, Deqing Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13599
• Pdf URL: https://arxiv.org/pdf/2404.13599
• Copy Paste: [[2404.13599]] "A good pun is its own reword": Can Large Language Models Understand Puns?(https://arxiv.org/abs/2404.13599)
• Keywords: large language model
• Abstract: Puns play a vital role in academic research due to their distinct structure and clear definition, which aid in the comprehensive analysis of linguistic humor. However, the understanding of puns in large language models (LLMs) has not been thoroughly examined, limiting their use in creative writing and humor creation. In this paper, we leverage three popular tasks, i.e., pun recognition, explanation and generation to systematically evaluate the capabilities of LLMs in pun understanding. In addition to adopting the automated evaluation metrics from prior research, we introduce new evaluation methods and metrics that are better suited to the in-context learning paradigm of LLMs. These new metrics offer a more rigorous assessment of an LLM's ability to understand puns and align more closely with human cognition than previous metrics. Our findings reveal the "lazy pun generation" pattern and identify the primary challenges LLMs encounter in understanding puns.

Title: CKGConv: General Graph Convolution with Continuous Kernels

• Authors: Liheng Ma, Soumyasundar Pal, Yitian Zhang, Jiaming Zhou, Yingxue Zhang, Mark Coates
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13604
• Pdf URL: https://arxiv.org/pdf/2404.13604
• Copy Paste: [[2404.13604]] CKGConv: General Graph Convolution with Continuous Kernels(https://arxiv.org/abs/2404.13604)
• Keywords: transformer
• Abstract: The existing definitions of graph convolution, either from spatial or spectral perspectives, are inflexible and not unified. Defining a general convolution operator in the graph domain is challenging due to the lack of canonical coordinates, the presence of irregular structures, and the properties of graph symmetries. In this work, we propose a novel graph convolution framework by parameterizing the kernels as continuous functions of pseudo-coordinates derived via graph positional encoding. We name this Continuous Kernel Graph Convolution (CKGConv). Theoretically, we demonstrate that CKGConv is flexible and expressive. CKGConv encompasses many existing graph convolutions, and exhibits the same expressiveness as graph transformers in terms of distinguishing non-isomorphic graphs. Empirically, we show that CKGConv-based Networks outperform existing graph convolutional networks and perform comparably to the best graph transformers across a variety of graph datasets.

Title: Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence

• Authors: Ripon Kumar Saha, Dehao Qin, Nianyi Li, Jinwei Ye, Suren Jayasuriya
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.13605
• Pdf URL: https://arxiv.org/pdf/2404.13605
• Copy Paste: [[2404.13605]] Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence(https://arxiv.org/abs/2404.13605)
• Keywords: transformer, segmentation
• Abstract: Tackling image degradation due to atmospheric turbulence, particularly in dynamic environment, remains a challenge for long-range imaging systems. Existing techniques have been primarily designed for static scenes or scenes with small motion. This paper presents the first segment-then-restore pipeline for restoring the videos of dynamic scenes in turbulent environment. We leverage mean optical flow with an unsupervised motion segmentation method to separate dynamic and static scene components prior to restoration. After camera shake compensation and segmentation, we introduce foreground/background enhancement leveraging the statistics of turbulence strength and a transformer model trained on a novel noise-based procedural turbulence generator for fast dataset augmentation. Benchmarked against existing restoration methods, our approach restores most of the geometric distortion and enhances sharpness for videos. We make our code, simulator, and data publicly available to advance the field of video restoration from turbulence: riponcs.github.io/TurbSegRes

Title: Attack on Scene Flow using Point Clouds

• Authors: Haniyeh Ehsani Oskouie, Mohammad-Shahram Moin, Shohreh Kasaei
• Subjects: cs.CV, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.13621
• Pdf URL: https://arxiv.org/pdf/2404.13621
• Copy Paste: [[2404.13621]] Attack on Scene Flow using Point Clouds(https://arxiv.org/abs/2404.13621)
• Keywords: attack, robust
• Abstract: Deep neural networks have made significant advancements in accurately estimating scene flow using point clouds, which is vital for many applications like video analysis, action recognition, and navigation. Robustness of these techniques, however, remains a concern, particularly in the face of adversarial attacks that have been proven to deceive state-of-the-art deep neural networks in many domains. Surprisingly, the robustness of scene flow networks against such attacks has not been thoroughly investigated. To address this problem, the proposed approach aims to bridge this gap by introducing adversarial white-box attacks specifically tailored for scene flow networks. Experimental results show that the generated adversarial examples obtain up to 33.7 relative degradation in average end-point error on the KITTI and FlyingThings3D datasets. The study also reveals the significant impact that attacks targeting point clouds in only one dimension or color channel have on average end-point error. Analyzing the success and failure of these attacks on the scene flow networks and their 2D optical flow network variants show a higher vulnerability for the optical flow networks.

Title: NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding

• Authors: Chunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng, Wei Fan, Haoran Li, Xin Liu, Hongming Zhang, Weiqi Wang, Yangqiu Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13627
• Pdf URL: https://arxiv.org/pdf/2404.13627
• Copy Paste: [[2404.13627]] NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding(https://arxiv.org/abs/2404.13627)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have sparked substantial interest and debate concerning their potential emergence of Theory of Mind (ToM) ability. Theory of mind evaluations currently focuses on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations, which lacks evaluation of machine ToM ability in real-world human interaction scenarios. This poses a pressing demand to develop new real-world scenario benchmarks. We introduce NegotiationToM, a new benchmark designed to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states (i.e., desires, beliefs, and intentions). Our benchmark builds upon the Belief-Desire-Intention (BDI) agent modeling theory and conducts the necessary empirical experiments to evaluate large language models. Our findings demonstrate that NegotiationToM is challenging for state-of-the-art LLMs, as they consistently perform significantly worse than humans, even when employing the chain-of-thought (CoT) method.

Title: Mixture of LoRA Experts

• Authors: Xun Wu, Shaohan Huang, Furu Wei
• Subjects: cs.CL, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.13628
• Pdf URL: https://arxiv.org/pdf/2404.13628
• Copy Paste: [[2404.13628]] Mixture of LoRA Experts(https://arxiv.org/abs/2404.13628)
• Keywords: generative
• Abstract: LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques. Due to the modular nature of LoRA's plug-and-play plugins, researchers have delved into the amalgamation of multiple LoRAs to empower models to excel across various downstream tasks. Nonetheless, extant approaches for LoRA fusion grapple with inherent challenges. Direct arithmetic merging may result in the loss of the original pre-trained model's generative capabilities or the distinct identity of LoRAs, thereby yielding suboptimal outcomes. On the other hand, Reference tuning-based fusion exhibits limitations concerning the requisite flexibility for the effective combination of multiple LoRAs. In response to these challenges, this paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection. The MoLE approach not only achieves superior LoRA fusion performance in comparison to direct arithmetic merging but also retains the crucial flexibility for combining LoRAs effectively. Extensive experimental evaluations conducted in both the Natural Language Processing (NLP) and Vision & Language (V&L) domains substantiate the efficacy of MoLE.

Title: Bt-GAN: Generating Fair Synthetic Healthdata via Bias-transforming Generative Adversarial Networks

• Authors: Resmi Ramachandranpillai, Md Fahim Sikder, David Bergström, Fredrik Heintz
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13634
• Pdf URL: https://arxiv.org/pdf/2404.13634
• Copy Paste: [[2404.13634]] Bt-GAN: Generating Fair Synthetic Healthdata via Bias-transforming Generative Adversarial Networks(https://arxiv.org/abs/2404.13634)
• Keywords: fair, explainability, generative
• Abstract: Synthetic data generation offers a promising solution to enhance the usefulness of Electronic Healthcare Records (EHR) by generating realistic de-identified data. However, the existing literature primarily focuses on the quality of synthetic health data, neglecting the crucial aspect of fairness in downstream predictions. Consequently, models trained on synthetic EHR have faced criticism for producing biased outcomes in target tasks. These biases can arise from either spurious correlations between features or the failure of models to accurately represent sub-groups. To address these concerns, we present Bias-transforming Generative Adversarial Networks (Bt-GAN), a GAN-based synthetic data generator specifically designed for the healthcare domain. In order to tackle spurious correlations (i), we propose an information-constrained Data Generation Process that enables the generator to learn a fair deterministic transformation based on a well-defined notion of algorithmic fairness. To overcome the challenge of capturing exact sub-group representations (ii), we incentivize the generator to preserve sub-group densities through score-based weighted sampling. This approach compels the generator to learn from underrepresented regions of the data manifold. We conduct extensive experiments using the MIMIC-III database. Our results demonstrate that Bt-GAN achieves SOTA accuracy while significantly improving fairness and minimizing bias amplification. We also perform an in-depth explainability analysis to provide additional evidence supporting the validity of our study. In conclusion, our research introduces a novel and professional approach to addressing the limitations of synthetic data generation in the healthcare domain. By incorporating fairness considerations and leveraging advanced techniques such as GANs, we pave the way for more reliable and unbiased predictions in healthcare applications.

Title: PEACH: Pretrained-embedding Explanation Across Contextual and Hierarchical Structure

• Authors: Feiqi Cao, Caren Han, Hyunsuk Chung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13645
• Pdf URL: https://arxiv.org/pdf/2404.13645
• Copy Paste: [[2404.13645]] PEACH: Pretrained-embedding Explanation Across Contextual and Hierarchical Structure(https://arxiv.org/abs/2404.13645)
• Keywords: interpretability
• Abstract: In this work, we propose a novel tree-based explanation technique, PEACH (Pretrained-embedding Explanation Across Contextual and Hierarchical Structure), that can explain how text-based documents are classified by using any pretrained contextual embeddings in a tree-based human-interpretable manner. Note that PEACH can adopt any contextual embeddings of the PLMs as a training input for the decision tree. Using the proposed PEACH, we perform a comprehensive analysis of several contextual embeddings on nine different NLP text classification benchmarks. This analysis demonstrates the flexibility of the model by applying several PLM contextual embeddings, its attribute selections, scaling, and clustering methods. Furthermore, we show the utility of explanations by visualising the feature selection and important trend of text classification via human-interpretable word-cloud-based trees, which clearly identify model mistakes and assist in dataset debugging. Besides interpretability, PEACH outperforms or is similar to those from pretrained models.

Title: Mean Aggregator Is More Robust Than Robust Aggregators Under Label Poisoning Attacks

• Authors: Jie Peng, Weiyu Li, Qing Ling
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13647
• Pdf URL: https://arxiv.org/pdf/2404.13647
• Copy Paste: [[2404.13647]] Mean Aggregator Is More Robust Than Robust Aggregators Under Label Poisoning Attacks(https://arxiv.org/abs/2404.13647)
• Keywords: attack, robust
• Abstract: Robustness to malicious attacks is of paramount importance for distributed learning. Existing works often consider the classical Byzantine attacks model, which assumes that some workers can send arbitrarily malicious messages to the server and disturb the aggregation steps of the distributed learning process. To defend against such worst-case Byzantine attacks, various robust aggregators have been proven effective and much superior to the often-used mean aggregator. In this paper, we show that robust aggregators are too conservative for a class of weak but practical malicious attacks, as known as label poisoning attacks, where the sample labels of some workers are poisoned. Surprisingly, we are able to show that the mean aggregator is more robust than the state-of-the-art robust aggregators in theory, given that the distributed data are sufficiently heterogeneous. In fact, the learning error of the mean aggregator is proven to be optimal in order. Experimental results corroborate our theoretical findings, demonstrating the superiority of the mean aggregator under label poisoning attacks.

Title: Data-independent Module-aware Pruning for Hierarchical Vision Transformers

• Authors: Yang He, Joey Tianyi Zhou
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13648
• Pdf URL: https://arxiv.org/pdf/2404.13648
• Copy Paste: [[2404.13648]] Data-independent Module-aware Pruning for Hierarchical Vision Transformers(https://arxiv.org/abs/2404.13648)
• Keywords: fair, transformer
• Abstract: Hierarchical vision transformers (ViTs) have two advantages over conventional ViTs. First, hierarchical ViTs achieve linear computational complexity with respect to image size by local self-attention. Second, hierarchical ViTs create hierarchical feature maps by merging image patches in deeper layers for dense prediction. However, existing pruning methods ignore the unique properties of hierarchical ViTs and use the magnitude value as the weight importance. This approach leads to two main drawbacks. First, the "local" attention weights are compared at a "global" level, which may cause some "locally" important weights to be pruned due to their relatively small magnitude "globally". The second issue with magnitude pruning is that it fails to consider the distinct weight distributions of the network, which are essential for extracting coarse to fine-grained features at various hierarchical levels. To solve the aforementioned issues, we have developed a Data-independent Module-Aware Pruning method (DIMAP) to compress hierarchical ViTs. To ensure that "local" attention weights at different hierarchical levels are compared fairly in terms of their contribution, we treat them as a module and examine their contribution by analyzing their information distortion. Furthermore, we introduce a novel weight metric that is solely based on weights and does not require input images, thereby eliminating the dependence on the patch merging process. Our method validates its usefulness and strengths on Swin Transformers of different sizes on ImageNet-1k classification. Notably, the top-5 accuracy drop is only 0.07% when we remove 52.5% FLOPs and 52.7% parameters of Swin-B. When we reduce 33.2% FLOPs and 33.2% parameters of Swin-S, we can even achieve a 0.8% higher relative top-5 accuracy than the original model. Code is available at: https://github.com/he-y/Data-independent-Module-Aware-Pruning

Title: LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing

• Authors: Tong Wang, Guanzhou Chen, Xiaodong Zhang, Chenxi Liu, Xiaoliang Tan, Jiaqi Wang, Chanjuan He, Wenlin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13659
• Pdf URL: https://arxiv.org/pdf/2404.13659
• Copy Paste: [[2404.13659]] LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing(https://arxiv.org/abs/2404.13659)
• Keywords: robust, extraction, transformer, segmentation
• Abstract: Despite the rapid evolution of semantic segmentation for land cover classification in high-resolution remote sensing imagery, integrating multiple data modalities such as Digital Surface Model (DSM), RGB, and Near-infrared (NIR) remains a challenge. Current methods often process only two types of data, missing out on the rich information that additional modalities can provide. Addressing this gap, we propose a novel \textbf{L}ightweight \textbf{M}ultimodal data \textbf{F}usion \textbf{Net}work (LMFNet) to accomplish the tasks of fusion and semantic segmentation of multimodal remote sensing images. LMFNet uniquely accommodates various data types simultaneously, including RGB, NirRG, and DSM, through a weight-sharing, multi-branch vision transformer that minimizes parameter count while ensuring robust feature extraction. Our proposed multimodal fusion module integrates a \textit{Multimodal Feature Fusion Reconstruction Layer} and \textit{Multimodal Feature Self-Attention Fusion Layer}, which can reconstruct and fuse multimodal features. Extensive testing on public datasets such as US3D, ISPRS Potsdam, and ISPRS Vaihingen demonstrates the effectiveness of LMFNet. Specifically, it achieves a mean Intersection over Union ($mIoU$) of 85.09\% on the US3D dataset, marking a significant improvement over existing methods. Compared to unimodal approaches, LMFNet shows a 10\% enhancement in $mIoU$ with only a 0.5M increase in parameter count. Furthermore, against bimodal methods, our approach with trilateral inputs enhances $mIoU$ by 0.46 percentage points.

Title: Trojan Detection in Large Language Models: Insights from The Trojan Detection Challenge

• Authors: Narek Maloyan, Ekansh Verma, Bulat Nutfullin, Bislan Ashinov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13660
• Pdf URL: https://arxiv.org/pdf/2404.13660
• Copy Paste: [[2404.13660]] Trojan Detection in Large Language Models: Insights from The Trojan Detection Challenge(https://arxiv.org/abs/2404.13660)
• Keywords: security, attack, robust, interpretability, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in various domains, but their vulnerability to trojan or backdoor attacks poses significant security risks. This paper explores the challenges and insights gained from the Trojan Detection Competition 2023 (TDC2023), which focused on identifying and evaluating trojan attacks on LLMs. We investigate the difficulty of distinguishing between intended and unintended triggers, as well as the feasibility of reverse engineering trojans in real-world scenarios. Our comparative analysis of various trojan detection methods reveals that achieving high Recall scores is significantly more challenging than obtaining high Reverse-Engineering Attack Success Rate (REASR) scores. The top-performing methods in the competition achieved Recall scores around 0.16, comparable to a simple baseline of randomly sampling sentences from a distribution similar to the given training prefixes. This finding raises questions about the detectability and recoverability of trojans inserted into the model, given only the harmful targets. Despite the inability to fully solve the problem, the competition has led to interesting observations about the viability of trojan detection and improved techniques for optimizing LLM input prompts. The phenomenon of unintended triggers and the difficulty in distinguishing them from intended triggers highlights the need for further research into the robustness and interpretability of LLMs. The TDC2023 has provided valuable insights into the challenges and opportunities associated with trojan detection in LLMs, laying the groundwork for future research in this area to ensure their safety and reliability in real-world applications.

Title: MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition

• Authors: Felix M. Schmitt-Koopmann, Elaine M. Huang, Hans-Peter Hutter, Thilo Stadelmann, Alireza Darvishy
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13667
• Pdf URL: https://arxiv.org/pdf/2404.13667
• Copy Paste: [[2404.13667]] MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition(https://arxiv.org/abs/2404.13667)
• Keywords: transformer
• Abstract: Printed mathematical expression recognition (MER) models are usually trained and tested using LaTeX-generated mathematical expressions (MEs) as input and the LaTeX source code as ground truth. As the same ME can be generated by various different LaTeX source codes, this leads to unwanted variations in the ground truth data that bias test performance results and hinder efficient learning. In addition, the use of only one font to generate the MEs heavily limits the generalization of the reported results to realistic scenarios. We propose a data-centric approach to overcome this problem, and present convincing experimental results: Our main contribution is an enhanced LaTeX normalization to map any LaTeX ME to a canonical form. Based on this process, we developed an improved version of the benchmark dataset im2latex-100k, featuring 30 fonts instead of one. Second, we introduce the real-world dataset realFormula, with MEs extracted from papers. Third, we developed a MER model, MathNet, based on a convolutional vision transformer, with superior results on all four test sets (im2latex-100k, im2latexv2, realFormula, and InftyMDB-1), outperforming the previous state of the art by up to 88.3%.

Title: FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization

• Authors: Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Hao Li, Ming Tang, Jinqiao Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13671
• Pdf URL: https://arxiv.org/pdf/2404.13671
• Copy Paste: [[2404.13671]] FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization(https://arxiv.org/abs/2404.13671)
• Keywords: robust, interpretability, large language model
• Abstract: Zero-shot anomaly detection (ZSAD) methods entail detecting anomalies directly without access to any known normal or abnormal samples within the target item categories. Existing approaches typically rely on the robust generalization capabilities of multimodal pretrained models, computing similarities between manually crafted textual features representing "normal" or "abnormal" semantics and image features to detect anomalies and localize anomalous patches. However, the generic descriptions of "abnormal" often fail to precisely match diverse types of anomalies across different object categories. Additionally, computing feature similarities for single patches struggles to pinpoint specific locations of anomalies with various sizes and scales. To address these issues, we propose a novel ZSAD method called FiLo, comprising two components: adaptively learned Fine-Grained Description (FG-Des) and position-enhanced High-Quality Localization (HQ-Loc). FG-Des introduces fine-grained anomaly descriptions for each category using Large Language Models (LLMs) and employs adaptively learned textual templates to enhance the accuracy and interpretability of anomaly detection. HQ-Loc, utilizing Grounding DINO for preliminary localization, position-enhanced text prompts, and Multi-scale Multi-shape Cross-modal Interaction (MMCI) module, facilitates more accurate localization of anomalies of different sizes and shapes. Experimental results on datasets like MVTec and VisA demonstrate that FiLo significantly improves the performance of ZSAD in both detection and localization, achieving state-of-the-art performance with an image-level AUC of 83.9% and a pixel-level AUC of 95.9% on the VisA dataset.

Title: A Dataset and Model for Realistic License Plate Deblurring

• Authors: Haoyan Gong, Yuzheng Feng, Zhenrong Zhang, Xianxu Hou, Jingxin Liu, Siqi Huang, Hongbin Liu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.13677
• Pdf URL: https://arxiv.org/pdf/2404.13677
• Copy Paste: [[2404.13677]] A Dataset and Model for Realistic License Plate Deblurring(https://arxiv.org/abs/2404.13677)
• Keywords: generative
• Abstract: Vehicle license plate recognition is a crucial task in intelligent traffic management systems. However, the challenge of achieving accurate recognition persists due to motion blur from fast-moving vehicles. Despite the widespread use of image synthesis approaches in existing deblurring and recognition algorithms, their effectiveness in real-world scenarios remains unproven. To address this, we introduce the first large-scale license plate deblurring dataset named License Plate Blur (LPBlur), captured by a dual-camera system and processed through a post-processing pipeline to avoid misalignment issues. Then, we propose a License Plate Deblurring Generative Adversarial Network (LPDGAN) to tackle the license plate deblurring: 1) a Feature Fusion Module to integrate multi-scale latent codes; 2) a Text Reconstruction Module to restore structure through textual modality; 3) a Partition Discriminator Module to enhance the model's perception of details in each letter. Extensive experiments validate the reliability of the LPBlur dataset for both model training and testing, showcasing that our proposed model outperforms other state-of-the-art motion deblurring methods in realistic license plate deblurring scenarios. The dataset and code are available at https://github.com/haoyGONG/LPDGAN.

Title: GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removal

• Authors: Yuxin Wang, Qianyi Wu, Guofeng Zhang, Dan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13679
• Pdf URL: https://arxiv.org/pdf/2404.13679
• Copy Paste: [[2404.13679]] GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removal(https://arxiv.org/abs/2404.13679)
• Keywords: robust
• Abstract: This paper tackles the intricate challenge of object removal to update the radiance field using the 3D Gaussian Splatting. The main challenges of this task lie in the preservation of geometric consistency and the maintenance of texture coherence in the presence of the substantial discrete nature of Gaussian primitives. We introduce a robust framework specifically designed to overcome these obstacles. The key insight of our approach is the enhancement of information exchange among visible and invisible areas, facilitating content restoration in terms of both geometry and texture. Our methodology begins with optimizing the positioning of Gaussian primitives to improve geometric consistency across both removed and visible areas, guided by an online registration process informed by monocular depth estimation. Following this, we employ a novel feature propagation mechanism to bolster texture coherence, leveraging a cross-attention design that bridges sampling Gaussians from both uncertain and certain areas. This innovative approach significantly refines the texture coherence within the final radiance field. Extensive experiments validate that our method not only elevates the quality of novel view synthesis for scenes undergoing object removal but also showcases notable efficiency gains in training and rendering speeds.

Title: Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis

• Authors: Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, Xuefeng Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13686
• Pdf URL: https://arxiv.org/pdf/2404.13686
• Copy Paste: [[2404.13686]] Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis(https://arxiv.org/abs/2404.13686)
• Keywords: diffusion
• Abstract: Recently, a series of diffusion-aware distillation algorithms have emerged to alleviate the computational overhead associated with the multi-step inference process of Diffusion Models (DMs). Current distillation techniques often dichotomize into two distinct aspects: i) ODE Trajectory Preservation; and ii) ODE Trajectory Reformulation. However, these approaches suffer from severe performance degradation or domain shifts. To address these limitations, we propose Hyper-SD, a novel framework that synergistically amalgamates the advantages of ODE Trajectory Preservation and Reformulation, while maintaining near-lossless performance during step compression. Firstly, we introduce Trajectory Segmented Consistency Distillation to progressively perform consistent distillation within pre-defined time-step segments, which facilitates the preservation of the original ODE trajectory from a higher-order perspective. Secondly, we incorporate human feedback learning to boost the performance of the model in a low-step regime and mitigate the performance loss incurred by the distillation process. Thirdly, we integrate score distillation to further improve the low-step generation capability of the model and offer the first attempt to leverage a unified LoRA to support the inference process at all steps. Extensive experiments and user studies demonstrate that Hyper-SD achieves SOTA performance from 1 to 8 inference steps for both SDXL and SD1.5. For example, Hyper-SDXL surpasses SDXL-Lightning by +0.68 in CLIP Score and +0.51 in Aes Score in the 1-step inference.

Title: Detecting Compromised IoT Devices Using Autoencoders with Sequential Hypothesis Testing

• Authors: Md Mainuddin, Zhenhai Duan, Yingfei Dong
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13690
• Pdf URL: https://arxiv.org/pdf/2404.13690
• Copy Paste: [[2404.13690]] Detecting Compromised IoT Devices Using Autoencoders with Sequential Hypothesis Testing(https://arxiv.org/abs/2404.13690)
• Keywords: security, protect, attack
• Abstract: IoT devices fundamentally lack built-in security mechanisms to protect themselves from security attacks. Existing works on improving IoT security mostly focus on detecting anomalous behaviors of IoT devices. However, these existing anomaly detection schemes may trigger an overwhelmingly large number of false alerts, rendering them unusable in detecting compromised IoT devices. In this paper we develop an effective and efficient framework, named CUMAD, to detect compromised IoT devices. Instead of directly relying on individual anomalous events, CUMAD aims to accumulate sufficient evidence in detecting compromised IoT devices, by integrating an autoencoder-based anomaly detection subsystem with a sequential probability ratio test (SPRT)-based sequential hypothesis testing subsystem. CUMAD can effectively reduce the number of false alerts in detecting compromised IoT devices, and moreover, it can detect compromised IoT devices quickly. Our evaluation studies based on the public-domain N-BaIoT dataset show that CUMAD can on average reduce the false positive rate from about 3.57% using only the autoencoder-based anomaly detection scheme to about 0.5%; in addition, CUMAD can detect compromised IoT devices quickly, with less than 5 observations on average.

Title: A Complete System for Automated 3D Semantic-Geometric Mapping of Corrosion in Industrial Environments

• Authors: Rui Pimentel de Figueiredo, Stefan Nordborg Eriksen, Ignacio Rodriguez, Simon Bøgh
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.13691
• Pdf URL: https://arxiv.org/pdf/2404.13691
• Copy Paste: [[2404.13691]] A Complete System for Automated 3D Semantic-Geometric Mapping of Corrosion in Industrial Environments(https://arxiv.org/abs/2404.13691)
• Keywords: segmentation
• Abstract: Corrosion, a naturally occurring process leading to the deterioration of metallic materials, demands diligent detection for quality control and the preservation of metal-based objects, especially within industrial contexts. Traditional techniques for corrosion identification, including ultrasonic testing, radio-graphic testing, and magnetic flux leakage, necessitate the deployment of expensive and bulky equipment on-site for effective data acquisition. An unexplored alternative involves employing lightweight, conventional camera systems, and state-of-the-art computer vision methods for its identification. In this work, we propose a complete system for semi-automated corrosion identification and mapping in industrial environments. We leverage recent advances in LiDAR-based methods for localization and mapping, with vision-based semantic segmentation deep learning techniques, in order to build semantic-geometric maps of industrial environments. Unlike previous corrosion identification systems available in the literature, our designed multi-modal system is low-cost, portable, semi-autonomous and allows collecting large datasets by untrained personnel. A set of experiments in an indoor laboratory environment, demonstrate quantitatively the high accuracy of the employed LiDAR based 3D mapping and localization system, with less then $0.05m$ and 0.02m average absolute and relative pose errors. Also, our data-driven semantic segmentation model, achieves around 70\% precision when trained with our pixel-wise manually annotated dataset.

Title: Semantic-Rearrangement-Based Multi-Level Alignment for Domain Generalized Segmentation

• Authors: Guanlong Jiao, Chenyangguang Zhang, Haonan Yin, Yu Mo, Biqing Huang, Hui Pan, Yi Luo, Jingxian Liu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13701
• Pdf URL: https://arxiv.org/pdf/2404.13701
• Copy Paste: [[2404.13701]] Semantic-Rearrangement-Based Multi-Level Alignment for Domain Generalized Segmentation(https://arxiv.org/abs/2404.13701)
• Keywords: robust, segmentation
• Abstract: Domain generalized semantic segmentation is an essential computer vision task, for which models only leverage source data to learn the capability of generalized semantic segmentation towards the unseen target domains. Previous works typically address this challenge by global style randomization or feature regularization. In this paper, we argue that given the observation that different local semantic regions perform different visual characteristics from the source domain to the target domain, methods focusing on global operations are hard to capture such regional discrepancies, thus failing to construct domain-invariant representations with the consistency from local to global level. Therefore, we propose the Semantic-Rearrangement-based Multi-Level Alignment (SRMA) to overcome this problem. SRMA first incorporates a Semantic Rearrangement Module (SRM), which conducts semantic region randomization to enhance the diversity of the source domain sufficiently. A Multi-Level Alignment module (MLA) is subsequently proposed with the help of such diversity to establish the global-regional-local consistent domain-invariant representations. By aligning features across randomized samples with domain-neutral knowledge at multiple levels, SRMA provides a more robust way to handle the source-target domain gap. Extensive experiments demonstrate the superiority of SRMA over the current state-of-the-art works on various benchmarks.

Title: Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Models

• Authors: Vitali Petsiuk, Kate Saenko
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13706
• Pdf URL: https://arxiv.org/pdf/2404.13706
• Copy Paste: [[2404.13706]] Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Models(https://arxiv.org/abs/2404.13706)
• Keywords: attack, diffusion
• Abstract: Motivated by ethical and legal concerns, the scientific community is actively developing methods to limit the misuse of Text-to-Image diffusion models for reproducing copyrighted, violent, explicit, or personal information in the generated images. Simultaneously, researchers put these newly developed safety measures to the test by assuming the role of an adversary to find vulnerabilities and backdoors in them. We use compositional property of diffusion models, which allows to leverage multiple prompts in a single image generation. This property allows us to combine other concepts, that should not have been affected by the inhibition, to reconstruct the vector, responsible for target concept generation, even though the direct computation of this vector is no longer accessible. We provide theoretical and empirical evidence why the proposed attacks are possible and discuss the implications of these findings for safe model deployment. We argue that it is essential to consider all possible approaches to image generation with diffusion models that can be employed by an adversary. Our work opens up the discussion about the implications of concept arithmetics and compositional inference for safety mechanisms in diffusion models. Content Advisory: This paper contains discussions and model-generated content that may be considered offensive. Reader discretion is advised. Project page: https://cs-people.bu.edu/vpetsiuk/arc

Title: SVGEditBench: A Benchmark Dataset for Quantitative Assessment of LLM's SVG Editing Capabilities

• Authors: Kunato Nishina, Yusuke Matsui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13710
• Pdf URL: https://arxiv.org/pdf/2404.13710
• Copy Paste: [[2404.13710]] SVGEditBench: A Benchmark Dataset for Quantitative Assessment of LLM's SVG Editing Capabilities(https://arxiv.org/abs/2404.13710)
• Keywords: large language model
• Abstract: Text-to-image models have shown progress in recent years. Along with this progress, generating vector graphics from text has also advanced. SVG is a popular format for vector graphics, and SVG represents a scene with XML text. Therefore, Large Language Models can directly process SVG code. Taking this into account, we focused on editing SVG with LLMs. For quantitative evaluation of LLMs' ability to edit SVG, we propose SVGEditBench. SVGEditBench is a benchmark for assessing the LLMs' ability to edit SVG code. We also show the GPT-4 and GPT-3.5 results when evaluated on the proposed benchmark. In the experiments, GPT-4 showed superior performance to GPT-3.5 both quantitatively and qualitatively. The dataset is available at https://github.com/mti-lab/SVGEditBench.

Title: ArtNeRF: A Stylized Neural Field for 3D-Aware Cartoonized Face Synthesis

• Authors: Zichen Tang, Hongyu Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13711
• Pdf URL: https://arxiv.org/pdf/2404.13711
• Copy Paste: [[2404.13711]] ArtNeRF: A Stylized Neural Field for 3D-Aware Cartoonized Face Synthesis(https://arxiv.org/abs/2404.13711)
• Keywords: robust, generative
• Abstract: Recent advances in generative visual models and neural radiance fields have greatly boosted 3D-aware image synthesis and stylization tasks. However, previous NeRF-based work is limited to single scene stylization, training a model to generate 3D-aware cartoon faces with arbitrary styles remains unsolved. We propose ArtNeRF, a novel face stylization framework derived from 3D-aware GAN to tackle this problem. In this framework, we utilize an expressive generator to synthesize stylized faces and a triple-branch discriminator module to improve the visual quality and style consistency of the generated faces. Specifically, a style encoder based on contrastive learning is leveraged to extract robust low-dimensional embeddings of style images, empowering the generator with the knowledge of various styles. To smooth the training process of cross-domain transfer learning, we propose an adaptive style blending module which helps inject style information and allows users to freely tune the level of stylization. We further introduce a neural rendering module to achieve efficient real-time rendering of images with higher resolutions. Extensive experiments demonstrate that ArtNeRF is versatile in generating high-quality 3D-aware cartoon faces with arbitrary styles.

Title: Interval Abstractions for Robust Counterfactual Explanations

• Authors: Junqi Jiang, Francesco Leofante, Antonio Rago, Francesca Toni
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13736
• Pdf URL: https://arxiv.org/pdf/2404.13736
• Copy Paste: [[2404.13736]] Interval Abstractions for Robust Counterfactual Explanations(https://arxiv.org/abs/2404.13736)
• Keywords: robust
• Abstract: Counterfactual Explanations (CEs) have emerged as a major paradigm in explainable AI research, providing recourse recommendations for users affected by the decisions of machine learning models. However, when slight changes occur in the parameters of the underlying model, CEs found by existing methods often become invalid for the updated models. The literature lacks a way to certify deterministic robustness guarantees for CEs under model changes, in that existing methods to improve CEs' robustness are heuristic, and the robustness performances are evaluated empirically using only a limited number of retrained models. To bridge this gap, we propose a novel interval abstraction technique for parametric machine learning models, which allows us to obtain provable robustness guarantees of CEs under the possibly infinite set of plausible model changes $\Delta$. We formalise our robustness notion as the $\Delta$-robustness for CEs, in both binary and multi-class classification settings. We formulate procedures to verify $\Delta$-robustness based on Mixed Integer Linear Programming, using which we further propose two algorithms to generate CEs that are $\Delta$-robust. In an extensive empirical study, we demonstrate how our approach can be used in practice by discussing two strategies for determining the appropriate hyperparameter in our method, and we quantitatively benchmark the CEs generated by eleven methods, highlighting the effectiveness of our algorithms in finding robust CEs.

Title: A Nasal Cytology Dataset for Object Detection and Deep Learning

• Authors: Mauro Camporeale, Giovanni Dimauro, Matteo Gelardi, Giorgia Iacobellis, Mattia Sebastiano Ladisa, Sergio Latrofa, Nunzia Lomonte
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13745
• Pdf URL: https://arxiv.org/pdf/2404.13745
• Copy Paste: [[2404.13745]] A Nasal Cytology Dataset for Object Detection and Deep Learning(https://arxiv.org/abs/2404.13745)
• Keywords: diffusion
• Abstract: Nasal Cytology is a new and efficient clinical technique to diagnose rhinitis and allergies that is not much widespread due to the time-consuming nature of cell counting; that is why AI-aided counting could be a turning point for the diffusion of this technique. In this article we present the first dataset of rhino-cytological field images: the NCD (Nasal Cytology Dataset), aimed to train and deploy Object Detection models to support physicians and biologists during clinical practice. The real distribution of the cytotypes, populating the nasal mucosa has been replicated, sampling images from slides of clinical patients, and manually annotating each cell found on them. The correspondent object detection task presents non'trivial issues associated with the strong class imbalancement, involving the rarest cell types. This work contributes to some of open challenges by presenting a novel machine learning-based approach to aid the automated detection and classification of nasal mucosa cells: the DETR and YOLO models shown good performance in detecting cells and classifying them correctly, revealing great potential to accelerate the work of rhinology experts.

Title: Embarrassingly Simple Unsupervised Aspect Based Sentiment Tuple Extraction

• Authors: Kevin Scaria, Abyn Scaria, Ben Scaria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13751
• Pdf URL: https://arxiv.org/pdf/2404.13751
• Copy Paste: [[2404.13751]] Embarrassingly Simple Unsupervised Aspect Based Sentiment Tuple Extraction(https://arxiv.org/abs/2404.13751)
• Keywords: extraction
• Abstract: Aspect Based Sentiment Analysis (ABSA) tasks involve the extraction of fine-grained sentiment tuples from sentences, aiming to discern the author's opinions. Conventional methodologies predominantly rely on supervised approaches; however, the efficacy of such methods diminishes in low-resource domains lacking labeled datasets since they often lack the ability to generalize across domains. To address this challenge, we propose a simple and novel unsupervised approach to extract opinion terms and the corresponding sentiment polarity for aspect terms in a sentence. Our experimental evaluations, conducted on four benchmark datasets, demonstrate compelling performance to extract the aspect oriented opinion words as well as assigning sentiment polarity. Additionally, unsupervised approaches for opinion word mining have not been explored and our work establishes a benchmark for the same.

Title: Towards General Conceptual Model Editing via Adversarial Representation Engineering

• Authors: Yihao Zhang, Zeming Wei, Jun Sun, Meng Sun
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR, math.OC
• Abstract URL: https://arxiv.org/abs/2404.13752
• Pdf URL: https://arxiv.org/pdf/2404.13752
• Copy Paste: [[2404.13752]] Towards General Conceptual Model Editing via Adversarial Representation Engineering(https://arxiv.org/abs/2404.13752)
• Keywords: generative, large language model
• Abstract: Recent research has introduced Representation Engineering (RepE) as a promising approach for understanding complex inner workings of large-scale models like Large Language Models (LLMs). However, finding practical and efficient methods to apply these representations for general and flexible model editing remains an open problem. Inspired by the Generative Adversarial Network (GAN) framework, we introduce a novel approach called Adversarial Representation Engineering (ARE). This method leverages RepE by using a representation sensor to guide the editing of LLMs, offering a unified and interpretable framework for conceptual model editing without degrading baseline performance. Our experiments on multiple conceptual editing confirm ARE's effectiveness. Code and data are available at https://github.com/Zhang-Yihao/Adversarial-Representation-Engineering.

Title: Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control

• Authors: Maria Mihaela Trusca, Wolf Nuyts, Jonathan Thomm, Robert Honig, Thomas Hofmann, Tinne Tuytelaars, Marie-Francine Moens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13766
• Pdf URL: https://arxiv.org/pdf/2404.13766
• Copy Paste: [[2404.13766]] Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control(https://arxiv.org/abs/2404.13766)
• Keywords: diffusion, transformer
• Abstract: Current diffusion models create photorealistic images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. This is evidenced by our novel image-graph alignment model called EPViT (Edge Prediction Vision Transformer) for the evaluation of image-text alignment. To alleviate the above problem, we propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. Additionally, the syntax structure of the prompt helps to disentangle the multimodal CLIP embeddings that are commonly used in T2I generation. The resulting DisCLIP embeddings and FCA are easily integrated in state-of-the-art diffusion models without additional training of these models. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.\footnote{Code and data will be made available upon acceptance.

Title: EncodeNet: A Framework for Boosting DNN Accuracy with Entropy-driven Generalized Converting Autoencoder

• Authors: Hasanul Mahmud, Kevin Desai, Palden Lama, Sushil K. Prasad
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13770
• Pdf URL: https://arxiv.org/pdf/2404.13770
• Copy Paste: [[2404.13770]] EncodeNet: A Framework for Boosting DNN Accuracy with Entropy-driven Generalized Converting Autoencoder(https://arxiv.org/abs/2404.13770)
• Keywords: extraction
• Abstract: Image classification is a fundamental task in computer vision, and the quest to enhance DNN accuracy without inflating model size or latency remains a pressing concern. We make a couple of advances in this regard, leading to a novel EncodeNet design and training framework. The first advancement involves Converting Autoencoders, a novel approach that transforms images into an easy-to-classify image of its class. Our prior work that applied the Converting Autoencoder and a simple classifier in tandem achieved moderate accuracy over simple datasets, such as MNIST and FMNIST. However, on more complex datasets like CIFAR-10, the Converting Autoencoder has a large reconstruction loss, making it unsuitable for enhancing DNN accuracy. To address these limitations, we generalize the design of Converting Autoencoders by leveraging a larger class of DNNs, those with architectures comprising feature extraction layers followed by classification layers. We incorporate a generalized algorithmic design of the Converting Autoencoder and intraclass clustering to identify representative images, leading to optimized image feature learning. Next, we demonstrate the effectiveness of our EncodeNet design and training framework, improving the accuracy of well-trained baseline DNNs while maintaining the overall model size. EncodeNet's building blocks comprise the trained encoder from our generalized Converting Autoencoders transferring knowledge to a lightweight classifier network - also extracted from the baseline DNN. Our experimental results demonstrate that EncodeNet improves the accuracy of VGG16 from 92.64% to 94.05% on CIFAR-10 and RestNet20 from 74.56% to 76.04% on CIFAR-100. It outperforms state-of-the-art techniques that rely on knowledge distillation and attention mechanisms, delivering higher accuracy for models of comparable size.

Title: Automated Text Mining of Experimental Methodologies from Biomedical Literature

• Authors: Ziqing Guo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13779
• Pdf URL: https://arxiv.org/pdf/2404.13779
• Copy Paste: [[2404.13779]] Automated Text Mining of Experimental Methodologies from Biomedical Literature(https://arxiv.org/abs/2404.13779)
• Keywords: generative
• Abstract: Biomedical literature is a rapidly expanding field of science and technology. Classification of biomedical texts is an essential part of biomedicine research, especially in the field of biology. This work proposes the fine-tuned DistilBERT, a methodology-specific, pre-trained generative classification language model for mining biomedicine texts. The model has proven its effectiveness in linguistic understanding capabilities and has reduced the size of BERT models by 40\% but by 60\% faster. The main objective of this project is to improve the model and assess the performance of the model compared to the non-fine-tuned model. We used DistilBert as a support model and pre-trained on a corpus of 32,000 abstracts and complete text articles; our results were impressive and surpassed those of traditional literature classification methods by using RNN or LSTM. Our aim is to integrate this highly specialised and specific model into different research industries.

Title: Evaluating Retrieval Quality in Retrieval-Augmented Generation

• Authors: Alireza Salemi, Hamed Zamani
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.13781
• Pdf URL: https://arxiv.org/pdf/2404.13781
• Copy Paste: [[2404.13781]] Evaluating Retrieval Quality in Retrieval-Augmented Generation(https://arxiv.org/abs/2404.13781)
• Keywords: large language model
• Abstract: Evaluating retrieval-augmented generation (RAG) presents challenges, particularly for retrieval models within these systems. Traditional end-to-end evaluation methods are computationally expensive. Furthermore, evaluation of the retrieval model's performance based on query-document relevance labels shows a small correlation with the RAG system's downstream performance. We propose a novel evaluation approach, eRAG, where each document in the retrieval list is individually utilized by the large language model within the RAG system. The output generated for each document is then evaluated based on the downstream task ground truth labels. In this manner, the downstream performance for each document serves as its relevance label. We employ various downstream task metrics to obtain document-level annotations and aggregate them using set-based or ranking metrics. Extensive experiments on a wide range of datasets demonstrate that eRAG achieves a higher correlation with downstream RAG performance compared to baseline methods, with improvements in Kendall's $\tau$ correlation ranging from 0.168 to 0.494. Additionally, eRAG offers significant computational advantages, improving runtime and consuming up to 50 times less GPU memory than end-to-end evaluation.

Title: Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images

• Authors: Ali Naseh, Katherine Thai, Mohit Iyyer, Amir Houmansadr
• Subjects: cs.CR, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13784
• Pdf URL: https://arxiv.org/pdf/2404.13784
• Copy Paste: [[2404.13784]] Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images(https://arxiv.org/abs/2404.13784)
• Keywords: security, attack
• Abstract: With the digital imagery landscape rapidly evolving, image stocks and AI-generated image marketplaces have become central to visual media. Traditional stock images now exist alongside innovative platforms that trade in prompts for AI-generated visuals, driven by sophisticated APIs like DALL-E 3 and Midjourney. This paper studies the possibility of employing multi-modal models with enhanced visual understanding to mimic the outputs of these platforms, introducing an original attack strategy. Our method leverages fine-tuned CLIP models, a multi-label classifier, and the descriptive capabilities of GPT-4V to create prompts that generate images similar to those available in marketplaces and from premium stock image providers, yet at a markedly lower expense. In presenting this strategy, we aim to spotlight a new class of economic and security considerations within the realm of digital imagery. Our findings, supported by both automated metrics and human assessment, reveal that comparable visual content can be produced for a fraction of the prevailing market prices ($0.23 - $0.27 per image), emphasizing the need for awareness and strategic discussions about the integrity of digital media in an increasingly AI-integrated landscape. Our work also contributes to the field by assembling a dataset consisting of approximately 19 million prompt-image pairs generated by the popular Midjourney platform, which we plan to release publicly.

Title: How to Inverting the Leverage Score Distribution?

• Authors: Zhihang Li, Zhao Song, Weixin Wang, Junze Yin, Zheng Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13785
• Pdf URL: https://arxiv.org/pdf/2404.13785
• Copy Paste: [[2404.13785]] How to Inverting the Leverage Score Distribution?(https://arxiv.org/abs/2404.13785)
• Keywords: security
• Abstract: Leverage score is a fundamental problem in machine learning and theoretical computer science. It has extensive applications in regression analysis, randomized algorithms, and neural network inversion. Despite leverage scores are widely used as a tool, in this paper, we study a novel problem, namely the inverting leverage score problem. We analyze to invert the leverage score distributions back to recover model parameters. Specifically, given a leverage score $\sigma \in \mathbb{R}^n$, the matrix $A \in \mathbb{R}^{n \times d}$, and the vector $b \in \mathbb{R}^n$, we analyze the non-convex optimization problem of finding $x \in \mathbb{R}^d$ to minimize $\| \mathrm{diag}( \sigma ) - I_n \circ (A(x) (A(x)^\top A(x) )^{-1} A(x)^\top ) \|_F$, where $A(x):= S(x)^{-1} A \in \mathbb{R}^{n \times d} $, $S(x) := \mathrm{diag}(s(x)) \in \mathbb{R}^{n \times n}$ and $s(x) : = Ax - b \in \mathbb{R}^n$. Our theoretical studies include computing the gradient and Hessian, demonstrating that the Hessian matrix is positive definite and Lipschitz, and constructing first-order and second-order algorithms to solve this regression problem. Our work combines iterative shrinking and the induction hypothesis to ensure global convergence rates for the Newton method, as well as the properties of Lipschitz and strong convexity to guarantee the performance of gradient descent. This important study on inverting statistical leverage opens up numerous new applications in interpretation, data recovery, and security.

Title: Universal Fingerprint Generation: Controllable Diffusion Model with Multimodal Conditions

• Authors: Steven A. Grosz, Anil K. Jain
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13791
• Pdf URL: https://arxiv.org/pdf/2404.13791
• Copy Paste: [[2404.13791]] Universal Fingerprint Generation: Controllable Diffusion Model with Multimodal Conditions(https://arxiv.org/abs/2404.13791)
• Keywords: privacy, biometric, diffusion
• Abstract: The utilization of synthetic data for fingerprint recognition has garnered increased attention due to its potential to alleviate privacy concerns surrounding sensitive biometric data. However, current methods for generating fingerprints have limitations in creating impressions of the same finger with useful intra-class variations. To tackle this challenge, we present GenPrint, a framework to produce fingerprint images of various types while maintaining identity and offering humanly understandable control over different appearance factors such as fingerprint class, acquisition type, sensor device, and quality level. Unlike previous fingerprint generation approaches, GenPrint is not confined to replicating style characteristics from the training dataset alone: it enables the generation of novel styles from unseen devices without requiring additional fine-tuning. To accomplish these objectives, we developed GenPrint using latent diffusion models with multimodal conditions (text and image) for consistent generation of style and identity. Our experiments leverage a variety of publicly available datasets for training and evaluation. Results demonstrate the benefits of GenPrint in terms of identity preservation, explainable control, and universality of generated images. Importantly, the GenPrint-generated images yield comparable or even superior accuracy to models trained solely on real data and further enhances performance when augmenting the diversity of existing real fingerprint datasets.

Title: Lightweight Connective Detection Using Gradient Boosting

• Authors: Mustafa Erolcan Er, Murathan Kurfalı, Deniz Zeyrek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13793
• Pdf URL: https://arxiv.org/pdf/2404.13793
• Copy Paste: [[2404.13793]] Lightweight Connective Detection Using Gradient Boosting(https://arxiv.org/abs/2404.13793)
• Keywords: robust
• Abstract: In this work, we introduce a lightweight discourse connective detection system. Employing gradient boosting trained on straightforward, low-complexity features, this proposed approach sidesteps the computational demands of the current approaches that rely on deep neural networks. Considering its simplicity, our approach achieves competitive results while offering significant gains in terms of time even on CPU. Furthermore, the stable performance across two unrelated languages suggests the robustness of our system in the multilingual scenario. The model is designed to support the annotation of discourse relations, particularly in scenarios with limited resources, while minimizing performance loss.

Title: Enforcing Conditional Independence for Fair Representation Learning and Causal Image Generation

• Authors: Jensen Hwa, Qingyu Zhao, Aditya Lahiri, Adnan Masood, Babak Salimi, Ehsan Adeli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13798
• Pdf URL: https://arxiv.org/pdf/2404.13798
• Copy Paste: [[2404.13798]] Enforcing Conditional Independence for Fair Representation Learning and Causal Image Generation(https://arxiv.org/abs/2404.13798)
• Keywords: protect, fair, diffusion
• Abstract: Conditional independence (CI) constraints are critical for defining and evaluating fairness in machine learning, as well as for learning unconfounded or causal representations. Traditional methods for ensuring fairness either blindly learn invariant features with respect to a protected variable (e.g., race when classifying sex from face images) or enforce CI relative to the protected attribute only on the model output (e.g., the sex label). Neither of these methods are effective in enforcing CI in high-dimensional feature spaces. In this paper, we focus on a nascent approach characterizing the CI constraint in terms of two Jensen-Shannon divergence terms, and we extend it to high-dimensional feature spaces using a novel dynamic sampling strategy. In doing so, we introduce a new training paradigm that can be applied to any encoder architecture. We are able to enforce conditional independence of the diffusion autoencoder latent representation with respect to any protected attribute under the equalized odds constraint and show that this approach enables causal image generation with controllable latent spaces. Our experimental results demonstrate that our approach can achieve high accuracy on downstream tasks while upholding equality of odds.

Title: From LLM to NMT: Advancing Low-Resource Machine Translation with Claude

• Authors: Maxim Enis, Mark Hopkins
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13813
• Pdf URL: https://arxiv.org/pdf/2404.13813
• Copy Paste: [[2404.13813]] From LLM to NMT: Advancing Low-Resource Machine Translation with Claude(https://arxiv.org/abs/2404.13813)
• Keywords: large language model
• Abstract: We show that Claude 3 Opus, a large language model (LLM) released by Anthropic in March 2024, exhibits stronger machine translation competence than other LLMs. Though we find evidence of data contamination with Claude on FLORES-200, we curate new benchmarks that corroborate the effectiveness of Claude for low-resource machine translation into English. We find that Claude has remarkable \textit{resource efficiency} -- the degree to which the quality of the translation model depends on a language pair's resource level. Finally, we show that advancements in LLM translation can be compressed into traditional neural machine translation (NMT) models. Using Claude to generate synthetic data, we demonstrate that knowledge distillation advances the state-of-the-art in Yoruba-English translation, meeting or surpassing strong baselines like NLLB-54B and Google Translate.

Title: Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference

• Authors: Yujin Han, Difan Zou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13815
• Pdf URL: https://arxiv.org/pdf/2404.13815
• Copy Paste: [[2404.13815]] Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference(https://arxiv.org/abs/2404.13815)
• Keywords: robust
• Abstract: Standard empirical risk minimization (ERM) models may prioritize learning spurious correlations between spurious features and true labels, leading to poor accuracy on groups where these correlations do not hold. Mitigating this issue often requires expensive spurious attribute (group) labels or relies on trained ERM models to infer group labels when group information is unavailable. However, the significant performance gap in worst-group accuracy between using pseudo group labels and using oracle group labels inspires us to consider further improving group robustness through preciser group inference. Therefore, we propose GIC, a novel method that accurately infers group labels, resulting in improved worst-group performance. GIC trains a spurious attribute classifier based on two key properties of spurious correlations: (1) high correlation between spurious attributes and true labels, and (2) variability in this correlation between datasets with different group distributions. Empirical studies on multiple datasets demonstrate the effectiveness of GIC in inferring group labels, and combining GIC with various downstream invariant learning methods improves worst-group accuracy, showcasing its powerful flexibility. Additionally, through analyzing the misclassifications in GIC, we identify an interesting phenomenon called semantic consistency, which may contribute to better decoupling the association between spurious attributes and labels, thereby mitigating spurious correlation.

Title: HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild

• Authors: Supreeth Narasimhaswamy, Huy Anh Nguyen, Lihan Huang, Minh Hoai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13819
• Pdf URL: https://arxiv.org/pdf/2404.13819
• Copy Paste: [[2404.13819]] HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild(https://arxiv.org/abs/2404.13819)
• Keywords: transformer, segmentation
• Abstract: We address the challenging task of identifying, segmenting, and tracking hand-held objects, which is crucial for applications such as human action segmentation and performance evaluation. This task is particularly challenging due to heavy occlusion, rapid motion, and the transitory nature of objects being hand-held, where an object may be held, released, and subsequently picked up again. To tackle these challenges, we have developed a novel transformer-based architecture called HOIST-Former. HOIST-Former is adept at spatially and temporally segmenting hands and objects by iteratively pooling features from each other, ensuring that the processes of identification, segmentation, and tracking of hand-held objects depend on the hands' positions and their contextual appearance. We further refine HOIST-Former with a contact loss that focuses on areas where hands are in contact with objects. Moreover, we also contribute an in-the-wild video dataset called HOIST, which comprises 4,125 videos complete with bounding boxes, segmentation masks, and tracking IDs for hand-held objects. Through experiments on the HOIST dataset and two additional public datasets, we demonstrate the efficacy of HOIST-Former in segmenting and tracking hand-held objects.

Title: Swap It Like Its Hot: Segmentation-based spoof attacks on eye-tracking images

• Authors: Anish S. Narkar, Brendan David-John
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13827
• Pdf URL: https://arxiv.org/pdf/2404.13827
• Copy Paste: [[2404.13827]] Swap It Like Its Hot: Segmentation-based spoof attacks on eye-tracking images(https://arxiv.org/abs/2404.13827)
• Keywords: secure, defense, attack, biometric, segmentation
• Abstract: Video-based eye trackers capture the iris biometric and enable authentication to secure user identity. However, biometric authentication is susceptible to spoofing another user's identity through physical or digital manipulation. The current standard to identify physical spoofing attacks on eye-tracking sensors uses liveness detection. Liveness detection classifies gaze data as real or fake, which is sufficient to detect physical presentation attacks. However, such defenses cannot detect a spoofing attack when real eye image inputs are digitally manipulated to swap the iris pattern of another person. We propose IrisSwap as a novel attack on gaze-based liveness detection. IrisSwap allows attackers to segment and digitally swap in a victim's iris pattern to fool iris authentication. Both offline and online attacks produce gaze data that deceives the current state-of-the-art defense models at rates up to 58% and motivates the need to develop more advanced authentication methods for eye trackers.

Title: C2F-SemiCD: A Coarse-to-Fine Semi-Supervised Change Detection Method Based on Consistency Regularization in High-Resolution Remote Sensing Images

• Authors: Chengxi Han, Chen Wu, Meiqi Hu, Jiepan Li, Hongruixuan Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13838
• Pdf URL: https://arxiv.org/pdf/2404.13838
• Copy Paste: [[2404.13838]] C2F-SemiCD: A Coarse-to-Fine Semi-Supervised Change Detection Method Based on Consistency Regularization in High-Resolution Remote Sensing Images(https://arxiv.org/abs/2404.13838)
• Keywords: extraction
• Abstract: A high-precision feature extraction model is crucial for change detection (CD). In the past, many deep learning-based supervised CD methods learned to recognize change feature patterns from a large number of labelled bi-temporal images, whereas labelling bi-temporal remote sensing images is very expensive and often time-consuming; therefore, we propose a coarse-to-fine semi-supervised CD method based on consistency regularization (C2F-SemiCD), which includes a coarse-to-fine CD network with a multiscale attention mechanism (C2FNet) and a semi-supervised update method. Among them, the C2FNet network gradually completes the extraction of change features from coarse-grained to fine-grained through multiscale feature fusion, channel attention mechanism, spatial attention mechanism, global context module, feature refine module, initial aggregation module, and final aggregation module. The semi-supervised update method uses the mean teacher method. The parameters of the student model are updated to the parameters of the teacher Model by using the exponential moving average (EMA) method. Through extensive experiments on three datasets and meticulous ablation studies, including crossover experiments across datasets, we verify the significant effectiveness and efficiency of the proposed C2F-SemiCD method. The code will be open at: https://github.com/ChengxiHAN/C2F-SemiCDand-C2FNet.

Title: Fair Concurrent Training of Multiple Models in Federated Learning

• Authors: Marie Siew, Haoran Zhang, Jong-Ik Park, Yuezhou Liu, Yichen Ruan, Lili Su, Stratis Ioannidis, Edmund Yeh, Carlee Joe-Wong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13841
• Pdf URL: https://arxiv.org/pdf/2404.13841
• Copy Paste: [[2404.13841]] Fair Concurrent Training of Multiple Models in Federated Learning(https://arxiv.org/abs/2404.13841)
• Keywords: federate, fair
• Abstract: Federated learning (FL) enables collaborative learning across multiple clients. In most FL work, all clients train a single learning task. However, the recent proliferation of FL applications may increasingly require multiple FL tasks to be trained simultaneously, sharing clients' computing and communication resources, which we call Multiple-Model Federated Learning (MMFL). Current MMFL algorithms use naive average-based client-task allocation schemes that can lead to unfair performance when FL tasks have heterogeneous difficulty levels, e.g., tasks with larger models may need more rounds and data to train. Just as naively allocating resources to generic computing jobs with heterogeneous resource needs can lead to unfair outcomes, naive allocation of clients to FL tasks can lead to unfairness, with some tasks having excessively long training times, or lower converged accuracies. Furthermore, in the FL setting, since clients are typically not paid for their training effort, we face a further challenge that some clients may not even be willing to train some tasks, e.g., due to high computational costs, which may exacerbate unfairness in training outcomes across tasks. We address both challenges by firstly designing FedFairMMFL, a difficulty-aware algorithm that dynamically allocates clients to tasks in each training round. We provide guarantees on airness and FedFairMMFL's convergence rate. We then propose a novel auction design that incentivizes clients to train multiple tasks, so as to fairly distribute clients' training efforts across the tasks. We show how our fairness-based learning and incentive mechanisms impact training convergence and finally evaluate our algorithm with multiple sets of learning tasks on real world datasets.

Title: EventLens: Leveraging Event-Aware Pretraining and Cross-modal Linking Enhances Visual Commonsense Reasoning

• Authors: Mingjie Ma, Zhihuan Yu, Yichao Ma, Guohui Li
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13847
• Pdf URL: https://arxiv.org/pdf/2404.13847
• Copy Paste: [[2404.13847]] EventLens: Leveraging Event-Aware Pretraining and Cross-modal Linking Enhances Visual Commonsense Reasoning(https://arxiv.org/abs/2404.13847)
• Keywords: large language model
• Abstract: Visual Commonsense Reasoning (VCR) is a cognitive task, challenging models to answer visual questions requiring human commonsense, and to provide rationales explaining why the answers are correct. With emergence of Large Language Models (LLMs), it is natural and imperative to explore their applicability to VCR. However, VCR task demands more external knowledge to tackle its challenging questions, necessitating special designs to activate LLMs' commonsense reasoning abilities. Also, most existing Multimodal LLMs adopted an abstraction of entire input image, which makes it difficult to comprehend VCR's unique co-reference tags between image regions and text, posing challenges for fine-grained alignment. To address these issues, we propose EventLens that leverages Event-Aware Pretraining and Cross-modal Linking and EnhanceS VCR. First, by emulating the cognitive process of human reasoning, an Event-Aware Pretraining auxiliary task is introduced to better activate LLM's global comprehension of intricate scenarios. Second, during fine-tuning, we further utilize reference tags to bridge RoI features with texts, while preserving both modality semantics. Finally, we use instruct-style prompts to narrow the gap between pretraining and fine-tuning, and task-specific adapters to better integrate LLM's inherent knowledge with new commonsense. Experimental results show the effectiveness of our proposed auxiliary task and fine-grained linking strategy.

Title: ICST-DNET: An Interpretable Causal Spatio-Temporal Diffusion Network for Traffic Speed Prediction

• Authors: Yi Rong, Yingchi Mao, Yinqiu Liu, Ling Chen, Xiaoming He, Dusit Niyato
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.13853
• Pdf URL: https://arxiv.org/pdf/2404.13853
• Copy Paste: [[2404.13853]] ICST-DNET: An Interpretable Causal Spatio-Temporal Diffusion Network for Traffic Speed Prediction(https://arxiv.org/abs/2404.13853)
• Keywords: interpretability, diffusion
• Abstract: Traffic speed prediction is significant for intelligent navigation and congestion alleviation. However, making accurate predictions is challenging due to three factors: 1) traffic diffusion, i.e., the spatial and temporal causality existing between the traffic conditions of multiple neighboring roads, 2) the poor interpretability of traffic data with complicated spatio-temporal correlations, and 3) the latent pattern of traffic speed fluctuations over time, such as morning and evening rush. Jointly considering these factors, in this paper, we present a novel architecture for traffic speed prediction, called Interpretable Causal Spatio-Temporal Diffusion Network (ICST-DNET). Specifically, ICST-DENT consists of three parts, namely the Spatio-Temporal Causality Learning (STCL), Causal Graph Generation (CGG), and Speed Fluctuation Pattern Recognition (SFPR) modules. First, to model the traffic diffusion within road networks, an STCL module is proposed to capture both the temporal causality on each individual road and the spatial causality in each road pair. The CGG module is then developed based on STCL to enhance the interpretability of the traffic diffusion procedure from the temporal and spatial perspectives. Specifically, a time causality matrix is generated to explain the temporal causality between each road's historical and future traffic conditions. For spatial causality, we utilize causal graphs to visualize the diffusion process in road pairs. Finally, to adapt to traffic speed fluctuations in different scenarios, we design a personalized SFPR module to select the historical timesteps with strong influences for learning the pattern of traffic speed fluctuations. Extensive experimental results prove that ICST-DNET can outperform all existing baselines, as evidenced by the higher prediction accuracy, ability to explain causality, and adaptability to different scenarios.

Title: Understanding the role of FFNs in driving multilingual behaviour in LLMs

• Authors: Sunit Bhattacharya, Ondřej Bojar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13855
• Pdf URL: https://arxiv.org/pdf/2404.13855
• Copy Paste: [[2404.13855]] Understanding the role of FFNs in driving multilingual behaviour in LLMs(https://arxiv.org/abs/2404.13855)
• Keywords: large language model
• Abstract: Multilingualism in Large Language Models (LLMs) is an yet under-explored area. In this paper, we conduct an in-depth analysis of the multilingual capabilities of a family of a Large Language Model, examining its architecture, activation patterns, and processing mechanisms across languages. We introduce novel metrics to probe the model's multilingual behaviour at different layers and shed light on the impact of architectural choices on multilingual processing. Our findings reveal different patterns of multilinugal processing in the sublayers of Feed-Forward Networks of the models. Furthermore, we uncover the phenomenon of "over-layerization" in certain model configurations, where increasing layer depth without corresponding adjustments to other parameters may degrade model performance. Through comparisons within and across languages, we demonstrate the interplay between model architecture, layer depth, and multilingual processing capabilities of LLMs trained on multiple languages.

Title: Unveiling and Mitigating Generalized Biases of DNNs through the Intrinsic Dimensions of Perceptual Manifolds

• Authors: Yanbiao Ma, Licheng Jiao, Fang Liu, Lingling Li, Wenping Ma, Shuyuan Yang, Xu Liu, Puhua Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13859
• Pdf URL: https://arxiv.org/pdf/2404.13859
• Copy Paste: [[2404.13859]] Unveiling and Mitigating Generalized Biases of DNNs through the Intrinsic Dimensions of Perceptual Manifolds(https://arxiv.org/abs/2404.13859)
• Keywords: fair
• Abstract: Building fair deep neural networks (DNNs) is a crucial step towards achieving trustworthy artificial intelligence. Delving into deeper factors that affect the fairness of DNNs is paramount and serves as the foundation for mitigating model biases. However, current methods are limited in accurately predicting DNN biases, relying solely on the number of training samples and lacking more precise measurement tools. Here, we establish a geometric perspective for analyzing the fairness of DNNs, comprehensively exploring how DNNs internally shape the intrinsic geometric characteristics of datasets-the intrinsic dimensions (IDs) of perceptual manifolds, and the impact of IDs on the fairness of DNNs. Based on multiple findings, we propose Intrinsic Dimension Regularization (IDR), which enhances the fairness and performance of models by promoting the learning of concise and ID-balanced class perceptual manifolds. In various image recognition benchmark tests, IDR significantly mitigates model bias while improving its performance.

Title: Distributional Black-Box Model Inversion Attack with Multi-Agent Reinforcement Learning

• Authors: Huan Bao, Kaimin Wei, Yongdong Wu, Jin Qian, Robert H. Deng
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.13860
• Pdf URL: https://arxiv.org/pdf/2404.13860
• Copy Paste: [[2404.13860]] Distributional Black-Box Model Inversion Attack with Multi-Agent Reinforcement Learning(https://arxiv.org/abs/2404.13860)
• Keywords: privacy, attack, generative
• Abstract: A Model Inversion (MI) attack based on Generative Adversarial Networks (GAN) aims to recover the private training data from complex deep learning models by searching codes in the latent space. However, they merely search a deterministic latent space such that the found latent code is usually suboptimal. In addition, the existing distributional MI schemes assume that an attacker can access the structures and parameters of the target model, which is not always viable in practice. To overcome the above shortcomings, this paper proposes a novel Distributional Black-Box Model Inversion (DBB-MI) attack by constructing the probabilistic latent space for searching the target privacy data. Specifically, DBB-MI does not need the target model parameters or specialized GAN training. Instead, it finds the latent probability distribution by combining the output of the target model with multi-agent reinforcement learning techniques. Then, it randomly chooses latent codes from the latent probability distribution for recovering the private data. As the latent probability distribution closely aligns with the target privacy data in latent space, the recovered data will leak the privacy of training samples of the target model significantly. Abundant experiments conducted on diverse datasets and networks show that the present DBB-MI has better performance than state-of-the-art in attack accuracy, K-nearest neighbor feature distance, and Peak Signal-to-Noise Ratio.

Title: PM-VIS: High-Performance Box-Supervised Video Instance Segmentation

• Authors: Zhangjing Yang, Dun Liu, Wensheng Cheng, Jinqiao Wang, Yi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13863
• Pdf URL: https://arxiv.org/pdf/2404.13863
• Copy Paste: [[2404.13863]] PM-VIS: High-Performance Box-Supervised Video Instance Segmentation(https://arxiv.org/abs/2404.13863)
• Keywords: segmentation
• Abstract: Labeling pixel-wise object masks in videos is a resource-intensive and laborious process. Box-supervised Video Instance Segmentation (VIS) methods have emerged as a viable solution to mitigate the labor-intensive annotation process. . In practical applications, the two-step approach is not only more flexible but also exhibits a higher recognition accuracy. Inspired by the recent success of Segment Anything Model (SAM), we introduce a novel approach that aims at harnessing instance box annotations from multiple perspectives to generate high-quality instance pseudo masks, thus enriching the information contained in instance annotations. We leverage ground-truth boxes to create three types of pseudo masks using the HQ-SAM model, the box-supervised VIS model (IDOL-BoxInst), and the VOS model (DeAOT) separately, along with three corresponding optimization mechanisms. Additionally, we introduce two ground-truth data filtering methods, assisted by high-quality pseudo masks, to further enhance the training dataset quality and improve the performance of fully supervised VIS methods. To fully capitalize on the obtained high-quality Pseudo Masks, we introduce a novel algorithm, PM-VIS, to integrate mask losses into IDOL-BoxInst. Our PM-VIS model, trained with high-quality pseudo mask annotations, demonstrates strong ability in instance mask prediction, achieving state-of-the-art performance on the YouTube-VIS 2019, YouTube-VIS 2021, and OVIS validation sets, notably narrowing the gap between box-supervised and fully supervised VIS methods.

Title: Context-Enhanced Language Models for Generating Multi-Paper Citations

• Authors: Avinash Anand, Kritarth Prasad, Ujjwal Goel, Mohit Gupta, Naman Lal, Astha Verma, Rajiv Ratn Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13865
• Pdf URL: https://arxiv.org/pdf/2404.13865
• Copy Paste: [[2404.13865]] Context-Enhanced Language Models for Generating Multi-Paper Citations(https://arxiv.org/abs/2404.13865)
• Keywords: large language model
• Abstract: Citation text plays a pivotal role in elucidating the connection between scientific documents, demanding an in-depth comprehension of the cited paper. Constructing citations is often time-consuming, requiring researchers to delve into extensive literature and grapple with articulating relevant content. To address this challenge, the field of citation text generation (CTG) has emerged. However, while earlier methods have primarily centered on creating single-sentence citations, practical scenarios frequently necessitate citing multiple papers within a single paragraph. To bridge this gap, we propose a method that leverages Large Language Models (LLMs) to generate multi-citation sentences. Our approach involves a single source paper and a collection of target papers, culminating in a coherent paragraph containing multi-sentence citation text. Furthermore, we introduce a curated dataset named MCG-S2ORC, composed of English-language academic research papers in Computer Science, showcasing multiple citation instances. In our experiments, we evaluate three LLMs LLaMA, Alpaca, and Vicuna to ascertain the most effective model for this endeavor. Additionally, we exhibit enhanced performance by integrating knowledge graphs from target papers into the prompts for generating citation text. This research underscores the potential of harnessing LLMs for citation generation, opening a compelling avenue for exploring the intricate connections between scientific documents.

Title: Texture-aware and Shape-guided Transformer for Sequential DeepFake Detection

• Authors: Yunfei Li, Jiaran Zhou, Xin Wang, Junyu Dong, Yuezun Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13873
• Pdf URL: https://arxiv.org/pdf/2404.13873
• Copy Paste: [[2404.13873]] Texture-aware and Shape-guided Transformer for Sequential DeepFake Detection(https://arxiv.org/abs/2404.13873)
• Keywords: transformer
• Abstract: Sequential DeepFake detection is an emerging task that aims to predict the manipulation sequence in order. Existing methods typically formulate it as an image-to-sequence problem, employing conventional Transformer architectures for detection. However, these methods lack dedicated design and consequently result in limited performance. In this paper, we propose a novel Texture-aware and Shape-guided Transformer to enhance detection performance. Our method features four major improvements. Firstly, we describe a texture-aware branch that effectively captures subtle manipulation traces with the Diversiform Pixel Difference Attention module. Then we introduce a Bidirectional Interaction Cross-attention module that seeks deep correlations among spatial and sequential features, enabling effective modeling of complex manipulation traces. To further enhance the cross-attention, we describe a Shape-guided Gaussian mapping strategy, providing initial priors of the manipulation shape. Finally, observing that the latter manipulation in a sequence may influence traces left in the earlier one, we intriguingly invert the prediction order from forward to backward, leading to notable gains as expected. Extensive experimental results demonstrate that our method outperforms others by a large margin, highlighting the superiority of our method.

Title: VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models

• Authors: Haoyi Qiu, Wenbo Hu, Zi-Yi Dou, Nanyun Peng
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13874
• Pdf URL: https://arxiv.org/pdf/2404.13874
• Copy Paste: [[2404.13874]] VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models(https://arxiv.org/abs/2404.13874)
• Keywords: large language model
• Abstract: Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs, undermining their reliability. A comprehensive quantitative evaluation is necessary to identify and understand the extent of hallucinations in these models. However, existing benchmarks are often limited in scope, focusing mainly on object hallucinations. Furthermore, current evaluation methods struggle to effectively address the subtle semantic distinctions between model outputs and reference data, as well as the balance between hallucination and informativeness. To address these issues, we introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases. Moreover, we propose an large language model (LLM)-based two-stage evaluation framework that generalizes the popular CHAIR metric and incorporates both faithfulness and coverage into the evaluation. Experiments on 10 established LVLMs demonstrate that our evaluation metric is more comprehensive and better correlated with humans than existing work when evaluating on our challenging human annotated benchmark dataset. Our work also highlights the critical balance between faithfulness and coverage of model outputs, and encourages future works to address hallucinations in LVLMs while keeping their outputs informative.

Title: Explicit Lipschitz Value Estimation Enhances Policy Robustness Against Perturbation

• Authors: Xulin Chen, Ruipeng Liu, Garrett E. Katz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13879
• Pdf URL: https://arxiv.org/pdf/2404.13879
• Copy Paste: [[2404.13879]] Explicit Lipschitz Value Estimation Enhances Policy Robustness Against Perturbation(https://arxiv.org/abs/2404.13879)
• Keywords: robust
• Abstract: In robotic control tasks, policies trained by reinforcement learning (RL) in simulation often experience a performance drop when deployed on physical hardware, due to modeling error, measurement error, and unpredictable perturbations in the real world. Robust RL methods account for this issue by approximating a worst-case value function during training, but they can be sensitive to approximation errors in the value function and its gradient before training is complete. In this paper, we hypothesize that Lipschitz regularization can help condition the approximated value function gradients, leading to improved robustness after training. We test this hypothesis by combining Lipschitz regularization with an application of Fast Gradient Sign Method to reduce approximation errors when evaluating the value function under adversarial perturbations. Our empirical results demonstrate the benefits of this approach over prior work on a number of continuous control benchmarks.

Title: Regional Style and Color Transfer

• Authors: Zhicheng Ding, Panfeng Li, Qikai Yang, Xinyu Shen, Siyang Li, Qingtian Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13880
• Pdf URL: https://arxiv.org/pdf/2404.13880
• Copy Paste: [[2404.13880]] Regional Style and Color Transfer(https://arxiv.org/abs/2404.13880)
• Keywords: segmentation
• Abstract: This paper presents a novel contribution to the field of regional style transfer. Existing methods often suffer from the drawback of applying style homogeneously across the entire image, leading to stylistic inconsistencies or foreground object twisted when applied to image with foreground elements such as person figures. To address this limitation, we propose a new approach that leverages a segmentation network to precisely isolate foreground objects within the input image. Subsequently, style transfer is applied exclusively to the background region. The isolated foreground objects are then carefully reintegrated into the style-transferred background. To enhance the visual coherence between foreground and background, a color transfer step is employed on the foreground elements prior to their rein-corporation. Finally, we utilize feathering techniques to achieve a seamless amalgamation of foreground and background, resulting in a visually unified and aesthetically pleasing final composition. Extensive evaluations demonstrate that our proposed approach yields significantly more natural stylistic transformations compared to conventional methods.

Title: Towards Better Text-to-Image Generation Alignment via Attention Modulation

• Authors: Yihang Wu, Xiao Cao, Kaixin Li, Zitan Chen, Haonan Wang, Lei Meng, Zhiyong Huang
• Subjects: cs.CL, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.13899
• Pdf URL: https://arxiv.org/pdf/2404.13899
• Copy Paste: [[2404.13899]] Towards Better Text-to-Image Generation Alignment via Attention Modulation(https://arxiv.org/abs/2404.13899)
• Keywords: diffusion
• Abstract: In text-to-image generation tasks, the advancements of diffusion models have facilitated the fidelity of generated results. However, these models encounter challenges when processing text prompts containing multiple entities and attributes. The uneven distribution of attention results in the issues of entity leakage and attribute misalignment. Training from scratch to address this issue requires numerous labeled data and is resource-consuming. Motivated by this, we propose an attribution-focusing mechanism, a training-free phase-wise mechanism by modulation of attention for diffusion model. One of our core ideas is to guide the model to concentrate on the corresponding syntactic components of the prompt at distinct timesteps. To achieve this, we incorporate a temperature control mechanism within the early phases of the self-attention modules to mitigate entity leakage issues. An object-focused masking scheme and a phase-wise dynamic weight control mechanism are integrated into the cross-attention modules, enabling the model to discern the affiliation of semantic information between entities more effectively. The experimental results in various alignment scenarios demonstrate that our model attain better image-text alignment with minimal additional computational cost.

Title: Accelerating Image Generation with Sub-path Linear Approximation Model

• Authors: Chen Xu, Tianhui Song, Weixin Feng, Xubin Li, Tiezheng Ge, Bo Zheng, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13903
• Pdf URL: https://arxiv.org/pdf/2404.13903
• Copy Paste: [[2404.13903]] Accelerating Image Generation with Sub-path Linear Approximation Model(https://arxiv.org/abs/2404.13903)
• Keywords: diffusion, generative
• Abstract: Diffusion models have significantly advanced the state of the art in image, audio, and video generation tasks. However, their applications in practical scenarios are hindered by slow inference speed. Drawing inspiration from the approximation strategies utilized in consistency models, we propose the Sub-path Linear Approximation Model (SLAM), which accelerates diffusion models while maintaining high-quality image generation. SLAM treats the PF-ODE trajectory as a series of PF-ODE sub-paths divided by sampled points, and harnesses sub-path linear (SL) ODEs to form a progressive and continuous error estimation along each individual PF-ODE sub-path. The optimization on such SL-ODEs allows SLAM to construct denoising mappings with smaller cumulative approximated errors. An efficient distillation method is also developed to facilitate the incorporation of more advanced diffusion models, such as latent diffusion models. Our extensive experimental results demonstrate that SLAM achieves an efficient training regimen, requiring only 6 A100 GPU days to produce a high-quality generative model capable of 2 to 4-step generation with high performance. Comprehensive evaluations on LAION, MS COCO 2014, and MS COCO 2017 datasets also illustrate that SLAM surpasses existing acceleration methods in few-step generation tasks, achieving state-of-the-art performance both on FID and the quality of the generated images.

Title: Generating Attractive and Authentic Copywriting from Customer Reviews

• Authors: Yu-Xiang Lin, Wei-Yun Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13906
• Pdf URL: https://arxiv.org/pdf/2404.13906
• Copy Paste: [[2404.13906]] Generating Attractive and Authentic Copywriting from Customer Reviews(https://arxiv.org/abs/2404.13906)
• Keywords: large language model
• Abstract: The goal of product copywriting is to capture the interest of potential buyers by emphasizing the features of products through text descriptions. As e-commerce platforms offer a wide range of services, it's becoming essential to dynamically adjust the styles of these auto-generated descriptions. Typical approaches to copywriting generation often rely solely on specified product attributes, which may result in dull and repetitive content. To tackle this issue, we propose to generate copywriting based on customer reviews, as they provide firsthand practical experiences with products, offering a richer source of information than just product attributes. We have developed a sequence-to-sequence framework, enhanced with reinforcement learning, to produce copywriting that is attractive, authentic, and rich in information. Our framework outperforms all existing baseline and zero-shot large language models, including LLaMA-2-chat-7B and GPT-3.5, in terms of both attractiveness and faithfulness. Furthermore, this work features the use of LLMs for aspect-based summaries collection and argument allure assessment. Experiments demonstrate the effectiveness of using LLMs for marketing domain corpus construction. The code and the dataset is publicly available at: https://github.com/YuXiangLin1234/Copywriting-Generation.

Title: Integrated Gradient Correlation: a Dataset-wise Attribution Method

• Authors: Pierre Lelièvre, Chien-Chung Chen (National Taiwan University)
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.13910
• Pdf URL: https://arxiv.org/pdf/2404.13910
• Copy Paste: [[2404.13910]] Integrated Gradient Correlation: a Dataset-wise Attribution Method(https://arxiv.org/abs/2404.13910)
• Keywords: interpretability
• Abstract: Attribution methods are primarily designed to study the distribution of input component contributions to individual model predictions. However, some research applications require a summary of attribution patterns across the entire dataset to facilitate the interpretability of the scrutinized models. In this paper, we present a new method called Integrated Gradient Correlation (IGC) that relates dataset-wise attributions to a model prediction score and enables region-specific analysis by a direct summation over associated components. We demonstrate our method on scalar predictions with the study of image feature representation in the brain from fMRI neural signals and the estimation of neural population receptive fields (NSD dataset), as well as on categorical predictions with the investigation of handwritten digit recognition (MNIST dataset). The resulting IGC attributions show selective patterns, revealing underlying model strategies coherent with their respective objectives.

Title: Navigating the Path of Writing: Outline-guided Text Generation with Large Language Models

• Authors: Yukyung Lee, Soonwon Ka, Bokyung Son, Pilsung Kang, Jaewook Kang
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.13919
• Pdf URL: https://arxiv.org/pdf/2404.13919
• Copy Paste: [[2404.13919]] Navigating the Path of Writing: Outline-guided Text Generation with Large Language Models(https://arxiv.org/abs/2404.13919)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have significantly impacted the writing process, enabling collaborative content creation and enhancing productivity. However, generating high-quality, user-aligned text remains challenging. In this paper, we propose Writing Path, a framework that uses explicit outlines to guide LLMs in generating goal-oriented, high-quality pieces of writing. Our approach draws inspiration from structured writing planning and reasoning paths, focusing on capturing and reflecting user intentions throughout the writing process. We construct a diverse dataset from unstructured blog posts to benchmark writing performance and introduce a comprehensive evaluation framework assessing the quality of outlines and generated texts. Our evaluations with GPT-3.5-turbo, GPT-4, and HyperCLOVA X demonstrate that the Writing Path approach significantly enhances text quality according to both LLMs and human evaluations. This study highlights the potential of integrating writing-specific techniques into LLMs to enhance their ability to meet the diverse writing needs of users.

Title: MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets

• Authors: Zeyu Li, Ruitong Gan, Chuanchen Luo, Yuxi Wang, Jiaheng Liu, Ziwei Zhu Man Zhang, Qing Li, Xucheng Yin, Zhaoxiang Zhang, Junran Peng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13923
• Pdf URL: https://arxiv.org/pdf/2404.13923
• Copy Paste: [[2404.13923]] MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets(https://arxiv.org/abs/2404.13923)
• Keywords: diffusion, generative
• Abstract: Driven by powerful image diffusion models, recent research has achieved the automatic creation of 3D objects from textual or visual guidance. By performing score distillation sampling (SDS) iteratively across different views, these methods succeed in lifting 2D generative prior to the 3D space. However, such a 2D generative image prior bakes the effect of illumination and shadow into the texture. As a result, material maps optimized by SDS inevitably involve spurious correlated components. The absence of precise material definition makes it infeasible to relight the generated assets reasonably in novel scenes, which limits their application in downstream scenarios. In contrast, humans can effortlessly circumvent this ambiguity by deducing the material of the object from its appearance and semantics. Motivated by this insight, we propose MaterialSeg3D, a 3D asset material generation framework to infer underlying material from the 2D semantic prior. Based on such a prior model, we devise a mechanism to parse material in 3D space. We maintain a UV stack, each map of which is unprojected from a specific viewpoint. After traversing all viewpoints, we fuse the stack through a weighted voting scheme and then employ region unification to ensure the coherence of the object parts. To fuel the learning of semantics prior, we collect a material dataset, named Materialized Individual Objects (MIO), which features abundant images, diverse categories, and accurate annotations. Extensive quantitative and qualitative experiments demonstrate the effectiveness of our method.

Title: MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkit

• Authors: Boning Zhang, Chengxi Li, Kai Fan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13925
• Pdf URL: https://arxiv.org/pdf/2404.13925
• Copy Paste: [[2404.13925]] MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkit(https://arxiv.org/abs/2404.13925)
• Keywords: robust, fair, large language model
• Abstract: Large language models (LLMs) have been explored in a variety of reasoning tasks including solving of mathematical problems. Each math dataset typically includes its own specially designed evaluation script, which, while suitable for its intended use, lacks generalizability across different datasets. Consequently, updates and adaptations to these evaluation tools tend to occur without being systematically reported, leading to inconsistencies and obstacles to fair comparison across studies. To bridge this gap, we introduce a comprehensive mathematical evaluation toolkit that not only utilizes a python computer algebra system (CAS) for its numerical accuracy, but also integrates an optional LLM, known for its considerable natural language processing capabilities. To validate the effectiveness of our toolkit, we manually annotated two distinct datasets. Our experiments demonstrate that the toolkit yields more robust evaluation results compared to prior works, even without an LLM. Furthermore, when an LLM is incorporated, there is a notable enhancement. The code for our method will be made available at \url{https://github.com/MARIO-Math-Reasoning/math_evaluation}.

Title: A User-Centric Benchmark for Evaluating Large Language Models

• Authors: Jiayin Wang, Fengran Mo, Weizhi Ma, Peijie Sun, Min Zhang, Jian-Yun Nie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13940
• Pdf URL: https://arxiv.org/pdf/2404.13940
• Copy Paste: [[2404.13940]] A User-Centric Benchmark for Evaluating Large Language Models(https://arxiv.org/abs/2404.13940)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are essential tools to collaborate with users on different tasks. Evaluating their performance to serve users' needs in real-world scenarios is important. While many benchmarks have been created, they mainly focus on specific predefined model abilities. Few have covered the intended utilization of LLMs by real users. To address this oversight, we propose benchmarking LLMs from a user perspective in both dataset construction and evaluation designs. We first collect 1863 real-world use cases with 15 LLMs from a user study with 712 participants from 23 countries. These self-reported cases form the User Reported Scenarios(URS) dataset with a categorization of 7 user intents. Secondly, on this authentic multi-cultural dataset, we benchmark 10 LLM services on their efficacy in satisfying user needs. Thirdly, we show that our benchmark scores align well with user-reported experience in LLM interactions across diverse intents, both of which emphasize the overlook of subjective scenarios. In conclusion, our study proposes to benchmark LLMs from a user-centric perspective, aiming to facilitate evaluations that better reflect real user needs. The benchmark dataset and code are available at https://github.com/Alice1998/URS.

Title: Gorgeous: Create Your Desired Character Facial Makeup from Any Ideas

• Authors: Jia Wei Sii, Chee Seng Chan
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.13944
• Pdf URL: https://arxiv.org/pdf/2404.13944
• Copy Paste: [[2404.13944]] Gorgeous: Create Your Desired Character Facial Makeup from Any Ideas(https://arxiv.org/abs/2404.13944)
• Keywords: diffusion
• Abstract: Contemporary makeup transfer methods primarily focus on replicating makeup from one face to another, considerably limiting their use in creating diverse and creative character makeup essential for visual storytelling. Such methods typically fail to address the need for uniqueness and contextual relevance, specifically aligning with character and story settings as they depend heavily on existing facial makeup in reference images. This approach also presents a significant challenge when attempting to source a perfectly matched facial makeup style, further complicating the creation of makeup designs inspired by various story elements, such as theme, background, and props that do not necessarily feature faces. To address these limitations, we introduce $Gorgeous$, a novel diffusion-based makeup application method that goes beyond simple transfer by innovatively crafting unique and thematic facial makeup. Unlike traditional methods, $Gorgeous$ does not require the presence of a face in the reference images. Instead, it draws artistic inspiration from a minimal set of three to five images, which can be of any type, and transforms these elements into practical makeup applications directly on the face. Our comprehensive experiments demonstrate that $Gorgeous$ can effectively generate distinctive character facial makeup inspired by the chosen thematic reference images. This approach opens up new possibilities for integrating broader story elements into character makeup, thereby enhancing the narrative depth and visual impact in storytelling.

Title: Dual Model Replacement:invisible Multi-target Backdoor Attack based on Federal Learning

• Authors: Rong Wang, Guichen Zhou, Mingjun Gao, Yunpeng Xiao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13946
• Pdf URL: https://arxiv.org/pdf/2404.13946
• Copy Paste: [[2404.13946]] Dual Model Replacement:invisible Multi-target Backdoor Attack based on Federal Learning(https://arxiv.org/abs/2404.13946)
• Keywords: security, attack, robust, federate
• Abstract: In recent years, the neural network backdoor hidden in the parameters of the federated learning model has been proved to have great security risks. Considering the characteristics of trigger generation, data poisoning and model training in backdoor attack, this paper designs a backdoor attack method based on federated learning. Firstly, aiming at the concealment of the backdoor trigger, a TrojanGan steganography model with encoder-decoder structure is designed. The model can encode specific attack information as invisible noise and attach it to the image as a backdoor trigger, which improves the concealment and data transformations of the backdoor trigger.Secondly, aiming at the problem of single backdoor trigger mode, an image poisoning attack method called combination trigger attack is proposed. This method realizes multi-backdoor triggering by multiplexing combined triggers and improves the robustness of backdoor attacks. Finally, aiming at the problem that the local training mechanism leads to the decrease of the success rate of backdoor attack, a dual model replacement backdoor attack algorithm based on federated learning is designed. This method can improve the success rate of backdoor attack while maintaining the performance of the federated learning aggregation model. Experiments show that the attack strategy in this paper can not only achieve high backdoor concealment and diversification of trigger forms under federated learning, but also achieve good attack success rate in multi-target attacks.door concealment and diversification of trigger forms but also achieve good results in multi-target attacks.

Title: Boter: Bootstrapping Knowledge Selection and Question Answering for Knowledge-based VQA

• Authors: Dongze Hao, Qunbo Wang, Longteng Guo, Jie Jiang, Jing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13947
• Pdf URL: https://arxiv.org/pdf/2404.13947
• Copy Paste: [[2404.13947]] Boter: Bootstrapping Knowledge Selection and Question Answering for Knowledge-based VQA(https://arxiv.org/abs/2404.13947)
• Keywords: robust, large language model
• Abstract: Knowledge-based Visual Question Answering (VQA) requires models to incorporate external knowledge to respond to questions about visual content. Previous methods mostly follow the "retrieve and generate" paradigm. Initially, they utilize a pre-trained retriever to fetch relevant knowledge documents, subsequently employing them to generate answers. While these methods have demonstrated commendable performance in the task, they possess limitations: (1) they employ an independent retriever to acquire knowledge solely based on the similarity between the query and knowledge embeddings, without assessing whether the knowledge document is truly conducive to helping answer the question; (2) they convert the image into text and then conduct retrieval and answering in natural language space, which may not ensure comprehensive acquisition of all image information. To address these limitations, we propose Boter, a novel framework designed to bootstrap knowledge selection and question answering by leveraging the robust multimodal perception capabilities of the Multimodal Large Language Model (MLLM). The framework consists of two modules: Selector and Answerer, where both are initialized by the MLLM and parameter-efficiently finetuned in a simple cycle: find key knowledge in the retrieved knowledge documents using the Selector, and then use them to finetune the Answerer to predict answers; obtain the pseudo-labels of key knowledge documents based on the predictions of the Answerer and weak supervision labels, and then finetune the Selector to select key knowledge; repeat. Our framework significantly enhances the performance of the baseline on the challenging open-domain Knowledge-based VQA benchmark, OK-VQA, achieving a state-of-the-art accuracy of 62.83%.

Title: Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations

• Authors: Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, Jong C. Park
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13948
• Pdf URL: https://arxiv.org/pdf/2404.13948
• Copy Paste: [[2404.13948]] Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations(https://arxiv.org/abs/2404.13948)
• Keywords: attack, robust, large language model
• Abstract: The robustness of recent Large Language Models (LLMs) has become increasingly crucial as their applicability expands across various domains and real-world applications. Retrieval-Augmented Generation (RAG) is a promising solution for addressing the limitations of LLMs, yet existing studies on the robustness of RAG often overlook the interconnected relationships between RAG components or the potential threats prevalent in real-world databases, such as minor textual errors. In this work, we investigate two underexplored aspects when assessing the robustness of RAG: 1) vulnerability to noisy documents through low-level perturbations and 2) a holistic evaluation of RAG robustness. Furthermore, we introduce a novel attack method, the Genetic Attack on RAG (\textit{GARAG}), which targets these aspects. Specifically, GARAG is designed to reveal vulnerabilities within each component and test the overall system functionality against noisy documents. We validate RAG robustness by applying our \textit{GARAG} to standard QA datasets, incorporating diverse retrievers and LLMs. The experimental results show that GARAG consistently achieves high attack success rates. Also, it significantly devastates the performance of each component and their synergy, highlighting the substantial risk that minor textual inaccuracies pose in disrupting RAG systems in the real world.

Title: PeLiCal: Targetless Extrinsic Calibration via Penetrating Lines for RGB-D Cameras with Limited Co-visibility

• Authors: Jaeho Shin, Seungsang Yun, Ayoung Kim
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.13949
• Pdf URL: https://arxiv.org/pdf/2404.13949
• Copy Paste: [[2404.13949]] PeLiCal: Targetless Extrinsic Calibration via Penetrating Lines for RGB-D Cameras with Limited Co-visibility(https://arxiv.org/abs/2404.13949)
• Keywords: robust
• Abstract: RGB-D cameras are crucial in robotic perception, given their ability to produce images augmented with depth data. However, their limited FOV often requires multiple cameras to cover a broader area. In multi-camera RGB-D setups, the goal is typically to reduce camera overlap, optimizing spatial coverage with as few cameras as possible. The extrinsic calibration of these systems introduces additional complexities. Existing methods for extrinsic calibration either necessitate specific tools or highly depend on the accuracy of camera motion estimation. To address these issues, we present PeLiCal, a novel line-based calibration approach for RGB-D camera systems exhibiting limited overlap. Our method leverages long line features from surroundings, and filters out outliers with a novel convergence voting algorithm, achieving targetless, real-time, and outlier-robust performance compared to existing methods. We open source our implementation on \url{https://github.com/joomeok/PeLiCal.git}.

Title: 360VOTS: Visual Object Tracking and Segmentation in Omnidirectional Videos

• Authors: Yinzhe Xu, Huajian Huang, Yingshu Chen, Sai-Kit Yeung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13953
• Pdf URL: https://arxiv.org/pdf/2404.13953
• Copy Paste: [[2404.13953]] 360VOTS: Visual Object Tracking and Segmentation in Omnidirectional Videos(https://arxiv.org/abs/2404.13953)
• Keywords: segmentation
• Abstract: Visual object tracking and segmentation in omnidirectional videos are challenging due to the wide field-of-view and large spherical distortion brought by 360{\deg} images. To alleviate these problems, we introduce a novel representation, extended bounding field-of-view (eBFoV), for target localization and use it as the foundation of a general 360 tracking framework which is applicable for both omnidirectional visual object tracking and segmentation tasks. Building upon our previous work on omnidirectional visual object tracking (360VOT), we propose a comprehensive dataset and benchmark that incorporates a new component called omnidirectional video object segmentation (360VOS). The 360VOS dataset includes 290 sequences accompanied by dense pixel-wise masks and covers a broader range of target categories. To support both the development and evaluation of algorithms in this domain, we divide the dataset into a training subset with 170 sequences and a testing subset with 120 sequences. Furthermore, we tailor evaluation metrics for both omnidirectional tracking and segmentation to ensure rigorous assessment. Through extensive experiments, we benchmark state-of-the-art approaches and demonstrate the effectiveness of our proposed 360 tracking framework and training dataset. Homepage: https://360vots.hkustvgd.com/

Title: How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO

• Authors: Man Tik Ng, Hui Tung Tse, Jen-tse Huang, Jingjing Li, Wenxuan Wang, Michael R. Lyu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13957
• Pdf URL: https://arxiv.org/pdf/2404.13957
• Copy Paste: [[2404.13957]] How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO(https://arxiv.org/abs/2404.13957)
• Keywords: large language model
• Abstract: The role-play ability of Large Language Models (LLMs) has emerged as a popular research direction. However, existing studies focus on imitating well-known public figures or fictional characters, overlooking the potential for simulating ordinary individuals. Such an oversight limits the potential for advancements in digital human clones and non-player characters in video games. To bridge this gap, we introduce ECHO, an evaluative framework inspired by the Turing test. This framework engages the acquaintances of the target individuals to distinguish between human and machine-generated responses. Notably, our framework focuses on emulating average individuals rather than historical or fictional figures, presenting a unique advantage to apply the Turing Test. We evaluated three role-playing LLMs using ECHO, with GPT-3.5 and GPT-4 serving as foundational models, alongside the online application GPTs from OpenAI. Our results demonstrate that GPT-4 more effectively deceives human evaluators, and GPTs achieves a leading success rate of 48.3%. Furthermore, we investigated whether LLMs could discern between human-generated and machine-generated texts. While GPT-4 can identify differences, it could not determine which texts were human-produced. Our code and results of reproducing the role-playing LLMs are made publicly available via https://github.com/CUHK-ARISE/ECHO.

Title: An Economic Solution to Copyright Challenges of Generative AI

• Authors: iachen T. Wang, Zhun Deng, Hiroaki Chiba-Okabe, Boaz Barak, Weijie J. Su
• Subjects: cs.LG, econ.GN, stat.ME
• Abstract URL: https://arxiv.org/abs/2404.13964
• Pdf URL: https://arxiv.org/pdf/2404.13964
• Copy Paste: [[2404.13964]] An Economic Solution to Copyright Challenges of Generative AI(https://arxiv.org/abs/2404.13964)
• Keywords: fair, generative
• Abstract: Generative artificial intelligence (AI) systems are trained on large data corpora to generate new pieces of text, images, videos, and other media. There is growing concern that such systems may infringe on the copyright interests of training data contributors. To address the copyright challenges of generative AI, we propose a framework that compensates copyright owners proportionally to their contributions to the creation of AI-generated content. The metric for contributions is quantitatively determined by leveraging the probabilistic nature of modern generative AI models and using techniques from cooperative game theory in economics. This framework enables a platform where AI developers benefit from access to high-quality training data, thus improving model performance. Meanwhile, copyright owners receive fair compensation, driving the continued provision of relevant data for generative model training. Experiments demonstrate that our framework successfully identifies the most relevant data sources used in artwork generation, ensuring a fair and interpretable distribution of revenues among copyright owners.

Title: Protecting Your LLMs with Information Bottleneck

• Authors: Zichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang, Lei Song, Tianchun Wang, Chunlin Chen, Wei Cheng, Jiang Bian
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.13968
• Pdf URL: https://arxiv.org/pdf/2404.13968
• Copy Paste: [[2404.13968]] Protecting Your LLMs with Information Bottleneck(https://arxiv.org/abs/2404.13968)
• Keywords: security, protect, defense, attack, large language model
• Abstract: The advent of large language models (LLMs) has revolutionized the field of natural language processing, yet they might be attacked to produce harmful content. Despite efforts to ethically align LLMs, these are often fragile and can be circumvented by jailbreaking attacks through optimized or manual adversarial prompts. To address this, we introduce the Information Bottleneck Protector (IBProtector), a defense mechanism grounded in the information bottleneck principle, and we modify the objective to avoid trivial solutions. The IBProtector selectively compresses and perturbs prompts, facilitated by a lightweight and trainable extractor, preserving only essential information for the target LLMs to respond with the expected answer. Moreover, we further consider a situation where the gradient is not visible to be compatible with any LLM. Our empirical evaluations show that IBProtector outperforms current defense methods in mitigating jailbreak attempts, without overly affecting response quality or inference speed. Its effectiveness and adaptability across various attack methods and target LLMs underscore the potential of IBProtector as a novel, transferable defense that bolsters the security of LLMs without requiring modifications to the underlying models.

Title: Modelling Technique for GDPR-compliance: Toward a Comprehensive Solution

• Authors: Naila Azam, Anna Lito Michala, Shuja Ansari, Nguyen Truong
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.13979
• Pdf URL: https://arxiv.org/pdf/2404.13979
• Copy Paste: [[2404.13979]] Modelling Technique for GDPR-compliance: Toward a Comprehensive Solution(https://arxiv.org/abs/2404.13979)
• Keywords: security, privacy, protect
• Abstract: Data-driven applications and services have been increasingly deployed in all aspects of life including healthcare and medical services in which a huge amount of personal data is collected, aggregated, and processed in a centralised server from various sources. As a consequence, preserving the data privacy and security of these applications is of paramount importance. Since May 2018, the new data protection legislation in the EU/UK, namely the General Data Protection Regulation (GDPR), has come into force and this has called for a critical need for modelling compliance with the GDPR's sophisticated requirements. Existing threat modelling techniques are not designed to model GDPR compliance, particularly in a complex system where personal data is collected, processed, manipulated, and shared with third parties. In this paper, we present a novel comprehensive solution for developing a threat modelling technique to address threats of non-compliance and mitigate them by taking GDPR requirements as the baseline and combining them with the existing security and privacy modelling techniques (i.e., \textit{STRIDE} and \textit{LINDDUN}, respectively). For this purpose, we propose a new data flow diagram integrated with the GDPR principles, develop a knowledge base for the non-compliance threats, and leverage an inference engine for reasoning the GDPR non-compliance threats over the knowledge base. Finally, we demonstrate our solution for threats of non-compliance with legal basis and accountability in a telehealth system to show the feasibility and effectiveness of the proposed solution.

Title: RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance

• Authors: Chengrui Wang, Pengfei Liu, Min Zhou, Ming Zeng, Xubin Li, Tiezheng Ge, Bo zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13984
• Pdf URL: https://arxiv.org/pdf/2404.13984
• Copy Paste: [[2404.13984]] RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance(https://arxiv.org/abs/2404.13984)
• Keywords: diffusion
• Abstract: Although diffusion models can generate high-quality human images, their applications are limited by the instability in generating hands with correct structures. Some previous works mitigate the problem by considering hand structure yet struggle to maintain style consistency between refined malformed hands and other image regions. In this paper, we aim to solve the problem of inconsistency regarding hand structure and style. We propose a conditional diffusion-based framework RHanDS to refine the hand region with the help of decoupled structure and style guidance. Specifically, the structure guidance is the hand mesh reconstructed from the malformed hand, serving to correct the hand structure. The style guidance is a hand image, e.g., the malformed hand itself, and is employed to furnish the style reference for hand refining. In order to suppress the structure leakage when referencing hand style and effectively utilize hand data to improve the capability of the model, we build a multi-style hand dataset and introduce a twostage training strategy. In the first stage, we use paired hand images for training to generate hands with the same style as the reference. In the second stage, various hand images generated based on the human mesh are used for training to enable the model to gain control over the hand structure. We evaluate our method and counterparts on the test dataset of the proposed multi-style hand dataset. The experimental results show that RHanDS can effectively refine hands structure- and style- correctly compared with previous methods. The codes and datasets will be available soon.

Title: Information Re-Organization Improves Reasoning in Large Language Models

• Authors: Xiaoxia Cheng, Zeqi Tan, Weiming Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13985
• Pdf URL: https://arxiv.org/pdf/2404.13985
• Copy Paste: [[2404.13985]] Information Re-Organization Improves Reasoning in Large Language Models(https://arxiv.org/abs/2404.13985)
• Keywords: large language model
• Abstract: Improving the reasoning capabilities of large language models (LLMs) has attracted considerable interest. Recent approaches primarily focus on improving the reasoning process to yield a more precise final answer. However, in scenarios involving contextually aware reasoning, these methods neglect the importance of first identifying logical relationships from the context before proceeding with the reasoning. This oversight could lead to a superficial understanding and interaction with the context, potentially undermining the quality and reliability of the reasoning outcomes. In this paper, we propose an information re-organization (InfoRE) method before proceeding with the reasoning to enhance the reasoning ability of LLMs. We first perform a re-organization processing of the contextual content, e.g., documents or paragraphs, to obtain logical relationships. Then, we utilize the re-organized information in the reasoning process. This enables LLMs to deeply understand the contextual content by clearly perceiving these logical relationships. To demonstrate the effectiveness of our approach in improving the reasoning ability, we conduct experiments using Llama2-70B, GPT-3.5, and GPT-4 on various contextually aware multi-hop reasoning tasks. Using only a zero-shot setting, our method achieves an average improvement of 3\% across all tasks, highlighting its potential to improve the reasoning performance of LLMs. Our source code is available at https://github.com/hustcxx/InfoRE.

Title: Dynamic Proxy Domain Generalizes the Crowd Localization by Better Binary Segmentation

• Authors: Junyu Gao, Da Zhang, Xuelong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13992
• Pdf URL: https://arxiv.org/pdf/2404.13992
• Copy Paste: [[2404.13992]] Dynamic Proxy Domain Generalizes the Crowd Localization by Better Binary Segmentation(https://arxiv.org/abs/2404.13992)
• Keywords: segmentation
• Abstract: Crowd localization targets on predicting each instance precise location within an image. Current advanced methods propose the pixel-wise binary classification to tackle the congested prediction, in which the pixel-level thresholds binarize the prediction confidence of being the pedestrian head. Since the crowd scenes suffer from extremely varying contents, counts and scales, the confidence-threshold learner is fragile and under-generalized encountering domain knowledge shift. Moreover, at the most time, the target domain is agnostic in training. Hence, it is imperative to exploit how to enhance the generalization of confidence-threshold locator to the latent target domain. In this paper, we propose a Dynamic Proxy Domain (DPD) method to generalize the learner under domain shift. Concretely, based on the theoretical analysis to the generalization error risk upper bound on the latent target domain to a binary classifier, we propose to introduce a generated proxy domain to facilitate generalization. Then, based on the theory, we design a DPD algorithm which is composed by a training paradigm and proxy domain generator to enhance the domain generalization of the confidence-threshold learner. Besides, we conduct our method on five kinds of domain shift scenarios, demonstrating the effectiveness on generalizing the crowd localization. Our code will be available at https://github.com/zhangda1018/DPD.

Title: Challenges in automatic and selective plant-clearing

• Authors: Fabrice Mayran de Chamisso, Loïc Cotten, Valentine Dhers, Thomas Lompech, Florian Seywert, Arnaud Susset
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13996
• Pdf URL: https://arxiv.org/pdf/2404.13996
• Copy Paste: [[2404.13996]] Challenges in automatic and selective plant-clearing(https://arxiv.org/abs/2404.13996)
• Keywords: robust, segmentation
• Abstract: With the advent of multispectral imagery and AI, there have been numerous works on automatic plant segmentation for purposes such as counting, picking, health monitoring, localized pesticide delivery, etc. In this paper, we tackle the related problem of automatic and selective plant-clearing in a sustainable forestry context, where an autonomous machine has to detect and avoid specific plants while clearing any weeds which may compete with the species being cultivated. Such an autonomous system requires a high level of robustness to weather conditions, plant variability, terrain and weeds while remaining cheap and easy to maintain. We notably discuss the lack of robustness of spectral imagery, investigate the impact of the reference database's size and discuss issues specific to AI systems operating in uncontrolled environments.

Title: SIGY: Breaking Intel SGX Enclaves with Malicious Exceptions & Signals

• Authors: Supraja Sridhara, Andrin Bertschi, Benedict Schlüter, Shweta Shinde
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.13998
• Pdf URL: https://arxiv.org/pdf/2404.13998
• Copy Paste: [[2404.13998]] SIGY: Breaking Intel SGX Enclaves with Malicious Exceptions & Signals(https://arxiv.org/abs/2404.13998)
• Keywords: attack
• Abstract: User programs recover from hardware exceptions and respond to signals by executing custom handlers that they register specifically for such events. We present SIGY attack, which abuses this programming model on Intel SGX to break the confidentiality and integrity guarantees of enclaves. SIGY uses the untrusted OS to deliver fake hardware events and injects fake signals in an enclave at any point. Such unintended execution of benign program-defined handlers in an enclave corrupts its state and violates execution integrity. 7 runtimes and library OSes (OpenEnclave, Gramine, Scone, Asylo, Teaclave, Occlum, EnclaveOS) are vulnerable to SIGY. 8 languages supported in Intel SGX have programming constructs that are vulnerable to SIGY. We use SIGY to demonstrate 4 proof of concept exploits on webservers (Nginx, Node.js) to leak secrets and data analytics workloads in different languages (C and Java) to break execution integrity.

Title: CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignment

• Authors: Kanglei Zhou, Junlin Li, Ruizhi Cai, Liyuan Wang, Xingxing Zhang, Xiaohui Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.13999
• Pdf URL: https://arxiv.org/pdf/2404.13999
• Copy Paste: [[2404.13999]] CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignment(https://arxiv.org/abs/2404.13999)
• Keywords: interpretability, large language model
• Abstract: Action Quality Assessment (AQA) is pivotal for quantifying actions across domains like sports and medical care. Existing methods often rely on pre-trained backbones from large-scale action recognition datasets to boost performance on smaller AQA datasets. However, this common strategy yields suboptimal results due to the inherent struggle of these backbones to capture the subtle cues essential for AQA. Moreover, fine-tuning on smaller datasets risks overfitting. To address these issues, we propose Coarse-to-Fine Instruction Alignment (CoFInAl). Inspired by recent advances in large language model tuning, CoFInAl aligns AQA with broader pre-trained tasks by reformulating it as a coarse-to-fine classification task. Initially, it learns grade prototypes for coarse assessment and then utilizes fixed sub-grade prototypes for fine-grained assessment. This hierarchical approach mirrors the judging process, enhancing interpretability within the AQA framework. Experimental results on two long-term AQA datasets demonstrate CoFInAl achieves state-of-the-art performance with significant correlation gains of 5.49% and 3.55% on Rhythmic Gymnastics and Fis-V, respectively. Our code is available at https://github.com/ZhouKanglei/CoFInAl_AQA.

Title: Distilled Datamodel with Reverse Gradient Matching

• Authors: Jingwen Ye, Ruonan Yu, Songhua Liu, Xinchao Wang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14006
• Pdf URL: https://arxiv.org/pdf/2404.14006
• Copy Paste: [[2404.14006]] Distilled Datamodel with Reverse Gradient Matching(https://arxiv.org/abs/2404.14006)
• Keywords: interpretability
• Abstract: The proliferation of large-scale AI models trained on extensive datasets has revolutionized machine learning. With these models taking on increasingly central roles in various applications, the need to understand their behavior and enhance interpretability has become paramount. To investigate the impact of changes in training data on a pre-trained model, a common approach is leave-one-out retraining. This entails systematically altering the training dataset by removing specific samples to observe resulting changes within the model. However, retraining the model for each altered dataset presents a significant computational challenge, given the need to perform this operation for every dataset variation. In this paper, we introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. During the offline training phase, we approximate the influence of training data on the target model through a distilled synset, formulated as a reversed gradient matching problem. For online evaluation, we expedite the leave-one-out process using the synset, which is then utilized to compute the attribution matrix based on the evaluation objective. Experimental evaluations, including training data attribution and assessments of data quality, demonstrate that our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.

Title: Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting

• Authors: Weili Zeng, Yichao Yan, Qi Zhu, Zhuo Chen, Pengzhi Chu, Weiming Zhao, Xiaokang Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14007
• Pdf URL: https://arxiv.org/pdf/2404.14007
• Copy Paste: [[2404.14007]] Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting(https://arxiv.org/abs/2404.14007)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) customization aims to create images that embody specific visual concepts delineated in textual descriptions. However, existing works still face a main challenge, concept overfitting. To tackle this challenge, we first analyze overfitting, categorizing it into concept-agnostic overfitting, which undermines non-customized concept knowledge, and concept-specific overfitting, which is confined to customize on limited modalities, i.e, backgrounds, layouts, styles. To evaluate the overfitting degree, we further introduce two metrics, i.e, Latent Fisher divergence and Wasserstein metric to measure the distribution changes of non-customized and customized concept respectively. Drawing from the analysis, we propose Infusion, a T2I customization method that enables the learning of target concepts to avoid being constrained by limited training modalities, while preserving non-customized knowledge. Remarkably, Infusion achieves this feat with remarkable efficiency, requiring a mere 11KB of trained parameters. Extensive experiments also demonstrate that our approach outperforms state-of-the-art methods in both single and multi-concept customized generation.

Title: Ungeneralizable Examples

• Authors: Jingwen Ye, Xinchao Wang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14016
• Pdf URL: https://arxiv.org/pdf/2404.14016
• Copy Paste: [[2404.14016]] Ungeneralizable Examples(https://arxiv.org/abs/2404.14016)
• Keywords: privacy, protect, attack
• Abstract: The training of contemporary deep learning models heavily relies on publicly available data, posing a risk of unauthorized access to online data and raising concerns about data privacy. Current approaches to creating unlearnable data involve incorporating small, specially designed noises, but these methods strictly limit data usability, overlooking its potential usage in authorized scenarios. In this paper, we extend the concept of unlearnable data to conditional data learnability and introduce \textbf{U}n\textbf{G}eneralizable \textbf{E}xamples (UGEs). UGEs exhibit learnability for authorized users while maintaining unlearnability for potential hackers. The protector defines the authorized network and optimizes UGEs to match the gradients of the original data and its ungeneralizable version, ensuring learnability. To prevent unauthorized learning, UGEs are trained by maximizing a designated distance loss in a common feature space. Additionally, to further safeguard the authorized side from potential attacks, we introduce additional undistillation optimization. Experimental results on multiple datasets and various networks demonstrate that the proposed UGEs framework preserves data usability while reducing training performance on hacker networks, even under different types of attacks.

Title: A Multimodal Feature Distillation with CNN-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

• Authors: Ming Kang, Fung Fung Ting, Raphaël C.-W. Phan, Zongyuan Ge, Chee-Ming Ting
• Subjects: cs.CV, eess.SP, stat.AP
• Abstract URL: https://arxiv.org/abs/2404.14019
• Pdf URL: https://arxiv.org/pdf/2404.14019
• Copy Paste: [[2404.14019]] A Multimodal Feature Distillation with CNN-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities(https://arxiv.org/abs/2404.14019)
• Keywords: transformer, segmentation
• Abstract: Existing brain tumor segmentation methods usually utilize multiple Magnetic Resonance Imaging (MRI) modalities in brain tumor images for segmentation, which can achieve better segmentation performance. However, in clinical applications, some modalities are missing due to resource constraints, leading to severe degradation in the performance of methods applying complete modality segmentation. In this paper, we propose a Multimodal feature distillation with Convolutional Neural Network (CNN)-Transformer hybrid network (MCTSeg) for accurate brain tumor segmentation with missing modalities. We first design a Multimodal Feature Distillation (MFD) module to distill feature-level multimodal knowledge into different unimodality to extract complete modality information. We further develop a Unimodal Feature Enhancement (UFE) module to model the relationship between global and local information semantically. Finally, we build a Cross-Modal Fusion (CMF) module to explicitly align the global correlations among different modalities even when some modalities are missing. Complementary features within and across different modalities are refined via the CNN-Transformer hybrid architectures in both the UFE and CMF modules, where local and global dependencies are both captured. Our ablation study demonstrates the importance of the proposed modules with CNN-Transformer networks and the convolutional blocks in Transformer for improving the performance of brain tumor segmentation with missing modalities. Extensive experiments on the BraTS2018 and BraTS2020 datasets show that the proposed MCTSeg framework outperforms the state-of-the-art methods in missing modalities cases. Our code is available at: https://github.com/mkang315/MCTSeg.

Title: Collaborative Perception Datasets in Autonomous Driving: A Survey

• Authors: Melih Yazgan, Mythra Varun Akkanapragada, J. Marius Zoellner
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.14022
• Pdf URL: https://arxiv.org/pdf/2404.14022
• Copy Paste: [[2404.14022]] Collaborative Perception Datasets in Autonomous Driving: A Survey(https://arxiv.org/abs/2404.14022)
• Keywords: security, privacy
• Abstract: This survey offers a comprehensive examination of collaborative perception datasets in the context of Vehicle-to-Infrastructure (V2I), Vehicle-to-Vehicle (V2V), and Vehicle-to-Everything (V2X). It highlights the latest developments in large-scale benchmarks that accelerate advancements in perception tasks for autonomous vehicles. The paper systematically analyzes a variety of datasets, comparing them based on aspects such as diversity, sensor setup, quality, public availability, and their applicability to downstream tasks. It also highlights the key challenges such as domain shift, sensor setup limitations, and gaps in dataset diversity and availability. The importance of addressing privacy and security concerns in the development of datasets is emphasized, regarding data sharing and dataset creation. The conclusion underscores the necessity for comprehensive, globally accessible datasets and collaborative efforts from both technological and research communities to overcome these challenges and fully harness the potential of autonomous driving.

Title: OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks

• Authors: Sophia Sirko-Galouchenko, Alexandre Boulch, Spyros Gidaris, Andrei Bursuc, Antonin Vobecky, Patrick Pérez, Renaud Marlet
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14027
• Pdf URL: https://arxiv.org/pdf/2404.14027
• Copy Paste: [[2404.14027]] OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks(https://arxiv.org/abs/2404.14027)
• Keywords: segmentation
• Abstract: We introduce a self-supervised pretraining method, called OcFeat, for camera-only Bird's-Eye-View (BEV) segmentation networks. With OccFeat, we pretrain a BEV network via occupancy prediction and feature distillation tasks. Occupancy prediction provides a 3D geometric understanding of the scene to the model. However, the geometry learned is class-agnostic. Hence, we add semantic information to the model in the 3D space through distillation from a self-supervised pretrained image foundation model. Models pretrained with our method exhibit improved BEV semantic segmentation performance, particularly in low-data scenarios. Moreover, empirical results affirm the efficacy of integrating feature distillation with 3D occupancy prediction in our pretraining approach.

Title: PointDifformer: Robust Point Cloud Registration With Neural Diffusion and Transformer

• Authors: Rui She, Qiyu Kang, Sijie Wang, Wee Peng Tay, Kai Zhao, Yang Song, Tianyu Geng, Yi Xu, Diego Navarro Navarro, Andreas Hartmannsgruber
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14034
• Pdf URL: https://arxiv.org/pdf/2404.14034
• Copy Paste: [[2404.14034]] PointDifformer: Robust Point Cloud Registration With Neural Diffusion and Transformer(https://arxiv.org/abs/2404.14034)
• Keywords: robust, diffusion, transformer
• Abstract: Point cloud registration is a fundamental technique in 3-D computer vision with applications in graphics, autonomous driving, and robotics. However, registration tasks under challenging conditions, under which noise or perturbations are prevalent, can be difficult. We propose a robust point cloud registration approach that leverages graph neural partial differential equations (PDEs) and heat kernel signatures. Our method first uses graph neural PDE modules to extract high dimensional features from point clouds by aggregating information from the 3-D point neighborhood, thereby enhancing the robustness of the feature representations. Then, we incorporate heat kernel signatures into an attention mechanism to efficiently obtain corresponding keypoints. Finally, a singular value decomposition (SVD) module with learnable weights is used to predict the transformation between two point clouds. Empirical experiments on a 3-D point cloud dataset demonstrate that our approach not only achieves state-of-the-art performance for point cloud registration but also exhibits better robustness to additive noise or 3-D shape perturbations.

Title: GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting

• Authors: Hongyun Yu, Zhan Qu, Qihang Yu, Jianchuan Chen, Zhonghua Jiang, Zhiwen Chen, Shengyu Zhang, Jimin Xu, Fei Wu, Chengfei Lv, Gang Yu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.14037
• Pdf URL: https://arxiv.org/pdf/2404.14037
• Copy Paste: [[2404.14037]] GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting(https://arxiv.org/abs/2404.14037)
• Keywords: extraction
• Abstract: Recent works on audio-driven talking head synthesis using Neural Radiance Fields (NeRF) have achieved impressive results. However, due to inadequate pose and expression control caused by NeRF implicit representation, these methods still have some limitations, such as unsynchronized or unnatural lip movements, and visual jitter and artifacts. In this paper, we propose GaussianTalker, a novel method for audio-driven talking head synthesis based on 3D Gaussian Splatting. With the explicit representation property of 3D Gaussians, intuitive control of the facial motion is achieved by binding Gaussians to 3D facial models. GaussianTalker consists of two modules, Speaker-specific Motion Translator and Dynamic Gaussian Renderer. Speaker-specific Motion Translator achieves accurate lip movements specific to the target speaker through universalized audio feature extraction and customized lip motion generation. Dynamic Gaussian Renderer introduces Speaker-specific BlendShapes to enhance facial detail representation via a latent pose, delivering stable and realistic rendered videos. Extensive experimental results suggest that GaussianTalker outperforms existing state-of-the-art methods in talking head synthesis, delivering precise lip synchronization and exceptional visual quality. Our method achieves rendering speeds of 130 FPS on NVIDIA RTX4090 GPU, significantly exceeding the threshold for real-time rendering performance, and can potentially be deployed on other hardware platforms.

Title: Surgical-DeSAM: Decoupling SAM for Instrument Segmentation in Robotic Surgery

• Authors: Yuyang Sheng, Sophia Bano, Matthew J. Clarkson, Mobarakol Islam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14040
• Pdf URL: https://arxiv.org/pdf/2404.14040
• Copy Paste: [[2404.14040]] Surgical-DeSAM: Decoupling SAM for Instrument Segmentation in Robotic Surgery(https://arxiv.org/abs/2404.14040)
• Keywords: transformer, segmentation
• Abstract: Purpose: The recent Segment Anything Model (SAM) has demonstrated impressive performance with point, text or bounding box prompts, in various applications. However, in safety-critical surgical tasks, prompting is not possible due to (i) the lack of per-frame prompts for supervised learning, (ii) it is unrealistic to prompt frame-by-frame in a real-time tracking application, and (iii) it is expensive to annotate prompts for offline applications. Methods: We develop Surgical-DeSAM to generate automatic bounding box prompts for decoupling SAM to obtain instrument segmentation in real-time robotic surgery. We utilise a commonly used detection architecture, DETR, and fine-tuned it to obtain bounding box prompt for the instruments. We then empolyed decoupling SAM (DeSAM) by replacing the image encoder with DETR encoder and fine-tune prompt encoder and mask decoder to obtain instance segmentation for the surgical instruments. To improve detection performance, we adopted the Swin-transformer to better feature representation. Results: The proposed method has been validated on two publicly available datasets from the MICCAI surgical instruments segmentation challenge EndoVis 2017 and 2018. The performance of our method is also compared with SOTA instrument segmentation methods and demonstrated significant improvements with dice metrics of 89.62 and 90.70 for the EndoVis 2017 and 2018. Conclusion: Our extensive experiments and validations demonstrate that Surgical-DeSAM enables real-time instrument segmentation without any additional prompting and outperforms other SOTA segmentation methods.

Title: CloudFort: Enhancing Robustness of 3D Point Cloud Classification Against Backdoor Attacks via Spatial Partitioning and Ensemble Prediction

• Authors: Wenhao Lan, Yijun Yang, Haihua Shen, Shan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14042
• Pdf URL: https://arxiv.org/pdf/2404.14042
• Copy Paste: [[2404.14042]] CloudFort: Enhancing Robustness of 3D Point Cloud Classification Against Backdoor Attacks via Spatial Partitioning and Ensemble Prediction(https://arxiv.org/abs/2404.14042)
• Keywords: security, defense, attack, robust
• Abstract: The increasing adoption of 3D point cloud data in various applications, such as autonomous vehicles, robotics, and virtual reality, has brought about significant advancements in object recognition and scene understanding. However, this progress is accompanied by new security challenges, particularly in the form of backdoor attacks. These attacks involve inserting malicious information into the training data of machine learning models, potentially compromising the model's behavior. In this paper, we propose CloudFort, a novel defense mechanism designed to enhance the robustness of 3D point cloud classifiers against backdoor attacks. CloudFort leverages spatial partitioning and ensemble prediction techniques to effectively mitigate the impact of backdoor triggers while preserving the model's performance on clean data. We evaluate the effectiveness of CloudFort through extensive experiments, demonstrating its strong resilience against the Point Cloud Backdoor Attack (PCBA). Our results show that CloudFort significantly enhances the security of 3D point cloud classification models without compromising their accuracy on benign samples. Furthermore, we explore the limitations of CloudFort and discuss potential avenues for future research in the field of 3D point cloud security. The proposed defense mechanism represents a significant step towards ensuring the trustworthiness and reliability of point-cloud-based systems in real-world applications.

Title: LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generation

• Authors: Keheng Wang, Feiyu Duan, Peiguang Li, Sirui Wang, Xunliang Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14043
• Pdf URL: https://arxiv.org/pdf/2404.14043
• Copy Paste: [[2404.14043]] LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generation(https://arxiv.org/abs/2404.14043)
• Keywords: extraction
• Abstract: Retrieval-Augmented Generation (RAG) demonstrates great value in alleviating outdated knowledge or hallucination by supplying LLMs with updated and relevant knowledge. However, there are still several difficulties for RAG in understanding complex multi-hop query and retrieving relevant documents, which require LLMs to perform reasoning and retrieve step by step. Inspired by human's reasoning process in which they gradually search for the required information, it is natural to ask whether the LLMs could notice the missing information in each reasoning step. In this work, we first experimentally verified the ability of LLMs to extract information as well as to know the missing. Based on the above discovery, we propose a Missing Information Guided Retrieve-Extraction-Solving paradigm (MIGRES), where we leverage the identification of missing information to generate a targeted query that steers the subsequent knowledge retrieval. Besides, we design a sentence-level re-ranking filtering approach to filter the irrelevant content out from document, along with the information extraction capability of LLMs to extract useful information from cleaned-up documents, which in turn to bolster the overall efficacy of RAG. Extensive experiments conducted on multiple public datasets reveal the superiority of the proposed MIGRES method, and analytical experiments demonstrate the effectiveness of our proposed modules.

Title: How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study

• Authors: Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, Michele Magno
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14047
• Pdf URL: https://arxiv.org/pdf/2404.14047
• Copy Paste: [[2404.14047]] How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study(https://arxiv.org/abs/2404.14047)
• Keywords: large language model
• Abstract: Meta's LLaMA family has become one of the most powerful open-source Large Language Model (LLM) series. Notably, LLaMA3 models have recently been released and achieve impressive performance across various with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-limited scenarios, we explore LLaMA3's capabilities when quantized to low bit-width. This exploration holds the potential to unveil new insights and challenges for low-bit quantization of LLaMA3 and other forthcoming LLMs, especially in addressing performance degradation problems that suffer in LLM compression. Specifically, we evaluate the 10 existing post-training quantization and LoRA-finetuning methods of LLaMA3 on 1-8 bits and diverse datasets to comprehensively reveal LLaMA3's low-bit quantization performance. Our experiment results indicate that LLaMA3 still suffers non-negligent degradation in these scenarios, especially in ultra-low bit-width. This highlights the significant performance gap under low bit-width that needs to be bridged in future developments. We expect that this empirical study will prove valuable in advancing future models, pushing the LLMs to lower bit-width with higher accuracy for being practical. Our project is released on https://github.com/Macaronlin/LLaMA3-Quantization and quantized LLaMA3 models are released in https://huggingface.co/LLMQ.

Title: RingID: Rethinking Tree-Ring Watermarking for Enhanced Multi-Key Identification

• Authors: Hai Ci, Pei Yang, Yiren Song, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14055
• Pdf URL: https://arxiv.org/pdf/2404.14055
• Copy Paste: [[2404.14055]] RingID: Rethinking Tree-Ring Watermarking for Enhanced Multi-Key Identification(https://arxiv.org/abs/2404.14055)
• Keywords: attack, robust, watermark, diffusion
• Abstract: We revisit Tree-Ring Watermarking, a recent diffusion model watermarking method that demonstrates great robustness to various attacks. We conduct an in-depth study on it and reveal that the distribution shift unintentionally introduced by the watermarking process, apart from watermark pattern matching, contributes to its exceptional robustness. Our investigation further exposes inherent flaws in its original design, particularly in its ability to identify multiple distinct keys, where distribution shift offers no assistance. Based on these findings and analysis, we present RingID for enhanced multi-key identification. It consists of a novel multi-channel heterogeneous watermarking approach designed to seamlessly amalgamate distinctive advantages from diverse watermarks. Coupled with a series of suggested enhancements, RingID exhibits substantial advancements in multi-key identification.

Title: FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learning

• Authors: Yinlin Zhu, Xunkai Li, Zhengyu Wu, Di Wu, Miao Hu, Rong-Hua Li
• Subjects: cs.LG, cs.AI, cs.DB, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.14061
• Pdf URL: https://arxiv.org/pdf/2404.14061
• Copy Paste: [[2404.14061]] FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learning(https://arxiv.org/abs/2404.14061)
• Keywords: federate, data-free
• Abstract: Subgraph federated learning (subgraph-FL) is a new distributed paradigm that facilitates the collaborative training of graph neural networks (GNNs) by multi-client subgraphs. Unfortunately, a significant challenge of subgraph-FL arises from subgraph heterogeneity, which stems from node and topology variation, causing the impaired performance of the global GNN. Despite various studies, they have not yet thoroughly investigated the impact mechanism of subgraph heterogeneity. To this end, we decouple node and topology variation, revealing that they correspond to differences in label distribution and structure homophily. Remarkably, these variations lead to significant differences in the class-wise knowledge reliability of multiple local GNNs, misguiding the model aggregation with varying degrees. Building on this insight, we propose topology-aware data-free knowledge distillation technology (FedTAD), enhancing reliable knowledge transfer from the local model to the global model. Extensive experiments on six public datasets consistently demonstrate the superiority of FedTAD over state-of-the-art baselines.

Title: GatedLexiconNet: A Comprehensive End-to-End Handwritten Paragraph Text Recognition System

• Authors: Lalita Kumari, Sukhdeep Singh, Vaibhav Varish Singh Rathore, Anuj Sharma
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14062
• Pdf URL: https://arxiv.org/pdf/2404.14062
• Copy Paste: [[2404.14062]] GatedLexiconNet: A Comprehensive End-to-End Handwritten Paragraph Text Recognition System(https://arxiv.org/abs/2404.14062)
• Keywords: segmentation
• Abstract: The Handwritten Text Recognition problem has been a challenge for researchers for the last few decades, especially in the domain of computer vision, a subdomain of pattern recognition. Variability of texts amongst writers, cursiveness, and different font styles of handwritten texts with degradation of historical text images make it a challenging problem. Recognizing scanned document images in neural network-based systems typically involves a two-step approach: segmentation and recognition. However, this method has several drawbacks. These shortcomings encompass challenges in identifying text regions, analyzing layout diversity within pages, and establishing accurate ground truth segmentation. Consequently, these processes are prone to errors, leading to bottlenecks in achieving high recognition accuracies. Thus, in this study, we present an end-to-end paragraph recognition system that incorporates internal line segmentation and gated convolutional layers based encoder. The gating is a mechanism that controls the flow of information and allows to adaptively selection of the more relevant features in handwritten text recognition models. The attention module plays an important role in performing internal line segmentation, allowing the page to be processed line-by-line. During the decoding step, we have integrated a connectionist temporal classification-based word beam search decoder as a post-processing step. In this work, we have extended existing LexiconNet by carefully applying and utilizing gated convolutional layers in the existing deep neural network. Our results at line and page levels also favour our new GatedLexiconNet. This study reported character error rates of 2.27% on IAM, 0.9% on RIMES, and 2.13% on READ-16, and word error rates of 5.73% on IAM, 2.76% on RIMES, and 6.52% on READ-2016 datasets.

Title: Towards Robust Trajectory Representations: Isolating Environmental Confounders with Causal Learning

• Authors: Kang Luo, Yuanshao Zhu, Wei Chen, Kun Wang, Zhengyang Zhou, Sijie Ruan, Yuxuan Liang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14073
• Pdf URL: https://arxiv.org/pdf/2404.14073
• Copy Paste: [[2404.14073]] Towards Robust Trajectory Representations: Isolating Environmental Confounders with Causal Learning(https://arxiv.org/abs/2404.14073)
• Keywords: robust, interpretability
• Abstract: Trajectory modeling refers to characterizing human movement behavior, serving as a pivotal step in understanding mobility patterns. Nevertheless, existing studies typically ignore the confounding effects of geospatial context, leading to the acquisition of spurious correlations and limited generalization capabilities. To bridge this gap, we initially formulate a Structural Causal Model (SCM) to decipher the trajectory representation learning process from a causal perspective. Building upon the SCM, we further present a Trajectory modeling framework (TrajCL) based on Causal Learning, which leverages the backdoor adjustment theory as an intervention tool to eliminate the spurious correlations between geospatial context and trajectories. Extensive experiments on two real-world datasets verify that TrajCL markedly enhances performance in trajectory classification tasks while showcasing superior generalization and interpretability.

Title: DPTraj-PM: Differentially Private Trajectory Synthesis Using Prefix Tree and Markov Process

• Authors: Nana Wang, Mohan Kankanhalli
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.14106
• Pdf URL: https://arxiv.org/pdf/2404.14106
• Copy Paste: [[2404.14106]] DPTraj-PM: Differentially Private Trajectory Synthesis Using Prefix Tree and Markov Process(https://arxiv.org/abs/2404.14106)
• Keywords: privacy, protect
• Abstract: The increasing use of GPS-enabled devices has generated a large amount of trajectory data. These data offer us vital insights to understand the movements of individuals and populations, benefiting a broad range of applications from transportation planning to epidemic modeling. However, improper release of trajectory data is increasing concerns on individual privacy. Previous attempts either lack strong privacy guarantees, or fail to preserve sufficient basic characteristics of the original data. In this paper, we propose DPTraj-PM, a method to synthesize trajectory dataset under the differential privacy (DP) framework while ensures high data utility. Based on the assumption that an individual's trajectory could be mainly determined by the initial trajectory segment (which depicts the starting point and the initial direction) and the next location point, DPTraj-PM discretizes the raw trajectories into neighboring cells, and models them by combining a prefix tree structure and an m-order Markov process. After adding noise to the model under differential privacy, DPTraj-PM generates a synthetic dataset from the noisy model to enable a wider spectrum of data mining and modeling tasks. The output traces crafted by DPTraj-PM not only preserves the patterns and variability in individuals' mobility behaviors, but also protects individual privacy. Experiments on two real-world datasets demonstrate that DPTraj-PM substantially outperforms the state-of-the-art techniques in terms of data utility. Our code is available at https://github.com/wnn5/DP-PrefixTreeMarkov.

Title: Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?

• Authors: Dawei Zhu, Pinzhen Chen, Miaoran Zhang, Barry Haddow, Xiaoyu Shen, Dietrich Klakow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14122
• Pdf URL: https://arxiv.org/pdf/2404.14122
• Copy Paste: [[2404.14122]] Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?(https://arxiv.org/abs/2404.14122)
• Keywords: large language model
• Abstract: Traditionally, success in multilingual machine translation can be attributed to three key factors in training data: large volume, diverse translation directions, and high quality. In the current practice of fine-tuning large language models (LLMs) for translation, we revisit the importance of all these factors. We find that LLMs display strong translation capability after being fine-tuned on as few as 32 training instances, and that fine-tuning on a single translation direction effectively enables LLMs to translate in multiple directions. However, the choice of direction is critical: fine-tuning LLMs with English on the target side can lead to task misinterpretation, which hinders translations into non-English languages. A similar problem arises when noise is introduced into the target side of parallel data, especially when the target language is well-represented in the LLM's pre-training. In contrast, noise in an under-represented language has a less pronounced effect. Our findings suggest that attaining successful alignment hinges on teaching the model to maintain a "superficial" focus, thereby avoiding the learning of erroneous biases beyond translation.

Title: CRNet: A Detail-Preserving Network for Unified Image Restoration and Enhancement Task

• Authors: Kangzhen Yang, Tao Hu, Kexin Dai, Genggeng Chen, Yu Cao, Wei Dong, Peng Wu, Yanning Zhang, Qingsen Yan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.14132
• Pdf URL: https://arxiv.org/pdf/2404.14132
• Copy Paste: [[2404.14132]] CRNet: A Detail-Preserving Network for Unified Image Restoration and Enhancement Task(https://arxiv.org/abs/2404.14132)
• Keywords: secure
• Abstract: In real-world scenarios, images captured often suffer from blurring, noise, and other forms of image degradation, and due to sensor limitations, people usually can only obtain low dynamic range images. To achieve high-quality images, researchers have attempted various image restoration and enhancement operations on photographs, including denoising, deblurring, and high dynamic range imaging. However, merely performing a single type of image enhancement still cannot yield satisfactory images. In this paper, to deal with the challenge above, we propose the Composite Refinement Network (CRNet) to address this issue using multiple exposure images. By fully integrating information-rich multiple exposure inputs, CRNet can perform unified image restoration and enhancement. To improve the quality of image details, CRNet explicitly separates and strengthens high and low-frequency information through pooling layers, using specially designed Multi-Branch Blocks for effective fusion of these frequencies. To increase the receptive field and fully integrate input features, CRNet employs the High-Frequency Enhancement Module, which includes large kernel convolutions and an inverted bottleneck ConvFFN. Our model secured third place in the first track of the Bracketing Image Restoration and Enhancement Challenge, surpassing previous SOTA models in both testing metrics and visual quality.

Title: Text in the Dark: Extremely Low-Light Text Image Enhancement

• Authors: Che-Tsung Lin, Chun Chet Ng, Zhi Qin Tan, Wan Jun Nah, Xinyu Wang, Jie Long Kew, Pohao Hsu, Shang Hong Lai, Chee Seng Chan, Christopher Zach
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14135
• Pdf URL: https://arxiv.org/pdf/2404.14135
• Copy Paste: [[2404.14135]] Text in the Dark: Extremely Low-Light Text Image Enhancement(https://arxiv.org/abs/2404.14135)
• Keywords: extraction
• Abstract: Extremely low-light text images are common in natural scenes, making scene text detection and recognition challenging. One solution is to enhance these images using low-light image enhancement methods before text extraction. However, previous methods often do not try to particularly address the significance of low-level features, which are crucial for optimal performance on downstream scene text tasks. Further research is also hindered by the lack of extremely low-light text datasets. To address these limitations, we propose a novel encoder-decoder framework with an edge-aware attention module to focus on scene text regions during enhancement. Our proposed method uses novel text detection and edge reconstruction losses to emphasize low-level scene text features, leading to successful text extraction. Additionally, we present a Supervised Deep Curve Estimation (Supervised-DCE) model to synthesize extremely low-light images based on publicly available scene text datasets such as ICDAR15 (IC15). We also labeled texts in the extremely low-light See In the Dark (SID) and ordinary LOw-Light (LOL) datasets to allow for objective assessment of extremely low-light image enhancement through scene text tasks. Extensive experiments show that our model outperforms state-of-the-art methods in terms of both image quality and scene text metrics on the widely-used LOL, SID, and synthetic IC15 datasets. Code and dataset will be released publicly at https://github.com/chunchet-ng/Text-in-the-Dark.

Title: Offensive AI: Enhancing Directory Brute-forcing Attack with the Use of Language Models

• Authors: Alberto Castagnaro, Mauro Conti, Luca Pajola
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.14138
• Pdf URL: https://arxiv.org/pdf/2404.14138
• Copy Paste: [[2404.14138]] Offensive AI: Enhancing Directory Brute-forcing Attack with the Use of Language Models(https://arxiv.org/abs/2404.14138)
• Keywords: security, attack
• Abstract: Web Vulnerability Assessment and Penetration Testing (Web VAPT) is a comprehensive cybersecurity process that uncovers a range of vulnerabilities which, if exploited, could compromise the integrity of web applications. In a VAPT, it is common to perform a \textit{Directory brute-forcing Attack}, aiming at the identification of accessible directories of a target website. Current commercial solutions are inefficient as they are based on brute-forcing strategies that use wordlists, resulting in enormous quantities of trials for a small amount of success. Offensive AI is a recent paradigm that integrates AI-based technologies in cyber attacks. In this work, we explore whether AI can enhance the directory enumeration process and propose a novel Language Model-based framework. Our experiments -- conducted in a testbed consisting of 1 million URLs from different web application domains (universities, hospitals, government, companies) -- demonstrate the superiority of the LM-based attack, with an average performance increase of 969%.

Title: Multidimensional Interpolants

• Authors: Dohoon Lee, Kyogu Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14161
• Pdf URL: https://arxiv.org/pdf/2404.14161
• Copy Paste: [[2404.14161]] Multidimensional Interpolants(https://arxiv.org/abs/2404.14161)
• Keywords: generative
• Abstract: In the domain of differential equation-based generative modeling, conventional approaches often rely on single-dimensional scalar values as interpolation coefficients during both training and inference phases. In this work, we introduce, for the first time, a multidimensional interpolant that extends these coefficients into multiple dimensions, leveraging the stochastic interpolant framework. Additionally, we propose a novel path optimization problem tailored to adaptively determine multidimensional inference trajectories, with a predetermined differential equation solver and a fixed number of function evaluations. Our solution involves simulation dynamics coupled with adversarial training to optimize the inference path. Notably, employing a multidimensional interpolant during training improves the model's inference performance, even in the absence of path optimization. When the adaptive, multidimensional path derived from our optimization process is employed, it yields further performance gains, even with fixed solver configurations. The introduction of multidimensional interpolants not only enhances the efficacy of models but also opens up a new domain for exploration in training and inference methodologies, emphasizing the potential of multidimensional paths as an untapped frontier.

Title: FLDM-VTON: Faithful Latent Diffusion Model for Virtual Try-on

• Authors: Chenhui Wang, Tao Chen, Zhihao Chen, Zhizhong Huang, Taoran Jiang, Qi Wang, Hongming Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14162
• Pdf URL: https://arxiv.org/pdf/2404.14162
• Copy Paste: [[2404.14162]] FLDM-VTON: Faithful Latent Diffusion Model for Virtual Try-on(https://arxiv.org/abs/2404.14162)
• Keywords: diffusion, generative
• Abstract: Despite their impressive generative performance, latent diffusion model-based virtual try-on (VTON) methods lack faithfulness to crucial details of the clothes, such as style, pattern, and text. To alleviate these issues caused by the diffusion stochastic nature and latent supervision, we propose a novel Faithful Latent Diffusion Model for VTON, termed FLDM-VTON. FLDM-VTON improves the conventional latent diffusion process in three major aspects. First, we propose incorporating warped clothes as both the starting point and local condition, supplying the model with faithful clothes priors. Second, we introduce a novel clothes flattening network to constrain generated try-on images, providing clothes-consistent faithful supervision. Third, we devise a clothes-posterior sampling for faithful inference, further enhancing the model performance over conventional clothes-agnostic Gaussian sampling. Extensive experimental results on the benchmark VITON-HD and Dress Code datasets demonstrate that our FLDM-VTON outperforms state-of-the-art baselines and is able to generate photo-realistic try-on images with faithful clothing details.

Title: New Solutions Based on the Generalized Eigenvalue Problem for the Data Collaboration Analysis

• Authors: Yuta Kawakami, Yuichi Takano, Akira Imakura
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.14164
• Pdf URL: https://arxiv.org/pdf/2404.14164
• Copy Paste: [[2404.14164]] New Solutions Based on the Generalized Eigenvalue Problem for the Data Collaboration Analysis(https://arxiv.org/abs/2404.14164)
• Keywords: protect, segmentation
• Abstract: In recent years, the accumulation of data across various institutions has garnered attention for the technology of confidential data analysis, which improves analytical accuracy by sharing data between multiple institutions while protecting sensitive information. Among these methods, Data Collaboration Analysis (DCA) is noted for its efficiency in terms of computational cost and communication load, facilitating data sharing and analysis across different institutions while safeguarding confidential information. However, existing optimization problems for determining the necessary collaborative functions have faced challenges, such as the optimal solution for the collaborative representation often being a zero matrix and the difficulty in understanding the process of deriving solutions. This research addresses these issues by formulating the optimization problem through the segmentation of matrices into column vectors and proposing a solution method based on the generalized eigenvalue problem. Additionally, we demonstrate methods for constructing collaborative functions more effectively through weighting and the selection of efficient algorithms suited to specific situations. Experiments using real-world datasets have shown that our proposed formulation and solution for the collaborative function optimization problem achieve superior predictive accuracy compared to existing methods.

Title: Face2Face: Label-driven Facial Retouching Restoration

• Authors: Guanhua Zhao, Yu Gu, Xuhan Sheng, Yujie Hu, Jian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14177
• Pdf URL: https://arxiv.org/pdf/2404.14177
• Copy Paste: [[2404.14177]] Face2Face: Label-driven Facial Retouching Restoration(https://arxiv.org/abs/2404.14177)
• Keywords: diffusion
• Abstract: With the popularity of social media platforms such as Instagram and TikTok, and the widespread availability and convenience of retouching tools, an increasing number of individuals are utilizing these tools to beautify their facial photographs. This poses challenges for fields that place high demands on the authenticity of photographs, such as identity verification and social media. By altering facial images, users can easily create deceptive images, leading to the dissemination of false information. This may pose challenges to the reliability of identity verification systems and social media, and even lead to online fraud. To address this issue, some work has proposed makeup removal methods, but they still lack the ability to restore images involving geometric deformations caused by retouching. To tackle the problem of facial retouching restoration, we propose a framework, dubbed Face2Face, which consists of three components: a facial retouching detector, an image restoration model named FaceR, and a color correction module called Hierarchical Adaptive Instance Normalization (H-AdaIN). Firstly, the facial retouching detector predicts a retouching label containing three integers, indicating the retouching methods and their corresponding degrees. Then FaceR restores the retouched image based on the predicted retouching label. Finally, H-AdaIN is applied to address the issue of color shift arising from diffusion models. Extensive experiments demonstrate the effectiveness of our framework and each module.

Title: BCFPL: Binary classification ConvNet based Fast Parking space recognition with Low resolution image

• Authors: Shuo Zhang, Xin Chen, Zixuan Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14198
• Pdf URL: https://arxiv.org/pdf/2404.14198
• Copy Paste: [[2404.14198]] BCFPL: Binary classification ConvNet based Fast Parking space recognition with Low resolution image(https://arxiv.org/abs/2404.14198)
• Keywords: privacy, protect
• Abstract: The automobile plays an important role in the economic activities of mankind, especially in the metropolis. Under the circumstances, the demand of quick search for available parking spaces has become a major concern for the automobile drivers. Meanwhile, the public sense of privacy is also awaking, the image-based parking space recognition methods lack the attention of privacy protection. In this paper, we proposed a binary convolutional neural network with lightweight design structure named BCFPL, which can be used to train with low-resolution parking space images and offer a reasonable recognition result. The images of parking space were collected from various complex environments, including different weather, occlusion conditions, and various camera angles. We conducted the training and testing progresses among different datasets and partial subsets. The experimental results show that the accuracy of BCFPL does not decrease compared with the original resolution image directly, and can reach the average level of the existing mainstream method. BCFPL also has low hardware requirements and fast recognition speed while meeting the privacy requirements, so it has application potential in intelligent city construction and automatic driving field.

Title: EnzChemRED, a rich enzyme chemistry relation extraction dataset

• Authors: Po-Ting Lai, Elisabeth Coudert, Lucila Aimo, Kristian Axelsen, Lionel Breuza, Edouard de Castro, Marc Feuermann, Anne Morgat, Lucille Pourcel, Ivo Pedruzzi, Sylvain Poux, Nicole Redaschi, Catherine Rivoire, Anastasia Sveshnikova, Chih-Hsuan Wei, Robert Leaman, Ling Luo, Zhiyong Lu, Alan Bridge
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14209
• Pdf URL: https://arxiv.org/pdf/2404.14209
• Copy Paste: [[2404.14209]] EnzChemRED, a rich enzyme chemistry relation extraction dataset(https://arxiv.org/abs/2404.14209)
• Keywords: extraction, fair
• Abstract: Expert curation is essential to capture knowledge of enzyme functions from the scientific literature in FAIR open knowledgebases but cannot keep pace with the rate of new discoveries and new publications. In this work we present EnzChemRED, for Enzyme Chemistry Relation Extraction Dataset, a new training and benchmarking dataset to support the development of Natural Language Processing (NLP) methods such as (large) language models that can assist enzyme curation. EnzChemRED consists of 1,210 expert curated PubMed abstracts in which enzymes and the chemical reactions they catalyze are annotated using identifiers from the UniProt Knowledgebase (UniProtKB) and the ontology of Chemical Entities of Biological Interest (ChEBI). We show that fine-tuning pre-trained language models with EnzChemRED can significantly boost their ability to identify mentions of proteins and chemicals in text (Named Entity Recognition, or NER) and to extract the chemical conversions in which they participate (Relation Extraction, or RE), with average F1 score of 86.30% for NER, 86.66% for RE for chemical conversion pairs, and 83.79% for RE for chemical conversion pairs and linked enzymes. We combine the best performing methods after fine-tuning using EnzChemRED to create an end-to-end pipeline for knowledge extraction from text and apply this to abstracts at PubMed scale to create a draft map of enzyme functions in literature to guide curation efforts in UniProtKB and the reaction knowledgebase Rhea. The EnzChemRED corpus is freely available at https://ftp.expasy.org/databases/rhea/nlp/.

Title: Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction

• Authors: Zheye Deng, Chunkit Chan, Weiqi Wang, Yuxi Sun, Wei Fan, Tianshi Zheng, Yauwai Yim, Yangqiu Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14215
• Pdf URL: https://arxiv.org/pdf/2404.14215
• Copy Paste: [[2404.14215]] Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction(https://arxiv.org/abs/2404.14215)
• Keywords: extraction, large language model
• Abstract: The task of condensing large chunks of textual information into concise and structured tables has gained attention recently due to the emergence of Large Language Models (LLMs) and their potential benefit for downstream tasks, such as text summarization and text mining. Previous approaches often generate tables that directly replicate information from the text, limiting their applicability in broader contexts, as text-to-table generation in real-life scenarios necessitates information extraction, reasoning, and integration. However, there is a lack of both datasets and methodologies towards this task. In this paper, we introduce LiveSum, a new benchmark dataset created for generating summary tables of competitions based on real-time commentary texts. We evaluate the performances of state-of-the-art LLMs on this task in both fine-tuning and zero-shot settings, and additionally propose a novel pipeline called $T^3$(Text-Tuple-Table) to improve their performances. Extensive experimental results demonstrate that LLMs still struggle with this task even after fine-tuning, while our approach can offer substantial performance gains without explicit training. Further analyses demonstrate that our method exhibits strong generalization abilities, surpassing previous approaches on several other text-to-table datasets. Our code and data can be found at https://github.com/HKUST-KnowComp/LiveSum-TTT.

Title: Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

• Authors: Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, et al. (31 additional authors not shown)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14219
• Pdf URL: https://arxiv.org/pdf/2404.14219
• Copy Paste: [[2404.14219]] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone(https://arxiv.org/abs/2404.14219)
• Keywords: robust
• Abstract: We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench).

Title: MultiBooth: Towards Generating All Your Concepts in an Image from Text

• Authors: Chenyang Zhu, Kai Li, Yue Ma, Chunming He, Li Xiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14239
• Pdf URL: https://arxiv.org/pdf/2404.14239
• Copy Paste: [[2404.14239]] MultiBooth: Towards Generating All Your Concepts in an Image from Text(https://arxiv.org/abs/2404.14239)
• Keywords: diffusion
• Abstract: This paper introduces MultiBooth, a novel and efficient technique for multi-concept customization in image generation from text. Despite the significant advancements in customized generation methods, particularly with the success of diffusion models, existing methods often struggle with multi-concept scenarios due to low concept fidelity and high inference cost. MultiBooth addresses these issues by dividing the multi-concept generation process into two phases: a single-concept learning phase and a multi-concept integration phase. During the single-concept learning phase, we employ a multi-modal image encoder and an efficient concept encoding technique to learn a concise and discriminative representation for each concept. In the multi-concept integration phase, we use bounding boxes to define the generation area for each concept within the cross-attention map. This method enables the creation of individual concepts within their specified regions, thereby facilitating the formation of multi-concept images. This strategy not only improves concept fidelity but also reduces additional inference cost. MultiBooth surpasses various baselines in both qualitative and quantitative evaluations, showcasing its superior performance and computational efficiency. Project Page: https://multibooth.github.io/

Title: AI-Generated Faces in the Real World: A Large-Scale Case Study of Twitter Profile Images

• Authors: Jonas Ricker, Dennis Assenmacher, Thorsten Holz, Asja Fischer, Erwin Quiring
• Subjects: cs.CR, cs.AI, cs.CY, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2404.14244
• Pdf URL: https://arxiv.org/pdf/2404.14244
• Copy Paste: [[2404.14244]] AI-Generated Faces in the Real World: A Large-Scale Case Study of Twitter Profile Images(https://arxiv.org/abs/2404.14244)
• Keywords: generative
• Abstract: Recent advances in the field of generative artificial intelligence (AI) have blurred the lines between authentic and machine-generated content, making it almost impossible for humans to distinguish between such media. One notable consequence is the use of AI-generated images for fake profiles on social media. While several types of disinformation campaigns and similar incidents have been reported in the past, a systematic analysis has been lacking. In this work, we conduct the first large-scale investigation of the prevalence of AI-generated profile pictures on Twitter. We tackle the challenges of a real-world measurement study by carefully integrating various data sources and designing a multi-stage detection pipeline. Our analysis of nearly 15 million Twitter profile pictures shows that 0.052% were artificially generated, confirming their notable presence on the platform. We comprehensively examine the characteristics of these accounts and their tweet content, and uncover patterns of coordinated inauthentic behavior. The results also reveal several motives, including spamming and political amplification campaigns. Our research reaffirms the need for effective detection and mitigation strategies to cope with the potential negative effects of generative AI in the future.

Title: Chain of trust: Unraveling the references among Common Criteria certified products

• Authors: Adam Janovsky, Łukasz Chmielewski, Petr Svenda, Jan Jancar, Vashek Matyas
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.14246
• Pdf URL: https://arxiv.org/pdf/2404.14246
• Copy Paste: [[2404.14246]] Chain of trust: Unraveling the references among Common Criteria certified products(https://arxiv.org/abs/2404.14246)
• Keywords: security
• Abstract: With 5394 security certificates of IT products and systems, the Common Criteria for Information Technology Security Evaluation have bred an ecosystem entangled with various kind of relations between the certified products. Yet, the prevalence and nature of dependencies among Common Criteria certified products remains largely unexplored. This study devises a novel method for building the graph of references among the Common Criteria certified products, determining the different contexts of references with a supervised machine-learning algorithm, and measuring how often the references constitute actual dependencies between the certified products. With the help of the resulting reference graph, this work identifies just a dozen of certified components that are relied on by at least 10% of the whole ecosystem -- making them a prime target for malicious actors. The impact of their compromise is assessed and potentially problematic references to archived products are discussed.

Title: From Modalities to Styles: Rethinking the Domain Gap in Heterogeneous Face Recognition

• Authors: Anjith George, Sebastien Marcel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14247
• Pdf URL: https://arxiv.org/pdf/2404.14247
• Copy Paste: [[2404.14247]] From Modalities to Styles: Rethinking the Domain Gap in Heterogeneous Face Recognition(https://arxiv.org/abs/2404.14247)
• Keywords: robust
• Abstract: Heterogeneous Face Recognition (HFR) focuses on matching faces from different domains, for instance, thermal to visible images, making Face Recognition (FR) systems more versatile for challenging scenarios. However, the domain gap between these domains and the limited large-scale datasets in the target HFR modalities make it challenging to develop robust HFR models from scratch. In our work, we view different modalities as distinct styles and propose a method to modulate feature maps of the target modality to address the domain gap. We present a new Conditional Adaptive Instance Modulation (CAIM ) module that seamlessly fits into existing FR networks, turning them into HFR-ready systems. The CAIM block modulates intermediate feature maps, efficiently adapting to the style of the source modality and bridging the domain gap. Our method enables end-to-end training using a small set of paired samples. We extensively evaluate the proposed approach on various challenging HFR benchmarks, showing that it outperforms state-of-the-art methods. The source code and protocols for reproducing the findings will be made publicly available

Title: CLIP-GS: CLIP-Informed Gaussian Splatting for Real-time and View-consistent 3D Semantic Understanding

• Authors: Guibiao Liao, Jiankun Li, Zhenyu Bao, Xiaoqing Ye, Jingdong Wang, Qing Li, Kanglin Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14249
• Pdf URL: https://arxiv.org/pdf/2404.14249
• Copy Paste: [[2404.14249]] CLIP-GS: CLIP-Informed Gaussian Splatting for Real-time and View-consistent 3D Semantic Understanding(https://arxiv.org/abs/2404.14249)
• Keywords: robust, segmentation
• Abstract: The recent 3D Gaussian Splatting (GS) exhibits high-quality and real-time synthesis of novel views in 3D scenes. Currently, it primarily focuses on geometry and appearance modeling, while lacking the semantic understanding of scenes. To bridge this gap, we present CLIP-GS, which integrates semantics from Contrastive Language-Image Pre-Training (CLIP) into Gaussian Splatting to efficiently comprehend 3D environments without annotated semantic data. In specific, rather than straightforwardly learning and rendering high-dimensional semantic features of 3D Gaussians, which significantly diminishes the efficiency, we propose a Semantic Attribute Compactness (SAC) approach. SAC exploits the inherent unified semantics within objects to learn compact yet effective semantic representations of 3D Gaussians, enabling highly efficient rendering (>100 FPS). Additionally, to address the semantic ambiguity, caused by utilizing view-inconsistent 2D CLIP semantics to supervise Gaussians, we introduce a 3D Coherent Self-training (3DCS) strategy, resorting to the multi-view consistency originated from the 3D model. 3DCS imposes cross-view semantic consistency constraints by leveraging refined, self-predicted pseudo-labels derived from the trained 3D Gaussian model, thereby enhancing precise and view-consistent segmentation results. Extensive experiments demonstrate that our method remarkably outperforms existing state-of-the-art approaches, achieving improvements of 17.29% and 20.81% in mIoU metric on Replica and ScanNet datasets, respectively, while maintaining real-time rendering speed. Furthermore, our approach exhibits superior performance even with sparse input data, verifying the robustness of our method.

Title: Deep Learning as Ricci Flow

• Authors: Anthony Baptista, Alessandro Barp, Tapabrata Chakraborti, Chris Harbron, Ben D. MacArthur, Christopher R. S. Banerji
• Subjects: cs.LG, math.DG
• Abstract URL: https://arxiv.org/abs/2404.14265
• Pdf URL: https://arxiv.org/pdf/2404.14265
• Copy Paste: [[2404.14265]] Deep Learning as Ricci Flow(https://arxiv.org/abs/2404.14265)
• Keywords: explainability
• Abstract: Deep neural networks (DNNs) are powerful tools for approximating the distribution of complex data. It is known that data passing through a trained DNN classifier undergoes a series of geometric and topological simplifications. While some progress has been made toward understanding these transformations in neural networks with smooth activation functions, an understanding in the more general setting of non-smooth activation functions, such as the rectified linear unit (ReLU), which tend to perform better, is required. Here we propose that the geometric transformations performed by DNNs during classification tasks have parallels to those expected under Hamilton's Ricci flow - a tool from differential geometry that evolves a manifold by smoothing its curvature, in order to identify its topology. To illustrate this idea, we present a computational framework to quantify the geometric changes that occur as data passes through successive layers of a DNN, and use this framework to motivate a notion of `global Ricci network flow' that can be used to assess a DNN's ability to disentangle complex data geometries to solve classification problems. By training more than $1,500$ DNN classifiers of different widths and depths on synthetic and real-world data, we show that the strength of global Ricci network flow-like behaviour correlates with accuracy for well-trained DNNs, independently of depth, width and data set. Our findings motivate the use of tools from differential and discrete geometry to the problem of explainability in deep learning.

Title: What do Transformers Know about Government?

• Authors: Jue Hou, Anisia Katinskaia, Lari Kotilainen, Sathianpong Trangcasanchai, Anh-Duc Vu, Roman Yangarber
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14270
• Pdf URL: https://arxiv.org/pdf/2404.14270
• Copy Paste: [[2404.14270]] What do Transformers Know about Government?(https://arxiv.org/abs/2404.14270)
• Keywords: transformer
• Abstract: This paper investigates what insights about linguistic features and what knowledge about the structure of natural language can be obtained from the encodings in transformer language models.In particular, we explore how BERT encodes the government relation between constituents in a sentence. We use several probing classifiers, and data from two morphologically rich languages. Our experiments show that information about government is encoded across all transformer layers, but predominantly in the early layers of the model. We find that, for both languages, a small number of attention heads encode enough information about the government relations to enable us to train a classifier capable of discovering new, previously unknown types of government, never seen in the training data. Currently, data is lacking for the research community working on grammatical constructions, and government in particular. We release the Government Bank -- a dataset defining the government relations for thousands of lemmas in the languages in our experiments.

Title: Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagation

• Authors: Paulo Yanez Sarmiento, Simon Witzke, Nadja Klein, Bernhard Y. Renard
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14271
• Pdf URL: https://arxiv.org/pdf/2404.14271
• Copy Paste: [[2404.14271]] Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagation(https://arxiv.org/abs/2404.14271)
• Keywords: explainability
• Abstract: Explainability is a key component in many applications involving deep neural networks (DNNs). However, current explanation methods for DNNs commonly leave it to the human observer to distinguish relevant explanations from spurious noise. This is not feasible anymore when going from easily human-accessible data such as images to more complex data such as genome sequences. To facilitate the accessibility of DNN outputs from such complex data and to increase explainability, we present a modification of the widely used explanation method layer-wise relevance propagation. Our approach enforces sparsity directly by pruning the relevance propagation for the different layers. Thereby, we achieve sparser relevance attributions for the input features as well as for the intermediate layers. As the relevance propagation is input-specific, we aim to prune the relevance propagation rather than the underlying model architecture. This allows to prune different neurons for different inputs and hence, might be more appropriate to the local nature of explanation methods. To demonstrate the efficacy of our method, we evaluate it on two types of data, images and genomic sequences. We show that our modification indeed leads to noise reduction and concentrates relevance on the most important features compared to the baseline.

Title: RESFM: Robust Equivariant Multiview Structure from Motion

• Authors: Fadi Khatib, Yoni Kasten, Dror Moran, Meirav Galun, Ronen Basri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14280
• Pdf URL: https://arxiv.org/pdf/2404.14280
• Copy Paste: [[2404.14280]] RESFM: Robust Equivariant Multiview Structure from Motion(https://arxiv.org/abs/2404.14280)
• Keywords: robust
• Abstract: Multiview Structure from Motion is a fundamental and challenging computer vision problem. A recent deep-based approach was proposed utilizing matrix equivariant architectures for the simultaneous recovery of camera pose and 3D scene structure from large image collections. This work however made the unrealistic assumption that the point tracks given as input are clean of outliers. Here we propose an architecture suited to dealing with outliers by adding an inlier/outlier classifying module that respects the model equivariance and by adding a robust bundle adjustment step. Experiments demonstrate that our method can be successfully applied in realistic settings that include large image collections and point tracks extracted with common heuristics and include many outliers.

Title: A Survey on Efficient Inference for Large Language Models

• Authors: Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, Yu Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14294
• Pdf URL: https://arxiv.org/pdf/2404.14294
• Copy Paste: [[2404.14294]] A Survey on Efficient Inference for Large Language Models(https://arxiv.org/abs/2404.14294)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have attracted extensive attention due to their remarkable performance across various tasks. However, the substantial computational and memory requirements of LLM inference pose challenges for deployment in resource-constrained scenarios. Efforts within the field have been directed towards developing techniques aimed at enhancing the efficiency of LLM inference. This paper presents a comprehensive survey of the existing literature on efficient LLM inference. We start by analyzing the primary causes of the inefficient LLM inference, i.e., the large model size, the quadratic-complexity attention operation, and the auto-regressive decoding approach. Then, we introduce a comprehensive taxonomy that organizes the current literature into data-level, model-level, and system-level optimization. Moreover, the paper includes comparative experiments on representative methods within critical sub-fields to provide quantitative insights. Last but not least, we provide some knowledge summary and discuss future research directions.

Title: Marking: Visual Grading with Highlighting Errors and Annotating Missing Bits

• Authors: Shashank Sonkar, Naiming Liu, Debshila B. Mallick, Richard G. Baraniuk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14301
• Pdf URL: https://arxiv.org/pdf/2404.14301
• Copy Paste: [[2404.14301]] Marking: Visual Grading with Highlighting Errors and Annotating Missing Bits(https://arxiv.org/abs/2404.14301)
• Keywords: transformer
• Abstract: In this paper, we introduce "Marking", a novel grading task that enhances automated grading systems by performing an in-depth analysis of student responses and providing students with visual highlights. Unlike traditional systems that provide binary scores, "marking" identifies and categorizes segments of the student response as correct, incorrect, or irrelevant and detects omissions from gold answers. We introduce a new dataset meticulously curated by Subject Matter Experts specifically for this task. We frame "Marking" as an extension of the Natural Language Inference (NLI) task, which is extensively explored in the field of Natural Language Processing. The gold answer and the student response play the roles of premise and hypothesis in NLI, respectively. We subsequently train language models to identify entailment, contradiction, and neutrality from student response, akin to NLI, and with the added dimension of identifying omissions from gold answers. Our experimental setup involves the use of transformer models, specifically BERT and RoBERTa, and an intelligent training step using the e-SNLI dataset. We present extensive baseline results highlighting the complexity of the "Marking" task, which sets a clear trajectory for the upcoming study. Our work not only opens up new avenues for research in AI-powered educational assessment tools, but also provides a valuable benchmark for the AI in education community to engage with and improve upon in the future. The code and dataset can be found at https://github.com/luffycodes/marking.

Title: Towards Better Adversarial Purification via Adversarial Denoising Diffusion Training

• Authors: Yiming Liu, Kezhao Liu, Yao Xiao, Ziyi Dong, Xiaogang Xu, Pengxu Wei, Liang Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14309
• Pdf URL: https://arxiv.org/pdf/2404.14309
• Copy Paste: [[2404.14309]] Towards Better Adversarial Purification via Adversarial Denoising Diffusion Training(https://arxiv.org/abs/2404.14309)
• Keywords: attack, robust, diffusion
• Abstract: Recently, diffusion-based purification (DBP) has emerged as a promising approach for defending against adversarial attacks. However, previous studies have used questionable methods to evaluate the robustness of DBP models, their explanations of DBP robustness also lack experimental support. We re-examine DBP robustness using precise gradient, and discuss the impact of stochasticity on DBP robustness. To better explain DBP robustness, we assess DBP robustness under a novel attack setting, Deterministic White-box, and pinpoint stochasticity as the main factor in DBP robustness. Our results suggest that DBP models rely on stochasticity to evade the most effective attack direction, rather than directly countering adversarial perturbations. To improve the robustness of DBP models, we propose Adversarial Denoising Diffusion Training (ADDT). This technique uses Classifier-Guided Perturbation Optimization (CGPO) to generate adversarial perturbation through guidance from a pre-trained classifier, and uses Rank-Based Gaussian Mapping (RBGM) to convert adversarial pertubation into a normal Gaussian distribution. Empirical results show that ADDT improves the robustness of DBP models. Further experiments confirm that ADDT equips DBP models with the ability to directly counter adversarial perturbations.

Title: Automated Long Answer Grading with RiceChem Dataset

• Authors: Shashank Sonkar, Kangqi Ni, Lesa Tran Lu, Kristi Kincaid, John S. Hutchinson, Richard G. Baraniuk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14316
• Pdf URL: https://arxiv.org/pdf/2404.14316
• Copy Paste: [[2404.14316]] Automated Long Answer Grading with RiceChem Dataset(https://arxiv.org/abs/2404.14316)
• Keywords: large language model
• Abstract: We introduce a new area of study in the field of educational Natural Language Processing: Automated Long Answer Grading (ALAG). Distinguishing itself from Automated Short Answer Grading (ASAG) and Automated Essay Grading (AEG), ALAG presents unique challenges due to the complexity and multifaceted nature of fact-based long answers. To study ALAG, we introduce RiceChem, a dataset derived from a college chemistry course, featuring real student responses to long-answer questions with an average word count notably higher than typical ASAG datasets. We propose a novel approach to ALAG by formulating it as a rubric entailment problem, employing natural language inference models to verify whether each criterion, represented by a rubric item, is addressed in the student's response. This formulation enables the effective use of MNLI for transfer learning, significantly improving the performance of models on the RiceChem dataset. We demonstrate the importance of rubric-based formulation in ALAG, showcasing its superiority over traditional score-based approaches in capturing the nuances of student responses. We also investigate the performance of models in cold start scenarios, providing valuable insights into the practical deployment considerations in educational settings. Lastly, we benchmark state-of-the-art open-sourced Large Language Models (LLMs) on RiceChem and compare their results to GPT models, highlighting the increased complexity of ALAG compared to ASAG. Despite leveraging the benefits of a rubric-based approach and transfer learning from MNLI, the lower performance of LLMs on RiceChem underscores the significant difficulty posed by the ALAG task. With this work, we offer a fresh perspective on grading long, fact-based answers and introduce a new dataset to stimulate further research in this important area. Code: \url{https://github.com/luffycodes/Automated-Long-Answer-Grading}.

Title: Machine Learning Techniques for MRI Data Processing at Expanding Scale

• Authors: Taro Langner
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14326
• Pdf URL: https://arxiv.org/pdf/2404.14326
• Copy Paste: [[2404.14326]] Machine Learning Techniques for MRI Data Processing at Expanding Scale(https://arxiv.org/abs/2404.14326)
• Keywords: secure, federate
• Abstract: Imaging sites around the world generate growing amounts of medical scan data with ever more versatile and affordable technology. Large-scale studies acquire MRI for tens of thousands of participants, together with metadata ranging from lifestyle questionnaires to biochemical assays, genetic analyses and more. These large datasets encode substantial information about human health and hold considerable potential for machine learning training and analysis. This chapter examines ongoing large-scale studies and the challenge of distribution shifts between them. Transfer learning for overcoming such shifts is discussed, together with federated learning for safe access to distributed training data securely held at multiple institutions. Finally, representation learning is reviewed as a methodology for encoding embeddings that express abstract relationships in multi-modal input formats.

Title: X-Ray: A Sequential 3D Representation for Generation

• Authors: Tao Hu, Wenhang Ge, Yuyang Zhao, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14329
• Pdf URL: https://arxiv.org/pdf/2404.14329
• Copy Paste: [[2404.14329]] X-Ray: A Sequential 3D Representation for Generation(https://arxiv.org/abs/2404.14329)
• Keywords: diffusion
• Abstract: In this paper, we introduce X-Ray, an innovative approach to 3D generation that employs a new sequential representation, drawing inspiration from the depth-revealing capabilities of X-Ray scans to meticulously capture both the external and internal features of objects. Central to our method is the utilization of ray casting techniques originating from the camera's viewpoint, meticulously recording the geometric and textural details encountered across all intersected surfaces. This process efficiently condenses complete objects or scenes into a multi-frame format, just like videos. Such a structure ensures the 3D representation is composed solely of critical surface information. Highlighting the practicality and adaptability of our X-Ray representation, we showcase its utility in synthesizing 3D objects, employing a network architecture akin to that used in video diffusion models. The outcomes reveal our representation's superior performance in enhancing both the accuracy and efficiency of 3D synthesis, heralding new directions for ongoing research and practical implementations in the field.

Title: Automatic Discovery of Visual Circuits

• Authors: Achyuta Rajaram, Neil Chowdhury, Antonio Torralba, Jacob Andreas, Sarah Schwettmann
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14349
• Pdf URL: https://arxiv.org/pdf/2404.14349
• Copy Paste: [[2404.14349]] Automatic Discovery of Visual Circuits(https://arxiv.org/abs/2404.14349)
• Keywords: attack
• Abstract: To date, most discoveries of network subcomponents that implement human-interpretable computations in deep vision models have involved close study of single units and large amounts of human labor. We explore scalable methods for extracting the subgraph of a vision model's computational graph that underlies recognition of a specific visual concept. We introduce a new method for identifying these subgraphs: specifying a visual concept using a few examples, and then tracing the interdependence of neuron activations across layers, or their functional connectivity. We find that our approach extracts circuits that causally affect model output, and that editing these circuits can defend large pretrained models from adversarial attacks.

Title: Calc-CMU at SemEval-2024 Task 7: Pre-Calc -- Learning to Use the Calculator Improves Numeracy in Language Models

• Authors: Vishruth Veerendranath, Vishwa Shah, Kshitish Ghate
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14355
• Pdf URL: https://arxiv.org/pdf/2404.14355
• Copy Paste: [[2404.14355]] Calc-CMU at SemEval-2024 Task 7: Pre-Calc -- Learning to Use the Calculator Improves Numeracy in Language Models(https://arxiv.org/abs/2404.14355)
• Keywords: generative
• Abstract: Quantitative and numerical comprehension in language is an important task in many fields like education and finance, but still remains a challenging task for language models. While tool and calculator usage has shown to be helpful to improve mathematical reasoning in large pretrained decoder-only language models, this remains unexplored for smaller language models with encoders. In this paper, we propose Pre-Calc, a simple pre-finetuning objective of learning to use the calculator for both encoder-only and encoder-decoder architectures, formulated as a discriminative and generative task respectively. We pre-train BERT and RoBERTa for discriminative calculator use and Flan-T5 for generative calculator use on the MAWPS, SVAMP, and AsDiv-A datasets, which improves performance on downstream tasks that require numerical understanding. Our code and data are available at https://github.com/calc-cmu/pre-calc.

Title: Better Synthetic Data by Retrieving and Transforming Existing Datasets

• Authors: Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, Graham Neubig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.14361
• Pdf URL: https://arxiv.org/pdf/2404.14361
• Copy Paste: [[2404.14361]] Better Synthetic Data by Retrieving and Transforming Existing Datasets(https://arxiv.org/abs/2404.14361)
• Keywords: large language model
• Abstract: Despite recent advances in large language models, building dependable and deployable NLP models typically requires abundant, high-quality training data. However, task-specific data is not available for many use cases, and manually curating task-specific data is labor-intensive. Recent work has studied prompt-driven synthetic data generation using large language models, but these generated datasets tend to lack complexity and diversity. To address these limitations, we introduce a method, \textit{DataTune}, to make better use of existing, publicly available datasets to improve automatic dataset generation. DataTune performs dataset transformation, enabling the repurposing of publicly available datasets into a format that is directly aligned with the specific requirements of target tasks. On a diverse set of language-based tasks from the BIG-Bench benchmark, we find that finetuning language models via DataTune improves over a few-shot prompting baseline by 49\% and improves over existing methods that use synthetic or retrieved training data by 34\%. We find that dataset transformation significantly increases the diversity and difficulty of generated data on many tasks. We integrate DataTune into an open-source repository to make this method accessible to the community: https://github.com/neulab/prompt2model.

Title: Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

• Authors: Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, Aviral Kumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14367
• Pdf URL: https://arxiv.org/pdf/2404.14367
• Copy Paste: [[2404.14367]] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data(https://arxiv.org/abs/2404.14367)
• Keywords: large language model
• Abstract: Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning. Different methods come with different implementation tradeoffs and performance differences, and existing empirical findings present different conclusions, for instance, some results show that online RL is quite important to attain good fine-tuning results, while others find (offline) contrastive or even purely supervised methods sufficient. This raises a natural question: what kind of approaches are important for fine-tuning with preference data and why? In this paper, we answer this question by performing a rigorous analysis of a number of fine-tuning techniques on didactic and full-scale LLM problems. Our main finding is that, in general, approaches that use on-policy sampling or attempt to push down the likelihood on certain responses (i.e., employ a "negative gradient") outperform offline and maximum likelihood objectives. We conceptualize our insights and unify methods that use on-policy sampling or negative gradient under a notion of mode-seeking objectives for categorical distributions. Mode-seeking objectives are able to alter probability mass on specific bins of a categorical distribution at a fast rate compared to maximum likelihood, allowing them to relocate masses across bins more effectively. Our analysis prescribes actionable insights for preference fine-tuning of LLMs and informs how data should be collected for maximal improvement.

Title: Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph

• Authors: Xiaochen Kev Gao, Feng Yao, Kewen Zhao, Beilei He, Animesh Kumar, Vish Krishnan, Jingbo Shang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14372
• Pdf URL: https://arxiv.org/pdf/2404.14372
• Copy Paste: [[2404.14372]] Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph(https://arxiv.org/abs/2404.14372)
• Keywords: large language model
• Abstract: Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval pre-diction task and unveil that simple domain-specific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. Specifically, we first extend the embedding-based state-of-the-art (SOTA) by scaling up its backbone model with various sizes of open-source LLMs, then explore prompt-based methods to harness proprietary LLMs' potential, but find the best results close to random guessing, underlining the ineffectiveness of model scaling-up. Hence, we propose a novel Fine-grained cLAim depeNdency (FLAN) Graph through meticulous patent data analyses, capturing the inherent dependencies across segments of the patent text. As it is model-agnostic, we apply cost-effective graph models to our FLAN Graph to obtain representations for approval prediction. Extensive experiments and detailed analyses prove that incorporating FLAN Graph via various graph models consistently outperforms all LLM baselines significantly. We hope that our observations and analyses in this paper can bring more attention to this challenging task and prompt further research into the limitations of LLMs. Our source code and dataset can be obtained from this http URL

Title: TAVGBench: Benchmarking Text to Audible-Video Generation

• Authors: Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, Yuchao Dai
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.14381
• Pdf URL: https://arxiv.org/pdf/2404.14381
• Copy Paste: [[2404.14381]] TAVGBench: Benchmarking Text to Audible-Video Generation(https://arxiv.org/abs/2404.14381)
• Keywords: diffusion
• Abstract: The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both audio and video elements. To support research in this field, we have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1.7 million clips with a total duration of 11.8 thousand hours. We propose an automatic annotation pipeline to ensure each audible video has detailed descriptions for both its audio and video contents. We also introduce the Audio-Visual Harmoni score (AVHScore) to provide a quantitative measure of the alignment between the generated audio and video modalities. Additionally, we present a baseline model for TAVG called TAVDiffusion, which uses a two-stream latent diffusion model to provide a fundamental starting point for further research in this area. We achieve the alignment of audio and video by employing cross-attention and contrastive learning. Through extensive experiments and evaluations on TAVGBench, we demonstrate the effectiveness of our proposed model under both conventional metrics and our proposed metrics.

Title: A Survey on Self-Evolution of Large Language Models

• Authors: Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, Jingren Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.14387
• Pdf URL: https://arxiv.org/pdf/2404.14387
• Copy Paste: [[2404.14387]] A Survey on Self-Evolution of Large Language Models(https://arxiv.org/abs/2404.14387)
• Keywords: large language model
• Abstract: Large language models (LLMs) have significantly advanced in various fields and intelligent agent applications. However, current LLMs that learn from human or external model supervision are costly and may face performance ceilings as task complexity and diversity increase. To address this issue, self-evolution approaches that enable LLM to autonomously acquire, refine, and learn from experiences generated by the model itself are rapidly growing. This new training paradigm inspired by the human experiential learning process offers the potential to scale LLMs towards superintelligence. In this work, we present a comprehensive survey of self-evolution approaches in LLMs. We first propose a conceptual framework for self-evolution and outline the evolving process as iterative cycles composed of four phases: experience acquisition, experience refinement, updating, and evaluation. Second, we categorize the evolution objectives of LLMs and LLM-based agents; then, we summarize the literature and provide taxonomy and insights for each module. Lastly, we pinpoint existing challenges and propose future directions to improve self-evolution frameworks, equipping researchers with critical insights to fast-track the development of self-evolving LLMs.

Title: RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?

• Authors: Adrian de Wynter, Ishaan Watts, Nektar Ege Altıntoprak, Tua Wongsangaroonsri, Minghui Zhang, Noura Farra, Lena Baur, Samantha Claudet, Pavel Gajdusek, Can Gören, Qilong Gu, Anna Kaminska, Tomasz Kaminski, Ruby Kuo, Akiko Kyuba, Jongho Lee, Kartik Mathur, Petter Merok, Ivana Milovanović, Nani Paananen, Vesa-Matti Paananen, Anna Pavlenko, Bruno Pereira Vidal, Luciano Strika, Yueh Tsao, Davide Turcato, Oleksandr Vakhno, Judit Velcsov, Anna Vickers, Stéphanie Visser, Herdyan Widarmanto, Andrey Zaikin, Si-Qing Chen
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14397
• Pdf URL: https://arxiv.org/pdf/2404.14397
• Copy Paste: [[2404.14397]] RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?(https://arxiv.org/abs/2404.14397)
• Keywords: large language model
• Abstract: Large language models (LLMs) and small language models (SLMs) are being adopted at remarkable speed, although their safety still remains a serious concern. With the advent of multilingual S/LLMs, the question now becomes a matter of scale: can we expand multilingual safety evaluations of these models with the same velocity at which they are deployed? To this end we introduce RTP-LX, a human-transcreated and human-annotated corpus of toxic prompts and outputs in 28 languages. RTP-LX follows participatory design practices, and a portion of the corpus is especially designed to detect culturally-specific toxic language. We evaluate seven S/LLMs on their ability to detect toxic content in a culturally-sensitive, multilingual scenario. We find that, although they typically score acceptably in terms of accuracy, they have low agreement with human judges when judging holistically the toxicity of a prompt, and have difficulty discerning harm in context-dependent scenarios, particularly with subtle-yet-harmful content (e.g. microagressions, bias). We release of this dataset to contribute to further reduce harmful uses of these models and improve their safe deployment.

Title: GeoDiffuser: Geometry-Based Image Editing with Diffusion Models

• Authors: Rahul Sajnani, Jeroen Vanbaar, Jie Min, Kapil Katyal, Srinath Sridhar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14403
• Pdf URL: https://arxiv.org/pdf/2404.14403
• Copy Paste: [[2404.14403]] GeoDiffuser: Geometry-Based Image Editing with Diffusion Models(https://arxiv.org/abs/2404.14403)
• Keywords: diffusion, generative
• Abstract: The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However, these methods are bespoke, imprecise, require additional information, or are limited to only 2D image edits. We present GeoDiffuser, a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images, for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally located. Given a natural image and user input, we segment the foreground object using SAM and estimate a corresponding transform which is used by our optimization approach for editing. GeoDiffuser can perform common 2D and 3D edits like object translation, 3D rotation, and removal. We present quantitative results, including a perceptual study, that shows how our approach is better than existing methods. Visit https://ivl.cs.brown.edu/research/geodiffuser.html for more information.

Title: Hyp-OC: Hyperbolic One Class Classification for Face Anti-Spoofing

• Authors: Kartik Narayan, Vishal M. Patel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14406
• Pdf URL: https://arxiv.org/pdf/2404.14406
• Copy Paste: [[2404.14406]] Hyp-OC: Hyperbolic One Class Classification for Face Anti-Spoofing(https://arxiv.org/abs/2404.14406)
• Keywords: security, attack
• Abstract: Face recognition technology has become an integral part of modern security systems and user authentication processes. However, these systems are vulnerable to spoofing attacks and can easily be circumvented. Most prior research in face anti-spoofing (FAS) approaches it as a two-class classification task where models are trained on real samples and known spoof attacks and tested for detection performance on unknown spoof attacks. However, in practice, FAS should be treated as a one-class classification task where, while training, one cannot assume any knowledge regarding the spoof samples a priori. In this paper, we reformulate the face anti-spoofing task from a one-class perspective and propose a novel hyperbolic one-class classification framework. To train our network, we use a pseudo-negative class sampled from the Gaussian distribution with a weighted running mean and propose two novel loss functions: (1) Hyp-PC: Hyperbolic Pairwise Confusion loss, and (2) Hyp-CE: Hyperbolic Cross Entropy loss, which operate in the hyperbolic space. Additionally, we employ Euclidean feature clipping and gradient clipping to stabilize the training in the hyperbolic space. To the best of our knowledge, this is the first work extending hyperbolic embeddings for face anti-spoofing in a one-class manner. With extensive experiments on five benchmark datasets: Rose-Youtu, MSU-MFSD, CASIA-MFSD, Idiap Replay-Attack, and OULU-NPU, we demonstrate that our method significantly outperforms the state-of-the-art, achieving better spoof detection performance.

Title: SpaceByte: Towards Deleting Tokenization from Large Language Modeling

• Authors: Kevin Slagle
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.14408
• Pdf URL: https://arxiv.org/pdf/2404.14408
• Copy Paste: [[2404.14408]] SpaceByte: Towards Deleting Tokenization from Large Language Modeling(https://arxiv.org/abs/2404.14408)
• Keywords: transformer, large language model
• Abstract: Tokenization is widely used in large language models because it significantly improves performance. However, tokenization imposes several disadvantages, such as performance biases, increased adversarial vulnerability, decreased character-level modeling performance, and increased modeling complexity. To address these disadvantages without sacrificing performance, we propose SpaceByte, a novel byte-level decoder architecture that closes the performance gap between byte-level and subword autoregressive language modeling. SpaceByte consists of a byte-level Transformer model, but with extra larger transformer blocks inserted in the middle of the layers. We find that performance is significantly improved by applying these larger blocks only after certain bytes, such as space characters, which typically denote word boundaries. Our experiments show that for a fixed training and inference compute budget, SpaceByte outperforms other byte-level architectures and roughly matches the performance of tokenized Transformer architectures.

Title: Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses

• Authors: Inhee Lee, Byungjun Kim, Hanbyul Joo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14410
• Pdf URL: https://arxiv.org/pdf/2404.14410
• Copy Paste: [[2404.14410]] Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses(https://arxiv.org/abs/2404.14410)
• Keywords: diffusion
• Abstract: In this paper, we present a method to reconstruct the world and multiple dynamic humans in 3D from a monocular video input. As a key idea, we represent both the world and multiple humans via the recently emerging 3D Gaussian Splatting (3D-GS) representation, enabling to conveniently and efficiently compose and render them together. In particular, we address the scenarios with severely limited and sparse observations in 3D human reconstruction, a common challenge encountered in the real world. To tackle this challenge, we introduce a novel approach to optimize the 3D-GS representation in a canonical space by fusing the sparse cues in the common space, where we leverage a pre-trained 2D diffusion model to synthesize unseen views while keeping the consistency with the observed 2D appearances. We demonstrate our method can reconstruct high-quality animatable 3D humans in various challenging examples, in the presence of occlusion, image crops, few-shot, and extremely sparse observations. After reconstruction, our method is capable of not only rendering the scene in any novel views at arbitrary time instances, but also editing the 3D scene by removing individual humans or applying different motions for each human. Through various experiments, we demonstrate the quality and efficiency of our methods over alternative existing approaches.

Title: AutoAD III: The Prequel -- Back to the Pixels

• Authors: Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, Andrew Zisserman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.14412
• Pdf URL: https://arxiv.org/pdf/2404.14412
• Copy Paste: [[2404.14412]] AutoAD III: The Prequel -- Back to the Pixels(https://arxiv.org/abs/2404.14412)
• Keywords: large language model
• Abstract: Generating Audio Description (AD) for movies is a challenging task that requires fine-grained visual understanding and an awareness of the characters and their names. Currently, visual language models for AD generation are limited by a lack of suitable training data, and also their evaluation is hampered by using performance measures not specialized to the AD domain. In this paper, we make three contributions: (i) We propose two approaches for constructing AD datasets with aligned video data, and build training and evaluation datasets using these. These datasets will be publicly released; (ii) We develop a Q-former-based architecture which ingests raw video and generates AD, using frozen pre-trained visual encoders and large language models; and (iii) We provide new evaluation metrics to benchmark AD quality that are well-matched to human performance. Taken together, we improve the state of the art on AD generation.