2024-03-18

Title: On Unsupervised Image-to-image translation and GAN stability

• Authors: BahaaEddin AlAila, Zahra Jandaghi, Abolfazl Farahani, Mohammad Ziad Al-Saad
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.09646
• Pdf URL: https://arxiv.org/pdf/2403.09646
• Copy Paste: [[2403.09646]] On Unsupervised Image-to-image translation and GAN stability(https://arxiv.org/abs/2403.09646)
• Keywords: generative, segmentation
• Abstract: The problem of image-to-image translation is one that is intruiging and challenging at the same time, for the impact potential it can have on a wide variety of other computer vision applications like colorization, inpainting, segmentation and others. Given the high-level of sophistication needed to extract patterns from one domain and successfully applying them to another, especially, in a completely unsupervised (unpaired) manner, this problem has gained much attention as of the last few years. It is one of the first problems where successful applications to deep generative models, and especially Generative Adversarial Networks achieved astounding results that are actually of realworld impact, rather than just a show of theoretical prowess; the such that has been dominating the GAN world. In this work, we study some of the failure cases of a seminal work in the field, CycleGAN [1] and hypothesize that they are GAN-stability related, and propose two general models to try to alleviate these problems. We also reach the same conclusion of the problem being ill-posed that has been also circulating in the literature lately.

Title: Precision Agriculture: Crop Mapping using Machine Learning and Sentinel-2 Satellite Imagery

• Authors: Kui Zhao, Siyang Wu, Chang Liu, Yue Wu, Natalia Efremova
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.09651
• Pdf URL: https://arxiv.org/pdf/2403.09651
• Copy Paste: [[2403.09651]] Precision Agriculture: Crop Mapping using Machine Learning and Sentinel-2 Satellite Imagery(https://arxiv.org/abs/2403.09651)
• Keywords: security
• Abstract: Food security has grown in significance due to the changing climate and its warming effects. To support the rising demand for agricultural products and to minimize the negative impact of climate change and mass cultivation, precision agriculture has become increasingly important for crop cultivation. This study employs deep learning and pixel-based machine learning methods to accurately segment lavender fields for precision agriculture, utilizing various spectral band combinations extracted from Sentinel-2 satellite imagery. Our fine-tuned final model, a U-Net architecture, can achieve a Dice coefficient of 0.8324. Additionally, our investigation highlights the unexpected efficacy of the pixel-based method and the RGB spectral band combination in this task.

Title: STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models

• Authors: Pum Jun Kim, Seojun Kim, Jaejun Yoo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09669
• Pdf URL: https://arxiv.org/pdf/2403.09669
• Copy Paste: [[2403.09669]] STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models(https://arxiv.org/abs/2403.09669)
• Keywords: generative
• Abstract: Image generative models have made significant progress in generating realistic and diverse images, supported by comprehensive guidance from various evaluation metrics. However, current video generative models struggle to generate even short video clips, with limited tools that provide insights for improvements. Current video evaluation metrics are simple adaptations of image metrics by switching the embeddings with video embedding networks, which may underestimate the unique characteristics of video. Our analysis reveals that the widely used Frechet Video Distance (FVD) has a stronger emphasis on the spatial aspect than the temporal naturalness of video and is inherently constrained by the input size of the embedding networks used, limiting it to 16 frames. Additionally, it demonstrates considerable instability and diverges from human evaluations. To address the limitations, we propose STREAM, a new video evaluation metric uniquely designed to independently evaluate spatial and temporal aspects. This feature allows comprehensive analysis and evaluation of video generative models from various perspectives, unconstrained by video length. We provide analytical and experimental evidence demonstrating that STREAM provides an effective evaluation tool for both visual and temporal quality of videos, offering insights into area of improvement for video generative models. To the best of our knowledge, STREAM is the first evaluation metric that can separately assess the temporal and spatial aspects of videos. Our code is available at STREAM.

Title: Navigating the Peril of Generated Alternative Facts: A ChatGPT-4 Fabricated Omega Variant Case as a Cautionary Tale in Medical Misinformation

• Authors: Malik Sallam, Jan Egger, Rainer Roehrig, Behrus Puladi
• Subjects: cs.CL, cs.CY, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09674
• Pdf URL: https://arxiv.org/pdf/2403.09674
• Copy Paste: [[2403.09674]] Navigating the Peril of Generated Alternative Facts: A ChatGPT-4 Fabricated Omega Variant Case as a Cautionary Tale in Medical Misinformation(https://arxiv.org/abs/2403.09674)
• Keywords: large language model
• Abstract: In an era where artificial intelligence (AI) intertwines with medical research, the delineation of truth becomes increasingly complex. This study ostensibly examines a purported novel SARS-CoV-2 variant, dubbed the Omega variant, showcasing 31 unique mutations in the S gene region. However, the real undercurrent of this narrative is a demonstration of the ease with which AI, specifically ChatGPT-4, can fabricate convincing yet entirely fictional scientific data. The so-called Omega variant was identified in a fully vaccinated, previously infected 35-year-old male presenting with severe COVID-19 symptoms. Through a detailed, albeit artificial, genomic analysis and contact tracing, this study mirrors the rigorous methodology of genuine case reports, thereby setting the stage for a compelling but entirely constructed narrative. The entire case study was generated by ChatGPT-4, a large language model by OpenAI. The fabricated Omega variant features an ensemble of mutations, including N501Y and E484K, known for enhancing ACE2 receptor affinity, alongside L452R and P681H, ostensibly indicative of immune evasion. This variant's contrived interaction dynamics - severe symptoms in a vaccinated individual versus mild ones in unvaccinated contacts - were designed to mimic real-world complexities, including suggestions of antibody-dependent enhancement (ADE). While the Omega variant is a product of AI-generated fiction, the implications of this exercise are real and profound. The ease with which AI can generate believable but false scientific information, as illustrated in this case, raises significant concerns about the potential for misinformation in medicine. This study, therefore, serves as a cautionary tale, emphasizing the necessity for critical evaluation of sources, especially in an age where AI tools like ChatGPT are becoming increasingly sophisticated and widespread in their use.

Title: Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases

• Authors: Rio Aguina-Kang, Maxim Gumin, Do Heon Han, Stewart Morris, Seung Jean Yoo, Aditya Ganeshan, R. Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, Daniel Ritchie
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.09675
• Pdf URL: https://arxiv.org/pdf/2403.09675
• Copy Paste: [[2403.09675]] Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases(https://arxiv.org/abs/2403.09675)
• Keywords: generative, large language model
• Abstract: We present a system for generating indoor scenes in response to text prompts. The prompts are not limited to a fixed vocabulary of scene descriptions, and the objects in generated scenes are not restricted to a fixed set of object categories -- we call this setting indoor scene generation. Unlike most prior work on indoor scene generation, our system does not require a large training dataset of existing 3D scenes. Instead, it leverages the world knowledge encoded in pre-trained large language models (LLMs) to synthesize programs in a domain-specific layout language that describe objects and spatial relations between them. Executing such a program produces a specification of a constraint satisfaction problem, which the system solves using a gradient-based optimization scheme to produce object positions and orientations. To produce object geometry, the system retrieves 3D meshes from a database. Unlike prior work which uses databases of category-annotated, mutually-aligned meshes, we develop a pipeline using vision-language models (VLMs) to retrieve meshes from massive databases of un-annotated, inconsistently-aligned meshes. Experimental evaluations show that our system outperforms generative models trained on 3D data for traditional, closed-universe scene generation tasks; it also outperforms a recent LLM-based layout generation method on open-universe scene generation.

Title: Unmasking the Shadows of AI: Investigating Deceptive Capabilities in Large Language Models

• Authors: Linge Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09676
• Pdf URL: https://arxiv.org/pdf/2403.09676
• Copy Paste: [[2403.09676]] Unmasking the Shadows of AI: Investigating Deceptive Capabilities in Large Language Models(https://arxiv.org/abs/2403.09676)
• Keywords: large language model
• Abstract: This research critically navigates the intricate landscape of AI deception, concentrating on deceptive behaviours of Large Language Models (LLMs). My objective is to elucidate this issue, examine the discourse surrounding it, and subsequently delve into its categorization and ramifications. The essay initiates with an evaluation of the AI Safety Summit 2023 (ASS) and introduction of LLMs, emphasising multidimensional biases that underlie their deceptive behaviours.The literature review covers four types of deception categorised: Strategic deception, Imitation, Sycophancy, and Unfaithful Reasoning, along with the social implications and risks they entail. Lastly, I take an evaluative stance on various aspects related to navigating the persistent challenges of the deceptive AI. This encompasses considerations of international collaborative governance, the reconfigured engagement of individuals with AI, proposal of practical adjustments, and specific elements of digital education.

Title: ViT-MUL: A Baseline Study on Recent Machine Unlearning Methods Applied to Vision Transformers

• Authors: Ikhyun Cho, Changyeon Park, Julia Hockenmaier
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09681
• Pdf URL: https://arxiv.org/pdf/2403.09681
• Copy Paste: [[2403.09681]] ViT-MUL: A Baseline Study on Recent Machine Unlearning Methods Applied to Vision Transformers(https://arxiv.org/abs/2403.09681)
• Keywords: transformer
• Abstract: Machine unlearning (MUL) is an arising field in machine learning that seeks to erase the learned information of specific training data points from a trained model. Despite the recent active research in MUL within computer vision, the majority of work has focused on ResNet-based models. Given that Vision Transformers (ViT) have become the predominant model architecture, a detailed study of MUL specifically tailored to ViT is essential. In this paper, we present comprehensive experiments on ViTs using recent MUL algorithms and datasets. We anticipate that our experiments, ablation studies, and findings could provide valuable insights and inspire further research in this field.

Title: Counterfactual Image Editing

• Authors: Yushu Pan, Elias Bareinboim
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09683
• Pdf URL: https://arxiv.org/pdf/2403.09683
• Copy Paste: [[2403.09683]] Counterfactual Image Editing(https://arxiv.org/abs/2403.09683)
• Keywords: generative
• Abstract: Counterfactual image editing is an important task in generative AI, which asks how an image would look if certain features were different. The current literature on the topic focuses primarily on changing individual features while remaining silent about the causal relationships between these features, as present in the real world. In this paper, we formalize the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images through a special type of model called augmented structural causal models (ASCMs). Second, we show two fundamental impossibility results: (1) counterfactual editing is impossible from i.i.d. image samples and their corresponding labels alone; (2) even when the causal relationships between the latent generative factors and images are available, no guarantees regarding the output of the model can be provided. Third, we propose a relaxation for this challenging problem by approximating non-identifiable counterfactual distributions with a new family of counterfactual-consistent estimators. This family exhibits the desirable property of preserving features that the user cares about across both factual and counterfactual worlds. Finally, we develop an efficient algorithm to generate counterfactual images by leveraging neural causal models.

Title: Shapley Values-Powered Framework for Fair Reward Split in Content Produced by GenAI

• Authors: Alex Glinsky, Alexey Sokolsky
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09700
• Pdf URL: https://arxiv.org/pdf/2403.09700
• Copy Paste: [[2403.09700]] Shapley Values-Powered Framework for Fair Reward Split in Content Produced by GenAI(https://arxiv.org/abs/2403.09700)
• Keywords: fair, diffusion, generative
• Abstract: It is evident that, currently, generative models are surpassed in quality by human professionals. However, with the advancements in Artificial Intelligence, this gap will narrow, leading to scenarios where individuals who have dedicated years of their lives to mastering a skill become obsolete due to their high costs, which are inherently linked to the time they require to complete a task -- a task that AI could accomplish in minutes or seconds. To avoid future social upheavals, we must, even now, contemplate how to fairly assess the contributions of such individuals in training generative models and how to compensate them for the reduction or complete loss of their incomes. In this work, we propose a method to structure collaboration between model developers and data providers. To achieve this, we employ Shapley Values to quantify the contribution of artist(s) in an image generated by the Stable Diffusion-v1.5 model and to equitably allocate the reward among them.

Title: Generator-Guided Crowd Reaction Assessment

• Authors: Sohom Ghosh, Chung-Chi Chen, Sudip Kumar Naskar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09702
• Pdf URL: https://arxiv.org/pdf/2403.09702
• Copy Paste: [[2403.09702]] Generator-Guided Crowd Reaction Assessment(https://arxiv.org/abs/2403.09702)
• Keywords: generative, large language model
• Abstract: In the realm of social media, understanding and predicting post reach is a significant challenge. This paper presents a Crowd Reaction AssessMent (CReAM) task designed to estimate if a given social media post will receive more reaction than another, a particularly essential task for digital marketers and content writers. We introduce the Crowd Reaction Estimation Dataset (CRED), consisting of pairs of tweets from The White House with comparative measures of retweet count. The proposed Generator-Guided Estimation Approach (GGEA) leverages generative Large Language Models (LLMs), such as ChatGPT, FLAN-UL2, and Claude, to guide classification models for making better predictions. Our results reveal that a fine-tuned FLANG-RoBERTa model, utilizing a cross-encoder architecture with tweet content and responses generated by Claude, performs optimally. We further use a T5-based paraphraser to generate paraphrases of a given post and demonstrate GGEA's ability to predict which post will elicit the most reactions. We believe this novel application of LLMs provides a significant advancement in predicting social media post reach.

Title: Concept-aware Data Construction Improves In-context Learning of Language Models

• Authors: Michal Štefánik, Marek Kadlčík, Petr Sojka
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09703
• Pdf URL: https://arxiv.org/pdf/2403.09703
• Copy Paste: [[2403.09703]] Concept-aware Data Construction Improves In-context Learning of Language Models(https://arxiv.org/abs/2403.09703)
• Keywords: robust, transformer
• Abstract: Many recent language models (LMs) are capable of in-context learning (ICL), manifested in the LMs' ability to perform a new task solely from natural-language instruction. Previous work curating in-context learners assumes that ICL emerges from a vast over-parametrization or the scale of multi-task training. However, recent theoretical work attributes the ICL ability to concept-dependent training data and creates functional in-context learners even in small-scale, synthetic settings. In this work, we practically explore this newly identified axis of ICL quality. We propose Concept-aware Training (CoAT), a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. We find that by using CoAT, pre-trained transformers can learn to better utilise new latent concepts from demonstrations and that such ability makes ICL more robust to the functional deficiencies of the previous models. Finally, we show that concept-aware in-context learning is more effective for a majority of new tasks when compared to traditional instruction tuning, resulting in a performance comparable to the previous in-context learners using magnitudes of more training data.

Title: Alignment Studio: Aligning Large Language Models to Particular Contextual Regulations

• Authors: Swapnaja Achintalwar, Ioana Baldini, Djallel Bouneffouf, Joan Byamugisha, Maria Chang, Pierre Dognin, Eitan Farchi, Ndivhuwo Makondo, Aleksandra Mojsilovic, Manish Nagireddy, Karthikeyan Natesan Ramamurthy, Inkit Padhi, Orna Raz, Jesus Rios, Prasanna Sattigeri, Moninder Singh, Siphiwe Thwala, Rosario A. Uceda-Sosa, Kush R. Varshney
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09704
• Pdf URL: https://arxiv.org/pdf/2403.09704
• Copy Paste: [[2403.09704]] Alignment Studio: Aligning Large Language Models to Particular Contextual Regulations(https://arxiv.org/abs/2403.09704)
• Keywords: large language model
• Abstract: The alignment of large language models is usually done by model providers to add or control behaviors that are common or universally understood across use cases and contexts. In contrast, in this article, we present an approach and architecture that empowers application developers to tune a model to their particular values, social norms, laws and other regulations, and orchestrate between potentially conflicting requirements in context. We lay out three main components of such an Alignment Studio architecture: Framers, Instructors, and Auditors that work in concert to control the behavior of a language model. We illustrate this approach with a running example of aligning a company's internal-facing enterprise chatbot to its business conduct guidelines.

Title: A Novel Nuanced Conversation Evaluation Framework for Large Language Models in Mental Health

• Authors: Alexander Marrapese, Basem Suleiman, Imdad Ullah, Juno Kim
• Subjects: cs.CL, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2403.09705
• Pdf URL: https://arxiv.org/pdf/2403.09705
• Copy Paste: [[2403.09705]] A Novel Nuanced Conversation Evaluation Framework for Large Language Models in Mental Health(https://arxiv.org/abs/2403.09705)
• Keywords: large language model
• Abstract: Understanding the conversation abilities of Large Language Models (LLMs) can help lead to its more cautious and appropriate deployment. This is especially important for safety-critical domains like mental health, where someone's life may depend on the exact wording of a response to an urgent question. In this paper, we propose a novel framework for evaluating the nuanced conversation abilities of LLMs. Within it, we develop a series of quantitative metrics developed from literature on using psychotherapy conversation analysis literature. While we ensure that our framework and metrics are transferable by researchers to relevant adjacent domains, we apply them to the mental health field. We use our framework to evaluate several popular frontier LLMs, including some GPT and Llama models, through a verified mental health dataset. Our results show that GPT4 Turbo can perform significantly more similarly to verified therapists than other selected LLMs. We conduct additional analysis to examine how LLM conversation performance varies across specific mental health topics. Our results indicate that GPT4 Turbo performs well in achieving high correlation with verified therapists in particular topics such as Parenting and Relationships. We believe our contributions will help researchers develop better LLMs that, in turn, will more positively support people's lives.

Title: Linguistic Structure Induction from Language Models

• Authors: Omar Momen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09714
• Pdf URL: https://arxiv.org/pdf/2403.09714
• Copy Paste: [[2403.09714]] Linguistic Structure Induction from Language Models(https://arxiv.org/abs/2403.09714)
• Keywords: transformer
• Abstract: Linear sequences of words are implicitly represented in our brains by hierarchical structures that organize the composition of words in sentences. Linguists formalize different frameworks to model this hierarchy; two of the most common syntactic frameworks are Constituency and Dependency. Constituency represents sentences as nested groups of phrases, while dependency represents a sentence by assigning relations between its words. Recently, the pursuit of intelligent machines has produced Language Models (LMs) capable of solving many language tasks with a human-level performance. Many studies now question whether LMs implicitly represent syntactic hierarchies. This thesis focuses on producing constituency and dependency structures from LMs in an unsupervised setting. I review the critical methods in this field and highlight a line of work that utilizes a numerical representation for binary constituency trees (Syntactic Distance). I present a detailed study on StructFormer (SF) (Shen et al., 2021), which retrofits a transformer encoder architecture with a parser network to produce constituency and dependency structures. I present six experiments to analyze and address this field's challenges; experiments include investigating the effect of repositioning the parser network within the SF architecture, evaluating subword-based induced trees, and benchmarking the models developed in the thesis experiments on linguistic tasks. Models benchmarking is performed by participating in the BabyLM challenge, published at CoNLL 2023 (Momen et al., 2023). The results of this thesis encourage further development in the direction of retrofitting transformer-based models to induce syntactic structures, supported by the acceptable performance of SF in different experimental settings and the observed limitations that require innovative solutions to advance the state of syntactic structure induction.

Title: Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Language

• Authors: Vitaly Shalumov, Harel Haskey, Yuval Solaz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09719
• Pdf URL: https://arxiv.org/pdf/2403.09719
• Copy Paste: [[2403.09719]] Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Language(https://arxiv.org/abs/2403.09719)
• Keywords: extraction
• Abstract: In this paper, we introduce summarization MevakerSumm and conclusion extraction MevakerConc datasets for the Hebrew language based on the State Comptroller and Ombudsman of Israel reports, along with two auxiliary datasets. We accompany these datasets with models for conclusion extraction (HeConE, HeConEspc) and conclusion allocation (HeCross). All of the code, datasets, and model checkpoints used in this work are publicly available.

Title: Fine-tuning vs Prompting, Can Language Models Understand Human Values?

• Authors: Pingwei Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09720
• Pdf URL: https://arxiv.org/pdf/2403.09720
• Copy Paste: [[2403.09720]] Fine-tuning vs Prompting, Can Language Models Understand Human Values?(https://arxiv.org/abs/2403.09720)
• Keywords: large language model
• Abstract: Accurately handling the underlying support values in sentences is crucial for understanding the speaker's tendencies, yet it poses a challenging task in natural language understanding (NLU). In this article, we explore the potential of fine-tuning and prompt tuning in this downstream task, using the Human Value Detection 2023. Additionally, we attempt to validate whether models can effectively solve the problem based on the knowledge acquired during the pre-training stage. Simultaneously, our interest lies in the capabilities of large language models (LLMs) aligned with RLHF in this task, and some preliminary attempts are presented.

Title: A Semantic Mention Graph Augmented Model for Document-Level Event Argument Extraction

• Authors: Jian Zhang, Changlin Yang, Haiping Zhu, Qika Lin, Fangzhi Xu, Jun Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09721
• Pdf URL: https://arxiv.org/pdf/2403.09721
• Copy Paste: [[2403.09721]] A Semantic Mention Graph Augmented Model for Document-Level Event Argument Extraction(https://arxiv.org/abs/2403.09721)
• Keywords: extraction, transformer
• Abstract: Document-level Event Argument Extraction (DEAE) aims to identify arguments and their specific roles from an unstructured document. The advanced approaches on DEAE utilize prompt-based methods to guide pre-trained language models (PLMs) in extracting arguments from input documents. They mainly concentrate on establishing relations between triggers and entity mentions within documents, leaving two unresolved problems: a) independent modeling of entity mentions; b) document-prompt isolation. To this end, we propose a semantic mention Graph Augmented Model (GAM) to address these two problems in this paper. Firstly, GAM constructs a semantic mention graph that captures relations within and between documents and prompts, encompassing co-existence, co-reference and co-type relations. Furthermore, we introduce an ensembled graph transformer module to address mentions and their three semantic relations effectively. Later, the graph-augmented encoder-decoder module incorporates the relation-specific graph into the input embedding of PLMs and optimizes the encoder section with topology information, enhancing the relations comprehensively. Extensive experiments on the RAMS and WikiEvents datasets demonstrate the effectiveness of our approach, surpassing baseline methods and achieving a new state-of-the-art performance.

Title: ClaimVer: Explainable Claim-Level Verification and Evidence Attribution of Text Through Knowledge Graphs

• Authors: Preetam Prabhu Srikar Dammu, Himanshu Naidu, Mouly Dewan, YoungMin Kim, Tanya Roosta, Aman Chadha, Chirag Shah
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09724
• Pdf URL: https://arxiv.org/pdf/2403.09724
• Copy Paste: [[2403.09724]] ClaimVer: Explainable Claim-Level Verification and Evidence Attribution of Text Through Knowledge Graphs(https://arxiv.org/abs/2403.09724)
• Keywords: explainability
• Abstract: In the midst of widespread misinformation and disinformation through social media and the proliferation of AI-generated texts, it has become increasingly difficult for people to validate and trust information they encounter. Many fact-checking approaches and tools have been developed, but they often lack appropriate explainability or granularity to be useful in various contexts. A text validation method that is easy to use, accessible, and can perform fine-grained evidence attribution has become crucial. More importantly, building user trust in such a method requires presenting the rationale behind each prediction, as research shows this significantly influences people's belief in automated systems. It is also paramount to localize and bring users' attention to the specific problematic content, instead of providing simple blanket labels. In this paper, we present $\textit{ClaimVer, a human-centric framework}$ tailored to meet users' informational and verification needs by generating rich annotations and thereby reducing cognitive load. Designed to deliver comprehensive evaluations of texts, it highlights each claim, verifies it against a trusted knowledge graph (KG), presents the evidence, and provides succinct, clear explanations for each claim prediction. Finally, our framework introduces an attribution score, enhancing applicability across a wide range of downstream tasks.

Title: Investigating the performance of Retrieval-Augmented Generation and fine-tuning for the development of AI-driven knowledge-based systems

• Authors: Robert Lakatos, Peter Pollner, Andras Hajdu, Tamas Joo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09727
• Pdf URL: https://arxiv.org/pdf/2403.09727
• Copy Paste: [[2403.09727]] Investigating the performance of Retrieval-Augmented Generation and fine-tuning for the development of AI-driven knowledge-based systems(https://arxiv.org/abs/2403.09727)
• Keywords: generative, large language model
• Abstract: The development of generative large language models (G-LLM) opened up new opportunities for the development of new types of knowledge-based systems similar to ChatGPT, Bing, or Gemini. Fine-tuning (FN) and Retrieval-Augmented Generation (RAG) are the techniques that can be used to implement domain adaptation for the development of G-LLM-based knowledge systems. In our study, using ROUGE, BLEU, METEOR scores, and cosine similarity, we compare and examine the performance of RAG and FN for the GPT-J-6B, OPT-6.7B, LlaMA, LlaMA-2 language models. Based on measurements shown on different datasets, we demonstrate that RAG-based constructions are more efficient than models produced with FN. We point out that connecting RAG and FN is not trivial, because connecting FN models with RAG can cause a decrease in performance. Furthermore, we outline a simple RAG-based architecture which, on average, outperforms the FN models by 16% in terms of the ROGUE score, 15% in the case of the BLEU score, and 53% based on the cosine similarity. This shows the significant advantage of RAG over FN in terms of hallucination, which is not offset by the fact that the average 8% better METEOR score of FN models indicates greater creativity compared to RAG.

Title: Simulating Weighted Automata over Sequences and Trees with Transformers

• Authors: Michael Rizvi, Maude Lizaire, Clara Lacroce, Guillaume Rabusseau
• Subjects: cs.CL, cs.AI, cs.CC
• Abstract URL: https://arxiv.org/abs/2403.09728
• Pdf URL: https://arxiv.org/pdf/2403.09728
• Copy Paste: [[2403.09728]] Simulating Weighted Automata over Sequences and Trees with Transformers(https://arxiv.org/abs/2403.09728)
• Keywords: transformer
• Abstract: Transformers are ubiquitous models in the natural language processing (NLP) community and have shown impressive empirical successes in the past few years. However, little is understood about how they reason and the limits of their computational capabilities. These models do not process data sequentially, and yet outperform sequential neural models such as RNNs. Recent work has shown that these models can compactly simulate the sequential reasoning abilities of deterministic finite automata (DFAs). This leads to the following question: can transformers simulate the reasoning of more complex finite state machines? In this work, we show that transformers can simulate weighted finite automata (WFAs), a class of models which subsumes DFAs, as well as weighted tree automata (WTA), a generalization of weighted automata to tree structured inputs. We prove these claims formally and provide upper bounds on the sizes of the transformer models needed as a function of the number of states the target automata. Empirically, we perform synthetic experiments showing that transformers are able to learn these compact solutions via standard gradient-based training.

Title: PET-SQL: A Prompt-enhanced Two-stage Text-to-SQL Framework with Cross-consistency

• Authors: Zhishuai Li, Xiang Wang, Jingjing Zhao, Sun Yang, Guoqing Du, Xiaoru Hu, Bin Zhang, Yuxiao Ye, Ziyue Li, Rui Zhao, Hangyu Mao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09732
• Pdf URL: https://arxiv.org/pdf/2403.09732
• Copy Paste: [[2403.09732]] PET-SQL: A Prompt-enhanced Two-stage Text-to-SQL Framework with Cross-consistency(https://arxiv.org/abs/2403.09732)
• Keywords: large language model
• Abstract: Recent advancements in Text-to-SQL (Text2SQL) emphasize stimulating the large language models (LLM) on in-context learning, achieving significant results. Nevertheless, they face challenges when dealing with verbose database information and complex user intentions. This paper presents a two-stage framework to enhance the performance of current LLM-based natural language to SQL systems. We first introduce a novel prompt representation, called reference-enhanced representation, which includes schema information and randomly sampled cell values from tables to instruct LLMs in generating SQL queries. Then, in the first stage, question-SQL pairs are retrieved as few-shot demonstrations, prompting the LLM to generate a preliminary SQL (PreSQL). After that, the mentioned entities in PreSQL are parsed to conduct schema linking, which can significantly compact the useful information. In the second stage, with the linked schema, we simplify the prompt's schema information and instruct the LLM to produce the final SQL. Finally, as the post-refinement module, we propose using cross-consistency across different LLMs rather than self-consistency within a particular LLM. Our methods achieve new SOTA results on the Spider benchmark, with an execution accuracy of 87.6%.

Title: OverleafCopilot: Empowering Academic Writing in Overleaf with Large Language Models

• Authors: Haomin Wen, Zhenjie Wei, Yan Lin, Jiyuan Wang, Yuxuan Liang, Huaiyu Wan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09733
• Pdf URL: https://arxiv.org/pdf/2403.09733
• Copy Paste: [[2403.09733]] OverleafCopilot: Empowering Academic Writing in Overleaf with Large Language Models(https://arxiv.org/abs/2403.09733)
• Keywords: privacy, large language model
• Abstract: The rapid development of Large Language Models (LLMs) has facilitated a variety of applications from different domains. In this technical report, we explore the integration of LLMs and the popular academic writing tool, Overleaf, to enhance the efficiency and quality of academic writing. To achieve the above goal, there are three challenges: i) including seamless interaction between Overleaf and LLMs, ii) establishing reliable communication with the LLM provider, and iii) ensuring user privacy. To address these challenges, we present OverleafCopilot, the first-ever tool (i.e., a browser extension) that seamlessly integrates LLMs and Overleaf, enabling researchers to leverage the power of LLMs while writing papers. Specifically, we first propose an effective framework to bridge LLMs and Overleaf. Then, we developed PromptGenius, a website for researchers to easily find and share high-quality up-to-date prompts. Thirdly, we propose an agent command system to help researchers quickly build their customizable agents. OverleafCopilot (https://chromewebstore.google.com/detail/overleaf-copilot/eoadabdpninlhkkbhngoddfjianhlghb ) has been on the Chrome Extension Store, which now serves thousands of researchers. Additionally, the code of PromptGenius is released at https://github.com/wenhaomin/ChatGPT-PromptGenius. We believe our work has the potential to revolutionize academic writing practices, empowering researchers to produce higher-quality papers in less time.

Title: Do Large Language Models Solve ARC Visual Analogies Like People Do?

• Authors: Gustaw Opiełka, Hannes Rosenbusch, Veerle Vijverberg, Claire E. Stevenson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09734
• Pdf URL: https://arxiv.org/pdf/2403.09734
• Copy Paste: [[2403.09734]] Do Large Language Models Solve ARC Visual Analogies Like People Do?(https://arxiv.org/abs/2403.09734)
• Keywords: large language model
• Abstract: The Abstraction Reasoning Corpus (ARC) is a visual analogical reasoning test designed for humans and machines (Chollet, 2019). We compared human and large language model (LLM) performance on a new child-friendly set of ARC items. Results show that both children and adults outperform most LLMs on these tasks. Error analysis revealed a similar "fallback" solution strategy in LLMs and young children, where part of the analogy is simply copied. In addition, we found two other error types, one based on seemingly grasping key concepts (e.g., Inside-Outside) and the other based on simple combinations of analogy input matrices. On the whole, "concept" errors were more common in humans, and "matrix" errors were more common in LLMs. This study sheds new light on LLM reasoning ability and the extent to which we can use error analyses and comparisons with human development to understand how LLMs solve visual analogies.

Title: A Sophisticated Framework for the Accurate Detection of Phishing Websites

• Authors: Asif Newaz, Farhan Shahriyar Haq, Nadim Ahmed
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09735
• Pdf URL: https://arxiv.org/pdf/2403.09735
• Copy Paste: [[2403.09735]] A Sophisticated Framework for the Accurate Detection of Phishing Websites(https://arxiv.org/abs/2403.09735)
• Keywords: privacy, attack
• Abstract: Phishing is an increasingly sophisticated form of cyberattack that is inflicting huge financial damage to corporations throughout the globe while also jeopardizing individuals' privacy. Attackers are constantly devising new methods of launching such assaults and detecting them has become a daunting task. Many different techniques have been suggested, each with its own pros and cons. While machine learning-based techniques have been most successful in identifying such attacks, they continue to fall short in terms of performance and generalizability. This paper proposes a comprehensive methodology for detecting phishing websites. The goal is to design a system that is capable of accurately distinguishing phishing websites from legitimate ones and provides generalized performance over a broad variety of datasets. A combination of feature selection, greedy algorithm, cross-validation, and deep learning methods have been utilized to construct a sophisticated stacking ensemble classifier. Extensive experimentation on four different phishing datasets was conducted to evaluate the performance of the proposed technique. The proposed algorithm outperformed the other existing phishing detection models obtaining accuracy of 97.49%, 98.23%, 97.48%, and 98.20% on dataset-1 (UCI Phishing Websites Dataset), dataset-2 (Phishing Dataset for Machine Learning: Feature Evaluation), dataset-3 (Phishing Websites Dataset), and dataset-4 (Web page phishing detection), respectively. The high accuracy values obtained across all datasets imply the models' generalizability and effectiveness in the accurate identification of phishing websites.

Title: Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation

• Authors: Se-eun Yoon, Zhankui He, Jessica Maria Echterhoff, Julian McAuley
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.09738
• Pdf URL: https://arxiv.org/pdf/2403.09738
• Copy Paste: [[2403.09738]] Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation(https://arxiv.org/abs/2403.09738)
• Keywords: generative, large language model
• Abstract: Synthetic users are cost-effective proxies for real users in the evaluation of conversational recommender systems. Large language models show promise in simulating human-like behavior, raising the question of their ability to represent a diverse population of users. We introduce a new protocol to measure the degree to which language models can accurately emulate human behavior in conversational recommendation. This protocol is comprised of five tasks, each designed to evaluate a key property that a synthetic user should exhibit: choosing which items to talk about, expressing binary preferences, expressing open-ended preferences, requesting recommendations, and giving feedback. Through evaluation of baseline simulators, we demonstrate these tasks effectively reveal deviations of language models from human behavior, and offer insights on how to reduce the deviations with model selection and prompting strategies.

Title: The Human Factor in Detecting Errors of Large Language Models: A Systematic Literature Review and Future Research Directions

• Authors: Christian A. Schiller
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09743
• Pdf URL: https://arxiv.org/pdf/2403.09743
• Copy Paste: [[2403.09743]] The Human Factor in Detecting Errors of Large Language Models: A Systematic Literature Review and Future Research Directions(https://arxiv.org/abs/2403.09743)
• Keywords: large language model
• Abstract: The launch of ChatGPT by OpenAI in November 2022 marked a pivotal moment for Artificial Intelligence, introducing Large Language Models (LLMs) to the mainstream and setting new records in user adoption. LLMs, particularly ChatGPT, trained on extensive internet data, demonstrate remarkable conversational capabilities across various domains, suggesting a significant impact on the workforce. However, these models are susceptible to errors - "hallucinations" and omissions, generating incorrect or incomplete information. This poses risks especially in contexts where accuracy is crucial, such as legal compliance, medicine or fine-grained process frameworks. There are both technical and human solutions to cope with this isse. This paper explores the human factors that enable users to detect errors in LLM outputs, a critical component in mitigating risks associated with their use in professional settings. Understanding these factors is essential for organizations aiming to leverage LLM technology efficiently, guiding targeted training and deployment strategies to enhance error detection by users. This approach not only aims to optimize the use of LLMs but also to prevent potential downstream issues stemming from reliance on inaccurate model responses. The research emphasizes the balance between technological advancement and human insight in maximizing the benefits of LLMs while minimizing the risks, particularly in areas where precision is paramount. This paper performs a systematic literature research on this research topic, analyses and synthesizes the findings, and outlines future research directions. Literature selection cut-off date is January 11th 2024.

Title: Evaluating the Application of Large Language Models to Generate Feedback in Programming Education

• Authors: Sven Jacobs, Steffen Jaschke
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.09744
• Pdf URL: https://arxiv.org/pdf/2403.09744
• Copy Paste: [[2403.09744]] Evaluating the Application of Large Language Models to Generate Feedback in Programming Education(https://arxiv.org/abs/2403.09744)
• Keywords: large language model
• Abstract: This study investigates the application of large language models, specifically GPT-4, to enhance programming education. The research outlines the design of a web application that uses GPT-4 to provide feedback on programming tasks, without giving away the solution. A web application for working on programming tasks was developed for the study and evaluated with 51 students over the course of one semester. The results show that most of the feedback generated by GPT-4 effectively addressed code errors. However, challenges with incorrect suggestions and hallucinated issues indicate the need for further improvements.

Title: Re-Search for The Truth: Multi-round Retrieval-augmented Large Language Models are Strong Fake News Detectors

• Authors: Guanghua Li, Wensheng Lu, Wei Zhang, Defu Lian, Kezhong Lu, Rui Mao, Kai Shu, Hao Liao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09747
• Pdf URL: https://arxiv.org/pdf/2403.09747
• Copy Paste: [[2403.09747]] Re-Search for The Truth: Multi-round Retrieval-augmented Large Language Models are Strong Fake News Detectors(https://arxiv.org/abs/2403.09747)
• Keywords: interpretability, generative, large language model
• Abstract: The proliferation of fake news has had far-reaching implications on politics, the economy, and society at large. While Fake news detection methods have been employed to mitigate this issue, they primarily depend on two essential elements: the quality and relevance of the evidence, and the effectiveness of the verdict prediction mechanism. Traditional methods, which often source information from static repositories like Wikipedia, are limited by outdated or incomplete data, particularly for emerging or rare claims. Large Language Models (LLMs), known for their remarkable reasoning and generative capabilities, introduce a new frontier for fake news detection. However, like traditional methods, LLM-based solutions also grapple with the limitations of stale and long-tail knowledge. Additionally, retrieval-enhanced LLMs frequently struggle with issues such as low-quality evidence retrieval and context length constraints. To address these challenges, we introduce a novel, retrieval-augmented LLMs framework--the first of its kind to automatically and strategically extract key evidence from web sources for claim verification. Employing a multi-round retrieval strategy, our framework ensures the acquisition of sufficient, relevant evidence, thereby enhancing performance. Comprehensive experiments across three real-world datasets validate the framework's superiority over existing methods. Importantly, our model not only delivers accurate verdicts but also offers human-readable explanations to improve result interpretability.

Title: Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Models

• Authors: Zhuoqun Li, Hongyu Lin, Yaojie Lu, Hao Xiang, Xianpei Han, Le Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09750
• Pdf URL: https://arxiv.org/pdf/2403.09750
• Copy Paste: [[2403.09750]] Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Models(https://arxiv.org/abs/2403.09750)
• Keywords: large language model
• Abstract: Declarative knowledge and procedural knowledge are two key parts in meta-cognitive theory, and these two hold significant importance in pre-training and inference of LLMs. However, a comprehensive analysis comparing these two types of knowledge is lacking, primarily due to challenges in definition, probing and quantitative assessment. In this paper, we explore from a new perspective by providing ground-truth knowledge for LLMs and evaluating the effective score. Through extensive experiments with widely-used datasets and models, we get conclusions: (1) In most tasks, benefits from declarative knowledge are greater than those from procedural knowledge. (2) Profits of procedural knowledge are larger than declarative knowledge only in reasoning tasks with simple logic. (3) As pre-training progresses and size increases, model ability to utilize both kinds of knowledge significantly improves, but in different speed. We do detailed analysis for the findings and this can provide primary guidance for evaluation and enhancement of large language models.

Title: What Was Your Prompt? A Remote Keylogging Attack on AI Assistants

• Authors: Roy Weiss, Daniel Ayzenshteyn, Guy Amit, Yisroel Mirsky
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09751
• Pdf URL: https://arxiv.org/pdf/2403.09751
• Copy Paste: [[2403.09751]] What Was Your Prompt? A Remote Keylogging Attack on AI Assistants(https://arxiv.org/abs/2403.09751)
• Keywords: attack, large language model
• Abstract: AI assistants are becoming an integral part of society, used for asking advice or help in personal and confidential issues. In this paper, we unveil a novel side-channel that can be used to read encrypted responses from AI Assistants over the web: the token-length side-channel. We found that many vendors, including OpenAI and Microsoft, have this side-channel. However, inferring the content of a response from a token-length sequence alone proves challenging. This is because tokens are akin to words, and responses can be several sentences long leading to millions of grammatically correct sentences. In this paper, we show how this can be overcome by (1) utilizing the power of a large language model (LLM) to translate these sequences, (2) providing the LLM with inter-sentence context to narrow the search space and (3) performing a known-plaintext attack by fine-tuning the model on the target model's writing style. Using these methods, we were able to accurately reconstruct 29\% of an AI assistant's responses and successfully infer the topic from 55\% of them. To demonstrate the threat, we performed the attack on OpenAI's ChatGPT-4 and Microsoft's Copilot on both browser and API traffic.

Title: Explainable Machine Learning-Based Security and Privacy Protection Framework for Internet of Medical Things Systems

• Authors: Ayoub Si-ahmed, Mohammed Ali Al-Garadi, Narhimene Boustia
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09752
• Pdf URL: https://arxiv.org/pdf/2403.09752
• Copy Paste: [[2403.09752]] Explainable Machine Learning-Based Security and Privacy Protection Framework for Internet of Medical Things Systems(https://arxiv.org/abs/2403.09752)
• Keywords: security, privacy, protect, attack, federate
• Abstract: The Internet of Medical Things (IoMT) transcends traditional medical boundaries, enabling a transition from reactive treatment to proactive prevention. This innovative method revolutionizes healthcare by facilitating early disease detection and tailored care, particularly in chronic disease management, where IoMT automates treatments based on real-time health data collection. Nonetheless, its benefits are countered by significant security challenges that endanger the lives of its users due to the sensitivity and value of the processed data, thereby attracting malicious interests. Moreover, the utilization of wireless communication for data transmission exposes medical data to interception and tampering by cybercriminals. Additionally, anomalies may arise due to human errors, network interference, or hardware malfunctions. In this context, anomaly detection based on Machine Learning (ML) is an interesting solution, but it comes up against obstacles in terms of explicability and protection of privacy. To address these challenges, a new framework for Intrusion Detection Systems (IDS) is introduced, leveraging Artificial Neural Networks (ANN) for intrusion detection while utilizing Federated Learning (FL) for privacy preservation. Additionally, eXplainable Artificial Intelligence (XAI) methods are incorporated to enhance model explanation and interpretation. The efficacy of the proposed framework is evaluated and compared with centralized approaches using multiple datasets containing network and medical data, simulating various attack types impacting the confidentiality, integrity, and availability of medical and physiological data. The results obtained offer compelling evidence that the FL method performs comparably to the centralized method, demonstrating high performance. Additionally, it affords the dual advantage of safeguarding privacy and providing model explanation.

Title: Emotional Intelligence Through Artificial Intelligence : NLP and Deep Learning in the Analysis of Healthcare Texts

• Authors: Prashant Kumar Nag, Amit Bhagat, R. Vishnu Priya, Deepak kumar Khare
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2403.09762
• Pdf URL: https://arxiv.org/pdf/2403.09762
• Copy Paste: [[2403.09762]] Emotional Intelligence Through Artificial Intelligence : NLP and Deep Learning in the Analysis of Healthcare Texts(https://arxiv.org/abs/2403.09762)
• Keywords: generative
• Abstract: This manuscript presents a methodical examination of the utilization of Artificial Intelligence in the assessment of emotions in texts related to healthcare, with a particular focus on the incorporation of Natural Language Processing and deep learning technologies. We scrutinize numerous research studies that employ AI to augment sentiment analysis, categorize emotions, and forecast patient outcomes based on textual information derived from clinical narratives, patient feedback on medications, and online health discussions. The review demonstrates noteworthy progress in the precision of algorithms used for sentiment classification, the prognostic capabilities of AI models for neurodegenerative diseases, and the creation of AI-powered systems that offer support in clinical decision-making. Remarkably, the utilization of AI applications has exhibited an enhancement in personalized therapy plans by integrating patient sentiment and contributing to the early identification of mental health disorders. There persist challenges, which encompass ensuring the ethical application of AI, safeguarding patient confidentiality, and addressing potential biases in algorithmic procedures. Nevertheless, the potential of AI to revolutionize healthcare practices is unmistakable, offering a future where healthcare is not only more knowledgeable and efficient but also more empathetic and centered around the needs of patients. This investigation underscores the transformative influence of AI on healthcare, delivering a comprehensive comprehension of its role in examining emotional content in healthcare texts and highlighting the trajectory towards a more compassionate approach to patient care. The findings advocate for a harmonious synergy between AI's analytical capabilities and the human aspects of healthcare.

Title: An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models

• Authors: Haochen Luo, Jindong Gu, Fengyuan Liu, Philip Torr
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09766
• Pdf URL: https://arxiv.org/pdf/2403.09766
• Copy Paste: [[2403.09766]] An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models(https://arxiv.org/abs/2403.09766)
• Keywords: attack
• Abstract: Different from traditional task-specific vision models, recent large VLMs can readily adapt to different vision tasks by simply using different textual instructions, i.e., prompts. However, a well-known concern about traditional task-specific vision models is that they can be misled by imperceptible adversarial perturbations. Furthermore, the concern is exacerbated by the phenomenon that the same adversarial perturbations can fool different task-specific models. Given that VLMs rely on prompts to adapt to different tasks, an intriguing question emerges: Can a single adversarial image mislead all predictions of VLMs when a thousand different prompts are given? This question essentially introduces a novel perspective on adversarial transferability: cross-prompt adversarial transferability. In this work, we propose the Cross-Prompt Attack (CroPA). This proposed method updates the visual adversarial perturbation with learnable prompts, which are designed to counteract the misleading effects of the adversarial image. By doing this, CroPA significantly improves the transferability of adversarial examples across prompts. Extensive experiments are conducted to verify the strong cross-prompt adversarial transferability of CroPA with prevalent VLMs including Flamingo, BLIP-2, and InstructBLIP in various different tasks. Our source code is available at \url{https://github.com/Haochen-Luo/CroPA}.

Title: Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models

• Authors: Yifan Li, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09792
• Pdf URL: https://arxiv.org/pdf/2403.09792
• Copy Paste: [[2403.09792]] Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models(https://arxiv.org/abs/2403.09792)
• Keywords: attack, large language model
• Abstract: In this paper, we study the harmlessness alignment problem of multimodal large language models~(MLLMs). We conduct a systematic empirical analysis of the harmlessness performance of representative MLLMs and reveal that the image input poses the alignment vulnerability of MLLMs. Inspired by this, we propose a novel jailbreak method named HADES, which hides and amplifies the harmfulness of the malicious intent within the text input, using meticulously crafted images. Experimental results show that HADES can effectively jailbreak existing MLLMs, which achieves an average Attack Success Rate~(ASR) of 90.26% for LLaVA-1.5 and 71.60% for Gemini Pro Vision. Our code and data will be publicly released.

Title: Helpful or Harmful? Exploring the Efficacy of Large Language Models for Online Grooming Prevention

• Authors: Ellie Prosser, Matthew Edwards
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09795
• Pdf URL: https://arxiv.org/pdf/2403.09795
• Copy Paste: [[2403.09795]] Helpful or Harmful? Exploring the Efficacy of Large Language Models for Online Grooming Prevention(https://arxiv.org/abs/2403.09795)
• Keywords: generative, large language model
• Abstract: Powerful generative Large Language Models (LLMs) are becoming popular tools amongst the general public as question-answering systems, and are being utilised by vulnerable groups such as children. With children increasingly interacting with these tools, it is imperative for researchers to scrutinise the safety of LLMs, especially for applications that could lead to serious outcomes, such as online child safety queries. In this paper, the efficacy of LLMs for online grooming prevention is explored both for identifying and avoiding grooming through advice generation, and the impact of prompt design on model performance is investigated by varying the provided context and prompt specificity. In results reflecting over 6,000 LLM interactions, we find that no models were clearly appropriate for online grooming prevention, with an observed lack of consistency in behaviours, and potential for harmful answer generation, especially from open-source models. We outline where and how models fall short, providing suggestions for improvement, and identify prompt designs that heavily altered model performance in troubling ways, with findings that can be used to inform best practice usage guides.

Title: BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects

• Authors: Tomas Hodan, Martin Sundermeyer, Yann Labbe, Van Nguyen Nguyen, Gu Wang, Eric Brachmann, Bertram Drost, Vincent Lepetit, Carsten Rother, Jiri Matas
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.09799
• Pdf URL: https://arxiv.org/pdf/2403.09799
• Copy Paste: [[2403.09799]] BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects(https://arxiv.org/abs/2403.09799)
• Keywords: segmentation
• Abstract: We present the evaluation methodology, datasets and results of the BOP Challenge 2023, the fifth in a series of public competitions organized to capture the state of the art in model-based 6D object pose estimation from an RGB/RGB-D image and related tasks. Besides the three tasks from 2022 (model-based 2D detection, 2D segmentation, and 6D localization of objects seen during training), the 2023 challenge introduced new variants of these tasks focused on objects unseen during training. In the new tasks, methods were required to learn new objects during a short onboarding stage (max 5 minutes, 1 GPU) from provided 3D object models. The best 2023 method for 6D localization of unseen objects (GenFlow) notably reached the accuracy of the best 2020 method for seen objects (CosyPose), although being noticeably slower. The best 2023 method for seen objects (GPose) achieved a moderate accuracy improvement but a significant 43% run-time improvement compared to the best 2022 counterpart (GDRNPP). Since 2017, the accuracy of 6D localization of seen objects has improved by more than 50% (from 56.9 to 85.6 AR_C). The online evaluation system stays open and is available at: this http URL

Title: On the Utility of 3D Hand Poses for Action Recognition

• Authors: Md Salman Shamil, Dibyadip Chatterjee, Fadime Sener, Shugao Ma, Angela Yao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09805
• Pdf URL: https://arxiv.org/pdf/2403.09805
• Copy Paste: [[2403.09805]] On the Utility of 3D Hand Poses for Action Recognition(https://arxiv.org/abs/2403.09805)
• Keywords: transformer
• Abstract: 3D hand poses are an under-explored modality for action recognition. Poses are compact yet informative and can greatly benefit applications with limited compute budgets. However, poses alone offer an incomplete understanding of actions, as they cannot fully capture objects and environments with which humans interact. To efficiently model hand-object interactions, we propose HandFormer, a novel multimodal transformer. HandFormer combines 3D hand poses at a high temporal resolution for fine-grained motion modeling with sparsely sampled RGB frames for encoding scene semantics. Observing the unique characteristics of hand poses, we temporally factorize hand modeling and represent each joint by its short-term trajectories. This factorized pose representation combined with sparse RGB samples is remarkably efficient and achieves high accuracy. Unimodal HandFormer with only hand poses outperforms existing skeleton-based methods at 5x fewer FLOPs. With RGB, we achieve new state-of-the-art performance on Assembly101 and H2O with significant improvements in egocentric action recognition.

Title: Self-Supervised Learning for Time Series: Contrastive or Generative?

• Authors: Ziyu Liu, Azadeh Alavi, Minyi Li, Xiang Zhang
• Subjects: cs.LG, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2403.09809
• Pdf URL: https://arxiv.org/pdf/2403.09809
• Copy Paste: [[2403.09809]] Self-Supervised Learning for Time Series: Contrastive or Generative?(https://arxiv.org/abs/2403.09809)
• Keywords: fair, generative
• Abstract: Self-supervised learning (SSL) has recently emerged as a powerful approach to learning representations from large-scale unlabeled data, showing promising results in time series analysis. The self-supervised representation learning can be categorized into two mainstream: contrastive and generative. In this paper, we will present a comprehensive comparative study between contrastive and generative methods in time series. We first introduce the basic frameworks for contrastive and generative SSL, respectively, and discuss how to obtain the supervision signal that guides the model optimization. We then implement classical algorithms (SimCLR vs. MAE) for each type and conduct a comparative analysis in fair settings. Our results provide insights into the strengths and weaknesses of each approach and offer practical recommendations for choosing suitable SSL methods. We also discuss the implications of our findings for the broader field of representation learning and propose future research directions. All the code and data are released at \url{https://github.com/DL4mHealth/SSL_Comparison}.

Title: Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks

• Authors: Zhifan Sun, Antonio Valerio Miceli-Barone
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09832
• Pdf URL: https://arxiv.org/pdf/2403.09832
• Copy Paste: [[2403.09832]] Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks(https://arxiv.org/abs/2403.09832)
• Keywords: attack, large language model
• Abstract: Large Language Models (LLMs) are increasingly becoming the preferred foundation platforms for many Natural Language Processing tasks such as Machine Translation, owing to their quality often comparable to or better than task-specific models, and the simplicity of specifying the task through natural language instructions or in-context examples. Their generality, however, opens them up to subversion by end users who may embed into their requests instructions that cause the model to behave in unauthorized and possibly unsafe ways. In this work we study these Prompt Injection Attacks (PIAs) on multiple families of LLMs on a Machine Translation task, focusing on the effects of model size on the attack success rates. We introduce a new benchmark data set and we discover that on multiple language pairs and injected prompts written in English, larger models under certain conditions may become more susceptible to successful attacks, an instance of the Inverse Scaling phenomenon (McKenzie et al., 2023). To our knowledge, this is the first work to study non-trivial LLM scaling behaviour in a multi-lingual setting.

Title: MARVIS: Motion & Geometry Aware Real and Virtual Image Segmentation

• Authors: Jiayi Wu, Xiaomin Lin, Shahriar Negahdaripour, Cornelia Fermüller, Yiannis Aloimonos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09850
• Pdf URL: https://arxiv.org/pdf/2403.09850
• Copy Paste: [[2403.09850]] MARVIS: Motion & Geometry Aware Real and Virtual Image Segmentation(https://arxiv.org/abs/2403.09850)
• Keywords: segmentation
• Abstract: Tasks such as autonomous navigation, 3D reconstruction, and object recognition near the water surfaces are crucial in marine robotics applications. However, challenges arise due to dynamic disturbances, e.g., light reflections and refraction from the random air-water interface, irregular liquid flow, and similar factors, which can lead to potential failures in perception and navigation systems. Traditional computer vision algorithms struggle to differentiate between real and virtual image regions, significantly complicating tasks. A virtual image region is an apparent representation formed by the redirection of light rays, typically through reflection or refraction, creating the illusion of an object's presence without its actual physical location. This work proposes a novel approach for segmentation on real and virtual image regions, exploiting synthetic images combined with domain-invariant information, a Motion Entropy Kernel, and Epipolar Geometric Consistency. Our segmentation network does not need to be re-trained if the domain changes. We show this by deploying the same segmentation network in two different domains: simulation and the real world. By creating realistic synthetic images that mimic the complexities of the water surface, we provide fine-grained training data for our network (MARVIS) to discern between real and virtual images effectively. By motion & geometry-aware design choices and through comprehensive experimental analysis, we achieve state-of-the-art real-virtual image segmentation performance in unseen real world domain, achieving an IoU over 78% and a F1-Score over 86% while ensuring a small computational footprint. MARVIS offers over 43 FPS (8 FPS) inference rates on a single GPU (CPU core). Our code and dataset are available here https://github.com/jiayi-wu-umd/MARVIS.

Title: FakeWatch: A Framework for Detecting Fake News to Ensure Credible Elections

• Authors: Shaina Raza, Tahniat Khan, Drai Paulen-Patterson, Veronica Chatrath, Mizanur Rahman, Oluwanifemi Bamgbose
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09858
• Pdf URL: https://arxiv.org/pdf/2403.09858
• Copy Paste: [[2403.09858]] FakeWatch: A Framework for Detecting Fake News to Ensure Credible Elections(https://arxiv.org/abs/2403.09858)
• Keywords: robust
• Abstract: In today's technologically driven world, the rapid spread of fake news, particularly during critical events like elections, poses a growing threat to the integrity of information. To tackle this challenge head-on, we introduce FakeWatch, a comprehensive framework carefully designed to detect fake news. Leveraging a newly curated dataset of North American election-related news articles, we construct robust classification models. Our framework integrates a model hub comprising of both traditional machine learning (ML) techniques and cutting-edge Language Models (LMs) to discern fake news effectively. Our overarching objective is to provide the research community with adaptable and precise classification models adept at identifying the ever-evolving landscape of misinformation. Quantitative evaluations of fake news classifiers on our dataset reveal that, while state-of-the-art LMs exhibit a slight edge over traditional ML models, classical models remain competitive due to their balance of accuracy and computational efficiency. Additionally, qualitative analyses shed light on patterns within fake news articles. This research lays the groundwork for future endeavors aimed at combating misinformation, particularly concerning electoral processes. We provide our labeled data and model publicly for use and reproducibility.

Title: ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Image

• Authors: Fangqiang Ding, Yunzhou Zhu, Xiangyu Wen, Chris Xiaoxuan Lu
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09871
• Pdf URL: https://arxiv.org/pdf/2403.09871
• Copy Paste: [[2403.09871]] ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Image(https://arxiv.org/abs/2403.09871)
• Keywords: robust, transformer
• Abstract: In this work, we present ThermoHands, a new benchmark for thermal image-based egocentric 3D hand pose estimation, aimed at overcoming challenges like varying lighting and obstructions (e.g., handwear). The benchmark includes a diverse dataset from 28 subjects performing hand-object and hand-virtual interactions, accurately annotated with 3D hand poses through an automated process. We introduce a bespoken baseline method, TheFormer, utilizing dual transformer modules for effective egocentric 3D hand pose estimation in thermal imagery. Our experimental results highlight TheFormer's leading performance and affirm thermal imaging's effectiveness in enabling robust 3D hand pose estimation in adverse conditions.

Title: Sabiá-2: A New Generation of Portuguese Large Language Models

• Authors: Thales Sales Almeida, Hugo Abonizio, Rodrigo Nogueira, Ramon Pires
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09887
• Pdf URL: https://arxiv.org/pdf/2403.09887
• Copy Paste: [[2403.09887]] Sabiá-2: A New Generation of Portuguese Large Language Models(https://arxiv.org/abs/2403.09887)
• Keywords: large language model
• Abstract: We introduce Sabi\'a-2, a family of large language models trained on Portuguese texts. The models are evaluated on a diverse range of exams, including entry-level tests for Brazilian universities, professional certification exams, and graduate-level exams for various disciplines such as accounting, economics, engineering, law and medicine. Our results reveal that our best model so far, Sabi\'a-2 Medium, matches or surpasses GPT-4's performance in 23 out of 64 exams and outperforms GPT-3.5 in 58 out of 64 exams. Notably, specialization has a significant impact on a model's performance without the need to increase its size, allowing us to offer Sabi\'a-2 Medium at a price per token that is 10 times cheaper than GPT-4. Finally, we identified that math and coding are key abilities that need improvement.

Title: Fisher Mask Nodes for Language Model Merging

• Authors: Thennal D K, Ganesh Nathan, Suchithra M S
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09891
• Pdf URL: https://arxiv.org/pdf/2403.09891
• Copy Paste: [[2403.09891]] Fisher Mask Nodes for Language Model Merging(https://arxiv.org/abs/2403.09891)
• Keywords: transformer
• Abstract: Fine-tuning pre-trained models provides significant advantages in downstream performance. The ubiquitous nature of pre-trained models such as BERT and its derivatives in natural language processing has also led to a proliferation of task-specific fine-tuned models. As these models typically only perform one task well, additional training or ensembling is required in multi-task scenarios. The growing field of model merging provides a solution, dealing with the challenge of combining multiple task-specific models into a single multi-task model. In this study, we introduce a novel model merging method for Transformers, combining insights from previous work in Fisher-weighted averaging and the use of Fisher information in model pruning. Utilizing the Fisher information of mask nodes within the Transformer architecture, we devise a computationally efficient weighted-averaging scheme. Our method exhibits a regular and significant performance increase across various models in the BERT family, outperforming full-scale Fisher-weighted averaging in a fraction of the computational cost, with baseline performance improvements of up to +6.5 and a speedup of 57.4x. Our results prove the potential of our method in current multi-task learning environments and suggest its scalability and adaptability to new model architectures and learning scenarios.

Title: Robust Subgraph Learning by Monitoring Early Training Representations

• Authors: Sepideh Neshatfar, Salimeh Yasaei Sekeh
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.09901
• Pdf URL: https://arxiv.org/pdf/2403.09901
• Copy Paste: [[2403.09901]] Robust Subgraph Learning by Monitoring Early Training Representations(https://arxiv.org/abs/2403.09901)
• Keywords: attack, robust
• Abstract: Graph neural networks (GNNs) have attracted significant attention for their outstanding performance in graph learning and node classification tasks. However, their vulnerability to adversarial attacks, particularly through susceptible nodes, poses a challenge in decision-making. The need for robust graph summarization is evident in adversarial challenges resulting from the propagation of attacks throughout the entire graph. In this paper, we address both performance and adversarial robustness in graph input by introducing the novel technique SHERD (Subgraph Learning Hale through Early Training Representation Distances). SHERD leverages information from layers of a partially trained graph convolutional network (GCN) to detect susceptible nodes during adversarial attacks using standard distance metrics. The method identifies "vulnerable (bad)" nodes and removes such nodes to form a robust subgraph while maintaining node classification performance. Through our experiments, we demonstrate the increased performance of SHERD in enhancing robustness by comparing the network's performance on original and subgraph inputs against various baselines alongside existing adversarial attacks. Our experiments across multiple datasets, including citation datasets such as Cora, Citeseer, and Pubmed, as well as microanatomical tissue structures of cell graphs in the placenta, highlight that SHERD not only achieves substantial improvement in robust performance but also outperforms several baselines in terms of node classification accuracy and computational complexity.

Title: FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models

• Authors: Kai Yi, Georg Meinhardt, Laurent Condat, Peter Richtárik
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.09904
• Pdf URL: https://arxiv.org/pdf/2403.09904
• Copy Paste: [[2403.09904]] FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models(https://arxiv.org/abs/2403.09904)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with a central server, while being respectful of privacy. A critical bottleneck in FL is the communication cost. A pivotal strategy to mitigate this burden is \emph{Local Training}, which involves running multiple local stochastic gradient descent iterations between communication phases. Our work is inspired by the innovative \emph{Scaffnew} algorithm, which has considerably advanced the reduction of communication complexity in FL. We introduce FedComLoc (Federated Compressed and Local Training), integrating practical and effective compression into \emph{Scaffnew} to further enhance communication efficiency. Extensive experiments, using the popular TopK compressor and quantization, demonstrate its prowess in substantially reducing communication overheads in heterogeneous settings.

Title: ProMark: Proactive Diffusion Watermarking for Causal Attribution

• Authors: Vishal Asnani, John Collomosse, Tu Bui, Xiaoming Liu, Shruti Agarwal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09914
• Pdf URL: https://arxiv.org/pdf/2403.09914
• Copy Paste: [[2403.09914]] ProMark: Proactive Diffusion Watermarking for Causal Attribution(https://arxiv.org/abs/2403.09914)
• Keywords: watermark, diffusion, generative
• Abstract: Generative AI (GenAI) is transforming creative workflows through the capability to synthesize and manipulate images via high-level prompts. Yet creatives are not well supported to receive recognition or reward for the use of their content in GenAI training. To this end, we propose ProMark, a causal attribution technique to attribute a synthetically generated image to its training data concepts like objects, motifs, templates, artists, or styles. The concept information is proactively embedded into the input training images using imperceptible watermarks, and the diffusion models (unconditional or conditional) are trained to retain the corresponding watermarks in generated images. We show that we can embed as many as $2^{16}$ unique watermarks into the training data, and each training image can contain more than one watermark. ProMark can maintain image quality whilst outperforming correlation-based attribution. Finally, several qualitative examples are presented, providing the confidence that the presence of the watermark conveys a causative relationship between training data and synthetic images.

Title: Robust Light-Weight Facial Affective Behavior Recognition with CLIP

• Authors: Li Lin, Sarah Papabathini, Xin Wang, Shu Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09915
• Pdf URL: https://arxiv.org/pdf/2403.09915
• Copy Paste: [[2403.09915]] Robust Light-Weight Facial Affective Behavior Recognition with CLIP(https://arxiv.org/abs/2403.09915)
• Keywords: robust
• Abstract: Human affective behavior analysis aims to delve into human expressions and behaviors to deepen our understanding of human emotions. Basic expression categories (EXPR) and Action Units (AUs) are two essential components in this analysis, which categorize emotions and break down facial movements into elemental units, respectively. Despite advancements, existing approaches in expression classification and AU detection often necessitate complex models and substantial computational resources, limiting their applicability in everyday settings. In this work, we introduce the first lightweight framework adept at efficiently tackling both expression classification and AU detection. This framework employs a frozen CLIP image encoder alongside a trainable multilayer perceptron (MLP), enhanced with Conditional Value at Risk (CVaR) for robustness and a loss landscape flattening strategy for improved generalization. Experimental results on the Aff-wild2 dataset demonstrate superior performance in comparison to the baseline while maintaining minimal computational demands, offering a practical solution for affective behavior analysis. The code is available at https://github.com/Purdue-M2/Affective_Behavior_Analysis_M2_PURDUE

Title: Attention-based Class-Conditioned Alignment for Multi-Source Domain Adaptive Object Detection

• Authors: Atif Belal, Akhil Meethal, Francisco Perdigon Romero, Marco Pedersoli, Eric Granger
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09918
• Pdf URL: https://arxiv.org/pdf/2403.09918
• Copy Paste: [[2403.09918]] Attention-based Class-Conditioned Alignment for Multi-Source Domain Adaptive Object Detection(https://arxiv.org/abs/2403.09918)
• Keywords: robust
• Abstract: Domain adaptation methods for object detection (OD) strive to mitigate the impact of distribution shifts by promoting feature alignment across source and target domains. Multi-source domain adaptation (MSDA) allows leveraging multiple annotated source datasets, and unlabeled target data to improve the accuracy and robustness of the detection model. Most state-of-the-art MSDA methods for OD perform feature alignment in a class-agnostic manner. This is challenging since the objects have unique modal information due to variations in object appearance across domains. A recent prototype-based approach proposed a class-wise alignment, yet it suffers from error accumulation due to noisy pseudo-labels which can negatively affect adaptation with imbalanced data. To overcome these limitations, we propose an attention-based class-conditioned alignment scheme for MSDA that aligns instances of each object category across domains. In particular, an attention module coupled with an adversarial domain classifier allows learning domain-invariant and class-specific instance representations. Experimental results on multiple benchmarking MSDA datasets indicate that our method outperforms the state-of-the-art methods and is robust to class imbalance. Our code is available at https://github.com/imatif17/ACIA.

Title: Recurrent Drafter for Fast Speculative Decoding in Large Language Models

• Authors: Aonan Zhang, Chong Wang, Yi Wang, Xuanyu Zhang, Yunfei Cheng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09919
• Pdf URL: https://arxiv.org/pdf/2403.09919
• Copy Paste: [[2403.09919]] Recurrent Drafter for Fast Speculative Decoding in Large Language Models(https://arxiv.org/abs/2403.09919)
• Keywords: transformer, large language model
• Abstract: In this paper, we introduce an improved approach of speculative decoding aimed at enhancing the efficiency of serving large language models. Our method capitalizes on the strengths of two established techniques: the classic two-model speculative decoding approach, and the more recent single-model approach, Medusa. Drawing inspiration from Medusa, our approach adopts a single-model strategy for speculative decoding. However, our method distinguishes itself by employing a single, lightweight draft head with a recurrent dependency design, akin in essence to the small, draft model uses in classic speculative decoding, but without the complexities of the full transformer architecture. And because of the recurrent dependency, we can use beam search to swiftly filter out undesired candidates with the draft head. The outcome is a method that combines the simplicity of single-model design and avoids the need to create a data-dependent tree attention structure only for inference in Medusa. We empirically demonstrate the effectiveness of the proposed method on several popular open source language models, along with a comprehensive analysis of the trade-offs involved in adopting this approach.

Title: Predicting Generalization of AI Colonoscopy Models to Unseen Data

• Authors: Joel Shor, Carson McNeil, Yotam Intrator, Joseph R Ledsam, Hiro-o Yamano, Daisuke Tsurumaru, Hiroki Kayama, Atsushi Hamabe, Koji Ando, Mitsuhiko Ota, Haruei Ogino, Hiroshi Nakase, Kaho Kobayashi, Masaaki Miyo, Eiji Oki, Ichiro Takemasa, Ehud Rivlin, Roman Goldenberg
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.09920
• Pdf URL: https://arxiv.org/pdf/2403.09920
• Copy Paste: [[2403.09920]] Predicting Generalization of AI Colonoscopy Models to Unseen Data(https://arxiv.org/abs/2403.09920)
• Keywords: robust
• Abstract: Background and aims Generalizability of AI colonoscopy algorithms is important for wider adoption in clinical practice. However, current techniques for evaluating performance on unseen data require expensive and time-intensive labels. Methods We use a "Masked Siamese Network" (MSN) to identify novel phenomena in unseen data and predict polyp detector performance. MSN is trained to predict masked out regions of polyp images, without any labels. We test MSN's ability to be trained on data only from Israel and detect unseen techniques, narrow-band imaging (NBI) and chromendoscoy (CE), on colonoscopes from Japan (354 videos, 128 hours). We also test MSN's ability to predict performance of Computer Aided Detection (CADe) of polyps on colonoscopies from both countries, even though MSN is not trained on data from Japan. Results MSN correctly identifies NBI and CE as less similar to Israel whitelight than Japan whitelight (bootstrapped z-test, |z| > 496, p < 10-8 for both) using the label-free Frechet distance. MSN detects NBI with 99% accuracy, predicts CE better than our heuristic (90% vs 79% accuracy) despite being trained only on whitelight, and is the only method that is robust to noisy labels. MSN predicts CADe polyp detector performance on in-domain Israel and out-of-domain Japan colonoscopies (r=0.79, 0.37 respectively). With few examples of Japan detector performance to train on, MSN prediction of Japan performance improves (r=0.56). Conclusion Our technique can identify distribution shifts in clinical data and can predict CADe detector performance on unseen data, without labels. Our self-supervised approach can aid in detecting when data in practice is different from training, such as between hospitals or data has meaningfully shifted from training. MSN has potential for application to medical image domains beyond colonoscopy.

Title: Quantization Effects on Neural Networks Perception: How would quantization change the perceptual field of vision models?

• Authors: Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani, Alessandro Bruno
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09939
• Pdf URL: https://arxiv.org/pdf/2403.09939
• Copy Paste: [[2403.09939]] Quantization Effects on Neural Networks Perception: How would quantization change the perceptual field of vision models?(https://arxiv.org/abs/2403.09939)
• Keywords: interpretability
• Abstract: Neural network quantization is an essential technique for deploying models on resource-constrained devices. However, its impact on model perceptual fields, particularly regarding class activation maps (CAMs), remains a significant area of investigation. In this study, we explore how quantization alters the spatial recognition ability of the perceptual field of vision models, shedding light on the alignment between CAMs and visual saliency maps across various architectures. Leveraging a dataset of 10,000 images from ImageNet, we rigorously evaluate six diverse foundational CNNs: VGG16, ResNet50, EfficientNet, MobileNet, SqueezeNet, and DenseNet. We uncover nuanced changes in CAMs and their alignment with human visual saliency maps through systematic quantization techniques applied to these models. Our findings reveal the varying sensitivities of different architectures to quantization and underscore its implications for real-world applications in terms of model performance and interpretability. The primary contribution of this work revolves around deepening our understanding of neural network quantization, providing insights crucial for deploying efficient and interpretable models in practical settings.

Title: Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries

• Authors: Swetha Ganesh, Jiayu Chen, Gugan Thoppe, Vaneet Aggarwal
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2403.09940
• Pdf URL: https://arxiv.org/pdf/2403.09940
• Copy Paste: [[2403.09940]] Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries(https://arxiv.org/abs/2403.09940)
• Keywords: robust, federate
• Abstract: Federated Reinforcement Learning (FRL) allows multiple agents to collaboratively build a decision making policy without sharing raw trajectories. However, if a small fraction of these agents are adversarial, it can lead to catastrophic results. We propose a policy gradient based approach that is robust to adversarial agents which can send arbitrary values to the server. Under this setting, our results form the first global convergence guarantees with general parametrization. These results demonstrate resilience with adversaries, while achieving sample complexity of order $\tilde{\mathcal{O}}\left( \frac{1}{\epsilon^2} \left( \frac{1}{N-f} + \frac{f^2}{(N-f)^2}\right)\right)$, where $N$ is the total number of agents and $f$ is the number of adversarial agents.

Title: Shifting Focus: From Global Semantics to Local Prominent Features in Swin-Transformer for Knee Osteoarthritis Severity Assessment

• Authors: Aymen Sekhri, Marouane Tliba, Mohamed Amine Kerkouri, Yassine Nasser, Aladine Chetouani, Alessandro Bruno, Rachid Jennane
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09947
• Pdf URL: https://arxiv.org/pdf/2403.09947
• Copy Paste: [[2403.09947]] Shifting Focus: From Global Semantics to Local Prominent Features in Swin-Transformer for Knee Osteoarthritis Severity Assessment(https://arxiv.org/abs/2403.09947)
• Keywords: robust, transformer
• Abstract: Conventional imaging diagnostics frequently encounter bottlenecks due to manual inspection, which can lead to delays and inconsistencies. Although deep learning offers a pathway to automation and enhanced accuracy, foundational models in computer vision often emphasize global context at the expense of local details, which are vital for medical imaging diagnostics. To address this, we harness the Swin Transformer's capacity to discern extended spatial dependencies within images through the hierarchical framework. Our novel contribution lies in refining local feature representations, orienting them specifically toward the final distribution of the classifier. This method ensures that local features are not only preserved but are also enriched with task-specific information, enhancing their relevance and detail at every hierarchical level. By implementing this strategy, our model demonstrates significant robustness and precision, as evidenced by extensive validation of two established benchmarks for Knee OsteoArthritis (KOA) grade classification. These results highlight our approach's effectiveness and its promising implications for the future of medical imaging diagnostics. Our implementation is available on https://github.com/mtliba/KOA_NLCS2024

Title: RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training

• Authors: Zhixiu Lu, Hailong Li, Lili He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09948
• Pdf URL: https://arxiv.org/pdf/2403.09948
• Copy Paste: [[2403.09948]] RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training(https://arxiv.org/abs/2403.09948)
• Keywords: robust
• Abstract: The integration of artificial intelligence (AI) with radiology has marked a transformative era in medical diagnostics. Vision foundation models have been adopted to enhance radiologic imaging analysis. However, the distinct complexities of radiological imaging, including the interpretation of 2D and 3D radiological data, pose unique challenges that existing models, trained on general non-medical images, fail to address adequately. To bridge this gap and capitalize on the diagnostic precision required in medical imaging, we introduce RadCLIP: a pioneering cross-modal foundational model that harnesses Contrastive Language-Image Pre-training (CLIP) to refine radiologic image analysis. RadCLIP incorporates a novel 3D slice pooling mechanism tailored for volumetric image analysis and is trained using a comprehensive and diverse dataset of radiologic image-text pairs. Our evaluations demonstrate that RadCLIP effectively aligns radiological images with their corresponding textual annotations, and in the meantime, offers a robust vision backbone for radiologic imagery with significant promise.

Title: Search-based Ordered Password Generation of Autoregressive Neural Networks

• Authors: Min Jin, Junbin Ye, Rongxuan Shen, Huaxing Lu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.09954
• Pdf URL: https://arxiv.org/pdf/2403.09954
• Copy Paste: [[2403.09954]] Search-based Ordered Password Generation of Autoregressive Neural Networks(https://arxiv.org/abs/2403.09954)
• Keywords: security, attack
• Abstract: Passwords are the most widely used method of authentication and password guessing is the essential part of password cracking and password security research. The progress of deep learning technology provides a promising way to improve the efficiency of password guessing. However, current research on neural network password guessing methods mostly focuses on model structure and has overlooked the generation method. Due to the randomness of sampling, not only the generated passwords have a large number of duplicates, but also the order in which passwords generated is random, leading to inefficient password attacks. In this paper, we propose SOPG, a search-based ordered password generation method, which enables the password guessing model based on autoregressive neural network to generate passwords in approximately descending order of probability. Experiment on comparison of SOPG and Random sampling shows passwords generated by SOPG do not repeat, and when they reach the same cover rate, SOPG requires fewer inferences and far fewer generated passwords than Random sampling, which brings great efficiency improvement to subsequent password attacks. We build SOPGesGPT, a password guessing model based on GPT, using SOPG to generate passwords. Compared with the most influential models OMEN, FLA, PassGAN, VAEPass and the latest model PassGPT in one-site test, experiments show that SOPGesGPT is far ahead in terms of both effective rate and cover rate. As to cover rate that everyone recognizes, SOPGesGPT reaches 35.06%, which is 254%, 298%, 421%, 380%, 81% higher than OMEN, FLA, PassGAN, VAEPass, and PassGPT respectively.

Title: ViTCN: Vision Transformer Contrastive Network For Reasoning

• Authors: Bo Song, Yuanhao Xu, Yichao Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09962
• Pdf URL: https://arxiv.org/pdf/2403.09962
• Copy Paste: [[2403.09962]] ViTCN: Vision Transformer Contrastive Network For Reasoning(https://arxiv.org/abs/2403.09962)
• Keywords: transformer, large language model
• Abstract: Machine learning models have achieved significant milestones in various domains, for example, computer vision models have an exceptional result in object recognition, and in natural language processing, where Large Language Models (LLM) like GPT can start a conversation with human-like proficiency. However, abstract reasoning remains a challenge for these models, Can AI really thinking like a human? still be a question yet to be answered. Raven Progressive Matrices (RPM) is a metric designed to assess human reasoning capabilities. It presents a series of eight images as a problem set, where the participant should try to discover the underlying rules among these images and select the most appropriate image from eight possible options that best completes the sequence. This task always be used to test human reasoning abilities and IQ. Zhang et al proposed a dataset called RAVEN which can be used to test Machine Learning model abstract reasoning ability. In this paper, we purposed Vision Transformer Contrastive Network which build on previous work with the Contrastive Perceptual Inference network (CoPiNet), which set a new benchmark for permutationinvariant models Raven Progressive Matrices by incorporating contrast effects from psychology, cognition, and education, and extends this foundation by leveraging the cutting-edge Vision Transformer architecture. This integration aims to further refine the machine ability to process and reason about spatial-temporal information from pixel-level inputs and global wise features on RAVEN dataset.

Title: Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction

• Authors: Ziyang Xu, Keqin Peng, Liang Ding, Dacheng Tao, Xiliang Lu
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.09963
• Pdf URL: https://arxiv.org/pdf/2403.09963
• Copy Paste: [[2403.09963]] Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction(https://arxiv.org/abs/2403.09963)
• Keywords: extraction
• Abstract: Recent research shows that pre-trained language models (PLMs) suffer from "prompt bias" in factual knowledge extraction, i.e., prompts tend to introduce biases toward specific labels. However, the extent and impact of prompt bias within the model remain underexplored. In response, this paper quantifies the bias with various types of prompts and assesses their impact on different benchmarks. We show that: 1) all prompts in the experiments exhibit non-negligible bias, with gradient-based prompts like AutoPrompt and OptiPrompt displaying significantly higher levels of bias; 2) prompt bias can amplify benchmark accuracy unreasonably by overfitting the test datasets, especially on imbalanced datasets like LAMA. Based on these findings, we propose a representation-based approach to mitigate the prompt bias during inference time. Specifically, we first estimate the biased representation using prompt-only querying, and then remove it from the model's internal representations to generate the debiased representations, which are used to produce the final debiased outputs. Experiments across various prompts, PLMs, and benchmarks show that our approach can not only correct the overfitted performance caused by prompt bias, but also significantly improve the prompt retrieval capability (up to 10% absolute performance gain). Our findings shed new light on the underlying predicting mechanisms of prompt-based queries in PLMs. Hopefully, our plug-and-play approach can be a golden standard to strengthen PLMs toward reliable knowledge bases. Code and data are released in https://github.com/FelliYang/PromptBias.

Title: Prediction of Vessel Arrival Time to Pilotage Area Using Multi-Data Fusion and Deep Learning

• Authors: Xiaocai Zhang, Xiuju Fu, Zhe Xiao, Haiyan Xu, Xiaoyang Wei, Jimmy Koh, Daichi Ogawa, Zheng Qin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.09969
• Pdf URL: https://arxiv.org/pdf/2403.09969
• Copy Paste: [[2403.09969]] Prediction of Vessel Arrival Time to Pilotage Area Using Multi-Data Fusion and Deep Learning(https://arxiv.org/abs/2403.09969)
• Keywords: extraction
• Abstract: This paper investigates the prediction of vessels' arrival time to the pilotage area using multi-data fusion and deep learning approaches. Firstly, the vessel arrival contour is extracted based on Multivariate Kernel Density Estimation (MKDE) and clustering. Secondly, multiple data sources, including Automatic Identification System (AIS), pilotage booking information, and meteorological data, are fused before latent feature extraction. Thirdly, a Temporal Convolutional Network (TCN) framework that incorporates a residual mechanism is constructed to learn the hidden arrival patterns of the vessels. Extensive tests on two real-world data sets from Singapore have been conducted and the following promising results have been obtained: 1) fusion of pilotage booking information and meteorological data improves the prediction accuracy, with pilotage booking information having a more significant impact; 2) using discrete embedding for the meteorological data performs better than using continuous embedding; 3) the TCN outperforms the state-of-the-art baseline methods in regression tasks, exhibiting Mean Absolute Error (MAE) ranging from 4.58 min to 4.86 min; and 4) approximately 89.41% to 90.61% of the absolute prediction residuals fall within a time frame of 10 min.

Title: Think Twice Before Assure: Confidence Estimation for Large Language Models through Reflection on Multiple Answers

• Authors: Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09972
• Pdf URL: https://arxiv.org/pdf/2403.09972
• Copy Paste: [[2403.09972]] Think Twice Before Assure: Confidence Estimation for Large Language Models through Reflection on Multiple Answers(https://arxiv.org/abs/2403.09972)
• Keywords: large language model
• Abstract: Confidence estimation aiming to evaluate output trustability is crucial for the application of large language models (LLM), especially the black-box ones. Existing confidence estimation of LLM is typically not calibrated due to the overconfidence of LLM on its generated incorrect answers. Existing approaches addressing the overconfidence issue are hindered by a significant limitation that they merely consider the confidence of one answer generated by LLM. To tackle this limitation, we propose a novel paradigm that thoroughly evaluates the trustability of multiple candidate answers to mitigate the overconfidence on incorrect answers. Building upon this paradigm, we introduce a two-step framework, which firstly instructs LLM to reflect and provide justifications for each answer, and then aggregates the justifications for comprehensive confidence estimation. This framework can be integrated with existing confidence estimation approaches for superior calibration. Experimental results on six datasets of three tasks demonstrate the rationality and effectiveness of the proposed framework.

Title: EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba

• Authors: Xiaohuan Pei, Tao Huang, Chang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09977
• Pdf URL: https://arxiv.org/pdf/2403.09977
• Copy Paste: [[2403.09977]] EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba(https://arxiv.org/abs/2403.09977)
• Keywords: extraction, transformer
• Abstract: Prior efforts in light-weight model development mainly centered on CNN and Transformer-based designs yet faced persistent challenges. CNNs adept at local feature extraction compromise resolution while Transformers offer global reach but escalate computational demands $\mathcal{O}(N^2)$. This ongoing trade-off between accuracy and efficiency remains a significant hurdle. Recently, state space models (SSMs), such as Mamba, have shown outstanding performance and competitiveness in various tasks such as language modeling and computer vision, while reducing the time complexity of global information extraction to $\mathcal{O}(N)$. Inspired by this, this work proposes to explore the potential of visual state space models in light-weight model design and introduce a novel efficient model variant dubbed EfficientVMamba. Concretely, our EfficientVMamba integrates a atrous-based selective scan approach by efficient skip sampling, constituting building blocks designed to harness both global and local representational features. Additionally, we investigate the integration between SSM blocks and convolutions, and introduce an efficient visual state space block combined with an additional convolution branch, which further elevate the model performance. Experimental results show that, EfficientVMamba scales down the computational complexity while yields competitive results across a variety of vision tasks. For example, our EfficientVMamba-S with $1.3$G FLOPs improves Vim-Ti with $1.5$G FLOPs by a large margin of $5.6\%$ accuracy on ImageNet. Code is available at: \url{https://github.com/TerryPei/EfficientVMamba}.

Title: Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting

• Authors: Zhiqi Li, Yiming Chen, Lingzhe Zhao, Peidong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09981
• Pdf URL: https://arxiv.org/pdf/2403.09981
• Copy Paste: [[2403.09981]] Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting(https://arxiv.org/abs/2403.09981)
• Keywords: robust, diffusion
• Abstract: While text-to-3D and image-to-3D generation tasks have received considerable attention, one important but under-explored field between them is controllable text-to-3D generation, which we mainly focus on in this work. To address this task, 1) we introduce Multi-view ControlNet (MVControl), a novel neural network architecture designed to enhance existing pre-trained multi-view diffusion models by integrating additional input conditions, such as edge, depth, normal, and scribble maps. Our innovation lies in the introduction of a conditioning module that controls the base diffusion model using both local and global embeddings, which are computed from the input condition images and camera poses. Once trained, MVControl is able to offer 3D diffusion guidance for optimization-based 3D generation. And, 2) we propose an efficient multi-stage 3D generation pipeline that leverages the benefits of recent large reconstruction models and score distillation algorithm. Building upon our MVControl architecture, we employ a unique hybrid diffusion guidance method to direct the optimization process. In pursuit of efficiency, we adopt 3D Gaussians as our representation instead of the commonly used implicit representations. We also pioneer the use of SuGaR, a hybrid representation that binds Gaussians to mesh triangle faces. This approach alleviates the issue of poor geometry in 3D Gaussians and enables the direct sculpting of fine-grained geometry on the mesh. Extensive experiments demonstrate that our method achieves robust generalization and enables the controllable generation of high-quality 3D content.

Title: MEDPNet: Achieving High-Precision Adaptive Registration for Complex Die Castings

• Authors: Yu Du, Yu Song, Ce Guo, Xiaojing Tian, Dong Liu, Ming Cong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.09996
• Pdf URL: https://arxiv.org/pdf/2403.09996
• Copy Paste: [[2403.09996]] MEDPNet: Achieving High-Precision Adaptive Registration for Complex Die Castings(https://arxiv.org/abs/2403.09996)
• Keywords: robust, transformer
• Abstract: Due to their complex spatial structure and diverse geometric features, achieving high-precision and robust point cloud registration for complex Die Castings has been a significant challenge in the die-casting industry. Existing point cloud registration methods primarily optimize network models using well-established high-quality datasets, often neglecting practical application in real scenarios. To address this gap, this paper proposes a high-precision adaptive registration method called Multiscale Efficient Deep Closest Point (MEDPNet) and introduces a die-casting point cloud dataset, DieCastCloud, specifically designed to tackle the challenges of point cloud registration in the die-casting industry. The MEDPNet method performs coarse die-casting point cloud data registration using the Efficient-DCP method, followed by precision registration using the Multiscale feature fusion dual-channel registration (MDR) method. We enhance the modeling capability and computational efficiency of the model by replacing the attention mechanism of the Transformer in DCP with Efficient Attention and implementing a collaborative scale mechanism through the combination of serial and parallel blocks. Additionally, we propose the MDR method, which utilizes multilayer perceptrons (MLP), Normal Distributions Transform (NDT), and Iterative Closest Point (ICP) to achieve learnable adaptive fusion, enabling high-precision, scalable, and noise-resistant global point cloud registration. Our proposed method demonstrates excellent performance compared to state-of-the-art geometric and learning-based registration methods when applied to complex die-casting point cloud data.

Title: Identifying Health Risks from Family History: A Survey of Natural Language Processing Techniques

• Authors: Xiang Dai, Sarvnaz Karimi, Nathan O'Callaghan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.09997
• Pdf URL: https://arxiv.org/pdf/2403.09997
• Copy Paste: [[2403.09997]] Identifying Health Risks from Family History: A Survey of Natural Language Processing Techniques(https://arxiv.org/abs/2403.09997)
• Keywords: extraction
• Abstract: Electronic health records include information on patients' status and medical history, which could cover the history of diseases and disorders that could be hereditary. One important use of family history information is in precision health, where the goal is to keep the population healthy with preventative measures. Natural Language Processing (NLP) and machine learning techniques can assist with identifying information that could assist health professionals in identifying health risks before a condition is developed in their later years, saving lives and reducing healthcare costs. We survey the literature on the techniques from the NLP field that have been developed to utilise digital health records to identify risks of familial diseases. We highlight that rule-based methods are heavily investigated and are still actively used for family history extraction. Still, more recent efforts have been put into building neural models based on large-scale pre-trained language models. In addition to the areas where NLP has successfully been utilised, we also identify the areas where more research is needed to unlock the value of patients' records regarding data collection, task formulation and downstream applications.

Title: FBPT: A Fully Binary Point Transformer

• Authors: Zhixing Hou, Yuzhang Shang, Yan Yan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.09998
• Pdf URL: https://arxiv.org/pdf/2403.09998
• Copy Paste: [[2403.09998]] FBPT: A Fully Binary Point Transformer(https://arxiv.org/abs/2403.09998)
• Keywords: transformer
• Abstract: This paper presents a novel Fully Binary Point Cloud Transformer (FBPT) model which has the potential to be widely applied and expanded in the fields of robotics and mobile devices. By compressing the weights and activations of a 32-bit full-precision network to 1-bit binary values, the proposed binary point cloud Transformer network significantly reduces the storage footprint and computational resource requirements of neural network models for point cloud processing tasks, compared to full-precision point cloud networks. However, achieving a fully binary point cloud Transformer network, where all parts except the modules specific to the task are binary, poses challenges and bottlenecks in quantizing the activations of Q, K, V and self-attention in the attention module, as they do not adhere to simple probability distributions and can vary with input data. Furthermore, in our network, the binary attention module undergoes a degradation of the self-attention module due to the uniform distribution that occurs after the softmax operation. The primary focus of this paper is on addressing the performance degradation issue caused by the use of binary point cloud Transformer modules. We propose a novel binarization mechanism called dynamic-static hybridization. Specifically, our approach combines static binarization of the overall network model with fine granularity dynamic binarization of data-sensitive components. Furthermore, we make use of a novel hierarchical training scheme to obtain the optimal model and binarization parameters. These above improvements allow the proposed binarization method to outperform binarization methods applied to convolution neural networks when used in point cloud Transformer structures. To demonstrate the superiority of our algorithm, we conducted experiments on two different tasks: point cloud classification and place recognition.

Title: Federated Learning with Anomaly Detection via Gradient and Reconstruction Analysis

• Authors: Zahir Alsulaimawi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.10000
• Pdf URL: https://arxiv.org/pdf/2403.10000
• Copy Paste: [[2403.10000]] Federated Learning with Anomaly Detection via Gradient and Reconstruction Analysis(https://arxiv.org/abs/2403.10000)
• Keywords: security, privacy, attack, robust, federate
• Abstract: In the evolving landscape of Federated Learning (FL), the challenge of ensuring data integrity against poisoning attacks is paramount, particularly for applications demanding stringent privacy preservation. Traditional anomaly detection strategies often struggle to adapt to the distributed nature of FL, leaving a gap our research aims to bridge. We introduce a novel framework that synergizes gradient-based analysis with autoencoder-driven data reconstruction to detect and mitigate poisoned data with unprecedented precision. Our approach uniquely combines detecting anomalous gradient patterns with identifying reconstruction errors, significantly enhancing FL model security. Validated through extensive experiments on MNIST and CIFAR-10 datasets, our method outperforms existing solutions by 15\% in anomaly detection accuracy while maintaining a minimal false positive rate. This robust performance, consistent across varied data types and network sizes, underscores our framework's potential in securing FL deployments in critical domains such as healthcare and finance. By setting new benchmarks for anomaly detection within FL, our work paves the way for future advancements in distributed learning security.

Title: Visual Foundation Models Boost Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation

• Authors: Jingyi Xu, Weidong Yang, Lingdong Kong, Youquan Liu, Rui Zhang, Qingyuan Zhou, Ben Fei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10001
• Pdf URL: https://arxiv.org/pdf/2403.10001
• Copy Paste: [[2403.10001]] Visual Foundation Models Boost Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation(https://arxiv.org/abs/2403.10001)
• Keywords: segmentation
• Abstract: Unsupervised domain adaptation (UDA) is vital for alleviating the workload of labeling 3D point cloud data and mitigating the absence of labels when facing a newly defined domain. Various methods of utilizing images to enhance the performance of cross-domain 3D segmentation have recently emerged. However, the pseudo labels, which are generated from models trained on the source domain and provide additional supervised signals for the unseen domain, are inadequate when utilized for 3D segmentation due to their inherent noisiness and consequently restrict the accuracy of neural networks. With the advent of 2D visual foundation models (VFMs) and their abundant knowledge prior, we propose a novel pipeline VFMSeg to further enhance the cross-modal unsupervised domain adaptation framework by leveraging these models. In this work, we study how to harness the knowledge priors learned by VFMs to produce more accurate labels for unlabeled target domains and improve overall performance. We first utilize a multi-modal VFM, which is pre-trained on large scale image-text pairs, to provide supervised labels (VFM-PL) for images and point clouds from the target domain. Then, another VFM trained on fine-grained 2D masks is adopted to guide the generation of semantically augmented images and point clouds to enhance the performance of neural networks, which mix the data from source and target domains like view frustums (FrustumMixing). Finally, we merge class-wise prediction across modalities to produce more accurate annotations for unlabeled target domains. Our method is evaluated on various autonomous driving datasets and the results demonstrate a significant improvement for 3D segmentation task.

Title: ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real Images

• Authors: Xiangtian Xue, Jiasong Wu, Youyong Kong, Lotfi Senhadji, Huazhong Shu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10004
• Pdf URL: https://arxiv.org/pdf/2403.10004
• Copy Paste: [[2403.10004]] ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real Images(https://arxiv.org/abs/2403.10004)
• Keywords: diffusion, transformer, generative
• Abstract: We present a novel image editing scenario termed Text-grounded Object Generation (TOG), defined as generating a new object in the real image spatially conditioned by textual descriptions. Existing diffusion models exhibit limitations of spatial perception in complex real-world scenes, relying on additional modalities to enforce constraints, and TOG imposes heightened challenges on scene comprehension under the weak supervision of linguistic information. We propose a universal framework ST-LDM based on Swin-Transformer, which can be integrated into any latent diffusion model with training-free backward guidance. ST-LDM encompasses a global-perceptual autoencoder with adaptable compression scales and hierarchical visual features, parallel with deformable multimodal transformer to generate region-wise guidance for the subsequent denoising process. We transcend the limitation of traditional attention mechanisms that only focus on existing visual features by introducing deformable feature alignment to hierarchically refine spatial positioning fused with multi-scale visual and linguistic information. Extensive Experiments demonstrate that our model enhances the localization of attention mechanisms while preserving the generative capabilities inherent to diffusion models.

Title: Securing Federated Learning with Control-Flow Attestation: A Novel Framework for Enhanced Integrity and Resilience against Adversarial Attacks

• Authors: Zahir Alsulaimawi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.10005
• Pdf URL: https://arxiv.org/pdf/2403.10005
• Copy Paste: [[2403.10005]] Securing Federated Learning with Control-Flow Attestation: A Novel Framework for Enhanced Integrity and Resilience against Adversarial Attacks(https://arxiv.org/abs/2403.10005)
• Keywords: secure, security, privacy, attack, federate
• Abstract: The advent of Federated Learning (FL) as a distributed machine learning paradigm has introduced new cybersecurity challenges, notably adversarial attacks that threaten model integrity and participant privacy. This study proposes an innovative security framework inspired by Control-Flow Attestation (CFA) mechanisms, traditionally used in cybersecurity, to ensure software execution integrity. By integrating digital signatures and cryptographic hashing within the FL framework, we authenticate and verify the integrity of model updates across the network, effectively mitigating risks associated with model poisoning and adversarial interference. Our approach, novel in its application of CFA principles to FL, ensures contributions from participating nodes are authentic and untampered, thereby enhancing system resilience without compromising computational efficiency or model performance. Empirical evaluations on benchmark datasets, MNIST and CIFAR-10, demonstrate our framework's effectiveness, achieving a 100\% success rate in integrity verification and authentication and notable resilience against adversarial attacks. These results validate the proposed security enhancements and open avenues for more secure, reliable, and privacy-conscious distributed machine learning solutions. Our work bridges a critical gap between cybersecurity and distributed machine learning, offering a foundation for future advancements in secure FL.

Title: Linear optimal transport subspaces for point set classification

• Authors: Mohammad Shifat E Rabbi, Naqib Sad Pathan, Shiying Li, Yan Zhuang, Abu Hasnat Mohammad Rubaiyat, Gustavo K Rohde
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10015
• Pdf URL: https://arxiv.org/pdf/2403.10015
• Copy Paste: [[2403.10015]] Linear optimal transport subspaces for point set classification(https://arxiv.org/abs/2403.10015)
• Keywords: robust
• Abstract: Learning from point sets is an essential component in many computer vision and machine learning applications. Native, unordered, and permutation invariant set structure space is challenging to model, particularly for point set classification under spatial deformations. Here we propose a framework for classifying point sets experiencing certain types of spatial deformations, with a particular emphasis on datasets featuring affine deformations. Our approach employs the Linear Optimal Transport (LOT) transform to obtain a linear embedding of set-structured data. Utilizing the mathematical properties of the LOT transform, we demonstrate its capacity to accommodate variations in point sets by constructing a convex data space, effectively simplifying point set classification problems. Our method, which employs a nearest-subspace algorithm in the LOT space, demonstrates label efficiency, non-iterative behavior, and requires no hyper-parameter tuning. It achieves competitive accuracies compared to state-of-the-art methods across various point set classification tasks. Furthermore, our approach exhibits robustness in out-of-distribution scenarios where training and test distributions vary in terms of deformation magnitudes.

Title: Lost in Overlap: Exploring Watermark Collision in LLMs

• Authors: Yiyang Luo, Ke Lin, Chao Gu
• Subjects: cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.10020
• Pdf URL: https://arxiv.org/pdf/2403.10020
• Copy Paste: [[2403.10020]] Lost in Overlap: Exploring Watermark Collision in LLMs(https://arxiv.org/abs/2403.10020)
• Keywords: watermark, large language model
• Abstract: The proliferation of large language models (LLMs) in generating content raises concerns about text copyright. Watermarking methods, particularly logit-based approaches, embed imperceptible identifiers into text to address these challenges. However, the widespread use of watermarking across diverse LLMs has led to an inevitable issue known as watermark collision during common tasks like question answering and paraphrasing. This study focuses on dual watermark collisions, where two watermarks are present simultaneously in the same text. The research demonstrates that watermark collision poses a threat to detection performance for detectors of both upstream and downstream watermark algorithms.

Title: Time-Frequency Jointed Imperceptible Adversarial Attack to Brainprint Recognition with Deep Learning Models

• Authors: Hangjie Yi, Yuhang Ming, Dongjun Liu, Wanzeng Kong
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.10021
• Pdf URL: https://arxiv.org/pdf/2403.10021
• Copy Paste: [[2403.10021]] Time-Frequency Jointed Imperceptible Adversarial Attack to Brainprint Recognition with Deep Learning Models(https://arxiv.org/abs/2403.10021)
• Keywords: attack, biometric
• Abstract: EEG-based brainprint recognition with deep learning models has garnered much attention in biometric identification. Yet, studies have indicated vulnerability to adversarial attacks in deep learning models with EEG inputs. In this paper, we introduce a novel adversarial attack method that jointly attacks time-domain and frequency-domain EEG signals by employing wavelet transform. Different from most existing methods which only target time-domain EEG signals, our method not only takes advantage of the time-domain attack's potent adversarial strength but also benefits from the imperceptibility inherent in frequency-domain attack, achieving a better balance between attack performance and imperceptibility. Extensive experiments are conducted in both white- and grey-box scenarios and the results demonstrate that our attack method achieves state-of-the-art attack performance on three datasets and three deep-learning models. In the meanwhile, the perturbations in the signals attacked by our method are barely perceptible to the human visual system.

Title: Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers

• Authors: Sanghyeok Lee, Joonmyung Choi, Hyunwoo J. Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10030
• Pdf URL: https://arxiv.org/pdf/2403.10030
• Copy Paste: [[2403.10030]] Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers(https://arxiv.org/abs/2403.10030)
• Keywords: transformer
• Abstract: Vision Transformer (ViT) has emerged as a prominent backbone for computer vision. For more efficient ViTs, recent works lessen the quadratic cost of the self-attention layer by pruning or fusing the redundant tokens. However, these works faced the speed-accuracy trade-off caused by the loss of information. Here, we argue that token fusion needs to consider diverse relations between tokens to minimize information loss. In this paper, we propose a Multi-criteria Token Fusion (MCTF), that gradually fuses the tokens based on multi-criteria (e.g., similarity, informativeness, and size of fused tokens). Further, we utilize the one-step-ahead attention, which is the improved approach to capture the informativeness of the tokens. By training the model equipped with MCTF using a token reduction consistency, we achieve the best speed-accuracy trade-off in the image classification (ImageNet1K). Experimental results prove that MCTF consistently surpasses the previous reduction methods with and without training. Specifically, DeiT-T and DeiT-S with MCTF reduce FLOPs by about 44% while improving the performance (+0.5%, and +0.3%) over the base model, respectively. We also demonstrate the applicability of MCTF in various Vision Transformers (e.g., T2T-ViT, LV-ViT), achieving at least 31% speedup without performance degradation. Code is available at https://github.com/mlvlab/MCTF.

Title: SparseFusion: Efficient Sparse Multi-Modal Fusion Framework for Long-Range 3D Perception

• Authors: Yiheng Li, Hongyang Li, Zehao Huang, Hong Chang, Naiyan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10036
• Pdf URL: https://arxiv.org/pdf/2403.10036
• Copy Paste: [[2403.10036]] SparseFusion: Efficient Sparse Multi-Modal Fusion Framework for Long-Range 3D Perception(https://arxiv.org/abs/2403.10036)
• Keywords: transformer
• Abstract: Multi-modal 3D object detection has exhibited significant progress in recent years. However, most existing methods can hardly scale to long-range scenarios due to their reliance on dense 3D features, which substantially escalate computational demands and memory usage. In this paper, we introduce SparseFusion, a novel multi-modal fusion framework fully built upon sparse 3D features to facilitate efficient long-range perception. The core of our method is the Sparse View Transformer module, which selectively lifts regions of interest in 2D image space into the unified 3D space. The proposed module introduces sparsity from both semantic and geometric aspects which only fill grids that foreground objects potentially reside in. Comprehensive experiments have verified the efficiency and effectiveness of our framework in long-range 3D perception. Remarkably, on the long-range Argoverse2 dataset, SparseFusion reduces memory footprint and accelerates the inference by about two times compared to dense detectors. It also achieves state-of-the-art performance with mAP of 41.2% and CDS of 32.1%. The versatility of SparseFusion is also validated in the temporal object detection task and 3D lane detection task. Codes will be released upon acceptance.

Title: Knowledge Condensation and Reasoning for Knowledge-based VQA

• Authors: Dongze Hao, Jian Jia, Longteng Guo, Qunbo Wang, Te Yang, Yan Li, Yanhua Cheng, Bo Wang, Quan Chen, Han Li, Jing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10037
• Pdf URL: https://arxiv.org/pdf/2403.10037
• Copy Paste: [[2403.10037]] Knowledge Condensation and Reasoning for Knowledge-based VQA(https://arxiv.org/abs/2403.10037)
• Keywords: large language model
• Abstract: Knowledge-based visual question answering (KB-VQA) is a challenging task, which requires the model to leverage external knowledge for comprehending and answering questions grounded in visual content. Recent studies retrieve the knowledge passages from external knowledge bases and then use them to answer questions. However, these retrieved knowledge passages often contain irrelevant or noisy information, which limits the performance of the model. To address the challenge, we propose two synergistic models: Knowledge Condensation model and Knowledge Reasoning model. We condense the retrieved knowledge passages from two perspectives. First, we leverage the multimodal perception and reasoning ability of the visual-language models to distill concise knowledge concepts from retrieved lengthy passages, ensuring relevance to both the visual content and the question. Second, we leverage the text comprehension ability of the large language models to summarize and condense the passages into the knowledge essence which helps answer the question. These two types of condensed knowledge are then seamlessly integrated into our Knowledge Reasoning model, which judiciously navigates through the amalgamated information to arrive at the conclusive answer. Extensive experiments validate the superiority of the proposed method. Compared to previous methods, our method achieves state-of-the-art performance on knowledge-based VQA datasets (65.1% on OK-VQA and 60.1% on A-OKVQA) without resorting to the knowledge produced by GPT-3 (175B).

Title: Rethinking Low-quality Optical Flow in Unsupervised Surgical Instrument Segmentation

• Authors: Peiran Wu, Yang Liu, Jiayu Huo, Gongyu Zhang, Christos Bergeles, Rachel Sparks, Prokar Dasgupta, Alejandro Granados, Sebastien Ourselin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10039
• Pdf URL: https://arxiv.org/pdf/2403.10039
• Copy Paste: [[2403.10039]] Rethinking Low-quality Optical Flow in Unsupervised Surgical Instrument Segmentation(https://arxiv.org/abs/2403.10039)
• Keywords: segmentation
• Abstract: Video-based surgical instrument segmentation plays an important role in robot-assisted surgeries. Unlike supervised settings, unsupervised segmentation relies heavily on motion cues, which are challenging to discern due to the typically lower quality of optical flow in surgical footage compared to natural scenes. This presents a considerable burden for the advancement of unsupervised segmentation techniques. In our work, we address the challenge of enhancing model performance despite the inherent limitations of low-quality optical flow. Our methodology employs a three-pronged approach: extracting boundaries directly from the optical flow, selectively discarding frames with inferior flow quality, and employing a fine-tuning process with variable frame rates. We thoroughly evaluate our strategy on the EndoVis2017 VOS dataset and Endovis2017 Challenge dataset, where our model demonstrates promising results, achieving a mean Intersection-over-Union (mIoU) of 0.75 and 0.72, respectively. Our findings suggest that our approach can greatly decrease the need for manual annotations in clinical environments and may facilitate the annotation process for new datasets. The code is available at https://github.com/wpr1018001/Rethinking-Low-quality-Optical-Flow.git

Title: SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model

• Authors: Tao Wu, Xuewei Li, Zhongang Qi, Di Hu, Xintao Wang, Ying Shan, Xi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10044
• Pdf URL: https://arxiv.org/pdf/2403.10044
• Copy Paste: [[2403.10044]] SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model(https://arxiv.org/abs/2403.10044)
• Keywords: diffusion
• Abstract: Controllable spherical panoramic image generation holds substantial applicative potential across a variety of domains.However, it remains a challenging task due to the inherent spherical distortion and geometry characteristics, resulting in low-quality content generation.In this paper, we introduce a novel framework of SphereDiffusion to address these unique challenges, for better generating high-quality and precisely controllable spherical panoramic images.For the spherical distortion characteristic, we embed the semantics of the distorted object with text encoding, then explicitly construct the relationship with text-object correspondence to better use the pre-trained knowledge of the planar images.Meanwhile, we employ a deformable technique to mitigate the semantic deviation in latent space caused by spherical distortion.For the spherical geometry characteristic, in virtue of spherical rotation invariance, we improve the data diversity and optimization objectives in the training process, enabling the model to better learn the spherical geometry characteristic.Furthermore, we enhance the denoising process of the diffusion model, enabling it to effectively use the learned geometric characteristic to ensure the boundary continuity of the generated images.With these specific techniques, experiments on Structured3D dataset show that SphereDiffusion significantly improves the quality of controllable spherical image generation and relatively reduces around 35% FID on average.

Title: Towards Adversarially Robust Dataset Distillation by Curvature Regularization

• Authors: Eric Xue, Yijiang Li, Haoyang Liu, Yifan Shen, Haohan Wang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10045
• Pdf URL: https://arxiv.org/pdf/2403.10045
• Copy Paste: [[2403.10045]] Towards Adversarially Robust Dataset Distillation by Curvature Regularization(https://arxiv.org/abs/2403.10045)
• Keywords: attack, robust
• Abstract: Dataset distillation (DD) allows datasets to be distilled to fractions of their original size while preserving the rich distributional information so that models trained on the distilled datasets can achieve a comparable accuracy while saving significant computational loads. Recent research in this area has been focusing on improving the accuracy of models trained on distilled datasets. In this paper, we aim to explore a new perspective of DD. We study how to embed adversarial robustness in distilled datasets, so that models trained on these datasets maintain the high accuracy and meanwhile acquire better adversarial robustness. We propose a new method that achieves this goal by incorporating curvature regularization into the distillation process with much less computational overhead than standard adversarial training. Extensive empirical experiments suggest that our method not only outperforms standard adversarial training on both accuracy and robustness with less computation overhead but is also capable of generating robust distilled datasets that can withstand various adversarial attacks.

Title: TextBlockV2: Towards Precise-Detection-Free Scene Text Spotting with Pre-trained Language Model

• Authors: Jiahao Lyu, Jin Wei, Gangyan Zeng, Zeng Li, Enze Xie, Wei Wang, Yu Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10047
• Pdf URL: https://arxiv.org/pdf/2403.10047
• Copy Paste: [[2403.10047]] TextBlockV2: Towards Precise-Detection-Free Scene Text Spotting with Pre-trained Language Model(https://arxiv.org/abs/2403.10047)
• Keywords: large language model
• Abstract: Existing scene text spotters are designed to locate and transcribe texts from images. However, it is challenging for a spotter to achieve precise detection and recognition of scene texts simultaneously. Inspired by the glimpse-focus spotting pipeline of human beings and impressive performances of Pre-trained Language Models (PLMs) on visual tasks, we ask: 1) "Can machines spot texts without precise detection just like human beings?", and if yes, 2) "Is text block another alternative for scene text spotting other than word or character?" To this end, our proposed scene text spotter leverages advanced PLMs to enhance performance without fine-grained detection. Specifically, we first use a simple detector for block-level text detection to obtain rough positional information. Then, we finetune a PLM using a large-scale OCR dataset to achieve accurate recognition. Benefiting from the comprehensive language knowledge gained during the pre-training phase, the PLM-based recognition module effectively handles complex scenarios, including multi-line, reversed, occluded, and incomplete-detection texts. Taking advantage of the fine-tuned language model on scene recognition benchmarks and the paradigm of text block detection, extensive experiments demonstrate the superior performance of our scene text spotter across multiple public benchmarks. Additionally, we attempt to spot texts directly from an entire scene image to demonstrate the potential of PLMs, even Large Language Models (LLMs).

Title: Control and Automation for Industrial Production Storage Zone: Generation of Optimal Route Using Image Processing

• Authors: Bejamin A. Huerfano, Fernando Jimenez
• Subjects: cs.CV, eess.IV, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.10054
• Pdf URL: https://arxiv.org/pdf/2403.10054
• Copy Paste: [[2403.10054]] Control and Automation for Industrial Production Storage Zone: Generation of Optimal Route Using Image Processing(https://arxiv.org/abs/2403.10054)
• Keywords: segmentation
• Abstract: Digital image processing (DIP) is of great importance in validating and guaranteeing parameters that ensure the quality of mass-produced products. Therefore, this article focused on developing an industrial automation method for a zone of a production line model using the DIP. The neo-cascade methodology employed allowed for defining each of the stages in an adequate way, ensuring the inclusion of the relevant methods for its development, which finally incurred in the modeling, design, implementation, and testing of an optimal route generation system for a warehouse area, using DIP with optimization guidelines, in conjunction with an embedded platform and the connection to programmable logic controllers (PLCs) for its execution. The system was based on the OpenCV library; tool focused on artificial vision, which was implemented on an object-oriented programming (OOP) platform based on Java language. It generated the optimal route for the automation of processes in a scale warehouse area, using the segmentation of objects and the optimization of flow in networks as pillars, ending with the connection to PLCs as a method of action, which in case of implementation would eliminate constraints such as process inefficiency, the use of manpower to perform these tasks, inadequate use of resources, among others

Title: Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning

• Authors: Yongquan He, Xuancheng Huang, Minghao Tang, Lingxun Meng, Xiang Li, Wei Lin, Wenyuan Zhang, Yifu Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10056
• Pdf URL: https://arxiv.org/pdf/2403.10056
• Copy Paste: [[2403.10056]] Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning(https://arxiv.org/abs/2403.10056)
• Keywords: large language model
• Abstract: Instruction tuning for large language models (LLMs) can drive them to produce results consistent with human goals in specific downstream tasks. However, the process of continual instruction tuning (CIT) for LLMs may bring about the catastrophic forgetting (CF) problem, where previously learned abilities are degraded. Recent methods try to alleviate the CF problem by modifying models or replaying data, which may only remember the surface-level pattern of instructions and get confused on held-out tasks. In this paper, we propose a novel continual instruction tuning method based on Key-part Information Gain (KPIG). Our method computes the information gain on masked parts to dynamically replay data and refine the training objective, which enables LLMs to capture task-aware information relevant to the correct response and alleviate overfitting to general descriptions in instructions. In addition, we propose two metrics, P-score and V-score, to measure the generalization and instruction-following abilities of LLMs. Experiments demonstrate our method achieves superior performance on both seen and held-out tasks.

Title: RID-TWIN: An end-to-end pipeline for automatic face de-identification in videos

• Authors: Anirban Mukherjee, Monjoy Narayan Choudhury, Dinesh Babu Jayagopi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10058
• Pdf URL: https://arxiv.org/pdf/2403.10058
• Copy Paste: [[2403.10058]] RID-TWIN: An end-to-end pipeline for automatic face de-identification in videos(https://arxiv.org/abs/2403.10058)
• Keywords: privacy, generative
• Abstract: Face de-identification in videos is a challenging task in the domain of computer vision, primarily used in privacy-preserving applications. Despite the considerable progress achieved through generative vision models, there remain multiple challenges in the latest approaches. They lack a comprehensive discussion and evaluation of aspects such as realism, temporal coherence, and preservation of non-identifiable features. In our work, we propose RID-Twin: a novel pipeline that leverages the state-of-the-art generative models, and decouples identity from motion to perform automatic face de-identification in videos. We investigate the task from a holistic point of view and discuss how our approach addresses the pertinent existing challenges in this domain. We evaluate the performance of our methodology on the widely employed VoxCeleb2 dataset, and also a custom dataset designed to accommodate the limitations of certain behavioral variations absent in the VoxCeleb2 dataset. We discuss the implications and advantages of our work and suggest directions for future research.

Title: Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling

• Authors: Baoquan Zhang, Huaibin Wang, Luo Chuyao, Xutao Li, Liang Guotao, Yunming Ye, Xiaochen Qi, Yao He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10071
• Pdf URL: https://arxiv.org/pdf/2403.10071
• Copy Paste: [[2403.10071]] Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling(https://arxiv.org/abs/2403.10071)
• Keywords: robust
• Abstract: Vector-Quantized Image Modeling (VQIM) is a fundamental research problem in image synthesis, which aims to represent an image with a discrete token sequence. Existing studies effectively address this problem by learning a discrete codebook from scratch and in a code-independent manner to quantize continuous representations into discrete tokens. However, learning a codebook from scratch and in a code-independent manner is highly challenging, which may be a key reason causing codebook collapse, i.e., some code vectors can rarely be optimized without regard to the relationship between codes and good codebook priors such that die off finally. In this paper, inspired by pretrained language models, we find that these language models have actually pretrained a superior codebook via a large number of text corpus, but such information is rarely exploited in VQIM. To this end, we propose a novel codebook transfer framework with part-of-speech, called VQCT, which aims to transfer a well-trained codebook from pretrained language models to VQIM for robust codebook learning. Specifically, we first introduce a pretrained codebook from language models and part-of-speech knowledge as priors. Then, we construct a vision-related codebook with these priors for achieving codebook transfer. Finally, a novel codebook transfer network is designed to exploit abundant semantic relationships between codes contained in pretrained codebooks for robust VQIM codebook learning. Experimental results on four datasets show that our VQCT method achieves superior VQIM performance over previous state-of-the-art methods.

Title: Revisiting Adversarial Training under Long-Tailed Distributions

• Authors: Xinli Yue, Ningping Mou, Qian Wang, Lingchen Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10073
• Pdf URL: https://arxiv.org/pdf/2403.10073
• Copy Paste: [[2403.10073]] Revisiting Adversarial Training under Long-Tailed Distributions(https://arxiv.org/abs/2403.10073)
• Keywords: attack, robust
• Abstract: Deep neural networks are vulnerable to adversarial attacks, often leading to erroneous outputs. Adversarial training has been recognized as one of the most effective methods to counter such attacks. However, existing adversarial training techniques have predominantly been tested on balanced datasets, whereas real-world data often exhibit a long-tailed distribution, casting doubt on the efficacy of these methods in practical scenarios. In this paper, we delve into adversarial training under long-tailed distributions. Through an analysis of the previous work "RoBal", we discover that utilizing Balanced Softmax Loss alone can achieve performance comparable to the complete RoBal approach while significantly reducing training overheads. Additionally, we reveal that, similar to uniform distributions, adversarial training under long-tailed distributions also suffers from robust overfitting. To address this, we explore data augmentation as a solution and unexpectedly discover that, unlike results obtained with balanced data, data augmentation not only effectively alleviates robust overfitting but also significantly improves robustness. We further investigate the reasons behind the improvement of robustness through data augmentation and identify that it is attributable to the increased diversity of examples. Extensive experiments further corroborate that data augmentation alone can significantly improve robustness. Finally, building on these findings, we demonstrate that compared to RoBal, the combination of BSL and data augmentation leads to a +6.66% improvement in model robustness under AutoAttack on CIFAR-10-LT. Our code is available at https://github.com/NISPLab/AT-BSL .

Title: A survey of synthetic data augmentation methods in computer vision

• Authors: Alhassan Mumuni, Fuseini Mumuni, Nana Kobina Gerrar
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10075
• Pdf URL: https://arxiv.org/pdf/2403.10075
• Copy Paste: [[2403.10075]] A survey of synthetic data augmentation methods in computer vision(https://arxiv.org/abs/2403.10075)
• Keywords: generative
• Abstract: The standard approach to tackling computer vision problems is to train deep convolutional neural network (CNN) models using large-scale image datasets which are representative of the target task. However, in many scenarios, it is often challenging to obtain sufficient image data for the target task. Data augmentation is a way to mitigate this challenge. A common practice is to explicitly transform existing images in desired ways so as to create the required volume and variability of training data necessary to achieve good generalization performance. In situations where data for the target domain is not accessible, a viable workaround is to synthesize training data from scratch--i.e., synthetic data augmentation. This paper presents an extensive review of synthetic data augmentation techniques. It covers data synthesis approaches based on realistic 3D graphics modeling, neural style transfer (NST), differential neural rendering, and generative artificial intelligence (AI) techniques such as generative adversarial networks (GANs) and variational autoencoders (VAEs). For each of these classes of methods, we focus on the important data generation and augmentation techniques, general scope of application and specific use-cases, as well as existing limitations and possible workarounds. Additionally, we provide a summary of common synthetic datasets for training computer vision models, highlighting the main features, application domains and supported tasks. Finally, we discuss the effectiveness of synthetic data augmentation methods. Since this is the first paper to explore synthetic data augmentation methods in great detail, we are hoping to equip readers with the necessary background information and in-depth knowledge of existing methods and their attendant issues.

Title: Benchmarking Adversarial Robustness of Image Shadow Removal with Shadow-adaptive Attacks

• Authors: Chong Wang, Yi Yu, Lanqing Guo, Bihan Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10076
• Pdf URL: https://arxiv.org/pdf/2403.10076
• Copy Paste: [[2403.10076]] Benchmarking Adversarial Robustness of Image Shadow Removal with Shadow-adaptive Attacks(https://arxiv.org/abs/2403.10076)
• Keywords: attack, robust
• Abstract: Shadow removal is a task aimed at erasing regional shadows present in images and reinstating visually pleasing natural scenes with consistent illumination. While recent deep learning techniques have demonstrated impressive performance in image shadow removal, their robustness against adversarial attacks remains largely unexplored. Furthermore, many existing attack frameworks typically allocate a uniform budget for perturbations across the entire input image, which may not be suitable for attacking shadow images. This is primarily due to the unique characteristic of spatially varying illumination within shadow images. In this paper, we propose a novel approach, called shadow-adaptive adversarial attack. Different from standard adversarial attacks, our attack budget is adjusted based on the pixel intensity in different regions of shadow images. Consequently, the optimized adversarial noise in the shadowed regions becomes visually less perceptible while permitting a greater tolerance for perturbations in non-shadow regions. The proposed shadow-adaptive attacks naturally align with the varying illumination distribution in shadow images, resulting in perturbations that are less conspicuous. Building on this, we conduct a comprehensive empirical evaluation of existing shadow removal methods, subjecting them to various levels of attack on publicly available datasets.

Title: DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models

• Authors: Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, Yiqun Liu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.10081
• Pdf URL: https://arxiv.org/pdf/2403.10081
• Copy Paste: [[2403.10081]] DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models(https://arxiv.org/abs/2403.10081)
• Keywords: large language model
• Abstract: Dynamic retrieval augmented generation (RAG) paradigm actively decides when and what to retrieve during the text generation process of Large Language Models (LLMs). There are two key elements of this paradigm: identifying the optimal moment to activate the retrieval module (deciding when to retrieve) and crafting the appropriate query once retrieval is triggered (determining what to retrieve). However, current dynamic RAG methods fall short in both aspects. Firstly, the strategies for deciding when to retrieve often rely on static rules. Moreover, the strategies for deciding what to retrieve typically limit themselves to the LLM's most recent sentence or the last few tokens, while the LLM's real-time information needs may span across the entire context. To overcome these limitations, we introduce a new framework, DRAGIN, i.e., Dynamic Retrieval Augmented Generation based on the real-time Information Needs of LLMs. Our framework is specifically designed to make decisions on when and what to retrieve based on the LLM's real-time information needs during the text generation process. We evaluate DRAGIN along with existing methods comprehensively over 4 knowledge-intensive generation datasets. Experimental results show that DRAGIN achieves superior performance on all tasks, demonstrating the effectiveness of our method. We have open-sourced all the code, data, and models in GitHub: https://github.com/oneal2000/DRAGIN/tree/main

Title: CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Manner

• Authors: Tingbing Yan, Wenzheng Zeng, Yang Xiao, Xingyu Tong, Bo Tan, Zhiwen Fang, Zhiguo Cao, Joey Tianyi Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10082
• Pdf URL: https://arxiv.org/pdf/2403.10082
• Copy Paste: [[2403.10082]] CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Manner(https://arxiv.org/abs/2403.10082)
• Keywords: large language model
• Abstract: Most existing one-shot skeleton-based action recognition focuses on raw low-level information (e.g., joint location), and may suffer from local information loss and low generalization ability. To alleviate these, we propose to leverage text description generated from large language models (LLM) that contain high-level human knowledge, to guide feature learning, in a global-local-global way. Particularly, during training, we design $2$ prompts to gain global and local text descriptions of each action from an LLM. We first utilize the global text description to guide the skeleton encoder focus on informative joints (i.e.,global-to-local). Then we build non-local interaction between local text and joint features, to form the final global representation (i.e., local-to-global). To mitigate the asymmetry issue between the training and inference phases, we further design a dual-branch architecture that allows the model to perform novel class inference without any text input, also making the additional inference cost neglectable compared with the base skeleton encoder. Extensive experiments on three different benchmarks show that CrossGLG consistently outperforms the existing SOTA methods with large margins, and the inference cost (model size) is only $2.8$\% than the previous SOTA. CrossGLG can also serve as a plug-and-play module that can substantially enhance the performance of different SOTA skeleton encoders with a neglectable cost during inference. The source code will be released soon.

Title: VRHCF: Cross-Source Point Cloud Registration via Voxel Representation and Hierarchical Correspondence Filtering

• Authors: Guiyu Zhao, Zewen Du, Zhentao Guo, Hongbin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10085
• Pdf URL: https://arxiv.org/pdf/2403.10085
• Copy Paste: [[2403.10085]] VRHCF: Cross-Source Point Cloud Registration via Voxel Representation and Hierarchical Correspondence Filtering(https://arxiv.org/abs/2403.10085)
• Keywords: robust
• Abstract: Addressing the challenges posed by the substantial gap in point cloud data collected from diverse sensors, achieving robust cross-source point cloud registration becomes a formidable task. In response, we present a novel framework for point cloud registration with broad applicability, suitable for both homologous and cross-source registration scenarios. To tackle the issues arising from different densities and distributions in cross-source point cloud data, we introduce a feature representation based on spherical voxels. Furthermore, addressing the challenge of numerous outliers and mismatches in cross-source registration, we propose a hierarchical correspondence filtering approach. This method progressively filters out mismatches, yielding a set of high-quality correspondences. Our method exhibits versatile applicability and excels in both traditional homologous registration and challenging cross-source registration scenarios. Specifically, in homologous registration using the 3DMatch dataset, we achieve the highest registration recall of 95.1% and an inlier ratio of 87.8%. In cross-source point cloud registration, our method attains the best RR on the 3DCSR dataset, demonstrating a 9.3 percentage points improvement. The code is available at https://github.com/GuiyuZhao/VRHCF.

Title: Monkeypox disease recognition model based on improved SE-InceptionV3

• Authors: Junzhuo Chen, Zonghan Lu, Shitong Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10087
• Pdf URL: https://arxiv.org/pdf/2403.10087
• Copy Paste: [[2403.10087]] Monkeypox disease recognition model based on improved SE-InceptionV3(https://arxiv.org/abs/2403.10087)
• Keywords: robust
• Abstract: In the wake of the global spread of monkeypox, accurate disease recognition has become crucial. This study introduces an improved SE-InceptionV3 model, embedding the SENet module and incorporating L2 regularization into the InceptionV3 framework to enhance monkeypox disease detection. Utilizing the Kaggle monkeypox dataset, which includes images of monkeypox and similar skin conditions, our model demonstrates a noteworthy accuracy of 96.71% on the test set, outperforming conventional methods and deep learning models. The SENet modules channel attention mechanism significantly elevates feature representation, while L2 regularization ensures robust generalization. Extensive experiments validate the models superiority in precision, recall, and F1 score, highlighting its effectiveness in differentiating monkeypox lesions in diverse and complex cases. The study not only provides insights into the application of advanced CNN architectures in medical diagnostics but also opens avenues for further research in model optimization and hyperparameter tuning for enhanced disease recognition. https://github.com/jzc777/SE-inceptionV3-L2

Title: Specification and Enforcement of Activity Dependency Policies using XACML

• Authors: Tanjila Mawla, Maanak Gupta, Ravi Sandhu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.10092
• Pdf URL: https://arxiv.org/pdf/2403.10092
• Copy Paste: [[2403.10092]] Specification and Enforcement of Activity Dependency Policies using XACML(https://arxiv.org/abs/2403.10092)
• Keywords: security
• Abstract: The evolving smart and interconnected systems are designed to operate with minimal human intervention. Devices within these smart systems often engage in prolonged operations based on sensor data and contextual factors. Recently, an Activity-Centric Access Control (ACAC) model has been introduced to regulate these prolonged operations, referred to as activities, which undergo state changes over extended duration of time. Dependencies among different activities can influence and restrict the execution of one another, necessitating active and real-time monitoring of the dependencies between activities to prevent security violation. In the ACAC model, the activity dependencies, denoted as "D", is considered as a decision parameter for controlling a requested activity. These dependencies must be evaluated throughout all phases of an activity's life cycle. To ensure the consistency of access control rules across diverse domains and applications, a standard policy language is essential. We propose a policy framework adapting the widely-used eXtensible Access Control Markup Language (XACML) , referred to as $\mathrm{XACML_{AD}}$, to specify the activity dependency policies. This work involves extending the syntax and semantics of XACML by introducing new elements to check dependent activities' states and handle state updates on dependent activities. In addition to the language extension, we present the enforcement architecture and data flow model of evaluating policies for activity dependencies. The integration of the proposed $\mathrm{XACML_{AD}}$ policy framework and the enforcement of the policies supports dependency evaluation, necessary updates and continuous enforcement of policies to control an activity throughout its life cycle. We implement the enforcement architecture exploiting the $\mathrm{XACML_{AD}}$ policy framework and discuss the performance evaluation results.

Title: RangeLDM: Fast Realistic LiDAR Point Cloud Generation

• Authors: Qianjiang Hu, Zhimin Zhang, Wei Hu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.10094
• Pdf URL: https://arxiv.org/pdf/2403.10094
• Copy Paste: [[2403.10094]] RangeLDM: Fast Realistic LiDAR Point Cloud Generation(https://arxiv.org/abs/2403.10094)
• Keywords: robust, diffusion, generative
• Abstract: Autonomous driving demands high-quality LiDAR data, yet the cost of physical LiDAR sensors presents a significant scaling-up challenge. While recent efforts have explored deep generative models to address this issue, they often consume substantial computational resources with slow generation speeds while suffering from a lack of realism. To address these limitations, we introduce RangeLDM, a novel approach for rapidly generating high-quality range-view LiDAR point clouds via latent diffusion models. We achieve this by correcting range-view data distribution for accurate projection from point clouds to range images via Hough voting, which has a critical impact on generative learning. We then compress the range images into a latent space with a variational autoencoder, and leverage a diffusion model to enhance expressivity. Additionally, we instruct the model to preserve 3D structural fidelity by devising a range-guided discriminator. Experimental results on KITTI-360 and nuScenes datasets demonstrate both the robust expressiveness and fast speed of our LiDAR point cloud generation.

Title: DiffMAC: Diffusion Manifold Hallucination Correction for High Generalization Blind Face Restoration

• Authors: Nan Gao, Jia Li, Huaibo Huang, Zhi Zeng, Ke Shang, Shuwu Zhang, Ran He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10098
• Pdf URL: https://arxiv.org/pdf/2403.10098
• Copy Paste: [[2403.10098]] DiffMAC: Diffusion Manifold Hallucination Correction for High Generalization Blind Face Restoration(https://arxiv.org/abs/2403.10098)
• Keywords: diffusion
• Abstract: Blind face restoration (BFR) is a highly challenging problem due to the uncertainty of degradation patterns. Current methods have low generalization across photorealistic and heterogeneous domains. In this paper, we propose a Diffusion-Information-Diffusion (DID) framework to tackle diffusion manifold hallucination correction (DiffMAC), which achieves high-generalization face restoration in diverse degraded scenes and heterogeneous domains. Specifically, the first diffusion stage aligns the restored face with spatial feature embedding of the low-quality face based on AdaIN, which synthesizes degradation-removal results but with uncontrollable artifacts for some hard cases. Based on Stage I, Stage II considers information compression using manifold information bottleneck (MIB) and finetunes the first diffusion model to improve facial fidelity. DiffMAC effectively fights against blind degradation patterns and synthesizes high-quality faces with attribute and identity consistencies. Experimental results demonstrate the superiority of DiffMAC over state-of-the-art methods, with a high degree of generalization in real-world and heterogeneous settings. The source code and models will be public.

Title: Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoning

• Authors: Hang Zhang, Wenxiao Zhang, Haoxuan Qu, Jun Liu
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.10107
• Pdf URL: https://arxiv.org/pdf/2403.10107
• Copy Paste: [[2403.10107]] Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoning(https://arxiv.org/abs/2403.10107)
• Keywords: large language model
• Abstract: Human-centered dynamic scene understanding plays a pivotal role in enhancing the capability of robotic and autonomous systems, in which Video-based Human-Object Interaction (V-HOI) detection is a crucial task in semantic scene understanding, aimed at comprehensively understanding HOI relationships within a video to benefit the behavioral decisions of mobile robots and autonomous driving systems. Although previous V-HOI detection models have made significant strides in accurate detection on specific datasets, they still lack the general reasoning ability like human beings to effectively induce HOI relationships. In this study, we propose V-HOI Multi-LLMs Collaborated Reasoning (V-HOI MLCR), a novel framework consisting of a series of plug-and-play modules that could facilitate the performance of current V-HOI detection models by leveraging the strong reasoning ability of different off-the-shelf pre-trained large language models (LLMs). We design a two-stage collaboration system of different LLMs for the V-HOI task. Specifically, in the first stage, we design a Cross-Agents Reasoning scheme to leverage the LLM conduct reasoning from different aspects. In the second stage, we perform Multi-LLMs Debate to get the final reasoning answer based on the different knowledge in different LLMs. Additionally, we devise an auxiliary training strategy that utilizes CLIP, a large vision-language model to enhance the base V-HOI models' discriminative ability to better cooperate with LLMs. We validate the superiority of our design by demonstrating its effectiveness in improving the prediction accuracy of the base V-HOI model via reasoning from multiple perspectives.

Title: Instance-optimal Clipping for Summation Problems in the Shuffle Model of Differential Privacy

• Authors: Wei Dong, Qiyao Luo, Giulia Fanti, Elaine Shi, Ke Yi
• Subjects: cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2403.10116
• Pdf URL: https://arxiv.org/pdf/2403.10116
• Copy Paste: [[2403.10116]] Instance-optimal Clipping for Summation Problems in the Shuffle Model of Differential Privacy(https://arxiv.org/abs/2403.10116)
• Keywords: privacy
• Abstract: Differentially private mechanisms achieving worst-case optimal error bounds (e.g., the classical Laplace mechanism) are well-studied in the literature. However, when typical data are far from the worst case, \emph{instance-specific} error bounds -- which depend on the largest value in the dataset -- are more meaningful. For example, consider the sum estimation problem, where each user has an integer $x_i$ from the domain $\{0,1,\dots,U\}$ and we wish to estimate $\sum_i x_i$. This has a worst-case optimal error of $O(U/\varepsilon)$, while recent work has shown that the clipping mechanism can achieve an instance-optimal error of $O(\max_i x_i \cdot \log\log U /\varepsilon)$. Under the shuffle model, known instance-optimal protocols are less communication-efficient. The clipping mechanism also works in the shuffle model, but requires two rounds: Round one finds the clipping threshold, and round two does the clipping and computes the noisy sum of the clipped data. In this paper, we show how these two seemingly sequential steps can be done simultaneously in one round using just $1+o(1)$ messages per user, while maintaining the instance-optimal error bound. We also extend our technique to the high-dimensional sum estimation problem and sparse vector aggregation (a.k.a. frequency estimation under user-level differential privacy). Our experiments show order-of-magnitude improvements of our protocols in terms of error compared with prior work.

Title: Depth-induced Saliency Comparison Network for Diagnosis of Alzheimer's Disease via Jointly Analysis of Visual Stimuli and Eye Movements

• Authors: Yu Liu, Wenlin Zhang, Shaochu Wang, Fangyu Zuo, Peiguang Jing, Yong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10124
• Pdf URL: https://arxiv.org/pdf/2403.10124
• Copy Paste: [[2403.10124]] Depth-induced Saliency Comparison Network for Diagnosis of Alzheimer's Disease via Jointly Analysis of Visual Stimuli and Eye Movements(https://arxiv.org/abs/2403.10124)
• Keywords: robust
• Abstract: Early diagnosis of Alzheimer's Disease (AD) is very important for following medical treatments, and eye movements under special visual stimuli may serve as a potential non-invasive biomarker for detecting cognitive abnormalities of AD patients. In this paper, we propose an Depth-induced saliency comparison network (DISCN) for eye movement analysis, which may be used for diagnosis the Alzheimers disease. In DISCN, a salient attention module fuses normal eye movements with RGB and depth maps of visual stimuli using hierarchical salient attention (SAA) to evaluate comprehensive saliency maps, which contain information from both visual stimuli and normal eye movement behaviors. In addition, we introduce serial attention module (SEA) to emphasis the most abnormal eye movement behaviors to reduce personal bias for a more robust result. According to our experiments, the DISCN achieves consistent validity in classifying the eye movements between the AD patients and normal controls.

Title: TransLandSeg: A Transfer Learning Approach for Landslide Semantic Segmentation Based on Vision Foundation Model

• Authors: Changhong Hou, Junchuan Yu, Daqing Ge, Liu Yang, Laidian Xi, Yunxuan Pang, Yi Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10127
• Pdf URL: https://arxiv.org/pdf/2403.10127
• Copy Paste: [[2403.10127]] TransLandSeg: A Transfer Learning Approach for Landslide Semantic Segmentation Based on Vision Foundation Model(https://arxiv.org/abs/2403.10127)
• Keywords: segmentation
• Abstract: Landslides are one of the most destructive natural disasters in the world, posing a serious threat to human life and safety. The development of foundation models has provided a new research paradigm for large-scale landslide detection. The Segment Anything Model (SAM) has garnered widespread attention in the field of image segmentation. However, our experiment found that SAM performed poorly in the task of landslide segmentation. We propose TransLandSeg, which is a transfer learning approach for landslide semantic segmentation based on a vision foundation model (VFM). TransLandSeg outperforms traditional semantic segmentation models on both the Landslide4Sense dataset and the Bijie landslide dataset. Our proposed adaptive transfer learning (ATL) architecture enables the powerful segmentation capability of SAM to be transferred to landslide detection by training only 1.3% of the number of the parameters of SAM, which greatly improves the training efficiency of the model. Finally we also conducted ablation experiments on models with different ATL structures, concluded that the deployment location and residual connection of ATL play an important role in TransLandSeg accuracy improvement.

Title: RAFT: Adapting Language Model to Domain Specific RAG

• Authors: Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, Joseph E. Gonzalez
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10131
• Pdf URL: https://arxiv.org/pdf/2403.10131
• Copy Paste: [[2403.10131]] RAFT: Adapting Language Model to Domain Specific RAG(https://arxiv.org/abs/2403.10131)
• Keywords: large language model
• Abstract: Pretraining Large Language Models (LLMs) on large corpora of textual data is now a standard paradigm. When using these LLMs for many downstream applications, it is common to additionally bake in new knowledge (e.g., time-critical news, or private domain knowledge) into the pretrained model either through RAG-based-prompting, or fine-tuning. However, the optimal methodology for the model to gain such new knowledge remains an open question. In this paper, we present Retrieval Augmented FineTuning (RAFT), a training recipe that improves the model's ability to answer questions in a "open-book" in-domain settings. In RAFT, given a question, and a set of retrieved documents, we train the model to ignore those documents that don't help in answering the question, which we call, distractor documents. RAFT accomplishes this by citing verbatim the right sequence from the relevant document that would help answer the question. This coupled with RAFT's chain-of-thought-style response helps improve the model's ability to reason. In domain-specific RAG, RAFT consistently improves the model's performance across PubMed, HotpotQA, and Gorilla datasets, presenting a post-training recipe to improve pre-trained LLMs to in-domain RAG. RAFT's code and demo are open-sourced at github.com/ShishirPatil/gorilla.

Title: E4C: Enhance Editability for Text-Based Image Editing by Harnessing Efficient CLIP Guidance

• Authors: Tianrui Huang, Pu Cao, Lu Yang, Chun Liu, Mengjie Hu, Zhiwei Liu, Qing Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10133
• Pdf URL: https://arxiv.org/pdf/2403.10133
• Copy Paste: [[2403.10133]] E4C: Enhance Editability for Text-Based Image Editing by Harnessing Efficient CLIP Guidance(https://arxiv.org/abs/2403.10133)
• Keywords: diffusion
• Abstract: Diffusion-based image editing is a composite process of preserving the source image content and generating new content or applying modifications. While current editing approaches have made improvements under text guidance, most of them have only focused on preserving the information of the input image, disregarding the importance of editability and alignment to the target prompt. In this paper, we prioritize the editability by proposing a zero-shot image editing method, named \textbf{E}nhance \textbf{E}ditability for text-based image \textbf{E}diting via \textbf{E}fficient \textbf{C}LIP guidance (\textbf{E4C}), which only requires inference-stage optimization to explicitly enhance the edibility and text alignment. Specifically, we develop a unified dual-branch feature-sharing pipeline that enables the preservation of the structure or texture of the source image while allowing the other to be adapted based on the editing task. We further integrate CLIP guidance into our pipeline by utilizing our novel random-gateway optimization mechanism to efficiently enhance the semantic alignment with the target prompt. Comprehensive quantitative and qualitative experiments demonstrate that our method effectively resolves the text alignment issues prevalent in existing methods while maintaining the fidelity to the source image, and performs well across a wide range of editing tasks.

Title: NLP Verification: Towards a General Methodology for Certifying Robustness

• Authors: Marco Casadio, Tanvi Dinkar, Ekaterina Komendantskaya, Luca Arnaboldi, Omri Isac, Matthew L. Daggitt, Guy Katz, Verena Rieser, Oliver Lemon
• Subjects: cs.CL, cs.AI, cs.LG, cs.LO, cs.PL
• Abstract URL: https://arxiv.org/abs/2403.10144
• Pdf URL: https://arxiv.org/pdf/2403.10144
• Copy Paste: [[2403.10144]] NLP Verification: Towards a General Methodology for Certifying Robustness(https://arxiv.org/abs/2403.10144)
• Keywords: attack, robust
• Abstract: Deep neural networks have exhibited substantial success in the field of Natural Language Processing (NLP) and ensuring their safety and reliability is crucial: there are safety critical contexts where such models must be robust to variability or attack, and give guarantees over their output. Unlike Computer Vision, NLP lacks a unified verification methodology and, despite recent advancements in literature, they are often light on the pragmatical issues of NLP verification. In this paper, we make an attempt to distil and evaluate general components of an NLP verification pipeline, that emerges from the progress in the field to date. Our contributions are two-fold. Firstly, we give a general characterisation of verifiable subspaces that result from embedding sentences into continuous spaces. We identify, and give an effective method to deal with, the technical challenge of semantic generalisability of verified subspaces; and propose it as a standard metric in the NLP verification pipelines (alongside with the standard metrics of model accuracy and model verifiability). Secondly, we propose a general methodology to analyse the effect of the embedding gap, a problem that refers to the discrepancy between verification of geometric subpspaces on the one hand, and semantic meaning of sentences which the geometric subspaces are supposed to represent, on the other hand. In extreme cases, poor choices in embedding of sentences may invalidate verification results. We propose a number of practical NLP methods that can help to identify the effects of the embedding gap; and in particular we propose the metric of falsifiability of semantic subpspaces as another fundamental metric to be reported as part of the NLP verification pipeline. We believe that together these general principles pave the way towards a more consolidated and effective development of this new domain.

Title: GGRt: Towards Generalizable 3D Gaussians without Pose Priors in Real-Time

• Authors: Hao Li, Yuanyuan Gao, Dingwen Zhang, Chenming Wu, Yalun Dai, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, Junwei Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10147
• Pdf URL: https://arxiv.org/pdf/2403.10147
• Copy Paste: [[2403.10147]] GGRt: Towards Generalizable 3D Gaussians without Pose Priors in Real-Time(https://arxiv.org/abs/2403.10147)
• Keywords: robust
• Abstract: This paper presents GGRt, a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses, complexity in processing high-resolution images, and lengthy optimization processes, thus facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in real-world scenarios. Specifically, we design a novel joint learning framework that consists of an Iterative Pose Optimization Network (IPO-Net) and a Generalizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism, the proposed framework can inherently estimate robust relative pose information from the image observations and thus primarily alleviate the requirement of real camera poses. Moreover, we implement a deferred back-propagation mechanism that enables high-resolution training and inference, overcoming the resolution constraints of previous methods. To enhance the speed and efficiency, we further introduce a progressive Gaussian cache module that dynamically adjusts during training and inference. As the first pose-free generalizable 3D-GS framework, GGRt achieves inference at $\ge$ 5 FPS and real-time rendering at $\ge$ 100 FPS. Through extensive experimentation, we demonstrate that our method outperforms existing NeRF-based pose-free techniques in terms of inference speed and effectiveness. It can also approach the real pose-based 3D-GS methods. Our contributions provide a significant leap forward for the integration of computer vision and computer graphics into practical applications, offering state-of-the-art results on LLFF, KITTI, and Waymo Open datasets and enabling real-time rendering for immersive experiences.

Title: Improving Medical Multi-modal Contrastive Learning with Expert Annotations

• Authors: Yogesh Kumar, Pekka Marttinen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10153
• Pdf URL: https://arxiv.org/pdf/2403.10153
• Copy Paste: [[2403.10153]] Improving Medical Multi-modal Contrastive Learning with Expert Annotations(https://arxiv.org/abs/2403.10153)
• Keywords: large language model
• Abstract: We introduce eCLIP, an enhanced version of the CLIP model that integrates expert annotations in the form of radiologist eye-gaze heatmaps. It tackles key challenges in contrastive multi-modal medical imaging analysis, notably data scarcity and the "modality gap" -- a significant disparity between image and text embeddings that diminishes the quality of representations and hampers cross-modal interoperability. eCLIP integrates a heatmap processor and leverages mixup augmentation to efficiently utilize the scarce expert annotations, thus boosting the model's learning effectiveness. eCLIP is designed to be generally applicable to any variant of CLIP without requiring any modifications of the core architecture. Through detailed evaluations across several tasks, including zero-shot inference, linear probing, cross-modal retrieval, and Retrieval Augmented Generation (RAG) of radiology reports using a frozen Large Language Model, eCLIP showcases consistent improvements in embedding quality. The outcomes reveal enhanced alignment and uniformity, affirming eCLIP's capability to harness high-quality annotations for enriched multi-modal analysis in the medical imaging domain.

Title: Functional Graph Convolutional Networks: A unified multi-task and multi-modal learning framework to facilitate health and social-care insights

• Authors: Tobia Boschi, Francesca Bonin, Rodrigo Ordonez-Hurtado, Cécile Rosseau, Alessandra Pascale, John Dinsmore
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10158
• Pdf URL: https://arxiv.org/pdf/2403.10158
• Copy Paste: [[2403.10158]] Functional Graph Convolutional Networks: A unified multi-task and multi-modal learning framework to facilitate health and social-care insights(https://arxiv.org/abs/2403.10158)
• Keywords: interpretability
• Abstract: This paper introduces a novel Functional Graph Convolutional Network (funGCN) framework that combines Functional Data Analysis and Graph Convolutional Networks to address the complexities of multi-task and multi-modal learning in digital health and longitudinal studies. With the growing importance of health solutions to improve health care and social support, ensure healthy lives, and promote well-being at all ages, funGCN offers a unified approach to handle multivariate longitudinal data for multiple entities and ensures interpretability even with small sample sizes. Key innovations include task-specific embedding components that manage different data types, the ability to perform classification, regression, and forecasting, and the creation of a knowledge graph for insightful data interpretation. The efficacy of funGCN is validated through simulation experiments and a real-data application.

Title: CoReEcho: Continuous Representation Learning for 2D+time Echocardiography Analysis

• Authors: Fadillah Adamsyah Maani, Numan Saeed, Aleksandr Matsun, Mohammad Yaqub
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10164
• Pdf URL: https://arxiv.org/pdf/2403.10164
• Copy Paste: [[2403.10164]] CoReEcho: Continuous Representation Learning for 2D+time Echocardiography Analysis(https://arxiv.org/abs/2403.10164)
• Keywords: robust
• Abstract: Deep learning (DL) models have been advancing automatic medical image analysis on various modalities, including echocardiography, by offering a comprehensive end-to-end training pipeline. This approach enables DL models to regress ejection fraction (EF) directly from 2D+time echocardiograms, resulting in superior performance. However, the end-to-end training pipeline makes the learned representations less explainable. The representations may also fail to capture the continuous relation among echocardiogram clips, indicating the existence of spurious correlations, which can negatively affect the generalization. To mitigate this issue, we propose CoReEcho, a novel training framework emphasizing continuous representations tailored for direct EF regression. Our extensive experiments demonstrate that CoReEcho: 1) outperforms the current state-of-the-art (SOTA) on the largest echocardiography dataset (EchoNet-Dynamic) with MAE of 3.90 & R2 of 82.44, and 2) provides robust and generalizable features that transfer more effectively in related downstream tasks. The code is publicly available at https://github.com/fadamsyah/CoReEcho.

Title: SemanticHuman-HD: High-Resolution Semantic Disentangled 3D Human Generation

• Authors: Peng Zheng, Tao Liu, Zili Yi, Rui Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10166
• Pdf URL: https://arxiv.org/pdf/2403.10166
• Copy Paste: [[2403.10166]] SemanticHuman-HD: High-Resolution Semantic Disentangled 3D Human Generation(https://arxiv.org/abs/2403.10166)
• Keywords: generative
• Abstract: With the development of neural radiance fields and generative models, numerous methods have been proposed for learning 3D human generation from 2D images. These methods allow control over the pose of the generated 3D human and enable rendering from different viewpoints. However, none of these methods explore semantic disentanglement in human image synthesis, i.e., they can not disentangle the generation of different semantic parts, such as the body, tops, and bottoms. Furthermore, existing methods are limited to synthesize images at $512^2$ resolution due to the high computational cost of neural radiance fields. To address these limitations, we introduce SemanticHuman-HD, the first method to achieve semantic disentangled human image synthesis. Notably, SemanticHuman-HD is also the first method to achieve 3D-aware image synthesis at $1024^2$ resolution, benefiting from our proposed 3D-aware super-resolution module. By leveraging the depth maps and semantic masks as guidance for the 3D-aware super-resolution, we significantly reduce the number of sampling points during volume rendering, thereby reducing the computational cost. Our comparative experiments demonstrate the superiority of our method. The effectiveness of each proposed component is also verified through ablation studies. Moreover, our method opens up exciting possibilities for various applications, including 3D garment generation, semantic-aware image synthesis, controllable image synthesis, and out-of-domain image synthesis.

Title: Explainability through uncertainty: Trustworthy decision-making with neural networks

• Authors: Arthur Thuy, Dries F. Benoit
• Subjects: cs.LG, cs.CY, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.10168
• Pdf URL: https://arxiv.org/pdf/2403.10168
• Copy Paste: [[2403.10168]] Explainability through uncertainty: Trustworthy decision-making with neural networks(https://arxiv.org/abs/2403.10168)
• Keywords: robust, explainability
• Abstract: Uncertainty is a key feature of any machine learning model and is particularly important in neural networks, which tend to be overconfident. This overconfidence is worrying under distribution shifts, where the model performance silently degrades as the data distribution diverges from the training data distribution. Uncertainty estimation offers a solution to overconfident models, communicating when the output should (not) be trusted. Although methods for uncertainty estimation have been developed, they have not been explicitly linked to the field of explainable artificial intelligence (XAI). Furthermore, literature in operations research ignores the actionability component of uncertainty estimation and does not consider distribution shifts. This work proposes a general uncertainty framework, with contributions being threefold: (i) uncertainty estimation in ML models is positioned as an XAI technique, giving local and model-specific explanations; (ii) classification with rejection is used to reduce misclassifications by bringing a human expert in the loop for uncertain observations; (iii) the framework is applied to a case study on neural networks in educational data mining subject to distribution shifts. Uncertainty as XAI improves the model's trustworthiness in downstream decision-making tasks, giving rise to more actionable and robust machine learning systems in operations research.

Title: Animate Your Motion: Turning Still Images into Dynamic Videos

• Authors: Mingxiao Li, Bo Wan, Marie-Francine Moens, Tinne Tuytelaars
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10179
• Pdf URL: https://arxiv.org/pdf/2403.10179
• Copy Paste: [[2403.10179]] Animate Your Motion: Turning Still Images into Dynamic Videos(https://arxiv.org/abs/2403.10179)
• Keywords: diffusion
• Abstract: In recent years, diffusion models have made remarkable strides in text-to-video generation, sparking a quest for enhanced control over video outputs to more accurately reflect user intentions. Traditional efforts predominantly focus on employing either semantic cues, like images or depth maps, or motion-based conditions, like moving sketches or object bounding boxes. Semantic inputs offer a rich scene context but lack detailed motion specificity; conversely, motion inputs provide precise trajectory information but miss the broader semantic narrative. For the first time, we integrate both semantic and motion cues within a diffusion model for video generation, as demonstrated in Fig 1. To this end, we introduce the Scene and Motion Conditional Diffusion (SMCD), a novel methodology for managing multimodal inputs. It incorporates a recognized motion conditioning module and investigates various approaches to integrate scene conditions, promoting synergy between different modalities. For model training, we separate the conditions for the two modalities, introducing a two-stage training pipeline. Experimental results demonstrate that our design significantly enhances video quality, motion precision, and semantic coherence.

Title: Taiyi: A high-performance CKKS accelerator for Practical Fully Homomorphic Encryption

• Authors: Shengyu Fan, Xianglong Deng, Zhuoyu Tian, Zhicheng Hu, Liang Chang, Rui Hou, Dan Meng, Mingzhe Zhang
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2403.10188
• Pdf URL: https://arxiv.org/pdf/2403.10188
• Copy Paste: [[2403.10188]] Taiyi: A high-performance CKKS accelerator for Practical Fully Homomorphic Encryption(https://arxiv.org/abs/2403.10188)
• Keywords: security
• Abstract: Fully Homomorphic Encryption (FHE), a novel cryptographic theory enabling computation directly on ciphertext data, offers significant security benefits but is hampered by substantial performance overhead. In recent years, a series of accelerator designs have significantly enhanced the performance of FHE applications, bringing them closer to real-world applicability. However, these accelerators face challenges related to large on-chip memory and area. Additionally, FHE algorithms undergo rapid development, rendering the previous accelerator designs less perfectly adapted to the evolving landscape of optimized FHE applications. In this paper, we conducted a detailed analysis of existing applications with the new FHE method, making two key observations: 1) the bottleneck of FHE applications shifts from NTT to the inner-product operation, and 2) the optimal {\alpha} of KeySwitch changes with the decrease in multiplicative level. Based on these observations, we designed an accelerator named Taiyi, which includes specific hardware for the inner-product operation and optimizes the NTT and BConv operations through algorithmic derivation. A comparative evaluation of Taiyi against previous state-of-the-art designs reveals an average performance improvement of 1.5x and reduces the area overhead by 15.7%.

Title: Generative Region-Language Pretraining for Open-Ended Object Detection

• Authors: Chuang Lin, Yi Jiang, Lizhen Qu, Zehuan Yuan, Jianfei Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10191
• Pdf URL: https://arxiv.org/pdf/2403.10191
• Copy Paste: [[2403.10191]] Generative Region-Language Pretraining for Open-Ended Object Detection(https://arxiv.org/abs/2403.10191)
• Keywords: generative
• Abstract: In recent research, significant attention has been devoted to the open-vocabulary object detection task, aiming to generalize beyond the limited number of classes labeled during training and detect objects described by arbitrary category names at inference. Compared with conventional object detection, open vocabulary object detection largely extends the object detection categories. However, it relies on calculating the similarity between image regions and a set of arbitrary category names with a pretrained vision-and-language model. This implies that, despite its open-set nature, the task still needs the predefined object categories during the inference stage. This raises the question: What if we do not have exact knowledge of object categories during inference? In this paper, we call such a new setting as generative open-ended object detection, which is a more general and practical problem. To address it, we formulate object detection as a generative problem and propose a simple framework named GenerateU, which can detect dense objects and generate their names in a free-form way. Particularly, we employ Deformable DETR as a region proposal generator with a language model translating visual regions to object names. To assess the free-form object detection task, we introduce an evaluation method designed to quantitatively measure the performance of generative outcomes. Extensive experiments demonstrate strong zero-shot detection performance of our GenerateU. For example, on the LVIS dataset, our GenerateU achieves comparable results to the open-vocabulary object detection method GLIP, even though the category names are not seen by GenerateU during inference. Code is available at: https:// github.com/FoundationVision/GenerateU .

Title: Read between the lines -- Functionality Extraction From READMEs

• Authors: Prince Kumar, Srikanth Tamilselvam, Dinesh Garg
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10205
• Pdf URL: https://arxiv.org/pdf/2403.10205
• Copy Paste: [[2403.10205]] Read between the lines -- Functionality Extraction From READMEs(https://arxiv.org/abs/2403.10205)
• Keywords: extraction, large language model
• Abstract: While text summarization is a well-known NLP task, in this paper, we introduce a novel and useful variant of it called functionality extraction from Git README files. Though this task is a text2text generation at an abstract level, it involves its own peculiarities and challenges making existing text2text generation systems not very useful. The motivation behind this task stems from a recent surge in research and development activities around the use of large language models for code-related tasks, such as code refactoring, code summarization, etc. We also release a human-annotated dataset called FuncRead, and develop a battery of models for the task. Our exhaustive experimentation shows that small size fine-tuned models beat any baseline models that can be designed using popular black-box or white-box large language models (LLMs) such as ChatGPT and Bard. Our best fine-tuned 7 Billion CodeLlama model exhibit 70% and 20% gain on the F1 score against ChatGPT and Bard respectively.

Title: BlindDiff: Empowering Degradation Modelling in Diffusion Models for Blind Image Super-Resolution

• Authors: Feng Li, Yixuan Wu, Zichao Liang, Runmin Cong, Huihui Bai, Yao Zhao, Meng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10211
• Pdf URL: https://arxiv.org/pdf/2403.10211
• Copy Paste: [[2403.10211]] BlindDiff: Empowering Degradation Modelling in Diffusion Models for Blind Image Super-Resolution(https://arxiv.org/abs/2403.10211)
• Keywords: diffusion, transformer
• Abstract: Diffusion models (DM) have achieved remarkable promise in image super-resolution (SR). However, most of them are tailored to solving non-blind inverse problems with fixed known degradation settings, limiting their adaptability to real-world applications that involve complex unknown degradations. In this work, we propose BlindDiff, a DM-based blind SR method to tackle the blind degradation settings in SISR. BlindDiff seamlessly integrates the MAP-based optimization into DMs, which constructs a joint distribution of the low-resolution (LR) observation, high-resolution (HR) data, and degradation kernels for the data and kernel priors, and solves the blind SR problem by unfolding MAP approach along with the reverse process. Unlike most DMs, BlindDiff firstly presents a modulated conditional transformer (MCFormer) that is pre-trained with noise and kernel constraints, further serving as a posterior sampler to provide both priors simultaneously. Then, we plug a simple yet effective kernel-aware gradient term between adjacent sampling iterations that guides the diffusion model to learn degradation consistency knowledge. This also enables to joint refine the degradation model as well as HR images by observing the previous denoised sample. With the MAP-based reverse diffusion process, we show that BlindDiff advocates alternate optimization for blur kernel estimation and HR image restoration in a mutual reinforcing manner. Experiments on both synthetic and real-world datasets show that BlindDiff achieves the state-of-the-art performance with significant model complexity reduction compared to recent DM-based methods. Code will be available at \url{https://github.com/lifengcs/BlindDiff}

Title: Exploring Optical Flow Inclusion into nnU-Net Framework for Surgical Instrument Segmentation

• Authors: Marcos Fernández-Rodríguez, Bruno Silva, Sandro Queirós, Helena R. Torres, Bruno Oliveira, Pedro Morais, Lukas R. Buschle, Jorge Correia-Pinto, Estevão Lima, João L. Vilaça
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10216
• Pdf URL: https://arxiv.org/pdf/2403.10216
• Copy Paste: [[2403.10216]] Exploring Optical Flow Inclusion into nnU-Net Framework for Surgical Instrument Segmentation(https://arxiv.org/abs/2403.10216)
• Keywords: segmentation
• Abstract: Surgical instrument segmentation in laparoscopy is essential for computer-assisted surgical systems. Despite the Deep Learning progress in recent years, the dynamic setting of laparoscopic surgery still presents challenges for precise segmentation. The nnU-Net framework excelled in semantic segmentation analyzing single frames without temporal information. The framework's ease of use, including its ability to be automatically configured, and its low expertise requirements, have made it a popular base framework for comparisons. Optical flow (OF) is a tool commonly used in video tasks to estimate motion and represent it in a single frame, containing temporal information. This work seeks to employ OF maps as an additional input to the nnU-Net architecture to improve its performance in the surgical instrument segmentation task, taking advantage of the fact that instruments are the main moving objects in the surgical field. With this new input, the temporal component would be indirectly added without modifying the architecture. Using CholecSeg8k dataset, three different representations of movement were estimated and used as new inputs, comparing them with a baseline model. Results showed that the use of OF maps improves the detection of classes with high movement, even when these are scarce in the dataset. To further improve performance, future work may focus on implementing other OF-preserving augmentations.

Title: From Chaos to Clarity: Time Series Anomaly Detection in Astronomical Observations

• Authors: Xinli Hao, Yile Chen, Chen Yang, Zhihui Du, Chaohong Ma, Chao Wu, Xiaofeng Meng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10220
• Pdf URL: https://arxiv.org/pdf/2403.10220
• Copy Paste: [[2403.10220]] From Chaos to Clarity: Time Series Anomaly Detection in Astronomical Observations(https://arxiv.org/abs/2403.10220)
• Keywords: transformer
• Abstract: With the development of astronomical facilities, large-scale time series data observed by these facilities is being collected. Analyzing anomalies in these astronomical observations is crucial for uncovering potential celestial events and physical phenomena, thus advancing the scientific research process. However, existing time series anomaly detection methods fall short in tackling the unique characteristics of astronomical observations where each star is inherently independent but interfered by random concurrent noise, resulting in a high rate of false alarms. To overcome the challenges, we propose AERO, a novel two-stage framework tailored for unsupervised anomaly detection in astronomical observations. In the first stage, we employ a Transformer-based encoder-decoder architecture to learn the normal temporal patterns on each variate (i.e., star) in alignment with the characteristic of variate independence. In the second stage, we enhance the graph neural network with a window-wise graph structure learning to tackle the occurrence of concurrent noise characterized by spatial and temporal randomness. In this way, AERO is not only capable of distinguishing normal temporal patterns from potential anomalies but also effectively differentiating concurrent noise, thus decreasing the number of false alarms. We conducted extensive experiments on three synthetic datasets and three real-world datasets. The results demonstrate that AERO outperforms the compared baselines. Notably, compared to the state-of-the-art model, AERO improves the F1-score by up to 8.76% and 2.63% on synthetic and real-world datasets respectively.

Title: HawkEye: Training Video-Text LLMs for Grounding Text in Videos

• Authors: Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, Dongyan Zhao
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10228
• Pdf URL: https://arxiv.org/pdf/2403.10228
• Copy Paste: [[2403.10228]] HawkEye: Training Video-Text LLMs for Grounding Text in Videos(https://arxiv.org/abs/2403.10228)
• Keywords: robust, large language model
• Abstract: Video-text Large Language Models (video-text LLMs) have shown remarkable performance in answering questions and holding conversations on simple videos. However, they perform almost the same as random on grounding text queries in long and complicated videos, having little ability to understand and reason about temporal information, which is the most fundamental difference between videos and images. In this paper, we propose HawkEye, one of the first video-text LLMs that can perform temporal video grounding in a fully text-to-text manner. To collect training data that is applicable for temporal video grounding, we construct InternVid-G, a large-scale video-text corpus with segment-level captions and negative spans, with which we introduce two new time-aware training objectives to video-text LLMs. We also propose a coarse-grained method of representing segments in videos, which is more robust and easier for LLMs to learn and follow than other alternatives. Extensive experiments show that HawkEye is better at temporal video grounding and comparable on other video-text tasks with existing video-text LLMs, which verifies its superior video-text multi-modal understanding abilities.

Title: FDGaussian: Fast Gaussian Splatting from Single Image via Geometric-aware Diffusion Model

• Authors: Qijun Feng, Zhen Xing, Zuxuan Wu, Yu-Gang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10242
• Pdf URL: https://arxiv.org/pdf/2403.10242
• Copy Paste: [[2403.10242]] FDGaussian: Fast Gaussian Splatting from Single Image via Geometric-aware Diffusion Model(https://arxiv.org/abs/2403.10242)
• Keywords: diffusion
• Abstract: Reconstructing detailed 3D objects from single-view images remains a challenging task due to the limited information available. In this paper, we introduce FDGaussian, a novel two-stage framework for single-image 3D reconstruction. Recent methods typically utilize pre-trained 2D diffusion models to generate plausible novel views from the input image, yet they encounter issues with either multi-view inconsistency or lack of geometric fidelity. To overcome these challenges, we propose an orthogonal plane decomposition mechanism to extract 3D geometric features from the 2D input, enabling the generation of consistent multi-view images. Moreover, we further accelerate the state-of-the-art Gaussian Splatting incorporating epipolar attention to fuse images from different viewpoints. We demonstrate that FDGaussian generates images with high consistency across different views and reconstructs high-quality 3D objects, both qualitatively and quantitatively. More examples can be found at our website https://qjfeng.net/FDGaussian/.

Title: Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning

• Authors: Meixuan Li, Tianyu Li, Guoqing Wang, Peng Wang, Yang Yang, Heng Tao Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10252
• Pdf URL: https://arxiv.org/pdf/2403.10252
• Copy Paste: [[2403.10252]] Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning(https://arxiv.org/abs/2403.10252)
• Keywords: segmentation
• Abstract: In this study, we address the intricate challenge of multi-task dense prediction, encompassing tasks such as semantic segmentation, depth estimation, and surface normal estimation, particularly when dealing with partially annotated data (MTPSL). The complexity arises from the absence of complete task labels for each training image. Given the inter-related nature of these pixel-wise dense tasks, our focus is on mining and capturing cross-task relationships. Existing solutions typically rely on learning global image representations for global cross-task image matching, imposing constraints that, unfortunately, sacrifice the finer structures within the images. Attempting local matching as a remedy faces hurdles due to the lack of precise region supervision, making local alignment a challenging endeavor. The introduction of Segment Anything Model (SAM) sheds light on addressing local alignment challenges by providing free and high-quality solutions for region detection. Leveraging SAM-detected regions, the subsequent challenge lies in aligning the representations within these regions. Diverging from conventional methods that directly learn a monolithic image representation, our proposal involves modeling region-wise representations using Gaussian Distributions. Aligning these distributions between corresponding regions from different tasks imparts higher flexibility and capacity to capture intra-region structures, accommodating a broader range of tasks. This innovative approach significantly enhances our ability to effectively capture cross-task relationships, resulting in improved overall performance in partially supervised multi-task dense prediction scenarios. Extensive experiments conducted on two widely used benchmarks underscore the superior effectiveness of our proposed method, showcasing state-of-the-art performance even when compared to fully supervised methods.

Title: Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification

• Authors: Pingping Zhang, Yuhao Wang, Yang Liu, Zhengzheng Tu, Huchuan Lu
• Subjects: cs.CV, cs.IR, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.10254
• Pdf URL: https://arxiv.org/pdf/2403.10254
• Copy Paste: [[2403.10254]] Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification(https://arxiv.org/abs/2403.10254)
• Keywords: robust, transformer
• Abstract: Single-modal object re-identification (ReID) faces great challenges in maintaining robustness within complex visual scenarios. In contrast, multi-modal object ReID utilizes complementary information from diverse modalities, showing great potentials for practical applications. However, previous methods may be easily affected by irrelevant backgrounds and usually ignore the modality gaps. To address above issues, we propose a novel learning framework named \textbf{EDITOR} to select diverse tokens from vision Transformers for multi-modal object ReID. We begin with a shared vision Transformer to extract tokenized features from different input modalities. Then, we introduce a Spatial-Frequency Token Selection (SFTS) module to adaptively select object-centric tokens with both spatial and frequency information. Afterwards, we employ a Hierarchical Masked Aggregation (HMA) module to facilitate feature interactions within and across modalities. Finally, to further reduce the effect of backgrounds, we propose a Background Consistency Constraint (BCC) and an Object-Centric Feature Refinement (OCFR). They are formulated as two new loss functions, which improve the feature discrimination with background suppression. As a result, our framework can generate more discriminative features for multi-modal object ReID. Extensive experiments on three multi-modal ReID benchmarks verify the effectiveness of our methods. The code is available at https://github.com/924973292/EDITOR.

Title: Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder

• Authors: Jinseok Kim, Tae-Kyun Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10255
• Pdf URL: https://arxiv.org/pdf/2403.10255
• Copy Paste: [[2403.10255]] Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder(https://arxiv.org/abs/2403.10255)
• Keywords: diffusion
• Abstract: Super-resolution (SR) and image generation are important tasks in computer vision and are widely adopted in real-world applications. Most existing methods, however, generate images only at fixed-scale magnification and suffer from over-smoothing and artifacts. Additionally, they do not offer enough diversity of output images nor image consistency at different scales. Most relevant work applied Implicit Neural Representation (INR) to the denoising diffusion model to obtain continuous-resolution yet diverse and high-quality SR results. Since this model operates in the image space, the larger the resolution of image is produced, the more memory and inference time is required, and it also does not maintain scale-specific consistency. We propose a novel pipeline that can super-resolve an input image or generate from a random noise a novel image at arbitrary scales. The method consists of a pretrained auto-encoder, a latent diffusion model, and an implicit neural decoder, and their learning strategies. The proposed method adopts diffusion processes in a latent space, thus efficient, yet aligned with output image space decoded by MLPs at arbitrary scales. More specifically, our arbitrary-scale decoder is designed by the symmetric decoder w/o up-scaling from the pretrained auto-encoder, and Local Implicit Image Function (LIIF) in series. The latent diffusion process is learnt by the denoising and the alignment losses jointly. Errors in output images are backpropagated via the fixed decoder, improving the quality of output images. In the extensive experiments using multiple public benchmarks on the two tasks i.e. image super-resolution and novel image generation at arbitrary scales, the proposed method outperforms relevant methods in metrics of image quality, diversity and scale consistency. It is significantly better than the relevant prior-art in the inference speed and memory usage.

Title: Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models

• Authors: Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, Lidong Bing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10258
• Pdf URL: https://arxiv.org/pdf/2403.10258
• Copy Paste: [[2403.10258]] Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models(https://arxiv.org/abs/2403.10258)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated strong multilingual capabilities; yet, they are mostly English-centric due to the imbalanced training corpora. Existing works leverage this phenomenon to improve their multilingual performances on NLP tasks. In this work, we extend the evaluation from NLP tasks to real user queries. We find that even though translation into English can help improve the performance of multilingual NLP tasks for English-centric LLMs, it may not be optimal for all scenarios. For culture-related tasks that need deep language understanding, prompting in the native language proves to be more promising since it can capture the nuances related to culture and language. Therefore, we advocate for more efforts towards the development of strong multilingual LLMs instead of just English-centric LLMs.

Title: Towards Generalizable Deepfake Video Detection with Thumbnail Layout and Graph Reasoning

• Authors: Yuting Xu, Jian Liang, Lijun Sheng, Xiao-Yu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10261
• Pdf URL: https://arxiv.org/pdf/2403.10261
• Copy Paste: [[2403.10261]] Towards Generalizable Deepfake Video Detection with Thumbnail Layout and Graph Reasoning(https://arxiv.org/abs/2403.10261)
• Keywords: security, diffusion
• Abstract: The deepfake threats to society and cybersecurity have provoked significant public apprehension, driving intensified efforts within the realm of deepfake video detection. Current video-level methods are mostly based on {3D CNNs} resulting in high computational demands, although have achieved good performance. This paper introduces an elegantly simple yet effective strategy named Thumbnail Layout (TALL), which transforms a video clip into a pre-defined layout to realize the preservation of spatial and temporal dependencies. This transformation process involves sequentially masking frames at the same positions within each frame. These frames are then resized into sub-frames and reorganized into the predetermined layout, forming thumbnails. TALL is model-agnostic and has remarkable simplicity, necessitating only minimal code modifications. Furthermore, we introduce a graph reasoning block (GRB) and semantic consistency (SC) loss to strengthen TALL, culminating in TALL++. GRB enhances interactions between different semantic regions to capture semantic-level inconsistency clues. The semantic consistency loss imposes consistency constraints on semantic features to improve model generalization ability. Extensive experiments on intra-dataset, cross-dataset, diffusion-generated image detection, and deepfake generation method recognition show that TALL++ achieves results surpassing or comparable to the state-of-the-art methods, demonstrating the effectiveness of our approaches for various deepfake detection problems. The code is available at https://github.com/rainy-xu/TALL4Deepfake.

Title: A Question on the Explainability of Large Language Models and the Word-Level Univariate First-Order Plausibility Assumption

• Authors: Jeremie Bogaert, Francois-Xavier Standaert
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10275
• Pdf URL: https://arxiv.org/pdf/2403.10275
• Copy Paste: [[2403.10275]] A Question on the Explainability of Large Language Models and the Word-Level Univariate First-Order Plausibility Assumption(https://arxiv.org/abs/2403.10275)
• Keywords: explainability, transformer, large language model
• Abstract: The explanations of large language models have recently been shown to be sensitive to the randomness used for their training, creating a need to characterize this sensitivity. In this paper, we propose a characterization that questions the possibility to provide simple and informative explanations for such models. To this end, we give statistical definitions for the explanations' signal, noise and signal-to-noise ratio. We highlight that, in a typical case study where word-level univariate explanations are analyzed with first-order statistical tools, the explanations of simple feature-based models carry more signal and less noise than those of transformer ones. We then discuss the possibility to improve these results with alternative definitions of signal and noise that would capture more complex explanations and analysis methods, while also questioning the tradeoff with their plausibility for readers.

Title: Team Trifecta at Factify5WQA: Setting the Standard in Fact Verification with Fine-Tuning

• Authors: Shang-Hsuan Chiang, Ming-Chih Lo, Lin-Wei Chao, Wen-Chih Peng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10281
• Pdf URL: https://arxiv.org/pdf/2403.10281
• Copy Paste: [[2403.10281]] Team Trifecta at Factify5WQA: Setting the Standard in Fact Verification with Fine-Tuning(https://arxiv.org/abs/2403.10281)
• Keywords: secure, large language model
• Abstract: In this paper, we present Pre-CoFactv3, a comprehensive framework comprised of Question Answering and Text Classification components for fact verification. Leveraging In-Context Learning, Fine-tuned Large Language Models (LLMs), and the FakeNet model, we address the challenges of fact verification. Our experiments explore diverse approaches, comparing different Pre-trained LLMs, introducing FakeNet, and implementing various ensemble methods. Notably, our team, Trifecta, secured first place in the AAAI-24 Factify 3.0 Workshop, surpassing the baseline accuracy by 103% and maintaining a 70% lead over the second competitor. This success underscores the efficacy of our approach and its potential contributions to advancing fact verification research.

Title: Few-Shot Image Classification and Segmentation as Visual Question Answering Using Vision-Language Models

• Authors: Tian Meng, Yang Tao, Ruilin Lyu, Wuliang Yin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10287
• Pdf URL: https://arxiv.org/pdf/2403.10287
• Copy Paste: [[2403.10287]] Few-Shot Image Classification and Segmentation as Visual Question Answering Using Vision-Language Models(https://arxiv.org/abs/2403.10287)
• Keywords: segmentation
• Abstract: The task of few-shot image classification and segmentation (FS-CS) involves classifying and segmenting target objects in a query image, given only a few examples of the target classes. We introduce the Vision-Instructed Segmentation and Evaluation (VISE) method that transforms the FS-CS problem into the Visual Question Answering (VQA) problem, utilising Vision-Language Models (VLMs), and addresses it in a training-free manner. By enabling a VLM to interact with off-the-shelf vision models as tools, the proposed method is capable of classifying and segmenting target objects using only image-level labels. Specifically, chain-of-thought prompting and in-context learning guide the VLM to answer multiple-choice questions like a human; vision models such as YOLO and Segment Anything Model (SAM) assist the VLM in completing the task. The modular framework of the proposed method makes it easily extendable. Our approach achieves state-of-the-art performance on the Pascal-5i and COCO-20i datasets.

Title: Formal Security Analysis of the AMD SEV-SNP Software Interface

• Authors: Petar Paradžik, Ante Derek, Marko Horvat
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.10296
• Pdf URL: https://arxiv.org/pdf/2403.10296
• Copy Paste: [[2403.10296]] Formal Security Analysis of the AMD SEV-SNP Software Interface(https://arxiv.org/abs/2403.10296)
• Keywords: secure, security, protect
• Abstract: AMD Secure Encrypted Virtualization technologies enable confidential computing by protecting virtual machines from highly privileged software such as hypervisors. In this work, we develop the first, comprehensive symbolic model of the software interface of the latest SEV iteration called SEV Secure Nested Paging (SEV-SNP). Our model covers remote attestation, key derivation, page swap and live migration. We analyze the security of the software interface of SEV-SNP by verifying critical secrecy, authentication, attestation and freshness properties, and find that the platform-agnostic nature of messages exchanged between SNP guests and the AMD Secure Processor firmware presents a weakness of the design. We show multiple ways of exploiting this weakness, including the compromise of attestation report integrity, and suggest slight modifications to the design which let third parties detect guest migrations to vulnerable platforms

Title: Leveraging Neural Radiance Field in Descriptor Synthesis for Keypoints Scene Coordinate Regression

• Authors: Huy-Hoang Bui, Bach-Thuan Bui, Dinh-Tuan Tran, Joo-Ho Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10297
• Pdf URL: https://arxiv.org/pdf/2403.10297
• Copy Paste: [[2403.10297]] Leveraging Neural Radiance Field in Descriptor Synthesis for Keypoints Scene Coordinate Regression(https://arxiv.org/abs/2403.10297)
• Keywords: privacy
• Abstract: Classical structural-based visual localization methods offer high accuracy but face trade-offs in terms of storage, speed, and privacy. A recent innovation, keypoint scene coordinate regression (KSCR) named D2S addresses these issues by leveraging graph attention networks to enhance keypoint relationships and predict their 3D coordinates using a simple multilayer perceptron (MLP). Camera pose is then determined via PnP+RANSAC, using established 2D-3D correspondences. While KSCR achieves competitive results, rivaling state-of-the-art image-retrieval methods like HLoc across multiple benchmarks, its performance is hindered when data samples are limited due to the deep learning model's reliance on extensive data. This paper proposes a solution to this challenge by introducing a pipeline for keypoint descriptor synthesis using Neural Radiance Field (NeRF). By generating novel poses and feeding them into a trained NeRF model to create new views, our approach enhances the KSCR's generalization capabilities in data-scarce environments. The proposed system could significantly improve localization accuracy by up to 50\% and cost only a fraction of time for data synthesis. Furthermore, its modular design allows for the integration of multiple NeRFs, offering a versatile and efficient solution for visual localization. The implementation is publicly available at: https://github.com/ais-lab/DescriptorSynthesis4Feat2Map.

Title: Uni-SMART: Universal Science Multimodal Analysis and Research Transformer

• Authors: Hengxing Cai, Xiaochen Cai, Shuwen Yang, Jiankun Wang, Lin Yao, Zhifeng Gao, Junhan Chang, Sihang Li, Mingjun Xu, Changxin Wang, Hongshuai Wang, Yongge Li, Mujie Lin, Yaqi Li, Yuqi Yin, Linfeng Zhang, Guolin Ke
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10301
• Pdf URL: https://arxiv.org/pdf/2403.10301
• Copy Paste: [[2403.10301]] Uni-SMART: Universal Science Multimodal Analysis and Research Transformer(https://arxiv.org/abs/2403.10301)
• Keywords: transformer, large language model
• Abstract: In scientific research and its application, scientific literature analysis is crucial as it allows researchers to build on the work of others. However, the fast growth of scientific knowledge has led to a massive increase in scholarly articles, making in-depth literature analysis increasingly challenging and time-consuming. The emergence of Large Language Models (LLMs) has offered a new way to address this challenge. Known for their strong abilities in summarizing texts, LLMs are seen as a potential tool to improve the analysis of scientific literature. However, existing LLMs have their own limits. Scientific literature often includes a wide range of multimodal elements, such as molecular structure, tables, and charts, which are hard for text-focused LLMs to understand and analyze. This issue points to the urgent need for new solutions that can fully understand and analyze multimodal content in scientific literature. To answer this demand, we present Uni-SMART (Universal Science Multimodal Analysis and Research Transformer), an innovative model designed for in-depth understanding of multimodal scientific literature. Through rigorous quantitative evaluation across several domains, Uni-SMART demonstrates superior performance over leading text-focused LLMs. Furthermore, our exploration extends to practical applications, including patent infringement detection and nuanced analysis of charts. These applications not only highlight Uni-SMART's adaptability but also its potential to revolutionize how we interact with scientific literature.

Title: Interactive Trimming against Evasive Online Data Manipulation Attacks: A Game-Theoretic Approach

• Authors: Yue Fu, Qingqing Ye, Rong Du, Haibo Hu
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2403.10313
• Pdf URL: https://arxiv.org/pdf/2403.10313
• Copy Paste: [[2403.10313]] Interactive Trimming against Evasive Online Data Manipulation Attacks: A Game-Theoretic Approach(https://arxiv.org/abs/2403.10313)
• Keywords: privacy, defense, attack
• Abstract: With the exponential growth of data and its crucial impact on our lives and decision-making, the integrity of data has become a significant concern. Malicious data poisoning attacks, where false values are injected into the data, can disrupt machine learning processes and lead to severe consequences. To mitigate these attacks, distance-based defenses, such as trimming, have been proposed, but they can be easily evaded by white-box attackers. The evasiveness and effectiveness of poisoning attack strategies are two sides of the same coin, making game theory a promising approach. However, existing game-theoretical models often overlook the complexities of online data poisoning attacks, where strategies must adapt to the dynamic process of data collection. In this paper, we present an interactive game-theoretical model to defend online data manipulation attacks using the trimming strategy. Our model accommodates a complete strategy space, making it applicable to strong evasive and colluding adversaries. Leveraging the principle of least action and the Euler-Lagrange equation from theoretical physics, we derive an analytical model for the game-theoretic process. To demonstrate its practical usage, we present a case study in a privacy-preserving data collection system under local differential privacy where a non-deterministic utility function is adopted. Two strategies are devised from this analytical model, namely, Tit-for-tat and Elastic. We conduct extensive experiments on real-world datasets, which showcase the effectiveness and accuracy of these two strategies.

Title: Unsupervised Threat Hunting using Continuous Bag-of-Terms-and-Time (CBoTT)

• Authors: Varol Kayhan, Shivendu Shivendu, Rouzbeh Behnia, Clinton Daniel, Manish Agrawal
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10327
• Pdf URL: https://arxiv.org/pdf/2403.10327
• Copy Paste: [[2403.10327]] Unsupervised Threat Hunting using Continuous Bag-of-Terms-and-Time (CBoTT)(https://arxiv.org/abs/2403.10327)
• Keywords: security
• Abstract: Threat hunting is sifting through system logs to detect malicious activities that might have bypassed existing security measures. It can be performed in several ways, one of which is based on detecting anomalies. We propose an unsupervised framework, called continuous bag-of-terms-and-time (CBoTT), and publish its application programming interface (API) to help researchers and cybersecurity analysts perform anomaly-based threat hunting among SIEM logs geared toward process auditing on endpoint devices. Analyses show that our framework consistently outperforms benchmark approaches. When logs are sorted by likelihood of being an anomaly (from most likely to least), our approach identifies anomalies at higher percentiles (between 1.82-6.46) while benchmark approaches identify the same anomalies at lower percentiles (between 3.25-80.92). This framework can be used by other researchers to conduct benchmark analyses and cybersecurity analysts to find anomalies in SIEM logs.

Title: The cool and the cruel: separating hard parts of LWE secrets

• Authors: Niklas Nolte, Mohamed Malhou, Emily Wenger, Samuel Stevens, Cathy Li, François Charton, Kristin Lauter
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.10328
• Pdf URL: https://arxiv.org/pdf/2403.10328
• Copy Paste: [[2403.10328]] The cool and the cruel: separating hard parts of LWE secrets(https://arxiv.org/abs/2403.10328)
• Keywords: attack
• Abstract: Sparse binary LWE secrets are under consideration for standardization for Homomorphic Encryption and its applications to private computation. Known attacks on sparse binary LWE secrets include the sparse dual attack and the hybrid sparse dual-meet in the middle attack which requires significant memory. In this paper, we provide a new statistical attack with low memory requirement. The attack relies on some initial lattice reduction. The key observation is that, after lattice reduction is applied to the rows of a q-ary-like embedded random matrix $\mathbf A$, the entries with high variance are concentrated in the early columns of the extracted matrix. This allows us to separate out the "hard part" of the LWE secret. We can first solve the sub-problem of finding the "cruel" bits of the secret in the early columns, and then find the remaining "cool" bits in linear time. We use statistical techniques to distinguish distributions to identify both the cruel and the cool bits of the secret. We provide concrete attack timings for recovering secrets in dimensions $n=256$, $512$, and $768$. For the lattice reduction stage, we leverage recent improvements in lattice reduction (e.g. flatter) applied in parallel. We also apply our new attack in the RLWE setting for $2$-power cyclotomic rings, showing that these RLWE instances are much more vulnerable to this attack than LWE.

Title: Towards Non-Adversarial Algorithmic Recourse

• Authors: Tobias Leemann, Martin Pawelczyk, Bardh Prenkaj, Gjergji Kasneci
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10330
• Pdf URL: https://arxiv.org/pdf/2403.10330
• Copy Paste: [[2403.10330]] Towards Non-Adversarial Algorithmic Recourse(https://arxiv.org/abs/2403.10330)
• Keywords: attack, robust
• Abstract: The streams of research on adversarial examples and counterfactual explanations have largely been growing independently. This has led to several recent works trying to elucidate their similarities and differences. Most prominently, it has been argued that adversarial examples, as opposed to counterfactual explanations, have a unique characteristic in that they lead to a misclassification compared to the ground truth. However, the computational goals and methodologies employed in existing counterfactual explanation and adversarial example generation methods often lack alignment with this requirement. Using formal definitions of adversarial examples and counterfactual explanations, we introduce non-adversarial algorithmic recourse and outline why in high-stakes situations, it is imperative to obtain counterfactual explanations that do not exhibit adversarial characteristics. We subsequently investigate how different components in the objective functions, e.g., the machine learning model or cost function used to measure distance, determine whether the outcome can be considered an adversarial example or not. Our experiments on common datasets highlight that these design choices are often more critical in deciding whether recourse is non-adversarial than whether recourse or attack algorithms are used. Furthermore, we show that choosing a robust and accurate machine learning model results in less adversarial recourse desired in practice.

Title: How Powerful Potential of Attention on Image Restoration?

• Authors: Cong Wang, Jinshan Pan, Yeying Jin, Liyan Wang, Wei Wang, Gang Fu, Wenqi Ren, Xiaochun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10336
• Pdf URL: https://arxiv.org/pdf/2403.10336
• Copy Paste: [[2403.10336]] How Powerful Potential of Attention on Image Restoration?(https://arxiv.org/abs/2403.10336)
• Keywords: transformer
• Abstract: Transformers have demonstrated their effectiveness in image restoration tasks. Existing Transformer architectures typically comprise two essential components: multi-head self-attention and feed-forward network (FFN). The former captures long-range pixel dependencies, while the latter enables the model to learn complex patterns and relationships in the data. Previous studies have demonstrated that FFNs are key-value memories \cite{geva2020transformer}, which are vital in modern Transformer architectures. In this paper, we conduct an empirical study to explore the potential of attention mechanisms without using FFN and provide novel structures to demonstrate that removing FFN is flexible for image restoration. Specifically, we propose Continuous Scaling Attention (\textbf{CSAttn}), a method that computes attention continuously in three stages without using FFN. To achieve competitive performance, we propose a series of key components within the attention. Our designs provide a closer look at the attention mechanism and reveal that some simple operations can significantly affect the model performance. We apply our \textbf{CSAttn} to several image restoration tasks and show that our model can outperform CNN-based and Transformer-based image restoration approaches.

Title: Investigating grammatical abstraction in language models using few-shot learning of novel noun gender

• Authors: Priyanka Sukumaran, Conor Houghton, Nina Kazanina
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10338
• Pdf URL: https://arxiv.org/pdf/2403.10338
• Copy Paste: [[2403.10338]] Investigating grammatical abstraction in language models using few-shot learning of novel noun gender(https://arxiv.org/abs/2403.10338)
• Keywords: transformer
• Abstract: Humans can learn a new word and infer its grammatical properties from very few examples. They have an abstract notion of linguistic properties like grammatical gender and agreement rules that can be applied to novel syntactic contexts and words. Drawing inspiration from psycholinguistics, we conduct a noun learning experiment to assess whether an LSTM and a decoder-only transformer can achieve human-like abstraction of grammatical gender in French. Language models were tasked with learning the gender of a novel noun embedding from a few examples in one grammatical agreement context and predicting agreement in another, unseen context. We find that both language models effectively generalise novel noun gender from one to two learning examples and apply the learnt gender across agreement contexts, albeit with a bias for the masculine gender category. Importantly, the few-shot updates were only applied to the embedding layers, demonstrating that models encode sufficient gender information within the word embedding space. While the generalisation behaviour of models suggests that they represent grammatical gender as an abstract category, like humans, further work is needed to explore the details of how exactly this is implemented. For a comparative perspective with human behaviour, we conducted an analogous one-shot novel noun gender learning experiment, which revealed that native French speakers, like language models, also exhibited a masculine gender bias and are not excellent one-shot learners either.

Title: Generation is better than Modification: Combating High Class Homophily Variance in Graph Anomaly Detection

• Authors: Rui Zhang, Dawei Cheng, Xin Liu, Jie Yang, Yi Ouyang, Xian Wu, Yefeng Zheng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10339
• Pdf URL: https://arxiv.org/pdf/2403.10339
• Copy Paste: [[2403.10339]] Generation is better than Modification: Combating High Class Homophily Variance in Graph Anomaly Detection(https://arxiv.org/abs/2403.10339)
• Keywords: attack, robust
• Abstract: Graph-based anomaly detection is currently an important research topic in the field of graph neural networks (GNNs). We find that in graph anomaly detection, the homophily distribution differences between different classes are significantly greater than those in homophilic and heterophilic graphs. For the first time, we introduce a new metric called Class Homophily Variance, which quantitatively describes this phenomenon. To mitigate its impact, we propose a novel GNN model named Homophily Edge Generation Graph Neural Network (HedGe). Previous works typically focused on pruning, selecting or connecting on original relationships, and we refer to these methods as modifications. Different from these works, our method emphasizes generating new relationships with low class homophily variance, using the original relationships as an auxiliary. HedGe samples homophily adjacency matrices from scratch using a self-attention mechanism, and leverages nodes that are relevant in the feature space but not directly connected in the original graph. Additionally, we modify the loss function to punish the generation of unnecessary heterophilic edges by the model. Extensive comparison experiments demonstrate that HedGe achieved the best performance across multiple benchmark datasets, including anomaly detection and edgeless node classification. The proposed model also improves the robustness under the novel Heterophily Attack with increased class homophily variance on other graph classification tasks.

Title: Thermal-NeRF: Neural Radiance Fields from an Infrared Camera

• Authors: Tianxiang Ye, Qi Wu, Junyuan Deng, Guoqing Liu, Liu Liu, Songpengcheng Xia, Liang Pang, Wenxian Yu, Ling Pei
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.10340
• Pdf URL: https://arxiv.org/pdf/2403.10340
• Copy Paste: [[2403.10340]] Thermal-NeRF: Neural Radiance Fields from an Infrared Camera(https://arxiv.org/abs/2403.10340)
• Keywords: robust
• Abstract: In recent years, Neural Radiance Fields (NeRFs) have demonstrated significant potential in encoding highly-detailed 3D geometry and environmental appearance, positioning themselves as a promising alternative to traditional explicit representation for 3D scene reconstruction. However, the predominant reliance on RGB imaging presupposes ideal lighting conditions: a premise frequently unmet in robotic applications plagued by poor lighting or visual obstructions. This limitation overlooks the capabilities of infrared (IR) cameras, which excel in low-light detection and present a robust alternative under such adverse scenarios. To tackle these issues, we introduce Thermal-NeRF, the first method that estimates a volumetric scene representation in the form of a NeRF solely from IR imaging. By leveraging a thermal mapping and structural thermal constraint derived from the thermal characteristics of IR imaging, our method showcasing unparalleled proficiency in recovering NeRFs in visually degraded scenes where RGB-based methods fall short. We conduct extensive experiments to demonstrate that Thermal-NeRF can achieve superior quality compared to existing methods. Furthermore, we contribute a dataset for IR-based NeRF applications, paving the way for future research in IR NeRF reconstruction.

Title: Denoising Task Difficulty-based Curriculum for Training Diffusion Models

• Authors: Jin-Young Kim, Hyojun Go, Soonwoo Kwon, Hyun-Gyoon Kim
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10348
• Pdf URL: https://arxiv.org/pdf/2403.10348
• Copy Paste: [[2403.10348]] Denoising Task Difficulty-based Curriculum for Training Diffusion Models(https://arxiv.org/abs/2403.10348)
• Keywords: diffusion, generative
• Abstract: Diffusion-based generative models have emerged as powerful tools in the realm of generative modeling. Despite extensive research on denoising across various timesteps and noise levels, a conflict persists regarding the relative difficulties of the denoising tasks. While various studies argue that lower timesteps present more challenging tasks, others contend that higher timesteps are more difficult. To address this conflict, our study undertakes a comprehensive examination of task difficulties, focusing on convergence behavior and changes in relative entropy between consecutive probability distributions across timesteps. Our observational study reveals that denoising at earlier timesteps poses challenges characterized by slower convergence and higher relative entropy, indicating increased task difficulty at these lower timesteps. Building on these observations, we introduce an easy-to-hard learning scheme, drawing from curriculum learning, to enhance the training process of diffusion models. By organizing timesteps or noise levels into clusters and training models with descending orders of difficulty, we facilitate an order-aware training regime, progressing from easier to harder denoising tasks, thereby deviating from the conventional approach of training diffusion models simultaneously across all timesteps. Our approach leads to improved performance and faster convergence by leveraging the benefits of curriculum learning, while maintaining orthogonality with existing improvements in diffusion training techniques. We validate these advantages through comprehensive experiments in image generation tasks, including unconditional, class-conditional, and text-to-image generation.

Title: TriSum: Learning Summarization Ability from Large Language Models with Structured Rationale

• Authors: Pengcheng Jiang, Cao Xiao, Zifeng Wang, Parminder Bhatia, Jimeng Sun, Jiawei Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10351
• Pdf URL: https://arxiv.org/pdf/2403.10351
• Copy Paste: [[2403.10351]] TriSum: Learning Summarization Ability from Large Language Models with Structured Rationale(https://arxiv.org/abs/2403.10351)
• Keywords: privacy, interpretability, large language model
• Abstract: The advent of large language models (LLMs) has significantly advanced natural language processing tasks like text summarization. However, their large size and computational demands, coupled with privacy concerns in data transmission, limit their use in resource-constrained and privacy-centric settings. To overcome this, we introduce TriSum, a framework for distilling LLMs' text summarization abilities into a compact, local model. Initially, LLMs extract a set of aspect-triple rationales and summaries, which are refined using a dual-scoring method for quality. Next, a smaller local model is trained with these tasks, employing a curriculum learning strategy that evolves from simple to complex tasks. Our method enhances local model performance on various benchmarks (CNN/DailyMail, XSum, and ClinicalTrial), outperforming baselines by 4.5%, 8.5%, and 7.4%, respectively. It also improves interpretability by providing insights into the summarization rationale.

Title: Open Stamped Parts Dataset

• Authors: Sara Antiles, Sachin S. Talathi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10369
• Pdf URL: https://arxiv.org/pdf/2403.10369
• Copy Paste: [[2403.10369]] Open Stamped Parts Dataset(https://arxiv.org/abs/2403.10369)
• Keywords: segmentation
• Abstract: We present the Open Stamped Parts Dataset (OSPD), featuring synthetic and real images of stamped metal sheets for auto manufacturing. The real part images, captured from 7 cameras, consist of 7,980 unlabeled images and 1,680 labeled images. In addition, we have compiled a defect dataset by overlaying synthetically generated masks on 10% of the holes. The synthetic dataset replicates the real manufacturing environment in terms of lighting and part placement relative to the cameras. The synthetic data includes 7,980 training images, 1,680 validation images and 1,680 test images, each with bounding box and segmentation mask annotations around all holes. 10% of the holes in the synthetic data mimic defects generated in the real image dataset. We trained a hole-detection model on the synthetic-OSPD, achieving a modified recall score of 67.2% and a precision of 94.4% . We anticipate researchers in the auto manufacturing and broader machine learning and computer vision communities using OSPD to advance the state of the art in defect detection of stamped holes in the metalsheet stamping process. The dataset is available for download at: https://tinyurl.com/hm6xatd7

Title: An Energy-Efficient Ensemble Approach for Mitigating Data Incompleteness in IoT Applications

• Authors: Yousef AlShehri, Lakshmish Ramaswamy
• Subjects: cs.LG, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.10371
• Pdf URL: https://arxiv.org/pdf/2403.10371
• Copy Paste: [[2403.10371]] An Energy-Efficient Ensemble Approach for Mitigating Data Incompleteness in IoT Applications(https://arxiv.org/abs/2403.10371)
• Keywords: robust
• Abstract: Machine Learning (ML) is becoming increasingly important for IoT-based applications. However, the dynamic and ad-hoc nature of many IoT ecosystems poses unique challenges to the efficacy of ML algorithms. One such challenge is data incompleteness, which is manifested as missing sensor readings. Many factors, including sensor failures and/or network disruption, can cause data incompleteness. Furthermore, most IoT systems are severely power-constrained. It is important that we build IoT-based ML systems that are robust against data incompleteness while simultaneously being energy efficient. This paper presents an empirical study of SECOE - a recent technique for alleviating data incompleteness in IoT - with respect to its energy bottlenecks. Towards addressing the energy bottlenecks of SECOE, we propose ENAMLE - a proactive, energy-aware technique for mitigating the impact of concurrent missing data. ENAMLE is unique in the sense that it builds an energy-aware ensemble of sub-models, each trained with a subset of sensors chosen carefully based on their correlations. Furthermore, at inference time, ENAMLE adaptively alters the number of the ensemble of models based on the amount of missing data rate and the energy-accuracy trade-off. ENAMLE's design includes several novel mechanisms for minimizing energy consumption while maintaining accuracy. We present extensive experimental studies on two distinct datasets that demonstrate the energy efficiency of ENAMLE and its ability to alleviate sensor failures.

Title: PASTA: Towards Flexible and Efficient HDR Imaging Via Progressively Aggregated Spatio-Temporal Aligment

• Authors: Xiaoning Liu, Ao Li, Zongwei Wu, Yapeng Du, Le Zhang, Yulun Zhang, Radu Timofte, Ce Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10376
• Pdf URL: https://arxiv.org/pdf/2403.10376
• Copy Paste: [[2403.10376]] PASTA: Towards Flexible and Efficient HDR Imaging Via Progressively Aggregated Spatio-Temporal Aligment(https://arxiv.org/abs/2403.10376)
• Keywords: transformer
• Abstract: Leveraging Transformer attention has led to great advancements in HDR deghosting. However, the intricate nature of self-attention introduces practical challenges, as existing state-of-the-art methods often demand high-end GPUs or exhibit slow inference speeds, especially for high-resolution images like 2K. Striking an optimal balance between performance and latency remains a critical concern. In response, this work presents PASTA, a novel Progressively Aggregated Spatio-Temporal Alignment framework for HDR deghosting. Our approach achieves effectiveness and efficiency by harnessing hierarchical representation during feature distanglement. Through the utilization of diverse granularities within the hierarchical structure, our method substantially boosts computational speed and optimizes the HDR imaging workflow. In addition, we explore within-scale feature modeling with local and global attention, gradually merging and refining them in a coarse-to-fine fashion. Experimental results showcase PASTA's superiority over current SOTA methods in both visual quality and performance metrics, accompanied by a substantial 3-fold (x3) increase in inference speed.

Title: Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding

• Authors: Pengkun Liu, Yikai Wang, Fuchun Sun, Jiafang Li, Hang Xiao, Hongxiang Xue, Xinzhou Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10395
• Pdf URL: https://arxiv.org/pdf/2403.10395
• Copy Paste: [[2403.10395]] Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding(https://arxiv.org/abs/2403.10395)
• Keywords: diffusion
• Abstract: Encouraged by the growing availability of pre-trained 2D diffusion models, image-to-3D generation by leveraging Score Distillation Sampling (SDS) is making remarkable progress. Most existing methods combine novel-view lifting from 2D diffusion models which usually take the reference image as a condition while applying hard L2 image supervision at the reference view. Yet heavily adhering to the image is prone to corrupting the inductive knowledge of the 2D diffusion model leading to flat or distorted 3D generation frequently. In this work, we reexamine image-to-3D in a novel perspective and present Isotropic3D, an image-to-3D generation pipeline that takes only an image CLIP embedding as input. Isotropic3D allows the optimization to be isotropic w.r.t. the azimuth angle by solely resting on the SDS loss. The core of our framework lies in a two-stage diffusion model fine-tuning. Firstly, we fine-tune a text-to-3D diffusion model by substituting its text encoder with an image encoder, by which the model preliminarily acquires image-to-image capabilities. Secondly, we perform fine-tuning using our Explicit Multi-view Attention (EMA) which combines noisy multi-view images with the noise-free reference image as an explicit condition. CLIP embedding is sent to the diffusion model throughout the whole process while reference images are discarded once after fine-tuning. As a result, with a single image CLIP embedding, Isotropic3D is capable of generating multi-view mutually consistent images and also a 3D model with more symmetrical and neat content, well-proportioned geometry, rich colored texture, and less distortion compared with existing image-to-3D methods while still preserving the similarity to the reference image to a large extent. The project page is available at https://isotropic3d.github.io/. The code and models are available at https://github.com/pkunliu/Isotropic3D.

Title: SocialGenPod: Privacy-Friendly Generative AI Social Web Applications with Decentralised Personal Data Stores

• Authors: Vidminas Vizgirda (1), Rui Zhao (2), Naman Goel (2) ((1) University of Edinburgh, (2) University of Oxford)
• Subjects: cs.CR, cs.CY, cs.IR, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.10408
• Pdf URL: https://arxiv.org/pdf/2403.10408
• Copy Paste: [[2403.10408]] SocialGenPod: Privacy-Friendly Generative AI Social Web Applications with Decentralised Personal Data Stores(https://arxiv.org/abs/2403.10408)
• Keywords: secure, privacy, generative, large language model
• Abstract: We present SocialGenPod, a decentralised and privacy-friendly way of deploying generative AI Web applications. Unlike centralised Web and data architectures that keep user data tied to application and service providers, we show how one can use Solid -- a decentralised Web specification -- to decouple user data from generative AI applications. We demonstrate SocialGenPod using a prototype that allows users to converse with different Large Language Models, optionally leveraging Retrieval Augmented Generation to generate answers grounded in private documents stored in any Solid Pod that the user is allowed to access, directly or indirectly. SocialGenPod makes use of Solid access control mechanisms to give users full control of determining who has access to data stored in their Pods. SocialGenPod keeps all user data (chat history, app configuration, personal documents, etc) securely in the user's personal Pod; separate from specific model or application providers. Besides better privacy controls, this approach also enables portability across different services and applications. Finally, we discuss challenges, posed by the large compute requirements of state-of-the-art models, that future research in this area should address. Our prototype is open-source and available at: https://github.com/Vidminas/socialgenpod/.

Title: Real-Time Image Segmentation via Hybrid Convolutional-Transformer Architecture Search

• Authors: Hongyuan Yu, Cheng Wan, Mengchen Liu, Dongdong Chen, Bin Xiao, Xiyang Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10413
• Pdf URL: https://arxiv.org/pdf/2403.10413
• Copy Paste: [[2403.10413]] Real-Time Image Segmentation via Hybrid Convolutional-Transformer Architecture Search(https://arxiv.org/abs/2403.10413)
• Keywords: transformer, segmentation
• Abstract: Image segmentation is one of the most fundamental problems in computer vision and has drawn a lot of attentions due to its vast applications in image understanding and autonomous driving. However, designing effective and efficient segmentation neural architectures is a labor-intensive process that may require lots of trials by human experts. In this paper, we address the challenge of integrating multi-head self-attention into high resolution representation CNNs efficiently, by leveraging architecture search. Manually replacing convolution layers with multi-head self-attention is non-trivial due to the costly overhead in memory to maintain high resolution. By contrast, we develop a multi-target multi-branch supernet method, which not only fully utilizes the advantages of high-resolution features, but also finds the proper location for placing multi-head self-attention module. Our search algorithm is optimized towards multiple objective s (e.g., latency and mIoU) and capable of finding architectures on Pareto frontier with arbitrary number of branches in a single search. We further present a series of model via Hybrid Convolutional-Transformer Architecture Search (HyCTAS) method that searched for the best hybrid combination of light-weight convolution layers and memory-efficient self-attention layers between branches from different resolutions and fuse to high resolution for both efficiency and effectiveness. Extensive experiments demonstrate that HyCTAS outperforms previous methods on semantic segmentation task. Code and models are available at \url{https://github.com/MarvinYu1995/HyCTAS}.

Title: Robust Sparse Estimation for Gaussians with Optimal Error under Huber Contamination

• Authors: Ilias Diakonikolas, Daniel M. Kane, Sushrut Karmalkar, Ankit Pensia, Thanasis Pittas
• Subjects: cs.LG, cs.DS, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.10416
• Pdf URL: https://arxiv.org/pdf/2403.10416
• Copy Paste: [[2403.10416]] Robust Sparse Estimation for Gaussians with Optimal Error under Huber Contamination(https://arxiv.org/abs/2403.10416)
• Keywords: robust
• Abstract: We study Gaussian sparse estimation tasks in Huber's contamination model with a focus on mean estimation, PCA, and linear regression. For each of these tasks, we give the first sample and computationally efficient robust estimators with optimal error guarantees, within constant factors. All prior efficient algorithms for these tasks incur quantitatively suboptimal error. Concretely, for Gaussian robust $k$-sparse mean estimation on $\mathbb{R}^d$ with corruption rate $\epsilon>0$, our algorithm has sample complexity $(k^2/\epsilon^2)\mathrm{polylog}(d/\epsilon)$, runs in sample polynomial time, and approximates the target mean within $\ell_2$-error $O(\epsilon)$. Previous efficient algorithms inherently incur error $\Omega(\epsilon \sqrt{\log(1/\epsilon)})$. At the technical level, we develop a novel multidimensional filtering method in the sparse regime that may find other applications.

Title: Structured Evaluation of Synthetic Tabular Data

• Authors: Scott Cheng-Hsin Yang, Baxter Eaves, Michael Schmidt, Ken Swanson, Patrick Shafto
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.10424
• Pdf URL: https://arxiv.org/pdf/2403.10424
• Copy Paste: [[2403.10424]] Structured Evaluation of Synthetic Tabular Data(https://arxiv.org/abs/2403.10424)
• Keywords: privacy
• Abstract: Tabular data is common yet typically incomplete, small in volume, and access-restricted due to privacy concerns. Synthetic data generation offers potential solutions. Many metrics exist for evaluating the quality of synthetic tabular data; however, we lack an objective, coherent interpretation of the many metrics. To address this issue, we propose an evaluation framework with a single, mathematical objective that posits that the synthetic data should be drawn from the same distribution as the observed data. Through various structural decomposition of the objective, this framework allows us to reason for the first time the completeness of any set of metrics, as well as unifies existing metrics, including those that stem from fidelity considerations, downstream application, and model-based approaches. Moreover, the framework motivates model-free baselines and a new spectrum of metrics. We evaluate structurally informed synthesizers and synthesizers powered by deep learning. Using our structured framework, we show that synthetic data generators that explicitly represent tabular structure outperform other methods, especially on smaller datasets.

Title: Using an LLM to Turn Sign Spottings into Spoken Language Sentences

• Authors: Ozge Mercanoglu Sincan, Necati Cihan Camgoz, Richard Bowden
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10434
• Pdf URL: https://arxiv.org/pdf/2403.10434
• Copy Paste: [[2403.10434]] Using an LLM to Turn Sign Spottings into Spoken Language Sentences(https://arxiv.org/abs/2403.10434)
• Keywords: large language model
• Abstract: Sign Language Translation (SLT) is a challenging task that aims to generate spoken language sentences from sign language videos. In this paper, we introduce a hybrid SLT approach, Spotter+GPT, that utilizes a sign spotter and a pretrained large language model to improve SLT performance. Our method builds upon the strengths of both components. The videos are first processed by the spotter, which is trained on a linguistic sign language dataset, to identify individual signs. These spotted signs are then passed to the powerful language model, which transforms them into coherent and contextually appropriate spoken language sentences.

Title: Optimal Block-Level Draft Verification for Accelerating Speculative Decoding

• Authors: Ziteng Sun, Jae Hun Ro, Ahmad Beirami, Ananda Theertha Suresh
• Subjects: cs.LG, cs.CL, cs.DS, cs.IT
• Abstract URL: https://arxiv.org/abs/2403.10444
• Pdf URL: https://arxiv.org/pdf/2403.10444
• Copy Paste: [[2403.10444]] Optimal Block-Level Draft Verification for Accelerating Speculative Decoding(https://arxiv.org/abs/2403.10444)
• Keywords: large language model
• Abstract: Speculative decoding has shown to be an effective method for lossless acceleration of large language models (LLMs) during inference. In each iteration, the algorithm first uses a smaller model to draft a block of tokens. The tokens are then verified by the large model in parallel and only a subset of tokens will be kept to guarantee that the final output follows the distribution of the large model. In all of the prior speculative decoding works, the draft verification is performed token-by-token independently. In this work, we propose a better draft verification algorithm that provides additional wall-clock speedup without incurring additional computation cost and draft tokens. We first formulate the draft verification step as a block-level optimal transport problem. The block-level formulation allows us to consider a wider range of draft verification algorithms and obtain a higher number of accepted tokens in expectation in one draft block. We propose a verification algorithm that achieves the optimal accepted length for the block-level transport problem. We empirically evaluate our proposed block-level verification algorithm in a wide range of tasks and datasets, and observe consistent improvements in wall-clock speedup when compared to token-level verification algorithm. To the best of our knowledge, our work is the first to establish improvement over speculative decoding through a better draft verification algorithm.

Title: Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases

• Authors: Jiarui Li, Ye Yuan, Zehua Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10446
• Pdf URL: https://arxiv.org/pdf/2403.10446
• Copy Paste: [[2403.10446]] Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases(https://arxiv.org/abs/2403.10446)
• Keywords: large language model
• Abstract: We proposed an end-to-end system design towards utilizing Retrieval Augmented Generation (RAG) to improve the factual accuracy of Large Language Models (LLMs) for domain-specific and time-sensitive queries related to private knowledge-bases. Our system integrates RAG pipeline with upstream datasets processing and downstream performance evaluation. Addressing the challenge of LLM hallucinations, we finetune models with a curated dataset which originates from CMU's extensive resources and annotated with the teacher model. Our experiments demonstrate the system's effectiveness in generating more accurate answers to domain-specific and time-sensitive inquiries. The results also revealed the limitations of fine-tuning LLMs with small-scale and skewed datasets. This research highlights the potential of RAG systems in augmenting LLMs with external datasets for improved performance in knowledge-intensive tasks. Our code and models are available on Github.

Title: Robust Shape Fitting for 3D Scene Abstraction

• Authors: Florian Kluger, Eric Brachmann, Michael Ying Yang, Bodo Rosenhahn
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10452
• Pdf URL: https://arxiv.org/pdf/2403.10452
• Copy Paste: [[2403.10452]] Robust Shape Fitting for 3D Scene Abstraction(https://arxiv.org/abs/2403.10452)
• Keywords: robust
• Abstract: Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.

Title: Introducing Adaptive Continuous Adversarial Training (ACAT) to Enhance ML Robustness

• Authors: Mohamed elShehaby, Aditya Kotha, Ashraf Matrawy
• Subjects: cs.LG, cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.10461
• Pdf URL: https://arxiv.org/pdf/2403.10461
• Copy Paste: [[2403.10461]] Introducing Adaptive Continuous Adversarial Training (ACAT) to Enhance ML Robustness(https://arxiv.org/abs/2403.10461)
• Keywords: security, defense, attack, robust
• Abstract: Machine Learning (ML) is susceptible to adversarial attacks that aim to trick ML models, making them produce faulty predictions. Adversarial training was found to increase the robustness of ML models against these attacks. However, in network and cybersecurity, obtaining labeled training and adversarial training data is challenging and costly. Furthermore, concept drift deepens the challenge, particularly in dynamic domains like network and cybersecurity, and requires various models to conduct periodic retraining. This letter introduces Adaptive Continuous Adversarial Training (ACAT) to continuously integrate adversarial training samples into the model during ongoing learning sessions, using real-world detected adversarial data, to enhance model resilience against evolving adversarial threats. ACAT is an adaptive defense mechanism that utilizes periodic retraining to effectively counter adversarial attacks while mitigating catastrophic forgetting. Our approach also reduces the total time required for adversarial sample detection, especially in environments such as network security where the rate of attacks could be very high. Traditional detection processes that involve two stages may result in lengthy procedures. Experimental results using a SPAM detection dataset demonstrate that with ACAT, the accuracy of the SPAM filter increased from 69% to over 88% after just three retraining sessions. Furthermore, ACAT outperforms conventional adversarial sample detectors, providing faster decision times, up to four times faster in some cases.

Title: Approximate Nullspace Augmented Finetuning for Robust Vision Transformers

• Authors: Haoyang Liu, Aditya Singh, Yijiang Li, Haohan Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10476
• Pdf URL: https://arxiv.org/pdf/2403.10476
• Copy Paste: [[2403.10476]] Approximate Nullspace Augmented Finetuning for Robust Vision Transformers(https://arxiv.org/abs/2403.10476)
• Keywords: robust, transformer
• Abstract: Enhancing the robustness of deep learning models, particularly in the realm of vision transformers (ViTs), is crucial for their real-world deployment. In this work, we provide a finetuning approach to enhance the robustness of vision transformers inspired by the concept of nullspace from linear algebra. Our investigation centers on whether a vision transformer can exhibit resilience to input variations akin to the nullspace property in linear mappings, implying that perturbations sampled from this nullspace do not influence the model's output when added to the input. Firstly, we show that for many pretrained ViTs, a non-trivial nullspace exists due to the presence of the patch embedding layer. Secondly, as nullspace is a concept associated with linear algebra, we demonstrate that it is possible to synthesize approximate nullspace elements for the non-linear blocks of ViTs employing an optimisation strategy. Finally, we propose a fine-tuning strategy for ViTs wherein we augment the training data with synthesized approximate nullspace noise. After finetuning, we find that the model demonstrates robustness to adversarial and natural image perbutations alike.

Title: Joint Multimodal Transformer for Dimensional Emotional Recognition in the Wild

• Authors: Paul Waligora, Osama Zeeshan, Haseeb Aslam, Soufiane Belharbi, Alessandro Lameiras Koerich, Marco Pedersoli, Simon Bacon, Eric Granger
• Subjects: cs.CV, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.10488
• Pdf URL: https://arxiv.org/pdf/2403.10488
• Copy Paste: [[2403.10488]] Joint Multimodal Transformer for Dimensional Emotional Recognition in the Wild(https://arxiv.org/abs/2403.10488)
• Keywords: transformer
• Abstract: Audiovisual emotion recognition (ER) in videos has immense potential over unimodal performance. It effectively leverages the inter- and intra-modal dependencies between visual and auditory modalities. This work proposes a novel audio-visual emotion recognition system utilizing a joint multimodal transformer architecture with key-based cross-attention. This framework aims to exploit the complementary nature of audio and visual cues (facial expressions and vocal patterns) in videos, leading to superior performance compared to solely relying on a single modality. The proposed model leverages separate backbones for capturing intra-modal temporal dependencies within each modality (audio and visual). Subsequently, a joint multimodal transformer architecture integrates the individual modality embeddings, enabling the model to effectively capture inter-modal (between audio and visual) and intra-modal (within each modality) relationships. Extensive evaluations on the challenging Affwild2 dataset demonstrate that the proposed model significantly outperforms baseline and state-of-the-art methods in ER tasks.

Title: Mitigating Dialogue Hallucination for Large Multi-modal Models via Adversarial Instruction Tuning

• Authors: Dongmin Park, Zhaofang Qian, Guangxing Han, Ser-Nam Lim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10492
• Pdf URL: https://arxiv.org/pdf/2403.10492
• Copy Paste: [[2403.10492]] Mitigating Dialogue Hallucination for Large Multi-modal Models via Adversarial Instruction Tuning(https://arxiv.org/abs/2403.10492)
• Keywords: attack, robust
• Abstract: Mitigating hallucinations of Large Multi-modal Models(LMMs) is crucial to enhance their reliability for general-purpose assistants. This paper shows that such hallucinations of LMMs can be significantly exacerbated by preceding user-system dialogues. To precisely measure this, we first present an evaluation benchmark by extending popular multi-modal benchmark datasets with prepended hallucinatory dialogues generated by our novel Adversarial Question Generator, which can automatically generate image-related yet adversarial dialogues by adopting adversarial attacks on LMMs. On our benchmark, the zero-shot performance of state-of-the-art LMMs dropped significantly for both the VQA and Captioning tasks. Next, we further reveal this hallucination is mainly due to the prediction bias toward preceding dialogues rather than visual content. To reduce this bias, we propose Adversarial Instruction Tuning that robustly fine-tunes LMMs on augmented multi-modal instruction-following datasets with hallucinatory dialogues. Extensive experiments show that our proposed approach successfully reduces dialogue hallucination while maintaining or even improving performance.

Title: Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study

• Authors: Chenguang Wang, Ruoxi Jia, Xin Liu, Dawn Song
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10499
• Pdf URL: https://arxiv.org/pdf/2403.10499
• Copy Paste: [[2403.10499]] Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study(https://arxiv.org/abs/2403.10499)
• Keywords: attack, robust
• Abstract: Pre-training image representations from the raw text about images enables zero-shot vision transfer to downstream tasks. Through pre-training on millions of samples collected from the internet, multimodal foundation models, such as CLIP, produce state-of-the-art zero-shot results that often reach competitiveness with fully supervised methods without the need for task-specific training. Besides the encouraging performance on classification accuracy, it is reported that these models close the robustness gap by matching the performance of supervised models trained on ImageNet under natural distribution shift. Because robustness is critical to real-world applications, especially safety-critical ones, in this paper, we present a comprehensive evaluation based on a large-scale robustness benchmark covering 7 natural, 3 synthetic distribution shifts, and 11 adversarial attacks. We use CLIP as a pilot study. We show that CLIP leads to a significant robustness drop compared to supervised ImageNet models on our benchmark, especially under synthetic distribution shift and adversarial attacks. Furthermore, data overlap analysis suggests that the observed robustness under natural distribution shifts could be attributed, at least in part, to data overlap. In summary, our evaluation shows a comprehensive evaluation of robustness is necessary; and there is a significant need to improve the robustness of zero-shot multimodal models.

Title: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction

• Authors: Anshul Gupta, Samy Tafasca, Arya Farkhondeh, Pierre Vuillecard, Jean-Marc Odobez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.10511
• Pdf URL: https://arxiv.org/pdf/2403.10511
• Copy Paste: [[2403.10511]] A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction(https://arxiv.org/abs/2403.10511)
• Keywords: transformer
• Abstract: Gaze following and social gaze prediction are fundamental tasks providing insights into human communication behaviors, intent, and social interactions. Most previous approaches addressed these tasks separately, either by designing highly specialized social gaze models that do not generalize to other social gaze tasks or by considering social gaze inference as an ad-hoc post-processing of the gaze following task. Furthermore, the vast majority of gaze following approaches have proposed static models that can handle only one person at a time, therefore failing to take advantage of social interactions and temporal dynamics. In this paper, we address these limitations and introduce a novel framework to jointly predict the gaze target and social gaze label for all people in the scene. The framework comprises of: (i) a temporal, transformer-based architecture that, in addition to image tokens, handles person-specific tokens capturing the gaze information related to each individual; (ii) a new dataset, VSGaze, that unifies annotation types across multiple gaze following and social gaze datasets. We show that our model trained on VSGaze can address all tasks jointly, and achieves state-of-the-art results for multi-person gaze following and social gaze prediction.

Title: FeatUp: A Model-Agnostic Framework for Features at Any Resolution

• Authors: Stephanie Fu, Mark Hamilton, Laura Brandt, Axel Feldman, Zhoutong Zhang, William T. Freeman
• Subjects: cs.CV, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10516
• Pdf URL: https://arxiv.org/pdf/2403.10516
• Copy Paste: [[2403.10516]] FeatUp: A Model-Agnostic Framework for Features at Any Resolution(https://arxiv.org/abs/2403.10516)
• Keywords: segmentation
• Abstract: Deep features are a cornerstone of computer vision research, capturing image semantics and enabling the community to solve downstream tasks even in the zero- or few-shot regime. However, these features often lack the spatial resolution to directly perform dense prediction tasks like segmentation and depth prediction because models aggressively pool information over large areas. In this work, we introduce FeatUp, a task- and model-agnostic framework to restore lost spatial information in deep features. We introduce two variants of FeatUp: one that guides features with high-resolution signal in a single forward pass, and one that fits an implicit model to a single image to reconstruct features at any resolution. Both approaches use a multi-view consistency loss with deep analogies to NeRFs. Our features retain their original semantics and can be swapped into existing applications to yield resolution and performance gains even without re-training. We show that FeatUp significantly outperforms other feature upsampling and image super-resolution approaches in class activation map generation, transfer learning for segmentation and depth prediction, and end-to-end training for semantic segmentation.

Title: VideoAgent: Long-form Video Understanding with Large Language Model as Agent

• Authors: Xiaohan Wang, Yuhui Zhang, Orr Zohar, Serena Yeung-Levy
• Subjects: cs.CV, cs.AI, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.10517
• Pdf URL: https://arxiv.org/pdf/2403.10517
• Copy Paste: [[2403.10517]] VideoAgent: Long-form Video Understanding with Large Language Model as Agent(https://arxiv.org/abs/2403.10517)
• Keywords: large language model
• Abstract: Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation models serving as tools to translate and retrieve visual information. Evaluated on the challenging EgoSchema and NExT-QA benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only 8.4 and 8.2 frames used on average. These results demonstrate superior effectiveness and efficiency of our method over the current state-of-the-art methods, highlighting the potential of agent-based approaches in advancing long-form video understanding.

Title: Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives

• Authors: Ronghui Li, YuXiang Zhang, Yachao Zhang, Hongwen Zhang, Jie Guo, Yan Zhang, Yebin Liu, Xiu Li
• Subjects: cs.CV, cs.GR, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.10518
• Pdf URL: https://arxiv.org/pdf/2403.10518
• Copy Paste: [[2403.10518]] Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives(https://arxiv.org/abs/2403.10518)
• Keywords: diffusion
• Abstract: We propose Lodge, a network capable of generating extremely long dance sequences conditioned on given music. We design Lodge as a two-stage coarse to fine diffusion architecture, and propose the characteristic dance primitives that possess significant expressiveness as intermediate representations between two diffusion models. The first stage is global diffusion, which focuses on comprehending the coarse-level music-dance correlation and production characteristic dance primitives. In contrast, the second-stage is the local diffusion, which parallelly generates detailed motion sequences under the guidance of the dance primitives and choreographic rules. In addition, we propose a Foot Refine Block to optimize the contact between the feet and the ground, enhancing the physical realism of the motion. Our approach can parallelly generate dance sequences of extremely long length, striking a balance between global choreographic patterns and local motion quality and expressiveness. Extensive experiments validate the efficacy of our method.

Title: Strong and Controllable Blind Image Decomposition

• Authors: Zeyu Zhang, Junlin Han, Chenhui Gou, Hongdong Li, Liang Zheng
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.10520
• Pdf URL: https://arxiv.org/pdf/2403.10520
• Copy Paste: [[2403.10520]] Strong and Controllable Blind Image Decomposition(https://arxiv.org/abs/2403.10520)
• Keywords: protect, watermark
• Abstract: Blind image decomposition aims to decompose all components present in an image, typically used to restore a multi-degraded input image. While fully recovering the clean image is appealing, in some scenarios, users might want to retain certain degradations, such as watermarks, for copyright protection. To address this need, we add controllability to the blind image decomposition process, allowing users to enter which types of degradation to remove or retain. We design an architecture named controllable blind image decomposition network. Inserted in the middle of U-Net structure, our method first decomposes the input feature maps and then recombines them according to user instructions. Advantageously, this functionality is implemented at minimal computational cost: decomposition and recombination are all parameter-free. Experimentally, our system excels in blind image decomposition tasks and can outputs partially or fully restored images that well reflect user intentions. Furthermore, we evaluate and configure different options for the network structure and loss functions. This, combined with the proposed decomposition-and-recombination method, yields an efficient and competitive system for blind image decomposition, compared with current state-of-the-art methods.