2024-03-29

Title: JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition

• Authors: Chang Sun, Hong Yang, Bo Qin
• Subjects: cs.CV, cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.18843
• Pdf URL: https://arxiv.org/pdf/2403.18843
• Copy Paste: [[2403.18843]] JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition(https://arxiv.org/abs/2403.18843)
• Keywords: robust, extraction, generative
• Abstract: Visual Speech Recognition (VSR) tasks are generally recognized to have a lower theoretical performance ceiling than Automatic Speech Recognition (ASR), owing to the inherent limitations of conveying semantic information visually. To mitigate this challenge, this paper introduces an advanced knowledge distillation approach using a Joint-Embedding Predictive Architecture (JEPA), named JEP-KD, designed to more effectively utilize audio features during model training. Central to JEP-KD is the inclusion of a generative network within the embedding layer, which enhances the video encoder's capacity for semantic feature extraction and brings it into closer alignment with the audio features from a pre-trained ASR model's encoder. This approach aims to progressively reduce the performance gap between VSR and ASR. Moreover, a comprehensive multimodal, multistage training regimen for the JEP-KD framework is established, bolstering the robustness and efficacy of the training process. Experiment results demonstrate that JEP-KD significantly improves the performance of VSR models and demonstrates versatility across different VSR platforms, indicating its potential for broader application within other multimodal tasks.

Title: Targeted Visualization of the Backbone of Encoder LLMs

• Authors: Isaac Roberts, Alexander Schulz, Luca Hermes, Barbara Hammer
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.18872
• Pdf URL: https://arxiv.org/pdf/2403.18872
• Copy Paste: [[2403.18872]] Targeted Visualization of the Backbone of Encoder LLMs(https://arxiv.org/abs/2403.18872)
• Keywords: attack, explainability, large language model
• Abstract: Attention based Large Language Models (LLMs) are the state-of-the-art in natural language processing (NLP). The two most common architectures are encoders such as BERT, and decoders like the GPT models. Despite the success of encoder models, on which we focus in this work, they also bear several risks, including issues with bias or their susceptibility for adversarial attacks, signifying the necessity for explainable AI to detect such issues. While there does exist various local explainability methods focusing on the prediction of single inputs, global methods based on dimensionality reduction for classification inspection, which have emerged in other domains and that go further than just using t-SNE in the embedding space, are not widely spread in NLP. To reduce this gap, we investigate the application of DeepView, a method for visualizing a part of the decision function together with a data set in two dimensions, to the NLP domain. While in previous work, DeepView has been used to inspect deep image classification models, we demonstrate how to apply it to BERT-based NLP classifiers and investigate its usability in this domain, including settings with adversarially perturbed input samples and pre-trained, fine-tuned, and multi-task models.

Title: AIC-UNet: Anatomy-informed Cascaded UNet for Robust Multi-Organ Segmentation

• Authors: Young Seok Jeon, Hongfei Yang, Huazhu Fu, Mengling Feng
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.18878
• Pdf URL: https://arxiv.org/pdf/2403.18878
• Copy Paste: [[2403.18878]] AIC-UNet: Anatomy-informed Cascaded UNet for Robust Multi-Organ Segmentation(https://arxiv.org/abs/2403.18878)
• Keywords: robust, segmentation
• Abstract: Imposing key anatomical features, such as the number of organs, their shapes, sizes, and relative positions, is crucial for building a robust multi-organ segmentation model. Current attempts to incorporate anatomical features include broadening effective receptive fields (ERF) size with resource- and data-intensive modules such as self-attention or introducing organ-specific topology regularizers, which may not scale to multi-organ segmentation problems where inter-organ relation also plays a huge role. We introduce a new approach to impose anatomical constraints on any existing encoder-decoder segmentation model by conditioning model prediction with learnable anatomy prior. More specifically, given an abdominal scan, a part of the encoder spatially warps a learnable prior to align with the given input scan using thin plate spline (TPS) grid interpolation. The warped prior is then integrated during the decoding phase to guide the model for more anatomy-informed predictions. Code is available at \hyperlink{https://anonymous.4open.science/r/AIC-UNet-7048}{https://anonymous.4open.science/r/AIC-UNet-7048}.

Title: Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning

• Authors: Huiyi Wang, Haodong Lu, Lina Yao, Dong Gong
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.18886
• Pdf URL: https://arxiv.org/pdf/2403.18886
• Copy Paste: [[2403.18886]] Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning(https://arxiv.org/abs/2403.18886)
• Keywords: transformer
• Abstract: Continual learning aims to learn from a stream of continuously arriving data with minimum forgetting of previously learned knowledge. While previous works have explored the effectiveness of leveraging the generalizable knowledge from pre-trained models in continual learning, existing parameter-efficient fine-tuning approaches focus on the use of a predetermined or task-wise set of adapters or prompts. However, these approaches still suffer from forgetting due to task interference on jointly used parameters or restricted flexibility. The reliance on a static model architecture may lead to the allocation of excessive parameters that are not essential or, conversely, inadequate adaptation for downstream tasks, given that the scale and distribution of incoming data are unpredictable in continual learning. We propose Self-Expansion of pre-trained models with Modularized Adaptation (SEMA), a novel fine-tuning approach which automatically decides to reuse or add adapter modules on demand in continual learning, depending on whether drastic distribution shift that could not be handled by existing modules is detected at different representation levels. We design each adapter module to consist of an adapter and a representation descriptor, specifically, implemented as an autoencoder. The representation descriptor functions as a distributional shift indicator during training and triggers adapter expansion. For better usage of the adapters, an expandable weighting router is learned jointly for mixture of adapter outputs. By comparing with vision-transformer-based continual learning adaptation methods, we demonstrate that the proposed framework outperforms the state-of-the-art without memory rehearsal.

Title: A Geometric Explanation of the Likelihood OOD Detection Paradox

• Authors: Hamidreza Kamkari, Brendan Leigh Ross, Jesse C. Cresswell, Anthony L. Caterini, Rahul G. Krishnan, Gabriel Loaiza-Ganem
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.18910
• Pdf URL: https://arxiv.org/pdf/2403.18910
• Copy Paste: [[2403.18910]] A Geometric Explanation of the Likelihood OOD Detection Paradox(https://arxiv.org/abs/2403.18910)
• Keywords: diffusion, generative
• Abstract: Likelihood-based deep generative models (DGMs) commonly exhibit a puzzling behaviour: when trained on a relatively complex dataset, they assign higher likelihood values to out-of-distribution (OOD) data from simpler sources. Adding to the mystery, OOD samples are never generated by these DGMs despite having higher likelihoods. This two-pronged paradox has yet to be conclusively explained, making likelihood-based OOD detection unreliable. Our primary observation is that high-likelihood regions will not be generated if they contain minimal probability mass. We demonstrate how this seeming contradiction of large densities yet low probability mass can occur around data confined to low-dimensional manifolds. We also show that this scenario can be identified through local intrinsic dimension (LID) estimation, and propose a method for OOD detection which pairs the likelihoods and LID estimates obtained from a pre-trained DGM. Our method can be applied to normalizing flows and score-based diffusion models, and obtains results which match or surpass state-of-the-art OOD detection benchmarks using the same DGM backbones. Our code is available at https://github.com/layer6ai-labs/dgm_ood_detection.

Title: PLOT-TAL -- Prompt Learning with Optimal Transport for Few-Shot Temporal Action Localization

• Authors: Edward Fish, Jon Weinbren, Andrew Gilbert
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.18915
• Pdf URL: https://arxiv.org/pdf/2403.18915
• Copy Paste: [[2403.18915]] PLOT-TAL -- Prompt Learning with Optimal Transport for Few-Shot Temporal Action Localization(https://arxiv.org/abs/2403.18915)
• Keywords: robust
• Abstract: This paper introduces a novel approach to temporal action localization (TAL) in few-shot learning. Our work addresses the inherent limitations of conventional single-prompt learning methods that often lead to overfitting due to the inability to generalize across varying contexts in real-world videos. Recognizing the diversity of camera views, backgrounds, and objects in videos, we propose a multi-prompt learning framework enhanced with optimal transport. This design allows the model to learn a set of diverse prompts for each action, capturing general characteristics more effectively and distributing the representation to mitigate the risk of overfitting. Furthermore, by employing optimal transport theory, we efficiently align these prompts with action features, optimizing for a comprehensive representation that adapts to the multifaceted nature of video data. Our experiments demonstrate significant improvements in action localization accuracy and robustness in few-shot settings on the standard challenging datasets of THUMOS-14 and EpicKitchens100, highlighting the efficacy of our multi-prompt optimal transport approach in overcoming the challenges of conventional few-shot TAL methods.

Title: CPR: Retrieval Augmented Generation for Copyright Protection

• Authors: Aditya Golatkar, Alessandro Achille, Luca Zancato, Yu-Xiang Wang, Ashwin Swaminathan, Stefano Soatto
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.18920
• Pdf URL: https://arxiv.org/pdf/2403.18920
• Copy Paste: [[2403.18920]] CPR: Retrieval Augmented Generation for Copyright Protection(https://arxiv.org/abs/2403.18920)
• Keywords: protect, attack, robust, diffusion
• Abstract: Retrieval Augmented Generation (RAG) is emerging as a flexible and robust technique to adapt models to private users data without training, to handle credit attribution, and to allow efficient machine unlearning at scale. However, RAG techniques for image generation may lead to parts of the retrieved samples being copied in the model's output. To reduce risks of leaking private information contained in the retrieved set, we introduce Copy-Protected generation with Retrieval (CPR), a new method for RAG with strong copyright protection guarantees in a mixed-private setting for diffusion models.CPR allows to condition the output of diffusion models on a set of retrieved images, while also guaranteeing that unique identifiable information about those example is not exposed in the generated outputs. In particular, it does so by sampling from a mixture of public (safe) distribution and private (user) distribution by merging their diffusion scores at inference. We prove that CPR satisfies Near Access Freeness (NAF) which bounds the amount of information an attacker may be able to extract from the generated images. We provide two algorithms for copyright protection, CPR-KL and CPR-Choose. Unlike previously proposed rejection-sampling-based NAF methods, our methods enable efficient copyright-protected sampling with a single run of backward diffusion. We show that our method can be applied to any pre-trained conditional diffusion model, such as Stable Diffusion or unCLIP. In particular, we empirically show that applying CPR on top of unCLIP improves quality and text-to-image alignment of the generated results (81.4 to 83.17 on TIFA benchmark), while enabling credit attribution, copy-right protection, and deterministic, constant time, unlearning.

Title: Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D

• Authors: Mukund Varma T, Peihao Wang, Zhiwen Fan, Zhangyang Wang, Hao Su, Ravi Ramamoorthi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.18922
• Pdf URL: https://arxiv.org/pdf/2403.18922
• Copy Paste: [[2403.18922]] Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D(https://arxiv.org/abs/2403.18922)
• Keywords: segmentation
• Abstract: In recent years, there has been an explosion of 2D vision models for numerous tasks such as semantic segmentation, style transfer or scene editing, enabled by large-scale 2D image datasets. At the same time, there has been renewed interest in 3D scene representations such as neural radiance fields from multi-view images. However, the availability of 3D or multiview data is still substantially limited compared to 2D image datasets, making extending 2D vision models to 3D data highly desirable but also very challenging. Indeed, extending a single 2D vision operator like scene editing to 3D typically requires a highly creative method specialized to that task and often requires per-scene optimization. In this paper, we ask the question of whether any 2D vision model can be lifted to make 3D consistent predictions. We answer this question in the affirmative; our new Lift3D method trains to predict unseen views on feature spaces generated by a few visual models (i.e. DINO and CLIP), but then generalizes to novel vision operators and tasks, such as style transfer, super-resolution, open vocabulary segmentation and image colorization; for some of these tasks, there is no comparable previous 3D method. In many cases, we even outperform state-of-the-art methods specialized for the task in question. Moreover, Lift3D is a zero-shot method, in the sense that it requires no task-specific training, nor scene-specific optimization.

Title: Enhancing Efficiency in Sparse Models with Sparser Selection

• Authors: Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, Zenglin Xu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.18926
• Pdf URL: https://arxiv.org/pdf/2403.18926
• Copy Paste: [[2403.18926]] Enhancing Efficiency in Sparse Models with Sparser Selection(https://arxiv.org/abs/2403.18926)
• Keywords: transformer
• Abstract: Sparse models, including sparse Mixture-of-Experts (MoE) models, have emerged as an effective approach for scaling Transformer models. However, they often suffer from computational inefficiency since a significant number of parameters are unnecessarily involved in computations via multiplying values by zero or low activation values. To address this issue, we present \tool, a novel MoE designed to enhance both the efficacy and efficiency of sparse MoE models. \tool leverages small experts and a threshold-based router to enable tokens to selectively engage only essential parameters. Our extensive experiments on language modeling and machine translation tasks demonstrate that \tool can enhance model performance while decreasing the computation load at MoE layers by over 50\% without sacrificing performance. Furthermore, we present the versatility of \tool by applying it to dense models, enabling sparse computation during inference. We provide a comprehensive analysis and make our code available at https://anonymous.4open.science/r/XMoE.

Title: Measuring Political Bias in Large Language Models: What Is Said and How It Is Said

• Authors: Yejin Bang, Delong Chen, Nayeon Lee, Pascale Fung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.18932
• Pdf URL: https://arxiv.org/pdf/2403.18932
• Copy Paste: [[2403.18932]] Measuring Political Bias in Large Language Models: What Is Said and How It Is Said(https://arxiv.org/abs/2403.18932)
• Keywords: large language model
• Abstract: We propose to measure political bias in LLMs by analyzing both the content and style of their generated content regarding political issues. Existing benchmarks and measures focus on gender and racial biases. However, political bias exists in LLMs and can lead to polarization and other harms in downstream applications. In order to provide transparency to users, we advocate that there should be fine-grained and explainable measures of political biases generated by LLMs. Our proposed measure looks at different political issues such as reproductive rights and climate change, at both the content (the substance of the generation) and the style (the lexical polarity) of such bias. We measured the political bias in eleven open-sourced LLMs and showed that our proposed framework is easily scalable to other topics and is explainable.

Title: Reshaping Free-Text Radiology Notes Into Structured Reports With Generative Transformers

• Authors: Laura Bergomi, Tommaso M. Buonocore, Paolo Antonazzo, Lorenzo Alberghi, Riccardo Bellazzi, Lorenzo Preda, Chandra Bortolotto, Enea Parimbelli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.18938
• Pdf URL: https://arxiv.org/pdf/2403.18938
• Copy Paste: [[2403.18938]] Reshaping Free-Text Radiology Notes Into Structured Reports With Generative Transformers(https://arxiv.org/abs/2403.18938)
• Keywords: transformer, generative, large language model
• Abstract: BACKGROUND: Radiology reports are typically written in a free-text format, making clinical information difficult to extract and use. Recently the adoption of structured reporting (SR) has been recommended by various medical societies thanks to the advantages it offers, e.g. standardization, completeness and information retrieval. We propose a pipeline to extract information from free-text radiology reports, that fits with the items of the reference SR registry proposed by a national society of interventional and medical radiology, focusing on CT staging of patients with lymphoma. METHODS: Our work aims to leverage the potential of Natural Language Processing (NLP) and Transformer-based models to deal with automatic SR registry filling. With the availability of 174 radiology reports, we investigate a rule-free generative Question Answering approach based on a domain-specific version of T5 (IT5). Two strategies (batch-truncation and ex-post combination) are implemented to comply with the model's context length limitations. Performance is evaluated in terms of strict accuracy, F1, and format accuracy, and compared with the widely used GPT-3.5 Large Language Model. A 5-point Likert scale questionnaire is used to collect human-expert feedback on the similarity between medical annotations and generated answers. RESULTS: The combination of fine-tuning and batch splitting allows IT5 to achieve notable results; it performs on par with GPT-3.5 albeit its size being a thousand times smaller in terms of parameters. Human-based assessment scores show a high correlation (Spearman's correlation coefficients>0.88, p-values<0.001) with AI performance metrics (F1) and confirm the superior ability of LLMs (i.e., GPT-3.5, 175B of parameters) in generating plausible human-like statements.

Title: Self-Supervised Interpretable Sensorimotor Learning via Latent Functional Modularity

• Authors: Hyunki Seong, David Hyunchul Shim
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.18947
• Pdf URL: https://arxiv.org/pdf/2403.18947
• Copy Paste: [[2403.18947]] Self-Supervised Interpretable Sensorimotor Learning via Latent Functional Modularity(https://arxiv.org/abs/2403.18947)
• Keywords: interpretability, explainability
• Abstract: We introduce MoNet, a novel method that combines end-to-end learning with modular network architectures for self-supervised and interpretable sensorimotor learning. MoNet is composed of three functionally distinct neural modules: Perception, Planning, and Control. Leveraging its inherent modularity through a cognition-guided contrastive loss function, MoNet efficiently learns task-specific decision-making processes in latent space, without requiring task-level supervision. Moreover, our method incorporates an online post-hoc explainability approach, which enhances the interpretability of the end-to-end inferences without a trade-off in sensorimotor performance. In real-world indoor environments, MoNet demonstrates effective visual autonomous navigation, surpassing baseline models by 11% to 47% in task specificity analysis. We further delve into the interpretability of our network through the post-hoc analysis of perceptual saliency maps and latent decision vectors. This offers insights into the incorporation of explainable artificial intelligence within the realm of robotic learning, encompassing both perceptual and behavioral perspectives.

Title: A Survey on Large Language Models from Concept to Implementation

• Authors: Chen Wang, Jin Zhao, Jiaqi Gong
• Subjects: cs.CL, cs.AI, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.18969
• Pdf URL: https://arxiv.org/pdf/2403.18969
• Copy Paste: [[2403.18969]] A Survey on Large Language Models from Concept to Implementation(https://arxiv.org/abs/2403.18969)
• Keywords: transformer, large language model
• Abstract: Recent advancements in Large Language Models (LLMs), particularly those built on Transformer architectures, have significantly broadened the scope of natural language processing (NLP) applications, transcending their initial use in chatbot technology. This paper investigates the multifaceted applications of these models, with an emphasis on the GPT series. This exploration focuses on the transformative impact of artificial intelligence (AI) driven tools in revolutionizing traditional tasks like coding and problem-solving, while also paving new paths in research and development across diverse industries. From code interpretation and image captioning to facilitating the construction of interactive systems and advancing computational domains, Transformer models exemplify a synergy of deep learning, data analysis, and neural network design. This survey provides an in-depth look at the latest research in Transformer models, highlighting their versatility and the potential they hold for transforming diverse application sectors, thereby offering readers a comprehensive understanding of the current and future landscape of Transformer-based LLMs in practical applications.

Title: A Novel Corpus of Annotated Medical Imaging Reports and Information Extraction Results Using BERT-based Language Models

• Authors: Namu Park, Kevin Lybarger, Giridhar Kaushik Ramachandran, Spencer Lewis, Aashka Damani, Ozlem Uzuner, Martin Gunn, Meliha Yetisgen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.18975
• Pdf URL: https://arxiv.org/pdf/2403.18975
• Copy Paste: [[2403.18975]] A Novel Corpus of Annotated Medical Imaging Reports and Information Extraction Results Using BERT-based Language Models(https://arxiv.org/abs/2403.18975)
• Keywords: extraction, transformer
• Abstract: Medical imaging is critical to the diagnosis, surveillance, and treatment of many health conditions, including oncological, neurological, cardiovascular, and musculoskeletal disorders, among others. Radiologists interpret these complex, unstructured images and articulate their assessments through narrative reports that remain largely unstructured. This unstructured narrative must be converted into a structured semantic representation to facilitate secondary applications such as retrospective analyses or clinical decision support. Here, we introduce the Corpus of Annotated Medical Imaging Reports (CAMIR), which includes 609 annotated radiology reports from three imaging modality types: Computed Tomography, Magnetic Resonance Imaging, and Positron Emission Tomography-Computed Tomography. Reports were annotated using an event-based schema that captures clinical indications, lesions, and medical problems. Each event consists of a trigger and multiple arguments, and a majority of the argument types, including anatomy, normalize the spans to pre-defined concepts to facilitate secondary use. CAMIR uniquely combines a granular event structure and concept normalization. To extract CAMIR events, we explored two BERT (Bi-directional Encoder Representation from Transformers)-based architectures, including an existing architecture (mSpERT) that jointly extracts all event information and a multi-step approach (PL-Marker++) that we augmented for the CAMIR schema.

Title: "Sorry, Come Again?" Prompting -- Enhancing Comprehension and Diminishing Hallucination with [PAUSE]-injected Optimal Paraphrasing

• Authors: Vipula Rawte, S.M Towhidul Islam Tonmoy, S M Mehedi Zaman, Prachi Priya, Aman Chadha, Amit P. Sheth, Amitava Das
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.18976
• Pdf URL: https://arxiv.org/pdf/2403.18976
• Copy Paste: [[2403.18976]] "Sorry, Come Again?" Prompting -- Enhancing Comprehension and Diminishing Hallucination with [PAUSE]-injected Optimal Paraphrasing(https://arxiv.org/abs/2403.18976)
• Keywords: large language model
• Abstract: Hallucination has emerged as the most vulnerable aspect of contemporary Large Language Models (LLMs). In this paper, we introduce the Sorry, Come Again (SCA) prompting, aimed to avoid LLM hallucinations by enhancing comprehension through: (i) optimal paraphrasing and (ii) injecting [PAUSE] tokens to delay LLM generation. First, we provide an in-depth analysis of linguistic nuances: formality, readability, and concreteness of prompts for 21 LLMs, and elucidate how these nuances contribute to hallucinated generation. Prompts with lower readability, formality, or concreteness pose comprehension challenges for LLMs, similar to those faced by humans. In such scenarios, an LLM tends to speculate and generate content based on its imagination (associative memory) to fill these information gaps. Although these speculations may occasionally align with factual information, their accuracy is not assured, often resulting in hallucination. Recent studies reveal that an LLM often neglects the middle sections of extended prompts, a phenomenon termed as lost in the middle. While a specific paraphrase may suit one LLM, the same paraphrased version may elicit a different response from another LLM. Therefore, we propose an optimal paraphrasing technique to identify the most comprehensible paraphrase of a given prompt, evaluated using Integrated Gradient (and its variations) to guarantee that the LLM accurately processes all words. While reading lengthy sentences, humans often pause at various points to better comprehend the meaning read thus far. We have fine-tuned an LLM with injected [PAUSE] tokens, allowing the LLM to pause while reading lengthier prompts. This has brought several key contributions: (i) determining the optimal position to inject [PAUSE], (ii) determining the number of [PAUSE] tokens to be inserted, and (iii) introducing reverse proxy tuning to fine-tune the LLM for [PAUSE] insertion.

Title: TextCraftor: Your Text Encoder Can be Image Quality Controller

• Authors: Yanyu Li, Xian Liu, Anil Kag, Ju Hu, Yerlan Idelbayev, Dhritiman Sagar, Yanzhi Wang, Sergey Tulyakov, Jian Ren
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.18978
• Pdf URL: https://arxiv.org/pdf/2403.18978
• Copy Paste: [[2403.18978]] TextCraftor: Your Text Encoder Can be Image Quality Controller(https://arxiv.org/abs/2403.18978)
• Keywords: diffusion, generative, large language model
• Abstract: Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have revolutionized the field of content generation, enabling significant advancements in areas like image editing and video synthesis. Despite their formidable capabilities, these models are not without their limitations. It is still challenging to synthesize an image that aligns well with the input text, and multiple runs with carefully crafted prompts are required to achieve satisfactory results. To mitigate these limitations, numerous studies have endeavored to fine-tune the pre-trained diffusion models, i.e., UNet, utilizing various technologies. Yet, amidst these efforts, a pivotal question of text-to-image diffusion model training has remained largely unexplored: Is it possible and feasible to fine-tune the text encoder to improve the performance of text-to-image diffusion models? Our findings reveal that, instead of replacing the CLIP text encoder used in Stable Diffusion with other large language models, we can enhance it through our proposed fine-tuning approach, TextCraftor, leading to substantial improvements in quantitative benchmarks and human assessments. Interestingly, our technique also empowers controllable image generation through the interpolation of different text encoders fine-tuned with various rewards. We also demonstrate that TextCraftor is orthogonal to UNet finetuning, and can be combined to further improve generative quality.

Title: Robustness and Visual Explanation for Black Box Image, Video, and ECG Signal Classification with Reinforcement Learning

• Authors: Soumyendu Sarkar, Ashwin Ramesh Babu, Sajad Mousavi, Vineet Gundecha, Avisek Naug, Sahand Ghorbanpour
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV, cs.MA
• Abstract URL: https://arxiv.org/abs/2403.18985
• Pdf URL: https://arxiv.org/pdf/2403.18985
• Copy Paste: [[2403.18985]] Robustness and Visual Explanation for Black Box Image, Video, and ECG Signal Classification with Reinforcement Learning(https://arxiv.org/abs/2403.18985)
• Keywords: attack, robust, interpretability
• Abstract: We present a generic Reinforcement Learning (RL) framework optimized for crafting adversarial attacks on different model types spanning from ECG signal analysis (1D), image classification (2D), and video classification (3D). The framework focuses on identifying sensitive regions and inducing misclassifications with minimal distortions and various distortion types. The novel RL method outperforms state-of-the-art methods for all three applications, proving its efficiency. Our RL approach produces superior localization masks, enhancing interpretability for image classification and ECG analysis models. For applications such as ECG analysis, our platform highlights critical ECG segments for clinicians while ensuring resilience against prevalent distortions. This comprehensive tool aims to bolster both resilience with adversarial training and transparency across varied applications and data types.

Title: Dealing with Imbalanced Classes in Bot-IoT Dataset

• Authors: Jesse Atuhurra, Takanori Hara, Yuanyu Zhang, Masahiro Sasabe, Shoji Kasahara
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.18989
• Pdf URL: https://arxiv.org/pdf/2403.18989
• Copy Paste: [[2403.18989]] Dealing with Imbalanced Classes in Bot-IoT Dataset(https://arxiv.org/abs/2403.18989)
• Keywords: protect, attack, robust
• Abstract: With the rapidly spreading usage of Internet of Things (IoT) devices, a network intrusion detection system (NIDS) plays an important role in detecting and protecting various types of attacks in the IoT network. To evaluate the robustness of the NIDS in the IoT network, the existing work proposed a realistic botnet dataset in the IoT network (Bot-IoT dataset) and applied it to machine learning-based anomaly detection. This dataset contains imbalanced normal and attack packets because the number of normal packets is much smaller than that of attack ones. The nature of imbalanced data may make it difficult to identify the minority class correctly. In this thesis, to address the class imbalance problem in the Bot-IoT dataset, we propose a binary classification method with synthetic minority over-sampling techniques (SMOTE). The proposed classifier aims to detect attack packets and overcome the class imbalance problem using the SMOTE algorithm. Through numerical results, we demonstrate the proposed classifier's fundamental characteristics and the impact of imbalanced data on its performance.

Title: Envisioning MedCLIP: A Deep Dive into Explainability for Medical Vision-Language Models

• Authors: Anees Ur Rehman Hashmi, Dwarikanath Mahapatra, Mohammad Yaqub
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.18996
• Pdf URL: https://arxiv.org/pdf/2403.18996
• Copy Paste: [[2403.18996]] Envisioning MedCLIP: A Deep Dive into Explainability for Medical Vision-Language Models(https://arxiv.org/abs/2403.18996)
• Keywords: explainability
• Abstract: Explaining Deep Learning models is becoming increasingly important in the face of daily emerging multimodal models, particularly in safety-critical domains like medical imaging. However, the lack of detailed investigations into the performance of explainability methods on these models is widening the gap between their development and safe deployment. In this work, we analyze the performance of various explainable AI methods on a vision-language model, MedCLIP, to demystify its inner workings. We also provide a simple methodology to overcome the shortcomings of these methods. Our work offers a different new perspective on the explainability of a recent well-known VLM in the medical domain and our assessment method is generalizable to other current and possible future VLMs.

Title: Cross--domain Fiber Cluster Shape Analysis for Language Performance Cognitive Score Prediction

• Authors: Yui Lo, Yuqian Chen, Dongnan Liu, Wan Liu, Leo Zekelman, Fan Zhang, Yogesh Rathi, Nikos Makris, Alexandra J. Golby, Weidong Cai, Lauren J. O'Donnell
• Subjects: cs.CV, cs.AI, eess.IV, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2403.19001
• Pdf URL: https://arxiv.org/pdf/2403.19001
• Copy Paste: [[2403.19001]] Cross--domain Fiber Cluster Shape Analysis for Language Performance Cognitive Score Prediction(https://arxiv.org/abs/2403.19001)
• Keywords: diffusion, transformer
• Abstract: Shape plays an important role in computer graphics, offering informative features to convey an object's morphology and functionality. Shape analysis in brain imaging can help interpret structural and functionality correlations of the human brain. In this work, we investigate the shape of the brain's 3D white matter connections and its potential predictive relationship to human cognitive function. We reconstruct brain connections as sequences of 3D points using diffusion magnetic resonance imaging (dMRI) tractography. To describe each connection, we extract 12 shape descriptors in addition to traditional dMRI connectivity and tissue microstructure features. We introduce a novel framework, Shape--fused Fiber Cluster Transformer (SFFormer), that leverages a multi-head cross-attention feature fusion module to predict subject-specific language performance based on dMRI tractography. We assess the performance of the method on a large dataset including 1065 healthy young adults. The results demonstrate that both the transformer-based SFFormer model and its inter/intra feature fusion with shape, microstructure, and connectivity are informative, and together, they improve the prediction of subject-specific language performance scores. Overall, our results indicate that the shape of the brain's connections is predictive of human language function.

Title: Towards Sustainable SecureML: Quantifying Carbon Footprint of Adversarial Machine Learning

• Authors: Syed Mhamudul Hasan, Abdur R. Shahid, Ahmed Imteaj
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.19009
• Pdf URL: https://arxiv.org/pdf/2403.19009
• Copy Paste: [[2403.19009]] Towards Sustainable SecureML: Quantifying Carbon Footprint of Adversarial Machine Learning(https://arxiv.org/abs/2403.19009)
• Keywords: secure, security, defense, attack, robust
• Abstract: The widespread adoption of machine learning (ML) across various industries has raised sustainability concerns due to its substantial energy usage and carbon emissions. This issue becomes more pressing in adversarial ML, which focuses on enhancing model security against different network-based attacks. Implementing defenses in ML systems often necessitates additional computational resources and network security measures, exacerbating their environmental impacts. In this paper, we pioneer the first investigation into adversarial ML's carbon footprint, providing empirical evidence connecting greater model robustness to higher emissions. Addressing the critical need to quantify this trade-off, we introduce the Robustness Carbon Trade-off Index (RCTI). This novel metric, inspired by economic elasticity principles, captures the sensitivity of carbon emissions to changes in adversarial robustness. We demonstrate the RCTI through an experiment involving evasion attacks, analyzing the interplay between robustness against attacks, performance, and carbon emissions.

Title: WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion

• Authors: Khiem Vuong, N. Dinesh Reddy, Robert Tamburo, Srinivasa G. Narasimhan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19022
• Pdf URL: https://arxiv.org/pdf/2403.19022
• Copy Paste: [[2403.19022]] WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion(https://arxiv.org/abs/2403.19022)
• Keywords: robust, segmentation
• Abstract: Current methods for 2D and 3D object understanding struggle with severe occlusions in busy urban environments, partly due to the lack of large-scale labeled ground-truth annotations for learning occlusion. In this work, we introduce a novel framework for automatically generating a large, realistic dataset of dynamic objects under occlusions using freely available time-lapse imagery. By leveraging off-the-shelf 2D (bounding box, segmentation, keypoint) and 3D (pose, shape) predictions as pseudo-groundtruth, unoccluded 3D objects are identified automatically and composited into the background in a clip-art style, ensuring realistic appearances and physically accurate occlusion configurations. The resulting clip-art image with pseudo-groundtruth enables efficient training of object reconstruction methods that are robust to occlusions. Our method demonstrates significant improvements in both 2D and 3D reconstruction, particularly in scenarios with heavily occluded objects like vehicles and people in urban scenes.

Title: Egocentric Scene-aware Human Trajectory Prediction

• Authors: Weizhuo Wang, C. Karen Liu, Monroe Kennedy III
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19026
• Pdf URL: https://arxiv.org/pdf/2403.19026
• Copy Paste: [[2403.19026]] Egocentric Scene-aware Human Trajectory Prediction(https://arxiv.org/abs/2403.19026)
• Keywords: diffusion
• Abstract: Wearable collaborative robots stand to assist human wearers who need fall prevention assistance or wear exoskeletons. Such a robot needs to be able to predict the ego motion of the wearer based on egocentric vision and the surrounding scene. In this work, we leveraged body-mounted cameras and sensors to anticipate the trajectory of human wearers through complex surroundings. To facilitate research in ego-motion prediction, we have collected a comprehensive walking scene navigation dataset centered on the user's perspective. We present a method to predict human motion conditioning on the surrounding static scene. Our method leverages a diffusion model to produce a distribution of potential future trajectories, taking into account the user's observation of the environment. We introduce a compact representation to encode the user's visual memory of the surroundings, as well as an efficient sample-generating technique to speed up real-time inference of a diffusion model. We ablate our model and compare it to baselines, and results show that our model outperforms existing methods on key metrics of collision avoidance and trajectory mode coverage.

Title: Evaluating Large Language Models for Health-Related Text Classification Tasks with Public Social Media Data

• Authors: Yuting Guo, Anthony Ovadje, Mohammed Ali Al-Garadi, Abeed Sarker
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19031
• Pdf URL: https://arxiv.org/pdf/2403.19031
• Copy Paste: [[2403.19031]] Evaluating Large Language Models for Health-Related Text Classification Tasks with Public Social Media Data(https://arxiv.org/abs/2403.19031)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable success in NLP tasks. However, there is a paucity of studies that attempt to evaluate their performances on social media-based health-related natural language processing tasks, which have traditionally been difficult to achieve high scores in. We benchmarked one supervised classic machine learning model based on Support Vector Machines (SVMs), three supervised pretrained language models (PLMs) based on RoBERTa, BERTweet, and SocBERT, and two LLM based classifiers (GPT3.5 and GPT4), across 6 text classification tasks. We developed three approaches for leveraging LLMs for text classification: employing LLMs as zero-shot classifiers, us-ing LLMs as annotators to annotate training data for supervised classifiers, and utilizing LLMs with few-shot examples for augmentation of manually annotated data. Our comprehensive experiments demonstrate that employ-ing data augmentation using LLMs (GPT-4) with relatively small human-annotated data to train lightweight supervised classification models achieves superior results compared to training with human-annotated data alone. Supervised learners also outperform GPT-4 and GPT-3.5 in zero-shot settings. By leveraging this data augmentation strategy, we can harness the power of LLMs to develop smaller, more effective domain-specific NLP models. LLM-annotated data without human guidance for training light-weight supervised classification models is an ineffective strategy. However, LLM, as a zero-shot classifier, shows promise in excluding false negatives and potentially reducing the human effort required for data annotation. Future investigations are imperative to explore optimal training data sizes and the optimal amounts of augmented data.

Title: Illicit object detection in X-ray images using Vision Transformers

• Authors: Jorgen Cani, Ioannis Mademlis, Adamantia Anna Rebolledo Chrysochoou, Georgios Th. Papadopoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19043
• Pdf URL: https://arxiv.org/pdf/2403.19043
• Copy Paste: [[2403.19043]] Illicit object detection in X-ray images using Vision Transformers(https://arxiv.org/abs/2403.19043)
• Keywords: security, transformer
• Abstract: Illicit object detection is a critical task performed at various high-security locations, including airports, train stations, subways, and ports. The continuous and tedious work of examining thousands of X-ray images per hour can be mentally taxing. Thus, Deep Neural Networks (DNNs) can be used to automate the X-ray image analysis process, improve efficiency and alleviate the security officers' inspection burden. The neural architectures typically utilized in relevant literature are Convolutional Neural Networks (CNNs), with Vision Transformers (ViTs) rarely employed. In order to address this gap, this paper conducts a comprehensive evaluation of relevant ViT architectures on illicit item detection in X-ray images. This study utilizes both Transformer and hybrid backbones, such as SWIN and NextViT, and detectors, such as DINO and RT-DETR. The results demonstrate the remarkable accuracy of the DINO Transformer detector in the low-data regime, the impressive real-time performance of YOLOv8, and the effectiveness of the hybrid NextViT backbone.

Title: LITA: Language Instructed Temporal-Localization Assistant

• Authors: De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, Jan Kautz
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19046
• Pdf URL: https://arxiv.org/pdf/2403.19046
• Copy Paste: [[2403.19046]] LITA: Language Instructed Temporal-Localization Assistant(https://arxiv.org/abs/2403.19046)
• Keywords: large language model
• Abstract: There has been tremendous progress in multimodal Large Language Models (LLMs). Recent works have extended these models to video input with promising instruction following capabilities. However, an important missing piece is temporal localization. These models cannot accurately answer the "When?" questions. We identify three key aspects that limit their temporal localization capabilities: (i) time representation, (ii) architecture, and (iii) data. We address these shortcomings by proposing Language Instructed Temporal-Localization Assistant (LITA) with the following features: (1) We introduce time tokens that encode timestamps relative to the video length to better represent time in videos. (2) We introduce SlowFast tokens in the architecture to capture temporal information at fine temporal resolution. (3) We emphasize temporal localization data for LITA. In addition to leveraging existing video datasets with timestamps, we propose a new task, Reasoning Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for learning and evaluating this task. Reasoning temporal localization requires both the reasoning and temporal localization of Video LLMs. LITA demonstrates strong performance on this challenging task, nearly doubling the temporal mean intersection-over-union (mIoU) of baselines. In addition, we show that our emphasis on temporal localization also substantially improves video-based text generation compared to existing Video LLMs, including a 36% relative improvement of Temporal Understanding. Code is available at: https://github.com/NVlabs/LITA

Title: Detecting Generative Parroting through Overfitting Masked Autoencoders

• Authors: Saeid Asgari Taghanaki, Joseph Lambourne
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19050
• Pdf URL: https://arxiv.org/pdf/2403.19050
• Copy Paste: [[2403.19050]] Detecting Generative Parroting through Overfitting Masked Autoencoders(https://arxiv.org/abs/2403.19050)
• Keywords: generative
• Abstract: The advent of generative AI models has revolutionized digital content creation, yet it introduces challenges in maintaining copyright integrity due to generative parroting, where models mimic their training data too closely. Our research presents a novel approach to tackle this issue by employing an overfitted Masked Autoencoder (MAE) to detect such parroted samples effectively. We establish a detection threshold based on the mean loss across the training dataset, allowing for the precise identification of parroted content in modified datasets. Preliminary evaluations demonstrate promising results, suggesting our method's potential to ensure ethical use and enhance the legal compliance of generative models.

Title: CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems

• Authors: Amin Abolghasemi, Zhaochun Ren, Arian Askari, Mohammad Aliannejadi, Maarten de Rijke, Suzan Verberne
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19056
• Pdf URL: https://arxiv.org/pdf/2403.19056
• Copy Paste: [[2403.19056]] CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems(https://arxiv.org/abs/2403.19056)
• Keywords: robust, large language model
• Abstract: An important unexplored aspect in previous work on user satisfaction estimation for Task-Oriented Dialogue (TOD) systems is their evaluation in terms of robustness for the identification of user dissatisfaction: current benchmarks for user satisfaction estimation in TOD systems are highly skewed towards dialogues for which the user is satisfied. The effect of having a more balanced set of satisfaction labels on performance is unknown. However, balancing the data with more dissatisfactory dialogue samples requires further data collection and human annotation, which is costly and time-consuming. In this work, we leverage large language models (LLMs) and unlock their ability to generate satisfaction-aware counterfactual dialogues to augment the set of original dialogues of a test collection. We gather human annotations to ensure the reliability of the generated samples. We evaluate two open-source LLMs as user satisfaction estimators on our augmented collection against state-of-the-art fine-tuned models. Our experiments show that when used as few-shot user satisfaction estimators, open-source LLMs show higher robustness to the increase in the number of dissatisfaction labels in the test collection than the fine-tuned state-of-the-art models. Our results shed light on the need for data augmentation approaches for user satisfaction estimation in TOD systems. We release our aligned counterfactual dialogues, which are curated by human annotation, to facilitate further research on this topic.

Title: Equity in Healthcare: Analyzing Disparities in Machine Learning Predictions of Diabetic Patient Readmissions

• Authors: Zainab Al-Zanbouri, Gauri Sharma, Shaina Raza
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19057
• Pdf URL: https://arxiv.org/pdf/2403.19057
• Copy Paste: [[2403.19057]] Equity in Healthcare: Analyzing Disparities in Machine Learning Predictions of Diabetic Patient Readmissions(https://arxiv.org/abs/2403.19057)
• Keywords: fair
• Abstract: This study investigates how machine learning (ML) models can predict hospital readmissions for diabetic patients fairly and accurately across different demographics (age, gender, race). We compared models like Deep Learning, Generalized Linear Models, Gradient Boosting Machines (GBM), and Naive Bayes. GBM stood out with an F1-score of 84.3% and accuracy of 82.2%, accurately predicting readmissions across demographics. A fairness analysis was conducted across all the models. GBM minimized disparities in predictions, achieving balanced results across genders and races. It showed low False Discovery Rates (FDR) (6-7%) and False Positive Rates (FPR) (5%) for both genders. Additionally, FDRs remained low for racial groups, such as African Americans (8%) and Asians (7%). Similarly, FPRs were consistent across age groups (4%) for both patients under 40 and those above 40, indicating its precision and ability to reduce bias. These findings emphasize the importance of choosing ML models carefully to ensure both accuracy and fairness for all patients. By showcasing effectiveness of various models with fairness metrics, this study promotes personalized medicine and the need for fair ML algorithms in healthcare. This can ultimately reduce disparities and improve outcomes for diabetic patients of all backgrounds.

Title: Generative Quanta Color Imaging

• Authors: Vishal Purohit, Junjie Luo, Yiheng Chi, Qi Guo, Stanley H. Chan, Qiang Qiu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19066
• Pdf URL: https://arxiv.org/pdf/2403.19066
• Copy Paste: [[2403.19066]] Generative Quanta Color Imaging(https://arxiv.org/abs/2403.19066)
• Keywords: generative
• Abstract: The astonishing development of single-photon cameras has created an unprecedented opportunity for scientific and industrial imaging. However, the high data throughput generated by these 1-bit sensors creates a significant bottleneck for low-power applications. In this paper, we explore the possibility of generating a color image from a single binary frame of a single-photon camera. We evidently find this problem being particularly difficult to standard colorization approaches due to the substantial degree of exposure variation. The core innovation of our paper is an exposure synthesis model framed under a neural ordinary differential equation (Neural ODE) that allows us to generate a continuum of exposures from a single observation. This innovation ensures consistent exposure in binary images that colorizers take on, resulting in notably enhanced colorization. We demonstrate applications of the method in single-image and burst colorization and show superior generative performance over baselines. Project website can be found at https://vishal-s-p.github.io/projects/2023/generative_quanta_color.html.

Title: Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach

• Authors: Wei Dong, Xing Zhang, Bihui Chen, Dawei Yan, Zhijun Lin, Qingsen Yan, Peng Wang, Yang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19067
• Pdf URL: https://arxiv.org/pdf/2403.19067
• Copy Paste: [[2403.19067]] Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach(https://arxiv.org/abs/2403.19067)
• Keywords: transformer
• Abstract: Parameter-efficient fine-tuning for pre-trained Vision Transformers aims to adeptly tailor a model to downstream tasks by learning a minimal set of new adaptation parameters while preserving the frozen majority of pre-trained parameters. Striking a balance between retaining the generalizable representation capacity of the pre-trained model and acquiring task-specific features poses a key challenge. Currently, there is a lack of focus on guiding this delicate trade-off. In this study, we approach the problem from the perspective of Singular Value Decomposition (SVD) of pre-trained parameter matrices, providing insights into the tuning dynamics of existing methods. Building upon this understanding, we propose a Residual-based Low-Rank Rescaling (RLRR) fine-tuning strategy. This strategy not only enhances flexibility in parameter tuning but also ensures that new parameters do not deviate excessively from the pre-trained model through a residual design. Extensive experiments demonstrate that our method achieves competitive performance across various downstream image classification tasks, all while maintaining comparable new parameters. We believe this work takes a step forward in offering a unified perspective for interpreting existing methods and serves as motivation for the development of new approaches that move closer to effectively considering the crucial trade-off mentioned above. Our code is available at \href{https://github.com/zstarN70/RLRR.git}{https://github.com/zstarN70/RLRR.git}.

Title: AssetHarvester: A Static Analysis Tool for Detecting Assets Protected by Secrets in Software Artifacts

• Authors: Setu Kumar Basak, K. Virgil English, Ken Ogura, Vitesh Kambara, Bradley Reaves, Laurie Williams
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.19072
• Pdf URL: https://arxiv.org/pdf/2403.19072
• Copy Paste: [[2403.19072]] AssetHarvester: A Static Analysis Tool for Detecting Assets Protected by Secrets in Software Artifacts(https://arxiv.org/abs/2403.19072)
• Keywords: protect
• Abstract: GitGuardian monitored secrets exposure in public GitHub repositories and reported developers leaked over 12 million secrets (database and other credentials) in 2023, indicating a 113% surge from 2021. Despite the availability of secret detection tools, developers ignore the tools' reported warnings because of false positives (25%-99%). However, each secret protects assets of different values accessible through asset identifiers (a DNS name and a public or private IP address). The asset information for a secret can aid developers in filtering false positives and prioritizing secret removal from the source code. However, existing secret detection tools do not provide the asset information, thus presenting difficulty to developers in filtering secrets only by looking at the secret value or finding the assets manually for each reported secret. The goal of our study is to aid software practitioners in prioritizing secrets removal by providing the assets information protected by the secrets through our novel static analysis tool. We present AssetHarvester, a static analysis tool to detect secret-asset pairs in a repository. Since the location of the asset can be distant from where the secret is defined, we investigated secret-asset co-location patterns and found four patterns. To identify the secret-asset pairs of the four patterns, we utilized three approaches (pattern matching, data flow analysis, and fast-approximation heuristics). We curated a benchmark of 1,791 secret-asset pairs of four database types extracted from 188 public GitHub repositories to evaluate the performance of AssetHarvester. AssetHarvester demonstrates precision of (97%), recall (90%), and F1-score (94%) in detecting secret-asset pairs. Our findings indicate that data flow analysis employed in AssetHarvester detects secret-asset pairs with 0% false positives and aids in improving the recall of secret detection tools.

Title: MMCert: Provable Defense against Adversarial Attacks to Multi-modal Models

• Authors: Yanting Wang, Hongye Fu, Wei Zou, Jinyuan Jia
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.19080
• Pdf URL: https://arxiv.org/pdf/2403.19080
• Copy Paste: [[2403.19080]] MMCert: Provable Defense against Adversarial Attacks to Multi-modal Models(https://arxiv.org/abs/2403.19080)
• Keywords: defense, attack, robust, segmentation
• Abstract: Different from a unimodal model whose input is from a single modality, the input (called multi-modal input) of a multi-modal model is from multiple modalities such as image, 3D points, audio, text, etc. Similar to unimodal models, many existing studies show that a multi-modal model is also vulnerable to adversarial perturbation, where an attacker could add small perturbation to all modalities of a multi-modal input such that the multi-modal model makes incorrect predictions for it. Existing certified defenses are mostly designed for unimodal models, which achieve sub-optimal certified robustness guarantees when extended to multi-modal models as shown in our experimental results. In our work, we propose MMCert, the first certified defense against adversarial attacks to a multi-modal model. We derive a lower bound on the performance of our MMCert under arbitrary adversarial attacks with bounded perturbations to both modalities (e.g., in the context of auto-driving, we bound the number of changed pixels in both RGB image and depth image). We evaluate our MMCert using two benchmark datasets: one for the multi-modal road segmentation task and the other for the multi-modal emotion recognition task. Moreover, we compare our MMCert with a state-of-the-art certified defense extended from unimodal models. Our experimental results show that our MMCert outperforms the baseline.

Title: Enhancing Conformal Prediction Using E-Test Statistics

• Authors: A.A.Balinsky, A.D.Balinsky
• Subjects: cs.LG, cs.AI, math.ST
• Abstract URL: https://arxiv.org/abs/2403.19082
• Pdf URL: https://arxiv.org/pdf/2403.19082
• Copy Paste: [[2403.19082]] Enhancing Conformal Prediction Using E-Test Statistics(https://arxiv.org/abs/2403.19082)
• Keywords: robust
• Abstract: Conformal Prediction (CP) serves as a robust framework that quantifies uncertainty in predictions made by Machine Learning (ML) models. Unlike traditional point predictors, CP generates statistically valid prediction regions, also known as prediction intervals, based on the assumption of data exchangeability. Typically, the construction of conformal predictions hinges on p-values. This paper, however, ventures down an alternative path, harnessing the power of e-test statistics to augment the efficacy of conformal predictions by introducing a BB-predictor (bounded from the below predictor).

Title: Learning From Correctness Without Prompting Makes LLM Efficient Reasoner

• Authors: Yuxuan Yao, Han Wu, Zhijiang Guo, Biyan Zhou, Jiahui Gao, Sichun Luo, Hanxu Hou, Xiaojin Fu, Linqi Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19094
• Pdf URL: https://arxiv.org/pdf/2403.19094
• Copy Paste: [[2403.19094]] Learning From Correctness Without Prompting Makes LLM Efficient Reasoner(https://arxiv.org/abs/2403.19094)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated outstanding performance across various tasks, yet they still exhibit limitations such as hallucination, unfaithful reasoning, and toxic content. One potential approach to mitigate these issues is learning from human or external feedback (e.g. tools). In this paper, we introduce an intrinsic self-correct reasoning framework for LLMs that eliminates the need for human feedback, external tools, and handcraft prompts. The proposed framework, based on a multi-step reasoning paradigm \textbf{Le}arning from \textbf{Co}rrectness (\textsc{LeCo}), improves reasoning performance without needing to learn from errors. This paradigm prioritizes learning from correct reasoning steps, and a unique method to measure confidence for each reasoning step based on generation logits. Experimental results across various multi-step reasoning tasks demonstrate the effectiveness of the framework in improving reasoning performance with reduced token consumption.

Title: AAPMT: AGI Assessment Through Prompt and Metric Transformer

• Authors: Benhao Huang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19101
• Pdf URL: https://arxiv.org/pdf/2403.19101
• Copy Paste: [[2403.19101]] AAPMT: AGI Assessment Through Prompt and Metric Transformer(https://arxiv.org/abs/2403.19101)
• Keywords: transformer
• Abstract: The emergence of text-to-image models marks a significant milestone in the evolution of AI-generated images (AGIs), expanding their use in diverse domains like design, entertainment, and more. Despite these breakthroughs, the quality of AGIs often remains suboptimal, highlighting the need for effective evaluation methods. These methods are crucial for assessing the quality of images relative to their textual descriptions, and they must accurately mirror human perception. Substantial progress has been achieved in this domain, with innovative techniques such as BLIP and DBCNN contributing significantly. However, recent studies, including AGIQA-3K, reveal a notable discrepancy between current methods and state-of-the-art (SOTA) standards. This gap emphasizes the necessity for a more sophisticated and precise evaluation metric. In response, our objective is to develop a model that could give ratings for metrics, which focuses on parameters like perceptual quality, authenticity, and the correspondence between text and image, that more closely aligns with human perception. In our paper, we introduce a range of effective methods, including prompt designs and the Metric Transformer. The Metric Transformer is a novel structure inspired by the complex interrelationships among various AGI quality metrics. The code is available at https://github.com/huskydoge/CS3324-Digital-Image-Processing/tree/main/Assignment1

Title: Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation

• Authors: Yutong He, Alexander Robey, Naoki Murata, Yiding Jiang, Joshua Williams, George J. Pappas, Hamed Hassani, Yuki Mitsufuji, Ruslan Salakhutdinov, J. Zico Kolter
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19103
• Pdf URL: https://arxiv.org/pdf/2403.19103
• Copy Paste: [[2403.19103]] Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation(https://arxiv.org/abs/2403.19103)
• Keywords: diffusion, generative, large language model
• Abstract: Prompt engineering is effective for controlling the output of text-to-image (T2I) generative models, but it is also laborious due to the need for manually crafted prompts. This challenge has spurred the development of algorithms for automated prompt generation. However, these methods often struggle with transferability across T2I models, require white-box access to the underlying model, and produce non-intuitive prompts. In this work, we introduce PRISM, an algorithm that automatically identifies human-interpretable and transferable prompts that can effectively generate desired concepts given only black-box access to T2I models. Inspired by large language model (LLM) jailbreaking, PRISM leverages the in-context learning ability of LLMs to iteratively refine the candidate prompts distribution for given reference images. Our experiments demonstrate the versatility and effectiveness of PRISM in generating accurate prompts for objects, styles and images across multiple T2I models, including Stable Diffusion, DALL-E, and Midjourney.

Title: Synthetic Medical Imaging Generation with Generative Adversarial Networks For Plain Radiographs

• Authors: John R. McNulty, Lee Kho, Alexandria L. Case, Charlie Fornaca, Drew Johnston, David Slater, Joshua M. Abzug, Sybil A. Russell
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19107
• Pdf URL: https://arxiv.org/pdf/2403.19107
• Copy Paste: [[2403.19107]] Synthetic Medical Imaging Generation with Generative Adversarial Networks For Plain Radiographs(https://arxiv.org/abs/2403.19107)
• Keywords: privacy, generative
• Abstract: In medical imaging, access to data is commonly limited due to patient privacy restrictions and the issue that it can be difficult to acquire enough data in the case of rare diseases.[1] The purpose of this investigation was to develop a reusable open-source synthetic image generation pipeline, the GAN Image Synthesis Tool (GIST), that is easy to use as well as easy to deploy. The pipeline helps to improve and standardize AI algorithms in the digital health space by generating high quality synthetic image data that is not linked to specific patients. Its image generation capabilities include the ability to generate imaging of pathologies or injuries with low incidence rates. This improvement of digital health AI algorithms could improve diagnostic accuracy, aid in patient care, decrease medicolegal claims, and ultimately decrease the overall cost of healthcare. The pipeline builds on existing Generative Adversarial Networks (GANs) algorithms, and preprocessing and evaluation steps were included for completeness. For this work, we focused on ensuring the pipeline supports radiography, with a focus on synthetic knee and elbow x-ray images. In designing the pipeline, we evaluated the performance of current GAN architectures, studying the performance on available x-ray data. We show that the pipeline is capable of generating high quality and clinically relevant images based on a lay person's evaluation and the Fr\'echet Inception Distance (FID) metric.

Title: Patch Spatio-Temporal Relation Prediction for Video Anomaly Detection

• Authors: Hao Shen, Lu Shi, Wanru Xu, Yigang Cen, Linna Zhang, Gaoyun An
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19111
• Pdf URL: https://arxiv.org/pdf/2403.19111
• Copy Paste: [[2403.19111]] Patch Spatio-Temporal Relation Prediction for Video Anomaly Detection(https://arxiv.org/abs/2403.19111)
• Keywords: transformer
• Abstract: Video Anomaly Detection (VAD), aiming to identify abnormalities within a specific context and timeframe, is crucial for intelligent Video Surveillance Systems. While recent deep learning-based VAD models have shown promising results by generating high-resolution frames, they often lack competence in preserving detailed spatial and temporal coherence in video frames. To tackle this issue, we propose a self-supervised learning approach for VAD through an inter-patch relationship prediction task. Specifically, we introduce a two-branch vision transformer network designed to capture deep visual features of video frames, addressing spatial and temporal dimensions responsible for modeling appearance and motion patterns, respectively. The inter-patch relationship in each dimension is decoupled into inter-patch similarity and the order information of each patch. To mitigate memory consumption, we convert the order information prediction task into a multi-label learning problem, and the inter-patch similarity prediction task into a distance matrix regression problem. Comprehensive experiments demonstrate the effectiveness of our method, surpassing pixel-generation-based methods by a significant margin across three public benchmarks. Additionally, our approach outperforms other self-supervised learning-based methods.

Title: Uncover the Premeditated Attacks: Detecting Exploitable Reentrancy Vulnerabilities by Identifying Attacker Contracts

• Authors: Shuo Yang, Jiachi Chen, Mingyuan Huang, Zibin Zheng, Yuan Huang
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.19112
• Pdf URL: https://arxiv.org/pdf/2403.19112
• Copy Paste: [[2403.19112]] Uncover the Premeditated Attacks: Detecting Exploitable Reentrancy Vulnerabilities by Identifying Attacker Contracts(https://arxiv.org/abs/2403.19112)
• Keywords: attack
• Abstract: Reentrancy, a notorious vulnerability in smart contracts, has led to millions of dollars in financial loss. However, current smart contract vulnerability detection tools suffer from a high false positive rate in identifying contracts with reentrancy vulnerabilities. Moreover, only a small portion of the detected reentrant contracts can actually be exploited by hackers, making these tools less effective in securing the Ethereum ecosystem in practice. In this paper, we propose BlockWatchdog, a tool that focuses on detecting reentrancy vulnerabilities by identifying attacker contracts. These attacker contracts are deployed by hackers to exploit vulnerable contracts automatically. By focusing on attacker contracts, BlockWatchdog effectively detects truly exploitable reentrancy vulnerabilities by identifying reentrant call flow. Additionally, BlockWatchdog is capable of detecting new types of reentrancy vulnerabilities caused by poor designs when using ERC tokens or user-defined interfaces, which cannot be detected by current rule-based tools. We implement BlockWatchdog using cross-contract static dataflow techniques based on attack logic obtained from an empirical study that analyzes attacker contracts from 281 attack incidents. BlockWatchdog is evaluated on 421,889 Ethereum contract bytecodes and identifies 113 attacker contracts that target 159 victim contracts, leading to the theft of Ether and tokens valued at approximately 908.6 million USD. Notably, only 18 of the identified 159 victim contracts can be reported by current reentrancy detection tools.

Title: FACTOID: FACtual enTailment fOr hallucInation Detection

• Authors: Vipula Rawte, S.M Towhidul Islam Tonmoy, Krishnav Rajbangshi, Shravani Nag, Aman Chadha, Amit P. Sheth, Amitava Das
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19113
• Pdf URL: https://arxiv.org/pdf/2403.19113
• Copy Paste: [[2403.19113]] FACTOID: FACtual enTailment fOr hallucInation Detection(https://arxiv.org/abs/2403.19113)
• Keywords: large language model
• Abstract: The widespread adoption of Large Language Models (LLMs) has facilitated numerous benefits. However, hallucination is a significant concern. In response, Retrieval Augmented Generation (RAG) has emerged as a highly promising paradigm to improve LLM outputs by grounding them in factual information. RAG relies on textual entailment (TE) or similar methods to check if the text produced by LLMs is supported or contradicted, compared to retrieved documents. This paper argues that conventional TE methods are inadequate for spotting hallucinations in content generated by LLMs. For instance, consider a prompt about the 'USA's stance on the Ukraine war''. The AI-generated text states, ...U.S. President Barack Obama says the U.S. will not put troops in Ukraine...'' However, during the war the U.S. president is Joe Biden which contradicts factual reality. Moreover, current TE systems are unable to accurately annotate the given text and identify the exact portion that is contradicted. To address this, we introduces a new type of TE called ``Factual Entailment (FE).'', aims to detect factual inaccuracies in content generated by LLMs while also highlighting the specific text segment that contradicts reality. We present FACTOID (FACTual enTAILment for hallucInation Detection), a benchmark dataset for FE. We propose a multi-task learning (MTL) framework for FE, incorporating state-of-the-art (SoTA) long text embeddings such as e5-mistral-7b-instruct, along with GPT-3, SpanBERT, and RoFormer. The proposed MTL architecture for FE achieves an avg. 40\% improvement in accuracy on the FACTOID benchmark compared to SoTA TE methods. As FE automatically detects hallucinations, we assessed 15 modern LLMs and ranked them using our proposed Auto Hallucination Vulnerability Index (HVI_auto). This index quantifies and offers a comparative scale to evaluate and rank LLMs according to their hallucinations.

Title: MFORT-QA: Multi-hop Few-shot Open Rich Table Question Answering

• Authors: Che Guan, Mengyu Huang, Peng Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19116
• Pdf URL: https://arxiv.org/pdf/2403.19116
• Copy Paste: [[2403.19116]] MFORT-QA: Multi-hop Few-shot Open Rich Table Question Answering(https://arxiv.org/abs/2403.19116)
• Keywords: large language model
• Abstract: In today's fast-paced industry, professionals face the challenge of summarizing a large number of documents and extracting vital information from them on a daily basis. These metrics are frequently hidden away in tables and/or their nested hyperlinks. To address this challenge, the approach of Table Question Answering (QA) has been developed to extract the relevant information. However, traditional Table QA training tasks that provide a table and an answer(s) from a gold cell coordinate(s) for a question may not always ensure extracting the accurate answer(s). Recent advancements in Large Language Models (LLMs) have opened up new possibilities for extracting information from tabular data using prompts. In this paper, we introduce the Multi-hop Few-shot Open Rich Table QA (MFORT-QA) approach, which consists of two major steps. The first step involves Few-Shot Learning (FSL), where relevant tables and associated contexts of hyperlinks are retrieved based on a given question. The retrieved content is then used to construct few-shot prompts as inputs to an LLM, such as ChatGPT. To tackle the challenge of answering complex questions, the second step leverages Chain-of-thought (CoT) prompting to decompose the complex question into a sequential chain of questions and reasoning thoughts in a multi-hop manner. Retrieval-Augmented Generation (RAG) enhances this process by retrieving relevant tables and contexts of hyperlinks that are relevant to the resulting reasoning thoughts and questions. These additional contexts are then used to supplement the prompt used in the first step, resulting in more accurate answers from an LLM. Empirical results from OTT-QA demonstrate that our abstractive QA approach significantly improves the accuracy of extractive Table QA methods.

Title: Code Comparison Tuning for Code Large Language Models

• Authors: Yufan Jiang, Qiaozhi He, Xiaomin Zhuang, Zhihua Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19121
• Pdf URL: https://arxiv.org/pdf/2403.19121
• Copy Paste: [[2403.19121]] Code Comparison Tuning for Code Large Language Models(https://arxiv.org/abs/2403.19121)
• Keywords: large language model
• Abstract: We present Code Comparison Tuning (CCT), a simple and effective tuning method for code large language models (Code LLMs) to better handle subtle code errors. Specifically, we integrate the concept of comparison into instruction tuning, both at the token and sequence levels, enabling the model to discern even the slightest deviations in code. To compare the original code with an erroneous version containing manually added code errors, we use token-level preference loss for detailed token-level comparisons. Additionally, we combine code segments to create a new instruction tuning sample for sequence-level comparisons, enhancing the model's bug-fixing capability. Experimental results on the HumanEvalFix benchmark show that CCT surpasses instruction tuning in pass@1 scores by up to 4 points across diverse code LLMs, and extensive analysis demonstrates the effectiveness of our method.

Title: OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition

• Authors: Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, Zhibo Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19128
• Pdf URL: https://arxiv.org/pdf/2403.19128
• Copy Paste: [[2403.19128]] OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition(https://arxiv.org/abs/2403.19128)
• Keywords: extraction, generative, large language model
• Abstract: Recently, visually-situated text parsing (VsTP) has experienced notable advancements, driven by the increasing demand for automated document understanding and the emergence of Generative Large Language Models (LLMs) capable of processing document-based questions. Various methods have been proposed to address the challenging problem of VsTP. However, due to the diversified targets and heterogeneous schemas, previous works usually design task-specific architectures and objectives for individual tasks, which inadvertently leads to modal isolation and complex workflow. In this paper, we propose a unified paradigm for parsing visually-situated text across diverse scenarios. Specifically, we devise a universal model, called OmniParser, which can simultaneously handle three typical visually-situated text parsing tasks: text spotting, key information extraction, and table recognition. In OmniParser, all tasks share the unified encoder-decoder architecture, the unified objective: point-conditioned text generation, and the unified input & output representation: prompt & structured sequences. Extensive experiments demonstrate that the proposed OmniParser achieves state-of-the-art (SOTA) or highly competitive performances on 7 datasets for the three visually-situated text parsing tasks, despite its unified, concise design. The code is available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery.

Title: Compressing Large Language Models by Streamlining the Unimportant Layer

• Authors: Xiaodong Chen, Yuxuan Hu, Jing Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19135
• Pdf URL: https://arxiv.org/pdf/2403.19135
• Copy Paste: [[2403.19135]] Compressing Large Language Models by Streamlining the Unimportant Layer(https://arxiv.org/abs/2403.19135)
• Keywords: transformer, large language model
• Abstract: Large language models (LLM) have been extensively applied in various natural language tasks and domains, but their applicability is constrained by the large number of parameters of the models. Consequently, there is an increasing emphasis on compact models that exhibit high performance. In this study, we observe that different layers in LLM have varying degrees of perturbation on the hidden states, which allows us to identify less important layers. Based on this phenomenon, we propose LLM-Streamline, which consists of two parts: layer pruning, where we remove a set of consecutive layers with the lowest importance in the model according to the target sparsity; and layer replacement, where we train a lightweight model to substitute the pruned layers, thereby mitigating the performance degradation caused by pruning. In our experiments, we utilize structures such as a multi-layer perceptron (MLP) and a transformer layer as lightweight models and ultimately demonstrate that a single MLP can effectively fit the pruned layers. Comprehensive experiments show that our proposed method, LLM-Streamline, outperforms previous state-of-the-art (SOTA) model pruning methods.

Title: QNCD: Quantization Noise Correction for Diffusion Models

• Authors: Huanpeng Chu, Wei Wu, Chengjie Zang, Kun Yuan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19140
• Pdf URL: https://arxiv.org/pdf/2403.19140
• Copy Paste: [[2403.19140]] QNCD: Quantization Noise Correction for Diffusion Models(https://arxiv.org/abs/2403.19140)
• Keywords: diffusion
• Abstract: Diffusion models have revolutionized image synthesis, setting new benchmarks in quality and creativity. However, their widespread adoption is hindered by the intensive computation required during the iterative denoising process. Post-training quantization (PTQ) presents a solution to accelerate sampling, aibeit at the expense of sample quality, extremely in low-bit settings. Addressing this, our study introduces a unified Quantization Noise Correction Scheme (QNCD), aimed at minishing quantization noise throughout the sampling process. We identify two primary quantization challenges: intra and inter quantization noise. Intra quantization noise, mainly exacerbated by embeddings in the resblock module, extends activation quantization ranges, increasing disturbances in each single denosing step. Besides, inter quantization noise stems from cumulative quantization deviations across the entire denoising process, altering data distributions step-by-step. QNCD combats these through embedding-derived feature smoothing for eliminating intra quantization noise and an effective runtime noise estimatiation module for dynamicly filtering inter quantization noise. Extensive experiments demonstrate that our method outperforms previous quantization methods for diffusion models, achieving lossless results in W4A8 and W8A8 quantization settings on ImageNet (LDM-4). Code is available at: https://github.com/huanpengchu/QNCD

Title: Tiny Graph Neural Networks for Radio Resource Management

• Authors: Ahmad Ghasemi, Hossein Pishro-Nik
• Subjects: cs.LG, cs.NI, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.19143
• Pdf URL: https://arxiv.org/pdf/2403.19143
• Copy Paste: [[2403.19143]] Tiny Graph Neural Networks for Radio Resource Management(https://arxiv.org/abs/2403.19143)
• Keywords: robust
• Abstract: The surge in demand for efficient radio resource management has necessitated the development of sophisticated yet compact neural network architectures. In this paper, we introduce a novel approach to Graph Neural Networks (GNNs) tailored for radio resource management by presenting a new architecture: the Low Rank Message Passing Graph Neural Network (LR-MPGNN). The cornerstone of LR-MPGNN is the implementation of a low-rank approximation technique that substitutes the conventional linear layers with their low-rank counterparts. This innovative design significantly reduces the model size and the number of parameters. We evaluate the performance of the proposed LR-MPGNN model based on several key metrics: model size, number of parameters, weighted sum rate of the communication system, and the distribution of eigenvalues of weight matrices. Our extensive evaluations demonstrate that the LR-MPGNN model achieves a sixtyfold decrease in model size, and the number of model parameters can be reduced by up to 98%. Performance-wise, the LR-MPGNN demonstrates robustness with a marginal 2% reduction in the best-case scenario in the normalized weighted sum rate compared to the original MPGNN model. Additionally, the distribution of eigenvalues of the weight matrices in the LR-MPGNN model is more uniform and spans a wider range, suggesting a strategic redistribution of weights.

Title: MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation

• Authors: Seyeon Kim, Siyoon Jin, Jihye Park, Kihong Kim, Jiyoung Kim, Jisu Nam, Seungryong Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19144
• Pdf URL: https://arxiv.org/pdf/2403.19144
• Copy Paste: [[2403.19144]] MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation(https://arxiv.org/abs/2403.19144)
• Keywords: diffusion
• Abstract: Conventional GAN-based models for talking head generation often suffer from limited quality and unstable training. Recent approaches based on diffusion models aimed to address these limitations and improve fidelity. However, they still face challenges, including extensive sampling times and difficulties in maintaining temporal consistency due to the high stochasticity of diffusion models. To overcome these challenges, we propose a novel motion-disentangled diffusion model for high-quality talking head generation, dubbed MoDiTalker. We introduce the two modules: audio-to-motion (AToM), designed to generate a synchronized lip motion from audio, and motion-to-video (MToV), designed to produce high-quality head video following the generated motion. AToM excels in capturing subtle lip movements by leveraging an audio attention mechanism. In addition, MToV enhances temporal consistency by leveraging an efficient tri-plane representation. Our experiments conducted on standard benchmarks demonstrate that our model achieves superior performance compared to existing models. We also provide comprehensive ablation studies and user study results.

Title: Towards Understanding Dual BN In Hybrid Adversarial Training

• Authors: Chenshuang Zhang, Chaoning Zhang, Kang Zhang, Axi Niu, Junmo Kim, In So Kweon
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19150
• Pdf URL: https://arxiv.org/pdf/2403.19150
• Copy Paste: [[2403.19150]] Towards Understanding Dual BN In Hybrid Adversarial Training(https://arxiv.org/abs/2403.19150)
• Keywords: robust
• Abstract: There is a growing concern about applying batch normalization (BN) in adversarial training (AT), especially when the model is trained on both adversarial samples and clean samples (termed Hybrid-AT). With the assumption that adversarial and clean samples are from two different domains, a common practice in prior works is to adopt Dual BN, where BN and BN are used for adversarial and clean branches, respectively. A popular belief for motivating Dual BN is that estimating normalization statistics of this mixture distribution is challenging and thus disentangling it for normalization achieves stronger robustness. In contrast to this belief, we reveal that disentangling statistics plays a less role than disentangling affine parameters in model training. This finding aligns with prior work (Rebuffi et al., 2023), and we build upon their research for further investigations. We demonstrate that the domain gap between adversarial and clean samples is not very large, which is counter-intuitive considering the significant influence of adversarial perturbation on the model accuracy. We further propose a two-task hypothesis which serves as the empirical foundation and a unified framework for Hybrid-AT improvement. We also investigate Dual BN in test-time and reveal that affine parameters characterize the robustness during inference. Overall, our work sheds new light on understanding the mechanism of Dual BN in Hybrid-AT and its underlying justification.

Title: Disentangling Length from Quality in Direct Preference Optimization

• Authors: Ryan Park, Rafael Rafailov, Stefano Ermon, Chelsea Finn
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19159
• Pdf URL: https://arxiv.org/pdf/2403.19159
• Copy Paste: [[2403.19159]] Disentangling Length from Quality in Direct Preference Optimization(https://arxiv.org/abs/2403.19159)
• Keywords: large language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) has been a crucial component in the recent success of Large Language Models. However, RLHF is know to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is less helpful and objective. A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO). Unlike classical RLHF, DPO does not train a separate reward model or use reinforcement learning directly, so previous approaches developed to control verbosity cannot be directly applied to this setting. Our work makes several contributions. For the first time, we study the length problem in the DPO setting, showing significant exploitation in DPO and linking it to out-of-distribution bootstrapping. We then develop a principled but simple regularization strategy that prevents length exploitation, while still maintaining improvements in model quality. We demonstrate these effects across datasets on summarization and dialogue, where we achieve up to 20\% improvement in win rates when controlling for length, despite the GPT4 judge's well-known verbosity bias.

Title: RecDiffusion: Rectangling for Image Stitching with Diffusion Models

• Authors: Tianhao Zhou, Haipeng Li, Ziyi Wang, Ao Luo, Chen-Lin Zhang, Jiajun Li, Bing Zeng, Shuaicheng Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19164
• Pdf URL: https://arxiv.org/pdf/2403.19164
• Copy Paste: [[2403.19164]] RecDiffusion: Rectangling for Image Stitching with Diffusion Models(https://arxiv.org/abs/2403.19164)
• Keywords: diffusion
• Abstract: Image stitching from different captures often results in non-rectangular boundaries, which is often considered unappealing. To solve non-rectangular boundaries, current solutions involve cropping, which discards image content, inpainting, which can introduce unrelated content, or warping, which can distort non-linear features and introduce artifacts. To overcome these issues, we introduce a novel diffusion-based learning framework, \textbf{RecDiffusion}, for image stitching rectangling. This framework combines Motion Diffusion Models (MDM) to generate motion fields, effectively transitioning from the stitched image's irregular borders to a geometrically corrected intermediary. Followed by Content Diffusion Models (CDM) for image detail refinement. Notably, our sampling process utilizes a weighted map to identify regions needing correction during each iteration of CDM. Our RecDiffusion ensures geometric accuracy and overall visual appeal, surpassing all previous methods in both quantitative and qualitative measures when evaluated on public benchmarks. Code is released at https://github.com/lhaippp/RecDiffusion.

Title: Evaluating Fair Feature Selection in Machine Learning for Healthcare

• Authors: Md Rahat Shahriar Zawad, Peter Washington
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.19165
• Pdf URL: https://arxiv.org/pdf/2403.19165
• Copy Paste: [[2403.19165]] Evaluating Fair Feature Selection in Machine Learning for Healthcare(https://arxiv.org/abs/2403.19165)
• Keywords: fair
• Abstract: With the universal adoption of machine learning in healthcare, the potential for the automation of societal biases to further exacerbate health disparities poses a significant risk. We explore algorithmic fairness from the perspective of feature selection. Traditional feature selection methods identify features for better decision making by removing resource-intensive, correlated, or non-relevant features but overlook how these factors may differ across subgroups. To counter these issues, we evaluate a fair feature selection method that considers equal importance to all demographic groups. We jointly considered a fairness metric and an error metric within the feature selection process to ensure a balance between minimizing both bias and global classification error. We tested our approach on three publicly available healthcare datasets. On all three datasets, we observed improvements in fairness metrics coupled with a minimal degradation of balanced accuracy. Our approach addresses both distributive and procedural fairness within the fair machine learning context.

Title: Mitigating Misleading Chain-of-Thought Reasoning with Selective Filtering

• Authors: Yexin Wu, Zhuosheng Zhang, Hai Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19167
• Pdf URL: https://arxiv.org/pdf/2403.19167
• Copy Paste: [[2403.19167]] Mitigating Misleading Chain-of-Thought Reasoning with Selective Filtering(https://arxiv.org/abs/2403.19167)
• Keywords: large language model
• Abstract: Large language models have manifested remarkable capabilities by leveraging chain-of-thought (CoT) reasoning techniques to solve intricate questions through step-by-step reasoning chains. Despite its success, the efficacy of such reasoning is inherently contingent upon the quality of CoT. However, flawless CoT reasoning cannot be guaranteed due to the presence of indecomposable questions and the potential for erroneous reasoning chains, particularly in the case of small-scale language models. To tackle this challenge, we propose a novel approach called the selective filtering reasoner (SelF-Reasoner) that assesses the entailment relationship between the question and the candidate reasoning chain. Then, we proceed with CoT reasoning when the reasoning chain demonstrates confidence; otherwise, we opt to predict the answer directly. SelF-Reasoner improves the fine-tuned T5 baseline consistently over the ScienceQA, ECQA, and LastLetter tasks. Code is available at \texttt{https://github.com/LibroWu/SelF-Reasoner}.

Title: Rethinking Information Loss in Medical Image Segmentation with Various-sized Targets

• Authors: Tianyi Liu, Zhaorui Tan, Kaizhu Huang, Haochuan Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19177
• Pdf URL: https://arxiv.org/pdf/2403.19177
• Copy Paste: [[2403.19177]] Rethinking Information Loss in Medical Image Segmentation with Various-sized Targets(https://arxiv.org/abs/2403.19177)
• Keywords: segmentation
• Abstract: Medical image segmentation presents the challenge of segmenting various-size targets, demanding the model to effectively capture both local and global information. Despite recent efforts using CNNs and ViTs to predict annotations of different scales, these approaches often struggle to effectively balance the detection of targets across varying sizes. Simply utilizing local information from CNNs and global relationships from ViTs without considering potential significant divergence in latent feature distributions may result in substantial information loss. To address this issue, in this paper, we will introduce a novel Stagger Network (SNet) and argues that a well-designed fusion structure can mitigate the divergence in latent feature distributions between CNNs and ViTs, thereby reducing information loss. Specifically, to emphasize both global dependencies and local focus, we design a Parallel Module to bridge the semantic gap. Meanwhile, we propose the Stagger Module, trying to fuse the selected features that are more semantically similar. An Information Recovery Module is further adopted to recover complementary information back to the network. As a key contribution, we theoretically analyze that the proposed parallel and stagger strategies would lead to less information loss, thus certifying the SNet's rationale. Experimental results clearly proved that the proposed SNet excels comparisons with recent SOTAs in segmenting on the Synapse dataset where targets are in various sizes. Besides, it also demonstrates superiority on the ACDC and the MoNuSeg datasets where targets are with more consistent dimensions.

Title: Enhancing Trust and Privacy in Distributed Networks: A Comprehensive Survey on Blockchain-based Federated Learning

• Authors: Ji Liu, Chunlu Chen, Yu Li, Lin Sun, Yulun Song, Jingbo Zhou, Bo Jing, Dejing Dou
• Subjects: cs.CR, cs.AI, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19178
• Pdf URL: https://arxiv.org/pdf/2403.19178
• Copy Paste: [[2403.19178]] Enhancing Trust and Privacy in Distributed Networks: A Comprehensive Survey on Blockchain-based Federated Learning(https://arxiv.org/abs/2403.19178)
• Keywords: secure, security, privacy, federate
• Abstract: While centralized servers pose a risk of being a single point of failure, decentralized approaches like blockchain offer a compelling solution by implementing a consensus mechanism among multiple entities. Merging distributed computing with cryptographic techniques, decentralized technologies introduce a novel computing paradigm. Blockchain ensures secure, transparent, and tamper-proof data management by validating and recording transactions via consensus across network nodes. Federated Learning (FL), as a distributed machine learning framework, enables participants to collaboratively train models while safeguarding data privacy by avoiding direct raw data exchange. Despite the growing interest in decentralized methods, their application in FL remains underexplored. This paper presents a thorough investigation into Blockchain-based FL (BCFL), spotlighting the synergy between blockchain's security features and FL's privacy-preserving model training capabilities. First, we present the taxonomy of BCFL from three aspects, including decentralized, separate networks, and reputation-based architectures. Then, we summarize the general architecture of BCFL systems, providing a comprehensive perspective on FL architectures informed by blockchain. Afterward, we analyze the application of BCFL in healthcare, IoT, and other privacy-sensitive areas. Finally, we identify future research directions of BCFL.

Title: Text Data-Centric Image Captioning with Interactive Prompts

• Authors: Yiyu Wang, Hao Luo, Jungang Xu, Yingfei Sun, Fan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19193
• Pdf URL: https://arxiv.org/pdf/2403.19193
• Copy Paste: [[2403.19193]] Text Data-Centric Image Captioning with Interactive Prompts(https://arxiv.org/abs/2403.19193)
• Keywords: generative
• Abstract: Supervised image captioning approaches have made great progress, but it is challenging to collect high-quality human-annotated image-text data. Recently, large-scale vision and language models (e.g., CLIP) and large-scale generative language models (e.g., GPT-2) have shown strong performances in various tasks, which also provide some new solutions for image captioning with web paired data, unpaired data or even text-only data. Among them, the mainstream solution is to project image embeddings into the text embedding space with the assistance of consistent representations between image-text pairs from the CLIP model. However, the current methods still face several challenges in adapting to the diversity of data configurations in a unified solution, accurately estimating image-text embedding bias, and correcting unsatisfactory prediction results in the inference stage. This paper proposes a new Text data-centric approach with Interactive Prompts for image Captioning, named TIPCap. 1) We consider four different settings which gradually reduce the dependence on paired data. 2) We construct a mapping module driven by multivariate Gaussian distribution to mitigate the modality gap, which is applicable to the above four different settings. 3) We propose a prompt interaction module that can incorporate optional prompt information before generating captions. Extensive experiments show that our TIPCap outperforms other weakly or unsupervised image captioning methods and achieves a new state-of-the-art performance on two widely used datasets, i.e., MS-COCO and Flickr30K.

Title: From Activation to Initialization: Scaling Insights for Optimizing Neural Fields

• Authors: Hemanth Saratchandran, Sameera Ramasinghe, Simon Lucey
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19205
• Pdf URL: https://arxiv.org/pdf/2403.19205
• Copy Paste: [[2403.19205]] From Activation to Initialization: Scaling Insights for Optimizing Neural Fields(https://arxiv.org/abs/2403.19205)
• Keywords: robust
• Abstract: In the realm of computer vision, Neural Fields have gained prominence as a contemporary tool harnessing neural networks for signal representation. Despite the remarkable progress in adapting these networks to solve a variety of problems, the field still lacks a comprehensive theoretical framework. This article aims to address this gap by delving into the intricate interplay between initialization and activation, providing a foundational basis for the robust optimization of Neural Fields. Our theoretical insights reveal a deep-seated connection among network initialization, architectural choices, and the optimization process, emphasizing the need for a holistic approach when designing cutting-edge Neural Fields.

Title: Dual-Personalizing Adapter for Federated Foundation Models

• Authors: Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, Michael Blumenstein
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19211
• Pdf URL: https://arxiv.org/pdf/2403.19211
• Copy Paste: [[2403.19211]] Dual-Personalizing Adapter for Federated Foundation Models(https://arxiv.org/abs/2403.19211)
• Keywords: privacy, federate, large language model
• Abstract: Recently, foundation models, particularly large language models (LLMs), have demonstrated an impressive ability to adapt to various tasks by fine-tuning large amounts of instruction data. Notably, federated foundation models emerge as a privacy preservation method to fine-tune models collaboratively under federated learning (FL) settings by leveraging many distributed datasets with non-IID data. To alleviate communication and computation overhead, parameter-efficient methods are introduced for efficiency, and some research adapted personalization methods to federated foundation models for better user preferences alignment. However, a critical gap in existing research is the neglect of test-time distribution shifts in real-world applications. Therefore, to bridge this gap, we propose a new setting, termed test-time personalization, which not only concentrates on the targeted local task but also extends to other tasks that exhibit test-time distribution shifts. To address challenges in this new setting, we explore a simple yet effective solution to learn a comprehensive foundation model. Specifically, a dual-personalizing adapter architecture (FedDPA) is proposed, comprising a global adapter and a local adapter for addressing test-time distribution shifts and personalization, respectively. Additionally, we introduce an instance-wise dynamic weighting mechanism to optimize the balance between the global and local adapters, enhancing overall performance. The effectiveness of the proposed method has been evaluated on benchmark datasets across different NLP tasks.

Title: Learning Multiple Representations with Inconsistency-Guided Detail Regularization for Mask-Guided Matting

• Authors: Weihao Jiang, Zhaozhi Xie, Yuxiang Lu, Longjie Qi, Jingyong Cai, Hiroyuki Uchiyama, Bin Chen, Yue Ding, Hongtao Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19213
• Pdf URL: https://arxiv.org/pdf/2403.19213
• Copy Paste: [[2403.19213]] Learning Multiple Representations with Inconsistency-Guided Detail Regularization for Mask-Guided Matting(https://arxiv.org/abs/2403.19213)
• Keywords: segmentation
• Abstract: Mask-guided matting networks have achieved significant improvements and have shown great potential in practical applications in recent years. However, simply learning matting representation from synthetic and lack-of-real-world-diversity matting data, these approaches tend to overfit low-level details in wrong regions, lack generalization to objects with complex structures and real-world scenes such as shadows, as well as suffer from interference of background lines or textures. To address these challenges, in this paper, we propose a novel auxiliary learning framework for mask-guided matting models, incorporating three auxiliary tasks: semantic segmentation, edge detection, and background line detection besides matting, to learn different and effective representations from different types of data and annotations. Our framework and model introduce the following key aspects: (1) to learn real-world adaptive semantic representation for objects with diverse and complex structures under real-world scenes, we introduce extra semantic segmentation and edge detection tasks on more diverse real-world data with segmentation annotations; (2) to avoid overfitting on low-level details, we propose a module to utilize the inconsistency between learned segmentation and matting representations to regularize detail refinement; (3) we propose a novel background line detection task into our auxiliary learning framework, to suppress interference of background lines or textures. In addition, we propose a high-quality matting benchmark, Plant-Mat, to evaluate matting methods on complex structures. Extensively quantitative and qualitative results show that our approach outperforms state-of-the-art mask-guided methods.

Title: Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality

• Authors: Sishuo Chen, Lei Li, Shuhuai Ren, Rundong Gao, Yuanxin Liu, Xiaohan Bi, Xu Sun, Lu Hou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19221
• Pdf URL: https://arxiv.org/pdf/2403.19221
• Copy Paste: [[2403.19221]] Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality(https://arxiv.org/abs/2403.19221)
• Keywords: robust
• Abstract: Video paragraph captioning (VPC) involves generating detailed narratives for long videos, utilizing supportive modalities such as speech and event boundaries. However, the existing models are constrained by the assumption of constant availability of a single auxiliary modality, which is impractical given the diversity and unpredictable nature of real-world scenarios. To this end, we propose a Missing-Resistant framework MR-VPC that effectively harnesses all available auxiliary inputs and maintains resilience even in the absence of certain modalities. Under this framework, we propose the Multimodal VPC (MVPC) architecture integrating video, speech, and event boundary inputs in a unified manner to process various auxiliary inputs. Moreover, to fortify the model against incomplete data, we introduce DropAM, a data augmentation strategy that randomly omits auxiliary inputs, paired with DistillAM, a regularization target that distills knowledge from teacher models trained on modality-complete data, enabling efficient learning in modality-deficient environments. Through exhaustive experimentation on YouCook2 and ActivityNet Captions, MR-VPC has proven to deliver superior performance on modality-complete and modality-missing test data. This work highlights the significance of developing resilient VPC models and paves the way for more adaptive, robust multimodal video understanding.

Title: Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment

• Authors: Angchi Xu, Wei-Shi Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19225
• Pdf URL: https://arxiv.org/pdf/2403.19225
• Copy Paste: [[2403.19225]] Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment(https://arxiv.org/abs/2403.19225)
• Keywords: segmentation
• Abstract: Weakly-supervised action segmentation is a task of learning to partition a long video into several action segments, where training videos are only accompanied by transcripts (ordered list of actions). Most of existing methods need to infer pseudo segmentation for training by serial alignment between all frames and the transcript, which is time-consuming and hard to be parallelized while training. In this work, we aim to escape from this inefficient alignment with massive but redundant frames, and instead to directly localize a few action transitions for pseudo segmentation generation, where a transition refers to the change from an action segment to its next adjacent one in the transcript. As the true transitions are submerged in noisy boundaries due to intra-segment visual variation, we propose a novel Action-Transition-Aware Boundary Alignment (ATBA) framework to efficiently and effectively filter out noisy boundaries and detect transitions. In addition, to boost the semantic learning in the case that noise is inevitably present in the pseudo segmentation, we also introduce video-level losses to utilize the trusted video-level supervision. Extensive experiments show the effectiveness of our approach on both performance and training speed.

Title: DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation

• Authors: Haonan Lin, Mengmeng Wang, Yan Chen, Wenbin An, Yuzhe Yao, Guang Dai, Qianying Wang, Yong Liu, Jingdong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19235
• Pdf URL: https://arxiv.org/pdf/2403.19235
• Copy Paste: [[2403.19235]] DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation(https://arxiv.org/abs/2403.19235)
• Keywords: diffusion
• Abstract: While large-scale pre-trained text-to-image models can synthesize diverse and high-quality human-centered images, novel challenges arise with a nuanced task of "identity fine editing": precisely modifying specific features of a subject while maintaining its inherent identity and context. Existing personalization methods either require time-consuming optimization or learning additional encoders, adept in "identity re-contextualization". However, they often struggle with detailed and sensitive tasks like human face editing. To address these challenges, we introduce DreamSalon, a noise-guided, staged-editing framework, uniquely focusing on detailed image manipulations and identity-context preservation. By discerning editing and boosting stages via the frequency and gradient of predicted noises, DreamSalon first performs detailed manipulations on specific features in the editing stage, guided by high-frequency information, and then employs stochastic denoising in the boosting stage to improve image quality. For more precise editing, DreamSalon semantically mixes source and target textual prompts, guided by differences in their embedding covariances, to direct the model's focus on specific manipulation areas. Our experiments demonstrate DreamSalon's ability to efficiently and faithfully edit fine details on human faces, outperforming existing methods both qualitatively and quantitatively.

Title: Taming Lookup Tables for Efficient Image Retouching

• Authors: Sidi Yang, Binxiao Huang, Mingdeng Cao, Yatai Ji, Hanzhong Guo, Ngai Wong, Yujiu Yang
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.19238
• Pdf URL: https://arxiv.org/pdf/2403.19238
• Copy Paste: [[2403.19238]] Taming Lookup Tables for Efficient Image Retouching(https://arxiv.org/abs/2403.19238)
• Keywords: robust
• Abstract: The widespread use of high-definition screens in edge devices, such as end-user cameras, smartphones, and televisions, is spurring a significant demand for image enhancement. Existing enhancement models often optimize for high performance while falling short of reducing hardware inference time and power consumption, especially on edge devices with constrained computing and storage resources. To this end, we propose Image Color Enhancement Lookup Table (ICELUT) that adopts LUTs for extremely efficient edge inference, without any convolutional neural network (CNN). During training, we leverage pointwise (1x1) convolution to extract color information, alongside a split fully connected layer to incorporate global information. Both components are then seamlessly converted into LUTs for hardware-agnostic deployment. ICELUT achieves near-state-of-the-art performance and remarkably low power consumption. We observe that the pointwise network structure exhibits robust scalability, upkeeping the performance even with a heavily downsampled 32x32 input image. These enable ICELUT, the first-ever purely LUT-based image enhancer, to reach an unprecedented speed of 0.4ms on GPU and 7ms on CPU, at least one order faster than any CNN solution. Codes are available at https://github.com/Stephen0808/ICELUT.

Title: RTracker: Recoverable Tracking via PN Tree Structured Memory

• Authors: Yuqing Huang, Xin Li, Zikun Zhou, Yaowei Wang, Zhenyu He, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19242
• Pdf URL: https://arxiv.org/pdf/2403.19242
• Copy Paste: [[2403.19242]] RTracker: Recoverable Tracking via PN Tree Structured Memory(https://arxiv.org/abs/2403.19242)
• Keywords: robust
• Abstract: Existing tracking methods mainly focus on learning better target representation or developing more robust prediction models to improve tracking performance. While tracking performance has significantly improved, the target loss issue occurs frequently due to tracking failures, complete occlusion, or out-of-view situations. However, considerably less attention is paid to the self-recovery issue of tracking methods, which is crucial for practical applications. To this end, we propose a recoverable tracking framework, RTracker, that uses a tree-structured memory to dynamically associate a tracker and a detector to enable self-recovery ability. Specifically, we propose a Positive-Negative Tree-structured memory to chronologically store and maintain positive and negative target samples. Upon the PN tree memory, we develop corresponding walking rules for determining the state of the target and define a set of control flows to unite the tracker and the detector in different tracking scenarios. Our core idea is to use the support samples of positive and negative target categories to establish a relative distance-based criterion for a reliable assessment of target loss. The favorable performance in comparison against the state-of-the-art methods on numerous challenging benchmarks demonstrates the effectiveness of the proposed algorithm.

Title: Sine Activated Low-Rank Matrices for Parameter Efficient Learning

• Authors: Yiping Ji, Hemanth Saratchandran, Cameron Gordon, Zeyu Zhang, Simon Lucey
• Subjects: cs.LG, cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2403.19243
• Pdf URL: https://arxiv.org/pdf/2403.19243
• Copy Paste: [[2403.19243]] Sine Activated Low-Rank Matrices for Parameter Efficient Learning(https://arxiv.org/abs/2403.19243)
• Keywords: transformer, large language model
• Abstract: Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learning. These techniques significantly lower the number of parameters, striking a balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a novel theoretical framework that integrates a sinusoidal function within the low-rank decomposition process. This approach not only preserves the benefits of the parameter efficiency characteristic of low-rank methods but also increases the decomposition's rank, thereby enhancing model accuracy. Our method proves to be an adaptable enhancement for existing low-rank models, as evidenced by its successful application in Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF), and 3D shape modeling. This demonstrates the wide-ranging potential and efficiency of our proposed technique.

Title: MPXGAT: An Attention based Deep Learning Model for Multiplex Graphs Embedding

• Authors: Marco Bongiovanni, Luca Gallo, Roberto Grasso, Alfredo Pulvirenti
• Subjects: cs.LG, cs.DM, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.19246
• Pdf URL: https://arxiv.org/pdf/2403.19246
• Copy Paste: [[2403.19246]] MPXGAT: An Attention based Deep Learning Model for Multiplex Graphs Embedding(https://arxiv.org/abs/2403.19246)
• Keywords: robust
• Abstract: Graph representation learning has rapidly emerged as a pivotal field of study. Despite its growing popularity, the majority of research has been confined to embedding single-layer graphs, which fall short in representing complex systems with multifaceted relationships. To bridge this gap, we introduce MPXGAT, an innovative attention-based deep learning model tailored to multiplex graph embedding. Leveraging the robustness of Graph Attention Networks (GATs), MPXGAT captures the structure of multiplex networks by harnessing both intra-layer and inter-layer connections. This exploitation facilitates accurate link prediction within and across the network's multiple layers. Our comprehensive experimental evaluation, conducted on various benchmark datasets, confirms that MPXGAT consistently outperforms state-of-the-art competing algorithms.

Title: Genos: General In-Network Unsupervised Intrusion Detection by Rule Extraction

• Authors: Ruoyu Li, Qing Li, Yu Zhang, Dan Zhao, Xi Xiao, Yong Jiang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.19248
• Pdf URL: https://arxiv.org/pdf/2403.19248
• Copy Paste: [[2403.19248]] Genos: General In-Network Unsupervised Intrusion Detection by Rule Extraction(https://arxiv.org/abs/2403.19248)
• Keywords: attack, extraction, interpretability
• Abstract: Anomaly-based network intrusion detection systems (A-NIDS) use unsupervised models to detect unforeseen attacks. However, existing A-NIDS solutions suffer from low throughput, lack of interpretability, and high maintenance costs. Recent in-network intelligence (INI) exploits programmable switches to offer line-rate deployment of NIDS. Nevertheless, current in-network NIDS are either model-specific or only apply to supervised models. In this paper, we propose Genos, a general in-network framework for unsupervised A-NIDS by rule extraction, which consists of a Model Compiler, a Model Interpreter, and a Model Debugger. Specifically, observing benign data are multimodal and usually located in multiple subspaces in the feature space, we utilize a divide-and-conquer approach for model-agnostic rule extraction. In the Model Compiler, we first propose a tree-based clustering algorithm to partition the feature space into subspaces, then design a decision boundary estimation mechanism to approximate the source model in each subspace. The Model Interpreter interprets predictions by important attributes to aid network operators in understanding the predictions. The Model Debugger conducts incremental updating to rectify errors by only fine-tuning rules on affected subspaces, thus reducing maintenance costs. We implement a prototype using physical hardware, and experiments demonstrate its superior performance of 100 Gbps throughput, great interpretability, and trivial updating overhead.

Title: Imperceptible Protection against Style Imitation from Diffusion Models

• Authors: Namhyuk Ahn, Wonhyuk Ahn, KiYoon Yoo, Daesik Kim, Seung-Hun Nam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19254
• Pdf URL: https://arxiv.org/pdf/2403.19254
• Copy Paste: [[2403.19254]] Imperceptible Protection against Style Imitation from Diffusion Models(https://arxiv.org/abs/2403.19254)
• Keywords: protect, diffusion
• Abstract: Recent progress in diffusion models has profoundly enhanced the fidelity of image generation. However, this has raised concerns about copyright infringements. While prior methods have introduced adversarial perturbations to prevent style imitation, most are accompanied by the degradation of artworks' visual quality. Recognizing the importance of maintaining this, we develop a visually improved protection method that preserves its protection capability. To this end, we create a perceptual map to identify areas most sensitive to human eyes. We then adjust the protection intensity guided by an instance-aware refinement. We also integrate a perceptual constraints bank to further improve the imperceptibility. Results show that our method substantially elevates the quality of the protected image without compromising on protection efficacy.

Title: NaijaHate: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data

• Authors: Manuel Tonneau, Pedro Vitor Quinta de Castro, Karim Lasri, Ibrahim Farouq, Lakshminarayanan Subramanian, Victor Orozco-Olvera, Samuel Fraiberger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19260
• Pdf URL: https://arxiv.org/pdf/2403.19260
• Copy Paste: [[2403.19260]] NaijaHate: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data(https://arxiv.org/abs/2403.19260)
• Keywords: protect, robust
• Abstract: To address the global issue of hateful content proliferating in online platforms, hate speech detection (HSD) models are typically developed on datasets collected in the United States, thereby failing to generalize to English dialects from the Majority World. Furthermore, HSD models are often evaluated on curated samples, raising concerns about overestimating model performance in real-world settings. In this work, we introduce NaijaHate, the first dataset annotated for HSD which contains a representative sample of Nigerian tweets. We demonstrate that HSD evaluated on biased datasets traditionally used in the literature largely overestimates real-world performance on representative data. We also propose NaijaXLM-T, a pretrained model tailored to the Nigerian Twitter context, and establish the key role played by domain-adaptive pretraining and finetuning in maximizing HSD performance. Finally, we show that in this context, a human-in-the-loop approach to content moderation where humans review 1% of Nigerian tweets flagged as hateful would enable to moderate 60% of all hateful content. Taken together, these results pave the way towards robust HSD systems and a better protection of social media users from hateful content in low-resource settings.

Title: sDPO: Don't Use Your Data All at Once

• Authors: Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, Chanjun Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19270
• Pdf URL: https://arxiv.org/pdf/2403.19270
• Copy Paste: [[2403.19270]] sDPO: Don't Use Your Data All at Once(https://arxiv.org/abs/2403.19270)
• Keywords: large language model
• Abstract: As development of large language models (LLM) progresses, aligning them with human preferences has become increasingly important. We propose stepwise DPO (sDPO), an extension of the recently popularized direct preference optimization (DPO) for alignment tuning. This approach involves dividing the available preference datasets and utilizing them in a stepwise manner, rather than employing it all at once. We demonstrate that this method facilitates the use of more precisely aligned reference models within the DPO training framework. Furthermore, sDPO trains the final model to be more performant, even outperforming other popular LLMs with more parameters.

Title: Fine-Tuning Language Models with Reward Learning on Policy

• Authors: Hao Lang, Fei Huang, Yongbin Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19279
• Pdf URL: https://arxiv.org/pdf/2403.19279
• Copy Paste: [[2403.19279]] Fine-Tuning Language Models with Reward Learning on Policy(https://arxiv.org/abs/2403.19279)
• Keywords: robust, large language model
• Abstract: Reinforcement learning from human feedback (RLHF) has emerged as an effective approach to aligning large language models (LLMs) to human preferences. RLHF contains three steps, i.e., human preference collecting, reward learning, and policy optimization, which are usually performed serially. Despite its popularity, however, (fixed) reward models may suffer from inaccurate off-distribution, since policy optimization continuously shifts LLMs' data distribution. Repeatedly collecting new preference data from the latest LLMs may alleviate this issue, which unfortunately makes the resulting system more complicated and difficult to optimize. In this paper, we propose reward learning on policy (RLP), an unsupervised framework that refines a reward model using policy samples to keep it on-distribution. Specifically, an unsupervised multi-view learning method is introduced to learn robust representations of policy samples. Meanwhile, a synthetic preference generation approach is developed to simulate high-quality preference data with policy outputs. Extensive experiments on three benchmark datasets show that RLP consistently outperforms the state-of-the-art. Our code is available at \url{https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/rlp}.

Title: Ungrammatical-syntax-based In-context Example Selection for Grammatical Error Correction

• Authors: Chenming Tang, Fanyi Qu, Yunfang Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19283
• Pdf URL: https://arxiv.org/pdf/2403.19283
• Copy Paste: [[2403.19283]] Ungrammatical-syntax-based In-context Example Selection for Grammatical Error Correction(https://arxiv.org/abs/2403.19283)
• Keywords: large language model
• Abstract: In the era of large language models (LLMs), in-context learning (ICL) stands out as an effective prompting strategy that explores LLMs' potency across various tasks. However, applying LLMs to grammatical error correction (GEC) is still a challenging task. In this paper, we propose a novel ungrammatical-syntax-based in-context example selection strategy for GEC. Specifically, we measure similarity of sentences based on their syntactic structures with diverse algorithms, and identify optimal ICL examples sharing the most similar ill-formed syntax to the test input. Additionally, we carry out a two-stage process to further improve the quality of selection results. On benchmark English GEC datasets, empirical results show that our proposed ungrammatical-syntax-based strategies outperform commonly-used word-matching or semantics-based methods with multiple LLMs. This indicates that for a syntax-oriented task like GEC, paying more attention to syntactic information can effectively boost LLMs' performance. Our code will be publicly available after the publication of this paper.

Title: Going Beyond Word Matching: Syntax Improves In-context Example Selection for Machine Translation

• Authors: Chenming Tang, Zhixiang Wang, Yunfang Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19285
• Pdf URL: https://arxiv.org/pdf/2403.19285
• Copy Paste: [[2403.19285]] Going Beyond Word Matching: Syntax Improves In-context Example Selection for Machine Translation(https://arxiv.org/abs/2403.19285)
• Keywords: large language model
• Abstract: In-context learning (ICL) is the trending prompting strategy in the era of large language models (LLMs), where a few examples are demonstrated to evoke LLMs' power for a given task. How to select informative examples remains an open issue. Previous works on in-context example selection for machine translation (MT) focus on superficial word-level features while ignoring deep syntax-level knowledge. In this paper, we propose a syntax-based in-context example selection method for MT, by computing the syntactic similarity between dependency trees using Polynomial Distance. In addition, we propose an ensemble strategy combining examples selected by both word-level and syntax-level criteria. Experimental results between English and 6 common languages indicate that syntax can effectively enhancing ICL for MT, obtaining the highest COMET scores on 11 out of 12 translation directions.

Title: FlowDepth: Decoupling Optical Flow for Self-Supervised Monocular Depth Estimation

• Authors: Yiyang Sun, Zhiyuan Xu, Xiaonian Wang, Jing Yao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19294
• Pdf URL: https://arxiv.org/pdf/2403.19294
• Copy Paste: [[2403.19294]] FlowDepth: Decoupling Optical Flow for Self-Supervised Monocular Depth Estimation(https://arxiv.org/abs/2403.19294)
• Keywords: fair
• Abstract: Self-supervised multi-frame methods have currently achieved promising results in depth estimation. However, these methods often suffer from mismatch problems due to the moving objects, which break the static assumption. Additionally, unfairness can occur when calculating photometric errors in high-freq or low-texture regions of the images. To address these issues, existing approaches use additional semantic priori black-box networks to separate moving objects and improve the model only at the loss level. Therefore, we propose FlowDepth, where a Dynamic Motion Flow Module (DMFM) decouples the optical flow by a mechanism-based approach and warps the dynamic regions thus solving the mismatch problem. For the unfairness of photometric errors caused by high-freq and low-texture regions, we use Depth-Cue-Aware Blur (DCABlur) and Cost-Volume sparsity loss respectively at the input and the loss level to solve the problem. Experimental results on the KITTI and Cityscapes datasets show that our method outperforms the state-of-the-art methods.

Title: Post Quantum Cryptography & its Comparison with Classical Cryptography

• Authors: Tanmay Tripathi, Abhinav Awasthi, Shaurya Pratap Singh, Atul Chaturvedi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.19299
• Pdf URL: https://arxiv.org/pdf/2403.19299
• Copy Paste: [[2403.19299]] Post Quantum Cryptography & its Comparison with Classical Cryptography(https://arxiv.org/abs/2403.19299)
• Keywords: secure, security
• Abstract: Cryptography plays a pivotal role in safeguarding sensitive information and facilitating secure communication. Classical cryptography relies on mathematical computations, whereas quantum cryptography operates on the principles of quantum mechanics, offering a new frontier in secure communication. Quantum cryptographic systems introduce novel dimensions to security, capable of detecting and thwarting eavesdropping attempts. By contrasting quantum cryptography with its classical counterpart, it becomes evident how quantum mechanics revolutionizes the landscape of secure communication.

Title: MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation

• Authors: Yu Li, Shenyu Zhang, Rui Wu, Xiutian Huang, Yongrui Chen, Wenhao Xu, Guilin Qi, Dehai Min
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19305
• Pdf URL: https://arxiv.org/pdf/2403.19305
• Copy Paste: [[2403.19305]] MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation(https://arxiv.org/abs/2403.19305)
• Keywords: generative, large language model
• Abstract: Recent advancements in generative Large Language Models(LLMs) have been remarkable, however, the quality of the text generated by these models often reveals persistent issues. Evaluating the quality of text generated by these models, especially in open-ended text, has consistently presented a significant challenge. Addressing this, recent work has explored the possibility of using LLMs as evaluators. While using a single LLM as an evaluation agent shows potential, it is filled with significant uncertainty and instability. To address these issues, we propose the MATEval: A "Multi-Agent Text Evaluation framework" where all agents are played by LLMs like GPT-4. The MATEval framework emulates human collaborative discussion methods, integrating multiple agents' interactions to evaluate open-ended text. Our framework incorporates self-reflection and Chain-of-Thought (CoT) strategies, along with feedback mechanisms, enhancing the depth and breadth of the evaluation process and guiding discussions towards consensus, while the framework generates comprehensive evaluation reports, including error localization, error types and scoring. Experimental results show that our framework outperforms existing open-ended text evaluation methods and achieves the highest correlation with human evaluation, which confirms the effectiveness and advancement of our framework in addressing the uncertainties and instabilities in evaluating LLMs-generated text. Furthermore, our framework significantly improves the efficiency of text evaluation and model iteration in industrial scenarios.

Title: TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios

• Authors: Xiaokang Zhang, Jing Zhang, Zeyao Ma, Yang Li, Bohan Zhang, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, Jifan Yu, Shu Zhao, Juanzi Li, Jie Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19318
• Pdf URL: https://arxiv.org/pdf/2403.19318
• Copy Paste: [[2403.19318]] TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios(https://arxiv.org/abs/2403.19318)
• Keywords: robust, large language model
• Abstract: We introduce TableLLM, a robust large language model (LLM) with 13 billion parameters, purpose-built for proficiently handling tabular data manipulation tasks, whether they are embedded within documents or spreadsheets, catering to real-world office scenarios. We propose a distant supervision method for training, which comprises a reasoning process extension strategy, aiding in training LLMs to understand reasoning patterns more effectively as well as a cross-way validation strategy, ensuring the quality of the automatically generated data. To evaluate the performance of TableLLM, we have crafted a benchmark tailored to address both document and spreadsheet formats as well as constructed a well-organized evaluation pipeline capable of handling both scenarios. Thorough evaluations underscore the advantages of TableLLM when compared to various existing general-purpose and tabular data-focused LLMs. We have publicly released the model checkpoint, source code, benchmarks, and a web application for user interaction.

Title: Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation

• Authors: Yujin Chen, Yinyu Nie, Benjamin Ummenhofer, Reiner Birkl, Michael Paulitsch, Matthias Müller, Matthias Nießner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19319
• Pdf URL: https://arxiv.org/pdf/2403.19319
• Copy Paste: [[2403.19319]] Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation(https://arxiv.org/abs/2403.19319)
• Keywords: extraction, generative
• Abstract: We present Mesh2NeRF, an approach to derive ground-truth radiance fields from textured meshes for 3D generation tasks. Many 3D generative approaches represent 3D scenes as radiance fields for training. Their ground-truth radiance fields are usually fitted from multi-view renderings from a large-scale synthetic 3D dataset, which often results in artifacts due to occlusions or under-fitting issues. In Mesh2NeRF, we propose an analytic solution to directly obtain ground-truth radiance fields from 3D meshes, characterizing the density field with an occupancy function featuring a defined surface thickness, and determining view-dependent color through a reflection function considering both the mesh and environment lighting. Mesh2NeRF extracts accurate radiance fields which provides direct supervision for training generative NeRFs and single scene representation. We validate the effectiveness of Mesh2NeRF across various tasks, achieving a noteworthy 3.12dB improvement in PSNR for view synthesis in single scene representation on the ABO dataset, a 0.69 PSNR enhancement in the single-view conditional generation of ShapeNet Cars, and notably improved mesh extraction from NeRF in the unconditional generation of Objaverse Mugs.

Title: Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models

• Authors: Jiaxing Chen, Yuxuan Liu, Dehu Li, Xiang An, Ziyong Feng, Yongle Zhao, Yin Xie
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19322
• Pdf URL: https://arxiv.org/pdf/2403.19322
• Copy Paste: [[2403.19322]] Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models(https://arxiv.org/abs/2403.19322)
• Keywords: large language model
• Abstract: The surge of Multimodal Large Language Models (MLLMs), given their prominent emergent capabilities in instruction following and reasoning, has greatly advanced the field of visual reasoning. However, constrained by their non-lossless image tokenization, most MLLMs fall short of comprehensively capturing details of text and objects, especially in high-resolution images. To address this, we propose P2G, a novel framework for plug-and-play grounding of reasoning in MLLMs. Specifically, P2G exploits the tool-usage potential of MLLMs to employ expert agents to achieve on-the-fly grounding to critical visual and textual objects of image, thus achieving deliberate reasoning via multimodal prompting. We further create P2GB, a benchmark aimed at assessing MLLMs' ability to understand inter-object relationships and text in challenging high-resolution images. Comprehensive experiments on visual reasoning tasks demonstrate the superiority of P2G. Noteworthy, P2G achieved comparable performance with GPT-4V on P2GB, with a 7B backbone. Our work highlights the potential of plug-and-play grounding of reasoning and opens up a promising alternative beyond model scaling.

Title: MedBN: Robust Test-Time Adaptation against Malicious Test Samples

• Authors: Hyejin Park, Jeongyeon Hwang, Sunung Mun, Sangdon Park, Jungseul Ok
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19326
• Pdf URL: https://arxiv.org/pdf/2403.19326
• Copy Paste: [[2403.19326]] MedBN: Robust Test-Time Adaptation against Malicious Test Samples(https://arxiv.org/abs/2403.19326)
• Keywords: security, attack, robust
• Abstract: Test-time adaptation (TTA) has emerged as a promising solution to address performance decay due to unforeseen distribution shifts between training and test data. While recent TTA methods excel in adapting to test data variations, such adaptability exposes a model to vulnerability against malicious examples, an aspect that has received limited attention. Previous studies have uncovered security vulnerabilities within TTA even when a small proportion of the test batch is maliciously manipulated. In response to the emerging threat, we propose median batch normalization (MedBN), leveraging the robustness of the median for statistics estimation within the batch normalization layer during test-time inference. Our method is algorithm-agnostic, thus allowing seamless integration with existing TTA frameworks. Our experimental results on benchmark datasets, including CIFAR10-C, CIFAR100-C and ImageNet-C, consistently demonstrate that MedBN outperforms existing approaches in maintaining robust performance across different attack scenarios, encompassing both instant and cumulative attacks. Through extensive experiments, we show that our approach sustains the performance even in the absence of attacks, achieving a practical balance between robustness and performance.

Title: Test-Time Domain Generalization for Face Anti-Spoofing

• Authors: Qianyu Zhou, Ke-Yue Zhang, Taiping Yao, Xuequan Lu, Shouhong Ding, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19334
• Pdf URL: https://arxiv.org/pdf/2403.19334
• Copy Paste: [[2403.19334]] Test-Time Domain Generalization for Face Anti-Spoofing(https://arxiv.org/abs/2403.19334)
• Keywords: attack
• Abstract: Face Anti-Spoofing (FAS) is pivotal in safeguarding facial recognition systems against presentation attacks. While domain generalization (DG) methods have been developed to enhance FAS performance, they predominantly focus on learning domain-invariant features during training, which may not guarantee generalizability to unseen data that differs largely from the source distributions. Our insight is that testing data can serve as a valuable resource to enhance the generalizability beyond mere evaluation for DG FAS. In this paper, we introduce a novel Test-Time Domain Generalization (TTDG) framework for FAS, which leverages the testing data to boost the model's generalizability. Our method, consisting of Test-Time Style Projection (TTSP) and Diverse Style Shifts Simulation (DSSS), effectively projects the unseen data to the seen domain space. In particular, we first introduce the innovative TTSP to project the styles of the arbitrarily unseen samples of the testing distribution to the known source space of the training distributions. We then design the efficient DSSS to synthesize diverse style shifts via learnable style bases with two specifically designed losses in a hyperspherical feature space. Our method eliminates the need for model updates at the test time and can be seamlessly integrated into not only the CNN but also ViT backbones. Comprehensive experiments on widely used cross-domain FAS benchmarks demonstrate our method's state-of-the-art performance and effectiveness.

Title: IVLMap: Instance-Aware Visual Language Grounding for Consumer Robot Navigation

• Authors: Jiacui Huang, Hongtao Zhang, Mingbo Zhao, Zhou Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19336
• Pdf URL: https://arxiv.org/pdf/2403.19336
• Copy Paste: [[2403.19336]] IVLMap: Instance-Aware Visual Language Grounding for Consumer Robot Navigation(https://arxiv.org/abs/2403.19336)
• Keywords: large language model
• Abstract: Vision-and-Language Navigation (VLN) is a challenging task that requires a robot to navigate in photo-realistic environments with human natural language promptings. Recent studies aim to handle this task by constructing the semantic spatial map representation of the environment, and then leveraging the strong ability of reasoning in large language models for generalizing code for guiding the robot navigation. However, these methods face limitations in instance-level and attribute-level navigation tasks as they cannot distinguish different instances of the same object. To address this challenge, we propose a new method, namely, Instance-aware Visual Language Map (IVLMap), to empower the robot with instance-level and attribute-level semantic mapping, where it is autonomously constructed by fusing the RGBD video data collected from the robot agent with special-designed natural language map indexing in the bird's-in-eye view. Such indexing is instance-level and attribute-level. In particular, when integrated with a large language model, IVLMap demonstrates the capability to i) transform natural language into navigation targets with instance and attribute information, enabling precise localization, and ii) accomplish zero-shot end-to-end navigation tasks based on natural language commands. Extensive navigation experiments are conducted. Simulation results illustrate that our method can achieve an average improvement of 14.4\% in navigation accuracy. Code and demo are released at https://ivlmap.github.io/.

Title: Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

• Authors: Hyunbyung Park, Sukyung Lee, Gyoungjin Gim, Yungi Kim, Dahyun Kim, Chanjun Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19340
• Pdf URL: https://arxiv.org/pdf/2403.19340
• Copy Paste: [[2403.19340]] Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models(https://arxiv.org/abs/2403.19340)
• Keywords: large language model
• Abstract: To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of custom processors with block-based interface in Dataverse allows users to readily and efficiently use Dataverse to build their own ETL pipeline. We hope that Dataverse will serve as a vital tool for LLM development and open source the entire library to welcome community contribution. Additionally, we provide a concise, two-minute video demonstration of our system, illustrating its capabilities and implementation.

Title: Large Language Models Are Unconscious of Unreasonability in Math Problems

• Authors: Jingyuan Ma, Damai Dai, Zhifang Sui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19346
• Pdf URL: https://arxiv.org/pdf/2403.19346
• Copy Paste: [[2403.19346]] Large Language Models Are Unconscious of Unreasonability in Math Problems(https://arxiv.org/abs/2403.19346)
• Keywords: large language model
• Abstract: Large language models (LLMs) demonstrate substantial capabilities in solving math problems. However, they tend to produce hallucinations when given questions containing unreasonable errors. In this paper, we study the behavior of LLMs when faced with unreasonable math problems and further explore their potential to address these problems. First, we construct the Unreasonable Math Problem (UMP) benchmark to examine the error detection ability of LLMs. Experiments show that LLMs are able to detect unreasonable errors, but still fail in generating non-hallucinatory content. In order to improve their ability of error detection and correction, we further design a strategic prompt template called Critical Calculation and Conclusion(CCC). With CCC, LLMs can better self-evaluate and detect unreasonable errors in math questions, making them more reliable and safe in practical application scenarios.

Title: Risk prediction of pathological gambling on social media

• Authors: Angelina Parfenova, Marianne Clausel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19358
• Pdf URL: https://arxiv.org/pdf/2403.19358
• Copy Paste: [[2403.19358]] Risk prediction of pathological gambling on social media(https://arxiv.org/abs/2403.19358)
• Keywords: interpretability
• Abstract: This paper addresses the problem of risk prediction on social media data, specifically focusing on the classification of Reddit users as having a pathological gambling disorder. To tackle this problem, this paper focuses on incorporating temporal and emotional features into the model. The preprocessing phase involves dealing with the time irregularity of posts by padding sequences. Two baseline architectures are used for preliminary evaluation: BERT classifier on concatenated posts per user and GRU with LSTM on sequential data. Experimental results demonstrate that the sequential models outperform the concatenation-based model. The results of the experiments conclude that the incorporation of a time decay layer (TD) and passing the emotion classification layer (EmoBERTa) through LSTM improves the performance significantly. Experiments concluded that the addition of a self-attention layer didn't significantly improve the performance of the model, however provided easily interpretable attention scores. The developed architecture with the inclusion of EmoBERTa and TD layers achieved a high F1 score, beating existing benchmarks on pathological gambling dataset. Future work may involve the early prediction of risk factors associated with pathological gambling disorder and testing models on other datasets. Overall, this research highlights the significance of the sequential processing of posts including temporal and emotional features to boost the predictive power, as well as adding an attention layer for interpretability.

Title: EthioMT: Parallel Corpus for Low-resource Ethiopian Languages

• Authors: Atnafu Lambebo Tonja, Olga Kolesnikova, Alexander Gelbukh, Jugal Kalita
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19365
• Pdf URL: https://arxiv.org/pdf/2403.19365
• Copy Paste: [[2403.19365]] EthioMT: Parallel Corpus for Low-resource Ethiopian Languages(https://arxiv.org/abs/2403.19365)
• Keywords: transformer
• Abstract: Recent research in natural language processing (NLP) has achieved impressive performance in tasks such as machine translation (MT), news classification, and question-answering in high-resource languages. However, the performance of MT leaves much to be desired for low-resource languages. This is due to the smaller size of available parallel corpora in these languages, if such corpora are available at all. NLP in Ethiopian languages suffers from the same issues due to the unavailability of publicly accessible datasets for NLP tasks, including MT. To help the research community and foster research for Ethiopian languages, we introduce EthioMT -- a new parallel corpus for 15 languages. We also create a new benchmark by collecting a dataset for better-researched languages in Ethiopia. We evaluate the newly collected corpus and the benchmark dataset for 23 Ethiopian languages using transformer and fine-tuning approaches.

Title: PointCloud-Text Matching: Benchmark Datasets and a Baseline

• Authors: Yanglin Feng, Yang Qin, Dezhong Peng, Hongyuan Zhu, Xi Peng, Peng Hu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19386
• Pdf URL: https://arxiv.org/pdf/2403.19386
• Copy Paste: [[2403.19386]] PointCloud-Text Matching: Benchmark Datasets and a Baseline(https://arxiv.org/abs/2403.19386)
• Keywords: robust
• Abstract: In this paper, we present and study a new instance-level retrieval task: PointCloud-Text Matching~(PTM), which aims to find the exact cross-modal instance that matches a given point-cloud query or text query. PTM could be applied to various scenarios, such as indoor/urban-canyon localization and scene retrieval. However, there exists no suitable and targeted dataset for PTM in practice. Therefore, we construct three new PTM benchmark datasets, namely 3D2T-SR, 3D2T-NR, and 3D2T-QA. We observe that the data is challenging and with noisy correspondence due to the sparsity, noise, or disorder of point clouds and the ambiguity, vagueness, or incompleteness of texts, which make existing cross-modal matching methods ineffective for PTM. To tackle these challenges, we propose a PTM baseline, named Robust PointCloud-Text Matching method (RoMa). RoMa consists of two modules: a Dual Attention Perception module (DAP) and a Robust Negative Contrastive Learning module (RNCL). Specifically, DAP leverages token-level and feature-level attention to adaptively focus on useful local and global features, and aggregate them into common representations, thereby reducing the adverse impact of noise and ambiguity. To handle noisy correspondence, RNCL divides negative pairs, which are much less error-prone than positive pairs, into clean and noisy subsets, and assigns them forward and reverse optimization directions respectively, thus enhancing robustness against noisy correspondence. We conduct extensive experiments on our benchmarks and demonstrate the superiority of our RoMa.

Title: Checkpoint Merging via Bayesian Optimization in LLM Pretraining

• Authors: Deyuan Liu, Zecheng Wang, Bingning Wang, Weipeng Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li, Dianbo Sui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19390
• Pdf URL: https://arxiv.org/pdf/2403.19390
• Copy Paste: [[2403.19390]] Checkpoint Merging via Bayesian Optimization in LLM Pretraining(https://arxiv.org/abs/2403.19390)
• Keywords: robust, large language model
• Abstract: The rapid proliferation of large language models (LLMs) such as GPT-4 and Gemini underscores the intense demand for resources during their training processes, posing significant challenges due to substantial computational and environmental costs. To alleviate this issue, we propose checkpoint merging in pretraining LLM. This method utilizes LLM checkpoints with shared training trajectories, and is rooted in an extensive search space exploration for the best merging weight via Bayesian optimization. Through various experiments, we demonstrate that: (1) Our proposed methodology exhibits the capacity to augment pretraining, presenting an opportunity akin to obtaining substantial benefits at minimal cost; (2) Our proposed methodology, despite requiring a given held-out dataset, still demonstrates robust generalization capabilities across diverse domains, a pivotal aspect in pretraining.

Title: Tabular Learning: Encoding for Entity and Context Embeddings

• Authors: Fredy Reusser
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2403.19405
• Pdf URL: https://arxiv.org/pdf/2403.19405
• Copy Paste: [[2403.19405]] Tabular Learning: Encoding for Entity and Context Embeddings(https://arxiv.org/abs/2403.19405)
• Keywords: transformer
• Abstract: Examining the effect of different encoding techniques on entity and context embeddings, the goal of this work is to challenge commonly used Ordinal encoding for tabular learning. Applying different preprocessing methods and network architectures over several datasets resulted in a benchmark on how the encoders influence the learning outcome of the networks. By keeping the test, validation and training data consistent, results have shown that ordinal encoding is not the most suited encoder for categorical data in terms of preprocessing the data and thereafter, classifying the target variable correctly. A better outcome was achieved, encoding the features based on string similarities by computing a similarity matrix as input for the network. This is the case for both, entity and context embeddings, where the transformer architecture showed improved performance for Ordinal and Similarity encoding with regard to multi-label classification tasks.

Title: Towards Temporally Consistent Referring Video Object Segmentation

• Authors: Bo Miao, Mohammed Bennamoun, Yongsheng Gao, Mubarak Shah, Ajmal Mian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19407
• Pdf URL: https://arxiv.org/pdf/2403.19407
• Copy Paste: [[2403.19407]] Towards Temporally Consistent Referring Video Object Segmentation(https://arxiv.org/abs/2403.19407)
• Keywords: robust, segmentation
• Abstract: Referring Video Object Segmentation (R-VOS) methods face challenges in maintaining consistent object segmentation due to temporal context variability and the presence of other visually similar objects. We propose an end-to-end R-VOS paradigm that explicitly models temporal instance consistency alongside the referring segmentation. Specifically, we introduce a novel hybrid memory that facilitates inter-frame collaboration for robust spatio-temporal matching and propagation. Features of frames with automatically generated high-quality reference masks are propagated to segment the remaining frames based on multi-granularity association to achieve temporally consistent R-VOS. Furthermore, we propose a new Mask Consistency Score (MCS) metric to evaluate the temporal consistency of video segmentation. Extensive experiments demonstrate that our approach enhances temporal consistency by a significant margin, leading to top-ranked performance on popular R-VOS benchmarks, i.e., Ref-YouTube-VOS (67.1%) and Ref-DAVIS17 (65.6%).

Title: BP4ER: Bootstrap Prompting for Explicit Reasoning in Medical Dialogue Generation

• Authors: Yuhong He, Yongqi Zhang, Shizhu He, Jun Wan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19414
• Pdf URL: https://arxiv.org/pdf/2403.19414
• Copy Paste: [[2403.19414]] BP4ER: Bootstrap Prompting for Explicit Reasoning in Medical Dialogue Generation(https://arxiv.org/abs/2403.19414)
• Keywords: large language model
• Abstract: Medical dialogue generation (MDG) has gained increasing attention due to its substantial practical value. Previous works typically employ a sequence-to-sequence framework to generate medical responses by modeling dialogue context as sequential text with annotated medical entities. While these methods have been successful in generating fluent responses, they fail to provide process explanations of reasoning and require extensive entity annotation. To address these limitations, we propose the method Bootstrap Prompting for Explicit Reasoning in MDG (BP4ER), which explicitly model MDG's multi-step reasoning process and iteratively enhance this reasoning process. We employ a least-to-most prompting strategy to guide a large language model (LLM) in explicit reasoning, breaking down MDG into simpler sub-questions. These sub-questions build on answers from previous ones. Additionally, we also introduce two distinct bootstrapping techniques for prompting, which autonomously correct errors and facilitate the LLM's explicit reasoning. This approach eliminates the need for entity annotation and increases the transparency of the MDG process by explicitly generating the intermediate reasoning chain. The experimental findings on the two public datasets indicate that BP4ER outperforms state-of-the-art methods in terms of both objective and subjective evaluation metrics.

Title: OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion

• Authors: Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, Cewu Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19417
• Pdf URL: https://arxiv.org/pdf/2403.19417
• Copy Paste: [[2403.19417]] OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion(https://arxiv.org/abs/2403.19417)
• Keywords: large language model
• Abstract: We present OAKINK2, a dataset of bimanual object manipulation tasks for complex daily activities. In pursuit of constructing the complex tasks into a structured representation, OAKINK2 introduces three level of abstraction to organize the manipulation tasks: Affordance, Primitive Task, and Complex Task. OAKINK2 features on an object-centric perspective for decoding the complex tasks, treating them as a sequence of object affordance fulfillment. The first level, Affordance, outlines the functionalities that objects in the scene can afford, the second level, Primitive Task, describes the minimal interaction units that humans interact with the object to achieve its affordance, and the third level, Complex Task, illustrates how Primitive Tasks are composed and interdependent. OAKINK2 dataset provides multi-view image streams and precise pose annotations for the human body, hands and various interacting objects. This extensive collection supports applications such as interaction reconstruction and motion synthesis. Based on the 3-level abstraction of OAKINK2, we explore a task-oriented framework for Complex Task Completion (CTC). CTC aims to generate a sequence of bimanual manipulation to achieve task objectives. Within the CTC framework, we employ Large Language Models (LLMs) to decompose the complex task objectives into sequences of Primitive Tasks and have developed a Motion Fulfillment Model that generates bimanual hand motion for each Primitive Task. OAKINK2 datasets and models are available at https://oakink.net/v2.

Title: Fairness in Ranking: Robustness through Randomization without the Protected Attribute

• Authors: Andrii Kliachkin, Eleni Psaroudaki, Jakub Marecek, Dimitris Fotakis
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.19419
• Pdf URL: https://arxiv.org/pdf/2403.19419
• Copy Paste: [[2403.19419]] Fairness in Ranking: Robustness through Randomization without the Protected Attribute(https://arxiv.org/abs/2403.19419)
• Keywords: protect, robust, fair
• Abstract: There has been great interest in fairness in machine learning, especially in relation to classification problems. In ranking-related problems, such as in online advertising, recommender systems, and HR automation, much work on fairness remains to be done. Two complications arise: first, the protected attribute may not be available in many applications. Second, there are multiple measures of fairness of rankings, and optimization-based methods utilizing a single measure of fairness of rankings may produce rankings that are unfair with respect to other measures. In this work, we propose a randomized method for post-processing rankings, which do not require the availability of the protected attribute. In an extensive numerical study, we show the robustness of our methods with respect to P-Fairness and effectiveness with respect to Normalized Discounted Cumulative Gain (NDCG) from the baseline ranking, improving on previously proposed methods.

Title: Burst Super-Resolution with Diffusion Models for Improving Perceptual Quality

• Authors: Kyotaro Tokoro, Kazutoshi Akita, Norimichi Ukita
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19428
• Pdf URL: https://arxiv.org/pdf/2403.19428
• Copy Paste: [[2403.19428]] Burst Super-Resolution with Diffusion Models for Improving Perceptual Quality(https://arxiv.org/abs/2403.19428)
• Keywords: diffusion
• Abstract: While burst LR images are useful for improving the SR image quality compared with a single LR image, prior SR networks accepting the burst LR images are trained in a deterministic manner, which is known to produce a blurry SR image. In addition, it is difficult to perfectly align the burst LR images, making the SR image more blurry. Since such blurry images are perceptually degraded, we aim to reconstruct the sharp high-fidelity boundaries. Such high-fidelity images can be reconstructed by diffusion models. However, prior SR methods using the diffusion model are not properly optimized for the burst SR task. Specifically, the reverse process starting from a random sample is not optimized for image enhancement and restoration methods, including burst SR. In our proposed method, on the other hand, burst LR features are used to reconstruct the initial burst SR image that is fed into an intermediate step in the diffusion model. This reverse process from the intermediate step 1) skips diffusion steps for reconstructing the global structure of the image and 2) focuses on steps for refining detailed textures. Our experimental results demonstrate that our method can improve the scores of the perceptual quality metrics. Code: https://github.com/placerkyo/BSRD

Title: BAMM: Bidirectional Autoregressive Motion Model

• Authors: Ekkasit Pinyoanuntapong, Muhammad Usama Saleem, Pu Wang, Minwoo Lee, Srijan Das, Chen Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19435
• Pdf URL: https://arxiv.org/pdf/2403.19435
• Copy Paste: [[2403.19435]] BAMM: Bidirectional Autoregressive Motion Model(https://arxiv.org/abs/2403.19435)
• Keywords: diffusion, transformer, generative
• Abstract: Generating human motion from text has been dominated by denoising motion models either through diffusion or generative masking process. However, these models face great limitations in usability by requiring prior knowledge of the motion length. Conversely, autoregressive motion models address this limitation by adaptively predicting motion endpoints, at the cost of degraded generation quality and editing capabilities. To address these challenges, we propose Bidirectional Autoregressive Motion Model (BAMM), a novel text-to-motion generation framework. BAMM consists of two key components: (1) a motion tokenizer that transforms 3D human motion into discrete tokens in latent space, and (2) a masked self-attention transformer that autoregressively predicts randomly masked tokens via a hybrid attention masking strategy. By unifying generative masked modeling and autoregressive modeling, BAMM captures rich and bidirectional dependencies among motion tokens, while learning the probabilistic mapping from textual inputs to motion outputs with dynamically-adjusted motion sequence length. This feature enables BAMM to simultaneously achieving high-quality motion generation with enhanced usability and built-in motion editability. Extensive experiments on HumanML3D and KIT-ML datasets demonstrate that BAMM surpasses current state-of-the-art methods in both qualitative and quantitative measures.

Title: SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Control

• Authors: Binyuan Huang, Yuqing Wen, Yucheng Zhao, Yaosi Hu, Yingfei Liu, Fan Jia, Weixin Mao, Tiancai Wang, Chi Zhang, Chang Wen Chen, Zhenzhong Chen, Xiangyu Zhang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.19438
• Pdf URL: https://arxiv.org/pdf/2403.19438
• Copy Paste: [[2403.19438]] SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Control(https://arxiv.org/abs/2403.19438)
• Keywords: generative
• Abstract: Autonomous driving progress relies on large-scale annotated datasets. In this work, we explore the potential of generative models to produce vast quantities of freely-labeled data for autonomous driving applications and present SubjectDrive, the first model proven to scale generative data production in a way that could continuously improve autonomous driving applications. We investigate the impact of scaling up the quantity of generative data on the performance of downstream perception models and find that enhancing data diversity plays a crucial role in effectively scaling generative data production. Therefore, we have developed a novel model equipped with a subject control mechanism, which allows the generative model to leverage diverse external data sources for producing varied and useful data. Extensive evaluations confirm SubjectDrive's efficacy in generating scalable autonomous driving training data, marking a significant step toward revolutionizing data production methods in this field.

Title: Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model

• Authors: Qi Gou, Cam-Tu Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19443
• Pdf URL: https://arxiv.org/pdf/2403.19443
• Copy Paste: [[2403.19443]] Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model(https://arxiv.org/abs/2403.19443)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have become increasingly popular due to their ability to process and generate natural language. However, as they are trained on massive datasets of text, LLMs can inherit harmful biases and produce outputs that are not aligned with human values. This paper studies two main approaches to LLM alignment: Reinforcement Learning with Human Feedback (RLHF) and contrastive learning-based methods like Direct Preference Optimization (DPO). By analyzing the stability and robustness of RLHF and DPO, we propose MPO (Mixed Preference Optimization), a novel method that mitigates the weaknesses of both approaches. Specifically, we propose a two-stage training procedure: first train DPO on an easy dataset, and then perform RLHF on a difficult set with DPO model being the reference model. Here, the easy and difficult sets are constructed by a well-trained reward model that splits response pairs into those with large gaps of reward (easy), and those with small gaps (difficult). The first stage allows us to obtain a relatively optimal policy (LLM) model quickly, whereas the second stage refines LLM with online RLHF, thus mitigating the distribution shift issue associated with DPO. Experiments are conducted on two public alignment datasets, namely HH-RLHF and TLDR, demonstrating the effectiveness of MPO, both in terms of GPT4 and human evaluation.

Title: JDocQA: Japanese Document Question Answering Dataset for Generative Language Models

• Authors: Eri Onami, Shuhei Kurita, Taiki Miyanishi, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19454
• Pdf URL: https://arxiv.org/pdf/2403.19454
• Copy Paste: [[2403.19454]] JDocQA: Japanese Document Question Answering Dataset for Generative Language Models(https://arxiv.org/abs/2403.19454)
• Keywords: generative, large language model
• Abstract: Document question answering is a task of question answering on given documents such as reports, slides, pamphlets, and websites, and it is a truly demanding task as paper and electronic forms of documents are so common in our society. This is known as a quite challenging task because it requires not only text understanding but also understanding of figures and tables, and hence visual question answering (VQA) methods are often examined in addition to textual approaches. We introduce Japanese Document Question Answering (JDocQA), a large-scale document-based QA dataset, essentially requiring both visual and textual information to answer questions, which comprises 5,504 documents in PDF format and annotated 11,600 question-and-answer instances in Japanese. Each QA instance includes references to the document pages and bounding boxes for the answer clues. We incorporate multiple categories of questions and unanswerable questions from the document for realistic question-answering applications. We empirically evaluate the effectiveness of our dataset with text-based large language models (LLMs) and multimodal models. Incorporating unanswerable questions in finetuning may contribute to harnessing the so-called hallucination generation.

Title: Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communication

• Authors: Mingze Sun, Chao Xu, Xinyu Jiang, Yang Liu, Baigui Sun, Ruqi Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19467
• Pdf URL: https://arxiv.org/pdf/2403.19467
• Copy Paste: [[2403.19467]] Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communication(https://arxiv.org/abs/2403.19467)
• Keywords: transformer
• Abstract: In this paper, we introduce an innovative task focused on human communication, aiming to generate 3D holistic human motions for both speakers and listeners. Central to our approach is the incorporation of factorization to decouple audio features and the combination of textual semantic information, thereby facilitating the creation of more realistic and coordinated movements. We separately train VQ-VAEs with respect to the holistic motions of both speaker and listener. We consider the real-time mutual influence between the speaker and the listener and propose a novel chain-like transformer-based auto-regressive model specifically designed to characterize real-world communication scenarios effectively which can generate the motions of both the speaker and the listener simultaneously. These designs ensure that the results we generate are both coordinated and diverse. Our approach demonstrates state-of-the-art performance on two benchmark datasets. Furthermore, we introduce the HoCo holistic communication dataset, which is a valuable resource for future research. Our HoCo dataset and code will be released for research purposes upon acceptance.

Title: Benchmarking Implicit Neural Representation and Geometric Rendering in Real-Time RGB-D SLAM

• Authors: Tongyan Hua, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19473
• Pdf URL: https://arxiv.org/pdf/2403.19473
• Copy Paste: [[2403.19473]] Benchmarking Implicit Neural Representation and Geometric Rendering in Real-Time RGB-D SLAM(https://arxiv.org/abs/2403.19473)
• Keywords: robust, fair
• Abstract: Implicit neural representation (INR), in combination with geometric rendering, has recently been employed in real-time dense RGB-D SLAM. Despite active research endeavors being made, there lacks a unified protocol for fair evaluation, impeding the evolution of this area. In this work, we establish, to our knowledge, the first open-source benchmark framework to evaluate the performance of a wide spectrum of commonly used INRs and rendering functions for mapping and localization. The goal of our benchmark is to 1) gain an intuition of how different INRs and rendering functions impact mapping and localization and 2) establish a unified evaluation protocol w.r.t. the design choices that may impact the mapping and localization. With the framework, we conduct a large suite of experiments, offering various insights in choosing the INRs and geometric rendering functions: for example, the dense feature grid outperforms other INRs (e.g. tri-plane and hash grid), even when geometric and color features are jointly encoded for memory efficiency. To extend the findings into the practical scenario, a hybrid encoding strategy is proposed to bring the best of the accuracy and completion from the grid-based and decomposition-based INRs. We further propose explicit hybrid encoding for high-fidelity dense grid mapping to comply with the RGB-D SLAM system that puts the premise on robustness and computation efficiency.

Title: Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment

• Authors: Alireza Ganjdanesh, Shangqian Gao, Heng Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19490
• Pdf URL: https://arxiv.org/pdf/2403.19490
• Copy Paste: [[2403.19490]] Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment(https://arxiv.org/abs/2403.19490)
• Keywords: secure
• Abstract: Structural model pruning is a prominent approach used for reducing the computational cost of Convolutional Neural Networks (CNNs) before their deployment on resource-constrained devices. Yet, the majority of proposed ideas require a pretrained model before pruning, which is costly to secure. In this paper, we propose a novel structural pruning approach to jointly learn the weights and structurally prune architectures of CNN models. The core element of our method is a Reinforcement Learning (RL) agent whose actions determine the pruning ratios of the CNN model's layers, and the resulting model's accuracy serves as its reward. We conduct the joint training and pruning by iteratively training the model's weights and the agent's policy, and we regularize the model's weights to align with the selected structure by the agent. The evolving model's weights result in a dynamic reward function for the agent, which prevents using prominent episodic RL methods with stationary environment assumption for our purpose. We address this challenge by designing a mechanism to model the complex changing dynamics of the reward function and provide a representation of it to the RL agent. To do so, we take a learnable embedding for each training epoch and employ a recurrent model to calculate a representation of the changing environment. We train the recurrent model and embeddings using a decoder model to reconstruct observed rewards. Such a design empowers our agent to effectively leverage episodic observations along with the environment representations to learn a proper policy to determine performant sub-networks of the CNN model. Our extensive experiments on CIFAR-10 and ImageNet using ResNets and MobileNets demonstrate the effectiveness of our method.

Title: Segmentation tool for images of cracks

• Authors: Andrii Kompanets, Remco Duits, Davide Leonetti, Nicky van den Berg, H.H. (Bert)Snijder
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19492
• Pdf URL: https://arxiv.org/pdf/2403.19492
• Copy Paste: [[2403.19492]] Segmentation tool for images of cracks(https://arxiv.org/abs/2403.19492)
• Keywords: segmentation
• Abstract: Safety-critical infrastructures, such as bridges, are periodically inspected to check for existing damage, such as fatigue cracks and corrosion, and to guarantee the safe use of the infrastructure. Visual inspection is the most frequent type of general inspection, despite the fact that its detection capability is rather limited, especially for fatigue cracks. Machine learning algorithms can be used for augmenting the capability of classical visual inspection of bridge structures, however, the implementation of such an algorithm requires a massive annotated training dataset, which is time-consuming to produce. This paper proposes a semi-automatic crack segmentation tool that eases the manual segmentation of cracks on images needed to create a training dataset for a machine learning algorithm. Also, it can be used to measure the geometry of the crack. This tool makes use of an image processing algorithm, which was initially developed for the analysis of vascular systems on retinal images. The algorithm relies on a multi-orientation wavelet transform, which is applied to the image to construct the so-called "orientation scores", i.e. a modified version of the image. Afterwards, the filtered orientation scores are used to formulate an optimal path problem that identifies the crack. The globally optimal path between manually selected crack endpoints is computed, using a state-of-the-art geometric tracking method. The pixel-wise segmentation is done afterwards using the obtained crack path. The proposed method outperforms fully automatic methods and shows potential to be an adequate alternative to the manual data annotation.

Title: Surface-based parcellation and vertex-wise analysis of ultra high-resolution ex vivo 7 tesla MRI in neurodegenerative diseases

• Authors: Pulkit Khandelwal, Michael Tran Duong, Constanza Fuentes, Amanda Denning, Winifred Trotman, Ranjit Ittyerah, Alejandra Bahena, Theresa Schuck, Marianna Gabrielyan, Karthik Prabhakaran, Daniel Ohm, Gabor Mizsei, John Robinson, Monica Munoz, John Detre, Edward Lee, David Irwin, Corey McMillan, M. Dylan Tisdall, Sandhitsu Das, David Wolk, Paul A. Yushkevich
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19497
• Pdf URL: https://arxiv.org/pdf/2403.19497
• Copy Paste: [[2403.19497]] Surface-based parcellation and vertex-wise analysis of ultra high-resolution ex vivo 7 tesla MRI in neurodegenerative diseases(https://arxiv.org/abs/2403.19497)
• Keywords: generative, segmentation
• Abstract: Magnetic resonance imaging (MRI) is the standard modality to understand human brain structure and function in vivo (antemortem). Decades of research in human neuroimaging has led to the widespread development of methods and tools to provide automated volume-based segmentations and surface-based parcellations which help localize brain functions to specialized anatomical regions. Recently ex vivo (postmortem) imaging of the brain has opened-up avenues to study brain structure at sub-millimeter ultra high-resolution revealing details not possible to observe with in vivo MRI. Unfortunately, there has been limited methodological development in ex vivo MRI primarily due to lack of datasets and limited centers with such imaging resources. Therefore, in this work, we present one-of-its-kind dataset of 82 ex vivo T2w whole brain hemispheres MRI at 0.3 mm isotropic resolution spanning Alzheimer's disease and related dementias. We adapted and developed a fast and easy-to-use automated surface-based pipeline to parcellate, for the first time, ultra high-resolution ex vivo brain tissue at the native subject space resolution using the Desikan-Killiany-Tourville (DKT) brain atlas. This allows us to perform vertex-wise analysis in the template space and thereby link morphometry measures with pathology measurements derived from histology. We will open-source our dataset docker container, Jupyter notebooks for ready-to-use out-of-the-box set of tools and command line options to advance ex vivo MRI clinical brain imaging research on the project webpage.

Title: Client-supervised Federated Learning: Towards One-model-for-all Personalization

• Authors: Peng Yan, Guodong Long
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19499
• Pdf URL: https://arxiv.org/pdf/2403.19499
• Copy Paste: [[2403.19499]] Client-supervised Federated Learning: Towards One-model-for-all Personalization(https://arxiv.org/abs/2403.19499)
• Keywords: robust, federate
• Abstract: Personalized Federated Learning (PerFL) is a new machine learning paradigm that delivers personalized models for diverse clients under federated learning settings. Most PerFL methods require extra learning processes on a client to adapt a globally shared model to the client-specific personalized model using its own local data. However, the model adaptation process in PerFL is still an open challenge in the stage of model deployment and test time. This work tackles the challenge by proposing a novel federated learning framework to learn only one robust global model to achieve competitive performance to those personalized models on unseen/test clients in the FL system. Specifically, we design a new Client-Supervised Federated Learning (FedCS) to unravel clients' bias on instances' latent representations so that the global model can learn both client-specific and client-agnostic knowledge. Experimental study shows that the FedCS can learn a robust FL global model for the changing data distributions of unseen/test clients. The FedCS's global model can be directly deployed to the test clients while achieving comparable performance to other personalized FL methods that require model adaptation.

Title: Phonetic Segmentation of the UCLA Phonetics Lab Archive

• Authors: Eleanor Chodroff, Blaž Pažon, Annie Baker, Steven Moran
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.19509
• Pdf URL: https://arxiv.org/pdf/2403.19509
• Copy Paste: [[2403.19509]] Phonetic Segmentation of the UCLA Phonetics Lab Archive(https://arxiv.org/abs/2403.19509)
• Keywords: segmentation
• Abstract: Research in speech technologies and comparative linguistics depends on access to diverse and accessible speech data. The UCLA Phonetics Lab Archive is one of the earliest multilingual speech corpora, with long-form audio recordings and phonetic transcriptions for 314 languages (Ladefoged et al., 2009). Recently, 95 of these languages were time-aligned with word-level phonetic transcriptions (Li et al., 2021). Here we present VoxAngeles, a corpus of audited phonetic transcriptions and phone-level alignments of the UCLA Phonetics Lab Archive, which uses the 95-language CMU re-release as our starting point. VoxAngeles also includes word- and phone-level segmentations from the original UCLA corpus, as well as phonetic measurements of word and phone durations, vowel formants, and vowel f0. This corpus enhances the usability of the original data, particularly for quantitative phonetic typology, as demonstrated through a case study of vowel intrinsic f0. We also discuss the utility of the VoxAngeles corpus for general research and pedagogy in crosslinguistic phonetics, as well as for low-resource and multilingual speech technologies. VoxAngeles is free to download and use under a CC-BY-NC 4.0 license.

Title: On the Robustness of LDP Protocols for Numerical Attributes under Data Poisoning Attacks

• Authors: Xiaoguang Li, Zitao Li, Ninghui Li, Wenhai Sun
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.19510
• Pdf URL: https://arxiv.org/pdf/2403.19510
• Copy Paste: [[2403.19510]] On the Robustness of LDP Protocols for Numerical Attributes under Data Poisoning Attacks(https://arxiv.org/abs/2403.19510)
• Keywords: security, privacy, attack, robust
• Abstract: Recent studies reveal that local differential privacy (LDP) protocols are vulnerable to data poisoning attacks where an attacker can manipulate the final estimate on the server by leveraging the characteristics of LDP and sending carefully crafted data from a small fraction of controlled local clients. This vulnerability raises concerns regarding the robustness and reliability of LDP in hostile environments. In this paper, we conduct a systematic investigation of the robustness of state-of-the-art LDP protocols for numerical attributes, i.e., categorical frequency oracles (CFOs) with binning and consistency, and distribution reconstruction. We evaluate protocol robustness through an attack-driven approach and propose new metrics for cross-protocol attack gain measurement. The results indicate that Square Wave and CFO-based protocols in the Server setting are more robust against the attack compared to the CFO-based protocols in the User setting. Our evaluation also unfolds new relationships between LDP security and its inherent design choices. We found that the hash domain size in local-hashing-based LDP has a profound impact on protocol robustness beyond the well-known effect on utility. Further, we propose a zero-shot attack detection by leveraging the rich reconstructed distribution information. The experiment show that our detection significantly improves the existing methods and effectively identifies data manipulation in challenging scenarios.

Title: Improving Clinical NLP Performance through Language Model-Generated Synthetic Clinical Data

• Authors: Shan Chen, Jack Gallifant, Marco Guevara, Yanjun Gao, Majid Afshar, Timothy Miller, Dmitriy Dligach, Danielle S. Bitterman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19511
• Pdf URL: https://arxiv.org/pdf/2403.19511
• Copy Paste: [[2403.19511]] Improving Clinical NLP Performance through Language Model-Generated Synthetic Clinical Data(https://arxiv.org/abs/2403.19511)
• Keywords: generative
• Abstract: Generative models have been showing potential for producing data in mass. This study explores the enhancement of clinical natural language processing performance by utilizing synthetic data generated from advanced language models. Promising results show feasible applications in such a high-stakes domain.

Title: Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Models

• Authors: Ang Lv, Kaiyi Zhang, Yuhan Chen, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, Rui Yan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19521
• Pdf URL: https://arxiv.org/pdf/2403.19521
• Copy Paste: [[2403.19521]] Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Models(https://arxiv.org/abs/2403.19521)
• Keywords: transformer
• Abstract: In this paper, we deeply explore the mechanisms employed by Transformer-based language models in factual recall tasks. In zero-shot scenarios, given a prompt like "The capital of France is," task-specific attention heads extract the topic entity, such as "France," from the context and pass it to subsequent MLPs to recall the required answer such as "Paris." We introduce a novel analysis method aimed at decomposing the outputs of the MLP into components understandable by humans. Through this method, we quantify the function of the MLP layer following these task-specific heads. In the residual stream, it either erases or amplifies the information originating from individual heads. Moreover, it generates a component that redirects the residual stream towards the direction of its expected answer. These zero-shot mechanisms are also employed in few-shot scenarios. Additionally, we observed a widely existent anti-overconfidence mechanism in the final layer of models, which suppresses correct predictions. We mitigate this suppression by leveraging our interpretation to improve factual recall performance. Our interpretations have been evaluated across various language models, from the GPT-2 families to 1.3B OPT, and across tasks covering different domains of factual knowledge.

Title: Instance-Adaptive and Geometric-Aware Keypoint Learning for Category-Level 6D Object Pose Estimation

• Authors: Xiao Lin, Wenfei Yang, Yuan Gao, Tianzhu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19527
• Pdf URL: https://arxiv.org/pdf/2403.19527
• Copy Paste: [[2403.19527]] Instance-Adaptive and Geometric-Aware Keypoint Learning for Category-Level 6D Object Pose Estimation(https://arxiv.org/abs/2403.19527)
• Keywords: robust
• Abstract: Category-level 6D object pose estimation aims to estimate the rotation, translation and size of unseen instances within specific categories. In this area, dense correspondence-based methods have achieved leading performance. However, they do not explicitly consider the local and global geometric information of different instances, resulting in poor generalization ability to unseen instances with significant shape variations. To deal with this problem, we propose a novel Instance-Adaptive and Geometric-Aware Keypoint Learning method for category-level 6D object pose estimation (AG-Pose), which includes two key designs: (1) The first design is an Instance-Adaptive Keypoint Detection module, which can adaptively detect a set of sparse keypoints for various instances to represent their geometric structures. (2) The second design is a Geometric-Aware Feature Aggregation module, which can efficiently integrate the local and global geometric information into keypoint features. These two modules can work together to establish robust keypoint-level correspondences for unseen instances, thus enhancing the generalization ability of the model.Experimental results on CAMERA25 and REAL275 datasets show that the proposed AG-Pose outperforms state-of-the-art methods by a large margin without category-specific shape priors.

Title: SecGraph: Towards SGX-based Efficient and Confidentiality-Preserving Graph Search

• Authors: Qiuhao Wang, Xu Yang, Saiyu Qi, Yong Qi
• Subjects: cs.CR, cs.DB, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.19531
• Pdf URL: https://arxiv.org/pdf/2403.19531
• Copy Paste: [[2403.19531]] SecGraph: Towards SGX-based Efficient and Confidentiality-Preserving Graph Search(https://arxiv.org/abs/2403.19531)
• Keywords: security, privacy
• Abstract: Graphs have more expressive power and are widely researched in various search demand scenarios, compared with traditional relational and XML models. Today, many graph search services have been deployed on a third-party server, which can alleviate users from the burdens of maintaining large-scale graphs and huge computation costs. Nevertheless, outsourcing graph search services to the third-party server may invade users' privacy. PeGraph was recently proposed to achieve the encrypted search over the social graph. The main idea of PeGraph is to maintain two data structures XSet and TSet motivated by the OXT technology to support encrypted conductive search. However, PeGraph still has some limitations. First, PeGraph suffers from high communication and computation costs in search operations. Second, PeGraph cannot support encrypted search over dynamic graphs. In this paper, we propose an SGX-based efficient and confidentiality-preserving graph search scheme SecGraph that can support insertion and deletion operations. We first design a new proxy-token generation method to reduce the communication cost. Then, we design an LDCF-encoded XSet based on the Logarithmic Dynamic Cuckoo Filter to reduce the computation cost. Finally, we design a new dynamic version of TSet named Twin-TSet to enable encrypted search over dynamic graphs. We have demonstrated the confidentiality preservation property of SecGraph through rigorous security analysis. Experiment results show that SecGraph yields up to 208x improvement in search time compared with PeGraph and the communication cost in PeGraph is up to 540x larger than that in SecGraph.

Title: De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts

• Authors: Yuzheng Wang, Dingkang Yang, Zhaoyu Chen, Yang Liu, Siao Liu, Wenqiang Zhang, Lihua Zhang, Lizhe Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19539
• Pdf URL: https://arxiv.org/pdf/2403.19539
• Copy Paste: [[2403.19539]] De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts(https://arxiv.org/abs/2403.19539)
• Keywords: data-free
• Abstract: Data-Free Knowledge Distillation (DFKD) is a promising task to train high-performance small models to enhance actual deployment without relying on the original training data. Existing methods commonly avoid relying on private data by utilizing synthetic or sampled data. However, a long-overlooked issue is that the severe distribution shifts between their substitution and original data, which manifests as huge differences in the quality of images and class proportions. The harmful shifts are essentially the confounder that significantly causes performance bottlenecks. To tackle the issue, this paper proposes a novel perspective with causal inference to disentangle the student models from the impact of such shifts. By designing a customized causal graph, we first reveal the causalities among the variables in the DFKD task. Subsequently, we propose a Knowledge Distillation Causal Intervention (KDCI) framework based on the backdoor adjustment to de-confound the confounder. KDCI can be flexibly combined with most existing state-of-the-art baselines. Experiments in combination with six representative DFKD methods demonstrate the effectiveness of our KDCI, which can obviously help existing methods under almost all settings, \textit{e.g.}, improving the baseline by up to 15.54\% accuracy on the CIFAR-100 dataset.

Title: WaterJudge: Quality-Detection Trade-off when Watermarking Large Language Models

• Authors: Piotr Molenda, Adian Liusie, Mark J. F. Gales
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19548
• Pdf URL: https://arxiv.org/pdf/2403.19548
• Copy Paste: [[2403.19548]] WaterJudge: Quality-Detection Trade-off when Watermarking Large Language Models(https://arxiv.org/abs/2403.19548)
• Keywords: watermark, generative
• Abstract: Watermarking generative-AI systems, such as LLMs, has gained considerable interest, driven by their enhanced capabilities across a wide range of tasks. Although current approaches have demonstrated that small, context-dependent shifts in the word distributions can be used to apply and detect watermarks, there has been little work in analyzing the impact that these perturbations have on the quality of generated texts. Balancing high detectability with minimal performance degradation is crucial in terms of selecting the appropriate watermarking setting; therefore this paper proposes a simple analysis framework where comparative assessment, a flexible NLG evaluation framework, is used to assess the quality degradation caused by a particular watermark setting. We demonstrate that our framework provides easy visualization of the quality-detection trade-off of watermark settings, enabling a simple solution to find an LLM watermark operating point that provides a well-balanced performance. This approach is applied to two different summarization systems and a translation system, enabling cross-model analysis for a task, and cross-task analysis.

Title: Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Dataset

• Authors: Janis Goldzycher, Paul Röttger, Gerold Schneider
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19559
• Pdf URL: https://arxiv.org/pdf/2403.19559
• Copy Paste: [[2403.19559]] Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Dataset(https://arxiv.org/abs/2403.19559)
• Keywords: robust
• Abstract: Hate speech detection models are only as good as the data they are trained on. Datasets sourced from social media suffer from systematic gaps and biases, leading to unreliable models with simplistic decision boundaries. Adversarial datasets, collected by exploiting model weaknesses, promise to fix this problem. However, adversarial data collection can be slow and costly, and individual annotators have limited creativity. In this paper, we introduce GAHD, a new German Adversarial Hate speech Dataset comprising ca.\ 11k examples. During data collection, we explore new strategies for supporting annotators, to create more diverse adversarial examples more efficiently and provide a manual analysis of annotator disagreements for each strategy. Our experiments show that the resulting dataset is challenging even for state-of-the-art hate speech detection models, and that training on GAHD clearly improves model robustness. Further, we find that mixing multiple support strategies is most advantageous. We make GAHD publicly available at https://github.com/jagol/gahd.

Title: Swarm Characteristics Classification Using Neural Networks

• Authors: Donald W. Peltier III, Isaac Kaminer, Abram Clark, Marko Orescanin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19572
• Pdf URL: https://arxiv.org/pdf/2403.19572
• Copy Paste: [[2403.19572]] Swarm Characteristics Classification Using Neural Networks(https://arxiv.org/abs/2403.19572)
• Keywords: security, defense, attack, robust
• Abstract: Understanding the characteristics of swarming autonomous agents is critical for defense and security applications. This article presents a study on using supervised neural network time series classification (NN TSC) to predict key attributes and tactics of swarming autonomous agents for military contexts. Specifically, NN TSC is applied to infer two binary attributes - communication and proportional navigation - which combine to define four mutually exclusive swarm tactics. We identify a gap in literature on using NNs for swarm classification and demonstrate the effectiveness of NN TSC in rapidly deducing intelligence about attacking swarms to inform counter-maneuvers. Through simulated swarm-vs-swarm engagements, we evaluate NN TSC performance in terms of observation window requirements, noise robustness, and scalability to swarm size. Key findings show NNs can predict swarm behaviors with 97% accuracy using short observation windows of 20 time steps, while also demonstrating graceful degradation down to 80% accuracy under 50% noise, as well as excellent scalability to swarm sizes from 10 to 100 agents. These capabilities are promising for real-time decision-making support in defense scenarios by rapidly inferring insights about swarm behavior.

Title: A Public and Reproducible Assessment of the Topics API on Real Data

• Authors: Yohan Beugin, Patrick McDaniel
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.19577
• Pdf URL: https://arxiv.org/pdf/2403.19577
• Copy Paste: [[2403.19577]] A Public and Reproducible Assessment of the Topics API on Real Data(https://arxiv.org/abs/2403.19577)
• Keywords: privacy
• Abstract: The Topics API for the web is Google's privacy-enhancing alternative to replace third-party cookies. Results of prior work have led to an ongoing discussion between Google and research communities about the capability of Topics to trade off both utility and privacy. The central point of contention is largely around the realism of the datasets used in these analyses and their reproducibility; researchers using data collected on a small sample of users or generating synthetic datasets, while Google's results are inferred from a private dataset. In this paper, we complement prior research by performing a reproducible assessment of the latest version of the Topics API on the largest and publicly available dataset of real browsing histories. First, we measure how unique and stable real users' interests are over time. Then, we evaluate if Topics can be used to fingerprint the users from these real browsing traces by adapting methodologies from prior privacy studies. Finally, we call on web actors to perform and enable reproducible evaluations by releasing anonymized distributions. We find that 46%, 55%, and 60% of the 1207 users in the dataset are uniquely re-identified across websites after only 1, 2, and 3 observations of their topics by advertisers, respectively. This paper shows on real data that Topics does not provide the same privacy guarantees to all users, further highlighting the need for public and reproducible evaluations of the claims made by new web proposals.

Title: The Bad Batches: Enhancing Self-Supervised Learning in Image Classification Through Representative Batch Curation

• Authors: Ozgu Goksu, Nicolas Pugeault
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19579
• Pdf URL: https://arxiv.org/pdf/2403.19579
• Copy Paste: [[2403.19579]] The Bad Batches: Enhancing Self-Supervised Learning in Image Classification Through Representative Batch Curation(https://arxiv.org/abs/2403.19579)
• Keywords: robust
• Abstract: The pursuit of learning robust representations without human supervision is a longstanding challenge. The recent advancements in self-supervised contrastive learning approaches have demonstrated high performance across various representation learning challenges. However, current methods depend on the random transformation of training examples, resulting in some cases of unrepresentative positive pairs that can have a large impact on learning. This limitation not only impedes the convergence of the learning process but the robustness of the learnt representation as well as requiring larger batch sizes to improve robustness to such bad batches. This paper attempts to alleviate the influence of false positive and false negative pairs by employing pairwise similarity calculations through the Fr\'echet ResNet Distance (FRD), thereby obtaining robust representations from unlabelled data. The effectiveness of the proposed method is substantiated by empirical results, where a linear classifier trained on self-supervised contrastive representations achieved an impressive 87.74\% top-1 accuracy on STL10 and 99.31\% on the Flower102 dataset. These results emphasize the potential of the proposed approach in pushing the boundaries of the state-of-the-art in self-supervised contrastive learning, particularly for image classification tasks.

Title: DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs

• Authors: Donghyun Kim, Byeongho Heo, Dongyoon Han
• Subjects: cs.CV, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2403.19588
• Pdf URL: https://arxiv.org/pdf/2403.19588
• Copy Paste: [[2403.19588]] DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs(https://arxiv.org/abs/2403.19588)
• Keywords: transformer, segmentation
• Abstract: This paper revives Densely Connected Convolutional Networks (DenseNets) and reveals the underrated effectiveness over predominant ResNet-style architectures. We believe DenseNets' potential was overlooked due to untouched training methods and traditional design elements not fully revealing their capabilities. Our pilot study shows dense connections through concatenation are strong, demonstrating that DenseNets can be revitalized to compete with modern architectures. We methodically refine suboptimal components - architectural adjustments, block redesign, and improved training recipes towards widening DenseNets and boosting memory efficiency while keeping concatenation shortcuts. Our models, employing simple architectural elements, ultimately surpass Swin Transformer, ConvNeXt, and DeiT-III - key architectures in the residual learning lineage. Furthermore, our models exhibit near state-of-the-art performance on ImageNet-1K, competing with the very recent models and downstream tasks, ADE20k semantic segmentation, and COCO object detection/instance segmentation. Finally, we provide empirical analyses that uncover the merits of the concatenation over additive shortcuts, steering a renewed preference towards DenseNet-style designs. Our code is available at https://github.com/naver-ai/rdnet.

Title: Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformers

• Authors: Pingcheng Dong, Yonghao Tan, Dong Zhang, Tianwei Ni, Xuejiao Liu, Yu Liu, Peng Luo, Luhong Liang, Shih-Yang Liu, Xijie Huang, Huaiyu Zhu, Yun Pan, Fengwei An, Kwang-Ting Cheng
• Subjects: cs.LG, cs.AR, cs.NE
• Abstract URL: https://arxiv.org/abs/2403.19591
• Pdf URL: https://arxiv.org/pdf/2403.19591
• Copy Paste: [[2403.19591]] Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformers(https://arxiv.org/abs/2403.19591)
• Keywords: transformer, segmentation
• Abstract: Non-linear functions are prevalent in Transformers and their lightweight variants, incurring substantial and frequently underestimated hardware costs. Previous state-of-the-art works optimize these operations by piece-wise linear approximation and store the parameters in look-up tables (LUT), but most of them require unfriendly high-precision arithmetics such as FP/INT 32 and lack consideration of integer-only INT quantization. This paper proposed a genetic LUT-Approximation algorithm namely GQA-LUT that can automatically determine the parameters with quantization awareness. The results demonstrate that GQA-LUT achieves negligible degradation on the challenging semantic segmentation task for both vanilla and linear Transformer models. Besides, proposed GQA-LUT enables the employment of INT8-based LUT-Approximation that achieves an area savings of 81.3~81.7% and a power reduction of 79.3~80.2% compared to the high-precision FP/INT 32 alternatives. Code is available at https:// github.com/PingchengDong/GQA-LUT.

Title: Frame by Familiar Frame: Understanding Replication in Video Diffusion Models

• Authors: Aimon Rahman, Malsha V. Perera, Vishal M. Patel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19593
• Pdf URL: https://arxiv.org/pdf/2403.19593
• Copy Paste: [[2403.19593]] Frame by Familiar Frame: Understanding Replication in Video Diffusion Models(https://arxiv.org/abs/2403.19593)
• Keywords: diffusion
• Abstract: Building on the momentum of image generation diffusion models, there is an increasing interest in video-based diffusion models. However, video generation poses greater challenges due to its higher-dimensional nature, the scarcity of training data, and the complex spatiotemporal relationships involved. Image generation models, due to their extensive data requirements, have already strained computational resources to their limits. There have been instances of these models reproducing elements from the training samples, leading to concerns and even legal disputes over sample replication. Video diffusion models, which operate with even more constrained datasets and are tasked with generating both spatial and temporal content, may be more prone to replicating samples from their training sets. Compounding the issue, these models are often evaluated using metrics that inadvertently reward replication. In our paper, we present a systematic investigation into the phenomenon of sample replication in video diffusion models. We scrutinize various recent diffusion models for video synthesis, assessing their tendency to replicate spatial and temporal content in both unconditional and conditional generation scenarios. Our study identifies strategies that are less likely to lead to replication. Furthermore, we propose new evaluation strategies that take replication into account, offering a more accurate measure of a model's ability to generate the original content.

Title: Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model

• Authors: Zhicai Wang, Longhui Wei, Tan Wang, Heyu Chen, Yanbin Hao, Xiang Wang, Xiangnan He, Qi Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19600
• Pdf URL: https://arxiv.org/pdf/2403.19600
• Copy Paste: [[2403.19600]] Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model(https://arxiv.org/abs/2403.19600)
• Keywords: diffusion, generative
• Abstract: Text-to-image (T2I) generative models have recently emerged as a powerful tool, enabling the creation of photo-realistic images and giving rise to a multitude of applications. However, the effective integration of T2I models into fundamental image classification tasks remains an open question. A prevalent strategy to bolster image classification performance is through augmenting the training set with synthetic images generated by T2I models. In this study, we scrutinize the shortcomings of both current generative and conventional data augmentation techniques. Our analysis reveals that these methods struggle to produce images that are both faithful (in terms of foreground objects) and diverse (in terms of background contexts) for domain-specific concepts. To tackle this challenge, we introduce an innovative inter-class data augmentation method known as Diff-Mix (https://github.com/Zhicaiwww/Diff-Mix), which enriches the dataset by performing image translations between classes. Our empirical results demonstrate that Diff-Mix achieves a better balance between faithfulness and diversity, leading to a marked improvement in performance across diverse image classification scenarios, including few-shot, conventional, and long-tail classifications for domain-specific datasets.

Title: Retrieval-Enhanced Knowledge Editing for Multi-Hop Question Answering in Language Models

• Authors: Yucheng Shi, Qiaoyu Tan, Xuansheng Wu, Shaochen Zhong, Kaixiong Zhou, Ninghao Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.19631
• Pdf URL: https://arxiv.org/pdf/2403.19631
• Copy Paste: [[2403.19631]] Retrieval-Enhanced Knowledge Editing for Multi-Hop Question Answering in Language Models(https://arxiv.org/abs/2403.19631)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown proficiency in question-answering tasks but often struggle to integrate real-time knowledge updates, leading to potentially outdated or inaccurate responses. This problem becomes even more challenging when dealing with multi-hop questions since they require LLMs to update and integrate multiple knowledge pieces relevant to the questions. To tackle the problem, we propose the Retrieval-Augmented model Editing (RAE) framework tailored for multi-hop question answering. RAE first retrieves edited facts and then refines the language model through in-context learning. Specifically, our retrieval approach, based on mutual information maximization, leverages the reasoning abilities of LLMs to identify chain facts that na\"ive similarity-based searches might miss. Additionally, our framework incorporates a pruning strategy to eliminate redundant information from the retrieved facts, which enhances the editing accuracy and mitigates the hallucination problem. Our framework is supported by theoretical justification for its fact retrieval efficacy. Finally, comprehensive evaluation across various LLMs validates RAE's ability in providing accurate answers with updated knowledge.

Title: Siamese Vision Transformers are Scalable Audio-visual Learners

• Authors: Yan-Bo Lin, Gedas Bertasius
• Subjects: cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.19638
• Pdf URL: https://arxiv.org/pdf/2403.19638
• Copy Paste: [[2403.19638]] Siamese Vision Transformers are Scalable Audio-visual Learners(https://arxiv.org/abs/2403.19638)
• Keywords: robust, transformer
• Abstract: Traditional audio-visual methods rely on independent audio and visual backbones, which is costly and not scalable. In this work, we investigate using an audio-visual siamese network (AVSiam) for efficient and scalable audio-visual pretraining. Our framework uses a single shared vision transformer backbone to process audio and visual inputs, improving its parameter efficiency, reducing the GPU memory footprint, and allowing us to scale our method to larger datasets and model sizes. We pretrain our model using a contrastive audio-visual matching objective with a multi-ratio random masking scheme, which enables our model to process larger audio-visual instance batches, helpful for contrastive learning. Unlike prior audio-visual methods, our method can robustly handle audio, visual, and audio-visual inputs with a single shared ViT backbone. Furthermore, despite using the shared backbone for both modalities, AVSiam achieves competitive or even better results than prior methods on AudioSet and VGGSound for audio-visual classification and retrieval. Our code is available at https://github.com/GenjiB/AVSiam

Title: GANTASTIC: GAN-based Transfer of Interpretable Directions for Disentangled Image Editing in Text-to-Image Diffusion Models

• Authors: Yusuf Dalva, Hidir Yesiltepe, Pinar Yanardag
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19645
• Pdf URL: https://arxiv.org/pdf/2403.19645
• Copy Paste: [[2403.19645]] GANTASTIC: GAN-based Transfer of Interpretable Directions for Disentangled Image Editing in Text-to-Image Diffusion Models(https://arxiv.org/abs/2403.19645)
• Keywords: diffusion, generative
• Abstract: The rapid advancement in image generation models has predominantly been driven by diffusion models, which have demonstrated unparalleled success in generating high-fidelity, diverse images from textual prompts. Despite their success, diffusion models encounter substantial challenges in the domain of image editing, particularly in executing disentangled edits-changes that target specific attributes of an image while leaving irrelevant parts untouched. In contrast, Generative Adversarial Networks (GANs) have been recognized for their success in disentangled edits through their interpretable latent spaces. We introduce GANTASTIC, a novel framework that takes existing directions from pre-trained GAN models-representative of specific, controllable attributes-and transfers these directions into diffusion-based models. This novel approach not only maintains the generative quality and diversity that diffusion models are known for but also significantly enhances their capability to perform precise, targeted image edits, thereby leveraging the best of both worlds.

Title: Change-Agent: Towards Interactive Comprehensive Change Interpretation and Analysis from Change Detection and Change Captioning

• Authors: Chenyang Liu, Keyan Chen, Haotian Zhang, Zipeng Qi, Zhengxia Zou, Zhenwei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19646
• Pdf URL: https://arxiv.org/pdf/2403.19646
• Copy Paste: [[2403.19646]] Change-Agent: Towards Interactive Comprehensive Change Interpretation and Analysis from Change Detection and Change Captioning(https://arxiv.org/abs/2403.19646)
• Keywords: large language model
• Abstract: Monitoring changes in the Earth's surface is crucial for understanding natural processes and human impacts, necessitating precise and comprehensive interpretation methodologies. Remote sensing satellite imagery offers a unique perspective for monitoring these changes, leading to the emergence of remote sensing image change interpretation (RSICI) as a significant research focus. Current RSICI technology encompasses change detection and change captioning, each with its limitations in providing comprehensive interpretation. To address this, we propose an interactive Change-Agent which integrates a multi-level change interpretation (MCI) model as eyes and a large language model (LLM) as the brain. Our Change-Agent can follow user instructions to achieve comprehensive change interpretation and insightful analysis according to user instructions, such as change detection and change captioning, change object counting, change cause analysis, etc. Our proposed MCI model contains two branches of pixel-level change detection and semantic-level change captioning, in which multiple BI-temporal Iterative Interaction (BI3) layers utilize Local Perception Enhancement (LPE) and the Global Difference Fusion Attention (GDFA) modules to enhance the model's discriminative feature representation capabilities. To train the MCI model, we build the LEVIR-MCI dataset with change masks and captions of bi-temporal images. Extensive experiments demonstrate the effectiveness of the proposed change interpretation model and highlight the promising potential of our Change-Agent in facilitating comprehensive and intelligent interpretation of surface changes. We will make our dataset and codebase of the change interpretation model and Change-Agent publicly available to facilitate future research at https://github.com/Chen-Yang-Liu/Change-Agent

Title: Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models

• Authors: Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, Aaron Mueller
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.19647
• Pdf URL: https://arxiv.org/pdf/2403.19647
• Copy Paste: [[2403.19647]] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models(https://arxiv.org/abs/2403.19647)
• Keywords: interpretability
• Abstract: We introduce methods for discovering and applying sparse feature circuits. These are causally implicated subnetworks of human-interpretable features for explaining language model behaviors. Circuits identified in prior work consist of polysemantic and difficult-to-interpret units like attention heads or neurons, rendering them unsuitable for many downstream applications. In contrast, sparse feature circuits enable detailed understanding of unanticipated mechanisms. Because they are based on fine-grained units, sparse feature circuits are useful for downstream tasks: We introduce SHIFT, where we improve the generalization of a classifier by ablating features that a human judges to be task-irrelevant. Finally, we demonstrate an entirely unsupervised and scalable interpretability pipeline by discovering thousands of sparse feature circuits for automatically discovered model behaviors.

Title: MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions

• Authors: Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, Ming-Wei Chang
• Subjects: cs.CV, cs.AI, cs.CL, cs.IR, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.19651
• Pdf URL: https://arxiv.org/pdf/2403.19651
• Copy Paste: [[2403.19651]] MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions(https://arxiv.org/abs/2403.19651)
• Keywords: large language model
• Abstract: Image retrieval, i.e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures. Recent work leverages text instructions to allow users to more freely express their search intents. However, existing work primarily focuses on image pairs that are visually similar and/or can be characterized by a small set of pre-defined relations. The core thesis of this paper is that text instructions can enable retrieving images with richer relations beyond visual similarity. To show this, we introduce MagicLens, a series of self-supervised image retrieval models that support open-ended instructions. MagicLens is built on a key novel insight: image pairs that naturally occur on the same web pages contain a wide range of implicit relations (e.g., inside view of), and we can bring those implicit relations explicit by synthesizing instructions via large multimodal models (LMMs) and large language models (LLMs). Trained on 36.7M (query image, instruction, target image) triplets with rich semantic relations mined from the web, MagicLens achieves comparable or better results on eight benchmarks of various image retrieval tasks than prior state-of-the-art (SOTA) methods. Remarkably, it outperforms previous SOTA but with a 50X smaller model size on multiple benchmarks. Additional human analyses on a 1.4M-image unseen corpus further demonstrate the diversity of search intents supported by MagicLens.

Title: InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction

• Authors: Sirui Xu, Ziyin Wang, Yu-Xiong Wang, Liang-Yan Gui
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.19652
• Pdf URL: https://arxiv.org/pdf/2403.19652
• Copy Paste: [[2403.19652]] InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction(https://arxiv.org/abs/2403.19652)
• Keywords: diffusion, large language model
• Abstract: Text-conditioned human motion generation has experienced significant advancements with diffusion models trained on extensive motion capture data and corresponding textual annotations. However, extending such success to 3D dynamic human-object interaction (HOI) generation faces notable challenges, primarily due to the lack of large-scale interaction data and comprehensive descriptions that align with these interactions. This paper takes the initiative and showcases the potential of generating human-object interactions without direct training on text-interaction pair data. Our key insight in achieving this is that interaction semantics and dynamics can be decoupled. Being unable to learn interaction semantics through supervised training, we instead leverage pre-trained large models, synergizing knowledge from a large language model and a text-to-motion model. While such knowledge offers high-level control over interaction semantics, it cannot grasp the intricacies of low-level interaction dynamics. To overcome this issue, we further introduce a world model designed to comprehend simple physics, modeling how human actions influence object motion. By integrating these components, our novel framework, InterDreamer, is able to generate text-aligned 3D HOI sequences in a zero-shot manner. We apply InterDreamer to the BEHAVE and CHAIRS datasets, and our comprehensive experimental analysis demonstrates its capability to generate realistic and coherent interaction sequences that seamlessly align with the text directives.

Title: Detecting Image Attribution for Text-to-Image Diffusion Models in RGB and Beyond

• Authors: Katherine Xu, Lingzhi Zhang, Jianbo Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19653
• Pdf URL: https://arxiv.org/pdf/2403.19653
• Copy Paste: [[2403.19653]] Detecting Image Attribution for Text-to-Image Diffusion Models in RGB and Beyond(https://arxiv.org/abs/2403.19653)
• Keywords: diffusion
• Abstract: Modern text-to-image (T2I) diffusion models can generate images with remarkable realism and creativity. These advancements have sparked research in fake image detection and attribution, yet prior studies have not fully explored the practical and scientific dimensions of this task. In addition to attributing images to 12 state-of-the-art T2I generators, we provide extensive analyses on what inference stage hyperparameters and image modifications are discernible. Our experiments reveal that initialization seeds are highly detectable, along with other subtle variations in the image generation process to some extent. We further investigate what visual traces are leveraged in image attribution by perturbing high-frequency details and employing mid-level representations of image style and structure. Notably, altering high-frequency information causes only slight reductions in accuracy, and training an attributor on style representations outperforms training on RGB images. Our analyses underscore that fake images are detectable and attributable at various levels of visual granularity than previously explored.

Title: RSMamba: Remote Sensing Image Classification with State Space Model

• Authors: Keyan Chen, Bowen Chen, Chenyang Liu, Wenyuan Li, Zhengxia Zou, Zhenwei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19654
• Pdf URL: https://arxiv.org/pdf/2403.19654
• Copy Paste: [[2403.19654]] RSMamba: Remote Sensing Image Classification with State Space Model(https://arxiv.org/abs/2403.19654)
• Keywords: transformer
• Abstract: Remote sensing image classification forms the foundation of various understanding tasks, serving a crucial function in remote sensing image interpretation. The recent advancements of Convolutional Neural Networks (CNNs) and Transformers have markedly enhanced classification accuracy. Nonetheless, remote sensing scene classification remains a significant challenge, especially given the complexity and diversity of remote sensing scenarios and the variability of spatiotemporal resolutions. The capacity for whole-image understanding can provide more precise semantic cues for scene discrimination. In this paper, we introduce RSMamba, a novel architecture for remote sensing image classification. RSMamba is based on the State Space Model (SSM) and incorporates an efficient, hardware-aware design known as the Mamba. It integrates the advantages of both a global receptive field and linear modeling complexity. To overcome the limitation of the vanilla Mamba, which can only model causal sequences and is not adaptable to two-dimensional image data, we propose a dynamic multi-path activation mechanism to augment Mamba's capacity to model non-causal data. Notably, RSMamba maintains the inherent modeling mechanism of the vanilla Mamba, yet exhibits superior performance across multiple remote sensing image classification datasets. This indicates that RSMamba holds significant potential to function as the backbone of future visual foundation models. The code will be available at \url{https://github.com/KyanChen/RSMamba}.

Title: GaussianCube: Structuring Gaussian Splatting using Optimal Transport for 3D Generative Modeling

• Authors: Bowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, Baining Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.19655
• Pdf URL: https://arxiv.org/pdf/2403.19655
• Copy Paste: [[2403.19655]] GaussianCube: Structuring Gaussian Splatting using Optimal Transport for 3D Generative Modeling(https://arxiv.org/abs/2403.19655)
• Keywords: diffusion, generative
• Abstract: 3D Gaussian Splatting (GS) have achieved considerable improvement over Neural Radiance Fields in terms of 3D fitting fidelity and rendering speed. However, this unstructured representation with scattered Gaussians poses a significant challenge for generative modeling. To address the problem, we introduce GaussianCube, a structured GS representation that is both powerful and efficient for generative modeling. We achieve this by first proposing a modified densification-constrained GS fitting algorithm which can yield high-quality fitting results using a fixed number of free Gaussians, and then re-arranging the Gaussians into a predefined voxel grid via Optimal Transport. The structured grid representation allows us to use standard 3D U-Net as our backbone in diffusion generative modeling without elaborate designs. Extensive experiments conducted on ShapeNet and OmniObject3D show that our model achieves state-of-the-art generation results both qualitatively and quantitatively, underscoring the potential of GaussianCube as a powerful and versatile 3D representation.