2024-04-08

Title: Federated Unlearning for Human Activity Recognition

• Authors: Kongyang Chen, Dongping zhang, Yaping Chai, Weibin Zhang, Shaowei Wang, Jiaxing Shen
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.03659
• Pdf URL: https://arxiv.org/pdf/2404.03659
• Copy Paste: [[2404.03659]] Federated Unlearning for Human Activity Recognition(https://arxiv.org/abs/2404.03659)
• Keywords: privacy, protect, membership infer, federate
• Abstract: The rapid evolution of Internet of Things (IoT) technology has spurred the widespread adoption of Human Activity Recognition (HAR) in various daily life domains. Federated Learning (FL) is frequently utilized to build a global HAR model by aggregating user contributions without transmitting raw individual data. Despite substantial progress in user privacy protection with FL, challenges persist. Regulations like the General Data Protection Regulation (GDPR) empower users to request data removal, raising a new query in FL: How can a HAR client request data removal without compromising other clients' privacy? In response, we propose a lightweight machine unlearning method for refining the FL HAR model by selectively removing a portion of a client's training data. Our method employs a third-party dataset unrelated to model training. Using KL divergence as a loss function for fine-tuning, we aim to align the predicted probability distribution on forgotten data with the third-party dataset. Additionally, we introduce a membership inference evaluation method to assess unlearning effectiveness. Experimental results across diverse datasets show our method achieves unlearning accuracy comparable to \textit{retraining} methods, resulting in speedups ranging from hundreds to thousands.

Title: RL for Consistency Models: Faster Reward Guided Text-to-Image Generation

• Authors: Owen Oertell, Jonathan D. Chang, Yiyi Zhang, Kianté Brantley, Wen Sun
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03673
• Pdf URL: https://arxiv.org/pdf/2404.03673
• Copy Paste: [[2404.03673]] RL for Consistency Models: Faster Reward Guided Text-to-Image Generation(https://arxiv.org/abs/2404.03673)
• Keywords: diffusion, generative
• Abstract: Reinforcement learning (RL) has improved guided image generation with diffusion models by directly optimizing rewards that capture image quality, aesthetics, and instruction following capabilities. However, the resulting generative policies inherit the same iterative sampling process of diffusion models that causes slow generation. To overcome this limitation, consistency models proposed learning a new class of generative models that directly map noise to data, resulting in a model that can generate an image in as few as one sampling iteration. In this work, to optimize text-to-image generative models for task specific rewards and enable fast training and inference, we propose a framework for fine-tuning consistency models via RL. Our framework, called Reinforcement Learning for Consistency Model (RLCM), frames the iterative inference process of a consistency model as an RL procedure. RLCM improves upon RL fine-tuned diffusion models on text-to-image generation capabilities and trades computation during inference time for sample quality. Experimentally, we show that RLCM can adapt text-to-image consistency models to objectives that are challenging to express with prompting, such as image compressibility, and those derived from human feedback, such as aesthetic quality. Comparing to RL finetuned diffusion models, RLCM trains significantly faster, improves the quality of the generation measured under the reward objectives, and speeds up the inference procedure by generating high quality images with as few as two inference steps. Our code is available at https://rlcm.owenoertell.com

Title: Stream of Search (SoS): Learning to Search in Language

• Authors: Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, Noah D. Goodman
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03683
• Pdf URL: https://arxiv.org/pdf/2404.03683
• Copy Paste: [[2404.03683]] Stream of Search (SoS): Learning to Search in Language(https://arxiv.org/abs/2404.03683)
• Keywords: transformer
• Abstract: Language models are rarely shown fruitful mistakes while training. They then struggle to look beyond the next token, suffering from a snowballing of errors and struggling to predict the consequence of their actions several steps ahead. In this paper, we show how language models can be taught to search by representing the process of search in language, as a flattened string -- a stream of search (SoS). We propose a unified language for search that captures an array of different symbolic search strategies. We demonstrate our approach using the simple yet difficult game of Countdown, where the goal is to combine input numbers with arithmetic operations to reach a target number. We pretrain a transformer-based language model from scratch on a dataset of streams of search generated by heuristic solvers. We find that SoS pretraining increases search accuracy by 25% over models trained to predict only the optimal search trajectory. We further finetune this model with two policy improvement methods: Advantage-Induced Policy Alignment (APA) and Self-Taught Reasoner (STaR). The finetuned SoS models solve 36% of previously unsolved problems, including problems that cannot be solved by any of the heuristic solvers. Our results indicate that language models can learn to solve problems via search, self-improve to flexibly use different search strategies, and potentially discover new ones.

Title: Personalized Federated Learning for Spatio-Temporal Forecasting: A Dual Semantic Alignment-Based Contrastive Approach

• Authors: Qingxiang Liu, Sheng Sun, Yuxuan Liang, Jingjing Xue, Min Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03702
• Pdf URL: https://arxiv.org/pdf/2404.03702
• Copy Paste: [[2404.03702]] Personalized Federated Learning for Spatio-Temporal Forecasting: A Dual Semantic Alignment-Based Contrastive Approach(https://arxiv.org/abs/2404.03702)
• Keywords: federate
• Abstract: The existing federated learning (FL) methods for spatio-temporal forecasting fail to capture the inherent spatio-temporal heterogeneity, which calls for personalized FL (PFL) methods to model the spatio-temporally variant patterns. While contrastive learning approach is promising in addressing spatio-temporal heterogeneity, the existing methods are noneffective in determining negative pairs and can hardly apply to PFL paradigm. To tackle this limitation, we propose a novel PFL method, named Federated dUal sEmantic aLignment-based contraStive learning (FUELS), which can adaptively align positive and negative pairs based on semantic similarity, thereby injecting precise spatio-temporal heterogeneity into the latent representation space by auxiliary contrastive tasks. From temporal perspective, a hard negative filtering module is introduced to dynamically align heterogeneous temporal representations for the supplemented intra-client contrastive task. From spatial perspective, we design lightweight-but-efficient prototypes as client-level semantic representations, based on which the server evaluates spatial similarity and yields client-customized global prototypes for the supplemented inter-client contrastive task. Extensive experiments demonstrate that FUELS outperforms state-of-the-art methods, with communication cost decreasing by around 94%.

Title: Improvement of Performance in Freezing of Gait detection in Parkinsons Disease using Transformer networks and a single waist worn triaxial accelerometer

• Authors: Luis Sigcha, Luigi Borzì, Ignacio Pavón, Nélson Costa, Susana Costa, Pedro Arezes, Juan-Manuel López, Guillermo De Arcas
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.03704
• Pdf URL: https://arxiv.org/pdf/2404.03704
• Copy Paste: [[2404.03704]] Improvement of Performance in Freezing of Gait detection in Parkinsons Disease using Transformer networks and a single waist worn triaxial accelerometer(https://arxiv.org/abs/2404.03704)
• Keywords: transformer
• Abstract: Freezing of gait (FOG) is one of the most incapacitating symptoms in Parkinsons disease, affecting more than 50 percent of patients in advanced stages of the disease. The presence of FOG may lead to falls and a loss of independence with a consequent reduction in the quality of life. Wearable technology and artificial intelligence have been used for automatic FOG detection to optimize monitoring. However, differences between laboratory and daily-life conditions present challenges for the implementation of reliable detection systems. Consequently, improvement of FOG detection methods remains important to provide accurate monitoring mechanisms intended for free-living and real-time use. This paper presents advances in automatic FOG detection using a single body-worn triaxial accelerometer and a novel classification algorithm based on Transformers and convolutional networks. This study was performed with data from 21 patients who manifested FOG episodes while performing activities of daily living in a home setting. Results indicate that the proposed FOG-Transformer can bring a significant improvement in FOG detection using leave-one-subject-out cross-validation (LOSO CV). These results bring opportunities for the implementation of accurate monitoring systems for use in ambulatory or home settings.

Title: Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study

• Authors: Zechun Niu, Jiaxin Mao, Qingyao Ai, Ji-Rong Wen
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.03707
• Pdf URL: https://arxiv.org/pdf/2404.03707
• Copy Paste: [[2404.03707]] Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study(https://arxiv.org/abs/2404.03707)
• Keywords: robust
• Abstract: Counterfactual learning to rank (CLTR) has attracted extensive attention in the IR community for its ability to leverage massive logged user interaction data to train ranking models. While the CLTR models can be theoretically unbiased when the user behavior assumption is correct and the propensity estimation is accurate, their effectiveness is usually empirically evaluated via simulation-based experiments due to a lack of widely-available, large-scale, real click logs. However, the mainstream simulation-based experiments are somewhat limited as they often feature a single, deterministic production ranker and simplified user simulation models to generate the synthetic click logs. As a result, the robustness of CLTR models in complex and diverse situations is largely unknown and needs further investigation. To address this problem, in this paper, we aim to investigate the robustness of existing CLTR models in a reproducibility study with extensive simulation-based experiments that (1) use both deterministic and stochastic production rankers, each with different ranking performance, and (2) leverage multiple user simulation models with different user behavior assumptions. We find that the DLA models and IPS-DCM show better robustness under various simulation settings than IPS-PBM and PRS with offline propensity estimation. Besides, the existing CLTR models often fail to outperform the naive click baselines when the production ranker has relatively high ranking performance or certain randomness, which suggests an urgent need for developing new CLTR algorithms that work for these settings.

Title: Explaining Explainability: Understanding Concept Activation Vectors

• Authors: Angus Nicolson, Lisa Schut, J. Alison Noble, Yarin Gal
• Subjects: cs.LG, cs.AI, cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.03713
• Pdf URL: https://arxiv.org/pdf/2404.03713
• Copy Paste: [[2404.03713]] Explaining Explainability: Understanding Concept Activation Vectors(https://arxiv.org/abs/2404.03713)
• Keywords: interpretability, explainability
• Abstract: Recent interpretability methods propose using concept-based explanations to translate the internal representations of deep learning models into a language that humans are familiar with: concepts. This requires understanding which concepts are present in the representation space of a neural network. One popular method for finding concepts is Concept Activation Vectors (CAVs), which are learnt using a probe dataset of concept exemplars. In this work, we investigate three properties of CAVs. CAVs may be: (1) inconsistent between layers, (2) entangled with different concepts, and (3) spatially dependent. Each property provides both challenges and opportunities in interpreting models. We introduce tools designed to detect the presence of these properties, provide insight into how they affect the derived explanations, and provide recommendations to minimise their impact. Understanding these properties can be used to our advantage. For example, we introduce spatially dependent CAVs to test if a model is translation invariant with respect to a specific concept and class. Our experiments are performed on ImageNet and a new synthetic dataset, Elements. Elements is designed to capture a known ground truth relationship between concepts and classes. We release this dataset to facilitate further research in understanding and evaluating interpretability methods.

Title: Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

• Authors: Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03715
• Pdf URL: https://arxiv.org/pdf/2404.03715
• Copy Paste: [[2404.03715]] Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences(https://arxiv.org/abs/2404.03715)
• Keywords: large language model
• Abstract: This paper studies post-training large language models (LLMs) using preference feedback from a powerful oracle to help a model iteratively improve over itself. The typical approach for post-training LLMs involves Reinforcement Learning from Human Feedback (RLHF), which traditionally separates reward learning and subsequent policy optimization. However, such a reward maximization approach is limited by the nature of "point-wise" rewards (such as Bradley-Terry model), which fails to express complex intransitive or cyclic preference relations. While advances on RLHF show reward learning and policy optimization can be merged into a single contrastive objective for stability, they yet still remain tethered to the reward maximization framework. Recently, a new wave of research sidesteps the reward maximization presumptions in favor of directly optimizing over "pair-wise" or general preferences. In this paper, we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm that marries the simplicity and stability of contrastive learning with theoretical generality from optimizing general preferences. Because DNO is a batched on-policy algorithm using a regression-based objective, its implementation is straightforward and efficient. Moreover, DNO enjoys monotonic improvement across iterations that help it improve even over a strong teacher (such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of 33% on AlpacaEval 2.0 (even after controlling for response length), an absolute gain of 26% (7% to 33%) over the initializing model. It outperforms models with far more parameters, including Mistral Large, Self-Rewarding LM (70B parameters), and older versions of GPT-4.

Title: SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection

• Authors: Bradley P. Allen, Fina Polat, Paul Groth
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03732
• Pdf URL: https://arxiv.org/pdf/2404.03732
• Copy Paste: [[2404.03732]] SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection(https://arxiv.org/abs/2404.03732)
• Keywords: large language model
• Abstract: We describe the University of Amsterdam Intelligent Data Engineering Lab team's entry for the SemEval-2024 Task 6 competition. The SHROOM-INDElab system builds on previous work on using prompt programming and in-context learning with large language models (LLMs) to build classifiers for hallucination detection, and extends that work through the incorporation of context-specific definition of task, role, and target concept, and automated generation of examples for use in a few-shot prompting approach. The resulting system achieved fourth-best and sixth-best performance in the model-agnostic track and model-aware tracks for Task 6, respectively, and evaluation using the validation sets showed that the system's classification decisions were consistent with those of the crowd-sourced human labellers. We further found that a zero-shot approach provided better accuracy than a few-shot approach using automatically generated examples. Code for the system described in this paper is available on Github.

Title: SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer

• Authors: Zijie Wu, Chaohui Yu, Yanqin Jiang, Chenjie Cao, Fan Wang, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03736
• Pdf URL: https://arxiv.org/pdf/2404.03736
• Copy Paste: [[2404.03736]] SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer(https://arxiv.org/abs/2404.03736)
• Keywords: generative
• Abstract: Recent advances in 2D/3D generative models enable the generation of dynamic 3D objects from a single-view video. Existing approaches utilize score distillation sampling to form the dynamic scene as dynamic NeRF or dense 3D Gaussians. However, these methods struggle to strike a balance among reference view alignment, spatio-temporal consistency, and motion fidelity under single-view conditions due to the implicit nature of NeRF or the intricate dense Gaussian motion prediction. To address these issues, this paper proposes an efficient, sparse-controlled video-to-4D framework named SC4D, that decouples motion and appearance to achieve superior video-to-4D generation. Moreover, we introduce Adaptive Gaussian (AG) initialization and Gaussian Alignment (GA) loss to mitigate shape degeneration issue, ensuring the fidelity of the learned motion and shape. Comprehensive experimental results demonstrate that our method surpasses existing methods in both quality and efficiency. In addition, facilitated by the disentangled modeling of motion and appearance of SC4D, we devise a novel application that seamlessly transfers the learned motion onto a diverse array of 4D entities according to textual descriptions.

Title: Test Time Training for Industrial Anomaly Segmentation

• Authors: Alex Costanzino, Pierluigi Zama Ramirez, Mirko Del Moro, Agostino Aiezzo, Giuseppe Lisanti, Samuele Salti, Luigi Di Stefano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03743
• Pdf URL: https://arxiv.org/pdf/2404.03743
• Copy Paste: [[2404.03743]] Test Time Training for Industrial Anomaly Segmentation(https://arxiv.org/abs/2404.03743)
• Keywords: segmentation
• Abstract: Anomaly Detection and Segmentation (AD&S) is crucial for industrial quality control. While existing methods excel in generating anomaly scores for each pixel, practical applications require producing a binary segmentation to identify anomalies. Due to the absence of labeled anomalies in many real scenarios, standard practices binarize these maps based on some statistics derived from a validation set containing only nominal samples, resulting in poor segmentation performance. This paper addresses this problem by proposing a test time training strategy to improve the segmentation performance. Indeed, at test time, we can extract rich features directly from anomalous samples to train a classifier that can discriminate defects effectively. Our general approach can work downstream to any AD&S method that provides an anomaly score map as output, even in multimodal settings. We demonstrate the effectiveness of our approach over baselines through extensive experimentation and evaluation on MVTec AD and MVTec 3D-AD.

Title: Localized Distributional Robustness in Submodular Multi-Task Subset Selection

• Authors: Ege C. Kaya, Abolfazl Hashemi
• Subjects: cs.LG, eess.SP, math.OC
• Abstract URL: https://arxiv.org/abs/2404.03759
• Pdf URL: https://arxiv.org/pdf/2404.03759
• Copy Paste: [[2404.03759]] Localized Distributional Robustness in Submodular Multi-Task Subset Selection(https://arxiv.org/abs/2404.03759)
• Keywords: robust
• Abstract: In this work, we approach the problem of multi-task submodular optimization with the perspective of local distributional robustness, within the neighborhood of a reference distribution which assigns an importance score to each task. We initially propose to introduce a regularization term which makes use of the relative entropy to the standard multi-task objective. We then demonstrate through duality that this novel formulation itself is equivalent to the maximization of a submodular function, which may be efficiently carried out through standard greedy selection methods. This approach bridges the existing gap in the optimization of performance-robustness trade-offs in multi-task subset selection. To numerically validate our theoretical results, we test the proposed method in two different setting, one involving the selection of satellites in low Earth orbit constellations in the context of a sensor selection problem, and the other involving an image summarization task using neural networks. Our method is compared with two other algorithms focused on optimizing the performance of the worst-case task, and on directly optimizing the performance on the reference distribution itself. We conclude that our novel formulation produces a solution that is locally distributional robust, and computationally inexpensive.

Title: CONCERT: Covariate-Elaborated Robust Local Information Transfer with Conditional Spike-and-Slab Prior

• Authors: Ruqian Zhang, Yijiao Zhang, Annie Qu, Zhongyi Zhu, Juan Shen
• Subjects: cs.LG, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.03764
• Pdf URL: https://arxiv.org/pdf/2404.03764
• Copy Paste: [[2404.03764]] CONCERT: Covariate-Elaborated Robust Local Information Transfer with Conditional Spike-and-Slab Prior(https://arxiv.org/abs/2404.03764)
• Keywords: robust
• Abstract: The popularity of transfer learning stems from the fact that it can borrow information from useful auxiliary datasets. Existing statistical transfer learning methods usually adopt a global similarity measure between the source data and the target data, which may lead to inefficiency when only local information is shared. In this paper, we propose a novel Bayesian transfer learning method named "CONCERT" to allow robust local information transfer for high-dimensional data analysis. A novel conditional spike-and-slab prior is introduced in the joint distribution of target and source parameters for information transfer. By incorporating covariate-specific priors, we can characterize the local similarities and make the sources work collaboratively to help improve the performance on the target. Distinguished from existing work, CONCERT is a one-step procedure, which achieves variable selection and information transfer simultaneously. Variable selection consistency is established for our CONCERT. To make our algorithm scalable, we adopt the variational Bayes framework to facilitate implementation. Extensive experiments and a genetic data analysis demonstrate the validity and the advantage of CONCERT over existing cutting-edge transfer learning methods. We also extend our CONCERT to the logistical models with numerical studies showing its superiority over other methods.

Title: R5Detect: Detecting Control-Flow Attacks from Standard RISC-V Enclaves

• Authors: Davide Bove, Lukas Panzer
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.03771
• Pdf URL: https://arxiv.org/pdf/2404.03771
• Copy Paste: [[2404.03771]] R5Detect: Detecting Control-Flow Attacks from Standard RISC-V Enclaves(https://arxiv.org/abs/2404.03771)
• Keywords: security, protect, attack
• Abstract: Embedded and Internet-of-Things (IoT) devices are ubiquitous today, and the uprising of several botnets based on them (e.g., Mirai, Ripple20) raises issues about the security of such devices. Especially low-power devices often lack support for modern system security measures, such as stack integrity, Non-eXecutable bits or strong cryptography. In this work, we present R5Detect, a security monitoring software that detects and prevents control-flow attacks on unmodified RISC-V standard architectures. With a novel combination of different protection techniques, it can run on embedded and low-power IoT devices, which may lack proper security features. R5Detect implements a memory-protected shadow stack to prevent runtime modifications, as well as a heuristics detection based on Hardware Performance Counters to detect control-flow integrity violations. Our results indicate that regular software can be protected against different degrees of control-flow manipulations with an average performance overhead of below 5 %. We implement and evaluate R5Detect on standard low-power RISC-V devices and show that such security features can be effectively used with minimal hardware support.

Title: Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learning

• Authors: Noah Golowich, Ankur Moitra, Dhruv Rohatgi
• Subjects: cs.LG, cs.CC, cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2404.03774
• Pdf URL: https://arxiv.org/pdf/2404.03774
• Copy Paste: [[2404.03774]] Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learning(https://arxiv.org/abs/2404.03774)
• Keywords: robust
• Abstract: Supervised learning is often computationally easy in practice. But to what extent does this mean that other modes of learning, such as reinforcement learning (RL), ought to be computationally easy by extension? In this work we show the first cryptographic separation between RL and supervised learning, by exhibiting a class of block MDPs and associated decoding functions where reward-free exploration is provably computationally harder than the associated regression problem. We also show that there is no computationally efficient algorithm for reward-directed RL in block MDPs, even when given access to an oracle for this regression problem. It is known that being able to perform regression in block MDPs is necessary for finding a good policy; our results suggest that it is not sufficient. Our separation lower bound uses a new robustness property of the Learning Parities with Noise (LPN) hardness assumption, which is crucial in handling the dependent nature of RL data. We argue that separations and oracle lower bounds, such as ours, are a more meaningful way to prove hardness of learning because the constructions better reflect the practical reality that supervised learning by itself is often not the computational bottleneck.

Title: Flattening the Parent Bias: Hierarchical Semantic Segmentation in the Poincaré Ball

• Authors: Simon Weber, Barış Zöngür, Nikita Araslanov, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03778
• Pdf URL: https://arxiv.org/pdf/2404.03778
• Copy Paste: [[2404.03778]] Flattening the Parent Bias: Hierarchical Semantic Segmentation in the Poincaré Ball(https://arxiv.org/abs/2404.03778)
• Keywords: segmentation
• Abstract: Hierarchy is a natural representation of semantic taxonomies, including the ones routinely used in image segmentation. Indeed, recent work on semantic segmentation reports improved accuracy from supervised training leveraging hierarchical label structures. Encouraged by these results, we revisit the fundamental assumptions behind that work. We postulate and then empirically verify that the reasons for the observed improvement in segmentation accuracy may be entirely unrelated to the use of the semantic hierarchy. To demonstrate this, we design a range of cross-domain experiments with a representative hierarchical approach. We find that on the new testing domains, a flat (non-hierarchical) segmentation network, in which the parents are inferred from the children, has superior segmentation accuracy to the hierarchical approach across the board. Complementing these findings and inspired by the intrinsic properties of hyperbolic spaces, we study a more principled approach to hierarchical segmentation using the Poincar\'e ball model. The hyperbolic representation largely outperforms the previous (Euclidean) hierarchical approach as well and is on par with our flat Euclidean baseline in terms of segmentation accuracy. However, it additionally exhibits surprisingly strong calibration quality of the parent nodes in the semantic hierarchy, especially on the more challenging domains. Our combined analysis suggests that the established practice of hierarchical segmentation may be limited to in-domain settings, whereas flat classifiers generalize substantially better, especially if they are modeled in the hyperbolic space.

Title: Quantifying Uncertainty in Motion Prediction with Variational Bayesian Mixture

• Authors: Juanwu Lu, Can Cui, Yunsheng Ma, Aniket Bera, Ziran Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03789
• Pdf URL: https://arxiv.org/pdf/2404.03789
• Copy Paste: [[2404.03789]] Quantifying Uncertainty in Motion Prediction with Variational Bayesian Mixture(https://arxiv.org/abs/2404.03789)
• Keywords: robust, generative
• Abstract: Safety and robustness are crucial factors in developing trustworthy autonomous vehicles. One essential aspect of addressing these factors is to equip vehicles with the capability to predict future trajectories for all moving objects in the surroundings and quantify prediction uncertainties. In this paper, we propose the Sequential Neural Variational Agent (SeNeVA), a generative model that describes the distribution of future trajectories for a single moving object. Our approach can distinguish Out-of-Distribution data while quantifying uncertainty and achieving competitive performance compared to state-of-the-art methods on the Argoverse 2 and INTERACTION datasets. Specifically, a 0.446 meters minimum Final Displacement Error, a 0.203 meters minimum Average Displacement Error, and a 5.35% Miss Rate are achieved on the INTERACTION test set. Extensive qualitative and quantitative analysis is also provided to evaluate the proposed model. Our open-source code is available at https://github.com/PurdueDigitalTwin/seneva.

Title: Language-Guided Instance-Aware Domain-Adaptive Panoptic Segmentation

• Authors: Elham Amin Mansour, Ozan Unal, Suman Saha, Benjamin Bejar, Luc Van Gool
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03799
• Pdf URL: https://arxiv.org/pdf/2404.03799
• Copy Paste: [[2404.03799]] Language-Guided Instance-Aware Domain-Adaptive Panoptic Segmentation(https://arxiv.org/abs/2404.03799)
• Keywords: robust, segmentation
• Abstract: The increasing relevance of panoptic segmentation is tied to the advancements in autonomous driving and AR/VR applications. However, the deployment of such models has been limited due to the expensive nature of dense data annotation, giving rise to unsupervised domain adaptation (UDA). A key challenge in panoptic UDA is reducing the domain gap between a labeled source and an unlabeled target domain while harmonizing the subtasks of semantic and instance segmentation to limit catastrophic interference. While considerable progress has been achieved, existing approaches mainly focus on the adaptation of semantic segmentation. In this work, we focus on incorporating instance-level adaptation via a novel instance-aware cross-domain mixing strategy IMix. IMix significantly enhances the panoptic quality by improving instance segmentation performance. Specifically, we propose inserting high-confidence predicted instances from the target domain onto source images, retaining the exhaustiveness of the resulting pseudo-labels while reducing the injected confirmation bias. Nevertheless, such an enhancement comes at the cost of degraded semantic performance, attributed to catastrophic forgetting. To mitigate this issue, we regularize our semantic branch by employing CLIP-based domain alignment (CDA), exploiting the domain-robustness of natural language prompts. Finally, we present an end-to-end model incorporating these two mechanisms called LIDAPS, achieving state-of-the-art results on all popular panoptic UDA benchmarks.

Title: Learning Social Fairness Preferences from Non-Expert Stakeholder Opinions in Kidney Placement

• Authors: Mukund Telukunta, Sukruth Rao, Gabriella Stickney, Venkata Sriram Siddardh Nadendla, Casey Canfield
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.03800
• Pdf URL: https://arxiv.org/pdf/2404.03800
• Copy Paste: [[2404.03800]] Learning Social Fairness Preferences from Non-Expert Stakeholder Opinions in Kidney Placement(https://arxiv.org/abs/2404.03800)
• Keywords: fair
• Abstract: Modern kidney placement incorporates several intelligent recommendation systems which exhibit social discrimination due to biases inherited from training data. Although initial attempts were made in the literature to study algorithmic fairness in kidney placement, these methods replace true outcomes with surgeons' decisions due to the long delays involved in recording such outcomes reliably. However, the replacement of true outcomes with surgeons' decisions disregards expert stakeholders' biases as well as social opinions of other stakeholders who do not possess medical expertise. This paper alleviates the latter concern and designs a novel fairness feedback survey to evaluate an acceptance rate predictor (ARP) that predicts a kidney's acceptance rate in a given kidney-match pair. The survey is launched on Prolific, a crowdsourcing platform, and public opinions are collected from 85 anonymous crowd participants. A novel social fairness preference learning algorithm is proposed based on minimizing social feedback regret computed using a novel logit-based fairness feedback model. The proposed model and learning algorithm are both validated using simulation experiments as well as Prolific data. Public preferences towards group fairness notions in the context of kidney placement have been estimated and discussed in detail. The specific ARP tested in the Prolific survey has been deemed fair by the participants.

Title: Effective Lymph Nodes Detection in CT Scans Using Location Debiased Query Selection and Contrastive Query Representation in Transformer

• Authors: Qinji Yu, Yirui Wang, Ke Yan, Haoshen Li, Dazhou Guo, Li Zhang, Le Lu, Na Shen, Qifeng Wang, Xiaowei Ding, Xianghua Ye, Dakai Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03819
• Pdf URL: https://arxiv.org/pdf/2404.03819
• Copy Paste: [[2404.03819]] Effective Lymph Nodes Detection in CT Scans Using Location Debiased Query Selection and Contrastive Query Representation in Transformer(https://arxiv.org/abs/2404.03819)
• Keywords: transformer
• Abstract: Lymph node (LN) assessment is a critical, indispensable yet very challenging task in the routine clinical workflow of radiology and oncology. Accurate LN analysis is essential for cancer diagnosis, staging, and treatment planning. Finding scatteredly distributed, low-contrast clinically relevant LNs in 3D CT is difficult even for experienced physicians under high inter-observer variations. Previous automatic LN detection works typically yield limited recall and high false positives (FPs) due to adjacent anatomies with similar image intensities, shapes, or textures (vessels, muscles, esophagus, etc). In this work, we propose a new LN DEtection TRansformer, named LN-DETR, to achieve more accurate performance. By enhancing the 2D backbone with a multi-scale 2.5D feature fusion to incorporate 3D context explicitly, more importantly, we make two main contributions to improve the representation quality of LN queries. 1) Considering that LN boundaries are often unclear, an IoU prediction head and a location debiased query selection are proposed to select LN queries of higher localization accuracy as the decoder query's initialization. 2) To reduce FPs, query contrastive learning is employed to explicitly reinforce LN queries towards their best-matched ground-truth queries over unmatched query predictions. Trained and tested on 3D CT scans of 1067 patients (with 10,000+ labeled LNs) via combining seven LN datasets from different body parts (neck, chest, and abdomen) and pathologies/cancers, our method significantly improves the performance of previous leading methods by > 4-5% average recall at the same FP rates in both internal and external testing. We further evaluate on the universal lesion detection task using NIH DeepLesion benchmark, and our method achieves the top performance of 88.46% averaged recall across 0.5 to 4 FPs per image, compared with other leading reported results.

Title: An Investigation into Misuse of Java Security APIs by Large Language Models

• Authors: Zahra Mousavi, Chadni Islam, Kristen Moore, Alsharif Abuadbba, Muhammad Ali Babar
• Subjects: cs.CR, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.03823
• Pdf URL: https://arxiv.org/pdf/2404.03823
• Copy Paste: [[2404.03823]] An Investigation into Misuse of Java Security APIs by Large Language Models(https://arxiv.org/abs/2404.03823)
• Keywords: secure, security, large language model
• Abstract: The increasing trend of using Large Language Models (LLMs) for code generation raises the question of their capability to generate trustworthy code. While many researchers are exploring the utility of code generation for uncovering software vulnerabilities, one crucial but often overlooked aspect is the security Application Programming Interfaces (APIs). APIs play an integral role in upholding software security, yet effectively integrating security APIs presents substantial challenges. This leads to inadvertent misuse by developers, thereby exposing software to vulnerabilities. To overcome these challenges, developers may seek assistance from LLMs. In this paper, we systematically assess ChatGPT's trustworthiness in code generation for security API use cases in Java. To conduct a thorough evaluation, we compile an extensive collection of 48 programming tasks for 5 widely used security APIs. We employ both automated and manual approaches to effectively detect security API misuse in the code generated by ChatGPT for these tasks. Our findings are concerning: around 70% of the code instances across 30 attempts per task contain security API misuse, with 20 distinct misuse types identified. Moreover, for roughly half of the tasks, this rate reaches 100%, indicating that there is a long way to go before developers can rely on ChatGPT to securely implement security API code.

Title: Outlier-Efficient Hopfield Layers for Large Transformer-Based Models

• Authors: Jerry Yao-Chieh Hu, Pei-Hsuan Chang, Robin Luo, Hong-Yu Chen, Weijian Li, Wei-Po Wang, Han Liu
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.03828
• Pdf URL: https://arxiv.org/pdf/2404.03828
• Copy Paste: [[2404.03828]] Outlier-Efficient Hopfield Layers for Large Transformer-Based Models(https://arxiv.org/abs/2404.03828)
• Keywords: transformer
• Abstract: We introduce an Outlier-Efficient Modern Hopfield Model (termed $\mathtt{OutEffHop}$) and use it to address the outlier-induced challenge of quantizing gigantic transformer-based models. Our main contribution is a novel associative memory model facilitating \textit{outlier-efficient} associative memory retrievals. Interestingly, this memory model manifests a model-based interpretation of an outlier-efficient attention mechanism ($\text{Softmax}_1$): it is an approximation of the memory retrieval process of $\mathtt{OutEffHop}$. Methodologically, this allows us to debut novel outlier-efficient Hopfield layers a powerful attention alternative with superior post-quantization performance. Theoretically, the Outlier-Efficient Modern Hopfield Model retains and improves the desirable properties of the standard modern Hopfield models, including fixed point convergence and exponential storage capacity. Empirically, we demonstrate the proposed model's efficacy across large-scale transformer-based and Hopfield-based models (including BERT, OPT, ViT and STanHop-Net), benchmarking against state-of-the-art methods including $\mathtt{Clipped\_Softmax}$ and $\mathtt{Gated\_Attention}$. Notably, $\mathtt{OutEffHop}$ achieves on average $\sim$22+\% reductions in both average kurtosis and maximum infinity norm of model outputs accross 4 models.

Title: BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model

• Authors: Chenwei Xu, Yu-Chao Huang, Jerry Yao-Chieh Hu, Weijian Li, Ammar Gilani, Hsi-Sheng Goan, Han Liu
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.03830
• Pdf URL: https://arxiv.org/pdf/2404.03830
• Copy Paste: [[2404.03830]] BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model(https://arxiv.org/abs/2404.03830)
• Keywords: robust
• Abstract: We introduce the \textbf{B}i-Directional \textbf{S}parse \textbf{Hop}field Network (\textbf{BiSHop}), a novel end-to-end framework for deep tabular learning. BiSHop handles the two major challenges of deep tabular learning: non-rotationally invariant data structure and feature sparsity in tabular data. Our key motivation comes from the recent established connection between associative memory and attention mechanisms. Consequently, BiSHop uses a dual-component approach, sequentially processing data both column-wise and row-wise through two interconnected directional learning modules. Computationally, these modules house layers of generalized sparse modern Hopfield layers, a sparse extension of the modern Hopfield model with adaptable sparsity. Methodologically, BiSHop facilitates multi-scale representation learning, capturing both intra-feature and inter-feature interactions, with adaptive sparsity at each scale. Empirically, through experiments on diverse real-world datasets, we demonstrate that BiSHop surpasses current SOTA methods with significantly less HPO runs, marking it a robust solution for deep tabular learning.

Title: SleepVST: Sleep Staging from Near-Infrared Video Signals using Pre-Trained Transformers

• Authors: Jonathan F. Carter, João Jorge, Oliver Gibson, Lionel Tarassenko
• Subjects: cs.CV, cs.HC, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2404.03831
• Pdf URL: https://arxiv.org/pdf/2404.03831
• Copy Paste: [[2404.03831]] SleepVST: Sleep Staging from Near-Infrared Video Signals using Pre-Trained Transformers(https://arxiv.org/abs/2404.03831)
• Keywords: robust, transformer
• Abstract: Advances in camera-based physiological monitoring have enabled the robust, non-contact measurement of respiration and the cardiac pulse, which are known to be indicative of the sleep stage. This has led to research into camera-based sleep monitoring as a promising alternative to "gold-standard" polysomnography, which is cumbersome, expensive to administer, and hence unsuitable for longer-term clinical studies. In this paper, we introduce SleepVST, a transformer model which enables state-of-the-art performance in camera-based sleep stage classification (sleep staging). After pre-training on contact sensor data, SleepVST outperforms existing methods for cardio-respiratory sleep staging on the SHHS and MESA datasets, achieving total Cohen's kappa scores of 0.75 and 0.77 respectively. We then show that SleepVST can be successfully transferred to cardio-respiratory waveforms extracted from video, enabling fully contact-free sleep staging. Using a video dataset of 50 nights, we achieve a total accuracy of 78.8\% and a Cohen's $\kappa$ of 0.71 in four-class video-based sleep staging, setting a new state-of-the-art in the domain.

Title: An ExplainableFair Framework for Prediction of Substance Use Disorder Treatment Completion

• Authors: Mary M. Lucas, Xiaoyang Wang, Chia-Hsuan Chang, Christopher C. Yang, Jacqueline E. Braughton, Quyen M. Ngo
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.03833
• Pdf URL: https://arxiv.org/pdf/2404.03833
• Copy Paste: [[2404.03833]] An ExplainableFair Framework for Prediction of Substance Use Disorder Treatment Completion(https://arxiv.org/abs/2404.03833)
• Keywords: fair, explainability
• Abstract: Fairness of machine learning models in healthcare has drawn increasing attention from clinicians, researchers, and even at the highest level of government. On the other hand, the importance of developing and deploying interpretable or explainable models has been demonstrated, and is essential to increasing the trustworthiness and likelihood of adoption of these models. The objective of this study was to develop and implement a framework for addressing both these issues - fairness and explainability. We propose an explainable fairness framework, first developing a model with optimized performance, and then using an in-processing approach to mitigate model biases relative to the sensitive attributes of race and sex. We then explore and visualize explanations of the model changes that lead to the fairness enhancement process through exploring the changes in importance of features. Our resulting-fairness enhanced models retain high sensitivity with improved fairness and explanations of the fairness-enhancement that may provide helpful insights for healthcare providers to guide clinical decision-making and resource allocation.

Title: PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model

• Authors: Amrin Kareem, Jean Lahoud, Hisham Cholakkal
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03836
• Pdf URL: https://arxiv.org/pdf/2404.03836
• Copy Paste: [[2404.03836]] PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model(https://arxiv.org/abs/2404.03836)
• Keywords: segmentation
• Abstract: Recent advancements in 3D perception systems have significantly improved their ability to perform visual recognition tasks such as segmentation. However, these systems still heavily rely on explicit human instruction to identify target objects or categories, lacking the capability to actively reason and comprehend implicit user intentions. We introduce a novel segmentation task known as reasoning part segmentation for 3D objects, aiming to output a segmentation mask based on complex and implicit textual queries about specific parts of a 3D object. To facilitate evaluation and benchmarking, we present a large 3D dataset comprising over 60k instructions paired with corresponding ground-truth part segmentation annotations specifically curated for reasoning-based 3D part segmentation. We propose a model that is capable of segmenting parts of 3D objects based on implicit textual queries and generating natural language explanations corresponding to 3D object segmentation requests. Experiments show that our method achieves competitive performance to models that use explicit queries, with the additional abilities to identify part concepts, reason about them, and complement them with world knowledge. Our source code, dataset, and trained models are available at https://github.com/AmrinKareem/PARIS3D.

Title: Mitigating Heterogeneity in Federated Multimodal Learning with Biomedical Vision-Language Pre-training

• Authors: Zitao Shuai, Liyue Shen
• Subjects: cs.LG, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03854
• Pdf URL: https://arxiv.org/pdf/2404.03854
• Copy Paste: [[2404.03854]] Mitigating Heterogeneity in Federated Multimodal Learning with Biomedical Vision-Language Pre-training(https://arxiv.org/abs/2404.03854)
• Keywords: privacy, protect, robust, federate
• Abstract: Vision-language pre-training (VLP) has arised as an efficient scheme for multimodal representation learning, but it requires large-scale multimodal data for pre-training, making it an obstacle especially for biomedical applications. To overcome the data limitation, federated learning (FL) can be a promising strategy to scale up the dataset for biomedical VLP while protecting data privacy. However, client data are often heterogeneous in real-world scenarios, and we observe that local training on heterogeneous client data would distort the multimodal representation learning and lead to biased cross-modal alignment. To address this challenge, we propose Federated distributional Robust Guidance-Based (FedRGB) learning framework for federated VLP with robustness to data heterogeneity. Specifically, we utilize a guidance-based local training scheme to reduce feature distortions, and employ a distribution-based min-max optimization to learn unbiased cross-modal alignment. The experiments on real-world datasets show our method successfully promotes efficient federated multimodal learning for biomedical VLP with data heterogeneity.

Title: Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data

• Authors: Jingyu Zhang, Marc Marone, Tianjian Li, Benjamin Van Durme, Daniel Khashabi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03862
• Pdf URL: https://arxiv.org/pdf/2404.03862
• Copy Paste: [[2404.03862]] Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data(https://arxiv.org/abs/2404.03862)
• Keywords: membership infer, large language model
• Abstract: For humans to trust the fluent generations of large language models (LLMs), they must be able to verify their correctness against trusted, external sources. Recent efforts aim to increase verifiability through citations of retrieved documents or post-hoc provenance. However, such citations are prone to mistakes that further complicate their verifiability. To address these limitations, we tackle the verifiability goal with a different philosophy: we trivialize the verification process by developing models that quote verbatim statements from trusted sources in pre-training data. We propose Quote-Tuning, which demonstrates the feasibility of aligning LLMs to leverage memorized information and quote from pre-training data. Quote-Tuning quantifies quoting against large corpora with efficient membership inference tools, and uses the amount of quotes as an implicit reward signal to construct a synthetic preference dataset for quoting, without any human annotation. Next, the target model is aligned to quote using preference optimization algorithms. Experimental results show that Quote-Tuning significantly increases the percentage of LLM generation quoted verbatim from high-quality pre-training documents by 55% to 130% relative to untuned models while maintaining response quality. Further experiments demonstrate that Quote-Tuning generalizes quoting to out-of-domain data, is applicable in different tasks, and provides additional benefits to truthfulness. Quote-Tuning not only serves as a hassle-free method to increase quoting but also opens up avenues for improving LLM trustworthiness through better verifiability.

Title: FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping

• Authors: Ajay Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Shiwei Liu, Tianlong Chen, Aditya Akella
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03865
• Pdf URL: https://arxiv.org/pdf/2404.03865
• Copy Paste: [[2404.03865]] FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping(https://arxiv.org/abs/2404.03865)
• Keywords: large language model
• Abstract: Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent achieving remarkable success in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges for autoregressive token-by-token generation. To mitigate computation overload incurred during generation, several early-exit and layer-dropping strategies have been proposed. Despite some promising success due to the redundancy across LLMs layers on metrics like Rough-L/BLUE, our careful knowledge-intensive evaluation unveils issues such as generation collapse, hallucination of wrong facts, and noticeable performance drop even at the trivial exit ratio of 10-15% of layers. We attribute these errors primarily to ineffective handling of the KV cache through state copying during early-exit. In this work, we observed the saturation of computationally expensive feed-forward blocks of LLM layers and proposed FFN-SkipLLM, which is a novel fine-grained skip strategy of autoregressive LLMs. More specifically, FFN-SkipLLM is an input-adaptive feed-forward skipping strategy that can skip 25-30% of FFN blocks of LLMs with marginal change in performance on knowledge-intensive generation tasks without any requirement to handle KV cache. Our extensive experiments and ablation across benchmarks like MT-Bench, Factoid-QA, and variable-length text summarization illustrate how our simple and ease-at-use method can facilitate faster autoregressive decoding.

Title: Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction

• Authors: Bowen Zhang, Harold Soh
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03868
• Pdf URL: https://arxiv.org/pdf/2404.03868
• Copy Paste: [[2404.03868]] Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction(https://arxiv.org/abs/2404.03868)
• Keywords: extraction, large language model
• Abstract: In this work, we are interested in automated methods for knowledge graph creation (KGC) from input text. Progress on large language models (LLMs) has prompted a series of recent works applying them to KGC, e.g., via zero/few-shot prompting. Despite successes on small domain-specific datasets, these models face difficulties scaling up to text common in many real-world applications. A principal issue is that in prior methods, the KG schema has to be included in the LLM prompt to generate valid triplets; larger and more complex schema easily exceed the LLMs' context window length. To address this problem, we propose a three-phase framework named Extract-Define-Canonicalize (EDC): open information extraction followed by schema definition and post-hoc canonicalization. EDC is flexible in that it can be applied to settings where a pre-defined target schema is available and when it is not; in the latter case, it constructs a schema automatically and applies self-canonicalization. To further improve performance, we introduce a trained component that retrieves schema elements relevant to the input text; this improves the LLMs' extraction performance in a retrieval-augmented generation-like manner. We demonstrate on three KGC benchmarks that EDC is able to extract high-quality triplets without any parameter tuning and with significantly larger schemas compared to prior works.

Title: Optimizing Convolutional Neural Networks for Identifying Invasive Pollinator Apis Mellifera and Finding a Ligand drug to Protect California's Biodiversity

• Authors: Arnav Swaroop
• Subjects: cs.LG, q-bio.BM, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2404.03870
• Pdf URL: https://arxiv.org/pdf/2404.03870
• Copy Paste: [[2404.03870]] Optimizing Convolutional Neural Networks for Identifying Invasive Pollinator Apis Mellifera and Finding a Ligand drug to Protect California's Biodiversity(https://arxiv.org/abs/2404.03870)
• Keywords: protect
• Abstract: In North America, there are many diverse species of native bees crucial for the environment, who are the primary pollinators of most native floral species. The Californian agriculture industry imports European honeybees (Apis Mellifera) primarily for pollinating almonds. Unfortunately, this has resulted in the unintended consequence of disrupting the native ecosystem and threatening many native bee species as they are outcompeted for food. Our first step for protecting the native species is identification with the use of a Convolutional Neural Network (CNN) to differentiate common native bee species from invasive ones. Removing invasive colonies efficiently without harming native species is difficult as pesticides cause myriad diseases in native species. Our approach seeks to prevent the formation of new queens, causing the colony's collapse. Workers secrete royal jelly, a substance that causes fertility and longevity; it is fed to future honeybee queens. Targeting the production of this substance is safe as no native species use it; small organic molecules (ligands) prevent the proteins Apisimin and MRJP1 from combining and producing an oligomer used to form the substance. Ideal ligands bind to only one of these proteins preventing them from joining together: they have a high affinity for one receptor and a significantly lower affinity for the other. We optimized the CNN to provide a framework for creating Machine Learning models that excel at differentiating between subspecies of insects by measuring the effects of image alteration and class grouping on model performance. The CNN is able to achieve an accuracy of 82% in differentiating between invasive and native bee species; 3 ligands have been identified as effective. Our new approach offers a promising solution to curb the spread of invasive bees within California through an identification and neutralization method.

Title: PrivShape: Extracting Shapes in Time Series under User-Level Local Differential Privacy

• Authors: Yulian Mao, Qingqing Ye, Haibo Hu, Qi Wang, Kai Huang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.03873
• Pdf URL: https://arxiv.org/pdf/2404.03873
• Copy Paste: [[2404.03873]] PrivShape: Extracting Shapes in Time Series under User-Level Local Differential Privacy(https://arxiv.org/abs/2404.03873)
• Keywords: privacy, protect
• Abstract: Time series have numerous applications in finance, healthcare, IoT, and smart city. In many of these applications, time series typically contain personal data, so privacy infringement may occur if they are released directly to the public. Recently, local differential privacy (LDP) has emerged as the state-of-the-art approach to protecting data privacy. However, existing works on LDP-based collections cannot preserve the shape of time series. A recent work, PatternLDP, attempts to address this problem, but it can only protect a finite group of elements in a time series due to {\omega}-event level privacy guarantee. In this paper, we propose PrivShape, a trie-based mechanism under user-level LDP to protect all elements. PrivShape first transforms a time series to reduce its length, and then adopts trie-expansion and two-level refinement to improve utility. By extensive experiments on real-world datasets, we demonstrate that PrivShape outperforms PatternLDP when adapted for offline use, and can effectively extract frequent shapes.

Title: VELLET: Verifiable Embedded Wallet for Securing Authenticity and Integrity

• Authors: Hiroki Watanabe, Kohei Ichihara, Takumi Aita
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.03874
• Pdf URL: https://arxiv.org/pdf/2404.03874
• Copy Paste: [[2404.03874]] VELLET: Verifiable Embedded Wallet for Securing Authenticity and Integrity(https://arxiv.org/abs/2404.03874)
• Keywords: security, attack
• Abstract: The blockchain ecosystem, particularly with the rise of Web3 and Non-Fungible Tokens (NFTs), has experienced a significant increase in users and applications. However, this expansion is challenged by the need to connect early adopters with a wider user base. A notable difficulty in this process is the complex interfaces of blockchain wallets, which can be daunting for those familiar with traditional payment methods. To address this issue, the category of "embedded wallets" has emerged as a promising solution. These wallets are seamlessly integrated into the front-end of decentralized applications (Dapps), simplifying the onboarding process for users and making access more widely available. However, our insights indicate that this simplification introduces a trade-off between ease of use and security. Embedded wallets lack transparency and auditability, leading to obscured transactions by the front end and a pronounced risk of fraud and phishing attacks. This paper proposes a new protocol to enhance the security of embedded wallets. Our VELLET protocol introduces a wallet verifier that can match the audit trail of embedded wallets on smart contracts, incorporating a process to verify authenticity and integrity. In the implementation architecture of the VELLET protocol, we suggest using the Text Record feature of the Ethereum Name Service (ENS), known as a decentralized domain name service, to serve as a repository for managing the audit trails of smart contracts. This approach has been demonstrated to reduce the necessity for new smart contract development and operational costs, proving cost-effective through a proof-of-concept. This protocol is a vital step in reducing security risks associated with embedded wallets, ensuring their convenience does not undermine user security and trust.

Title: Increasing Fairness in Classification of Out of Distribution Data for Facial Recognition

• Authors: Gianluca Barone, Aashrit Cunchala, Rudy Nunez
• Subjects: cs.CV, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03876
• Pdf URL: https://arxiv.org/pdf/2404.03876
• Copy Paste: [[2404.03876]] Increasing Fairness in Classification of Out of Distribution Data for Facial Recognition(https://arxiv.org/abs/2404.03876)
• Keywords: fair
• Abstract: Standard classification theory assumes that the distribution of images in the test and training sets are identical. Unfortunately, real-life scenarios typically feature unseen data ("out-of-distribution data") which is different from data in the training distribution("in-distribution"). This issue is most prevalent in social justice problems where data from under-represented groups may appear in the test data without representing an equal proportion of the training data. This may result in a model returning confidently wrong decisions and predictions. We are interested in the following question: Can the performance of a neural network improve on facial images of out-of-distribution data when it is trained simultaneously on multiple datasets of in-distribution data? We approach this problem by incorporating the Outlier Exposure model and investigate how the model's performance changes when other datasets of facial images were implemented. We observe that the accuracy and other metrics of the model can be increased by applying Outlier Exposure, incorporating a trainable weight parameter to increase the machine's emphasis on outlier images, and by re-weighting the importance of different class labels. We also experimented with whether sorting the images and determining outliers via image features would have more of an effect on the metrics than sorting by average pixel value. Our goal was to make models not only more accurate but also more fair by scanning a more expanded range of images. We also tested the datasets in reverse order to see whether a more fair dataset with balanced features has an effect on the model's accuracy.

Title: Beyond the Bridge: Contention-Based Covert and Side Channel Attacks on Multi-GPU Interconnect

• Authors: Yicheng Zhang, Ravan Nazaraliyev, Sankha Baran Dutta, Nael Abu-Ghazaleh, Andres Marquez, Kevin Barker
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.03877
• Pdf URL: https://arxiv.org/pdf/2404.03877
• Copy Paste: [[2404.03877]] Beyond the Bridge: Contention-Based Covert and Side Channel Attacks on Multi-GPU Interconnect(https://arxiv.org/abs/2404.03877)
• Keywords: attack
• Abstract: High-speed interconnects, such as NVLink, are integral to modern multi-GPU systems, acting as a vital link between CPUs and GPUs. This study highlights the vulnerability of multi-GPU systems to covert and side channel attacks due to congestion on interconnects. An adversary can infer private information about a victim's activities by monitoring NVLink congestion without needing special permissions. Leveraging this insight, we develop a covert channel attack across two GPUs with a bandwidth of 45.5 kbps and a low error rate, and introduce a side channel attack enabling attackers to fingerprint applications through the shared NVLink interconnect.

Title: A Bi-consolidating Model for Joint Relational Triple Extraction

• Authors: Xiaocheng Luo, Yanping Chen, Ruixue Tang, Ruizhang Huang, Yongbin Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03881
• Pdf URL: https://arxiv.org/pdf/2404.03881
• Copy Paste: [[2404.03881]] A Bi-consolidating Model for Joint Relational Triple Extraction(https://arxiv.org/abs/2404.03881)
• Keywords: extraction
• Abstract: Current methods to extract relational triples directly make a prediction based on a possible entity pair in a raw sentence without depending on entity recognition. The task suffers from a serious semantic overlapping problem, in which several relation triples may share one or two entities in a sentence. It is weak to learn discriminative semantic features relevant to a relation triple. In this paper, based on a two-dimensional sentence representation, a bi-consolidating model is proposed to address this problem by simultaneously reinforcing the local and global semantic features relevant to a relation triple. This model consists of a local consolidation component and a global consolidation component. The first component uses a pixel difference convolution to enhance semantic information of a possible triple representation from adjacent regions and mitigate noise in neighbouring neighbours. The second component strengthens the triple representation based a channel attention and a spatial attention, which has the advantage to learn remote semantic dependencies in a sentence. They are helpful to improve the performance of both entity identification and relation type classification in relation triple extraction. After evaluated on several publish datasets, it achieves competitive performance. Analytical experiments demonstrate the effectiveness of our model for relational triple extraction and give motivation for other natural language processing tasks.

Title: SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models

• Authors: Hyeonwoo Kim, Gyoungjin Gim, Yungi Kim, Jihoo Kim, Byungju Kim, Wonseok Lee, Chanjun Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03887
• Pdf URL: https://arxiv.org/pdf/2404.03887
• Copy Paste: [[2404.03887]] SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models(https://arxiv.org/abs/2404.03887)
• Keywords: large language model
• Abstract: This study presents a novel learning approach designed to enhance both mathematical reasoning and problem-solving abilities of Large Language Models (LLMs). We focus on integrating the Chain-of-Thought (CoT) and the Program-of-Thought (PoT) learning, hypothesizing that prioritizing the learning of mathematical reasoning ability is helpful for the amplification of problem-solving ability. Thus, the initial learning with CoT is essential for solving challenging mathematical problems. To this end, we propose a sequential learning approach, named SAAS (Solving Ability Amplification Strategy), which strategically transitions from CoT learning to PoT learning. Our empirical study, involving an extensive performance comparison using several benchmarks, demonstrates that our SAAS achieves state-of-the-art (SOTA) performance. The results underscore the effectiveness of our sequential learning approach, marking a significant advancement in the field of mathematical reasoning in LLMs.

Title: Enhancing Breast Cancer Diagnosis in Mammography: Evaluation and Integration of Convolutional Neural Networks and Explainable AI

• Authors: Maryam Ahmed, Tooba Bibi, Rizwan Ahmed Khan, Sidra Nasir
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.03892
• Pdf URL: https://arxiv.org/pdf/2404.03892
• Copy Paste: [[2404.03892]] Enhancing Breast Cancer Diagnosis in Mammography: Evaluation and Integration of Convolutional Neural Networks and Explainable AI(https://arxiv.org/abs/2404.03892)
• Keywords: fair, interpretability
• Abstract: The study introduces an integrated framework combining Convolutional Neural Networks (CNNs) and Explainable Artificial Intelligence (XAI) for the enhanced diagnosis of breast cancer using the CBIS-DDSM dataset. Utilizing a fine-tuned ResNet50 architecture, our investigation not only provides effective differentiation of mammographic images into benign and malignant categories but also addresses the opaque "black-box" nature of deep learning models by employing XAI methodologies, namely Grad-CAM, LIME, and SHAP, to interpret CNN decision-making processes for healthcare professionals. Our methodology encompasses an elaborate data preprocessing pipeline and advanced data augmentation techniques to counteract dataset limitations, and transfer learning using pre-trained networks, such as VGG-16, DenseNet and ResNet was employed. A focal point of our study is the evaluation of XAI's effectiveness in interpreting model predictions, highlighted by utilising the Hausdorff measure to assess the alignment between AI-generated explanations and expert annotations quantitatively. This approach plays a critical role for XAI in promoting trustworthiness and ethical fairness in AI-assisted diagnostics. The findings from our research illustrate the effective collaboration between CNNs and XAI in advancing diagnostic methods for breast cancer, thereby facilitating a more seamless integration of advanced AI technologies within clinical settings. By enhancing the interpretability of AI-driven decisions, this work lays the groundwork for improved collaboration between AI systems and medical practitioners, ultimately enriching patient care. Furthermore, the implications of our research extend well beyond the current methodologies, advocating for subsequent inquiries into the integration of multimodal data and the refinement of AI explanations to satisfy the needs of clinical practice.

Title: VoltaVision: A Transfer Learning model for electronic component classification

• Authors: Anas Mohammad Ishfaqul Muktadir Osmani, Taimur Rahman, Salekul Islam
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03898
• Pdf URL: https://arxiv.org/pdf/2404.03898
• Copy Paste: [[2404.03898]] VoltaVision: A Transfer Learning model for electronic component classification(https://arxiv.org/abs/2404.03898)
• Keywords: robust
• Abstract: In this paper, we analyze the effectiveness of transfer learning on classifying electronic components. Transfer learning reuses pre-trained models to save time and resources in building a robust classifier rather than learning from scratch. Our work introduces a lightweight CNN, coined as VoltaVision, and compares its performance against more complex models. We test the hypothesis that transferring knowledge from a similar task to our target domain yields better results than state-of-the-art models trained on general datasets. Our dataset and code for this work are available at https://github.com/AnasIshfaque/VoltaVision.

Title: Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models

• Authors: Gihyun Kwon, Simon Jenni, Dingzeyu Li, Joon-Young Lee, Jong Chul Ye, Fabian Caba Heilbron
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03913
• Pdf URL: https://arxiv.org/pdf/2404.03913
• Copy Paste: [[2404.03913]] Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models(https://arxiv.org/abs/2404.03913)
• Keywords: diffusion
• Abstract: While there has been significant progress in customizing text-to-image generation models, generating images that combine multiple personalized concepts remains challenging. In this work, we introduce Concept Weaver, a method for composing customized text-to-image diffusion models at inference time. Specifically, the method breaks the process into two steps: creating a template image aligned with the semantics of input prompts, and then personalizing the template using a concept fusion strategy. The fusion strategy incorporates the appearance of the target concepts into the template image while retaining its structural details. The results indicate that our method can generate multiple custom concepts with higher identity fidelity compared to alternative approaches. Furthermore, the method is shown to seamlessly handle more than two concepts and closely follow the semantic meaning of the input prompt without blending appearances across different subjects.

Title: Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models

• Authors: Bowen Zhang, Kehua Chang, Chunping Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03921
• Pdf URL: https://arxiv.org/pdf/2404.03921
• Copy Paste: [[2404.03921]] Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models(https://arxiv.org/abs/2404.03921)
• Keywords: generative, large language model
• Abstract: Sentence Embedding stands as a fundamental task within the realm of Natural Language Processing, finding extensive application in search engines, expert systems, and question-and-answer platforms. With the continuous evolution of large language models such as LLaMA and Mistral, research on sentence embedding has recently achieved notable breakthroughs. However, these advancements mainly pertain to fine-tuning scenarios, leaving explorations into computationally efficient direct inference methods for sentence representation in a nascent stage. This paper endeavors to bridge this research gap. Through comprehensive experimentation, we challenge the widely held belief in the necessity of an Explicit One-word Limitation for deriving sentence embeddings from Pre-trained Language Models (PLMs). We demonstrate that this approach, while beneficial for generative models under direct inference scenario, is not imperative for discriminative models or the fine-tuning of generative PLMs. This discovery sheds new light on the design of manual templates in future studies. Building upon this insight, we propose two innovative prompt engineering techniques capable of further enhancing the expressive power of PLMs' raw embeddings: Pretended Chain of Thought and Knowledge Enhancement. We confirm their effectiveness across various PLM types and provide a detailed exploration of the underlying factors contributing to their success.

Title: Learning Correlation Structures for Vision Transformers

• Authors: Manjin Kim, Paul Hongsuck Seo, Cordelia Schmid, Minsu Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03924
• Pdf URL: https://arxiv.org/pdf/2404.03924
• Copy Paste: [[2404.03924]] Learning Correlation Structures for Vision Transformers(https://arxiv.org/abs/2404.03924)
• Keywords: transformer
• Abstract: We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention. StructSA generates attention maps by recognizing space-time structures of key-query correlations via convolution and uses them to dynamically aggregate local contexts of value features. This effectively leverages rich structural patterns in images and videos such as scene layouts, object motion, and inter-object relations. Using StructSA as a main building block, we develop the structural vision transformer (StructViT) and evaluate its effectiveness on both image and video classification tasks, achieving state-of-the-art results on ImageNet-1K, Kinetics-400, Something-Something V1 & V2, Diving-48, and FineGym.

Title: Re-pseudonymization Strategies for Smart Meter Data Are Not Robust to Deep Learning Profiling Attacks

• Authors: Ana-Maria Cretu, Miruna Rusu, Yves-Alexandre de Montjoye
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03948
• Pdf URL: https://arxiv.org/pdf/2404.03948
• Copy Paste: [[2404.03948]] Re-pseudonymization Strategies for Smart Meter Data Are Not Robust to Deep Learning Profiling Attacks(https://arxiv.org/abs/2404.03948)
• Keywords: attack, robust, transformer
• Abstract: Smart meters, devices measuring the electricity and gas consumption of a household, are currently being deployed at a fast rate throughout the world. The data they collect are extremely useful, including in the fight against climate change. However, these data and the information that can be inferred from them are highly sensitive. Re-pseudonymization, i.e., the frequent replacement of random identifiers over time, is widely used to share smart meter data while mitigating the risk of re-identification. We here show how, in spite of re-pseudonymization, households' consumption records can be pieced together with high accuracy in large-scale datasets. We propose the first deep learning-based profiling attack against re-pseudonymized smart meter data. Our attack combines neural network embeddings, which are used to extract features from weekly consumption records and are tailored to the smart meter identification task, with a nearest neighbor classifier. We evaluate six neural networks architectures as the embedding model. Our results suggest that the Transformer and CNN-LSTM architectures vastly outperform previous methods as well as other architectures, successfully identifying the correct household 73.4% of the time among 5139 households based on electricity and gas consumption records (54.5% for electricity only). We further show that the features extracted by the embedding model maintain their effectiveness when transferred to a set of users disjoint from the one used to train the model. Finally, we extensively evaluate the robustness of our results. Taken together, our results strongly suggest that even frequent re-pseudonymization strategies can be reversed, strongly limiting their ability to prevent re-identification in practice.

Title: Transformers for molecular property prediction: Lessons learned from the past five years

• Authors: Afnan Sultan, Jochen Sieg, Miriam Mathea, Andrea Volkamer
• Subjects: cs.LG, cs.CL, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2404.03969
• Pdf URL: https://arxiv.org/pdf/2404.03969
• Copy Paste: [[2404.03969]] Transformers for molecular property prediction: Lessons learned from the past five years(https://arxiv.org/abs/2404.03969)
• Keywords: protect, robust, transformer
• Abstract: Molecular Property Prediction (MPP) is vital for drug discovery, crop protection, and environmental science. Over the last decades, diverse computational techniques have been developed, from using simple physical and chemical properties and molecular fingerprints in statistical models and classical machine learning to advanced deep learning approaches. In this review, we aim to distill insights from current research on employing transformer models for MPP. We analyze the currently available models and explore key questions that arise when training and fine-tuning a transformer model for MPP. These questions encompass the choice and scale of the pre-training data, optimal architecture selections, and promising pre-training objectives. Our analysis highlights areas not yet covered in current research, inviting further exploration to enhance the field's understanding. Additionally, we address the challenges in comparing different models, emphasizing the need for standardized data splitting and robust statistical analysis.

Title: SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials

• Authors: Mathilde Aguiar, Pierre Zweigenbaum, Nona Naderi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03977
• Pdf URL: https://arxiv.org/pdf/2404.03977
• Copy Paste: [[2404.03977]] SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials(https://arxiv.org/abs/2404.03977)
• Keywords: generative, large language model
• Abstract: This paper describes our submission to Task 2 of SemEval-2024: Safe Biomedical Natural Language Inference for Clinical Trials. The Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) consists of a Textual Entailment (TE) task focused on the evaluation of the consistency and faithfulness of Natural Language Inference (NLI) models applied to Clinical Trial Reports (CTR). We test 2 distinct approaches, one based on finetuning and ensembling Masked Language Models and the other based on prompting Large Language Models using templates, in particular, using Chain-Of-Thought and Contrastive Chain-Of-Thought. Prompting Flan-T5-large in a 2-shot setting leads to our best system that achieves 0.57 F1 score, 0.64 Faithfulness, and 0.56 Consistency.

Title: Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Study

• Authors: Myrthe Reuver, Suzan Verberne, Antske Fokkens
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03987
• Pdf URL: https://arxiv.org/pdf/2404.03987
• Copy Paste: [[2404.03987]] Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Study(https://arxiv.org/abs/2404.03987)
• Keywords: robust
• Abstract: For a viewpoint-diverse news recommender, identifying whether two news articles express the same viewpoint is essential. One way to determine "same or different" viewpoint is stance detection. In this paper, we investigate the robustness of operationalization choices for few-shot stance detection, with special attention to modelling stance across different topics. Our experiments test pre-registered hypotheses on stance detection. Specifically, we compare two stance task definitions (Pro/Con versus Same Side Stance), two LLM architectures (bi-encoding versus cross-encoding), and adding Natural Language Inference knowledge, with pre-trained RoBERTa models trained with shots of 100 examples from 7 different stance detection datasets. Some of our hypotheses and claims from earlier work can be confirmed, while others give more inconsistent results. The effect of the Same Side Stance definition on performance differs per dataset and is influenced by other modelling choices. We found no relationship between the number of training topics in the training shots and performance. In general, cross-encoding out-performs bi-encoding, and adding NLI training to our models gives considerable improvement, but these results are not consistent across all datasets. Our results indicate that it is essential to include multiple datasets and systematic modelling experiments when aiming to find robust modelling choices for the concept `stance'.

Title: Demonstration Guided Multi-Objective Reinforcement Learning

• Authors: Junlin Lu, Patrick Mannion, Karl Mason
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03997
• Pdf URL: https://arxiv.org/pdf/2404.03997
• Copy Paste: [[2404.03997]] Demonstration Guided Multi-Objective Reinforcement Learning(https://arxiv.org/abs/2404.03997)
• Keywords: robust
• Abstract: Multi-objective reinforcement learning (MORL) is increasingly relevant due to its resemblance to real-world scenarios requiring trade-offs between multiple objectives. Catering to diverse user preferences, traditional reinforcement learning faces amplified challenges in MORL. To address the difficulty of training policies from scratch in MORL, we introduce demonstration-guided multi-objective reinforcement learning (DG-MORL). This novel approach utilizes prior demonstrations, aligns them with user preferences via corner weight support, and incorporates a self-evolving mechanism to refine suboptimal demonstrations. Our empirical studies demonstrate DG-MORL's superiority over existing MORL algorithms, establishing its robustness and efficacy, particularly under challenging conditions. We also provide an upper bound of the algorithm's sample complexity.

Title: Finsler-Laplace-Beltrami Operators with Application to Shape Analysis

• Authors: Simon Weber, Thomas Dagès, Maolin Gao, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03999
• Pdf URL: https://arxiv.org/pdf/2404.03999
• Copy Paste: [[2404.03999]] Finsler-Laplace-Beltrami Operators with Application to Shape Analysis(https://arxiv.org/abs/2404.03999)
• Keywords: diffusion
• Abstract: The Laplace-Beltrami operator (LBO) emerges from studying manifolds equipped with a Riemannian metric. It is often called the Swiss army knife of geometry processing as it allows to capture intrinsic shape information and gives rise to heat diffusion, geodesic distances, and a multitude of shape descriptors. It also plays a central role in geometric deep learning. In this work, we explore Finsler manifolds as a generalization of Riemannian manifolds. We revisit the Finsler heat equation and derive a Finsler heat kernel and a Finsler-Laplace-Beltrami Operator (FLBO): a novel theoretically justified anisotropic Laplace-Beltrami operator (ALBO). In experimental evaluations we demonstrate that the proposed FLBO is a valuable alternative to the traditional Riemannian-based LBO and ALBOs for spatial filtering and shape correspondence estimation. We hope that the proposed Finsler heat kernel and the FLBO will inspire further exploration of Finsler geometry in the computer vision community.

Title: Continual Learning with Weight Interpolation

• Authors: Jędrzej Kozal, Jan Wasilewski, Bartosz Krawczyk, Michał Woźniak
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04002
• Pdf URL: https://arxiv.org/pdf/2404.04002
• Copy Paste: [[2404.04002]] Continual Learning with Weight Interpolation(https://arxiv.org/abs/2404.04002)
• Keywords: robust
• Abstract: Continual learning poses a fundamental challenge for modern machine learning systems, requiring models to adapt to new tasks while retaining knowledge from previous ones. Addressing this challenge necessitates the development of efficient algorithms capable of learning from data streams and accumulating knowledge over time. This paper proposes a novel approach to continual learning utilizing the weight consolidation method. Our method, a simple yet powerful technique, enhances robustness against catastrophic forgetting by interpolating between old and new model weights after each novel task, effectively merging two models to facilitate exploration of local minima emerging after arrival of new concepts. Moreover, we demonstrate that our approach can complement existing rehearsal-based replay approaches, improving their accuracy and further mitigating the forgetting phenomenon. Additionally, our method provides an intuitive mechanism for controlling the stability-plasticity trade-off. Experimental results showcase the significant performance enhancement to state-of-the-art experience replay algorithms the proposed weight consolidation approach offers. Our algorithm can be downloaded from https://github.com/jedrzejkozal/weight-interpolation-cl.

Title: BuDDIE: A Business Document Dataset for Multi-task Information Extraction

• Authors: Ran Zmigrod, Dongsheng Wang, Mathieu Sibue, Yulong Pei, Petr Babkin, Ivan Brugere, Xiaomo Liu, Nacho Navarro, Antony Papadimitriou, William Watson, Zhiqiang Ma, Armineh Nourbakhsh, Sameena Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04003
• Pdf URL: https://arxiv.org/pdf/2404.04003
• Copy Paste: [[2404.04003]] BuDDIE: A Business Document Dataset for Multi-task Information Extraction(https://arxiv.org/abs/2404.04003)
• Keywords: extraction, large language model
• Abstract: The field of visually rich document understanding (VRDU) aims to solve a multitude of well-researched NLP tasks in a multi-modal domain. Several datasets exist for research on specific tasks of VRDU such as document classification (DC), key entity extraction (KEE), entity linking, visual question answering (VQA), inter alia. These datasets cover documents like invoices and receipts with sparse annotations such that they support one or two co-related tasks (e.g., entity extraction and entity linking). Unfortunately, only focusing on a single specific of documents or task is not representative of how documents often need to be processed in the wild - where variety in style and requirements is expected. In this paper, we introduce BuDDIE (Business Document Dataset for Information Extraction), the first multi-task dataset of 1,665 real-world business documents that contains rich and dense annotations for DC, KEE, and VQA. Our dataset consists of publicly available business entity documents from US state government websites. The documents are structured and vary in their style and layout across states and types (e.g., forms, certificates, reports, etc.). We provide data variety and quality metrics for BuDDIE as well as a series of baselines for each task. Our baselines cover traditional textual, multi-modal, and large language model approaches to VRDU.

Title: From Theory to Comprehension: A Comparative Study of Differential Privacy and $k$-Anonymity

• Authors: Saskia Nuñez von Voigt, Luise Mehner, Florian Tschorsch
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.04006
• Pdf URL: https://arxiv.org/pdf/2404.04006
• Copy Paste: [[2404.04006]] From Theory to Comprehension: A Comparative Study of Differential Privacy and $k$-Anonymity(https://arxiv.org/abs/2404.04006)
• Keywords: privacy, protect
• Abstract: The notion of $\varepsilon$-differential privacy is a widely used concept of providing quantifiable privacy to individuals. However, it is unclear how to explain the level of privacy protection provided by a differential privacy mechanism with a set $\varepsilon$. In this study, we focus on users' comprehension of the privacy protection provided by a differential privacy mechanism. To do so, we study three variants of explaining the privacy protection provided by differential privacy: (1) the original mathematical definition; (2) $\varepsilon$ translated into a specific privacy risk; and (3) an explanation using the randomized response technique. We compare users' comprehension of privacy protection employing these explanatory models with their comprehension of privacy protection of $k$-anonymity as baseline comprehensibility. Our findings suggest that participants' comprehension of differential privacy protection is enhanced by the privacy risk model and the randomized response-based model. Moreover, our results confirm our intuition that privacy protection provided by $k$-anonymity is more comprehensible.

Title: InstructHumans: Editing Animated 3D Human Textures with Instructions

• Authors: Jiayin Zhu, Linlin Yang, Angela Yao
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.04037
• Pdf URL: https://arxiv.org/pdf/2404.04037
• Copy Paste: [[2404.04037]] InstructHumans: Editing Animated 3D Human Textures with Instructions(https://arxiv.org/abs/2404.04037)
• Keywords: diffusion, generative
• Abstract: We present InstructHumans, a novel framework for instruction-driven 3D human texture editing. Existing text-based editing methods use Score Distillation Sampling (SDS) to distill guidance from generative models. This work shows that naively using such scores is harmful to editing as they destroy consistency with the source avatar. Instead, we propose an alternate SDS for Editing (SDS-E) that selectively incorporates subterms of SDS across diffusion timesteps. We further enhance SDS-E with spatial smoothness regularization and gradient-based viewpoint sampling to achieve high-quality edits with sharp and high-fidelity detailing. InstructHumans significantly outperforms existing 3D editing methods, consistent with the initial avatar while faithful to the textual instructions. Project page: https://jyzhu.top/instruct-humans .

Title: Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer

• Authors: Hele-Andra Kuulmets, Taido Purason, Agnes Luhtaru, Mark Fishel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04042
• Pdf URL: https://arxiv.org/pdf/2404.04042
• Copy Paste: [[2404.04042]] Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer(https://arxiv.org/abs/2404.04042)
• Keywords: large language model
• Abstract: This paper explores cost-efficient methods to adapt pretrained Large Language Models (LLMs) to new lower-resource languages, with a specific focus on Estonian. Leveraging the Llama 2 model, we investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. Our results demonstrate that even a relatively small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances results on Estonian. Furthermore, we showcase cross-lingual knowledge transfer from high-quality English instructions to Estonian, resulting in improvements in commonsense reasoning and multi-turn conversation capabilities. Our best model, named \textsc{Llammas}, represents the first open-source instruction-following LLM for Estonian. Additionally, we publish Alpaca-est, the first general task instruction dataset for Estonia. These contributions mark the initial progress in the direction of developing open-source LLMs for Estonian.

Title: No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation

• Authors: Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Jiaming Liu, Han Xiao, Chaoyou Fu, Hao Dong, Peng Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04050
• Pdf URL: https://arxiv.org/pdf/2404.04050
• Copy Paste: [[2404.04050]] No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation(https://arxiv.org/abs/2404.04050)
• Keywords: segmentation
• Abstract: To reduce the reliance on large-scale datasets, recent works in 3D segmentation resort to few-shot learning. Current 3D few-shot segmentation methods first pre-train models on 'seen' classes, and then evaluate their generalization performance on 'unseen' classes. However, the prior pre-training stage not only introduces excessive time overhead but also incurs a significant domain gap on 'unseen' classes. To tackle these issues, we propose a Non-parametric Network for few-shot 3D Segmentation, Seg-NN, and its Parametric variant, Seg-PN. Without training, Seg-NN extracts dense representations by hand-crafted filters and achieves comparable performance to existing parametric models. Due to the elimination of pre-training, Seg-NN can alleviate the domain gap issue and save a substantial amount of time. Based on Seg-NN, Seg-PN only requires training a lightweight QUEry-Support Transferring (QUEST) module, which enhances the interaction between the support set and query set. Experiments suggest that Seg-PN outperforms previous state-of-the-art method by +4.19% and +7.71% mIoU on S3DIS and ScanNet datasets respectively, while reducing training time by -90%, indicating its effectiveness and efficiency.

Title: Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation

• Authors: Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, Hai Huang
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.04057
• Pdf URL: https://arxiv.org/pdf/2404.04057
• Copy Paste: [[2404.04057]] Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation(https://arxiv.org/abs/2404.04057)
• Keywords: diffusion, data-free, generative
• Abstract: We introduce Score identity Distillation (SiD), an innovative data-free method that distills the generative capabilities of pretrained diffusion models into a single-step generator. SiD not only facilitates an exponentially fast reduction in Fr\'echet inception distance (FID) during distillation but also approaches or even exceeds the FID performance of the original teacher diffusion models. By reformulating forward diffusion processes as semi-implicit distributions, we leverage three score-related identities to create an innovative loss mechanism. This mechanism achieves rapid FID reduction by training the generator using its own synthesized images, eliminating the need for real data or reverse-diffusion-based generation, all accomplished within significantly shortened generation time. Upon evaluation across four benchmark datasets, the SiD algorithm demonstrates high iteration efficiency during distillation and surpasses competing distillation approaches, whether they are one-step or few-step, data-free, or dependent on training data, in terms of generation quality. This achievement not only redefines the benchmarks for efficiency and effectiveness in diffusion distillation but also in the broader field of diffusion-based generation. Our PyTorch implementation will be publicly accessible on GitHub.

Title: CLUE: A Clinical Language Understanding Evaluation for LLMs

• Authors: Amin Dada, Marie Bauer, Amanda Butler Contreras, Osman Alperen Koraş, Constantin Marc Seibold, Kaleb E Smith, Jens Kleesiek
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04067
• Pdf URL: https://arxiv.org/pdf/2404.04067
• Copy Paste: [[2404.04067]] CLUE: A Clinical Language Understanding Evaluation for LLMs(https://arxiv.org/abs/2404.04067)
• Keywords: privacy, large language model
• Abstract: Large Language Models (LLMs) have shown the potential to significantly contribute to patient care, diagnostics, and administrative processes. Emerging biomedical LLMs address healthcare-specific challenges, including privacy demands and computational constraints. However, evaluation of these models has primarily been limited to non-clinical tasks, which do not reflect the complexity of practical clinical applications. Additionally, there has been no thorough comparison between biomedical and general-domain LLMs for clinical tasks. To fill this gap, we present the Clinical Language Understanding Evaluation (CLUE), a benchmark tailored to evaluate LLMs on real-world clinical tasks. CLUE includes two novel datasets derived from MIMIC IV discharge letters and four existing tasks designed to test the practical applicability of LLMs in healthcare settings. Our evaluation covers several biomedical and general domain LLMs, providing insights into their clinical performance and applicability. CLUE represents a step towards a standardized approach to evaluating and developing LLMs in healthcare to align future model development with the real-world needs of clinical application. We publish our evaluation and data generation scripts: https://github.com/dadaamin/CLUE

Title: Assessing the quality of information extraction

• Authors: Filip Seitl, Tomáš Kovářík, Soheyla Mirshahi, Jan Kryštůfek, Rastislav Dujava, Matúš Ondreička, Herbert Ullrich, Petr Gronat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04068
• Pdf URL: https://arxiv.org/pdf/2404.04068
• Copy Paste: [[2404.04068]] Assessing the quality of information extraction(https://arxiv.org/abs/2404.04068)
• Keywords: extraction, large language model
• Abstract: Advances in large language models have notably enhanced the efficiency of information extraction from unstructured and semi-structured data sources. As these technologies become integral to various applications, establishing an objective measure for the quality of information extraction becomes imperative. However, the scarcity of labeled data presents significant challenges to this endeavor. In this paper, we introduce an automatic framework to assess the quality of the information extraction and its completeness. The framework focuses on information extraction in the form of entity and its properties. We discuss how to handle the input/output size limitations of the large language models and analyze their performance when iteratively extracting the information. Finally, we introduce metrics to evaluate the quality of the extraction and provide an extensive discussion on how to interpret the metrics.

Title: Hierarchical Neural Additive Models for Interpretable Demand Forecasts

• Authors: Leif Feddersen, Catherine Cleophas
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.04070
• Pdf URL: https://arxiv.org/pdf/2404.04070
• Copy Paste: [[2404.04070]] Hierarchical Neural Additive Models for Interpretable Demand Forecasts(https://arxiv.org/abs/2404.04070)
• Keywords: interpretability
• Abstract: Demand forecasts are the crucial basis for numerous business decisions, ranging from inventory management to strategic facility planning. While machine learning (ML) approaches offer accuracy gains, their interpretability and acceptance are notoriously lacking. Addressing this dilemma, we introduce Hierarchical Neural Additive Models for time series (HNAM). HNAM expands upon Neural Additive Models (NAM) by introducing a time-series specific additive model with a level and interacting covariate components. Covariate interactions are only allowed according to a user-specified interaction hierarchy. For example, weekday effects may be estimated independently of other covariates, whereas a holiday effect may depend on the weekday and an additional promotion may depend on both former covariates that are lower in the interaction hierarchy. Thereby, HNAM yields an intuitive forecasting interface in which analysts can observe the contribution for each known covariate. We evaluate the proposed approach and benchmark its performance against other state-of-the-art machine learning and statistical models extensively on real-world retail data. The results reveal that HNAM offers competitive prediction performance whilst providing plausible explanations.

Title: Dynamic Prompt Optimizing for Text-to-Image Generation

• Authors: Wenyi Mo, Tianyu Zhang, Yalong Bai, Bing Su, Ji-Rong Wen, Qing Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04095
• Pdf URL: https://arxiv.org/pdf/2404.04095
• Copy Paste: [[2404.04095]] Dynamic Prompt Optimizing for Text-to-Image Generation(https://arxiv.org/abs/2404.04095)
• Keywords: diffusion, generative
• Abstract: Text-to-image generative models, specifically those based on diffusion models like Imagen and Stable Diffusion, have made substantial advancements. Recently, there has been a surge of interest in the delicate refinement of text prompts. Users assign weights or alter the injection time steps of certain words in the text prompts to improve the quality of generated images. However, the success of fine-control prompts depends on the accuracy of the text prompts and the careful selection of weights and time steps, which requires significant manual intervention. To address this, we introduce the \textbf{P}rompt \textbf{A}uto-\textbf{E}diting (PAE) method. Besides refining the original prompts for image generation, we further employ an online reinforcement learning strategy to explore the weights and injection time steps of each word, leading to the dynamic fine-control prompts. The reward function during training encourages the model to consider aesthetic score, semantic consistency, and user preferences. Experimental results demonstrate that our proposed method effectively improves the original prompts, generating visually more appealing images while maintaining semantic alignment. Code is available at https://github.com/Mowenyii/PAE.

Title: You Can Use But Cannot Recognize: Preserving Visual Privacy in Deep Neural Networks

• Authors: Qiushi Li, Yan Zhang, Ju Ren, Qi Li, Yaoxue Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04098
• Pdf URL: https://arxiv.org/pdf/2404.04098
• Copy Paste: [[2404.04098]] You Can Use But Cannot Recognize: Preserving Visual Privacy in Deep Neural Networks(https://arxiv.org/abs/2404.04098)
• Keywords: privacy, protect
• Abstract: Image data have been extensively used in Deep Neural Network (DNN) tasks in various scenarios, e.g., autonomous driving and medical image analysis, which incurs significant privacy concerns. Existing privacy protection techniques are unable to efficiently protect such data. For example, Differential Privacy (DP) that is an emerging technique protects data with strong privacy guarantee cannot effectively protect visual features of exposed image dataset. In this paper, we propose a novel privacy-preserving framework VisualMixer that protects the training data of visual DNN tasks by pixel shuffling, while not injecting any noises. VisualMixer utilizes a new privacy metric called Visual Feature Entropy (VFE) to effectively quantify the visual features of an image from both biological and machine vision aspects. In VisualMixer, we devise a task-agnostic image obfuscation method to protect the visual privacy of data for DNN training and inference. For each image, it determines regions for pixel shuffling in the image and the sizes of these regions according to the desired VFE. It shuffles pixels both in the spatial domain and in the chromatic channel space in the regions without injecting noises so that it can prevent visual features from being discerned and recognized, while incurring negligible accuracy loss. Extensive experiments on real-world datasets demonstrate that VisualMixer can effectively preserve the visual privacy with negligible accuracy loss, i.e., at average 2.35 percentage points of model accuracy loss, and almost no performance degradation on model training.

Title: Robust Preference Optimization with Provable Noise Tolerance for LLMs

• Authors: Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, Jieping Ye
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04102
• Pdf URL: https://arxiv.org/pdf/2404.04102
• Copy Paste: [[2404.04102]] Robust Preference Optimization with Provable Noise Tolerance for LLMs(https://arxiv.org/abs/2404.04102)
• Keywords: robust, large language model
• Abstract: The preference alignment aims to enable large language models (LLMs) to generate responses that conform to human values, which is essential for developing general AI systems. Ranking-based methods -- a promising class of alignment approaches -- learn human preferences from datasets containing response pairs by optimizing the log-likelihood margins between preferred and dis-preferred responses. However, due to the inherent differences in annotators' preferences, ranking labels of comparisons for response pairs are unavoidably noisy. This seriously hurts the reliability of existing ranking-based methods. To address this problem, we propose a provably noise-tolerant preference alignment method, namely RObust Preference Optimization (ROPO). To the best of our knowledge, ROPO is the first preference alignment method with noise-tolerance guarantees. The key idea of ROPO is to dynamically assign conservative gradient weights to response pairs with high label uncertainty, based on the log-likelihood margins between the responses. By effectively suppressing the gradients of noisy samples, our weighting strategy ensures that the expected risk has the same gradient direction independent of the presence and proportion of noise. Experiments on three open-ended text generation tasks with four base models ranging in size from 2.8B to 13B demonstrate that ROPO significantly outperforms existing ranking-based methods.

Title: GNNBENCH: Fair and Productive Benchmarking for Single-GPU GNN System

• Authors: Yidong Gong, Pradeep Kumar
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.04118
• Pdf URL: https://arxiv.org/pdf/2404.04118
• Copy Paste: [[2404.04118]] GNNBENCH: Fair and Productive Benchmarking for Single-GPU GNN System(https://arxiv.org/abs/2404.04118)
• Keywords: fair
• Abstract: We hypothesize that the absence of a standardized benchmark has allowed several fundamental pitfalls in GNN System design and evaluation that the community has overlooked. In this work, we propose GNNBench, a plug-and-play benchmarking platform focused on system innovation. GNNBench presents a new protocol to exchange their captive tensor data, supports custom classes in System APIs, and allows automatic integration of the same system module to many deep learning frameworks, such as PyTorch and TensorFlow. To demonstrate the importance of such a benchmark framework, we integrated several GNN systems. Our results show that integration with GNNBench helped us identify several measurement issues that deserve attention from the community.

Title: No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

• Authors: Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H.S. Torr, Adel Bibi, Samuel Albanie, Matthias Bethge
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04125
• Pdf URL: https://arxiv.org/pdf/2404.04125
• Copy Paste: [[2404.04125]] No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance(https://arxiv.org/abs/2404.04125)
• Keywords: diffusion
• Abstract: Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation. However, it is unclear how meaningful the notion of "zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during "zero-shot" evaluation. In this work, we ask: How is the performance of multimodal models on downstream concepts influenced by the frequency of these concepts in their pretraining datasets? We comprehensively investigate this question across 34 models and five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M, LAION-Aesthetics), generating over 300GB of data artifacts. We consistently find that, far from exhibiting "zero-shot" generalization, multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets, and testing on purely synthetic data distributions. Furthermore, upon benchmarking models on long-tailed data sampled based on our analysis, we demonstrate that multimodal models across the board perform poorly. We contribute this long-tail test set as the "Let it Wag!" benchmark to further research in this direction. Taken together, our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.

Title: Smart Contract Languages: a comparative analysis

• Authors: Massimo Bartoletti, Lorenzo Benetollo, Michele Bugliesi, Silvia Crafa, Giacomo Dal Sasso, Roberto Pettinau, Andrea Pinna, Mattia Piras, Sabina Rossi, Stefano Salis, Alvise Spanò, Viacheslav Tkachenko, Roberto Tonelli, Roberto Zunino
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2404.04129
• Pdf URL: https://arxiv.org/pdf/2404.04129
• Copy Paste: [[2404.04129]] Smart Contract Languages: a comparative analysis(https://arxiv.org/abs/2404.04129)
• Keywords: secure, security
• Abstract: Decentralized blockchain platforms support the secure exchange of assets among users without relying on trusted third parties. These exchanges are programmed with smart contracts, computer programs directly executed by blockchain nodes. Multiple smart contract languages are available nowadays to developers, each with its own distinctive features, strengths, and weaknesses. In this paper, we examine the smart contract languages used in six major blockchain platforms: Ethereum, Solana, Cardano, Algorand, Aptos, and Tezos. Starting with a high-level overview of their design choices, we provide a comprehensive assessment that focuses on programming style, security, code readability, and usability, drawing on an original benchmark that encompasses a common set of use cases across all the smart contract languages under examination.

Title: Precision Guided Approach to Mitigate Data Poisoning Attacks in Federated Learning

• Authors: K Naveen Kumar, C Krishna Mohan, Aravind Machiry
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04139
• Pdf URL: https://arxiv.org/pdf/2404.04139
• Copy Paste: [[2404.04139]] Precision Guided Approach to Mitigate Data Poisoning Attacks in Federated Learning(https://arxiv.org/abs/2404.04139)
• Keywords: privacy, defense, attack, robust, federate
• Abstract: Federated Learning (FL) is a collaborative learning paradigm enabling participants to collectively train a shared machine learning model while preserving the privacy of their sensitive data. Nevertheless, the inherent decentralized and data-opaque characteristics of FL render its susceptibility to data poisoning attacks. These attacks introduce malformed or malicious inputs during local model training, subsequently influencing the global model and resulting in erroneous predictions. Current FL defense strategies against data poisoning attacks either involve a trade-off between accuracy and robustness or necessitate the presence of a uniformly distributed root dataset at the server. To overcome these limitations, we present FedZZ, which harnesses a zone-based deviating update (ZBDU) mechanism to effectively counter data poisoning attacks in FL. Further, we introduce a precision-guided methodology that actively characterizes these client clusters (zones), which in turn aids in recognizing and discarding malicious updates at the server. Our evaluation of FedZZ across two widely recognized datasets: CIFAR10 and EMNIST, demonstrate its efficacy in mitigating data poisoning attacks, surpassing the performance of prevailing state-of-the-art methodologies in both single and multi-client attack scenarios and varying attack volumes. Notably, FedZZ also functions as a robust client selection strategy, even in highly non-IID and attack-free scenarios. Moreover, in the face of escalating poisoning rates, the model accuracy attained by FedZZ displays superior resilience compared to existing techniques. For instance, when confronted with a 50% presence of malicious clients, FedZZ sustains an accuracy of 67.43%, while the accuracy of the second-best solution, FL-Defender, diminishes to 43.36%.

Title: Improving Detection in Aerial Images by Capturing Inter-Object Relationships

• Authors: Botao Ren, Botian Xu, Yifan Pu, Jingyi Wang, Zhidong Deng
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04140
• Pdf URL: https://arxiv.org/pdf/2404.04140
• Copy Paste: [[2404.04140]] Improving Detection in Aerial Images by Capturing Inter-Object Relationships(https://arxiv.org/abs/2404.04140)
• Keywords: transformer
• Abstract: In many image domains, the spatial distribution of objects in a scene exhibits meaningful patterns governed by their semantic relationships. In most modern detection pipelines, however, the detection proposals are processed independently, overlooking the underlying relationships between objects. In this work, we introduce a transformer-based approach to capture these inter-object relationships to refine classification and regression outcomes for detected objects. Building on two-stage detectors, we tokenize the region of interest (RoI) proposals to be processed by a transformer encoder. Specific spatial and geometric relations are incorporated into the attention weights and adaptively modulated and regularized. Experimental results demonstrate that the proposed method achieves consistent performance improvement on three benchmarks including DOTA-v1.0, DOTA-v1.5, and HRSC 2016, especially ranking first on both DOTA-v1.5 and HRSC 2016. Specifically, our new method has an increase of 1.59 mAP on DOTA-v1.0, 4.88 mAP on DOTA-v1.5, and 2.1 mAP on HRSC 2016, respectively, compared to the baselines.

Title: MarsSeg: Mars Surface Semantic Segmentation with Multi-level Extractor and Connector

• Authors: Junbo Li, Keyan Chen, Gengju Tian, Lu Li, Zhenwei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04155
• Pdf URL: https://arxiv.org/pdf/2404.04155
• Copy Paste: [[2404.04155]] MarsSeg: Mars Surface Semantic Segmentation with Multi-level Extractor and Connector(https://arxiv.org/abs/2404.04155)
• Keywords: extraction, segmentation
• Abstract: The segmentation and interpretation of the Martian surface play a pivotal role in Mars exploration, providing essential data for the trajectory planning and obstacle avoidance of rovers. However, the complex topography, similar surface features, and the lack of extensive annotated data pose significant challenges to the high-precision semantic segmentation of the Martian surface. To address these challenges, we propose a novel encoder-decoder based Mars segmentation network, termed MarsSeg. Specifically, we employ an encoder-decoder structure with a minimized number of down-sampling layers to preserve local details. To facilitate a high-level semantic understanding across the shadow multi-level feature maps, we introduce a feature enhancement connection layer situated between the encoder and decoder. This layer incorporates Mini Atrous Spatial Pyramid Pooling (Mini-ASPP), Polarized Self-Attention (PSA), and Strip Pyramid Pooling Module (SPPM). The Mini-ASPP and PSA are specifically designed for shadow feature enhancement, thereby enabling the expression of local details and small objects. Conversely, the SPPM is employed for deep feature enhancement, facilitating the extraction of high-level semantic category-related information. Experimental results derived from the Mars-Seg and AI4Mars datasets substantiate that the proposed MarsSeg outperforms other state-of-the-art methods in segmentation performance, validating the efficacy of each proposed component.

Title: Noisy Label Processing for Classification: A Survey

• Authors: Mengting Li, Chuang Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04159
• Pdf URL: https://arxiv.org/pdf/2404.04159
• Copy Paste: [[2404.04159]] Noisy Label Processing for Classification: A Survey(https://arxiv.org/abs/2404.04159)
• Keywords: robust
• Abstract: In recent years, deep neural networks (DNNs) have gained remarkable achievement in computer vision tasks, and the success of DNNs often depends greatly on the richness of data. However, the acquisition process of data and high-quality ground truth requires a lot of manpower and money. In the long, tedious process of data annotation, annotators are prone to make mistakes, resulting in incorrect labels of images, i.e., noisy labels. The emergence of noisy labels is inevitable. Moreover, since research shows that DNNs can easily fit noisy labels, the existence of noisy labels will cause significant damage to the model training process. Therefore, it is crucial to combat noisy labels for computer vision tasks, especially for classification tasks. In this survey, we first comprehensively review the evolution of different deep learning approaches for noisy label combating in the image classification task. In addition, we also review different noise patterns that have been proposed to design robust algorithms. Furthermore, we explore the inner pattern of real-world label noise and propose an algorithm to generate a synthetic label noise pattern guided by real-world data. We test the algorithm on the well-known real-world dataset CIFAR-10N to form a new real-world data-guided synthetic benchmark and evaluate some typical noise-robust methods on the benchmark.

Title: Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

• Authors: Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Binhang Yuan, Wenhu Chen, Jie Fu, Ge Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04167
• Pdf URL: https://arxiv.org/pdf/2404.04167
• Copy Paste: [[2404.04167]] Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model(https://arxiv.org/abs/2404.04167)
• Keywords: large language model
• Abstract: In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remarkable performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including a detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.

Title: Do Sentence Transformers Learn Quasi-Geospatial Concepts from General Text?

• Authors: Ilya Ilyankou, Aldo Lipani, Stefano Cavazzi, Xiaowei Gao, James Haworth
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04169
• Pdf URL: https://arxiv.org/pdf/2404.04169
• Copy Paste: [[2404.04169]] Do Sentence Transformers Learn Quasi-Geospatial Concepts from General Text?(https://arxiv.org/abs/2404.04169)
• Keywords: transformer
• Abstract: Sentence transformers are language models designed to perform semantic search. This study investigates the capacity of sentence transformers, fine-tuned on general question-answering datasets for asymmetric semantic search, to associate descriptions of human-generated routes across Great Britain with queries often used to describe hiking experiences. We find that sentence transformers have some zero-shot capabilities to understand quasi-geospatial concepts, such as route types and difficulty, suggesting their potential utility for routing recommendation systems.

Title: Reliable Feature Selection for Adversarially Robust Cyber-Attack Detection

• Authors: João Vitorino, Miguel Silva, Eva Maia, Isabel Praça
• Subjects: cs.CR, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.04188
• Pdf URL: https://arxiv.org/pdf/2404.04188
• Copy Paste: [[2404.04188]] Reliable Feature Selection for Adversarially Robust Cyber-Attack Detection(https://arxiv.org/abs/2404.04188)
• Keywords: security, attack, robust
• Abstract: The growing cybersecurity threats make it essential to use high-quality data to train Machine Learning (ML) models for network traffic analysis, without noisy or missing data. By selecting the most relevant features for cyber-attack detection, it is possible to improve both the robustness and computational efficiency of the models used in a cybersecurity system. This work presents a feature selection and consensus process that combines multiple methods and applies them to several network datasets. Two different feature sets were selected and were used to train multiple ML models with regular and adversarial training. Finally, an adversarial evasion robustness benchmark was performed to analyze the reliability of the different feature sets and their impact on the susceptibility of the models to adversarial examples. By using an improved dataset with more data diversity, selecting the best time-related features and a more specific feature set, and performing adversarial training, the ML models were able to achieve a better adversarially robust generalization. The robustness of the models was significantly improved without their generalization to regular traffic flows being affected, without increases of false alarms, and without requiring too many computational resources, which enables a reliable detection of suspicious activity and perturbed traffic flows in enterprise computer networks.

Title: Social Skill Training with Large Language Models

• Authors: Diyi Yang, Caleb Ziems, William Held, Omar Shaikh, Michael S. Bernstein, John Mitchell
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.04204
• Pdf URL: https://arxiv.org/pdf/2404.04204
• Copy Paste: [[2404.04204]] Social Skill Training with Large Language Models(https://arxiv.org/abs/2404.04204)
• Keywords: large language model
• Abstract: People rely on social skills like conflict resolution to communicate effectively and to thrive in both work and personal life. However, practice environments for social skills are typically out of reach for most people. How can we make social skill training more available, accessible, and inviting? Drawing upon interdisciplinary research from communication and psychology, this perspective paper identifies social skill barriers to enter specialized fields. Then we present a solution that leverages large language models for social skill training via a generic framework. Our AI Partner, AI Mentor framework merges experiential learning with realistic practice and tailored feedback. This work ultimately calls for cross-disciplinary innovation to address the broader implications for workforce development and social equality.

Title: Enhancing IoT Intelligence: A Transformer-based Reinforcement Learning Methodology

• Authors: Gaith Rjoub, Saidul Islam, Jamal Bentahar, Mohammed Amin Almaiah, Rana Alrawashdeh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04205
• Pdf URL: https://arxiv.org/pdf/2404.04205
• Copy Paste: [[2404.04205]] Enhancing IoT Intelligence: A Transformer-based Reinforcement Learning Methodology(https://arxiv.org/abs/2404.04205)
• Keywords: transformer
• Abstract: The proliferation of the Internet of Things (IoT) has led to an explosion of data generated by interconnected devices, presenting both opportunities and challenges for intelligent decision-making in complex environments. Traditional Reinforcement Learning (RL) approaches often struggle to fully harness this data due to their limited ability to process and interpret the intricate patterns and dependencies inherent in IoT applications. This paper introduces a novel framework that integrates transformer architectures with Proximal Policy Optimization (PPO) to address these challenges. By leveraging the self-attention mechanism of transformers, our approach enhances RL agents' capacity for understanding and acting within dynamic IoT environments, leading to improved decision-making processes. We demonstrate the effectiveness of our method across various IoT scenarios, from smart home automation to industrial control systems, showing marked improvements in decision-making efficiency and adaptability. Our contributions include a detailed exploration of the transformer's role in processing heterogeneous IoT data, a comprehensive evaluation of the framework's performance in diverse environments, and a benchmark against traditional RL methods. The results indicate significant advancements in enabling RL agents to navigate the complexities of IoT ecosystems, highlighting the potential of our approach to revolutionize intelligent automation and decision-making in the IoT landscape.

Title: Robust Gaussian Splatting

• Authors: François Darmon, Lorenzo Porzi, Samuel Rota-Bulò, Peter Kontschieder
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04211
• Pdf URL: https://arxiv.org/pdf/2404.04211
• Copy Paste: [[2404.04211]] Robust Gaussian Splatting(https://arxiv.org/abs/2404.04211)
• Keywords: robust
• Abstract: In this paper, we address common error sources for 3D Gaussian Splatting (3DGS) including blur, imperfect camera poses, and color inconsistencies, with the goal of improving its robustness for practical applications like reconstructions from handheld phone captures. Our main contribution involves modeling motion blur as a Gaussian distribution over camera poses, allowing us to address both camera pose refinement and motion blur correction in a unified way. Additionally, we propose mechanisms for defocus blur compensation and for addressing color in-consistencies caused by ambient light, shadows, or due to camera-related factors like varying white balancing settings. Our proposed solutions integrate in a seamless way with the 3DGS formulation while maintaining its benefits in terms of training efficiency and rendering speed. We experimentally validate our contributions on relevant benchmark datasets including Scannet++ and Deblur-NeRF, obtaining state-of-the-art results and thus consistent improvements over relevant baselines.

Title: Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation

• Authors: Ji-Jia Wu, Andy Chia-Hao Chang, Chieh-Yu Chuang, Chun-Pei Chen, Yu-Lun Liu, Min-Hung Chen, Hou-Ning Hu, Yung-Yu Chuang, Yen-Yu Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04231
• Pdf URL: https://arxiv.org/pdf/2404.04231
• Copy Paste: [[2404.04231]] Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation(https://arxiv.org/abs/2404.04231)
• Keywords: segmentation
• Abstract: This paper addresses text-supervised semantic segmentation, aiming to learn a model capable of segmenting arbitrary visual concepts within images by using only image-text pairs without dense annotations. Existing methods have demonstrated that contrastive learning on image-text pairs effectively aligns visual segments with the meanings of texts. We notice that there is a discrepancy between text alignment and semantic segmentation: A text often consists of multiple semantic concepts, whereas semantic segmentation strives to create semantically homogeneous segments. To address this issue, we propose a novel framework, Image-Text Co-Decomposition (CoDe), where the paired image and text are jointly decomposed into a set of image regions and a set of word segments, respectively, and contrastive learning is developed to enforce region-word alignment. To work with a vision-language model, we present a prompt learning mechanism that derives an extra representation to highlight an image segment or a word segment of interest, with which more effective features can be extracted from that segment. Comprehensive experimental results demonstrate that our method performs favorably against existing text-supervised semantic segmentation methods on six benchmark datasets.

Title: player2vec: A Language Modeling Approach to Understand Player Behavior in Games

• Authors: Tianze Wang, Maryam Honari-Jahromi, Styliani Katsarou, Olga Mikheeva, Theodoros Panagiotakopoulos, Sahar Asadi, Oleg Smirnov
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04234
• Pdf URL: https://arxiv.org/pdf/2404.04234
• Copy Paste: [[2404.04234]] player2vec: A Language Modeling Approach to Understand Player Behavior in Games(https://arxiv.org/abs/2404.04234)
• Keywords: transformer
• Abstract: Methods for learning latent user representations from historical behavior logs have gained traction for recommendation tasks in e-commerce, content streaming, and other settings. However, this area still remains relatively underexplored in video and mobile gaming contexts. In this work, we present a novel method for overcoming this limitation by extending a long-range Transformer model from the natural language processing domain to player behavior data. We discuss specifics of behavior tracking in games and propose preprocessing and tokenization approaches by viewing in-game events in an analogous way to words in sentences, thus enabling learning player representations in a self-supervised manner in the absence of ground-truth annotations. We experimentally demonstrate the efficacy of the proposed approach in fitting the distribution of behavior events by evaluating intrinsic language modeling metrics. Furthermore, we qualitatively analyze the emerging structure of the learned embedding space and show its value for generating insights into behavior patterns to inform downstream applications.

Title: Cleared for Takeoff? Compositional & Conditional Reasoning may be the Achilles Heel to (Flight-Booking) Language Agents

• Authors: Harsh Kohli, Huan Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04237
• Pdf URL: https://arxiv.org/pdf/2404.04237
• Copy Paste: [[2404.04237]] Cleared for Takeoff? Compositional & Conditional Reasoning may be the Achilles Heel to (Flight-Booking) Language Agents(https://arxiv.org/abs/2404.04237)
• Keywords: large language model
• Abstract: The rapid progress of large language models (LLMs) has seen them excel and frequently surpass human performance on standard benchmarks. This has enabled many downstream applications, such as LLM agents, to rely on their sophisticated reasoning to navigate complex task requirements. However, LLMs are known to unexpectedly falter in simple tasks and under seemingly straightforward circumstances - underscoring the need for better and more diverse evaluation setups to measure their true capabilities. To this end, we choose to study compositional and conditional reasoning, two cornerstones of human cognition, and introduce GroundCocoa - a lexically diverse benchmark connecting these reasoning skills to the real-world problem of flight booking. Our task involves aligning detailed user preferences with available flight options presented in a multiple-choice format. Results indicate a significant disparity in performance among current state-of-the-art LLMs with even the best performing model, GPT-4 Turbo, not exceeding 67% accuracy despite advanced prompting techniques.

Title: Dynamic Conditional Optimal Transport through Simulation-Free Flows

• Authors: Gavin Kerrigan, Giosue Migliorini, Padhraic Smyth
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04240
• Pdf URL: https://arxiv.org/pdf/2404.04240
• Copy Paste: [[2404.04240]] Dynamic Conditional Optimal Transport through Simulation-Free Flows(https://arxiv.org/abs/2404.04240)
• Keywords: generative
• Abstract: We study the geometry of conditional optimal transport (COT) and prove a dynamical formulation which generalizes the Benamou-Brenier Theorem. With these tools, we propose a simulation-free flow-based method for conditional generative modeling. Our method couples an arbitrary source distribution to a specified target distribution through a triangular COT plan. We build on the framework of flow matching to train a conditional generative model by approximating the geodesic path of measures induced by this COT plan. Our theory and methods are applicable in the infinite-dimensional setting, making them well suited for inverse problems. Empirically, we demonstrate our proposed method on two image-to-image translation tasks and an infinite-dimensional Bayesian inverse problem.

Title: Physical Property Understanding from Language-Embedded Feature Fields

• Authors: Albert J. Zhai, Yuan Shen, Emily Y. Chen, Gloria X. Wang, Xinlei Wang, Sheng Wang, Kaiyu Guan, Shenlong Wang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04242
• Pdf URL: https://arxiv.org/pdf/2404.04242
• Copy Paste: [[2404.04242]] Physical Property Understanding from Language-Embedded Feature Fields(https://arxiv.org/abs/2404.04242)
• Keywords: large language model
• Abstract: Can computers perceive the physical properties of objects solely through vision? Research in cognitive science and vision science has shown that humans excel at identifying materials and estimating their physical properties based purely on visual appearance. In this paper, we present a novel approach for dense prediction of the physical properties of objects using a collection of images. Inspired by how humans reason about physics through vision, we leverage large language models to propose candidate materials for each object. We then construct a language-embedded point cloud and estimate the physical properties of each 3D point using a zero-shot kernel regression approach. Our method is accurate, annotation-free, and applicable to any object in the open world. Experiments demonstrate the effectiveness of the proposed approach in various physical property reasoning tasks, such as estimating the mass of common objects, as well as other properties like friction and hardness.

Title: Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models

• Authors: Sangwon Jang, Jaehyeong Jo, Kimin Lee, Sung Ju Hwang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04243
• Pdf URL: https://arxiv.org/pdf/2404.04243
• Copy Paste: [[2404.04243]] Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models(https://arxiv.org/abs/2404.04243)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models have shown remarkable success in generating a personalized subject based on a few reference images. However, current methods struggle with handling multiple subjects simultaneously, often resulting in mixed identities with combined attributes from different subjects. In this work, we present MuDI, a novel framework that enables multi-subject personalization by effectively decoupling identities from multiple subjects. Our main idea is to utilize segmented subjects generated by the Segment Anything Model for both training and inference, as a form of data augmentation for training and initialization for the generation process. Our experiments demonstrate that MuDI can produce high-quality personalized images without identity mixing, even for highly similar subjects as shown in Figure 1. In human evaluation, MuDI shows twice as many successes for personalizing multiple subjects without identity mixing over existing baselines and is preferred over 70% compared to the strongest baseline. More results are available at https://mudi-t2i.github.io/.

Title: Evaluating Adversarial Robustness: A Comparison Of FGSM, Carlini-Wagner Attacks, And The Role of Distillation as Defense Mechanism

• Authors: Trilokesh Ranjan Sarkar, Nilanjan Das, Pralay Sankar Maitra, Bijoy Some, Ritwik Saha, Orijita Adhikary, Bishal Bose, Jaydip Sen
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04245
• Pdf URL: https://arxiv.org/pdf/2404.04245
• Copy Paste: [[2404.04245]] Evaluating Adversarial Robustness: A Comparison Of FGSM, Carlini-Wagner Attacks, And The Role of Distillation as Defense Mechanism(https://arxiv.org/abs/2404.04245)
• Keywords: defense, attack, robust
• Abstract: This technical report delves into an in-depth exploration of adversarial attacks specifically targeted at Deep Neural Networks (DNNs) utilized for image classification. The study also investigates defense mechanisms aimed at bolstering the robustness of machine learning models. The research focuses on comprehending the ramifications of two prominent attack methodologies: the Fast Gradient Sign Method (FGSM) and the Carlini-Wagner (CW) approach. These attacks are examined concerning three pre-trained image classifiers: Resnext50_32x4d, DenseNet-201, and VGG-19, utilizing the Tiny-ImageNet dataset. Furthermore, the study proposes the robustness of defensive distillation as a defense mechanism to counter FGSM and CW attacks. This defense mechanism is evaluated using the CIFAR-10 dataset, where CNN models, specifically resnet101 and Resnext50_32x4d, serve as the teacher and student models, respectively. The proposed defensive distillation model exhibits effectiveness in thwarting attacks such as FGSM. However, it is noted to remain susceptible to more sophisticated techniques like the CW attack. The document presents a meticulous validation of the proposed scheme. It provides detailed and comprehensive results, elucidating the efficacy and limitations of the defense mechanisms employed. Through rigorous experimentation and analysis, the study offers insights into the dynamics of adversarial attacks on DNNs, as well as the effectiveness of defensive strategies in mitigating their impact.

Title: Watermark-based Detection and Attribution of AI-Generated Content

• Authors: Zhengyuan Jiang, Moyang Guo, Yuepeng Hu, Neil Zhenqiang Gong
• Subjects: cs.CR, cs.AI, cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04254
• Pdf URL: https://arxiv.org/pdf/2404.04254
• Copy Paste: [[2404.04254]] Watermark-based Detection and Attribution of AI-Generated Content(https://arxiv.org/abs/2404.04254)
• Keywords: robust, watermark, generative
• Abstract: Several companies--such as Google, Microsoft, and OpenAI--have deployed techniques to watermark AI-generated content to enable proactive detection. However, existing literature mainly focuses on user-agnostic detection. Attribution aims to further trace back the user of a generative-AI service who generated a given content detected as AI-generated. Despite its growing importance, attribution is largely unexplored. In this work, we aim to bridge this gap by providing the first systematic study on watermark-based, user-aware detection and attribution of AI-generated content. Specifically, we theoretically study the detection and attribution performance via rigorous probabilistic analysis. Moreover, we develop an efficient algorithm to select watermarks for the users to enhance attribution performance. Both our theoretical and empirical results show that watermark-based detection and attribution inherit the accuracy and (non-)robustness properties of the watermarking method.

Title: Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation

• Authors: Zifu Wan, Yuhao Wang, Silong Yong, Pingping Zhang, Simon Stepputtis, Katia Sycara, Yaqi Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04256
• Pdf URL: https://arxiv.org/pdf/2404.04256
• Copy Paste: [[2404.04256]] Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation(https://arxiv.org/abs/2404.04256)
• Keywords: robust, transformer, segmentation
• Abstract: Multi-modal semantic segmentation significantly enhances AI agents' perception and scene understanding, especially under adverse conditions like low-light or overexposed environments. Leveraging additional modalities (X-modality) like thermal and depth alongside traditional RGB provides complementary information, enabling more robust and reliable segmentation. In this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic segmentation, utilizing the Selective Structured State Space Model, Mamba. Unlike conventional methods that rely on CNNs, with their limited local receptive fields, or Vision Transformers (ViTs), which offer global receptive fields at the cost of quadratic complexity, our model achieves global receptive fields coverage with linear complexity. By employing a Siamese encoder and innovating a Mamba fusion mechanism, we effectively select essential information from different modalities. A decoder is then developed to enhance the channel-wise modeling ability of the model. Our method, Sigma, is rigorously evaluated on both RGB-Thermal and RGB-Depth segmentation tasks, demonstrating its superiority and marking the first successful application of State Space Models (SSMs) in multi-modal perception tasks. Code is available at https://github.com/zifuwan/Sigma.