2024-03-25

Title: Towards a Framework for Deep Learning Certification in Safety-Critical Applications Using Inherently Safe Design and Run-Time Error Detection

• Authors: Romeo Valentin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14678
• Pdf URL: https://arxiv.org/pdf/2403.14678
• Copy Paste: [[2403.14678]] Towards a Framework for Deep Learning Certification in Safety-Critical Applications Using Inherently Safe Design and Run-Time Error Detection(https://arxiv.org/abs/2403.14678)
• Keywords: attack, robust
• Abstract: Although an ever-growing number of applications employ deep learning based systems for prediction, decision-making, or state estimation, almost no certification processes have been established that would allow such systems to be deployed in safety-critical applications. In this work we consider real-world problems arising in aviation and other safety-critical areas, and investigate their requirements for a certified model. To this end, we investigate methodologies from the machine learning research community aimed towards verifying robustness and reliability of deep learning systems, and evaluate these methodologies with regard to their applicability to real-world problems. Then, we establish a new framework towards deep learning certification based on (i) inherently safe design, and (ii) run-time error detection. Using a concrete use case from aviation, we show how deep learning models can recover disentangled variables through the use of weakly-supervised representation learning. We argue that such a system design is inherently less prone to common model failures, and can be verified to encode underlying mechanisms governing the data. Then, we investigate four techniques related to the run-time safety of a model, namely (i) uncertainty quantification, (ii) out-of-distribution detection, (iii) feature collapse, and (iv) adversarial attacks. We evaluate each for their applicability and formulate a set of desiderata that a certified model should fulfill. Finally, we propose a novel model structure that exhibits all desired properties discussed in this work, and is able to make regression and uncertainty predictions, as well as detect out-of-distribution inputs, while requiring no regression labels to train. We conclude with a discussion of the current state and expected future progress of deep learning certification, and its industrial and social implications.

Title: Deep Generative Domain Adaptation with Temporal Relation Knowledge for Cross-User Activity Recognition

• Authors: Xiaozhou Ye, Kevin I-Kai Wang
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14682
• Pdf URL: https://arxiv.org/pdf/2403.14682
• Copy Paste: [[2403.14682]] Deep Generative Domain Adaptation with Temporal Relation Knowledge for Cross-User Activity Recognition(https://arxiv.org/abs/2403.14682)
• Keywords: generative
• Abstract: In human activity recognition (HAR), the assumption that training and testing data are independent and identically distributed (i.i.d.) often fails, particularly in cross-user scenarios where data distributions vary significantly. This discrepancy highlights the limitations of conventional domain adaptation methods in HAR, which typically overlook the inherent temporal relations in time-series data. To bridge this gap, our study introduces a Conditional Variational Autoencoder with Universal Sequence Mapping (CVAE-USM) approach, which addresses the unique challenges of time-series domain adaptation in HAR by relaxing the i.i.d. assumption and leveraging temporal relations to align data distributions effectively across different users. This method combines the strengths of Variational Autoencoder (VAE) and Universal Sequence Mapping (USM) to capture and utilize common temporal patterns between users for improved activity recognition. Our results, evaluated on two public HAR datasets (OPPT and PAMAP2), demonstrate that CVAE-USM outperforms existing state-of-the-art methods, offering a more accurate and generalizable solution for cross-user activity recognition.

Title: FOCIL: Finetune-and-Freeze for Online Class Incremental Learning by Training Randomly Pruned Sparse Experts

• Authors: Murat Onur Yildirim, Elif Ceren Gok Yildirim, Decebal Constantin Mocanu, Joaquin Vanschoren
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14684
• Pdf URL: https://arxiv.org/pdf/2403.14684
• Copy Paste: [[2403.14684]] FOCIL: Finetune-and-Freeze for Online Class Incremental Learning by Training Randomly Pruned Sparse Experts(https://arxiv.org/abs/2403.14684)
• Keywords: privacy
• Abstract: Class incremental learning (CIL) in an online continual learning setting strives to acquire knowledge on a series of novel classes from a data stream, using each data point only once for training. This is more realistic compared to offline modes, where it is assumed that all data from novel class(es) is readily available. Current online CIL approaches store a subset of the previous data which creates heavy overhead costs in terms of both memory and computation, as well as privacy issues. In this paper, we propose a new online CIL approach called FOCIL. It fine-tunes the main architecture continually by training a randomly pruned sparse subnetwork for each task. Then, it freezes the trained connections to prevent forgetting. FOCIL also determines the sparsity level and learning rate per task adaptively and ensures (almost) zero forgetting across all tasks without storing any replay data. Experimental results on 10-Task CIFAR100, 20-Task CIFAR100, and 100-Task TinyImagenet, demonstrate that our method outperforms the SOTA by a large margin. The code is publicly available at https://github.com/muratonuryildirim/FOCIL.

Title: Cyclical Log Annealing as a Learning Rate Scheduler

• Authors: Philip Naveen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14685
• Pdf URL: https://arxiv.org/pdf/2403.14685
• Copy Paste: [[2403.14685]] Cyclical Log Annealing as a Learning Rate Scheduler(https://arxiv.org/abs/2403.14685)
• Keywords: transformer, generative
• Abstract: A learning rate scheduler is a predefined set of instructions for varying search stepsizes during model training processes. This paper introduces a new logarithmic method using harsh restarting of step sizes through stochastic gradient descent. Cyclical log annealing implements the restart pattern more aggressively to maybe allow the usage of more greedy algorithms on the online convex optimization framework. The algorithm was tested on the CIFAR-10 image datasets, and seemed to perform analogously with cosine annealing on large transformer-enhanced residual neural networks. Future experiments would involve testing the scheduler in generative adversarial networks and finding the best parameters for the scheduler with more experiments.

Title: FedSR: A Semi-Decentralized Federated Learning Algorithm for Non-IIDness in IoT System

• Authors: Jianjun Huang, Lixin Ye, Li Kang
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.14718
• Pdf URL: https://arxiv.org/pdf/2403.14718
• Copy Paste: [[2403.14718]] FedSR: A Semi-Decentralized Federated Learning Algorithm for Non-IIDness in IoT System(https://arxiv.org/abs/2403.14718)
• Keywords: security, privacy, protect, federate
• Abstract: In the Industrial Internet of Things (IoT), a large amount of data will be generated every day. Due to privacy and security issues, it is difficult to collect all these data together to train deep learning models, thus the federated learning, a distributed machine learning paradigm that protects data privacy, has been widely used in IoT. However, in practical federated learning, the data distributions usually have large differences across devices, and the heterogeneity of data will deteriorate the performance of the model. Moreover, federated learning in IoT usually has a large number of devices involved in training, and the limited communication resource of cloud servers become a bottleneck for training. To address the above issues, in this paper, we combine centralized federated learning with decentralized federated learning to design a semi-decentralized cloud-edge-device hierarchical federated learning framework, which can mitigate the impact of data heterogeneity, and can be deployed at lage scale in IoT. To address the effect of data heterogeneity, we use an incremental subgradient optimization algorithm in each ring cluster to improve the generalization ability of the ring cluster models. Our extensive experiments show that our approach can effectively mitigate the impact of data heterogeneity and alleviate the communication bottleneck in cloud servers.

Title: Bypassing LLM Watermarks with Color-Aware Substitutions

• Authors: Qilong Wu, Varun Chandrasekaran
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14719
• Pdf URL: https://arxiv.org/pdf/2403.14719
• Copy Paste: [[2403.14719]] Bypassing LLM Watermarks with Color-Aware Substitutions(https://arxiv.org/abs/2403.14719)
• Keywords: attack, robust, watermark, large language model
• Abstract: Watermarking approaches are proposed to identify if text being circulated is human or large language model (LLM) generated. The state-of-the-art watermarking strategy of Kirchenbauer et al. (2023a) biases the LLM to generate specific (``green'') tokens. However, determining the robustness of this watermarking method is an open problem. Existing attack methods fail to evade detection for longer text segments. We overcome this limitation, and propose {\em Self Color Testing-based Substitution (SCTS)}, the first ``color-aware'' attack. SCTS obtains color information by strategically prompting the watermarked LLM and comparing output tokens frequencies. It uses this information to determine token colors, and substitutes green tokens with non-green ones. In our experiments, SCTS successfully evades watermark detection using fewer number of edits than related work. Additionally, we show both theoretically and empirically that SCTS can remove the watermark for arbitrarily long watermarked text.

Title: Defending Against Indirect Prompt Injection Attacks With Spotlighting

• Authors: Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, Emre Kiciman
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14720
• Pdf URL: https://arxiv.org/pdf/2403.14720
• Copy Paste: [[2403.14720]] Defending Against Indirect Prompt Injection Attacks With Spotlighting(https://arxiv.org/abs/2403.14720)
• Keywords: security, defense, attack, robust, large language model
• Abstract: Large Language Models (LLMs), while powerful, are built and trained to process a single text input. In common applications, multiple inputs can be processed by concatenating them together into a single stream of text. However, the LLM is unable to distinguish which sections of prompt belong to various input sources. Indirect prompt injection attacks take advantage of this vulnerability by embedding adversarial instructions into untrusted data being processed alongside user commands. Often, the LLM will mistake the adversarial instructions as user commands to be followed, creating a security vulnerability in the larger system. We introduce spotlighting, a family of prompt engineering techniques that can be used to improve LLMs' ability to distinguish among multiple sources of input. The key insight is to utilize transformations of an input to provide a reliable and continuous signal of its provenance. We evaluate spotlighting as a defense against indirect prompt injection attacks, and find that it is a robust defense that has minimal detrimental impact to underlying NLP tasks. Using GPT-family models, we find that spotlighting reduces the attack success rate from greater than {50}\% to below {2}\% in our experiments with minimal impact on task efficacy.

Title: Six Levels of Privacy: A Framework for Financial Synthetic Data

• Authors: Tucker Balch, Vamsi K. Potluru, Deepak Paramanand, Manuela Veloso
• Subjects: cs.CR, cs.LG, q-fin.ST
• Abstract URL: https://arxiv.org/abs/2403.14724
• Pdf URL: https://arxiv.org/pdf/2403.14724
• Copy Paste: [[2403.14724]] Six Levels of Privacy: A Framework for Financial Synthetic Data(https://arxiv.org/abs/2403.14724)
• Keywords: privacy, protect, defense, attack
• Abstract: Synthetic Data is increasingly important in financial applications. In addition to the benefits it provides, such as improved financial modeling and better testing procedures, it poses privacy risks as well. Such data may arise from client information, business information, or other proprietary sources that must be protected. Even though the process by which Synthetic Data is generated serves to obscure the original data to some degree, the extent to which privacy is preserved is hard to assess. Accordingly, we introduce a hierarchy of ``levels'' of privacy that are useful for categorizing Synthetic Data generation methods and the progressively improved protections they offer. While the six levels were devised in the context of financial applications, they may also be appropriate for other industries as well. Our paper includes: A brief overview of Financial Synthetic Data, how it can be used, how its value can be assessed, privacy risks, and privacy attacks. We close with details of the ``Six Levels'' that include defenses against those attacks.

Title: Jailbreaking is Best Solved by Definition

• Authors: Taeyoun Kim, Suhas Kotha, Aditi Raghunathan
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14725
• Pdf URL: https://arxiv.org/pdf/2403.14725
• Copy Paste: [[2403.14725]] Jailbreaking is Best Solved by Definition(https://arxiv.org/abs/2403.14725)
• Keywords: defense, attack, robust
• Abstract: The rise of "jailbreak" attacks on language models has led to a flurry of defenses aimed at preventing the output of undesirable responses. In this work, we critically examine the two stages of the defense pipeline: (i) the definition of what constitutes unsafe outputs, and (ii) the enforcement of the definition via methods such as input processing or fine-tuning. We cast severe doubt on the efficacy of existing enforcement mechanisms by showing that they fail to defend even for a simple definition of unsafe outputs--outputs that contain the word "purple". In contrast, post-processing outputs is perfectly robust for such a definition. Drawing on our results, we present our position that the real challenge in defending jailbreaks lies in obtaining a good definition of unsafe responses: without a good definition, no enforcement strategy can succeed, but with a good definition, output processing already serves as a robust baseline albeit with inference-time overheads.

Title: Reversible Jump Attack to Textual Classifiers with Modification Reduction

• Authors: Mingze Ni, Zhensu Sun, Wei Liu
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14731
• Pdf URL: https://arxiv.org/pdf/2403.14731
• Copy Paste: [[2403.14731]] Reversible Jump Attack to Textual Classifiers with Modification Reduction(https://arxiv.org/abs/2403.14731)
• Keywords: attack
• Abstract: Recent studies on adversarial examples expose vulnerabilities of natural language processing (NLP) models. Existing techniques for generating adversarial examples are typically driven by deterministic hierarchical rules that are agnostic to the optimal adversarial examples, a strategy that often results in adversarial samples with a suboptimal balance between magnitudes of changes and attack successes. To this end, in this research we propose two algorithms, Reversible Jump Attack (RJA) and Metropolis-Hasting Modification Reduction (MMR), to generate highly effective adversarial examples and to improve the imperceptibility of the examples, respectively. RJA utilizes a novel randomization mechanism to enlarge the search space and efficiently adapts to a number of perturbed words for adversarial examples. With these generated adversarial examples, MMR applies the Metropolis-Hasting sampler to enhance the imperceptibility of adversarial examples. Extensive experiments demonstrate that RJA-MMR outperforms current state-of-the-art methods in attack performance, imperceptibility, fluency and grammar correctness.

Title: FedMef: Towards Memory-efficient Federated Dynamic Pruning

• Authors: Hong Huang, Weiming Zhuang, Chen Chen, Lingjuan Lyu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.14737
• Pdf URL: https://arxiv.org/pdf/2403.14737
• Copy Paste: [[2403.14737]] FedMef: Towards Memory-efficient Federated Dynamic Pruning(https://arxiv.org/abs/2403.14737)
• Keywords: federate
• Abstract: Federated learning (FL) promotes decentralized training while prioritizing data confidentiality. However, its application on resource-constrained devices is challenging due to the high demand for computation and memory resources to train deep learning models. Neural network pruning techniques, such as dynamic pruning, could enhance model efficiency, but directly adopting them in FL still poses substantial challenges, including post-pruning performance degradation, high activation memory usage, etc. To address these challenges, we propose FedMef, a novel and memory-efficient federated dynamic pruning framework. FedMef comprises two key components. First, we introduce the budget-aware extrusion that maintains pruning efficiency while preserving post-pruning performance by salvaging crucial information from parameters marked for pruning within a given budget. Second, we propose scaled activation pruning to effectively reduce activation memory footprints, which is particularly beneficial for deploying FL to memory-limited devices. Extensive experiments demonstrate the effectiveness of our proposed FedMef. In particular, it achieves a significant reduction of 28.5% in memory footprint compared to state-of-the-art methods while obtaining superior accuracy.

Title: A task of anomaly detection for a smart satellite Internet of things system

• Authors: Zilong Shao
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.14738
• Pdf URL: https://arxiv.org/pdf/2403.14738
• Copy Paste: [[2403.14738]] A task of anomaly detection for a smart satellite Internet of things system(https://arxiv.org/abs/2403.14738)
• Keywords: security, attack, interpretability, generative
• Abstract: When the equipment is working, real-time collection of environmental sensor data for anomaly detection is one of the key links to prevent industrial process accidents and network attacks and ensure system security. However, under the environment with specific real-time requirements, the anomaly detection for environmental sensors still faces the following difficulties: (1) The complex nonlinear correlation characteristics between environmental sensor data variables lack effective expression methods, and the distribution between the data is difficult to be captured. (2) it is difficult to ensure the real-time monitoring requirements by using complex machine learning models, and the equipment cost is too high. (3) Too little sample data leads to less labeled data in supervised learning. This paper proposes an unsupervised deep learning anomaly detection system. Based on the generative adversarial network and self-attention mechanism, considering the different feature information contained in the local subsequences, it automatically learns the complex linear and nonlinear dependencies between environmental sensor variables, and uses the anomaly score calculation method combining reconstruction error and discrimination error. It can monitor the abnormal points of real sensor data with high real-time performance and can run on the intelligent satellite Internet of things system, which is suitable for the real working environment. Anomaly detection outperforms baseline methods in most cases and has good interpretability, which can be used to prevent industrial accidents and cyber-attacks for monitoring environmental sensors.

Title: VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understanding

• Authors: Ahmad Mahmood, Ashmal Vayani, Muzammal Naseer, Salman Khan, Fahad Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14743
• Pdf URL: https://arxiv.org/pdf/2403.14743
• Copy Paste: [[2403.14743]] VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understanding(https://arxiv.org/abs/2403.14743)
• Keywords: robust, large language model
• Abstract: Recent studies have demonstrated the effectiveness of Large Language Models (LLMs) as reasoning modules that can deconstruct complex tasks into more manageable sub-tasks, particularly when applied to visual reasoning tasks for images. In contrast, this paper introduces a Video Understanding and Reasoning Framework (VURF) based on the reasoning power of LLMs. Ours is a novel approach to extend the utility of LLMs in the context of video tasks, leveraging their capacity to generalize from minimal input and output demonstrations within a contextual framework. By presenting LLMs with pairs of instructions and their corresponding high-level programs, we harness their contextual learning capabilities to generate executable visual programs for video understanding. To enhance program's accuracy and robustness, we implement two important strategies. Firstly, we employ a feedback-generation approach, powered by GPT-3.5, to rectify errors in programs utilizing unsupported functions. Secondly, taking motivation from recent works on self refinement of LLM outputs, we introduce an iterative procedure for improving the quality of the in-context examples by aligning the initial outputs to the outputs that would have been generated had the LLM not been bound by the structure of the in-context examples. Our results on several video-specific tasks, including visual QA, video anticipation, pose estimation and multi-video QA illustrate the efficacy of these enhancements in improving the performance of visual programming approaches for video tasks. Our Codes and data will be publicly released.

Title: Can 3D Vision-Language Models Truly Understand Natural Language?

• Authors: Weipeng Deng, Runyu Ding, Jihan Yang, Jiahui Liu, Yijiang Li, Xiaojuan Qi, Edith Ngai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14760
• Pdf URL: https://arxiv.org/pdf/2403.14760
• Copy Paste: [[2403.14760]] Can 3D Vision-Language Models Truly Understand Natural Language?(https://arxiv.org/abs/2403.14760)
• Keywords: robust
• Abstract: Rapid advancements in 3D vision-language (3D-VL) tasks have opened up new avenues for human interaction with embodied agents or robots using natural language. Despite this progress, we find a notable limitation: existing 3D-VL models exhibit sensitivity to the styles of language input, struggling to understand sentences with the same semantic meaning but written in different variants. This observation raises a critical question: Can 3D vision-language models truly understand natural language? To test the language understandability of 3D-VL models, we first propose a language robustness task for systematically assessing 3D-VL models across various tasks, benchmarking their performance when presented with different language style variants. Importantly, these variants are commonly encountered in applications requiring direct interaction with humans, such as embodied robotics, given the diversity and unpredictability of human language. We propose a 3D Language Robustness Dataset, designed based on the characteristics of human language, to facilitate the systematic study of robustness. Our comprehensive evaluation uncovers a significant drop in the performance of all existing models across various 3D-VL tasks. Even the state-of-the-art 3D-LLM fails to understand some variants of the same sentences. Further in-depth analysis suggests that the existing models have a fragile and biased fusion module, which stems from the low diversity of the existing dataset. Finally, we propose a training-free module driven by LLM, which improves language robustness. Datasets and code will be available at github.

Title: Improving Robustness to Model Inversion Attacks via Sparse Coding Architectures

• Authors: Sayanton V. Dibbo, Adam Breuer, Juston Moore, Michael Teti
• Subjects: cs.CV, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14772
• Pdf URL: https://arxiv.org/pdf/2403.14772
• Copy Paste: [[2403.14772]] Improving Robustness to Model Inversion Attacks via Sparse Coding Architectures(https://arxiv.org/abs/2403.14772)
• Keywords: privacy, defense, attack, robust
• Abstract: Recent model inversion attack algorithms permit adversaries to reconstruct a neural network's private training data just by repeatedly querying the network and inspecting its outputs. In this work, we develop a novel network architecture that leverages sparse-coding layers to obtain superior robustness to this class of attacks. Three decades of computer science research has studied sparse coding in the context of image denoising, object recognition, and adversarial misclassification settings, but to the best of our knowledge, its connection to state-of-the-art privacy vulnerabilities remains unstudied. However, sparse coding architectures suggest an advantageous means to defend against model inversion attacks because they allow us to control the amount of irrelevant private information encoded in a network's intermediate representations in a manner that can be computed efficiently during training and that is known to have little effect on classification accuracy. Specifically, compared to networks trained with a variety of state-of-the-art defenses, our sparse-coding architectures maintain comparable or higher classification accuracy while degrading state-of-the-art training data reconstructions by factors of 1.1 to 18.3 across a variety of reconstruction quality metrics (PSNR, SSIM, FID). This performance advantage holds across 5 datasets ranging from CelebA faces to medical images and CIFAR-10, and across various state-of-the-art SGD-based and GAN-based inversion attacks, including Plug-&-Play attacks. We provide a cluster-ready PyTorch codebase to promote research and standardize defense evaluations.

Title: StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text

• Authors: Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, Humphrey Shi
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.14773
• Pdf URL: https://arxiv.org/pdf/2403.14773
• Copy Paste: [[2403.14773]] StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text(https://arxiv.org/abs/2403.14773)
• Keywords: diffusion
• Abstract: Text-to-video diffusion models enable the generation of high-quality videos that follow text instructions, making it easy to create diverse and individual content. However, existing approaches mostly focus on high-quality short video generation (typically 16 or 24 frames), ending up with hard-cuts when naively extended to the case of long video synthesis. To overcome these limitations, we introduce StreamingT2V, an autoregressive approach for long video generation of 80, 240, 600, 1200 or more frames with smooth transitions. The key components are:(i) a short-term memory block called conditional attention module (CAM), which conditions the current generation on the features extracted from the previous chunk via an attentional mechanism, leading to consistent chunk transitions, (ii) a long-term memory block called appearance preservation module, which extracts high-level scene and object features from the first video chunk to prevent the model from forgetting the initial scene, and (iii) a randomized blending approach that enables to apply a video enhancer autoregressively for infinitely long videos without inconsistencies between chunks. Experiments show that StreamingT2V generates high motion amount. In contrast, all competing image-to-video methods are prone to video stagnation when applied naively in an autoregressive manner. Thus, we propose with StreamingT2V a high-quality seamless text-to-long video generator that outperforms competitors with consistency and motion. Our code will be available at: https://github.com/Picsart-AI-Research/StreamingT2V

Title: Few-Shot Adversarial Prompt Learning on Vision-Language Models

• Authors: Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, Tongliang Liu
• Subjects: cs.CV, cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14774
• Pdf URL: https://arxiv.org/pdf/2403.14774
• Copy Paste: [[2403.14774]] Few-Shot Adversarial Prompt Learning on Vision-Language Models(https://arxiv.org/abs/2403.14774)
• Keywords: robust
• Abstract: The vulnerability of deep neural networks to imperceptible adversarial perturbations has attracted widespread attention. Inspired by the success of vision-language foundation models, previous efforts achieved zero-shot adversarial robustness by aligning adversarial visual features with text supervision. However, in practice, they are still unsatisfactory due to several issues, including heavy adaptation cost, suboptimal text supervision, and uncontrolled natural generalization capacity. In this paper, to address these issues, we propose a few-shot adversarial prompt framework where adapting input sequences with limited data makes significant adversarial robustness improvement. Specifically, we achieve this by providing adversarially correlated text supervision that is end-to-end learned from adversarial examples. We also propose a novel training objective that enhances the consistency of multi-modal features while encourages differentiated uni-modal features between natural and adversarial examples. The proposed framework gives access to learn adversarial text supervision, which provides superior cross-modal adversarial alignment and matches state-of-the-art zero-shot adversarial robustness with only 1% training data.

Title: Diffusion Attack: Leveraging Stable Diffusion for Naturalistic Image Attacking

• Authors: Qianyu Guo, Jiaming Fu, Yawen Lu, Dongming Gan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.14778
• Pdf URL: https://arxiv.org/pdf/2403.14778
• Copy Paste: [[2403.14778]] Diffusion Attack: Leveraging Stable Diffusion for Naturalistic Image Attacking(https://arxiv.org/abs/2403.14778)
• Keywords: security, attack, diffusion
• Abstract: In Virtual Reality (VR), adversarial attack remains a significant security threat. Most deep learning-based methods for physical and digital adversarial attacks focus on enhancing attack performance by crafting adversarial examples that contain large printable distortions that are easy for human observers to identify. However, attackers rarely impose limitations on the naturalness and comfort of the appearance of the generated attack image, resulting in a noticeable and unnatural attack. To address this challenge, we propose a framework to incorporate style transfer to craft adversarial inputs of natural styles that exhibit minimal detectability and maximum natural appearance, while maintaining superior attack capabilities.

Title: Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance

• Authors: Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, Siyu Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14781
• Pdf URL: https://arxiv.org/pdf/2403.14781
• Copy Paste: [[2403.14781]] Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance(https://arxiv.org/abs/2403.14781)
• Keywords: diffusion, generative
• Abstract: In this study, we introduce a methodology for human image animation by leveraging a 3D human parametric model within a latent diffusion framework to enhance shape alignment and motion guidance in curernt human generative techniques. The methodology utilizes the SMPL(Skinned Multi-Person Linear) model as the 3D human parametric model to establish a unified representation of body shape and pose. This facilitates the accurate capture of intricate human geometry and motion characteristics from source videos. Specifically, we incorporate rendered depth images, normal maps, and semantic maps obtained from SMPL sequences, alongside skeleton-based motion guidance, to enrich the conditions to the latent diffusion model with comprehensive 3D shape and detailed pose attributes. A multi-layer motion fusion module, integrating self-attention mechanisms, is employed to fuse the shape and motion latent representations in the spatial domain. By representing the 3D human parametric model as the motion guidance, we can perform parametric shape alignment of the human body between the reference image and the source video motion. Experimental evaluations conducted on benchmark datasets demonstrate the methodology's superior ability to generate high-quality human animations that accurately capture both pose and shape variations. Furthermore, our approach also exhibits superior generalization capabilities on the proposed wild dataset. Project page: https://fudan-generative-vision.github.io/champ.

Title: Multi-Agent VQA: Exploring Multi-Agent Foundation Models in Zero-Shot Visual Question Answering

• Authors: Bowen Jiang, Zhijun Zhuang, Shreyas S. Shivakumar, Dan Roth, Camillo J. Taylor
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2403.14783
• Pdf URL: https://arxiv.org/pdf/2403.14783
• Copy Paste: [[2403.14783]] Multi-Agent VQA: Exploring Multi-Agent Foundation Models in Zero-Shot Visual Question Answering(https://arxiv.org/abs/2403.14783)
• Keywords: robust
• Abstract: This work explores the zero-shot capabilities of foundation models in Visual Question Answering (VQA) tasks. We propose an adaptive multi-agent system, named Multi-Agent VQA, to overcome the limitations of foundation models in object detection and counting by using specialized agents as tools. Unlike existing approaches, our study focuses on the system's performance without fine-tuning it on specific VQA datasets, making it more practical and robust in the open world. We present preliminary experimental results under zero-shot scenarios and highlight some failure cases, offering new directions for future research.

Title: On the exploitation of DCT statistics for cropping detectors

• Authors: Claudio Vittorio Ragaglia, Francesco Guarnera, Sebastiano Battiato
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14789
• Pdf URL: https://arxiv.org/pdf/2403.14789
• Copy Paste: [[2403.14789]] On the exploitation of DCT statistics for cropping detectors(https://arxiv.org/abs/2403.14789)
• Keywords: security
• Abstract: {The study of frequency components derived from Discrete Cosine Transform (DCT) has been widely used in image analysis. In recent years it has been observed that significant information can be extrapolated from them about the lifecycle of the image, but no study has focused on the analysis between them and the source resolution of the image. In this work, we investigated a novel image resolution classifier that employs DCT statistics with the goal to detect the original resolution of images; in particular the insight was exploited to address the challenge of identifying cropped images. Training a Machine Learning (ML) classifier on entire images (not cropped), the generated model can leverage this information to detect cropping. The results demonstrate the classifier's reliability in distinguishing between cropped and not cropped images, providing a dependable estimation of their original resolution. This advancement has significant implications for image processing applications, including digital security, authenticity verification, and visual quality analysis, by offering a new tool for detecting image manipulations and enhancing qualitative image assessment. This work opens new perspectives in the field, with potential to transform image analysis and usage across multiple domains.}

Title: Latent Diffusion Models for Attribute-Preserving Image Anonymization

• Authors: Luca Piano, Pietro Basci, Fabrizio Lamberti, Lia Morra
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14790
• Pdf URL: https://arxiv.org/pdf/2403.14790
• Copy Paste: [[2403.14790]] Latent Diffusion Models for Attribute-Preserving Image Anonymization(https://arxiv.org/abs/2403.14790)
• Keywords: privacy, protect, diffusion, generative
• Abstract: Generative techniques for image anonymization have great potential to generate datasets that protect the privacy of those depicted in the images, while achieving high data fidelity and utility. Existing methods have focused extensively on preserving facial attributes, but failed to embrace a more comprehensive perspective that considers the scene and background into the anonymization process. This paper presents, to the best of our knowledge, the first approach to image anonymization based on Latent Diffusion Models (LDMs). Every element of a scene is maintained to convey the same meaning, yet manipulated in a way that makes re-identification difficult. We propose two LDMs for this purpose: CAMOUFLaGE-Base exploits a combination of pre-trained ControlNets, and a new controlling mechanism designed to increase the distance between the real and anonymized images. CAMOFULaGE-Light is based on the Adapter technique, coupled with an encoding designed to efficiently represent the attributes of different persons in a scene. The former solution achieves superior performance on most metrics and benchmarks, while the latter cuts the inference time in half at the cost of fine-tuning a lightweight module. We show through extensive experimental comparison that the proposed method is competitive with the state-of-the-art concerning identity obfuscation whilst better preserving the original content of the image and tackling unresolved challenges that current solutions fail to address.

Title: Preventing Catastrophic Forgetting through Memory Networks in Continuous Detection

• Authors: Gaurav Bhatt, James Ross, Leonid Sigal
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14797
• Pdf URL: https://arxiv.org/pdf/2403.14797
• Copy Paste: [[2403.14797]] Preventing Catastrophic Forgetting through Memory Networks in Continuous Detection(https://arxiv.org/abs/2403.14797)
• Keywords: transformer, segmentation
• Abstract: Modern pre-trained architectures struggle to retain previous information while undergoing continuous fine-tuning on new tasks. Despite notable progress in continual classification, systems designed for complex vision tasks such as detection or segmentation still struggle to attain satisfactory performance. In this work, we introduce a memory-based detection transformer architecture to adapt a pre-trained DETR-style detector to new tasks while preserving knowledge from previous tasks. We propose a novel localized query function for efficient information retrieval from memory units, aiming to minimize forgetting. Furthermore, we identify a fundamental challenge in continual detection referred to as background relegation. This arises when object categories from earlier tasks reappear in future tasks, potentially without labels, leading them to be implicitly treated as background. This is an inevitable issue in continual detection or segmentation. The introduced continual optimization technique effectively tackles this challenge. Finally, we assess the performance of our proposed system on continual detection benchmarks and demonstrate that our approach surpasses the performance of existing state-of-the-art resulting in 5-7% improvements on MS-COCO and PASCAL-VOC on the task of continual detection.

Title: The opportunities and risks of large language models in mental health

• Authors: Hannah R. Lawrence, Renee A. Schneider, Susan B. Rubin, Maja J. Mataric, Daniel J. McDuff, Megan Jones Bell
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14814
• Pdf URL: https://arxiv.org/pdf/2403.14814
• Copy Paste: [[2403.14814]] The opportunities and risks of large language models in mental health(https://arxiv.org/abs/2403.14814)
• Keywords: large language model
• Abstract: Global rates of mental health concerns are rising and there is increasing realization that existing models of mental healthcare will not adequately expand to meet the demand. With the emergence of large language models (LLMs) has come great optimism regarding their promise to create novel, large-scale solutions to support mental health. Despite their nascence, LLMs have already been applied to mental health-related tasks. In this review, we summarize the extant literature on efforts to use LLMs to provide mental health education, assessment, and intervention and highlight key opportunities for positive impact in each area. We then highlight risks associated with LLMs application to mental health and encourage adoption of strategies to mitigate these risks. The urgent need for mental health support must be balanced with responsible development, testing, and deployment of mental health LLMs. Especially critical is ensuring that mental health LLMs are fine-tuned for mental health, enhance mental health equity, adhere to ethical standards, and that people, including those with lived experience with mental health concerns, are involved in all stages from development through deployment. Prioritizing these efforts will minimize potential harms to mental health and maximize the likelihood that LLMs will positively impact mental health globally.

Title: Learning Gaussian Representation for Eye Fixation Prediction

• Authors: Peipei Song, Jing Zhang, Piotr Koniusz, Nick Barnes
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14821
• Pdf URL: https://arxiv.org/pdf/2403.14821
• Copy Paste: [[2403.14821]] Learning Gaussian Representation for Eye Fixation Prediction(https://arxiv.org/abs/2403.14821)
• Keywords: robust
• Abstract: Existing eye fixation prediction methods perform the mapping from input images to the corresponding dense fixation maps generated from raw fixation points. However, due to the stochastic nature of human fixation, the generated dense fixation maps may be a less-than-ideal representation of human fixation. To provide a robust fixation model, we introduce Gaussian Representation for eye fixation modeling. Specifically, we propose to model the eye fixation map as a mixture of probability distributions, namely a Gaussian Mixture Model. In this new representation, we use several Gaussian distribution components as an alternative to the provided fixation map, which makes the model more robust to the randomness of fixation. Meanwhile, we design our framework upon some lightweight backbones to achieve real-time fixation prediction. Experimental results on three public fixation prediction datasets (SALICON, MIT1003, TORONTO) demonstrate that our method is fast and effective.

Title: Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing

• Authors: Alberto Baldrati, Davide Morelli, Marcella Cornia, Marco Bertini, Rita Cucchiara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14828
• Pdf URL: https://arxiv.org/pdf/2403.14828
• Copy Paste: [[2403.14828]] Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing(https://arxiv.org/abs/2403.14828)
• Keywords: diffusion
• Abstract: Fashion illustration is a crucial medium for designers to convey their creative vision and transform design concepts into tangible representations that showcase the interplay between clothing and the human body. In the context of fashion design, computer vision techniques have the potential to enhance and streamline the design process. Departing from prior research primarily focused on virtual try-on, this paper tackles the task of multimodal-conditioned fashion image editing. Our approach aims to generate human-centric fashion images guided by multimodal prompts, including text, human body poses, garment sketches, and fabric textures. To address this problem, we propose extending latent diffusion models to incorporate these multiple modalities and modifying the structure of the denoising network, taking multimodal prompts as input. To condition the proposed architecture on fabric textures, we employ textual inversion techniques and let diverse cross-attention layers of the denoising network attend to textual and texture information, thus incorporating different granularity conditioning details. Given the lack of datasets for the task, we extend two existing fashion datasets, Dress Code and VITON-HD, with multimodal annotations. Experimental evaluations demonstrate the effectiveness of our proposed approach in terms of realism and coherence concerning the provided multimodal inputs.

Title: Osmosis: RGBD Diffusion Prior for Underwater Image Restoration

• Authors: Opher Bar Nathan, Deborah Levy, Tali Treibitz, Dan Rosenbaum
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14837
• Pdf URL: https://arxiv.org/pdf/2403.14837
• Copy Paste: [[2403.14837]] Osmosis: RGBD Diffusion Prior for Underwater Image Restoration(https://arxiv.org/abs/2403.14837)
• Keywords: diffusion
• Abstract: Underwater image restoration is a challenging task because of strong water effects that increase dramatically with distance. This is worsened by lack of ground truth data of clean scenes without water. Diffusion priors have emerged as strong image restoration priors. However, they are often trained with a dataset of the desired restored output, which is not available in our case. To overcome this critical issue, we show how to leverage in-air images to train diffusion priors for underwater restoration. We also observe that only color data is insufficient, and augment the prior with a depth channel. We train an unconditional diffusion model prior on the joint space of color and depth, using standard RGBD datasets of natural outdoor scenes in air. Using this prior together with a novel guidance method based on the underwater image formation model, we generate posterior samples of clean images, removing the water effects. Even though our prior did not see any underwater images during training, our method outperforms state-of-the-art baselines for image restoration on very challenging scenes. Data, models and code are published in the project page.

Title: TAMS: Translation-Assisted Morphological Segmentation

• Authors: Enora Rice, Ali Marashian, Luke Gessler, Alexis Palmer, Katharina von der Wense
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14840
• Pdf URL: https://arxiv.org/pdf/2403.14840
• Copy Paste: [[2403.14840]] TAMS: Translation-Assisted Morphological Segmentation(https://arxiv.org/abs/2403.14840)
• Keywords: segmentation
• Abstract: Canonical morphological segmentation is the process of analyzing words into the standard (aka underlying) forms of their constituent morphemes. This is a core task in language documentation, and NLP systems have the potential to dramatically speed up this process. But in typical language documentation settings, training data for canonical morpheme segmentation is scarce, making it difficult to train high quality models. However, translation data is often much more abundant, and, in this work, we present a method that attempts to leverage this data in the canonical segmentation task. We propose a character-level sequence-to-sequence model that incorporates representations of translations obtained from pretrained high-resource monolingual language models as an additional signal. Our model outperforms the baseline in a super-low resource setting but yields mixed results on training splits with more data. While further work is needed to make translations useful in higher-resource settings, our model shows promise in severely resource-constrained settings.

Title: KeyPoint Relative Position Encoding for Face Recognition

• Authors: Minchul Kim, Yiyang Su, Feng Liu, Anil Jain, Xiaoming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14852
• Pdf URL: https://arxiv.org/pdf/2403.14852
• Copy Paste: [[2403.14852]] KeyPoint Relative Position Encoding for Face Recognition(https://arxiv.org/abs/2403.14852)
• Keywords: robust
• Abstract: In this paper, we address the challenge of making ViT models more robust to unseen affine transformations. Such robustness becomes useful in various recognition tasks such as face recognition when image alignment failures occur. We propose a novel method called KP-RPE, which leverages key points (e.g.~facial landmarks) to make ViT more resilient to scale, translation, and pose variations. We begin with the observation that Relative Position Encoding (RPE) is a good way to bring affine transform generalization to ViTs. RPE, however, can only inject the model with prior knowledge that nearby pixels are more important than far pixels. Keypoint RPE (KP-RPE) is an extension of this principle, where the significance of pixels is not solely dictated by their proximity but also by their relative positions to specific keypoints within the image. By anchoring the significance of pixels around keypoints, the model can more effectively retain spatial relationships, even when those relationships are disrupted by affine transformations. We show the merit of KP-RPE in face and gait recognition. The experimental results demonstrate the effectiveness in improving face recognition performance from low-quality images, particularly where alignment is prone to failure. Code and pre-trained models are available.

Title: Structuring the Chaos: Enabling Small Business Cyber-Security Risks & Assets Modelling with a UML Class Model

• Authors: Tracy Tam, Asha Rao, Joanne Hall
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.14872
• Pdf URL: https://arxiv.org/pdf/2403.14872
• Copy Paste: [[2403.14872]] Structuring the Chaos: Enabling Small Business Cyber-Security Risks & Assets Modelling with a UML Class Model(https://arxiv.org/abs/2403.14872)
• Keywords: security
• Abstract: Small businesses are increasingly adopting IT, and consequently becoming more vulnerable to cyber-incidents. Whilst small businesses are aware of the cyber-security risks, many struggle with implementing mitigations. Some of these can be traced to fundamental differences in the characteristics of small business versus large enterprises where modern cyber-security solutions are widely deployed. Small business specific cyber-security tools are needed. Currently available cyber-security tools and standards assume technical expertise and time resources often not practical for small businesses. Cyber-security competes with other roles that small business owners take on, e.g. cleaning, sales etc. A small business model, salient and implementable at-scale, with simplified non-specialist terminologies and presentation is needed to encourage sustained participation of all stakeholders, not just technical ones. We propose a new UML class (Small IT Data (SITD)) model to support the often chaotic information-gathering phase of a small business' first foray into cyber-security. The SITD model is designed in the UML format to help small business implement technical solutions. The SITD model structure stays relevant by using generic classes and structures that evolve with technology and environmental changes. The SITD model keeps security decisions proportionate to the business by highlighting relationships between business strategy tasks and IT infrastructure. We construct a set of design principles to address small business cyber-security needs. Model components are designed in response to these needs. The uses of the SITD model are then demonstrated and design principles validated by examining a case study of a real small business operational and IT information. The SITD model's ability to illustrate breach information is also demonstrated using the NotPetya incident.

Title: WeatherProof: Leveraging Language Guidance for Semantic Segmentation in Adverse Weather

• Authors: Blake Gella, Howard Zhang, Rishi Upadhyay, Tiffany Chang, Nathan Wei, Matthew Waliman, Yunhao Bao, Celso de Melo, Alex Wong, Achuta Kadambi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14874
• Pdf URL: https://arxiv.org/pdf/2403.14874
• Copy Paste: [[2403.14874]] WeatherProof: Leveraging Language Guidance for Semantic Segmentation in Adverse Weather(https://arxiv.org/abs/2403.14874)
• Keywords: robust, segmentation
• Abstract: We propose a method to infer semantic segmentation maps from images captured under adverse weather conditions. We begin by examining existing models on images degraded by weather conditions such as rain, fog, or snow, and found that they exhibit a large performance drop as compared to those captured under clear weather. To control for changes in scene structures, we propose WeatherProof, the first semantic segmentation dataset with accurate clear and adverse weather image pairs that share an underlying scene. Through this dataset, we analyze the error modes in existing models and found that they were sensitive to the highly complex combination of different weather effects induced on the image during capture. To improve robustness, we propose a way to use language as guidance by identifying contributions of adverse weather conditions and injecting that as "side information". Models trained using our language guidance exhibit performance gains by up to 10.2% in mIoU on WeatherProof, up to 8.44% in mIoU on the widely used ACDC dataset compared to standard training techniques, and up to 6.21% in mIoU on the ACDC dataset as compared to previous SOTA methods.

Title: DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation

• Authors: Zeeshan Hayder, Xuming He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14886
• Pdf URL: https://arxiv.org/pdf/2403.14886
• Copy Paste: [[2403.14886]] DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation(https://arxiv.org/abs/2403.14886)
• Keywords: transformer
• Abstract: Scene graph generation aims to capture detailed spatial and semantic relationships between objects in an image, which is challenging due to incomplete labelling, long-tailed relationship categories, and relational semantic overlap. Existing Transformer-based methods either employ distinct queries for objects and predicates or utilize holistic queries for relation triplets and hence often suffer from limited capacity in learning low-frequency relationships. In this paper, we present a new Transformer-based method, called DSGG, that views scene graph detection as a direct graph prediction problem based on a unique set of graph-aware queries. In particular, each graph-aware query encodes a compact representation of both the node and all of its relations in the graph, acquired through the utilization of a relaxed sub-graph matching during the training process. Moreover, to address the problem of relational semantic overlap, we utilize a strategy for relation distillation, aiming to efficiently learn multiple instances of semantic relationships. Extensive experiments on the VG and the PSG datasets show that our model achieves state-of-the-art results, showing a significant improvement of 3.5\% and 6.7\% in mR@50 and mR@100 for the scene-graph generation task and achieves an even more substantial improvement of 8.5\% and 10.3\% in mR@50 and mR@100 for the panoptic scene graph generation task. Code is available at \url{https://github.com/zeeshanhayder/DSGG}.

Title: AutoRE: Document-Level Relation Extraction with Large Language Models

• Authors: Xue Lilong, Zhang Dan, Dong Yuxiao, Tang Jie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14888
• Pdf URL: https://arxiv.org/pdf/2403.14888
• Copy Paste: [[2403.14888]] AutoRE: Document-Level Relation Extraction with Large Language Models(https://arxiv.org/abs/2403.14888)
• Keywords: extraction, large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional abilities in comprehending and generating text, motivating numerous researchers to utilize them for Information Extraction (IE) purposes, including Relation Extraction (RE). Nonetheless, most existing methods are predominantly designed for Sentence-level Relation Extraction (SentRE) tasks, which typically encompass a restricted set of relations and triplet facts within a single sentence. Furthermore, certain approaches resort to treating relations as candidate choices integrated into prompt templates, leading to inefficient processing and suboptimal performance when tackling Document-Level Relation Extraction (DocRE) tasks, which entail handling multiple relations and triplet facts distributed across a given document, posing distinct challenges. To overcome these limitations, we introduce AutoRE, an end-to-end DocRE model that adopts a novel RE extraction paradigm named RHF (Relation-Head-Facts). Unlike existing approaches, AutoRE does not rely on the assumption of known relation options, making it more reflective of real-world scenarios. Additionally, we have developed an easily extensible RE framework using a Parameters Efficient Fine Tuning (PEFT) algorithm (QLoRA). Our experiments on the RE-DocRED dataset showcase AutoRE's best performance, achieving state-of-the-art results, surpassing TAG by 10.03% and 9.03% respectively on the dev and test set.

Title: Stance Reasoner: Zero-Shot Stance Detection on Social Media with Explicit Reasoning

• Authors: Maksym Taranukhin, Vered Shwartz, Evangelos Milios
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14895
• Pdf URL: https://arxiv.org/pdf/2403.14895
• Copy Paste: [[2403.14895]] Stance Reasoner: Zero-Shot Stance Detection on Social Media with Explicit Reasoning(https://arxiv.org/abs/2403.14895)
• Keywords: extraction
• Abstract: Social media platforms are rich sources of opinionated content. Stance detection allows the automatic extraction of users' opinions on various topics from such content. We focus on zero-shot stance detection, where the model's success relies on (a) having knowledge about the target topic; and (b) learning general reasoning strategies that can be employed for new topics. We present Stance Reasoner, an approach to zero-shot stance detection on social media that leverages explicit reasoning over background knowledge to guide the model's inference about the document's stance on a target. Specifically, our method uses a pre-trained language model as a source of world knowledge, with the chain-of-thought in-context learning approach to generate intermediate reasoning steps. Stance Reasoner outperforms the current state-of-the-art models on 3 Twitter datasets, including fully supervised models. It can better generalize across targets, while at the same time providing explicit and interpretable explanations for its predictions.

Title: Geometric Generative Models based on Morphological Equivariant PDEs and GANs

• Authors: El Hadji S. Diop, Thierno Fall, Alioune Mbengue, Mohamed Daoudi
• Subjects: cs.CV, eess.IV, math.DG
• Abstract URL: https://arxiv.org/abs/2403.14897
• Pdf URL: https://arxiv.org/pdf/2403.14897
• Copy Paste: [[2403.14897]] Geometric Generative Models based on Morphological Equivariant PDEs and GANs(https://arxiv.org/abs/2403.14897)
• Keywords: extraction, interpretability, generative
• Abstract: Content and image generation consist in creating or generating data from noisy information by extracting specific features such as texture, edges, and other thin image structures. We are interested here in generative models, and two main problems are addressed. Firstly, the improvements of specific feature extraction while accounting at multiscale levels intrinsic geometric features; and secondly, the equivariance of the network to reduce its complexity and provide a geometric interpretability. To proceed, we propose a geometric generative model based on an equivariant partial differential equation (PDE) for group convolution neural networks (G-CNNs), so called PDE-G-CNNs, built on morphology operators and generative adversarial networks (GANs). Equivariant morphological PDE layers are composed of multiscale dilations and erosions formulated in Riemannian manifolds, while group symmetries are defined on a Lie group. We take advantage of the Lie group structure to properly integrate the equivariance in layers, and are able to use the Riemannian metric to solve the multiscale morphological operations. Each point of the Lie group is associated with a unique point in the manifold, which helps us derive a metric on the Riemannian manifold from a tensor field invariant under the Lie group so that the induced metric has the same symmetries. The proposed geometric morphological GAN (GM-GAN) is obtained by using the proposed morphological equivariant convolutions in PDE-G-CNNs to bring nonlinearity in classical CNNs. GM-GAN is evaluated on MNIST data and compared with GANs. Preliminary results show that GM-GAN model outperforms classical GAN.

Title: Web-based Melanoma Detection

• Authors: SangHyuk Kim, Edward Gaibor, Daniel Haehn
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14898
• Pdf URL: https://arxiv.org/pdf/2403.14898
• Copy Paste: [[2403.14898]] Web-based Melanoma Detection(https://arxiv.org/abs/2403.14898)
• Keywords: fair
• Abstract: Melanoma is the most aggressive form of skin cancer, and early detection can significantly increase survival rates and prevent cancer spread. However, developing reliable automated detection techniques is difficult due to the lack of standardized datasets and evaluation methods. This study introduces a unified melanoma classification approach that supports 54 combinations of 11 datasets and 24 state-of-the-art deep learning architectures. It enables a fair comparison of 1,296 experiments and results in a lightweight model deployable to the web-based MeshNet architecture named Mela-D. This approach can run up to 33x faster by reducing parameters 24x to yield an analogous 88.8\% accuracy comparable with ResNet50 on previously unseen images. This allows efficient and accurate melanoma detection in real-world settings that can run on consumer-level hardware.

Title: Snail: Secure Single Iteration Localization

• Authors: James Choncholas, Pujith Kachana, André Mateus, Gregoire Phillips, Ada Gavrilovska
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.14916
• Pdf URL: https://arxiv.org/pdf/2403.14916
• Copy Paste: [[2403.14916]] Snail: Secure Single Iteration Localization(https://arxiv.org/abs/2403.14916)
• Keywords: secure, security, privacy
• Abstract: Localization is a computer vision task by which the position and orientation of a camera is determined from an image and environmental map. We propose a method for performing localization in a privacy preserving manner supporting two scenarios: first, when the image and map are held by a client who wishes to offload localization to untrusted third parties, and second, when the image and map are held separately by untrusting parties. Privacy preserving localization is necessary when the image and map are confidential, and offloading conserves on-device power and frees resources for other tasks. To accomplish this we integrate existing localization methods and secure multi-party computation (MPC), specifically garbled circuits, yielding proof-based security guarantees in contrast to existing obfuscation-based approaches which recent related work has shown vulnerable. We present two approaches to localization, a baseline data-oblivious adaptation of localization suitable for garbled circuits and our novel Single Iteration Localization. Our technique improves overall performance while maintaining confidentiality of the input image, map, and output pose at the expense of increased communication rounds but reduced computation and communication required per round. Single Iteration Localization is over two orders of magnitude faster than a straightforward application of garbled circuits to localization enabling real-world usage in the first robot to offload localization without revealing input images, environmental map, position, or orientation to offload servers.

Title: CODA: A COst-efficient Test-time Domain Adaptation Mechanism for HAR

• Authors: Minghui Qiu (DSA, Hong Kong University of Science and Technology,Guangzhou), Yandao Huang (CSE, Hong Kong University of Science and Technology), Lin Chen (DSA, Hong Kong University of Science and Technology, Guangzhou), Lu Wang (CSSE, Shenzhen University), Kaishun Wu (DSA and IoT, Hong Kong University of Science and Technology, Guangzhou)
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.14922
• Pdf URL: https://arxiv.org/pdf/2403.14922
• Copy Paste: [[2403.14922]] CODA: A COst-efficient Test-time Domain Adaptation Mechanism for HAR(https://arxiv.org/abs/2403.14922)
• Keywords: robust
• Abstract: In recent years, emerging research on mobile sensing has led to novel scenarios that enhance daily life for humans, but dynamic usage conditions often result in performance degradation when systems are deployed in real-world settings. Existing solutions typically employ one-off adaptation schemes based on neural networks, which struggle to ensure robustness against uncertain drifting conditions in human-centric sensing scenarios. In this paper, we propose CODA, a COst-efficient Domain Adaptation mechanism for mobile sensing that addresses real-time drifts from the data distribution perspective with active learning theory, ensuring cost-efficient adaptation directly on the device. By incorporating a clustering loss and importance-weighted active learning algorithm, CODA retains the relationship between different clusters during cost-effective instance-level updates, preserving meaningful structure within the data distribution. We also showcase its generalization by seamlessly integrating it with Neural Network-based solutions for Human Activity Recognition tasks. Through meticulous evaluations across diverse datasets, including phone-based, watch-based, and integrated sensor-based sensing tasks, we demonstrate the feasibility and potential of online adaptation with CODA. The promising results achieved by CODA, even without learnable parameters, also suggest the possibility of realizing unobtrusive adaptation through specific application designs with sufficient feedback.

Title: Attention-Driven Reasoning: Unlocking the Potential of Large Language Models

• Authors: Bingli Liao, Danilo Vasconcellos Vargas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14932
• Pdf URL: https://arxiv.org/pdf/2403.14932
• Copy Paste: [[2403.14932]] Attention-Driven Reasoning: Unlocking the Potential of Large Language Models(https://arxiv.org/abs/2403.14932)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities, but their reasoning abilities and underlying mechanisms remain poorly understood. We present a novel approach to enhance LLMs' reasoning through attention mechanism optimization, without additional training data. We identify inefficiencies in the attention distribution caused by non-semantic tokens and propose an algorithm to re-balance the skewed distribution, enabling the model to abstract more nuanced knowledge. Our experiments demonstrate significantly improved reasoning capabilities, particularly for non-STEM questions. We provide insights into the role of attention patterns in LLMs' reasoning and propose a method to enhance these abilities, paving the way for more powerful and versatile language models.

Title: On Zero-Shot Counterspeech Generation by LLMs

• Authors: Punyajoy Saha, Aalok Agrawal, Abhik Jana, Chris Biemann, Animesh Mukherjee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14938
• Pdf URL: https://arxiv.org/pdf/2403.14938
• Copy Paste: [[2403.14938]] On Zero-Shot Counterspeech Generation by LLMs(https://arxiv.org/abs/2403.14938)
• Keywords: generative, large language model
• Abstract: With the emergence of numerous Large Language Models (LLM), the usage of such models in various Natural Language Processing (NLP) applications is increasing extensively. Counterspeech generation is one such key task where efforts are made to develop generative models by fine-tuning LLMs with hatespeech - counterspeech pairs, but none of these attempts explores the intrinsic properties of large language models in zero-shot settings. In this work, we present a comprehensive analysis of the performances of four LLMs namely GPT-2, DialoGPT, ChatGPT and FlanT5 in zero-shot settings for counterspeech generation, which is the first of its kind. For GPT-2 and DialoGPT, we further investigate the deviation in performance with respect to the sizes (small, medium, large) of the models. On the other hand, we propose three different prompting strategies for generating different types of counterspeech and analyse the impact of such strategies on the performance of the models. Our analysis shows that there is an improvement in generation quality for two datasets (17%), however the toxicity increase (25%) with increase in model size. Considering type of model, GPT-2 and FlanT5 models are significantly better in terms of counterspeech quality but also have high toxicity as compared to DialoGPT. ChatGPT are much better at generating counter speech than other models across all metrics. In terms of prompting, we find that our proposed strategies help in improving counter speech generation across all the models.

Title: STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians

• Authors: Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, Yao Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14939
• Pdf URL: https://arxiv.org/pdf/2403.14939
• Copy Paste: [[2403.14939]] STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians(https://arxiv.org/abs/2403.14939)
• Keywords: robust, diffusion, generative
• Abstract: Recent progress in pre-trained diffusion models and 3D generation have spurred interest in 4D content creation. However, achieving high-fidelity 4D generation with spatial-temporal consistency remains a challenge. In this work, we propose STAG4D, a novel framework that combines pre-trained diffusion models with dynamic 3D Gaussian splatting for high-fidelity 4D generation. Drawing inspiration from 3D generation techniques, we utilize a multi-view diffusion model to initialize multi-view images anchoring on the input video frames, where the video can be either real-world captured or generated by a video diffusion model. To ensure the temporal consistency of the multi-view sequence initialization, we introduce a simple yet effective fusion strategy to leverage the first frame as a temporal anchor in the self-attention computation. With the almost consistent multi-view sequences, we then apply the score distillation sampling to optimize the 4D Gaussian point cloud. The 4D Gaussian spatting is specially crafted for the generation task, where an adaptive densification strategy is proposed to mitigate the unstable Gaussian gradient for robust optimization. Notably, the proposed pipeline does not require any pre-training or fine-tuning of diffusion networks, offering a more accessible and practical solution for the 4D generation task. Extensive experiments demonstrate that our method outperforms prior 4D generation works in rendering quality, spatial-temporal consistency, and generation robustness, setting a new state-of-the-art for 4D generation from diverse inputs, including text, image, and video.

Title: Unifying Lane-Level Traffic Prediction from a Graph Structural Perspective: Benchmark and Baseline

• Authors: Shuhao Li, Yue Cui, Jingyi Xu, Libin Li, Lingkai Meng, Weidong Yang, Fan Zhang, Xiaofang Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14941
• Pdf URL: https://arxiv.org/pdf/2403.14941
• Copy Paste: [[2403.14941]] Unifying Lane-Level Traffic Prediction from a Graph Structural Perspective: Benchmark and Baseline(https://arxiv.org/abs/2403.14941)
• Keywords: fair
• Abstract: Traffic prediction has long been a focal and pivotal area in research, witnessing both significant strides from city-level to road-level predictions in recent years. With the advancement of Vehicle-to-Everything (V2X) technologies, autonomous driving, and large-scale models in the traffic domain, lane-level traffic prediction has emerged as an indispensable direction. However, further progress in this field is hindered by the absence of comprehensive and unified evaluation standards, coupled with limited public availability of data and code. This paper extensively analyzes and categorizes existing research in lane-level traffic prediction, establishes a unified spatial topology structure and prediction tasks, and introduces a simple baseline model, GraphMLP, based on graph structure and MLP networks. We have replicated codes not publicly available in existing studies and, based on this, thoroughly and fairly assessed various models in terms of effectiveness, efficiency, and applicability, providing insights for practical applications. Additionally, we have released three new datasets and corresponding codes to accelerate progress in this field, all of which can be found on https://github.com/ShuhaoLii/TITS24LaneLevel-Traffic-Benchmark.

Title: CLIP-VQDiffusion : Langauge Free Training of Text To Image generation using CLIP and vector quantized diffusion model

• Authors: Seungdae Han, Joohee Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14944
• Pdf URL: https://arxiv.org/pdf/2403.14944
• Copy Paste: [[2403.14944]] CLIP-VQDiffusion : Langauge Free Training of Text To Image generation using CLIP and vector quantized diffusion model(https://arxiv.org/abs/2403.14944)
• Keywords: diffusion
• Abstract: There has been a significant progress in text conditional image generation models. Recent advancements in this field depend not only on improvements in model structures, but also vast quantities of text-image paired datasets. However, creating these kinds of datasets is very costly and requires a substantial amount of labor. Famous face datasets don't have corresponding text captions, making it difficult to develop text conditional image generation models on these datasets. Some research has focused on developing text to image generation models using only images without text captions. Here, we propose CLIP-VQDiffusion, which leverage the pretrained CLIP model to provide multimodal text-image representations and strong image generation capabilities. On the FFHQ dataset, our model outperformed previous state-of-the-art methods by 4.4% in clipscore and generated very realistic images even when the text was both in and out of distribution. The pretrained models and codes will soon be available at https://github.com/INFINIQ-AI1/CLIPVQDiffusion

Title: KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation

• Authors: Xindi Luo, Zequn Sun, Jing Zhao, Zhe Zhao, Wei Hu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.14950
• Pdf URL: https://arxiv.org/pdf/2403.14950
• Copy Paste: [[2403.14950]] KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation(https://arxiv.org/abs/2403.14950)
• Keywords: robust, large language model
• Abstract: Parameter-efficient finetuning (PEFT) is a key technique for adapting large language models (LLMs) to downstream tasks. In this paper, we study leveraging knowledge graph embeddings to improve the effectiveness of PEFT. We propose a knowledgeable adaptation method called KnowLA. It inserts an adaptation layer into an LLM to integrate the embeddings of entities appearing in the input text. The adaptation layer is trained in combination with LoRA on instruction data. Experiments on six benchmarks with two popular LLMs and three knowledge graphs demonstrate the effectiveness and robustness of KnowLA. We show that \modelname can help activate the relevant parameterized knowledge in an LLM to answer a question without changing its parameters or input prompts.

Title: Evidence-Driven Retrieval Augmented Response Generation for Online Misinformation

• Authors: Zhenrui Yue, Huimin Zeng, Yimeng Lu, Lanyu Shang, Yang Zhang, Dong Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.14952
• Pdf URL: https://arxiv.org/pdf/2403.14952
• Copy Paste: [[2403.14952]] Evidence-Driven Retrieval Augmented Response Generation for Online Misinformation(https://arxiv.org/abs/2403.14952)
• Keywords: large language model
• Abstract: The proliferation of online misinformation has posed significant threats to public interest. While numerous online users actively participate in the combat against misinformation, many of such responses can be characterized by the lack of politeness and supporting facts. As a solution, text generation approaches are proposed to automatically produce counter-misinformation responses. Nevertheless, existing methods are often trained end-to-end without leveraging external knowledge, resulting in subpar text quality and excessively repetitive responses. In this paper, we propose retrieval augmented response generation for online misinformation (RARG), which collects supporting evidence from scientific sources and generates counter-misinformation responses based on the evidences. In particular, our RARG consists of two stages: (1) evidence collection, where we design a retrieval pipeline to retrieve and rerank evidence documents using a database comprising over 1M academic articles; (2) response generation, in which we align large language models (LLMs) to generate evidence-based responses via reinforcement learning from human feedback (RLHF). We propose a reward function to maximize the utilization of the retrieved evidence while maintaining the quality of the generated text, which yields polite and factual responses that clearly refutes misinformation. To demonstrate the effectiveness of our method, we study the case of COVID-19 and perform extensive experiments with both in- and cross-domain datasets, where RARG consistently outperforms baselines by generating high-quality counter-misinformation responses.

Title: Enabling Physical Localization of Uncooperative Cellular Devices

• Authors: Taekkyung Oh, Sangwook Bae, Junho Ahn, Yonghwa Lee, Dinh-Tuan Hoang, Min Suk Kang, Nils Ole Tippenhauer, Yongdae Kim
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.14963
• Pdf URL: https://arxiv.org/pdf/2403.14963
• Copy Paste: [[2403.14963]] Enabling Physical Localization of Uncooperative Cellular Devices(https://arxiv.org/abs/2403.14963)
• Keywords: attack
• Abstract: In cellular networks, it can become necessary for authorities to physically locate user devices for tracking criminals or illegal devices. While cellular operators can provide authorities with cell information the device is camping on, fine-grained localization is still required. Therefore, the authorized agents trace the device by monitoring its uplink signals. However, tracking the uplink signal source without its cooperation is challenging even for operators and authorities. Particularly, three challenges remain for fine-grained localization: i) localization works only if devices generate enough uplink traffic reliably over time, ii) the target device might generate its uplink traffic with significantly low power, and iii) cellular repeater may add too much noise to true uplink signals. While these challenges present practical hurdles for localization, they have been overlooked in prior works. In this work, we investigate the impact of these real-world challenges on cellular localization and propose an Uncooperative Multiangulation Attack (UMA) that addresses these challenges. UMA can 1) force a target device to transmit traffic continuously, 2) boost the target's signal strength to the maximum, and 3) uniquely distinguish traffic from the target and the repeaters. Notably, the UMA technique works without privilege on cellular operators or user devices, which makes it operate on any LTE network. Our evaluations show that UMA effectively resolves the challenges in real-world environments when devices are not cooperative for localization. Our approach exploits the current cellular design vulnerabilities, which we have responsibly disclosed to GSMA.

Title: DreamFlow: High-Quality Text-to-3D Generation by Approximating Probability Flow

• Authors: Kyungmin Lee, Kihyuk Sohn, Jinwoo Shin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14966
• Pdf URL: https://arxiv.org/pdf/2403.14966
• Copy Paste: [[2403.14966]] DreamFlow: High-Quality Text-to-3D Generation by Approximating Probability Flow(https://arxiv.org/abs/2403.14966)
• Keywords: diffusion, generative
• Abstract: Recent progress in text-to-3D generation has been achieved through the utilization of score distillation methods: they make use of the pre-trained text-to-image (T2I) diffusion models by distilling via the diffusion model training objective. However, such an approach inevitably results in the use of random timesteps at each update, which increases the variance of the gradient and ultimately prolongs the optimization process. In this paper, we propose to enhance the text-to-3D optimization by leveraging the T2I diffusion prior in the generative sampling process with a predetermined timestep schedule. To this end, we interpret text-to3D optimization as a multi-view image-to-image translation problem, and propose a solution by approximating the probability flow. By leveraging the proposed novel optimization algorithm, we design DreamFlow, a practical three-stage coarseto-fine text-to-3D optimization framework that enables fast generation of highquality and high-resolution (i.e., 1024x1024) 3D contents. For example, we demonstrate that DreamFlow is 5 times faster than the existing state-of-the-art text-to-3D method, while producing more photorealistic 3D contents. Visit our project page (https://kyungmnlee.github.io/dreamflow.github.io/) for visualizations.

Title: AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies

• Authors: Rui Wang, Dengpan Ye, Long Tang, Yunming Zhang, Jiacheng Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14974
• Pdf URL: https://arxiv.org/pdf/2403.14974
• Copy Paste: [[2403.14974]] AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies(https://arxiv.org/abs/2403.14974)
• Keywords: transformer
• Abstract: With the continuous improvements of deepfake methods, forgery messages have transitioned from single-modality to multi-modal fusion, posing new challenges for existing forgery detection algorithms. In this paper, we propose AVT2-DWF, the Audio-Visual dual Transformers grounded in Dynamic Weight Fusion, which aims to amplify both intra- and cross-modal forgery cues, thereby enhancing detection capabilities. AVT2-DWF adopts a dual-stage approach to capture both spatial characteristics and temporal dynamics of facial expressions. This is achieved through a face transformer with an n-frame-wise tokenization strategy encoder and an audio transformer encoder. Subsequently, it uses multi-modal conversion with dynamic weight fusion to address the challenge of heterogeneous information fusion between audio and visual modalities. Experiments on DeepfakeTIMIT, FakeAVCeleb, and DFDC datasets indicate that AVT2-DWF achieves state-of-the-art performance intra- and cross-dataset Deepfake detection. Code is available at https://github.com/raining-dev/AVT2-DWF.

Title: MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughts

• Authors: Md Nishat Raihan, Dhiman Goswami, Al Nahian Bin Emran, Sadiya Sayara Chowdhury Puspo, Amrita Ganguly, Marcos Zampieri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14982
• Pdf URL: https://arxiv.org/pdf/2403.14982
• Copy Paste: [[2403.14982]] MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughts(https://arxiv.org/abs/2403.14982)
• Keywords: large language model
• Abstract: Our paper presents team MasonTigers submission to the SemEval-2024 Task 9 - which provides a dataset of puzzles for testing natural language understanding. We employ large language models (LLMs) to solve this task through several prompting techniques. Zero-shot and few-shot prompting generate reasonably good results when tested with proprietary LLMs, compared to the open-source models. We obtain further improved results with chain-of-thought prompting, an iterative prompting method that breaks down the reasoning process step-by-step. We obtain our best results by utilizing an ensemble of chain-of-thought prompts, placing 2nd in the word puzzle subtask and 13th in the sentence puzzle subtask. The strong performance of prompted LLMs demonstrates their capability for complex reasoning when provided with a decomposition of the thought process. Our work sheds light on how step-wise explanatory prompts can unlock more of the knowledge encoded in the parameters of large models.

Title: FileDES: A Secure Scalable and Succinct Decentralized Encrypted Storage Network

• Authors: Minghui Xu, Jiahao Zhang, Hechuan Guo, Xiuzhen Cheng, Dongxiao Yu, Qin Hu, Yijun Li, Yipu Wu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.14985
• Pdf URL: https://arxiv.org/pdf/2403.14985
• Copy Paste: [[2403.14985]] FileDES: A Secure Scalable and Succinct Decentralized Encrypted Storage Network(https://arxiv.org/abs/2403.14985)
• Keywords: secure, privacy, attack
• Abstract: Decentralized Storage Network (DSN) is an emerging technology that challenges traditional cloud-based storage systems by consolidating storage capacities from independent providers and coordinating to provide decentralized storage and retrieval services. However, current DSNs face several challenges associated with data privacy and efficiency of the proof systems. To address these issues, we propose FileDES (\uline{D}ecentralized \uline{E}ncrypted \uline{S}torage), which incorporates three essential elements: privacy preservation, scalable storage proof, and batch verification. FileDES provides encrypted data storage while maintaining data availability, with a scalable Proof of Encrypted Storage (PoES) algorithm that is resilient to Sybil and Generation attacks. Additionally, we introduce a rollup-based batch verification approach to simultaneously verify multiple files using publicly verifiable succinct proofs. We conducted a comparative evaluation on FileDES, Filecoin, Storj and Sia under various conditions, including a WAN composed of up to 120 geographically dispersed nodes. Our protocol outperforms the others in terms of proof generation/verification efficiency, storage costs, and scalability.

Title: Generative Active Learning for Image Synthesis Personalization

• Authors: Xulu Zhang, Wengyu Zhang, Xiao-Yong Wei, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, Qing Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14987
• Pdf URL: https://arxiv.org/pdf/2403.14987
• Copy Paste: [[2403.14987]] Generative Active Learning for Image Synthesis Personalization(https://arxiv.org/abs/2403.14987)
• Keywords: generative
• Abstract: This paper presents a pilot study that explores the application of active learning, traditionally studied in the context of discriminative models, to generative models. We specifically focus on image synthesis personalization tasks. The primary challenge in conducting active learning on generative models lies in the open-ended nature of querying, which differs from the closed form of querying in discriminative models that typically target a single concept. We introduce the concept of anchor directions to transform the querying process into a semi-open problem. We propose a direction-based uncertainty sampling strategy to enable generative active learning and tackle the exploitation-exploration dilemma. Extensive experiments are conducted to validate the effectiveness of our approach, demonstrating that an open-source model can achieve superior performance compared to closed-source models developed by large companies, such as Google's StyleDrop. The source code is available at https://github.com/zhangxulu1996/GAL4Personalization.

Title: Risk and Response in Large Language Models: Evaluating Key Threat Categories

• Authors: Bahareh Harandizadeh, Abel Salinas, Fred Morstatter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14988
• Pdf URL: https://arxiv.org/pdf/2403.14988
• Copy Paste: [[2403.14988]] Risk and Response in Large Language Models: Evaluating Key Threat Categories(https://arxiv.org/abs/2403.14988)
• Keywords: security, attack, large language model
• Abstract: This paper explores the pressing issue of risk assessment in Large Language Models (LLMs) as they become increasingly prevalent in various applications. Focusing on how reward models, which are designed to fine-tune pretrained LLMs to align with human values, perceive and categorize different types of risks, we delve into the challenges posed by the subjective nature of preference-based training data. By utilizing the Anthropic Red-team dataset, we analyze major risk categories, including Information Hazards, Malicious Uses, and Discrimination/Hateful content. Our findings indicate that LLMs tend to consider Information Hazards less harmful, a finding confirmed by a specially developed regression model. Additionally, our analysis shows that LLMs respond less stringently to Information Hazards compared to other risks. The study further reveals a significant vulnerability of LLMs to jailbreaking attacks in Information Hazard scenarios, highlighting a critical security concern in LLM risk assessment and emphasizing the need for improved AI safety measures.

Title: MasonTigers at SemEval-2024 Task 8: Performance Analysis of Transformer-based Models on Machine-Generated Text Detection

• Authors: Sadiya Sayara Chowdhury Puspo, Md Nishat Raihan, Dhiman Goswami, Al Nahian Bin Emran, Amrita Ganguly, Ozlem Uzuner
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14989
• Pdf URL: https://arxiv.org/pdf/2403.14989
• Copy Paste: [[2403.14989]] MasonTigers at SemEval-2024 Task 8: Performance Analysis of Transformer-based Models on Machine-Generated Text Detection(https://arxiv.org/abs/2403.14989)
• Keywords: transformer
• Abstract: This paper presents the MasonTigers entry to the SemEval-2024 Task 8 - Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection. The task encompasses Binary Human-Written vs. Machine-Generated Text Classification (Track A), Multi-Way Machine-Generated Text Classification (Track B), and Human-Machine Mixed Text Detection (Track C). Our best performing approaches utilize mainly the ensemble of discriminator transformer models along with sentence transformer and statistical machine learning approaches in specific cases. Moreover, zero-shot prompting and fine-tuning of FLAN-T5 are used for Track A and B.

Title: MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness

• Authors: Dhiman Goswami, Sadiya Sayara Chowdhury Puspo, Md Nishat Raihan, Al Nahian Bin Emran, Amrita Ganguly, Marcos Zampieri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.14990
• Pdf URL: https://arxiv.org/pdf/2403.14990
• Copy Paste: [[2403.14990]] MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness(https://arxiv.org/abs/2403.14990)
• Keywords: transformer
• Abstract: This paper presents the MasonTigers entry to the SemEval-2024 Task 1 - Semantic Textual Relatedness. The task encompasses supervised (Track A), unsupervised (Track B), and cross-lingual (Track C) approaches across 14 different languages. MasonTigers stands out as one of the two teams who participated in all languages across the three tracks. Our approaches achieved rankings ranging from 11th to 21st in Track A, from 1st to 8th in Track B, and from 5th to 12th in Track C. Adhering to the task-specific constraints, our best performing approaches utilize ensemble of statistical machine learning approaches combined with language-specific BERT based models and sentence transformers.

Title: Improve Cross-domain Mixed Sampling with Guidance Training for Adaptive Segmentation

• Authors: Wenlve Zhou, Zhiheng Zhou, Tianlei Wang, Delu Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.14995
• Pdf URL: https://arxiv.org/pdf/2403.14995
• Copy Paste: [[2403.14995]] Improve Cross-domain Mixed Sampling with Guidance Training for Adaptive Segmentation(https://arxiv.org/abs/2403.14995)
• Keywords: segmentation
• Abstract: Unsupervised Domain Adaptation (UDA) endeavors to adjust models trained on a source domain to perform well on a target domain without requiring additional annotations. In the context of domain adaptive semantic segmentation, which tackles UDA for dense prediction, the goal is to circumvent the need for costly pixel-level annotations. Typically, various prevailing methods baseline rely on constructing intermediate domains via cross-domain mixed sampling techniques to mitigate the performance decline caused by domain gaps. However, such approaches generate synthetic data that diverge from real-world distributions, potentially leading the model astray from the true target distribution. To address this challenge, we propose a novel auxiliary task called Guidance Training. This task facilitates the effective utilization of cross-domain mixed sampling techniques while mitigating distribution shifts from the real world. Specifically, Guidance Training guides the model to extract and reconstruct the target-domain feature distribution from mixed data, followed by decoding the reconstructed target-domain features to make pseudo-label predictions. Importantly, integrating Guidance Training incurs minimal training overhead and imposes no additional inference burden. We demonstrate the efficacy of our approach by integrating it with existing methods, consistently improving performance. The implementation will be available at https://github.com/Wenlve-Zhou/Guidance-Training.

Title: Magic for the Age of Quantized DNNs

• Authors: Yoshihide Sawada, Ryuji Saiin, Kazuma Suetake
• Subjects: cs.LG, cs.AI, cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2403.14999
• Pdf URL: https://arxiv.org/pdf/2403.14999
• Copy Paste: [[2403.14999]] Magic for the Age of Quantized DNNs(https://arxiv.org/abs/2403.14999)
• Keywords: large language model
• Abstract: Recently, the number of parameters in DNNs has explosively increased, as exemplified by LLMs (Large Language Models), making inference on small-scale computers more difficult. Model compression technology is, therefore, essential for integration into products. In this paper, we propose a method of quantization-aware training. We introduce a novel normalization (Layer-Batch Normalization) that is independent of the mini-batch size and does not require any additional computation cost during inference. Then, we quantize the weights by the scaled round-clip function with the weight standardization. We also quantize activation functions using the same function and apply surrogate gradients to train the model with both quantized weights and the quantized activation functions. We call this method Magic for the age of Quantised DNNs (MaQD). Experimental results show that our quantization method can be achieved with minimal accuracy degradation.

Title: ParFormer: Vision Transformer Baseline with Parallel Local Global Token Mixer and Convolution Attention Patch Embedding

• Authors: Novendra Setyawan, Ghufron Wahyu Kurniawan, Chi-Chia Sun, Jun-Wei Hsieh, Hui-Kai Su, Wen-Kai Kuo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15004
• Pdf URL: https://arxiv.org/pdf/2403.15004
• Copy Paste: [[2403.15004]] ParFormer: Vision Transformer Baseline with Parallel Local Global Token Mixer and Convolution Attention Patch Embedding(https://arxiv.org/abs/2403.15004)
• Keywords: extraction, transformer
• Abstract: This work presents ParFormer as an enhanced transformer architecture that allows the incorporation of different token mixers into a single stage, hence improving feature extraction capabilities. Integrating both local and global data allows for precise representation of short- and long-range spatial relationships without the need for computationally intensive methods such as shifting windows. Along with the parallel token mixer encoder, We offer the Convolutional Attention Patch Embedding (CAPE) as an enhancement of standard patch embedding to improve token mixer extraction with a convolutional attention module. Our comprehensive evaluation demonstrates that our ParFormer outperforms CNN-based and state-of-the-art transformer-based architectures in image classification and several complex tasks such as object recognition. The proposed CAPE has been demonstrated to benefit the overall MetaFormer architecture, even while utilizing the Identity Mapping Token Mixer, resulting in a 0.5\% increase in accuracy. The ParFormer models outperformed ConvNeXt and Swin Transformer for the pure convolution and transformer model in accuracy. Furthermore, our model surpasses the current leading hybrid transformer by reaching competitive Top-1 scores in the ImageNet-1K classification test. Specifically, our model variants with 11M, 23M, and 34M parameters achieve scores of 80.4\%, 82.1\%, and 83.1\%, respectively. Code: https://github.com/novendrastywn/ParFormer-CAPE-2024

Title: Clean-image Backdoor Attacks

• Authors: Dazhong Rong, Shuheng Shen, Xinyi Fu, Peng Qian, Jianhai Chen, Qinming He, Xing Fu, Weiqiang Wang
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15010
• Pdf URL: https://arxiv.org/pdf/2403.15010
• Copy Paste: [[2403.15010]] Clean-image Backdoor Attacks(https://arxiv.org/abs/2403.15010)
• Keywords: secure, attack, robust, fair
• Abstract: To gather a significant quantity of annotated training data for high-performance image classification models, numerous companies opt to enlist third-party providers to label their unlabeled data. This practice is widely regarded as secure, even in cases where some annotated errors occur, as the impact of these minor inaccuracies on the final performance of the models is negligible and existing backdoor attacks require attacker's ability to poison the training images. Nevertheless, in this paper, we propose clean-image backdoor attacks which uncover that backdoors can still be injected via a fraction of incorrect labels without modifying the training images. Specifically, in our attacks, the attacker first seeks a trigger feature to divide the training images into two parts: those with the feature and those without it. Subsequently, the attacker falsifies the labels of the former part to a backdoor class. The backdoor will be finally implanted into the target model after it is trained on the poisoned data. During the inference phase, the attacker can activate the backdoor in two ways: slightly modifying the input image to obtain the trigger feature, or taking an image that naturally has the trigger feature as input. We conduct extensive experiments to demonstrate the effectiveness and practicality of our attacks. According to the experimental results, we conclude that our attacks seriously jeopardize the fairness and robustness of image classification models, and it is necessary to be vigilant about the incorrect labels in outsourced labeling.

Title: Cell Tracking according to Biological Needs -- Strong Mitosis-aware Random-finite Sets Tracker with Aleatoric Uncertainty

• Authors: Timo Kaiser, Maximilian Schier, Bodo Rosenhahn
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15011
• Pdf URL: https://arxiv.org/pdf/2403.15011
• Copy Paste: [[2403.15011]] Cell Tracking according to Biological Needs -- Strong Mitosis-aware Random-finite Sets Tracker with Aleatoric Uncertainty(https://arxiv.org/abs/2403.15011)
• Keywords: segmentation
• Abstract: Cell tracking and segmentation assist biologists in extracting insights from large-scale microscopy time-lapse data. Driven by local accuracy metrics, current tracking approaches often suffer from a lack of long-term consistency. To address this issue, we introduce an uncertainty estimation technique for neural tracking-by-regression frameworks and incorporate it into our novel extended Poisson multi-Bernoulli mixture tracker. Our uncertainty estimation identifies uncertain associations within high-performing tracking-by-regression methods using problem-specific test-time augmentations. Leveraging this uncertainty, along with a novel mitosis-aware assignment problem formulation, our tracker resolves false associations and mitosis detections stemming from long-term conflicts. We evaluate our approach on nine competitive datasets and demonstrate that it outperforms the current state-of-the-art on biologically relevant metrics substantially, achieving improvements by a factor of approximately $5.75$. Furthermore, we uncover new insights into the behavior of tracking-by-regression uncertainty.

Title: Vehicle Detection Performance in Nordic Region

• Authors: Hamam Mokayed, Rajkumar Saini, Oluwatosin Adewumi, Lama Alkhaled, Bjorn Backe, Palaiahnakote Shivakumara, Olle Hagner, Yan Chai Hum
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15017
• Pdf URL: https://arxiv.org/pdf/2403.15017
• Copy Paste: [[2403.15017]] Vehicle Detection Performance in Nordic Region(https://arxiv.org/abs/2403.15017)
• Keywords: robust, transformer
• Abstract: This paper addresses the critical challenge of vehicle detection in the harsh winter conditions in the Nordic regions, characterized by heavy snowfall, reduced visibility, and low lighting. Due to their susceptibility to environmental distortions and occlusions, traditional vehicle detection methods have struggled in these adverse conditions. The advanced proposed deep learning architectures brought promise, yet the unique difficulties of detecting vehicles in Nordic winters remain inadequately addressed. This study uses the Nordic Vehicle Dataset (NVD), which has UAV images from northern Sweden, to evaluate the performance of state-of-the-art vehicle detection algorithms under challenging weather conditions. Our methodology includes a comprehensive evaluation of single-stage, two-stage, and transformer-based detectors against the NVD. We propose a series of enhancements tailored to each detection framework, including data augmentation, hyperparameter tuning, transfer learning, and novel strategies designed explicitly for the DETR model. Our findings not only highlight the limitations of current detection systems in the Nordic environment but also offer promising directions for enhancing these algorithms for improved robustness and accuracy in vehicle detection amidst the complexities of winter landscapes. The code and the dataset are available at https://nvd.ltu-ai.dev

Title: BSNet: Box-Supervised Simulation-assisted Mean Teacher for 3D Instance Segmentation

• Authors: Jiahao Lu, Jiacheng Deng, Tianzhu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15019
• Pdf URL: https://arxiv.org/pdf/2403.15019
• Copy Paste: [[2403.15019]] BSNet: Box-Supervised Simulation-assisted Mean Teacher for 3D Instance Segmentation(https://arxiv.org/abs/2403.15019)
• Keywords: transformer, segmentation
• Abstract: 3D instance segmentation (3DIS) is a crucial task, but point-level annotations are tedious in fully supervised settings. Thus, using bounding boxes (bboxes) as annotations has shown great potential. The current mainstream approach is a two-step process, involving the generation of pseudo-labels from box annotations and the training of a 3DIS network with the pseudo-labels. However, due to the presence of intersections among bboxes, not every point has a determined instance label, especially in overlapping areas. To generate higher quality pseudo-labels and achieve more precise weakly supervised 3DIS results, we propose the Box-Supervised Simulation-assisted Mean Teacher for 3D Instance Segmentation (BSNet), which devises a novel pseudo-labeler called Simulation-assisted Transformer. The labeler consists of two main components. The first is Simulation-assisted Mean Teacher, which introduces Mean Teacher for the first time in this task and constructs simulated samples to assist the labeler in acquiring prior knowledge about overlapping areas. To better model local-global structure, we also propose Local-Global Aware Attention as the decoder for teacher and student labelers. Extensive experiments conducted on the ScanNetV2 and S3DIS datasets verify the superiority of our designs. Code is available at \href{https://github.com/peoplelu/BSNet}{https://github.com/peoplelu/BSNet}.

Title: Robust Conformal Prediction under Distribution Shift via Physics-Informed Structural Causal Model

• Authors: Rui Xu, Yue Sun, Chao Chen, Parv Venkitasubramaniam, Sihong Xie
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.15025
• Pdf URL: https://arxiv.org/pdf/2403.15025
• Copy Paste: [[2403.15025]] Robust Conformal Prediction under Distribution Shift via Physics-Informed Structural Causal Model(https://arxiv.org/abs/2403.15025)
• Keywords: robust
• Abstract: Uncertainty is critical to reliable decision-making with machine learning. Conformal prediction (CP) handles uncertainty by predicting a set on a test input, hoping the set to cover the true label with at least $(1-\alpha)$ confidence. This coverage can be guaranteed on test data even if the marginal distributions $P_X$ differ between calibration and test datasets. However, as it is common in practice, when the conditional distribution $P_{Y|X}$ is different on calibration and test data, the coverage is not guaranteed and it is essential to measure and minimize the coverage loss under distributional shift at \textit{all} possible confidence levels. To address these issues, we upper bound the coverage difference at all levels using the cumulative density functions of calibration and test conformal scores and Wasserstein distance. Inspired by the invariance of physics across data distributions, we propose a physics-informed structural causal model (PI-SCM) to reduce the upper bound. We validated that PI-SCM can improve coverage robustness along confidence level and test domain on a traffic speed prediction task and an epidemic spread task with multiple real-world datasets.

Title: Grey-informed neural network for time-series forecasting

• Authors: Wanli Xie, Ruibin Zhao, Zhenguo Xu, Tingting Liang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15027
• Pdf URL: https://arxiv.org/pdf/2403.15027
• Copy Paste: [[2403.15027]] Grey-informed neural network for time-series forecasting(https://arxiv.org/abs/2403.15027)
• Keywords: interpretability
• Abstract: Neural network models have shown outstanding performance and successful resolutions to complex problems in various fields. However, the majority of these models are viewed as black-box, requiring a significant amount of data for development. Consequently, in situations with limited data, constructing appropriate models becomes challenging due to the lack of transparency and scarcity of data. To tackle these challenges, this study suggests the implementation of a grey-informed neural network (GINN). The GINN ensures that the output of the neural network follows the differential equation model of the grey system, improving interpretability. Moreover, incorporating prior knowledge from grey system theory enables traditional neural networks to effectively handle small data samples. Our proposed model has been observed to uncover underlying patterns in the real world and produce reliable forecasts based on empirical data.

Title: Tie-Breaking Rule Based on Partial Proof of Work in a Blockchain

• Authors: Akira Sakurai, Kazuyuki Shudo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15030
• Pdf URL: https://arxiv.org/pdf/2403.15030
• Copy Paste: [[2403.15030]] Tie-Breaking Rule Based on Partial Proof of Work in a Blockchain(https://arxiv.org/abs/2403.15030)
• Keywords: attack
• Abstract: Numerous methods have been proposed for suppressing intentional forks by attackers in blockchain systems. Among these, last-generated rules, which select the latest chain among chains in a tie, are effective methods that do not require significant changes to the blockchain protocol. However, existing methods either require a trusted third party or rely on timestamps that attackers can manipulate which makes applying a last-generated rule to existing systems such as Bitcoin challenging. To address these issues, we propose a last-generated rule that can be easily applied to existing proof of work blockchain systems. Our method uses partial proof of work, which does not function as a block, as a time standard with finer granularity. Only weak synchronization, which is already met by existing systems, is required for effective functioning. We evaluated the proposed method through a detailed analysis that is lacking in existing works. In networks that adopt our method, the proportion of the attacker hashrate necessary for selfish mining was approximately 0.31479 or higher, regardless of the block propagation capability of the attacker. Furthermore, we demonstrated through extended selfish mining that the impact of Match against pre-generated block, which is a concern in all last-generated rules, can be mitigated with appropriate parameter settings.

Title: An Integrated Neighborhood and Scale Information Network for Open-Pit Mine Change Detection in High-Resolution Remote Sensing Images

• Authors: Zilin Xie, Kangning Li, Jinbao Jiang, Jinzhong Yang, Xiaojun Qiao, Deshuai Yuan, Cheng Nie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15032
• Pdf URL: https://arxiv.org/pdf/2403.15032
• Copy Paste: [[2403.15032]] An Integrated Neighborhood and Scale Information Network for Open-Pit Mine Change Detection in High-Resolution Remote Sensing Images(https://arxiv.org/abs/2403.15032)
• Keywords: protect, extraction
• Abstract: Open-pit mine change detection (CD) in high-resolution (HR) remote sensing images plays a crucial role in mineral development and environmental protection. Significant progress has been made in this field in recent years, largely due to the advancement of deep learning techniques. However, existing deep-learning-based CD methods encounter challenges in effectively integrating neighborhood and scale information, resulting in suboptimal performance. Therefore, by exploring the influence patterns of neighborhood and scale information, this paper proposes an Integrated Neighborhood and Scale Information Network (INSINet) for open-pit mine CD in HR remote sensing images. Specifically, INSINet introduces 8-neighborhood-image information to acquire a larger receptive field, improving the recognition of center image boundary regions. Drawing on techniques of skip connection, deep supervision, and attention mechanism, the multi-path deep supervised attention (MDSA) module is designed to enhance multi-scale information fusion and change feature extraction. Experimental analysis reveals that incorporating neighborhood and scale information enhances the F1 score of INSINet by 6.40%, with improvements of 3.08% and 3.32% respectively. INSINet outperforms existing methods with an Overall Accuracy of 97.69%, Intersection over Union of 71.26%, and F1 score of 83.22%. INSINet shows significance for open-pit mine CD in HR remote sensing images.

Title: Toward Tiny and High-quality Facial Makeup with Data Amplify Learning

• Authors: Qiaoqiao Jin, Xuanhong Chen, Meiguang Jin, Ying Cheng, Rui Shi, Yucheng Zheng, Yupeng Zhu, Bingbing Ni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15033
• Pdf URL: https://arxiv.org/pdf/2403.15033
• Copy Paste: [[2403.15033]] Toward Tiny and High-quality Facial Makeup with Data Amplify Learning(https://arxiv.org/abs/2403.15033)
• Keywords: diffusion
• Abstract: Contemporary makeup approaches primarily hinge on unpaired learning paradigms, yet they grapple with the challenges of inaccurate supervision (e.g., face misalignment) and sophisticated facial prompts (including face parsing, and landmark detection). These challenges prohibit low-cost deployment of facial makeup models, especially on mobile devices. To solve above problems, we propose a brand-new learning paradigm, termed "Data Amplify Learning (DAL)," alongside a compact makeup model named "TinyBeauty." The core idea of DAL lies in employing a Diffusion-based Data Amplifier (DDA) to "amplify" limited images for the model training, thereby enabling accurate pixel-to-pixel supervision with merely a handful of annotations. Two pivotal innovations in DDA facilitate the above training approach: (1) A Residual Diffusion Model (RDM) is designed to generate high-fidelity detail and circumvent the detail vanishing problem in the vanilla diffusion models; (2) A Fine-Grained Makeup Module (FGMM) is proposed to achieve precise makeup control and combination while retaining face identity. Coupled with DAL, TinyBeauty necessitates merely 80K parameters to achieve a state-of-the-art performance without intricate face prompts. Meanwhile, TinyBeauty achieves a remarkable inference speed of up to 460 fps on the iPhone 13. Extensive experiments show that DAL can produce highly competitive makeup models using only 5 image pairs.

Title: LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

• Authors: Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W. Mahoney, Kurt Keutzer, Amir Gholami
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15042
• Pdf URL: https://arxiv.org/pdf/2403.15042
• Copy Paste: [[2403.15042]] LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement(https://arxiv.org/abs/2403.15042)
• Keywords: large language model
• Abstract: Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the low-data regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a teacher LLM to generate synthetic data based on these incorrect data points, which are then added back into the training data. This approach amplifies the signal from incorrectly predicted data points by the LLM during training and reintegrates them into the dataset to focus on more challenging examples for the LLM. Our results show that LLM2LLM significantly enhances the performance of LLMs in the low-data regime, outperforming both traditional fine-tuning and other data augmentation baselines. LLM2LLM reduces the dependence on labor-intensive data curation and paves the way for more scalable and performant LLM solutions, allowing us to tackle data-constrained domains and tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime using a LLaMA2-7B student model.

Title: DP-Dueling: Learning from Preference Feedback without Compromising User Privacy

• Authors: Aadirupa Saha, Hilal Asi
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15045
• Pdf URL: https://arxiv.org/pdf/2403.15045
• Copy Paste: [[2403.15045]] DP-Dueling: Learning from Preference Feedback without Compromising User Privacy(https://arxiv.org/abs/2403.15045)
• Keywords: privacy
• Abstract: We consider the well-studied dueling bandit problem, where a learner aims to identify near-optimal actions using pairwise comparisons, under the constraint of differential privacy. We consider a general class of utility-based preference matrices for large (potentially unbounded) decision spaces and give the first differentially private dueling bandit algorithm for active learning with user preferences. Our proposed algorithms are computationally efficient with near-optimal performance, both in terms of the private and non-private regret bound. More precisely, we show that when the decision space is of finite size $K$, our proposed algorithm yields order optimal $O\Big(\sum_{i = 2}^K\log\frac{KT}{\Delta_i} + \frac{K}{\epsilon}\Big)$ regret bound for pure $\epsilon$-DP, where $\Delta_i$ denotes the suboptimality gap of the $i$-th arm. We also present a matching lower bound analysis which proves the optimality of our algorithms. Finally, we extend our results to any general decision space in $d$-dimensions with potentially infinite arms and design an $\epsilon$-DP algorithm with regret $\tilde{O} \left( \frac{d^6}{\kappa \epsilon } + \frac{ d\sqrt{T }}{\kappa} \right)$, providing privacy for free when $T \gg d$.

Title: Cartoon Hallucinations Detection: Pose-aware In Context Visual Learning

• Authors: Bumsoo Kim, Wonseop Shin, Kyuchul Lee, Sanghyun Seo
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.15048
• Pdf URL: https://arxiv.org/pdf/2403.15048
• Copy Paste: [[2403.15048]] Cartoon Hallucinations Detection: Pose-aware In Context Visual Learning(https://arxiv.org/abs/2403.15048)
• Keywords: generative
• Abstract: Large-scale Text-to-Image (TTI) models have become a common approach for generating training data in various generative fields. However, visual hallucinations, which contain perceptually critical defects, remain a concern, especially in non-photorealistic styles like cartoon characters. We propose a novel visual hallucination detection system for cartoon character images generated by TTI models. Our approach leverages pose-aware in-context visual learning (PA-ICVL) with Vision-Language Models (VLMs), utilizing both RGB images and pose information. By incorporating pose guidance from a fine-tuned pose estimator, we enable VLMs to make more accurate decisions. Experimental results demonstrate significant improvements in identifying visual hallucinations compared to baseline methods relying solely on RGB images. This research advances TTI models by mitigating visual hallucinations, expanding their potential in non-photorealistic domains.

Title: The Power of Bamboo: On the Post-Compromise Security for Searchable Symmetric Encryption

• Authors: Tianyang Chen, Peng Xu, Stjepan Picek, Bo Luo, Willy Susilo, Hai Jin, Kaitai Liang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15052
• Pdf URL: https://arxiv.org/pdf/2403.15052
• Copy Paste: [[2403.15052]] The Power of Bamboo: On the Post-Compromise Security for Searchable Symmetric Encryption(https://arxiv.org/abs/2403.15052)
• Keywords: secure, security, privacy
• Abstract: Dynamic searchable symmetric encryption (DSSE) enables users to delegate the keyword search over dynamically updated encrypted databases to an honest-but-curious server without losing keyword privacy. This paper studies a new and practical security risk to DSSE, namely, secret key compromise (e.g., a user's secret key is leaked or stolen), which threatens all the security guarantees offered by existing DSSE schemes. To address this open problem, we introduce the notion of searchable encryption with key-update (SEKU) that provides users with the option of non-interactive key updates. We further define the notion of post-compromise secure with respect to leakage functions to study whether DSSE schemes can still provide data security after the client's secret key is compromised. We demonstrate that post-compromise security is achievable with a proposed protocol called ``Bamboo". Interestingly, the leakage functions of Bamboo satisfy the requirements for both forward and backward security. We conduct a performance evaluation of Bamboo using a real-world dataset and compare its runtime efficiency with the existing forward-and-backward secure DSSE schemes. The result shows that Bamboo provides strong security with better or comparable performance.

Title: MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration

• Authors: Zhichao Wei, Qingkun Su, Long Qin, Weizhi Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15059
• Pdf URL: https://arxiv.org/pdf/2403.15059
• Copy Paste: [[2403.15059]] MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration(https://arxiv.org/abs/2403.15059)
• Keywords: diffusion
• Abstract: Recent advances in tuning-free personalized image generation based on diffusion models are impressive. However, to improve subject fidelity, existing methods either retrain the diffusion model or infuse it with dense visual embeddings, both of which suffer from poor generalization and efficiency. Also, these methods falter in multi-subject image generation due to the unconstrained cross-attention mechanism. In this paper, we propose MM-Diff, a unified and tuning-free image personalization framework capable of generating high-fidelity images of both single and multiple subjects in seconds. Specifically, to simultaneously enhance text consistency and subject fidelity, MM-Diff employs a vision encoder to transform the input image into CLS and patch embeddings. CLS embeddings are used on the one hand to augment the text embeddings, and on the other hand together with patch embeddings to derive a small number of detail-rich subject embeddings, both of which are efficiently integrated into the diffusion model through the well-designed multimodal cross-attention mechanism. Additionally, MM-Diff introduces cross-attention map constraints during the training phase, ensuring flexible multi-subject image sampling during inference without any predefined inputs (e.g., layout). Extensive experiments demonstrate the superior performance of MM-Diff over other leading methods.

Title: Towards a Comprehensive, Efficient and Promptable Anatomic Structure Segmentation Model using 3D Whole-body CT Scans

• Authors: Heng Guo, Jianfeng Zhang, Jiaxing Huang, Tony C. W. Mok, Dazhou Guo, Ke Yan, Le Lu, Dakai Jin, Minfeng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15063
• Pdf URL: https://arxiv.org/pdf/2403.15063
• Copy Paste: [[2403.15063]] Towards a Comprehensive, Efficient and Promptable Anatomic Structure Segmentation Model using 3D Whole-body CT Scans(https://arxiv.org/abs/2403.15063)
• Keywords: segmentation
• Abstract: Segment anything model (SAM) demonstrates strong generalization ability on natural image segmentation. However, its direct adaption in medical image segmentation tasks shows significant performance drops with inferior accuracy and unstable results. It may also requires an excessive number of prompt points to obtain a reasonable accuracy. For segmenting 3D radiological CT or MRI scans, a 2D SAM model has to separately handle hundreds of 2D slices. Although quite a few studies explore adapting SAM into medical image volumes, the efficiency of 2D adaption methods is unsatisfactory and 3D adaptation methods only capable of segmenting specific organs/tumors. In this work, we propose a comprehensive and scalable 3D SAM model for whole-body CT segmentation, named CT-SAM3D. Instead of adapting SAM, we propose a 3D promptable segmentation model using a (nearly) fully labeled CT dataset. To train CT-SAM3D effectively, ensuring the model's accurate responses to higher-dimensional spatial prompts is crucial, and 3D patch-wise training is required due to GPU memory constraints. For this purpose, we propose two key technical developments: 1) a progressively and spatially aligned prompt encoding method to effectively encode click prompts in local 3D space; and 2) a cross-patch prompt learning scheme to capture more 3D spatial context, which is beneficial for reducing the editing workloads when interactively prompting on large organs. CT-SAM3D is trained and validated using a curated dataset of 1204 CT scans containing 107 whole-body anatomies, reporting significantly better quantitative performance against all previous SAM-derived models by a large margin with much fewer click prompts. Our model can handle segmenting unseen organ as well. Code, data, and our 3D interactive segmentation tool with quasi-real-time responses will be made publicly available.

Title: Recent Trends in 3D Reconstruction of General Non-Rigid Scenes

• Authors: Raza Yunus, Jan Eric Lenssen, Michael Niemeyer, Yiyi Liao, Christian Rupprecht, Christian Theobalt, Gerard Pons-Moll, Jia-Bin Huang, Vladislav Golyanik, Eddy Ilg
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.15064
• Pdf URL: https://arxiv.org/pdf/2403.15064
• Copy Paste: [[2403.15064]] Recent Trends in 3D Reconstruction of General Non-Rigid Scenes(https://arxiv.org/abs/2403.15064)
• Keywords: generative
• Abstract: Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision. It enables the synthesizing of photorealistic novel views, useful for the movie industry and AR/VR applications. It also facilitates the content creation necessary in computer games and AR/VR by avoiding laborious manual design processes. Further, such models are fundamental for intelligent computing systems that need to interpret real-world scenes and actions to act and interact safely with the human world. Notably, the world surrounding us is dynamic, and reconstructing models of dynamic, non-rigidly moving scenes is a severely underconstrained and challenging problem. This state-of-the-art report (STAR) offers the reader a comprehensive summary of state-of-the-art techniques with monocular and multi-view inputs such as data from RGB and RGB-D sensors, among others, conveying an understanding of different approaches, their potential applications, and promising further research directions. The report covers 3D reconstruction of general non-rigid scenes and further addresses the techniques for scene decomposition, editing and controlling, and generalizable and generative modeling. More specifically, we first review the common and fundamental concepts necessary to understand and navigate the field and then discuss the state-of-the-art techniques by reviewing recent approaches that use traditional and machine-learning-based neural representations, including a discussion on the newly enabled applications. The STAR is concluded with a discussion of the remaining limitations and open challenges.

Title: Real-time Threat Detection Strategies for Resource-constrained Devices

• Authors: Mounia Hamidouche, Biniam Fisseha Demissie, Bilel Cherif
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15078
• Pdf URL: https://arxiv.org/pdf/2403.15078
• Copy Paste: [[2403.15078]] Real-time Threat Detection Strategies for Resource-constrained Devices(https://arxiv.org/abs/2403.15078)
• Keywords: security, attack
• Abstract: As more devices connect to the internet, it becomes crucial to address their limitations and basic security needs. While much research focuses on utilizing ML and DL to tackle security challenges, there is often a tendency to overlook the practicality and feasibility of implementing these methods in real-time settings. This oversight stems from the constrained processing power and memory of certain devices (IoT devices), as well as concerns about the generalizability of these approaches. Focusing on the detection of DNS-tunneling attacks in a router as a case study, we present an end-to-end process designed to effectively address these challenges. The process spans from developing a lightweight DNS-tunneling detection model to integrating it into a resource-constrained device for real-time detection. Through our experiments, we demonstrate that utilizing stateless features for training the ML model, along with features chosen to be independent of the network configuration, leads to highly accurate results. The deployment of this carefully crafted model, optimized for embedded devices across diverse environments, resulted in high DNS-tunneling attack detection with minimal latency. With this work, we aim to encourage solutions that strike a balance between theoretical advancements and the practical applicability of ML approaches in the ever-evolving landscape of device security.

Title: Evaluating the Influence of Multi-Factor Authentication and Recovery Settings on the Security and Accessibility of User Accounts

• Authors: Andre Büttner, Nils Gruschka
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15080
• Pdf URL: https://arxiv.org/pdf/2403.15080
• Copy Paste: [[2403.15080]] Evaluating the Influence of Multi-Factor Authentication and Recovery Settings on the Security and Accessibility of User Accounts(https://arxiv.org/abs/2403.15080)
• Keywords: security
• Abstract: Nowadays, most online services offer different authentication methods that users can set up for multi-factor authentication but also as a recovery method. This configuration must be done thoroughly to prevent an adversary's access while ensuring the legitimate user does not lose access to their account. This is particularly important for fundamental everyday services, where either failure would have severe consequences. Nevertheless, little research has been done on the authentication of actual users regarding security and the risk of being locked out of their accounts. To foster research in this direction, this paper presents a study on the account settings of Google and Apple users. Considering the multi-factor authentication configuration and recovery options, we analyzed the account security and lock-out risks. Our results provide insights into the usage of multi-factor authentication in practice, show significant security differences between Google and Apple accounts, and reveal that many users would miss access to their accounts when losing a single authentication device.

Title: Cell Variational Information Bottleneck Network

• Authors: Zhonghua Zhai, Chen Ju, Jinsong Lan, Shuai Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15082
• Pdf URL: https://arxiv.org/pdf/2403.15082
• Copy Paste: [[2403.15082]] Cell Variational Information Bottleneck Network(https://arxiv.org/abs/2403.15082)
• Keywords: robust
• Abstract: In this work, we propose Cell Variational Information Bottleneck Network (cellVIB), a convolutional neural network using information bottleneck mechanism, which can be combined with the latest feedforward network architecture in an end-to-end training method. Our Cell Variational Information Bottleneck Network is constructed by stacking VIB cells, which generate feature maps with uncertainty. As layers going deeper, the regularization effect will gradually increase, instead of directly adding excessive regular constraints to the output layer of the model as in Deep VIB. Under each VIB cell, the feedforward process learns an independent mean term and an standard deviation term, and predicts the Gaussian distribution based on them. The feedback process is based on reparameterization trick for effective training. This work performs an extensive analysis on MNIST dataset to verify the effectiveness of each VIB cells, and provides an insightful analysis on how the VIB cells affect mutual information. Experiments conducted on CIFAR-10 also prove that our cellVIB is robust against noisy labels during training and against corrupted images during testing. Then, we validate our method on PACS dataset, whose results show that the VIB cells can significantly improve the generalization performance of the basic model. Finally, in a more complex representation learning task, face recognition, our network structure has also achieved very competitive results.

Title: SIMAP: A simplicial-map layer for neural networks

• Authors: Rocio Gonzalez-Diaz, Miguel A. Gutiérrez-Naranjo, Eduardo Paluzo-Hidalgo
• Subjects: cs.LG, math.AT
• Abstract URL: https://arxiv.org/abs/2403.15083
• Pdf URL: https://arxiv.org/pdf/2403.15083
• Copy Paste: [[2403.15083]] SIMAP: A simplicial-map layer for neural networks(https://arxiv.org/abs/2403.15083)
• Keywords: interpretability
• Abstract: In this paper, we present SIMAP, a novel layer integrated into deep learning models, aimed at enhancing the interpretability of the output. The SIMAP layer is an enhanced version of Simplicial-Map Neural Networks (SMNNs), an explainable neural network based on support sets and simplicial maps (functions used in topology to transform shapes while preserving their structural connectivity). The novelty of the methodology proposed in this paper is two-fold: Firstly, SIMAP layers work in combination with other deep learning architectures as an interpretable layer substituting classic dense final layers. Secondly, unlike SMNNs, the support set is based on a fixed maximal simplex, the barycentric subdivision being efficiently computed with a matrix-based multiplication algorithm.

Title: CHisIEC: An Information Extraction Corpus for Ancient Chinese History

• Authors: Xuemei Tang, Zekun Deng, Qi Su, Hao Yang, Jun Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15088
• Pdf URL: https://arxiv.org/pdf/2403.15088
• Copy Paste: [[2403.15088]] CHisIEC: An Information Extraction Corpus for Ancient Chinese History(https://arxiv.org/abs/2403.15088)
• Keywords: robust, extraction, large language model
• Abstract: Natural Language Processing (NLP) plays a pivotal role in the realm of Digital Humanities (DH) and serves as the cornerstone for advancing the structural analysis of historical and cultural heritage texts. This is particularly true for the domains of named entity recognition (NER) and relation extraction (RE). In our commitment to expediting ancient history and culture, we present the ``Chinese Historical Information Extraction Corpus''(CHisIEC). CHisIEC is a meticulously curated dataset designed to develop and evaluate NER and RE tasks, offering a resource to facilitate research in the field. Spanning a remarkable historical timeline encompassing data from 13 dynasties spanning over 1830 years, CHisIEC epitomizes the extensive temporal range and text heterogeneity inherent in Chinese historical documents. The dataset encompasses four distinct entity types and twelve relation types, resulting in a meticulously labeled dataset comprising 14,194 entities and 8,609 relations. To establish the robustness and versatility of our dataset, we have undertaken comprehensive experimentation involving models of various sizes and paradigms. Additionally, we have evaluated the capabilities of Large Language Models (LLMs) in the context of tasks related to ancient Chinese history. The dataset and code are available at \url{https://github.com/tangxuemei1995/CHisIEC}.

Title: IFSENet : Harnessing Sparse Iterations for Interactive Few-shot Segmentation Excellence

• Authors: Shreyas Chandgothia, Ardhendu Sekhar, Amit Sethi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15089
• Pdf URL: https://arxiv.org/pdf/2403.15089
• Copy Paste: [[2403.15089]] IFSENet : Harnessing Sparse Iterations for Interactive Few-shot Segmentation Excellence(https://arxiv.org/abs/2403.15089)
• Keywords: segmentation
• Abstract: Training a computer vision system to segment a novel class typically requires collecting and painstakingly annotating lots of images with objects from that class. Few-shot segmentation techniques reduce the required number of images to learn to segment a new class, but careful annotations of object boundaries are still required. On the other hand, interactive segmentation techniques only focus on incrementally improving the segmentation of one object at a time (typically, using clicks given by an expert) in a class-agnostic manner. We combine the two concepts to drastically reduce the effort required to train segmentation models for novel classes. Instead of trivially feeding interactive segmentation masks as ground truth to a few-shot segmentation model, we propose IFSENet, which can accept sparse supervision on a single or few support images in the form of clicks to generate masks on support (training, at least clicked upon once) as well as query (test, never clicked upon) images. To trade-off effort for accuracy flexibly, the number of images and clicks can be incrementally added to the support set to further improve the segmentation of support as well as query images. The proposed model approaches the accuracy of previous state-of-the-art few-shot segmentation models with considerably lower annotation effort (clicks instead of maps), when tested on Pascal and SBD datasets on query images. It also works well as an interactive segmentation method on support images.

Title: Text clustering with LLM embeddings

• Authors: Alina Petukhova, Joao P. Matos-Carvalho, Nuno Fachada
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15112
• Pdf URL: https://arxiv.org/pdf/2403.15112
• Copy Paste: [[2403.15112]] Text clustering with LLM embeddings(https://arxiv.org/abs/2403.15112)
• Keywords: large language model
• Abstract: Text clustering is an important approach for organising the growing amount of digital content, helping to structure and find hidden patterns in uncategorised data. In this research, we investigated how different textual embeddings - particularly those used in large language models (LLMs) - and clustering algorithms affect how text datasets are clustered. A series of experiments were conducted to assess how embeddings influence clustering results, the role played by dimensionality reduction through summarisation, and embedding size adjustment. Results reveal that LLM embeddings excel at capturing the nuances of structured language, while BERT leads the lightweight options in performance. In addition, we find that increasing embedding dimensionality and summarisation techniques do not uniformly improve clustering efficiency, suggesting that these strategies require careful analysis to use in real-life models. These results highlight a complex balance between the need for nuanced text representation and computational feasibility in text clustering applications. This study extends traditional text clustering frameworks by incorporating embeddings from LLMs, thereby paving the way for improved methodologies and opening new avenues for future research in various types of textual analysis.

Title: An Open-World, Diverse, Cross-Spatial-Temporal Benchmark for Dynamic Wild Person Re-Identification

• Authors: Lei Zhang, Xiaowei Fu, Fuxiang Huang, Yi Yang, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15119
• Pdf URL: https://arxiv.org/pdf/2403.15119
• Copy Paste: [[2403.15119]] An Open-World, Diverse, Cross-Spatial-Temporal Benchmark for Dynamic Wild Person Re-Identification(https://arxiv.org/abs/2403.15119)
• Keywords: privacy, protect
• Abstract: Person re-identification (ReID) has made great strides thanks to the data-driven deep learning techniques. However, the existing benchmark datasets lack diversity, and models trained on these data cannot generalize well to dynamic wild scenarios. To meet the goal of improving the explicit generalization of ReID models, we develop a new Open-World, Diverse, Cross-Spatial-Temporal dataset named OWD with several distinct features. 1) Diverse collection scenes: multiple independent open-world and highly dynamic collecting scenes, including streets, intersections, shopping malls, etc. 2) Diverse lighting variations: long time spans from daytime to nighttime with abundant illumination changes. 3) Diverse person status: multiple camera networks in all seasons with normal/adverse weather conditions and diverse pedestrian appearances (e.g., clothes, personal belongings, poses, etc.). 4) Protected privacy: invisible faces for privacy critical applications. To improve the implicit generalization of ReID, we further propose a Latent Domain Expansion (LDE) method to develop the potential of source data, which decouples discriminative identity-relevant and trustworthy domain-relevant features and implicitly enforces domain-randomized identity feature space expansion with richer domain diversity to facilitate domain invariant representations. Our comprehensive evaluations with most benchmark datasets in the community are crucial for progress, although this work is far from the grand goal toward open-world and dynamic wild applications.

Title: SYNCS: Synthetic Data and Contrastive Self-Supervised Training for Central Sulcus Segmentation

• Authors: Vladyslav Zalevskyi, Kristoffer Hougaard Madsen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15121
• Pdf URL: https://arxiv.org/pdf/2403.15121
• Copy Paste: [[2403.15121]] SYNCS: Synthetic Data and Contrastive Self-Supervised Training for Central Sulcus Segmentation(https://arxiv.org/abs/2403.15121)
• Keywords: segmentation
• Abstract: Bipolar disorder (BD) and schizophrenia (SZ) are severe mental disorders with profound societal impact. Identifying risk markers early is crucial for understanding disease progression and enabling preventive measures. The Danish High Risk and Resilience Study (VIA) focuses on understanding early disease processes, particularly in children with familial high risk (FHR). Understanding structural brain changes associated with these diseases during early stages is essential for effective interventions. The central sulcus (CS) is a prominent brain landmark related to brain regions involved in motor and sensory processing. Analyzing CS morphology can provide valuable insights into neurodevelopmental abnormalities in the FHR group. However, segmenting the central sulcus (CS) presents challenges due to its variability, especially in adolescents. This study introduces two novel approaches to improve CS segmentation: synthetic data generation to model CS variability and self-supervised pre-training with multi-task learning to adapt models to new cohorts. These methods aim to enhance segmentation performance across diverse populations, eliminating the need for extensive preprocessing.

Title: Transfer CLIP for Generalizable Image Denoising

• Authors: Jun Cheng, Dong Liang, Shan Tan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.15132
• Pdf URL: https://arxiv.org/pdf/2403.15132
• Copy Paste: [[2403.15132]] Transfer CLIP for Generalizable Image Denoising(https://arxiv.org/abs/2403.15132)
• Keywords: robust, segmentation
• Abstract: Image denoising is a fundamental task in computer vision. While prevailing deep learning-based supervised and self-supervised methods have excelled in eliminating in-distribution noise, their susceptibility to out-of-distribution (OOD) noise remains a significant challenge. The recent emergence of contrastive language-image pre-training (CLIP) model has showcased exceptional capabilities in open-world image recognition and segmentation. Yet, the potential for leveraging CLIP to enhance the robustness of low-level tasks remains largely unexplored. This paper uncovers that certain dense features extracted from the frozen ResNet image encoder of CLIP exhibit distortion-invariant and content-related properties, which are highly desirable for generalizable denoising. Leveraging these properties, we devise an asymmetrical encoder-decoder denoising network, which incorporates dense features including the noisy image and its multi-scale features from the frozen ResNet encoder of CLIP into a learnable image decoder to achieve generalizable denoising. The progressive feature augmentation strategy is further proposed to mitigate feature overfitting and improve the robustness of the learnable decoder. Extensive experiments and comparisons conducted across diverse OOD noises, including synthetic noise, real-world sRGB noise, and low-dose CT image noise, demonstrate the superior generalization ability of our method.

Title: Deep Generative Model based Rate-Distortion for Image Downscaling Assessment

• Authors: Yuanbang Liang, Bhavesh Garg, Paul L Rosin, Yipeng Qin
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.15139
• Pdf URL: https://arxiv.org/pdf/2403.15139
• Copy Paste: [[2403.15139]] Deep Generative Model based Rate-Distortion for Image Downscaling Assessment(https://arxiv.org/abs/2403.15139)
• Keywords: generative
• Abstract: In this paper, we propose Image Downscaling Assessment by Rate-Distortion (IDA-RD), a novel measure to quantitatively evaluate image downscaling algorithms. In contrast to image-based methods that measure the quality of downscaled images, ours is process-based that draws ideas from rate-distortion theory to measure the distortion incurred during downscaling. Our main idea is that downscaling and super-resolution (SR) can be viewed as the encoding and decoding processes in the rate-distortion model, respectively, and that a downscaling algorithm that preserves more details in the resulting low-resolution (LR) images should lead to less distorted high-resolution (HR) images in SR. In other words, the distortion should increase as the downscaling algorithm deteriorates. However, it is non-trivial to measure this distortion as it requires the SR algorithm to be blind and stochastic. Our key insight is that such requirements can be met by recent SR algorithms based on deep generative models that can find all matching HR images for a given LR image on their learned image manifolds. Extensive experimental results show the effectiveness of our IDA-RD measure.

Title: Modular Deep Active Learning Framework for Image Annotation: A Technical Report for the Ophthalmo-AI Project

• Authors: Md Abdul Kadir, Hasan Md Tusfiqur Alam, Pascale Maul, Hans-Jürgen Profitlich, Moritz Wolf, Daniel Sonntag
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15143
• Pdf URL: https://arxiv.org/pdf/2403.15143
• Copy Paste: [[2403.15143]] Modular Deep Active Learning Framework for Image Annotation: A Technical Report for the Ophthalmo-AI Project(https://arxiv.org/abs/2403.15143)
• Keywords: segmentation
• Abstract: Image annotation is one of the most essential tasks for guaranteeing proper treatment for patients and tracking progress over the course of therapy in the field of medical imaging and disease diagnosis. However, manually annotating a lot of 2D and 3D imaging data can be extremely tedious. Deep Learning (DL) based segmentation algorithms have completely transformed this process and made it possible to automate image segmentation. By accurately segmenting medical images, these algorithms can greatly minimize the time and effort necessary for manual annotation. Additionally, by incorporating Active Learning (AL) methods, these segmentation algorithms can perform far more effectively with a smaller amount of ground truth data. We introduce MedDeepCyleAL, an end-to-end framework implementing the complete AL cycle. It provides researchers with the flexibility to choose the type of deep learning model they wish to employ and includes an annotation tool that supports the classification and segmentation of medical images. The user-friendly interface allows for easy alteration of the AL and DL model settings through a configuration file, requiring no prior programming experience. While MedDeepCyleAL can be applied to any kind of image data, we have specifically applied it to ophthalmology data in this project.

Title: A Multimodal Approach for Cross-Domain Image Retrieval

• Authors: Lucas Iijima, Tania Stathaki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15152
• Pdf URL: https://arxiv.org/pdf/2403.15152
• Copy Paste: [[2403.15152]] A Multimodal Approach for Cross-Domain Image Retrieval(https://arxiv.org/abs/2403.15152)
• Keywords: generative
• Abstract: Image generators are gaining vast amount of popularity and have rapidly changed how digital content is created. With the latest AI technology, millions of high quality images are being generated by the public, which are constantly motivating the research community to push the limits of generative models to create more complex and realistic images. This paper focuses on Cross-Domain Image Retrieval (CDIR) which can be used as an additional tool to inspect collections of generated images by determining the level of similarity between images in a dataset. An ideal retrieval system would be able to generalize to unseen complex images from multiple domains (e.g., photos, drawings and paintings). To address this goal, we propose a novel caption-matching approach that leverages multimodal language-vision architectures pre-trained on large datasets. The method is tested on DomainNet and Office-Home datasets and consistently achieves state-of-the-art performance over the latest approaches in the literature for cross-domain image retrieval. In order to verify the effectiveness with AI-generated images, the method was also put to test with a database composed by samples collected from Midjourney, which is a widely used generative platform for content creation.

Title: Self-Improvement for Neural Combinatorial Optimization: Sample without Replacement, but Improvement

• Authors: Jonathan Pirnay, Dominik G. Grimm
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15180
• Pdf URL: https://arxiv.org/pdf/2403.15180
• Copy Paste: [[2403.15180]] Self-Improvement for Neural Combinatorial Optimization: Sample without Replacement, but Improvement(https://arxiv.org/abs/2403.15180)
• Keywords: transformer
• Abstract: Current methods for end-to-end constructive neural combinatorial optimization usually train a policy using behavior cloning from expert solutions or policy gradient methods from reinforcement learning. While behavior cloning is straightforward, it requires expensive expert solutions, and policy gradient methods are often computationally demanding and complex to fine-tune. In this work, we bridge the two and simplify the training process by sampling multiple solutions for random instances using the current model in each epoch and then selecting the best solution as an expert trajectory for supervised imitation learning. To achieve progressively improving solutions with minimal sampling, we introduce a method that combines round-wise Stochastic Beam Search with an update strategy derived from a provable policy improvement. This strategy refines the policy between rounds by utilizing the advantage of the sampled sequences with almost no computational overhead. We evaluate our approach on the Traveling Salesman Problem and the Capacitated Vehicle Routing Problem. The models trained with our method achieve comparable performance and generalization to those trained with expert data. Additionally, we apply our method to the Job Shop Scheduling Problem using a transformer-based architecture and outperform existing state-of-the-art methods by a wide margin.

Title: PDE-CNNs: Axiomatic Derivations and Applications

• Authors: Gijs Bellaard, Sei Sakata, Bart M. N. Smets, Remco Duits
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15182
• Pdf URL: https://arxiv.org/pdf/2403.15182
• Copy Paste: [[2403.15182]] PDE-CNNs: Axiomatic Derivations and Applications(https://arxiv.org/abs/2403.15182)
• Keywords: interpretability
• Abstract: PDE-based Group Convolutional Neural Networks (PDE-G-CNNs) utilize solvers of geometrically meaningful evolution PDEs as substitutes for the conventional components in G-CNNs. PDE-G-CNNs offer several key benefits all at once: fewer parameters, inherent equivariance, better performance, data efficiency, and geometric interpretability. In this article we focus on Euclidean equivariant PDE-G-CNNs where the feature maps are two dimensional throughout. We call this variant of the framework a PDE-CNN. We list several practically desirable axioms and derive from these which PDEs should be used in a PDE-CNN. Here our approach to geometric learning via PDEs is inspired by the axioms of classical linear and morphological scale-space theory, which we generalize by introducing semifield-valued signals. Furthermore, we experimentally confirm for small networks that PDE-CNNs offer fewer parameters, better performance, and data efficiency in comparison to CNNs. We also investigate what effect the use of different semifields has on the performance of the models.

Title: ECHO: Efficient Off-Chain Payments and Cross-Chain Swaps for Cryptocurrencies

• Authors: Di Wu, Jian Liu, Zhengwei Hou, Wu Wen, Kui Ren
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.15191
• Pdf URL: https://arxiv.org/pdf/2403.15191
• Copy Paste: [[2403.15191]] ECHO: Efficient Off-Chain Payments and Cross-Chain Swaps for Cryptocurrencies(https://arxiv.org/abs/2403.15191)
• Keywords: fair
• Abstract: In this paper, we present ECHO, a TEE-based layer-2 solution that tackles two crucial challenges in the realm of cryptocurrencies: off-chain payments and cross-chain swaps. It offers three notable features: - Channel-free off-chain payments: it allows a payer to make direct payments to anyone without requiring any on-chain relationship or intermediary channels. - Real-time yet decentralized cross-chain swaps: it is the first known solution that enables real-time cross-chain swaps without relying on a central server. This novel feature is made possible through a ground-breaking fair exchange protocol. - TEE crash-tolerance: it offers two solutions to handle TEE crashes, one of which involves an innovative application of time-lock puzzles in this context. We evaluate ECHO on a network consists of 1000 nodes and the evaluation results show that ECHO can achieve 7000 TPS

Title: Your Image is My Video: Reshaping the Receptive Field via Image-To-Video Differentiable AutoAugmentation and Fusion

• Authors: Sofia Casarin, Cynthia I. Ugwu, Sergio Escalera, Oswald Lanz
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15194
• Pdf URL: https://arxiv.org/pdf/2403.15194
• Copy Paste: [[2403.15194]] Your Image is My Video: Reshaping the Receptive Field via Image-To-Video Differentiable AutoAugmentation and Fusion(https://arxiv.org/abs/2403.15194)
• Keywords: segmentation
• Abstract: The landscape of deep learning research is moving towards innovative strategies to harness the true potential of data. Traditionally, emphasis has been on scaling model architectures, resulting in large and complex neural networks, which can be difficult to train with limited computational resources. However, independently of the model size, data quality (i.e. amount and variability) is still a major factor that affects model generalization. In this work, we propose a novel technique to exploit available data through the use of automatic data augmentation for the tasks of image classification and semantic segmentation. We introduce the first Differentiable Augmentation Search method (DAS) to generate variations of images that can be processed as videos. Compared to previous approaches, DAS is extremely fast and flexible, allowing the search on very large search spaces in less than a GPU day. Our intuition is that the increased receptive field in the temporal dimension provided by DAS could lead to benefits also to the spatial receptive field. More specifically, we leverage DAS to guide the reshaping of the spatial receptive field by selecting task-dependant transformations. As a result, compared to standard augmentation alternatives, we improve in terms of accuracy on ImageNet, Cifar10, Cifar100, Tiny-ImageNet, Pascal-VOC-2012 and CityScapes datasets when plugging-in our DAS over different light-weight video backbones.

Title: Cryptic Bytes: WebAssembly Obfuscation for Evading Cryptojacking Detection

• Authors: Håkon Harnes, Donn Morrison
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15197
• Pdf URL: https://arxiv.org/pdf/2403.15197
• Copy Paste: [[2403.15197]] Cryptic Bytes: WebAssembly Obfuscation for Evading Cryptojacking Detection(https://arxiv.org/abs/2403.15197)
• Keywords: secure, security, robust
• Abstract: WebAssembly has gained significant traction as a high-performance, secure, and portable compilation target for the Web and beyond. However, its growing adoption has also introduced new security challenges. One such threat is cryptojacking, where websites mine cryptocurrencies on visitors' devices without their knowledge or consent, often through the use of WebAssembly. While detection methods have been proposed, research on circumventing them remains limited. In this paper, we present the most comprehensive evaluation of code obfuscation techniques for WebAssembly to date, assessing their effectiveness, detectability, and overhead across multiple abstraction levels. We obfuscate a diverse set of applications, including utilities, games, and crypto miners, using state-of-the-art obfuscation tools like Tigress and wasm-mutate, as well as our novel tool, emcc-obf. Our findings suggest that obfuscation can effectively produce dissimilar WebAssembly binaries, with Tigress proving most effective, followed by emcc-obf and wasm-mutate. The impact on the resulting native code is also significant, although the V8 engine's TurboFan optimizer can reduce native code size by 30\% on average. Notably, we find that obfuscation can successfully evade state-of-the-art cryptojacking detectors. Although obfuscation can introduce substantial performance overheads, we demonstrate how obfuscation can be used for evading detection with minimal overhead in real-world scenarios by strategically applying transformations. These insights are valuable for researchers, providing a foundation for developing more robust detection methods. Additionally, we make our dataset of over 20,000 obfuscated WebAssembly binaries and the emcc-obf tool publicly available to stimulate further research.

Title: Robust optimization for adversarial learning with finite sample complexity guarantees

• Authors: André Bertolace, Konstatinos Gatsis, Kostas Margellos
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.15207
• Pdf URL: https://arxiv.org/pdf/2403.15207
• Copy Paste: [[2403.15207]] Robust optimization for adversarial learning with finite sample complexity guarantees(https://arxiv.org/abs/2403.15207)
• Keywords: attack, robust
• Abstract: Decision making and learning in the presence of uncertainty has attracted significant attention in view of the increasing need to achieve robust and reliable operations. In the case where uncertainty stems from the presence of adversarial attacks this need is becoming more prominent. In this paper we focus on linear and nonlinear classification problems and propose a novel adversarial training method for robust classifiers, inspired by Support Vector Machine (SVM) margins. We view robustness under a data driven lens, and derive finite sample complexity bounds for both linear and non-linear classifiers in binary and multi-class scenarios. Notably, our bounds match natural classifiers' complexity. Our algorithm minimizes a worst-case surrogate loss using Linear Programming (LP) and Second Order Cone Programming (SOCP) for linear and non-linear models. Numerical experiments on the benchmark MNIST and CIFAR10 datasets show our approach's comparable performance to state-of-the-art methods, without needing adversarial examples during training. Our work offers a comprehensive framework for enhancing binary linear and non-linear classifier robustness, embedding robustness in learning under the presence of adversaries.

Title: VPAS: Publicly Verifiable and Privacy-Preserving Aggregate Statistics on Distributed Datasets

• Authors: Mohammed Alghazwi, Dewi Davies-Batista, Dimka Karastoyanova, Fatih Turkmen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15208
• Pdf URL: https://arxiv.org/pdf/2403.15208
• Copy Paste: [[2403.15208]] VPAS: Publicly Verifiable and Privacy-Preserving Aggregate Statistics on Distributed Datasets(https://arxiv.org/abs/2403.15208)
• Keywords: secure, privacy
• Abstract: Aggregate statistics play an important role in extracting meaningful insights from distributed data while preserving privacy. A growing number of application domains, such as healthcare, utilize these statistics in advancing research and improving patient care. In this work, we explore the challenge of input validation and public verifiability within privacy-preserving aggregation protocols. We address the scenario in which a party receives data from multiple sources and must verify the validity of the input and correctness of the computations over this data to third parties, such as auditors, while ensuring input data privacy. To achieve this, we propose the "VPAS" protocol, which satisfies these requirements. Our protocol utilizes homomorphic encryption for data privacy, and employs Zero-Knowledge Proofs (ZKP) and a blockchain system for input validation and public verifiability. We constructed VPAS by extending existing verifiable encryption schemes into secure protocols that enable N clients to encrypt, aggregate, and subsequently release the final result to a collector in a verifiable manner. We implemented and experimentally evaluated VPAS with regard to encryption costs, proof generation, and verification. The findings indicate that the overhead associated with verifiability in our protocol is 10x lower than that incurred by simply using conventional zkSNARKs. This enhanced efficiency makes it feasible to apply input validation with public verifiability across a wider range of applications or use cases that can tolerate moderate computational overhead associated with proof generation.

Title: MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral Pedestrian Detection

• Authors: Taeheon Kim, Sangyun Chung, Damin Yeom, Youngjoon Yu, Hak Gu Kim, Yong Man Ro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15209
• Pdf URL: https://arxiv.org/pdf/2403.15209
• Copy Paste: [[2403.15209]] MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral Pedestrian Detection(https://arxiv.org/abs/2403.15209)
• Keywords: large language model
• Abstract: Multispectral pedestrian detection is attractive for around-the-clock applications due to the complementary information between RGB and thermal modalities. However, current models often fail to detect pedestrians in obvious cases, especially due to the modality bias learned from statistically biased datasets. From these problems, we anticipate that maybe understanding the complementary information itself is difficult to achieve from vision-only models. Accordingly, we propose a novel Multispectral Chain-of-Thought Detection (MSCoTDet) framework, which incorporates Large Language Models (LLMs) to understand the complementary information at the semantic level and further enhance the fusion process. Specifically, we generate text descriptions of the pedestrian in each RGB and thermal modality and design a Multispectral Chain-of-Thought (MSCoT) prompting, which models a step-by-step process to facilitate cross-modal reasoning at the semantic level and perform accurate detection. Moreover, we design a Language-driven Multi-modal Fusion (LMF) strategy that enables fusing vision-driven and language-driven detections. Extensive experiments validate that MSCoTDet improves multispectral pedestrian detection.

Title: Anytime, Anywhere, Anyone: Investigating the Feasibility of Segment Anything Model for Crowd-Sourcing Medical Image Annotations

• Authors: Pranav Kulkarni, Adway Kanhere, Dharmam Savani, Andrew Chan, Devina Chatterjee, Paul H. Yi, Vishwa S. Parekh
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15218
• Pdf URL: https://arxiv.org/pdf/2403.15218
• Copy Paste: [[2403.15218]] Anytime, Anywhere, Anyone: Investigating the Feasibility of Segment Anything Model for Crowd-Sourcing Medical Image Annotations(https://arxiv.org/abs/2403.15218)
• Keywords: segmentation
• Abstract: Curating annotations for medical image segmentation is a labor-intensive and time-consuming task that requires domain expertise, resulting in "narrowly" focused deep learning (DL) models with limited translational utility. Recently, foundation models like the Segment Anything Model (SAM) have revolutionized semantic segmentation with exceptional zero-shot generalizability across various domains, including medical imaging, and hold a lot of promise for streamlining the annotation process. However, SAM has yet to be evaluated in a crowd-sourced setting to curate annotations for training 3D DL segmentation models. In this work, we explore the potential of SAM for crowd-sourcing "sparse" annotations from non-experts to generate "dense" segmentation masks for training 3D nnU-Net models, a state-of-the-art DL segmentation model. Our results indicate that while SAM-generated annotations exhibit high mean Dice scores compared to ground-truth annotations, nnU-Net models trained on SAM-generated annotations perform significantly worse than nnU-Net models trained on ground-truth annotations ($p<0.001$, all).

Title: Differentially Private Ad Conversion Measurement

• Authors: John Delaney, Badih Ghazi, Charlie Harrison, Christina Ilvento, Ravi Kumar, Pasin Manurangsi, Martin Pal, Karthik Prabhakar, Mariana Raykova
• Subjects: cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2403.15224
• Pdf URL: https://arxiv.org/pdf/2403.15224
• Copy Paste: [[2403.15224]] Differentially Private Ad Conversion Measurement(https://arxiv.org/abs/2403.15224)
• Keywords: privacy
• Abstract: In this work, we study ad conversion measurement, a central functionality in digital advertising, where an advertiser seeks to estimate advertiser website (or mobile app) conversions attributed to ad impressions that users have interacted with on various publisher websites (or mobile apps). Using differential privacy (DP), a notion that has gained in popularity due to its strong mathematical guarantees, we develop a formal framework for private ad conversion measurement. In particular, we define the notion of an operationally valid configuration of the attribution rule, DP adjacency relation, contribution bounding scope and enforcement point. We then provide, for the set of configurations that most commonly arises in practice, a complete characterization, which uncovers a delicate interplay between attribution and privacy.

Title: Attacking with Something That Does Not Exist: Low-Rate Flood with 'Proof of Non-Existence' Can Exhaust DNS Resolver CPU

• Authors: Olivia Gruza (1), Elias Heftrig (1), Oliver Jacobsen (1), Haya Schulmann (1), Niklas Vogel (1), Michael Waidner (2, 3, 4) ((1) Goethe-Universität Frankfurt, (2) National Research Center for Applied Cybersecurity ATHENE, (3) Fraunhofer Institute for Secure Information Technology SIT, (4) Technische Universität Darmstadt)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15233
• Pdf URL: https://arxiv.org/pdf/2403.15233
• Copy Paste: [[2403.15233]] Attacking with Something That Does Not Exist: Low-Rate Flood with 'Proof of Non-Existence' Can Exhaust DNS Resolver CPU(https://arxiv.org/abs/2403.15233)
• Keywords: attack
• Abstract: NSEC3 is a proof of non-existence in DNSSEC, which provides an authenticated assertion that a queried resource does not exist in the target domain. NSEC3 consists of alphabetically sorted hashed names before and after the queried hostname. To make dictionary attacks harder, the hash function can be applied in multiple iterations, which however also increases the load on the DNS resolver during the computation of the SHA-1 hashes in NSEC3 records. Concerns about the load created by the computation of NSEC3 records on the DNS resolvers were already considered in the NSEC3 specifications RFC5155 and RFC9276. In February 2024, the potential of NSEC3 to exhaust DNS resolvers' resources was assigned a CVE-2023-50868, confirming that extra iterations of NSEC3 created substantial load. However, there is no published evaluation of the attack and the impact of the attack on the resolvers was not clarified. In this work we perform the first evaluation of the NSEC3-encloser attack against DNS resolver implementations and find that the NSEC3-encloser attack can still create a 72x increase in CPU instruction count, despite the victim resolver following RFC5155 recommendations in limiting hash iteration counts. The impact of the attack varies across the different DNS resolvers, but we show that with a sufficient volume of DNS packets the attack can increase CPU load and cause packet loss. We find that at a rate of 150 malicious NSEC3 records per second, depending on the DNS implementation, the loss rate of benign DNS requests varies between 2.7% and 30%. We provide a detailed description and implementation the NSEC3-encloser attack along with evaluation against five popular DNS resolver implementations. We also develop the first analysis how each NSEC3 parameter impacts the load inflicted on the victim resolver during NSEC3-encloser attack.

Title: Shadow Generation for Composite Image Using Diffusion model

• Authors: Qingyang Liu, Junqi You, Jianting Wang, Xinhao Tao, Bo Zhang, Li Niu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15234
• Pdf URL: https://arxiv.org/pdf/2403.15234
• Copy Paste: [[2403.15234]] Shadow Generation for Composite Image Using Diffusion model(https://arxiv.org/abs/2403.15234)
• Keywords: diffusion
• Abstract: In the realm of image composition, generating realistic shadow for the inserted foreground remains a formidable challenge. Previous works have developed image-to-image translation models which are trained on paired training data. However, they are struggling to generate shadows with accurate shapes and intensities, hindered by data scarcity and inherent task complexity. In this paper, we resort to foundation model with rich prior knowledge of natural shadow images. Specifically, we first adapt ControlNet to our task and then propose intensity modulation modules to improve the shadow intensity. Moreover, we extend the small-scale DESOBA dataset to DESOBAv2 using a novel data acquisition pipeline. Experimental results on both DESOBA and DESOBAv2 datasets as well as real composite images demonstrate the superior capability of our model for shadow generation task. The dataset, code, and model are released at https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBAv2.

Title: IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection

• Authors: Junbo Yin, Jianbing Shen, Runnan Chen, Wei Li, Ruigang Yang, Pascal Frossard, Wenguan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15241
• Pdf URL: https://arxiv.org/pdf/2403.15241
• Copy Paste: [[2403.15241]] IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection(https://arxiv.org/abs/2403.15241)
• Keywords: transformer
• Abstract: Bird's eye view (BEV) representation has emerged as a dominant solution for describing 3D space in autonomous driving scenarios. However, objects in the BEV representation typically exhibit small sizes, and the associated point cloud context is inherently sparse, which leads to great challenges for reliable 3D perception. In this paper, we propose IS-Fusion, an innovative multimodal fusion framework that jointly captures the Instance- and Scene-level contextual information. IS-Fusion essentially differs from existing approaches that only focus on the BEV scene-level fusion by explicitly incorporating instance-level multimodal information, thus facilitating the instance-centric tasks like 3D object detection. It comprises a Hierarchical Scene Fusion (HSF) module and an Instance-Guided Fusion (IGF) module. HSF applies Point-to-Grid and Grid-to-Region transformers to capture the multimodal scene context at different granularities. IGF mines instance candidates, explores their relationships, and aggregates the local multimodal context for each instance. These instances then serve as guidance to enhance the scene feature and yield an instance-aware BEV representation. On the challenging nuScenes benchmark, IS-Fusion outperforms all the published multimodal works to date. Code is available at: https://github.com/yinjunbo/IS-Fusion.

Title: A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness

• Authors: Zhenyu Sun, Ermin Wei
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2403.15244
• Pdf URL: https://arxiv.org/pdf/2403.15244
• Copy Paste: [[2403.15244]] A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness(https://arxiv.org/abs/2403.15244)
• Keywords: robust
• Abstract: Classical convergence analyses for optimization algorithms rely on the widely-adopted uniform smoothness assumption. However, recent experimental studies have demonstrated that many machine learning problems exhibit non-uniform smoothness, meaning the smoothness factor is a function of the model parameter instead of a universal constant. In particular, it has been observed that the smoothness grows with respect to the gradient norm along the training trajectory. Motivated by this phenomenon, the recently introduced $(L_0, L_1)$-smoothness is a more general notion, compared to traditional $L$-smoothness, that captures such positive relationship between smoothness and gradient norm. Under this type of non-uniform smoothness, existing literature has designed stochastic first-order algorithms by utilizing gradient clipping techniques to obtain the optimal $\mathcal{O}(\epsilon^{-3})$ sample complexity for finding an $\epsilon$-approximate first-order stationary solution. Nevertheless, the studies of quasi-Newton methods are still lacking. Considering higher accuracy and more robustness for quasi-Newton methods, in this paper we propose a fast stochastic quasi-Newton method when there exists non-uniformity in smoothness. Leveraging gradient clipping and variance reduction, our algorithm can achieve the best-known $\mathcal{O}(\epsilon^{-3})$ sample complexity and enjoys convergence speedup with simple hyperparameter tuning. Our numerical experiments show that our proposed algorithm outperforms the state-of-the-art approaches.

Title: Reasoning-Enhanced Object-Centric Learning for Videos

• Authors: Jian Li, Pu Ren, Yang Liu, Hao Sun
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15245
• Pdf URL: https://arxiv.org/pdf/2403.15245
• Copy Paste: [[2403.15245]] Reasoning-Enhanced Object-Centric Learning for Videos(https://arxiv.org/abs/2403.15245)
• Keywords: transformer
• Abstract: Object-centric learning aims to break down complex visual scenes into more manageable object representations, enhancing the understanding and reasoning abilities of machine learning systems toward the physical world. Recently, slot-based video models have demonstrated remarkable proficiency in segmenting and tracking objects, but they overlook the importance of the effective reasoning module. In the real world, reasoning and predictive abilities play a crucial role in human perception and object tracking; in particular, these abilities are closely related to human intuitive physics. Inspired by this, we designed a novel reasoning module called the Slot-based Time-Space Transformer with Memory buffer (STATM) to enhance the model's perception ability in complex scenes. The memory buffer primarily serves as storage for slot information from upstream modules, the Slot-based Time-Space Transformer makes predictions through slot-based spatiotemporal attention computations and fusion. Our experiment results on various datasets show that STATM can significantly enhance object-centric learning capabilities of slot-based video models.

Title: Self-Supervised Backbone Framework for Diverse Agricultural Vision Tasks

• Authors: Sudhir Sornapudi (1), Rajhans Singh (1) ((1) Corteva Agriscience, Indianapolis, USA)
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.15248
• Pdf URL: https://arxiv.org/pdf/2403.15248
• Copy Paste: [[2403.15248]] Self-Supervised Backbone Framework for Diverse Agricultural Vision Tasks(https://arxiv.org/abs/2403.15248)
• Keywords: robust
• Abstract: Computer vision in agriculture is game-changing with its ability to transform farming into a data-driven, precise, and sustainable industry. Deep learning has empowered agriculture vision to analyze vast, complex visual data, but heavily rely on the availability of large annotated datasets. This remains a bottleneck as manual labeling is error-prone, time-consuming, and expensive. The lack of efficient labeling approaches inspired us to consider self-supervised learning as a paradigm shift, learning meaningful feature representations from raw agricultural image data. In this work, we explore how self-supervised representation learning unlocks the potential applicability to diverse agriculture vision tasks by eliminating the need for large-scale annotated datasets. We propose a lightweight framework utilizing SimCLR, a contrastive learning approach, to pre-train a ResNet-50 backbone on a large, unannotated dataset of real-world agriculture field images. Our experimental analysis and results indicate that the model learns robust features applicable to a broad range of downstream agriculture tasks discussed in the paper. Additionally, the reduced reliance on annotated data makes our approach more cost-effective and accessible, paving the way for broader adoption of computer vision in agriculture.

Title: Spectral Motion Alignment for Video Motion Transfer using Diffusion Models

• Authors: Geon Yeong Park, Hyeonho Jeong, Sang Wan Lee, Jong Chul Ye
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15249
• Pdf URL: https://arxiv.org/pdf/2403.15249
• Copy Paste: [[2403.15249]] Spectral Motion Alignment for Video Motion Transfer using Diffusion Models(https://arxiv.org/abs/2403.15249)
• Keywords: diffusion
• Abstract: The evolution of diffusion models has greatly impacted video generation and understanding. Particularly, text-to-video diffusion models (VDMs) have significantly facilitated the customization of input video with target appearance, motion, etc. Despite these advances, challenges persist in accurately distilling motion information from video frames. While existing works leverage the consecutive frame residual as the target motion vector, they inherently lack global motion context and are vulnerable to frame-wise distortions. To address this, we present Spectral Motion Alignment (SMA), a novel framework that refines and aligns motion vectors using Fourier and wavelet transforms. SMA learns motion patterns by incorporating frequency-domain regularization, facilitating the learning of whole-frame global motion dynamics, and mitigating spatial artifacts. Extensive experiments demonstrate SMA's efficacy in improving motion transfer while maintaining computational efficiency and compatibility across various video customization frameworks.

Title: Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach

• Authors: Kun Sun, Rong Wang, Haitao Liu, Anders Søgaard
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15250
• Pdf URL: https://arxiv.org/pdf/2403.15250
• Copy Paste: [[2403.15250]] Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach(https://arxiv.org/abs/2403.15250)
• Keywords: robust
• Abstract: Amidst the rapid evolution of LLMs, the significance of evaluation in comprehending and propelling these models forward is increasingly paramount. Evaluations have revealed that factors such as scaling, training types, architectures and other factors profoundly impact the performance of LLMs. However, the extent and nature of these impacts continue to be subjects of debate because most assessments have been restricted to a limited number of models and data points. Clarifying the effects of these factors on performance scores can be more effectively achieved through a statistical lens. Our study embarks on a thorough re-examination of these LLMs, targeting the inadequacies in current evaluation methods. With the advent of a uniform evaluation framework, our research leverages an expansive dataset of evaluation results, introducing a comprehensive statistical methodology. This includes the application of ANOVA, Tukey HSD tests, GAMM, and clustering technique, offering a robust and transparent approach to deciphering LLM performance data. Contrary to prevailing findings, our results challenge assumptions about emergent abilities and the influence of given training types and architectures in LLMs. These findings furnish new perspectives on the characteristics, intrinsic nature, and developmental trajectories of LLMs. By providing straightforward and reliable methods to scrutinize and reassess LLM performance data, this study contributes a nuanced perspective on LLM efficiency and potentials.

Title: Federated Bayesian Deep Learning: The Application of Statistical Aggregation Methods to Bayesian Models

• Authors: John Fischer, Marko Orescanin, Justin Loomis, Patrick McClure
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.15263
• Pdf URL: https://arxiv.org/pdf/2403.15263
• Copy Paste: [[2403.15263]] Federated Bayesian Deep Learning: The Application of Statistical Aggregation Methods to Bayesian Models(https://arxiv.org/abs/2403.15263)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) is an approach to training machine learning models that takes advantage of multiple distributed datasets while maintaining data privacy and reducing communication costs associated with sharing local datasets. Aggregation strategies have been developed to pool or fuse the weights and biases of distributed deterministic models; however, modern deterministic deep learning (DL) models are often poorly calibrated and lack the ability to communicate a measure of epistemic uncertainty in prediction, which is desirable for remote sensing platforms and safety-critical applications. Conversely, Bayesian DL models are often well calibrated and capable of quantifying and communicating a measure of epistemic uncertainty along with a competitive prediction accuracy. Unfortunately, because the weights and biases in Bayesian DL models are defined by a probability distribution, simple application of the aggregation methods associated with FL schemes for deterministic models is either impossible or results in sub-optimal performance. In this work, we use independent and identically distributed (IID) and non-IID partitions of the CIFAR-10 dataset and a fully variational ResNet-20 architecture to analyze six different aggregation strategies for Bayesian DL models. Additionally, we analyze the traditional federated averaging approach applied to an approximate Bayesian Monte Carlo dropout model as a lightweight alternative to more complex variational inference methods in FL. We show that aggregation strategy is a key hyperparameter in the design of a Bayesian FL system with downstream effects on accuracy, calibration, uncertainty quantification, training stability, and client compute requirements.

Title: Parametric PDE Control with Deep Reinforcement Learning and Differentiable L0-Sparse Polynomial Policies

• Authors: Nicolò Botteghi, Urban Fasel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15267
• Pdf URL: https://arxiv.org/pdf/2403.15267
• Copy Paste: [[2403.15267]] Parametric PDE Control with Deep Reinforcement Learning and Differentiable L0-Sparse Polynomial Policies(https://arxiv.org/abs/2403.15267)
• Keywords: robust, interpretability, diffusion
• Abstract: Optimal control of parametric partial differential equations (PDEs) is crucial in many applications in engineering and science. In recent years, the progress in scientific machine learning has opened up new frontiers for the control of parametric PDEs. In particular, deep reinforcement learning (DRL) has the potential to solve high-dimensional and complex control problems in a large variety of applications. Most DRL methods rely on deep neural network (DNN) control policies. However, for many dynamical systems, DNN-based control policies tend to be over-parametrized, which means they need large amounts of training data, show limited robustness, and lack interpretability. In this work, we leverage dictionary learning and differentiable L$_0$ regularization to learn sparse, robust, and interpretable control policies for parametric PDEs. Our sparse policy architecture is agnostic to the DRL method and can be used in different policy-gradient and actor-critic DRL algorithms without changing their policy-optimization procedure. We test our approach on the challenging tasks of controlling parametric Kuramoto-Sivashinsky and convection-diffusion-reaction PDEs. We show that our method (1) outperforms baseline DNN-based DRL policies, (2) allows for the derivation of interpretable equations of the learned optimal control laws, and (3) generalizes to unseen parameters of the PDE without retraining the policies.

Title: Imagination Augmented Generation: Learning to Imagine Richer Context for Question Answering over Large Language Models

• Authors: Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, Shengping Liu, Jun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15268
• Pdf URL: https://arxiv.org/pdf/2403.15268
• Copy Paste: [[2403.15268]] Imagination Augmented Generation: Learning to Imagine Richer Context for Question Answering over Large Language Models(https://arxiv.org/abs/2403.15268)
• Keywords: large language model
• Abstract: Retrieval-Augmented-Generation and Gener-ation-Augmented-Generation have been proposed to enhance the knowledge required for question answering over Large Language Models (LLMs). However, the former depends on external resources, and both require incorporating the explicit documents into the context, which results in longer contexts that lead to more resource consumption. Recent works indicate that LLMs have modeled rich knowledge, albeit not effectively triggered or activated. Inspired by this, we propose a novel knowledge-augmented framework, Imagination-Augmented-Generation (IAG), which simulates the human capacity to compensate for knowledge deficits while answering questions solely through imagination, without relying on external resources. Guided by IAG, we propose an imagine richer context method for question answering (IMcQA), which obtains richer context through the following two modules: explicit imagination by generating a short dummy document with long context compress and implicit imagination with HyperNetwork for generating adapter weights. Experimental results on three datasets demonstrate that IMcQA exhibits significant advantages in both open-domain and closed-book settings, as well as in both in-distribution performance and out-of-distribution generalizations. Our code will be available at https://github.com/Xnhyacinth/IAG.

Title: From Hardware Fingerprint to Access Token: Enhancing the Authentication on IoT Devices

• Authors: Yue Xiao, Yi He, Xiaoli Zhang, Qian Wang, Renjie Xie, Kun Sun, Ke Xu, Qi Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.15271
• Pdf URL: https://arxiv.org/pdf/2403.15271
• Copy Paste: [[2403.15271]] From Hardware Fingerprint to Access Token: Enhancing the Authentication on IoT Devices(https://arxiv.org/abs/2403.15271)
• Keywords: secure, attack, steal
• Abstract: The proliferation of consumer IoT products in our daily lives has raised the need for secure device authentication and access control. Unfortunately, these resource-constrained devices typically use token-based authentication, which is vulnerable to token compromise attacks that allow attackers to impersonate the devices and perform malicious operations by stealing the access token. Using hardware fingerprints to secure their authentication is a promising way to mitigate these threats. However, once attackers have stolen some hardware fingerprints (e.g., via MitM attacks), they can bypass the hardware authentication by training a machine learning model to mimic fingerprints or reusing these fingerprints to craft forge requests. In this paper, we present MCU-Token, a secure hardware fingerprinting framework for MCU-based IoT devices even if the cryptographic mechanisms (e.g., private keys) are compromised. MCU-Token can be easily integrated with various IoT devices by simply adding a short hardware fingerprint-based token to the existing payload. To prevent the reuse of this token, we propose a message mapping approach that binds the token to a specific request via generating the hardware fingerprints based on the request payload. To defeat the machine learning attacks, we mix the valid fingerprints with poisoning data so that attackers cannot train a usable model with the leaked tokens. MCU-Token can defend against armored adversary who may replay, craft, and offload the requests via MitM or use both hardware (e.g., use identical devices) and software (e.g., machine learning attacks) strategies to mimic the fingerprints. The system evaluation shows that MCU-Token can achieve high accuracy (over 97%) with a low overhead across various IoT devices and application scenarios.

Title: Event Temporal Relation Extraction based on Retrieval-Augmented on LLMs

• Authors: Xiaobin Zhang, Liangjun Zang, Qianwen Liu, Shuchong Wei, Songlin Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15273
• Pdf URL: https://arxiv.org/pdf/2403.15273
• Copy Paste: [[2403.15273]] Event Temporal Relation Extraction based on Retrieval-Augmented on LLMs(https://arxiv.org/abs/2403.15273)
• Keywords: extraction, large language model
• Abstract: Event temporal relation (TempRel) is a primary subject of the event relation extraction task. However, the inherent ambiguity of TempRel increases the difficulty of the task. With the rise of prompt engineering, it is important to design effective prompt templates and verbalizers to extract relevant knowledge. The traditional manually designed templates struggle to extract precise temporal knowledge. This paper introduces a novel retrieval-augmented TempRel extraction approach, leveraging knowledge retrieved from large language models (LLMs) to enhance prompt templates and verbalizers. Our method capitalizes on the diverse capabilities of various LLMs to generate a wide array of ideas for template and verbalizer design. Our proposed method fully exploits the potential of LLMs for generation tasks and contributes more knowledge to our design. Empirical evaluations across three widely recognized datasets demonstrate the efficacy of our method in improving the performance of event temporal relation extraction tasks.

Title: Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions

• Authors: Max Dallabetta, Conrad Dobberstein, Adrian Breiding, Alan Akbik
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.15279
• Pdf URL: https://arxiv.org/pdf/2403.15279
• Copy Paste: [[2403.15279]] Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions(https://arxiv.org/abs/2403.15279)
• Keywords: extraction
• Abstract: This paper introduces Fundus, a user-friendly news scraper that enables users to obtain millions of high-quality news articles with just a few lines of code. Unlike existing news scrapers, we use manually crafted, bespoke content extractors that are specifically tailored to the formatting guidelines of each supported online newspaper. This allows us to optimize our scraping for quality such that retrieved news articles are textually complete and without HTML artifacts. Further, our framework combines both crawling (retrieving HTML from the web or large web archives) and content extraction into a single pipeline. By providing a unified interface for a predefined collection of newspapers, we aim to make Fundus broadly usable even for non-technical users. This paper gives an overview of the framework, discusses our design choices, and presents a comparative evaluation against other popular news scrapers. Our evaluation shows that Fundus yields significantly higher quality extractions (complete and artifact-free news articles) than prior work. The framework is available on GitHub under https://github.com/flairNLP/fundus and can be simply installed using pip.

Title: Controlled Training Data Generation with Diffusion Models

• Authors: Teresa Yeo, Andrei Atanov, Harold Benoit, Aleksandr Alekseev, Ruchira Ray, Pooya Esmaeil Akhoondi, Amir Zamir
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15309
• Pdf URL: https://arxiv.org/pdf/2403.15309
• Copy Paste: [[2403.15309]] Controlled Training Data Generation with Diffusion Models(https://arxiv.org/abs/2403.15309)
• Keywords: diffusion, generative
• Abstract: In this work, we present a method to control a text-to-image generative model to produce training data specifically "useful" for supervised learning. Unlike previous works that employ an open-loop approach and pre-define prompts to generate new data using either a language model or human expertise, we develop an automated closed-loop system which involves two feedback mechanisms. The first mechanism uses feedback from a given supervised model and finds adversarial prompts that result in image generations that maximize the model loss. While these adversarial prompts result in diverse data informed by the model, they are not informed of the target distribution, which can be inefficient. Therefore, we introduce the second feedback mechanism that guides the generation process towards a certain target distribution. We call the method combining these two mechanisms Guided Adversarial Prompts. We perform our evaluations on different tasks, datasets and architectures, with different types of distribution shifts (spuriously correlated data, unseen domains) and demonstrate the efficiency of the proposed feedback mechanisms compared to open-loop approaches.

Title: CO-Fun: A German Dataset on Company Outsourcing in Fund Prospectuses for Named Entity Recognition and Relation Extraction

• Authors: Neda Foroutan, Markus Schröder, Andreas Dengel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15322
• Pdf URL: https://arxiv.org/pdf/2403.15322
• Copy Paste: [[2403.15322]] CO-Fun: A German Dataset on Company Outsourcing in Fund Prospectuses for Named Entity Recognition and Relation Extraction(https://arxiv.org/abs/2403.15322)
• Keywords: extraction
• Abstract: The process of cyber mapping gives insights in relationships among financial entities and service providers. Centered around the outsourcing practices of companies within fund prospectuses in Germany, we introduce a dataset specifically designed for named entity recognition and relation extraction tasks. The labeling process on 948 sentences was carried out by three experts which yields to 5,969 annotations for four entity types (Outsourcing, Company, Location and Software) and 4,102 relation annotations (Outsourcing-Company, Company-Location). State-of-the-art deep learning models were trained to recognize entities and extract relations showing first promising results. An anonymized version of the dataset, along with guidelines and the code used for model training, are publicly available at https://www.dfki.uni-kl.de/cybermapping/data/CO-Fun-1.0-anonymized.zip.

Title: Fully automated workflow for the design of patient-specific orthopaedic implants: application to total knee arthroplasty

• Authors: Aziliz Guezou-Philippe, Arnaud Clavé, Ehouarn Maguet, Ludivine Maintier, Charles Garraud, Jean-Rassaire Fouefack, Valérie Burdin, Eric Stindel, Guillaume Dardenne
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15353
• Pdf URL: https://arxiv.org/pdf/2403.15353
• Copy Paste: [[2403.15353]] Fully automated workflow for the design of patient-specific orthopaedic implants: application to total knee arthroplasty(https://arxiv.org/abs/2403.15353)
• Keywords: segmentation
• Abstract: Arthroplasty is commonly performed to treat joint osteoarthritis, reducing pain and improving mobility. While arthroplasty has known several technical improvements, a significant share of patients are still unsatisfied with their surgery. Personalised arthroplasty improves surgical outcomes however current solutions require delays, making it difficult to integrate in clinical routine. We propose a fully automated workflow to design patient-specific implants, presented for total knee arthroplasty, the most widely performed arthroplasty in the world nowadays. The proposed pipeline first uses artificial neural networks to segment the proximal and distal extremities of the femur and tibia. Then the full bones are reconstructed using augmented statistical shape models, combining shape and landmarks information. Finally, 77 morphological parameters are computed to design patient-specific implants. The developed workflow has been trained using 91 CT scans of lower limb and evaluated on 41 CT scans manually segmented, in terms of accuracy and execution time. The workflow accuracy was $0.4\pm0.2mm$ for the segmentation, $1.2\pm0.4mm$ for the full bones reconstruction, and $2.8\pm2.2mm$ for the anatomical landmarks determination. The custom implants fitted the patients' anatomy with $0.6\pm0.2mm$ accuracy. The whole process from segmentation to implants' design lasted about 5 minutes. The proposed workflow allows for a fast and reliable personalisation of knee implants, directly from the patient CT image without requiring any manual intervention. It establishes a patient-specific pre-operative planning for TKA in a very short time making it easily available for all patients. Combined with efficient implant manufacturing techniques, this solution could help answer the growing number of arthroplasties while reducing complications and improving the patients' satisfaction.

Title: Neural Plasticity-Inspired Foundation Model for Observing the Earth Crossing Modalities

• Authors: Zhitong Xiong, Yi Wang, Fahong Zhang, Adam J. Stewart, Joëlle Hanna, Damian Borth, Ioannis Papoutsis, Bertrand Le Saux, Gustau Camps-Valls, Xiao Xiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.15356
• Pdf URL: https://arxiv.org/pdf/2403.15356
• Copy Paste: [[2403.15356]] Neural Plasticity-Inspired Foundation Model for Observing the Earth Crossing Modalities(https://arxiv.org/abs/2403.15356)
• Keywords: transformer
• Abstract: The development of foundation models has revolutionized our ability to interpret the Earth's surface using satellite observational data. Traditional models have been siloed, tailored to specific sensors or data types like optical, radar, and hyperspectral, each with its own unique characteristics. This specialization hinders the potential for a holistic analysis that could benefit from the combined strengths of these diverse data sources. Our novel approach introduces the Dynamic One-For-All (DOFA) model, leveraging the concept of neural plasticity in brain science to integrate various data modalities into a single framework adaptively. This dynamic hypernetwork, adjusting to different wavelengths, enables a single versatile Transformer jointly trained on data from five sensors to excel across 12 distinct Earth observation tasks, including sensors never seen during pretraining. DOFA's innovative design offers a promising leap towards more accurate, efficient, and unified Earth observation analysis, showcasing remarkable adaptability and performance in harnessing the potential of multimodal Earth observation data.

Title: SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series

• Authors: Badri N. Patro, Vijay S. Agneeswaran
• Subjects: cs.CV, cs.LG, eess.IV, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.15360
• Pdf URL: https://arxiv.org/pdf/2403.15360
• Copy Paste: [[2403.15360]] SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series(https://arxiv.org/abs/2403.15360)
• Keywords: transformer
• Abstract: Transformers have widely adopted attention networks for sequence mixing and MLPs for channel mixing, playing a pivotal role in achieving breakthroughs across domains. However, recent literature highlights issues with attention networks, including low inductive bias and quadratic complexity concerning input sequence length. State Space Models (SSMs) like S4 and others (Hippo, Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address the above issues to help handle longer sequence lengths. Mamba, while being the state-of-the-art SSM, has a stability issue when scaled to large networks for computer vision datasets. We propose SiMBA, a new architecture that introduces Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations and uses the Mamba block for sequence modeling. Extensive performance studies across image and time-series benchmarks demonstrate that SiMBA outperforms existing SSMs, bridging the performance gap with state-of-the-art transformers. Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet and transfer learning benchmarks such as Stanford Car and Flower as well as task learning benchmarks as well as seven time series benchmark datasets. The project page is available on this website ~\url{https://github.com/badripatro/Simba}.

Title: Learning Topological Representations for Deep Image Understanding

• Authors: Xiaoling Hu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15361
• Pdf URL: https://arxiv.org/pdf/2403.15361
• Copy Paste: [[2403.15361]] Learning Topological Representations for Deep Image Understanding(https://arxiv.org/abs/2403.15361)
• Keywords: segmentation
• Abstract: In many scenarios, especially biomedical applications, the correct delineation of complex fine-scaled structures such as neurons, tissues, and vessels is critical for downstream analysis. Despite the strong predictive power of deep learning methods, they do not provide a satisfactory representation of these structures, thus creating significant barriers in scalable annotation and downstream analysis. In this dissertation, we tackle such challenges by proposing novel representations of these topological structures in a deep learning framework. We leverage the mathematical tools from topological data analysis, i.e., persistent homology and discrete Morse theory, to develop principled methods for better segmentation and uncertainty estimation, which will become powerful tools for scalable annotation.

Title: CoLLEGe: Concept Embedding Generation for Large Language Models

• Authors: Ryan Teehan, Brenden Lake, Mengye Ren
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.15362
• Pdf URL: https://arxiv.org/pdf/2403.15362
• Copy Paste: [[2403.15362]] CoLLEGe: Concept Embedding Generation for Large Language Models(https://arxiv.org/abs/2403.15362)
• Keywords: robust, large language model
• Abstract: Current language models are unable to quickly learn new concepts on the fly, often requiring a more involved finetuning process to learn robustly. Prompting in-context is not robust to context distractions, and often fails to confer much information about the new concepts. Classic methods for few-shot word learning in NLP, relying on global word vectors, are less applicable to large language models. In this paper, we introduce a novel approach named CoLLEGe (Concept Learning with Language Embedding Generation) to modernize few-shot concept learning. CoLLEGe is a meta-learning framework capable of generating flexible embeddings for new concepts using a small number of example sentences or definitions. Our primary meta-learning objective is simply to facilitate a language model to make next word predictions in forthcoming sentences, making it compatible with language model pretraining. We design a series of tasks to test new concept learning in challenging real-world scenarios, including new word acquisition, definition inference, and verbal reasoning, and demonstrate that our method succeeds in each setting without task-specific training.

Title: Towards Knowledge-Grounded Natural Language Understanding and Generation

• Authors: Chenxi Whitehouse
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15364
• Pdf URL: https://arxiv.org/pdf/2403.15364
• Copy Paste: [[2403.15364]] Towards Knowledge-Grounded Natural Language Understanding and Generation(https://arxiv.org/abs/2403.15364)
• Keywords: transformer
• Abstract: This thesis investigates how natural language understanding and generation with transformer models can benefit from grounding the models with knowledge representations and addresses the following key research questions: (i) Can knowledge of entities extend its benefits beyond entity-centric tasks, such as entity linking? (ii) How can we faithfully and effectively extract such structured knowledge from raw text, especially noisy web text? (iii) How do other types of knowledge, beyond structured knowledge, contribute to improving NLP tasks? Studies in this thesis find that incorporating relevant and up-to-date knowledge of entities benefits fake news detection, and entity-focused code-switching significantly enhances zero-shot cross-lingual transfer on entity-centric tasks. In terms of effective and faithful approaches to extracting structured knowledge, it is observed that integrating negative examples and training with entity planning significantly improves performance. Additionally, it is established that other general forms of knowledge, such as parametric and distilled knowledge, enhance multimodal and multilingual knowledge-intensive tasks. This research shows the tangible benefits of diverse knowledge integration and motivates further exploration in this direction.

Title: A Transfer Attack to Image Watermarks

• Authors: Yuepeng Hu, Zhengyuan Jiang, Moyang Guo, Neil Gong
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15365
• Pdf URL: https://arxiv.org/pdf/2403.15365
• Copy Paste: [[2403.15365]] A Transfer Attack to Image Watermarks(https://arxiv.org/abs/2403.15365)
• Keywords: attack, robust, watermark
• Abstract: Watermark has been widely deployed by industry to detect AI-generated images. The robustness of such watermark-based detector against evasion attacks in the white-box and black-box settings is well understood in the literature. However, the robustness in the no-box setting is much less understood. In particular, multiple studies claimed that image watermark is robust in such setting. In this work, we propose a new transfer evasion attack to image watermark in the no-box setting. Our transfer attack adds a perturbation to a watermarked image to evade multiple surrogate watermarking models trained by the attacker itself, and the perturbed watermarked image also evades the target watermarking model. Our major contribution is to show that, both theoretically and empirically, watermark-based AI-generated image detector is not robust to evasion attacks even if the attacker does not have access to the watermarking model nor the detection API.

Title: Can large language models explore in-context?

• Authors: Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, Aleksandrs Slivkins
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15371
• Pdf URL: https://arxiv.org/pdf/2403.15371
• Copy Paste: [[2403.15371]] Can large language models explore in-context?(https://arxiv.org/abs/2403.15371)
• Keywords: robust, large language model
• Abstract: We investigate the extent to which contemporary Large Language Models (LLMs) can engage in exploration, a core capability in reinforcement learning and decision making. We focus on native performance of existing LLMs, without training interventions. We deploy LLMs as agents in simple multi-armed bandit environments, specifying the environment description and interaction history entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5, GPT-4, and Llama2, using a variety of prompt designs, and find that the models do not robustly engage in exploration without substantial interventions: i) Across all of our experiments, only one configuration resulted in satisfactory exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally summarized interaction history, presented as sufficient statistics; ii) All other configurations did not result in robust exploratory behavior, including those with chain-of-thought reasoning but unsummarized history. Although these findings can be interpreted positively, they suggest that external summarization -- which may not be possible in more complex settings -- is important for obtaining desirable behavior from LLM agents. We conclude that non-trivial algorithmic interventions, such as fine-tuning or dataset curation, may be required to empower LLM-based decision making agents in complex settings.

Title: LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis

• Authors: Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, Xiaohui Zeng
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15385
• Pdf URL: https://arxiv.org/pdf/2403.15385
• Copy Paste: [[2403.15385]] LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis(https://arxiv.org/abs/2403.15385)
• Keywords: robust, diffusion
• Abstract: Recent text-to-3D generation approaches produce impressive 3D results but require time-consuming optimization that can take up to an hour per prompt. Amortized methods like ATT3D optimize multiple prompts simultaneously to improve efficiency, enabling fast text-to-3D synthesis. However, they cannot capture high-frequency geometry and texture details and struggle to scale to large prompt sets, so they generalize poorly. We introduce LATTE3D, addressing these limitations to achieve fast, high-quality generation on a significantly larger prompt set. Key to our method is 1) building a scalable architecture and 2) leveraging 3D data during optimization through 3D-aware diffusion priors, shape regularization, and model initialization to achieve robustness to diverse and complex training prompts. LATTE3D amortizes both neural field and textured surface generation to produce highly detailed textured meshes in a single forward pass. LATTE3D generates 3D objects in 400ms, and can be further enhanced with fast test-time optimization.

Title: LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

• Authors: Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, Yan Yan
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.15388
• Pdf URL: https://arxiv.org/pdf/2403.15388
• Copy Paste: [[2403.15388]] LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models(https://arxiv.org/abs/2403.15388)
• Keywords: transformer, large language model
• Abstract: Large Multimodal Models (LMMs) have shown significant reasoning capabilities by connecting a visual encoder and a large language model. LMMs typically use a fixed amount of visual tokens, such as the penultimate layer features in the CLIP visual encoder, as the prefix content. Recent LMMs incorporate more complex visual inputs, such as high-resolution images and videos, which increase the number of visual tokens significantly. However, due to the design of the Transformer architecture, computational costs associated with these models tend to increase quadratically with the number of input tokens. To tackle this problem, we explore a token reduction mechanism and find, similar to prior work, that many visual tokens are spatially redundant. Based on this, we propose PruMerge, a novel adaptive visual token reduction approach, which largely reduces the number of visual tokens while maintaining comparable model performance. We first select the unpruned visual tokens based on their similarity to class tokens and spatial tokens. We then cluster the pruned tokens based on key similarity and merge the clustered tokens with the unpruned tokens to supplement their information. Empirically, when applied to LLaVA-1.5, our approach can compress the visual tokens by 14.4 times on average, and achieve comparable performance across diverse visual question-answering and reasoning tasks. Code and checkpoints are at https://llava-prumerge.github.io/.

Title: DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data

• Authors: Hanrong Ye, Dan Xu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.15389
• Pdf URL: https://arxiv.org/pdf/2403.15389
• Copy Paste: [[2403.15389]] DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data(https://arxiv.org/abs/2403.15389)
• Keywords: diffusion
• Abstract: Recently, there has been an increased interest in the practical problem of learning multiple dense scene understanding tasks from partially annotated data, where each training sample is only labeled for a subset of the tasks. The missing of task labels in training leads to low-quality and noisy predictions, as can be observed from state-of-the-art methods. To tackle this issue, we reformulate the partially-labeled multi-task dense prediction as a pixel-level denoising problem, and propose a novel multi-task denoising diffusion framework coined as DiffusionMTL. It designs a joint diffusion and denoising paradigm to model a potential noisy distribution in the task prediction or feature maps and generate rectified outputs for different tasks. To exploit multi-task consistency in denoising, we further introduce a Multi-Task Conditioning strategy, which can implicitly utilize the complementary nature of the tasks to help learn the unlabeled tasks, leading to an improvement in the denoising performance of the different tasks. Extensive quantitative and qualitative experiments demonstrate that the proposed multi-task denoising diffusion model can significantly improve multi-task prediction maps, and outperform the state-of-the-art methods on three challenging multi-task benchmarks, under two different partial-labeling evaluation settings. The code is available at https://prismformore.github.io/diffusionmtl/.