2023-12-19

secure

Title: Healthcare Policy Compliance: A Blockchain Smart Contract-Based Approach. (arXiv:2312.10214v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.10214
• Code URL: null
• Copy Paste: [[2312.10214]] Healthcare Policy Compliance: A Blockchain Smart Contract-Based Approach(http://arxiv.org/abs/2312.10214)
• Summary:

  This paper addresses the critical challenge of ensuring healthcare policy compliance in the context of Electronic Health Records (EHRs). Despite stringent regulations like HIPAA, significant gaps in policy compliance often remain undetected until a data breach occurs. To bridge this gap, we propose a novel blockchain-powered, smart contract-based access control model. This model is specifically designed to enforce patient-provider agreements (PPAs) and other relevant policies, thereby ensuring both policy compliance and provenance. Our approach integrates components of informed consent into PPAs, employing blockchain smart contracts to automate and secure policy enforcement. The authorization module utilizes these contracts to make informed access decisions, recording all actions in a transparent, immutable blockchain ledger. This system not only ensures that policies are rigorously applied but also maintains a verifiable record of all actions taken, thus facilitating an easy audit and proving compliance. We implement this model in a private Ethereum blockchain setup, focusing on maintaining the integrity and lineage of policies and ensuring that audit trails are accurately and securely recorded. The Proof of Compliance (PoC) consensus mechanism enables decentralized, independent auditor nodes to verify compliance status based on the audit trails recorded. Experimental evaluation demonstrates the effectiveness of the proposed model in a simulated healthcare environment. The results show that our approach not only strengthens policy compliance and provenance but also enhances the transparency and accountability of the entire process. In summary, this paper presents a comprehensive, blockchain-based solution to a longstanding problem in healthcare data management, offering a robust framework for ensuring policy compliance and provenance through smart contracts and blockchain technology.

security

privacy

Title: Privacy-Aware Document Visual Question Answering. (arXiv:2312.10108v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10108
• Code URL: null
• Copy Paste: [[2312.10108]] Privacy-Aware Document Visual Question Answering(http://arxiv.org/abs/2312.10108)
• Summary:

  Document Visual Question Answering (DocVQA) is a fast growing branch of document understanding. Despite the fact that documents contain sensitive or copyrighted information, none of the current DocVQA methods offers strong privacy guarantees.

  In this work, we explore privacy in the domain of DocVQA for the first time. We highlight privacy issues in state of the art multi-modal LLM models used for DocVQA, and explore possible solutions.

  Specifically, we focus on the invoice processing use case as a realistic, widely used scenario for document understanding, and propose a large scale DocVQA dataset comprising invoice documents and associated questions and answers. We employ a federated learning scheme, that reflects the real-life distribution of documents in different businesses, and we explore the use case where the ID of the invoice issuer is the sensitive information to be protected.

  We demonstrate that non-private models tend to memorise, behaviour that can lead to exposing private information. We then evaluate baseline training schemes employing federated learning and differential privacy in this multi-modal scenario, where the sensitive information might be exposed through any of the two input modalities: vision (document image) or language (OCR tokens).

  Finally, we design an attack exploiting the memorisation effect of the model, and demonstrate its effectiveness in probing different DocVQA models.

protect

defense

attack

Title: Closing the Gap: Achieving Better Accuracy-Robustness Tradeoffs Against Query-Based Attacks. (arXiv:2312.10132v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10132
• Code URL: https://github.com/RUB-InfSec/closing_the_gap
• Copy Paste: [[2312.10132]] Closing the Gap: Achieving Better Accuracy-Robustness Tradeoffs Against Query-Based Attacks(http://arxiv.org/abs/2312.10132)
• Summary:

  Although promising, existing defenses against query-based attacks share a common limitation: they offer increased robustness against attacks at the price of a considerable accuracy drop on clean samples. In this work, we show how to efficiently establish, at test-time, a solid tradeoff between robustness and accuracy when mitigating query-based attacks. Given that these attacks necessarily explore low-confidence regions, our insight is that activating dedicated defenses, such as RND (Qin et al., NeuRIPS 2021) and Random Image Transformations (Xie et al., ICLR 2018), only for low-confidence inputs is sufficient to prevent them. Our approach is independent of training and supported by theory. We verify the effectiveness of our approach for various existing defenses by conducting extensive experiments on CIFAR-10, CIFAR-100, and ImageNet. Our results confirm that our proposal can indeed enhance these defenses by providing better tradeoffs between robustness and accuracy when compared to state-of-the-art approaches while being completely training-free.

robust

Title: Forging Tokens for Improved Storage-efficient Training. (arXiv:2312.10105v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10105
• Code URL: null
• Copy Paste: [[2312.10105]] Forging Tokens for Improved Storage-efficient Training(http://arxiv.org/abs/2312.10105)
• Summary:

  Recent advancements in Deep Neural Network (DNN) models have significantly improved performance across computer vision tasks. However, achieving highly generalizable and high-performing vision models requires extensive datasets, leading to large storage requirements. This storage challenge poses a critical bottleneck for scaling up vision models. Motivated by the success of discrete representations, SeiT proposes to use Vector-Quantized (VQ) feature vectors (i.e., tokens) as network inputs for vision classification. However, applying traditional data augmentations to tokens faces challenges due to input domain shift. To address this issue, we introduce TokenAdapt and ColorAdapt, simple yet effective token-based augmentation strategies. TokenAdapt realigns token embedding space for compatibility with spatial augmentations, preserving the model's efficiency without requiring fine-tuning. Additionally, ColorAdapt addresses color-based augmentations for tokens inspired by Adaptive Instance Normalization (AdaIN). We evaluate our approach across various scenarios, including storage-efficient ImageNet-1k classification, fine-grained classification, robustness benchmarks, and ADE-20k semantic segmentation. Experimental results demonstrate consistent performance improvement in diverse experiments. Code is available at https://github.com/naver-ai/tokenadapt.

Title: Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learning. (arXiv:2312.10116v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10116
• Code URL: null
• Copy Paste: [[2312.10116]] Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learning(http://arxiv.org/abs/2312.10116)
• Summary:

  The effectiveness of active learning largely depends on the sampling efficiency of the acquisition function. Expected Loss Reduction (ELR) focuses on a Bayesian estimate of the reduction in classification error, and more general costs fit in the same framework. We propose Bayesian Estimate of Mean Proper Scores (BEMPS) to estimate the increase in strictly proper scores such as log probability or negative mean square error within this framework. We also prove convergence results for this general class of costs. To facilitate better experimentation with the new acquisition functions, we develop a complementary batch AL algorithm that encourages diversity in the vector of expected changes in scores for unlabeled data. To allow high-performance classifiers, we combine deep ensembles, and dynamic validation set construction on pretrained models, and further speed up the ensemble process with the idea of Monte Carlo Dropout. Extensive experiments on both texts and images show that the use of mean square error and log probability with BEMPS yields robust acquisition functions and well-calibrated classifiers, and consistently outperforms the others tested. The advantages of BEMPS over the others are further supported by a set of qualitative analyses, where we visualise their sampling behaviour using data maps and t-SNE plots.

Title: Video-based Surgical Skill Assessment using Tree-based Gaussian Process Classifier. (arXiv:2312.10208v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10208
• Code URL: null
• Copy Paste: [[2312.10208]] Video-based Surgical Skill Assessment using Tree-based Gaussian Process Classifier(http://arxiv.org/abs/2312.10208)
• Summary:

  assessment using video data and to showcase the effectiveness of the proposed approach in evaluating surgeon proficiency, its potential for targeted training interventions, and quality assurance in surgical departments. The pipeline incorporates a representation flow convolutional neural network and a novel tree-based Gaussian process classifier, which is robust to noise, while being computationally efficient. Additionally, new kernels are introduced to enhance accuracy. The performance of the pipeline is evaluated using the JIGSAWS dataset. Comparative analysis with existing literature reveals significant improvement in accuracy and betterment in computation cost. The proposed pipeline contributes to computational efficiency and accuracy improvement in surgical skill assessment using video data. Results of our study based on comments of our colleague surgeons show that the proposed method has the potential to facilitate skill improvement among surgery fellows and enhance patient safety through targeted training interventions and quality assurance in surgical departments.

Title: T-MAE: Temporal Masked Autoencoders for Point Cloud Representation Learning. (arXiv:2312.10217v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10217
• Code URL: null
• Copy Paste: [[2312.10217]] T-MAE: Temporal Masked Autoencoders for Point Cloud Representation Learning(http://arxiv.org/abs/2312.10217)
• Summary:

  The scarcity of annotated data in outdoor point cloud segmentation poses a significant obstacle in harnessing the modeling capabilities of advanced networks like transformers. Consequently, scholars have been actively investigating efficacious self-supervised pre-training strategies, e.g. contrasting learning and reconstruction-based pretext tasks. Nevertheless, temporal information, which is inherent in the LiDAR point cloud sequence, is consistently disregarded. To better utilize this property, we propose an effective pre-training strategy, namely Temporal Masked AutoEncoders (T-MAE), which takes as input temporally adjacent frames and learns temporal dependency. A SiamWCA backbone, containing a Siamese encoder and a window-based cross-attention (WCA) module, is established for the two-frame input. Taking into account that the motion of an ego-vehicle alters the illumination angles of the same instance, temporal modeling also serves as a robust and natural data augmentation, enhancing the comprehension of target objects. Moreover, instead of utilizing consecutive frames, it is more cost-effective and powerful by using distant historical frames. SiamWCA is a powerful architecture but heavily relies on annotated data. With our T-MAE pre-training strategy, we achieve the best performance on the Waymo dataset among self-supervised learning methods. Comprehensive experiments are conducted to validate all components of our proposal. Upon acceptance, the source code will be made accessible.

Title: Towards Context-Aware Domain Generalization: Representing Environments with Permutation-Invariant Networks. (arXiv:2312.10107v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10107
• Code URL: null
• Copy Paste: [[2312.10107]] Towards Context-Aware Domain Generalization: Representing Environments with Permutation-Invariant Networks(http://arxiv.org/abs/2312.10107)
• Summary:

  In this work, we show that information about the context of an input $X$ can improve the predictions of deep learning models when applied in new domains or production environments. We formalize the notion of context as a permutation-invariant representation of a set of data points that originate from the same environment/domain as the input itself. These representations are jointly learned with a standard supervised learning objective, providing incremental information about the unknown outcome. Furthermore, we offer a theoretical analysis of the conditions under which our approach can, in principle, yield benefits, and formulate two necessary criteria that can be easily verified in practice. Additionally, we contribute insights into the kind of distribution shifts for which our approach promises robustness. Our empirical evaluation demonstrates the effectiveness of our approach for both low-dimensional and high-dimensional data sets. Finally, we demonstrate that we can reliably detect scenarios where a model is tasked with unwarranted extrapolation in out-of-distribution (OOD) domains, identifying potential failure cases. Consequently, we showcase a method to select between the most predictive and the most robust model, circumventing the well-known trade-off between predictive performance and robustness.

Title: Building symmetries into data-driven manifold dynamics models for complex flows. (arXiv:2312.10235v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10235
• Code URL: null
• Copy Paste: [[2312.10235]] Building symmetries into data-driven manifold dynamics models for complex flows(http://arxiv.org/abs/2312.10235)
• Summary:

  Symmetries in a dynamical system provide an opportunity to dramatically improve the performance of data-driven models. For fluid flows, such models are needed for tasks related to design, understanding, prediction, and control. In this work we exploit the symmetries of the Navier-Stokes equations (NSE) and use simulation data to find the manifold where the long-time dynamics live, which has many fewer degrees of freedom than the full state representation, and the evolution equation for the dynamics on that manifold. We call this method ''symmetry charting''. The first step is to map to a ''fundamental chart'', which is a region in the state space of the flow to which all other regions can be mapped by a symmetry operation. To map to the fundamental chart we identify a set of indicators from the Fourier transform that uniquely identify the symmetries of the system. We then find a low-dimensional coordinate representation of the data in the fundamental chart with the use of an autoencoder. We use a variation called an implicit rank minimizing autoencoder with weight decay, which in addition to compressing the dimension of the data, also gives estimates of how many dimensions are needed to represent the data: i.e. the dimension of the invariant manifold of the long-time dynamics. Finally, we learn dynamics on this manifold with the use of neural ordinary differential equations. We apply symmetry charting to two-dimensional Kolmogorov flow in a chaotic bursting regime. This system has a continuous translation symmetry, and discrete rotation and shift-reflect symmetries. With this framework we observe that less data is needed to learn accurate data-driven models, more robust estimates of the manifold dimension are obtained, equivariance of the NSE is satisfied, better short-time tracking with respect to the true data is observed, and long-time statistics are correctly captured.

biometric

steal

extraction

Title: Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation. (arXiv:2312.10113v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10113
• Code URL: https://github.com/guoqincode/focus-on-your-instruction
• Copy Paste: [[2312.10113]] Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation(http://arxiv.org/abs/2312.10113)
• Summary:

  Recently, diffusion-based methods, like InstructPix2Pix (IP2P), have achieved effective instruction-based image editing, requiring only natural language instructions from the user. However, these methods often inadvertently alter unintended areas and struggle with multi-instruction editing, resulting in compromised outcomes. To address these issues, we introduce the Focus on Your Instruction (FoI), a method designed to ensure precise and harmonious editing across multiple instructions without extra training or test-time optimization. In the FoI, we primarily emphasize two aspects: (1) precisely extracting regions of interest for each instruction and (2) guiding the denoising process to concentrate within these regions of interest. For the first objective, we identify the implicit grounding capability of IP2P from the cross-attention between instruction and image, then develop an effective mask extraction method. For the second objective, we introduce a cross attention modulation module for rough isolation of target editing regions and unrelated regions. Additionally, we introduce a mask-guided disentangle sampling strategy to further ensure clear region isolation. Experimental results demonstrate that FoI surpasses existing methods in both quantitative and qualitative evaluations, especially excelling in multi-instruction editing task.

Title: Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization. (arXiv:2312.10165v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10165
• Code URL: https://github.com/ynanwu/mabn
• Copy Paste: [[2312.10165]] Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization(http://arxiv.org/abs/2312.10165)
• Summary:

  Test-time domain adaptation aims to adapt the model trained on source domains to unseen target domains using a few unlabeled images. Emerging research has shown that the label and domain information is separately embedded in the weight matrix and batch normalization (BN) layer. Previous works normally update the whole network naively without explicitly decoupling the knowledge between label and domain. As a result, it leads to knowledge interference and defective distribution adaptation. In this work, we propose to reduce such learning interference and elevate the domain knowledge learning by only manipulating the BN layer. However, the normalization step in BN is intrinsically unstable when the statistics are re-estimated from a few samples. We find that ambiguities can be greatly reduced when only updating the two affine parameters in BN while keeping the source domain statistics. To further enhance the domain knowledge extraction from unlabeled data, we construct an auxiliary branch with label-independent self-supervised learning (SSL) to provide supervision. Moreover, we propose a bi-level optimization based on meta-learning to enforce the alignment of two learning objectives of auxiliary and main branches. The goal is to use the auxiliary branch to adapt the domain and benefit main task for subsequent inference. Our method keeps the same computational cost at inference as the auxiliary branch can be thoroughly discarded after adaptation. Extensive experiments show that our method outperforms the prior works on five WILDS real-world domain shift datasets. Our method can also be integrated with methods with label-dependent optimization to further push the performance boundary. Our code is available at https://github.com/ynanwu/MABN.

membership infer

federate

Title: 3FM: Multi-modal Meta-learning for Federated Tasks. (arXiv:2312.10179v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10179
• Code URL: https://github.com/minhtcai/MLMF
• Copy Paste: [[2312.10179]] 3FM: Multi-modal Meta-learning for Federated Tasks(http://arxiv.org/abs/2312.10179)
• Summary:

  We present a novel approach in the domain of federated learning (FL), particularly focusing on addressing the challenges posed by modality heterogeneity, variability in modality availability across clients, and the prevalent issue of missing data. We introduce a meta-learning framework specifically designed for multimodal federated tasks. Our approach is motivated by the need to enable federated models to robustly adapt when exposed to new modalities, a common scenario in FL where clients often differ in the number of available modalities. The effectiveness of our proposed framework is demonstrated through extensive experimentation on an augmented MNIST dataset, enriched with audio and sign language data. We demonstrate that the proposed algorithm achieves better performance than the baseline on a subset of missing modality scenarios with careful tuning of the meta-learning rates. This is a shortened report, and our work will be extended and updated soon.

fair

Title: An Information-Flow Perspective on Algorithmic Fairness. (arXiv:2312.10128v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.10128
• Code URL: null
• Copy Paste: [[2312.10128]] An Information-Flow Perspective on Algorithmic Fairness(http://arxiv.org/abs/2312.10128)
• Summary:

  This work presents insights gained by investigating the relationship between algorithmic fairness and the concept of secure information flow. The problem of enforcing secure information flow is well-studied in the context of information security: If secret information may "flow" through an algorithm or program in such a way that it can influence the program's output, then that is considered insecure information flow as attackers could potentially observe (parts of) the secret.

  There is a strong correspondence between secure information flow and algorithmic fairness: if protected attributes such as race, gender, or age are treated as secret program inputs, then secure information flow means that these ``secret'' attributes cannot influence the result of a program. While most research in algorithmic fairness evaluation concentrates on studying the impact of algorithms (often treating the algorithm as a black-box), the concepts derived from information flow can be used both for the analysis of disparate treatment as well as disparate impact w.r.t. a structural causal model.

  In this paper, we examine the relationship between quantitative as well as qualitative information-flow properties and fairness. Moreover, based on this duality, we derive a new quantitative notion of fairness called fairness spread, which can be easily analyzed using quantitative information flow and which strongly relates to counterfactual fairness. We demonstrate that off-the-shelf tools for information-flow properties can be used in order to formally analyze a program's algorithmic fairness properties, including the new notion of fairness spread as well as established notions such as demographic parity.

Title: Coupling Fairness and Pruning in a Single Run: a Bi-level Optimization Perspective. (arXiv:2312.10181v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10181
• Code URL: null
• Copy Paste: [[2312.10181]] Coupling Fairness and Pruning in a Single Run: a Bi-level Optimization Perspective(http://arxiv.org/abs/2312.10181)
• Summary:

  Deep neural networks have demonstrated remarkable performance in various tasks. With a growing need for sparse deep learning, model compression techniques, especially pruning, have gained significant attention. However, conventional pruning techniques can inadvertently exacerbate algorithmic bias, resulting in unequal predictions. To address this, we define a fair pruning task where a sparse model is derived subject to fairness requirements. In particular, we propose a framework to jointly optimize the pruning mask and weight update processes with fairness constraints. This framework is engineered to compress models that maintain performance while ensuring fairness in a single execution. To this end, we formulate the fair pruning problem as a novel constrained bi-level optimization task and derive efficient and effective solving strategies. We design experiments spanning various datasets and settings to validate our proposed method. Our empirical analysis contrasts our framework with several mainstream pruning strategies, emphasizing our method's superiority in maintaining model fairness, performance, and efficiency.

interpretability

Title: A Generic Stochastic Hybrid Car-following Model Based on Approximate Bayesian Computation. (arXiv:2312.10042v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10042
• Code URL: null
• Copy Paste: [[2312.10042]] A Generic Stochastic Hybrid Car-following Model Based on Approximate Bayesian Computation(http://arxiv.org/abs/2312.10042)
• Summary:

  Car following (CF) models are fundamental to describing traffic dynamics. However, the CF behavior of human drivers is highly stochastic and nonlinear. As a result, identifying the best CF model has been challenging and controversial despite decades of research. Introduction of automated vehicles has further complicated this matter as their CF controllers remain proprietary, though their behavior appears different than human drivers. This paper develops a stochastic learning approach to integrate multiple CF models, rather than relying on a single model. The framework is based on approximate Bayesian computation that probabilistically concatenates a pool of CF models based on their relative likelihood of describing observed behavior. The approach, while data-driven, retains physical tractability and interpretability. Evaluation results using two datasets show that the proposed approach can better reproduce vehicle trajectories for both human driven and automated vehicles than any single CF model considered.

explainability

watermark

diffusion

Title: Plasticine3D: Non-rigid 3D editting with text guidance. (arXiv:2312.10111v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10111
• Code URL: null
• Copy Paste: [[2312.10111]] Plasticine3D: Non-rigid 3D editting with text guidance(http://arxiv.org/abs/2312.10111)
• Summary:

  With the help of Score Distillation Sampling(SDS) and the rapid development of various trainable 3D representations, Text-to-Image(T2I) diffusion models have been applied to 3D generation tasks and achieved considerable results. There are also some attempts toward the task of editing 3D objects leveraging this Text-to-3D pipeline. However, most methods currently focus on adding additional geometries, overwriting textures or both. But few of them can perform non-rigid transformation of 3D objects. For those who can perform non-rigid editing, on the other hand, suffer from low-resolution, lack of fidelity and poor flexibility. In order to address these issues, we present: Plasticine3D, a general, high-fidelity, photo-realistic and controllable non-rigid editing pipeline. Firstly, our work divides the editing process into a geometry editing stage and a texture editing stage to achieve more detailed and photo-realistic results ; Secondly, in order to perform non-rigid transformation with controllable results while maintain the fidelity towards original 3D models in the same time, we propose a multi-view-embedding(MVE) optimization strategy to ensure that the diffusion model learns the overall features of the original object and an embedding-fusion(EF) to control the degree of editing by adjusting the value of the fusing rate. We also design a geometry processing step before optimizing on the base geometry to cope with different needs of various editing tasks. Further more, to fully leverage the geometric prior from the original 3D object, we provide an optional replacement of score distillation sampling named score projection sampling(SPS) which enables us to directly perform optimization from the origin 3D mesh in most common median non-rigid editing scenarios. We demonstrate the effectiveness of our method on both the non-rigid 3D editing task and general 3D editing task.

Title: MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D Human Generation. (arXiv:2312.10120v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10120
• Code URL: null
• Copy Paste: [[2312.10120]] MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D Human Generation(http://arxiv.org/abs/2312.10120)
• Summary:

  Recent months have witnessed rapid progress in 3D generation based on diffusion models. Most advances require fine-tuning existing 2D Stable Diffsuions into multi-view settings or tedious distilling operations and hence fall short of 3D human generation due to the lack of diverse 3D human datasets. We present an alternative scheme named MVHuman to generate human radiance fields from text guidance, with consistent multi-view images directly sampled from pre-trained Stable Diffsuions without any fine-tuning or distilling. Our core is a multi-view sampling strategy to tailor the denoising processes of the pre-trained network for generating consistent multi-view images. It encompasses view-consistent conditioning, replacing the original noises with ``consistency-guided noises'', optimizing latent codes, as well as utilizing cross-view attention layers. With the multi-view images through the sampling process, we adopt geometry refinement and 3D radiance field generation followed by a subsequent neural blending scheme for free-view rendering. Extensive experiments demonstrate the efficacy of our method, as well as its superiority to state-of-the-art 3D human generation methods.

Title: Tell Me What You See: Text-Guided Real-World Image Denoising. (arXiv:2312.10191v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10191
• Code URL: null
• Copy Paste: [[2312.10191]] Tell Me What You See: Text-Guided Real-World Image Denoising(http://arxiv.org/abs/2312.10191)
• Summary:

  Image reconstruction in low-light conditions is a challenging problem. Many solutions have been proposed for it, where the main approach is trying to learn a good prior of natural images along with modeling the true statistics of the noise in the scene. In the presence of very low lighting conditions, such approaches are usually not enough, and additional information is required, e.g., in the form of using multiple captures. In this work, we suggest as an alternative to add a description of the scene as prior, which can be easily done by the photographer who is capturing the scene. Using a text-conditioned diffusion model, we show that adding image caption information improves significantly the image reconstruction in low-light conditions on both synthetic and real-world images.

noise learning

data-free

transformer

Title: UniAR: Unifying Human Attention and Response Prediction on Visual Content. (arXiv:2312.10175v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10175
• Code URL: null
• Copy Paste: [[2312.10175]] UniAR: Unifying Human Attention and Response Prediction on Visual Content(http://arxiv.org/abs/2312.10175)
• Summary:

  Progress in human behavior modeling involves understanding both implicit, early-stage perceptual behavior such as human attention and explicit, later-stage behavior such as subjective ratings/preferences. Yet, most prior research has focused on modeling implicit and explicit human behavior in isolation. Can we build a unified model of human attention and preference behavior that reliably works across diverse types of visual content? Such a model would enable predicting subjective feedback such as overall satisfaction or aesthetic quality ratings, along with the underlying human attention or interaction heatmaps and viewing order, enabling designers and content-creation models to optimize their creation for human-centric improvements. In this paper, we propose UniAR -- a unified model that predicts both implicit and explicit human behavior across different types of visual content. UniAR leverages a multimodal transformer, featuring distinct prediction heads for each facet, and predicts attention heatmap, scanpath or viewing order, and subjective rating/preference. We train UniAR on diverse public datasets spanning natural images, web pages and graphic designs, and achieve leading performance on multiple benchmarks across different image domains and various behavior modeling tasks. Potential applications include providing instant feedback on the effectiveness of UIs/digital designs/images, and serving as a reward model to further optimize design/image creation.

Title: SoloPose: One-Shot Kinematic 3D Human Pose Estimation with Video Data Augmentation. (arXiv:2312.10195v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10195
• Code URL: null
• Copy Paste: [[2312.10195]] SoloPose: One-Shot Kinematic 3D Human Pose Estimation with Video Data Augmentation(http://arxiv.org/abs/2312.10195)
• Summary:

  While recent two-stage many-to-one deep learning models have demonstrated great success in 3D human pose estimation, such models are inefficient ways to detect 3D key points in a sequential video relative to one-shot and many-to-many models. Another key drawback of two-stage and many-to-one models is that errors in the first stage will be passed onto the second stage. In this paper, we introduce SoloPose, a novel one-shot, many-to-many spatio-temporal transformer model for kinematic 3D human pose estimation of video. SoloPose is further fortified by HeatPose, a 3D heatmap based on Gaussian Mixture Model distributions that factors target key points as well as kinematically adjacent key points. Finally, we address data diversity constraints with the 3D AugMotion Toolkit, a methodology to augment existing 3D human pose datasets, specifically by projecting four top public 3D human pose datasets (Humans3.6M, MADS, AIST Dance++, MPI INF 3DHP) into a novel dataset (Humans7.1M) with a universal coordinate system. Extensive experiments are conducted on Human3.6M as well as the augmented Humans7.1M dataset, and SoloPose demonstrates superior results relative to the state-of-the-art approaches.

Title: Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference. (arXiv:2312.10193v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10193
• Code URL: https://github.com/bartwojcik/adaptive_computation_modules
• Copy Paste: [[2312.10193]] Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference(http://arxiv.org/abs/2312.10193)
• Summary:

  The computational cost of transformer models makes them inefficient in low-latency or low-power applications. While techniques such as quantization or linear attention can reduce the computational load, they may incur a reduction in accuracy. In addition, globally reducing the cost for all inputs may be sub-optimal. We observe that for each layer, the full width of the layer may be needed only for a small subset of tokens inside a batch and that the "effective" width needed to process a token can vary from layer to layer. Motivated by this observation, we introduce the Adaptive Computation Module (ACM), a generic module that dynamically adapts its computational load to match the estimated difficulty of the input on a per-token basis. An ACM consists of a sequence of learners that progressively refine the output of their preceding counterparts. An additional gating mechanism determines the optimal number of learners to execute for each token. We also describe a distillation technique to replace any pre-trained model with an "ACMized" variant. The distillation phase is designed to be highly parallelizable across layers while being simple to plug-and-play into existing networks. Our evaluation of transformer models in computer vision and speech recognition demonstrates that substituting layers with ACMs significantly reduces inference costs without degrading the downstream accuracy for a wide interval of user-defined budgets.

generative

Title: NM-FlowGAN: Modeling sRGB Noise with a Hybrid Approach based on Normalizing Flows and Generative Adversarial Networks. (arXiv:2312.10112v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10112
• Code URL: https://github.com/YoungJooHan/NM-FlowGAN
• Copy Paste: [[2312.10112]] NM-FlowGAN: Modeling sRGB Noise with a Hybrid Approach based on Normalizing Flows and Generative Adversarial Networks(http://arxiv.org/abs/2312.10112)
• Summary:

  Modeling and synthesizing real sRGB noise is crucial for various low-level vision tasks. The distribution of real sRGB noise is highly complex and affected by a multitude of factors, making its accurate modeling extremely challenging. Therefore, recent studies have proposed methods that employ data-driven generative models, such as generative adversarial networks (GAN) and Normalizing Flows. These studies achieve more accurate modeling of sRGB noise compared to traditional noise modeling methods. However, there are performance limitations due to the inherent characteristics of each generative model. To address this issue, we propose NM-FlowGAN, a hybrid approach that exploits the strengths of both GAN and Normalizing Flows. We simultaneously employ a pixel-wise noise modeling network based on Normalizing Flows, and spatial correlation modeling networks based on GAN. In our experiments, our NM-FlowGAN outperforms other baselines on the sRGB noise synthesis task. Moreover, the denoising neural network, trained with synthesized image pairs from our model, also shows superior performance compared to other baselines. Our code is available at: https://github.com/YoungJooHan/NM-FlowGAN

Title: Data-Efficient Multimodal Fusion on a Single GPU. (arXiv:2312.10144v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10144
• Code URL: https://github.com/layer6ai-labs/fusemix
• Copy Paste: [[2312.10144]] Data-Efficient Multimodal Fusion on a Single GPU(http://arxiv.org/abs/2312.10144)
• Summary:

  The goal of multimodal alignment is to learn a single latent space that is shared between multimodal inputs. The most powerful models in this space have been trained using massive datasets of paired inputs and large-scale computational resources, making them prohibitively expensive to train in many practical scenarios. We surmise that existing unimodal encoders pre-trained on large amounts of unimodal data should provide an effective bootstrap to create multimodal models from unimodal ones at much lower costs. We therefore propose FuseMix, a multimodal augmentation scheme that operates on the latent spaces of arbitrary pre-trained unimodal encoders. Using FuseMix for multimodal alignment, we achieve competitive performance -- and in certain cases outperform state-of-the art methods -- in both image-text and audio-text retrieval, with orders of magnitude less compute and data: for example, we outperform CLIP on the Flickr30K text-to-image retrieval task with $\sim \! 600\times$ fewer GPU days and $\sim \! 80\times$ fewer image-text pairs. Additionally, we show how our method can be applied to convert pre-trained text-to-image generative models into audio-to-image ones. Code is available at: https://github.com/layer6ai-labs/fusemix.

Title: Towards the Unification of Generative and Discriminative Visual Foundation Model: A Survey. (arXiv:2312.10163v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10163
• Code URL: null
• Copy Paste: [[2312.10163]] Towards the Unification of Generative and Discriminative Visual Foundation Model: A Survey(http://arxiv.org/abs/2312.10163)
• Summary:

  The advent of foundation models, which are pre-trained on vast datasets, has ushered in a new era of computer vision, characterized by their robustness and remarkable zero-shot generalization capabilities. Mirroring the transformative impact of foundation models like large language models (LLMs) in natural language processing, visual foundation models (VFMs) have become a catalyst for groundbreaking developments in computer vision. This review paper delineates the pivotal trajectories of VFMs, emphasizing their scalability and proficiency in generative tasks such as text-to-image synthesis, as well as their adeptness in discriminative tasks including image segmentation. While generative and discriminative models have historically charted distinct paths, we undertake a comprehensive examination of the recent strides made by VFMs in both domains, elucidating their origins, seminal breakthroughs, and pivotal methodologies. Additionally, we collate and discuss the extensive resources that facilitate the development of VFMs and address the challenges that pave the way for future research endeavors. A crucial direction for forthcoming innovation is the amalgamation of generative and discriminative paradigms. The nascent application of generative models within discriminative contexts signifies the early stages of this confluence. This survey aspires to be a contemporary compendium for scholars and practitioners alike, charting the course of VFMs and illuminating their multifaceted landscape.

large language model

Title: GSVA: Generalized Segmentation via Multimodal Large Language Models. (arXiv:2312.10103v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10103
• Code URL: null
• Copy Paste: [[2312.10103]] GSVA: Generalized Segmentation via Multimodal Large Language Models(http://arxiv.org/abs/2312.10103)
• Summary:

  Generalized Referring Expression Segmentation (GRES) extends the scope of classic RES to referring to multiple objects in one expression or identifying the empty targets absent in the image. GRES poses challenges in modeling the complex spatial relationships of the instances in the image and identifying non-existing referents. Recently, Multimodal Large Language Models (MLLMs) have shown tremendous progress in these complicated vision-language tasks. Connecting Large Language Models (LLMs) and vision models, MLLMs are proficient in understanding contexts with visual inputs. Among them, LISA, as a representative, adopts a special [SEG] token to prompt a segmentation mask decoder, e.g., SAM, to enable MLLMs in the RES task. However, existing solutions to of GRES remain unsatisfactory since current segmentation MLLMs cannot properly handle the cases where users might reference multiple subjects in a singular prompt or provide descriptions incongruent with any image target. In this paper, we propose Generalized Segmentation Vision Assistant (GSVA) to address this gap. Specifically, GSVA reuses the [SEG] token to prompt the segmentation model towards supporting multiple mask references simultaneously and innovatively learns to generate a [REJ] token to reject the null targets explicitly. Experiments validate GSVA's efficacy in resolving the GRES issue, marking a notable enhancement and setting a new record on the GRES benchmark gRefCOCO dataset. GSVA also proves effective across various classic referring expression segmentation and comprehension tasks.

Title: Student as an Inherent Denoiser of Noisy Teacher. (arXiv:2312.10185v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10185
• Code URL: null
• Copy Paste: [[2312.10185]] Student as an Inherent Denoiser of Noisy Teacher(http://arxiv.org/abs/2312.10185)
• Summary:

  Knowledge distillation (KD) has been widely employed to transfer knowledge from a large language model (LLM) to a specialized model in low-data regimes through pseudo label learning. However, pseudo labels generated by teacher models are usually noisy and may influence KD performance. This study delves into KD with noisy teachers and uncovers that the student model can already generate more accurate predictions than the teacher labels used to train it during KD, indicating its inherent ability to denoise noisy teacher labels. Motivated by this finding, we propose Peer-Advised KD to improve vanilla KD from noisy teachers. Experiments show that Peer-Advised KD can outperform LLM by approximately 5% with 50 human-labeled data, and even competitive to standard supervised finetuning with 750 human-labeled data.

Title: Low-resource classification of mobility functioning information in clinical sentences using large language models. (arXiv:2312.10202v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.10202
• Code URL: null
• Copy Paste: [[2312.10202]] Low-resource classification of mobility functioning information in clinical sentences using large language models(http://arxiv.org/abs/2312.10202)
• Summary:

  Objective: Function is increasingly recognized as an important indicator of whole-person health. This study evaluates the ability of publicly available large language models (LLMs) to accurately identify the presence of functioning information from clinical notes. We explore various strategies to improve the performance on this task. Materials and Methods: We collect a balanced binary classification dataset of 1000 sentences from the Mobility NER dataset, which was curated from n2c2 clinical notes. For evaluation, we construct zero-shot and few-shot prompts to query the LLMs whether a given sentence contains mobility functioning information. Two sampling techniques, random sampling and k-nearest neighbor (kNN)-based sampling, are used to select the few-shot examples. Furthermore, we apply a parameter-efficient prompt-based fine-tuning method to the LLMs and evaluate their performance under various training settings. Results: Flan-T5-xxl outperforms all other models in both zero-shot and few-shot settings, achieving a F1 score of 0.865 with a single demonstrative example selected by kNN sampling. In prompt-based fine-tuning experiments, this foundation model also demonstrates superior performance across all low-resource settings, particularly achieving an impressive F1 score of 0.922 using the full training dataset. The smaller model, Flan-T5-xl, requires fine-tuning with only 2.3M additional parameters to achieve comparable performance to the fully fine-tuned Gatortron-base model, both surpassing 0.9 F1 score. Conclusion: Open-source instruction-tuned LLMs demonstrate impressive in-context learning capability in the mobility functioning classification task. The performance of these models can be further improved by continuing fine-tuning on a task-specific dataset.

segmentation

Title: FoMo-Bench: a multi-modal, multi-scale and multi-task Forest Monitoring Benchmark for remote sensing foundation models. (arXiv:2312.10114v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10114
• Code URL: null
• Copy Paste: [[2312.10114]] FoMo-Bench: a multi-modal, multi-scale and multi-task Forest Monitoring Benchmark for remote sensing foundation models(http://arxiv.org/abs/2312.10114)
• Summary:

  Forests are an essential part of Earth's ecosystems and natural systems, as well as providing services on which humanity depends, yet they are rapidly changing as a result of land use decisions and climate change. Understanding and mitigating negative effects requires parsing data on forests at global scale from a broad array of sensory modalities, and recently many such problems have been approached using machine learning algorithms for remote sensing. To date, forest-monitoring problems have largely been approached in isolation. Inspired by the rise of foundation models for computer vision and remote sensing, we here present the first unified Forest Monitoring Benchmark (FoMo-Bench). FoMo-Bench consists of 15 diverse datasets encompassing satellite, aerial, and inventory data, covering a variety of geographical regions, and including multispectral, red-green-blue, synthetic aperture radar (SAR) and LiDAR data with various temporal, spatial and spectral resolutions. FoMo-Bench includes multiple types of forest-monitoring tasks, spanning classification, segmentation, and object detection. To further enhance the diversity of tasks and geographies represented in FoMo-Bench, we introduce a novel global dataset, TalloS, combining satellite imagery with ground-based annotations for tree species classification, spanning 1,000+ hierarchical taxonomic levels (species, genus, family). Finally, we propose FoMo-Net, a foundation model baseline designed for forest monitoring with the flexibility to process any combination of commonly used sensors in remote sensing. This work aims to inspire research collaborations between machine learning and forest biology researchers in exploring scalable multi-modal and multi-task models for forest monitoring. All code and data will be made publicly available.

Title: Gradient-based Parameter Selection for Efficient Fine-Tuning. (arXiv:2312.10136v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.10136
• Code URL: null
• Copy Paste: [[2312.10136]] Gradient-based Parameter Selection for Efficient Fine-Tuning(http://arxiv.org/abs/2312.10136)
• Summary:

  With the growing size of pre-trained models, full fine-tuning and storing all the parameters for various downstream tasks is costly and infeasible. In this paper, we propose a new parameter-efficient fine-tuning method, Gradient-based Parameter Selection (GPS), demonstrating that only tuning a few selected parameters from the pre-trained model while keeping the remainder of the model frozen can generate similar or better performance compared with the full model fine-tuning method. Different from the existing popular and state-of-the-art parameter-efficient fine-tuning approaches, our method does not introduce any additional parameters and computational costs during both the training and inference stages. Another advantage is the model-agnostic and non-destructive property, which eliminates the need for any other design specific to a particular model. Compared with the full fine-tuning, GPS achieves 3.33% (91.78% vs. 88.45%, FGVC) and 9.61% (73.1% vs. 65.57%, VTAB) improvement of the accuracy with tuning only 0.36% parameters of the pre-trained model on average over 24 image classification tasks; it also demonstrates a significant improvement of 17% and 16.8% in mDice and mIoU, respectively, on medical image segmentation task. Moreover, GPS achieves state-of-the-art performance compared with existing PEFT methods.

Title: WordScape: a Pipeline to extract multilingual, visually rich Documents with Layout Annotations from Web Crawl Data. (arXiv:2312.10188v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.10188
• Code URL: null
• Copy Paste: [[2312.10188]] WordScape: a Pipeline to extract multilingual, visually rich Documents with Layout Annotations from Web Crawl Data(http://arxiv.org/abs/2312.10188)
• Summary:

  We introduce WordScape, a novel pipeline for the creation of cross-disciplinary, multilingual corpora comprising millions of pages with annotations for document layout detection. Relating visual and textual items on document pages has gained further significance with the advent of multimodal models. Various approaches proved effective for visual question answering or layout segmentation. However, the interplay of text, tables, and visuals remains challenging for a variety of document understanding tasks. In particular, many models fail to generalize well to diverse domains and new languages due to insufficient availability of training data. WordScape addresses these limitations. Our automatic annotation pipeline parses the Open XML structure of Word documents obtained from the web, jointly providing layout-annotated document images and their textual representations. In turn, WordScape offers unique properties as it (1) leverages the ubiquity of the Word file format on the internet, (2) is readily accessible through the Common Crawl web corpus, (3) is adaptive to domain-specific documents, and (4) offers culturally and linguistically diverse document pages with natural semantic structure and high-quality text. Together with the pipeline, we will additionally release 9.5M urls to word documents which can be processed using WordScape to create a dataset of over 40M pages. Finally, we investigate the quality of text and layout annotations extracted by WordScape, assess the impact on document understanding benchmarks, and demonstrate that manual labeling costs can be substantially reduced.