2023-12-27

secure

security

privacy

protect

Title: SODA: Protecting Proprietary Information in On-Device Machine Learning Models. (arXiv:2312.15036v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.15036
• Code URL: null
• Copy Paste: [[2312.15036]] SODA: Protecting Proprietary Information in On-Device Machine Learning Models(http://arxiv.org/abs/2312.15036)
• Summary:

  The growth of low-end hardware has led to a proliferation of machine learning-based services in edge applications. These applications gather contextual information about users and provide some services, such as personalized offers, through a machine learning (ML) model. A growing practice has been to deploy such ML models on the user's device to reduce latency, maintain user privacy, and minimize continuous reliance on a centralized source. However, deploying ML models on the user's edge device can leak proprietary information about the service provider. In this work, we investigate on-device ML models that are used to provide mobile services and demonstrate how simple attacks can leak proprietary information of the service provider. We show that different adversaries can easily exploit such models to maximize their profit and accomplish content theft. Motivated by the need to thwart such attacks, we present an end-to-end framework, SODA, for deploying and serving on edge devices while defending against adversarial usage. Our results demonstrate that SODA can detect adversarial usage with 89% accuracy in less than 50 queries with minimal impact on service performance, latency, and storage.

defense

attack

Title: Adaptive Domain Inference Attack. (arXiv:2312.15088v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.15088
• Code URL: null
• Copy Paste: [[2312.15088]] Adaptive Domain Inference Attack(http://arxiv.org/abs/2312.15088)
• Summary:

  As deep neural networks are increasingly deployed in sensitive application domains, such as healthcare and security, it's necessary to understand what kind of sensitive information can be inferred from these models. Existing model-targeted attacks all assume the attacker has known the application domain or training data distribution, which plays an essential role in successful attacks. Can removing the domain information from model APIs protect models from these attacks? This paper studies this critical problem. Unfortunately, even with minimal knowledge, i.e., accessing the model as an unnamed function without leaking the meaning of input and output, the proposed adaptive domain inference attack (ADI) can still successfully estimate relevant subsets of training data. We show that the extracted relevant data can significantly improve, for instance, the performance of model-inversion attacks. Specifically, the ADI method utilizes a concept hierarchy built on top of a large collection of available public and private datasets and a novel algorithm to adaptively tune the likelihood of leaf concepts showing up in the unseen training data. The ADI attack not only extracts partial training data at the concept level, but also converges fast and requires much fewer target-model accesses than another domain inference attack, GDI.

Title: Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation. (arXiv:2312.15112v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.15112
• Code URL: null
• Copy Paste: [[2312.15112]] Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation(http://arxiv.org/abs/2312.15112)
• Summary:

  Knowledge distillation aims to train a compact student network using soft supervision from a larger teacher network and hard supervision from ground truths. However, determining an optimal knowledge fusion ratio that balances these supervisory signals remains challenging. Prior methods generally resort to a constant or heuristic-based fusion ratio, which often falls short of a proper balance. In this study, we introduce a novel adaptive method for learning a sample-wise knowledge fusion ratio, exploiting both the correctness of teacher and student, as well as how well the student mimics the teacher on each sample. Our method naturally leads to the intra-sample trilateral geometric relations among the student prediction ($S$), teacher prediction ($T$), and ground truth ($G$). To counterbalance the impact of outliers, we further extend to the inter-sample relations, incorporating the teacher's global average prediction $\bar{T}$ for samples within the same class. A simple neural network then learns the implicit mapping from the intra- and inter-sample relations to an adaptive, sample-wise knowledge fusion ratio in a bilevel-optimization manner. Our approach provides a simple, practical, and adaptable solution for knowledge distillation that can be employed across various architectures and model sizes. Extensive experiments demonstrate consistent improvements over other loss re-weighting methods on image classification, attack detection, and click-through rate prediction.

robust

Title: AS-XAI: Self-supervised Automatic Semantic Interpretation for CNN. (arXiv:2312.14935v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14935
• Code URL: null
• Copy Paste: [[2312.14935]] AS-XAI: Self-supervised Automatic Semantic Interpretation for CNN(http://arxiv.org/abs/2312.14935)
• Summary:

  Explainable artificial intelligence (XAI) aims to develop transparent explanatory approaches for "black-box" deep learning models. However,it remains difficult for existing methods to achieve the trade-off of the three key criteria in interpretability, namely, reliability, causality, and usability, which hinder their practical applications. In this paper, we propose a self-supervised automatic semantic interpretable explainable artificial intelligence (AS-XAI) framework, which utilizes transparent orthogonal embedding semantic extraction spaces and row-centered principal component analysis (PCA) for global semantic interpretation of model decisions in the absence of human interference, without additional computational costs. In addition, the invariance of filter feature high-rank decomposition is used to evaluate model sensitivity to different semantic concepts. Extensive experiments demonstrate that robust and orthogonal semantic spaces can be automatically extracted by AS-XAI, providing more effective global interpretability for convolutional neural networks (CNNs) and generating human-comprehensible explanations. The proposed approach offers broad fine-grained extensible practical applications, including shared semantic interpretation under out-of-distribution (OOD) categories, auxiliary explanations for species that are challenging to distinguish, and classification explanations from various perspectives.

Title: SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes. (arXiv:2312.14937v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14937
• Code URL: null
• Copy Paste: [[2312.14937]] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes(http://arxiv.org/abs/2312.14937)
• Summary:

  Novel view synthesis for dynamic scenes is still a challenging problem in computer vision and graphics. Recently, Gaussian splatting has emerged as a robust technique to represent static scenes and enable high-quality and real-time novel view synthesis. Building upon this technique, we propose a new representation that explicitly decomposes the motion and appearance of dynamic scenes into sparse control points and dense Gaussians, respectively. Our key idea is to use sparse control points, significantly fewer in number than the Gaussians, to learn compact 6 DoF transformation bases, which can be locally interpolated through learned interpolation weights to yield the motion field of 3D Gaussians. We employ a deformation MLP to predict time-varying 6 DoF transformations for each control point, which reduces learning complexities, enhances learning abilities, and facilitates obtaining temporal and spatial coherent motion patterns. Then, we jointly learn the 3D Gaussians, the canonical space locations of control points, and the deformation MLP to reconstruct the appearance, geometry, and dynamics of 3D scenes. During learning, the location and number of control points are adaptively adjusted to accommodate varying motion complexities in different regions, and an ARAP loss following the principle of as rigid as possible is developed to enforce spatial continuity and local rigidity of learned motions. Finally, thanks to the explicit sparse motion representation and its decomposition from appearance, our method can enable user-controlled motion editing while retaining high-fidelity appearances. Extensive experiments demonstrate that our approach outperforms existing approaches on novel view synthesis with a high rendering speed and enables novel appearance-preserved motion editing applications.

Title: Robust Sclera Segmentation for Skin-tone Agnostic Face Image Quality Assessment. (arXiv:2312.15102v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.15102
• Code URL: null
• Copy Paste: [[2312.15102]] Robust Sclera Segmentation for Skin-tone Agnostic Face Image Quality Assessment(http://arxiv.org/abs/2312.15102)
• Summary:

  Face image quality assessment (FIQA) is crucial for obtaining good face recognition performance. FIQA algorithms should be robust and insensitive to demographic factors. The eye sclera has a consistent whitish color in all humans regardless of their age, ethnicity and skin-tone. This work proposes a robust sclera segmentation method that is suitable for face images in the enrolment and the border control face recognition scenarios. It shows how the statistical analysis of the sclera pixels produces features that are invariant to skin-tone, age and ethnicity and thus can be incorporated into FIQA algorithms to make them agnostic to demographic factors.

Title: Bridging AI and Clinical Practice: Integrating Automated Sleep Scoring Algorithm with Uncertainty-Guided Physician Review. (arXiv:2312.14996v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14996
• Code URL: null
• Copy Paste: [[2312.14996]] Bridging AI and Clinical Practice: Integrating Automated Sleep Scoring Algorithm with Uncertainty-Guided Physician Review(http://arxiv.org/abs/2312.14996)
• Summary:

  Purpose: This study aims to enhance the clinical use of automated sleep-scoring algorithms by incorporating an uncertainty estimation approach to efficiently assist clinicians in the manual review of predicted hypnograms, a necessity due to the notable inter-scorer variability inherent in polysomnography (PSG) databases. Our efforts target the extent of review required to achieve predefined agreement levels, examining both in-domain and out-of-domain data, and considering subjects diagnoses. Patients and methods: Total of 19578 PSGs from 13 open-access databases were used to train U-Sleep, a state-of-the-art sleep-scoring algorithm. We leveraged a comprehensive clinical database of additional 8832 PSGs, covering a full spectrum of ages and sleep-disorders, to refine the U-Sleep, and to evaluate different uncertainty-quantification approaches, including our novel confidence network. The ID data consisted of PSGs scored by over 50 physicians, and the two OOD sets comprised recordings each scored by a unique senior physician. Results: U-Sleep demonstrated robust performance, with Cohen's kappa (K) at 76.2% on ID and 73.8-78.8% on OOD data. The confidence network excelled at identifying uncertain predictions, achieving AUROC scores of 85.7% on ID and 82.5-85.6% on OOD data. Independently of sleep-disorder status, statistical evaluations revealed significant differences in confidence scores between aligning vs discording predictions, and significant correlations of confidence scores with classification performance metrics. To achieve K of at least 90% with physician intervention, examining less than 29.0% of uncertain epochs was required, substantially reducing physicians workload, and facilitating near-perfect agreement.

Title: Recourse under Model Multiplicity via Argumentative Ensembling. (arXiv:2312.15097v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.15097
• Code URL: https://github.com/junqi-jiang/recourse_under_model_multiplicity
• Copy Paste: [[2312.15097]] Recourse under Model Multiplicity via Argumentative Ensembling(http://arxiv.org/abs/2312.15097)
• Summary:

  Model Multiplicity (MM) arises when multiple, equally performing machine learning models can be trained to solve the same prediction task. Recent studies show that models obtained under MM may produce inconsistent predictions for the same input. When this occurs, it becomes challenging to provide counterfactual explanations (CEs), a common means for offering recourse recommendations to individuals negatively affected by models' predictions. In this paper, we formalise this problem, which we name recourse-aware ensembling, and identify several desirable properties which methods for solving it should satisfy. We show that existing ensembling methods, naturally extended in different ways to provide CEs, fail to satisfy these properties. We then introduce argumentative ensembling, deploying computational argumentation to guarantee robustness of CEs to MM, while also accommodating customisable user preferences. We show theoretically and experimentally that argumentative ensembling satisfies properties which the existing methods lack, and that the trade-offs are minimal wrt accuracy.

biometric

steal

extraction

Title: Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention. (arXiv:2312.15033v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.15033
• Code URL: null
• Copy Paste: [[2312.15033]] Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention(http://arxiv.org/abs/2312.15033)
• Summary:

  Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. However, the enigmatic ``black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. While past approaches, such as attention visualization, pivotal subnetwork extraction, and concept-based analyses, offer some insight, they often focus on either local or global explanations within a single dimension, occasionally falling short in providing comprehensive clarity. In response, we propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs. Our framework, termed SparseCBM, innovatively integrates sparsity to elucidate three intertwined layers of interpretation: input, subnetwork, and concept levels. In addition, the newly introduced dimension of interpretable inference-time intervention facilitates dynamic adjustments to the model during deployment. Through rigorous empirical evaluations on real-world datasets, we demonstrate that SparseCBM delivers a profound understanding of LLM behaviors, setting it apart in both interpreting and ameliorating model inaccuracies. Codes are provided in supplements.

membership infer

federate

Title: Federated Q-Learning: Linear Regret Speedup with Low Communication Cost. (arXiv:2312.15023v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.15023
• Code URL: null
• Copy Paste: [[2312.15023]] Federated Q-Learning: Linear Regret Speedup with Low Communication Cost(http://arxiv.org/abs/2312.15023)
• Summary:

  In this paper, we consider federated reinforcement learning for tabular episodic Markov Decision Processes (MDP) where, under the coordination of a central server, multiple agents collaboratively explore the environment and learn an optimal policy without sharing their raw data. While linear speedup in the number of agents has been achieved for some metrics, such as convergence rate and sample complexity, in similar settings, it is unclear whether it is possible to design a model-free algorithm to achieve linear regret speedup with low communication cost. We propose two federated Q-Learning algorithms termed as FedQ-Hoeffding and FedQ-Bernstein, respectively, and show that the corresponding total regrets achieve a linear speedup compared with their single-agent counterparts when the time horizon is sufficiently large, while the communication cost scales logarithmically in the total number of time steps $T$. Those results rely on an event-triggered synchronization mechanism between the agents and the server, a novel step size selection when the server aggregates the local estimates of the state-action values to form the global estimates, and a set of new concentration inequalities to bound the sum of non-martingale differences. This is the first work showing that linear regret speedup and logarithmic communication cost can be achieved by model-free algorithms in federated reinforcement learning.

fair

Title: Gaussian Harmony: Attaining Fairness in Diffusion-based Face Generation Models. (arXiv:2312.14976v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14976
• Code URL: null
• Copy Paste: [[2312.14976]] Gaussian Harmony: Attaining Fairness in Diffusion-based Face Generation Models(http://arxiv.org/abs/2312.14976)
• Summary:

  Diffusion models have achieved great progress in face generation. However, these models amplify the bias in the generation process, leading to an imbalance in distribution of sensitive attributes such as age, gender and race. This paper proposes a novel solution to this problem by balancing the facial attributes of the generated images. We mitigate the bias by localizing the means of the facial attributes in the latent space of the diffusion model using Gaussian mixture models (GMM). Our motivation for choosing GMMs over other clustering frameworks comes from the flexible latent structure of diffusion model. Since each sampling step in diffusion models follows a Gaussian distribution, we show that fitting a GMM model helps us to localize the subspace responsible for generating a specific attribute. Furthermore, our method does not require retraining, we instead localize the subspace on-the-fly and mitigate the bias for generating a fair dataset. We evaluate our approach on multiple face attribute datasets to demonstrate the effectiveness of our approach. Our results demonstrate that our approach leads to a more fair data generation in terms of representational fairness while preserving the quality of generated samples.

Title: Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases. (arXiv:2312.15011v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.15011
• Code URL: https://github.com/qi-zhangyang/gemini-vs-gpt4v
• Copy Paste: [[2312.15011]] Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases(http://arxiv.org/abs/2312.15011)
• Summary:

  The rapidly evolving sector of Multi-modal Large Language Models (MLLMs) is at the forefront of integrating linguistic and visual processing in artificial intelligence. This paper presents an in-depth comparative study of two pioneering models: Google's Gemini and OpenAI's GPT-4V(ision). Our study involves a multi-faceted evaluation of both models across key dimensions such as Vision-Language Capability, Interaction with Humans, Temporal Understanding, and assessments in both Intelligence and Emotional Quotients. The core of our analysis delves into the distinct visual comprehension abilities of each model. We conducted a series of structured experiments to evaluate their performance in various industrial application scenarios, offering a comprehensive perspective on their practical utility. We not only involve direct performance comparisons but also include adjustments in prompts and scenarios to ensure a balanced and fair analysis. Our findings illuminate the unique strengths and niches of both models. GPT-4V distinguishes itself with its precision and succinctness in responses, while Gemini excels in providing detailed, expansive answers accompanied by relevant imagery and links. These understandings not only shed light on the comparative merits of Gemini and GPT-4V but also underscore the evolving landscape of multimodal foundation models, paving the way for future advancements in this area. After the comparison, we attempted to achieve better results by combining the two models. Finally, We would like to express our profound gratitude to the teams behind GPT-4V and Gemini for their pioneering contributions to the field. Our acknowledgments are also extended to the comprehensive qualitative analysis presented in 'Dawn' by Yang et al. This work, with its extensive collection of image samples, prompts, and GPT-4V-related results, provided a foundational basis for our analysis.

interpretability

Title: SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology. (arXiv:2312.15010v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.15010
• Code URL: null
• Copy Paste: [[2312.15010]] SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathology(http://arxiv.org/abs/2312.15010)
• Summary:

  Introducing interpretability and reasoning into Multiple Instance Learning (MIL) methods for Whole Slide Image (WSI) analysis is challenging, given the complexity of gigapixel slides. Traditionally, MIL interpretability is limited to identifying salient regions deemed pertinent for downstream tasks, offering little insight to the end-user (pathologist) regarding the rationale behind these selections. To address this, we propose Self-Interpretable MIL (SI-MIL), a method intrinsically designed for interpretability from the very outset. SI-MIL employs a deep MIL framework to guide an interpretable branch grounded on handcrafted pathological features, facilitating linear predictions. Beyond identifying salient regions, SI-MIL uniquely provides feature-level interpretations rooted in pathological insights for WSIs. Notably, SI-MIL, with its linear prediction constraints, challenges the prevalent myth of an inevitable trade-off between model interpretability and performance, demonstrating competitive results compared to state-of-the-art methods on WSI-level prediction tasks across three cancer types. In addition, we thoroughly benchmark the local- and global-interpretability of SI-MIL in terms of statistical analysis, a domain expert study, and desiderata of interpretability, namely, user-friendliness and faithfulness.

explainability

watermark

diffusion

Title: Unraveling the Temporal Dynamics of the Unet in Diffusion Models. (arXiv:2312.14965v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14965
• Code URL: null
• Copy Paste: [[2312.14965]] Unraveling the Temporal Dynamics of the Unet in Diffusion Models(http://arxiv.org/abs/2312.14965)
• Summary:

  Diffusion models have garnered significant attention since they can effectively learn complex multivariate Gaussian distributions, resulting in diverse, high-quality outcomes. They introduce Gaussian noise into training data and reconstruct the original data iteratively. Central to this iterative process is a single Unet, adapting across time steps to facilitate generation. Recent work revealed the presence of composition and denoising phases in this generation process, raising questions about the Unets' varying roles. Our study dives into the dynamic behavior of Unets within denoising diffusion probabilistic models (DDPM), focusing on (de)convolutional blocks and skip connections across time steps. We propose an analytical method to systematically assess the impact of time steps and core Unet components on the final output. This method eliminates components to study causal relations and investigate their influence on output changes. The main purpose is to understand the temporal dynamics and identify potential shortcuts during inference. Our findings provide valuable insights into the various generation phases during inference and shed light on the Unets' usage patterns across these phases. Leveraging these insights, we identify redundancies in GLIDE (an improved DDPM) and improve inference time by ~27% with minimal degradation in output quality. Our ultimate goal is to guide more informed optimization strategies for inference and influence new model designs.

Title: Emage: Non-Autoregressive Text-to-Image Generation. (arXiv:2312.14988v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14988
• Code URL: null
• Copy Paste: [[2312.14988]] Emage: Non-Autoregressive Text-to-Image Generation(http://arxiv.org/abs/2312.14988)
• Summary:

  Autoregressive and diffusion models drive the recent breakthroughs on text-to-image generation. Despite their huge success of generating high-realistic images, a common shortcoming of these models is their high inference latency - autoregressive models run more than a thousand times successively to produce image tokens and diffusion models convert Gaussian noise into images with many hundreds of denoising steps. In this work, we explore non-autoregressive text-to-image models that efficiently generate hundreds of image tokens in parallel. We develop many model variations with different learning and inference strategies, initialized text encoders, etc. Compared with autoregressive baselines that needs to run one thousand times, our model only runs 16 times to generate images of competitive quality with an order of magnitude lower inference latency. Our non-autoregressive model with 346M parameters generates an image of 256$\times$256 with about one second on one V100 GPU.

Title: Synthetic images aid the recognition of human-made art forgeries. (arXiv:2312.14998v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14998
• Code URL: null
• Copy Paste: [[2312.14998]] Synthetic images aid the recognition of human-made art forgeries(http://arxiv.org/abs/2312.14998)
• Summary:

  Previous research has shown that Artificial Intelligence is capable of distinguishing between authentic paintings by a given artist and human-made forgeries with remarkable accuracy, provided sufficient training. However, with the limited amount of existing known forgeries, augmentation methods for forgery detection are highly desirable. In this work, we examine the potential of incorporating synthetic artworks into training datasets to enhance the performance of forgery detection. Our investigation focuses on paintings by Vincent van Gogh, for which we release the first dataset specialized for forgery detection. To reinforce our results, we conduct the same analyses on the artists Amedeo Modigliani and Raphael. We train a classifier to distinguish original artworks from forgeries. For this, we use human-made forgeries and imitations in the style of well-known artists and augment our training sets with images in a similar style generated by Stable Diffusion and StyleGAN. We find that the additional synthetic forgeries consistently improve the detection of human-made forgeries. In addition, we find that, in line with previous research, the inclusion of synthetic forgeries in the training also enables the detection of AI-generated forgeries, especially if created using a similar generator.

Title: FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing. (arXiv:2312.15004v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.15004
• Code URL: null
• Copy Paste: [[2312.15004]] FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing(http://arxiv.org/abs/2312.15004)
• Summary:

  Text-driven motion generation has achieved substantial progress with the emergence of diffusion models. However, existing methods still struggle to generate complex motion sequences that correspond to fine-grained descriptions, depicting detailed and accurate spatio-temporal actions. This lack of fine controllability limits the usage of motion generation to a larger audience. To tackle these challenges, we present FineMoGen, a diffusion-based motion generation and editing framework that can synthesize fine-grained motions, with spatial-temporal composition to the user instructions. Specifically, FineMoGen builds upon diffusion model with a novel transformer architecture dubbed Spatio-Temporal Mixture Attention (SAMI). SAMI optimizes the generation of the global attention template from two perspectives: 1) explicitly modeling the constraints of spatio-temporal composition; and 2) utilizing sparsely-activated mixture-of-experts to adaptively extract fine-grained features. To facilitate a large-scale study on this new fine-grained motion generation task, we contribute the HuMMan-MoGen dataset, which consists of 2,968 videos and 102,336 fine-grained spatio-temporal descriptions. Extensive experiments validate that FineMoGen exhibits superior motion generation quality over state-of-the-art methods. Notably, FineMoGen further enables zero-shot motion editing capabilities with the aid of modern large language models (LLM), which faithfully manipulates motion sequences with fine-grained instructions. Project Page: https://mingyuan-zhang.github.io/projects/FineMoGen.html

Title: Automatic Tooth Arrangement with Joint Features of Point and Mesh Representations via Diffusion Probabilistic Models. (arXiv:2312.15139v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.15139
• Code URL: null
• Copy Paste: [[2312.15139]] Automatic Tooth Arrangement with Joint Features of Point and Mesh Representations via Diffusion Probabilistic Models(http://arxiv.org/abs/2312.15139)
• Summary:

  Tooth arrangement is a crucial step in orthodontics treatment, in which aligning teeth could improve overall well-being, enhance facial aesthetics, and boost self-confidence. To improve the efficiency of tooth arrangement and minimize errors associated with unreasonable designs by inexperienced practitioners, some deep learning-based tooth arrangement methods have been proposed. Currently, most existing approaches employ MLPs to model the nonlinear relationship between tooth features and transformation matrices to achieve tooth arrangement automatically. However, the limited datasets (which to our knowledge, have not been made public) collected from clinical practice constrain the applicability of existing methods, making them inadequate for addressing diverse malocclusion issues. To address this challenge, we propose a general tooth arrangement neural network based on the diffusion probabilistic model. Conditioned on the features extracted from the dental model, the diffusion probabilistic model can learn the distribution of teeth transformation matrices from malocclusion to normal occlusion by gradually denoising from a random variable, thus more adeptly managing real orthodontic data. To take full advantage of effective features, we exploit both mesh and point cloud representations by designing different encoding networks to extract the tooth (local) and jaw (global) features, respectively. In addition to traditional metrics ADD, PA-ADD, CSA, and ME_{rot}, we propose a new evaluation metric based on dental arch curves to judge whether the generated teeth meet the individual normal occlusion. Experimental results demonstrate that our proposed method achieves state-of-the-art tooth alignment results and satisfactory occlusal relationships between dental arches. We will publish the code and dataset.

Title: Diffusion Models for Generative Artificial Intelligence: An Introduction for Applied Mathematicians. (arXiv:2312.14977v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14977
• Code URL: null
• Copy Paste: [[2312.14977]] Diffusion Models for Generative Artificial Intelligence: An Introduction for Applied Mathematicians(http://arxiv.org/abs/2312.14977)
• Summary:

  Generative artificial intelligence (AI) refers to algorithms that create synthetic but realistic output. Diffusion models currently offer state of the art performance in generative AI for images. They also form a key component in more general tools, including text-to-image generators and large language models. Diffusion models work by adding noise to the available training data and then learning how to reverse the process. The reverse operation may then be applied to new random data in order to produce new outputs. We provide a brief introduction to diffusion models for applied mathematicians and statisticians. Our key aims are (a) to present illustrative computational examples, (b) to give a careful derivation of the underlying mathematical formulas involved, and (c) to draw a connection with partial differential equation (PDE) diffusion models. We provide code for the computational experiments. We hope that this topic will be of interest to advanced undergraduate students and postgraduate students. Portions of the material may also provide useful motivational examples for those who teach courses in stochastic processes, inference, machine learning, PDEs or scientific computing.

noise learning

data-free

transformer

Title: TPTNet: A Data-Driven Temperature Prediction Model Based on Turbulent Potential Temperature. (arXiv:2312.14980v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14980
• Code URL: null
• Copy Paste: [[2312.14980]] TPTNet: A Data-Driven Temperature Prediction Model Based on Turbulent Potential Temperature(http://arxiv.org/abs/2312.14980)
• Summary:

  A data-driven model for predicting the surface temperature using neural networks was proposed to alleviate the computational burden of numerical weather prediction (NWP). Our model, named TPTNet uses only 2m temperature measured at the weather stations of the South Korean Peninsula as input to predict the local temperature at finite forecast hours. The turbulent fluctuation component of the temperature was extracted from the station measurements by separating the climatology component accounting for the yearly and daily variations. The effect of station altitude was then compensated by introducing a potential temperature. The resulting turbulent potential temperature data at irregularly distributed stations were used as input for predicting the turbulent potential temperature at forecast hours through three trained networks based on convolutional neural network (CNN), Swin Transformer, and a graphic neural network (GNN). The prediction performance of our network was compared with that of persistence and NWP, confirming that our model outperformed NWP for up to 12 forecast hours.

generative

Title: Latents2Semantics: Leveraging the Latent Space of Generative Models for Localized Style Manipulation of Face Images. (arXiv:2312.15037v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.15037
• Code URL: null
• Copy Paste: [[2312.15037]] Latents2Semantics: Leveraging the Latent Space of Generative Models for Localized Style Manipulation of Face Images(http://arxiv.org/abs/2312.15037)
• Summary:

  With the metaverse slowly becoming a reality and given the rapid pace of developments toward the creation of digital humans, the need for a principled style editing pipeline for human faces is bound to increase manifold. We cater to this need by introducing the Latents2Semantics Autoencoder (L2SAE), a Generative Autoencoder model that facilitates highly localized editing of style attributes of several Regions of Interest (ROIs) in face images. The L2SAE learns separate latent representations for encoded images' structure and style information. Thus, allowing for structure-preserving style editing of the chosen ROIs. The encoded structure representation is a multichannel 2D tensor with reduced spatial dimensions, which captures both local and global structure properties. The style representation is a 1D tensor that captures global style attributes. In our framework, we slice the structure representation to build strong and disentangled correspondences with different ROIs. Consequentially, style editing of the chosen ROIs amounts to a simple combination of (a) the ROI-mask generated from the sliced structure representation and (b) the decoded image with global style changes, generated from the manipulated (using Gaussian noise) global style and unchanged structure tensor. Style editing sans additional human supervision is a significant win over SOTA style editing pipelines because most existing works require additional human effort (supervision) post-training for attributing semantic meaning to style edits. We also do away with iterative-optimization-based inversion or determining controllable latent directions post-training, which requires additional computationally expensive operations. We provide qualitative and quantitative results for the same over multiple applications, such as selective style editing and swapping using test images sampled from several datasets.

Title: EGAIN: Extended GAn INversion. (arXiv:2312.15116v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.15116
• Code URL: null
• Copy Paste: [[2312.15116]] EGAIN: Extended GAn INversion(http://arxiv.org/abs/2312.15116)
• Summary:

  Generative Adversarial Networks (GANs) have witnessed significant advances in recent years, generating increasingly higher quality images, which are non-distinguishable from real ones. Recent GANs have proven to encode features in a disentangled latent space, enabling precise control over various semantic attributes of the generated facial images such as pose, illumination, or gender. GAN inversion, which is projecting images into the latent space of a GAN, opens the door for the manipulation of facial semantics of real face images. This is useful for numerous applications such as evaluating the performance of face recognition systems. In this work, EGAIN, an architecture for constructing GAN inversion models, is presented. This architecture explicitly addresses some of the shortcomings in previous GAN inversion models. A specific model with the same name, egain, based on this architecture is also proposed, demonstrating superior reconstruction quality over state-of-the-art models, and illustrating the validity of the EGAIN architecture.

large language model

Title: Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models. (arXiv:2312.15099v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.15099
• Code URL: https://github.com/cactilab/hateguard
• Copy Paste: [[2312.15099]] Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models(http://arxiv.org/abs/2312.15099)
• Summary:

  Online hate is an escalating problem that negatively impacts the lives of Internet users, and is also subject to rapid changes due to evolving events, resulting in new waves of online hate that pose a critical threat. Detecting and mitigating these new waves present two key challenges: it demands reasoning-based complex decision-making to determine the presence of hateful content, and the limited availability of training samples hinders updating the detection model. To address this critical issue, we present a novel framework called HATEGUARD for effectively moderating new waves of online hate. HATEGUARD employs a reasoning-based approach that leverages the recently introduced chain-of-thought (CoT) prompting technique, harnessing the capabilities of large language models (LLMs). HATEGUARD further achieves prompt-based zero-shot detection by automatically generating and updating detection prompts with new derogatory terms and targets in new wave samples to effectively address new waves of online hate. To demonstrate the effectiveness of our approach, we compile a new dataset consisting of tweets related to three recently witnessed new waves: the 2022 Russian invasion of Ukraine, the 2021 insurrection of the US Capitol, and the COVID-19 pandemic. Our studies reveal crucial longitudinal patterns in these new waves concerning the evolution of events and the pressing need for techniques to rapidly update existing moderation tools to counteract them. Comparative evaluations against state-of-the-art tools illustrate the superiority of our framework, showcasing a substantial 22.22% to 83.33% improvement in detecting the three new waves of online hate. Our work highlights the severe threat posed by the emergence of new waves of online hate and represents a paradigm shift in addressing this threat practically.

segmentation

Title: FoodLMM: A Versatile Food Assistant using Large Multi-modal Model. (arXiv:2312.14991v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14991
• Code URL: null
• Copy Paste: [[2312.14991]] FoodLMM: A Versatile Food Assistant using Large Multi-modal Model(http://arxiv.org/abs/2312.14991)
• Summary:

  Large Multi-modal Models (LMMs) have made impressive progress in many vision-language tasks. Nevertheless, the performance of general LMMs in specific domains is still far from satisfactory. This paper proposes FoodLMM, a versatile food assistant based on LMMs with various capabilities, including food recognition, ingredient recognition, recipe generation, nutrition estimation, food segmentation and multi-round conversation. To facilitate FoodLMM to deal with tasks beyond pure text output, we introduce a series of novel task-specific tokens and heads, enabling the model to predict food nutritional values and multiple segmentation masks. We adopt a two-stage training strategy. In the first stage, we utilize multiple public food benchmarks for multi-task learning by leveraging instruct-following paradigm. In the second stage, we construct a multi-round conversation and a reasoning segmentation datasets to fine-tune the model, enabling it to conduct professional dialogues and generate segmentation masks based on complex reasoning in food domain. Our fine-tuned FoodLMM achieves state-of-the-art results across several food benchmarks. We will make our code, models and datasets publicly available.

Title: Automated forest inventory: analysis of high-density airborne LiDAR point clouds with 3D deep learning. (arXiv:2312.15084v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.15084
• Code URL: null
• Copy Paste: [[2312.15084]] Automated forest inventory: analysis of high-density airborne LiDAR point clouds with 3D deep learning(http://arxiv.org/abs/2312.15084)
• Summary:

  Detailed forest inventories are critical for sustainable and flexible management of forest resources, to conserve various ecosystem services. Modern airborne laser scanners deliver high-density point clouds with great potential for fine-scale forest inventory and analysis, but automatically partitioning those point clouds into meaningful entities like individual trees or tree components remains a challenge. The present study aims to fill this gap and introduces a deep learning framework that is able to perform such a segmentation across diverse forest types and geographic regions. From the segmented data, we then derive relevant biophysical parameters of individual trees as well as stands. The system has been tested on FOR-Instance, a dataset of point clouds that have been acquired in five different countries using surveying drones. The segmentation back-end achieves over 85% F-score for individual trees, respectively over 73% mean IoU across five semantic categories: ground, low vegetation, stems, live branches and dead branches. Building on the segmentation results our pipeline then densely calculates biophysical features of each individual tree (height, crown diameter, crown volume, DBH, and location) and properties per stand (digital terrain model and stand density). Especially crown-related features are in most cases retrieved with high accuracy, whereas the estimates for DBH and location are less reliable, due to the airborne scanning setup.