diffusion

Title: Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression. (arXiv:2311.10794v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10794
• Code URL: null
• Copy Paste: [[2311.10794]] Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression(http://arxiv.org/abs/2311.10794)
• Summary:

  We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models (LDMs) in a distinct domain with high visual quality, prompt alignment and scene diversity. We choose sticker image generation as the target domain, as the images significantly differ from photorealistic samples typically generated by large-scale LDMs. We start with a competent text-to-image model, like Emu, and show that relying on prompt engineering with a photorealistic model to generate stickers leads to poor prompt alignment and scene diversity. To overcome these drawbacks, we first finetune Emu on millions of sticker-like images collected using weak supervision to elicit diversity. Next, we curate human-in-the-loop (HITL) Alignment and Style datasets from model generations, and finetune to improve prompt alignment and style alignment respectively. Sequential finetuning on these datasets poses a tradeoff between better style alignment and prompt alignment gains. To address this tradeoff, we propose a novel fine-tuning method called Style Tailoring, which jointly fits the content and style distribution and achieves best tradeoff. Evaluation results show our method improves visual quality by 14%, prompt alignment by 16.2% and scene diversity by 15.3%, compared to prompt engineering the base Emu model for stickers generation.

Title: Make Pixels Dance: High-Dynamic Video Generation. (arXiv:2311.10982v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10982
• Code URL: null
• Copy Paste: [[2311.10982]] Make Pixels Dance: High-Dynamic Video Generation(http://arxiv.org/abs/2311.10982)
• Summary:

  Creating high-dynamic videos such as motion-rich actions and sophisticated visual effects poses a significant challenge in the field of artificial intelligence. Unfortunately, current state-of-the-art video generation methods, primarily focusing on text-to-video generation, tend to produce video clips with minimal motions despite maintaining high fidelity. We argue that relying solely on text instructions is insufficient and suboptimal for video generation. In this paper, we introduce PixelDance, a novel approach based on diffusion models that incorporates image instructions for both the first and last frames in conjunction with text instructions for video generation. Comprehensive experimental results demonstrate that PixelDance trained with public data exhibits significantly better proficiency in synthesizing videos with complex scenes and intricate motions, setting a new standard for video generation.

Title: Behavior Optimized Image Generation. (arXiv:2311.10995v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10995
• Code URL: null
• Copy Paste: [[2311.10995]] Behavior Optimized Image Generation(http://arxiv.org/abs/2311.10995)
• Summary:

  The last few years have witnessed great success on image generation, which has crossed the acceptance thresholds of aesthetics, making it directly applicable to personal and commercial applications. However, images, especially in marketing and advertising applications, are often created as a means to an end as opposed to just aesthetic concerns. The goal can be increasing sales, getting more clicks, likes, or image sales (in the case of stock businesses). Therefore, the generated images need to perform well on these key performance indicators (KPIs), in addition to being aesthetically good. In this paper, we make the first endeavor to answer the question of "How can one infuse the knowledge of the end-goal within the image generation process itself to create not just better-looking images but also "better-performing'' images?''. We propose BoigLLM, an LLM that understands both image content and user behavior. BoigLLM knows how an image should look to get a certain required KPI. We show that BoigLLM outperforms 13x larger models such as GPT-3.5 and GPT-4 in this task, demonstrating that while these state-of-the-art models can understand images, they lack information on how these images perform in the real world. To generate actual pixels of behavior-conditioned images, we train a diffusion-based model (BoigSD) to align with a proposed BoigLLM-defined reward. We show the performance of the overall pipeline on two datasets covering two different behaviors: a stock dataset with the number of forward actions as the KPI and a dataset containing tweets with the total likes as the KPI, denoted as BoigBench. To advance research in the direction of utility-driven image generation and understanding, we release BoigBench, a benchmark dataset containing 168 million enterprise tweets with their media, brand account names, time of post, and total likes.

Title: Improving Adversarial Transferability by Stable Diffusion. (arXiv:2311.11017v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.11017
• Code URL: null
• Copy Paste: [[2311.11017]] Improving Adversarial Transferability by Stable Diffusion(http://arxiv.org/abs/2311.11017)
• Summary:

  Deep neural networks (DNNs) are susceptible to adversarial examples, which introduce imperceptible perturbations to benign samples, deceiving DNN predictions. While some attack methods excel in the white-box setting, they often struggle in the black-box scenario, particularly against models fortified with defense mechanisms. Various techniques have emerged to enhance the transferability of adversarial attacks for the black-box scenario. Among these, input transformation-based attacks have demonstrated their effectiveness. In this paper, we explore the potential of leveraging data generated by Stable Diffusion to boost adversarial transferability. This approach draws inspiration from recent research that harnessed synthetic data generated by Stable Diffusion to enhance model generalization. In particular, previous work has highlighted the correlation between the presence of both real and synthetic data and improved model generalization. Building upon this insight, we introduce a novel attack method called Stable Diffusion Attack Method (SDAM), which incorporates samples generated by Stable Diffusion to augment input images. Furthermore, we propose a fast variant of SDAM to reduce computational overhead while preserving high adversarial transferability. Our extensive experimental results demonstrate that our method outperforms state-of-the-art baselines by a substantial margin. Moreover, our approach is compatible with existing transfer-based attacks to further enhance adversarial transferability.

Title: A Study on Altering the Latent Space of Pretrained Text to Speech Models for Improved Expressiveness. (arXiv:2311.10804v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10804
• Code URL: null
• Copy Paste: [[2311.10804]] A Study on Altering the Latent Space of Pretrained Text to Speech Models for Improved Expressiveness(http://arxiv.org/abs/2311.10804)
• Summary:

  This report explores the challenge of enhancing expressiveness control in Text-to-Speech (TTS) models by augmenting a frozen pretrained model with a Diffusion Model that is conditioned on joint semantic audio/text embeddings. The paper identifies the challenges encountered when working with a VAE-based TTS model and evaluates different image-to-image methods for altering latent speech features. Our results offer valuable insights into the complexities of adding expressiveness control to TTS systems and open avenues for future research in this direction.

self-supervised

Title: Domain Generalization of 3D Object Detection by Density-Resampling. (arXiv:2311.10845v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10845
• Code URL: null
• Copy Paste: [[2311.10845]] Domain Generalization of 3D Object Detection by Density-Resampling(http://arxiv.org/abs/2311.10845)
• Summary:

  Point-cloud-based 3D object detection suffers from performance degradation when encountering data with novel domain gaps. To tackle it, the single-domain generalization (SDG) aims to generalize the detection model trained in a limited single source domain to perform robustly on unexplored domains. In this paper, we propose an SDG method to improve the generalizability of 3D object detection to unseen target domains. Unlike prior SDG works for 3D object detection solely focusing on data augmentation, our work introduces a novel data augmentation method and contributes a new multi-task learning strategy in the methodology. Specifically, from the perspective of data augmentation, we design a universal physical-aware density-based data augmentation (PDDA) method to mitigate the performance loss stemming from diverse point densities. From the learning methodology viewpoint, we develop a multi-task learning for 3D object detection: during source training, besides the main standard detection task, we leverage an auxiliary self-supervised 3D scene restoration task to enhance the comprehension of the encoder on background and foreground details for better recognition and detection of objects. Furthermore, based on the auxiliary self-supervised task, we propose the first test-time adaptation method for domain generalization of 3D object detection, which efficiently adjusts the encoder's parameters to adapt to unseen target domains during testing time, to further bridge domain gaps. Extensive cross-dataset experiments covering "Car", "Pedestrian", and "Cyclist" detections, demonstrate our method outperforms state-of-the-art SDG methods and even overpass unsupervised domain adaptation methods under some circumstances. The code will be made publicly available.

Title: Multi-entity Video Transformers for Fine-Grained Video Representation Learning. (arXiv:2311.10873v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10873
• Code URL: https://github.com/facebookresearch/video_rep_learning
• Copy Paste: [[2311.10873]] Multi-entity Video Transformers for Fine-Grained Video Representation Learning(http://arxiv.org/abs/2311.10873)
• Summary:

  The area of temporally fine-grained video representation learning aims to generate frame-by-frame representations for temporally dense tasks. In this work, we advance the state-of-the-art for this area by re-examining the design of transformer architectures for video representation learning. A salient aspect of our self-supervised method is the improved integration of spatial information in the temporal pipeline by representing multiple entities per frame. Prior works use late fusion architectures that reduce frames to a single dimensional vector before any cross-frame information is shared, while our method represents each frame as a group of entities or tokens. Our Multi-entity Video Transformer (MV-Former) architecture achieves state-of-the-art results on multiple fine-grained video benchmarks. MV-Former leverages image features from self-supervised ViTs, and employs several strategies to maximize the utility of the extracted features while also avoiding the need to fine-tune the complex ViT backbone. This includes a Learnable Spatial Token Pooling strategy, which is used to identify and extract features for multiple salient regions per frame. Our experiments show that MV-Former not only outperforms previous self-supervised methods, but also surpasses some prior works that use additional supervision or training data. When combined with additional pre-training data from Kinetics-400, MV-Former achieves a further performance boost. The code for MV-Former is available at https://github.com/facebookresearch/video_rep_learning.

Title: Point Cloud Self-supervised Learning via 3D to Multi-view Masked Autoencoder. (arXiv:2311.10887v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10887
• Code URL: null
• Copy Paste: [[2311.10887]] Point Cloud Self-supervised Learning via 3D to Multi-view Masked Autoencoder(http://arxiv.org/abs/2311.10887)
• Summary:

  In recent years, the field of 3D self-supervised learning has witnessed significant progress, resulting in the emergence of Multi-Modality Masked AutoEncoders (MAE) methods that leverage both 2D images and 3D point clouds for pre-training. However, a notable limitation of these approaches is that they do not fully utilize the multi-view attributes inherent in 3D point clouds, which is crucial for a deeper understanding of 3D structures. Building upon this insight, we introduce a novel approach employing a 3D to multi-view masked autoencoder to fully harness the multi-modal attributes of 3D point clouds. To be specific, our method uses the encoded tokens from 3D masked point clouds to generate original point clouds and multi-view depth images across various poses. This approach not only enriches the model's comprehension of geometric structures but also leverages the inherent multi-modal properties of point clouds. Our experiments illustrate the effectiveness of the proposed method for different tasks and under different settings. Remarkably, our method outperforms state-of-the-art counterparts by a large margin in a variety of downstream tasks, including 3D object classification, few-shot learning, part segmentation, and 3D object detection. Code will be available at: https://github.com/Zhimin-C/Multiview-MAE

Title: Single-shot Phase Retrieval from a Fractional Fourier Transform Perspective. (arXiv:2311.10950v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10950
• Code URL: null
• Copy Paste: [[2311.10950]] Single-shot Phase Retrieval from a Fractional Fourier Transform Perspective(http://arxiv.org/abs/2311.10950)
• Summary:

  The realm of classical phase retrieval concerns itself with the arduous task of recovering a signal from its Fourier magnitude measurements, which are fraught with inherent ambiguities. A single-exposure intensity measurement is commonly deemed insufficient for the reconstruction of the primal signal, given that the absent phase component is imperative for the inverse transformation. In this work, we present a novel single-shot phase retrieval paradigm from a fractional Fourier transform (FrFT) perspective, which involves integrating the FrFT-based physical measurement model within a self-supervised reconstruction scheme. Specifically, the proposed FrFT-based measurement model addresses the aliasing artifacts problem in the numerical calculation of Fresnel diffraction, featuring adaptability to both short-distance and long-distance propagation scenarios. Moreover, the intensity measurement in the FrFT domain proves highly effective in alleviating the ambiguities of phase retrieval and relaxing the previous conditions on oversampled or multiple measurements in the Fourier domain. Furthermore, the proposed self-supervised reconstruction approach harnesses the fast discrete algorithm of FrFT alongside untrained neural network priors, thereby attaining preeminent results. Through numerical simulations, we demonstrate that both amplitude and phase objects can be effectively retrieved from a single-shot intensity measurement using the proposed approach and provide a promising technique for support-free coherent diffraction imaging.

Title: Lesion Search with Self-supervised Learning. (arXiv:2311.11014v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.11014
• Code URL: null
• Copy Paste: [[2311.11014]] Lesion Search with Self-supervised Learning(http://arxiv.org/abs/2311.11014)
• Summary:

  Content-based image retrieval (CBIR) with self-supervised learning (SSL) accelerates clinicians' interpretation of similar images without manual annotations. We develop a CBIR from the contrastive learning SimCLR and incorporate a generalized-mean (GeM) pooling followed by L2 normalization to classify lesion types and retrieve similar images before clinicians' analysis. Results have shown improved performance. We additionally build an open-source application for image analysis and retrieval. The application is easy to integrate, relieving manual efforts and suggesting the potential to support clinicians' everyday activities.

Title: SORTAD: Self-Supervised Optimized Random Transformations for Anomaly Detection in Tabular Data. (arXiv:2311.11018v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.11018
• Code URL: null
• Copy Paste: [[2311.11018]] SORTAD: Self-Supervised Optimized Random Transformations for Anomaly Detection in Tabular Data(http://arxiv.org/abs/2311.11018)
• Summary:

  We consider a self-supervised approach to anomaly detection in tabular data. Random transformations are applied to the data, and then each transformation is identified based on its output. These predicted transformations are used to identify anomalies. In tabular data this approach faces many challenges that are related to the uncorrelated nature of the data. These challenges affect the transformations that should be used, as well as the use of their predictions. To this end, we propose SORTAD, a novel algorithm that is tailor-made to solve these challenges. SORTAD optimally chooses random transformations that help the classification process, and have a scoring function that is more sensitive to the changes in the transformations classification prediction encountered in tabular data. SORTAD achieved state-of-the-art results on multiple commonly used anomaly detection data sets, as well as in the overall results across all data sets tested.

foundation model

Title: EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge. (arXiv:2311.10986v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10986
• Code URL: null
• Copy Paste: [[2311.10986]] EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge(http://arxiv.org/abs/2311.10986)
• Summary:

  Deep Learning (DL) models have been widely deployed on IoT devices with the help of advancements in DL algorithms and chips. However, the limited resources of edge devices make these on-device DL models hard to be generalizable to diverse environments and tasks. Although the recently emerged foundation models (FMs) show impressive generalization power, how to effectively leverage the rich knowledge of FMs on resource-limited edge devices is still not explored. In this paper, we propose EdgeFM, a novel edge-cloud cooperative system with open-set recognition capability. EdgeFM selectively uploads unlabeled data to query the FM on the cloud and customizes the specific knowledge and architectures for edge models. Meanwhile, EdgeFM conducts dynamic model switching at run-time taking into account both data uncertainty and dynamic network variations, which ensures the accuracy always close to the original FM. We implement EdgeFM using two FMs on two edge platforms. We evaluate EdgeFM on three public datasets and two self-collected datasets. Results show that EdgeFM can reduce the end-to-end latency up to 3.2x and achieve 34.3% accuracy increase compared with the baseline.

generative

Title: Partially Randomizing Transformer Weights for Dialogue Response Diversity. (arXiv:2311.10943v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10943
• Code URL: null
• Copy Paste: [[2311.10943]] Partially Randomizing Transformer Weights for Dialogue Response Diversity(http://arxiv.org/abs/2311.10943)
• Summary:

  Despite recent progress in generative open-domain dialogue, the issue of low response diversity persists. Prior works have addressed this issue via either novel objective functions, alternative learning approaches such as variational frameworks, or architectural extensions such as the Randomized Link (RL) Transformer. However, these approaches typically entail either additional difficulties during training/inference, or a significant increase in model size and complexity. Hence, we propose the \underline{Pa}rtially \underline{Ra}ndomized trans\underline{Former} (PaRaFormer), a simple extension of the transformer which involves freezing the weights of selected layers after random initialization. Experimental results reveal that the performance of the PaRaformer is comparable to that of the aforementioned approaches, despite not entailing any additional training difficulty or increase in model complexity.

Title: Journey of Hallucination-minimized Generative AI Solutions for Financial Decision Makers. (arXiv:2311.10961v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10961
• Code URL: null
• Copy Paste: [[2311.10961]] Journey of Hallucination-minimized Generative AI Solutions for Financial Decision Makers(http://arxiv.org/abs/2311.10961)
• Summary:

  Generative AI has significantly reduced the entry barrier to the domain of AI owing to the ease of use and core capabilities of automation, translation, and intelligent actions in our day to day lives. Currently, Large language models (LLMs) that power such chatbots are being utilized primarily for their automation capabilities for software monitoring, report generation etc. and for specific personalized question answering capabilities, on a limited scope and scale. One major limitation of the currently evolving family of LLMs is 'hallucinations', wherein inaccurate responses are reported as factual. Hallucinations are primarily caused by biased training data, ambiguous prompts and inaccurate LLM parameters, and they majorly occur while combining mathematical facts with language-based context. Thus, monitoring and controlling for hallucinations becomes necessary when designing solutions that are meant for decision makers. In this work we present the three major stages in the journey of designing hallucination-minimized LLM-based solutions that are specialized for the decision makers of the financial domain, namely: prototyping, scaling and LLM evolution using human feedback. These three stages and the novel data to answer generation modules presented in this work are necessary to ensure that the Generative AI chatbots, autonomous reports and alerts are reliable and high-quality to aid key decision-making processes.

Title: PACOL: Poisoning Attacks Against Continual Learners. (arXiv:2311.10919v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10919
• Code URL: null
• Copy Paste: [[2311.10919]] PACOL: Poisoning Attacks Against Continual Learners(http://arxiv.org/abs/2311.10919)
• Summary:

  Continual learning algorithms are typically exposed to untrusted sources that contain training data inserted by adversaries and bad actors. An adversary can insert a small number of poisoned samples, such as mislabeled samples from previously learned tasks, or intentional adversarial perturbed samples, into the training datasets, which can drastically reduce the model's performance. In this work, we demonstrate that continual learning systems can be manipulated by malicious misinformation and present a new category of data poisoning attacks specific for continual learners, which we refer to as {\em Poisoning Attacks Against Continual Learners} (PACOL). The effectiveness of labeling flipping attacks inspires PACOL; however, PACOL produces attack samples that do not change the sample's label and produce an attack that causes catastrophic forgetting. A comprehensive set of experiments shows the vulnerability of commonly used generative replay and regularization-based continual learning approaches against attack methods. We evaluate the ability of label-flipping and a new adversarial poison attack, namely PACOL proposed in this work, to force the continual learning system to forget the knowledge of a learned task(s). More specifically, we compared the performance degradation of continual learning systems trained on benchmark data streams with and without poisoning attacks. Moreover, we discuss the stealthiness of the attacks in which we test the success rate of data sanitization defense and other outlier detection-based defenses for filtering out adversarial samples.

Title: Compact and Intuitive Airfoil Parameterization Method through Physics-aware Variational Autoencoder. (arXiv:2311.10921v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10921
• Code URL: null
• Copy Paste: [[2311.10921]] Compact and Intuitive Airfoil Parameterization Method through Physics-aware Variational Autoencoder(http://arxiv.org/abs/2311.10921)
• Summary:

  Airfoil shape optimization plays a critical role in the design of high-performance aircraft. However, the high-dimensional nature of airfoil representation causes the challenging problem known as the "curse of dimensionality". To overcome this problem, numerous airfoil parameterization methods have been developed, which can be broadly classified as polynomial-based and data-driven approaches. Each of these methods has desirable characteristics such as flexibility, parsimony, feasibility, and intuitiveness, but a single approach that encompasses all of these attributes has yet to be found. For example, polynomial-based methods struggle to balance parsimony and flexibility, while data-driven methods lack in feasibility and intuitiveness. In recent years, generative models, such as generative adversarial networks and variational autoencoders, have shown promising potential in airfoil parameterization. However, these models still face challenges related to intuitiveness due to their black-box nature. To address this issue, we developed a novel airfoil parameterization method using physics-aware variational autoencoder. The proposed method not only explicitly separates the generation of thickness and camber distributions to produce smooth and non-intersecting airfoils, thereby improving feasibility, but it also directly aligns its latent dimensions with geometric features of the airfoil, significantly enhancing intuitiveness. Finally, extensive comparative studies were performed to demonstrate the effectiveness of our approach.

Title: Wasserstein Convergence Guarantees for a General Class of Score-Based Generative Models. (arXiv:2311.11003v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.11003
• Code URL: null
• Copy Paste: [[2311.11003]] Wasserstein Convergence Guarantees for a General Class of Score-Based Generative Models(http://arxiv.org/abs/2311.11003)
• Summary:

  Score-based generative models (SGMs) is a recent class of deep generative models with state-of-the-art performance in many applications. In this paper, we establish convergence guarantees for a general class of SGMs in 2-Wasserstein distance, assuming accurate score estimates and smooth log-concave data distribution. We specialize our result to several concrete SGMs with specific choices of forward processes modelled by stochastic differential equations, and obtain an upper bound on the iteration complexity for each model, which demonstrates the impacts of different choices of the forward processes. We also provide a lower bound when the data distribution is Gaussian. Numerically, we experiment SGMs with different forward processes, some of which are newly proposed in this paper, for unconditional image generation on CIFAR-10. We find that the experimental results are in good agreement with our theoretical predictions on the iteration complexity, and the models with our newly proposed forward processes can outperform existing models.

anomaly

in-context

Title: Enhancing Machine Translation through Advanced In-Context Learning: A Methodological Strategy for GPT-4 Improvement. (arXiv:2311.10765v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10765
• Code URL: null
• Copy Paste: [[2311.10765]] Enhancing Machine Translation through Advanced In-Context Learning: A Methodological Strategy for GPT-4 Improvement(http://arxiv.org/abs/2311.10765)
• Summary:

  The challenge of improving translation accuracy in GPT-4 is being addressed by harnessing a method known as in-context learning. This paper introduces a strategic approach to utilize in-context learning specifically for machine translation, aiming to significantly boost accuracy. The crux of this method lies in the judicious selection of demonstrations that are most effective for in-context learning. By selecting these examples carefully, GPT-4 can utilize them to achieve remarkably accurate machine translations, eliminating the need for task-specific fine-tuning. This technique is anchored in the semantic similarities between the user's prompt and the chosen dataset. Sentences from this dataset, carefully picked for their relevance and clarity, serve as potent demonstrations for in-context learning. This approach not only enhances translation accuracy but also enriches the understanding of nuanced linguistic structures. It represents a significant step forward in machine learning, leveraging the inherent capabilities of GPT-4 to provide translations that are not only accurate but also contextually rich and linguistically sophisticated. This method demonstrates the potential of in-context learning in overcoming language barriers, opening new avenues for cross-cultural communication and global collaboration.