2024-02-09

Title: Quantifying Similarity: Text-Mining Approaches to Evaluate ChatGPT and Google Bard Content in Relation to BioMedical Literature

• Authors: Jakub Klimczak, Ahmed Abdeen Hamed
• Subjects: cs.CL, cs.DL, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.05116
• Pdf URL: https://arxiv.org/pdf/2402.05116
• Copy Paste: [[2402.05116]] Quantifying Similarity: Text-Mining Approaches to Evaluate ChatGPT and Google Bard Content in Relation to BioMedical Literature(https://arxiv.org/abs/2402.05116)
• Keywords: generative
• Abstract: Background: The emergence of generative AI tools, empowered by Large Language Models (LLMs), has shown powerful capabilities in generating content. To date, the assessment of the usefulness of such content, generated by what is known as prompt engineering, has become an interesting research question. Objectives Using the mean of prompt engineering, we assess the similarity and closeness of such contents to real literature produced by scientists. Methods In this exploratory analysis, (1) we prompt-engineer ChatGPT and Google Bard to generate clinical content to be compared with literature counterparts, (2) we assess the similarities of the contents generated by comparing them with counterparts from biomedical literature. Our approach is to use text-mining approaches to compare documents and associated bigrams and to use network analysis to assess the terms' centrality. Results The experiments demonstrated that ChatGPT outperformed Google Bard in cosine document similarity (38% to 34%), Jaccard document similarity (23% to 19%), TF-IDF bigram similarity (47% to 41%), and term network centrality (degree and closeness). We also found new links that emerged in ChatGPT bigram networks that did not exist in literature bigram networks. Conclusions: The obtained similarity results show that ChatGPT outperformed Google Bard in document similarity, bigrams, and degree and closeness centrality. We also observed that ChatGPT offers linkage to terms that are connected in the literature. Such connections could inspire asking interesting questions and generate new hypotheses.

Title: Illuminate: A novel approach for depression detection with explainable analysis and proactive therapy using prompt engineering

• Authors: Aryan Agrawal
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05127
• Pdf URL: https://arxiv.org/pdf/2402.05127
• Copy Paste: [[2402.05127]] Illuminate: A novel approach for depression detection with explainable analysis and proactive therapy using prompt engineering(https://arxiv.org/abs/2402.05127)
• Keywords: generative
• Abstract: This paper introduces a novel paradigm for depression detection and treatment using advanced Large Language Models (LLMs): Generative Pre-trained Transformer 4 (GPT-4), Llama 2 chat, and Gemini. These LLMs are fine-tuned with specialized prompts to diagnose, explain, and suggest therapeutic interventions for depression. A unique few-shot prompting method enhances the models' ability to analyze and explain depressive symptoms based on the DSM-5 criteria. In the interaction phase, the models engage in empathetic dialogue management, drawing from resources like PsychDB and a Cognitive Behavioral Therapy (CBT) Guide, fostering supportive interactions with individuals experiencing major depressive disorders. Additionally, the research introduces the Illuminate Database, enriched with various CBT modules, aiding in personalized therapy recommendations. The study evaluates LLM performance using metrics such as F1 scores, Precision, Recall, Cosine similarity, and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) across different test sets, demonstrating their effectiveness. This comprehensive approach blends cutting-edge AI with established psychological methods, offering new possibilities in mental health care and showcasing the potential of LLMs in revolutionizing depression diagnosis and treatment strategies.

Title: LB-KBQA: Large-language-model and BERT based Knowledge-Based Question and Answering System

• Authors: Yan Zhao, Zhongyun Li, Jiaxing Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.05130
• Pdf URL: https://arxiv.org/pdf/2402.05130
• Copy Paste: [[2402.05130]] LB-KBQA: Large-language-model and BERT based Knowledge-Based Question and Answering System(https://arxiv.org/abs/2402.05130)
• Keywords: generative
• Abstract: Generative Artificial Intelligence (AI), because of its emergent abilities, has empowered various fields, one typical of which is large language models (LLMs). One of the typical application fields of Generative AI is large language models (LLMs), and the natural language understanding capability of LLM is dramatically improved when compared with conventional AI-based methods. The natural language understanding capability has always been a barrier to the intent recognition performance of the Knowledge-Based-Question-and-Answer (KBQA) system, which arises from linguistic diversity and the newly appeared intent. Conventional AI-based methods for intent recognition can be divided into semantic parsing-based and model-based approaches. However, both of the methods suffer from limited resources in intent recognition. To address this issue, we propose a novel KBQA system based on a Large Language Model(LLM) and BERT (LB-KBQA). With the help of generative AI, our proposed method could detect newly appeared intent and acquire new knowledge. In experiments on financial domain question answering, our model has demonstrated superior effectiveness.

Title: $λ$-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space

• Authors: Maitreya Patel, Sangmin Jung, Chitta Baral, Yezhou Yang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.05195
• Pdf URL: https://arxiv.org/pdf/2402.05195
• Copy Paste: [[2402.05195]] $λ$-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space(https://arxiv.org/abs/2402.05195)
• Keywords: diffusion, generative
• Abstract: Despite the recent advances in personalized text-to-image (P-T2I) generative models, subject-driven T2I remains challenging. The primary bottlenecks include 1) Intensive training resource requirements, 2) Hyper-parameter sensitivity leading to inconsistent outputs, and 3) Balancing the intricacies of novel visual concept and composition alignment. We start by re-iterating the core philosophy of T2I diffusion models to address the above limitations. Predominantly, contemporary subject-driven T2I approaches hinge on Latent Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention layers. While LDMs offer distinct advantages, P-T2I methods' reliance on the latent space of these diffusion models significantly escalates resource demands, leading to inconsistent results and necessitating numerous iterations for a single desired image. Recently, ECLIPSE has demonstrated a more resource-efficient pathway for training UnCLIP-based T2I models, circumventing the need for diffusion text-to-image priors. Building on this, we introduce $\lambda$-ECLIPSE. Our method illustrates that effective P-T2I does not necessarily depend on the latent space of diffusion models. $\lambda$-ECLIPSE achieves single, multi-subject, and edge-guided T2I personalization with just 34M parameters and is trained on a mere 74 GPU hours using 1.6M image-text interleaved data. Through extensive experiments, we also establish that $\lambda$-ECLIPSE surpasses existing baselines in composition alignment while preserving concept alignment performance, even with significantly lower resource utilization.

Title: QGFN: Controllable Greediness with Action Values

• Authors: Elaine Lau, Stephen Zhewen Lu, Ling Pan, Doina Precup, Emmanuel Bengio
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05234
• Pdf URL: https://arxiv.org/pdf/2402.05234
• Copy Paste: [[2402.05234]] QGFN: Controllable Greediness with Action Values(https://arxiv.org/abs/2402.05234)
• Keywords: generative
• Abstract: Generative Flow Networks (GFlowNets; GFNs) are a family of reward/energy-based generative methods for combinatorial objects, capable of generating diverse and high-utility samples. However, biasing GFNs towards producing high-utility samples is non-trivial. In this work, we leverage connections between GFNs and reinforcement learning (RL) and propose to combine the GFN policy with an action-value estimate, $Q$, to create greedier sampling policies which can be controlled by a mixing parameter. We show that several variants of the proposed method, QGFN, are able to improve on the number of high-reward samples generated in a variety of tasks without sacrificing diversity.

Title: SPAD : Spatially Aware Multiview Diffusers

• Authors: Yash Kant, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, Igor Gilitschenski, Aliaksandr Siarohin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05235
• Pdf URL: https://arxiv.org/pdf/2402.05235
• Copy Paste: [[2402.05235]] SPAD : Spatially Aware Multiview Diffusers(https://arxiv.org/abs/2402.05235)
• Keywords: diffusion
• Abstract: We present SPAD, a novel approach for creating consistent multi-view images from text prompts or single images. To enable multi-view generation, we repurpose a pretrained 2D diffusion model by extending its self-attention layers with cross-view interactions, and fine-tune it on a high quality subset of Objaverse. We find that a naive extension of the self-attention proposed in prior work (e.g. MVDream) leads to content copying between views. Therefore, we explicitly constrain the cross-view attention based on epipolar geometry. To further enhance 3D consistency, we utilize Plucker coordinates derived from camera rays and inject them as positional encoding. This enables SPAD to reason over spatial proximity in 3D well. In contrast to recent works that can only generate views at fixed azimuth and elevation, SPAD offers full camera control and achieves state-of-the-art results in novel view synthesis on unseen objects from the Objaverse and Google Scanned Objects datasets. Finally, we demonstrate that text-to-3D generation using SPAD prevents the multi-face Janus issue. See more details at our webpage: https://yashkant.github.io/spad

Title: Investigating Generalization Behaviours of Generative Flow Networks

• Authors: Lazar Atanackovic, Emmanuel Bengio
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05309
• Pdf URL: https://arxiv.org/pdf/2402.05309
• Copy Paste: [[2402.05309]] Investigating Generalization Behaviours of Generative Flow Networks(https://arxiv.org/abs/2402.05309)
• Keywords: generative
• Abstract: Generative Flow Networks (GFlowNets, GFNs) are a generative framework for learning unnormalized probability mass functions over discrete spaces. Since their inception, GFlowNets have proven to be useful for learning generative models in applications where the majority of the discrete space is unvisited during training. This has inspired some to hypothesize that GFlowNets, when paired with deep neural networks (DNNs), have favourable generalization properties. In this work, we empirically verify some of the hypothesized mechanisms of generalization of GFlowNets. In particular, we find that the functions that GFlowNets learn to approximate have an implicit underlying structure which facilitate generalization. We also find that GFlowNets are sensitive to being trained offline and off-policy; however, the reward implicitly learned by GFlowNets is robust to changes in the training distribution.

Title: Descanning: From Scanned to the Original Images with a Color Correction Diffusion Model

• Authors: Junghun Cha, Ali Haider, Seoyun Yang, Hoeyeong Jin, Subin Yang, A. F. M. Shahab Uddin, Jaehyoung Kim, Soo Ye Kim, Sung-Ho Bae
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.05350
• Pdf URL: https://arxiv.org/pdf/2402.05350
• Copy Paste: [[2402.05350]] Descanning: From Scanned to the Original Images with a Color Correction Diffusion Model(https://arxiv.org/abs/2402.05350)
• Keywords: diffusion
• Abstract: A significant volume of analog information, i.e., documents and images, have been digitized in the form of scanned copies for storing, sharing, and/or analyzing in the digital world. However, the quality of such contents is severely degraded by various distortions caused by printing, storing, and scanning processes in the physical world. Although restoring high-quality content from scanned copies has become an indispensable task for many products, it has not been systematically explored, and to the best of our knowledge, no public datasets are available. In this paper, we define this problem as Descanning and introduce a new high-quality and large-scale dataset named DESCAN-18K. It contains 18K pairs of original and scanned images collected in the wild containing multiple complex degradations. In order to eliminate such complex degradations, we propose a new image restoration model called DescanDiffusion consisting of a color encoder that corrects the global color degradation and a conditional denoising diffusion probabilistic model (DDPM) that removes local degradations. To further improve the generalization ability of DescanDiffusion, we also design a synthetic data generation scheme by reproducing prominent degradations in scanned images. We demonstrate that our DescanDiffusion outperforms other baselines including commercial restoration products, objectively and subjectively, via comprehensive experiments and analyses.

Title: Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models

• Authors: Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, Jian Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05375
• Pdf URL: https://arxiv.org/pdf/2402.05375
• Copy Paste: [[2402.05375]] Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models(https://arxiv.org/abs/2402.05375)
• Keywords: diffusion
• Abstract: The success of recent text-to-image diffusion models is largely due to their capacity to be guided by a complex text prompt, which enables users to precisely describe the desired content. However, these models struggle to effectively suppress the generation of undesired content, which is explicitly requested to be omitted from the generated image in the prompt. In this paper, we analyze how to manipulate the text embeddings and remove unwanted content from them. We introduce two contributions, which we refer to as $\textit{soft-weighted regularization}$ and $\textit{inference-time text embedding optimization}$. The first regularizes the text embedding matrix and effectively suppresses the undesired content. The second method aims to further suppress the unwanted content generation of the prompt, and encourages the generation of desired content. We evaluate our method quantitatively and qualitatively on extensive experiments, validating its effectiveness. Furthermore, our method is generalizability to both the pixel-space diffusion models (i.e. DeepFloyd-IF) and the latent-space diffusion models (i.e. Stable Diffusion).

Title: Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts

• Authors: Zhili Liu, Kai Chen, Jianhua Han, Lanqing Hong, Hang Xu, Zhenguo Li, James T. Kwok
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05382
• Pdf URL: https://arxiv.org/pdf/2402.05382
• Copy Paste: [[2402.05382]] Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts(https://arxiv.org/abs/2402.05382)
• Keywords: self-supervised
• Abstract: Masked Autoencoder~(MAE) is a prevailing self-supervised learning method that achieves promising results in model pre-training. However, when the various downstream tasks have data distributions different from the pre-training data, the semantically irrelevant pre-training information might result in negative transfer, impeding MAE's scalability. To address this issue, we propose a novel MAE-based pre-training paradigm, Mixture of Cluster-conditional Experts (MoCE), which can be trained once but provides customized pre-training models for diverse downstream tasks. Different from the mixture of experts (MoE), our MoCE trains each expert only with semantically relevant images by using cluster-conditional gates. Thus, each downstream task can be allocated to its customized model pre-trained with data most similar to the downstream data. Experiments on a collection of 11 downstream tasks show that MoCE outperforms the vanilla MAE by 2.45\% on average. It also obtains new state-of-the-art self-supervised learning results on detection and segmentation.

Title: In-Context Principle Learning from Mistakes

• Authors: Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, Uri Alon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.05403
• Pdf URL: https://arxiv.org/pdf/2402.05403
• Copy Paste: [[2402.05403]] In-Context Principle Learning from Mistakes(https://arxiv.org/abs/2402.05403)
• Keywords: in-context
• Abstract: In-context learning (ICL, also known as few-shot prompting) has been the standard method of adapting LLMs to downstream tasks, by learning from a few input-output examples. Nonetheless, all ICL-based approaches only learn from correct input-output pairs. In this paper, we revisit this paradigm, by learning more from the few given input-output examples. We introduce Learning Principles (LEAP): First, we intentionally induce the model to make mistakes on these few examples; then we reflect on these mistakes, and learn explicit task-specific "principles" from them, which help solve similar problems and avoid common mistakes; finally, we prompt the model to answer unseen test questions using the original few-shot examples and these learned general principles. We evaluate LEAP on a wide range of benchmarks, including multi-hop question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning, and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the strongest available LLMs such as GPT-3.5-turbo, GPT-4, GPT-4 turbo and Claude-2.1. For example, LEAP improves over the standard few-shot prompting using GPT-4 by 7.5% in DROP, and by 3.3% in HotpotQA. Importantly, LEAP does not require any more input or examples than the standard few-shot prompting settings.

Title: MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis

• Authors: Dewei Zhou, You Li, Fan Ma, Zongxin Yang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05408
• Pdf URL: https://arxiv.org/pdf/2402.05408
• Copy Paste: [[2402.05408]] MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis(https://arxiv.org/abs/2402.05408)
• Keywords: diffusion
• Abstract: We present a Multi-Instance Generation (MIG) task, simultaneously generating multiple instances with diverse controls in one image. Given a set of predefined coordinates and their corresponding descriptions, the task is to ensure that generated instances are accurately at the designated locations and that all instances' attributes adhere to their corresponding description. This broadens the scope of current research on Single-instance generation, elevating it to a more versatile and practical dimension. Inspired by the idea of divide and conquer, we introduce an innovative approach named Multi-Instance Generation Controller (MIGC) to address the challenges of the MIG task. Initially, we break down the MIG task into several subtasks, each involving the shading of a single instance. To ensure precise shading for each instance, we introduce an instance enhancement attention mechanism. Lastly, we aggregate all the shaded instances to provide the necessary information for accurately generating multiple instances in stable diffusion (SD). To evaluate how well generation models perform on the MIG task, we provide a COCO-MIG benchmark along with an evaluation pipeline. Extensive experiments were conducted on the proposed COCO-MIG benchmark, as well as on various commonly used benchmarks. The evaluation results illustrate the exceptional control capabilities of our model in terms of quantity, position, attribute, and interaction.

Title: DiffTOP: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning

• Authors: Weikang Wan, Yufei Wang, Zackory Erickson, David Held
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.05421
• Pdf URL: https://arxiv.org/pdf/2402.05421
• Copy Paste: [[2402.05421]] DiffTOP: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning(https://arxiv.org/abs/2402.05421)
• Keywords: diffusion
• Abstract: This paper introduces DiffTOP, which utilizes Differentiable Trajectory OPtimization as the policy representation to generate actions for deep reinforcement and imitation learning. Trajectory optimization is a powerful and widely used algorithm in control, parameterized by a cost and a dynamics function. The key to our approach is to leverage the recent progress in differentiable trajectory optimization, which enables computing the gradients of the loss with respect to the parameters of trajectory optimization. As a result, the cost and dynamics functions of trajectory optimization can be learned end-to-end. DiffTOP addresses the ``objective mismatch'' issue of prior model-based RL algorithms, as the dynamics model in DiffTOP is learned to directly maximize task performance by differentiating the policy gradient loss through the trajectory optimization process. We further benchmark DiffTOP for imitation learning on standard robotic manipulation task suites with high-dimensional sensory observations and compare our method to feed-forward policy classes as well as Energy-Based Models (EBM) and Diffusion. Across 15 model-based RL tasks and 13 imitation learning tasks with high-dimensional image and point cloud inputs, DiffTOP outperforms prior state-of-the-art methods in both domains.

Title: Scalable Wasserstein Gradient Flow for Generative Modeling through Unbalanced Optimal Transport

• Authors: Jaemoo Choi, Jaewoong Choi, Myungjoo Kang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05443
• Pdf URL: https://arxiv.org/pdf/2402.05443
• Copy Paste: [[2402.05443]] Scalable Wasserstein Gradient Flow for Generative Modeling through Unbalanced Optimal Transport(https://arxiv.org/abs/2402.05443)
• Keywords: generative
• Abstract: Wasserstein Gradient Flow (WGF) describes the gradient dynamics of probability density within the Wasserstein space. WGF provides a promising approach for conducting optimization over the probability distributions. Numerically approximating the continuous WGF requires the time discretization method. The most well-known method for this is the JKO scheme. In this regard, previous WGF models employ the JKO scheme and parametrize transport map for each JKO step. However, this approach results in quadratic training complexity $O(K^2)$ with the number of JKO step $K$. This severely limits the scalability of WGF models. In this paper, we introduce a scalable WGF-based generative model, called Semi-dual JKO (S-JKO). Our model is based on the semi-dual form of the JKO step, derived from the equivalence between the JKO step and the Unbalanced Optimal Transport. Our approach reduces the training complexity to $O(K)$. We demonstrate that our model significantly outperforms existing WGF-based generative models, achieving FID scores of 2.62 on CIFAR-10 and 6.19 on CelebA-HQ-256, which are comparable to state-of-the-art image generative models.

Title: It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

• Authors: Chen Chen, Ruizhe Li, Yuchen Hu, Sabato Marco Siniscalchi, Pin-Yu Chen, Ensiong Chng, Chao-Han Huck Yang
• Subjects: cs.CL, cs.AI, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.05457
• Pdf URL: https://arxiv.org/pdf/2402.05457
• Copy Paste: [[2402.05457]] It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition(https://arxiv.org/abs/2402.05457)
• Keywords: generative
• Abstract: Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output. Specifically, an LLM is utilized to carry out a direct mapping from the N-best hypotheses list generated by an ASR system to the predicted output transcription. However, despite its effectiveness, GER introduces extra data uncertainty since the LLM is trained without taking into account acoustic information available in the speech signal. In this work, we aim to overcome such a limitation by infusing acoustic information before generating the predicted transcription through a novel late fusion solution termed Uncertainty-Aware Dynamic Fusion (UADF). UADF is a multimodal fusion approach implemented into an auto-regressive decoding process and works in two stages: (i) It first analyzes and calibrates the token-level LLM decision, and (ii) it then dynamically assimilates the information from the acoustic modality. Experimental evidence collected from various ASR tasks shows that UADF surpasses existing fusion mechanisms in several ways. It yields significant improvements in word error rate (WER) while mitigating data uncertainty issues in LLM and addressing the poor generalization relied with sole modality during fusion. We also demonstrate that UADF seamlessly adapts to audio-visual speech recognition.

Title: Implicit Diffusion: Efficient Optimization through Stochastic Sampling

• Authors: Pierre Marion, Anna Korba, Peter Bartlett, Mathieu Blondel, Valentin De Bortoli, Arnaud Doucet, Felipe Llinares-López, Courtney Paquette, Quentin Berthet
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05468
• Pdf URL: https://arxiv.org/pdf/2402.05468
• Copy Paste: [[2402.05468]] Implicit Diffusion: Efficient Optimization through Stochastic Sampling(https://arxiv.org/abs/2402.05468)
• Keywords: diffusion
• Abstract: We present a new algorithm to optimize distributions defined implicitly by parameterized stochastic diffusions. Doing so allows us to modify the outcome distribution of sampling processes by optimizing over their parameters. We introduce a general framework for first-order optimization of these processes, that performs jointly, in a single loop, optimization and sampling steps. This approach is inspired by recent advances in bilevel optimization and automatic implicit differentiation, leveraging the point of view of sampling as optimization over the space of probability distributions. We provide theoretical guarantees on the performance of our method, as well as experimental results demonstrating its effectiveness in real-world settings.

Title: Determining the severity of Parkinson's disease in patients using a multi task neural network

• Authors: María Teresa García-Ordás, José Alberto Benítez-Andrades, Jose Aveleira-Mata, José-Manuel Alija-Pérez, Carmen Benavides
• Subjects: cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.05491
• Pdf URL: https://arxiv.org/pdf/2402.05491
• Copy Paste: [[2402.05491]] Determining the severity of Parkinson's disease in patients using a multi task neural network(https://arxiv.org/abs/2402.05491)
• Keywords: generative
• Abstract: Parkinson's disease is easy to diagnose when it is advanced, but it is very difficult to diagnose in its early stages. Early diagnosis is essential to be able to treat the symptoms. It impacts on daily activities and reduces the quality of life of both the patients and their families and it is also the second most prevalent neurodegenerative disorder after Alzheimer in people over the age of 60. Most current studies on the prediction of Parkinson's severity are carried out in advanced stages of the disease. In this work, the study analyzes a set of variables that can be easily extracted from voice analysis, making it a very non-intrusive technique. In this paper, a method based on different deep learning techniques is proposed with two purposes. On the one hand, to find out if a person has severe or non-severe Parkinson's disease, and on the other hand, to determine by means of regression techniques the degree of evolution of the disease in a given patient. The UPDRS (Unified Parkinson's Disease Rating Scale) has been used by taking into account both the motor and total labels, and the best results have been obtained using a mixed multi-layer perceptron (MLP) that classifies and regresses at the same time and the most important features of the data obtained are taken as input, using an autoencoder. A success rate of 99.15% has been achieved in the problem of predicting whether a person suffers from severe Parkinson's disease or non-severe Parkinson's disease. In the degree of disease involvement prediction problem case, a MSE (Mean Squared Error) of 0.15 has been obtained. Using a full deep learning pipeline for data preprocessing and classification has proven to be very promising in the field Parkinson's outperforming the state-of-the-art proposals.

Title: NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning

• Authors: Yufeng Zhao, Yoshihiro Sakai, Naoya Inoue
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.05515
• Pdf URL: https://arxiv.org/pdf/2402.05515
• Copy Paste: [[2402.05515]] NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning(https://arxiv.org/abs/2402.05515)
• Keywords: in-context
• Abstract: In-Context Learning (ICL) is suffering from unsatisfactory performance and under-calibration due to high prior bias and unfaithful confidence. Some previous works fine-tuned language models for better ICL performance with enormous datasets and computing costs. In this paper, we propose NoisyICL, simply perturbing the model parameters by random noises to strive for better performance and calibration. Our experiments on 2 models and 12 downstream datasets show that NoisyICL can help ICL produce more accurate predictions. Our further analysis indicates that NoisyICL enables the model to provide more fair predictions, and also with less unfaithful confidence. Therefore, we believe that NoisyICL is an effective calibration of ICL. Our experimental code is uploaded to Github.

Title: Buffer Overflow in Mixture of Experts

• Authors: Jamie Hayes, Ilia Shumailov, Itay Yona
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05526
• Pdf URL: https://arxiv.org/pdf/2402.05526
• Copy Paste: [[2402.05526]] Buffer Overflow in Mixture of Experts(https://arxiv.org/abs/2402.05526)
• Keywords: foundation model
• Abstract: Mixture of Experts (MoE) has become a key ingredient for scaling large foundation models while keeping inference costs steady. We show that expert routing strategies that have cross-batch dependencies are vulnerable to attacks. Malicious queries can be sent to a model and can affect a model's output on other benign queries if they are grouped in the same batch. We demonstrate this via a proof-of-concept attack in a toy experimental setting.

Title: Asynchronous Diffusion Learning with Agent Subsampling and Local Updates

• Authors: Elsa Rizk, Kun Yuan, Ali H. Sayed
• Subjects: cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2402.05529
• Pdf URL: https://arxiv.org/pdf/2402.05529
• Copy Paste: [[2402.05529]] Asynchronous Diffusion Learning with Agent Subsampling and Local Updates(https://arxiv.org/abs/2402.05529)
• Keywords: diffusion
• Abstract: In this work, we examine a network of agents operating asynchronously, aiming to discover an ideal global model that suits individual local datasets. Our assumption is that each agent independently chooses when to participate throughout the algorithm and the specific subset of its neighbourhood with which it will cooperate at any given moment. When an agent chooses to take part, it undergoes multiple local updates before conveying its outcomes to the sub-sampled neighbourhood. Under this setup, we prove that the resulting asynchronous diffusion strategy is stable in the mean-square error sense and provide performance guarantees specifically for the federated learning setting. We illustrate the findings with numerical simulations.

Title: Scalable Diffusion Models with State Space Backbone

• Authors: Zhengcong Fei, Mingyuan Fan, Changqian Yu, Junshi Huang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.05608
• Pdf URL: https://arxiv.org/pdf/2402.05608
• Copy Paste: [[2402.05608]] Scalable Diffusion Models with State Space Backbone(https://arxiv.org/abs/2402.05608)
• Keywords: diffusion
• Abstract: This paper presents a new exploration into a category of diffusion models built upon state space architecture. We endeavor to train diffusion models for image data, wherein the traditional U-Net backbone is supplanted by a state space backbone, functioning on raw patches or latent space. Given its notable efficacy in accommodating long-range dependencies, Diffusion State Space Models (DiS) are distinguished by treating all inputs including time, condition, and noisy image patches as tokens. Our assessment of DiS encompasses both unconditional and class-conditional image generation scenarios, revealing that DiS exhibits comparable, if not superior, performance to CNN-based or Transformer-based U-Net architectures of commensurate size. Furthermore, we analyze the scalability of DiS, gauged by the forward pass complexity quantified in Gflops. DiS models with higher Gflops, achieved through augmentation of depth/width or augmentation of input tokens, consistently demonstrate lower FID. In addition to demonstrating commendable scalability characteristics, DiS-H/2 models in latent space achieve performance levels akin to prior diffusion models on class-conditional ImageNet benchmarks at the resolution of 256$\times$256 and 512$\times$512, while significantly reducing the computational burden. The code and models are available at: https://github.com/feizc/DiS.

Title: Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks

• Authors: Ben Fauber
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05616
• Pdf URL: https://arxiv.org/pdf/2402.05616
• Copy Paste: [[2402.05616]] Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks(https://arxiv.org/abs/2402.05616)
• Keywords: generative
• Abstract: We propose that small pretrained foundational generative language models with millions of parameters can be utilized as a general learning framework for sequence-based tasks. Our proposal overcomes the computational resource, skill set, and timeline challenges associated with training neural networks and language models from scratch. Further, our approach focuses on creating small and highly specialized models that can accurately execute a challenging task of which the base model is incapable of performing. We demonstrate that 125M, 350M, and 1.3B parameter pretrained foundational language models can be instruction fine-tuned with 10,000-to-1,000,000 instruction examples to achieve near state-of-the-art results on challenging cheminformatics tasks. We also demonstrate the role of successive language model fine-tuning epochs on improved outcomes, as well as the importance of both data formatting and pretrained foundational language model selection for instruction fine-tuning success.

Title: Real-time Holistic Robot Pose Estimation with Unknown States

• Authors: Shikun Ban, Juling Fan, Wentao Zhu, Xiaoxuan Ma, Yu Qiao, Yizhou Wang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.05655
• Pdf URL: https://arxiv.org/pdf/2402.05655
• Copy Paste: [[2402.05655]] Real-time Holistic Robot Pose Estimation with Unknown States(https://arxiv.org/abs/2402.05655)
• Keywords: self-supervised
• Abstract: Estimating robot pose from RGB images is a crucial problem in computer vision and robotics. While previous methods have achieved promising performance, most of them presume full knowledge of robot internal states, e.g. ground-truth robot joint angles, which are not always available in real-world scenarios. On the other hand, existing approaches that estimate robot pose without joint state priors suffer from heavy computation burdens and thus cannot support real-time applications. This work addresses the urgent need for efficient robot pose estimation with unknown states. We propose an end-to-end pipeline for real-time, holistic robot pose estimation from a single RGB image, even in the absence of known robot states. Our method decomposes the problem into estimating camera-to-robot rotation, robot state parameters, keypoint locations, and root depth. We further design a corresponding neural network module for each task. This approach allows for learning multi-facet representations and facilitates sim-to-real transfer through self-supervised learning. Notably, our method achieves inference with a single feedforward, eliminating the need for costly test-time iterative optimization. As a result, it delivers a 12-time speed boost with state-of-the-art accuracy, enabling real-time holistic robot pose estimation for the first time. Code is available at https://oliverbansk.github.io/Holistic-Robot-Pose/.

Title: DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer

• Authors: Zhiyuan Ma, Xiangyu Zhu, Guojun Qi, Chen Qian, Zhaoxiang Zhang, Zhen Lei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.05712
• Pdf URL: https://arxiv.org/pdf/2402.05712
• Copy Paste: [[2402.05712]] DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer(https://arxiv.org/abs/2402.05712)
• Keywords: diffusion
• Abstract: Speech-driven 3D facial animation is important for many multimedia applications. Recent work has shown promise in using either Diffusion models or Transformer architectures for this task. However, their mere aggregation does not lead to improved performance. We suspect this is due to a shortage of paired audio-4D data, which is crucial for the Transformer to effectively perform as a denoiser within the Diffusion framework. To tackle this issue, we present DiffSpeaker, a Transformer-based network equipped with novel biased conditional attention modules. These modules serve as substitutes for the traditional self/cross-attention in standard Transformers, incorporating thoughtfully designed biases that steer the attention mechanisms to concentrate on both the relevant task-specific and diffusion-related conditions. We also explore the trade-off between accurate lip synchronization and non-verbal facial expressions within the Diffusion paradigm. Experiments show our model not only achieves state-of-the-art performance on existing benchmarks, but also fast inference speed owing to its ability to generate facial motions in parallel.

Title: In-Context Learning Can Re-learn Forbidden Tasks

• Authors: Sophie Xhonneux, David Dobre, Jian Tang, Gauthier Gidel, Dhanya Sridhar
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.05723
• Pdf URL: https://arxiv.org/pdf/2402.05723
• Copy Paste: [[2402.05723]] In-Context Learning Can Re-learn Forbidden Tasks(https://arxiv.org/abs/2402.05723)
• Keywords: in-context
• Abstract: Despite significant investment into safety training, large language models (LLMs) deployed in the real world still suffer from numerous vulnerabilities. One perspective on LLM safety training is that it algorithmically forbids the model from answering toxic or harmful queries. To assess the effectiveness of safety training, in this work, we study forbidden tasks, i.e., tasks the model is designed to refuse to answer. Specifically, we investigate whether in-context learning (ICL) can be used to re-learn forbidden tasks despite the explicit fine-tuning of the model to refuse them. We first examine a toy example of refusing sentiment classification to demonstrate the problem. Then, we use ICL on a model fine-tuned to refuse to summarise made-up news articles. Finally, we investigate whether ICL can undo safety training, which could represent a major security risk. For the safety task, we look at Vicuna-7B, Starling-7B, and Llama2-7B. We show that the attack works out-of-the-box on Starling-7B and Vicuna-7B but fails on Llama2-7B. Finally, we propose an ICL attack that uses the chat template tokens like a prompt injection attack to achieve a better attack success rate on Vicuna-7B and Starling-7B. Trigger Warning: the appendix contains LLM-generated text with violence, suicide, and misinformation.

Title: CTGAN: Semantic-guided Conditional Texture Generator for 3D Shapes

• Authors: Yi-Ting Pan, Chai-Rong Lee, Shu-Ho Fan, Jheng-Wei Su, Jia-Bin Huang, Yung-Yu Chuang, Hung-Kuo Chu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05728
• Pdf URL: https://arxiv.org/pdf/2402.05728
• Copy Paste: [[2402.05728]] CTGAN: Semantic-guided Conditional Texture Generator for 3D Shapes(https://arxiv.org/abs/2402.05728)
• Keywords: generative
• Abstract: The entertainment industry relies on 3D visual content to create immersive experiences, but traditional methods for creating textured 3D models can be time-consuming and subjective. Generative networks such as StyleGAN have advanced image synthesis, but generating 3D objects with high-fidelity textures is still not well explored, and existing methods have limitations. We propose the Semantic-guided Conditional Texture Generator (CTGAN), producing high-quality textures for 3D shapes that are consistent with the viewing angle while respecting shape semantics. CTGAN utilizes the disentangled nature of StyleGAN to finely manipulate the input latent codes, enabling explicit control over both the style and structure of the generated textures. A coarse-to-fine encoder architecture is introduced to enhance control over the structure of the resulting textures via input segmentation. Experimental results show that CTGAN outperforms existing methods on multiple quality metrics and achieves state-of-the-art performance on texture generation in both conditional and unconditional settings.

Title: Stable Autonomous Flow Matching

• Authors: Christopher Iliffe Sprague, Arne Elofsson, Hossein Azizpour
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.05774
• Pdf URL: https://arxiv.org/pdf/2402.05774
• Copy Paste: [[2402.05774]] Stable Autonomous Flow Matching(https://arxiv.org/abs/2402.05774)
• Keywords: generative
• Abstract: In contexts where data samples represent a physically stable state, it is often assumed that the data points represent the local minima of an energy landscape. In control theory, it is well-known that energy can serve as an effective Lyapunov function. Despite this, connections between control theory and generative models in the literature are sparse, even though there are several machine learning applications with physically stable data points. In this paper, we focus on such data and a recent class of deep generative models called flow matching. We apply tools of stochastic stability for time-independent systems to flow matching models. In doing so, we characterize the space of flow matching models that are amenable to this treatment, as well as draw connections to other control theory principles. We demonstrate our theoretical results on two examples.

Title: AvatarMMC: 3D Head Avatar Generation and Editing with Multi-Modal Conditioning

• Authors: Wamiq Reyaz Para, Abdelrahman Eldesokey, Zhenyu Li, Pradyumna Reddy, Jiankang Deng, Peter Wonka
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.05803
• Pdf URL: https://arxiv.org/pdf/2402.05803
• Copy Paste: [[2402.05803]] AvatarMMC: 3D Head Avatar Generation and Editing with Multi-Modal Conditioning(https://arxiv.org/abs/2402.05803)
• Keywords: diffusion, generative
• Abstract: We introduce an approach for 3D head avatar generation and editing with multi-modal conditioning based on a 3D Generative Adversarial Network (GAN) and a Latent Diffusion Model (LDM). 3D GANs can generate high-quality head avatars given a single or no condition. However, it is challenging to generate samples that adhere to multiple conditions of different modalities. On the other hand, LDMs excel at learning complex conditional distributions. To this end, we propose to exploit the conditioning capabilities of LDMs to enable multi-modal control over the latent space of a pre-trained 3D GAN. Our method can generate and edit 3D head avatars given a mixture of control signals such as RGB input, segmentation masks, and global attributes. This provides better control over the generation and editing of synthetic avatars both globally and locally. Experiments show that our proposed approach outperforms a solely GAN-based approach both qualitatively and quantitatively on generation and editing tasks. To the best of our knowledge, our approach is the first to introduce multi-modal conditioning to 3D avatar generation and editing. \\href{avatarmmc-sig24.github.io}{Project Page}

Title: Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seeking

• Authors: Nikhil Sharma, Q. Vera Liao, Ziang Xiao
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.05880
• Pdf URL: https://arxiv.org/pdf/2402.05880
• Copy Paste: [[2402.05880]] Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seeking(https://arxiv.org/abs/2402.05880)
• Keywords: generative
• Abstract: Large language models (LLMs) powered conversational search systems have already been used by hundreds of millions of people, and are believed to bring many benefits over conventional search. However, while decades of research and public discourse interrogated the risk of search systems in increasing selective exposure and creating echo chambers -- limiting exposure to diverse opinions and leading to opinion polarization, little is known about such a risk of LLM-powered conversational search. We conduct two experiments to investigate: 1) whether and how LLM-powered conversational search increases selective exposure compared to conventional search; 2) whether and how LLMs with opinion biases that either reinforce or challenge the user's view change the effect. Overall, we found that participants engaged in more biased information querying with LLM-powered conversational search, and an opinionated LLM reinforcing their views exacerbated this bias. These results present critical implications for the development of LLMs and conversational search systems, and the policy governing these technologies.

Title: Collaborative Control for Geometry-Conditioned PBR Image Generation

• Authors: Shimon Vainer, Mark Boss, Mathias Parger, Konstantin Kutsy, Dante De Nigris, Ciara Rowles, Nicolas Perony, Simon Donné
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.05919
• Pdf URL: https://arxiv.org/pdf/2402.05919
• Copy Paste: [[2402.05919]] Collaborative Control for Geometry-Conditioned PBR Image Generation(https://arxiv.org/abs/2402.05919)
• Keywords: generative
• Abstract: Current 3D content generation builds on generative models that output RGB images. Modern graphics pipelines, however, require physically-based rendering (PBR) material properties. We propose to model the PBR image distribution directly to avoid photometric inaccuracies in RGB generation and the inherent ambiguity in extracting PBR from RGB. Existing paradigms for cross-modal finetuning are not suited for PBR generation due to a lack of data and the high dimensionality of the output modalities: we overcome both challenges by retaining a frozen RGB model and tightly linking a newly trained PBR model using a novel cross-network communication paradigm. As the base RGB model is fully frozen, the proposed method does not risk catastrophic forgetting during finetuning and remains compatible with techniques such as IPAdapter pretrained for the base RGB model. We validate our design choices, robustness to data sparsity, and compare against existing paradigms with an extensive experimental section.

Title: Time Series Diffusion in the Frequency Domain

• Authors: Jonathan Crabbé, Nicolas Huynh, Jan Stanczuk, Mihaela van der Schaar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.05933
• Pdf URL: https://arxiv.org/pdf/2402.05933
• Copy Paste: [[2402.05933]] Time Series Diffusion in the Frequency Domain(https://arxiv.org/abs/2402.05933)
• Keywords: diffusion, generative
• Abstract: Fourier analysis has been an instrumental tool in the development of signal processing. This leads us to wonder whether this framework could similarly benefit generative modelling. In this paper, we explore this question through the scope of time series diffusion models. More specifically, we analyze whether representing time series in the frequency domain is a useful inductive bias for score-based diffusion models. By starting from the canonical SDE formulation of diffusion in the time domain, we show that a dual diffusion process occurs in the frequency domain with an important nuance: Brownian motions are replaced by what we call mirrored Brownian motions, characterized by mirror symmetries among their components. Building on this insight, we show how to adapt the denoising score matching approach to implement diffusion models in the frequency domain. This results in frequency diffusion models, which we compare to canonical time diffusion models. Our empirical evaluation on real-world datasets, covering various domains like healthcare and finance, shows that frequency diffusion models better capture the training distribution than time diffusion models. We explain this observation by showing that time series from these datasets tend to be more localized in the frequency domain than in the time domain, which makes them easier to model in the former case. All our observations point towards impactful synergies between Fourier analysis and diffusion models.

Title: InstaGen: Enhancing Object Detection by Training on Synthetic Dataset

• Authors: Chengjian Feng, Yujie Zhong, Zequn Jie, Weidi Xie, Lin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05937
• Pdf URL: https://arxiv.org/pdf/2402.05937
• Copy Paste: [[2402.05937]] InstaGen: Enhancing Object Detection by Training on Synthetic Dataset(https://arxiv.org/abs/2402.05937)
• Keywords: diffusion, generative
• Abstract: In this paper, we introduce a novel paradigm to enhance the ability of object detector, e.g., expanding categories or improving detection performance, by training on synthetic dataset generated from diffusion models. Specifically, we integrate an instance-level grounding head into a pre-trained, generative diffusion model, to augment it with the ability of localising arbitrary instances in the generated images. The grounding head is trained to align the text embedding of category names with the regional visual feature of the diffusion model, using supervision from an off-the-shelf object detector, and a novel self-training scheme on (novel) categories not covered by the detector. This enhanced version of diffusion model, termed as InstaGen, can serve as a data synthesizer for object detection. We conduct thorough experiments to show that, object detector can be enhanced while training on the synthetic dataset from InstaGen, demonstrating superior performance over existing state-of-the-art methods in open-vocabulary (+4.5 AP) and data-sparse (+1.2 to 5.2 AP) scenarios.