2024-01-29

Title: Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Search

• Authors: Yanjie Li, Weijun Li, Lina Yu, Min Wu, Jingyi Liu, Wenqiang Li, Meilan Hao, Shu Wei, Yusong Deng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14424
• Pdf URL: https://arxiv.org/pdf/2401.14424
• Copy Paste: [[2401.14424]] Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Search(https://arxiv.org/abs/2401.14424)
• Keywords: generative
• Abstract: Finding a concise and interpretable mathematical formula that accurately describes the relationship between each variable and the predicted value in the data is a crucial task in scientific research, as well as a significant challenge in artificial intelligence. This problem is referred to as symbolic regression, which is an NP-hard problem. Last year, a symbolic regression method based on Monte Carlo Tree Search (MCTS) was proposed and sota was obtained on multiple datasets. While this algorithm has shown considerable improvement in recovering target expressions compared to previous methods, the lack of guidance during the MCTS process severely hampers its search efficiency. Recently, some algorithms have added a pre-trained policy network to guide the search of MCTS, but the pre-trained policy network generalizes poorly. To balance efficiency and generality, we propose SR-GPT combining ideas from AlphaZero. SR-GPT is a new symbolic regression algorithm that combines MCTS with a Generative Pre-Trained Transformer (GPT). By using GPT to guide the MCTS process, the search efficiency of MCTS is significantly improved. Next, we utilize the MCTS results to further refine the GPT, enhancing its capabilities and providing more accurate guidance for the MCTS process. MCTS and GPT are coupled together and optimize each other until the target expression is successfully determined. We conducted extensive evaluations of SR-GPT using 222 expressions sourced from over 10 different symbolic regression datasets. The experimental results demonstrate that SR-GPT outperforms existing state-of-the-art algorithms in accurately recovering symbolic expressions both with and without added noise.

Title: K-QA: A Real-World Medical Q&A Benchmark

• Authors: Itay Manes, Naama Ronn, David Cohen, Ran Ilan Ber, Zehavi Horowitz-Kugler, Gabriel Stanovsky
• Subjects: cs.CL, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14493
• Pdf URL: https://arxiv.org/pdf/2401.14493
• Copy Paste: [[2401.14493]] K-QA: A Real-World Medical Q&A Benchmark(https://arxiv.org/abs/2401.14493)
• Keywords: in-context
• Abstract: Ensuring the accuracy of responses provided by large language models (LLMs) is crucial, particularly in clinical settings where incorrect information may directly impact patient health. To address this challenge, we construct K-QA, a dataset containing 1,212 patient questions originating from real-world conversations held on K Health (an AI-driven clinical platform). We employ a panel of in-house physicians to answer and manually decompose a subset of K-QA into self-contained statements. Additionally, we formulate two NLI-based evaluation metrics approximating recall and precision: (1) comprehensiveness, measuring the percentage of essential clinical information in the generated answer and (2) hallucination rate, measuring the number of statements from the physician-curated response contradicted by the LLM answer. Finally, we use K-QA along with these metrics to evaluate several state-of-the-art models, as well as the effect of in-context learning and medically-oriented augmented retrieval schemes developed by the authors. Our findings indicate that in-context learning improves the comprehensiveness of the models, and augmented retrieval is effective in reducing hallucinations. We make K-QA available to to the community to spur research into medically accurate NLP applications.

Title: Evaluating GPT-3.5's Awareness and Summarization Abilities for European Constitutional Texts with Shared Topics

• Authors: Candida M. Greco, A. Tagarelli
• Subjects: cs.CL, cs.AI, cs.CY, cs.DL, physics.soc-ph
• Abstract URL: https://arxiv.org/abs/2401.14524
• Pdf URL: https://arxiv.org/pdf/2401.14524
• Copy Paste: [[2401.14524]] Evaluating GPT-3.5's Awareness and Summarization Abilities for European Constitutional Texts with Shared Topics(https://arxiv.org/abs/2401.14524)
• Keywords: generative
• Abstract: Constitutions are foundational legal documents that underpin the governmental and societal structures. As such, they are a reflection of a nation's cultural and social uniqueness, but also contribute to establish topics of universal importance, like citizens' rights and duties (RD). In this work, using the renowned GPT-3.5, we leverage generative large language models to understand constitutional passages that transcend national boundaries. A key contribution of our study is the introduction of a novel application of abstractive summarization on a multi-source collection of constitutional texts, with a focus on European countries' constitution passages related to RD topics. Our results show the meaningfulness of GPT-3.5 to produce informative, coherent and faithful summaries capturing RD topics across European countries.

Title: CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process

• Authors: Guangyi Chen, Yifan Shen, Zhenhao Chen, Xiangchen Song, Yuewen Sun, Weiran Yao, Xiao Liu, Kun Zhang
• Subjects: cs.LG, cs.CV, stat.ME
• Abstract URL: https://arxiv.org/abs/2401.14535
• Pdf URL: https://arxiv.org/pdf/2401.14535
• Copy Paste: [[2401.14535]] CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process(https://arxiv.org/abs/2401.14535)
• Keywords: generative
• Abstract: Identifying the underlying time-delayed latent causal processes in sequential data is vital for grasping temporal dynamics and making downstream reasoning. While some recent methods can robustly identify these latent causal variables, they rely on strict assumptions about the invertible generation process from latent variables to observed data. However, these assumptions are often hard to satisfy in real-world applications containing information loss. For instance, the visual perception process translates a 3D space into 2D images, or the phenomenon of persistence of vision incorporates historical data into current perceptions. To address this challenge, we establish an identifiability theory that allows for the recovery of independent latent components even when they come from a nonlinear and non-invertible mix. Using this theory as a foundation, we propose a principled approach, CaRiNG, to learn the CAusal RepresentatIon of Non-invertible Generative temporal data with identifiability guarantees. Specifically, we utilize temporal context to recover lost latent information and apply the conditions in our theory to guide the training process. Through experiments conducted on synthetic datasets, we validate that our CaRiNG method reliably identifies the causal process, even when the generation process is non-invertible. Moreover, we demonstrate that our approach considerably improves temporal understanding and reasoning in practical applications.

Title: Revisiting Active Learning in the Era of Vision Foundation Models

• Authors: Sanket Rajan Gupte, Josiah Aklilu, Jeffrey J. Nirschl, Serena Yeung-Levy
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14555
• Pdf URL: https://arxiv.org/pdf/2401.14555
• Copy Paste: [[2401.14555]] Revisiting Active Learning in the Era of Vision Foundation Models(https://arxiv.org/abs/2401.14555)
• Keywords: foundation model
• Abstract: Foundation vision or vision-language models are trained on large unlabeled or noisy data and learn robust representations that can achieve impressive zero- or few-shot performance on diverse tasks. Given these properties, they are a natural fit for active learning (AL), which aims to maximize labeling efficiency, but the full potential of foundation models has not been explored in the context of AL, specifically in the low-budget regime. In this work, we evaluate how foundation models influence three critical components of effective AL, namely, 1) initial labeled pool selection, 2) ensuring diverse sampling, and 3) the trade-off between representative and uncertainty sampling. We systematically study how the robust representations of foundation models (DINOv2, OpenCLIP) challenge existing findings in active learning. Our observations inform the principled construction of a new simple and elegant AL strategy that balances uncertainty estimated via dropout with sample diversity. We extensively test our strategy on many challenging image classification benchmarks, including natural images as well as out-of-domain biomedical images that are relatively understudied in the AL literature. Source code will be made available.

Title: Language Modelling Approaches to Adaptive Machine Translation

• Authors: Yasmin Moslem
• Subjects: cs.CL, cs.AI, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.14559
• Pdf URL: https://arxiv.org/pdf/2401.14559
• Copy Paste: [[2401.14559]] Language Modelling Approaches to Adaptive Machine Translation(https://arxiv.org/abs/2401.14559)
• Keywords: in-context
• Abstract: Consistency is a key requirement of high-quality translation. It is especially important to adhere to pre-approved terminology and adapt to corrected translations in domain-specific projects. Machine translation (MT) has achieved significant progress in the area of domain adaptation. However, in-domain data scarcity is common in translation settings, due to the lack of specialised datasets and terminology, or inconsistency and inaccuracy of available in-domain translations. In such scenarios where there is insufficient in-domain data to fine-tune MT models, producing translations that are consistent with the relevant context is challenging. While real-time adaptation can make use of smaller amounts of in-domain data to improve the translation on the fly, it remains challenging due to supported context limitations and efficiency constraints. Large language models (LLMs) have recently shown interesting capabilities of in-context learning, where they learn to replicate certain input-output text generation patterns, without further fine-tuning. Such capabilities have opened new horizons for domain-specific data augmentation and real-time adaptive MT. This work attempts to address two main relevant questions: 1) in scenarios involving human interaction and continuous feedback, can we employ language models to improve the quality of adaptive MT at inference time? and 2) in the absence of sufficient in-domain data, can we use pre-trained large-scale language models to improve the process of MT domain adaptation?

Title: Diffusion Stochastic Optimization for Min-Max Problems

• Authors: Haoyuan Cai, Sulaiman A. Alghunaim, Ali H. Sayed
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2401.14585
• Pdf URL: https://arxiv.org/pdf/2401.14585
• Copy Paste: [[2401.14585]] Diffusion Stochastic Optimization for Min-Max Problems(https://arxiv.org/abs/2401.14585)
• Keywords: diffusion, generative
• Abstract: The optimistic gradient method is useful in addressing minimax optimization problems. Motivated by the observation that the conventional stochastic version suffers from the need for a large batch size on the order of $\mathcal{O}(\varepsilon^{-2})$ to achieve an $\varepsilon$-stationary solution, we introduce and analyze a new formulation termed Diffusion Stochastic Same-Sample Optimistic Gradient (DSS-OG). We prove its convergence and resolve the large batch issue by establishing a tighter upper bound, under the more general setting of nonconvex Polyak-Lojasiewicz (PL) risk functions. We also extend the applicability of the proposed method to the distributed scenario, where agents communicate with their neighbors via a left-stochastic protocol. To implement DSS-OG, we can query the stochastic gradient oracles in parallel with some extra memory overhead, resulting in a complexity comparable to its conventional counterpart. To demonstrate the efficacy of the proposed algorithm, we conduct tests by training generative adversarial networks.

Title: Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learning

• Authors: Tao He, Tongtong Wu, Dongyang Zhang, Guiduo Duan, Ke Qin, Yuan-Fang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14626
• Pdf URL: https://arxiv.org/pdf/2401.14626
• Copy Paste: [[2401.14626]] Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learning(https://arxiv.org/abs/2401.14626)
• Keywords: in-context
• Abstract: Scene graph generation (SGG) endeavors to predict visual relationships between pairs of objects within an image. Prevailing SGG methods traditionally assume a one-off learning process for SGG. This conventional paradigm may necessitate repetitive training on all previously observed samples whenever new relationships emerge, mitigating the risk of forgetting previously acquired knowledge. This work seeks to address this pitfall inherent in a suite of prior relationship predictions. Motivated by the achievements of in-context learning in pretrained language models, our approach imbues the model with the capability to predict relationships and continuously acquire novel knowledge without succumbing to catastrophic forgetting. To achieve this goal, we introduce a novel and pragmatic framework for scene graph generation, namely Lifelong Scene Graph Generation (LSGG), where tasks, such as predicates, unfold in a streaming fashion. In this framework, the model is constrained to exclusive training on the present task, devoid of access to previously encountered training data, except for a limited number of exemplars, but the model is tasked with inferring all predicates it has encountered thus far. Rigorous experiments demonstrate the superiority of our proposed method over state-of-the-art SGG models in the context of LSGG across a diverse array of metrics. Besides, extensive experiments on the two mainstream benchmark datasets, VG and Open-Image(v6), show the superiority of our proposed model to a number of competitive SGG models in terms of continuous learning and conventional settings. Moreover, comprehensive ablation experiments demonstrate the effectiveness of each component in our model.

Title: Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support

• Authors: Xiaojun Wu, Dixiang Zhang, Ruyi Gan, Junyu Lu, Ziwei Wu, Renliang Sun, Jiaxing Zhang, Pingjian Zhang, Yan Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14688
• Pdf URL: https://arxiv.org/pdf/2401.14688
• Copy Paste: [[2401.14688]] Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support(https://arxiv.org/abs/2401.14688)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}{this https URL}, fostering further research and collaboration in this domain.

Title: A Survey on Video Prediction: From Deterministic to Generative Approaches

• Authors: Ruibo Ming, Zhewei Huang, Zhuoxuan Ju, Jianming Hu, Lihui Peng, Shuchang Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14718
• Pdf URL: https://arxiv.org/pdf/2401.14718
• Copy Paste: [[2401.14718]] A Survey on Video Prediction: From Deterministic to Generative Approaches(https://arxiv.org/abs/2401.14718)
• Keywords: generative
• Abstract: Video prediction, a fundamental task in computer vision, aims to enable models to generate sequences of future frames based on existing video content. This task has garnered widespread application across various domains. In this paper, we comprehensively survey both historical and contemporary works in this field, encompassing the most widely used datasets and algorithms. Our survey scrutinizes the challenges and evolving landscape of video prediction within the realm of computer vision. We propose a novel taxonomy centered on the stochastic nature of video prediction algorithms. This taxonomy accentuates the gradual transition from deterministic to generative prediction methodologies, underlining significant advancements and shifts in approach.

Title: Large Language Model Adaptation for Financial Sentiment Analysis

• Authors: Pau Rodriguez Inserte, Mariam Nakhlé, Raheel Qader, Gaetan Caillaut, Jingshu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14777
• Pdf URL: https://arxiv.org/pdf/2401.14777
• Copy Paste: [[2401.14777]] Large Language Model Adaptation for Financial Sentiment Analysis(https://arxiv.org/abs/2401.14777)
• Keywords: foundation model, generative
• Abstract: Natural language processing (NLP) has recently gained relevance within financial institutions by providing highly valuable insights into companies and markets' financial documents. However, the landscape of the financial domain presents extra challenges for NLP, due to the complexity of the texts and the use of specific terminology. Generalist language models tend to fall short in tasks specifically tailored for finance, even when using large language models (LLMs) with great natural language understanding and generative capabilities. This paper presents a study on LLM adaptation methods targeted at the financial domain and with high emphasis on financial sentiment analysis. To this purpose, two foundation models with less than 1.5B parameters have been adapted using a wide range of strategies. We show that through careful fine-tuning on both financial documents and instructions, these foundation models can be adapted to the target domain. Moreover, we observe that small LLMs have comparable performance to larger scale models, while being more efficient in terms of parameters and data. In addition to the models, we show how to generate artificial instructions through LLMs to augment the number of samples of the instruction dataset.

Title: Deep Variational Privacy Funnel: General Modeling with Applications in Face Recognition

• Authors: Behrooz Razeghi, Parsa Rahimi, Sébastien Marcel
• Subjects: cs.CV, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14792
• Pdf URL: https://arxiv.org/pdf/2401.14792
• Copy Paste: [[2401.14792]] Deep Variational Privacy Funnel: General Modeling with Applications in Face Recognition(https://arxiv.org/abs/2401.14792)
• Keywords: generative
• Abstract: In this study, we harness the information-theoretic Privacy Funnel (PF) model to develop a method for privacy-preserving representation learning using an end-to-end training framework. We rigorously address the trade-off between obfuscation and utility. Both are quantified through the logarithmic loss, a measure also recognized as self-information loss. This exploration deepens the interplay between information-theoretic privacy and representation learning, offering substantive insights into data protection mechanisms for both discriminative and generative models. Importantly, we apply our model to state-of-the-art face recognition systems. The model demonstrates adaptability across diverse inputs, from raw facial images to both derived or refined embeddings, and is competent in tasks such as classification, reconstruction, and generation.

Title: ChemDFM: Dialogue Foundation Model for Chemistry

• Authors: Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, Guodong Shen, Xin Chen, Kai Yu
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2401.14818
• Pdf URL: https://arxiv.org/pdf/2401.14818
• Copy Paste: [[2401.14818]] ChemDFM: Dialogue Foundation Model for Chemistry(https://arxiv.org/abs/2401.14818)
• Keywords: foundation model
• Abstract: Large language models (LLMs) have established great success in the general domain of natural language processing. Their emerging task generalization and free-form dialogue capabilities can greatly help to design Chemical General Intelligence (CGI) to assist real-world research in chemistry. However, the existence of specialized language and knowledge in the field of chemistry, such as the highly informative SMILES notation, hinders the performance of general-domain LLMs in chemistry. To this end, we develop ChemDFM, the first LLM towards CGI. ChemDFM-13B is trained on 34B tokens from chemical literature, textbooks, and instructions as well as various data from the general domain. Therefore, it can store, understand, and reason over chemical knowledge and languages while still possessing advanced free-form language comprehension capabilities. Extensive quantitative evaluation shows that ChemDFM can significantly outperform the representative open-sourced LLMs. Moreover, ChemDFM can also surpass GPT-4 on a great portion of chemical tasks, despite the significant size difference. Further qualitative evaluations demonstrate the efficiency and effectiveness of ChemDFM in real-world research scenarios. We will open-source the ChemDFM model soon.

Title: Text Image Inpainting via Global Structure-Guided Diffusion Models

• Authors: Shipeng Zhu, Pengfei Fang, Chenjie Zhu, Zuoyan Zhao, Qiang Xu, Hui Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14832
• Pdf URL: https://arxiv.org/pdf/2401.14832
• Copy Paste: [[2401.14832]] Text Image Inpainting via Global Structure-Guided Diffusion Models(https://arxiv.org/abs/2401.14832)
• Keywords: diffusion
• Abstract: Real-world text can be damaged by corrosion issues caused by environmental or human factors, which hinder the preservation of the complete styles of texts, e.g., texture and structure. These corrosion issues, such as graffiti signs and incomplete signatures, bring difficulties in understanding the texts, thereby posing significant challenges to downstream applications, e.g., scene text recognition and signature identification. Notably, current inpainting techniques often fail to adequately address this problem and have difficulties restoring accurate text images along with reasonable and consistent styles. Formulating this as an open problem of text image inpainting, this paper aims to build a benchmark to facilitate its study. In doing so, we establish two specific text inpainting datasets which contain scene text images and handwritten text images, respectively. Each of them includes images revamped by real-life and synthetic datasets, featuring pairs of original images, corrupted images, and other assistant information. On top of the datasets, we further develop a novel neural framework, Global Structure-guided Diffusion Model (GSDM), as a potential solution. Leveraging the global structure of the text as a prior, the proposed GSDM develops an efficient diffusion model to recover clean texts. The efficacy of our approach is demonstrated by thorough empirical study, including a substantial boost in both recognition accuracy and image quality. These findings not only highlight the effectiveness of our method but also underscore its potential to enhance the broader field of text image understanding and processing. Code and datasets are available at: https://github.com/blackprotoss/GSDM.

Title: Memory-Inspired Temporal Prompt Interaction for Text-Image Classification

• Authors: Xinyao Yu, Hao Sun, Ziwei Niu, Rui Qin, Zhenjia Bai, Yen-Wei Chen, Lanfen Lin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14856
• Pdf URL: https://arxiv.org/pdf/2401.14856
• Copy Paste: [[2401.14856]] Memory-Inspired Temporal Prompt Interaction for Text-Image Classification(https://arxiv.org/abs/2401.14856)
• Keywords: foundation model
• Abstract: In recent years, large-scale pre-trained multimodal models (LMM) generally emerge to integrate the vision and language modalities, achieving considerable success in various natural language processing and computer vision tasks. The growing size of LMMs, however, results in a significant computational cost for fine-tuning these models for downstream tasks. Hence, prompt-based interaction strategy is studied to align modalities more efficiently. In this contex, we propose a novel prompt-based multimodal interaction strategy inspired by human memory strategy, namely Memory-Inspired Temporal Prompt Interaction (MITP). Our proposed method involves in two stages as in human memory strategy: the acquiring stage, and the consolidation and activation stage. We utilize temporal prompts on intermediate layers to imitate the acquiring stage, leverage similarity-based prompt interaction to imitate memory consolidation, and employ prompt generation strategy to imitate memory activation. The main strength of our paper is that we interact the prompt vectors on intermediate layers to leverage sufficient information exchange between modalities, with compressed trainable parameters and memory usage. We achieve competitive results on several datasets with relatively small memory usage and 2.0M of trainable parameters (about 1% of the pre-trained foundation model).

Title: DAM: Diffusion Activation Maximization for 3D Global Explanations

• Authors: Hanxiao Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14938
• Pdf URL: https://arxiv.org/pdf/2401.14938
• Copy Paste: [[2401.14938]] DAM: Diffusion Activation Maximization for 3D Global Explanations(https://arxiv.org/abs/2401.14938)
• Keywords: diffusion
• Abstract: In recent years, the performance of point cloud models has been rapidly improved. However, due to the limited amount of relevant explainability studies, the unreliability and opacity of these black-box models may lead to potential risks in applications where human lives are at stake, e.g. autonomous driving or healthcare. This work proposes a DDPM-based point cloud global explainability method (DAM) that leverages Point Diffusion Transformer (PDT), a novel point-wise symmetric model, with dual-classifier guidance to generate high-quality global explanations. In addition, an adapted path gradient integration method for DAM is proposed, which not only provides a global overview of the saliency maps for point cloud categories, but also sheds light on how the attributions of the explanations vary during the generation process. Extensive experiments indicate that our method outperforms existing ones in terms of perceptibility, representativeness, and diversity, with a significant reduction in generation time. Our code is available at: https://github.com/Explain3D/DAM

Title: Annotated Hands for Generative Models

• Authors: Yue Yang, Atith N Gandhi, Greg Turk
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.15075
• Pdf URL: https://arxiv.org/pdf/2401.15075
• Copy Paste: [[2401.15075]] Annotated Hands for Generative Models(https://arxiv.org/abs/2401.15075)
• Keywords: diffusion, generative
• Abstract: Generative models such as GANs and diffusion models have demonstrated impressive image generation capabilities. Despite these successes, these systems are surprisingly poor at creating images with hands. We propose a novel training framework for generative models that substantially improves the ability of such systems to create hand images. Our approach is to augment the training images with three additional channels that provide annotations to hands in the image. These annotations provide additional structure that coax the generative model to produce higher quality hand images. We demonstrate this approach on two different generative models: a generative adversarial network and a diffusion model. We demonstrate our method both on a new synthetic dataset of hand images and also on real photographs that contain hands. We measure the improved quality of the generated hands through higher confidence in finger joint identification using an off-the-shelf hand detector.