2024-01-25

Title: CCA: Collaborative Competitive Agents for Image Editing

• Authors: Tiankai Hang, Shuyang Gu, Dong Chen, Xin Geng, Baining Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13011
• Pdf URL: https://arxiv.org/pdf/2401.13011
• Copy Paste: [[2401.13011]] CCA: Collaborative Competitive Agents for Image Editing(https://arxiv.org/abs/2401.13011)
• Keywords: generative
• Abstract: This paper presents a novel generative model, Collaborative Competitive Agents (CCA), which leverages the capabilities of multiple Large Language Models (LLMs) based agents to execute complex tasks. Drawing inspiration from Generative Adversarial Networks (GANs), the CCA system employs two equal-status generator agents and a discriminator agent. The generators independently process user instructions and generate results, while the discriminator evaluates the outputs, and provides feedback for the generator agents to further reflect and improve the generation results. Unlike the previous generative model, our system can obtain the intermediate steps of generation. This allows each generator agent to learn from other successful executions due to its transparency, enabling a collaborative competition that enhances the quality and robustness of the system's results. The primary focus of this study is image editing, demonstrating the CCA's ability to handle intricate instructions robustly. The paper's main contributions include the introduction of a multi-agent-based generative model with controllable intermediate steps and iterative optimization, a detailed examination of agent relationships, and comprehensive experiments on image editing. Code is available at \href{https://github.com/TiankaiHang/CCA}{https://github.com/TiankaiHang/CCA}.

Title: Contractive Diffusion Probabilistic Models

• Authors: Wenpin Tang, Hanyang Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13115
• Pdf URL: https://arxiv.org/pdf/2401.13115
• Copy Paste: [[2401.13115]] Contractive Diffusion Probabilistic Models(https://arxiv.org/abs/2401.13115)
• Keywords: diffusion, generative
• Abstract: Diffusion probabilistic models (DPMs) have emerged as a promising technology in generative modeling. The success of DPMs relies on two ingredients: time reversal of Markov diffusion processes and score matching. Most existing work implicitly assumes that score matching is close to perfect, while this assumption is questionable. In view of possibly unguaranteed score matching, we propose a new criterion -- the contraction of backward sampling in the design of DPMs. This leads to a novel class of contractive DPMs (CDPMs), including contractive Ornstein-Uhlenbeck (OU) processes and contractive sub-variance preserving (sub-VP) stochastic differential equations (SDEs). The key insight is that the contraction in the backward process narrows score matching errors, as well as discretization error. Thus, the proposed CDPMs are robust to both sources of error. Our proposal is supported by theoretical results, and is corroborated by experiments. Notably, contractive sub-VP shows the best performance among all known SDE-based DPMs on the CIFAR-10 dataset.

Title: Compositional Generative Inverse Design

• Authors: Tailin Wu, Takashi Maruyama, Long Wei, Tao Zhang, Yilun Du, Gianluca Iaccarino, Jure Leskovec
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2401.13171
• Pdf URL: https://arxiv.org/pdf/2401.13171
• Copy Paste: [[2401.13171]] Compositional Generative Inverse Design(https://arxiv.org/abs/2401.13171)
• Keywords: diffusion, generative
• Abstract: Inverse design, where we seek to design input variables in order to optimize an underlying objective function, is an important problem that arises across fields such as mechanical engineering to aerospace engineering. Inverse design is typically formulated as an optimization problem, with recent works leveraging optimization across learned dynamics models. However, as models are optimized they tend to fall into adversarial modes, preventing effective sampling. We illustrate that by instead optimizing over the learned energy function captured by the diffusion model, we can avoid such adversarial examples and significantly improve design performance. We further illustrate how such a design system is compositional, enabling us to combine multiple different diffusion models representing subcomponents of our desired system to design systems with every specified component. In an N-body interaction task and a challenging 2D multi-airfoil design task, we demonstrate that by composing the learned diffusion model at test time, our method allows us to design initial states and boundary shapes that are more complex than those in the training data. Our method outperforms state-of-the-art neural inverse design method by an average of 41.5% in prediction MAE and 14.3% in design objective for the N-body dataset and discovers formation flying to minimize drag in the multi-airfoil design task. Project website and code can be found at https://github.com/AI4Science-WestlakeU/cindm.

Title: Towards Multi-domain Face Landmark Detection with Synthetic Data from Diffusion model

• Authors: Yuanming Li, Gwantae Kim, Jeong-gi Kwak, Bon-hwa Ku, Hanseok Ko
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13191
• Pdf URL: https://arxiv.org/pdf/2401.13191
• Copy Paste: [[2401.13191]] Towards Multi-domain Face Landmark Detection with Synthetic Data from Diffusion model(https://arxiv.org/abs/2401.13191)
• Keywords: diffusion
• Abstract: Recently, deep learning-based facial landmark detection for in-the-wild faces has achieved significant improvement. However, there are still challenges in face landmark detection in other domains (e.g. cartoon, caricature, etc). This is due to the scarcity of extensively annotated training data. To tackle this concern, we design a two-stage training approach that effectively leverages limited datasets and the pre-trained diffusion model to obtain aligned pairs of landmarks and face in multiple domains. In the first stage, we train a landmark-conditioned face generation model on a large dataset of real faces. In the second stage, we fine-tune the above model on a small dataset of image-landmark pairs with text prompts for controlling the domain. Our new designs enable our method to generate high-quality synthetic paired datasets from multiple domains while preserving the alignment between landmarks and facial features. Finally, we fine-tuned a pre-trained face landmark detection model on the synthetic dataset to achieve multi-domain face landmark detection. Our qualitative and quantitative results demonstrate that our method outperforms existing methods on multi-domain face landmark detection.

Title: Boosting the Transferability of Adversarial Examples via Local Mixup and Adaptive Step Size

• Authors: Junlin Liu, Xinchen Lyu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13205
• Pdf URL: https://arxiv.org/pdf/2401.13205
• Copy Paste: [[2401.13205]] Boosting the Transferability of Adversarial Examples via Local Mixup and Adaptive Step Size(https://arxiv.org/abs/2401.13205)
• Keywords: generative
• Abstract: Adversarial examples are one critical security threat to various visual applications, where injected human-imperceptible perturbations can confuse the output.Generating transferable adversarial examples in the black-box setting is crucial but challenging in practice. Existing input-diversity-based methods adopt different image transformations, but may be inefficient due to insufficient input diversity and an identical perturbation step size. Motivated by the fact that different image regions have distinctive weights in classification, this paper proposes a black-box adversarial generative framework by jointly designing enhanced input diversity and adaptive step sizes. We design local mixup to randomly mix a group of transformed adversarial images, strengthening the input diversity. For precise adversarial generation, we project the perturbation into the $tanh$ space to relax the boundary constraint. Moreover, the step sizes of different regions can be dynamically adjusted by integrating a second-order momentum.Extensive experiments on ImageNet validate that our framework can achieve superior transferability compared to state-of-the-art baselines.

Title: Multitask Active Learning for Graph Anomaly Detection

• Authors: Wenjing Chang, Kay Liu, Kaize Ding, Philip S. Yu, Jianjun Yu
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.13210
• Pdf URL: https://arxiv.org/pdf/2401.13210
• Copy Paste: [[2401.13210]] Multitask Active Learning for Graph Anomaly Detection(https://arxiv.org/abs/2401.13210)
• Keywords: anomaly
• Abstract: In the web era, graph machine learning has been widely used on ubiquitous graph-structured data. As a pivotal component for bolstering web security and enhancing the robustness of graph-based applications, the significance of graph anomaly detection is continually increasing. While Graph Neural Networks (GNNs) have demonstrated efficacy in supervised and semi-supervised graph anomaly detection, their performance is contingent upon the availability of sufficient ground truth labels. The labor-intensive nature of identifying anomalies from complex graph structures poses a significant challenge in real-world applications. Despite that, the indirect supervision signals from other tasks (e.g., node classification) are relatively abundant. In this paper, we propose a novel MultItask acTIve Graph Anomaly deTEction framework, namely MITIGATE. Firstly, by coupling node classification tasks, MITIGATE obtains the capability to detect out-of-distribution nodes without known anomalies. Secondly, MITIGATE quantifies the informativeness of nodes by the confidence difference across tasks, allowing samples with conflicting predictions to provide informative yet not excessively challenging information for subsequent training. Finally, to enhance the likelihood of selecting representative nodes that are distant from known patterns, MITIGATE adopts a masked aggregation mechanism for distance measurement, considering both inherent features of nodes and current labeled status. Empirical studies on four datasets demonstrate that MITIGATE significantly outperforms the state-of-the-art methods for anomaly detection. Our code is publicly available at: https://github.com/AhaChang/MITIGATE.

Title: Scalable Link Prediction on Large-Scale Heterogeneous Graphs with Large Language Models

• Authors: Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei, Xueqi Chen
• Subjects: cs.CL, cs.AI, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.13227
• Pdf URL: https://arxiv.org/pdf/2401.13227
• Copy Paste: [[2401.13227]] Scalable Link Prediction on Large-Scale Heterogeneous Graphs with Large Language Models(https://arxiv.org/abs/2401.13227)
• Keywords: self-supervised
• Abstract: Exploring the application of large-scale language models to graph learning is a novel endeavor. However, the vast amount of information inherent in large graphs poses significant challenges to this process. This paper focuses on the link prediction task and introduces LPNL (Link Prediction via Natural Language), a framework based on a large language model designed for scalable link prediction on large-scale heterogeneous graphs.We design novel prompts for link prediction that articulate graph details in natural language. We propose a two-stage sampling pipeline to extract crucial information from large-scale heterogeneous graphs, and a divide-and-conquer strategy to control the input token count within predefined limits, addressing the challenge of overwhelming information. We fine-tune a T5 model based on our self-supervised learning designed for for link prediction. Extensive experiments on a large public heterogeneous graphs demonstrate that LPNL outperforms various advanced baselines, highlighting its remarkable performance in link prediction tasks on large-scale graphs.

Title: Adaptive Crowdsourcing Via Self-Supervised Learning

• Authors: Anmol Kagrecha, Henrik Marklund, Benjamin Van Roy, Hong Jun Jeon, Richard Zeckhauser
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.13239
• Pdf URL: https://arxiv.org/pdf/2401.13239
• Copy Paste: [[2401.13239]] Adaptive Crowdsourcing Via Self-Supervised Learning(https://arxiv.org/abs/2401.13239)
• Keywords: self-supervised
• Abstract: Common crowdsourcing systems average estimates of a latent quantity of interest provided by many crowdworkers to produce a group estimate. We develop a new approach -- just-predict-others -- that leverages self-supervised learning and a novel aggregation scheme. This approach adapts weights assigned to crowdworkers based on estimates they provided for previous quantities. When skills vary across crowdworkers or their estimates correlate, the weighted sum offers a more accurate group estimate than the average. Existing algorithms such as expectation maximization can, at least in principle, produce similarly accurate group estimates. However, their computational requirements become onerous when complex models, such as neural networks, are required to express relationships among crowdworkers. Just-predict-others accommodates such complexity as well as many other practical challenges. We analyze the efficacy of just-predict-others through theoretical and computational studies. Among other things, we establish asymptotic optimality as the number of engagements per crowdworker grows.

Title: Dual-modal Dynamic Traceback Learning for Medical Report Generation

• Authors: Shuchang Ye, Mingyuan Meng, Mingjian Li, Dagan Feng, Jinman Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13267
• Pdf URL: https://arxiv.org/pdf/2401.13267
• Copy Paste: [[2401.13267]] Dual-modal Dynamic Traceback Learning for Medical Report Generation(https://arxiv.org/abs/2401.13267)
• Keywords: generative
• Abstract: With increasing reliance on medical imaging in clinical practices, automated report generation from medical images is in great demand. Existing report generation methods typically adopt an encoder-decoder deep learning framework to build a uni-directional image-to-report mapping. However, such a framework ignores the bi-directional mutual associations between images and reports, thus incurring difficulties in associating the intrinsic medical meanings between them. Recent generative representation learning methods have demonstrated the benefits of dual-modal learning from both image and text modalities. However, these methods exhibit two major drawbacks for medical report generation: 1) they tend to capture morphological information and have difficulties in capturing subtle pathological semantic information, and 2) they predict masked text rely on both unmasked images and text, inevitably degrading performance when inference is based solely on images. In this study, we propose a new report generation framework with dual-modal dynamic traceback learning (DTrace) to overcome the two identified drawbacks and enable dual-modal learning for medical report generation. To achieve this, our DTrace introduces a traceback mechanism to control the semantic validity of generated content via self-assessment. Further, our DTrace introduces a dynamic learning strategy to adapt to various proportions of image and text input, enabling report generation without reliance on textual input during inference. Extensive experiments on two well-benchmarked datasets (IU-Xray and MIMIC-CXR) show that our DTrace outperforms state-of-the-art medical report generation methods.

Title: Audio-Infused Automatic Image Colorization by Exploiting Audio Scene Semantics

• Authors: Pengcheng Zhao, Yanxiang Chen, Yang Zhao, Wei Jia, Zhao Zhang, Ronggang Wang, Richang Hong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13270
• Pdf URL: https://arxiv.org/pdf/2401.13270
• Copy Paste: [[2401.13270]] Audio-Infused Automatic Image Colorization by Exploiting Audio Scene Semantics(https://arxiv.org/abs/2401.13270)
• Keywords: self-supervised
• Abstract: Automatic image colorization is inherently an ill-posed problem with uncertainty, which requires an accurate semantic understanding of scenes to estimate reasonable colors for grayscale images. Although recent interaction-based methods have achieved impressive performance, it is still a very difficult task to infer realistic and accurate colors for automatic colorization. To reduce the difficulty of semantic understanding of grayscale scenes, this paper tries to utilize corresponding audio, which naturally contains extra semantic information about the same scene. Specifically, a novel audio-infused automatic image colorization (AIAIC) network is proposed, which consists of three stages. First, we take color image semantics as a bridge and pretrain a colorization network guided by color image semantics. Second, the natural co-occurrence of audio and video is utilized to learn the color semantic correlations between audio and visual scenes. Third, the implicit audio semantic representation is fed into the pretrained network to finally realize the audio-guided colorization. The whole process is trained in a self-supervised manner without human annotation. In addition, an audiovisual colorization dataset is established for training and testing. Experiments demonstrate that audio guidance can effectively improve the performance of automatic colorization, especially for some scenes that are difficult to understand only from visual modality.

Title: RefreshNet: Learning Multiscale Dynamics through Hierarchical Refreshing

• Authors: Junaid Farooq, Danish Rafiq, Pantelis R. Vlachas, Mohammad Abid Bazaz
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13282
• Pdf URL: https://arxiv.org/pdf/2401.13282
• Copy Paste: [[2401.13282]] RefreshNet: Learning Multiscale Dynamics through Hierarchical Refreshing(https://arxiv.org/abs/2401.13282)
• Keywords: diffusion
• Abstract: Forecasting complex system dynamics, particularly for long-term predictions, is persistently hindered by error accumulation and computational burdens. This study presents RefreshNet, a multiscale framework developed to overcome these challenges, delivering an unprecedented balance between computational efficiency and predictive accuracy. RefreshNet incorporates convolutional autoencoders to identify a reduced order latent space capturing essential features of the dynamics, and strategically employs multiple recurrent neural network (RNN) blocks operating at varying temporal resolutions within the latent space, thus allowing the capture of latent dynamics at multiple temporal scales. The unique "refreshing" mechanism in RefreshNet allows coarser blocks to reset inputs of finer blocks, effectively controlling and alleviating error accumulation. This design demonstrates superiority over existing techniques regarding computational efficiency and predictive accuracy, especially in long-term forecasting. The framework is validated using three benchmark applications: the FitzHugh-Nagumo system, the Reaction-Diffusion equation, and Kuramoto-Sivashinsky dynamics. RefreshNet significantly outperforms state-of-the-art methods in long-term forecasting accuracy and speed, marking a significant advancement in modeling complex systems and opening new avenues in understanding and predicting their behavior.

Title: Classification of Radiologically Isolated Syndrome and Clinically Isolated Syndrome with Machine-Learning Techniques

• Authors: V Mato-Abad, A Labiano-Fontcuberta, S Rodriguez-Yanez, R Garcia-Vazquez, CR Munteanu, J Andrade-Garda, A Domingo-Santos, V Galan Sanchez-Seco, Y Aladro, ML Martinez-Gines, L Ayuso, J Benito-Leon
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13301
• Pdf URL: https://arxiv.org/pdf/2401.13301
• Copy Paste: [[2401.13301]] Classification of Radiologically Isolated Syndrome and Clinically Isolated Syndrome with Machine-Learning Techniques(https://arxiv.org/abs/2401.13301)
• Keywords: diffusion
• Abstract: Background and purpose: The unanticipated detection by magnetic resonance imaging (MRI) in the brain of asymptomatic subjects of white matter lesions suggestive of multiple sclerosis (MS) has been named radiologically isolated syndrome (RIS). As the difference between early MS [i.e. clinically isolated syndrome (CIS)] and RIS is the occurrence of a clinical event, it is logical to improve detection of the subclinical form without interfering with MRI as there are radiological diagnostic criteria for that. Our objective was to use machine-learning classification methods to identify morphometric measures that help to discriminate patients with RIS from those with CIS. Methods: We used a multimodal 3-T MRI approach by combining MRI biomarkers (cortical thickness, cortical and subcortical grey matter volume, and white matter integrity) of a cohort of 17 patients with RIS and 17 patients with CIS for single-subject level classification. Results: The best proposed models to predict the diagnosis of CIS and RIS were based on the Naive Bayes, Bagging and Multilayer Perceptron classifiers using only three features: the left rostral middle frontal gyrus volume and the fractional anisotropy values in the right amygdala and right lingual gyrus. The Naive Bayes obtained the highest accuracy [overall classification, 0.765; area under the receiver operating characteristic (AUROC), 0.782]. Conclusions: A machine-learning approach applied to multimodal MRI data may differentiate between the earliest clinical expressions of MS (CIS and RIS) with an accuracy of 78%. Keywords: Bagging; Multilayer Perceptron; Naive Bayes classifier; clinically isolated syndrome; diffusion tensor imaging; machine-learning; magnetic resonance imaging; multiple sclerosis; radiologically isolated syndrome.

Title: MaLA-500: Massive Language Adaptation of Large Language Models

• Authors: Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13303
• Pdf URL: https://arxiv.org/pdf/2401.13303
• Copy Paste: [[2401.13303]] MaLA-500: Massive Language Adaptation of Large Language Models(https://arxiv.org/abs/2401.13303)
• Keywords: in-context
• Abstract: Large language models have advanced the state of the art in natural language processing. However, their predominant design for English or a limited set of languages creates a substantial gap in their effectiveness for low-resource languages. To bridge this gap, we introduce MaLA-500, a novel large language model designed to cover an extensive range of 534 languages. To train MaLA-500, we employ vocabulary extension and continued pretraining on LLaMA 2 with Glot500-c. Our experiments on SIB-200 show that MaLA-500 achieves state-of-the-art in-context learning results. We release MaLA-500 at https://huggingface.co/MaLA-LM

Title: Generating Synthetic Health Sensor Data for Privacy-Preserving Wearable Stress Detection

• Authors: Lucas Lange, Nils Wenzlitschke, Erhard Rahm
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.13327
• Pdf URL: https://arxiv.org/pdf/2401.13327
• Copy Paste: [[2401.13327]] Generating Synthetic Health Sensor Data for Privacy-Preserving Wearable Stress Detection(https://arxiv.org/abs/2401.13327)
• Keywords: generative
• Abstract: Smartwatch health sensor data is increasingly utilized in smart health applications and patient monitoring, including stress detection. However, such medical data often comprises sensitive personal information and is resource-intensive to acquire for research purposes. In response to this challenge, we introduce the privacy-aware synthetization of multi-sensor smartwatch health readings related to moments of stress. Our method involves the generation of synthetic sequence data through Generative Adversarial Networks (GANs), coupled with the implementation of Differential Privacy (DP) safeguards for protecting patient information during model training. To ensure the integrity of our synthetic data, we employ a range of quality assessments and monitor the plausibility between synthetic and original data. To test the usefulness, we create private machine learning models on a commonly used, albeit small, stress detection dataset, exploring strategies for enhancing the existing data foundation with our synthetic data. Through our GAN-based augmentation methods, we observe improvements in model performance, both in non-private (0.45% F1) and private (11.90-15.48% F1) training scenarios. We underline the potential of differentially private synthetic data in optimizing utility-privacy trade-offs, especially with limited availability of real training samples.

Title: Generative Video Diffusion for Unseen Cross-Domain Video Moment Retrieval

• Authors: Dezhao Luo, Jiabo Huang, Shaogang Gong, Hailin Jin, Yang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13329
• Pdf URL: https://arxiv.org/pdf/2401.13329
• Copy Paste: [[2401.13329]] Generative Video Diffusion for Unseen Cross-Domain Video Moment Retrieval(https://arxiv.org/abs/2401.13329)
• Keywords: diffusion, generative
• Abstract: Video Moment Retrieval (VMR) requires precise modelling of fine-grained moment-text associations to capture intricate visual-language relationships. Due to the lack of a diverse and generalisable VMR dataset to facilitate learning scalable moment-text associations, existing methods resort to joint training on both source and target domain videos for cross-domain applications. Meanwhile, recent developments in vision-language multimodal models pre-trained on large-scale image-text and/or video-text pairs are only based on coarse associations (weakly labelled). They are inadequate to provide fine-grained moment-text correlations required for cross-domain VMR. In this work, we solve the problem of unseen cross-domain VMR, where certain visual and textual concepts do not overlap across domains, by only utilising target domain sentences (text prompts) without accessing their videos. To that end, we explore generative video diffusion for fine-grained editing of source videos controlled by the target sentences, enabling us to simulate target domain videos. We address two problems in video editing for optimising unseen domain VMR: (1) generation of high-quality simulation videos of different moments with subtle distinctions, (2) selection of simulation videos that complement existing source training videos without introducing harmful noise or unnecessary repetitions. On the first problem, we formulate a two-stage video diffusion generation controlled simultaneously by (1) the original video structure of a source video, (2) subject specifics, and (3) a target sentence prompt. This ensures fine-grained variations between video moments. On the second problem, we introduce a hybrid selection mechanism that combines two quantitative metrics for noise filtering and one qualitative metric for leveraging VMR prediction on simulation video selection.

Title: UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion

• Authors: Wei Li, Xue Xu, Jiachen Liu, Xinyan Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13388
• Pdf URL: https://arxiv.org/pdf/2401.13388
• Copy Paste: [[2401.13388]] UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion(https://arxiv.org/abs/2401.13388)
• Keywords: diffusion
• Abstract: Existing text-to-image diffusion models primarily generate images from text prompts. However, the inherent conciseness of textual descriptions poses challenges in faithfully synthesizing images with intricate details, such as specific entities or scenes. This paper presents \textbf{UNIMO-G}, a simple multimodal conditional diffusion framework that operates on multimodal prompts with interleaved textual and visual inputs, which demonstrates a unified ability for both text-driven and subject-driven image generation. UNIMO-G comprises two core components: a Multimodal Large Language Model (MLLM) for encoding multimodal prompts, and a conditional denoising diffusion network for generating images based on the encoded multimodal input. We leverage a two-stage training strategy to effectively train the framework: firstly pre-training on large-scale text-image pairs to develop conditional image generation capabilities, and then instruction tuning with multimodal prompts to achieve unified image generation proficiency. A well-designed data processing pipeline involving language grounding and image segmentation is employed to construct multi-modal prompts. UNIMO-G excels in both text-to-image generation and zero-shot subject-driven synthesis, and is notably effective in generating high-fidelity images from complex multimodal prompts involving multiple image entities.

Title: Generative Human Motion Stylization in Latent Space

• Authors: Chuan Guo, Yuxuan Mu, Xinxin Zuo, Peng Dai, Youliang Yan, Juwei Lu, Li Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13505
• Pdf URL: https://arxiv.org/pdf/2401.13505
• Copy Paste: [[2401.13505]] Generative Human Motion Stylization in Latent Space(https://arxiv.org/abs/2401.13505)
• Keywords: generative
• Abstract: Human motion stylization aims to revise the style of an input motion while keeping its content unaltered. Unlike existing works that operate directly in pose space, we leverage the latent space of pretrained autoencoders as a more expressive and robust representation for motion extraction and infusion. Building upon this, we present a novel generative model that produces diverse stylization results of a single motion (latent) code. During training, a motion code is decomposed into two coding components: a deterministic content code, and a probabilistic style code adhering to a prior distribution; then a generator massages the random combination of content and style codes to reconstruct the corresponding motion codes. Our approach is versatile, allowing the learning of probabilistic style space from either style labeled or unlabeled motions, providing notable flexibility in stylization as well. In inference, users can opt to stylize a motion using style cues from a reference motion or a label. Even in the absence of explicit style input, our model facilitates novel re-stylization by sampling from the unconditional style prior distribution. Experimental results show that our proposed stylization models, despite their lightweight design, outperform the state-of-the-arts in style reeanactment, content preservation, and generalization across various applications and settings. Project Page: https://yxmu.foo/GenMoStyle

Title: SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation

• Authors: Dong Zhang, Xin Zhang, Jun Zhan, Shimin Li, Yaqian Zhou, Xipeng Qiu
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.13527
• Pdf URL: https://arxiv.org/pdf/2401.13527
• Copy Paste: [[2401.13527]] SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation(https://arxiv.org/abs/2401.13527)
• Keywords: in-context
• Abstract: Benefiting from effective speech modeling, current Speech Large Language Models (SLLMs) have demonstrated exceptional capabilities in in-context speech generation and efficient generalization to unseen speakers. However, the prevailing information modeling process is encumbered by certain redundancies, leading to inefficiencies in speech generation. We propose Chain-of-Information Generation (CoIG), a method for decoupling semantic and perceptual information in large-scale speech generation. Building on this, we develop SpeechGPT-Gen, an 8-billion-parameter SLLM efficient in semantic and perceptual information modeling. It comprises an autoregressive model based on LLM for semantic information modeling and a non-autoregressive model employing flow matching for perceptual information modeling. Additionally, we introduce the novel approach of infusing semantic information into the prior distribution to enhance the efficiency of flow matching. Extensive experimental results demonstrate that SpeechGPT-Gen markedly excels in zero-shot text-to-speech, zero-shot voice conversion, and speech-to-speech dialogue, underscoring CoIG's remarkable proficiency in capturing and modeling speech's semantic and perceptual dimensions. Code and models are available at https://github.com/0nutation/SpeechGPT.

Title: Interleaving One-Class and Weakly-Supervised Models with Adaptive Thresholding for Unsupervised Video Anomaly Detection

• Authors: Yongwei Nie, Hao Huang, Chengjiang Long, Qing Zhang, Pradipta Maji, Hongmin Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13551
• Pdf URL: https://arxiv.org/pdf/2401.13551
• Copy Paste: [[2401.13551]] Interleaving One-Class and Weakly-Supervised Models with Adaptive Thresholding for Unsupervised Video Anomaly Detection(https://arxiv.org/abs/2401.13551)
• Keywords: anomaly
• Abstract: Without human annotations, a typical Unsupervised Video Anomaly Detection (UVAD) method needs to train two models that generate pseudo labels for each other. In previous work, the two models are closely entangled with each other, and it is not known how to upgrade their method without modifying their training framework significantly. Second, previous work usually adopts fixed thresholding to obtain pseudo labels, however the user-specified threshold is not reliable which inevitably introduces errors into the training process. To alleviate these two problems, we propose a novel interleaved framework that alternately trains a One-Class Classification (OCC) model and a Weakly-Supervised (WS) model for UVAD. The OCC or WS models in our method can be easily replaced with other OCC or WS models, which facilitates our method to upgrade with the most recent developments in both fields. For handling the fixed thresholding problem, we break through the conventional cognitive boundary and propose a weighted OCC model that can be trained on both normal and abnormal data. We also propose an adaptive mechanism for automatically finding the optimal threshold for the WS model in a loose to strict manner. Experiments demonstrate that the proposed UVAD method outperforms previous approaches.

Title: Benchmarking the Fairness of Image Upsampling Methods

• Authors: Mike Laszkiewicz, Imant Daunhawer, Julia E. Vogt, Asja Fischer, Johannes Lederer
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13555
• Pdf URL: https://arxiv.org/pdf/2401.13555
• Copy Paste: [[2401.13555]] Benchmarking the Fairness of Image Upsampling Methods(https://arxiv.org/abs/2401.13555)
• Keywords: generative
• Abstract: Recent years have witnessed a rapid development of deep generative models for creating synthetic media, such as images and videos. While the practical applications of these models in everyday tasks are enticing, it is crucial to assess the inherent risks regarding their fairness. In this work, we introduce a comprehensive framework for benchmarking the performance and fairness of conditional generative models. We develop a set of metrics$\unicode{x2013}$inspired by their supervised fairness counterparts$\unicode{x2013}$to evaluate the models on their fairness and diversity. Focusing on the specific application of image upsampling, we create a benchmark covering a wide variety of modern upsampling methods. As part of the benchmark, we introduce UnfairFace, a subset of FairFace that replicates the racial distribution of common large-scale face datasets. Our empirical study highlights the importance of using an unbiased training set and reveals variations in how the algorithms respond to dataset imbalances. Alarmingly, we find that none of the considered methods produces statistically fair and diverse results.

Title: Towards Efficient and Effective Deep Clustering with Dynamic Grouping and Prototype Aggregation

• Authors: Haixin Zhang, Dong Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13581
• Pdf URL: https://arxiv.org/pdf/2401.13581
• Copy Paste: [[2401.13581]] Towards Efficient and Effective Deep Clustering with Dynamic Grouping and Prototype Aggregation(https://arxiv.org/abs/2401.13581)
• Keywords: self-supervised
• Abstract: Previous contrastive deep clustering methods mostly focus on instance-level information while overlooking the member relationship within groups/clusters, which may significantly undermine their representation learning and clustering capability. Recently, some group-contrastive methods have been developed, which, however, typically rely on the samples of the entire dataset to obtain pseudo labels and lack the ability to efficiently update the group assignments in a batch-wise manner. To tackle these critical issues, we present a novel end-to-end deep clustering framework with dynamic grouping and prototype aggregation, termed as DigPro. Specifically, the proposed dynamic grouping extends contrastive learning from instance-level to group-level, which is effective and efficient for timely updating groups. Meanwhile, we perform contrastive learning on prototypes in a spherical feature space, termed as prototype aggregation, which aims to maximize the inter-cluster distance. Notably, with an expectation-maximization framework, DigPro simultaneously takes advantage of compact intra-cluster connections, well-separated clusters, and efficient group updating during the self-supervised training. Extensive experiments on six image benchmarks demonstrate the superior performance of our approach over the state-of-the-art. Code is available at https://github.com/Regan-Zhang/DigPro.

Title: DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning

• Authors: Xinghao Wang, Junliang He, Pengyu Wang, Yunhua Zhou, Tianxiang Sun, Xipeng Qiu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13621
• Pdf URL: https://arxiv.org/pdf/2401.13621
• Copy Paste: [[2401.13621]] DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning(https://arxiv.org/abs/2401.13621)
• Keywords: self-supervised
• Abstract: Contrastive-learning-based methods have dominated sentence representation learning. These methods regularize the representation space by pulling similar sentence representations closer and pushing away the dissimilar ones and have been proven effective in various NLP tasks, e.g., semantic textual similarity (STS) tasks. However, it is challenging for these methods to learn fine-grained semantics as they only learn from the inter-sentence perspective, i.e., their supervision signal comes from the relationship between data samples. In this work, we propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective. By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form. Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks, standing up well in comparison to contrastive-learning-based methods. Notably, the proposed intra-sentence denoising objective complements existing inter-sentence contrastive methodologies and can be integrated with them to further enhance performance. Our code is available at https://github.com/xinghaow99/DenoSent.

Title: Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild

• Authors: Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang, Jingwen He, Yu Qiao, Chao Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13627
• Pdf URL: https://arxiv.org/pdf/2401.13627
• Copy Paste: [[2401.13627]] Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild(https://arxiv.org/abs/2401.13627)
• Keywords: generative
• Abstract: We introduce SUPIR (Scaling-UP Image Restoration), a groundbreaking image restoration method that harnesses generative prior and the power of model scaling up. Leveraging multi-modal techniques and advanced generative prior, SUPIR marks a significant advance in intelligent and realistic image restoration. As a pivotal catalyst within SUPIR, model scaling dramatically enhances its capabilities and demonstrates new potential for image restoration. We collect a dataset comprising 20 million high-resolution, high-quality images for model training, each enriched with descriptive text annotations. SUPIR provides the capability to restore images guided by textual prompts, broadening its application scope and potential. Moreover, we introduce negative-quality prompts to further improve perceptual quality. We also develop a restoration-guided sampling method to suppress the fidelity issue encountered in generative-based restoration. Experiments demonstrate SUPIR's exceptional restoration effects and its novel capacity to manipulate restoration through textual prompts.