2024-04-08

Title: RL for Consistency Models: Faster Reward Guided Text-to-Image Generation

• Authors: Owen Oertell, Jonathan D. Chang, Yiyi Zhang, Kianté Brantley, Wen Sun
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03673
• Pdf URL: https://arxiv.org/pdf/2404.03673
• Copy Paste: [[2404.03673]] RL for Consistency Models: Faster Reward Guided Text-to-Image Generation(https://arxiv.org/abs/2404.03673)
• Keywords: diffusion, generative
• Abstract: Reinforcement learning (RL) has improved guided image generation with diffusion models by directly optimizing rewards that capture image quality, aesthetics, and instruction following capabilities. However, the resulting generative policies inherit the same iterative sampling process of diffusion models that causes slow generation. To overcome this limitation, consistency models proposed learning a new class of generative models that directly map noise to data, resulting in a model that can generate an image in as few as one sampling iteration. In this work, to optimize text-to-image generative models for task specific rewards and enable fast training and inference, we propose a framework for fine-tuning consistency models via RL. Our framework, called Reinforcement Learning for Consistency Model (RLCM), frames the iterative inference process of a consistency model as an RL procedure. RLCM improves upon RL fine-tuned diffusion models on text-to-image generation capabilities and trades computation during inference time for sample quality. Experimentally, we show that RLCM can adapt text-to-image consistency models to objectives that are challenging to express with prompting, such as image compressibility, and those derived from human feedback, such as aesthetic quality. Comparing to RL finetuned diffusion models, RLCM trains significantly faster, improves the quality of the generation measured under the reward objectives, and speeds up the inference procedure by generating high quality images with as few as two inference steps. Our code is available at https://rlcm.owenoertell.com

Title: SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection

• Authors: Bradley P. Allen, Fina Polat, Paul Groth
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03732
• Pdf URL: https://arxiv.org/pdf/2404.03732
• Copy Paste: [[2404.03732]] SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection(https://arxiv.org/abs/2404.03732)
• Keywords: in-context
• Abstract: We describe the University of Amsterdam Intelligent Data Engineering Lab team's entry for the SemEval-2024 Task 6 competition. The SHROOM-INDElab system builds on previous work on using prompt programming and in-context learning with large language models (LLMs) to build classifiers for hallucination detection, and extends that work through the incorporation of context-specific definition of task, role, and target concept, and automated generation of examples for use in a few-shot prompting approach. The resulting system achieved fourth-best and sixth-best performance in the model-agnostic track and model-aware tracks for Task 6, respectively, and evaluation using the validation sets showed that the system's classification decisions were consistent with those of the crowd-sourced human labellers. We further found that a zero-shot approach provided better accuracy than a few-shot approach using automatically generated examples. Code for the system described in this paper is available on Github.

Title: SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer

• Authors: Zijie Wu, Chaohui Yu, Yanqin Jiang, Chenjie Cao, Fan Wang, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03736
• Pdf URL: https://arxiv.org/pdf/2404.03736
• Copy Paste: [[2404.03736]] SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer(https://arxiv.org/abs/2404.03736)
• Keywords: generative
• Abstract: Recent advances in 2D/3D generative models enable the generation of dynamic 3D objects from a single-view video. Existing approaches utilize score distillation sampling to form the dynamic scene as dynamic NeRF or dense 3D Gaussians. However, these methods struggle to strike a balance among reference view alignment, spatio-temporal consistency, and motion fidelity under single-view conditions due to the implicit nature of NeRF or the intricate dense Gaussian motion prediction. To address these issues, this paper proposes an efficient, sparse-controlled video-to-4D framework named SC4D, that decouples motion and appearance to achieve superior video-to-4D generation. Moreover, we introduce Adaptive Gaussian (AG) initialization and Gaussian Alignment (GA) loss to mitigate shape degeneration issue, ensuring the fidelity of the learned motion and shape. Comprehensive experimental results demonstrate that our method surpasses existing methods in both quality and efficiency. In addition, facilitated by the disentangled modeling of motion and appearance of SC4D, we devise a novel application that seamlessly transfers the learned motion onto a diverse array of 4D entities according to textual descriptions.

Title: Test Time Training for Industrial Anomaly Segmentation

• Authors: Alex Costanzino, Pierluigi Zama Ramirez, Mirko Del Moro, Agostino Aiezzo, Giuseppe Lisanti, Samuele Salti, Luigi Di Stefano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03743
• Pdf URL: https://arxiv.org/pdf/2404.03743
• Copy Paste: [[2404.03743]] Test Time Training for Industrial Anomaly Segmentation(https://arxiv.org/abs/2404.03743)
• Keywords: anomaly
• Abstract: Anomaly Detection and Segmentation (AD&S) is crucial for industrial quality control. While existing methods excel in generating anomaly scores for each pixel, practical applications require producing a binary segmentation to identify anomalies. Due to the absence of labeled anomalies in many real scenarios, standard practices binarize these maps based on some statistics derived from a validation set containing only nominal samples, resulting in poor segmentation performance. This paper addresses this problem by proposing a test time training strategy to improve the segmentation performance. Indeed, at test time, we can extract rich features directly from anomalous samples to train a classifier that can discriminate defects effectively. Our general approach can work downstream to any AD&S method that provides an anomaly score map as output, even in multimodal settings. We demonstrate the effectiveness of our approach over baselines through extensive experimentation and evaluation on MVTec AD and MVTec 3D-AD.

Title: Quantifying Uncertainty in Motion Prediction with Variational Bayesian Mixture

• Authors: Juanwu Lu, Can Cui, Yunsheng Ma, Aniket Bera, Ziran Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.03789
• Pdf URL: https://arxiv.org/pdf/2404.03789
• Copy Paste: [[2404.03789]] Quantifying Uncertainty in Motion Prediction with Variational Bayesian Mixture(https://arxiv.org/abs/2404.03789)
• Keywords: generative
• Abstract: Safety and robustness are crucial factors in developing trustworthy autonomous vehicles. One essential aspect of addressing these factors is to equip vehicles with the capability to predict future trajectories for all moving objects in the surroundings and quantify prediction uncertainties. In this paper, we propose the Sequential Neural Variational Agent (SeNeVA), a generative model that describes the distribution of future trajectories for a single moving object. Our approach can distinguish Out-of-Distribution data while quantifying uncertainty and achieving competitive performance compared to state-of-the-art methods on the Argoverse 2 and INTERACTION datasets. Specifically, a 0.446 meters minimum Final Displacement Error, a 0.203 meters minimum Average Displacement Error, and a 5.35% Miss Rate are achieved on the INTERACTION test set. Extensive qualitative and quantitative analysis is also provided to evaluate the proposed model. Our open-source code is available at https://github.com/PurdueDigitalTwin/seneva.

Title: Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models

• Authors: Gihyun Kwon, Simon Jenni, Dingzeyu Li, Joon-Young Lee, Jong Chul Ye, Fabian Caba Heilbron
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.03913
• Pdf URL: https://arxiv.org/pdf/2404.03913
• Copy Paste: [[2404.03913]] Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models(https://arxiv.org/abs/2404.03913)
• Keywords: diffusion
• Abstract: While there has been significant progress in customizing text-to-image generation models, generating images that combine multiple personalized concepts remains challenging. In this work, we introduce Concept Weaver, a method for composing customized text-to-image diffusion models at inference time. Specifically, the method breaks the process into two steps: creating a template image aligned with the semantics of input prompts, and then personalizing the template using a concept fusion strategy. The fusion strategy incorporates the appearance of the target concepts into the template image while retaining its structural details. The results indicate that our method can generate multiple custom concepts with higher identity fidelity compared to alternative approaches. Furthermore, the method is shown to seamlessly handle more than two concepts and closely follow the semantic meaning of the input prompt without blending appearances across different subjects.

Title: Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models

• Authors: Bowen Zhang, Kehua Chang, Chunping Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03921
• Pdf URL: https://arxiv.org/pdf/2404.03921
• Copy Paste: [[2404.03921]] Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models(https://arxiv.org/abs/2404.03921)
• Keywords: generative
• Abstract: Sentence Embedding stands as a fundamental task within the realm of Natural Language Processing, finding extensive application in search engines, expert systems, and question-and-answer platforms. With the continuous evolution of large language models such as LLaMA and Mistral, research on sentence embedding has recently achieved notable breakthroughs. However, these advancements mainly pertain to fine-tuning scenarios, leaving explorations into computationally efficient direct inference methods for sentence representation in a nascent stage. This paper endeavors to bridge this research gap. Through comprehensive experimentation, we challenge the widely held belief in the necessity of an Explicit One-word Limitation for deriving sentence embeddings from Pre-trained Language Models (PLMs). We demonstrate that this approach, while beneficial for generative models under direct inference scenario, is not imperative for discriminative models or the fine-tuning of generative PLMs. This discovery sheds new light on the design of manual templates in future studies. Building upon this insight, we propose two innovative prompt engineering techniques capable of further enhancing the expressive power of PLMs' raw embeddings: Pretended Chain of Thought and Knowledge Enhancement. We confirm their effectiveness across various PLM types and provide a detailed exploration of the underlying factors contributing to their success.

Title: Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solving

• Authors: Gulsum Yigit, Mehmet Fatih Amasyali
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03938
• Pdf URL: https://arxiv.org/pdf/2404.03938
• Copy Paste: [[2404.03938]] Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solving(https://arxiv.org/abs/2404.03938)
• Keywords: in-context
• Abstract: Math Word Problem (MWP) solving presents a challenging task in Natural Language Processing (NLP). This study aims to provide MWP solvers with a more diverse training set, ultimately improving their ability to solve various math problems. We propose several methods for data augmentation by modifying the problem texts and equations, such as synonym replacement, rule-based: question replacement, and rule based: reversing question methodologies over two English MWP datasets. This study extends by introducing a new in-context learning augmentation method, employing the Llama-7b language model. This approach involves instruction-based prompting for rephrasing the math problem texts. Performance evaluations are conducted on 9 baseline models, revealing that augmentation methods outperform baseline models. Moreover, concatenating examples generated by various augmentation methods further improves performance.

Title: SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials

• Authors: Mathilde Aguiar, Pierre Zweigenbaum, Nona Naderi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.03977
• Pdf URL: https://arxiv.org/pdf/2404.03977
• Copy Paste: [[2404.03977]] SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials(https://arxiv.org/abs/2404.03977)
• Keywords: generative
• Abstract: This paper describes our submission to Task 2 of SemEval-2024: Safe Biomedical Natural Language Inference for Clinical Trials. The Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) consists of a Textual Entailment (TE) task focused on the evaluation of the consistency and faithfulness of Natural Language Inference (NLI) models applied to Clinical Trial Reports (CTR). We test 2 distinct approaches, one based on finetuning and ensembling Masked Language Models and the other based on prompting Large Language Models using templates, in particular, using Chain-Of-Thought and Contrastive Chain-Of-Thought. Prompting Flan-T5-large in a 2-shot setting leads to our best system that achieves 0.57 F1 score, 0.64 Faithfulness, and 0.56 Consistency.

Title: Finsler-Laplace-Beltrami Operators with Application to Shape Analysis

• Authors: Simon Weber, Thomas Dagès, Maolin Gao, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.03999
• Pdf URL: https://arxiv.org/pdf/2404.03999
• Copy Paste: [[2404.03999]] Finsler-Laplace-Beltrami Operators with Application to Shape Analysis(https://arxiv.org/abs/2404.03999)
• Keywords: diffusion
• Abstract: The Laplace-Beltrami operator (LBO) emerges from studying manifolds equipped with a Riemannian metric. It is often called the Swiss army knife of geometry processing as it allows to capture intrinsic shape information and gives rise to heat diffusion, geodesic distances, and a multitude of shape descriptors. It also plays a central role in geometric deep learning. In this work, we explore Finsler manifolds as a generalization of Riemannian manifolds. We revisit the Finsler heat equation and derive a Finsler heat kernel and a Finsler-Laplace-Beltrami Operator (FLBO): a novel theoretically justified anisotropic Laplace-Beltrami operator (ALBO). In experimental evaluations we demonstrate that the proposed FLBO is a valuable alternative to the traditional Riemannian-based LBO and ALBOs for spatial filtering and shape correspondence estimation. We hope that the proposed Finsler heat kernel and the FLBO will inspire further exploration of Finsler geometry in the computer vision community.

Title: InstructHumans: Editing Animated 3D Human Textures with Instructions

• Authors: Jiayin Zhu, Linlin Yang, Angela Yao
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.04037
• Pdf URL: https://arxiv.org/pdf/2404.04037
• Copy Paste: [[2404.04037]] InstructHumans: Editing Animated 3D Human Textures with Instructions(https://arxiv.org/abs/2404.04037)
• Keywords: diffusion, generative
• Abstract: We present InstructHumans, a novel framework for instruction-driven 3D human texture editing. Existing text-based editing methods use Score Distillation Sampling (SDS) to distill guidance from generative models. This work shows that naively using such scores is harmful to editing as they destroy consistency with the source avatar. Instead, we propose an alternate SDS for Editing (SDS-E) that selectively incorporates subterms of SDS across diffusion timesteps. We further enhance SDS-E with spatial smoothness regularization and gradient-based viewpoint sampling to achieve high-quality edits with sharp and high-fidelity detailing. InstructHumans significantly outperforms existing 3D editing methods, consistent with the initial avatar while faithful to the textual instructions. Project page: https://jyzhu.top/instruct-humans .

Title: Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation

• Authors: Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, Hai Huang
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.04057
• Pdf URL: https://arxiv.org/pdf/2404.04057
• Copy Paste: [[2404.04057]] Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation(https://arxiv.org/abs/2404.04057)
• Keywords: diffusion, generative
• Abstract: We introduce Score identity Distillation (SiD), an innovative data-free method that distills the generative capabilities of pretrained diffusion models into a single-step generator. SiD not only facilitates an exponentially fast reduction in Fr\'echet inception distance (FID) during distillation but also approaches or even exceeds the FID performance of the original teacher diffusion models. By reformulating forward diffusion processes as semi-implicit distributions, we leverage three score-related identities to create an innovative loss mechanism. This mechanism achieves rapid FID reduction by training the generator using its own synthesized images, eliminating the need for real data or reverse-diffusion-based generation, all accomplished within significantly shortened generation time. Upon evaluation across four benchmark datasets, the SiD algorithm demonstrates high iteration efficiency during distillation and surpasses competing distillation approaches, whether they are one-step or few-step, data-free, or dependent on training data, in terms of generation quality. This achievement not only redefines the benchmarks for efficiency and effectiveness in diffusion distillation but also in the broader field of diffusion-based generation. Our PyTorch implementation will be publicly accessible on GitHub.

Title: Fusing Dictionary Learning and Support Vector Machines for Unsupervised Anomaly Detection

• Authors: Paul Irofti, Iulian-Andrei Hîji, Andrei Pătraşcu, Nicolae Cleju
• Subjects: cs.LG, cs.CR, math.NA
• Abstract URL: https://arxiv.org/abs/2404.04064
• Pdf URL: https://arxiv.org/pdf/2404.04064
• Copy Paste: [[2404.04064]] Fusing Dictionary Learning and Support Vector Machines for Unsupervised Anomaly Detection(https://arxiv.org/abs/2404.04064)
• Keywords: anomaly
• Abstract: We study in this paper the improvement of one-class support vector machines (OC-SVM) through sparse representation techniques for unsupervised anomaly detection. As Dictionary Learning (DL) became recently a common analysis technique that reveals hidden sparse patterns of data, our approach uses this insight to endow unsupervised detection with more control on pattern finding and dimensions. We introduce a new anomaly detection model that unifies the OC-SVM and DL residual functions into a single composite objective, subsequently solved through K-SVD-type iterative algorithms. A closed-form of the alternating K-SVD iteration is explicitly derived for the new composite model and practical implementable schemes are discussed. The standard DL model is adapted for the Dictionary Pair Learning (DPL) context, where the usual sparsity constraints are naturally eliminated. Finally, we extend both objectives to the more general setting that allows the use of kernel functions. The empirical convergence properties of the resulting algorithms are provided and an in-depth analysis of their parametrization is performed while also demonstrating their numerical performance in comparison with existing methods.

Title: Dynamic Prompt Optimizing for Text-to-Image Generation

• Authors: Wenyi Mo, Tianyu Zhang, Yalong Bai, Bing Su, Ji-Rong Wen, Qing Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04095
• Pdf URL: https://arxiv.org/pdf/2404.04095
• Copy Paste: [[2404.04095]] Dynamic Prompt Optimizing for Text-to-Image Generation(https://arxiv.org/abs/2404.04095)
• Keywords: diffusion, generative
• Abstract: Text-to-image generative models, specifically those based on diffusion models like Imagen and Stable Diffusion, have made substantial advancements. Recently, there has been a surge of interest in the delicate refinement of text prompts. Users assign weights or alter the injection time steps of certain words in the text prompts to improve the quality of generated images. However, the success of fine-control prompts depends on the accuracy of the text prompts and the careful selection of weights and time steps, which requires significant manual intervention. To address this, we introduce the \textbf{P}rompt \textbf{A}uto-\textbf{E}diting (PAE) method. Besides refining the original prompts for image generation, we further employ an online reinforcement learning strategy to explore the weights and injection time steps of each word, leading to the dynamic fine-control prompts. The reward function during training encourages the model to consider aesthetic score, semantic consistency, and user preferences. Experimental results demonstrate that our proposed method effectively improves the original prompts, generating visually more appealing images while maintaining semantic alignment. Code is available at https://github.com/Mowenyii/PAE.

Title: 3D Facial Expressions through Analysis-by-Neural-Synthesis

• Authors: George Retsinas, Panagiotis P. Filntisis, Radek Danecek, Victoria F. Abrevaya, Anastasios Roussos, Timo Bolkart, Petros Maragos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04104
• Pdf URL: https://arxiv.org/pdf/2404.04104
• Copy Paste: [[2404.04104]] 3D Facial Expressions through Analysis-by-Neural-Synthesis(https://arxiv.org/abs/2404.04104)
• Keywords: self-supervised
• Abstract: While existing methods for 3D face reconstruction from in-the-wild images excel at recovering the overall face shape, they commonly miss subtle, extreme, asymmetric, or rarely observed expressions. We improve upon these methods with SMIRK (Spatial Modeling for Image-based Reconstruction of Kinesics), which faithfully reconstructs expressive 3D faces from images. We identify two key limitations in existing methods: shortcomings in their self-supervised training formulation, and a lack of expression diversity in the training images. For training, most methods employ differentiable rendering to compare a predicted face mesh with the input image, along with a plethora of additional loss functions. This differentiable rendering loss not only has to provide supervision to optimize for 3D face geometry, camera, albedo, and lighting, which is an ill-posed optimization problem, but the domain gap between rendering and input image further hinders the learning process. Instead, SMIRK replaces the differentiable rendering with a neural rendering module that, given the rendered predicted mesh geometry, and sparsely sampled pixels of the input image, generates a face image. As the neural rendering gets color information from sampled image pixels, supervising with neural rendering-based reconstruction loss can focus solely on the geometry. Further, it enables us to generate images of the input identity with varying expressions while training. These are then utilized as input to the reconstruction model and used as supervision with ground truth geometry. This effectively augments the training data and enhances the generalization for diverse expressions. Our qualitative, quantitative and particularly our perceptual evaluations demonstrate that SMIRK achieves the new state-of-the art performance on accurate expression reconstruction. Project webpage: https://georgeretsi.github.io/smirk/.

Title: No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

• Authors: Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H.S. Torr, Adel Bibi, Samuel Albanie, Matthias Bethge
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04125
• Pdf URL: https://arxiv.org/pdf/2404.04125
• Copy Paste: [[2404.04125]] No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance(https://arxiv.org/abs/2404.04125)
• Keywords: diffusion
• Abstract: Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation. However, it is unclear how meaningful the notion of "zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during "zero-shot" evaluation. In this work, we ask: How is the performance of multimodal models on downstream concepts influenced by the frequency of these concepts in their pretraining datasets? We comprehensively investigate this question across 34 models and five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M, LAION-Aesthetics), generating over 300GB of data artifacts. We consistently find that, far from exhibiting "zero-shot" generalization, multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets, and testing on purely synthetic data distributions. Furthermore, upon benchmarking models on long-tailed data sampled based on our analysis, we demonstrate that multimodal models across the board perform poorly. We contribute this long-tail test set as the "Let it Wag!" benchmark to further research in this direction. Taken together, our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.

Title: player2vec: A Language Modeling Approach to Understand Player Behavior in Games

• Authors: Tianze Wang, Maryam Honari-Jahromi, Styliani Katsarou, Olga Mikheeva, Theodoros Panagiotakopoulos, Sahar Asadi, Oleg Smirnov
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04234
• Pdf URL: https://arxiv.org/pdf/2404.04234
• Copy Paste: [[2404.04234]] player2vec: A Language Modeling Approach to Understand Player Behavior in Games(https://arxiv.org/abs/2404.04234)
• Keywords: self-supervised
• Abstract: Methods for learning latent user representations from historical behavior logs have gained traction for recommendation tasks in e-commerce, content streaming, and other settings. However, this area still remains relatively underexplored in video and mobile gaming contexts. In this work, we present a novel method for overcoming this limitation by extending a long-range Transformer model from the natural language processing domain to player behavior data. We discuss specifics of behavior tracking in games and propose preprocessing and tokenization approaches by viewing in-game events in an analogous way to words in sentences, thus enabling learning player representations in a self-supervised manner in the absence of ground-truth annotations. We experimentally demonstrate the efficacy of the proposed approach in fitting the distribution of behavior events by evaluating intrinsic language modeling metrics. Furthermore, we qualitatively analyze the emerging structure of the learned embedding space and show its value for generating insights into behavior patterns to inform downstream applications.

Title: Dynamic Conditional Optimal Transport through Simulation-Free Flows

• Authors: Gavin Kerrigan, Giosue Migliorini, Padhraic Smyth
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04240
• Pdf URL: https://arxiv.org/pdf/2404.04240
• Copy Paste: [[2404.04240]] Dynamic Conditional Optimal Transport through Simulation-Free Flows(https://arxiv.org/abs/2404.04240)
• Keywords: generative
• Abstract: We study the geometry of conditional optimal transport (COT) and prove a dynamical formulation which generalizes the Benamou-Brenier Theorem. With these tools, we propose a simulation-free flow-based method for conditional generative modeling. Our method couples an arbitrary source distribution to a specified target distribution through a triangular COT plan. We build on the framework of flow matching to train a conditional generative model by approximating the geodesic path of measures induced by this COT plan. Our theory and methods are applicable in the infinite-dimensional setting, making them well suited for inverse problems. Empirically, we demonstrate our proposed method on two image-to-image translation tasks and an infinite-dimensional Bayesian inverse problem.

Title: Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models

• Authors: Sangwon Jang, Jaehyeong Jo, Kimin Lee, Sung Ju Hwang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04243
• Pdf URL: https://arxiv.org/pdf/2404.04243
• Copy Paste: [[2404.04243]] Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models(https://arxiv.org/abs/2404.04243)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models have shown remarkable success in generating a personalized subject based on a few reference images. However, current methods struggle with handling multiple subjects simultaneously, often resulting in mixed identities with combined attributes from different subjects. In this work, we present MuDI, a novel framework that enables multi-subject personalization by effectively decoupling identities from multiple subjects. Our main idea is to utilize segmented subjects generated by the Segment Anything Model for both training and inference, as a form of data augmentation for training and initialization for the generation process. Our experiments demonstrate that MuDI can produce high-quality personalized images without identity mixing, even for highly similar subjects as shown in Figure 1. In human evaluation, MuDI shows twice as many successes for personalizing multiple subjects without identity mixing over existing baselines and is preferred over 70% compared to the strongest baseline. More results are available at https://mudi-t2i.github.io/.

Title: Watermark-based Detection and Attribution of AI-Generated Content

• Authors: Zhengyuan Jiang, Moyang Guo, Yuepeng Hu, Neil Zhenqiang Gong
• Subjects: cs.CR, cs.AI, cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04254
• Pdf URL: https://arxiv.org/pdf/2404.04254
• Copy Paste: [[2404.04254]] Watermark-based Detection and Attribution of AI-Generated Content(https://arxiv.org/abs/2404.04254)
• Keywords: generative
• Abstract: Several companies--such as Google, Microsoft, and OpenAI--have deployed techniques to watermark AI-generated content to enable proactive detection. However, existing literature mainly focuses on user-agnostic detection. Attribution aims to further trace back the user of a generative-AI service who generated a given content detected as AI-generated. Despite its growing importance, attribution is largely unexplored. In this work, we aim to bridge this gap by providing the first systematic study on watermark-based, user-aware detection and attribution of AI-generated content. Specifically, we theoretically study the detection and attribution performance via rigorous probabilistic analysis. Moreover, we develop an efficient algorithm to select watermarks for the users to enhance attribution performance. Both our theoretical and empirical results show that watermark-based detection and attribution inherit the accuracy and (non-)robustness properties of the watermarking method.