2024-02-05

Title: Self-Supervised Contrastive Pre-Training for Multivariate Point Processes

• Authors: Xiao Shou, Dharmashankar Subramanian, Debarun Bhattacharjya, Tian Gao, Kristin P. Bennet
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00987
• Pdf URL: https://arxiv.org/pdf/2402.00987
• Copy Paste: [[2402.00987]] Self-Supervised Contrastive Pre-Training for Multivariate Point Processes(https://arxiv.org/abs/2402.00987)
• Keywords: self-supervised, foundation model
• Abstract: Self-supervision is one of the hallmarks of representation learning in the increasingly popular suite of foundation models including large language models such as BERT and GPT-3, but it has not been pursued in the context of multivariate event streams, to the best of our knowledge. We introduce a new paradigm for self-supervised learning for multivariate point processes using a transformer encoder. Specifically, we design a novel pre-training strategy for the encoder where we not only mask random event epochs but also insert randomly sampled "void" epochs where an event does not occur; this differs from the typical discrete-time pretext tasks such as word-masking in BERT but expands the effectiveness of masking to better capture continuous-time dynamics. To improve downstream tasks, we introduce a contrasting module that compares real events to simulated void instances. The pre-trained model can subsequently be fine-tuned on a potentially much smaller event dataset, similar conceptually to the typical transfer of popular pre-trained language models. We demonstrate the effectiveness of our proposed paradigm on the next-event prediction task using synthetic datasets and 3 real applications, observing a relative performance boost of as high as up to 20% compared to state-of-the-art models.

Title: A Cost-Efficient Approach for Creating Virtual Fitting Room using Generative Adversarial Networks (GANs)

• Authors: Kirolos Attallah, Girgis Zaky, Nourhan Abdelrhim, Kyrillos Botros, Amjad Dife, Nermin Negied
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00994
• Pdf URL: https://arxiv.org/pdf/2402.00994
• Copy Paste: [[2402.00994]] A Cost-Efficient Approach for Creating Virtual Fitting Room using Generative Adversarial Networks (GANs)(https://arxiv.org/abs/2402.00994)
• Keywords: generative
• Abstract: Customers all over the world want to see how the clothes fit them or not before purchasing. Therefore, customers by nature prefer brick-and-mortar clothes shopping so they can try on products before purchasing them. But after the Pandemic of COVID19 many sellers either shifted to online shopping or closed their fitting rooms which made the shopping process hesitant and doubtful. The fact that the clothes may not be suitable for their buyers after purchase led us to think about using new AI technologies to create an online platform or a virtual fitting room (VFR) in the form of a mobile application and a deployed model using a webpage that can be embedded later to any online store where they can try on any number of cloth items without physically trying them. Besides, it will save much searching time for their needs. Furthermore, it will reduce the crowding and headache in the physical shops by applying the same technology using a special type of mirror that will enable customers to try on faster. On the other hand, from business owners' perspective, this project will highly increase their online sales, besides, it will save the quality of the products by avoiding physical trials issues. The main approach used in this work is applying Generative Adversarial Networks (GANs) combined with image processing techniques to generate one output image from two input images which are the person image and the cloth image. This work achieved results that outperformed the state-of-the-art approaches found in literature.

Title: mmID: High-Resolution mmWave Imaging for Human Identification

• Authors: Sakila S. Jayaweera, Sai Deepika Regani, Yuqian Hu, Beibei Wang, K. J. Ray Liu
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.00996
• Pdf URL: https://arxiv.org/pdf/2402.00996
• Copy Paste: [[2402.00996]] mmID: High-Resolution mmWave Imaging for Human Identification(https://arxiv.org/abs/2402.00996)
• Keywords: generative
• Abstract: Achieving accurate human identification through RF imaging has been a persistent challenge, primarily attributed to the limited aperture size and its consequent impact on imaging resolution. The existing imaging solution enables tasks such as pose estimation, activity recognition, and human tracking based on deep neural networks by estimating skeleton joints. In contrast to estimating joints, this paper proposes to improve imaging resolution by estimating the human figure as a whole using conditional generative adversarial networks (cGAN). In order to reduce training complexity, we use an estimated spatial spectrum using the MUltiple SIgnal Classification (MUSIC) algorithm as input to the cGAN. Our system generates environmentally independent, high-resolution images that can extract unique physical features useful for human identification. We use a simple convolution layers-based classification network to obtain the final identification result. From the experimental results, we show that resolution of the image produced by our trained generator is high enough to enable human identification. Our finding indicates high-resolution accuracy with 5% mean silhouette difference to the Kinect device. Extensive experiments in different environments on multiple testers demonstrate that our system can achieve 93% overall test accuracy in unseen environments for static human target identification.

Title: AI-generated faces free from racial and gender stereotypes

• Authors: Nouar AlDahoul, Talal Rahwan, Yasir Zaki
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01002
• Pdf URL: https://arxiv.org/pdf/2402.01002
• Copy Paste: [[2402.01002]] AI-generated faces free from racial and gender stereotypes(https://arxiv.org/abs/2402.01002)
• Keywords: diffusion, generative
• Abstract: Text-to-image generative AI models such as Stable Diffusion are used daily by millions worldwide. However, many have raised concerns regarding how these models amplify racial and gender stereotypes. To study this phenomenon, we develop a classifier to predict the race, gender, and age group of any given face image, and show that it achieves state-of-the-art performance. Using this classifier, we quantify biases in Stable Diffusion across six races, two genders, five age groups, 32 professions, and eight attributes. We then propose novel debiasing solutions that outperform state-of-the-art alternatives. Additionally, we examine the degree to which Stable Diffusion depicts individuals of the same race as being similar to one another. This analysis reveals a high degree of stereotyping, e.g., depicting most middle eastern males as being dark-skinned, bearded, and wearing a traditional headdress. We address these limitations by proposing yet another novel solution that increases facial diversity across genders and racial groups. Our solutions are open-sourced and made publicly available.

Title: IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition

• Authors: Zikang Leng, Amitrajit Bhattacharjee, Hrudhai Rajasekhar, Lizhe Zhang, Elizabeth Bruda, Hyeokhyen Kwon, Thomas Plötz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01049
• Pdf URL: https://arxiv.org/pdf/2402.01049
• Copy Paste: [[2402.01049]] IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition(https://arxiv.org/abs/2402.01049)
• Keywords: generative
• Abstract: One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality (IMU). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which open up IMUGPT for practical use cases beyond a mere proof of concept.

Title: Chameleon: Foundation Models for Fairness-aware Multi-modal Data Augmentation to Enhance Coverage of Minorities

• Authors: Mahdi Erfanian, H. V. Jagadish, Abolfazl Asudeh
• Subjects: cs.LG, cs.CY, cs.DB
• Abstract URL: https://arxiv.org/abs/2402.01071
• Pdf URL: https://arxiv.org/pdf/2402.01071
• Copy Paste: [[2402.01071]] Chameleon: Foundation Models for Fairness-aware Multi-modal Data Augmentation to Enhance Coverage of Minorities(https://arxiv.org/abs/2402.01071)
• Keywords: foundation model, generative
• Abstract: The potential harms of the under-representation of minorities in training data, particularly in multi-modal settings, is a well-recognized concern. While there has been extensive effort in detecting such under-representation, resolution has remained a challenge. With recent advancements in generative AI, large language models and foundation models have emerged as versatile tools across various domains. In this paper, we propose Chameleon, a system that efficiently utilizes these tools to augment a data set with a minimal addition of synthetically generated tuples, in order to enhance the coverage of the under-represented groups. Our system follows a rejection sampling approach to ensure the generated tuples have a high quality and follow the underlying distribution. In order to minimize the rejection chance of the generated tuples, we propose multiple strategies for providing a guide for the foundation model. Our experiment results, in addition to confirming the efficiency of our proposed algorithms, illustrate the effectiveness of our approach, as the unfairness of the model in a downstream task significantly dropped after data repair using Chameleon.

Title: Compositional Generative Modeling: A Single Model is Not All You Need

• Authors: Yilun Du, Leslie Kaelbling
• Subjects: cs.LG, cs.AI, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.01103
• Pdf URL: https://arxiv.org/pdf/2402.01103
• Copy Paste: [[2402.01103]] Compositional Generative Modeling: A Single Model is Not All You Need(https://arxiv.org/abs/2402.01103)
• Keywords: generative
• Abstract: Large monolithic generative models trained on massive amounts of data have become an increasingly dominant approach in AI research. In this paper, we argue that we should instead construct large generative systems by composing smaller generative models together. We show how such a compositional generative approach enables us to learn distributions in a more data-efficient manner, enabling generalization to parts of the data distribution unseen at training time. We further show how this enables us to program and construct new generative models for tasks completely unseen at training. Finally, we show that in many cases, we can discover separate compositional components from data.

Title: A Survey for Foundation Models in Autonomous Driving

• Authors: Haoxiang Gao, Yaqian Li, Kaiwen Long, Ming Yang, Yiqing Shen
• Subjects: cs.LG, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.01105
• Pdf URL: https://arxiv.org/pdf/2402.01105
• Copy Paste: [[2402.01105]] A Survey for Foundation Models in Autonomous Driving(https://arxiv.org/abs/2402.01105)
• Keywords: foundation model
• Abstract: The advent of foundation models has revolutionized the fields of natural language processing and computer vision, paving the way for their application in autonomous driving (AD). This survey presents a comprehensive review of more than 40 research papers, demonstrating the role of foundation models in enhancing AD. Large language models contribute to planning and simulation in AD, particularly through their proficiency in reasoning, code generation and translation. In parallel, vision foundation models are increasingly adapted for critical tasks such as 3D object detection and tracking, as well as creating realistic driving scenarios for simulation and testing. Multi-modal foundation models, integrating diverse inputs, exhibit exceptional visual understanding and spatial reasoning, crucial for end-to-end AD. This survey not only provides a structured taxonomy, categorizing foundation models based on their modalities and functionalities within the AD domain but also delves into the methods employed in current research. It identifies the gaps between existing foundation models and cutting-edge AD approaches, thereby charting future research directions and proposing a roadmap for bridging these gaps.

Title: A Single Simple Patch is All You Need for AI-generated Image Detection

• Authors: Jiaxuan Chen, Jieteng Yao, Li Niu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01123
• Pdf URL: https://arxiv.org/pdf/2402.01123
• Copy Paste: [[2402.01123]] A Single Simple Patch is All You Need for AI-generated Image Detection(https://arxiv.org/abs/2402.01123)
• Keywords: generative
• Abstract: The recent development of generative models unleashes the potential of generating hyper-realistic fake images. To prevent the malicious usage of fake images, AI-generated image detection aims to distinguish fake images from real images. Nevertheless, existing methods usually suffer from poor generalizability across different generators. In this work, we propose an embarrassingly simple approach named SSP, i.e., feeding the noise pattern of a Single Simple Patch (SSP) to a binary classifier, which could achieve 14.6% relative improvement over the recent method on GenImage dataset. Our SSP method is very robust and generalizable, which could serve as a simple and competitive baseline for the future methods.

Title: Root Cause Analysis In Microservice Using Neural Granger Causal Discovery

• Authors: Cheng-Ming Lin, Ching Chang, Wei-Yao Wang, Kuang-Da Wang, Wen-Chih Peng
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.01140
• Pdf URL: https://arxiv.org/pdf/2402.01140
• Copy Paste: [[2402.01140]] Root Cause Analysis In Microservice Using Neural Granger Causal Discovery(https://arxiv.org/abs/2402.01140)
• Keywords: anomaly
• Abstract: In recent years, microservices have gained widespread adoption in IT operations due to their scalability, maintenance, and flexibility. However, it becomes challenging for site reliability engineers (SREs) to pinpoint the root cause due to the complex relationships in microservices when facing system malfunctions. Previous research employed structured learning methods (e.g., PC-algorithm) to establish causal relationships and derive root causes from causal graphs. Nevertheless, they ignored the temporal order of time series data and failed to leverage the rich information inherent in the temporal relationships. For instance, in cases where there is a sudden spike in CPU utilization, it can lead to an increase in latency for other microservices. However, in this scenario, the anomaly in CPU utilization occurs before the latency increase, rather than simultaneously. As a result, the PC-algorithm fails to capture such characteristics. To address these challenges, we propose RUN, a novel approach for root cause analysis using neural Granger causal discovery with contrastive learning. RUN enhances the backbone encoder by integrating contextual information from time series, and leverages a time series forecasting model to conduct neural Granger causal discovery. In addition, RUN incorporates Pagerank with a personalization vector to efficiently recommend the top-k root causes. Extensive experiments conducted on the synthetic and real-world microservice-based datasets demonstrate that RUN noticeably outperforms the state-of-the-art root cause analysis methods. Moreover, we provide an analysis scenario for the sock-shop case to showcase the practicality and efficacy of RUN in microservice-based applications. Our code is publicly available at https://github.com/zmlin1998/RUN.

Title: Learning Network Representations with Disentangled Graph Auto-Encoder

• Authors: Di Fan, Chuanhou Gao
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01143
• Pdf URL: https://arxiv.org/pdf/2402.01143
• Copy Paste: [[2402.01143]] Learning Network Representations with Disentangled Graph Auto-Encoder(https://arxiv.org/abs/2402.01143)
• Keywords: generative
• Abstract: The (variational) graph auto-encoder is extensively employed for learning representations of graph-structured data. However, the formation of real-world graphs is a complex and heterogeneous process influenced by latent factors. Existing encoders are fundamentally holistic, neglecting the entanglement of latent factors. This not only makes graph analysis tasks less effective but also makes it harder to understand and explain the representations. Learning disentangled graph representations with (variational) graph auto-encoder poses significant challenges, and remains largely unexplored in the existing literature. In this article, we introduce the Disentangled Graph Auto-Encoder (DGA) and Disentangled Variational Graph Auto-Encoder (DVGA), approaches that leverage generative models to learn disentangled representations. Specifically, we first design a disentangled graph convolutional network with multi-channel message-passing layers, as the encoder aggregating information related to each disentangled latent factor. Subsequently, a component-wise flow is applied to each channel to enhance the expressive capabilities of disentangled variational graph auto-encoder. Additionally, we design a factor-wise decoder, considering the characteristics of disentangled representations. In order to further enhance the independence among representations, we introduce independence constraints on mapping channels for different latent factors. Empirical experiments on both synthetic and real-world datasets show the superiority of our proposed method compared to several state-of-the-art baselines.

Title: CABINET: Content Relevance based Noise Reduction for Table Question Answering

• Authors: Sohan Patnaik, Heril Changwal, Milan Aggarwal, Sumita Bhatia, Yaman Kumar, Balaji Krishnamurthy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01155
• Pdf URL: https://arxiv.org/pdf/2402.01155
• Copy Paste: [[2402.01155]] CABINET: Content Relevance based Noise Reduction for Table Question Answering(https://arxiv.org/abs/2402.01155)
• Keywords: in-context
• Abstract: Table understanding capability of Large Language Models (LLMs) has been extensively studied through the task of question-answering (QA) over tables. Typically, only a small part of the whole table is relevant to derive the answer for a given question. The irrelevant parts act as noise and are distracting information, resulting in sub-optimal performance due to the vulnerability of LLMs to noise. To mitigate this, we propose CABINET (Content RelevAnce-Based NoIse ReductioN for TablE QuesTion-Answering) - a framework to enable LLMs to focus on relevant tabular data by suppressing extraneous information. CABINET comprises an Unsupervised Relevance Scorer (URS), trained differentially with the QA LLM, that weighs the table content based on its relevance to the input question before feeding it to the question-answering LLM (QA LLM). To further aid the relevance scorer, CABINET employs a weakly supervised module that generates a parsing statement describing the criteria of rows and columns relevant to the question and highlights the content of corresponding table cells. CABINET significantly outperforms various tabular LLM baselines, as well as GPT3-based in-context learning methods, is more robust to noise, maintains outperformance on tables of varying sizes, and establishes new SoTA performance on WikiTQ, FeTaQA, and WikiSQL datasets. We release our code and datasets at https://github.com/Sohanpatnaik106/CABINET_QA.

Title: Enhanced Urban Region Profiling with Adversarial Self-Supervised Learning

• Authors: Weiliang Chan, Qianqian Ren, Jinbao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01163
• Pdf URL: https://arxiv.org/pdf/2402.01163
• Copy Paste: [[2402.01163]] Enhanced Urban Region Profiling with Adversarial Self-Supervised Learning(https://arxiv.org/abs/2402.01163)
• Keywords: self-supervised
• Abstract: Urban region profiling is pivotal for smart cities, but mining fine-grained semantics from noisy and incomplete urban data remains challenging. In response, we propose a novel self-supervised graph collaborative filtering model for urban region embedding called EUPAS. Specifically, region heterogeneous graphs containing human mobility data, point of interests (POIs) information, and geographic neighborhood details for each region are fed into the model, which generates region embeddings that preserve intra-region and inter-region dependencies through GCNs and multi-head attention. Meanwhile, we introduce spatial perturbation augmentation to generate positive samples that are semantically similar and spatially close to the anchor, preparing for subsequent contrastive learning. Furthermore, adversarial training is employed to construct an effective pretext task by generating strong positive pairs and mining hard negative pairs for the region embeddings. Finally, we jointly optimize supervised and self-supervised learning to encourage the model to capture the high-level semantics of region embeddings while ignoring the noisy and unimportant details. Extensive experiments on real-world datasets demonstrate the superiority of our model over state-of-the-art methods.

Title: A Comprehensive Survey on 3D Content Generation

• Authors: Jian Liu, Xiaoshui Huang, Tianyu Huang, Lu Chen, Yuenan Hou, Shixiang Tang, Ziwei Liu, Wanli Ouyang, Wangmeng Zuo, Junjun Jiang, Xianming Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01166
• Pdf URL: https://arxiv.org/pdf/2402.01166
• Copy Paste: [[2402.01166]] A Comprehensive Survey on 3D Content Generation(https://arxiv.org/abs/2402.01166)
• Keywords: generative
• Abstract: Recent years have witnessed remarkable advances in artificial intelligence generated content(AIGC), with diverse input modalities, e.g., text, image, video, audio and 3D. The 3D is the most close visual modality to real-world 3D environment and carries enormous knowledge. The 3D content generation shows both academic and practical values while also presenting formidable technical challenges. This review aims to consolidate developments within the burgeoning domain of 3D content generation. Specifically, a new taxonomy is proposed that categorizes existing approaches into three types: 3D native generative methods, 2D prior-based 3D generative methods, and hybrid 3D generative methods. The survey covers approximately 60 papers spanning the major techniques. Besides, we discuss limitations of current 3D content generation techniques, and point out open challenges as well as promising directions for future work. Accompanied with this survey, we have established a project website where the resources on 3D content generation research are provided. The project page is available at https://github.com/hitcslj/Awesome-AIGC-3D.

Title: Towards a Unified Language Model for Knowledge-Intensive Tasks Utilizing External Corpus

• Authors: Xiaoxi Li, Zhicheng Dou, Yujia Zhou, Fangchao Liu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.01176
• Pdf URL: https://arxiv.org/pdf/2402.01176
• Copy Paste: [[2402.01176]] Towards a Unified Language Model for Knowledge-Intensive Tasks Utilizing External Corpus(https://arxiv.org/abs/2402.01176)
• Keywords: generative
• Abstract: The advent of large language models (LLMs) has showcased their efficacy across various domains, yet they often hallucinate, especially in knowledge-intensive tasks that require external knowledge sources. To improve factual accuracy of language models, retrieval-augmented generation (RAG) has emerged as a popular solution. However, traditional retrieval modules often rely on large-scale document indexes, which can be disconnected from generative tasks. Through generative retrieval (GR) approach, language models can achieve superior retrieval performance by directly generating relevant document identifiers (DocIDs). However, the relationship between GR and downstream tasks, as well as the potential of LLMs in GR, remains unexplored. In this paper, we present a unified language model that utilizes external corpus to handle various knowledge-intensive tasks by seamlessly integrating generative retrieval, closed-book generation, and RAG. In order to achieve effective retrieval and generation through a unified continuous decoding process, we introduce the following mechanisms: (1) a ranking-oriented DocID decoding strategy, which improves ranking ability by directly learning from a DocID ranking list; (2) a continuous generation strategy to facilitate effective and efficient RAG; (3) well-designed auxiliary DocID understanding tasks to enhance the model's comprehension of DocIDs and their relevance to downstream tasks. Our approach is evaluated on the widely used KILT benchmark using two variants of backbone models: an encoder-decoder T5 model and a decoder-only LLM, Llama2. Experimental results showcase the superior performance of our models in both retrieval and downstream knowledge-intensive tasks.

Title: In-Context Learning for Few-Shot Nested Named Entity Recognition

• Authors: Meishan Zhang, Bin Wang, Hao Fei, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01182
• Pdf URL: https://arxiv.org/pdf/2402.01182
• Copy Paste: [[2402.01182]] In-Context Learning for Few-Shot Nested Named Entity Recognition(https://arxiv.org/abs/2402.01182)
• Keywords: in-context
• Abstract: In nested Named entity recognition (NER), entities are nested with each other, and thus requiring more data annotations to address. This leads to the development of few-shot nested NER, where the prevalence of pretrained language models with in-context learning (ICL) offers promising solutions. In this work, we introduce an effective and innovative ICL framework for the setting of few-shot nested NER. We improve the ICL prompt by devising a novel example demonstration selection mechanism, EnDe retriever. In EnDe retriever, we employ contrastive learning to perform three types of representation learning, in terms of semantic similarity, boundary similarity, and label similarity, to generate high-quality demonstration examples. Extensive experiments over three nested NER and four flat NER datasets demonstrate the efficacy of our system.

Title: Segment Any Change

• Authors: Zhuo Zheng, Yanfei Zhong, Liangpei Zhang, Stefano Ermon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01188
• Pdf URL: https://arxiv.org/pdf/2402.01188
• Copy Paste: [[2402.01188]] Segment Any Change(https://arxiv.org/abs/2402.01188)
• Keywords: foundation model
• Abstract: Visual foundation models have achieved remarkable results in zero-shot image classification and segmentation, but zero-shot change detection remains an open problem. In this paper, we propose the segment any change models (AnyChange), a new type of change detection model that supports zero-shot prediction and generalization on unseen change types and data distributions. AnyChange is built on the segment anything model (SAM) via our training-free adaptation method, bitemporal latent matching. By revealing and exploiting intra-image and inter-image semantic similarities in SAM's latent space, bitemporal latent matching endows SAM with zero-shot change detection capabilities in a training-free way. We also propose a point query mechanism to enable AnyChange's zero-shot object-centric change detection capability. We perform extensive experiments to confirm the effectiveness of AnyChange for zero-shot change detection. AnyChange sets a new record on the SECOND benchmark for unsupervised change detection, exceeding the previous SOTA by up to 4.4% F$_1$ score, and achieving comparable accuracy with negligible manual annotations (1 pixel per image) for supervised change detection.

Title: Unsupervised Generation of Pseudo Normal PET from MRI with Diffusion Model for Epileptic Focus Localization

• Authors: Wentao Chen, Jiwei Li, Xichen Xu, Hui Huang, Siyu Yuan, Miao Zhang, Tianming Xu, Jie Luo, Weimin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01191
• Pdf URL: https://arxiv.org/pdf/2402.01191
• Copy Paste: [[2402.01191]] Unsupervised Generation of Pseudo Normal PET from MRI with Diffusion Model for Epileptic Focus Localization(https://arxiv.org/abs/2402.01191)
• Keywords: diffusion
• Abstract: [$^{18}$F]fluorodeoxyglucose (FDG) positron emission tomography (PET) has emerged as a crucial tool in identifying the epileptic focus, especially in cases where magnetic resonance imaging (MRI) diagnosis yields indeterminate results. FDG PET can provide the metabolic information of glucose and help identify abnormal areas that are not easily found through MRI. However, the effectiveness of FDG PET-based assessment and diagnosis depends on the selection of a healthy control group. The healthy control group typically consists of healthy individuals similar to epilepsy patients in terms of age, gender, and other aspects for providing normal FDG PET data, which will be used as a reference for enhancing the accuracy and reliability of the epilepsy diagnosis. However, significant challenges arise when a healthy PET control group is unattainable. Yaakub \emph{et al.} have previously introduced a Pix2PixGAN-based method for MRI to PET translation. This method used paired MRI and FDG PET scans from healthy individuals for training, and produced pseudo normal FDG PET images from patient MRIs that are subsequently used for lesion detection. However, this approach requires a large amount of high-quality, paired MRI and PET images from healthy control subjects, which may not always be available. In this study, we investigated unsupervised learning methods for unpaired MRI to PET translation for generating pseudo normal FDG PET for epileptic focus localization. Two deep learning methods, CycleGAN and SynDiff, were employed, and we found that diffusion-based method achieved improved performance in accurately localizing the epileptic focus.

Title: Conditional Normalizing Flows for Active Learning of Coarse-Grained Molecular Representations

• Authors: Henrik Schopmans, Pascal Friederich
• Subjects: cs.LG, cs.AI, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2402.01195
• Pdf URL: https://arxiv.org/pdf/2402.01195
• Copy Paste: [[2402.01195]] Conditional Normalizing Flows for Active Learning of Coarse-Grained Molecular Representations(https://arxiv.org/abs/2402.01195)
• Keywords: generative
• Abstract: Efficient sampling of the Boltzmann distribution of molecular systems is a long-standing challenge. Recently, instead of generating long molecular dynamics simulations, generative machine learning methods such as normalizing flows have been used to learn the Boltzmann distribution directly, without samples. However, this approach is susceptible to mode collapse and thus often does not explore the full configurational space. In this work, we address this challenge by separating the problem into two levels, the fine-grained and coarse-grained degrees of freedom. A normalizing flow conditioned on the coarse-grained space yields a probabilistic connection between the two levels. To explore the configurational space, we employ coarse-grained simulations with active learning which allows us to update the flow and make all-atom potential energy evaluations only when necessary. Using alanine dipeptide as an example, we show that our methods obtain a speedup to molecular dynamics simulations of approximately 15.9 to 216.2 compared to the speedup of 4.5 of the current state-of-the-art machine learning approach.

Title: Structured World Modeling via Semantic Vector Quantization

• Authors: Yi-Fu Wu, Minseung Lee, Sungjin Ahn
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01203
• Pdf URL: https://arxiv.org/pdf/2402.01203
• Copy Paste: [[2402.01203]] Structured World Modeling via Semantic Vector Quantization(https://arxiv.org/abs/2402.01203)
• Keywords: generative
• Abstract: Neural discrete representations are crucial components of modern neural networks. However, their main limitation is that the primary strategies such as VQ-VAE can only provide representations at the patch level. Therefore, one of the main goals of representation learning, acquiring structured, semantic, and compositional abstractions such as the color and shape of an object, remains elusive. In this paper, we present the first approach to semantic neural discrete representation learning. The proposed model, called Semantic Vector-Quantized Variational Autoencoder (SVQ), leverages recent advances in unsupervised object-centric learning to address this limitation. Specifically, we observe that a simple approach quantizing at the object level poses a significant challenge and propose constructing scene representations hierarchically, from low-level discrete concept schemas to object representations. Additionally, we suggest a novel method for structured semantic world modeling by training a prior over these representations, enabling the ability to generate images by sampling the semantic properties of the objects in the scene. In experiments on various 2D and 3D object-centric datasets, we find that our model achieves superior generation performance compared to non-semantic vector quantization methods such as VQ-VAE and previous object-centric generative models. Furthermore, we find that the semantic discrete representations can solve downstream scene understanding tasks that require reasoning about the properties of different objects in the scene.

Title: A Survey on Self-Supervised Learning for Non-Sequential Tabular Data

• Authors: Wei-Yao Wang, Wei-Wei Du, Derek Xu, Wei Wang, Wen-Chih Peng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01204
• Pdf URL: https://arxiv.org/pdf/2402.01204
• Copy Paste: [[2402.01204]] A Survey on Self-Supervised Learning for Non-Sequential Tabular Data(https://arxiv.org/abs/2402.01204)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) has been incorporated into many state-of-the-art models in various domains, where SSL defines pretext tasks based on unlabeled datasets to learn contextualized and robust representations. Recently, SSL has been a new trend in exploring the representation learning capability in the realm of tabular data, which is more challenging due to not having explicit relations for learning descriptive representations. This survey aims to systematically review and summarize the recent progress and challenges of SSL for non-sequential tabular data (SSL4NS-TD). We first present a formal definition of NS-TD and clarify its correlation to related studies. Then, these approaches are categorized into three groups -- predictive learning, contrastive learning, and hybrid learning, with their motivations and strengths of representative methods within each direction. On top of this, application issues of SSL4NS-TD are presented, including automatic data engineering, cross-table transferability, and domain knowledge integration. In addition, we elaborate on existing benchmarks and datasets for NS-TD applications to discuss the performance of existing tabular models. Finally, we discuss the challenges of SSL4NS-TD and provide potential directions for future research. We expect our work to be useful in terms of encouraging more research on lowering the barrier to entry SSL for the tabular domain and improving the foundations for implicit tabular data.

Title: Taming Uncertainty in Sparse-view Generalizable NeRF via Indirect Diffusion Guidance

• Authors: Yaokun Li, Chao Gou, Guang Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01217
• Pdf URL: https://arxiv.org/pdf/2402.01217
• Copy Paste: [[2402.01217]] Taming Uncertainty in Sparse-view Generalizable NeRF via Indirect Diffusion Guidance(https://arxiv.org/abs/2402.01217)
• Keywords: diffusion, generative
• Abstract: Neural Radiance Fields (NeRF) have demonstrated effectiveness in synthesizing novel views. However, their reliance on dense inputs and scene-specific optimization has limited their broader applicability. Generalizable NeRFs (Gen-NeRF), while intended to address this, often produce blurring artifacts in unobserved regions with sparse inputs, which are full of uncertainty. In this paper, we aim to diminish the uncertainty in Gen-NeRF for plausible renderings. We assume that NeRF's inability to effectively mitigate this uncertainty stems from its inherent lack of generative capacity. Therefore, we innovatively propose an Indirect Diffusion-guided NeRF framework, termed ID-NeRF, to address this uncertainty from a generative perspective by leveraging a distilled diffusion prior as guidance. Specifically, to avoid model confusion caused by directly regularizing with inconsistent samplings as in previous methods, our approach introduces a strategy to indirectly inject the inherently missing imagination into the learned implicit function through a diffusion-guided latent space. Empirical evaluation across various benchmarks demonstrates the superior performance of our approach in handling uncertainty with sparse inputs.

Title: PRIME: Protect Your Videos From Malicious Editing

• Authors: Guanlin Li, Shuai Yang, Jie Zhang, Tianwei Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01239
• Pdf URL: https://arxiv.org/pdf/2402.01239
• Copy Paste: [[2402.01239]] PRIME: Protect Your Videos From Malicious Editing(https://arxiv.org/abs/2402.01239)
• Keywords: generative
• Abstract: With the development of generative models, the quality of generated content keeps increasing. Recently, open-source models have made it surprisingly easy to manipulate and edit photos and videos, with just a few simple prompts. While these cutting-edge technologies have gained popularity, they have also given rise to concerns regarding the privacy and portrait rights of individuals. Malicious users can exploit these tools for deceptive or illegal purposes. Although some previous works focus on protecting photos against generative models, we find there are still gaps between protecting videos and images in the aspects of efficiency and effectiveness. Therefore, we introduce our protection method, PRIME, to significantly reduce the time cost and improve the protection performance. Moreover, to evaluate our proposed protection method, we consider both objective metrics and human subjective metrics. Our evaluation results indicate that PRIME only costs 8.3% GPU hours of the cost of the previous state-of-the-art method and achieves better protection results on both human evaluation and objective metrics. Code can be found in https://github.com/GuanlinLee/prime.

Title: Can Shape-Infused Joint Embeddings Improve Image-Conditioned 3D Diffusion?

• Authors: Cristian Sbrolli, Paolo Cudrano, Matteo Matteucci
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01241
• Pdf URL: https://arxiv.org/pdf/2402.01241
• Copy Paste: [[2402.01241]] Can Shape-Infused Joint Embeddings Improve Image-Conditioned 3D Diffusion?(https://arxiv.org/abs/2402.01241)
• Keywords: diffusion, generative
• Abstract: Recent advancements in deep generative models, particularly with the application of CLIP (Contrastive Language Image Pretraining) to Denoising Diffusion Probabilistic Models (DDPMs), have demonstrated remarkable effectiveness in text to image generation. The well structured embedding space of CLIP has also been extended to image to shape generation with DDPMs, yielding notable results. Despite these successes, some fundamental questions arise: Does CLIP ensure the best results in shape generation from images? Can we leverage conditioning to bring explicit 3D knowledge into the generative process and obtain better quality? This study introduces CISP (Contrastive Image Shape Pre training), designed to enhance 3D shape synthesis guided by 2D images. CISP aims to enrich the CLIP framework by aligning 2D images with 3D shapes in a shared embedding space, specifically capturing 3D characteristics potentially overlooked by CLIP's text image focus. Our comprehensive analysis assesses CISP's guidance performance against CLIP guided models, focusing on generation quality, diversity, and coherence of the produced shapes with the conditioning image. We find that, while matching CLIP in generation quality and diversity, CISP substantially improves coherence with input images, underscoring the value of incorporating 3D knowledge into generative models. These findings suggest a promising direction for advancing the synthesis of 3D visual content by integrating multimodal systems with 3D representations.

Title: Can MLLMs Perform Text-to-Image In-Context Learning?

• Authors: Yuchen Zeng, Wonjun Kang, Yicong Chen, Hyung Il Koo, Kangwook Lee
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01293
• Pdf URL: https://arxiv.org/pdf/2402.01293
• Copy Paste: [[2402.01293]] Can MLLMs Perform Text-to-Image In-Context Learning?(https://arxiv.org/abs/2402.01293)
• Keywords: in-context
• Abstract: The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present CoBSAT, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation. To overcome these challenges, we explore strategies like fine-tuning and Chain-of-Thought prompting, demonstrating notable improvements. Our code and dataset are available at \url{https://github.com/UW-Madison-Lee-Lab/CoBSAT}.

Title: Simulator-Free Visual Domain Randomization via Video Games

• Authors: Chintan Trivedi, Nemanja Rašajski, Konstantinos Makantasis, Antonios Liapis, Georgios N. Yannakakis
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01335
• Pdf URL: https://arxiv.org/pdf/2402.01335
• Copy Paste: [[2402.01335]] Simulator-Free Visual Domain Randomization via Video Games(https://arxiv.org/abs/2402.01335)
• Keywords: foundation model
• Abstract: Domain randomization is an effective computer vision technique for improving transferability of vision models across visually distinct domains exhibiting similar content. Existing approaches, however, rely extensively on tweaking complex and specialized simulation engines that are difficult to construct, subsequently affecting their feasibility and scalability. This paper introduces BehAVE, a video understanding framework that uniquely leverages the plethora of existing commercial video games for domain randomization, without requiring access to their simulation engines. Under BehAVE (1) the inherent rich visual diversity of video games acts as the source of randomization and (2) player behavior -- represented semantically via textual descriptions of actions -- guides the *alignment* of videos with similar content. We test BehAVE on 25 games of the first-person shooter (FPS) genre across various video and text foundation models and we report its robustness for domain randomization. BehAVE successfully aligns player behavioral patterns and is able to zero-shot transfer them to multiple unseen FPS games when trained on just one FPS game. In a more challenging setting, BehAVE manages to improve the zero-shot transferability of foundation models to unseen FPS games (up to 22%) even when trained on a game of a different genre (Minecraft). Code and dataset can be found at https://github.com/nrasajski/BehAVE.

Title: Cheating Suffix: Targeted Attack to Text-To-Image Diffusion Models with Multi-Modal Priors

• Authors: Dingcheng Yang, Yang Bai, Xiaojun Jia, Yang Liu, Xiaochun Cao, Wenjian Yu
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01369
• Pdf URL: https://arxiv.org/pdf/2402.01369
• Copy Paste: [[2402.01369]] Cheating Suffix: Targeted Attack to Text-To-Image Diffusion Models with Multi-Modal Priors(https://arxiv.org/abs/2402.01369)
• Keywords: diffusion
• Abstract: Diffusion models have been widely deployed in various image generation tasks, demonstrating an extraordinary connection between image and text modalities. However, they face challenges of being maliciously exploited to generate harmful or sensitive images by appending a specific suffix to the original prompt. Existing works mainly focus on using single-modal information to conduct attacks, which fails to utilize multi-modal features and results in less than satisfactory performance. Integrating multi-modal priors (MMP), i.e. both text and image features, we propose a targeted attack method named MMP-Attack in this work. Specifically, the goal of MMP-Attack is to add a target object into the image content while simultaneously removing the original object. The MMP-Attack shows a notable advantage over existing works with superior universality and transferability, which can effectively attack commercial text-to-image (T2I) models such as DALL-E 3. To the best of our knowledge, this marks the first successful attempt of transfer-based attack to commercial T2I models. Our code is publicly available at \url{https://github.com/ydc123/MMP-Attack}.

Title: A Probabilistic Model to explain Self-Supervised Representation Learning

• Authors: Alice Bizeul, Bernhard Schölkopf, Carl Allen
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01399
• Pdf URL: https://arxiv.org/pdf/2402.01399
• Copy Paste: [[2402.01399]] A Probabilistic Model to explain Self-Supervised Representation Learning(https://arxiv.org/abs/2402.01399)
• Keywords: self-supervised, generative
• Abstract: Self-supervised learning (SSL) learns representations by leveraging an auxiliary unsupervised task, such as classifying semantically related samples, e.g. different data augmentations or modalities. Of the many approaches to SSL, contrastive methods, e.g. SimCLR, CLIP and VicREG, have gained attention for learning representations that achieve downstream performance close to that of supervised learning. However, a theoretical understanding of the mechanism behind these methods eludes. We propose a generative latent variable model for the data and show that several families of discriminative self-supervised algorithms, including contrastive methods, approximately induce its latent structure over representations, providing a unifying theoretical framework. We also justify links to mutual information and the use of a projection head. Fitting our model generatively, as SimVE, improves performance over previous VAE methods on common benchmarks (e.g. FashionMNIST, CIFAR10, CelebA), narrows the gap to discriminative methods on _content_ classification and, as our analysis predicts, outperforms them where _style_ information is required, taking a step toward task-agnostic representations.

Title: EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation

• Authors: Guanwen Feng, Haoran Cheng, Yunan Li, Zhiyuan Ma, Chaoneng Li, Zhihao Qian, Qiguang Miao, Chi-Man Pun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01422
• Pdf URL: https://arxiv.org/pdf/2402.01422
• Copy Paste: [[2402.01422]] EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation(https://arxiv.org/abs/2402.01422)
• Keywords: generative
• Abstract: Implementing fine-grained emotion control is crucial for emotion generation tasks because it enhances the expressive capability of the generative model, allowing it to accurately and comprehensively capture and express various nuanced emotional states, thereby improving the emotional quality and personalization of generated content. Generating fine-grained facial animations that accurately portray emotional expressions using only a portrait and an audio recording presents a challenge. In order to address this challenge, we propose a visual attribute-guided audio decoupler. This enables the obtention of content vectors solely related to the audio content, enhancing the stability of subsequent lip movement coefficient predictions. To achieve more precise emotional expression, we introduce a fine-grained emotion coefficient prediction module. Additionally, we propose an emotion intensity control method using a fine-grained emotion matrix. Through these, effective control over emotional expression in the generated videos and finer classification of emotion intensity are accomplished. Subsequently, a series of 3DMM coefficient generation networks are designed to predict 3D coefficients, followed by the utilization of a rendering network to generate the final video. Our experimental results demonstrate that our proposed method, EmoSpeaker, outperforms existing emotional talking face generation methods in terms of expression variation and lip synchronization. Project page: https://peterfanfan.github.io/EmoSpeaker/

Title: Cross-view Masked Diffusion Transformers for Person Image Synthesis

• Authors: Trung X. Pham, Zhang Kang, Chang D. Yoo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01516
• Pdf URL: https://arxiv.org/pdf/2402.01516
• Copy Paste: [[2402.01516]] Cross-view Masked Diffusion Transformers for Person Image Synthesis(https://arxiv.org/abs/2402.01516)
• Keywords: diffusion
• Abstract: We present X-MDPT (Cross-view Masked Diffusion Prediction Transformers), a novel diffusion model designed for pose-guided human image generation. X-MDPT distinguishes itself by employing masked diffusion transformers that operate on latent patches, a departure from the commonly-used Unet structures in existing works. The model comprises three key modules: 1) a denoising diffusion Transformer, 2) an aggregation network that consolidates conditions into a single vector for the diffusion process, and 3) a mask cross-prediction module that enhances representation learning with semantic information from the reference image. X-MDPT demonstrates scalability, improving FID, SSIM, and LPIPS with larger models. Despite its simple design, our model outperforms state-of-the-art approaches on the DeepFashion dataset while exhibiting efficiency in terms of training parameters, training time, and inference speed. Our compact 33MB model achieves an FID of 7.42, surpassing a prior Unet latent diffusion approach (FID 8.07) using only $11\times$ fewer parameters. Our best model surpasses the pixel-based diffusion with $\frac{2}{3}$ of the parameters and achieves $5.43 \times$ faster inference.

Title: Closing the Gap in Human Behavior Analysis: A Pipeline for Synthesizing Trimodal Data

• Authors: Christian Stippel, Thomas Heitzinger, Rafael Sterzinger, Martin Kampel
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01537
• Pdf URL: https://arxiv.org/pdf/2402.01537
• Copy Paste: [[2402.01537]] Closing the Gap in Human Behavior Analysis: A Pipeline for Synthesizing Trimodal Data(https://arxiv.org/abs/2402.01537)
• Keywords: generative
• Abstract: In pervasive machine learning, especially in Human Behavior Analysis (HBA), RGB has been the primary modality due to its accessibility and richness of information. However, linked with its benefits are challenges, including sensitivity to lighting conditions and privacy concerns. One possibility to overcome these vulnerabilities is to resort to different modalities. For instance, thermal is particularly adept at accentuating human forms, while depth adds crucial contextual layers. Despite their known benefits, only a few HBA-specific datasets that integrate these modalities exist. To address this shortage, our research introduces a novel generative technique for creating trimodal, i.e., RGB, thermal, and depth, human-focused datasets. This technique capitalizes on human segmentation masks derived from RGB images, combined with thermal and depth backgrounds that are sourced automatically. With these two ingredients, we synthesize depth and thermal counterparts from existing RGB data utilizing conditional image-to-image translation. By employing this approach, we generate trimodal data that can be leveraged to train models for settings with limited data, bad lightning conditions, or privacy-sensitive areas.

Title: SLYKLatent, a Learning Framework for Facial Features Estimation

• Authors: Samuel Adebayo, Joost C. Dessing, Seán McLoone
• Subjects: cs.CV, cs.HC, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.01555
• Pdf URL: https://arxiv.org/pdf/2402.01555
• Copy Paste: [[2402.01555]] SLYKLatent, a Learning Framework for Facial Features Estimation(https://arxiv.org/abs/2402.01555)
• Keywords: self-supervised
• Abstract: In this research, we present SLYKLatent, a novel approach for enhancing gaze estimation by addressing appearance instability challenges in datasets due to aleatoric uncertainties, covariant shifts, and test domain generalization. SLYKLatent utilizes Self-Supervised Learning for initial training with facial expression datasets, followed by refinement with a patch-based tri-branch network and an inverse explained variance-weighted training loss function. Our evaluation on benchmark datasets achieves an 8.7% improvement on Gaze360, rivals top MPIIFaceGaze results, and leads on a subset of ETH-XGaze by 13%, surpassing existing methods by significant margins. Adaptability tests on RAF-DB and Affectnet show 86.4% and 60.9% accuracies, respectively. Ablation studies confirm the effectiveness of SLYKLatent's novel components. This approach has strong potential in human-robot interaction.

Title: Boximator: Generating Rich and Controllable Motions for Video Synthesis

• Authors: Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, Hang Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01566
• Pdf URL: https://arxiv.org/pdf/2402.01566
• Copy Paste: [[2402.01566]] Boximator: Generating Rich and Controllable Motions for Video Synthesis(https://arxiv.org/abs/2402.01566)
• Keywords: diffusion
• Abstract: Generating rich and controllable motion is a pivotal challenge in video synthesis. We propose Boximator, a new approach for fine-grained motion control. Boximator introduces two constraint types: hard box and soft box. Users select objects in the conditional frame using hard boxes and then use either type of boxes to roughly or rigorously define the object's position, shape, or motion path in future frames. Boximator functions as a plug-in for existing video diffusion models. Its training process preserves the base model's knowledge by freezing the original weights and training only the control module. To address training challenges, we introduce a novel self-tracking technique that greatly simplifies the learning of box-object correlations. Empirically, Boximator achieves state-of-the-art video quality (FVD) scores, improving on two base models, and further enhanced after incorporating box constraints. Its robust motion controllability is validated by drastic increases in the bounding box alignment metric. Human evaluation also shows that users favor Boximator generation results over the base model.

Title: NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

• Authors: Jingyuan Sun, Mingxiao Li, Zijiao Chen, Marie-Francine Moens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01590
• Pdf URL: https://arxiv.org/pdf/2402.01590
• Copy Paste: [[2402.01590]] NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties(https://arxiv.org/abs/2402.01590)
• Keywords: diffusion
• Abstract: In the pursuit to understand the intricacies of human brain's visual processing, reconstructing dynamic visual experiences from brain activities emerges as a challenging yet fascinating endeavor. While recent advancements have achieved success in reconstructing static images from non-invasive brain recordings, the domain of translating continuous brain activities into video format remains underexplored. In this work, we introduce NeuroCine, a novel dual-phase framework to targeting the inherent challenges of decoding fMRI data, such as noises, spatial redundancy and temporal lags. This framework proposes spatial masking and temporal interpolation-based augmentation for contrastive learning fMRI representations and a diffusion model enhanced by dependent prior noise for video generation. Tested on a publicly available fMRI dataset, our method shows promising results, outperforming the previous state-of-the-art models by a notable margin of ${20.97\%}$, ${31.00\%}$ and ${12.30\%}$ respectively on decoding the brain activities of three subjects in the fMRI dataset, as measured by SSIM. Additionally, our attention analysis suggests that the model aligns with existing brain structures and functions, indicating its biological plausibility and interpretability.

Title: Style Vectors for Steering Generative Large Language Model

• Authors: Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, Tobias Hecking
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01618
• Pdf URL: https://arxiv.org/pdf/2402.01618
• Copy Paste: [[2402.01618]] Style Vectors for Steering Generative Large Language Model(https://arxiv.org/abs/2402.01618)
• Keywords: generative
• Abstract: This research explores strategies for steering the output of large language models (LLMs) towards specific styles, such as sentiment, emotion, or writing style, by adding style vectors to the activations of hidden layers during text generation. We show that style vectors can be simply computed from recorded layer activations for input texts in a specific style in contrast to more complex training-based approaches. Through a series of experiments, we demonstrate the effectiveness of activation engineering using such style vectors to influence the style of generated text in a nuanced and parameterisable way, distinguishing it from prompt engineering. The presented research constitutes a significant step towards developing more adaptive and effective AI-empowered interactive systems.

Title: KB-Plugin: A Plug-and-play Framework for Large Language Models to Induce Programs over Low-resourced Knowledge Bases

• Authors: Jiajie Zhang, Shulin Cao, Linmei Hu, Ling Feng, Lei Hou, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01619
• Pdf URL: https://arxiv.org/pdf/2402.01619
• Copy Paste: [[2402.01619]] KB-Plugin: A Plug-and-play Framework for Large Language Models to Induce Programs over Low-resourced Knowledge Bases(https://arxiv.org/abs/2402.01619)
• Keywords: self-supervised
• Abstract: Program induction (PI) has become a promising paradigm for using knowledge bases (KBs) to help large language models (LLMs) answer complex knowledge-intensive questions. Nonetheless, PI typically relies on a large number of parallel question-program pairs to make the LLM aware of the schema of the given KB, and is thus challenging for many low-resourced KBs that lack annotated data. To this end, we propose KB-Plugin, a plug-and-play framework that enables LLMs to induce programs over any low-resourced KB. Firstly, KB-Plugin adopts self-supervised learning to encode the detailed schema information of a given KB into a pluggable module, namely schema plugin. Secondly, KB-Plugin utilizes abundant annotated data from a rich-resourced KB to train another pluggable module, namely PI plugin, which can help the LLM extract question-relevant schema information from the schema plugin of any KB and utilize this information to induce programs over this KB. Experiments on five heterogeneous KBQA datasets show that KB-Plugin achieves better or comparable performance with 25$\times$ smaller backbone LLM compared to SoTA PI methods for low-resourced KBs, and even approaches the performance of supervised methods. Our code and data are available at https://github.com/THU-KEG/KB-Plugin.

Title: Stochastic Two Points Method for Deep Model Zeroth-order Optimization

• Authors: Yijiang Pang, Jiayu Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01621
• Pdf URL: https://arxiv.org/pdf/2402.01621
• Copy Paste: [[2402.01621]] Stochastic Two Points Method for Deep Model Zeroth-order Optimization(https://arxiv.org/abs/2402.01621)
• Keywords: foundation model
• Abstract: Large foundation models, such as large language models, have performed exceptionally well in various application scenarios. Building or fully fine-tuning such large models is usually prohibitive due to either hardware budget or lack of access to backpropagation. The zeroth-order methods offer a promising direction for tackling this challenge, where only forward passes are needed to update the model. This paper introduces an efficient Stochastic Two-Point (S2P) approach within the gradient-free regime. We present the theoretical convergence properties of S2P under the general and relaxed smoothness assumptions. The theoretical properties also shed light on a faster and more stable S2P variant, Accelerated S2P (AS2P), through exploiting our new convergence properties that better represent the dynamics of deep models in training. Our comprehensive empirical results show that AS2P is highly effective in optimizing objectives for large deep models, including language models, and outperforms standard methods across various model types and scales, with 2 $\times$ speed-up in training over most conducted tasks.