2024-02-02

Title: TrackGPT -- A generative pre-trained transformer for cross-domain entity trajectory forecasting

• Authors: Nicholas Stroh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00066
• Pdf URL: https://arxiv.org/pdf/2402.00066
• Copy Paste: [[2402.00066]] TrackGPT -- A generative pre-trained transformer for cross-domain entity trajectory forecasting(https://arxiv.org/abs/2402.00066)
• Keywords: generative
• Abstract: The forecasting of entity trajectories at future points in time is a critical capability gap in applications across both Commercial and Defense sectors. Transformers, and specifically Generative Pre-trained Transformer (GPT) networks have recently revolutionized several fields of Artificial Intelligence, most notably Natural Language Processing (NLP) with the advent of Large Language Models (LLM) like OpenAI's ChatGPT. In this research paper, we introduce TrackGPT, a GPT-based model for entity trajectory forecasting that has shown utility across both maritime and air domains, and we expect to perform well in others. TrackGPT stands as a pioneering GPT model capable of producing accurate predictions across diverse entity time series datasets, demonstrating proficiency in generating both long-term forecasts with sustained accuracy and short-term forecasts with high precision. We present benchmarks against state-of-the-art deep learning techniques, showing that TrackGPT's forecasting capability excels in terms of accuracy, reliability, and modularity. Importantly, TrackGPT achieves these results while remaining domain-agnostic and requiring minimal data features (only location and time) compared to models achieving similar performance. In conclusion, our findings underscore the immense potential of applying GPT architectures to the task of entity trajectory forecasting, exemplified by the innovative TrackGPT model.

Title: Multimodal Neurodegenerative Disease Subtyping Explained by ChatGPT

• Authors: Diego Machado Reyes, Hanqing Chao, Juergen Hahn, Li Shen, Pingkun Yan
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00137
• Pdf URL: https://arxiv.org/pdf/2402.00137
• Copy Paste: [[2402.00137]] Multimodal Neurodegenerative Disease Subtyping Explained by ChatGPT(https://arxiv.org/abs/2402.00137)
• Keywords: generative
• Abstract: Alzheimer's disease (AD) is the most prevalent neurodegenerative disease; yet its currently available treatments are limited to stopping disease progression. Moreover, effectiveness of these treatments is not guaranteed due to the heterogenetiy of the disease. Therefore, it is essential to be able to identify the disease subtypes at a very early stage. Current data driven approaches are able to classify the subtypes at later stages of AD or related disorders, but struggle when predicting at the asymptomatic or prodromal stage. Moreover, most existing models either lack explainability behind the classification or only use a single modality for the assessment, limiting scope of its analysis. Thus, we propose a multimodal framework that uses early-stage indicators such as imaging, genetics and clinical assessments to classify AD patients into subtypes at early stages. Similarly, we build prompts and use large language models, such as ChatGPT, to interpret the findings of our model. In our framework, we propose a tri-modal co-attention mechanism (Tri-COAT) to explicitly learn the cross-modal feature associations. Our proposed model outperforms baseline models and provides insight into key cross-modal feature associations supported by known biological mechanisms.

Title: Fully Data-Driven Model for Increasing Sampling Rate Frequency of Seismic Data using Super-Resolution Generative Adversarial Networks

• Authors: Navid Gholizadeh, Javad Katebi
• Subjects: cs.LG, eess.SP, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2402.00153
• Pdf URL: https://arxiv.org/pdf/2402.00153
• Copy Paste: [[2402.00153]] Fully Data-Driven Model for Increasing Sampling Rate Frequency of Seismic Data using Super-Resolution Generative Adversarial Networks(https://arxiv.org/abs/2402.00153)
• Keywords: generative
• Abstract: High-quality data is one of the key requirements for any engineering application. In earthquake engineering practice, accurate data is pivotal in predicting the response of structure or damage detection process in an Structural Health Monitoring (SHM) application with less uncertainty. However, obtaining high-resolution data is fraught with challenges, such as significant costs, extensive data channels, and substantial storage requirements. To address these challenges, this study employs super-resolution generative adversarial networks (SRGANs) to improve the resolution of time-history data such as the data obtained by a sensor network in an SHM application, marking the first application of SRGANs in earthquake engineering domain. The time-series data are transformed into RGB values, converting raw data into images. SRGANs are then utilized to upscale these low-resolution images, thereby enhancing the overall sensor resolution. This methodology not only offers potential reductions in data storage requirements but also simplifies the sensor network, which could result in lower installation and maintenance costs. The proposed SRGAN method is rigorously evaluated using real seismic data, and its performance is compared with traditional enhancement techniques. The findings of this study pave the way for cost-effective and efficient improvements in the resolution of sensors used in SHM systems, with promising implications for the safety and sustainability of infrastructures worldwide.

Title: De-identification is not always enough

• Authors: Atiquer Rahman Sarkar, Yao-Shun Chuang, Noman Mohammed, Xiaoqian Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00179
• Pdf URL: https://arxiv.org/pdf/2402.00179
• Copy Paste: [[2402.00179]] De-identification is not always enough(https://arxiv.org/abs/2402.00179)
• Keywords: generative
• Abstract: For sharing privacy-sensitive data, de-identification is commonly regarded as adequate for safeguarding privacy. Synthetic data is also being considered as a privacy-preserving alternative. Recent successes with numerical and tabular data generative models and the breakthroughs in large generative language models raise the question of whether synthetically generated clinical notes could be a viable alternative to real notes for research purposes. In this work, we demonstrated that (i) de-identification of real clinical notes does not protect records against a membership inference attack, (ii) proposed a novel approach to generate synthetic clinical notes using the current state-of-the-art large language models, (iii) evaluated the performance of the synthetically generated notes in a clinical domain task, and (iv) proposed a way to mount a membership inference attack where the target model is trained with synthetic data. We observed that when synthetically generated notes closely match the performance of real data, they also exhibit similar privacy concerns to the real data. Whether other approaches to synthetically generated clinical notes could offer better trade-offs and become a better alternative to sensitive real notes warrants further investigation.

Title: LRDif: Diffusion Models for Under-Display Camera Emotion Recognition

• Authors: Zhifeng Wang, Kaihao Zhang, Ramesh Sankaranarayana
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00250
• Pdf URL: https://arxiv.org/pdf/2402.00250
• Copy Paste: [[2402.00250]] LRDif: Diffusion Models for Under-Display Camera Emotion Recognition(https://arxiv.org/abs/2402.00250)
• Keywords: diffusion
• Abstract: This study introduces LRDif, a novel diffusion-based framework designed specifically for facial expression recognition (FER) within the context of under-display cameras (UDC). To address the inherent challenges posed by UDC's image degradation, such as reduced sharpness and increased noise, LRDif employs a two-stage training strategy that integrates a condensed preliminary extraction network (FPEN) and an agile transformer network (UDCformer) to effectively identify emotion labels from UDC images. By harnessing the robust distribution mapping capabilities of Diffusion Models (DMs) and the spatial dependency modeling strength of transformers, LRDif effectively overcomes the obstacles of noise and distortion inherent in UDC environments. Comprehensive experiments on standard FER datasets including RAF-DB, KDEF, and FERPlus, LRDif demonstrate state-of-the-art performance, underscoring its potential in advancing FER applications. This work not only addresses a significant gap in the literature by tackling the UDC challenge in FER but also sets a new benchmark for future research in the field.

Title: Self-supervised learning of video representations from a child's perspective

• Authors: A. Emin Orhan, Wentao Wang, Alex N. Wang, Mengye Ren, Brenden M. Lake
• Subjects: cs.CV, cs.LG, cs.NE, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2402.00300
• Pdf URL: https://arxiv.org/pdf/2402.00300
• Copy Paste: [[2402.00300]] Self-supervised learning of video representations from a child's perspective(https://arxiv.org/abs/2402.00300)
• Keywords: self-supervised
• Abstract: Children learn powerful internal models of the world around them from a few years of egocentric visual experience. Can such internal models be learned from a child's visual experience with highly generic learning algorithms or do they require strong inductive biases? Recent advances in collecting large-scale, longitudinal, developmentally realistic video datasets and generic self-supervised learning (SSL) algorithms are allowing us to begin to tackle this nature vs. nurture question. However, existing work typically focuses on image-based SSL algorithms and visual capabilities that can be learned from static images (e.g. object recognition), thus ignoring temporal aspects of the world. To close this gap, here we train self-supervised video models on longitudinal, egocentric headcam recordings collected from a child over a two year period in their early development (6-31 months). The resulting models are highly effective at facilitating the learning of action concepts from a small number of labeled examples; they have favorable data size scaling properties; and they display emergent video interpolation capabilities. Video models also learn more robust object representations than image-based models trained with the exact same data. These results suggest that important temporal aspects of a child's internal model of the world may be learnable from their visual experience using highly generic learning algorithms and without strong inductive biases.

Title: Machine Unlearning for Image-to-Image Generative Models

• Authors: Guihong Li, Hsiang Hsu, Chun-Fu (Richard)Chen, Radu Marculescu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00351
• Pdf URL: https://arxiv.org/pdf/2402.00351
• Copy Paste: [[2402.00351]] Machine Unlearning for Image-to-Image Generative Models(https://arxiv.org/abs/2402.00351)
• Keywords: generative
• Abstract: Machine unlearning has emerged as a new paradigm to deliberately forget data samples from a given model in order to adhere to stringent regulations. However, existing machine unlearning methods have been primarily focused on classification models, leaving the landscape of unlearning for generative models relatively unexplored. This paper serves as a bridge, addressing the gap by providing a unifying framework of machine unlearning for image-to-image generative models. Within this framework, we propose a computationally-efficient algorithm, underpinned by rigorous theoretical analysis, that demonstrates negligible performance degradation on the retain samples, while effectively removing the information from the forget samples. Empirical studies on two large-scale datasets, ImageNet-1K and Places-365, further show that our algorithm does not rely on the availability of the retain samples, which further complies with data retention policy. To our best knowledge, this work is the first that represents systemic, theoretical, empirical explorations of machine unlearning specifically tailored for image-to-image generative models. Our code is available at https://github.com/jpmorganchase/l2l-generator-unlearning.

Title: InfMAE: A Foundation Model in Infrared Modality

• Authors: Fangcen Liu, Chenqiang Gao, Yaming Zhang, Junjie Guo, Jinhao Wang, Deyu Meng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00407
• Pdf URL: https://arxiv.org/pdf/2402.00407
• Copy Paste: [[2402.00407]] InfMAE: A Foundation Model in Infrared Modality(https://arxiv.org/abs/2402.00407)
• Keywords: self-supervised, foundation model
• Abstract: In recent years, the foundation models have swept the computer vision field and facilitated the development of various tasks within different modalities. However, it remains an open question on how to design an infrared foundation model. In this paper, we propose InfMAE, a foundation model in infrared modality. We release an infrared dataset, called Inf30 to address the problem of lacking large-scale data for self-supervised learning in the infrared vision community. Besides, we design an information-aware masking strategy, which is suitable for infrared images. This masking strategy allows for a greater emphasis on the regions with richer information in infrared images during the self-supervised learning process, which is conducive to learning the generalized representation. In addition, we adopt a multi-scale encoder to enhance the performance of the pre-trained encoders in downstream tasks. Finally, based on the fact that infrared images do not have a lot of details and texture information, we design an infrared decoder module, which further improves the performance of downstream tasks. Extensive experiments show that our proposed method InfMAE outperforms other supervised methods and self-supervised learning methods in three downstream tasks. Our code will be made public at https://github.com/liufangcen/InfMAE.

Title: Short: Benchmarking transferable adversarial attacks

• Authors: Zhibo Jin, Jiayu Zhang, Zhiyu Zhu, Huaming Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00418
• Pdf URL: https://arxiv.org/pdf/2402.00418
• Copy Paste: [[2402.00418]] Short: Benchmarking transferable adversarial attacks(https://arxiv.org/abs/2402.00418)
• Keywords: generative
• Abstract: The robustness of deep learning models against adversarial attacks remains a pivotal concern. This study presents, for the first time, an exhaustive review of the transferability aspect of adversarial attacks. It systematically categorizes and critically evaluates various methodologies developed to augment the transferability of adversarial attacks. This study encompasses a spectrum of techniques, including Generative Structure, Semantic Similarity, Gradient Editing, Target Modification, and Ensemble Approach. Concurrently, this paper introduces a benchmark framework \textit{TAA-Bench}, integrating ten leading methodologies for adversarial attack transferability, thereby providing a standardized and systematic platform for comparative analysis across diverse model architectures. Through comprehensive scrutiny, we delineate the efficacy and constraints of each method, shedding light on their underlying operational principles and practical utility. This review endeavors to be a quintessential resource for both scholars and practitioners in the field, charting the complex terrain of adversarial transferability and setting a foundation for future explorations in this vital sector. The associated codebase is accessible at: https://github.com/KxPlaug/TAA-Bench

Title: A Survey of Data-Efficient Graph Learning

• Authors: Wei Ju, Siyu Yi, Yifan Wang, Qingqing Long, Junyu Luo, Zhiping Xiao, Ming Zhang
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.00447
• Pdf URL: https://arxiv.org/pdf/2402.00447
• Copy Paste: [[2402.00447]] A Survey of Data-Efficient Graph Learning(https://arxiv.org/abs/2402.00447)
• Keywords: self-supervised
• Abstract: Graph-structured data, prevalent in domains ranging from social networks to biochemical analysis, serve as the foundation for diverse real-world systems. While graph neural networks demonstrate proficiency in modeling this type of data, their success is often reliant on significant amounts of labeled data, posing a challenge in practical scenarios with limited annotation resources. To tackle this problem, tremendous efforts have been devoted to enhancing graph machine learning performance under low-resource settings by exploring various approaches to minimal supervision. In this paper, we introduce a novel concept of Data-Efficient Graph Learning (DEGL) as a research frontier, and present the first survey that summarizes the current progress of DEGL. We initiate by highlighting the challenges inherent in training models with large labeled data, paving the way for our exploration into DEGL. Next, we systematically review recent advances on this topic from several key aspects, including self-supervised graph learning, semi-supervised graph learning, and few-shot graph learning. Also, we state promising directions for future research, contributing to the evolution of graph machine learning.

Title: Dual-Student Knowledge Distillation Networks for Unsupervised Anomaly Detection

• Authors: Liyi Yao, Shaobing Gao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00448
• Pdf URL: https://arxiv.org/pdf/2402.00448
• Copy Paste: [[2402.00448]] Dual-Student Knowledge Distillation Networks for Unsupervised Anomaly Detection(https://arxiv.org/abs/2402.00448)
• Keywords: anomaly
• Abstract: Due to the data imbalance and the diversity of defects, student-teacher networks (S-T) are favored in unsupervised anomaly detection, which explores the discrepancy in feature representation derived from the knowledge distillation process to recognize anomalies. However, vanilla S-T network is not stable. Employing identical structures to construct the S-T network may weaken the representative discrepancy on anomalies. But using different structures can increase the likelihood of divergent performance on normal data. To address this problem, we propose a novel dual-student knowledge distillation (DSKD) architecture. Different from other S-T networks, we use two student networks a single pre-trained teacher network, where the students have the same scale but inverted structures. This framework can enhance the distillation effect to improve the consistency in recognition of normal data, and simultaneously introduce diversity for anomaly representation. To explore high-dimensional semantic information to capture anomaly clues, we employ two strategies. First, a pyramid matching mode is used to perform knowledge distillation on multi-scale feature maps in the intermediate layers of networks. Second, an interaction is facilitated between the two student networks through a deep feature embedding module, which is inspired by real-world group discussions. In terms of classification, we obtain pixel-wise anomaly segmentation maps by measuring the discrepancy between the output feature maps of the teacher and student networks, from which an anomaly score is computed for sample-wise determination. We evaluate DSKD on three benchmark datasets and probe the effects of internal modules through ablation experiments. The results demonstrate that DSKD can achieve exceptional performance on small models like ResNet18 and effectively improve vanilla S-T networks.

Title: Masked Conditional Diffusion Model for Enhancing Deepfake Detection

• Authors: Tiewen Chen, Shanmin Yang, Shu Hu, Zhenghan Fang, Ying Fu, Xi Wu, Xin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00541
• Pdf URL: https://arxiv.org/pdf/2402.00541
• Copy Paste: [[2402.00541]] Masked Conditional Diffusion Model for Enhancing Deepfake Detection(https://arxiv.org/abs/2402.00541)
• Keywords: diffusion
• Abstract: Recent studies on deepfake detection have achieved promising results when training and testing faces are from the same dataset. However, their results severely degrade when confronted with forged samples that the model has not yet seen during training. In this paper, deepfake data to help detect deepfakes. this paper present we put a new insight into diffusion model-based data augmentation, and propose a Masked Conditional Diffusion Model (MCDM) for enhancing deepfake detection. It generates a variety of forged faces from a masked pristine one, encouraging the deepfake detection model to learn generic and robust representations without overfitting to special artifacts. Extensive experiments demonstrate that forgery images generated with our method are of high quality and helpful to improve the performance of deepfake detection models.

Title: Diffusion-based Light Field Synthesis

• Authors: Ruisheng Gao, Yutong Liu, Zeyu Xiao, Zhiwei Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00575
• Pdf URL: https://arxiv.org/pdf/2402.00575
• Copy Paste: [[2402.00575]] Diffusion-based Light Field Synthesis(https://arxiv.org/abs/2402.00575)
• Keywords: diffusion, generative
• Abstract: Light fields (LFs), conducive to comprehensive scene radiance recorded across angular dimensions, find wide applications in 3D reconstruction, virtual reality, and computational photography.However, the LF acquisition is inevitably time-consuming and resource-intensive due to the mainstream acquisition strategy involving manual capture or laborious software synthesis.Given such a challenge, we introduce LFdiff, a straightforward yet effective diffusion-based generative framework tailored for LF synthesis, which adopts only a single RGB image as input.LFdiff leverages disparity estimated by a monocular depth estimation network and incorporates two distinctive components: a novel condition scheme and a noise estimation network tailored for LF data.Specifically, we design a position-aware warping condition scheme, enhancing inter-view geometry learning via a robust conditional signal.We then propose DistgUnet, a disentanglement-based noise estimation network, to harness comprehensive LF representations.Extensive experiments demonstrate that LFdiff excels in synthesizing visually pleasing and disparity-controllable light fields with enhanced generalization capability.Additionally, comprehensive results affirm the broad applicability of the generated LF data, spanning applications like LF super-resolution and refocusing.

Title: CapHuman: Capture Your Moments in Parallel Universes

• Authors: Chao Liang, Fan Ma, Linchao Zhu, Yingying Deng, Yi Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00627
• Pdf URL: https://arxiv.org/pdf/2402.00627
• Copy Paste: [[2402.00627]] CapHuman: Capture Your Moments in Parallel Universes(https://arxiv.org/abs/2402.00627)
• Keywords: diffusion, generative
• Abstract: We concentrate on a novel human-centric image synthesis task, that is, given only one reference facial photograph, it is expected to generate specific individual images with diverse head positions, poses, and facial expressions in different contexts. To accomplish this goal, we argue that our generative model should be capable of the following favorable characteristics: (1) a strong visual and semantic understanding of our world and human society for basic object and human image generation. (2) generalizable identity preservation ability. (3) flexible and fine-grained head control. Recently, large pre-trained text-to-image diffusion models have shown remarkable results, serving as a powerful generative foundation. As a basis, we aim to unleash the above two capabilities of the pre-trained model. In this work, we present a new framework named CapHuman. We embrace the ``encode then learn to align" paradigm, which enables generalizable identity preservation for new individuals without cumbersome tuning at inference. CapHuman encodes identity features and then learns to align them into the latent space. Moreover, we introduce the 3D facial prior to equip our model with control over the human head in a flexible and 3D-consistent manner. Extensive qualitative and quantitative analyses demonstrate our CapHuman can produce well-identity-preserved, photo-realistic, and high-fidelity portraits with content-rich representations and various head renditions, superior to established baselines. Code and checkpoint will be released at https://github.com/VamosC/CapHuman.

Title: SeFi-IDE: Semantic-Fidelity Identity Embedding for Personalized Diffusion-Based Generation

• Authors: Yang Li, Songlin Yang, Wei Wang, Jing Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00631
• Pdf URL: https://arxiv.org/pdf/2402.00631
• Copy Paste: [[2402.00631]] SeFi-IDE: Semantic-Fidelity Identity Embedding for Personalized Diffusion-Based Generation(https://arxiv.org/abs/2402.00631)
• Keywords: diffusion, generative
• Abstract: Advanced diffusion-based Text-to-Image (T2I) models, such as the Stable Diffusion Model, have made significant progress in generating diverse and high-quality images using text prompts alone. However, T2I models are unable to accurately map identities (IDs) when non-famous users require personalized image generation. The main problem is that existing T2I models do not learn the ID-image alignments of new users. The previous methods either failed to accurately fit the face region or lost the interactive generative ability with other existing concepts in T2I models (i.e., unable to generate other concepts described in given prompts such as scenes, actions, and facial attributes). In this paper, we focus on accurate and semantic-fidelity ID embedding into the Stable Diffusion Model for personalized generation. We address this challenge from two perspectives: face-wise region fitting, and semantic-fidelity token optimization. Specifically, we first visualize the attention overfit problem, and propose a face-wise attention loss to fit the face region instead of the whole target image. This key trick significantly enhances the ID accuracy and interactive generative ability with other existing concepts. Then, we optimize one ID representation as multiple per-stage tokens where each token contains two disentangled features. This expansion of the textual conditioning space enhances semantic-fidelity control. Extensive experiments validate that our results exhibit superior ID accuracy and manipulation ability compared to previous methods.

Title: Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning

• Authors: Jitao Sang, Yuhang Wang, Jing Zhang, Yanxu Zhu, Chao Kong, Junhong Ye, Shuyu Wei, Jinlin Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00667
• Pdf URL: https://arxiv.org/pdf/2402.00667
• Copy Paste: [[2402.00667]] Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning(https://arxiv.org/abs/2402.00667)
• Keywords: in-context
• Abstract: This paper presents a follow-up study to OpenAI's recent superalignment work on Weak-to-Strong Generalization (W2SG). Superalignment focuses on ensuring that high-level AI systems remain consistent with human values and intentions when dealing with complex, high-risk tasks. The W2SG framework has opened new possibilities for empirical research in this evolving field. Our study simulates two phases of superalignment under the W2SG framework: the development of general superhuman models and the progression towards superintelligence. In the first phase, based on human supervision, the quality of weak supervision is enhanced through a combination of scalable oversight and ensemble learning, reducing the capability gap between weak teachers and strong students. In the second phase, an automatic alignment evaluator is employed as the weak supervisor. By recursively updating this auto aligner, the capabilities of the weak teacher models are synchronously enhanced, achieving weak-to-strong supervision over stronger student models.We also provide an initial validation of the proposed approach for the first phase. Using the SciQ task as example, we explore ensemble learning for weak teacher models through bagging and boosting. Scalable oversight is explored through two auxiliary settings: human-AI interaction and AI-AI debate. Additionally, the paper discusses the impact of improved weak supervision on enhancing weak-to-strong generalization based on in-context learning. Experiment code and dataset will be released at https://github.com/ADaM-BJTU/W2SG.

Title: Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders

• Authors: Yingji Zhang, Danilo S. Carvalho, Marco Valentino, Ian Pratt-Hartmann, Andre Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00723
• Pdf URL: https://arxiv.org/pdf/2402.00723
• Copy Paste: [[2402.00723]] Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders(https://arxiv.org/abs/2402.00723)
• Keywords: generative
• Abstract: Achieving precise semantic control over the latent spaces of Variational AutoEncoders (VAEs) holds significant value for downstream tasks in NLP as the underlying generative mechanisms could be better localised, explained and improved upon. Recent research, however, has struggled to achieve consistent results, primarily due to the inevitable loss of semantic information in the variational bottleneck and limited control over the decoding mechanism. To overcome these challenges, we investigate discrete latent spaces in Vector Quantized Variational AutoEncoders (VQVAEs) to improve semantic control and generation in Transformer-based VAEs. In particular, We propose T5VQVAE, a novel model that leverages the controllability of VQVAEs to guide the self-attention mechanism in T5 at the token-level, exploiting its full generalization capabilities. Experimental results indicate that T5VQVAE outperforms existing state-of-the-art VAE models, including Optimus, in terms of controllability and preservation of semantic information across different tasks such as auto-encoding of sentences and mathematical expressions, text transfer, and inference. Moreover, T5VQVAE exhibits improved inference capabilities, suggesting potential applications for downstream natural language and symbolic reasoning tasks.

Title: Benefits of Transformer: In-Context Learning in Linear Regression Tasks with Unstructured Data

• Authors: Yue Xing, Xiaofeng Lin, Namjoon Suh, Qifan Song, Guang Cheng
• Subjects: cs.LG, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.00743
• Pdf URL: https://arxiv.org/pdf/2402.00743
• Copy Paste: [[2402.00743]] Benefits of Transformer: In-Context Learning in Linear Regression Tasks with Unstructured Data(https://arxiv.org/abs/2402.00743)
• Keywords: in-context
• Abstract: In practice, it is observed that transformer-based models can learn concepts in context in the inference stage. While existing literature, e.g., \citet{zhang2023trained,huang2023context}, provide theoretical explanations on this in-context learning ability, they assume the input $x_i$ and the output $y_i$ for each sample are embedded in the same token (i.e., structured data). However, in reality, they are presented in two tokens (i.e., unstructured data \cite{wibisono2023role}). In this case, this paper conducts experiments in linear regression tasks to study the benefits of the architecture of transformers and provides some corresponding theoretical intuitions to explain why the transformer can learn from unstructured data. We study the exact components in a transformer that facilitate the in-context learning. In particular, we observe that (1) a transformer with two layers of softmax (self-)attentions with look-ahead attention mask can learn from the prompt if $y_i$ is in the token next to $x_i$ for each example; (2) positional encoding can further improve the performance; and (3) multi-head attention with a high input embedding dimension has a better prediction performance than single-head attention.

Title: Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement

• Authors: Xin Quan, Marco Valentino, Louise A. Dennis, André Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00745
• Pdf URL: https://arxiv.org/pdf/2402.00745
• Copy Paste: [[2402.00745]] Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement(https://arxiv.org/abs/2402.00745)
• Keywords: in-context
• Abstract: An increasing amount of research in Natural Language Inference (NLI) focuses on the application and evaluation of Large Language Models (LLMs) and their reasoning capabilities. Despite their success, however, LLMs are still prone to factual errors and inconsistencies in their explanations, offering limited control and interpretability for inference in complex domains. In this paper, we focus on ethical NLI, investigating how hybrid neuro-symbolic techniques can enhance the logical validity and alignment of ethical explanations produced by LLMs. Specifically, we present an abductive-deductive framework named Logic-Explainer, which integrates LLMs with an external backward-chaining solver to refine step-wise natural language explanations and jointly verify their correctness, reduce incompleteness and minimise redundancy. An extensive empirical analysis demonstrates that Logic-Explainer can improve explanations generated via in-context learning methods and Chain-of-Thought (CoT) on challenging ethical NLI tasks, while, at the same time, producing formal proofs describing and supporting models' reasoning. As ethical NLI requires commonsense reasoning to identify underlying moral violations, our results suggest the effectiveness of neuro-symbolic methods for multi-step NLI more broadly, opening new opportunities to enhance the logical consistency, reliability, and alignment of LLMs.

Title: Unlearnable Algorithms for In-context Learning

• Authors: Andrei Muresanu, Anvith Thudi, Michael R. Zhang, Nicolas Papernot
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00751
• Pdf URL: https://arxiv.org/pdf/2402.00751
• Copy Paste: [[2402.00751]] Unlearnable Algorithms for In-context Learning(https://arxiv.org/abs/2402.00751)
• Keywords: in-context
• Abstract: Machine unlearning is a desirable operation as models get increasingly deployed on data with unknown provenance. However, achieving exact unlearning -- obtaining a model that matches the model distribution when the data to be forgotten was never used -- is challenging or inefficient, often requiring significant retraining. In this paper, we focus on efficient unlearning methods for the task adaptation phase of a pretrained large language model (LLM). We observe that an LLM's ability to do in-context learning for task adaptation allows for efficient exact unlearning of task adaptation training data. We provide an algorithm for selecting few-shot training examples to prepend to the prompt given to an LLM (for task adaptation), ERASE, whose unlearning operation cost is independent of model and dataset size, meaning it scales to large models and datasets. We additionally compare our approach to fine-tuning approaches and discuss the trade-offs between the two approaches. This leads us to propose a new holistic measure of unlearning cost which accounts for varying inference costs, and conclude that in-context learning can often be more favourable than fine-tuning for deployments involving unlearning requests.

Title: Building Expressive and Tractable Probabilistic Generative Models: A Review

• Authors: Sahil Sidheekh, Sriraam Natarajan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00759
• Pdf URL: https://arxiv.org/pdf/2402.00759
• Copy Paste: [[2402.00759]] Building Expressive and Tractable Probabilistic Generative Models: A Review(https://arxiv.org/abs/2402.00759)
• Keywords: generative
• Abstract: We present a comprehensive survey of the advancements and techniques in the field of tractable probabilistic generative modeling, primarily focusing on Probabilistic Circuits (PCs). We provide a unified perspective on the inherent trade-offs between expressivity and the tractability, highlighting the design principles and algorithmic extensions that have enabled building expressive and efficient PCs, and provide a taxonomy of the field. We also discuss recent efforts to build deep and hybrid PCs by fusing notions from deep neural models, and outline the challenges and open questions that can guide future research in this evolving field.

Title: AnimateLCM: Accelerating the Animation of Personalized Diffusion Models and Adapters with Decoupled Consistency Learning

• Authors: Fu-Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, Hongsheng Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00769
• Pdf URL: https://arxiv.org/pdf/2402.00769
• Copy Paste: [[2402.00769]] AnimateLCM: Accelerating the Animation of Personalized Diffusion Models and Adapters with Decoupled Consistency Learning(https://arxiv.org/abs/2402.00769)
• Keywords: diffusion
• Abstract: Video diffusion models has been gaining increasing attention for its ability to produce videos that are both coherent and of high fidelity. However, the iterative denoising process makes it computationally intensive and time-consuming, thus limiting its applications. Inspired by the Consistency Model (CM) that distills pretrained image diffusion models to accelerate the sampling with minimal steps and its successful extension Latent Consistency Model (LCM) on conditional image generation, we propose AnimateLCM, allowing for high-fidelity video generation within minimal steps. Instead of directly conducting consistency learning on the raw video dataset, we propose a decoupled consistency learning strategy that decouples the distillation of image generation priors and motion generation priors, which improves the training efficiency and enhance the generation visual quality. Additionally, to enable the combination of plug-and-play adapters in stable diffusion community to achieve various functions (e.g., ControlNet for controllable generation). we propose an efficient strategy to adapt existing adapters to our distilled text-conditioned video consistency model or train adapters from scratch without harming the sampling speed. We validate the proposed strategy in image-conditioned video generation and layout-conditioned video generation, all achieving top-performing results. Experimental results validate the effectiveness of our proposed method. Code and weights will be made public. More details are available at https://github.com/G-U-N/AnimateLCM.

Title: ReAGent: Towards A Model-agnostic Feature Attribution Method for Generative Language Models

• Authors: Zhixue Zhao, Boxuan Shan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00794
• Pdf URL: https://arxiv.org/pdf/2402.00794
• Copy Paste: [[2402.00794]] ReAGent: Towards A Model-agnostic Feature Attribution Method for Generative Language Models(https://arxiv.org/abs/2402.00794)
• Keywords: generative
• Abstract: Feature attribution methods (FAs), such as gradients and attention, are widely employed approaches to derive the importance of all input features to the model predictions. Existing work in natural language processing has mostly focused on developing and testing FAs for encoder-only language models (LMs) in classification tasks. However, it is unknown if it is faithful to use these FAs for decoder-only models on text generation, due to the inherent differences between model architectures and task settings respectively. Moreover, previous work has demonstrated that there is no `one-wins-all' FA across models and tasks. This makes the selection of a FA computationally expensive for large LMs since input importance derivation often requires multiple forward and backward passes including gradient computations that might be prohibitive even with access to large compute. To address these issues, we present a model-agnostic FA for generative LMs called Recursive Attribution Generator (ReAGent). Our method updates the token importance distribution in a recursive manner. For each update, we compute the difference in the probability distribution over the vocabulary for predicting the next token between using the original input and using a modified version where a part of the input is replaced with RoBERTa predictions. Our intuition is that replacing an important token in the context should have resulted in a larger change in the model's confidence in predicting the token than replacing an unimportant token. Our method can be universally applied to any generative LM without accessing internal model weights or additional training and fine-tuning, as most other FAs require. We extensively compare the faithfulness of ReAGent with seven popular FAs across six decoder-only LMs of various sizes. The results show that our method consistently provides more faithful token importance distributions.

Title: LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law

• Authors: Toni J.B. Liu, Nicolas Boullé, Raphaël Sarfati, Christopher J. Earls
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00795
• Pdf URL: https://arxiv.org/pdf/2402.00795
• Copy Paste: [[2402.00795]] LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law(https://arxiv.org/abs/2402.00795)
• Keywords: in-context
• Abstract: Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. In this paper, we study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our results show that LLaMA 2, a language model trained primarily on texts, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.

Title: Distilling Conditional Diffusion Models for Offline Reinforcement Learning through Trajectory Stitching

• Authors: Shangzhe Li, Xinhua Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00807
• Pdf URL: https://arxiv.org/pdf/2402.00807
• Copy Paste: [[2402.00807]] Distilling Conditional Diffusion Models for Offline Reinforcement Learning through Trajectory Stitching(https://arxiv.org/abs/2402.00807)
• Keywords: diffusion, generative
• Abstract: Deep generative models have recently emerged as an effective approach to offline reinforcement learning. However, their large model size poses challenges in computation. We address this issue by proposing a knowledge distillation method based on data augmentation. In particular, high-return trajectories are generated from a conditional diffusion model, and they are blended with the original trajectories through a novel stitching algorithm that leverages a new reward generator. Applying the resulting dataset to behavioral cloning, the learned shallow policy whose size is much smaller outperforms or nearly matches deep generative planners on several D4RL benchmarks.

Title: Position Paper: Bayesian Deep Learning in the Age of Large-Scale AI

• Authors: Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, Aliaksandr Hubin, Alexander Immer, Theofanis Karaletsos, Mohammad Emtiyaz Khan, Agustinus Kristiadi, Yingzhen Li, Jose Miguel Hernandez Lobato, Stephan Mandt, Christopher Nemeth, Michael A. Osborne, Tim G. J. Rudner, David Rügamer, Yee Whye Teh, Max Welling, Andrew Gordon Wilson, Ruqi Zhang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.00809
• Pdf URL: https://arxiv.org/pdf/2402.00809
• Copy Paste: [[2402.00809]] Position Paper: Bayesian Deep Learning in the Age of Large-Scale AI(https://arxiv.org/abs/2402.00809)
• Keywords: foundation model
• Abstract: In the current landscape of deep learning research, there is a predominant emphasis on achieving high predictive accuracy in supervised tasks involving large image and language datasets. However, a broader perspective reveals a multitude of overlooked metrics, tasks, and data types, such as uncertainty, active and continual learning, and scientific data, that demand attention. Bayesian deep learning (BDL) constitutes a promising avenue, offering advantages across these diverse settings. This paper posits that BDL can elevate the capabilities of deep learning. It revisits the strengths of BDL, acknowledges existing challenges, and highlights some exciting research avenues aimed at addressing these obstacles. Looking ahead, the discussion focuses on possible ways to combine large-scale foundation models with BDL to unlock their full potential.

Title: SLIM: Skill Learning with Multiple Critics

• Authors: David Emukpere, Bingbing Wu, Julien Perez
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.00823
• Pdf URL: https://arxiv.org/pdf/2402.00823
• Copy Paste: [[2402.00823]] SLIM: Skill Learning with Multiple Critics(https://arxiv.org/abs/2402.00823)
• Keywords: self-supervised
• Abstract: Self-supervised skill learning aims to acquire useful behaviors that leverage the underlying dynamics of the environment. Latent variable models, based on mutual information maximization, have been particularly successful in this task but still struggle in the context of robotic manipulation. As it requires impacting a possibly large set of degrees of freedom composing the environment, mutual information maximization fails alone in producing useful manipulation behaviors. To address this limitation, we introduce SLIM, a multi-critic learning approach for skill discovery with a particular focus on robotic manipulation. Our main insight is that utilizing multiple critics in an actor-critic framework to gracefully combine multiple reward functions leads to a significant improvement in latent-variable skill discovery for robotic manipulation while overcoming possible interference occurring among rewards which hinders convergence to useful skills. Furthermore, in the context of tabletop manipulation, we demonstrate the applicability of our novel skill discovery approach to acquire safe and efficient motor primitives in a hierarchical reinforcement learning fashion and leverage them through planning, surpassing the state-of-the-art approaches for skill discovery by a large margin.

Title: BootsTAP: Bootstrapped Training for Tracking-Any-Point

• Authors: Carl Doersch, Yi Yang, Dilara Gokay, Pauline Luc, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ross Goroshin, João Carreira, Andrew Zisserman
• Subjects: cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.00847
• Pdf URL: https://arxiv.org/pdf/2402.00847
• Copy Paste: [[2402.00847]] BootsTAP: Bootstrapped Training for Tracking-Any-Point(https://arxiv.org/abs/2402.00847)
• Keywords: self-supervised
• Abstract: To endow models with greater understanding of physics and motion, it is useful to enable them to perceive how solid surfaces move and deform in real scenes. This can be formalized as Tracking-Any-Point (TAP), which requires the algorithm to be able to track any point corresponding to a solid surface in a video, potentially densely in space and time. Large-scale ground-truth training data for TAP is only available in simulation, which currently has limited variety of objects and motion. In this work, we demonstrate how large-scale, unlabeled, uncurated real-world data can improve a TAP model with minimal architectural changes, using a self-supervised student-teacher setup. We demonstrate state-of-the-art performance on the TAP-Vid benchmark surpassing previous results by a wide margin: for example, TAP-Vid-DAVIS performance improves from 61.3% to 66.4%, and TAP-Vid-Kinetics from 57.2% to 61.5%.

Title: SymbolicAI: A framework for logic-based approaches combining generative models and solvers

• Authors: Marius-Constantin Dinu, Claudiu Leoveanu-Condrei, Markus Holzleitner, Werner Zellinger, Sepp Hochreiter
• Subjects: cs.LG, cs.AI, cs.SC, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.00854
• Pdf URL: https://arxiv.org/pdf/2402.00854
• Copy Paste: [[2402.00854]] SymbolicAI: A framework for logic-based approaches combining generative models and solvers(https://arxiv.org/abs/2402.00854)
• Keywords: foundation model, generative
• Abstract: We introduce SymbolicAI, a versatile and modular framework employing a logic-based approach to concept learning and flow management in generative processes. SymbolicAI enables the seamless integration of generative models with a diverse range of solvers by treating large language models (LLMs) as semantic parsers that execute tasks based on both natural and formal language instructions, thus bridging the gap between symbolic reasoning and generative AI. We leverage probabilistic programming principles to tackle complex tasks, and utilize differentiable and classical programming paradigms with their respective strengths. The framework introduces a set of polymorphic, compositional, and self-referential operations for data stream manipulation, aligning LLM outputs with user objectives. As a result, we can transition between the capabilities of various foundation models endowed with zero- and few-shot learning capabilities and specialized, fine-tuned models or solvers proficient in addressing specific problems. In turn, the framework facilitates the creation and evaluation of explainable computational graphs. We conclude by introducing a quality measure and its empirical score for evaluating these computational graphs, and propose a benchmark that compares various state-of-the-art LLMs across a set of complex workflows. We refer to the empirical score as the "Vector Embedding for Relational Trajectory Evaluation through Cross-similarity", or VERTEX score for short. The framework codebase and benchmark are linked below.

Title: Can Large Language Models Understand Context?

• Authors: Yilun Zhu, Joel Ruben Antony Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, Bo-Hsiang Tseng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00858
• Pdf URL: https://arxiv.org/pdf/2402.00858
• Copy Paste: [[2402.00858]] Can Large Language Models Understand Context?(https://arxiv.org/abs/2402.00858)
• Keywords: generative, in-context
• Abstract: Understanding context is key to understanding human language, an ability which Large Language Models (LLMs) have been increasingly seen to demonstrate to an impressive extent. However, though the evaluation of LLMs encompasses various domains within the realm of Natural Language Processing, limited attention has been paid to probing their linguistic capability of understanding contextual features. This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. This benchmark comprises of four distinct tasks and nine datasets, all featuring prompts designed to assess the models' ability to understand context. First, we evaluate the performance of LLMs under the in-context learning pretraining scenario. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. Second, as LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings. We find that 3-bit post-training quantization leads to varying degrees of performance reduction on our benchmark. We conduct an extensive analysis of these scenarios to substantiate our experimental results.

Title: ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields

• Authors: Jiahua Dong, Yu-Xiong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00864
• Pdf URL: https://arxiv.org/pdf/2402.00864
• Copy Paste: [[2402.00864]] ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields(https://arxiv.org/abs/2402.00864)
• Keywords: diffusion
• Abstract: We introduce ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that explicitly propagate the editing information across different views, thus ensuring multi-view consistency. For geometric regularization, we leverage the depth information derived from NeRF to establish image correspondences between different views. For learned regularization, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is publicly available.

Title: AToM: Amortized Text-to-Mesh using 2D Diffusion

• Authors: Guocheng Qian, Junli Cao, Aliaksandr Siarohin, Yash Kant, Chaoyang Wang, Michael Vasilkovsky, Hsin-Ying Lee, Yuwei Fang, Ivan Skorokhodov, Peiye Zhuang, Igor Gilitschenski, Jian Ren, Bernard Ghanem, Kfir Aberman, Sergey Tulyakov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00867
• Pdf URL: https://arxiv.org/pdf/2402.00867
• Copy Paste: [[2402.00867]] AToM: Amortized Text-to-Mesh using 2D Diffusion(https://arxiv.org/abs/2402.00867)
• Keywords: diffusion
• Abstract: We introduce Amortized Text-to-Mesh (AToM), a feed-forward text-to-mesh framework optimized across multiple text prompts simultaneously. In contrast to existing text-to-3D methods that often entail time-consuming per-prompt optimization and commonly output representations other than polygonal meshes, AToM directly generates high-quality textured meshes in less than 1 second with around 10 times reduction in the training cost, and generalizes to unseen prompts. Our key idea is a novel triplane-based text-to-mesh architecture with a two-stage amortized optimization strategy that ensures stable training and enables scalability. Through extensive experiments on various prompt benchmarks, AToM significantly outperforms state-of-the-art amortized approaches with over 4 times higher accuracy (in DF415 dataset) and produces more distinguishable and higher-quality 3D outputs. AToM demonstrates strong generalizability, offering finegrained 3D assets for unseen interpolated prompts without further optimization during inference, unlike per-prompt solutions.