2024-02-07

Title: Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentations

• Authors: Helen Qu, Sang Michael Xie
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.03325
• Pdf URL: https://arxiv.org/pdf/2402.03325
• Copy Paste: [[2402.03325]] Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentations(https://arxiv.org/abs/2402.03325)
• Keywords: self-supervised
• Abstract: Models trained on a labeled source domain (e.g., labeled images from wildlife camera traps) often generalize poorly when deployed on an out-of-distribution (OOD) target domain (e.g., images from new camera trap locations). In the domain adaptation setting where unlabeled target data is available, self-supervised pretraining (e.g., masked autoencoding or contrastive learning) is a promising method to mitigate this performance drop. Pretraining improves OOD error when the generic data augmentations used (e.g., masking or cropping) connect the source and target domains, which may be far apart in the input space. In this paper, we show on real-world tasks that standard fine-tuning after pretraining does not consistently improve OOD error over simply training from scratch on labeled source data. To better leverage pretraining for distribution shifts, we propose Connect Later: after pretraining with generic augmentations, fine-tune with targeted augmentations designed with knowledge of the distribution shift. Pretraining learns good representations within the source and target domains, while targeted augmentations connect the domains better during fine-tuning. Connect Later improves average OOD error over standard fine-tuning and supervised learning with targeted augmentations on 4 real-world datasets: Connect Later achieves the state-of-the-art on astronomical time-series classification (AstroClassification) by 2.5%, wildlife species identification (iWildCam-WILDS) with ResNet-50 by 0.9%, and tumor identification (Camelyon17-WILDS) with DenseNet121 by 1.1%; as well as best performance on a new dataset for astronomical time-series redshift prediction (Redshifts) by 0.03 RMSE (11% relative). Code and datasets are available at https://github.com/helenqu/connect-later.

Title: Large-scale Generative AI Models Lack Visual Number Sense

• Authors: Alberto Testolin, Kuinan Hou, Marco Zorzi
• Subjects: cs.CV, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.03328
• Pdf URL: https://arxiv.org/pdf/2402.03328
• Copy Paste: [[2402.03328]] Large-scale Generative AI Models Lack Visual Number Sense(https://arxiv.org/abs/2402.03328)
• Keywords: foundation model, generative
• Abstract: Humans can readily judge the number of objects in a visual scene, even without counting, and such a skill has been documented in a variety of animal species and in babies prior to language development and formal schooling. Numerical judgments are error-free for small sets, while for larger collections responses become approximate, with variability increasing proportionally to the target number. This response pattern is observed for items of all kinds, despite variation in object features (such as color or shape), suggesting that our visual number sense relies on abstract representations of numerosity. Here, we investigated whether generative Artificial Intelligence (AI) models based on large-scale transformer architectures can reliably name the number of objects in simple visual stimuli or generate images containing a target number of items in the 1-10 range. Surprisingly, none of the foundation models considered performed in a human-like way: They all made striking errors even with small numbers, the response variability often did not increase in a systematic way, and the pattern of errors varied with object category. Our findings demonstrate that advanced AI systems still lack a basic ability that supports an intuitive understanding of numbers, which in humans is foundational for numeracy and mathematical development.

Title: Unsupervised Salient Patch Selection for Data-Efficient Reinforcement Learning

• Authors: Zhaohui Jiang, Paul Weng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.03329
• Pdf URL: https://arxiv.org/pdf/2402.03329
• Copy Paste: [[2402.03329]] Unsupervised Salient Patch Selection for Data-Efficient Reinforcement Learning(https://arxiv.org/abs/2402.03329)
• Keywords: self-supervised
• Abstract: To improve the sample efficiency of vision-based deep reinforcement learning (RL), we propose a novel method, called SPIRL, to automatically extract important patches from input images. Following Masked Auto-Encoders, SPIRL is based on Vision Transformer models pre-trained in a self-supervised fashion to reconstruct images from randomly-sampled patches. These pre-trained models can then be exploited to detect and select salient patches, defined as hard to reconstruct from neighboring patches. In RL, the SPIRL agent processes selected salient patches via an attention module. We empirically validate SPIRL on Atari games to test its data-efficiency against relevant state-of-the-art methods, including some traditional model-based methods and keypoint-based models. In addition, we analyze our model's interpretability capabilities.

Title: Denoising Diffusion via Image-Based Rendering

• Authors: Titas Anciukevicius, Fabian Manhardt, Federico Tombari, Paul Henderson
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.03445
• Pdf URL: https://arxiv.org/pdf/2402.03445
• Copy Paste: [[2402.03445]] Denoising Diffusion via Image-Based Rendering(https://arxiv.org/abs/2402.03445)
• Keywords: diffusion, generative
• Abstract: Generating 3D scenes is a challenging open problem, which requires synthesizing plausible content that is fully consistent in 3D space. While recent methods such as neural radiance fields excel at view synthesis and 3D reconstruction, they cannot synthesize plausible details in unobserved regions since they lack a generative capability. Conversely, existing generative methods are typically not capable of reconstructing detailed, large-scale scenes in the wild, as they use limited-capacity 3D scene representations, require aligned camera poses, or rely on additional regularizers. In this work, we introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. To achieve this, we make three contributions. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes, dynamically allocating more capacity as needed to capture details visible in each image. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images without the need for any additional supervision signal such as masks or depths. This supports 3D reconstruction and generation in a unified architecture. Third, we develop a principled approach to avoid trivial 3D solutions when integrating the image-based rendering with the diffusion model, by dropping out representations of some images. We evaluate the model on several challenging datasets of real and synthetic images, and demonstrate superior results on generation, novel view synthesis and 3D reconstruction.

Title: Constrained Multiview Representation for Self-supervised Contrastive Learning

• Authors: Siyuan Dai, Kai Ye, Kun Zhao, Ge Cui, Haoteng Tang, Liang Zhan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03456
• Pdf URL: https://arxiv.org/pdf/2402.03456
• Copy Paste: [[2402.03456]] Constrained Multiview Representation for Self-supervised Contrastive Learning(https://arxiv.org/abs/2402.03456)
• Keywords: self-supervised
• Abstract: Representation learning constitutes a pivotal cornerstone in contemporary deep learning paradigms, offering a conduit to elucidate distinctive features within the latent space and interpret the deep models. Nevertheless, the inherent complexity of anatomical patterns and the random nature of lesion distribution in medical image segmentation pose significant challenges to the disentanglement of representations and the understanding of salient features. Methods guided by the maximization of mutual information, particularly within the framework of contrastive learning, have demonstrated remarkable success and superiority in decoupling densely intertwined representations. However, the effectiveness of contrastive learning highly depends on the quality of the positive and negative sample pairs, i.e. the unselected average mutual information among multi-views would obstruct the learning strategy so the selection of the views is vital. In this work, we introduce a novel approach predicated on representation distance-based mutual information (MI) maximization for measuring the significance of different views, aiming at conducting more efficient contrastive learning and representation disentanglement. Additionally, we introduce an MI re-ranking strategy for representation selection, benefiting both the continuous MI estimating and representation significance distance measuring. Specifically, we harness multi-view representations extracted from the frequency domain, re-evaluating their significance based on mutual information across varying frequencies, thereby facilitating a multifaceted contrastive learning approach to bolster semantic comprehension. The statistical results under the five metrics demonstrate that our proposed framework proficiently constrains the MI maximization-driven representation selection and steers the multi-view contrastive learning process.

Title: Stochastic Modified Flows for Riemannian Stochastic Gradient Descent

• Authors: Benjamin Gess, Sebastian Kassing, Nimit Rana
• Subjects: cs.LG, math.OC, math.PR, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.03467
• Pdf URL: https://arxiv.org/pdf/2402.03467
• Copy Paste: [[2402.03467]] Stochastic Modified Flows for Riemannian Stochastic Gradient Descent(https://arxiv.org/abs/2402.03467)
• Keywords: diffusion
• Abstract: We give quantitative estimates for the rate of convergence of Riemannian stochastic gradient descent (RSGD) to Riemannian gradient flow and to a diffusion process, the so-called Riemannian stochastic modified flow (RSMF). Using tools from stochastic differential geometry we show that, in the small learning rate regime, RSGD can be approximated by the solution to the RSMF driven by an infinite-dimensional Wiener process. The RSMF accounts for the random fluctuations of RSGD and, thereby, increases the order of approximation compared to the deterministic Riemannian gradient flow. The RSGD is build using the concept of a retraction map, that is, a cost efficient approximation of the exponential map, and we prove quantitative bounds for the weak error of the diffusion approximation under assumptions on the retraction map, the geometry of the manifold, and the random estimators of the gradient.

Title: Hyper-Diffusion: Estimating Epistemic and Aleatoric Uncertainty with a Single Model

• Authors: Matthew A. Chan, Maria J. Molina, Christopher A. Metzler
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03478
• Pdf URL: https://arxiv.org/pdf/2402.03478
• Copy Paste: [[2402.03478]] Hyper-Diffusion: Estimating Epistemic and Aleatoric Uncertainty with a Single Model(https://arxiv.org/abs/2402.03478)
• Keywords: diffusion
• Abstract: Estimating and disentangling epistemic uncertainty (uncertainty that can be reduced with more training data) and aleatoric uncertainty (uncertainty that is inherent to the task at hand) is critically important when applying machine learning (ML) to high-stakes applications such as medical imaging and weather forecasting. Conditional diffusion models' breakthrough ability to accurately and efficiently sample from the posterior distribution of a dataset now makes uncertainty estimation conceptually straightforward: One need only train and sample from a large ensemble of diffusion models. Unfortunately, training such an ensemble becomes computationally intractable as the complexity of the model architecture grows. In this work we introduce a new approach to ensembling, hyper-diffusion, which allows one to accurately estimate epistemic and aleatoric uncertainty with a single model. Unlike existing Monte Carlo dropout based single-model ensembling methods, hyper-diffusion offers the same prediction accuracy as multi-model ensembles. We validate our approach on two distinct tasks: x-ray computed tomography (CT) reconstruction and weather temperature forecasting.

Title: ICED: Zero-Shot Transfer in Reinforcement Learning via In-Context Environment Design

• Authors: Samuel Garcin, James Doran, Shangmin Guo, Christopher G. Lucas, Stefano V. Albrecht
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.03479
• Pdf URL: https://arxiv.org/pdf/2402.03479
• Copy Paste: [[2402.03479]] ICED: Zero-Shot Transfer in Reinforcement Learning via In-Context Environment Design(https://arxiv.org/abs/2402.03479)
• Keywords: in-context
• Abstract: Autonomous agents trained using deep reinforcement learning (RL) often lack the ability to successfully generalise to new environments, even when they share characteristics with the environments they have encountered during training. In this work, we investigate how the sampling of individual environment instances, or levels, affects the zero-shot generalisation (ZSG) ability of RL agents. We discover that, for deep actor-critic architectures sharing their base layers, prioritising levels according to their value loss minimises the mutual information between the agent's internal representation and the set of training levels in the generated training data. This provides a novel theoretical justification for the implicit regularisation achieved by certain adaptive sampling strategies. We then turn our attention to unsupervised environment design (UED) methods, which have more control over the data generation mechanism. We find that existing UED methods can significantly shift the training distribution, which translates to low ZSG performance. To prevent both overfitting and distributional shift, we introduce in-context environment design (ICED). ICED generates levels using a variational autoencoder trained over an initial set of level parameters, reducing distributional shift, and achieves significant improvements in ZSG over adaptive level sampling strategies and UED methods.

Title: An Inpainting-Infused Pipeline for Attire and Background Replacement

• Authors: Felipe Rodrigues Perche-Mahlow, André Felipe-Zanella, William Alberto Cruz-Castañeda, Marcellus Amadeus
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.03501
• Pdf URL: https://arxiv.org/pdf/2402.03501
• Copy Paste: [[2402.03501]] An Inpainting-Infused Pipeline for Attire and Background Replacement(https://arxiv.org/abs/2402.03501)
• Keywords: diffusion, generative
• Abstract: In recent years, groundbreaking advancements in Generative Artificial Intelligence (GenAI) have triggered a transformative paradigm shift, significantly influencing various domains. In this work, we specifically explore an integrated approach, leveraging advanced techniques in GenAI and computer vision emphasizing image manipulation. The methodology unfolds through several stages, including depth estimation, the creation of inpaint masks based on depth information, the generation and replacement of backgrounds utilizing Stable Diffusion in conjunction with Latent Consistency Models (LCMs), and the subsequent replacement of clothes and application of aesthetic changes through an inpainting pipeline. Experiments conducted in this study underscore the methodology's efficacy, highlighting its potential to produce visually captivating content. The convergence of these advanced techniques allows users to input photographs of individuals and manipulate them to modify clothing and background based on specific prompts without manually input inpainting masks, effectively placing the subjects within the vast landscape of creative imagination.

Title: Online Feature Updates Improve Online (Generalized) Label Shift Adaptation

• Authors: Ruihan Wu, Siddhartha Datta, Yi Su, Dheeraj Baby, Yu-Xiang Wang, Kilian Q. Weinberger
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.03545
• Pdf URL: https://arxiv.org/pdf/2402.03545
• Copy Paste: [[2402.03545]] Online Feature Updates Improve Online (Generalized) Label Shift Adaptation(https://arxiv.org/abs/2402.03545)
• Keywords: self-supervised
• Abstract: This paper addresses the prevalent issue of label shift in an online setting with missing labels, where data distributions change over time and obtaining timely labels is challenging. While existing methods primarily focus on adjusting or updating the final layer of a pre-trained classifier, we explore the untapped potential of enhancing feature representations using unlabeled data at test-time. Our novel method, Online Label Shift adaptation with Online Feature Updates (OLS-OFU), leverages self-supervised learning to refine the feature extraction process, thereby improving the prediction model. Theoretical analyses confirm that OLS-OFU reduces algorithmic regret by capitalizing on self-supervised learning for feature refinement. Empirical studies on various datasets, under both online label shift and generalized label shift conditions, underscore the effectiveness and robustness of OLS-OFU, especially in cases of domain shifts.

Title: AnaMoDiff: 2D Analogical Motion Diffusion via Disentangled Denoising

• Authors: Maham Tanveer, Yizhi Wang, Ruiqi Wang, Nanxuan Zhao, Ali Mahdavi-Amiri, Hao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03549
• Pdf URL: https://arxiv.org/pdf/2402.03549
• Copy Paste: [[2402.03549]] AnaMoDiff: 2D Analogical Motion Diffusion via Disentangled Denoising(https://arxiv.org/abs/2402.03549)
• Keywords: diffusion
• Abstract: We present AnaMoDiff, a novel diffusion-based method for 2D motion analogies that is applied to raw, unannotated videos of articulated characters. Our goal is to accurately transfer motions from a 2D driving video onto a source character, with its identity, in terms of appearance and natural movement, well preserved, even when there may be significant discrepancies between the source and driving characters in their part proportions and movement speed and styles. Our diffusion model transfers the input motion via a latent optical flow (LOF) network operating in a noised latent space, which is spatially aware, efficient to process compared to the original RGB videos, and artifact-resistant through the diffusion denoising process even amid dense movements. To accomplish both motion analogy and identity preservation, we train our denoising model in a feature-disentangled manner, operating at two noise levels. While identity-revealing features of the source are learned via conventional noise injection, motion features are learned from LOF-warped videos by only injecting noise with large values, with the stipulation that motion properties involving pose and limbs are encoded by higher-level features. Experiments demonstrate that our method achieves the best trade-off between motion analogy and identity preservation.

Title: Path Signatures and Graph Neural Networks for Slow Earthquake Analysis: Better Together?

• Authors: Hans Riess, Manolis Veveakis, Michael M. Zavlanos
• Subjects: cs.LG, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2402.03558
• Pdf URL: https://arxiv.org/pdf/2402.03558
• Copy Paste: [[2402.03558]] Path Signatures and Graph Neural Networks for Slow Earthquake Analysis: Better Together?(https://arxiv.org/abs/2402.03558)
• Keywords: diffusion
• Abstract: The path signature, having enjoyed recent success in the machine learning community, is a theoretically-driven method for engineering features from irregular paths. On the other hand, graph neural networks (GNN), neural architectures for processing data on graphs, excel on tasks with irregular domains, such as sensor networks. In this paper, we introduce a novel approach, Path Signature Graph Convolutional Neural Networks (PS-GCNN), integrating path signatures into graph convolutional neural networks (GCNN), and leveraging the strengths of both path signatures, for feature extraction, and GCNNs, for handling spatial interactions. We apply our method to analyze slow earthquake sequences, also called slow slip events (SSE), utilizing data from GPS timeseries, with a case study on a GPS sensor network on the east coast of New Zealand's north island. We also establish benchmarks for our method on simulated stochastic differential equations, which model similar reaction-diffusion phenomenon. Our methodology shows promise for future advancement in earthquake prediction and sensor network analysis.

Title: Projected Generative Diffusion Models for Constraint Satisfaction

• Authors: Jacob K Christopher, Stephen Baek, Ferdinando Fioretto
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.03559
• Pdf URL: https://arxiv.org/pdf/2402.03559
• Copy Paste: [[2402.03559]] Projected Generative Diffusion Models for Constraint Satisfaction(https://arxiv.org/abs/2402.03559)
• Keywords: diffusion, generative
• Abstract: Generative diffusion models excel at robustly synthesizing coherent content from raw noise through a sequential process. However, their direct application in scenarios requiring outputs to adhere to specific, stringent criteria faces several severe challenges. This paper aims at overcome these challenges and introduces Projected Generative Diffusion Models (PGDM), an approach that recast traditional diffusion models sampling into a constrained-optimization problem. This enables the application of an iterative projections method to ensure that generated data faithfully adheres to specified constraints or physical principles. This paper provides theoretical support for the ability of PGDM to synthesize outputs from a feasible subdistribution under a restricted class of constraints while also providing large empirical evidence in the case of complex non-convex constraints and ordinary differential equations. These capabilities are demonstrated by physics-informed motion in video generation, trajectory optimization in path planning, and morphometric properties adherence in material science.

Title: Diffusion World Model

• Authors: Zihan Ding, Amy Zhang, Yuandong Tian, Qinqing Zheng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.03570
• Pdf URL: https://arxiv.org/pdf/2402.03570
• Copy Paste: [[2402.03570]] Diffusion World Model(https://arxiv.org/abs/2402.03570)
• Keywords: diffusion, generative
• Abstract: We introduce Diffusion World Model (DWM), a conditional diffusion model capable of predicting multistep future states and rewards concurrently. As opposed to traditional one-step dynamics models, DWM offers long-horizon predictions in a single forward pass, eliminating the need for recursive quires. We integrate DWM into model-based value estimation, where the short-term return is simulated by future trajectories sampled from DWM. In the context of offline reinforcement learning, DWM can be viewed as a conservative value regularization through generative modeling. Alternatively, it can be seen as a data source that enables offline Q-learning with synthetic data. Our experiments on the D4RL dataset confirm the robustness of DWM to long-horizon simulation. In terms of absolute performance, DWM significantly surpasses one-step dynamics models with a $44\%$ performance gain, and achieves state-of-the-art performance.

Title: Neural Network Approximators for Marginal MAP in Probabilistic Circuits

• Authors: Shivvrat Arya, Tahrima Rahman, Vibhav Gogate
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.03621
• Pdf URL: https://arxiv.org/pdf/2402.03621
• Copy Paste: [[2402.03621]] Neural Network Approximators for Marginal MAP in Probabilistic Circuits(https://arxiv.org/abs/2402.03621)
• Keywords: self-supervised
• Abstract: Probabilistic circuits (PCs) such as sum-product networks efficiently represent large multi-variate probability distributions. They are preferred in practice over other probabilistic representations such as Bayesian and Markov networks because PCs can solve marginal inference (MAR) tasks in time that scales linearly in the size of the network. Unfortunately, the maximum-a-posteriori (MAP) and marginal MAP (MMAP) tasks remain NP-hard in these models. Inspired by the recent work on using neural networks for generating near-optimal solutions to optimization problems such as integer linear programming, we propose an approach that uses neural networks to approximate (M)MAP inference in PCs. The key idea in our approach is to approximate the cost of an assignment to the query variables using a continuous multilinear function, and then use the latter as a loss function. The two main benefits of our new method are that it is self-supervised and after the neural network is learned, it requires only linear time to output a solution. We evaluate our new approach on several benchmark datasets and show that it outperforms three competing linear time approximations, max-product inference, max-marginal inference and sequential estimation, which are used in practice to solve MMAP tasks in PCs.

Title: Lens: A Foundation Model for Network Traffic

• Authors: Qineng Wang, Chen Qian, Xiaochang Li, Ziyu Yao, Huajie Shao
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.03646
• Pdf URL: https://arxiv.org/pdf/2402.03646
• Copy Paste: [[2402.03646]] Lens: A Foundation Model for Network Traffic(https://arxiv.org/abs/2402.03646)
• Keywords: foundation model, generative
• Abstract: Network traffic refers to the amount of information being sent and received over the internet or any system that connects computers. Analyzing and understanding network traffic is vital for improving network security and management. However, the analysis of network traffic poses great challenges due to the unique characteristics of data packets, such as heterogeneous headers and encrypted payload lacking semantics. To capture the latent semantics of traffic, a few studies have adopted pre-training techniques based on the Transformer encoder or decoder to learn the representations from large-scale traffic data. However, these methods typically excel only in traffic understanding (classification) or traffic generation tasks. To address this issue, we develop Lens, a foundational network traffic model that leverages the T5 architecture to learn the pre-trained representations from large-scale unlabeled data. Harnessing the strength of the encoder-decoder framework, which captures the global information while preserving the generative ability, our model can better learn the representations from large-scale network traffic. To further enhance pre-training performance, we design a novel loss that integrates three distinct tasks, namely Masked Span Prediction (MSP), Packet Order Prediction (POP), and Homologous Traffic Prediction (HTP). Evaluation results on multiple benchmark datasets demonstrate that the proposed Lens outperforms the baselines in most downstream tasks related to both traffic understanding and traffic generation. Notably, it also requires considerably less labeled data for fine-tuning compared to current methods.

Title: Reviewing FID and SID Metrics on Generative Adversarial Networks

• Authors: Ricardo de Deijn, Aishwarya Batra, Brandon Koch, Naseef Mansoor, Hema Makkena
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.03654
• Pdf URL: https://arxiv.org/pdf/2402.03654
• Copy Paste: [[2402.03654]] Reviewing FID and SID Metrics on Generative Adversarial Networks(https://arxiv.org/abs/2402.03654)
• Keywords: generative
• Abstract: The growth of generative adversarial network (GAN) models has increased the ability of image processing and provides numerous industries with the technology to produce realistic image transformations. However, with the field being recently established there are new evaluation metrics that can further this research. Previous research has shown the Fr\'echet Inception Distance (FID) to be an effective metric when testing these image-to-image GANs in real-world applications. Signed Inception Distance (SID), a founded metric in 2023, expands on FID by allowing unsigned distances. This paper uses public datasets that consist of fa\c{c}ades, cityscapes, and maps within Pix2Pix and CycleGAN models. After training these models are evaluated on both inception distance metrics which measure the generating performance of the trained models. Our findings indicate that usage of the metric SID incorporates an efficient and effective metric to complement, or even exceed the ability shown using the FID for the image-to-image GANs

Title: Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue

• Authors: Kun Ouyang, Liqiang Jing, Xuemeng Song, Meng Liu, Yupeng Hu, Liqiang Nie
• Subjects: cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.03658
• Pdf URL: https://arxiv.org/pdf/2402.03658
• Copy Paste: [[2402.03658]] Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue(https://arxiv.org/abs/2402.03658)
• Keywords: generative
• Abstract: Sarcasm Explanation in Dialogue (SED) is a new yet challenging task, which aims to generate a natural language explanation for the given sarcastic dialogue that involves multiple modalities (i.e., utterance, video, and audio). Although existing studies have achieved great success based on the generative pretrained language model BART, they overlook exploiting the sentiments residing in the utterance, video and audio, which are vital clues for sarcasm explanation. In fact, it is non-trivial to incorporate sentiments for boosting SED performance, due to three main challenges: 1) diverse effects of utterance tokens on sentiments; 2) gap between video-audio sentiment signals and the embedding space of BART; and 3) various relations among utterances, utterance sentiments, and video-audio sentiments. To tackle these challenges, we propose a novel sEntiment-enhanceD Graph-based multimodal sarcasm Explanation framework, named EDGE. In particular, we first propose a lexicon-guided utterance sentiment inference module, where a heuristic utterance sentiment refinement strategy is devised. We then develop a module named Joint Cross Attention-based Sentiment Inference (JCA-SI) by extending the multimodal sentiment analysis model JCA to derive the joint sentiment label for each video-audio clip. Thereafter, we devise a context-sentiment graph to comprehensively model the semantic relations among the utterances, utterance sentiments, and video-audio sentiments, to facilitate sarcasm explanation generation. Extensive experiments on the publicly released dataset WITS verify the superiority of our model over cutting-edge methods.

Title: Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models

• Authors: Kelvin J.L. Koa, Yunshan Ma, Ritchie Ng, Tat-Seng Chua
• Subjects: cs.LG, cs.CL, q-fin.ST
• Abstract URL: https://arxiv.org/abs/2402.03659
• Pdf URL: https://arxiv.org/pdf/2402.03659
• Copy Paste: [[2402.03659]] Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models(https://arxiv.org/abs/2402.03659)
• Keywords: generative
• Abstract: Explaining stock predictions is generally a difficult task for traditional non-generative deep learning models, where explanations are limited to visualizing the attention weights on important texts. Today, Large Language Models (LLMs) present a solution to this problem, given their known capabilities to generate human-readable explanations for their decision-making process. However, the task of stock prediction remains challenging for LLMs, as it requires the ability to weigh the varying impacts of chaotic social texts on stock prices. The problem gets progressively harder with the introduction of the explanation component, which requires LLMs to explain verbally why certain factors are more important than the others. On the other hand, to fine-tune LLMs for such a task, one would need expert-annotated samples of explanation for every stock movement in the training set, which is expensive and impractical to scale. To tackle these issues, we propose our Summarize-Explain-Predict (SEP) framework, which utilizes a self-reflective agent and Proximal Policy Optimization (PPO) to let a LLM teach itself how to generate explainable stock predictions in a fully autonomous manner. The reflective agent learns how to explain past stock movements through self-reasoning, while the PPO trainer trains the model to generate the most likely explanations from input texts. The training samples for the PPO trainer are also the responses generated during the reflective process, which eliminates the need for human annotators. Using our SEP framework, we fine-tune a LLM that can outperform both traditional deep-learning and LLM methods in prediction accuracy and Matthews correlation coefficient for the stock classification task. To justify the generalization capability of our framework, we further test it on the portfolio construction task, and demonstrate its effectiveness through various portfolio metrics.

Title: QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning

• Authors: Haoxuan Wang, Yuzhang Shang, Zhihang Yuan, Junyi Wu, Yan Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03666
• Pdf URL: https://arxiv.org/pdf/2402.03666
• Copy Paste: [[2402.03666]] QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning(https://arxiv.org/abs/2402.03666)
• Keywords: diffusion
• Abstract: Diffusion models have achieved remarkable success in image generation tasks, yet their practical deployment is restrained by the high memory and time consumption. While quantization paves a way for diffusion model compression and acceleration, existing methods totally fail when the models are quantized to low-bits. In this paper, we unravel three properties in quantized diffusion models that compromise the efficacy of current methods: imbalanced activation distributions, imprecise temporal information, and vulnerability to perturbations of specific modules. To alleviate the intensified low-bit quantization difficulty stemming from the distribution imbalance, we propose finetuning the quantized model to better adapt to the activation distribution. Building on this idea, we identify two critical types of quantized layers: those holding vital temporal information and those sensitive to reduced bit-width, and finetune them to mitigate performance degradation with efficiency. We empirically verify that our approach modifies the activation distribution and provides meaningful temporal information, facilitating easier and more accurate quantization. Our method is evaluated over three high-resolution image generation tasks and achieves state-of-the-art performance under various bit-width settings, as well as being the first method to generate readable images on full 4-bit (i.e. W4A4) Stable Diffusion.

Title: Pard: Permutation-Invariant Autoregressive Diffusion for Graph Generation

• Authors: Lingxiao Zhao, Xueying Ding, Leman Akoglu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.03687
• Pdf URL: https://arxiv.org/pdf/2402.03687
• Copy Paste: [[2402.03687]] Pard: Permutation-Invariant Autoregressive Diffusion for Graph Generation(https://arxiv.org/abs/2402.03687)
• Keywords: diffusion
• Abstract: Graph generation has been dominated by autoregressive models due to their simplicity and effectiveness, despite their sensitivity to ordering. Yet diffusion models have garnered increasing attention, as they offer comparable performance while being permutation-invariant. Current graph diffusion models generate graphs in a one-shot fashion, but they require extra features and thousands of denoising steps to achieve optimal performance. We introduce PARD, a Permutation-invariant Auto Regressive Diffusion model that integrates diffusion models with autoregressive methods. PARD harnesses the effectiveness and efficiency of the autoregressive model while maintaining permutation invariance without ordering sensitivity. Specifically, we show that contrary to sets, elements in a graph are not entirely unordered and there is a unique partial order for nodes and edges. With this partial order, PARD generates a graph in a block-by-block, autoregressive fashion, where each block's probability is conditionally modeled by a shared diffusion model with an equivariant network. To ensure efficiency while being expressive, we further propose a higher-order graph transformer, which integrates transformer with PPGN. Like GPT, we extend the higher-order graph transformer to support parallel training of all blocks. Without any extra features, PARD achieves state-of-the-art performance on molecular and non-molecular datasets, and scales to large datasets like MOSES containing 1.9M molecules.

Title: Improving and Unifying Discrete&Continuous-time Discrete Denoising Diffusion

• Authors: Lingxiao Zhao, Xueying Ding, Lijun Yu, Leman Akoglu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.03701
• Pdf URL: https://arxiv.org/pdf/2402.03701
• Copy Paste: [[2402.03701]] Improving and Unifying Discrete&Continuous-time Discrete Denoising Diffusion(https://arxiv.org/abs/2402.03701)
• Keywords: diffusion
• Abstract: Discrete diffusion models have seen a surge of attention with applications on naturally discrete data such as language and graphs. Although discrete-time discrete diffusion has been established for a while, only recently Campbell et al. (2022) introduced the first framework for continuous-time discrete diffusion. However, their training and sampling processes differ significantly from the discrete-time version, necessitating nontrivial approximations for tractability. In this paper, we first present a series of mathematical simplifications of the variational lower bound that enable more accurate and easy-to-optimize training for discrete diffusion. In addition, we derive a simple formulation for backward denoising that enables exact and accelerated sampling, and importantly, an elegant unification of discrete-time and continuous-time discrete diffusion. Thanks to simpler analytical formulations, both forward and now also backward probabilities can flexibly accommodate any noise distribution, including different noise distributions for multi-element objects. Experiments show that our proposed USD3 (for Unified Simplified Discrete Denoising Diffusion) outperform all SOTA baselines on established datasets. We open-source our unified code at https://github.com/LingxiaoShawn/USD3.

Title: FoolSDEdit: Deceptively Steering Your Edits Towards Targeted Attribute-aware Distribution

• Authors: Qi Zhou, Dongxia Wang, Tianlin Li, Zhihong Xu, Yang Liu, Kui Ren, Wenhai Wang, Qing Guo
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.03705
• Pdf URL: https://arxiv.org/pdf/2402.03705
• Copy Paste: [[2402.03705]] FoolSDEdit: Deceptively Steering Your Edits Towards Targeted Attribute-aware Distribution(https://arxiv.org/abs/2402.03705)
• Keywords: diffusion, generative
• Abstract: Guided image synthesis methods, like SDEdit based on the diffusion model, excel at creating realistic images from user inputs such as stroke paintings. However, existing efforts mainly focus on image quality, often overlooking a key point: the diffusion model represents a data distribution, not individual images. This introduces a low but critical chance of generating images that contradict user intentions, raising ethical concerns. For example, a user inputting a stroke painting with female characteristics might, with some probability, get male faces from SDEdit. To expose this potential vulnerability, we aim to build an adversarial attack forcing SDEdit to generate a specific data distribution aligned with a specified attribute (e.g., female), without changing the input's attribute characteristics. We propose the Targeted Attribute Generative Attack (TAGA), using an attribute-aware objective function and optimizing the adversarial noise added to the input stroke painting. Empirical studies reveal that traditional adversarial noise struggles with TAGA, while natural perturbations like exposure and motion blur easily alter generated images' attributes. To execute effective attacks, we introduce FoolSDEdit: We design a joint adversarial exposure and blur attack, adding exposure and motion blur to the stroke painting and optimizing them together. We optimize the execution strategy of various perturbations, framing it as a network architecture search problem. We create the SuperPert, a graph representing diverse execution strategies for different perturbations. After training, we obtain the optimized execution strategy for effective TAGA against SDEdit. Comprehensive experiments on two datasets show our method compelling SDEdit to generate a targeted attribute-aware data distribution, significantly outperforming baselines.

Title: Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models

• Authors: Jianyuan Guo, Hanting Chen, Chengcheng Wang, Kai Han, Chang Xu, Yunhe Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03749
• Pdf URL: https://arxiv.org/pdf/2402.03749
• Copy Paste: [[2402.03749]] Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models(https://arxiv.org/abs/2402.03749)
• Keywords: foundation model
• Abstract: Recent advancements in large language models have sparked interest in their extraordinary and near-superhuman capabilities, leading researchers to explore methods for evaluating and optimizing these abilities, which is called superalignment. In this context, our paper delves into the realm of vision foundation models, focusing on the concept of weak-to-strong generalization, which involves using a weaker model to supervise a stronger one, aiming to enhance the latter's capabilities beyond the former's limits. We introduce a novel and adaptively adjustable loss function for weak-to-strong supervision. Our comprehensive experiments span various scenarios, including few-shot learning, transfer learning, noisy label learning, and common knowledge distillation settings. The results are striking: our approach not only exceeds the performance benchmarks set by strong-to-strong generalization but also surpasses the outcomes of fine-tuning strong models with whole datasets. This compelling evidence underscores the significant potential of weak-to-strong generalization, showcasing its capability to substantially elevate the performance of vision foundation models. The code is available at https://github.com/ggjy/vision_weak_to_strong.

Title: AirPhyNet: Harnessing Physics-Guided Neural Networks for Air Quality Prediction

• Authors: Kethmi Hirushini Hettige, Jiahao Ji, Shili Xiang, Cheng Long, Gao Cong, Jingyuan Wang
• Subjects: cs.LG, cs.AI, physics.app-ph
• Abstract URL: https://arxiv.org/abs/2402.03784
• Pdf URL: https://arxiv.org/pdf/2402.03784
• Copy Paste: [[2402.03784]] AirPhyNet: Harnessing Physics-Guided Neural Networks for Air Quality Prediction(https://arxiv.org/abs/2402.03784)
• Keywords: diffusion
• Abstract: Air quality prediction and modelling plays a pivotal role in public health and environment management, for individuals and authorities to make informed decisions. Although traditional data-driven models have shown promise in this domain, their long-term prediction accuracy can be limited, especially in scenarios with sparse or incomplete data and they often rely on black-box deep learning structures that lack solid physical foundation leading to reduced transparency and interpretability in predictions. To address these limitations, this paper presents a novel approach named Physics guided Neural Network for Air Quality Prediction (AirPhyNet). Specifically, we leverage two well-established physics principles of air particle movement (diffusion and advection) by representing them as differential equation networks. Then, we utilize a graph structure to integrate physics knowledge into a neural network architecture and exploit latent representations to capture spatio-temporal relationships within the air quality data. Experiments on two real-world benchmark datasets demonstrate that AirPhyNet outperforms state-of-the-art models for different testing scenarios including different lead time (24h, 48h, 72h), sparse data and sudden change prediction, achieving reduction in prediction errors up to 10%. Moreover, a case study further validates that our model captures underlying physical processes of particle movement and generates accurate predictions with real physical meaning.

Title: Weakly Supervised Anomaly Detection via Knowledge-Data Alignment

• Authors: Haihong Zhao, Chenyi Zi, Yang Liu, Chen Zhang, Yan Zhou, Jia Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.03785
• Pdf URL: https://arxiv.org/pdf/2402.03785
• Copy Paste: [[2402.03785]] Weakly Supervised Anomaly Detection via Knowledge-Data Alignment(https://arxiv.org/abs/2402.03785)
• Keywords: anomaly
• Abstract: Anomaly detection (AD) plays a pivotal role in numerous web-based applications, including malware detection, anti-money laundering, device failure detection, and network fault analysis. Most methods, which rely on unsupervised learning, are hard to reach satisfactory detection accuracy due to the lack of labels. Weakly Supervised Anomaly Detection (WSAD) has been introduced with a limited number of labeled anomaly samples to enhance model performance. Nevertheless, it is still challenging for models, trained on an inadequate amount of labeled data, to generalize to unseen anomalies. In this paper, we introduce a novel framework Knowledge-Data Alignment (KDAlign) to integrate rule knowledge, typically summarized by human experts, to supplement the limited labeled data. Specifically, we transpose these rules into the knowledge space and subsequently recast the incorporation of knowledge as the alignment of knowledge and data. To facilitate this alignment, we employ the Optimal Transport (OT) technique. We then incorporate the OT distance as an additional loss term to the original objective function of WSAD methodologies. Comprehensive experimental results on five real-world datasets demonstrate that our proposed KDAlign framework markedly surpasses its state-of-the-art counterparts, achieving superior performance across various anomaly types.

Title: Energy-based Domain-Adaptive Segmentation with Depth Guidance

• Authors: Jinjing Zhu, Zhedong Hu, Tae-Kyun Kim, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03795
• Pdf URL: https://arxiv.org/pdf/2402.03795
• Copy Paste: [[2402.03795]] Energy-based Domain-Adaptive Segmentation with Depth Guidance(https://arxiv.org/abs/2402.03795)
• Keywords: self-supervised
• Abstract: Recent endeavors have been made to leverage self-supervised depth estimation as guidance in unsupervised domain adaptation (UDA) for semantic segmentation. Prior arts, however, overlook the discrepancy between semantic and depth features, as well as the reliability of feature fusion, thus leading to suboptimal segmentation performance. To address this issue, we propose a novel UDA framework called SMART (croSs doMain semAntic segmentation based on eneRgy esTimation) that utilizes Energy-Based Models (EBMs) to obtain task-adaptive features and achieve reliable feature fusion for semantic segmentation with self-supervised depth estimates. Our framework incorporates two novel components: energy-based feature fusion (EB2F) and energy-based reliable fusion Assessment (RFA) modules. The EB2F module produces task-adaptive semantic and depth features by explicitly measuring and reducing their discrepancy using Hopfield energy for better feature fusion. The RFA module evaluates the reliability of the feature fusion using an energy score to improve the effectiveness of depth guidance. Extensive experiments on two datasets demonstrate that our method achieves significant performance gains over prior works, validating the effectiveness of our energy-based learning approach.

Title: Masked Graph Autoencoder with Non-discrete Bandwidths

• Authors: Ziwen Zhao, Yuhua Li, Yixiong Zou, Jiliang Tang, Ruixuan Li
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.03814
• Pdf URL: https://arxiv.org/pdf/2402.03814
• Copy Paste: [[2402.03814]] Masked Graph Autoencoder with Non-discrete Bandwidths(https://arxiv.org/abs/2402.03814)
• Keywords: self-supervised
• Abstract: Masked graph autoencoders have emerged as a powerful graph self-supervised learning method that has yet to be fully explored. In this paper, we unveil that the existing discrete edge masking and binary link reconstruction strategies are insufficient to learn topologically informative representations, from the perspective of message propagation on graph neural networks. These limitations include blocking message flows, vulnerability to over-smoothness, and suboptimal neighborhood discriminability. Inspired by these understandings, we explore non-discrete edge masks, which are sampled from a continuous and dispersive probability distribution instead of the discrete Bernoulli distribution. These masks restrict the amount of output messages for each edge, referred to as "bandwidths". We propose a novel, informative, and effective topological masked graph autoencoder using bandwidth masking and a layer-wise bandwidth prediction objective. We demonstrate its powerful graph topological learning ability both theoretically and empirically. Our proposed framework outperforms representative baselines in both self-supervised link prediction (improving the discrete edge reconstructors by at most 20%) and node classification on numerous datasets, solely with a structure-learning pretext. Our implementation is available at https://github.com/Newiz430/Bandana.

Title: OASim: an Open and Adaptive Simulator based on Neural Rendering for Autonomous Driving

• Authors: Guohang Yan, Jiahao Pi, Jianfei Guo, Zhaotong Luo, Min Dou, Nianchen Deng, Qiusheng Huang, Daocheng Fu, Licheng Wen, Pinlong Cai, Xing Gao, Xinyu Cai, Bo Zhang, Xuemeng Yang, Yeqi Bai, Hongbin Zhou, Botian Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03830
• Pdf URL: https://arxiv.org/pdf/2402.03830
• Copy Paste: [[2402.03830]] OASim: an Open and Adaptive Simulator based on Neural Rendering for Autonomous Driving(https://arxiv.org/abs/2402.03830)
• Keywords: generative
• Abstract: With deep learning and computer vision technology development, autonomous driving provides new solutions to improve traffic safety and efficiency. The importance of building high-quality datasets is self-evident, especially with the rise of end-to-end autonomous driving algorithms in recent years. Data plays a core role in the algorithm closed-loop system. However, collecting real-world data is expensive, time-consuming, and unsafe. With the development of implicit rendering technology and in-depth research on using generative models to produce data at scale, we propose OASim, an open and adaptive simulator and autonomous driving data generator based on implicit neural rendering. It has the following characteristics: (1) High-quality scene reconstruction through neural implicit surface reconstruction technology. (2) Trajectory editing of the ego vehicle and participating vehicles. (3) Rich vehicle model library that can be freely selected and inserted into the scene. (4) Rich sensors model library where you can select specified sensors to generate data. (5) A highly customizable data generation system can generate data according to user needs. We demonstrate the high quality and fidelity of the generated data through perception performance evaluation on the Carla simulator and real-world data acquisition. Code is available at https://github.com/PJLab-ADG/OASim.

Title: Rethinking Skill Extraction in the Job Market Domain using Large Language Models

• Authors: Khanh Cao Nguyen, Mike Zhang, Syrielle Montariol, Antoine Bosselut
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.03832
• Pdf URL: https://arxiv.org/pdf/2402.03832
• Copy Paste: [[2402.03832]] Rethinking Skill Extraction in the Job Market Domain using Large Language Models(https://arxiv.org/abs/2402.03832)
• Keywords: in-context
• Abstract: Skill Extraction involves identifying skills and qualifications mentioned in documents such as job postings and resumes. The task is commonly tackled by training supervised models using a sequence labeling approach with BIO tags. However, the reliance on manually annotated data limits the generalizability of such approaches. Moreover, the common BIO setting limits the ability of the models to capture complex skill patterns and handle ambiguous mentions. In this paper, we explore the use of in-context learning to overcome these challenges, on a benchmark of 6 uniformized skill extraction datasets. Our approach leverages the few-shot learning capabilities of large language models (LLMs) to identify and extract skills from sentences. We show that LLMs, despite not being on par with traditional supervised models in terms of performance, can better handle syntactically complex skill mentions in skill extraction tasks.

Title: On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models

• Authors: Christian Horvat, Jean-Pascal Pfister
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.03845
• Pdf URL: https://arxiv.org/pdf/2402.03845
• Copy Paste: [[2402.03845]] On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models(https://arxiv.org/abs/2402.03845)
• Keywords: diffusion, generative
• Abstract: Diffusion models are generative models that have recently demonstrated impressive performances in terms of sampling quality and density estimation in high dimensions. They rely on a forward continuous diffusion process and a backward continuous denoising process, which can be described by a time-dependent vector field and is used as a generative model. In the original formulation of the diffusion model, this vector field is assumed to be the score function (i.e. it is the gradient of the log-probability at a given time in the diffusion process). Curiously, on the practical side, most studies on diffusion models implement this vector field as a neural network function and do not constrain it be the gradient of some energy function (that is, most studies do not constrain the vector field to be conservative). Even though some studies investigated empirically whether such a constraint will lead to a performance gain, they lead to contradicting results and failed to provide analytical results. Here, we provide three analytical results regarding the extent of the modeling freedom of this vector field. {Firstly, we propose a novel decomposition of vector fields into a conservative component and an orthogonal component which satisfies a given (gauge) freedom. Secondly, from this orthogonal decomposition, we show that exact density estimation and exact sampling is achieved when the conservative component is exactly equals to the true score and therefore conservativity is neither necessary nor sufficient to obtain exact density estimation and exact sampling. Finally, we show that when it comes to inferring local information of the data manifold, constraining the vector field to be conservative is desirable.

Title: ANLS* -- A Universal Document Processing Metric for Generative Large Language Models

• Authors: David Peer, Philemon Schöpf, Volckmar Nebendahl, Alexander Rietzler, Sebastian Stabinger
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.03848
• Pdf URL: https://arxiv.org/pdf/2402.03848
• Copy Paste: [[2402.03848]] ANLS* -- A Universal Document Processing Metric for Generative Large Language Models(https://arxiv.org/abs/2402.03848)
• Keywords: generative
• Abstract: Traditionally, discriminative models have been the predominant choice for tasks like document classification and information extraction. These models make predictions that fall into a limited number of predefined classes, facilitating a binary true or false evaluation and enabling the direct calculation of metrics such as the F1 score. However, recent advancements in generative large language models (GLLMs) have prompted a shift in the field due to their enhanced zero-shot capabilities, which eliminate the need for a downstream dataset and computationally expensive fine-tuning. However, evaluating GLLMs presents a challenge as the binary true or false evaluation used for discriminative models is not applicable to the predictions made by GLLMs. This paper introduces a new metric for generative models called ANLS* for evaluating a wide variety of tasks, including information extraction and classification tasks. The ANLS* metric extends existing ANLS metrics as a drop-in-replacement and is still compatible with previously reported ANLS scores. An evaluation of 7 different datasets and 3 different GLLMs using the ANLS* metric is also provided, demonstrating the importance of the proposed metric. We also benchmark a novel approach to generate prompts for documents, called SFT, against other prompting techniques such as LATIN. In 15 out of 21 cases, SFT outperforms other techniques and improves the state-of-the-art, sometimes by as much as $15$ percentage points. Sources are available at https://github.com/deepopinion/anls_star_metric

Title: MOMENT: A Family of Open Time-series Foundation Models

• Authors: Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, Artur Dubrawski
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.03885
• Pdf URL: https://arxiv.org/pdf/2402.03885
• Copy Paste: [[2402.03885]] MOMENT: A Family of Open Time-series Foundation Models(https://arxiv.org/abs/2402.03885)
• Keywords: foundation model
• Abstract: We introduce MOMENT, a family of open-source foundation models for general-purpose time-series analysis. Pre-training large models on time-series data is challenging due to (1) the absence of a large and cohesive public time-series repository, and (2) diverse time-series characteristics which make multi-dataset training onerous. Additionally, (3) experimental benchmarks to evaluate these models, especially in scenarios with limited resources, time, and supervision, are still in their nascent stages. To address these challenges, we compile a large and diverse collection of public time-series, called the Time-series Pile, and systematically tackle time-series-specific challenges to unlock large-scale multi-dataset pre-training. Finally, we build on recent work to design a benchmark to evaluate time-series foundation models on diverse tasks and datasets in limited supervision settings. Experiments on this benchmark demonstrate the effectiveness of our pre-trained models with minimal data and task-specific fine-tuning. Finally, we present several interesting empirical observations about large pre-trained time-series models. Our code is available anonymously at anonymous.4open.science/r/BETT-773F/.

Title: EscherNet: A Generative Model for Scalable View Synthesis

• Authors: Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, Andrew J. Davison
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03908
• Pdf URL: https://arxiv.org/pdf/2402.03908
• Copy Paste: [[2402.03908]] EscherNet: A Generative Model for Scalable View Synthesis(https://arxiv.org/abs/2402.03908)
• Keywords: diffusion, generative
• Abstract: We introduce EscherNet, a multi-view conditioned diffusion model for view synthesis. EscherNet learns implicit and generative 3D representations coupled with a specialised camera positional encoding, allowing precise and continuous relative control of the camera transformation between an arbitrary number of reference and target views. EscherNet offers exceptional generality, flexibility, and scalability in view synthesis -- it can generate more than 100 consistent target views simultaneously on a single consumer-grade GPU, despite being trained with a fixed number of 3 reference views to 3 target views. As a result, EscherNet not only addresses zero-shot novel view synthesis, but also naturally unifies single- and multi-image 3D reconstruction, combining these diverse tasks into a single, cohesive framework. Our extensive experiments demonstrate that EscherNet achieves state-of-the-art performance in multiple benchmarks, even when compared to methods specifically tailored for each individual problem. This remarkable versatility opens up new directions for designing scalable neural architectures for 3D vision. Project page: \url{https://kxhit.github.io/EscherNet}.

Title: IMUSIC: IMU-based Facial Expression Capture

• Authors: Youjia Wang, Yiwen Wu, Ruiqian Li, Hengan Zhou, Hongyang Lin, Yingwenqi Jiang, Yingsheng Zhu, Guanpeng Long, Jingya Wang, Lan Xu, Jingyi Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03944
• Pdf URL: https://arxiv.org/pdf/2402.03944
• Copy Paste: [[2402.03944]] IMUSIC: IMU-based Facial Expression Capture(https://arxiv.org/abs/2402.03944)
• Keywords: diffusion
• Abstract: For facial motion capture and analysis, the dominated solutions are generally based on visual cues, which cannot protect privacy and are vulnerable to occlusions. Inertial measurement units (IMUs) serve as potential rescues yet are mainly adopted for full-body motion capture. In this paper, we propose IMUSIC to fill the gap, a novel path for facial expression capture using purely IMU signals, significantly distant from previous visual solutions.The key design in our IMUSIC is a trilogy. We first design micro-IMUs to suit facial capture, companion with an anatomy-driven IMU placement scheme. Then, we contribute a novel IMU-ARKit dataset, which provides rich paired IMU/visual signals for diverse facial expressions and performances. Such unique multi-modality brings huge potential for future directions like IMU-based facial behavior analysis. Moreover, utilizing IMU-ARKit, we introduce a strong baseline approach to accurately predict facial blendshape parameters from purely IMU signals. Specifically, we tailor a Transformer diffusion model with a two-stage training strategy for this novel tracking task. The IMUSIC framework empowers us to perform accurate facial capture in scenarios where visual methods falter and simultaneously safeguard user privacy. We conduct extensive experiments about both the IMU configuration and technical components to validate the effectiveness of our IMUSIC approach. Notably, IMUSIC enables various potential and novel applications, i.e., privacy-protecting facial capture, hybrid capture against occlusions, or detecting minute facial movements that are often invisible through visual cues. We will release our dataset and implementations to enrich more possibilities of facial capture and analysis in our community.

Title: In-context learning agents are asymmetric belief updaters

• Authors: Johannes A. Schubert, Akshay K. Jagadish, Marcel Binz, Eric Schulz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.03969
• Pdf URL: https://arxiv.org/pdf/2402.03969
• Copy Paste: [[2402.03969]] In-context learning agents are asymmetric belief updaters(https://arxiv.org/abs/2402.03969)
• Keywords: in-context
• Abstract: We study the in-context learning dynamics of large language models (LLMs) using three instrumental learning tasks adapted from cognitive psychology. We find that LLMs update their beliefs in an asymmetric manner and learn more from better-than-expected outcomes than from worse-than-expected ones. Furthermore, we show that this effect reverses when learning about counterfactual feedback and disappears when no agency is implied. We corroborate these findings by investigating idealized in-context learning agents derived through meta-reinforcement learning, where we observe similar patterns. Taken together, our results contribute to our understanding of how in-context learning works by highlighting that the framing of a problem significantly influences how learning occurs, a phenomenon also observed in human cognition.

Title: Controllable Diverse Sampling for Diffusion Based Motion Behavior Forecasting

• Authors: Yiming Xu, Hao Cheng, Monika Sester
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.03981
• Pdf URL: https://arxiv.org/pdf/2402.03981
• Copy Paste: [[2402.03981]] Controllable Diverse Sampling for Diffusion Based Motion Behavior Forecasting(https://arxiv.org/abs/2402.03981)
• Keywords: diffusion, generative
• Abstract: In autonomous driving tasks, trajectory prediction in complex traffic environments requires adherence to real-world context conditions and behavior multimodalities. Existing methods predominantly rely on prior assumptions or generative models trained on curated data to learn road agents' stochastic behavior bounded by scene constraints. However, they often face mode averaging issues due to data imbalance and simplistic priors, and could even suffer from mode collapse due to unstable training and single ground truth supervision. These issues lead the existing methods to a loss of predictive diversity and adherence to the scene constraints. To address these challenges, we introduce a novel trajectory generator named Controllable Diffusion Trajectory (CDT), which integrates map information and social interactions into a Transformer-based conditional denoising diffusion model to guide the prediction of future trajectories. To ensure multimodality, we incorporate behavioral tokens to direct the trajectory's modes, such as going straight, turning right or left. Moreover, we incorporate the predicted endpoints as an alternative behavioral token into the CDT model to facilitate the prediction of accurate trajectories. Extensive experiments on the Argoverse 2 benchmark demonstrate that CDT excels in generating diverse and scene-compliant trajectories in complex urban settings.

Title: Space Group Constrained Crystal Generation

• Authors: Rui Jiao, Wenbing Huang, Yu Liu, Deli Zhao, Yang Liu
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2402.03992
• Pdf URL: https://arxiv.org/pdf/2402.03992
• Copy Paste: [[2402.03992]] Space Group Constrained Crystal Generation(https://arxiv.org/abs/2402.03992)
• Keywords: diffusion
• Abstract: Crystals are the foundation of numerous scientific and industrial applications. While various learning-based approaches have been proposed for crystal generation, existing methods seldom consider the space group constraint which is crucial in describing the geometry of crystals and closely relevant to many desirable properties. However, considering space group constraint is challenging owing to its diverse and nontrivial forms. In this paper, we reduce the space group constraint into an equivalent formulation that is more tractable to be handcrafted into the generation process. In particular, we translate the space group constraint into two parts: the basis constraint of the invariant logarithmic space of the lattice matrix and the Wyckoff position constraint of the fractional coordinates. Upon the derived constraints, we then propose DiffCSP++, a novel diffusion model that has enhanced a previous work DiffCSP by further taking space group constraint into account. Experiments on several popular datasets verify the benefit of the involvement of the space group constraint, and show that our DiffCSP++ achieves promising performance on crystal structure prediction, ab initio crystal generation and controllable generation with customized space groups.

Title: Polyp-DDPM: Diffusion-Based Semantic Polyp Synthesis for Enhanced Segmentation

• Authors: Zolnamar Dorjsembe, Hsing-Kuo Pao, Furen Xiao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04031
• Pdf URL: https://arxiv.org/pdf/2402.04031
• Copy Paste: [[2402.04031]] Polyp-DDPM: Diffusion-Based Semantic Polyp Synthesis for Enhanced Segmentation(https://arxiv.org/abs/2402.04031)
• Keywords: diffusion
• Abstract: This study introduces Polyp-DDPM, a diffusion-based method for generating realistic images of polyps conditioned on masks, aimed at enhancing the segmentation of gastrointestinal (GI) tract polyps. Our approach addresses the challenges of data limitations, high annotation costs, and privacy concerns associated with medical images. By conditioning the diffusion model on segmentation masks-binary masks that represent abnormal areas-Polyp-DDPM outperforms state-of-the-art methods in terms of image quality (achieving a Frechet Inception Distance (FID) score of 78.47, compared to scores above 83.79) and segmentation performance (achieving an Intersection over Union (IoU) of 0.7156, versus less than 0.6694 for synthetic images from baseline models and 0.7067 for real data). Our method generates a high-quality, diverse synthetic dataset for training, thereby enhancing polyp segmentation models to be comparable with real images and offering greater data augmentation capabilities to improve segmentation models. The source code and pretrained weights for Polyp-DDPM are made publicly available at https://github.com/mobaidoctor/polyp-ddpm.

Title: Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning

• Authors: Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas B. Schön, Per Mattsson
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.04080
• Pdf URL: https://arxiv.org/pdf/2402.04080
• Copy Paste: [[2402.04080]] Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning(https://arxiv.org/abs/2402.04080)
• Keywords: diffusion
• Abstract: This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distribution into a standard Gaussian and then samples actions conditioned on the environment state with a corresponding reverse-time SDE, like a typical diffusion policy. We show that such an SDE has a solution that we can use to calculate the log probability of the policy, yielding an entropy regularizer that improves the exploration of offline datasets. To mitigate the impact of inaccurate value functions from out-of-distribution data points, we further propose to learn the lower confidence bound of Q-ensembles for more robust policy improvement. By combining the entropy-regularized diffusion policy with Q-ensembles in offline RL, our method achieves state-of-the-art performance on most tasks in D4RL benchmarks. Code is available at \href{https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble}{https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble}.

Title: Improved Generalization of Weight Space Networks via Augmentations

• Authors: Aviv Shamsian, Aviv Navon, David W. Zhang, Yan Zhang, Ethan Fetaya, Gal Chechik, Haggai Maron
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04081
• Pdf URL: https://arxiv.org/pdf/2402.04081
• Copy Paste: [[2402.04081]] Improved Generalization of Weight Space Networks via Augmentations(https://arxiv.org/abs/2402.04081)
• Keywords: self-supervised
• Abstract: Learning in deep weight spaces (DWS), where neural networks process the weights of other neural networks, is an emerging research direction, with applications to 2D and 3D neural fields (INRs, NeRFs), as well as making inferences about other types of neural networks. Unfortunately, weight space models tend to suffer from substantial overfitting. We empirically analyze the reasons for this overfitting and find that a key reason is the lack of diversity in DWS datasets. While a given object can be represented by many different weight configurations, typical INR training sets fail to capture variability across INRs that represent the same object. To address this, we explore strategies for data augmentation in weight spaces and propose a MixUp method adapted for weight spaces. We demonstrate the effectiveness of these methods in two setups. In classification, they improve performance similarly to having up to 10 times more data. In self-supervised contrastive learning, they yield substantial 5-10% gains in downstream classification.

Title: VRMM: A Volumetric Relightable Morphable Head Model

• Authors: Haotian Yang, Mingwu Zheng, Chongyang Ma, Yu-Kun Lai, Pengfei Wan, Haibin Huang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.04101
• Pdf URL: https://arxiv.org/pdf/2402.04101
• Copy Paste: [[2402.04101]] VRMM: A Volumetric Relightable Morphable Head Model(https://arxiv.org/abs/2402.04101)
• Keywords: self-supervised, generative
• Abstract: In this paper, we introduce the Volumetric Relightable Morphable Model (VRMM), a novel volumetric and parametric facial prior for 3D face modeling. While recent volumetric prior models offer improvements over traditional methods like 3D Morphable Models (3DMMs), they face challenges in model learning and personalized reconstructions. Our VRMM overcomes these by employing a novel training framework that efficiently disentangles and encodes latent spaces of identity, expression, and lighting into low-dimensional representations. This framework, designed with self-supervised learning, significantly reduces the constraints for training data, making it more feasible in practice. The learned VRMM offers relighting capabilities and encompasses a comprehensive range of expressions. We demonstrate the versatility and effectiveness of VRMM through various applications like avatar generation, facial reconstruction, and animation. Additionally, we address the common issue of overfitting in generative volumetric models with a novel prior-preserving personalization framework based on VRMM. Such an approach enables accurate 3D face reconstruction from even a single portrait input. Our experiments showcase the potential of VRMM to significantly enhance the field of 3D face modeling.

Title: Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains

• Authors: Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Martin Jaggi, Hyeji Kim, Michael Gastpar
• Subjects: cs.LG, cs.CL, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.04161
• Pdf URL: https://arxiv.org/pdf/2402.04161
• Copy Paste: [[2402.04161]] Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains(https://arxiv.org/abs/2402.04161)
• Keywords: generative
• Abstract: In recent years, attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. A key ingredient behind their success is the generative pretraining procedure, during which these models are trained on a large text corpus in an auto-regressive manner. To shed light on this phenomenon, we propose a new framework that allows both theory and systematic experiments to study the sequential modeling capabilities of transformers through the lens of Markov chains. Inspired by the Markovianity of natural languages, we model the data as a Markovian source and utilize this framework to systematically study the interplay between the data-distributional properties, the transformer architecture, the learnt distribution, and the final model performance. In particular, we theoretically characterize the loss landscape of single-layer transformers and show the existence of global minima and bad local minima contingent upon the specific data characteristics and the transformer architecture. Backed by experiments, we demonstrate that our theoretical findings are in congruence with the empirical results. We further investigate these findings in the broader context of higher order Markov chains and deeper architectures, and outline open problems in this arena. Code is available at \url{https://github.com/Bond1995/Markov}.

Title: SHIELD : An Evaluation Benchmark for Face Spoofing and Forgery Detection with Multimodal Large Language Models

• Authors: Yichen Shi, Yuhao Gao, Yingxin Lai, Hongyang Wang, Jun Feng, Lei He, Jun Wan, Changsheng Chen, Zitong Yu, Xiaochun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04178
• Pdf URL: https://arxiv.org/pdf/2402.04178
• Copy Paste: [[2402.04178]] SHIELD : An Evaluation Benchmark for Face Spoofing and Forgery Detection with Multimodal Large Language Models(https://arxiv.org/abs/2402.04178)
• Keywords: diffusion
• Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable problem-solving capabilities in various vision fields (e.g., generic object recognition and grounding) based on strong visual semantic representation and language reasoning ability. However, whether MLLMs are sensitive to subtle visual spoof/forged clues and how they perform in the domain of face attack detection (e.g., face spoofing and forgery detection) is still unexplored. In this paper, we introduce a new benchmark, namely SHIELD, to evaluate the ability of MLLMs on face spoofing and forgery detection. Specifically, we design true/false and multiple-choice questions to evaluate multimodal face data in these two face security tasks. For the face anti-spoofing task, we evaluate three different modalities (i.e., RGB, infrared, depth) under four types of presentation attacks (i.e., print attack, replay attack, rigid mask, paper mask). For the face forgery detection task, we evaluate GAN-based and diffusion-based data with both visual and acoustic modalities. Each question is subjected to both zero-shot and few-shot tests under standard and chain of thought (COT) settings. The results indicate that MLLMs hold substantial potential in the face security domain, offering advantages over traditional specific models in terms of interpretability, multimodal flexible reasoning, and joint face spoof and forgery detection. Additionally, we develop a novel Multi-Attribute Chain of Thought (MA-COT) paradigm for describing and judging various task-specific and task-irrelevant attributes of face images, which provides rich task-related knowledge for subtle spoof/forged clue mining. Extensive experiments in separate face anti-spoofing, separate face forgery detection, and joint detection tasks demonstrate the effectiveness of the proposed MA-COT. The project is available at https$:$//github.com/laiyingxin2/SHIELD

Title: Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks

• Authors: Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, Dimitris Papailiopoulos
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04248
• Pdf URL: https://arxiv.org/pdf/2402.04248
• Copy Paste: [[2402.04248]] Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks(https://arxiv.org/abs/2402.04248)
• Keywords: in-context
• Abstract: State-space models (SSMs), such as Mamba Gu & Dao (2034), have been proposed as alternatives to Transformer networks in language modeling, by incorporating gating, convolutions, and input-dependent token selection to mitigate the quadratic cost of multi-head attention. Although SSMs exhibit competitive performance, their in-context learning (ICL) capabilities, a remarkable emergent property of modern language models that enables task execution without parameter optimization, remain underexplored compared to Transformers. In this study, we evaluate the ICL performance of SSMs, focusing on Mamba, against Transformer models across various tasks. Our results show that SSMs perform comparably to Transformers in standard regression ICL tasks, while outperforming them in tasks like sparse parity learning. However, SSMs fall short in tasks involving non-standard retrieval functionality. To address these limitations, we introduce a hybrid model, \variant, that combines Mamba with attention blocks, surpassing individual models in tasks where they struggle independently. Our findings suggest that hybrid architectures offer promising avenues for enhancing ICL in language models.

Title: EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters

• Authors: Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Xinlong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04252
• Pdf URL: https://arxiv.org/pdf/2402.04252
• Copy Paste: [[2402.04252]] EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters(https://arxiv.org/abs/2402.04252)
• Keywords: foundation model
• Abstract: Scaling up contrastive language-image pretraining (CLIP) is critical for empowering both vision and multimodal models. We present EVA-CLIP-18B, the largest and most powerful open-source CLIP model to date, with 18-billion parameters. With only 6-billion training samples seen, EVA-CLIP-18B achieves an exceptional 80.7% zero-shot top-1 accuracy averaged across 27 widely recognized image classification benchmarks, outperforming its forerunner EVA-CLIP (5-billion parameters) and other open-source CLIP models by a large margin. Remarkably, we observe a consistent performance improvement with the model size scaling of EVA-CLIP, despite maintaining a constant training dataset of 2-billion image-text pairs from LAION-2B and COYO-700M. This dataset is openly available and much smaller than the in-house datasets (e.g., DFN-5B, WebLI-10B) employed in other state-of-the-art CLIP models. EVA-CLIP-18B demonstrates the potential of EVA-style weak-to-strong visual model scaling. With our model weights made publicly available, we hope to facilitate future research in vision and multimodal foundation models.