2024-10-31

Title: Accelerating Augmentation Invariance Pretraining

• Authors: Jinhong Lin, Cheng-En Wu, Yibing Wei, Pedro Morgado
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22364
• Pdf URL: https://arxiv.org/pdf/2410.22364
• Copy Paste: [[2410.22364]] Accelerating Augmentation Invariance Pretraining(https://arxiv.org/abs/2410.22364)
• Keywords: self-supervised
• Abstract: Our work tackles the computational challenges of contrastive learning methods, particularly for the pretraining of Vision Transformers (ViTs). Despite the effectiveness of contrastive learning, the substantial computational resources required for training often hinder their practical application. To mitigate this issue, we propose an acceleration framework, leveraging ViT's unique ability to generalize across inputs of varying sequence lengths. Our method employs a mix of sequence compression strategies, including randomized token dropout and flexible patch scaling, to reduce the cost of gradient estimation and accelerate convergence. We further provide an in-depth analysis of the gradient estimation error of various acceleration strategies as well as their impact on downstream tasks, offering valuable insights into the trade-offs between acceleration and performance. We also propose a novel procedure to identify an optimal acceleration schedule to adjust the sequence compression ratios to the training progress, ensuring efficient training without sacrificing downstream performance. Our approach significantly reduces computational overhead across various self-supervised learning algorithms on large-scale datasets. In ImageNet, our method achieves speedups of 4$\times$ in MoCo, 3.3$\times$ in SimCLR, and 2.5$\times$ in DINO, demonstrating substantial efficiency gains.

Title: Unpacking SDXL Turbo: Interpreting Text-to-Image Models with Sparse Autoencoders

• Authors: Viacheslav Surkov, Chris Wendler, Mikhail Terekhov, Justin Deschenaux, Robert West, Caglar Gulcehre
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22366
• Pdf URL: https://arxiv.org/pdf/2410.22366
• Copy Paste: [[2410.22366]] Unpacking SDXL Turbo: Interpreting Text-to-Image Models with Sparse Autoencoders(https://arxiv.org/abs/2410.22366)
• Keywords: diffusion, generative
• Abstract: Sparse autoencoders (SAEs) have become a core ingredient in the reverse engineering of large-language models (LLMs). For LLMs, they have been shown to decompose intermediate representations that often are not interpretable directly into sparse sums of interpretable features, facilitating better control and subsequent analysis. However, similar analyses and approaches have been lacking for text-to-image models. We investigated the possibility of using SAEs to learn interpretable features for a few-step text-to-image diffusion models, such as SDXL Turbo. To this end, we train SAEs on the updates performed by transformer blocks within SDXL Turbo's denoising U-net. We find that their learned features are interpretable, causally influence the generation process, and reveal specialization among the blocks. In particular, we find one block that deals mainly with image composition, one that is mainly responsible for adding local details, and one for color, illumination, and style. Therefore, our work is an important first step towards better understanding the internals of generative text-to-image models like SDXL Turbo and showcases the potential of features learned by SAEs for the visual domain. Code is available at this https URL

Title: Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance

• Authors: Dongmin Park, Sebin Kim, Taehong Moon, Minkyu Kim, Kangwook Lee, Jaewoong Cho
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22376
• Pdf URL: https://arxiv.org/pdf/2410.22376
• Copy Paste: [[2410.22376]] Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance(https://arxiv.org/abs/2410.22376)
• Keywords: diffusion
• Abstract: State-of-the-art text-to-image (T2I) diffusion models often struggle to generate rare compositions of concepts, e.g., objects with unusual attributes. In this paper, we show that the compositional generation power of diffusion models on such rare concepts can be significantly enhanced by the Large Language Model (LLM) guidance. We start with empirical and theoretical analysis, demonstrating that exposing frequent concepts relevant to the target rare concepts during the diffusion sampling process yields more accurate concept composition. Based on this, we propose a training-free approach, R2F, that plans and executes the overall rare-to-frequent concept guidance throughout the diffusion inference by leveraging the abundant semantic knowledge in LLMs. Our framework is flexible across any pre-trained diffusion models and LLMs, and can be seamlessly integrated with the region-guided diffusion approaches. Extensive experiments on three datasets, including our newly proposed benchmark, RareBench, containing various prompts with rare compositions of concepts, R2F significantly surpasses existing models including SD3.0 and FLUX by up to 28.1%p in T2I alignment. Code is available at this https URL.

Title: Discrete Modeling via Boundary Conditional Diffusion Processes

• Authors: Yuxuan Gu, Xiaocheng Feng, Lei Huang, Yingsheng Wu, Zekun Zhou, Weihong Zhong, Kun Zhu, Bing Qin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22380
• Pdf URL: https://arxiv.org/pdf/2410.22380
• Copy Paste: [[2410.22380]] Discrete Modeling via Boundary Conditional Diffusion Processes(https://arxiv.org/abs/2410.22380)
• Keywords: diffusion
• Abstract: We present an novel framework for efficiently and effectively extending the powerful continuous diffusion processes to discrete modeling. Previous approaches have suffered from the discrepancy between discrete data and continuous modeling. Our study reveals that the absence of guidance from discrete boundaries in learning probability contours is one of the main reasons. To address this issue, we propose a two-step forward process that first estimates the boundary as a prior distribution and then rescales the forward trajectory to construct a boundary conditional diffusion model. The reverse process is proportionally adjusted to guarantee that the learned contours yield more precise discrete data. Experimental results indicate that our approach achieves strong performance in both language modeling and discrete image generation tasks. In language modeling, our approach surpasses previous state-of-the-art continuous diffusion language models in three translation tasks and a summarization task, while also demonstrating competitive performance compared to auto-regressive transformers. Moreover, our method achieves comparable results to continuous diffusion models when using discrete ordinal pixels and establishes a new state-of-the-art for categorical image generation on the Cifar-10 dataset.

Title: Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss

• Authors: José Manuel de Frutos, Manuel A. Vázquez, Pablo Olmos, Joaquín Míguez
• Subjects: cs.LG, cs.AI, stat.CO, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.22381
• Pdf URL: https://arxiv.org/pdf/2410.22381
• Copy Paste: [[2410.22381]] Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss(https://arxiv.org/abs/2410.22381)
• Keywords: generative
• Abstract: Traditional implicit generative models are capable of learning highly complex data distributions. However, their training involves distinguishing real data from synthetically generated data using adversarial discriminators, which can lead to unstable training dynamics and mode dropping issues. In this work, we build on the \textit{invariant statistical loss} (ISL) method introduced in \cite{de2024training}, and extend it to handle heavy-tailed and multivariate data distributions. The data generated by many real-world phenomena can only be properly characterised using heavy-tailed probability distributions, and traditional implicit methods struggle to effectively capture their asymptotic behavior. To address this problem, we introduce a generator trained with ISL, that uses input noise from a generalised Pareto distribution (GPD). We refer to this generative scheme as Pareto-ISL for conciseness. Our experiments demonstrate that Pareto-ISL accurately models the tails of the distributions while still effectively capturing their central characteristics. The original ISL function was conceived for 1D data sets. When the actual data is $n$-dimensional, a straightforward extension of the method was obtained by targeting the $n$ marginal distributions of the data. This approach is computationally infeasible and ineffective in high-dimensional spaces. To overcome this, we extend the 1D approach using random projections and define a new loss function suited for multivariate data, keeping problems tractable by adjusting the number of projections. We assess its performance in multidimensional generative modeling and explore its potential as a pretraining technique for generative adversarial networks (GANs) to prevent mode collapse, reporting promising results and highlighting its robustness across various hyperparameter settings.

Title: Embedding Watermarks in Diffusion Process for Model Intellectual Property Protection

• Authors: Jijia Yang, Sen Peng, Xiaohua Jia
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.22445
• Pdf URL: https://arxiv.org/pdf/2410.22445
• Copy Paste: [[2410.22445]] Embedding Watermarks in Diffusion Process for Model Intellectual Property Protection(https://arxiv.org/abs/2410.22445)
• Keywords: diffusion
• Abstract: In practical application, the widespread deployment of diffusion models often necessitates substantial investment in training. As diffusion models find increasingly diverse applications, concerns about potential misuse highlight the imperative for robust intellectual property protection. Current protection strategies either employ backdoor-based methods, integrating a watermark task as a simpler training objective with the main model task, or embedding watermarks directly into the final output samples. However, the former approach is fragile compared to existing backdoor defense techniques, while the latter fundamentally alters the expected output. In this work, we introduce a novel watermarking framework by embedding the watermark into the whole diffusion process, and theoretically ensure that our final output samples contain no additional information. Furthermore, we utilize statistical algorithms to verify the watermark from internally generated model samples without necessitating triggers as conditions. Detailed theoretical analysis and experimental validation demonstrate the effectiveness of our proposed method.

Title: Brain age identification from diffusion MRI synergistically predicts neurodegenerative disease

• Authors: Chenyu Gao, Michael E. Kim, Karthik Ramadass, Praitayini Kanakaraj, Aravind R. Krishnan, Adam M. Saunders, Nancy R. Newlin, Ho Hin Lee, Qi Yang, Warren D. Taylor, Brian D. Boyd, Lori L. Beason-Held, Susan M. Resnick, Lisa L. Barnes, David A. Bennett, Katherine D. Van Schaik, Derek B. Archer, Timothy J. Hohman, Angela L. Jefferson, Ivana Išgum, Daniel Moyer, Yuankai Huo, Kurt G. Schilling, Lianrui Zuo, Shunxing Bao, Nazirah Mohd Khairi, Zhiyuan Li, Christos Davatzikos, Bennett A. Landman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22454
• Pdf URL: https://arxiv.org/pdf/2410.22454
• Copy Paste: [[2410.22454]] Brain age identification from diffusion MRI synergistically predicts neurodegenerative disease(https://arxiv.org/abs/2410.22454)
• Keywords: diffusion, generative
• Abstract: Estimated brain age from magnetic resonance image (MRI) and its deviation from chronological age can provide early insights into potential neurodegenerative diseases, supporting early detection and implementation of prevention strategies. Diffusion MRI (dMRI), a widely used modality for brain age estimation, presents an opportunity to build an earlier biomarker for neurodegenerative disease prediction because it captures subtle microstructural changes that precede more perceptible macrostructural changes. However, the coexistence of macro- and micro-structural information in dMRI raises the question of whether current dMRI-based brain age estimation models are leveraging the intended microstructural information or if they inadvertently rely on the macrostructural information. To develop a microstructure-specific brain age, we propose a method for brain age identification from dMRI that minimizes the model's use of macrostructural information by non-rigidly registering all images to a standard template. Imaging data from 13,398 participants across 12 datasets were used for the training and evaluation. We compare our brain age models, trained with and without macrostructural information minimized, with an architecturally similar T1-weighted (T1w) MRI-based brain age model and two state-of-the-art T1w MRI-based brain age models that primarily use macrostructural information. We observe difference between our dMRI-based brain age and T1w MRI-based brain age across stages of neurodegeneration, with dMRI-based brain age being older than T1w MRI-based brain age in participants transitioning from cognitively normal (CN) to mild cognitive impairment (MCI), but younger in participants already diagnosed with Alzheimer's disease (AD). Approximately 4 years before MCI diagnosis, dMRI-based brain age yields better performance than T1w MRI-based brain ages in predicting transition from CN to MCI.

Title: Learning Identifiable Factorized Causal Representations of Cellular Responses

• Authors: Haiyi Mao, Romain Lopez, Kai Liu, Jan-Christian Huetter, David Richmond, Panayiotis Benos, Lin Qiu
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.22472
• Pdf URL: https://arxiv.org/pdf/2410.22472
• Copy Paste: [[2410.22472]] Learning Identifiable Factorized Causal Representations of Cellular Responses(https://arxiv.org/abs/2410.22472)
• Keywords: generative
• Abstract: The study of cells and their responses to genetic or chemical perturbations promises to accelerate the discovery of therapeutic targets. However, designing adequate and insightful models for such data is difficult because the response of a cell to perturbations essentially depends on its biological context (e.g., genetic background or cell type). For example, while discovering therapeutic targets, one may want to enrich for drugs that specifically target a certain cell type. This challenge emphasizes the need for methods that explicitly take into account potential interactions between drugs and contexts. Towards this goal, we propose a novel Factorized Causal Representation (FCR) learning method that reveals causal structure in single-cell perturbation data from several cell lines. Based on the framework of identifiable deep generative models, FCR learns multiple cellular representations that are disentangled, comprised of covariate-specific ($\mathbf{z}_x$), treatment-specific ($\mathbf{z}_{t}$), and interaction-specific ($\mathbf{z}_{tx}$) blocks. Based on recent advances in non-linear ICA theory, we prove the component-wise identifiability of $\mathbf{z}_{tx}$ and block-wise identifiability of $\mathbf{z}_t$ and $\mathbf{z}_x$. Then, we present our implementation of FCR, and empirically demonstrate that it outperforms state-of-the-art baselines in various tasks across four single-cell datasets.

Title: Unlocking Point Processes through Point Set Diffusion

• Authors: David Lüdke, Enric Rabasseda Raventós, Marcel Kollovieh, Stephan Günnemann
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.22493
• Pdf URL: https://arxiv.org/pdf/2410.22493
• Copy Paste: [[2410.22493]] Unlocking Point Processes through Point Set Diffusion(https://arxiv.org/abs/2410.22493)
• Keywords: diffusion
• Abstract: Point processes model the distribution of random point sets in mathematical spaces, such as spatial and temporal domains, with applications in fields like seismology, neuroscience, and economics. Existing statistical and machine learning models for point processes are predominantly constrained by their reliance on the characteristic intensity function, introducing an inherent trade-off between efficiency and flexibility. In this paper, we introduce Point Set Diffusion, a diffusion-based latent variable model that can represent arbitrary point processes on general metric spaces without relying on the intensity function. By directly learning to stochastically interpolate between noise and data point sets, our approach enables efficient, parallel sampling and flexible generation for complex conditional tasks defined on the metric space. Experiments on synthetic and real-world datasets demonstrate that Point Set Diffusion achieves state-of-the-art performance in unconditional and conditional generation of spatial and spatiotemporal point processes while providing up to orders of magnitude faster sampling than autoregressive baselines.

Title: FairSkin: Fair Diffusion for Skin Disease Image Generation

• Authors: Ruichen Zhang, Yuguang Yao, Zhen Tan, Zhiming Li, Pan Wang, Jingtong Hu, Sijia Liu, Tianlong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22551
• Pdf URL: https://arxiv.org/pdf/2410.22551
• Copy Paste: [[2410.22551]] FairSkin: Fair Diffusion for Skin Disease Image Generation(https://arxiv.org/abs/2410.22551)
• Keywords: diffusion
• Abstract: Image generation is a prevailing technique for clinical data augmentation for advancing diagnostic accuracy and reducing healthcare disparities. Diffusion Model (DM) has become a leading method in generating synthetic medical images, but it suffers from a critical twofold bias: (1) The quality of images generated for Caucasian individuals is significantly higher, as measured by the Frechet Inception Distance (FID). (2) The ability of the downstream-task learner to learn critical features from disease images varies across different skin tones. These biases pose significant risks, particularly in skin disease detection, where underrepresentation of certain skin tones can lead to misdiagnosis or neglect of specific conditions. To address these challenges, we propose FairSkin, a novel DM framework that mitigates these biases through a three-level resampling mechanism, ensuring fairer representation across racial and disease categories. Our approach significantly improves the diversity and quality of generated images, contributing to more equitable skin disease detection in clinical settings.

Title: Unsupervised Multimodal Fusion of In-process Sensor Data for Advanced Manufacturing Process Monitoring

• Authors: Matthew McKinney, Anthony Garland, Dale Cillessen, Jesse Adamczyk, Dan Bolintineanu, Michael Heiden, Elliott Fowler, Brad L. Boyce
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.22558
• Pdf URL: https://arxiv.org/pdf/2410.22558
• Copy Paste: [[2410.22558]] Unsupervised Multimodal Fusion of In-process Sensor Data for Advanced Manufacturing Process Monitoring(https://arxiv.org/abs/2410.22558)
• Keywords: anomaly
• Abstract: Effective monitoring of manufacturing processes is crucial for maintaining product quality and operational efficiency. Modern manufacturing environments generate vast amounts of multimodal data, including visual imagery from various perspectives and resolutions, hyperspectral data, and machine health monitoring information such as actuator positions, accelerometer readings, and temperature measurements. However, interpreting this complex, high-dimensional data presents significant challenges, particularly when labeled datasets are unavailable. This paper presents a novel approach to multimodal sensor data fusion in manufacturing processes, inspired by the Contrastive Language-Image Pre-training (CLIP) model. We leverage contrastive learning techniques to correlate different data modalities without the need for labeled data, developing encoders for five distinct modalities: visual imagery, audio signals, laser position (x and y coordinates), and laser power measurements. By compressing these high-dimensional datasets into low-dimensional representational spaces, our approach facilitates downstream tasks such as process control, anomaly detection, and quality assurance. We evaluate the effectiveness of our approach through experiments, demonstrating its potential to enhance process monitoring capabilities in advanced manufacturing systems. This research contributes to smart manufacturing by providing a flexible, scalable framework for multimodal data fusion that can adapt to diverse manufacturing environments and sensor configurations.

Title: Unpicking Data at the Seams: VAEs, Disentanglement and Independent Components

• Authors: Carl Allen
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.22559
• Pdf URL: https://arxiv.org/pdf/2410.22559
• Copy Paste: [[2410.22559]] Unpicking Data at the Seams: VAEs, Disentanglement and Independent Components(https://arxiv.org/abs/2410.22559)
• Keywords: diffusion, generative
• Abstract: Disentanglement, or identifying salient statistically independent factors of the data, is of interest in many areas of machine learning and statistics, with relevance to synthetic data generation with controlled properties, robust classification of features, parsimonious encoding, and a greater understanding of the generative process underlying the data. Disentanglement arises in several generative paradigms, including Variational Autoencoders (VAEs), Generative Adversarial Networks and diffusion models. Particular progress has recently been made in understanding disentanglement in VAEs, where the choice of diagonal posterior covariance matrices is shown to promote mutual orthogonality between columns of the decoder's Jacobian. We continue this thread to show how this linear independence translates to statistical independence, completing the chain in understanding how the VAE's objective identifies independent components of, or disentangles, the data.

Title: Flow Matching for Posterior Inference with Simulator Feedback

• Authors: Benjamin Holzschuh, Nils Thuerey
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.22573
• Pdf URL: https://arxiv.org/pdf/2410.22573
• Copy Paste: [[2410.22573]] Flow Matching for Posterior Inference with Simulator Feedback(https://arxiv.org/abs/2410.22573)
• Keywords: generative
• Abstract: Flow-based generative modeling is a powerful tool for solving inverse problems in physical sciences that can be used for sampling and likelihood evaluation with much lower inference times than traditional methods. We propose to refine flows with additional control signals based on a simulator. Control signals can include gradients and a problem-specific cost function if the simulator is differentiable, or they can be fully learned from the simulator output. In our proposed method, we pretrain the flow network and include feedback from the simulator exclusively for finetuning, therefore requiring only a small amount of additional parameters and compute. We motivate our design choices on several benchmark problems for simulation-based inference and evaluate flow matching with simulator feedback against classical MCMC methods for modeling strong gravitational lens systems, a challenging inverse problem in astronomy. We demonstrate that including feedback from the simulator improves the accuracy by $53\%$, making it competitive with traditional techniques while being up to $67$x faster for inference.

Title: BENCHAGENTS: Automated Benchmark Creation with Agent Interaction

• Authors: Natasha Butt, Varun Chandrasekaran, Neel Joshi, Besmira Nushi, Vidhisha Balachandran
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22584
• Pdf URL: https://arxiv.org/pdf/2410.22584
• Copy Paste: [[2410.22584]] BENCHAGENTS: Automated Benchmark Creation with Agent Interaction(https://arxiv.org/abs/2410.22584)
• Keywords: generative
• Abstract: Evaluations are limited by benchmark availability. As models evolve, there is a need to create benchmarks that can measure progress on new generative capabilities. However, creating new benchmarks through human annotations is slow and expensive, restricting comprehensive evaluations for any capability. We introduce BENCHAGENTS, a framework that methodically leverages large language models (LLMs) to automate benchmark creation for complex capabilities while inherently ensuring data and metric quality. BENCHAGENTS decomposes the benchmark creation process into planning, generation, data verification, and evaluation, each of which is executed by an LLM agent. These agents interact with each other and utilize human-in-the-loop feedback from benchmark developers to explicitly improve and flexibly control data diversity and quality. We use BENCHAGENTS to create benchmarks to evaluate capabilities related to planning and constraint satisfaction during text generation. We then use these benchmarks to study seven state-of-the-art models and extract new insights on common failure modes and model differences.

Title: PV-VTT: A Privacy-Centric Dataset for Mission-Specific Anomaly Detection and Natural Language Interpretation

• Authors: Ryozo Masukawa, Sanggeon Yun, Yoshiki Yamaguchi, Mohsen Imani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22623
• Pdf URL: https://arxiv.org/pdf/2410.22623
• Copy Paste: [[2410.22623]] PV-VTT: A Privacy-Centric Dataset for Mission-Specific Anomaly Detection and Natural Language Interpretation(https://arxiv.org/abs/2410.22623)
• Keywords: anomaly
• Abstract: Video crime detection is a significant application of computer vision and artificial intelligence. However, existing datasets primarily focus on detecting severe crimes by analyzing entire video clips, often neglecting the precursor activities (i.e., privacy violations) that could potentially prevent these crimes. To address this limitation, we present PV-VTT (Privacy Violation Video To Text), a unique multimodal dataset aimed at identifying privacy violations. PV-VTT provides detailed annotations for both video and text in scenarios. To ensure the privacy of individuals in the videos, we only provide video feature vectors, avoiding the release of any raw video data. This privacy-focused approach allows researchers to use the dataset while protecting participant confidentiality. Recognizing that privacy violations are often ambiguous and context-dependent, we propose a Graph Neural Network (GNN)-based video description model. Our model generates a GNN-based prompt with image for Large Language Model (LLM), which deliver cost-effective and high-quality video descriptions. By leveraging a single video frame along with relevant text, our method reduces the number of input tokens required, maintaining descriptive quality while optimizing LLM API-usage. Extensive experiments validate the effectiveness and interpretability of our approach in video description tasks and flexibility of our PV-VTT dataset.

Title: CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentation

• Authors: Ziyang Gong, Zhixiang Wei, Di Wang, Xianzheng Ma, Hongruixuan Chen, Yuru Jia, Yupeng Deng, Zhenming Ji, Xiangwei Zhu, Naoto Yokoya, Jing Zhang, Bo Du, Liangpei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22629
• Pdf URL: https://arxiv.org/pdf/2410.22629
• Copy Paste: [[2410.22629]] CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentation(https://arxiv.org/abs/2410.22629)
• Keywords: foundation model
• Abstract: The field of Remote Sensing Domain Generalization (RSDG) has emerged as a critical and valuable research frontier, focusing on developing models that generalize effectively across diverse scenarios. Despite the substantial domain gaps in RS images that are characterized by variabilities such as location, wavelength, and sensor type, research in this area remains underexplored: (1) Current cross-domain methods primarily focus on Domain Adaptation (DA), which adapts models to predefined domains rather than to unseen ones; (2) Few studies targeting the RSDG issue, especially for semantic segmentation tasks, where existing models are developed for specific unknown domains, struggling with issues of underfitting on other unknown scenarios; (3) Existing RS foundation models tend to prioritize in-domain performance over cross-domain generalization. To this end, we introduce the first vision foundation model for RSDG semantic segmentation, CrossEarth. CrossEarth demonstrates strong cross-domain generalization through a specially designed data-level Earth-Style Injection pipeline and a model-level Multi-Task Training pipeline. In addition, for the semantic segmentation task, we have curated an RSDG benchmark comprising 28 cross-domain settings across various regions, spectral bands, platforms, and climates, providing a comprehensive framework for testing the generalizability of future RSDG models. Extensive experiments on this benchmark demonstrate the superiority of CrossEarth over existing state-of-the-art methods.

Title: Consistency Diffusion Bridge Models

• Authors: Guande He, Kaiwen Zheng, Jianfei Chen, Fan Bao, Jun Zhu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22637
• Pdf URL: https://arxiv.org/pdf/2410.22637
• Copy Paste: [[2410.22637]] Consistency Diffusion Bridge Models(https://arxiv.org/abs/2410.22637)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DMs) have become the dominant paradigm of generative modeling in a variety of domains by learning stochastic processes from noise to data. Recently, diffusion denoising bridge models (DDBMs), a new formulation of generative modeling that builds stochastic processes between fixed data endpoints based on a reference diffusion process, have achieved empirical success across tasks with coupled data distribution, such as image-to-image translation. However, DDBM's sampling process typically requires hundreds of network evaluations to achieve decent performance, which may impede their practical deployment due to high computational demands. In this work, inspired by the recent advance of consistency models in DMs, we tackle this problem by learning the consistency function of the probability-flow ordinary differential equation (PF-ODE) of DDBMs, which directly predicts the solution at a starting step given any point on the ODE trajectory. Based on a dedicated general-form ODE solver, we propose two paradigms: consistency bridge distillation and consistency bridge training, which is flexible to apply on DDBMs with broad design choices. Experimental results show that our proposed method could sample $4\times$ to $50\times$ faster than the base DDBM and produce better visual quality given the same step in various tasks with pixel resolution ranging from $64 \times 64$ to $256 \times 256$, as well as supporting downstream tasks such as semantic interpolation in the data space.

Title: FlowDCN: Exploring DCN-like Architectures for Fast Image Generation with Arbitrary Resolution

• Authors: Shuai Wang, Zexian Li, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22655
• Pdf URL: https://arxiv.org/pdf/2410.22655
• Copy Paste: [[2410.22655]] FlowDCN: Exploring DCN-like Architectures for Fast Image Generation with Arbitrary Resolution(https://arxiv.org/abs/2410.22655)
• Keywords: diffusion, generative
• Abstract: Arbitrary-resolution image generation still remains a challenging task in AIGC, as it requires handling varying resolutions and aspect ratios while maintaining high visual quality. Existing transformer-based diffusion methods suffer from quadratic computation cost and limited resolution extrapolation capabilities, making them less effective for this task. In this paper, we propose FlowDCN, a purely convolution-based generative model with linear time and memory complexity, that can efficiently generate high-quality images at arbitrary resolutions. Equipped with a new design of learnable group-wise deformable convolution block, our FlowDCN yields higher flexibility and capability to handle different resolutions with a single model. FlowDCN achieves the state-of-the-art 4.30 sFID on $256\times256$ ImageNet Benchmark and comparable resolution extrapolation results, surpassing transformer-based counterparts in terms of convergence speed (only $\frac{1}{5}$ images), visual quality, parameters ($8\%$ reduction) and FLOPs ($20\%$ reduction). We believe FlowDCN offers a promising solution to scalable and flexible image synthesis.

Title: Incremental Learning of Retrievable Skills For Efficient Continual Task Adaptation

• Authors: Daehee Lee, Minjong Yoo, Woo Kyung Kim, Wonje Choi, Honguk Woo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22658
• Pdf URL: https://arxiv.org/pdf/2410.22658
• Copy Paste: [[2410.22658]] Incremental Learning of Retrievable Skills For Efficient Continual Task Adaptation(https://arxiv.org/abs/2410.22658)
• Keywords: foundation model
• Abstract: Continual Imitation Learning (CiL) involves extracting and accumulating task knowledge from demonstrations across multiple stages and tasks to achieve a multi-task policy. With recent advancements in foundation models, there has been a growing interest in adapter-based CiL approaches, where adapters are established parameter-efficiently for tasks newly demonstrated. While these approaches isolate parameters for specific tasks and tend to mitigate catastrophic forgetting, they limit knowledge sharing among different demonstrations. We introduce IsCiL, an adapter-based CiL framework that addresses this limitation of knowledge sharing by incrementally learning shareable skills from different demonstrations, thus enabling sample-efficient task adaptation using the skills particularly in non-stationary CiL environments. In IsCiL, demonstrations are mapped into the state embedding space, where proper skills can be retrieved upon input states through prototype-based memory. These retrievable skills are incrementally learned on their corresponding adapters. Our CiL experiments with complex tasks in Franka-Kitchen and Meta-World demonstrate robust performance of IsCiL in both task adaptation and sample-efficiency. We also show a simple extension of IsCiL for task unlearning scenarios.

Title: Exactly Minimax-Optimal Locally Differentially Private Sampling

• Authors: Hyun-Young Park, Shahab Asoodeh, Si-Hyeon Lee
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.22699
• Pdf URL: https://arxiv.org/pdf/2410.22699
• Copy Paste: [[2410.22699]] Exactly Minimax-Optimal Locally Differentially Private Sampling(https://arxiv.org/abs/2410.22699)
• Keywords: generative
• Abstract: The sampling problem under local differential privacy has recently been studied with potential applications to generative models, but a fundamental analysis of its privacy-utility trade-off (PUT) remains incomplete. In this work, we define the fundamental PUT of private sampling in the minimax sense, using the f-divergence between original and sampling distributions as the utility measure. We characterize the exact PUT for both finite and continuous data spaces under some mild conditions on the data distributions, and propose sampling mechanisms that are universally optimal for all f-divergences. Our numerical experiments demonstrate the superiority of our mechanisms over baselines, in terms of theoretical utilities for finite data space and of empirical utilities for continuous data space.

Title: MIXAD: Memory-Induced Explainable Time Series Anomaly Detection

• Authors: Minha Kim, Kishor Kumar Bhaumik, Amin Ahsan Ali, Simon S. Woo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22735
• Pdf URL: https://arxiv.org/pdf/2410.22735
• Copy Paste: [[2410.22735]] MIXAD: Memory-Induced Explainable Time Series Anomaly Detection(https://arxiv.org/abs/2410.22735)
• Keywords: anomaly
• Abstract: For modern industrial applications, accurately detecting and diagnosing anomalies in multivariate time series data is essential. Despite such need, most state-of-the-art methods often prioritize detection performance over model interpretability. Addressing this gap, we introduce MIXAD (Memory-Induced Explainable Time Series Anomaly Detection), a model designed for interpretable anomaly detection. MIXAD leverages a memory network alongside spatiotemporal processing units to understand the intricate dynamics and topological structures inherent in sensor relationships. We also introduce a novel anomaly scoring method that detects significant shifts in memory activation patterns during anomalies. Our approach not only ensures decent detection performance but also outperforms state-of-the-art baselines by 34.30% and 34.51% in interpretability metrics.

Title: FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Images

• Authors: Zheng Yu, Yaohua Wang, Siying Cui, Aixi Zhang, Wei-Long Zheng, Senzhang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22771
• Pdf URL: https://arxiv.org/pdf/2410.22771
• Copy Paste: [[2410.22771]] FuseAnyPart: Diffusion-Driven Facial Parts Swapping via Multiple Reference Images(https://arxiv.org/abs/2410.22771)
• Keywords: diffusion
• Abstract: Facial parts swapping aims to selectively transfer regions of interest from the source image onto the target image while maintaining the rest of the target image unchanged. Most studies on face swapping designed specifically for full-face swapping, are either unable or significantly limited when it comes to swapping individual facial parts, which hinders fine-grained and customized character designs. However, designing such an approach specifically for facial parts swapping is challenged by a reasonable multiple reference feature fusion, which needs to be both efficient and effective. To overcome this challenge, FuseAnyPart is proposed to facilitate the seamless "fuse-any-part" customization of the face. In FuseAnyPart, facial parts from different people are assembled into a complete face in latent space within the Mask-based Fusion Module. Subsequently, the consolidated feature is dispatched to the Addition-based Injection Module for fusion within the UNet of the diffusion model to create novel characters. Extensive experiments qualitatively and quantitatively validate the superiority and robustness of FuseAnyPart. Source codes are available at this https URL.

Title: Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models

• Authors: Arash Marioriyad, Parham Rezaei, Mahdieh Soleymani Baghshah, Mohammad Hossein Rohban
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22775
• Pdf URL: https://arxiv.org/pdf/2410.22775
• Copy Paste: [[2410.22775]] Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models(https://arxiv.org/abs/2410.22775)
• Keywords: diffusion, generative
• Abstract: Text-to-image (T2I) generative models, such as Stable Diffusion and DALL-E, have shown remarkable proficiency in producing high-quality, realistic, and natural images from textual descriptions. However, these models sometimes fail to accurately capture all the details specified in the input prompts, particularly concerning entities, attributes, and spatial relationships. This issue becomes more pronounced when the prompt contains novel or complex compositions, leading to what are known as compositional generation failure modes. Recently, a new open-source diffusion-based T2I model, FLUX, has been introduced, demonstrating strong performance in high-quality image generation. Additionally, autoregressive T2I models like LlamaGen have claimed competitive visual quality performance compared to diffusion-based models. In this study, we evaluate the compositional generation capabilities of these newly introduced models against established models using the T2I-CompBench benchmark. Our findings reveal that LlamaGen, as a vanilla autoregressive model, is not yet on par with state-of-the-art diffusion models for compositional generation tasks under the same criteria, such as model size and inference time. On the other hand, the open-source diffusion-based model FLUX exhibits compositional generation capabilities comparable to the state-of-the-art closed-source model DALL-E3.

Title: Universality of the $\pi^2/6$ Pathway in Avoiding Model Collapse

• Authors: Apratim Dey, David Donoho
• Subjects: cs.LG, cs.AI, cs.ET, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.22812
• Pdf URL: https://arxiv.org/pdf/2410.22812
• Copy Paste: [[2410.22812]] Universality of the $\pi^2/6$ Pathway in Avoiding Model Collapse(https://arxiv.org/abs/2410.22812)
• Keywords: generative
• Abstract: Researchers in empirical machine learning recently spotlighted their fears of so-called Model Collapse. They imagined a discard workflow, where an initial generative model is trained with real data, after which the real data are discarded, and subsequently, the model generates synthetic data on which a new model is trained. They came to the conclusion that models degenerate as model-fitting generations proceed. However, other researchers considered an augment workflow, where the original real data continue to be used in each generation of training, augmented by synthetic data from models fit in all earlier generations. Empirical results on canonical datasets and learning procedures confirmed the occurrence of model collapse under the discard workflow and avoidance of model collapse under the augment workflow. Under the augment workflow, theoretical evidence also confirmed avoidance in particular instances; specifically, Gerstgrasser et al. (2024) found that for classical Linear Regression, test risk at any later generation is bounded by a moderate multiple, viz. pi-squared-over-6 of the test risk of training with the original real data alone. Some commentators questioned the generality of theoretical conclusions based on the generative model assumed in Gerstgrasser et al. (2024): could similar conclusions be reached for other task/model pairings? In this work, we demonstrate the universality of the pi-squared-over-6 augment risk bound across a large family of canonical statistical models, offering key insights into exactly why collapse happens under the discard workflow and is avoided under the augment workflow. In the process, we provide a framework that is able to accommodate a large variety of workflows (beyond discard and augment), thereby enabling an experimenter to judge the comparative merits of multiple different workflows by simulating a simple Gaussian process.

Title: Danoliteracy of Generative, Large Language Models

• Authors: Søren Vejlgaard Holm, Lars Kai Hansen, Martin Carsten Nielsen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22839
• Pdf URL: https://arxiv.org/pdf/2410.22839
• Copy Paste: [[2410.22839]] Danoliteracy of Generative, Large Language Models(https://arxiv.org/abs/2410.22839)
• Keywords: generative
• Abstract: The language technology moonshot moment of Generative, Large Language Models (GLLMs) was not limited to English: These models brought a surge of technological applications, investments and hype to low-resource languages as well. However, the capabilities of these models in languages such as Danish were until recently difficult to verify beyond qualitative demonstrations due to a lack of applicable evaluation corpora. We present a GLLM benchmark to evaluate Danoliteracy, a measure of Danish language and cultural competency, across eight diverse scenarios such Danish citizenship tests and abstractive social media question answering. This limited-size benchmark is found to produce a robust ranking that correlates to human feedback at $\rho \sim 0.8$ with GPT-4 and Claude Opus models achieving the highest rankings. Analyzing these model results across scenarios, we find one strong underlying factor explaining $95\%$ of scenario performance variance for GLLMs in Danish, suggesting a $g$ factor of model consistency in language adaption.

Title: Conditioned quantum-assisted deep generative surrogate for particle-calorimeter interactions

• Authors: J. Quetzalcoatl Toledo-Marin, Sebastian Gonzalez, Hao Jia, Ian Lu, Deniz Sogutlu, Abhishek Abhishek, Colin Gay, Eric Paquet, Roger Melko, Geoffrey C. Fox, Maximilian Swiatlowski, Wojciech Fedorko
• Subjects: cs.LG, cs.AI, hep-ph, physics.comp-ph, physics.ins-det
• Abstract URL: https://arxiv.org/abs/2410.22870
• Pdf URL: https://arxiv.org/pdf/2410.22870
• Copy Paste: [[2410.22870]] Conditioned quantum-assisted deep generative surrogate for particle-calorimeter interactions(https://arxiv.org/abs/2410.22870)
• Keywords: generative
• Abstract: Particle collisions at accelerators such as the Large Hadron Collider, recorded and analyzed by experiments such as ATLAS and CMS, enable exquisite measurements of the Standard Model and searches for new phenomena. Simulations of collision events at these detectors have played a pivotal role in shaping the design of future experiments and analyzing ongoing ones. However, the quest for accuracy in Large Hadron Collider (LHC) collisions comes at an imposing computational cost, with projections estimating the need for millions of CPU-years annually during the High Luminosity LHC (HL-LHC) run \cite{collaboration2022atlas}. Simulating a single LHC event with \textsc{Geant4} currently devours around 1000 CPU seconds, with simulations of the calorimeter subdetectors in particular imposing substantial computational demands \cite{rousseau2023experimental}. To address this challenge, we propose a conditioned quantum-assisted deep generative model. Our model integrates a conditioned variational autoencoder (VAE) on the exterior with a conditioned Restricted Boltzmann Machine (RBM) in the latent space, providing enhanced expressiveness compared to conventional VAEs. The RBM nodes and connections are meticulously engineered to enable the use of qubits and couplers on D-Wave's Pegasus-structured \textit{Advantage} quantum annealer (QA) for sampling. We introduce a novel method for conditioning the quantum-assisted RBM using \textit{flux biases}. We further propose a novel adaptive mapping to estimate the effective inverse temperature in quantum annealers. The effectiveness of our framework is illustrated using Dataset 2 of the CaloChallenge \cite{calochallenge}.

Title: Eliciting Critical Reasoning in Retrieval-Augmented Language Models via Contrastive Explanations

• Authors: Leonardo Ranaldi, Marco Valentino, Andrè Freitas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22874
• Pdf URL: https://arxiv.org/pdf/2410.22874
• Copy Paste: [[2410.22874]] Eliciting Critical Reasoning in Retrieval-Augmented Language Models via Contrastive Explanations(https://arxiv.org/abs/2410.22874)
• Keywords: in-context
• Abstract: Retrieval-augmented generation (RAG) has emerged as a critical mechanism in contemporary NLP to support Large Language Models(LLMs) in systematically accessing richer factual context. However, the integration of RAG mechanisms brings its inherent challenges, as LLMs need to deal with potentially noisy contexts. Recent studies have shown that LLMs still struggle to critically analyse RAG-based in-context information, a limitation that may lead to incorrect inferences and hallucinations. In this paper, we investigate how to elicit critical reasoning in RAG via contrastive explanations. In particular, we propose Contrastive-RAG (C-RAG), a framework that (i) retrieves relevant documents given a query, (ii) selects and exemplifies relevant passages, and (iii) generates explanations that explicitly contrast the relevance of the passages to (iv) support the final answer. We show the impact of C-RAG building contrastive reasoning demonstrations from LLMs to instruct smaller models for retrieval-augmented tasks. Extensive experiments demonstrate that C-RAG improves state-of-the-art RAG models while (a) requiring significantly fewer prompts and demonstrations and (b) being robust to perturbations in the retrieved documents.

Title: Adaptive Paradigm Synergy: Can a Cross-Paradigm Objective Enhance Long-Tailed Learning?

• Authors: Haowen Xiao, Guanghui Liu, Xinyi Gao, Yang Li, Fengmao Lv, Jielei Chu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22883
• Pdf URL: https://arxiv.org/pdf/2410.22883
• Copy Paste: [[2410.22883]] Adaptive Paradigm Synergy: Can a Cross-Paradigm Objective Enhance Long-Tailed Learning?(https://arxiv.org/abs/2410.22883)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) has achieved impressive results across several computer vision tasks, even rivaling supervised methods. However, its performance degrades on real-world datasets with long-tailed distributions due to difficulties in capturing inherent class imbalances. Although supervised long-tailed learning offers significant insights, the absence of labels in SSL prevents direct transfer of these this http URL bridge this gap, we introduce Adaptive Paradigm Synergy (APS), a cross-paradigm objective that seeks to unify the strengths of both paradigms. Our approach reexamines contrastive learning from a spatial structure perspective, dynamically adjusting the uniformity of latent space structure through adaptive temperature tuning. Furthermore, we draw on a re-weighting strategy from supervised learning to compensate for the shortcomings of temperature adjustment in explicit quantity this http URL experiments on commonly used long-tailed datasets demonstrate that APS improves performance effectively and efficiently. Our findings reveal the potential for deeper integration between supervised and self-supervised learning, paving the way for robust models that handle real-world class imbalance.

Title: HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models

• Authors: Shengkai Zhang, Nianhong Jiao, Tian Li, Chaojie Yang, Chenhui Xue, Boya Niu, Jun Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22901
• Pdf URL: https://arxiv.org/pdf/2410.22901
• Copy Paste: [[2410.22901]] HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models(https://arxiv.org/abs/2410.22901)
• Keywords: diffusion, foundation model
• Abstract: We propose an effective method for inserting adapters into text-to-image foundation models, which enables the execution of complex downstream tasks while preserving the generalization ability of the base model. The core idea of this method is to optimize the attention mechanism related to 2D feature maps, which enhances the performance of the adapter. This approach was validated on the task of meme video generation and achieved significant results. We hope this work can provide insights for post-training tasks of large text-to-image models. Additionally, as this method demonstrates good compatibility with SD1.5 derivative models, it holds certain value for the open-source community. Therefore, we will release the related code (\url{this https URL}).

Title: CopRA: A Progressive LoRA Training Strategy

• Authors: Zhan Zhuang, Xiequn Wang, Yulong Zhang, Wei Li, Yu Zhang, Ying Wei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22911
• Pdf URL: https://arxiv.org/pdf/2410.22911
• Copy Paste: [[2410.22911]] CopRA: A Progressive LoRA Training Strategy(https://arxiv.org/abs/2410.22911)
• Keywords: foundation model
• Abstract: Low-Rank Adaptation (LoRA) is a parameter-efficient technique for rapidly fine-tuning foundation models. In standard LoRA training dynamics, models tend to quickly converge to a local optimum near the initialization. However, this local optimum may not be ideal for out-of-distribution data or tasks such as merging and pruning. In this work, we propose a novel progressive training strategy for LoRA with random layer dropping. This strategy also optimizes the Shapley value of LoRA parameters in each layer, treating each layer as a player in a cooperative game. We refer to this method as Cooperative LoRA (CopRA). Our experimental results demonstrate that parameters trained with CopRA exhibit linear mode connectivity, which enables efficient model merging. This also paves the way for federated learning and multi-task learning via LoRA merging. Additionally, by optimizing the Shapley value, CopRA shows superior performance in pruning tasks.

Title: Simulation-Free Training of Neural ODEs on Paired Data

• Authors: Semin Kim, Jaehoon Yoo, Jinwoo Kim, Yeonwoo Cha, Saehoon Kim, Seunghoon Hong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22918
• Pdf URL: https://arxiv.org/pdf/2410.22918
• Copy Paste: [[2410.22918]] Simulation-Free Training of Neural ODEs on Paired Data(https://arxiv.org/abs/2410.22918)
• Keywords: generative
• Abstract: In this work, we investigate a method for simulation-free training of Neural Ordinary Differential Equations (NODEs) for learning deterministic mappings between paired data. Despite the analogy of NODEs as continuous-depth residual networks, their application in typical supervised learning tasks has not been popular, mainly due to the large number of function evaluations required by ODE solvers and numerical instability in gradient estimation. To alleviate this problem, we employ the flow matching framework for simulation-free training of NODEs, which directly regresses the parameterized dynamics function to a predefined target velocity field. Contrary to generative tasks, however, we show that applying flow matching directly between paired data can often lead to an ill-defined flow that breaks the coupling of the data pairs (e.g., due to crossing trajectories). We propose a simple extension that applies flow matching in the embedding space of data pairs, where the embeddings are learned jointly with the dynamic function to ensure the validity of the flow which is also easier to learn. We demonstrate the effectiveness of our method on both regression and classification tasks, where our method outperforms existing NODEs with a significantly lower number of function evaluations. The code is available at this https URL.

Title: Multi-Agent Large Language Models for Conversational Task-Solving

• Authors: Jonas Becker
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22932
• Pdf URL: https://arxiv.org/pdf/2410.22932
• Copy Paste: [[2410.22932]] Multi-Agent Large Language Models for Conversational Task-Solving(https://arxiv.org/abs/2410.22932)
• Keywords: generative
• Abstract: In an era where single large language models have dominated the landscape of artificial intelligence for years, multi-agent systems arise as new protagonists in conversational task-solving. While previous studies have showcased their potential in reasoning tasks and creative endeavors, an analysis of their limitations concerning the conversational paradigms and the impact of individual agents is missing. It remains unascertained how multi-agent discussions perform across tasks of varying complexity and how the structure of these conversations influences the process. To fill that gap, this work systematically evaluates multi-agent systems across various discussion paradigms, assessing their strengths and weaknesses in both generative tasks and question-answering tasks. Alongside the experiments, I propose a taxonomy of 20 multi-agent research studies from 2022 to 2024, followed by the introduction of a framework for deploying multi-agent LLMs in conversational task-solving. I demonstrate that while multi-agent systems excel in complex reasoning tasks, outperforming a single model by leveraging expert personas, they fail on basic tasks. Concretely, I identify three challenges that arise: 1) While longer discussions enhance reasoning, agents fail to maintain conformity to strict task requirements, which leads to problem drift, making shorter conversations more effective for basic tasks. 2) Prolonged discussions risk alignment collapse, raising new safety concerns for these systems. 3) I showcase discussion monopolization through long generations, posing the problem of fairness in decision-making for tasks like summarization. This work uncovers both the potential and challenges that arise with multi-agent interaction and varying conversational paradigms, providing insights into how future research could improve the efficiency, performance, and safety of multi-agent LLMs.

Title: Dynamic Threshold-based Two-layer Online Unsupervised Anomaly Detector

• Authors: Yachao Yuan, Yu Huang, Yali Yuan, Jin Wang
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.22967
• Pdf URL: https://arxiv.org/pdf/2410.22967
• Copy Paste: [[2410.22967]] Dynamic Threshold-based Two-layer Online Unsupervised Anomaly Detector(https://arxiv.org/abs/2410.22967)
• Keywords: anomaly
• Abstract: The proliferation of the Internet of Things (IoT) has heightened the vulnerability to cyber threats, making it imperative to develop Anomaly Detection Systems (ADSs) capable of adapting to emerging or novel attacks. Prior research has predominantly concentrated on offline unsupervised learning techniques to protect ADSs, which are impractical for real-world applications. Furthermore, these studies often rely heavily on the assumption of known legitimate behaviors and fall short of meeting the interpretability requirements in security contexts, thereby hindering their practical adoption. In response, this paper introduces Adaptive NAD, a comprehensive framework aimed at enhancing and interpreting online unsupervised anomaly detection within security domains. We propose an interpretable two-layer anomaly detection approach that generates dependable, high-confidence pseudo-labels. Subsequently, we incorporate an online learning mechanism that updates Adaptive NAD using an innovative threshold adjustment method to accommodate new threats. Experimental findings reveal that Adaptive NAD surpasses state-of-the-art solutions by achieving improvements of over 5.4% and 23.0% in SPAUC on the CIC-Darknet2020 and CIC-DoHBrw-2020 datasets, respectively. The code for Adaptive NAD is publicly available at this https URL.

Title: Private Synthetic Text Generation with Diffusion Models

• Authors: Sebastian Ochs, Ivan Habernal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22971
• Pdf URL: https://arxiv.org/pdf/2410.22971
• Copy Paste: [[2410.22971]] Private Synthetic Text Generation with Diffusion Models(https://arxiv.org/abs/2410.22971)
• Keywords: diffusion
• Abstract: How capable are diffusion models of generating synthetics texts? Recent research shows their strengths, with performance reaching that of auto-regressive LLMs. But are they also good in generating synthetic data if the training was under differential privacy? Here the evidence is missing, yet the promises from private image generation look strong. In this paper we address this open question by extensive experiments. At the same time, we critically assess (and reimplement) previous works on synthetic private text generation with LLMs and reveal some unmet assumptions that might have led to violating the differential privacy guarantees. Our results partly contradict previous non-private findings and show that fully open-source LLMs outperform diffusion models in the privacy regime. Our complete source codes, datasets, and experimental setup is publicly available to foster future research.

Title: LumiSculpt: A Consistency Lighting Control Network for Video Generation

• Authors: Yuxin Zhang, Dandan Zheng, Biao Gong, Jingdong Chen, Ming Yang, Weiming Dong, Changsheng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22979
• Pdf URL: https://arxiv.org/pdf/2410.22979
• Copy Paste: [[2410.22979]] LumiSculpt: A Consistency Lighting Control Network for Video Generation(https://arxiv.org/abs/2410.22979)
• Keywords: diffusion
• Abstract: Lighting plays a pivotal role in ensuring the naturalness of video generation, significantly influencing the aesthetic quality of the generated content. However, due to the deep coupling between lighting and the temporal features of videos, it remains challenging to disentangle and model independent and coherent lighting attributes, limiting the ability to control lighting in video generation. In this paper, inspired by the established controllable T2I models, we propose LumiSculpt, which, for the first time, enables precise and consistent lighting control in T2V generation this http URL equips the video generation with strong interactive capabilities, allowing the input of custom lighting reference image sequences. Furthermore, the core learnable plug-and-play module of LumiSculpt facilitates remarkable control over lighting intensity, position, and trajectory in latent video diffusion models based on the advanced DiT this http URL, to effectively train LumiSculpt and address the issue of insufficient lighting data, we construct LumiHuman, a new lightweight and flexible dataset for portrait lighting of images and videos. Experimental results demonstrate that LumiSculpt achieves precise and high-quality lighting control in video generation.

Title: Toward Understanding In-context vs. In-weight Learning

• Authors: Bryan Chan, Xinyi Chen, András György, Dale Schuurmans
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.23042
• Pdf URL: https://arxiv.org/pdf/2410.23042
• Copy Paste: [[2410.23042]] Toward Understanding In-context vs. In-weight Learning(https://arxiv.org/abs/2410.23042)
• Keywords: in-context
• Abstract: It has recently been demonstrated empirically that in-context learning emerges in transformers when certain distributional properties are present in the training data, but this ability can also diminish upon further training. We provide a new theoretical understanding of these phenomena by identifying simplified distributional properties that give rise to the emergence and eventual disappearance of in-context learning. We do so by first analyzing a simplified model that uses a gating mechanism to choose between an in-weight and an in-context predictor. Through a combination of a generalization error and regret analysis we identify conditions where in-context and in-weight learning emerge. These theoretical findings are then corroborated experimentally by comparing the behaviour of a full transformer on the simplified distributions to that of the stylized model, demonstrating aligned results. We then extend the study to a full large language model, showing how fine-tuning on various collections of natural language prompts can elicit similar in-context and in-weight learning behaviour.

Title: Controlling Language and Diffusion Models by Transporting Activations

• Authors: Pau Rodriguez, Arno Blaas, Michal Klein, Luca Zappella, Nicholas Apostoloff, Marco Cuturi, Xavier Suau
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.23054
• Pdf URL: https://arxiv.org/pdf/2410.23054
• Copy Paste: [[2410.23054]] Controlling Language and Diffusion Models by Transporting Activations(https://arxiv.org/abs/2410.23054)
• Keywords: diffusion, generative
• Abstract: The increasing capabilities of large generative models and their ever more widespread deployment have raised concerns about their reliability, safety, and potential misuse. To address these issues, recent works have proposed to control model generation by steering model activations in order to effectively induce or prevent the emergence of concepts or behaviors in the generated output. In this paper we introduce Activation Transport (AcT), a general framework to steer activations guided by optimal transport theory that generalizes many previous activation-steering works. AcT is modality-agnostic and provides fine-grained control over the model behavior with negligible computational overhead, while minimally impacting model abilities. We experimentally show the effectiveness and versatility of our approach by addressing key challenges in large language models (LLMs) and text-to-image diffusion models (T2Is). For LLMs, we show that AcT can effectively mitigate toxicity, induce arbitrary concepts, and increase their truthfulness. In T2Is, we show how AcT enables fine-grained style control and concept negation.

Title: CNN Explainability with Multivector Tucker Saliency Maps for Self-Supervised Models

• Authors: Aymene Mohammed Bouayed, Samuel Deslauriers-Gauthier, Adrian Iaccovelli, David Naccache
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.23072
• Pdf URL: https://arxiv.org/pdf/2410.23072
• Copy Paste: [[2410.23072]] CNN Explainability with Multivector Tucker Saliency Maps for Self-Supervised Models(https://arxiv.org/abs/2410.23072)
• Keywords: self-supervised
• Abstract: Interpreting the decisions of Convolutional Neural Networks (CNNs) is essential for understanding their behavior, yet explainability remains a significant challenge, particularly for self-supervised models. Most existing methods for generating saliency maps rely on ground truth labels, restricting their use to supervised tasks. EigenCAM is the only notable label-independent alternative, leveraging Singular Value Decomposition to generate saliency maps applicable across CNN models, but it does not fully exploit the tensorial structure of feature maps. In this work, we introduce the Tucker Saliency Map (TSM) method, which applies Tucker tensor decomposition to better capture the inherent structure of feature maps, producing more accurate singular vectors and values. These are used to generate high-fidelity saliency maps, effectively highlighting objects of interest in the input. We further extend EigenCAM and TSM into multivector variants -Multivec-EigenCAM and Multivector Tucker Saliency Maps (MTSM)- which utilize all singular vectors and values, further improving saliency map quality. Quantitative evaluations on supervised classification models demonstrate that TSM, Multivec-EigenCAM, and MTSM achieve competitive performance with label-dependent methods. Moreover, TSM enhances explainability by approximately 50% over EigenCAM for both supervised and self-supervised models. Multivec-EigenCAM and MTSM further advance state-of-the-art explainability performance on self-supervised models, with MTSM achieving the best results.

Title: S3PT: Scene Semantics and Structure Guided Clustering to Boost Self-Supervised Pre-Training for Autonomous Driving

• Authors: Maciej K. Wozniak, Hariprasath Govindarajan, Marvin Klingner, Camille Maurice, Ravi Kiran, Senthil Yogamani
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.23085
• Pdf URL: https://arxiv.org/pdf/2410.23085
• Copy Paste: [[2410.23085]] S3PT: Scene Semantics and Structure Guided Clustering to Boost Self-Supervised Pre-Training for Autonomous Driving(https://arxiv.org/abs/2410.23085)
• Keywords: self-supervised
• Abstract: Recent self-supervised clustering-based pre-training techniques like DINO and Cribo have shown impressive results for downstream detection and segmentation tasks. However, real-world applications such as autonomous driving face challenges with imbalanced object class and size distributions and complex scene geometries. In this paper, we propose S3PT a novel scene semantics and structure guided clustering to provide more scene-consistent objectives for self-supervised training. Specifically, our contributions are threefold: First, we incorporate semantic distribution consistent clustering to encourage better representation of rare classes such as motorcycles or animals. Second, we introduce object diversity consistent spatial clustering, to handle imbalanced and diverse object sizes, ranging from large background areas to small objects such as pedestrians and traffic signs. Third, we propose a depth-guided spatial clustering to regularize learning based on geometric information of the scene, thus further refining region separation on the feature level. Our learned representations significantly improve performance in downstream semantic segmentation and 3D object detection tasks on the nuScenes, nuImages, and Cityscapes datasets and show promising domain translation properties.

Title: CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense

• Authors: Mingkun Zhang, Keping Bi, Wei Chen, Quanrun Chen, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.23091
• Pdf URL: https://arxiv.org/pdf/2410.23091
• Copy Paste: [[2410.23091]] CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense(https://arxiv.org/abs/2410.23091)
• Keywords: diffusion
• Abstract: Despite ongoing efforts to defend neural classifiers from adversarial attacks, they remain vulnerable, especially to unseen attacks. In contrast, humans are difficult to be cheated by subtle manipulations, since we make judgments only based on essential factors. Inspired by this observation, we attempt to model label generation with essential label-causative factors and incorporate label-non-causative factors to assist data generation. For an adversarial example, we aim to discriminate the perturbations as non-causative factors and make predictions only based on the label-causative factors. Concretely, we propose a casual diffusion model (CausalDiff) that adapts diffusion models for conditional data generation and disentangles the two types of casual factors by learning towards a novel casual information bottleneck objective. Empirically, CausalDiff has significantly outperformed state-of-the-art defense methods on various unseen attacks, achieving an average robustness of 86.39% (+4.01%) on CIFAR-10, 56.25% (+3.13%) on CIFAR-100, and 82.62% (+4.93%) on GTSRB (German Traffic Sign Recognition Benchmark).

Title: Comparative Analysis of Demonstration Selection Algorithms for LLM In-Context Learning

• Authors: Dong Shu, Mengnan Du
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.23099
• Pdf URL: https://arxiv.org/pdf/2410.23099
• Copy Paste: [[2410.23099]] Comparative Analysis of Demonstration Selection Algorithms for LLM In-Context Learning(https://arxiv.org/abs/2410.23099)
• Keywords: in-context
• Abstract: In-context learning can help Large Language Models (LLMs) to adapt new tasks without additional training. However, this performance heavily depends on the quality of the demonstrations, driving research into effective demonstration selection algorithms to optimize this process. These algorithms assist users in selecting the best $k$ input-label pairs (demonstration examples) based on a given test input, enabling LLMs to in-context learn the relationship between the provided examples and the test inputs. Despite all the proposed demonstration selection algorithms, their efficiency and effectiveness remain unclear. This lack of clarity make it difficult to apply these algorithms in real-world scenarios and poses challenges for future research aimed at developing improved methods. This paper revisits six proposed algorithms, evaluating them on five datasets from both efficiency and effectiveness perspectives. Our experiments reveal significant variations in algorithm performance across different tasks, with some methods struggling to outperform random selection in certain scenarios. We also find that increasing the number of demonstrations does not always lead to better performance, and that there are often trade-offs between accuracy and computational efficiency. Our code is available at this https URL.

Title: Controllable Game Level Generation: Assessing the Effect of Negative Examples in GAN Models

• Authors: Mahsa Bazzaz, Seth Cooper
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.23108
• Pdf URL: https://arxiv.org/pdf/2410.23108
• Copy Paste: [[2410.23108]] Controllable Game Level Generation: Assessing the Effect of Negative Examples in GAN Models(https://arxiv.org/abs/2410.23108)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) are unsupervised models designed to learn and replicate a target distribution. The vanilla versions of these models can be extended to more controllable models. Conditional Generative Adversarial Networks (CGANs) extend vanilla GANs by conditioning both the generator and discriminator on some additional information (labels). Controllable models based on complementary learning, such as Rumi-GAN, have been introduced. Rumi-GANs leverage negative examples to enhance the generator's ability to learn positive examples. We evaluate the performance of two controllable GAN variants, CGAN and Rumi-GAN, in generating game levels targeting specific constraints of interest: playability and controllability. This evaluation is conducted under two scenarios: with and without the inclusion of negative examples. The goal is to determine whether incorporating negative examples helps the GAN models avoid generating undesirable outputs. Our findings highlight the strengths and weaknesses of each method in enforcing the generation of specific conditions when generating outputs based on given positive and negative examples.

Title: Revisiting MAE pre-training for 3D medical image segmentation

• Authors: Tassilo Wald, Constantin Ulrich, Stanislav Lukyanenko, Andrei Goncharov, Alberto Paderno, Leander Maerkisch, Paul F. Jäger, Klaus Maier-Hein
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.23132
• Pdf URL: https://arxiv.org/pdf/2410.23132
• Copy Paste: [[2410.23132]] Revisiting MAE pre-training for 3D medical image segmentation(https://arxiv.org/abs/2410.23132)
• Keywords: self-supervised
• Abstract: Self-Supervised Learning (SSL) presents an exciting opportunity to unlock the potential of vast, untapped clinical datasets, for various downstream applications that suffer from the scarcity of labeled data. While SSL has revolutionized fields like natural language processing and computer vision, their adoption in 3D medical image computing has been limited by three key pitfalls: Small pre-training dataset sizes, architectures inadequate for 3D medical image analysis, and insufficient evaluation practices. We address these issues by i) leveraging a large-scale dataset of 44k 3D brain MRI volumes and ii) using a Residual Encoder U-Net architecture within the state-of-the-art nnU-Net framework. iii) A robust development framework, incorporating 5 development and 8 testing brain MRI segmentation datasets, allowed performance-driven design decisions to optimize the simple concept of Masked Auto Encoders (MAEs) for 3D CNNs. The resulting model not only surpasses previous SSL methods but also outperforms the strong nnU-Net baseline by an average of approximately 3 Dice points. Furthermore, our model demonstrates exceptional stability, achieving the highest average rank of 2 out of 7 methods, compared to the second-best method's mean rank of 3.

Title: Directional anomaly detection

• Authors: Oliver Urs Lenz, Matthijs van Leeuwen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.23158
• Pdf URL: https://arxiv.org/pdf/2410.23158
• Copy Paste: [[2410.23158]] Directional anomaly detection(https://arxiv.org/abs/2410.23158)
• Keywords: anomaly
• Abstract: Semi-supervised anomaly detection is based on the principle that potential anomalies are those records that look different from normal training data. However, in some cases we are specifically interested in anomalies that correspond to high attribute values (or low, but not both). We present two asymmetrical distance measures that take this directionality into account: ramp distance and signed distance. Through experiments on synthetic and real-life datasets we show that ramp distance performs as well or better than the absolute distance traditionally used in anomaly detection. While signed distance also performs well on synthetic data, it performs substantially poorer on real-life datasets. We argue that this reflects the fact that in practice, good scores on some attributes should not be allowed to compensate for bad scores on others.

Title: FlexTSF: A Universal Forecasting Model for Time Series with Variable Regularities

• Authors: Jingge Xiao, Yile Chen, Gao Cong, Wolfgang Nejdl, Simon Gottschalk
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.23160
• Pdf URL: https://arxiv.org/pdf/2410.23160
• Copy Paste: [[2410.23160]] FlexTSF: A Universal Forecasting Model for Time Series with Variable Regularities(https://arxiv.org/abs/2410.23160)
• Keywords: self-supervised, foundation model
• Abstract: Developing a foundation model for time series forecasting across diverse domains has attracted significant attention in recent years. Existing works typically assume regularly sampled, well-structured data, limiting their applicability to more generalized scenarios where time series often contain missing values, unequal sequence lengths, and irregular time intervals between measurements. To cover diverse domains and handle variable regularities, we propose FlexTSF, a universal time series forecasting model that possesses better generalization and natively support both regular and irregular time series. FlexTSF produces forecasts in an autoregressive manner and incorporates three novel designs: VT-Norm, a normalization strategy to ablate data domain barriers, IVP Patcher, a patching module to learn representations from flexibly structured time series, and LED attention, an attention mechanism to seamlessly integrate these two and propagate forecasts with awareness of domain and time information. Experiments on 12 datasets show that FlexTSF outperforms state-of-the-art forecasting models respectively designed for regular and irregular time series. Furthermore, after self-supervised pre-training, FlexTSF shows exceptional performance in both zero-shot and few-show settings for time series forecasting.

Title: TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters

• Authors: Haiyang Wang, Yue Fan, Muhammad Ferjad Naeem, Yongqin Xian, Jan Eric Lenssen, Liwei Wang, Federico Tombari, Bernt Schiele
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.23168
• Pdf URL: https://arxiv.org/pdf/2410.23168
• Copy Paste: [[2410.23168]] TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters(https://arxiv.org/abs/2410.23168)
• Keywords: foundation model
• Abstract: Transformers have become the predominant architecture in foundation models due to their excellent performance across various domains. However, the substantial cost of scaling these models remains a significant concern. This problem arises primarily from their dependence on a fixed number of parameters within linear projections. When architectural modifications (e.g., channel dimensions) are introduced, the entire model typically requires retraining from scratch. As model sizes continue growing, this strategy results in increasingly high computational costs and becomes unsustainable. To overcome this problem, we introduce TokenFormer, a natively scalable architecture that leverages the attention mechanism not only for computations among input tokens but also for interactions between tokens and model parameters, thereby enhancing architectural flexibility. By treating model parameters as tokens, we replace all the linear projections in Transformers with our token-parameter attention layer, where input tokens act as queries and model parameters as keys and values. This reformulation allows for progressive and efficient scaling without necessitating retraining from scratch. Our model scales from 124M to 1.4B parameters by incrementally adding new key-value parameter pairs, achieving performance comparable to Transformers trained from scratch while greatly reducing training costs. Code and models are available at \url{this https URL}.

Title: HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms

• Authors: Kiran Kokilepersaud, Seulgi Kim, Mohit Prabhushankar, Ghassan AlRegib
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.23200
• Pdf URL: https://arxiv.org/pdf/2410.23200
• Copy Paste: [[2410.23200]] HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms(https://arxiv.org/abs/2410.23200)
• Keywords: self-supervised
• Abstract: In this paper, we propose an algorithm that can be used on top of a wide variety of self-supervised (SSL) approaches to take advantage of hierarchical structures that emerge during training. SSL approaches typically work through some invariance term to ensure consistency between similar samples and a regularization term to prevent global dimensional collapse. Dimensional collapse refers to data representations spanning a lower-dimensional subspace. Recent work has demonstrated that the representation space of these algorithms gradually reflects a semantic hierarchical structure as training progresses. Data samples of the same hierarchical grouping tend to exhibit greater dimensional collapse locally compared to the dataset as a whole due to sharing features in common with each other. Ideally, SSL algorithms would take advantage of this hierarchical emergence to have an additional regularization term to account for this local dimensional collapse effect. However, the construction of existing SSL algorithms does not account for this property. To address this, we propose an adaptive algorithm that performs a weighted decomposition of the denominator of the InfoNCE loss into two terms: local hierarchical and global collapse regularization respectively. This decomposition is based on an adaptive threshold that gradually lowers to reflect the emerging hierarchical structure of the representation space throughout training. It is based on an analysis of the cosine similarity distribution of samples in a batch. We demonstrate that this hierarchical emergence exploitation (HEX) approach can be integrated across a wide variety of SSL algorithms. Empirically, we show performance improvements of up to 5.6% relative improvement over baseline SSL approaches on classification accuracy on Imagenet with 100 epochs of training.

Title: Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks

• Authors: Michael Matthews, Michael Beukman, Chris Lu, Jakob Foerster
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.23208
• Pdf URL: https://arxiv.org/pdf/2410.23208
• Copy Paste: [[2410.23208]] Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks(https://arxiv.org/abs/2410.23208)
• Keywords: self-supervised
• Abstract: While large models trained with self-supervised learning on offline datasets have shown remarkable capabilities in text and image domains, achieving the same generalisation for agents that act in sequential decision problems remains an open challenge. In this work, we take a step towards this goal by procedurally generating tens of millions of 2D physics-based tasks and using these to train a general reinforcement learning (RL) agent for physical control. To this end, we introduce Kinetix: an open-ended space of physics-based RL environments that can represent tasks ranging from robotic locomotion and grasping to video games and classic RL environments, all within a unified framework. Kinetix makes use of our novel hardware-accelerated physics engine Jax2D that allows us to cheaply simulate billions of environment steps during training. Our trained agent exhibits strong physical reasoning capabilities, being able to zero-shot solve unseen human-designed environments. Furthermore, fine-tuning this general agent on tasks of interest shows significantly stronger performance than training an RL agent *tabula rasa*. This includes solving some environments that standard RL training completely fails at. We believe this demonstrates the feasibility of large scale, mixed-quality pre-training for online RL and we hope that Kinetix will serve as a useful framework to investigate this further.

Title: Partial Channel Dependence with Channel Masks for Time Series Foundation Models

• Authors: Seunghan Lee, Taeyoung Park, Kibok Lee
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.23222
• Pdf URL: https://arxiv.org/pdf/2410.23222
• Copy Paste: [[2410.23222]] Partial Channel Dependence with Channel Masks for Time Series Foundation Models(https://arxiv.org/abs/2410.23222)
• Keywords: foundation model, anomaly
• Abstract: Recent advancements in foundation models have been successfully extended to the time series (TS) domain, facilitated by the emergence of large-scale TS datasets. However, previous efforts have primarily focused on designing model architectures to address explicit heterogeneity among datasets such as various numbers of channels, while often overlooking implicit heterogeneity such as varying dependencies between channels. In this work, we introduce the concept of partial channel dependence (PCD), which enables a more sophisticated adjustment of channel dependencies based on dataset-specific information. To achieve PCD, we propose a channel mask that captures the relationships between channels within a dataset using two key components: 1) a correlation matrix that encodes relative dependencies between channels, and 2) domain parameters that learn the absolute dependencies specific to each dataset, refining the correlation matrix. We validate the effectiveness of PCD across four tasks in TS including forecasting, classification, imputation, and anomaly detection, under diverse settings, including few-shot and zero-shot scenarios with both TS foundation models and single-task models. Code is available at this https URL.

Title: TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models

• Authors: Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, Arman Cohan
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.23266
• Pdf URL: https://arxiv.org/pdf/2410.23266
• Copy Paste: [[2410.23266]] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models(https://arxiv.org/abs/2410.23266)
• Keywords: foundation model
• Abstract: Existing benchmarks often highlight the remarkable performance achieved by state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal context for video understanding. However, how well do the models truly perform visual temporal reasoning? Our study of existing benchmarks shows that this capability of MFMs is likely overestimated as many questions can be solved by using a single, few, or out-of-order frames. To systematically examine current visual temporal reasoning tasks, we propose three principles with corresponding metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame Information Disparity. Following these principles, we introduce TOMATO, Temporal Reasoning Multimodal Evaluation, a novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding. TOMATO comprises 1,484 carefully curated, human-annotated questions spanning six tasks (i.e., action count, direction, rotation, shape & trend, velocity & frequency, and visual cues), applied to 1,417 videos, including 805 self-recorded and -generated videos, that encompass human-centric, real-world, and simulated scenarios. Our comprehensive evaluation reveals a human-model performance gap of 57.3% with the best-performing model. Moreover, our in-depth analysis uncovers more fundamental limitations beyond this gap in current MFMs. While they can accurately recognize events in isolated frames, they fail to interpret these frames as a continuous sequence. We believe TOMATO will serve as a crucial testbed for evaluating the next-generation MFMs and as a call to the community to develop AI systems capable of comprehending human world dynamics through the video modality.

Title: Multi-student Diffusion Distillation for Better One-step Generators

• Authors: Yanke Song, Jonathan Lorraine, Weili Nie, Karsten Kreis, James Lucas
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.23274
• Pdf URL: https://arxiv.org/pdf/2410.23274
• Copy Paste: [[2410.23274]] Multi-student Diffusion Distillation for Better One-step Generators(https://arxiv.org/abs/2410.23274)
• Keywords: diffusion
• Abstract: Diffusion models achieve high-quality sample generation at the cost of a lengthy multistep inference procedure. To overcome this, diffusion distillation techniques produce student generators capable of matching or surpassing the teacher in a single step. However, the student model's inference speed is limited by the size of the teacher architecture, preventing real-time generation for computationally heavy applications. In this work, we introduce Multi-Student Distillation (MSD), a framework to distill a conditional teacher diffusion model into multiple single-step generators. Each student generator is responsible for a subset of the conditioning data, thereby obtaining higher generation quality for the same capacity. MSD trains multiple distilled students, allowing smaller sizes and, therefore, faster inference. Also, MSD offers a lightweight quality boost over single-student distillation with the same architecture. We demonstrate MSD is effective by training multiple same-sized or smaller students on single-step distillation using distribution matching and adversarial distillation techniques. With smaller students, MSD gets competitive results with faster inference for single-step generation. Using 4 same-sized students, MSD sets a new state-of-the-art for one-step image generation: FID 1.20 on ImageNet-64x64 and 8.20 on zero-shot COCO2014.

Title: SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation

• Authors: Yining Hong, Beide Liu, Maxine Wu, Yuanhao Zhai, Kai-Wei Chang, Lingjie Li, Kevin Lin, Chung-Ching Lin, Jianfeng Wang, Zhengyuan Yang, Yingnian Wu, Lijuan Wang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.23277
• Pdf URL: https://arxiv.org/pdf/2410.23277
• Copy Paste: [[2410.23277]] SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation(https://arxiv.org/abs/2410.23277)
• Keywords: diffusion
• Abstract: Human beings are endowed with a complementary learning system, which bridges the slow learning of general world dynamics with fast storage of episodic memory from a new experience. Previous video generation models, however, primarily focus on slow learning by pre-training on vast amounts of data, overlooking the fast learning phase crucial for episodic memory storage. This oversight leads to inconsistencies across temporally distant frames when generating longer videos, as these frames fall beyond the model's context window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation. Our approach incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference-time fast learning strategy based on a temporal LoRA module. Specifically, the fast learning process updates its temporal LoRA parameters based on local inputs and outputs, thereby efficiently storing episodic memory in its parameters. We further propose a slow-fast learning loop algorithm that seamlessly integrates the inner fast learning loop into the outer slow learning loop, enabling the recall of prior multi-episode experiences for context-aware skill learning. To facilitate the slow learning of an approximate world model, we collect a large-scale dataset of 200k videos with language action annotations, covering a wide range of scenarios. Extensive experiments show that SlowFast-VGen outperforms baselines across various metrics for action-driven video generation, achieving an FVD score of 514 compared to 782, and maintaining consistency in longer videos, with an average of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm significantly enhances performances on long-horizon planning tasks as well. Project Website: this https URL

Title: RelationBooth: Towards Relation-Aware Customized Object Generation

• Authors: Qingyu Shi, Lu Qi, Jianzong Wu, Jinbin Bai, Jingbo Wang, Yunhai Tong, Xiangtai Li, Ming-Husang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.23280
• Pdf URL: https://arxiv.org/pdf/2410.23280
• Copy Paste: [[2410.23280]] RelationBooth: Towards Relation-Aware Customized Object Generation(https://arxiv.org/abs/2410.23280)
• Keywords: diffusion
• Abstract: Customized image generation is crucial for delivering personalized content based on user-provided image prompts, aligning large-scale text-to-image diffusion models with individual needs. However, existing models often overlook the relationships between customized objects in generated images. Instead, this work addresses that gap by focusing on relation-aware customized image generation, which aims to preserve the identities from image prompts while maintaining the predicate relations described in text prompts. Specifically, we introduce RelationBooth, a framework that disentangles identity and relation learning through a well-curated dataset. Our training data consists of relation-specific images, independent object images containing identity information, and text prompts to guide relation generation. Then, we propose two key modules to tackle the two main challenges: generating accurate and natural relations, especially when significant pose adjustments are required, and avoiding object confusion in cases of overlap. First, we introduce a keypoint matching loss that effectively guides the model in adjusting object poses closely tied to their relationships. Second, we incorporate local features from the image prompts to better distinguish between objects, preventing confusion in overlapping cases. Extensive results on three benchmarks demonstrate the superiority of RelationBooth in generating precise relations while preserving object identities across a diverse set of objects and relations. The source code and trained models will be made available to the public.

Title: Provable acceleration for diffusion models under minimal assumptions

• Authors: Gen Li, Changxiao Cai
• Subjects: cs.LG, cs.AI, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.23285
• Pdf URL: https://arxiv.org/pdf/2410.23285
• Copy Paste: [[2410.23285]] Provable acceleration for diffusion models under minimal assumptions(https://arxiv.org/abs/2410.23285)
• Keywords: diffusion
• Abstract: While score-based diffusion models have achieved exceptional sampling quality, their sampling speeds are often limited by the high computational burden of score function evaluations. Despite the recent remarkable empirical advances in speeding up the score-based samplers, theoretical understanding of acceleration techniques remains largely limited. To bridge this gap, we propose a novel training-free acceleration scheme for stochastic samplers. Under minimal assumptions -- namely, $L^2$-accurate score estimates and a finite second-moment condition on the target distribution -- our accelerated sampler provably achieves $\varepsilon$-accuracy in total variation within $\widetilde{O}(d^{5/4}/\sqrt{\varepsilon})$ iterations, thereby significantly improving upon the $\widetilde{O}(d/\varepsilon)$ iteration complexity of standard score-based samplers. Notably, our convergence theory does not rely on restrictive assumptions on the target distribution or higher-order score estimation guarantees.

Title: ReferEverything: Towards Segmenting Everything We Can Speak of in Videos

• Authors: Anurag Bagchi, Zhipeng Bao, Yu-Xiong Wang, Pavel Tokmakov, Martial Hebert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.23287
• Pdf URL: https://arxiv.org/pdf/2410.23287
• Copy Paste: [[2410.23287]] ReferEverything: Towards Segmenting Everything We Can Speak of in Videos(https://arxiv.org/abs/2410.23287)
• Keywords: diffusion, generative
• Abstract: We present REM, a framework for segmenting a wide range of concepts in video that can be described through natural language. Our method capitalizes on visual-language representations learned by video diffusion models on Internet-scale datasets. A key insight of our approach is preserving as much of the generative model's original representation as possible, while fine-tuning it on narrow-domain Referral Object Segmentation datasets. As a result, our framework can accurately segment and track rare and unseen objects, despite being trained on object masks from a limited set of categories. Additionally, it can generalize to non-object dynamic concepts, such as waves crashing in the ocean, as demonstrated in our newly introduced benchmark for Referral Video Process Segmentation (Ref-VPS). Our experiments show that REM performs on par with state-of-the-art approaches on in-domain datasets, like Ref-DAVIS, while outperforming them by up to twelve points in terms of region similarity on out-of-domain data, leveraging the power of Internet-scale pre-training.