2025-01-29

Title: Self-supervised Graph Transformer with Contrastive Learning for Brain Connectivity Analysis towards Improving Autism Detection

• Authors: Yicheng Leng, Syed Muhammad Anwar, Islem Rekik, Sen He, Eung-Joo Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16346
• Pdf URL: https://arxiv.org/pdf/2501.16346
• Copy Paste: [[2501.16346]] Self-supervised Graph Transformer with Contrastive Learning for Brain Connectivity Analysis towards Improving Autism Detection(https://arxiv.org/abs/2501.16346)
• Keywords: self-supervised
• Abstract: Functional Magnetic Resonance Imaging (fMRI) provides useful insights into the brain function both during task or rest. Representing fMRI data using correlation matrices is found to be a reliable method of analyzing the inherent connectivity of the brain in the resting and active states. Graph Neural Networks (GNNs) have been widely used for brain network analysis due to their inherent explainability capability. In this work, we introduce a novel framework using contrastive self-supervised learning graph transformers, incorporating a brain network transformer encoder with random graph alterations. The proposed network leverages both contrastive learning and graph alterations to effectively train the graph transformer for autism detection. Our approach, tested on Autism Brain Imaging Data Exchange (ABIDE) data, demonstrates superior autism detection, achieving an AUROC of 82.6 and an accuracy of 74%, surpassing current state-of-the-art methods.

Title: An Integrated Approach to AI-Generated Content in e-health

• Authors: Tasnim Ahmed, Salimur Choudhury
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16348
• Pdf URL: https://arxiv.org/pdf/2501.16348
• Copy Paste: [[2501.16348]] An Integrated Approach to AI-Generated Content in e-health(https://arxiv.org/abs/2501.16348)
• Keywords: diffusion, generative
• Abstract: Artificial Intelligence-Generated Content, a subset of Generative Artificial Intelligence, holds significant potential for advancing the e-health sector by generating diverse forms of data. In this paper, we propose an end-to-end class-conditioned framework that addresses the challenge of data scarcity in health applications by generating synthetic medical images and text data, evaluating on practical applications such as retinopathy detection, skin infections and mental health assessments. Our framework integrates Diffusion and Large Language Models (LLMs) to generate data that closely match real-world patterns, which is essential for improving downstream task performance and model robustness in e-health applications. Experimental results demonstrate that the synthetic images produced by the proposed diffusion model outperform traditional GAN architectures. Similarly, in the text modality, data generated by uncensored LLM achieves significantly better alignment with real-world data than censored models in replicating the authentic tone.

Title: Risk-Informed Diffusion Transformer for Long-Tail Trajectory Prediction in the Crash Scenario

• Authors: Junlan Chen, Pei Liu, Zihao Zhang, Hongyi Zhao, Yufei Ji, Ziyuan Pu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16349
• Pdf URL: https://arxiv.org/pdf/2501.16349
• Copy Paste: [[2501.16349]] Risk-Informed Diffusion Transformer for Long-Tail Trajectory Prediction in the Crash Scenario(https://arxiv.org/abs/2501.16349)
• Keywords: diffusion
• Abstract: Trajectory prediction methods have been widely applied in autonomous driving technologies. Although the overall performance accuracy of trajectory prediction is relatively high, the lack of trajectory data in critical scenarios in the training data leads to the long-tail phenomenon. Normally, the trajectories of the tail data are more critical and more difficult to predict and may include rare scenarios such as crashes. To solve this problem, we extracted the trajectory data from real-world crash scenarios, which contain more long-tail data. Meanwhile, based on the trajectory data in this scenario, we integrated graph-based risk information and diffusion with transformer and proposed the Risk-Informed Diffusion Transformer (RI-DiT) trajectory prediction method. Extensive experiments were conducted on trajectory data in the real-world crash scenario, and the results show that the algorithm we proposed has good performance. When predicting the data of the tail 10\% (Top 10\%), the minADE and minFDE indicators are 0.016/2.667 m. At the same time, we showed the trajectory conditions of different long-tail distributions. The distribution of trajectory data is closer to the tail, the less smooth the trajectory is. Through the trajectory data in real-world crash scenarios, Our work expands the methods to overcome the long-tail challenges in trajectory prediction. Our method, RI-DiT, integrates inverse time to collision (ITTC) and the feature of traffic flow, which can predict long-tail trajectories more accurately and improve the safety of autonomous driving systems.

Title: The OpenLAM Challenges

• Authors: Anyang Peng, Xinzijian Liu, Ming-Yu Guo, Linfeng Zhang, Han Wang
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2501.16358
• Pdf URL: https://arxiv.org/pdf/2501.16358
• Copy Paste: [[2501.16358]] The OpenLAM Challenges(https://arxiv.org/abs/2501.16358)
• Keywords: foundation model, generative
• Abstract: Inspired by the success of Large Language Models (LLMs), the development of Large Atom Models (LAMs) has gained significant momentum in scientific computation. Since 2022, the Deep Potential team has been actively pretraining LAMs and launched the OpenLAM Initiative to develop an open-source foundation model spanning the periodic table. A core objective is establishing comprehensive benchmarks for reliable LAM evaluation, addressing limitations in existing datasets. As a first step, the LAM Crystal Philately competition has collected over 19.8 million valid structures, including 1 million on the OpenLAM convex hull, driving advancements in generative modeling and materials science applications.

Title: Multivariate Time Series Anomaly Detection by Capturing Coarse-Grained Intra- and Inter-Variate Dependencies

• Authors: Yongzheng Xie, Hongyu Zhang, Muhammad Ali Babar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16364
• Pdf URL: https://arxiv.org/pdf/2501.16364
• Copy Paste: [[2501.16364]] Multivariate Time Series Anomaly Detection by Capturing Coarse-Grained Intra- and Inter-Variate Dependencies(https://arxiv.org/abs/2501.16364)
• Keywords: anomaly
• Abstract: Multivariate time series anomaly detection is essential for failure management in web application operations, as it directly influences the effectiveness and timeliness of implementing remedial or preventive measures. This task is often framed as a semi-supervised learning problem, where only normal data are available for model training, primarily due to the labor-intensive nature of data labeling and the scarcity of anomalous data. Existing semi-supervised methods often detect anomalies by capturing intra-variate temporal dependencies and/or inter-variate relationships to learn normal patterns, flagging timestamps that deviate from these patterns as anomalies. However, these approaches often fail to capture salient intra-variate temporal and inter-variate dependencies in time series due to their focus on excessively fine granularity, leading to suboptimal performance. In this study, we introduce MtsCID, a novel semi-supervised multivariate time series anomaly detection method. MtsCID employs a dual network architecture: one network operates on the attention maps of multi-scale intra-variate patches for coarse-grained temporal dependency learning, while the other works on variates to capture coarse-grained inter-variate relationships through convolution and interaction with sinusoidal prototypes. This design enhances the ability to capture the patterns from both intra-variate temporal dependencies and inter-variate relationships, resulting in improved performance. Extensive experiments across seven widely used datasets demonstrate that MtsCID achieves performance comparable or superior to state-of-the-art benchmark methods.

Title: Foundation Models for CPS-IoT: Opportunities and Challenges

• Authors: Ozan Baris, Yizhuo Chen, Gaofeng Dong, Liying Han, Tomoyoshi Kimura, Pengrui Quan, Ruijie Wang, Tianchen Wang, Tarek Abdelzaher, Mario Bergés, Paul Pu Liang, Mani Srivastava
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2501.16368
• Pdf URL: https://arxiv.org/pdf/2501.16368
• Copy Paste: [[2501.16368]] Foundation Models for CPS-IoT: Opportunities and Challenges(https://arxiv.org/abs/2501.16368)
• Keywords: foundation model
• Abstract: Methods from machine learning (ML) have transformed the implementation of Perception-Cognition-Communication-Action loops in Cyber-Physical Systems (CPS) and the Internet of Things (IoT), replacing mechanistic and basic statistical models with those derived from data. However, the first generation of ML approaches, which depend on supervised learning with annotated data to create task-specific models, faces significant limitations in scaling to the diverse sensor modalities, deployment configurations, application tasks, and operating dynamics characterizing real-world CPS-IoT systems. The success of task-agnostic foundation models (FMs), including multimodal large language models (LLMs), in addressing similar challenges across natural language, computer vision, and human speech has generated considerable enthusiasm for and exploration of FMs and LLMs as flexible building blocks in CPS-IoT analytics pipelines, promising to reduce the need for costly task-specific engineering. Nonetheless, a significant gap persists between the current capabilities of FMs and LLMs in the CPS-IoT domain and the requirements they must meet to be viable for CPS-IoT applications. In this paper, we analyze and characterize this gap through a thorough examination of the state of the art and our research, which extends beyond it in various dimensions. Based on the results of our analysis and research, we identify essential desiderata that CPS-IoT domain-specific FMs and LLMs must satisfy to bridge this gap. We also propose actions by CPS-IoT researchers to collaborate in developing key community resources necessary for establishing FMs and LLMs as foundational tools for the next generation of CPS-IoT systems.

Title: UDiTQC: U-Net-Style Diffusion Transformer for Quantum Circuit Synthesis

• Authors: Zhiwei Chen, Hao Tang
• Subjects: cs.LG, cs.AI, quant-ph
• Abstract URL: https://arxiv.org/abs/2501.16380
• Pdf URL: https://arxiv.org/pdf/2501.16380
• Copy Paste: [[2501.16380]] UDiTQC: U-Net-Style Diffusion Transformer for Quantum Circuit Synthesis(https://arxiv.org/abs/2501.16380)
• Keywords: diffusion, generative
• Abstract: Quantum computing is a transformative technology with wide-ranging applications, and efficient quantum circuit generation is crucial for unlocking its full potential. Current diffusion model approaches based on U-Net architectures, while promising, encounter challenges related to computational efficiency and modeling global context. To address these issues, we propose UDiT,a novel U-Net-style Diffusion Transformer architecture, which combines U-Net's strengths in multi-scale feature extraction with the Transformer's ability to model global context. We demonstrate the framework's effectiveness on two tasks: entanglement generation and unitary compilation, where UDiTQC consistently outperforms existing methods. Additionally, our framework supports tasks such as masking and editing circuits to meet specific physical property requirements. This dual advancement, improving quantum circuit synthesis and refining generative model architectures, marks a significant milestone in the convergence of quantum computing and machine learning research.

Title: DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation

• Authors: Han Sun, Rui Gong, Ismail Nejjar, Olga Fink
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16410
• Pdf URL: https://arxiv.org/pdf/2501.16410
• Copy Paste: [[2501.16410]] DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation(https://arxiv.org/abs/2501.16410)
• Keywords: foundation model
• Abstract: Current unsupervised domain adaptation (UDA) methods for semantic segmentation typically assume identical class labels between the source and target domains. This assumption ignores the label-level domain gap, which is common in real-world scenarios, thus limiting their ability to identify finer-grained or novel categories without requiring extensive manual annotation. A promising direction to address this limitation lies in recent advancements in foundation models, which exhibit strong generalization abilities due to their rich prior knowledge. However, these models often struggle with domain-specific nuances and underrepresented fine-grained categories. To address these challenges, we introduce DynAlign, a framework that integrates UDA with foundation models to bridge both the image-level and label-level domain gaps. Our approach leverages prior semantic knowledge to align source categories with target categories that can be novel, more fine-grained, or named differently (e.g., vehicle to {car, truck, bus}). Foundation models are then employed for precise segmentation and category reassignment. To further enhance accuracy, we propose a knowledge fusion approach that dynamically adapts to varying scene contexts. DynAlign generates accurate predictions in a new target label space without requiring any manual annotations, allowing seamless adaptation to new taxonomies through either model retraining or direct inference. Experiments on the street scene semantic segmentation benchmarks GTA to Mapillary Vistas and GTA to IDD validate the effectiveness of our approach, achieving a significant improvement over existing methods. Our code will be publicly available.

Title: Detecting Zero-Day Attacks in Digital Substations via In-Context Learning

• Authors: Faizan Manzoor, Vanshaj Khattar, Akila Herath, Clifton Black, Matthew C Nielsen, Junho Hong, Chen-Ching Liu, Ming Jin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16453
• Pdf URL: https://arxiv.org/pdf/2501.16453
• Copy Paste: [[2501.16453]] Detecting Zero-Day Attacks in Digital Substations via In-Context Learning(https://arxiv.org/abs/2501.16453)
• Keywords: in-context
• Abstract: The occurrences of cyber attacks on the power grids have been increasing every year, with novel attack techniques emerging every year. In this paper, we address the critical challenge of detecting novel/zero-day attacks in digital substations that employ the IEC-61850 communication protocol. While many heuristic and machine learning (ML)-based methods have been proposed for attack detection in IEC-61850 digital substations, generalization to novel or zero-day attacks remains challenging. We propose an approach that leverages the in-context learning (ICL) capability of the transformer architecture, the fundamental building block of large language models. The ICL approach enables the model to detect zero-day attacks and learn from a few examples of that attack without explicit retraining. Our experiments on the IEC-61850 dataset demonstrate that the proposed method achieves more than $85\%$ detection accuracy on zero-day attacks while the existing state-of-the-art baselines fail. This work paves the way for building more secure and resilient digital substations of the future.

Title: SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments

• Authors: Simon Dahan, Gabriel Bénédict, Logan Z. J. Williams, Yourong Guo, Daniel Rueckert, Robert Leech, Emma C. Robinson
• Subjects: cs.LG, cs.AI, eess.AS, eess.IV, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2501.16471
• Pdf URL: https://arxiv.org/pdf/2501.16471
• Copy Paste: [[2501.16471]] SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments(https://arxiv.org/abs/2501.16471)
• Keywords: self-supervised
• Abstract: Current AI frameworks for brain decoding and encoding, typically train and test models within the same datasets. This limits their utility for brain computer interfaces (BCI) or neurofeedback, for which it would be useful to pool experiences across individuals to better simulate stimuli not sampled during training. A key obstacle to model generalisation is the degree of variability of inter-subject cortical organisation, which makes it difficult to align or compare cortical signals across participants. In this paper we address this through the use of surface vision transformers, which build a generalisable model of cortical functional dynamics, through encoding the topography of cortical networks and their interactions as a moving image across a surface. This is then combined with tri-modal self-supervised contrastive (CLIP) alignment of audio, video, and fMRI modalities to enable the retrieval of visual and auditory stimuli from patterns of cortical activity (and vice-versa). We validate our approach on 7T task-fMRI data from 174 healthy participants engaged in the movie-watching experiment from the Human Connectome Project (HCP). Results show that it is possible to detect which movie clips an individual is watching purely from their brain activity, even for individuals and movies not seen during training. Further analysis of attention maps reveals that our model captures individual patterns of brain activity that reflect semantic and visual systems. This opens the door to future personalised simulations of brain function. Code & pre-trained models will be made available at this https URL, processed data for training will be available upon request at this https URL.

Title: Towards Robust Stability Prediction in Smart Grids: GAN-based Approach under Data Constraints and Adversarial Challenges

• Authors: Emad Efatinasab, Alessandro Brighente, Denis Donadel, Mauro Conti, Mirco Rampazzo
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.16490
• Pdf URL: https://arxiv.org/pdf/2501.16490
• Copy Paste: [[2501.16490]] Towards Robust Stability Prediction in Smart Grids: GAN-based Approach under Data Constraints and Adversarial Challenges(https://arxiv.org/abs/2501.16490)
• Keywords: generative
• Abstract: Smart grids are critical for addressing the growing energy demand due to global population growth and urbanization. They enhance efficiency, reliability, and sustainability by integrating renewable energy. Ensuring their availability and safety requires advanced operational control and safety measures. Researchers employ AI and machine learning to assess grid stability, but challenges like the lack of datasets and cybersecurity threats, including adversarial attacks, persist. In particular, data scarcity is a key issue: obtaining grid instability instances is tough due to the need for significant expertise, resources, and time. However, they are essential to test novel research advancements and security mitigations. In this paper, we introduce a novel framework to detect instability in smart grids by employing only stable data. It relies on a Generative Adversarial Network (GAN) where the generator is trained to create instability data that are used along with stable data to train the discriminator. Moreover, we include a new adversarial training layer to improve robustness against adversarial attacks. Our solution, tested on a dataset composed of real-world stable and unstable samples, achieve accuracy up to 97.5\% in predicting grid stability and up to 98.9\% in detecting adversarial attacks. Moreover, we implemented our model in a single-board computer demonstrating efficient real-time decision-making with an average response time of less than 7ms. Our solution improves prediction accuracy and resilience while addressing data scarcity in smart grid management.

Title: How well can LLMs Grade Essays in Arabic?

• Authors: Rayed Ghazawi, Edwin Simpson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16516
• Pdf URL: https://arxiv.org/pdf/2501.16516
• Copy Paste: [[2501.16516]] How well can LLMs Grade Essays in Arabic?(https://arxiv.org/abs/2501.16516)
• Keywords: generative, in-context
• Abstract: This research assesses the effectiveness of state-of-the-art large language models (LLMs), including ChatGPT, Llama, Aya, Jais, and ACEGPT, in the task of Arabic automated essay scoring (AES) using the AR-AES dataset. It explores various evaluation methodologies, including zero-shot, few-shot in-context learning, and fine-tuning, and examines the influence of instruction-following capabilities through the inclusion of marking guidelines within the prompts. A mixed-language prompting strategy, integrating English prompts with Arabic content, was implemented to improve model comprehension and performance. Among the models tested, ACEGPT demonstrated the strongest performance across the dataset, achieving a Quadratic Weighted Kappa (QWK) of 0.67, but was outperformed by a smaller BERT-based model with a QWK of 0.88. The study identifies challenges faced by LLMs in processing Arabic, including tokenization complexities and higher computational demands. Performance variation across different courses underscores the need for adaptive models capable of handling diverse assessment formats and highlights the positive impact of effective prompt engineering on improving LLM outputs. To the best of our knowledge, this study is the first to empirically evaluate the performance of multiple generative Large Language Models (LLMs) on Arabic essays using authentic student data.

Title: PackDiT: Joint Human Motion and Text Generation via Mutual Prompting

• Authors: Zhongyu Jiang, Wenhao Chai, Zhuoran Zhou, Cheng-Yen Yang, Hsiang-Wei Huang, Jenq-Neng Hwang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.16551
• Pdf URL: https://arxiv.org/pdf/2501.16551
• Copy Paste: [[2501.16551]] PackDiT: Joint Human Motion and Text Generation via Mutual Prompting(https://arxiv.org/abs/2501.16551)
• Keywords: diffusion, generative
• Abstract: Human motion generation has advanced markedly with the advent of diffusion models. Most recent studies have concentrated on generating motion sequences based on text prompts, commonly referred to as text-to-motion generation. However, the bidirectional generation of motion and text, enabling tasks such as motion-to-text alongside text-to-motion, has been largely unexplored. This capability is essential for aligning diverse modalities and supports unconditional generation. In this paper, we introduce PackDiT, the first diffusion-based generative model capable of performing various tasks simultaneously, including motion generation, motion prediction, text generation, text-to-motion, motion-to-text, and joint motion-text generation. Our core innovation leverages mutual blocks to integrate multiple diffusion transformers (DiTs) across different modalities seamlessly. We train PackDiT on the HumanML3D dataset, achieving state-of-the-art text-to-motion performance with an FID score of 0.106, along with superior results in motion prediction and in-between tasks. Our experiments further demonstrate that diffusion models are effective for motion-to-text generation, achieving performance comparable to that of autoregressive models.

Title: LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation

• Authors: Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, Fatih Porikli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16559
• Pdf URL: https://arxiv.org/pdf/2501.16559
• Copy Paste: [[2501.16559]] LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation(https://arxiv.org/abs/2501.16559)
• Keywords: diffusion, foundation model
• Abstract: The rising popularity of large foundation models has led to a heightened demand for parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), which offer performance comparable to full model fine-tuning while requiring only a few additional parameters tailored to the specific base model. When such base models are deprecated and replaced, all associated LoRA modules must be retrained, requiring access to either the original training data or a substantial amount of synthetic data that mirrors the original distribution. However, the original data is often inaccessible due to privacy or licensing issues, and generating synthetic data may be impractical and insufficiently representative. These factors complicate the fine-tuning process considerably. To address this challenge, we introduce a new adapter, Cross-Model Low-Rank Adaptation (LoRA-X), which enables the training-free transfer of LoRA parameters across source and target models, eliminating the need for original or synthetic training data. Our approach imposes the adapter to operate within the subspace of the source base model. This constraint is necessary because our prior knowledge of the target model is limited to its weights, and the criteria for ensuring the adapter's transferability are restricted to the target base model's weights and subspace. To facilitate the transfer of LoRA parameters of the source model to a target model, we employ the adapter only in the layers of the target model that exhibit an acceptable level of subspace similarity. Our extensive experiments demonstrate the effectiveness of LoRA-X for text-to-image generation, including Stable Diffusion v1.5 and Stable Diffusion XL.

Title: Fine-Tuned Language Models as Space Systems Controllers

• Authors: Enrico M. Zucchelli, Di Wu, Julia Briden, Christian Hofmann, Victor Rodriguez-Fernandez, Richard Linares
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2501.16588
• Pdf URL: https://arxiv.org/pdf/2501.16588
• Copy Paste: [[2501.16588]] Fine-Tuned Language Models as Space Systems Controllers(https://arxiv.org/abs/2501.16588)
• Keywords: foundation model
• Abstract: Large language models (LLMs), or foundation models (FMs), are pretrained transformers that coherently complete sentences auto-regressively. In this paper, we show that LLMs can control simplified space systems after some additional training, called fine-tuning. We look at relatively small language models, ranging between 7 and 13 billion parameters. We focus on four problems: a three-dimensional spring toy problem, low-thrust orbit transfer, low-thrust cislunar control, and powered descent guidance. The fine-tuned LLMs are capable of controlling systems by generating sufficiently accurate outputs that are multi-dimensional vectors with up to 10 significant digits. We show that for several problems the amount of data required to perform fine-tuning is smaller than what is generally required of traditional deep neural networks (DNNs), and that fine-tuned LLMs are good at generalizing outside of the training dataset. Further, the same LLM can be fine-tuned with data from different problems, with only minor performance degradation with respect to LLMs trained for a single application. This work is intended as a first step towards the development of a general space systems controller.

Title: CascadeV: An Implementation of Wurstchen Architecture for Video Generation

• Authors: Wenfeng Lin, Jiangchuan Wei, Boyuan Liu, Yichen Zhang, Shiyue Yan, Mingyu Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16612
• Pdf URL: https://arxiv.org/pdf/2501.16612
• Copy Paste: [[2501.16612]] CascadeV: An Implementation of Wurstchen Architecture for Video Generation(https://arxiv.org/abs/2501.16612)
• Keywords: diffusion
• Abstract: Recently, with the tremendous success of diffusion models in the field of text-to-image (T2I) generation, increasing attention has been directed toward their potential in text-to-video (T2V) applications. However, the computational demands of diffusion models pose significant challenges, particularly in generating high-resolution videos with high frame rates. In this paper, we propose CascadeV, a cascaded latent diffusion model (LDM), that is capable of producing state-of-the-art 2K resolution videos. Experiments demonstrate that our cascaded model achieves a higher compression ratio, substantially reducing the computational challenges associated with high-quality video generation. We also implement a spatiotemporal alternating grid 3D attention mechanism, which effectively integrates spatial and temporal information, ensuring superior consistency across the generated video frames. Furthermore, our model can be cascaded with existing T2V models, theoretically enabling a 4$\times$ increase in resolution or frames per second without any fine-tuning. Our code is available at this https URL.

Title: Few-Shot Optimized Framework for Hallucination Detection in Resource-Limited NLP Systems

• Authors: Baraa Hikal, Ahmed Nasreldin, Ali Hamdi, Ammar Mohammed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.16616
• Pdf URL: https://arxiv.org/pdf/2501.16616
• Copy Paste: [[2501.16616]] Few-Shot Optimized Framework for Hallucination Detection in Resource-Limited NLP Systems(https://arxiv.org/abs/2501.16616)
• Keywords: generative
• Abstract: Hallucination detection in text generation remains an ongoing struggle for natural language processing (NLP) systems, frequently resulting in unreliable outputs in applications such as machine translation and definition modeling. Existing methods struggle with data scarcity and the limitations of unlabeled datasets, as highlighted by the SHROOM shared task at SemEval-2024. In this work, we propose a novel framework to address these challenges, introducing DeepSeek Few-shot optimization to enhance weak label generation through iterative prompt engineering. We achieved high-quality annotations that considerably enhanced the performance of downstream models by restructuring data to align with instruct generative models. We further fine-tuned the Mistral-7B-Instruct-v0.3 model on these optimized annotations, enabling it to accurately detect hallucinations in resource-limited settings. Combining this fine-tuned model with ensemble learning strategies, our approach achieved 85.5% accuracy on the test set, setting a new benchmark for the SHROOM task. This study demonstrates the effectiveness of data restructuring, few-shot optimization, and fine-tuning in building scalable and robust hallucination detection frameworks for resource-constrained NLP systems.

Title: Predicting 3D representations for Dynamic Scenes

• Authors: Di Qi, Tong Yang, Beining Wang, Xiangyu Zhang, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16617
• Pdf URL: https://arxiv.org/pdf/2501.16617
• Copy Paste: [[2501.16617]] Predicting 3D representations for Dynamic Scenes(https://arxiv.org/abs/2501.16617)
• Keywords: self-supervised
• Abstract: We present a novel framework for dynamic radiance field prediction given monocular video streams. Unlike previous methods that primarily focus on predicting future frames, our method goes a step further by generating explicit 3D representations of the dynamic scene. The framework builds on two core designs. First, we adopt an ego-centric unbounded triplane to explicitly represent the dynamic physical world. Second, we develop a 4D-aware transformer to aggregate features from monocular videos to update the triplane. Coupling these two designs enables us to train the proposed model with large-scale monocular videos in a self-supervised manner. Our model achieves top results in dynamic radiance field prediction on NVIDIA dynamic scenes, demonstrating its strong performance on 4D physical world modeling. Besides, our model shows a superior generalizability to unseen scenarios. Notably, we find that our approach emerges capabilities for geometry and semantic learning.

Title: Molecular-driven Foundation Model for Oncologic Pathology

• Authors: Anurag Vaidya, Andrew Zhang, Guillaume Jaume, Andrew H. Song, Tong Ding, Sophia J. Wagner, Ming Y. Lu, Paul Doucet, Harry Robertson, Cristina Almagro-Perez, Richard J. Chen, Dina ElHarouni, Georges Ayoub, Connor Bossi, Keith L. Ligon, Georg Gerber, Long Phi Le, Faisal Mahmood
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16652
• Pdf URL: https://arxiv.org/pdf/2501.16652
• Copy Paste: [[2501.16652]] Molecular-driven Foundation Model for Oncologic Pathology(https://arxiv.org/abs/2501.16652)
• Keywords: foundation model
• Abstract: Foundation models are reshaping computational pathology by enabling transfer learning, where models pre-trained on vast datasets can be adapted for downstream diagnostic, prognostic, and therapeutic response tasks. Despite these advances, foundation models are still limited in their ability to encode the entire gigapixel whole-slide images without additional training and often lack complementary multimodal data. Here, we introduce Threads, a slide-level foundation model capable of generating universal representations of whole-slide images of any size. Threads was pre-trained using a multimodal learning approach on a diverse cohort of 47,171 hematoxylin and eosin (H&E)-stained tissue sections, paired with corresponding genomic and transcriptomic profiles - the largest such paired dataset to be used for foundation model development to date. This unique training paradigm enables Threads to capture the tissue's underlying molecular composition, yielding powerful representations applicable to a wide array of downstream tasks. In extensive benchmarking across 54 oncology tasks, including clinical subtyping, grading, mutation prediction, immunohistochemistry status determination, treatment response prediction, and survival prediction, Threads outperformed all baselines while demonstrating remarkable generalizability and label efficiency. It is particularly well suited for predicting rare events, further emphasizing its clinical utility. We intend to make the model publicly available for the broader community.

Title: Federated Learning for Efficient Condition Monitoring and Anomaly Detection in Industrial Cyber-Physical Systems

• Authors: William Marfo, Deepak K. Tosh, Shirley V. Moore
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16666
• Pdf URL: https://arxiv.org/pdf/2501.16666
• Copy Paste: [[2501.16666]] Federated Learning for Efficient Condition Monitoring and Anomaly Detection in Industrial Cyber-Physical Systems(https://arxiv.org/abs/2501.16666)
• Keywords: anomaly
• Abstract: Detecting and localizing anomalies in cyber-physical systems (CPS) has become increasingly challenging as systems grow in complexity, particularly due to varying sensor reliability and node failures in distributed environments. While federated learning (FL) provides a foundation for distributed model training, existing approaches often lack mechanisms to address these CPS-specific challenges. This paper introduces an enhanced FL framework with three key innovations: adaptive model aggregation based on sensor reliability, dynamic node selection for resource optimization, and Weibull-based checkpointing for fault tolerance. The proposed framework ensures reliable condition monitoring while tackling the computational and reliability challenges of industrial CPS deployments. Experiments on the NASA Bearing and Hydraulic System datasets demonstrate superior performance compared to state-of-the-art FL methods, achieving 99.5% AUC-ROC in anomaly detection and maintaining accuracy even under node failures. Statistical validation using the Mann-Whitney U test confirms significant improvements, with a p-value less than 0.05, in both detection accuracy and computational efficiency across various operational scenarios.

Title: Data-Free Model-Related Attacks: Unleashing the Potential of Generative AI

• Authors: Dayong Ye, Tianqing Zhu, Shang Wang, Bo Liu, Leo Yu Zhang, Wanlei Zhou, Yang Zhang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16671
• Pdf URL: https://arxiv.org/pdf/2501.16671
• Copy Paste: [[2501.16671]] Data-Free Model-Related Attacks: Unleashing the Potential of Generative AI(https://arxiv.org/abs/2501.16671)
• Keywords: generative
• Abstract: Generative AI technology has become increasingly integrated into our daily lives, offering powerful capabilities to enhance productivity. However, these same capabilities can be exploited by adversaries for malicious purposes. While existing research on adversarial applications of generative AI predominantly focuses on cyberattacks, less attention has been given to attacks targeting deep learning models. In this paper, we introduce the use of generative AI for facilitating model-related attacks, including model extraction, membership inference, and model inversion. Our study reveals that adversaries can launch a variety of model-related attacks against both image and text models in a data-free and black-box manner, achieving comparable performance to baseline methods that have access to the target models' training data and parameters in a white-box manner. This research serves as an important early warning to the community about the potential risks associated with generative AI-powered attacks on deep learning models.

Title: Polyp-Gen: Realistic and Diverse Polyp Image Generation for Endoscopic Dataset Expansion

• Authors: Shengyuan Liu, Zhen Chen, Qiushi Yang, Weihao Yu, Di Dong, Jiancong Hu, Yixuan Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16679
• Pdf URL: https://arxiv.org/pdf/2501.16679
• Copy Paste: [[2501.16679]] Polyp-Gen: Realistic and Diverse Polyp Image Generation for Endoscopic Dataset Expansion(https://arxiv.org/abs/2501.16679)
• Keywords: diffusion
• Abstract: Automated diagnostic systems (ADS) have shown significant potential in the early detection of polyps during endoscopic examinations, thereby reducing the incidence of colorectal cancer. However, due to high annotation costs and strict privacy concerns, acquiring high-quality endoscopic images poses a considerable challenge in the development of ADS. Despite recent advancements in generating synthetic images for dataset expansion, existing endoscopic image generation algorithms failed to accurately generate the details of polyp boundary regions and typically required medical priors to specify plausible locations and shapes of polyps, which limited the realism and diversity of the generated images. To address these limitations, we present Polyp-Gen, the first full-automatic diffusion-based endoscopic image generation framework. Specifically, we devise a spatial-aware diffusion training scheme with a lesion-guided loss to enhance the structural context of polyp boundary regions. Moreover, to capture medical priors for the localization of potential polyp areas, we introduce a hierarchical retrieval-based sampling strategy to match similar fine-grained spatial features. In this way, our Polyp-Gen can generate realistic and diverse endoscopic images for building reliable ADS. Extensive experiments demonstrate the state-of-the-art generation quality, and the synthetic images can improve the downstream polyp detection task. Additionally, our Polyp-Gen has shown remarkable zero-shot generalizability on other datasets. The source code is available at this https URL.

Title: 3D-MoE: A Mixture-of-Experts Multi-modal LLM for 3D Vision and Pose Diffusion via Rectified Flow

• Authors: Yueen Ma, Yuzheng Zhuang, Jianye Hao, Irwin King
• Subjects: cs.CL, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.16698
• Pdf URL: https://arxiv.org/pdf/2501.16698
• Copy Paste: [[2501.16698]] 3D-MoE: A Mixture-of-Experts Multi-modal LLM for 3D Vision and Pose Diffusion via Rectified Flow(https://arxiv.org/abs/2501.16698)
• Keywords: diffusion
• Abstract: 3D vision and spatial reasoning have long been recognized as preferable for accurately perceiving our three-dimensional world, especially when compared with traditional visual reasoning based on 2D images. Due to the difficulties in collecting high-quality 3D data, research in this area has only recently gained momentum. With the advent of powerful large language models (LLMs), multi-modal LLMs for 3D vision have been developed over the past few years. However, most of these models focus primarily on the vision encoder for 3D data. In this paper, we propose converting existing densely activated LLMs into mixture-of-experts (MoE) models, which have proven effective for multi-modal data processing. In addition to leveraging these models' instruction-following capabilities, we further enable embodied task planning by attaching a diffusion head, Pose-DiT, that employs a novel rectified flow diffusion scheduler. Experimental results on 3D question answering and task-planning tasks demonstrate that our 3D-MoE framework achieves improved performance with fewer activated parameters.

Title: Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models

• Authors: Huijie Liu, Jingyun Wang, Shuai Ma, Jie Hu, Xiaoming Wei, Guoliang Kang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16714
• Pdf URL: https://arxiv.org/pdf/2501.16714
• Copy Paste: [[2501.16714]] Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models(https://arxiv.org/abs/2501.16714)
• Keywords: diffusion
• Abstract: Motion customization aims to adapt the diffusion model (DM) to generate videos with the motion specified by a set of video clips with the same motion concept. To realize this goal, the adaptation of DM should be possible to model the specified motion concept, without compromising the ability to generate diverse appearances. Thus, the key to solving this problem lies in how to separate the motion concept from the appearance in the adaptation process of DM. Typical previous works explore different ways to represent and insert a motion concept into large-scale pretrained text-to-video diffusion models, e.g., learning a motion LoRA, using latent noise residuals, etc. While those methods can encode the motion concept, they also inevitably encode the appearance in the reference videos, resulting in weakened appearance generation capability. In this paper, we follow the typical way to learn a motion LoRA to encode the motion concept, but propose two novel strategies to enhance motion-appearance separation, including temporal attention purification (TAP) and appearance highway (AH). Specifically, we assume that in the temporal attention module, the pretrained Value embeddings are sufficient to serve as basic components needed by producing a new motion. Thus, in TAP, we choose only to reshape the temporal attention with motion LoRAs so that Value embeddings can be reorganized to produce a new motion. Further, in AH, we alter the starting point of each skip connection in U-Net from the output of each temporal attention module to the output of each spatial attention module. Extensive experiments demonstrate that compared to previous works, our method can generate videos with appearance more aligned with the text descriptions and motion more consistent with the reference videos.

Title: Outlier Synthesis via Hamiltonian Monte Carlo for Out-of-Distribution Detection

• Authors: Hengzhuang Li, Teng Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.16718
• Pdf URL: https://arxiv.org/pdf/2501.16718
• Copy Paste: [[2501.16718]] Outlier Synthesis via Hamiltonian Monte Carlo for Out-of-Distribution Detection(https://arxiv.org/abs/2501.16718)
• Keywords: generative
• Abstract: Out-of-distribution (OOD) detection is crucial for developing trustworthy and reliable machine learning systems. Recent advances in training with auxiliary OOD data demonstrate efficacy in enhancing detection capabilities. Nonetheless, these methods heavily rely on acquiring a large pool of high-quality natural outliers. Some prior methods try to alleviate this problem by synthesizing virtual outliers but suffer from either poor quality or high cost due to the monotonous sampling strategy and the heavy-parameterized generative models. In this paper, we overcome all these problems by proposing the Hamiltonian Monte Carlo Outlier Synthesis (HamOS) framework, which views the synthesis process as sampling from Markov chains. Based solely on the in-distribution data, the Markov chains can extensively traverse the feature space and generate diverse and representative outliers, hence exposing the model to miscellaneous potential OOD scenarios. The Hamiltonian Monte Carlo with sampling acceptance rate almost close to 1 also makes our framework enjoy great efficiency. By empirically competing with SOTA baselines on both standard and large-scale benchmarks, we verify the efficacy and efficiency of our proposed HamOS.

Title: One Head Eight Arms: Block Matrix based Low Rank Adaptation for CLIP-based Few-Shot Learning

• Authors: Chunpeng Zhou, Qianqian Shen, Zhi Yu, Jiajun Bu, Haishuai Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16720
• Pdf URL: https://arxiv.org/pdf/2501.16720
• Copy Paste: [[2501.16720]] One Head Eight Arms: Block Matrix based Low Rank Adaptation for CLIP-based Few-Shot Learning(https://arxiv.org/abs/2501.16720)
• Keywords: foundation model
• Abstract: Recent advancements in fine-tuning Vision-Language Foundation Models (VLMs) have garnered significant attention for their effectiveness in downstream few-shot learning this http URL these recent approaches exhibits some performance improvements, they often suffer from excessive training parameters and high computational costs. To address these challenges, we propose a novel Block matrix-based low-rank adaptation framework, called Block-LoRA, for fine-tuning VLMs on downstream few-shot tasks. Inspired by recent work on Low-Rank Adaptation (LoRA), Block-LoRA partitions the original low-rank decomposition matrix of LoRA into a series of sub-matrices while sharing all down-projection sub-matrices. This structure not only reduces the number of training parameters, but also transforms certain complex matrix multiplication operations into simpler matrix addition, significantly lowering the computational cost of fine-tuning. Notably, Block-LoRA enables fine-tuning CLIP on the ImageNet few-shot benchmark using a single 24GB GPU. We also show that Block-LoRA has the more tighter bound of generalization error than vanilla LoRA. Without bells and whistles, extensive experiments demonstrate that Block-LoRA achieves competitive performance compared to state-of-the-art CLIP-based few-shot methods, while maintaining a low training parameters count and reduced computational overhead.

Title: Consistency Diffusion Models for Single-Image 3D Reconstruction with Priors

• Authors: Chenru Jiang, Chengrui Zhang, Xi Yang, Jie Sun, Kaizhu Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16737
• Pdf URL: https://arxiv.org/pdf/2501.16737
• Copy Paste: [[2501.16737]] Consistency Diffusion Models for Single-Image 3D Reconstruction with Priors(https://arxiv.org/abs/2501.16737)
• Keywords: diffusion
• Abstract: This paper delves into the study of 3D point cloud reconstruction from a single image. Our objective is to develop the Consistency Diffusion Model, exploring synergistic 2D and 3D priors in the Bayesian framework to ensure superior consistency in the reconstruction process, a challenging yet critical requirement in this field. Specifically, we introduce a pioneering training framework under diffusion models that brings two key innovations. First, we convert 3D structural priors derived from the initial 3D point cloud as a bound term to increase evidence in the variational Bayesian framework, leveraging these robust intrinsic priors to tightly govern the diffusion training process and bolster consistency in reconstruction. Second, we extract and incorporate 2D priors from the single input image, projecting them onto the 3D point cloud to enrich the guidance for diffusion training. Our framework not only sidesteps potential model learning shifts that may arise from directly imposing additional constraints during training but also precisely transposes the 2D priors into the 3D domain. Extensive experimental evaluations reveal that our approach sets new benchmarks in both synthetic and real-world datasets. The code is included with the submission.

Title: LLM Assisted Anomaly Detection Service for Site Reliability Engineers: Enhancing Cloud Infrastructure Resilience

• Authors: Nimesh Jha, Shuxin Lin, Srideepika Jayaraman, Kyle Frohling, Christodoulos Constantinides, Dhaval Patel
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.16744
• Pdf URL: https://arxiv.org/pdf/2501.16744
• Copy Paste: [[2501.16744]] LLM Assisted Anomaly Detection Service for Site Reliability Engineers: Enhancing Cloud Infrastructure Resilience(https://arxiv.org/abs/2501.16744)
• Keywords: foundation model, anomaly
• Abstract: This paper introduces a scalable Anomaly Detection Service with a generalizable API tailored for industrial time-series data, designed to assist Site Reliability Engineers (SREs) in managing cloud infrastructure. The service enables efficient anomaly detection in complex data streams, supporting proactive identification and resolution of issues. Furthermore, it presents an innovative approach to anomaly modeling in cloud infrastructure by utilizing Large Language Models (LLMs) to understand key components, their failure modes, and behaviors. A suite of algorithms for detecting anomalies is offered in univariate and multivariate time series data, including regression-based, mixture-model-based, and semi-supervised approaches. We provide insights into the usage patterns of the service, with over 500 users and 200,000 API calls in a year. The service has been successfully applied in various industrial settings, including IoT-based AI applications. We have also evaluated our system on public anomaly benchmarks to show its effectiveness. By leveraging it, SREs can proactively identify potential issues before they escalate, reducing downtime and improving response times to incidents, ultimately enhancing the overall customer experience. We plan to extend the system to include time series foundation models, enabling zero-shot anomaly detection capabilities.

Title: ITVTON:Virtual Try-On Diffusion Transformer Model Based on Integrated Image and Text

• Authors: Haifeng Ni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16757
• Pdf URL: https://arxiv.org/pdf/2501.16757
• Copy Paste: [[2501.16757]] ITVTON:Virtual Try-On Diffusion Transformer Model Based on Integrated Image and Text(https://arxiv.org/abs/2501.16757)
• Keywords: diffusion
• Abstract: Recent advancements in virtual fitting for characters and clothing have leveraged diffusion models to improve the realism of garment fitting. However, challenges remain in handling complex scenes and poses, which can result in unnatural garment fitting and poorly rendered intricate patterns. In this work, we introduce ITVTON, a novel method that enhances clothing-character interactions by combining clothing and character images along spatial channels as inputs, thereby improving fitting accuracy for the inpainting model. Additionally, we incorporate integrated textual descriptions from multiple images to boost the realism of the generated visual effects. To optimize computational efficiency, we limit training to the attention parameters within a single diffusion transformer (Single-DiT) block. To more rigorously address the complexities of real-world scenarios, we curated training samples from the IGPair dataset, thereby enhancing ITVTON's performance across diverse environments. Extensive experiments demonstrate that ITVTON outperforms baseline methods both qualitatively and quantitatively, setting a new standard for virtual fitting tasks.

Title: AdaSemSeg: An Adaptive Few-shot Semantic Segmentation of Seismic Facies

• Authors: Surojit Saha, Ross Whitaker
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.16760
• Pdf URL: https://arxiv.org/pdf/2501.16760
• Copy Paste: [[2501.16760]] AdaSemSeg: An Adaptive Few-shot Semantic Segmentation of Seismic Facies(https://arxiv.org/abs/2501.16760)
• Keywords: self-supervised
• Abstract: Automated interpretation of seismic images using deep learning methods is challenging because of the limited availability of training data. Few-shot learning is a suitable learning paradigm in such scenarios due to its ability to adapt to a new task with limited supervision (small training budget). Existing few-shot semantic segmentation (FSSS) methods fix the number of target classes. Therefore, they do not support joint training on multiple datasets varying in the number of classes. In the context of the interpretation of seismic facies, fixing the number of target classes inhibits the generalization capability of a model trained on one facies dataset to another, which is likely to have a different number of facies. To address this shortcoming, we propose a few-shot semantic segmentation method for interpreting seismic facies that can adapt to the varying number of facies across the dataset, dubbed the AdaSemSeg. In general, the backbone network of FSSS methods is initialized with the statistics learned from the ImageNet dataset for better performance. The lack of such a huge annotated dataset for seismic images motivates using a self-supervised algorithm on seismic datasets to initialize the backbone network. We have trained the AdaSemSeg on three public seismic facies datasets with different numbers of facies and evaluated the proposed method on multiple metrics. The performance of the AdaSemSeg on unseen datasets (not used in training) is better than the prototype-based few-shot method and baselines.

Title: DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation

• Authors: Chenguo Lin, Panwang Pan, Bangbang Yang, Zeming Li, Yadong Mu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16764
• Pdf URL: https://arxiv.org/pdf/2501.16764
• Copy Paste: [[2501.16764]] DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation(https://arxiv.org/abs/2501.16764)
• Keywords: diffusion, generative
• Abstract: Recent advancements in 3D content generation from text or a single image struggle with limited high-quality 3D datasets and inconsistency from 2D multi-view generation. We introduce DiffSplat, a novel 3D generative framework that natively generates 3D Gaussian splats by taming large-scale text-to-image diffusion models. It differs from previous 3D generative models by effectively utilizing web-scale 2D priors while maintaining 3D consistency in a unified model. To bootstrap the training, a lightweight reconstruction model is proposed to instantly produce multi-view Gaussian splat grids for scalable dataset curation. In conjunction with the regular diffusion loss on these grids, a 3D rendering loss is introduced to facilitate 3D coherence across arbitrary views. The compatibility with image diffusion models enables seamless adaptions of numerous techniques for image generation to the 3D realm. Extensive experiments reveal the superiority of DiffSplat in text- and image-conditioned generation tasks and downstream applications. Thorough ablation studies validate the efficacy of each critical design choice and provide insights into the underlying mechanism.

Title: Beyond-Labels: Advancing Open-Vocabulary Segmentation With Vision-Language Models

• Authors: Muhammad Atta ur Rahman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16769
• Pdf URL: https://arxiv.org/pdf/2501.16769
• Copy Paste: [[2501.16769]] Beyond-Labels: Advancing Open-Vocabulary Segmentation With Vision-Language Models(https://arxiv.org/abs/2501.16769)
• Keywords: self-supervised, foundation model
• Abstract: Self-supervised learning can resolve numerous image or linguistic processing problems when effectively trained. This study investigated simple yet efficient methods for adaping previously learned foundation models for open-vocabulary semantic segmentation tasks. Our research proposed "Beyond-Labels," a lightweight transformer-based fusion module that uses a handful of image segmentation data to fuse frozen image representations with language concepts. Furthermore, we efficiently captured positional information in images using Fourier embeddings, thus improving the generalization across various image sizes. Extensive ablation tests were performed to investigate the important components of our proposed method; when tested against the common benchmark PASCAL-5i, it demonstrated superior performance despite being trained on frozen image and language characteristics.

Title: FlexMotion: Lightweight, Physics-Aware, and Controllable Human Motion Generation

• Authors: Arvin Tashakori, Arash Tashakori, Gongbo Yang, Z. Jane Wang, Peyman Servati
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.16778
• Pdf URL: https://arxiv.org/pdf/2501.16778
• Copy Paste: [[2501.16778]] FlexMotion: Lightweight, Physics-Aware, and Controllable Human Motion Generation(https://arxiv.org/abs/2501.16778)
• Keywords: diffusion
• Abstract: Lightweight, controllable, and physically plausible human motion synthesis is crucial for animation, virtual reality, robotics, and human-computer interaction applications. Existing methods often compromise between computational efficiency, physical realism, or spatial controllability. We propose FlexMotion, a novel framework that leverages a computationally lightweight diffusion model operating in the latent space, eliminating the need for physics simulators and enabling fast and efficient training. FlexMotion employs a multimodal pre-trained Transformer encoder-decoder, integrating joint locations, contact forces, joint actuations and muscle activations to ensure the physical plausibility of the generated motions. FlexMotion also introduces a plug-and-play module, which adds spatial controllability over a range of motion parameters (e.g., joint locations, joint actuations, contact forces, and muscle activations). Our framework achieves realistic motion generation with improved efficiency and control, setting a new benchmark for human motion synthesis. We evaluate FlexMotion on extended datasets and demonstrate its superior performance in terms of realism, physical plausibility, and controllability.

Title: A Stochastic Dynamical Theory of LLM Self-Adversariality: Modeling Severity Drift as a Critical Process

• Authors: Jack David Carson
• Subjects: cs.CL, cs.AI, nlin.AO
• Abstract URL: https://arxiv.org/abs/2501.16783
• Pdf URL: https://arxiv.org/pdf/2501.16783
• Copy Paste: [[2501.16783]] A Stochastic Dynamical Theory of LLM Self-Adversariality: Modeling Severity Drift as a Critical Process(https://arxiv.org/abs/2501.16783)
• Keywords: diffusion
• Abstract: This paper introduces a continuous-time stochastic dynamical framework for understanding how large language models (LLMs) may self-amplify latent biases or toxicity through their own chain-of-thought reasoning. The model posits an instantaneous "severity" variable $x(t) \in [0,1]$ evolving under a stochastic differential equation (SDE) with a drift term $\mu(x)$ and diffusion $\sigma(x)$. Crucially, such a process can be consistently analyzed via the Fokker--Planck approach if each incremental step behaves nearly Markovian in severity space. The analysis investigates critical phenomena, showing that certain parameter regimes create phase transitions from subcritical (self-correcting) to supercritical (runaway severity). The paper derives stationary distributions, first-passage times to harmful thresholds, and scaling laws near critical points. Finally, it highlights implications for agents and extended LLM reasoning models: in principle, these equations might serve as a basis for formal verification of whether a model remains stable or propagates bias over repeated inferences.

Title: Algorithm for Automatic Legislative Text Consolidation

• Authors: Matias Etcheverry, Thibaud Real, Pauline Chavallard
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.16794
• Pdf URL: https://arxiv.org/pdf/2501.16794
• Copy Paste: [[2501.16794]] Algorithm for Automatic Legislative Text Consolidation(https://arxiv.org/abs/2501.16794)
• Keywords: generative
• Abstract: This study introduces a method for automating the consolidation process in a legal context, a time-consuming task traditionally performed by legal professionals. We present a generative approach that processes legislative texts to automatically apply amendments. Our method employs light quantized generative model, fine-tuned with LoRA, to generate accurate and reliable amended texts. To the authors knowledge, this is the first time generative models are used on legislative text consolidation. Our dataset is publicly available on HuggingFace1. Experimental results demonstrate a significant improvement in efficiency, offering faster updates to legal documents. A full automated pipeline of legislative text consolidation can be done in a few hours, with a success rate of more than 63% on a difficult bill.

Title: Can Transformers Learn Full Bayesian Inference in Context?

• Authors: Arik Reuter, Tim G. J. Rudner, Vincent Fortuin, David Rügamer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.16825
• Pdf URL: https://arxiv.org/pdf/2501.16825
• Copy Paste: [[2501.16825]] Can Transformers Learn Full Bayesian Inference in Context?(https://arxiv.org/abs/2501.16825)
• Keywords: in-context
• Abstract: Transformers have emerged as the dominant architecture in the field of deep learning, with a broad range of applications and remarkable in-context learning (ICL) capabilities. While not yet fully understood, ICL has already proved to be an intriguing phenomenon, allowing transformers to learn in context -- without requiring further training. In this paper, we further advance the understanding of ICL by demonstrating that transformers can perform full Bayesian inference for commonly used statistical models in context. More specifically, we introduce a general framework that builds on ideas from prior fitted networks and continuous normalizing flows which enables us to infer complex posterior distributions for methods such as generalized linear models and latent factor models. Extensive experiments on real-world datasets demonstrate that our ICL approach yields posterior samples that are similar in quality to state-of-the-art MCMC or variational inference methods not operating in context.

Title: Flow Matching: Markov Kernels, Stochastic Processes and Transport Plans

• Authors: Christian Wald, Gabriele Steidl
• Subjects: cs.LG, math.PR
• Abstract URL: https://arxiv.org/abs/2501.16839
• Pdf URL: https://arxiv.org/pdf/2501.16839
• Copy Paste: [[2501.16839]] Flow Matching: Markov Kernels, Stochastic Processes and Transport Plans(https://arxiv.org/abs/2501.16839)
• Keywords: generative
• Abstract: Among generative neural models, flow matching techniques stand out for their simple applicability and good scaling properties. Here, velocity fields of curves connecting a simple latent and a target distribution are learned. Then the corresponding ordinary differential equation can be used to sample from a target distribution, starting in samples from the latent one. This paper reviews from a mathematical point of view different techniques to learn the velocity fields of absolutely continuous curves in the Wasserstein geometry. We show how the velocity fields can be characterized and learned via i) transport plans (couplings) between latent and target distributions, ii) Markov kernels and iii) stochastic processes, where the latter two include the coupling approach, but are in general broader. Besides this main goal, we show how flow matching can be used for solving Bayesian inverse problems, where the definition of conditional Wasserstein distances plays a central role. Finally, we briefly address continuous normalizing flows and score matching techniques, which approach the learning of velocity fields of curves from other directions.

Title: Adversarial Masked Autoencoder Purifier with Defense Transferability

• Authors: Yuan-Chih Chen, Chun-Shien Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16904
• Pdf URL: https://arxiv.org/pdf/2501.16904
• Copy Paste: [[2501.16904]] Adversarial Masked Autoencoder Purifier with Defense Transferability(https://arxiv.org/abs/2501.16904)
• Keywords: diffusion
• Abstract: The study of adversarial defense still struggles to combat with advanced adversarial attacks. In contrast to most prior studies that rely on the diffusion model for test-time defense to remarkably increase the inference time, we propose Masked AutoEncoder Purifier (MAEP), which integrates Masked AutoEncoder (MAE) into an adversarial purifier framework for test-time purification. While MAEP achieves promising adversarial robustness, it particularly features model defense transferability and attack generalization without relying on using additional data that is different from the training dataset. To our knowledge, MAEP is the first study of adversarial purifier based on MAE. Extensive experimental results demonstrate that our method can not only maintain clear accuracy with only a slight drop but also exhibit a close gap between the clean and robust accuracy. Notably, MAEP trained on CIFAR10 achieves state-of-the-art performance even when tested directly on ImageNet, outperforming existing diffusion-based models trained specifically on ImageNet.

Title: TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models

• Authors: Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, Takuya Akiba
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.16937
• Pdf URL: https://arxiv.org/pdf/2501.16937
• Copy Paste: [[2501.16937]] TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models(https://arxiv.org/abs/2501.16937)
• Keywords: foundation model
• Abstract: Causal language models have demonstrated remarkable capabilities, but their size poses significant challenges for deployment in resource-constrained environments. Knowledge distillation, a widely-used technique for transferring knowledge from a large teacher model to a small student model, presents a promising approach for model compression. A significant remaining issue lies in the major differences between teacher and student models, namely the substantial capacity gap, mode averaging, and mode collapse, which pose barriers during distillation. To address these issues, we introduce $\textit{Temporally Adaptive Interpolated Distillation (TAID)}$, a novel knowledge distillation approach that dynamically interpolates student and teacher distributions through an adaptive intermediate distribution, gradually shifting from the student's initial distribution towards the teacher's distribution. We provide a theoretical analysis demonstrating TAID's ability to prevent mode collapse and empirically show its effectiveness in addressing the capacity gap while balancing mode averaging and mode collapse. Our comprehensive experiments demonstrate TAID's superior performance across various model sizes and architectures in both instruction tuning and pre-training scenarios. Furthermore, we showcase TAID's practical impact by developing two state-of-the-art compact foundation models: $\texttt{TAID-LLM-1.5B}$ for language tasks and $\texttt{TAID-VLM-2B}$ for vision-language tasks. These results demonstrate TAID's effectiveness in creating high-performing and efficient models, advancing the development of more accessible AI technologies.

Title: Few Edges Are Enough: Few-Shot Network Attack Detection with Graph Neural Networks

• Authors: Tristan Bilot, Nour El Madhoun, Khaldoun Al Agha, Anis Zouaoui
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.16964
• Pdf URL: https://arxiv.org/pdf/2501.16964
• Copy Paste: [[2501.16964]] Few Edges Are Enough: Few-Shot Network Attack Detection with Graph Neural Networks(https://arxiv.org/abs/2501.16964)
• Keywords: self-supervised
• Abstract: Detecting cyberattacks using Graph Neural Networks (GNNs) has seen promising results recently. Most of the state-of-the-art models that leverage these techniques require labeled examples, hard to obtain in many real-world scenarios. To address this issue, unsupervised learning and Self-Supervised Learning (SSL) have emerged as interesting approaches to reduce the dependency on labeled data. Nonetheless, these methods tend to yield more anomalous detection algorithms rather than effective attack detection systems. This paper introduces Few Edges Are Enough (FEAE), a GNN-based architecture trained with SSL and Few-Shot Learning (FSL) to better distinguish between false positive anomalies and actual attacks. To maximize the potential of few-shot examples, our model employs a hybrid self-supervised objective that combines the advantages of contrastive-based and reconstruction-based SSL. By leveraging only a minimal number of labeled attack events, represented as attack edges, FEAE achieves competitive performance on two well-known network datasets compared to both supervised and unsupervised methods. Remarkably, our experimental results unveil that employing only 1 malicious event for each attack type in the dataset is sufficient to achieve substantial improvements. FEAE not only outperforms self-supervised GNN baselines but also surpasses some supervised approaches on one of the datasets.

Title: Modulating CNN Features with Pre-Trained ViT Representations for Open-Vocabulary Object Detection

• Authors: Xiangyu Gao, Yu Dai, Benliu Qiu, Hongliang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16981
• Pdf URL: https://arxiv.org/pdf/2501.16981
• Copy Paste: [[2501.16981]] Modulating CNN Features with Pre-Trained ViT Representations for Open-Vocabulary Object Detection(https://arxiv.org/abs/2501.16981)
• Keywords: generative
• Abstract: Owing to large-scale image-text contrastive training, pre-trained vision language model (VLM) like CLIP shows superior open-vocabulary recognition ability. Most existing open-vocabulary object detectors attempt to utilize the pre-trained VLM to attain generative representation. F-ViT uses the pre-trained visual encoder as the backbone network and freezes it during training. However, the frozen backbone doesn't benefit from the labeled data to strengthen the representation. Therefore, we propose a novel two-branch backbone network design, named as ViT-Feature-Modulated Multi-Scale Convolutional network (VMCNet). VMCNet consists of a trainable convolutional branch, a frozen pre-trained ViT branch and a feature modulation module. The trainable CNN branch could be optimized with labeled data while the frozen pre-trained ViT branch could keep the representation ability derived from large-scale pre-training. Then, the proposed feature modulation module could modulate the multi-scale CNN features with the representations from ViT branch. With the proposed mixed structure, detector is more likely to discover novel categories. Evaluated on two popular benchmarks, our method boosts the detection performance on novel category and outperforms the baseline. On OV-COCO, the proposed method achieves 44.3 AP$_{50}^{\mathrm{novel}}$ with ViT-B/16 and 48.5 AP$_{50}^{\mathrm{novel}}$ with ViT-L/14. On OV-LVIS, VMCNet with ViT-B/16 and ViT-L/14 reaches 27.8 and 38.4 mAP$_{r}$.

Title: FedEFM: Federated Endovascular Foundation Model with Unseen Data

• Authors: Tuong Do, Nghia Vu, Tudor Jianu, Baoru Huang, Minh Vu, Jionglong Su, Erman Tjiputra, Quang D. Tran, Te-Chuan Chiu, Anh Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.16992
• Pdf URL: https://arxiv.org/pdf/2501.16992
• Copy Paste: [[2501.16992]] FedEFM: Federated Endovascular Foundation Model with Unseen Data(https://arxiv.org/abs/2501.16992)
• Keywords: foundation model
• Abstract: In endovascular surgery, the precise identification of catheters and guidewires in X-ray images is essential for reducing intervention risks. However, accurately segmenting catheter and guidewire structures is challenging due to the limited availability of labeled data. Foundation models offer a promising solution by enabling the collection of similar domain data to train models whose weights can be fine-tuned for downstream tasks. Nonetheless, large-scale data collection for training is constrained by the necessity of maintaining patient privacy. This paper proposes a new method to train a foundation model in a decentralized federated learning setting for endovascular intervention. To ensure the feasibility of the training, we tackle the unseen data issue using differentiable Earth Mover's Distance within a knowledge distillation framework. Once trained, our foundation model's weights provide valuable initialization for downstream tasks, thereby enhancing task-specific performance. Intensive experiments show that our approach achieves new state-of-the-art results, contributing to advancements in endovascular intervention and robotic-assisted endovascular surgery, while addressing the critical issue of data sharing in the medical domain.

Title: MAUCell: An Adaptive Multi-Attention Framework for Video Frame Prediction

• Authors: Shreyam Gupta (1), P. Agrawal (2), Priyam Gupta (3) ((1) Indian Institute of Technology (BHU), Varanasi, India, (2) University of Colorado, Boulder, USA, (3) Intelligent Field Robotic Systems (IFROS), University of Girona, Spain)
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.16997
• Pdf URL: https://arxiv.org/pdf/2501.16997
• Copy Paste: [[2501.16997]] MAUCell: An Adaptive Multi-Attention Framework for Video Frame Prediction(https://arxiv.org/abs/2501.16997)
• Keywords: generative, anomaly
• Abstract: Temporal sequence modeling stands as the fundamental foundation for video prediction systems and real-time forecasting operations as well as anomaly detection applications. The achievement of accurate predictions through efficient resource consumption remains an ongoing issue in contemporary temporal sequence modeling. We introduce the Multi-Attention Unit (MAUCell) which combines Generative Adversarial Networks (GANs) and spatio-temporal attention mechanisms to improve video frame prediction capabilities. Our approach implements three types of attention models to capture intricate motion sequences. A dynamic combination of these attention outputs allows the model to reach both advanced decision accuracy along with superior quality while remaining computationally efficient. The integration of GAN elements makes generated frames appear more true to life therefore the framework creates output sequences which mimic real-world footage. The new design system maintains equilibrium between temporal continuity and spatial accuracy to deliver reliable video prediction. Through a comprehensive evaluation methodology which merged the perceptual LPIPS measurement together with classic tests MSE, MAE, SSIM and PSNR exhibited enhancing capabilities than contemporary approaches based on direct benchmark tests of Moving MNIST, KTH Action, and CASIA-B (Preprocessed) datasets. Our examination indicates that MAUCell shows promise for operational time requirements. The research findings demonstrate how GANs work best with attention mechanisms to create better applications for predicting video sequences.

Title: Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding

• Authors: Akash Kumar, Zsolt Kira, Yogesh Singh Rawat
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.17053
• Pdf URL: https://arxiv.org/pdf/2501.17053
• Copy Paste: [[2501.17053]] Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding(https://arxiv.org/abs/2501.17053)
• Keywords: foundation model
• Abstract: In this work, we focus on Weakly Supervised Spatio-Temporal Video Grounding (WSTVG). It is a multimodal task aimed at localizing specific subjects spatio-temporally based on textual queries without bounding box supervision. Motivated by recent advancements in multi-modal foundation models for grounding tasks, we first explore the potential of state-of-the-art object detection models for WSTVG. Despite their robust zero-shot capabilities, our adaptation reveals significant limitations, including inconsistent temporal predictions, inadequate understanding of complex queries, and challenges in adapting to difficult scenarios. We propose CoSPaL (Contextual Self-Paced Learning), a novel approach which is designed to overcome these limitations. CoSPaL integrates three core components: (1) Tubelet Phrase Grounding (TPG), which introduces spatio-temporal prediction by linking textual queries to tubelets; (2) Contextual Referral Grounding (CRG), which improves comprehension of complex queries by extracting contextual information to refine object identification over time; and (3) Self-Paced Scene Understanding (SPS), a training paradigm that progressively increases task difficulty, enabling the model to adapt to complex scenarios by transitioning from coarse to fine-grained understanding.

Title: DINOSTAR: Deep Iterative Neural Object Detector Self-Supervised Training for Roadside LiDAR Applications

• Authors: Muhammad Shahbaz, Shaurya Agarwal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.17076
• Pdf URL: https://arxiv.org/pdf/2501.17076
• Copy Paste: [[2501.17076]] DINOSTAR: Deep Iterative Neural Object Detector Self-Supervised Training for Roadside LiDAR Applications(https://arxiv.org/abs/2501.17076)
• Keywords: self-supervised
• Abstract: Recent advancements in deep-learning methods for object detection in point-cloud data have enabled numerous roadside applications, fostering improvements in transportation safety and management. However, the intricate nature of point-cloud data poses significant challenges for human-supervised labeling, resulting in substantial expenditures of time and capital. This paper addresses the issue by developing an end-to-end, scalable, and self-supervised framework for training deep object detectors tailored for roadside point-cloud data. The proposed framework leverages self-supervised, statistically modeled teachers to train off-the-shelf deep object detectors, thus circumventing the need for human supervision. The teacher models follow fine-tuned set standard practices of background filtering, object clustering, bounding-box fitting, and classification to generate noisy labels. It is presented that by training the student model over the combined noisy annotations from multitude of teachers enhances its capacity to discern background/foreground more effectively and forces it to learn diverse point-cloud-representations for object categories of interest. The evaluations, involving publicly available roadside datasets and state-of-art deep object detectors, demonstrate that the proposed framework achieves comparable performance to deep object detectors trained on human-annotated labels, despite not utilizing such human-annotations in its training process.

Title: Unlocking Transparent Alignment Through Enhanced Inverse Constitutional AI for Principle Extraction

• Authors: Carl-Leander Henneking, Claas Beger
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.17112
• Pdf URL: https://arxiv.org/pdf/2501.17112
• Copy Paste: [[2501.17112]] Unlocking Transparent Alignment Through Enhanced Inverse Constitutional AI for Principle Extraction(https://arxiv.org/abs/2501.17112)
• Keywords: in-context
• Abstract: Traditional methods for aligning Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on implicit principles, limiting interpretability. Constitutional AI (CAI) offers an explicit, rule-based framework for guiding model outputs. Building on this, we refine the Inverse Constitutional AI (ICAI) algorithm, which extracts constitutions from preference datasets. By improving principle generation, clustering, and embedding processes, our approach enhances the accuracy and generalizability of extracted principles across synthetic and real-world datasets. While in-context alignment yields modest improvements, our results highlight the potential of these principles to foster more transparent and adaptable alignment methods, offering a promising direction for future advancements beyond traditional fine-tuning.

Title: IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait

• Authors: Han Yang, Enis Simsar, Sotiris Anagnostidi, Yanlong Zang, Thomas Hofmann, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.17159
• Pdf URL: https://arxiv.org/pdf/2501.17159
• Copy Paste: [[2501.17159]] IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait(https://arxiv.org/abs/2501.17159)
• Keywords: diffusion, in-context
• Abstract: Existing diffusion models show great potential for identity-preserving generation. However, personalized portrait generation remains challenging due to the diversity in user profiles, including variations in appearance and lighting conditions. To address these challenges, we propose IC-Portrait, a novel framework designed to accurately encode individual identities for personalized portrait generation. Our key insight is that pre-trained diffusion models are fast learners (e.g.,100 ~ 200 steps) for in-context dense correspondence matching, which motivates the two major designs of our IC-Portrait framework. Specifically, we reformulate portrait generation into two sub-tasks: 1) Lighting-Aware Stitching: we find that masking a high proportion of the input image, e.g., 80%, yields a highly effective self-supervisory representation learning of reference image lighting. 2) View-Consistent Adaptation: we leverage a synthetic view-consistent profile dataset to learn the in-context correspondence. The reference profile can then be warped into arbitrary poses for strong spatial-aligned view conditioning. Coupling these two designs by simply concatenating latents to form ControlNet-like supervision and modeling, enables us to significantly enhance the identity preservation fidelity and stability. Extensive evaluations demonstrate that IC-Portrait consistently outperforms existing state-of-the-art methods both quantitatively and qualitatively, with particularly notable improvements in visual qualities. Furthermore, IC-Portrait even demonstrates 3D-aware relighting capabilities.

Title: CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation

• Authors: Nikolai Kalischek, Michael Oechsle, Fabian Manhardt, Philipp Henzler, Konrad Schindler, Federico Tombari
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.17162
• Pdf URL: https://arxiv.org/pdf/2501.17162
• Copy Paste: [[2501.17162]] CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation(https://arxiv.org/abs/2501.17162)
• Keywords: diffusion
• Abstract: We introduce a novel method for generating 360° panoramas from text prompts or images. Our approach leverages recent advances in 3D generation by employing multi-view diffusion models to jointly synthesize the six faces of a cubemap. Unlike previous methods that rely on processing equirectangular projections or autoregressive generation, our method treats each face as a standard perspective image, simplifying the generation process and enabling the use of existing multi-view diffusion models. We demonstrate that these models can be adapted to produce high-quality cubemaps without requiring correspondence-aware attention layers. Our model allows for fine-grained text control, generates high resolution panorama images and generalizes well beyond its training set, whilst achieving state-of-the-art results, both qualitatively and quantitatively. Project page: this https URL