2025-01-06

Title: LS-GAN: Human Motion Synthesis with Latent-space GANs

• Authors: Avinash Amballa, Gayathri Akkinapalli, Vinitra Muralikrishnan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01449
• Pdf URL: https://arxiv.org/pdf/2501.01449
• Copy Paste: [[2501.01449]] LS-GAN: Human Motion Synthesis with Latent-space GANs(https://arxiv.org/abs/2501.01449)
• Keywords: diffusion, generative
• Abstract: Human motion synthesis conditioned on textual input has gained significant attention in recent years due to its potential applications in various domains such as gaming, film production, and virtual reality. Conditioned Motion synthesis takes a text input and outputs a 3D motion corresponding to the text. While previous works have explored motion synthesis using raw motion data and latent space representations with diffusion models, these approaches often suffer from high training and inference times. In this paper, we introduce a novel framework that utilizes Generative Adversarial Networks (GANs) in the latent space to enable faster training and inference while achieving results comparable to those of the state-of-the-art diffusion methods. We perform experiments on the HumanML3D, HumanAct12 benchmarks and demonstrate that a remarkably simple GAN in the latent space achieves a FID of 0.482 with more than 91% in FLOPs reduction compared to latent diffusion model. Our work opens up new possibilities for efficient and high-quality motion synthesis using latent space GANs.

Title: Geometry Matters: Benchmarking Scientific ML Approaches for Flow Prediction around Complex Geometries

• Authors: Ali Rabeh, Ethan Herron, Aditya Balu, Soumik Sarkar, Chinmay Hegde, Adarsh Krishnamurthy, Baskar Ganapathysubramanian
• Subjects: cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2501.01453
• Pdf URL: https://arxiv.org/pdf/2501.01453
• Copy Paste: [[2501.01453]] Geometry Matters: Benchmarking Scientific ML Approaches for Flow Prediction around Complex Geometries(https://arxiv.org/abs/2501.01453)
• Keywords: foundation model
• Abstract: Rapid yet accurate simulations of fluid dynamics around complex geometries is critical in a variety of engineering and scientific applications, including aerodynamics and biomedical flows. However, while scientific machine learning (SciML) has shown promise, most studies are constrained to simple geometries, leaving complex, real-world scenarios underexplored. This study addresses this gap by benchmarking diverse SciML models, including neural operators and vision transformer-based foundation models, for fluid flow prediction over intricate geometries. Using a high-fidelity dataset of steady-state flows across various geometries, we evaluate the impact of geometric representations -- Signed Distance Fields (SDF) and binary masks -- on model accuracy, scalability, and generalization. Central to this effort is the introduction of a novel, unified scoring framework that integrates metrics for global accuracy, boundary layer fidelity, and physical consistency to enable a robust, comparative evaluation of model performance. Our findings demonstrate that foundation models significantly outperform neural operators, particularly in data-limited scenarios, and that SDF representations yield superior results with sufficient training data. Despite these advancements, all models struggle with out-of-distribution generalization, highlighting a critical challenge for future SciML applications. By advancing both evaluation methodologies and modeling capabilities, this work paves the way for robust and scalable ML solutions for fluid dynamics across complex geometries.

Title: Unraveling Indirect In-Context Learning Using Influence Functions

• Authors: Hadi Askari, Shivanshu Gupta, Terry Tong, Fei Wang, Anshuman Chhabra, Muhao Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01473
• Pdf URL: https://arxiv.org/pdf/2501.01473
• Copy Paste: [[2501.01473]] Unraveling Indirect In-Context Learning Using Influence Functions(https://arxiv.org/abs/2501.01473)
• Keywords: in-context
• Abstract: This work introduces a novel paradigm for generalized In-Context Learning (ICL), termed Indirect In-Context Learning. In Indirect ICL, we explore demonstration selection strategies tailored for two distinct real-world scenarios: Mixture of Tasks and Noisy Demonstrations. We systematically evaluate the effectiveness of Influence Functions (IFs) as a selection tool for these settings, highlighting the potential for IFs to better capture the informativeness of examples within the demonstration pool. For the Mixture of Tasks setting, demonstrations are drawn from 28 diverse tasks, including MMLU, BigBench, StrategyQA, and CommonsenseQA. We demonstrate that combining BertScore-Recall (BSR) with an IF surrogate model can significantly improve performance, leading to average absolute accuracy gains of 0.37\% and 1.45\% for 3-shot and 5-shot setups when compared to traditional ICL metrics. In the Noisy Demonstrations setting, we examine scenarios where demonstrations might be mislabeled. Our experiments show that reweighting traditional ICL selectors (BSR and Cosine Similarity) with IF-based selectors boosts accuracy by an average of 2.90\% for Cosine Similarity and 2.94\% for BSR on noisy GLUE benchmarks. In sum, we propose a robust framework for demonstration selection that generalizes beyond traditional ICL, offering valuable insights into the role of IFs for Indirect ICL.

Title: Explainable Brain Age Gap Prediction in Neurodegenerative Conditions using coVariance Neural Networks

• Authors: Saurabh Sihag, Gonzalo Mateos, Alejandro Ribeiro
• Subjects: cs.LG, eess.SP, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2501.01510
• Pdf URL: https://arxiv.org/pdf/2501.01510
• Copy Paste: [[2501.01510]] Explainable Brain Age Gap Prediction in Neurodegenerative Conditions using coVariance Neural Networks(https://arxiv.org/abs/2501.01510)
• Keywords: generative
• Abstract: Brain age is the estimate of biological age derived from neuroimaging datasets using machine learning algorithms. Increasing \textit{brain age gap} characterized by an elevated brain age relative to the chronological age can reflect increased vulnerability to neurodegeneration and cognitive decline. Hence, brain age gap is a promising biomarker for monitoring brain health. However, black-box machine learning approaches to brain age gap prediction have limited practical utility. Recent studies on coVariance neural networks (VNN) have proposed a relatively transparent deep learning pipeline for neuroimaging data analyses, which possesses two key features: (i) inherent \textit{anatomically interpretablity} of derived biomarkers; and (ii) a methodologically interpretable perspective based on \textit{linkage with eigenvectors of anatomic covariance matrix}. In this paper, we apply the VNN-based approach to study brain age gap using cortical thickness features for various prevalent neurodegenerative conditions. Our results reveal distinct anatomic patterns for brain age gap in Alzheimer's disease, frontotemporal dementia, and atypical Parkinsonian disorders. Furthermore, we demonstrate that the distinct anatomic patterns of brain age gap are linked with the differences in how VNN leverages the eigenspectrum of the anatomic covariance matrix, thus lending explainability to the reported results.

Title: SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers

• Authors: Bhavna Gopal, Huanrui Yang, Mark Horton, Yiran Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01529
• Pdf URL: https://arxiv.org/pdf/2501.01529
• Copy Paste: [[2501.01529]] SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers(https://arxiv.org/abs/2501.01529)
• Keywords: foundation model
• Abstract: Vision transformers (ViTs) have become essential backbones in advanced computer vision applications and multi-modal foundation models. Despite their strengths, ViTs remain vulnerable to adversarial perturbations, comparable to or even exceeding the vulnerability of convolutional neural networks (CNNs). Furthermore, the large parameter count and complex architecture of ViTs make them particularly prone to adversarial overfitting, often compromising both clean and adversarial accuracy. This paper mitigates adversarial overfitting in ViTs through a novel, layer-selective fine-tuning approach: SAFER. Instead of optimizing the entire model, we identify and selectively fine-tune a small subset of layers most susceptible to overfitting, applying sharpness-aware minimization to these layers while freezing the rest of the model. Our method consistently enhances both clean and adversarial accuracy over baseline approaches. Typical improvements are around 5%, with some cases achieving gains as high as 20% across various ViT architectures and datasets.

Title: BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery

• Authors: Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Louise Li, Aditi Bhaskar, Mohammed Zaman, Noah D. Goodman
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01540
• Pdf URL: https://arxiv.org/pdf/2501.01540
• Copy Paste: [[2501.01540]] BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery(https://arxiv.org/abs/2501.01540)
• Keywords: generative
• Abstract: Understanding the world and explaining it with scientific theories is a central aspiration of artificial intelligence research. Proposing theories, designing experiments to test them, and then revising them based on data are fundamental to scientific discovery. Despite the significant promise of LLM-based scientific agents, no benchmarks systematically test LLM's ability to propose scientific models, collect experimental data, and revise them in light of new data. We introduce BoxingGym, a benchmark with 10 environments for systematically evaluating both experimental design (e.g. collecting data to test a scientific theory) and model discovery (e.g. proposing and revising scientific theories). To enable tractable and quantitative evaluation, we implement each environment as a generative probabilistic model with which a scientific agent can run interactive experiments. These probabilistic models are drawn from various real-world scientific domains ranging from psychology to ecology. To quantitatively evaluate a scientific agent's ability to collect informative experimental data, we compute the expected information gain (EIG), an information-theoretic quantity which measures how much an experiment reduces uncertainty about the parameters of a generative model. A good scientific theory is a concise and predictive explanation. Therefore, to quantitatively evaluate model discovery, we ask a scientific agent to explain their model and then assess whether this explanation enables another scientific agent to make reliable predictions about this environment. In addition to this explanation-based evaluation, we compute standard model evaluation metrics such as prediction errors. We find that current LLMs, such as GPT-4o, struggle with both experimental design and model discovery. We find that augmenting the LLM-based agent with an explicit statistical model does not reliably improve these results.

Title: Multivariate Time Series Anomaly Detection using DiffGAN Model

• Authors: Guangqiang Wu, Fu Zhang
• Subjects: cs.LG, math.ST
• Abstract URL: https://arxiv.org/abs/2501.01591
• Pdf URL: https://arxiv.org/pdf/2501.01591
• Copy Paste: [[2501.01591]] Multivariate Time Series Anomaly Detection using DiffGAN Model(https://arxiv.org/abs/2501.01591)
• Keywords: diffusion, generative, anomaly
• Abstract: In recent years, some researchers have applied diffusion models to multivariate time series anomaly detection. The partial diffusion strategy, which depends on the diffusion steps, is commonly used for anomaly detection in these models. However, different diffusion steps have an impact on the reconstruction of the original data, thereby impacting the effectiveness of anomaly detection. To address this issue, we propose a novel method named DiffGAN, which adds a generative adversarial network component to the denoiser of diffusion model. This addition allows for the simultaneous generation of noisy data and prediction of diffusion steps. Compared to multiple state-of-the-art reconstruction models, experimental results demonstrate that DiffGAN achieves superior performance in anomaly detection.

Title: Few-shot Implicit Function Generation via Equivariance

• Authors: Suizhi Huang, Xingyi Yang, Hongtao Lu, Xinchao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01601
• Pdf URL: https://arxiv.org/pdf/2501.01601
• Copy Paste: [[2501.01601]] Few-shot Implicit Function Generation via Equivariance(https://arxiv.org/abs/2501.01601)
• Keywords: diffusion
• Abstract: Implicit Neural Representations (INRs) have emerged as a powerful framework for representing continuous signals. However, generating diverse INR weights remains challenging due to limited training data. We introduce Few-shot Implicit Function Generation, a new problem setup that aims to generate diverse yet functionally consistent INR weights from only a few examples. This is challenging because even for the same signal, the optimal INRs can vary significantly depending on their initializations. To tackle this, we propose EquiGen, a framework that can generate new INRs from limited data. The core idea is that functionally similar networks can be transformed into one another through weight permutations, forming an equivariance group. By projecting these weights into an equivariant latent space, we enable diverse generation within these groups, even with few examples. EquiGen implements this through an equivariant encoder trained via contrastive learning and smooth augmentation, an equivariance-guided diffusion process, and controlled perturbations in the equivariant subspace. Experiments on 2D image and 3D shape INR datasets demonstrate that our approach effectively generates diverse INR weights while preserving their functional properties in few-shot scenarios.

Title: ICPC: In-context Prompt Compression with Faster Inference

• Authors: Ziyang Yu, Yuyu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01625
• Pdf URL: https://arxiv.org/pdf/2501.01625
• Copy Paste: [[2501.01625]] ICPC: In-context Prompt Compression with Faster Inference(https://arxiv.org/abs/2501.01625)
• Keywords: in-context
• Abstract: Despite the recent success of Large Language Models (LLMs), it remains challenging to feed LLMs with long prompts due to the fixed size of LLM inputs. As a remedy, prompt compression becomes a promising solution by removing redundant tokens in the prompt. However, using LLM in the existing works requires additional computation resources and leads to memory overheads. To address it, we propose ICPC (In-context Prompt Compression), a novel and scalable prompt compression method that adaptively reduces the prompt length. The key idea of ICPC is to calculate the probability of each word appearing in the prompt using encoders and calculate information carried by each word through the information function, which effectively reduces the information loss during prompt compression and increases the speed of compression. Empirically, we demonstrate that ICPC can effectively compress long texts of different categories and thus achieve better performance and speed on different types of NLP tasks.

Title: A Probabilistic Model for Node Classification in Directed Graphs

• Authors: Diego Huerta, Gerardo Arizmendi
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.01630
• Pdf URL: https://arxiv.org/pdf/2501.01630
• Copy Paste: [[2501.01630]] A Probabilistic Model for Node Classification in Directed Graphs(https://arxiv.org/abs/2501.01630)
• Keywords: generative
• Abstract: In this work, we present a probabilistic model for directed graphs where nodes have attributes and labels. This model serves as a generative classifier capable of predicting the labels of unseen nodes using either maximum likelihood or maximum a posteriori estimations. The predictions made by this model are highly interpretable, contrasting with some common methods for node classification, such as graph neural networks. We applied the model to two datasets, demonstrating predictive performance that is competitive with, and even superior to, state-of-the-art methods. One of the datasets considered is adapted from the Math Genealogy Project, which has not previously been utilized for this purpose. Consequently, we evaluated several classification algorithms on this dataset to compare the performance of our model and provide benchmarks for this new resource.

Title: ACE: Anti-Editing Concept Erasure in Text-to-Image Models

• Authors: Zihao Wang, Yuxiang Wei, Fan Li, Renjing Pei, Hang Xu, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01633
• Pdf URL: https://arxiv.org/pdf/2501.01633
• Copy Paste: [[2501.01633]] ACE: Anti-Editing Concept Erasure in Text-to-Image Models(https://arxiv.org/abs/2501.01633)
• Keywords: diffusion
• Abstract: Recent advance in text-to-image diffusion models have significantly facilitated the generation of high-quality images, but also raising concerns about the illegal creation of harmful content, such as copyrighted images. Existing concept erasure methods achieve superior results in preventing the production of erased concept from prompts, but typically perform poorly in preventing undesired editing. To address this issue, we propose an Anti-Editing Concept Erasure (ACE) method, which not only erases the target concept during generation but also filters out it during editing. Specifically, we propose to inject the erasure guidance into both conditional and the unconditional noise prediction, enabling the model to effectively prevent the creation of erasure concepts during both editing and generation. Furthermore, a stochastic correction guidance is introduced during training to address the erosion of unrelated concepts. We conducted erasure editing experiments with representative editing methods (i.e., LEDITS++ and MasaCtrl) to erase IP characters, and the results indicate that our ACE effectively filters out target concepts in both types of edits. Additional experiments on erasing explicit concepts and artistic styles further demonstrate that our ACE performs favorably against state-of-the-art methods. Our code will be publicly available at this https URL.

Title: Uncertainty and Energy based Loss Guided Semi-Supervised Semantic Segmentation

• Authors: Rini Smita Thakur, Vinod K. Kurmi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01640
• Pdf URL: https://arxiv.org/pdf/2501.01640
• Copy Paste: [[2501.01640]] Uncertainty and Energy based Loss Guided Semi-Supervised Semantic Segmentation(https://arxiv.org/abs/2501.01640)
• Keywords: generative
• Abstract: Semi-supervised (SS) semantic segmentation exploits both labeled and unlabeled images to overcome tedious and costly pixel-level annotation problems. Pseudolabel supervision is one of the core approaches of training networks with both pseudo labels and ground-truth labels. This work uses aleatoric or data uncertainty and energy based modeling in intersection-union pseudo supervised this http URL aleatoric uncertainty is modeling the inherent noise variations of the data in a network with two predictive branches. The per-pixel variance parameter obtained from the network gives a quantitative idea about the data uncertainty. Moreover, energy-based loss realizes the potential of generative modeling on the downstream SS segmentation task. The aleatoric and energy loss are applied in conjunction with pseudo-intersection labels, pseudo-union labels, and ground-truth on the respective network branch. The comparative analysis with state-of-the-art methods has shown improvement in performance metrics.

Title: Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning

• Authors: Danni Peng, Yuan Wang, Huazhu Fu, Jinpeng Jiang, Yong Liu, Rick Siow Mong Goh, Qingsong Wei
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.01653
• Pdf URL: https://arxiv.org/pdf/2501.01653
• Copy Paste: [[2501.01653]] Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning(https://arxiv.org/abs/2501.01653)
• Keywords: foundation model
• Abstract: Personalized federated learning (PFL) studies effective model personalization to address the data heterogeneity issue among clients in traditional federated learning (FL). Existing PFL approaches mainly generate personalized models by relying solely on the clients' latest updated models while ignoring their previous updates, which may result in suboptimal personalized model learning. To bridge this gap, we propose a novel framework termed pFedSeq, designed for personalizing adapters to fine-tune a foundation model in FL. In pFedSeq, the server maintains and trains a sequential learner, which processes a sequence of past adapter updates from clients and generates calibrations for personalized adapters. To effectively capture the cross-client and cross-step relations hidden in previous updates and generate high-performing personalized adapters, pFedSeq adopts the powerful selective state space model (SSM) as the architecture of sequential learner. Through extensive experiments on four public benchmark datasets, we demonstrate the superiority of pFedSeq over state-of-the-art PFL methods.

Title: Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models

• Authors: Lei Tang, Jinghui Qin, Wenxuan Ye, Hao Tan, Zhijing Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01679
• Pdf URL: https://arxiv.org/pdf/2501.01679
• Copy Paste: [[2501.01679]] Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models(https://arxiv.org/abs/2501.01679)
• Keywords: in-context
• Abstract: Recently, Large language models (LLMs) with in-context learning have demonstrated remarkable potential in handling neural machine translation. However, existing evidence shows that LLMs are prompt-sensitive and it is sub-optimal to apply the fixed prompt to any input for downstream machine translation tasks. To address this issue, we propose an adaptive few-shot prompting (AFSP) framework to automatically select suitable translation demonstrations for various source input sentences to further elicit the translation capability of an LLM for better machine translation. First, we build a translation demonstration retrieval module based on LLM's embedding to retrieve top-k semantic-similar translation demonstrations from aligned parallel translation corpus. Rather than using other embedding models for semantic demonstration retrieval, we build a hybrid demonstration retrieval module based on the embedding layer of the deployed LLM to build better input representation for retrieving more semantic-related translation demonstrations. Then, to ensure better semantic consistency between source inputs and target outputs, we force the deployed LLM itself to generate multiple output candidates in the target language with the help of translation demonstrations and rerank these candidates. Besides, to better evaluate the effectiveness of our AFSP framework on the latest language and extend the research boundary of neural machine translation, we construct a high-quality diplomatic Chinese-English parallel dataset that consists of 5,528 parallel Chinese-English sentences. Finally, extensive experiments on the proposed diplomatic Chinese-English parallel dataset and the United Nations Parallel Corpus (Chinese-English part) show the effectiveness and superiority of our proposed AFSP.

Title: MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders

• Authors: Jiajun Cao, Yuan Zhang, Tao Huang, Ming Lu, Qizhe Zhang, Ruichuan An, Ningning MA, Shanghang Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01709
• Pdf URL: https://arxiv.org/pdf/2501.01709
• Copy Paste: [[2501.01709]] MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders(https://arxiv.org/abs/2501.01709)
• Keywords: foundation model
• Abstract: Visual encoders are fundamental components in vision-language models (VLMs), each showcasing unique strengths derived from various pre-trained visual foundation models. To leverage the various capabilities of these encoders, recent studies incorporate multiple encoders within a single VLM, leading to a considerable increase in computational cost. In this paper, we present Mixture-of-Visual-Encoder Knowledge Distillation (MoVE-KD), a novel framework that distills the unique proficiencies of multiple vision encoders into a single, efficient encoder model. Specifically, to mitigate conflicts and retain the unique characteristics of each teacher encoder, we employ low-rank adaptation (LoRA) and mixture-of-experts (MoEs) to selectively activate specialized knowledge based on input features, enhancing both adaptability and efficiency. To regularize the KD process and enhance performance, we propose an attention-based distillation strategy that adaptively weighs the different visual encoders and emphasizes valuable visual tokens, reducing the burden of replicating comprehensive but distinct features from multiple teachers. Comprehensive experiments on popular VLMs, such as LLaVA and LLaVA-NeXT, validate the effectiveness of our method. The code will be released.

Title: AR4D: Autoregressive 4D Generation from Monocular Videos

• Authors: Hanxin Zhu, Tianyu He, Xiqian Yu, Junliang Guo, Zhibo Chen, Jiang Bian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01722
• Pdf URL: https://arxiv.org/pdf/2501.01722
• Copy Paste: [[2501.01722]] AR4D: Autoregressive 4D Generation from Monocular Videos(https://arxiv.org/abs/2501.01722)
• Keywords: generative
• Abstract: Recent advancements in generative models have ignited substantial interest in dynamic 3D content creation (\ie, 4D generation). Existing approaches primarily rely on Score Distillation Sampling (SDS) to infer novel-view videos, typically leading to issues such as limited diversity, spatial-temporal inconsistency and poor prompt alignment, due to the inherent randomness of SDS. To tackle these problems, we propose AR4D, a novel paradigm for SDS-free 4D generation. Specifically, our paradigm consists of three stages. To begin with, for a monocular video that is either generated or captured, we first utilize pre-trained expert models to create a 3D representation of the first frame, which is further fine-tuned to serve as the canonical space. Subsequently, motivated by the fact that videos happen naturally in an autoregressive manner, we propose to generate each frame's 3D representation based on its previous frame's representation, as this autoregressive generation manner can facilitate more accurate geometry and motion estimation. Meanwhile, to prevent overfitting during this process, we introduce a progressive view sampling strategy, utilizing priors from pre-trained large-scale 3D reconstruction models. To avoid appearance drift introduced by autoregressive generation, we further incorporate a refinement stage based on a global deformation field and the geometry of each frame's 3D representation. Extensive experiments have demonstrated that AR4D can achieve state-of-the-art 4D generation without SDS, delivering greater diversity, improved spatial-temporal consistency, and better alignment with input prompts.

Title: Adverse Weather Conditions Augmentation of LiDAR Scenes with Latent Diffusion Models

• Authors: Andrea Matteazzi, Pascal Colling, Michael Arnold, Dietmar Tutsch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01761
• Pdf URL: https://arxiv.org/pdf/2501.01761
• Copy Paste: [[2501.01761]] Adverse Weather Conditions Augmentation of LiDAR Scenes with Latent Diffusion Models(https://arxiv.org/abs/2501.01761)
• Keywords: diffusion, generative
• Abstract: LiDAR scenes constitute a fundamental source for several autonomous driving applications. Despite the existence of several datasets, scenes from adverse weather conditions are rarely available. This limits the robustness of downstream machine learning models, and restrains the reliability of autonomous driving systems in particular locations and seasons. Collecting feature-diverse scenes under adverse weather conditions is challenging due to seasonal limitations. Generative models are therefore essentials, especially for generating adverse weather conditions for specific driving scenarios. In our work, we propose a latent diffusion process constituted by autoencoder and latent diffusion models. Moreover, we leverage the clear condition LiDAR scenes with a postprocessing step to improve the realism of the generated adverse weather condition scenes.

Title: LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extraction

• Authors: Er Jin, Qihui Feng, Yongli Mou, Stefan Decker, Gerhard Lakemeyer, Oliver Simons, Johannes Stegmaier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01767
• Pdf URL: https://arxiv.org/pdf/2501.01767
• Copy Paste: [[2501.01767]] LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extraction(https://arxiv.org/abs/2501.01767)
• Keywords: anomaly
• Abstract: Logical image understanding involves interpreting and reasoning about the relationships and consistency within an image's visual content. This capability is essential in applications such as industrial inspection, where logical anomaly detection is critical for maintaining high-quality standards and minimizing costly recalls. Previous research in anomaly detection (AD) has relied on prior knowledge for designing algorithms, which often requires extensive manual annotations, significant computing power, and large amounts of data for training. Autoregressive, multimodal Vision Language Models (AVLMs) offer a promising alternative due to their exceptional performance in visual reasoning across various domains. Despite this, their application to logical AD remains unexplored. In this work, we investigate using AVLMs for logical AD and demonstrate that they are well-suited to the task. Combining AVLMs with format embedding and a logic reasoner, we achieve SOTA performance on public benchmarks, MVTec LOCO AD, with an AUROC of 86.0% and F1-max of 83.7%, along with explanations of anomalies. This significantly outperforms the existing SOTA method by a large margin.

Title: Ingredients: Blending Custom Photos with Video Diffusion Transformers

• Authors: Zhengcong Fei, Debang Li, Di Qiu, Changqian Yu, Mingyuan Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01790
• Pdf URL: https://arxiv.org/pdf/2501.01790
• Copy Paste: [[2501.01790]] Ingredients: Blending Custom Photos with Video Diffusion Transformers(https://arxiv.org/abs/2501.01790)
• Keywords: diffusion, generative
• Abstract: This paper presents a powerful framework to customize video creations by incorporating multiple specific identity (ID) photos, with video diffusion Transformers, referred to as \texttt{Ingredients}. Generally, our method consists of three primary modules: (\textbf{i}) a facial extractor that captures versatile and precise facial features for each human ID from both global and local perspectives; (\textbf{ii}) a multi-scale projector that maps face embeddings into the contextual space of image query in video diffusion transformers; (\textbf{iii}) an ID router that dynamically combines and allocates multiple ID embedding to the corresponding space-time regions. Leveraging a meticulously curated text-video dataset and a multi-stage training protocol, \texttt{Ingredients} demonstrates superior performance in turning custom photos into dynamic and personalized video content. Qualitative evaluations highlight the advantages of proposed method, positioning it as a significant advancement toward more effective generative video control tools in Transformer-based architecture, compared to existing methods. The data, code, and model weights are publicly available at: \url{this https URL}.

Title: Creating Artificial Students that Never Existed: Leveraging Large Language Models and CTGANs for Synthetic Data Generation

• Authors: Mohammad Khalil, Farhad Vadiee, Ronas Shakya, Qinyi Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01793
• Pdf URL: https://arxiv.org/pdf/2501.01793
• Copy Paste: [[2501.01793]] Creating Artificial Students that Never Existed: Leveraging Large Language Models and CTGANs for Synthetic Data Generation(https://arxiv.org/abs/2501.01793)
• Keywords: generative
• Abstract: In this study, we explore the growing potential of AI and deep learning technologies, particularly Generative Adversarial Networks (GANs) and Large Language Models (LLMs), for generating synthetic tabular data. Access to quality students data is critical for advancing learning analytics, but privacy concerns and stricter data protection regulations worldwide limit their availability and usage. Synthetic data offers a promising alternative. We investigate whether synthetic data can be leveraged to create artificial students for serving learning analytics models. Using the popular GAN model CTGAN and three LLMs- GPT2, DistilGPT2, and DialoGPT, we generate synthetic tabular student data. Our results demonstrate the strong potential of these methods to produce high-quality synthetic datasets that resemble real students data. To validate our findings, we apply a comprehensive set of utility evaluation metrics to assess the statistical and predictive performance of the synthetic data and compare the different generator models used, specially the performance of LLMs. Our study aims to provide the learning analytics community with valuable insights into the use of synthetic data, laying the groundwork for expanding the field methodological toolbox with new innovative approaches for learning analytics data generation.

Title: Time Series Language Model for Descriptive Caption Generation

• Authors: Mohamed Trabelsi, Aidan Boyd, Jin Cao, Huseyin Uzunalioglu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01832
• Pdf URL: https://arxiv.org/pdf/2501.01832
• Copy Paste: [[2501.01832]] Time Series Language Model for Descriptive Caption Generation(https://arxiv.org/abs/2501.01832)
• Keywords: foundation model, in-context
• Abstract: The automatic generation of representative natural language descriptions for observable patterns in time series data enhances interpretability, simplifies analysis and increases cross-domain utility of temporal data. While pre-trained foundation models have made considerable progress in natural language processing (NLP) and computer vision (CV), their application to time series analysis has been hindered by data scarcity. Although several large language model (LLM)-based methods have been proposed for time series forecasting, time series captioning is under-explored in the context of LLMs. In this paper, we introduce TSLM, a novel time series language model designed specifically for time series captioning. TSLM operates as an encoder-decoder model, leveraging both text prompts and time series data representations to capture subtle temporal patterns across multiple phases and generate precise textual descriptions of time series inputs. TSLM addresses the data scarcity problem in time series captioning by first leveraging an in-context prompting synthetic data generation, and second denoising the generated data via a novel cross-modal dense retrieval scoring applied to time series-caption pairs. Experimental findings on various time series captioning datasets demonstrate that TSLM outperforms existing state-of-the-art approaches from multiple data modalities by a significant margin.

Title: GoBERT: Gene Ontology Graph Informed BERT for Universal Gene Function Prediction

• Authors: Yuwei Miao, Yuzhi Guo, Hehuan Ma, Jingquan Yan, Feng Jiang, Rui Liao, Junzhou Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01930
• Pdf URL: https://arxiv.org/pdf/2501.01930
• Copy Paste: [[2501.01930]] GoBERT: Gene Ontology Graph Informed BERT for Universal Gene Function Prediction(https://arxiv.org/abs/2501.01930)
• Keywords: self-supervised
• Abstract: Exploring the functions of genes and gene products is crucial to a wide range of fields, including medical research, evolutionary biology, and environmental science. However, discovering new functions largely relies on expensive and exhaustive wet lab experiments. Existing methods of automatic function annotation or prediction mainly focus on protein function prediction with sequence, 3D-structures or protein family information. In this study, we propose to tackle the gene function prediction problem by exploring Gene Ontology graph and annotation with BERT (GoBERT) to decipher the underlying relationships among gene functions. Our proposed novel function prediction task utilizes existing functions as inputs and generalizes the function prediction to gene and gene products. Specifically, two pre-train tasks are designed to jointly train GoBERT to capture both explicit and implicit relations of functions. Neighborhood prediction is a self-supervised multi-label classification task that captures the explicit function relations. Specified masking and recovering task helps GoBERT in finding implicit patterns among functions. The pre-trained GoBERT possess the ability to predict novel functions for various gene and gene products based on known functional annotations. Extensive experiments, biological case studies, and ablation studies are conducted to demonstrate the superiority of our proposed GoBERT.

Title: Bridging Classification and Segmentation in Osteosarcoma Assessment via Foundation and Discrete Diffusion Models

• Authors: Manh Duong Nguyen, Dac Thai Nguyen, Trung Viet Nguyen, Homi Yamada, Huy Hieu Pham, Phi Le Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.01932
• Pdf URL: https://arxiv.org/pdf/2501.01932
• Copy Paste: [[2501.01932]] Bridging Classification and Segmentation in Osteosarcoma Assessment via Foundation and Discrete Diffusion Models(https://arxiv.org/abs/2501.01932)
• Keywords: diffusion, foundation model
• Abstract: Osteosarcoma, the most common primary bone cancer, often requires accurate necrosis assessment from whole slide images (WSIs) for effective treatment planning and prognosis. However, manual assessments are subjective and prone to variability. In response, we introduce FDDM, a novel framework bridging the gap between patch classification and region-based segmentation. FDDM operates in two stages: patch-based classification, followed by region-based refinement, enabling cross-patch information intergation. Leveraging a newly curated dataset of osteosarcoma images, FDDM demonstrates superior segmentation performance, achieving up to a 10% improvement mIOU and a 32.12% enhancement in necrosis rate estimation over state-of-the-art methods. This framework sets a new benchmark in osteosarcoma assessment, highlighting the potential of foundation models and diffusion-based refinements in complex medical imaging tasks.

Title: MADGEN -- Mass-Spec attends to De Novo Molecular generation

• Authors: Yinkai Wang, Xiaohui Chen, Liping Liu, Soha Hassoun
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01950
• Pdf URL: https://arxiv.org/pdf/2501.01950
• Copy Paste: [[2501.01950]] MADGEN -- Mass-Spec attends to De Novo Molecular generation(https://arxiv.org/abs/2501.01950)
• Keywords: generative
• Abstract: The annotation (assigning structural chemical identities) of MS/MS spectra remains a significant challenge due to the enormous molecular diversity in biological samples and the limited scope of reference databases. Currently, the vast majority of spectral measurements remain in the "dark chemical space" without structural annotations. To improve annotation, we propose MADGEN (Mass-spec Attends to De Novo Molecular GENeration), a scaffold-based method for de novo molecular structure generation guided by mass spectrometry data. MADGEN operates in two stages: scaffold retrieval and spectra-conditioned molecular generation starting with the scaffold. In the first stage, given an MS/MS spectrum, we formulate scaffold retrieval as a ranking problem and employ contrastive learning to align mass spectra with candidate molecular scaffolds. In the second stage, starting from the retrieved scaffold, we employ the MS/MS spectrum to guide an attention-based generative model to generate the final molecule. Our approach constrains the molecular generation search space, reducing its complexity and improving generation accuracy. We evaluate MADGEN on three datasets (NIST23, CANOPUS, and MassSpecGym) and evaluate MADGEN's performance with a predictive scaffold retriever and with an oracle retriever. We demonstrate the effectiveness of using attention to integrate spectral information throughout the generation process to achieve strong results with the oracle retriever.