2024-09-20

Title: ARTICLE: Annotator Reliability Through In-Context Learning

• Authors: Sujan Dutta, Deepak Pandita, Tharindu Cyril Weerasooriya, Marcos Zampieri, Christopher M. Homan, Ashiqur R. KhudaBukhsh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12218
• Pdf URL: https://arxiv.org/pdf/2409.12218
• Copy Paste: [[2409.12218]] ARTICLE: Annotator Reliability Through In-Context Learning(https://arxiv.org/abs/2409.12218)
• Keywords: in-context
• Abstract: Ensuring annotator quality in training and evaluation data is a key piece of machine learning in NLP. Tasks such as sentiment analysis and offensive speech detection are intrinsically subjective, creating a challenging scenario for traditional quality assessment approaches because it is hard to distinguish disagreement due to poor work from that due to differences of opinions between sincere annotators. With the goal of increasing diverse perspectives in annotation while ensuring consistency, we propose \texttt{ARTICLE}, an in-context learning (ICL) framework to estimate annotation quality through self-consistency. We evaluate this framework on two offensive speech datasets using multiple LLMs and compare its performance with traditional methods. Our findings indicate that \texttt{ARTICLE} can be used as a robust method for identifying reliable annotators, hence improving data quality.

Title: Sparks of Artificial General Intelligence(AGI) in Semiconductor Material Science: Early Explorations into the Next Frontier of Generative AI-Assisted Electron Micrograph Analysis

• Authors: Sakhinana Sagar Srinivas, Geethan Sannidhi, Sreeja Gangasani, Chidaksh Ravuru, Venkataramana Runkana
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12244
• Pdf URL: https://arxiv.org/pdf/2409.12244
• Copy Paste: [[2409.12244]] Sparks of Artificial General Intelligence(AGI) in Semiconductor Material Science: Early Explorations into the Next Frontier of Generative AI-Assisted Electron Micrograph Analysis(https://arxiv.org/abs/2409.12244)
• Keywords: generative, in-context
• Abstract: Characterizing materials with electron micrographs poses significant challenges for automated labeling due to the complex nature of nanomaterial structures. To address this, we introduce a fully automated, end-to-end pipeline that leverages recent advances in Generative AI. It is designed for analyzing and understanding the microstructures of semiconductor materials with effectiveness comparable to that of human experts, contributing to the pursuit of Artificial General Intelligence (AGI) in nanomaterial identification. Our approach utilizes Large MultiModal Models (LMMs) such as GPT-4V, alongside text-to-image models like DALLE-3. We integrate a GPT-4 guided Visual Question Answering (VQA) method to analyze nanomaterial images, generate synthetic nanomaterial images via DALLE-3, and employ in-context learning with few-shot prompting in GPT-4V for accurate nanomaterial identification. Our method surpasses traditional techniques by enhancing the precision of nanomaterial identification and optimizing the process for high-throughput screening.

Title: User-friendly Foundation Model Adapters for Multivariate Time Series Classification

• Authors: Vasilii Feofanov, Romain Ilbert, Malik Tiomoko, Themis Palpanas, Ievgen Redko
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.12264
• Pdf URL: https://arxiv.org/pdf/2409.12264
• Copy Paste: [[2409.12264]] User-friendly Foundation Model Adapters for Multivariate Time Series Classification(https://arxiv.org/abs/2409.12264)
• Keywords: foundation model
• Abstract: Foundation models, while highly effective, are often resource-intensive, requiring substantial inference time and memory. This paper addresses the challenge of making these models more accessible with limited computational resources by exploring dimensionality reduction techniques. Our goal is to enable users to run large pre-trained foundation models on standard GPUs without sacrificing performance. We investigate classical methods such as Principal Component Analysis alongside neural network-based adapters, aiming to reduce the dimensionality of multivariate time series data while preserving key features. Our experiments show up to a 10x speedup compared to the baseline model, without performance degradation, and enable up to 4.5x more datasets to fit on a single GPU, paving the way for more user-friendly and scalable foundation models.

Title: Provable In-Context Learning of Linear Systems and Linear Elliptic PDEs with Transformers

• Authors: Frank Cole, Yulong Lu, Riley O'Neill, Tianhao Zhang
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.12293
• Pdf URL: https://arxiv.org/pdf/2409.12293
• Copy Paste: [[2409.12293]] Provable In-Context Learning of Linear Systems and Linear Elliptic PDEs with Transformers(https://arxiv.org/abs/2409.12293)
• Keywords: foundation model, in-context
• Abstract: Foundation models for natural language processing, powered by the transformer architecture, exhibit remarkable in-context learning (ICL) capabilities, allowing pre-trained models to adapt to downstream tasks using few-shot prompts without updating their weights. Recently, transformer-based foundation models have also emerged as versatile tools for solving scientific problems, particularly in the realm of partial differential equations (PDEs). However, the theoretical foundations of the ICL capabilities in these scientific models remain largely unexplored. This work develops a rigorous error analysis for transformer-based ICL applied to solution operators associated with a family of linear elliptic PDEs. We first demonstrate that a linear transformer, defined by a linear self-attention layer, can provably learn in-context to invert linear systems arising from the spatial discretization of PDEs. This is achieved by deriving theoretical scaling laws for the prediction risk of the proposed linear transformers in terms of spatial discretization size, the number of training tasks, and the lengths of prompts used during training and inference. These scaling laws also enable us to establish quantitative error bounds for learning PDE solutions. Furthermore, we quantify the adaptability of the pre-trained transformer on downstream PDE tasks that experience distribution shifts in both tasks (represented by PDE coefficients) and input covariates (represented by the source term). To analyze task distribution shifts, we introduce a novel concept of task diversity and characterize the transformer's prediction error in terms of the magnitude of task shift, assuming sufficient diversity in the pre-training tasks. We also establish sufficient conditions to ensure task diversity. Finally, we validate the ICL-capabilities of transformers through extensive numerical experiments.

Title: Self-Supervised Pre-training Tasks for an fMRI Time-series Transformer in Autism Detection

• Authors: Yinchi Zhou, Peiyu Duan, Yuexi Du, Nicha C. Dvornek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12304
• Pdf URL: https://arxiv.org/pdf/2409.12304
• Copy Paste: [[2409.12304]] Self-Supervised Pre-training Tasks for an fMRI Time-series Transformer in Autism Detection(https://arxiv.org/abs/2409.12304)
• Keywords: self-supervised
• Abstract: Autism Spectrum Disorder (ASD) is a neurodevelopmental condition that encompasses a wide variety of symptoms and degrees of impairment, which makes the diagnosis and treatment challenging. Functional magnetic resonance imaging (fMRI) has been extensively used to study brain activity in ASD, and machine learning methods have been applied to analyze resting state fMRI (rs-fMRI) data. However, fewer studies have explored the recent transformer-based models on rs-fMRI data. Given the superiority of transformer models in capturing long-range dependencies in sequence data, we have developed a transformer-based self-supervised framework that directly analyzes time-series fMRI data without computing functional connectivity. To address over-fitting in small datasets and enhance the model performance, we propose self-supervised pre-training tasks to reconstruct the randomly masked fMRI time-series data, investigating the effects of various masking strategies. We then finetune the model for the ASD classification task and evaluate it using two public datasets and five-fold cross-validation with different amounts of training data. The experiments show that randomly masking entire ROIs gives better model performance than randomly masking time points in the pre-training step, resulting in an average improvement of 10.8% for AUC and 9.3% for subject accuracy compared with the transformer model trained from scratch across different levels of training data availability. Our code is available on GitHub.

Title: Understanding Implosion in Text-to-Image Generative Models

• Authors: Wenxin Ding, Cathy Y. Li, Shawn Shan, Ben Y. Zhao, Haitao Zheng
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12314
• Pdf URL: https://arxiv.org/pdf/2409.12314
• Copy Paste: [[2409.12314]] Understanding Implosion in Text-to-Image Generative Models(https://arxiv.org/abs/2409.12314)
• Keywords: diffusion, generative
• Abstract: Recent works show that text-to-image generative models are surprisingly vulnerable to a variety of poisoning attacks. Empirical results find that these models can be corrupted by altering associations between individual text prompts and associated visual features. Furthermore, a number of concurrent poisoning attacks can induce "model implosion," where the model becomes unable to produce meaningful images for unpoisoned prompts. These intriguing findings highlight the absence of an intuitive framework to understand poisoning attacks on these models. In this work, we establish the first analytical framework on robustness of image generative models to poisoning attacks, by modeling and analyzing the behavior of the cross-attention mechanism in latent diffusion models. We model cross-attention training as an abstract problem of "supervised graph alignment" and formally quantify the impact of training data by the hardness of alignment, measured by an Alignment Difficulty (AD) metric. The higher the AD, the harder the alignment. We prove that AD increases with the number of individual prompts (or concepts) poisoned. As AD grows, the alignment task becomes increasingly difficult, yielding highly distorted outcomes that frequently map meaningful text prompts to undefined or meaningless visual representations. As a result, the generative model implodes and outputs random, incoherent images at large. We validate our analytical framework through extensive experiments, and we confirm and explain the unexpected (and unexplained) effect of model implosion while producing new, unforeseen insights. Our work provides a useful tool for studying poisoning attacks against diffusion models and their defenses.

Title: Depth Estimation Based on 3D Gaussian Splatting Siamese Defocus

• Authors: Jinchang Zhang, Ningning Xu, Hao Zhang, Guoyu Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12323
• Pdf URL: https://arxiv.org/pdf/2409.12323
• Copy Paste: [[2409.12323]] Depth Estimation Based on 3D Gaussian Splatting Siamese Defocus(https://arxiv.org/abs/2409.12323)
• Keywords: self-supervised
• Abstract: Depth estimation is a fundamental task in 3D geometry. While stereo depth estimation can be achieved through triangulation methods, it is not as straightforward for monocular methods, which require the integration of global and local information. The Depth from Defocus (DFD) method utilizes camera lens models and parameters to recover depth information from blurred images and has been proven to perform well. However, these methods rely on All-In-Focus (AIF) images for depth estimation, which is nearly impossible to obtain in real-world applications. To address this issue, we propose a self-supervised framework based on 3D Gaussian splatting and Siamese networks. By learning the blur levels at different focal distances of the same scene in the focal stack, the framework predicts the defocus map and Circle of Confusion (CoC) from a single defocused image, using the defocus map as input to DepthNet for monocular depth estimation. The 3D Gaussian splatting model renders defocused images using the predicted CoC, and the differences between these and the real defocused images provide additional supervision signals for the Siamese Defocus self-supervised network. This framework has been validated on both artificially synthesized and real blurred datasets. Subsequent quantitative and visualization experiments demonstrate that our proposed framework is highly effective as a DFD method.

Title: Look Through Masks: Towards Masked Face Recognition with De-Occlusion Distillation

• Authors: Chenyu Li, Shiming Ge, Daichi Zhang, Jia Li
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12385
• Pdf URL: https://arxiv.org/pdf/2409.12385
• Copy Paste: [[2409.12385]] Look Through Masks: Towards Masked Face Recognition with De-Occlusion Distillation(https://arxiv.org/abs/2409.12385)
• Keywords: generative
• Abstract: Many real-world applications today like video surveillance and urban governance need to address the recognition of masked faces, where content replacement by diverse masks often brings in incomplete appearance and ambiguous representation, leading to a sharp drop in accuracy. Inspired by recent progress on amodal perception, we propose to migrate the mechanism of amodal completion for the task of masked face recognition with an end-to-end de-occlusion distillation framework, which consists of two modules. The \textit{de-occlusion} module applies a generative adversarial network to perform face completion, which recovers the content under the mask and eliminates appearance ambiguity. The \textit{distillation} module takes a pre-trained general face recognition model as the teacher and transfers its knowledge to train a student for completed faces using massive online synthesized face pairs. Especially, the teacher knowledge is represented with structural relations among instances in multiple orders, which serves as a posterior regularization to enable the adaptation. In this way, the knowledge can be fully distilled and transferred to identify masked faces. Experiments on synthetic and realistic datasets show the efficacy of the proposed approach.

Title: Frequency-Guided Spatial Adaptation for Camouflaged Object Detection

• Authors: Shizhou Zhang, Dexuan Kong, Yinghui Xing, Yue Lu, Lingyan Ran, Guoqiang Liang, Hexu Wang, Yanning Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12421
• Pdf URL: https://arxiv.org/pdf/2409.12421
• Copy Paste: [[2409.12421]] Frequency-Guided Spatial Adaptation for Camouflaged Object Detection(https://arxiv.org/abs/2409.12421)
• Keywords: foundation model
• Abstract: Camouflaged object detection (COD) aims to segment camouflaged objects which exhibit very similar patterns with the surrounding environment. Recent research works have shown that enhancing the feature representation via the frequency information can greatly alleviate the ambiguity problem between the foreground objects and the background.With the emergence of vision foundation models, like InternImage, Segment Anything Model etc, adapting the pretrained model on COD tasks with a lightweight adapter module shows a novel and promising research direction. Existing adapter modules mainly care about the feature adaptation in the spatial domain. In this paper, we propose a novel frequency-guided spatial adaptation method for COD task. Specifically, we transform the input features of the adapter into frequency domain. By grouping and interacting with frequency components located within non overlapping circles in the spectrogram, different frequency components are dynamically enhanced or weakened, making the intensity of image details and contour features adaptively adjusted. At the same time, the features that are conducive to distinguishing object and background are highlighted, indirectly implying the position and shape of camouflaged object. We conduct extensive experiments on four widely adopted benchmark datasets and the proposed method outperforms 26 state-of-the-art methods with large margins. Code will be released.

Title: Zero-to-Strong Generalization: Eliciting Strong Capabilities of Large Language Models Iteratively without Gold Labels

• Authors: Chaoqun Liu, Qin Chao, Wenxuan Zhang, Xiaobao Wu, Boyang Li, Anh Tuan Luu, Lidong Bing
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12425
• Pdf URL: https://arxiv.org/pdf/2409.12425
• Copy Paste: [[2409.12425]] Zero-to-Strong Generalization: Eliciting Strong Capabilities of Large Language Models Iteratively without Gold Labels(https://arxiv.org/abs/2409.12425)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance through supervised fine-tuning or in-context learning using gold labels. However, this paradigm is limited by the availability of gold labels, while in certain scenarios, LLMs may need to perform tasks that are too complex for humans to provide such labels. To tackle this challenge, this study explores whether solely utilizing unlabeled data can elicit strong model capabilities. We propose a new paradigm termed zero-to-strong generalization. We iteratively prompt LLMs to annotate unlabeled data and retain high-quality labels by filtering. Surprisingly, we obverse that this iterative process gradually unlocks LLMs' potential on downstream tasks. Our experiments on extensive classification and reasoning tasks confirm the effectiveness of our proposed framework. Our analysis indicates that this paradigm is effective for both in-context learning and fine-tuning, and for various model sizes.

Title: FlexiTex: Enhancing Texture Generation with Visual Guidance

• Authors: DaDong Jiang, Xianghui Yang, Zibo Zhao, Sheng Zhang, Jiaao Yu, Zeqiang Lai, Shaoxiong Yang, Chunchao Guo, Xiaobo Zhou, Zhihui Ke
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12431
• Pdf URL: https://arxiv.org/pdf/2409.12431
• Copy Paste: [[2409.12431]] FlexiTex: Enhancing Texture Generation with Visual Guidance(https://arxiv.org/abs/2409.12431)
• Keywords: diffusion, generative
• Abstract: Recent texture generation methods achieve impressive results due to the powerful generative prior they leverage from large-scale text-to-image diffusion models. However, abstract textual prompts are limited in providing global textural or shape information, which results in the texture generation methods producing blurry or inconsistent patterns. To tackle this, we present FlexiTex, embedding rich information via visual guidance to generate a high-quality texture. The core of FlexiTex is the Visual Guidance Enhancement module, which incorporates more specific information from visual guidance to reduce ambiguity in the text prompt and preserve high-frequency details. To further enhance the visual guidance, we introduce a Direction-Aware Adaptation module that automatically designs direction prompts based on different camera poses, avoiding the Janus problem and maintaining semantically global consistency. Benefiting from the visual guidance, FlexiTex produces quantitatively and qualitatively sound results, demonstrating its potential to advance texture generation for real-world applications.

Title: FoME: A Foundation Model for EEG using Adaptive Temporal-Lateral Attention Scaling

• Authors: Enze Shi, Kui Zhao, Qilong Yuan, Jiaqi Wang, Huawen Hu, Sigang Yu, Shu Zhang
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.12454
• Pdf URL: https://arxiv.org/pdf/2409.12454
• Copy Paste: [[2409.12454]] FoME: A Foundation Model for EEG using Adaptive Temporal-Lateral Attention Scaling(https://arxiv.org/abs/2409.12454)
• Keywords: foundation model
• Abstract: Electroencephalography (EEG) is a vital tool to measure and record brain activity in neuroscience and clinical applications, yet its potential is constrained by signal heterogeneity, low signal-to-noise ratios, and limited labeled datasets. In this paper, we propose FoME (Foundation Model for EEG), a novel approach using adaptive temporal-lateral attention scaling to address above-mentioned challenges. FoME is pre-trained on a diverse 1.7TB dataset of scalp and intracranial EEG recordings, comprising 745M parameters trained for 1,096k steps. Our model introduces two key innovations: a time-frequency fusion embedding technique and an adaptive time-lateral attention scaling (ATLAS) mechanism. These components synergistically capture complex temporal and spectral EEG dynamics, enabling FoME to adapt to varying patterns across diverse data streams and facilitate robust multi-channel modeling. Evaluations across four downstream tasks demonstrate FoME's superior performance in classification and forecasting applications, consistently achieving state-of-the-art results. To conclude, FoME establishes a new paradigm for EEG analysis, offering a versatile foundation that advances brain-computer interfaces, clinical diagnostics, and cognitive research across neuroscience and related fields. Our code will be available at this https URL.

Title: Bayesian-Optimized One-Step Diffusion Model with Knowledge Distillation for Real-Time 3D Human Motion Prediction

• Authors: Sibo Tian, Minghui Zheng, Xiao Liang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.12456
• Pdf URL: https://arxiv.org/pdf/2409.12456
• Copy Paste: [[2409.12456]] Bayesian-Optimized One-Step Diffusion Model with Knowledge Distillation for Real-Time 3D Human Motion Prediction(https://arxiv.org/abs/2409.12456)
• Keywords: diffusion, generative
• Abstract: Human motion prediction is a cornerstone of human-robot collaboration (HRC), as robots need to infer the future movements of human workers based on past motion cues to proactively plan their motion, ensuring safety in close collaboration scenarios. The diffusion model has demonstrated remarkable performance in predicting high-quality motion samples with reasonable diversity, but suffers from a slow generative process which necessitates multiple model evaluations, hindering real-world applications. To enable real-time prediction, in this work, we propose training a one-step multi-layer perceptron-based (MLP-based) diffusion model for motion prediction using knowledge distillation and Bayesian optimization. Our method contains two steps. First, we distill a pretrained diffusion-based motion predictor, TransFusion, directly into a one-step diffusion model with the same denoiser architecture. Then, to further reduce the inference time, we remove the computationally expensive components from the original denoiser and use knowledge distillation once again to distill the obtained one-step diffusion model into an even smaller model based solely on MLPs. Bayesian optimization is used to tune the hyperparameters for training the smaller diffusion model. Extensive experimental studies are conducted on benchmark datasets, and our model can significantly improve the inference speed, achieving real-time prediction without noticeable degradation in performance.

Title: HSIGene: A Foundation Model For Hyperspectral Image Generation

• Authors: Li Pang, Datao Tang, Shuang Xu, Deyu Meng, Xiangyong Cao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.12470
• Pdf URL: https://arxiv.org/pdf/2409.12470
• Copy Paste: [[2409.12470]] HSIGene: A Foundation Model For Hyperspectral Image Generation(https://arxiv.org/abs/2409.12470)
• Keywords: diffusion, foundation model
• Abstract: Hyperspectral image (HSI) plays a vital role in various fields such as agriculture and environmental monitoring. However, due to the expensive acquisition cost, the number of hyperspectral images is limited, degenerating the performance of downstream tasks. Although some recent studies have attempted to employ diffusion models to synthesize HSIs, they still struggle with the scarcity of HSIs, affecting the reliability and diversity of the generated images. Some studies propose to incorporate multi-modal data to enhance spatial diversity, but the spectral fidelity cannot be ensured. In addition, existing HSI synthesis models are typically uncontrollable or only support single-condition control, limiting their ability to generate accurate and reliable HSIs. To alleviate these issues, we propose HSIGene, a novel HSI generation foundation model which is based on latent diffusion and supports multi-condition control, allowing for more precise and reliable HSI generation. To enhance the spatial diversity of the training data while preserving spectral fidelity, we propose a new data augmentation method based on spatial super-resolution, in which HSIs are upscaled first, and thus abundant training patches could be obtained by cropping the high-resolution HSIs. In addition, to improve the perceptual quality of the augmented data, we introduce a novel two-stage HSI super-resolution framework, which first applies RGB bands super-resolution and then utilizes our proposed Rectangular Guided Attention Network (RGAN) for guided HSI super-resolution. Experiments demonstrate that the proposed model is capable of generating a vast quantity of realistic HSIs for downstream tasks such as denoising and super-resolution. The code and models are available at this https URL.

Title: Reference Dataset and Benchmark for Reconstructing Laser Parameters from On-axis Video in Powder Bed Fusion of Bulk Stainless Steel

• Authors: Cyril Blanc, Ayyoub Ahar, Kurt De Grave
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12475
• Pdf URL: https://arxiv.org/pdf/2409.12475
• Copy Paste: [[2409.12475]] Reference Dataset and Benchmark for Reconstructing Laser Parameters from On-axis Video in Powder Bed Fusion of Bulk Stainless Steel(https://arxiv.org/abs/2409.12475)
• Keywords: anomaly
• Abstract: We present RAISE-LPBF, a large dataset on the effect of laser power and laser dot speed in powder bed fusion (LPBF) of 316L stainless steel bulk material, monitored by on-axis 20k FPS video. Both process parameters are independently sampled for each scan line from a continuous distribution, so interactions of different parameter choices can be investigated. The data can be used to derive statistical properties of LPBF, as well as to build anomaly detectors. We provide example source code for loading the data, baseline machine learning models and results, and a public benchmark to evaluate predictive models.

Title: Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generation

• Authors: Chenyu Wang, Shuo Yan, Yixuan Chen, Yujiang Wang, Mingzhi Dong, Xiaochen Yang, Dongsheng Li, Robert P. Dick, Qin Lv, Fan Yang, Tun Lu, Ning Gu, Li Shang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12532
• Pdf URL: https://arxiv.org/pdf/2409.12532
• Copy Paste: [[2409.12532]] Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generation(https://arxiv.org/abs/2409.12532)
• Keywords: diffusion
• Abstract: Video generation using diffusion-based models is constrained by high computational costs due to the frame-wise iterative diffusion process. This work presents a Diffusion Reuse MOtion (Dr. Mo) network to accelerate latent video generation. Our key discovery is that coarse-grained noises in earlier denoising steps have demonstrated high motion consistency across consecutive video frames. Following this observation, Dr. Mo propagates those coarse-grained noises onto the next frame by incorporating carefully designed, lightweight inter-frame motions, eliminating massive computational redundancy in frame-wise diffusion models. The more sensitive and fine-grained noises are still acquired via later denoising steps, which can be essential to retain visual qualities. As such, deciding which intermediate steps should switch from motion-based propagations to denoising can be a crucial problem and a key tradeoff between efficiency and quality. Dr. Mo employs a meta-network named Denoising Step Selector (DSS) to dynamically determine desirable intermediate steps across video frames. Extensive evaluations on video generation and editing tasks have shown that Dr. Mo can substantially accelerate diffusion models in video tasks with improved visual qualities.

Title: Improving Cone-Beam CT Image Quality with Knowledge Distillation-Enhanced Diffusion Model in Imbalanced Data Settings

• Authors: Joonil Hwang, Sangjoon Park, NaHyeon Park, Seungryong Cho, Jin Sung Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12539
• Pdf URL: https://arxiv.org/pdf/2409.12539
• Copy Paste: [[2409.12539]] Improving Cone-Beam CT Image Quality with Knowledge Distillation-Enhanced Diffusion Model in Imbalanced Data Settings(https://arxiv.org/abs/2409.12539)
• Keywords: diffusion
• Abstract: In radiation therapy (RT), the reliance on pre-treatment computed tomography (CT) images encounter challenges due to anatomical changes, necessitating adaptive planning. Daily cone-beam CT (CBCT) imaging, pivotal for therapy adjustment, falls short in tissue density accuracy. To address this, our innovative approach integrates diffusion models for CT image generation, offering precise control over data synthesis. Leveraging a self-training method with knowledge distillation, we maximize CBCT data during therapy, complemented by sparse paired fan-beam CTs. This strategy, incorporated into state-of-the-art diffusion-based models, surpasses conventional methods like Pix2pix and CycleGAN. A meticulously curated dataset of 2800 paired CBCT and CT scans, supplemented by 4200 CBCT scans, undergoes preprocessing and teacher model training, including the Brownian Bridge Diffusion Model (BBDM). Pseudo-label CT images are generated, resulting in a dataset combining 5600 CT images with corresponding CBCT images. Thorough evaluation using MSE, SSIM, PSNR and LPIPS demonstrates superior performance against Pix2pix and CycleGAN. Our approach shows promise in generating high-quality CT images from CBCT scans in RT.

Title: CF-GO-Net: A Universal Distribution Learner via Characteristic Function Networks with Graph Optimizers

• Authors: Zeyang Yu, Shengxi Li, Danilo Mandic
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12610
• Pdf URL: https://arxiv.org/pdf/2409.12610
• Copy Paste: [[2409.12610]] CF-GO-Net: A Universal Distribution Learner via Characteristic Function Networks with Graph Optimizers(https://arxiv.org/abs/2409.12610)
• Keywords: generative
• Abstract: Generative models aim to learn the distribution of datasets, such as images, so as to be able to generate samples that statistically resemble real data. However, learning the underlying probability distribution can be very challenging and intractable. To this end, we introduce an approach which employs the characteristic function (CF), a probabilistic descriptor that directly corresponds to the distribution. However, unlike the probability density function (pdf), the characteristic function not only always exists, but also provides an additional degree of freedom, hence enhances flexibility in learning distributions. This removes the critical dependence on pdf-based assumptions, which limit the applicability of traditional methods. While several works have attempted to use CF in generative modeling, they often impose strong constraints on the training process. In contrast, our approach calculates the distance between query points in the CF domain, which is an unconstrained and well defined problem. Next, to deal with the sampling strategy, which is crucial to model performance, we propose a graph neural network (GNN)-based optimizer for the sampling process, which identifies regions where the difference between CFs is most significant. In addition, our method allows the use of a pre-trained model, such as a well-trained autoencoder, and is capable of learning directly in its feature space, without modifying its parameters. This offers a flexible and robust approach to generative modeling, not only provides broader applicability and improved performance, but also equips any latent space world with the ability to become a generative model.

Title: Image inpainting for corrupted images by using the semi-super resolution GAN

• Authors: Mehrshad Momen-Tayefeh, Mehrdad Momen-Tayefeh, Amir Ali Ghafourian Ghahramani
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.12636
• Pdf URL: https://arxiv.org/pdf/2409.12636
• Copy Paste: [[2409.12636]] Image inpainting for corrupted images by using the semi-super resolution GAN(https://arxiv.org/abs/2409.12636)
• Keywords: generative
• Abstract: Image inpainting is a valuable technique for enhancing images that have been corrupted. The primary challenge in this research revolves around the extent of corruption in the input image that the deep learning model must restore. To address this challenge, we introduce a Generative Adversarial Network (GAN) for learning and replicating the missing pixels. Additionally, we have developed a distinct variant of the Super-Resolution GAN (SRGAN), which we refer to as the Semi-SRGAN (SSRGAN). Furthermore, we leveraged three diverse datasets to assess the robustness and accuracy of our proposed model. Our training process involves varying levels of pixel corruption to attain optimal accuracy and generate high-quality images.

Title: Deep generative models as an adversarial attack strategy for tabular machine learning

• Authors: Salijona Dyrmishi, Mihaela Cătălina Stoian, Eleonora Giunchiglia, Maxime Cordy
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12642
• Pdf URL: https://arxiv.org/pdf/2409.12642
• Copy Paste: [[2409.12642]] Deep generative models as an adversarial attack strategy for tabular machine learning(https://arxiv.org/abs/2409.12642)
• Keywords: generative
• Abstract: Deep Generative Models (DGMs) have found application in computer vision for generating adversarial examples to test the robustness of machine learning (ML) systems. Extending these adversarial techniques to tabular ML presents unique challenges due to the distinct nature of tabular data and the necessity to preserve domain constraints in adversarial examples. In this paper, we adapt four popular tabular DGMs into adversarial DGMs (AdvDGMs) and evaluate their effectiveness in generating realistic adversarial examples that conform to domain constraints.

Title: Exploring Large Language Models for Product Attribute Value Identification

• Authors: Kassem Sabeh, Mouna Kacimi, Johann Gamper, Robert Litschko, Barbara Plank
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.12695
• Pdf URL: https://arxiv.org/pdf/2409.12695
• Copy Paste: [[2409.12695]] Exploring Large Language Models for Product Attribute Value Identification(https://arxiv.org/abs/2409.12695)
• Keywords: in-context
• Abstract: Product attribute value identification (PAVI) involves automatically identifying attributes and their values from product information, enabling features like product search, recommendation, and comparison. Existing methods primarily rely on fine-tuning pre-trained language models, such as BART and T5, which require extensive task-specific training data and struggle to generalize to new attributes. This paper explores large language models (LLMs), such as LLaMA and Mistral, as data-efficient and robust alternatives for PAVI. We propose various strategies: comparing one-step and two-step prompt-based approaches in zero-shot settings and utilizing parametric and non-parametric knowledge through in-context learning examples. We also introduce a dense demonstration retriever based on a pre-trained T5 model and perform instruction fine-tuning to explicitly train LLMs on task-specific instructions. Extensive experiments on two product benchmarks show that our two-step approach significantly improves performance in zero-shot settings, and instruction fine-tuning further boosts performance when using training data, demonstrating the practical benefits of using LLMs for PAVI.

Title: Generation and Editing of Mandrill Faces: Application to Sex Editing and Assessment

• Authors: Nicolas M. Dibot, Julien P. Renoult, William Puech
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12705
• Pdf URL: https://arxiv.org/pdf/2409.12705
• Copy Paste: [[2409.12705]] Generation and Editing of Mandrill Faces: Application to Sex Editing and Assessment(https://arxiv.org/abs/2409.12705)
• Keywords: generative
• Abstract: Generative AI has seen major developments in recent years, enhancing the realism of synthetic images, also known as computer-generated images. In addition, generative AI has also made it possible to modify specific image characteristics through image editing. Previous work has developed methods based on generative adversarial networks (GAN) for generating realistic images, in particular faces, but also to modify specific features. However, this work has never been applied to specific animal species. Moreover, the assessment of the results has been generally done subjectively, rather than quantitatively. In this paper, we propose an approach based on methods for generating images of faces of male or female mandrills, a non-human primate. The main novelty of proposed method is the ability to edit their sex by identifying a sex axis in the latent space of a specific GAN. In addition, we have developed an assessment of the sex levels based on statistical features extracted from real image distributions. The experimental results we obtained from a specific database are not only realistic, but also accurate, meeting a need for future work in behavioral experiments with wild mandrills.

Title: DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Input

• Authors: Qijian Tian, Xin Tan, Yuan Xie, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12753
• Pdf URL: https://arxiv.org/pdf/2409.12753
• Copy Paste: [[2409.12753]] DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Input(https://arxiv.org/abs/2409.12753)
• Keywords: self-supervised
• Abstract: We propose DrivingForward, a feed-forward Gaussian Splatting model that reconstructs driving scenes from flexible surround-view input. Driving scene images from vehicle-mounted cameras are typically sparse, with limited overlap, and the movement of the vehicle further complicates the acquisition of camera extrinsics. To tackle these challenges and achieve real-time reconstruction, we jointly train a pose network, a depth network, and a Gaussian network to predict the Gaussian primitives that represent the driving scenes. The pose network and depth network determine the position of the Gaussian primitives in a self-supervised manner, without using depth ground truth and camera extrinsics during training. The Gaussian network independently predicts primitive parameters from each input image, including covariance, opacity, and spherical harmonics coefficients. At the inference stage, our model can achieve feed-forward reconstruction from flexible multi-frame surround-view input. Experiments on the nuScenes dataset show that our model outperforms existing state-of-the-art feed-forward and scene-optimized reconstruction methods in terms of reconstruction.

Title: EventDance++: Language-guided Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition

• Authors: Xu Zheng, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12778
• Pdf URL: https://arxiv.org/pdf/2409.12778
• Copy Paste: [[2409.12778]] EventDance++: Language-guided Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition(https://arxiv.org/abs/2409.12778)
• Keywords: self-supervised
• Abstract: In this paper, we address the challenging problem of cross-modal (image-to-events) adaptation for event-based recognition without accessing any labeled source image data. This task is arduous due to the substantial modality gap between images and events. With only a pre-trained source model available, the key challenge lies in extracting knowledge from this model and effectively transferring knowledge to the event-based domain. Inspired by the natural ability of language to convey semantics across different modalities, we propose EventDance++, a novel framework that tackles this unsupervised source-free cross-modal adaptation problem from a language-guided perspective. We introduce a language-guided reconstruction-based modality bridging (L-RMB) module, which reconstructs intensity frames from events in a self-supervised manner. Importantly, it leverages a vision-language model to provide further supervision, enriching the surrogate images and enhancing modality bridging. This enables the creation of surrogate images to extract knowledge (i.e., labels) from the source model. On top, we propose a multi-representation knowledge adaptation (MKA) module to transfer knowledge to target models, utilizing multiple event representations to capture the spatiotemporal characteristics of events fully. The L-RMB and MKA modules are jointly optimized to achieve optimal performance in bridging the modality gap. Experiments on three benchmark datasets demonstrate that EventDance++ performs on par with methods that utilize source data, validating the effectiveness of our language-guided approach in event-based recognition.

Title: FoodPuzzle: Developing Large Language Model Agents as Flavor Scientists

• Authors: Tenghao Huang, Donghee Lee, John Sweeney, Jiatong Shi, Emily Steliotes, Matthew Lange, Jonathan May, Muhao Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12832
• Pdf URL: https://arxiv.org/pdf/2409.12832
• Copy Paste: [[2409.12832]] FoodPuzzle: Developing Large Language Model Agents as Flavor Scientists(https://arxiv.org/abs/2409.12832)
• Keywords: in-context
• Abstract: Flavor development in the food industry is increasingly challenged by the need for rapid innovation and precise flavor profile creation. Traditional flavor research methods typically rely on iterative, subjective testing, which lacks the efficiency and scalability required for modern demands. This paper presents three contributions to address the challenges. Firstly, we define a new problem domain for scientific agents in flavor science, conceptualized as the generation of hypotheses for flavor profile sourcing and understanding. To facilitate research in this area, we introduce the FoodPuzzle, a challenging benchmark consisting of 978 food items and 1,766 flavor molecules profiles. We propose a novel Scientific Agent approach, integrating in-context learning and retrieval augmented techniques to generate grounded hypotheses in the domain of food science. Experimental results indicate that our model significantly surpasses traditional methods in flavor profile prediction tasks, demonstrating its potential to transform flavor development practices.

Title: Unveiling and Manipulating Concepts in Time Series Foundation Models

• Authors: Michał Wiliński, Mononito Goswami, Nina Żukowska, Willa Potosnak, Artur Dubrawski
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12915
• Pdf URL: https://arxiv.org/pdf/2409.12915
• Copy Paste: [[2409.12915]] Unveiling and Manipulating Concepts in Time Series Foundation Models(https://arxiv.org/abs/2409.12915)
• Keywords: foundation model
• Abstract: Time series foundation models promise to be powerful tools for a wide range of applications. However, little is known about the concepts that these models learn and how can we manipulate them in the latent space. Our study bridges these gaps by identifying concepts learned by these models, localizing them to specific parts of the model, and steering model predictions along these conceptual directions, using synthetic time series data. Our results show that MOMENT, a state-of-the-art foundation model, can discern distinct time series patterns, and that this ability peaks in the middle layers of the network. Moreover, we show that model outputs can be steered using insights from its activations (e.g., by introducing periodic trends to initially constant signals through intervention during inference). Our findings underscore the importance of synthetic data in studying and steering time series foundation models and intervening throughout the whole model (using steering matrices), instead of a single layer.

Title: MaskMol: Knowledge-guided Molecular Image Pre-Training Framework for Activity Cliffs

• Authors: Zhixiang Cheng, Hongxin Xiang, Pengsen Ma, Li Zeng, Xin Jin, Xixi Yang, Jianxin Lin, Yang Deng, Bosheng Song, Xinxin Feng, Changhui Deng, Xiangxiang Zeng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12926
• Pdf URL: https://arxiv.org/pdf/2409.12926
• Copy Paste: [[2409.12926]] MaskMol: Knowledge-guided Molecular Image Pre-Training Framework for Activity Cliffs(https://arxiv.org/abs/2409.12926)
• Keywords: self-supervised
• Abstract: Activity cliffs, which refer to pairs of molecules that are structurally similar but show significant differences in their potency, can lead to model representation collapse and make the model challenging to distinguish them. Our research indicates that as molecular similarity increases, graph-based methods struggle to capture these nuances, whereas image-based approaches effectively retain the distinctions. Thus, we developed MaskMol, a knowledge-guided molecular image self-supervised learning framework. MaskMol accurately learns the representation of molecular images by considering multiple levels of molecular knowledge, such as atoms, bonds, and substructures. By utilizing pixel masking tasks, MaskMol extracts fine-grained information from molecular images, overcoming the limitations of existing deep learning models in identifying subtle structural changes. Experimental results demonstrate MaskMol's high accuracy and transferability in activity cliff estimation and compound potency prediction across 20 different macromolecular targets, outperforming 25 state-of-the-art deep learning and machine learning approaches. Visualization analyses reveal MaskMol's high biological interpretability in identifying activity cliff-relevant molecular substructures. Notably, through MaskMol, we identified candidate EP4 inhibitors that could be used to treat tumors. This study not only raises awareness about activity cliffs but also introduces a novel method for molecular image representation learning and virtual screening, advancing drug discovery and providing new insights into structure-activity relationships (SAR).

Title: The Gaussian Discriminant Variational Autoencoder (GdVAE): A Self-Explainable Model with Counterfactual Explanations

• Authors: Anselm Haselhoff, Kevin Trelenberg, Fabian Küppers, Jonas Schneider
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12952
• Pdf URL: https://arxiv.org/pdf/2409.12952
• Copy Paste: [[2409.12952]] The Gaussian Discriminant Variational Autoencoder (GdVAE): A Self-Explainable Model with Counterfactual Explanations(https://arxiv.org/abs/2409.12952)
• Keywords: generative
• Abstract: Visual counterfactual explanation (CF) methods modify image concepts, e.g, shape, to change a prediction to a predefined outcome while closely resembling the original query image. Unlike self-explainable models (SEMs) and heatmap techniques, they grant users the ability to examine hypothetical "what-if" scenarios. Previous CF methods either entail post-hoc training, limiting the balance between transparency and CF quality, or demand optimization during inference. To bridge the gap between transparent SEMs and CF methods, we introduce the GdVAE, a self-explainable model based on a conditional variational autoencoder (CVAE), featuring a Gaussian discriminant analysis (GDA) classifier and integrated CF explanations. Full transparency is achieved through a generative classifier that leverages class-specific prototypes for the downstream task and a closed-form solution for CFs in the latent space. The consistency of CFs is improved by regularizing the latent space with the explainer function. Extensive comparisons with existing approaches affirm the effectiveness of our method in producing high-quality CF explanations while preserving transparency. Code and models are public.

Title: 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion

• Authors: Zhaoxi Chen, Jiaxiang Tang, Yuhao Dong, Ziang Cao, Fangzhou Hong, Yushi Lan, Tengfei Wang, Haozhe Xie, Tong Wu, Shunsuke Saito, Liang Pan, Dahua Lin, Ziwei Liu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.12957
• Pdf URL: https://arxiv.org/pdf/2409.12957
• Copy Paste: [[2409.12957]] 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion(https://arxiv.org/abs/2409.12957)
• Keywords: diffusion, generative
• Abstract: The increasing demand for high-quality 3D assets across various industries necessitates efficient and automated 3D content creation. Despite recent advancements in 3D generative models, existing methods still face challenges with optimization speed, geometric fidelity, and the lack of assets for physically based rendering (PBR). In this paper, we introduce 3DTopia-XL, a scalable native 3D generative model designed to overcome these limitations. 3DTopia-XL leverages a novel primitive-based 3D representation, PrimX, which encodes detailed shape, albedo, and material field into a compact tensorial format, facilitating the modeling of high-resolution geometry with PBR assets. On top of the novel representation, we propose a generative framework based on Diffusion Transformer (DiT), which comprises 1) Primitive Patch Compression, 2) and Latent Primitive Diffusion. 3DTopia-XL learns to generate high-quality 3D assets from textual or visual inputs. We conduct extensive qualitative and quantitative experiments to demonstrate that 3DTopia-XL significantly outperforms existing methods in generating high-quality 3D assets with fine-grained textures and materials, efficiently bridging the quality gap between generative models and real-world applications.

Title: LVCD: Reference-based Lineart Video Colorization with Diffusion Models

• Authors: Zhitong Huang, Mohan Zhang, Jing Liao
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.12960
• Pdf URL: https://arxiv.org/pdf/2409.12960
• Copy Paste: [[2409.12960]] LVCD: Reference-based Lineart Video Colorization with Diffusion Models(https://arxiv.org/abs/2409.12960)
• Keywords: diffusion, generative
• Abstract: We propose the first video diffusion framework for reference-based lineart video colorization. Unlike previous works that rely solely on image generative models to colorize lineart frame by frame, our approach leverages a large-scale pretrained video diffusion model to generate colorized animation videos. This approach leads to more temporally consistent results and is better equipped to handle large motions. Firstly, we introduce Sketch-guided ControlNet which provides additional control to finetune an image-to-video diffusion model for controllable video synthesis, enabling the generation of animation videos conditioned on lineart. We then propose Reference Attention to facilitate the transfer of colors from the reference frame to other frames containing fast and expansive motions. Finally, we present a novel scheme for sequential sampling, incorporating the Overlapped Blending Module and Prev-Reference Attention, to extend the video diffusion model beyond its original fixed-length limitation for long video colorization. Both qualitative and quantitative results demonstrate that our method significantly outperforms state-of-the-art techniques in terms of frame and video quality, as well as temporal consistency. Moreover, our method is capable of generating high-quality, long temporal-consistent animation videos with large motions, which is not achievable in previous works. Our code and model are available at this https URL.