2024-10-16

Title: High-Fidelity 3D Lung CT Synthesis in ARDS Swine Models Using Score-Based 3D Residual Diffusion Models

• Authors: Siyeop Yoon, Yujin Oh, Xiang Li, Yi Xin, Maurizio Cereda, Quanzheng Li
• Subjects: cs.CV, cs.LG, physics.med-ph
• Abstract URL: https://arxiv.org/abs/2410.10826
• Pdf URL: https://arxiv.org/pdf/2410.10826
• Copy Paste: [[2410.10826]] High-Fidelity 3D Lung CT Synthesis in ARDS Swine Models Using Score-Based 3D Residual Diffusion Models(https://arxiv.org/abs/2410.10826)
• Keywords: diffusion
• Abstract: Acute respiratory distress syndrome (ARDS) is a severe condition characterized by lung inflammation and respiratory failure, with a high mortality rate of approximately 40%. Traditional imaging methods, such as chest X-rays, provide only two-dimensional views, limiting their effectiveness in fully assessing lung pathology. Three-dimensional (3D) computed tomography (CT) offers a more comprehensive visualization, enabling detailed analysis of lung aeration, atelectasis, and the effects of therapeutic interventions. However, the routine use of CT in ARDS management is constrained by practical challenges and risks associated with transporting critically ill patients to remote scanners. In this study, we synthesize high-fidelity 3D lung CT from 2D generated X-ray images with associated physiological parameters using a score-based 3D residual diffusion model. Our preliminary results demonstrate that this approach can produce high-quality 3D CT images that are validated with ground truth, offering a promising solution for enhancing ARDS management.

Title: LLaCA: Multimodal Large Language Continual Assistant

• Authors: Jingyang Qiao, Zhizhong Zhang, Xin Tan, Yanyun Qu, Shouhong Ding, Yuan Xie
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.10868
• Pdf URL: https://arxiv.org/pdf/2410.10868
• Copy Paste: [[2410.10868]] LLaCA: Multimodal Large Language Continual Assistant(https://arxiv.org/abs/2410.10868)
• Keywords: foundation model
• Abstract: Instruction tuning guides the Multimodal Large Language Models (MLLMs) in aligning different modalities by designing text instructions, which seems to be an essential technique to enhance the capabilities and controllability of foundation models. In this framework, Multimodal Continual Instruction Tuning (MCIT) is adopted to continually instruct MLLMs to follow human intent in sequential datasets. We observe existing gradient update would heavily destroy the tuning performance on previous datasets and the zero-shot ability during continual instruction tuning. Exponential Moving Average (EMA) update policy owns the ability to trace previous parameters, which can aid in decreasing forgetting. However, its stable balance weight cannot deal with the ever-changing datasets, leading to the out-of-balance between plasticity and stability of MLLMs. In this paper, we propose a method called Multimodal Large Language Continual Assistant (LLaCA) to address the challenge. Starting from the trade-off prerequisite and EMA update, we propose the plasticity and stability ideal condition. Based on Taylor expansion in the loss function, we find the optimal balance weight is basically according to the gradient information and previous parameters. We automatically determine the balance weight and significantly improve the performance. Through comprehensive experiments on LLaVA-1.5 in a continual visual-question-answering benchmark, compared with baseline, our approach not only highly improves anti-forgetting ability (with reducing forgetting from 22.67 to 2.68), but also significantly promotes continual tuning performance (with increasing average accuracy from 41.31 to 61.89). Our code will be published soon.

Title: Graph Masked Autoencoder for Spatio-Temporal Graph Learning

• Authors: Qianru Zhang, Haixin Wang, Siu-Ming Yiu, Hongzhi Yin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.10915
• Pdf URL: https://arxiv.org/pdf/2410.10915
• Copy Paste: [[2410.10915]] Graph Masked Autoencoder for Spatio-Temporal Graph Learning(https://arxiv.org/abs/2410.10915)
• Keywords: self-supervised, generative
• Abstract: Effective spatio-temporal prediction frameworks play a crucial role in urban sensing applications, including traffic analysis, human mobility behavior modeling, and citywide crime prediction. However, the presence of data noise and label sparsity in spatio-temporal data presents significant challenges for existing neural network models in learning effective and robust region representations. To address these challenges, we propose a novel spatio-temporal graph masked autoencoder paradigm that explores generative self-supervised learning for effective spatio-temporal data augmentation. Our proposed framework introduces a spatial-temporal heterogeneous graph neural encoder that captures region-wise dependencies from heterogeneous data sources, enabling the modeling of diverse spatial dependencies. In our spatio-temporal self-supervised learning paradigm, we incorporate a masked autoencoding mechanism on node representations and structures. This mechanism automatically distills heterogeneous spatio-temporal dependencies across regions over time, enhancing the learning process of dynamic region-wise spatial correlations. To validate the effectiveness of our STGMAE framework, we conduct extensive experiments on various spatio-temporal mining tasks. We compare our approach against state-of-the-art baselines. The results of these evaluations demonstrate the superiority of our proposed framework in terms of performance and its ability to address the challenges of spatial and temporal data noise and sparsity in practical urban sensing scenarios.

Title: Cultural Heritage 3D Reconstruction with Diffusion Networks

• Authors: Pablo Jaramillo, Ivan Sipiran
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.10927
• Pdf URL: https://arxiv.org/pdf/2410.10927
• Copy Paste: [[2410.10927]] Cultural Heritage 3D Reconstruction with Diffusion Networks(https://arxiv.org/abs/2410.10927)
• Keywords: diffusion, generative
• Abstract: This article explores the use of recent generative AI algorithms for repairing cultural heritage objects, leveraging a conditional diffusion model designed to reconstruct 3D point clouds effectively. Our study evaluates the model's performance across general and cultural heritage-specific settings. Results indicate that, with considerations for object variability, the diffusion model can accurately reproduce cultural heritage geometries. Despite encountering challenges like data diversity and outlier sensitivity, the model demonstrates significant potential in artifact restoration research. This work lays groundwork for advancing restoration methodologies for ancient artifacts using AI technologies.

Title: Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs

• Authors: Haozhen Zhang, Tao Feng, Jiaxuan You
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11001
• Pdf URL: https://arxiv.org/pdf/2410.11001
• Copy Paste: [[2410.11001]] Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs(https://arxiv.org/abs/2410.11001)
• Keywords: self-supervised
• Abstract: Retrieval-augmented generation (RAG) has revitalized Large Language Models (LLMs) by injecting non-parametric factual knowledge. Compared with long-context LLMs, RAG is considered an effective summarization tool in a more concise and lightweight manner, which can interact with LLMs multiple times using diverse queries to get comprehensive responses. However, the LLM-generated historical responses, which contain potentially insightful information, are largely neglected and discarded by existing approaches, leading to suboptimal results. In this paper, we propose \textit{graph of records} (\textbf{GoR}), which leverages historical responses generated by LLMs to enhance RAG for long-context global summarization. Inspired by the \textit{retrieve-then-generate} paradigm of RAG, we construct a graph by establishing an edge between the retrieved text chunks and the corresponding LLM-generated response. To further uncover the intricate correlations between them, GoR further features a \textit{graph neural network} and an elaborately designed \textit{BERTScore}-based objective for self-supervised model training, enabling seamless supervision signal backpropagation between reference summaries and node embeddings. We comprehensively compare GoR with 12 baselines across four long-context summarization datasets, and the results indicate that our proposed method reaches the best performance e.g., 15\%, 8\%, and 19\% improvement over retrievers w.r.t. Rouge-L, Rouge-1, and Rouge-2 on the WCEP dataset). Extensive experiments further demonstrate the effectiveness of GoR. Code is available at this https URL

Title: Effective Self-Mining of In-Context Examples for Unsupervised Machine Translation with LLMs

• Authors: Abdellah El Mekki, Muhammad Abdul-Mageed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.11006
• Pdf URL: https://arxiv.org/pdf/2410.11006
• Copy Paste: [[2410.11006]] Effective Self-Mining of In-Context Examples for Unsupervised Machine Translation with LLMs(https://arxiv.org/abs/2410.11006)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have demonstrated impressive performance on a wide range of natural language processing (NLP) tasks, primarily through in-context learning (ICL). In ICL, the LLM is provided with examples that represent a given task such that it learns to generate answers for test inputs. However, access to these in-context examples is not guaranteed especially for low-resource or massively multilingual tasks. In this work, we propose an unsupervised approach to mine in-context examples for machine translation (MT), enabling unsupervised MT (UMT) across different languages. Our approach begins with word-level mining to acquire word translations that are then used to perform sentence-level mining. As the quality of mined parallel pairs may not be optimal due to noise or mistakes, we introduce a filtering criterion to select the optimal in-context examples from a pool of unsupervised parallel sentences. We evaluate our approach using two multilingual LLMs on 288 directions from the FLORES-200 dataset and analyze the impact of various linguistic features on performance. Our findings demonstrate the effectiveness of our unsupervised approach in mining in-context examples for MT, leading to better or comparable translation performance as translation with regular in-context samples (extracted from human-annotated data), while also outperforming the other state-of-the-art UMT methods by an average of $7$ BLEU points.

Title: Beyond Fixed Topologies: Unregistered Training and Comprehensive Evaluation Metrics for 3D Talking Heads

• Authors: Federico Nocentini, Thomas Besnier, Claudio Ferrari, Sylvain Arguillere, Stefano Berretti, Mohamed Daoudi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11041
• Pdf URL: https://arxiv.org/pdf/2410.11041
• Copy Paste: [[2410.11041]] Beyond Fixed Topologies: Unregistered Training and Comprehensive Evaluation Metrics for 3D Talking Heads(https://arxiv.org/abs/2410.11041)
• Keywords: diffusion
• Abstract: Generating speech-driven 3D talking heads presents numerous challenges; among those is dealing with varying mesh topologies. Existing methods require a registered setting, where all meshes share a common topology: a point-wise correspondence across all meshes the model can animate. While simplifying the problem, it limits applicability as unseen meshes must adhere to the training topology. This work presents a framework capable of animating 3D faces in arbitrary topologies, including real scanned data. Our approach relies on a model leveraging heat diffusion over meshes to overcome the fixed topology constraint. We explore two training settings: a supervised one, in which training sequences share a fixed topology within a sequence but any mesh can be animated at test time, and an unsupervised one, which allows effective training with varying mesh structures. Additionally, we highlight the limitations of current evaluation metrics and propose new metrics for better lip-syncing evaluation between speech and facial movements. Our extensive evaluation shows our approach performs favorably compared to fixed topology techniques, setting a new benchmark by offering a versatile and high-fidelity solution for 3D talking head generation.

Title: Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models

• Authors: Cheng Lu, Yang Song
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.11081
• Pdf URL: https://arxiv.org/pdf/2410.11081
• Copy Paste: [[2410.11081]] Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models(https://arxiv.org/abs/2410.11081)
• Keywords: diffusion, generative
• Abstract: Consistency models (CMs) are a powerful class of diffusion-based generative models optimized for fast sampling. Most existing CMs are trained using discretized timesteps, which introduce additional hyperparameters and are prone to discretization errors. While continuous-time formulations can mitigate these issues, their success has been limited by training instability. To address this, we propose a simplified theoretical framework that unifies previous parameterizations of diffusion models and CMs, identifying the root causes of instability. Based on this analysis, we introduce key improvements in diffusion process parameterization, network architecture, and training objectives. These changes enable us to train continuous-time CMs at an unprecedented scale, reaching 1.5B parameters on ImageNet 512x512. Our proposed training algorithm, using only two sampling steps, achieves FID scores of 2.06 on CIFAR-10, 1.48 on ImageNet 64x64, and 1.88 on ImageNet 512x512, narrowing the gap in FID scores with the best existing diffusion models to within 10%.

Title: EchoApex: A General-Purpose Vision Foundation Model for Echocardiography

• Authors: Abdoul Aziz Amadou, Yue Zhang, Sebastien Piat, Paul Klein, Ingo Schmuecking, Tiziano Passerini, Puneet Sharma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.11092
• Pdf URL: https://arxiv.org/pdf/2410.11092
• Copy Paste: [[2410.11092]] EchoApex: A General-Purpose Vision Foundation Model for Echocardiography(https://arxiv.org/abs/2410.11092)
• Keywords: self-supervised, foundation model
• Abstract: Quantitative evaluation of echocardiography is essential for precise assessment of cardiac condition, monitoring disease progression, and guiding treatment decisions. The diverse nature of echo images, including variations in probe types, manufacturers, and pathologies, poses challenges for developing artificial intelligent models that can generalize across different clinical practice. We introduce EchoApex, the first general-purpose vision foundation model echocardiography with applications on a variety of clinical practice. Leveraging self-supervised learning, EchoApex is pretrained on over 20 million echo images from 11 clinical centres. By incorporating task-specific decoders and adapter modules, we demonstrate the effectiveness of EchoApex on 4 different kind of clinical applications with 28 sub-tasks, including view classification, interactive structure segmentation, left ventricle hypertrophy detection and automated ejection fraction estimation from view sequences. Compared to state-of-the-art task-specific models, EchoApex attains improved performance with a unified image encoding architecture, demonstrating the benefits of model pretraining at scale with in-domain data. Furthermore, EchoApex illustrates the potential for developing a general-purpose vision foundation model tailored specifically for echocardiography, capable of addressing a diverse range of clinical applications with high efficiency and efficacy.

Title: IsoChronoMeter: A simple and effective isochronic translation evaluation metric

• Authors: Nikolai Rozanov, Vikentiy Pankov, Dmitrii Mukhutdinov, Dima Vypirailenko
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.11127
• Pdf URL: https://arxiv.org/pdf/2410.11127
• Copy Paste: [[2410.11127]] IsoChronoMeter: A simple and effective isochronic translation evaluation metric(https://arxiv.org/abs/2410.11127)
• Keywords: generative
• Abstract: Machine translation (MT) has come a long way and is readily employed in production systems to serve millions of users daily. With the recent advances in generative AI, a new form of translation is becoming possible - video dubbing. This work motivates the importance of isochronic translation, especially in the context of automatic dubbing, and introduces `IsoChronoMeter' (ICM). ICM is a simple yet effective metric to measure isochrony of translations in a scalable and resource-efficient way without the need for gold data, based on state-of-the-art text-to-speech (TTS) duration predictors. We motivate IsoChronoMeter and demonstrate its effectiveness. Using ICM we demonstrate the shortcomings of state-of-the-art translation systems and show the need for new methods. We release the code at this URL: \url{this https URL}.

Title: Sensor Deprivation Attacks for Stealthy UAV Manipulation

• Authors: Alessandro Erba, John H. Castellanos, Sahil Sihag, Saman Zonouz, Nils Ole Tippenhauer
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.11131
• Pdf URL: https://arxiv.org/pdf/2410.11131
• Copy Paste: [[2410.11131]] Sensor Deprivation Attacks for Stealthy UAV Manipulation(https://arxiv.org/abs/2410.11131)
• Keywords: anomaly
• Abstract: Unmanned Aerial Vehicles autonomously perform tasks with the use of state-of-the-art control algorithms. These control algorithms rely on the freshness and correctness of sensor readings. Incorrect control actions lead to catastrophic destabilization of the process. In this work, we propose a multi-part \emph{Sensor Deprivation Attacks} (SDAs), aiming to stealthily impact process control via sensor reconfiguration. In the first part, the attacker will inject messages on local buses that connect to the sensor. The injected message reconfigures the sensors, e.g.,~to suspend the sensing. In the second part, those manipulation primitives are selectively used to cause adversarial sensor values at the controller, transparently to the data consumer. In the third part, the manipulated sensor values lead to unwanted control actions (e.g. a drone crash). We experimentally investigate all three parts of our proposed attack. Our findings show that i)~reconfiguring sensors can have surprising effects on reported sensor values, and ii)~the attacker can stall the overall Kalman Filter state estimation, leading to a complete stop of control computations. As a result, the UAV becomes destabilized, leading to a crash or significant deviation from its planned trajectory (over 30 meters). We also propose an attack synthesis methodology that optimizes the timing of these SDA manipulations, maximizing their impact. Notably, our results demonstrate that these SDAs evade detection by state-of-the-art UAV anomaly detectors. Our work shows that attacks on sensors are not limited to continuously inducing random measurements, and demonstrate that sensor reconfiguration can completely stall the drone controller. In our experiments, state-of-the-art UAV controller software and countermeasures are unable to handle such manipulations. Hence, we also discuss new corresponding countermeasures.

Title: Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs

• Authors: Severi Rissanen, Markus Heinonen, Arno Solin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11149
• Pdf URL: https://arxiv.org/pdf/2410.11149
• Copy Paste: [[2410.11149]] Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs(https://arxiv.org/abs/2410.11149)
• Keywords: diffusion, generative
• Abstract: The covariance for clean data given a noisy observation is an important quantity in many conditional generation methods for diffusion models. Current methods require heavy test-time computation, altering the standard diffusion training process or denoiser architecture, or making heavy approximations. We propose a new framework that sidesteps these issues by using covariance information that is available for free from training data and the curvature of the generative trajectory, which is linked to the covariance through the second-order Tweedie's formula. We integrate these sources of information using {\em (i)} a novel method to transfer covariance estimates across noise levels and (ii) low-rank updates in a given noise level. We validate the method on linear inverse problems, where it outperforms recent baselines, especially with fewer diffusion steps.

Title: MANet: Fine-Tuning Segment Anything Model for Multimodal Remote Sensing Semantic Segmentation

• Authors: Xianping Ma, Xiaokang Zhang, Man-On Pun, Bo Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11160
• Pdf URL: https://arxiv.org/pdf/2410.11160
• Copy Paste: [[2410.11160]] MANet: Fine-Tuning Segment Anything Model for Multimodal Remote Sensing Semantic Segmentation(https://arxiv.org/abs/2410.11160)
• Keywords: foundation model
• Abstract: Multimodal remote sensing data, collected from a variety of sensors, provide a comprehensive and integrated perspective of the Earth's surface. By employing multimodal fusion techniques, semantic segmentation offers more detailed insights into geographic scenes compared to single-modality approaches. Building upon recent advancements in vision foundation models, particularly the Segment Anything Model (SAM), this study introduces a novel Multimodal Adapter-based Network (MANet) for multimodal remote sensing semantic segmentation. At the core of this approach is the development of a Multimodal Adapter (MMAdapter), which fine-tunes SAM's image encoder to effectively leverage the model's general knowledge for multimodal data. In addition, a pyramid-based Deep Fusion Module (DFM) is incorporated to further integrate high-level geographic features across multiple scales before decoding. This work not only introduces a novel network for multimodal fusion, but also demonstrates, for the first time, SAM's powerful generalization capabilities with Digital Surface Model (DSM) data. Experimental results on two well-established fine-resolution multimodal remote sensing datasets, ISPRS Vaihingen and ISPRS Potsdam, confirm that the proposed MANet significantly surpasses current models in the task of multimodal semantic segmentation. The source code for this work will be accessible at this https URL.

Title: Synthesizing Proton-Density Fat Fraction and $R_2^*$ from 2-point Dixon MRI with Generative Machine Learning

• Authors: Suma Anand, Kaiwen Xu, Colm O'Dushlaine, Sumit Mukherjee
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11186
• Pdf URL: https://arxiv.org/pdf/2410.11186
• Copy Paste: [[2410.11186]] Synthesizing Proton-Density Fat Fraction and $R_2^*$ from 2-point Dixon MRI with Generative Machine Learning(https://arxiv.org/abs/2410.11186)
• Keywords: generative
• Abstract: Magnetic Resonance Imaging (MRI) is the gold standard for measuring fat and iron content non-invasively in the body via measures known as Proton Density Fat Fraction (PDFF) and $R_2^*$, respectively. However, conventional PDFF and $R_2^*$ quantification methods operate on MR images voxel-wise and require at least three measurements to estimate three quantities: water, fat, and $R_2^*$. Alternatively, the two-point Dixon MRI protocol is widely used and fast because it acquires only two measurements; however, these cannot be used to estimate three quantities voxel-wise. Leveraging the fact that neighboring voxels have similar values, we propose using a generative machine learning approach to learn PDFF and $R_2^*$ from Dixon MRI. We use paired Dixon-IDEAL data from UK Biobank in the liver and a Pix2Pix conditional GAN to demonstrate the first large-scale $R_2^*$ imputation from two-point Dixon MRIs. Using our proposed approach, we synthesize PDFF and $R_2^*$ maps that show significantly greater correlation with ground-truth than conventional voxel-wise baselines.

Title: Athena: Retrieval-augmented Legal Judgment Prediction with Large Language Models

• Authors: Xiao Peng, Liang Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.11195
• Pdf URL: https://arxiv.org/pdf/2410.11195
• Copy Paste: [[2410.11195]] Athena: Retrieval-augmented Legal Judgment Prediction with Large Language Models(https://arxiv.org/abs/2410.11195)
• Keywords: in-context
• Abstract: Recently, large language models (LLMs) like ChatGPT, LLaMA, and Claude have prevailed in countless domains, including legal scenarios. With LLMs' rapid technological progress, the development of prompt engineering (PE) as an interface between the LLMs and real-world applications has drawn the attention of all developers. Various PE methods have been proposed to overcome real-world challenges, such as few-shot prompting, chain-of-thought, and retrieval-augmented generation (RAG). However, RAG for legal judgment prediction (LJP) is still underexplored. To address this, we propose "Athena", a novel framework cultivating RAG as a core preprocess component to enhance LLMs' performance on specialized tasks. Athena constructs a knowledge base for accusations, attached with a semantic retrieval mechanism through vectorization. Our experiments show that Athena's overall performance has improved significantly, achieving state-of-the-art results on the CAIL2018 dataset. Our ablation study on the in-context window size parameter further reproduces LLMs' "lost-in-the-middle" phenomenon with a relative positional variation. And with moderate hyper-parameter-tuning, we can achieve at most 95% of accuracy accordingly. We also study the impact of query rewriting and data distribution, providing possible directions for future research based on former analyses.

Title: SplitSEE: A Splittable Self-supervised Framework for Single-Channel EEG Representation Learning

• Authors: Rikuto Kotoge, Zheng Chen, Tasuku Kimura, Yasuko Matsubara, Takufumi Yanagisawa, Haruhiko Kishima, Yasushi Sakurai
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.11200
• Pdf URL: https://arxiv.org/pdf/2410.11200
• Copy Paste: [[2410.11200]] SplitSEE: A Splittable Self-supervised Framework for Single-Channel EEG Representation Learning(https://arxiv.org/abs/2410.11200)
• Keywords: self-supervised
• Abstract: While end-to-end multi-channel electroencephalography (EEG) learning approaches have shown significant promise, their applicability is often constrained in neurological diagnostics, such as intracranial EEG resources. When provided with a single-channel EEG, how can we learn representations that are robust to multi-channels and scalable across varied tasks, such as seizure prediction? In this paper, we present SplitSEE, a structurally splittable framework designed for effective temporal-frequency representation learning in single-channel EEG. The key concept of SplitSEE is a self-supervised framework incorporating a deep clustering task. Given an EEG, we argue that the time and frequency domains are two distinct perspectives, and hence, learned representations should share the same cluster assignment. To this end, we first propose two domain-specific modules that independently learn domain-specific representation and address the temporal-frequency tradeoff issue in conventional spectrogram-based methods. Then, we introduce a novel clustering loss to measure the information similarity. This encourages representations from both domains to coherently describe the same input by assigning them a consistent cluster. SplitSEE leverages a pre-training-to-fine-tuning framework within a splittable architecture and has following properties: (a) Effectiveness: it learns representations solely from single-channel EEG but has even outperformed multi-channel baselines. (b) Robustness: it shows the capacity to adapt across different channels with low performance variance. Superior performance is also achieved with our collected clinical dataset. (c) Scalability: With just one fine-tuning epoch, SplitSEE achieves high and stable performance using partial model layers.

Title: Error Diffusion: Post Training Quantization with Block-Scaled Number Formats for Neural Networks

• Authors: Alireza Khodamoradi, Kristof Denolf, Eric Dellinger
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11203
• Pdf URL: https://arxiv.org/pdf/2410.11203
• Copy Paste: [[2410.11203]] Error Diffusion: Post Training Quantization with Block-Scaled Number Formats for Neural Networks(https://arxiv.org/abs/2410.11203)
• Keywords: diffusion
• Abstract: Quantization reduces the model's hardware costs, such as data movement, storage, and operations like multiply and addition. It also affects the model's behavior by degrading the output quality. Therefore, there is a need for methods that preserve the model's behavior when quantizing model parameters. More exotic numerical encodings, such as block-scaled number formats, have shown advantages for utilizing a fixed bit budget to encode model parameters. This paper presents error diffusion (ED), a hyperparameter-free method for post-training quantization with support for block-scaled data formats. Our approach does not rely on backpropagation or Hessian information. We describe how to improve the quantization process by viewing the neural model as a composite function and diffusing the quantization error in every layer. In addition, we introduce TensorCast, an open-source library based on PyTorch to emulate a variety of number formats, including the block-scaled ones, to aid the research in neural model quantization. We demonstrate the efficacy of our algorithm through rigorous testing on various architectures, including vision and large language models (LLMs), where it consistently delivers competitive results. Our experiments confirm that block-scaled data formats provide a robust choice for post-training quantization and could be used effectively to enhance the practical deployment of advanced neural networks.

Title: DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models

• Authors: Zhengyang Yu, Zhaoyuan Yang, Jing Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11208
• Pdf URL: https://arxiv.org/pdf/2410.11208
• Copy Paste: [[2410.11208]] DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models(https://arxiv.org/abs/2410.11208)
• Keywords: diffusion
• Abstract: Recent text-to-image personalization methods have shown great promise in teaching a diffusion model user-specified concepts given a few images for reusing the acquired concepts in a novel context. With massive efforts being dedicated to personalized generation, a promising extension is personalized editing, namely to edit an image using personalized concepts, which can provide a more precise guidance signal than traditional textual guidance. To address this, a straightforward solution is to incorporate a personalized diffusion model with a text-driven editing framework. However, such a solution often shows unsatisfactory editability on the source image. To address this, we propose DreamSteerer, a plug-in method for augmenting existing T2I personalization methods. Specifically, we enhance the source image conditioned editability of a personalized diffusion model via a novel Editability Driven Score Distillation (EDSD) objective. Moreover, we identify a mode trapping issue with EDSD, and propose a mode shifting regularization with spatial feature guided sampling to avoid such an issue. We further employ two key modifications to the Delta Denoising Score framework that enable high-fidelity local editing with personalized concepts. Extensive experiments validate that DreamSteerer can significantly improve the editability of several T2I personalization baselines while being computationally efficient.

Title: Sampling Strategies for Creation of a Benchmark for Dialectal Sentiment Classification

• Authors: Dipankar Srirag, Jordan Painter, Aditya Joshi, Diptesh Kanojia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.11216
• Pdf URL: https://arxiv.org/pdf/2410.11216
• Copy Paste: [[2410.11216]] Sampling Strategies for Creation of a Benchmark for Dialectal Sentiment Classification(https://arxiv.org/abs/2410.11216)
• Keywords: self-supervised
• Abstract: This paper investigates data sampling strategies to create a benchmark for dialectal sentiment classification of Google Places reviews written in English. Based on location-based filtering, we collect a self-supervised dataset of reviews in Australian (Australian English), Indian (Indian English), and British (British English) English with self-supervised sentiment labels (1-star to 5-star). We employ sampling techniques based on label semantics, review length, and sentiment proportion and report performances on three fine-tuned BERT-based models. Our multi-dialect evaluation provides pointers to challenging scenarios for inner-circle (Australian English and British English) as well as non-native dialects (Indian English) of English, highlighting the need for more diverse benchmarks.

Title: MF-LAL: Drug Compound Generation Using Multi-Fidelity Latent Space Active Learning

• Authors: Peter Eckmann, Dongxia Wu, Germano Heinzelmann, Michael K Gilson, Rose Yu
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.11226
• Pdf URL: https://arxiv.org/pdf/2410.11226
• Copy Paste: [[2410.11226]] MF-LAL: Drug Compound Generation Using Multi-Fidelity Latent Space Active Learning(https://arxiv.org/abs/2410.11226)
• Keywords: generative
• Abstract: Current generative models for drug discovery primarily use molecular docking as an oracle to guide the generation of active compounds. However, such models are often not useful in practice because even compounds with high docking scores do not consistently show experimental activity. More accurate methods for activity prediction exist, such as molecular dynamics based binding free energy calculations, but they are too computationally expensive to use in a generative model. To address this challenge, we propose Multi-Fidelity Latent space Active Learning (MF-LAL), a generative modeling framework that integrates a set of oracles with varying cost-accuracy tradeoffs. Unlike previous approaches that separately learn the surrogate model and generative model, MF-LAL combines the generative and multi-fidelity surrogate models into a single framework, allowing for more accurate activity prediction and higher quality samples. We train MF-LAL with a novel active learning algorithm to further reduce computational cost. Our experiments on two disease-relevant proteins show that MF-LAL produces compounds with significantly better binding free energy scores than other single and multi-fidelity approaches.

Title: Learning Diffusion Model from Noisy Measurement using Principled Expectation-Maximization Method

• Authors: Weimin Bai, Weiheng Tang, Enze Ye, Siyi Chen, Wenzheng Chen, He Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11241
• Pdf URL: https://arxiv.org/pdf/2410.11241
• Copy Paste: [[2410.11241]] Learning Diffusion Model from Noisy Measurement using Principled Expectation-Maximization Method(https://arxiv.org/abs/2410.11241)
• Keywords: diffusion
• Abstract: Diffusion models have demonstrated exceptional ability in modeling complex image distributions, making them versatile plug-and-play priors for solving imaging inverse problems. However, their reliance on large-scale clean datasets for training limits their applicability in scenarios where acquiring clean data is costly or impractical. Recent approaches have attempted to learn diffusion models directly from corrupted measurements, but these methods either lack theoretical convergence guarantees or are restricted to specific types of data corruption. In this paper, we propose a principled expectation-maximization (EM) framework that iteratively learns diffusion models from noisy data with arbitrary corruption types. Our framework employs a plug-and-play Monte Carlo method to accurately estimate clean images from noisy measurements, followed by training the diffusion model using the reconstructed images. This process alternates between estimation and training until convergence. We evaluate the performance of our method across various imaging tasks, including inpainting, denoising, and deblurring. Experimental results demonstrate that our approach enables the learning of high-fidelity diffusion priors from noisy data, significantly enhancing reconstruction quality in imaging inverse problems.

Title: In-Context Learning for Long-Context Sentiment Analysis on Infrastructure Project Opinions

• Authors: Alireza Shamshiri, Kyeong Rok Ryu, June Young Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.11265
• Pdf URL: https://arxiv.org/pdf/2410.11265
• Copy Paste: [[2410.11265]] In-Context Learning for Long-Context Sentiment Analysis on Infrastructure Project Opinions(https://arxiv.org/abs/2410.11265)
• Keywords: in-context
• Abstract: Large language models (LLMs) have achieved impressive results across various tasks. However, they still struggle with long-context documents. This study evaluates the performance of three leading LLMs: GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro on lengthy, complex, and opinion-varying documents concerning infrastructure projects, under both zero-shot and few-shot scenarios. Our results indicate that GPT-4o excels in zero-shot scenarios for simpler, shorter documents, while Claude 3.5 Sonnet surpasses GPT-4o in handling more complex, sentiment-fluctuating opinions. In few-shot scenarios, Claude 3.5 Sonnet outperforms overall, while GPT-4o shows greater stability as the number of demonstrations increases.

Title: Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent

• Authors: Bo Chen, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.11268
• Pdf URL: https://arxiv.org/pdf/2410.11268
• Copy Paste: [[2410.11268]] Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent(https://arxiv.org/abs/2410.11268)
• Keywords: in-context
• Abstract: In-context learning has been recognized as a key factor in the success of Large Language Models (LLMs). It refers to the model's ability to learn patterns on the fly from provided in-context examples in the prompt during inference. Previous studies have demonstrated that the Transformer architecture used in LLMs can implement a single-step gradient descent update by processing in-context examples in a single forward pass. Recent work has further shown that, during in-context learning, a looped Transformer can implement multi-step gradient descent updates in forward passes. However, their theoretical results require an exponential number of in-context examples, $n = \exp(\Omega(T))$, where $T$ is the number of loops or passes, to achieve a reasonably low error. In this paper, we study linear looped Transformers in-context learning on linear vector generation tasks. We show that linear looped Transformers can implement multi-step gradient descent efficiently for in-context learning. Our results demonstrate that as long as the input data has a constant condition number, e.g., $n = O(d)$, the linear looped Transformers can achieve a small error by multi-step gradient descent during in-context learning. Furthermore, our preliminary experiments validate our theoretical analysis. Our findings reveal that the Transformer architecture possesses a stronger in-context learning capability than previously understood, offering new insights into the mechanisms behind LLMs and potentially guiding the better design of efficient inference algorithms for LLMs.

Title: Reducing Source-Private Bias in Extreme Universal Domain Adaptation

• Authors: Hung-Chieh Fang, Po-Yi Lu, Hsuan-Tien Lin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11271
• Pdf URL: https://arxiv.org/pdf/2410.11271
• Copy Paste: [[2410.11271]] Reducing Source-Private Bias in Extreme Universal Domain Adaptation(https://arxiv.org/abs/2410.11271)
• Keywords: self-supervised
• Abstract: Universal Domain Adaptation (UniDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain without assuming how much the label-sets of the two domains intersect. The goal of UniDA is to achieve robust performance on the target domain across different intersection levels. However, existing literature has not sufficiently explored performance under extreme intersection levels. Our experiments reveal that state-of-the-art methods struggle when the source domain has significantly more non-overlapping classes than overlapping ones, a setting we refer to as Extreme UniDA. In this paper, we demonstrate that classical partial domain alignment, which focuses on aligning only overlapping-class data between domains, is limited in mitigating the bias of feature extractors toward source-private classes in extreme UniDA scenarios. We argue that feature extractors trained with source supervised loss distort the intrinsic structure of the target data due to the inherent differences between source-private classes and the target data. To mitigate this bias, we propose using self-supervised learning to preserve the structure of the target data. Our approach can be easily integrated into existing frameworks. We apply the proposed approach to two distinct training paradigms-adversarial-based and optimal-transport-based-and show consistent improvements across various intersection levels, with significant gains in extreme UniDA settings.

Title: Cognitive Overload Attack:Prompt Injection for Long Context

• Authors: Bibek Upadhayay, Vahid Behzadan, Amin Karbasi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.11272
• Pdf URL: https://arxiv.org/pdf/2410.11272
• Copy Paste: [[2410.11272]] Cognitive Overload Attack:Prompt Injection for Long Context(https://arxiv.org/abs/2410.11272)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in performing tasks across various domains without needing explicit retraining. This capability, known as In-Context Learning (ICL), while impressive, exposes LLMs to a variety of adversarial prompts and jailbreaks that manipulate safety-trained LLMs into generating undesired or harmful output. In this paper, we propose a novel interpretation of ICL in LLMs through the lens of cognitive neuroscience, by drawing parallels between learning in human cognition with ICL. We applied the principles of Cognitive Load Theory in LLMs and empirically validate that similar to human cognition, LLMs also suffer from cognitive overload a state where the demand on cognitive processing exceeds the available capacity of the model, leading to potential errors. Furthermore, we demonstrated how an attacker can exploit ICL to jailbreak LLMs through deliberately designed prompts that induce cognitive overload on LLMs, thereby compromising the safety mechanisms of LLMs. We empirically validate this threat model by crafting various cognitive overload prompts and show that advanced models such as GPT-4, Claude-3.5 Sonnet, Claude-3 OPUS, Llama-3-70B-Instruct, Gemini-1.0-Pro, and Gemini-1.5-Pro can be successfully jailbroken, with attack success rates of up to 99.99%. Our findings highlight critical vulnerabilities in LLMs and underscore the urgency of developing robust safeguards. We propose integrating insights from cognitive load theory into the design and evaluation of LLMs to better anticipate and mitigate the risks of adversarial attacks. By expanding our experiments to encompass a broader range of models and by highlighting vulnerabilities in LLMs' ICL, we aim to ensure the development of safer and more reliable AI systems.

Title: Shallow diffusion networks provably learn hidden low-dimensional structure

• Authors: Nicholas M. Boffi, Arthur Jacot, Stephen Tu, Ingvar Ziemann
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.11275
• Pdf URL: https://arxiv.org/pdf/2410.11275
• Copy Paste: [[2410.11275]] Shallow diffusion networks provably learn hidden low-dimensional structure(https://arxiv.org/abs/2410.11275)
• Keywords: diffusion, generative
• Abstract: Diffusion-based generative models provide a powerful framework for learning to sample from a complex target distribution. The remarkable empirical success of these models applied to high-dimensional signals, including images and video, stands in stark contrast to classical results highlighting the curse of dimensionality for distribution recovery. In this work, we take a step towards understanding this gap through a careful analysis of learning diffusion models over the Barron space of single layer neural networks. In particular, we show that these shallow models provably adapt to simple forms of low dimensional structure, thereby avoiding the curse of dimensionality. We combine our results with recent analyses of sampling with diffusion models to provide an end-to-end sample complexity bound for learning to sample from structured distributions. Importantly, our results do not require specialized architectures tailored to particular latent structures, and instead rely on the low-index structure of the Barron space to adapt to the underlying distribution.

Title: ILAEDA: An Imitation Learning Based Approach for Automatic Exploratory Data Analysis

• Authors: Abhijit Manatkar, Devarsh Patel, Hima Patel, Naresh Manwani
• Subjects: cs.LG, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2410.11276
• Pdf URL: https://arxiv.org/pdf/2410.11276
• Copy Paste: [[2410.11276]] ILAEDA: An Imitation Learning Based Approach for Automatic Exploratory Data Analysis(https://arxiv.org/abs/2410.11276)
• Keywords: generative
• Abstract: Automating end-to-end Exploratory Data Analysis (AutoEDA) is a challenging open problem, often tackled through Reinforcement Learning (RL) by learning to predict a sequence of analysis operations (FILTER, GROUP, etc). Defining rewards for each operation is a challenging task and existing methods rely on various \emph{interestingness measures} to craft reward functions to capture the importance of each operation. In this work, we argue that not all of the essential features of what makes an operation important can be accurately captured mathematically using rewards. We propose an AutoEDA model trained through imitation learning from expert EDA sessions, bypassing the need for manually defined interestingness measures. Our method, based on generative adversarial imitation learning (GAIL), generalizes well across datasets, even with limited expert data. We also introduce a novel approach for generating synthetic EDA demonstrations for training. Our method outperforms the existing state-of-the-art end-to-end EDA approach on benchmarks by upto 3x, showing strong performance and generalization, while naturally capturing diverse interestingness measures in generated EDA sessions.

Title: Contrastive learning of cell state dynamics in response to perturbations

• Authors: Soorya Pradeep, Alishba Imran, Ziwen Liu, Taylla Milena Theodoro, Eduardo Hirata-Miyasaki, Ivan Ivanov, Madhura Bhave, Sudip Khadka, Hunter Woosley, Carolina Arias, Shalin B. Mehta
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.11281
• Pdf URL: https://arxiv.org/pdf/2410.11281
• Copy Paste: [[2410.11281]] Contrastive learning of cell state dynamics in response to perturbations(https://arxiv.org/abs/2410.11281)
• Keywords: self-supervised
• Abstract: We introduce DynaCLR, a self-supervised framework for modeling cell dynamics via contrastive learning of representations of time-lapse datasets. Live cell imaging of cells and organelles is widely used to analyze cellular responses to perturbations. Human annotation of dynamic cell states captured by time-lapse perturbation datasets is laborious and prone to bias. DynaCLR integrates single-cell tracking with time-aware contrastive learning to map images of cells at neighboring time points to neighboring embeddings. Mapping the morphological dynamics of cells to a temporally regularized embedding space makes the annotation, classification, clustering, or interpretation of the cell states more quantitative and efficient. We illustrate the features and applications of DynaCLR with the following experiments: analyzing the kinetics of viral infection in human cells, detecting transient changes in cell morphology due to cell division, and mapping the dynamics of organelles due to viral infection. Models trained with DynaCLR consistently achieve $>95\%$ accuracy for infection state classification, enable the detection of transient cell states and reliably embed unseen experiments. DynaCLR provides a flexible framework for comparative analysis of cell state dynamics due to perturbations, such as infection, gene knockouts, and drugs. We provide PyTorch-based implementations of the model training and inference pipeline (this https URL) and a user interface (this https URL) for the visualization and annotation of trajectories of cells in the real space and the embedding space.

Title: AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignment

• Authors: Pankayaraj Pathmanathan, Udari Madhushani Sehwag, Michael-Andrei Panaitescu-Liess, Furong Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11283
• Pdf URL: https://arxiv.org/pdf/2410.11283
• Copy Paste: [[2410.11283]] AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignment(https://arxiv.org/abs/2410.11283)
• Keywords: generative
• Abstract: With the growing adoption of reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs), the risk of backdoor installation during alignment has increased, leading to unintended and harmful behaviors. Existing backdoor triggers are typically limited to fixed word patterns, making them detectable during data cleaning and easily removable post-poisoning. In this work, we explore the use of prompt-specific paraphrases as backdoor triggers, enhancing their stealth and resistance to removal during LLM alignment. We propose AdvBDGen, an adversarially fortified generative fine-tuning framework that automatically generates prompt-specific backdoors that are effective, stealthy, and transferable across models. AdvBDGen employs a generator-discriminator pair, fortified by an adversary, to ensure the installability and stealthiness of backdoors. It enables the crafting and successful installation of complex triggers using as little as 3% of the fine-tuning data. Once installed, these backdoors can jailbreak LLMs during inference, demonstrate improved stability against perturbations compared to traditional constant triggers, and are more challenging to remove. These findings underscore an urgent need for the research community to develop more robust defenses against adversarial backdoor threats in LLM alignment.

Title: Data Selection for Task-Specific Model Finetuning

• Authors: Zifan Liu, Amin Karbasi, Theodoros Rekatsinas
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.11303
• Pdf URL: https://arxiv.org/pdf/2410.11303
• Copy Paste: [[2410.11303]] Data Selection for Task-Specific Model Finetuning(https://arxiv.org/abs/2410.11303)
• Keywords: foundation model
• Abstract: Finetuning foundation models for specific tasks is an emerging paradigm in modern machine learning. The efficacy of task-specific finetuning largely depends on the selection of appropriate training data. We present a framework to select data for task-specific model finetuning, guided by a small but representative set of examples from the target task. To do so, we formulate data selection for task-specific finetuning as an optimization problem with a distribution alignment loss based on optimal transport to capture the discrepancy between the selected data and the target distribution. In addition, we add a regularizer to encourage the diversity of the selected data and incorporate kernel density estimation into the regularizer to reduce the negative effects of near-duplicates among the candidate data. We connect our optimization problem to nearest neighbor search and design efficient algorithms to compute the optimal solution based on approximate nearest neighbor search techniques. We evaluate our method on data selection for both continued pretraining and instruction tuning of language models. We show that instruction tuning using data selected by our method with a 1% selection ratio often outperforms using the full dataset and beats the baseline selection methods by 1.5 points in F1 score on average.

Title: CONSULT: Contrastive Self-Supervised Learning for Few-shot Tumor Detection

• Authors: Sin Chee Chin, Xuan Zhang, Lee Yeong Khang, Wenming Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11307
• Pdf URL: https://arxiv.org/pdf/2410.11307
• Copy Paste: [[2410.11307]] CONSULT: Contrastive Self-Supervised Learning for Few-shot Tumor Detection(https://arxiv.org/abs/2410.11307)
• Keywords: self-supervised, anomaly
• Abstract: Artificial intelligence aids in brain tumor detection via MRI scans, enhancing the accuracy and reducing the workload of medical professionals. However, in scenarios with extremely limited medical images, traditional deep learning approaches tend to fail due to the absence of anomalous images. Anomaly detection also suffers from ineffective feature extraction due to vague training process. Our work introduces a novel two-stage anomaly detection algorithm called CONSULT (CONtrastive Self-sUpervised Learning for few-shot Tumor detection). The first stage of CONSULT fine-tunes a pre-trained feature extractor specifically for MRI brain images, using a synthetic data generation pipeline to create tumor-like data. This process overcomes the lack of anomaly samples and enables the integration of attention mechanisms to focus on anomalous image segments. The first stage is to overcome the shortcomings of current anomaly detection in extracting features in high-variation data by incorporating Context-Aware Contrastive Learning and Self-supervised Feature Adversarial Learning. The second stage of CONSULT uses PatchCore for conventional feature extraction via the fine-tuned weights from the first stage. To summarize, we propose a self-supervised training scheme for anomaly detection, enhancing model performance and data reliability. Furthermore, our proposed contrastive loss, Tritanh Loss, stabilizes learning by offering a unique solution all while enhancing gradient flow. Finally, CONSULT achieves superior performance in few-shot brain tumor detection, demonstrating significant improvements over PatchCore by 9.4%, 12.9%, 10.2%, and 6.0% for 2, 4, 6, and 8 shots, respectively, while training exclusively on healthy images.

Title: Evolutionary Retrofitting

• Authors: Mathurin Videau (TAU), Mariia Zameshina (LIGM), Alessandro Leite (TAU), Laurent Najman (LIGM), Marc Schoenauer (TAU), Olivier Teytaud (TAU)
• Subjects: cs.LG, cs.NE, math.OC
• Abstract URL: https://arxiv.org/abs/2410.11330
• Pdf URL: https://arxiv.org/pdf/2410.11330
• Copy Paste: [[2410.11330]] Evolutionary Retrofitting(https://arxiv.org/abs/2410.11330)
• Keywords: diffusion, generative
• Abstract: AfterLearnER (After Learning Evolutionary Retrofitting) consists in applying non-differentiable optimization, including evolutionary methods, to refine fully-trained machine learning models by optimizing a set of carefully chosen parameters or hyperparameters of the model, with respect to some actual, exact, and hence possibly non-differentiable error signal, performed on a subset of the standard validation set. The efficiency of AfterLearnER is demonstrated by tackling non-differentiable signals such as threshold-based criteria in depth sensing, the word error rate in speech re-synthesis, image quality in 3D generative adversarial networks (GANs), image generation via Latent Diffusion Models (LDM), the number of kills per life at Doom, computational accuracy or BLEU in code translation, and human appreciations in image synthesis. In some cases, this retrofitting is performed dynamically at inference time by taking into account user inputs. The advantages of AfterLearnER are its versatility (no gradient is needed), the possibility to use non-differentiable feedback including human evaluations, the limited overfitting, supported by a theoretical study and its anytime behavior. Last but not least, AfterLearnER requires only a minimal amount of feedback, i.e., a few dozens to a few hundreds of scalars, rather than the tens of thousands needed in most related published works. Compared to fine-tuning (typically using the same loss, and gradient-based optimization on a smaller but still big dataset at a fine grain), AfterLearnER uses a minimum amount of data on the real objective function without requiring differentiability.

Title: DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation

• Authors: Jaehyun Park, Yunho Kim, Sejin Kim, Byung-Jun Lee, Sundong Kim
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.11338
• Pdf URL: https://arxiv.org/pdf/2410.11338
• Copy Paste: [[2410.11338]] DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation(https://arxiv.org/abs/2410.11338)
• Keywords: diffusion
• Abstract: We propose a novel offline reinforcement learning (offline RL) approach, introducing the Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation (DIAR) framework. We address two key challenges in offline RL: out-of-distribution samples and long-horizon problems. We leverage diffusion models to learn state-action sequence distributions and incorporate value functions for more balanced and adaptive decision-making. DIAR introduces an Adaptive Revaluation mechanism that dynamically adjusts decision lengths by comparing current and future state values, enabling flexible long-term decision-making. Furthermore, we address Q-value overestimation by combining Q-network learning with a value function guided by a diffusion model. The diffusion model generates diverse latent trajectories, enhancing policy robustness and generalization. As demonstrated in tasks like Maze2D, AntMaze, and Kitchen, DIAR consistently outperforms state-of-the-art algorithms in long-horizon, sparse-reward environments.

Title: Enhance Graph Alignment for Large Language Models

• Authors: Haitong Luo, Xuying Meng, Suhang Wang, Tianxiang Zhao, Fali Wang, Hanyun Cao, Yujun Zhang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.11370
• Pdf URL: https://arxiv.org/pdf/2410.11370
• Copy Paste: [[2410.11370]] Enhance Graph Alignment for Large Language Models(https://arxiv.org/abs/2410.11370)
• Keywords: self-supervised, foundation model
• Abstract: Graph-structured data is prevalent in the real world. Recently, due to the powerful emergent capabilities, Large Language Models (LLMs) have shown promising performance in modeling graphs. The key to effectively applying LLMs on graphs is converting graph data into a format LLMs can comprehend. Graph-to-token approaches are popular in enabling LLMs to process graph information. They transform graphs into sequences of tokens and align them with text tokens through instruction tuning, where self-supervised instruction tuning helps LLMs acquire general knowledge about graphs, and supervised fine-tuning specializes LLMs for the downstream tasks on graphs. Despite their initial success, we find that existing methods have a misalignment between self-supervised tasks and supervised downstream tasks, resulting in negative transfer from self-supervised fine-tuning to downstream tasks. To address these issues, we propose Graph Alignment Large Language Models (GALLM) to benefit from aligned task templates. In the self-supervised tuning stage, we introduce a novel text matching task using templates aligned with downstream tasks. In the task-specific tuning stage, we propose two category prompt methods that learn supervision information from additional explanation with further aligned templates. Experimental evaluations on four datasets demonstrate substantial improvements in supervised learning, multi-dataset generalizability, and particularly in zero-shot capability, highlighting the model's potential as a graph foundation model.

Title: DRACO: A Denoising-Reconstruction Autoencoder for Cryo-EM

• Authors: Yingjun Shen, Haizhao Dai, Qihe Chen, Yan Zeng, Jiakai Zhang, Yuan Pei, Jingyi Yu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.11373
• Pdf URL: https://arxiv.org/pdf/2410.11373
• Copy Paste: [[2410.11373]] DRACO: A Denoising-Reconstruction Autoencoder for Cryo-EM(https://arxiv.org/abs/2410.11373)
• Keywords: self-supervised, foundation model
• Abstract: Foundation models in computer vision have demonstrated exceptional performance in zero-shot and few-shot tasks by extracting multi-purpose features from large-scale datasets through self-supervised pre-training methods. However, these models often overlook the severe corruption in cryogenic electron microscopy (cryo-EM) images by high-level noises. We introduce DRACO, a Denoising-Reconstruction Autoencoder for CryO-EM, inspired by the Noise2Noise (N2N) approach. By processing cryo-EM movies into odd and even images and treating them as independent noisy observations, we apply a denoising-reconstruction hybrid training scheme. We mask both images to create denoising and reconstruction tasks. For DRACO's pre-training, the quality of the dataset is essential, we hence build a high-quality, diverse dataset from an uncurated public database, including over 270,000 movies or micrographs. After pre-training, DRACO naturally serves as a generalizable cryo-EM image denoiser and a foundation model for various cryo-EM downstream tasks. DRACO demonstrates the best performance in denoising, micrograph curation, and particle picking tasks compared to state-of-the-art baselines. We will release the code, pre-trained models, and the curated dataset to stimulate further research.

Title: Augmentation-Driven Metric for Balancing Preservation and Modification in Text-Guided Image Editing

• Authors: Yoonjeon Kim, Soohyun Ryu, Yeonsung Jung, Hyunkoo Lee, Joowon Kim, June Yong Yang, Jaeryong Hwang, Eunho Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.11374
• Pdf URL: https://arxiv.org/pdf/2410.11374
• Copy Paste: [[2410.11374]] Augmentation-Driven Metric for Balancing Preservation and Modification in Text-Guided Image Editing(https://arxiv.org/abs/2410.11374)
• Keywords: generative
• Abstract: The development of vision-language and generative models has significantly advanced text-guided image editing, which seeks \textit{preservation} of core elements in the source image while implementing \textit{modifications} based on the target text. However, in the absence of evaluation metrics specifically tailored for text-guided image editing, existing metrics are limited in balancing the consideration of preservation and modification. Especially, our analysis reveals that CLIPScore, the most commonly used metric, tends to favor modification and ignore core attributes to be preserved, resulting in inaccurate evaluations. To address this problem, we propose \texttt{AugCLIP}, \black{which balances preservation and modification by estimating the representation of an ideal edited image that aligns with the target text with minimum alteration on the source image. We augment detailed textual descriptions on the source image and the target text using a multi-modal large language model, to model a hyperplane that separates CLIP space into source or target. The representation of the ideal edited image is an orthogonal projection of the source image into the hyperplane, which encapsulates the relative importance of each attribute considering the interdependent relationships.} Our extensive experiments on five benchmark datasets, encompassing a diverse range of editing scenarios, demonstrate that \texttt{AugCLIP} aligns remarkably well with human evaluation standards compared to existing metrics. The code for evaluation will be open-sourced to contribute to the community.

Title: Hessian-Informed Flow Matching

• Authors: Christopher Iliffe Sprague, Arne Elofsson, Hossein Azizpour
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.11433
• Pdf URL: https://arxiv.org/pdf/2410.11433
• Copy Paste: [[2410.11433]] Hessian-Informed Flow Matching(https://arxiv.org/abs/2410.11433)
• Keywords: generative
• Abstract: Modeling complex systems that evolve toward equilibrium distributions is important in various physical applications, including molecular dynamics and robotic control. These systems often follow the stochastic gradient descent of an underlying energy function, converging to stationary distributions around energy minima. The local covariance of these distributions is shaped by the energy landscape's curvature, often resulting in anisotropic characteristics. While flow-based generative models have gained traction in generating samples from equilibrium distributions in such applications, they predominately employ isotropic conditional probability paths, limiting their ability to capture such covariance structures. In this paper, we introduce Hessian-Informed Flow Matching (HI-FM), a novel approach that integrates the Hessian of an energy function into conditional flows within the flow matching framework. This integration allows HI-FM to account for local curvature and anisotropic covariance structures. Our approach leverages the linearization theorem from dynamical systems and incorporates additional considerations such as time transformations and equivariance. Empirical evaluations on the MNIST and Lennard-Jones particles datasets demonstrate that HI-FM improves the likelihood of test samples.

Title: A Simple Approach to Unifying Diffusion-based Conditional Generation

• Authors: Xirui Li, Charles Herrmann, Kelvin C.K. Chan, Yinxiao Li, Deqing Sun, Chao Ma, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11439
• Pdf URL: https://arxiv.org/pdf/2410.11439
• Copy Paste: [[2410.11439]] A Simple Approach to Unifying Diffusion-based Conditional Generation(https://arxiv.org/abs/2410.11439)
• Keywords: diffusion
• Abstract: Recent progress in image generation has sparked research into controlling these models through condition signals, with various methods addressing specific challenges in conditional generation. Instead of proposing another specialized technique, we introduce a simple, unified framework to handle diverse conditional generation tasks involving a specific image-condition correlation. By learning a joint distribution over a correlated image pair (e.g. image and depth) with a diffusion model, our approach enables versatile capabilities via different inference-time sampling schemes, including controllable image generation (e.g. depth to image), estimation (e.g. image to depth), signal guidance, joint generation (image & depth), and coarse control. Previous attempts at unification often introduce significant complexity through multi-stage training, architectural modification, or increased parameter counts. In contrast, our simple formulation requires a single, computationally efficient training stage, maintains the standard model input, and adds minimal learned parameters (15% of the base model). Moreover, our model supports additional capabilities like non-spatially aligned and coarse conditioning. Extensive results show that our single model can produce comparable results with specialized methods and better results than prior unified methods. We also demonstrate that multiple models can be effectively combined for multi-signal conditional generation.

Title: On Championing Foundation Models: From Explainability to Interpretability

• Authors: Shi Fu, Yuzhu Chen, Yingjie Wang, Dacheng Tao
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.11444
• Pdf URL: https://arxiv.org/pdf/2410.11444
• Copy Paste: [[2410.11444]] On Championing Foundation Models: From Explainability to Interpretability(https://arxiv.org/abs/2410.11444)
• Keywords: foundation model
• Abstract: Understanding the inner mechanisms of black-box foundation models (FMs) is essential yet challenging in artificial intelligence and its applications. Over the last decade, the long-running focus has been on their explainability, leading to the development of post-hoc explainable methods to rationalize the specific decisions already made by black-box FMs. However, these explainable methods have certain limitations in terms of faithfulness, detail capture and resource requirement. Consequently, in response to these issues, a new class of interpretable methods should be considered to unveil the underlying mechanisms in an accurate, comprehensive, heuristic and resource-light way. This survey aims to review interpretable methods that comply with the aforementioned principles and have been successfully applied to FMs. These methods are deeply rooted in machine learning theory, covering the analysis of generalization performance, expressive capability, and dynamic behavior. They provide a thorough interpretation of the entire workflow of FMs, ranging from the inference capability and training dynamics to their ethical implications. Ultimately, drawing upon these interpretations, this review identifies the next frontier research directions for FMs.

Title: Can sparse autoencoders make sense of latent representations?

• Authors: Viktoria Schuster
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11468
• Pdf URL: https://arxiv.org/pdf/2410.11468
• Copy Paste: [[2410.11468]] Can sparse autoencoders make sense of latent representations?(https://arxiv.org/abs/2410.11468)
• Keywords: generative
• Abstract: Sparse autoencoders (SAEs) have lately been used to uncover interpretable latent features in large language models. Here, we explore their potential for decomposing latent representations in complex and high-dimensional biological data, where the underlying variables are often unknown. On simulated data we show that generative hidden variables can be captured in learned representations in the form of superpositions. The degree to which they are learned depends on the completeness of the representations. Superpositions, however, are not identifiable if these generative variables are unknown. SAEs can to some extent recover these variables, yielding interpretable features. Applied to single-cell multi-omics data, we show that an SAE can uncover key biological processes such as carbon dioxide transport and ion homeostasis, which are crucial for red blood cell differentiation and immune function. Our findings highlight how SAEs can be used in advancing interpretability in biological and other scientific domains.

Title: InvSeg: Test-Time Prompt Inversion for Semantic Segmentation

• Authors: Jiayi Lin, Jiabo Huang, Jian Hu, Shaogang Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11473
• Pdf URL: https://arxiv.org/pdf/2410.11473
• Copy Paste: [[2410.11473]] InvSeg: Test-Time Prompt Inversion for Semantic Segmentation(https://arxiv.org/abs/2410.11473)
• Keywords: diffusion
• Abstract: Visual-textual correlations in the attention maps derived from text-to-image diffusion models are proven beneficial to dense visual prediction tasks, e.g., semantic segmentation. However, a significant challenge arises due to the input distributional discrepancy between the context-rich sentences used for image generation and the isolated class names typically employed in semantic segmentation, hindering the diffusion models from capturing accurate visual-textual correlations. To solve this, we propose InvSeg, a test-time prompt inversion method that tackles open-vocabulary semantic segmentation by inverting image-specific visual context into text prompt embedding space, leveraging structure information derived from the diffusion model's reconstruction process to enrich text prompts so as to associate each class with a structure-consistent mask. Specifically, we introduce Contrastive Soft Clustering (CSC) to align derived masks with the image's structure information, softly selecting anchors for each class and calculating weighted distances to push inner-class pixels closer while separating inter-class pixels, thereby ensuring mask distinction and internal consistency. By incorporating sample-specific context, InvSeg learns context-rich text prompts in embedding space and achieves accurate semantic alignment across modalities. Experiments show that InvSeg achieves state-of-the-art performance on the PASCAL VOC and Context datasets. Project page: this https URL.

Title: How Transformers Implement Induction Heads: Approximation and Optimization Analysis

• Authors: Mingze Wang, Ruoxi Yu, Weinan E, Lei Wu
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.11474
• Pdf URL: https://arxiv.org/pdf/2410.11474
• Copy Paste: [[2410.11474]] How Transformers Implement Induction Heads: Approximation and Optimization Analysis(https://arxiv.org/abs/2410.11474)
• Keywords: in-context
• Abstract: Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remain limited. A recent work (Elhage et al., 2021) identified a "rich" in-context mechanism known as induction head, contrasting with "lazy" $n$-gram models that overlook long-range dependencies. In this work, we provide both approximation and optimization analyses of how transformers implement induction heads. In the approximation analysis, we formalize both standard and generalized induction head mechanisms, and examine how transformers can efficiently implement them, with an emphasis on the distinct role of each transformer submodule. For the optimization analysis, we study the training dynamics on a synthetic mixed target, composed of a 4-gram and an in-context 2-gram component. This setting enables us to precisely characterize the entire training process and uncover an {\em abrupt transition} from lazy (4-gram) to rich (induction head) mechanisms as training progresses.

Title: Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations

• Authors: M. Germán-Morales, A.J. Rivera-Rivas, M.J. del Jesus Díaz, C.J. Carmona
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11539
• Pdf URL: https://arxiv.org/pdf/2410.11539
• Copy Paste: [[2410.11539]] Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations(https://arxiv.org/abs/2410.11539)
• Keywords: generative
• Abstract: High computational power and the availability of large datasets have supported the development of Foundational Models. They are a new emerging technique widely used in Generative Artificial Intelligence, characterized by their scalability and their use in Transfer Learning. The enormous and heterogeneous amounts of data used in their initial training phase, known as pre-training, give them a higher generalization capacity than any other specific model, constituting a solid base that can be adapted or adjusted to a wide range of tasks, increasing their applicability. This study proposes LLIAM, the Llama Lora-Integrated Autorregresive Model. Low-Rank Adaptations are used to enhance the knowledge of the model with diverse time series datasets, known as the fine-tuning phase. To illustrate the capabilities of our proposal, two sets of experiments have been carried out that obtained favorable and promising results with lower training times than other Deep Learning approaches. With this work, we also encourage the use of available resources (such as these pre-trained models) to avoid unnecessary and costly training, narrowing the gap between the goals of traditional Artificial Intelligence and those specified by the definition of Green Artificial Intelligence.

Title: PaSTe: Improving the Efficiency of Visual Anomaly Detection at the Edge

• Authors: Manuel Barusco, Francesco Borsatti, Davide Dalle Pezze, Francesco Paissan, Elisabetta Farella, Gian Antonio Susto
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11591
• Pdf URL: https://arxiv.org/pdf/2410.11591
• Copy Paste: [[2410.11591]] PaSTe: Improving the Efficiency of Visual Anomaly Detection at the Edge(https://arxiv.org/abs/2410.11591)
• Keywords: anomaly
• Abstract: Visual Anomaly Detection (VAD) has gained significant research attention for its ability to identify anomalous images and pinpoint the specific areas responsible for the anomaly. A key advantage of VAD is its unsupervised nature, which eliminates the need for costly and time-consuming labeled data collection. However, despite its potential for real-world applications, the literature has given limited focus to resource-efficient VAD, particularly for deployment on edge devices. This work addresses this gap by leveraging lightweight neural networks to reduce memory and computation requirements, enabling VAD deployment on resource-constrained edge devices. We benchmark the major VAD algorithms within this framework and demonstrate the feasibility of edge-based VAD using the well-known MVTec dataset. Furthermore, we introduce a novel algorithm, Partially Shared Teacher-student (PaSTe), designed to address the high resource demands of the existing Student Teacher Feature Pyramid Matching (STFPM) approach. Our results show that PaSTe decreases the inference time by 25%, while reducing the training time by 33% and peak RAM usage during training by 76%. These improvements make the VAD process significantly more efficient, laying a solid foundation for real-world deployment on edge devices.

Title: Federated Learning framework for LoRaWAN-enabled IIoT communication: A case study

• Authors: Oscar Torres Sanchez, Guilherme Borges, Duarte Raposo, André Rodrigues, Fernando Boavida, Jorge Sá Silva
• Subjects: cs.LG, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2410.11612
• Pdf URL: https://arxiv.org/pdf/2410.11612
• Copy Paste: [[2410.11612]] Federated Learning framework for LoRaWAN-enabled IIoT communication: A case study(https://arxiv.org/abs/2410.11612)
• Keywords: anomaly
• Abstract: The development of intelligent Industrial Internet of Things (IIoT) systems promises to revolutionize operational and maintenance practices, driving improvements in operational efficiency. Anomaly detection within IIoT architectures plays a crucial role in preventive maintenance and spotting irregularities in industrial components. However, due to limited message and processing capacity, traditional Machine Learning (ML) faces challenges in deploying anomaly detection models in resource-constrained environments like LoRaWAN. On the other hand, Federated Learning (FL) solves this problem by enabling distributed model training, addressing privacy concerns, and minimizing data transmission. This study explores using FL for anomaly detection in industrial and civil construction machinery architectures that use IIoT prototypes with LoRaWAN communication. The process leverages an optimized autoencoder neural network structure and compares federated models with centralized ones. Despite uneven data distribution among machine clients, FL demonstrates effectiveness, with a mean F1 score (of 94.77), accuracy (of 92.30), TNR (of 90.65), and TPR (92.93), comparable to centralized models, considering airtime of trainning messages of 52.8 min. Local model evaluations on each machine highlight adaptability. At the same time, the performed analysis identifies message requirements, minimum training hours, and optimal round/epoch configurations for FL in LoRaWAN, guiding future implementations in constrained industrial environments.

Title: VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI

• Authors: Sijie Cheng, Kechen Fang, Yangyang Yu, Sicheng Zhou, Bohao Li, Ye Tian, Tingguang Li, Lei Han, Yang Liu
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.11623
• Pdf URL: https://arxiv.org/pdf/2410.11623
• Copy Paste: [[2410.11623]] VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI(https://arxiv.org/abs/2410.11623)
• Keywords: foundation model
• Abstract: Recent advancements in Multi-modal Large Language Models (MLLMs) have opened new avenues for applications in Embodied AI. Building on previous work, EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating egocentric video understanding capabilities. To bridge the gap between MLLMs and low-level control in Embodied AI, we design four key interrelated tasks: video question-answering, hierarchy planning, visual grounding and reward modeling. To minimize manual annotation costs, we develop an automatic data generation pipeline based on the Ego4D dataset, leveraging the prior knowledge and multimodal capabilities of GPT-4o. Three human annotators then filter the generated data to ensure diversity and quality, resulting in the VidEgoThink benchmark. We conduct extensive experiments with three types of models: API-based MLLMs, open-source image-based MLLMs, and open-source video-based MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform poorly across all tasks related to egocentric video understanding. These findings suggest that foundation models still require significant advancements to be effectively applied to first-person scenarios in Embodied AI. In conclusion, VidEgoThink reflects a research trend towards employing MLLMs for egocentric vision, akin to human capabilities, enabling active observation and interaction in the complex real-world environments.

Title: Simultaneous Diffusion Sampling for Conditional LiDAR Generation

• Authors: Ryan Faulkner, Luke Haub, Simon Ratcliffe, Anh-Dzung Doan, Ian Reid, Tat-Jun Chin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11628
• Pdf URL: https://arxiv.org/pdf/2410.11628
• Copy Paste: [[2410.11628]] Simultaneous Diffusion Sampling for Conditional LiDAR Generation(https://arxiv.org/abs/2410.11628)
• Keywords: diffusion, generative
• Abstract: By enabling capturing of 3D point clouds that reflect the geometry of the immediate environment, LiDAR has emerged as a primary sensor for autonomous systems. If a LiDAR scan is too sparse, occluded by obstacles, or too small in range, enhancing the point cloud scan by while respecting the geometry of the scene is useful for downstream tasks. Motivated by the explosive growth of interest in generative methods in vision, conditional LiDAR generation is starting to take off. This paper proposes a novel simultaneous diffusion sampling methodology to generate point clouds conditioned on the 3D structure of the scene as seen from multiple views. The key idea is to impose multi-view geometric constraints on the generation process, exploiting mutual information for enhanced results. Our method begins by recasting the input scan to multiple new viewpoints around the scan, thus creating multiple synthetic LiDAR scans. Then, the synthetic and input LiDAR scans simultaneously undergo conditional generation according to our methodology. Results show that our method can produce accurate and geometrically consistent enhancements to point cloud scans, allowing it to outperform existing methods by a large margin in a variety of benchmarks.

Title: Feature-guided score diffusion for sampling conditional densities

• Authors: Zahra Kadkhodaie, Stéphane Mallat, Eero P. Simoncelli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11646
• Pdf URL: https://arxiv.org/pdf/2410.11646
• Copy Paste: [[2410.11646]] Feature-guided score diffusion for sampling conditional densities(https://arxiv.org/abs/2410.11646)
• Keywords: diffusion
• Abstract: Score diffusion methods can learn probability densities from samples. The score of the noise-corrupted density is estimated using a deep neural network, which is then used to iteratively transport a Gaussian white noise density to a target density. Variants for conditional densities have been developed, but correct estimation of the corresponding scores is difficult. We avoid these difficulties by introducing an algorithm that guides the diffusion with a projected score. The projection pushes the image feature vector towards the feature vector centroid of the target class. The projected score and the feature vectors are learned by the same network. Specifically, the image feature vector is defined as the spatial averages of the channels activations in select layers of the network. Optimizing the projected score for denoising loss encourages image feature vectors of each class to cluster around their centroids. It also leads to the separations of the centroids. We show that these centroids provide a low-dimensional Euclidean embedding of the class conditional densities. We demonstrate that the algorithm can generate high quality and diverse samples from the conditioning class. Conditional generation can be performed using feature vectors interpolated between those of the training set, demonstrating out-of-distribution generalization.

Title: Degradation Oriented and Regularized Network for Real-World Depth Super-Resolution

• Authors: Zhengxue Wang, Zhiqiang Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11666
• Pdf URL: https://arxiv.org/pdf/2410.11666
• Copy Paste: [[2410.11666]] Degradation Oriented and Regularized Network for Real-World Depth Super-Resolution(https://arxiv.org/abs/2410.11666)
• Keywords: self-supervised
• Abstract: Recently, existing RGB-guided depth super-resolution methods achieve excellent performance based on the assumption of fixed and known degradation (e.g., bicubic downsampling). However, in real-world scenarios, the captured depth often suffers from unconventional and agnostic degradation due to sensor limitations and the complexity of imaging environments (e.g., low reflective surface, illumination). Their performance significantly declines when these real degradation differ from their assumptions. To address these issues, we propose a Degradation Oriented and Regularized Network, DORNet, which pays more attention on learning degradation representation of low-resolution depth that can provide targeted guidance for depth recovery. Specifically, we first design a self-supervised Degradation Learning to model the discriminative degradation representation of low-resolution depth using routing selection-based Degradation Regularization. Then, we present a Degradation Awareness that recursively conducts multiple Degradation-Oriented Feature Transformations, each of which selectively embeds RGB information into the depth based on the learned degradation representation. Extensive experimental results on both real and synthetic datasets demonstrate that our method achieves state-of-the-art performance.

Title: Generative Image Steganography Based on Point Cloud

• Authors: Zhong Yangjie, Liu Jia, Liu Meiqi, Ke Yan
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.11673
• Pdf URL: https://arxiv.org/pdf/2410.11673
• Copy Paste: [[2410.11673]] Generative Image Steganography Based on Point Cloud(https://arxiv.org/abs/2410.11673)
• Keywords: generative
• Abstract: In deep steganography, the model size is usually related to the underlying mesh resolution, and a separate neural network needs to be trained as a message extractor. In this paper, we propose a generative image steganography based on point cloud representation, which represents image data as a point cloud, learns the distribution of the point cloud data, and represents it in the form of a continuous function. This method breaks through the limitation of the image resolution, and can generate images with arbitrary resolution according to the actual need, and omits the need for explicit data for image steganography. At the same time, using a fixed point cloud extractor transfers the training of the network to the point cloud data, which saves the training time and avoids the risk of exposing the steganography behavior caused by the transmission of the message extractor. Experiments prove that the steganographic images generated by the scheme have very high image quality and the accuracy of message extraction reaches more than 99%.

Title: A Survey of Low-shot Vision-Language Model Adaptation via Representer Theorem

• Authors: Kun Ding, Ying Wang, Gaofeng Meng, Shiming Xiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11686
• Pdf URL: https://arxiv.org/pdf/2410.11686
• Copy Paste: [[2410.11686]] A Survey of Low-shot Vision-Language Model Adaptation via Representer Theorem(https://arxiv.org/abs/2410.11686)
• Keywords: foundation model
• Abstract: The advent of pre-trained vision-language foundation models has revolutionized the field of zero/few-shot (i.e., low-shot) image recognition. The key challenge to address under the condition of limited training data is how to fine-tune pre-trained vision-language models in a parameter-efficient manner. Previously, numerous approaches tackling this challenge have been proposed. Meantime, a few survey papers are also published to summarize these works. However, there still lacks a unified computational framework to integrate existing methods together, identify their nature and support in-depth comparison. As such, this survey paper first proposes a unified computational framework from the perspective of Representer Theorem and then derives many of the existing methods by specializing this framework. Thereafter, a comparative analysis is conducted to uncover the differences and relationships between existing methods. Based on the analyses, some possible variants to improve the existing works are presented. As a demonstration, we extend existing methods by modeling inter-class correlation between representers in reproducing kernel Hilbert space (RKHS), which is implemented by exploiting the closed-form solution of kernel ridge regression. Extensive experiments on 11 datasets are conducted to validate the effectiveness of this method. Toward the end of this paper, we discuss the limitations and provide further research directions.

Title: State-space models can learn in-context by gradient descent

• Authors: Neeraj Mohan Sushma, Yudou Tian, Harshvardhan Mestha, Nicolo Colombo, David Kappel, Anand Subramoney
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2410.11687
• Pdf URL: https://arxiv.org/pdf/2410.11687
• Copy Paste: [[2410.11687]] State-space models can learn in-context by gradient descent(https://arxiv.org/abs/2410.11687)
• Keywords: foundation model, in-context
• Abstract: Deep state-space models (Deep SSMs) have shown capabilities for in-context learning on autoregressive tasks, similar to transformers. However, the architectural requirements and mechanisms enabling this in recurrent networks remain unclear. This study demonstrates that state-space model architectures can perform gradient-based learning and use it for in-context learning. We prove that a single structured state-space model layer, augmented with local self-attention, can reproduce the outputs of an implicit linear model with least squares loss after one step of gradient descent. Our key insight is that the diagonal linear recurrent layer can act as a gradient accumulator, which can be `applied' to the parameters of the implicit regression model. We validate our construction by training randomly initialized augmented SSMs on simple linear regression tasks. The empirically optimized parameters match the theoretical ones, obtained analytically from the implicit model construction. Extensions to multi-step linear and non-linear regression yield consistent results. The constructed SSM encompasses features of modern deep state-space models, with the potential for scalable training and effectiveness even in general tasks. The theoretical construction elucidates the role of local self-attention and multiplicative interactions in recurrent architectures as the key ingredients for enabling the expressive power typical of foundation models.

Title: Visual Fixation-Based Retinal Prosthetic Simulation

• Authors: Yuli Wu, Do Dinh Tan Nguyen, Henning Konermann, Rüveyda Yilmaz, Peter Walter, Johannes Stegmaier
• Subjects: cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2410.11688
• Pdf URL: https://arxiv.org/pdf/2410.11688
• Copy Paste: [[2410.11688]] Visual Fixation-Based Retinal Prosthetic Simulation(https://arxiv.org/abs/2410.11688)
• Keywords: self-supervised, foundation model
• Abstract: This study proposes a retinal prosthetic simulation framework driven by visual fixations, inspired by the saccade mechanism, and assesses performance improvements through end-to-end optimization in a classification task. Salient patches are predicted from input images using the self-attention map of a vision transformer to mimic visual fixations. These patches are then encoded by a trainable U-Net and simulated using the pulse2percept framework to predict visual percepts. By incorporating a learnable encoder, we aim to optimize the visual information transmitted to the retinal implant, addressing both the limited resolution of the electrode array and the distortion between the input stimuli and resulting phosphenes. The predicted percepts are evaluated using the self-supervised DINOv2 foundation model, with an optional learnable linear layer for classification accuracy. On a subset of the ImageNet validation set, the fixation-based framework achieves a classification accuracy of 87.72%, using computational parameters based on a real subject's physiological data, significantly outperforming the downsampling-based accuracy of 40.59% and approaching the healthy upper bound of 92.76%. Our approach shows promising potential for producing more semantically understandable percepts with the limited resolution available in retinal prosthetics.

Title: Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems

• Authors: Jason Hu, Bowen Song, Jeffrey A. Fessler, Liyue Shen
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.11730
• Pdf URL: https://arxiv.org/pdf/2410.11730
• Copy Paste: [[2410.11730]] Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems(https://arxiv.org/abs/2410.11730)
• Keywords: diffusion, self-supervised
• Abstract: Diffusion models have achieved excellent success in solving inverse problems due to their ability to learn strong image priors, but existing approaches require a large training dataset of images that should come from the same distribution as the test dataset. When the training and test distributions are mismatched, artifacts and hallucinations can occur in reconstructed images due to the incorrect priors. In this work, we systematically study out of distribution (OOD) problems where a known training distribution is first provided. We first study the setting where only a single measurement obtained from the unknown test distribution is available. Next we study the setting where a very small sample of data belonging to the test distribution is available, and our goal is still to reconstruct an image from a measurement that came from the test distribution. In both settings, we use a patch-based diffusion prior that learns the image distribution solely from patches. Furthermore, in the first setting, we include a self-supervised loss that helps the network output maintain consistency with the measurement. Extensive experiments show that in both settings, the patch-based method can obtain high quality image reconstructions that can outperform whole-image models and can compete with methods that have access to large in-distribution training datasets. Furthermore, we show how whole-image models are prone to memorization and overfitting, leading to artifacts in the reconstructions, while a patch-based model can resolve these issues.

Title: On the Training Convergence of Transformers for In-Context Classification

• Authors: Wei Shen, Ruida Zhou, Jing Yang, Cong Shen
• Subjects: cs.LG, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.11778
• Pdf URL: https://arxiv.org/pdf/2410.11778
• Copy Paste: [[2410.11778]] On the Training Convergence of Transformers for In-Context Classification(https://arxiv.org/abs/2410.11778)
• Keywords: in-context
• Abstract: While transformers have demonstrated impressive capacities for in-context learning (ICL) in practice, theoretical understanding of the underlying mechanism enabling transformers to perform ICL is still in its infant stage. This work aims to theoretically study the training dynamics of transformers for in-context classification tasks. We demonstrate that, for in-context classification of Gaussian mixtures under certain assumptions, a single-layer transformer trained via gradient descent converges to a globally optimal model at a linear rate. We further quantify the impact of the training and testing prompt lengths on the ICL inference error of the trained transformer. We show that when the lengths of training and testing prompts are sufficiently large, the prediction of the trained transformer approaches the Bayes-optimal classifier. Experimental results corroborate the theoretical findings.

Title: Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability

• Authors: Tsz Ting Chung, Leyang Cui, Lemao Liu, Xinting Huang, Shuming Shi, Dit-Yan Yeung
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11786
• Pdf URL: https://arxiv.org/pdf/2410.11786
• Copy Paste: [[2410.11786]] Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability(https://arxiv.org/abs/2410.11786)
• Keywords: self-supervised, in-context
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in a wide range of natural language processing tasks when leveraging in-context learning. To mitigate the additional computational and financial costs associated with in-context learning, several prompt compression methods have been proposed to compress the in-context learning prompts. Despite their success, these methods face challenges with transferability due to model-specific compression, or rely on external training data, such as GPT-4. In this paper, we investigate the ability of LLMs to develop a unified compression method that discretizes uninformative tokens, utilizing a self-supervised pre-training technique. By introducing a small number of parameters during the continual pre-training, the proposed Selection-p produces a probability for each input token, indicating whether to preserve or discard it. Experiments show Selection-p achieves state-of-the-art performance across numerous classification tasks, achieving compression rates of up to 10 times while experiencing only a marginal 0.8% decrease in performance. Moreover, it exhibits superior transferability to different models compared to prior work. Additionally, we further analyze how Selection-p helps maintain performance on in-context learning with long contexts.

Title: Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices

• Authors: Zhiyuan Ma, Yuzhu Zhang, Guoli Jia, Liangliang Zhao, Yichao Ma, Mingjie Ma, Gaofeng Liu, Kaiyan Zhang, Jianjun Li, Bowen Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11795
• Pdf URL: https://arxiv.org/pdf/2410.11795
• Copy Paste: [[2410.11795]] Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices(https://arxiv.org/abs/2410.11795)
• Keywords: diffusion, generative
• Abstract: As one of the most popular and sought-after generative models in the recent years, diffusion models have sparked the interests of many researchers and steadily shown excellent advantage in various generative tasks such as image synthesis, video generation, molecule design, 3D scene rendering and multimodal generation, relying on their dense theoretical principles and reliable application practices. The remarkable success of these recent efforts on diffusion models comes largely from progressive design principles and efficient architecture, training, inference, and deployment methodologies. However, there has not been a comprehensive and in-depth review to summarize these principles and practices to help the rapid understanding and application of diffusion models. In this survey, we provide a new efficiency-oriented perspective on these existing efforts, which mainly focuses on the profound principles and efficient practices in architecture designs, model training, fast inference and reliable deployment, to guide further theoretical research, algorithm migration and model application for new scenarios in a reader-friendly way. \url{this https URL}

Title: FoundTS: Comprehensive and Unified Benchmarking of Foundation Models for Time Series Forecasting

• Authors: Zhe Li, Xiangfei Qiu, Peng Chen, Yihang Wang, Hanyin Cheng, Yang Shu, Jilin Hu, Chenjuan Guo, Aoying Zhou, Qingsong Wen, Christian S. Jensen, Bin Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11802
• Pdf URL: https://arxiv.org/pdf/2410.11802
• Copy Paste: [[2410.11802]] FoundTS: Comprehensive and Unified Benchmarking of Foundation Models for Time Series Forecasting(https://arxiv.org/abs/2410.11802)
• Keywords: foundation model
• Abstract: Time Series Forecasting (TSF) is key functionality in numerous fields, including in finance, weather services, and energy management. While TSF methods are emerging these days, many of them require domain-specific data collection and model training and struggle with poor generalization performance on new domains. Foundation models aim to overcome this limitation. Pre-trained on large-scale language or time series data, they exhibit promising inferencing capabilities in new or unseen data. This has spurred a surge in new TSF foundation models. We propose a new benchmark, FoundTS, to enable thorough and fair evaluation and comparison of such models. FoundTS covers a variety of TSF foundation models, including those based on large language models and those pretrained on time series. Next, FoundTS supports different forecasting strategies, including zero-shot, few-shot, and full-shot, thereby facilitating more thorough evaluations. Finally, FoundTS offers a pipeline that standardizes evaluation processes such as dataset splitting, loading, normalization, and few-shot sampling, thereby facilitating fair evaluations. Building on this, we report on an extensive evaluation of TSF foundation models on a broad range of datasets from diverse domains and with different statistical characteristics. Specifically, we identify pros and cons and inherent limitations of existing foundation models, and we identify directions for future model design. We make our code and datasets available at this https URL.

Title: SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing

• Authors: Zhiyuan Zhang, DongDong Chen, Jing Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11815
• Pdf URL: https://arxiv.org/pdf/2410.11815
• Copy Paste: [[2410.11815]] SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing(https://arxiv.org/abs/2410.11815)
• Keywords: diffusion, generative
• Abstract: Scene graphs offer a structured, hierarchical representation of images, with nodes and edges symbolizing objects and the relationships among them. It can serve as a natural interface for image editing, dramatically improving precision and flexibility. Leveraging this benefit, we introduce a new framework that integrates large language model (LLM) with Text2Image generative model for scene graph-based image editing. This integration enables precise modifications at the object level and creative recomposition of scenes without compromising overall image integrity. Our approach involves two primary stages: 1) Utilizing a LLM-driven scene parser, we construct an image's scene graph, capturing key objects and their interrelationships, as well as parsing fine-grained attributes such as object masks and descriptions. These annotations facilitate concept learning with a fine-tuned diffusion model, representing each object with an optimized token and detailed description prompt. 2) During the image editing phase, a LLM editing controller guides the edits towards specific areas. These edits are then implemented by an attention-modulated diffusion editor, utilizing the fine-tuned model to perform object additions, deletions, replacements, and adjustments. Through extensive experiments, we demonstrate that our framework significantly outperforms existing image editing methods in terms of editing precision and scene aesthetics.

Title: Jigsaw++: Imagining Complete Shape Priors for Object Reassembly

• Authors: Jiaxin Lu, Gang Hua, Qixing Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11816
• Pdf URL: https://arxiv.org/pdf/2410.11816
• Copy Paste: [[2410.11816]] Jigsaw++: Imagining Complete Shape Priors for Object Reassembly(https://arxiv.org/abs/2410.11816)
• Keywords: generative
• Abstract: The automatic assembly problem has attracted increasing interest due to its complex challenges that involve 3D representation. This paper introduces Jigsaw++, a novel generative method designed to tackle the multifaceted challenges of reconstruction for the reassembly problem. Existing approach focusing primarily on piecewise information for both part and fracture assembly, often overlooking the integration of complete object prior. Jigsaw++ distinguishes itself by learning a category-agnostic shape prior of complete objects. It employs the proposed "retargeting" strategy that effectively leverages the output of any existing assembly method to generate complete shape reconstructions. This capability allows it to function orthogonally to the current methods. Through extensive evaluations on Breaking Bad dataset and PartNet, Jigsaw++ has demonstrated its effectiveness, reducing reconstruction errors and enhancing the precision of shape reconstruction, which sets a new direction for future reassembly model developments.

Title: Improving Long-Text Alignment for Text-to-Image Diffusion Models

• Authors: Luping Liu, Chao Du, Tianyu Pang, Zehan Wang, Chongxuan Li, Dong Xu
• Subjects: cs.CV, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.11817
• Pdf URL: https://arxiv.org/pdf/2410.11817
• Copy Paste: [[2410.11817]] Improving Long-Text Alignment for Text-to-Image Diffusion Models(https://arxiv.org/abs/2410.11817)
• Keywords: diffusion, foundation model
• Abstract: The rapid advancement of text-to-image (T2I) diffusion models has enabled them to generate unprecedented results from given texts. However, as text inputs become longer, existing encoding methods like CLIP face limitations, and aligning the generated images with long texts becomes challenging. To tackle these issues, we propose LongAlign, which includes a segment-level encoding method for processing long texts and a decomposed preference optimization method for effective alignment training. For segment-level encoding, long texts are divided into multiple segments and processed separately. This method overcomes the maximum input length limits of pretrained encoding models. For preference optimization, we provide decomposed CLIP-based preference models to fine-tune diffusion models. Specifically, to utilize CLIP-based preference models for T2I alignment, we delve into their scoring mechanisms and find that the preference scores can be decomposed into two components: a text-relevant part that measures T2I alignment and a text-irrelevant part that assesses other visual aspects of human preference. Additionally, we find that the text-irrelevant part contributes to a common overfitting problem during fine-tuning. To address this, we propose a reweighting strategy that assigns different weights to these two components, thereby reducing overfitting and enhancing alignment. After fine-tuning $512 \times 512$ Stable Diffusion (SD) v1.5 for about 20 hours using our method, the fine-tuned SD outperforms stronger foundation models in T2I alignment, such as PixArt-$\alpha$ and Kandinsky v2.2. The code is available at this https URL.

Title: Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws

• Authors: Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, J. Zico Kolter
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.11820
• Pdf URL: https://arxiv.org/pdf/2410.11820
• Copy Paste: [[2410.11820]] Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws(https://arxiv.org/abs/2410.11820)
• Keywords: foundation model
• Abstract: The composition of pretraining data is a key determinant of foundation models' performance, but there is no standard guideline for allocating a limited computational budget across different data sources. Most current approaches either rely on extensive experiments with smaller models or dynamic data adjustments that also require proxy models, both of which significantly increase the workflow complexity and computational overhead. In this paper, we introduce Adaptive Data Optimization (ADO), an algorithm that optimizes data distributions in an online fashion, concurrent with model training. Unlike existing techniques, ADO does not require external knowledge, proxy models, or modifications to the model update. Instead, ADO uses per-domain scaling laws to estimate the learning potential of each domain during training and adjusts the data mixture accordingly, making it more scalable and easier to integrate. Experiments demonstrate that ADO can achieve comparable or better performance than prior methods while maintaining computational efficiency across different computation scales, offering a practical solution for dynamically adjusting data distribution without sacrificing flexibility or increasing costs. Beyond its practical benefits, ADO also provides a new perspective on data collection strategies via scaling laws.

Title: On the Effectiveness of Dataset Alignment for Fake Image Detection

• Authors: Anirudh Sundara Rajan, Utkarsh Ojha, Jedidiah Schloesser, Yong Jae Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11835
• Pdf URL: https://arxiv.org/pdf/2410.11835
• Copy Paste: [[2410.11835]] On the Effectiveness of Dataset Alignment for Fake Image Detection(https://arxiv.org/abs/2410.11835)
• Keywords: diffusion, generative
• Abstract: As latent diffusion models (LDMs) democratize image generation capabilities, there is a growing need to detect fake images. A good detector should focus on the generative models fingerprints while ignoring image properties such as semantic content, resolution, file format, etc. Fake image detectors are usually built in a data driven way, where a model is trained to separate real from fake images. Existing works primarily investigate network architecture choices and training recipes. In this work, we argue that in addition to these algorithmic choices, we also require a well aligned dataset of real/fake images to train a robust detector. For the family of LDMs, we propose a very simple way to achieve this: we reconstruct all the real images using the LDMs autoencoder, without any denoising operation. We then train a model to separate these real images from their reconstructions. The fakes created this way are extremely similar to the real ones in almost every aspect (e.g., size, aspect ratio, semantic content), which forces the model to look for the LDM decoders artifacts. We empirically show that this way of creating aligned real/fake datasets, which also sidesteps the computationally expensive denoising process, helps in building a detector that focuses less on spurious correlations, something that a very popular existing method is susceptible to. Finally, to demonstrate just how effective the alignment in a dataset can be, we build a detector using images that are not natural objects, and present promising results. Overall, our work identifies the subtle but significant issues that arise when training a fake image detector and proposes a simple and inexpensive solution to address these problems.

Title: High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion

• Authors: Junhwa Hur, Charles Herrmann, Saurabh Saxena, Janne Kontkanen, Wei-Sheng Lai, Yichang Shih, Michael Rubinstein, David J. Fleet, Deqing Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.11838
• Pdf URL: https://arxiv.org/pdf/2410.11838
• Copy Paste: [[2410.11838]] High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion(https://arxiv.org/abs/2410.11838)
• Keywords: diffusion
• Abstract: Despite the recent progress, existing frame interpolation methods still struggle with processing extremely high resolution input and handling challenging cases such as repetitive textures, thin objects, and large motion. To address these issues, we introduce a patch-based cascaded pixel diffusion model for frame interpolation, HiFI, that excels in these scenarios while achieving competitive performance on standard benchmarks. Cascades, which generate a series of images from low- to high-resolution, can help significantly with large or complex motion that require both global context for a coarse solution and detailed context for high resolution output. However, contrary to prior work on cascaded diffusion models which perform diffusion on increasingly large resolutions, we use a single model that always performs diffusion at the same resolution and upsamples by processing patches of the inputs and the prior solution. We show that this technique drastically reduces memory usage at inference time and also allows us to use a single model at test time, solving both frame interpolation and spatial up-sampling, saving training cost. We show that HiFI helps significantly with high resolution and complex repeated textures that require global context. HiFI demonstrates comparable or beyond state-of-the-art performance on multiple benchmarks (Vimeo, Xiph, X-Test, SEPE-8K). On our newly introduced dataset that focuses on particularly challenging cases, HiFI also significantly outperforms other baselines on these cases. Please visit our project page for video results: this https URL