2025-01-23

Title: Federated Discrete Denoising Diffusion Model for Molecular Generation with OpenFL

• Authors: Kevin Ta, Patrick Foley, Mattson Thieme, Abhishek Pandey, Prashant Shah
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.12523
• Pdf URL: https://arxiv.org/pdf/2501.12523
• Copy Paste: [[2501.12523]] Federated Discrete Denoising Diffusion Model for Molecular Generation with OpenFL(https://arxiv.org/abs/2501.12523)
• Keywords: diffusion
• Abstract: Generating unique molecules with biochemically desired properties to serve as viable drug candidates is a difficult task that requires specialized domain expertise. In recent years, diffusion models have shown promising results in accelerating the drug design process through AI-driven molecular generation. However, training these models requires massive amounts of data, which are often isolated in proprietary silos. OpenFL is a federated learning framework that enables privacy-preserving collaborative training across these decentralized data sites. In this work, we present a federated discrete denoising diffusion model that was trained using OpenFL. The federated model achieves comparable performance with a model trained on centralized data when evaluating the uniqueness and validity of the generated molecules. This demonstrates the utility of federated learning in the drug design process. OpenFL is available at: this https URL

Title: How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?

• Authors: Mirali Purohit, Gedeon Muhawenayo, Esther Rolf, Hannah Kerner
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12535
• Pdf URL: https://arxiv.org/pdf/2501.12535
• Copy Paste: [[2501.12535]] How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?(https://arxiv.org/abs/2501.12535)
• Keywords: foundation model
• Abstract: Foundation models have made rapid advances in many domains including Earth observation, where Geospatial Foundation Models (GFMs) can help address global challenges such as climate change, agriculture, and disaster response. Previous work on GFMs focused on tailoring model architecture and pre-text tasks, and did not investigate the impact of pre-training data selection on model performance. However, recent works from other domains show that the pre-training data distribution is an important factor influencing the performance of the foundation models. With this motivation, our research explores how the geographic distribution of pre-training data affects the performance of GFMs. We evaluated several pre-training data distributions by sampling different compositions from a global data pool. Our experiments with two GFMs on downstream tasks indicate that balanced and globally representative data compositions often outperform region-specific sampling, highlighting the importance of diversity and global coverage in pre-training data. Our results suggest that the most appropriate data sampling technique may depend on the specific GFM architecture. These findings will support the development of robust GFMs by incorporating quality pre-training data distributions, ultimately improving machine learning solutions for Earth observation.

Title: Compositional Instruction Following with Language Models and Reinforcement Learning

• Authors: Vanya Cohen, Geraud Nangue Tasse, Nakul Gopalan, Steven James, Matthew Gombolay, Ray Mooney, Benjamin Rosman
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.12539
• Pdf URL: https://arxiv.org/pdf/2501.12539
• Copy Paste: [[2501.12539]] Compositional Instruction Following with Language Models and Reinforcement Learning(https://arxiv.org/abs/2501.12539)
• Keywords: in-context
• Abstract: Combining reinforcement learning with language grounding is challenging as the agent needs to explore the environment while simultaneously learning multiple language-conditioned tasks. To address this, we introduce a novel method: the compositionally-enabled reinforcement learning language agent (CERLLA). Our method reduces the sample complexity of tasks specified with language by leveraging compositional policy representations and a semantic parser trained using reinforcement learning and in-context learning. We evaluate our approach in an environment requiring function approximation and demonstrate compositional generalization to novel tasks. Our method significantly outperforms the previous best non-compositional baseline in terms of sample complexity on 162 tasks designed to test compositional generalization. Our model attains a higher success rate and learns in fewer steps than the non-compositional baseline. It reaches a success rate equal to an oracle policy's upper-bound performance of 92%. With the same number of environment steps, the baseline only reaches a success rate of 80%.

Title: FedGrAINS: Personalized SubGraph Federated Learning with Adaptive Neighbor Sampling

• Authors: Emir Ceyani, Han Xie, Baturalp Buyukates, Carl Yang, Salman Avestimehr
• Subjects: cs.LG, cs.AI, cs.DC, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.12592
• Pdf URL: https://arxiv.org/pdf/2501.12592
• Copy Paste: [[2501.12592]] FedGrAINS: Personalized SubGraph Federated Learning with Adaptive Neighbor Sampling(https://arxiv.org/abs/2501.12592)
• Keywords: generative
• Abstract: Graphs are crucial for modeling relational and biological data. As datasets grow larger in real-world scenarios, the risk of exposing sensitive information increases, making privacy-preserving training methods like federated learning (FL) essential to ensure data security and compliance with privacy regulations. Recently proposed personalized subgraph FL methods have become the de-facto standard for training personalized Graph Neural Networks (GNNs) in a federated manner while dealing with the missing links across clients' subgraphs due to privacy restrictions. However, personalized subgraph FL faces significant challenges due to the heterogeneity in client subgraphs, such as degree distributions among the nodes, which complicate federated training of graph models. To address these challenges, we propose \textit{FedGrAINS}, a novel data-adaptive and sampling-based regularization method for subgraph FL. FedGrAINS leverages generative flow networks (GFlowNets) to evaluate node importance concerning clients' tasks, dynamically adjusting the message-passing step in clients' GNNs. This adaptation reflects task-optimized sampling aligned with a trajectory balance objective. Experimental results demonstrate that the inclusion of \textit{FedGrAINS} as a regularizer consistently improves the FL performance compared to baselines that do not leverage such regularization.

Title: T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation

• Authors: Lijun Li, Zhelun Shi, Xuhao Hu, Bowen Dong, Yiran Qin, Xihui Liu, Lu Sheng, Jing Shao
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.12612
• Pdf URL: https://arxiv.org/pdf/2501.12612
• Copy Paste: [[2501.12612]] T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation(https://arxiv.org/abs/2501.12612)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) models have rapidly advanced, enabling the generation of high-quality images from text prompts across various domains. However, these models present notable safety concerns, including the risk of generating harmful, biased, or private content. Current research on assessing T2I safety remains in its early stages. While some efforts have been made to evaluate models on specific safety dimensions, many critical risks remain unexplored. To address this gap, we introduce T2ISafety, a safety benchmark that evaluates T2I models across three key domains: toxicity, fairness, and bias. We build a detailed hierarchy of 12 tasks and 44 categories based on these three domains, and meticulously collect 70K corresponding prompts. Based on this taxonomy and prompt set, we build a large-scale T2I dataset with 68K manually annotated images and train an evaluator capable of detecting critical risks that previous work has failed to identify, including risks that even ultra-large proprietary models like GPTs cannot correctly detect. We evaluate 12 prominent diffusion models on T2ISafety and reveal several concerns including persistent issues with racial fairness, a tendency to generate toxic content, and significant variation in privacy protection across the models, even with defense methods like concept erasing. Data and evaluator are released under this https URL.

Title: EchoLM: Accelerating LLM Serving with Real-time Knowledge Distillation

• Authors: Yifan Yu, Yu Gan, Lily Tasi, Nikhil Sarda, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Henry M. Levy, David Culler
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.12689
• Pdf URL: https://arxiv.org/pdf/2501.12689
• Copy Paste: [[2501.12689]] EchoLM: Accelerating LLM Serving with Real-time Knowledge Distillation(https://arxiv.org/abs/2501.12689)
• Keywords: in-context
• Abstract: Large language models (LLMs) have excelled in various applications, yet serving them at scale is challenging due to their substantial resource demands and high latency. Our real-world studies reveal that over 60% of user requests to LLMs have semantically similar counterparts, suggesting the potential for knowledge sharing among requests. However, naively caching and reusing past responses leads to large quality degradation. In this paper, we introduce EchoLM, an in-context caching system that leverages historical requests as examples to guide response generation, enabling selective offloading of requests to more efficient LLMs. However, enabling this real-time knowledge transfer leads to intricate tradeoffs between response quality, latency, and system throughput at scale. For a new request, EchoLM identifies similar, high-utility examples and efficiently prepends them to the input for better response. At scale, EchoLM adaptively routes requests to LLMs of varying capabilities, accounting for response quality and serving loads. EchoLM employs a cost-aware cache replay mechanism to improve example quality and coverage offline, maximizing cache utility and runtime efficiency. Evaluations on millions of open-source requests demonstrate that EchoLM has a throughput improvement of 1.4-5.9x while reducing latency by 28-71% without hurting response quality on average.

Title: Anomaly Detection in Double-entry Bookkeeping Data by Federated Learning System with Non-model Sharing Approach

• Authors: Sota Mashiko, Yuji Kawamata, Tomoru Nakayama, Tetsuya Sakurai, Yukihiko Okada
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.12723
• Pdf URL: https://arxiv.org/pdf/2501.12723
• Copy Paste: [[2501.12723]] Anomaly Detection in Double-entry Bookkeeping Data by Federated Learning System with Non-model Sharing Approach(https://arxiv.org/abs/2501.12723)
• Keywords: anomaly
• Abstract: Anomaly detection is crucial in financial auditing and effective detection often requires obtaining large volumes of data from multiple organizations. However, confidentiality concerns hinder data sharing among audit firms. Although the federated learning (FL)-based approach, FedAvg, has been proposed to address this challenge, its use of mutiple communication rounds increases its overhead, limiting its practicality. In this study, we propose a novel framework employing Data Collaboration (DC) analysis -- a non-model share-type FL method -- to streamline model training into a single communication round. Our method first encodes journal entry data via dimensionality reduction to obtain secure intermediate representations, then transforms them into collaboration representations for building an autoencoder that detects anomalies. We evaluate our approach on a synthetic dataset and real journal entry data from multiple organizations. The results show that our method not only outperforms single-organization baselines but also exceeds FedAvg in non-i.i.d. experiments on real journal entry data that closely mirror real-world conditions. By preserving data confidentiality and reducing iterative communication, this study addresses a key auditing challenge -- ensuring data confidentiality while integrating knowledge from multiple audit firms. Our findings represent a significant advance in artificial intelligence-driven auditing and underscore the potential of FL methods in high-security domains.

Title: EvidenceMap: Unleashing the Power of Small Language Models with Evidence Analysis for Biomedical Question Answering

• Authors: Chang Zong, Jian Wan, Lei Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.12746
• Pdf URL: https://arxiv.org/pdf/2501.12746
• Copy Paste: [[2501.12746]] EvidenceMap: Unleashing the Power of Small Language Models with Evidence Analysis for Biomedical Question Answering(https://arxiv.org/abs/2501.12746)
• Keywords: generative
• Abstract: Current LLM-based approaches improve question answering performance by leveraging the internal reasoning abilities of models or incorporating external knowledge. However, when humans address professional problems, it is essential to explicitly analyze the multifaceted relationships from multiple pieces and diverse sources of evidence to achieve better answers. In this study, we propose a novel generative question answering framework for the biomedical domain, named EvidenceMap, which explicitly learns and incorporates evidence analysis with small language models (SLMs). The framework describes an evidence map for each question and fully utilizes an SLM to derive the representation of the supportive evaluation, the logical correlation, and the summarization of the related evidence, which facilitates an analysis-augmented generation with another SLM in an autoregressive way. Extensive experiments have shown that introducing an evidence analysis learning process can significantly outperform larger models and popular LLM reasoning methods.

Title: Certified Guidance for Planning with Deep Generative Models

• Authors: Francesco Giacomarra, Mehran Hosseini, Nicola Paoletti, Francesca Cairoli
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.12815
• Pdf URL: https://arxiv.org/pdf/2501.12815
• Copy Paste: [[2501.12815]] Certified Guidance for Planning with Deep Generative Models(https://arxiv.org/abs/2501.12815)
• Keywords: diffusion, generative
• Abstract: Deep generative models, such as generative adversarial networks and diffusion models, have recently emerged as powerful tools for planning tasks and behavior synthesis in autonomous systems. Various guidance strategies have been introduced to steer the generative process toward outputs that are more likely to satisfy the planning objectives. These strategies avoid the need for model retraining but do not provide any guarantee that the generated outputs will satisfy the desired planning objectives. To address this limitation, we introduce certified guidance, an approach that modifies a generative model, without retraining it, into a new model guaranteed to satisfy a given specification with probability one. We focus on Signal Temporal Logic specifications, which are rich enough to describe nontrivial planning tasks. Our approach leverages neural network verification techniques to systematically explore the latent spaces of the generative models, identifying latent regions that are certifiably correct with respect to the STL property of interest. We evaluate the effectiveness of our method on four planning benchmarks using GANs and diffusion models. Our results confirm that certified guidance produces generative models that are always correct, unlike existing guidance methods that are not certified.

Title: Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks

• Authors: Alessio Quercia, Erenus Yildiz, Zhuo Cao, Kai Krajsek, Abigail Morrison, Ira Assent, Hanno Scharr
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12824
• Pdf URL: https://arxiv.org/pdf/2501.12824
• Copy Paste: [[2501.12824]] Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks(https://arxiv.org/abs/2501.12824)
• Keywords: foundation model
• Abstract: Monocular depth estimation (MDE) is a challenging task in computer vision, often hindered by the cost and scarcity of high-quality labeled datasets. We tackle this challenge using auxiliary datasets from related vision tasks for an alternating training scheme with a shared decoder built on top of a pre-trained vision foundation model, while giving a higher weight to MDE. Through extensive experiments we demonstrate the benefits of incorporating various in-domain auxiliary datasets and tasks to improve MDE quality on average by ~11%. Our experimental analysis shows that auxiliary tasks have different impacts, confirming the importance of task selection, highlighting that quality gains are not achieved by merely adding data. Remarkably, our study reveals that using semantic segmentation datasets as Multi-Label Dense Classification (MLDC) often results in additional quality gains. Lastly, our method significantly improves the data efficiency for the considered MDE datasets, enhancing their quality while reducing their size by at least 80%. This paves the way for using auxiliary data from related tasks to improve MDE quality despite limited availability of high-quality labeled data. Code is available at this https URL.

Title: AMM-Diff: Adaptive Multi-Modality Diffusion Network for Missing Modality Imputation

• Authors: Aghiles Kebaili, Jérôme Lapuyade-Lahorgue, Pierre Vera, Su Ruan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12840
• Pdf URL: https://arxiv.org/pdf/2501.12840
• Copy Paste: [[2501.12840]] AMM-Diff: Adaptive Multi-Modality Diffusion Network for Missing Modality Imputation(https://arxiv.org/abs/2501.12840)
• Keywords: diffusion, self-supervised, generative
• Abstract: In clinical practice, full imaging is not always feasible, often due to complex acquisition protocols, stringent privacy regulations, or specific clinical needs. However, missing MR modalities pose significant challenges for tasks like brain tumor segmentation, especially in deep learning-based segmentation, as each modality provides complementary information crucial for improving accuracy. A promising solution is missing data imputation, where absent modalities are generated from available ones. While generative models have been widely used for this purpose, most state-of-the-art approaches are limited to single or dual target translations, lacking the adaptability to generate missing modalities based on varying input configurations. To address this, we propose an Adaptive Multi-Modality Diffusion Network (AMM-Diff), a novel diffusion-based generative model capable of handling any number of input modalities and generating the missing ones. We designed an Image-Frequency Fusion Network (IFFN) that learns a unified feature representation through a self-supervised pretext task across the full input modalities and their selected high-frequency Fourier components. The proposed diffusion model leverages this representation, encapsulating prior knowledge of the complete modalities, and combines it with an adaptive reconstruction strategy to achieve missing modality completion. Experimental results on the BraTS 2021 dataset demonstrate the effectiveness of our approach.

Title: CrossDiff: Diffusion Probabilistic Model With Cross-conditional Encoder-Decoder for Crack Segmentation

• Authors: Xianglong Shi, Yunhan Jiang, Xiaoheng Jiang, Mingling Xu, Yang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12860
• Pdf URL: https://arxiv.org/pdf/2501.12860
• Copy Paste: [[2501.12860]] CrossDiff: Diffusion Probabilistic Model With Cross-conditional Encoder-Decoder for Crack Segmentation(https://arxiv.org/abs/2501.12860)
• Keywords: diffusion
• Abstract: Crack Segmentation in industrial concrete surfaces is a challenging task because cracks usually exhibit intricate morphology with slender appearances. Traditional segmentation methods often struggle to accurately locate such cracks, leading to inefficiencies in maintenance and repair processes. In this paper, we propose a novel diffusion-based model with a cross-conditional encoder-decoder, named CrossDiff, which is the first to introduce the diffusion probabilistic model for the crack segmentation task. Specifically, CrossDiff integrates a cross-encoder and a cross-decoder into the diffusion model to constitute a cross-shaped diffusion model structure. The cross-encoder enhances the ability to retain crack details and the cross-decoder helps extract the semantic features of cracks. As a result, CrossDiff can better handle slender cracks. Extensive experiments were conducted on five challenging crack datasets including CFD, CrackTree200, DeepCrack, GAPs384, and Rissbilder. The results demonstrate that the proposed CrossDiff model achieves impressive performance, outperforming other state-of-the-art methods by 8.0% in terms of both Dice score and IoU. The code will be open-source soon.

Title: Generative AI Misuse Potential in Cyber Security Education: A Case Study of a UK Degree Program

• Authors: Carlton Shepherd
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.12883
• Pdf URL: https://arxiv.org/pdf/2501.12883
• Copy Paste: [[2501.12883]] Generative AI Misuse Potential in Cyber Security Education: A Case Study of a UK Degree Program(https://arxiv.org/abs/2501.12883)
• Keywords: generative
• Abstract: Recent advances in generative artificial intelligence (AI), such as ChatGPT, Google Gemini, and other large language models (LLMs), pose significant challenges to upholding academic integrity in higher education. This paper investigates the susceptibility of a Master's-level cyber security degree program at a UK Russell Group university, accredited by a leading national body, to LLM misuse. Through the application and extension of a quantitative assessment framework, we identify a high exposure to misuse, particularly in independent project- and report-based assessments. Contributing factors, including block teaching and a predominantly international cohort, are highlighted as potential amplifiers of these vulnerabilities. To address these challenges, we discuss the adoption of LLM-resistant assessments, detection tools, and the importance of fostering an ethical learning environment. These approaches aim to uphold academic standards while preparing students for the complexities of real-world cyber security.

Title: DocTTT: Test-Time Training for Handwritten Document Recognition Using Meta-Auxiliary Learning

• Authors: Wenhao Gu, Li Gu, Ziqiang Wang, Ching Yee Suen, Yang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12898
• Pdf URL: https://arxiv.org/pdf/2501.12898
• Copy Paste: [[2501.12898]] DocTTT: Test-Time Training for Handwritten Document Recognition Using Meta-Auxiliary Learning(https://arxiv.org/abs/2501.12898)
• Keywords: self-supervised
• Abstract: Despite recent significant advancements in Handwritten Document Recognition (HDR), the efficient and accurate recognition of text against complex backgrounds, diverse handwriting styles, and varying document layouts remains a practical challenge. Moreover, this issue is seldom addressed in academic research, particularly in scenarios with minimal annotated data available. In this paper, we introduce the DocTTT framework to address these challenges. The key innovation of our approach is that it uses test-time training to adapt the model to each specific input during testing. We propose a novel Meta-Auxiliary learning approach that combines Meta-learning and self-supervised Masked Autoencoder~(MAE). During testing, we adapt the visual representation parameters using a self-supervised MAE loss. During training, we learn the model parameters using a meta-learning framework, so that the model parameters are learned to adapt to a new input effectively. Experimental results show that our proposed method significantly outperforms existing state-of-the-art approaches on benchmark datasets.

Title: DynamicEarth: How Far are We from Open-Vocabulary Change Detection?

• Authors: Kaiyu Li, Xiangyong Cao, Yupeng Deng, Chao Pang, Zepeng Xin, Deyu Meng, Zhi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12931
• Pdf URL: https://arxiv.org/pdf/2501.12931
• Copy Paste: [[2501.12931]] DynamicEarth: How Far are We from Open-Vocabulary Change Detection?(https://arxiv.org/abs/2501.12931)
• Keywords: foundation model
• Abstract: Monitoring Earth's evolving land covers requires methods capable of detecting changes across a wide range of categories and contexts. Existing change detection methods are hindered by their dependency on predefined classes, reducing their effectiveness in open-world applications. To address this issue, we introduce open-vocabulary change detection (OVCD), a novel task that bridges vision and language to detect changes across any category. Considering the lack of high-quality data and annotation, we propose two training-free frameworks, M-C-I and I-M-C, which leverage and integrate off-the-shelf foundation models for the OVCD task. The insight behind the M-C-I framework is to discover all potential changes and then classify these changes, while the insight of I-M-C framework is to identify all targets of interest and then determine whether their states have changed. Based on these two frameworks, we instantiate to obtain several methods, e.g., SAM-DINOv2-SegEarth-OV, Grounding-DINO-SAM2-DINO, etc. Extensive evaluations on 5 benchmark datasets demonstrate the superior generalization and robustness of our OVCD methods over existing supervised and unsupervised methods. To support continued exploration, we release DynamicEarth, a dedicated codebase designed to advance research and application of OVCD. this https URL

Title: 3D Object Manipulation in a Single Image using Generative Models

• Authors: Ruisi Zhao, Zechuan Zhang, Zongxin Yang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12935
• Pdf URL: https://arxiv.org/pdf/2501.12935
• Copy Paste: [[2501.12935]] 3D Object Manipulation in a Single Image using Generative Models(https://arxiv.org/abs/2501.12935)
• Keywords: diffusion, generative
• Abstract: Object manipulation in images aims to not only edit the object's presentation but also gift objects with motion. Previous methods encountered challenges in concurrently handling static editing and dynamic generation, while also struggling to achieve fidelity in object appearance and scene lighting. In this work, we introduce \textbf{OMG3D}, a novel framework that integrates the precise geometric control with the generative power of diffusion models, thus achieving significant enhancements in visual performance. Our framework first converts 2D objects into 3D, enabling user-directed modifications and lifelike motions at the geometric level. To address texture realism, we propose CustomRefiner, a texture refinement module that pre-train a customized diffusion model, aligning the details and style of coarse renderings of 3D rough model with the original image, further refine the texture. Additionally, we introduce IllumiCombiner, a lighting processing module that estimates and corrects background lighting to match human visual perception, resulting in more realistic shadow effects. Extensive experiments demonstrate the outstanding visual performance of our approach in both static and dynamic scenarios. Remarkably, all these steps can be done using one NVIDIA 3090. Project page is at this https URL

Title: A Novel Tracking Framework for Devices in X-ray Leveraging Supplementary Cue-Driven Self-Supervised Features

• Authors: Saahil Islam, Venkatesh N. Murthy, Dominik Neumann, Serkan Cimen, Puneet Sharma, Andreas Maier, Dorin Comaniciu, Florin C. Ghesu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.12958
• Pdf URL: https://arxiv.org/pdf/2501.12958
• Copy Paste: [[2501.12958]] A Novel Tracking Framework for Devices in X-ray Leveraging Supplementary Cue-Driven Self-Supervised Features(https://arxiv.org/abs/2501.12958)
• Keywords: self-supervised
• Abstract: To restore proper blood flow in blocked coronary arteries via angioplasty procedure, accurate placement of devices such as catheters, balloons, and stents under live fluoroscopy or diagnostic angiography is crucial. Identified balloon markers help in enhancing stent visibility in X-ray sequences, while the catheter tip aids in precise navigation and co-registering vessel structures, reducing the need for contrast in angiography. However, accurate detection of these devices in interventional X-ray sequences faces significant challenges, particularly due to occlusions from contrasted vessels and other devices and distractions from surrounding, resulting in the failure to track such small objects. While most tracking methods rely on spatial correlation of past and current appearance, they often lack strong motion comprehension essential for navigating through these challenging conditions, and fail to effectively detect multiple instances in the scene. To overcome these limitations, we propose a self-supervised learning approach that enhances its spatio-temporal understanding by incorporating supplementary cues and learning across multiple representation spaces on a large dataset. Followed by that, we introduce a generic real-time tracking framework that effectively leverages the pretrained spatio-temporal network and also takes the historical appearance and trajectory data into account. This results in enhanced localization of multiple instances of device landmarks. Our method outperforms state-of-the-art methods in interventional X-ray device tracking, especially stability and robustness, achieving an 87% reduction in max error for balloon marker detection and a 61% reduction in max error for catheter tip detection.

Title: LiT: Delving into a Simplified Linear Diffusion Transformer for Image Generation

• Authors: Jiahao Wang, Ning Kang, Lewei Yao, Mengzhao Chen, Chengyue Wu, Songyang Zhang, Shuchen Xue, Yong Liu, Taiqiang Wu, Xihui Liu, Kaipeng Zhang, Shifeng Zhang, Wenqi Shao, Zhenguo Li, Ping Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.12976
• Pdf URL: https://arxiv.org/pdf/2501.12976
• Copy Paste: [[2501.12976]] LiT: Delving into a Simplified Linear Diffusion Transformer for Image Generation(https://arxiv.org/abs/2501.12976)
• Keywords: diffusion
• Abstract: In commonly used sub-quadratic complexity modules, linear attention benefits from simplicity and high parallelism, making it promising for image synthesis tasks. However, the architectural design and learning strategy for linear attention remain underexplored in this field. In this paper, we offer a suite of ready-to-use solutions for efficient linear diffusion Transformers. Our core contributions include: (1) Simplified Linear Attention using few heads, observing the free-lunch effect of performance without latency increase. (2) Weight inheritance from a fully pre-trained diffusion Transformer: initializing linear Transformer using pre-trained diffusion Transformer and loading all parameters except for those related to linear attention. (3) Hybrid knowledge distillation objective: using a pre-trained diffusion Transformer to help the training of the student linear Transformer, supervising not only the predicted noise but also the variance of the reverse diffusion process. These guidelines lead to our proposed Linear Diffusion Transformer (LiT), an efficient text-to-image Transformer that can be deployed offline on a laptop. Experiments show that in class-conditional 256*256 and 512*512 ImageNet benchmark LiT achieves highly competitive FID while reducing training steps by 80% and 77% compared to DiT. LiT also rivals methods based on Mamba or Gated Linear Attention. Besides, for text-to-image generation, LiT allows for the rapid synthesis of up to 1K resolution photorealistic images. Project page: this https URL.

Title: FlanEC: Exploring Flan-T5 for Post-ASR Error Correction

• Authors: Moreno La Quatra, Valerio Mario Salerno, Yu Tsao, Sabato Marco Siniscalchi
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.12979
• Pdf URL: https://arxiv.org/pdf/2501.12979
• Copy Paste: [[2501.12979]] FlanEC: Exploring Flan-T5 for Post-ASR Error Correction(https://arxiv.org/abs/2501.12979)
• Keywords: generative
• Abstract: In this paper, we present an encoder-decoder model leveraging Flan-T5 for post-Automatic Speech Recognition (ASR) Generative Speech Error Correction (GenSEC), and we refer to it as FlanEC. We explore its application within the GenSEC framework to enhance ASR outputs by mapping n-best hypotheses into a single output sentence. By utilizing n-best lists from ASR models, we aim to improve the linguistic correctness, accuracy, and grammaticality of final ASR transcriptions. Specifically, we investigate whether scaling the training data and incorporating diverse datasets can lead to significant improvements in post-ASR error correction. We evaluate FlanEC using the HyPoradise dataset, providing a comprehensive analysis of the model's effectiveness in this domain. Furthermore, we assess the proposed approach under different settings to evaluate model scalability and efficiency, offering valuable insights into the potential of instruction-tuned encoder-decoder models for this task.

Title: A Probabilistic Model for Self-Supervised Learning

• Authors: Maximilian Fleissner, Pascal Esser, Debarghya Ghoshdastidar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13031
• Pdf URL: https://arxiv.org/pdf/2501.13031
• Copy Paste: [[2501.13031]] A Probabilistic Model for Self-Supervised Learning(https://arxiv.org/abs/2501.13031)
• Keywords: self-supervised, generative
• Abstract: Self-supervised learning (SSL) aims to find meaningful representations from unlabeled data by encoding semantic similarities through data augmentations. Despite its current popularity, theoretical insights about SSL are still scarce. For example, it is not yet known whether commonly used SSL loss functions can be related to a statistical model, much in the same as OLS, generalized linear models or PCA naturally emerge as maximum likelihood estimates of an underlying generative process. In this short paper, we consider a latent variable statistical model for SSL that exhibits an interesting property: Depending on the informativeness of the data augmentations, the MLE of the model either reduces to PCA, or approaches a simple non-contrastive loss. We analyze the model and also empirically illustrate our findings.

Title: One-Class Domain Adaptation via Meta-Learning

• Authors: Stephanie Holly, Thomas Bierweiler, Stefan von Dosky, Ahmed Frikha, Clemens Heitzinger, Jana Eder
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13052
• Pdf URL: https://arxiv.org/pdf/2501.13052
• Copy Paste: [[2501.13052]] One-Class Domain Adaptation via Meta-Learning(https://arxiv.org/abs/2501.13052)
• Keywords: anomaly
• Abstract: The deployment of IoT (Internet of Things) sensor-based machine learning models in industrial systems for anomaly classification tasks poses significant challenges due to distribution shifts, as the training data acquired in controlled laboratory settings may significantly differ from real-time data in production environments. Furthermore, many real-world applications cannot provide a substantial number of labeled examples for each anomalous class in every new environment. It is therefore crucial to develop adaptable machine learning models that can be effectively transferred from one environment to another, enabling rapid adaptation using normal operational data. We extended this problem setting to an arbitrary classification task and formulated the one-class domain adaptation (OC-DA) problem setting. We took a meta-learning approach to tackle the challenge of OC-DA, and proposed a task sampling strategy to adapt any bi-level meta-learning algorithm to OC-DA. We modified the well-established model-agnostic meta-learning (MAML) algorithm and introduced the OC-DA MAML algorithm. We provided a theoretical analysis showing that OC-DA MAML optimizes for meta-parameters that enable rapid one-class adaptation across domains. The OC-DA MAML algorithm is evaluated on the Rainbow-MNIST meta-learning benchmark and on a real-world dataset of vibration-based sensor readings. The results show that OC-DA MAML significantly improves the performance on the target domains and outperforms MAML using the standard task sampling strategy.

Title: Beyond the Lungs: Extending the Field of View in Chest CT with Latent Diffusion Models

• Authors: Lianrui Zuo, Kaiwen Xu, Dingjie Su, Xin Yu, Aravind R. Krishnan, Yihao Liu, Shunxing Bao, Thomas Li, Kim L. Sandler, Fabien Maldonado, Bennett A. Landman
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.13068
• Pdf URL: https://arxiv.org/pdf/2501.13068
• Copy Paste: [[2501.13068]] Beyond the Lungs: Extending the Field of View in Chest CT with Latent Diffusion Models(https://arxiv.org/abs/2501.13068)
• Keywords: diffusion
• Abstract: The interconnection between the human lungs and other organs, such as the liver and kidneys, is crucial for understanding the underlying risks and effects of lung diseases and improving patient care. However, most research chest CT imaging is focused solely on the lungs due to considerations of cost and radiation dose. This restricted field of view (FOV) in the acquired images poses challenges to comprehensive analysis and hinders the ability to gain insights into the impact of lung diseases on other organs. To address this, we propose SCOPE (Spatial Coverage Optimization with Prior Encoding), a novel approach to capture the inter-organ relationships from CT images and extend the FOV of chest CT images. Our approach first trains a variational autoencoder (VAE) to encode 2D axial CT slices individually, then stacks the latent representations of the VAE to form a 3D context for training a latent diffusion model. Once trained, our approach extends the FOV of CT images in the z-direction by generating new axial slices in a zero-shot manner. We evaluated our approach on the National Lung Screening Trial (NLST) dataset, and results suggest that it effectively extends the FOV to include the liver and kidneys, which are not completely covered in the original NLST data acquisition. Quantitative results on a held-out whole-body dataset demonstrate that the generated slices exhibit high fidelity with acquired data, achieving an SSIM of 0.81.

Title: Robust Body Composition Analysis by Generating 3D CT Volumes from Limited 2D Slices

• Authors: Lianrui Zuo, Xin Yu, Dingjie Su, Kaiwen Xu, Aravind R. Krishnan, Yihao Liu, Shunxing Bao, Fabien Maldonado, Luigi Ferrucci, Bennett A. Landman
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.13071
• Pdf URL: https://arxiv.org/pdf/2501.13071
• Copy Paste: [[2501.13071]] Robust Body Composition Analysis by Generating 3D CT Volumes from Limited 2D Slices(https://arxiv.org/abs/2501.13071)
• Keywords: diffusion
• Abstract: Body composition analysis provides valuable insights into aging, disease progression, and overall health conditions. Due to concerns of radiation exposure, two-dimensional (2D) single-slice computed tomography (CT) imaging has been used repeatedly for body composition analysis. However, this approach introduces significant spatial variability that can impact the accuracy and robustness of the analysis. To mitigate this issue and facilitate body composition analysis, this paper presents a novel method to generate 3D CT volumes from limited number of 2D slices using a latent diffusion model (LDM). Our approach first maps 2D slices into a latent representation space using a variational autoencoder. An LDM is then trained to capture the 3D context of a stack of these latent representations. To accurately interpolate intermediateslices and construct a full 3D volume, we utilize body part regression to determine the spatial location and distance between the acquired slices. Experiments on both in-house and public 3D abdominal CT datasets demonstrate that the proposed method significantly enhances body composition analysis compared to traditional 2D-based analysis, with a reduced error rate from 23.3% to 15.2%.

Title: Real-Time Multi-Modal Subcomponent-Level Measurements for Trustworthy System Monitoring and Malware Detection

• Authors: Farshad Khorrami, Ramesh Karri, Prashanth Krishnamurthy
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.13081
• Pdf URL: https://arxiv.org/pdf/2501.13081
• Copy Paste: [[2501.13081]] Real-Time Multi-Modal Subcomponent-Level Measurements for Trustworthy System Monitoring and Malware Detection(https://arxiv.org/abs/2501.13081)
• Keywords: anomaly
• Abstract: With increasingly sophisticated cyber-adversaries able to access a wider repertoire of mechanisms to implant malware such as ransomware, CPU/GPU keyloggers, and stealthy kernel rootkits, there is an urgent need for techniques to detect and mitigate such attacks. While state of the art relies on digital and analog side channel measurements assuming trustworthiness of measurements obtained on the main processor, such an approach has limitations since processor-based side channel measurements are potentially untrustworthy. Sophisticated adversaries (especially in late stage cyber attacks when they have breached the computer and network security systems such as firewalls and antivirus and penetrated the computer's OS) can compromise user-space and kernel-space measurements. To address this key limitation of state of the art, we propose a "subcomponent-level" approach to collect side channel measurements so as to enable robust anomaly detection in a modern computer even when the main processor is compromised. Our proposed approach leverages the fact that modern computers are complex systems with multiple interacting subcomponents and measurements from subcomponents can be used to detect anomalies even when the main processor is no longer trustworthy. We develop mechanisms to obtain time series measurements of activity of several subcomponents and methodologies to process and fuse these measurements for anomaly detection. The subcomponents include network interface controller, GPU, CPU Hardware Performance Counters, CPU power, and keyboard. Our main hypothesis is that subcomponent measurements can enable detection of security threats without requiring a trustworthy main processor. By enabling real-time measurements from multiple subcomponents, the goal is to provide a deeper visibility into system operation, thereby yielding a powerful tool to track system operation and detect anomalies.

Title: Orchid: Image Latent Diffusion for Joint Appearance and Geometry Generation

• Authors: Akshay Krishnan, Xinchen Yan, Vincent Casser, Abhijit Kundu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13087
• Pdf URL: https://arxiv.org/pdf/2501.13087
• Copy Paste: [[2501.13087]] Orchid: Image Latent Diffusion for Joint Appearance and Geometry Generation(https://arxiv.org/abs/2501.13087)
• Keywords: diffusion
• Abstract: Diffusion models are state-of-the-art for image generation. Trained on large datasets, they capture expressive image priors that have been used for tasks like inpainting, depth, and (surface) normal prediction. However, these models are typically trained for one specific task, e.g., a separate model for each of color, depth, and normal prediction. Such models do not leverage the intrinsic correlation between appearance and geometry, often leading to inconsistent predictions. In this paper, we propose using a novel image diffusion prior that jointly encodes appearance and geometry. We introduce a diffusion model Orchid, comprising a Variational Autoencoder (VAE) to encode color, depth, and surface normals to a latent space, and a Latent Diffusion Model (LDM) for generating these joint latents. Orchid directly generates photo-realistic color images, relative depth, and surface normals from user-provided text, and can be used to create image-aligned partial 3D scenes seamlessly. It can also perform image-conditioned tasks like joint monocular depth and normal prediction and is competitive in accuracy to state-of-the-art methods designed for those tasks alone. Lastly, our model learns a joint prior that can be used zero-shot as a regularizer for many inverse problems that entangle appearance and geometry. For example, we demonstrate its effectiveness in color-depth-normal inpainting, showcasing its applicability to problems in 3D generation from sparse views.

Title: Robust Representation Consistency Model via Contrastive Denoising

• Authors: Jiachen Lei, Julius Berner, Jiongxiao Wang, Zhongzhu Chen, Zhongjia Ba, Kui Ren, Jun Zhu, Anima Anandkumar
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13094
• Pdf URL: https://arxiv.org/pdf/2501.13094
• Copy Paste: [[2501.13094]] Robust Representation Consistency Model via Contrastive Denoising(https://arxiv.org/abs/2501.13094)
• Keywords: diffusion, generative
• Abstract: Robustness is essential for deep neural networks, especially in security-sensitive applications. To this end, randomized smoothing provides theoretical guarantees for certifying robustness against adversarial perturbations. Recently, diffusion models have been successfully employed for randomized smoothing to purify noise-perturbed samples before making predictions with a standard classifier. While these methods excel at small perturbation radii, they struggle with larger perturbations and incur a significant computational overhead during inference compared to classical methods. To address this, we reformulate the generative modeling task along the diffusion trajectories in pixel space as a discriminative task in the latent space. Specifically, we use instance discrimination to achieve consistent representations along the trajectories by aligning temporally adjacent points. After fine-tuning based on the learned representations, our model enables implicit denoising-then-classification via a single prediction, substantially reducing inference costs. We conduct extensive experiments on various datasets and achieve state-of-the-art performance with minimal computation budget during inference. For example, our method outperforms the certified accuracy of diffusion-based methods on ImageNet across all perturbation radii by 5.3% on average, with up to 11.6% at larger radii, while reducing inference costs by 85$\times$ on average. Codes are available at: this https URL.

Title: VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding

• Authors: Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, Deli Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13106
• Pdf URL: https://arxiv.org/pdf/2501.13106
• Copy Paste: [[2501.13106]] VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding(https://arxiv.org/abs/2501.13106)
• Keywords: foundation model
• Abstract: In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation model for image and video understanding. The core design philosophy of VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the vision-centric training paradigm and vision-centric framework design. The key insight of our vision-centric training paradigm is that high-quality image-text data is crucial for both image and video understanding. Instead of preparing massive video-text datasets, we focus on constructing large-scale and high-quality image-text datasets. VideoLLaMA3 has four training stages: 1) vision-centric alignment stage, which warms up the vision encoder and projector; 2) vision-language pretraining stage, which jointly tunes the vision encoder, projector, and LLM with large-scale image-text data covering multiple types (including scene images, documents, charts) as well as text-only data. 3) multi-task fine-tuning stage, which incorporates image-text SFT data for downstream tasks and video-text data to establish a foundation for video understanding. 4) video-centric fine-tuning, which further improves the model's capability in video understanding. As for the framework design, to better capture fine-grained details in images, the pretrained vision encoder is adapted to encode images of varying sizes into vision tokens with corresponding numbers, rather than a fixed number of tokens. For video inputs, we reduce the number of vision tokens according to their similarity so that the representation of videos will be more precise and compact. Benefit from vision-centric designs, VideoLLaMA3 achieves compelling performances in both image and video understanding benchmarks.

Title: Accelerate High-Quality Diffusion Models with Inner Loop Feedback

• Authors: Matthew Gwilliam, Han Cai, Di Wu, Abhinav Shrivastava, Zhiyu Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.13107
• Pdf URL: https://arxiv.org/pdf/2501.13107
• Copy Paste: [[2501.13107]] Accelerate High-Quality Diffusion Models with Inner Loop Feedback(https://arxiv.org/abs/2501.13107)
• Keywords: diffusion
• Abstract: We propose Inner Loop Feedback (ILF), a novel approach to accelerate diffusion models' inference. ILF trains a lightweight module to predict future features in the denoising process by leveraging the outputs from a chosen diffusion backbone block at a given time step. This approach exploits two key intuitions; (1) the outputs of a given block at adjacent time steps are similar, and (2) performing partial computations for a step imposes a lower burden on the model than skipping the step entirely. Our method is highly flexible, since we find that the feedback module itself can simply be a block from the diffusion backbone, with all settings copied. Its influence on the diffusion forward can be tempered with a learnable scaling factor from zero initialization. We train this module using distillation losses; however, unlike some prior work where a full diffusion backbone serves as the student, our model freezes the backbone, training only the feedback module. While many efforts to optimize diffusion models focus on achieving acceptable image quality in extremely few steps (1-4 steps), our emphasis is on matching best case results (typically achieved in 20 steps) while significantly reducing runtime. ILF achieves this balance effectively, demonstrating strong performance for both class-to-image generation with diffusion transformer (DiT) and text-to-image generation with DiT-based PixArt-alpha and PixArt-sigma. The quality of ILF's 1.7x-1.8x speedups are confirmed by FID, CLIP score, CLIP Image Quality Assessment, ImageReward, and qualitative comparisons.