2025-01-31

Title: Shared DIFF Transformer

• Authors: Yueyang Cang, Yuhang Liu, Xiaoteng Zhang, Xiangju Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.17900
• Pdf URL: https://arxiv.org/pdf/2501.17900
• Copy Paste: [[2501.17900]] Shared DIFF Transformer(https://arxiv.org/abs/2501.17900)
• Keywords: in-context
• Abstract: DIFF Transformer improves attention allocation by enhancing focus on relevant context while suppressing noise. It introduces a differential attention mechanism that calculates the difference between two independently generated attention distributions, effectively reducing noise and promoting sparse attention patterns. However, the independent signal generation in DIFF Transformer results in parameter redundancy and suboptimal utilization of information. In this work, we propose Shared DIFF Transformer, which draws on the idea of a differential amplifier by introducing a shared base matrix to model global patterns and incorporating low-rank updates to enhance task-specific flexibility. This design significantly reduces parameter redundancy, improves efficiency, and retains strong noise suppression capabilities. Experimental results show that, compared to DIFF Transformer, our method achieves better performance in tasks such as long-sequence modeling, key information retrieval, and in-context learning. Our work provides a novel and efficient approach to optimizing differential attention mechanisms and advancing robust Transformer architectures.

Title: KoopAGRU: A Koopman-based Anomaly Detection in Time-Series using Gated Recurrent Units

• Authors: Issam Ait Yahia, Ismail Berrada
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.17976
• Pdf URL: https://arxiv.org/pdf/2501.17976
• Copy Paste: [[2501.17976]] KoopAGRU: A Koopman-based Anomaly Detection in Time-Series using Gated Recurrent Units(https://arxiv.org/abs/2501.17976)
• Keywords: anomaly
• Abstract: Anomaly detection in real-world time-series data is a challenging task due to the complex and nonlinear temporal dynamics involved. This paper introduces KoopAGRU, a new deep learning model designed to tackle this problem by combining Fast Fourier Transform (FFT), Deep Dynamic Mode Decomposition (DeepDMD), and Koopman theory. FFT allows KoopAGRU to decompose temporal data into time-variant and time-invariant components providing precise modeling of complex patterns. To better control these two components, KoopAGRU utilizes Gate Recurrent Unit (GRU) encoders to learn Koopman observables, enhancing the detection capability across multiple temporal scales. KoopAGRU is trained in a single process and offers fast inference times. Extensive tests on various benchmark datasets show that KoopAGRU outperforms other leading methods, achieving a new average F1-score of 90.88\% on the well-known anomalies detection task of times series datasets, and proves to be efficient and reliable in detecting anomalies in real-world scenarios.

Title: Generative AI for Vision: A Comprehensive Study of Frameworks and Applications

• Authors: Fouad Bousetouane
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18033
• Pdf URL: https://arxiv.org/pdf/2501.18033
• Copy Paste: [[2501.18033]] Generative AI for Vision: A Comprehensive Study of Frameworks and Applications(https://arxiv.org/abs/2501.18033)
• Keywords: diffusion, generative
• Abstract: Generative AI is transforming image synthesis, enabling the creation of high-quality, diverse, and photorealistic visuals across industries like design, media, healthcare, and autonomous systems. Advances in techniques such as image-to-image translation, text-to-image generation, domain transfer, and multimodal alignment have broadened the scope of automated visual content creation, supporting a wide spectrum of applications. These advancements are driven by models like Generative Adversarial Networks (GANs), conditional frameworks, and diffusion-based approaches such as Stable Diffusion. This work presents a structured classification of image generation techniques based on the nature of the input, organizing methods by input modalities like noisy vectors, latent representations, and conditional inputs. We explore the principles behind these models, highlight key frameworks including DALL-E, ControlNet, and DeepSeek Janus-Pro, and address challenges such as computational costs, data biases, and output alignment with user intent. By offering this input-centric perspective, this study bridges technical depth with practical insights, providing researchers and practitioners with a comprehensive resource to harness generative AI for real-world applications.

Title: SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders

• Authors: Bartosz Cywiński, Kamil Deja
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18052
• Pdf URL: https://arxiv.org/pdf/2501.18052
• Copy Paste: [[2501.18052]] SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders(https://arxiv.org/abs/2501.18052)
• Keywords: diffusion
• Abstract: Recent machine unlearning approaches offer promising solution for removing unwanted concepts from diffusion models. However, traditional methods, which largely rely on fine-tuning, provide little insight into the changes they introduce to the base model, making it unclear whether concepts are truly removed or only masked. In this work, we introduce SAeUron, a novel method leveraging features learned by sparse autoencoders (SAEs) to unlearn unwanted concepts in text-to-image diffusion models. First, we demonstrate that SAEs, trained in an unsupervised manner on activations from multiple denoising timesteps of the diffusion model, capture sparse and interpretable features corresponding to specific concepts. Building on this, we propose a method of selecting concept-specific features. This enables precise interventions on the model's activations to block targeted content while preserving the model's overall performance. Evaluation on the competitive UnlearnCanvas benchmark on object and style unlearning highlights SAeUron's state-of-the-art performance. Moreover, we show that with a single SAE, we can remove multiple concepts simultaneously and that in contrast to other methods, SAeUron dismisses the possibility of generating unwanted content, even under adversarial attack.

Title: Current Pathology Foundation Models are unrobust to Medical Center Differences

• Authors: Edwin D. de Jong, Eric Marcus, Jonas Teuwen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18055
• Pdf URL: https://arxiv.org/pdf/2501.18055
• Copy Paste: [[2501.18055]] Current Pathology Foundation Models are unrobust to Medical Center Differences(https://arxiv.org/abs/2501.18055)
• Keywords: foundation model
• Abstract: Pathology Foundation Models (FMs) hold great promise for healthcare. Before they can be used in clinical practice, it is essential to ensure they are robust to variations between medical centers. We measure whether pathology FMs focus on biological features like tissue and cancer type, or on the well known confounding medical center signatures introduced by staining procedure and other differences. We introduce the Robustness Index. This novel robustness metric reflects to what degree biological features dominate confounding features. Ten current publicly available pathology FMs are evaluated. We find that all current pathology foundation models evaluated represent the medical center to a strong degree. Significant differences in the robustness index are observed. Only one model so far has a robustness index greater than one, meaning biological features dominate confounding features, but only slightly. A quantitative approach to measure the influence of medical center differences on FM-based prediction performance is described. We analyze the impact of unrobustness on classification performance of downstream models, and find that cancer-type classification errors are not random, but specifically attributable to same-center confounders: images of other classes from the same medical center. We visualize FM embedding spaces, and find these are more strongly organized by medical centers than by biological factors. As a consequence, the medical center of origin is predicted more accurately than the tissue source and cancer type. The robustness index introduced here is provided with the aim of advancing progress towards clinical adoption of robust and reliable pathology FMs.

Title: Diverse Preference Optimization

• Authors: Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, Ilia Kulikov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18101
• Pdf URL: https://arxiv.org/pdf/2501.18101
• Copy Paste: [[2501.18101]] Diverse Preference Optimization(https://arxiv.org/abs/2501.18101)
• Keywords: generative
• Abstract: Post-training of language models, either through reinforcement learning, preference optimization or supervised finetuning, tends to sharpen the output probability distribution and reduce the diversity of generated responses. This is particularly a problem for creative generative tasks where varied responses are desired. %This impacts the ability to generate high quality synthetic data which is becoming a vital component of model training. In this work we introduce Diverse Preference Optimization (DivPO), an online optimization method which learns to generate much more diverse responses than standard pipelines, while maintaining the quality of the generations. In DivPO, preference pairs are selected by first considering a pool of responses, and a measure of diversity among them, and selecting chosen examples as being more rare but high quality, while rejected examples are more common, but low quality. DivPO results in generating 45.6% more diverse persona attributes, and an 74.6% increase in story diversity, while maintaining similar win rates as standard baselines.

Title: Self-supervised Quantized Representation for Seamlessly Integrating Knowledge Graphs with Large Language Models

• Authors: Qika Lin, Tianzhe Zhao, Kai He, Zhen Peng, Fangzhi Xu, Ling Huang, Jingying Ma, Mengling Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18119
• Pdf URL: https://arxiv.org/pdf/2501.18119
• Copy Paste: [[2501.18119]] Self-supervised Quantized Representation for Seamlessly Integrating Knowledge Graphs with Large Language Models(https://arxiv.org/abs/2501.18119)
• Keywords: self-supervised
• Abstract: Due to the presence of the natural gap between Knowledge Graph (KG) structures and the natural language, the effective integration of holistic structural information of KGs with Large Language Models (LLMs) has emerged as a significant question. To this end, we propose a two-stage framework to learn and apply quantized codes for each entity, aiming for the seamless integration of KGs with LLMs. Firstly, a self-supervised quantized representation (SSQR) method is proposed to compress both KG structural and semantic knowledge into discrete codes (\ie, tokens) that align the format of language sentences. We further design KG instruction-following data by viewing these learned codes as features to directly input to LLMs, thereby achieving seamless integration. The experiment results demonstrate that SSQR outperforms existing unsupervised quantized methods, producing more distinguishable codes. Further, the fine-tuned LLaMA2 and LLaMA3.1 also have superior performance on KG link prediction and triple classification tasks, utilizing only 16 tokens per entity instead of thousands in conventional prompting methods.

Title: Battery State of Health Estimation Using LLM Framework

• Authors: Aybars Yunusoglu, Dexter Le, Karn Tiwari, Murat Isik, I. Can Dikmen
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2501.18123
• Pdf URL: https://arxiv.org/pdf/2501.18123
• Copy Paste: [[2501.18123]] Battery State of Health Estimation Using LLM Framework(https://arxiv.org/abs/2501.18123)
• Keywords: anomaly
• Abstract: Battery health monitoring is critical for the efficient and reliable operation of electric vehicles (EVs). This study introduces a transformer-based framework for estimating the State of Health (SoH) and predicting the Remaining Useful Life (RUL) of lithium titanate (LTO) battery cells by utilizing both cycle-based and instantaneous discharge data. Testing on eight LTO cells under various cycling conditions over 500 cycles, we demonstrate the impact of charge durations on energy storage trends and apply Differential Voltage Analysis (DVA) to monitor capacity changes (dQ/dV) across voltage ranges. Our LLM model achieves superior performance, with a Mean Absolute Error (MAE) as low as 0.87\% and varied latency metrics that support efficient processing, demonstrating its strong potential for real-time integration into EVs. The framework effectively identifies early signs of degradation through anomaly detection in high-resolution data, facilitating predictive maintenance to prevent sudden battery failures and enhance energy efficiency.

Title: Continually Evolved Multimodal Foundation Models for Cancer Prognosis

• Authors: Jie Peng, Shuang Zhou, Longwei Yang, Yiran Song, Mohan Zhang, Kaixiong Zhou, Feng Xie, Mingquan Lin, Rui Zhang, Tianlong Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18170
• Pdf URL: https://arxiv.org/pdf/2501.18170
• Copy Paste: [[2501.18170]] Continually Evolved Multimodal Foundation Models for Cancer Prognosis(https://arxiv.org/abs/2501.18170)
• Keywords: foundation model
• Abstract: Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates. To enhance prediction accuracy, previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information. However, existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals, thus rendering sub-optimal generalizability and limited utility in real-world applications. Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities. To address these, we propose a continually evolving multi-modal foundation model. Extensive experiments on the TCGA dataset demonstrate the effectiveness of our approach, highlighting its potential to advance cancer prognosis by enabling robust and adaptive multimodal integration.

Title: In-Context Learning of Polynomial Kernel Regression in Transformers with GLU Layers

• Authors: Haoyuan Sun, Ali Jadbabaie, Navid Azizan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18187
• Pdf URL: https://arxiv.org/pdf/2501.18187
• Copy Paste: [[2501.18187]] In-Context Learning of Polynomial Kernel Regression in Transformers with GLU Layers(https://arxiv.org/abs/2501.18187)
• Keywords: in-context
• Abstract: Transformer-based models have demonstrated remarkable ability in in-context learning (ICL), where they can adapt to unseen tasks from a prompt with a few examples, without requiring parameter updates. Recent research has provided insight into how linear Transformers can perform ICL by implementing gradient descent estimators. In particular, it has been shown that the optimal linear self-attention (LSA) mechanism can implement one step of gradient descent with respect to a linear least-squares objective when trained on random linear regression tasks. However, the theoretical understanding of ICL for nonlinear function classes remains limited. In this work, we address this gap by first showing that LSA is inherently restricted to solving linear least-squares objectives and thus, the solutions in prior works cannot readily extend to nonlinear ICL tasks. To overcome this limitation, drawing inspiration from modern architectures, we study a mechanism that combines LSA with GLU-like feed-forward layers and show that this allows the model to perform one step of gradient descent on a polynomial kernel regression. Further, we characterize the scaling behavior of the resulting Transformer model, highlighting the necessary model size to effectively handle quadratic ICL tasks. Our findings highlight the distinct roles of attention and feed-forward layers in nonlinear ICL and identify key challenges when extending ICL to nonlinear function classes.

Title: GDformer: Going Beyond Subsequence Isolation for Multivariate Time Series Anomaly Detection

• Authors: Qingxiang Liu, Chenghao Liu, Sheng Sun, Di Yao, Yuxuan Liang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18196
• Pdf URL: https://arxiv.org/pdf/2501.18196
• Copy Paste: [[2501.18196]] GDformer: Going Beyond Subsequence Isolation for Multivariate Time Series Anomaly Detection(https://arxiv.org/abs/2501.18196)
• Keywords: anomaly
• Abstract: Unsupervised anomaly detection of multivariate time series is a challenging task, given the requirements of deriving a compact detection criterion without accessing the anomaly points. The existing methods are mainly based on reconstruction error or association divergence, which are both confined to isolated subsequences with limited horizons, hardly promising unified series-level criterion. In this paper, we propose the Global Dictionary-enhanced Transformer (GDformer) with a renovated dictionary-based cross attention mechanism to cultivate the global representations shared by all normal points in the entire series. Accordingly, the cross-attention maps reflect the correlation weights between the point and global representations, which naturally leads to the representation-wise similarity-based detection criterion. To foster more compact detection boundary, prototypes are introduced to capture the distribution of normal point-global correlation weights. GDformer consistently achieves state-of-the-art unsupervised anomaly detection performance on five real-world benchmark datasets. Further experiments validate the global dictionary has great transferability among various datasets. The code is available at this https URL.

Title: Free-T2M: Frequency Enhanced Text-to-Motion Diffusion Model With Consistency Loss

• Authors: Wenshuo Chen, Haozhe Jia, Songning Lai, Keming Wu, Hongru Xiao, Lijie Hu, Yutao Yue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18232
• Pdf URL: https://arxiv.org/pdf/2501.18232
• Copy Paste: [[2501.18232]] Free-T2M: Frequency Enhanced Text-to-Motion Diffusion Model With Consistency Loss(https://arxiv.org/abs/2501.18232)
• Keywords: diffusion
• Abstract: Rapid progress in text-to-motion generation has been largely driven by diffusion models. However, existing methods focus solely on temporal modeling, thereby overlooking frequency-domain analysis. We identify two key phases in motion denoising: the **semantic planning stage** and the **fine-grained improving stage**. To address these phases effectively, we propose **Fre**quency **e**nhanced **t**ext-**to**-**m**otion diffusion model (**Free-T2M**), incorporating stage-specific consistency losses that enhance the robustness of static features and improve fine-grained accuracy. Extensive experiments demonstrate the effectiveness of our method. Specifically, on StableMoFusion, our method reduces the FID from **0.189** to **0.051**, establishing a new SOTA performance within the diffusion architecture. These findings highlight the importance of incorporating frequency-domain insights into text-to-motion generation for more precise and robust results.

Title: A Video-grounded Dialogue Dataset and Metric for Event-driven Activities

• Authors: Wiradee Imrattanatrai, Masaki Asada, Kimihiro Hasegawa, Zhi-Qi Cheng, Ken Fukuda, Teruko Mitamura
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18324
• Pdf URL: https://arxiv.org/pdf/2501.18324
• Copy Paste: [[2501.18324]] A Video-grounded Dialogue Dataset and Metric for Event-driven Activities(https://arxiv.org/abs/2501.18324)
• Keywords: foundation model
• Abstract: This paper presents VDAct, a dataset for a Video-grounded Dialogue on Event-driven Activities, alongside VDEval, a session-based context evaluation metric specially designed for the task. Unlike existing datasets, VDAct includes longer and more complex video sequences that depict a variety of event-driven activities that require advanced contextual understanding for accurate response generation. The dataset comprises 3,000 dialogues with over 30,000 question-and-answer pairs, derived from 1,000 videos with diverse activity scenarios. VDAct displays a notably challenging characteristic due to its broad spectrum of activity scenarios and wide range of question types. Empirical studies on state-of-the-art vision foundation models highlight their limitations in addressing certain question types on our dataset. Furthermore, VDEval, which integrates dialogue session history and video content summaries extracted from our supplementary Knowledge Graphs to evaluate individual responses, demonstrates a significantly higher correlation with human assessments on the VDAct dataset than existing evaluation metrics that rely solely on the context of single dialogue turns.

Title: Causal Inference Real-Time Anomaly Detection with Synthetic Anomaly Monitoring (SAM)

• Authors: Emanuele Luzio, Moacir Antonelli Ponti
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18417
• Pdf URL: https://arxiv.org/pdf/2501.18417
• Copy Paste: [[2501.18417]] Causal Inference Real-Time Anomaly Detection with Synthetic Anomaly Monitoring (SAM)(https://arxiv.org/abs/2501.18417)
• Keywords: anomaly
• Abstract: Anomaly detection is essential for identifying rare and significant events across diverse domains such as finance, cybersecurity, and network monitoring. This paper presents Synthetic Anomaly Monitoring (SAM), an innovative approach that applies synthetic control methods from causal inference to improve both the accuracy and interpretability of anomaly detection processes. By modeling normal behavior through the treatment of each feature as a control unit, SAM identifies anomalies as deviations within this causal framework. We conducted extensive experiments comparing SAM with established benchmark models, including Isolation Forest, Local Outlier Factor (LOF), k-Nearest Neighbors (kNN), and One-Class Support Vector Machine (SVM), across five diverse datasets, including Credit Card Fraud, HTTP Dataset CSIC 2010, and KDD Cup 1999, among others. Our results demonstrate that SAM consistently delivers robust performance, highlighting its potential as a powerful tool for real-time anomaly detection in dynamic and complex environments.

Title: SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer

• Authors: Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, Bingchen Liu, Daquan Zhou, Song Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18427
• Pdf URL: https://arxiv.org/pdf/2501.18427
• Copy Paste: [[2501.18427]] SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer(https://arxiv.org/abs/2501.18427)
• Keywords: diffusion
• Abstract: This paper presents SANA-1.5, a linear Diffusion Transformer for efficient scaling in text-to-image generation. Building upon SANA-1.0, we introduce three key innovations: (1) Efficient Training Scaling: A depth-growth paradigm that enables scaling from 1.6B to 4.8B parameters with significantly reduced computational resources, combined with a memory-efficient 8-bit optimizer. (2) Model Depth Pruning: A block importance analysis technique for efficient model compression to arbitrary sizes with minimal quality loss. (3) Inference-time Scaling: A repeated sampling strategy that trades computation for model capacity, enabling smaller models to match larger model quality at inference time. Through these strategies, SANA-1.5 achieves a text-image alignment score of 0.72 on GenEval, which can be further improved to 0.80 through inference scaling, establishing a new SoTA on GenEval benchmark. These innovations enable efficient model scaling across different compute budgets while maintaining high quality, making high-quality image generation more accessible.

Title: GENIE: Generative Note Information Extraction model for structuring EHR data

• Authors: Huaiyuan Ying, Hongyi Yuan, Jinsen Lu, Zitian Qu, Yang Zhao, Zhengyun Zhao, Isaac Kohane, Tianxi Cai, Sheng Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.18435
• Pdf URL: https://arxiv.org/pdf/2501.18435
• Copy Paste: [[2501.18435]] GENIE: Generative Note Information Extraction model for structuring EHR data(https://arxiv.org/abs/2501.18435)
• Keywords: generative
• Abstract: Electronic Health Records (EHRs) hold immense potential for advancing healthcare, offering rich, longitudinal data that combines structured information with valuable insights from unstructured clinical notes. However, the unstructured nature of clinical text poses significant challenges for secondary applications. Traditional methods for structuring EHR free-text data, such as rule-based systems and multi-stage pipelines, are often limited by their time-consuming configurations and inability to adapt across clinical notes from diverse healthcare settings. Few systems provide a comprehensive attribute extraction for terminologies. While giant large language models (LLMs) like GPT-4 and LLaMA 405B excel at structuring tasks, they are slow, costly, and impractical for large-scale use. To overcome these limitations, we introduce GENIE, a Generative Note Information Extraction system that leverages LLMs to streamline the structuring of unstructured clinical text into usable data with standardized format. GENIE processes entire paragraphs in a single pass, extracting entities, assertion statuses, locations, modifiers, values, and purposes with high accuracy. Its unified, end-to-end approach simplifies workflows, reduces errors, and eliminates the need for extensive manual intervention. Using a robust data preparation pipeline and fine-tuned small scale LLMs, GENIE achieves competitive performance across multiple information extraction tasks, outperforming traditional tools like cTAKES and MetaMap and can handle extra attributes to be extracted. GENIE strongly enhances real-world applicability and scalability in healthcare systems. By open-sourcing the model and test data, we aim to encourage collaboration and drive further advancements in EHR structurization.

Title: Clustering Properties of Self-Supervised Learning

• Authors: Xi Weng, Jianing An, Xudong Ma, Binhang Qi, Jie Luo, Xi Yang, Jin Song Dong, Lei Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.18452
• Pdf URL: https://arxiv.org/pdf/2501.18452
• Copy Paste: [[2501.18452]] Clustering Properties of Self-Supervised Learning(https://arxiv.org/abs/2501.18452)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich representations with strong clustering properties, magically in the absence of label supervision. Despite this, few of them have explored leveraging these untapped properties to improve themselves. In this paper, we provide an evidence through various metrics that the encoder's output $encoding$ exhibits superior and more stable clustering properties compared to other components. Building on this insight, we propose a novel positive-feedback SSL method, termed Representation Soft Assignment (ReSA), which leverages the model's clustering properties to promote learning in a self-guided manner. Extensive experiments on standard SSL benchmarks reveal that models pretrained with ReSA outperform other state-of-the-art SSL methods by a significant margin. Finally, we analyze how ReSA facilitates better clustering properties, demonstrating that it effectively enhances clustering performance at both fine-grained and coarse-grained levels, shaping representations that are inherently more structured and semantically meaningful.

Title: Tuning Vision Foundation Model via Test-Time Prompt-Guided Training for VFSS Segmentations

• Authors: Chengxi Zeng, David Smithard, Alberto M Gambaruto, Tilo Burghardt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18474
• Pdf URL: https://arxiv.org/pdf/2501.18474
• Copy Paste: [[2501.18474]] Tuning Vision Foundation Model via Test-Time Prompt-Guided Training for VFSS Segmentations(https://arxiv.org/abs/2501.18474)
• Keywords: self-supervised, foundation model
• Abstract: Vision foundation models have demonstrated exceptional generalization capabilities in segmentation tasks for both generic and specialized images. However, a performance gap persists between foundation models and task-specific, specialized models. Fine-tuning foundation models on downstream datasets is often necessary to bridge this gap. Unfortunately, obtaining fully annotated ground truth for downstream datasets is both challenging and costly. To address this limitation, we propose a novel test-time training paradigm that enhances the performance of foundation models on downstream datasets without requiring full annotations. Specifically, our method employs simple point prompts to guide a test-time semi-self-supervised training task. The model learns by resolving the ambiguity of the point prompt through various augmentations. This approach directly tackles challenges in the medical imaging field, where acquiring annotations is both time-intensive and expensive. We conducted extensive experiments on our new Videofluoroscopy dataset (VFSS-5k) for the instance segmentation task, achieving an average Dice coefficient of 0.868 across 12 anatomies with a single model.

Title: CryptoDNA: A Machine Learning Paradigm for DDoS Detection in Healthcare IoT, Inspired by crypto jacking prevention Models

• Authors: Zag ElSayed, Ahmed Abdelgawad, Nelly Elsayed
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.18549
• Pdf URL: https://arxiv.org/pdf/2501.18549
• Copy Paste: [[2501.18549]] CryptoDNA: A Machine Learning Paradigm for DDoS Detection in Healthcare IoT, Inspired by crypto jacking prevention Models(https://arxiv.org/abs/2501.18549)
• Keywords: anomaly
• Abstract: The rapid integration of the Internet of Things (IoT) and Internet of Medical (IoM) devices in the healthcare industry has markedly improved patient care and hospital operations but has concurrently brought substantial risks. Distributed Denial-of-Service (DDoS) attacks present significant dangers, jeopardizing operational stability and patient safety. This study introduces CryptoDNA, an innovative machine learning detection framework influenced by cryptojacking detection methods, designed to identify and alleviate DDoS attacks in healthcare IoT settings. The proposed approach relies on behavioral analytics, including atypical resource usage and network activity patterns. Key features derived from cryptojacking-inspired methodologies include entropy-based analysis of traffic, time-series monitoring of device performance, and dynamic anomaly detection. A lightweight architecture ensures inter-compatibility with resource-constrained IoT devices while maintaining high detection accuracy. The proposed architecture and model were tested in real-world and synthetic datasets to demonstrate the model's superior performance, achieving over 96% accuracy with minimal computational overhead. Comparative analysis reveals its resilience against emerging attack vectors and scalability across diverse device ecosystems. By bridging principles from cryptojacking and DDoS detection, CryptoDNA offers a robust, innovative solution to fortify the healthcare IoT landscape against evolving cyber threats and highlights the potential of interdisciplinary approaches in adaptive cybersecurity defense mechanisms for critical healthcare infrastructures.

Title: DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models

• Authors: Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Zhi-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, Zian Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.18590
• Pdf URL: https://arxiv.org/pdf/2501.18590
• Copy Paste: [[2501.18590]] DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models(https://arxiv.org/abs/2501.18590)
• Keywords: diffusion
• Abstract: Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios. Therefore, we introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework. Leveraging powerful video diffusion model priors, the inverse rendering model accurately estimates G-buffers from real-world videos, providing an interface for image editing tasks, and training data for the rendering model. Conversely, our rendering model generates photorealistic images from G-buffers without explicit light transport simulation. Experiments demonstrate that DiffusionRenderer effectively approximates inverse and forwards rendering, consistently outperforming the state-of-the-art. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.

Title: Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models

• Authors: Hao Dong, Moru Liu, Kaiyang Zhou, Eleni Chatzi, Juho Kannala, Cyrill Stachniss, Olga Fink
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.18592
• Pdf URL: https://arxiv.org/pdf/2501.18592
• Copy Paste: [[2501.18592]] Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models(https://arxiv.org/abs/2501.18592)
• Keywords: foundation model
• Abstract: In real-world scenarios, achieving domain adaptation and generalization poses significant challenges, as models must adapt to or generalize across unknown target distributions. Extending these capabilities to unseen multimodal distributions, i.e., multimodal domain adaptation and generalization, is even more challenging due to the distinct characteristics of different modalities. Significant progress has been made over the years, with applications ranging from action recognition to semantic segmentation. Besides, the recent advent of large-scale pre-trained multimodal foundation models, such as CLIP, has inspired works leveraging these models to enhance adaptation and generalization performances or adapting them to downstream tasks. This survey provides the first comprehensive review of recent advances from traditional approaches to foundation models, covering: (1) Multimodal domain adaptation; (2) Multimodal test-time adaptation; (3) Multimodal domain generalization; (4) Domain adaptation and generalization with the help of multimodal foundation models; and (5) Adaptation of multimodal foundation models. For each topic, we formally define the problem and thoroughly review existing methods. Additionally, we analyze relevant datasets and applications, highlighting open challenges and potential future research directions. We maintain an active repository that contains up-to-date literature at this https URL.

Title: Diffusion Autoencoders are Scalable Image Tokenizers

• Authors: Yinbo Chen, Rohit Girdhar, Xiaolong Wang, Sai Saketh Rambhatla, Ishan Misra
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.18593
• Pdf URL: https://arxiv.org/pdf/2501.18593
• Copy Paste: [[2501.18593]] Diffusion Autoencoders are Scalable Image Tokenizers(https://arxiv.org/abs/2501.18593)
• Keywords: diffusion, self-supervised, generative
• Abstract: Tokenizing images into compact visual representations is a key step in learning efficient and high-quality image generative models. We present a simple diffusion tokenizer (DiTo) that learns compact visual representations for image generation models. Our key insight is that a single learning objective, diffusion L2 loss, can be used for training scalable image tokenizers. Since diffusion is already widely used for image generation, our insight greatly simplifies training such tokenizers. In contrast, current state-of-the-art tokenizers rely on an empirically found combination of heuristics and losses, thus requiring a complex training recipe that relies on non-trivially balancing different losses and pretrained supervised models. We show design decisions, along with theoretical grounding, that enable us to scale DiTo for learning competitive image representations. Our results show that DiTo is a simpler, scalable, and self-supervised alternative to the current state-of-the-art image tokenizer which is supervised. DiTo achieves competitive or better quality than state-of-the-art in image reconstruction and downstream image generation tasks.

Title: Foundational Models for 3D Point Clouds: A Survey and Outlook

• Authors: Vishal Thengane, Xiatian Zhu, Salim Bouzerdoum, Son Lam Phung, Yunpeng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.18594
• Pdf URL: https://arxiv.org/pdf/2501.18594
• Copy Paste: [[2501.18594]] Foundational Models for 3D Point Clouds: A Survey and Outlook(https://arxiv.org/abs/2501.18594)
• Keywords: foundation model
• Abstract: The 3D point cloud representation plays a crucial role in preserving the geometric fidelity of the physical world, enabling more accurate complex 3D environments. While humans naturally comprehend the intricate relationships between objects and variations through a multisensory system, artificial intelligence (AI) systems have yet to fully replicate this capacity. To bridge this gap, it becomes essential to incorporate multiple modalities. Models that can seamlessly integrate and reason across these modalities are known as foundation models (FMs). The development of FMs for 2D modalities, such as images and text, has seen significant progress, driven by the abundant availability of large-scale datasets. However, the 3D domain has lagged due to the scarcity of labelled data and high computational overheads. In response, recent research has begun to explore the potential of applying FMs to 3D tasks, overcoming these challenges by leveraging existing 2D knowledge. Additionally, language, with its capacity for abstract reasoning and description of the environment, offers a promising avenue for enhancing 3D understanding through large pre-trained language models (LLMs). Despite the rapid development and adoption of FMs for 3D vision tasks in recent years, there remains a gap in comprehensive and in-depth literature reviews. This article aims to address this gap by presenting a comprehensive overview of the state-of-the-art methods that utilize FMs for 3D visual understanding. We start by reviewing various strategies employed in the building of various 3D FMs. Then we categorize and summarize use of different FMs for tasks such as perception tasks. Finally, the article offers insights into future directions for research and development in this field. To help reader, we have curated list of relevant papers on the topic: this https URL.