2025-06-16

Title: Not All Clients Are Equal: Personalized Federated Learning on Heterogeneous Multi-Modal Clients

• Authors: Minhyuk Seo, Taeheon Kim, Hankook Lee, Jonghyun Choi, Tinne Tuytelaars
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2506.11024
• Pdf URL: https://arxiv.org/pdf/2506.11024
• Copy Paste: [[2506.11024]] Not All Clients Are Equal: Personalized Federated Learning on Heterogeneous Multi-Modal Clients(https://arxiv.org/abs/2506.11024)
• Keywords: foundation model
• Abstract: Foundation models have shown remarkable capabilities across diverse multi-modal tasks, but their centralized training raises privacy concerns and induces high transmission costs. In contrast, federated learning (FL) offers a distributed alternative without the need to share data. Recently, for the growing demand for personalizing AI models for different user purposes, personalized federated learning (PFL) has emerged. PFL allows each client to leverage the knowledge of other clients for further adaptation to individual user preferences, again without the need to share data. Despite its potential, most PFL studies remain confined to simulated environments, overlooking the data and model heterogeneity that arise in real-world scenarios. In contrast, we first consider large data heterogeneity, evaluating on a new benchmark for multi-modal PFL, spanning 40 distinct tasks with realistic data distribution shifts. We then consider model heterogeneity in that we do not assume that all clients share similar model architectures. To address data heterogeneity, we propose a task-similarity-aware model aggregation method that provides customized global models to each client. For model heterogeneity, we propose a dimension-invariant module that enables knowledge sharing across heterogeneous models. Empirical validations demonstrate that the proposed approach outperforms the state-of-the-art, excelling in both personalization and generalization capabilities.

Title: Evaluating Privacy-Utility Tradeoffs in Synthetic Smart Grid Data

• Authors: Andre Catarino, Rui Melo, Rui Abreu, Luis Cruz
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2506.11026
• Pdf URL: https://arxiv.org/pdf/2506.11026
• Copy Paste: [[2506.11026]] Evaluating Privacy-Utility Tradeoffs in Synthetic Smart Grid Data(https://arxiv.org/abs/2506.11026)
• Keywords: diffusion, generative
• Abstract: The widespread adoption of dynamic Time-of-Use (dToU) electricity tariffs requires accurately identifying households that would benefit from such pricing structures. However, the use of real consumption data poses serious privacy concerns, motivating the adoption of synthetic alternatives. In this study, we conduct a comparative evaluation of four synthetic data generation methods, Wasserstein-GP Generative Adversarial Networks (WGAN), Conditional Tabular GAN (CTGAN), Diffusion Models, and Gaussian noise augmentation, under different synthetic regimes. We assess classification utility, distribution fidelity, and privacy leakage. Our results show that architectural design plays a key role: diffusion models achieve the highest utility (macro-F1 up to 88.2%), while CTGAN provide the strongest resistance to reconstruction attacks. These findings highlight the potential of structured generative models for developing privacy-preserving, data-driven energy systems.

Title: Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model

• Authors: Xue Wang, Tian Zhou, Jinyang Gao, Bolin Ding, Jingren Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11029
• Pdf URL: https://arxiv.org/pdf/2506.11029
• Copy Paste: [[2506.11029]] Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model(https://arxiv.org/abs/2506.11029)
• Keywords: foundation model
• Abstract: We present a joint forecasting framework for time series prediction that contrasts with traditional direct or recursive methods. This framework achieves state-of-the-art performance for our designed foundation model, YingLong, and reveals a novel scaling effect: longer outputs significantly enhance model accuracy due to delayed chain-of-thought reasoning in our non-causal approach. YingLong is a non-causal, bidirectional attention encoder-only transformer trained through masked token recovery, aligning more effectively with language understanding tasks than with generation tasks. Additionally, we boost performance by tackling output variance with a multi-input ensemble. We release four foundation models ranging from 6M to 300M parameters, demonstrating superior results in zero-shot tasks on the ETT and Weather datasets. YingLong achieves more than 60% best performance. To ensure generalizability, we assessed the models using the GIFT-Eval benchmark, which comprises 23 time series datasets across 7 domains. Yinglong significantly outperformed the best time-series foundation models, end-to-end trained models by 14% and 44% in rank this http URL pretrained 300M model is available at this https URL

Title: CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models

• Authors: Aneesh Komanduri, Karuna Bhaila, Xintao Wu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2506.11034
• Pdf URL: https://arxiv.org/pdf/2506.11034
• Copy Paste: [[2506.11034]] CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models(https://arxiv.org/abs/2506.11034)
• Keywords: in-context
• Abstract: Large language models (LLMs) have shown remarkable ability in various language tasks, especially with their emergent in-context learning capability. Extending LLMs to incorporate visual inputs, large vision-language models (LVLMs) have shown impressive performance in tasks such as recognition and visual question answering (VQA). Despite increasing interest in the utility of LLMs in causal reasoning tasks such as causal discovery and counterfactual reasoning, there has been relatively little work showcasing the abilities of LVLMs on visual causal reasoning tasks. We take this opportunity to formally introduce a comprehensive causal reasoning benchmark for multi-modal in-context learning from LVLMs. Our CausalVLBench encompasses three representative tasks: causal structure inference, intervention target prediction, and counterfactual prediction. We evaluate the ability of state-of-the-art open-source LVLMs on our causal reasoning tasks across three causal representation learning datasets and demonstrate their fundamental strengths and weaknesses. We hope that our benchmark elucidates the drawbacks of existing vision-language models and motivates new directions and paradigms in improving the visual causal reasoning abilities of LVLMs.

Title: Angle Domain Guidance: Latent Diffusion Requires Rotation Rather Than Extrapolation

• Authors: Cheng Jin, Zhenyu Xiao, Chutao Liu, Yuantao Gu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11039
• Pdf URL: https://arxiv.org/pdf/2506.11039
• Copy Paste: [[2506.11039]] Angle Domain Guidance: Latent Diffusion Requires Rotation Rather Than Extrapolation(https://arxiv.org/abs/2506.11039)
• Keywords: diffusion
• Abstract: Classifier-free guidance (CFG) has emerged as a pivotal advancement in text-to-image latent diffusion models, establishing itself as a cornerstone technique for achieving high-quality image synthesis. However, under high guidance weights, where text-image alignment is significantly enhanced, CFG also leads to pronounced color distortions in the generated images. We identify that these distortions stem from the amplification of sample norms in the latent space. We present a theoretical framework that elucidates the mechanisms of norm amplification and anomalous diffusion phenomena induced by classifier-free guidance. Leveraging our theoretical insights and the latent space structure, we propose an Angle Domain Guidance (ADG) algorithm. ADG constrains magnitude variations while optimizing angular alignment, thereby mitigating color distortions while preserving the enhanced text-image alignment achieved at higher guidance weights. Experimental results demonstrate that ADG significantly outperforms existing methods, generating images that not only maintain superior text alignment but also exhibit improved color fidelity and better alignment with human perceptual preferences.

Title: Large Language models for Time Series Analysis: Techniques, Applications, and Challenges

• Authors: Feifei Shi, Xueyan Yin, Kang Wang, Wanyu Tu, Qifu Sun, Huansheng Ning
• Subjects: cs.LG, cs.CL, cs.ET
• Abstract URL: https://arxiv.org/abs/2506.11040
• Pdf URL: https://arxiv.org/pdf/2506.11040
• Copy Paste: [[2506.11040]] Large Language models for Time Series Analysis: Techniques, Applications, and Challenges(https://arxiv.org/abs/2506.11040)
• Keywords: foundation model
• Abstract: Time series analysis is pivotal in domains like financial forecasting and biomedical monitoring, yet traditional methods are constrained by limited nonlinear feature representation and long-term dependency capture. The emergence of Large Language Models (LLMs) offers transformative potential by leveraging their cross-modal knowledge integration and inherent attention mechanisms for time series analysis. However, the development of general-purpose LLMs for time series from scratch is still hindered by data diversity, annotation scarcity, and computational requirements. This paper presents a systematic review of pre-trained LLM-driven time series analysis, focusing on enabling techniques, potential applications, and open challenges. First, it establishes an evolutionary roadmap of AI-driven time series analysis, from the early machine learning era, through the emerging LLM-driven paradigm, to the development of native temporal foundation models. Second, it organizes and systematizes the technical landscape of LLM-driven time series analysis from a workflow perspective, covering LLMs' input, optimization, and lightweight stages. Finally, it critically examines novel real-world applications and highlights key open challenges that can guide future research and innovation. The work not only provides valuable insights into current advances but also outlines promising directions for future development. It serves as a foundational reference for both academic and industrial researchers, paving the way for the development of more efficient, generalizable, and interpretable systems of LLM-driven time series analysis.

Title: GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models

• Authors: Baoquan Zhang, Guangning Xu, Michael. K. Ng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11042
• Pdf URL: https://arxiv.org/pdf/2506.11042
• Copy Paste: [[2506.11042]] GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models(https://arxiv.org/abs/2506.11042)
• Keywords: foundation model, generative
• Abstract: Pretrained Foundation Models (PFMs) have transformed numerous applications by enabling efficient adaptation to customized tasks. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a resource-efficient alternative to full fine-tuning, especially leveraging reparameterized weights $\Delta W$ to adapt models for downstream tasks. However, a critical yet underexplored question remains: can we utilize well-pretrained weights $W_0$ to guide the update of task-specific $\Delta W$, avoiding inefficient training it from scratch? To end this, we propose Generative Parameter-Efficient Fine-Tuning (GenFT), a novel method that extracts structured, transferable information from $W_0$ for efficient $\Delta W$ training. To extract row and column structure information, GenFT applies row and column transformations to distill essential patterns from $W_0$. A tailored policy further decomposes $\Delta W$ into layer-shared and layer-specific components, balancing information reuse and individualized flexibility. GenFT is simple yet effective, achieving superior performance across CV and NLP tasks. Extensive experiments on VTAB-1K, FGVC, and GLUE benchmarks demonstrate that GenFT outperforms state-of-the-art PEFT methods, offering a new perspective for efficient model adaptation.

Title: PolyMicros: Bootstrapping a Foundation Model for Polycrystalline Material Structure

• Authors: Michael Buzzy, Andreas Robertson, Peng Chen, Surya Kalidindi
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2506.11055
• Pdf URL: https://arxiv.org/pdf/2506.11055
• Copy Paste: [[2506.11055]] PolyMicros: Bootstrapping a Foundation Model for Polycrystalline Material Structure(https://arxiv.org/abs/2506.11055)
• Keywords: foundation model, generative
• Abstract: Recent advances in Foundation Models for Materials Science are poised to revolutionize the discovery, manufacture, and design of novel materials with tailored properties and responses. Although great strides have been made, successes have been restricted to materials classes where multi-million sample data repositories can be readily curated (e.g., atomistic structures). Unfortunately, for many structural and functional materials (e.g., mesoscale structured metal alloys), such datasets are too costly or prohibitive to construct; instead, datasets are limited to very few examples. To address this challenge, we introduce a novel machine learning approach for learning from hyper-sparse, complex spatial data in scientific domains. Our core contribution is a physics-driven data augmentation scheme that leverages an ensemble of local generative models, trained on as few as five experimental observations, and coordinates them through a novel diversity curation strategy to generate a large-scale, physically diverse dataset. We utilize this framework to construct PolyMicros, the first Foundation Model for polycrystalline materials (a structural material class important across a broad range of industrial and scientific applications). We demonstrate the utility of PolyMicros by zero-shot solving several long standing challenges related to accelerating 3D experimental microscopy. Finally, we make both our models and datasets openly available to the community.

Title: Smotrom tvoja pa ander drogoj verden! Resurrecting Dead Pidgin with Generative Models: Russenorsk Case Study

• Authors: Alexey Tikhonov, Sergei Shteiner, Anna Bykova, Ivan P. Yamshchikov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.11065
• Pdf URL: https://arxiv.org/pdf/2506.11065
• Copy Paste: [[2506.11065]] Smotrom tvoja pa ander drogoj verden! Resurrecting Dead Pidgin with Generative Models: Russenorsk Case Study(https://arxiv.org/abs/2506.11065)
• Keywords: generative
• Abstract: Russenorsk, a pidgin language historically used in trade interactions between Russian and Norwegian speakers, represents a unique linguistic phenomenon. In this paper, we attempt to analyze its lexicon using modern large language models (LLMs), based on surviving literary sources. We construct a structured dictionary of the language, grouped by synonyms and word origins. Subsequently, we use this dictionary to formulate hypotheses about the core principles of word formation and grammatical structure in Russenorsk and show which hypotheses generated by large language models correspond to the hypotheses previously proposed ones in the academic literature. We also develop a "reconstruction" translation agent that generates hypothetical Russenorsk renderings of contemporary Russian and Norwegian texts.

Title: PRISM: A Transformer-based Language Model of Structured Clinical Event Data

• Authors: Lionel Levine, John Santerre, Alex S. Young, T. Barry Levine, Francis Campion, Majid Sarrafzadeh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11082
• Pdf URL: https://arxiv.org/pdf/2506.11082
• Copy Paste: [[2506.11082]] PRISM: A Transformer-based Language Model of Structured Clinical Event Data(https://arxiv.org/abs/2506.11082)
• Keywords: generative
• Abstract: We introduce PRISM (Predictive Reasoning in Sequential Medicine), a transformer-based architecture designed to model the sequential progression of clinical decision-making processes. Unlike traditional approaches that rely on isolated diagnostic classification, PRISM frames clinical trajectories as tokenized sequences of events - including diagnostic tests, laboratory results, and diagnoses - and learns to predict the most probable next steps in the patient diagnostic journey. Leveraging a large custom clinical vocabulary and an autoregressive training objective, PRISM demonstrates the ability to capture complex dependencies across longitudinal patient timelines. Experimental results show substantial improvements over random baselines in next-token prediction tasks, with generated sequences reflecting realistic diagnostic pathways, laboratory result progressions, and clinician ordering behaviors. These findings highlight the feasibility of applying generative language modeling techniques to structured medical event data, enabling applications in clinical decision support, simulation, and education. PRISM establishes a foundation for future advancements in sequence-based healthcare modeling, bridging the gap between machine learning architectures and real-world diagnostic reasoning.

Title: You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language Model

• Authors: Wenchong He, Liqian Peng, Zhe Jiang, Alex Go
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11103
• Pdf URL: https://arxiv.org/pdf/2506.11103
• Copy Paste: [[2506.11103]] You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language Model(https://arxiv.org/abs/2506.11103)
• Keywords: in-context
• Abstract: Large language models (LLMs) possess a remarkable ability to perform in-context learning (ICL), which enables them to handle multiple downstream tasks simultaneously without requiring task-specific fine-tuning. Recent studies have shown that even moderately sized LLMs, such as Mistral 7B, Gemma 7B and Llama-3 8B, can achieve ICL through few-shot in-context fine-tuning of all tasks at once. However, this approach still lags behind dedicated fine-tuning, where a separate model is trained for each individual task. In this paper, we propose a novel approach, Many-Shot In-Context Fine-tuning (ManyICL), which significantly narrows this performance gap by extending the principles of ICL to a many-shot setting. To unlock the full potential of ManyICL and address the inherent inefficiency of processing long sequences with numerous in-context examples, we propose a novel training objective. Instead of solely predicting the final answer, our approach treats every answer within the context as a supervised training target. This effectively shifts the role of many-shot examples from prompts to targets for autoregressive learning. Through extensive experiments on diverse downstream tasks, including classification, summarization, question answering, natural language inference, and math, we demonstrate that ManyICL substantially outperforms zero/few-shot fine-tuning and approaches the performance of dedicated fine-tuning. Furthermore, ManyICL significantly mitigates catastrophic forgetting issues observed in zero/few-shot fine-tuning. The code will be made publicly available upon publication.

Title: History-Aware Cross-Attention Reinforcement: Self-Supervised Multi Turn and Chain-of-Thought Fine-Tuning with vLLM

• Authors: Andrew Kiruluta, Andreas Lemos, Priscilla Burity
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.11108
• Pdf URL: https://arxiv.org/pdf/2506.11108
• Copy Paste: [[2506.11108]] History-Aware Cross-Attention Reinforcement: Self-Supervised Multi Turn and Chain-of-Thought Fine-Tuning with vLLM(https://arxiv.org/abs/2506.11108)
• Keywords: self-supervised
• Abstract: We present CAGSR-vLLM-MTC, an extension of our Self-Supervised Cross-Attention-Guided Reinforcement (CAGSR) framework, now implemented on the high-performance vLLM runtime, to address both multi-turn dialogue and chain-of-thought reasoning. Building upon our original single-turn approach, we first instrumented vLLM's C++/CUDA kernels to asynchronously capture per-layer, per-head cross-attention weights during generation. We then generalized our self-supervised reward function to accumulate attention signals over entire conversation histories and intermediate chain-of-thought steps. We discuss practical trade-offs, including an entropy-based clamping mechanism to prevent attention collapse on early context, and outline future directions for multi-party dialogues and hierarchical reasoning.

Title: Benchmarking Foundation Speech and Language Models for Alzheimer's Disease and Related Dementia Detection from Spontaneous Speech

• Authors: Jingyu Li, Lingchao Mao, Hairong Wang, Zhendong Wang, Xi Mao, Xuelei Sherry Ni
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2506.11119
• Pdf URL: https://arxiv.org/pdf/2506.11119
• Copy Paste: [[2506.11119]] Benchmarking Foundation Speech and Language Models for Alzheimer's Disease and Related Dementia Detection from Spontaneous Speech(https://arxiv.org/abs/2506.11119)
• Keywords: foundation model, generative
• Abstract: Background: Alzheimer's disease and related dementias (ADRD) are progressive neurodegenerative conditions where early detection is vital for timely intervention and care. Spontaneous speech contains rich acoustic and linguistic markers that may serve as non-invasive biomarkers for cognitive decline. Foundation models, pre-trained on large-scale audio or text data, produce high-dimensional embeddings encoding contextual and acoustic features. Methods: We used the PREPARE Challenge dataset, which includes audio recordings from over 1,600 participants with three cognitive statuses: healthy control (HC), mild cognitive impairment (MCI), and Alzheimer's Disease (AD). We excluded non-English, non-spontaneous, or poor-quality recordings. The final dataset included 703 (59.13%) HC, 81 (6.81%) MCI, and 405 (34.06%) AD cases. We benchmarked a range of open-source foundation speech and language models to classify cognitive status into the three categories. Results: The Whisper-medium model achieved the highest performance among speech models (accuracy = 0.731, AUC = 0.802). Among language models, BERT with pause annotation performed best (accuracy = 0.662, AUC = 0.744). ADRD detection using state-of-the-art automatic speech recognition (ASR) model-generated audio embeddings outperformed others. Including non-semantic features like pause patterns consistently improved text-based classification. Conclusion: This study introduces a benchmarking framework using foundation models and a clinically relevant dataset. Acoustic-based approaches -- particularly ASR-derived embeddings -- demonstrate strong potential for scalable, non-invasive, and cost-effective early detection of ADRD.

Title: SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models

• Authors: Hourun Zhu, Chengchao Shen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11120
• Pdf URL: https://arxiv.org/pdf/2506.11120
• Copy Paste: [[2506.11120]] SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models(https://arxiv.org/abs/2506.11120)
• Keywords: generative
• Abstract: In spite of strong performance achieved by LLMs, the costs of their deployment are unaffordable. For the compression of LLMs, gradient-based pruning methods present promising effectiveness. However, in these methods, the gradient computation with one-hot labels ignore the potential predictions on other words, thus missing key information for generative capability of the original model. To address this issue, we introduce a self-distillation loss during the pruning phase (rather than post-training) to fully exploit the predictions of the original model, thereby obtaining more accurate gradient information for pruning. Moreover, we find that, compared to attention modules, the predictions of LLM are less sensitive to multilayer perceptron (MLP) modules, which take up more than $5 \times$ parameters (LLaMA3.2-1.2B). To this end, we focus on the pruning of MLP modules, to significantly compress LLM without obvious performance degradation. Experimental results on extensive zero-shot benchmarks demonstrate that our method significantly outperforms existing pruning methods. Furthermore, our method achieves very competitive performance among 1B-scale open source LLMs. The source code and trained weights are available at this https URL.

Title: SUTA-LM: Bridging Test-Time Adaptation and Language Model Rescoring for Robust ASR

• Authors: Wei-Ping Huang, Guan-Ting Lin, Hung-yi Lee
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2506.11121
• Pdf URL: https://arxiv.org/pdf/2506.11121
• Copy Paste: [[2506.11121]] SUTA-LM: Bridging Test-Time Adaptation and Language Model Rescoring for Robust ASR(https://arxiv.org/abs/2506.11121)
• Keywords: generative
• Abstract: Despite progress in end-to-end ASR, real-world domain mismatches still cause performance drops, which Test-Time Adaptation (TTA) aims to mitigate by adjusting models during inference. Recent work explores combining TTA with external language models, using techniques like beam search rescoring or generative error correction. In this work, we identify a previously overlooked challenge: TTA can interfere with language model rescoring, revealing the nontrivial nature of effectively combining the two methods. Based on this insight, we propose SUTA-LM, a simple yet effective extension of SUTA, an entropy-minimization-based TTA approach, with language model rescoring. SUTA-LM first applies a controlled adaptation process guided by an auto-step selection mechanism leveraging both acoustic and linguistic information, followed by language model rescoring to refine the outputs. Experiments on 18 diverse ASR datasets show that SUTA-LM achieves robust results across a wide range of domains.

Title: Adaptive Object Detection with ESRGAN-Enhanced Resolution & Faster R-CNN

• Authors: Divya Swetha K, Ziaul Haque Choudhury, Hemanta Kumar Bhuyan, Biswajit Brahma, Nilayam Kumar Kamila
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11122
• Pdf URL: https://arxiv.org/pdf/2506.11122
• Copy Paste: [[2506.11122]] Adaptive Object Detection with ESRGAN-Enhanced Resolution & Faster R-CNN(https://arxiv.org/abs/2506.11122)
• Keywords: generative
• Abstract: In this study, proposes a method for improved object detection from the low-resolution images by integrating Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN) and Faster Region-Convolutional Neural Network (Faster R-CNN). ESRGAN enhances low-quality images, restoring details and improving clarity, while Faster R-CNN performs accurate object detection on the enhanced images. The combination of these techniques ensures better detection performance, even with poor-quality inputs, offering an effective solution for applications where image resolution is in consistent. ESRGAN is employed as a pre-processing step to enhance the low-resolution input image, effectively restoring lost details and improving overall image quality. Subsequently, the enhanced image is fed into the Faster R-CNN model for accurate object detection and localization. Experimental results demonstrate that this integrated approach yields superior performance compared to traditional methods applied directly to low-resolution images. The proposed framework provides a promising solution for applications where image quality is variable or limited, enabling more robust and reliable object detection in challenging scenarios. It achieves a balance between improved image quality and efficient object detection

Title: Image-Based Method For Measuring And Classification Of Iron Ore Pellets Using Star-Convex Polygons

• Authors: Artem Solomko, Oleg Kartashev, Andrey Golov, Mikhail Deulin, Vadim Valynkin, Vasily Kharin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11126
• Pdf URL: https://arxiv.org/pdf/2506.11126
• Copy Paste: [[2506.11126]] Image-Based Method For Measuring And Classification Of Iron Ore Pellets Using Star-Convex Polygons(https://arxiv.org/abs/2506.11126)
• Keywords: anomaly
• Abstract: We would like to present a comprehensive study on the classification of iron ore pellets, aimed at identifying quality violations in the final product, alongside the development of an innovative imagebased measurement method utilizing the StarDist algorithm, which is primarily employed in the medical field. This initiative is motivated by the necessity to accurately identify and analyze objects within densely packed and unstable environments. The process involves segmenting these objects, determining their contours, classifying them, and measuring their physical dimensions. This is crucial because the size distribution and classification of pellets such as distinguishing between nice (quality) and joint (caused by the presence of moisture or indicating a process of production failure) types are among the most significant characteristics that define the quality of the final product. Traditional algorithms, including image classification techniques using Vision Transformer (ViT), instance segmentation methods like Mask R-CNN, and various anomaly segmentation algorithms, have not yielded satisfactory results in this context. Consequently, we explored methodologies from related fields to enhance our approach. The outcome of our research is a novel method designed to detect objects with smoothed boundaries. This advancement significantly improves the accuracy of physical dimension measurements and facilitates a more precise analysis of size distribution among the iron ore pellets. By leveraging the strengths of the StarDist algorithm, we aim to provide a robust solution that addresses the challenges posed by the complex nature of pellet classification and measurement.

Title: GUIRoboTron-Speech: Towards Automated GUI Agents Based on Speech Instructions

• Authors: Wenkang Han, Zhixiong Zeng, Jing Huang, Shu Jiang, Liming Zheng, Longrong Yang, Haibo Qiu, Chang Yao, Jingyuan Chen, Lin Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11127
• Pdf URL: https://arxiv.org/pdf/2506.11127
• Copy Paste: [[2506.11127]] GUIRoboTron-Speech: Towards Automated GUI Agents Based on Speech Instructions(https://arxiv.org/abs/2506.11127)
• Keywords: foundation model
• Abstract: Autonomous agents for Graphical User Interfaces (GUIs) are revolutionizing human-computer interaction, yet their reliance on text-based instructions imposes limitations on accessibility and convenience, particularly in hands-free scenarios. To address this gap, we propose GUIRoboTron-Speech, the first end-to-end autonomous GUI agent that directly accepts speech instructions and on-device screenshots to predict actions. Confronted with the scarcity of speech-based GUI agent datasets, we initially generated high-quality speech instructions for training by leveraging a random timbre text-to-speech (TTS) model to convert existing text instructions. We then develop GUIRoboTron-Speech's capabilities through progressive grounding and planning training stages. A key contribution is a heuristic mixed-instruction training strategy designed to mitigate the modality imbalance inherent in pre-trained foundation models. Comprehensive experiments on several benchmark datasets validate the robust and superior performance of GUIRoboTron-Speech, demonstrating the significant potential and widespread applicability of speech as an effective instruction modality for driving GUI agents. Our code and datasets are available at this https URL.

Title: AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation

• Authors: Chao Liang, Jianwen Jiang, Wang Liao, Jiaqi Yang, Zerong zheng, Weihong Zeng, Han Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11144
• Pdf URL: https://arxiv.org/pdf/2506.11144
• Copy Paste: [[2506.11144]] AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation(https://arxiv.org/abs/2506.11144)
• Keywords: diffusion
• Abstract: Recent advancements in human video generation and animation tasks, driven by diffusion models, have achieved significant progress. However, expressive and realistic human animation remains challenging due to the trade-off between motion naturalness and visual fidelity. To address this, we propose \textbf{AlignHuman}, a framework that combines Preference Optimization as a post-training technique with a divide-and-conquer training strategy to jointly optimize these competing objectives. Our key insight stems from an analysis of the denoising process across timesteps: (1) early denoising timesteps primarily control motion dynamics, while (2) fidelity and human structure can be effectively managed by later timesteps, even if early steps are skipped. Building on this observation, we propose timestep-segment preference optimization (TPO) and introduce two specialized LoRAs as expert alignment modules, each targeting a specific dimension in its corresponding timestep interval. The LoRAs are trained using their respective preference data and activated in the corresponding intervals during inference to enhance motion naturalness and fidelity. Extensive experiments demonstrate that AlignHuman improves strong baselines and reduces NFEs during inference, achieving a 3.3$\times$ speedup (from 100 NFEs to 30 NFEs) with minimal impact on generation quality. Homepage: \href{this https URL}{this https URL}

Title: 3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks

• Authors: Xiaotang Gai, Jiaxiang Liu, Yichen Li, Zijie Meng, Jian Wu, Zuozhu Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11147
• Pdf URL: https://arxiv.org/pdf/2506.11147
• Copy Paste: [[2506.11147]] 3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks(https://arxiv.org/abs/2506.11147)
• Keywords: anomaly
• Abstract: Medical Visual Question Answering (Med-VQA) holds significant potential for clinical decision support, yet existing efforts primarily focus on 2D imaging with limited task diversity. This paper presents 3D-RAD, a large-scale dataset designed to advance 3D Med-VQA using radiology CT scans. The 3D-RAD dataset encompasses six diverse VQA tasks: anomaly detection, image observation, medical computation, existence detection, static temporal diagnosis, and longitudinal temporal diagnosis. It supports both open- and closed-ended questions while introducing complex reasoning challenges, including computational tasks and multi-stage temporal analysis, to enable comprehensive benchmarking. Extensive evaluations demonstrate that existing vision-language models (VLMs), especially medical VLMs exhibit limited generalization, particularly in multi-temporal tasks, underscoring the challenges of real-world 3D diagnostic reasoning. To drive future advancements, we release a high-quality training set 3D-RAD-T of 136,195 expert-aligned samples, showing that fine-tuning on this dataset could significantly enhance model performance. Our dataset and code, aiming to catalyze multimodal medical AI research and establish a robust foundation for 3D medical visual understanding, are publicly available at this https URL.

Title: LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs

• Authors: Melvin Wong, Yueming Lyu, Thiago Rios, Stefan Menzel, Yew-Soon Ong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11148
• Pdf URL: https://arxiv.org/pdf/2506.11148
• Copy Paste: [[2506.11148]] LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs(https://arxiv.org/abs/2506.11148)
• Keywords: generative
• Abstract: The emergence of generative artificial intelligence (GenAI) and large language models (LLMs) has revolutionized the landscape of digital content creation in different modalities. However, its potential use in Physical AI for engineering design, where the production of physically viable artifacts is paramount, remains vastly underexplored. The absence of physical knowledge in existing LLM-to-3D models often results in outputs detached from real-world physical constraints. To address this gap, we introduce LLM-to-Phy3D, a physically conform online 3D object generation that enables existing LLM-to-3D models to produce physically conforming 3D objects on the fly. LLM-to-Phy3D introduces a novel online black-box refinement loop that empowers large language models (LLMs) through synergistic visual and physics-based evaluations. By delivering directional feedback in an iterative refinement process, LLM-to-Phy3D actively drives the discovery of prompts that yield 3D artifacts with enhanced physical performance and greater geometric novelty relative to reference objects, marking a substantial contribution to AI-driven generative design. Systematic evaluations of LLM-to-Phy3D, supported by ablation studies in vehicle design optimization, reveal various LLM improvements gained by 4.5% to 106.7% in producing physically conform target domain 3D designs over conventional LLM-to-3D models. The encouraging results suggest the potential general use of LLM-to-Phy3D in Physical AI for scientific and engineering applications.

Title: Synthetic Geology -- Structural Geology Meets Deep Learning

• Authors: Simon Ghyselincks, Valeriia Okhmak, Stefano Zampini, George Turkiyyah, David Keyes, Eldad Haber
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11164
• Pdf URL: https://arxiv.org/pdf/2506.11164
• Copy Paste: [[2506.11164]] Synthetic Geology -- Structural Geology Meets Deep Learning(https://arxiv.org/abs/2506.11164)
• Keywords: foundation model, generative
• Abstract: Visualizing the first few kilometers of the Earth's subsurface, a long-standing challenge gating a virtually inexhaustible list of important applications, is coming within reach through deep learning. Building on techniques of generative artificial intelligence applied to voxelated images, we demonstrate a method that extends surface geological data supplemented by boreholes to a three-dimensional subsurface region by training a neural network. The Earth's land area having been extensively mapped for geological features, the bottleneck of this or any related technique is the availability of data below the surface. We close this data gap in the development of subsurface deep learning by designing a synthetic data-generator process that mimics eons of geological activity such as sediment compaction, volcanic intrusion, and tectonic dynamics to produce a virtually limitless number of samples of the near lithosphere. A foundation model trained on such synthetic data is able to generate a 3D image of the subsurface from a previously unseen map of surface topography and geology, showing increasing fidelity with increasing access to borehole data, depicting such structures as layers, faults, folds, dikes, and sills. We illustrate the early promise of the combination of a synthetic lithospheric generator with a trained neural network model using generative flow matching. Ultimately, such models will be fine-tuned on data from applicable campaigns, such as mineral prospecting in a given region. Though useful in itself, a regionally fine-tuned models may be employed not as an end but as a means: as an AI-based regularizer in a more traditional inverse problem application, in which the objective function represents the mismatch of additional data with physical models with applications in resource exploration, hazard assessment, and geotechnical engineering.

Title: Towards a general-purpose foundation model for fMRI analysis

• Authors: Cheng Wang, Yu Jiang, Zhihao Peng, Chenxin Li, Changbae Bang, Lin Zhao, Jinglei Lv, Jorge Sepulcre, Carl Yang, Lifang He, Tianming Liu, Daniel Barron, Quanzheng Li, Randy Hirschtick, Byung-Hoon Kim, Xiang Li, Yixuan Yuan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11167
• Pdf URL: https://arxiv.org/pdf/2506.11167
• Copy Paste: [[2506.11167]] Towards a general-purpose foundation model for fMRI analysis(https://arxiv.org/abs/2506.11167)
• Keywords: foundation model
• Abstract: Functional Magnetic Resonance Imaging (fMRI) is essential for studying brain function and diagnosing neurological disorders, but current analysis methods face reproducibility and transferability issues due to complex pre-processing and task-specific models. We introduce NeuroSTORM (Neuroimaging Foundation Model with Spatial-Temporal Optimized Representation Modeling), a generalizable framework that directly learns from 4D fMRI volumes and enables efficient knowledge transfer across diverse applications. NeuroSTORM is pre-trained on 28.65 million fMRI frames (>9,000 hours) from over 50,000 subjects across multiple centers and ages 5 to 100. Using a Mamba backbone and a shifted scanning strategy, it efficiently processes full 4D volumes. We also propose a spatial-temporal optimized pre-training approach and task-specific prompt tuning to improve transferability. NeuroSTORM outperforms existing methods across five tasks: age/gender prediction, phenotype prediction, disease diagnosis, fMRI-to-image retrieval, and task-based fMRI classification. It demonstrates strong clinical utility on datasets from hospitals in the U.S., South Korea, and Australia, achieving top performance in disease diagnosis and cognitive phenotype prediction. NeuroSTORM provides a standardized, open-source foundation model to improve reproducibility and transferability in fMRI-based clinical research.

Title: BrainMAP: Multimodal Graph Learning For Efficient Brain Disease Localization

• Authors: Nguyen Linh Dan Le, Jing Ren, Ciyuan Peng, Chengyao Xie, Bowen Li, Feng Xia
• Subjects: cs.CV, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2506.11178
• Pdf URL: https://arxiv.org/pdf/2506.11178
• Copy Paste: [[2506.11178]] BrainMAP: Multimodal Graph Learning For Efficient Brain Disease Localization(https://arxiv.org/abs/2506.11178)
• Keywords: diffusion, generative
• Abstract: Recent years have seen a surge in research focused on leveraging graph learning techniques to detect neurodegenerative diseases. However, existing graph-based approaches typically lack the ability to localize and extract the specific brain regions driving neurodegenerative pathology within the full connectome. Additionally, recent works on multimodal brain graph models often suffer from high computational complexity, limiting their practical use in resource-constrained devices. In this study, we present BrainMAP, a novel multimodal graph learning framework designed for precise and computationally efficient identification of brain regions affected by neurodegenerative diseases. First, BrainMAP utilizes an atlas-driven filtering approach guided by the AAL atlas to pinpoint and extract critical brain subgraphs. Unlike recent state-of-the-art methods, which model the entire brain network, BrainMAP achieves more than 50% reduction in computational overhead by concentrating on disease-relevant subgraphs. Second, we employ an advanced multimodal fusion process comprising cross-node attention to align functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI) data, coupled with an adaptive gating mechanism to blend and integrate these modalities dynamically. Experimental results demonstrate that BrainMAP outperforms state-of-the-art methods in computational efficiency, without compromising predictive accuracy.

Title: Can Time-Series Foundation Models Perform Building Energy Management Tasks?

• Authors: Ozan Baris Mulayim, Pengrui Quan, Liying Han, Xiaomin Ouyang, Dezhi Hong, Mario Bergés, Mani Srivastava
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2506.11250
• Pdf URL: https://arxiv.org/pdf/2506.11250
• Copy Paste: [[2506.11250]] Can Time-Series Foundation Models Perform Building Energy Management Tasks?(https://arxiv.org/abs/2506.11250)
• Keywords: foundation model
• Abstract: Building energy management (BEM) tasks require processing and learning from a variety of time-series data. Existing solutions rely on bespoke task- and data-specific models to perform these tasks, limiting their broader applicability. Inspired by the transformative success of Large Language Models (LLMs), Time-Series Foundation Models (TSFMs), trained on diverse datasets, have the potential to change this. Were TSFMs to achieve a level of generalizability across tasks and contexts akin to LLMs, they could fundamentally address the scalability challenges pervasive in BEM. To understand where they stand today, we evaluate TSFMs across four dimensions: (1) generalizability in zero-shot univariate forecasting, (2) forecasting with covariates for thermal behavior modeling, (3) zero-shot representation learning for classification tasks, and (4) robustness to performance metrics and varying operational conditions. Our results reveal that TSFMs exhibit \emph{limited} generalizability, performing only marginally better than statistical models on unseen datasets and modalities for univariate forecasting. Similarly, inclusion of covariates in TSFMs does not yield performance improvements, and their performance remains inferior to conventional models that utilize covariates. While TSFMs generate effective zero-shot representations for downstream classification tasks, they may remain inferior to statistical models in forecasting when statistical models perform test-time fitting. Moreover, TSFMs forecasting performance is sensitive to evaluation metrics, and they struggle in more complex building environments compared to statistical models. These findings underscore the need for targeted advancements in TSFM design, particularly their handling of covariates and incorporating context and temporal dynamics into prediction mechanisms, to develop more adaptable and scalable solutions for BEM.

Title: Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models

• Authors: Yuwen Tan, Boqing Gong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11253
• Pdf URL: https://arxiv.org/pdf/2506.11253
• Copy Paste: [[2506.11253]] Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models(https://arxiv.org/abs/2506.11253)
• Keywords: foundation model
• Abstract: Machine unlearning removes certain training data points and their influence on AI models (e.g., when a data owner revokes their decision to allow models to learn from the data). In this position paper, we propose to lift data-tracing machine unlearning to knowledge-tracing for foundation models (FMs). We support this position based on practical needs and insights from cognitive studies. Practically, tracing data cannot meet the diverse unlearning requests for FMs, which may be from regulators, enterprise users, product teams, etc., having no access to FMs' massive training data. Instead, it is convenient for these parties to issue an unlearning request about the knowledge or capability FMs (should not) possess. Cognitively, knowledge-tracing unlearning aligns with how the human brain forgets more closely than tracing individual training data points. Finally, we provide a concrete case study about a vision-language FM to illustrate how an unlearner might instantiate the knowledge-tracing machine unlearning paradigm.

Title: Domain-Constrained Diffusion Models to Synthesize Tabular Data: A Case Study in Power Systems

• Authors: Milad Hoseinpour, Vladimir Dvorkin
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2506.11281
• Pdf URL: https://arxiv.org/pdf/2506.11281
• Copy Paste: [[2506.11281]] Domain-Constrained Diffusion Models to Synthesize Tabular Data: A Case Study in Power Systems(https://arxiv.org/abs/2506.11281)
• Keywords: diffusion, generative
• Abstract: Growing concerns over privacy, security, and legal barriers are driving the rising demand for synthetic data across domains such as healthcare, finance, and energy. While generative models offer a promising solution to overcome these barriers, their utility depends on the incorporation of domain-specific knowledge. We propose to synthesize data using a guided diffusion model that integrates domain constraints directly into the generative process. We develop the model in the context of power systems, with potential applicability to other domains that involve tabular data. Specifically, we synthesize statistically representative and high-fidelity power flow datasets. To satisfy domain constraints, e.g., Kirchhoff laws, we introduce a gradient-based guidance to steer the sampling trajectory in a feasible direction. Numerical results demonstrate the effectiveness of our approach.

Title: TARDIS STRIDE: A Spatio-Temporal Road Image Dataset for Exploration and Autonomy

• Authors: Héctor Carrión, Yutong Bai, Víctor A. Hernández Castro, Kishan Panaganti, Ayush Zenith, Matthew Trang, Tony Zhang, Pietro Perona, Jitendra Malik
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11302
• Pdf URL: https://arxiv.org/pdf/2506.11302
• Copy Paste: [[2506.11302]] TARDIS STRIDE: A Spatio-Temporal Road Image Dataset for Exploration and Autonomy(https://arxiv.org/abs/2506.11302)
• Keywords: generative
• Abstract: World models aim to simulate environments and enable effective agent behavior. However, modeling real-world environments presents unique challenges as they dynamically change across both space and, crucially, time. To capture these composed dynamics, we introduce a Spatio-Temporal Road Image Dataset for Exploration (STRIDE) permuting 360-degree panoramic imagery into rich interconnected observation, state and action nodes. Leveraging this structure, we can simultaneously model the relationship between egocentric views, positional coordinates, and movement commands across both space and time. We benchmark this dataset via TARDIS, a transformer-based generative world model that integrates spatial and temporal dynamics through a unified autoregressive framework trained on STRIDE. We demonstrate robust performance across a range of agentic tasks such as controllable photorealistic image synthesis, instruction following, autonomous self-control, and state-of-the-art georeferencing. These results suggest a promising direction towards sophisticated generalist agents--capable of understanding and manipulating the spatial and temporal aspects of their material environments--with enhanced embodied reasoning capabilities. Training code, datasets, and model checkpoints are made available at this https URL.

Title: HyBiomass: Global Hyperspectral Imagery Benchmark Dataset for Evaluating Geospatial Foundation Models in Forest Aboveground Biomass Estimation

• Authors: Aaron Banze, Timothée Stassin, Nassim Ait Ali Braham, Rıdvan Salih Kuzu, Simon Besnard, Michael Schmitt
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2506.11314
• Pdf URL: https://arxiv.org/pdf/2506.11314
• Copy Paste: [[2506.11314]] HyBiomass: Global Hyperspectral Imagery Benchmark Dataset for Evaluating Geospatial Foundation Models in Forest Aboveground Biomass Estimation(https://arxiv.org/abs/2506.11314)
• Keywords: foundation model
• Abstract: Comprehensive evaluation of geospatial foundation models (Geo-FMs) requires benchmarking across diverse tasks, sensors, and geographic regions. However, most existing benchmark datasets are limited to segmentation or classification tasks, and focus on specific geographic areas. To address this gap, we introduce a globally distributed dataset for forest aboveground biomass (AGB) estimation, a pixel-wise regression task. This benchmark dataset combines co-located hyperspectral imagery (HSI) from the Environmental Mapping and Analysis Program (EnMAP) satellite and predictions of AGB density estimates derived from the Global Ecosystem Dynamics Investigation lidars, covering seven continental regions. Our experimental results on this dataset demonstrate that the evaluated Geo-FMs can match or, in some cases, surpass the performance of a baseline U-Net, especially when fine-tuning the encoder. We also find that the performance difference between the U-Net and Geo-FMs depends on the dataset size for each region and highlight the importance of the token patch size in the Vision Transformer backbone for accurate predictions in pixel-wise regression tasks. By releasing this globally distributed hyperspectral benchmark dataset, we aim to facilitate the development and evaluation of Geo-FMs for HSI applications. Leveraging this dataset additionally enables research into geographic bias and generalization capacity of Geo-FMs. The dataset and source code will be made publicly available.

Title: The Biased Samaritan: LLM biases in Perceived Kindness

• Authors: Jack H Fagan, Ruhaan Juyaal, Amy Yue-Ming Yu, Siya Pun
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2506.11361
• Pdf URL: https://arxiv.org/pdf/2506.11361
• Copy Paste: [[2506.11361]] The Biased Samaritan: LLM biases in Perceived Kindness(https://arxiv.org/abs/2506.11361)
• Keywords: generative
• Abstract: While Large Language Models (LLMs) have become ubiquitous in many fields, understanding and mitigating LLM biases is an ongoing issue. This paper provides a novel method for evaluating the demographic biases of various generative AI models. By prompting models to assess a moral patient's willingness to intervene constructively, we aim to quantitatively evaluate different LLMs' biases towards various genders, races, and ages. Our work differs from existing work by aiming to determine the baseline demographic identities for various commercial models and the relationship between the baseline and other demographics. We strive to understand if these biases are positive, neutral, or negative, and the strength of these biases. This paper can contribute to the objective assessment of bias in Large Language Models and give the user or developer the power to account for these biases in LLM output or in training future LLMs. Our analysis suggested two key findings: that models view the baseline demographic as a white middle-aged or young adult male; however, a general trend across models suggested that non-baseline demographics are more willing to help than the baseline. These methodologies allowed us to distinguish these two biases that are often tangled together.

Title: The Effect of Stochasticity in Score-Based Diffusion Sampling: a KL Divergence Analysis

• Authors: Bernardo P. Schaeffer, Ricardo M. S. Rosa, Glauco Valle
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11378
• Pdf URL: https://arxiv.org/pdf/2506.11378
• Copy Paste: [[2506.11378]] The Effect of Stochasticity in Score-Based Diffusion Sampling: a KL Divergence Analysis(https://arxiv.org/abs/2506.11378)
• Keywords: diffusion
• Abstract: Sampling in score-based diffusion models can be performed by solving either a probability flow ODE or a reverse-time stochastic differential equation (SDE) parameterized by an arbitrary stochasticity parameter. In this work, we study the effect of stochasticity on the generation process through bounds on the Kullback-Leibler (KL) divergence and complement the analysis with numerical and analytical examples. Our results apply to general forward SDEs with additive noise and Lipschitz-continuous score functions, and quantify how errors from the prior distribution and score approximation propagate under different choices of the stochasticity parameter. The theoretical bounds are derived using log-Sobolev inequalities for the marginals of the forward process, which enable a more effective control of the KL divergence decay along sampling. For exact score functions, we find that stochasticity acts as an error-correcting mechanism, decreasing KL divergence along the sampling trajectory. For an approximate score function, there is a trade-off between error correction and score error amplification, so that stochasticity can either improve or worsen the performance, depending on the structure of the score error. Numerical experiments on simple datasets and a fully analytical example are included to illustrate and enlighten the theoretical results.

Title: PPDiff: Diffusing in Hybrid Sequence-Structure Space for Protein-Protein Complex Design

• Authors: Zhenqiao Song, Tiaoxiao Li, Lei Li, Martin Renqiang Min
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2506.11420
• Pdf URL: https://arxiv.org/pdf/2506.11420
• Copy Paste: [[2506.11420]] PPDiff: Diffusing in Hybrid Sequence-Structure Space for Protein-Protein Complex Design(https://arxiv.org/abs/2506.11420)
• Keywords: diffusion
• Abstract: Designing protein-binding proteins with high affinity is critical in biomedical research and biotechnology. Despite recent advancements targeting specific proteins, the ability to create high-affinity binders for arbitrary protein targets on demand, without extensive rounds of wet-lab testing, remains a significant challenge. Here, we introduce PPDiff, a diffusion model to jointly design the sequence and structure of binders for arbitrary protein targets in a non-autoregressive manner. PPDiffbuilds upon our developed Sequence Structure Interleaving Network with Causal attention layers (SSINC), which integrates interleaved self-attention layers to capture global amino acid correlations, k-nearest neighbor (kNN) equivariant graph layers to model local interactions in three-dimensional (3D) space, and causal attention layers to simplify the intricate interdependencies within the protein sequence. To assess PPDiff, we curate PPBench, a general protein-protein complex dataset comprising 706,360 complexes from the Protein Data Bank (PDB). The model is pretrained on PPBenchand finetuned on two real-world applications: target-protein mini-binder complex design and antigen-antibody complex design. PPDiffconsistently surpasses baseline methods, achieving success rates of 50.00%, 23.16%, and 16.89% for the pretraining task and the two downstream applications, respectively.

Title: Auditing Data Provenance in Real-world Text-to-Image Diffusion Models for Privacy and Copyright Protection

• Authors: Jie Zhu, Leye Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11434
• Pdf URL: https://arxiv.org/pdf/2506.11434
• Copy Paste: [[2506.11434]] Auditing Data Provenance in Real-world Text-to-Image Diffusion Models for Privacy and Copyright Protection(https://arxiv.org/abs/2506.11434)
• Keywords: diffusion
• Abstract: Text-to-image diffusion model since its propose has significantly influenced the content creation due to its impressive generation capability. However, this capability depends on large-scale text-image datasets gathered from web platforms like social media, posing substantial challenges in copyright compliance and personal privacy leakage. Though there are some efforts devoted to explore approaches for auditing data provenance in text-to-image diffusion models, existing work has unrealistic assumptions that can obtain model internal knowledge, e.g., intermediate results, or the evaluation is not reliable. To fill this gap, we propose a completely black-box auditing framework called Feature Semantic Consistency-based Auditing (FSCA). It utilizes two types of semantic connections within the text-to-image diffusion model for auditing, eliminating the need for access to internal knowledge. To demonstrate the effectiveness of our FSCA framework, we perform extensive experiments on LAION-mi dataset and COCO dataset, and compare with eight state-of-the-art baseline approaches. The results show that FSCA surpasses previous baseline approaches across various metrics and different data distributions, showcasing the superiority of our FSCA. Moreover, we introduce a recall balance strategy and a threshold adjustment strategy, which collectively allows FSCA to reach up a user-level accuracy of 90% in a real-world auditing scenario with only 10 samples/user, highlighting its strong auditing potential in real-world applications. Our code is made available at this https URL.

Title: TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models

• Authors: Ziyang Luo, Nian Liu, Xuguang Yang, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Junwei Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11436
• Pdf URL: https://arxiv.org/pdf/2506.11436
• Copy Paste: [[2506.11436]] TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models(https://arxiv.org/abs/2506.11436)
• Keywords: foundation model
• Abstract: Audio-Visual Segmentation (AVS) faces a fundamental challenge of effectively aligning audio and visual modalities. While recent approaches leverage foundation models to address data scarcity, they often rely on single-modality knowledge or combine foundation models in an off-the-shelf manner, failing to address the cross-modal alignment challenge. In this paper, we present TAViS, a novel framework that \textbf{couples} the knowledge of multimodal foundation models (ImageBind) for cross-modal alignment and a segmentation foundation model (SAM2) for precise segmentation. However, effectively combining these models poses two key challenges: the difficulty in transferring the knowledge between SAM2 and ImageBind due to their different feature spaces, and the insufficiency of using only segmentation loss for supervision. To address these challenges, we introduce a text-bridged design with two key components: (1) a text-bridged hybrid prompting mechanism where pseudo text provides class prototype information while retaining modality-specific details from both audio and visual inputs, and (2) an alignment supervision strategy that leverages text as a bridge to align shared semantic concepts within audio-visual modalities. Our approach achieves superior performance on single-source, multi-source, semantic datasets, and excels in zero-shot settings.

Title: Uncertainty Awareness Enables Efficient Labeling for Cancer Subtyping in Digital Pathology

• Authors: Nirhoshan Sivaroopan, Chamuditha Jayanga Galappaththige, Chalani Ekanayake, Hasindri Watawana, Ranga Rodrigo, Chamira U. S. Edussooriya, Dushan N. Wadduwage
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11439
• Pdf URL: https://arxiv.org/pdf/2506.11439
• Copy Paste: [[2506.11439]] Uncertainty Awareness Enables Efficient Labeling for Cancer Subtyping in Digital Pathology(https://arxiv.org/abs/2506.11439)
• Keywords: self-supervised
• Abstract: Machine-learning-assisted cancer subtyping is a promising avenue in digital pathology. Cancer subtyping models, however, require careful training using expert annotations so that they can be inferred with a degree of known certainty (or uncertainty). To this end, we introduce the concept of uncertainty awareness into a self-supervised contrastive learning model. This is achieved by computing an evidence vector at every epoch, which assesses the model's confidence in its predictions. The derived uncertainty score is then utilized as a metric to selectively label the most crucial images that require further annotation, thus iteratively refining the training process. With just 1-10% of strategically selected annotations, we attain state-of-the-art performance in cancer subtyping on benchmark datasets. Our method not only strategically guides the annotation process to minimize the need for extensive labeled datasets, but also improves the precision and efficiency of classifications. This development is particularly beneficial in settings where the availability of labeled data is limited, offering a promising direction for future research and application in digital pathology.

Title: GaussMarker: Robust Dual-Domain Watermark for Diffusion Models

• Authors: Kecen Li, Zhicong Huang, Xinwen Hou, Cheng Hong
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11444
• Pdf URL: https://arxiv.org/pdf/2506.11444
• Copy Paste: [[2506.11444]] GaussMarker: Robust Dual-Domain Watermark for Diffusion Models(https://arxiv.org/abs/2506.11444)
• Keywords: diffusion
• Abstract: As Diffusion Models (DM) generate increasingly realistic images, related issues such as copyright and misuse have become a growing concern. Watermarking is one of the promising solutions. Existing methods inject the watermark into the single-domain of initial Gaussian noise for generation, which suffers from unsatisfactory robustness. This paper presents the first dual-domain DM watermarking approach using a pipelined injector to consistently embed watermarks in both the spatial and frequency domains. To further boost robustness against certain image manipulations and advanced attacks, we introduce a model-independent learnable Gaussian Noise Restorer (GNR) to refine Gaussian noise extracted from manipulated images and enhance detection robustness by integrating the detection scores of both watermarks. GaussMarker efficiently achieves state-of-the-art performance under eight image distortions and four advanced attacks across three versions of Stable Diffusion with better recall and lower false positive rates, as preferred in real applications.

Title: FAME: A Lightweight Spatio-Temporal Network for Model Attribution of Face-Swap Deepfakes

• Authors: Wasim Ahmad, Yan-Tsung Peng, Yuan-Hao Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11477
• Pdf URL: https://arxiv.org/pdf/2506.11477
• Copy Paste: [[2506.11477]] FAME: A Lightweight Spatio-Temporal Network for Model Attribution of Face-Swap Deepfakes(https://arxiv.org/abs/2506.11477)
• Keywords: generative
• Abstract: The widespread emergence of face-swap Deepfake videos poses growing risks to digital security, privacy, and media integrity, necessitating effective forensic tools for identifying the source of such manipulations. Although most prior research has focused primarily on binary Deepfake detection, the task of model attribution -- determining which generative model produced a given Deepfake -- remains underexplored. In this paper, we introduce FAME (Fake Attribution via Multilevel Embeddings), a lightweight and efficient spatio-temporal framework designed to capture subtle generative artifacts specific to different face-swap models. FAME integrates spatial and temporal attention mechanisms to improve attribution accuracy while remaining computationally efficient. We evaluate our model on three challenging and diverse datasets: Deepfake Detection and Manipulation (DFDM), FaceForensics++, and FakeAVCeleb. Results show that FAME consistently outperforms existing methods in both accuracy and runtime, highlighting its potential for deployment in real-world forensic and information security applications.

Title: Composite Data Augmentations for Synthetic Image Detection Against Real-World Perturbations

• Authors: Efthymia Amarantidou, Christos Koutlis, Symeon Papadopoulos, Panagiotis C. Petrantonakis
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11490
• Pdf URL: https://arxiv.org/pdf/2506.11490
• Copy Paste: [[2506.11490]] Composite Data Augmentations for Synthetic Image Detection Against Real-World Perturbations(https://arxiv.org/abs/2506.11490)
• Keywords: generative
• Abstract: The advent of accessible Generative AI tools enables anyone to create and spread synthetic images on social media, often with the intention to mislead, thus posing a significant threat to online information integrity. Most existing Synthetic Image Detection (SID) solutions struggle on generated images sourced from the Internet, as these are often altered by compression and other operations. To address this, our research enhances SID by exploring data augmentation combinations, leveraging a genetic algorithm for optimal augmentation selection, and introducing a dual-criteria optimization approach. These methods significantly improve model performance under real-world perturbations. Our findings provide valuable insights for developing detection models capable of identifying synthetic images across varying qualities and transformations, with the best-performing model achieving a mean average precision increase of +22.53% compared to models without augmentations. The implementation is available at this http URL.

Title: Task-Driven Discrete Representation Learning

• Authors: Tung-Long Vuong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11511
• Pdf URL: https://arxiv.org/pdf/2506.11511
• Copy Paste: [[2506.11511]] Task-Driven Discrete Representation Learning(https://arxiv.org/abs/2506.11511)
• Keywords: generative
• Abstract: In recent years, deep discrete representation learning (DRL) has achieved significant success across various domains. Most DRL frameworks (e.g., the widely used VQ-VAE and its variants) have primarily focused on generative settings, where the quality of a representation is implicitly gauged by the fidelity of its generation. In fact, the goodness of a discrete representation remain ambiguously defined across the literature. In this work, we adopt a practical approach that examines DRL from a task-driven perspective. We propose a unified framework that explores the usefulness of discrete features in relation to downstream tasks, with generation naturally viewed as one possible application. In this context, the properties of discrete representations as well as the way they benefit certain tasks are also relatively understudied. We therefore provide an additional theoretical analysis of the trade-off between representational capacity and sample complexity, shedding light on how discrete representation utilization impacts task performance. Finally, we demonstrate the flexibility and effectiveness of our framework across diverse applications.

Title: Prioritizing Alignment Paradigms over Task-Specific Model Customization in Time-Series LLMs

• Authors: Wei Li, Yunyao Cheng, Xinli Hao, Chaohong Ma, Yuxuan Liang, Bin Yang, Christian S.Jensen, Xiaofeng Meng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11512
• Pdf URL: https://arxiv.org/pdf/2506.11512
• Copy Paste: [[2506.11512]] Prioritizing Alignment Paradigms over Task-Specific Model Customization in Time-Series LLMs(https://arxiv.org/abs/2506.11512)
• Keywords: anomaly
• Abstract: Recent advances in Large Language Models (LLMs) have enabled unprecedented capabilities for time-series reasoning in diverse real-world applications, including medical, financial, and spatio-temporal domains. However, existing approaches typically focus on task-specific model customization, such as forecasting and anomaly detection, while overlooking the data itself, referred to as time-series primitives, which are essential for in-depth reasoning. This position paper advocates a fundamental shift in approaching time-series reasoning with LLMs: prioritizing alignment paradigms grounded in the intrinsic primitives of time series data over task-specific model customization. This realignment addresses the core limitations of current time-series reasoning approaches, which are often costly, inflexible, and inefficient, by systematically accounting for intrinsic structure of data before task engineering. To this end, we propose three alignment paradigms: Injective Alignment, Bridging Alignment, and Internal Alignment, which are emphasized by prioritizing different aspects of time-series primitives: domain, characteristic, and representation, respectively, to activate time-series reasoning capabilities of LLMs to enable economical, flexible, and efficient reasoning. We further recommend that practitioners adopt an alignment-oriented method to avail this instruction to select an appropriate alignment paradigm. Additionally, we categorize relevant literature into these alignment paradigms and outline promising research directions.

Title: Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning

• Authors: Chengye Li, Haiyun Liu, Yuanxi Li
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2506.11516
• Pdf URL: https://arxiv.org/pdf/2506.11516
• Copy Paste: [[2506.11516]] Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning(https://arxiv.org/abs/2506.11516)
• Keywords: in-context
• Abstract: In-context learning (ICL) allows large language models (LLMs) to solve novel tasks without weight updates. Despite its empirical success, the mechanism behind ICL remains poorly understood, limiting our ability to interpret, improve, and reliably apply it. In this paper, we propose a new theoretical perspective that interprets ICL as an implicit form of knowledge distillation (KD), where prompt demonstrations guide the model to form a task-specific reference model during inference. Under this view, we derive a Rademacher complexity-based generalization bound and prove that the bias of the distilled weights grows linearly with the Maximum Mean Discrepancy (MMD) between the prompt and target distributions. This theoretical framework explains several empirical phenomena and unifies prior gradient-based and distributional analyses. To the best of our knowledge, this is the first to formalize inference-time attention as a distillation process, which provides theoretical insights for future prompt engineering and automated demonstration selection.

Title: Robust Filtering -- Novel Statistical Learning and Inference Algorithms with Applications

• Authors: Aamir Hussain Chughtai
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2506.11530
• Pdf URL: https://arxiv.org/pdf/2506.11530
• Copy Paste: [[2506.11530]] Robust Filtering -- Novel Statistical Learning and Inference Algorithms with Applications(https://arxiv.org/abs/2506.11530)
• Keywords: diffusion, generative
• Abstract: State estimation or filtering serves as a fundamental task to enable intelligent decision-making in applications such as autonomous vehicles, robotics, healthcare monitoring, smart grids, intelligent transportation, and predictive maintenance. Standard filtering assumes prior knowledge of noise statistics to extract latent system states from noisy sensor data. However, real-world scenarios involve abnormalities like outliers, biases, drifts, and missing observations with unknown or partially known statistics, limiting conventional approaches. This thesis presents novel robust nonlinear filtering methods to mitigate these challenges. Based on insights from our filtering proposals, we extend the formulations to offline estimation/learning setups and propose smoothing extensions. Our methods leverage Bayesian inference frameworks, employing both deterministic and stochastic approximation techniques including Variational Inference (VI) and Particle Filters/Sequential Monte Carlo (SMC). We also study theoretical estimation limits using Bayesian Cramér-Rao bounds (BCRBs) in the context of measurement abnormalities. To validate the performance gains of the proposed methods, we perform simulations and experiments in scenarios including target tracking, indoor localization, 3D point cloud registration, mesh registration, and pose graph optimization. The fundamental nature of the work makes it useful in diverse applications, with possible future extensions toward developing outlier-robust machine learning pipelines, learning system dynamics from anomalous data, and addressing challenges in generative AI where standard diffusion models struggle with outliers, imbalanced datasets, and mode collapse.

Title: Learn to Preserve Personality: Federated Foundation Models in Recommendations

• Authors: Zhiwei Li, Guodong Long, Chunxu Zhang, Honglei Zhang, Jing Jiang, Chengqi Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11563
• Pdf URL: https://arxiv.org/pdf/2506.11563
• Copy Paste: [[2506.11563]] Learn to Preserve Personality: Federated Foundation Models in Recommendations(https://arxiv.org/abs/2506.11563)
• Keywords: foundation model
• Abstract: A core learning challenge for existed Foundation Models (FM) is striking the tradeoff between generalization with personalization, which is a dilemma that has been highlighted by various parameter-efficient adaptation techniques. Federated foundation models (FFM) provide a structural means to decouple shared knowledge from individual specific adaptations via decentralized processes. Recommendation systems offer a perfect testbed for FFMs, given their reliance on rich implicit feedback reflecting unique user characteristics. This position paper discusses a novel learning paradigm where FFMs not only harness their generalization capabilities but are specifically designed to preserve the integrity of user personality, illustrated thoroughly within the recommendation contexts. We envision future personal agents, powered by personalized adaptive FMs, guiding user decisions on content. Such an architecture promises a user centric, decentralized system where individuals maintain control over their personalized agents.

Title: A$^2$LC: Active and Automated Label Correction for Semantic Segmentation

• Authors: Youjin Jeon, Kyusik Cho, Suhan Woo, Euntai Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.11599
• Pdf URL: https://arxiv.org/pdf/2506.11599
• Copy Paste: [[2506.11599]] A$^2$LC: Active and Automated Label Correction for Semantic Segmentation(https://arxiv.org/abs/2506.11599)
• Keywords: foundation model
• Abstract: Active Label Correction (ALC) has emerged as a promising solution to the high cost and error-prone nature of manual pixel-wise annotation in semantic segmentation, by selectively identifying and correcting mislabeled data. Although recent work has improved correction efficiency by generating pseudo-labels using foundation models, substantial inefficiencies still remain. In this paper, we propose Active and Automated Label Correction for semantic segmentation (A$^2$LC), a novel and efficient ALC framework that integrates an automated correction stage into the conventional pipeline. Specifically, the automated correction stage leverages annotator feedback to perform label correction beyond the queried samples, thereby maximizing cost efficiency. In addition, we further introduce an adaptively balanced acquisition function that emphasizes underrepresented tail classes and complements the automated correction mechanism. Extensive experiments on Cityscapes and PASCAL VOC 2012 demonstrate that A$^2$LC significantly outperforms previous state-of-the-art methods. Notably, A$^2$LC achieves high efficiency by outperforming previous methods using only 20% of their budget, and demonstrates strong effectiveness by yielding a 27.23% performance improvement under an equivalent budget constraint on the Cityscapes dataset. The code will be released upon acceptance.

Title: Cross-Modal Clustering-Guided Negative Sampling for Self-Supervised Joint Learning from Medical Images and Reports

• Authors: Libin Lan, Hongxing Li, Zunhui Xia, Juan Zhou, Xiaofei Zhu, Yongmei Li, Yudong Zhang, Xin Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11674
• Pdf URL: https://arxiv.org/pdf/2506.11674
• Copy Paste: [[2506.11674]] Cross-Modal Clustering-Guided Negative Sampling for Self-Supervised Joint Learning from Medical Images and Reports(https://arxiv.org/abs/2506.11674)
• Keywords: self-supervised
• Abstract: Learning medical visual representations directly from paired images and reports through multimodal self-supervised learning has emerged as a novel and efficient approach to digital diagnosis in recent years. However, existing models suffer from several severe limitations. 1) neglecting the selection of negative samples, resulting in the scarcity of hard negatives and the inclusion of false negatives; 2) focusing on global feature extraction, but overlooking the fine-grained local details that are crucial for medical image recognition tasks; and 3) contrastive learning primarily targets high-level features but ignoring low-level details which are essential for accurate medical analysis. Motivated by these critical issues, this paper presents a Cross-Modal Cluster-Guided Negative Sampling (CM-CGNS) method with two-fold ideas. First, it extends the k-means clustering used for local text features in the single-modal domain to the multimodal domain through cross-modal attention. This improvement increases the number of negative samples and boosts the model representation capability. Second, it introduces a Cross-Modal Masked Image Reconstruction (CM-MIR) module that leverages local text-to-image features obtained via cross-modal attention to reconstruct masked local image regions. This module significantly strengthens the model's cross-modal information interaction capabilities and retains low-level image features essential for downstream tasks. By well handling the aforementioned limitations, the proposed CM-CGNS can learn effective and robust medical visual representations suitable for various recognition tasks. Extensive experimental results on classification, detection, and segmentation tasks across five downstream datasets show that our method outperforms state-of-the-art approaches on multiple metrics, verifying its superior performance.

Title: Geometry-Aware Edge Pooling for Graph Neural Networks

• Authors: Katharina Limbeck, Lydia Mezrag, Guy Wolf, Bastian Rieck
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11700
• Pdf URL: https://arxiv.org/pdf/2506.11700
• Copy Paste: [[2506.11700]] Geometry-Aware Edge Pooling for Graph Neural Networks(https://arxiv.org/abs/2506.11700)
• Keywords: diffusion
• Abstract: Graph Neural Networks (GNNs) have shown significant success for graph-based tasks. Motivated by the prevalence of large datasets in real-world applications, pooling layers are crucial components of GNNs. By reducing the size of input graphs, pooling enables faster training and potentially better generalisation. However, existing pooling operations often optimise for the learning task at the expense of fundamental graph structures and interpretability. This leads to unreliable performance across varying dataset types, downstream tasks and pooling ratios. Addressing these concerns, we propose novel graph pooling layers for structure aware pooling via edge collapses. Our methods leverage diffusion geometry and iteratively reduce a graph's size while preserving both its metric structure and structural diversity. We guide pooling using magnitude, an isometry-invariant diversity measure, which permits us to control the fidelity of the pooling process. Further, we use the spread of a metric space as a faster and more stable alternative ensuring computational efficiency. Empirical results demonstrate that our methods (i) achieve superior performance compared to alternative pooling layers across a range of diverse graph classification tasks, (ii) preserve key spectral properties of the input graphs, and (iii) retain high accuracy across varying pooling ratios.

Title: Quizzard@INOVA Challenge 2025 -- Track A: Plug-and-Play Technique in Interleaved Multi-Image Model

• Authors: Dinh Viet Cuong, Hoang-Bao Le, An Pham Ngoc Nguyen, Liting Zhou, Cathal Gurrin
• Subjects: cs.CV, cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2506.11737
• Pdf URL: https://arxiv.org/pdf/2506.11737
• Copy Paste: [[2506.11737]] Quizzard@INOVA Challenge 2025 -- Track A: Plug-and-Play Technique in Interleaved Multi-Image Model(https://arxiv.org/abs/2506.11737)
• Keywords: foundation model
• Abstract: This paper addresses two main objectives. Firstly, we demonstrate the impressive performance of the LLaVA-NeXT-interleave on 22 datasets across three different tasks: Multi-Image Reasoning, Documents and Knowledge-Based Understanding and Interactive Multi-Modal Communication. Secondly, we add the Dense Channel Integration (DCI) connector to the LLaVA-NeXT-Interleave and compare its performance against the standard model. We find that the standard model achieves the highest overall accuracy, excelling in vision-heavy tasks like VISION, NLVR2, and Fashion200K. Meanwhile, the DCI-enhanced version shows particular strength on datasets requiring deeper semantic coherence or structured change understanding such as MIT-States_PropertyCoherence and SlideVQA. Our results highlight the potential of combining powerful foundation models with plug-and-play techniques for Interleave tasks. The code is available at this https URL.

Title: DiffFuSR: Super-Resolution of all Sentinel-2 Multispectral Bands using Diffusion Models

• Authors: Muhammad Sarmad, Arnt-Børre Salberg, Michael Kampffmeyer
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2506.11764
• Pdf URL: https://arxiv.org/pdf/2506.11764
• Copy Paste: [[2506.11764]] DiffFuSR: Super-Resolution of all Sentinel-2 Multispectral Bands using Diffusion Models(https://arxiv.org/abs/2506.11764)
• Keywords: diffusion, generative
• Abstract: This paper presents DiffFuSR, a modular pipeline for super-resolving all 12 spectral bands of Sentinel-2 Level-2A imagery to a unified ground sampling distance (GSD) of 2.5 meters. The pipeline comprises two stages: (i) a diffusion-based super-resolution (SR) model trained on high-resolution RGB imagery from the NAIP and WorldStrat datasets, harmonized to simulate Sentinel-2 characteristics; and (ii) a learned fusion network that upscales the remaining multispectral bands using the super-resolved RGB image as a spatial prior. We introduce a robust degradation model and contrastive degradation encoder to support blind SR. Extensive evaluations of the proposed SR pipeline on the OpenSR benchmark demonstrate that the proposed method outperforms current SOTA baselines in terms of reflectance fidelity, spectral consistency, spatial alignment, and hallucination suppression. Furthermore, the fusion network significantly outperforms classical pansharpening approaches, enabling accurate enhancement of Sentinel-2's 20 m and 60 m bands. This study underscores the power of harmonized learning with generative priors and fusion strategies to create a modular framework for Sentinel-2 SR. Our code and models can be found at this https URL.

Title: CLIP Meets Diffusion: A Synergistic Approach to Anomaly Detection

• Authors: Byeongchan Lee, John Won, Seunghyun Lee, Jinwoo Shin
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11772
• Pdf URL: https://arxiv.org/pdf/2506.11772
• Copy Paste: [[2506.11772]] CLIP Meets Diffusion: A Synergistic Approach to Anomaly Detection(https://arxiv.org/abs/2506.11772)
• Keywords: diffusion, foundation model, generative, anomaly
• Abstract: Anomaly detection is a complex problem due to the ambiguity in defining anomalies, the diversity of anomaly types (e.g., local and global defect), and the scarcity of training data. As such, it necessitates a comprehensive model capable of capturing both low-level and high-level features, even with limited data. To address this, we propose CLIPFUSION, a method that leverages both discriminative and generative foundation models. Specifically, the CLIP-based discriminative model excels at capturing global features, while the diffusion-based generative model effectively captures local details, creating a synergistic and complementary approach. Notably, we introduce a methodology for utilizing cross-attention maps and feature maps extracted from diffusion models specifically for anomaly detection. Experimental results on benchmark datasets (MVTec-AD, VisA) demonstrate that CLIPFUSION consistently outperforms baseline methods, achieving outstanding performance in both anomaly segmentation and classification. We believe that our method underscores the effectiveness of multi-modal and multi-model fusion in tackling the multifaceted challenges of anomaly detection, providing a scalable solution for real-world applications.

Title: Self-supervised Learning of Echocardiographic Video Representations via Online Cluster Distillation

• Authors: Divyanshu Mishra, Mohammadreza Salehi, Pramit Saha, Olga Patey, Aris T. Papageorghiou, Yuki M. Asano, J. Alison Noble
• Subjects: cs.CV, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11777
• Pdf URL: https://arxiv.org/pdf/2506.11777
• Copy Paste: [[2506.11777]] Self-supervised Learning of Echocardiographic Video Representations via Online Cluster Distillation(https://arxiv.org/abs/2506.11777)
• Keywords: self-supervised, anomaly
• Abstract: Self-supervised learning (SSL) has achieved major advances in natural images and video understanding, but challenges remain in domains like echocardiography (heart ultrasound) due to subtle anatomical structures, complex temporal dynamics, and the current lack of domain-specific pre-trained models. Existing SSL approaches such as contrastive, masked modeling, and clustering-based methods struggle with high intersample similarity, sensitivity to low PSNR inputs common in ultrasound, or aggressive augmentations that distort clinically relevant features. We present DISCOVR (Distilled Image Supervision for Cross Modal Video Representation), a self-supervised dual branch framework for cardiac ultrasound video representation learning. DISCOVR combines a clustering-based video encoder that models temporal dynamics with an online image encoder that extracts fine-grained spatial semantics. These branches are connected through a semantic cluster distillation loss that transfers anatomical knowledge from the evolving image encoder to the video encoder, enabling temporally coherent representations enriched with fine-grained semantic understanding. Evaluated on six echocardiography datasets spanning fetal, pediatric, and adult populations, DISCOVR outperforms both specialized video anomaly detection methods and state-of-the-art video-SSL baselines in zero-shot and linear probing setups, and achieves superior segmentation transfer.

Title: SSPINNpose: A Self-Supervised PINN for Inertial Pose and Dynamics Estimation

• Authors: Markus Gambietz, Eva Dorschky, Altan Akat, Marcel Schöckel, Jörg Miehling, Anne D. Koelewijn
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11786
• Pdf URL: https://arxiv.org/pdf/2506.11786
• Copy Paste: [[2506.11786]] SSPINNpose: A Self-Supervised PINN for Inertial Pose and Dynamics Estimation(https://arxiv.org/abs/2506.11786)
• Keywords: self-supervised
• Abstract: Accurate real-time estimation of human movement dynamics, including internal joint moments and muscle forces, is essential for applications in clinical diagnostics and sports performance monitoring. Inertial measurement units (IMUs) provide a minimally intrusive solution for capturing motion data, particularly when used in sparse sensor configurations. However, current real-time methods rely on supervised learning, where a ground truth dataset needs to be measured with laboratory measurement systems, such as optical motion capture. These systems are known to introduce measurement and processing errors and often fail to generalize to real-world or previously unseen movements, necessitating new data collection efforts that are time-consuming and impractical. To overcome these limitations, we propose SSPINNpose, a self-supervised, physics-informed neural network that estimates joint kinematics and kinetics directly from IMU data, without requiring ground truth labels for training. We run the network output through a physics model of the human body to optimize physical plausibility and generate virtual measurement data. Using this virtual sensor data, the network is trained directly on the measured sensor data instead of a ground truth. When compared to optical motion capture, SSPINNpose is able to accurately estimate joint angles and joint moments at an RMSD of 8.7 deg and 4.9 BWBH%, respectively, for walking and running at speeds up to 4.9 m/s at a latency of 3.5 ms. Furthermore, the framework demonstrates robustness across sparse sensor configurations and can infer the anatomical locations of the sensors. These results underscore the potential of SSPINNpose as a scalable and adaptable solution for real-time biomechanical analysis in both laboratory and field environments.

Title: Measurement-aligned Flow for Inverse Problem

• Authors: Shaorong Zhang, Rob Brekelmans, Yunshu Wu, Greg Ver Steeg
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2506.11893
• Pdf URL: https://arxiv.org/pdf/2506.11893
• Copy Paste: [[2506.11893]] Measurement-aligned Flow for Inverse Problem(https://arxiv.org/abs/2506.11893)
• Keywords: diffusion
• Abstract: Diffusion models provide a powerful way to incorporate complex prior information for solving inverse problems. However, existing methods struggle to correctly incorporate guidance from conflicting signals in the prior and measurement, especially in the challenging setting of non-Gaussian or unknown noise. To bridge these gaps, we propose Measurement-Aligned Sampling (MAS), a novel framework for linear inverse problem solving that can more flexibly balance prior and measurement information. MAS unifies and extends existing approaches like DDNM and DAPS, and offers a new optimization perspective. MAS can generalize to handle known Gaussian noise, unknown or non-Gaussian noise types. Extensive experiments show that MAS consistently outperforms state-of-the-art methods across a range of tasks.

Title: Aligned Novel View Image and Geometry Synthesis via Cross-modal Attention Instillation

• Authors: Min-Seop Kwak, Junho Kim, Sangdoo Yun, Dongyoon Han, Taekyoung Kim, Seungryong Kim, Jin-Hwa Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11924
• Pdf URL: https://arxiv.org/pdf/2506.11924
• Copy Paste: [[2506.11924]] Aligned Novel View Image and Geometry Synthesis via Cross-modal Attention Instillation(https://arxiv.org/abs/2506.11924)
• Keywords: diffusion, generative
• Abstract: We introduce a diffusion-based framework that performs aligned novel view image and geometry generation via a warping-and-inpainting methodology. Unlike prior methods that require dense posed images or pose-embedded generative models limited to in-domain views, our method leverages off-the-shelf geometry predictors to predict partial geometries viewed from reference images, and formulates novel-view synthesis as an inpainting task for both image and geometry. To ensure accurate alignment between generated images and geometry, we propose cross-modal attention distillation, where attention maps from the image diffusion branch are injected into a parallel geometry diffusion branch during both training and inference. This multi-task approach achieves synergistic effects, facilitating geometrically robust image synthesis as well as well-defined geometry prediction. We further introduce proximity-based mesh conditioning to integrate depth and normal cues, interpolating between point cloud and filtering erroneously predicted geometry from influencing the generation process. Empirically, our method achieves high-fidelity extrapolative view synthesis on both image and geometry across a range of unseen scenes, delivers competitive reconstruction quality under interpolation settings, and produces geometrically aligned colored point clouds for comprehensive 3D completion. Project page is available at this https URL.

Title: Visual Pre-Training on Unlabeled Images using Reinforcement Learning

• Authors: Dibya Ghosh, Sergey Levine
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2506.11967
• Pdf URL: https://arxiv.org/pdf/2506.11967
• Copy Paste: [[2506.11967]] Visual Pre-Training on Unlabeled Images using Reinforcement Learning(https://arxiv.org/abs/2506.11967)
• Keywords: self-supervised
• Abstract: In reinforcement learning (RL), value-based algorithms learn to associate each observation with the states and rewards that are likely to be reached from it. We observe that many self-supervised image pre-training methods bear similarity to this formulation: learning features that associate crops of images with those of nearby views, e.g., by taking a different crop or color augmentation. In this paper, we complete this analogy and explore a method that directly casts pre-training on unlabeled image data like web crawls and video frames as an RL problem. We train a general value function in a dynamical system where an agent transforms an image by changing the view or adding image augmentations. Learning in this way resembles crop-consistency self-supervision, but through the reward function, offers a simple lever to shape feature learning using curated images or weakly labeled captions when they exist. Our experiments demonstrate improved representations when training on unlabeled images in the wild, including video data like EpicKitchens, scene data like COCO, and web-crawl data like CC12M.

Title: EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction

• Authors: Hsi-Che Lin, Yu-Chu Yu, Kai-Po Chang, Yu-Chiang Frank Wang
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2506.12015
• Pdf URL: https://arxiv.org/pdf/2506.12015
• Copy Paste: [[2506.12015]] EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction(https://arxiv.org/abs/2506.12015)
• Keywords: foundation model
• Abstract: Open-source foundation models have seen rapid adoption and development, enabling powerful general-purpose capabilities across diverse domains. However, fine-tuning large foundation models for domain-specific or personalized tasks remains prohibitively expensive for most users due to the significant memory overhead beyond that of inference. We introduce EMLoC, an Emulator-based Memory-efficient fine-tuning framework with LoRA Correction, which enables model fine-tuning within the same memory budget required for inference. EMLoC constructs a task-specific light-weight emulator using activation-aware singular value decomposition (SVD) on a small downstream calibration set. Fine-tuning then is performed on this lightweight emulator via LoRA. To tackle the misalignment between the original model and the compressed emulator, we propose a novel compensation algorithm to correct the fine-tuned LoRA module, which thus can be merged into the original model for inference. EMLoC supports flexible compression ratios and standard training pipelines, making it adaptable to a wide range of applications. Extensive experiments demonstrate that EMLoC outperforms other baselines across multiple datasets and modalities. Moreover, without quantization, EMLoC enables fine-tuning of a 38B model on a single 24GB consumer GPU-bringing efficient and practical model adaptation to individual users.