2024-11-14

Title: EAPCR: A Universal Feature Extractor for Scientific Data without Explicit Feature Relation Patterns

• Authors: Zhuohang Yu, Ling An, Yansong Li, Yu Wu, Zeyu Dong, Zhangdi Liu, Le Gao, Zhenyu Zhang, Chichun Zhou
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08164
• Pdf URL: https://arxiv.org/pdf/2411.08164
• Copy Paste: [[2411.08164]] EAPCR: A Universal Feature Extractor for Scientific Data without Explicit Feature Relation Patterns(https://arxiv.org/abs/2411.08164)
• Keywords: anomaly
• Abstract: Conventional methods, including Decision Tree (DT)-based methods, have been effective in scientific tasks, such as non-image medical diagnostics, system anomaly detection, and inorganic catalysis efficiency prediction. However, most deep-learning techniques have struggled to surpass or even match this level of success as traditional machine-learning methods. The primary reason is that these applications involve multi-source, heterogeneous data where features lack explicit relationships. This contrasts with image data, where pixels exhibit spatial relationships; textual data, where words have sequential dependencies; and graph data, where nodes are connected through established associations. The absence of explicit Feature Relation Patterns (FRPs) presents a significant challenge for deep learning techniques in scientific applications that are not image, text, and graph-based. In this paper, we introduce EAPCR, a universal feature extractor designed for data without explicit FRPs. Tested across various scientific tasks, EAPCR consistently outperforms traditional methods and bridges the gap where deep learning models fall short. To further demonstrate its robustness, we synthesize a dataset without explicit FRPs. While Kolmogorov-Arnold Network (KAN) and feature extractors like Convolutional Neural Networks (CNNs), Graph Convolutional Networks (GCNs), and Transformers struggle, EAPCR excels, demonstrating its robustness and superior performance in scientific tasks without FRPs.

Title: Latent Space Disentanglement in Diffusion Transformers Enables Precise Zero-shot Semantic Editing

• Authors: Zitao Shuai, Chenwei Wu, Zhengxu Tang, Bowen Song, Liyue Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08196
• Pdf URL: https://arxiv.org/pdf/2411.08196
• Copy Paste: [[2411.08196]] Latent Space Disentanglement in Diffusion Transformers Enables Precise Zero-shot Semantic Editing(https://arxiv.org/abs/2411.08196)
• Keywords: diffusion
• Abstract: Diffusion Transformers (DiTs) have recently achieved remarkable success in text-guided image generation. In image editing, DiTs project text and image inputs to a joint latent space, from which they decode and synthesize new images. However, it remains largely unexplored how multimodal information collectively forms this joint space and how they guide the semantics of the synthesized images. In this paper, we investigate the latent space of DiT models and uncover two key properties: First, DiT's latent space is inherently semantically disentangled, where different semantic attributes can be controlled by specific editing directions. Second, consistent semantic editing requires utilizing the entire joint latent space, as neither encoded image nor text alone contains enough semantic information. We show that these editing directions can be obtained directly from text prompts, enabling precise semantic control without additional training or mask annotations. Based on these insights, we propose a simple yet effective Encode-Identify-Manipulate (EIM) framework for zero-shot fine-grained image editing. Specifically, we first encode both the given source image and the text prompt that describes the image, to obtain the joint latent embedding. Then, using our proposed Hessian Score Distillation Sampling (HSDS) method, we identify editing directions that control specific target attributes while preserving other image features. These directions are guided by text prompts and used to manipulate the latent embeddings. Moreover, we propose a new metric to quantify the disentanglement degree of the latent space of diffusion models. Extensive experiment results on our new curated benchmark dataset and analysis demonstrate DiT's disentanglement properties and effectiveness of the EIM framework.

Title: Joint Diffusion models in Continual Learning

• Authors: Paweł Skierś, Kamil Deja
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08224
• Pdf URL: https://arxiv.org/pdf/2411.08224
• Copy Paste: [[2411.08224]] Joint Diffusion models in Continual Learning(https://arxiv.org/abs/2411.08224)
• Keywords: diffusion, self-supervised, generative
• Abstract: In this work, we introduce JDCL - a new method for continual learning with generative rehearsal based on joint diffusion models. Neural networks suffer from catastrophic forgetting defined as abrupt loss in the model's performance when retrained with additional data coming from a different distribution. Generative-replay-based continual learning methods try to mitigate this issue by retraining a model with a combination of new and rehearsal data sampled from a generative model. In this work, we propose to extend this idea by combining a continually trained classifier with a diffusion-based generative model into a single - jointly optimized neural network. We show that such shared parametrization, combined with the knowledge distillation technique allows for stable adaptation to new tasks without catastrophic forgetting. We evaluate our approach on several benchmarks, where it outperforms recent state-of-the-art generative replay techniques. Additionally, we extend our method to the semi-supervised continual learning setup, where it outperforms competing buffer-based replay techniques, and evaluate, in a self-supervised manner, the quality of trained representations.

Title: Retrieval Augmented Time Series Forecasting

• Authors: Kutay Tire, Ege Onur Taga, Muhammed Emrullah Ildiz, Samet Oymak
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08249
• Pdf URL: https://arxiv.org/pdf/2411.08249
• Copy Paste: [[2411.08249]] Retrieval Augmented Time Series Forecasting(https://arxiv.org/abs/2411.08249)
• Keywords: foundation model
• Abstract: Retrieval-augmented generation (RAG) is a central component of modern LLM systems, particularly in scenarios where up-to-date information is crucial for accurately responding to user queries or when queries exceed the scope of the training data. The advent of time-series foundation models (TSFM), such as Chronos, and the need for effective zero-shot forecasting performance across various time-series domains motivates the question: Do benefits of RAG similarly carry over to time series forecasting? In this paper, we advocate that the dynamic and event-driven nature of time-series data makes RAG a crucial component of TSFMs and introduce a principled RAG framework for time-series forecasting, called Retrieval Augmented Forecasting (RAF). Within RAF, we develop efficient strategies for retrieving related time-series examples and incorporating them into forecast. Through experiments and mechanistic studies, we demonstrate that RAF indeed improves the forecasting accuracy across diverse time series domains and the improvement is more significant for larger TSFM sizes.

Title: SDDBench: A Benchmark for Synthesizable Drug Design

• Authors: Songtao Liu, Zhengkai Tu, Hanjun Dai, Peng Liu
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.08306
• Pdf URL: https://arxiv.org/pdf/2411.08306
• Copy Paste: [[2411.08306]] SDDBench: A Benchmark for Synthesizable Drug Design(https://arxiv.org/abs/2411.08306)
• Keywords: generative
• Abstract: A significant challenge in wet lab experiments with current drug design generative models is the trade-off between pharmacological properties and synthesizability. Molecules predicted to have highly desirable properties are often difficult to synthesize, while those that are easily synthesizable tend to exhibit less favorable properties. As a result, evaluating the synthesizability of molecules in general drug design scenarios remains a significant challenge in the field of drug discovery. The commonly used synthetic accessibility (SA) score aims to evaluate the ease of synthesizing generated molecules, but it falls short of guaranteeing that synthetic routes can actually be found. Inspired by recent advances in top-down synthetic route generation, we propose a new, data-driven metric to evaluate molecule synthesizability. Our approach directly assesses the feasibility of synthetic routes for a given molecule through our proposed round-trip score. This novel metric leverages the synergistic duality between retrosynthetic planners and reaction predictors, both of which are trained on extensive reaction datasets. To demonstrate the efficacy of our method, we conduct a comprehensive evaluation of round-trip scores alongside search success rate across a range of representative molecule generative models. Code is available at this https URL.

Title: Motion Control for Enhanced Complex Action Video Generation

• Authors: Qiang Zhou, Shaofeng Zhang, Nianzu Yang, Ye Qian, Hao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08328
• Pdf URL: https://arxiv.org/pdf/2411.08328
• Copy Paste: [[2411.08328]] Motion Control for Enhanced Complex Action Video Generation(https://arxiv.org/abs/2411.08328)
• Keywords: diffusion
• Abstract: Existing text-to-video (T2V) models often struggle with generating videos with sufficiently pronounced or complex actions. A key limitation lies in the text prompt's inability to precisely convey intricate motion details. To address this, we propose a novel framework, MVideo, designed to produce long-duration videos with precise, fluid actions. MVideo overcomes the limitations of text prompts by incorporating mask sequences as an additional motion condition input, providing a clearer, more accurate representation of intended actions. Leveraging foundational vision models such as GroundingDINO and SAM2, MVideo automatically generates mask sequences, enhancing both efficiency and robustness. Our results demonstrate that, after training, MVideo effectively aligns text prompts with motion conditions to produce videos that simultaneously meet both criteria. This dual control mechanism allows for more dynamic video generation by enabling alterations to either the text prompt or motion condition independently, or both in tandem. Furthermore, MVideo supports motion condition editing and composition, facilitating the generation of videos with more complex actions. MVideo thus advances T2V motion generation, setting a strong benchmark for improved action depiction in current video diffusion models. Our project page is available at this https URL.

Title: Physics Informed Distillation for Diffusion Models

• Authors: Joshua Tian Jin Tee, Kang Zhang, Hee Suk Yoon, Dhananjaya Nagaraja Gowda, Chanwoo Kim, Chang D. Yoo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08378
• Pdf URL: https://arxiv.org/pdf/2411.08378
• Copy Paste: [[2411.08378]] Physics Informed Distillation for Diffusion Models(https://arxiv.org/abs/2411.08378)
• Keywords: diffusion, generative
• Abstract: Diffusion models have recently emerged as a potent tool in generative modeling. However, their inherent iterative nature often results in sluggish image generation due to the requirement for multiple model evaluations. Recent progress has unveiled the intrinsic link between diffusion models and Probability Flow Ordinary Differential Equations (ODEs), thus enabling us to conceptualize diffusion models as ODE systems. Simultaneously, Physics Informed Neural Networks (PINNs) have substantiated their effectiveness in solving intricate differential equations through implicit modeling of their solutions. Building upon these foundational insights, we introduce Physics Informed Distillation (PID), which employs a student model to represent the solution of the ODE system corresponding to the teacher diffusion model, akin to the principles employed in PINNs. Through experiments on CIFAR 10 and ImageNet 64x64, we observe that PID achieves performance comparable to recent distillation methods. Notably, it demonstrates predictable trends concerning method-specific hyperparameters and eliminates the need for synthetic dataset generation during the distillation process. Both of which contribute to its easy-to-use nature as a distillation approach for Diffusion Models. Our code and pre-trained checkpoint are publicly available at: this https URL.

Title: CLaSP: Learning Concepts for Time-Series Signals from Natural Language Supervision

• Authors: Aoi Ito, Kota Dohi, Yohei Kawaguchi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08397
• Pdf URL: https://arxiv.org/pdf/2411.08397
• Copy Paste: [[2411.08397]] CLaSP: Learning Concepts for Time-Series Signals from Natural Language Supervision(https://arxiv.org/abs/2411.08397)
• Keywords: foundation model
• Abstract: This paper proposes a foundation model called "CLaSP" that can search time series signals using natural language that describes the characteristics of the signals as queries. Previous efforts to represent time series signal data in natural language have had challenges in designing a conventional class of time series signal characteristics, formulating their quantification, and creating a dictionary of synonyms. To overcome these limitations, the proposed method introduces a neural network based on contrastive learning. This network is first trained using the datasets TRUCE and SUSHI, which consist of time series signals and their corresponding natural language descriptions. Previous studies have proposed vocabularies that data analysts use to describe signal characteristics, and SUSHI was designed to cover these terms. We believe that a neural network trained on these datasets will enable data analysts to search using natural language vocabulary. Furthermore, our method does not require a dictionary of predefined synonyms, and it leverages common sense knowledge embedded in a large-scale language model (LLM). Experimental results demonstrate that CLaSP enables natural language search of time series signal data and can accurately learn the points at which signal data changes.

Title: V2X-R: Cooperative LiDAR-4D Radar Fusion for 3D Object Detection with Denoising Diffusion

• Authors: Xun Huang, Jinlong Wang, Qiming Xia, Siheng Chen, Bisheng Yang, Cheng Wang, Chenglu Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08402
• Pdf URL: https://arxiv.org/pdf/2411.08402
• Copy Paste: [[2411.08402]] V2X-R: Cooperative LiDAR-4D Radar Fusion for 3D Object Detection with Denoising Diffusion(https://arxiv.org/abs/2411.08402)
• Keywords: diffusion
• Abstract: Current Vehicle-to-Everything (V2X) systems have significantly enhanced 3D object detection using LiDAR and camera data. However, these methods suffer from performance degradation in adverse weather conditions. The weatherrobust 4D radar provides Doppler and additional geometric information, raising the possibility of addressing this challenge. To this end, we present V2X-R, the first simulated V2X dataset incorporating LiDAR, camera, and 4D radar. V2X-R contains 12,079 scenarios with 37,727 frames of LiDAR and 4D radar point clouds, 150,908 images, and 170,859 annotated 3D vehicle bounding boxes. Subsequently, we propose a novel cooperative LiDAR-4D radar fusion pipeline for 3D object detection and implement it with various fusion strategies. To achieve weather-robust detection, we additionally propose a Multi-modal Denoising Diffusion (MDD) module in our fusion pipeline. MDD utilizes weather-robust 4D radar feature as a condition to prompt the diffusion model to denoise noisy LiDAR features. Experiments show that our LiDAR-4D radar fusion pipeline demonstrates superior performance in the V2X-R dataset. Over and above this, our MDD module further improved the performance of basic fusion model by up to 5.73%/6.70% in foggy/snowy conditions with barely disrupting normal performance. The dataset and code will be publicly available at: this https URL.

Title: A Heterogeneous Graph Neural Network Fusing Functional and Structural Connectivity for MCI Diagnosis

• Authors: Feiyu Yin, Yu Lei, Siyuan Dai, Wenwen Zeng, Guoqing Wu, Liang Zhan, Jinhua Yu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08424
• Pdf URL: https://arxiv.org/pdf/2411.08424
• Copy Paste: [[2411.08424]] A Heterogeneous Graph Neural Network Fusing Functional and Structural Connectivity for MCI Diagnosis(https://arxiv.org/abs/2411.08424)
• Keywords: diffusion
• Abstract: Brain connectivity alternations associated with brain disorders have been widely reported in resting-state functional imaging (rs-fMRI) and diffusion tensor imaging (DTI). While many dual-modal fusion methods based on graph neural networks (GNNs) have been proposed, they generally follow homogenous fusion ways ignoring rich heterogeneity of dual-modal information. To address this issue, we propose a novel method that integrates functional and structural connectivity based on heterogeneous graph neural networks (HGNNs) to better leverage the rich heterogeneity in dual-modal images. We firstly use blood oxygen level dependency and whiter matter structure information provided by rs-fMRI and DTI to establish homo-meta-path, capturing node relationships within the same modality. At the same time, we propose to establish hetero-meta-path based on structure-function coupling and brain community searching to capture relations among cross-modal nodes. Secondly, we further introduce a heterogeneous graph pooling strategy that automatically balances homo- and hetero-meta-path, effectively leveraging heterogeneous information and preventing feature confusion after pooling. Thirdly, based on the flexibility of heterogeneous graphs, we propose a heterogeneous graph data augmentation approach that can conveniently address the sample imbalance issue commonly seen in clinical diagnosis. We evaluate our method on ADNI-3 dataset for mild cognitive impairment (MCI) diagnosis. Experimental results indicate the proposed method is effective and superior to other algorithms, with a mean classification accuracy of 93.3%.

Title: Can MLLMs Guide Weakly-Supervised Temporal Action Localization Tasks?

• Authors: Quan Zhang, Yuxin Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08466
• Pdf URL: https://arxiv.org/pdf/2411.08466
• Copy Paste: [[2411.08466]] Can MLLMs Guide Weakly-Supervised Temporal Action Localization Tasks?(https://arxiv.org/abs/2411.08466)
• Keywords: foundation model
• Abstract: Recent breakthroughs in Multimodal Large Language Models (MLLMs) have gained significant recognition within the deep learning community, where the fusion of the Video Foundation Models (VFMs) and Large Language Models(LLMs) has proven instrumental in constructing robust video understanding systems, effectively surmounting constraints associated with predefined visual tasks. These sophisticated MLLMs exhibit remarkable proficiency in comprehending videos, swiftly attaining unprecedented performance levels across diverse benchmarks. However, their operation demands substantial memory and computational resources, underscoring the continued importance of traditional models in video comprehension tasks. In this paper, we introduce a novel learning paradigm termed MLLM4WTAL. This paradigm harnesses the potential of MLLM to offer temporal action key semantics and complete semantic priors for conventional Weakly-supervised Temporal Action Localization (WTAL) methods. MLLM4WTAL facilitates the enhancement of WTAL by leveraging MLLM guidance. It achieves this by integrating two distinct modules: Key Semantic Matching (KSM) and Complete Semantic Reconstruction (CSR). These modules work in tandem to effectively address prevalent issues like incomplete and over-complete outcomes common in WTAL methods. Rigorous experiments are conducted to validate the efficacy of our proposed approach in augmenting the performance of various heterogeneous WTAL models.

Title: HyperFace: Generating Synthetic Face Recognition Datasets by Exploring Face Embedding Hypersphere

• Authors: Hatef Otroshi Shahreza, Sébastien Marcel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08470
• Pdf URL: https://arxiv.org/pdf/2411.08470
• Copy Paste: [[2411.08470]] HyperFace: Generating Synthetic Face Recognition Datasets by Exploring Face Embedding Hypersphere(https://arxiv.org/abs/2411.08470)
• Keywords: generative
• Abstract: Face recognition datasets are often collected by crawling Internet and without individuals' consents, raising ethical and privacy concerns. Generating synthetic datasets for training face recognition models has emerged as a promising alternative. However, the generation of synthetic datasets remains challenging as it entails adequate inter-class and intra-class variations. While advances in generative models have made it easier to increase intra-class variations in face datasets (such as pose, illumination, etc.), generating sufficient inter-class variation is still a difficult task. In this paper, we formulate the dataset generation as a packing problem on the embedding space (represented on a hypersphere) of a face recognition model and propose a new synthetic dataset generation approach, called HyperFace. We formalize our packing problem as an optimization problem and solve it with a gradient descent-based approach. Then, we use a conditional face generator model to synthesize face images from the optimized embeddings. We use our generated datasets to train face recognition models and evaluate the trained models on several benchmarking real datasets. Our experimental results show that models trained with HyperFace achieve state-of-the-art performance in training face recognition using synthetic datasets.

Title: A survey on Graph Deep Representation Learning for Facial Expression Recognition

• Authors: Théo Gueuret, Akrem Sellami, Chaabane Djeraba
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08472
• Pdf URL: https://arxiv.org/pdf/2411.08472
• Copy Paste: [[2411.08472]] A survey on Graph Deep Representation Learning for Facial Expression Recognition(https://arxiv.org/abs/2411.08472)
• Keywords: diffusion
• Abstract: This comprehensive review delves deeply into the various methodologies applied to facial expression recognition (FER) through the lens of graph representation learning (GRL). Initially, we introduce the task of FER and the concepts of graph representation and GRL. Afterward, we discuss some of the most prevalent and valuable databases for this task. We explore promising approaches for graph representation in FER, including graph diffusion, spatio-temporal graphs, and multi-stream architectures. Finally, we identify future research opportunities and provide concluding remarks.

Title: Graph Neural Networks in Supply Chain Analytics and Optimization: Concepts, Perspectives, Dataset and Benchmarks

• Authors: Azmine Toushik Wasi, MD Shafikul Islam, Adipto Raihan Akib, Mahathir Mohammad Bappy
• Subjects: cs.LG, cs.CE, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.08550
• Pdf URL: https://arxiv.org/pdf/2411.08550
• Copy Paste: [[2411.08550]] Graph Neural Networks in Supply Chain Analytics and Optimization: Concepts, Perspectives, Dataset and Benchmarks(https://arxiv.org/abs/2411.08550)
• Keywords: anomaly
• Abstract: Graph Neural Networks (GNNs) have recently gained traction in transportation, bioinformatics, language and image processing, but research on their application to supply chain management remains limited. Supply chains are inherently graph-like, making them ideal for GNN methodologies, which can optimize and solve complex problems. The barriers include a lack of proper conceptual foundations, familiarity with graph applications in SCM, and real-world benchmark datasets for GNN-based supply chain research. To address this, we discuss and connect supply chains with graph structures for effective GNN application, providing detailed formulations, examples, mathematical definitions, and task guidelines. Additionally, we present a multi-perspective real-world benchmark dataset from a leading FMCG company in Bangladesh, focusing on supply chain planning. We discuss various supply chain tasks using GNNs and benchmark several state-of-the-art models on homogeneous and heterogeneous graphs across six supply chain analytics tasks. Our analysis shows that GNN-based models consistently outperform statistical Machine Learning and other Deep Learning models by around 10-30% in regression, 10-30% in classification and detection tasks, and 15-40% in anomaly detection tasks on designated metrics. With this work, we lay the groundwork for solving supply chain problems using GNNs, supported by conceptual discussions, methodological insights, and a comprehensive dataset.

Title: Zero-shot capability of SAM-family models for bone segmentation in CT scans

• Authors: Caroline Magg, Hoel Kervadec, Clara I. Sánchez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08629
• Pdf URL: https://arxiv.org/pdf/2411.08629
• Copy Paste: [[2411.08629]] Zero-shot capability of SAM-family models for bone segmentation in CT scans(https://arxiv.org/abs/2411.08629)
• Keywords: foundation model
• Abstract: The Segment Anything Model (SAM) and similar models build a family of promptable foundation models (FMs) for image and video segmentation. The object of interest is identified using prompts, such as bounding boxes or points. With these FMs becoming part of medical image segmentation, extensive evaluation studies are required to assess their strengths and weaknesses in clinical setting. Since the performance is highly dependent on the chosen prompting strategy, it is important to investigate different prompting techniques to define optimal guidelines that ensure effective use in medical image segmentation. Currently, no dedicated evaluation studies exist specifically for bone segmentation in CT scans, leaving a gap in understanding the performance for this task. Thus, we use non-iterative, ``optimal'' prompting strategies composed of bounding box, points and combinations to test the zero-shot capability of SAM-family models for bone CT segmentation on three different skeletal regions. Our results show that the best settings depend on the model type and size, dataset characteristics and objective to optimize. Overall, SAM and SAM2 prompted with a bounding box in combination with the center point for all the components of an object yield the best results across all tested settings. As the results depend on multiple factors, we provide a guideline for informed decision-making in 2D prompting with non-interactive, ''optimal'' prompts.

Title: Towards More Accurate Fake Detection on Images Generated from Advanced Generative and Neural Rendering Models

• Authors: Chengdong Dong, Vijayakumar Bhagavatula, Zhenyu Zhou, Ajay Kumar
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08642
• Pdf URL: https://arxiv.org/pdf/2411.08642
• Copy Paste: [[2411.08642]] Towards More Accurate Fake Detection on Images Generated from Advanced Generative and Neural Rendering Models(https://arxiv.org/abs/2411.08642)
• Keywords: diffusion, generative
• Abstract: The remarkable progress in neural-network-driven visual data generation, especially with neural rendering techniques like Neural Radiance Fields and 3D Gaussian splatting, offers a powerful alternative to GANs and diffusion models. These methods can produce high-fidelity images and lifelike avatars, highlighting the need for robust detection methods. In response, an unsupervised training technique is proposed that enables the model to extract comprehensive features from the Fourier spectrum magnitude, thereby overcoming the challenges of reconstructing the spectrum due to its centrosymmetric properties. By leveraging the spectral domain and dynamically combining it with spatial domain information, we create a robust multimodal detector that demonstrates superior generalization capabilities in identifying challenging synthetic images generated by the latest image synthesis techniques. To address the absence of a 3D neural rendering-based fake image database, we develop a comprehensive database that includes images generated by diverse neural rendering techniques, providing a robust foundation for evaluating and advancing detection methods.

Title: Accelerating Quasi-Static Time Series Simulations with Foundation Models

• Authors: Alban Puech, François Mirallès, Jonas Weiss, Vincent Mai, Alexandre Blondin Massé, Martin de Montigny, Thomas Brunschwiler, Hendrik F. Hamann
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08652
• Pdf URL: https://arxiv.org/pdf/2411.08652
• Copy Paste: [[2411.08652]] Accelerating Quasi-Static Time Series Simulations with Foundation Models(https://arxiv.org/abs/2411.08652)
• Keywords: foundation model
• Abstract: Quasi-static time series (QSTS) simulations have great potential for evaluating the grid's ability to accommodate the large-scale integration of distributed energy resources. However, as grids expand and operate closer to their limits, iterative power flow solvers, central to QSTS simulations, become computationally prohibitive and face increasing convergence issues. Neural power flow solvers provide a promising alternative, speeding up power flow computations by 3 to 4 orders of magnitude, though they are costly to train. In this paper, we envision how recently introduced grid foundation models could improve the economic viability of neural power flow solvers. Conceptually, these models amortize training costs by serving as a foundation for a range of grid operation and planning tasks beyond power flow solving, with only minimal fine-tuning required. We call for collaboration between the AI and power grid communities to develop and open-source these models, enabling all operators, even those with limited resources, to benefit from AI without building solutions from scratch.

Title: MikuDance: Animating Character Art with Mixed Motion Dynamics

• Authors: Jiaxu Zhang, Xianfang Zeng, Xin Chen, Wei Zuo, Gang Yu, Zhigang Tu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08656
• Pdf URL: https://arxiv.org/pdf/2411.08656
• Copy Paste: [[2411.08656]] MikuDance: Animating Character Art with Mixed Motion Dynamics(https://arxiv.org/abs/2411.08656)
• Keywords: diffusion
• Abstract: We propose MikuDance, a diffusion-based pipeline incorporating mixed motion dynamics to animate stylized character art. MikuDance consists of two key techniques: Mixed Motion Modeling and Mixed-Control Diffusion, to address the challenges of high-dynamic motion and reference-guidance misalignment in character art animation. Specifically, a Scene Motion Tracking strategy is presented to explicitly model the dynamic camera in pixel-wise space, enabling unified character-scene motion modeling. Building on this, the Mixed-Control Diffusion implicitly aligns the scale and body shape of diverse characters with motion guidance, allowing flexible control of local character motion. Subsequently, a Motion-Adaptive Normalization module is incorporated to effectively inject global scene motion, paving the way for comprehensive character art animation. Through extensive experiments, we demonstrate the effectiveness and generalizability of MikuDance across various character art and motion guidance, consistently producing high-quality animations with remarkable motion dynamics.

Title: Toward Human Understanding with Controllable Synthesis

• Authors: Hanz Cuevas-Velasquez, Priyanka Patel, Haiwen Feng, Michael Black
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08663
• Pdf URL: https://arxiv.org/pdf/2411.08663
• Copy Paste: [[2411.08663]] Toward Human Understanding with Controllable Synthesis(https://arxiv.org/abs/2411.08663)
• Keywords: generative
• Abstract: Training methods to perform robust 3D human pose and shape (HPS) estimation requires diverse training images with accurate ground truth. While BEDLAM demonstrates the potential of traditional procedural graphics to generate such data, the training images are clearly synthetic. In contrast, generative image models produce highly realistic images but without ground truth. Putting these methods together seems straightforward: use a generative model with the body ground truth as controlling signal. However, we find that, the more realistic the generated images, the more they deviate from the ground truth, making them inappropriate for training and evaluation. Enhancements of realistic details, such as clothing and facial expressions, can lead to subtle yet significant deviations from the ground truth, potentially misleading training models. We empirically verify that this misalignment causes the accuracy of HPS networks to decline when trained with generated images. To address this, we design a controllable synthesis method that effectively balances image realism with precise ground truth. We use this to create the Generative BEDLAM (Gen-B) dataset, which improves the realism of the existing synthetic BEDLAM dataset while preserving ground truth accuracy. We perform extensive experiments, with various noise-conditioning strategies, to evaluate the tradeoff between visual realism and HPS accuracy. We show, for the first time, that generative image models can be controlled by traditional graphics methods to produce training data that increases the accuracy of HPS methods.

Title: Weakly-Supervised Anomaly Detection in Surveillance Videos Based on Two-Stream I3D Convolution Network

• Authors: Sareh Soltani Nejad, Anwar Haque
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08755
• Pdf URL: https://arxiv.org/pdf/2411.08755
• Copy Paste: [[2411.08755]] Weakly-Supervised Anomaly Detection in Surveillance Videos Based on Two-Stream I3D Convolution Network(https://arxiv.org/abs/2411.08755)
• Keywords: anomaly
• Abstract: The widespread implementation of urban surveillance systems has necessitated more sophisticated techniques for anomaly detection to ensure enhanced public safety. This paper presents a significant advancement in the field of anomaly detection through the application of Two-Stream Inflated 3D (I3D) Convolutional Networks. These networks substantially outperform traditional 3D Convolutional Networks (C3D) by more effectively extracting spatial and temporal features from surveillance videos, thus improving the precision of anomaly detection. Our research advances the field by implementing a weakly supervised learning framework based on Multiple Instance Learning (MIL), which uniquely conceptualizes surveillance videos as collections of 'bags' that contain instances (video clips). Each instance is innovatively processed through a ranking mechanism that prioritizes clips based on their potential to display anomalies. This novel strategy not only enhances the accuracy and precision of anomaly detection but also significantly diminishes the dependency on extensive manual annotations. Moreover, through meticulous optimization of model settings, including the choice of optimizer, our approach not only establishes new benchmarks in the performance of anomaly detection systems but also offers a scalable and efficient solution for real-world surveillance applications. This paper contributes significantly to the field of computer vision by delivering a more adaptable, efficient, and context-aware anomaly detection system, which is poised to redefine practices in urban surveillance.

Title: Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation

• Authors: Yangyang Li, Xuanting Hao, Ronghua Shang, Licheng Jiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.08756
• Pdf URL: https://arxiv.org/pdf/2411.08756
• Copy Paste: [[2411.08756]] Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation(https://arxiv.org/abs/2411.08756)
• Keywords: self-supervised, generative
• Abstract: In view of the fact that semi- and self-supervised learning share a fundamental principle, effectively modeling knowledge from unlabeled data, various semi-supervised semantic segmentation methods have integrated representative self-supervised learning paradigms for further regularization. However, the potential of the state-of-the-art generative self-supervised paradigm, masked image modeling, has been scarcely studied. This paradigm learns the knowledge through establishing connections between the masked and visible parts of masked image, during the pixel reconstruction process. By inheriting and extending this insight, we successfully leverage masked image modeling to boost semi-supervised semantic segmentation. Specifically, we introduce a novel class-wise masked image modeling that independently reconstructs different image regions according to their respective classes. In this way, the mask-induced connections are established within each class, mitigating the semantic confusion that arises from plainly reconstructing images in basic masked image modeling. To strengthen these intra-class connections, we further develop a feature aggregation strategy that minimizes the distances between features corresponding to the masked and visible parts within the same class. Additionally, in semantic space, we explore the application of masked image modeling to enhance regularization. Extensive experiments conducted on well-known benchmarks demonstrate that our approach achieves state-of-the-art performance. The code will be available at this https URL.

Title: The Limited Impact of Medical Adaptation of Large Language and Vision-Language Models

• Authors: Daniel P. Jeong, Pranav Mani, Saurabh Garg, Zachary C. Lipton, Michael Oberst
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08870
• Pdf URL: https://arxiv.org/pdf/2411.08870
• Copy Paste: [[2411.08870]] The Limited Impact of Medical Adaptation of Large Language and Vision-Language Models(https://arxiv.org/abs/2411.08870)
• Keywords: foundation model
• Abstract: Several recent works seek to develop foundation models specifically for medical applications, adapting general-purpose large language models (LLMs) and vision-language models (VLMs) via continued pretraining on publicly available biomedical corpora. These works typically claim that such domain-adaptive pretraining (DAPT) improves performance on downstream medical tasks, such as answering medical licensing exam questions. In this paper, we compare ten public "medical" LLMs and two VLMs against their corresponding base models, arriving at a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models in the zero-/few-shot prompting and supervised fine-tuning regimes for medical question-answering (QA). For instance, across all tasks and model pairs we consider in the 3-shot setting, medical LLMs only outperform their base models in 22.7% of cases, reach a (statistical) tie in 36.8% of cases, and are significantly worse than their base models in the remaining 40.5% of cases. Our conclusions are based on (i) comparing each medical model head-to-head, directly against the corresponding base model; (ii) optimizing the prompts for each model separately in zero-/few-shot prompting; and (iii) accounting for statistical uncertainty in comparisons. While these basic practices are not consistently adopted in the literature, our ablations show that they substantially impact conclusions. Meanwhile, we find that after fine-tuning on specific QA tasks, medical LLMs can show performance improvements, but the benefits do not carry over to tasks based on clinical notes. Our findings suggest that state-of-the-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities, and offer recommendations to strengthen the conclusions of future studies.

Title: 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization

• Authors: Mijeong Kim, Jongwoo Lim, Bohyung Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08879
• Pdf URL: https://arxiv.org/pdf/2411.08879
• Copy Paste: [[2411.08879]] 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization(https://arxiv.org/abs/2411.08879)
• Keywords: diffusion
• Abstract: Novel view synthesis of dynamic scenes is becoming important in various applications, including augmented and virtual reality. We propose a novel 4D Gaussian Splatting (4DGS) algorithm for dynamic scenes from casually recorded monocular videos. To overcome the overfitting problem of existing work for these real-world videos, we introduce an uncertainty-aware regularization that identifies uncertain regions with few observations and selectively imposes additional priors based on diffusion models and depth smoothness on such regions. This approach improves both the performance of novel view synthesis and the quality of training image reconstruction. We also identify the initialization problem of 4DGS in fast-moving dynamic regions, where the Structure from Motion (SfM) algorithm fails to provide reliable 3D landmarks. To initialize Gaussian primitives in such regions, we present a dynamic region densification method using the estimated depth maps and scene flow. Our experiments show that the proposed method improves the performance of 4DGS reconstruction from a video captured by a handheld monocular camera and also exhibits promising results in few-shot static scene reconstruction.