2025-01-14

Title: Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models

• Authors: Xin Ding, Shijie Cao, Ting Cao, Zhibo Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06218
• Pdf URL: https://arxiv.org/pdf/2501.06218
• Copy Paste: [[2501.06218]] Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models(https://arxiv.org/abs/2501.06218)
• Keywords: diffusion, generative
• Abstract: Vision generative models have recently made significant advancements along two primary paradigms: diffusion-style and language-style, both of which have demonstrated excellent scaling laws. Quantization is crucial for efficiently deploying these models, as it reduces memory and computation costs. In this work, we systematically investigate the impact of quantization on these two paradigms. Surprisingly, despite achieving comparable performance in full precision, language-style models consistently outperform diffusion-style models across various quantization settings. This observation suggests that language-style models have superior bit-level scaling laws, offering a better tradeoff between model quality and total bits. To dissect this phenomenon, we conduct extensive experiments and find that the primary reason is the discrete representation space of language-style models, which is more tolerant of information loss during quantization. Furthermore, our analysis indicates that improving the bit-level scaling law of quantized vision generative models is challenging, with model distillation identified as a highly effective approach. Specifically, we propose TopKLD to optimize the transfer of distilled knowledge by balancing ``implicit knowledge'' and ``explicit knowledge'' during the distillation process. This approach elevates the bit-level scaling laws by one level across both integer and floating-point quantization settings.

Title: Generating and Detecting Various Types of Fake Image and Audio Content: A Review of Modern Deep Learning Technologies and Tools

• Authors: Arash Dehghani, Hossein Saberi
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06227
• Pdf URL: https://arxiv.org/pdf/2501.06227
• Copy Paste: [[2501.06227]] Generating and Detecting Various Types of Fake Image and Audio Content: A Review of Modern Deep Learning Technologies and Tools(https://arxiv.org/abs/2501.06227)
• Keywords: diffusion, generative
• Abstract: This paper reviews the state-of-the-art in deepfake generation and detection, focusing on modern deep learning technologies and tools based on the latest scientific advancements. The rise of deepfakes, leveraging techniques like Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Diffusion models and other generative models, presents significant threats to privacy, security, and democracy. This fake media can deceive individuals, discredit real people and organizations, facilitate blackmail, and even threaten the integrity of legal, political, and social systems. Therefore, finding appropriate solutions to counter the potential threats posed by this technology is essential. We explore various deepfake methods, including face swapping, voice conversion, reenactment and lip synchronization, highlighting their applications in both benign and malicious contexts. The review critically examines the ongoing "arms race" between deepfake generation and detection, analyzing the challenges in identifying manipulated contents. By examining current methods and highlighting future research directions, this paper contributes to a crucial understanding of this rapidly evolving field and the urgent need for robust detection strategies to counter the misuse of this powerful technology. While focusing primarily on audio, image, and video domains, this study allows the reader to easily grasp the latest advancements in deepfake generation and detection.

Title: Mechanics and Design of Metastructured Auxetic Patches with Bio-inspired Materials

• Authors: Yingbin Chen, Milad Arzani, Xuan Mu, Sophia Jin, Shaoping Xiao
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2501.06233
• Pdf URL: https://arxiv.org/pdf/2501.06233
• Copy Paste: [[2501.06233]] Mechanics and Design of Metastructured Auxetic Patches with Bio-inspired Materials(https://arxiv.org/abs/2501.06233)
• Keywords: generative
• Abstract: Metastructured auxetic patches, characterized by negative Poisson's ratios, offer unique mechanical properties that closely resemble the behavior of human tissues and organs. As a result, these patches have gained significant attention for their potential applications in organ repair and tissue regeneration. This study focuses on neural networks-based computational modeling of auxetic patches with a sinusoidal metastructure fabricated from silk fibroin, a bio-inspired material known for its biocompatibility and strength. The primary objective of this research is to introduce a novel, data-driven framework for patch design. To achieve this, we conducted experimental fabrication and mechanical testing to determine material properties and validate the corresponding finite element models. Finite element simulations were then employed to generate the necessary data, while greedy sampling, an active learning technique, was utilized to reduce the computational cost associated with data labeling. Two neural networks were trained to accurately predict Poisson's ratios and stresses for strains up to 15\%, respectively. Both models achieved $R^2$ scores exceeding 0.995, which indicates highly reliable predictions. Building on this, we developed a neural network-based design model capable of tailoring patch designs to achieve specific mechanical properties. This model demonstrated superior performance when compared to traditional optimization methods, such as genetic algorithms, by providing more efficient and precise design solutions. The proposed framework represents a significant advancement in the design of bio-inspired metastructures for medical applications, paving the way for future innovations in tissue engineering and regenerative medicine.

Title: Data-Driven Radio Propagation Modeling using Graph Neural Networks

• Authors: Adrien Bufort, Laurent Lebocq, Stefan Cathabard
• Subjects: cs.LG, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.06236
• Pdf URL: https://arxiv.org/pdf/2501.06236
• Copy Paste: [[2501.06236]] Data-Driven Radio Propagation Modeling using Graph Neural Networks(https://arxiv.org/abs/2501.06236)
• Keywords: generative
• Abstract: Modeling radio propagation is essential for wireless network design and performance optimization. Traditional methods rely on physics models of radio propagation, which can be inaccurate or inflexible. In this work, we propose using graph neural networks to learn radio propagation behaviors directly from real-world network data. Our approach converts the radio propagation environment into a graph representation, with nodes corresponding to locations and edges representing spatial and ray-tracing relationships between locations. The graph is generated by converting images of the environment into a graph structure, with specific relationships between nodes. The model is trained on this graph representation, using sensor measurements as target data. We demonstrate that the graph neural network, which learns to predict radio propagation directly from data, achieves competitive performance compared to traditional heuristic models. This data-driven approach outperforms classic numerical solvers in terms of both speed and accuracy. To the best of our knowledge, we are the first to apply graph neural networks to real-world radio propagation data to generate coverage maps, enabling generative models of signal propagation with point measurements only.

Title: Scalable Cosmic AI Inference using Cloud Serverless Computing with FMI

• Authors: Mills Staylor, Amirreza Dolatpour Fathkouhi, Md Khairul Islam, Kaleigh O'Hara, Ryan Ghiles Goudjil, Geoffrey Fox, Judy Fox
• Subjects: cs.CV, astro-ph.IM
• Abstract URL: https://arxiv.org/abs/2501.06249
• Pdf URL: https://arxiv.org/pdf/2501.06249
• Copy Paste: [[2501.06249]] Scalable Cosmic AI Inference using Cloud Serverless Computing with FMI(https://arxiv.org/abs/2501.06249)
• Keywords: foundation model
• Abstract: Large-scale astronomical image data processing and prediction is essential for astronomers, providing crucial insights into celestial objects, the universe's history, and its evolution. While modern deep learning models offer high predictive accuracy, they often demand substantial computational resources, making them resource-intensive and limiting accessibility. We introduce the Cloud-based Astronomy Inference (CAI) framework to address these challenges. This scalable solution integrates pre-trained foundation models with serverless cloud infrastructure through a Function-as-a-Service (FaaS) Message Interface (FMI). CAI enables efficient and scalable inference on astronomical images without extensive hardware. Using a foundation model for redshift prediction as a case study, our extensive experiments cover user devices, HPC (High-Performance Computing) servers, and Cloud. CAI's significant scalability improvement on large data sizes provides an accessible and effective tool for the astronomy community. The code is accessible at this https URL.

Title: Generative AI for Cel-Animation: A Survey

• Authors: Yunlong Tang, Junjia Guo, Pinxin Liu, Zhiyuan Wang, Hang Hua, Jia-Xing Zhong, Yunzhong Xiao, Chao Huang, Luchuan Song, Susan Liang, Yizhi Song, Liu He, Jing Bi, Mingqian Feng, Xinyang Li, Zeliang Zhang, Chenliang Xu
• Subjects: cs.CV, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.06250
• Pdf URL: https://arxiv.org/pdf/2501.06250
• Copy Paste: [[2501.06250]] Generative AI for Cel-Animation: A Survey(https://arxiv.org/abs/2501.06250)
• Keywords: diffusion, generative
• Abstract: Traditional Celluloid (Cel) Animation production pipeline encompasses multiple essential steps, including storyboarding, layout design, keyframe animation, inbetweening, and colorization, which demand substantial manual effort, technical expertise, and significant time investment. These challenges have historically impeded the efficiency and scalability of Cel-Animation production. The rise of generative artificial intelligence (GenAI), encompassing large language models, multimodal models, and diffusion models, offers innovative solutions by automating tasks such as inbetween frame generation, colorization, and storyboard creation. This survey explores how GenAI integration is revolutionizing traditional animation workflows by lowering technical barriers, broadening accessibility for a wider range of creators through tools like AniDoc, ToonCrafter, and AniSora, and enabling artists to focus more on creative expression and artistic innovation. Despite its potential, issues such as maintaining visual consistency, ensuring stylistic coherence, and addressing ethical considerations continue to pose challenges. Furthermore, this paper discusses future directions and explores potential advancements in AI-assisted animation. For further exploration and resources, please visit our GitHub repository: this https URL

Title: What Matters for In-Context Learning: A Balancing Act of Look-up and In-Weight Learning

• Authors: Jelena Bratulić, Sudhanshu Mittal, Christian Rupprecht, Thomas Brox
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06256
• Pdf URL: https://arxiv.org/pdf/2501.06256
• Copy Paste: [[2501.06256]] What Matters for In-Context Learning: A Balancing Act of Look-up and In-Weight Learning(https://arxiv.org/abs/2501.06256)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have demonstrated impressive performance in various tasks, including In-Context Learning (ICL), where the model performs new tasks by conditioning solely on the examples provided in the context, without updating the model's weights. While prior research has explored the roles of pretraining data and model architecture, the key mechanism behind ICL remains unclear. In this work, we systematically uncover properties present in LLMs that support the emergence of ICL. To disambiguate these factors, we conduct a study with a controlled dataset and data sequences using a deep autoregressive model. We show that conceptual repetitions in the data sequences are crucial for ICL, more so than previously indicated training data properties like burstiness or long-tail distribution. Conceptual repetitions could refer to $n$-gram repetitions in textual data or exact image copies in image sequence data. Such repetitions also offer other previously overlooked benefits such as reduced transiency in ICL performance. Furthermore, we show that the emergence of ICL depends on balancing the in-weight learning objective with the in-context solving ability during training.

Title: Quantum Down Sampling Filter for Variational Auto-encoder

• Authors: Farina Riaz, Fakhar Zaman, Hajime Suzuki, Sharif Abuadbba, David Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06259
• Pdf URL: https://arxiv.org/pdf/2501.06259
• Copy Paste: [[2501.06259]] Quantum Down Sampling Filter for Variational Auto-encoder(https://arxiv.org/abs/2501.06259)
• Keywords: generative
• Abstract: Variational Autoencoders (VAEs) are essential tools in generative modeling and image reconstruction, with their performance heavily influenced by the encoder-decoder architecture. This study aims to improve the quality of reconstructed images by enhancing their resolution and preserving finer details, particularly when working with low-resolution inputs (16x16 pixels), where traditional VAEs often yield blurred or in-accurate results. To address this, we propose a hybrid model that combines quantum computing techniques in the VAE encoder with convolutional neural networks (CNNs) in the decoder. By upscaling the resolution from 16x16 to 32x32 during the encoding process, our approach evaluates how the model reconstructs images with enhanced resolution while maintaining key features and structures. This method tests the model's robustness in handling image reconstruction and its ability to preserve essential details despite training on lower-resolution data. We evaluate our proposed down sampling filter for Quantum VAE (Q-VAE) on the MNIST and USPS datasets and compare it with classical VAEs and a variant called Classical Direct Passing VAE (CDP-VAE), which uses windowing pooling filters in the encoding process. Performance is assessed using metrics such as the Fréchet Inception Distance (FID) and Mean Squared Error (MSE), which measure the fidelity of reconstructed images. Our results demonstrate that the Q-VAE consistently outperforms both the Classical VAE and CDP-VAE, achieving significantly lower FID and MSE scores. Additionally, CDP-VAE yields better performance than C-VAE. These findings highlight the potential of quantum-enhanced VAEs to improve image reconstruction quality by enhancing resolution and preserving essential features, offering a promising direction for future applications in computer vision and synthetic data generation.

Title: Environmental large language model Evaluation (ELLE) dataset: A Benchmark for Evaluating Generative AI applications in Eco-environment Domain

• Authors: Jing Guo, Nan Li, Ming Xu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.06277
• Pdf URL: https://arxiv.org/pdf/2501.06277
• Copy Paste: [[2501.06277]] Environmental large language model Evaluation (ELLE) dataset: A Benchmark for Evaluating Generative AI applications in Eco-environment Domain(https://arxiv.org/abs/2501.06277)
• Keywords: generative
• Abstract: Generative AI holds significant potential for ecological and environmental applications such as monitoring, data analysis, education, and policy support. However, its effectiveness is limited by the lack of a unified evaluation framework. To address this, we present the Environmental Large Language model Evaluation (ELLE) question answer (QA) dataset, the first benchmark designed to assess large language models and their applications in ecological and environmental sciences. The ELLE dataset includes 1,130 question answer pairs across 16 environmental topics, categorized by domain, difficulty, and type. This comprehensive dataset standardizes performance assessments in these fields, enabling consistent and objective comparisons of generative AI performance. By providing a dedicated evaluation tool, ELLE dataset promotes the development and application of generative AI technologies for sustainable environmental outcomes. The dataset and code are available at this https URL and this https URL.

Title: Towards Iris Presentation Attack Detection with Foundation Models

• Authors: Juan E. Tapia, Lázaro Janier González-Soler, Christoph Busch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06312
• Pdf URL: https://arxiv.org/pdf/2501.06312
• Copy Paste: [[2501.06312]] Towards Iris Presentation Attack Detection with Foundation Models(https://arxiv.org/abs/2501.06312)
• Keywords: foundation model
• Abstract: Foundation models are becoming increasingly popular due to their strong generalization capabilities resulting from being trained on huge datasets. These generalization capabilities are attractive in areas such as NIR Iris Presentation Attack Detection (PAD), in which databases are limited in the number of subjects and diversity of attack instruments, and there is no correspondence between the bona fide and attack images because, most of the time, they do not belong to the same subjects. This work explores an iris PAD approach based on two foundation models, DinoV2 and VisualOpenClip. The results show that fine-tuning prediction with a small neural network as head overpasses the state-of-the-art performance based on deep learning approaches. However, systems trained from scratch have still reached better results if bona fide and attack images are available.

Title: MEt3R: Measuring Multi-View Consistency in Generated Images

• Authors: Mohammad Asim, Christopher Wewer, Thomas Wimmer, Bernt Schiele, Jan Eric Lenssen
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.06336
• Pdf URL: https://arxiv.org/pdf/2501.06336
• Copy Paste: [[2501.06336]] MEt3R: Measuring Multi-View Consistency in Generated Images(https://arxiv.org/abs/2501.06336)
• Keywords: diffusion, generative
• Abstract: We introduce MEt3R, a metric for multi-view consistency in generated images. Large-scale generative models for multi-view image generation are rapidly advancing the field of 3D inference from sparse observations. However, due to the nature of generative modeling, traditional reconstruction metrics are not suitable to measure the quality of generated outputs and metrics that are independent of the sampling procedure are desperately needed. In this work, we specifically address the aspect of consistency between generated multi-view images, which can be evaluated independently of the specific scene. Our approach uses DUSt3R to obtain dense 3D reconstructions from image pairs in a feed-forward manner, which are used to warp image contents from one view into the other. Then, feature maps of these images are compared to obtain a similarity score that is invariant to view-dependent effects. Using MEt3R, we evaluate the consistency of a large set of previous methods for novel view and video generation, including our open, multi-view latent diffusion model.

Title: Has an AI model been trained on your images?

• Authors: Matyas Bohacek, Hany Farid
• Subjects: cs.CV, cs.AI, cs.CR, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06399
• Pdf URL: https://arxiv.org/pdf/2501.06399
• Copy Paste: [[2501.06399]] Has an AI model been trained on your images?(https://arxiv.org/abs/2501.06399)
• Keywords: generative
• Abstract: From a simple text prompt, generative-AI image models can create stunningly realistic and creative images bounded, it seems, by only our imagination. These models have achieved this remarkable feat thanks, in part, to the ingestion of billions of images collected from nearly every corner of the internet. Many creators have understandably expressed concern over how their intellectual property has been ingested without their permission or a mechanism to opt out of training. As a result, questions of fair use and copyright infringement have quickly emerged. We describe a method that allows us to determine if a model was trained on a specific image or set of images. This method is computationally efficient and assumes no explicit knowledge of the model architecture or weights (so-called black-box membership inference). We anticipate that this method will be crucial for auditing existing models and, looking ahead, ensuring the fairer development and deployment of generative AI models.

Title: Mathematics of Digital Twins and Transfer Learning for PDE Models

• Authors: Yifei Zong, Alexandre Tartakovsky
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.06400
• Pdf URL: https://arxiv.org/pdf/2501.06400
• Copy Paste: [[2501.06400]] Mathematics of Digital Twins and Transfer Learning for PDE Models(https://arxiv.org/abs/2501.06400)
• Keywords: diffusion
• Abstract: We define a digital twin (DT) of a physical system governed by partial differential equations (PDEs) as a model for real-time simulations and control of the system behavior under changing conditions. We construct DTs using the Karhunen-Loève Neural Network (KL-NN) surrogate model and transfer learning (TL). The surrogate model allows fast inference and differentiability with respect to control parameters for control and optimization. TL is used to retrain the model for new conditions with minimal additional data. We employ the moment equations to analyze TL and identify parameters that can be transferred to new conditions. The proposed analysis also guides the control variable selection in DT to facilitate efficient TL. For linear PDE problems, the non-transferable parameters in the KL-NN surrogate model can be exactly estimated from a single solution of the PDE corresponding to the mean values of the control variables under new target conditions. Retraining an ML model with a single solution sample is known as one-shot learning, and our analysis shows that the one-shot TL is exact for linear PDEs. For nonlinear PDE problems, transferring of any parameters introduces errors. For a nonlinear diffusion PDE model, we find that for a relatively small range of control variables, some surrogate model parameters can be transferred without introducing a significant error, some can be approximately estimated from the mean-field equation, and the rest can be found using a linear residual least square problem or an ordinary linear least square problem if a small labeled dataset for new conditions is available. The former approach results in a one-shot TL while the latter approach is an example of a few-shot TL. Both methods are approximate for the nonlinear PDEs.

Title: Qffusion: Controllable Portrait Video Editing via Quadrant-Grid Attention Learning

• Authors: Maomao Li, Lijian Lin, Yunfei Liu, Ye Zhu, Yu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06438
• Pdf URL: https://arxiv.org/pdf/2501.06438
• Copy Paste: [[2501.06438]] Qffusion: Controllable Portrait Video Editing via Quadrant-Grid Attention Learning(https://arxiv.org/abs/2501.06438)
• Keywords: diffusion, generative
• Abstract: This paper presents Qffusion, a dual-frame-guided framework for portrait video editing. Specifically, we consider a design principle of ``animation for editing'', and train Qffusion as a general animation framework from two still reference images while we can use it for portrait video editing easily by applying modified start and end frames as references during inference. Leveraging the powerful generative power of Stable Diffusion, we propose a Quadrant-grid Arrangement (QGA) scheme for latent re-arrangement, which arranges the latent codes of two reference images and that of four facial conditions into a four-grid fashion, separately. Then, we fuse features of these two modalities and use self-attention for both appearance and temporal learning, where representations at different times are jointly modeled under QGA. Our Qffusion can achieve stable video editing without additional networks or complex training stages, where only the input format of Stable Diffusion is modified. Further, we propose a Quadrant-grid Propagation (QGP) inference strategy, which enjoys a unique advantage on stable arbitrary-length video generation by processing reference and condition frames recursively. Through extensive experiments, Qffusion consistently outperforms state-of-the-art techniques on portrait video editing.

Title: MedCT: A Clinical Terminology Graph for Generative AI Applications in Healthcare

• Authors: Ye Chen, Dongdong Huang, Haoyun Xu, Cong Fu, Lin Sheng, Qingli Zhou, Yuqiang Shen, Kai Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06465
• Pdf URL: https://arxiv.org/pdf/2501.06465
• Copy Paste: [[2501.06465]] MedCT: A Clinical Terminology Graph for Generative AI Applications in Healthcare(https://arxiv.org/abs/2501.06465)
• Keywords: foundation model, generative
• Abstract: We introduce the world's first clinical terminology for the Chinese healthcare community, namely MedCT, accompanied by a clinical foundation model MedBERT and an entity linking model MedLink. The MedCT system enables standardized and programmable representation of Chinese clinical data, successively stimulating the development of new medicines, treatment pathways, and better patient outcomes for the populous Chinese community. Moreover, the MedCT knowledge graph provides a principled mechanism to minimize the hallucination problem of large language models (LLMs), therefore achieving significant levels of accuracy and safety in LLM-based clinical applications. By leveraging the LLMs' emergent capabilities of generativeness and expressiveness, we were able to rapidly built a production-quality terminology system and deployed to real-world clinical field within three months, while classical terminologies like SNOMED CT have gone through more than twenty years development. Our experiments show that the MedCT system achieves state-of-the-art (SOTA) performance in semantic matching and entity linking tasks, not only for Chinese but also for English. We also conducted a longitudinal field experiment by applying MedCT and LLMs in a representative spectrum of clinical tasks, including electronic health record (EHR) auto-generation and medical document search for diagnostic decision making. Our study shows a multitude of values of MedCT for clinical workflows and patient outcomes, especially in the new genre of clinical LLM applications. We present our approach in sufficient engineering detail, such that implementing a clinical terminology for other non-English societies should be readily reproducible. We openly release our terminology, models and algorithms, along with real-world clinical datasets for the development.

Title: NVS-SQA: Exploring Self-Supervised Quality Representation Learning for Neurally Synthesized Scenes without References

• Authors: Qiang Qu, Yiran Shen, Xiaoming Chen, Yuk Ying Chung, Weidong Cai, Tongliang Liu
• Subjects: cs.CV, cs.AI, cs.HC, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.06488
• Pdf URL: https://arxiv.org/pdf/2501.06488
• Copy Paste: [[2501.06488]] NVS-SQA: Exploring Self-Supervised Quality Representation Learning for Neurally Synthesized Scenes without References(https://arxiv.org/abs/2501.06488)
• Keywords: self-supervised
• Abstract: Neural View Synthesis (NVS), such as NeRF and 3D Gaussian Splatting, effectively creates photorealistic scenes from sparse viewpoints, typically evaluated by quality assessment methods like PSNR, SSIM, and LPIPS. However, these full-reference methods, which compare synthesized views to reference views, may not fully capture the perceptual quality of neurally synthesized scenes (NSS), particularly due to the limited availability of dense reference views. Furthermore, the challenges in acquiring human perceptual labels hinder the creation of extensive labeled datasets, risking model overfitting and reduced generalizability. To address these issues, we propose NVS-SQA, a NSS quality assessment method to learn no-reference quality representations through self-supervision without reliance on human labels. Traditional self-supervised learning predominantly relies on the "same instance, similar representation" assumption and extensive datasets. However, given that these conditions do not apply in NSS quality assessment, we employ heuristic cues and quality scores as learning objectives, along with a specialized contrastive pair preparation process to improve the effectiveness and efficiency of learning. The results show that NVS-SQA outperforms 17 no-reference methods by a large margin (i.e., on average 109.5% in SRCC, 98.6% in PLCC, and 91.5% in KRCC over the second best) and even exceeds 16 full-reference methods across all evaluation metrics (i.e., 22.9% in SRCC, 19.1% in PLCC, and 18.6% in KRCC over the second best).

Title: Active Rule Mining for Multivariate Anomaly Detection in Radio Access Networks

• Authors: Ebenezer R. H. P. Isaac, Joseph H. R. Isaac
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06571
• Pdf URL: https://arxiv.org/pdf/2501.06571
• Copy Paste: [[2501.06571]] Active Rule Mining for Multivariate Anomaly Detection in Radio Access Networks(https://arxiv.org/abs/2501.06571)
• Keywords: anomaly
• Abstract: Multivariate anomaly detection finds its importance in diverse applications. Despite the existence of many detectors to solve this problem, one cannot simply define why an obtained anomaly inferred by the detector is anomalous. This reasoning is required for network operators to understand the root cause of the anomaly and the remedial action that should be taken to counteract its occurrence. Existing solutions in explainable AI may give cues to features that influence an anomaly, but they do not formulate generalizable rules that can be assessed by a domain expert. Furthermore, not all outliers are anomalous in a business sense. There is an unfulfilled need for a system that can interpret anomalies predicted by a multivariate anomaly detector and map these patterns to actionable rules. This paper aims to fulfill this need by proposing a semi-autonomous anomaly rule miner. The proposed method is applicable to both discrete and time series data and is tailored for radio access network (RAN) anomaly detection use cases. The proposed method is demonstrated in this paper with time series RAN data.

Title: Boundary-enhanced time series data imputation with long-term dependency diffusion models

• Authors: Chunjing Xiao, Xue Jiang, Xianghe Du, Wei Yang, Wei Lu, Xiaomin Wang, Kevin Chetty
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.06585
• Pdf URL: https://arxiv.org/pdf/2501.06585
• Copy Paste: [[2501.06585]] Boundary-enhanced time series data imputation with long-term dependency diffusion models(https://arxiv.org/abs/2501.06585)
• Keywords: diffusion
• Abstract: Data imputation is crucial for addressing challenges posed by missing values in multivariate time series data across various fields, such as healthcare, traffic, and economics, and has garnered significant attention. Among various methods, diffusion model-based approaches show notable performance improvements. However, existing methods often cause disharmonious boundaries between missing and known regions and overlook long-range dependencies in missing data estimation, leading to suboptimal results. To address these issues, we propose a Diffusion-based time Series Data Imputation (DSDI) framework. We develop a weight-reducing injection strategy that incorporates the predicted values of missing points with reducing weights into the reverse diffusion process to mitigate boundary inconsistencies. Further, we introduce a multi-scale S4-based U-Net, which combines hierarchical information from different levels via multi-resolution integration to capture long-term dependencies. Experimental results demonstrate that our model outperforms existing imputation methods.

Title: Exploring Pose-Based Anomaly Detection for Retail Security: A Real-World Shoplifting Dataset and Benchmark

• Authors: Narges Rashvand, Ghazal Alinezhad Noghre, Armin Danesh Pazho, Shanle Yao, Hamed Tabkhi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06591
• Pdf URL: https://arxiv.org/pdf/2501.06591
• Copy Paste: [[2501.06591]] Exploring Pose-Based Anomaly Detection for Retail Security: A Real-World Shoplifting Dataset and Benchmark(https://arxiv.org/abs/2501.06591)
• Keywords: anomaly
• Abstract: Shoplifting poses a significant challenge for retailers, resulting in billions of dollars in annual losses. Traditional security measures often fall short, highlighting the need for intelligent solutions capable of detecting shoplifting behaviors in real time. This paper frames shoplifting detection as an anomaly detection problem, focusing on the identification of deviations from typical shopping patterns. We introduce PoseLift, a privacy-preserving dataset specifically designed for shoplifting detection, addressing challenges such as data scarcity, privacy concerns, and model biases. PoseLift is built in collaboration with a retail store and contains anonymized human pose data from real-world scenarios. By preserving essential behavioral information while anonymizing identities, PoseLift balances privacy and utility. We benchmark state-of-the-art pose-based anomaly detection models on this dataset, evaluating performance using a comprehensive set of metrics. Our results demonstrate that pose-based approaches achieve high detection accuracy while effectively addressing privacy and bias concerns inherent in traditional methods. As one of the first datasets capturing real-world shoplifting behaviors, PoseLift offers researchers a valuable tool to advance computer vision ethically and will be publicly available to foster innovation and collaboration. The dataset is available at this https URL.

Title: EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses

• Authors: Shireesh Reddy Pyreddy, Tarannum Shaila Zaman
• Subjects: cs.LG, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.06597
• Pdf URL: https://arxiv.org/pdf/2501.06597
• Copy Paste: [[2501.06597]] EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses(https://arxiv.org/abs/2501.06597)
• Keywords: generative
• Abstract: The widespread adoption of generative AI has generated diverse opinions, with individuals expressing both support and criticism of its applications. This study investigates the emotional dynamics surrounding generative AI by analyzing human tweets referencing terms such as ChatGPT, OpenAI, Copilot, and LLMs. To further understand the emotional intelligence of ChatGPT, we examine its responses to selected tweets, highlighting differences in sentiment between human comments and LLM-generated responses. We introduce EmoXpt, a sentiment analysis framework designed to assess both human perspectives on generative AI and the sentiment embedded in ChatGPT's responses. Unlike prior studies that focus exclusively on human sentiment, EmoXpt uniquely evaluates the emotional expression of ChatGPT. Experimental results demonstrate that LLM-generated responses are notably more efficient, cohesive, and consistently positive than human responses.

Title: Personalized Preference Fine-tuning of Diffusion Models

• Authors: Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, Jiaming Song
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06655
• Pdf URL: https://arxiv.org/pdf/2501.06655
• Copy Paste: [[2501.06655]] Personalized Preference Fine-tuning of Diffusion Models(https://arxiv.org/abs/2501.06655)
• Keywords: diffusion
• Abstract: RLHF techniques like DPO can significantly improve the generation quality of text-to-image diffusion models. However, these methods optimize for a single reward that aligns model generation with population-level preferences, neglecting the nuances of individual users' beliefs or values. This lack of personalization limits the efficacy of these models. To bridge this gap, we introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences. With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way, enabling generalization to unseen users. Specifically, our approach (1) leverages a vision-language model (VLM) to extract personal preference embeddings from a small set of pairwise preference examples, and then (2) incorporates the embeddings into diffusion models through cross attention. Conditioning on user embeddings, the text-to-image models are fine-tuned with the DPO objective, simultaneously optimizing for alignment with the preferences of multiple users. Empirical results demonstrate that our method effectively optimizes for multiple reward functions and can interpolate between them during inference. In real-world user scenarios, with as few as four preference examples from a new user, our approach achieves an average win rate of 76\% over Stable Cascade, generating images that more accurately reflect specific user preferences.

Title: Tab-Shapley: Identifying Top-k Tabular Data Quality Insights

• Authors: Manisha Padala, Lokesh Nagalapatti, Atharv Tyagi, Ramasuri Narayanam, Shiv Kumar Saini
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.06685
• Pdf URL: https://arxiv.org/pdf/2501.06685
• Copy Paste: [[2501.06685]] Tab-Shapley: Identifying Top-k Tabular Data Quality Insights(https://arxiv.org/abs/2501.06685)
• Keywords: anomaly
• Abstract: We present an unsupervised method for aggregating anomalies in tabular datasets by identifying the top-k tabular data quality insights. Each insight consists of a set of anomalous attributes and the corresponding subsets of records that serve as evidence to the user. The process of identifying these insight blocks is challenging due to (i) the absence of labeled anomalies, (ii) the exponential size of the subset search space, and (iii) the complex dependencies among attributes, which obscure the true sources of anomalies. Simple frequency-based methods fail to capture these dependencies, leading to inaccurate results. To address this, we introduce Tab-Shapley, a cooperative game theory based framework that uses Shapley values to quantify the contribution of each attribute to the data's anomalous nature. While calculating Shapley values typically requires exponential time, we show that our game admits a closed-form solution, making the computation efficient. We validate the effectiveness of our approach through empirical analysis on real-world tabular datasets with ground-truth anomaly labels.

Title: Evaluating Sample Utility for Data Selection by Mimicking Model Weights

• Authors: Tzu-Heng Huang, Manjot Bilkhu, Frederic Sala, Javier Movellan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06708
• Pdf URL: https://arxiv.org/pdf/2501.06708
• Copy Paste: [[2501.06708]] Evaluating Sample Utility for Data Selection by Mimicking Model Weights(https://arxiv.org/abs/2501.06708)
• Keywords: foundation model
• Abstract: Foundation models rely on large-scale web-crawled datasets, which frequently contain noisy data, biases, and irrelevant content. Existing data selection techniques typically use human heuristics, downstream evaluation datasets, or specialized scoring models, and can overlook samples' utility in the training process. Instead, we propose a new approach, Mimic Score, a data quality metric that uses a pretrained reference model as a guide to assess the usefulness of data samples for training a new model. It relies on the alignment between the gradient of the new model parameters and the vector pointing toward the reference model in weight space. Samples that misalign with this direction are considered low-value and can be filtered out. Motivated by the Mimic score, we develop Grad-Mimic, a data selection framework that identifies and prioritizes useful samples, automating the selection process to create effective filters. Empirically, using Mimic scores to guide model training results in consistent performance gains across six image datasets and enhances the performance of CLIP models. Moreover, Mimic scores and their associated filters improve upon existing filtering methods and offer accurate estimation of dataset quality.

Title: F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Consistent Gaussian Splatting

• Authors: Yuxin Wang, Qianyi Wu, Dan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06714
• Pdf URL: https://arxiv.org/pdf/2501.06714
• Copy Paste: [[2501.06714]] F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Consistent Gaussian Splatting(https://arxiv.org/abs/2501.06714)
• Keywords: self-supervised
• Abstract: This paper tackles the problem of generalizable 3D-aware generation from monocular datasets, e.g., ImageNet. The key challenge of this task is learning a robust 3D-aware representation without multi-view or dynamic data, while ensuring consistent texture and geometry across different viewpoints. Although some baseline methods are capable of 3D-aware generation, the quality of the generated images still lags behind state-of-the-art 2D generation approaches, which excel in producing high-quality, detailed images. To address this severe limitation, we propose a novel feed-forward pipeline based on pixel-aligned Gaussian Splatting, coined as F3D-Gaus, which can produce more realistic and reliable 3D renderings from monocular inputs. In addition, we introduce a self-supervised cycle-consistent constraint to enforce cross-view consistency in the learned 3D representation. This training strategy naturally allows aggregation of multiple aligned Gaussian primitives and significantly alleviates the interpolation limitations inherent in single-view pixel-aligned Gaussian Splatting. Furthermore, we incorporate video model priors to perform geometry-aware refinement, enhancing the generation of fine details in wide-viewpoint scenarios and improving the model's capability to capture intricate 3D textures. Extensive experiments demonstrate that our approach not only achieves high-quality, multi-view consistent 3D-aware generation from monocular datasets, but also significantly improves training and inference efficiency.

Title: DRDT3: Diffusion-Refined Decision Test-Time Training Model

• Authors: Xingshuai Huang, Di Wu, Benoit Boulet
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06718
• Pdf URL: https://arxiv.org/pdf/2501.06718
• Copy Paste: [[2501.06718]] DRDT3: Diffusion-Refined Decision Test-Time Training Model(https://arxiv.org/abs/2501.06718)
• Keywords: diffusion, generative
• Abstract: Decision Transformer (DT), a trajectory modeling method, has shown competitive performance compared to traditional offline reinforcement learning (RL) approaches on various classic control tasks. However, it struggles to learn optimal policies from suboptimal, reward-labeled trajectories. In this study, we explore the use of conditional generative modeling to facilitate trajectory stitching given its high-quality data generation ability. Additionally, recent advancements in Recurrent Neural Networks (RNNs) have shown their linear complexity and competitive sequence modeling performance over Transformers. We leverage the Test-Time Training (TTT) layer, an RNN that updates hidden states during testing, to model trajectories in the form of DT. We introduce a unified framework, called Diffusion-Refined Decision TTT (DRDT3), to achieve performance beyond DT models. Specifically, we propose the Decision TTT (DT3) module, which harnesses the sequence modeling strengths of both self-attention and the TTT layer to capture recent contextual information and make coarse action predictions. We further integrate DT3 with the diffusion model using a unified optimization objective. With experiments on multiple tasks of Gym and AntMaze in the D4RL benchmark, our DT3 model without diffusion refinement demonstrates improved performance over standard DT, while DRDT3 further achieves superior results compared to state-of-the-art conventional offline RL and DT-based methods.

Title: Better Prompt Compression Without Multi-Layer Perceptrons

• Authors: Edouardo Honig, Andrew Lizarraga, Zijun Frank Zhang, Ying Nian Wu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06730
• Pdf URL: https://arxiv.org/pdf/2501.06730
• Copy Paste: [[2501.06730]] Better Prompt Compression Without Multi-Layer Perceptrons(https://arxiv.org/abs/2501.06730)
• Keywords: generative
• Abstract: Prompt compression is a promising approach to speeding up language model inference without altering the generative model. Prior works compress prompts into smaller sequences of learned tokens using an encoder that is trained as a LowRank Adaptation (LoRA) of the inference language model. However, we show that the encoder does not need to keep the original language model's architecture to achieve useful compression. We introduce the Attention-Only Compressor (AOC), which learns a prompt compression encoder after removing the multilayer perceptron (MLP) layers in the Transformer blocks of a language model, resulting in an encoder with roughly 67% less parameters compared to the original model. Intriguingly we find that, across a range of compression ratios up to 480x, AOC can better regenerate prompts and outperform a baseline compression encoder that is a LoRA of the inference language model without removing MLP layers. These results demonstrate that the architecture of prompt compression encoders does not need to be identical to that of the original decoder language model, paving the way for further research into architectures and approaches for prompt compression.

Title: Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models

• Authors: Michael Toker, Ido Galil, Hadas Orgad, Rinon Gal, Yoad Tewel, Gal Chechik, Yonatan Belinkov
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06751
• Pdf URL: https://arxiv.org/pdf/2501.06751
• Copy Paste: [[2501.06751]] Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models(https://arxiv.org/abs/2501.06751)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process. Typically, these prompts are extended to a fixed length by adding padding tokens before text encoding. Despite being a default practice, the influence of padding tokens on the image generation process has not been investigated. In this work, we conduct the first in-depth analysis of the role padding tokens play in T2I models. We develop two causal techniques to analyze how information is encoded in the representation of tokens across different components of the T2I pipeline. Using these techniques, we investigate when and how padding tokens impact the image generation process. Our findings reveal three distinct scenarios: padding tokens may affect the model's output during text encoding, during the diffusion process, or be effectively ignored. Moreover, we identify key relationships between these scenarios and the model's architecture (cross or self-attention) and its training process (frozen or trained text encoder). These insights contribute to a deeper understanding of the mechanisms of padding tokens, potentially informing future model design and training practices in T2I systems.

Title: ODPG: Outfitting Diffusion with Pose Guided Condition

• Authors: Seohyun Lee, Jintae Park, Sanghyeok Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06769
• Pdf URL: https://arxiv.org/pdf/2501.06769
• Copy Paste: [[2501.06769]] ODPG: Outfitting Diffusion with Pose Guided Condition(https://arxiv.org/abs/2501.06769)
• Keywords: diffusion, generative
• Abstract: Virtual Try-On (VTON) technology allows users to visualize how clothes would look on them without physically trying them on, gaining traction with the rise of digitalization and online shopping. Traditional VTON methods, often using Generative Adversarial Networks (GANs) and Diffusion models, face challenges in achieving high realism and handling dynamic poses. This paper introduces Outfitting Diffusion with Pose Guided Condition (ODPG), a novel approach that leverages a latent diffusion model with multiple conditioning inputs during the denoising process. By transforming garment, pose, and appearance images into latent features and integrating these features in a UNet-based denoising model, ODPG achieves non-explicit synthesis of garments on dynamically posed human images. Our experiments on the FashionTryOn and a subset of the DeepFashion dataset demonstrate that ODPG generates realistic VTON images with fine-grained texture details across various poses, utilizing an end-to-end architecture without the need for explicit garment warping processes. Future work will focus on generating VTON outputs in video format and on applying our attention mechanism, as detailed in the Method section, to other domains with limited data.

Title: Semantic-CD: Remote Sensing Image Semantic Change Detection towards Open-vocabulary Setting

• Authors: Yongshuo Zhu, Lu Li, Keyan Chen, Chenyang Liu, Fugen Zhou, Zhenwei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06808
• Pdf URL: https://arxiv.org/pdf/2501.06808
• Copy Paste: [[2501.06808]] Semantic-CD: Remote Sensing Image Semantic Change Detection towards Open-vocabulary Setting(https://arxiv.org/abs/2501.06808)
• Keywords: foundation model
• Abstract: Remote sensing image semantic change detection is a method used to analyze remote sensing images, aiming to identify areas of change as well as categorize these changes within images of the same location taken at different times. Traditional change detection methods often face challenges in generalizing across semantic categories in practical scenarios. To address this issue, we introduce a novel approach called Semantic-CD, specifically designed for semantic change detection in remote sensing images. This method incorporates the open vocabulary semantics from the vision-language foundation model, CLIP. By utilizing CLIP's extensive vocabulary knowledge, our model enhances its ability to generalize across categories and improves segmentation through fully decoupled multi-task learning, which includes both binary change detection and semantic change detection tasks. Semantic-CD consists of four main components: a bi-temporal CLIP visual encoder for extracting features from bi-temporal images, an open semantic prompter for creating semantic cost volume maps with open vocabulary, a binary change detection decoder for generating binary change detection masks, and a semantic change detection decoder for producing semantic labels. Experimental results on the SECOND dataset demonstrate that Semantic-CD achieves more accurate masks and reduces semantic classification errors, illustrating its effectiveness in applying semantic priors from vision-language foundation models to SCD tasks.

Title: RSRefSeg: Referring Remote Sensing Image Segmentation with Foundation Models

• Authors: Keyan Chen, Jiafan Zhang, Chenyang Liu, Zhengxia Zou, Zhenwei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06809
• Pdf URL: https://arxiv.org/pdf/2501.06809
• Copy Paste: [[2501.06809]] RSRefSeg: Referring Remote Sensing Image Segmentation with Foundation Models(https://arxiv.org/abs/2501.06809)
• Keywords: foundation model
• Abstract: Referring remote sensing image segmentation is crucial for achieving fine-grained visual understanding through free-format textual input, enabling enhanced scene and object extraction in remote sensing applications. Current research primarily utilizes pre-trained language models to encode textual descriptions and align them with visual modalities, thereby facilitating the expression of relevant visual features. However, these approaches often struggle to establish robust alignments between fine-grained semantic concepts, leading to inconsistent representations across textual and visual information. To address these limitations, we introduce a referring remote sensing image segmentation foundational model, RSRefSeg. RSRefSeg leverages CLIP for visual and textual encoding, employing both global and local textual semantics as filters to generate referring-related visual activation features in the latent space. These activated features then serve as input prompts for SAM, which refines the segmentation masks through its robust visual generalization capabilities. Experimental results on the RRSIS-D dataset demonstrate that RSRefSeg outperforms existing methods, underscoring the effectiveness of foundational models in enhancing multimodal task comprehension. The code is available at \url{this https URL}.

Title: A General Framework for Inference-time Scaling and Steering of Diffusion Models

• Authors: Raghav Singhal, Zachary Horvitz, Ryan Teehan, Mengye Ren, Zhou Yu, Kathleen McKeown, Rajesh Ranganath
• Subjects: cs.LG, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06848
• Pdf URL: https://arxiv.org/pdf/2501.06848
• Copy Paste: [[2501.06848]] A General Framework for Inference-time Scaling and Steering of Diffusion Models(https://arxiv.org/abs/2501.06848)
• Keywords: diffusion
• Abstract: Diffusion models produce impressive results in modalities ranging from images and video to protein design and text. However, generating samples with user-specified properties remains a challenge. Recent research proposes fine-tuning models to maximize rewards that capture desired properties, but these methods require expensive training and are prone to mode collapse. In this work, we propose Feynman Kac (FK) steering, an inference-time framework for steering diffusion models with reward functions. FK steering works by sampling a system of multiple interacting diffusion processes, called particles, and resampling particles at intermediate steps based on scores computed using functions called potentials. Potentials are defined using rewards for intermediate states and are selected such that a high value indicates that the particle will yield a high-reward sample. We explore various choices of potentials, intermediate rewards, and samplers. We evaluate FK steering on text-to-image and text diffusion models. For steering text-to-image models with a human preference reward, we find that FK steering a 0.8B parameter model outperforms a 2.6B parameter fine-tuned model on prompt fidelity, with faster sampling and no training. For steering text diffusion models with rewards for text quality and specific text attributes, we find that FK steering generates lower perplexity, more linguistically acceptable outputs and enables gradient-free control of attributes like toxicity. Our results demonstrate that inference-time scaling and steering of diffusion models, even with off-the-shelf rewards, can provide significant sample quality gains and controllability benefits. Code is available at this https URL .

Title: Transfer Learning of Tabular Data by Finetuning Large Language Models

• Authors: Shourav B. Rabbani, Ibna Kowsar, Manar D. Samad
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06863
• Pdf URL: https://arxiv.org/pdf/2501.06863
• Copy Paste: [[2501.06863]] Transfer Learning of Tabular Data by Finetuning Large Language Models(https://arxiv.org/abs/2501.06863)
• Keywords: generative
• Abstract: Despite the artificial intelligence (AI) revolution, deep learning has yet to achieve much success with tabular data due to heterogeneous feature space and limited sample sizes without viable transfer learning. The new era of generative AI, powered by large language models (LLM), brings unprecedented learning opportunities to diverse data and domains. This paper investigates the effectiveness of an LLM application programming interface (API) and transfer learning of LLM in tabular data classification. LLM APIs respond to input text prompts with tokenized data and instructions, whereas transfer learning finetunes an LLM for a target classification task. This paper proposes an end-to-end finetuning of LLM to demonstrate cross-data transfer learning on ten benchmark data sets when large pre-trained tabular data models do not exist to facilitate transfer learning. The proposed LLM finetuning method outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features - a standard feature size for tabular data sets. The transfer learning approach uses a fraction of the computational cost of other deep learning or API-based solutions while ensuring competitive or superior classification performance.

Title: Synthetic Prior for Few-Shot Drivable Head Avatar Inversion

• Authors: Wojciech Zielonka, Stephan J. Garbin, Alexandros Lattas, George Kopanas, Paulo Gotardo, Thabo Beeler, Justus Thies, Timo Bolkart
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06903
• Pdf URL: https://arxiv.org/pdf/2501.06903
• Copy Paste: [[2501.06903]] Synthetic Prior for Few-Shot Drivable Head Avatar Inversion(https://arxiv.org/abs/2501.06903)
• Keywords: generative
• Abstract: We present SynShot, a novel method for the few-shot inversion of a drivable head avatar based on a synthetic prior. We tackle two major challenges. First, training a controllable 3D generative network requires a large number of diverse sequences, for which pairs of images and high-quality tracked meshes are not always available. Second, state-of-the-art monocular avatar models struggle to generalize to new views and expressions, lacking a strong prior and often overfitting to a specific viewpoint distribution. Inspired by machine learning models trained solely on synthetic data, we propose a method that learns a prior model from a large dataset of synthetic heads with diverse identities, expressions, and viewpoints. With few input images, SynShot fine-tunes the pretrained synthetic prior to bridge the domain gap, modeling a photorealistic head avatar that generalizes to novel expressions and viewpoints. We model the head avatar using 3D Gaussian splatting and a convolutional encoder-decoder that outputs Gaussian parameters in UV texture space. To account for the different modeling complexities over parts of the head (e.g., skin vs hair), we embed the prior with explicit control for upsampling the number of per-part primitives. Compared to state-of-the-art monocular methods that require thousands of real training images, SynShot significantly improves novel view and expression synthesis.

Title: Deep Learning and Foundation Models for Weather Prediction: A Survey

• Authors: Jimeng Shi, Azam Shirali, Bowen Jin, Sizhe Zhou, Wei Hu, Rahuul Rangaraj, Shaowen Wang, Jiawei Han, Zhaonan Wang, Upmanu Lall, Yanzhao Wu, Leonardo Bobadilla, Giri Narasimhan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06907
• Pdf URL: https://arxiv.org/pdf/2501.06907
• Copy Paste: [[2501.06907]] Deep Learning and Foundation Models for Weather Prediction: A Survey(https://arxiv.org/abs/2501.06907)
• Keywords: foundation model, generative
• Abstract: Physics-based numerical models have been the bedrock of atmospheric sciences for decades, offering robust solutions but often at the cost of significant computational resources. Deep learning (DL) models have emerged as powerful tools in meteorology, capable of analyzing complex weather and climate data by learning intricate dependencies and providing rapid predictions once trained. While these models demonstrate promising performance in weather prediction, often surpassing traditional physics-based methods, they still face critical challenges. This paper presents a comprehensive survey of recent deep learning and foundation models for weather prediction. We propose a taxonomy to classify existing models based on their training paradigms: deterministic predictive learning, probabilistic generative learning, and pre-training and fine-tuning. For each paradigm, we delve into the underlying model architectures, address major challenges, offer key insights, and propose targeted directions for future research. Furthermore, we explore real-world applications of these methods and provide a curated summary of open-source code repositories and widely used datasets, aiming to bridge research advancements with practical implementations while fostering open and trustworthy scientific practices in adopting cutting-edge artificial intelligence for weather prediction. The related sources are available at this https URL DL-Foundation-Models-Weather.

Title: Comparison of Autoencoders for tokenization of ASL datasets

• Authors: Vouk Praun-Petrovic, Aadhvika Koundinya, Lavanya Prahallad
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06942
• Pdf URL: https://arxiv.org/pdf/2501.06942
• Copy Paste: [[2501.06942]] Comparison of Autoencoders for tokenization of ASL datasets(https://arxiv.org/abs/2501.06942)
• Keywords: diffusion, generative
• Abstract: Generative AI, powered by large language models (LLMs), has revolutionized applications across text, audio, images, and video. This study focuses on developing and evaluating encoder-decoder architectures for the American Sign Language (ASL) image dataset, consisting of 87,000 images across 29 hand sign classes. Three approaches were compared: Feedforward Autoencoders, Convolutional Autoencoders, and Diffusion Autoencoders. The Diffusion Autoencoder outperformed the others, achieving the lowest mean squared error (MSE) and highest Mean Opinion Score (MOS) due to its probabilistic noise modeling and iterative denoising capabilities. The Convolutional Autoencoder demonstrated effective spatial feature extraction but lacked the robustness of the diffusion process, while the Feedforward Autoencoder served as a baseline with limitations in handling complex image data. Objective and subjective evaluations confirmed the superiority of the Diffusion Autoencoder for high-fidelity image reconstruction, emphasizing its potential in multimodal AI applications such as sign language recognition and generation. This work provides critical insights into designing robust encoder-decoder systems to advance multimodal AI capabilities.

Title: Generative Artificial Intelligence-Supported Pentesting: A Comparison between Claude Opus, GPT-4, and Copilot

• Authors: Antonio López Martínez, Alejandro Cano, Antonio Ruiz-Martínez
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06963
• Pdf URL: https://arxiv.org/pdf/2501.06963
• Copy Paste: [[2501.06963]] Generative Artificial Intelligence-Supported Pentesting: A Comparison between Claude Opus, GPT-4, and Copilot(https://arxiv.org/abs/2501.06963)
• Keywords: generative
• Abstract: The advent of Generative Artificial Intelligence (GenAI) has brought a significant change to our society. GenAI can be applied across numerous fields, with particular relevance in cybersecurity. Among the various areas of application, its use in penetration testing (pentesting) or ethical hacking processes is of special interest. In this paper, we have analyzed the potential of leading generic-purpose GenAI tools-Claude Opus, GPT-4 from ChatGPT, and Copilot-in augmenting the penetration testing process as defined by the Penetration Testing Execution Standard (PTES). Our analysis involved evaluating each tool across all PTES phases within a controlled virtualized environment. The findings reveal that, while these tools cannot fully automate the pentesting process, they provide substantial support by enhancing efficiency and effectiveness in specific tasks. Notably, all tools demonstrated utility; however, Claude Opus consistently outperformed the others in our experimental scenarios.

Title: TFLAG:Towards Practical APT Detection via Deviation-Aware Learning on Temporal Provenance Graph

• Authors: Wenhan Jiang, Tingting Chai, Hongri Liu, Kai Wang, Hongke Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.06997
• Pdf URL: https://arxiv.org/pdf/2501.06997
• Copy Paste: [[2501.06997]] TFLAG:Towards Practical APT Detection via Deviation-Aware Learning on Temporal Provenance Graph(https://arxiv.org/abs/2501.06997)
• Keywords: self-supervised, anomaly
• Abstract: Advanced Persistent Threat (APT) have grown increasingly complex and concealed, posing formidable challenges to existing Intrusion Detection Systems in identifying and mitigating these attacks. Recent studies have incorporated graph learning techniques to extract detailed information from provenance graphs, enabling the detection of attacks with greater granularity. Nevertheless, existing studies have largely overlooked the continuous yet subtle temporal variations in the structure of provenance graphs, which may correspond to surreptitious perturbation anomalies in ongoing APT attacks. Therefore, we introduce TFLAG, an advanced anomaly detection framework that for the first time integrates the structural dynamic extraction capabilities of temporal graph model with the anomaly delineation abilities of deviation networks to pinpoint covert attack activities in provenance graphs. This self-supervised integration framework leverages the graph model to extract neighbor interaction data under continuous temporal changes from historical benign behaviors within provenance graphs, while simultaneously utilizing deviation networks to accurately distinguish authentic attack activities from false positive deviations due to unexpected subtle perturbations. The experimental results indicate that, through a comprehensive design that utilizes both attribute and temporal information, it can accurately identify the time windows associated with APT attack behaviors without prior knowledge (e.g., labeled data samples), demonstrating superior accuracy compared to current state-of-the-art methods in differentiating between attack events and system false positive events.

Title: Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps

• Authors: Henry Li, Ronen Basri, Yuval Kluger
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06999
• Pdf URL: https://arxiv.org/pdf/2501.06999
• Copy Paste: [[2501.06999]] Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps(https://arxiv.org/abs/2501.06999)
• Keywords: diffusion, generative
• Abstract: Cascaded models are multi-scale generative models with a marked capacity for producing perceptually impressive samples at high resolutions. In this work, we show that they can also be excellent likelihood models, so long as we overcome a fundamental difficulty with probabilistic multi-scale models: the intractability of the likelihood function. Chiefly, in cascaded models each intermediary scale introduces extraneous variables that cannot be tractably marginalized out for likelihood evaluation. This issue vanishes by modeling the diffusion process on latent spaces induced by a class of transformations we call hierarchical volume-preserving maps, which decompose spatially structured data in a hierarchical fashion without introducing local distortions in the latent space. We demonstrate that two such maps are well-known in the literature for multiscale modeling: Laplacian pyramids and wavelet transforms. Not only do such reparameterizations allow the likelihood function to be directly expressed as a joint likelihood over the scales, we show that the Laplacian pyramid and wavelet transform also produces significant improvements to the state-of-the-art on a selection of benchmarks in likelihood modeling, including density estimation, lossless compression, and out-of-distribution detection. Investigating the theoretical basis of our empirical gains we uncover deep connections to score matching under the Earth Mover's Distance (EMD), which is a well-known surrogate for perceptual similarity. Code can be found at \href{this https URL}{this https url}.

Title: Detection of AI Deepfake and Fraud in Online Payments Using GAN-Based Models

• Authors: Zong Ke, Shicheng Zhou, Yining Zhou, Chia Hong Chang, Rong Zhang
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07033
• Pdf URL: https://arxiv.org/pdf/2501.07033
• Copy Paste: [[2501.07033]] Detection of AI Deepfake and Fraud in Online Payments Using GAN-Based Models(https://arxiv.org/abs/2501.07033)
• Keywords: generative
• Abstract: This study explores the use of Generative Adversarial Networks (GANs) to detect AI deepfakes and fraudulent activities in online payment systems. With the growing prevalence of deepfake technology, which can manipulate facial features in images and videos, the potential for fraud in online transactions has escalated. Traditional security systems struggle to identify these sophisticated forms of fraud. This research proposes a novel GAN-based model that enhances online payment security by identifying subtle manipulations in payment images. The model is trained on a dataset consisting of real-world online payment images and deepfake images generated using advanced GAN architectures, such as StyleGAN and DeepFake. The results demonstrate that the proposed model can accurately distinguish between legitimate transactions and deepfakes, achieving a high detection rate above 95%. This approach significantly improves the robustness of payment systems against AI-driven fraud. The paper contributes to the growing field of digital security, offering insights into the application of GANs for fraud detection in financial services. Keywords- Payment Security, Image Recognition, Generative Adversarial Networks, AI Deepfake, Fraudulent Activities

Title: Explore the Use of Time Series Foundation Model for Car-Following Behavior Analysis

• Authors: Luwei Zeng, Runze Yan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07034
• Pdf URL: https://arxiv.org/pdf/2501.07034
• Copy Paste: [[2501.07034]] Explore the Use of Time Series Foundation Model for Car-Following Behavior Analysis(https://arxiv.org/abs/2501.07034)
• Keywords: foundation model
• Abstract: Modeling car-following behavior is essential for traffic simulation, analyzing driving patterns, and understanding complex traffic flows with varying levels of autonomous vehicles. Traditional models like the Safe Distance Model and Intelligent Driver Model (IDM) require precise parameter calibration and often lack generality due to simplified assumptions about driver behavior. While machine learning and deep learning methods capture complex patterns, they require large labeled datasets. Foundation models provide a more efficient alternative. Pre-trained on vast, diverse time series datasets, they can be applied directly to various tasks without the need for extensive re-training. These models generalize well across domains, and with minimal fine-tuning, they can be adapted to specific tasks like car-following behavior prediction. In this paper, we apply Chronos, a state-of-the-art public time series foundation model, to analyze car-following behavior using the Open ACC dataset. Without fine-tuning, Chronos outperforms traditional models like IDM and Exponential smoothing with trend and seasonality (ETS), and achieves similar results to deep learning models such as DeepAR and TFT, with an RMSE of 0.60. After fine-tuning, Chronos reduces the error to an RMSE of 0.53, representing a 33.75% improvement over IDM and a 12-37% reduction compared to machine learning models like ETS and deep learning models including DeepAR, WaveNet, and TFT. This demonstrates the potential of foundation models to significantly advance transportation research, offering a scalable, adaptable, and highly accurate approach to predicting and simulating car-following behaviors.

Title: Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective

• Authors: Jinjing Zhu, Songze Li, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07040
• Pdf URL: https://arxiv.org/pdf/2501.07040
• Copy Paste: [[2501.07040]] Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective(https://arxiv.org/abs/2501.07040)
• Keywords: in-context
• Abstract: Conventional knowledge distillation (KD) approaches are designed for the student model to predict similar output as the teacher model for each sample. Unfortunately, the relationship across samples with same class is often neglected. In this paper, we explore to redefine the knowledge in distillation, capturing the relationship between each sample and its corresponding in-context samples (a group of similar samples with the same or different classes), and perform KD from an in-context sample retrieval perspective. As KD is a type of learned label smoothing regularization (LSR), we first conduct a theoretical analysis showing that the teacher's knowledge from the in-context samples is a crucial contributor to regularize the student training with the corresponding samples. Buttressed by the analysis, we propose a novel in-context knowledge distillation (IC-KD) framework that shows its superiority across diverse KD paradigms (offline, online, and teacher-free KD). Firstly, we construct a feature memory bank from the teacher model and retrieve in-context samples for each corresponding sample through retrieval-based learning. We then introduce Positive In-Context Distillation (PICD) to reduce the discrepancy between a sample from the student and the aggregated in-context samples with the same class from the teacher in the logit space. Moreover, Negative In-Context Distillation (NICD) is introduced to separate a sample from the student and the in-context samples with different classes from the teacher in the logit space. Extensive experiments demonstrate that IC-KD is effective across various types of KD, and consistently achieves state-of-the-art performance on CIFAR-100 and ImageNet datasets.

Title: SFC-GAN: A Generative Adversarial Network for Brain Functional and Structural Connectome Translation

• Authors: Yee-Fan Tan, Jun Lin Liow, Pei-Sze Tan, Fuad Noman, Raphael C.-W. Phan, Hernando Ombao, Chee-Ming Ting
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07055
• Pdf URL: https://arxiv.org/pdf/2501.07055
• Copy Paste: [[2501.07055]] SFC-GAN: A Generative Adversarial Network for Brain Functional and Structural Connectome Translation(https://arxiv.org/abs/2501.07055)
• Keywords: diffusion, generative
• Abstract: Modern brain imaging technologies have enabled the detailed reconstruction of human brain connectomes, capturing structural connectivity (SC) from diffusion MRI and functional connectivity (FC) from functional MRI. Understanding the intricate relationships between SC and FC is vital for gaining deeper insights into the brain's functional and organizational mechanisms. However, obtaining both SC and FC modalities simultaneously remains challenging, hindering comprehensive analyses. Existing deep generative models typically focus on synthesizing a single modality or unidirectional translation between FC and SC, thereby missing the potential benefits of bi-directional translation, especially in scenarios where only one connectome is available. Therefore, we propose Structural-Functional Connectivity GAN (SFC-GAN), a novel framework for bidirectional translation between SC and FC. This approach leverages the CycleGAN architecture, incorporating convolutional layers to effectively capture the spatial structures of brain connectomes. To preserve the topological integrity of these connectomes, we employ a structure-preserving loss that guides the model in capturing both global and local connectome patterns while maintaining symmetry. Our framework demonstrates superior performance in translating between SC and FC, outperforming baseline models in similarity and graph property evaluations compared to ground truth data, each translated modality can be effectively utilized for downstream classification.

Title: Enhancing Image Generation Fidelity via Progressive Prompts

• Authors: Zhen Xiong, Yuqi Li, Chuanguang Yang, Tiao Tan, Zhihong Zhu, Siyuan Li, Yue Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07070
• Pdf URL: https://arxiv.org/pdf/2501.07070
• Copy Paste: [[2501.07070]] Enhancing Image Generation Fidelity via Progressive Prompts(https://arxiv.org/abs/2501.07070)
• Keywords: diffusion
• Abstract: The diffusion transformer (DiT) architecture has attracted significant attention in image generation, achieving better fidelity, performance, and diversity. However, most existing DiT - based image generation methods focus on global - aware synthesis, and regional prompt control has been less explored. In this paper, we propose a coarse - to - fine generation pipeline for regional prompt - following generation. Specifically, we first utilize the powerful large language model (LLM) to generate both high - level descriptions of the image (such as content, topic, and objects) and low - level descriptions (such as details and style). Then, we explore the influence of cross - attention layers at different depths. We find that deeper layers are always responsible for high - level content control, while shallow layers handle low - level content control. Various prompts are injected into the proposed regional cross - attention control for coarse - to - fine generation. By using the proposed pipeline, we enhance the controllability of DiT - based image generation. Extensive quantitative and qualitative results show that our pipeline can improve the performance of the generated images.

Title: Label Calibration in Source Free Domain Adaptation

• Authors: Shivangi Rai, Rini Smita Thakur, Kunal Jangid, Vinod K Kurmi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07072
• Pdf URL: https://arxiv.org/pdf/2501.07072
• Copy Paste: [[2501.07072]] Label Calibration in Source Free Domain Adaptation(https://arxiv.org/abs/2501.07072)
• Keywords: self-supervised
• Abstract: Source-free domain adaptation (SFDA) utilizes a pre-trained source model with unlabeled target data. Self-supervised SFDA techniques generate pseudolabels from the pre-trained source model, but these pseudolabels often contain noise due to domain discrepancies between the source and target domains. Traditional self-supervised SFDA techniques rely on deterministic model predictions using the softmax function, leading to unreliable pseudolabels. In this work, we propose to introduce predictive uncertainty and softmax calibration for pseudolabel refinement using evidential deep learning. The Dirichlet prior is placed over the output of the target network to capture uncertainty using evidence with a single forward pass. Furthermore, softmax calibration solves the translation invariance problem to assist in learning with noisy labels. We incorporate a combination of evidential deep learning loss and information maximization loss with calibrated softmax in both prior and non-prior target knowledge SFDA settings. Extensive experimental analysis shows that our method outperforms other state-of-the-art methods on benchmark datasets.

Title: D3MES: Diffusion Transformer with multihead equivariant self-attention for 3D molecule generation

• Authors: Zhejun Zhang, Yuanping Chen, Shibing Chu
• Subjects: cs.LG, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2501.07077
• Pdf URL: https://arxiv.org/pdf/2501.07077
• Copy Paste: [[2501.07077]] D3MES: Diffusion Transformer with multihead equivariant self-attention for 3D molecule generation(https://arxiv.org/abs/2501.07077)
• Keywords: diffusion, generative
• Abstract: Understanding and predicting the diverse conformational states of molecules is crucial for advancing fields such as chemistry, material science, and drug development. Despite significant progress in generative models, accurately generating complex and biologically or material-relevant molecular structures remains a major challenge. In this work, we introduce a diffusion model for three-dimensional (3D) molecule generation that combines a classifiable diffusion model, Diffusion Transformer, with multihead equivariant self-attention. This method addresses two key challenges: correctly attaching hydrogen atoms in generated molecules through learning representations of molecules after hydrogen atoms are removed; and overcoming the limitations of existing models that cannot generate molecules across multiple classes simultaneously. The experimental results demonstrate that our model not only achieves state-of-the-art performance across several key metrics but also exhibits robustness and versatility, making it highly suitable for early-stage large-scale generation processes in molecular design, followed by validation and further screening to obtain molecules with specific properties.

Title: Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models

• Authors: Yongyu Mu, Hengyu Li, Junxin Wang, Xiaoxuan Zhou, Chenglong Wang, Yingfeng Luo, Qiaozhi He, Tong Xiao, Guocheng Chen, Jingbo Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.07086
• Pdf URL: https://arxiv.org/pdf/2501.07086
• Copy Paste: [[2501.07086]] Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models(https://arxiv.org/abs/2501.07086)
• Keywords: in-context
• Abstract: Previous work on augmenting large multimodal models (LMMs) for text-to-image (T2I) generation has focused on enriching the input space of in-context learning (ICL). This includes providing a few demonstrations and optimizing image descriptions to be more detailed and logical. However, as demand for more complex and flexible image descriptions grows, enhancing comprehension of input text within the ICL paradigm remains a critical yet underexplored area. In this work, we extend this line of research by constructing parallel multilingual prompts aimed at harnessing the multilingual capabilities of LMMs. More specifically, we translate the input text into several languages and provide the models with both the original text and the translations. Experiments on two LMMs across 3 benchmarks show that our method, PMT2I, achieves superior performance in general, compositional, and fine-grained assessments, especially in human preference alignment. Additionally, with its advantage of generating more diverse images, PMT2I significantly outperforms baseline prompts when incorporated with reranking methods. Our code and parallel multilingual data can be found at this https URL.

Title: Matching Free Depth Recovery from Structured Light

• Authors: Zhuohang Yu, Kai Wang, Juyong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07113
• Pdf URL: https://arxiv.org/pdf/2501.07113
• Copy Paste: [[2501.07113]] Matching Free Depth Recovery from Structured Light(https://arxiv.org/abs/2501.07113)
• Keywords: self-supervised
• Abstract: We present a novel approach for depth estimation from images captured by structured light systems. Unlike many previous methods that rely on image matching process, our approach uses a density voxel grid to represent scene geometry, which is trained via self-supervised differentiable volume rendering. Our method leverages color fields derived from projected patterns in structured light systems during the rendering process, enabling the isolated optimization of the geometry field. This contributes to faster convergence and high-quality output. Additionally, we incorporate normalized device coordinates (NDC), a distortion loss, and a novel surface-based color loss to enhance geometric fidelity. Experimental results demonstrate that our method outperforms existing matching-based techniques in geometric performance for few-shot scenarios, achieving approximately a 60% reduction in average estimated depth errors on synthetic scenes and about 30% on real-world captured scenes. Furthermore, our approach delivers fast training, with a speed roughly three times faster than previous matching-free methods that employ implicit representations.

Title: AlphaNet: Scaling Up Local Frame-based Atomistic Foundation Model

• Authors: Bangchen Yin, Jiaao Wang, Weitao Du, Pengbo Wang, Penghua Ying, Haojun Jia, Zisheng Zhang, Yuanqi Du, Carla P. Gomes, Chenru Duan, Hai Xiao, Graeme Henkelman
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07155
• Pdf URL: https://arxiv.org/pdf/2501.07155
• Copy Paste: [[2501.07155]] AlphaNet: Scaling Up Local Frame-based Atomistic Foundation Model(https://arxiv.org/abs/2501.07155)
• Keywords: foundation model
• Abstract: We present AlphaNet, a local frame-based equivariant model designed to achieve both accurate and efficient simulations for atomistic systems. Recently, machine learning force fields (MLFFs) have gained prominence in molecular dynamics simulations due to their advantageous efficiency-accuracy balance compared to classical force fields and quantum mechanical calculations, alongside their transferability across various systems. Despite the advancements in improving model accuracy, the efficiency and scalability of MLFFs remain significant obstacles in practical applications. AlphaNet enhances computational efficiency and accuracy by leveraging the local geometric structures of atomic environments through the construction of equivariant local frames and learnable frame transitions. We substantiate the efficacy of AlphaNet across diverse datasets, including defected graphene, formate decomposition, zeolites, and surface reactions. AlphaNet consistently surpasses well-established models, such as NequIP and DeepPot, in terms of both energy and force prediction accuracy. Notably, AlphaNet offers one of the best trade-offs between computational efficiency and accuracy among existing models. Moreover, AlphaNet exhibits scalability across a broad spectrum of system and dataset sizes, affirming its versatility.

Title: Anomalous Agreement: How to find the Ideal Number of Anomaly Classes in Correlated, Multivariate Time Series Data

• Authors: Ferdinand Rewicki, Joachim Denzler, Julia Niebling
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.07172
• Pdf URL: https://arxiv.org/pdf/2501.07172
• Copy Paste: [[2501.07172]] Anomalous Agreement: How to find the Ideal Number of Anomaly Classes in Correlated, Multivariate Time Series Data(https://arxiv.org/abs/2501.07172)
• Keywords: anomaly
• Abstract: Detecting and classifying abnormal system states is critical for condition monitoring, but supervised methods often fall short due to the rarity of anomalies and the lack of labeled data. Therefore, clustering is often used to group similar abnormal behavior. However, evaluating cluster quality without ground truth is challenging, as existing measures such as the Silhouette Score (SSC) only evaluate the cohesion and separation of clusters and ignore possible prior knowledge about the data. To address this challenge, we introduce the Synchronized Anomaly Agreement Index (SAAI), which exploits the synchronicity of anomalies across multivariate time series to assess cluster quality. We demonstrate the effectiveness of SAAI by showing that maximizing SAAI improves accuracy on the task of finding the true number of anomaly classes K in correlated time series by 0.23 compared to SSC and by 0.32 compared to X-Means. We also show that clusters obtained by maximizing SAAI are easier to interpret compared to SSC.

Title: EdgeTAM: On-Device Track Anything Model

• Authors: Chong Zhou, Chenchen Zhu, Yunyang Xiong, Saksham Suri, Fanyi Xiao, Lemeng Wu, Raghuraman Krishnamoorthi, Bo Dai, Chen Change Loy, Vikas Chandra, Bilge Soran
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07256
• Pdf URL: https://arxiv.org/pdf/2501.07256
• Copy Paste: [[2501.07256]] EdgeTAM: On-Device Track Anything Model(https://arxiv.org/abs/2501.07256)
• Keywords: foundation model
• Abstract: On top of Segment Anything Model (SAM), SAM 2 further extends its capability from image to video inputs through a memory bank mechanism and obtains a remarkable performance compared with previous methods, making it a foundation model for video segmentation task. In this paper, we aim at making SAM 2 much more efficient so that it even runs on mobile devices while maintaining a comparable performance. Despite several works optimizing SAM for better efficiency, we find they are not sufficient for SAM 2 because they all focus on compressing the image encoder, while our benchmark shows that the newly introduced memory attention blocks are also the latency bottleneck. Given this observation, we propose EdgeTAM, which leverages a novel 2D Spatial Perceiver to reduce the computational cost. In particular, the proposed 2D Spatial Perceiver encodes the densely stored frame-level memories with a lightweight Transformer that contains a fixed set of learnable queries. Given that video segmentation is a dense prediction task, we find preserving the spatial structure of the memories is essential so that the queries are split into global-level and patch-level groups. We also propose a distillation pipeline that further improves the performance without inference overhead. As a result, EdgeTAM achieves 87.7, 70.0, 72.3, and 71.7 J&F on DAVIS 2017, MOSE, SA-V val, and SA-V test, while running at 16 FPS on iPhone 15 Pro Max.

Title: Skip Mamba Diffusion for Monocular 3D Semantic Scene Completion

• Authors: Li Liang, Naveed Akhtar, Jordan Vice, Xiangrui Kong, Ajmal Saeed Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07260
• Pdf URL: https://arxiv.org/pdf/2501.07260
• Copy Paste: [[2501.07260]] Skip Mamba Diffusion for Monocular 3D Semantic Scene Completion(https://arxiv.org/abs/2501.07260)
• Keywords: diffusion, generative
• Abstract: 3D semantic scene completion is critical for multiple downstream tasks in autonomous systems. It estimates missing geometric and semantic information in the acquired scene data. Due to the challenging real-world conditions, this task usually demands complex models that process multi-modal data to achieve acceptable performance. We propose a unique neural model, leveraging advances from the state space and diffusion generative modeling to achieve remarkable 3D semantic scene completion performance with monocular image input. Our technique processes the data in the conditioned latent space of a variational autoencoder where diffusion modeling is carried out with an innovative state space technique. A key component of our neural network is the proposed Skimba (Skip Mamba) denoiser, which is adept at efficiently processing long-sequence data. The Skimba diffusion model is integral to our 3D scene completion network, incorporating a triple Mamba structure, dimensional decomposition residuals and varying dilations along three directions. We also adopt a variant of this network for the subsequent semantic segmentation stage of our method. Extensive evaluation on the standard SemanticKITTI and SSCBench-KITTI360 datasets show that our approach not only outperforms other monocular techniques by a large margin, it also achieves competitive performance against stereo methods. The code is available at this https URL

Title: Foundation Models at Work: Fine-Tuning for Fairness in Algorithmic Hiring

• Authors: Buse Sibel Korkmaz, Rahul Nair, Elizabeth M. Daly, Evangelos Anagnostopoulos, Christos Varytimidis, Antonio del Rio Chanona
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07324
• Pdf URL: https://arxiv.org/pdf/2501.07324
• Copy Paste: [[2501.07324]] Foundation Models at Work: Fine-Tuning for Fairness in Algorithmic Hiring(https://arxiv.org/abs/2501.07324)
• Keywords: foundation model, generative
• Abstract: Foundation models require fine-tuning to ensure their generative outputs align with intended results for specific tasks. Automating this fine-tuning process is challenging, as it typically needs human feedback that can be expensive to acquire. We present AutoRefine, a method that leverages reinforcement learning for targeted fine-tuning, utilizing direct feedback from measurable performance improvements in specific downstream tasks. We demonstrate the method for a problem arising in algorithmic hiring platforms where linguistic biases influence a recommendation system. In this setting, a generative model seeks to rewrite given job specifications to receive more diverse candidate matches from a recommendation engine which matches jobs to candidates. Our model detects and regulates biases in job descriptions to meet diversity and fairness criteria. The experiments on a public hiring dataset and a real-world hiring platform showcase how large language models can assist in identifying and mitigation biases in the real world.

Title: Deep Generative Clustering with VAEs and Expectation-Maximization

• Authors: Michael Adipoetra, Ségolène Martin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.07358
• Pdf URL: https://arxiv.org/pdf/2501.07358
• Copy Paste: [[2501.07358]] Deep Generative Clustering with VAEs and Expectation-Maximization(https://arxiv.org/abs/2501.07358)
• Keywords: generative
• Abstract: We propose a novel deep clustering method that integrates Variational Autoencoders (VAEs) into the Expectation-Maximization (EM) framework. Our approach models the probability distribution of each cluster with a VAE and alternates between updating model parameters by maximizing the Evidence Lower Bound (ELBO) of the log-likelihood and refining cluster assignments based on the learned distributions. This enables effective clustering and generation of new samples from each cluster. Unlike existing VAE-based methods, our approach eliminates the need for a Gaussian Mixture Model (GMM) prior or additional regularization techniques. Experiments on MNIST and FashionMNIST demonstrate superior clustering performance compared to state-of-the-art methods.

Title: Enhancing Retrieval-Augmented Generation: A Study of Best Practices

• Authors: Siran Li, Linus Stenzel, Carsten Eickhoff, Seyed Ali Bahrainian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07391
• Pdf URL: https://arxiv.org/pdf/2501.07391
• Copy Paste: [[2501.07391]] Enhancing Retrieval-Augmented Generation: A Study of Best Practices(https://arxiv.org/abs/2501.07391)
• Keywords: in-context
• Abstract: Retrieval-Augmented Generation (RAG) systems have recently shown remarkable advancements by integrating retrieval mechanisms into language models, enhancing their ability to produce more accurate and contextually relevant responses. However, the influence of various components and configurations within RAG systems remains underexplored. A comprehensive understanding of these elements is essential for tailoring RAG systems to complex retrieval tasks and ensuring optimal performance across diverse applications. In this paper, we develop several advanced RAG system designs that incorporate query expansion, various novel retrieval strategies, and a novel Contrastive In-Context Learning RAG. Our study systematically investigates key factors, including language model size, prompt design, document chunk size, knowledge base size, retrieval stride, query expansion techniques, Contrastive In-Context Learning knowledge bases, multilingual knowledge bases, and Focus Mode retrieving relevant context at sentence-level. Through extensive experimentation, we provide a detailed analysis of how these factors influence response quality. Our findings offer actionable insights for developing RAG systems, striking a balance between contextual richness and retrieval-generation efficiency, thereby paving the way for more adaptable and high-performing RAG frameworks in diverse real-world scenarios. Our code and implementation details are publicly available.

Title: OCORD: Open-Campus Object Removal Dataset

• Authors: Shuo Zhang, Runpu Wei, Kongming Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07397
• Pdf URL: https://arxiv.org/pdf/2501.07397
• Copy Paste: [[2501.07397]] OCORD: Open-Campus Object Removal Dataset(https://arxiv.org/abs/2501.07397)
• Keywords: diffusion, generative
• Abstract: The rapid advancements in generative models, particularly diffusion-based techniques, have revolutionized image inpainting tasks by enabling the generation of high-fidelity and diverse content. However, object removal remains under-explored as a specific subset of inpainting, facing challenges such as inadequate semantic understanding and the unintended generation of artifacts. Existing datasets for object removal often rely on synthetic data, which fails to align with real-world scenarios, limiting model performance. Although some real-world datasets address these issues partially, they suffer from scalability, annotation inefficiencies, and limited realism in physical phenomena such as lighting and shadows. To address these limitations, this paper introduces a novel approach to object removal by constructing a high-resolution real-world dataset through long-duration video capture with fixed camera settings. Leveraging advanced tools such as Grounding-DINO, Segment-Anything-Model, and MASA for automated annotation, we provides image, background, and mask pairs while significantly reducing annotation time and labor. With our efficient annotation pipeline, we release the first fully open, high-resolution real-world dataset for object removal, and improved performance in object removal tasks through fine-tuning of pre-trained diffusion models.

Title: Diff-Ensembler: Learning to Ensemble 2D Diffusion Models for Volume-to-Volume Medical Image Translation

• Authors: Xiyue Zhu, Dou Hoon Kwark, Ruike Zhu, Kaiwen Hong, Yiqi Tao, Shirui Luo, Yudu Li, Zhi-Pei Liang, Volodymyr Kindratenko
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07430
• Pdf URL: https://arxiv.org/pdf/2501.07430
• Copy Paste: [[2501.07430]] Diff-Ensembler: Learning to Ensemble 2D Diffusion Models for Volume-to-Volume Medical Image Translation(https://arxiv.org/abs/2501.07430)
• Keywords: diffusion
• Abstract: Despite success in volume-to-volume translations in medical images, most existing models struggle to effectively capture the inherent volumetric distribution using 3D representations. The current state-of-the-art approach combines multiple 2D-based networks through weighted averaging, thereby neglecting the 3D spatial structures. Directly training 3D models in medical imaging presents significant challenges due to high computational demands and the need for large-scale datasets. To address these challenges, we introduce Diff-Ensembler, a novel hybrid 2D-3D model for efficient and effective volumetric translations by ensembling perpendicularly trained 2D diffusion models with a 3D network in each diffusion step. Moreover, our model can naturally be used to ensemble diffusion models conditioned on different modalities, allowing flexible and accurate fusion of input conditions. Extensive experiments demonstrate that Diff-Ensembler attains superior accuracy and volumetric realism in 3D medical image super-resolution and modality translation. We further demonstrate the strength of our model's volumetric realism using tumor segmentation as a downstream task.

Title: PrecipDiff: Leveraging image diffusion models to enhance satellite-based precipitation observations

• Authors: Ting-Yu Dai, Hayato Ushijima-Mwesigwa
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07447
• Pdf URL: https://arxiv.org/pdf/2501.07447
• Copy Paste: [[2501.07447]] PrecipDiff: Leveraging image diffusion models to enhance satellite-based precipitation observations(https://arxiv.org/abs/2501.07447)
• Keywords: diffusion
• Abstract: A recent report from the World Meteorological Organization (WMO) highlights that water-related disasters have caused the highest human losses among natural disasters over the past 50 years, with over 91\% of deaths occurring in low-income countries. This disparity is largely due to the lack of adequate ground monitoring stations, such as weather surveillance radars (WSR), which are expensive to install. For example, while the US and Europe combined possess over 600 WSRs, Africa, despite having almost one and half times their landmass, has fewer than 40. To address this issue, satellite-based observations offer a global, near-real-time monitoring solution. However, they face several challenges like accuracy, bias, and low spatial resolution. This study leverages the power of diffusion models and residual learning to address these limitations in a unified framework. We introduce the first diffusion model for correcting the inconsistency between different precipitation products. Our method demonstrates the effectiveness in downscaling satellite precipitation estimates from 10 km to 1 km resolution. Extensive experiments conducted in the Seattle region demonstrate significant improvements in accuracy, bias reduction, and spatial detail. Importantly, our approach achieves these results using only precipitation data, showcasing the potential of a purely computer vision-based approach for enhancing satellite precipitation products and paving the way for further advancements in this domain.

Title: RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment

• Authors: Difei Gu, Yunhe Gao, Yang Zhou, Mu Zhou, Dimitris Metaxas
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07525
• Pdf URL: https://arxiv.org/pdf/2501.07525
• Copy Paste: [[2501.07525]] RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment(https://arxiv.org/abs/2501.07525)
• Keywords: generative
• Abstract: Automated chest radiographs interpretation requires both accurate disease classification and detailed radiology report generation, presenting a significant challenge in the clinical workflow. Current approaches either focus on classification accuracy at the expense of interpretability or generate detailed but potentially unreliable reports through image captioning techniques. In this study, we present RadAlign, a novel framework that combines the predictive accuracy of vision-language models (VLMs) with the reasoning capabilities of large language models (LLMs). Inspired by the radiologist's workflow, RadAlign first employs a specialized VLM to align visual features with key medical concepts, achieving superior disease classification with an average AUC of 0.885 across multiple diseases. These recognized medical conditions, represented as text-based concepts in the aligned visual-language space, are then used to prompt LLM-based report generation. Enhanced by a retrieval-augmented generation mechanism that grounds outputs in similar historical cases, RadAlign delivers superior report quality with a GREEN score of 0.678, outperforming state-of-the-art methods' 0.634. Our framework maintains strong clinical interpretability while reducing hallucinations, advancing automated medical imaging and report analysis through integrated predictive and generative AI. Code is available at this https URL.

Title: IP-FaceDiff: Identity-Preserving Facial Video Editing with Diffusion

• Authors: Tharun Anand, Aryan Garg, Kaushik Mitra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07530
• Pdf URL: https://arxiv.org/pdf/2501.07530
• Copy Paste: [[2501.07530]] IP-FaceDiff: Identity-Preserving Facial Video Editing with Diffusion(https://arxiv.org/abs/2501.07530)
• Keywords: diffusion
• Abstract: Facial video editing has become increasingly important for content creators, enabling the manipulation of facial expressions and attributes. However, existing models encounter challenges such as poor editing quality, high computational costs and difficulties in preserving facial identity across diverse edits. Additionally, these models are often constrained to editing predefined facial attributes, limiting their flexibility to diverse editing prompts. To address these challenges, we propose a novel facial video editing framework that leverages the rich latent space of pre-trained text-to-image (T2I) diffusion models and fine-tune them specifically for facial video editing tasks. Our approach introduces a targeted fine-tuning scheme that enables high quality, localized, text-driven edits while ensuring identity preservation across video frames. Additionally, by using pre-trained T2I models during inference, our approach significantly reduces editing time by 80%, while maintaining temporal consistency throughout the video sequence. We evaluate the effectiveness of our approach through extensive testing across a wide range of challenging scenarios, including varying head poses, complex action sequences, and diverse facial expressions. Our method consistently outperforms existing techniques, demonstrating superior performance across a broad set of metrics and benchmarks.

Title: Confident Pseudo-labeled Diffusion Augmentation for Canine Cardiomegaly Detection

• Authors: Shiman Zhang, Lakshmikar Reddy Polamreddy, Youshan Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07533
• Pdf URL: https://arxiv.org/pdf/2501.07533
• Copy Paste: [[2501.07533]] Confident Pseudo-labeled Diffusion Augmentation for Canine Cardiomegaly Detection(https://arxiv.org/abs/2501.07533)
• Keywords: diffusion
• Abstract: Canine cardiomegaly, marked by an enlarged heart, poses serious health risks if undetected, requiring accurate diagnostic methods. Current detection models often rely on small, poorly annotated datasets and struggle to generalize across diverse imaging conditions, limiting their real-world applicability. To address these issues, we propose a Confident Pseudo-labeled Diffusion Augmentation (CDA) model for identifying canine cardiomegaly. Our approach addresses the challenge of limited high-quality training data by employing diffusion models to generate synthetic X-ray images and annotate Vertebral Heart Score key points, thereby expanding the dataset. We also employ a pseudo-labeling strategy with Monte Carlo Dropout to select high-confidence labels, refine the synthetic dataset, and improve accuracy. Iteratively incorporating these labels enhances the model's performance, overcoming the limitations of existing approaches. Experimental results show that the CDA model outperforms traditional methods, achieving state-of-the-art accuracy in canine cardiomegaly detection. The code implementation is available at this https URL.

Title: Training-Free Motion-Guided Video Generation with Enhanced Temporal Consistency Using Motion Consistency Loss

• Authors: Xinyu Zhang, Zicheng Duan, Dong Gong, Lingqiao Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07563
• Pdf URL: https://arxiv.org/pdf/2501.07563
• Copy Paste: [[2501.07563]] Training-Free Motion-Guided Video Generation with Enhanced Temporal Consistency Using Motion Consistency Loss(https://arxiv.org/abs/2501.07563)
• Keywords: diffusion, foundation model
• Abstract: In this paper, we address the challenge of generating temporally consistent videos with motion guidance. While many existing methods depend on additional control modules or inference-time fine-tuning, recent studies suggest that effective motion guidance is achievable without altering the model architecture or requiring extra training. Such approaches offer promising compatibility with various video generation foundation models. However, existing training-free methods often struggle to maintain consistent temporal coherence across frames or to follow guided motion accurately. In this work, we propose a simple yet effective solution that combines an initial-noise-based approach with a novel motion consistency loss, the latter being our key innovation. Specifically, we capture the inter-frame feature correlation patterns of intermediate features from a video diffusion model to represent the motion pattern of the reference video. We then design a motion consistency loss to maintain similar feature correlation patterns in the generated video, using the gradient of this loss in the latent space to guide the generation process for precise motion control. This approach improves temporal consistency across various motion control tasks while preserving the benefits of a training-free setup. Extensive experiments show that our method sets a new standard for efficient, temporally coherent video generation.

Title: UnCommon Objects in 3D

• Authors: Xingchen Liu, Piyush Tayal, Jianyuan Wang, Jesus Zarzar, Tom Monnier, Konstantinos Tertikas, Jiali Duan, Antoine Toisoul, Jason Y. Zhang, Natalia Neverova, Andrea Vedaldi, Roman Shapovalov, David Novotny
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2501.07574
• Pdf URL: https://arxiv.org/pdf/2501.07574
• Copy Paste: [[2501.07574]] UnCommon Objects in 3D(https://arxiv.org/abs/2501.07574)
• Keywords: generative
• Abstract: We introduce Uncommon Objects in 3D (uCO3D), a new object-centric dataset for 3D deep learning and 3D generative AI. uCO3D is the largest publicly-available collection of high-resolution videos of objects with 3D annotations that ensures full-360$^{\circ}$ coverage. uCO3D is significantly more diverse than MVImgNet and CO3Dv2, covering more than 1,000 object categories. It is also of higher quality, due to extensive quality checks of both the collected videos and the 3D annotations. Similar to analogous datasets, uCO3D contains annotations for 3D camera poses, depth maps and sparse point clouds. In addition, each object is equipped with a caption and a 3D Gaussian Splat reconstruction. We train several large 3D models on MVImgNet, CO3Dv2, and uCO3D and obtain superior results using the latter, showing that uCO3D is better for learning applications.