2024-09-04

Title: DivDiff: A Conditional Diffusion Model for Diverse Human Motion Prediction

• Authors: Hua Yu, Yaqing Hou, Wenbin Pei, Qiang Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.00014
• Pdf URL: https://arxiv.org/pdf/2409.00014
• Copy Paste: [[2409.00014]] DivDiff: A Conditional Diffusion Model for Diverse Human Motion Prediction(https://arxiv.org/abs/2409.00014)
• Keywords: diffusion, generative
• Abstract: Diverse human motion prediction (HMP) aims to predict multiple plausible future motions given an observed human motion sequence. It is a challenging task due to the diversity of potential human motions while ensuring an accurate description of future human motions. Current solutions are either low-diversity or limited in expressiveness. Recent denoising diffusion models (DDPM) hold potential generative capabilities in generative tasks. However, introducing DDPM directly into diverse HMP incurs some issues. Although DDPM can increase the diversity of the potential patterns of human motions, the predicted human motions become implausible over time because of the significant noise disturbances in the forward process of DDPM. This phenomenon leads to the predicted human motions being hard to control, seriously impacting the quality of predicted motions and restricting their practical applicability in real-world scenarios. To alleviate this, we propose a novel conditional diffusion-based generative model, called DivDiff, to predict more diverse and realistic human motions. Specifically, the DivDiff employs DDPM as our backbone and incorporates Discrete Cosine Transform (DCT) and transformer mechanisms to encode the observed human motion sequence as a condition to instruct the reverse process of DDPM. More importantly, we design a diversified reinforcement sampling function (DRSF) to enforce human skeletal constraints on the predicted human motions. DRSF utilizes the acquired information from human skeletal as prior knowledge, thereby reducing significant disturbances introduced during the forward process. Extensive results received in the experiments on two widely-used datasets (Human3.6M and HumanEva-I) demonstrate that our model obtains competitive performance on both diversity and accuracy.

Title: Generative-Adversarial Networks for Low-Resource Language Data Augmentation in Machine Translation

• Authors: Linda Zeng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.00071
• Pdf URL: https://arxiv.org/pdf/2409.00071
• Copy Paste: [[2409.00071]] Generative-Adversarial Networks for Low-Resource Language Data Augmentation in Machine Translation(https://arxiv.org/abs/2409.00071)
• Keywords: generative
• Abstract: Neural Machine Translation (NMT) systems struggle when translating to and from low-resource languages, which lack large-scale data corpora for models to use for training. As manual data curation is expensive and time-consuming, we propose utilizing a generative-adversarial network (GAN) to augment low-resource language data. When training on a very small amount of language data (under 20,000 sentences) in a simulated low-resource setting, our model shows potential at data augmentation, generating monolingual language data with sentences such as "ask me that healthy lunch im cooking up," and "my grandfather work harder than your grandfather before." Our novel data augmentation approach takes the first step in investigating the capability of GANs in low-resource NMT, and our results suggest that there is promise for future extension of GANs to low-resource NMT.

Title: Towards Human-Level Understanding of Complex Process Engineering Schematics: A Pedagogical, Introspective Multi-Agent Framework for Open-Domain Question Answering

• Authors: Sagar Srinivas Sakhinana, Geethan Sannidhi, Venkataramana Runkana
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.00082
• Pdf URL: https://arxiv.org/pdf/2409.00082
• Copy Paste: [[2409.00082]] Towards Human-Level Understanding of Complex Process Engineering Schematics: A Pedagogical, Introspective Multi-Agent Framework for Open-Domain Question Answering(https://arxiv.org/abs/2409.00082)
• Keywords: generative
• Abstract: In the chemical and process industries, Process Flow Diagrams (PFDs) and Piping and Instrumentation Diagrams (P&IDs) are critical for design, construction, and maintenance. Recent advancements in Generative AI, such as Large Multimodal Models (LMMs) like GPT4 (Omni), have shown promise in understanding and interpreting process diagrams for Visual Question Answering (VQA). However, proprietary models pose data privacy risks, and their computational complexity prevents knowledge editing for domain-specific customization on consumer hardware. To overcome these challenges, we propose a secure, on-premises enterprise solution using a hierarchical, multi-agent Retrieval Augmented Generation (RAG) framework for open-domain question answering (ODQA) tasks, offering enhanced data privacy, explainability, and cost-effectiveness. Our novel multi-agent framework employs introspective and specialized sub-agents using open-source, small-scale multimodal models with the ReAct (Reason+Act) prompting technique for PFD and P&ID analysis, integrating multiple information sources to provide accurate and contextually relevant answers. Our approach, supported by iterative self-correction, aims to deliver superior performance in ODQA tasks. We conducted rigorous experimental studies, and the empirical results validated the proposed approach effectiveness.

Title: Genetic Approach to Mitigate Hallucination in Generative IR

• Authors: Hrishikesh Kulkarni, Nazli Goharian, Ophir Frieder, Sean MacAvaney
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.00085
• Pdf URL: https://arxiv.org/pdf/2409.00085
• Copy Paste: [[2409.00085]] Genetic Approach to Mitigate Hallucination in Generative IR(https://arxiv.org/abs/2409.00085)
• Keywords: generative
• Abstract: Generative language models hallucinate. That is, at times, they generate factually flawed responses. These inaccuracies are particularly insidious because the responses are fluent and well-articulated. We focus on the task of Grounded Answer Generation (part of Generative IR), which aims to produce direct answers to a user's question based on results retrieved from a search engine. We address hallucination by adapting an existing genetic generation approach with a new 'balanced fitness function' consisting of a cross-encoder model for relevance and an n-gram overlap metric to promote grounding. Our balanced fitness function approach quadruples the grounded answer generation accuracy while maintaining high relevance.

Title: A Generative Adversarial Network-based Method for LiDAR-Assisted Radar Image Enhancement

• Authors: Thakshila Thilakanayake, Oscar De Silva, Thumeera R. Wanasinghe, George K. Mann, Awantha Jayasiri
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.00196
• Pdf URL: https://arxiv.org/pdf/2409.00196
• Copy Paste: [[2409.00196]] A Generative Adversarial Network-based Method for LiDAR-Assisted Radar Image Enhancement(https://arxiv.org/abs/2409.00196)
• Keywords: generative
• Abstract: This paper presents a generative adversarial network (GAN) based approach for radar image enhancement. Although radar sensors remain robust for operations under adverse weather conditions, their application in autonomous vehicles (AVs) is commonly limited by the low-resolution data they produce. The primary goal of this study is to enhance the radar images to better depict the details and features of the environment, thereby facilitating more accurate object identification in AVs. The proposed method utilizes high-resolution, two-dimensional (2D) projected light detection and ranging (LiDAR) point clouds as ground truth images and low-resolution radar images as inputs to train the GAN. The ground truth images were obtained through two main steps. First, a LiDAR point cloud map was generated by accumulating raw LiDAR scans. Then, a customized LiDAR point cloud cropping and projection method was employed to obtain 2D projected LiDAR point clouds. The inference process of the proposed method relies solely on radar images to generate an enhanced version of them. The effectiveness of the proposed method is demonstrated through both qualitative and quantitative results. These results show that the proposed method can generate enhanced images with clearer object representation compared to the input radar images, even under adverse weather conditions.

Title: ProGRes: Prompted Generative Rescoring on ASR n-Best

• Authors: Ada Defne Tur, Adel Moumen, Mirco Ravanelli
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.00217
• Pdf URL: https://arxiv.org/pdf/2409.00217
• Copy Paste: [[2409.00217]] ProGRes: Prompted Generative Rescoring on ASR n-Best(https://arxiv.org/abs/2409.00217)
• Keywords: generative
• Abstract: Large Language Models (LLMs) have shown their ability to improve the performance of speech recognizers by effectively rescoring the n-best hypotheses generated during the beam search process. However, the best way to exploit recent generative instruction-tuned LLMs for hypothesis rescoring is still unclear. This paper proposes a novel method that uses instruction-tuned LLMs to dynamically expand the n-best speech recognition hypotheses with new hypotheses generated through appropriately-prompted LLMs. Specifically, we introduce a new zero-shot method for ASR n-best rescoring, which combines confidence scores, LLM sequence scoring, and prompt-based hypothesis generation. We compare Llama-3-Instruct, GPT-3.5 Turbo, and GPT-4 Turbo as prompt-based generators with Llama-3 as sequence scorer LLM. We evaluated our approach using different speech recognizers and observed significant relative improvement in the word error rate (WER) ranging from 5% to 25%.

Title: Spatially-Aware Diffusion Models with Cross-Attention for Global Field Reconstruction with Sparse Observations

• Authors: Yilin Zhuang, Sibo Cheng, Karthik Duraisamy
• Subjects: cs.LG, cs.AI, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2409.00230
• Pdf URL: https://arxiv.org/pdf/2409.00230
• Copy Paste: [[2409.00230]] Spatially-Aware Diffusion Models with Cross-Attention for Global Field Reconstruction with Sparse Observations(https://arxiv.org/abs/2409.00230)
• Keywords: diffusion
• Abstract: Diffusion models have gained attention for their ability to represent complex distributions and incorporate uncertainty, making them ideal for robust predictions in the presence of noisy or incomplete data. In this study, we develop and enhance score-based diffusion models in field reconstruction tasks, where the goal is to estimate complete spatial fields from partial observations. We introduce a condition encoding approach to construct a tractable mapping mapping between observed and unobserved regions using a learnable integration of sparse observations and interpolated fields as an inductive bias. With refined sensing representations and an unraveled temporal dimension, our method can handle arbitrary moving sensors and effectively reconstruct fields. Furthermore, we conduct a comprehensive benchmark of our approach against a deterministic interpolation-based method across various static and time-dependent PDEs. Our study attempts to addresses the gap in strong baselines for evaluating performance across varying sampling hyperparameters, noise levels, and conditioning methods. Our results show that diffusion models with cross-attention and the proposed conditional encoding generally outperform other methods under noisy conditions, although the deterministic method excels with noiseless data. Additionally, both the diffusion models and the deterministic method surpass the numerical approach in accuracy and computational cost for the steady problem. We also demonstrate the ability of the model to capture possible reconstructions and improve the accuracy of fused results in covariance-based correction tasks using ensemble sampling.

Title: Self-Supervised Learning for Building Robust Pediatric Chest X-ray Classification Models

• Authors: Sheng Cheng, Zbigniew A. Starosolski, Devika Subramanian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00231
• Pdf URL: https://arxiv.org/pdf/2409.00231
• Copy Paste: [[2409.00231]] Self-Supervised Learning for Building Robust Pediatric Chest X-ray Classification Models(https://arxiv.org/abs/2409.00231)
• Keywords: self-supervised
• Abstract: Recent advancements in deep learning for Medical Artificial Intelligence have demonstrated that models can match the diagnostic performance of clinical experts in adult chest X-ray (CXR) interpretation. However, their application in the pediatric context remains limited due to the scarcity of large annotated pediatric image datasets. Additionally, significant challenges arise from the substantial variability in pediatric CXR images across different hospitals and the diverse age range of patients from 0 to 18 years. To address these challenges, we propose SCC, a novel approach that combines transfer learning with self-supervised contrastive learning, augmented by an unsupervised contrast enhancement technique. Transfer learning from a well-trained adult CXR model mitigates issues related to the scarcity of pediatric training data. Contrastive learning with contrast enhancement focuses on the lungs, reducing the impact of image variations and producing high-quality embeddings across diverse pediatric CXR images. We train SCC on one pediatric CXR dataset and evaluate its performance on two other pediatric datasets from different sources. Our results show that SCC's out-of-distribution (zero-shot) performance exceeds regular transfer learning in terms of AUC by 13.6% and 34.6% on the two test datasets. Moreover, with few-shot learning using 10 times fewer labeled images, SCC matches the performance of regular transfer learning trained on the entire labeled dataset. To test the generality of the framework, we verify its performance on three benchmark breast cancer datasets. Starting from a model trained on natural images and fine-tuned on one breast dataset, SCC outperforms the fully supervised learning baseline on the other two datasets in terms of AUC by 3.6% and 5.5% in zero-shot learning.

Title: AWRaCLe: All-Weather Image Restoration using Visual In-Context Learning

• Authors: Sudarshan Rajagopalan, Vishal M. Patel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00263
• Pdf URL: https://arxiv.org/pdf/2409.00263
• Copy Paste: [[2409.00263]] AWRaCLe: All-Weather Image Restoration using Visual In-Context Learning(https://arxiv.org/abs/2409.00263)
• Keywords: in-context
• Abstract: All-Weather Image Restoration (AWIR) under adverse weather conditions is a challenging task due to the presence of different types of degradations. Prior research in this domain relies on extensive training data but lacks the utilization of additional contextual information for restoration guidance. Consequently, the performance of existing methods is limited by the degradation cues that are learnt from individual training samples. Recent advancements in visual in-context learning have introduced generalist models that are capable of addressing multiple computer vision tasks simultaneously by using the information present in the provided context as a prior. In this paper, we propose All-Weather Image Restoration using Visual In-Context Learning (AWRaCLe), a novel approach for AWIR that innovatively utilizes degradation-specific visual context information to steer the image restoration process. To achieve this, AWRaCLe incorporates Degradation Context Extraction (DCE) and Context Fusion (CF) to seamlessly integrate degradation-specific features from the context into an image restoration network. The proposed DCE and CF blocks leverage CLIP features and incorporate attention mechanisms to adeptly learn and fuse contextual information. These blocks are specifically designed for visual in-context learning under all-weather conditions and are crucial for effective context utilization. Through extensive experiments, we demonstrate the effectiveness of AWRaCLe for all-weather restoration and show that our method advances the state-of-the-art in AWIR.

Title: Training-Free Sketch-Guided Diffusion with Latent Optimization

• Authors: Sandra Zhang Ding, Jiafeng Mao, Kiyoharu Aizawa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00313
• Pdf URL: https://arxiv.org/pdf/2409.00313
• Copy Paste: [[2409.00313]] Training-Free Sketch-Guided Diffusion with Latent Optimization(https://arxiv.org/abs/2409.00313)
• Keywords: diffusion
• Abstract: Based on recent advanced diffusion models, Text-to-image (T2I) generation models have demonstrated their capabilities in generating diverse and high-quality images. However, leveraging their potential for real-world content creation, particularly in providing users with precise control over the image generation result, poses a significant challenge. In this paper, we propose an innovative training-free pipeline that extends existing text-to-image generation models to incorporate a sketch as an additional condition. To generate new images with a layout and structure closely resembling the input sketch, we find that these core features of a sketch can be tracked with the cross-attention maps of diffusion models. We introduce latent optimization, a method that refines the noisy latent at each intermediate step of the generation process using cross-attention maps to ensure that the generated images closely adhere to the desired structure outlined in the reference sketch. Through latent optimization, our method enhances the fidelity and accuracy of image generation, offering users greater control and customization options in content creation.

Title: Evaluating the Effectiveness of Large Language Models in Representing and Understanding Movement Trajectories

• Authors: Yuhan Ji, Song Gao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.00335
• Pdf URL: https://arxiv.org/pdf/2409.00335
• Copy Paste: [[2409.00335]] Evaluating the Effectiveness of Large Language Models in Representing and Understanding Movement Trajectories(https://arxiv.org/abs/2409.00335)
• Keywords: foundation model
• Abstract: This research focuses on assessing the ability of AI foundation models in representing the trajectories of movements. We utilize one of the large language models (LLMs) (i.e., GPT-J) to encode the string format of trajectories and then evaluate the effectiveness of the LLM-based representation for trajectory data analysis. The experiments demonstrate that while the LLM-based embeddings can preserve certain trajectory distance metrics (i.e., the correlation coefficients exceed 0.74 between the Cosine distance derived from GPT-J embeddings and the Hausdorff and Dynamic Time Warping distances on raw trajectories), challenges remain in restoring numeric values and retrieving spatial neighbors in movement trajectory analytics. In addition, the LLMs can understand the spatiotemporal dependency contained in trajectories and have good accuracy in location prediction tasks. This research highlights the need for improvement in terms of capturing the nuances and complexities of the underlying geospatial data and integrating domain knowledge to support various GeoAI applications using LLMs.

Title: LightPure: Realtime Adversarial Image Purification for Mobile Devices Using Diffusion Models

• Authors: Hossein Khalili, Seongbin Park, Vincent Li, Brandan Bright, Ali Payani, Ramana Rao Kompella, Nader Sehatbakhsh
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00340
• Pdf URL: https://arxiv.org/pdf/2409.00340
• Copy Paste: [[2409.00340]] LightPure: Realtime Adversarial Image Purification for Mobile Devices Using Diffusion Models(https://arxiv.org/abs/2409.00340)
• Keywords: diffusion, generative
• Abstract: Autonomous mobile systems increasingly rely on deep neural networks for perception and decision-making. While effective, these systems are vulnerable to adversarial machine learning attacks where minor input perturbations can significantly impact outcomes. Common countermeasures involve adversarial training and/or data or network transformation. These methods, though effective, require full access to typically proprietary classifiers and are costly for large models. Recent solutions propose purification models, which add a "purification" layer before classification, eliminating the need to modify the classifier directly. Despite their effectiveness, these methods are compute-intensive, making them unsuitable for mobile systems where resources are limited and low latency is essential. This paper introduces LightPure, a new method that enhances adversarial image purification. It improves the accuracy of existing purification methods and provides notable enhancements in speed and computational efficiency, making it suitable for mobile devices with limited resources. Our approach uses a two-step diffusion and one-shot Generative Adversarial Network (GAN) framework, prioritizing latency without compromising robustness. We propose several new techniques to achieve a reasonable balance between classification accuracy and adversarial robustness while maintaining desired latency. We design and implement a proof-of-concept on a Jetson Nano board and evaluate our method using various attack scenarios and datasets. Our results show that LightPure can outperform existing methods by up to 10x in terms of latency while achieving higher accuracy and robustness for various attack scenarios. This method offers a scalable and effective solution for real-world mobile systems.

Title: RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learning

• Authors: Kunming Su, Qiuxia Wu, Panpan Cai, Xiaogang Zhu, Xuequan Lu, Zhiyong Wang, Kun Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00353
• Pdf URL: https://arxiv.org/pdf/2409.00353
• Copy Paste: [[2409.00353]] RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learning(https://arxiv.org/abs/2409.00353)
• Keywords: self-supervised
• Abstract: Masked point modeling methods have recently achieved great success in self-supervised learning for point cloud data. However, these methods are sensitive to rotations and often exhibit sharp performance drops when encountering rotational variations. In this paper, we propose a novel Rotation-Invariant Masked AutoEncoders (RI-MAE) to address two major challenges: 1) achieving rotation-invariant latent representations, and 2) facilitating self-supervised reconstruction in a rotation-invariant manner. For the first challenge, we introduce RI-Transformer, which features disentangled geometry content, rotation-invariant relative orientation and position embedding mechanisms for constructing rotation-invariant point cloud latent space. For the second challenge, a novel dual-branch student-teacher architecture is devised. It enables the self-supervised learning via the reconstruction of masked patches within the learned rotation-invariant latent space. Each branch is based on an RI-Transformer, and they are connected with an additional RI-Transformer predictor. The teacher encodes all point patches, while the student solely encodes unmasked ones. Finally, the predictor predicts the latent features of the masked patches using the output latent embeddings from the student, supervised by the outputs from the teacher. Extensive experiments demonstrate that our method is robust to rotations, achieving the state-of-the-art performance on various downstream tasks.

Title: Towards understanding Diffusion Models (on Graphs)

• Authors: Solveig Klepper
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.00374
• Pdf URL: https://arxiv.org/pdf/2409.00374
• Copy Paste: [[2409.00374]] Towards understanding Diffusion Models (on Graphs)(https://arxiv.org/abs/2409.00374)
• Keywords: diffusion
• Abstract: Diffusion models have emerged from various theoretical and methodological perspectives, each offering unique insights into their underlying principles. In this work, we provide an overview of the most prominent approaches, drawing attention to their striking analogies -- namely, how seemingly diverse methodologies converge to a similar mathematical formulation of the core problem. While our ultimate goal is to understand these models in the context of graphs, we begin by conducting experiments in a simpler setting to build foundational insights. Through an empirical investigation of different diffusion and sampling techniques, we explore three critical questions: (1) What role does noise play in these models? (2) How significantly does the choice of the sampling method affect outcomes? (3) What function is the neural network approximating, and is high complexity necessary for optimal performance? Our findings aim to enhance the understanding of diffusion models and in the long run their application in graph machine learning.

Title: Self-supervised Fusarium Head Blight Detection with Hyperspectral Image and Feature Mining

• Authors: Yu-Fan Lin, Ching-Heng Cheng, Bo-Cheng Qiu, Cheng-Jun Kang, Chia-Ming Lee, Chih-Chung Hsu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.00395
• Pdf URL: https://arxiv.org/pdf/2409.00395
• Copy Paste: [[2409.00395]] Self-supervised Fusarium Head Blight Detection with Hyperspectral Image and Feature Mining(https://arxiv.org/abs/2409.00395)
• Keywords: self-supervised
• Abstract: Fusarium Head Blight (FHB) is a serious fungal disease affecting wheat (including durum), barley, oats, other small cereal grains, and corn. Effective monitoring and accurate detection of FHB are crucial to ensuring stable and reliable food security. Traditionally, trained agronomists and surveyors perform manual identification, a method that is labor-intensive, impractical, and challenging to scale. With the advancement of deep learning and Hyper-spectral Imaging (HSI) and Remote Sensing (RS) technologies, employing deep learning, particularly Convolutional Neural Networks (CNNs), has emerged as a promising solution. Notably, wheat infected with serious FHB may exhibit significant differences on the spectral compared to mild FHB one, which is particularly advantageous for hyperspectral image-based methods. In this study, we propose a self-unsupervised classification method based on HSI endmember extraction strategy and top-K bands selection, designed to analyze material signatures in HSIs to derive discriminative feature representations. This approach does not require expensive device or complicate algorithm design, making it more suitable for practical uses. Our method has been effectively validated in the Beyond Visible Spectrum: AI for Agriculture Challenge 2024. The source code is easy to reproduce and available at {this https URL}.

Title: COSMo: CLIP Talks on Open-Set Multi-Target Domain Adaptation

• Authors: Munish Monga, Sachin Kumar Giroh, Ankit Jha, Mainak Singha, Biplab Banerjee, Jocelyn Chanussot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00397
• Pdf URL: https://arxiv.org/pdf/2409.00397
• Copy Paste: [[2409.00397]] COSMo: CLIP Talks on Open-Set Multi-Target Domain Adaptation(https://arxiv.org/abs/2409.00397)
• Keywords: foundation model
• Abstract: Multi-Target Domain Adaptation (MTDA) entails learning domain-invariant information from a single source domain and applying it to multiple unlabeled target domains. Yet, existing MTDA methods predominantly focus on addressing domain shifts within visual features, often overlooking semantic features and struggling to handle unknown classes, resulting in what is known as Open-Set (OS) MTDA. While large-scale vision-language foundation models like CLIP show promise, their potential for MTDA remains largely unexplored. This paper introduces COSMo, a novel method that learns domain-agnostic prompts through source domain-guided prompt learning to tackle the MTDA problem in the prompt space. By leveraging a domain-specific bias network and separate prompts for known and unknown classes, COSMo effectively adapts across domain and class shifts. To the best of our knowledge, COSMo is the first method to address Open-Set Multi-Target DA (OSMTDA), offering a more realistic representation of real-world scenarios and addressing the challenges of both open-set and multi-target DA. COSMo demonstrates an average improvement of $5.1\%$ across three challenging datasets: Mini-DomainNet, Office-31, and Office-Home, compared to other related DA methods adapted to operate within the OSMTDA setting. Code is available at: this https URL

Title: Geospatial foundation models for image analysis: evaluating and enhancing NASA-IBM Prithvi's domain adaptability

• Authors: Chia-Yu Hsu, Wenwen Li, Sizhe Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.00489
• Pdf URL: https://arxiv.org/pdf/2409.00489
• Copy Paste: [[2409.00489]] Geospatial foundation models for image analysis: evaluating and enhancing NASA-IBM Prithvi's domain adaptability(https://arxiv.org/abs/2409.00489)
• Keywords: foundation model
• Abstract: Research on geospatial foundation models (GFMs) has become a trending topic in geospatial artificial intelligence (AI) research due to their potential for achieving high generalizability and domain adaptability, reducing model training costs for individual researchers. Unlike large language models, such as ChatGPT, constructing visual foundation models for image analysis, particularly in remote sensing, encountered significant challenges such as formulating diverse vision tasks into a general problem framework. This paper evaluates the recently released NASA-IBM GFM Prithvi for its predictive performance on high-level image analysis tasks across multiple benchmark datasets. Prithvi was selected because it is one of the first open-source GFMs trained on time-series of high-resolution remote sensing imagery. A series of experiments were designed to assess Prithvi's performance as compared to other pre-trained task-specific AI models in geospatial image analysis. New strategies, including band adaptation, multi-scale feature generation, and fine-tuning techniques, are introduced and integrated into an image analysis pipeline to enhance Prithvi's domain adaptation capability and improve model performance. In-depth analyses reveal Prithvi's strengths and weaknesses, offering insights for both improving Prithvi and developing future visual foundation models for geospatial tasks.

Title: Accurate Compression of Text-to-Image Diffusion Models via Vector Quantization

• Authors: Vage Egiazarian, Denis Kuznedelev, Anton Voronov, Ruslan Svirschevski, Michael Goin, Daniil Pavlov, Dan Alistarh, Dmitry Baranchuk
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00492
• Pdf URL: https://arxiv.org/pdf/2409.00492
• Copy Paste: [[2409.00492]] Accurate Compression of Text-to-Image Diffusion Models via Vector Quantization(https://arxiv.org/abs/2409.00492)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models have emerged as a powerful framework for high-quality image generation given textual prompts. Their success has driven the rapid development of production-grade diffusion models that consistently increase in size and already contain billions of parameters. As a result, state-of-the-art text-to-image models are becoming less accessible in practice, especially in resource-limited environments. Post-training quantization (PTQ) tackles this issue by compressing the pretrained model weights into lower-bit representations. Recent diffusion quantization techniques primarily rely on uniform scalar quantization, providing decent performance for the models compressed to 4 bits. This work demonstrates that more versatile vector quantization (VQ) may achieve higher compression rates for large-scale text-to-image diffusion models. Specifically, we tailor vector-based PTQ methods to recent billion-scale text-to-image models (SDXL and SDXL-Turbo), and show that the diffusion models of 2B+ parameters compressed to around 3 bits using VQ exhibit the similar image quality and textual alignment as previous 4-bit compression techniques.

Title: RevCD -- Reversed Conditional Diffusion for Generalized Zero-Shot Learning

• Authors: William Heyden, Habib Ullah, M. Salman Siddiqui, Fadi Al Machot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00511
• Pdf URL: https://arxiv.org/pdf/2409.00511
• Copy Paste: [[2409.00511]] RevCD -- Reversed Conditional Diffusion for Generalized Zero-Shot Learning(https://arxiv.org/abs/2409.00511)
• Keywords: diffusion
• Abstract: In Generalized Zero-Shot Learning (GZSL), we aim to recognize both seen and unseen categories using a model trained only on seen categories. In computer vision, this translates into a classification problem, where knowledge from seen categories is transferred to unseen categories by exploiting the relationships between visual features and available semantic information, such as text corpora or manual annotations. However, learning this joint distribution is costly and requires one-to-one training with corresponding semantic information. We present a reversed conditional Diffusion-based model (RevCD) that mitigates this issue by generating semantic features synthesized from visual inputs by leveraging Diffusion models' conditional mechanisms. Our RevCD model consists of a cross Hadamard-Addition embedding of a sinusoidal time schedule and a multi-headed visual transformer for attention-guided embeddings. The proposed approach introduces three key innovations. First, we reverse the process of generating semantic space based on visual data, introducing a novel loss function that facilitates more efficient knowledge transfer. Second, we apply Diffusion models to zero-shot learning - a novel approach that exploits their strengths in capturing data complexity. Third, we demonstrate our model's performance through a comprehensive cross-dataset evaluation. The complete code will be available on GitHub.

Title: EraseDraw: Learning to Insert Objects by Erasing Them from Images

• Authors: Alper Canberk, Maksym Bondarenko, Ege Ozguroglu, Ruoshi Liu, Carl Vondrick
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00522
• Pdf URL: https://arxiv.org/pdf/2409.00522
• Copy Paste: [[2409.00522]] EraseDraw: Learning to Insert Objects by Erasing Them from Images(https://arxiv.org/abs/2409.00522)
• Keywords: diffusion
• Abstract: Creative processes such as painting often involve creating different components of an image one by one. Can we build a computational model to perform this task? Prior works often fail by making global changes to the image, inserting objects in unrealistic spatial locations, and generating inaccurate lighting details. We observe that while state-of-the-art models perform poorly on object insertion, they can remove objects and erase the background in natural images very well. Inverting the direction of object removal, we obtain high-quality data for learning to insert objects that are spatially, physically, and optically consistent with the surroundings. With this scalable automatic data generation pipeline, we can create a dataset for learning object insertion, which is used to train our proposed text conditioned diffusion model. Qualitative and quantitative experiments have shown that our model achieves state-of-the-art results in object insertion, particularly for in-the-wild images. We show compelling results on diverse insertion prompts and images across various this http URL addition, we automate iterative insertion by combining our insertion model with beam search guided by CLIP.

Title: Incremental Open-set Domain Adaptation

• Authors: Sayan Rakshit, Hmrishav Bandyopadhyay, Nibaran Das, Biplab Banerjee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00530
• Pdf URL: https://arxiv.org/pdf/2409.00530
• Copy Paste: [[2409.00530]] Incremental Open-set Domain Adaptation(https://arxiv.org/abs/2409.00530)
• Keywords: generative
• Abstract: Catastrophic forgetting makes neural network models unstable when learning visual domains consecutively. The neural network model drifts to catastrophic forgetting-induced low performance of previously learnt domains when training with new domains. We illuminate this current neural network model weakness and develop a forgetting-resistant incremental learning strategy. Here, we propose a new unsupervised incremental open-set domain adaptation (IOSDA) issue for image classification. Open-set domain adaptation adds complexity to the incremental domain adaptation issue since each target domain has more classes than the Source domain. In IOSDA, the model learns training with domain streams phase by phase in incremented time. Inference uses test data from all target domains without revealing their identities. We proposed IOSDA-Net, a two-stage learning pipeline, to solve the problem. The first module replicates prior domains from random noise using a generative framework and creates a pseudo source domain. In the second step, this pseudo source is adapted to the present target domain. We test our model on Office-Home, DomainNet, and UPRN-RSDA, a newly curated optical remote sensing dataset.

Title: Data Augmentation for Image Classification using Generative AI

• Authors: Fazle Rahat, M Shifat Hossain, Md Rubel Ahmed, Sumit Kumar Jha, Rickard Ewetz
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.00547
• Pdf URL: https://arxiv.org/pdf/2409.00547
• Copy Paste: [[2409.00547]] Data Augmentation for Image Classification using Generative AI(https://arxiv.org/abs/2409.00547)
• Keywords: diffusion, generative
• Abstract: Scaling laws dictate that the performance of AI models is proportional to the amount of available data. Data augmentation is a promising solution to expanding the dataset size. Traditional approaches focused on augmentation using rotation, translation, and resizing. Recent approaches use generative AI models to improve dataset diversity. However, the generative methods struggle with issues such as subject corruption and the introduction of irrelevant artifacts. In this paper, we propose the Automated Generative Data Augmentation (AGA). The framework combines the utility of large language models (LLMs), diffusion models, and segmentation models to augment data. AGA preserves foreground authenticity while ensuring background diversity. Specific contributions include: i) segment and superclass based object extraction, ii) prompt diversity with combinatorial complexity using prompt decomposition, and iii) affine subject manipulation. We evaluate AGA against state-of-the-art (SOTA) techniques on three representative datasets, ImageNet, CUB, and iWildCam. The experimental evaluation demonstrates an accuracy improvement of 15.6% and 23.5% for in and out-of-distribution data compared to baseline models, respectively. There is also a 64.3% improvement in SIC score compared to the baselines.

Title: FADE: Few-shot/zero-shot Anomaly Detection Engine using Large Vision-Language Model

• Authors: Yuanwei Li, Elizaveta Ivanova, Martins Bruveris
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00556
• Pdf URL: https://arxiv.org/pdf/2409.00556
• Copy Paste: [[2409.00556]] FADE: Few-shot/zero-shot Anomaly Detection Engine using Large Vision-Language Model(https://arxiv.org/abs/2409.00556)
• Keywords: anomaly
• Abstract: Automatic image anomaly detection is important for quality inspection in the manufacturing industry. The usual unsupervised anomaly detection approach is to train a model for each object class using a dataset of normal samples. However, a more realistic problem is zero-/few-shot anomaly detection where zero or only a few normal samples are available. This makes the training of object-specific models challenging. Recently, large foundation vision-language models have shown strong zero-shot performance in various downstream tasks. While these models have learned complex relationships between vision and language, they are not specifically designed for the tasks of anomaly detection. In this paper, we propose the Few-shot/zero-shot Anomaly Detection Engine (FADE) which leverages the vision-language CLIP model and adjusts it for the purpose of industrial anomaly detection. Specifically, we improve language-guided anomaly segmentation 1) by adapting CLIP to extract multi-scale image patch embeddings that are better aligned with language and 2) by automatically generating an ensemble of text prompts related to industrial anomaly detection. 3) We use additional vision-based guidance from the query and reference images to further improve both zero-shot and few-shot anomaly detection. On the MVTec-AD (and VisA) dataset, FADE outperforms other state-of-the-art methods in anomaly segmentation with pixel-AUROC of 89.6% (91.5%) in zero-shot and 95.4% (97.5%) in 1-normal-shot. Code is available at this https URL.

Title: Compositional 3D-aware Video Generation with LLM Director

• Authors: Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, Jiang Bian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00558
• Pdf URL: https://arxiv.org/pdf/2409.00558
• Copy Paste: [[2409.00558]] Compositional 3D-aware Video Generation with LLM Director(https://arxiv.org/abs/2409.00558)
• Keywords: diffusion, generative
• Abstract: Significant progress has been made in text-to-video generation through the use of powerful generative models and large-scale internet data. However, substantial challenges remain in precisely controlling individual concepts within the generated video, such as the motion and appearance of specific characters and the movement of viewpoints. In this work, we propose a novel paradigm that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models. Specifically, given an input textual prompt, our scheme consists of three stages: 1) We leverage LLM as the director to first decompose the complex query into several sub-prompts that indicate individual concepts within the video~(\textit{e.g.}, scene, objects, motions), then we let LLM to invoke pre-trained expert models to obtain corresponding 3D representations of concepts. 2) To compose these representations, we prompt multi-modal LLM to produce coarse guidance on the scales and coordinates of trajectories for the objects. 3) To make the generated frames adhere to natural image distribution, we further leverage 2D diffusion priors and use Score Distillation Sampling to refine the composition. Extensive experiments demonstrate that our method can generate high-fidelity videos from text with diverse motion and flexible control over each concept. Project page: \url{this https URL}.

Title: McCaD: Multi-Contrast MRI Conditioned, Adaptive Adversarial Diffusion Model for High-Fidelity MRI Synthesis

• Authors: Sanuwani Dayarathna, Kh Tohidul Islam, Bohan Zhuang, Guang Yang, Jianfei Cai, Meng Law, Zhaolin Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00585
• Pdf URL: https://arxiv.org/pdf/2409.00585
• Copy Paste: [[2409.00585]] McCaD: Multi-Contrast MRI Conditioned, Adaptive Adversarial Diffusion Model for High-Fidelity MRI Synthesis(https://arxiv.org/abs/2409.00585)
• Keywords: diffusion
• Abstract: Magnetic Resonance Imaging (MRI) is instrumental in clinical diagnosis, offering diverse contrasts that provide comprehensive diagnostic information. However, acquiring multiple MRI contrasts is often constrained by high costs, long scanning durations, and patient discomfort. Current synthesis methods, typically focused on single-image contrasts, fall short in capturing the collective nuances across various contrasts. Moreover, existing methods for multi-contrast MRI synthesis often fail to accurately map feature-level information across multiple imaging contrasts. We introduce McCaD (Multi-Contrast MRI Conditioned Adaptive Adversarial Diffusion), a novel framework leveraging an adversarial diffusion model conditioned on multiple contrasts for high-fidelity MRI synthesis. McCaD significantly enhances synthesis accuracy by employing a multi-scale, feature-guided mechanism, incorporating denoising and semantic encoders. An adaptive feature maximization strategy and a spatial feature-attentive loss have been introduced to capture more intrinsic features across multiple contrasts. This facilitates a precise and comprehensive feature-guided denoising process. Extensive experiments on tumor and healthy multi-contrast MRI datasets demonstrated that the McCaD outperforms state-of-the-art baselines quantitively and qualitatively. The code is provided with supplementary materials.

Title: Seed-to-Seed: Image Translation in Diffusion Seed Space

• Authors: Or Greenberg, Eran Kishon, Dani Lischinski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00654
• Pdf URL: https://arxiv.org/pdf/2409.00654
• Copy Paste: [[2409.00654]] Seed-to-Seed: Image Translation in Diffusion Seed Space(https://arxiv.org/abs/2409.00654)
• Keywords: diffusion
• Abstract: We introduce Seed-to-Seed Translation (StS), a novel approach for Image-to-Image Translation using diffusion models (DMs), aimed at translations that require close adherence to the structure of the source image. In contrast to existing methods that modify images during the diffusion sampling process, we leverage the semantic information encoded within the space of inverted seeds of a pretrained DM, dubbed as the seed-space. We demonstrate that inverted seeds can be used for discriminative tasks, and can also be manipulated to achieve desired transformations in an unpaired image-to-image translation setting. Our method involves training an sts-GAN, an unpaired translation model between source and target seeds, based on CycleGAN. The final translated images are obtained by initiating the DM's sampling process from the translated seeds. A ControlNet is used to ensure the structural preservation of the input image. We demonstrate the effectiveness of our approach for the task of translating automotive scenes, showcasing superior performance compared to existing GAN-based and diffusion-based methods, as well as for several other unpaired image translation tasks. Our approach offers a fresh perspective on leveraging the semantic information encoded within the seed-space of pretrained DMs for effective image editing and manipulation.

Title: Comprehensive Botnet Detection by Mitigating Adversarial Attacks, Navigating the Subtleties of Perturbation Distances and Fortifying Predictions with Conformal Layers

• Authors: Rahul Yumlembam, Biju Issac, Seibu Mary Jacob, Longzhi Yang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.00667
• Pdf URL: https://arxiv.org/pdf/2409.00667
• Copy Paste: [[2409.00667]] Comprehensive Botnet Detection by Mitigating Adversarial Attacks, Navigating the Subtleties of Perturbation Distances and Fortifying Predictions with Conformal Layers(https://arxiv.org/abs/2409.00667)
• Keywords: generative
• Abstract: Botnets are computer networks controlled by malicious actors that present significant cybersecurity challenges. They autonomously infect, propagate, and coordinate to conduct cybercrimes, necessitating robust detection methods. This research addresses the sophisticated adversarial manipulations posed by attackers, aiming to undermine machine learning-based botnet detection systems. We introduce a flow-based detection approach, leveraging machine learning and deep learning algorithms trained on the ISCX and ISOT datasets. The detection algorithms are optimized using the Genetic Algorithm and Particle Swarm Optimization to obtain a baseline detection method. The Carlini & Wagner (C&W) attack and Generative Adversarial Network (GAN) generate deceptive data with subtle perturbations, targeting each feature used for classification while preserving their semantic and syntactic relationships, which ensures that the adversarial samples retain meaningfulness and realism. An in-depth analysis of the required L2 distance from the original sample for the malware sample to misclassify is performed across various iteration checkpoints, showing different levels of misclassification at different L2 distances of the Pertrub sample from the original sample. Our work delves into the vulnerability of various models, examining the transferability of adversarial examples from a Neural Network surrogate model to Tree-based algorithms. Subsequently, models that initially misclassified the perturbed samples are retrained, enhancing their resilience and detection capabilities. In the final phase, a conformal prediction layer is integrated, significantly rejecting incorrect predictions, of 58.20 % in the ISCX dataset and 98.94 % in the ISOT dataset.

Title: Curriculum Prompting Foundation Models for Medical Image Segmentation

• Authors: Xiuqi Zheng, Yuhang Zhang, Haoran Zhang, Hongrui Liang, Xueqi Bao, Zhuqing Jiang, Qicheng Lao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.00695
• Pdf URL: https://arxiv.org/pdf/2409.00695
• Copy Paste: [[2409.00695]] Curriculum Prompting Foundation Models for Medical Image Segmentation(https://arxiv.org/abs/2409.00695)
• Keywords: foundation model
• Abstract: Adapting large pre-trained foundation models, e.g., SAM, for medical image segmentation remains a significant challenge. A crucial step involves the formulation of a series of specialized prompts that incorporate specific clinical instructions. Past works have been heavily reliant on a singular type of prompt for each instance, necessitating manual input of an ideally correct prompt, which is less efficient. To tackle this issue, we propose to utilize prompts of different granularity, which are sourced from original images to provide a broader scope of clinical insights. However, combining prompts of varying types can pose a challenge due to potential conflicts. In response, we have designed a coarse-to-fine mechanism, referred to as curriculum prompting, that progressively integrates prompts of different types. Through extensive experiments on three public medical datasets across various modalities, we demonstrate the effectiveness of our proposed approach, which not only automates the prompt generation process but also yields superior performance compared to other SAM-based medical image segmentation methods. Code is available at: this https URL.

Title: ReMOVE: A Reference-free Metric for Object Erasure

• Authors: Aditya Chandrasekar, Goirik Chakrabarty, Jai Bardhan, Ramya Hebbalaguppe, Prathosh AP
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.00707
• Pdf URL: https://arxiv.org/pdf/2409.00707
• Copy Paste: [[2409.00707]] ReMOVE: A Reference-free Metric for Object Erasure(https://arxiv.org/abs/2409.00707)
• Keywords: diffusion
• Abstract: We introduce $\texttt{ReMOVE}$, a novel reference-free metric for assessing object erasure efficacy in diffusion-based image editing models post-generation. Unlike existing measures such as LPIPS and CLIPScore, $\texttt{ReMOVE}$ addresses the challenge of evaluating inpainting without a reference image, common in practical scenarios. It effectively distinguishes between object removal and replacement. This is a key issue in diffusion models due to stochastic nature of image generation. Traditional metrics fail to align with the intuitive definition of inpainting, which aims for (1) seamless object removal within masked regions (2) while preserving the background continuity. $\texttt{ReMOVE}$ not only correlates with state-of-the-art metrics and aligns with human perception but also captures the nuanced aspects of the inpainting process, providing a finer-grained evaluation of the generated outputs.

Title: LPUWF-LDM: Enhanced Latent Diffusion Model for Precise Late-phase UWF-FA Generation on Limited Dataset

• Authors: Zhaojie Fang, Xiao Yu, Guanyu Zhou, Ke Zhuang, Yifei Chen, Ruiquan Ge, Changmiao Wang, Gangyong Jia, Qing Wu, Juan Ye, Maimaiti Nuliqiman, Peifang Xu, Ahmed Elazab
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.00726
• Pdf URL: https://arxiv.org/pdf/2409.00726
• Copy Paste: [[2409.00726]] LPUWF-LDM: Enhanced Latent Diffusion Model for Precise Late-phase UWF-FA Generation on Limited Dataset(https://arxiv.org/abs/2409.00726)
• Keywords: diffusion
• Abstract: Ultra-Wide-Field Fluorescein Angiography (UWF-FA) enables precise identification of ocular diseases using sodium fluorescein, which can be potentially harmful. Existing research has developed methods to generate UWF-FA from Ultra-Wide-Field Scanning Laser Ophthalmoscopy (UWF-SLO) to reduce the adverse reactions associated with injections. However, these methods have been less effective in producing high-quality late-phase UWF-FA, particularly in lesion areas and fine details. Two primary challenges hinder the generation of high-quality late-phase UWF-FA: the scarcity of paired UWF-SLO and early/late-phase UWF-FA datasets, and the need for realistic generation at lesion sites and potential blood leakage regions. This study introduces an improved latent diffusion model framework to generate high-quality late-phase UWF-FA from limited paired UWF images. To address the challenges as mentioned earlier, our approach employs a module utilizing Cross-temporal Regional Difference Loss, which encourages the model to focus on the differences between early and late phases. Additionally, we introduce a low-frequency enhanced noise strategy in the diffusion forward process to improve the realism of medical images. To further enhance the mapping capability of the variational autoencoder module, especially with limited datasets, we implement a Gated Convolutional Encoder to extract additional information from conditional images. Our Latent Diffusion Model for Ultra-Wide-Field Late-Phase Fluorescein Angiography (LPUWF-LDM) effectively reconstructs fine details in late-phase UWF-FA and achieves state-of-the-art results compared to other existing methods when working with limited datasets. Our source code is available at: this https URL.

Title: Generating Physical Dynamics under Priors

• Authors: Zihan Zhou, Xiaoxue Wang, Tianshu Yu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.00730
• Pdf URL: https://arxiv.org/pdf/2409.00730
• Copy Paste: [[2409.00730]] Generating Physical Dynamics under Priors(https://arxiv.org/abs/2409.00730)
• Keywords: diffusion, generative
• Abstract: Generating physically feasible dynamics in a data-driven context is challenging, especially when adhering to physical priors expressed in specific equations or formulas. Existing methodologies often overlook the integration of physical priors, resulting in violation of basic physical laws and suboptimal performance. In this paper, we introduce a novel framework that seamlessly incorporates physical priors into diffusion-based generative models to address this limitation. Our approach leverages two categories of priors: 1) distributional priors, such as roto-translational invariance, and 2) physical feasibility priors, including energy and momentum conservation laws and PDE constraints. By embedding these priors into the generative process, our method can efficiently generate physically realistic dynamics, encompassing trajectories and flows. Empirical evaluations demonstrate that our method produces high-quality dynamics across a diverse array of physical phenomena with remarkable robustness, underscoring its potential to advance data-driven studies in AI4Physics. Our contributions signify a substantial advancement in the field of generative modeling, offering a robust solution to generate accurate and physically consistent dynamics.

Title: A Critical Analysis on Machine Learning Techniques for Video-based Human Activity Recognition of Surveillance Systems: A Review

• Authors: Shahriar Jahan, Roknuzzaman, Md Robiul Islam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00731
• Pdf URL: https://arxiv.org/pdf/2409.00731
• Copy Paste: [[2409.00731]] A Critical Analysis on Machine Learning Techniques for Video-based Human Activity Recognition of Surveillance Systems: A Review(https://arxiv.org/abs/2409.00731)
• Keywords: anomaly
• Abstract: Upsurging abnormal activities in crowded locations such as airports, train stations, bus stops, shopping malls, etc., urges the necessity for an intelligent surveillance system. An intelligent surveillance system can differentiate between normal and suspicious activities from real-time video analysis that will enable to take appropriate measures regarding the level of an anomaly instantaneously and efficiently. Video-based human activity recognition has intrigued many researchers with its pressing issues and a variety of applications ranging from simple hand gesture recognition to crucial behavior recognition in a surveillance system. This paper provides a critical survey of video-based Human Activity Recognition (HAR) techniques beginning with an examination of basic approaches for detecting and recognizing suspicious behavior followed by a critical analysis of machine learning and deep learning techniques such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Hidden Markov Model (HMM), K-means Clustering etc. A detailed investigation and comparison are done on these learning techniques on the basis of feature extraction techniques, parameter initialization, and optimization algorithms, accuracy, etc. The purpose of this review is to prioritize positive schemes and to assist researchers with emerging advancements in this field's future endeavors. This paper also pragmatically discusses existing challenges in the field of HAR and examines the prospects in the field.

Title: Self-Supervised Vision Transformers for Writer Retrieval

• Authors: Tim Raven, Arthur Matei, Gernot A. Fink
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.00751
• Pdf URL: https://arxiv.org/pdf/2409.00751
• Copy Paste: [[2409.00751]] Self-Supervised Vision Transformers for Writer Retrieval(https://arxiv.org/abs/2409.00751)
• Keywords: self-supervised
• Abstract: While methods based on Vision Transformers (ViT) have achieved state-of-the-art performance in many domains, they have not yet been applied successfully in the domain of writer retrieval. The field is dominated by methods using handcrafted features or features extracted from Convolutional Neural Networks. In this work, we bridge this gap and present a novel method that extracts features from a ViT and aggregates them using VLAD encoding. The model is trained in a self-supervised fashion without any need for labels. We show that extracting local foreground features is superior to using the ViT's class token in the context of writer retrieval. We evaluate our method on two historical document collections. We set a new state-at-of-art performance on the Historical-WI dataset (83.1\% mAP), and the HisIR19 dataset (95.0\% mAP). Additionally, we demonstrate that our ViT feature extractor can be directly applied to modern datasets such as the CVL database (98.6\% mAP) without any fine-tuning.

Title: SITUATE: Indoor Human Trajectory Prediction through Geometric Features and Self-Supervised Vision Representation

• Authors: Luigi Capogrosso, Andrea Toaiari, Andrea Avogaro, Uzair Khan, Aditya Jivoji, Franco Fummi, Marco Cristani
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.00774
• Pdf URL: https://arxiv.org/pdf/2409.00774
• Copy Paste: [[2409.00774]] SITUATE: Indoor Human Trajectory Prediction through Geometric Features and Self-Supervised Vision Representation(https://arxiv.org/abs/2409.00774)
• Keywords: self-supervised
• Abstract: Patterns of human motion in outdoor and indoor environments are substantially different due to the scope of the environment and the typical intentions of people therein. While outdoor trajectory forecasting has received significant attention, indoor forecasting is still an underexplored research area. This paper proposes SITUATE, a novel approach to cope with indoor human trajectory prediction by leveraging equivariant and invariant geometric features and a self-supervised vision representation. The geometric learning modules model the intrinsic symmetries and human movements inherent in indoor spaces. This concept becomes particularly important because self-loops at various scales and rapid direction changes often characterize indoor trajectories. On the other hand, the vision representation module is used to acquire spatial-semantic information about the environment to predict users' future locations more accurately. We evaluate our method through comprehensive experiments on the two most famous indoor trajectory forecasting datasets, i.e., THÖR and Supermarket, obtaining state-of-the-art performance. Furthermore, we also achieve competitive results in outdoor scenarios, showing that indoor-oriented forecasting models generalize better than outdoor-oriented ones. The source code is available at this https URL.

Title: Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Models

• Authors: Martin Mayr, Marcel Dreier, Florian Kordon, Mathias Seuret, Jochen Zöllner, Fei Wu, Andreas Maier, Vincent Christlein
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00786
• Pdf URL: https://arxiv.org/pdf/2409.00786
• Copy Paste: [[2409.00786]] Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Models(https://arxiv.org/abs/2409.00786)
• Keywords: diffusion
• Abstract: The imitation of cursive handwriting is mainly limited to generating handwritten words or lines. Multiple synthetic outputs must be stitched together to create paragraphs or whole pages, whereby consistency and layout information are lost. To close this gap, we propose a method for imitating handwriting at the paragraph level that also works for unseen writing styles. Therefore, we introduce a modified latent diffusion model that enriches the encoder-decoder mechanism with specialized loss functions that explicitly preserve the style and content. We enhance the attention mechanism of the diffusion model with adaptive 2D positional encoding and the conditioning mechanism to work with two modalities simultaneously: a style image and the target text. This significantly improves the realism of the generated handwriting. Our approach sets a new benchmark in our comprehensive evaluation. It outperforms all existing imitation methods at both line and paragraph levels, considering combined style and content preservation.

Title: Diffusion based multi-domain neuroimaging harmonization method with preservation of anatomical details

• Authors: Haoyu Lan, Bino A. Varghese, Nasim Sheikh-Bahaei, Farshid Sepehrband, Arthur W Toga, Jeiran Choupan
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.00807
• Pdf URL: https://arxiv.org/pdf/2409.00807
• Copy Paste: [[2409.00807]] Diffusion based multi-domain neuroimaging harmonization method with preservation of anatomical details(https://arxiv.org/abs/2409.00807)
• Keywords: diffusion, generative
• Abstract: Multi-center neuroimaging studies face technical variability due to batch differences across sites, which potentially hinders data aggregation and impacts study reliability.Recent efforts in neuroimaging harmonization have aimed to minimize these technical gaps and reduce technical variability across batches. While Generative Adversarial Networks (GAN) has been a prominent method for addressing image harmonization tasks, GAN-harmonized images suffer from artifacts or anatomical distortions. Given the advancements of denoising diffusion probabilistic model which produces high-fidelity images, we have assessed the efficacy of the diffusion model for neuroimaging harmonization. we have demonstrated the diffusion model's superior capability in harmonizing images from multiple domains, while GAN-based methods are limited to harmonizing images between two domains per model. Our experiments highlight that the learned domain invariant anatomical condition reinforces the model to accurately preserve the anatomical details while differentiating batch differences at each diffusion step. Our proposed method has been tested on two public neuroimaging dataset ADNI1 and ABIDE II, yielding harmonization results with consistent anatomy preservation and superior FID score compared to the GAN-based methods. We have conducted multiple analysis including extensive quantitative and qualitative evaluations against the baseline models, ablation study showcasing the benefits of the learned conditions, and improvements in the consistency of perivascular spaces (PVS) segmentation through harmonization.

Title: Curvy: A Parametric Cross-section based Surface Reconstruction

• Authors: Aradhya N. Mathur, Apoorv Khattar, Ojaswa Sharma
• Subjects: cs.CV, cs.CG, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.00829
• Pdf URL: https://arxiv.org/pdf/2409.00829
• Copy Paste: [[2409.00829]] Curvy: A Parametric Cross-section based Surface Reconstruction(https://arxiv.org/abs/2409.00829)
• Keywords: generative
• Abstract: In this work, we present a novel approach for reconstructing shape point clouds using planar sparse cross-sections with the help of generative modeling. We present unique challenges pertaining to the representation and reconstruction in this problem setting. Most methods in the classical literature lack the ability to generalize based on object class and employ complex mathematical machinery to reconstruct reliable surfaces. We present a simple learnable approach to generate a large number of points from a small number of input cross-sections over a large dataset. We use a compact parametric polyline representation using adaptive splitting to represent the cross-sections and perform learning using a Graph Neural Network to reconstruct the underlying shape in an adaptive manner reducing the dependence on the number of cross-sections provided.

Title: Image-to-Lidar Relational Distillation for Autonomous Driving Data

• Authors: Anas Mahmoud, Ali Harakeh, Steven Waslander
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00845
• Pdf URL: https://arxiv.org/pdf/2409.00845
• Copy Paste: [[2409.00845]] Image-to-Lidar Relational Distillation for Autonomous Driving Data(https://arxiv.org/abs/2409.00845)
• Keywords: foundation model
• Abstract: Pre-trained on extensive and diverse multi-modal datasets, 2D foundation models excel at addressing 2D tasks with little or no downstream supervision, owing to their robust representations. The emergence of 2D-to-3D distillation frameworks has extended these capabilities to 3D models. However, distilling 3D representations for autonomous driving datasets presents challenges like self-similarity, class imbalance, and point cloud sparsity, hindering the effectiveness of contrastive distillation, especially in zero-shot learning contexts. Whereas other methodologies, such as similarity-based distillation, enhance zero-shot performance, they tend to yield less discriminative representations, diminishing few-shot performance. We investigate the gap in structure between the 2D and the 3D representations that result from state-of-the-art distillation frameworks and reveal a significant mismatch between the two. Additionally, we demonstrate that the observed structural gap is negatively correlated with the efficacy of the distilled representations on zero-shot and few-shot 3D semantic segmentation. To bridge this gap, we propose a relational distillation framework enforcing intra-modal and cross-modal constraints, resulting in distilled 3D representations that closely capture the structure of the 2D representation. This alignment significantly enhances 3D representation performance over those learned through contrastive distillation in zero-shot segmentation tasks. Furthermore, our relational loss consistently improves the quality of 3D representations in both in-distribution and out-of-distribution few-shot segmentation tasks, outperforming approaches that rely on the similarity loss.

Title: VQ-Flow: Taming Normalizing Flows for Multi-Class Anomaly Detection via Hierarchical Vector Quantization

• Authors: Yixuan Zhou, Xing Xu, Zhe Sun, Jingkuan Song, Andrzej Cichocki, Heng Tao Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00942
• Pdf URL: https://arxiv.org/pdf/2409.00942
• Copy Paste: [[2409.00942]] VQ-Flow: Taming Normalizing Flows for Multi-Class Anomaly Detection via Hierarchical Vector Quantization(https://arxiv.org/abs/2409.00942)
• Keywords: anomaly
• Abstract: Normalizing flows, a category of probabilistic models famed for their capabilities in modeling complex data distributions, have exhibited remarkable efficacy in unsupervised anomaly detection. This paper explores the potential of normalizing flows in multi-class anomaly detection, wherein the normal data is compounded with multiple classes without providing class labels. Through the integration of vector quantization (VQ), we empower the flow models to distinguish different concepts of multi-class normal data in an unsupervised manner, resulting in a novel flow-based unified method, named VQ-Flow. Specifically, our VQ-Flow leverages hierarchical vector quantization to estimate two relative codebooks: a Conceptual Prototype Codebook (CPC) for concept distinction and its concomitant Concept-Specific Pattern Codebook (CSPC) to capture concept-specific normal patterns. The flow models in VQ-Flow are conditioned on the concept-specific patterns captured in CSPC, capable of modeling specific normal patterns associated with different concepts. Moreover, CPC further enables our VQ-Flow for concept-aware distribution modeling, faithfully mimicking the intricate multi-class normal distribution through a mixed Gaussian distribution reparametrized on the conceptual prototypes. Through the introduction of vector quantization, the proposed VQ-Flow advances the state-of-the-art in multi-class anomaly detection within a unified training scheme, yielding the Det./Loc. AUROC of 99.5%/98.3% on MVTec AD. The codebase is publicly available at this https URL.

Title: IVGF: The Fusion-Guided Infrared and Visible General Framework

• Authors: Fangcen Liu, Chenqiang Gao, Fang Chen, Pengcheng Li, Junjie Guo, Deyu Meng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00973
• Pdf URL: https://arxiv.org/pdf/2409.00973
• Copy Paste: [[2409.00973]] IVGF: The Fusion-Guided Infrared and Visible General Framework(https://arxiv.org/abs/2409.00973)
• Keywords: foundation model
• Abstract: Infrared and visible dual-modality tasks such as semantic segmentation and object detection can achieve robust performance even in extreme scenes by fusing complementary information. Most current methods design task-specific frameworks, which are limited in generalization across multiple tasks. In this paper, we propose a fusion-guided infrared and visible general framework, IVGF, which can be easily extended to many high-level vision tasks. Firstly, we adopt the SOTA infrared and visible foundation models to extract the general representations. Then, to enrich the semantics information of these general representations for high-level vision tasks, we design the feature enhancement module and token enhancement module for feature maps and tokens, respectively. Besides, the attention-guided fusion module is proposed for effectively fusing by exploring the complementary information of two modalities. Moreover, we also adopt the cutout&mix augmentation strategy to conduct the data augmentation, which further improves the ability of the model to mine the regional complementary between the two modalities. Extensive experiments show that the IVGF outperforms state-of-the-art dual-modality methods in the semantic segmentation and object detection tasks. The detailed ablation studies demonstrate the effectiveness of each module, and another experiment explores the anti-missing modality ability of the proposed method in the dual-modality semantic segmentation task.

Title: Self-Supervised Multi-Scale Network for Blind Image Deblurring via Alternating Optimization

• Authors: Lening Guo, Jing Yu, Ning Zhang, Chuangbai Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.00988
• Pdf URL: https://arxiv.org/pdf/2409.00988
• Copy Paste: [[2409.00988]] Self-Supervised Multi-Scale Network for Blind Image Deblurring via Alternating Optimization(https://arxiv.org/abs/2409.00988)
• Keywords: self-supervised
• Abstract: Blind image deblurring is a challenging low-level vision task that involves estimating the unblurred image when the blur kernel is unknown. In this paper, we present a self-supervised multi-scale blind image deblurring method to jointly estimate the latent image and the blur kernel via alternating optimization. In the image estimation step, we construct a multi-scale generator network with multiple inputs and multiple outputs to collaboratively estimate latent images at various scales, supervised by an image pyramid constructed from only the blurred image. This generator places architectural constraints on the network and avoids the need for mathematical expression of image priors. In the blur kernel estimation step, the blur kernel at each scale is independently estimated with a direct solution to a quadratic regularized least-squares model for its flexible adaptation to the proposed multi-scale generator for image estimation. Thanks to the collaborative estimation across multiple scales, our method avoids the computationally intensive coarse-to-fine propagation and additional image deblurring processes used in traditional mathematical optimization-based methods. Quantitative and qualitative experimental results on synthetic and realistic datasets demonstrate the superior performance of our method, especially for handling large and real-world blurs.

Title: 3D Priors-Guided Diffusion for Blind Face Restoration

• Authors: Xiaobin Lu, Xiaobin Hu, Jun Luo, Ben Zhu, Yaping Ruan, Wenqi Ren
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.00991
• Pdf URL: https://arxiv.org/pdf/2409.00991
• Copy Paste: [[2409.00991]] 3D Priors-Guided Diffusion for Blind Face Restoration(https://arxiv.org/abs/2409.00991)
• Keywords: diffusion, generative
• Abstract: Blind face restoration endeavors to restore a clear face image from a degraded counterpart. Recent approaches employing Generative Adversarial Networks (GANs) as priors have demonstrated remarkable success in this field. However, these methods encounter challenges in achieving a balance between realism and fidelity, particularly in complex degradation scenarios. To inherit the exceptional realism generative ability of the diffusion model and also constrained by the identity-aware fidelity, we propose a novel diffusion-based framework by embedding the 3D facial priors as structure and identity constraints into a denoising diffusion process. Specifically, in order to obtain more accurate 3D prior representations, the 3D facial image is reconstructed by a 3D Morphable Model (3DMM) using an initial restored face image that has been processed by a pretrained restoration network. A customized multi-level feature extraction method is employed to exploit both structural and identity information of 3D facial images, which are then mapped into the noise estimation process. In order to enhance the fusion of identity information into the noise estimation, we propose a Time-Aware Fusion Block (TAFB). This module offers a more efficient and adaptive fusion of weights for denoising, considering the dynamic nature of the denoising process in the diffusion model, which involves initial structure refinement followed by texture detail enhancement.Extensive experiments demonstrate that our network performs favorably against state-of-the-art algorithms on synthetic and real-world datasets for blind face restoration.

Title: From Bird's-Eye to Street View: Crafting Diverse and Condition-Aligned Images with Latent Diffusion Model

• Authors: Xiaojie Xu, Tianshuo Xu, Fulong Ma, Yingcong Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01014
• Pdf URL: https://arxiv.org/pdf/2409.01014
• Copy Paste: [[2409.01014]] From Bird's-Eye to Street View: Crafting Diverse and Condition-Aligned Images with Latent Diffusion Model(https://arxiv.org/abs/2409.01014)
• Keywords: diffusion, generative
• Abstract: We explore Bird's-Eye View (BEV) generation, converting a BEV map into its corresponding multi-view street images. Valued for its unified spatial representation aiding multi-sensor fusion, BEV is pivotal for various autonomous driving applications. Creating accurate street-view images from BEV maps is essential for portraying complex traffic scenarios and enhancing driving algorithms. Concurrently, diffusion-based conditional image generation models have demonstrated remarkable outcomes, adept at producing diverse, high-quality, and condition-aligned results. Nonetheless, the training of these models demands substantial data and computational resources. Hence, exploring methods to fine-tune these advanced models, like Stable Diffusion, for specific conditional generation tasks emerges as a promising avenue. In this paper, we introduce a practical framework for generating images from a BEV layout. Our approach comprises two main components: the Neural View Transformation and the Street Image Generation. The Neural View Transformation phase converts the BEV map into aligned multi-view semantic segmentation maps by learning the shape correspondence between the BEV and perspective views. Subsequently, the Street Image Generation phase utilizes these segmentations as a condition to guide a fine-tuned latent diffusion model. This finetuning process ensures both view and style consistency. Our model leverages the generative capacity of large pretrained diffusion models within traffic contexts, effectively yielding diverse and condition-coherent street view images.

Title: A Perspective on Literary Metaphor in the Context of Generative AI

• Authors: Imke van Heerden, Anil Bas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01053
• Pdf URL: https://arxiv.org/pdf/2409.01053
• Copy Paste: [[2409.01053]] A Perspective on Literary Metaphor in the Context of Generative AI(https://arxiv.org/abs/2409.01053)
• Keywords: generative
• Abstract: At the intersection of creative text generation and literary theory, this study explores the role of literary metaphor and its capacity to generate a range of meanings. In this regard, literary metaphor is vital to the development of any particular language. To investigate whether the inclusion of original figurative language improves textual quality, we trained an LSTM-based language model in Afrikaans. The network produces phrases containing compellingly novel figures of speech. Specifically, the emphasis falls on how AI might be utilised as a defamiliarisation technique, which disrupts expected uses of language to augment poetic expression. Providing a literary perspective on text generation, the paper raises thought-provoking questions on aesthetic value, interpretation and evaluation.

Title: Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation

• Authors: Qihua Chen, Yue Ma, Hongfa Wang, Junkun Yuan, Wenzhe Zhao, Qi Tian, Hongmei Wang, Shaobo Min, Qifeng Chen, Wei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01055
• Pdf URL: https://arxiv.org/pdf/2409.01055
• Copy Paste: [[2409.01055]] Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation(https://arxiv.org/abs/2409.01055)
• Keywords: diffusion
• Abstract: This paper explores higher-resolution video outpainting with extensive content generation. We point out common issues faced by existing methods when attempting to largely outpaint videos: the generation of low-quality content and limitations imposed by GPU memory. To address these challenges, we propose a diffusion-based method called \textit{Follow-Your-Canvas}. It builds upon two core designs. First, instead of employing the common practice of "single-shot" outpainting, we distribute the task across spatial windows and seamlessly merge them. It allows us to outpaint videos of any size and resolution without being constrained by GPU memory. Second, the source video and its relative positional relation are injected into the generation process of each window. It makes the generated spatial layout within each window harmonize with the source video. Coupling with these two designs enables us to generate higher-resolution outpainting videos with rich content while keeping spatial and temporal consistency. Follow-Your-Canvas excels in large-scale video outpainting, e.g., from 512X512 to 1152X2048 (9X), while producing high-quality and aesthetically pleasing results. It achieves the best quantitative results across various resolution and scale setups. The code is released on this https URL

Title: DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing

• Authors: Xiaolong Wang, Zhi-Qi Cheng, Jue Wang, Xiaojiang Peng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01086
• Pdf URL: https://arxiv.org/pdf/2409.01086
• Copy Paste: [[2409.01086]] DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing(https://arxiv.org/abs/2409.01086)
• Keywords: diffusion
• Abstract: Fashion image editing is a crucial tool for designers to convey their creative ideas by visualizing design concepts interactively. Current fashion image editing techniques, though advanced with multimodal prompts and powerful diffusion models, often struggle to accurately identify editing regions and preserve the desired garment texture detail. To address these challenges, we introduce a new multimodal fashion image editing architecture based on latent diffusion models, called Detail-Preserved Diffusion Models (DPDEdit). DPDEdit guides the fashion image generation of diffusion models by integrating text prompts, region masks, human pose images, and garment texture images. To precisely locate the editing region, we first introduce Grounded-SAM to predict the editing region based on the user's textual description, and then combine it with other conditions to perform local editing. To transfer the detail of the given garment texture into the target fashion image, we propose a texture injection and refinement mechanism. Specifically, this mechanism employs a decoupled cross-attention layer to integrate textual descriptions and texture images, and incorporates an auxiliary U-Net to preserve the high-frequency details of generated garment texture. Additionally, we extend the VITON-HD dataset using a multimodal large language model to generate paired samples with texture images and textual descriptions. Extensive experiments show that our DPDEdit outperforms state-of-the-art methods in terms of image fidelity and coherence with the given multimodal inputs.

Title: Pre-Trained Language Models for Keyphrase Prediction: A Review

• Authors: Muhammad Umair, Tangina Sultana, Young-Koo Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01087
• Pdf URL: https://arxiv.org/pdf/2409.01087
• Copy Paste: [[2409.01087]] Pre-Trained Language Models for Keyphrase Prediction: A Review(https://arxiv.org/abs/2409.01087)
• Keywords: self-supervised
• Abstract: Keyphrase Prediction (KP) is essential for identifying keyphrases in a document that can summarize its content. However, recent Natural Language Processing (NLP) advances have developed more efficient KP models using deep learning techniques. The limitation of a comprehensive exploration jointly both keyphrase extraction and generation using pre-trained language models spotlights a critical gap in the literature, compelling our survey paper to bridge this deficiency and offer a unified and in-depth analysis to address limitations in previous surveys. This paper extensively examines the topic of pre-trained language models for keyphrase prediction (PLM-KP), which are trained on large text corpora via different learning (supervisor, unsupervised, semi-supervised, and self-supervised) techniques, to provide respective insights into these two types of tasks in NLP, precisely, Keyphrase Extraction (KPE) and Keyphrase Generation (KPG). We introduce appropriate taxonomies for PLM-KPE and KPG to highlight these two main tasks of NLP. Moreover, we point out some promising future directions for predicting keyphrases.

Title: Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning

• Authors: Jinglin Liang, Jin Zhong, Hanlin Gu, Zhongqi Lu, Xingxing Tang, Gang Dai, Shuangping Huang, Lixin Fan, Qiang Yang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01128
• Pdf URL: https://arxiv.org/pdf/2409.01128
• Copy Paste: [[2409.01128]] Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning(https://arxiv.org/abs/2409.01128)
• Keywords: diffusion
• Abstract: Federated Class Continual Learning (FCCL) merges the challenges of distributed client learning with the need for seamless adaptation to new classes without forgetting old ones. The key challenge in FCCL is catastrophic forgetting, an issue that has been explored to some extent in Continual Learning (CL). However, due to privacy preservation requirements, some conventional methods, such as experience replay, are not directly applicable to FCCL.Existing FCCL methods mitigate forgetting by generating historical data through federated training of GANs or data-free knowledge distillation. However, these approaches often suffer from unstable training of generators or low-quality generated data, limiting their guidance for the this http URL address this challenge, we propose a novel method of data replay based on diffusion models. Instead of training a diffusion model, we employ a pre-trained conditional diffusion model to reverse-engineer each class, searching the corresponding input conditions for each class within the model's input space, significantly reducing computational resources and time consumption while ensuring effective generation. Furthermore, we enhance the classifier's domain generalization ability on generated and real data through contrastive learning, indirectly improving the representational capability of generated data for real data. Comprehensive experiments demonstrate that our method significantly outperforms existing baselines.Code is available at this https URL.

Title: Generating Synthetic Satellite Imagery for Rare Objects: An Empirical Comparison of Models and Metrics

• Authors: Tuong Vy Nguyen, Johannes Hoster, Alexander Glaser, Kristian Hildebrand, Felix Biessmann
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.01138
• Pdf URL: https://arxiv.org/pdf/2409.01138
• Copy Paste: [[2409.01138]] Generating Synthetic Satellite Imagery for Rare Objects: An Empirical Comparison of Models and Metrics(https://arxiv.org/abs/2409.01138)
• Keywords: generative
• Abstract: Generative deep learning architectures can produce realistic, high-resolution fake imagery -- with potentially drastic societal implications. A key question in this context is: How easy is it to generate realistic imagery, in particular for niche domains. The iterative process required to achieve specific image content is difficult to automate and control. Especially for rare classes, it remains difficult to assess fidelity, meaning whether generative approaches produce realistic imagery and alignment, meaning how (well) the generation can be guided by human input. In this work, we present a large-scale empirical evaluation of generative architectures which we fine-tuned to generate synthetic satellite imagery. We focus on nuclear power plants as an example of a rare object category - as there are only around 400 facilities worldwide, this restriction is exemplary for many other scenarios in which training and test data is limited by the restricted number of occurrences of real-world examples. We generate synthetic imagery by conditioning on two kinds of modalities, textual input and image input obtained from a game engine that allows for detailed specification of the building layout. The generated images are assessed by commonly used metrics for automatic evaluation and then compared with human judgement from our conducted user studies to assess their trustworthiness. Our results demonstrate that even for rare objects, generation of authentic synthetic satellite imagery with textual or detailed building layouts is feasible. In line with previous work, we find that automated metrics are often not aligned with human perception -- in fact, we find strong negative correlations between commonly used image quality metrics and human ratings.

Title: Backdoor Defense through Self-Supervised and Generative Learning

• Authors: Ivan Sabolić, Ivan Grubišić, Siniša Šegvić
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.01185
• Pdf URL: https://arxiv.org/pdf/2409.01185
• Copy Paste: [[2409.01185]] Backdoor Defense through Self-Supervised and Generative Learning(https://arxiv.org/abs/2409.01185)
• Keywords: self-supervised, generative
• Abstract: Backdoor attacks change a small portion of training data by introducing hand-crafted triggers and rewiring the corresponding labels towards a desired target class. Training on such data injects a backdoor which causes malicious inference in selected test samples. Most defenses mitigate such attacks through various modifications of the discriminative learning procedure. In contrast, this paper explores an approach based on generative modelling of per-class distributions in a self-supervised representation space. Interestingly, these representations get either preserved or heavily disturbed under recent backdoor attacks. In both cases, we find that per-class generative models allow to detect poisoned data and cleanse the dataset. Experiments show that training on cleansed dataset greatly reduces the attack success rate and retains the accuracy on benign inputs.

Title: OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model

• Authors: Liuhan Chen, Zongjian Li, Bin Lin, Bin Zhu, Qian Wang, Shenghai Yuan, Xing Zhou, Xinghua Cheng, Li Yuan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.01199
• Pdf URL: https://arxiv.org/pdf/2409.01199
• Copy Paste: [[2409.01199]] OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model(https://arxiv.org/abs/2409.01199)
• Keywords: diffusion
• Abstract: Variational Autoencoder (VAE), compressing videos into latent representations, is a crucial preceding component of Latent Video Diffusion Models (LVDMs). With the same reconstruction quality, the more sufficient the VAE's compression for videos is, the more efficient the LVDMs are. However, most LVDMs utilize 2D image VAE, whose compression for videos is only in the spatial dimension and often ignored in the temporal dimension. How to conduct temporal compression for videos in a VAE to obtain more concise latent representations while promising accurate reconstruction is seldom explored. To fill this gap, we propose an omni-dimension compression VAE, named OD-VAE, which can temporally and spatially compress videos. Although OD-VAE's more sufficient compression brings a great challenge to video reconstruction, it can still achieve high reconstructed accuracy by our fine design. To obtain a better trade-off between video reconstruction quality and compression speed, four variants of OD-VAE are introduced and analyzed. In addition, a novel tail initialization is designed to train OD-VAE more efficiently, and a novel inference strategy is proposed to enable OD-VAE to handle videos of arbitrary length with limited GPU memory. Comprehensive experiments on video reconstruction and LVDM-based video generation demonstrate the effectiveness and efficiency of our proposed methods.

Title: GAS: Generative Activation-Aided Asynchronous Split Federated Learning

• Authors: Jiarong Yang, Yuan Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.01251
• Pdf URL: https://arxiv.org/pdf/2409.01251
• Copy Paste: [[2409.01251]] GAS: Generative Activation-Aided Asynchronous Split Federated Learning(https://arxiv.org/abs/2409.01251)
• Keywords: generative
• Abstract: Split Federated Learning (SFL) splits and collaboratively trains a shared model between clients and server, where clients transmit activations and client-side models to server for updates. Recent SFL studies assume synchronous transmission of activations and client-side models from clients to server. However, due to significant variations in computational and communication capabilities among clients, activations and client-side models arrive at server asynchronously. The delay caused by asynchrony significantly degrades the performance of SFL. To address this issue, we consider an asynchronous SFL framework, where an activation buffer and a model buffer are embedded on the server to manage the asynchronously transmitted activations and client-side models, respectively. Furthermore, as asynchronous activation transmissions cause the buffer to frequently receive activations from resource-rich clients, leading to biased updates of the server-side model, we propose Generative activations-aided Asynchronous SFL (GAS). In GAS, the server maintains an activation distribution for each label based on received activations and generates activations from these distributions according to the degree of bias. These generative activations are then used to assist in updating the server-side model, ensuring more accurate updates. We derive a tighter convergence bound, and our experiments demonstrate the effectiveness of the proposed method.

Title: Topological degree as a discrete diagnostic for disentanglement, with applications to the $\Delta$VAE

• Authors: Mahefa Ratsisetraina Ravelonanosy, Vlado Menkovski, Jacobus W. Portegies
• Subjects: cs.LG, cs.AI, math.AT
• Abstract URL: https://arxiv.org/abs/2409.01303
• Pdf URL: https://arxiv.org/pdf/2409.01303
• Copy Paste: [[2409.01303]] Topological degree as a discrete diagnostic for disentanglement, with applications to the $\Delta$VAE(https://arxiv.org/abs/2409.01303)
• Keywords: diffusion
• Abstract: We investigate the ability of Diffusion Variational Autoencoder ($\Delta$VAE) with unit sphere $\mathcal{S}^2$ as latent space to capture topological and geometrical structure and disentangle latent factors in datasets. For this, we introduce a new diagnostic of disentanglement: namely the topological degree of the encoder, which is a map from the data manifold to the latent space. By using tools from homology theory, we derive and implement an algorithm that computes this degree. We use the algorithm to compute the degree of the encoder of models that result from the training procedure. Our experimental results show that the $\Delta$VAE achieves relatively small LSBD scores, and that regardless of the degree after initialization, the degree of the encoder after training becomes $-1$ or $+1$, which implies that the resulting encoder is at least homotopic to a homeomorphism.

Title: LoGex: Improved tail detection of extremely rare histopathology classes via guided diffusion

• Authors: Maximilian Mueller, Matthias Hein
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.01317
• Pdf URL: https://arxiv.org/pdf/2409.01317
• Copy Paste: [[2409.01317]] LoGex: Improved tail detection of extremely rare histopathology classes via guided diffusion(https://arxiv.org/abs/2409.01317)
• Keywords: diffusion
• Abstract: In realistic medical settings, the data are often inherently long-tailed, with most samples concentrated in a few classes and a long tail of rare classes, usually containing just a few samples. This distribution presents a significant challenge because rare conditions are critical to detect and difficult to classify due to limited data. In this paper, rather than attempting to classify rare classes, we aim to detect these as out-of-distribution data reliably. We leverage low-rank adaption (LoRA) and diffusion guidance to generate targeted synthetic data for the detection problem. We significantly improve the OOD detection performance on a challenging histopathological task with only ten samples per tail class without losing classification accuracy on the head classes.

Title: Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing

• Authors: Vadim Titov, Madina Khalmatova, Alexandra Ivanova, Dmitry Vetrov, Aibek Alanov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01322
• Pdf URL: https://arxiv.org/pdf/2409.01322
• Copy Paste: [[2409.01322]] Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing(https://arxiv.org/abs/2409.01322)
• Keywords: diffusion, generative
• Abstract: Despite recent advances in large-scale text-to-image generative models, manipulating real images with these models remains a challenging problem. The main limitations of existing editing methods are that they either fail to perform with consistent quality on a wide range of image edits or require time-consuming hyperparameter tuning or fine-tuning of the diffusion model to preserve the image-specific appearance of the input image. We propose a novel approach that is built upon a modified diffusion sampling process via the guidance mechanism. In this work, we explore the self-guidance technique to preserve the overall structure of the input image and its local regions appearance that should not be edited. In particular, we explicitly introduce layout-preserving energy functions that are aimed to save local and global structures of the source image. Additionally, we propose a noise rescaling mechanism that allows to preserve noise distribution by balancing the norms of classifier-free guidance and our proposed guiders during generation. Such a guiding approach does not require fine-tuning the diffusion model and exact inversion process. As a result, the proposed method provides a fast and high-quality editing mechanism. In our experiments, we show through human evaluation and quantitative analysis that the proposed method allows to produce desired editing which is more preferable by humans and also achieves a better trade-off between editing quality and preservation of the original image. Our code is available at this https URL.

Title: SPDiffusion: Semantic Protection Diffusion for Multi-concept Text-to-image Generation

• Authors: Yang Zhang, Rui Zhang, Xuecheng Nie, Haochen Li, Jikun Chen, Yifan Hao, Xin Zhang, Luoqi Liu, Ling Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01327
• Pdf URL: https://arxiv.org/pdf/2409.01327
• Copy Paste: [[2409.01327]] SPDiffusion: Semantic Protection Diffusion for Multi-concept Text-to-image Generation(https://arxiv.org/abs/2409.01327)
• Keywords: diffusion
• Abstract: Recent text-to-image models have achieved remarkable success in generating high-quality images. However, when tasked with multi-concept generation which creates images containing multiple characters or objects, existing methods often suffer from attribute confusion, resulting in severe text-image inconsistency. We found that attribute confusion occurs when a certain region of the latent features attend to multiple or incorrect prompt tokens. In this work, we propose novel Semantic Protection Diffusion (SPDiffusion) to protect the semantics of regions from the influence of irrelevant tokens, eliminating the confusion of non-corresponding attributes. In the SPDiffusion framework, we design a Semantic Protection Mask (SP-Mask) to represent the relevance of the regions and the tokens, and propose a Semantic Protection Cross-Attention (SP-Attn) to shield the influence of irrelevant tokens on specific regions in the generation process. To evaluate our method, we created a diverse multi-concept benchmark, and SPDiffusion achieves state-of-the-art results on this benchmark, proving its effectiveness. Our method can be combined with many other application methods or backbones, such as ControlNet, Story Diffusion, PhotoMaker and PixArt-alpha to enhance their multi-concept capabilities, demonstrating strong compatibility and scalability.

Title: Pediatric brain tumor classification using digital histopathology and deep learning: evaluation of SOTA methods on a multi-center Swedish cohort

• Authors: Iulian Emil Tampu, Per Nyman, Christoforos Spyretos, Ida Blystad, Alia Shamikh, Gabriela Prochazka, Teresita Díaz de Ståhl, Johanna Sandgren, Peter Lundberg, Neda Haj-Hosseini
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01330
• Pdf URL: https://arxiv.org/pdf/2409.01330
• Copy Paste: [[2409.01330]] Pediatric brain tumor classification using digital histopathology and deep learning: evaluation of SOTA methods on a multi-center Swedish cohort(https://arxiv.org/abs/2409.01330)
• Keywords: foundation model
• Abstract: Brain tumors are the most common solid tumors in children and young adults, but the scarcity of large histopathology datasets has limited the application of computational pathology in this group. This study implements two weakly supervised multiple-instance learning (MIL) approaches on patch-features obtained from state-of-the-art histology-specific foundation models to classify pediatric brain tumors in hematoxylin and eosin whole slide images (WSIs) from a multi-center Swedish cohort. WSIs from 540 subjects (age 8.5$\pm$4.9 years) diagnosed with brain tumor were gathered from the six Swedish university hospitals. Instance (patch)-level features were obtained from WSIs using three pre-trained feature extractors: ResNet50, UNI and CONCH. Instances were aggregated using attention-based MIL (ABMIL) or clustering-constrained attention MIL (CLAM) for patient-level classification. Models were evaluated on three classification tasks based on the hierarchical classification of pediatric brain tumors: tumor category, family and type. Model generalization was assessed by training on data from two of the centers and testing on data from four other centers. Model interpretability was evaluated through attention-mapping. The highest classification performance was achieved using UNI features and AMBIL aggregation, with Matthew's correlation coefficient of 0.86$\pm$0.04, 0.63$\pm$0.04, and 0.53$\pm$0.05, for tumor category, family and type classification, respectively. When evaluating generalization, models utilizing UNI and CONCH features outperformed those using ResNet50. However, the drop in performance from the in-site to out-of-site testing was similar across feature extractors. These results show the potential of state-of-the-art computational pathology methods in diagnosing pediatric brain tumors at different hierarchical levels with fair generalizability on a multi-center national dataset.

Title: Target-Driven Distillation: Consistency Distillation with Target Timestep Selection and Decoupled Guidance

• Authors: Cunzheng Wang, Ziyuan Guo, Yuxuan Duan, Huaxia Li, Nemo Chen, Xu Tang, Yao Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01347
• Pdf URL: https://arxiv.org/pdf/2409.01347
• Copy Paste: [[2409.01347]] Target-Driven Distillation: Consistency Distillation with Target Timestep Selection and Decoupled Guidance(https://arxiv.org/abs/2409.01347)
• Keywords: diffusion, generative
• Abstract: Consistency distillation methods have demonstrated significant success in accelerating generative tasks of diffusion models. However, since previous consistency distillation methods use simple and straightforward strategies in selecting target timesteps, they usually struggle with blurs and detail losses in generated images. To address these limitations, we introduce Target-Driven Distillation (TDD), which (1) adopts a delicate selection strategy of target timesteps, increasing the training efficiency; (2) utilizes decoupled guidances during training, making TDD open to post-tuning on guidance scale during inference periods; (3) can be optionally equipped with non-equidistant sampling and x0 clipping, enabling a more flexible and accurate way for image sampling. Experiments verify that TDD achieves state-of-the-art performance in few-step generation, offering a better choice among consistency distillation models.

Title: PatternPaint: Generating Layout Patterns Using Generative AI and Inpainting Techniques

• Authors: Guanglei Zhou, Bhargav Korrapati, Gaurav Rajavendra Reddy, Jiang Hu, Yiran Chen, Dipto G. Thakurta
• Subjects: cs.CV, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.01348
• Pdf URL: https://arxiv.org/pdf/2409.01348
• Copy Paste: [[2409.01348]] PatternPaint: Generating Layout Patterns Using Generative AI and Inpainting Techniques(https://arxiv.org/abs/2409.01348)
• Keywords: generative
• Abstract: Generation of VLSI layout patterns is essential for a wide range of Design For Manufacturability (DFM) studies. In this study, we investigate the potential of generative machine learning models for creating design rule legal metal layout patterns. Our results demonstrate that the proposed model can generate legal patterns in complex design rule settings and achieves a high diversity score. The designed system, with its flexible settings, supports both pattern generation with localized changes, and design rule violation correction. Our methodology is validated on Intel 18A Process Design Kit (PDK) and can produce a wide range of DRC-compliant pattern libraries with only 20 starter patterns.

Title: Membership Inference Attacks Against In-Context Learning

• Authors: Rui Wen, Zheng Li, Michael Backes, Yang Zhang
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.01380
• Pdf URL: https://arxiv.org/pdf/2409.01380
• Copy Paste: [[2409.01380]] Membership Inference Attacks Against In-Context Learning(https://arxiv.org/abs/2409.01380)
• Keywords: in-context
• Abstract: Adapting Large Language Models (LLMs) to specific tasks introduces concerns about computational efficiency, prompting an exploration of efficient methods such as In-Context Learning (ICL). However, the vulnerability of ICL to privacy attacks under realistic assumptions remains largely unexplored. In this work, we present the first membership inference attack tailored for ICL, relying solely on generated texts without their associated probabilities. We propose four attack strategies tailored to various constrained scenarios and conduct extensive experiments on four popular large language models. Empirical results show that our attacks can accurately determine membership status in most cases, e.g., 95\% accuracy advantage against LLaMA, indicating that the associated risks are much higher than those shown by existing probability-based attacks. Additionally, we propose a hybrid attack that synthesizes the strengths of the aforementioned strategies, achieving an accuracy advantage of over 95\% in most cases. Furthermore, we investigate three potential defenses targeting data, instruction, and output. Results demonstrate combining defenses from orthogonal dimensions significantly reduces privacy leakage and offers enhanced privacy assurances.

Title: Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization

• Authors: Gao Tianci, Dmitriev D. Dmitry, Konstantin A. Neusypin, Yang Bo, Rao Shengren
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.01427
• Pdf URL: https://arxiv.org/pdf/2409.01427
• Copy Paste: [[2409.01427]] Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization(https://arxiv.org/abs/2409.01427)
• Keywords: diffusion
• Abstract: Recent advancements in reinforcement learning (RL) have been fueled by large-scale data and deep neural networks, particularly for high-dimensional and complex tasks. Online RL methods like Proximal Policy Optimization (PPO) are effective in dynamic scenarios but require substantial real-time data, posing challenges in resource-constrained or slow simulation environments. Offline RL addresses this by pre-learning policies from large datasets, though its success depends on the quality and diversity of the data. This work proposes a framework that enhances PPO algorithms by incorporating a diffusion model to generate high-quality virtual trajectories for offline datasets. This approach improves exploration and sample efficiency, leading to significant gains in cumulative rewards, convergence speed, and strategy stability in complex tasks. Our contributions are threefold: we explore the potential of diffusion models in RL, particularly for offline datasets, extend the application of online RL to offline environments, and experimentally validate the performance improvements of PPO with diffusion models. These findings provide new insights and methods for applying RL to high-dimensional, complex tasks. Finally, we open-source our code at this https URL

Title: PoliPrompt: A High-Performance Cost-Effective LLM-Based Text Classification Framework for Political Science

• Authors: Menglin Liu, Ge Shi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01466
• Pdf URL: https://arxiv.org/pdf/2409.01466
• Copy Paste: [[2409.01466]] PoliPrompt: A High-Performance Cost-Effective LLM-Based Text Classification Framework for Political Science(https://arxiv.org/abs/2409.01466)
• Keywords: in-context
• Abstract: Recent advancements in large language models (LLMs) have opened new avenues for enhancing text classification efficiency in political science, surpassing traditional machine learning methods that often require extensive feature engineering, human labeling, and task-specific training. However, their effectiveness in achieving high classification accuracy remains questionable. This paper introduces a three-stage in-context learning approach that leverages LLMs to improve classification accuracy while minimizing experimental costs. Our method incorporates automatic enhanced prompt generation, adaptive exemplar selection, and a consensus mechanism that resolves discrepancies between two weaker LLMs, refined by an advanced LLM. We validate our approach using datasets from the BBC news reports, Kavanaugh Supreme Court confirmation, and 2018 election campaign ads. The results show significant improvements in classification F1 score (+0.36 for zero-shot classification) with manageable economic costs (-78% compared with human labeling), demonstrating that our method effectively addresses the limitations of traditional machine learning while offering a scalable and reliable solution for text analysis in political science.

Title: Masked Mixers for Language Generation and Retrieval

• Authors: Benjamin L. Badger
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.01482
• Pdf URL: https://arxiv.org/pdf/2409.01482
• Copy Paste: [[2409.01482]] Masked Mixers for Language Generation and Retrieval(https://arxiv.org/abs/2409.01482)
• Keywords: generative
• Abstract: Attention mechanisms that confer selective focus on a strict subset of input elements are nearly ubiquitous in language models today. We posit there to be downside to the use of attention: most information present in the input is necessarily lost. In support of this idea we observe poor input representation accuracy in transformers, but find more accurate representation in what we term masked mixers which replace self-attention with masked convolutions. Applied to TinyStories the masked mixer learns causal language tasks more efficiently than early transformer implementations and somewhat less efficiently than optimized, current implementations. The most efficient learning algorithm observed for this dataset is a transformer-masked mixer hybrid, suggesting that these models learn in an orthogonal manner. We hypothesized that the information loss exhibited by transformers would be much more detrimental to retrieval than generation, and to test this we introduce an efficient training approach for retrieval models based on existing generative model embeddings. With this method, embeddings from masked mixers are found to result in far better summary-to-story retrieval compared to embeddings from transformers.

Title: EarthGen: Generating the World from Top-Down Views

• Authors: Ansh Sharma, Albert Xiao, Praneet Rathi, Rohit Kundu, Albert Zhai, Yuan Shen, Shenlong Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01491
• Pdf URL: https://arxiv.org/pdf/2409.01491
• Copy Paste: [[2409.01491]] EarthGen: Generating the World from Top-Down Views(https://arxiv.org/abs/2409.01491)
• Keywords: diffusion, generative
• Abstract: In this work, we present a novel method for extensive multi-scale generative terrain modeling. At the core of our model is a cascade of superresolution diffusion models that can be combined to produce consistent images across multiple resolutions. Pairing this concept with a tiled generation method yields a scalable system that can generate thousands of square kilometers of realistic Earth surfaces at high resolution. We evaluate our method on a dataset collected from Bing Maps and show that it outperforms super-resolution baselines on the extreme super-resolution task of 1024x zoom. We also demonstrate its ability to create diverse and coherent scenes via an interactive gigapixel-scale generated map. Finally, we demonstrate how our system can be extended to enable novel content creation applications including controllable world generation and 3D scene generation.

Title: The Compressor-Retriever Architecture for Language Model OS

• Authors: Yuan Yang, Siheng Xiong, Ehsan Shareghi, Faramarz Fekri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.01495
• Pdf URL: https://arxiv.org/pdf/2409.01495
• Copy Paste: [[2409.01495]] The Compressor-Retriever Architecture for Language Model OS(https://arxiv.org/abs/2409.01495)
• Keywords: in-context
• Abstract: Recent advancements in large language models (LLMs) have significantly enhanced their capacity to aggregate and process information across multiple modalities, enabling them to perform a wide range of tasks such as multimodal data querying, tool usage, web interactions, and handling long documents. These capabilities pave the way for transforming LLMs from mere chatbots into general-purpose agents capable of interacting with the real world. This paper explores the concept of using a language model as the core component of an operating system (OS), effectively acting as a CPU that processes data stored in a context window, which functions as RAM. A key challenge in realizing such an LM OS is managing the life-long context and ensuring statefulness across sessions, a feature limited by the current session-based interaction paradigm due to context window size limit. To address this, we introduce compressor-retriever, a model-agnostic architecture designed for life-long context management. Unlike other long-context solutions such as retrieval-augmented generation, our approach exclusively uses the base model's forward function to compress and retrieve context, ensuring end-to-end differentiability. Preliminary experiments demonstrate the effectiveness of this architecture in in-context learning tasks, marking a step towards the development of a fully stateful LLM OS. Project repo available at: this https URL

Title: AMG: Avatar Motion Guided Video Generation

• Authors: Zhangsihao Yang, Mengyi Shan, Mohammad Farazi, Wenhui Zhu, Yanxi Chen, Xuanzhao Dong, Yalin Wang
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.01502
• Pdf URL: https://arxiv.org/pdf/2409.01502
• Copy Paste: [[2409.01502]] AMG: Avatar Motion Guided Video Generation(https://arxiv.org/abs/2409.01502)
• Keywords: diffusion, generative
• Abstract: Human video generation task has gained significant attention with the advancement of deep generative models. Generating realistic videos with human movements is challenging in nature, due to the intricacies of human body topology and sensitivity to visual artifacts. The extensively studied 2D media generation methods take advantage of massive human media datasets, but struggle with 3D-aware control; whereas 3D avatar-based approaches, while offering more freedom in control, lack photorealism and cannot be harmonized seamlessly with background scene. We propose AMG, a method that combines the 2D photorealism and 3D controllability by conditioning video diffusion models on controlled rendering of 3D avatars. We additionally introduce a novel data processing pipeline that reconstructs and renders human avatar movements from dynamic camera videos. AMG is the first method that enables multi-person diffusion video generation with precise control over camera positions, human motions, and background style. We also demonstrate through extensive evaluation that it outperforms existing human video generation methods conditioned on pose sequences or driving videos in terms of realism and adaptability.

Title: Improving Robustness of Spectrogram Classifiers with Neural Stochastic Differential Equations

• Authors: Joel Brogan, Olivera Kotevska, Anibely Torres, Sumit Jha, Mark Adams
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01532
• Pdf URL: https://arxiv.org/pdf/2409.01532
• Copy Paste: [[2409.01532]] Improving Robustness of Spectrogram Classifiers with Neural Stochastic Differential Equations(https://arxiv.org/abs/2409.01532)
• Keywords: anomaly
• Abstract: Signal analysis and classification is fraught with high levels of noise and perturbation. Computer-vision-based deep learning models applied to spectrograms have proven useful in the field of signal classification and detection; however, these methods aren't designed to handle the low signal-to-noise ratios inherent within non-vision signal processing tasks. While they are powerful, they are currently not the method of choice in the inherently noisy and dynamic critical infrastructure domain, such as smart-grid sensing, anomaly detection, and non-intrusive load monitoring.

Title: Think Twice Before Recognizing: Large Multimodal Models for General Fine-grained Traffic Sign Recognition

• Authors: Yaozong Gan, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.01534
• Pdf URL: https://arxiv.org/pdf/2409.01534
• Copy Paste: [[2409.01534]] Think Twice Before Recognizing: Large Multimodal Models for General Fine-grained Traffic Sign Recognition(https://arxiv.org/abs/2409.01534)
• Keywords: in-context
• Abstract: We propose a new strategy called think twice before recognizing to improve fine-grained traffic sign recognition (TSR). Fine-grained TSR in the wild is difficult due to the complex road conditions, and existing approaches particularly struggle with cross-country TSR when data is lacking. Our strategy achieves effective fine-grained TSR by stimulating the multiple-thinking capability of large multimodal models (LMM). We introduce context, characteristic, and differential descriptions to design multiple thinking processes for the LMM. The context descriptions with center coordinate prompt optimization help the LMM to locate the target traffic sign in the original road images containing multiple traffic signs and filter irrelevant answers through the proposed prior traffic sign hypothesis. The characteristic description is based on few-shot in-context learning of template traffic signs, which decreases the cross-domain difference and enhances the fine-grained recognition capability of the LMM. The differential descriptions of similar traffic signs optimize the multimodal thinking capability of the LMM. The proposed method is independent of training data and requires only simple and uniform instructions. We conducted extensive experiments on three benchmark datasets and two real-world datasets from different countries, and the proposed method achieves state-of-the-art TSR results on all five datasets.

Title: It is Time to Develop an Auditing Framework to Promote Value Aware Chatbots

• Authors: Yanchen Wang, Lisa Singh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.01539
• Pdf URL: https://arxiv.org/pdf/2409.01539
• Copy Paste: [[2409.01539]] It is Time to Develop an Auditing Framework to Promote Value Aware Chatbots(https://arxiv.org/abs/2409.01539)
• Keywords: generative
• Abstract: The launch of ChatGPT in November 2022 marked the beginning of a new era in AI, the availability of generative AI tools for everyone to use. ChatGPT and other similar chatbots boast a wide range of capabilities from answering student homework questions to creating music and art. Given the large amounts of human data chatbots are built on, it is inevitable that they will inherit human errors and biases. These biases have the potential to inflict significant harm or increase inequity on different subpopulations. Because chatbots do not have an inherent understanding of societal values, they may create new content that is contrary to established norms. Examples of concerning generated content includes child pornography, inaccurate facts, and discriminatory posts. In this position paper, we argue that the speed of advancement of this technology requires us, as computer and data scientists, to mobilize and develop a values-based auditing framework containing a community established standard set of measurements to monitor the health of different chatbots and LLMs. To support our argument, we use a simple audit template to share the results of basic audits we conduct that are focused on measuring potential bias in search engine style tasks, code generation, and story generation. We identify responses from GPT 3.5 and GPT 4 that are both consistent and not consistent with values derived from existing law. While the findings come as no surprise, they do underscore the urgency of developing a robust auditing framework for openly sharing results in a consistent way so that mitigation strategies can be developed by the academic community, government agencies, and companies when our values are not being adhered to. We conclude this paper with recommendations for value-based strategies for improving the technologies.

Title: Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs

• Authors: Zhuo Li, Yuhao Du, Jinpeng Hu, Xiang Wan, Anningzhe Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01552
• Pdf URL: https://arxiv.org/pdf/2409.01552
• Copy Paste: [[2409.01552]] Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs(https://arxiv.org/abs/2409.01552)
• Keywords: in-context
• Abstract: Large language models (LLMs) have shown success in generating high-quality responses. In order to achieve better alignment with LLMs with human preference, various works are proposed based on specific optimization process, which, however, is not suitable to Black-Box LLMs like GPT-4, due to inaccessible parameters. In Black-Box LLMs case, their performance is highly dependent on the quality of the provided prompts. Existing methods to enhance response quality often involve a prompt refinement model, yet these approaches potentially suffer from semantic inconsistencies between the refined and original prompts, and typically overlook the relationship between them. To address these challenges, we introduce a self-instructed in-context learning framework that empowers LLMs to deliver more effective responses by generating reliable derived prompts to construct informative contextual environments. Our approach incorporates a self-instructed reinforcement learning mechanism, enabling direct interaction with the response model during derived prompt generation for better alignment. We then formulate querying as an in-context learning task, using responses from LLMs combined with the derived prompts to establish a contextual demonstration for the original prompt. This strategy ensures alignment with the original query, reduces discrepancies from refined prompts, and maximizes the LLMs' in-context learning capability. Extensive experiments demonstrate that the proposed method not only generates more reliable derived prompts but also significantly enhances LLMs' ability to deliver more effective responses, including Black-Box models such as GPT-4.

Title: CT-SDM: A Sampling Diffusion Model for Sparse-View CT Reconstruction across All Sampling Rates

• Authors: Liutao Yang, Jiahao Huang, Guang Yang, Daoqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01571
• Pdf URL: https://arxiv.org/pdf/2409.01571
• Copy Paste: [[2409.01571]] CT-SDM: A Sampling Diffusion Model for Sparse-View CT Reconstruction across All Sampling Rates(https://arxiv.org/abs/2409.01571)
• Keywords: diffusion
• Abstract: Sparse views X-ray computed tomography has emerged as a contemporary technique to mitigate radiation dose. Because of the reduced number of projection views, traditional reconstruction methods can lead to severe artifacts. Recently, research studies utilizing deep learning methods has made promising progress in removing artifacts for Sparse-View Computed Tomography (SVCT). However, given the limitations on the generalization capability of deep learning models, current methods usually train models on fixed sampling rates, affecting the usability and flexibility of model deployment in real clinical settings. To address this issue, our study proposes a adaptive reconstruction method to achieve high-performance SVCT reconstruction at any sampling rate. Specifically, we design a novel imaging degradation operator in the proposed sampling diffusion model for SVCT (CT-SDM) to simulate the projection process in the sinogram domain. Thus, the CT-SDM can gradually add projection views to highly undersampled measurements to generalize the full-view sinograms. By choosing an appropriate starting point in diffusion inference, the proposed model can recover the full-view sinograms from any sampling rate with only one trained model. Experiments on several datasets have verified the effectiveness and robustness of our approach, demonstrating its superiority in reconstructing high-quality images from sparse-view CT scans across various sampling rates.

Title: AdaComp: Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models

• Authors: Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, Zhiming Zheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01579
• Pdf URL: https://arxiv.org/pdf/2409.01579
• Copy Paste: [[2409.01579]] AdaComp: Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models(https://arxiv.org/abs/2409.01579)
• Keywords: generative
• Abstract: Retrieved documents containing noise will hinder RAG from detecting answer clues and make the inference process slow and expensive. Therefore, context compression is necessary to enhance its accuracy and efficiency. Existing context compression methods use extractive or generative models to retain the most query-relevant sentences or apply the information bottleneck theory to preserve sufficient information. However, these methods may face issues such as over-compression or high computational costs. We observe that the retriever often ranks relevant documents at the top, but the exact number of documents needed to answer the query is uncertain due to the impact of query complexity and retrieval quality: complex queries like multi-hop questions may require retaining more documents than simpler queries, and a low-quality retrieval may need to rely on more documents to generate accurate outputs. Therefore, determining the minimum number of required documents (compression rate) is still a challenge for RAG. In this paper, we introduce AdaComp, a low-cost extractive context compression method that adaptively determines the compression rate based on both query complexity and retrieval quality. Specifically, we first annotate the minimum top-k documents necessary for the RAG system to answer the current query as the compression rate and then construct triplets of the query, retrieved documents, and its compression rate. Then, we use this triplet dataset to train a compression-rate predictor. Experiments on three QA datasets and one conversational Muiti-doc QA dataset show that AdaComp significantly reduces inference costs while maintaining performance nearly identical to uncompressed models, achieving a balance between efficiency and performance.

Title: DiVE: DiT-based Video Generation with Enhanced Control

• Authors: Junpeng Jiang, Gangyi Hong, Lijun Zhou, Enhui Ma, Hengtong Hu, Xia Zhou, Jie Xiang, Fan Liu, Kaicheng Yu, Haiyang Sun, Kun Zhan, Peng Jia, Miao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01595
• Pdf URL: https://arxiv.org/pdf/2409.01595
• Copy Paste: [[2409.01595]] DiVE: DiT-based Video Generation with Enhanced Control(https://arxiv.org/abs/2409.01595)
• Keywords: diffusion
• Abstract: Generating high-fidelity, temporally consistent videos in autonomous driving scenarios faces a significant challenge, e.g. problematic maneuvers in corner cases. Despite recent video generation works are proposed to tackcle the mentioned problem, i.e. models built on top of Diffusion Transformers (DiT), works are still missing which are targeted on exploring the potential for multi-view videos generation scenarios. Noticeably, we propose the first DiT-based framework specifically designed for generating temporally and multi-view consistent videos which precisely match the given bird's-eye view layouts control. Specifically, the proposed framework leverages a parameter-free spatial view-inflated attention mechanism to guarantee the cross-view consistency, where joint cross-attention modules and ControlNet-Transformer are integrated to further improve the precision of control. To demonstrate our advantages, we extensively investigate the qualitative comparisons on nuScenes dataset, particularly in some most challenging corner cases. In summary, the effectiveness of our proposed method in producing long, controllable, and highly consistent videos under difficult conditions is proven to be effective.

Title: A Time-Intensity Aware Pipeline for Generating Late-Stage Breast DCE-MRI using Generative Adversarial Models

• Authors: Ruben D. Fonnegra, Maria Liliana Hernández, Juan C. Caicedo, Gloria M. Díaz
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.01596
• Pdf URL: https://arxiv.org/pdf/2409.01596
• Copy Paste: [[2409.01596]] A Time-Intensity Aware Pipeline for Generating Late-Stage Breast DCE-MRI using Generative Adversarial Models(https://arxiv.org/abs/2409.01596)
• Keywords: generative
• Abstract: Contrast-enhancement pattern analysis is critical in breast magnetic resonance imaging (MRI) to distinguish benign from probably malignant tumors. However, contrast-enhanced image acquisitions are time-consuming and very expensive. As an alternative to physical acquisition, this paper proposes a comprehensive pipeline for the generation of accurate long-term (late) contrast-enhanced breast MRI from the early counterpart. The proposed strategy focuses on preserving the contrast agent pattern in the enhanced regions while maintaining visual properties in the entire synthesized images. To that end, a novel loss function that leverages the biological behavior of contrast agent (CA) in tissue, given by the Time-Intensity (TI) enhancement curve, is proposed to optimize a pixel-attention based generative model. In addition, unlike traditional normalization and standardization methods, we developed a new normalization strategy that maintains the contrast enhancement pattern across the image sequences at multiple timestamps. This ensures the prevalence of the CA pattern after image preprocessing, unlike conventional approaches. Furthermore, in order to objectively evaluate the clinical quality of the synthesized images, two metrics are also introduced to measure the differences between the TI curves of enhanced regions of the acquired and synthesized images. The experimental results showed that the proposed strategy generates images that significantly outperform diagnostic quality in contrast-enhanced regions while maintaining the spatial features of the entire image. This results suggest a potential use of synthetic late enhanced images generated via deep learning in clinical scenarios.

Title: Data-driven topology design based on principal component analysis for 3D structural design problems

• Authors: Jun Yang, Kentaro Yaji, Shintaro Yamasaki
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2409.01607
• Pdf URL: https://arxiv.org/pdf/2409.01607
• Copy Paste: [[2409.01607]] Data-driven topology design based on principal component analysis for 3D structural design problems(https://arxiv.org/abs/2409.01607)
• Keywords: generative
• Abstract: Topology optimization is a structural design methodology widely utilized to address engineering challenges. However, sensitivity-based topology optimization methods struggle to solve optimization problems characterized by strong non-linearity. Leveraging the sensitivity-free nature and high capacity of deep generative models, data-driven topology design (DDTD) methodology is considered an effective solution to this problem. Despite this, the training effectiveness of deep generative models diminishes when input size exceeds a threshold while maintaining high degrees of freedom is crucial for accurately characterizing complex structures. To resolve the conflict between the both, we propose DDTD based on principal component analysis (PCA). Its core idea is to replace the direct training of deep generative models with material distributions by using a principal component score matrix obtained from PCA computation and to obtain the generated material distributions with new features through the restoration process. We apply the proposed PCA-based DDTD to the problem of minimizing the maximum stress in 3D structural mechanics and demonstrate it can effectively address the current challenges faced by DDTD that fail to handle 3D structural design problems. Various experiments are conducted to demonstrate the effectiveness and practicability of the proposed PCA-based DDTD.

Title: CTG-KrEW: Generating Synthetic Structured Contextually Correlated Content by Conditional Tabular GAN with K-Means Clustering and Efficient Word Embedding

• Authors: Riya Samanta, Bidyut Saha, Soumya K. Ghosh, Sajal K. Das
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.01628
• Pdf URL: https://arxiv.org/pdf/2409.01628
• Copy Paste: [[2409.01628]] CTG-KrEW: Generating Synthetic Structured Contextually Correlated Content by Conditional Tabular GAN with K-Means Clustering and Efficient Word Embedding(https://arxiv.org/abs/2409.01628)
• Keywords: generative
• Abstract: Conditional Tabular Generative Adversarial Networks (CTGAN) and their various derivatives are attractive for their ability to efficiently and flexibly create synthetic tabular data, showcasing strong performance and adaptability. However, there are certain critical limitations to such models. The first is their inability to preserve the semantic integrity of contextually correlated words or phrases. For instance, skillset in freelancer profiles is one such attribute where individual skills are semantically interconnected and indicative of specific domain interests or qualifications. The second challenge of traditional approaches is that, when applied to generate contextually correlated tabular content, besides generating semantically shallow content, they consume huge memory resources and CPU time during the training stage. To address these problems, we introduce a novel framework, CTGKrEW (Conditional Tabular GAN with KMeans Clustering and Word Embedding), which is adept at generating realistic synthetic tabular data where attributes are collections of semantically and contextually coherent words. CTGKrEW is trained and evaluated using a dataset from Upwork, a realworld freelancing platform. Comprehensive experiments were conducted to analyze the variability, contextual similarity, frequency distribution, and associativity of the generated data, along with testing the framework's system feasibility. CTGKrEW also takes around 99\% less CPU time and 33\% less memory footprints than the conventional approach. Furthermore, we developed KrEW, a web application to facilitate the generation of realistic data containing skill-related information. This application, available at this https URL, is freely accessible to both the general public and the research community.

Title: Unveiling Advanced Frequency Disentanglement Paradigm for Low-Light Image Enhancement

• Authors: Kun Zhou, Xinyu Lin, Wenbo Li, Xiaogang Xu, Yuanhao Cai, Zhonghang Liu, Xiaoguang Han, Jiangbo Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01641
• Pdf URL: https://arxiv.org/pdf/2409.01641
• Copy Paste: [[2409.01641]] Unveiling Advanced Frequency Disentanglement Paradigm for Low-Light Image Enhancement(https://arxiv.org/abs/2409.01641)
• Keywords: diffusion
• Abstract: Previous low-light image enhancement (LLIE) approaches, while employing frequency decomposition techniques to address the intertwined challenges of low frequency (e.g., illumination recovery) and high frequency (e.g., noise reduction), primarily focused on the development of dedicated and complex networks to achieve improved performance. In contrast, we reveal that an advanced disentanglement paradigm is sufficient to consistently enhance state-of-the-art methods with minimal computational overhead. Leveraging the image Laplace decomposition scheme, we propose a novel low-frequency consistency method, facilitating improved frequency disentanglement optimization. Our method, seamlessly integrating with various models such as CNNs, Transformers, and flow-based and diffusion models, demonstrates remarkable adaptability. Noteworthy improvements are showcased across five popular benchmarks, with up to 7.68dB gains on PSNR achieved for six state-of-the-art models. Impressively, our approach maintains efficiency with only 88K extra parameters, setting a new standard in the challenging realm of low-light image enhancement.

Title: Classifier-Free Diffusion-Based Weakly-Supervised Approach for Health Indicator Derivation in Rotating Machines: Advancing Early Fault Detection and Condition Monitoring

• Authors: Wenyang Hu, Gaetan Frusque, Tianyang Wang, Fulei Chu, Olga Fink
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.01676
• Pdf URL: https://arxiv.org/pdf/2409.01676
• Copy Paste: [[2409.01676]] Classifier-Free Diffusion-Based Weakly-Supervised Approach for Health Indicator Derivation in Rotating Machines: Advancing Early Fault Detection and Condition Monitoring(https://arxiv.org/abs/2409.01676)
• Keywords: diffusion, anomaly
• Abstract: Deriving health indicators of rotating machines is crucial for their maintenance. However, this process is challenging for the prevalent adopted intelligent methods since they may take the whole data distributions, not only introducing noise interference but also lacking the explainability. To address these issues, we propose a diffusion-based weakly-supervised approach for deriving health indicators of rotating machines, enabling early fault detection and continuous monitoring of condition evolution. This approach relies on a classifier-free diffusion model trained using healthy samples and a few anomalies. This model generates healthy samples. and by comparing the differences between the original samples and the generated ones in the envelope spectrum, we construct an anomaly map that clearly identifies faults. Health indicators are then derived, which can explain the fault types and mitigate noise interference. Comparative studies on two cases demonstrate that the proposed method offers superior health monitoring effectiveness and robustness compared to baseline models.

Title: Interpreting Outliers in Time Series Data through Decoding Autoencoder

• Authors: Patrick Knab, Sascha Marton, Christian Bartelt, Robert Fuder
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01713
• Pdf URL: https://arxiv.org/pdf/2409.01713
• Copy Paste: [[2409.01713]] Interpreting Outliers in Time Series Data through Decoding Autoencoder(https://arxiv.org/abs/2409.01713)
• Keywords: anomaly
• Abstract: Outlier detection is a crucial analytical tool in various fields. In critical systems like manufacturing, malfunctioning outlier detection can be costly and safety-critical. Therefore, there is a significant need for explainable artificial intelligence (XAI) when deploying opaque models in such environments. This study focuses on manufacturing time series data from a German automotive supply industry. We utilize autoencoders to compress the entire time series and then apply anomaly detection techniques to its latent features. For outlier interpretation, we (i) adopt widely used XAI techniques to the autoencoder's encoder. Additionally, (ii) we propose AEE, Aggregated Explanatory Ensemble, a novel approach that fuses explanations of multiple XAI techniques into a single, more expressive interpretation. For evaluation of explanations, (iii) we propose a technique to measure the quality of encoder explanations quantitatively. Furthermore, we qualitatively assess the effectiveness of outlier explanations with domain expertise.

Title: State-of-the-art Advances of Deep-learning Linguistic Steganalysis Research

• Authors: Yihao Wang, Ru Zhang, Yifan Tang, Jianyi Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.01780
• Pdf URL: https://arxiv.org/pdf/2409.01780
• Copy Paste: [[2409.01780]] State-of-the-art Advances of Deep-learning Linguistic Steganalysis Research(https://arxiv.org/abs/2409.01780)
• Keywords: generative
• Abstract: With the evolution of generative linguistic steganography techniques, conventional steganalysis falls short in robustly quantifying the alterations induced by steganography, thereby complicating detection. Consequently, the research paradigm has pivoted towards deep-learning-based linguistic steganalysis. This study offers a comprehensive review of existing contributions and evaluates prevailing developmental trajectories. Specifically, we first provided a formalized exposition of the general formulas for linguistic steganalysis, while comparing the differences between this field and the domain of text classification. Subsequently, we classified the existing work into two levels based on vector space mapping and feature extraction models, thereby comparing the research motivations, model advantages, and other details. A comparative analysis of the experiments is conducted to assess the performances. Finally, the challenges faced by this field are discussed, and several directions for future development and key issues that urgently need to be addressed are proposed.

Title: LLM-GAN: Construct Generative Adversarial Network Through Large Language Models For Explainable Fake News Detection

• Authors: Yifeng Wang, Zhouhong Gu, Siwei Zhang, Suhang Zheng, Tao Wang, Tianyu Li, Hongwei Feng, Yanghua Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.01787
• Pdf URL: https://arxiv.org/pdf/2409.01787
• Copy Paste: [[2409.01787]] LLM-GAN: Construct Generative Adversarial Network Through Large Language Models For Explainable Fake News Detection(https://arxiv.org/abs/2409.01787)
• Keywords: generative
• Abstract: Explainable fake news detection predicts the authenticity of news items with annotated explanations. Today, Large Language Models (LLMs) are known for their powerful natural language understanding and explanation generation abilities. However, presenting LLMs for explainable fake news detection remains two main challenges. Firstly, fake news appears reasonable and could easily mislead LLMs, leaving them unable to understand the complex news-faking process. Secondly, utilizing LLMs for this task would generate both correct and incorrect explanations, which necessitates abundant labor in the loop. In this paper, we propose LLM-GAN, a novel framework that utilizes prompting mechanisms to enable an LLM to become Generator and Detector and for realistic fake news generation and detection. Our results demonstrate LLM-GAN's effectiveness in both prediction performance and explanation quality. We further showcase the integration of LLM-GAN to a cloud-native AI platform to provide better fake news detection service in the cloud.

Title: Towards Generative Class Prompt Learning for Few-shot Visual Recognition

• Authors: Soumitri Chattopadhyay, Sanket Biswas, Emanuele Vivoli, Josep Lladós
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.01835
• Pdf URL: https://arxiv.org/pdf/2409.01835
• Copy Paste: [[2409.01835]] Towards Generative Class Prompt Learning for Few-shot Visual Recognition(https://arxiv.org/abs/2409.01835)
• Keywords: diffusion, generative
• Abstract: Although foundational vision-language models (VLMs) have proven to be very successful for various semantic discrimination tasks, they still struggle to perform faithfully for fine-grained categorization. Moreover, foundational models trained on one domain do not generalize well on a different domain without fine-tuning. We attribute these to the limitations of the VLM's semantic representations and attempt to improve their fine-grained visual awareness using generative modeling. Specifically, we propose two novel methods: Generative Class Prompt Learning (GCPL) and Contrastive Multi-class Prompt Learning (CoMPLe). Utilizing text-to-image diffusion models, GCPL significantly improves the visio-linguistic synergy in class embeddings by conditioning on few-shot exemplars with learnable class prompts. CoMPLe builds on this foundation by introducing a contrastive learning component that encourages inter-class separation during the generative optimization process. Our empirical results demonstrate that such a generative class prompt learning approach substantially outperform existing methods, offering a better alternative to few shot image recognition challenges. The source code will be made available at: this https URL.

Title: CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention

• Authors: Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang, Yanbo Zheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.01876
• Pdf URL: https://arxiv.org/pdf/2409.01876
• Copy Paste: [[2409.01876]] CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention(https://arxiv.org/abs/2409.01876)
• Keywords: diffusion
• Abstract: Diffusion-based video generation technology has advanced significantly, catalyzing a proliferation of research in human animation. However, the majority of these studies are confined to same-modality driving settings, with cross-modality human body animation remaining relatively underexplored. In this paper, we introduce, an end-to-end audio-driven human animation framework that ensures hand integrity, identity consistency, and natural motion. The key design of CyberHost is the Region Codebook Attention mechanism, which improves the generation quality of facial and hand animations by integrating fine-grained local features with learned motion pattern priors. Furthermore, we have developed a suite of human-prior-guided training strategies, including body movement map, hand clarity score, pose-aligned reference feature, and local enhancement supervision, to improve synthesis results. To our knowledge, CyberHost is the first end-to-end audio-driven human diffusion model capable of facilitating zero-shot video generation within the scope of human body. Extensive experiments demonstrate that CyberHost surpasses previous works in both quantitative and qualitative aspects.

Title: Map-Assisted Remote-Sensing Image Compression at Extremely Low Bitrates

• Authors: Yixuan Ye, Ce Wang, Wanjie Sun, Zhenzhong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.01935
• Pdf URL: https://arxiv.org/pdf/2409.01935
• Copy Paste: [[2409.01935]] Map-Assisted Remote-Sensing Image Compression at Extremely Low Bitrates(https://arxiv.org/abs/2409.01935)
• Keywords: diffusion, generative
• Abstract: Remote-sensing (RS) image compression at extremely low bitrates has always been a challenging task in practical scenarios like edge device storage and narrow bandwidth transmission. Generative models including VAEs and GANs have been explored to compress RS images into extremely low-bitrate streams. However, these generative models struggle to reconstruct visually plausible images due to the highly ill-posed nature of extremely low-bitrate image compression. To this end, we propose an image compression framework that utilizes a pre-trained diffusion model with powerful natural image priors to achieve high-realism reconstructions. However, diffusion models tend to hallucinate small structures and textures due to the significant information loss at limited bitrates. Thus, we introduce vector maps as semantic and structural guidance and propose a novel image compression approach named Map-Assisted Generative Compression (MAGC). MAGC employs a two-stage pipeline to compress and decompress RS images at extremely low bitrates. The first stage maps an image into a latent representation, which is then further compressed in a VAE architecture to save bitrates and serves as implicit guidance in the subsequent diffusion process. The second stage conducts a conditional diffusion model to generate a visually pleasing and semantically accurate result using implicit guidance and explicit semantic guidance. Quantitative and qualitative comparisons show that our method outperforms standard codecs and other learning-based methods in terms of perceptual quality and semantic accuracy. The dataset and code will be publicly available at this https URL.

Title: Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey

• Authors: Ruiyao Xu, Kaize Ding
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.01980
• Pdf URL: https://arxiv.org/pdf/2409.01980
• Copy Paste: [[2409.01980]] Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey(https://arxiv.org/abs/2409.01980)
• Keywords: generative, anomaly
• Abstract: Detecting anomalies or out-of-distribution (OOD) samples is critical for maintaining the reliability and trustworthiness of machine learning systems. Recently, Large Language Models (LLMs) have demonstrated their effectiveness not only in natural language processing but also in broader applications due to their advanced comprehension and generative capabilities. The integration of LLMs into anomaly and OOD detection marks a significant shift from the traditional paradigm in the field. This survey focuses on the problem of anomaly and OOD detection under the context of LLMs. We propose a new taxonomy to categorize existing approaches into three classes based on the role played by LLMs. Following our proposed taxonomy, we further discuss the related work under each of the categories and finally discuss potential challenges and directions for future research in this field. We also provide an up-to-date reading list of relevant papers.

Title: PMT-MAE: Dual-Branch Self-Supervised Learning with Distillation for Efficient Point Cloud Classification

• Authors: Qiang Zheng, Chao Zhang, Jian Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.02007
• Pdf URL: https://arxiv.org/pdf/2409.02007
• Copy Paste: [[2409.02007]] PMT-MAE: Dual-Branch Self-Supervised Learning with Distillation for Efficient Point Cloud Classification(https://arxiv.org/abs/2409.02007)
• Keywords: self-supervised
• Abstract: Advances in self-supervised learning are essential for enhancing feature extraction and understanding in point cloud processing. This paper introduces PMT-MAE (Point MLP-Transformer Masked Autoencoder), a novel self-supervised learning framework for point cloud classification. PMT-MAE features a dual-branch architecture that integrates Transformer and MLP components to capture rich features. The Transformer branch leverages global self-attention for intricate feature interactions, while the parallel MLP branch processes tokens through shared fully connected layers, offering a complementary feature transformation pathway. A fusion mechanism then combines these features, enhancing the model's capacity to learn comprehensive 3D representations. Guided by the sophisticated teacher model Point-M2AE, PMT-MAE employs a distillation strategy that includes feature distillation during pre-training and logit distillation during fine-tuning, ensuring effective knowledge transfer. On the ModelNet40 classification task, achieving an accuracy of 93.6\% without employing voting strategy, PMT-MAE surpasses the baseline Point-MAE (93.2\%) and the teacher Point-M2AE (93.4\%), underscoring its ability to learn discriminative 3D point cloud representations. Additionally, this framework demonstrates high efficiency, requiring only 40 epochs for both pre-training and fine-tuning. PMT-MAE's effectiveness and efficiency render it well-suited for scenarios with limited computational resources, positioning it as a promising solution for practical point cloud analysis.

Title: ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

• Authors: Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, Yonghong Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.02048
• Pdf URL: https://arxiv.org/pdf/2409.02048
• Copy Paste: [[2409.02048]] ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis(https://arxiv.org/abs/2409.02048)
• Keywords: diffusion
• Abstract: Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. In this work, we propose \textbf{ViewCrafter}, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images with the prior of video diffusion model. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames with precise camera pose control. To further enlarge the generation range of novel views, we tailored an iterative view synthesis strategy together with a camera trajectory planning algorithm to progressively extend the 3D clues and the areas covered by the novel views. With ViewCrafter, we can facilitate various applications, such as immersive experiences with real-time rendering by efficiently optimizing a 3D-GS representation using the reconstructed 3D points and the generated novel views, and scene-level text-to-3D generation for more imaginative content creation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in synthesizing high-fidelity and consistent novel views.

Title: Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text

• Authors: Michael Burnham, Kayla Kahn, Ryan Yank Wang, Rachel X. Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.02078
• Pdf URL: https://arxiv.org/pdf/2409.02078
• Copy Paste: [[2409.02078]] Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text(https://arxiv.org/abs/2409.02078)
• Keywords: generative
• Abstract: Social scientists quickly adopted large language models due to their ability to annotate documents without supervised training, an ability known as zero-shot learning. However, due to their compute demands, cost, and often proprietary nature, these models are often at odds with replication and open science standards. This paper introduces the Political DEBATE (DeBERTa Algorithm for Textual Entailment) language models for zero-shot and few-shot classification of political documents. These models are not only as good, or better than, state-of-the art large language models at zero and few-shot classification, but are orders of magnitude more efficient and completely open source. By training the models on a simple random sample of 10-25 documents, they can outperform supervised classifiers trained on hundreds or thousands of documents and state-of-the-art generative models with complex, engineered prompts. Additionally, we release the PolNLI dataset used to train these models -- a corpus of over 200,000 political documents with highly accurate labels across over 800 classification tasks.

Title: DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos

• Authors: Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, Ying Shan
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.02095
• Pdf URL: https://arxiv.org/pdf/2409.02095
• Copy Paste: [[2409.02095]] DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos(https://arxiv.org/abs/2409.02095)
• Keywords: diffusion
• Abstract: Despite significant advancements in monocular depth estimation for static images, estimating video depth in the open world remains challenging, since open-world videos are extremely diverse in content, motion, camera movement, and length. We present DepthCrafter, an innovative method for generating temporally consistent long depth sequences with intricate details for open-world videos, without requiring any supplementary information such as camera poses or optical flow. DepthCrafter achieves generalization ability to open-world videos by training a video-to-depth model from a pre-trained image-to-video diffusion model, through our meticulously designed three-stage training strategy with the compiled paired video-depth datasets. Our training approach enables the model to generate depth sequences with variable lengths at one time, up to 110 frames, and harvest both precise depth details and rich content diversity from realistic and synthetic datasets. We also propose an inference strategy that processes extremely long videos through segment-wise estimation and seamless stitching. Comprehensive evaluations on multiple datasets reveal that DepthCrafter achieves state-of-the-art performance in open-world video depth estimation under zero-shot settings. Furthermore, DepthCrafter facilitates various downstream applications, including depth-based visual effects and conditional video generation.

Title: LinFusion: 1 GPU, 1 Minute, 16K Image

• Authors: Songhua Liu, Weihao Yu, Zhenxiong Tan, Xinchao Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.02097
• Pdf URL: https://arxiv.org/pdf/2409.02097
• Copy Paste: [[2409.02097]] LinFusion: 1 GPU, 1 Minute, 16K Image(https://arxiv.org/abs/2409.02097)
• Keywords: diffusion
• Abstract: Modern diffusion models, particularly those utilizing a Transformer-based UNet for denoising, rely heavily on self-attention operations to manage complex spatial relationships, thus achieving impressive generation performance. However, this existing paradigm faces significant challenges in generating high-resolution visual content due to its quadratic time and memory complexity with respect to the number of spatial tokens. To address this limitation, we aim at a novel linear attention mechanism as an alternative in this paper. Specifically, we begin our exploration from recently introduced models with linear complexity, e.g., Mamba, Mamba2, and Gated Linear Attention, and identify two key features-attention normalization and non-causal inference-that enhance high-resolution visual generation performance. Building on these insights, we introduce a generalized linear attention paradigm, which serves as a low-rank approximation of a wide spectrum of popular linear token mixers. To save the training cost and better leverage pre-trained models, we initialize our models and distill the knowledge from pre-trained StableDiffusion (SD). We find that the distilled model, termed LinFusion, achieves performance on par with or superior to the original SD after only modest training, while significantly reducing time and memory complexity. Extensive experiments on SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion delivers satisfactory zero-shot cross-resolution generation performance, generating high-resolution images like 16K resolution. Moreover, it is highly compatible with pre-trained SD components, such as ControlNet and IP-Adapter, requiring no adaptation efforts. Codes are available at this https URL.