2024-05-16

Title: Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Video

• Authors: Tomoya Sugihara, Shuntaro Masuda, Ling Xiao, Toshihiko Yamasaki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Video(https://arxiv.org/abs/)
• Keywords: self-supervised
• Abstract: Current video summarization methods primarily depend on supervised computer vision techniques, which demands time-consuming manual annotations. Further, the annotations are always subjective which make this task more challenging. To address these issues, we analyzed the feasibility in transforming the video summarization into a text summary task and leverage Large Language Models (LLMs) to boost video summarization. This paper proposes a novel self-supervised framework for video summarization guided by LLMs. Our method begins by generating captions for video frames, which are then synthesized into text summaries by LLMs. Subsequently, we measure semantic distance between the frame captions and the text summary. It's worth noting that we propose a novel loss function to optimize our model according to the diversity of the video. Finally, the summarized video can be generated by selecting the frames whose captions are similar with the text summary. Our model achieves competitive results against other state-of-the-art methods and paves a novel pathway in video summarization.

Title: CLIP with Quality Captions: A Strong Pretraining for Vision Tasks

• Authors: Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Oncel Tuzel
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] CLIP with Quality Captions: A Strong Pretraining for Vision Tasks(https://arxiv.org/abs/)
• Keywords: self-supervised
• Abstract: CLIP models perform remarkably well on zero-shot classification and retrieval tasks. But recent studies have shown that learnt representations in CLIP are not well suited for dense prediction tasks like object detection, semantic segmentation or depth estimation. More recently, multi-stage training methods for CLIP models was introduced to mitigate the weak performance of CLIP on downstream tasks. In this work, we find that simply improving the quality of captions in image-text datasets improves the quality of CLIP's visual representations, resulting in significant improvement on downstream dense prediction vision tasks. In fact, we find that CLIP pretraining with good quality captions can surpass recent supervised, self-supervised and weakly supervised pretraining methods. We show that when CLIP model with ViT-B/16 as image encoder is trained on well aligned image-text pairs it obtains 12.1% higher mIoU and 11.5% lower RMSE on semantic segmentation and depth estimation tasks over recent state-of-the-art Masked Image Modeling (MIM) pretraining methods like Masked Autoencoder (MAE). We find that mobile architectures also benefit significantly from CLIP pretraining. A recent mobile vision architecture, MCi2, with CLIP pretraining obtains similar performance as Swin-L, pretrained on ImageNet-22k for semantic segmentation task while being 6.1$\times$ smaller. Moreover, we show that improving caption quality results in $10\times$ data efficiency when finetuning for dense prediction tasks.

Title: Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysis

• Authors: Alexandre Englebert, Anne-Sophie Collin, Olivier Cornu, Christophe De Vleeschouwer
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysis(https://arxiv.org/abs/)
• Keywords: self-supervised, anomaly
• Abstract: This paper proposes leveraging vision-language pretraining on bone X-rays paired with French reports to address downstream tasks of interest on bone radiography. A practical processing pipeline is introduced to anonymize and process French medical reports. Pretraining then consists in the self-supervised alignment of visual and textual embedding spaces derived from deep model encoders. The resulting image encoder is then used to handle various downstream tasks, including quantification of osteoarthritis, estimation of bone age on pediatric wrists, bone fracture and anomaly detection. Our approach demonstrates competitive performance on downstream tasks, compared to alternatives requiring a significantly larger amount of human expert annotations. Our work stands as the first study to integrate French reports to shape the embedding space devoted to bone X-Rays representations, capitalizing on the large quantity of paired images and reports data available in an hospital. By relying on generic vision-laguage deep models in a language-specific scenario, it contributes to the deployement of vision models for wider healthcare applications.

Title: Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis

• Authors: Yao Fu
• Subjects: cs.LG, cs.AI, cs.CL, cs.DC
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: Transformer-based long context generative models power emerging AI applications like hour-long video understanding and project-level coding agent. Deploying long context transformers (e.g., 100K to 10M tokens) is prohibitively expensive compared to short context (e.g., 4K tokens) model variants. Reducing the cost of long-context transformers is becoming a pressing research and engineering challenge starting from the year of 2024. This work describes a concurrent programming framework for quantitatively analyzing the efficiency challenges in serving multiple long-context requests under limited size of GPU high-bandwidth memory (HBM) regime. We give a detailed analysis of how all additional computational costs, compared to 4K context, trace back to \textit{one single source: the large size of the KV cache}. We use a 34B GPT-3.5 level model of 50K context on A100 NVLink as a running example, and describe how its large KV cache causes four types of deployment challenges: (1) prefilling long inputs takes much longer compute time and GPU memory than short inputs; (2) after prefilling, the large KV cache residing on the GPU HBM substantially restricts the number of concurrent users being served; (3) during decoding, repeatedly reading the KV cache from HBM to SM largely increases latency; (4) when KV cache memory overflows, swapping it from HBM to DDR causes significant context switching latency. We use this framework to analyze existing works and identify possibilities of combining them to build end-to-end systems. Overall, this work offers a foundational framework for analyzing long context transformer deployment and identifies directions towards reducing the inference cost of 1M context to be as cheap as 4K.

Title: Deep Learning in Earthquake Engineering: A Comprehensive Review

• Authors: Yazhou Xie
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Deep Learning in Earthquake Engineering: A Comprehensive Review(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: This article surveys the growing interest in utilizing Deep Learning (DL) as a powerful tool to address challenging problems in earthquake engineering. Despite decades of advancement in domain knowledge, issues such as uncertainty in earthquake occurrence, unpredictable seismic loads, nonlinear structural responses, and community engagement remain difficult to tackle using domain-specific methods. DL offers promising solutions by leveraging its data-driven capacity for nonlinear mapping, sequential data modeling, automatic feature extraction, dimensionality reduction, optimal decision-making, etc. However, the literature lacks a comprehensive review that systematically covers a consistent scope intersecting DL and earthquake engineering. To bridge the gap, the article first discusses methodological advances to elucidate various applicable DL techniques, such as multi-layer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), autoencoder (AE), transfer learning (TL), reinforcement learning (RL), and graph neural network (GNN). A thorough research landscape is then disclosed by exploring various DL applications across different research topics, including vision-based seismic damage assessment and structural characterization, seismic demand and damage state prediction, seismic response history prediction, regional seismic risk assessment and community resilience, ground motion (GM) for engineering use, seismic response control, and the inverse problem of system/damage identification. Suitable DL techniques for each research topic are identified, emphasizing the preeminence of CNN for vision-based tasks, RNN for sequential data, RL for community resilience, and unsupervised learning for GM analysis. The article also discusses opportunities and challenges for leveraging DL in earthquake engineering research and practice.

Title: SMART: Towards Pre-trained Missing-Aware Model for Patient Health Status Prediction

• Authors: Zhihao Yu, Xu Chu, Yujie Jin, Yasha Wang, Junfeng Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] SMART: Towards Pre-trained Missing-Aware Model for Patient Health Status Prediction(https://arxiv.org/abs/)
• Keywords: self-supervised
• Abstract: Electronic health record (EHR) data has emerged as a valuable resource for analyzing patient health status. However, the prevalence of missing data in EHR poses significant challenges to existing methods, leading to spurious correlations and suboptimal predictions. While various imputation techniques have been developed to address this issue, they often obsess unnecessary details and may introduce additional noise when making clinical predictions. To tackle this problem, we propose SMART, a Self-Supervised Missing-Aware RepresenTation Learning approach for patient health status prediction, which encodes missing information via elaborated attentions and learns to impute missing values through a novel self-supervised pre-training approach that reconstructs missing data representations in the latent space. By adopting missing-aware attentions and focusing on learning higher-order representations, SMART promotes better generalization and robustness to missing data. We validate the effectiveness of SMART through extensive experiments on six EHR tasks, demonstrating its superiority over state-of-the-art methods.

Title: CTS: A Consistency-Based Medical Image Segmentation Model

• Authors: Kejia Zhang, Lan Zhang, Haiwei Pan, Baolong Yu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] CTS: A Consistency-Based Medical Image Segmentation Model(https://arxiv.org/abs/)
• Keywords: diffusion, generative
• Abstract: In medical image segmentation tasks, diffusion models have shown significant potential. However, mainstream diffusion models suffer from drawbacks such as multiple sampling times and slow prediction results. Recently, consistency models, as a standalone generative network, have resolved this issue. Compared to diffusion models, consistency models can reduce the sampling times to once, not only achieving similar generative effects but also significantly speeding up training and prediction. However, they are not suitable for image segmentation tasks, and their application in the medical imaging field has not yet been explored. Therefore, this paper applies the consistency model to medical image segmentation tasks, designing multi-scale feature signal supervision modes and loss function guidance to achieve model convergence. Experiments have verified that the CTS model can obtain better medical image segmentation results with a single sampling during the test phase.

Title: Response Matching for generating materials and molecules

• Authors: Bingqing Cheng
• Subjects: cs.LG, cond-mat.mtrl-sci, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Response Matching for generating materials and molecules(https://arxiv.org/abs/)
• Keywords: diffusion, generative
• Abstract: Machine learning has recently emerged as a powerful tool for generating new molecular and material structures. The success of state-of-the-art models stems from their ability to incorporate physical symmetries, such as translation, rotation, and periodicity. Here, we present a novel generative method called Response Matching (RM), which leverages the fact that each stable material or molecule exists at the minimum of its potential energy surface. Consequently, any perturbation induces a response in energy and stress, driving the structure back to equilibrium. Matching to such response is closely related to score matching in diffusion models. By employing the combination of a machine learning interatomic potential and random structure search as the denoising model, RM exploits the locality of atomic interactions, and inherently respects permutation, translation, rotation, and periodic invariances. RM is the first model to handle both molecules and bulk materials under the same framework. We demonstrate the efficiency and generalization of RM across three systems: a small organic molecular dataset, stable crystals from the Materials Project, and one-shot learning on a single diamond configuration.

Title: RSHazeDiff: A Unified Fourier-aware Diffusion Model for Remote Sensing Image Dehazing

• Authors: Jiamei Xiong, Xuefeng Yan, Yongzhen Wang, Wei Zhao, Xiao-Ping Zhang, Mingqiang Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] RSHazeDiff: A Unified Fourier-aware Diffusion Model for Remote Sensing Image Dehazing(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: Haze severely degrades the visual quality of remote sensing images and hampers the performance of automotive navigation, intelligent monitoring, and urban management. The emerging denoising diffusion probabilistic model (DDPM) exhibits the significant potential for dense haze removal with its strong generation ability. Since remote sensing images contain extensive small-scale texture structures, it is important to effectively restore image details from hazy images. However, current wisdom of DDPM fails to preserve image details and color fidelity well, limiting its dehazing capacity for remote sensing images. In this paper, we propose a novel unified Fourier-aware diffusion model for remote sensing image dehazing, termed RSHazeDiff. From a new perspective, RSHazeDiff explores the conditional DDPM to improve image quality in dense hazy scenarios, and it makes three key contributions. First, RSHazeDiff refines the training phase of diffusion process by performing noise estimation and reconstruction constraints in a coarse-to-fine fashion. Thus, it remedies the unpleasing results caused by the simple noise estimation constraint in DDPM. Second, by taking the frequency information as important prior knowledge during iterative sampling steps, RSHazeDiff can preserve more texture details and color fidelity in dehazed images. Third, we design a global compensated learning module to utilize the Fourier transform to capture the global dependency features of input images, which can effectively mitigate the effects of boundary artifacts when processing fixed-size patches. Experiments on both synthetic and real-world benchmarks validate the favorable performance of RSHazeDiff over multiple state-of-the-art methods. Source code will be released at this https URL.

Title: Towards Next-Generation Steganalysis: LLMs Unleash the Power of Detecting Steganography

• Authors: Minhao Bai. Jinshuai Yang, Kaiyi Pang, Huili Wang, Yongfeng Huang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Towards Next-Generation Steganalysis: LLMs Unleash the Power of Detecting Steganography(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: Linguistic steganography provides convenient implementation to hide messages, particularly with the emergence of AI generation technology. The potential abuse of this technology raises security concerns within societies, calling for powerful linguistic steganalysis to detect carrier containing steganographic messages. Existing methods are limited to finding distribution differences between steganographic texts and normal texts from the aspect of symbolic statistics. However, the distribution differences of both kinds of texts are hard to build precisely, which heavily hurts the detection ability of the existing methods in realistic scenarios. To seek a feasible way to construct practical steganalysis in real world, this paper propose to employ human-like text processing abilities of large language models (LLMs) to realize the difference from the aspect of human perception, addition to traditional statistic aspect. Specifically, we systematically investigate the performance of LLMs in this task by modeling it as a generative paradigm, instead of traditional classification paradigm. Extensive experiment results reveal that generative LLMs exhibit significant advantages in linguistic steganalysis and demonstrate performance trends distinct from traditional approaches. Results also reveal that LLMs outperform existing baselines by a wide margin, and the domain-agnostic ability of LLMs makes it possible to train a generic steganalysis model (Both codes and trained models are openly available in this https URL).

Title: SOEDiff: Efficient Distillation for Small Object Editing

• Authors: Qihe Pan, Zicheng Wang, Zhen Zhao, Yiming Wu, Sifan Long, Haoran Liang, Ronghua Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] SOEDiff: Efficient Distillation for Small Object Editing(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: In this paper, we delve into a new task known as small object editing (SOE), which focuses on text-based image inpainting within a constrained, small-sized area. Despite the remarkable success have been achieved by current image inpainting approaches, their application to the SOE task generally results in failure cases such as Object Missing, Text-Image Mismatch, and Distortion. These failures stem from the limited use of small-sized objects in training datasets and the downsampling operations employed by U-Net models, which hinders accurate generation. To overcome these challenges, we introduce a novel training-based approach, SOEDiff, aimed at enhancing the capability of baseline models like StableDiffusion in editing small-sized objects while minimizing training costs. Specifically, our method involves two key components: SO-LoRA, which efficiently fine-tunes low-rank matrices, and Cross-Scale Score Distillation loss, which leverages high-resolution predictions from the pre-trained teacher diffusion model. Our method presents significant improvements on the test dataset collected from MSCOCO and OpenImage, validating the effectiveness of our proposed method in small object editing. In particular, when comparing SOEDiff with SD-I model on the OpenImage-f dataset, we observe a 0.99 improvement in CLIP-Score and a reduction of 2.87 in FID. Our project page can be found in this https URL.

Title: A Hierarchically Feature Reconstructed Autoencoder for Unsupervised Anomaly Detection

• Authors: Honghui Chen, Pingping Chen, Huan Mao, Mengxi Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A Hierarchically Feature Reconstructed Autoencoder for Unsupervised Anomaly Detection(https://arxiv.org/abs/)
• Keywords: anomaly
• Abstract: Anomaly detection and localization without any manual annotations and prior knowledge is a challenging task under the setting of unsupervised learning. The existing works achieve excellent performance in the anomaly detection, but with complex networks or cumbersome pipelines. To address this issue, this paper explores a simple but effective architecture in the anomaly detection. It consists of a well pre-trained encoder to extract hierarchical feature representations and a decoder to reconstruct these intermediate features from the encoder. In particular, it does not require any data augmentations and anomalous images for training. The anomalies can be detected when the decoder fails to reconstruct features well, and then errors of hierarchical feature reconstruction are aggregated into an anomaly map to achieve anomaly localization. The difference comparison between those features of encoder and decode lead to more accurate and robust localization results than the comparison in single feature or pixel-by-pixel comparison in the conventional works. Experiment results show that the proposed method outperforms the state-of-the-art methods on MNIST, Fashion-MNIST, CIFAR-10, and MVTec Anomaly Detection datasets on both anomaly detection and localization.

Title: QMedShield: A Novel Quantum Chaos-based Image Encryption Scheme for Secure Medical Image Storage in the Cloud

• Authors: Arun Amaithi Rajan, Vetriselvi V
• Subjects: cs.CR, cs.MM
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] QMedShield: A Novel Quantum Chaos-based Image Encryption Scheme for Secure Medical Image Storage in the Cloud(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: In the age of digital technology, medical images play a crucial role in the healthcare industry which aids surgeons in making precise decisions and reducing the diagnosis time. However, the storage of large amounts of these images in third-party cloud services raises privacy and security concerns. There are a lot of classical security mechanisms to protect them. Although, the advent of quantum computing entails the development of quantum-based encryption models for healthcare. Hence, we introduce a novel quantum chaos-based encryption scheme for medical images in this article. The model comprises bit-plane scrambling, quantum logistic map, quantum operations in the diffusion phase and hybrid chaotic map, DNA encoding, and computations in the confusion phase to transform the plain medical image into a cipher medical image. The proposed scheme has been evaluated using multiple statistical measures and validated against more attacks such as differential attacks with three different medical datasets. Hence the introduced encryption model has proved to be attack-resistant and robust than other existing image encryption schemes, ensuring the secure storage of medical images in cloud environments.

Title: Dance Any Beat: Blending Beats with Visuals in Dance Video Generation

• Authors: Xuanchen Wang, Heng Wang, Dongnan Liu, Weidong Cai
• Subjects: cs.CV, cs.AI, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Dance Any Beat: Blending Beats with Visuals in Dance Video Generation(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: The task of generating dance from music is crucial, yet current methods, which mainly produce joint sequences, lead to outputs that lack intuitiveness and complicate data collection due to the necessity for precise joint annotations. We introduce a Dance Any Beat Diffusion model, namely DabFusion, that employs music as a conditional input to directly create dance videos from still images, utilizing conditional image-to-video generation principles. This approach pioneers the use of music as a conditioning factor in image-to-video synthesis. Our method unfolds in two stages: training an auto-encoder to predict latent optical flow between reference and driving frames, eliminating the need for joint annotation, and training a U-Net-based diffusion model to produce these latent optical flows guided by music rhythm encoded by CLAP. Although capable of producing high-quality dance videos, the baseline model struggles with rhythm alignment. We enhance the model by adding beat information, improving synchronization. We introduce a 2D motion-music alignment score (2D-MM Align) for quantitative assessment. Evaluated on the AIST++ dataset, our enhanced model shows marked improvements in 2D-MM Align score and established metrics. Video results can be found on our project page: this https URL.

Title: DeCoDEx: Confounder Detector Guidance for Improved Diffusion-based Counterfactual Explanations

• Authors: Nima Fathi, Amar Kumar, Brennan Nichyporuk, Mohammad Havaei, Tal Arbel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] DeCoDEx: Confounder Detector Guidance for Improved Diffusion-based Counterfactual Explanations(https://arxiv.org/abs/)
• Keywords: diffusion
• Abstract: Deep learning classifiers are prone to latching onto dominant confounders present in a dataset rather than on the causal markers associated with the target class, leading to poor generalization and biased predictions. Although explainability via counterfactual image generation has been successful at exposing the problem, bias mitigation strategies that permit accurate explainability in the presence of dominant and diverse artifacts remain unsolved. In this work, we propose the DeCoDEx framework and show how an external, pre-trained binary artifact detector can be leveraged during inference to guide a diffusion-based counterfactual image generator towards accurate explainability. Experiments on the CheXpert dataset, using both synthetic artifacts and real visual artifacts (support devices), show that the proposed method successfully synthesizes the counterfactual images that change the causal pathology markers associated with Pleural Effusion while preserving or ignoring the visual artifacts. Augmentation of ERM and Group-DRO classifiers with the DeCoDEx generated images substantially improves the results across underrepresented groups that are out of distribution for each class. The code is made publicly available at this https URL.

Title: SARATR-X: A Foundation Model for Synthetic Aperture Radar Images Target Recognition

• Authors: Weijie L, Wei Yang, Yuenan Hou, Li Liu, Yongxiang Liu, Xiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] SARATR-X: A Foundation Model for Synthetic Aperture Radar Images Target Recognition(https://arxiv.org/abs/)
• Keywords: self-supervised, foundation model
• Abstract: Synthetic aperture radar (SAR) is essential in actively acquiring information for Earth observation. SAR Automatic Target Recognition (ATR) focuses on detecting and classifying various target categories under different image conditions. The current deep learning-based SAR ATR methods are typically designed for specific datasets and applications. Various target characteristics, scene background information, and sensor parameters across ATR datasets challenge the generalization of those methods. This paper aims to achieve general SAR ATR based on a foundation model with Self-Supervised Learning (SSL). Our motivation is to break through the specific dataset and condition limitations and obtain universal perceptual capabilities across the target, scene, and sensor. A foundation model named SARATR-X is proposed with the following four aspects: pre-training dataset, model backbone, SSL, and evaluation task. First, we integrated 14 datasets with various target categories and imaging conditions as a pre-training dataset. Second, different model backbones were discussed to find the most suitable approaches for remote-sensing images. Third, we applied two-stage training and SAR gradient features to ensure the diversity and scalability of SARATR-X. Finally, SARATR-X has achieved competitive and superior performance on 5 datasets with 8 task settings, which shows that the foundation model can achieve universal SAR ATR. We believe it is time to embrace fundamental models for SAR image interpretation in the era of increasing big data.

Title: Time-Equivariant Contrastive Learning for Degenerative Disease Progression in Retinal OCT

• Authors: Taha Emre, Arunava Chakravarty, Dmitrii Lachinov, Antoine Rivail, Ursula Schmidt-Erfurth, Hrvoje Bogunović
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Time-Equivariant Contrastive Learning for Degenerative Disease Progression in Retinal OCT(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: Contrastive pretraining provides robust representations by ensuring their invariance to different image transformations while simultaneously preventing representational collapse. Equivariant contrastive learning, on the other hand, provides representations sensitive to specific image transformations while remaining invariant to others. By introducing equivariance to time-induced transformations, such as disease-related anatomical changes in longitudinal imaging, the model can effectively capture such changes in the representation space. In this work, we pro-pose a Time-equivariant Contrastive Learning (TC) method. First, an encoder embeds two unlabeled scans from different time points of the same patient into the representation space. Next, a temporal equivariance module is trained to predict the representation of a later visit based on the representation from one of the previous visits and the corresponding time interval with a novel regularization loss term while preserving the invariance property to irrelevant image transformations. On a large longitudinal dataset, our model clearly outperforms existing equivariant contrastive methods in predicting progression from intermediate age-related macular degeneration (AMD) to advanced wet-AMD within a specified time-window.

Title: Global-Local Image Perceptual Score (GLIPS): Evaluating Photorealistic Quality of AI-Generated Images

• Authors: Memoona Aziz (Western University, Canada), Umair Rehman (Western University, Canada), Muhammad Umair Danish (Western University, Canada), Katarina Grolinger (Western University, Canada)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Global-Local Image Perceptual Score (GLIPS): Evaluating Photorealistic Quality of AI-Generated Images(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: This paper introduces the Global-Local Image Perceptual Score (GLIPS), an image metric designed to assess the photorealistic image quality of AI-generated images with a high degree of alignment to human visual perception. Traditional metrics such as FID and KID scores do not align closely with human evaluations. The proposed metric incorporates advanced transformer-based attention mechanisms to assess local similarity and Maximum Mean Discrepancy (MMD) to evaluate global distributional similarity. To evaluate the performance of GLIPS, we conducted a human study on photorealistic image quality. Comprehensive tests across various generative models demonstrate that GLIPS consistently outperforms existing metrics like FID, SSIM, and MS-SSIM in terms of correlation with human scores. Additionally, we introduce the Interpolative Binning Scale (IBS), a refined scaling method that enhances the interpretability of metric scores by aligning them more closely with human evaluative standards. The proposed metric and scaling approach not only provides more reliable assessments of AI-generated images but also suggest pathways for future enhancements in image generation technologies.

Title: A Survey On Text-to-3D Contents Generation In The Wild

• Authors: Chenhan Jiang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] A Survey On Text-to-3D Contents Generation In The Wild(https://arxiv.org/abs/)
• Keywords: generative
• Abstract: 3D content creation plays a vital role in various applications, such as gaming, robotics simulation, and virtual reality. However, the process is labor-intensive and time-consuming, requiring skilled designers to invest considerable effort in creating a single 3D asset. To address this challenge, text-to-3D generation technologies have emerged as a promising solution for automating 3D creation. Leveraging the success of large vision language models, these techniques aim to generate 3D content based on textual descriptions. Despite recent advancements in this area, existing solutions still face significant limitations in terms of generation quality and efficiency. In this survey, we conduct an in-depth investigation of the latest text-to-3D creation methods. We provide a comprehensive background on text-to-3D creation, including discussions on datasets employed in training and evaluation metrics used to assess the quality of generated 3D models. Then, we delve into the various 3D representations that serve as the foundation for the 3D generation process. Furthermore, we present a thorough comparison of the rapidly growing literature on generative pipelines, categorizing them into feedforward generators, optimization-based generation, and view reconstruction approaches. By examining the strengths and weaknesses of these methods, we aim to shed light on their respective capabilities and limitations. Lastly, we point out several promising avenues for future research. With this survey, we hope to inspire researchers further to explore the potential of open-vocabulary text-conditioned 3D content creation.