2024-07-30

Title: ScalingGaussian: Enhancing 3D Content Creation with Generative Gaussian Splatting

• Authors: Shen Chen, Jiale Zhou, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, Lei Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19035
• Pdf URL: https://arxiv.org/pdf/2407.19035
• Copy Paste: [[2407.19035]] ScalingGaussian: Enhancing 3D Content Creation with Generative Gaussian Splatting(https://arxiv.org/abs/2407.19035)
• Keywords: diffusion, generative
• Abstract: The creation of high-quality 3D assets is paramount for applications in digital heritage preservation, entertainment, and robotics. Traditionally, this process necessitates skilled professionals and specialized software for the modeling, texturing, and rendering of 3D objects. However, the rising demand for 3D assets in gaming and virtual reality (VR) has led to the creation of accessible image-to-3D technologies, allowing non-professionals to produce 3D content and decreasing dependence on expert input. Existing methods for 3D content generation struggle to simultaneously achieve detailed textures and strong geometric consistency. We introduce a novel 3D content creation framework, ScalingGaussian, which combines 3D and 2D diffusion models to achieve detailed textures and geometric consistency in generated 3D assets. Initially, a 3D diffusion model generates point clouds, which are then densified through a process of selecting local regions, introducing Gaussian noise, followed by using local density-weighted selection. To refine the 3D gaussians, we utilize a 2D diffusion model with Score Distillation Sampling (SDS) loss, guiding the 3D Gaussians to clone and split. Finally, the 3D Gaussians are converted into meshes, and the surface textures are optimized using Mean Square Error(MSE) and Gradient Profile Prior(GPP) losses. Our method addresses the common issue of sparse point clouds in 3D diffusion, resulting in improved geometric structure and detailed textures. Experiments on image-to-3D tasks demonstrate that our approach efficiently generates high-quality 3D assets.

Title: UniForensics: Face Forgery Detection via General Facial Representation

• Authors: Ziyuan Fang, Hanqing Zhao, Tianyi Wei, Wenbo Zhou, Ming Wan, Zhanyi Wang, Weiming Zhang, Nenghai Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19079
• Pdf URL: https://arxiv.org/pdf/2407.19079
• Copy Paste: [[2407.19079]] UniForensics: Face Forgery Detection via General Facial Representation(https://arxiv.org/abs/2407.19079)
• Keywords: self-supervised
• Abstract: Previous deepfake detection methods mostly depend on low-level textural features vulnerable to perturbations and fall short of detecting unseen forgery methods. In contrast, high-level semantic features are less susceptible to perturbations and not limited to forgery-specific artifacts, thus having stronger generalization. Motivated by this, we propose a detection method that utilizes high-level semantic features of faces to identify inconsistencies in temporal domain. We introduce UniForensics, a novel deepfake detection framework that leverages a transformer-based video classification network, initialized with a meta-functional face encoder for enriched facial representation. In this way, we can take advantage of both the powerful spatio-temporal model and the high-level semantic information of faces. Furthermore, to leverage easily accessible real face data and guide the model in focusing on spatio-temporal features, we design a Dynamic Video Self-Blending (DVSB) method to efficiently generate training samples with diverse spatio-temporal forgery traces using real facial videos. Based on this, we advance our framework with a two-stage training approach: The first stage employs a novel self-supervised contrastive learning, where we encourage the network to focus on forgery traces by impelling videos generated by the same forgery process to have similar representations. On the basis of the representation learned in the first stage, the second stage involves fine-tuning on face forgery detection dataset to build a deepfake detector. Extensive experiments validates that UniForensics outperforms existing face forgery methods in generalization ability and robustness. In particular, our method achieves 95.3\% and 77.2\% cross dataset AUC on the challenging Celeb-DFv2 and DFDC respectively.

Title: Many-Shot In-Context Learning for Molecular Inverse Design

• Authors: Saeed Moayedpour, Alejandro Corrochano-Navarro, Faryad Sahneh, Shahriar Noroozizadeh, Alexander Koetter, Jiri Vymetal, Lorenzo Kogler-Anele, Pablo Mas, Yasser Jangjou, Sizhen Li, Michael Bailey, Marc Bianciotto, Hans Matter, Christoph Grebner, Gerhard Hessler, Ziv Bar-Joseph, Sven Jager
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19089
• Pdf URL: https://arxiv.org/pdf/2407.19089
• Copy Paste: [[2407.19089]] Many-Shot In-Context Learning for Molecular Inverse Design(https://arxiv.org/abs/2407.19089)
• Keywords: generative, in-context
• Abstract: Large Language Models (LLMs) have demonstrated great performance in few-shot In-Context Learning (ICL) for a variety of generative and discriminative chemical design tasks. The newly expanded context windows of LLMs can further improve ICL capabilities for molecular inverse design and lead optimization. To take full advantage of these capabilities we developed a new semi-supervised learning method that overcomes the lack of experimental data available for many-shot ICL. Our approach involves iterative inclusion of LLM generated molecules with high predicted performance, along with experimental data. We further integrated our method in a multi-modal LLM which allows for the interactive modification of generated molecular structures using text instructions. As we show, the new method greatly improves upon existing ICL methods for molecular design while being accessible and easy to use for scientists.

Title: Few-Shot Medical Image Segmentation with Large Kernel Attention

• Authors: Xiaoxiao Wu, Xiaowei Chen, Zhenguo Gao, Shulei Qu, Yuanyuan Qiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19148
• Pdf URL: https://arxiv.org/pdf/2407.19148
• Copy Paste: [[2407.19148]] Few-Shot Medical Image Segmentation with Large Kernel Attention(https://arxiv.org/abs/2407.19148)
• Keywords: anomaly
• Abstract: Medical image segmentation has witnessed significant advancements with the emergence of deep learning. However, the reliance of most neural network models on a substantial amount of annotated data remains a challenge for medical image segmentation. To address this issue, few-shot segmentation methods based on meta-learning have been employed. Presently, the methods primarily focus on aligning the support set and query set to enhance performance, but this approach hinders further improvement of the model's effectiveness. In this paper, our objective is to propose a few-shot medical segmentation model that acquire comprehensive feature representation capabilities, which will boost segmentation accuracy by capturing both local and long-range features. To achieve this, we introduce a plug-and-play attention module that dynamically enhances both query and support features, thereby improving the representativeness of the extracted features. Our model comprises four key modules: a dual-path feature extractor, an attention module, an adaptive prototype prediction module, and a multi-scale prediction fusion module. Specifically, the dual-path feature extractor acquires multi-scale features by obtaining features of 32{\times}32 size and 64{\times}64 size. The attention module follows the feature extractor and captures local and long-range information. The adaptive prototype prediction module automatically adjusts the anomaly score threshold to predict prototypes, while the multi-scale fusion prediction module integrates prediction masks of various scales to produce the final segmentation result. We conducted experiments on publicly available MRI datasets, namely CHAOS and CMR, and compared our method with other advanced techniques. The results demonstrate that our method achieves state-of-the-art performance.

Title: Revisit Self-supervised Depth Estimation with Local Structure-from-Motion

• Authors: Shengjie Zhu, Xiaoming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19166
• Pdf URL: https://arxiv.org/pdf/2407.19166
• Copy Paste: [[2407.19166]] Revisit Self-supervised Depth Estimation with Local Structure-from-Motion(https://arxiv.org/abs/2407.19166)
• Keywords: self-supervised
• Abstract: Both self-supervised depth estimation and Structure-from-Motion (SfM) recover scene depth from RGB videos. Despite sharing a similar objective, the two approaches are disconnected. Prior works of self-supervision backpropagate losses defined within immediate neighboring frames. Instead of learning-through-loss, this work proposes an alternative scheme by performing local SfM. First, with calibrated RGB or RGB-D images, we employ a depth and correspondence estimator to infer depthmaps and pair-wise correspondence maps. Then, a novel bundle-RANSAC-adjustment algorithm jointly optimizes camera poses and one depth adjustment for each depthmap. Finally, we fix camera poses and employ a NeRF, however, without a neural network, for dense triangulation and geometric verification. Poses, depth adjustments, and triangulated sparse depths are our outputs. For the first time, we show self-supervision within $5$ frames already benefits SoTA supervised depth and correspondence models.

Title: Reducing Spurious Correlation for Federated Domain Generalization

• Authors: Shuran Ma, Weiying Xie, Daixun Li, Haowei Li, Yunsong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19174
• Pdf URL: https://arxiv.org/pdf/2407.19174
• Copy Paste: [[2407.19174]] Reducing Spurious Correlation for Federated Domain Generalization(https://arxiv.org/abs/2407.19174)
• Keywords: self-supervised
• Abstract: The rapid development of multimedia has provided a large amount of data with different distributions for visual tasks, forming different domains. Federated Learning (FL) can efficiently use this diverse data distributed on different client media in a decentralized manner through model sharing. However, in open-world scenarios, there is a challenge: global models may struggle to predict well on entirely new domain data captured by certain media, which were not encountered during training. Existing methods still rely on strong statistical correlations between samples and labels to address this issue, which can be misleading, as some features may establish spurious short-cut correlations with the predictions. To comprehensively address this challenge, we introduce FedCD (Cross-Domain Invariant Federated Learning), an overall optimization framework at both the local and global levels. We introduce the Spurious Correlation Intervener (SCI), which employs invariance theory to locally generate interventers for features in a self-supervised manner to reduce the model's susceptibility to spurious correlated features. Our approach requires no sharing of data or features, only the gradients related to the model. Additionally, we develop the simple yet effective Risk Extrapolation Aggregation strategy (REA), determining aggregation coefficients through mathematical optimization to facilitate global causal invariant predictions. Extensive experiments and ablation studies highlight the effectiveness of our approach. In both classification and object detection generalization tasks, our method outperforms the baselines by an average of at least 1.45% in Acc, 4.8% and 1.27% in mAP50.

Title: Data Processing Techniques for Modern Multimodal Models

• Authors: Yinheng Li, Han Ding, Hang Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19180
• Pdf URL: https://arxiv.org/pdf/2407.19180
• Copy Paste: [[2407.19180]] Data Processing Techniques for Modern Multimodal Models(https://arxiv.org/abs/2407.19180)
• Keywords: diffusion
• Abstract: Data processing plays an significant role in current multimodal model training. In this paper. we provide an comprehensive review of common data processing techniques used in modern multimodal model training with a focus on diffusion models and multimodal large language models (MLLMs). We summarized all techniques into four categories: data quality, data quantity, data distribution and data safety. We further present our findings in the choice of data process methods in different type of models. This study aims to provide guidance to multimodal models developers with effective data processing techniques.

Title: Faster Image2Video Generation: A Closer Look at CLIP Image Embedding's Impact on Spatio-Temporal Cross-Attentions

• Authors: Ashkan Taghipour, Morteza Ghahremani, Mohammed Bennamoun, Aref Miri Rekavandi, Zinuo Li, Hamid Laga, Farid Boussaid
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19205
• Pdf URL: https://arxiv.org/pdf/2407.19205
• Copy Paste: [[2407.19205]] Faster Image2Video Generation: A Closer Look at CLIP Image Embedding's Impact on Spatio-Temporal Cross-Attentions(https://arxiv.org/abs/2407.19205)
• Keywords: diffusion
• Abstract: This paper investigates the role of CLIP image embeddings within the Stable Video Diffusion (SVD) framework, focusing on their impact on video generation quality and computational efficiency. Our findings indicate that CLIP embeddings, while crucial for aesthetic quality, do not significantly contribute towards the subject and background consistency of video outputs. Moreover, the computationally expensive cross-attention mechanism can be effectively replaced by a simpler linear layer. This layer is computed only once at the first diffusion inference step, and its output is then cached and reused throughout the inference process, thereby enhancing efficiency while maintaining high-quality outputs. Building on these insights, we introduce the VCUT, a training-free approach optimized for efficiency within the SVD architecture. VCUT eliminates temporal cross-attention and replaces spatial cross-attention with a one-time computed linear layer, significantly reducing computational load. The implementation of VCUT leads to a reduction of up to 322T Multiple-Accumulate Operations (MACs) per video and a decrease in model parameters by up to 50M, achieving a 20% reduction in latency compared to the baseline. Our approach demonstrates that conditioning during the Semantic Binding stage is sufficient, eliminating the need for continuous computation across all inference steps and setting a new standard for efficient video generation.

Title: Radio Frequency Signal based Human Silhouette Segmentation: A Sequential Diffusion Approach

• Authors: Penghui Wen, Kun Hu, Dong Yuan, Zhiyuan Ning, Changyang Li, Zhiyong Wang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.19244
• Pdf URL: https://arxiv.org/pdf/2407.19244
• Copy Paste: [[2407.19244]] Radio Frequency Signal based Human Silhouette Segmentation: A Sequential Diffusion Approach(https://arxiv.org/abs/2407.19244)
• Keywords: diffusion
• Abstract: Radio frequency (RF) signals have been proved to be flexible for human silhouette segmentation (HSS) under complex environments. Existing studies are mainly based on a one-shot approach, which lacks a coherent projection ability from the RF domain. Additionally, the spatio-temporal patterns have not been fully explored for human motion dynamics in HSS. Therefore, we propose a two-stage Sequential Diffusion Model (SDM) to progressively synthesize high-quality segmentation jointly with the considerations on motion dynamics. Cross-view transformation blocks are devised to guide the diffusion model in a multi-scale manner for comprehensively characterizing human related patterns in an individual frame such as directional projection from signal planes. Moreover, spatio-temporal blocks are devised to fine-tune the frame-level model to incorporate spatio-temporal contexts and motion dynamics, enhancing the consistency of the segmentation maps. Comprehensive experiments on a public benchmark -- HIBER demonstrate the state-of-the-art performance of our method with an IoU 0.732. Our code is available at this https URL.

Title: Fine-Grained Scene Graph Generation via Sample-Level Bias Prediction

• Authors: Yansheng Li, Tingzhu Wang, Kang Wu, Linlin Wang, Xin Guo, Wenbin Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19259
• Pdf URL: https://arxiv.org/pdf/2407.19259
• Copy Paste: [[2407.19259]] Fine-Grained Scene Graph Generation via Sample-Level Bias Prediction(https://arxiv.org/abs/2407.19259)
• Keywords: generative
• Abstract: Scene Graph Generation (SGG) aims to explore the relationships between objects in images and obtain scene summary graphs, thereby better serving downstream tasks. However, the long-tailed problem has adversely affected the scene graph's quality. The predictions are dominated by coarse-grained relationships, lacking more informative fine-grained ones. The union region of one object pair (i.e., one sample) contains rich and dedicated contextual information, enabling the prediction of the sample-specific bias for refining the original relationship prediction. Therefore, we propose a novel Sample-Level Bias Prediction (SBP) method for fine-grained SGG (SBG). Firstly, we train a classic SGG model and construct a correction bias set by calculating the margin between the ground truth label and the predicted label with one classic SGG model. Then, we devise a Bias-Oriented Generative Adversarial Network (BGAN) that learns to predict the constructed correction biases, which can be utilized to correct the original predictions from coarse-grained relationships to fine-grained ones. The extensive experimental results on VG, GQA, and VG-1800 datasets demonstrate that our SBG outperforms the state-of-the-art methods in terms of Average@K across three mainstream SGG models: Motif, VCtree, and Transformer. Compared to dataset-level correction methods on VG, SBG shows a significant average improvement of 5.6%, 3.9%, and 3.2% on Average@K for tasks PredCls, SGCls, and SGDet, respectively. The code will be available at this https URL.

Title: Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications

• Authors: Till Speicher, Mohammad Aflah Khan, Qinyuan Wu, Vedant Nanda, Soumi Das, Bishwamittra Ghosh, Krishna P. Gummadi, Evimaria Terzi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.19262
• Pdf URL: https://arxiv.org/pdf/2407.19262
• Copy Paste: [[2407.19262]] Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications(https://arxiv.org/abs/2407.19262)
• Keywords: in-context
• Abstract: Understanding whether and to what extent large language models (LLMs) have memorised training data has important implications for the reliability of their output and the privacy of their training data. In order to cleanly measure and disentangle memorisation from other phenomena (e.g. in-context learning), we create an experimental framework that is based on repeatedly exposing LLMs to random strings. Our framework allows us to better understand the dynamics, i.e., the behaviour of the model, when repeatedly exposing it to random strings. Using our framework, we make several striking observations: (a) we find consistent phases of the dynamics across families of models (Pythia, Phi and Llama2), (b) we identify factors that make some strings easier to memorise than others, and (c) we identify the role of local prefixes and global context in memorisation. We also show that sequential exposition to different random strings has a significant effect on memorisation. Our results, often surprising, have significant downstream implications in the study and usage of LLMs.

Title: CoLiDR: Concept Learning using Aggregated Disentangled Representations

• Authors: Sanchit Sinha, Guangzhi Xiong, Aidong Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19300
• Pdf URL: https://arxiv.org/pdf/2407.19300
• Copy Paste: [[2407.19300]] CoLiDR: Concept Learning using Aggregated Disentangled Representations(https://arxiv.org/abs/2407.19300)
• Keywords: generative
• Abstract: Interpretability of Deep Neural Networks using concept-based models offers a promising way to explain model behavior through human-understandable concepts. A parallel line of research focuses on disentangling the data distribution into its underlying generative factors, in turn explaining the data generation process. While both directions have received extensive attention, little work has been done on explaining concepts in terms of generative factors to unify mathematically disentangled representations and human-understandable concepts as an explanation for downstream tasks. In this paper, we propose a novel method CoLiDR - which utilizes a disentangled representation learning setup for learning mutually independent generative factors and subsequently learns to aggregate the said representations into human-understandable concepts using a novel aggregation/decomposition module. Experiments are conducted on datasets with both known and unknown latent generative factors. Our method successfully aggregates disentangled generative factors into concepts while maintaining parity with state-of-the-art concept-based approaches. Quantitative and visual analysis of the learned aggregation procedure demonstrates the advantages of our work compared to commonly used concept-based models over four challenging datasets. Lastly, our work is generalizable to an arbitrary number of concepts and generative factors - making it flexible enough to be suitable for various types of data.

Title: Parameter-Efficient Fine-Tuning via Circular Convolution

• Authors: Aochuan Chen, Ziqi Gao, Zijing Liu, Yu Li, Jia Li
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19342
• Pdf URL: https://arxiv.org/pdf/2407.19342
• Copy Paste: [[2407.19342]] Parameter-Efficient Fine-Tuning via Circular Convolution(https://arxiv.org/abs/2407.19342)
• Keywords: foundation model
• Abstract: Low-Rank Adaptation (LoRA) has gained popularity for fine-tuning large foundation models, leveraging low-rank matrices $\mathbf{A}$ and $\mathbf{B}$ to represent weight changes (\textit{i.e.,} $\Delta \mathbf{W} = \mathbf{B} \mathbf{A}$). This method reduces trainable parameters and mitigates heavy memory consumption associated with full delta matrices by sequentially multiplying $\mathbf{A}$ and $\mathbf{B}$ with the activation. Despite its success, the intrinsic low-rank characteristic may limit its performance. Although several variants have been proposed to address this issue, they often overlook the crucial computational and memory efficiency brought by LoRA. In this paper, we propose \underline{C}ir\underline{c}ular \underline{C}onvolution \underline{A}daptation (C$^3$A), which not only achieves high-rank adaptation with enhanced performance but also excels in both computational power and memory utilization. Extensive experiments demonstrate that C$^3$A consistently outperforms LoRA and its variants across various fine-tuning tasks.

Title: Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment

• Authors: Max Wilcoxson, Morten Svendgård, Ria Doshi, Dylan Davis, Reya Vir, Anant Sahai
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19346
• Pdf URL: https://arxiv.org/pdf/2407.19346
• Copy Paste: [[2407.19346]] Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment(https://arxiv.org/abs/2407.19346)
• Keywords: in-context
• Abstract: Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly.

Title: Learning to Select the Best Forecasting Tasks for Clinical Outcome Prediction

• Authors: Yuan Xue, Nan Du, Anne Mottram, Martin Seneviratne, Andrew M. Dai
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19359
• Pdf URL: https://arxiv.org/pdf/2407.19359
• Copy Paste: [[2407.19359]] Learning to Select the Best Forecasting Tasks for Clinical Outcome Prediction(https://arxiv.org/abs/2407.19359)
• Keywords: self-supervised
• Abstract: We propose to meta-learn an a self-supervised patient trajectory forecast learning rule by meta-training on a meta-objective that directly optimizes the utility of the patient representation over the subsequent clinical outcome prediction. This meta-objective directly targets the usefulness of a representation generated from unlabeled clinical measurement forecast for later supervised tasks. The meta-learned can then be directly used in target risk prediction, and the limited available samples can be used for further fine-tuning the model performance. The effectiveness of our approach is tested on a real open source patient EHR dataset MIMIC-III. We are able to demonstrate that our attention-based patient state representation approach can achieve much better performance for predicting target risk with low resources comparing with both direct supervised learning and pretraining with all-observation trajectory forecast.

Title: ClickDiff: Click to Induce Semantic Contact Map for Controllable Grasp Generation with Diffusion Models

• Authors: Peiming Li, Ziyi Wang, Mengyuan Liu, Hong Liu, Chen Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19370
• Pdf URL: https://arxiv.org/pdf/2407.19370
• Copy Paste: [[2407.19370]] ClickDiff: Click to Induce Semantic Contact Map for Controllable Grasp Generation with Diffusion Models(https://arxiv.org/abs/2407.19370)
• Keywords: diffusion
• Abstract: Grasp generation aims to create complex hand-object interactions with a specified object. While traditional approaches for hand generation have primarily focused on visibility and diversity under scene constraints, they tend to overlook the fine-grained hand-object interactions such as contacts, resulting in inaccurate and undesired grasps. To address these challenges, we propose a controllable grasp generation task and introduce ClickDiff, a controllable conditional generation model that leverages a fine-grained Semantic Contact Map (SCM). Particularly when synthesizing interactive grasps, the method enables the precise control of grasp synthesis through either user-specified or algorithmically predicted Semantic Contact Map. Specifically, to optimally utilize contact supervision constraints and to accurately model the complex physical structure of hands, we propose a Dual Generation Framework. Within this framework, the Semantic Conditional Module generates reasonable contact maps based on fine-grained contact information, while the Contact Conditional Module utilizes contact maps alongside object point clouds to generate realistic grasps. We evaluate the evaluation criteria applicable to controllable grasp generation. Both unimanual and bimanual generation experiments on GRAB and ARCTIC datasets verify the validity of our proposed method, demonstrating the efficacy and robustness of ClickDiff, even with previously unseen objects. Our code is available at this https URL.

Title: Deep State-Space Generative Model For Correlated Time-to-Event Predictions

• Authors: Yuan Xue, Denny Zhou, Nan Du, Andrew M. Dai, Zhen Xu, Kun Zhang, Claire Cui
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.19371
• Pdf URL: https://arxiv.org/pdf/2407.19371
• Copy Paste: [[2407.19371]] Deep State-Space Generative Model For Correlated Time-to-Event Predictions(https://arxiv.org/abs/2407.19371)
• Keywords: generative
• Abstract: Capturing the inter-dependencies among multiple types of clinically-critical events is critical not only to accurate future event prediction, but also to better treatment planning. In this work, we propose a deep latent state-space generative model to capture the interactions among different types of correlated clinical events (e.g., kidney failure, mortality) by explicitly modeling the temporal dynamics of patients' latent states. Based on these learned patient states, we further develop a new general discrete-time formulation of the hazard rate function to estimate the survival distribution of patients with significantly improved accuracy. Extensive evaluations over real EMR data show that our proposed model compares favorably to various state-of-the-art baselines. Furthermore, our method also uncovers meaningful insights about the latent correlations among mortality and different types of organ failures.

Title: X-Fake: Juggling Utility Evaluation and Explanation of Simulated SAR Images

• Authors: Zhongling Huang, Yihan Zhuang, Zipei Zhong, Feng Xu, Gong Cheng, Junwei Han
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.19436
• Pdf URL: https://arxiv.org/pdf/2407.19436
• Copy Paste: [[2407.19436]] X-Fake: Juggling Utility Evaluation and Explanation of Simulated SAR Images(https://arxiv.org/abs/2407.19436)
• Keywords: generative
• Abstract: SAR image simulation has attracted much attention due to its great potential to supplement the scarce training data for deep learning algorithms. Consequently, evaluating the quality of the simulated SAR image is crucial for practical applications. The current literature primarily uses image quality assessment techniques for evaluation that rely on human observers' perceptions. However, because of the unique imaging mechanism of SAR, these techniques may produce evaluation results that are not entirely valid. The distribution inconsistency between real and simulated data is the main obstacle that influences the utility of simulated SAR images. To this end, we propose a novel trustworthy utility evaluation framework with a counterfactual explanation for simulated SAR images for the first time, denoted as X-Fake. It unifies a probabilistic evaluator and a causal explainer to achieve a trustworthy utility assessment. We construct the evaluator using a probabilistic Bayesian deep model to learn the posterior distribution, conditioned on real data. Quantitatively, the predicted uncertainty of simulated data can reflect the distribution discrepancy. We build the causal explainer with an introspective variational auto-encoder to generate high-resolution counterfactuals. The latent code of IntroVAE is finally optimized with evaluation indicators and prior information to generate the counterfactual explanation, thus revealing the inauthentic details of simulated data explicitly. The proposed framework is validated on four simulated SAR image datasets obtained from electromagnetic models and generative artificial intelligence approaches. The results demonstrate the proposed X-Fake framework outperforms other IQA methods in terms of utility. Furthermore, the results illustrate that the generated counterfactual explanations are trustworthy, and can further improve the data utility in applications.

Title: \textsc{Perm}: A Parametric Representation for Multi-Style 3D Hair Modeling

• Authors: Chengan He, Xin Sun, Zhixin Shu, Fujun Luan, Sören Pirk, Jorge Alejandro Amador Herrera, Dominik L. Michels, Tuanfeng Y. Wang, Meng Zhang, Holly Rushmeier, Yi Zhou
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2407.19451
• Pdf URL: https://arxiv.org/pdf/2407.19451
• Copy Paste: [[2407.19451]] \textsc{Perm}: A Parametric Representation for Multi-Style 3D Hair Modeling(https://arxiv.org/abs/2407.19451)
• Keywords: generative
• Abstract: We present \textsc{Perm}, a learned parametric model of human 3D hair designed to facilitate various hair-related applications. Unlike previous work that jointly models the global hair shape and local strand details, we propose to disentangle them using a PCA-based strand representation in the frequency domain, thereby allowing more precise editing and output control. Specifically, we leverage our strand representation to fit and decompose hair geometry textures into low- to high-frequency hair structures. These decomposed textures are later parameterized with different generative models, emulating common stages in the hair modeling process. We conduct extensive experiments to validate the architecture design of \textsc{Perm}, and finally deploy the trained model as a generic prior to solve task-agnostic problems, further showcasing its flexibility and superiority in tasks such as 3D hair parameterization, hairstyle interpolation, single-view hair reconstruction, and hair-conditioned image generation. Our code and data will be available at: \url{this https URL}.

Title: FIND: Fine-tuning Initial Noise Distribution with Policy Optimization for Diffusion Models

• Authors: Changgu Chen, Libing Yang, Xiaoyan Yang, Lianggangxu Chen, Gaoqi He, CHangbo Wang, Yang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19453
• Pdf URL: https://arxiv.org/pdf/2407.19453
• Copy Paste: [[2407.19453]] FIND: Fine-tuning Initial Noise Distribution with Policy Optimization for Diffusion Models(https://arxiv.org/abs/2407.19453)
• Keywords: diffusion
• Abstract: In recent years, large-scale pre-trained diffusion models have demonstrated their outstanding capabilities in image and video generation tasks. However, existing models tend to produce visual objects commonly found in the training dataset, which diverges from user input prompts. The underlying reason behind the inaccurate generated results lies in the model's difficulty in sampling from specific intervals of the initial noise distribution corresponding to the prompt. Moreover, it is challenging to directly optimize the initial distribution, given that the diffusion process involves multiple denoising steps. In this paper, we introduce a Fine-tuning Initial Noise Distribution (FIND) framework with policy optimization, which unleashes the powerful potential of pre-trained diffusion networks by directly optimizing the initial distribution to align the generated contents with user-input prompts. To this end, we first reformulate the diffusion denoising procedure as a one-step Markov decision process and employ policy optimization to directly optimize the initial distribution. In addition, a dynamic reward calibration module is proposed to ensure training stability during optimization. Furthermore, we introduce a ratio clipping algorithm to utilize historical data for network training and prevent the optimized distribution from deviating too far from the original policy to restrain excessive optimization magnitudes. Extensive experiments demonstrate the effectiveness of our method in both text-to-image and text-to-video tasks, surpassing SOTA methods in achieving consistency between prompts and the generated content. Our method achieves 10 times faster than the SOTA approach. Our homepage is available at \url{this https URL}.

Title: White Matter Geometry-Guided Score-Based Diffusion Model for Tissue Microstructure Imputation in Tractography Imaging

• Authors: Yui Lo, Yuqian Chen, Fan Zhang, Dongnan Liu, Leo Zekelman, Suheyla Cetin-Karayumak, Yogesh Rathi, Weidong Cai, Lauren J. O'Donnell
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19460
• Pdf URL: https://arxiv.org/pdf/2407.19460
• Copy Paste: [[2407.19460]] White Matter Geometry-Guided Score-Based Diffusion Model for Tissue Microstructure Imputation in Tractography Imaging(https://arxiv.org/abs/2407.19460)
• Keywords: diffusion
• Abstract: Parcellation of white matter tractography provides anatomical features for disease prediction, anatomical tract segmentation, surgical brain mapping, and non-imaging phenotype classifications. However, parcellation does not always reach 100% accuracy due to various factors, including inter-individual anatomical variability and the quality of neuroimaging scan data. The failure to identify parcels causes a problem of missing microstructure data values, which is especially challenging for downstream tasks that analyze large brain datasets. In this work, we propose a novel deep-learning model to impute tissue microstructure: the White Matter Geometry-guided Diffusion (WMG-Diff) model. Specifically, we first propose a deep score-based guided diffusion model to impute tissue microstructure for diffusion magnetic resonance imaging (dMRI) tractography fiber clusters. Second, we propose a white matter atlas geometric relationship-guided denoising function to guide the reverse denoising process at the subject-specific level. Third, we train and evaluate our model on a large dataset with 9342 subjects. Comprehensive experiments for tissue microstructure imputation and a downstream non-imaging phenotype prediction task demonstrate that our proposed WMG-Diff outperforms state-of-the-art methods.

Title: MVPbev: Multi-view Perspective Image Generation from BEV with Test-time Controllability and Generalizability

• Authors: Buyu Liu, Kai Wang, Yansong Liu, Jun Bao, Tingting Han, Jun Yu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.19468
• Pdf URL: https://arxiv.org/pdf/2407.19468
• Copy Paste: [[2407.19468]] MVPbev: Multi-view Perspective Image Generation from BEV with Test-time Controllability and Generalizability(https://arxiv.org/abs/2407.19468)
• Keywords: diffusion
• Abstract: This work aims to address the multi-view perspective RGB generation from text prompts given Bird-Eye-View(BEV) semantics. Unlike prior methods that neglect layout consistency, lack the ability to handle detailed text prompts, or are incapable of generalizing to unseen view points, MVPbev simultaneously generates cross-view consistent images of different perspective views with a two-stage design, allowing object-level control and novel view generation at test-time. Specifically, MVPbev firstly projects given BEV semantics to perspective view with camera parameters, empowering the model to generalize to unseen view points. Then we introduce a multi-view attention module where special initialization and de-noising processes are introduced to explicitly enforce local consistency among overlapping views w.r.t. cross-view homography. Last but not least, MVPbev further allows test-time instance-level controllability by refining a pre-trained text-to-image diffusion model. Our extensive experiments on NuScenes demonstrate that our method is capable of generating high-resolution photorealistic images from text descriptions with thousands of training samples, surpassing the state-of-the-art methods under various evaluation metrics. We further demonstrate the advances of our method in terms of generalizability and controllability with the help of novel evaluation metrics and comprehensive human analysis. Our code, data, and model can be found in \url{this https URL}.

Title: Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation

• Authors: Tz-Ying Wu, Kyle Min, Subarna Tripathi, Nuno Vasconcelos
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.19520
• Pdf URL: https://arxiv.org/pdf/2407.19520
• Copy Paste: [[2407.19520]] Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation(https://arxiv.org/abs/2407.19520)
• Keywords: foundation model
• Abstract: Video understanding typically requires fine-tuning the large backbone when adapting to new domains. In this paper, we leverage the egocentric video foundation models (Ego-VFMs) based on video-language pre-training and propose a parameter-efficient adaptation for egocentric video tasks, namely Ego-VPA. It employs a local sparse approximation for each video frame/text feature using the basis prompts, and the selected basis prompts are used to synthesize video/text prompts. Since the basis prompts are shared across frames and modalities, it models context fusion and cross-modal transfer in an efficient fashion. Experiments show that Ego-VPA excels in lightweight adaptation (with only 0.84% learnable parameters), largely improving over baselines and reaching the performance of full fine-tuning.

Title: Robust Fast Adaptation from Adversarially Explicit Task Distribution Generation

• Authors: Cheems Wang, Yiqin Lv, Yixiu Mao, Yun Qu, Yi Xu, Xiangyang Ji
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.19523
• Pdf URL: https://arxiv.org/pdf/2407.19523
• Copy Paste: [[2407.19523]] Robust Fast Adaptation from Adversarially Explicit Task Distribution Generation(https://arxiv.org/abs/2407.19523)
• Keywords: generative
• Abstract: Meta-learning is a practical learning paradigm to transfer skills across tasks from a few examples. Nevertheless, the existence of task distribution shifts tends to weaken meta-learners' generalization capability, particularly when the task distribution is naively hand-crafted or based on simple priors that fail to cover typical scenarios sufficiently. Here, we consider explicitly generative modeling task distributions placed over task identifiers and propose robustifying fast adaptation from adversarial training. Our approach, which can be interpreted as a model of a Stackelberg game, not only uncovers the task structure during problem-solving from an explicit generative model but also theoretically increases the adaptation robustness in worst cases. This work has practical implications, particularly in dealing with task distribution shifts in meta-learning, and contributes to theoretical insights in the field. Our method demonstrates its robustness in the presence of task subpopulation shifts and improved performance over SOTA baselines in extensive experiments. The project is available at this https URL.

Title: VersusDebias: Universal Zero-Shot Debiasing for Text-to-Image Models via SLM-Based Prompt Engineering and Generative Adversary

• Authors: Hanjun Luo, Ziye Deng, Haoyu Huang, Xuecheng Liu, Ruizhe Chen, Zuozhu Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19524
• Pdf URL: https://arxiv.org/pdf/2407.19524
• Copy Paste: [[2407.19524]] VersusDebias: Universal Zero-Shot Debiasing for Text-to-Image Models via SLM-Based Prompt Engineering and Generative Adversary(https://arxiv.org/abs/2407.19524)
• Keywords: generative
• Abstract: With the rapid development of Text-to-Image models, biases in human image generation against demographic groups social attract more and more concerns. Existing methods are designed based on certain models with fixed prompts, unable to accommodate the trend of high-speed updating of Text-to-Image (T2I) models and variable prompts in practical scenes. Additionally, they fail to consider the possibility of hallucinations, leading to deviations between expected and actual results. To address this issue, we introduce VersusDebias, a novel and universal debiasing framework for biases in T2I models, consisting of one generative adversarial mechanism (GAM) and one debiasing generation mechanism using a small language model (SLM). The self-adaptive GAM generates specialized attribute arrays for each prompts for diminishing the influence of hallucinations from T2I models. The SLM uses prompt engineering to generate debiased prompts for the T2I model, providing zero-shot debiasing ability and custom optimization for different models. Extensive experiments demonstrate VersusDebias's capability to rectify biases on arbitrary models across multiple protected attributes simultaneously, including gender, race, and age. Furthermore, VersusDebias outperforms existing methods in both zero-shot and few-shot situations, illustrating its extraordinary utility. Our work is openly accessible to the research community to ensure the reproducibility.

Title: Temporal Feature Matters: A Framework for Diffusion Model Quantization

• Authors: Yushi Huang, Ruihao Gong, Xianglong Liu, Jing Liu, Yuhang Li, Jiwen Lu, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19547
• Pdf URL: https://arxiv.org/pdf/2407.19547
• Copy Paste: [[2407.19547]] Temporal Feature Matters: A Framework for Diffusion Model Quantization(https://arxiv.org/abs/2407.19547)
• Keywords: diffusion
• Abstract: The Diffusion models, widely used for image generation, face significant challenges related to their broad applicability due to prolonged inference times and high memory demands. Efficient Post-Training Quantization (PTQ) is crucial to address these issues in traditional models. Unlike those models, diffusion models critically rely on the time-step $t$ for effective multi-round denoising. Typically, $t$ from the finite set $\{1, \ldots, T\}$ is encoded into a hypersensitive temporal feature by several modules, entirely independent of the sampling data. However, existing PTQ methods do not optimize these modules individually. Instead, they employ unsuitable reconstruction objectives and complex calibration methods, leading to significant disturbances in the temporal feature and denoising trajectory. To address these challenges, we introduce a novel quantization framework: 1)~TIB-based Maintenance: Based on our innovative Temporal Information Block~(TIB) definition, Temporal Information-aware Reconstruction~(TIAR) and Finite Set Calibration~(FSC) are developed to efficiently align full precision temporal features. 2)~Cache-based Maintenance: Instead of indirect and complex optimization for the related modules, pre-computing and caching quantized counterparts of temporal features are developed to minimize errors. 3)~Disturbance-aware Selection: Employ temporal feature errors to guide a fine-grained selection for superior maintenance. This framework preserves most of the temporal information and ensures high-quality end-to-end generation. Extensive testing on various datasets and diffusion models confirms our superior results. Notably, our approach closely matches the performance of the full-precision model under 4-bit quantization. Furthermore, the quantized SD-XL model achieves hardware acceleration of 2.20$\times$ on CPU and 5.76$\times$ on GPU demonstrating its efficiency.

Title: Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle

• Authors: Zhenyu Tang, Junwu Zhang, Xinhua Cheng, Wangbo Yu, Chaoran Feng, Yatian Pang, Bin Lin, Li Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19548
• Pdf URL: https://arxiv.org/pdf/2407.19548
• Copy Paste: [[2407.19548]] Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle(https://arxiv.org/abs/2407.19548)
• Keywords: diffusion
• Abstract: Recent 3D large reconstruction models typically employ a two-stage process, including first generate multi-view images by a multi-view diffusion model, and then utilize a feed-forward model to reconstruct images to 3D content.However, multi-view diffusion models often produce low-quality and inconsistent images, adversely affecting the quality of the final 3D reconstruction. To address this issue, we propose a unified 3D generation framework called Cycle3D, which cyclically utilizes a 2D diffusion-based generation module and a feed-forward 3D reconstruction module during the multi-step diffusion process. Concretely, 2D diffusion model is applied for generating high-quality texture, and the reconstruction model guarantees multi-view consistency.Moreover, 2D diffusion model can further control the generated content and inject reference-view information for unseen views, thereby enhancing the diversity and texture consistency of 3D generation during the denoising process. Extensive experiments demonstrate the superior ability of our method to create 3D content with high-quality and consistency compared with state-of-the-art baselines.

Title: Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models

• Authors: Jifeng Wang, Kaouther Messaoud, Yuejiang Liu, Juergen Gall, Alexandre Alahi
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.19564
• Pdf URL: https://arxiv.org/pdf/2407.19564
• Copy Paste: [[2407.19564]] Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models(https://arxiv.org/abs/2407.19564)
• Keywords: self-supervised
• Abstract: Recent progress in motion forecasting has been substantially driven by self-supervised pre-training. However, adapting pre-trained models for specific downstream tasks, especially motion prediction, through extensive fine-tuning is often inefficient. This inefficiency arises because motion prediction closely aligns with the masked pre-training tasks, and traditional full fine-tuning methods fail to fully leverage this alignment. To address this, we introduce Forecast-PEFT, a fine-tuning strategy that freezes the majority of the model's parameters, focusing adjustments on newly introduced prompts and adapters. This approach not only preserves the pre-learned representations but also significantly reduces the number of parameters that need retraining, thereby enhancing efficiency. This tailored strategy, supplemented by our method's capability to efficiently adapt to different datasets, enhances model efficiency and ensures robust performance across datasets without the need for extensive retraining. Our experiments show that Forecast-PEFT outperforms traditional full fine-tuning methods in motion prediction tasks, achieving higher accuracy with only 17% of the trainable parameters typically required. Moreover, our comprehensive adaptation, Forecast-FT, further improves prediction performance, evidencing up to a 9.6% enhancement over conventional baseline methods. Code will be available at this https URL.

Title: Bridging the Gap: Studio-like Avatar Creation from a Monocular Phone Capture

• Authors: ShahRukh Athar, Shunsuke Saito, Zhengyu Yang, Stanislav Pidhorsky, Chen Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19593
• Pdf URL: https://arxiv.org/pdf/2407.19593
• Copy Paste: [[2407.19593]] Bridging the Gap: Studio-like Avatar Creation from a Monocular Phone Capture(https://arxiv.org/abs/2407.19593)
• Keywords: diffusion
• Abstract: Creating photorealistic avatars for individuals traditionally involves extensive capture sessions with complex and expensive studio devices like the LightStage system. While recent strides in neural representations have enabled the generation of photorealistic and animatable 3D avatars from quick phone scans, they have the capture-time lighting baked-in, lack facial details and have missing regions in areas such as the back of the ears. Thus, they lag in quality compared to studio-captured avatars. In this paper, we propose a method that bridges this gap by generating studio-like illuminated texture maps from short, monocular phone captures. We do this by parameterizing the phone texture maps using the $W^+$ space of a StyleGAN2, enabling near-perfect reconstruction. Then, we finetune a StyleGAN2 by sampling in the $W^+$ parameterized space using a very small set of studio-captured textures as an adversarial training signal. To further enhance the realism and accuracy of facial details, we super-resolve the output of the StyleGAN2 using carefully designed diffusion model that is guided by image gradients of the phone-captured texture map. Once trained, our method excels at producing studio-like facial texture maps from casual monocular smartphone videos. Demonstrating its capabilities, we showcase the generation of photorealistic, uniformly lit, complete avatars from monocular phone captures. \href{this http URL}{The project page can be found here.}

Title: AgEval: A Benchmark for Zero-Shot and Few-Shot Plant Stress Phenotyping with Multimodal LLMs

• Authors: Muhammad Arbab Arshad, Talukder Zaki Jubery, Tirtho Roy, Rim Nassiri, Asheesh K. Singh, Arti Singh, Chinmay Hegde, Baskar Ganapathysubramanian, Aditya Balu, Adarsh Krishnamurthy, Soumik Sarkar
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19617
• Pdf URL: https://arxiv.org/pdf/2407.19617
• Copy Paste: [[2407.19617]] AgEval: A Benchmark for Zero-Shot and Few-Shot Plant Stress Phenotyping with Multimodal LLMs(https://arxiv.org/abs/2407.19617)
• Keywords: in-context
• Abstract: Plant stress phenotyping traditionally relies on expert assessments and specialized models, limiting scalability in agriculture. Recent advances in multimodal large language models (LLMs) offer potential solutions to this challenge. We present AgEval, a benchmark comprising 12 diverse plant stress phenotyping tasks, to evaluate these models' capabilities. Our study assesses zero-shot and few-shot in-context learning performance of state-of-the-art models, including Claude, GPT, Gemini, and LLaVA. Results show significant performance improvements with few-shot learning, with F1 scores increasing from 46.24% to 73.37% in 8-shot identification for the best-performing model. Few-shot examples from other classes in the dataset have negligible or negative impacts, although having the exact category example helps to increase performance by 15.38%. We also quantify the consistency of model performance across different classes within each task, finding that the coefficient of variance (CV) ranges from 26.02% to 58.03% across models, implying that subject matter expertise is needed - of 'difficult' classes - to achieve reliability in performance. AgEval establishes baseline metrics for multimodal LLMs in agricultural applications, offering insights into their promise for enhancing plant stress phenotyping at scale. Benchmark and code can be accessed at: https://anonymous.4open.science/r/AgEval/

Title: Towards a Knowledge guided Multimodal Foundation Model for Spatio-Temporal Remote Sensing Applications

• Authors: Praveen Ravirathinam, Ankush Khandelwal, Rahul Ghosh, Vipin Kumar
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.19660
• Pdf URL: https://arxiv.org/pdf/2407.19660
• Copy Paste: [[2407.19660]] Towards a Knowledge guided Multimodal Foundation Model for Spatio-Temporal Remote Sensing Applications(https://arxiv.org/abs/2407.19660)
• Keywords: foundation model
• Abstract: In recent years, there is increased interest in foundation models for geoscience due to vast amount of earth observing satellite imagery. Existing remote sensing foundation models make use of the various sources of spectral imagery to create large models pretrained on masked reconstruction task. The embeddings from these foundation models are then used for various downstream remote sensing applications. In this paper we propose a foundational modeling framework for remote sensing geoscience applications, that goes beyond these traditional single modality masked autoencoder family of foundation models. This framework leverages the knowledge guided principles that the spectral imagery captures the impact of the physical drivers on the environmental system, and that the relationship between them is governed by the characteristics of the system. Specifically, our method, called MultiModal Variable Step Forecasting (MM-VSF), uses mutlimodal data (spectral imagery and weather) as its input and a variable step forecasting task as its pretraining objective. In our evaluation we show forecasting of satellite imagery using weather can be used as an effective pretraining task for foundation models. We further show the effectiveness of the embeddings from MM-VSF on the downstream task of pixel wise crop mapping, when compared with a model trained in the traditional setting of single modality input and masked reconstruction based pretraining.

Title: Structural damage detection via hierarchical damage information with volumetric assessment

• Authors: Isaac Osei Agyemang, Jianwen Chen, Liaoyuan Zeng, Isaac Adjei-Mensah, Daniel Acheampong, Gordon Owusu Boateng, Adu Asare Baffour
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19694
• Pdf URL: https://arxiv.org/pdf/2407.19694
• Copy Paste: [[2407.19694]] Structural damage detection via hierarchical damage information with volumetric assessment(https://arxiv.org/abs/2407.19694)
• Keywords: generative
• Abstract: Image environments and noisy labels hinder deep learning-based inference models in structural damage detection. Post-detection, there is the challenge of reliance on manual assessments of detected damages. As a result, Guided-DetNet, characterized by Generative Attention Module (GAM), Hierarchical Elimination Algorithm (HEA), and Volumetric Contour Visual Assessment (VCVA), is proposed to mitigate complex image environments, noisy labeling, and post-detection manual assessment of structural damages. GAM leverages cross-horizontal and cross-vertical patch merging and cross foreground-background feature fusion to generate varied features to mitigate complex image environments. HEA addresses noisy labeling using hierarchical relationships among classes to refine instances given an image by eliminating unlikely class categories. VCVA assesses the severity of detected damages via volumetric representation and quantification leveraging the Dirac delta distribution. A comprehensive quantitative study, two robustness tests, and an application scenario based on the PEER Hub Image-Net dataset substantiate Guided-DetNet's promising performances. Guided-DetNet outperformed the best-compared models in a triple classification task by a difference of not less than 3% and not less than 2% in a dual detection task under varying metrics.

Title: Multiscale Representation Enhanced Temporal Flow Fusion Model for Long-Term Workload Forecasting

• Authors: Shiyu Wang, Zhixuan Chu, Yinbo Sun, Yu Liu, Yuliang Guo, Yang Chen, Huiyang Jian, Lintao Ma, Xingyu Lu, Jun Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19697
• Pdf URL: https://arxiv.org/pdf/2407.19697
• Copy Paste: [[2407.19697]] Multiscale Representation Enhanced Temporal Flow Fusion Model for Long-Term Workload Forecasting(https://arxiv.org/abs/2407.19697)
• Keywords: self-supervised
• Abstract: Accurate workload forecasting is critical for efficient resource management in cloud computing systems, enabling effective scheduling and autoscaling. Despite recent advances with transformer-based forecasting models, challenges remain due to the non-stationary, nonlinear characteristics of workload time series and the long-term dependencies. In particular, inconsistent performance between long-term history and near-term forecasts hinders long-range predictions. This paper proposes a novel framework leveraging self-supervised multiscale representation learning to capture both long-term and near-term workload patterns. The long-term history is encoded through multiscale representations while the near-term observations are modeled via temporal flow fusion. These representations of different scales are fused using an attention mechanism and characterized with normalizing flows to handle non-Gaussian/non-linear distributions of time series. Extensive experiments on 9 benchmarks demonstrate superiority over existing methods.

Title: Sensor Selection via GFlowNets: A Deep Generative Modeling Framework to Navigate Combinatorial Complexity

• Authors: Spilios Evmorfos, Zhaoyi Xu, Athina Petropulu
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2407.19736
• Pdf URL: https://arxiv.org/pdf/2407.19736
• Copy Paste: [[2407.19736]] Sensor Selection via GFlowNets: A Deep Generative Modeling Framework to Navigate Combinatorial Complexity(https://arxiv.org/abs/2407.19736)
• Keywords: generative
• Abstract: The performance of sensor arrays in sensing and wireless communications improves with more elements, but this comes at the cost of increased energy consumption and hardware expense. This work addresses the challenge of selecting $k$ sensor elements from a set of $m$ to optimize a generic Quality-of-Service metric. Evaluating all $\binom{m}{k}$ possible sensor subsets is impractical, leading to prior solutions using convex relaxations, greedy algorithms, and supervised learning approaches. The current paper proposes a new framework that employs deep generative modeling, treating sensor selection as a deterministic Markov Decision Process where sensor subsets of size $k$ arise as terminal states. Generative Flow Networks (GFlowNets) are employed to model an action distribution conditioned on the state. Sampling actions from the aforementioned distribution ensures that the probability of arriving at a terminal state is proportional to the performance of the corresponding subset. Applied to a standard sensor selection scenario, the developed approach outperforms popular methods which are based on convex optimization and greedy algorithms. Finally, a multiobjective formulation of the proposed approach is adopted and applied on the sparse antenna array design for Integrated Sensing and Communication (ISAC) systems. The multiobjective variation is shown to perform well in managing the trade-off between radar and communication performance.

Title: Introducing a new hyper-parameter for RAG: Context Window Utilization

• Authors: Kush Juvekar, Anupam Purwar
• Subjects: cs.CL, cs.ET
• Abstract URL: https://arxiv.org/abs/2407.19794
• Pdf URL: https://arxiv.org/pdf/2407.19794
• Copy Paste: [[2407.19794]] Introducing a new hyper-parameter for RAG: Context Window Utilization(https://arxiv.org/abs/2407.19794)
• Keywords: generative
• Abstract: This paper introduces a new hyper-parameter for Retrieval-Augmented Generation (RAG) systems called Context Window Utilization. RAG systems enhance generative models by incorporating relevant information retrieved from external knowledge bases, improving the factual accuracy and contextual relevance of generated responses. The size of the text chunks retrieved and processed is a critical factor influencing RAG performance. This study aims to identify the optimal chunk size that maximizes answer generation quality. Through systematic experimentation, we analyze the effects of varying chunk sizes on the efficiency and effectiveness of RAG frameworks. Our findings reveal that an optimal chunk size balances the trade-off between providing sufficient context and minimizing irrelevant information. These insights are crucial for enhancing the design and implementation of RAG systems, underscoring the importance of selecting an appropriate chunk size to achieve superior performance.

Title: Synthetic Thermal and RGB Videos for Automatic Pain Assessment utilizing a Vision-MLP Architecture

• Authors: Stefanos Gkikas, Manolis Tsiknakis
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19811
• Pdf URL: https://arxiv.org/pdf/2407.19811
• Copy Paste: [[2407.19811]] Synthetic Thermal and RGB Videos for Automatic Pain Assessment utilizing a Vision-MLP Architecture(https://arxiv.org/abs/2407.19811)
• Keywords: generative
• Abstract: Pain assessment is essential in developing optimal pain management protocols to alleviate suffering and prevent functional decline in patients. Consequently, reliable and accurate automatic pain assessment systems are essential for continuous and effective patient monitoring. This study presents synthetic thermal videos generated by Generative Adversarial Networks integrated into the pain recognition pipeline and evaluates their efficacy. A framework consisting of a Vision-MLP and a Transformer-based module is utilized, employing RGB and synthetic thermal videos in unimodal and multimodal settings. Experiments conducted on facial videos from the BioVid database demonstrate the effectiveness of synthetic thermal videos and underline the potential advantages of it.

Title: Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability

• Authors: Jorge García-Carrasco, Alejandro Maté, Juan Trujillo
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.19842
• Pdf URL: https://arxiv.org/pdf/2407.19842
• Copy Paste: [[2407.19842]] Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability(https://arxiv.org/abs/2407.19842)
• Keywords: self-supervised, generative
• Abstract: Large Language Models (LLMs), characterized by being trained on broad amounts of data in a self-supervised manner, have shown impressive performance across a wide range of tasks. Indeed, their generative abilities have aroused interest on the application of LLMs across a wide range of contexts. However, neural networks in general, and LLMs in particular, are known to be vulnerable to adversarial attacks, where an imperceptible change to the input can mislead the output of the model. This is a serious concern that impedes the use of LLMs on high-stakes applications, such as healthcare, where a wrong prediction can imply serious consequences. Even though there are many efforts on making LLMs more robust to adversarial attacks, there are almost no works that study \emph{how} and \emph{where} these vulnerabilities that make LLMs prone to adversarial attacks happen. Motivated by these facts, we explore how to localize and understand vulnerabilities, and propose a method, based on Mechanistic Interpretability (MI) techniques, to guide this process. Specifically, this method enables us to detect vulnerabilities related to a concrete task by (i) obtaining the subset of the model that is responsible for that task, (ii) generating adversarial samples for that task, and (iii) using MI techniques together with the previous samples to discover and understand the possible vulnerabilities. We showcase our method on a pretrained GPT-2 Small model carrying out the task of predicting 3-letter acronyms to demonstrate its effectiveness on locating and understanding concrete vulnerabilities of the model.

Title: Normality Addition via Normality Detection in Industrial Image Anomaly Detection Models

• Authors: Jihun Yi, Dahuin Jung, Sungroh Yoon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19849
• Pdf URL: https://arxiv.org/pdf/2407.19849
• Copy Paste: [[2407.19849]] Normality Addition via Normality Detection in Industrial Image Anomaly Detection Models(https://arxiv.org/abs/2407.19849)
• Keywords: anomaly
• Abstract: The task of image anomaly detection (IAD) aims to identify deviations from normality in image data. These anomalies are patterns that deviate significantly from what the IAD model has learned from the data during training. However, in real-world scenarios, the criteria for what constitutes normality often change, necessitating the reclassification of previously anomalous instances as normal. To address this challenge, we propose a new scenario termed "normality addition," involving the post-training adjustment of decision boundaries to incorporate new normalities. To address this challenge, we propose a method called Normality Addition via Normality Detection (NAND), leveraging a vision-language model. NAND performs normality detection which detect patterns related to the intended normality within images based on textual descriptions. We then modify the results of a pre-trained IAD model to implement this normality addition. Using the benchmark dataset in IAD, MVTec AD, we establish an evaluation protocol for the normality addition task and empirically demonstrate the effectiveness of the NAND method.

Title: Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learning

• Authors: Leen Kweider, Maissa Abou Kassem, Ubai Sandouk
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19860
• Pdf URL: https://arxiv.org/pdf/2407.19860
• Copy Paste: [[2407.19860]] Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learning(https://arxiv.org/abs/2407.19860)
• Keywords: anomaly
• Abstract: The deployment of artificial intelligence (AI) in decision-making applications requires ensuring an appropriate level of safety and reliability, particularly in changing environments that contain a large number of unknown observations. To address this challenge, we propose a novel safe reinforcement learning (RL) approach that utilizes an anomalous state sequence to enhance RL safety. Our proposed solution Safe Reinforcement Learning with Anomalous State Sequences (AnoSeqs) consists of two stages. First, we train an agent in a non-safety-critical offline 'source' environment to collect safe state sequences. Next, we use these safe sequences to build an anomaly detection model that can detect potentially unsafe state sequences in a 'target' safety-critical environment where failures can have high costs. The estimated risk from the anomaly detection model is utilized to train a risk-averse RL policy in the target environment; this involves adjusting the reward function to penalize the agent for visiting anomalous states deemed unsafe by our anomaly model. In experiments on multiple safety-critical benchmarking environments including self-driving cars, our solution approach successfully learns safer policies and proves that sequential anomaly detection can provide an effective supervisory signal for training safety-aware RL agents

Title: Self-Supervised Learning for Text Recognition: A Critical Survey

• Authors: Carlos Penarrubia, Jose J. Valero-Mas, Jorge Calvo-Zaragoza
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19889
• Pdf URL: https://arxiv.org/pdf/2407.19889
• Copy Paste: [[2407.19889]] Self-Supervised Learning for Text Recognition: A Critical Survey(https://arxiv.org/abs/2407.19889)
• Keywords: self-supervised
• Abstract: Text Recognition (TR) refers to the research area that focuses on retrieving textual information from images, a topic that has seen significant advancements in the last decade due to the use of Deep Neural Networks (DNN). However, these solutions often necessitate vast amounts of manually labeled or synthetic data. Addressing this challenge, Self-Supervised Learning (SSL) has gained attention by utilizing large datasets of unlabeled data to train DNN, thereby generating meaningful and robust representations. Although SSL was initially overlooked in TR because of its unique characteristics, recent years have witnessed a surge in the development of SSL methods specifically for this field. This rapid development, however, has led to many methods being explored independently, without taking previous efforts in methodology or comparison into account, thereby hindering progress in the field of research. This paper, therefore, seeks to consolidate the use of SSL in the field of TR, offering a critical and comprehensive overview of the current state of the art. We will review and analyze the existing methods, compare their results, and highlight inconsistencies in the current literature. This thorough analysis aims to provide general insights into the field, propose standardizations, identify new research directions, and foster its proper development.

Title: Cell Culture Assistive Application for Precipitation Image Diagnosis

• Authors: Takato Yasuno
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19913
• Pdf URL: https://arxiv.org/pdf/2407.19913
• Copy Paste: [[2407.19913]] Cell Culture Assistive Application for Precipitation Image Diagnosis(https://arxiv.org/abs/2407.19913)
• Keywords: generative, anomaly
• Abstract: In regenerative medicine research, we experimentally design the composition of chemical medium. We add different components to 384-well plates and culture the biological cells. We monitor the condition of the cells and take time-lapse bioimages for morphological assay. In particular, precipitation can appear as artefacts in the image and contaminate the noise in the imaging assay. Inspecting precipitates is a tedious task for the observer, and differences in experience can lead to variations in judgement from person to person. The machine learning approach will remove the burden of human inspection and provide consistent inspection. In addition, precipitation features are as small as 10-20 {\mu}m. A 1200 pixel square well image resized under a resolution of 2.82 {\mu}m/pixel will result in a reduction in precipitation features. Dividing the well images into 240-pixel squares and learning without resizing preserves the resolution of the original image. In this study, we developed an application to automatically detect precipitation on 384-well plates utilising optical microscope images. We apply MN-pair contrastive clustering to extract precipitation classes from approximately 20,000 patch images. To detect precipitation features, we compare deeper FCDDs detectors with optional backbones and build a machine learning pipeline to detect precipitation from the maximum score of quadruplet well images using isolation Forest algorithm, where the anomaly score is ranged from zero to one. Furthermore, using this application we can visualise precipitation situ heatmap on a 384-well plate.

Title: FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention

• Authors: Yu Lu, Yuanzhi Liang, Linchao Zhu, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19918
• Pdf URL: https://arxiv.org/pdf/2407.19918
• Copy Paste: [[2407.19918]] FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention(https://arxiv.org/abs/2407.19918)
• Keywords: diffusion
• Abstract: Video diffusion models have made substantial progress in various video generation applications. However, training models for long video generation tasks require significant computational and data resources, posing a challenge to developing long video diffusion models. This paper investigates a straightforward and training-free approach to extend an existing short video diffusion model (e.g. pre-trained on 16-frame videos) for consistent long video generation (e.g. 128 frames). Our preliminary observation has found that directly applying the short video diffusion model to generate long videos can lead to severe video quality degradation. Further investigation reveals that this degradation is primarily due to the distortion of high-frequency components in long videos, characterized by a decrease in spatial high-frequency components and an increase in temporal high-frequency components. Motivated by this, we propose a novel solution named FreeLong to balance the frequency distribution of long video features during the denoising process. FreeLong blends the low-frequency components of global video features, which encapsulate the entire video sequence, with the high-frequency components of local video features that focus on shorter subsequences of frames. This approach maintains global consistency while incorporating diverse and high-quality spatiotemporal details from local videos, enhancing both the consistency and fidelity of long video generation. We evaluated FreeLong on multiple base video diffusion models and observed significant improvements. Additionally, our method supports coherent multi-prompt generation, ensuring both visual coherence and seamless transitions between scenes.

Title: Boosting Graph Foundation Model from Structural Perspective

• Authors: Yao Cheng, Yige Zhao, Jianxiang Yu, Xiang Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.19941
• Pdf URL: https://arxiv.org/pdf/2407.19941
• Copy Paste: [[2407.19941]] Boosting Graph Foundation Model from Structural Perspective(https://arxiv.org/abs/2407.19941)
• Keywords: foundation model
• Abstract: Graph foundation models have recently attracted significant attention due to its strong generalizability. Although existing methods resort to language models to learn unified semantic representations across domains, they disregard the unique structural characteristics of graphs from different domains. To address the problem, in this paper, we boost graph foundation model from structural perspective and propose BooG. The model constructs virtual super nodes to unify structural characteristics of graph data from different domains. Specifically, the super nodes fuse the information of anchor nodes and class labels, where each anchor node captures the information of a node or a graph instance to be classified. Instead of using the raw graph structure, we connect super nodes to all nodes within their neighborhood by virtual edges. This new structure allows for effective information aggregation while unifying cross-domain structural characteristics. Additionally, we propose a novel pre-training objective based on contrastive learning, which learns more expressive representations for graph data and generalizes effectively to different domains and downstream tasks. Experimental results on various datasets and tasks demonstrate the superior performance of BooG. We provide our code and data here: https://anonymous.4open.science/r/BooG-EE42/.

Title: Can I trust my anomaly detection system? A case study based on explainable AI

• Authors: Muhammad Rashid, Elvio Amparore, Enrico Ferrari, Damiano Verda
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19951
• Pdf URL: https://arxiv.org/pdf/2407.19951
• Copy Paste: [[2407.19951]] Can I trust my anomaly detection system? A case study based on explainable AI(https://arxiv.org/abs/2407.19951)
• Keywords: generative, anomaly
• Abstract: Generative models based on variational autoencoders are a popular technique for detecting anomalies in images in a semi-supervised context. A common approach employs the anomaly score to detect the presence of anomalies, and it is known to reach high level of accuracy on benchmark datasets. However, since anomaly scores are computed from reconstruction disparities, they often obscure the detection of various spurious features, raising concerns regarding their actual efficacy. This case study explores the robustness of an anomaly detection system based on variational autoencoder generative models through the use of eXplainable AI methods. The goal is to get a different perspective on the real performances of anomaly detectors that use reconstruction differences. In our case study we discovered that, in many cases, samples are detected as anomalous for the wrong or misleading factors.

Title: FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models

• Authors: Mingzhao Yang, Shangchao Su, Bin Li, Xiangyang Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19953
• Pdf URL: https://arxiv.org/pdf/2407.19953
• Copy Paste: [[2407.19953]] FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models(https://arxiv.org/abs/2407.19953)
• Keywords: diffusion
• Abstract: In recent years, the attention towards One-Shot Federated Learning (OSFL) has been driven by its capacity to minimize communication. With the development of the diffusion model (DM), several methods employ the DM for OSFL, utilizing model parameters, image features, or textual prompts as mediums to transfer the local client knowledge to the server. However, these mediums often require public datasets or the uniform feature extractor, significantly limiting their practicality. In this paper, we propose FedDEO, a Description-Enhanced One-Shot Federated Learning Method with DMs, offering a novel exploration of utilizing the DM in OSFL. The core idea of our method involves training local descriptions on the clients, serving as the medium to transfer the knowledge of the distributed clients to the server. Firstly, we train local descriptions on the client data to capture the characteristics of client distributions, which are then uploaded to the server. On the server, the descriptions are used as conditions to guide the DM in generating synthetic datasets that comply with the distributions of various clients, enabling the training of the aggregated model. Theoretical analyses and sufficient quantitation and visualization experiments on three large-scale real-world datasets demonstrate that through the training of local descriptions, the server is capable of generating synthetic datasets with high quality and diversity. Consequently, with advantages in communication and privacy protection, the aggregated model outperforms compared FL or diffusion-based OSFL methods and, on some clients, outperforms the performance ceiling of centralized training.

Title: Reproducibility Study of "ITI-GEN: Inclusive Text-to-Image Generation"

• Authors: Daniel Gallo Fernández, Răzvan-Andrei Matisan, Alejandro Monroy Muñoz, Janusz Partyka
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19996
• Pdf URL: https://arxiv.org/pdf/2407.19996
• Copy Paste: [[2407.19996]] Reproducibility Study of "ITI-GEN: Inclusive Text-to-Image Generation"(https://arxiv.org/abs/2407.19996)
• Keywords: generative
• Abstract: Text-to-image generative models often present issues regarding fairness with respect to certain sensitive attributes, such as gender or skin tone. This study aims to reproduce the results presented in "ITI-GEN: Inclusive Text-to-Image Generation" by Zhang et al. (2023a), which introduces a model to improve inclusiveness in these kinds of models. We show that most of the claims made by the authors about ITI-GEN hold: it improves the diversity and quality of generated images, it is scalable to different domains, it has plug-and-play capabilities, and it is efficient from a computational point of view. However, ITI-GEN sometimes uses undesired attributes as proxy features and it is unable to disentangle some pairs of (correlated) attributes such as gender and baldness. In addition, when the number of considered attributes increases, the training time grows exponentially and ITI-GEN struggles to generate inclusive images for all elements in the joint distribution. To solve these issues, we propose using Hard Prompt Search with negative prompting, a method that does not require training and that handles negation better than vanilla Hard Prompt Search. Nonetheless, Hard Prompt Search (with or without negative prompting) cannot be used for continuous attributes that are hard to express in natural language, an area where ITI-GEN excels as it is guided by images during training. Finally, we propose combining ITI-GEN and Hard Prompt Search with negative prompting.

Title: ImagiNet: A Multi-Content Dataset for Generalizable Synthetic Image Detection via Contrastive Learning

• Authors: Delyan Boychev, Radostin Cholakov
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.20020
• Pdf URL: https://arxiv.org/pdf/2407.20020
• Copy Paste: [[2407.20020]] ImagiNet: A Multi-Content Dataset for Generalizable Synthetic Image Detection via Contrastive Learning(https://arxiv.org/abs/2407.20020)
• Keywords: diffusion, self-supervised, generative
• Abstract: Generative models, such as diffusion models (DMs), variational autoencoders (VAEs), and generative adversarial networks (GANs), produce images with a level of authenticity that makes them nearly indistinguishable from real photos and artwork. While this capability is beneficial for many industries, the difficulty of identifying synthetic images leaves online media platforms vulnerable to impersonation and misinformation attempts. To support the development of defensive methods, we introduce ImagiNet, a high-resolution and balanced dataset for synthetic image detection, designed to mitigate potential biases in existing resources. It contains 200K examples, spanning four content categories: photos, paintings, faces, and uncategorized. Synthetic images are produced with open-source and proprietary generators, whereas real counterparts of the same content type are collected from public datasets. The structure of ImagiNet allows for a two-track evaluation system: i) classification as real or synthetic and ii) identification of the generative model. To establish a baseline, we train a ResNet-50 model using a self-supervised contrastive objective (SelfCon) for each track. The model demonstrates state-of-the-art performance and high inference speed across established benchmarks, achieving an AUC of up to 0.99 and balanced accuracy ranging from 86% to 95%, even under social network conditions that involve compression and resizing. Our data and code are available at this https URL.

Title: Aircraft Trajectory Segmentation-based Contrastive Coding: A Framework for Self-supervised Trajectory Representation

• Authors: Thaweerath Phisannupawong, Joshua Julian Damanik, Han-Lim Choi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.20028
• Pdf URL: https://arxiv.org/pdf/2407.20028
• Copy Paste: [[2407.20028]] Aircraft Trajectory Segmentation-based Contrastive Coding: A Framework for Self-supervised Trajectory Representation(https://arxiv.org/abs/2407.20028)
• Keywords: self-supervised
• Abstract: Air traffic trajectory recognition has gained significant interest within the air traffic management community, particularly for fundamental tasks such as classification and clustering. This paper introduces Aircraft Trajectory Segmentation-based Contrastive Coding (ATSCC), a novel self-supervised time series representation learning framework designed to capture semantic information in air traffic trajectory data. The framework leverages the segmentable characteristic of trajectories and ensures consistency within the self-assigned segments. Intensive experiments were conducted on datasets from three different airports, totaling four datasets, comparing the learned representation's performance of downstream classification and clustering with other state-of-the-art representation learning techniques. The results show that ATSCC outperforms these methods by aligning with the labels defined by aeronautical procedures. ATSCC is adaptable to various airport configurations and scalable to incomplete trajectories. This research has expanded upon existing capabilities, achieving these improvements independently without predefined inputs such as airport configurations, maneuvering procedures, or labeled data.

Title: MaskInversion: Localized Embeddings via Optimization of Explainability Maps

• Authors: Walid Bousselham, Sofian Chaybouti, Christian Rupprecht, Vittorio Ferrari, Hilde Kuehne
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.20034
• Pdf URL: https://arxiv.org/pdf/2407.20034
• Copy Paste: [[2407.20034]] MaskInversion: Localized Embeddings via Optimization of Explainability Maps(https://arxiv.org/abs/2407.20034)
• Keywords: foundation model
• Abstract: Vision-language foundation models such as CLIP have achieved tremendous results in global vision-language alignment, but still show some limitations in creating representations for specific image regions. % To address this problem, we propose MaskInversion, a method that leverages the feature representations of pre-trained foundation models, such as CLIP, to generate a context-aware embedding for a query image region specified by a mask at test time. MaskInversion starts with initializing an embedding token and compares its explainability map, derived from the foundation model, to the query mask. The embedding token is then subsequently refined to approximate the query region by minimizing the discrepancy between its explainability map and the query mask. During this process, only the embedding vector is updated, while the underlying foundation model is kept frozen allowing to use MaskInversion with any pre-trained model. As deriving the explainability map involves computing its gradient, which can be expensive, we propose a gradient decomposition strategy that simplifies this computation. The learned region representation can be used for a broad range of tasks, including open-vocabulary class retrieval, referring expression comprehension, as well as for localized captioning and image generation. We evaluate the proposed method on all those tasks on several datasets such as PascalVOC, MSCOCO, RefCOCO, and OpenImagesV7 and show its capabilities compared to other SOTA approaches.

Title: Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning

• Authors: Liyuan Mao, Haoran Xu, Weinan Zhang, Xianyuan Zhan, Amy Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.20109
• Pdf URL: https://arxiv.org/pdf/2407.20109
• Copy Paste: [[2407.20109]] Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning(https://arxiv.org/abs/2407.20109)
• Keywords: diffusion
• Abstract: One important property of DIstribution Correction Estimation (DICE) methods is that the solution is the optimal stationary distribution ratio between the optimized and data collection policy. In this work, we show that DICE-based methods can be viewed as a transformation from the behavior distribution to the optimal policy distribution. Based on this, we propose a novel approach, Diffusion-DICE, that directly performs this transformation using diffusion models. We find that the optimal policy's score function can be decomposed into two terms: the behavior policy's score function and the gradient of a guidance term which depends on the optimal distribution ratio. The first term can be obtained from a diffusion model trained on the dataset and we propose an in-sample learning objective to learn the second term. Due to the multi-modality contained in the optimal policy distribution, the transformation in Diffusion-DICE may guide towards those local-optimal modes. We thus generate a few candidate actions and carefully select from them to approach global-optimum. Different from all other diffusion-based offline RL methods, the guide-then-select paradigm in Diffusion-DICE only uses in-sample actions for training and brings minimal error exploitation in the value function. We use a didatic toycase example to show how previous diffusion-based methods fail to generate optimal actions due to leveraging these errors and how Diffusion-DICE successfully avoids that. We then conduct extensive experiments on benchmark datasets to show the strong performance of Diffusion-DICE.

Title: Adaptive Self-supervised Robust Clustering for Unstructured Data with Unknown Cluster Number

• Authors: Chen-Lu Ding, Jiancan Wu, Wei Lin, Shiyang Shen, Xiang Wang, Yancheng Yuan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.20119
• Pdf URL: https://arxiv.org/pdf/2407.20119
• Copy Paste: [[2407.20119]] Adaptive Self-supervised Robust Clustering for Unstructured Data with Unknown Cluster Number(https://arxiv.org/abs/2407.20119)
• Keywords: self-supervised
• Abstract: We introduce a novel self-supervised deep clustering approach tailored for unstructured data without requiring prior knowledge of the number of clusters, termed Adaptive Self-supervised Robust Clustering (ASRC). In particular, ASRC adaptively learns the graph structure and edge weights to capture both local and global structural information. The obtained graph enables us to learn clustering-friendly feature representations by an enhanced graph auto-encoder with contrastive learning technique. It further leverages the clustering results adaptively obtained by robust continuous clustering (RCC) to generate prototypes for negative sampling, which can further contribute to promoting consistency among positive pairs and enlarging the gap between positive and negative samples. ASRC obtains the final clustering results by applying RCC to the learned feature representations with their consistent graph structure and edge weights. Extensive experiments conducted on seven benchmark datasets demonstrate the efficacy of ASRC, demonstrating its superior performance over other popular clustering models. Notably, ASRC even outperforms methods that rely on prior knowledge of the number of clusters, highlighting its effectiveness in addressing the challenges of clustering unstructured data.

Title: DDAP: Dual-Domain Anti-Personalization against Text-to-Image Diffusion Models

• Authors: Jing Yang, Runping Xi, Yingxin Lai, Xun Lin, Zitong Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.20141
• Pdf URL: https://arxiv.org/pdf/2407.20141
• Copy Paste: [[2407.20141]] DDAP: Dual-Domain Anti-Personalization against Text-to-Image Diffusion Models(https://arxiv.org/abs/2407.20141)
• Keywords: diffusion
• Abstract: Diffusion-based personalized visual content generation technologies have achieved significant breakthroughs, allowing for the creation of specific objects by just learning from a few reference photos. However, when misused to fabricate fake news or unsettling content targeting individuals, these technologies could cause considerable societal harm. To address this problem, current methods generate adversarial samples by adversarially maximizing the training loss, thereby disrupting the output of any personalized generation model trained with these samples. However, the existing methods fail to achieve effective defense and maintain stealthiness, as they overlook the intrinsic properties of diffusion models. In this paper, we introduce a novel Dual-Domain Anti-Personalization framework (DDAP). Specifically, we have developed Spatial Perturbation Learning (SPL) by exploiting the fixed and perturbation-sensitive nature of the image encoder in personalized generation. Subsequently, we have designed a Frequency Perturbation Learning (FPL) method that utilizes the characteristics of diffusion models in the frequency domain. The SPL disrupts the overall texture of the generated images, while the FPL focuses on image details. By alternating between these two methods, we construct the DDAP framework, effectively harnessing the strengths of both domains. To further enhance the visual quality of the adversarial samples, we design a localization module to accurately capture attentive areas while ensuring the effectiveness of the attack and avoiding unnecessary disturbances in the background. Extensive experiments on facial benchmarks have shown that the proposed DDAP enhances the disruption of personalized generation models while also maintaining high quality in adversarial samples, making it more effective in protecting privacy in practical applications.

Title: Diffusion Feedback Helps CLIP See Better

• Authors: Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, Xinlong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.20171
• Pdf URL: https://arxiv.org/pdf/2407.20171
• Copy Paste: [[2407.20171]] Diffusion Feedback Helps CLIP See Better(https://arxiv.org/abs/2407.20171)
• Keywords: diffusion, self-supervised, generative
• Abstract: Contrastive Language-Image Pre-training (CLIP), which excels at abstracting open-world representations across domains and modalities, has become a foundation for a variety of vision and multimodal tasks. However, recent studies reveal that CLIP has severe visual shortcomings, such as which can hardly distinguish orientation, quantity, color, structure, etc. These visual shortcomings also limit the perception capabilities of multimodal large language models (MLLMs) built on CLIP. The main reason could be that the image-text pairs used to train CLIP are inherently biased, due to the lack of the distinctiveness of the text and the diversity of images. In this work, we present a simple post-training approach for CLIP models, which largely overcomes its visual shortcomings via a self-supervised diffusion process. We introduce DIVA, which uses the DIffusion model as a Visual Assistant for CLIP. Specifically, DIVA leverages generative feedback from text-to-image diffusion models to optimize CLIP representations, with only images (without corresponding text). We demonstrate that DIVA improves CLIP's performance on the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities to a large extent (e.g., 3-7%), and enhances the performance of MLLMs and vision models on multimodal understanding and segmentation tasks. Extensive evaluation on 29 image classification and retrieval benchmarks confirms that our framework preserves CLIP's strong zero-shot capabilities. The code will be available at this https URL.

Title: Towards Localized Fine-Grained Control for Facial Expression Generation

• Authors: Tuomas Varanka, Huai-Qian Khor, Yante Li, Mengting Wei, Hanwei Kung, Nicu Sebe, Guoying Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.20175
• Pdf URL: https://arxiv.org/pdf/2407.20175
• Copy Paste: [[2407.20175]] Towards Localized Fine-Grained Control for Facial Expression Generation(https://arxiv.org/abs/2407.20175)
• Keywords: generative
• Abstract: Generative models have surged in popularity recently due to their ability to produce high-quality images and video. However, steering these models to produce images with specific attributes and precise control remains challenging. Humans, particularly their faces, are central to content generation due to their ability to convey rich expressions and intent. Current generative models mostly generate flat neutral expressions and characterless smiles without authenticity. Other basic expressions like anger are possible, but are limited to the stereotypical expression, while other unconventional facial expressions like doubtful are difficult to reliably generate. In this work, we propose the use of AUs (action units) for facial expression control in face generation. AUs describe individual facial muscle movements based on facial anatomy, allowing precise and localized control over the intensity of facial movements. By combining different action units, we unlock the ability to create unconventional facial expressions that go beyond typical emotional models, enabling nuanced and authentic reactions reflective of real-world expressions. The proposed method can be seamlessly integrated with both text and image prompts using adapters, offering precise and intuitive control of the generated results. Code and dataset are available in {this https URL}.

Title: SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction

• Authors: Çağhan Köksal, Ghazal Ghazaei, Felix Holm, Azade Farshad, Nassir Navab
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.20214
• Pdf URL: https://arxiv.org/pdf/2407.20214
• Copy Paste: [[2407.20214]] SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction(https://arxiv.org/abs/2407.20214)
• Keywords: foundation model
• Abstract: Graph-based holistic scene representations facilitate surgical workflow understanding and have recently demonstrated significant success. However, this task is often hindered by the limited availability of densely annotated surgical scene data. In this work, we introduce an end-to-end framework for the generation and optimization of surgical scene graphs on a downstream task. Our approach leverages the flexibility of graph-based spectral clustering and the generalization capability of foundation models to generate unsupervised scene graphs with learnable properties. We reinforce the initial spatial graph with sparse temporal connections using local matches between consecutive frames to predict temporally consistent clusters across a temporal neighborhood. By jointly optimizing the spatiotemporal relations and node features of the dynamic scene graph with the downstream task of phase segmentation, we address the costly and annotation-burdensome task of semantic scene comprehension and scene graph generation in surgical videos using only weak surgical phase labels. Further, by incorporating effective intermediate scene representation disentanglement steps within the pipeline, our solution outperforms the SOTA on the CATARACTS dataset by 8% accuracy and 10% F1 score in surgical workflow recognition

Title: Improving 2D Feature Representations by 3D-Aware Fine-Tuning

• Authors: Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, Jan Eric Lenssen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.20229
• Pdf URL: https://arxiv.org/pdf/2407.20229
• Copy Paste: [[2407.20229]] Improving 2D Feature Representations by 3D-Aware Fine-Tuning(https://arxiv.org/abs/2407.20229)
• Keywords: foundation model
• Abstract: Current visual foundation models are trained purely on unstructured 2D data, limiting their understanding of 3D structure of objects and scenes. In this work, we show that fine-tuning on 3D-aware data improves the quality of emerging semantic features. We design a method to lift semantic 2D features into an efficient 3D Gaussian representation, which allows us to re-render them for arbitrary views. Using the rendered 3D-aware features, we design a fine-tuning strategy to transfer such 3D awareness into a 2D foundation model. We demonstrate that models fine-tuned in that way produce features that readily improve downstream task performance in semantic segmentation and depth estimation through simple linear probing. Notably, though fined-tuned on a single indoor dataset, the improvement is transferable to a variety of indoor datasets and out-of-domain datasets. We hope our study encourages the community to consider injecting 3D awareness when training 2D foundation models. Project page: this https URL.

Title: Specify and Edit: Overcoming Ambiguity in Text-Based Image Editing

• Authors: Ekaterina Iakovleva, Fabio Pizzati, Philip Torr, Stéphane Lathuilière
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.20232
• Pdf URL: https://arxiv.org/pdf/2407.20232
• Copy Paste: [[2407.20232]] Specify and Edit: Overcoming Ambiguity in Text-Based Image Editing(https://arxiv.org/abs/2407.20232)
• Keywords: diffusion
• Abstract: Text-based editing diffusion models exhibit limited performance when the user's input instruction is ambiguous. To solve this problem, we propose $\textit{Specify ANd Edit}$ (SANE), a zero-shot inference pipeline for diffusion-based editing systems. We use a large language model (LLM) to decompose the input instruction into specific instructions, i.e. well-defined interventions to apply to the input image to satisfy the user's request. We benefit from the LLM-derived instructions along the original one, thanks to a novel denoising guidance strategy specifically designed for the task. Our experiments with three baselines and on two datasets demonstrate the benefits of SANE in all setups. Moreover, our pipeline improves the interpretability of editing models, and boosts the output diversity. We also demonstrate that our approach can be applied to any edit, whether ambiguous or not. Our code is public at this https URL.