2024-07-24

Title: A Survey on Trustworthiness in Foundation Models for Medical Image Analysis

• Authors: Congzhen Shi, Ryan Rezai, Jiaxi Yang, Qi Dou, Xiaoxiao Li
• Subjects: cs.CV, cs.AI, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15851
• Pdf URL: https://arxiv.org/pdf/2407.15851
• Copy Paste: [[2407.15851]] A Survey on Trustworthiness in Foundation Models for Medical Image Analysis(https://arxiv.org/abs/2407.15851)
• Keywords: foundation model
• Abstract: The rapid advancement of foundation models in medical imaging represents a significant leap toward enhancing diagnostic accuracy and personalized treatment. However, the deployment of foundation models in healthcare necessitates a rigorous examination of their trustworthiness, encompassing privacy, robustness, reliability, explainability, and fairness. The current body of survey literature on foundation models in medical imaging reveals considerable gaps, particularly in the area of trustworthiness. Additionally, extant surveys on the trustworthiness of foundation models fail to address their specific variations and applications within the medical imaging domain. This survey paper reviews the current research on foundation models in the major medical imaging applications, with a focus on segmentation, medical report generation, medical question and answering (Q&A), and disease diagnosis, which includes trustworthiness discussion in their manuscripts. We explore the complex challenges of making foundation models for medical image analysis trustworthy, associated with each application, and summarize the current concerns and strategies to enhance trustworthiness. Furthermore, we explore the future promises of these models in revolutionizing patient care. Our analysis underscores the imperative for advancing towards trustworthy AI in medical image analysis, advocating for a balanced approach that fosters innovation while ensuring ethical and equitable healthcare delivery.

Title: Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Survey

• Authors: Chenyu Zhang, Mingwang Hu, Wenhui Li, Lanjun Wang
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.15861
• Pdf URL: https://arxiv.org/pdf/2407.15861
• Copy Paste: [[2407.15861]] Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Survey(https://arxiv.org/abs/2407.15861)
• Keywords: diffusion
• Abstract: Recently, the text-to-image diffusion model has gained considerable attention from the community due to its exceptional image generation capability. A representative model, Stable Diffusion, amassed more than 10 million users within just two months of its release. This surge in popularity has facilitated studies on the robustness and safety of the model, leading to the proposal of various adversarial attack methods. Simultaneously, there has been a marked increase in research focused on defense methods to improve the robustness and safety of these models. In this survey, we provide a comprehensive review of the literature on adversarial attacks and defenses targeting text-to-image diffusion models. We begin with an overview of text-to-image diffusion models, followed by an introduction to a taxonomy of adversarial attacks and an in-depth review of existing attack methods. We then present a detailed analysis of current defense methods that improve model robustness and safety. Finally, we discuss ongoing challenges and explore promising future research directions. For a complete list of the adversarial attack and defense methods covered in this survey, please refer to our curated repository at this https URL.

Title: SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization

• Authors: Rui Xie, Asad Ul Haq, Linsen Ma, Krystal Sun, Sanchari Sen, Swagath Venkataramani, Liu Liu, Tong Zhang
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2407.15866
• Pdf URL: https://arxiv.org/pdf/2407.15866
• Copy Paste: [[2407.15866]] SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization(https://arxiv.org/abs/2407.15866)
• Keywords: generative
• Abstract: Recent studies have revealed that, during the inference on generative AI models such as transformer, the importance of different weights exhibits substantial context-dependent variations. This naturally manifests a promising potential of adaptively configuring weight quantization to improve the generative AI inference efficiency. Although configurable weight quantization can readily leverage the hardware support of variable-precision arithmetics in modern GPU and AI accelerators, little prior research has studied how one could exploit variable weight quantization to proportionally improve the AI model memory access speed and energy efficiency. Motivated by the rapidly maturing CXL ecosystem, this work develops a CXL-based design solution to fill this gap. The key is to allow CXL memory controllers play an active role in supporting and exploiting runtime configurable weight quantization. Using transformer as a representative generative AI model, we carried out experiments that well demonstrate the effectiveness of the proposed design solution.

Title: A reinforcement learning strategy to automate and accelerate h/p-multigrid solvers

• Authors: David Huergo, Laura Alonso, Saumitra Joshi, Adrian Juanicoteca, Gonzalo Rubio, Esteban Ferrer
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2407.15872
• Pdf URL: https://arxiv.org/pdf/2407.15872
• Copy Paste: [[2407.15872]] A reinforcement learning strategy to automate and accelerate h/p-multigrid solvers(https://arxiv.org/abs/2407.15872)
• Keywords: diffusion
• Abstract: We explore a reinforcement learning strategy to automate and accelerate h/p-multigrid methods in high-order solvers. Multigrid methods are very efficient but require fine-tuning of numerical parameters, such as the number of smoothing sweeps per level and the correction fraction (i.e., proportion of the corrected solution that is transferred from a coarser grid to a finer grid). The objective of this paper is to use a proximal policy optimization algorithm to automatically tune the multigrid parameters and, by doing so, improve stability and efficiency of the h/p-multigrid strategy. Our findings reveal that the proposed reinforcement learning h/p-multigrid approach significantly accelerates and improves the robustness of steady-state simulations for one dimensional advection-diffusion and nonlinear Burgers' equations, when discretized using high-order h/p methods, on uniform and nonuniform grids.

Title: Decentralized Federated Anomaly Detection in Smart Grids: A P2P Gossip Approach

• Authors: Muhammad Akbar Husnoo, Adnan Anwar, Md Enamul Haque, A. N. Mahmood
• Subjects: cs.CR, cs.AI, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15879
• Pdf URL: https://arxiv.org/pdf/2407.15879
• Copy Paste: [[2407.15879]] Decentralized Federated Anomaly Detection in Smart Grids: A P2P Gossip Approach(https://arxiv.org/abs/2407.15879)
• Keywords: anomaly
• Abstract: The increasing security and privacy concerns in the Smart Grid sector have led to a significant demand for robust intrusion detection systems within critical smart grid infrastructure. To address the challenges posed by privacy preservation and decentralized power system zones with distinct data ownership, Federated Learning (FL) has emerged as a promising privacy-preserving solution which facilitates collaborative training of attack detection models without necessitating the sharing of raw data. However, FL presents several implementation limitations in the power system domain due to its heavy reliance on a centralized aggregator and the risks of privacy leakage during model update transmission. To overcome these technical bottlenecks, this paper introduces a novel decentralized federated anomaly detection scheme based on two main gossip protocols namely Random Walk and Epidemic. Our findings indicate that the Random Walk protocol exhibits superior performance compared to the Epidemic protocol, highlighting its efficacy in decentralized federated learning environments. Experimental validation of the proposed framework utilizing publicly available industrial control systems datasets demonstrates superior attack detection accuracy while safeguarding data confidentiality and mitigating the impact of communication latency and stragglers. Furthermore, our approach yields a notable 35% improvement in training time compared to conventional FL, underscoring the efficacy and robustness of our decentralized learning method.

Title: Diff4VS: HIV-inhibiting Molecules Generation with Classifier Guidance Diffusion for Virtual Screening

• Authors: Jiaqing Lyu, Changjie Chen, Bing Liang, Yijia Zhang
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2407.15880
• Pdf URL: https://arxiv.org/pdf/2407.15880
• Copy Paste: [[2407.15880]] Diff4VS: HIV-inhibiting Molecules Generation with Classifier Guidance Diffusion for Virtual Screening(https://arxiv.org/abs/2407.15880)
• Keywords: diffusion, generative
• Abstract: The AIDS epidemic has killed 40 million people and caused serious global problems. The identification of new HIV-inhibiting molecules is of great importance for combating the AIDS epidemic. Here, the Classifier Guidance Diffusion model and ligand-based virtual screening strategy are combined to discover potential HIV-inhibiting molecules for the first time. We call it Diff4VS. An extra classifier is trained using the HIV molecule dataset, and the gradient of the classifier is used to guide the Diffusion to generate HIV-inhibiting molecules. Experiments show that Diff4VS can generate more candidate HIV-inhibiting molecules than other methods. Inspired by ligand-based virtual screening, a new metric DrugIndex is proposed. The DrugIndex is the ratio of the proportion of candidate drug molecules in the generated molecule to the proportion of candidate drug molecules in the training set. DrugIndex provides a new evaluation method for evolving molecular generative models from a pharmaceutical perspective. Besides, we report a new phenomenon observed when using molecule generation models for virtual screening. Compared to real molecules, the generated molecules have a lower proportion that is highly similar to known drug molecules. We call it Degradation in molecule generation. Based on the data analysis, the Degradation may result from the difficulty of generating molecules with a specific structure in the generative model. Our research contributes to the application of generative models in drug design from method, metric, and phenomenon analysis.

Title: CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models

• Authors: Zheng Chong, Xiao Dong, Haoxiang Li, Shiyue Zhang, Wenqing Zhang, Xujie Zhang, Hanqing Zhao, Xiaodan Liang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15886
• Pdf URL: https://arxiv.org/pdf/2407.15886
• Copy Paste: [[2407.15886]] CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models(https://arxiv.org/abs/2407.15886)
• Keywords: diffusion
• Abstract: Virtual try-on methods based on diffusion models achieve realistic try-on effects but often replicate the backbone network as a ReferenceNet or use additional image encoders to process condition inputs, leading to high training and inference costs. In this work, we rethink the necessity of ReferenceNet and image encoders and innovate the interaction between garment and person by proposing CatVTON, a simple and efficient virtual try-on diffusion model. CatVTON facilitates the seamless transfer of in-shop or worn garments of any category to target persons by simply concatenating them in spatial dimensions as inputs. The efficiency of our model is demonstrated in three aspects: (1) Lightweight network: Only the original diffusion modules are used, without additional network modules. The text encoder and cross-attentions for text injection in the backbone are removed, reducing the parameters by 167.02M. (2) Parameter-efficient training: We identified the try-on relevant modules through experiments and achieved high-quality try-on effects by training only 49.57M parameters, approximately 5.51 percent of the backbone network's parameters. (3) Simplified inference: CatVTON eliminates all unnecessary conditions and preprocessing steps, including pose estimation, human parsing, and text input, requiring only a garment reference, target person image, and mask for the virtual try-on process. Extensive experiments demonstrate that CatVTON achieves superior qualitative and quantitative results with fewer prerequisites and trainable parameters than baseline methods. Furthermore, CatVTON shows good generalization in in-the-wild scenarios despite using open-source datasets with only 73K samples.

Title: Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models

• Authors: Aayush Saxena, Arit Kumar Bishwas, Ayush Ashok Mishra, Ryan Armstrong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15904
• Pdf URL: https://arxiv.org/pdf/2407.15904
• Copy Paste: [[2407.15904]] Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models(https://arxiv.org/abs/2407.15904)
• Keywords: generative
• Abstract: Deep learning models have achieved tremendous success in most of the industries in recent years. The evolution of these models has also led to an increase in the model size and energy requirement, making it difficult to deploy in production on low compute devices. An increase in the number of connected devices around the world warrants compressed models that can be easily deployed at the local devices with low compute capacity and power accessibility. A wide range of solutions have been proposed by different researchers to reduce the size and complexity of such models, prominent among them are, Weight Quantization, Parameter Pruning, Network Pruning, low-rank representation, weights sharing, neural architecture search, knowledge distillation etc. In this research work, we investigate the performance impacts on various trained deep learning models, compressed using quantization and pruning techniques. We implemented both, quantization and pruning, compression techniques on popular deep learning models used in the image classification, object detection, language models and generative models-based problem statements. We also explored performance of various large language models (LLMs) after quantization and low rank adaptation. We used the standard evaluation metrics (model's size, accuracy, and inference time) for all the related problem statements and concluded this paper by discussing the challenges and future work.

Title: The Shadow of Fraud: The Emerging Danger of AI-powered Social Engineering and its Possible Cure

• Authors: Jingru Yu, Yi Yu, Xuhong Wang, Yilun Lin, Manzhi Yang, Yu Qiao, Fei-Yue Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.15912
• Pdf URL: https://arxiv.org/pdf/2407.15912
• Copy Paste: [[2407.15912]] The Shadow of Fraud: The Emerging Danger of AI-powered Social Engineering and its Possible Cure(https://arxiv.org/abs/2407.15912)
• Keywords: diffusion
• Abstract: Social engineering (SE) attacks remain a significant threat to both individuals and organizations. The advancement of Artificial Intelligence (AI), including diffusion models and large language models (LLMs), has potentially intensified these threats by enabling more personalized and convincing attacks. This survey paper categorizes SE attack mechanisms, analyzes their evolution, and explores methods for measuring these threats. It highlights the challenges in raising awareness about the risks of AI-enhanced SE attacks and offers insights into developing proactive and adaptable defense strategies. Additionally, we introduce a categorization of the evolving nature of AI-powered social engineering attacks into "3E phases": Enlarging, wherein the magnitude of attacks expands through the leverage of digital media; Enriching, introducing novel attack vectors and techniques; and Emerging, signifying the advent of novel threats and methods. Moreover, we emphasize the necessity for a robust framework to assess the risk of AI-powered SE attacks. By identifying and addressing gaps in existing research, we aim to guide future studies and encourage the development of more effective defenses against the growing threat of AI-powered social engineering.

Title: Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models

• Authors: Raza Imam, Hanan Gani, Muhammad Huzaifa, Karthik Nandakumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.15913
• Pdf URL: https://arxiv.org/pdf/2407.15913
• Copy Paste: [[2407.15913]] Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models(https://arxiv.org/abs/2407.15913)
• Keywords: self-supervised
• Abstract: The conventional modus operandi for adapting pre-trained vision-language models (VLMs) during test-time involves tuning learnable prompts, ie, test-time prompt tuning. This paper introduces Test-Time Low-rank adaptation (TTL) as an alternative to prompt tuning for zero-shot generalization of large-scale VLMs. Taking inspiration from recent advancements in efficiently fine-tuning large language models, TTL offers a test-time parameter-efficient adaptation approach that updates the attention weights of the transformer encoder by maximizing prediction confidence. The self-supervised confidence maximization objective is specified using a weighted entropy loss that enforces consistency among predictions of augmented samples. TTL introduces only a small amount of trainable parameters for low-rank adapters in the model space while keeping the prompts and backbone frozen. Extensive experiments on a variety of natural distribution and cross-domain tasks show that TTL can outperform other techniques for test-time optimization of VLMs in strict zero-shot settings. Specifically, TTL outperforms test-time prompt tuning baselines with a significant improvement on average. Our code is available at at this https URL.

Title: Multilingual Fine-Grained News Headline Hallucination Detection

• Authors: Jiaming Shen, Tianqi Liu, Jialu Liu, Zhen Qin, Jay Pavagadhi, Simon Baumgartner, Michael Bendersky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15975
• Pdf URL: https://arxiv.org/pdf/2407.15975
• Copy Paste: [[2407.15975]] Multilingual Fine-Grained News Headline Hallucination Detection(https://arxiv.org/abs/2407.15975)
• Keywords: in-context
• Abstract: The popularity of automated news headline generation has surged with advancements in pre-trained language models. However, these models often suffer from the ``hallucination'' problem, where the generated headline is not fully supported by its source article. Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types. In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts. We conduct extensive experiments on this dataset under two settings. First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset's challenges and utilities. Second, we test various large language models' in-context learning abilities and propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric. We release this dataset to foster further research in multilingual, fine-grained headline hallucination detection.

Title: KaPQA: Knowledge-Augmented Product Question-Answering

• Authors: Swetha Eppalapally, Daksh Dangi, Chaithra Bhat, Ankita Gupta, Ruiyi Zhang, Shubham Agarwal, Karishma Bagga, Seunghyun Yoon, Nedim Lipka, Ryan A. Rossi, Franck Dernoncourt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16073
• Pdf URL: https://arxiv.org/pdf/2407.16073
• Copy Paste: [[2407.16073]] KaPQA: Knowledge-Augmented Product Question-Answering(https://arxiv.org/abs/2407.16073)
• Keywords: generative
• Abstract: Question-answering for domain-specific applications has recently attracted much interest due to the latest advancements in large language models (LLMs). However, accurately assessing the performance of these applications remains a challenge, mainly due to the lack of suitable benchmarks that effectively simulate real-world scenarios. To address this challenge, we introduce two product question-answering (QA) datasets focused on Adobe Acrobat and Photoshop products to help evaluate the performance of existing models on domain-specific product QA tasks. Additionally, we propose a novel knowledge-driven RAG-QA framework to enhance the performance of the models in the product QA task. Our experiments demonstrated that inducing domain knowledge through query reformulation allowed for increased retrieval and generative performance when compared to standard RAG-QA methods. This improvement, however, is slight, and thus illustrates the challenge posed by the datasets introduced.

Title: Universal Spectral Transfer with Physical Prior-Informed Deep Generative Learning

• Authors: Yanmin Zhu, Loza F. Tadesse
• Subjects: cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.16094
• Pdf URL: https://arxiv.org/pdf/2407.16094
• Copy Paste: [[2407.16094]] Universal Spectral Transfer with Physical Prior-Informed Deep Generative Learning(https://arxiv.org/abs/2407.16094)
• Keywords: generative
• Abstract: Spectroscopy is a powerful analytical technique for characterizing matter across physical and biological realms1-5. However, its fundamental principle necessitates specialized instrumentation per physical phenomena probed, limiting broad adoption and use in all relevant research. In this study, we introduce SpectroGen, a novel physical prior-informed deep generative model for generating relevant spectral signatures across modalities using experimentally collected spectral input only from a single modality. We achieve this by reimagining the representation of spectral data as mathematical constructs of distributions instead of their traditional physical and molecular state representations. The results from 319 standard mineral samples tested demonstrate generating with 99% correlation and 0.01 root mean square error with superior resolution than experimentally acquired ground truth spectra. We showed transferring capability across Raman, Infrared, and X-ray Diffraction modalities with Gaussian, Lorentzian, and Voigt distribution priors respectively6-10. This approach however is globally generalizable for any spectral input that can be represented by a distribution prior, making it universally applicable. We believe our work revolutionizes the application sphere of spectroscopy, which has traditionally been limited by access to the required sophisticated and often expensive equipment towards accelerating material, pharmaceutical, and biological discoveries.

Title: Fr\'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos

• Authors: Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, Renjie Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16124
• Pdf URL: https://arxiv.org/pdf/2407.16124
• Copy Paste: [[2407.16124]] Fr\'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos(https://arxiv.org/abs/2407.16124)
• Keywords: generative
• Abstract: Significant advancements have been made in video generative models recently. Unlike image generation, video generation presents greater challenges, requiring not only generating high-quality frames but also ensuring temporal consistency across these frames. Despite the impressive progress, research on metrics for evaluating the quality of generated videos, especially concerning temporal and motion consistency, remains underexplored. To bridge this research gap, we propose Fréchet Video Motion Distance (FVMD) metric, which focuses on evaluating motion consistency in video generation. Specifically, we design explicit motion features based on key point tracking, and then measure the similarity between these features via the Fréchet distance. We conduct sensitivity analysis by injecting noise into real videos to verify the effectiveness of FVMD. Further, we carry out a large-scale human study, demonstrating that our metric effectively detects temporal noise and aligns better with human perceptions of generated video quality than existing metrics. Additionally, our motion features can consistently improve the performance of Video Quality Assessment (VQA) models, indicating that our approach is also applicable to unary video quality evaluation. Code is available at this https URL.

Title: Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problems

• Authors: Sojin Lee, Dogyun Park, Inho Kong, Hyunwoo J. Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16125
• Pdf URL: https://arxiv.org/pdf/2407.16125
• Copy Paste: [[2407.16125]] Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problems(https://arxiv.org/abs/2407.16125)
• Keywords: diffusion
• Abstract: Recent studies on inverse problems have proposed posterior samplers that leverage the pre-trained diffusion models as powerful priors. These attempts have paved the way for using diffusion models in a wide range of inverse problems. However, the existing methods entail computationally demanding iterative sampling procedures and optimize a separate solution for each measurement, which leads to limited scalability and lack of generalization capability across unseen samples. To address these limitations, we propose a novel approach, Diffusion prior-based Amortized Variational Inference (DAVI) that solves inverse problems with a diffusion prior from an amortized variational inference perspective. Specifically, instead of separate measurement-wise optimization, our amortized inference learns a function that directly maps measurements to the implicit posterior distributions of corresponding clean data, enabling a single-step posterior sampling even for unseen measurements. Extensive experiments on image restoration tasks, e.g., Gaussian deblur, 4$\times$ super-resolution, and box inpainting with two benchmark datasets, demonstrate our approach's superior performance over strong baselines. Code is available at this https URL.

Title: Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion

• Authors: Yang Liu, Xiaobin Tian, Zequn Sun, Wei Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16127
• Pdf URL: https://arxiv.org/pdf/2407.16127
• Copy Paste: [[2407.16127]] Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion(https://arxiv.org/abs/2407.16127)
• Keywords: generative
• Abstract: Traditional knowledge graph (KG) completion models learn embeddings to predict missing facts. Recent works attempt to complete KGs in a text-generation manner with large language models (LLMs). However, they need to ground the output of LLMs to KG entities, which inevitably brings errors. In this paper, we present a finetuning framework, DIFT, aiming to unleash the KG completion ability of LLMs and avoid grounding errors. Given an incomplete fact, DIFT employs a lightweight model to obtain candidate entities and finetunes an LLM with discrimination instructions to select the correct one from the given candidates. To improve performance while reducing instruction data, DIFT uses a truncated sampling method to select useful facts for finetuning and injects KG embeddings into the LLM. Extensive experiments on benchmark datasets demonstrate the effectiveness of our proposed framework.

Title: Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Data

• Authors: Hengyu Fu, Zehao Dou, Jiawei Guo, Mengdi Wang, Minshuo Chen
• Subjects: cs.LG, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.16134
• Pdf URL: https://arxiv.org/pdf/2407.16134
• Copy Paste: [[2407.16134]] Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Data(https://arxiv.org/abs/2407.16134)
• Keywords: diffusion
• Abstract: Diffusion Transformer, the backbone of Sora for video generation, successfully scales the capacity of diffusion models, pioneering new avenues for high-fidelity sequential data generation. Unlike static data such as images, sequential data consists of consecutive data frames indexed by time, exhibiting rich spatial and temporal dependencies. These dependencies represent the underlying dynamic model and are critical to validate the generated data. In this paper, we make the first theoretical step towards bridging diffusion transformers for capturing spatial-temporal dependencies. Specifically, we establish score approximation and distribution estimation guarantees of diffusion transformers for learning Gaussian process data with covariance functions of various decay patterns. We highlight how the spatial-temporal dependencies are captured and affect learning efficiency. Our study proposes a novel transformer approximation theory, where the transformer acts to unroll an algorithm. We support our theoretical results by numerical experiments, providing strong evidence that spatial-temporal dependencies are captured within attention layers, aligning with our approximation theory.

Title: Diffusion Models as Optimizers for Efficient Planning in Offline RL

• Authors: Renming Huang, Yunqiang Pei, Guoqing Wang, Yangming Zhang, Yang Yang, Peng Wang, Hengtao Shen
• Subjects: cs.LG, cs.AI, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.16142
• Pdf URL: https://arxiv.org/pdf/2407.16142
• Copy Paste: [[2407.16142]] Diffusion Models as Optimizers for Efficient Planning in Offline RL(https://arxiv.org/abs/2407.16142)
• Keywords: diffusion
• Abstract: Diffusion models have shown strong competitiveness in offline reinforcement learning tasks by formulating decision-making as sequential generation. However, the practicality of these methods is limited due to the lengthy inference processes they require. In this paper, we address this problem by decomposing the sampling process of diffusion models into two decoupled subprocesses: 1) generating a feasible trajectory, which is a time-consuming process, and 2) optimizing the trajectory. With this decomposition approach, we are able to partially separate efficiency and quality factors, enabling us to simultaneously gain efficiency advantages and ensure quality assurance. We propose the Trajectory Diffuser, which utilizes a faster autoregressive model to handle the generation of feasible trajectories while retaining the trajectory optimization process of diffusion models. This allows us to achieve more efficient planning without sacrificing capability. To evaluate the effectiveness and efficiency of the Trajectory Diffuser, we conduct experiments on the D4RL benchmarks. The results demonstrate that our method achieves $\it 3$-$\it 10 \times$ faster inference speed compared to previous sequence modeling methods, while also outperforming them in terms of overall performance. this https URL Keywords: Reinforcement Learning and Efficient Planning and Diffusion Model

Title: Learning Trimodal Relation for AVQA with Missing Modality

• Authors: Kyu Ri Park, Hong Joo Lee, Jung Uk Kim
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.16171
• Pdf URL: https://arxiv.org/pdf/2407.16171
• Copy Paste: [[2407.16171]] Learning Trimodal Relation for AVQA with Missing Modality(https://arxiv.org/abs/2407.16171)
• Keywords: diffusion
• Abstract: Recent Audio-Visual Question Answering (AVQA) methods rely on complete visual and audio input to answer questions accurately. However, in real-world scenarios, issues such as device malfunctions and data transmission errors frequently result in missing audio or visual modality. In such cases, existing AVQA methods suffer significant performance degradation. In this paper, we propose a framework that ensures robust AVQA performance even when a modality is missing. First, we propose a Relation-aware Missing Modal (RMM) generator with Relation-aware Missing Modal Recalling (RMMR) loss to enhance the ability of the generator to recall missing modal information by understanding the relationships and context among the available modalities. Second, we design an Audio-Visual Relation-aware (AVR) diffusion model with Audio-Visual Enhancing (AVE) loss to further enhance audio-visual features by leveraging the relationships and shared cues between the audio-visual modalities. As a result, our method can provide accurate answers by effectively utilizing available information even when input modalities are missing. We believe our method holds potential applications not only in AVQA research but also in various multi-modal scenarios.

Title: No Re-Train, More Gain: Upgrading Backbones with Diffusion Model for Few-Shot Segmentation

• Authors: Shuai Chen, Fanman Meng, Chenhao Wu, Haoran Wei, Runtong Zhang, Qingbo Wu, Linfeng Xu, Hongliang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16182
• Pdf URL: https://arxiv.org/pdf/2407.16182
• Copy Paste: [[2407.16182]] No Re-Train, More Gain: Upgrading Backbones with Diffusion Model for Few-Shot Segmentation(https://arxiv.org/abs/2407.16182)
• Keywords: diffusion, generative
• Abstract: Few-Shot Segmentation (FSS) aims to segment novel classes using only a few annotated images. Despite considerable process under pixel-wise support annotation, current FSS methods still face three issues: the inflexibility of backbone upgrade without re-training, the inability to uniformly handle various types of annotations (e.g., scribble, bounding box, mask and text), and the difficulty in accommodating different annotation quantity. To address these issues simultaneously, we propose DiffUp, a novel FSS method that conceptualizes the FSS task as a conditional generative problem using a diffusion process. For the first issue, we introduce a backbone-agnostic feature transformation module that converts different segmentation cues into unified coarse priors, facilitating seamless backbone upgrade without re-training. For the second issue, due to the varying granularity of transformed priors from diverse annotation types, we conceptualize these multi-granular transformed priors as analogous to noisy intermediates at different steps of a diffusion model. This is implemented via a self-conditioned modulation block coupled with a dual-level quality modulation branch. For the third issue, we incorporates an uncertainty-aware information fusion module that harmonizing the variability across zero-shot, one-shot and many-shot scenarios. Evaluated through rigorous benchmarks, DiffUp significantly outperforms existing FSS models in terms of flexibility and accuracy.

Title: CloudFixer: Test-Time Adaptation for 3D Point Clouds via Diffusion-Guided Geometric Transformation

• Authors: Hajin Shim, Changhun Kim, Eunho Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16193
• Pdf URL: https://arxiv.org/pdf/2407.16193
• Copy Paste: [[2407.16193]] CloudFixer: Test-Time Adaptation for 3D Point Clouds via Diffusion-Guided Geometric Transformation(https://arxiv.org/abs/2407.16193)
• Keywords: diffusion
• Abstract: 3D point clouds captured from real-world sensors frequently encompass noisy points due to various obstacles, such as occlusion, limited resolution, and variations in scale. These challenges hinder the deployment of pre-trained point cloud recognition models trained on clean point clouds, leading to significant performance degradation. While test-time adaptation (TTA) strategies have shown promising results on this issue in the 2D domain, their application to 3D point clouds remains under-explored. Among TTA methods, an input adaptation approach, which directly converts test instances to the source domain using a pre-trained diffusion model, has been proposed in the 2D domain. Despite its robust TTA performance in practical situations, naively adopting this into the 3D domain may be suboptimal due to the neglect of inherent properties of point clouds, and its prohibitive computational cost. Motivated by these limitations, we propose CloudFixer, a test-time input adaptation method tailored for 3D point clouds, employing a pre-trained diffusion model. Specifically, CloudFixer optimizes geometric transformation parameters with carefully designed objectives that leverage the geometric properties of point clouds. We also substantially improve computational efficiency by avoiding backpropagation through the diffusion model and a prohibitive generation process. Furthermore, we propose an online model adaptation strategy by aligning the original model prediction with that of the adapted input. Extensive experiments showcase the superiority of CloudFixer over various TTA baselines, excelling in handling common corruptions and natural distribution shifts across diverse real-world scenarios. Our code is available at this https URL

Title: Figure it Out: Analyzing-based Jailbreak Attack on Large Language Models

• Authors: Shi Lin, Rongchang Li, Xun Wang, Changting Lin, Wenpeng Xing, Meng Han
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16205
• Pdf URL: https://arxiv.org/pdf/2407.16205
• Copy Paste: [[2407.16205]] Figure it Out: Analyzing-based Jailbreak Attack on Large Language Models(https://arxiv.org/abs/2407.16205)
• Keywords: generative
• Abstract: The rapid development of Large Language Models (LLMs) has brought remarkable generative capabilities across diverse tasks. However, despite the impressive achievements, these models still have numerous security vulnerabilities, particularly when faced with jailbreak attacks. Therefore, by investigating jailbreak attacks, we can uncover hidden weaknesses in LLMs and guide us in developing more robust defense mechanisms to fortify their security. In this paper, we further explore the boundary of jailbreak attacks on LLMs and propose Analyzing-based Jailbreak (ABJ). This effective jailbreak attack method takes advantage of LLMs' growing analyzing and reasoning capability and reveals their underlying vulnerabilities when facing analysis-based tasks. We conduct a detailed evaluation of ABJ across various open-source and closed-source LLMs, which achieves 94.8% Attack Success Rate (ASR) and 1.06 Attack Efficiency (AE) on GPT-4-turbo-0409, demonstrating state-of-the-art attack effectiveness and efficiency. Our research highlights the importance of prioritizing and enhancing the safety of LLMs to mitigate the risks of misuse.

Title: Diff-Shadow: Global-guided Diffusion Model for Shadow Removal

• Authors: Jinting Luo, Ru Li, Chengzhi Jiang, Mingyan Han, Xiaoming Zhang, Ting Jiang, Haoqiang Fan, Shuaicheng Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16214
• Pdf URL: https://arxiv.org/pdf/2407.16214
• Copy Paste: [[2407.16214]] Diff-Shadow: Global-guided Diffusion Model for Shadow Removal(https://arxiv.org/abs/2407.16214)
• Keywords: diffusion
• Abstract: We propose Diff-Shadow, a global-guided diffusion model for high-quality shadow removal. Previous transformer-based approaches can utilize global information to relate shadow and non-shadow regions but are limited in their synthesis ability and recover images with obvious boundaries. In contrast, diffusion-based methods can generate better content but ignore global information, resulting in inconsistent illumination. In this work, we combine the advantages of diffusion models and global guidance to realize shadow-free restoration. Specifically, we propose a parallel UNets architecture: 1) the local branch performs the patch-based noise estimation in the diffusion process, and 2) the global branch recovers the low-resolution shadow-free images. A Reweight Cross Attention (RCA) module is designed to integrate global contextural information of non-shadow regions into the local branch. We further design a Global-guided Sampling Strategy (GSS) that mitigates patch boundary issues and ensures consistent illumination across shaded and unshaded regions in the recovered image. Comprehensive experiments on three publicly standard datasets ISTD, ISTD+, and SRD have demonstrated the effectiveness of Diff-Shadow. Compared to state-of-the-art methods, our method achieves a significant improvement in terms of PSNR, increasing from 32.33dB to 33.69dB on the SRD dataset. Codes will be released.

Title: A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More

• Authors: Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Zixu (James)Zhu, Xiang-Bo Mao, Sitaram Asur, Na (Claire)Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16216
• Pdf URL: https://arxiv.org/pdf/2407.16216
• Copy Paste: [[2407.16216]] A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More(https://arxiv.org/abs/2407.16216)
• Keywords: self-supervised
• Abstract: With advancements in self-supervised learning, the availability of trillions tokens in a pre-training corpus, instruction fine-tuning, and the development of large Transformers with billions of parameters, large language models (LLMs) are now capable of generating factual and coherent responses to human queries. However, the mixed quality of training data can lead to the generation of undesired responses, presenting a significant challenge. Over the past two years, various methods have been proposed from different perspectives to enhance LLMs, particularly in aligning them with human expectation. Despite these efforts, there has not been a comprehensive survey paper that categorizes and details these approaches. In this work, we aim to address this gap by categorizing these papers into distinct topics and providing detailed explanations of each alignment method, thereby helping readers gain a thorough understanding of the current state of the field.

Title: OutfitAnyone: Ultra-high Quality Virtual Try-On for Any Clothing and Any Person

• Authors: Ke Sun, Jian Cao, Qi Wang, Linrui Tian, Xindi Zhang, Lian Zhuo, Bang Zhang, Liefeng Bo, Wenbo Zhou, Weiming Zhang, Daiheng Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16224
• Pdf URL: https://arxiv.org/pdf/2407.16224
• Copy Paste: [[2407.16224]] OutfitAnyone: Ultra-high Quality Virtual Try-On for Any Clothing and Any Person(https://arxiv.org/abs/2407.16224)
• Keywords: diffusion
• Abstract: Virtual Try-On (VTON) has become a transformative technology, empowering users to experiment with fashion without ever having to physically try on clothing. However, existing methods often struggle with generating high-fidelity and detail-consistent results. While diffusion models, such as Stable Diffusion series, have shown their capability in creating high-quality and photorealistic images, they encounter formidable challenges in conditional generation scenarios like VTON. Specifically, these models struggle to maintain a balance between control and consistency when generating images for virtual clothing trials. OutfitAnyone addresses these limitations by leveraging a two-stream conditional diffusion model, enabling it to adeptly handle garment deformation for more lifelike results. It distinguishes itself with scalability-modulating factors such as pose, body shape and broad applicability, extending from anime to in-the-wild images. OutfitAnyone's performance in diverse scenarios underscores its utility and readiness for real-world deployment. For more details and animated results, please see \url{this https URL}.

Title: Self-Reasoning Assistant Learning for non-Abelian Gauge Fields Design

• Authors: Jinyang Sun, Xi Chen, Xiumei Wang, Dandan Zhu, Xingping Zhou
• Subjects: cs.LG, cond-mat.mes-hall, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16255
• Pdf URL: https://arxiv.org/pdf/2407.16255
• Copy Paste: [[2407.16255]] Self-Reasoning Assistant Learning for non-Abelian Gauge Fields Design(https://arxiv.org/abs/2407.16255)
• Keywords: diffusion
• Abstract: Non-Abelian braiding has attracted substantial attention because of its pivotal role in describing the exchange behaviour of anyons, in which the input and outcome of non-Abelian braiding are connected by a unitary matrix. Implementing braiding in a classical system can assist the experimental investigation of non-Abelian physics. However, the design of non-Abelian gauge fields faces numerous challenges stemmed from the intricate interplay of group structures, Lie algebra properties, representation theory, topology, and symmetry breaking. The extreme diversity makes it a powerful tool for the study of condensed matter physics. Whereas the widely used artificial intelligence with data-driven approaches has greatly promoted the development of physics, most works are limited on the data-to-data design. Here we propose a self-reasoning assistant learning framework capable of directly generating non-Abelian gauge fields. This framework utilizes the forward diffusion process to capture and reproduce the complex patterns and details inherent in the target distribution through continuous transformation. Then the reverse diffusion process is used to make the generated data closer to the distribution of the original situation. Thus, it owns strong self-reasoning capabilities, allowing to automatically discover the feature representation and capture more subtle relationships from the dataset. Moreover, the self-reasoning eliminates the need for manual feature engineering and simplifies the process of model building. Our framework offers a disruptive paradigm shift to parse complex physical processes, automatically uncovering patterns from massive datasets.

Title: DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priors

• Authors: Zizheng Yan, Jiapeng Zhou, Fanpeng Meng, Yushuang Wu, Lingteng Qiu, Zisheng Ye, Shuguang Cui, Guanying Chen, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16260
• Pdf URL: https://arxiv.org/pdf/2407.16260
• Copy Paste: [[2407.16260]] DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priors(https://arxiv.org/abs/2407.16260)
• Keywords: diffusion
• Abstract: Text-to-3D generation has recently seen significant progress. To enhance its practicality in real-world applications, it is crucial to generate multiple independent objects with interactions, similar to layer-compositing in 2D image editing. However, existing text-to-3D methods struggle with this task, as they are designed to generate either non-independent objects or independent objects lacking spatially plausible interactions. Addressing this, we propose DreamDissector, a text-to-3D method capable of generating multiple independent objects with interactions. DreamDissector accepts a multi-object text-to-3D NeRF as input and produces independent textured meshes. To achieve this, we introduce the Neural Category Field (NeCF) for disentangling the input NeRF. Additionally, we present the Category Score Distillation Sampling (CSDS), facilitated by a Deep Concept Mining (DCM) module, to tackle the concept gap issue in diffusion models. By leveraging NeCF and CSDS, we can effectively derive sub-NeRFs from the original scene. Further refinement enhances geometry and texture. Our experimental results validate the effectiveness of DreamDissector, providing users with novel means to control 3D synthesis at the object level and potentially opening avenues for various creative applications in the future.

Title: Federated Learning for Face Recognition via Intra-subject Self-supervised Learning

• Authors: Hansol Kim, Hoyeol Choi, Youngjun Kwak
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16289
• Pdf URL: https://arxiv.org/pdf/2407.16289
• Copy Paste: [[2407.16289]] Federated Learning for Face Recognition via Intra-subject Self-supervised Learning(https://arxiv.org/abs/2407.16289)
• Keywords: self-supervised
• Abstract: Federated Learning (FL) for face recognition aggregates locally optimized models from individual clients to construct a generalized face recognition model. However, previous studies present two major challenges: insufficient incorporation of self-supervised learning and the necessity for clients to accommodate multiple subjects. To tackle these limitations, we propose FedFS (Federated Learning for personalized Face recognition via intra-subject Self-supervised learning framework), a novel federated learning architecture tailored to train personalized face recognition models without imposing subjects. Our proposed FedFS comprises two crucial components that leverage aggregated features of the local and global models to cooperate with representations of an off-the-shelf model. These components are (1) adaptive soft label construction, utilizing dot product operations to reformat labels within intra-instances, and (2) intra-subject self-supervised learning, employing cosine similarity operations to strengthen robust intra-subject representations. Additionally, we introduce a regularization loss to prevent overfitting and ensure the stability of the optimized model. To assess the effectiveness of FedFS, we conduct comprehensive experiments on the DigiFace-1M and VGGFace datasets, demonstrating superior performance compared to previous methods.

Title: Harmonizing Visual Text Comprehension and Generation

• Authors: Zhen Zhao, Jingqun Tang, Binghong Wu, Chunhui Lin, Shu Wei, Hao Liu, Xin Tan, Zhizhong Zhang, Can Huang, Yuan Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16364
• Pdf URL: https://arxiv.org/pdf/2407.16364
• Copy Paste: [[2407.16364]] Harmonizing Visual Text Comprehension and Generation(https://arxiv.org/abs/2407.16364)
• Keywords: generative
• Abstract: In this work, we present TextHarmony, a unified and versatile multimodal generative model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities. To overcome this challenge, existing approaches resort to modality-specific data for supervised fine-tuning, necessitating distinct model instances. We propose Slide-LoRA, which dynamically aggregates modality-specific and modality-agnostic LoRA experts, partially decoupling the multimodal generation space. Slide-LoRA harmonizes the generation of vision and language within a singular model instance, thereby facilitating a more unified generative process. Additionally, we develop a high-quality image caption dataset, DetailedTextCaps-100K, synthesized with a sophisticated closed-source MLLM to enhance visual text generation capabilities further. Comprehensive experiments across various benchmarks demonstrate the effectiveness of the proposed approach. Empowered by Slide-LoRA, TextHarmony achieves comparable performance to modality-specific fine-tuning results with only a 2% increase in parameters and shows an average improvement of 2.5% in visual text comprehension tasks and 4.0% in visual text generation tasks. Our work delineates the viability of an integrated approach to multimodal generation within the visual text domain, setting a foundation for subsequent inquiries.

Title: Evolutionary Prompt Design for LLM-Based Post-ASR Error Correction

• Authors: Rithik Sachdev, Zhong-Qiu Wang, Chao-Han Huck Yang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.16370
• Pdf URL: https://arxiv.org/pdf/2407.16370
• Copy Paste: [[2407.16370]] Evolutionary Prompt Design for LLM-Based Post-ASR Error Correction(https://arxiv.org/abs/2407.16370)
• Keywords: generative, in-context
• Abstract: Building upon the strength of modern large language models (LLMs), generative error correction (GEC) has emerged as a promising paradigm that can elevate the performance of modern automatic speech recognition (ASR) systems. One representative approach is to leverage in-context learning to prompt LLMs so that a better hypothesis can be generated by the LLMs based on a carefully-designed prompt and an $N$-best list of hypotheses produced by ASR systems. However, it is yet unknown whether the existing prompts are the most effective ones for the task of post-ASR error correction. In this context, this paper first explores alternative prompts to identify an initial set of effective prompts, and then proposes to employ an evolutionary prompt optimization algorithm to refine the initial prompts. Evaluations results on the CHiME-4 subset of the Task $1$ of the SLT $2024$ GenSEC challenge show the effectiveness and potential of the proposed algorithms.

Title: TookaBERT: A Step Forward for Persian NLU

• Authors: MohammadAli SadraeiJavaheri, Ali Moghaddaszadeh, Milad Molazadeh, Fariba Naeiji, Farnaz Aghababaloo, Hamideh Rafiee, Zahra Amirmahani, Tohid Abedini, Fatemeh Zahra Sheikhi, Amirmohammad Salehoof
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16382
• Pdf URL: https://arxiv.org/pdf/2407.16382
• Copy Paste: [[2407.16382]] TookaBERT: A Step Forward for Persian NLU(https://arxiv.org/abs/2407.16382)
• Keywords: foundation model
• Abstract: The field of natural language processing (NLP) has seen remarkable advancements, thanks to the power of deep learning and foundation models. Language models, and specifically BERT, have been key players in this progress. In this study, we trained and introduced two new BERT models using Persian data. We put our models to the test, comparing them to seven existing models across 14 diverse Persian natural language understanding (NLU) tasks. The results speak for themselves: our larger model outperforms the competition, showing an average improvement of at least +2.8 points. This highlights the effectiveness and potential of our new BERT models for Persian NLU tasks.

Title: Securing Tomorrow's Smart Cities: Investigating Software Security in Internet of Vehicles and Deep Learning Technologies

• Authors: Ridhi Jain, Norbert Tihanyi, Mohamed Amine Ferrag
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.16410
• Pdf URL: https://arxiv.org/pdf/2407.16410
• Copy Paste: [[2407.16410]] Securing Tomorrow's Smart Cities: Investigating Software Security in Internet of Vehicles and Deep Learning Technologies(https://arxiv.org/abs/2407.16410)
• Keywords: anomaly
• Abstract: Integrating Deep Learning (DL) techniques in the Internet of Vehicles (IoV) introduces many security challenges and issues that require thorough examination. This literature review delves into the inherent vulnerabilities and risks associated with DL in IoV systems, shedding light on the multifaceted nature of security threats. Through an extensive analysis of existing research, we explore potential threats posed by DL algorithms, including adversarial attacks, data privacy breaches, and model poisoning. Additionally, we investigate the impact of DL on critical aspects of IoV security, such as intrusion detection, anomaly detection, and secure communication protocols. Our review emphasizes the complexities of ensuring the robustness, reliability, and trustworthiness of DL-based IoV systems, given the dynamic and interconnected nature of vehicular networks. Furthermore, we discuss the need for novel security solutions tailored to address these challenges effectively and enhance the security posture of DL-enabled IoV environments. By offering insights into these critical issues, this chapter aims to stimulate further research, innovation, and collaboration in securing DL techniques within the context of the IoV, thereby fostering a safer and more resilient future for vehicular communication and connectivity.

Title: MonoWAD: Weather-Adaptive Diffusion Model for Robust Monocular 3D Object Detection

• Authors: Youngmin Oh, Hyung-Il Kim, Seong Tae Kim, Jung Uk Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16448
• Pdf URL: https://arxiv.org/pdf/2407.16448
• Copy Paste: [[2407.16448]] MonoWAD: Weather-Adaptive Diffusion Model for Robust Monocular 3D Object Detection(https://arxiv.org/abs/2407.16448)
• Keywords: diffusion
• Abstract: Monocular 3D object detection is an important challenging task in autonomous driving. Existing methods mainly focus on performing 3D detection in ideal weather conditions, characterized by scenarios with clear and optimal visibility. However, the challenge of autonomous driving requires the ability to handle changes in weather conditions, such as foggy weather, not just clear weather. We introduce MonoWAD, a novel weather-robust monocular 3D object detector with a weather-adaptive diffusion model. It contains two components: (1) the weather codebook to memorize the knowledge of the clear weather and generate a weather-reference feature for any input, and (2) the weather-adaptive diffusion model to enhance the feature representation of the input feature by incorporating a weather-reference feature. This serves an attention role in indicating how much improvement is needed for the input feature according to the weather conditions. To achieve this goal, we introduce a weather-adaptive enhancement loss to enhance the feature representation under both clear and foggy weather conditions. Extensive experiments under various weather conditions demonstrate that MonoWAD achieves weather-robust monocular 3D object detection. The code and dataset are released at this https URL.

Title: qMRI Diffusor: Quantitative T1 Mapping of the Brain using a Denoising Diffusion Probabilistic Model

• Authors: Shishuai Wang, Hua Ma, Juan A. Hernandez-Tamames, Stefan Klein, Dirk H.J. Poot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16477
• Pdf URL: https://arxiv.org/pdf/2407.16477
• Copy Paste: [[2407.16477]] qMRI Diffusor: Quantitative T1 Mapping of the Brain using a Denoising Diffusion Probabilistic Model(https://arxiv.org/abs/2407.16477)
• Keywords: diffusion, generative
• Abstract: Quantitative MRI (qMRI) offers significant advantages over weighted images by providing objective parameters related to tissue properties. Deep learning-based methods have demonstrated effectiveness in estimating quantitative maps from series of weighted images. In this study, we present qMRI Diffusor, a novel approach to qMRI utilising deep generative models. Specifically, we implemented denoising diffusion probabilistic models (DDPM) for T1 quantification in the brain, framing the estimation of quantitative maps as a conditional generation task. The proposed method is compared with the residual neural network (ResNet) and the recurrent inference machine (RIM) on both phantom and in vivo data. The results indicate that our method achieves improved accuracy and precision in parameter estimation, along with superior visual performance. Moreover, our method inherently incorporates stochasticity, enabling straightforward quantification of uncertainty. Hence, the proposed method holds significant promise for quantitative MR mapping.

Title: ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation

• Authors: Zhenhua Wu, Yanlin Jin, Liangdong Qiu, Xiaoguang Han, Xiang Wan, Guanbin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16508
• Pdf URL: https://arxiv.org/pdf/2407.16508
• Copy Paste: [[2407.16508]] ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation(https://arxiv.org/abs/2407.16508)
• Keywords: self-supervised
• Abstract: Visualizing colonoscopy is crucial for medical auxiliary diagnosis to prevent undetected polyps in areas that are not fully observed. Traditional feature-based and depth-based reconstruction approaches usually end up with undesirable results due to incorrect point matching or imprecise depth estimation in realistic colonoscopy videos. Modern deep-based methods often require a sufficient number of ground truth samples, which are generally hard to obtain in optical colonoscopy. To address this issue, self-supervised and domain adaptation methods have been explored. However, these methods neglect geometry constraints and exhibit lower accuracy in predicting detailed depth. We thus propose a novel reconstruction pipeline with a bi-directional adaptation architecture named ToDER to get precise depth estimations. Furthermore, we carefully design a TNet module in our adaptation architecture to yield geometry constraints and obtain better depth quality. Estimated depth is finally utilized to reconstruct a reliable colon model for visualization. Experimental results demonstrate that our approach can precisely predict depth maps in both realistic and synthetic colonoscopy videos compared with other self-supervised and domain adaptation methods. Our method on realistic colonoscopy also shows the great potential for visualizing unobserved regions and preventing misdiagnoses.

Title: DreamVTON: Customizing 3D Virtual Try-on with Personalized Diffusion Models

• Authors: Zhenyu Xie, Haoye Dong, Yufei Gao, Zehua Ma, Xiaodan Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16511
• Pdf URL: https://arxiv.org/pdf/2407.16511
• Copy Paste: [[2407.16511]] DreamVTON: Customizing 3D Virtual Try-on with Personalized Diffusion Models(https://arxiv.org/abs/2407.16511)
• Keywords: diffusion, generative
• Abstract: Image-based 3D Virtual Try-ON (VTON) aims to sculpt the 3D human according to person and clothes images, which is data-efficient (i.e., getting rid of expensive 3D data) but challenging. Recent text-to-3D methods achieve remarkable improvement in high-fidelity 3D human generation, demonstrating its potential for 3D virtual try-on. Inspired by the impressive success of personalized diffusion models (e.g., Dreambooth and LoRA) for 2D VTON, it is straightforward to achieve 3D VTON by integrating the personalization technique into the diffusion-based text-to-3D framework. However, employing the personalized module in a pre-trained diffusion model (e.g., StableDiffusion (SD)) would degrade the model's capability for multi-view or multi-domain synthesis, which is detrimental to the geometry and texture optimization guided by Score Distillation Sampling (SDS) loss. In this work, we propose a novel customizing 3D human try-on model, named \textbf{DreamVTON}, to separately optimize the geometry and texture of the 3D human. Specifically, a personalized SD with multi-concept LoRA is proposed to provide the generative prior about the specific person and clothes, while a Densepose-guided ControlNet is exploited to guarantee consistent prior about body pose across various camera views. Besides, to avoid the inconsistent multi-view priors from the personalized SD dominating the optimization, DreamVTON introduces a template-based optimization mechanism, which employs mask templates for geometry shape learning and normal/RGB templates for geometry/texture details learning. Furthermore, for the geometry optimization phase, DreamVTON integrates a normal-style LoRA into personalized SD to enhance normal map generative prior, facilitating smooth geometry modeling.

Title: Assessing In-context Learning and Fine-tuning for Topic Classification of German Web Data

• Authors: Julian Schelb, Roberto Ulloa, Andreas Spitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16516
• Pdf URL: https://arxiv.org/pdf/2407.16516
• Copy Paste: [[2407.16516]] Assessing In-context Learning and Fine-tuning for Topic Classification of German Web Data(https://arxiv.org/abs/2407.16516)
• Keywords: in-context
• Abstract: Researchers in the political and social sciences often rely on classification models to analyze trends in information consumption by examining browsing histories of millions of webpages. Automated scalable methods are necessary due to the impracticality of manual labeling. In this paper, we model the detection of topic-related content as a binary classification task and compare the accuracy of fine-tuned pre-trained encoder models against in-context learning strategies. Using only a few hundred annotated data points per topic, we detect content related to three German policies in a database of scraped webpages. We compare multilingual and monolingual models, as well as zero and few-shot approaches, and investigate the impact of negative sampling strategies and the combination of URL & content-based features. Our results show that a small sample of annotated data is sufficient to train an effective classifier. Fine-tuning encoder-based models yields better results than in-context learning. Classifiers using both URL & content-based features perform best, while using URLs alone provides adequate results when content is unavailable.

Title: MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence

• Authors: Canyu Zhao, Mingyu Liu, Wen Wang, Jianlong Yuan, Hao Chen, Bo Zhang, Chunhua Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16655
• Pdf URL: https://arxiv.org/pdf/2407.16655
• Copy Paste: [[2407.16655]] MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence(https://arxiv.org/abs/2407.16655)
• Keywords: diffusion
• Abstract: Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities. Homepage: this https URL.

Title: SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation

• Authors: Pengfei Chen, Lingxi Xie, Xinyue Huo, Xuehui Yu, Xiaopeng Zhang, Yingfei Sun, Zhenjun Han, Qi Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16682
• Pdf URL: https://arxiv.org/pdf/2407.16682
• Copy Paste: [[2407.16682]] SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation(https://arxiv.org/abs/2407.16682)
• Keywords: foundation model
• Abstract: The Segment Anything model (SAM) has shown a generalized ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges. This paper presents SAM-CP, a simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation. Specifically, given a set of classes (in texts) and a set of SAM patches, the Type-I prompt judges whether a SAM patch aligns with a text label, and the Type-II prompt judges whether two SAM patches with the same text label also belong to the same instance. To decrease the complexity in dealing with a large number of semantic classes and patches, we establish a unified framework that calculates the affinity between (semantic and instance) queries and SAM patches and merges patches with high affinity to the query. Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains. In particular, it achieves state-of-the-art performance in open-vocabulary segmentation. Our research offers a novel and generalized methodology for equipping vision foundation models like SAM with multi-grained semantic perception abilities.

Title: Can Large Language Models Automatically Jailbreak GPT-4V?

• Authors: Yuanwei Wu, Yue Huang, Yixin Liu, Xiang Li, Pan Zhou, Lichao Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16686
• Pdf URL: https://arxiv.org/pdf/2407.16686
• Copy Paste: [[2407.16686]] Can Large Language Models Automatically Jailbreak GPT-4V?(https://arxiv.org/abs/2407.16686)
• Keywords: in-context
• Abstract: GPT-4V has attracted considerable attention due to its extraordinary capacity for integrating and processing multimodal information. At the same time, its ability of face recognition raises new safety concerns of privacy leakage. Despite researchers' efforts in safety alignment through RLHF or preprocessing filters, vulnerabilities might still be exploited. In our study, we introduce AutoJailbreak, an innovative automatic jailbreak technique inspired by prompt optimization. We leverage Large Language Models (LLMs) for red-teaming to refine the jailbreak prompt and employ weak-to-strong in-context learning prompts to boost efficiency. Furthermore, we present an effective search method that incorporates early stopping to minimize optimization time and token expenditure. Our experiments demonstrate that AutoJailbreak significantly surpasses conventional methods, achieving an Attack Success Rate (ASR) exceeding 95.3\%. This research sheds light on strengthening GPT-4V security, underscoring the potential for LLMs to be exploited in compromising GPT-4V integrity.

Title: Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack

• Authors: Xiaoyue Xu, Qinyuan Ye, Xiang Ren
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16695
• Pdf URL: https://arxiv.org/pdf/2407.16695
• Copy Paste: [[2407.16695]] Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack(https://arxiv.org/abs/2407.16695)
• Keywords: in-context
• Abstract: We introduce Lifelong ICL, a problem setting that challenges long-context language models (LMs) to learn from a sequence of language tasks through in-context learning (ICL). We further introduce Task Haystack, an evaluation suite dedicated to assessing and diagnosing how long-context LMs utilizes contexts in Lifelong ICL. When given a task instruction and test inputs, long-context LMs are expected to leverage the relevant demonstrations in the Lifelong ICL prompt, avoid distraction and interference from other tasks, and achieve test accuracies that are not significantly worse than the Single-task ICL baseline. Task Haystack draws inspiration from the widely-adopted "needle-in-a-haystack" (NIAH) evaluation, but presents new and unique challenges. It demands that models (1) utilize the contexts with deeper understanding, rather than resorting to simple copying and pasting; (2) navigate through long streams of evolving topics and tasks, which closely approximates the complexities of real-world usage of long-context LMs. Additionally, Task Haystack inherits the controllability aspect of NIAH, providing model developers with tools and visualizations to identify model vulnerabilities effectively. We benchmark 12 long-context LMs using Task Haystack. We find that state-of-the-art closed models such as GPT-4o still struggle in this setting, failing 15% of the cases on average, while all open-weight models we evaluate further lack behind by a large margin, failing up to 61% of the cases. In our controlled analysis, we identify factors such as distraction and recency bias as contributors to these failure cases. Further, we observe declines in performance when task instructions are paraphrased at test time or when ICL demonstrations are repeated excessively, raising concerns about the robustness, instruction understanding, and true context utilization of current long-context LMs.

Title: PartGLEE: A Foundation Model for Recognizing and Parsing Any Objects

• Authors: Junyi Li, Junfeng Wu, Weizhi Zhao, Song Bai, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16696
• Pdf URL: https://arxiv.org/pdf/2407.16696
• Copy Paste: [[2407.16696]] PartGLEE: A Foundation Model for Recognizing and Parsing Any Objects(https://arxiv.org/abs/2407.16696)
• Keywords: foundation model
• Abstract: We present PartGLEE, a part-level foundation model for locating and identifying both objects and parts in images. Through a unified framework, PartGLEE accomplishes detection, segmentation, and grounding of instances at any granularity in the open world scenario. Specifically, we propose a Q-Former to construct the hierarchical relationship between objects and parts, parsing every object into corresponding semantic parts. By incorporating a large amount of object-level data, the hierarchical relationships can be extended, enabling PartGLEE to recognize a rich variety of parts. We conduct comprehensive studies to validate the effectiveness of our method, PartGLEE achieves the state-of-the-art performance across various part-level tasks and obtain competitive results on object-level tasks. The proposed PartGLEE significantly enhances hierarchical modeling capabilities and part-level perception over our previous GLEE model. Further analysis indicates that the hierarchical cognitive ability of PartGLEE is able to facilitate a detailed comprehension in images for mLLMs. The model and code will be released at this https URL .

Title: Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions

• Authors: Fabio Tosi, Pierluigi Zama Ramirez, Matteo Poggi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16698
• Pdf URL: https://arxiv.org/pdf/2407.16698
• Copy Paste: [[2407.16698]] Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions(https://arxiv.org/abs/2407.16698)
• Keywords: diffusion
• Abstract: We present a novel approach designed to address the complexities posed by challenging, out-of-distribution data in the single-image depth estimation task. Starting with images that facilitate depth prediction due to the absence of unfavorable factors, we systematically generate new, user-defined scenes with a comprehensive set of challenges and associated depth information. This is achieved by leveraging cutting-edge text-to-image diffusion models with depth-aware control, known for synthesizing high-quality image content from textual prompts while preserving the coherence of 3D structure between generated and source imagery. Subsequent fine-tuning of any monocular depth network is carried out through a self-distillation protocol that takes into account images generated using our strategy and its own depth predictions on simple, unchallenging scenes. Experiments on benchmarks tailored for our purposes demonstrate the effectiveness and versatility of our proposal.