2024-05-06

Title: Mitigating LLM Hallucinations via Conformal Abstention

• Authors: Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, Nenad Tomasev
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2405.01563
• Pdf URL: https://arxiv.org/pdf/2405.01563
• Copy Paste: [[2405.01563]] Mitigating LLM Hallucinations via Conformal Abstention(https://arxiv.org/abs/2405.01563)
• Keywords: generative
• Abstract: We develop a principled procedure for determining when a large language model (LLM) should abstain from responding (e.g., by saying "I don't know") in a general domain, instead of resorting to possibly "hallucinating" a non-sensical or incorrect answer. Building on earlier approaches that use self-consistency as a more reliable measure of model confidence, we propose using the LLM itself to self-evaluate the similarity between each of its sampled responses for a given query. We then further leverage conformal prediction techniques to develop an abstention procedure that benefits from rigorous theoretical guarantees on the hallucination rate (error rate). Experimentally, our resulting conformal abstention method reliably bounds the hallucination rate on various closed-book, open-domain generative question answering datasets, while also maintaining a significantly less conservative abstention rate on a dataset with long responses (Temporal Sequences) compared to baselines using log-probability scores to quantify uncertainty, while achieveing comparable performance on a dataset with short answers (TriviaQA). To evaluate the experiments automatically, one needs to determine if two responses are equivalent given a question. Following standard practice, we use a thresholded similarity function to determine if two responses match, but also provide a method for calibrating the threshold based on conformal prediction, with theoretical guarantees on the accuracy of the match prediction, which might be of independent interest.

Title: GPT-4 passes most of the 297 written Polish Board Certification Examinations

• Authors: Jakub Pokrywka, Jeremi Kaczmarek, Edward Gorzelańczyk
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01589
• Pdf URL: https://arxiv.org/pdf/2405.01589
• Copy Paste: [[2405.01589]] GPT-4 passes most of the 297 written Polish Board Certification Examinations(https://arxiv.org/abs/2405.01589)
• Keywords: generative
• Abstract: Introduction: Recently, the effectiveness of Large Language Models (LLMs) has increased rapidly, allowing them to be used in a great number of applications. However, the risks posed by the generation of false information through LLMs significantly limit their applications in sensitive areas such as healthcare, highlighting the necessity for rigorous validations to determine their utility and reliability. To date, no study has extensively compared the performance of LLMs on Polish medical examinations across a broad spectrum of specialties on a very large dataset. Objectives: This study evaluated the performance of three Generative Pretrained Transformer (GPT) models on the Polish Board Certification Exam (Pa\'nstwowy Egzamin Specjalizacyjny, PES) dataset, which consists of 297 tests. Methods: We developed a software program to download and process PES exams and tested the performance of GPT models using OpenAI Application Programming Interface. Results: Our findings reveal that GPT-3.5 did not pass any of the analyzed exams. In contrast, the GPT-4 models demonstrated the capability to pass the majority of the exams evaluated, with the most recent model, gpt-4-0125, successfully passing 222 (75%) of them. The performance of the GPT models varied significantly, displaying excellence in exams related to certain specialties while completely failing others. Conclusions: The significant progress and impressive performance of LLM models hold great promise for the increased application of AI in the field of medicine in Poland. For instance, this advancement could lead to the development of AI-based medical assistants for healthcare professionals, enhancing the efficiency and accuracy of medical services.

Title: Simplifying Multimodality: Unimodal Approach to Multimodal Challenges in Radiology with General-Domain Large Language Model

• Authors: Seonhee Cho, Choonghan Kim, Jiho Lee, Chetan Chilkunda, Sujin Choi, Joo Heung Yoon
• Subjects: cs.CL, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2405.01591
• Pdf URL: https://arxiv.org/pdf/2405.01591
• Copy Paste: [[2405.01591]] Simplifying Multimodality: Unimodal Approach to Multimodal Challenges in Radiology with General-Domain Large Language Model(https://arxiv.org/abs/2405.01591)
• Keywords: in-context
• Abstract: Recent advancements in Large Multimodal Models (LMMs) have attracted interest in their generalization capability with only a few samples in the prompt. This progress is particularly relevant to the medical domain, where the quality and sensitivity of data pose unique challenges for model training and application. However, the dependency on high-quality data for effective in-context learning raises questions about the feasibility of these models when encountering with the inevitable variations and errors inherent in real-world medical data. In this paper, we introduce MID-M, a novel framework that leverages the in-context learning capabilities of a general-domain Large Language Model (LLM) to process multimodal data via image descriptions. MID-M achieves a comparable or superior performance to task-specific fine-tuned LMMs and other general-domain ones, without the extensive domain-specific training or pre-training on multimodal data, with significantly fewer parameters. This highlights the potential of leveraging general-domain LLMs for domain-specific tasks and offers a sustainable and cost-effective alternative to traditional LMM developments. Moreover, the robustness of MID-M against data quality issues demonstrates its practical utility in real-world medical domain applications.

Title: Efficient Sample-Specific Encoder Perturbations

• Authors: Yassir Fathullah, Mark J. F. Gales
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01601
• Pdf URL: https://arxiv.org/pdf/2405.01601
• Copy Paste: [[2405.01601]] Efficient Sample-Specific Encoder Perturbations(https://arxiv.org/abs/2405.01601)
• Keywords: foundation model
• Abstract: Encoder-decoder foundation models have displayed state-of-the-art performance on a range of autoregressive sequence tasks. This paper proposes a simple and lightweight modification to such systems to control the behaviour according to a specific attribute of interest. This paper proposes a novel inference-efficient approach to modifying the behaviour of an encoder-decoder system according to a specific attribute of interest. Specifically, we show that a small proxy network can be used to find a sample-by-sample perturbation of the encoder output of a frozen foundation model to trigger the decoder to generate improved decodings. This work explores a specific realization of this framework focused on improving the COMET performance of Flan-T5 on Machine Translation and the WER of Whisper foundation models on Speech Recognition. Results display consistent improvements in performance evaluated through COMET and WER respectively. Furthermore, experiments also show that the proxies are robust to the exact nature of the data used to train them and can extend to other domains.

Title: Unifying and extending Precision Recall metrics for assessing generative models

• Authors: Benjamin Sykes, Loic Simon, Julien Rabin
• Subjects: cs.LG, cs.AI, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2405.01611
• Pdf URL: https://arxiv.org/pdf/2405.01611
• Copy Paste: [[2405.01611]] Unifying and extending Precision Recall metrics for assessing generative models(https://arxiv.org/abs/2405.01611)
• Keywords: generative
• Abstract: With the recent success of generative models in image and text, the evaluation of generative models has gained a lot of attention. Whereas most generative models are compared in terms of scalar values such as Frechet Inception Distance (FID) or Inception Score (IS), in the last years (Sajjadi et al., 2018) proposed a definition of precision-recall curve to characterize the closeness of two distributions. Since then, various approaches to precision and recall have seen the light (Kynkaanniemi et al., 2019; Naeem et al., 2020; Park & Kim, 2023). They center their attention on the extreme values of precision and recall, but apart from this fact, their ties are elusive. In this paper, we unify most of these approaches under the same umbrella, relying on the work of (Simon et al., 2019). Doing so, we were able not only to recover entire curves, but also to expose the sources of the accounted pitfalls of the concerned metrics. We also provide consistency results that go well beyond the ones presented in the corresponding literature. Last, we study the different behaviors of the curves obtained experimentally.

Title: S4: Self-Supervised Sensing Across the Spectrum

• Authors: Jayanth Shenoy, Xinjian Davis Zhang, Shlok Mehrotra, Bill Tao, Rem Yang, Han Zhao, Deepak Vasisht
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01656
• Pdf URL: https://arxiv.org/pdf/2405.01656
• Copy Paste: [[2405.01656]] S4: Self-Supervised Sensing Across the Spectrum(https://arxiv.org/abs/2405.01656)
• Keywords: self-supervised
• Abstract: Satellite image time series (SITS) segmentation is crucial for many applications like environmental monitoring, land cover mapping and agricultural crop type classification. However, training models for SITS segmentation remains a challenging task due to the lack of abundant training data, which requires fine grained annotation. We propose S4 a new self-supervised pre-training approach that significantly reduces the requirement for labeled training data by utilizing two new insights: (a) Satellites capture images in different parts of the spectrum such as radio frequencies, and visible frequencies. (b) Satellite imagery is geo-registered allowing for fine-grained spatial alignment. We use these insights to formulate pre-training tasks in S4. We also curate m2s2-SITS, a large-scale dataset of unlabeled, spatially-aligned, multi-modal and geographic specific SITS that serves as representative pre-training data for S4. Finally, we evaluate S4 on multiple SITS segmentation datasets and demonstrate its efficacy against competing baselines while using limited labeled data.

Title: Generative AI in Cybersecurity

• Authors: Shivani Metta, Isaac Chang, Jack Parker, Michael P. Roman, Arturo F. Ehuan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.01674
• Pdf URL: https://arxiv.org/pdf/2405.01674
• Copy Paste: [[2405.01674]] Generative AI in Cybersecurity(https://arxiv.org/abs/2405.01674)
• Keywords: generative
• Abstract: The dawn of Generative Artificial Intelligence (GAI), characterized by advanced models such as Generative Pre-trained Transformers (GPT) and other Large Language Models (LLMs), has been pivotal in reshaping the field of data analysis, pattern recognition, and decision-making processes. This surge in GAI technology has ushered in not only innovative opportunities for data processing and automation but has also introduced significant cybersecurity challenges. As GAI rapidly progresses, it outstrips the current pace of cybersecurity protocols and regulatory frameworks, leading to a paradox wherein the same innovations meant to safeguard digital infrastructures also enhance the arsenal available to cyber criminals. These adversaries, adept at swiftly integrating and exploiting emerging technologies, may utilize GAI to develop malware that is both more covert and adaptable, thus complicating traditional cybersecurity efforts. The acceleration of GAI presents an ambiguous frontier for cybersecurity experts, offering potent tools for threat detection and response, while concurrently providing cyber attackers with the means to engineer more intricate and potent malware. Through the joint efforts of Duke Pratt School of Engineering, Coalfire, and Safebreach, this research undertakes a meticulous analysis of how malicious agents are exploiting GAI to augment their attack strategies, emphasizing a critical issue for the integrity of future cybersecurity initiatives. The study highlights the critical need for organizations to proactively identify and develop more complex defensive strategies to counter the sophisticated employment of GAI in malware creation.

Title: Leveraging Prompt-Learning for Structured Information Extraction from Crohn's Disease Radiology Reports in a Low-Resource Language

• Authors: Liam Hazan, Gili Focht, Naama Gavrielov, Roi Reichart, Talar Hagopian, Mary-Louise C. Greer, Ruth Cytter Kuint, Dan Turner, Moti Freiman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01682
• Pdf URL: https://arxiv.org/pdf/2405.01682
• Copy Paste: [[2405.01682]] Leveraging Prompt-Learning for Structured Information Extraction from Crohn's Disease Radiology Reports in a Low-Resource Language(https://arxiv.org/abs/2405.01682)
• Keywords: generative
• Abstract: Automatic conversion of free-text radiology reports into structured data using Natural Language Processing (NLP) techniques is crucial for analyzing diseases on a large scale. While effective for tasks in widely spoken languages like English, generative large language models (LLMs) typically underperform with less common languages and can pose potential risks to patient privacy. Fine-tuning local NLP models is hindered by the skewed nature of real-world medical datasets, where rare findings represent a significant data imbalance. We introduce SMP-BERT, a novel prompt learning method that leverages the structured nature of reports to overcome these challenges. In our studies involving a substantial collection of Crohn's disease radiology reports in Hebrew (over 8,000 patients and 10,000 reports), SMP-BERT greatly surpassed traditional fine-tuning methods in performance, notably in detecting infrequent conditions (AUC: 0.99 vs 0.94, F1: 0.84 vs 0.34). SMP-BERT empowers more accurate AI diagnostics available for low-resource languages.

Title: Adapting Self-Supervised Learning for Computational Pathology

• Authors: Eric Zimmermann, Neil Tenenholtz, James Hall, George Shaikovski, Michal Zelechowski, Adam Casson, Fausto Milletari, Julian Viret, Eugene Vorontsov, Siqi Liu, Kristen Severson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01688
• Pdf URL: https://arxiv.org/pdf/2405.01688
• Copy Paste: [[2405.01688]] Adapting Self-Supervised Learning for Computational Pathology(https://arxiv.org/abs/2405.01688)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) has emerged as a key technique for training networks that can generalize well to diverse tasks without task-specific supervision. This property makes SSL desirable for computational pathology, the study of digitized images of tissues, as there are many target applications and often limited labeled training samples. However, SSL algorithms and models have been primarily developed in the field of natural images and whether their performance can be improved by adaptation to particular domains remains an open question. In this work, we present an investigation of modifications to SSL for pathology data, specifically focusing on the DINOv2 algorithm. We propose alternative augmentations, regularization functions, and position encodings motivated by the characteristics of pathology images. We evaluate the impact of these changes on several benchmarks to demonstrate the value of tailored approaches.

Title: Language-Enhanced Latent Representations for Out-of-Distribution Detection in Autonomous Driving

• Authors: Zhenjiang Mao, Dong-You Jhong, Ao Wang, Ivan Ruchkin
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2405.01691
• Pdf URL: https://arxiv.org/pdf/2405.01691
• Copy Paste: [[2405.01691]] Language-Enhanced Latent Representations for Out-of-Distribution Detection in Autonomous Driving(https://arxiv.org/abs/2405.01691)
• Keywords: foundation model, anomaly
• Abstract: Out-of-distribution (OOD) detection is essential in autonomous driving, to determine when learning-based components encounter unexpected inputs. Traditional detectors typically use encoder models with fixed settings, thus lacking effective human interaction capabilities. With the rise of large foundation models, multimodal inputs offer the possibility of taking human language as a latent representation, thus enabling language-defined OOD detection. In this paper, we use the cosine similarity of image and text representations encoded by the multimodal model CLIP as a new representation to improve the transparency and controllability of latent encodings used for visual anomaly detection. We compare our approach with existing pre-trained encoders that can only produce latent representations that are meaningless from the user's standpoint. Our experiments on realistic driving data show that the language-based latent representation performs better than the traditional representation of the vision encoder and helps improve the detection performance when combined with standard representations.

Title: Long Tail Image Generation Through Feature Space Augmentation and Iterated Learning

• Authors: Rafael Elberg, Denis Parra, Mircea Petrache
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01705
• Pdf URL: https://arxiv.org/pdf/2405.01705
• Copy Paste: [[2405.01705]] Long Tail Image Generation Through Feature Space Augmentation and Iterated Learning(https://arxiv.org/abs/2405.01705)
• Keywords: diffusion
• Abstract: Image and multimodal machine learning tasks are very challenging to solve in the case of poorly distributed data. In particular, data availability and privacy restrictions exacerbate these hurdles in the medical domain. The state of the art in image generation quality is held by Latent Diffusion models, making them prime candidates for tackling this problem. However, a few key issues still need to be solved, such as the difficulty in generating data from under-represented classes and a slow inference process. To mitigate these issues, we propose a new method for image augmentation in long-tailed data based on leveraging the rich latent space of pre-trained Stable Diffusion Models. We create a modified separable latent space to mix head and tail class examples. We build this space via Iterated Learning of underlying sparsified embeddings, which we apply to task-specific saliency maps via a K-NN approach. Code is available at https://github.com/SugarFreeManatee/Feature-Space-Augmentation-and-Iterated-Learning

Title: A deep causal inference model for fully-interpretable travel behaviour analysis

• Authors: Kimia Kamal, Bilal Farooq
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01708
• Pdf URL: https://arxiv.org/pdf/2405.01708
• Copy Paste: [[2405.01708]] A deep causal inference model for fully-interpretable travel behaviour analysis(https://arxiv.org/abs/2405.01708)
• Keywords: generative
• Abstract: Transport policy assessment often involves causal questions, yet the causal inference capabilities of traditional travel behavioural models are at best limited. We present the deep CAusal infeRence mOdel for traveL behavIour aNAlysis (CAROLINA), a framework that explicitly models causality in travel behaviour, enhances predictive accuracy, and maintains interpretability by leveraging causal inference, deep learning, and traditional discrete choice modelling. Within this framework, we introduce a Generative Counterfactual model for forecasting human behaviour by adapting the Normalizing Flow method. Through the case studies of virtual reality-based pedestrian crossing behaviour, revealed preference travel behaviour from London, and synthetic data, we demonstrate the effectiveness of our proposed models in uncovering causal relationships, prediction accuracy, and assessing policy interventions. Our results show that intervention mechanisms that can reduce pedestrian stress levels lead to a 38.5% increase in individuals experiencing shorter waiting times. Reducing the travel distances in London results in a 47% increase in sustainable travel modes.

Title: Question Suggestion for Conversational Shopping Assistants Using Product Metadata

• Authors: Nikhita Vedula, Oleg Rokhlenko, Shervin Malmasi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.01738
• Pdf URL: https://arxiv.org/pdf/2405.01738
• Copy Paste: [[2405.01738]] Question Suggestion for Conversational Shopping Assistants Using Product Metadata(https://arxiv.org/abs/2405.01738)
• Keywords: generative, in-context
• Abstract: Digital assistants have become ubiquitous in e-commerce applications, following the recent advancements in Information Retrieval (IR), Natural Language Processing (NLP) and Generative Artificial Intelligence (AI). However, customers are often unsure or unaware of how to effectively converse with these assistants to meet their shopping needs. In this work, we emphasize the importance of providing customers a fast, easy to use, and natural way to interact with conversational shopping assistants. We propose a framework that employs Large Language Models (LLMs) to automatically generate contextual, useful, answerable, fluent and diverse questions about products, via in-context learning and supervised fine-tuning. Recommending these questions to customers as helpful suggestions or hints to both start and continue a conversation can result in a smoother and faster shopping experience with reduced conversation overhead and friction. We perform extensive offline evaluations, and discuss in detail about potential customer impact, and the type, length and latency of our generated product questions if incorporated into a real-world shopping assistant.

Title: The Psychosocial Impacts of Generative AI Harms

• Authors: Faye-Marie Vassel, Evan Shieh, Cassidy R. Sugimoto, Thema Monroe-White
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.01740
• Pdf URL: https://arxiv.org/pdf/2405.01740
• Copy Paste: [[2405.01740]] The Psychosocial Impacts of Generative AI Harms(https://arxiv.org/abs/2405.01740)
• Keywords: generative
• Abstract: The rapid emergence of generative Language Models (LMs) has led to growing concern about the impacts that their unexamined adoption may have on the social well-being of diverse user groups. Meanwhile, LMs are increasingly being adopted in K-20 schools and one-on-one student settings with minimal investigation of potential harms associated with their deployment. Motivated in part by real-world/everyday use cases (e.g., an AI writing assistant) this paper explores the potential psychosocial harms of stories generated by five leading LMs in response to open-ended prompting. We extend findings of stereotyping harms analyzing a total of 150K 100-word stories related to student classroom interactions. Examining patterns in LM-generated character demographics and representational harms (i.e., erasure, subordination, and stereotyping) we highlight particularly egregious vignettes, illustrating the ways LM-generated outputs may influence the experiences of users with marginalized and minoritized identities, and emphasizing the need for a critical understanding of the psychosocial impacts of generative AI tools when deployed and utilized in diverse social contexts.

Title: Efficient and Economic Large Language Model Inference with Attention Offloading

• Authors: Shaoyuan Chen, Yutong Lin, Mingxing Zhang, Yongwei Wu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2405.01814
• Pdf URL: https://arxiv.org/pdf/2405.01814
• Copy Paste: [[2405.01814]] Efficient and Economic Large Language Model Inference with Attention Offloading(https://arxiv.org/abs/2405.01814)
• Keywords: generative
• Abstract: Transformer-based large language models (LLMs) exhibit impressive performance in generative tasks but introduce significant challenges in real-world serving due to inefficient use of the expensive, computation-optimized accelerators. This mismatch arises from the autoregressive nature of LLMs, where the generation phase comprises operators with varying resource demands. Specifically, the attention operator is memory-intensive, exhibiting a memory access pattern that clashes with the strengths of modern accelerators, especially as context length increases. To enhance the efficiency and cost-effectiveness of LLM serving, we introduce the concept of attention offloading. This approach leverages a collection of cheap, memory-optimized devices for the attention operator while still utilizing high-end accelerators for other parts of the model. This heterogeneous setup ensures that each component is tailored to its specific workload, maximizing overall performance and cost efficiency. Our comprehensive analysis and experiments confirm the viability of splitting the attention computation over multiple devices. Also, the communication bandwidth required between heterogeneous devices proves to be manageable with prevalent networking technologies. To further validate our theory, we develop Lamina, an LLM inference system that incorporates attention offloading. Experimental results indicate that Lamina can provide 1.48x-12.1x higher estimated throughput per dollar than homogeneous solutions.

Title: A Novel Approach to Guard from Adversarial Attacks using Stable Diffusion

• Authors: Trinath Sai Subhash Reddy Pittala, Uma Maheswara Rao Meleti, Geethakrishna Puligundla
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.01838
• Pdf URL: https://arxiv.org/pdf/2405.01838
• Copy Paste: [[2405.01838]] A Novel Approach to Guard from Adversarial Attacks using Stable Diffusion(https://arxiv.org/abs/2405.01838)
• Keywords: diffusion
• Abstract: Recent developments in adversarial machine learning have highlighted the importance of building robust AI systems to protect against increasingly sophisticated attacks. While frameworks like AI Guardian are designed to defend against these threats, they often rely on assumptions that can limit their effectiveness. For example, they may assume attacks only come from one direction or include adversarial images in their training data. Our proposal suggests a different approach to the AI Guardian framework. Instead of including adversarial examples in the training process, we propose training the AI system without them. This aims to create a system that is inherently resilient to a wider range of attacks. Our method focuses on a dynamic defense strategy using stable diffusion that learns continuously and models threats comprehensively. We believe this approach can lead to a more generalized and robust defense against adversarial attacks. In this paper, we outline our proposed approach, including the theoretical basis, experimental design, and expected impact on improving AI security against adversarial threats.

Title: Defect Image Sample Generation With Diffusion Prior for Steel Surface Defect Recognition

• Authors: Yichun Tai, Kun Yang, Tao Peng, Zhenzhen Huang, Zhijiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.01872
• Pdf URL: https://arxiv.org/pdf/2405.01872
• Copy Paste: [[2405.01872]] Defect Image Sample Generation With Diffusion Prior for Steel Surface Defect Recognition(https://arxiv.org/abs/2405.01872)
• Keywords: diffusion, generative
• Abstract: The task of steel surface defect recognition is an industrial problem with great industry values. The data insufficiency is the major challenge in training a robust defect recognition network. Existing methods have investigated to enlarge the dataset by generating samples with generative models. However, their generation quality is still limited by the insufficiency of defect image samples. To this end, we propose Stable Surface Defect Generation (StableSDG), which transfers the vast generation distribution embedded in Stable Diffusion model for steel surface defect image generation. To tackle with the distinctive distribution gap between steel surface images and generated images of the diffusion model, we propose two processes. First, we align the distribution by adapting parameters of the diffusion model, adopted both in the token embedding space and network parameter space. Besides, in the generation process, we propose image-oriented generation rather than from pure Gaussian noises. We conduct extensive experiments on steel surface defect dataset, demonstrating state-of-the-art performance on generating high-quality samples and training recognition models, and both designed processes are significant for the performance.

Title: Exploring Combinatorial Problem Solving with Large Language Models: A Case Study on the Travelling Salesman Problem Using GPT-3.5 Turbo

• Authors: Mahmoud Masoud, Ahmed Abdelhay, Mohammed Elhenawy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.01997
• Pdf URL: https://arxiv.org/pdf/2405.01997
• Copy Paste: [[2405.01997]] Exploring Combinatorial Problem Solving with Large Language Models: A Case Study on the Travelling Salesman Problem Using GPT-3.5 Turbo(https://arxiv.org/abs/2405.01997)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) are deep learning models designed to generate text based on textual input. Although researchers have been developing these models for more complex tasks such as code generation and general reasoning, few efforts have explored how LLMs can be applied to combinatorial problems. In this research, we investigate the potential of LLMs to solve the Travelling Salesman Problem (TSP). Utilizing GPT-3.5 Turbo, we conducted experiments employing various approaches, including zero-shot in-context learning, few-shot in-context learning, and chain-of-thoughts (CoT). Consequently, we fine-tuned GPT-3.5 Turbo to solve a specific problem size and tested it using a set of various instance sizes. The fine-tuned models demonstrated promising performance on problems identical in size to the training instances and generalized well to larger problems. Furthermore, to improve the performance of the fine-tuned model without incurring additional training costs, we adopted a self-ensemble approach to improve the quality of the solutions.

Title: M${^2}$Depth: Self-supervised Two-Frame Multi-camera Metric Depth Estimation

• Authors: Yingshuang Zou, Yikang Ding, Xi Qiu, Haoqian Wang, Haotian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02004
• Pdf URL: https://arxiv.org/pdf/2405.02004
• Copy Paste: [[2405.02004]] M${^2}$Depth: Self-supervised Two-Frame Multi-camera Metric Depth Estimation(https://arxiv.org/abs/2405.02004)
• Keywords: self-supervised
• Abstract: This paper presents a novel self-supervised two-frame multi-camera metric depth estimation network, termed M${^2}$Depth, which is designed to predict reliable scale-aware surrounding depth in autonomous driving. Unlike the previous works that use multi-view images from a single time-step or multiple time-step images from a single camera, M${^2}$Depth takes temporally adjacent two-frame images from multiple cameras as inputs and produces high-quality surrounding depth. We first construct cost volumes in spatial and temporal domains individually and propose a spatial-temporal fusion module that integrates the spatial-temporal information to yield a strong volume presentation. We additionally combine the neural prior from SAM features with internal features to reduce the ambiguity between foreground and background and strengthen the depth edges. Extensive experimental results on nuScenes and DDAD benchmarks show M${^2}$Depth achieves state-of-the-art performance. More results can be found in https://heiheishuang.xyz/M2Depth .

Title: DiffMap: Enhancing Map Segmentation with Map Prior Using Diffusion Model

• Authors: Peijin Jia, Tuopu Wen, Ziang Luo, Mengmeng Yang, Kun Jiang, Zhiquan Lei, Xuewei Tang, Ziyuan Liu, Le Cui, Kehua Sheng, Bo Zhang, Diange Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02008
• Pdf URL: https://arxiv.org/pdf/2405.02008
• Copy Paste: [[2405.02008]] DiffMap: Enhancing Map Segmentation with Map Prior Using Diffusion Model(https://arxiv.org/abs/2405.02008)
• Keywords: diffusion
• Abstract: Constructing high-definition (HD) maps is a crucial requirement for enabling autonomous driving. In recent years, several map segmentation algorithms have been developed to address this need, leveraging advancements in Bird's-Eye View (BEV) perception. However, existing models still encounter challenges in producing realistic and consistent semantic map layouts. One prominent issue is the limited utilization of structured priors inherent in map segmentation masks. In light of this, we propose DiffMap, a novel approach specifically designed to model the structured priors of map segmentation masks using latent diffusion model. By incorporating this technique, the performance of existing semantic segmentation methods can be significantly enhanced and certain structural errors present in the segmentation outputs can be effectively rectified. Notably, the proposed module can be seamlessly integrated into any map segmentation model, thereby augmenting its capability to accurately delineate semantic information. Furthermore, through extensive visualization analysis, our model demonstrates superior proficiency in generating results that more accurately reflect real-world map layouts, further validating its efficacy in improving the quality of the generated maps.

Title: Advancing Pre-trained Teacher: Towards Robust Feature Discrepancy for Anomaly Detection

• Authors: Canhui Tang, Sanping Zhou, Yizhe Li, Yonghao Dong, Le Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02068
• Pdf URL: https://arxiv.org/pdf/2405.02068
• Copy Paste: [[2405.02068]] Advancing Pre-trained Teacher: Towards Robust Feature Discrepancy for Anomaly Detection(https://arxiv.org/abs/2405.02068)
• Keywords: anomaly
• Abstract: With the wide application of knowledge distillation between an ImageNet pre-trained teacher model and a learnable student model, industrial anomaly detection has witnessed a significant achievement in the past few years. The success of knowledge distillation mainly relies on how to keep the feature discrepancy between the teacher and student model, in which it assumes that: (1) the teacher model can jointly represent two different distributions for the normal and abnormal patterns, while (2) the student model can only reconstruct the normal distribution. However, it still remains a challenging issue to maintain these ideal assumptions in practice. In this paper, we propose a simple yet effective two-stage industrial anomaly detection framework, termed as AAND, which sequentially performs Anomaly Amplification and Normality Distillation to obtain robust feature discrepancy. In the first anomaly amplification stage, we propose a novel Residual Anomaly Amplification (RAA) module to advance the pre-trained teacher encoder. With the exposure of synthetic anomalies, it amplifies anomalies via residual generation while maintaining the integrity of pre-trained model. It mainly comprises a Matching-guided Residual Gate and an Attribute-scaling Residual Generator, which can determine the residuals' proportion and characteristic, respectively. In the second normality distillation stage, we further employ a reverse distillation paradigm to train a student decoder, in which a novel Hard Knowledge Distillation (HKD) loss is built to better facilitate the reconstruction of normal patterns. Comprehensive experiments on the MvTecAD, VisA, and MvTec3D-RGB datasets show that our method achieves state-of-the-art performance.

Title: Probablistic Restoration with Adaptive Noise Sampling for 3D Human Pose Estimation

• Authors: Xianzhou Zeng, Hao Qin, Ming Kong, Luyuan Chen, Qiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02114
• Pdf URL: https://arxiv.org/pdf/2405.02114
• Copy Paste: [[2405.02114]] Probablistic Restoration with Adaptive Noise Sampling for 3D Human Pose Estimation(https://arxiv.org/abs/2405.02114)
• Keywords: generative
• Abstract: The accuracy and robustness of 3D human pose estimation (HPE) are limited by 2D pose detection errors and 2D to 3D ill-posed challenges, which have drawn great attention to Multi-Hypothesis HPE research. Most existing MH-HPE methods are based on generative models, which are computationally expensive and difficult to train. In this study, we propose a Probabilistic Restoration 3D Human Pose Estimation framework (PRPose) that can be integrated with any lightweight single-hypothesis model. Specifically, PRPose employs a weakly supervised approach to fit the hidden probability distribution of the 2D-to-3D lifting process in the Single-Hypothesis HPE model and then reverse-map the distribution to the 2D pose input through an adaptive noise sampling strategy to generate reasonable multi-hypothesis samples effectively. Extensive experiments on 3D HPE benchmarks (Human3.6M and MPI-INF-3DHP) highlight the effectiveness and efficiency of PRPose. Code is available at: https://github.com/xzhouzeng/PRPose.

Title: Mapping the Unseen: Unified Promptable Panoptic Mapping with Dynamic Labeling using Foundation Models

• Authors: Mohamad Al Mdfaa, Raghad Salameh, Sergey Zagoruyko, Gonzalo Ferrer
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2405.02162
• Pdf URL: https://arxiv.org/pdf/2405.02162
• Copy Paste: [[2405.02162]] Mapping the Unseen: Unified Promptable Panoptic Mapping with Dynamic Labeling using Foundation Models(https://arxiv.org/abs/2405.02162)
• Keywords: foundation model
• Abstract: In the field of robotics and computer vision, efficient and accurate semantic mapping remains a significant challenge due to the growing demand for intelligent machines that can comprehend and interact with complex environments. Conventional panoptic mapping methods, however, are limited by predefined semantic classes, thus making them ineffective for handling novel or unforeseen objects. In response to this limitation, we introduce the Unified Promptable Panoptic Mapping (UPPM) method. UPPM utilizes recent advances in foundation models to enable real-time, on-demand label generation using natural language prompts. By incorporating a dynamic labeling strategy into traditional panoptic mapping techniques, UPPM provides significant improvements in adaptability and versatility while maintaining high performance levels in map reconstruction. We demonstrate our approach on real-world and simulated datasets. Results show that UPPM can accurately reconstruct scenes and segment objects while generating rich semantic labels through natural language interactions. A series of ablation experiments validated the advantages of foundation model-based labeling over fixed label sets.

Title: Self-Supervised Learning for Real-World Super-Resolution from Dual and Multiple Zoomed Observations

• Authors: Zhilu Zhang, Ruohao Wang, Hongzhi Zhang, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02171
• Pdf URL: https://arxiv.org/pdf/2405.02171
• Copy Paste: [[2405.02171]] Self-Supervised Learning for Real-World Super-Resolution from Dual and Multiple Zoomed Observations(https://arxiv.org/abs/2405.02171)
• Keywords: self-supervised
• Abstract: In this paper, we consider two challenging issues in reference-based super-resolution (RefSR) for smartphone, (i) how to choose a proper reference image, and (ii) how to learn RefSR in a self-supervised manner. Particularly, we propose a novel self-supervised learning approach for real-world RefSR from observations at dual and multiple camera zooms. Firstly, considering the popularity of multiple cameras in modern smartphones, the more zoomed (telephoto) image can be naturally leveraged as the reference to guide the super-resolution (SR) of the lesser zoomed (ultra-wide) image, which gives us a chance to learn a deep network that performs SR from the dual zoomed observations (DZSR). Secondly, for self-supervised learning of DZSR, we take the telephoto image instead of an additional high-resolution image as the supervision information, and select a center patch from it as the reference to super-resolve the corresponding ultra-wide image patch. To mitigate the effect of the misalignment between ultra-wide low-resolution (LR) patch and telephoto ground-truth (GT) image during training, we first adopt patch-based optical flow alignment and then design an auxiliary-LR to guide the deforming of the warped LR features. To generate visually pleasing results, we present local overlapped sliced Wasserstein loss to better represent the perceptual difference between GT and output in the feature space. During testing, DZSR can be directly deployed to super-solve the whole ultra-wide image with the reference of the telephoto image. In addition, we further take multiple zoomed observations to explore self-supervised RefSR, and present a progressive fusion scheme for the effective utilization of reference images. Experiments show that our methods achieve better quantitative and qualitative performance against state-of-the-arts. Codes are available at https://github.com/cszhilu1998/SelfDZSR_PlusPlus.

Title: A Flow-Based Model for Conditional and Probabilistic Electricity Consumption Profile Generation and Prediction

• Authors: Weijie Xia, Chenguang Wang, Peter Palensky, Pedro P. Vergara
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2405.02180
• Pdf URL: https://arxiv.org/pdf/2405.02180
• Copy Paste: [[2405.02180]] A Flow-Based Model for Conditional and Probabilistic Electricity Consumption Profile Generation and Prediction(https://arxiv.org/abs/2405.02180)
• Keywords: generative
• Abstract: Residential Load Profile (RLP) generation and prediction are critical for the operation and planning of distribution networks, particularly as diverse low-carbon technologies are increasingly integrated. This paper introduces a novel flow-based generative model, termed Full Convolutional Profile Flow (FCPFlow), which is uniquely designed for both conditional and unconditional RLP generation, and for probabilistic load forecasting. By introducing two new layers--the invertible linear layer and the invertible normalization layer--the proposed FCPFlow architecture shows three main advantages compared to traditional statistical and contemporary deep generative models: 1) it is well-suited for RLP generation under continuous conditions, such as varying weather and annual electricity consumption, 2) it shows superior scalability in different datasets compared to traditional statistical, and 3) it also demonstrates better modeling capabilities in capturing the complex correlation of RLPs compared with deep generative models.

Title: Subgraph2vec: A random walk-based algorithm for embedding knowledge graphs

• Authors: Elika Bozorgi, Saber Soleimani, Sakher Khalil Alqaiidi, Hamid Reza Arabnia, Krzysztof Kochut
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.02240
• Pdf URL: https://arxiv.org/pdf/2405.02240
• Copy Paste: [[2405.02240]] Subgraph2vec: A random walk-based algorithm for embedding knowledge graphs(https://arxiv.org/abs/2405.02240)
• Keywords: anomaly
• Abstract: Graph is an important data representation which occurs naturally in the real world applications \cite{goyal2018graph}. Therefore, analyzing graphs provides users with better insights in different areas such as anomaly detection \cite{ma2021comprehensive}, decision making \cite{fan2023graph}, clustering \cite{tsitsulin2023graph}, classification \cite{wang2021mixup} and etc. However, most of these methods require high levels of computational time and space. We can use other ways like embedding to reduce these costs. Knowledge graph (KG) embedding is a technique that aims to achieve the vector representation of a KG. It represents entities and relations of a KG in a low-dimensional space while maintaining the semantic meanings of them. There are different methods for embedding graphs including random walk-based methods such as node2vec, metapath2vec and regpattern2vec. However, most of these methods bias the walks based on a rigid pattern usually hard-coded in the algorithm. In this work, we introduce \textit{subgraph2vec} for embedding KGs where walks are run inside a user-defined subgraph. We use this embedding for link prediction and prove our method has better performance in most cases in comparison with the previous ones.

Title: DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos

• Authors: Wen-Hsuan Chu, Lei Ke, Katerina Fragkiadaki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02280
• Pdf URL: https://arxiv.org/pdf/2405.02280
• Copy Paste: [[2405.02280]] DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos(https://arxiv.org/abs/2405.02280)
• Keywords: diffusion, generative
• Abstract: Existing VLMs can track in-the-wild 2D video objects while current generative models provide powerful visual priors for synthesizing novel views for the highly under-constrained 2D-to-3D object lifting. Building upon this exciting progress, we present DreamScene4D, the first approach that can generate three-dimensional dynamic scenes of multiple objects from monocular in-the-wild videos with large object motion across occlusions and novel viewpoints. Our key insight is to design a "decompose-then-recompose" scheme to factorize both the whole video scene and each object's 3D motion. We first decompose the video scene by using open-vocabulary mask trackers and an adapted image diffusion model to segment, track, and amodally complete the objects and background in the video. Each object track is mapped to a set of 3D Gaussians that deform and move in space and time. We also factorize the observed motion into multiple components to handle fast motion. The camera motion can be inferred by re-rendering the background to match the video frames. For the object motion, we first model the object-centric deformation of the objects by leveraging rendering losses and multi-view generative priors in an object-centric frame, then optimize object-centric to world-frame transformations by comparing the rendered outputs against the perceived pixel and optical flow. Finally, we recompose the background and objects and optimize for relative object scales using monocular depth prediction guidance. We show extensive results on the challenging DAVIS, Kubric, and self-captured videos, detail some limitations, and provide future directions. Besides 4D scene generation, our results show that DreamScene4D enables accurate 2D point motion tracking by projecting the inferred 3D trajectories to 2D, while never explicitly trained to do so.