2024-04-11

Title: Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation

• Authors: Luca Barsellotti, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06542
• Pdf URL: https://arxiv.org/pdf/2404.06542
• Copy Paste: [[2404.06542]] Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation(https://arxiv.org/abs/2404.06542)
• Keywords: diffusion, segmentation
• Abstract: Open-vocabulary semantic segmentation aims at segmenting arbitrary categories expressed in textual form. Previous works have trained over large amounts of image-caption pairs to enforce pixel-level multimodal alignments. However, captions provide global information about the semantics of a given image but lack direct localization of individual concepts. Further, training on large-scale datasets inevitably brings significant computational costs. In this paper, we propose FreeDA, a training-free diffusion-augmented method for open-vocabulary semantic segmentation, which leverages the ability of diffusion models to visually localize generated concepts and local-global similarities to match class-agnostic regions with semantic classes. Our approach involves an offline stage in which textual-visual reference embeddings are collected, starting from a large set of captions and leveraging visual and semantic contexts. At test time, these are queried to support the visual matching process, which is carried out by jointly considering class-agnostic regions and global semantic similarities. Extensive analyses demonstrate that FreeDA achieves state-of-the-art performance on five datasets, surpassing previous methods by more than 7.0 average points in terms of mIoU and without requiring any training.

Title: The Impact of Print-and-Scan in Heterogeneous Morph Evaluation Scenarios

• Authors: Richard E. Neddo, Zander W. Blasingame, Chen Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06559
• Pdf URL: https://arxiv.org/pdf/2404.06559
• Copy Paste: [[2404.06559]] The Impact of Print-and-Scan in Heterogeneous Morph Evaluation Scenarios(https://arxiv.org/abs/2404.06559)
• Keywords: attack
• Abstract: Face morphing attacks present an emerging threat to the face recognition system. On top of that, printing and scanning the morphed images could obscure the artifacts generated during the morphing process, which makes morphed image detection even harder. In this work, we investigate the impact that printing and scanning has on morphing attacks through a series of heterogeneous tests. Our experiments show that we can increase the possibility of a false match by up to 5.64% for DiM and 16.00% for StyleGAN2 when providing an image that has been printed and scanned, regardless it is morphed or bona fide, to a Face Recognition (FR) system. Likewise, using Frechet Inception Distance (FID) metric, strictly print-scanned morph attacks performed on average 9.185% stronger than non-print-scanned digital morphs.

Title: MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection

• Authors: Haoyang He, Yuhu Bai, Jiangning Zhang, Qingdong He, Hongxu Chen, Zhenye Gan, Chengjie Wang, Xiangtai Li, Guanzhong Tian, Lei Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06564
• Pdf URL: https://arxiv.org/pdf/2404.06564
• Copy Paste: [[2404.06564]] MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection(https://arxiv.org/abs/2404.06564)
• Keywords: transformer
• Abstract: Recent advancements in anomaly detection have seen the efficacy of CNN- and transformer-based approaches. However, CNNs struggle with long-range dependencies, while transformers are burdened by quadratic computational complexity. Mamba-based models, with their superior long-range modeling and linear efficiency, have garnered substantial attention. This study pioneers the application of Mamba to multi-class unsupervised anomaly detection, presenting MambaAD, which consists of a pre-trained encoder and a Mamba decoder featuring Locality-Enhanced State Space (LSS) modules at multi-scales. The proposed LSS module, integrating parallel cascaded (Hybrid State Space) HSS blocks and multi-kernel convolutions operations, effectively captures both long-range and local information. The HSS block, utilizing (Hybrid Scanning) HS encoders, encodes feature maps into five scanning methods and eight directions, thereby strengthening global connections through the (State Space Model) SSM. The use of Hilbert scanning and eight directions significantly improves feature sequence modeling. Comprehensive experiments on six diverse anomaly detection datasets and seven metrics demonstrate SoTA performance, substantiating the method's effectiveness.

Title: Less is More for Improving Automatic Evaluation of Factual Consistency

• Authors: Tong Wang, Ninad Kulkarni, Yanjun Qi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06579
• Pdf URL: https://arxiv.org/pdf/2404.06579
• Copy Paste: [[2404.06579]] Less is More for Improving Automatic Evaluation of Factual Consistency(https://arxiv.org/abs/2404.06579)
• Keywords: robust, large language model
• Abstract: Assessing the factual consistency of automatically generated texts in relation to source context is crucial for developing reliable natural language generation applications. Recent literature proposes AlignScore which uses a unified alignment model to evaluate factual consistency and substantially outperforms previous methods across many benchmark tasks. In this paper, we take a closer look of datasets used in AlignScore and uncover an unexpected finding: utilizing a smaller number of data points can actually improve performance. We process the original AlignScore training dataset to remove noise, augment with robustness-enhanced samples, and utilize a subset comprising 10\% of the data to train an improved factual consistency evaluation model, we call LIM-RA (Less Is More for Robust AlignScore). LIM-RA demonstrates superior performance, consistently outperforming AlignScore and other strong baselines like ChatGPT across four benchmarks (two utilizing traditional natural language generation datasets and two focused on large language model outputs). Our experiments show that LIM-RA achieves the highest score on 24 of the 33 test datasets, while staying competitive on the rest, establishing the new state-of-the-art benchmarks.

Title: FMDA-OT: Federated Multi-source Domain Adaptation Through Optimal Transport

• Authors: Omar Ghannou, Younès Bennani
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06599
• Pdf URL: https://arxiv.org/pdf/2404.06599
• Copy Paste: [[2404.06599]] FMDA-OT: Federated Multi-source Domain Adaptation Through Optimal Transport(https://arxiv.org/abs/2404.06599)
• Keywords: privacy, federate
• Abstract: Multi-source Domain Adaptation (MDA) aims to adapt models trained on multiple labeled source domains to an unlabeled target domain. In this paper, we introduce our approach as a collaborative MDA framework, which comprises two adaptation phases. Firstly, we conduct domain adaptation for each source individually with the target, utilizing optimal transport. Then, in the second phase, which constitutes the final part of the framework, we design the architecture of centralized federated learning to collaborate the N models representing the N sources. This architecture offers the advantage of using the sources without accessing their data, thus resolving data privacy issues inherent in domain adaptation. Additionally, during this phase, the server guides and fine-tunes the adaptation using a small number of pseudo-labeled samples available in the target domain, referred to as the target validation subset of the dataset.

Title: FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations

• Authors: Jane Dwivedi-Yu, Raaz Dwivedi, Timo Schick
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06619
• Pdf URL: https://arxiv.org/pdf/2404.06619
• Copy Paste: [[2404.06619]] FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations(https://arxiv.org/abs/2404.06619)
• Keywords: robust, fair, generative
• Abstract: The accurate evaluation of differential treatment in language models to specific groups is critical to ensuring a positive and safe user experience. An ideal evaluation should have the properties of being robust, extendable to new groups or attributes, and being able to capture biases that appear in typical usage (rather than just extreme, rare cases). Relatedly, bias evaluation should surface not only egregious biases but also ones that are subtle and commonplace, such as a likelihood for talking about appearances with regard to women. We present FairPair, an evaluation framework for assessing differential treatment that occurs during ordinary usage. FairPair operates through counterfactual pairs, but crucially, the paired continuations are grounded in the same demographic group, which ensures equivalent comparison. Additionally, unlike prior work, our method factors in the inherent variability that comes from the generation process itself by measuring the sampling variability. We present an evaluation of several commonly used generative models and a qualitative analysis that indicates a preference for discussing family and hobbies with regard to women.

Title: What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models

• Authors: Jeongrok Yu, Seong Ug Kim, Jacob Choi, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06621
• Pdf URL: https://arxiv.org/pdf/2404.06621
• Copy Paste: [[2404.06621]] What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models(https://arxiv.org/abs/2404.06621)
• Keywords: robust, transformer
• Abstract: Bias is a disproportionate prejudice in favor of one side against another. Due to the success of transformer-based Masked Language Models (MLMs) and their impact on many NLP tasks, a systematic evaluation of bias in these models is needed more than ever. While many studies have evaluated gender bias in English MLMs, only a few works have been conducted for the task in other languages. This paper proposes a multilingual approach to estimate gender bias in MLMs from 5 languages: Chinese, English, German, Portuguese, and Spanish. Unlike previous work, our approach does not depend on parallel corpora coupled with English to detect gender bias in other languages using multilingual lexicons. Moreover, a novel model-based method is presented to generate sentence pairs for a more robust analysis of gender bias, compared to the traditional lexicon-based method. For each language, both the lexicon-based and model-based methods are applied to create two datasets respectively, which are used to evaluate gender bias in an MLM specifically trained for that language using one existing and 3 new scoring metrics. Our results show that the previous approach is data-sensitive and not stable as it does not remove contextual dependencies irrelevant to gender. In fact, the results often flip when different scoring metrics are used on the same dataset, suggesting that gender bias should be studied on a large dataset using multiple evaluation metrics for best practice.

Title: Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformers

• Authors: Dipam Goswami, Bartłomiej Twardowski, Joost van de Weijer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06622
• Pdf URL: https://arxiv.org/pdf/2404.06622
• Copy Paste: [[2404.06622]] Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformers(https://arxiv.org/abs/2404.06622)
• Keywords: transformer
• Abstract: Few-shot class-incremental learning (FSCIL) aims to adapt the model to new classes from very few data (5 samples) without forgetting the previously learned classes. Recent works in many-shot CIL (MSCIL) (using all available training data) exploited pre-trained models to reduce forgetting and achieve better plasticity. In a similar fashion, we use ViT models pre-trained on large-scale datasets for few-shot settings, which face the critical issue of low plasticity. FSCIL methods start with a many-shot first task to learn a very good feature extractor and then move to the few-shot setting from the second task onwards. While the focus of most recent studies is on how to learn the many-shot first task so that the model generalizes to all future few-shot tasks, we explore in this work how to better model the few-shot data using pre-trained models, irrespective of how the first task is trained. Inspired by recent works in MSCIL, we explore how using higher-order feature statistics can influence the classification of few-shot classes. We identify the main challenge of obtaining a good covariance matrix from few-shot data and propose to calibrate the covariance matrix for new classes based on semantic similarity to the many-shot base classes. Using the calibrated feature statistics in combination with existing methods significantly improves few-shot continual classification on several FSCIL benchmarks. Code is available at https://github.com/dipamgoswami/FSCIL-Calibration.

Title: Current Affairs: A Measurement Study of Deployment and Security Trends in EV Charging Infrastructure

• Authors: Marcell Szakály, Sebastian Köhler, Ivan Martinovic
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06635
• Pdf URL: https://arxiv.org/pdf/2404.06635
• Copy Paste: [[2404.06635]] Current Affairs: A Measurement Study of Deployment and Security Trends in EV Charging Infrastructure(https://arxiv.org/abs/2404.06635)
• Keywords: security, attack, fair
• Abstract: The deployment of electric vehicle charging infrastructure is occurring at a rapid pace. Simultaneously, existing standards, such as ISO 15118, which defines critical charging communication, are being improved and further developed. In this paper, we conduct a measurement study of already deployed DC charging stations to analyze the current state of deployment for various protocols. We present the adoption of TLS, and various EV charging protocols with a direct security impact, as well as observations about the Signal Level Attenuation Characterization (SLAC) process, and encryption keys. Our results indicate that even recently installed charging stations (December 2023) do not adhere to the latest version of the standard, leaving them vulnerable to attacks. We found that 84% of the surveyed charging stations do not implement Transport Layer Security (TLS), and are thus unable to implement the latest versions of the ISO 15118 protocol, leaving them vulnerable to attacks already demonstrated years ago. Finally, we observe and document anomalous behavior and violations of the standard.

Title: Federated learning model for predicting major postoperative complications

• Authors: Yonggi Park, Yuanfang Ren, Benjamin Shickel, Ziyuan Guan, Ayush Patela, Yingbo Ma, Zhenhong Hu, Tyler J. Loftus, Parisa Rashidi, Tezcan Ozrazgat-Baslanti, Azra Bihorac
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.06641
• Pdf URL: https://arxiv.org/pdf/2404.06641
• Copy Paste: [[2404.06641]] Federated learning model for predicting major postoperative complications(https://arxiv.org/abs/2404.06641)
• Keywords: privacy, protect, robust, federate
• Abstract: Background: The accurate prediction of postoperative complication risk using Electronic Health Records (EHR) and artificial intelligence shows great potential. Training a robust artificial intelligence model typically requires large-scale and diverse datasets. In reality, collecting medical data often encounters challenges surrounding privacy protection. Methods: This retrospective cohort study includes adult patients who were admitted to UFH Gainesville (GNV) (n = 79,850) and Jacksonville (JAX) (n = 28,636) for any type of inpatient surgical procedure. Using perioperative and intraoperative features, we developed federated learning models to predict nine major postoperative complications (i.e., prolonged intensive care unit stay and mechanical ventilation). We compared federated learning models with local learning models trained on a single site and central learning models trained on pooled dataset from two centers. Results: Our federated learning models achieved the area under the receiver operating characteristics curve (AUROC) values ranged from 0.81 for wound complications to 0.92 for prolonged ICU stay at UFH GNV center. At UFH JAX center, these values ranged from 0.73-0.74 for wound complications to 0.92-0.93 for hospital mortality. Federated learning models achieved comparable AUROC performance to central learning models, except for prolonged ICU stay, where the performance of federated learning models was slightly higher than central learning models at UFH GNV center, but slightly lower at UFH JAX center. In addition, our federated learning model obtained comparable performance to the best local learning model at each center, demonstrating strong generalizability. Conclusion: Federated learning is shown to be a useful tool to train robust and generalizable models from large scale data across multiple institutions where data protection barriers are high.

Title: Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?

• Authors: Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, Mohammad Hossein Rohban
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06644
• Pdf URL: https://arxiv.org/pdf/2404.06644
• Copy Paste: [[2404.06644]] Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?(https://arxiv.org/abs/2404.06644)
• Keywords: generative, large language model
• Abstract: Evaluating Large Language Models (LLMs) is challenging due to their generative nature, necessitating precise evaluation methodologies. Additionally, non-English LLM evaluation lags behind English, resulting in the absence or weakness of LLMs for many languages. In response to this necessity, we introduce Khayyam Challenge (also known as PersianMMLU), a meticulously curated collection comprising 20,192 four-choice questions sourced from 38 diverse tasks extracted from Persian examinations, spanning a wide spectrum of subjects, complexities, and ages. The primary objective of the Khayyam Challenge is to facilitate the rigorous evaluation of LLMs that support the Persian language. Distinctive features of the Khayyam Challenge are (i) its comprehensive coverage of various topics, including literary comprehension, mathematics, sciences, logic, intelligence testing, etc., aimed at assessing different facets of LLMs such as language comprehension, reasoning, and information retrieval across various educational stages, from lower primary school to upper secondary school (ii) its inclusion of rich metadata such as human response rates, difficulty levels, and descriptive answers (iii) its utilization of new data to avoid data contamination issues prevalent in existing frameworks (iv) its use of original, non-translated data tailored for Persian speakers, ensuring the framework is free from translation challenges and errors while encompassing cultural nuances (v) its inherent scalability for future data updates and evaluations without requiring special human effort. Previous works lacked an evaluation framework that combined all of these features into a single comprehensive benchmark. Furthermore, we evaluate a wide range of existing LLMs that support the Persian language, with statistical analyses and interpretations of their outputs.

Title: FlameFinder: Illuminating Obscured Fire through Smoke with Attentive Deep Metric Learning

• Authors: Hossein Rajoli, Sahand Khoshdel, Fatemeh Afghah, Xiaolong Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06653
• Pdf URL: https://arxiv.org/pdf/2404.06653
• Copy Paste: [[2404.06653]] FlameFinder: Illuminating Obscured Fire through Smoke with Attentive Deep Metric Learning(https://arxiv.org/abs/2404.06653)
• Keywords: interpretability, segmentation
• Abstract: FlameFinder is a deep metric learning (DML) framework designed to accurately detect flames, even when obscured by smoke, using thermal images from firefighter drones during wildfire monitoring. Traditional RGB cameras struggle in such conditions, but thermal cameras can capture smoke-obscured flame features. However, they lack absolute thermal reference points, leading to false positives.To address this issue, FlameFinder utilizes paired thermal-RGB images for training. By learning latent flame features from smoke-free samples, the model becomes less biased towards relative thermal gradients. In testing, it identifies flames in smoky patches by analyzing their equivalent thermal-domain distribution. This method improves performance using both supervised and distance-based clustering metrics.The framework incorporates a flame segmentation method and a DML-aided detection framework. This includes utilizing center loss (CL), triplet center loss (TCL), and triplet cosine center loss (TCCL) to identify optimal cluster representatives for classification. However, the dominance of center loss over the other losses leads to the model missing features sensitive to them. To address this limitation, an attention mechanism is proposed. This mechanism allows for non-uniform feature contribution, amplifying the critical role of cosine and triplet loss in the DML framework. Additionally, it improves interpretability, class discrimination, and decreases intra-class variance. As a result, the proposed model surpasses the baseline by 4.4% in the FLAME2 dataset and 7% in the FLAME3 dataset for unobscured flame detection accuracy. Moreover, it demonstrates enhanced class separation in obscured scenarios compared to VGG19, ResNet18, and three backbone models tailored for flame detection.

Title: Efficient Denoising using Score Embedding in Score-based Diffusion Models

• Authors: Andrew S. Na, William Gao, Justin W.L. Wan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06661
• Pdf URL: https://arxiv.org/pdf/2404.06661
• Copy Paste: [[2404.06661]] Efficient Denoising using Score Embedding in Score-based Diffusion Models(https://arxiv.org/abs/2404.06661)
• Keywords: diffusion
• Abstract: It is well known that training a denoising score-based diffusion models requires tens of thousands of epochs and a substantial number of image data to train the model. In this paper, we propose to increase the efficiency in training score-based diffusion models. Our method allows us to decrease the number of epochs needed to train the diffusion model. We accomplish this by solving the log-density Fokker-Planck (FP) Equation numerically to compute the score \textit{before} training. The pre-computed score is embedded into the image to encourage faster training under slice Wasserstein distance. Consequently, it also allows us to decrease the number of images we need to train the neural network to learn an accurate score. We demonstrate through our numerical experiments the improved performance of our proposed method compared to standard score-based diffusion models. Our proposed method achieves a similar quality to the standard method meaningfully faster.

Title: Multi-modal Document Presentation Attack Detection With Forensics Trace Disentanglement

• Authors: Changsheng Chen, Yongyi Deng, Liangwei Lin, Zitong Yu, Zhimao Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06663
• Pdf URL: https://arxiv.org/pdf/2404.06663
• Copy Paste: [[2404.06663]] Multi-modal Document Presentation Attack Detection With Forensics Trace Disentanglement(https://arxiv.org/abs/2404.06663)
• Keywords: protect, attack, transformer
• Abstract: Document Presentation Attack Detection (DPAD) is an important measure in protecting the authenticity of a document image. However, recent DPAD methods demand additional resources, such as manual effort in collecting additional data or knowing the parameters of acquisition devices. This work proposes a DPAD method based on multi-modal disentangled traces (MMDT) without the above drawbacks. We first disentangle the recaptured traces by a self-supervised disentanglement and synthesis network to enhance the generalization capacity in document images with different contents and layouts. Then, unlike the existing DPAD approaches that rely only on data in the RGB domain, we propose to explicitly employ the disentangled recaptured traces as new modalities in the transformer backbone through adaptive multi-modal adapters to fuse RGB/trace features efficiently. Visualization of the disentangled traces confirms the effectiveness of the proposed method in different document contents. Extensive experiments on three benchmark datasets demonstrate the superiority of our MMDT method on representing forensic traces of recapturing distortion.

Title: CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge

• Authors: Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, Yejin Choi
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.06664
• Pdf URL: https://arxiv.org/pdf/2404.06664
• Copy Paste: [[2404.06664]] CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge(https://arxiv.org/abs/2404.06664)
• Keywords: large language model
• Abstract: Frontier large language models (LLMs) are developed by researchers and practitioners with skewed cultural backgrounds and on datasets with skewed sources. However, LLMs' (lack of) multicultural knowledge cannot be effectively assessed with current methods for developing benchmarks. Existing multicultural evaluations primarily rely on expensive and restricted human annotations or potentially outdated internet resources. Thus, they struggle to capture the intricacy, dynamics, and diversity of cultural norms. LLM-generated benchmarks are promising, yet risk propagating the same biases they are meant to measure. To synergize the creativity and expert cultural knowledge of human annotators and the scalability and standardizability of LLM-based automation, we introduce CulturalTeaming, an interactive red-teaming system that leverages human-AI collaboration to build truly challenging evaluation dataset for assessing the multicultural knowledge of LLMs, while improving annotators' capabilities and experiences. Our study reveals that CulturalTeaming's various modes of AI assistance support annotators in creating cultural questions, that modern LLMs fail at, in a gamified manner. Importantly, the increased level of AI assistance (e.g., LLM-generated revision hints) empowers users to create more difficult questions with enhanced perceived creativity of themselves, shedding light on the promises of involving heavier AI assistance in modern evaluation dataset creation procedures. Through a series of 1-hour workshop sessions, we gather CULTURALBENCH-V0.1, a compact yet high-quality evaluation dataset with users' red-teaming attempts, that different families of modern LLMs perform with accuracy ranging from 37.7% to 72.2%, revealing a notable gap in LLMs' multicultural proficiency.

Title: Deep Generative Data Assimilation in Multimodal Setting

• Authors: Yongquan Qu, Juan Nathaniel, Shuolin Li, Pierre Gentine
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06665
• Pdf URL: https://arxiv.org/pdf/2404.06665
• Copy Paste: [[2404.06665]] Deep Generative Data Assimilation in Multimodal Setting(https://arxiv.org/abs/2404.06665)
• Keywords: robust, diffusion, generative
• Abstract: Robust integration of physical knowledge and data is key to improve computational simulations, such as Earth system models. Data assimilation is crucial for achieving this goal because it provides a systematic framework to calibrate model outputs with observations, which can include remote sensing imagery and ground station measurements, with uncertainty quantification. Conventional methods, including Kalman filters and variational approaches, inherently rely on simplifying linear and Gaussian assumptions, and can be computationally expensive. Nevertheless, with the rapid adoption of data-driven methods in many areas of computational sciences, we see the potential of emulating traditional data assimilation with deep learning, especially generative models. In particular, the diffusion-based probabilistic framework has large overlaps with data assimilation principles: both allows for conditional generation of samples with a Bayesian inverse framework. These models have shown remarkable success in text-conditioned image generation or image-controlled video synthesis. Likewise, one can frame data assimilation as observation-conditioned state calibration. In this work, we propose SLAMS: Score-based Latent Assimilation in Multimodal Setting. Specifically, we assimilate in-situ weather station data and ex-situ satellite imagery to calibrate the vertical temperature profiles, globally. Through extensive ablation, we demonstrate that SLAMS is robust even in low-resolution, noisy, and sparse data settings. To our knowledge, our work is the first to apply deep generative framework for multimodal data assimilation using real-world datasets; an important step for building robust computational simulators, including the next-generation Earth system models. Our code is available at: https://github.com/yongquan-qu/SLAMS

Title: SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models

• Authors: Xinfeng Li, Yuchen Yang, Jiangyi Deng, Chen Yan, Yanjiao Chen, Xiaoyu Ji, Wenyuan Xu
• Subjects: cs.CV, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06666
• Pdf URL: https://arxiv.org/pdf/2404.06666
• Copy Paste: [[2404.06666]] SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models(https://arxiv.org/abs/2404.06666)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) models, such as Stable Diffusion, have exhibited remarkable performance in generating high-quality images from text descriptions in recent years. However, text-to-image models may be tricked into generating not-safe-for-work (NSFW) content, particularly in sexual scenarios. Existing countermeasures mostly focus on filtering inappropriate inputs and outputs, or suppressing improper text embeddings, which can block explicit NSFW-related content (e.g., naked or sexy) but may still be vulnerable to adversarial prompts inputs that appear innocent but are ill-intended. In this paper, we present SafeGen, a framework to mitigate unsafe content generation by text-to-image models in a text-agnostic manner. The key idea is to eliminate unsafe visual representations from the model regardless of the text input. In this way, the text-to-image model is resistant to adversarial prompts since unsafe visual representations are obstructed from within. Extensive experiments conducted on four datasets demonstrate SafeGen's effectiveness in mitigating unsafe content generation while preserving the high-fidelity of benign images. SafeGen outperforms eight state-of-the-art baseline methods and achieves 99.1% sexual content removal performance. Furthermore, our constructed benchmark of adversarial prompts provides a basis for future development and evaluation of anti-NSFW-generation methods.

Title: Forecasting the Future with Future Technologies: Advancements in Large Meteorological Models

• Authors: Hailong Shu, Yue Wang, Weiwei Song, Huichuang Guo, Zhen Song
• Subjects: cs.LG, cs.AI, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2404.06668
• Pdf URL: https://arxiv.org/pdf/2404.06668
• Copy Paste: [[2404.06668]] Forecasting the Future with Future Technologies: Advancements in Large Meteorological Models(https://arxiv.org/abs/2404.06668)
• Keywords: transformer
• Abstract: The field of meteorological forecasting has undergone a significant transformation with the integration of large models, especially those employing deep learning techniques. This paper reviews the advancements and applications of these models in weather prediction, emphasizing their role in transforming traditional forecasting methods. Models like FourCastNet, Pangu-Weather, GraphCast, ClimaX, and FengWu have made notable contributions by providing accurate, high-resolution forecasts, surpassing the capabilities of traditional Numerical Weather Prediction (NWP) models. These models utilize advanced neural network architectures, such as Convolutional Neural Networks (CNNs), Graph Neural Networks (GNNs), and Transformers, to process diverse meteorological data, enhancing predictive accuracy across various time scales and spatial resolutions. The paper addresses challenges in this domain, including data acquisition and computational demands, and explores future opportunities for model optimization and hardware advancements. It underscores the integration of artificial intelligence with conventional meteorological techniques, promising improved weather prediction accuracy and a significant contribution to addressing climate-related challenges. This synergy positions large models as pivotal in the evolving landscape of meteorological forecasting.

Title: Toward Cross-Layer Energy Optimizations in Machine Learning Systems

• Authors: Jae-Won Chung, Mosharaf Chowdhury
• Subjects: cs.LG, cs.AR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.06675
• Pdf URL: https://arxiv.org/pdf/2404.06675
• Copy Paste: [[2404.06675]] Toward Cross-Layer Energy Optimizations in Machine Learning Systems(https://arxiv.org/abs/2404.06675)
• Keywords: generative, large language model
• Abstract: The enormous energy consumption of machine learning (ML) and generative AI workloads shows no sign of waning, taking a toll on operating costs, power delivery, and environmental sustainability. Despite a long line of research on energy-efficient hardware, we found that software plays a critical role in ML energy optimization through two recent works: Zeus and Perseus. This is especially true for large language models (LLMs) because their model sizes and, therefore, energy demands are growing faster than hardware efficiency improvements. Therefore, we advocate for a cross-layer approach for energy optimizations in ML systems, where hardware provides architectural support that pushes energy-efficient software further, while software leverages and abstracts the hardware to develop techniques that bring hardware-agnostic energy-efficiency gains.

Title: Topological Feature Search Method for Multichannel EEG: Application in ADHD classification

• Authors: Tianming Cai, Guoying Zhao, Junbin Zang, Chen Zong, Zhidong Zhang, Chenyang Xue
• Subjects: cs.LG, eess.SP, stat.AP
• Abstract URL: https://arxiv.org/abs/2404.06676
• Pdf URL: https://arxiv.org/pdf/2404.06676
• Copy Paste: [[2404.06676]] Topological Feature Search Method for Multichannel EEG: Application in ADHD classification(https://arxiv.org/abs/2404.06676)
• Keywords: robust
• Abstract: In recent years, the preliminary diagnosis of Attention Deficit Hyperactivity Disorder (ADHD) using electroencephalography (EEG) has garnered attention from researchers. EEG, known for its expediency and efficiency, plays a pivotal role in the diagnosis and treatment of ADHD. However, the non-stationarity of EEG signals and inter-subject variability pose challenges to the diagnostic and classification processes. Topological Data Analysis (TDA) offers a novel perspective for ADHD classification, diverging from traditional time-frequency domain features. Yet, conventional TDA models are restricted to single-channel time series and are susceptible to noise, leading to the loss of topological features in persistence diagrams.This paper presents an enhanced TDA approach applicable to multi-channel EEG in ADHD. Initially, optimal input parameters for multi-channel EEG are determined. Subsequently, each channel's EEG undergoes phase space reconstruction (PSR) followed by the utilization of k-Power Distance to Measure (k-PDTM) for approximating ideal point clouds. Then, multi-dimensional time series are re-embedded, and TDA is applied to obtain topological feature information. Gaussian function-based Multivariate Kernel Density Estimation (MKDE) is employed in the merger persistence diagram to filter out desired topological feature mappings. Finally, persistence image (PI) method is utilized to extract topological features, and the influence of various weighting functions on the results is discussed.The effectiveness of our method is evaluated using the IEEE ADHD dataset. Results demonstrate that the accuracy, sensitivity, and specificity reach 85.60%, 83.61%, and 88.33%, respectively. Compared to traditional TDA methods, our method was effectively improved and outperforms typical nonlinear descriptors. These findings indicate that our method exhibits higher precision and robustness.

Title: Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncology

• Authors: Shashi Kant Gupta, Aditya Basu, Bradley Taylor, Anai Kothari, Hrituraj Singh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06680
• Pdf URL: https://arxiv.org/pdf/2404.06680
• Copy Paste: [[2404.06680]] Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncology(https://arxiv.org/abs/2404.06680)
• Keywords: generative, large language model
• Abstract: Retrieving information from EHR systems is essential for answering specific questions about patient journeys and improving the delivery of clinical care. Despite this fact, most EHR systems still rely on keyword-based searches. With the advent of generative large language models (LLMs), retrieving information can lead to better search and summarization capabilities. Such retrievers can also feed Retrieval-augmented generation (RAG) pipelines to answer any query. However, the task of retrieving information from EHR real-world clinical data contained within EHR systems in order to solve several downstream use cases is challenging due to the difficulty in creating query-document support pairs. We provide a blueprint for creating such datasets in an affordable manner using large language models. Our method results in a retriever that is 30-50 F-1 points better than propriety counterparts such as Ada and Mistral for oncology data elements. We further compare our model, called Onco-Retriever, against fine-tuned PubMedBERT model as well. We conduct an extensive manual evaluation on real-world EHR data along with latency analysis of the different models and provide a path forward for healthcare organizations to build domain-specific retrievers.

Title: Atlas-X Equity Financing: Unlocking New Methods to Securely Obfuscate Axe Inventory Data Based on Differential Privacy

• Authors: Antigoni Polychroniadou, Gabriele Cipriani, Richard Hua, Tucker Balch
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06686
• Pdf URL: https://arxiv.org/pdf/2404.06686
• Copy Paste: [[2404.06686]] Atlas-X Equity Financing: Unlocking New Methods to Securely Obfuscate Axe Inventory Data Based on Differential Privacy(https://arxiv.org/abs/2404.06686)
• Keywords: secure, privacy, fair
• Abstract: Banks publish daily a list of available securities/assets (axe list) to selected clients to help them effectively locate Long (buy) or Short (sell) trades at reduced financing rates. This reduces costs for the bank, as the list aggregates the bank's internal firm inventory per asset for all clients of long as well as short trades. However, this is somewhat problematic: (1) the bank's inventory is revealed; (2) trades of clients who contribute to the aggregated list, particularly those deemed large, are revealed to other clients. Clients conducting sizable trades with the bank and possessing a portion of the aggregated asset exceeding $50\%$ are considered to be concentrated clients. This could potentially reveal a trading concentrated client's activity to their competitors, thus providing an unfair advantage over the market. Atlas-X Axe Obfuscation, powered by new differential private methods, enables a bank to obfuscate its published axe list on a daily basis while under continual observation, thus maintaining an acceptable inventory Profit and Loss (P&L) cost pertaining to the noisy obfuscated axe list while reducing the clients' trading activity leakage. Our main differential private innovation is a differential private aggregator for streams (time series data) of both positive and negative integers under continual observation. For the last two years, Atlas-X system has been live in production across three major regions-USA, Europe, and Asia-at J.P. Morgan, a major financial institution, facilitating significant profitability. To our knowledge, it is the first differential privacy solution to be deployed in the financial sector. We also report benchmarks of our algorithm based on (anonymous) real and synthetic data to showcase the quality of our obfuscation and its success in production.

Title: Binomial Self-compensation for Motion Error in Dynamic 3D Scanning

• Authors: Geyou Zhang, Ce Zhu, Kai Liu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.06693
• Pdf URL: https://arxiv.org/pdf/2404.06693
• Copy Paste: [[2404.06693]] Binomial Self-compensation for Motion Error in Dynamic 3D Scanning(https://arxiv.org/abs/2404.06693)
• Keywords: robust
• Abstract: Phase shifting profilometry (PSP) is favored in high-precision 3D scanning due to its high accuracy, robustness, and pixel-wise property. However, a fundamental assumption of PSP that the object should remain static is violated in dynamic measurement, making PSP susceptible to object moving, resulting in ripple-like errors in the point clouds. We propose a pixel-wise and frame-wise loopable binomial self-compensation (BSC) algorithm to effectively and flexibly eliminate motion error in the four-step PSP. Our mathematical model demonstrates that by summing successive motion-affected phase frames weighted by binomial coefficients, motion error exponentially diminishes as the binomial order increases, accomplishing automatic error compensation through the motion-affected phase sequence, without the assistance of any intermediate variable. Extensive experiments show that our BSC outperforms the existing methods in reducing motion error, while achieving a depth map frame rate equal to the camera's acquisition rate (90 fps), enabling high-accuracy 3D reconstruction with a quasi-single-shot frame rate.

Title: How to Craft Backdoors with Unlabeled Data Alone?

• Authors: Yifei Wang, Wenhan Ma, Yisen Wang
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06694
• Pdf URL: https://arxiv.org/pdf/2404.06694
• Copy Paste: [[2404.06694]] How to Craft Backdoors with Unlabeled Data Alone?(https://arxiv.org/abs/2404.06694)
• Keywords: security, attack
• Abstract: Relying only on unlabeled data, Self-supervised learning (SSL) can learn rich features in an economical and scalable way. As the drive-horse for building foundation models, SSL has received a lot of attention recently with wide applications, which also raises security concerns where backdoor attack is a major type of threat: if the released dataset is maliciously poisoned, backdoored SSL models can behave badly when triggers are injected to test samples. The goal of this work is to investigate this potential risk. We notice that existing backdoors all require a considerable amount of \emph{labeled} data that may not be available for SSL. To circumvent this limitation, we explore a more restrictive setting called no-label backdoors, where we only have access to the unlabeled data alone, where the key challenge is how to select the proper poison set without using label information. We propose two strategies for poison selection: clustering-based selection using pseudolabels, and contrastive selection derived from the mutual information principle. Experiments on CIFAR-10 and ImageNet-100 show that both no-label backdoors are effective on many SSL methods and outperform random poisoning by a large margin. Code will be available at https://github.com/PKU-ML/nlb.

Title: Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting

• Authors: Hao Lu, Jiaqi Tang, Xinli Xu, Xu Cao, Yunpeng Zhang, Guoqing Wang, Dalong Du, Hao Chen, Yingcong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06700
• Pdf URL: https://arxiv.org/pdf/2404.06700
• Copy Paste: [[2404.06700]] Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting(https://arxiv.org/abs/2404.06700)
• Keywords: robust
• Abstract: The emergence of Multi-Camera 3D Object Detection (MC3D-Det), facilitated by bird's-eye view (BEV) representation, signifies a notable progression in 3D object detection. Scaling MC3D-Det training effectively accommodates varied camera parameters and urban landscapes, paving the way for the MC3D-Det foundation model. However, the multi-view fusion stage of the MC3D-Det method relies on the ill-posed monocular perception during training rather than surround refinement ability, leading to what we term "surround refinement degradation". To this end, our study presents a weak-to-strong eliciting framework aimed at enhancing surround refinement while maintaining robust monocular perception. Specifically, our framework employs weakly tuned experts trained on distinct subsets, and each is inherently biased toward specific camera configurations and scenarios. These biased experts can learn the perception of monocular degeneration, which can help the multi-view fusion stage to enhance surround refinement abilities. Moreover, a composite distillation strategy is proposed to integrate the universal knowledge of 2D foundation models and task-specific information. Finally, for MC3D-Det joint training, the elaborate dataset merge strategy is designed to solve the problem of inconsistent camera numbers and camera parameters. We set up a multiple dataset joint training benchmark for MC3D-Det and adequately evaluated existing methods. Further, we demonstrate the proposed framework brings a generalized and significant boost over multiple baselines. Our code is at \url{https://github.com/EnVision-Research/Scale-BEV}.

Title: Convolution-based Probability Gradient Loss for Semantic Segmentation

• Authors: Guohang Shan, Shuangcheng Jia
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06704
• Pdf URL: https://arxiv.org/pdf/2404.06704
• Copy Paste: [[2404.06704]] Convolution-based Probability Gradient Loss for Semantic Segmentation(https://arxiv.org/abs/2404.06704)
• Keywords: segmentation
• Abstract: In this paper, we introduce a novel Convolution-based Probability Gradient (CPG) loss for semantic segmentation. It employs convolution kernels similar to the Sobel operator, capable of computing the gradient of pixel intensity in an image. This enables the computation of gradients for both ground-truth and predicted category-wise probabilities. It enhances network performance by maximizing the similarity between these two probability gradients. Moreover, to specifically enhance accuracy near the object's boundary, we extract the object boundary based on the ground-truth probability gradient and exclusively apply the CPG loss to pixels belonging to boundaries. CPG loss proves to be highly convenient and effective. It establishes pixel relationships through convolution, calculating errors from a distinct dimension compared to pixel-wise loss functions such as cross-entropy loss. We conduct qualitative and quantitative analyses to evaluate the impact of the CPG loss on three well-established networks (DeepLabv3-Resnet50, HRNetV2-OCR, and LRASPP_MobileNet_V3_Large) across three standard segmentation datasets (Cityscapes, COCO-Stuff, ADE20K). Our extensive experimental results consistently and significantly demonstrate that the CPG loss enhances the mean Intersection over Union.

Title: CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers

• Authors: Longwei Zou, Qingyang Wang, Han Zhao, Jiangang Kong, Yi Yang, Yangdong Deng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06709
• Pdf URL: https://arxiv.org/pdf/2404.06709
• Copy Paste: [[2404.06709]] CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers(https://arxiv.org/abs/2404.06709)
• Keywords: large language model
• Abstract: The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on the LLaMA-33B model, while maintaining a close level of performance.

Title: MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Education

• Authors: Murong Yue, Wijdane Mifdal, Yixuan Zhang, Jennifer Suh, Ziyu Yao
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.06711
• Pdf URL: https://arxiv.org/pdf/2404.06711
• Copy Paste: [[2404.06711]] MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Education(https://arxiv.org/abs/2404.06711)
• Keywords: large language model
• Abstract: Mathematical modeling (MM) is considered a fundamental skill for students in STEM disciplines. Practicing the MM skill is often the most effective when students can engage in group discussion and collaborative problem-solving. However, due to unevenly distributed teachers and educational resources needed to monitor such group activities, students do not always receive equal opportunities for this practice. Excitingly, large language models (LLMs) have recently demonstrated strong capability in both modeling mathematical problems and simulating characters with different traits and properties. Drawing inspiration from the advancement of LLMs, in this work, we present MATHVC, the very first LLM-powered virtual classroom containing multiple LLM-simulated student characters, with whom a human student can practice their MM skill. To encourage each LLM character's behaviors to be aligned with their specified math-relevant properties (termed "characteristics alignment") and the overall conversational procedure to be close to an authentic student MM discussion (termed "conversational procedural alignment"), we proposed three innovations: integrating MM domain knowledge into the simulation, defining a symbolic schema as the ground for character simulation, and designing a meta planner at the platform level to drive the conversational procedure. Through experiments and ablation studies, we confirmed the effectiveness of our simulation approach and showed the promise for MATHVC to benefit real-life students in the future.

Title: Sparse Points to Dense Clouds: Enhancing 3D Detection with Limited LiDAR Data

• Authors: Aakash Kumar, Chen Chen, Ajmal Mian, Neils Lobo, Mubarak Shah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06715
• Pdf URL: https://arxiv.org/pdf/2404.06715
• Copy Paste: [[2404.06715]] Sparse Points to Dense Clouds: Enhancing 3D Detection with Limited LiDAR Data(https://arxiv.org/abs/2404.06715)
• Keywords: robust
• Abstract: 3D detection is a critical task that enables machines to identify and locate objects in three-dimensional space. It has a broad range of applications in several fields, including autonomous driving, robotics and augmented reality. Monocular 3D detection is attractive as it requires only a single camera, however, it lacks the accuracy and robustness required for real world applications. High resolution LiDAR on the other hand, can be expensive and lead to interference problems in heavy traffic given their active transmissions. We propose a balanced approach that combines the advantages of monocular and point cloud-based 3D detection. Our method requires only a small number of 3D points, that can be obtained from a low-cost, low-resolution sensor. Specifically, we use only 512 points, which is just 1% of a full LiDAR frame in the KITTI dataset. Our method reconstructs a complete 3D point cloud from this limited 3D information combined with a single image. The reconstructed 3D point cloud and corresponding image can be used by any multi-modal off-the-shelf detector for 3D object detection. By using the proposed network architecture with an off-the-shelf multi-modal 3D detector, the accuracy of 3D detection improves by 20% compared to the state-of-the-art monocular detection methods and 6% to 9% compare to the baseline multi-modal methods on KITTI and JackRabbot datasets.

Title: Poisoning Prevention in Federated Learning and Differential Privacy via Stateful Proofs of Execution

• Authors: Norrathep Rattanavipanon, Ivan de Oliviera Nunes
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06721
• Pdf URL: https://arxiv.org/pdf/2404.06721
• Copy Paste: [[2404.06721]] Poisoning Prevention in Federated Learning and Differential Privacy via Stateful Proofs of Execution(https://arxiv.org/abs/2404.06721)
• Keywords: security, privacy, attack, robust, federate
• Abstract: The rise in IoT-driven distributed data analytics, coupled with increasing privacy concerns, has led to a demand for effective privacy-preserving and federated data collection/model training mechanisms. In response, approaches such as Federated Learning (FL) and Local Differential Privacy (LDP) have been proposed and attracted much attention over the past few years. However, they still share the common limitation of being vulnerable to poisoning attacks wherein adversaries compromising edge devices feed forged (a.k.a. poisoned) data to aggregation back-ends, undermining the integrity of FL/LDP results. In this work, we propose a system-level approach to remedy this issue based on a novel security notion of Proofs of Stateful Execution (PoSX) for IoT/embedded devices' software. To realize the PoSX concept, we design SLAPP: a System-Level Approach for Poisoning Prevention. SLAPP leverages commodity security features of embedded devices - in particular ARM TrustZoneM security extensions - to verifiably bind raw sensed data to their correct usage as part of FL/LDP edge device routines. As a consequence, it offers robust security guarantees against poisoning. Our evaluation, based on real-world prototypes featuring multiple cryptographic primitives and data collection schemes, showcases SLAPP's security and low overhead.

Title: Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision

• Authors: Yingbo Ma, Suraj Kolla, Zhenhong Hu, Dhruv Kaliraman, Victoria Nolan, Ziyuan Guan, Yuanfang Ren, Brooke Armfield, Tezcan Ozrazgat-Baslanti, Jeremy A. Balch, Tyler J. Loftus, Parisa Rashidi, Azra Bihorac, Benjamin Shickel
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06723
• Pdf URL: https://arxiv.org/pdf/2404.06723
• Copy Paste: [[2404.06723]] Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision(https://arxiv.org/abs/2404.06723)
• Keywords: transformer
• Abstract: Modern electronic health records (EHRs) hold immense promise in tracking personalized patient health trajectories through sequential deep learning, owing to their extensive breadth, scale, and temporal granularity. Nonetheless, how to effectively leverage multiple modalities from EHRs poses significant challenges, given its complex characteristics such as high dimensionality, multimodality, sparsity, varied recording frequencies, and temporal irregularities. To this end, this paper introduces a novel multimodal contrastive learning framework, specifically focusing on medical time series and clinical notes. To tackle the challenge of sparsity and irregular time intervals in medical time series, the framework integrates temporal cross-attention transformers with a dynamic embedding and tokenization scheme for learning multimodal feature representations. To harness the interconnected relationships between medical time series and clinical notes, the framework equips a global contrastive loss, aligning a patient's multimodal feature representations with the corresponding discharge summaries. Since discharge summaries uniquely pertain to individual patients and represent a holistic view of the patient's hospital stay, machine learning models are led to learn discriminative multimodal features via global contrasting. Extensive experiments with a real-world EHR dataset demonstrated that our framework outperformed state-of-the-art approaches on the exemplar task of predicting the occurrence of nine postoperative complications for more than 120,000 major inpatient surgeries using multimodal data from UF health system split among three hospitals (UF Health Gainesville, UF Health Jacksonville, and UF Health Jacksonville-North).

Title: Disguised Copyright Infringement of Latent Diffusion Model

• Authors: Yiwei Lu, Matthew Y.R. Yang, Zuoqiu Liu, Gautam Kamath, Yaoliang Yu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06737
• Pdf URL: https://arxiv.org/pdf/2404.06737
• Copy Paste: [[2404.06737]] Disguised Copyright Infringement of Latent Diffusion Model(https://arxiv.org/abs/2404.06737)
• Keywords: diffusion, generative
• Abstract: Copyright infringement may occur when a generative model produces samples substantially similar to some copyrighted data that it had access to during the training phase. The notion of access usually refers to including copyrighted samples directly in the training dataset, which one may inspect to identify an infringement. We argue that such visual auditing largely overlooks a concealed copyright infringement, where one constructs a disguise that looks drastically different from the copyrighted sample yet still induces the effect of training Latent Diffusion Models on it. Such disguises only require indirect access to the copyrighted material and cannot be visually distinguished, thus easily circumventing the current auditing tools. In this paper, we provide a better understanding of such disguised copyright infringement by uncovering the disguises generation algorithm, the revelation of the disguises, and importantly, how to detect them to augment the existing toolbox. Additionally, we introduce a broader notion of acknowledgment for comprehending such indirect access.

Title: Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checking

• Authors: Xiaokang Zhang, Zijun Yao, Jing Zhang, Kaifeng Yun, Jifan Yu, Juanzi Li, Jie Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06742
• Pdf URL: https://arxiv.org/pdf/2404.06742
• Copy Paste: [[2404.06742]] Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checking(https://arxiv.org/abs/2404.06742)
• Keywords: large language model
• Abstract: Detecting non-factual content is a longstanding goal to increase the trustworthiness of large language models (LLMs) generations. Current factuality probes, trained using humanannotated labels, exhibit limited transferability to out-of-distribution content, while online selfconsistency checking imposes extensive computation burden due to the necessity of generating multiple outputs. This paper proposes PINOSE, which trains a probing model on offline self-consistency checking results, thereby circumventing the need for human-annotated data and achieving transferability across diverse data distributions. As the consistency check process is offline, PINOSE reduces the computational burden of generating multiple responses by online consistency verification. Additionally, it examines various aspects of internal states prior to response decoding, contributing to more effective detection of factual inaccuracies. Experiment results on both factuality detection and question answering benchmarks show that PINOSE achieves surpassing results than existing factuality detection methods. Our code and datasets are publicly available on this anonymized repository.

Title: DiffusionDialog: A Diffusion Model for Diverse Dialog Generation with Latent Space

• Authors: Jianxiang Xiang, Zhenhua Liu, Haodong Liu, Yin Bai, Jia Cheng, Wenliang Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06760
• Pdf URL: https://arxiv.org/pdf/2404.06760
• Copy Paste: [[2404.06760]] DiffusionDialog: A Diffusion Model for Diverse Dialog Generation with Latent Space(https://arxiv.org/abs/2404.06760)
• Keywords: diffusion
• Abstract: In real-life conversations, the content is diverse, and there exists the one-to-many problem that requires diverse generation. Previous studies attempted to introduce discrete or Gaussian-based continuous latent variables to address the one-to-many problem, but the diversity is limited. Recently, diffusion models have made breakthroughs in computer vision, and some attempts have been made in natural language processing. In this paper, we propose DiffusionDialog, a novel approach to enhance the diversity of dialogue generation with the help of diffusion model. In our approach, we introduce continuous latent variables into the diffusion model. The problem of using latent variables in the dialog task is how to build both an effective prior of the latent space and an inferring process to obtain the proper latent given the context. By combining the encoder and latent-based diffusion model, we encode the response's latent representation in a continuous space as the prior, instead of fixed Gaussian distribution or simply discrete ones. We then infer the latent by denoising step by step with the diffusion model. The experimental results show that our model greatly enhances the diversity of dialog responses while maintaining coherence. Furthermore, in further analysis, we find that our diffusion model achieves high inference efficiency, which is the main challenge of applying diffusion models in natural language processing.

Title: Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems

• Authors: Zhengyuan Liu, Stella Xin Yin, Geyu Lin, Nancy F. Chen
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.06762
• Pdf URL: https://arxiv.org/pdf/2404.06762
• Copy Paste: [[2404.06762]] Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems(https://arxiv.org/abs/2404.06762)
• Keywords: large language model
• Abstract: Intelligent Tutoring Systems (ITSs) can provide personalized and self-paced learning experience. The emergence of large language models (LLMs) further enables better human-machine interaction, and facilitates the development of conversational ITSs in various disciplines such as math and language learning. In dialogic teaching, recognizing and adapting to individual characteristics can significantly enhance student engagement and learning efficiency. However, characterizing and simulating student's persona remain challenging in training and evaluating conversational ITSs. In this work, we propose a framework to construct profiles of different student groups by refining and integrating both cognitive and noncognitive aspects, and leverage LLMs for personality-aware student simulation in a language learning scenario. We further enhance the framework with multi-aspect validation, and conduct extensive analysis from both teacher and student perspectives. Our experimental results show that state-of-the-art LLMs can produce diverse student responses according to the given language ability and personality traits, and trigger teacher's adaptive scaffolding strategies.

Title: Adapting LLaMA Decoder to Vision Transformer

• Authors: Jiahao Wang, Wenqi Shao, Mengzhao Chen, Chengyue Wu, Yong Liu, Kaipeng Zhang, Songyang Zhang, Kai Chen, Ping Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06773
• Pdf URL: https://arxiv.org/pdf/2404.06773
• Copy Paste: [[2404.06773]] Adapting LLaMA Decoder to Vision Transformer(https://arxiv.org/abs/2404.06773)
• Keywords: transformer, large language model, segmentation
• Abstract: This work examines whether decoder-only Transformers such as LLaMA, which were originally designed for large language models (LLMs), can be adapted to the computer vision field. We first "LLaMAfy" a standard ViT step-by-step to align with LLaMA's architecture, and find that directly applying a casual mask to the self-attention brings an attention collapse issue, resulting in the failure to the network training. We suggest to reposition the class token behind the image tokens with a post-sequence class token technique to overcome this challenge, enabling causal self-attention to efficiently capture the entire image's information. Additionally, we develop a soft mask strategy that gradually introduces a casual mask to the self-attention at the onset of training to facilitate the optimization behavior. The tailored model, dubbed as image LLaMA (iLLaMA), is akin to LLaMA in architecture and enables direct supervised learning. Its causal self-attention boosts computational efficiency and learns complex representation by elevating attention map ranks. iLLaMA rivals the performance with its encoder-only counterparts, achieving 75.1% ImageNet top-1 accuracy with only 5.7M parameters. Scaling the model to ~310M and pre-training on ImageNet-21K further enhances the accuracy to 86.0%. Extensive experiments demonstrate iLLaMA's reliable properties: calibration, shape-texture bias, quantization compatibility, ADE20K segmentation and CIFAR transfer learning. We hope our study can kindle fresh views to visual model design in the wave of LLMs. Pre-trained models and codes are available here.

Title: Logit Calibration and Feature Contrast for Robust Federated Learning on Non-IID Data

• Authors: Yu Qiao, Chaoning Zhang, Apurba Adhikary, Choong Seon Hong
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06776
• Pdf URL: https://arxiv.org/pdf/2404.06776
• Copy Paste: [[2404.06776]] Logit Calibration and Feature Contrast for Robust Federated Learning on Non-IID Data(https://arxiv.org/abs/2404.06776)
• Keywords: privacy, defense, attack, robust, federate
• Abstract: Federated learning (FL) is a privacy-preserving distributed framework for collaborative model training on devices in edge networks. However, challenges arise due to vulnerability to adversarial examples (AEs) and the non-independent and identically distributed (non-IID) nature of data distribution among devices, hindering the deployment of adversarially robust and accurate learning models at the edge. While adversarial training (AT) is commonly acknowledged as an effective defense strategy against adversarial attacks in centralized training, we shed light on the adverse effects of directly applying AT in FL that can severely compromise accuracy, especially in non-IID challenges. Given this limitation, this paper proposes FatCC, which incorporates local logit \underline{C}alibration and global feature \underline{C}ontrast into the vanilla federated adversarial training (\underline{FAT}) process from both logit and feature perspectives. This approach can effectively enhance the federated system's robust accuracy (RA) and clean accuracy (CA). First, we propose logit calibration, where the logits are calibrated during local adversarial updates, thereby improving adversarial robustness. Second, FatCC introduces feature contrast, which involves a global alignment term that aligns each local representation with unbiased global features, thus further enhancing robustness and accuracy in federated adversarial environments. Extensive experiments across multiple datasets demonstrate that FatCC achieves comparable or superior performance gains in both CA and RA compared to other baselines.

Title: Efficient and Scalable Chinese Vector Font Generation via Component Composition

• Authors: Jinyu Song, Weitao You, Shuhui Shi, Shuxuan Guo, Lingyun Sun, Wei Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.06779
• Pdf URL: https://arxiv.org/pdf/2404.06779
• Copy Paste: [[2404.06779]] Efficient and Scalable Chinese Vector Font Generation via Component Composition(https://arxiv.org/abs/2404.06779)
• Keywords: transformer
• Abstract: Chinese vector font generation is challenging due to the complex structure and huge amount of Chinese characters. Recent advances remain limited to generating a small set of characters with simple structure. In this work, we first observe that most Chinese characters can be disassembled into frequently-reused components. Therefore, we introduce the first efficient and scalable Chinese vector font generation approach via component composition, allowing generating numerous vector characters from a small set of components. To achieve this, we collect a large-scale dataset that contains over \textit{90K} Chinese characters with their components and layout information. Upon the dataset, we propose a simple yet effective framework based on spatial transformer networks (STN) and multiple losses tailored to font characteristics to learn the affine transformation of the components, which can be directly applied to the B\'ezier curves, resulting in Chinese characters in vector format. Our qualitative and quantitative experiments have demonstrated that our method significantly surpasses the state-of-the-art vector font generation methods in generating large-scale complex Chinese characters in both font generation and zero-shot font extension.

Title: Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior

• Authors: Fan Lu, Kwan-Yee Lin, Yan Xu, Hongsheng Li, Guang Chen, Changjun Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06780
• Pdf URL: https://arxiv.org/pdf/2404.06780
• Copy Paste: [[2404.06780]] Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior(https://arxiv.org/abs/2404.06780)
• Keywords: interpretability, diffusion
• Abstract: Text-to-3D generation has achieved remarkable success via large-scale text-to-image diffusion models. Nevertheless, there is no paradigm for scaling up the methodology to urban scale. Urban scenes, characterized by numerous elements, intricate arrangement relationships, and vast scale, present a formidable barrier to the interpretability of ambiguous textual descriptions for effective model optimization. In this work, we surmount the limitations by introducing a compositional 3D layout representation into text-to-3D paradigm, serving as an additional prior. It comprises a set of semantic primitives with simple geometric structures and explicit arrangement relationships, complementing textual descriptions and enabling steerable generation. Upon this, we propose two modifications -- (1) We introduce Layout-Guided Variational Score Distillation to address model optimization inadequacies. It conditions the score distillation sampling process with geometric and semantic constraints of 3D layouts. (2) To handle the unbounded nature of urban scenes, we represent 3D scene with a Scalable Hash Grid structure, incrementally adapting to the growing scale of urban scenes. Extensive experiments substantiate the capability of our framework to scale text-to-3D generation to large-scale urban scenes that cover over 1000m driving distance for the first time. We also present various scene editing demonstrations, showing the powers of steerable urban scene generation. Website: https://urbanarchitect.github.io.

Title: Private Wasserstein Distance with Random Noises

• Authors: Wenqian Li, Haozhi Wang, Zhe Huang, Yan Pang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06787
• Pdf URL: https://arxiv.org/pdf/2404.06787
• Copy Paste: [[2404.06787]] Private Wasserstein Distance with Random Noises(https://arxiv.org/abs/2404.06787)
• Keywords: privacy, attack, robust, federate
• Abstract: Wasserstein distance is a principle measure of data divergence from a distributional standpoint. However, its application becomes challenging in the context of data privacy, where sharing raw data is restricted. Prior attempts have employed techniques like Differential Privacy or Federated optimization to approximate Wasserstein distance. Nevertheless, these approaches often lack accuracy and robustness against potential attack. In this study, we investigate the underlying triangular properties within the Wasserstein space, leading to a straightforward solution named TriangleWad. This approach enables the computation of Wasserstein distance between datasets stored across different entities. Notably, TriangleWad is 20 times faster, making raw data information truly invisible, enhancing resilience against attacks, and without sacrificing estimation accuracy. Through comprehensive experimentation across various tasks involving both image and text data, we demonstrate its superior performance and generalizations.

Title: Extracting Clean and Balanced Subset for Noisy Long-tailed Classification

• Authors: Zhuo Li, He Zhao, Zhen Li, Tongliang Liu, Dandan Guo, Xiang Wan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06795
• Pdf URL: https://arxiv.org/pdf/2404.06795
• Copy Paste: [[2404.06795]] Extracting Clean and Balanced Subset for Noisy Long-tailed Classification(https://arxiv.org/abs/2404.06795)
• Keywords: robust
• Abstract: Real-world datasets usually are class-imbalanced and corrupted by label noise. To solve the joint issue of long-tailed distribution and label noise, most previous works usually aim to design a noise detector to distinguish the noisy and clean samples. Despite their effectiveness, they may be limited in handling the joint issue effectively in a unified way. In this work, we develop a novel pseudo labeling method using class prototypes from the perspective of distribution matching, which can be solved with optimal transport (OT). By setting a manually-specific probability measure and using a learned transport plan to pseudo-label the training samples, the proposed method can reduce the side-effects of noisy and long-tailed data simultaneously. Then we introduce a simple yet effective filter criteria by combining the observed labels and pseudo labels to obtain a more balanced and less noisy subset for a robust model training. Extensive experiments demonstrate that our method can extract this class-balanced subset with clean labels, which brings effective performance gains for long-tailed classification with label noise.

Title: MedRG: Medical Report Grounding with Multi-modal Large Language Model

• Authors: Ke Zou, Yang Bai, Zhihao Chen, Yang Zhou, Yidi Chen, Kai Ren, Meng Wang, Xuedong Yuan, Xiaojing Shen, Huazhu Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06798
• Pdf URL: https://arxiv.org/pdf/2404.06798
• Copy Paste: [[2404.06798]] MedRG: Medical Report Grounding with Multi-modal Large Language Model(https://arxiv.org/abs/2404.06798)
• Keywords: extraction, large language model
• Abstract: Medical Report Grounding is pivotal in identifying the most relevant regions in medical images based on a given phrase query, a critical aspect in medical image analysis and radiological diagnosis. However, prevailing visual grounding approaches necessitate the manual extraction of key phrases from medical reports, imposing substantial burdens on both system efficiency and physicians. In this paper, we introduce a novel framework, Medical Report Grounding (MedRG), an end-to-end solution for utilizing a multi-modal Large Language Model to predict key phrase by incorporating a unique token, BOX, into the vocabulary to serve as an embedding for unlocking detection capabilities. Subsequently, the vision encoder-decoder jointly decodes the hidden embedding and the input medical image, generating the corresponding grounding box. The experimental results validate the effectiveness of MedRG, surpassing the performance of the existing state-of-the-art medical phrase grounding methods. This study represents a pioneering exploration of the medical report grounding task, marking the first-ever endeavor in this domain.

Title: Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation

• Authors: Ruotong Pan, Boxi Cao, Hongyu Lin, Xianpei Han, Jia Zheng, Sirui Wang, Xunliang Cai, Le Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06809
• Pdf URL: https://arxiv.org/pdf/2404.06809
• Copy Paste: [[2404.06809]] Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation(https://arxiv.org/abs/2404.06809)
• Keywords: robust, large language model
• Abstract: The rapid development of large language models has led to the widespread adoption of Retrieval-Augmented Generation (RAG), which integrates external knowledge to alleviate knowledge bottlenecks and mitigate hallucinations. However, the existing RAG paradigm inevitably suffers from the impact of flawed information introduced during the retrieval phrase, thereby diminishing the reliability and correctness of the generated outcomes. In this paper, we propose Credibility-aware Generation (CAG), a universally applicable framework designed to mitigate the impact of flawed information in RAG. At its core, CAG aims to equip models with the ability to discern and process information based on its credibility. To this end, we propose an innovative data transformation framework that generates data based on credibility, thereby effectively endowing models with the capability of CAG. Furthermore, to accurately evaluate the models' capabilities of CAG, we construct a comprehensive benchmark covering three critical real-world scenarios. Experimental results demonstrate that our model can effectively understand and utilize credibility for generation, significantly outperform other models with retrieval augmentation, and exhibit resilience against the disruption caused by noisy documents, thereby maintaining robust performance. Moreover, our model supports customized credibility, offering a wide range of potential applications.

Title: Emotion-cause pair extraction method based on multi-granularity information and multi-module interaction

• Authors: Mingrui Fu, Weijiang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06812
• Pdf URL: https://arxiv.org/pdf/2404.06812
• Copy Paste: [[2404.06812]] Emotion-cause pair extraction method based on multi-granularity information and multi-module interaction(https://arxiv.org/abs/2404.06812)
• Keywords: extraction, transformer
• Abstract: The purpose of emotion-cause pair extraction is to extract the pair of emotion clauses and cause clauses. On the one hand, the existing methods do not take fully into account the relationship between the emotion extraction of two auxiliary tasks. On the other hand, the existing two-stage model has the problem of error propagation. In addition, existing models do not adequately address the emotion and cause-induced locational imbalance of samples. To solve these problems, an end-to-end multitasking model (MM-ECPE) based on shared interaction between GRU, knowledge graph and transformer modules is proposed. Furthermore, based on MM-ECPE, in order to use the encoder layer to better solve the problem of imbalanced distribution of clause distances between clauses and emotion clauses, we propose a novel encoding based on BERT, sentiment lexicon, and position-aware interaction module layer of emotion motif pair retrieval model (MM-ECPE(BERT)). The model first fully models the interaction between different tasks through the multi-level sharing module, and mines the shared information between emotion-cause pair extraction and the emotion extraction and cause extraction. Second, to solve the imbalanced distribution of emotion clauses and cause clauses problem, suitable labels are screened out according to the knowledge graph path length and task-specific features are constructed so that the model can focus on extracting pairs with corresponding emotion-cause relationships. Experimental results on the ECPE benchmark dataset show that the proposed model achieves good performance, especially on position-imbalanced samples.

Title: Zero-shot Point Cloud Completion Via 2D Priors

• Authors: Tianxin Huang, Zhiwen Yan, Yuyang Zhao, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06814
• Pdf URL: https://arxiv.org/pdf/2404.06814
• Copy Paste: [[2404.06814]] Zero-shot Point Cloud Completion Via 2D Priors(https://arxiv.org/abs/2404.06814)
• Keywords: diffusion
• Abstract: 3D point cloud completion is designed to recover complete shapes from partially observed point clouds. Conventional completion methods typically depend on extensive point cloud data for training %, with their effectiveness often constrained to object categories similar to those seen during training. In contrast, we propose a zero-shot framework aimed at completing partially observed point clouds across any unseen categories. Leveraging point rendering via Gaussian Splatting, we develop techniques of Point Cloud Colorization and Zero-shot Fractal Completion that utilize 2D priors from pre-trained diffusion models to infer missing regions. Experimental results on both synthetic and real-world scanned point clouds demonstrate that our approach outperforms existing methods in completing a variety of objects without any requirement for specific training data.

Title: Security Assessment of the LG Cryptosystem

• Authors: Étienne Burle, Hervé Talé Kalachi, Freddy Lende Metouke, Ayoub Otmani
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06815
• Pdf URL: https://arxiv.org/pdf/2404.06815
• Copy Paste: [[2404.06815]] Security Assessment of the LG Cryptosystem(https://arxiv.org/abs/2404.06815)
• Keywords: security, attack
• Abstract: The LG cryptosystem is a public-key encryption scheme in the rank metric using the recent family of $\lambdav-$Gabidulin codes and introduced in 2019 by Lau and Tan. In this paper, we present a cryptanalysis showing that the security of several parameters of the scheme have been overestimated. We also show the existence of some weak keys allowing an attacker to find in polynomial time an alternative private key.

Title: Enc2DB: A Hybrid and Adaptive Encrypted Query Processing Framework

• Authors: Hui Li, Jingwen Shi, Qi Tian, Zheng Li, Yan Fu, Bingqing Shen, Yaofeng Tu
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2404.06819
• Pdf URL: https://arxiv.org/pdf/2404.06819
• Copy Paste: [[2404.06819]] Enc2DB: A Hybrid and Adaptive Encrypted Query Processing Framework(https://arxiv.org/abs/2404.06819)
• Keywords: secure, privacy
• Abstract: As cloud computing gains traction, data owners are outsourcing their data to cloud service providers (CSPs) for Database Service (DBaaS), bringing in a deviation of data ownership and usage, and intensifying privacy concerns, especially with potential breaches by hackers or CSP insiders. To address that, encrypted database services propose encrypting every tuple and query statement before submitting to the CSP, ensuring data confidentiality when the CSP is honest-but-curious, or even compromised. Existing solutions either employ property preserving cryptography schemes, which can perform certain operations over ciphertext without decrypting the data over the CSP, or utilize trusted execution environment (TEE) to safeguard data and computations from the CSP. Based on these efforts, we introduce Enc2DB, a novel secure database system, following a hybrid strategy on PostgreSQL and openGauss. We present a micro-benchmarking test and self-adaptive mode switch strategy that can dynamically choose the best execution path (cryptography or TEE) to answer a given query. Besides, we also design and implement a ciphertext index compatible with native cost model and query optimizers to accelerate query processing. Empirical study over TPC-C test justifies that Enc2DB outperforms pure TEE and cryptography solutions, and our ciphertext index implementation also outperforms the state-of-the-art cryptographic-based system.

Title: Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge

• Authors: Li Zhou, Taelin Karidi, Nicolas Garneau, Yong Cao, Wanlong Liu, Wenyu Chen, Daniel Hershcovich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06833
• Pdf URL: https://arxiv.org/pdf/2404.06833
• Copy Paste: [[2404.06833]] Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge(https://arxiv.org/abs/2404.06833)
• Keywords: robust, large language model
• Abstract: Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet often lack a robust methodology to dissect these phenomena comprehensively. Our work aims to bridge this gap by delving into the Food domain, a universally relevant yet culturally diverse aspect of human life. We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings. By leveraging templates in six different languages, we investigate how LLMs interact with language-specific and cultural knowledge. Our findings reveal that (1) LLMs demonstrate a pronounced bias towards food knowledge prevalent in the United States; (2) Incorporating relevant cultural context significantly improves LLMs' ability to access cultural knowledge; (3) The efficacy of LLMs in capturing cultural nuances is highly dependent on the interplay between the probing language, the specific model architecture, and the cultural context in question. This research underscores the complexity of integrating cultural understanding into LLMs and emphasizes the importance of culturally diverse datasets to mitigate biases and enhance model performance across different cultural domains.

Title: Tuning-Free Adaptive Style Incorporation for Structure-Consistent Text-Driven Style Transfer

• Authors: Yanqi Ge, Jiaqi Liu, Qingnan Fan, Xi Jiang, Ye Huang, Shuai Qin, Hong Gu, Wen Li, Lixin Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06835
• Pdf URL: https://arxiv.org/pdf/2404.06835
• Copy Paste: [[2404.06835]] Tuning-Free Adaptive Style Incorporation for Structure-Consistent Text-Driven Style Transfer(https://arxiv.org/abs/2404.06835)
• Keywords: diffusion
• Abstract: In this work, we target the task of text-driven style transfer in the context of text-to-image (T2I) diffusion models. The main challenge is consistent structure preservation while enabling effective style transfer effects. The past approaches in this field directly concatenate the content and style prompts for a prompt-level style injection, leading to unavoidable structure distortions. In this work, we propose a novel solution to the text-driven style transfer task, namely, Adaptive Style Incorporation~(ASI), to achieve fine-grained feature-level style incorporation. It consists of the Siamese Cross-Attention~(SiCA) to decouple the single-track cross-attention to a dual-track structure to obtain separate content and style features, and the Adaptive Content-Style Blending (AdaBlending) module to couple the content and style information from a structure-consistent manner. Experimentally, our method exhibits much better performance in both structure preservation and stylized effects.

Title: O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation

• Authors: Muer Tie, Julong Wei, Zhengjun Wang, Ke Wu, Shansuai Yuan, Kaizhao Zhang, Jie Jia, Jieru Zhao, Zhongxue Gan, Wenchao Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06836
• Pdf URL: https://arxiv.org/pdf/2404.06836
• Copy Paste: [[2404.06836]] O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation(https://arxiv.org/abs/2404.06836)
• Keywords: segmentation
• Abstract: Online construction of open-ended language scenes is crucial for robotic applications, where open-vocabulary interactive scene understanding is required. Recently, neural implicit representation has provided a promising direction for online interactive mapping. However, implementing open-vocabulary scene understanding capability into online neural implicit mapping still faces three challenges: lack of local scene updating ability, blurry spatial hierarchical semantic segmentation and difficulty in maintaining multi-view consistency. To this end, we proposed O2V-mapping, which utilizes voxel-based language and geometric features to create an open-vocabulary field, thus allowing for local updates during online training process. Additionally, we leverage a foundational model for image segmentation to extract language features on object-level entities, achieving clear segmentation boundaries and hierarchical semantic features. For the purpose of preserving consistency in 3D object properties across different viewpoints, we propose a spatial adaptive voxel adjustment mechanism and a multi-view weight selection method. Extensive experiments on open-vocabulary object localization and semantic segmentation demonstrate that O2V-mapping achieves online construction of language scenes while enhancing accuracy, outperforming the previous SOTA method.

Title: Simpler becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified Corpora?

• Authors: Miriam Anschütz, Edoardo Mosca, Georg Groh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06838
• Pdf URL: https://arxiv.org/pdf/2404.06838
• Copy Paste: [[2404.06838]] Simpler becomes Harder: Do LLMs Exhibit a Coherent Behavior on Simplified Corpora?(https://arxiv.org/abs/2404.06838)
• Keywords: attack
• Abstract: Text simplification seeks to improve readability while retaining the original content and meaning. Our study investigates whether pre-trained classifiers also maintain such coherence by comparing their predictions on both original and simplified inputs. We conduct experiments using 11 pre-trained models, including BERT and OpenAI's GPT 3.5, across six datasets spanning three languages. Additionally, we conduct a detailed analysis of the correlation between prediction change rates and simplification types/strengths. Our findings reveal alarming inconsistencies across all languages and models. If not promptly addressed, simplified inputs can be easily exploited to craft zero-iteration model-agnostic adversarial attacks with success rates of up to 50%

Title: UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion

• Authors: Junsheng Zhou, Weiqi Zhang, Baorui Ma, Kanle Shi, Yu-Shen Liu, Zhizhong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06851
• Pdf URL: https://arxiv.org/pdf/2404.06851
• Copy Paste: [[2404.06851]] UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion(https://arxiv.org/abs/2404.06851)
• Keywords: diffusion
• Abstract: Diffusion models have shown remarkable results for image generation, editing and inpainting. Recent works explore diffusion models for 3D shape generation with neural implicit functions, i.e., signed distance function and occupancy function. However, they are limited to shapes with closed surfaces, which prevents them from generating diverse 3D real-world contents containing open surfaces. In this work, we present UDiFF, a 3D diffusion model for unsigned distance fields (UDFs) which is capable to generate textured 3D shapes with open surfaces from text conditions or unconditionally. Our key idea is to generate UDFs in spatial-frequency domain with an optimal wavelet transformation, which produces a compact representation space for UDF generation. Specifically, instead of selecting an appropriate wavelet transformation which requires expensive manual efforts and still leads to large information loss, we propose a data-driven approach to learn the optimal wavelet transformation for UDFs. We evaluate UDiFF to show our advantages by numerical and visual comparisons with the latest methods on widely used benchmarks. Page: https://weiqi-zhang.github.io/UDiFF.

Title: Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata

• Authors: Jinghong Chen, Weizhe Lin, Jingbiao Mei, Bill Byrne
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06854
• Pdf URL: https://arxiv.org/pdf/2404.06854
• Copy Paste: [[2404.06854]] Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata(https://arxiv.org/abs/2404.06854)
• Keywords: transformer
• Abstract: The Directed Acyclic Transformer is a fast non-autoregressive (NAR) model that performs well in Neural Machine Translation. Two issues prevent its application to general Natural Language Generation (NLG) tasks: frequent Out-Of-Vocabulary (OOV) errors and the inability to faithfully generate entity names. We introduce Control-DAG, a constrained decoding algorithm for our Directed Acyclic T5 (DA-T5) model which offers lexical, vocabulary and length control. We show that Control-DAG significantly enhances DA-T5 on the Schema Guided Dialogue and the DART datasets, establishing strong NAR results for Task-Oriented Dialogue and Data-to-Text NLG.

Title: RESSCAL3D: Resolution Scalable 3D Semantic Segmentation of Point Clouds

• Authors: Remco Royen, Adrian Munteanu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06863
• Pdf URL: https://arxiv.org/pdf/2404.06863
• Copy Paste: [[2404.06863]] RESSCAL3D: Resolution Scalable 3D Semantic Segmentation of Point Clouds(https://arxiv.org/abs/2404.06863)
• Keywords: segmentation
• Abstract: While deep learning-based methods have demonstrated outstanding results in numerous domains, some important functionalities are missing. Resolution scalability is one of them. In this work, we introduce a novel architecture, dubbed RESSCAL3D, providing resolution-scalable 3D semantic segmentation of point clouds. In contrast to existing works, the proposed method does not require the whole point cloud to be available to start inference. Once a low-resolution version of the input point cloud is available, first semantic predictions can be generated in an extremely fast manner. This enables early decision-making in subsequent processing steps. As additional points become available, these are processed in parallel. To improve performance, features from previously computed scales are employed as prior knowledge at the current scale. Our experiments show that RESSCAL3D is 31-62% faster than the non-scalable baseline while keeping a limited impact on performance. To the best of our knowledge, the proposed method is the first to propose a resolution-scalable approach for 3D semantic segmentation of point clouds based on deep learning.

Title: Fine color guidance in diffusion models and its application to image compression at extremely low bitrates

• Authors: Tom Bordin, Thomas Maugey
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06865
• Pdf URL: https://arxiv.org/pdf/2404.06865
• Copy Paste: [[2404.06865]] Fine color guidance in diffusion models and its application to image compression at extremely low bitrates(https://arxiv.org/abs/2404.06865)
• Keywords: diffusion
• Abstract: This study addresses the challenge of, without training or fine-tuning, controlling the global color aspect of images generated with a diffusion model. We rewrite the guidance equations to ensure that the outputs are closer to a known color map, and this without hindering the quality of the generation. Our method leads to new guidance equations. We show in the color guidance context that, the scaling of the guidance should not decrease but remains high throughout the diffusion process. In a second contribution, our guidance is applied in a compression framework, we combine both semantic and general color information on the image to decode the images at low cost. We show that our method is effective at improving fidelity and realism of compressed images at extremely low bit rates, when compared to other classical or more semantic oriented approaches.

Title: O-TALC: Steps Towards Combating Oversegmentation within Online Action Segmentation

• Authors: Matthew Kent Myers, Nick Wright, A. Stephen McGough, Nicholas Martin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06894
• Pdf URL: https://arxiv.org/pdf/2404.06894
• Copy Paste: [[2404.06894]] O-TALC: Steps Towards Combating Oversegmentation within Online Action Segmentation(https://arxiv.org/abs/2404.06894)
• Keywords: segmentation
• Abstract: Online temporal action segmentation shows a strong potential to facilitate many HRI tasks where extended human action sequences must be tracked and understood in real time. Traditional action segmentation approaches, however, operate in an offline two stage approach, relying on computationally expensive video wide features for segmentation, rendering them unsuitable for online HRI applications. In order to facilitate online action segmentation on a stream of incoming video data, we introduce two methods for improved training and inference of backbone action recognition models, allowing them to be deployed directly for online frame level classification. Firstly, we introduce surround dense sampling whilst training to facilitate training vs. inference clip matching and improve segment boundary predictions. Secondly, we introduce an Online Temporally Aware Label Cleaning (O-TALC) strategy to explicitly reduce oversegmentation during online inference. As our methods are backbone invariant, they can be deployed with computationally efficient spatio-temporal action recognition models capable of operating in real time with a small segmentation latency. We show our method outperforms similar online action segmentation work as well as matches the performance of many offline models with access to full temporal resolution when operating on challenging fine-grained datasets.

Title: DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting

• Authors: Shijie Zhou, Zhiwen Fan, Dejia Xu, Haoran Chang, Pradyumna Chari, Tejas Bharadwaj, Suya You, Zhangyang Wang, Achuta Kadambi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06903
• Pdf URL: https://arxiv.org/pdf/2404.06903
• Copy Paste: [[2404.06903]] DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting(https://arxiv.org/abs/2404.06903)
• Keywords: diffusion, generative
• Abstract: The increasing demand for virtual reality applications has highlighted the significance of crafting immersive 3D assets. We present a text-to-3D 360$^{\circ}$ scene generation pipeline that facilitates the creation of comprehensive 360$^{\circ}$ scenes for in-the-wild environments in a matter of minutes. Our approach utilizes the generative power of a 2D diffusion model and prompt self-refinement to create a high-quality and globally coherent panoramic image. This image acts as a preliminary "flat" (2D) scene representation. Subsequently, it is lifted into 3D Gaussians, employing splatting techniques to enable real-time exploration. To produce consistent 3D geometry, our pipeline constructs a spatially coherent structure by aligning the 2D monocular depth into a globally optimized point cloud. This point cloud serves as the initial state for the centroids of 3D Gaussians. In order to address invisible issues inherent in single-view inputs, we impose semantic and geometric constraints on both synthesized and input camera views as regularizations. These guide the optimization of Gaussians, aiding in the reconstruction of unseen regions. In summary, our method offers a globally consistent 3D scene within a 360$^{\circ}$ perspective, providing an enhanced immersive experience over existing techniques. Project website at: this http URL

Title: Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation

• Authors: Thomas Merth, Qichen Fu, Mohammad Rastegari, Mahyar Najibi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06910
• Pdf URL: https://arxiv.org/pdf/2404.06910
• Copy Paste: [[2404.06910]] Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation(https://arxiv.org/abs/2404.06910)
• Keywords: transformer, large language model
• Abstract: Despite the successes of large language models (LLMs), they exhibit significant drawbacks, particularly when processing long contexts. Their inference cost scales quadratically with respect to sequence length, making it expensive for deployment in some real-world text processing applications, such as retrieval-augmented generation (RAG). Additionally, LLMs also exhibit the "distraction phenomenon," where irrelevant context in the prompt degrades output quality. To address these drawbacks, we propose a novel RAG prompting methodology, superposition prompting, which can be directly applied to pre-trained transformer-based LLMs without the need for fine-tuning. At a high level, superposition prompting allows the LLM to process input documents in parallel prompt paths, discarding paths once they are deemed irrelevant. We demonstrate the capability of our method to simultaneously enhance time efficiency across a variety of question-answering benchmarks using multiple pre-trained LLMs. Furthermore, our technique significantly improves accuracy when the retrieved context is large relative the context the model was trained on. For example, our approach facilitates an 93x reduction in compute time while improving accuracy by 43\% on the NaturalQuestions-Open dataset with the MPT-7B instruction-tuned model over naive RAG.

Title: HRVDA: High-Resolution Visual Document Assistant

• Authors: Chaohu Liu, Kun Yin, Haoyu Cao, Xinghua Jiang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, Linli Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06918
• Pdf URL: https://arxiv.org/pdf/2404.06918
• Copy Paste: [[2404.06918]] HRVDA: High-Resolution Visual Document Assistant(https://arxiv.org/abs/2404.06918)
• Keywords: large language model
• Abstract: Leveraging vast training data, multimodal large language models (MLLMs) have demonstrated formidable general visual comprehension capabilities and achieved remarkable performance across various tasks. However, their performance in visual document understanding still leaves much room for improvement. This discrepancy is primarily attributed to the fact that visual document understanding is a fine-grained prediction task. In natural scenes, MLLMs typically use low-resolution images, leading to a substantial loss of visual information. Furthermore, general-purpose MLLMs do not excel in handling document-oriented instructions. In this paper, we propose a High-Resolution Visual Document Assistant (HRVDA), which bridges the gap between MLLMs and visual document understanding. This model employs a content filtering mechanism and an instruction filtering module to separately filter out the content-agnostic visual tokens and instruction-agnostic visual tokens, thereby achieving efficient model training and inference for high-resolution images. In addition, we construct a document-oriented visual instruction tuning dataset and apply a multi-stage training strategy to enhance the model's document modeling capabilities. Extensive experiments demonstrate that our model achieves state-of-the-art performance across multiple document understanding datasets, while maintaining training efficiency and inference speed comparable to low-resolution models.

Title: GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications

• Authors: Shishir G. Patil, Tianjun Zhang, Vivian Fang, Noppapon C., Roy Huang, Aaron Hao, Martin Casado, Joseph E. Gonzalez, Raluca Ada Popa, Ion Stoica
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06921
• Pdf URL: https://arxiv.org/pdf/2404.06921
• Copy Paste: [[2404.06921]] GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications(https://arxiv.org/abs/2404.06921)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are evolving beyond their classical role of providing information within dialogue systems to actively engaging with tools and performing actions on real-world applications and services. Today, humans verify the correctness and appropriateness of the LLM-generated outputs (e.g., code, functions, or actions) before putting them into real-world execution. This poses significant challenges as code comprehension is well known to be notoriously difficult. In this paper, we study how humans can efficiently collaborate with, delegate to, and supervise autonomous LLMs in the future. We argue that in many cases, "post-facto validation" - verifying the correctness of a proposed action after seeing the output - is much easier than the aforementioned "pre-facto validation" setting. The core concept behind enabling a post-facto validation system is the integration of an intuitive undo feature, and establishing a damage confinement for the LLM-generated actions as effective strategies to mitigate the associated risks. Using this, a human can now either revert the effect of an LLM-generated output or be confident that the potential risk is bounded. We believe this is critical to unlock the potential for LLM agents to interact with applications and services with limited (post-facto) human involvement. We describe the design and implementation of our open-source runtime for executing LLM actions, Gorilla Execution Engine (GoEX), and present open research questions towards realizing the goal of LLMs and applications interacting with each other with minimal human supervision. We release GoEX at https://github.com/ShishirPatil/gorilla/.

Title: MetaCheckGPT -- A Multi-task Hallucination Detection Using LLM Uncertainty and Meta-models

• Authors: Rahul Mehta, Andrew Hoblitzell, Jack O'Keefe, Hyeju Jang, Vasudeva Varma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06948
• Pdf URL: https://arxiv.org/pdf/2404.06948
• Copy Paste: [[2404.06948]] MetaCheckGPT -- A Multi-task Hallucination Detection Using LLM Uncertainty and Meta-models(https://arxiv.org/abs/2404.06948)
• Keywords: robust, large language model
• Abstract: This paper presents our winning solution for the SemEval-2024 Task 6 competition. We propose a meta-regressor framework of large language models (LLMs) for model evaluation and integration that achieves the highest scores on the leader board. Our approach leverages uncertainty signals present in a diverse basket of LLMs to detect hallucinations more robustly.

Title: Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy

• Authors: Yijin Liu, Fandong Meng, Jie Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06954
• Pdf URL: https://arxiv.org/pdf/2404.06954
• Copy Paste: [[2404.06954]] Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy(https://arxiv.org/abs/2404.06954)
• Keywords: large language model
• Abstract: Recently, dynamic computation methods have shown notable acceleration for Large Language Models (LLMs) by skipping several layers of computations through elaborate heuristics or additional predictors. However, in the decoding process of existing approaches, different samples are assigned different computational budgets, which cannot guarantee a stable and precise acceleration effect. Furthermore, existing approaches generally skip multiple contiguous layers at the bottom or top of the layers, leading to a drastic change in the model's layer-wise representations, and thus a consequent performance degeneration. Therefore, we propose a Unified Layer Skipping strategy, which selects the number of layers to skip computation based solely on the target speedup ratio, and then skips the corresponding number of intermediate layer computations in a balanced manner. Since the Unified Layer Skipping strategy is independent of input samples, it naturally supports popular acceleration techniques such as batch decoding and KV caching, thus demonstrating more practicality for real-world applications. Experimental results on two common tasks, i.e., machine translation and text summarization, indicate that given a target speedup ratio, the Unified Layer Skipping strategy significantly enhances both the inference performance and the actual model throughput over existing dynamic approaches.

Title: Adversarial purification for no-reference image-quality metrics: applicability study and new methods

• Authors: Aleksandr Gushchin, Anna Chistyakova, Vladislav Minashkin, Anastasia Antsiferova, Dmitriy Vatolin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06957
• Pdf URL: https://arxiv.org/pdf/2404.06957
• Copy Paste: [[2404.06957]] Adversarial purification for no-reference image-quality metrics: applicability study and new methods(https://arxiv.org/abs/2404.06957)
• Keywords: attack
• Abstract: Recently, the area of adversarial attacks on image quality metrics has begun to be explored, whereas the area of defences remains under-researched. In this study, we aim to cover that case and check the transferability of adversarial purification defences from image classifiers to IQA methods. In this paper, we apply several widespread attacks on IQA models and examine the success of the defences against them. The purification methodologies covered different preprocessing techniques, including geometrical transformations, compression, denoising, and modern neural network-based methods. Also, we address the challenge of assessing the efficacy of a defensive methodology by proposing ways to estimate output visual quality and the success of neutralizing attacks. Defences were tested against attack on three IQA metrics -- Linearity, MetaIQA and SPAQ. The code for attacks and defences is available at: (link is hidden for a blind review).

Title: Advancing Real-time Pandemic Forecasting Using Large Language Models: A COVID-19 Case Study

• Authors: Hongru Du, Jianan Zhao, Yang Zhao, Shaochong Xu, Xihong Lin, Yiran Chen, Lauren M. Gardner, Hao (Frank)Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06962
• Pdf URL: https://arxiv.org/pdf/2404.06962
• Copy Paste: [[2404.06962]] Advancing Real-time Pandemic Forecasting Using Large Language Models: A COVID-19 Case Study(https://arxiv.org/abs/2404.06962)
• Keywords: robust, large language model
• Abstract: Forecasting the short-term spread of an ongoing disease outbreak is a formidable challenge due to the complexity of contributing factors, some of which can be characterized through interlinked, multi-modality variables such as epidemiological time series data, viral biology, population demographics, and the intersection of public policy and human behavior. Existing forecasting model frameworks struggle with the multifaceted nature of relevant data and robust results translation, which hinders their performances and the provision of actionable insights for public health decision-makers. Our work introduces PandemicLLM, a novel framework with multi-modal Large Language Models (LLMs) that reformulates real-time forecasting of disease spread as a text reasoning problem, with the ability to incorporate real-time, complex, non-numerical information that previously unattainable in traditional forecasting models. This approach, through a unique AI-human cooperative prompt design and time series representation learning, encodes multi-modal data for LLMs. The model is applied to the COVID-19 pandemic, and trained to utilize textual public health policies, genomic surveillance, spatial, and epidemiological time series data, and is subsequently tested across all 50 states of the U.S. Empirically, PandemicLLM is shown to be a high-performing pandemic forecasting framework that effectively captures the impact of emerging variants and can provide timely and accurate predictions. The proposed PandemicLLM opens avenues for incorporating various pandemic-related data in heterogeneous formats and exhibits performance benefits over existing models. This study illuminates the potential of adapting LLMs and representation learning to enhance pandemic forecasting, illustrating how AI innovations can strengthen pandemic responses and crisis management in the future.

Title: V-MAD: Video-based Morphing Attack Detection in Operational Scenarios

• Authors: Guido Borghi, Annalisa Franco, Nicolò Di Domenico, Matteo Ferrara, Davide Maltoni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06963
• Pdf URL: https://arxiv.org/pdf/2404.06963
• Copy Paste: [[2404.06963]] V-MAD: Video-based Morphing Attack Detection in Operational Scenarios(https://arxiv.org/abs/2404.06963)
• Keywords: attack, robust
• Abstract: In response to the rising threat of the face morphing attack, this paper introduces and explores the potential of Video-based Morphing Attack Detection (V-MAD) systems in real-world operational scenarios. While current morphing attack detection methods primarily focus on a single or a pair of images, V-MAD is based on video sequences, exploiting the video streams often acquired by face verification tools available, for instance, at airport gates. Through this study, we show for the first time the advantages that the availability of multiple probe frames can bring to the morphing attack detection task, especially in scenarios where the quality of probe images is varied and might be affected, for instance, by pose or illumination variations. Experimental results on a real operational database demonstrate that video sequences represent valuable information for increasing the robustness and performance of morphing attack detection systems.

Title: FiP: a Fixed-Point Approach for Causal Generative Modeling

• Authors: Meyer Scetbon, Joel Jennings, Agrin Hilmkil, Cheng Zhang, Chao Ma
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.06969
• Pdf URL: https://arxiv.org/pdf/2404.06969
• Copy Paste: [[2404.06969]] FiP: a Fixed-Point Approach for Causal Generative Modeling(https://arxiv.org/abs/2404.06969)
• Keywords: transformer, generative
• Abstract: Modeling true world data-generating processes lies at the heart of empirical science. Structural Causal Models (SCMs) and their associated Directed Acyclic Graphs (DAGs) provide an increasingly popular answer to such problems by defining the causal generative process that transforms random noise into observations. However, learning them from observational data poses an ill-posed and NP-hard inverse problem in general. In this work, we propose a new and equivalent formalism that do not require DAGs to describe them, viewed as fixed-point problems on the causally ordered variables, and show three important cases where they can be uniquely recovered given the topological ordering (TO). To the best of our knowledge, we obtain the most general recovery results when the TO is known. Based on our theoretical findings, we design a two-stage causal generative model that first infers the causal order from observations in a zero-shot manner, thus by-passing the search, and then learns the generative fixed-point SCM on the ordered variables. To infer TOs from observations, we propose to amortize the learning of TOs on generated datasets by sequentially predicting the leaves of graphs seen during training. To learn fixed-point SCMs, we design a transformer-based architecture that exploits a new attention mechanism enabling the modeling of causal structures, and show that this parameterization is consistent with our formalism. Finally, we conduct an extensive evaluation of each method individually, and show that when combined, our model outperforms various baselines on generated out-of-distribution problems.

Title: Toward industrial use of continual learning : new metrics proposal for class incremental learning

• Authors: Konaté Mohamed Abbas, Anne-Françoise Yao, Thierry Chateau, Pierre Bouges
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06972
• Pdf URL: https://arxiv.org/pdf/2404.06972
• Copy Paste: [[2404.06972]] Toward industrial use of continual learning : new metrics proposal for class incremental learning(https://arxiv.org/abs/2404.06972)
• Keywords: fair
• Abstract: In this paper, we investigate continual learning performance metrics used in class incremental learning strategies for continual learning (CL) using some high performing methods. We investigate especially mean task accuracy. First, we show that it lacks of expressiveness through some simple experiments to capture performance. We show that monitoring average tasks performance is over optimistic and can lead to misleading conclusions for future real life industrial uses. Then, we propose first a simple metric, Minimal Incremental Class Accuracy (MICA) which gives a fair and more useful evaluation of different continual learning methods. Moreover, in order to provide a simple way to easily compare different methods performance in continual learning, we derive another single scalar metric that take into account the learning performance variation as well as our newly introduced metric.

Title: Accurate Tennis Court Line Detection on Amateur Recorded Matches

• Authors: Sameer Agrawal, Ragoth Sundararajan, Vishak Sagar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06977
• Pdf URL: https://arxiv.org/pdf/2404.06977
• Copy Paste: [[2404.06977]] Accurate Tennis Court Line Detection on Amateur Recorded Matches(https://arxiv.org/abs/2404.06977)
• Keywords: robust
• Abstract: Typically, tennis court line detection is done by running Hough-Line-Detection to find straight lines in the image, and then computing a transformation matrix from the detected lines to create the final court structure. We propose numerous improvements and enhancements to this algorithm, including using pretrained State-of-the-Art shadow-removal and object-detection ML models to make our line-detection more robust. Compared to the original algorithm, our method can accurately detect lines on amateur, dirty courts. When combined with a robust ball-tracking system, our method will enable accurate, automatic refereeing for amateur and professional tennis matches alike.

Title: Event Grounded Criminal Court View Generation withCooperative (Large) Language Models

• Authors: Linan Yue, Qi Liu, Lili Zhao, Li Wang, Weibo Gao, Yanqing An
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07001
• Pdf URL: https://arxiv.org/pdf/2404.07001
• Copy Paste: [[2404.07001]] Event Grounded Criminal Court View Generation withCooperative (Large) Language Models(https://arxiv.org/abs/2404.07001)
• Keywords: extraction
• Abstract: With the development of legal intelligence, Criminal Court View Generation has attracted much attention as a crucial task of legal intelligence, which aims to generate concise and coherent texts that summarize case facts and provide explanations for verdicts. Existing researches explore the key information in case facts to yield the court views. Most of them employ a coarse-grained approach that partitions the facts into broad segments (e.g., verdict-related sentences) to make predictions. However, this approach fails to capture the complex details present in the case facts, such as various criminal elements and legal events. To this end, in this paper, we propose an Event Grounded Generation (EGG) method for criminal court view generation with cooperative (Large) Language Models, which introduces the fine-grained event information into the generation. Specifically, we first design a LLMs-based extraction method that can extract events in case facts without massive annotated events. Then, we incorporate the extracted events into court view generation by merging case facts and events. Besides, considering the computational burden posed by the use of LLMs in the extraction phase of EGG, we propose a LLMs-free EGG method that can eliminate the requirement for event extraction using LLMs in the inference phase. Extensive experimental results on a real-world dataset clearly validate the effectiveness of our proposed method.

Title: LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models

• Authors: Igor Tufanov, Karen Hambardzumyan, Javier Ferrando, Elena Voita
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07004
• Pdf URL: https://arxiv.org/pdf/2404.07004
• Copy Paste: [[2404.07004]] LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models(https://arxiv.org/abs/2404.07004)
• Keywords: interpretability, transformer
• Abstract: We present the LM Transparency Tool (LM-TT), an open-source interactive toolkit for analyzing the internal workings of Transformer-based language models. Differently from previously existing tools that focus on isolated parts of the decision-making process, our framework is designed to make the entire prediction process transparent, and allows tracing back model behavior from the top-layer representation to very fine-grained parts of the model. Specifically, it (1) shows the important part of the whole input-to-output information flow, (2) allows attributing any changes done by a model block to individual attention heads and feed-forward neurons, (3) allows interpreting the functions of those heads or neurons. A crucial part of this pipeline is showing the importance of specific model components at each step. As a result, we are able to look at the roles of model components only in cases where they are important for a prediction. Since knowing which components should be inspected is key for analyzing large models where the number of these components is extremely high, we believe our tool will greatly support the interpretability community both in research settings and in practical applications.

Title: Knowledge graphs for empirical concept retrieval

• Authors: Lenka Tětková, Teresa Karen Scheidt, Maria Mandrup Fogh, Ellen Marie Gaunby Jørgensen, Finn Årup Nielsen, Lars Kai Hansen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07008
• Pdf URL: https://arxiv.org/pdf/2404.07008
• Copy Paste: [[2404.07008]] Knowledge graphs for empirical concept retrieval(https://arxiv.org/abs/2404.07008)
• Keywords: robust, explainability
• Abstract: Concept-based explainable AI is promising as a tool to improve the understanding of complex models at the premises of a given user, viz.\ as a tool for personalized explainability. An important class of concept-based explainability methods is constructed with empirically defined concepts, indirectly defined through a set of positive and negative examples, as in the TCAV approach (Kim et al., 2018). While it is appealing to the user to avoid formal definitions of concepts and their operationalization, it can be challenging to establish relevant concept datasets. Here, we address this challenge using general knowledge graphs (such as, e.g., Wikidata or WordNet) for comprehensive concept definition and present a workflow for user-driven data collection in both text and image domains. The concepts derived from knowledge graphs are defined interactively, providing an opportunity for personalization and ensuring that the concepts reflect the user's intentions. We test the retrieved concept datasets on two concept-based explainability methods, namely concept activation vectors (CAVs) and concept activation regions (CARs) (Crabbe and van der Schaar, 2022). We show that CAVs and CARs based on these empirical concept datasets provide robust and accurate explanations. Importantly, we also find good alignment between the models' representations of concepts and the structure of knowledge graphs, i.e., human representations. This supports our conclusion that knowledge graph-based concepts are relevant for XAI.

Title: A Mathematical Theory for Learning Semantic Languages by Abstract Learners

• Authors: Kuo-Yu Liao, Cheng-Shang Chang, Y.-W. Peter Hong
• Subjects: cs.CL, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07009
• Pdf URL: https://arxiv.org/pdf/2404.07009
• Copy Paste: [[2404.07009]] A Mathematical Theory for Learning Semantic Languages by Abstract Learners(https://arxiv.org/abs/2404.07009)
• Keywords: large language model
• Abstract: Recent advances in Large Language Models (LLMs) have demonstrated the emergence of capabilities (learned skills) when the number of system parameters and the size of training data surpass certain thresholds. The exact mechanisms behind such phenomena are not fully understood and remain a topic of active research. Inspired by the skill-text bipartite graph model presented in [1] for modeling semantic language, we develop a mathematical theory to explain the emergence of learned skills, taking the learning (or training) process into account. Our approach models the learning process for skills in the skill-text bipartite graph as an iterative decoding process in Low-Density Parity Check (LDPC) codes and Irregular Repetition Slotted ALOHA (IRSA). Using density evolution analysis, we demonstrate the emergence of learned skills when the ratio of the size of training texts to the number of skills exceeds a certain threshold. Our analysis also yields a scaling law for testing errors relative to the size of training texts. Upon completion of the training, we propose a method for semantic compression and discuss its application in semantic communication.

Title: Diffusion-based inpainting of incomplete Euclidean distance matrices of trajectories generated by a fractional Brownian motion

• Authors: Alexander Lobashev, Kirill Polovnikov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07029
• Pdf URL: https://arxiv.org/pdf/2404.07029
• Copy Paste: [[2404.07029]] Diffusion-based inpainting of incomplete Euclidean distance matrices of trajectories generated by a fractional Brownian motion(https://arxiv.org/abs/2404.07029)
• Keywords: diffusion, generative
• Abstract: Fractional Brownian trajectories (fBm) feature both randomness and strong scale-free correlations, challenging generative models to reproduce the intrinsic memory characterizing the underlying process. Here we test a diffusion probabilistic model on a specific dataset of corrupted images corresponding to incomplete Euclidean distance matrices of fBm at various memory exponents $H$. Our dataset implies uniqueness of the data imputation in the regime of low missing ratio, where the remaining partial graph is rigid, providing the ground truth for the inpainting. We find that the conditional diffusion generation stably reproduces the statistics of missing fBm-distributed distances for different values of $H$ exponent. Furthermore, while diffusion models have been recently shown to remember samples from the training database, we show that diffusion-based inpainting behaves qualitatively different from the database search with the increasing database size. Finally, we apply our fBm-trained diffusion model with $H=1/3$ for completion of chromosome distance matrices obtained in single-cell microscopy experiments, showing its superiority over the standard bioinformatics algorithms. Our source code is available on GitHub at https://github.com/alobashev/diffusion_fbm.

Title: An Evidential-enhanced Tri-Branch Consistency Learning Method for Semi-supervised Medical Image Segmentation

• Authors: Zhenxi Zhang, Heng Zhou, Xiaoran Shi, Ran Ran, Chunna Tian, Feng Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07032
• Pdf URL: https://arxiv.org/pdf/2404.07032
• Copy Paste: [[2404.07032]] An Evidential-enhanced Tri-Branch Consistency Learning Method for Semi-supervised Medical Image Segmentation(https://arxiv.org/abs/2404.07032)
• Keywords: segmentation
• Abstract: Semi-supervised segmentation presents a promising approach for large-scale medical image analysis, effectively reducing annotation burdens while achieving comparable performance. This methodology holds substantial potential for streamlining the segmentation process and enhancing its feasibility within clinical settings for translational investigations. While cross-supervised training, based on distinct co-training sub-networks, has become a prevalent paradigm for this task, addressing critical issues such as predication disagreement and label-noise suppression requires further attention and progress in cross-supervised training. In this paper, we introduce an Evidential Tri-Branch Consistency learning framework (ETC-Net) for semi-supervised medical image segmentation. ETC-Net employs three branches: an evidential conservative branch, an evidential progressive branch, and an evidential fusion branch. The first two branches exhibit complementary characteristics, allowing them to address prediction diversity and enhance training stability. We also integrate uncertainty estimation from the evidential learning into cross-supervised training, mitigating the negative impact of erroneous supervision signals. Additionally, the evidential fusion branch capitalizes on the complementary attributes of the first two branches and leverages an evidence-based Dempster-Shafer fusion strategy, supervised by more reliable and accurate pseudo-labels of unlabeled data. Extensive experiments conducted on LA, Pancreas-CT, and ACDC datasets demonstrate that ETC-Net surpasses other state-of-the-art methods for semi-supervised segmentation. The code will be made available in the near future at https://github.com/Medsemiseg.

Title: Remote Scheduler Contention Attacks

• Authors: Stefan Gast, Jonas Juffinger, Lukas Maar, Christoph Royer, Andreas Kogler, Daniel Gruss (Graz University of Technology)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.07042
• Pdf URL: https://arxiv.org/pdf/2404.07042
• Copy Paste: [[2404.07042]] Remote Scheduler Contention Attacks(https://arxiv.org/abs/2404.07042)
• Keywords: attack
• Abstract: In this paper, we investigate unexplored aspects of scheduler contention: We systematically study the leakage of all scheduler queues on AMD Zen 3 and show that all queues leak. We mount the first scheduler contention attacks on Zen 4, with a novel measurement method evoking an out-of-order race condition, more precise than the state of the art. We demonstrate the first inter-keystroke timing attacks based on scheduler contention, with an F1 score of $\geq$ 99.5 % and a standard deviation below 4 ms from the ground truth. Our end-to-end JavaScript attack transmits across Firefox instances, bypassing cross-origin policies and site isolation, with 891.9 bit/s (Zen 3) and 940.7 bit/s (Zen 4).

Title: Identification of Fine-grained Systematic Errors via Controlled Scene Generation

• Authors: Valentyn Boreiko, Matthias Hein, Jan Hendrik Metzen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07045
• Pdf URL: https://arxiv.org/pdf/2404.07045
• Copy Paste: [[2404.07045]] Identification of Fine-grained Systematic Errors via Controlled Scene Generation(https://arxiv.org/abs/2404.07045)
• Keywords: generative
• Abstract: Many safety-critical applications, especially in autonomous driving, require reliable object detectors. They can be very effectively assisted by a method to search for and identify potential failures and systematic errors before these detectors are deployed. Systematic errors are characterized by combinations of attributes such as object location, scale, orientation, and color, as well as the composition of their respective backgrounds. To identify them, one must rely on something other than real images from a test set because they do not account for very rare but possible combinations of attributes. To overcome this limitation, we propose a pipeline for generating realistic synthetic scenes with fine-grained control, allowing the creation of complex scenes with multiple objects. Our approach, BEV2EGO, allows for a realistic generation of the complete scene with road-contingent control that maps 2D bird's-eye view (BEV) scene configurations to a first-person view (EGO). In addition, we propose a benchmark for controlled scene generation to select the most appropriate generative outpainting model for BEV2EGO. We further use it to perform a systematic analysis of multiple state-of-the-art object detection models and discover differences between them.

Title: Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study

• Authors: Alessandro Stolfo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07060
• Pdf URL: https://arxiv.org/pdf/2404.07060
• Copy Paste: [[2404.07060]] Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study(https://arxiv.org/abs/2404.07060)
• Keywords: robust, large language model
• Abstract: We present an empirical study of groundedness in long-form question answering (LFQA) by retrieval-augmented large language models (LLMs). In particular, we evaluate whether every generated sentence is grounded in the retrieved documents or the model's pre-training data. Across 3 datasets and 4 model families, our findings reveal that a significant fraction of generated sentences are consistently ungrounded, even when those sentences contain correct ground-truth answers. Additionally, we examine the impacts of factors such as model size, decoding strategy, and instruction tuning on groundedness. Our results show that while larger models tend to ground their outputs more effectively, a significant portion of correct answers remains compromised by hallucinations. This study provides novel insights into the groundedness challenges in LFQA and underscores the necessity for more robust mechanisms in LLMs to mitigate the generation of ungrounded content.

Title: Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?

• Authors: Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, Yongfeng Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07066
• Pdf URL: https://arxiv.org/pdf/2404.07066
• Copy Paste: [[2404.07066]] Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?(https://arxiv.org/abs/2404.07066)
• Keywords: large language model
• Abstract: This paper studies the phenomenon that different concepts are learned in different layers of large language models, i.e. more difficult concepts are fully acquired with deeper layers. We define the difficulty of concepts by the level of abstraction, and here it is crudely categorized by factual, emotional, and inferential. Each category contains a spectrum of tasks, arranged from simple to complex. For example, within the factual dimension, tasks range from lie detection to categorizing mathematical problems. We employ a probing technique to extract representations from different layers of the model and apply these to classification tasks. Our findings reveal that models tend to efficiently classify simpler tasks, indicating that these concepts are learned in shallower layers. Conversely, more complex tasks may only be discernible at deeper layers, if at all. This paper explores the implications of these findings for our understanding of model learning processes and internal representations. Our implementation is available at \url{https://github.com/Luckfort/CD}.

Title: Implicit Multi-Spectral Transformer: An Lightweight and Effective Visible to Infrared Image Translation Model

• Authors: Yijia Chen, Pinghua Chen, Xiangxin Zhou, Yingtie Lei, Ziyang Zhou, Mingxian Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07072
• Pdf URL: https://arxiv.org/pdf/2404.07072
• Copy Paste: [[2404.07072]] Implicit Multi-Spectral Transformer: An Lightweight and Effective Visible to Infrared Image Translation Model(https://arxiv.org/abs/2404.07072)
• Keywords: transformer, generative
• Abstract: In the field of computer vision, visible light images often exhibit low contrast in low-light conditions, presenting a significant challenge. While infrared imagery provides a potential solution, its utilization entails high costs and practical limitations. Recent advancements in deep learning, particularly the deployment of Generative Adversarial Networks (GANs), have facilitated the transformation of visible light images to infrared images. However, these methods often experience unstable training phases and may produce suboptimal outputs. To address these issues, we propose a novel end-to-end Transformer-based model that efficiently converts visible light images into high-fidelity infrared images. Initially, the Texture Mapping Module and Color Perception Adapter collaborate to extract texture and color features from the visible light image. The Dynamic Fusion Aggregation Module subsequently integrates these features. Finally, the transformation into an infrared image is refined through the synergistic action of the Color Perception Adapter and the Enhanced Perception Attention mechanism. Comprehensive benchmarking experiments confirm that our model outperforms existing methods, producing infrared images of markedly superior quality, both qualitatively and quantitatively. Furthermore, the proposed model enables more effective downstream applications for infrared images than other methods.

Title: VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning

• Authors: Alexandros Xenos, Niki Maria Foteinopoulou, Ioanna Ntinou, Ioannis Patras, Georgios Tzimiropoulos
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.07078
• Pdf URL: https://arxiv.org/pdf/2404.07078
• Copy Paste: [[2404.07078]] VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning(https://arxiv.org/abs/2404.07078)
• Keywords: transformer
• Abstract: Recognising emotions in context involves identifying the apparent emotions of an individual, taking into account contextual cues from the surrounding scene. Previous approaches to this task have involved the design of explicit scene-encoding architectures or the incorporation of external scene-related information, such as captions. However, these methods often utilise limited contextual information or rely on intricate training pipelines. In this work, we leverage the groundbreaking capabilities of Vision-and-Large-Language Models (VLLMs) to enhance in-context emotion classification without introducing complexity to the training process in a two-stage approach. In the first stage, we propose prompting VLLMs to generate descriptions in natural language of the subject's apparent emotion relative to the visual context. In the second stage, the descriptions are used as contextual information and, along with the image input, are used to train a transformer-based architecture that fuses text and visual features before the final classification task. Our experimental results show that the text and image features have complementary information, and our fused architecture significantly outperforms the individual modalities without any complex training methods. We evaluate our approach on three different datasets, namely, EMOTIC, CAER-S, and BoLD, and achieve state-of-the-art or comparable accuracy across all datasets and metrics compared to much more complex approaches. The code will be made publicly available on github: https://github.com/NickyFot/EmoCommonSense.git

Title: Minimizing Chebyshev Prototype Risk Magically Mitigates the Perils of Overfitting

• Authors: Nathaniel Dean, Dilip Sarkar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07083
• Pdf URL: https://arxiv.org/pdf/2404.07083
• Copy Paste: [[2404.07083]] Minimizing Chebyshev Prototype Risk Magically Mitigates the Perils of Overfitting(https://arxiv.org/abs/2404.07083)
• Keywords: extraction
• Abstract: Overparameterized deep neural networks (DNNs), if not sufficiently regularized, are susceptible to overfitting their training examples and not generalizing well to test data. To discourage overfitting, researchers have developed multicomponent loss functions that reduce intra-class feature correlation and maximize inter-class feature distance in one or more layers of the network. By analyzing the penultimate feature layer activations output by a DNN's feature extraction section prior to the linear classifier, we find that modified forms of the intra-class feature covariance and inter-class prototype separation are key components of a fundamental Chebyshev upper bound on the probability of misclassification, which we designate the Chebyshev Prototype Risk (CPR). While previous approaches' covariance loss terms scale quadratically with the number of network features, our CPR bound indicates that an approximate covariance loss in log-linear time is sufficient to reduce the bound and is scalable to large architectures. We implement the terms of the CPR bound into our Explicit CPR (exCPR) loss function and observe from empirical results on multiple datasets and network architectures that our training algorithm reduces overfitting and improves upon previous approaches in many settings. Our code is available $\href{https://github.com/Deano1718/Regularization_exCPR}{here}$.

Title: Dynamic Generation of Personalities with Large Language Models

• Authors: Jianzhi Liu, Hexiang Gu, Tianyu Zheng, Liuyu Xiang, Huijia Wu, Jie Fu, Zhaofeng He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07084
• Pdf URL: https://arxiv.org/pdf/2404.07084
• Copy Paste: [[2404.07084]] Dynamic Generation of Personalities with Large Language Models(https://arxiv.org/abs/2404.07084)
• Keywords: large language model
• Abstract: In the realm of mimicking human deliberation, large language models (LLMs) show promising performance, thereby amplifying the importance of this research area. Deliberation is influenced by both logic and personality. However, previous studies predominantly focused on the logic of LLMs, neglecting the exploration of personality aspects. In this work, we introduce Dynamic Personality Generation (DPG), a dynamic personality generation method based on Hypernetworks. Initially, we embed the Big Five personality theory into GPT-4 to form a personality assessment machine, enabling it to evaluate characters' personality traits from dialogues automatically. We propose a new metric to assess personality generation capability based on this evaluation method. Then, we use this personality assessment machine to evaluate dialogues in script data, resulting in a personality-dialogue dataset. Finally, we fine-tune DPG on the personality-dialogue dataset. Experiments prove that DPG's personality generation capability is stronger after fine-tuning on this dataset than traditional fine-tuning methods, surpassing prompt-based GPT-4.

Title: Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection

• Authors: Linas Nasvytis, Kai Sandbrink, Jakob Foerster, Tim Franzmeyer, Christian Schroeder de Witt
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07099
• Pdf URL: https://arxiv.org/pdf/2404.07099
• Copy Paste: [[2404.07099]] Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection(https://arxiv.org/abs/2404.07099)
• Keywords: extraction
• Abstract: While reinforcement learning (RL) algorithms have been successfully applied across numerous sequential decision-making problems, their generalization to unforeseen testing environments remains a significant concern. In this paper, we study the problem of out-of-distribution (OOD) detection in RL, which focuses on identifying situations at test time that RL agents have not encountered in their training environments. We first propose a clarification of terminology for OOD detection in RL, which aligns it with the literature from other machine learning domains. We then present new benchmark scenarios for OOD detection, which introduce anomalies with temporal autocorrelation into different components of the agent-environment loop. We argue that such scenarios have been understudied in the current literature, despite their relevance to real-world situations. Confirming our theoretical predictions, our experimental results suggest that state-of-the-art OOD detectors are not able to identify such anomalies. To address this problem, we propose a novel method for OOD detection, which we call DEXTER (Detection via Extraction of Time Series Representations). By treating environment observations as time series data, DEXTER extracts salient time series features, and then leverages an ensemble of isolation forest algorithms to detect anomalies. We find that DEXTER can reliably identify anomalies across benchmark scenarios, exhibiting superior performance compared to both state-of-the-art OOD detectors and high-dimensional changepoint detectors adopted from statistics.

Title: Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs

• Authors: Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Suhang Wang, Yu Meng, Jiawei Han
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07103
• Pdf URL: https://arxiv.org/pdf/2404.07103
• Copy Paste: [[2404.07103]] Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs(https://arxiv.org/abs/2404.07103)
• Keywords: large language model
• Abstract: Large language models (LLMs), while exhibiting exceptional performance, suffer from hallucinations, especially on knowledge-intensive tasks. Existing works propose to augment LLMs with individual text units retrieved from external knowledge corpora to alleviate the issue. However, in many domains, texts are interconnected (e.g., academic papers in a bibliographic graph are linked by citations and co-authorships) which form a (text-attributed) graph. The knowledge in such graphs is encoded not only in single texts/nodes but also in their associated connections. To facilitate the research of augmenting LLMs with graphs, we manually construct a Graph Reasoning Benchmark dataset called GRBench, containing 1,740 questions that can be answered with the knowledge from 10 domain graphs. Then, we propose a simple and effective framework called Graph Chain-of-thought (Graph-CoT) to augment LLMs with graphs by encouraging LLMs to reason on the graph iteratively. Each Graph-CoT iteration consists of three sub-steps: LLM reasoning, LLM-graph interaction, and graph execution. We conduct systematic experiments with three LLM backbones on GRBench, where Graph-CoT outperforms the baselines consistently. The code is available at https://github.com/PeterGriffinJin/Graph-CoT.

Title: 3DMambaComplete: Exploring Structured State Space Model for Point Cloud Completion

• Authors: Yixuan Li, Weidong Yang, Ben Fei
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.07106
• Pdf URL: https://arxiv.org/pdf/2404.07106
• Copy Paste: [[2404.07106]] 3DMambaComplete: Exploring Structured State Space Model for Point Cloud Completion(https://arxiv.org/abs/2404.07106)
• Keywords: transformer
• Abstract: Point cloud completion aims to generate a complete and high-fidelity point cloud from an initially incomplete and low-quality input. A prevalent strategy involves leveraging Transformer-based models to encode global features and facilitate the reconstruction process. However, the adoption of pooling operations to obtain global feature representations often results in the loss of local details within the point cloud. Moreover, the attention mechanism inherent in Transformers introduces additional computational complexity, rendering it challenging to handle long sequences effectively. To address these issues, we propose 3DMambaComplete, a point cloud completion network built on the novel Mamba framework. It comprises three modules: HyperPoint Generation encodes point cloud features using Mamba's selection mechanism and predicts a set of Hyperpoints. A specific offset is estimated, and the down-sampled points become HyperPoints. The HyperPoint Spread module disperses these HyperPoints across different spatial locations to avoid concentration. Finally, a deformation method transforms the 2D mesh representation of HyperPoints into a fine-grained 3D structure for point cloud reconstruction. Extensive experiments conducted on various established benchmarks demonstrate that 3DMambaComplete surpasses state-of-the-art point cloud completion methods, as confirmed by qualitative and quantitative analyses.

Title: From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLMs-based Applications

• Authors: Yongqiang Ma, Lizhi Qin, Jiawei Liu, Yangyang Kang, Yue Zhang, Wei Lu, Xiaozhong Liu, Qikai Cheng
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.07108
• Pdf URL: https://arxiv.org/pdf/2404.07108
• Copy Paste: [[2404.07108]] From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLMs-based Applications(https://arxiv.org/abs/2404.07108)
• Keywords: large language model
• Abstract: Evaluating large language models (LLMs) is fundamental, particularly in the context of practical applications. Conventional evaluation methods, typically designed primarily for LLM development, yield numerical scores that ignore the user experience. Therefore, our study shifts the focus from model-centered to human-centered evaluation in the context of AI-powered writing assistance applications. Our proposed metric, termed ``Revision Distance,'' utilizes LLMs to suggest revision edits that mimic the human writing process. It is determined by counting the revision edits generated by LLMs. Benefiting from the generated revision edit details, our metric can provide a self-explained text evaluation result in a human-understandable manner beyond the context-independent score. Our results show that for the easy-writing task, ``Revision Distance'' is consistent with established metrics (ROUGE, Bert-score, and GPT-score), but offers more insightful, detailed feedback and better distinguishes between texts. Moreover, in the context of challenging academic writing tasks, our metric still delivers reliable evaluations where other metrics tend to struggle. Furthermore, our metric also holds significant potential for scenarios lacking reference texts.

Title: Unfolding ADMM for Enhanced Subspace Clustering of Hyperspectral Images

• Authors: Xianlu Li, Nicolas Nadisic, Shaoguang Huang, Aleksandra Pižurica
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07112
• Pdf URL: https://arxiv.org/pdf/2404.07112
• Copy Paste: [[2404.07112]] Unfolding ADMM for Enhanced Subspace Clustering of Hyperspectral Images(https://arxiv.org/abs/2404.07112)
• Keywords: interpretability
• Abstract: Deep subspace clustering methods are now prominent in clustering, typically using fully connected networks and a self-representation loss function. However, these methods often struggle with overfitting and lack interpretability. In this paper, we explore an alternative clustering approach based on deep unfolding. By unfolding iterative optimization methods into neural networks, this approach offers enhanced interpretability and reliability compared to data-driven deep learning methods, and greater adaptability and generalization than model-based approaches. Hence, unfolding has become widely used in inverse imaging problems, such as image restoration, reconstruction, and super-resolution, but has not been sufficiently explored yet in the context of clustering. In this work, we introduce an innovative clustering architecture for hyperspectral images (HSI) by unfolding an iterative solver based on the Alternating Direction Method of Multipliers (ADMM) for sparse subspace clustering. To our knowledge, this is the first attempt to apply unfolding ADMM for computing the self-representation matrix in subspace clustering. Moreover, our approach captures well the structural characteristics of HSI data by employing the K nearest neighbors algorithm as part of a structure preservation module. Experimental evaluation of three established HSI datasets shows clearly the potential of the unfolding approach in HSI clustering and even demonstrates superior performance compared to state-of-the-art techniques.

Title: Continuous Language Model Interpolation for Dynamic and Controllable Text Generation

• Authors: Sara Kangaslahti, David Alvarez-Melis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07117
• Pdf URL: https://arxiv.org/pdf/2404.07117
• Copy Paste: [[2404.07117]] Continuous Language Model Interpolation for Dynamic and Controllable Text Generation(https://arxiv.org/abs/2404.07117)
• Keywords: large language model
• Abstract: As large language models (LLMs) have gained popularity for a variety of use cases, making them adaptable and controllable has become increasingly important, especially for user-facing applications. While the existing literature on LLM adaptation primarily focuses on finding a model (or models) that optimizes a single predefined objective, here we focus on the challenging case where the model must dynamically adapt to diverse -- and often changing -- user preferences. For this, we leverage adaptation methods based on linear weight interpolation, casting them as continuous multi-domain interpolators that produce models with specific prescribed generation characteristics on-the-fly. Specifically, we use low-rank updates to fine-tune a base model to various different domains, yielding a set of anchor models with distinct generation profiles. Then, we use the weight updates of these anchor models to parametrize the entire (infinite) class of models contained within their convex hull. We empirically show that varying the interpolation weights yields predictable and consistent change in the model outputs with respect to all of the controlled attributes. We find that there is little entanglement between most attributes and identify and discuss the pairs of attributes for which this is not the case. Our results suggest that linearly interpolating between the weights of fine-tuned models facilitates predictable, fine-grained control of model outputs with respect to multiple stylistic characteristics simultaneously.

Title: Measuring proximity to standard planes during fetal brain ultrasound scanning

• Authors: Chiara Di Vece, Antonio Cirigliano, Meala Le Lous, Raffaele Napolitano, Anna L. David, Donald Peebles, Pierre Jannin, Francisco Vasconcelos, Danail Stoyanov
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07124
• Pdf URL: https://arxiv.org/pdf/2404.07124
• Copy Paste: [[2404.07124]] Measuring proximity to standard planes during fetal brain ultrasound scanning(https://arxiv.org/abs/2404.07124)
• Keywords: segmentation
• Abstract: This paper introduces a novel pipeline designed to bring ultrasound (US) plane pose estimation closer to clinical use for more effective navigation to the standard planes (SPs) in the fetal brain. We propose a semi-supervised segmentation model utilizing both labeled SPs and unlabeled 3D US volume slices. Our model enables reliable segmentation across a diverse set of fetal brain images. Furthermore, the model incorporates a classification mechanism to identify the fetal brain precisely. Our model not only filters out frames lacking the brain but also generates masks for those containing it, enhancing the relevance of plane pose regression in clinical settings. We focus on fetal brain navigation from 2D ultrasound (US) video analysis and combine this model with a US plane pose regression network to provide sensorless proximity detection to SPs and non-SPs planes; we emphasize the importance of proximity detection to SPs for guiding sonographers, offering a substantial advantage over traditional methods by allowing earlier and more precise adjustments during scanning. We demonstrate the practical applicability of our approach through validation on real fetal scan videos obtained from sonographers of varying expertise levels. Our findings demonstrate the potential of our approach to complement existing fetal US technologies and advance prenatal diagnostic practices.

Title: What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation

• Authors: Aaditya K. Singh, Ted Moskovitz, Felix Hill, Stephanie C.Y. Chan, Andrew M. Saxe
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07129
• Pdf URL: https://arxiv.org/pdf/2404.07129
• Copy Paste: [[2404.07129]] What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation(https://arxiv.org/abs/2404.07129)
• Keywords: robust, interpretability, transformer
• Abstract: In-context learning is a powerful emergent ability in transformer models. Prior work in mechanistic interpretability has identified a circuit element that may be critical for in-context learning -- the induction head (IH), which performs a match-and-copy operation. During training of large transformers on natural language data, IHs emerge around the same time as a notable phase change in the loss. Despite the robust evidence for IHs and this interesting coincidence with the phase change, relatively little is known about the diversity and emergence dynamics of IHs. Why is there more than one IH, and how are they dependent on each other? Why do IHs appear all of a sudden, and what are the subcircuits that enable them to emerge? We answer these questions by studying IH emergence dynamics in a controlled setting by training on synthetic data. In doing so, we develop and share a novel optogenetics-inspired causal framework for modifying activations throughout training. Using this framework, we delineate the diverse and additive nature of IHs. By clamping subsets of activations throughout training, we then identify three underlying subcircuits that interact to drive IH formation, yielding the phase change. Furthermore, these subcircuits shed light on data-dependent properties of formation, such as phase change timing, already showing the promise of this more in-depth understanding of subcircuits that need to "go right" for an induction head.

Title: Towards Robustness of Text-to-Visualization Translation against Lexical and Phrasal Variability

• Authors: Jinwei Lu, Yuanfeng Song, Haodi Zhang, Chen Zhang, Raymond Chi-Wing Wong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07135
• Pdf URL: https://arxiv.org/pdf/2404.07135
• Copy Paste: [[2404.07135]] Towards Robustness of Text-to-Visualization Translation against Lexical and Phrasal Variability(https://arxiv.org/abs/2404.07135)
• Keywords: robust
• Abstract: Text-to-Vis is an emerging task in the natural language processing (NLP) area that aims to automatically generate data visualizations from natural language questions (NLQs). Despite their progress, existing text-to-vis models often heavily rely on lexical matching between words in the questions and tokens in data schemas. This overreliance on lexical matching may lead to a diminished level of model robustness against input variations. In this study, we thoroughly examine the robustness of current text-to-vis models, an area that has not previously been explored. In particular, we construct the first robustness dataset nvBench-Rob, which contains diverse lexical and phrasal variations based on the original text-to-vis benchmark nvBench. Then, we found that the performance of existing text-to-vis models on this new dataset dramatically drops, implying that these methods exhibit inadequate robustness overall. Finally, we propose a novel framework based on Retrieval-Augmented Generation (RAG) technique, named GRED, specifically designed to address input perturbations in these two variants. The framework consists of three parts: NLQ-Retrieval Generator, Visualization Query-Retrieval Retuner and Annotation-based Debugger, which are used to tackle the challenges posed by natural language variants, programming style differences and data schema variants, respectively. Extensive experimental evaluations show that, compared to the state-of-the-art model RGVisNet in the Text-to-Vis field, RGDR performs better in terms of model robustness, with a 32% increase in accuracy on the proposed nvBench-Rob dataset.

Title: Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

• Authors: Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.07143
• Pdf URL: https://arxiv.org/pdf/2404.07143
• Copy Paste: [[2404.07143]] Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention(https://arxiv.org/abs/2404.07143)
• Keywords: transformer, large language model
• Abstract: This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.

Title: Lost in Translation: Modern Neural Networks Still Struggle With Small Realistic Image Transformations

• Authors: Ofir Shifman, Yair Weiss
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07153
• Pdf URL: https://arxiv.org/pdf/2404.07153
• Copy Paste: [[2404.07153]] Lost in Translation: Modern Neural Networks Still Struggle With Small Realistic Image Transformations(https://arxiv.org/abs/2404.07153)
• Keywords: robust
• Abstract: Deep neural networks that achieve remarkable performance in image classification have previously been shown to be easily fooled by tiny transformations such as a one pixel translation of the input image. In order to address this problem, two approaches have been proposed in recent years. The first approach suggests using huge datasets together with data augmentation in the hope that a highly varied training set will teach the network to learn to be invariant. The second approach suggests using architectural modifications based on sampling theory to deal explicitly with image translations. In this paper, we show that these approaches still fall short in robustly handling 'natural' image translations that simulate a subtle change in camera orientation. Our findings reveal that a mere one-pixel translation can result in a significant change in the predicted image representation for approximately 40% of the test images in state-of-the-art models (e.g. open-CLIP trained on LAION-2B or DINO-v2) , while models that are explicitly constructed to be robust to cyclic translations can still be fooled with 1 pixel realistic (non-cyclic) translations 11% of the time. We present Robust Inference by Crop Selection: a simple method that can be proven to achieve any desired level of consistency, although with a modest tradeoff with the model's accuracy. Importantly, we demonstrate how employing this method reduces the ability to fool state-of-the-art models with a 1 pixel translation to less than 5% while suffering from only a 1% drop in classification accuracy. Additionally, we show that our method can be easy adjusted to deal with circular shifts as well. In such case we achieve 100% robustness to integer shifts with state-of-the-art accuracy, and with no need for any further training.

Title: A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks

• Authors: Neel Mishra, Bamdev Mishra, Pratik Jawanpuria, Pawan Kumar
• Subjects: cs.LG, math.NA, math.OC
• Abstract URL: https://arxiv.org/abs/2404.07172
• Pdf URL: https://arxiv.org/pdf/2404.07172
• Copy Paste: [[2404.07172]] A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks(https://arxiv.org/abs/2404.07172)
• Keywords: generative
• Abstract: A novel first-order method is proposed for training generative adversarial networks (GANs). It modifies the Gauss-Newton method to approximate the min-max Hessian and uses the Sherman-Morrison inversion formula to calculate the inverse. The method corresponds to a fixed-point method that ensures necessary contraction. To evaluate its effectiveness, numerical experiments are conducted on various datasets commonly used in image generation tasks, such as MNIST, Fashion MNIST, CIFAR10, FFHQ, and LSUN. Our method is capable of generating high-fidelity images with greater diversity across multiple datasets. It also achieves the highest inception score for CIFAR10 among all compared methods, including state-of-the-art second-order methods. Additionally, its execution time is comparable to that of first-order min-max methods.

Title: Self-supervised Monocular Depth Estimation on Water Scenes via Specular Reflection Prior

• Authors: Zhengyang Lu, Ying Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07176
• Pdf URL: https://arxiv.org/pdf/2404.07176
• Copy Paste: [[2404.07176]] Self-supervised Monocular Depth Estimation on Water Scenes via Specular Reflection Prior(https://arxiv.org/abs/2404.07176)
• Keywords: segmentation
• Abstract: Monocular depth estimation from a single image is an ill-posed problem for computer vision due to insufficient reliable cues as the prior knowledge. Besides the inter-frame supervision, namely stereo and adjacent frames, extensive prior information is available in the same frame. Reflections from specular surfaces, informative intra-frame priors, enable us to reformulate the ill-posed depth estimation task as a multi-view synthesis. This paper proposes the first self-supervision for deep-learning depth estimation on water scenes via intra-frame priors, known as reflection supervision and geometrical constraints. In the first stage, a water segmentation network is performed to separate the reflection components from the entire image. Next, we construct a self-supervised framework to predict the target appearance from reflections, perceived as other perspectives. The photometric re-projection error, incorporating SmoothL1 and a novel photometric adaptive SSIM, is formulated to optimize pose and depth estimation by aligning the transformed virtual depths and source ones. As a supplement, the water surface is determined from real and virtual camera positions, which complement the depth of the water area. Furthermore, to alleviate these laborious ground truth annotations, we introduce a large-scale water reflection scene (WRS) dataset rendered from Unreal Engine 4. Extensive experiments on the WRS dataset prove the feasibility of the proposed method compared to state-of-the-art depth estimation techniques.

Title: Move Anything with Layered Scene Diffusion

• Authors: Jiawei Ren, Mengmeng Xu, Jui-Chieh Wu, Ziwei Liu, Tao Xiang, Antoine Toisoul
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07178
• Pdf URL: https://arxiv.org/pdf/2404.07178
• Copy Paste: [[2404.07178]] Move Anything with Layered Scene Diffusion(https://arxiv.org/abs/2404.07178)
• Keywords: diffusion
• Abstract: Diffusion models generate images with an unprecedented level of quality, but how can we freely rearrange image layouts? Recent works generate controllable scenes via learning spatially disentangled latent codes, but these methods do not apply to diffusion models due to their fixed forward process. In this work, we propose SceneDiffusion to optimize a layered scene representation during the diffusion sampling process. Our key insight is that spatial disentanglement can be obtained by jointly denoising scene renderings at different spatial layouts. Our generated scenes support a wide range of spatial editing operations, including moving, resizing, cloning, and layer-wise appearance editing operations, including object restyling and replacing. Moreover, a scene can be generated conditioned on a reference image, thus enabling object moving for in-the-wild images. Notably, this approach is training-free, compatible with general text-to-image diffusion models, and responsive in less than a second.

Title: InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models

• Authors: Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.07191
• Pdf URL: https://arxiv.org/pdf/2404.07191
• Copy Paste: [[2404.07191]] InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models(https://arxiv.org/abs/2404.07191)
• Keywords: extraction, diffusion, generative
• Abstract: We present InstantMesh, a feed-forward framework for instant 3D mesh generation from a single image, featuring state-of-the-art generation quality and significant training scalability. By synergizing the strengths of an off-the-shelf multiview diffusion model and a sparse-view reconstruction model based on the LRM architecture, InstantMesh is able to create diverse 3D assets within 10 seconds. To enhance the training efficiency and exploit more geometric supervisions, e.g, depths and normals, we integrate a differentiable iso-surface extraction module into our framework and directly optimize on the mesh representation. Experimental results on public datasets demonstrate that InstantMesh significantly outperforms other latest image-to-3D baselines, both qualitatively and quantitatively. We release all the code, weights, and demo of InstantMesh, with the intention that it can make substantial contributions to the community of 3D generative AI and empower both researchers and content creators.

Title: RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion

• Authors: Jaidev Shriram, Alex Trevithick, Lingjie Liu, Ravi Ramamoorthi
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07199
• Pdf URL: https://arxiv.org/pdf/2404.07199
• Copy Paste: [[2404.07199]] RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion(https://arxiv.org/abs/2404.07199)
• Keywords: diffusion
• Abstract: We introduce RealmDreamer, a technique for generation of general forward-facing 3D scenes from text descriptions. Our technique optimizes a 3D Gaussian Splatting representation to match complex text prompts. We initialize these splats by utilizing the state-of-the-art text-to-image generators, lifting their samples into 3D, and computing the occlusion volume. We then optimize this representation across multiple views as a 3D inpainting task with image-conditional diffusion models. To learn correct geometric structure, we incorporate a depth diffusion model by conditioning on the samples from the inpainting model, giving rich geometric structure. Finally, we finetune the model using sharpened samples from image generators. Notably, our technique does not require video or multi-view data and can synthesize a variety of high-quality 3D scenes in different styles, consisting of multiple objects. Its generality additionally allows 3D synthesis from a single image.

Title: UMBRAE: Unified Multimodal Decoding of Brain Signals

• Authors: Weihao Xia, Raoul de Charette, Cengiz Öztireli, Jing-Hao Xue
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07202
• Pdf URL: https://arxiv.org/pdf/2404.07202
• Copy Paste: [[2404.07202]] UMBRAE: Unified Multimodal Decoding of Brain Signals(https://arxiv.org/abs/2404.07202)
• Keywords: large language model
• Abstract: We address prevailing challenges of the brain-powered research, departing from the observation that the literature hardly recover accurate spatial information and require subject-specific models. To address these challenges, we propose UMBRAE, a unified multimodal decoding of brain signals. First, to extract instance-level conceptual and spatial details from neural signals, we introduce an efficient universal brain encoder for multimodal-brain alignment and recover object descriptions at multiple levels of granularity from subsequent multimodal large language model (MLLM). Second, we introduce a cross-subject training strategy mapping subject-specific features to a common feature space. This allows a model to be trained on multiple subjects without extra resources, even yielding superior results compared to subject-specific models. Further, we demonstrate this supports weakly-supervised adaptation to new subjects, with only a fraction of the total training data. Experiments demonstrate that UMBRAE not only achieves superior results in the newly introduced tasks but also outperforms methods in well established tasks. To assess our method, we construct and share with the community a comprehensive brain understanding benchmark BrainHub. Our code and benchmark are available at https://weihaox.github.io/UMBRAE.

Title: GoodDrag: Towards Good Practices for Drag Editing with Diffusion Models

• Authors: Zewei Zhang, Huan Liu, Jun Chen, Xiangyu Xu
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.07206
• Pdf URL: https://arxiv.org/pdf/2404.07206
• Copy Paste: [[2404.07206]] GoodDrag: Towards Good Practices for Drag Editing with Diffusion Models(https://arxiv.org/abs/2404.07206)
• Keywords: diffusion
• Abstract: In this paper, we introduce GoodDrag, a novel approach to improve the stability and image quality of drag editing. Unlike existing methods that struggle with accumulated perturbations and often result in distortions, GoodDrag introduces an AlDD framework that alternates between drag and denoising operations within the diffusion process, effectively improving the fidelity of the result. We also propose an information-preserving motion supervision operation that maintains the original features of the starting point for precise manipulation and artifact reduction. In addition, we contribute to the benchmarking of drag editing by introducing a new dataset, Drag100, and developing dedicated quality assessment metrics, Dragging Accuracy Index and Gemini Score, utilizing Large Multimodal Models. Extensive experiments demonstrate that the proposed GoodDrag compares favorably against the state-of-the-art approaches both qualitatively and quantitatively. The project page is https://gooddrag.github.io.