2024-02-13

Title: Adversarial Text Purification: A Large Language Model Approach for Defense

• Authors: Raha Moraffah, Shubh Khandelwal, Amrita Bhattacharjee, Huan Liu
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06655
• Pdf URL: https://arxiv.org/pdf/2402.06655
• Copy Paste: [[2402.06655]] Adversarial Text Purification: A Large Language Model Approach for Defense(https://arxiv.org/abs/2402.06655)
• Keywords: generative
• Abstract: Adversarial purification is a defense mechanism for safeguarding classifiers against adversarial attacks without knowing the type of attacks or training of the classifier. These techniques characterize and eliminate adversarial perturbations from the attacked inputs, aiming to restore purified samples that retain similarity to the initially attacked ones and are correctly classified by the classifier. Due to the inherent challenges associated with characterizing noise perturbations for discrete inputs, adversarial text purification has been relatively unexplored. In this paper, we investigate the effectiveness of adversarial purification methods in defending text classifiers. We propose a novel adversarial text purification that harnesses the generative capabilities of Large Language Models (LLMs) to purify adversarial text without the need to explicitly characterize the discrete noise perturbations. We utilize prompt engineering to exploit LLMs for recovering the purified examples for given adversarial examples such that they are semantically similar and correctly classified. Our proposed method demonstrates remarkable performance over various classifiers, improving their accuracy under the attack by over 65% on average.

Title: Explainable Adversarial Learning Framework on Physical Layer Secret Keys Combating Malicious Reconfigurable Intelligent Surface

• Authors: Zhuangkun Wei, Wenxiu Hu, Weisi Guo
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06663
• Pdf URL: https://arxiv.org/pdf/2402.06663
• Copy Paste: [[2402.06663]] Explainable Adversarial Learning Framework on Physical Layer Secret Keys Combating Malicious Reconfigurable Intelligent Surface(https://arxiv.org/abs/2402.06663)
• Keywords: generative
• Abstract: The development of reconfigurable intelligent surfaces (RIS) is a double-edged sword to physical layer security (PLS). Whilst a legitimate RIS can yield beneficial impacts including increased channel randomness to enhance physical layer secret key generation (PL-SKG), malicious RIS can poison legitimate channels and crack most of existing PL-SKGs. In this work, we propose an adversarial learning framework between legitimate parties (namely Alice and Bob) to address this Man-in-the-middle malicious RIS (MITM-RIS) eavesdropping. First, the theoretical mutual information gap between legitimate pairs and MITM-RIS is deduced. Then, Alice and Bob leverage generative adversarial networks (GANs) to learn to achieve a common feature surface that does not have mutual information overlap with MITM-RIS. Next, we aid signal processing interpretation of black-box neural networks by using a symbolic explainable AI (xAI) representation. These symbolic terms of dominant neurons aid feature engineering-based validation and future design of PLS common feature space. Simulation results show that our proposed GAN-based and symbolic-based PL-SKGs can achieve high key agreement rates between legitimate users, and is even resistant to MITM-RIS Eve with the knowledge of legitimate feature generation (NNs or formulas). This therefore paves the way to secure wireless communications with untrusted reflective devices in future 6G.

Title: Entropy-Regularized Token-Level Policy Optimization for Large Language Models

• Authors: Muning Wen, Cheng Deng, Jun Wang, Weinan Zhang, Ying Wen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06700
• Pdf URL: https://arxiv.org/pdf/2402.06700
• Copy Paste: [[2402.06700]] Entropy-Regularized Token-Level Policy Optimization for Large Language Models(https://arxiv.org/abs/2402.06700)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. Traditional approaches often depend on meticulously designed prompts, high-quality examples, or additional reward models for in-context learning, supervised fine-tuning, or RLHF. Reinforcement learning (RL) presents a dynamic alternative for LLMs to overcome these dependencies by engaging directly with task-specific environments. Nonetheless, it faces significant hurdles: 1) instability stemming from the exponentially vast action space requiring exploration; 2) challenges in assigning token-level credit based on action-level reward signals, resulting in discord between maximizing rewards and accurately modeling corpus data. In response to these challenges, we introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. At the heart of ETPO is our novel per-token soft Bellman update, designed to harmonize the RL process with the principles of language modeling. This methodology decomposes the Q-function update from a coarse action-level view to a more granular token-level perspective, backed by theoretical proof of optimization consistency. Crucially, this decomposition renders linear time complexity in action exploration. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks; results show that ETPO achieves effective performance improvement on the CodeLlama-7B model and surpasses a variant PPO baseline inherited from RLHF. This underlines ETPO's potential as a robust method for refining the interactive decision-making capabilities of LLMs.

Title: NICE: To Optimize In-Context Examples or Not?

• Authors: Pragya Srivastava, Satvik Golechha, Amit Deshpande, Amit Sharma
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06733
• Pdf URL: https://arxiv.org/pdf/2402.06733
• Copy Paste: [[2402.06733]] NICE: To Optimize In-Context Examples or Not?(https://arxiv.org/abs/2402.06733)
• Keywords: in-context
• Abstract: Recent works have shown that large language models (LLMs) work remarkably well on a wide range of tasks through in-context learning and optimization of in-context examples (ICE). However, most of these studies assume either a fixed or no instruction provided in the prompt, leading to the apparent consensus that the optimization of in-context examples is critical for better performance. We challenge this consensus for instruction-tuned LLMs by investigating the necessity of optimizing in-context examples when task-specific instructions are provided, and find that there are tasks for which various ways of optimizing in-context examples yield diminishing returns. We introduce a task-specific metric called \metriclong{} (\metric) that quantifies the learnability of tasks from a given instruction, and provides a heuristic that helps decide whether to optimize for instructions or ICE for any new task. On a wide range of tasks and a systematically created instruction set with gradually added details, we validate our hypothesis empirically by computing \metric with query-dependent bins of examples, comparing different instructions with ICE selection methods, and performing label perturbation experiments. We conclude that tasks can be divided into two broad classes based on the \metric metric, where the returns on ICE optimization follow predictable trends when instructions are provided in the prompt.

Title: ExGRG: Explicitly-Generated Relation Graph for Self-Supervised Representation Learning

• Authors: Mahdi Naseri, Mahdi Biparva
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06737
• Pdf URL: https://arxiv.org/pdf/2402.06737
• Copy Paste: [[2402.06737]] ExGRG: Explicitly-Generated Relation Graph for Self-Supervised Representation Learning(https://arxiv.org/abs/2402.06737)
• Keywords: self-supervised
• Abstract: Self-supervised Learning (SSL) has emerged as a powerful technique in pre-training deep learning models without relying on expensive annotated labels, instead leveraging embedded signals in unlabeled data. While SSL has shown remarkable success in computer vision tasks through intuitive data augmentation, its application to graph-structured data poses challenges due to the semantic-altering and counter-intuitive nature of graph augmentations. Addressing this limitation, this paper introduces a novel non-contrastive SSL approach to Explicitly Generate a compositional Relation Graph (ExGRG) instead of relying solely on the conventional augmentation-based implicit relation graph. ExGRG offers a framework for incorporating prior domain knowledge and online extracted information into the SSL invariance objective, drawing inspiration from the Laplacian Eigenmap and Expectation-Maximization (EM). Employing an EM perspective on SSL, our E-step involves relation graph generation to identify candidates to guide the SSL invariance objective, and M-step updates the model parameters by integrating the derived relational information. Extensive experimentation on diverse node classification datasets demonstrates the superiority of our method over state-of-the-art techniques, affirming ExGRG as an effective adoption of SSL for graph representation learning.

Title: EntGPT: Linking Generative Large Language Models with Knowledge Bases

• Authors: Yifan Ding, Amrit Poudel, Qingkai Zeng, Tim Weninger, Balaji Veeramani, Sanmitra Bhattacharya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06738
• Pdf URL: https://arxiv.org/pdf/2402.06738
• Copy Paste: [[2402.06738]] EntGPT: Linking Generative Large Language Models with Knowledge Bases(https://arxiv.org/abs/2402.06738)
• Keywords: generative
• Abstract: The ability of Large Language Models (LLMs) to generate factually correct output remains relatively unexplored due to the lack of fact-checking and knowledge grounding during training and inference. In this work, we aim to address this challenge through the Entity Disambiguation (ED) task. We first consider prompt engineering, and design a three-step hard-prompting method to probe LLMs' ED performance without supervised fine-tuning (SFT). Overall, the prompting method improves the micro-F_1 score of the original vanilla models by a large margin, on some cases up to 36% and higher, and obtains comparable performance across 10 datasets when compared to existing methods with SFT. We further improve the knowledge grounding ability through instruction tuning (IT) with similar prompts and responses. The instruction-tuned model not only achieves higher micro-F1 score performance as compared to several baseline methods on supervised entity disambiguation tasks with an average micro-F_1 improvement of 2.1% over the existing baseline models, but also obtains higher accuracy on six Question Answering (QA) tasks in the zero-shot setting. Our methodologies apply to both open- and closed-source LLMs.

Title: Transfer learning with generative models for object detection on limited datasets

• Authors: Matteo Paiano, Stefano Martina, Carlotta Giannelli, Filippo Caruso
• Subjects: cs.CV, cond-mat.dis-nn, cs.AI, cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2402.06784
• Pdf URL: https://arxiv.org/pdf/2402.06784
• Copy Paste: [[2402.06784]] Transfer learning with generative models for object detection on limited datasets(https://arxiv.org/abs/2402.06784)
• Keywords: diffusion, generative
• Abstract: The availability of data is limited in some fields, especially for object detection tasks, where it is necessary to have correctly labeled bounding boxes around each object. A notable example of such data scarcity is found in the domain of marine biology, where it is useful to develop methods to automatically detect submarine species for environmental monitoring. To address this data limitation, the state-of-the-art machine learning strategies employ two main approaches. The first involves pretraining models on existing datasets before generalizing to the specific domain of interest. The second strategy is to create synthetic datasets specifically tailored to the target domain using methods like copy-paste techniques or ad-hoc simulators. The first strategy often faces a significant domain shift, while the second demands custom solutions crafted for the specific task. In response to these challenges, here we propose a transfer learning framework that is valid for a generic scenario. In this framework, generated images help to improve the performances of an object detector in a few-real data regime. This is achieved through a diffusion-based generative model that was pretrained on large generic datasets, and is not trained on the task-specific domain. We validate our approach on object detection tasks, specifically focusing on fishes in an underwater environment, and on the more common domain of cars in an urban setting. Our method achieves detection performance comparable to models trained on thousands of images, using only a few hundreds of input data. Our results pave the way for new generative AI-based protocols for machine learning applications in various domains, for instance ranging from geophysics to biology and medicine.

Title: Generative Nowcasting of Marine Fog Visibility in the Grand Banks area and Sable Island in Canada

• Authors: Eren Gultepe, Sen Wang, Byron Blomquist, Harindra J.S. Fernando, O. Patrick Kreidl, David J. Delene, Ismail Gultepe
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2402.06800
• Pdf URL: https://arxiv.org/pdf/2402.06800
• Copy Paste: [[2402.06800]] Generative Nowcasting of Marine Fog Visibility in the Grand Banks area and Sable Island in Canada(https://arxiv.org/abs/2402.06800)
• Keywords: generative
• Abstract: This study presents the application of generative deep learning techniques to evaluate marine fog visibility nowcasting using the FATIMA (Fog and turbulence interactions in the marine atmosphere) campaign observations collected during July 2022 in the North Atlantic in the Grand Banks area and vicinity of Sable Island (SI), northeast of Canada. The measurements were collected using the Vaisala Forward Scatter Sensor model FD70 and Weather Transmitter model WXT50, and Gill R3A ultrasonic anemometer mounted on the Research Vessel Atlantic Condor. To perform nowcasting, the time series of fog visibility (Vis), wind speed, dew point depression, and relative humidity with respect to water were preprocessed to have lagged time step features. Generative nowcasting of Vis time series for lead times of 30 and 60 minutes were performed using conditional generative adversarial networks (cGAN) regression at visibility thresholds of Vis < 1 km and < 10 km. Extreme gradient boosting (XGBoost) was used as a baseline method for comparison against cGAN. At the 30 min lead time, Vis was best predicted with cGAN at Vis < 1 km (RMSE = 0.151 km) and with XGBoost at Vis < 10 km (RMSE = 2.821 km). At the 60 min lead time, Vis was best predicted with XGBoost at Vis < 1 km (RMSE = 0.167 km) and Vis < 10 km (RMSE = 3.508 km), but the cGAN RMSE was similar to XGBoost. Despite nowcasting Vis at 30 min being quite difficult, the ability of the cGAN model to track the variation in Vis at 1 km suggests that there is potential for generative analysis of marine fog visibility using observational meteorological parameters.

Title: Towards Principled Assessment of Tabular Data Synthesis Algorithms

• Authors: Yuntao Du, Ninghui Li
• Subjects: cs.CR, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06806
• Pdf URL: https://arxiv.org/pdf/2402.06806
• Copy Paste: [[2402.06806]] Towards Principled Assessment of Tabular Data Synthesis Algorithms(https://arxiv.org/abs/2402.06806)
• Keywords: diffusion
• Abstract: Data synthesis has been advocated as an important approach for utilizing data while protecting data privacy. A large number of tabular data synthesis algorithms (which we call synthesizers) have been proposed. Some synthesizers satisfy Differential Privacy, while others aim to provide privacy in a heuristic fashion. A comprehensive understanding of the strengths and weaknesses of these synthesizers remains elusive due to lacking principled evaluation metrics and missing head-to-head comparisons of newly developed synthesizers that take advantage of diffusion models and large language models with state-of-the-art marginal-based synthesizers. In this paper, we present a principled and systematic evaluation framework for assessing tabular data synthesis algorithms. Specifically, we examine and critique existing evaluation metrics, and introduce a set of new metrics in terms of fidelity, privacy, and utility to address their limitations. Based on the proposed metrics, we also devise a unified objective for tuning, which can consistently improve the quality of synthetic data for all methods. We conducted extensive evaluations of 8 different types of synthesizers on 12 datasets and identified some interesting findings, which offer new directions for privacy-preserving data synthesis.

Title: Discriminative Adversarial Unlearning

• Authors: Rohan Sharma, Shijie Zhou, Kaiyi Ji, Changyou Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06864
• Pdf URL: https://arxiv.org/pdf/2402.06864
• Copy Paste: [[2402.06864]] Discriminative Adversarial Unlearning(https://arxiv.org/abs/2402.06864)
• Keywords: self-supervised
• Abstract: We introduce a novel machine unlearning framework founded upon the established principles of the min-max optimization paradigm. We capitalize on the capabilities of strong Membership Inference Attacks (MIA) to facilitate the unlearning of specific samples from a trained model. We consider the scenario of two networks, the attacker $\mathbf{A}$ and the trained defender $\mathbf{D}$ pitted against each other in an adversarial objective, wherein the attacker aims at teasing out the information of the data to be unlearned in order to infer membership, and the defender unlearns to defend the network against the attack, whilst preserving its general performance. The algorithm can be trained end-to-end using backpropagation, following the well known iterative min-max approach in updating the attacker and the defender. We additionally incorporate a self-supervised objective effectively addressing the feature space discrepancies between the forget set and the validation set, enhancing unlearning performance. Our proposed algorithm closely approximates the ideal benchmark of retraining from scratch for both random sample forgetting and class-wise forgetting schemes on standard machine-unlearning datasets. Specifically, on the class unlearning scheme, the method demonstrates near-optimal performance and comprehensively overcomes known methods over the random sample forgetting scheme across all metrics and multiple network pruning strategies.

Title: GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators

• Authors: Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Dong Zhang, Zhehuai Chen, Eng Siong Chng
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.06894
• Pdf URL: https://arxiv.org/pdf/2402.06894
• Copy Paste: [[2402.06894]] GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators(https://arxiv.org/abs/2402.06894)
• Keywords: generative
• Abstract: Recent advances in large language models (LLMs) have stepped forward the development of multilingual speech and machine translation by its reduced representation errors and incorporated external knowledge. However, both translation tasks typically utilize beam search decoding and top-1 hypothesis selection for inference. These techniques struggle to fully exploit the rich information in the diverse N-best hypotheses, making them less optimal for translation tasks that require a single, high-quality output sequence. In this paper, we propose a new generative paradigm for translation tasks, namely "GenTranslate", which builds upon LLMs to generate better results from the diverse translation versions in N-best list. Leveraging the rich linguistic knowledge and strong reasoning abilities of LLMs, our new paradigm can integrate the rich information in N-best candidates to generate a higher-quality translation result. Furthermore, to support LLM finetuning, we build and release a HypoTranslate dataset that contains over 592K hypotheses-translation pairs in 11 languages. Experiments on various speech and machine translation benchmarks (e.g., FLEURS, CoVoST-2, WMT) demonstrate that our GenTranslate significantly outperforms the state-of-the-art model.

Title: SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data

• Authors: Hsuan-Fu Wang, Yi-Jen Shih, Heng-Jui Chang, Layne Berry, Puyuan Peng, Hung-yi Lee, Hsin-Min Wang, David Harwath
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.06959
• Pdf URL: https://arxiv.org/pdf/2402.06959
• Copy Paste: [[2402.06959]] SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data(https://arxiv.org/abs/2402.06959)
• Keywords: self-supervised
• Abstract: The recently proposed visually grounded speech model SpeechCLIP is an innovative framework that bridges speech and text through images via CLIP without relying on text transcription. On this basis, this paper introduces two extensions to SpeechCLIP. First, we apply the Continuous Integrate-and-Fire (CIF) module to replace a fixed number of CLS tokens in the cascaded architecture. Second, we propose a new hybrid architecture that merges the cascaded and parallel architectures of SpeechCLIP into a multi-task learning framework. Our experimental evaluation is performed on the Flickr8k and SpokenCOCO datasets. The results show that in the speech keyword extraction task, the CIF-based cascaded SpeechCLIP model outperforms the previous cascaded SpeechCLIP model using a fixed number of CLS tokens. Furthermore, through our hybrid architecture, cascaded task learning boosts the performance of the parallel branch in image-speech retrieval tasks.

Title: Synthesizing CTA Image Data for Type-B Aortic Dissection using Stable Diffusion Models

• Authors: Ayman Abaid, Muhammad Ali Farooq, Niamh Hynes, Peter Corcoran, Ihsan Ullah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06969
• Pdf URL: https://arxiv.org/pdf/2402.06969
• Copy Paste: [[2402.06969]] Synthesizing CTA Image Data for Type-B Aortic Dissection using Stable Diffusion Models(https://arxiv.org/abs/2402.06969)
• Keywords: diffusion, generative
• Abstract: Stable Diffusion (SD) has gained a lot of attention in recent years in the field of Generative AI thus helping in synthesizing medical imaging data with distinct features. The aim is to contribute to the ongoing effort focused on overcoming the limitations of data scarcity and improving the capabilities of ML algorithms for cardiovascular image processing. Therefore, in this study, the possibility of generating synthetic cardiac CTA images was explored by fine-tuning stable diffusion models based on user defined text prompts, using only limited number of CTA images as input. A comprehensive evaluation of the synthetic data was conducted by incorporating both quantitative analysis and qualitative assessment, where a clinician assessed the quality of the generated data. It has been shown that Cardiac CTA images can be successfully generated using using Text to Image (T2I) stable diffusion model. The results demonstrate that the tuned T2I CTA diffusion model was able to generate images with features that are typically unique to acute type B aortic dissection (TBAD) medical conditions.

Title: In-Context Data Distillation with TabPFN

• Authors: Junwei Ma, Valentin Thomas, Guangwei Yu, Anthony Caterini
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06971
• Pdf URL: https://arxiv.org/pdf/2402.06971
• Copy Paste: [[2402.06971]] In-Context Data Distillation with TabPFN(https://arxiv.org/abs/2402.06971)
• Keywords: foundation model, in-context
• Abstract: Foundation models have revolutionized tasks in computer vision and natural language processing. However, in the realm of tabular data, tree-based models like XGBoost continue to dominate. TabPFN, a transformer model tailored for tabular data, mirrors recent foundation models in its exceptional in-context learning capability, being competitive with XGBoost's performance without the need for task-specific training or hyperparameter tuning. Despite its promise, TabPFN's applicability is hindered by its data size constraint, limiting its use in real-world scenarios. To address this, we present in-context data distillation (ICD), a novel methodology that effectively eliminates these constraints by optimizing TabPFN's context. ICD efficiently enables TabPFN to handle significantly larger datasets with a fixed memory budget, improving TabPFN's quadratic memory complexity but at the cost of a linear number of tuning steps. Notably, TabPFN, enhanced with ICD, demonstrates very strong performance against established tree-based models and modern deep learning methods on 48 large tabular datasets from OpenML.

Title: Self-Correcting Self-Consuming Loops for Generative Model Training

• Authors: Nate Gillman, Michael Freeman, Daksh Aggarwal, Chia-Hong Hsu, Calvin Luo, Yonglong Tian, Chen Sun
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.07087
• Pdf URL: https://arxiv.org/pdf/2402.07087
• Copy Paste: [[2402.07087]] Self-Correcting Self-Consuming Loops for Generative Model Training(https://arxiv.org/abs/2402.07087)
• Keywords: generative
• Abstract: As synthetic data becomes higher quality and proliferates on the internet, machine learning models are increasingly trained on a mix of human- and machine-generated data. Despite the successful stories of using synthetic data for representation learning, using synthetic data for generative model training creates "self-consuming loops" which may lead to training instability or even collapse, unless certain conditions are met. Our paper aims to stabilize self-consuming generative model training. Our theoretical results demonstrate that by introducing an idealized correction function, which maps a data point to be more likely under the true data distribution, self-consuming loops can be made exponentially more stable. We then propose self-correction functions, which rely on expert knowledge (e.g. the laws of physics programmed in a simulator), and aim to approximate the idealized corrector automatically and at scale. We empirically validate the effectiveness of self-correcting self-consuming loops on the challenging human motion synthesis task, and observe that it successfully avoids model collapse, even when the ratio of synthetic data to real data is as high as 100%.

Title: A Benchmark for Multi-modal Foundation Models on Low-level Vision: from Single Images to Pairs

• Authors: Zicheng Zhang, Haoning Wu, Erli Zhang, Guangtao Zhai, Weisi Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.07116
• Pdf URL: https://arxiv.org/pdf/2402.07116
• Copy Paste: [[2402.07116]] A Benchmark for Multi-modal Foundation Models on Low-level Vision: from Single Images to Pairs(https://arxiv.org/abs/2402.07116)
• Keywords: foundation model
• Abstract: The rapid development of Multi-modality Large Language Models (MLLMs) has navigated a paradigm shift in computer vision, moving towards versatile foundational models. However, evaluating MLLMs in low-level visual perception and understanding remains a yet-to-explore domain. To this end, we design benchmark settings to emulate human language responses related to low-level vision: the low-level visual perception (A1) via visual question answering related to low-level attributes (e.g. clarity, lighting); and the low-level visual description (A2), on evaluating MLLMs for low-level text descriptions. Furthermore, given that pairwise comparison can better avoid ambiguity of responses and has been adopted by many human experiments, we further extend the low-level perception-related question-answering and description evaluations of MLLMs from single images to image pairs. Specifically, for perception (A1), we carry out the LLVisionQA+ dataset, comprising 2,990 single images and 1,999 image pairs each accompanied by an open-ended question about its low-level features; for description (A2), we propose the LLDescribe+ dataset, evaluating MLLMs for low-level descriptions on 499 single images and 450 pairs. Additionally, we evaluate MLLMs on assessment (A3) ability, i.e. predicting score, by employing a softmax-based approach to enable all MLLMs to generate quantifiable quality ratings, tested against human opinions in 7 image quality assessment (IQA) datasets. With 24 MLLMs under evaluation, we demonstrate that several MLLMs have decent low-level visual competencies on single images, but only GPT-4V exhibits higher accuracy on pairwise comparisons than single image evaluations (like humans). We hope that our benchmark will motivate further research into uncovering and enhancing these nascent capabilities of MLLMs. Datasets will be available at https://github.com/Q-Future/Q-Bench.

Title: Two-Stage Multi-task Self-Supervised Learning for Medical Image Segmentation

• Authors: Binyan Hu, A. K. Qin
• Subjects: cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.07119
• Pdf URL: https://arxiv.org/pdf/2402.07119
• Copy Paste: [[2402.07119]] Two-Stage Multi-task Self-Supervised Learning for Medical Image Segmentation(https://arxiv.org/abs/2402.07119)
• Keywords: self-supervised
• Abstract: Medical image segmentation has been significantly advanced by deep learning (DL) techniques, though the data scarcity inherent in medical applications poses a great challenge to DL-based segmentation methods. Self-supervised learning offers a solution by creating auxiliary learning tasks from the available dataset and then leveraging the knowledge acquired from solving auxiliary tasks to help better solve the target segmentation task. Different auxiliary tasks may have different properties and thus can help the target task to different extents. It is desired to leverage their complementary advantages to enhance the overall assistance to the target task. To achieve this, existing methods often adopt a joint training paradigm, which co-solves segmentation and auxiliary tasks by integrating their losses or intermediate gradients. However, direct coupling of losses or intermediate gradients risks undesirable interference because the knowledge acquired from solving each auxiliary task at every training step may not always benefit the target task. To address this issue, we propose a two-stage training approach. In the first stage, the target segmentation task will be independently co-solved with each auxiliary task in both joint training and pre-training modes, with the better model selected via validation performance. In the second stage, the models obtained with respect to each auxiliary task are converted into a single model using an ensemble knowledge distillation method. Our approach allows for making best use of each auxiliary task to create multiple elite segmentation models and then combine them into an even more powerful model. We employed five auxiliary tasks of different proprieties in our approach and applied it to train the U-Net model on an X-ray pneumothorax segmentation dataset. Experimental results demonstrate the superiority of our approach over several existing methods.

Title: An attempt to generate new bridge types from latent space of denoising diffusion Implicit model

• Authors: Hongjun Zhang
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.07129
• Pdf URL: https://arxiv.org/pdf/2402.07129
• Copy Paste: [[2402.07129]] An attempt to generate new bridge types from latent space of denoising diffusion Implicit model(https://arxiv.org/abs/2402.07129)
• Keywords: diffusion
• Abstract: Use denoising diffusion implicit model for bridge-type innovation. The process of adding noise and denoising to an image can be likened to the process of a corpse rotting and a detective restoring the scene of a victim being killed, to help beginners understand. Through an easy-to-understand algebraic method, derive the function formulas for adding noise and denoising, making it easier for beginners to master the mathematical principles of the model. Using symmetric structured image dataset of three-span beam bridge, arch bridge, cable-stayed bridge and suspension bridge , based on Python programming language, TensorFlow and Keras deep learning platform framework , denoising diffusion implicit model is constructed and trained. From the latent space sampling, new bridge types with asymmetric structures can be generated. Denoising diffusion implicit model can organically combine different structural components on the basis of human original bridge types, and create new bridge types.

Title: GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting

• Authors: Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang, Deqing Sun, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.07207
• Pdf URL: https://arxiv.org/pdf/2402.07207
• Copy Paste: [[2402.07207]] GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting(https://arxiv.org/abs/2402.07207)
• Keywords: diffusion, generative
• Abstract: We present GALA3D, generative 3D GAussians with LAyout-guided control, for effective compositional text-to-3D generation. We first utilize large language models (LLMs) to generate the initial layout and introduce a layout-guided 3D Gaussian representation for 3D content generation with adaptive geometric constraints. We then propose an object-scene compositional optimization mechanism with conditioned diffusion to collaboratively generate realistic 3D scenes with consistent geometry, texture, scale, and accurate interactions among multiple objects while simultaneously adjusting the coarse layout priors extracted from the LLMs to align with the generated scene. Experiments show that GALA3D is a user-friendly, end-to-end framework for state-of-the-art scene-level 3D content generation and controllable editing while ensuring the high fidelity of object-level entities within the scene. Source codes and models will be available at https://gala3d.github.io/.

Title: Towards Fast Stochastic Sampling in Diffusion Generative Models

• Authors: Kushagra Pandey, Maja Rudolph, Stephan Mandt
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.07211
• Pdf URL: https://arxiv.org/pdf/2402.07211
• Copy Paste: [[2402.07211]] Towards Fast Stochastic Sampling in Diffusion Generative Models(https://arxiv.org/abs/2402.07211)
• Keywords: diffusion, generative
• Abstract: Diffusion models suffer from slow sample generation at inference time. Despite recent efforts, improving the sampling efficiency of stochastic samplers for diffusion models remains a promising direction. We propose Splitting Integrators for fast stochastic sampling in pre-trained diffusion models in augmented spaces. Commonly used in molecular dynamics, splitting-based integrators attempt to improve sampling efficiency by cleverly alternating between numerical updates involving the data, auxiliary, or noise variables. However, we show that a naive application of splitting integrators is sub-optimal for fast sampling. Consequently, we propose several principled modifications to naive splitting samplers for improving sampling efficiency and denote the resulting samplers as Reduced Splitting Integrators. In the context of Phase Space Langevin Diffusion (PSLD) [Pandey \& Mandt, 2023] on CIFAR-10, our stochastic sampler achieves an FID score of 2.36 in only 100 network function evaluations (NFE) as compared to 2.63 for the best baselines.

Title: Rethinking Graph Masked Autoencoders through Alignment and Uniformity

• Authors: Liang Wang, Xiang Tao, Qiang Liu, Shu Wu, Liang Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.07225
• Pdf URL: https://arxiv.org/pdf/2402.07225
• Copy Paste: [[2402.07225]] Rethinking Graph Masked Autoencoders through Alignment and Uniformity(https://arxiv.org/abs/2402.07225)
• Keywords: self-supervised, generative
• Abstract: Self-supervised learning on graphs can be bifurcated into contrastive and generative methods. Contrastive methods, also known as graph contrastive learning (GCL), have dominated graph self-supervised learning in the past few years, but the recent advent of graph masked autoencoder (GraphMAE) rekindles the momentum behind generative methods. Despite the empirical success of GraphMAE, there is still a dearth of theoretical understanding regarding its efficacy. Moreover, while both generative and contrastive methods have been shown to be effective, their connections and differences have yet to be thoroughly investigated. Therefore, we theoretically build a bridge between GraphMAE and GCL, and prove that the node-level reconstruction objective in GraphMAE implicitly performs context-level GCL. Based on our theoretical analysis, we further identify the limitations of the GraphMAE from the perspectives of alignment and uniformity, which have been considered as two key properties of high-quality representations in GCL. We point out that GraphMAE's alignment performance is restricted by the masking strategy, and the uniformity is not strictly guaranteed. To remedy the aforementioned limitations, we propose an Alignment-Uniformity enhanced Graph Masked AutoEncoder, named AUG-MAE. Specifically, we propose an easy-to-hard adversarial masking strategy to provide hard-to-align samples, which improves the alignment performance. Meanwhile, we introduce an explicit uniformity regularizer to ensure the uniformity of the learned representations. Experimental results on benchmark datasets demonstrate the superiority of our model over existing state-of-the-art methods.

Title: TransGPT: Multi-modal Generative Pre-trained Transformer for Transportation

• Authors: Peng Wang, Xiang Wei, Fangxu Hu, Wenjuan Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.07233
• Pdf URL: https://arxiv.org/pdf/2402.07233
• Copy Paste: [[2402.07233]] TransGPT: Multi-modal Generative Pre-trained Transformer for Transportation(https://arxiv.org/abs/2402.07233)
• Keywords: generative
• Abstract: Natural language processing (NLP) is a key component of intelligent transportation systems (ITS), but it faces many challenges in the transportation domain, such as domain-specific knowledge and data, and multi-modal inputs and outputs. This paper presents TransGPT, a novel (multi-modal) large language model for the transportation domain, which consists of two independent variants: TransGPT-SM for single-modal data and TransGPT-MM for multi-modal data. TransGPT-SM is finetuned on a single-modal Transportation dataset (STD) that contains textual data from various sources in the transportation domain. TransGPT-MM is finetuned on a multi-modal Transportation dataset (MTD) that we manually collected from three areas of the transportation domain: driving tests, traffic signs, and landmarks. We evaluate TransGPT on several benchmark datasets for different tasks in the transportation domain, and show that it outperforms baseline models on most tasks. We also showcase the potential applications of TransGPT for traffic analysis and modeling, such as generating synthetic traffic scenarios, explaining traffic phenomena, answering traffic-related questions, providing traffic recommendations, and generating traffic reports. This work advances the state-of-the-art of NLP in the transportation domain and provides a useful tool for ITS researchers and practitioners.

Title: DIMON: Learning Solution Operators of Partial Differential Equations on a Diffeomorphic Family of Domains

• Authors: Minglang Yin, Nicolas Charon, Ryan Brody, Lu Lu, Natalia Trayanova, Mauro Maggioni
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2402.07250
• Pdf URL: https://arxiv.org/pdf/2402.07250
• Copy Paste: [[2402.07250]] DIMON: Learning Solution Operators of Partial Differential Equations on a Diffeomorphic Family of Domains(https://arxiv.org/abs/2402.07250)
• Keywords: diffusion
• Abstract: The solution of a PDE over varying initial/boundary conditions on multiple domains is needed in a wide variety of applications, but it is computationally expensive if the solution is computed de novo whenever the initial/boundary conditions of the domain change. We introduce a general operator learning framework, called DIffeomorphic Mapping Operator learNing (DIMON) to learn approximate PDE solutions over a family of domains $\{\Omega_{\theta}}_\theta$, that learns the map from initial/boundary conditions and domain $\Omega_\theta$ to the solution of the PDE, or to specified functionals thereof. DIMON is based on transporting a given problem (initial/boundary conditions and domain $\Omega_{\theta}$) to a problem on a reference domain $\Omega_{0}$, where training data from multiple problems is used to learn the map to the solution on $\Omega_{0}$, which is then re-mapped to the original domain $\Omega_{\theta}$. We consider several problems to demonstrate the performance of the framework in learning both static and time-dependent PDEs on non-rigid geometries; these include solving the Laplace equation, reaction-diffusion equations, and a multiscale PDE that characterizes the electrical propagation on the left ventricle. This work paves the way toward the fast prediction of PDE solutions on a family of domains and the application of neural operators in engineering and precision medicine.

Title: Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy

• Authors: Simon Ging, María A. Bravo, Thomas Brox
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.07270
• Pdf URL: https://arxiv.org/pdf/2402.07270
• Copy Paste: [[2402.07270]] Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy(https://arxiv.org/abs/2402.07270)
• Keywords: generative
• Abstract: The evaluation of text-generative vision-language models is a challenging yet crucial endeavor. By addressing the limitations of existing Visual Question Answering (VQA) benchmarks and proposing innovative evaluation methodologies, our research seeks to advance our understanding of these models' capabilities. We propose a novel VQA benchmark based on well-known visual classification datasets which allows a granular evaluation of text-generative vision-language models and their comparison with discriminative vision-language models. To improve the assessment of coarse answers on fine-grained classification tasks, we suggest using the semantic hierarchy of the label space to ask automatically generated follow-up questions about the ground-truth category. Finally, we compare traditional NLP and LLM-based metrics for the problem of evaluating model predictions given ground-truth answers. We perform a human evaluation study upon which we base our decision on the final metric. We apply our benchmark to a suite of vision-language models and show a detailed comparison of their abilities on object, action, and attribute classification. Our contributions aim to lay the foundation for more precise and meaningful assessments, facilitating targeted progress in the exciting field of vision-language modeling.

Title: Can Tree Based Approaches Surpass Deep Learning in Anomaly Detection? A Benchmarking Study

• Authors: Santonu Sarkar, Shanay Mehta, Nicole Fernandes, Jyotirmoy Sarkar, Snehanshu Saha
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.07281
• Pdf URL: https://arxiv.org/pdf/2402.07281
• Copy Paste: [[2402.07281]] Can Tree Based Approaches Surpass Deep Learning in Anomaly Detection? A Benchmarking Study(https://arxiv.org/abs/2402.07281)
• Keywords: anomaly
• Abstract: Detection of anomalous situations for complex mission-critical systems holds paramount importance when their service continuity needs to be ensured. A major challenge in detecting anomalies from the operational data arises due to the imbalanced class distribution problem since the anomalies are supposed to be rare events. This paper evaluates a diverse array of machine learning-based anomaly detection algorithms through a comprehensive benchmark study. The paper contributes significantly by conducting an unbiased comparison of various anomaly detection algorithms, spanning classical machine learning including various tree-based approaches to deep learning and outlier detection methods. The inclusion of 104 publicly available and a few proprietary industrial systems datasets enhances the diversity of the study, allowing for a more realistic evaluation of algorithm performance and emphasizing the importance of adaptability to real-world scenarios. The paper dispels the deep learning myth, demonstrating that though powerful, deep learning is not a universal solution in this case. We observed that recently proposed tree-based evolutionary algorithms outperform in many scenarios. We noticed that tree-based approaches catch a singleton anomaly in a dataset where deep learning methods fail. On the other hand, classical SVM performs the best on datasets with more than 10% anomalies, implying that such scenarios can be best modeled as a classification problem rather than anomaly detection. To our knowledge, such a study on a large number of state-of-the-art algorithms using diverse data sets, with the objective of guiding researchers and practitioners in making informed algorithmic choices, has not been attempted earlier.

Title: Assessing Generalization for Subpopulation Representative Modeling via In-Context Learning

• Authors: Gabriel Simmons, Vladislav Savinov
• Subjects: cs.LG, cs.AI, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.07368
• Pdf URL: https://arxiv.org/pdf/2402.07368
• Copy Paste: [[2402.07368]] Assessing Generalization for Subpopulation Representative Modeling via In-Context Learning(https://arxiv.org/abs/2402.07368)
• Keywords: in-context
• Abstract: This study evaluates the ability of Large Language Model (LLM)-based Subpopulation Representative Models (SRMs) to generalize from empirical data, utilizing in-context learning with data from the 2016 and 2020 American National Election Studies. We explore generalization across response variables and demographic subgroups. While conditioning with empirical data improves performance on the whole, the benefit of in-context learning varies considerably across demographics, sometimes hurting performance for one demographic while helping performance for others. The inequitable benefits of in-context learning for SRM present a challenge for practitioners implementing SRMs, and for decision-makers who might come to rely on them. Our work highlights a need for fine-grained benchmarks captured from diverse subpopulations that test not only fidelity but generalization.

Title: Diff-RNTraj: A Structure-aware Diffusion Model for Road Network-constrained Trajectory Generation

• Authors: Tonglong Wei, Youfang Lin, Shengnan Guo, Yan Lin, Yiheng Huang, Chenyang Xiang, Yuqing Bai, Menglu Ya, Huaiyu Wan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.07369
• Pdf URL: https://arxiv.org/pdf/2402.07369
• Copy Paste: [[2402.07369]] Diff-RNTraj: A Structure-aware Diffusion Model for Road Network-constrained Trajectory Generation(https://arxiv.org/abs/2402.07369)
• Keywords: diffusion
• Abstract: Trajectory data is essential for various applications as it records the movement of vehicles. However, publicly available trajectory datasets remain limited in scale due to privacy concerns, which hinders the development of trajectory data mining and trajectory-based applications. To address this issue, some methods for generating synthetic trajectories have been proposed to expand the scale of the dataset. However, all existing methods generate trajectories in the geographical coordinate system, which poses two limitations for their utilization in practical applications: 1) the inability to ensure that the generated trajectories are constrained on the road. 2) the lack of road-related information. In this paper, we propose a new problem to meet the practical application need, \emph{i.e.}, road network-constrained trajectory (RNTraj) generation, which can directly generate trajectories on the road network with road-related information. RNTraj is a hybrid type of data, in which each point is represented by a discrete road segment and a continuous moving rate. To generate RNTraj, we design a diffusion model called Diff-RNTraj. This model can effectively handle the hybrid RNTraj using a continuous diffusion framework by incorporating a pre-training strategy to embed hybrid RNTraj into continuous representations. During the sampling stage, a RNTraj decoder is designed to map the continuous representation generated by the diffusion model back to the hybrid RNTraj format. Furthermore, Diff-RNTraj introduces a novel loss function to enhance the spatial validity of the generated trajectories. Extensive experiments conducted on two real-world trajectory datasets demonstrate the effectiveness of the proposed model.

Title: SelfSwapper: Self-Supervised Face Swapping via Shape Agnostic Masked AutoEncoder

• Authors: Jaeseong Lee, Junha Hyung, Sohyun Jeong, Jaegul Choo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.07370
• Pdf URL: https://arxiv.org/pdf/2402.07370
• Copy Paste: [[2402.07370]] SelfSwapper: Self-Supervised Face Swapping via Shape Agnostic Masked AutoEncoder(https://arxiv.org/abs/2402.07370)
• Keywords: self-supervised
• Abstract: Face swapping has gained significant attention for its varied applications. The majority of previous face swapping approaches have relied on the seesaw game training scheme, which often leads to the instability of the model training and results in undesired samples with blended identities due to the target identity leakage problem. This paper introduces the Shape Agnostic Masked AutoEncoder (SAMAE) training scheme, a novel self-supervised approach designed to enhance face swapping model training. Our training scheme addresses the limitations of traditional training methods by circumventing the conventional seesaw game and introducing clear ground truth through its self-reconstruction training regime. It effectively mitigates identity leakage by masking facial regions of the input images and utilizing learned disentangled identity and non-identity features. Additionally, we tackle the shape misalignment problem with new techniques including perforation confusion and random mesh scaling, and establishes a new state-of-the-art, surpassing other baseline methods, preserving both identity and non-identity attributes, without sacrificing on either aspect.

Title: Chain-of-Layer: Iteratively Prompting Large Language Models for Taxonomy Induction from Limited Examples

• Authors: Qingkai Zeng, Yuyang Bai, Zhaoxuan Tan, Shangbin Feng, Zhenwen Liang, Zhihan Zhang, Meng Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.07386
• Pdf URL: https://arxiv.org/pdf/2402.07386
• Copy Paste: [[2402.07386]] Chain-of-Layer: Iteratively Prompting Large Language Models for Taxonomy Induction from Limited Examples(https://arxiv.org/abs/2402.07386)
• Keywords: in-context
• Abstract: Automatic taxonomy induction is crucial for web search, recommendation systems, and question answering. Manual curation of taxonomies is expensive in terms of human effort, making automatic taxonomy construction highly desirable. In this work, we introduce Chain-of-Layer which is an in-context learning framework designed to induct taxonomies from a given set of entities. Chain-of-Layer breaks down the task into selecting relevant candidate entities in each layer and gradually building the taxonomy from top to bottom. To minimize errors, we introduce the Ensemble-based Ranking Filter to reduce the hallucinated content generated at each iteration. Through extensive experiments, we demonstrate that Chain-of-Layer achieves state-of-the-art performance on four real-world benchmarks.

Title: Conditional Generative Models are Sufficient to Sample from Any Causal Effect Estimand

• Authors: Md Musfiqur Rahman, Matt Jordan, Murat Kocaoglu
• Subjects: cs.LG, cs.AI, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.07419
• Pdf URL: https://arxiv.org/pdf/2402.07419
• Copy Paste: [[2402.07419]] Conditional Generative Models are Sufficient to Sample from Any Causal Effect Estimand(https://arxiv.org/abs/2402.07419)
• Keywords: diffusion, generative
• Abstract: Causal inference from observational data has recently found many applications in machine learning. While sound and complete algorithms exist to compute causal effects, many of these algorithms require explicit access to conditional likelihoods over the observational distribution, which is difficult to estimate in the high-dimensional regime, such as with images. To alleviate this issue, researchers have approached the problem by simulating causal relations with neural models and obtained impressive results. However, none of these existing approaches can be applied to generic scenarios such as causal graphs on image data with latent confounders, or obtain conditional interventional samples. In this paper, we show that any identifiable causal effect given an arbitrary causal graph can be computed through push-forward computations of conditional generative models. Based on this result, we devise a diffusion-based approach to sample from any (conditional) interventional distribution on image data. To showcase our algorithm's performance, we conduct experiments on a Colored MNIST dataset having both the treatment ($X$) and the target variables ($Y$) as images and obtain interventional samples from $P(y|do(x))$. As an application of our algorithm, we evaluate two large conditional generative models that are pre-trained on the CelebA dataset by analyzing the strength of spurious correlations and the level of disentanglement they achieve.

Title: SALAD: Smart AI Language Assistant Daily

• Authors: Ragib Amin Nihal
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.07431
• Pdf URL: https://arxiv.org/pdf/2402.07431
• Copy Paste: [[2402.07431]] SALAD: Smart AI Language Assistant Daily(https://arxiv.org/abs/2402.07431)
• Keywords: diffusion
• Abstract: SALAD is an AI-driven language-learning application designed to help foreigners learn Japanese. It offers translations in Kanji-Kana-Romaji, speech recognition, translated audio, vocabulary tracking, grammar explanations, and songs generated from newly learned words. The app targets beginners and intermediate learners, aiming to make language acquisition more accessible and enjoyable. SALAD uses daily translations to enhance fluency and comfort in communication with native speakers. The primary objectives include effective Japanese language learning, user engagement, and progress tracking. A survey by us found that 39% of foreigners in Japan face discomfort in conversations with Japanese speakers. Over 60% of foreigners expressed confidence in SALAD's ability to enhance their Japanese language skills. The app uses large language models, speech recognition, and diffusion models to bridge the language gap and foster a more inclusive community in Japan.

Title: Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial

• Authors: Wenpin Tang, Hanyang Zhao
• Subjects: cs.LG, math.HO
• Abstract URL: https://arxiv.org/abs/2402.07487
• Pdf URL: https://arxiv.org/pdf/2402.07487
• Copy Paste: [[2402.07487]] Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial(https://arxiv.org/abs/2402.07487)
• Keywords: diffusion
• Abstract: This is an expository article on the score-based diffusion models, with a particular focus on the formulation via stochastic differential equations (SDE). After a gentle introduction, we discuss the two pillars in the diffusion modeling -- sampling and score matching, which encompass the SDE/ODE sampling, score matching efficiency, the consistency model, and reinforcement learning. Short proofs are given to illustrate the main idea of the stated results. The article is primarily for introducing the beginners to the field, and practitioners may also find some analysis useful in designing new models or algorithms.

Title: MAFIA: Multi-Adapter Fused Inclusive LanguAge Models

• Authors: Prachi Jain, Ashutosh Sathe, Varun Gumma, Kabir Ahuja, Sunayana Sitaram
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.07519
• Pdf URL: https://arxiv.org/pdf/2402.07519
• Copy Paste: [[2402.07519]] MAFIA: Multi-Adapter Fused Inclusive LanguAge Models(https://arxiv.org/abs/2402.07519)
• Keywords: generative
• Abstract: Pretrained Language Models (PLMs) are widely used in NLP for various tasks. Recent studies have identified various biases that such models exhibit and have proposed methods to correct these biases. However, most of the works address a limited set of bias dimensions independently such as gender, race, or religion. Moreover, the methods typically involve finetuning the full model to maintain the performance on the downstream task. In this work, we aim to modularly debias a pretrained language model across multiple dimensions. Previous works extensively explored debiasing PLMs using limited US-centric counterfactual data augmentation (CDA). We use structured knowledge and a large generative model to build a diverse CDA across multiple bias dimensions in a semi-automated way. We highlight how existing debiasing methods do not consider interactions between multiple societal biases and propose a debiasing model that exploits the synergy amongst various societal biases and enables multi-bias debiasing simultaneously. An extensive evaluation on multiple tasks and languages demonstrates the efficacy of our approach.

Title: Only the Curve Shape Matters: Training Foundation Models for Zero-Shot Multivariate Time Series Forecasting through Next Curve Shape Prediction

• Authors: Cheng Feng, Long Huang, Denis Krompass
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.07570
• Pdf URL: https://arxiv.org/pdf/2402.07570
• Copy Paste: [[2402.07570]] Only the Curve Shape Matters: Training Foundation Models for Zero-Shot Multivariate Time Series Forecasting through Next Curve Shape Prediction(https://arxiv.org/abs/2402.07570)
• Keywords: foundation model
• Abstract: We present General Time Transformer (GTT), an encoder-only style foundation model for zero-shot multivariate time series forecasting. GTT is pretrained on a large dataset of 200M high-quality time series samples spanning diverse domains. In our proposed framework, the task of multivariate time series forecasting is formulated as a channel-wise next curve shape prediction problem, where each time series sample is represented as a sequence of non-overlapping curve shapes with a unified numerical magnitude. GTT is trained to predict the next curve shape based on a window of past curve shapes in a channel-wise manner. Experimental results demonstrate that GTT exhibits superior zero-shot multivariate forecasting capabilities on unseen time series datasets, even surpassing state-of-the-art supervised baselines. Additionally, we investigate the impact of varying GTT model parameters and training dataset scales, observing that the scaling law also holds in the context of zero-shot multivariate time series forecasting.

Title: Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model

• Authors: Mark Rowland, Li Kevin Wenliang, Rémi Munos, Clare Lyle, Yunhao Tang, Will Dabney
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.07598
• Pdf URL: https://arxiv.org/pdf/2402.07598
• Copy Paste: [[2402.07598]] Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model(https://arxiv.org/abs/2402.07598)
• Keywords: generative
• Abstract: We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions with a generative model (up to logarithmic factors), resolving an open question of Zhang et al. (2023). Our analysis provides new theoretical results on categorical approaches to distributional RL, and also introduces a new distributional Bellman equation, the stochastic categorical CDF Bellman equation, which we expect to be of independent interest. We also provide an experimental study comparing several model-based distributional RL algorithms, with several takeaways for practitioners.

Title: Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping

• Authors: Haoyu Wang, Guozheng Ma, Ziqiao Meng, Zeyu Qin, Li Shen, Zhong Zhang, Bingzhe Wu, Liu Liu, Yatao Bian, Tingyang Xu, Xueqian Wang, Peilin Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.07610
• Pdf URL: https://arxiv.org/pdf/2402.07610
• Copy Paste: [[2402.07610]] Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping(https://arxiv.org/abs/2402.07610)
• Keywords: in-context
• Abstract: Self-alignment is an effective way to reduce the cost of human annotation while ensuring promising model capability. However, most current methods complete the data collection and training steps in a single round, which may overlook the continuously improving ability of self-aligned models. This gives rise to a key query: What if we do multi-time bootstrapping self-alignment? Does this strategy enhance model performance or lead to rapid degradation? In this paper, our pioneering exploration delves into the impact of bootstrapping self-alignment on large language models. Our findings reveal that bootstrapping self-alignment markedly surpasses the single-round approach, by guaranteeing data diversity from in-context learning. To further exploit the capabilities of bootstrapping, we investigate and adjust the training order of data, which yields improved performance of the model. Drawing on these findings, we propose Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced few-shot ability to boost zero or one-shot performance. Based on easy-to-hard training recipe, we propose SOFT+ which further boost self-alignment's performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across various classification and generation tasks, highlighting the potential of bootstrapping self-alignment on continually enhancing model alignment performance.

Title: Universal link predictor by In-context Learning

• Authors: Kaiwen Dong, Haitao Mao, Zhichun Guo, Nitesh V. Chawla
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.07738
• Pdf URL: https://arxiv.org/pdf/2402.07738
• Copy Paste: [[2402.07738]] Universal link predictor by In-context Learning(https://arxiv.org/abs/2402.07738)
• Keywords: in-context
• Abstract: Link prediction is a crucial task in graph machine learning, where the goal is to infer missing or future links within a graph. Traditional approaches leverage heuristic methods based on widely observed connectivity patterns, offering broad applicability and generalizability without the need for model training. Despite their utility, these methods are limited by their reliance on human-derived heuristics and lack the adaptability of data-driven approaches. Conversely, parametric link predictors excel in automatically learning the connectivity patterns from data and achieving state-of-the-art but fail short to directly transfer across different graphs. Instead, it requires the cost of extensive training and hyperparameter optimization to adapt to the target graph. In this work, we introduce the Universal Link Predictor (UniLP), a novel model that combines the generalizability of heuristic approaches with the pattern learning capabilities of parametric models. UniLP is designed to autonomously identify connectivity patterns across diverse graphs, ready for immediate application to any unseen graph dataset without targeted training. We address the challenge of conflicting connectivity patterns-arising from the unique distributions of different graphs-through the implementation of In-context Learning (ICL). This approach allows UniLP to dynamically adjust to various target graphs based on contextual demonstrations, thereby avoiding negative transfer. Through rigorous experimentation, we demonstrate UniLP's effectiveness in adapting to new, unseen graphs at test time, showcasing its ability to perform comparably or even outperform parametric models that have been finetuned for specific datasets. Our findings highlight UniLP's potential to set a new standard in link prediction, combining the strengths of heuristic and parametric methods in a single, versatile framework.

Title: Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search

• Authors: Yifei Yuan, Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke, Wai Lam
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.07742
• Pdf URL: https://arxiv.org/pdf/2402.07742
• Copy Paste: [[2402.07742]] Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search(https://arxiv.org/abs/2402.07742)
• Keywords: generative
• Abstract: In mixed-initiative conversational search systems, clarifying questions are used to help users who struggle to express their intentions in a single query. These questions aim to uncover user's information needs and resolve query ambiguities. We hypothesize that in scenarios where multimodal information is pertinent, the clarification process can be improved by using non-textual information. Therefore, we propose to add images to clarifying questions and formulate the novel task of asking multimodal clarifying questions in open-domain, mixed-initiative conversational search systems. To facilitate research into this task, we collect a dataset named Melon that contains over 4k multimodal clarifying questions, enriched with over 14k images. We also propose a multimodal query clarification model named Marto and adopt a prompt-based, generative fine-tuning strategy to perform the training of different stages with different prompts. Several analyses are conducted to understand the importance of multimodal contents during the query clarification phase. Experimental results indicate that the addition of images leads to significant improvements of up to 90% in retrieval performance when selecting the relevant images. Extensive analyses are also performed to show the superiority of Marto compared with discriminative baselines in terms of effectiveness and efficiency.

Title: Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models

• Authors: Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Zhenguo Li, Wei Bi, Lingpeng Kong
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.07754
• Pdf URL: https://arxiv.org/pdf/2402.07754
• Copy Paste: [[2402.07754]] Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models(https://arxiv.org/abs/2402.07754)
• Keywords: diffusion
• Abstract: Diffusion models have gained attention in text processing, offering many potential advantages over traditional autoregressive models. This work explores the integration of diffusion models and Chain-of-Thought (CoT), a well-established technique to improve the reasoning ability in autoregressive language models. We propose Diffusion-of-Thought (DoT), allowing reasoning steps to diffuse over time through the diffusion process. In contrast to traditional autoregressive language models that make decisions in a left-to-right, token-by-token manner, DoT offers more flexibility in the trade-off between computation and reasoning performance. Our experimental results demonstrate the effectiveness of DoT in multi-digit multiplication and grade school math problems. Additionally, DoT showcases promising self-correction abilities and benefits from existing reasoning-enhancing techniques like self-consistency decoding. Our findings contribute to the understanding and development of reasoning capabilities in diffusion language models.

Title: Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model

• Authors: Mikail Khona, Maya Okawa, Jan Hula, Rahul Ramesh, Kento Nishi, Robert Dick, Ekdeep Singh Lubana, Hidenori Tanaka
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.07757
• Pdf URL: https://arxiv.org/pdf/2402.07757
• Copy Paste: [[2402.07757]] Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model(https://arxiv.org/abs/2402.07757)
• Keywords: in-context
• Abstract: Stepwise inference protocols, such as scratchpads and chain-of-thought, help language models solve complex problems by decomposing them into a sequence of simpler subproblems. Despite the significant gain in performance achieved via these protocols, the underlying mechanisms of stepwise inference have remained elusive. To address this, we propose to study autoregressive Transformer models on a synthetic task that embodies the multi-step nature of problems where stepwise inference is generally most useful. Specifically, we define a graph navigation problem wherein a model is tasked with traversing a path from a start to a goal node on the graph. Despite is simplicity, we find we can empirically reproduce and analyze several phenomena observed at scale: (i) the stepwise inference reasoning gap, the cause of which we find in the structure of the training data; (ii) a diversity-accuracy tradeoff in model generations as sampling temperature varies; (iii) a simplicity bias in the model's output; and (iv) compositional generalization and a primacy bias with in-context exemplars. Overall, our work introduces a grounded, synthetic framework for studying stepwise inference and offers mechanistic hypotheses that can lay the foundation for a deeper understanding of this phenomenon.

Title: Multi-Intent Attribute-Aware Text Matching in Searching

• Authors: Mingzhe Li, Xiuying Chen, Jing Xiang, Qishen Zhang, Changsheng Ma, Chenchen Dai, Jinxiong Chang, Zhongyi Liu, Guannan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.07788
• Pdf URL: https://arxiv.org/pdf/2402.07788
• Copy Paste: [[2402.07788]] Multi-Intent Attribute-Aware Text Matching in Searching(https://arxiv.org/abs/2402.07788)
• Keywords: self-supervised
• Abstract: Text matching systems have become a fundamental service in most searching platforms. For instance, they are responsible for matching user queries to relevant candidate items, or rewriting the user-input query to a pre-selected high-performing one for a better search experience. In practice, both the queries and items often contain multiple attributes, such as the category of the item and the location mentioned in the query, which represent condensed key information that is helpful for matching. However, most of the existing works downplay the effectiveness of attributes by integrating them into text representations as supplementary information. Hence, in this work, we focus on exploring the relationship between the attributes from two sides. Since attributes from two ends are often not aligned in terms of number and type, we propose to exploit the benefit of attributes by multiple-intent modeling. The intents extracted from attributes summarize the diverse needs of queries and provide rich content of items, which are more refined and abstract, and can be aligned for paired inputs. Concretely, we propose a multi-intent attribute-aware matching model (MIM), which consists of three main components: attribute-aware encoder, multi-intent modeling, and intent-aware matching. In the attribute-aware encoder, the text and attributes are weighted and processed through a scaled attention mechanism with regard to the attributes' importance. Afterward, the multi-intent modeling extracts intents from two ends and aligns them. Herein, we come up with a distribution loss to ensure the learned intents are diverse but concentrated, and a kullback-leibler divergence loss that aligns the learned intents. Finally, in the intent-aware matching, the intents are evaluated by a self-supervised masking task, and then incorporated to output the final matching result.

Title: Retrieval-Augmented Thought Process as Sequential Decision Making

• Authors: Thomas Pouplin, Hao Sun, Samuel Holt, Mihaela van der Schaar
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.07812
• Pdf URL: https://arxiv.org/pdf/2402.07812
• Copy Paste: [[2402.07812]] Retrieval-Augmented Thought Process as Sequential Decision Making(https://arxiv.org/abs/2402.07812)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have demonstrated their strong ability to assist people and show "sparks of intelligence". However, several open challenges hinder their wider application: such as concerns over privacy, tendencies to produce hallucinations, and difficulties in handling long contexts. In this work, we address those challenges by introducing the Retrieval-Augmented Thought Process (RATP). Given access to external knowledge, RATP formulates the thought generation of LLMs as a multiple-step decision process. To optimize such a thought process, RATP leverages Monte-Carlo Tree Search, and learns a Q-value estimator that permits cost-efficient inference. In addressing the task of question-answering with private data, where ethical and security concerns limit LLM training methods, RATP achieves a 50% improvement over existing in-context retrieval-augmented language models.

Title: Injecting Wiktionary to improve token-level contextual representations using contrastive learning

• Authors: Anna Mosolova, Marie Candito, Carlos Ramisch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.07817
• Pdf URL: https://arxiv.org/pdf/2402.07817
• Copy Paste: [[2402.07817]] Injecting Wiktionary to improve token-level contextual representations using contrastive learning(https://arxiv.org/abs/2402.07817)
• Keywords: in-context
• Abstract: While static word embeddings are blind to context, for lexical semantics tasks context is rather too present in contextual word embeddings, vectors of same-meaning occurrences being too different (Ethayarajh, 2019). Fine-tuning pre-trained language models (PLMs) using contrastive learning was proposed, leveraging automatically self-augmented examples (Liu et al., 2021b). In this paper, we investigate how to inject a lexicon as an alternative source of supervision, using the English Wiktionary. We also test how dimensionality reduction impacts the resulting contextual word embeddings. We evaluate our approach on the Word-In-Context (WiC) task, in the unsupervised setting (not using the training set). We achieve new SoTA result on the original WiC test set. We also propose two new WiC test sets for which we show that our fine-tuning method achieves substantial improvements. We also observe improvements, although modest, for the semantic frame induction task. Although we experimented on English to allow comparison with related work, our method is adaptable to the many languages for which large Wiktionaries exist.

Title: Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model

• Authors: Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, Sara Hooker
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.07827
• Pdf URL: https://arxiv.org/pdf/2402.07827
• Copy Paste: [[2402.07827]] Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model(https://arxiv.org/abs/2402.07827)
• Keywords: generative
• Abstract: Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages. What does it take to broaden access to breakthroughs beyond first-class citizen languages? Our work introduces Aya, a massively multilingual generative language model that follows instructions in 101 languages of which over 50% are considered as lower-resourced. Aya outperforms mT0 and BLOOMZ on the majority of tasks while covering double the number of languages. We introduce extensive new evaluation suites that broaden the state-of-art for multilingual eval across 99 languages -- including discriminative and generative tasks, human evaluation, and simulated win rates that cover both held-out tasks and in-distribution performance. Furthermore, we conduct detailed investigations on the optimal finetuning mixture composition, data pruning, as well as the toxicity, bias, and safety of our models. We open-source our instruction datasets and our model at https://hf.co/CohereForAI/aya-101

Title: Generative Modeling of Discrete Joint Distributions by E-Geodesic Flow Matching on Assignment Manifolds

• Authors: Bastian Boll, Daniel Gonzalez-Alvarado, Christoph Schnörr
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.07846
• Pdf URL: https://arxiv.org/pdf/2402.07846
• Copy Paste: [[2402.07846]] Generative Modeling of Discrete Joint Distributions by E-Geodesic Flow Matching on Assignment Manifolds(https://arxiv.org/abs/2402.07846)
• Keywords: generative
• Abstract: This paper introduces a novel generative model for discrete distributions based on continuous normalizing flows on the submanifold of factorizing discrete measures. Integration of the flow gradually assigns categories and avoids issues of discretizing the latent continuous model like rounding, sample truncation etc. General non-factorizing discrete distributions capable of representing complex statistical dependencies of structured discrete data, can be approximated by embedding the submanifold into a the meta-simplex of all joint discrete distributions and data-driven averaging. Efficient training of the generative model is demonstrated by matching the flow of geodesics of factorizing discrete distributions. Various experiments underline the approach's broad applicability.

Title: PoisonedRAG: Knowledge Poisoning Attacks to Retrieval-Augmented Generation of Large Language Models

• Authors: Wei Zou, Runpeng Geng, Binghui Wang, Jinyuan Jia
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.07867
• Pdf URL: https://arxiv.org/pdf/2402.07867
• Copy Paste: [[2402.07867]] PoisonedRAG: Knowledge Poisoning Attacks to Retrieval-Augmented Generation of Large Language Models(https://arxiv.org/abs/2402.07867)
• Keywords: generative
• Abstract: Large language models (LLMs) have achieved remarkable success due to their exceptional generative capabilities. Despite their success, they also have inherent limitations such as a lack of up-to-date knowledge and hallucination. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate those limitations. In particular, given a question, RAG retrieves relevant knowledge from a knowledge database to augment the input of the LLM. For instance, the retrieved knowledge could be a set of top-k texts that are most semantically similar to the given question when the knowledge database contains millions of texts collected from Wikipedia. As a result, the LLM could utilize the retrieved knowledge as the context to generate an answer for the given question. Existing studies mainly focus on improving the accuracy or efficiency of RAG, leaving its security largely unexplored. We aim to bridge the gap in this work. Particularly, we propose PoisonedRAG , a set of knowledge poisoning attacks to RAG, where an attacker could inject a few poisoned texts into the knowledge database such that the LLM generates an attacker-chosen target answer for an attacker-chosen target question. We formulate knowledge poisoning attacks as an optimization problem, whose solution is a set of poisoned texts. Depending on the background knowledge (e.g., black-box and white-box settings) of an attacker on the RAG, we propose two solutions to solve the optimization problem, respectively. Our results on multiple benchmark datasets and LLMs show our attacks could achieve 90% attack success rates when injecting 5 poisoned texts for each target question into a database with millions of texts. We also evaluate recent defenses and our results show they are insufficient to defend against our attacks, highlighting the need for new defenses.