2024-02-21

Title: Turn Waste into Worth: Rectifying Top-$k$ Router of MoE

• Authors: Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei, Zhangyue Yin, Yunhua Zhou, Linyang Li, Tianxiang Sun, Hang Yan, Dahua Lin, Xipeng Qiu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12399
• Pdf URL: https://arxiv.org/pdf/2402.12399
• Copy Paste: [[2402.12399]] Turn Waste into Worth: Rectifying Top-$k$ Router of MoE(https://arxiv.org/abs/2402.12399)
• Keywords: large language model
• Abstract: Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top-$k$ routing mechanism suffers from redundancy computation and memory costs due to the unbalanced routing. Some experts are overflow, where the exceeding tokens are dropped. While some experts are vacant, which are padded with zeros, negatively impacting model performance. To address the dropped tokens and padding, we propose the Rectify-Router, comprising the Intra-GPU Rectification and the Fill-in Rectification. The Intra-GPU Rectification handles dropped tokens, efficiently routing them to experts within the GPU where they are located to avoid inter-GPU communication. The Fill-in Rectification addresses padding by replacing padding tokens with the tokens that have high routing scores. Our experimental results demonstrate that the Intra-GPU Rectification and the Fill-in Rectification effectively handle dropped tokens and padding, respectively. Furthermore, the combination of them achieves superior performance, surpassing the accuracy of the vanilla top-1 router by 4.7%.

Title: Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation

• Authors: Hyunjune Shin, Dong-Wan Choi
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12406
• Pdf URL: https://arxiv.org/pdf/2402.12406
• Copy Paste: [[2402.12406]] Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation(https://arxiv.org/abs/2402.12406)
• Keywords: robust, data-free
• Abstract: Data-free knowledge distillation (DFKD) aims to distill pretrained knowledge to a student model with the help of a generator without using original data. In such data-free scenarios, achieving stable performance of DFKD is essential due to the unavailability of validation data. Unfortunately, this paper has discovered that existing DFKD methods are quite sensitive to different teacher models, occasionally showing catastrophic failures of distillation, even when using well-trained teacher models. Our observation is that the generator in DFKD is not always guaranteed to produce precise yet diverse samples using the existing representative strategy of minimizing both class-prior and adversarial losses. Through our empirical study, we focus on the fact that class-prior not only decreases the diversity of generated samples, but also cannot completely address the problem of generating unexpectedly low-quality samples depending on teacher models. In this paper, we propose the teacher-agnostic data-free knowledge distillation (TA-DFKD) method, with the goal of more robust and stable performance regardless of teacher models. Our basic idea is to assign the teacher model a lenient expert role for evaluating samples, rather than a strict supervisor that enforces its class-prior on the generator. Specifically, we design a sample selection approach that takes only clean samples verified by the teacher model without imposing restrictions on the power of generating diverse samples. Through extensive experiments, we show that our method successfully achieves both robustness and training stability across various teacher models, while outperforming the existing DFKD methods.

Title: ModelGPT: Unleashing LLM's Capabilities for Tailored Model Generation

• Authors: Zihao Tang, Zheqi Lv, Shengyu Zhang, Fei Wu, Kun Kuang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12408
• Pdf URL: https://arxiv.org/pdf/2402.12408
• Copy Paste: [[2402.12408]] ModelGPT: Unleashing LLM's Capabilities for Tailored Model Generation(https://arxiv.org/abs/2402.12408)
• Keywords: large language model
• Abstract: The rapid advancement of Large Language Models (LLMs) has revolutionized various sectors by automating routine tasks, marking a step toward the realization of Artificial General Intelligence (AGI). However, they still struggle to accommodate the diverse and specific needs of users and simplify the utilization of AI models for the average user. In response, we propose ModelGPT, a novel framework designed to determine and generate AI models specifically tailored to the data or task descriptions provided by the user, leveraging the capabilities of LLMs. Given user requirements, ModelGPT is able to provide tailored models at most 270x faster than the previous paradigms (e.g. all-parameter or LoRA finetuning). Comprehensive experiments on NLP, CV, and Tabular datasets attest to the effectiveness of our framework in making AI models more accessible and user-friendly. Our code is available at https://github.com/IshiKura-a/ModelGPT.

Title: Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architectures

• Authors: Akash Guna R.T, Arnav Chavan, Deepak Gupta
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.12418
• Pdf URL: https://arxiv.org/pdf/2402.12418
• Copy Paste: [[2402.12418]] Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architectures(https://arxiv.org/abs/2402.12418)
• Keywords: transformer
• Abstract: Conventional scaling of neural networks typically involves designing a base network and growing different dimensions like width, depth, etc. of the same by some predefined scaling factors. We introduce an automated scaling approach leveraging second-order loss landscape information. Our method is flexible towards skip connections a mainstay in modern vision transformers. Our training-aware method jointly scales and trains transformers without additional training iterations. Motivated by the hypothesis that not all neurons need uniform depth complexity, our approach embraces depth heterogeneity. Extensive evaluations on DeiT-S with ImageNet100 show a 2.5% accuracy gain and 10% parameter efficiency improvement over conventional scaling. Scaled networks demonstrate superior performance upon training small scale datasets from scratch. We introduce the first intact scaling mechanism for vision transformers, a step towards efficient model scaling.

Title: The (R)Evolution of Multimodal Large Language Models: A Survey

• Authors: Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara
• Subjects: cs.CV, cs.AI, cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.12451
• Pdf URL: https://arxiv.org/pdf/2402.12451
• Copy Paste: [[2402.12451]] The (R)Evolution of Multimodal Large Language Models: A Survey(https://arxiv.org/abs/2402.12451)
• Keywords: generative, large language model
• Abstract: Connecting text and visual modalities plays an essential role in generative intelligence. For this reason, inspired by the success of large language models, significant research efforts are being devoted to the development of Multimodal Large Language Models (MLLMs). These models can seamlessly integrate visual and textual modalities, both as input and output, while providing a dialogue-based interface and instruction-following capabilities. In this paper, we provide a comprehensive review of recent visual-based MLLMs, analyzing their architectural choices, multimodal alignment strategies, and training techniques. We also conduct a detailed analysis of these models across a wide range of tasks, including visual grounding, image generation and editing, visual understanding, and domain-specific applications. Additionally, we compile and describe training datasets and evaluation benchmarks, conducting comparisons among existing models in terms of performance and computational requirements. Overall, this survey offers a comprehensive overview of the current state of the art, laying the groundwork for future MLLMs.

Title: Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?

• Authors: Nishant Balepur, Abhilasha Ravichander, Rachel Rudinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12483
• Pdf URL: https://arxiv.org/pdf/2402.12483
• Copy Paste: [[2402.12483]] Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?(https://arxiv.org/abs/2402.12483)
• Keywords: robust, large language model
• Abstract: Multiple-choice question answering (MCQA) is often used to evaluate large language models (LLMs). To see if MCQA assesses LLMs as intended, we probe if LLMs can perform MCQA with choices-only prompts, where models must select the correct answer only from the choices. In three MCQA datasets and four LLMs, this prompt bests a majority baseline in 11/12 cases, with up to 0.33 accuracy gain. To help explain this behavior, we conduct an in-depth, black-box analysis on memorization, choice dynamics, and question inference. Our key findings are threefold. First, we find no evidence that the choices-only accuracy stems from memorization alone. Second, priors over individual choices do not fully explain choices-only accuracy, hinting that LLMs use the group dynamics of choices. Third, LLMs have some ability to infer a relevant question from choices, and surprisingly can sometimes even match the original question. We hope to motivate the use of stronger baselines in MCQA benchmarks, the design of robust MCQA datasets, and further efforts to explain LLM decision-making.

Title: Integrating kNN with Foundation Models for Adaptable and Privacy-Aware Image Classification

• Authors: Sebastian Doerrich, Tobias Archut, Francesco Di Salvo, Christian Ledig
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.12500
• Pdf URL: https://arxiv.org/pdf/2402.12500
• Copy Paste: [[2402.12500]] Integrating kNN with Foundation Models for Adaptable and Privacy-Aware Image Classification(https://arxiv.org/abs/2402.12500)
• Keywords: privacy, robust, interpretability
• Abstract: Traditional deep learning models implicity encode knowledge limiting their transparency and ability to adapt to data changes. Yet, this adaptability is vital for addressing user data privacy concerns. We address this limitation by storing embeddings of the underlying training data independently of the model weights, enabling dynamic data modifications without retraining. Specifically, our approach integrates the $k$-Nearest Neighbor ($k$-NN) classifier with a vision-based foundation model, pre-trained self-supervised on natural images, enhancing interpretability and adaptability. We share open-source implementations of a previously unpublished baseline method as well as our performance-improving contributions. Quantitative experiments confirm improved classification across established benchmark datasets and the method's applicability to distinct medical image classification tasks. Additionally, we assess the method's robustness in continual learning and data removal scenarios. The approach exhibits great promise for bridging the gap between foundation models' performance and challenges tied to data privacy. The source code is available at https://github.com/TobArc/privacy-aware-image-classification-with-kNN.

Title: Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection

• Authors: Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, Heng Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12501
• Pdf URL: https://arxiv.org/pdf/2402.12501
• Copy Paste: [[2402.12501]] Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection(https://arxiv.org/abs/2402.12501)
• Keywords: large language model
• Abstract: Data selection in instruction tuning emerges as a pivotal process for acquiring high-quality data and training instruction-following large language models (LLMs), but it is still a new and unexplored research area for vision-language models (VLMs). Existing data selection approaches on LLMs either rely on single unreliable scores, or use downstream tasks for selection, which is time-consuming and can lead to potential over-fitting on the chosen evaluation datasets. To address this challenge, we introduce a novel dataset selection method, Self-Filter, that utilizes the VLM itself as a filter. This approach is inspired by the observation that VLMs benefit from training with the most challenging instructions. Self-Filter operates in two stages. In the first stage, we devise a scoring network to evaluate the difficulty of training instructions, which is co-trained with the VLM. In the second stage, we use the trained score net to measure the difficulty of each instruction, select the most challenging samples, and penalize similar samples to encourage diversity. Comprehensive experiments on LLaVA and MiniGPT-4 show that Self-Filter can reach better results compared to full data settings with merely about 15% samples, and can achieve superior performance against competitive baselines.

Title: PARCv2: Physics-aware Recurrent Convolutional Neural Networks for Spatiotemporal Dynamics Modeling

• Authors: Phong C.H. Nguyen, Xinlun Cheng, Shahab Arfaza, Pradeep Seshadri, Yen T. Nguyen, Munho Kim, Sanghun Choi, H.S. Udaykumar, Stephen Baek
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12503
• Pdf URL: https://arxiv.org/pdf/2402.12503
• Copy Paste: [[2402.12503]] PARCv2: Physics-aware Recurrent Convolutional Neural Networks for Spatiotemporal Dynamics Modeling(https://arxiv.org/abs/2402.12503)
• Keywords: diffusion
• Abstract: Modeling unsteady, fast transient, and advection-dominated physics problems is a pressing challenge for physics-aware deep learning (PADL). The physics of complex systems is governed by large systems of partial differential equations (PDEs) and ancillary constitutive models with nonlinear structures, as well as evolving state fields exhibiting sharp gradients and rapidly deforming material interfaces. Here, we investigate an inductive bias approach that is versatile and generalizable to model generic nonlinear field evolution problems. Our study focuses on the recent physics-aware recurrent convolutions (PARC), which incorporates a differentiator-integrator architecture that inductively models the spatiotemporal dynamics of generic physical systems. We extend the capabilities of PARC to simulate unsteady, transient, and advection-dominant systems. The extended model, referred to as PARCv2, is equipped with differential operators to model advection-reaction-diffusion equations, as well as a hybrid integral solver for stable, long-time predictions. PARCv2 is tested on both standard benchmark problems in fluid dynamics, namely Burgers and Navier-Stokes equations, and then applied to more complex shock-induced reaction problems in energetic materials. We evaluate the behavior of PARCv2 in comparison to other physics-informed and learning bias models and demonstrate its potential to model unsteady and advection-dominant dynamics regimes.

Title: Induced Model Matching: How Restricted Models Can Help Larger Ones

• Authors: Usama Muneeb, Mesrob I. Ohannessian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12513
• Pdf URL: https://arxiv.org/pdf/2402.12513
• Copy Paste: [[2402.12513]] Induced Model Matching: How Restricted Models Can Help Larger Ones(https://arxiv.org/abs/2402.12513)
• Keywords: transformer
• Abstract: We consider scenarios where a very accurate predictive model using restricted features is available at the time of training of a larger, full-featured, model. This restricted model may be thought of as "side-information", derived either from an auxiliary exhaustive dataset or on the same dataset, by forcing the restriction. How can the restricted model be useful to the full model? We propose an approach for transferring the knowledge of the restricted model to the full model, by aligning the full model's context-restricted performance with that of the restricted model's. We call this methodology Induced Model Matching (IMM) and first illustrate its general applicability by using logistic regression as a toy example. We then explore IMM's use in language modeling, the application that initially inspired it, and where it offers an explicit foundation in contrast to the implicit use of restricted models in techniques such as noising. We demonstrate the methodology on both LSTM and transformer full models, using $N$-grams as restricted models. To further illustrate the potential of the principle whenever it is much cheaper to collect restricted rather than full information, we conclude with a simple RL example where POMDP policies can improve learned MDP policies via IMM.

Title: System Identification of Neural Systems: Going Beyond Images to Modelling Dynamics

• Authors: Mai Gamal, Mohamed Rashad, Eman Ehab, Seif Eldawlatly, Mennatullah Siam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12519
• Pdf URL: https://arxiv.org/pdf/2402.12519
• Copy Paste: [[2402.12519]] System Identification of Neural Systems: Going Beyond Images to Modelling Dynamics(https://arxiv.org/abs/2402.12519)
• Keywords: transformer
• Abstract: Vast literature has compared the recordings of biological neurons in the brain to deep neural networks. The ultimate goal is to interpret deep networks or to better understand and encode biological neural systems. Recently, there has been a debate on whether system identification is possible and how much it can tell us about the brain computation. System identification recognizes whether one model is more valid to represent the brain computation over another. Nonetheless, previous work did not consider the time aspect and how video and dynamics (e.g., motion) modelling in deep networks relate to these biological neural systems within a large-scale comparison. Towards this end, we propose a system identification study focused on comparing single image vs. video understanding models with respect to the visual cortex recordings. Our study encompasses two sets of experiments; a real environment setup and a simulated environment setup. The study also encompasses more than 30 models and, unlike prior works, we focus on convolutional vs. transformer-based, single vs. two-stream, and fully vs. self-supervised video understanding models. The goal is to capture a greater variety of architectures that model dynamics. As such, this signifies the first large-scale study of video understanding models from a neuroscience perspective. Our results in the simulated experiments, show that system identification can be attained to a certain level in differentiating image vs. video understanding models. Moreover, we provide key insights on how video understanding models predict visual cortex responses; showing video understanding better than image understanding models, convolutional models are better in the early-mid regions than transformer based except for multiscale transformers that are still good in predicting these regions, and that two-stream models are better than single stream.

Title: An evaluation of Deep Learning based stereo dense matching dataset shift from aerial images and a large scale stereo dataset

• Authors: Teng Wu, Bruno Vallet, Marc Pierrot-Deseilligny, Ewelina Rupnik
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12522
• Pdf URL: https://arxiv.org/pdf/2402.12522
• Copy Paste: [[2402.12522]] An evaluation of Deep Learning based stereo dense matching dataset shift from aerial images and a large scale stereo dataset(https://arxiv.org/abs/2402.12522)
• Keywords: robust
• Abstract: Dense matching is crucial for 3D scene reconstruction since it enables the recovery of scene 3D geometry from image acquisition. Deep Learning (DL)-based methods have shown effectiveness in the special case of epipolar stereo disparity estimation in the computer vision community. DL-based methods depend heavily on the quality and quantity of training datasets. However, generating ground-truth disparity maps for real scenes remains a challenging task in the photogrammetry community. To address this challenge, we propose a method for generating ground-truth disparity maps directly from Light Detection and Ranging (LiDAR) and images to produce a large and diverse dataset for six aerial datasets across four different areas and two areas with different resolution images. We also introduce a LiDAR-to-image co-registration refinement to the framework that takes special precautions regarding occlusions and refrains from disparity interpolation to avoid precision loss. Evaluating 11 dense matching methods across datasets with diverse scene types, image resolutions, and geometric configurations, which are deeply investigated in dataset shift, GANet performs best with identical training and testing data, and PSMNet shows robustness across different datasets, and we proposed the best strategy for training with a limit dataset. We will also provide the dataset and training models; more information can be found at https://github.com/whuwuteng/Aerial_Stereo_Dataset.

Title: LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Tasks

• Authors: Truong Thanh Hung Nguyen, Tobias Clement, Phuc Truong Loc Nguyen, Nils Kemmerzell, Van Binh Truong, Vo Thanh Khang Nguyen, Mohamed Abdelaal, Hung Cao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12525
• Pdf URL: https://arxiv.org/pdf/2402.12525
• Copy Paste: [[2402.12525]] LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Tasks(https://arxiv.org/abs/2402.12525)
• Keywords: explainability, segmentation
• Abstract: LangXAI is a framework that integrates Explainable Artificial Intelligence (XAI) with advanced vision models to generate textual explanations for visual recognition tasks. Despite XAI advancements, an understanding gap persists for end-users with limited domain knowledge in artificial intelligence and computer vision. LangXAI addresses this by furnishing text-based explanations for classification, object detection, and semantic segmentation model outputs to end-users. Preliminary results demonstrate LangXAI's enhanced plausibility, with high BERTScore across tasks, fostering a more transparent and reliable AI framework on vision tasks for end-users.

Title: The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning

• Authors: Anya Sims, Cong Lu, Yee Whye Teh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12527
• Pdf URL: https://arxiv.org/pdf/2402.12527
• Copy Paste: [[2402.12527]] The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning(https://arxiv.org/abs/2402.12527)
• Keywords: robust
• Abstract: Offline reinforcement learning aims to enable agents to be trained from pre-collected datasets, however, this comes with the added challenge of estimating the value of behavior not covered in the dataset. Model-based methods offer a solution by allowing agents to collect additional synthetic data via rollouts in a learned dynamics model. The prevailing theoretical understanding is that this can then be viewed as online reinforcement learning in an approximate dynamics model, and any remaining gap is therefore assumed to be due to the imperfect dynamics model. Surprisingly, however, we find that if the learned dynamics model is replaced by the true error-free dynamics, existing model-based methods completely fail. This reveals a major misconception. Our subsequent investigation finds that the general procedure used in model-based algorithms results in the existence of a set of edge-of-reach states which trigger pathological value overestimation and collapse in Bellman-based algorithms. We term this the edge-of-reach problem. Based on this, we fill some gaps in existing theory and also explain how prior model-based methods are inadvertently addressing the true underlying edge-of-reach problem. Finally, we propose Reach-Aware Value Learning (RAVL), a simple and robust method that directly addresses the edge-of-reach problem and achieves strong performance across both proprioceptive and pixel-based benchmarks. Code open-sourced at: https://github.com/anyasims/edge-of-reach.

Title: Improving Deep Generative Models on Many-To-One Image-to-Image Translation

• Authors: Sagar Saxena, Mohammad Nayeem Teli
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12531
• Pdf URL: https://arxiv.org/pdf/2402.12531
• Copy Paste: [[2402.12531]] Improving Deep Generative Models on Many-To-One Image-to-Image Translation(https://arxiv.org/abs/2402.12531)
• Keywords: diffusion, generative
• Abstract: Deep generative models have been applied to multiple applications in image- to-image translation. Generative Adversarial Networks and Diffusion Models have presented impressive results, setting new state-of-the-art results on these tasks. Most methods have symmetric setups across the different domains in a dataset. These methods assume that all domains have either multiple modalities or only one modality. However, there are many datasets that have a many-to-one relationship between two domains. In this work, we first introduce a Colorized MNIST dataset and a Color-Recall score that can provide a simple benchmark for evaluating models on many-to-one translation. We then introduce a new asymmetric framework to improve existing deep generative models on many-to-one image-to- image translation. We apply this framework to StarGAN V2 and show that in both unsupervised and semi-supervised settings, the performance of this new model improves on many-to-one image-to-image translation.

Title: Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics

• Authors: Siqi Miao, Zhiyuan Lu, Mia Liu, Javier Duarte, Pan Li
• Subjects: cs.LG, hep-ex
• Abstract URL: https://arxiv.org/abs/2402.12535
• Pdf URL: https://arxiv.org/pdf/2402.12535
• Copy Paste: [[2402.12535]] Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics(https://arxiv.org/abs/2402.12535)
• Keywords: transformer
• Abstract: This study introduces a novel transformer model optimized for large-scale point cloud processing in scientific domains such as high-energy physics (HEP) and astrophysics. Addressing the limitations of graph neural networks and standard transformers, our model integrates local inductive bias and achieves near-linear complexity with hardware-friendly regular operations. One contribution of this work is the quantitative analysis of the error-complexity tradeoff of various sparsification techniques for building efficient transformers. Our findings highlight the superiority of using locality-sensitive hashing (LSH), especially OR \& AND-construction LSH, in kernel approximation for large-scale point cloud data with local inductive bias. Based on this finding, we propose LSH-based Efficient Point Transformer (\textbf{HEPT}), which combines E$^2$LSH with OR \& AND constructions and is built upon regular computations. HEPT demonstrates remarkable performance in two critical yet time-consuming HEP tasks, significantly outperforming existing GNNs and transformers in accuracy and computational speed, marking a significant advancement in geometric deep learning and large-scale scientific data processing. Our code is available at \url{https://github.com/Graph-COM/HEPT}.

Title: Designing High-Performing Networks for Multi-Scale Computer Vision

• Authors: Cédric Picron
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12536
• Pdf URL: https://arxiv.org/pdf/2402.12536
• Copy Paste: [[2402.12536]] Designing High-Performing Networks for Multi-Scale Computer Vision(https://arxiv.org/abs/2402.12536)
• Keywords: fair
• Abstract: Since the emergence of deep learning, the computer vision field has flourished with models improving at a rapid pace on more and more complex tasks. We distinguish three main ways to improve a computer vision model: (1) improving the data aspect by for example training on a large, more diverse dataset, (2) improving the training aspect by for example designing a better optimizer, and (3) improving the network architecture (or network for short). In this thesis, we chose to improve the latter, i.e. improving the network designs of computer vision models. More specifically, we investigate new network designs for multi-scale computer vision tasks, which are tasks requiring to make predictions about concepts at different scales. The goal of these new network designs is to outperform existing baseline designs from the literature. Specific care is taken to make sure the comparisons are fair, by guaranteeing that the different network designs were trained and evaluated with the same settings. Code is publicly available at https://github.com/CedricPicron/DetSeg.

Title: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning

• Authors: Kaan Ozkara, Bruce Huang, Ruida Zhou, Suhas Diggavi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12537
• Pdf URL: https://arxiv.org/pdf/2402.12537
• Copy Paste: [[2402.12537]] Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning(https://arxiv.org/abs/2402.12537)
• Keywords: federate, diffusion
• Abstract: Statistical heterogeneity of clients' local data is an important characteristic in federated learning, motivating personalized algorithms tailored to the local data statistics. Though there has been a plethora of algorithms proposed for personalized supervised learning, discovering the structure of local data through personalized unsupervised learning is less explored. We initiate a systematic study of such personalized unsupervised learning by developing algorithms based on optimization criteria inspired by a hierarchical Bayesian statistical framework. We develop adaptive algorithms that discover the balance between using limited local data and collaborative information. We do this in the context of two unsupervised learning tasks: personalized dimensionality reduction and personalized diffusion models. We develop convergence analyses for our adaptive algorithms which illustrate the dependence on problem parameters (e.g., heterogeneity, local sample size). We also develop a theoretical framework for personalized diffusion models, which shows the benefits of collaboration even under heterogeneity. We finally evaluate our proposed algorithms using synthetic and real data, demonstrating the effective sample amplification for personalized tasks, induced through collaboration, despite data heterogeneity.

Title: TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness

• Authors: Danna Zheng, Danyang Liu, Mirella Lapata, Jeff Z. Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12545
• Pdf URL: https://arxiv.org/pdf/2402.12545
• Copy Paste: [[2402.12545]] TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness(https://arxiv.org/abs/2402.12545)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities across various domains, prompting a surge in their practical applications. However, concerns have arisen regarding the trustworthiness of LLMs outputs, particularly in closed-book question-answering tasks, where non-experts may struggle to identify inaccuracies due to the absence of contextual or ground truth information. This paper introduces TrustScore, a framework based on the concept of Behavioral Consistency, which evaluates whether an LLMs response aligns with its intrinsic knowledge. Additionally, TrustScore can seamlessly integrate with fact-checking methods, which assesses alignment with external knowledge sources. The experimental results show that TrustScore achieves strong correlations with human judgments, surpassing existing reference-free metrics, and achieving results on par with reference-based metrics.

Title: Creating a Fine Grained Entity Type Taxonomy Using LLMs

• Authors: Michael Gunn, Dohyun Park, Nidhish Kamath
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12557
• Pdf URL: https://arxiv.org/pdf/2402.12557
• Copy Paste: [[2402.12557]] Creating a Fine Grained Entity Type Taxonomy Using LLMs(https://arxiv.org/abs/2402.12557)
• Keywords: extraction
• Abstract: In this study, we investigate the potential of GPT-4 and its advanced iteration, GPT-4 Turbo, in autonomously developing a detailed entity type taxonomy. Our objective is to construct a comprehensive taxonomy, starting from a broad classification of entity types - including objects, time, locations, organizations, events, actions, and subjects - similar to existing manually curated taxonomies. This classification is then progressively refined through iterative prompting techniques, leveraging GPT-4's internal knowledge base. The result is an extensive taxonomy comprising over 5000 nuanced entity types, which demonstrates remarkable quality upon subjective evaluation. We employed a straightforward yet effective prompting strategy, enabling the taxonomy to be dynamically expanded. The practical applications of this detailed taxonomy are diverse and significant. It facilitates the creation of new, more intricate branches through pattern-based combinations and notably enhances information extraction tasks, such as relation extraction and event argument extraction. Our methodology not only introduces an innovative approach to taxonomy creation but also opens new avenues for applying such taxonomies in various computational linguistics and AI-related fields.

Title: CausalGym: Benchmarking causal interpretability methods on linguistic tasks

• Authors: Aryaman Arora, Dan Jurafsky, Christopher Potts
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12560
• Pdf URL: https://arxiv.org/pdf/2402.12560
• Copy Paste: [[2402.12560]] CausalGym: Benchmarking causal interpretability methods on linguistic tasks(https://arxiv.org/abs/2402.12560)
• Keywords: interpretability
• Abstract: Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e.g., surprisal comparisons). At the same time, research in model interpretability has begun to illuminate the abstract causal mechanisms shaping LM behavior. To help bring these strands of research closer together, we introduce CausalGym. We adapt and expand the SyntaxGym suite of tasks to benchmark the ability of interpretability methods to causally affect model behaviour. To illustrate how CausalGym can be used, we study the pythia models (14M--6.9B) and assess the causal efficacy of a wide range of interpretability methods, including linear probing and distributed alignment search (DAS). We find that DAS outperforms the other methods, and so we use it to study the learning trajectory of two difficult linguistic phenomena in pythia-1b: negative polarity item licensing and filler--gap dependencies. Our analysis shows that the mechanism implementing both of these tasks is learned in discrete stages, not gradually.

Title: Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models

• Authors: Loka Li, Guangyi Chen, Yusheng Su, Zhenhao Chen, Yixuan Zhang, Eric Xing, Kun Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12563
• Pdf URL: https://arxiv.org/pdf/2402.12563
• Copy Paste: [[2402.12563]] Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models(https://arxiv.org/abs/2402.12563)
• Keywords: large language model
• Abstract: The recent success of Large Language Models (LLMs) has catalyzed an increasing interest in their self-correction capabilities. This paper presents a comprehensive investigation into the intrinsic self-correction of LLMs, attempting to address the ongoing debate about its feasibility. Our research has identified an important latent factor - the ``confidence'' of LLMs - during the self-correction process. Overlooking this factor may cause the models to over-criticize themselves, resulting in unreliable conclusions regarding the efficacy of self-correction. We have experimentally observed that LLMs possess the capability to understand the ``confidence'' in their own responses. It motivates us to develop an ``If-or-Else'' (IoE) prompting framework, designed to guide LLMs in assessing their own ``confidence'', facilitating intrinsic self-corrections. We conduct extensive experiments and demonstrate that our IoE-based Prompt can achieve a consistent improvement regarding the accuracy of self-corrected responses over the initial answers. Our study not only sheds light on the underlying factors affecting self-correction in LLMs, but also introduces a practical framework that utilizes the IoE prompting principle to efficiently improve self-correction capabilities with ``confidence''. The code is available at \url{https://github.com/MBZUAI-CLeaR/IoE-Prompting.git}.

Title: FairProof : Confidential and Certifiable Fairness for Neural Networks

• Authors: Chhavi Yadav, Amrita Roy Chowdhury, Dan Boneh, Kamalika Chaudhuri
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.12572
• Pdf URL: https://arxiv.org/pdf/2402.12572
• Copy Paste: [[2402.12572]] FairProof : Confidential and Certifiable Fairness for Neural Networks(https://arxiv.org/abs/2402.12572)
• Keywords: privacy, fair
• Abstract: Machine learning models are increasingly used in societal applications, yet legal and privacy concerns demand that they very often be kept confidential. Consequently, there is a growing distrust about the fairness properties of these models in the minds of consumers, who are often at the receiving end of model predictions. To this end, we propose FairProof - a system that uses Zero-Knowledge Proofs (a cryptographic primitive) to publicly verify the fairness of a model, while maintaining confidentiality. We also propose a fairness certification algorithm for fully-connected neural networks which is befitting to ZKPs and is used in this system. We implement FairProof in Gnark and demonstrate empirically that our system is practically feasible.

Title: Evolving AI Collectives to Enhance Human Diversity and Enable Self-Regulation

• Authors: Shiyang Lai, Yujin Potter, Junsol Kim, Richard Zhuang, Dawn Song, James Evans
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.12590
• Pdf URL: https://arxiv.org/pdf/2402.12590
• Copy Paste: [[2402.12590]] Evolving AI Collectives to Enhance Human Diversity and Enable Self-Regulation(https://arxiv.org/abs/2402.12590)
• Keywords: large language model
• Abstract: Large language models steer their behaviors based on texts generated by others. This capacity and their increasing prevalence in online settings portend that they will intentionally or unintentionally "program" one another and form emergent AI subjectivities, relationships, and collectives. Here, we call upon the research community to investigate these "society-like" properties of interacting artificial intelligences to increase their rewards and reduce their risks for human society and the health of online environments. We use a simple model and its outputs to illustrate how such emergent, decentralized AI collectives can expand the bounds of human diversity and reduce the risk of toxic, anti-social behavior online. Finally, we discuss opportunities for AI self-moderation and address ethical issues and design challenges associated with creating and maintaining decentralized AI collectives.

Title: Standardize: Aligning Language Models with Expert-Defined Standards for Content Generation

• Authors: Joseph Marvin Imperial, Gail Forey, Harish Tayyar Madabushi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12593
• Pdf URL: https://arxiv.org/pdf/2402.12593
• Copy Paste: [[2402.12593]] Standardize: Aligning Language Models with Expert-Defined Standards for Content Generation(https://arxiv.org/abs/2402.12593)
• Keywords: large language model
• Abstract: Domain experts across engineering, healthcare, and education follow strict standards for producing quality content such as technical manuals, medication instructions, and children's reading materials. However, current works in controllable text generation have yet to explore using these standards as references for control. Towards this end, we introduce Standardize, a retrieval-style in-context learning-based framework to guide large language models to align with expert-defined standards. Focusing on English language standards in the education domain as a use case, we consider the Common European Framework of Reference for Languages (CEFR) and Common Core Standards (CCS) for the task of open-ended content generation. Our findings show that models can gain 40% to 100% increase in precise accuracy for Llama2 and GPT-4, respectively, demonstrating that the use of knowledge artifacts extracted from standards and integrating them in the generation process can effectively guide models to produce better standard-aligned content.

Title: Generative AI Security: Challenges and Countermeasures

• Authors: Banghua Zhu, Norman Mu, Jiantao Jiao, David Wagner
• Subjects: cs.CR, cs.AI, cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12617
• Pdf URL: https://arxiv.org/pdf/2402.12617
• Copy Paste: [[2402.12617]] Generative AI Security: Challenges and Countermeasures(https://arxiv.org/abs/2402.12617)
• Keywords: security, generative
• Abstract: Generative AI's expanding footprint across numerous industries has led to both excitement and increased scrutiny. This paper delves into the unique security challenges posed by Generative AI, and outlines potential research directions for managing these risks.

Title: Indiscriminate Data Poisoning Attacks on Pre-trained Feature Extractors

• Authors: Yiwei Lu, Matthew Y.R. Yang, Gautam Kamath, Yaoliang Yu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.12626
• Pdf URL: https://arxiv.org/pdf/2402.12626
• Copy Paste: [[2402.12626]] Indiscriminate Data Poisoning Attacks on Pre-trained Feature Extractors(https://arxiv.org/abs/2402.12626)
• Keywords: security, attack
• Abstract: Machine learning models have achieved great success in supervised learning tasks for end-to-end training, which requires a large amount of labeled data that is not always feasible. Recently, many practitioners have shifted to self-supervised learning methods that utilize cheap unlabeled data to learn a general feature extractor via pre-training, which can be further applied to personalized downstream tasks by simply training an additional linear layer with limited labeled data. However, such a process may also raise concerns regarding data poisoning attacks. For instance, indiscriminate data poisoning attacks, which aim to decrease model utility by injecting a small number of poisoned data into the training set, pose a security risk to machine learning models, but have only been studied for end-to-end supervised learning. In this paper, we extend the exploration of the threat of indiscriminate attacks on downstream tasks that apply pre-trained feature extractors. Specifically, we propose two types of attacks: (1) the input space attacks, where we modify existing attacks to directly craft poisoned data in the input space. However, due to the difficulty of optimization under constraints, we further propose (2) the feature targeted attacks, where we mitigate the challenge with three stages, firstly acquiring target parameters for the linear head; secondly finding poisoned features by treating the learned feature representations as a dataset; and thirdly inverting the poisoned features back to the input space. Our experiments examine such attacks in popular downstream tasks of fine-tuning on the same dataset and transfer learning that considers domain adaptation. Empirical results reveal that transfer learning is more vulnerable to our attacks. Additionally, input space attacks are a strong threat if no countermeasures are posed, but are otherwise weaker than feature targeted attacks.

Title: YOLO-Ant: A Lightweight Detector via Depthwise Separable Convolutional and Large Kernel Design for Antenna Interference Source Detection

• Authors: Xiaoyu Tang, Xingming Chen, Jintao Cheng, Jin Wu, Rui Fan, Chengxi Zhang, Zebo Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12641
• Pdf URL: https://arxiv.org/pdf/2402.12641
• Copy Paste: [[2402.12641]] YOLO-Ant: A Lightweight Detector via Depthwise Separable Convolutional and Large Kernel Design for Antenna Interference Source Detection(https://arxiv.org/abs/2402.12641)
• Keywords: extraction, transformer
• Abstract: In the era of 5G communication, removing interference sources that affect communication is a resource-intensive task. The rapid development of computer vision has enabled unmanned aerial vehicles to perform various high-altitude detection tasks. Because the field of object detection for antenna interference sources has not been fully explored, this industry lacks dedicated learning samples and detection models for this specific task. In this article, an antenna dataset is created to address important antenna interference source detection issues and serves as the basis for subsequent research. We introduce YOLO-Ant, a lightweight CNN and transformer hybrid detector specifically designed for antenna interference source detection. Specifically, we initially formulated a lightweight design for the network depth and width, ensuring that subsequent investigations were conducted within a lightweight framework. Then, we propose a DSLK-Block module based on depthwise separable convolution and large convolution kernels to enhance the network's feature extraction ability, effectively improving small object detection. To address challenges such as complex backgrounds and large interclass differences in antenna detection, we construct DSLKVit-Block, a powerful feature extraction module that combines DSLK-Block and transformer structures. Considering both its lightweight design and accuracy, our method not only achieves optimal performance on the antenna dataset but also yields competitive results on public datasets.

Title: DiffusionNOCS: Managing Symmetry and Uncertainty in Sim2Real Multi-Modal Category-level Pose Estimation

• Authors: Takuya Ikeda, Sergey Zakharov, Tianyi Ko, Muhammad Zubair Irshad, Robert Lee, Katherine Liu, Rares Ambrus, Koichi Nishiwaki
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.12647
• Pdf URL: https://arxiv.org/pdf/2402.12647
• Copy Paste: [[2402.12647]] DiffusionNOCS: Managing Symmetry and Uncertainty in Sim2Real Multi-Modal Category-level Pose Estimation(https://arxiv.org/abs/2402.12647)
• Keywords: diffusion
• Abstract: This paper addresses the challenging problem of category-level pose estimation. Current state-of-the-art methods for this task face challenges when dealing with symmetric objects and when attempting to generalize to new environments solely through synthetic data training. In this work, we address these challenges by proposing a probabilistic model that relies on diffusion to estimate dense canonical maps crucial for recovering partial object shapes as well as establishing correspondences essential for pose estimation. Furthermore, we introduce critical components to enhance performance by leveraging the strength of the diffusion models with multi-modal input representations. We demonstrate the effectiveness of our method by testing it on a range of real datasets. Despite being trained solely on our generated synthetic data, our approach achieves state-of-the-art performance and unprecedented generalization qualities, outperforming baselines, even those specifically trained on the target domain.

Title: OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification

• Authors: Yifan Peng, Yui Sudo, Muhammad Shakeel, Shinji Watanabe
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.12654
• Pdf URL: https://arxiv.org/pdf/2402.12654
• Copy Paste: [[2402.12654]] OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification(https://arxiv.org/abs/2402.12654)
• Keywords: robust
• Abstract: There has been an increasing interest in large speech models that can perform multiple speech processing tasks in a single model. Such models usually adopt the encoder-decoder or decoder-only architecture due to their popularity and good performance in many domains. However, autoregressive models can be slower during inference compared to non-autoregressive models and also have potential risks of hallucination. Though prior studies observed promising results of non-autoregressive models for certain tasks at small scales, it remains unclear if they can be scaled to speech-to-text generation in diverse languages and tasks. Inspired by the Open Whisper-style Speech Model (OWSM) project, we propose OWSM-CTC, a novel encoder-only speech foundation model based on Connectionist Temporal Classification (CTC). It is trained on 180k hours of public audio data for multilingual automatic speech recognition (ASR), speech translation (ST), and language identification (LID). Compared to encoder-decoder OWSM, our OWSM-CTC achieves competitive results on ASR and up to 25% relative improvement on ST, while it is more robust and 3 to 4 times faster for inference. OWSM-CTC also improves the long-form ASR result with 20x speed-up. We will publicly release our codebase, pre-trained model, and training logs to promote open science in speech foundation models.

Title: The FinBen: An Holistic Financial Benchmark for Large Language Models

• Authors: Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, Zhiyang Deng, Yuechen Jiang, Zhiyuan Yao, Haohang Li, Yangyang Yu, Gang Hu, Jiajia Huang, Xiao-Yang Liu, Alejandro Lopez-Lira, Benyou Wang, Yanzhao Lai, Hao Wang, Min Peng, Sophia Ananiadou, Jimin Huang
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2402.12659
• Pdf URL: https://arxiv.org/pdf/2402.12659
• Copy Paste: [[2402.12659]] The FinBen: An Holistic Financial Benchmark for Large Language Models(https://arxiv.org/abs/2402.12659)
• Keywords: extraction, large language model
• Abstract: LLMs have transformed NLP and shown promise in various fields, yet their potential in finance is underexplored due to a lack of thorough evaluations and the complexity of financial tasks. This along with the rapid development of LLMs, highlights the urgent need for a systematic financial evaluation benchmark for LLMs. In this paper, we introduce FinBen, the first comprehensive open-sourced evaluation benchmark, specifically designed to thoroughly assess the capabilities of LLMs in the financial domain. FinBen encompasses 35 datasets across 23 financial tasks, organized into three spectrums of difficulty inspired by the Cattell-Horn-Carroll theory, to evaluate LLMs' cognitive abilities in inductive reasoning, associative memory, quantitative reasoning, crystallized intelligence, and more. Our evaluation of 15 representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals insights into their strengths and limitations within the financial domain. The findings indicate that GPT-4 leads in quantification, extraction, numerical reasoning, and stock trading, while Gemini shines in generation and forecasting; however, both struggle with complex extraction and forecasting, showing a clear need for targeted enhancements. Instruction tuning boosts simple task performance but falls short in improving complex reasoning and forecasting abilities. FinBen seeks to continuously evaluate LLMs in finance, fostering AI development with regular updates of tasks and models.

Title: SoftQE: Learned Representations of Queries Expanded by LLMs

• Authors: Varad Pimpalkhute, John Heyer, Xusen Yin, Sameer Gupta
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12663
• Pdf URL: https://arxiv.org/pdf/2402.12663
• Copy Paste: [[2402.12663]] SoftQE: Learned Representations of Queries Expanded by LLMs(https://arxiv.org/abs/2402.12663)
• Keywords: large language model
• Abstract: We investigate the integration of Large Language Models (LLMs) into query encoders to improve dense retrieval without increasing latency and cost, by circumventing the dependency on LLMs at inference time. SoftQE incorporates knowledge from LLMs by mapping embeddings of input queries to those of the LLM-expanded queries. While improvements over various strong baselines on in-domain MS-MARCO metrics are marginal, SoftQE improves performance by 2.83 absolute percentage points on average on five out-of-domain BEIR tasks.

Title: Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies

• Authors: Xiangyu Liu, Chenghao Deng, Yanchao Sun, Yongyuan Liang, Furong Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12673
• Pdf URL: https://arxiv.org/pdf/2402.12673
• Copy Paste: [[2402.12673]] Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies(https://arxiv.org/abs/2402.12673)
• Keywords: defense, attack, robust
• Abstract: In light of the burgeoning success of reinforcement learning (RL) in diverse real-world applications, considerable focus has been directed towards ensuring RL policies are robust to adversarial attacks during test time. Current approaches largely revolve around solving a minimax problem to prepare for potential worst-case scenarios. While effective against strong attacks, these methods often compromise performance in the absence of attacks or the presence of only weak attacks. To address this, we study policy robustness under the well-accepted state-adversarial attack model, extending our focus beyond only worst-case attacks. We first formalize this task at test time as a regret minimization problem and establish its intrinsic hardness in achieving sublinear regret when the baseline policy is from a general continuous policy class, $\Pi$. This finding prompts us to \textit{refine} the baseline policy class $\Pi$ prior to test time, aiming for efficient adaptation within a finite policy class $\Tilde{\Pi}$, which can resort to an adversarial bandit subroutine. In light of the importance of a small, finite $\Tilde{\Pi}$, we propose a novel training-time algorithm to iteratively discover \textit{non-dominated policies}, forming a near-optimal and minimal $\Tilde{\Pi}$, thereby ensuring both robustness and test-time efficiency. Empirical validation on the Mujoco corroborates the superiority of our approach in terms of natural and robust performance, as well as adaptability to various attack scenarios.

Title: Learning on manifolds without manifold learning

• Authors: H. N. Mhaskar, Ryan O'Dowd
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.12687
• Pdf URL: https://arxiv.org/pdf/2402.12687
• Copy Paste: [[2402.12687]] Learning on manifolds without manifold learning(https://arxiv.org/abs/2402.12687)
• Keywords: extraction
• Abstract: Function approximation based on data drawn randomly from an unknown distribution is an important problem in machine learning. In contrast to the prevalent paradigm of solving this problem by minimizing a loss functional, we have given a direct one-shot construction together with optimal error bounds under the manifold assumption; i.e., one assumes that the data is sampled from an unknown sub-manifold of a high dimensional Euclidean space. A great deal of research deals with obtaining information about this manifold, such as the eigendecomposition of the Laplace-Beltrami operator or coordinate charts, and using this information for function approximation. This two step approach implies some extra errors in the approximation stemming from basic quantities of the data in addition to the errors inherent in function approximation. In Neural Networks, 132:253268, 2020, we have proposed a one-shot direct method to achieve function approximation without requiring the extraction of any information about the manifold other than its dimension. However, one cannot pin down the class of approximants used in that paper. In this paper, we view the unknown manifold as a sub-manifold of an ambient hypersphere and study the question of constructing a one-shot approximation using the spherical polynomials based on the hypersphere. Our approach does not require preprocessing of the data to obtain information about the manifold other than its dimension. We give optimal rates of approximation for relatively "rough" functions.

Title: Robust-Wide: Robust Watermarking against Instruction-driven Image Editing

• Authors: Runyi Hu, Jie Zhang, Tianwei Zhang, Jiwei Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.12688
• Pdf URL: https://arxiv.org/pdf/2402.12688
• Copy Paste: [[2402.12688]] Robust-Wide: Robust Watermarking against Instruction-driven Image Editing(https://arxiv.org/abs/2402.12688)
• Keywords: robust, extraction, watermark
• Abstract: Instruction-driven image editing allows users to quickly edit an image according to text instructions in a forward pass. Nevertheless, malicious users can easily exploit this technique to create fake images, which could cause a crisis of trust and harm the rights of the original image owners. Watermarking is a common solution to trace such malicious behavior. Unfortunately, instruction-driven image editing can significantly change the watermarked image at the semantic level, making it less robust and effective. We propose Robust-Wide, the first robust watermarking methodology against instruction-driven image editing. Specifically, we adopt the widely-used encoder-decoder framework for watermark embedding and extraction. To achieve robustness against semantic distortions, we introduce a novel Partial Instruction-driven Denoising Sampling Guidance (PIDSG) module, which consists of a large variety of instruction injections and substantial modifications of images at different semantic levels. With PIDSG, the encoder tends to embed the watermark into more robust and semantic-aware areas, which remains in existence even after severe image editing. Experiments demonstrate that Robust-Wide can effectively extract the watermark from the edited image with a low bit error rate of nearly 2.6% for 64-bit watermark messages. Meanwhile, it only induces a neglectable influence on the visual quality and editability of the original images. Moreover, Robust-Wide holds general robustness against different sampling configurations and other image editing methods such as ControlNet-InstructPix2Pix, MagicBrush, Inpainting and DDIM Inversion.

Title: Tree-Planted Transformers: Large Language Models with Implicit Syntactic Supervision

• Authors: Ryo Yoshida, Taiga Someya, Yohei Oseki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12691
• Pdf URL: https://arxiv.org/pdf/2402.12691
• Copy Paste: [[2402.12691]] Tree-Planted Transformers: Large Language Models with Implicit Syntactic Supervision(https://arxiv.org/abs/2402.12691)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable success thanks to scalability on large text corpora, but have some drawback in training efficiency. In contrast, Syntactic Language Models (SLMs) can be trained efficiently to reach relatively high performance thanks to syntactic supervision, but have trouble with scalability. Thus, given these complementary advantages of LLMs and SLMs, it is necessary to develop an architecture that integrates the scalability of LLMs with the training efficiency of SLMs, namely Syntactic Large Language Models (SLLM). In this paper, we propose a novel method dubbed tree-planting: implicitly "plant" trees into attention weights of Transformer LMs to reflect syntactic structures of natural language. Specifically, Transformer LMs trained with tree-planting will be called Tree-Planted Transformers (TPT), which learn syntax on small treebanks via tree-planting and then scale on large text corpora via continual learning with syntactic scaffolding. Targeted syntactic evaluations on the SyntaxGym benchmark demonstrated that TPTs, despite the lack of explicit syntactic supervision, significantly outperformed various SLMs with explicit syntactic supervision that generate hundreds of syntactic structures in parallel, suggesting that tree-planting and TPTs are the promising foundation for SLLMs.

Title: Learning Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition

• Authors: Yuke Li, Guangyi Chen, Ben Abramowitz, Stefano Anzellott, Donglai Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12706
• Pdf URL: https://arxiv.org/pdf/2402.12706
• Copy Paste: [[2402.12706]] Learning Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition(https://arxiv.org/abs/2402.12706)
• Keywords: generative
• Abstract: Few-shot action recognition aims at quickly adapting a pre-trained model to the novel data with a distribution shift using only a limited number of samples. Key challenges include how to identify and leverage the transferable knowledge learned by the pre-trained model. Our central hypothesis is that temporal invariance in the dynamic system between latent variables lends itself to transferability (domain-invariance). We therefore propose DITeD, or Domain-Invariant Temporal Dynamics for knowledge transfer. To detect the temporal invariance part, we propose a generative framework with a two-stage training strategy during pre-training. Specifically, we explicitly model invariant dynamics including temporal dynamic generation and transitions, and the variant visual and domain encoders. Then we pre-train the model with the self-supervised signals to learn the representation. After that, we fix the whole representation model and tune the classifier. During adaptation, we fix the transferable temporal dynamics and update the image encoder. The efficacy of our approach is revealed by the superior accuracy of DITeD over leading alternatives across standard few-shot action recognition datasets. Moreover, we validate that the learned temporal dynamic transition and temporal dynamic generation modules possess transferable qualities.

Title: MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction

• Authors: Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, Rakesh Ranjan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12712
• Pdf URL: https://arxiv.org/pdf/2402.12712
• Copy Paste: [[2402.12712]] MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction(https://arxiv.org/abs/2402.12712)
• Keywords: diffusion, generative
• Abstract: This paper presents a neural architecture MVDiffusion++ for 3D object reconstruction that synthesizes dense and high-resolution views of an object given one or a few images without camera poses. MVDiffusion++ achieves superior flexibility and scalability with two surprisingly simple ideas: 1) A ``pose-free architecture'' where standard self-attention among 2D latent features learns 3D consistency across an arbitrary number of conditional and generation views without explicitly using camera pose information; and 2) A ``view dropout strategy'' that discards a substantial number of output views during training, which reduces the training-time memory footprint and enables dense and high-resolution view synthesis at test time. We use the Objaverse for training and the Google Scanned Objects for evaluation with standard novel view synthesis and 3D reconstruction metrics, where MVDiffusion++ significantly outperforms the current state of the arts. We also demonstrate a text-to-3D application example by combining MVDiffusion++ with a text-to-image generative model.

Title: Are Large Language Models Rational Investors?

• Authors: Yuhang Zhou, Yuchen Ni, Xiang Liu, Jian Zhang, Sen Liu, Guangnan Ye, Hongfeng Chai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12713
• Pdf URL: https://arxiv.org/pdf/2402.12713
• Copy Paste: [[2402.12713]] Are Large Language Models Rational Investors?(https://arxiv.org/abs/2402.12713)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are progressively being adopted in financial analysis to harness their extensive knowledge base for interpreting complex market data and trends. However, their application in the financial domain is challenged by intrinsic biases (i.e., risk-preference bias) and a superficial grasp of market intricacies, underscoring the need for a thorough assessment of their financial insight. This study introduces a novel framework, Financial Bias Indicators (FBI), to critically evaluate the financial rationality of LLMs, focusing on their ability to discern and navigate the subtleties of financial information and to identify any irrational biases that might skew market analysis. Our research adopts an innovative methodology to measure financial rationality, integrating principles of behavioral finance to scrutinize the biases and decision-making patterns of LLMs. We conduct a comprehensive evaluation of 19 leading LLMs, considering factors such as model scale, training datasets, input strategies, etc. The findings reveal varying degrees of financial irrationality among the models, influenced by their design and training. Models trained specifically on financial datasets might exhibit greater irrationality, and it's possible that even larger financial language models (FinLLMs) could display more biases than smaller, more generalized models. This outcomes provide profound insights into how these elements affect the financial rationality of LLMs, indicating that targeted training and structured input methods could improve model performance. This work enriches our understanding of LLMs' strengths and weaknesses in financial applications, laying the groundwork for the development of more dependable and rational financial analysis tools.

Title: Equivariant Pretrained Transformer for Unified Geometric Learning on Multi-Domain 3D Molecules

• Authors: Rui Jiao, Xiangzhe Kong, Ziyang Yu, Wenbing Huang, Yang Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12714
• Pdf URL: https://arxiv.org/pdf/2402.12714
• Copy Paste: [[2402.12714]] Equivariant Pretrained Transformer for Unified Geometric Learning on Multi-Domain 3D Molecules(https://arxiv.org/abs/2402.12714)
• Keywords: transformer
• Abstract: Pretraining on a large number of unlabeled 3D molecules has showcased superiority in various scientific applications. However, prior efforts typically focus on pretraining models on a specific domain, either proteins or small molecules, missing the opportunity to leverage the cross-domain knowledge. To mitigate this gap, we introduce Equivariant Pretrained Transformer (EPT), a novel pretraining framework designed to harmonize the geometric learning of small molecules and proteins. To be specific, EPT unifies the geometric modeling of multi-domain molecules via the block-enhanced representation that can attend a broader context of each atom. Upon transformer framework, EPT is further enhanced with E(3) equivariance to facilitate the accurate representation of 3D structures. Another key innovation of EPT is its block-level pretraining task, which allows for joint pretraining on datasets comprising both small molecules and proteins. Experimental evaluations on a diverse group of benchmarks, including ligand binding affinity prediction, molecular property prediction, and protein property prediction, show that EPT significantly outperforms previous SOTA methods for affinity prediction, and achieves the best or comparable performance with existing domain-specific pretraining models for other tasks.

Title: Spurious Correlations in Machine Learning: A Survey

• Authors: Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, Xia Hu, Aidong Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12715
• Pdf URL: https://arxiv.org/pdf/2402.12715
• Copy Paste: [[2402.12715]] Spurious Correlations in Machine Learning: A Survey(https://arxiv.org/abs/2402.12715)
• Keywords: robust
• Abstract: Machine learning systems are known to be sensitive to spurious correlations between biased features of the inputs (e.g., background, texture, and secondary objects) and the corresponding labels. These features and their correlations with the labels are known as "spurious" because they tend to change with shifts in real-world data distributions, which can negatively impact the model's generalization and robustness. In this survey, we provide a comprehensive review of this issue, along with a taxonomy of current state-of-the-art methods for addressing spurious correlations in machine learning models. Additionally, we summarize existing datasets, benchmarks, and metrics to aid future research. The paper concludes with a discussion of the recent advancements and future research challenges in this field, aiming to provide valuable insights for researchers in the related domains.

Title: Revisiting the Information Capacity of Neural Network Watermarks: Upper Bound Estimation and Beyond

• Authors: Fangqi Li, Haodong Zhao, Wei Du, Shilin Wang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12720
• Pdf URL: https://arxiv.org/pdf/2402.12720
• Copy Paste: [[2402.12720]] Revisiting the Information Capacity of Neural Network Watermarks: Upper Bound Estimation and Beyond(https://arxiv.org/abs/2402.12720)
• Keywords: secure, attack, robust, watermark
• Abstract: To trace the copyright of deep neural networks, an owner can embed its identity information into its model as a watermark. The capacity of the watermark quantify the maximal volume of information that can be verified from the watermarked model. Current studies on capacity focus on the ownership verification accuracy under ordinary removal attacks and fail to capture the relationship between robustness and fidelity. This paper studies the capacity of deep neural network watermarks from an information theoretical perspective. We propose a new definition of deep neural network watermark capacity analogous to channel capacity, analyze its properties, and design an algorithm that yields a tight estimation of its upper bound under adversarial overwriting. We also propose a universal non-invasive method to secure the transmission of the identity message beyond capacity by multiple rounds of ownership verification. Our observations provide evidence for neural network owners and defenders that are curious about the tradeoff between the integrity of their ownership and the performance degradation of their products.

Title: Diffusion Posterior Sampling is Computationally Intractable

• Authors: Shivam Gupta, Ajil Jalal, Aditya Parulekar, Eric Price, Zhiyang Xun
• Subjects: cs.LG, cs.AI, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.12727
• Pdf URL: https://arxiv.org/pdf/2402.12727
• Copy Paste: [[2402.12727]] Diffusion Posterior Sampling is Computationally Intractable(https://arxiv.org/abs/2402.12727)
• Keywords: diffusion
• Abstract: Diffusion models are a remarkably effective way of learning and sampling from a distribution $p(x)$. In posterior sampling, one is also given a measurement model $p(y \mid x)$ and a measurement $y$, and would like to sample from $p(x \mid y)$. Posterior sampling is useful for tasks such as inpainting, super-resolution, and MRI reconstruction, so a number of recent works have given algorithms to heuristically approximate it; but none are known to converge to the correct distribution in polynomial time. In this paper we show that posterior sampling is \emph{computationally intractable}: under the most basic assumption in cryptography -- that one-way functions exist -- there are instances for which \emph{every} algorithm takes superpolynomial time, even though \emph{unconditional} sampling is provably fast. We also show that the exponential-time rejection sampling algorithm is essentially optimal under the stronger plausible assumption that there are one-way functions that take exponential time to invert.

Title: Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answering

• Authors: Junnan Dong, Qinggang Zhang, Huachi Zhou, Daochen Zha, Pai Zheng, Xiao Huang
• Subjects: cs.CV, cs.AI, cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12728
• Pdf URL: https://arxiv.org/pdf/2402.12728
• Copy Paste: [[2402.12728]] Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answering(https://arxiv.org/abs/2402.12728)
• Keywords: large language model
• Abstract: Knowledge-based visual question answering (KVQA) has been extensively studied to answer visual questions with external knowledge, e.g., knowledge graphs (KGs). While several attempts have been proposed to leverage large language models (LLMs) as an implicit knowledge source, it remains challenging since LLMs may generate hallucinations. Moreover, multiple knowledge sources, e.g., images, KGs and LLMs, cannot be readily aligned for complex scenarios. To tackle these, we present a novel modality-aware integration with LLMs for KVQA (MAIL). It carefully leverages multimodal knowledge for both image understanding and knowledge reasoning. Specifically, (i) we propose a two-stage prompting strategy with LLMs to densely embody the image into a scene graph with detailed visual features; (ii) We construct a coupled concept graph by linking the mentioned entities with external facts. (iii) A tailored pseudo-siamese graph medium fusion is designed for sufficient multimodal fusion. We utilize the shared mentioned entities in two graphs as mediums to bridge a tight inter-modal exchange, while maximally preserving insightful intra-modal learning by constraining the fusion within mediums. Extensive experiments on two benchmark datasets show the superiority of MAIL with 24x less resources.

Title: UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation

• Authors: Shubhashis Roy Dipta, Sai Vallurupalli
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12730
• Pdf URL: https://arxiv.org/pdf/2402.12730
• Copy Paste: [[2402.12730]] UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation(https://arxiv.org/abs/2402.12730)
• Keywords: large language model
• Abstract: This paper describes the system we developed for SemEval-2024 Task 1, "Semantic Textual Relatedness for African and Asian Languages." The aim of the task is to build a model that can identify semantic textual relatedness (STR) between two sentences of a target language belonging to a collection of African and Asian languages. We participated in Subtasks A and C and explored supervised and cross-lingual training leveraging large language models (LLMs). Pre-trained large language models have been extensively used for machine translation and semantic similarity. Using a combination of machine translation and sentence embedding LLMs, we developed a unified STR model, TranSem, for subtask A and fine-tuned the T5 family of models on the STR data, FineSem, for use in subtask C. Our model results for 7 languages in subtask A were better than the official baseline for 3 languages and on par with the baseline for the remaining 4 languages. Our model results for the 12 languages in subtask C resulted in 1st place for Africaans, 2nd place for Indonesian, and 3rd place for English with low performance for the remaining 9 languages.

Title: CST: Calibration Side-Tuning for Parameter and Memory Efficient Transfer Learning

• Authors: Feng Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12736
• Pdf URL: https://arxiv.org/pdf/2402.12736
• Copy Paste: [[2402.12736]] CST: Calibration Side-Tuning for Parameter and Memory Efficient Transfer Learning(https://arxiv.org/abs/2402.12736)
• Keywords: transformer
• Abstract: Achieving a universally high accuracy in object detection is quite challenging, and the mainstream focus in the industry currently lies on detecting specific classes of objects. However, deploying one or multiple object detection networks requires a certain amount of GPU memory for training and storage capacity for inference. This presents challenges in terms of how to effectively coordinate multiple object detection tasks under resource-constrained conditions. This paper introduces a lightweight fine-tuning strategy called Calibration side tuning, which integrates aspects of adapter tuning and side tuning to adapt the successful techniques employed in transformers for use with ResNet. The Calibration side tuning architecture that incorporates maximal transition calibration, utilizing a small number of additional parameters to enhance network performance while maintaining a smooth training process. Furthermore, this paper has conducted an analysis on multiple fine-tuning strategies and have implemented their application within ResNet, thereby expanding the research on fine-tuning strategies for object detection networks. Besides, this paper carried out extensive experiments using five benchmark datasets. The experimental results demonstrated that this method outperforms other compared state-of-the-art techniques, and a better balance between the complexity and performance of the finetune schemes is achieved.

Title: Guarantee Regions for Local Explanations

• Authors: Marton Havasi, Sonali Parbhoo, Finale Doshi-Velez
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12737
• Pdf URL: https://arxiv.org/pdf/2402.12737
• Copy Paste: [[2402.12737]] Guarantee Regions for Local Explanations(https://arxiv.org/abs/2402.12737)
• Keywords: interpretability
• Abstract: Interpretability methods that utilise local surrogate models (e.g. LIME) are very good at describing the behaviour of the predictive model at a point of interest, but they are not guaranteed to extrapolate to the local region surrounding the point. However, overfitting to the local curvature of the predictive model and malicious tampering can significantly limit extrapolation. We propose an anchor-based algorithm for identifying regions in which local explanations are guaranteed to be correct by explicitly describing those intervals along which the input features can be trusted. Our method produces an interpretable feature-aligned box where the prediction of the local surrogate model is guaranteed to match the predictive model. We demonstrate that our algorithm can be used to find explanations with larger guarantee regions that better cover the data manifold compared to existing baselines. We also show how our method can identify misleading local explanations with significantly poorer guarantee regions.

Title: Can Large Language Models be Used to Provide Psychological Counselling? An Analysis of GPT-4-Generated Responses Using Role-play Dialogues

• Authors: Michimasa Inaba, Mariko Ukiyo, Keiko Takamizo
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.12738
• Pdf URL: https://arxiv.org/pdf/2402.12738
• Copy Paste: [[2402.12738]] Can Large Language Models be Used to Provide Psychological Counselling? An Analysis of GPT-4-Generated Responses Using Role-play Dialogues(https://arxiv.org/abs/2402.12738)
• Keywords: large language model
• Abstract: Mental health care poses an increasingly serious challenge to modern societies. In this context, there has been a surge in research that utilizes information technologies to address mental health problems, including those aiming to develop counseling dialogue systems. However, there is a need for more evaluations of the performance of counseling dialogue systems that use large language models. For this study, we collected counseling dialogue data via role-playing scenarios involving expert counselors, and the utterances were annotated with the intentions of the counselors. To determine the feasibility of a dialogue system in real-world counseling scenarios, third-party counselors evaluated the appropriateness of responses from human counselors and those generated by GPT-4 in identical contexts in role-play dialogue data. Analysis of the evaluation results showed that the responses generated by GPT-4 were competitive with those of human counselors.

Title: MuLan: Multimodal-LLM Agent for Progressive Multi-Object Diffusion

• Authors: Sen Li, Ruochen Wang, Cho-Jui Hsieh, Minhao Cheng, Tianyi Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12741
• Pdf URL: https://arxiv.org/pdf/2402.12741
• Copy Paste: [[2402.12741]] MuLan: Multimodal-LLM Agent for Progressive Multi-Object Diffusion(https://arxiv.org/abs/2402.12741)
• Keywords: diffusion, large language model
• Abstract: Existing text-to-image models still struggle to generate images of multiple objects, especially in handling their spatial positions, relative sizes, overlapping, and attribute bindings. In this paper, we develop a training-free Multimodal-LLM agent (MuLan) to address these challenges by progressive multi-object generation with planning and feedback control, like a human painter. MuLan harnesses a large language model (LLM) to decompose a prompt to a sequence of sub-tasks, each generating only one object conditioned on previously generated objects by stable diffusion. Unlike existing LLM-grounded methods, MuLan only produces a high-level plan at the beginning while the exact size and location of each object are determined by an LLM and attention guidance upon each sub-task. Moreover, MuLan adopts a vision-language model (VLM) to provide feedback to the image generated in each sub-task and control the diffusion model to re-generate the image if it violates the original prompt. Hence, each model in every step of MuLan only needs to address an easy sub-task it is specialized for. We collect 200 prompts containing multi-objects with spatial relationships and attribute bindings from different benchmarks to evaluate MuLan. The results demonstrate the superiority of MuLan in generating multiple objects over baselines. The code is available on https://github.com/measure-infinity/mulan-code.

Title: APT-MMF: An advanced persistent threat actor attribution method based on multimodal and multilevel feature fusion

• Authors: Nan Xiao, Bo Lang, Ting Wang, Yikai Chen
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12743
• Pdf URL: https://arxiv.org/pdf/2402.12743
• Copy Paste: [[2402.12743]] APT-MMF: An advanced persistent threat actor attribution method based on multimodal and multilevel feature fusion(https://arxiv.org/abs/2402.12743)
• Keywords: defense, interpretability
• Abstract: Threat actor attribution is a crucial defense strategy for combating advanced persistent threats (APTs). Cyber threat intelligence (CTI), which involves analyzing multisource heterogeneous data from APTs, plays an important role in APT actor attribution. The current attribution methods extract features from different CTI perspectives and employ machine learning models to classify CTI reports according to their threat actors. However, these methods usually extract only one kind of feature and ignore heterogeneous information, especially the attributes and relations of indicators of compromise (IOCs), which form the core of CTI. To address these problems, we propose an APT actor attribution method based on multimodal and multilevel feature fusion (APT-MMF). First, we leverage a heterogeneous attributed graph to characterize APT reports and their IOC information. Then, we extract and fuse multimodal features, including attribute type features, natural language text features and topological relationship features, to construct comprehensive node representations. Furthermore, we design multilevel heterogeneous graph attention networks to learn the deep hidden features of APT report nodes; these networks integrate IOC type-level, metapath-based neighbor node-level, and metapath semantic-level attention. Utilizing multisource threat intelligence, we construct a heterogeneous attributed graph dataset for verification purposes. The experimental results show that our method not only outperforms the existing methods but also demonstrates its good interpretability for attribution analysis tasks.

Title: Me LLaMA: Foundation Large Language Models for Medical Applications

• Authors: Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, Huan He, Lucila Ohno-Machido, Yonghui Wu, Hua Xu, Jiang Bian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12749
• Pdf URL: https://arxiv.org/pdf/2402.12749
• Copy Paste: [[2402.12749]] Me LLaMA: Foundation Large Language Models for Medical Applications(https://arxiv.org/abs/2402.12749)
• Keywords: large language model
• Abstract: Recent large language models (LLMs) like ChatGPT and LLaMA have shown great promise in many AI applications. However, their performance on medical tasks is suboptimal and can be further improved by training on large domain-specific datasets. This study introduces Me LLaMA, a medical LLM family including foundation models - Me LLaMA 13/70B and their chat-enhanced versions - Me LLaMA 13/70B-chat, developed through the continual pre-training and instruction tuning of LLaMA2 using large medical data. Our domain-specific data suite for training and evaluation, includes a large-scale continual pre-training dataset with 129B tokens, an instruction tuning dataset with 214k samples, and a medical evaluation benchmark (MIBE) across six tasks with 14 datasets. Our extensive evaluation using MIBE shows that Me LLaMA models surpass existing open-source medical LLMs in zero-shot and few-shot learning and outperform commercial giants like ChatGPT on 6 out of 8 datasets and GPT-4 in 3 out of 8 datasets. In addition, we empirically investigated the catastrophic forgetting problem, and our results show that Me LLaMA models outperform other medical LLMs. Me LLaMA is one of the first and largest open-source foundational LLMs designed for the medical domain, using both biomedical and clinical data. It exhibits superior performance across both general and medical tasks compared to other medical LLMs, rendering it an attractive choice for medical AI applications. All resources are available at: https://github.com/BIDS-Xu-Lab/Me-LLaMA.

Title: Model Composition for Multimodal Large Language Models

• Authors: Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, Yang Liu
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12750
• Pdf URL: https://arxiv.org/pdf/2402.12750
• Copy Paste: [[2402.12750]] Model Composition for Multimodal Large Language Models(https://arxiv.org/abs/2402.12750)
• Keywords: large language model
• Abstract: Recent developments in Multimodal Large Language Models (MLLMs) have shown rapid progress, moving towards the goal of creating versatile MLLMs that understand inputs from various modalities. However, existing methods typically rely on joint training with paired multimodal instruction data, which is resource-intensive and challenging to extend to new modalities. In this paper, we propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model. Our basic implementation, NaiveMC, demonstrates the effectiveness of this paradigm by reusing modality encoders and merging LLM parameters. Furthermore, we introduce DAMC to address parameter interference and mismatch issues during the merging process, thereby enhancing the model performance. To facilitate research in this area, we propose MCUB, a benchmark for assessing ability of MLLMs to understand inputs from diverse modalities. Experiments on this benchmark and four other multimodal understanding tasks show significant improvements over baselines, proving that model composition can create a versatile model capable of processing inputs from multiple modalities.

Title: Fingerprint Presentation Attack Detector Using Global-Local Model

• Authors: Haozhe Liu, Wentian Zhang, Feng Liu, Haoqian Wu, Linlin Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12754
• Pdf URL: https://arxiv.org/pdf/2402.12754
• Copy Paste: [[2402.12754]] Fingerprint Presentation Attack Detector Using Global-Local Model(https://arxiv.org/abs/2402.12754)
• Keywords: attack
• Abstract: The vulnerability of automated fingerprint recognition systems (AFRSs) to presentation attacks (PAs) promotes the vigorous development of PA detection (PAD) technology. However, PAD methods have been limited by information loss and poor generalization ability, resulting in new PA materials and fingerprint sensors. This paper thus proposes a global-local model-based PAD (RTK-PAD) method to overcome those limitations to some extent. The proposed method consists of three modules, called: 1) the global module; 2) the local module; and 3) the rethinking module. By adopting the cut-out-based global module, a global spoofness score predicted from nonlocal features of the entire fingerprint images can be achieved. While by using the texture in-painting-based local module, a local spoofness score predicted from fingerprint patches is obtained. The two modules are not independent but connected through our proposed rethinking module by localizing two discriminative patches for the local module based on the global spoofness score. Finally, the fusion spoofness score by averaging the global and local spoofness scores is used for PAD. Our experimental results evaluated on LivDet 2017 show that the proposed RTK-PAD can achieve an average classification error (ACE) of 2.28% and a true detection rate (TDR) of 91.19% when the false detection rate (FDR) equals 1.0%, which significantly outperformed the state-of-the-art methods by $\sim$10% in terms of TDR (91.19% versus 80.74%).

Title: FGAD: Self-boosted Knowledge Distillation for An Effective Federated Graph Anomaly Detection Framework

• Authors: Jinyu Cai, Yunhe Zhang, Zhoumin Lu, Wenzhong Guo, See-kiong Ng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12761
• Pdf URL: https://arxiv.org/pdf/2402.12761
• Copy Paste: [[2402.12761]] FGAD: Self-boosted Knowledge Distillation for An Effective Federated Graph Anomaly Detection Framework(https://arxiv.org/abs/2402.12761)
• Keywords: privacy, robust, federate
• Abstract: Graph anomaly detection (GAD) aims to identify anomalous graphs that significantly deviate from other ones, which has raised growing attention due to the broad existence and complexity of graph-structured data in many real-world scenarios. However, existing GAD methods usually execute with centralized training, which may lead to privacy leakage risk in some sensitive cases, thereby impeding collaboration among organizations seeking to collectively develop robust GAD models. Although federated learning offers a promising solution, the prevalent non-IID problems and high communication costs present significant challenges, particularly pronounced in collaborations with graph data distributed among different participants. To tackle these challenges, we propose an effective federated graph anomaly detection framework (FGAD). We first introduce an anomaly generator to perturb the normal graphs to be anomalous, and train a powerful anomaly detector by distinguishing generated anomalous graphs from normal ones. Then, we leverage a student model to distill knowledge from the trained anomaly detector (teacher model), which aims to maintain the personality of local models and alleviate the adverse impact of non-IID problems. Moreover, we design an effective collaborative learning mechanism that facilitates the personalization preservation of local models and significantly reduces communication costs among clients. Empirical results of the GAD tasks on non-IID graphs compared with state-of-the-art baselines demonstrate the superiority and efficiency of the proposed FGAD method.

Title: Two-stage Rainfall-Forecasting Diffusion Model

• Authors: XuDong Ling, ChaoRong Li, FengQing Qin, LiHong Zhu, Yuanyuan Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12779
• Pdf URL: https://arxiv.org/pdf/2402.12779
• Copy Paste: [[2402.12779]] Two-stage Rainfall-Forecasting Diffusion Model(https://arxiv.org/abs/2402.12779)
• Keywords: robust, diffusion
• Abstract: Deep neural networks have made great achievements in rainfall prediction.However, the current forecasting methods have certain limitations, such as with blurry generated images and incorrect spatial positions. To overcome these challenges, we propose a Two-stage Rainfall-Forecasting Diffusion Model (TRDM) aimed at improving the accuracy of long-term rainfall forecasts and addressing the imbalance in performance between temporal and spatial modeling. TRDM is a two-stage method for rainfall prediction tasks. The task of the first stage is to capture robust temporal information while preserving spatial information under low-resolution conditions. The task of the second stage is to reconstruct the low-resolution images generated in the first stage into high-resolution images. We demonstrate state-of-the-art results on the MRMS and Swedish radar datasets. Our project is open source and available on GitHub at: \href{https://github.com/clearlyzerolxd/TRDM}{https://github.com/clearlyzerolxd/TRDM}.

Title: Tackling Byzantine Clients in Federated Learning

• Authors: Youssef Allouah, Sadegh Farhadkhani, Rachid GuerraouI, Nirupam Gupta, Rafael Pinot, Geovani Rizk, Sasha Voitovych
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12780
• Pdf URL: https://arxiv.org/pdf/2402.12780
• Copy Paste: [[2402.12780]] Tackling Byzantine Clients in Federated Learning(https://arxiv.org/abs/2402.12780)
• Keywords: robust, federate
• Abstract: The possibility of adversarial (a.k.a., {\em Byzantine}) clients makes federated learning (FL) prone to arbitrary manipulation. The natural approach to robustify FL against adversarial clients is to replace the simple averaging operation at the server in the standard $\mathsf{FedAvg}$ algorithm by a \emph{robust averaging rule}. While a significant amount of work has been devoted to studying the convergence of federated {\em robust averaging} (which we denote by $\mathsf{FedRo}$), prior work has largely ignored the impact of {\em client subsampling} and {\em local steps}, two fundamental FL characteristics. While client subsampling increases the effective fraction of Byzantine clients, local steps increase the drift between the local updates computed by honest (i.e., non-Byzantine) clients. Consequently, a careless deployment of $\mathsf{FedRo}$ could yield poor performance. We validate this observation by presenting an in-depth analysis of $\mathsf{FedRo}$ tightly analyzing the impact of client subsampling and local steps. Specifically, we present a sufficient condition on client subsampling for nearly-optimal convergence of $\mathsf{FedRo}$ (for smooth non-convex loss). Also, we show that the rate of improvement in learning accuracy {\em diminishes} with respect to the number of clients subsampled, as soon as the sample size exceeds a threshold value. Interestingly, we also observe that under a careful choice of step-sizes, the learning error due to Byzantine clients decreases with the number of local steps. We validate our theory by experiments on the FEMNIST and CIFAR-$10$ image classification tasks.

Title: Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations

• Authors: Guan-Ting Lin, Cheng-Han Chiang, Hung-yi Lee
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.12786
• Pdf URL: https://arxiv.org/pdf/2402.12786
• Copy Paste: [[2402.12786]] Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations(https://arxiv.org/abs/2402.12786)
• Keywords: large language model
• Abstract: In spoken dialogue, even if two current turns are the same sentence, their responses might still differ when they are spoken in different styles. The spoken styles, containing paralinguistic and prosodic information, mark the most significant difference between text and speech modality. When using text-only LLMs to model spoken dialogue, text-only LLMs cannot give different responses based on the speaking style of the current turn. In this paper, we focus on enabling LLMs to listen to the speaking styles and respond properly. Our goal is to teach the LLM that "even if the sentences are identical if they are spoken in different styles, their corresponding responses might be different". Since there is no suitable dataset for achieving this goal, we collect a speech-to-speech dataset, StyleTalk, with the following desired characteristics: when two current speeches have the same content but are spoken in different styles, their responses will be different. To teach LLMs to understand and respond properly to the speaking styles, we propose the Spoken-LLM framework that can model the linguistic content and the speaking styles. We train Spoken-LLM using the StyleTalk dataset and devise a two-stage training pipeline to help the Spoken-LLM better learn the speaking styles. Based on extensive experiments, we show that Spoken-LLM outperforms text-only baselines and prior speech LLMs methods.

Title: RhythmFormer: Extracting rPPG Signals Based on Hierarchical Temporal Periodic Transformer

• Authors: Bochao Zou, Zizheng Guo, Jiansheng Chen, Huimin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12788
• Pdf URL: https://arxiv.org/pdf/2402.12788
• Copy Paste: [[2402.12788]] RhythmFormer: Extracting rPPG Signals Based on Hierarchical Temporal Periodic Transformer(https://arxiv.org/abs/2402.12788)
• Keywords: transformer
• Abstract: Remote photoplethysmography (rPPG) is a non-contact method for detecting physiological signals based on facial videos, holding high potential in various applications such as healthcare, affective computing, anti-spoofing, etc. Due to the periodicity nature of rPPG, the long-range dependency capturing capacity of the Transformer was assumed to be advantageous for such signals. However, existing approaches have not conclusively demonstrated the superior performance of Transformer over traditional convolutional neural network methods, this gap may stem from a lack of thorough exploration of rPPG periodicity. In this paper, we propose RhythmFormer, a fully end-to-end transformer-based method for extracting rPPG signals by explicitly leveraging the quasi-periodic nature of rPPG. The core module, Hierarchical Temporal Periodic Transformer, hierarchically extracts periodic features from multiple temporal scales. It utilizes dynamic sparse attention based on periodicity in the temporal domain, allowing for fine-grained modeling of rPPG features. Furthermore, a fusion stem is proposed to guide self-attention to rPPG features effectively, and it can be easily transferred to existing methods to enhance their performance significantly. RhythmFormer achieves state-of-the-art performance with fewer parameters and reduced computational complexity in comprehensive experiments compared to previous approaches. The codes are available at https://github.com/zizheng-guo/RhythmFormer.

Title: Fair Classifiers Without Fair Training: An Influence-Guided Data Sampling Approach

• Authors: Jinlong Pang, Jialu Wang, Zhaowei Zhu, Yuanshun Yao, Chen Qian, Yang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12789
• Pdf URL: https://arxiv.org/pdf/2402.12789
• Copy Paste: [[2402.12789]] Fair Classifiers Without Fair Training: An Influence-Guided Data Sampling Approach(https://arxiv.org/abs/2402.12789)
• Keywords: fair
• Abstract: A fair classifier should ensure the benefit of people from different groups, while the group information is often sensitive and unsuitable for model training. Therefore, learning a fair classifier but excluding sensitive attributes in the training dataset is important. In this paper, we study learning fair classifiers without implementing fair training algorithms to avoid possible leakage of sensitive information. Our theoretical analyses validate the possibility of this approach, that traditional training on a dataset with an appropriate distribution shift can reduce both the upper bound for fairness disparity and model generalization error, indicating that fairness and accuracy can be improved simultaneously with simply traditional training. We then propose a tractable solution to progressively shift the original training data during training by sampling influential data, where the sensitive attribute of new data is not accessed in sampling or used in training. Extensive experiments on real-world data demonstrate the effectiveness of our proposed algorithm.

Title: From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition

• Authors: Kimji N. Pellano, Inga Strümke, Espen Alexander F. Ihlen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12790
• Pdf URL: https://arxiv.org/pdf/2402.12790
• Copy Paste: [[2402.12790]] From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition(https://arxiv.org/abs/2402.12790)
• Keywords: security
• Abstract: The advancement of deep learning in human activity recognition (HAR) using 3D skeleton data is critical for applications in healthcare, security, sports, and human-computer interaction. This paper tackles a well-known gap in the field, which is the lack of testing in the applicability and reliability of XAI evaluation metrics in the skeleton-based HAR domain. We have tested established XAI metrics namely faithfulness and stability on Class Activation Mapping (CAM) and Gradient-weighted Class Activation Mapping (Grad-CAM) to address this problem. The study also introduces a perturbation method that respects human biomechanical constraints to ensure realistic variations in human movement. Our findings indicate that \textit{faithfulness} may not be a reliable metric in certain contexts, such as with the EfficientGCN model. Conversely, stability emerges as a more dependable metric when there is slight input data perturbations. CAM and Grad-CAM are also found to produce almost identical explanations, leading to very similar XAI metric performance. This calls for the need for more diversified metrics and new XAI methods applied in skeleton-based HAR.

Title: Radar-Based Recognition of Static Hand Gestures in American Sign Language

• Authors: Christian Schuessler, Wenxuan Zhang, Johanna Bräunig, Marcel Hoffmann, Michael Stelzig, Martin Vossiek
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.12800
• Pdf URL: https://arxiv.org/pdf/2402.12800
• Copy Paste: [[2402.12800]] Radar-Based Recognition of Static Hand Gestures in American Sign Language(https://arxiv.org/abs/2402.12800)
• Keywords: privacy
• Abstract: In the fast-paced field of human-computer interaction (HCI) and virtual reality (VR), automatic gesture recognition has become increasingly essential. This is particularly true for the recognition of hand signs, providing an intuitive way to effortlessly navigate and control VR and HCI applications. Considering increased privacy requirements, radar sensors emerge as a compelling alternative to cameras. They operate effectively in low-light conditions without capturing identifiable human details, thanks to their lower resolution and distinct wavelength compared to visible light. While previous works predominantly deploy radar sensors for dynamic hand gesture recognition based on Doppler information, our approach prioritizes classification using an imaging radar that operates on spatial information, e.g. image-like data. However, generating large training datasets required for neural networks (NN) is a time-consuming and challenging process, often falling short of covering all potential scenarios. Acknowledging these challenges, this study explores the efficacy of synthetic data generated by an advanced radar ray-tracing simulator. This simulator employs an intuitive material model that can be adjusted to introduce data diversity. Despite exclusively training the NN on synthetic data, it demonstrates promising performance when put to the test with real measurement data. This emphasizes the practicality of our methodology in overcoming data scarcity challenges and advancing the field of automatic gesture recognition in VR and HCI applications.

Title: Few shot clinical entity recognition in three languages: Masked language models outperform LLM prompting

• Authors: Marco Naguib, Xavier Tannier, Aurélie Névéol
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12801
• Pdf URL: https://arxiv.org/pdf/2402.12801
• Copy Paste: [[2402.12801]] Few shot clinical entity recognition in three languages: Masked language models outperform LLM prompting(https://arxiv.org/abs/2402.12801)
• Keywords: large language model
• Abstract: Large Language Models are becoming the go-to solution for many natural language processing tasks, including in specialized domains where their few-shot capacities are expected to yield high performance in low-resource settings. Herein, we aim to assess the performance of Large Language Models for few shot clinical entity recognition in multiple languages. We evaluate named entity recognition in English, French and Spanish using 8 in-domain (clinical) and 6 out-domain gold standard corpora. We assess the performance of 10 auto-regressive language models using prompting and 16 masked language models used for text encoding in a biLSTM-CRF supervised tagger. We create a few-shot set-up by limiting the amount of annotated data available to 100 sentences. Our experiments show that although larger prompt-based models tend to achieve competitive F-measure for named entity recognition outside the clinical domain, this level of performance does not carry over to the clinical domain where lighter supervised taggers relying on masked language models perform better, even with the performance drop incurred from the few-shot set-up. In all experiments, the CO2 impact of masked language models is inferior to that of auto-regressive models. Results are consistent over the three languages and suggest that few-shot learning using Large language models is not production ready for named entity recognition in the clinical domain. Instead, models could be used for speeding-up the production of gold standard annotated data.

Title: SymBa: Symbolic Backward Chaining for Multi-step Natural Language Reasoning

• Authors: Jinu Lee, Wonseok Hwang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12806
• Pdf URL: https://arxiv.org/pdf/2402.12806
• Copy Paste: [[2402.12806]] SymBa: Symbolic Backward Chaining for Multi-step Natural Language Reasoning(https://arxiv.org/abs/2402.12806)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have recently demonstrated remarkable reasoning ability as in Chain-of-thought prompting, but faithful multi-step reasoning remains a challenge. We specifically focus on backward chaining, where the query is recursively decomposed using logical rules until proven. To address the limitations of current backward chaining implementations, we propose SymBa (Symbolic Backward Chaining). In SymBa, the symbolic top-down solver controls the entire proof process and the LLM is called to generate a single reasoning step only when the solver encounters a dead end. By this novel solver-LLM integration, while being able to produce an interpretable, structured proof, SymBa achieves significant improvement in performance, proof faithfulness, and efficiency in diverse multi-step reasoning benchmarks (ProofWriter, Birds-Electricity, GSM8k, CLUTRR-TF, ECtHR Article 6) compared to backward chaining baselines.

Title: Fine-Tuning, Prompting, In-Context Learning and Instruction-Tuning: How Many Labelled Samples Do We Need?

• Authors: Branislav Pecher, Ivan Srba, Maria Bielikova
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12819
• Pdf URL: https://arxiv.org/pdf/2402.12819
• Copy Paste: [[2402.12819]] Fine-Tuning, Prompting, In-Context Learning and Instruction-Tuning: How Many Labelled Samples Do We Need?(https://arxiv.org/abs/2402.12819)
• Keywords: large language model
• Abstract: When solving a task with limited labelled data, researchers can either use a general large language model without further update, or use the few examples to tune a specialised smaller model. When enough labels are available, the specialised models outperform the general ones on many NLP tasks. In this work, we aim to investigate how many labelled samples are required for the specialised models to achieve this superior performance, while taking the results variance into consideration. Observing the behaviour of prompting, in-context learning, fine-tuning and instruction-tuning, identifying their break-even points when increasing number of labelled training samples across three tasks of varying complexity, we find that the specialised models often need only few samples ($100-1000$) to be on par or better than the general ones. At the same time, the amount of required labelled data strongly depends on the task complexity and results variance.

Title: Identifying Factual Inconsistency in Summaries: Towards Effective Utilization of Large Language Model

• Authors: Liyan Xu, Zhenlin Su, Mo Yu, Jin Xu, Jinho D. Choi, Jie Zhou, Fei Liu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12821
• Pdf URL: https://arxiv.org/pdf/2402.12821
• Copy Paste: [[2402.12821]] Identifying Factual Inconsistency in Summaries: Towards Effective Utilization of Large Language Model(https://arxiv.org/abs/2402.12821)
• Keywords: large language model
• Abstract: Factual inconsistency poses a significant hurdle for the commercial deployment of abstractive summarizers. Under this Large Language Model (LLM) era, this work focuses around two important questions: what is the best way to leverage LLM for factual inconsistency detection, and how could we distill a smaller LLM with both high efficiency and efficacy? Three zero-shot paradigms are firstly proposed and evaluated across five diverse datasets: direct inference on the entire summary or each summary window; entity verification through question generation and answering. Experiments suggest that LLM itself is capable to resolve this task train-free under the proper paradigm design, surpassing strong trained baselines by 2.8% on average. To further promote practical utility, we then propose training strategies aimed at distilling smaller open-source LLM that learns to score the entire summary at once with high accuracy, which outperforms the zero-shot approaches by much larger LLM, serving as an effective and efficient ready-to-use scorer.

Title: PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs

• Authors: An Liu, Zonghan Yang, Zhenhe Zhang, Qingyuan Hu, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12835
• Pdf URL: https://arxiv.org/pdf/2402.12835
• Copy Paste: [[2402.12835]] PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs(https://arxiv.org/abs/2402.12835)
• Keywords: large language model
• Abstract: While Large language models (LLMs) have demonstrated considerable capabilities across various natural language tasks, they often fall short of the performance achieved by domain-specific state-of-the-art models. One potential approach to enhance domain-specific capabilities of LLMs involves fine-tuning them using corresponding datasets. However, this method can be both resource and time-intensive, and not applicable to closed-source commercial LLMs. In this paper, we propose Preference Adaptation for Enhancing Domain-specific Abilities of LLMs (PANDA), a method designed to augment the domain-specific capabilities of LLMs by leveraging insights from the response preference of expert models without requiring fine-tuning. Our experimental results reveal that PANDA significantly enhances the domain-specific ability of LLMs on text classification and interactive decision tasks. Moreover, LLM with PANDA even outperforms the expert model that being learned on 4 tasks of ScienceWorld. This finding highlights the potential of exploring tuning-free approaches to achieve weak-to-strong generalization.

Title: PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning

• Authors: Gyeongman Kim, Doohyuk Jang, Eunho Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12842
• Pdf URL: https://arxiv.org/pdf/2402.12842
• Copy Paste: [[2402.12842]] PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning(https://arxiv.org/abs/2402.12842)
• Keywords: generative, large language model
• Abstract: Recent advancements in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression. While knowledge distillation (KD) is a prominent method for this, research on KD for generative language models like LLMs is relatively sparse, and the approach of distilling student-friendly knowledge, which has shown promising performance in KD for classification models, remains unexplored in generative language models. To explore this approach, we propose PromptKD, a simple yet effective method that utilizes prompt tuning - for the first time in KD - to enable generative language models to transfer student-friendly knowledge. Unlike previous works in classification that require fine-tuning the entire teacher model for extracting student-friendly knowledge, PromptKD achieves similar effects by adding a small number of prompt tokens and tuning only the prompt with student guidance. Extensive experiments on instruction-following datasets using the GPT-2 model family show that PromptKD achieves state-of-the-art performance while adding only 0.0007% of the teacher's parameters as prompts. Further analysis suggests that distilling student-friendly knowledge alleviates exposure bias effectively throughout the entire training process, leading to performance enhancements.

Title: SolarPanel Segmentation :Self-Supervised Learning for Imperfect Datasets

• Authors: Sankarshanaa Sagaram, Aditya Kasliwal, Krish Didwania, Laven Srivastava, Pallavi Kailas, Ujjwal Verma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12843
• Pdf URL: https://arxiv.org/pdf/2402.12843
• Copy Paste: [[2402.12843]] SolarPanel Segmentation :Self-Supervised Learning for Imperfect Datasets(https://arxiv.org/abs/2402.12843)
• Keywords: robust, segmentation
• Abstract: The increasing adoption of solar energy necessitates advanced methodologies for monitoring and maintenance to ensure optimal performance of solar panel installations. A critical component in this context is the accurate segmentation of solar panels from aerial or satellite imagery, which is essential for identifying operational issues and assessing efficiency. This paper addresses the significant challenges in panel segmentation, particularly the scarcity of annotated data and the labour-intensive nature of manual annotation for supervised learning. We explore and apply Self-Supervised Learning (SSL) to solve these challenges. We demonstrate that SSL significantly enhances model generalization under various conditions and reduces dependency on manually annotated data, paving the way for robust and adaptable solar panel segmentation solutions.

Title: Instruction-tuned Language Models are Better Knowledge Learners

• Authors: Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, Srinivasan Iyer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12847
• Pdf URL: https://arxiv.org/pdf/2402.12847
• Copy Paste: [[2402.12847]] Instruction-tuned Language Models are Better Knowledge Learners(https://arxiv.org/abs/2402.12847)
• Keywords: large language model
• Abstract: In order for large language model (LLM)-based assistants to effectively adapt to evolving information needs, it must be possible to update their factual knowledge through continued training on new data. The standard recipe for doing so involves continued pre-training on new documents followed by instruction-tuning on question-answer (QA) pairs. However, we find that LLMs trained with this recipe struggle to answer questions, even though the perplexity of documents is minimized. We found that QA pairs are generally straightforward, while documents are more complex, weaving many factual statements together in an intricate manner. Therefore, we hypothesize that it is beneficial to expose LLMs to QA pairs before continued pre-training on documents so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed through questions. Based on this, we propose pre-instruction-tuning (PIT), a method that instruction-tunes on questions prior to training on documents. This contrasts with standard instruction-tuning, which learns how to extract knowledge after training on documents. Extensive experiments and ablation studies demonstrate that PIT significantly enhances the ability of LLMs to absorb knowledge from new documents, outperforming standard instruction-tuning by 17.8%.

Title: MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

• Authors: Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, Kang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12851
• Pdf URL: https://arxiv.org/pdf/2402.12851
• Copy Paste: [[2402.12851]] MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models(https://arxiv.org/abs/2402.12851)
• Keywords: large language model
• Abstract: Fine-tuning is often necessary to enhance the adaptability of Large Language Models (LLM) to downstream tasks. Nonetheless, the process of updating billions of parameters demands significant computational resources and training time, which poses a substantial obstacle to the widespread application of large-scale models in various scenarios. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prominent paradigm in recent research. However, current PEFT approaches that employ a limited set of global parameters (such as LoRA, which adds low-rank approximation matrices to all weights) face challenges in flexibly combining different computational modules in downstream tasks. In this work, we introduce a novel PEFT method: MoELoRA. We consider LoRA as Mixture of Experts (MoE), and to mitigate the random routing phenomenon observed in MoE, we propose the utilization of contrastive learning to encourage experts to learn distinct features. We conducted experiments on 11 tasks in math reasoning and common-sense reasoning benchmarks. With the same number of parameters, our approach outperforms LoRA significantly. In math reasoning, MoELoRA achieved an average performance that was 4.2% higher than LoRA, and demonstrated competitive performance compared to the 175B GPT-3.5 on several benchmarks.

Title: CCFC++: Enhancing Federated Clustering through Feature Decorrelation

• Authors: Jie Yan, Jing Liu, Yi-Zi Ning, Zhong-Yuan Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12852
• Pdf URL: https://arxiv.org/pdf/2402.12852
• Copy Paste: [[2402.12852]] CCFC++: Enhancing Federated Clustering through Feature Decorrelation(https://arxiv.org/abs/2402.12852)
• Keywords: robust, federate
• Abstract: In federated clustering, multiple data-holding clients collaboratively group data without exchanging raw data. This field has seen notable advancements through its marriage with contrastive learning, exemplified by Cluster-Contrastive Federated Clustering (CCFC). However, CCFC suffers from heterogeneous data across clients, leading to poor and unrobust performance. Our study conducts both empirical and theoretical analyses to understand the impact of heterogeneous data on CCFC. Findings indicate that increased data heterogeneity exacerbates dimensional collapse in CCFC, evidenced by increased correlations across multiple dimensions of the learned representations. To address this, we introduce a decorrelation regularizer to CCFC. Benefiting from the regularizer, the improved method effectively mitigates the detrimental effects of data heterogeneity, and achieves superior performance, as evidenced by a marked increase in NMI scores, with the gain reaching as high as 0.32 in the most pronounced case.

Title: Bounding Reconstruction Attack Success of Adversaries Without Data Priors

• Authors: Alexander Ziller, Anneliese Riess, Kristian Schwethelm, Tamara T. Mueller, Daniel Rueckert, Georgios Kaissis
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.12861
• Pdf URL: https://arxiv.org/pdf/2402.12861
• Copy Paste: [[2402.12861]] Bounding Reconstruction Attack Success of Adversaries Without Data Priors(https://arxiv.org/abs/2402.12861)
• Keywords: privacy, attack
• Abstract: Reconstruction attacks on machine learning (ML) models pose a strong risk of leakage of sensitive data. In specific contexts, an adversary can (almost) perfectly reconstruct training data samples from a trained model using the model's gradients. When training ML models with differential privacy (DP), formal upper bounds on the success of such reconstruction attacks can be provided. So far, these bounds have been formulated under worst-case assumptions that might not hold high realistic practicality. In this work, we provide formal upper bounds on reconstruction success under realistic adversarial settings against ML models trained with DP and support these bounds with empirical results. With this, we show that in realistic scenarios, (a) the expected reconstruction success can be bounded appropriately in different contexts and by different metrics, which (b) allows for a more educated choice of a privacy parameter.

Title: A Novel Protocol Using Captive Portals for FIDO2 Network Authentication

• Authors: Martiño Rivera-Dourado, Marcos Gestal, Alejandro Pazos, Jose Vázquez-Naya
• Subjects: cs.CR, cs.NI, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.12864
• Pdf URL: https://arxiv.org/pdf/2402.12864
• Copy Paste: [[2402.12864]] A Novel Protocol Using Captive Portals for FIDO2 Network Authentication(https://arxiv.org/abs/2402.12864)
• Keywords: security
• Abstract: FIDO2 authentication is starting to be applied in numerous web authentication services, aiming to replace passwords and their known vulnerabilities. However, this new authentication method has not been integrated yet with network authentication systems. In this paper, we introduce FIDO2CAP: FIDO2 Captive-portal Authentication Protocol. Our proposal describes a novel protocol for captive-portal network authentication using FIDO2 authenticators, as security keys and passkeys. For validating our proposal, we have developed a prototype of FIDO2CAP authentication in a mock scenario. Using this prototype, we performed an usability experiment with 15 real users. This work makes the first systematic approach for adapting network authentication to the new authentication paradigm relying on FIDO2 authentication.

Title: Backward Lens: Projecting Language Model Gradients into the Vocabulary Space

• Authors: Shahar Katz, Yonatan Belinkov, Mor Geva, Lior Wolf
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12865
• Pdf URL: https://arxiv.org/pdf/2402.12865
• Copy Paste: [[2402.12865]] Backward Lens: Projecting Language Model Gradients into the Vocabulary Space(https://arxiv.org/abs/2402.12865)
• Keywords: interpretability, transformer
• Abstract: Understanding how Transformer-based Language Models (LMs) learn and recall information is a key goal of the deep learning community. Recent interpretability methods project weights and hidden states obtained from the forward pass to the models' vocabularies, helping to uncover how information flows within LMs. In this work, we extend this methodology to LMs' backward pass and gradients. We first prove that a gradient matrix can be cast as a low-rank linear combination of its forward and backward passes' inputs. We then develop methods to project these gradients into vocabulary items and explore the mechanics of how new information is stored in the LMs' neurons.

Title: Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Data

• Authors: Dehai Min, Nan Hu, Rihui Jin, Nuo Lin, Jiaoyan Chen, Yongrui Chen, Yu Li, Guilin Qi, Yun Li, Nijun Li, Qianren Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12869
• Pdf URL: https://arxiv.org/pdf/2402.12869
• Copy Paste: [[2402.12869]] Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Data(https://arxiv.org/abs/2402.12869)
• Keywords: robust, large language model
• Abstract: Augmenting Large Language Models (LLMs) for Question Answering (QA) with domain specific data has attracted wide attention. However, domain data often exists in a hybrid format, including text and semi-structured tables, posing challenges for the seamless integration of information. Table-to-Text Generation is a promising solution by facilitating the transformation of hybrid data into a uniformly text-formatted corpus. Although this technique has been widely studied by the NLP community, there is currently no comparative analysis on how corpora generated by different table-to-text methods affect the performance of QA systems. In this paper, we address this research gap in two steps. First, we innovatively integrate table-to-text generation into the framework of enhancing LLM-based QA systems with domain hybrid data. Then, we utilize this framework in real-world industrial data to conduct extensive experiments on two types of QA systems (DSFT and RAG frameworks) with four representative methods: Markdown format, Template serialization, TPLM-based method, and LLM-based method. Based on the experimental results, we draw some empirical findings and explore the underlying reasons behind the success of some methods. We hope the findings of this work will provide a valuable reference for the academic and industrial communities in developing robust QA systems.

Title: Chain of Thought Empowers Transformers to Solve Inherently Serial Problems

• Authors: Zhiyuan Li, Hong Liu, Denny Zhou, Tengyu Ma
• Subjects: cs.LG, cs.CC, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.12875
• Pdf URL: https://arxiv.org/pdf/2402.12875
• Copy Paste: [[2402.12875]] Chain of Thought Empowers Transformers to Solve Inherently Serial Problems(https://arxiv.org/abs/2402.12875)
• Keywords: transformer, large language model
• Abstract: Instructing the model to generate a sequence of intermediate steps, a.k.a., a chain of thought (CoT), is a highly effective method to improve the accuracy of large language models (LLMs) on arithmetics and symbolic reasoning tasks. However, the mechanism behind CoT remains unclear. This work provides a theoretical understanding of the power of CoT for decoder-only transformers through the lens of expressiveness. Conceptually, CoT empowers the model with the ability to perform inherently serial computation, which is otherwise lacking in transformers, especially when depth is low. Given input length $n$, previous works have shown that constant-depth transformers with finite precision $\mathsf{poly}(n)$ embedding size can only solve problems in $\mathsf{TC}^0$ without CoT. We first show an even tighter expressiveness upper bound for constant-depth transformers with constant-bit precision, which can only solve problems in $\mathsf{AC}^0$, a proper subset of $ \mathsf{TC}^0$. However, with $T$ steps of CoT, constant-depth transformers using constant-bit precision and $O(\log n)$ embedding size can solve any problem solvable by boolean circuits of size $T$. Empirically, enabling CoT dramatically improves the accuracy for tasks that are hard for parallel computation, including the composition of permutation groups, iterated squaring, and circuit value problems, especially for low-depth transformers.

Title: Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Study

• Authors: Yuwen Yang, Yuxiang Lu, Suizhi Huang, Shalayiding Sirejiding, Hongtao Lu, Yue Ding
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.12876
• Pdf URL: https://arxiv.org/pdf/2402.12876
• Copy Paste: [[2402.12876]] Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Study(https://arxiv.org/abs/2402.12876)
• Keywords: federate
• Abstract: The innovative Federated Multi-Task Learning (FMTL) approach consolidates the benefits of Federated Learning (FL) and Multi-Task Learning (MTL), enabling collaborative model training on multi-task learning datasets. However, a comprehensive evaluation method, integrating the unique features of both FL and MTL, is currently absent in the field. This paper fills this void by introducing a novel framework, FMTL-Bench, for systematic evaluation of the FMTL paradigm. This benchmark covers various aspects at the data, model, and optimization algorithm levels, and comprises seven sets of comparative experiments, encapsulating a wide array of non-independent and identically distributed (Non-IID) data partitioning scenarios. We propose a systematic process for comparing baselines of diverse indicators and conduct a case study on communication expenditure, time, and energy consumption. Through our exhaustive experiments, we aim to provide valuable insights into the strengths and limitations of existing baseline methods, contributing to the ongoing discourse on optimal FMTL application in practical scenarios. The source code will be made available for results replication.

Title: Autism Detection in Speech - A Survey

• Authors: Nadine Probol, Margot Mieskes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12880
• Pdf URL: https://arxiv.org/pdf/2402.12880
• Copy Paste: [[2402.12880]] Autism Detection in Speech - A Survey(https://arxiv.org/abs/2402.12880)
• Keywords: transformer
• Abstract: There has been a range of studies of how autism is displayed in voice, speech, and language. We analyse studies from the biomedical, as well as the psychological domain, but also from the NLP domain in order to find linguistic, prosodic and acoustic cues that could indicate autism. Our survey looks at all three domains. We define autism and which comorbidities might influence the correct detection of the disorder. We especially look at observations such as verbal and semantic fluency, prosodic features, but also disfluencies and speaking rate. We also show word-based approaches and describe machine learning and transformer-based approaches both on the audio data as well as the transcripts. Lastly, we conclude, while there already is a lot of research, female patients seem to be severely under-researched. Also, most NLP research focuses on traditional machine learning methods instead of transformers which could be beneficial in this context. Additionally, we were unable to find research combining both features from audio and transcripts.

Title: GRAFFORD: A Benchmark Dataset for Testing the Knowledge of Object Affordances of Language and Vision Models

• Authors: Sayantan Adak, Daivik Agrawal, Animesh Mukherjee, Somak Aditya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12881
• Pdf URL: https://arxiv.org/pdf/2402.12881
• Copy Paste: [[2402.12881]] GRAFFORD: A Benchmark Dataset for Testing the Knowledge of Object Affordances of Language and Vision Models(https://arxiv.org/abs/2402.12881)
• Keywords: transformer
• Abstract: We investigate the knowledge of object affordances in pre-trained language models (LMs) and pre-trained Vision-Language models (VLMs). Transformers-based large pre-trained language models (PTLM) learn contextual representation from massive amounts of unlabeled text and are shown to perform impressively in downstream NLU tasks. In parallel, a growing body of literature shows that PTLMs fail inconsistently and non-intuitively, showing a lack of reasoning and grounding. To take a first step toward quantifying the effect of grounding (or lack thereof), we curate a novel and comprehensive dataset of object affordances -- GrAFFORD, characterized by 15 affordance classes. Unlike affordance datasets collected in vision and language domains, we annotate in-the-wild sentences with objects and affordances. Experimental results reveal that PTLMs exhibit limited reasoning abilities when it comes to uncommon object affordances. We also observe that pre-trained VLMs do not necessarily capture object affordances effectively. Through few-shot fine-tuning, we demonstrate improvement in affordance knowledge in PTLMs and VLMs. Our research contributes a novel dataset for language grounding tasks, and presents insights into LM capabilities, advancing the understanding of object affordances. Codes and data are available at https://github.com/sayantan11995/Affordance

Title: BFT-DSN: A Byzantine Fault Tolerant Decentralized Storage Network

• Authors: Hechuan Guo, Minghui Xu, Jiahao Zhang, Chunchi Liu, Rajiv Ranjan, Dongxiao Yu, Xiuzhen Cheng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.12889
• Pdf URL: https://arxiv.org/pdf/2402.12889
• Copy Paste: [[2402.12889]] BFT-DSN: A Byzantine Fault Tolerant Decentralized Storage Network(https://arxiv.org/abs/2402.12889)
• Keywords: robust
• Abstract: With the rapid development of blockchain and its applications, the amount of data stored on decentralized storage networks (DSNs) has grown exponentially. DSNs bring together affordable storage resources from around the world to provide robust, decentralized storage services for tens of thousands of decentralized applications (dApps). However, existing DSNs do not offer verifiability when implementing erasure coding for redundant storage, making them vulnerable to Byzantine encoders. Additionally, there is a lack of Byzantine fault-tolerant consensus for optimal resilience in DSNs. This paper introduces BFT-DSN, a Byzantine fault-tolerant decentralized storage network designed to address these challenges. BFT-DSN combines storage-weighted BFT consensus with erasure coding and incorporates homomorphic fingerprints and weighted threshold signatures for decentralized verification. The implementation of BFT-DSN demonstrates its comparable performance in terms of storage cost and latency as well as superior performance in Byzantine resilience when compared to existing industrial decentralized storage networks.

Title: RealCompo: Dynamic Equilibrium between Realism and Compositionality Improves Text-to-Image Diffusion Models

• Authors: Xinchen Zhang, Ling Yang, Yaqi Cai, Zhaochen Yu, Jiake Xie, Ye Tian, Minkai Xu, Yong Tang, Yujiu Yang, Bin Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12908
• Pdf URL: https://arxiv.org/pdf/2402.12908
• Copy Paste: [[2402.12908]] RealCompo: Dynamic Equilibrium between Realism and Compositionality Improves Text-to-Image Diffusion Models(https://arxiv.org/abs/2402.12908)
• Keywords: diffusion
• Abstract: Diffusion models have achieved remarkable advancements in text-to-image generation. However, existing models still have many difficulties when faced with multiple-object compositional generation. In this paper, we propose a new training-free and transferred-friendly text-to-image generation framework, namely RealCompo, which aims to leverage the advantages of text-to-image and layout-to-image models to enhance both realism and compositionality of the generated images. An intuitive and novel balancer is proposed to dynamically balance the strengths of the two models in denoising process, allowing plug-and-play use of any model without extra training. Extensive experiments show that our RealCompo consistently outperforms state-of-the-art text-to-image models and layout-to-image models in multiple-object compositional generation while keeping satisfactory realism and compositionality of the generated images. Code is available at https://github.com/YangLing0818/RealCompo

Title: Large Language Model-based Human-Agent Collaboration for Complex Task Solving

• Authors: Xueyang Feng, Zhi-Yuan Chen, Yujia Qin, Yankai Lin, Xu Chen, Zhiyuan Liu, Ji-Rong Wen
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.12914
• Pdf URL: https://arxiv.org/pdf/2402.12914
• Copy Paste: [[2402.12914]] Large Language Model-based Human-Agent Collaboration for Complex Task Solving(https://arxiv.org/abs/2402.12914)
• Keywords: large language model
• Abstract: In recent developments within the research community, the integration of Large Language Models (LLMs) in creating fully autonomous agents has garnered significant interest. Despite this, LLM-based agents frequently demonstrate notable shortcomings in adjusting to dynamic environments and fully grasping human needs. In this work, we introduce the problem of LLM-based human-agent collaboration for complex task-solving, exploring their synergistic potential. In addition, we propose a Reinforcement Learning-based Human-Agent Collaboration method, ReHAC. This approach includes a policy model designed to determine the most opportune stages for human intervention within the task-solving process. We construct a human-agent collaboration dataset to train this policy model in an offline reinforcement learning environment. Our validation tests confirm the model's effectiveness. The results demonstrate that the synergistic efforts of humans and LLM-based agents significantly improve performance in complex tasks, primarily through well-planned, limited human intervention. Datasets and code are available at: https://github.com/XueyangFeng/ReHAC.

Title: Advancements in Point Cloud-Based 3D Defect Detection and Classification for Industrial Systems: A Comprehensive Survey

• Authors: Anju Rani, Daniel Ortiz-Arroyo, Petar Durdevic
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12923
• Pdf URL: https://arxiv.org/pdf/2402.12923
• Copy Paste: [[2402.12923]] Advancements in Point Cloud-Based 3D Defect Detection and Classification for Industrial Systems: A Comprehensive Survey(https://arxiv.org/abs/2402.12923)
• Keywords: segmentation
• Abstract: In recent years, 3D point clouds (PCs) have gained significant attention due to their diverse applications across various fields such as computer vision (CV), condition monitoring, virtual reality, robotics, autonomous driving etc. Deep learning (DL) has proven effective in leveraging 3D PCs to address various challenges previously encountered in 2D vision. However, the application of deep neural networks (DNN) to process 3D PCs presents its own set of challenges. To address these challenges, numerous methods have been proposed. This paper provides an in-depth review of recent advancements in DL-based condition monitoring (CM) using 3D PCs, with a specific focus on defect shape classification and segmentation within industrial applications for operational and maintenance purposes. Recognizing the crucial role of these aspects in industrial maintenance, the paper provides insightful observations that offer perspectives on the strengths and limitations of the reviewed DL-based PC processing methods. This synthesis of knowledge aims to contribute to the understanding and enhancement of CM processes, particularly within the framework of remaining useful life (RUL), in industrial systems.

Title: CLIPping the Deception: Adapting Vision-Language Models for Universal Deepfake Detection

• Authors: Sohail Ahmed Khan, Duc-Tien Dang-Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12927
• Pdf URL: https://arxiv.org/pdf/2402.12927
• Copy Paste: [[2402.12927]] CLIPping the Deception: Adapting Vision-Language Models for Universal Deepfake Detection(https://arxiv.org/abs/2402.12927)
• Keywords: diffusion, generative
• Abstract: The recent advancements in Generative Adversarial Networks (GANs) and the emergence of Diffusion models have significantly streamlined the production of highly realistic and widely accessible synthetic content. As a result, there is a pressing need for effective general purpose detection mechanisms to mitigate the potential risks posed by deepfakes. In this paper, we explore the effectiveness of pre-trained vision-language models (VLMs) when paired with recent adaptation methods for universal deepfake detection. Following previous studies in this domain, we employ only a single dataset (ProGAN) in order to adapt CLIP for deepfake detection. However, in contrast to prior research, which rely solely on the visual part of CLIP while ignoring its textual component, our analysis reveals that retaining the text part is crucial. Consequently, the simple and lightweight Prompt Tuning based adaptation strategy that we employ outperforms the previous SOTA approach by 5.01% mAP and 6.61% accuracy while utilizing less than one third of the training data (200k images as compared to 720k). To assess the real-world applicability of our proposed models, we conduct a comprehensive evaluation across various scenarios. This involves rigorous testing on images sourced from 21 distinct datasets, including those generated by GANs-based, Diffusion-based and Commercial tools.

Title: GRAPHGINI: Fostering Individual and Group Fairness in Graph Neural Networks

• Authors: Anuj Kumar Sirohi, Anjali Gupta, Sayan Ranu, Sandeep Kumar, Amitabha Bagchi
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.12937
• Pdf URL: https://arxiv.org/pdf/2402.12937
• Copy Paste: [[2402.12937]] GRAPHGINI: Fostering Individual and Group Fairness in Graph Neural Networks(https://arxiv.org/abs/2402.12937)
• Keywords: fair
• Abstract: We address the growing apprehension that GNNs, in the absence of fairness constraints, might produce biased decisions that disproportionately affect underprivileged groups or individuals. Departing from previous work, we introduce for the first time a method for incorporating the Gini coefficient as a measure of fairness to be used within the GNN framework. Our proposal, GRAPHGINI, works with the two different goals of individual and group fairness in a single system, while maintaining high prediction accuracy. GRAPHGINI enforces individual fairness through learnable attention scores that help in aggregating more information through similar nodes. A heuristic-based maximum Nash social welfare constraint ensures the maximum possible group fairness. Both the individual fairness constraint and the group fairness constraint are stated in terms of a differentiable approximation of the Gini coefficient. This approximation is a contribution that is likely to be of interest even beyond the scope of the problem studied in this paper. Unlike other state-of-the-art, GRAPHGINI automatically balances all three optimization objectives (utility, individual, and group fairness) of the GNN and is free from any manual tuning of weight parameters. Extensive experimentation on real-world datasets showcases the efficacy of GRAPHGINI in making significant improvements in individual fairness compared to all currently available state-of-the-art methods while maintaining utility and group equality.

Title: Stochastic Approximation Approach to Federated Machine Learning

• Authors: Srihari P V, Bharath Bhikkaji
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12945
• Pdf URL: https://arxiv.org/pdf/2402.12945
• Copy Paste: [[2402.12945]] Stochastic Approximation Approach to Federated Machine Learning(https://arxiv.org/abs/2402.12945)
• Keywords: robust, federate
• Abstract: This paper examines Federated learning (FL) in a Stochastic Approximation (SA) framework. FL is a collaborative way to train neural network models across various participants or clients without centralizing their data. Each client will train a model on their respective data and send the weights across to a the server periodically for aggregation. The server aggregates these weights which are then used by the clients to re-initialize their neural network and continue the training. SA is an iterative algorithm that uses approximate sample gradients and tapering step size to locate a minimizer of a cost function. In this paper the clients use a stochastic approximation iterate to update the weights of its neural network. It is shown that the aggregated weights track an autonomous ODE. Numerical simulations are performed and the results are compared with standard algorithms like FedAvg and FedProx. It is observed that the proposed algorithm is robust and gives more reliable estimates of the weights, in particular when the clients data are not identically distributed.

Title: Cell Graph Transformer for Nuclei Classification

• Authors: Wei Lou, Guanbin Li, Xiang Wan, Haofeng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12946
• Pdf URL: https://arxiv.org/pdf/2402.12946
• Copy Paste: [[2402.12946]] Cell Graph Transformer for Nuclei Classification(https://arxiv.org/abs/2402.12946)
• Keywords: transformer
• Abstract: Nuclei classification is a critical step in computer-aided diagnosis with histopathology images. In the past, various methods have employed graph neural networks (GNN) to analyze cell graphs that model inter-cell relationships by considering nuclei as vertices. However, they are limited by the GNN mechanism that only passes messages among local nodes via fixed edges. To address the issue, we develop a cell graph transformer (CGT) that treats nodes and edges as input tokens to enable learnable adjacency and information exchange among all nodes. Nevertheless, training the transformer with a cell graph presents another challenge. Poorly initialized features can lead to noisy self-attention scores and inferior convergence, particularly when processing the cell graphs with numerous connections. Thus, we further propose a novel topology-aware pretraining method that leverages a graph convolutional network (GCN) to learn a feature extractor. The pre-trained features may suppress unreasonable correlations and hence ease the finetuning of CGT. Experimental results suggest that the proposed cell graph transformer with topology-aware pretraining significantly improves the nuclei classification results, and achieves the state-of-the-art performance. Code and models are available at https://github.com/lhaof/CGT

Title: GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick

• Authors: Jiayi Fu, Xuandong Zhao, Ruihan Yang, Yuansen Zhang, Jiangjie Chen, Yanghua Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12948
• Pdf URL: https://arxiv.org/pdf/2402.12948
• Copy Paste: [[2402.12948]] GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick(https://arxiv.org/abs/2402.12948)
• Keywords: watermark, large language model
• Abstract: Large language models (LLMs) excellently generate human-like text, but also raise concerns about misuse in fake news and academic dishonesty. Decoding-based watermark, particularly the GumbelMax-trick-based watermark(GM watermark), is a standout solution for safeguarding machine-generated texts due to its notable detectability. However, GM watermark encounters a major challenge with generation diversity, always yielding identical outputs for the same prompt, negatively impacting generation diversity and user experience. To overcome this limitation, we propose a new type of GM watermark, the Logits-Addition watermark, and its three variants, specifically designed to enhance diversity. Among these, the GumbelSoft watermark (a softmax variant of the Logits-Addition watermark) demonstrates superior performance in high diversity settings, with its AUROC score outperforming those of the two alternative variants by 0.1 to 0.3 and surpassing other decoding-based watermarking methods by a minimum of 0.1.

Title: Conditional Logical Message Passing Transformer for Complex Query Answering

• Authors: Chongzhi Zhang, Zhiping Peng, Junhao Zheng, Qianli Ma
• Subjects: cs.LG, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2402.12954
• Pdf URL: https://arxiv.org/pdf/2402.12954
• Copy Paste: [[2402.12954]] Conditional Logical Message Passing Transformer for Complex Query Answering(https://arxiv.org/abs/2402.12954)
• Keywords: transformer
• Abstract: Complex Query Answering (CQA) over Knowledge Graphs (KGs) is a challenging task. Given that KGs are usually incomplete, neural models are proposed to solve CQA by performing multi-hop logical reasoning. However, most of them cannot perform well on both one-hop and multi-hop queries simultaneously. Recent work proposes a logical message passing mechanism based on the pre-trained neural link predictors. While effective on both one-hop and multi-hop queries, it ignores the difference between the constant and variable nodes in a query graph. In addition, during the node embedding update stage, this mechanism cannot dynamically measure the importance of different messages, and whether it can capture the implicit logical dependencies related to a node and received messages remains unclear. In this paper, we propose Conditional Logical Message Passing Transformer (CLMPT), which considers the difference between constants and variables in the case of using pre-trained neural link predictors and performs message passing conditionally on the node type. We empirically verified that this approach can reduce computational costs without affecting performance. Furthermore, CLMPT uses the transformer to aggregate received messages and update the corresponding node embedding. Through the self-attention mechanism, CLMPT can assign adaptive weights to elements in an input set consisting of received messages and the corresponding node and explicitly model logical dependencies between various elements. Experimental results show that CLMPT is a new state-of-the-art neural CQA model.

Title: Prompt Stealing Attacks Against Large Language Models

• Authors: Zeyang Sha, Yang Zhang
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12959
• Pdf URL: https://arxiv.org/pdf/2402.12959
• Copy Paste: [[2402.12959]] Prompt Stealing Attacks Against Large Language Models(https://arxiv.org/abs/2402.12959)
• Keywords: security, attack, steal, large language model
• Abstract: The increasing reliance on large language models (LLMs) such as ChatGPT in various fields emphasizes the importance of ``prompt engineering,'' a technology to improve the quality of model outputs. With companies investing significantly in expert prompt engineers and educational resources rising to meet market demand, designing high-quality prompts has become an intriguing challenge. In this paper, we propose a novel attack against LLMs, named prompt stealing attacks. Our proposed prompt stealing attack aims to steal these well-designed prompts based on the generated answers. The prompt stealing attack contains two primary modules: the parameter extractor and the prompt reconstruction. The goal of the parameter extractor is to figure out the properties of the original prompts. We first observe that most prompts fall into one of three categories: direct prompt, role-based prompt, and in-context prompt. Our parameter extractor first tries to distinguish the type of prompts based on the generated answers. Then, it can further predict which role or how many contexts are used based on the types of prompts. Following the parameter extractor, the prompt reconstructor can be used to reconstruct the original prompts based on the generated answers and the extracted features. The final goal of the prompt reconstructor is to generate the reversed prompts, which are similar to the original prompts. Our experimental results show the remarkable performance of our proposed attacks. Our proposed attacks add a new dimension to the study of prompt engineering and call for more attention to the security issues on LLMs.

Title: MapTrack: Tracking in the Map

• Authors: Fei Wang, Ruohui Zhang, Chenglin Chen, Min Yang, Yun Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12968
• Pdf URL: https://arxiv.org/pdf/2402.12968
• Copy Paste: [[2402.12968]] MapTrack: Tracking in the Map(https://arxiv.org/abs/2402.12968)
• Keywords: robust
• Abstract: Multi-Object Tracking (MOT) aims to maintain stable and uninterrupted trajectories for each target. Most state-of-the-art approaches first detect objects in each frame and then implement data association between new detections and existing tracks using motion models and appearance similarities. Despite achieving satisfactory results, occlusion and crowds can easily lead to missing and distorted detections, followed by missing and false associations. In this paper, we first revisit the classic tracker DeepSORT, enhancing its robustness over crowds and occlusion significantly by placing greater trust in predictions when detections are unavailable or of low quality in crowded and occluded scenes. Specifically, we propose a new framework comprising of three lightweight and plug-and-play algorithms: the probability map, the prediction map, and the covariance adaptive Kalman filter. The probability map identifies whether undetected objects have genuinely disappeared from view (e.g., out of the image or entered a building) or are only temporarily undetected due to occlusion or other reasons. Trajectories of undetected targets that are still within the probability map are extended by state estimations directly. The prediction map determines whether an object is in a crowd, and we prioritize state estimations over observations when severe deformation of observations occurs, accomplished through the covariance adaptive Kalman filter. The proposed method, named MapTrack, achieves state-of-the-art results on popular multi-object tracking benchmarks such as MOT17 and MOT20. Despite its superior performance, our method remains simple, online, and real-time. The code will be open-sourced later.

Title: GlórIA - A Generative and Open Large Language Model for Portuguese

• Authors: Ricardo Lopes, João Magalhães, David Semedo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12969
• Pdf URL: https://arxiv.org/pdf/2402.12969
• Copy Paste: [[2402.12969]] GlórIA - A Generative and Open Large Language Model for Portuguese(https://arxiv.org/abs/2402.12969)
• Keywords: robust, generative, large language model
• Abstract: Significant strides have been made in natural language tasks, largely attributed to the emergence of powerful large language models (LLMs). These models, pre-trained on extensive and diverse corpora, have become increasingly capable of comprehending the intricacies of language. Despite the abundance of LLMs for many high-resource languages, the availability of such models remains limited for European Portuguese. We introduce Gl\'orIA, a robust European Portuguese decoder LLM. To pre-train Gl\'orIA, we assembled a comprehensive PT-PT text corpus comprising 35 billion tokens from various sources. We present our pre-training methodology, followed by an assessment of the model's effectiveness on multiple downstream tasks. Additionally, to evaluate our models' language modeling capabilities, we introduce CALAME-PT (Context-Aware LAnguage Modeling Evaluation for Portuguese), the first Portuguese zero-shot language-modeling benchmark. Evaluation shows that Gl\'orIA significantly outperforms existing open PT decoder models in language modeling and that it can generate sound, knowledge-rich, and coherent PT-PT text. The model also exhibits strong potential for various downstream tasks.

Title: Visual Style Prompting with Swapping Self-Attention

• Authors: Jaeseok Jeong, Junho Kim, Yunjey Choi, Gayoung Lee, Youngjung Uh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12974
• Pdf URL: https://arxiv.org/pdf/2402.12974
• Copy Paste: [[2402.12974]] Visual Style Prompting with Swapping Self-Attention(https://arxiv.org/abs/2402.12974)
• Keywords: diffusion
• Abstract: In the evolving domain of text-to-image generation, diffusion models have emerged as powerful tools in content creation. Despite their remarkable capability, existing models still face challenges in achieving controlled generation with a consistent style, requiring costly fine-tuning or often inadequately transferring the visual elements due to content leakage. To address these challenges, we propose a novel approach, \ours, to produce a diverse range of images while maintaining specific style elements and nuances. During the denoising process, we keep the query from original features while swapping the key and value with those from reference features in the late self-attention layers. This approach allows for the visual style prompting without any fine-tuning, ensuring that generated images maintain a faithful style. Through extensive evaluation across various styles and text prompts, our method demonstrates superiority over existing approaches, best reflecting the style of the references and ensuring that resulting images match the text prompts most accurately. Our project page is available \href{https://curryjung.github.io/VisualStylePrompt/}{here}.

Title: The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis

• Authors: Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba O. Alabi, Xiaoyu Shen, Dietrich Klakow, Marius Mosbach
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12976
• Pdf URL: https://arxiv.org/pdf/2402.12976
• Copy Paste: [[2402.12976]] The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis(https://arxiv.org/abs/2402.12976)
• Keywords: large language model
• Abstract: In-context learning is a popular inference strategy where large language models solve a task using only a few labelled demonstrations without needing any parameter updates. Compared to work on monolingual (English) in-context learning, multilingual in-context learning is under-explored, and we lack an in-depth understanding of the role of demonstrations in this context. To address this gap, we conduct a multidimensional analysis of multilingual in-context learning, experimenting with 5 models from different model families, 9 datasets covering classification and generation tasks, and 56 typologically diverse languages. Our results reveal that the effectiveness of demonstrations varies significantly across models, tasks, and languages. We also find that Llama 2-Chat, GPT-3.5, and GPT-4 are largely insensitive to the quality of demonstrations. Instead, a carefully crafted template often eliminates the benefits of demonstrations for some tasks and languages altogether. These findings show that the importance of demonstrations might be overestimated. Our work highlights the need for granular evaluation across multiple axes towards a better understanding of in-context learning.

Title: Can GNN be Good Adapter for LLMs?

• Authors: Xuanwen Huang, Kaiqiao Han, Yang Yang, Dezheng Bao, Quanjin Tao, Ziwei Chai, Qi Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12984
• Pdf URL: https://arxiv.org/pdf/2402.12984
• Copy Paste: [[2402.12984]] Can GNN be Good Adapter for LLMs?(https://arxiv.org/abs/2402.12984)
• Keywords: large language model
• Abstract: Recently, large language models (LLMs) have demonstrated superior capabilities in understanding and zero-shot learning on textual data, promising significant advances for many text-related domains. In the graph domain, various real-world scenarios also involve textual data, where tasks and node features can be described by text. These text-attributed graphs (TAGs) have broad applications in social media, recommendation systems, etc. Thus, this paper explores how to utilize LLMs to model TAGs. Previous methods for TAG modeling are based on million-scale LMs. When scaled up to billion-scale LLMs, they face huge challenges in computational costs. Additionally, they also ignore the zero-shot inference capabilities of LLMs. Therefore, we propose GraphAdapter, which uses a graph neural network (GNN) as an efficient adapter in collaboration with LLMs to tackle TAGs. In terms of efficiency, the GNN adapter introduces only a few trainable parameters and can be trained with low computation costs. The entire framework is trained using auto-regression on node text (next token prediction). Once trained, GraphAdapter can be seamlessly fine-tuned with task-specific prompts for various downstream tasks. Through extensive experiments across multiple real-world TAGs, GraphAdapter based on Llama 2 gains an average improvement of approximately 5\% in terms of node classification. Furthermore, GraphAdapter can also adapt to other language models, including RoBERTa, GPT-2. The promising results demonstrate that GNNs can serve as effective adapters for LLMs in TAG modeling.

Title: Towards Robust Graph Incremental Learning on Evolving Graphs

• Authors: Junwei Su, Difan Zou, Zijun Zhang, Chuan Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12987
• Pdf URL: https://arxiv.org/pdf/2402.12987
• Copy Paste: [[2402.12987]] Towards Robust Graph Incremental Learning on Evolving Graphs(https://arxiv.org/abs/2402.12987)
• Keywords: robust
• Abstract: Incremental learning is a machine learning approach that involves training a model on a sequence of tasks, rather than all tasks at once. This ability to learn incrementally from a stream of tasks is crucial for many real-world applications. However, incremental learning is a challenging problem on graph-structured data, as many graph-related problems involve prediction tasks for each individual node, known as Node-wise Graph Incremental Learning (NGIL). This introduces non-independent and non-identically distributed characteristics in the sample data generation process, making it difficult to maintain the performance of the model as new tasks are added. In this paper, we focus on the inductive NGIL problem, which accounts for the evolution of graph structure (structural shift) induced by emerging tasks. We provide a formal formulation and analysis of the problem, and propose a novel regularization-based technique called Structural-Shift-Risk-Mitigation (SSRM) to mitigate the impact of the structural shift on catastrophic forgetting of the inductive NGIL problem. We show that the structural shift can lead to a shift in the input distribution for the existing tasks, and further lead to an increased risk of catastrophic forgetting. Through comprehensive empirical studies with several benchmark datasets, we demonstrate that our proposed method, Structural-Shift-Risk-Mitigation (SSRM), is flexible and easy to adapt to improve the performance of state-of-the-art GNN incremental learning frameworks in the inductive setting.

Title: TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification

• Authors: Martin Gubri, Dennis Ulmer, Hwaran Lee, Sangdoo Yun, Seong Joon Oh
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.12991
• Pdf URL: https://arxiv.org/pdf/2402.12991
• Copy Paste: [[2402.12991]] TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification(https://arxiv.org/abs/2402.12991)
• Keywords: protect, large language model
• Abstract: Large Language Model (LLM) services and models often come with legal rules on who can use them and how they must use them. Assessing the compliance of the released LLMs is crucial, as these rules protect the interests of the LLM contributor and prevent misuse. In this context, we describe the novel problem of Black-box Identity Verification (BBIV). The goal is to determine whether a third-party application uses a certain LLM through its chat function. We propose a method called Targeted Random Adversarial Prompt (TRAP) that identifies the specific LLM in use. We repurpose adversarial suffixes, originally proposed for jailbreaking, to get a pre-defined answer from the target LLM, while other models give random answers. TRAP detects the target LLMs with over 95% true positive rate at under 0.2% false positive rate even after a single interaction. TRAP remains effective even if the LLM has minor changes that do not significantly alter the original function.

Title: Phonotactic Complexity across Dialects

• Authors: Ryan Soh-Eun Shim, Kalvin Chang, David R. Mortensen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12998
• Pdf URL: https://arxiv.org/pdf/2402.12998
• Copy Paste: [[2402.12998]] Phonotactic Complexity across Dialects(https://arxiv.org/abs/2402.12998)
• Keywords: fair
• Abstract: Received wisdom in linguistic typology holds that if the structure of a language becomes more complex in one dimension, it will simplify in another, building on the assumption that all languages are equally complex (Joseph and Newmeyer, 2012). We study this claim on a micro-level, using a tightly-controlled sample of Dutch dialects (across 366 collection sites) and Min dialects (across 60 sites), which enables a more fair comparison across varieties. Even at the dialect level, we find empirical evidence for a tradeoff between word length and a computational measure of phonotactic complexity from a LSTM-based phone-level language model-a result previously documented only at the language level. A generalized additive model (GAM) shows that dialects with low phonotactic complexity concentrate around the capital regions, which we hypothesize to correspond to prior hypotheses that language varieties of greater or more diverse populations show reduced phonotactic complexity. We also experiment with incorporating the auxiliary task of predicting syllable constituency, but do not find an increase in the negative correlation observed.

Title: Investigating the Impact of Model Instability on Explanations and Uncertainty

• Authors: Sara Vera Marjanović, Isabelle Augenstein, Christina Lioma
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13006
• Pdf URL: https://arxiv.org/pdf/2402.13006
• Copy Paste: [[2402.13006]] Investigating the Impact of Model Instability on Explanations and Uncertainty(https://arxiv.org/abs/2402.13006)
• Keywords: robust, transformer
• Abstract: Explainable AI methods facilitate the understanding of model behaviour, yet, small, imperceptible perturbations to inputs can vastly distort explanations. As these explanations are typically evaluated holistically, before model deployment, it is difficult to assess when a particular explanation is trustworthy. Some studies have tried to create confidence estimators for explanations, but none have investigated an existing link between uncertainty and explanation quality. We artificially simulate epistemic uncertainty in text input by introducing noise at inference time. In this large-scale empirical study, we insert different levels of noise perturbations and measure the effect on the output of pre-trained language models and different uncertainty metrics. Realistic perturbations have minimal effect on performance and explanations, yet masking has a drastic effect. We find that high uncertainty doesn't necessarily imply low explanation plausibility; the correlation between the two metrics can be moderately positive when noise is exposed during the training process. This suggests that noise-augmented models may be better at identifying salient tokens when uncertain. Furthermore, when predictive and epistemic uncertainty measures are over-confident, the robustness of a saliency map to perturbation can indicate model stability issues. Integrated Gradients shows the overall greatest robustness to perturbation, while still showing model-specific patterns in performance; however, this phenomenon is limited to smaller Transformer-based language models.

Title: Code Needs Comments: Enhancing Code LLMs with Comment Augmentation

• Authors: Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, Dahua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13013
• Pdf URL: https://arxiv.org/pdf/2402.13013
• Copy Paste: [[2402.13013]] Code Needs Comments: Enhancing Code LLMs with Comment Augmentation(https://arxiv.org/abs/2402.13013)
• Keywords: large language model
• Abstract: The programming skill is one crucial ability for Large Language Models (LLMs), necessitating a deep understanding of programming languages (PLs) and their correlation with natural languages (NLs). We examine the impact of pre-training data on code-focused LLMs' performance by assessing the comment density as a measure of PL-NL alignment. Given the scarcity of code-comment aligned data in pre-training corpora, we introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language. We conducted experiments on three code-focused LLMs and observed consistent improvements in performance on two widely-used programming skill benchmarks. Notably, the model trained on the augmented data outperformed both the model used for generating comments and the model further trained on the data without augmentation.

Title: Understanding the effects of language-specific class imbalance in multilingual fine-tuning

• Authors: Vincent Jung, Lonneke van der Plas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13016
• Pdf URL: https://arxiv.org/pdf/2402.13016
• Copy Paste: [[2402.13016]] Understanding the effects of language-specific class imbalance in multilingual fine-tuning(https://arxiv.org/abs/2402.13016)
• Keywords: transformer, large language model
• Abstract: We study the effect of one type of imbalance often present in real-life multilingual classification datasets: an uneven distribution of labels across languages. We show evidence that fine-tuning a transformer-based Large Language Model (LLM) on a dataset with this imbalance leads to worse performance, a more pronounced separation of languages in the latent space, and the promotion of uninformative features. We modify the traditional class weighing approach to imbalance by calculating class weights separately for each language and show that this helps mitigate those detrimental effects. These results create awareness of the negative effects of language-specific class imbalance in multilingual fine-tuning and the way in which the model learns to rely on the separation of languages to perform the task.

Title: CFEVER: A Chinese Fact Extraction and VERification Dataset

• Authors: Ying-Jia Lin, Chun-Yi Lin, Chia-Jen Yeh, Yi-Ting Li, Yun-Yu Hu, Chih-Hao Hsu, Mei-Feng Lee, Hung-Yu Kao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13025
• Pdf URL: https://arxiv.org/pdf/2402.13025
• Copy Paste: [[2402.13025]] CFEVER: A Chinese Fact Extraction and VERification Dataset(https://arxiv.org/abs/2402.13025)
• Keywords: extraction
• Abstract: We present CFEVER, a Chinese dataset designed for Fact Extraction and VERification. CFEVER comprises 30,012 manually created claims based on content in Chinese Wikipedia. Each claim in CFEVER is labeled as "Supports", "Refutes", or "Not Enough Info" to depict its degree of factualness. Similar to the FEVER dataset, claims in the "Supports" and "Refutes" categories are also annotated with corresponding evidence sentences sourced from single or multiple pages in Chinese Wikipedia. Our labeled dataset holds a Fleiss' kappa value of 0.7934 for five-way inter-annotator agreement. In addition, through the experiments with the state-of-the-art approaches developed on the FEVER dataset and a simple baseline for CFEVER, we demonstrate that our dataset is a new rigorous benchmark for factual extraction and verification, which can be further used for developing automated systems to alleviate human fact-checking efforts. CFEVER is available at https://ikmlab.github.io/CFEVER.

Title: Learning to Check: Unleashing Potentials for Self-Correction in Large Language Models

• Authors: Che Zhang, Zhenyang Xiao, Chengcheng Han, Yixin Lian, Yuejian Fang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13035
• Pdf URL: https://arxiv.org/pdf/2402.13035
• Copy Paste: [[2402.13035]] Learning to Check: Unleashing Potentials for Self-Correction in Large Language Models(https://arxiv.org/abs/2402.13035)
• Keywords: large language model
• Abstract: Large language models (LLMs) have made significant strides in reasoning capabilities, with ongoing efforts to refine their reasoning through self-correction. However, recent studies suggest that self-correction can be limited or even counterproductive without external accurate knowledge, raising questions about the limits and effectiveness of self-correction. In this paper, we aim to enhance LLM's self-checking capabilities by meticulously designing training data, thereby improving the accuracy of self-correction. We conduct a detailed analysis of error types in mathematical reasoning and develop a tailored prompt, termed ``Step CoT Check''. Then we construct a checking-correction dataset for training models. After integrating the original CoT data and checking-correction data for training, we observe that models could improve their self-checking capabilities, thereby enhancing their self-correction capacity and eliminating the need for external feedback or ground truth labels to ascertain the endpoint of correction. We compare the performance of models fine-tuned with the ``Step CoT Check'' prompt against those refined using other promps within the context of checking-correction data. The ``Step CoT Check'' outperforms the other two check formats in model with lager parameters, providing more precise feedback thus achieving a higher rate of correctness. For reproducibility, all the datasets and codes are provided in \url{https://github.com/bammt/Learn-to-check}.

Title: SiLLM: Large Language Models for Simultaneous Machine Translation

• Authors: Shoutao Guo, Shaolei Zhang, Zhengrui Ma, Min Zhang, Yang Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13036
• Pdf URL: https://arxiv.org/pdf/2402.13036
• Copy Paste: [[2402.13036]] SiLLM: Large Language Models for Simultaneous Machine Translation(https://arxiv.org/abs/2402.13036)
• Keywords: transformer, large language model
• Abstract: Simultaneous Machine Translation (SiMT) generates translations while reading the source sentence, necessitating a policy to determine the optimal timing for reading and generating words. Despite the remarkable performance achieved by Large Language Models (LLM) across various NLP tasks, existing SiMT methods predominantly focus on conventional transformers, employing a single model to concurrently determine the policy and generate the translations. However, given the complexity of SiMT, it is challenging to effectively address both tasks with a single model. Therefore, there is a need to decouple the SiMT task into policy-decision and translation sub-tasks. We propose SiLLM, which delegates the two sub-tasks to separate agents, thereby incorporating LLM into SiMT. The policy-decision agent is managed by a conventional SiMT model, responsible for determining the translation policy. The translation agent, leveraging the capabilities of LLM, generates translation using the partial source sentence. The two agents collaborate to accomplish SiMT. To facilitate the application of token-level policies determined by conventional SiMT models to LLM, we propose a word-level policy adapted for LLM. Experiments on two datasets demonstrate that, with a small amount of data for fine-tuning LLM, SiLLM attains state-of-the-art performance.

Title: Text-Guided Molecule Generation with Diffusion Language Model

• Authors: Haisong Gong, Qiang Liu, Shu Wu, Liang Wang
• Subjects: cs.LG, cs.AI, cs.CE, cs.CL, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2402.13040
• Pdf URL: https://arxiv.org/pdf/2402.13040
• Copy Paste: [[2402.13040]] Text-Guided Molecule Generation with Diffusion Language Model(https://arxiv.org/abs/2402.13040)
• Keywords: diffusion
• Abstract: Text-guided molecule generation is a task where molecules are generated to match specific textual descriptions. Recently, most existing SMILES-based molecule generation methods rely on an autoregressive architecture. In this work, we propose the Text-Guided Molecule Generation with Diffusion Language Model (TGM-DLM), a novel approach that leverages diffusion models to address the limitations of autoregressive methods. TGM-DLM updates token embeddings within the SMILES string collectively and iteratively, using a two-phase diffusion generation process. The first phase optimizes embeddings from random noise, guided by the text description, while the second phase corrects invalid SMILES strings to form valid molecular representations. We demonstrate that TGM-DLM outperforms MolT5-Base, an autoregressive model, without the need for additional data resources. Our findings underscore the remarkable effectiveness of TGM-DLM in generating coherent and precise molecules with specific properties, opening new avenues in drug discovery and related scientific domains. Code will be released at: https://github.com/Deno-V/tgm-dlm.

Title: Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries

• Authors: Seanie Lee, Jianpeng Chen, Joris Driesen, Alexandru Coca, Anders Johannsen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13043
• Pdf URL: https://arxiv.org/pdf/2402.13043
• Copy Paste: [[2402.13043]] Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries(https://arxiv.org/abs/2402.13043)
• Keywords: large language model
• Abstract: Few-shot dialogue state tracking (DST) with Large Language Models (LLM) relies on an effective and efficient conversation retriever to find similar in-context examples for prompt learning. Previous works use raw dialogue context as search keys and queries, and a retriever is fine-tuned with annotated dialogues to achieve superior performance. However, the approach is less suited for scaling to new domains or new annotation languages, where fine-tuning data is unavailable. To address this problem, we handle the task of conversation retrieval based on text summaries of the conversations. A LLM-based conversation summarizer is adopted for query and key generation, which enables effective maximum inner product search. To avoid the extra inference cost brought by LLM-based conversation summarization, we further distill a light-weight conversation encoder which produces query embeddings without decoding summaries for test conversations. We validate our retrieval approach on MultiWOZ datasets with GPT-Neo-2.7B and LLaMA-7B/30B. The experimental results show a significant improvement over relevant baselines in real few-shot DST settings.

Title: Stable Knowledge Editing in Large Language Models

• Authors: Zihao Wei, Liang Pang, Hanxing Ding, Jingcheng Deng, Huawei Shen, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13048
• Pdf URL: https://arxiv.org/pdf/2402.13048
• Copy Paste: [[2402.13048]] Stable Knowledge Editing in Large Language Models(https://arxiv.org/abs/2402.13048)
• Keywords: large language model
• Abstract: Efficient knowledge editing of large language models is crucial for replacing obsolete information or incorporating specialized knowledge on a large scale. However, previous methods implicitly assume that knowledge is localized and isolated within the model, an assumption that oversimplifies the interconnected nature of model knowledge. The premise of localization results in an incomplete knowledge editing, whereas an isolated assumption may impair both other knowledge and general abilities. It introduces instability to the performance of the knowledge editing method. To transcend these assumptions, we introduce StableKE, a method adopts a novel perspective based on knowledge augmentation rather than knowledge localization. To overcome the expense of human labeling, StableKE integrates two automated knowledge augmentation strategies: Semantic Paraphrase Enhancement strategy, which diversifies knowledge descriptions to facilitate the teaching of new information to the model, and Contextual Description Enrichment strategy, expanding the surrounding knowledge to prevent the forgetting of related information. StableKE surpasses other knowledge editing methods, demonstrating stability both edited knowledge and multi-hop knowledge, while also preserving unrelated knowledge and general abilities. Moreover, StableKE can edit knowledge on ChatGPT.

Title: Identifying Semantic Induction Heads to Understand In-Context Learning

• Authors: Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Xipeng Qiu, Dahua Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13055
• Pdf URL: https://arxiv.org/pdf/2402.13055
• Copy Paste: [[2402.13055]] Identifying Semantic Induction Heads to Understand In-Context Learning(https://arxiv.org/abs/2402.13055)
• Keywords: transformer, large language model
• Abstract: Although large language models (LLMs) have demonstrated remarkable performance, the lack of transparency in their inference logic raises concerns about their trustworthiness. To gain a better understanding of LLMs, we conduct a detailed analysis of the operations of attention heads and aim to better understand the in-context learning of LLMs. Specifically, we investigate whether attention heads encode two types of relationships between tokens present in natural languages: the syntactic dependency parsed from sentences and the relation within knowledge graphs. We find that certain attention heads exhibit a pattern where, when attending to head tokens, they recall tail tokens and increase the output logits of those tail tokens. More crucially, the formulation of such semantic induction heads has a close correlation with the emergence of the in-context learning ability of language models. The study of semantic attention heads advances our understanding of the intricate operations of attention heads in transformers, and further provides new insights into the in-context learning of LLMs.

Title: Toward Fairness via Maximum Mean Discrepancy Regularization on Logits Space

• Authors: Hao-Wei Chung, Ching-Hao Chiu, Yu-Jen Chen, Yiyu Shi, Tsung-Yi Ho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.13061
• Pdf URL: https://arxiv.org/pdf/2402.13061
• Copy Paste: [[2402.13061]] Toward Fairness via Maximum Mean Discrepancy Regularization on Logits Space(https://arxiv.org/abs/2402.13061)
• Keywords: fair
• Abstract: Fairness has become increasingly pivotal in machine learning for high-risk applications such as machine learning in healthcare and facial recognition. However, we see the deficiency in the previous logits space constraint methods. Therefore, we propose a novel framework, Logits-MMD, that achieves the fairness condition by imposing constraints on output logits with Maximum Mean Discrepancy. Moreover, quantitative analysis and experimental results show that our framework has a better property that outperforms previous methods and achieves state-of-the-art on two facial recognition datasets and one animal dataset. Finally, we show experimental results and demonstrate that our debias approach achieves the fairness condition effectively.

Title: Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

• Authors: Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13064
• Pdf URL: https://arxiv.org/pdf/2402.13064
• Copy Paste: [[2402.13064]] Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models(https://arxiv.org/abs/2402.13064)
• Keywords: large language model
• Abstract: We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy.

Title: Mechanistic Neural Networks for Scientific Machine Learning

• Authors: Adeel Pervez, Francesco Locatello, Efstratios Gavves
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.13077
• Pdf URL: https://arxiv.org/pdf/2402.13077
• Copy Paste: [[2402.13077]] Mechanistic Neural Networks for Scientific Machine Learning(https://arxiv.org/abs/2402.13077)
• Keywords: interpretability
• Abstract: This paper presents Mechanistic Neural Networks, a neural network design for machine learning applications in the sciences. It incorporates a new Mechanistic Block in standard architectures to explicitly learn governing differential equations as representations, revealing the underlying dynamics of data and enhancing interpretability and efficiency in data modeling. Central to our approach is a novel Relaxed Linear Programming Solver (NeuRLP) inspired by a technique that reduces solving linear ODEs to solving linear programs. This integrates well with neural networks and surpasses the limitations of traditional ODE solvers enabling scalable GPU parallel processing. Overall, Mechanistic Neural Networks demonstrate their versatility for scientific machine learning applications, adeptly managing tasks from equation discovery to dynamic systems modeling. We prove their comprehensive capabilities in analyzing and interpreting complex scientific data across various applications, showing significant performance against specialized state-of-the-art methods.

Title: IT Intrusion Detection Using Statistical Learning and Testbed Measurements

• Authors: Xiaoxuan Wang, Rolf Stadler
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.13081
• Pdf URL: https://arxiv.org/pdf/2402.13081
• Copy Paste: [[2402.13081]] IT Intrusion Detection Using Statistical Learning and Testbed Measurements(https://arxiv.org/abs/2402.13081)
• Keywords: attack
• Abstract: We study automated intrusion detection in an IT infrastructure, specifically the problem of identifying the start of an attack, the type of attack, and the sequence of actions an attacker takes, based on continuous measurements from the infrastructure. We apply statistical learning methods, including Hidden Markov Model (HMM), Long Short-Term Memory (LSTM), and Random Forest Classifier (RFC) to map sequences of observations to sequences of predicted attack actions. In contrast to most related research, we have abundant data to train the models and evaluate their predictive power. The data comes from traces we generate on an in-house testbed where we run attacks against an emulated IT infrastructure. Central to our work is a machine-learning pipeline that maps measurements from a high-dimensional observation space to a space of low dimensionality or to a small set of observation symbols. Investigating intrusions in offline as well as online scenarios, we find that both HMM and LSTM can be effective in predicting attack start time, attack type, and attack actions. If sufficient training data is available, LSTM achieves higher prediction accuracy than HMM. HMM, on the other hand, requires less computational resources and less training data for effective prediction. Also, we find that the methods we study benefit from data produced by traditional intrusion detection systems like SNORT.

Title: How Does Selection Leak Privacy: Revisiting Private Selection and Improved Results for Hyper-parameter Tuning

• Authors: Zihang Xiang, Chenglong Wang, Di Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.13087
• Pdf URL: https://arxiv.org/pdf/2402.13087
• Copy Paste: [[2402.13087]] How Does Selection Leak Privacy: Revisiting Private Selection and Improved Results for Hyper-parameter Tuning(https://arxiv.org/abs/2402.13087)
• Keywords: privacy
• Abstract: We study the problem of guaranteeing Differential Privacy (DP) in hyper-parameter tuning, a crucial process in machine learning involving the selection of the best run from several. Unlike many private algorithms, including the prevalent DP-SGD, the privacy implications of tuning remain insufficiently understood. Recent works propose a generic private solution for the tuning process, yet a fundamental question still persists: is the current privacy bound for this solution tight? This paper contributes both positive and negative answers to this question. Initially, we provide studies affirming the current privacy analysis is indeed tight in a general sense. However, when we specifically study the hyper-parameter tuning problem, such tightness no longer holds. This is first demonstrated by applying privacy audit on the tuning process. Our findings underscore a substantial gap between the current theoretical privacy bound and the empirical bound derived even under the strongest audit setup. The gap found is not a fluke. Our subsequent study provides an improved privacy result for private hyper-parameter tuning due to its distinct properties. Our privacy results are also more generalizable compared to prior analyses that are only easily applicable in specific setups.

Title: Slot-VLM: SlowFast Slots for Video-Language Modeling

• Authors: Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, Yan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.13088
• Pdf URL: https://arxiv.org/pdf/2402.13088
• Copy Paste: [[2402.13088]] Slot-VLM: SlowFast Slots for Video-Language Modeling(https://arxiv.org/abs/2402.13088)
• Keywords: large language model
• Abstract: Video-Language Models (VLMs), powered by the advancements in Large Language Models (LLMs), are charting new frontiers in video understanding. A pivotal challenge is the development of an efficient method to encapsulate video content into a set of representative tokens to align with LLMs. In this work, we introduce Slot-VLM, a novel framework designed to generate semantically decomposed video tokens, in terms of object-wise and event-wise visual representations, to facilitate LLM inference. Particularly, we design a SlowFast Slots module, i.e., SF-Slots, that adaptively aggregates the dense video tokens from the CLIP vision encoder to a set of representative slots. In order to take into account both the spatial object details and the varied temporal dynamics, SF-Slots is built with a dual-branch structure. The Slow-Slots branch focuses on extracting object-centric slots from features at high spatial resolution but low (slow) frame sample rate, emphasizing detailed object information. Conversely, Fast-Slots branch is engineered to learn event-centric slots from high temporal sample rate but low spatial resolution features. These complementary slots are combined to form the vision context, serving as the input to the LLM for efficient question answering. Our experimental results demonstrate the effectiveness of our Slot-VLM, which achieves the state-of-the-art performance on video question-answering.

Title: Event-level Knowledge Editing

• Authors: Hao Peng, Xiaozhi Wang, Chunyang Li, Kaisheng Zeng, Jiangshan Duo, Yixin Cao, Lei Hou, Juanzi Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13093
• Pdf URL: https://arxiv.org/pdf/2402.13093
• Copy Paste: [[2402.13093]] Event-level Knowledge Editing(https://arxiv.org/abs/2402.13093)
• Keywords: large language model
• Abstract: Knowledge editing aims at updating knowledge of large language models (LLMs) to prevent them from becoming outdated. Existing work edits LLMs at the level of factual knowledge triplets. However, natural knowledge updates in the real world come from the occurrences of new events rather than direct changes in factual triplets. In this paper, we propose a new task setting: event-level knowledge editing, which directly edits new events into LLMs and improves over conventional triplet-level editing on (1) Efficiency. A single event edit leads to updates in multiple entailed knowledge triplets. (2) Completeness. Beyond updating factual knowledge, event-level editing also requires considering the event influences and updating LLMs' knowledge about future trends. We construct a high-quality event-level editing benchmark ELKEN, consisting of 1,515 event edits, 6,449 questions about factual knowledge, and 10,150 questions about future tendencies. We systematically evaluate the performance of various knowledge editing methods and LLMs on this benchmark. We find that ELKEN poses significant challenges to existing knowledge editing approaches. Our codes and dataset are publicly released to facilitate further research.

Title: ELAD: Explanation-Guided Large Language Models Active Distillation

• Authors: Yifei Zhang, Bo Pan, Chen Ling, Yuntong Hu, Liang Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13098
• Pdf URL: https://arxiv.org/pdf/2402.13098
• Copy Paste: [[2402.13098]] ELAD: Explanation-Guided Large Language Models Active Distillation(https://arxiv.org/abs/2402.13098)
• Keywords: large language model
• Abstract: The deployment and application of Large Language Models (LLMs) is hindered by their memory inefficiency, computational demands, and the high costs of API inferences. Traditional distillation methods, which transfer the capabilities of LLMs to smaller models, often fail to determine whether the knowledge has been sufficiently transferred, potentially resulting in high costs or incomplete distillation. In this paper, we propose an Explanation-Guided LLMs Active Distillation (ELAD) framework that employs an active learning strategy to optimize the balance between annotation costs and model performance. To improve efficient sample selection, we introduce an explanation-guided sample selection method that identifies samples challenging its reasoning by exploiting uncertainties in explanation steps. Additionally, we present a customized LLM-annotated explanation revision technique where the teacher model detects and corrects flaws in the student model's reasoning. Our experiments across various reasoning datasets demonstrate that our framework significantly enhances the efficiency of LLM knowledge distillation.

Title: CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models

• Authors: Yizhi LI, Ge Zhang, Xingwei Qu, Jiali Li, Zhaoqun Li, Zekun Wang, Hao Li, Ruibin Yuan, Yinghao Ma, Kai Zhang, Wangchunshu Zhou, Yiming Liang, Lei Zhang, Lei Ma, Jiajun Zhang, Zuowen Li, Stephen W. Huang, Chenghua Lin, Wenhu Chen, Jie Fu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13109
• Pdf URL: https://arxiv.org/pdf/2402.13109
• Copy Paste: [[2402.13109]] CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models(https://arxiv.org/abs/2402.13109)
• Keywords: large language model
• Abstract: The advancement of large language models (LLMs) has enhanced the ability to generalize across a wide range of unseen natural language processing (NLP) tasks through instruction-following. Yet, their effectiveness often diminishes in low-resource languages like Chinese, exacerbated by biased evaluations from data leakage, casting doubt on their true generalizability to new linguistic territories. In response, we introduce the Chinese Instruction-Following Benchmark (CIF-Bench), designed to evaluate the zero-shot generalizability of LLMs to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances across 20 categories. To mitigate evaluation bias, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance, totaling 45,000 data instances. Our evaluation of 28 selected LLMs reveals a noticeable performance gap, with the best model scoring only 52.9%, highlighting the limitations of LLMs in less familiar language and task contexts. This work aims to uncover the current limitations of LLMs in handling Chinese tasks, pushing towards the development of more culturally informed and linguistically diverse models with the released data and benchmark (https://yizhilll.github.io/CIF-Bench/).

Title: When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality

• Authors: Brielen Madureira, Patrick Kahardipraja, David Schlangen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13113
• Pdf URL: https://arxiv.org/pdf/2402.13113
• Copy Paste: [[2402.13113]] When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality(https://arxiv.org/abs/2402.13113)
• Keywords: transformer
• Abstract: Incremental models that process sentences one token at a time will sometimes encounter points where more than one interpretation is possible. Causal models are forced to output one interpretation and continue, whereas models that can revise may edit their previous output as the ambiguity is resolved. In this work, we look at how restart-incremental Transformers build and update internal states, in an effort to shed light on what processes cause revisions not viable in autoregressive models. We propose an interpretable way to analyse the incremental states, showing that their sequential structure encodes information on the garden path effect and its resolution. Our method brings insights on various bidirectional encoders for contextualised meaning representation and dependency parsing, contributing to show their advantage over causal models when it comes to revisions.

Title: A Survey on Knowledge Distillation of Large Language Models

• Authors: Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, Tianyi Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13116
• Pdf URL: https://arxiv.org/pdf/2402.13116
• Copy Paste: [[2402.13116]] A Survey on Knowledge Distillation of Large Language Models(https://arxiv.org/abs/2402.13116)
• Keywords: large language model
• Abstract: This survey presents an in-depth exploration of knowledge distillation (KD) techniques within the realm of Large Language Models (LLMs), spotlighting the pivotal role of KD in transferring sophisticated capabilities from proprietary giants such as GPT-4 to accessible, open-source models like LLaMA and Mistral. Amidst the evolving AI landscape, this work elucidates the critical disparities between proprietary and open-source LLMs, demonstrating how KD serves as an essential conduit for imbuing the latter with the former's advanced functionalities and nuanced understandings. Our survey is meticulously structured around three foundational pillars: algorithm, skill, and verticalization -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts. This work aims to provide an insightful guide for researchers and practitioners, offering a detailed overview of current methodologies in knowledge distillation and proposing future research directions. By bridging the gap between proprietary and open-source LLMs, this survey underscores the potential for more accessible, efficient, and sustainable AI solutions, fostering a more inclusive and equitable landscape in AI advancements. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs.

Title: Cross-Domain Transfer Learning with CoRTe: Consistent and Reliable Transfer from Black-Box to Lightweight Segmentation Model

• Authors: Claudia Cuttano, Antonio Tavera, Fabio Cermelli, Giuseppe Averta, Barbara Caputo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.13122
• Pdf URL: https://arxiv.org/pdf/2402.13122
• Copy Paste: [[2402.13122]] Cross-Domain Transfer Learning with CoRTe: Consistent and Reliable Transfer from Black-Box to Lightweight Segmentation Model(https://arxiv.org/abs/2402.13122)
• Keywords: segmentation
• Abstract: Many practical applications require training of semantic segmentation models on unlabelled datasets and their execution on low-resource hardware. Distillation from a trained source model may represent a solution for the first but does not account for the different distribution of the training data. Unsupervised domain adaptation (UDA) techniques claim to solve the domain shift, but in most cases assume the availability of the source data or an accessible white-box source model, which in practical applications are often unavailable for commercial and/or safety reasons. In this paper, we investigate a more challenging setting in which a lightweight model has to be trained on a target unlabelled dataset for semantic segmentation, under the assumption that we have access only to black-box source model predictions. Our method, named CoRTe, consists of (i) a pseudo-labelling function that extracts reliable knowledge from the black-box source model using its relative confidence, (ii) a pseudo label refinement method to retain and enhance the novel information learned by the student model on the target data, and (iii) a consistent training of the model using the extracted pseudo labels. We benchmark CoRTe on two synthetic-to-real settings, demonstrating remarkable results when using black-box models to transfer knowledge on lightweight models for a target data distribution.

Title: TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning

• Authors: Xiang Li, Yunshi Lan, Chao Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13125
• Pdf URL: https://arxiv.org/pdf/2402.13125
• Copy Paste: [[2402.13125]] TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning(https://arxiv.org/abs/2402.13125)
• Keywords: robust, large language model
• Abstract: Recently, numerous new benchmarks have been established to evaluate the performance of large language models (LLMs) via either computing a holistic score or employing another LLM as a judge. However, these approaches suffer from data leakage due to the open access of the benchmark and inflexible evaluation process. To address this issue, we introduce $\textbf{TreeEval}$, a benchmark-free evaluation method for LLMs that let a high-performance LLM host an irreproducible evaluation session and essentially avoids the data leakage. Moreover, this LLM performs as an examiner to raise up a series of questions under a topic with a tree planing strategy, which considers the current evaluation status to decide the next question generation and ensures the completeness and efficiency of the evaluation process. We evaluate $6$ models of different parameter sizes, including $7$B, $13$B, and $33$B, and ultimately achieved the highest correlation coefficient with AlpacaEval2.0 using only around $45$ questions. We also conduct more analysis to show the robustness and reliability of TreeEval. Our code can be accessed via the provided https://github.com/Ashura5/TreeEval.

Title: VGMShield: Mitigating Misuse of Video Generative Models

• Authors: Yan Pang, Yang Zhang, Tianhao Wang
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.13126
• Pdf URL: https://arxiv.org/pdf/2402.13126
• Copy Paste: [[2402.13126]] VGMShield: Mitigating Misuse of Video Generative Models(https://arxiv.org/abs/2402.13126)
• Keywords: generative
• Abstract: With the rapid advancement in video generation, people can conveniently utilize video generation models to create videos tailored to their specific desires. Nevertheless, there are also growing concerns about their potential misuse in creating and disseminating false information. In this work, we introduce VGMShield: a set of three straightforward but pioneering mitigations through the lifecycle of fake video generation. We start from \textit{fake video detection} trying to understand whether there is uniqueness in generated videos and whether we can differentiate them from real videos; then, we investigate the \textit{tracing} problem, which maps a fake video back to a model that generates it. Towards these, we propose to leverage pre-trained models that focus on {\it spatial-temporal dynamics} as the backbone to identify inconsistencies in videos. Through experiments on seven state-of-the-art open-source models, we demonstrate that current models still cannot perfectly handle spatial-temporal relationships, and thus, we can accomplish detection and tracing with nearly perfect accuracy. Furthermore, anticipating future generative model improvements, we propose a {\it prevention} method that adds invisible perturbations to images to make the generated videos look unreal. Together with fake video detection and tracing, our multi-faceted set of solutions can effectively mitigate misuse of video generative models.

Title: The Hidden Space of Transformer Language Adapters

• Authors: Jesujoba O. Alabi, Marius Mosbach, Matan Eyal, Dietrich Klakow, Mor Geva
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13137
• Pdf URL: https://arxiv.org/pdf/2402.13137
• Copy Paste: [[2402.13137]] The Hidden Space of Transformer Language Adapters(https://arxiv.org/abs/2402.13137)
• Keywords: transformer
• Abstract: We analyze the operation of transformer language adapters, which are small modules trained on top of a frozen language model to adapt its predictions to new target languages. We show that adapted predictions mostly evolve in the source language the model was trained on, while the target language becomes pronounced only in the very last layers of the model. Moreover, the adaptation process is gradual and distributed across layers, where it is possible to skip small groups of adapters without decreasing adaptation performance. Last, we show that adapters operate on top of the model's frozen representation space while largely preserving its structure, rather than on an 'isolated' subspace. Our findings provide a deeper view into the adaptation process of language models to new languages, showcasing the constraints imposed on it by the underlying model and introduces practical implications to enhance its efficiency.

Title: Neural Network Diffusion

• Authors: Kai Wang, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, Yang You
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.13144
• Pdf URL: https://arxiv.org/pdf/2402.13144
• Copy Paste: [[2402.13144]] Neural Network Diffusion(https://arxiv.org/abs/2402.13144)
• Keywords: diffusion
• Abstract: Diffusion models have achieved remarkable success in image and video generation. In this work, we demonstrate that diffusion models can also \textit{generate high-performing neural network parameters}. Our approach is simple, utilizing an autoencoder and a standard latent diffusion model. The autoencoder extracts latent representations of a subset of the trained network parameters. A diffusion model is then trained to synthesize these latent parameter representations from random noise. It then generates new representations that are passed through the autoencoder's decoder, whose outputs are ready to use as new subsets of network parameters. Across various architectures and datasets, our diffusion process consistently generates models of comparable or improved performance over trained networks, with minimal additional cost. Notably, we empirically find that the generated models perform differently with the trained networks. Our results encourage more exploration on the versatile use of diffusion models.

Title: CMDAG: A Chinese Metaphor Dataset with Annotated Grounds as CoT for Boosting Metaphor Generation

• Authors: Yujie Shao, Xinrong Yao, Xingwei Qu, Chenghua Lin, Shi Wang, Stephen W. Huang, Ge Zhang, Jie Fu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13145
• Pdf URL: https://arxiv.org/pdf/2402.13145
• Copy Paste: [[2402.13145]] CMDAG: A Chinese Metaphor Dataset with Annotated Grounds as CoT for Boosting Metaphor Generation(https://arxiv.org/abs/2402.13145)
• Keywords: generative
• Abstract: Metaphor is a prominent linguistic device in human language and literature, as they add color, imagery, and emphasis to enhance effective communication. This paper introduces a large-scale high quality annotated Chinese Metaphor Corpus, which comprises around 28K sentences drawn from a diverse range of Chinese literary sources, such as poems, prose, song lyrics, etc. To ensure the accuracy and consistency of our annotations, we introduce a comprehensive set of guidelines. These guidelines address the facets of metaphor annotation, including identifying tenors, vehicles, and grounds to handling the complexities of similes, personifications, juxtapositions, and hyperboles. Breaking tradition, our approach to metaphor generation emphasizes grounds and their distinct features rather than the conventional combination of tenors and vehicles. By integrating "ground" as a CoT (Chain of Thoughts) input, we are able to generate metaphors that resonate more with real-world intuition. We test generative models such as Belle, Baichuan, and Chinese-alpaca-33B using our annotated corpus. These models are able to generate creative and fluent metaphor sentences more frequently induced by selected samples from our dataset, demonstrating the value of our corpus for Chinese metaphor research. The code is available in the https://anonymous.4open.science/r/Chinese_Metaphor_Explanation-63F2.

Title: OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog

• Authors: Adnen Abdessaied, Manuel von Hochmeister, Andreas Bulling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.13146
• Pdf URL: https://arxiv.org/pdf/2402.13146
• Copy Paste: [[2402.13146]] OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog(https://arxiv.org/abs/2402.13146)
• Keywords: transformer, large language model
• Abstract: We present the Object Language Video Transformer (OLViT) - a novel model for video dialog operating over a multi-modal attention-based dialog state tracker. Existing video dialog models struggle with questions requiring both spatial and temporal localization within videos, long-term temporal reasoning, and accurate object tracking across multiple dialog turns. OLViT addresses these challenges by maintaining a global dialog state based on the output of an Object State Tracker (OST) and a Language State Tracker (LST): while the OST attends to the most important objects within the video, the LST keeps track of the most important linguistic co-references to previous dialog turns. In stark contrast to previous works, our approach is generic by nature and is therefore capable of learning continuous multi-modal dialog state representations of the most relevant objects and rounds. As a result, they can be seamlessly integrated into Large Language Models (LLMs) and offer high flexibility in dealing with different datasets and tasks. Evaluations on the challenging DVD (response classification) and SIMMC 2.1 (response generation) datasets show that OLViT achieves new state-of-the-art performance across both datasets.

Title: Defending Jailbreak Prompts via In-Context Adversarial Game

• Authors: Yujun Zhou, Yufei Han, Haomin Zhuang, Taicheng Guo, Kehan Guo, Zhenwen Liang, Hongyan Bao, Xiangliang Zhang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.13148
• Pdf URL: https://arxiv.org/pdf/2402.13148
• Copy Paste: [[2402.13148]] Defending Jailbreak Prompts via In-Context Adversarial Game(https://arxiv.org/abs/2402.13148)
• Keywords: security, defense, attack, large language model
• Abstract: Large Language Models (LLMs) demonstrate remarkable capabilities across diverse applications. However, concerns regarding their security, particularly the vulnerability to jailbreak attacks, persist. Drawing inspiration from adversarial training in deep learning and LLM agent learning processes, we introduce the In-Context Adversarial Game (ICAG) for defending against jailbreaks without the need for fine-tuning. ICAG leverages agent learning to conduct an adversarial game, aiming to dynamically extend knowledge to defend against jailbreaks. Unlike traditional methods that rely on static datasets, ICAG employs an iterative process to enhance both the defense and attack agents. This continuous improvement process strengthens defenses against newly generated jailbreak prompts. Our empirical studies affirm ICAG's efficacy, where LLMs safeguarded by ICAG exhibit significantly reduced jailbreak success rates across various attack scenarios. Moreover, ICAG demonstrates remarkable transferability to other LLMs, indicating its potential as a versatile defense mechanism.

Title: Formal Verification for Blockchain-based Insurance Claims Processing

• Authors: Roshan Lal Neupane, Ernest Bonnah, Bishnu Bhusal, Kiran Neupane, Khaza Anuarul Hoque, Prasad Calyam
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.13169
• Pdf URL: https://arxiv.org/pdf/2402.13169
• Copy Paste: [[2402.13169]] Formal Verification for Blockchain-based Insurance Claims Processing(https://arxiv.org/abs/2402.13169)
• Keywords: secure
• Abstract: Insurance claims processing involves multi-domain entities and multi-source data, along with a number of human-agent interactions. Use of Blockchain technology-based platform can significantly improve scalability and response time for processing of claims which are otherwise manually-intensive and time-consuming. However, the chaincodes involved within the processes that issue claims, approve or deny them as required, need to be formally verified to ensure secure and reliable processing of transactions in Blockchain. In this paper, we use a formal modeling approach to verify various processes and their underlying chaincodes relating to different stages in insurance claims processing viz., issuance, approval, denial, and flagging for fraud investigation by using linear temporal logic (LTL). We simulate the formalism on the chaincodes and analyze the breach of chaincodes via model checking.

Title: Benchmarking Retrieval-Augmented Generation for Medicine

• Authors: Guangzhi Xiong, Qiao Jin, Zhiyong Lu, Aidong Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13178
• Pdf URL: https://arxiv.org/pdf/2402.13178
• Copy Paste: [[2402.13178]] Benchmarking Retrieval-Augmented Generation for Medicine(https://arxiv.org/abs/2402.13178)
• Keywords: large language model
• Abstract: While large language models (LLMs) have achieved state-of-the-art performance on a wide range of medical question answering (QA) tasks, they still face challenges with hallucinations and outdated knowledge. Retrieval-augmented generation (RAG) is a promising solution and has been widely adopted. However, a RAG system can involve multiple flexible components, and there is a lack of best practices regarding the optimal RAG setting for various medical purposes. To systematically evaluate such systems, we propose the Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE), a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets. Using MIRAGE, we conducted large-scale experiments with over 1.8 trillion prompt tokens on 41 combinations of different corpora, retrievers, and backbone LLMs through the MedRAG toolkit introduced in this work. Overall, MedRAG improves the accuracy of six different LLMs by up to 18% over chain-of-thought prompting, elevating the performance of GPT-3.5 and Mixtral to GPT-4-level. Our results show that the combination of various medical corpora and retrievers achieves the best performance. In addition, we discovered a log-linear scaling property and the "lost-in-the-middle" effects in medical RAG. We believe our comprehensive evaluations can serve as practical guidelines for implementing RAG systems for medicine.

Title: What if LLMs Have Different World Views: Simulating Alien Civilizations with LLM-based Agents

• Authors: Mingyu Jin, Beichen Wang, Zhaoqian Xue, Suiyuan Zhu, Wenyue Hua, Hua Tang, Kai Mei, Mengnan Du, Yongfeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13184
• Pdf URL: https://arxiv.org/pdf/2402.13184
• Copy Paste: [[2402.13184]] What if LLMs Have Different World Views: Simulating Alien Civilizations with LLM-based Agents(https://arxiv.org/abs/2402.13184)
• Keywords: large language model
• Abstract: In this study, we introduce "CosmoAgent," an innovative artificial intelligence framework utilizing Large Language Models (LLMs) to simulate complex interactions between human and extraterrestrial civilizations, with a special emphasis on Stephen Hawking's cautionary advice about not sending radio signals haphazardly into the universe. The goal is to assess the feasibility of peaceful coexistence while considering potential risks that could threaten well-intentioned civilizations. Employing mathematical models and state transition matrices, our approach quantitatively evaluates the development trajectories of civilizations, offering insights into future decision-making at critical points of growth and saturation. Furthermore, the paper acknowledges the vast diversity in potential living conditions across the universe, which could foster unique cosmologies, ethical codes, and worldviews among various civilizations. Recognizing the Earth-centric bias inherent in current LLM designs, we propose the novel concept of using LLMs with diverse ethical paradigms and simulating interactions between entities with distinct moral principles. This innovative research provides a new way to understand complex inter-civilizational dynamics, expanding our perspective while pioneering novel strategies for conflict resolution, crucial for preventing interstellar conflicts. We have also released the code and datasets to enable further academic investigation into this interesting area of research. The code is available at https://github.com/agiresearch/AlienAgent.

Title: Bayesian Reward Models for LLM Alignment

• Authors: Adam X. Yang, Maxime Robeyns, Thomas Coste, Jun Wang, Haitham Bou-Ammar, Laurence Aitchison
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.13210
• Pdf URL: https://arxiv.org/pdf/2402.13210
• Copy Paste: [[2402.13210]] Bayesian Reward Models for LLM Alignment(https://arxiv.org/abs/2402.13210)
• Keywords: large language model
• Abstract: To ensure that large language model (LLM) responses are helpful and non-toxic, we usually fine-tune a reward model on human preference data. We then select policy responses with high rewards (best-of-n sampling) or further optimize the policy to produce responses with high rewards (reinforcement learning from human feedback). However, this process is vulnerable to reward overoptimization or hacking, in which the responses selected have high rewards due to errors in the reward model rather than a genuine preference. This is especially problematic as the prompt or response diverges from the training data. It should be possible to mitigate these issues by training a Bayesian reward model, which signals higher uncertainty further from the training data distribution. Therefore, we trained Bayesian reward models using Laplace-LoRA (Yang et al., 2024) and found that the resulting uncertainty estimates can successfully mitigate reward overoptimization in best-of-n sampling.

Title: Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation

• Authors: Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Seungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, Jinyoung Yeo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13211
• Pdf URL: https://arxiv.org/pdf/2402.13211
• Copy Paste: [[2402.13211]] Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation(https://arxiv.org/abs/2402.13211)
• Keywords: robust, large language model
• Abstract: Emotional Support Conversation (ESC) is a task aimed at alleviating individuals' emotional distress through daily conversation. Given its inherent complexity and non-intuitive nature, ESConv dataset incorporates support strategies to facilitate the generation of appropriate responses. Recently, despite the remarkable conversational ability of large language models (LLMs), previous studies have suggested that they often struggle with providing useful emotional support. Hence, this work initially analyzes the results of LLMs on ESConv, revealing challenges in selecting the correct strategy and a notable preference for a specific strategy. Motivated by these, we explore the impact of the inherent preference in LLMs on providing emotional support, and consequently, we observe that exhibiting high preference for specific strategies hinders effective emotional support, aggravating its robustness in predicting the appropriate strategy. Moreover, we conduct a methodological study to offer insights into the necessary approaches for LLMs to serve as proficient emotional supporters. Our findings emphasize that (1) low preference for specific strategies hinders the progress of emotional support, (2) external assistance helps reduce preference bias, and (3) LLMs alone cannot become good emotional supporters. These insights suggest promising avenues for future research to enhance the emotional intelligence of LLMs.

Title: Soft Self-Consistency Improves Language Model Agents

• Authors: Han Wang, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.13212
• Pdf URL: https://arxiv.org/pdf/2402.13212
• Copy Paste: [[2402.13212]] Soft Self-Consistency Improves Language Model Agents(https://arxiv.org/abs/2402.13212)
• Keywords: large language model
• Abstract: Generations from large language models (LLMs) can be improved by sampling and scoring multiple solutions to select a final answer. Current "sample and select" methods such as self-consistency (SC) rely on majority voting to score answers. However, when tasks have many distinct and valid answers, selection by voting requires a large number of samples. This makes SC prohibitively expensive for interactive tasks that involve generating multiple actions (answers) sequentially. After establishing that majority voting fails to provide consistent gains on such tasks, we demonstrate how to increase success rates by softening the scoring criterion. We introduce Soft Self-Consistency (Soft-SC), which replaces SC's discontinuous scoring with a continuous score computed from model likelihoods, allowing for selection even when actions are sparsely distributed. Soft-SC improves both performance and efficiency on long-horizon interactive tasks, requiring half as many samples as SC for comparable or better performance. For a fixed number of samples, Soft-SC leads to a 1.3% increase over SC in absolute success rate on writing bash programs, a 6.6% increase on online shopping (WebShop), and a 4.7% increase for an interactive household game (ALFWorld). Finally, we show that Soft-SC can be applied to both open-source and black-box models.

Title: Softmax Probabilities (Mostly) Predict Large Language Model Correctness on Multiple-Choice Q&A

• Authors: Benjamin Plaut, Khanh Nguyen, Tu Trinh
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.13213
• Pdf URL: https://arxiv.org/pdf/2402.13213
• Copy Paste: [[2402.13213]] Softmax Probabilities (Mostly) Predict Large Language Model Correctness on Multiple-Choice Q&A(https://arxiv.org/abs/2402.13213)
• Keywords: large language model
• Abstract: Although large language models (LLMs) perform impressively on many tasks, overconfidence remains a problem. We hypothesized that on multiple-choice Q&A tasks, wrong answers would be associated with smaller maximum softmax probabilities (MSPs) compared to correct answers. We comprehensively evaluate this hypothesis on ten open-source LLMs and five datasets, and find strong evidence for our hypothesis among models which perform well on the original Q&A task. For the six LLMs with the best Q&A performance, the AUROC derived from the MSP was better than random chance with p < 10^{-4} in 59/60 instances. Among those six LLMs, the average AUROC ranged from 60% to 69%. Leveraging these findings, we propose a multiple-choice Q&A task with an option to abstain and show that performance can be improved by selectively abstaining based on the MSP of the initial model response. We also run the same experiments with pre-softmax logits instead of softmax probabilities and find similar (but not identical) results.

Title: How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts

• Authors: Yusu Qian, Haotian Zhang, Yinfei Yang, Zhe Gan
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13220
• Pdf URL: https://arxiv.org/pdf/2402.13220
• Copy Paste: [[2402.13220]] How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts(https://arxiv.org/abs/2402.13220)
• Keywords: robust, large language model
• Abstract: The remarkable advancements in Multimodal Large Language Models (MLLMs) have not rendered them immune to challenges, particularly in the context of handling deceptive information in prompts, thus producing hallucinated responses under such conditions. To quantitatively assess this vulnerability, we present MAD-Bench, a carefully curated benchmark that contains 850 test samples divided into 6 categories, such as non-existent objects, count of objects, spatial relationship, and visual confusion. We provide a comprehensive analysis of popular MLLMs, ranging from GPT-4V, Gemini-Pro, to open-sourced models, such as LLaVA-1.5 and CogVLM. Empirically, we observe significant performance gaps between GPT-4V and other models; and previous robust instruction-tuned models, such as LRV-Instruction and LLaVA-RLHF, are not effective on this new benchmark. While GPT-4V achieves 75.02% accuracy on MAD-Bench, the accuracy of any other model in our experiments ranges from 5% to 35%. We further propose a remedy that adds an additional paragraph to the deceptive prompts to encourage models to think twice before answering the question. Surprisingly, this simple method can even double the accuracy; however, the absolute numbers are still too low to be satisfactory. We hope MAD-Bench can serve as a valuable benchmark to stimulate further research to enhance models' resilience against deceptive prompts.

Title: CHILI: Chemically-Informed Large-scale Inorganic Nanomaterials Dataset for Advancing Graph Machine Learning

• Authors: Ulrik Friis-Jensen, Frederik L. Johansen, Andy S. Anker, Erik B. Dam, Kirsten M. Ø. Jensen, Raghavendra Selvan
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.13221
• Pdf URL: https://arxiv.org/pdf/2402.13221
• Copy Paste: [[2402.13221]] CHILI: Chemically-Informed Large-scale Inorganic Nanomaterials Dataset for Advancing Graph Machine Learning(https://arxiv.org/abs/2402.13221)
• Keywords: generative
• Abstract: Advances in graph machine learning (ML) have been driven by applications in chemistry as graphs have remained the most expressive representations of molecules. While early graph ML methods focused primarily on small organic molecules, recently, the scope of graph ML has expanded to include inorganic materials. Modelling the periodicity and symmetry of inorganic crystalline materials poses unique challenges, which existing graph ML methods are unable to address. Moving to inorganic nanomaterials increases complexity as the scale of number of nodes within each graph can be broad ($10$ to $10^5$). The bulk of existing graph ML focuses on characterising molecules and materials by predicting target properties with graphs as input. However, the most exciting applications of graph ML will be in their generative capabilities, which is currently not at par with other domains such as images or text. We invite the graph ML community to address these open challenges by presenting two new chemically-informed large-scale inorganic (CHILI) nanomaterials datasets: A medium-scale dataset (with overall >6M nodes, >49M edges) of mono-metallic oxide nanomaterials generated from 12 selected crystal types (CHILI-3K) and a large-scale dataset (with overall >183M nodes, >1.2B edges) of nanomaterials generated from experimentally determined crystal structures (CHILI-100K). We define 11 property prediction tasks and 6 structure prediction tasks, which are of special interest for nanomaterial research. We benchmark the performance of a wide array of baseline methods and use these benchmarking results to highlight areas which need future work. To the best of our knowledge, CHILI-3K and CHILI-100K are the first open-source nanomaterial datasets of this scale -- both on the individual graph level and of the dataset as a whole -- and the only nanomaterials datasets with high structural and elemental diversity.

Title: RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian

• Authors: Adrian Cosma, Bogdan Iordache, Paolo Rosso
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.13222
• Pdf URL: https://arxiv.org/pdf/2402.13222
• Copy Paste: [[2402.13222]] RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian(https://arxiv.org/abs/2402.13222)
• Keywords: large language model
• Abstract: Recently, large language models (LLMs) have become increasingly powerful and have become capable of solving a plethora of tasks through proper instructions in natural language. However, the vast majority of testing suites assume that the instructions are written in English, the de facto prompting language. Code intelligence and problem solving still remain a difficult task, even for the most advanced LLMs. Currently, there are no datasets to measure the generalization power for code-generation models in a language other than English. In this work, we present RoCode, a competitive programming dataset, consisting of 2,642 problems written in Romanian, 11k solutions in C, C++ and Python and comprehensive testing suites for each problem. The purpose of RoCode is to provide a benchmark for evaluating the code intelligence of language models trained on Romanian / multilingual text as well as a fine-tuning set for pretrained Romanian models. Through our results and review of related works, we argue for the need to develop code models for languages other than English.

Title: AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learning

• Authors: Qiao Jin, Zhizheng Wang, Yifan Yang, Qingqing Zhu, Donald Wright, Thomas Huang, W John Wilbur, Zhe He, Andrew Taylor, Qingyu Chen, Zhiyong Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13225
• Pdf URL: https://arxiv.org/pdf/2402.13225
• Copy Paste: [[2402.13225]] AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learning(https://arxiv.org/abs/2402.13225)
• Keywords: large language model
• Abstract: Clinical calculators play a vital role in healthcare by offering accurate evidence-based predictions for various purposes such as prognosis. Nevertheless, their widespread utilization is frequently hindered by usability challenges, poor dissemination, and restricted functionality. Augmenting large language models with extensive collections of clinical calculators presents an opportunity to overcome these obstacles and improve workflow efficiency, but the scalability of the manual curation process poses a significant challenge. In response, we introduce AgentMD, a novel language agent capable of curating and applying clinical calculators across various clinical contexts. Using the published literature, AgentMD has automatically curated a collection of 2,164 diverse clinical calculators with executable functions and structured documentation, collectively named RiskCalcs. Manual evaluations show that RiskCalcs tools achieve an accuracy of over 80% on three quality metrics. At inference time, AgentMD can automatically select and apply the relevant RiskCalcs tools given any patient description. On the newly established RiskQA benchmark, AgentMD significantly outperforms chain-of-thought prompting with GPT-4 (87.7% vs. 40.9% in accuracy). Additionally, we also applied AgentMD to real-world clinical notes for analyzing both population-level and risk-level patient characteristics. In summary, our study illustrates the utility of language agents augmented with clinical calculators for healthcare analytics and patient care.

Title: Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

• Authors: Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, Colin White
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.13228
• Pdf URL: https://arxiv.org/pdf/2402.13228
• Copy Paste: [[2402.13228]] Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive(https://arxiv.org/abs/2402.13228)
• Keywords: large language model
• Abstract: Direct Preference Optimisation (DPO) is effective at significantly improving the performance of large language models (LLMs) on downstream tasks such as reasoning, summarisation, and alignment. Using pairs of preferred and dispreferred data, DPO models the \textit{relative} probability of picking one response over another. In this work, first we show theoretically that the standard DPO loss can lead to a \textit{reduction} of the model's likelihood of the preferred examples, as long as the relative probability between the preferred and dispreferred classes increases. We then show empirically that this phenomenon occurs when fine-tuning LLMs on common datasets, especially datasets in which the edit distance between pairs of completions is low. Using these insights, we design DPO-Positive (DPOP), a new loss function and training procedure which avoids this failure mode. Surprisingly, we also find that DPOP significantly outperforms DPO across a wide variety of datasets and downstream tasks, including datasets with high edit distances between completions. By fine-tuning with DPOP, we create and release Smaug-34B and Smaug-72B, which achieve state-of-the-art open-source performance. Notably, Smaug-72B is nearly 2\% better than any other open-source model on the HuggingFace Open LLM Leaderboard and becomes the first open-source LLM to surpass an average accuracy of 80\%.

Title: Investigating Cultural Alignment of Large Language Models

• Authors: Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, Mona Diab
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.13231
• Pdf URL: https://arxiv.org/pdf/2402.13231
• Copy Paste: [[2402.13231]] Investigating Cultural Alignment of Large Language Models(https://arxiv.org/abs/2402.13231)
• Keywords: large language model
• Abstract: The intricate relationship between language and culture has long been a subject of exploration within the realm of linguistic anthropology. Large Language Models (LLMs), promoted as repositories of collective human knowledge, raise a pivotal question: do these models genuinely encapsulate the diverse knowledge adopted by different cultures? Our study reveals that these models demonstrate greater cultural alignment along two dimensions -- firstly, when prompted with the dominant language of a specific culture, and secondly, when pretrained with a refined mixture of languages employed by that culture. We quantify cultural alignment by simulating sociological surveys, comparing model responses to those of actual survey participants as references. Specifically, we replicate a survey conducted in various regions of Egypt and the United States through prompting LLMs with different pretraining data mixtures in both Arabic and English with the personas of the real respondents and the survey questions. Further analysis reveals that misalignment becomes more pronounced for underrepresented personas and for culturally sensitive topics, such as those probing social values. Finally, we introduce Anthropological Prompting, a novel method leveraging anthropological reasoning to enhance cultural alignment. Our study emphasizes the necessity for a more balanced multilingual pretraining dataset to better represent the diversity of human experience and the plurality of different cultures with many implications on the topic of cross-lingual transfer.

Title: A Touch, Vision, and Language Dataset for Multimodal Alignment

• Authors: Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, Ken Goldberg
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.13232
• Pdf URL: https://arxiv.org/pdf/2402.13232
• Copy Paste: [[2402.13232]] A Touch, Vision, and Language Dataset for Multimodal Alignment(https://arxiv.org/abs/2402.13232)
• Keywords: generative
• Abstract: Touch is an important sensing modality for humans, but it has not yet been incorporated into a multimodal generative language model. This is partially due to the difficulty of obtaining natural language labels for tactile data and the complexity of aligning tactile readings with both visual observations and language descriptions. As a step towards bridging that gap, this work introduces a new dataset of 44K in-the-wild vision-touch pairs, with English language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V (90%). We use this dataset to train a vision-language-aligned tactile encoder for open-vocabulary classification and a touch-vision-language (TVL) model for text generation using the trained encoder. Results suggest that by incorporating touch, the TVL model improves (+29% classification accuracy) touch-vision-language alignment over existing models trained on any pair of those modalities. Although only a small fraction of the dataset is human-labeled, the TVL model demonstrates improved visual-tactile understanding over GPT-4V (+12%) and open-source vision-language models (+32%) on a new touch-vision understanding benchmark. Code and data: https://tactile-vlm.github.io.

Title: Federated Causal Discovery from Heterogeneous Data

• Authors: Loka Li, Ignavier Ng, Gongxu Luo, Biwei Huang, Guangyi Chen, Tongliang Liu, Bin Gu, Kun Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.13241
• Pdf URL: https://arxiv.org/pdf/2402.13241
• Copy Paste: [[2402.13241]] Federated Causal Discovery from Heterogeneous Data(https://arxiv.org/abs/2402.13241)
• Keywords: privacy, protect, federate
• Abstract: Conventional causal discovery methods rely on centralized data, which is inconsistent with the decentralized nature of data in many real-world situations. This discrepancy has motivated the development of federated causal discovery (FCD) approaches. However, existing FCD methods may be limited by their potentially restrictive assumptions of identifiable functional causal models or homogeneous data distributions, narrowing their applicability in diverse scenarios. In this paper, we propose a novel FCD method attempting to accommodate arbitrary causal models and heterogeneous data. We first utilize a surrogate variable corresponding to the client index to account for the data heterogeneity across different clients. We then develop a federated conditional independence test (FCIT) for causal skeleton discovery and establish a federated independent change principle (FICP) to determine causal directions. These approaches involve constructing summary statistics as a proxy of the raw data to protect data privacy. Owing to the nonparametric properties, FCIT and FICP make no assumption about particular functional forms, thereby facilitating the handling of arbitrary causal models. We conduct extensive experiments on synthetic and real datasets to show the efficacy of our method. The code is available at \url{https://github.com/lokali/FedCDH.git}.

Title: Improving Robustness for Joint Optimization of Camera Poses and Decomposed Low-Rank Tensorial Radiance Fields

• Authors: Bo-Yu Cheng, Wei-Chen Chiu, Yu-Lun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.13252
• Pdf URL: https://arxiv.org/pdf/2402.13252
• Copy Paste: [[2402.13252]] Improving Robustness for Joint Optimization of Camera Poses and Decomposed Low-Rank Tensorial Radiance Fields(https://arxiv.org/abs/2402.13252)
• Keywords: robust
• Abstract: In this paper, we propose an algorithm that allows joint refinement of camera pose and scene geometry represented by decomposed low-rank tensor, using only 2D images as supervision. First, we conduct a pilot study based on a 1D signal and relate our findings to 3D scenarios, where the naive joint pose optimization on voxel-based NeRFs can easily lead to sub-optimal solutions. Moreover, based on the analysis of the frequency spectrum, we propose to apply convolutional Gaussian filters on 2D and 3D radiance fields for a coarse-to-fine training schedule that enables joint camera pose optimization. Leveraging the decomposition property in decomposed low-rank tensor, our method achieves an equivalent effect to brute-force 3D convolution with only incurring little computational overhead. To further improve the robustness and stability of joint optimization, we also propose techniques of smoothed 2D supervision, randomly scaled kernel parameters, and edge-guided loss mask. Extensive quantitative and qualitative evaluations demonstrate that our proposed framework achieves superior performance in novel view synthesis as well as rapid convergence for optimization.

Title: CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples

• Authors: Jianrui Zhang, Mu Cai, Tengyang Xie, Yong Jae Lee
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.13254
• Pdf URL: https://arxiv.org/pdf/2402.13254
• Copy Paste: [[2402.13254]] CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples(https://arxiv.org/abs/2402.13254)
• Keywords: generative
• Abstract: We propose CounterCurate, a framework to comprehensively improve the visio-linguistic compositional reasoning capability for both contrastive and generative multimodal models. In particular, we identify two under-explored critical problems: the neglect of the physically grounded reasoning (counting and position understanding) and the potential of using highly capable text and image generation models for semantic counterfactual fine-tuning. Our work pioneers an approach that addresses these gaps. We first spotlight the near-chance performance of multimodal models like CLIP and LLaVA in physically grounded compositional reasoning. We then apply simple data augmentation using a grounded image generation model, GLIGEN, to generate finetuning data, resulting in significant performance improvements: +33% and +37% for CLIP and LLaVA, respectively, on our newly curated Flickr30k-Positions benchmark. Moreover, we exploit the capabilities of high-performing text generation and image generation models, specifically GPT-4V and DALLE-3, to curate challenging semantic counterfactuals, thereby further enhancing compositional reasoning capabilities on benchmarks such as SugarCrepe, where CounterCurate outperforms GPT-4V.