2024-02-27

Title: Large Scale Generative AI Text Applied to Sports and Music

• Authors: Aaron Baughman, Stephen Hammer, Rahul Agarwal, Gozde Akay, Eduardo Morales, Tony Johnson, Leonid Karlinsky, Rogerio Feris
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15514
• Pdf URL: https://arxiv.org/pdf/2402.15514
• Copy Paste: [[2402.15514]] Large Scale Generative AI Text Applied to Sports and Music(https://arxiv.org/abs/2402.15514)
• Keywords: generative
• Abstract: We address the problem of scaling up the production of media content, including commentary and personalized news stories, for large-scale sports and music events worldwide. Our approach relies on generative AI models to transform a large volume of multimodal data (e.g., videos, articles, real-time scoring feeds, statistics, and fact sheets) into coherent and fluent text. Based on this approach, we introduce, for the first time, an AI commentary system, which was deployed to produce automated narrations for highlight packages at the 2023 US Open, Wimbledon, and Masters tournaments. In the same vein, our solution was extended to create personalized content for ESPN Fantasy Football and stories about music artists for the Grammy awards. These applications were built using a common software architecture achieved a 15x speed improvement with an average Rouge-L of 82.00 and perplexity of 6.6. Our work was successfully deployed at the aforementioned events, supporting 90 million fans around the world with 8 billion page views, continuously pushing the bounds on what is possible at the intersection of sports, entertainment, and AI.

Title: Beware of Words: Evaluating the Lexical Richness of Conversational Large Language Models

• Authors: Gonzalo Martínez, José Alberto Hernández, Javier Conde, Pedro Reviriego, Elena Merino
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15518
• Pdf URL: https://arxiv.org/pdf/2402.15518
• Copy Paste: [[2402.15518]] Beware of Words: Evaluating the Lexical Richness of Conversational Large Language Models(https://arxiv.org/abs/2402.15518)
• Keywords: large language model
• Abstract: The performance of conversational Large Language Models (LLMs) in general, and of ChatGPT in particular, is currently being evaluated on many different tasks, from logical reasoning or maths to answering questions on a myriad of topics. Instead, much less attention is being devoted to the study of the linguistic features of the texts generated by these LLMs. This is surprising since LLMs are models for language, and understanding how they use the language is important. Indeed, conversational LLMs are poised to have a significant impact on the evolution of languages as they may eventually dominate the creation of new text. This means that for example, if conversational LLMs do not use a word it may become less and less frequent and eventually stop being used altogether. Therefore, evaluating the linguistic features of the text they produce and how those depend on the model parameters is the first step toward understanding the potential impact of conversational LLMs on the evolution of languages. In this paper, we consider the evaluation of the lexical richness of the text generated by LLMs and how it depends on the model parameters. A methodology is presented and used to conduct a comprehensive evaluation of lexical richness using ChatGPT as a case study. The results show how lexical richness depends on the version of ChatGPT and some of its parameters, such as the presence penalty, or on the role assigned to the model. The dataset and tools used in our analysis are released under open licenses with the goal of drawing the much-needed attention to the evaluation of the linguistic features of LLM-generated text.

Title: Detecting misinformation through Framing Theory: the Frame Element-based Model

• Authors: Guan Wang, Rebecca Frederick, Jinglong Duan, William Wong, Verica Rupar, Weihua Li, Quan Bai
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.15525
• Pdf URL: https://arxiv.org/pdf/2402.15525
• Copy Paste: [[2402.15525]] Detecting misinformation through Framing Theory: the Frame Element-based Model(https://arxiv.org/abs/2402.15525)
• Keywords: generative, large language model
• Abstract: In this paper, we delve into the rapidly evolving challenge of misinformation detection, with a specific focus on the nuanced manipulation of narrative frames - an under-explored area within the AI community. The potential for Generative AI models to generate misleading narratives underscores the urgency of this problem. Drawing from communication and framing theories, we posit that the presentation or 'framing' of accurate information can dramatically alter its interpretation, potentially leading to misinformation. We highlight this issue through real-world examples, demonstrating how shifts in narrative frames can transmute fact-based information into misinformation. To tackle this challenge, we propose an innovative approach leveraging the power of pre-trained Large Language Models and deep neural networks to detect misinformation originating from accurate facts portrayed under different frames. These advanced AI techniques offer unprecedented capabilities in identifying complex patterns within unstructured data critical for examining the subtleties of narrative frames. The objective of this paper is to bridge a significant research gap in the AI domain, providing valuable insights and methodologies for tackling framing-induced misinformation, thus contributing to the advancement of responsible and trustworthy AI technologies. Several experiments are intensively conducted and experimental results explicitly demonstrate the various impact of elements of framing theory proving the rationale of applying framing theory to increase the performance in misinformation detection.

Title: PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

• Authors: Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, Baobao Chang
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15527
• Pdf URL: https://arxiv.org/pdf/2402.15527
• Copy Paste: [[2402.15527]] PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain(https://arxiv.org/abs/2402.15527)
• Keywords: robust, large language model
• Abstract: We present PCA-Bench, a multimodal decision-making benchmark for evaluating the integrated capabilities of Multimodal Large Language Models (MLLMs). Departing from previous benchmarks focusing on simplistic tasks and individual model capability, PCA-Bench introduces three complex scenarios: autonomous driving, domestic robotics, and open-world games. Given task instructions and diverse contexts, the model is required to seamlessly integrate multiple capabilities of Perception, Cognition, and Action in a reasoning chain to make accurate decisions. Moreover, PCA-Bench features error localization capabilities, scrutinizing model inaccuracies in areas such as perception, knowledge, or reasoning. This enhances the reliability of deploying MLLMs. To balance accuracy and efficiency in evaluation, we propose PCA-Eval, an automatic evaluation protocol, and assess 10 prevalent MLLMs. The results reveal significant performance disparities between open-source models and powerful proprietary models like GPT-4 Vision. To address this, we introduce Embodied-Instruction-Evolution (EIE), an automatic framework for synthesizing instruction tuning examples in multimodal embodied environments. EIE generates 7,510 training examples in PCA-Bench and enhances the performance of open-source MLLMs, occasionally surpassing GPT-4 Vision (+3\% in decision accuracy), thereby validating the effectiveness of EIE. Our findings suggest that robust MLLMs like GPT4-Vision show promise for decision-making in embodied agents, opening new avenues for MLLM research.

Title: Evaluating the Performance of ChatGPT for Spam Email Detection

• Authors: Yuwei Wu, Shijing Si, Yugui Zhang, Jiawen Gu, Jedrek Wosik
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.15537
• Pdf URL: https://arxiv.org/pdf/2402.15537
• Copy Paste: [[2402.15537]] Evaluating the Performance of ChatGPT for Spam Email Detection(https://arxiv.org/abs/2402.15537)
• Keywords: security, large language model
• Abstract: Email continues to be a pivotal and extensively utilized communication medium within professional and commercial domains. Nonetheless, the prevalence of spam emails poses a significant challenge for users, disrupting their daily routines and diminishing productivity. Consequently, accurately identifying and filtering spam based on content has become crucial for cybersecurity. Recent advancements in natural language processing, particularly with large language models like ChatGPT, have shown remarkable performance in tasks such as question answering and text generation. However, its potential in spam identification remains underexplored. To fill in the gap, this study attempts to evaluate ChatGPT's capabilities for spam identification in both English and Chinese email datasets. We employ ChatGPT for spam email detection using in-context learning, which requires a prompt instruction and a few demonstrations. We also investigate how the training example size affects the performance of ChatGPT. For comparison, we also implement five popular benchmark methods, including naive Bayes, support vector machines (SVM), logistic regression (LR), feedforward dense neural networks (DNN), and BERT classifiers. Though extensive experiments, the performance of ChatGPT is significantly worse than deep supervised learning methods in the large English dataset, while it presents superior performance on the low-resourced Chinese dataset, even outperforming BERT in this case.

Title: Deep Networks Always Grok and Here is Why

• Authors: Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15555
• Pdf URL: https://arxiv.org/pdf/2402.15555
• Copy Paste: [[2402.15555]] Deep Networks Always Grok and Here is Why(https://arxiv.org/abs/2402.15555)
• Keywords: robust, transformer
• Abstract: Grokking, or delayed generalization, is a phenomenon where generalization in a deep neural network (DNN) occurs long after achieving near zero training error. Previous studies have reported the occurrence of grokking in specific controlled settings, such as DNNs initialized with large-norm parameters or transformers trained on algorithmic datasets. We demonstrate that grokking is actually much more widespread and materializes in a wide range of practical settings, such as training of a convolutional neural network (CNN) on CIFAR10 or a Resnet on Imagenette. We introduce the new concept of delayed robustness, whereby a DNN groks adversarial examples and becomes robust, long after interpolation and/or generalization. We develop an analytical explanation for the emergence of both delayed generalization and delayed robustness based on a new measure of the local complexity of a DNN's input-output mapping. Our local complexity measures the density of the so-called 'linear regions' (aka, spline partition regions) that tile the DNN input space, and serves as a utile progress measure for training. We provide the first evidence that for classification problems, the linear regions undergo a phase transition during training whereafter they migrate away from the training samples (making the DNN mapping smoother there) and towards the decision boundary (making the DNN mapping less smooth there). Grokking occurs post phase transition as a robust partition of the input space emerges thanks to the linearization of the DNN mapping around the training points. Website: https://bit.ly/grok-adversarial

Title: Fair Multivariate Adaptive Regression Splines for Ensuring Equity and Transparency

• Authors: Parian Haghighat, Denisa G'andara, Lulu Kang, Hadis Anahideh
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.15561
• Pdf URL: https://arxiv.org/pdf/2402.15561
• Copy Paste: [[2402.15561]] Fair Multivariate Adaptive Regression Splines for Ensuring Equity and Transparency(https://arxiv.org/abs/2402.15561)
• Keywords: fair
• Abstract: Predictive analytics is widely used in various domains, including education, to inform decision-making and improve outcomes. However, many predictive models are proprietary and inaccessible for evaluation or modification by researchers and practitioners, limiting their accountability and ethical design. Moreover, predictive models are often opaque and incomprehensible to the officials who use them, reducing their trust and utility. Furthermore, predictive models may introduce or exacerbate bias and inequity, as they have done in many sectors of society. Therefore, there is a need for transparent, interpretable, and fair predictive models that can be easily adopted and adapted by different stakeholders. In this paper, we propose a fair predictive model based on multivariate adaptive regression splines(MARS) that incorporates fairness measures in the learning process. MARS is a non-parametric regression model that performs feature selection, handles non-linear relationships, generates interpretable decision rules, and derives optimal splitting criteria on the variables. Specifically, we integrate fairness into the knot optimization algorithm and provide theoretical and empirical evidence of how it results in a fair knot placement. We apply our fairMARS model to real-world data and demonstrate its effectiveness in terms of accuracy and equity. Our paper contributes to the advancement of responsible and ethical predictive analytics for social good.

Title: Fast Adversarial Attacks on Language Models In One GPU Minute

• Authors: Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, Soheil Feizi
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15570
• Pdf URL: https://arxiv.org/pdf/2402.15570
• Copy Paste: [[2402.15570]] Fast Adversarial Attacks on Language Models In One GPU Minute(https://arxiv.org/abs/2402.15570)
• Keywords: security, privacy, attack, membership infer
• Abstract: In this paper, we introduce a novel class of fast, beam search-based adversarial attack (BEAST) for Language Models (LMs). BEAST employs interpretable parameters, enabling attackers to balance between attack speed, success rate, and the readability of adversarial prompts. The computational efficiency of BEAST facilitates us to investigate its applications on LMs for jailbreaking, eliciting hallucinations, and privacy attacks. Our gradient-free targeted attack can jailbreak aligned LMs with high attack success rates within one minute. For instance, BEAST can jailbreak Vicuna-7B-v1.5 under one minute with a success rate of 89% when compared to a gradient-based baseline that takes over an hour to achieve 70% success rate using a single Nvidia RTX A6000 48GB GPU. Additionally, we discover a unique outcome wherein our untargeted attack induces hallucinations in LM chatbots. Through human evaluations, we find that our untargeted attack causes Vicuna-7B-v1.5 to produce ~15% more incorrect outputs when compared to LM outputs in the absence of our attack. We also learn that 22% of the time, BEAST causes Vicuna to generate outputs that are not relevant to the original prompt. Further, we use BEAST to generate adversarial prompts in a few seconds that can boost the performance of existing membership inference attacks for LMs. We believe that our fast attack, BEAST, has the potential to accelerate research in LM security and privacy. Our codebase is publicly available at https://github.com/vinusankars/BEAST.

Title: Self-Supervised Pre-Training for Table Structure Recognition Transformer

• Authors: ShengYun Peng, Seongmin Lee, Xiaojing Wang, Rajarajeswari Balasubramaniyan, Duen Horng Chau
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15578
• Pdf URL: https://arxiv.org/pdf/2402.15578
• Copy Paste: [[2402.15578]] Self-Supervised Pre-Training for Table Structure Recognition Transformer(https://arxiv.org/abs/2402.15578)
• Keywords: fair, transformer
• Abstract: Table structure recognition (TSR) aims to convert tabular images into a machine-readable format. Although hybrid convolutional neural network (CNN)-transformer architecture is widely used in existing approaches, linear projection transformer has outperformed the hybrid architecture in numerous vision tasks due to its simplicity and efficiency. However, existing research has demonstrated that a direct replacement of CNN backbone with linear projection leads to a marked performance drop. In this work, we resolve the issue by proposing a self-supervised pre-training (SSP) method for TSR transformers. We discover that the performance gap between the linear projection transformer and the hybrid CNN-transformer can be mitigated by SSP of the visual encoder in the TSR model. We conducted reproducible ablation studies and open-sourced our code at https://github.com/poloclub/unitable to enhance transparency, inspire innovations, and facilitate fair comparisons in our domain as tables are a promising modality for representation learning.

Title: CI w/o TN: Context Injection without Task Name for Procedure Planning

• Authors: Xinjie Li
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15579
• Pdf URL: https://arxiv.org/pdf/2402.15579
• Copy Paste: [[2402.15579]] CI w/o TN: Context Injection without Task Name for Procedure Planning(https://arxiv.org/abs/2402.15579)
• Keywords: large language model
• Abstract: This paper explores the challenge of procedure planning in instructional videos, which involves creating goal-directed plans based on visual start and goal observations from videos. Previous research has tackled this problem with gradually weaker training supervision, from heavy intermediate visual observations or language instructions to task class supervision. However, with the advent of large language models, even given only the task name, these models can produce a detailed plan. In this study, we propose a much weaker setting without task name as supervision, which is not currently solvable by existing large language models since they require good prompts with sufficient information. Specifically, we hypothesize that previous intermediate supervisions can serve as context information, and we use captions of visual start and goal observations as a much cheaper form of supervision. This approach greatly reduces the labeling cost since the captions can be easily obtained by large pre-trained vision-language foundation models. Technically, we apply BLIP to generate captions as supervision to train the context feature with contrastive learning loss. Afterward, the context feature is fed into the generator to aid in plan generation. Our experiments on two datasets with varying scales demonstrate that our model can achieve comparable performance on multiple metrics, which validates our hypothesis.

Title: Cohere3D: Exploiting Temporal Coherence for Unsupervised Representation Learning of Vision-based Autonomous Driving

• Authors: Yichen Xie, Hongge Chen, Gregory P. Meyer, Yong Jae Lee, Eric M. Wolff, Masayoshi Tomizuka, Wei Zhan, Yuning Chai, Xin Huang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.15583
• Pdf URL: https://arxiv.org/pdf/2402.15583
• Copy Paste: [[2402.15583]] Cohere3D: Exploiting Temporal Coherence for Unsupervised Representation Learning of Vision-based Autonomous Driving(https://arxiv.org/abs/2402.15583)
• Keywords: robust, extraction
• Abstract: Due to the lack of depth cues in images, multi-frame inputs are important for the success of vision-based perception, prediction, and planning in autonomous driving. Observations from different angles enable the recovery of 3D object states from 2D image inputs if we can identify the same instance in different input frames. However, the dynamic nature of autonomous driving scenes leads to significant changes in the appearance and shape of each instance captured by the camera at different time steps. To this end, we propose a novel contrastive learning algorithm, Cohere3D, to learn coherent instance representations in a long-term input sequence robust to the change in distance and perspective. The learned representation aids in instance-level correspondence across multiple input frames in downstream tasks. In the pretraining stage, the raw point clouds from LiDAR sensors are utilized to construct the long-term temporal correspondence for each instance, which serves as guidance for the extraction of instance-level representation from the vision-based bird's eye-view (BEV) feature map. Cohere3D encourages a consistent representation for the same instance at different frames but distinguishes between representations of different instances. We evaluate our algorithm by finetuning the pretrained model on various downstream perception, prediction, and planning tasks. Results show a notable improvement in both data efficiency and task performance.

Title: State Space Models for Event Cameras

• Authors: Nikola Zubić, Mathias Gehrig, Davide Scaramuzza
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.15584
• Pdf URL: https://arxiv.org/pdf/2402.15584
• Copy Paste: [[2402.15584]] State Space Models for Event Cameras(https://arxiv.org/abs/2402.15584)
• Keywords: transformer
• Abstract: Today, state-of-the-art deep neural networks that process event-camera data first convert a temporal window of events into dense, grid-like input representations. As such, they exhibit poor generalizability when deployed at higher inference frequencies (i.e., smaller temporal windows) than the ones they were trained on. We address this challenge by introducing state-space models (SSMs) with learnable timescale parameters to event-based vision. This design adapts to varying frequencies without the need to retrain the network at different frequencies. Additionally, we investigate two strategies to counteract aliasing effects when deploying the model at higher frequencies. We comprehensively evaluate our approach against existing methods based on RNN and Transformer architectures across various benchmarks, including Gen1 and 1 Mpx event camera datasets. Our results demonstrate that SSM-based models train 33% faster and also exhibit minimal performance degradation when tested at higher frequencies than the training input. Traditional RNN and Transformer models exhibit performance drops of more than 20 mAP, with SSMs having a drop of 3.31 mAP, highlighting the effectiveness of SSMs in event-based vision tasks.

Title: Distilling Adversarial Robustness Using Heterogeneous Teachers

• Authors: Jieren Deng, Aaron Palmer, Rigel Mahmood, Ethan Rathbun, Jinbo Bi, Kaleel Mahmood, Derek Aguiar
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.15586
• Pdf URL: https://arxiv.org/pdf/2402.15586
• Copy Paste: [[2402.15586]] Distilling Adversarial Robustness Using Heterogeneous Teachers(https://arxiv.org/abs/2402.15586)
• Keywords: defense, attack, robust
• Abstract: Achieving resiliency against adversarial attacks is necessary prior to deploying neural network classifiers in domains where misclassification incurs substantial costs, e.g., self-driving cars or medical imaging. Recent work has demonstrated that robustness can be transferred from an adversarially trained teacher to a student model using knowledge distillation. However, current methods perform distillation using a single adversarial and vanilla teacher and consider homogeneous architectures (i.e., residual networks) that are susceptible to misclassify examples from similar adversarial subspaces. In this work, we develop a defense framework against adversarial attacks by distilling adversarial robustness using heterogeneous teachers (DARHT). In DARHT, the student model explicitly represents teacher logits in a student-teacher feature map and leverages multiple teachers that exhibit low adversarial example transferability (i.e., exhibit high performance on dissimilar adversarial examples). Experiments on classification tasks in both white-box and black-box scenarios demonstrate that DARHT achieves state-of-the-art clean and robust accuracies when compared to competing adversarial training and distillation methods in the CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. Comparisons with homogeneous and heterogeneous teacher sets suggest that leveraging teachers with low adversarial example transferability increases student model robustness.

Title: A Study of Shape Modeling Against Noise

• Authors: Cheng Long, Adrian Barbu
• Subjects: cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.15587
• Pdf URL: https://arxiv.org/pdf/2402.15587
• Copy Paste: [[2402.15587]] A Study of Shape Modeling Against Noise(https://arxiv.org/abs/2402.15587)
• Keywords: generative
• Abstract: Shape modeling is a challenging task with many potential applications in computer vision and medical imaging. There are many shape modeling methods in the literature, each with its advantages and applications. However, many shape modeling methods have difficulties handling shapes that have missing pieces or outliers. In this regard, this paper introduces shape denoising, a fundamental problem in shape modeling that lies at the core of many computer vision and medical imaging applications and has not received enough attention in the literature. The paper introduces six types of noise that can be used to perturb shapes as well as an objective measure for the noise level and for comparing methods on their shape denoising capabilities. Finally, the paper evaluates seven methods capable of accomplishing this task, of which six are based on deep learning, including some generative models.

Title: Prompting LLMs to Compose Meta-Review Drafts from Peer-Review Narratives of Scholarly Manuscripts

• Authors: Shubhra Kanti Karmaker Santu, Sanjeev Kumar Sinha, Naman Bansal, Alex Knipper, Souvika Sarkar, John Salvador, Yash Mahajan, Sri Guttikonda, Mousumi Akter, Matthew Freestone, Matthew C. Williams Jr
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.15589
• Pdf URL: https://arxiv.org/pdf/2402.15589
• Copy Paste: [[2402.15589]] Prompting LLMs to Compose Meta-Review Drafts from Peer-Review Narratives of Scholarly Manuscripts(https://arxiv.org/abs/2402.15589)
• Keywords: generative, large language model
• Abstract: One of the most important yet onerous tasks in the academic peer-reviewing process is composing meta-reviews, which involves understanding the core contributions, strengths, and weaknesses of a scholarly manuscript based on peer-review narratives from multiple experts and then summarizing those multiple experts' perspectives into a concise holistic overview. Given the latest major developments in generative AI, especially Large Language Models (LLMs), it is very compelling to rigorously study the utility of LLMs in generating such meta-reviews in an academic peer-review setting. In this paper, we perform a case study with three popular LLMs, i.e., GPT-3.5, LLaMA2, and PaLM2, to automatically generate meta-reviews by prompting them with different types/levels of prompts based on the recently proposed TELeR taxonomy. Finally, we perform a detailed qualitative study of the meta-reviews generated by the LLMs and summarize our findings and recommendations for prompting LLMs for this complex task.

Title: Differentially Private Fair Binary Classifications

• Authors: Hrad Ghoukasian, Shahab Asoodeh
• Subjects: cs.LG, cs.CR, cs.IT, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.15603
• Pdf URL: https://arxiv.org/pdf/2402.15603
• Copy Paste: [[2402.15603]] Differentially Private Fair Binary Classifications(https://arxiv.org/abs/2402.15603)
• Keywords: privacy, fair
• Abstract: In this work, we investigate binary classification under the constraints of both differential privacy and fairness. We first propose an algorithm based on the decoupling technique for learning a classifier with only fairness guarantee. This algorithm takes in classifiers trained on different demographic groups and generates a single classifier satisfying statistical parity. We then refine this algorithm to incorporate differential privacy. The performance of the final algorithm is rigorously examined in terms of privacy, fairness, and utility guarantees. Empirical evaluations conducted on the Adult and Credit Card datasets illustrate that our algorithm outperforms the state-of-the-art in terms of fairness guarantees, while maintaining the same level of privacy and utility.

Title: Training Nonlinear Transformers for Efficient In-Context Learning: A Theoretical Learning and Generalization Analysis

• Authors: Hongkang Li, Meng Wang, Songtao Lu, Xiaodong Cui, Pin-Yu Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.15607
• Pdf URL: https://arxiv.org/pdf/2402.15607
• Copy Paste: [[2402.15607]] Training Nonlinear Transformers for Efficient In-Context Learning: A Theoretical Learning and Generalization Analysis(https://arxiv.org/abs/2402.15607)
• Keywords: transformer, large language model
• Abstract: Transformer-based large language models have displayed impressive in-context learning capabilities, where a pre-trained model can handle new tasks without fine-tuning by simply augmenting the query with some input-output examples from that task. Despite the empirical success, the mechanics of how to train a Transformer to achieve ICL and the corresponding ICL capacity is mostly elusive due to the technical challenges of analyzing the nonconvex training problems resulting from the nonlinear self-attention and nonlinear activation in Transformers. To the best of our knowledge, this paper provides the first theoretical analysis of the training dynamics of Transformers with nonlinear self-attention and nonlinear MLP, together with the ICL generalization capability of the resulting model. Focusing on a group of binary classification tasks, we train Transformers using data from a subset of these tasks and quantify the impact of various factors on the ICL generalization performance on the remaining unseen tasks with and without data distribution shifts. We also analyze how different components in the learned Transformers contribute to the ICL performance. Furthermore, we provide the first theoretical analysis of how model pruning affects the ICL performance and prove that proper magnitude-based pruning can have a minimal impact on ICL while reducing inference costs. These theoretical findings are justified through numerical experiments.

Title: Towards Efficient Active Learning in NLP via Pretrained Representations

• Authors: Artem Vysogorets, Achintya Gopal
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15613
• Pdf URL: https://arxiv.org/pdf/2402.15613
• Copy Paste: [[2402.15613]] Towards Efficient Active Learning in NLP via Pretrained Representations(https://arxiv.org/abs/2402.15613)
• Keywords: large language model
• Abstract: Fine-tuning Large Language Models (LLMs) is now a common approach for text classification in a wide range of applications. When labeled documents are scarce, active learning helps save annotation efforts but requires retraining of massive models on each acquisition iteration. We drastically expedite this process by using pretrained representations of LLMs within the active learning loop and, once the desired amount of labeled data is acquired, fine-tuning that or even a different pretrained LLM on this labeled data to achieve the best performance. As verified on common text classification benchmarks with pretrained BERT and RoBERTa as the backbone, our strategy yields similar performance to fine-tuning all the way through the active learning loop but is orders of magnitude less computationally expensive. The data acquired with our procedure generalizes across pretrained networks, allowing flexibility in choosing the final model or updating it as newer versions get released.

Title: Reinforcement Learning-Based Approaches for Enhancing Security and Resilience in Smart Control: A Survey on Attack and Defense Methods

• Authors: Zheyu Zhang
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.15617
• Pdf URL: https://arxiv.org/pdf/2402.15617
• Copy Paste: [[2402.15617]] Reinforcement Learning-Based Approaches for Enhancing Security and Resilience in Smart Control: A Survey on Attack and Defense Methods(https://arxiv.org/abs/2402.15617)
• Keywords: secure, security, defense, attack
• Abstract: Reinforcement Learning (RL), one of the core paradigms in machine learning, learns to make decisions based on real-world experiences. This approach has significantly advanced AI applications across various domains, notably in smart grid optimization and smart home automation. However, the proliferation of RL in these critical sectors has also exposed them to sophisticated adversarial attacks that target the underlying neural network policies, compromising system integrity. Given the pivotal role of RL in enhancing the efficiency and sustainability of smart grids and the personalized convenience in smart homes, ensuring the security of these systems is paramount. This paper aims to bolster the resilience of RL frameworks within these specific contexts, addressing the unique challenges posed by the intricate and potentially adversarial environments of smart grids and smart homes. We provide a thorough review of the latest adversarial RL threats and outline effective defense strategies tailored to safeguard these applications. Our comparative analysis sheds light on the nuances of adversarial tactics against RL-driven smart systems and evaluates the defense mechanisms, focusing on their innovative contributions, limitations, and the compromises they entail. By concentrating on the smart grid and smart home scenarios, this survey equips ML developers and researchers with the insights needed to secure RL applications against emerging threats, ensuring their reliability and safety in our increasingly connected world.

Title: Language-Based User Profiles for Recommendation

• Authors: Joyce Zhou, Yijia Dai, Thorsten Joachims
• Subjects: cs.CL, cs.HC, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.15623
• Pdf URL: https://arxiv.org/pdf/2402.15623
• Copy Paste: [[2402.15623]] Language-Based User Profiles for Recommendation(https://arxiv.org/abs/2402.15623)
• Keywords: interpretability, large language model
• Abstract: Most conventional recommendation methods (e.g., matrix factorization) represent user profiles as high-dimensional vectors. Unfortunately, these vectors lack interpretability and steerability, and often perform poorly in cold-start settings. To address these shortcomings, we explore the use of user profiles that are represented as human-readable text. We propose the Language-based Factorization Model (LFM), which is essentially an encoder/decoder model where both the encoder and the decoder are large language models (LLMs). The encoder LLM generates a compact natural-language profile of the user's interests from the user's rating history. The decoder LLM uses this summary profile to complete predictive downstream tasks. We evaluate our LFM approach on the MovieLens dataset, comparing it against matrix factorization and an LLM model that directly predicts from the user's rating history. In cold-start settings, we find that our method can have higher accuracy than matrix factorization. Furthermore, we find that generating a compact and human-readable summary often performs comparably with or better than direct LLM prediction, while enjoying better interpretability and shorter model input length. Our results motivate a number of future research directions and potential improvements.

Title: MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

• Authors: Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, Xin Liu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.15627
• Pdf URL: https://arxiv.org/pdf/2402.15627
• Copy Paste: [[2402.15627]] MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs(https://arxiv.org/abs/2402.15627)
• Keywords: large language model
• Abstract: We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high efficiency throughout the training process (i.e., stability) is an important consideration in production given the long extent of LLM training jobs. Many hard stability issues only emerge at large scale, and in-depth observability is the key to address them. We develop a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the MFU by 1.34x compared to Megatron-LM. We share our operational experience in identifying and fixing failures and stragglers. We hope by articulating the problems and sharing our experience from a systems perspective, this work can inspire future LLM systems research.

Title: Fine-Grained Self-Endorsement Improves Factuality and Reasoning

• Authors: Ante Wang, Linfeng Song, Baolin Peng, Ye Tian, Lifeng Jin, Haitao Mi, Jinsong Su, Dong Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15631
• Pdf URL: https://arxiv.org/pdf/2402.15631
• Copy Paste: [[2402.15631]] Fine-Grained Self-Endorsement Improves Factuality and Reasoning(https://arxiv.org/abs/2402.15631)
• Keywords: large language model
• Abstract: This work studies improving large language model (LLM) generations at inference time by mitigating fact-conflicting hallucinations. Particularly, we propose a self-endorsement framework that leverages the fine-grained fact-level comparisons across multiple sampled responses. Compared with prior ensemble methods (Wang et al., 2022;Chen et al., 2023)) that perform response-level selection, our approach can better alleviate hallucinations, especially for longform generation tasks. Our approach can broadly benefit smaller and open-source LLMs as it mainly conducts simple content-based comparisons. Experiments on Biographies show that our method can effectively improve the factuality of generations with simple and intuitive prompts across different scales of LLMs. Besides, comprehensive analyses on TriviaQA and GSM8K demonstrate the potential of self-endorsement for broader application.

Title: Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Models

• Authors: Yanzheng Xiang, Hanqi Yan, Lin Gui, Yulan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15637
• Pdf URL: https://arxiv.org/pdf/2402.15637
• Copy Paste: [[2402.15637]] Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Models(https://arxiv.org/abs/2402.15637)
• Keywords: robust
• Abstract: In-context learning has become a popular paradigm in natural language processing. However, its performance can be significantly influenced by the order of in-context demonstration examples. In this paper, we found that causal language models (CausalLMs) are more sensitive to this order compared to prefix language models (PrefixLMs). We attribute this phenomenon to the auto-regressive attention masks within CausalLMs, which restrict each token from accessing information from subsequent tokens. This results in different receptive fields for samples at different positions, thereby leading to representation disparities across positions. To tackle this challenge, we introduce an unsupervised fine-tuning method, termed the Information-Augmented and Consistency-Enhanced approach. This approach utilizes contrastive learning to align representations of in-context examples across different positions and introduces a consistency loss to ensure similar representations for inputs with different permutations. This enhances the model's predictive consistency across permutations. Experimental results on four benchmarks suggest that our proposed method can reduce the sensitivity to the order of in-context examples and exhibit robust generalizability, particularly when demonstrations are sourced from a pool different from that used in the training phase, or when the number of in-context examples differs from what is used during training.

Title: Fair Resource Allocation in Multi-Task Learning

• Authors: Hao Ban, Kaiyi Ji
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.15638
• Pdf URL: https://arxiv.org/pdf/2402.15638
• Copy Paste: [[2402.15638]] Fair Resource Allocation in Multi-Task Learning(https://arxiv.org/abs/2402.15638)
• Keywords: fair
• Abstract: By jointly learning multiple tasks, multi-task learning (MTL) can leverage the shared knowledge across tasks, resulting in improved data efficiency and generalization performance. However, a major challenge in MTL lies in the presence of conflicting gradients, which can hinder the fair optimization of some tasks and subsequently impede MTL's ability to achieve better overall performance. Inspired by fair resource allocation in communication networks, we formulate the optimization of MTL as a utility maximization problem, where the loss decreases across tasks are maximized under different fairness measurements. To solve this problem, we propose FairGrad, a novel MTL optimization method. FairGrad not only enables flexible emphasis on certain tasks but also achieves a theoretical convergence guarantee. Extensive experiments demonstrate that our method can achieve state-of-the-art performance among gradient manipulation methods on a suite of multi-task benchmarks in supervised learning and reinforcement learning. Furthermore, we incorporate the idea of $\alpha$-fairness into loss functions of various MTL methods. Extensive empirical studies demonstrate that their performance can be significantly enhanced. Code is provided at \url{https://github.com/OptMN-Lab/fairgrad}.

Title: MambaIR: A Simple Baseline for Image Restoration with State-Space Model

• Authors: Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, Shu-Tao Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15648
• Pdf URL: https://arxiv.org/pdf/2402.15648
• Copy Paste: [[2402.15648]] MambaIR: A Simple Baseline for Image Restoration with State-Space Model(https://arxiv.org/abs/2402.15648)
• Keywords: transformer
• Abstract: Recent years have witnessed great progress in image restoration thanks to the advancements in modern deep neural networks e.g. Convolutional Neural Network and Transformer. However, existing restoration backbones are usually limited due to the inherent local reductive bias or quadratic computational complexity. Recently, Selective Structured State Space Model e.g., Mamba, has shown great potential for long-range dependencies modeling with linear complexity, but it is still under-explored in low-level computer vision. In this work, we introduce a simple but strong benchmark model, named MambaIR, for image restoration. In detail, we propose the Residual State Space Block as the core component, which employs convolution and channel attention to enhance the capabilities of the vanilla Mamba. In this way, our MambaIR takes advantage of local patch recurrence prior as well as channel interaction to produce restoration-specific feature representation. Extensive experiments demonstrate the superiority of our method, for example, MambaIR outperforms Transformer-based baseline SwinIR by up to 0.36dB, using similar computational cost but with a global receptive field. Code is available at \url{https://github.com/csguoh/MambaIR}.

Title: Low-Frequency Black-Box Backdoor Attack via Evolutionary Algorithm

• Authors: Yanqi Qiao, Dazhuang Liu, Rui Wang, Kaitai Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15653
• Pdf URL: https://arxiv.org/pdf/2402.15653
• Copy Paste: [[2402.15653]] Low-Frequency Black-Box Backdoor Attack via Evolutionary Algorithm(https://arxiv.org/abs/2402.15653)
• Keywords: defense, attack, robust, steal
• Abstract: While convolutional neural networks (CNNs) have achieved success in computer vision tasks, it is vulnerable to backdoor attacks. Such attacks could mislead the victim model to make attacker-chosen prediction with a specific trigger pattern. Until now, the trigger injection of existing attacks is mainly limited to spatial domain. Recent works take advantage of perceptual properties of planting specific patterns in the frequency domain, which only reflect indistinguishable pixel-wise perturbations in pixel domain. However, in the black-box setup, the inaccessibility of training process often renders more complex trigger designs. Existing frequency attacks simply handcraft the magnitude of spectrum, introducing anomaly frequency disparities between clean and poisoned data and taking risks of being removed by image processing operations (such as lossy compression and filtering). In this paper, we propose a robust low-frequency black-box backdoor attack (LFBA), which minimally perturbs low-frequency components of frequency spectrum and maintains the perceptual similarity in spatial space simultaneously. The key insight of our attack restrict the search for the optimal trigger to low-frequency region that can achieve high attack effectiveness, robustness against image transformation defenses and stealthiness in dual space. We utilize simulated annealing (SA), a form of evolutionary algorithm, to optimize the properties of frequency trigger including the number of manipulated frequency bands and the perturbation of each frequency component, without relying on the knowledge from the victim classifier. Extensive experiments on real-world datasets verify the effectiveness and robustness of LFBA against image processing operations and the state-of-the-art backdoor defenses, as well as its inherent stealthiness in both spatial and frequency space, making it resilient against frequency inspection.

Title: Learning Semilinear Neural Operators : A Unified Recursive Framework For Prediction And Data Assimilation

• Authors: Ashutosh Singh, Ricardo Augusto Borsoi, Deniz Erdogmus, Tales Imbiriba
• Subjects: cs.LG, cs.AI, math.NA
• Abstract URL: https://arxiv.org/abs/2402.15656
• Pdf URL: https://arxiv.org/pdf/2402.15656
• Copy Paste: [[2402.15656]] Learning Semilinear Neural Operators : A Unified Recursive Framework For Prediction And Data Assimilation(https://arxiv.org/abs/2402.15656)
• Keywords: robust
• Abstract: Recent advances in the theory of Neural Operators (NOs) have enabled fast and accurate computation of the solutions to complex systems described by partial differential equations (PDEs). Despite their great success, current NO-based solutions face important challenges when dealing with spatio-temporal PDEs over long time scales. Specifically, the current theory of NOs does not present a systematic framework to perform data assimilation and efficiently correct the evolution of PDE solutions over time based on sparsely sampled noisy measurements. In this paper, we propose a learning-based state-space approach to compute the solution operators to infinite-dimensional semilinear PDEs. Exploiting the structure of semilinear PDEs and the theory of nonlinear observers in function spaces, we develop a flexible recursive method that allows for both prediction and data assimilation by combining prediction and correction operations. The proposed framework is capable of producing fast and accurate predictions over long time horizons, dealing with irregularly sampled noisy measurements to correct the solution, and benefits from the decoupling between the spatial and temporal dynamics of this class of PDEs. We show through experiments on the Kuramoto-Sivashinsky, Navier-Stokes and Korteweg-de Vries equations that the proposed model is robust to noise and can leverage arbitrary amounts of measurements to correct its prediction over a long time horizon with little computational overhead.

Title: Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study

• Authors: Zhaoyue Sun, Gabriele Pergola, Byron C. Wallace, Yulan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15663
• Pdf URL: https://arxiv.org/pdf/2402.15663
• Copy Paste: [[2402.15663]] Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study(https://arxiv.org/abs/2402.15663)
• Keywords: extraction, large language model
• Abstract: With the advent of large language models (LLMs), there has been growing interest in exploring their potential for medical applications. This research aims to investigate the ability of LLMs, specifically ChatGPT, in the context of pharmacovigilance event extraction, of which the main goal is to identify and extract adverse events or potential therapeutic events from textual medical sources. We conduct extensive experiments to assess the performance of ChatGPT in the pharmacovigilance event extraction task, employing various prompts and demonstration selection strategies. The findings demonstrate that while ChatGPT demonstrates reasonable performance with appropriate demonstration selection strategies, it still falls short compared to fully fine-tuned small models. Additionally, we explore the potential of leveraging ChatGPT for data augmentation. However, our investigation reveals that the inclusion of synthesized data into fine-tuning may lead to a decrease in performance, possibly attributed to noise in the ChatGPT-generated labels. To mitigate this, we explore different filtering strategies and find that, with the proper approach, more stable performance can be achieved, although constant improvement remains elusive.

Title: General Purpose Image Encoder DINOv2 for Medical Image Registration

• Authors: Xinrui Song, Xuanang Xu, Pingkun Yan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15687
• Pdf URL: https://arxiv.org/pdf/2402.15687
• Copy Paste: [[2402.15687]] General Purpose Image Encoder DINOv2 for Medical Image Registration(https://arxiv.org/abs/2402.15687)
• Keywords: extraction
• Abstract: Existing medical image registration algorithms rely on either dataset specific training or local texture-based features to align images. The former cannot be reliably implemented without large modality-specific training datasets, while the latter lacks global semantics thus could be easily trapped at local minima. In this paper, we present a training-free deformable image registration method, DINO-Reg, leveraging a general purpose image encoder DINOv2 for image feature extraction. The DINOv2 encoder was trained using the ImageNet data containing natural images. We used the pretrained DINOv2 without any finetuning. Our method feeds the DINOv2 encoded features into a discrete optimizer to find the optimal deformable registration field. We conducted a series of experiments to understand the behavior and role of such a general purpose image encoder in the application of image registration. Combined with handcrafted features, our method won the first place in the recent OncoReg Challenge. To our knowledge, this is the first application of general vision foundation models in medical image registration.

Title: Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology

• Authors: Zhenhua Wang, Wei Xie, Baosheng Wang, Enze Wang, Zhiwen Gui, Shuoyoucheng Ma, Kai Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15690
• Pdf URL: https://arxiv.org/pdf/2402.15690
• Copy Paste: [[2402.15690]] Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology(https://arxiv.org/abs/2402.15690)
• Keywords: security, protect, attack, large language model
• Abstract: Large Language Models (LLMs) have gradually become the gateway for people to acquire new knowledge. However, attackers can break the model's security protection ("jail") to access restricted information, which is called "jailbreaking." Previous studies have shown the weakness of current LLMs when confronted with such jailbreaking attacks. Nevertheless, comprehension of the intrinsic decision-making mechanism within the LLMs upon receipt of jailbreak prompts is noticeably lacking. Our research provides a psychological explanation of the jailbreak prompts. Drawing on cognitive consistency theory, we argue that the key to jailbreak is guiding the LLM to achieve cognitive coordination in an erroneous direction. Further, we propose an automatic black-box jailbreaking method based on the Foot-in-the-Door (FITD) technique. This method progressively induces the model to answer harmful questions via multi-step incremental prompts. We instantiated a prototype system to evaluate the jailbreaking effectiveness on 8 advanced LLMs, yielding an average success rate of 83.9%. This study builds a psychological perspective on the explanatory insights into the intrinsic decision-making logic of LLMs.

Title: Orthogonal Gradient Boosting for Simpler Additive Rule Ensembles

• Authors: Fan Yang, Pierre Le Bodic, Michael Kamp, Mario Boley
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.15691
• Pdf URL: https://arxiv.org/pdf/2402.15691
• Copy Paste: [[2402.15691]] Orthogonal Gradient Boosting for Simpler Additive Rule Ensembles(https://arxiv.org/abs/2402.15691)
• Keywords: interpretability
• Abstract: Gradient boosting of prediction rules is an efficient approach to learn potentially interpretable yet accurate probabilistic models. However, actual interpretability requires to limit the number and size of the generated rules, and existing boosting variants are not designed for this purpose. Though corrective boosting refits all rule weights in each iteration to minimise prediction risk, the included rule conditions tend to be sub-optimal, because commonly used objective functions fail to anticipate this refitting. Here, we address this issue by a new objective function that measures the angle between the risk gradient vector and the projection of the condition output vector onto the orthogonal complement of the already selected conditions. This approach correctly approximate the ideal update of adding the risk gradient itself to the model and favours the inclusion of more general and thus shorter rules. As we demonstrate using a wide range of prediction tasks, this significantly improves the comprehensibility/accuracy trade-off of the fitted ensemble. Additionally, we show how objective values for related rule conditions can be computed incrementally to avoid any substantial computational overhead of the new method.

Title: CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learning

• Authors: Junyu Luo, Xiaochen Wang, Jiaqi Wang, Aofei Chang, Yaqing Wang, Fenglong Ma
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15700
• Pdf URL: https://arxiv.org/pdf/2402.15700
• Copy Paste: [[2402.15700]] CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learning(https://arxiv.org/abs/2402.15700)
• Keywords: extraction
• Abstract: Automatic International Classification of Diseases (ICD) coding plays a crucial role in the extraction of relevant information from clinical notes for proper recording and billing. One of the most important directions for boosting the performance of automatic ICD coding is modeling ICD code relations. However, current methods insufficiently model the intricate relationships among ICD codes and often overlook the importance of context in clinical notes. In this paper, we propose a novel approach, a contextualized and flexible framework, to enhance the learning of ICD code representations. Our approach, unlike existing methods, employs a dependent learning paradigm that considers the context of clinical notes in modeling all possible code relations. We evaluate our approach on six public ICD coding datasets and the experimental results demonstrate the effectiveness of our approach compared to state-of-the-art baselines.

Title: A Statistical Analysis of Wasserstein Autoencoders for Intrinsically Low-dimensional Data

• Authors: Saptarshi Chakraborty, Peter L. Bartlett
• Subjects: cs.LG, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.15710
• Pdf URL: https://arxiv.org/pdf/2402.15710
• Copy Paste: [[2402.15710]] A Statistical Analysis of Wasserstein Autoencoders for Intrinsically Low-dimensional Data(https://arxiv.org/abs/2402.15710)
• Keywords: interpretability
• Abstract: Variational Autoencoders (VAEs) have gained significant popularity among researchers as a powerful tool for understanding unknown distributions based on limited samples. This popularity stems partly from their impressive performance and partly from their ability to provide meaningful feature representations in the latent space. Wasserstein Autoencoders (WAEs), a variant of VAEs, aim to not only improve model efficiency but also interpretability. However, there has been limited focus on analyzing their statistical guarantees. The matter is further complicated by the fact that the data distributions to which WAEs are applied - such as natural images - are often presumed to possess an underlying low-dimensional structure within a high-dimensional feature space, which current theory does not adequately account for, rendering known bounds inefficient. To bridge the gap between the theory and practice of WAEs, in this paper, we show that WAEs can learn the data distributions when the network architectures are properly chosen. We show that the convergence rates of the expected excess risk in the number of samples for WAEs are independent of the high feature dimension, instead relying only on the intrinsic dimension of the data distribution.

Title: Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors

• Authors: Shengkun Ma, Jiale Han, Yi Liang, Bo Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15713
• Pdf URL: https://arxiv.org/pdf/2402.15713
• Copy Paste: [[2402.15713]] Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors(https://arxiv.org/abs/2402.15713)
• Keywords: extraction
• Abstract: Continual Few-shot Relation Extraction (CFRE) is a practical problem that requires the model to continuously learn novel relations while avoiding forgetting old ones with few labeled training data. The primary challenges are catastrophic forgetting and overfitting. This paper harnesses prompt learning to explore the implicit capabilities of pre-trained language models to address the above two challenges, thereby making language models better continual few-shot relation extractors. Specifically, we propose a Contrastive Prompt Learning framework, which designs prompt representation to acquire more generalized knowledge that can be easily adapted to old and new categories, and margin-based contrastive learning to focus more on hard samples, therefore alleviating catastrophic forgetting and overfitting issues. To further remedy overfitting in low-resource scenarios, we introduce an effective memory augmentation strategy that employs well-crafted prompts to guide ChatGPT in generating diverse samples. Extensive experiments demonstrate that our method outperforms state-of-the-art methods by a large margin and significantly mitigates catastrophic forgetting and overfitting in low-resource scenarios.

Title: LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Paper

• Authors: Daoyuan Wu, Shuai Wang, Yang Liu, Ning Liu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15727
• Pdf URL: https://arxiv.org/pdf/2402.15727
• Copy Paste: [[2402.15727]] LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Paper(https://arxiv.org/abs/2402.15727)
• Keywords: defense, attack, large language model
• Abstract: Jailbreaking is an emerging adversarial attack that bypasses the safety alignment deployed in off-the-shelf large language models (LLMs). A considerable amount of research exists proposing more effective jailbreak attacks, including the recent Greedy Coordinate Gradient (GCG) attack, jailbreak template-based attacks such as using "Do-Anything-Now" (DAN), and multilingual jailbreak. In contrast, the defensive side has been relatively less explored. This paper proposes a lightweight yet practical defense called SELFDEFEND, which can defend against all existing jailbreak attacks with minimal delay for jailbreak prompts and negligible delay for normal user prompts. Our key insight is that regardless of the kind of jailbreak strategies employed, they eventually need to include a harmful prompt (e.g., "how to make a bomb") in the prompt sent to LLMs, and we found that existing LLMs can effectively recognize such harmful prompts that violate their safety policies. Based on this insight, we design a shadow stack that concurrently checks whether a harmful prompt exists in the user prompt and triggers a checkpoint in the normal stack once a token of "No" or a harmful prompt is output. The latter could also generate an explainable LLM response to adversarial prompts. We demonstrate our idea of SELFDEFEND works in various jailbreak scenarios through manual analysis in GPT-3.5/4. We also list three future directions to further enhance SELFDEFEND.

Title: Clustering in Dynamic Environments: A Framework for Benchmark Dataset Generation With Heterogeneous Changes

• Authors: Danial Yazdani, Juergen Branke, Mohammad Sadegh Khorshidi, Mohammad Nabi Omidvar, Xiaodong Li, Amir H. Gandomi, Xin Yao
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.15731
• Pdf URL: https://arxiv.org/pdf/2402.15731
• Copy Paste: [[2402.15731]] Clustering in Dynamic Environments: A Framework for Benchmark Dataset Generation With Heterogeneous Changes(https://arxiv.org/abs/2402.15731)
• Keywords: robust
• Abstract: Clustering in dynamic environments is of increasing importance, with broad applications ranging from real-time data analysis and online unsupervised learning to dynamic facility location problems. While meta-heuristics have shown promising effectiveness in static clustering tasks, their application for tracking optimal clustering solutions or robust clustering over time in dynamic environments remains largely underexplored. This is partly due to a lack of dynamic datasets with diverse, controllable, and realistic dynamic characteristics, hindering systematic performance evaluations of clustering algorithms in various dynamic scenarios. This deficiency leads to a gap in our understanding and capability to effectively design algorithms for clustering in dynamic environments. To bridge this gap, this paper introduces the Dynamic Dataset Generator (DDG). DDG features multiple dynamic Gaussian components integrated with a range of heterogeneous, local, and global changes. These changes vary in spatial and temporal severity, patterns, and domain of influence, providing a comprehensive tool for simulating a wide range of dynamic scenarios.

Title: Privacy-Preserving State Estimation in the Presence of Eavesdroppers: A Survey

• Authors: Xinhao Yan, Guanzhong Zhou, Daniel E. Quevedo, Carlos Murguia, Bo Chen, Hailong Huang
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.15738
• Pdf URL: https://arxiv.org/pdf/2402.15738
• Copy Paste: [[2402.15738]] Privacy-Preserving State Estimation in the Presence of Eavesdroppers: A Survey(https://arxiv.org/abs/2402.15738)
• Keywords: privacy, protect, attack, steal
• Abstract: Networked systems are increasingly the target of cyberattacks that exploit vulnerabilities within digital communications, embedded hardware, and software. Arguably, the simplest class of attacks -- and often the first type before launching destructive integrity attacks -- are eavesdropping attacks, which aim to infer information by collecting system data and exploiting it for malicious purposes. A key technology of networked systems is state estimation, which leverages sensing and actuation data and first-principles models to enable trajectory planning, real-time monitoring, and control. However, state estimation can also be exploited by eavesdroppers to identify models and reconstruct states with the aim of, e.g., launching integrity (stealthy) attacks and inferring sensitive information. It is therefore crucial to protect disclosed system data to avoid an accurate state estimation by eavesdroppers. This survey presents a comprehensive review of existing literature on privacy-preserving state estimation methods, while also identifying potential limitations and research gaps. Our primary focus revolves around three types of methods: cryptography, data perturbation, and transmission scheduling, with particular emphasis on Kalman-like filters. Within these categories, we delve into the concepts of homomorphic encryption and differential privacy, which have been extensively investigated in recent years in the context of privacy-preserving state estimation. Finally, we shed light on several technical and fundamental challenges surrounding current methods and propose potential directions for future research.

Title: Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning

• Authors: Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, Yang You
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15751
• Pdf URL: https://arxiv.org/pdf/2402.15751
• Copy Paste: [[2402.15751]] Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning(https://arxiv.org/abs/2402.15751)
• Keywords: large language model
• Abstract: While fine-tuning large language models (LLMs) for specific tasks often yields impressive results, it comes at the cost of memory inefficiency due to back-propagation in gradient-based training. Memory-efficient Zeroth-order (MeZO) optimizers, recently proposed to address this issue, only require forward passes during training, making them more memory-friendly. However, the quality of gradient estimates in zeroth order optimization often depends on the data dimensionality, potentially explaining why MeZO still exhibits significant performance drops compared to standard fine-tuning across various tasks. Inspired by the success of Parameter-Efficient Fine-Tuning (PEFT), this paper introduces Sparse MeZO, a novel memory-efficient zeroth-order optimization approach that applies ZO only to a carefully chosen subset of parameters. We propose a simple yet effective parameter selection scheme that yields significant performance gains with Sparse-MeZO. Additionally, we develop a memory-optimized implementation for sparse masking, ensuring the algorithm requires only inference-level memory consumption, allowing Sparse-MeZO to fine-tune LLaMA-30b on a single A100 GPU. Experimental results illustrate that Sparse-MeZO consistently improves both performance and convergence speed over MeZO without any overhead. For example, it achieves a 9\% absolute accuracy improvement and 3.5x speedup over MeZO on the RTE task.

Title: HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition

• Authors: Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15754
• Pdf URL: https://arxiv.org/pdf/2402.15754
• Copy Paste: [[2402.15754]] HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition(https://arxiv.org/abs/2402.15754)
• Keywords: large language model
• Abstract: Large language models (LLMs) have emerged as a promising alternative to expensive human evaluations. However, the alignment and coverage of LLM-based evaluations are often limited by the scope and potential bias of the evaluation prompts and criteria. To address this challenge, we propose HD-Eval, a novel framework that iteratively aligns LLM-based evaluators with human preference via Hierarchical Criteria Decomposition. HD-Eval inherits the essence from the evaluation mindset of human experts and enhances the alignment of LLM-based evaluators by decomposing a given evaluation task into finer-grained criteria, aggregating them according to estimated human preferences, pruning insignificant criteria with attribution, and further decomposing significant criteria. By integrating these steps within an iterative alignment training process, we obtain a hierarchical decomposition of criteria that comprehensively captures aspects of natural language at multiple levels of granularity. Implemented as a white box, the human preference-guided aggregator is efficient to train and more explainable than relying solely on prompting, and its independence from model parameters makes it applicable to closed-source LLMs. Extensive experiments on three evaluation domains demonstrate the superiority of HD-Eval in further aligning state-of-the-art evaluators and providing deeper insights into the explanation of evaluation results and the task itself.

Title: Dental Severity Assessment through Few-shot Learning and SBERT Fine-tuning

• Authors: Mohammad Dehghani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15755
• Pdf URL: https://arxiv.org/pdf/2402.15755
• Copy Paste: [[2402.15755]] Dental Severity Assessment through Few-shot Learning and SBERT Fine-tuning(https://arxiv.org/abs/2402.15755)
• Keywords: large language model
• Abstract: Dental diseases have a significant impact on a considerable portion of the population, leading to various health issues that can detrimentally affect individuals' overall well-being. The integration of automated systems in oral healthcare has become increasingly crucial. Machine learning approaches offer a viable solution to address challenges such as diagnostic difficulties, inefficiencies, and errors in oral disease diagnosis. These methods prove particularly useful when physicians struggle to predict or diagnose diseases at their early stages. In this study, thirteen different machine learning, deep learning, and large language models were employed to determine the severity level of oral health issues based on radiologists' reports. The results revealed that the Few-shot learning with SBERT and Multi-Layer Perceptron model outperformed all other models across various experiments, achieving an impressive accuracy of 94.1% as the best result. Consequently, this model exhibits promise as a reliable tool for evaluating the severity of oral diseases, enabling patients to receive more effective treatment and aiding healthcare professionals in making informed decisions regarding resource allocation and the management of high-risk patients.

Title: Detection Is Tracking: Point Cloud Multi-Sweep Deep Learning Models Revisited

• Authors: Lingji Chen
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.15756
• Pdf URL: https://arxiv.org/pdf/2402.15756
• Copy Paste: [[2402.15756]] Detection Is Tracking: Point Cloud Multi-Sweep Deep Learning Models Revisited(https://arxiv.org/abs/2402.15756)
• Keywords: robust, fair
• Abstract: Conventional tracking paradigm takes in instantaneous measurements such as range and bearing, and produces object tracks across time. In applications such as autonomous driving, lidar measurements in the form of point clouds are usually passed through a "virtual sensor" realized by a deep learning model, to produce "measurements" such as bounding boxes, which are in turn ingested by a tracking module to produce object tracks. Very often multiple lidar sweeps are accumulated in a buffer to merge and become the input to the virtual sensor. We argue in this paper that such an input already contains temporal information, and therefore the virtual sensor output should also contain temporal information, not just instantaneous values for the time corresponding to the end of the buffer. In particular, we present the deep learning model called MULti-Sweep PAired Detector (MULSPAD) that produces, for each detected object, a pair of bounding boxes at both the end time and the beginning time of the input buffer. This is achieved with fairly straightforward changes in commonly used lidar detection models, and with only marginal extra processing, but the resulting symmetry is satisfying. Such paired detections make it possible not only to construct rudimentary trackers fairly easily, but also to construct more sophisticated trackers that can exploit the extra information conveyed by the pair and be robust to choices of motion models and object birth/death models. We have conducted preliminary training and experimentation using Waymo Open Dataset, which shows the efficacy of our proposed method.

Title: Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens

• Authors: Ziqian Zeng, Jiahong Yu, Qianshi Pang, Zihao Wang, Huiping Zhuang, Cen Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15758
• Pdf URL: https://arxiv.org/pdf/2402.15758
• Copy Paste: [[2402.15758]] Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens(https://arxiv.org/abs/2402.15758)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across various tasks. However, their widespread application is hindered by the resource-intensive decoding process. To address this challenge, current approaches have incorporated additional decoding heads to enable parallel prediction of multiple subsequent tokens, thereby achieving inference acceleration. Nevertheless, the accuracy of these decoding heads falls short of the auto-regressive decoding approach. In light of these limitations, we propose Chimera, a novel framework specifically designed for speculative sampling. Within this framework, we introduce a lightweight draft model that effectively utilizes previously generated tokens to predict subsequent words. To ensure both accuracy and efficiency, we present two strategies within the lightweight draft model. Firstly, we focus on capturing short-range dependencies at the bottom layer. Secondly, we leverage the readily available representations from the original LLM.Through empirical evaluation on the Vicuna and LlaMA-2 series, Chimera demonstrates impressive results, achieving an average latency speedup ratio of 2.7x compared to the vanilla auto-regressive decoding approach. This highlights the potential of our proposed framework in significantly improving the efficiency of large language models during the decoding process.

Title: Increasing SAM Zero-Shot Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation

• Authors: Zekun Jiang, Dongjie Cheng, Ziyuan Qin, Jun Gao, Qicheng Lao, Kang Li, Le Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15759
• Pdf URL: https://arxiv.org/pdf/2402.15759
• Copy Paste: [[2402.15759]] Increasing SAM Zero-Shot Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation(https://arxiv.org/abs/2402.15759)
• Keywords: large language model, segmentation
• Abstract: This study develops and evaluates a novel multimodal medical image zero-shot segmentation algorithm named Text-Visual-Prompt SAM (TV-SAM) without any manual annotations. TV-SAM incorporates and integrates large language model GPT-4, Vision Language Model GLIP, and Segment Anything Model (SAM), to autonomously generate descriptive text prompts and visual bounding box prompts from medical images, thereby enhancing SAM for zero-shot segmentation. Comprehensive evaluations are implemented on seven public datasets encompassing eight imaging modalities to demonstrate that TV-SAM can effectively segment unseen targets across various modalities without additional training, significantly outperforming SAM AUTO and GSAM, closely matching the performance of SAM BBOX with gold standard bounding box prompts, and surpassing the state-of-the-art on specific datasets like ISIC and WBC. The study indicates that TV-SAM serves as an effective multimodal medical image zero-shot segmentation algorithm, highlighting the significant contribution of GPT-4 to zero-shot segmentation. By integrating foundational models such as GPT-4, GLIP, and SAM, it could enhance the capability to address complex problems in specialized domains. The code is available at: https://github.com/JZK00/TV-SAM.

Title: Res-VMamba: Fine-Grained Food Category Visual Classification Using Selective State Space Models with Deep Residual Learning

• Authors: Chi-Sheng Chen, Guan-Ying Chen, Dong Zhou, Di Jiang, Dai-Shi Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15761
• Pdf URL: https://arxiv.org/pdf/2402.15761
• Copy Paste: [[2402.15761]] Res-VMamba: Fine-Grained Food Category Visual Classification Using Selective State Space Models with Deep Residual Learning(https://arxiv.org/abs/2402.15761)
• Keywords: transformer
• Abstract: Food classification is the foundation for developing food vision tasks and plays a key role in the burgeoning field of computational nutrition. Due to the complexity of food requiring fine-grained classification, recent academic research mainly modifies Convolutional Neural Networks (CNNs) and/or Vision Transformers (ViTs) to perform food category classification. However, to learn fine-grained features, the CNN backbone needs additional structural design, whereas ViT, containing the self-attention module, has increased computational complexity. In recent months, a new Sequence State Space (S4) model, through a Selection mechanism and computation with a Scan (S6), colloquially termed Mamba, has demonstrated superior performance and computation efficiency compared to the Transformer architecture. The VMamba model, which incorporates the Mamba mechanism into image tasks (such as classification), currently establishes the state-of-the-art (SOTA) on the ImageNet dataset. In this research, we introduce an academically underestimated food dataset CNFOOD-241, and pioneer the integration of a residual learning framework within the VMamba model to concurrently harness both global and local state features inherent in the original VMamba architectural design. The research results show that VMamba surpasses current SOTA models in fine-grained and food classification. The proposed Res-VMamba further improves the classification accuracy to 79.54\% without pretrained weight. Our findings elucidate that our proposed methodology establishes a new benchmark for SOTA performance in food recognition on the CNFOOD-241 dataset. The code can be obtained on GitHub: https://github.com/ChiShengChen/ResVMamba.

Title: Look Before You Leap: Problem Elaboration Prompting Improves Mathematical Reasoning in Large Language Models

• Authors: Haoran Liao, Jidong Tian, Shaohua Hu, Hao He, Yaohui Jin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15764
• Pdf URL: https://arxiv.org/pdf/2402.15764
• Copy Paste: [[2402.15764]] Look Before You Leap: Problem Elaboration Prompting Improves Mathematical Reasoning in Large Language Models(https://arxiv.org/abs/2402.15764)
• Keywords: robust, large language model
• Abstract: Large language models~(LLMs) have exhibited impressive performance across NLP tasks. So far they still face challenges in complex reasoning tasks and can be sensitive to input context. Despite significant efforts have been invested in enhancing reasoning process and improving prefix-prompts robustness, the crucial role of problem context has been overlooked. In this study, we propose a new approach to improve the mathematical capacities of LLMs, named Problem Elaboration Prompting~(PEP). Specifically, PEP decomposes and elucidates the problem context before reasoning, thus enhancing the global context modeling and reducing the parsing difficulties. Experiments on datasets demonstrate promising performances on complex reasoning and indicate the beneficial impact for ill-formed problems. For instance, with the GPT-3.5 model~(\texttt{text-davinci-003}), we observed a 9.93\% improvement with greedy decoding and 8.80\% improvement with self-consistency on GSM8k compared to the standard CoT. With ChatGPT~(\texttt{turbo}) and PEP, we achieve SOTA performances on SVAMP with 86.2\% and GSM8k with 90.98\%.

Title: Cryptanalysis and improvement of multimodal data encryption by machine-learning-based system

• Authors: Zakaria Tolba
• Subjects: cs.CR, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.15779
• Pdf URL: https://arxiv.org/pdf/2402.15779
• Copy Paste: [[2402.15779]] Cryptanalysis and improvement of multimodal data encryption by machine-learning-based system(https://arxiv.org/abs/2402.15779)
• Keywords: secure, security, privacy, protect, attack
• Abstract: With the rising popularity of the internet and the widespread use of networks and information systems via the cloud and data centers, the privacy and security of individuals and organizations have become extremely crucial. In this perspective, encryption consolidates effective technologies that can effectively fulfill these requirements by protecting public information exchanges. To achieve these aims, the researchers used a wide assortment of encryption algorithms to accommodate the varied requirements of this field, as well as focusing on complex mathematical issues during their work to substantially complicate the encrypted communication mechanism. as much as possible to preserve personal information while significantly reducing the possibility of attacks. Depending on how complex and distinct the requirements established by these various applications are, the potential of trying to break them continues to occur, and systems for evaluating and verifying the cryptographic algorithms implemented continue to be necessary. The best approach to analyzing an encryption algorithm is to identify a practical and efficient technique to break it or to learn ways to detect and repair weak aspects in algorithms, which is known as cryptanalysis. Experts in cryptanalysis have discovered several methods for breaking the cipher, such as discovering a critical vulnerability in mathematical equations to derive the secret key or determining the plaintext from the ciphertext. There are various attacks against secure cryptographic algorithms in the literature, and the strategies and mathematical solutions widely employed empower cryptanalysts to demonstrate their findings, identify weaknesses, and diagnose maintenance failures in algorithms.

Title: Holding Secrets Accountable: Auditing Privacy-Preserving Machine Learning

• Authors: Hidde Lycklama, Alexander Viand, Nicolas Küchler, Christian Knabenhans, Anwar Hithnawi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.15780
• Pdf URL: https://arxiv.org/pdf/2402.15780
• Copy Paste: [[2402.15780]] Holding Secrets Accountable: Auditing Privacy-Preserving Machine Learning(https://arxiv.org/abs/2402.15780)
• Keywords: privacy
• Abstract: Recent advancements in privacy-preserving machine learning are paving the way to extend the benefits of ML to highly sensitive data that, until now, have been hard to utilize due to privacy concerns and regulatory constraints. Simultaneously, there is a growing emphasis on enhancing the transparency and accountability of machine learning, including the ability to audit ML deployments. While ML auditing and PPML have both been the subjects of intensive research, they have predominately been examined in isolation. However, their combination is becoming increasingly important. In this work, we introduce Arc, an MPC framework for auditing privacy-preserving machine learning. At the core of our framework is a new protocol for efficiently verifying MPC inputs against succinct commitments at scale. We evaluate the performance of our framework when instantiated with our consistency protocol and compare it to hashing-based and homomorphic-commitment-based approaches, demonstrating that it is up to 10^4x faster and up to 10^6x more concise.

Title: IRConStyle: Image Restoration Framework Using Contrastive Learning and Style Transfer

• Authors: Dongqi Fan, Xin Zhao, Liang Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15784
• Pdf URL: https://arxiv.org/pdf/2402.15784
• Copy Paste: [[2402.15784]] IRConStyle: Image Restoration Framework Using Contrastive Learning and Style Transfer(https://arxiv.org/abs/2402.15784)
• Keywords: transformer, segmentation
• Abstract: Recently, the contrastive learning paradigm has achieved remarkable success in high-level tasks such as classification, detection, and segmentation. However, contrastive learning applied in low-level tasks, like image restoration, is limited, and its effectiveness is uncertain. This raises a question: Why does the contrastive learning paradigm not yield satisfactory results in image restoration? In this paper, we conduct in-depth analyses and propose three guidelines to address the above question. In addition, inspired by style transfer and based on contrastive learning, we propose a novel module for image restoration called \textbf{ConStyle}, which can be efficiently integrated into any U-Net structure network. By leveraging the flexibility of ConStyle, we develop a \textbf{general restoration network} for image restoration. ConStyle and the general restoration network together form an image restoration framework, namely \textbf{IRConStyle}. To demonstrate the capability and compatibility of ConStyle, we replace the general restoration network with transformer-based, CNN-based, and MLP-based networks, respectively. We perform extensive experiments on various image restoration tasks, including denoising, deblurring, deraining, and dehazing. The results on 19 benchmarks demonstrate that ConStyle can be integrated with any U-Net-based network and significantly enhance performance. For instance, ConStyle NAFNet significantly outperforms the original NAFNet on SOTS outdoor (dehazing) and Rain100H (deraining) datasets, with PSNR improvements of 4.16 dB and 3.58 dB with 85% fewer parameters.

Title: Gait-Based Privacy Protection for Smart Wearable Devices

• Authors: Yu Su, Yongjiao Li, Zhu Cao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.15797
• Pdf URL: https://arxiv.org/pdf/2402.15797
• Copy Paste: [[2402.15797]] Gait-Based Privacy Protection for Smart Wearable Devices(https://arxiv.org/abs/2402.15797)
• Keywords: secure, security, privacy, protect, attack, biometric
• Abstract: Smart wearable devices (SWDs) collect and store sensitive daily information of many people. Its primary method of identification is still the password unlocking method. However, several studies have shown serious security flaws in that method, which makes the privacy and security concerns of SWDs particularly urgent. Gait identification is well suited for SWDs because its built-in sensors can provide data support for identification. However, existing gait identification methods have low accuracy and neglect to protect the privacy of gait features. In addition, the SWD can be used as an internet of things device for users to share data. But few studies have used gait feature-based encryption schemes to protect the privacy of message interactions between SWDs and other devices. In this paper, we propose a gait identification network, a bi-directional long short-term memory network with an attention mechanism (ABLSTM), to improve the identification accuracy and a stochastic orthogonal transformation (SOT) scheme to protect the extracted gait features from leakage. In the experiments, ABLSTM achieves an accuracy of 95.28%, reducing previous error rate by 19.3%. The SOT scheme is proved to be resistant to the chosen plaintext attack (CPA) and is 30% faster than previous methods. A biometric-based encryption scheme is proposed to enable secure message interactions using gait features as keys after the gait identification stage is passed, and offers better protection of the gait features compared to previous schemes.

Title: Optimal Zero-Shot Detector for Multi-Armed Attacks

• Authors: Federica Granese, Marco Romanelli, Pablo Piantanida
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.15808
• Pdf URL: https://arxiv.org/pdf/2402.15808
• Copy Paste: [[2402.15808]] Optimal Zero-Shot Detector for Multi-Armed Attacks(https://arxiv.org/abs/2402.15808)
• Keywords: protect, defense, attack
• Abstract: This paper explores a scenario in which a malicious actor employs a multi-armed attack strategy to manipulate data samples, offering them various avenues to introduce noise into the dataset. Our central objective is to protect the data by detecting any alterations to the input. We approach this defensive strategy with utmost caution, operating in an environment where the defender possesses significantly less information compared to the attacker. Specifically, the defender is unable to utilize any data samples for training a defense model or verifying the integrity of the channel. Instead, the defender relies exclusively on a set of pre-existing detectors readily available ``off the shelf''. To tackle this challenge, we derive an innovative information-theoretic defense approach that optimally aggregates the decisions made by these detectors, eliminating the need for any training data. We further explore a practical use-case scenario for empirical evaluation, where the attacker possesses a pre-trained classifier and launches well-known adversarial attacks against it. Our experiments highlight the effectiveness of our proposed solution, even in scenarios that deviate from the optimal setup.

Title: A Generative Machine Learning Model for Material Microstructure 3D Reconstruction and Performance Evaluation

• Authors: Yilin Zheng, Zhigong Song
• Subjects: cs.LG, cond-mat.mtrl-sci, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15815
• Pdf URL: https://arxiv.org/pdf/2402.15815
• Copy Paste: [[2402.15815]] A Generative Machine Learning Model for Material Microstructure 3D Reconstruction and Performance Evaluation(https://arxiv.org/abs/2402.15815)
• Keywords: generative
• Abstract: The reconstruction of 3D microstructures from 2D slices is considered to hold significant value in predicting the spatial structure and physical properties of materials.The dimensional extension from 2D to 3D is viewed as a highly challenging inverse problem from the current technological perspective.Recently,methods based on generative adversarial networks have garnered widespread attention.However,they are still hampered by numerous limitations,including oversimplified models,a requirement for a substantial number of training samples,and difficulties in achieving model convergence during training.In light of this,a novel generative model that integrates the multiscale properties of U-net with and the generative capabilities of GAN has been proposed.Based on this,the innovative construction of a multi-scale channel aggregation module,a multi-scale hierarchical feature aggregation module and a convolutional block attention mechanism can better capture the properties of the material microstructure and extract the image information.The model's accuracy is further improved by combining the image regularization loss with the Wasserstein distance loss.In addition,this study utilizes the anisotropy index to accurately distinguish the nature of the image,which can clearly determine the isotropy and anisotropy of the image.It is also the first time that the generation quality of material samples from different domains is evaluated and the performance of the model itself is compared.The experimental results demonstrate that the present model not only shows a very high similarity between the generated 3D structures and real samples but is also highly consistent with real data in terms of statistical data analysis.

Title: BETA-UAV: Blockchain-based Efficient Authentication for Secure UAV Communication

• Authors: Sana Hafeez, Mahmoud A. Shawky, Mohammad Al-Quraan, Lina Mohjazi, Muhammad Ali Imran, Yao Sun
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.15817
• Pdf URL: https://arxiv.org/pdf/2402.15817
• Copy Paste: [[2402.15817]] BETA-UAV: Blockchain-based Efficient Authentication for Secure UAV Communication(https://arxiv.org/abs/2402.15817)
• Keywords: secure, security, privacy, attack
• Abstract: Unmanned aerial vehicles (UAV), an emerging architecture that embodies flying ad-hoc networks, face critical privacy and security challenges, mainly when engaged in data-sensitive missions. Therefore, message authentication is a crucial security feature in drone communications. This paper presents a Blockchain-based Efficient, and Trusted Authentication scheme for UAV communication, BETA-UAV, which exploits the inherent properties of blockchain technology concerning memorability and is immutable to record communication sessions via transactions using a smart contract. The smart contract in BETA-UAV allows participants to publish and call transactions from the blockchain network. Furthermore, transaction addresses are proof of freshness and trustworthiness for subsequent transmissions. Furthermore, we investigated their ability to resist active attacks, such as impersonation, replaying, and modification. In addition, we evaluate the gas costs associated with the functions of the smart contract by implementing a BETA-UAV on the Ethereum public blockchain. A comparison of the computation and communication overheads shows that the proposed approach can save significant costs over traditional techniques.

Title: Linguistic Intelligence in Large Language Models for Telecommunications

• Authors: Tasnim Ahmed, Nicola Piovesan, Antonio De Domenico, Salimur Choudhury
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15818
• Pdf URL: https://arxiv.org/pdf/2402.15818
• Copy Paste: [[2402.15818]] Linguistic Intelligence in Large Language Models for Telecommunications(https://arxiv.org/abs/2402.15818)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have emerged as a significant advancement in the field of Natural Language Processing (NLP), demonstrating remarkable capabilities in language generation and other language-centric tasks. Despite their evaluation across a multitude of analytical and reasoning tasks in various scientific domains, a comprehensive exploration of their knowledge and understanding within the realm of natural language tasks in the telecommunications domain is still needed. This study, therefore, seeks to evaluate the knowledge and understanding capabilities of LLMs within this domain. To achieve this, we conduct an exhaustive zero-shot evaluation of four prominent LLMs-Llama-2, Falcon, Mistral, and Zephyr. These models require fewer resources than ChatGPT, making them suitable for resource-constrained environments. Their performance is compared with state-of-the-art, fine-tuned models. To the best of our knowledge, this is the first work to extensively evaluate and compare the understanding of LLMs across multiple language-centric tasks in this domain. Our evaluation reveals that zero-shot LLMs can achieve performance levels comparable to the current state-of-the-art fine-tuned models. This indicates that pretraining on extensive text corpora equips LLMs with a degree of specialization, even within the telecommunications domain. We also observe that no single LLM consistently outperforms others, and the performance of different LLMs can fluctuate. Although their performance lags behind fine-tuned models, our findings underscore the potential of LLMs as a valuable resource for understanding various aspects of this field that lack large annotated data.

Title: Parameter-efficient Prompt Learning for 3D Point Cloud Understanding

• Authors: Hongyu Sun, Yongcai Wang, Wang Chen, Haoran Deng, Deying Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15823
• Pdf URL: https://arxiv.org/pdf/2402.15823
• Copy Paste: [[2402.15823]] Parameter-efficient Prompt Learning for 3D Point Cloud Understanding(https://arxiv.org/abs/2402.15823)
• Keywords: segmentation
• Abstract: This paper presents a parameter-efficient prompt tuning method, named PPT, to adapt a large multi-modal model for 3D point cloud understanding. Existing strategies are quite expensive in computation and storage, and depend on time-consuming prompt engineering. We address the problems from three aspects. Firstly, a PromptLearner module is devised to replace hand-crafted prompts with learnable contexts to automate the prompt tuning process. Then, we lock the pre-trained backbone instead of adopting the full fine-tuning paradigm to substantially improve the parameter efficiency. Finally, a lightweight PointAdapter module is arranged near target tasks to enhance prompt tuning for 3D point cloud understanding. Comprehensive experiments are conducted to demonstrate the superior parameter and data efficiency of the proposed method.Meanwhile, we obtain new records on 4 public datasets and multiple 3D tasks, i.e., point cloud recognition, few-shot learning, and part segmentation. The implementation is available at https://github.com/auniquesun/PPT.

Title: A New Secure Memory System for Efficient Data Protection and Access Pattern Obfuscation

• Authors: Haoran Geng, Yuezhi Che, Aaron Dingler, Michael Niemier, Xiaobo Sharon Hu
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2402.15824
• Pdf URL: https://arxiv.org/pdf/2402.15824
• Copy Paste: [[2402.15824]] A New Secure Memory System for Efficient Data Protection and Access Pattern Obfuscation(https://arxiv.org/abs/2402.15824)
• Keywords: secure, security, protect
• Abstract: As the reliance on secure memory environments permeates across applications, memory encryption is used to ensure memory security. However, most effective encryption schemes, such as the widely used AES-CTR, inherently introduce extra overheads, including those associated with counter storage and version number integrity checks. Moreover, encryption only protects data content, and it does not fully address the memory access pattern leakage. While Oblivious RAM (ORAM) aims to obscure these patterns, its high performance costs hinder practical applications. We introduce Secure Scattered Memory (SSM), an efficient scheme provides a comprehensive security solution that preserves the confidentiality of data content without traditional encryption, protects access patterns, and enables efficient integrity verification. Moving away from traditional encryption-centric methods, SSM offers a fresh approach to protecting data content while eliminating counter-induced overheads. Moreover, SSM is designed to inherently obscure memory access patterns, thereby significantly enhancing the confidentiality of memory data. In addition, SSM incorporates lightweight, thus integrated mechanisms for integrity assurance, protecting against data tampering. We also introduce SSM+, an extension that adapts Path ORAM to offer even greater security guarantees for both data content and memory access patterns, demonstrating its flexibility and efficiency. Experimental results show that SSM incurs only a 10% performance overhead compared to non-protected memory and offers a 15% improvement over AES-CTR mode memory protection. Notably, SSM+ provides an 20% improvement against Path ORAM integrated with Intel SGX under the highest security guarantees.

Title: Multiple Instance Learning for Glioma Diagnosis using Hematoxylin and Eosin Whole Slide Images: An Indian cohort Study

• Authors: Ekansh Chauhan, Amit Sharma, Megha S Uppin, C.V. Jawahar, Vinod P.K
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15832
• Pdf URL: https://arxiv.org/pdf/2402.15832
• Copy Paste: [[2402.15832]] Multiple Instance Learning for Glioma Diagnosis using Hematoxylin and Eosin Whole Slide Images: An Indian cohort Study(https://arxiv.org/abs/2402.15832)
• Keywords: extraction
• Abstract: Brain tumors represent a severe and life-threatening condition, demanding precise diagnosis and tailored treatment strategies. This study advances patient care with findings from rigorous multiple-instance-learning experimentations across various feature extractors and aggregators in brain tumor histopathology. It establishes new performance benchmarks in glioma subtype classification across multiple datasets, including a novel dataset focused on the Indian demographic (IPD-Brain), providing a valuable resource for existing research. Using a ResNet-50, pretrained on histopathology datasets, for feature extraction, combined with DTFD feature aggregator, our approach achieves state-of-the-art AUCs of 88.08 on IPD-Brain and 95.81 on TCGA-Brain dataset respectively for three-way glioma subtype classification. Moreover, it establishes new benchmarks in grading and detecting IHC molecular biomarkers (IDH1 (mutant R132H), TP53, ATRX, Ki-67) through H&E stained whole slide images for the IPD-Brain dataset. The work also highlights a significant correlation between the model decision-making processes and the diagnostic reasoning of pathologists, underscoring its capability to mimic professional diagnostic procedures.

Title: Prompt Perturbation Consistency Learning for Robust Language Models

• Authors: Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, Aram Galstyan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.15833
• Pdf URL: https://arxiv.org/pdf/2402.15833
• Copy Paste: [[2402.15833]] Prompt Perturbation Consistency Learning for Robust Language Models(https://arxiv.org/abs/2402.15833)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have demonstrated impressive performance on a number of natural language processing tasks, such as question answering and text summarization. However, their performance on sequence labeling tasks such as intent classification and slot filling (IC-SF), which is a central component in personal assistant systems, lags significantly behind discriminative models. Furthermore, there is a lack of substantive research on the robustness of LLMs to various perturbations in the input prompts. The contributions of this paper are three-fold. First, we show that fine-tuning sufficiently large LLMs can produce IC-SF performance comparable to discriminative models. Next, we systematically analyze the performance deterioration of those fine-tuned models due to three distinct yet relevant types of input perturbations - oronyms, synonyms, and paraphrasing. Finally, we propose an efficient mitigation approach, Prompt Perturbation Consistency Learning (PPCL), which works by regularizing the divergence between losses from clean and perturbed samples. Our experiments demonstrate that PPCL can recover on average 59% and 69% of the performance drop for IC and SF tasks, respectively. Furthermore, PPCL beats the data augmentation approach while using ten times fewer augmented data samples.

Title: RAUCA: A Novel Physical Adversarial Attack on Vehicle Detectors via Robust and Accurate Camouflage Generation

• Authors: Jiawei Zhou, Linye Lyu, Daojing He, Yu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15853
• Pdf URL: https://arxiv.org/pdf/2402.15853
• Copy Paste: [[2402.15853]] RAUCA: A Novel Physical Adversarial Attack on Vehicle Detectors via Robust and Accurate Camouflage Generation(https://arxiv.org/abs/2402.15853)
• Keywords: attack, robust
• Abstract: Adversarial camouflage is a widely used physical attack against vehicle detectors for its superiority in multi-view attack performance. One promising approach involves using differentiable neural renderers to facilitate adversarial camouflage optimization through gradient back-propagation. However, existing methods often struggle to capture environmental characteristics during the rendering process or produce adversarial textures that can precisely map to the target vehicle, resulting in suboptimal attack performance. Moreover, these approaches neglect diverse weather conditions, reducing the efficacy of generated camouflage across varying weather scenarios. To tackle these challenges, we propose a robust and accurate camouflage generation method, namely RAUCA. The core of RAUCA is a novel neural rendering component, Neural Renderer Plus (NRP), which can accurately project vehicle textures and render images with environmental characteristics such as lighting and weather. In addition, we integrate a multi-weather dataset for camouflage generation, leveraging the NRP to enhance the attack robustness. Experimental results on six popular object detectors show that RAUCA consistently outperforms existing methods in both simulation and real-world settings.

Title: FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology

• Authors: Yuanzhe Peng, Jieming Bian, Jie Xu
• Subjects: cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.15858
• Pdf URL: https://arxiv.org/pdf/2402.15858
• Copy Paste: [[2402.15858]] FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology(https://arxiv.org/abs/2402.15858)
• Keywords: privacy, federate
• Abstract: The fusion of complementary multimodal information is crucial in computational pathology for accurate diagnostics. However, existing multimodal learning approaches necessitate access to users' raw data, posing substantial privacy risks. While Federated Learning (FL) serves as a privacy-preserving alternative, it falls short in addressing the challenges posed by heterogeneous (yet possibly overlapped) modalities data across various hospitals. To bridge this gap, we propose a Federated Multi-Modal (FedMM) learning framework that federatedly trains multiple single-modal feature extractors to enhance subsequent classification performance instead of existing FL that aims to train a unified multimodal fusion model. Any participating hospital, even with small-scale datasets or limited devices, can leverage these federated trained extractors to perform local downstream tasks (e.g., classification) while ensuring data privacy. Through comprehensive evaluations of two publicly available datasets, we demonstrate that FedMM notably outperforms two baselines in accuracy and AUC metrics.

Title: SportQA: A Benchmark for Sports Understanding in Large Language Models

• Authors: Haotian Xia, Zhengbang Yang, Yuqing Wang, Rhys Tracy, Yun Zhao, Dongdong Huang, Zezhi Chen, Yan Zhu, Yuan-fang Wang, Weining Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15862
• Pdf URL: https://arxiv.org/pdf/2402.15862
• Copy Paste: [[2402.15862]] SportQA: A Benchmark for Sports Understanding in Large Language Models(https://arxiv.org/abs/2402.15862)
• Keywords: large language model
• Abstract: A deep understanding of sports, a field rich in strategic and dynamic content, is crucial for advancing Natural Language Processing (NLP). This holds particular significance in the context of evaluating and advancing Large Language Models (LLMs), given the existing gap in specialized benchmarks. To bridge this gap, we introduce SportQA, a novel benchmark specifically designed for evaluating LLMs in the context of sports understanding. SportQA encompasses over 70,000 multiple-choice questions across three distinct difficulty levels, each targeting different aspects of sports knowledge from basic historical facts to intricate, scenario-based reasoning tasks. We conducted a thorough evaluation of prevalent LLMs, mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting. Our results reveal that while LLMs exhibit competent performance in basic sports knowledge, they struggle with more complex, scenario-based sports reasoning, lagging behind human expertise. The introduction of SportQA marks a significant step forward in NLP, offering a tool for assessing and enhancing sports understanding in LLMs.

Title: Field-based Molecule Generation

• Authors: Alexandru Dumitrescu, Dani Korpela, Markus Heinonen, Yogesh Verma, Valerii Iakovlev, Vikas Garg, Harri Lähdesmäki
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2402.15864
• Pdf URL: https://arxiv.org/pdf/2402.15864
• Copy Paste: [[2402.15864]] Field-based Molecule Generation(https://arxiv.org/abs/2402.15864)
• Keywords: generative
• Abstract: This work introduces FMG, a field-based model for drug-like molecule generation. We show how the flexibility of this method provides crucial advantages over the prevalent, point-cloud based methods, and achieves competitive molecular stability generation. We tackle optical isomerism (enantiomers), a previously omitted molecular property that is crucial for drug safety and effectiveness, and thus account for all molecular geometry aspects. We demonstrate how previous methods are invariant to a group of transformations that includes enantiomer pairs, leading them invariant to the molecular R and S configurations, while our field-based generative model captures this property.

Title: HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models

• Authors: Li Pang, Xiangyu Rui, Long Cui, Hongzhong Wang, Deyu Meng, Xiangyong Cao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.15865
• Pdf URL: https://arxiv.org/pdf/2402.15865
• Copy Paste: [[2402.15865]] HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models(https://arxiv.org/abs/2402.15865)
• Keywords: diffusion
• Abstract: Hyperspectral image (HSI) restoration aims at recovering clean images from degraded observations and plays a vital role in downstream tasks. Existing model-based methods have limitations in accurately modeling the complex image characteristics with handcraft priors, and deep learning-based methods suffer from poor generalization ability. To alleviate these issues, this paper proposes an unsupervised HSI restoration framework with pre-trained diffusion model (HIR-Diff), which restores the clean HSIs from the product of two low-rank components, i.e., the reduced image and the coefficient matrix. Specifically, the reduced image, which has a low spectral dimension, lies in the image field and can be inferred from our improved diffusion model where a new guidance function with total variation (TV) prior is designed to ensure that the reduced image can be well sampled. The coefficient matrix can be effectively pre-estimated based on singular value decomposition (SVD) and rank-revealing QR (RRQR) factorization. Furthermore, a novel exponential noise schedule is proposed to accelerate the restoration process (about 5$\times$ acceleration for denoising) with little performance decrease. Extensive experimental results validate the superiority of our method in both performance and speed on a variety of HSI restoration tasks, including HSI denoising, noisy HSI super-resolution, and noisy HSI inpainting. The code is available at https://github.com/LiPang/HIRDiff.

Title: SemEval-2024 Task 8: Weighted Layer Averaging RoBERTa for Black-Box Machine-Generated Text Detection

• Authors: Ayan Datta, Aryan Chandramania, Radhika Mamidi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15873
• Pdf URL: https://arxiv.org/pdf/2402.15873
• Copy Paste: [[2402.15873]] SemEval-2024 Task 8: Weighted Layer Averaging RoBERTa for Black-Box Machine-Generated Text Detection(https://arxiv.org/abs/2402.15873)
• Keywords: large language model
• Abstract: This document contains the details of the authors' submission to the proceedings of SemEval 2024's Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection Subtask A (monolingual) and B. Detection of machine-generated text is becoming an increasingly important task, with the advent of large language models (LLMs). In this document, we lay out the techniques utilized for performing the same, along with the results obtained.

Title: Multimodal Instruction Tuning with Conditional Mixture of LoRA

• Authors: Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, Lifu Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15896
• Pdf URL: https://arxiv.org/pdf/2402.15896
• Copy Paste: [[2402.15896]] Multimodal Instruction Tuning with Conditional Mixture of LoRA(https://arxiv.org/abs/2402.15896)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in diverse tasks across different domains, with an increasing focus on improving their zero-shot generalization capabilities for unseen multimodal tasks. Multimodal instruction tuning has emerged as a successful strategy for achieving zero-shot generalization by fine-tuning pre-trained models on diverse multimodal tasks through instructions. As MLLMs grow in complexity and size, the need for parameter-efficient fine-tuning methods like Low-Rank Adaption (LoRA), which fine-tunes with a minimal set of parameters, becomes essential. However, applying LoRA in multimodal instruction tuning presents the challenge of task interference, which leads to performance degradation, especially when dealing with a broad array of multimodal tasks. To address this, this paper introduces a novel approach that integrates multimodal instruction tuning with Conditional Mixture-of-LoRA (MixLoRA). It innovates upon LoRA by dynamically constructing low-rank adaptation matrices tailored to the unique demands of each input instance, aiming to mitigate task interference. Experimental results on various multimodal evaluation datasets indicate that MixLoRA not only outperforms the conventional LoRA with the same or even higher ranks, demonstrating its efficacy and adaptability in diverse multimodal tasks.

Title: ESFL: Efficient Split Federated Learning over Resource-Constrained Heterogeneous Wireless Devices

• Authors: Guangyu Zhu, Yiqin Deng, Xianhao Chen, Haixia Zhang, Yuguang Fang, Tan F. Wong
• Subjects: cs.LG, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.15903
• Pdf URL: https://arxiv.org/pdf/2402.15903
• Copy Paste: [[2402.15903]] ESFL: Efficient Split Federated Learning over Resource-Constrained Heterogeneous Wireless Devices(https://arxiv.org/abs/2402.15903)
• Keywords: federate
• Abstract: Federated learning (FL) allows multiple parties (distributed devices) to train a machine learning model without sharing raw data. How to effectively and efficiently utilize the resources on devices and the central server is a highly interesting yet challenging problem. In this paper, we propose an efficient split federated learning algorithm (ESFL) to take full advantage of the powerful computing capabilities at a central server under a split federated learning framework with heterogeneous end devices (EDs). By splitting the model into different submodels between the server and EDs, our approach jointly optimizes user-side workload and server-side computing resource allocation by considering users' heterogeneity. We formulate the whole optimization problem as a mixed-integer non-linear program, which is an NP-hard problem, and develop an iterative approach to obtain an approximate solution efficiently. Extensive simulations have been conducted to validate the significantly increased efficiency of our ESFL approach compared with standard federated learning, split learning, and splitfed learning.

Title: Enhanced Droplet Analysis Using Generative Adversarial Networks

• Authors: Tan-Hanh Pham, Kim-Doang Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15909
• Pdf URL: https://arxiv.org/pdf/2402.15909
• Copy Paste: [[2402.15909]] Enhanced Droplet Analysis Using Generative Adversarial Networks(https://arxiv.org/abs/2402.15909)
• Keywords: generative
• Abstract: Precision devices play an important role in enhancing production quality and productivity in agricultural systems. Therefore, the optimization of these devices is essential in precision agriculture. Recently, with the advancements of deep learning, there have been several studies aiming to harness its capabilities for improving spray system performance. However, the effectiveness of these methods heavily depends on the size of the training dataset, which is expensive and time-consuming to collect. To address the challenge of insufficient training samples, this paper proposes an alternative solution by generating artificial images of droplets using generative adversarial networks (GAN). The GAN model is trained by using a small dataset captured by a high-speed camera and capable of generating images with progressively increasing resolution. The results demonstrate that the model can generate high-quality images with the size of $1024\times1024$. Furthermore, this research leverages recent advancements in computer vision and deep learning to develop a light droplet detector using the synthetic dataset. As a result, the detection model achieves a 16.06\% increase in mean average precision (mAP) when utilizing the synthetic dataset. To the best of our knowledge, this work stands as the first to employ a generative model for augmenting droplet detection. Its significance lies not only in optimizing nozzle design for constructing efficient spray systems but also in addressing the common challenge of insufficient data in various precision agriculture tasks. This work offers a critical contribution to conserving resources while striving for optimal and sustainable agricultural practices.

Title: PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails

• Authors: Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, Atul Prakash
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.15911
• Pdf URL: https://arxiv.org/pdf/2402.15911
• Copy Paste: [[2402.15911]] PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails(https://arxiv.org/abs/2402.15911)
• Keywords: defense, attack, large language model
• Abstract: Large language models (LLMs) are typically aligned to be harmless to humans. Unfortunately, recent work has shown that such models are susceptible to automated jailbreak attacks that induce them to generate harmful content. More recent LLMs often incorporate an additional layer of defense, a Guard Model, which is a second LLM that is designed to check and moderate the output response of the primary LLM. Our key contribution is to show a novel attack strategy, PRP, that is successful against several open-source (e.g., Llama 2) and closed-source (e.g., GPT 3.5) implementations of Guard Models. PRP leverages a two step prefix-based attack that operates by (a) constructing a universal adversarial prefix for the Guard Model, and (b) propagating this prefix to the response. We find that this procedure is effective across multiple threat models, including ones in which the adversary has no access to the Guard Model at all. Our work suggests that further advances are required on defenses and Guard Models before they can be considered effective.

Title: Predicting Outcomes in Video Games with Long Short Term Memory Networks

• Authors: Kittimate Chulajata, Sean Wu, Fabien Scalzo, Eun Sang Cha
• Subjects: cs.LG, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.15923
• Pdf URL: https://arxiv.org/pdf/2402.15923
• Copy Paste: [[2402.15923]] Predicting Outcomes in Video Games with Long Short Term Memory Networks(https://arxiv.org/abs/2402.15923)
• Keywords: transformer, large language model
• Abstract: Forecasting winners in E-sports with real-time analytics has the potential to further engage audiences watching major tournament events. However, making such real-time predictions is challenging due to unpredictable variables within the game involving diverse player strategies and decision-making. Our work attempts to enhance audience engagement within video game tournaments by introducing a real-time method of predicting wins. Our Long Short Term Memory Network (LSTMs) based approach enables efficient predictions of win-lose outcomes by only using the health indicator of each player as a time series. As a proof of concept, we evaluate our model's performance within a classic, two-player arcade game, Super Street Fighter II Turbo. We also benchmark our method against state of the art methods for time series forecasting; i.e. Transformer models found in large language models (LLMs). Finally, we open-source our data set and code in hopes of furthering work in predictive analysis for arcade games.

Title: Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA

• Authors: Wentao Mo, Yang Liu
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.15933
• Pdf URL: https://arxiv.org/pdf/2402.15933
• Copy Paste: [[2402.15933]] Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA(https://arxiv.org/abs/2402.15933)
• Keywords: transformer
• Abstract: In 3D Visual Question Answering (3D VQA), the scarcity of fully annotated data and limited visual content diversity hampers the generalization to novel scenes and 3D concepts (e.g., only around 800 scenes are utilized in ScanQA and SQA dataset). Current approaches resort supplement 3D reasoning with 2D information. However, these methods face challenges: either they use top-down 2D views that introduce overly complex and sometimes question-irrelevant visual clues, or they rely on globally aggregated scene/image-level representations from 2D VLMs, losing the fine-grained vision-language correlations. To overcome these limitations, our approach utilizes question-conditional 2D view selection procedure, pinpointing semantically relevant 2D inputs for crucial visual clues. We then integrate this 2D knowledge into the 3D-VQA system via a two-branch Transformer structure. This structure, featuring a Twin-Transformer design, compactly combines 2D and 3D modalities and captures fine-grained correlations between modalities, allowing them mutually augmenting each other. Integrating proposed mechanisms above, we present BridgeQA, that offers a fresh perspective on multi-modal transformer-based architectures for 3D-VQA. Experiments validate that BridgeQA achieves state-of-the-art on 3D-VQA datasets and significantly outperforms existing solutions. Code is available at $\href{https://github.com/matthewdm0816/BridgeQA}{\text{this URL}}$.

Title: Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models

• Authors: Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Ge Li
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.15938
• Pdf URL: https://arxiv.org/pdf/2402.15938
• Copy Paste: [[2402.15938]] Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models(https://arxiv.org/abs/2402.15938)
• Keywords: large language model
• Abstract: Recent statements about the impressive capabilities of large language models (LLMs) are usually supported by evaluating on open-access benchmarks. Considering the vast size and wide-ranging sources of LLMs' training data, it could explicitly or implicitly include test data, leading to LLMs being more susceptible to data contamination. However, due to the opacity of training data, the black-box access of models, and the rapid growth of synthetic training data, detecting and mitigating data contamination for LLMs faces significant challenges. In this paper, we propose CDD, which stands for Contamination Detection via output Distribution for LLMs. CDD necessitates only the sampled texts to detect data contamination, by identifying the peakedness of LLM's output distribution. To mitigate the impact of data contamination in evaluation, we also present TED: Trustworthy Evaluation via output Distribution, based on the correction of LLM's output distribution. To facilitate this study, we introduce two benchmarks, i.e., DetCon and ComiEval, for data contamination detection and contamination mitigation evaluation tasks. Extensive experimental results show that CDD achieves the average relative improvements of 21.8\%-30.2\% over other contamination detection approaches in terms of Accuracy, F1 Score, and AUC metrics, and can effectively detect contamination caused by the variants of test data. TED significantly mitigates performance improvements up to 66.9\% attributed to data contamination across 24 settings and 21 contamination degrees. In real-world applications, we reveal that ChatGPT exhibits a high potential to suffer from data contamination on HumanEval benchmark.

Title: Attention-GAN for Anomaly Detection: A Cutting-Edge Approach to Cybersecurity Threat Management

• Authors: Mohammed Abo Sen
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15945
• Pdf URL: https://arxiv.org/pdf/2402.15945
• Copy Paste: [[2402.15945]] Attention-GAN for Anomaly Detection: A Cutting-Edge Approach to Cybersecurity Threat Management(https://arxiv.org/abs/2402.15945)
• Keywords: security, defense, attack, generative
• Abstract: This paper proposes an innovative Attention-GAN framework for enhancing cybersecurity, focusing on anomaly detection. In response to the challenges posed by the constantly evolving nature of cyber threats, the proposed approach aims to generate diverse and realistic synthetic attack scenarios, thereby enriching the dataset and improving threat identification. Integrating attention mechanisms with Generative Adversarial Networks (GANs) is a key feature of the proposed method. The attention mechanism enhances the model's ability to focus on relevant features, essential for detecting subtle and complex attack patterns. In addition, GANs address the issue of data scarcity by generating additional varied attack data, encompassing known and emerging threats. This dual approach ensures that the system remains relevant and effective against the continuously evolving cyberattacks. The KDD Cup and CICIDS2017 datasets were used to validate this model, which exhibited significant improvements in anomaly detection. It achieved an accuracy of 99.69% on the KDD dataset and 97.93% on the CICIDS2017 dataset, with precision, recall, and F1-scores above 97%, demonstrating its effectiveness in recognizing complex attack patterns. This study contributes significantly to cybersecurity by providing a scalable and adaptable solution for anomaly detection in the face of sophisticated and dynamic cyber threats. The exploration of GANs for data augmentation highlights a promising direction for future research, particularly in situations where data limitations restrict the development of cybersecurity systems. The attention-GAN framework has emerged as a pioneering approach, setting a new benchmark for advanced cyber-defense strategies.

Title: GreenLLaMA: A Framework for Detoxification with Explanations

• Authors: Md Tawkat Islam Khondaker, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan
• Subjects: cs.LG, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.15951
• Pdf URL: https://arxiv.org/pdf/2402.15951
• Copy Paste: [[2402.15951]] GreenLLaMA: A Framework for Detoxification with Explanations(https://arxiv.org/abs/2402.15951)
• Keywords: robust
• Abstract: Prior works on detoxification are scattered in the sense that they do not cover all aspects of detoxification needed in a real-world scenario. Notably, prior works restrict the task of developing detoxification models to only a seen subset of platforms, leaving the question of how the models would perform on unseen platforms unexplored. Additionally, these works do not address non-detoxifiability, a phenomenon whereby the toxic text cannot be detoxified without altering the meaning. We propose GreenLLaMA, the first comprehensive end-to-end detoxification framework, which attempts to alleviate the aforementioned limitations. We first introduce a cross-platform pseudo-parallel corpus applying multi-step data processing and generation strategies leveraging ChatGPT. We then train a suite of detoxification models with our cross-platform corpus. We show that our detoxification models outperform the SoTA model trained with human-annotated parallel corpus. We further introduce explanation to promote transparency and trustworthiness. GreenLLaMA additionally offers a unique paraphrase detector especially dedicated for the detoxification task to tackle the non-detoxifiable cases. Through experimental analysis, we demonstrate the effectiveness of our cross-platform corpus and the robustness of GreenLLaMA against adversarial toxicity.

Title: ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysis

• Authors: Yuchen He, Zeqing Yuan, Yihong Wu, Liqi Cheng, Dazhen Deng, Yingcai Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15952
• Pdf URL: https://arxiv.org/pdf/2402.15952
• Copy Paste: [[2402.15952]] ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysis(https://arxiv.org/abs/2402.15952)
• Keywords: segmentation
• Abstract: The immense popularity of racket sports has fueled substantial demand in tactical analysis with broadcast videos. However, existing manual methods require laborious annotation, and recent attempts leveraging video perception models are limited to low-level annotations like ball trajectories, overlooking tactics that necessitate an understanding of stroke techniques. State-of-the-art action segmentation models also struggle with technique recognition due to frequent occlusions and motion-induced blurring in racket sports videos. To address these challenges, We propose ViSTec, a Video-based Sports Technique recognition model inspired by human cognition that synergizes sparse visual data with rich contextual insights. Our approach integrates a graph to explicitly model strategic knowledge in stroke sequences and enhance technique recognition with contextual inductive bias. A two-stage action perception model is jointly trained to align with the contextual knowledge in the graph. Experiments demonstrate that our method outperforms existing models by a significant margin. Case studies with experts from the Chinese national table tennis team validate our model's capacity to automate analysis for technical actions and tactical strategies. More details are available at: https://ViSTec2024.github.io/.

Title: Towards Robust Image Stitching: An Adaptive Resistance Learning against Compatible Attacks

• Authors: Zhiying Jiang, Xingyuan Li, Jinyuan Liu, Xin Fan, Risheng Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.15959
• Pdf URL: https://arxiv.org/pdf/2402.15959
• Copy Paste: [[2402.15959]] Towards Robust Image Stitching: An Adaptive Resistance Learning against Compatible Attacks(https://arxiv.org/abs/2402.15959)
• Keywords: attack, robust
• Abstract: Image stitching seamlessly integrates images captured from varying perspectives into a single wide field-of-view image. Such integration not only broadens the captured scene but also augments holistic perception in computer vision applications. Given a pair of captured images, subtle perturbations and distortions which go unnoticed by the human visual system tend to attack the correspondence matching, impairing the performance of image stitching algorithms. In light of this challenge, this paper presents the first attempt to improve the robustness of image stitching against adversarial attacks. Specifically, we introduce a stitching-oriented attack~(SoA), tailored to amplify the alignment loss within overlapping regions, thereby targeting the feature matching procedure. To establish an attack resistant model, we delve into the robustness of stitching architecture and develop an adaptive adversarial training~(AAT) to balance attack resistance with stitching precision. In this way, we relieve the gap between the routine adversarial training and benign models, ensuring resilience without quality compromise. Comprehensive evaluation across real-world and synthetic datasets validate the deterioration of SoA on stitching performance. Furthermore, AAT emerges as a more robust solution against adversarial perturbations, delivering superior stitching results. Code is available at:https://github.com/Jzy2017/TRIS.

Title: Direct Punjabi to English speech translation using discrete units

• Authors: Prabhjot Kaur, L. Andrew M. Bush, Weisong Shi
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.15967
• Pdf URL: https://arxiv.org/pdf/2402.15967
• Copy Paste: [[2402.15967]] Direct Punjabi to English speech translation using discrete units(https://arxiv.org/abs/2402.15967)
• Keywords: transformer
• Abstract: Speech-to-speech translation is yet to reach the same level of coverage as text-to-text translation systems. The current speech technology is highly limited in its coverage of over 7000 languages spoken worldwide, leaving more than half of the population deprived of such technology and shared experiences. With voice-assisted technology (such as social robots and speech-to-text apps) and auditory content (such as podcasts and lectures) on the rise, ensuring that the technology is available for all is more important than ever. Speech translation can play a vital role in mitigating technological disparity and creating a more inclusive society. With a motive to contribute towards speech translation research for low-resource languages, our work presents a direct speech-to-speech translation model for one of the Indic languages called Punjabi to English. Additionally, we explore the performance of using a discrete representation of speech called discrete acoustic units as input to the Transformer-based translation model. The model, abbreviated as Unit-to-Unit Translation (U2UT), takes a sequence of discrete units of the source language (the language being translated from) and outputs a sequence of discrete units of the target language (the language being translated to). Our results show that the U2UT model performs better than the Speech-to-Unit Translation (S2UT) model by a 3.69 BLEU score.

Title: CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous models

• Authors: Abhishek Singh, Gauri Gupta, Ritvik Kapila, Yichuan Shi, Alex Dang, Sheshank Shankar, Mohammed Ehab, Ramesh Raskar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15968
• Pdf URL: https://arxiv.org/pdf/2402.15968
• Copy Paste: [[2402.15968]] CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous models(https://arxiv.org/abs/2402.15968)
• Keywords: secure, privacy, federate
• Abstract: Federated Learning (FL) enables collaborative optimization of machine learning models across decentralized data by aggregating model parameters. Our approach extends this concept by aggregating "knowledge" derived from models, instead of model parameters. We present a novel framework called \codream, where clients collaboratively optimize randomly initialized data using federated optimization in the input data space, similar to how randomly initialized model parameters are optimized in FL. Our key insight is that jointly optimizing this data can effectively capture the properties of the global data distribution. Sharing knowledge in data space offers numerous benefits: (1) model-agnostic collaborative learning, i.e., different clients can have different model architectures; (2) communication that is independent of the model size, eliminating scalability concerns with model parameters; (3) compatibility with secure aggregation, thus preserving the privacy benefits of federated learning; (4) allowing of adaptive optimization of knowledge shared for personalized learning. We empirically validate \codream on standard FL tasks, demonstrating competitive performance despite not sharing model parameters. Our code: https://mitmedialab.github.io/codream.github.io/

Title: Likelihood-based Mitigation of Evaluation Bias in Large Language Models

• Authors: Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, Naoaki Okazaki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.15987
• Pdf URL: https://arxiv.org/pdf/2402.15987
• Copy Paste: [[2402.15987]] Likelihood-based Mitigation of Evaluation Bias in Large Language Models(https://arxiv.org/abs/2402.15987)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics. However, the likelihood, a measure of LLM's plausibility for a sentence, can vary due to superficial differences in sentences, such as word order and sentence structure. It is therefore possible that there might be a likelihood bias if LLMs are used for evaluation: they might overrate sentences with higher likelihoods while underrating those with lower likelihoods. In this paper, we investigate the presence and impact of likelihood bias in LLM-based evaluators. We also propose a method to mitigate the likelihood bias. Our method utilizes highly biased instances as few-shot examples for in-context learning. Our experiments in evaluating the data-to-text and grammatical error correction tasks reveal that several LLMs we test display a likelihood bias. Furthermore, our proposed method successfully mitigates this bias, also improving evaluation performance (in terms of correlation of models with human scores) significantly.

Title: Cross-Resolution Land Cover Classification Using Outdated Products and Transformers

• Authors: Huan Ni, Yubin Zhao, Haiyan Guan, Cheng Jiang, Yongshi Jie, Xing Wang, Yiyang Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16001
• Pdf URL: https://arxiv.org/pdf/2402.16001
• Copy Paste: [[2402.16001]] Cross-Resolution Land Cover Classification Using Outdated Products and Transformers(https://arxiv.org/abs/2402.16001)
• Keywords: transformer
• Abstract: Large-scale high-resolution land cover classification is a prerequisite for constructing Earth system models and addressing ecological and resource issues. Advancements in satellite sensor technology have led to an improvement in spatial resolution and wider coverage areas. Nevertheless, the lack of high-resolution labeled data is still a challenge, hindering the largescale application of land cover classification methods. In this paper, we propose a Transformerbased weakly supervised method for cross-resolution land cover classification using outdated data. First, to capture long-range dependencies without missing the fine-grained details of objects, we propose a U-Net-like Transformer based on a reverse difference mechanism (RDM) using dynamic sparse attention. Second, we propose an anti-noise loss calculation (ANLC) module based on optimal transport (OT). Anti-noise loss calculation identifies confident areas (CA) and vague areas (VA) based on the OT matrix, which relieves the impact of noises in outdated land cover products. By introducing a weakly supervised loss with weights and employing unsupervised loss, the RDM-based U-Net-like Transformer was trained. Remote sensing images with 1 m resolution and the corresponding ground-truths of six states in the United States were employed to validate the performance of the proposed method. The experiments utilized outdated land cover products with 30 m resolution from 2013 as training labels, and produced land cover maps with 1 m resolution from 2017. The results show the superiority of the proposed method compared to state-of-the-art methods. The code is available at https://github.com/yu-ni1989/ANLC-Former.

Title: Post-Quantum Cryptography Neural Network

• Authors: Abel C. H. Chen
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16002
• Pdf URL: https://arxiv.org/pdf/2402.16002
• Copy Paste: [[2402.16002]] Post-Quantum Cryptography Neural Network(https://arxiv.org/abs/2402.16002)
• Keywords: security, attack
• Abstract: In recent years, quantum computers and Shor quantum algorithm have posed a threat to current mainstream asymmetric cryptography methods (e.g. RSA and Elliptic Curve Cryptography (ECC)). Therefore, it is necessary to construct a Post-Quantum Cryptography (PQC) method to resist quantum computing attacks. Therefore, this study proposes a PQC-based neural network that maps a code-based PQC method to a neural network structure and enhances the security of ciphertexts with non-linear activation functions, random perturbation of ciphertexts, and uniform distribution of ciphertexts. In practical experiments, this study uses cellular network signals as a case study to demonstrate that encryption and decryption can be performed by the proposed PQC-based neural network with the uniform distribution of ciphertexts. In the future, the proposed PQC-based neural network could be applied to various applications.

Title: Adversarial-Robust Transfer Learning for Medical Imaging via Domain Assimilation

• Authors: Xiaohui Chen, Tie Luo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16005
• Pdf URL: https://arxiv.org/pdf/2402.16005
• Copy Paste: [[2402.16005]] Adversarial-Robust Transfer Learning for Medical Imaging via Domain Assimilation(https://arxiv.org/abs/2402.16005)
• Keywords: security, attack, robust
• Abstract: In the field of Medical Imaging, extensive research has been dedicated to leveraging its potential in uncovering critical diagnostic features in patients. Artificial Intelligence (AI)-driven medical diagnosis relies on sophisticated machine learning and deep learning models to analyze, detect, and identify diseases from medical images. Despite the remarkable performance of these models, characterized by high accuracy, they grapple with trustworthiness issues. The introduction of a subtle perturbation to the original image empowers adversaries to manipulate the prediction output, redirecting it to other targeted or untargeted classes. Furthermore, the scarcity of publicly available medical images, constituting a bottleneck for reliable training, has led contemporary algorithms to depend on pretrained models grounded on a large set of natural images -- a practice referred to as transfer learning. However, a significant {\em domain discrepancy} exists between natural and medical images, which causes AI models resulting from transfer learning to exhibit heightened {\em vulnerability} to adversarial attacks. This paper proposes a {\em domain assimilation} approach that introduces texture and color adaptation into transfer learning, followed by a texture preservation component to suppress undesired distortion. We systematically analyze the performance of transfer learning in the face of various adversarial attacks under different data modalities, with the overarching goal of fortifying the model's robustness and security in medical imaging tasks. The results demonstrate high effectiveness in reducing attack efficacy, contributing toward more trustworthy transfer learning in biomedical applications.

Title: From Noise to Clarity: Unraveling the Adversarial Suffix of Large Language Model Attacks via Translation of Text Embeddings

• Authors: Hao Wang, Hao Li, Minlie Huang, Lei Sha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16006
• Pdf URL: https://arxiv.org/pdf/2402.16006
• Copy Paste: [[2402.16006]] From Noise to Clarity: Unraveling the Adversarial Suffix of Large Language Model Attacks via Translation of Text Embeddings(https://arxiv.org/abs/2402.16006)
• Keywords: defense, attack, large language model
• Abstract: The safety defense methods of Large language models(LLMs) stays limited because the dangerous prompts are manually curated to just few known attack types, which fails to keep pace with emerging varieties. Recent studies found that attaching suffixes to harmful instructions can hack the defense of LLMs and lead to dangerous outputs. This method, while effective, leaves a gap in understanding the underlying mechanics of such adversarial suffix due to the non-readability and it can be relatively easily seen through by common defense methods such as perplexity filters.To cope with this challenge, in this paper, we propose an Adversarial Suffixes Embedding Translation Framework(ASETF) that are able to translate the unreadable adversarial suffixes into coherent, readable text, which makes it easier to understand and analyze the reasons behind harmful content generation by large language models. We conducted experiments on LLMs such as LLaMa2, Vicuna and using the Advbench dataset's harmful instructions. The results indicate that our method achieves a much better attack success rate to existing techniques, while significantly enhancing the textual fluency of the prompts. In addition, our approach can be generalized into a broader method for generating transferable adversarial suffixes that can successfully attack multiple LLMs, even black-box LLMs, such as ChatGPT and Gemini. As a result, the prompts generated through our method exhibit enriched semantic diversity, which potentially provides more adversarial examples for LLM defense methods.

Title: Unmasking Dementia Detection by Masking Input Gradients: A JSM Approach to Model Interpretability and Precision

• Authors: Yasmine Mustafa, Tie Luo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16008
• Pdf URL: https://arxiv.org/pdf/2402.16008
• Copy Paste: [[2402.16008]] Unmasking Dementia Detection by Masking Input Gradients: A JSM Approach to Model Interpretability and Precision(https://arxiv.org/abs/2402.16008)
• Keywords: interpretability
• Abstract: The evolution of deep learning and artificial intelligence has significantly reshaped technological landscapes. However, their effective application in crucial sectors such as medicine demands more than just superior performance, but trustworthiness as well. While interpretability plays a pivotal role, existing explainable AI (XAI) approaches often do not reveal {\em Clever Hans} behavior where a model makes (ungeneralizable) correct predictions using spurious correlations or biases in data. Likewise, current post-hoc XAI methods are susceptible to generating unjustified counterfactual examples. In this paper, we approach XAI with an innovative {\em model debugging} methodology realized through Jacobian Saliency Map (JSM). To cast the problem into a concrete context, we employ Alzheimer's disease (AD) diagnosis as the use case, motivated by its significant impact on human lives and the formidable challenge in its early detection, stemming from the intricate nature of its progression. We introduce an interpretable, multimodal model for AD classification over its multi-stage progression, incorporating JSM as a modality-agnostic tool that provides insights into volumetric changes indicative of brain abnormalities. Our extensive evaluation including ablation study manifests the efficacy of using JSM for model debugging and interpretation, while significantly enhancing model accuracy as well.

Title: Spectrum Extraction and Clipping for Implicitly Linear Layers

• Authors: Ali Ebrahimpour Boroojeny, Matus Telgarsky, Hari Sundaram
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16017
• Pdf URL: https://arxiv.org/pdf/2402.16017
• Copy Paste: [[2402.16017]] Spectrum Extraction and Clipping for Implicitly Linear Layers(https://arxiv.org/abs/2402.16017)
• Keywords: robust, extraction
• Abstract: We show the effectiveness of automatic differentiation in efficiently and correctly computing and controlling the spectrum of implicitly linear operators, a rich family of layer types including all standard convolutional and dense layers. We provide the first clipping method which is correct for general convolution layers, and illuminate the representational limitation that caused correctness issues in prior work. We study the effect of the batch normalization layers when concatenated with convolutional layers and show how our clipping method can be applied to their composition. By comparing the accuracy and performance of our algorithms to the state-of-the-art methods, using various experiments, we show they are more precise and efficient and lead to better generalization and adversarial robustness. We provide the code for using our methods at https://github.com/Ali-E/FastClip.

Title: FedFDP: Federated Learning with Fairness and Differential Privacy

• Authors: Xinpeng Ling, Jie Fu, Zhili Chen, Kuncan Wang, Huifa Li, Tong Cheng, Guanying Xu, Qin Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16028
• Pdf URL: https://arxiv.org/pdf/2402.16028
• Copy Paste: [[2402.16028]] FedFDP: Federated Learning with Fairness and Differential Privacy(https://arxiv.org/abs/2402.16028)
• Keywords: privacy, protect, attack, membership infer, federate, fair
• Abstract: Federated learning (FL) is a new machine learning paradigm to overcome the challenge of data silos and has garnered significant attention. However, through our observations, a globally effective trained model may performance disparities in different clients. This implies that the jointly trained models by clients may lead to unfair outcomes. On the other hand, relevant studies indicate that the transmission of gradients or models in federated learning can also give rise to privacy leakage issues, such as membership inference attacks. To address the first issue mentioned above, we propose a federated algorithm with fairness, termed FedFair. Building upon FedFair, we introduce privacy protection to form the FedFDP algorithm to address the second issue mentioned above. In FedFDP, we devise a fairness-aware clipping strategy to achieve differential privacy while adjusting fairness. Additionally, for the extra uploaded loss values, we present an adaptive clipping approach to maximize utility. Furthermore, we theoretically prove that our algorithm converges and ensures differential privacy. Lastly, Extensive experimental results demonstrate that FedFair and FedFDP significantly outperforms state-of-the-art solutions in terms of model performance and fairness. The code is accessible at https://anonymous.4open.science/r/FedFDP-E754.

Title: GraphWiz: An Instruction-Following Language Model for Graph Problems

• Authors: Nuo Chen, Yuhan Li, Jianheng Tang, Jia Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16029
• Pdf URL: https://arxiv.org/pdf/2402.16029
• Copy Paste: [[2402.16029]] GraphWiz: An Instruction-Following Language Model for Graph Problems(https://arxiv.org/abs/2402.16029)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved impressive success across several fields, but their proficiency in understanding and resolving complex graph problems is less explored. To bridge this gap, we introduce GraphInstruct, a novel and comprehensive instruction-tuning dataset designed to equip language models with the ability to tackle a broad spectrum of graph problems using explicit reasoning paths. Utilizing GraphInstruct, we build GraphWiz, an open-source language model capable of resolving various graph problem types while generating clear reasoning processes. To enhance the model's capability and reliability, we incorporate the Direct Preference Optimization (DPO) framework into the graph problem-solving context. The enhanced model, GraphWiz-DPO, achieves an average accuracy of 65% across nine tasks with different complexity levels, surpassing GPT-4 which has an average accuracy of 43.8%. Moreover, our research delves into the delicate balance between training data volume and model performance, highlighting the potential for overfitting with increased data. We also explore the transferability of the model's reasoning ability across different graph tasks, indicating the model's adaptability and practical application potential. Our investigation offers a new blueprint and valuable insights for developing LLMs specialized in graph reasoning and problem-solving.

Title: Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration

• Authors: Xin Mao, Feng-Lin Li, Huimin Xu, Wei Zhang, Anh Tuan Luu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16030
• Pdf URL: https://arxiv.org/pdf/2402.16030
• Copy Paste: [[2402.16030]] Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration(https://arxiv.org/abs/2402.16030)
• Keywords: robust, large language model
• Abstract: While Reinforcement Learning from Human Feedback (RLHF) significantly enhances the generation quality of Large Language Models (LLMs), recent studies have raised concerns regarding the complexity and instability associated with the Proximal Policy Optimization (PPO) algorithm, proposing a series of order-based calibration methods as viable alternatives. This paper delves further into current order-based methods, examining their inefficiencies in utilizing reward values and addressing misalignment issues. Building upon these findings, we propose a novel \textbf{V}alue-based \textbf{C}ali\textbf{B}ration (VCB) method to better align LLMs with human preferences. Experimental results demonstrate that VCB surpasses existing alignment methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and stability in diverse settings.

Title: Diving Deep into Regions: Exploiting Regional Information Transformer for Single Image Deraining

• Authors: Baiang Li, Zhao Zhang, Huan Zheng, Xiaogang Xu, Yanyan Wei, Jingyi Zhang, Jicong Fan, Meng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16033
• Pdf URL: https://arxiv.org/pdf/2402.16033
• Copy Paste: [[2402.16033]] Diving Deep into Regions: Exploiting Regional Information Transformer for Single Image Deraining(https://arxiv.org/abs/2402.16033)
• Keywords: robust, transformer
• Abstract: Transformer-based Single Image Deraining (SID) methods have achieved remarkable success, primarily attributed to their robust capability in capturing long-range interactions. However, we've noticed that current methods handle rain-affected and unaffected regions concurrently, overlooking the disparities between these areas, resulting in confusion between rain streaks and background parts, and inabilities to obtain effective interactions, ultimately resulting in suboptimal deraining outcomes. To address the above issue, we introduce the Region Transformer (Regformer), a novel SID method that underlines the importance of independently processing rain-affected and unaffected regions while considering their combined impact for high-quality image reconstruction. The crux of our method is the innovative Region Transformer Block (RTB), which integrates a Region Masked Attention (RMA) mechanism and a Mixed Gate Forward Block (MGFB). Our RTB is used for attention selection of rain-affected and unaffected regions and local modeling of mixed scales. The RMA generates attention maps tailored to these two regions and their interactions, enabling our model to capture comprehensive features essential for rain removal. To better recover high-frequency textures and capture more local details, we develop the MGFB as a compensation module to complete local mixed scale modeling. Extensive experiments demonstrate that our model reaches state-of-the-art performance, significantly improving the image deraining quality. Our code and trained models are publicly available.

Title: Text Understanding and Generation Using Transformer Models for Intelligent E-commerce Recommendations

• Authors: Yafei Xiang, Hanyi Yu, Yulu Gong, Shuning Huo, Mengran Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16035
• Pdf URL: https://arxiv.org/pdf/2402.16035
• Copy Paste: [[2402.16035]] Text Understanding and Generation Using Transformer Models for Intelligent E-commerce Recommendations(https://arxiv.org/abs/2402.16035)
• Keywords: privacy, protect, transformer, large language model
• Abstract: With the rapid development of artificial intelligence technology, Transformer structural pre-training model has become an important tool for large language model (LLM) tasks. In the field of e-commerce, these models are especially widely used, from text understanding to generating recommendation systems, which provide powerful technical support for improving user experience and optimizing service processes. This paper reviews the core application scenarios of Transformer pre-training model in e-commerce text understanding and recommendation generation, including but not limited to automatic generation of product descriptions, sentiment analysis of user comments, construction of personalized recommendation system and automated processing of customer service conversations. Through a detailed analysis of the model's working principle, implementation process, and application effects in specific cases, this paper emphasizes the unique advantages of pre-trained models in understanding complex user intentions and improving the quality of recommendations. In addition, the challenges and improvement directions for the future are also discussed, such as how to further improve the generalization ability of the model, the ability to handle large-scale data sets, and technical strategies to protect user privacy. Ultimately, the paper points out that the application of Transformer structural pre-training models in e-commerce has not only driven technological innovation, but also brought substantial benefits to merchants and consumers, and looking forward, these models will continue to play a key role in e-commerce and beyond.

Title: Deep Learning Approaches for Improving Question Answering Systems in Hepatocellular Carcinoma Research

• Authors: Shuning Huo, Yafei Xiang, Hanyi Yu, Mengran Zhu, Yulu Gong
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16038
• Pdf URL: https://arxiv.org/pdf/2402.16038
• Copy Paste: [[2402.16038]] Deep Learning Approaches for Improving Question Answering Systems in Hepatocellular Carcinoma Research(https://arxiv.org/abs/2402.16038)
• Keywords: robust
• Abstract: In recent years, advancements in natural language processing (NLP) have been fueled by deep learning techniques, particularly through the utilization of powerful computing resources like GPUs and TPUs. Models such as BERT and GPT-3, trained on vast amounts of data, have revolutionized language understanding and generation. These pre-trained models serve as robust bases for various tasks including semantic understanding, intelligent writing, and reasoning, paving the way for a more generalized form of artificial intelligence. NLP, as a vital application of AI, aims to bridge the gap between humans and computers through natural language interaction. This paper delves into the current landscape and future prospects of large-scale model-based NLP, focusing on the question-answering systems within this domain. Practical cases and developments in artificial intelligence-driven question-answering systems are analyzed to foster further exploration and research in the realm of large-scale NLP.

Title: EHRNoteQA: A Patient-Specific Question Answering Benchmark for Evaluating Large Language Models in Clinical Settings

• Authors: Sunjun Kweon, Jiyoun Kim, Heeyoung Kwak, Dongchul Cha, Hangyul Yoon, Kwanghyun Kim, Seunghyun Won, Edward Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16040
• Pdf URL: https://arxiv.org/pdf/2402.16040
• Copy Paste: [[2402.16040]] EHRNoteQA: A Patient-Specific Question Answering Benchmark for Evaluating Large Language Models in Clinical Settings(https://arxiv.org/abs/2402.16040)
• Keywords: large language model
• Abstract: This study introduces EHRNoteQA, a novel patient-specific question answering benchmark tailored for evaluating Large Language Models (LLMs) in clinical environments. Based on MIMIC-IV Electronic Health Record (EHR), a team of three medical professionals has curated the dataset comprising 962 unique questions, each linked to a specific patient's EHR clinical notes. What makes EHRNoteQA distinct from existing EHR-based benchmarks is as follows: Firstly, it is the first dataset to adopt a multi-choice question answering format, a design choice that effectively evaluates LLMs with reliable scores in the context of automatic evaluation, compared to other formats. Secondly, it requires an analysis of multiple clinical notes to answer a single question, reflecting the complex nature of real-world clinical decision-making where clinicians review extensive records of patient histories. Our comprehensive evaluation on various large language models showed that their scores on EHRNoteQA correlate more closely with their performance in addressing real-world medical questions evaluated by clinicians than their scores from other LLM benchmarks. This underscores the significance of EHRNoteQA in evaluating LLMs for medical applications and highlights its crucial role in facilitating the integration of LLMs into healthcare systems. The dataset will be made available to the public under PhysioNet credential access, promoting further research in this vital field.

Title: Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy

• Authors: Shuhai Zhang, Feng Liu, Jiahao Yang, Yifan Yang, Changsheng Li, Bo Han, Mingkui Tan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16041
• Pdf URL: https://arxiv.org/pdf/2402.16041
• Copy Paste: [[2402.16041]] Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy(https://arxiv.org/abs/2402.16041)
• Keywords: large language model
• Abstract: Large language models (LLMs) such as ChatGPT have exhibited remarkable performance in generating human-like texts. However, machine-generated texts (MGTs) may carry critical risks, such as plagiarism issues, misleading information, or hallucination issues. Therefore, it is very urgent and important to detect MGTs in many situations. Unfortunately, it is challenging to distinguish MGTs and human-written texts because the distributional discrepancy between them is often very subtle due to the remarkable performance of LLMs. In this paper, we seek to exploit \textit{maximum mean discrepancy} (MMD) to address this issue in the sense that MMD can well identify distributional discrepancies. However, directly training a detector with MMD using diverse MGTs will incur a significantly increased variance of MMD since MGTs may contain \textit{multiple text populations} due to various LLMs. This will severely impair MMD's ability to measure the difference between two samples. To tackle this, we propose a novel \textit{multi-population} aware optimization method for MMD called MMD-MP, which can \textit{avoid variance increases} and thus improve the stability to measure the distributional discrepancy. Relying on MMD-MP, we develop two methods for paragraph-based and sentence-based detection, respectively. Extensive experiments on various LLMs, \eg, GPT2 and ChatGPT, show superior detection performance of our MMD-MP. The source code is available at \url{https://github.com/ZSHsh98/MMD-MP}.

Title: LuaTaint: A Static Taint Analysis System for Web Interface Framework Vulnerability of IoT Devices

• Authors: Jiahui Xiang, Wenhai Wang, Tong Ye, Peiyu Liu
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.16043
• Pdf URL: https://arxiv.org/pdf/2402.16043
• Copy Paste: [[2402.16043]] LuaTaint: A Static Taint Analysis System for Web Interface Framework Vulnerability of IoT Devices(https://arxiv.org/abs/2402.16043)
• Keywords: secure, security, attack, large language model
• Abstract: IoT devices are currently facing continuous malicious attacks due to their widespread use. Among these IoT devices, web vulnerabilities are also widely exploited because of their inherent characteristics, such as improper permission controls and insecure interfaces. Recently, the embedded system web interface framework has become highly diverse, and specific vulnerabilities can arise if developers forget to detect user input parameters or if the detection process is not strict enough. Therefore, discovering vulnerabilities in the web interfaces of IoT devices accurately and comprehensively through an automated method is a major challenge. This paper aims to work out the challenge. We have developed an automated vulnerability detection system called LuaTaint for the typical web interface framework, LuCI. The system employs static taint analysis to address web security issues on mobile terminal platforms to ensure detection coverage. It integrates rules pertaining to page handler control logic within the taint detection process to improve its extensibility. We also implemented a post-processing step with the assistance of large language models to enhance accuracy and reduce the need for manual analysis. We have created a prototype of LuaTaint and tested it on 92 IoT firmwares from 8 well-known vendors. LuaTaint has discovered 68 unknown vulnerabilities.

Title: LLMs with Chain-of-Thought Are Non-Causal Reasoners

• Authors: Guangsheng Bao, Hongbo Zhang, Linyi Yang, Cunxiang Wang, Yue Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16048
• Pdf URL: https://arxiv.org/pdf/2402.16048
• Copy Paste: [[2402.16048]] LLMs with Chain-of-Thought Are Non-Causal Reasoners(https://arxiv.org/abs/2402.16048)
• Keywords: large language model
• Abstract: This paper explores the role of the Chain of Thought (CoT) in Large Language Models (LLMs) reasoning. Despite its potential to improve task performance, our analysis reveals a surprising frequency of correct answers following incorrect CoTs and vice versa. We employ causal analysis to assess the cause-effect relationship between CoTs/instructions and answers in LLMs, uncovering the Structural Causal Model (SCM) that LLMs approximate. By comparing the implied SCM with that of human reasoning, we highlight discrepancies between LLM and human reasoning processes. We further examine the factors influencing the causal structure of the implied SCM, revealing that in-context learning, supervised fine-tuning, and reinforcement learning on human feedback significantly impact the causal relations. We release the code and results at https://github.com/StevenZHB/CoT_Causal_Analysis.

Title: LSTP: Language-guided Spatial-Temporal Prompt Learning for Long-form Video-Text Understanding

• Authors: Yuxuan Wang, Yueqian Wang, Pengfei Wu, Jianxin Liang, Dongyan Zhao, Zilong Zheng
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16050
• Pdf URL: https://arxiv.org/pdf/2402.16050
• Copy Paste: [[2402.16050]] LSTP: Language-guided Spatial-Temporal Prompt Learning for Long-form Video-Text Understanding(https://arxiv.org/abs/2402.16050)
• Keywords: large language model
• Abstract: Despite progress in video-language modeling, the computational challenge of interpreting long-form videos in response to task-specific linguistic queries persists, largely due to the complexity of high-dimensional video data and the misalignment between language and visual cues over space and time. To tackle this issue, we introduce a novel approach called Language-guided Spatial-Temporal Prompt Learning (LSTP). This approach features two key components: a Temporal Prompt Sampler (TPS) with optical flow prior that leverages temporal information to efficiently extract relevant video content, and a Spatial Prompt Solver (SPS) that adeptly captures the intricate spatial relationships between visual and textual elements. By harmonizing TPS and SPS with a cohesive training strategy, our framework significantly enhances computational efficiency, temporal understanding, and spatial-temporal alignment. Empirical evaluations across two challenging tasks--video question answering and temporal question grounding in videos--using a variety of video-language pretrainings (VLPs) and large language models (LLMs) demonstrate the superior performance, speed, and versatility of our proposed LSTP paradigm.

Title: Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression

• Authors: Xinze Li, Zhenghao Liu, Chenyan Xiong, Shi Yu, Yukun Yan, Shuo Wang, Ge Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16058
• Pdf URL: https://arxiv.org/pdf/2402.16058
• Copy Paste: [[2402.16058]] Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression(https://arxiv.org/abs/2402.16058)
• Keywords: large language model
• Abstract: Large language models (LLMs) require lengthy prompts as the input context to produce output aligned with user intentions, a process that incurs extra costs during inference. In this paper, we propose the Gist COnditioned deCOding (Gist-COCO) model, introducing a novel method for compressing prompts which also can assist the prompt interpretation and engineering. Gist-COCO employs an encoder-decoder based language model and then incorporates an additional encoder as a plugin module to compress prompts with inputs using gist tokens. It finetunes the compression plugin module and uses the representations of gist tokens to emulate the raw prompts in the vanilla language model. By verbalizing the representations of gist tokens into gist prompts, the compression ability of Gist-COCO can be generalized to different LLMs with high compression rates. Our experiments demonstrate that Gist-COCO outperforms previous prompt compression models in both passage and instruction compression tasks. Further analysis on gist verbalization results suggests that our gist prompts serve different functions in aiding language models. They may directly provide potential answers, generate the chain-of-thought, or simply repeat the inputs. All data and codes are available at https://github.com/OpenMatch/Gist-COCO .

Title: How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study

• Authors: Tianjie Ju, Weiwei Sun, Wei Du, Xinwei Yuan, Zhaochun Ren, Gongshen Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16061
• Pdf URL: https://arxiv.org/pdf/2402.16061
• Copy Paste: [[2402.16061]] How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study(https://arxiv.org/abs/2402.16061)
• Keywords: generative, large language model
• Abstract: Previous work has showcased the intriguing capability of large language models (LLMs) in retrieving facts and processing context knowledge. However, only limited research exists on the layer-wise capability of LLMs to encode knowledge, which challenges our understanding of their internal mechanisms. In this paper, we devote the first attempt to investigate the layer-wise capability of LLMs through probing tasks. We leverage the powerful generative capability of ChatGPT to construct probing datasets, providing diverse and coherent evidence corresponding to various facts. We employ $\mathcal V$-usable information as the validation metric to better reflect the capability in encoding context knowledge across different layers. Our experiments on conflicting and newly acquired knowledge show that LLMs: (1) prefer to encode more context knowledge in the upper layers; (2) primarily encode context knowledge within knowledge-related entity tokens at lower layers while progressively expanding more knowledge within other tokens at upper layers; and (3) gradually forget the earlier context knowledge retained within the intermediate layers when provided with irrelevant evidence. Code is publicly available at https://github.com/Jometeorie/probing_llama.

Title: Citation-Enhanced Generation for LLM-based Chatbot

• Authors: Weitao Li, Junkai Li, Weizhi Ma, Yang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16063
• Pdf URL: https://arxiv.org/pdf/2402.16063
• Copy Paste: [[2402.16063]] Citation-Enhanced Generation for LLM-based Chatbot(https://arxiv.org/abs/2402.16063)
• Keywords: large language model
• Abstract: Large language models (LLMs) exhibit powerful general intelligence across diverse scenarios, including their integration into chatbots. However, a vital challenge of LLM-based chatbots is that they may produce hallucinated content in responses, which significantly limits their applicability. Various efforts have been made to alleviate hallucination, such as retrieval augmented generation and reinforcement learning with human feedback, but most of them require additional training and data annotation. In this paper, we propose a novel post-hoc \textbf{C}itation-\textbf{E}nhanced \textbf{G}eneration (\textbf{CEG}) approach combined with retrieval argumentation. Unlike previous studies that focus on preventing hallucinations during generation, our method addresses this issue in a post-hoc way. It incorporates a retrieval module to search for supporting documents relevant to the generated content, and employs a natural language inference-based citation generation module. Once the statements in the generated content lack of reference, our model can regenerate responses until all statements are supported by citations. Note that our method is a training-free plug-and-play plugin that is capable of various LLMs. Experiments on various hallucination-related datasets show our framework outperforms state-of-the-art methods in both hallucination detection and response regeneration on three benchmarks. Our codes and dataset will be publicly available.

Title: Behavioral Refinement via Interpolant-based Policy Diffusion

• Authors: Kaiqi Chen, Eugene Lim, Kelvin Lin, Yiyang Chen, Harold Soh
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.16075
• Pdf URL: https://arxiv.org/pdf/2402.16075
• Copy Paste: [[2402.16075]] Behavioral Refinement via Interpolant-based Policy Diffusion(https://arxiv.org/abs/2402.16075)
• Keywords: diffusion
• Abstract: Imitation learning empowers artificial agents to mimic behavior by learning from demonstrations. Recently, diffusion models, which have the ability to model high-dimensional and multimodal distributions, have shown impressive performance on imitation learning tasks. These models learn to shape a policy by diffusing actions (or states) from standard Gaussian noise. However, the target policy to be learned is often significantly different from Gaussian and this mismatch can result in poor performance when using a small number of diffusion steps (to improve inference speed) and under limited data. The key idea in this work is that initiating from a more informative source than Gaussian enables diffusion methods to overcome the above limitations. We contribute both theoretical results, a new method, and empirical findings that show the benefits of using an informative source policy. Our method, which we call BRIDGER, leverages the stochastic interpolants framework to bridge arbitrary policies, thus enabling a flexible approach towards imitation learning. It generalizes prior work in that standard Gaussians can still be applied, but other source policies can be used if available. In experiments on challenging benchmarks, BRIDGER outperforms state-of-the-art diffusion policies and we provide further analysis on design considerations when applying BRIDGER.

Title: Equivariant Frames and the Impossibility of Continuous Canonicalization

• Authors: Nadav Dym, Hannah Lawrence, Jonathan W. Siegel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16077
• Pdf URL: https://arxiv.org/pdf/2402.16077
• Copy Paste: [[2402.16077]] Equivariant Frames and the Impossibility of Continuous Canonicalization(https://arxiv.org/abs/2402.16077)
• Keywords: robust
• Abstract: Canonicalization provides an architecture-agnostic method for enforcing equivariance, with generalizations such as frame-averaging recently gaining prominence as a lightweight and flexible alternative to equivariant architectures. Recent works have found an empirical benefit to using probabilistic frames instead, which learn weighted distributions over group elements. In this work, we provide strong theoretical justification for this phenomenon: for commonly-used groups, there is no efficiently computable choice of frame that preserves continuity of the function being averaged. In other words, unweighted frame-averaging can turn a smooth, non-symmetric function into a discontinuous, symmetric function. To address this fundamental robustness problem, we formally define and construct \emph{weighted} frames, which provably preserve continuity, and demonstrate their utility by constructing efficient and continuous weighted frames for the actions of $SO(2)$, $SO(3)$, and $S_n$ on point clouds.

Title: Deep Homography Estimation for Visual Place Recognition

• Authors: Feng Lu, Shuting Dong, Lijun Zhang, Bingxi Liu, Xiangyuan Lan, Dongmei Jiang, Chun Yuan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16086
• Pdf URL: https://arxiv.org/pdf/2402.16086
• Copy Paste: [[2402.16086]] Deep Homography Estimation for Visual Place Recognition(https://arxiv.org/abs/2402.16086)
• Keywords: extraction, transformer
• Abstract: Visual place recognition (VPR) is a fundamental task for many applications such as robot localization and augmented reality. Recently, the hierarchical VPR methods have received considerable attention due to the trade-off between accuracy and efficiency. They usually first use global features to retrieve the candidate images, then verify the spatial consistency of matched local features for re-ranking. However, the latter typically relies on the RANSAC algorithm for fitting homography, which is time-consuming and non-differentiable. This makes existing methods compromise to train the network only in global feature extraction. Here, we propose a transformer-based deep homography estimation (DHE) network that takes the dense feature map extracted by a backbone network as input and fits homography for fast and learnable geometric verification. Moreover, we design a re-projection error of inliers loss to train the DHE network without additional homography labels, which can also be jointly trained with the backbone network to help it extract the features that are more suitable for local matching. Extensive experiments on benchmark datasets show that our method can outperform several state-of-the-art methods. And it is more than one order of magnitude faster than the mainstream hierarchical VPR methods using RANSAC. The code is released at https://github.com/Lu-Feng/DHE-VPR.

Title: How to Privately Tune Hyperparameters in Federated Learning? Insights from a Benchmark Study

• Authors: Natalija Mitic, Apostolos Pyrgelis, Sinem Sav
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16087
• Pdf URL: https://arxiv.org/pdf/2402.16087
• Copy Paste: [[2402.16087]] How to Privately Tune Hyperparameters in Federated Learning? Insights from a Benchmark Study(https://arxiv.org/abs/2402.16087)
• Keywords: privacy, federate
• Abstract: In this paper, we address the problem of privacy-preserving hyperparameter (HP) tuning for cross-silo federated learning (FL). We first perform a comprehensive measurement study that benchmarks various HP strategies suitable for FL. Our benchmarks show that the optimal parameters of the FL server, e.g., the learning rate, can be accurately and efficiently tuned based on the HPs found by each client on its local data. We demonstrate that HP averaging is suitable for iid settings, while density-based clustering can uncover the optimal set of parameters in non-iid ones. Then, to prevent information leakage from the exchange of the clients' local HPs, we design and implement PrivTuna, a novel framework for privacy-preserving HP tuning using multiparty homomorphic encryption. We use PrivTuna to implement privacy-preserving federated averaging and density-based clustering, and we experimentally evaluate its performance demonstrating its computation/communication efficiency and its precision in tuning hyperparameters.

Title: Bayesian Neural Network For Personalized Federated Learning Parameter Selection

• Authors: Mengen Luo, Ercan Engin Kuruoglu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16091
• Pdf URL: https://arxiv.org/pdf/2402.16091
• Copy Paste: [[2402.16091]] Bayesian Neural Network For Personalized Federated Learning Parameter Selection(https://arxiv.org/abs/2402.16091)
• Keywords: federate
• Abstract: Federated learning's poor performance in the presence of heterogeneous data remains one of the most pressing issues in the field. Personalized federated learning departs from the conventional paradigm in which all clients employ the same model, instead striving to discover an individualized model for each client to address the heterogeneity in the data. One of such approach involves personalizing specific layers of neural networks. However, prior endeavors have not provided a dependable rationale, and some have selected personalized layers that are entirely distinct and conflicting. In this work, we take a step further by proposing personalization at the elemental level, rather than the traditional layer-level personalization. To select personalized parameters, we introduce Bayesian neural networks and rely on the uncertainty they offer to guide our selection of personalized parameters. Finally, we validate our algorithm's efficacy on several real-world datasets, demonstrating that our proposed approach outperforms existing baselines.

Title: StochCA: A Novel Approach for Exploiting Pretrained Models with Cross-Attention

• Authors: Seungwon Seo, Suho Lee, Sangheum Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16092
• Pdf URL: https://arxiv.org/pdf/2402.16092
• Copy Paste: [[2402.16092]] StochCA: A Novel Approach for Exploiting Pretrained Models with Cross-Attention(https://arxiv.org/abs/2402.16092)
• Keywords: transformer
• Abstract: Utilizing large-scale pretrained models is a well-known strategy to enhance performance on various target tasks. It is typically achieved through fine-tuning pretrained models on target tasks. However, na\"{\i}ve fine-tuning may not fully leverage knowledge embedded in pretrained models. In this study, we introduce a novel fine-tuning method, called stochastic cross-attention (StochCA), specific to Transformer architectures. This method modifies the Transformer's self-attention mechanism to selectively utilize knowledge from pretrained models during fine-tuning. Specifically, in each block, instead of self-attention, cross-attention is performed stochastically according to the predefined probability, where keys and values are extracted from the corresponding block of a pretrained model. By doing so, queries and channel-mixing multi-layer perceptron layers of a target model are fine-tuned to target tasks to learn how to effectively exploit rich representations of pretrained models. To verify the effectiveness of StochCA, extensive experiments are conducted on benchmarks in the areas of transfer learning and domain generalization, where the exploitation of pretrained models is critical. Our experimental results show the superiority of StochCA over state-of-the-art approaches in both areas. Furthermore, we demonstrate that StochCA is complementary to existing approaches, i.e., it can be combined with them to further improve performance. Our code is available at https://github.com/daintlab/stochastic_cross_attention

Title: Bistochastically private release of data streams with zero delay

• Authors: Nicolas Ruiz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16094
• Pdf URL: https://arxiv.org/pdf/2402.16094
• Copy Paste: [[2402.16094]] Bistochastically private release of data streams with zero delay(https://arxiv.org/abs/2402.16094)
• Keywords: privacy, protect
• Abstract: Although the bulk of the research in privacy and statistical disclosure control is designed for static data, more and more data are often collected as continuous streams, and extensions of popular privacy tools and models have been proposed for this scenario. However, most of these proposals require buffers, where incoming individuals are momentarily stored, anonymized, and then released following a delay, thus considering a data stream as a succession of batches while it is by nature continuous. Having a delay unavoidably alters data freshness but also, more critically, inordinately exerts constraints on what can be achieved in terms of protection and information preservation. By considering randomized response, and specifically its recent bistochastic extension, in the context of dynamic data, this paper proposes a protocol for the anonymization of data streams that achieves zero delay while exhibiting formal privacy guarantees. Using a new tool in the privacy literature that introduces the concept of elementary plausible deniability, we show that it is feasible to achieve an atomic processing of individuals entering a stream, in-stead of proceeding by batches. We illustrate the application of the proposed approach by an empirical example.

Title: chainBoost: A Secure Performance Booster for Blockchain-based Resource Markets

• Authors: Zahra Motaqy, Mohamed E. Najd, Ghada Almashaqbeh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16095
• Pdf URL: https://arxiv.org/pdf/2402.16095
• Copy Paste: [[2402.16095]] chainBoost: A Secure Performance Booster for Blockchain-based Resource Markets(https://arxiv.org/abs/2402.16095)
• Keywords: secure, security, robust
• Abstract: Cryptocurrencies and blockchain technology provide an innovative model for reshaping digital services. Driven by the movement toward Web 3.0, recent systems started to provide distributed services, such as computation outsourcing or file storage, on top of the currency exchange medium. By allowing anyone to join and collect cryptocurrency payments for serving others, these systems create decentralized markets for trading digital resources. Yet, there is still a big gap between the promise of these markets and their practical viability. Existing initiatives are still early-stage and have already encountered security and efficiency obstacles. At the same time, existing work around promising ideas, specifically sidechains, fall short in exploiting their full potential in addressing these problems. To bridge this gap, we propose chainBoost, a secure performance booster for decentralized resource markets. It expedites service related operations, reduces the blockchain size, and supports flexible service-payment exchange modalities at low overhead. At its core, chainBoost employs a sidechain, that has a (security and semantic) mutual-dependence with the mainchain, to which the system offloads heavy/frequent operations. To enable it, we develop a novel sidechain architecture composed of temporary and permanent blocks, a block suppression mechanism to prune the sidechain, a syncing protocol to permit arbitrary data exchange between the two chains, and an autorecovery protocol to support robustness and resilience. We analyze the security of chainBoost, and implement a proof-of-concept prototype for a distributed file storage market as a use case. For a market handling around 2000 transactions per round, our experiments show up to 11x improvement in throughput and 94\% reduction in confirmation time. They also show that chainBoost can reduce the main blockchain size by around 90%.

Title: Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?

• Authors: Joris Baan, Raquel Fernández, Barbara Plank, Wilker Aziz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16102
• Pdf URL: https://arxiv.org/pdf/2402.16102
• Copy Paste: [[2402.16102]] Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?(https://arxiv.org/abs/2402.16102)
• Keywords: fair
• Abstract: With the rise of increasingly powerful and user-facing NLP systems, there is growing interest in assessing whether they have a good representation of uncertainty by evaluating the quality of their predictive distribution over outcomes. We identify two main perspectives that drive starkly different evaluation protocols. The first treats predictive probability as an indication of model confidence; the second as an indication of human label variation. We discuss their merits and limitations, and take the position that both are crucial for trustworthy and fair NLP systems, but that exploiting a single predictive distribution is limiting. We recommend tools and highlight exciting directions towards models with disentangled representations of uncertainty about predictions and uncertainty about human labels.

Title: Informed Meta-Learning

• Authors: Katarzyna Kobalczyk, Mihaela van der Schaar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16105
• Pdf URL: https://arxiv.org/pdf/2402.16105
• Copy Paste: [[2402.16105]] Informed Meta-Learning(https://arxiv.org/abs/2402.16105)
• Keywords: robust
• Abstract: In noisy and low-data regimes prevalent in real-world applications, an outstanding challenge of machine learning lies in effectively incorporating inductive biases that promote data efficiency and robustness. Meta-learning and informed ML stand out as two approaches for incorporating prior knowledge into the ML pipeline. While the former relies on a purely data-driven source of priors, the latter is guided by a formal representation of expert knowledge. This paper introduces a novel hybrid paradigm, informed meta-learning, seeking complementarity in cross-task knowledge sharing of humans and machines. We establish the foundational components of informed meta-learning and present a concrete instantiation of this framework--the Informed Neural Process. Through a series of illustrative and larger-scale experiments, we demonstrate the potential benefits of informed meta-learning in improving data efficiency and robustness to observational noise, task distribution shifts, and heterogeneity.

Title: FuseChat: Knowledge Fusion of Chat Models

• Authors: Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, Wei Bi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16107
• Pdf URL: https://arxiv.org/pdf/2402.16107
• Copy Paste: [[2402.16107]] FuseChat: Knowledge Fusion of Chat Models(https://arxiv.org/abs/2402.16107)
• Keywords: robust, large language model
• Abstract: While training large language models (LLMs) from scratch can indeed lead to models with distinct capabilities and strengths, this approach incurs substantial costs and may lead to potential redundancy in competencies. An alternative strategy is to combine existing LLMs into a more robust LLM, thereby diminishing the necessity for expensive pre-training. However, due to the diverse architectures of LLMs, direct parameter blending proves to be unfeasible. Recently, \textsc{FuseLLM} introduced the concept of knowledge fusion to transfer the collective knowledge of multiple structurally varied LLMs into a target LLM through lightweight continual training. In this report, we extend the scalability and flexibility of the \textsc{FuseLLM} framework to realize the fusion of chat LLMs, resulting in \textsc{FuseChat}. \textsc{FuseChat} comprises two main stages. Firstly, we undertake knowledge fusion for structurally and scale-varied source LLMs to derive multiple target LLMs of identical structure and size via lightweight fine-tuning. Then, these target LLMs are merged within the parameter space, wherein we propose a novel method for determining the merging weights based on the variation ratio of parameter matrices before and after fine-tuning. We validate our approach using three prominent chat LLMs with diverse architectures and scales, namely \texttt{NH2-Mixtral-8x7B}, \texttt{NH2-Solar-10.7B}, and \texttt{OpenChat-3.5-7B}. Experimental results spanning various chat domains demonstrate the superiority of \texttt{\textsc{FuseChat}-7B} across a broad spectrum of chat LLMs at 7B and 34B scales, even surpassing \texttt{GPT-3.5 (March)} and approaching \texttt{Mixtral-8x7B-Instruct}. Our code, model weights, and data are openly accessible at \url{https://github.com/fanqiwan/FuseLLM}.

Title: Towards Accurate Post-training Quantization for Reparameterized Models

• Authors: Luoming Zhang, Yefei He, Wen Fei, Zhenyu Lou, Weijia Wu, YangWei Ying, Hong Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16121
• Pdf URL: https://arxiv.org/pdf/2402.16121
• Copy Paste: [[2402.16121]] Towards Accurate Post-training Quantization for Reparameterized Models(https://arxiv.org/abs/2402.16121)
• Keywords: protect
• Abstract: Model reparameterization is a widely accepted technique for improving inference speed without compromising performance. However, current Post-training Quantization (PTQ) methods often lead to significant accuracy degradation when applied to reparameterized models. This is primarily caused by channel-specific and sample-specific outliers, which appear only at specific samples and channels and impact on the selection of quantization parameters. To address this issue, we propose RepAPQ, a novel framework that preserves the accuracy of quantized reparameterization models. Different from previous frameworks using Mean Squared Error (MSE) as a measurement, we utilize Mean Absolute Error (MAE) to mitigate the influence of outliers on quantization parameters. Our framework comprises two main components: Quantization Protecting Reparameterization and Across-block Calibration. For effective calibration, Quantization Protecting Reparameterization combines multiple branches into a single convolution with an affine layer. During training, the affine layer accelerates convergence and amplifies the output of the convolution to better accommodate samples with outliers. Additionally, Across-block Calibration leverages the measurement of stage output as supervision to address the gradient problem introduced by MAE and enhance the interlayer correlation with quantization parameters. Comprehensive experiments demonstrate the effectiveness of RepAPQ across various models and tasks. Our framework outperforms previous methods by approximately 1\% for 8-bit PTQ and 2\% for 6-bit PTQ, showcasing its superior performance. The code is available at \url{https://github.com/ilur98/DLMC-QUANT}.

Title: InstructEdit: Instruction-based Knowledge Editing for Large Language Models

• Authors: Bozhong Tian, Siyuan Cheng, Xiaozhuan Liang, Ningyu Zhang, Yi Hu, Kouying Xue, Yanjie Gou, Xi Chen, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16123
• Pdf URL: https://arxiv.org/pdf/2402.16123
• Copy Paste: [[2402.16123]] InstructEdit: Instruction-based Knowledge Editing for Large Language Models(https://arxiv.org/abs/2402.16123)
• Keywords: large language model
• Abstract: Knowledge editing for large language models can offer an efficient solution to alter a model's behavior without negatively impacting the overall performance. However, the current approach encounters issues with limited generalizability across tasks, necessitating one distinct editor for each task, which significantly hinders the broader applications. To address this, we take the first step to analyze the multi-task generalization issue in knowledge editing. Specifically, we develop an instruction-based editing technique, termed InstructEdit, which facilitates the editor's adaptation to various task performances simultaneously using simple instructions. With only one unified editor for each LLM, we empirically demonstrate that InstructEdit can improve the editor's control, leading to an average 14.86% increase in Reliability in multi-task editing setting. Furthermore, experiments involving holdout unseen task illustrate that InstructEdit consistently surpass previous strong baselines. To further investigate the underlying mechanisms of instruction-based knowledge editing, we analyze the principal components of the editing gradient directions, which unveils that instructions can help control optimization direction with stronger OOD generalization. Code and datasets will be available in https://github.com/zjunlp/EasyEdit.

Title: AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation

• Authors: Yasheng Sun, Wenqing Chu, Hang Zhou, Kaisiyuan Wang, Hideki Koike
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16124
• Pdf URL: https://arxiv.org/pdf/2402.16124
• Copy Paste: [[2402.16124]] AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation(https://arxiv.org/abs/2402.16124)
• Keywords: robust, interpretability, diffusion, generative, large language model
• Abstract: While considerable progress has been made in achieving accurate lip synchronization for 3D speech-driven talking face generation, the task of incorporating expressive facial detail synthesis aligned with the speaker's speaking status remains challenging. Our goal is to directly leverage the inherent style information conveyed by human speech for generating an expressive talking face that aligns with the speaking status. In this paper, we propose AVI-Talking, an Audio-Visual Instruction system for expressive Talking face generation. This system harnesses the robust contextual reasoning and hallucination capability offered by Large Language Models (LLMs) to instruct the realistic synthesis of 3D talking faces. Instead of directly learning facial movements from human speech, our two-stage strategy involves the LLMs first comprehending audio information and generating instructions implying expressive facial details seamlessly corresponding to the speech. Subsequently, a diffusion-based generative network executes these instructions. This two-stage process, coupled with the incorporation of LLMs, enhances model interpretability and provides users with flexibility to comprehend instructions and specify desired operations or modifications. Extensive experiments showcase the effectiveness of our approach in producing vivid talking faces with expressive facial movements and consistent emotional status.

Title: A statistical method for crack detection in 3D concrete images

• Authors: Vitalii Makogin, Duc Nguyen, Evgeny Spodarev
• Subjects: cs.CV, stat.AP
• Abstract URL: https://arxiv.org/abs/2402.16126
• Pdf URL: https://arxiv.org/pdf/2402.16126
• Copy Paste: [[2402.16126]] A statistical method for crack detection in 3D concrete images(https://arxiv.org/abs/2402.16126)
• Keywords: robust, segmentation
• Abstract: In practical applications, effectively segmenting cracks in large-scale computed tomography (CT) images holds significant importance for understanding the structural integrity of materials. However, classical methods and Machine Learning algorithms often incur high computational costs when dealing with the substantial size of input images. Hence, a robust algorithm is needed to pre-detect crack regions, enabling focused analysis and reducing computational overhead. The proposed approach addresses this challenge by offering a streamlined method for identifying crack regions in CT images with high probability. By efficiently identifying areas of interest, our algorithm allows for a more focused examination of potential anomalies within the material structure. Through comprehensive testing on both semi-synthetic and real 3D CT images, we validate the efficiency of our approach in enhancing crack segmentation while reducing computational resource requirements.

Title: LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting

• Authors: Haoxin Liu, Zhiyuan Zhao, Jindong Wang, Harshavardhan Kamarthi, B. Aditya Prakash
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16132
• Pdf URL: https://arxiv.org/pdf/2402.16132
• Copy Paste: [[2402.16132]] LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting(https://arxiv.org/abs/2402.16132)
• Keywords: large language model
• Abstract: Time-series forecasting (TSF) finds broad applications in real-world scenarios. Prompting off-the-shelf Large Language Models (LLMs) demonstrates strong zero-shot TSF capabilities while preserving computational efficiency. However, existing prompting methods oversimplify TSF as language next-token predictions, overlooking its dynamic nature and lack of integration with state-of-the-art prompt strategies such as Chain-of-Thought. Thus, we propose LSTPrompt, a novel approach for prompting LLMs in zero-shot TSF tasks. LSTPrompt decomposes TSF into short-term and long-term forecasting sub-tasks, tailoring prompts to each. LSTPrompt guides LLMs to regularly reassess forecasting mechanisms to enhance adaptability. Extensive evaluations demonstrate consistently better performance of LSTPrompt than existing prompting methods, and competitive results compared to foundation TSF models.

Title: What Generative Artificial Intelligence Means for Terminological Definitions

• Authors: Antonio San Martín
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16139
• Pdf URL: https://arxiv.org/pdf/2402.16139
• Copy Paste: [[2402.16139]] What Generative Artificial Intelligence Means for Terminological Definitions(https://arxiv.org/abs/2402.16139)
• Keywords: generative
• Abstract: This paper examines the impact of Generative Artificial Intelligence (GenAI) on the creation and consumption of terminological definitions. GenAI tools like ChatGPT present a mix of benefits and drawbacks compared to traditional terminological resources. ChatGPT excels in providing context-specific meanings in an interactive and customized fashion but faces challenges with accuracy. Terminological definitions in recognized resources will likely survive because of their reliability. From the point of view of the terminologist, tools like ChatGPT enable AI-assisted terminography, including post-editing terminography, as an approach blending AI efficiency with human expertise for faster definition creation.

Title: PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization

• Authors: Xiangdi Meng, Damai Dai, Weiyao Luo, Zhe Yang, Shaoxiang Wu, Xiaochen Wang, Peiyi Wang, Qingxiu Dong, Liang Chen, Zhifang Sui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16141
• Pdf URL: https://arxiv.org/pdf/2402.16141
• Copy Paste: [[2402.16141]] PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization(https://arxiv.org/abs/2402.16141)
• Keywords: large language model
• Abstract: Supervised fine-tuning is the most common method to adapt large language models (LLMs) to downstream tasks, but full fine-tuning LLMs requires massive computational resources. Recently, parameter-efficient fine-tuning (PEFT) methods have been widely studied due to its cost-effectiveness. LoRA is one of the most widely used methods, which assumes that the optimization process is essentially low-dimensional. Although LoRA fine-tuning is effective, there is still a performance gap compared to full fine-tuning, since its weight update is limited to low-rank matrices. In order to break the low-rank bottleneck in LoRA Optimization, we propose PeriodicLoRA (PLoRA), which accumulates low-rank update matrices multiple times to achieve a higher update rank. PLoRA has multiple training stages. During each stage, we still update only the LoRA weights. However, at the end of each stage, we unload the LoRA weights into the backbone parameters and then reinitialize the LoRA states. Experimental results show that PLoRA has stronger learning ability, approximately 1.8 times that of LoRA's learning ability at most, but it does not increase memory usage. Further, we introduce a momentum-based unloading strategy for PLoRA to mitigate the training instability.

Title: From Text to Transformation: A Comprehensive Review of Large Language Models' Versatility

• Authors: Pravneet Kaur, Gautam Siddharth Kashyap, Ankit Kumar, Md Tabrez Nafis, Sandeep Kumar, Vikrant Shokeen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16142
• Pdf URL: https://arxiv.org/pdf/2402.16142
• Copy Paste: [[2402.16142]] From Text to Transformation: A Comprehensive Review of Large Language Models' Versatility(https://arxiv.org/abs/2402.16142)
• Keywords: transformer, generative, large language model
• Abstract: This groundbreaking study explores the expanse of Large Language Models (LLMs), such as Generative Pre-Trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT) across varied domains ranging from technology, finance, healthcare to education. Despite their established prowess in Natural Language Processing (NLP), these LLMs have not been systematically examined for their impact on domains such as fitness, and holistic well-being, urban planning, climate modelling as well as disaster management. This review paper, in addition to furnishing a comprehensive analysis of the vast expanse and extent of LLMs' utility in diverse domains, recognizes the research gaps and realms where the potential of LLMs is yet to be harnessed. This study uncovers innovative ways in which LLMs can leave a mark in the fields like fitness and wellbeing, urban planning, climate modelling and disaster response which could inspire future researches and applications in the said avenues.

Title: Consensus learning: A novel decentralised ensemble learning paradigm

• Authors: Horia Magureanu, Naïri Usher
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.16157
• Pdf URL: https://arxiv.org/pdf/2402.16157
• Copy Paste: [[2402.16157]] Consensus learning: A novel decentralised ensemble learning paradigm(https://arxiv.org/abs/2402.16157)
• Keywords: privacy, attack, robust
• Abstract: The widespread adoption of large-scale machine learning models in recent years highlights the need for distributed computing for efficiency and scalability. This work introduces a novel distributed machine learning paradigm -- \emph{consensus learning} -- which combines classical ensemble methods with consensus protocols deployed in peer-to-peer systems. These algorithms consist of two phases: first, participants develop their models and submit predictions for any new data inputs; second, the individual predictions are used as inputs for a communication phase, which is governed by a consensus protocol. Consensus learning ensures user data privacy, while also inheriting the safety measures against Byzantine attacks from the underlying consensus mechanism. We provide a detailed theoretical analysis for a particular consensus protocol and compare the performance of the consensus learning ensemble with centralised ensemble learning algorithms. The discussion is supplemented by various numerical simulations, which describe the robustness of the algorithms against Byzantine participants.

Title: DistALANER: Distantly Supervised Active Learning Augmented Named Entity Recognition in the Open Source Software Ecosystem

• Authors: Somnath Banerjee, Avik Dutta, Aaditya Agrawal, Rima Hazra, Animesh Mukherjee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16159
• Pdf URL: https://arxiv.org/pdf/2402.16159
• Copy Paste: [[2402.16159]] DistALANER: Distantly Supervised Active Learning Augmented Named Entity Recognition in the Open Source Software Ecosystem(https://arxiv.org/abs/2402.16159)
• Keywords: extraction
• Abstract: This paper proposes a novel named entity recognition (NER) technique specifically tailored for the open-source software systems. Our approach aims to address the scarcity of annotated software data by employing a comprehensive two-step distantly supervised annotation process. This process strategically leverages language heuristics, unique lookup tables, external knowledge sources, and an active learning approach. By harnessing these powerful techniques, we not only enhance model performance but also effectively mitigate the limitations associated with cost and the scarcity of expert annotators. It is noteworthy that our framework significantly outperforms the state-of-the-art LLMs by a substantial margin. We also show the effectiveness of NER in the downstream task of relation extraction.

Title: Task Specific Pretraining with Noisy Labels for Remote sensing Image Segmentation

• Authors: Chenying Liu, Conrad Albrecht, Yi Wang, Xiao Xiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16164
• Pdf URL: https://arxiv.org/pdf/2402.16164
• Copy Paste: [[2402.16164]] Task Specific Pretraining with Noisy Labels for Remote sensing Image Segmentation(https://arxiv.org/abs/2402.16164)
• Keywords: robust, segmentation
• Abstract: In recent years, self-supervision has drawn a lot of attention in remote sensing society due to its ability to reduce the demand of exact labels in supervised deep learning model training. Self-supervision methods generally utilize image-level information to pretrain models in an unsupervised fashion. Though these pretrained encoders show effectiveness in many downstream tasks, their performance on segmentation tasks is often not as good as that on classification tasks. On the other hand, many easily available label sources (e.g., automatic labeling tools and land cover land use products) exist, which can provide a large amount of noisy labels for segmentation model training. In this work, we propose to explore the under-exploited potential of noisy labels for segmentation task specific pretraining, and exam its robustness when confronted with mismatched categories and different decoders during fine-tuning. Specifically, we inspect the impacts of noisy labels on different layers in supervised model training to serve as the basis of our work. Experiments on two datasets indicate the effectiveness of task specific supervised pretraining with noisy labels. The findings are expected to shed light on new avenues for improving the accuracy and versatility of pretraining strategies for remote sensing image segmentation.

Title: XAI-based gait analysis of patients walking with Knee-Ankle-Foot orthosis using video cameras

• Authors: Arnav Mishra, Aditi Shetkar, Ganesh M. Bapat, Rajdeep Ojha, Tanmay Tulsidas Verlekar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16175
• Pdf URL: https://arxiv.org/pdf/2402.16175
• Copy Paste: [[2402.16175]] XAI-based gait analysis of patients walking with Knee-Ankle-Foot orthosis using video cameras(https://arxiv.org/abs/2402.16175)
• Keywords: robust, explainability
• Abstract: Recent technological advancements in artificial intelligence and computer vision have enabled gait analysis on portable devices such as cell phones. However, most state-of-the-art vision-based systems still impose numerous constraints for capturing a patient's video, such as using a static camera and maintaining a specific distance from it. While these constraints are manageable under professional observation, they pose challenges in home settings. Another issue with most vision-based systems is their output, typically a classification label and confidence value, whose reliability is often questioned by medical professionals. This paper addresses these challenges by presenting a novel system for gait analysis robust to camera movements and providing explanations for its output. The study utilizes a dataset comprising videos of subjects wearing two types of Knee Ankle Foot Orthosis (KAFO), namely "Locked Knee" and "Semi-flexion," for mobility, along with metadata and ground truth for explanations. The ground truth highlights the statistical significance of seven features captured using motion capture systems to differentiate between the two gaits. To address camera movement challenges, the proposed system employs super-resolution and pose estimation during pre-processing. It then identifies the seven features - Stride Length, Step Length and Duration of single support of orthotic and non-orthotic leg, Cadence, and Speed - using the skeletal output of pose estimation. These features train a multi-layer perceptron, with its output explained by highlighting the features' contribution to classification. While most state-of-the-art systems struggle with processing the video or training on the proposed dataset, our system achieves an average accuracy of 94%. The model's explainability is validated using ground truth and can be considered reliable.

Title: How Can LLM Guide RL? A Value-Based Approach

• Authors: Shenao Zhang, Sirui Zheng, Shuqi Ke, Zhihan Liu, Wanxin Jin, Jianbo Yuan, Yingxiang Yang, Hongxia Yang, Zhaoran Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16181
• Pdf URL: https://arxiv.org/pdf/2402.16181
• Copy Paste: [[2402.16181]] How Can LLM Guide RL? A Value-Based Approach(https://arxiv.org/abs/2402.16181)
• Keywords: large language model
• Abstract: Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. However, RL algorithms may require extensive trial-and-error interactions to collect useful feedback for improvement. On the other hand, recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities for planning tasks, lacking the ability to autonomously refine their responses based on feedback. Therefore, in this paper, we study how the policy prior provided by the LLM can enhance the sample efficiency of RL algorithms. Specifically, we develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning, particularly when the difference between the ideal policy and the LLM-informed policy is small, which suggests that the initial policy is close to optimal, reducing the need for further exploration. Additionally, we present a practical algorithm SLINVIT that simplifies the construction of the value function and employs subgoals to reduce the search complexity. Our experiments across three interactive environments ALFWorld, InterCode, and BlocksWorld demonstrate that our method achieves state-of-the-art success rates and also surpasses previous RL and LLM approaches in terms of sample efficiency. Our code is available at https://github.com/agentification/Language-Integrated-VI.

Title: Attacking LLM Watermarks by Exploiting Their Strengths

• Authors: Qi Pang, Shengyuan Hu, Wenting Zheng, Virginia Smith
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16187
• Pdf URL: https://arxiv.org/pdf/2402.16187
• Copy Paste: [[2402.16187]] Attacking LLM Watermarks by Exploiting Their Strengths(https://arxiv.org/abs/2402.16187)
• Keywords: defense, attack, robust, watermark, generative
• Abstract: Advances in generative models have made it possible for AI-generated text, code, and images to mirror human-generated content in many applications. Watermarking, a technique that aims to embed information in the output of a model to verify its source, is useful for mitigating misuse of such AI-generated content. However, existing watermarking schemes remain surprisingly susceptible to attack. In particular, we show that desirable properties shared by existing LLM watermarking systems such as quality preservation, robustness, and public detection APIs can in turn make these systems vulnerable to various attacks. We rigorously study potential attacks in terms of common watermark design choices, and propose best practices and defenses for mitigation -- establishing a set of practical guidelines for embedding and detection of LLM watermarks.

Title: ARIN: Adaptive Resampling and Instance Normalization for Robust Blind Inpainting of Dunhuang Cave Paintings

• Authors: Alexander Schmidt, Prathmesh Madhu, Andreas Maier, Vincent Christlein, Ronak Kosti
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16188
• Pdf URL: https://arxiv.org/pdf/2402.16188
• Copy Paste: [[2402.16188]] ARIN: Adaptive Resampling and Instance Normalization for Robust Blind Inpainting of Dunhuang Cave Paintings(https://arxiv.org/abs/2402.16188)
• Keywords: robust
• Abstract: Image enhancement algorithms are very useful for real world computer vision tasks where image resolution is often physically limited by the sensor size. While state-of-the-art deep neural networks show impressive results for image enhancement, they often struggle to enhance real-world images. In this work, we tackle a real-world setting: inpainting of images from Dunhuang caves. The Dunhuang dataset consists of murals, half of which suffer from corrosion and aging. These murals feature a range of rich content, such as Buddha statues, bodhisattvas, sponsors, architecture, dance, music, and decorative patterns designed by different artists spanning ten centuries, which makes manual restoration challenging. We modify two different existing methods (CAR, HINet) that are based upon state-of-the-art (SOTA) super resolution and deblurring networks. We show that those can successfully inpaint and enhance these deteriorated cave paintings. We further show that a novel combination of CAR and HINet, resulting in our proposed inpainting network (ARIN), is very robust to external noise, especially Gaussian noise. To this end, we present a quantitative and qualitative comparison of our proposed approach with existing SOTA networks and winners of the Dunhuang challenge. One of the proposed methods HINet) represents the new state of the art and outperforms the 1st place of the Dunhuang Challenge, while our combination ARIN, which is robust to noise, is comparable to the 1st place. We also present and discuss qualitative results showing the impact of our method for inpainting on Dunhuang cave images.

Title: One-stage Prompt-based Continual Learning

• Authors: Youngeun Kim, Yuhang Li, Priyadarshini Panda
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16189
• Pdf URL: https://arxiv.org/pdf/2402.16189
• Copy Paste: [[2402.16189]] One-stage Prompt-based Continual Learning(https://arxiv.org/abs/2402.16189)
• Keywords: privacy, transformer
• Abstract: Prompt-based Continual Learning (PCL) has gained considerable attention as a promising continual learning solution as it achieves state-of-the-art performance while preventing privacy violation and memory overhead issues. Nonetheless, existing PCL approaches face significant computational burdens because of two Vision Transformer (ViT) feed-forward stages; one is for the query ViT that generates a prompt query to select prompts inside a prompt pool; the other one is a backbone ViT that mixes information between selected prompts and image tokens. To address this, we introduce a one-stage PCL framework by directly using the intermediate layer's token embedding as a prompt query. This design removes the need for an additional feed-forward stage for query ViT, resulting in ~50% computational cost reduction for both training and inference with marginal accuracy drop < 1%. We further introduce a Query-Pool Regularization (QR) loss that regulates the relationship between the prompt query and the prompt pool to improve representation power. The QR loss is only applied during training time, so there is no computational overhead at inference from the QR loss. With the QR loss, our approach maintains ~ 50% computational cost reduction during inference as well as outperforms the prior two-stage PCL methods by ~1.4% on public class-incremental continual learning benchmarks including CIFAR-100, ImageNet-R, and DomainNet.

Title: Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing

• Authors: Jiabao Ji, Bairu Hou, Alexander Robey, George J. Pappas, Hamed Hassani, Yang Zhang, Eric Wong, Shiyu Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16192
• Pdf URL: https://arxiv.org/pdf/2402.16192
• Copy Paste: [[2402.16192]] Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing(https://arxiv.org/abs/2402.16192)
• Keywords: defense, attack, robust, large language model
• Abstract: Aligned large language models (LLMs) are vulnerable to jailbreaking attacks, which bypass the safeguards of targeted LLMs and fool them into generating objectionable content. While initial defenses show promise against token-based threat models, there do not exist defenses that provide robustness against semantic attacks and avoid unfavorable trade-offs between robustness and nominal performance. To meet this need, we propose SEMANTICSMOOTH, a smoothing-based defense that aggregates the predictions of multiple semantically transformed copies of a given input prompt. Experimental results demonstrate that SEMANTICSMOOTH achieves state-of-the-art robustness against GCG, PAIR, and AutoDAN attacks while maintaining strong nominal performance on instruction following benchmarks such as InstructionFollowing and AlpacaEval. The codes will be publicly available at https://github.com/UCSB-NLP-Chang/SemanticSmooth.

Title: HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs

• Authors: Cem Uluoglakci, Tugba Taskaya Temizel (Middle East Technical University)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16211
• Pdf URL: https://arxiv.org/pdf/2402.16211
• Copy Paste: [[2402.16211]] HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs(https://arxiv.org/abs/2402.16211)
• Keywords: large language model
• Abstract: Hallucinations pose a significant challenge to the reliability and alignment of Large Language Models (LLMs), limiting their widespread acceptance beyond chatbot applications. Despite ongoing efforts, hallucinations remain a prevalent challenge in LLMs. The detection of hallucinations itself is also a formidable task, frequently requiring manual labeling or constrained evaluations. This paper introduces an automated scalable framework that combines benchmarking LLMs' hallucination tendencies with efficient hallucination detection. We leverage LLMs to generate challenging tasks related to hypothetical phenomena, subsequently employing them as agents for efficient hallucination detection. The framework is domain-agnostic, allowing the use of any language model for benchmark creation or evaluation in any domain. We introduce the publicly available HypoTermQA Benchmarking Dataset, on which state-of-the-art models' performance ranged between 3% and 11%, and evaluator agents demonstrated a 6% error rate in hallucination prediction. The proposed framework provides opportunities to test and improve LLMs. Additionally, it has the potential to generate benchmarking datasets tailored to specific domains, such as law, health, and finance.

Title: GARNN: An Interpretable Graph Attentive Recurrent Neural Network for Predicting Blood Glucose Levels via Multivariate Time Series

• Authors: Chengzhe Piao, Taiyu Zhu, Stephanie E Baldeweg, Paul Taylor, Pantelis Georgiou, Jiahao Sun, Jun Wang, Kezhi Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16230
• Pdf URL: https://arxiv.org/pdf/2402.16230
• Copy Paste: [[2402.16230]] GARNN: An Interpretable Graph Attentive Recurrent Neural Network for Predicting Blood Glucose Levels via Multivariate Time Series(https://arxiv.org/abs/2402.16230)
• Keywords: robust, interpretability
• Abstract: Accurate prediction of future blood glucose (BG) levels can effectively improve BG management for people living with diabetes, thereby reducing complications and improving quality of life. The state of the art of BG prediction has been achieved by leveraging advanced deep learning methods to model multi-modal data, i.e., sensor data and self-reported event data, organised as multi-variate time series (MTS). However, these methods are mostly regarded as ``black boxes'' and not entirely trusted by clinicians and patients. In this paper, we propose interpretable graph attentive recurrent neural networks (GARNNs) to model MTS, explaining variable contributions via summarizing variable importance and generating feature maps by graph attention mechanisms instead of post-hoc analysis. We evaluate GARNNs on four datasets, representing diverse clinical scenarios. Upon comparison with twelve well-established baseline methods, GARNNs not only achieve the best prediction accuracy but also provide high-quality temporal interpretability, in particular for postprandial glucose levels as a result of corresponding meal intake and insulin injection. These findings underline the potential of GARNN as a robust tool for improving diabetes care, bridging the gap between deep learning technology and real-world healthcare solutions.

Title: Learning Translations: Emergent Communication Pretraining for Cooperative Language Acquisition

• Authors: Dylan Cope, Peter McBurney
• Subjects: cs.LG, cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2402.16247
• Pdf URL: https://arxiv.org/pdf/2402.16247
• Copy Paste: [[2402.16247]] Learning Translations: Emergent Communication Pretraining for Cooperative Language Acquisition(https://arxiv.org/abs/2402.16247)
• Keywords: robust
• Abstract: In Emergent Communication (EC) agents learn to communicate with one another, but the protocols that they develop are specialised to their training community. This observation led to research into Zero-Shot Coordination (ZSC) for learning communication strategies that are robust to agents not encountered during training. However, ZSC typically assumes that no prior data is available about the agents that will be encountered in the zero-shot setting. In many cases, this presents an unnecessarily hard problem and rules out communication via preestablished conventions. We propose a novel AI challenge called a Cooperative Language Acquisition Problem (CLAP) in which the ZSC assumptions are relaxed by allowing a 'joiner' agent to learn from a dataset of interactions between agents in a target community. We propose and compare two methods for solving CLAPs: Imitation Learning (IL), and Emergent Communication pretraining and Translation Learning (ECTL), in which an agent is trained in self-play with EC and then learns from the data to translate between the emergent protocol and the target community's protocol.

Title: SeqTrack3D: Exploring Sequence Information for Robust 3D Point Cloud Tracking

• Authors: Yu Lin, Zhiheng Li, Yubo Cui, Zheng Fang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16249
• Pdf URL: https://arxiv.org/pdf/2402.16249
• Copy Paste: [[2402.16249]] SeqTrack3D: Exploring Sequence Information for Robust 3D Point Cloud Tracking(https://arxiv.org/abs/2402.16249)
• Keywords: robust
• Abstract: 3D single object tracking (SOT) is an important and challenging task for the autonomous driving and mobile robotics. Most existing methods perform tracking between two consecutive frames while ignoring the motion patterns of the target over a series of frames, which would cause performance degradation in the scenes with sparse points. To break through this limitation, we introduce Sequence-to-Sequence tracking paradigm and a tracker named SeqTrack3D to capture target motion across continuous frames. Unlike previous methods that primarily adopted three strategies: matching two consecutive point clouds, predicting relative motion, or utilizing sequential point clouds to address feature degradation, our SeqTrack3D combines both historical point clouds and bounding box sequences. This novel method ensures robust tracking by leveraging location priors from historical boxes, even in scenes with sparse points. Extensive experiments conducted on large-scale datasets show that SeqTrack3D achieves new state-of-the-art performances, improving by 6.00% on NuScenes and 14.13% on Waymo dataset. The code will be made public at https://github.com/aron-lin/seqtrack3d.

Title: Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Models

• Authors: Jinqian Chen, Jihua Zhu, Qinghai Zheng, Zhongyu Li, Zhiqiang Tian
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16255
• Pdf URL: https://arxiv.org/pdf/2402.16255
• Copy Paste: [[2402.16255]] Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Models(https://arxiv.org/abs/2402.16255)
• Keywords: federate
• Abstract: Federated learning encounters substantial challenges with heterogeneous data, leading to performance degradation and convergence issues. While considerable progress has been achieved in mitigating such an impact, the reliability aspect of federated models has been largely disregarded. In this study, we conduct extensive experiments to investigate the reliability of both generic and personalized federated models. Our exploration uncovers a significant finding: \textbf{federated models exhibit unreliability when faced with heterogeneous data}, demonstrating poor calibration on in-distribution test data and low uncertainty levels on out-of-distribution data. This unreliability is primarily attributed to the presence of biased projection heads, which introduce miscalibration into the federated models. Inspired by this observation, we propose the "Assembled Projection Heads" (APH) method for enhancing the reliability of federated models. By treating the existing projection head parameters as priors, APH randomly samples multiple initialized parameters of projection heads from the prior and further performs targeted fine-tuning on locally available data under varying learning rates. Such a head ensemble introduces parameter diversity into the deterministic model, eliminating the bias and producing reliable predictions via head averaging. We evaluate the effectiveness of the proposed APH method across three prominent federated benchmarks. Experimental results validate the efficacy of APH in model calibration and uncertainty estimation. Notably, APH can be seamlessly integrated into various federated approaches but only requires less than 30\% additional computation cost for 100$\times$ inferences within large models.

Title: Foundation Model Transparency Reports

• Authors: Rishi Bommasani, Kevin Klyman, Shayne Longpre, Betty Xiong, Sayash Kapoor, Nestor Maslej, Arvind Narayanan, Percy Liang
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.16268
• Pdf URL: https://arxiv.org/pdf/2402.16268
• Copy Paste: [[2402.16268]] Foundation Model Transparency Reports(https://arxiv.org/abs/2402.16268)
• Keywords: secure
• Abstract: Foundation models are critical digital technologies with sweeping societal impact that necessitates transparency. To codify how foundation model developers should provide transparency about the development and deployment of their models, we propose Foundation Model Transparency Reports, drawing upon the transparency reporting practices in social media. While external documentation of societal harms prompted social media transparency reports, our objective is to institutionalize transparency reporting for foundation models while the industry is still nascent. To design our reports, we identify 6 design principles given the successes and shortcomings of social media transparency reporting. To further schematize our reports, we draw upon the 100 transparency indicators from the Foundation Model Transparency Index. Given these indicators, we measure the extent to which they overlap with the transparency requirements included in six prominent government policies (e.g., the EU AI Act, the US Executive Order on Safe, Secure, and Trustworthy AI). Well-designed transparency reports could reduce compliance costs, in part due to overlapping regulatory requirements across different jurisdictions. We encourage foundation model developers to regularly publish transparency reports, building upon recommendations from the G7 and the White House.

Title: Few-Shot Learning for Annotation-Efficient Nucleus Instance Segmentation

• Authors: Yu Ming, Zihao Wu, Jie Yang, Danyi Li, Yuan Gao, Changxin Gao, Gui-Song Xia, Yuanqing Li, Li Liang, Jin-Gang Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16280
• Pdf URL: https://arxiv.org/pdf/2402.16280
• Copy Paste: [[2402.16280]] Few-Shot Learning for Annotation-Efficient Nucleus Instance Segmentation(https://arxiv.org/abs/2402.16280)
• Keywords: generative, segmentation
• Abstract: Nucleus instance segmentation from histopathology images suffers from the extremely laborious and expert-dependent annotation of nucleus instances. As a promising solution to this task, annotation-efficient deep learning paradigms have recently attracted much research interest, such as weakly-/semi-supervised learning, generative adversarial learning, etc. In this paper, we propose to formulate annotation-efficient nucleus instance segmentation from the perspective of few-shot learning (FSL). Our work was motivated by that, with the prosperity of computational pathology, an increasing number of fully-annotated datasets are publicly accessible, and we hope to leverage these external datasets to assist nucleus instance segmentation on the target dataset which only has very limited annotation. To achieve this goal, we adopt the meta-learning based FSL paradigm, which however has to be tailored in two substantial aspects before adapting to our task. First, since the novel classes may be inconsistent with those of the external dataset, we extend the basic definition of few-shot instance segmentation (FSIS) to generalized few-shot instance segmentation (GFSIS). Second, to cope with the intrinsic challenges of nucleus segmentation, including touching between adjacent cells, cellular heterogeneity, etc., we further introduce a structural guidance mechanism into the GFSIS network, finally leading to a unified Structurally-Guided Generalized Few-Shot Instance Segmentation (SGFSIS) framework. Extensive experiments on a couple of publicly accessible datasets demonstrate that, SGFSIS can outperform other annotation-efficient learning baselines, including semi-supervised learning, simple transfer learning, etc., with comparable performance to fully supervised learning with less than 5% annotations.

Title: PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering

• Authors: Yiming Du, Hongru Wang, Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong Wang, Kam-Fai Wong
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.16288
• Pdf URL: https://arxiv.org/pdf/2402.16288
• Copy Paste: [[2402.16288]] PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering(https://arxiv.org/abs/2402.16288)
• Keywords: large language model
• Abstract: Long-term memory plays a critical role in personal interaction, considering long-term memory can better leverage world knowledge, historical information, and preferences in dialogues. Our research introduces PerLTQA, an innovative QA dataset that combines semantic and episodic memories, including world knowledge, profiles, social relationships, events, and dialogues. This dataset is collected to investigate the use of personalized memories, focusing on social interactions and events in the QA task. PerLTQA features two types of memory and a comprehensive benchmark of 8,593 questions for 30 characters, facilitating the exploration and application of personalized memories in Large Language Models (LLMs). Based on PerLTQA, we propose a novel framework for memory integration and generation, consisting of three main components: Memory Classification, Memory Retrieval, and Memory Synthesis. We evaluate this framework using five LLMs and three retrievers. Experimental results demonstrate that BERT-based classification models significantly outperform LLMs such as ChatGLM3 and ChatGPT in the memory classification task. Furthermore, our study highlights the importance of effective memory integration in the QA task.

Title: Card-Based Overwriting Protocol for Equality Function and Applications

• Authors: Suthee Ruangwises, Tomoki Ono, Yoshiki Abe, Kyosuke Hatsugai, Mitsugu Iwamoto
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16290
• Pdf URL: https://arxiv.org/pdf/2402.16290
• Copy Paste: [[2402.16290]] Card-Based Overwriting Protocol for Equality Function and Applications(https://arxiv.org/abs/2402.16290)
• Keywords: secure
• Abstract: Research in the area of secure multi-party computation with an unconventional method of using a physical deck of playing cards began in 1989 when den Boar proposed a protocol to compute the logical AND function using five cards. Since then, the area has gained interest from many researchers and several card-based protocols to compute various functions have been developed. In this paper, we propose a card-based protocol called the overwriting protocol that can securely compute the $k$-candidate $n$-variable equality function $f: \{0,1,\ldots ,k-1\}^n \rightarrow \{0,1\}$. We also apply the technique used in this protocol to compute other similar functions.

Title: Decentralized Federated Unlearning on Blockchain

• Authors: Xiao Liu, Mingyuan Li, Xu Wang, Guangsheng Yu, Wei Ni, Lixiang Li, Haipeng Peng, Renping Liu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16294
• Pdf URL: https://arxiv.org/pdf/2402.16294
• Copy Paste: [[2402.16294]] Decentralized Federated Unlearning on Blockchain(https://arxiv.org/abs/2402.16294)
• Keywords: federate
• Abstract: Blockchained Federated Learning (FL) has been gaining traction for ensuring the integrity and traceability of FL processes. Blockchained FL involves participants training models locally with their data and subsequently publishing the models on the blockchain, forming a Directed Acyclic Graph (DAG)-like inheritance structure that represents the model relationship. However, this particular DAG-based structure presents challenges in updating models with sensitive data, due to the complexity and overhead involved. To address this, we propose Blockchained Federated Unlearning (BlockFUL), a generic framework that redesigns the blockchain structure using Chameleon Hash (CH) technology to mitigate the complexity of model updating, thereby reducing the computational and consensus costs of unlearning tasks.Furthermore, BlockFUL supports various federated unlearning methods, ensuring the integrity and traceability of model updates, whether conducted in parallel or serial. We conduct a comprehensive study of two typical unlearning methods, gradient ascent and re-training, demonstrating the efficient unlearning workflow in these two categories with minimal CH and block update operations. Additionally, we compare the computation and communication costs of these methods.

Title: MV-Swin-T: Mammogram Classification with Multi-view Swin Transformer

• Authors: Sushmita Sarker, Prithul Sarker, George Bebis, Alireza Tavakkoli
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16298
• Pdf URL: https://arxiv.org/pdf/2402.16298
• Copy Paste: [[2402.16298]] MV-Swin-T: Mammogram Classification with Multi-view Swin Transformer(https://arxiv.org/abs/2402.16298)
• Keywords: transformer
• Abstract: Traditional deep learning approaches for breast cancer classification has predominantly concentrated on single-view analysis. In clinical practice, however, radiologists concurrently examine all views within a mammography exam, leveraging the inherent correlations in these views to effectively detect tumors. Acknowledging the significance of multi-view analysis, some studies have introduced methods that independently process mammogram views, either through distinct convolutional branches or simple fusion strategies, inadvertently leading to a loss of crucial inter-view correlations. In this paper, we propose an innovative multi-view network exclusively based on transformers to address challenges in mammographic image classification. Our approach introduces a novel shifted window-based dynamic attention block, facilitating the effective integration of multi-view information and promoting the coherent transfer of this information between views at the spatial feature map level. Furthermore, we conduct a comprehensive comparative analysis of the performance and effectiveness of transformer-based models under diverse settings, employing the CBIS-DDSM and Vin-Dr Mammo datasets. Our code is publicly available at https://github.com/prithuls/MV-Swin-T

Title: Conformalized Selective Regression

• Authors: Anna Sokol, Nuno Moniz, Nitesh Chawla
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.16300
• Pdf URL: https://arxiv.org/pdf/2402.16300
• Copy Paste: [[2402.16300]] Conformalized Selective Regression(https://arxiv.org/abs/2402.16300)
• Keywords: fair
• Abstract: Should prediction models always deliver a prediction? In the pursuit of maximum predictive performance, critical considerations of reliability and fairness are often overshadowed, particularly when it comes to the role of uncertainty. Selective regression, also known as the "reject option," allows models to abstain from predictions in cases of considerable uncertainty. Initially proposed seven decades ago, approaches to selective regression have mostly focused on distribution-based proxies for measuring uncertainty, particularly conditional variance. However, this focus neglects the significant influence of model-specific biases on a model's performance. In this paper, we propose a novel approach to selective regression by leveraging conformal prediction, which provides grounded confidence measures for individual predictions based on model-specific biases. In addition, we propose a standardized evaluation framework to allow proper comparison of selective regression approaches. Via an extensive experimental approach, we demonstrate how our proposed approach, conformalized selective regression, demonstrates an advantage over multiple state-of-the-art baselines.

Title: Graph Diffusion Policy Optimization

• Authors: Yijing Liu, Chao Du, Tianyu Pang, Chongxuan Li, Wei Chen, Min Lin
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2402.16302
• Pdf URL: https://arxiv.org/pdf/2402.16302
• Copy Paste: [[2402.16302]] Graph Diffusion Policy Optimization(https://arxiv.org/abs/2402.16302)
• Keywords: diffusion
• Abstract: Recent research has made significant progress in optimizing diffusion models for specific downstream objectives, which is an important pursuit in fields such as graph generation for drug design. However, directly applying these models to graph diffusion presents challenges, resulting in suboptimal performance. This paper introduces graph diffusion policy optimization (GDPO), a novel approach to optimize graph diffusion models for arbitrary (e.g., non-differentiable) objectives using reinforcement learning. GDPO is based on an eager policy gradient tailored for graph diffusion models, developed through meticulous analysis and promising improved performance. Experimental results show that GDPO achieves state-of-the-art performance in various graph generation tasks with complex and diverse objectives. Code is available at https://github.com/sail-sg/GDPO.

Title: Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion

• Authors: Xuantong Liu, Tianyang Hu, Wenjia Wang, Kenji Kawaguchi, Yuan Yao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16305
• Pdf URL: https://arxiv.org/pdf/2402.16305
• Copy Paste: [[2402.16305]] Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion(https://arxiv.org/abs/2402.16305)
• Keywords: diffusion
• Abstract: As a dominant force in text-to-image generation tasks, Diffusion Probabilistic Models (DPMs) face a critical challenge in controllability, struggling to adhere strictly to complex, multi-faceted instructions. In this work, we aim to address this alignment challenge for conditional generation tasks. First, we provide an alternative view of state-of-the-art DPMs as a way of inverting advanced Vision-Language Models (VLMs). With this formulation, we naturally propose a training-free approach that bypasses the conventional sampling process associated with DPMs. By directly optimizing images with the supervision of discriminative VLMs, the proposed method can potentially achieve a better text-image alignment. As proof of concept, we demonstrate the pipeline with the pre-trained BLIP-2 model and identify several key designs for improved image generation. To further enhance the image fidelity, a Score Distillation Sampling module of Stable Diffusion is incorporated. By carefully balancing the two components during optimization, our method can produce high-quality images with near state-of-the-art performance on T2I-Compbench.

Title: Cross-domain Chinese Sentence Pattern Parsing

• Authors: Yingsi Yu, Cunliang Kong, Liner Yang, Meishan Zhang, Lin Zhu, Yujie Wang, Haozhe Lin, Maosong Sun, Erhong Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16311
• Pdf URL: https://arxiv.org/pdf/2402.16311
• Copy Paste: [[2402.16311]] Cross-domain Chinese Sentence Pattern Parsing(https://arxiv.org/abs/2402.16311)
• Keywords: large language model
• Abstract: Sentence Pattern Structure (SPS) parsing is a syntactic analysis method primarily employed in language teaching.Existing SPS parsers rely heavily on textbook corpora for training, lacking cross-domain capability.To overcome this constraint, this paper proposes an innovative approach leveraging large language models (LLMs) within a self-training framework. Partial syntactic rules from a source domain are combined with target domain sentences to dynamically generate training data, enhancing the adaptability of the parser to diverse domains.Experiments conducted on textbook and news domains demonstrate the effectiveness of the proposed method, outperforming rule-based baselines by 1.68 points on F1 metrics.

Title: Federated Contextual Cascading Bandits with Asynchronous Communication and Heterogeneous Users

• Authors: Hantao Yang, Xutong Liu, Zhiyong Wang, Hong Xie, John C. S. Lui, Defu Lian, Enhong Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16312
• Pdf URL: https://arxiv.org/pdf/2402.16312
• Copy Paste: [[2402.16312]] Federated Contextual Cascading Bandits with Asynchronous Communication and Heterogeneous Users(https://arxiv.org/abs/2402.16312)
• Keywords: federate
• Abstract: We study the problem of federated contextual combinatorial cascading bandits, where $|\mathcal{U}|$ agents collaborate under the coordination of a central server to provide tailored recommendations to the $|\mathcal{U}|$ corresponding users. Existing works consider either a synchronous framework, necessitating full agent participation and global synchronization, or assume user homogeneity with identical behaviors. We overcome these limitations by considering (1) federated agents operating in an asynchronous communication paradigm, where no mandatory synchronization is required and all agents communicate independently with the server, (2) heterogeneous user behaviors, where users can be stratified into $J \le |\mathcal{U}|$ latent user clusters, each exhibiting distinct preferences. For this setting, we propose a UCB-type algorithm with delicate communication protocols. Through theoretical analysis, we give sub-linear regret bounds on par with those achieved in the synchronous framework, while incurring only logarithmic communication costs. Empirical evaluation on synthetic and real-world datasets validates our algorithm's superior performance in terms of regrets and communication costs.

Title: Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answering

• Authors: Mingxu Tao, Dongyan Zhao, Yansong Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16313
• Pdf URL: https://arxiv.org/pdf/2402.16313
• Copy Paste: [[2402.16313]] Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answering(https://arxiv.org/abs/2402.16313)
• Keywords: large language model
• Abstract: Open-ended question answering requires models to find appropriate evidence to form well-reasoned, comprehensive and helpful answers. In practical applications, models also need to engage in extended discussions on potential scenarios closely relevant to the question. With augmentation of retrieval module, open-source Large Language Models (LLMs) can produce coherent answers often with different focuses, but are still sub-optimal in terms of reliable evidence selection and in-depth question analysis. In this paper, we propose a novel Chain-of-Discussion framework to leverage the synergy among multiple open-source LLMs aiming to provide \textbf{more correct} and \textbf{more comprehensive} answers for open-ended QA, although they are not strong enough individually. Our experiments show that discussions among multiple LLMs play a vital role in enhancing the quality of answers. We release our data and code at \url{https://github.com/kobayashikanna01/Chain-of-Discussion}.

Title: Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models

• Authors: Jeonghwan Kim, Heng Ji
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16315
• Pdf URL: https://arxiv.org/pdf/2402.16315
• Copy Paste: [[2402.16315]] Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models(https://arxiv.org/abs/2402.16315)
• Keywords: explainability, large language model
• Abstract: Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. While such capability is largely attributed to the rich world knowledge contained within the Large Language Models (LLMs), our work reveals their shortcomings in fine-grained visual categorization (FGVC) across six different benchmark settings. Most recent state-of-the-art LVLMs like LLaVa-1.5, InstructBLIP and GPT-4V not only severely deteriorate in terms of classification performance, e.g., average drop of 65.58 in EM for Stanford Dogs for LLaVA-1.5, but also struggle to generate an accurate explanation with detailed attributes based on the concept that appears within an input image despite their capability to generate holistic image-level descriptions. In-depth analyses show that instruction-tuned LVLMs exhibit modality gap, showing discrepancy when given textual and visual inputs that correspond to the same concept, preventing the image modality from leveraging the rich parametric knowledge within the LLMs. In an effort to further the community's endeavor in this direction, we propose a multiple granularity attribute-centric evaluation benchmark, Finer, which aims to establish a ground to evaluate LVLMs' fine-grained visual comprehension ability and provide significantly improved explainability.

Title: Gradient-Guided Modality Decoupling for Missing-Modality Robustness

• Authors: Hao Wang, Shengda Luo, Guosheng Hu, Jianguo Zhang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.16318
• Pdf URL: https://arxiv.org/pdf/2402.16318
• Copy Paste: [[2402.16318]] Gradient-Guided Modality Decoupling for Missing-Modality Robustness(https://arxiv.org/abs/2402.16318)
• Keywords: robust, segmentation
• Abstract: Multimodal learning with incomplete input data (missing modality) is practical and challenging. In this work, we conduct an in-depth analysis of this challenge and find that modality dominance has a significant negative impact on the model training, greatly degrading the missing modality performance. Motivated by Grad-CAM, we introduce a novel indicator, gradients, to monitor and reduce modality dominance which widely exists in the missing-modality scenario. In aid of this indicator, we present a novel Gradient-guided Modality Decoupling (GMD) method to decouple the dependency on dominating modalities. Specifically, GMD removes the conflicted gradient components from different modalities to achieve this decoupling, significantly improving the performance. In addition, to flexibly handle modal-incomplete data, we design a parameter-efficient Dynamic Sharing (DS) framework which can adaptively switch on/off the network parameters based on whether one modality is available. We conduct extensive experiments on three popular multimodal benchmarks, including BraTS 2018 for medical segmentation, CMU-MOSI, and CMU-MOSEI for sentiment analysis. The results show that our method can significantly outperform the competitors, showing the effectiveness of the proposed solutions. Our code is released here: https://github.com/HaoWang420/Gradient-guided-Modality-Decoupling.

Title: Data-freeWeight Compress and Denoise for Large Language Models

• Authors: Runyu Peng, Yunhua Zhou, Qipeng Guo, Yang Gao, Hang Yan, Xipeng Qiu, Dahua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16319
• Pdf URL: https://arxiv.org/pdf/2402.16319
• Copy Paste: [[2402.16319]] Data-freeWeight Compress and Denoise for Large Language Models(https://arxiv.org/abs/2402.16319)
• Keywords: data-free, large language model
• Abstract: Large Language Models (LLMs) are reshaping the research landscape in artificial intelligence, particularly as model parameters scale up significantly, unlocking remarkable capabilities across various domains. Nevertheless, the scalability of model parameters faces constraints due to limitations in GPU memory and computational speed. To address these constraints, various weight compression methods have emerged, such as Pruning and Quantization. Given the low-rank nature of weight matrices in language models, the reduction of weights through matrix decomposition undoubtedly holds significant potential and promise. In this paper, drawing upon the intrinsic structure of LLMs, we propose a novel approach termed Data-free Joint Rank-k Approximation for compressing the parameter matrices. Significantly, our method is characterized by without necessitating additional involvement of any corpus, while simultaneously preserving orthogonality in conjunction with pruning and quantization methods. We achieve a model pruning of 80% parameters while retaining 93.43% of the original performance without any calibration data. Additionally, we explore the fundamental properties of the weight matrix of LLMs undergone Rank-k Approximation and conduct comprehensive experiments to elucidate our hypothesis.

Title: BLO-SAM: Bi-level Optimization Based Overfitting-Preventing Finetuning of SAM

• Authors: Li Zhang, Youwei Liang, Pengtao Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16338
• Pdf URL: https://arxiv.org/pdf/2402.16338
• Copy Paste: [[2402.16338]] BLO-SAM: Bi-level Optimization Based Overfitting-Preventing Finetuning of SAM(https://arxiv.org/abs/2402.16338)
• Keywords: segmentation
• Abstract: The Segment Anything Model (SAM), a foundation model pretrained on millions of images and segmentation masks, has significantly advanced semantic segmentation, a fundamental task in computer vision. Despite its strengths, SAM encounters two major challenges. Firstly, it struggles with segmenting specific objects autonomously, as it relies on users to manually input prompts like points or bounding boxes to identify targeted objects. Secondly, SAM faces challenges in excelling at specific downstream tasks, like medical imaging, due to a disparity between the distribution of its pretraining data, which predominantly consists of general-domain images, and the data used in downstream tasks. Current solutions to these problems, which involve finetuning SAM, often lead to overfitting, a notable issue in scenarios with very limited data, like in medical imaging. To overcome these limitations, we introduce BLO-SAM, which finetunes SAM based on bi-level optimization (BLO). Our approach allows for automatic image segmentation without the need for manual prompts, by optimizing a learnable prompt embedding. Furthermore, it significantly reduces the risk of overfitting by training the model's weight parameters and the prompt embedding on two separate subsets of the training dataset, each at a different level of optimization. We apply BLO-SAM to diverse semantic segmentation tasks in general and medical domains. The results demonstrate BLO-SAM's superior performance over various state-of-the-art image semantic segmentation methods.

Title: Boosting Graph Pooling with Persistent Homology

• Authors: Chaolong Ying, Xinjian Zhao, Tianshu Yu
• Subjects: cs.LG, math.AT
• Abstract URL: https://arxiv.org/abs/2402.16346
• Pdf URL: https://arxiv.org/pdf/2402.16346
• Copy Paste: [[2402.16346]] Boosting Graph Pooling with Persistent Homology(https://arxiv.org/abs/2402.16346)
• Keywords: interpretability
• Abstract: Recently, there has been an emerging trend to integrate persistent homology (PH) into graph neural networks (GNNs) to enrich expressive power. However, naively plugging PH features into GNN layers always results in marginal improvement with low interpretability. In this paper, we investigate a novel mechanism for injecting global topological invariance into pooling layers using PH, motivated by the observation that filtration operation in PH naturally aligns graph pooling in a cut-off manner. In this fashion, message passing in the coarsened graph acts along persistent pooled topology, leading to improved performance. Experimentally, we apply our mechanism to a collection of graph pooling methods and observe consistent and substantial performance gain over several popular datasets, demonstrating its wide applicability and flexibility.

Title: CodeS: Towards Building Open-source Language Models for Text-to-SQL

• Authors: Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, Hong Chen
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2402.16347
• Pdf URL: https://arxiv.org/pdf/2402.16347
• Copy Paste: [[2402.16347]] CodeS: Towards Building Open-source Language Models for Text-to-SQL(https://arxiv.org/abs/2402.16347)
• Keywords: privacy, robust, large language model
• Abstract: Language models have shown promising performance on the task of translating natural language questions into SQL queries (Text-to-SQL). However, most of the state-of-the-art (SOTA) approaches rely on powerful yet closed-source large language models (LLMs), such as ChatGPT and GPT-4, which may have the limitations of unclear model architectures, data privacy risks, and expensive inference overheads. To address the limitations, we introduce CodeS, a series of pre-trained language models with parameters ranging from 1B to 15B, specifically designed for the text-to-SQL task. CodeS is a fully open-source language model, which achieves superior accuracy with much smaller parameter sizes. This paper studies the research challenges in building CodeS. To enhance the SQL generation abilities of CodeS, we adopt an incremental pre-training approach using a specifically curated SQL-centric corpus. Based on this, we address the challenges of schema linking and rapid domain adaptation through strategic prompt construction and a bi-directional data augmentation technique. We conduct comprehensive evaluations on multiple datasets, including the widely used Spider benchmark, the newly released BIRD benchmark, robustness-diagnostic benchmarks such as Spider-DK, Spider-Syn, Spider-Realistic, and Dr.Spider, as well as two real-world datasets created for financial and academic applications. The experimental results show that our CodeS achieves new SOTA accuracy and robustness on nearly all challenging text-to-SQL benchmarks.

Title: C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory

• Authors: Tianjiao Luo, Tim Pearce, Huayu Chen, Jianfei Chen, Jun Zhu
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.16349
• Pdf URL: https://arxiv.org/pdf/2402.16349
• Copy Paste: [[2402.16349]] C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory(https://arxiv.org/abs/2402.16349)
• Keywords: generative
• Abstract: Generative Adversarial Imitation Learning (GAIL) trains a generative policy to mimic a demonstrator. It uses on-policy Reinforcement Learning (RL) to optimize a reward signal derived from a GAN-like discriminator. A major drawback of GAIL is its training instability - it inherits the complex training dynamics of GANs, and the distribution shift introduced by RL. This can cause oscillations during training, harming its sample efficiency and final policy performance. Recent work has shown that control theory can help with the convergence of a GAN's training. This paper extends this line of work, conducting a control-theoretic analysis of GAIL and deriving a novel controller that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in a 'one-step' setting. Based on this, we propose a practical algorithm 'Controlled-GAIL' (C-GAIL). On MuJoCo tasks, our controlled variant is able to speed up the rate of convergence, reduce the range of oscillation and match the expert's distribution more closely both for vanilla GAIL and GAIL-DAC.

Title: Impression-CLIP: Contrastive Shape-Impression Embedding for Fonts

• Authors: Yugo Kubota, Daichi Haraguchi, Seiichi Uchida
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16350
• Pdf URL: https://arxiv.org/pdf/2402.16350
• Copy Paste: [[2402.16350]] Impression-CLIP: Contrastive Shape-Impression Embedding for Fonts(https://arxiv.org/abs/2402.16350)
• Keywords: robust
• Abstract: Fonts convey different impressions to readers. These impressions often come from the font shapes. However, the correlation between fonts and their impression is weak and unstable because impressions are subjective. To capture such weak and unstable cross-modal correlation between font shapes and their impressions, we propose Impression-CLIP, which is a novel machine-learning model based on CLIP (Contrastive Language-Image Pre-training). By using the CLIP-based model, font image features and their impression features are pulled closer, and font image features and unrelated impression features are pushed apart. This procedure realizes co-embedding between font image and their impressions. In our experiment, we perform cross-modal retrieval between fonts and impressions through co-embedding. The results indicate that Impression-CLIP achieves better retrieval accuracy than the state-of-the-art method. Additionally, our model shows the robustness to noise and missing tags.

Title: MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs

• Authors: Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16352
• Pdf URL: https://arxiv.org/pdf/2402.16352
• Copy Paste: [[2402.16352]] MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs(https://arxiv.org/abs/2402.16352)
• Keywords: large language model
• Abstract: Large language models (LLMs) have exhibited great potential in mathematical reasoning. However, there remains a performance gap in this area between existing open-source models and closed-source models such as GPT-4. In this paper, we introduce MathGenie, a novel method for generating diverse and reliable math problems from a small-scale problem-solution dataset (denoted as seed data). We augment the ground-truth solutions of our seed data and train a back-translation model to translate the augmented solutions back into new questions. Subsequently, we generate code-integrated solutions for the new questions. To ensure the correctness of the code-integrated solutions, we employ rationale-based strategy for solution verification. Various pretrained models, ranging from 7B to 70B, are trained on the newly curated data to test the effectiveness of the proposed augmentation technique, resulting in a family of models known as MathGenieLM. These models consistently outperform previous open-source models across five representative mathematical reasoning datasets, achieving state-of-the-art performance. In particular, MathGenieLM-InternLM2 achieves an accuracy of 87.7% on GSM8K and 55.7% on MATH, securing the best overall score among open-source language models.

Title: Language-guided Skill Learning with Temporal Variational Inference

• Authors: Haotian Fu, Pratyusha Sharma, Elias Stengel-Eskin, George Konidaris, Nicolas Le Roux, Marc-Alexandre Côté, Xingdi Yuan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16354
• Pdf URL: https://arxiv.org/pdf/2402.16354
• Copy Paste: [[2402.16354]] Language-guided Skill Learning with Temporal Variational Inference(https://arxiv.org/abs/2402.16354)
• Keywords: large language model, segmentation
• Abstract: We present an algorithm for skill discovery from expert demonstrations. The algorithm first utilizes Large Language Models (LLMs) to propose an initial segmentation of the trajectories. Following that, a hierarchical variational inference framework incorporates the LLM-generated segmentation information to discover reusable skills by merging trajectory segments. To further control the trade-off between compression and reusability, we introduce a novel auxiliary objective based on the Minimum Description Length principle that helps guide this skill discovery process. Our results demonstrate that agents equipped with our method are able to discover skills that help accelerate learning and outperform baseline skill learning approaches on new long-horizon tasks in BabyAI, a grid world navigation environment, as well as ALFRED, a household simulation environment.

Title: Feedback Efficient Online Fine-Tuning of Diffusion Models

• Authors: Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Sergey Levine, Tommaso Biancalani
• Subjects: cs.LG, cs.AI, q-bio.QM, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.16359
• Pdf URL: https://arxiv.org/pdf/2402.16359
• Copy Paste: [[2402.16359]] Feedback Efficient Online Fine-Tuning of Diffusion Models(https://arxiv.org/abs/2402.16359)
• Keywords: diffusion
• Abstract: Diffusion models excel at modeling complex data distributions, including those of images, proteins, and small molecules. However, in many cases, our goal is to model parts of the distribution that maximize certain properties: for example, we may want to generate images with high aesthetic quality, or molecules with high bioactivity. It is natural to frame this as a reinforcement learning (RL) problem, in which the objective is to fine-tune a diffusion model to maximize a reward function that corresponds to some property. Even with access to online queries of the ground-truth reward function, efficiently discovering high-reward samples can be challenging: they might have a low probability in the initial distribution, and there might be many infeasible samples that do not even have a well-defined reward (e.g., unnatural images or physically impossible molecules). In this work, we propose a novel reinforcement learning procedure that efficiently explores on the manifold of feasible samples. We present a theoretical analysis providing a regret guarantee, as well as empirical validation across three domains: images, biological sequences, and molecules.

Title: Layer-wise Regularized Dropout for Neural Language Models

• Authors: Shiwen Ni, Min Yang, Ruifeng Xu, Chengming Li, Xiping Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16361
• Pdf URL: https://arxiv.org/pdf/2402.16361
• Copy Paste: [[2402.16361]] Layer-wise Regularized Dropout for Neural Language Models(https://arxiv.org/abs/2402.16361)
• Keywords: transformer
• Abstract: Among the various pre-trained neural language models that are popular today, dropout is already an indispensable regularization technique. To solve the inconsistency between training and inference caused by the randomness of dropout, some studies use consistency training to regularize dropout at the output layer. In this paper, we propose a novel Layer-wise Regularized Dropout (LR-Drop), which is specially designed for Transformer-based Language models. Specifically, LR-Drop layer-wise regularizes each Transformer layer using the consistency training strategy. Each training sample passes through the two siamese sub-models sampled by dropout, and then LR-Drop forces the hidden states, multi-head attention matrices, and output distribution of the two siamese sub-models to be consistent. The proposed LR-Drop can be regarded as a "self-distillation" framework, in which each sub-model generated by dropout is the other's "teacher" model and "student" model. Through extensive experiments on 8 natural language understanding datasets, 6 neural machine translation datasets, and 1 abstractive summarization dataset (a total of 15 datasets), we show that LR-Drop achieves superior performances, including state-of-the-art results.

Title: LLM Inference Unveiled: Survey and Roofline Model Insights

• Authors: Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, Kurt Keutzer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16363
• Pdf URL: https://arxiv.org/pdf/2402.16363
• Copy Paste: [[2402.16363]] LLM Inference Unveiled: Survey and Roofline Model Insights(https://arxiv.org/abs/2402.16363)
• Keywords: large language model
• Abstract: The field of efficient Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Although the field has expanded and is vibrant, there hasn't been a concise framework that analyzes the various methods of LLM Inference to provide a clear understanding of this domain. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference techniques. This framework enables identifying the bottlenecks in LLM deployments and provides a deeper understanding of the practical aspects on real devices, thereby informing more effective strategies for deploying LLM. Furthermore, we systematically collate the latest advancements in efficient LLM inference, covering crucial areas such as weight optimization (e.g., Knowledge Distillation and Quantization), decoding algorithm improvements (e.g., Early Exit and Mixture-of-Expert), and both hardware and system-level enhancements. Distinguished by the integration of roofline model analysis, our survey provides a comprehensive and nuanced exploration of efficient LLM inference challenges and solutions. This distinctive approach not only showcases the current research landscape but also delivers valuable insights for practical implementation, positioning our work as an indispensable resource for researchers new to the field as well as for those seeking to deepen their understanding of efficient LLM deployment. The tool LLM-Viewer is open-sourced.

Title: Unraveling Babel: Exploring Multilingual Activation Patterns within Large Language Models

• Authors: Weize Liu, Yinlong Xu, Hongxia Xu, Jintai Chen, Xuming Hu, Jian Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16367
• Pdf URL: https://arxiv.org/pdf/2402.16367
• Copy Paste: [[2402.16367]] Unraveling Babel: Exploring Multilingual Activation Patterns within Large Language Models(https://arxiv.org/abs/2402.16367)
• Keywords: large language model
• Abstract: Recently, large language models (LLMs) have achieved tremendous breakthroughs in the field of language processing, yet their mechanisms in processing multiple languages remain agnostic. Therefore, in this work we study the multilingual activation patterns of LLMs. By transforming the original Large Language Models (LLMs) into a Mixture of Experts (MoE) architecture, we analyze the expert activation patterns when processing various languages and demonstrate the connections of these activation patterns at the level of language families. We discover the existence of non-language-specific neurons as well as language-specific activation neurons. Further exploration even showcases that merely leveraging high-frequency activation neurons can accelerate inference while maintaining comparable performance. These findings shed light on the LLMs' multilingual processing mechanism, and are of significant importance in guiding the multilingual training and model pruning of LLMs.

Title: Generative AI in Vision: A Survey on Models, Metrics and Applications

• Authors: Gaurav Raut, Apoorv Singh
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16369
• Pdf URL: https://arxiv.org/pdf/2402.16369
• Copy Paste: [[2402.16369]] Generative AI in Vision: A Survey on Models, Metrics and Applications(https://arxiv.org/abs/2402.16369)
• Keywords: diffusion, generative
• Abstract: Generative AI models have revolutionized various fields by enabling the creation of realistic and diverse data samples. Among these models, diffusion models have emerged as a powerful approach for generating high-quality images, text, and audio. This survey paper provides a comprehensive overview of generative AI diffusion and legacy models, focusing on their underlying techniques, applications across different domains, and their challenges. We delve into the theoretical foundations of diffusion models, including concepts such as denoising diffusion probabilistic models (DDPM) and score-based generative modeling. Furthermore, we explore the diverse applications of these models in text-to-image, image inpainting, and image super-resolution, along with others, showcasing their potential in creative tasks and data augmentation. By synthesizing existing research and highlighting critical advancements in this field, this survey aims to provide researchers and practitioners with a comprehensive understanding of generative AI diffusion and legacy models and inspire future innovations in this exciting area of artificial intelligence.

Title: Improving LLM-based Machine Translation with Systematic Self-Correction

• Authors: Zhaopeng Feng, Yan Zhang, Hao Li, Wenqiang Liu, Jun Lang, Yang Feng, Jian Wu, Zuozhu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16379
• Pdf URL: https://arxiv.org/pdf/2402.16379
• Copy Paste: [[2402.16379]] Improving LLM-based Machine Translation with Systematic Self-Correction(https://arxiv.org/abs/2402.16379)
• Keywords: interpretability, large language model
• Abstract: Large Language Models (LLMs) have achieved impressive results in Machine Translation (MT). However, careful evaluations by human reveal that the translations produced by LLMs still contain multiple errors. Importantly, feeding back such error information into the LLMs can lead to self-correction and result in improved translation performance. Motivated by these insights, we introduce a systematic LLM-based self-correcting translation framework, named TER, which stands for Translate, Estimate, and Refine, marking a significant step forward in this direction. Our findings demonstrate that 1) our self-correction framework successfully assists LLMs in improving their translation quality across a wide range of languages, whether it's from high-resource languages to low-resource ones or whether it's English-centric or centered around other languages; 2) TER exhibits superior systematicity and interpretability compared to previous methods; 3) different estimation strategies yield varied impacts on AI feedback, directly affecting the effectiveness of the final corrections. We further compare different LLMs and conduct various experiments involving self-correction and cross-model correction to investigate the potential relationship between the translation and evaluation capabilities of LLMs.

Title: Immunization against harmful fine-tuning attacks

• Authors: Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, Jan Batzner, Hassan Sajjad, Frank Rudzicz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16382
• Pdf URL: https://arxiv.org/pdf/2402.16382
• Copy Paste: [[2402.16382]] Immunization against harmful fine-tuning attacks(https://arxiv.org/abs/2402.16382)
• Keywords: attack, large language model
• Abstract: Approaches to aligning large language models (LLMs) with human values has focused on correcting misalignment that emerges from pretraining. However, this focus overlooks another source of misalignment: bad actors might purposely fine-tune LLMs to achieve harmful goals. In this paper, we present an emerging threat model that has arisen from alignment circumvention and fine-tuning attacks. However, lacking in previous works is a clear presentation of the conditions for effective defence. We propose a set of conditions for effective defence against harmful fine-tuning in LLMs called "Immunization conditions," which help us understand how we would construct and measure future defences. Using this formal framework for defence, we offer a synthesis of different research directions that might be persued to prevent harmful fine-tuning attacks and provide a demonstration of how to use these conditions experimentally showing early results of using an adversarial loss to immunize LLama2-7b-chat.

Title: MoZIP: A Multilingual Benchmark to Evaluate Large Language Models in Intellectual Property

• Authors: Shiwen Ni, Minghuan Tan, Yuelin Bai, Fuqiang Niu, Min Yang, Bowen Zhang, Ruifeng Xu, Xiaojun Chen, Chengming Li, Xiping Hu, Ye Li, Jianping Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16389
• Pdf URL: https://arxiv.org/pdf/2402.16389
• Copy Paste: [[2402.16389]] MoZIP: A Multilingual Benchmark to Evaluate Large Language Models in Intellectual Property(https://arxiv.org/abs/2402.16389)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated impressive performance in various natural language processing (NLP) tasks. However, there is limited understanding of how well LLMs perform in specific domains (e.g, the intellectual property (IP) domain). In this paper, we contribute a new benchmark, the first Multilingual-oriented quiZ on Intellectual Property (MoZIP), for the evaluation of LLMs in the IP domain. The MoZIP benchmark includes three challenging tasks: IP multiple-choice quiz (IPQuiz), IP question answering (IPQA), and patent matching (PatentMatch). In addition, we also develop a new IP-oriented multilingual large language model (called MoZi), which is a BLOOMZ-based model that has been supervised fine-tuned with multilingual IP-related text data. We evaluate our proposed MoZi model and four well-known LLMs (i.e., BLOOMZ, BELLE, ChatGLM and ChatGPT) on the MoZIP benchmark. Experimental results demonstrate that MoZi outperforms BLOOMZ, BELLE and ChatGLM by a noticeable margin, while it had lower scores compared with ChatGPT. Notably, the performance of current LLMs on the MoZIP benchmark has much room for improvement, and even the most powerful ChatGPT does not reach the passing level. Our source code, data, and models are available at \url{https://github.com/AI-for-Science/MoZi}.

Title: Placing Objects in Context via Inpainting for Out-of-distribution Segmentation

• Authors: Pau de Jorge, Riccardo Volpi, Puneet K. Dokania, Philip H. S. Torr, Gregory Rogez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16392
• Pdf URL: https://arxiv.org/pdf/2402.16392
• Copy Paste: [[2402.16392]] Placing Objects in Context via Inpainting for Out-of-distribution Segmentation(https://arxiv.org/abs/2402.16392)
• Keywords: diffusion, segmentation
• Abstract: When deploying a semantic segmentation model into the real world, it will inevitably be confronted with semantic classes unseen during training. Thus, to safely deploy such systems, it is crucial to accurately evaluate and improve their anomaly segmentation capabilities. However, acquiring and labelling semantic segmentation data is expensive and unanticipated conditions are long-tail and potentially hazardous. Indeed, existing anomaly segmentation datasets capture a limited number of anomalies, lack realism or have strong domain shifts. In this paper, we propose the Placing Objects in Context (POC) pipeline to realistically add any object into any image via diffusion models. POC can be used to easily extend any dataset with an arbitrary number of objects. In our experiments, we present different anomaly segmentation datasets based on POC-generated data and show that POC can improve the performance of recent state-of-the-art anomaly fine-tuning methods in several standardized benchmarks. POC is also effective to learn new classes. For example, we use it to edit Cityscapes samples by adding a subset of Pascal classes and show that models trained on such data achieve comparable performance to the Pascal-trained baseline. This corroborates the low sim-to-real gap of models trained on POC-generated images.

Title: Communication Optimal Unbalanced Private Set Union

• Authors: Jean-Guillaume Dumas (UGA, LJK, CASC), Alexis Galan (CASC), Bruno Grenet (CASC), Aude Maignan (CASC), Daniel S. Roche
• Subjects: cs.CR, cs.SC
• Abstract URL: https://arxiv.org/abs/2402.16393
• Pdf URL: https://arxiv.org/pdf/2402.16393
• Copy Paste: [[2402.16393]] Communication Optimal Unbalanced Private Set Union(https://arxiv.org/abs/2402.16393)
• Keywords: privacy
• Abstract: We consider the private set union (PSU) problem, where two parties each hold a private set of elements, and they want one of the parties (the receiver) to learn the union of the two sets and nothing else. Our protocols are targeted for the unbalanced case where the receiver's set size is larger than the sender's set size, with the goal of minimizing the costs for the sender both in terms of communication volume and local computation time. This setting is motivated by applications where the receiver has significantly more data (input set size) and computational resources than the sender which might be realized on a small, low-power device. Asymptotically, we achieve communication cost linear in the sender's (smaller) set size, and computation costs for sender and receiver which are nearly-linear in their respective set sizes. To our knowledge, ours is the first algorithm to achieve nearly-linear communication and computation for PSU in this unbalanced setting. Our protocols utilize fully homomorphic encryption (FHE) and, optionally, linearly homomorphic encryption (LHE) to perform the necessary computations while preserving privacy. The underlying computations are based on univariate polynomial arithmetic realized within homomorphic encryption, namely fast multiplication, modular reduction, and multi-point evaluation. These asymptotically fast HE polynomial arithmetic algorithms may be of independent interest.

Title: Investigating Deep Watermark Security: An Adversarial Transferability Perspective

• Authors: Biqing Qi, Junqi Gao, Yiang Luo, Jianxing Liu, Ligang Wu, Bowen Zhou
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16397
• Pdf URL: https://arxiv.org/pdf/2402.16397
• Copy Paste: [[2402.16397]] Investigating Deep Watermark Security: An Adversarial Transferability Perspective(https://arxiv.org/abs/2402.16397)
• Keywords: security, protect, attack, watermark, generative
• Abstract: The rise of generative neural networks has triggered an increased demand for intellectual property (IP) protection in generated content. Deep watermarking techniques, recognized for their flexibility in IP protection, have garnered significant attention. However, the surge in adversarial transferable attacks poses unprecedented challenges to the security of deep watermarking techniques-an area currently lacking systematic investigation. This study fills this gap by introducing two effective transferable attackers to assess the vulnerability of deep watermarks against erasure and tampering risks. Specifically, we initially define the concept of local sample density, utilizing it to deduce theorems on the consistency of model outputs. Upon discovering that perturbing samples towards high sample density regions (HSDR) of the target class enhances targeted adversarial transferability, we propose the Easy Sample Selection (ESS) mechanism and the Easy Sample Matching Attack (ESMA) method. Additionally, we propose the Bottleneck Enhanced Mixup (BEM) that integrates information bottleneck theory to reduce the generator's dependence on irrelevant noise. Experiments show a significant enhancement in the success rate of targeted transfer attacks for both ESMA and BEM-ESMA methods. We further conduct a comprehensive evaluation using ESMA and BEM-ESMA as measurements, considering model architecture and watermark encoding length, and achieve some impressive findings.

Title: Analysis of Embeddings Learned by End-to-End Machine Learning Eye Movement-driven Biometrics Pipeline

• Authors: Mehedi Hasan Raju, Lee Friedman, Dillon J Lohr, Oleg V Komogortsev
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16399
• Pdf URL: https://arxiv.org/pdf/2402.16399
• Copy Paste: [[2402.16399]] Analysis of Embeddings Learned by End-to-End Machine Learning Eye Movement-driven Biometrics Pipeline(https://arxiv.org/abs/2402.16399)
• Keywords: biometric
• Abstract: This paper expands on the foundational concept of temporal persistence in biometric systems, specifically focusing on the domain of eye movement biometrics facilitated by machine learning. Unlike previous studies that primarily focused on developing biometric authentication systems, our research delves into the embeddings learned by these systems, particularly examining their temporal persistence, reliability, and biometric efficacy in response to varying input data. Utilizing two publicly available eye-movement datasets, we employed the state-of-the-art Eye Know You Too machine learning pipeline for our analysis. We aim to validate whether the machine learning-derived embeddings in eye movement biometrics mirror the temporal persistence observed in traditional biometrics. Our methodology involved conducting extensive experiments to assess how different lengths and qualities of input data influence the performance of eye movement biometrics more specifically how it impacts the learned embeddings. We also explored the reliability and consistency of the embeddings under varying data conditions. Three key metrics (kendall's coefficient of concordance, intercorrelations, and equal error rate) were employed to quantitatively evaluate our findings. The results reveal while data length significantly impacts the stability of the learned embeddings, however, the intercorrelations among embeddings show minimal effect.

Title: From RAGs to riches: Using large language models to write documents for clinical trials

• Authors: Nigel Markey, Ilyass El-Mansouri, Gaetan Rensonnet, Casper van Langen, Christoph Meier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16406
• Pdf URL: https://arxiv.org/pdf/2402.16406
• Copy Paste: [[2402.16406]] From RAGs to riches: Using large language models to write documents for clinical trials(https://arxiv.org/abs/2402.16406)
• Keywords: large language model
• Abstract: Clinical trials require numerous documents to be written -- protocols, consent forms, clinical study reports and others. Large language models (LLMs) offer the potential to rapidly generate first versions of these documents, however there are concerns about the quality of their output Here we report an evaluation of LLMs in generating parts of one such document, clinical trial protocols. We find that an offthe-shelf LLM delivers reasonable results, especially when assessing content relevance and the correct use of terminology. However, deficiencies remain: specifically clinical thinking and logic, and appropriate use of references. To improve performance, we used retrieval-augmented generation (RAG) to prompt an LLM with accurate up-to-date information. As a result of using RAG, the writing quality of the LLM improves substantially, which has implications for the practical useability of LLMs in clinical trial-related writing.

Title: Outline-Guided Object Inpainting with Diffusion Models

• Authors: Markus Pobitzer, Filip Janicki, Mattia Rigotti, Cristiano Malossi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16421
• Pdf URL: https://arxiv.org/pdf/2402.16421
• Copy Paste: [[2402.16421]] Outline-Guided Object Inpainting with Diffusion Models(https://arxiv.org/abs/2402.16421)
• Keywords: robust, diffusion, segmentation
• Abstract: Instance segmentation datasets play a crucial role in training accurate and robust computer vision models. However, obtaining accurate mask annotations to produce high-quality segmentation datasets is a costly and labor-intensive process. In this work, we show how this issue can be mitigated by starting with small annotated instance segmentation datasets and augmenting them to effectively obtain a sizeable annotated dataset. We achieve that by creating variations of the available annotated object instances in a way that preserves the provided mask annotations, thereby resulting in new image-mask pairs to be added to the set of annotated images. Specifically, we generate new images using a diffusion-based inpainting model to fill out the masked area with a desired object class by guiding the diffusion through the object outline. We show that the object outline provides a simple, but also reliable and convenient training-free guidance signal for the underlying inpainting model that is often sufficient to fill out the mask with an object of the correct class without further text guidance and preserve the correspondence between generated images and the mask annotations with high precision. Our experimental results reveal that our method successfully generates realistic variations of object instances, preserving their shape characteristics while introducing diversity within the augmented area. We also show that the proposed method can naturally be combined with text guidance and other image augmentation techniques.

Title: Improving behavior based authentication against adversarial attack using XAI

• Authors: Dong Qin, George Amariucai, Daji Qiao, Yong Guan
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.16430
• Pdf URL: https://arxiv.org/pdf/2402.16430
• Copy Paste: [[2402.16430]] Improving behavior based authentication against adversarial attack using XAI(https://arxiv.org/abs/2402.16430)
• Keywords: security, defense, attack, robust
• Abstract: In recent years, machine learning models, especially deep neural networks, have been widely used for classification tasks in the security domain. However, these models have been shown to be vulnerable to adversarial manipulation: small changes learned by an adversarial attack model, when applied to the input, can cause significant changes in the output. Most research on adversarial attacks and corresponding defense methods focuses only on scenarios where adversarial samples are directly generated by the attack model. In this study, we explore a more practical scenario in behavior-based authentication, where adversarial samples are collected from the attacker. The generated adversarial samples from the model are replicated by attackers with a certain level of discrepancy. We propose an eXplainable AI (XAI) based defense strategy against adversarial attacks in such scenarios. A feature selector, trained with our method, can be used as a filter in front of the original authenticator. It filters out features that are more vulnerable to adversarial attacks or irrelevant to authentication, while retaining features that are more robust. Through comprehensive experiments, we demonstrate that our XAI based defense strategy is effective against adversarial attacks and outperforms other defense strategies, such as adversarial training and defensive distillation.

Title: RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions

• Authors: Yuansen Zhang, Xiao Wang, Zhiheng Xi, Han Xia, Tao Gui, Qi Zhang, Xuanjing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16431
• Pdf URL: https://arxiv.org/pdf/2402.16431
• Copy Paste: [[2402.16431]] RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions(https://arxiv.org/abs/2402.16431)
• Keywords: attack, robust, large language model
• Abstract: Large Language Models (LLMs) have showcased remarkable capabilities in following human instructions. However, recent studies have raised concerns about the robustness of LLMs when prompted with instructions combining textual adversarial samples. In this paper, drawing inspiration from recent works that LLMs are sensitive to the design of the instructions, we utilize instructions in code style, which are more structural and less ambiguous, to replace typically natural language instructions. Through this conversion, we provide LLMs with more precise instructions and strengthen the robustness of LLMs. Moreover, under few-shot scenarios, we propose a novel method to compose in-context demonstrations using both clean and adversarial samples (\textit{adversarial context method}) to further boost the robustness of the LLMs. Experiments on eight robustness datasets show that our method consistently outperforms prompting LLMs with natural language instructions. For example, with gpt-3.5-turbo, our method achieves an improvement of 5.68\% in test set accuracy and a reduction of 5.66 points in Attack Success Rate (ASR).

Title: Training Implicit Generative Models via an Invariant Statistical Loss

• Authors: José Manuel de Frutos, Pablo M. Olmos, Manuel A. Vázquez, Joaquín Míguez
• Subjects: cs.LG, cs.AI, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.16435
• Pdf URL: https://arxiv.org/pdf/2402.16435
• Copy Paste: [[2402.16435]] Training Implicit Generative Models via an Invariant Statistical Loss(https://arxiv.org/abs/2402.16435)
• Keywords: generative
• Abstract: Implicit generative models have the capability to learn arbitrary complex data distributions. On the downside, training requires telling apart real data from artificially-generated ones using adversarial discriminators, leading to unstable training and mode-dropping issues. As reported by Zahee et al. (2017), even in the one-dimensional (1D) case, training a generative adversarial network (GAN) is challenging and often suboptimal. In this work, we develop a discriminator-free method for training one-dimensional (1D) generative implicit models and subsequently expand this method to accommodate multivariate cases. Our loss function is a discrepancy measure between a suitably chosen transformation of the model samples and a uniform distribution; hence, it is invariant with respect to the true distribution of the data. We first formulate our method for 1D random variables, providing an effective solution for approximate reparameterization of arbitrary complex distributions. Then, we consider the temporal setting (both univariate and multivariate), in which we model the conditional distribution of each sample given the history of the process. We demonstrate through numerical simulations that this new method yields promising results, successfully learning true distributions in a variety of scenarios and mitigating some of the well-known problems that state-of-the-art implicit methods present.

Title: Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models

• Authors: Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16438
• Pdf URL: https://arxiv.org/pdf/2402.16438
• Copy Paste: [[2402.16438]] Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models(https://arxiv.org/abs/2402.16438)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) demonstrate remarkable multilingual capabilities without being pre-trained on specially curated multilingual parallel corpora. It remains a challenging problem to explain the underlying mechanisms by which LLMs process multilingual texts. In this paper, we delve into the composition of Transformer architectures in LLMs to pinpoint language-specific regions. Specially, we propose a novel detection method, language activation probability entropy (LAPE), to identify language-specific neurons within LLMs. Based on LAPE, we conduct comprehensive experiments on two representative LLMs, namely LLaMA-2 and BLOOM. Our findings indicate that LLMs' proficiency in processing a particular language is predominantly due to a small subset of neurons, primarily situated in the models' top and bottom layers. Furthermore, we showcase the feasibility to "steer" the output language of LLMs by selectively activating or deactivating language-specific neurons. Our research provides important evidence to the understanding and exploration of the multilingual capabilities of LLMs.

Title: ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectors

• Authors: Zhexin Zhang, Yida Lu, Jingyuan Ma, Di Zhang, Rui Li, Pei Ke, Hao Sun, Lei Sha, Zhifang Sui, Hongning Wang, Minlie Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16444
• Pdf URL: https://arxiv.org/pdf/2402.16444
• Copy Paste: [[2402.16444]] ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectors(https://arxiv.org/abs/2402.16444)
• Keywords: explainability, large language model
• Abstract: The safety of Large Language Models (LLMs) has gained increasing attention in recent years, but there still lacks a comprehensive approach for detecting safety issues within LLMs' responses in an aligned, customizable and explainable manner. In this paper, we propose ShieldLM, an LLM-based safety detector, which aligns with general human safety standards, supports customizable detection rules, and provides explanations for its decisions. To train ShieldLM, we compile a large bilingual dataset comprising 14,387 query-response pairs, annotating the safety of responses based on various safety standards. Through extensive experiments, we demonstrate that ShieldLM surpasses strong baselines across four test sets, showcasing remarkable customizability and explainability. Besides performing well on standard detection datasets, ShieldLM has also been shown to be effective in real-world situations as a safety evaluator for advanced LLMs. We release ShieldLM at \url{https://github.com/thu-coai/ShieldLM} to support accurate and explainable safety detection under various safety standards, contributing to the ongoing efforts to enhance the safety of LLMs.

Title: D-XCB: Data-independent Debiasing for Fair and Accurate Transformer-based Cyberbullying Detection

• Authors: Peiling Yi, Arkaitz Zubiaga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16458
• Pdf URL: https://arxiv.org/pdf/2402.16458
• Copy Paste: [[2402.16458]] D-XCB: Data-independent Debiasing for Fair and Accurate Transformer-based Cyberbullying Detection(https://arxiv.org/abs/2402.16458)
• Keywords: robust, fair, transformer
• Abstract: Swear words are a common proxy to collect datasets with cyberbullying incidents. Our focus is on measuring and mitigating biases derived from spurious associations between swear words and incidents occurring as a result of such data collection strategies. After demonstrating and quantifying these biases, we introduce ID-XCB, the first data-independent debiasing technique that combines adversarial training, bias constraints and debias fine-tuning approach aimed at alleviating model attention to bias-inducing words without impacting overall model performance. We explore ID-XCB on two popular session-based cyberbullying datasets along with comprehensive ablation and generalisation studies. We show that ID-XCB learns robust cyberbullying detection capabilities while mitigating biases, outperforming state-of-the-art debiasing methods in both performance and bias mitigation. Our quantitative and qualitative analyses demonstrate its generalisability to unseen data.

Title: Defending LLMs against Jailbreaking Attacks via Backtranslation

• Authors: Yihan Wang, Zhouxing Shi, Andrew Bai, Cho-Jui Hsieh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16459
• Pdf URL: https://arxiv.org/pdf/2402.16459
• Copy Paste: [[2402.16459]] Defending LLMs against Jailbreaking Attacks via Backtranslation(https://arxiv.org/abs/2402.16459)
• Keywords: defense, attack, large language model
• Abstract: Although many large language models (LLMs) have been trained to refuse harmful requests, they are still vulnerable to jailbreaking attacks, which rewrite the original prompt to conceal its harmful intent. In this paper, we propose a new method for defending LLMs against jailbreaking attacks by ``backtranslation''. Specifically, given an initial response generated by the target LLM from an input prompt, our backtranslation prompts a language model to infer an input prompt that can lead to the response. The inferred prompt is called the backtranslated prompt which tends to reveal the actual intent of the original prompt, since it is generated based on the LLM's response and is not directly manipulated by the attacker. We then run the target LLM again on the backtranslated prompt, and we refuse the original prompt if the model refuses the backtranslated prompt. We explain that the proposed defense provides several benefits on its effectiveness and efficiency. We empirically demonstrate that our defense significantly outperforms the baselines, in the cases that are hard for the baselines, and our defense also has little impact on the generation quality for benign input prompts.

Title: Unveiling Vulnerability of Self-Attention

• Authors: Khai Jiet Liong, Hongqiu Wu, Hai Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16470
• Pdf URL: https://arxiv.org/pdf/2402.16470
• Copy Paste: [[2402.16470]] Unveiling Vulnerability of Self-Attention(https://arxiv.org/abs/2402.16470)
• Keywords: attack, robust, transformer
• Abstract: Pre-trained language models (PLMs) are shown to be vulnerable to minor word changes, which poses a big threat to real-world systems. While previous studies directly focus on manipulating word inputs, they are limited by their means of generating adversarial samples, lacking generalization to versatile real-world attack. This paper studies the basic structure of transformer-based PLMs, the self-attention (SA) mechanism. (1) We propose a powerful perturbation technique \textit{HackAttend}, which perturbs the attention scores within the SA matrices via meticulously crafted attention masks. We show that state-of-the-art PLMs fall into heavy vulnerability that minor attention perturbations $(1\%)$ can produce a very high attack success rate $(98\%)$. Our paper expands the conventional text attack of word perturbations to more general structural perturbations. (2) We introduce \textit{S-Attend}, a novel smoothing technique that effectively makes SA robust via structural perturbations. We empirically demonstrate that this simple yet effective technique achieves robust performance on par with adversarial training when facing various text attackers. Code is publicly available at \url{github.com/liongkj/HackAttend}.

Title: Edge Detectors Can Make Deep Convolutional Neural Networks More Robust

• Authors: Jin Ding, Jie-Chao Zhao, Yong-Zhi Sun, Ping Tan, Jia-Wei Wang, Ji-En Ma, You-Tong Fang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16479
• Pdf URL: https://arxiv.org/pdf/2402.16479
• Copy Paste: [[2402.16479]] Edge Detectors Can Make Deep Convolutional Neural Networks More Robust(https://arxiv.org/abs/2402.16479)
• Keywords: attack, robust
• Abstract: Deep convolutional neural networks (DCNN for short) are vulnerable to examples with small perturbations. Improving DCNN's robustness is of great significance to the safety-critical applications, such as autonomous driving and industry automation. Inspired by the principal way that human eyes recognize objects, i.e., largely relying on the shape features, this paper first employs the edge detectors as layer kernels and designs a binary edge feature branch (BEFB for short) to learn the binary edge features, which can be easily integrated into any popular backbone. The four edge detectors can learn the horizontal, vertical, positive diagonal, and negative diagonal edge features, respectively, and the branch is stacked by multiple Sobel layers (using edge detectors as kernels) and one threshold layer. The binary edge features learned by the branch, concatenated with the texture features learned by the backbone, are fed into the fully connected layers for classification. We integrate the proposed branch into VGG16 and ResNet34, respectively, and conduct experiments on multiple datasets. Experimental results demonstrate the BEFB is lightweight and has no side effects on training. And the accuracy of the BEFB integrated models is better than the original ones on all datasets when facing FGSM, PGD, and C\&W attacks. Besides, BEFB integrated models equipped with the robustness enhancing techniques can achieve better classification accuracy compared to the original models. The work in this paper for the first time shows it is feasible to enhance the robustness of DCNNs through combining both shape-like features and texture features.

Title: Intelligent Known and Novel Aircraft Recognition -- A Shift from Classification to Similarity Learning for Combat Identification

• Authors: Ahmad Saeed, Haasha Bin Atif, Usman Habib, Mohsin Bilal
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16486
• Pdf URL: https://arxiv.org/pdf/2402.16486
• Copy Paste: [[2402.16486]] Intelligent Known and Novel Aircraft Recognition -- A Shift from Classification to Similarity Learning for Combat Identification(https://arxiv.org/abs/2402.16486)
• Keywords: robust
• Abstract: Precise aircraft recognition in low-resolution remote sensing imagery is a challenging yet crucial task in aviation, especially combat identification. This research addresses this problem with a novel, scalable, and AI-driven solution. The primary hurdle in combat identification in remote sensing imagery is the accurate recognition of Novel/Unknown types of aircraft in addition to Known types. Traditional methods, human expert-driven combat identification and image classification, fall short in identifying Novel classes. Our methodology employs similarity learning to discern features of a broad spectrum of military and civilian aircraft. It discerns both Known and Novel aircraft types, leveraging metric learning for the identification and supervised few-shot learning for aircraft type classification. To counter the challenge of limited low-resolution remote sensing data, we propose an end-to-end framework that adapts to the diverse and versatile process of military aircraft recognition by training a generalized embedder in fully supervised manner. Comparative analysis with earlier aircraft image classification methods shows that our approach is effective for aircraft image classification (F1-score Aircraft Type of 0.861) and pioneering for quantifying the identification of Novel types (F1-score Bipartitioning of 0.936). The proposed methodology effectively addresses inherent challenges in remote sensing data, thereby setting new standards in dataset quality. The research opens new avenues for domain experts and demonstrates unique capabilities in distinguishing various aircraft types, contributing to a more robust, domain-adapted potential for real-time aircraft recognition.

Title: SAND: Decoupling Sanitization from Fuzzing for Low Overhead

• Authors: Ziqiao Kong, Shaohua Li, Heqing Huang, Zhendong Su
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.16497
• Pdf URL: https://arxiv.org/pdf/2402.16497
• Copy Paste: [[2402.16497]] SAND: Decoupling Sanitization from Fuzzing for Low Overhead(https://arxiv.org/abs/2402.16497)
• Keywords: robust
• Abstract: Sanitizers provide robust test oracles for various software vulnerabilities. Fuzzing on sanitizer-enabled programs has been the best practice to find software bugs. Since sanitizers need to heavily instrument a target program to insert run-time checks, sanitizer-enabled programs have much higher overhead compared to normally built programs. In this paper, we present SAND, a new fuzzing framework that decouples sanitization from the fuzzing loop. SAND performs fuzzing on a normally built program and only invokes sanitizer-enabled programs when input is shown to be interesting. Since most of the generated inputs are not interesting, i.e., not bug-triggering, SAND allows most of the fuzzing time to be spent on the normally built program. To identify interesting inputs, we introduce execution pattern for a practical execution analysis on the normally built program. We realize SAND on top of AFL++ and evaluate it on 12 real-world programs. Our extensive evaluation highlights its effectiveness: on a period of 24 hours, compared to fuzzing on ASan/UBSan-enabled and MSan-enabled programs, SAND respectively achieves 2.6x and 15x throughput and detects 51% and 242% more bugs.

Title: LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments

• Authors: Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, Lijie Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16499
• Pdf URL: https://arxiv.org/pdf/2402.16499
• Copy Paste: [[2402.16499]] LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments(https://arxiv.org/abs/2402.16499)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have revealed their potential for achieving autonomous agents possessing human-level intelligence. However, existing benchmarks for evaluating LLM Agents either use static datasets, potentially leading to data leakage or focus only on single-agent scenarios, overlooking the complexities of multi-agent interactions. There is a lack of a benchmark that evaluates the diverse capabilities of LLM agents in multi-agent, dynamic environments. To this end, we introduce LLMArena, a novel and easily extensible framework for evaluating the diverse capabilities of LLM in multi-agent dynamic environments. LLMArena encompasses seven distinct gaming environments, employing Trueskill scoring to assess crucial abilities in LLM agents, including spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. We conduct an extensive experiment and human evaluation among different sizes and types of LLMs, showing that LLMs still have a significant journey ahead in their development towards becoming fully autonomous agents, especially in opponent modeling and team collaboration. We hope LLMArena could guide future research towards enhancing these capabilities in LLMs, ultimately leading to more sophisticated and practical applications in dynamic, multi-agent settings. The code and data will be available.

Title: Stochastic Conditional Diffusion Models for Semantic Image Synthesis

• Authors: Juyeon Ko, Inho Kong, Hyunwoo J. Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16506
• Pdf URL: https://arxiv.org/pdf/2402.16506
• Copy Paste: [[2402.16506]] Stochastic Conditional Diffusion Models for Semantic Image Synthesis(https://arxiv.org/abs/2402.16506)
• Keywords: robust, diffusion
• Abstract: Semantic image synthesis (SIS) is a task to generate realistic images corresponding to semantic maps (labels). It can be applied to diverse real-world practices such as photo editing or content creation. However, in real-world applications, SIS often encounters noisy user inputs. To address this, we propose Stochastic Conditional Diffusion Model (SCDM), which is a robust conditional diffusion model that features novel forward and generation processes tailored for SIS with noisy labels. It enhances robustness by stochastically perturbing the semantic label maps through Label Diffusion, which diffuses the labels with discrete diffusion. Through the diffusion of labels, the noisy and clean semantic maps become similar as the timestep increases, eventually becoming identical at $t=T$. This facilitates the generation of an image close to a clean image, enabling robust generation. Furthermore, we propose a class-wise noise schedule to differentially diffuse the labels depending on the class. We demonstrate that the proposed method generates high-quality samples through extensive experiments and analyses on benchmark datasets, including a novel experimental setup simulating human errors during real-world applications.

Title: Enhancement of 3D Camera Synthetic Training Data with Noise Models

• Authors: Katarína Osvaldová, Lukáš Gajdošech, Viktor Kocur, Martin Madaras
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16514
• Pdf URL: https://arxiv.org/pdf/2402.16514
• Copy Paste: [[2402.16514]] Enhancement of 3D Camera Synthetic Training Data with Noise Models(https://arxiv.org/abs/2402.16514)
• Keywords: segmentation
• Abstract: The goal of this paper is to assess the impact of noise in 3D camera-captured data by modeling the noise of the imaging process and applying it on synthetic training data. We compiled a dataset of specifically constructed scenes to obtain a noise model. We specifically model lateral noise, affecting the position of captured points in the image plane, and axial noise, affecting the position along the axis perpendicular to the image plane. The estimated models can be used to emulate noise in synthetic training data. The added benefit of adding artificial noise is evaluated in an experiment with rendered data for object segmentation. We train a series of neural networks with varying levels of noise in the data and measure their ability to generalize on real data. The results show that using too little or too much noise can hurt the networks' performance indicating that obtaining a model of noise from real scanners is beneficial for synthetic data generation.

Title: LLM-based Privacy Data Augmentation Guided by Knowledge Distillation with a Distribution Tutor for Medical Text Classification

• Authors: Yiping Song, Juhua Zhang, Zhiliang Tian, Yuxin Yang, Minlie Huang, Dongsheng Li
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16515
• Pdf URL: https://arxiv.org/pdf/2402.16515
• Copy Paste: [[2402.16515]] LLM-based Privacy Data Augmentation Guided by Knowledge Distillation with a Distribution Tutor for Medical Text Classification(https://arxiv.org/abs/2402.16515)
• Keywords: privacy, protect
• Abstract: As sufficient data are not always publically accessible for model training, researchers exploit limited data with advanced learning algorithms or expand the dataset via data augmentation (DA). Conducting DA in private domain requires private protection approaches (i.e. anonymization and perturbation), but those methods cannot provide protection guarantees. Differential privacy (DP) learning methods theoretically bound the protection but are not skilled at generating pseudo text samples with large models. In this paper, we transfer DP-based pseudo sample generation task to DP-based generated samples discrimination task, where we propose a DP-based DA method with a LLM and a DP-based discriminator for text classification on private domains. We construct a knowledge distillation model as the DP-based discriminator: teacher models, accessing private data, teaches students how to select private samples with calibrated noise to achieve DP. To constrain the distribution of DA's generation, we propose a DP-based tutor that models the noised private distribution and controls samples' generation with a low privacy cost. We theoretically analyze our model's privacy protection and empirically verify our model.

Title: Generative Pretrained Hierarchical Transformer for Time Series Forecasting

• Authors: Zhiding Liu, Jiqian Yang, Mingyue Cheng, Yucong Luo, Zhi Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16516
• Pdf URL: https://arxiv.org/pdf/2402.16516
• Copy Paste: [[2402.16516]] Generative Pretrained Hierarchical Transformer for Time Series Forecasting(https://arxiv.org/abs/2402.16516)
• Keywords: transformer, generative
• Abstract: Recent efforts have been dedicated to enhancing time series forecasting accuracy by introducing advanced network architectures and self-supervised pretraining strategies. Nevertheless, existing approaches still exhibit two critical drawbacks. Firstly, these methods often rely on a single dataset for training, limiting the model's generalizability due to the restricted scale of the training data. Secondly, the one-step generation schema is widely followed, which necessitates a customized forecasting head and overlooks the temporal dependencies in the output series, and also leads to increased training costs under different horizon length settings. To address these issues, we propose a novel generative pretrained hierarchical transformer architecture for forecasting, named GPHT. There are two aspects of key designs in GPHT. On the one hand, we advocate for constructing a mixed dataset for pretraining our model, comprising various datasets from diverse data scenarios. This approach significantly expands the scale of training data, allowing our model to uncover commonalities in time series data and facilitating improved transfer to specific datasets. On the other hand, GPHT employs an auto-regressive forecasting approach under the channel-independent assumption, effectively modeling temporal dependencies in the output series. Importantly, no customized forecasting head is required, enabling a single model to forecast at arbitrary horizon settings. We conduct sufficient experiments on eight datasets with mainstream self-supervised pretraining models and supervised models. The results demonstrated that GPHT surpasses the baseline models across various fine-tuning and zero/few-shot learning settings in the traditional long-term forecasting task, providing support for verifying the feasibility of pretrained time series large models.

Title: Q-FOX Learning: Breaking Tradition in Reinforcement Learning

• Authors: Mahmood Alqaseer, Yossra H. Ali, Tarik A. Rashid
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.16562
• Pdf URL: https://arxiv.org/pdf/2402.16562
• Copy Paste: [[2402.16562]] Q-FOX Learning: Breaking Tradition in Reinforcement Learning(https://arxiv.org/abs/2402.16562)
• Keywords: robust
• Abstract: Reinforcement learning (RL) is a subset of artificial intelligence (AI) where agents learn the best action by interacting with the environment, making it suitable for tasks that do not require labeled data or direct supervision. Hyperparameters (HP) tuning refers to choosing the best parameter that leads to optimal solutions in RL algorithms. Manual or random tuning of the HP may be a crucial process because variations in this parameter lead to changes in the overall learning aspects and different rewards. In this paper, a novel and automatic HP-tuning method called Q-FOX is proposed. This uses both the FOX optimizer, a new optimization method inspired by nature that mimics red foxes' hunting behavior, and the commonly used, easy-to-implement RL Q-learning algorithm to solve the problem of HP tuning. Moreover, a new objective function is proposed which prioritizes the reward over the mean squared error (MSE) and learning time (steps). Q-FOX has been evaluated on two OpenAI Gym environment control tasks: Cart Pole and Frozen Lake. It exposed greater cumulative rewards than HP tuning with other optimizers, such as PSO, GA, Bee, or randomly selected HP. The cumulative reward for the Cart Pole task was 32.08, and for the Frozen Lake task was 0.95. Despite the robustness of Q-FOX, it has limitations. It cannot be used directly in real-word problems before choosing the HP in a simulation environment because its processes work iteratively, making it time-consuming. The results indicate that Q-FOX has played an essential role in HP tuning for RL algorithms to effectively solve different control tasks.

Title: Aligning Large Language Models to a Domain-specific Graph Database

• Authors: Yuanyuan Liang, Keren Tan, Tingyu Xie, Wenbiao Tao, Siyuan Wang, Yunshi Lan, Weining Qian
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2402.16567
• Pdf URL: https://arxiv.org/pdf/2402.16567
• Copy Paste: [[2402.16567]] Aligning Large Language Models to a Domain-specific Graph Database(https://arxiv.org/abs/2402.16567)
• Keywords: large language model
• Abstract: Graph Databases (Graph DB) are widely applied in various fields, including finance, social networks, and medicine. However, translating Natural Language (NL) into the Graph Query Language (GQL), commonly known as NL2GQL, proves to be challenging due to its inherent complexity and specialized nature. Some approaches have sought to utilize Large Language Models (LLMs) to address analogous tasks like text2SQL. Nevertheless, when it comes to NL2GQL taskson a particular domain, the absence of domain-specific NL-GQL data pairs makes it difficult to establish alignment between LLMs and the graph DB. To address this challenge, we propose a well-defined pipeline. Specifically, we utilize ChatGPT to create NL-GQL data pairs based on the given graph DB with self-instruct. Then, we use the created data to fine-tune LLMs, thereby achieving alignment between LLMs and the graph DB. Additionally, during inference, we propose a method that extracts relevant schema to the queried NL as the input context to guide LLMs for generating accurate GQLs.We evaluate our method on two constructed datasets deriving from graph DBs in finance domain and medicine domain, namely FinGQL and MediGQL. Experimental results demonstrate that our method significantly outperforms a set of baseline methods, with improvements of 5.90 and 6.36 absolute points on EM, and 6.00 and 7.09 absolute points on EX, respectively.

Title: Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models

• Authors: Yifu Gao, Linbo Qiao, Zhigang Kan, Zhihua Wen, Yongquan He, Dongsheng Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16568
• Pdf URL: https://arxiv.org/pdf/2402.16568
• Copy Paste: [[2402.16568]] Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models(https://arxiv.org/abs/2402.16568)
• Keywords: generative, large language model
• Abstract: Temporal knowledge graph question answering (TKGQA) poses a significant challenge task, due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge. Although large language models (LLMs) have made considerable progress in their reasoning ability over structured data, their application to the TKGQA task is a relatively unexplored area. This paper first proposes a novel generative temporal knowledge graph question answering framework, GenTKGQA, which guides LLMs to answer temporal questions through two phases: Subgraph Retrieval and Answer Generation. First, we exploit LLM's intrinsic knowledge to mine temporal constraints and structural links in the questions without extra training, thus narrowing down the subgraph search space in both temporal and structural dimensions. Next, we design virtual knowledge indicators to fuse the graph neural network signals of the subgraph and the text representations of the LLM in a non-shallow way, which helps the open-source LLM deeply understand the temporal order and structural dependencies among the retrieved facts through instruction tuning. Experimental results demonstrate that our model outperforms state-of-the-art baselines, even achieving 100\% on the metrics for the simple question type.

Title: Multi-Bit Distortion-Free Watermarking for Large Language Models

• Authors: Massieh Kordi Boroujeny, Ya Jiang, Kai Zeng, Brian Mark
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16578
• Pdf URL: https://arxiv.org/pdf/2402.16578
• Copy Paste: [[2402.16578]] Multi-Bit Distortion-Free Watermarking for Large Language Models(https://arxiv.org/abs/2402.16578)
• Keywords: watermark, large language model
• Abstract: Methods for watermarking large language models have been proposed that distinguish AI-generated text from human-generated text by slightly altering the model output distribution, but they also distort the quality of the text, exposing the watermark to adversarial detection. More recently, distortion-free watermarking methods were proposed that require a secret key to detect the watermark. The prior methods generally embed zero-bit watermarks that do not provide additional information beyond tagging a text as being AI-generated. We extend an existing zero-bit distortion-free watermarking method by embedding multiple bits of meta-information as part of the watermark. We also develop a computationally efficient decoder that extracts the embedded information from the watermark with low bit error rate.

Title: Improving the JPEG-resistance of Adversarial Attacks on Face Recognition by Interpolation Smoothing

• Authors: Kefu Guo, Fengfan Zhou, Hefei Ling, Ping Li, Hui Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16586
• Pdf URL: https://arxiv.org/pdf/2402.16586
• Copy Paste: [[2402.16586]] Improving the JPEG-resistance of Adversarial Attacks on Face Recognition by Interpolation Smoothing(https://arxiv.org/abs/2402.16586)
• Keywords: attack
• Abstract: JPEG compression can significantly impair the performance of adversarial face examples, which previous adversarial attacks on face recognition (FR) have not adequately addressed. Considering this challenge, we propose a novel adversarial attack on FR that aims to improve the resistance of adversarial examples against JPEG compression. Specifically, during the iterative process of generating adversarial face examples, we interpolate the adversarial face examples into a smaller size. Then we utilize these interpolated adversarial face examples to create the adversarial examples in the next iteration. Subsequently, we restore the adversarial face examples to their original size by interpolating. Throughout the entire process, our proposed method can smooth the adversarial perturbations, effectively mitigating the presence of high-frequency signals in the crafted adversarial face examples that are typically eliminated by JPEG compression. Our experimental results demonstrate the effectiveness of our proposed method in improving the JPEG-resistance of adversarial face examples.

Title: PCR-99: A Practical Method for Point Cloud Registration with 99% Outliers

• Authors: Seong Hun Lee, Javier Civera, Patrick Vandewalle
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.16598
• Pdf URL: https://arxiv.org/pdf/2402.16598
• Copy Paste: [[2402.16598]] PCR-99: A Practical Method for Point Cloud Registration with 99% Outliers(https://arxiv.org/abs/2402.16598)
• Keywords: robust
• Abstract: We propose a robust method for point cloud registration that can handle both unknown scales and extreme outlier ratios. Our method, dubbed PCR-99, uses a deterministic 3-point sampling approach with two novel mechanisms that significantly boost the speed: (1) an improved ordering of the samples based on pairwise scale consistency, prioritizing the point correspondences that are more likely to be inliers, and (2) an efficient outlier rejection scheme based on triplet scale consistency, prescreening bad samples and reducing the number of hypotheses to be tested. Our evaluation shows that, up to 98% outlier ratio, the proposed method achieves comparable performance to the state of the art. At 99% outlier ratio, however, it outperforms the state of the art for both known-scale and unknown-scale problems. Especially for the latter, we observe a clear superiority in terms of robustness and speed.

Title: Rethinking Negative Instances for Generative Named Entity Recognition

• Authors: Yuyang Ding, Juntao Li, Pinzheng Wang, Zecheng Tang, Bowen Yan, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16602
• Pdf URL: https://arxiv.org/pdf/2402.16602
• Copy Paste: [[2402.16602]] Rethinking Negative Instances for Generative Named Entity Recognition(https://arxiv.org/abs/2402.16602)
• Keywords: generative, large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities for generalizing in unseen tasks. In the Named Entity Recognition (NER) task, recent advancements have seen the remarkable improvement of LLMs in a broad range of entity domains via instruction tuning, by adopting entity-centric schema. In this work, we explore the potential enhancement of the existing methods by incorporating negative instances into training. Our experiments reveal that negative instances contribute to remarkable improvements by (1) introducing contextual information, and (2) clearly delineating label boundaries. Furthermore, we introduce a novel and efficient algorithm named Hierarchical Matching, which is tailored to transform unstructured predictions into structured entities. By integrating these components, we present GNER, a Generative NER system that shows improved zero-shot performance across unseen entity domains. Our comprehensive evaluation illustrates our system's superiority, surpassing state-of-the-art (SoTA) methods by 11 $F_1$ score in zero-shot evaluation.

Title: Understanding the Dataset Practitioners Behind Large Language Model Development

• Authors: Crystal Qian, Emily Reif, Minsuk Kahng
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.16611
• Pdf URL: https://arxiv.org/pdf/2402.16611
• Copy Paste: [[2402.16611]] Understanding the Dataset Practitioners Behind Large Language Model Development(https://arxiv.org/abs/2402.16611)
• Keywords: large language model
• Abstract: As large language models (LLMs) become more advanced and impactful, it is increasingly important to scrutinize the data that they rely upon and produce. What is it to be a dataset practitioner doing this work? We approach this in two parts: first, we define the role of "dataset practitioner" by performing a retrospective analysis on the responsibilities of teams contributing to LLM development at Google. Then, we conduct semi-structured interviews with a cross-section of these practitioners (N=10). We find that data quality is the top priority. To evaluate data quality, practitioners either rely on their own intuition or write custom evaluation logic. There is a lack of consensus across practitioners on what quality is and how to evaluate it. We discuss potential reasons for this phenomenon and opportunities for alignment.

Title: Long-Context Language Modeling with Parallel Context Encoding

• Authors: Howard Yen, Tianyu Gao, Danqi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16617
• Pdf URL: https://arxiv.org/pdf/2402.16617
• Copy Paste: [[2402.16617]] Long-Context Language Modeling with Parallel Context Encoding(https://arxiv.org/abs/2402.16617)
• Keywords: transformer, large language model
• Abstract: Extending large language models (LLMs) to process longer inputs is crucial for numerous applications. However, the considerable computational cost of transformers, coupled with limited generalization of positional encoding, restricts the size of their context window. We introduce Context Expansion with Parallel Encoding (CEPE), a framework that can be applied to any existing decoder-only LLMs to extend their context window. CEPE adopts a small encoder to process long inputs chunk by chunk and enables the frozen decoder to leverage additional contexts via cross-attention. CEPE is efficient, generalizable, and versatile: trained with 8K-token documents, CEPE extends the context window of LLAMA-2 to 128K tokens, offering 10x the throughput with only 1/6 of the memory. CEPE yields strong performance on language modeling and in-context learning. CEPE also excels in retrieval-augmented applications, while existing long-context models degenerate with retrieved contexts. We further introduce a CEPE variant that can extend the context window of instruction-tuned models with only unlabeled data, and showcase its effectiveness on LLAMA-2-CHAT, leading to a strong instruction-following model that can leverage very long context on downstream tasks.

Title: Cross-Modal Contextualized Diffusion Models for Text-Guided Visual Generation and Editing

• Authors: Ling Yang, Zhilong Zhang, Zhaochen Yu, Jingwei Liu, Minkai Xu, Stefano Ermon, Bin Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16627
• Pdf URL: https://arxiv.org/pdf/2402.16627
• Copy Paste: [[2402.16627]] Cross-Modal Contextualized Diffusion Models for Text-Guided Visual Generation and Editing(https://arxiv.org/abs/2402.16627)
• Keywords: diffusion
• Abstract: Conditional diffusion models have exhibited superior performance in high-fidelity text-guided visual generation and editing. Nevertheless, prevailing text-guided visual diffusion models primarily focus on incorporating text-visual relationships exclusively into the reverse process, often disregarding their relevance in the forward process. This inconsistency between forward and reverse processes may limit the precise conveyance of textual semantics in visual synthesis results. To address this issue, we propose a novel and general contextualized diffusion model (ContextDiff) by incorporating the cross-modal context encompassing interactions and alignments between text condition and visual sample into forward and reverse processes. We propagate this context to all timesteps in the two processes to adapt their trajectories, thereby facilitating cross-modal conditional modeling. We generalize our contextualized diffusion to both DDPMs and DDIMs with theoretical derivations, and demonstrate the effectiveness of our model in evaluations with two challenging tasks: text-to-image generation, and text-to-video editing. In each task, our ContextDiff achieves new state-of-the-art performance, significantly enhancing the semantic alignment between text condition and generated samples, as evidenced by quantitative and qualitative evaluations. Our code is available at https://github.com/YangLing0818/ContextDiff

Title: Domain Embeddings for Generating Complex Descriptions of Concepts in Italian Language

• Authors: Alessandro Maisto
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16632
• Pdf URL: https://arxiv.org/pdf/2402.16632
• Copy Paste: [[2402.16632]] Domain Embeddings for Generating Complex Descriptions of Concepts in Italian Language(https://arxiv.org/abs/2402.16632)
• Keywords: extraction
• Abstract: In this work, we propose a Distributional Semantic resource enriched with linguistic and lexical information extracted from electronic dictionaries, designed to address the challenge of bridging the gap between the continuous semantic values represented by distributional vectors and the discrete descriptions offered by general semantics theory. Recently, many researchers have concentrated on the nexus between embeddings and a comprehensive theory of semantics and meaning. This often involves decoding the representation of word meanings in Distributional Models into a set of discrete, manually constructed properties such as semantic primitives or features, using neural decoding techniques. Our approach introduces an alternative strategy grounded in linguistic data. We have developed a collection of domain-specific co-occurrence matrices, derived from two sources: a classification of Italian nouns categorized into 4 semantic traits and 20 concrete noun sub-categories, and a list of Italian verbs classified according to their semantic classes. In these matrices, the co-occurrence values for each word are calculated exclusively with a defined set of words pertinent to a particular lexical domain. The resource comprises 21 domain-specific matrices, one comprehensive matrix, and a Graphical User Interface. Our model facilitates the generation of reasoned semantic descriptions of concepts by selecting matrices directly associated with concrete conceptual knowledge, such as a matrix based on location nouns and the concept of animal habitats. We assessed the utility of the resource through two experiments, achieving promising outcomes in both: the automatic classification of animal nouns and the extraction of animal features.

Title: RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation

• Authors: Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16667
• Pdf URL: https://arxiv.org/pdf/2402.16667
• Copy Paste: [[2402.16667]] RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation(https://arxiv.org/abs/2402.16667)
• Keywords: generative, large language model
• Abstract: Generative models have demonstrated considerable potential in software engineering, particularly in tasks such as code generation and debugging. However, their utilization in the domain of code documentation generation remains underexplored. To this end, we introduce RepoAgent, a large language model powered open-source framework aimed at proactively generating, maintaining, and updating code documentation. Through both qualitative and quantitative evaluations, we have validated the effectiveness of our approach, showing that RepoAgent excels in generating high-quality repository-level documentation. The code and results are publicly accessible at https://github.com/OpenBMB/RepoAgent.

Title: StructLM: Towards Building Generalist Models for Structured Knowledge Grounding

• Authors: Alex Zhuang, Ge Zhang, Tianyu Zheng, Xinrun Du, Junjie Wang, Weiming Ren, Stephen W. Huang, Jie Fu, Xiang Yue, Wenhu Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16671
• Pdf URL: https://arxiv.org/pdf/2402.16671
• Copy Paste: [[2402.16671]] StructLM: Towards Building Generalist Models for Structured Knowledge Grounding(https://arxiv.org/abs/2402.16671)
• Keywords: large language model
• Abstract: Structured data sources, such as tables, graphs, and databases, are ubiquitous knowledge sources. Despite the demonstrated capabilities of large language models (LLMs) on plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as StructLM, based on the Code-LLaMA architecture, ranging from 7B to 34B parameters. Our StructLM series surpasses task-specific models on 14 out of 18 evaluated datasets and establishes new SoTA achievements on 7 SKG tasks. Furthermore, StructLM demonstrates exceptional generalization across 6 novel SKG tasks. Contrary to expectations, we observe that scaling model size offers marginal benefits, with StructLM-34B showing only slight improvements over StructLM-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level.

Title: ConSept: Continual Semantic Segmentation via Adapter-based Vision Transformer

• Authors: Bowen Dong, Guanglei Yang, Wangmeng Zuo, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16674
• Pdf URL: https://arxiv.org/pdf/2402.16674
• Copy Paste: [[2402.16674]] ConSept: Continual Semantic Segmentation via Adapter-based Vision Transformer(https://arxiv.org/abs/2402.16674)
• Keywords: transformer, segmentation
• Abstract: In this paper, we delve into the realm of vision transformers for continual semantic segmentation, a problem that has not been sufficiently explored in previous literature. Empirical investigations on the adaptation of existing frameworks to vanilla ViT reveal that incorporating visual adapters into ViTs or fine-tuning ViTs with distillation terms is advantageous for enhancing the segmentation capability of novel classes. These findings motivate us to propose Continual semantic Segmentation via Adapter-based ViT, namely ConSept. Within the simplified architecture of ViT with linear segmentation head, ConSept integrates lightweight attention-based adapters into vanilla ViTs. Capitalizing on the feature adaptation abilities of these adapters, ConSept not only retains superior segmentation ability for old classes, but also attains promising segmentation quality for novel classes. To further harness the intrinsic anti-catastrophic forgetting ability of ConSept and concurrently enhance the segmentation capabilities for both old and new classes, we propose two key strategies: distillation with a deterministic old-classes boundary for improved anti-catastrophic forgetting, and dual dice losses to regularize segmentation maps, thereby improving overall segmentation performance. Extensive experiments show the effectiveness of ConSept on multiple continual semantic segmentation benchmarks under overlapped or disjoint settings. Code will be publicly available at \url{https://github.com/DongSky/ConSept}.

Title: HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization

• Authors: Qiwei Peng, Yekun Chai, Xuhong Li
• Subjects: cs.CL, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.16694
• Pdf URL: https://arxiv.org/pdf/2402.16694
• Copy Paste: [[2402.16694]] HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization(https://arxiv.org/abs/2402.16694)
• Keywords: large language model
• Abstract: Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at \url{https://github.com/FloatAI/HumanEval-XL}.

Title: Look Before You Leap: Towards Decision-Aware and Generalizable Tool-Usage for Large Language Models

• Authors: Anchun Gui, Jian Li, Yong Dai, Nan Du, Han Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16696
• Pdf URL: https://arxiv.org/pdf/2402.16696
• Copy Paste: [[2402.16696]] Look Before You Leap: Towards Decision-Aware and Generalizable Tool-Usage for Large Language Models(https://arxiv.org/abs/2402.16696)
• Keywords: large language model
• Abstract: Tool-augmented large language models (LLMs) are attracting widespread attention when accessing up-to-date knowledge and alleviating hallucination issues. Nowadays, advanced closed-source LLMs (e.g., ChatGPT) have demonstrated surprising tool-usage capabilities through prompting and in-context learning techniques. To empower the capabilities of open-source LLMs (e.g., LLaMA) in manipulating tools, current efforts focus on either template-driven or token-triggered tool-usage. However, the former hampers LLMs' flexibility to address diverse user's queries due to constrained tool interactions, while the latter limits the generalizability when engaging with new tools, since tool-usage learning is based on task- and tool-specific datasets. To alleviate these concerns, in this paper, we propose a decision-aware and generalizable tool-usage framework (DEER). Specifically, we first construct the tool-usage samples with multiple decision branches via an automatic generation pipeline, thereby inspiring the decision-making awareness of LLMs under diverse scenarios. Meanwhile, we propose a novel tool sampling strategy to enhance the generalizability of LLMs over unseen tools. Extensive experiments demonstrate that our proposed DEER is effective and significantly outperforms baselines across various datasets.

Title: Generating Effective Ensembles for Sentiment Analysis

• Authors: Itay Etelis, Avi Rosenfeld, Abraham Itzhak Weinberg, David Sarne
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16700
• Pdf URL: https://arxiv.org/pdf/2402.16700
• Copy Paste: [[2402.16700]] Generating Effective Ensembles for Sentiment Analysis(https://arxiv.org/abs/2402.16700)
• Keywords: transformer
• Abstract: In recent years, transformer models have revolutionized Natural Language Processing (NLP), achieving exceptional results across various tasks, including Sentiment Analysis (SA). As such, current state-of-the-art approaches for SA predominantly rely on transformer models alone, achieving impressive accuracy levels on benchmark datasets. In this paper, we show that the key for further improving the accuracy of such ensembles for SA is to include not only transformers, but also traditional NLP models, despite the inferiority of the latter compared to transformer models. However, as we empirically show, this necessitates a change in how the ensemble is constructed, specifically relying on the Hierarchical Ensemble Construction (HEC) algorithm we present. Our empirical studies across eight canonical SA datasets reveal that ensembles incorporating a mix of model types, structured via HEC, significantly outperform traditional ensembles. Finally, we provide a comparative analysis of the performance of the HEC and GPT-4, demonstrating that while GPT-4 closely approaches state-of-the-art SA methods, it remains outperformed by our proposed ensemble strategy.

Title: SelectIT: Selective Instruction Tuning for Large Language Models via Uncertainty-Aware Self-Reflection

• Authors: Liangxin Liu, Xuebo Liu, Derek F. Wong, Dongfang Li, Ziyi Wang, Baotian Hu, Min Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16705
• Pdf URL: https://arxiv.org/pdf/2402.16705
• Copy Paste: [[2402.16705]] SelectIT: Selective Instruction Tuning for Large Language Models via Uncertainty-Aware Self-Reflection(https://arxiv.org/abs/2402.16705)
• Keywords: robust, large language model
• Abstract: Instruction tuning (IT) is crucial to tailoring large language models (LLMs) towards human-centric interactions. Recent advancements have shown that the careful selection of a small, high-quality subset of IT data can significantly enhance the performance of LLMs. Despite this, common approaches often rely on additional models or data sets, which increases costs and limits widespread adoption. In this work, we propose a novel approach, termed SelectIT, that capitalizes on the foundational capabilities of the LLM itself. Specifically, we exploit the intrinsic uncertainty present in LLMs to more effectively select high-quality IT data, without the need for extra resources. Furthermore, we introduce a novel IT dataset, the Selective Alpaca, created by applying SelectIT to the Alpaca-GPT4 dataset. Empirical results demonstrate that IT using Selective Alpaca leads to substantial model ability enhancement. The robustness of SelectIT has also been corroborated in various foundation models and domain-specific tasks. Our findings suggest that longer and more computationally intensive IT data may serve as superior sources of IT, offering valuable insights for future research in this area. Data, code, and scripts are freely available at https://github.com/Blue-Raincoat/SelectIT.

Title: CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models

• Authors: Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, Xuanjing Huang
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16717
• Pdf URL: https://arxiv.org/pdf/2402.16717
• Copy Paste: [[2402.16717]] CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models(https://arxiv.org/abs/2402.16717)
• Keywords: security, attack, large language model
• Abstract: Adversarial misuse, particularly through `jailbreaking' that circumvents a model's safety and ethical protocols, poses a significant challenge for Large Language Models (LLMs). This paper delves into the mechanisms behind such successful attacks, introducing a hypothesis for the safety mechanism of aligned LLMs: intent security recognition followed by response generation. Grounded in this hypothesis, we propose CodeChameleon, a novel jailbreak framework based on personalized encryption tactics. To elude the intent security recognition phase, we reformulate tasks into a code completion format, enabling users to encrypt queries using personalized encryption functions. To guarantee response generation functionality, we embed a decryption function within the instructions, which allows the LLM to decrypt and execute the encrypted queries successfully. We conduct extensive experiments on 7 LLMs, achieving state-of-the-art average Attack Success Rate (ASR). Remarkably, our method achieves an 86.6\% ASR on GPT-4-1106.

Title: Interpreting Grokked Transformers in Complex Modular Arithmetic

• Authors: Hiroki Furuta, Minegishi Gouki, Yusuke Iwasawa, Yutaka Matsuo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16726
• Pdf URL: https://arxiv.org/pdf/2402.16726
• Copy Paste: [[2402.16726]] Interpreting Grokked Transformers in Complex Modular Arithmetic(https://arxiv.org/abs/2402.16726)
• Keywords: transformer
• Abstract: Grokking has been actively explored to reveal the mystery of delayed generalization. Identifying interpretable algorithms inside the grokked models is a suggestive hint to understanding its mechanism. In this work, beyond the simplest and well-studied modular addition, we observe the internal circuits learned through grokking in complex modular arithmetic via interpretable reverse engineering, which highlights the significant difference in their dynamics: subtraction poses a strong asymmetry on Transformer; multiplication requires cosine-biased components at all the frequencies in a Fourier domain; polynomials often result in the superposition of the patterns from elementary arithmetic, but clear patterns do not emerge in challenging cases; grokking can easily occur even in higher-degree formulas with basic symmetric and alternating expressions. We also introduce the novel progress measure for modular arithmetic; Fourier Frequency Sparsity and Fourier Coefficient Ratio, which not only indicate the late generalization but also characterize distinctive internal representations of grokked models per modular operation. Our empirical analysis emphasizes the importance of holistic evaluation among various combinations.

Title: Neural Mesh Fusion: Unsupervised 3D Planar Surface Understanding

• Authors: Farhad G. Zanjani, Hong Cai, Yinhao Zhu, Leyla Mirvakhabova, Fatih Porikli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16739
• Pdf URL: https://arxiv.org/pdf/2402.16739
• Copy Paste: [[2402.16739]] Neural Mesh Fusion: Unsupervised 3D Planar Surface Understanding(https://arxiv.org/abs/2402.16739)
• Keywords: segmentation
• Abstract: This paper presents Neural Mesh Fusion (NMF), an efficient approach for joint optimization of polygon mesh from multi-view image observations and unsupervised 3D planar-surface parsing of the scene. In contrast to implicit neural representations, NMF directly learns to deform surface triangle mesh and generate an embedding for unsupervised 3D planar segmentation through gradient-based optimization directly on the surface mesh. The conducted experiments show that NMF obtains competitive results compared to state-of-the-art multi-view planar reconstruction, while not requiring any ground-truth 3D or planar supervision. Moreover, NMF is significantly more computationally efficient compared to implicit neural rendering-based scene reconstruction approaches.

Title: A Comprehensive Evaluation of Quantization Strategies for Large Language Models

• Authors: Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, Deyi Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16775
• Pdf URL: https://arxiv.org/pdf/2402.16775
• Copy Paste: [[2402.16775]] A Comprehensive Evaluation of Quantization Strategies for Large Language Models(https://arxiv.org/abs/2402.16775)
• Keywords: large language model
• Abstract: Increasing the number of parameters in large language models (LLMs) usually improves performance in downstream tasks but raises compute and memory costs, making deployment difficult in resource-limited settings. Quantization techniques, which reduce the bits needed for model weights or activations with minimal performance loss, have become popular due to the rise of LLMs. However, most quantization studies use pre-trained LLMs, and the impact of quantization on instruction-tuned LLMs and the relationship between perplexity and benchmark performance of quantized LLMs are not well understood. Evaluation of quantized LLMs is often limited to language modeling and a few classification tasks, leaving their performance on other benchmarks unclear. To address these gaps, we propose a structured evaluation framework consisting of three critical dimensions: (1) knowledge \& capacity, (2) alignment, and (3) efficiency, and conduct extensive experiments across ten diverse benchmarks. Our experimental results indicate that LLMs with 4-bit quantization can retain performance comparable to their non-quantized counterparts, and perplexity can serve as a proxy metric for quantized LLMs on most benchmarks. Furthermore, quantized LLMs with larger parameter scales can outperform smaller LLMs. Despite the memory savings achieved through quantization, it can also slow down the inference speed of LLMs. Consequently, substantial engineering efforts and hardware support are imperative to achieve a balanced optimization of decoding speed and memory consumption in the context of quantized LLMs.

Title: Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models

• Authors: Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, Dirk Hovy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16786
• Pdf URL: https://arxiv.org/pdf/2402.16786
• Copy Paste: [[2402.16786]] Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models(https://arxiv.org/abs/2402.16786)
• Keywords: robust, large language model
• Abstract: Much recent work seeks to evaluate values and opinions in large language models (LLMs) using multiple-choice surveys and questionnaires. Most of this work is motivated by concerns around real-world LLM applications. For example, politically-biased LLMs may subtly influence society when they are used by millions of people. Such real-world concerns, however, stand in stark contrast to the artificiality of current evaluations: real users do not typically ask LLMs survey questions. Motivated by this discrepancy, we challenge the prevailing constrained evaluation paradigm for values and opinions in LLMs and explore more realistic unconstrained evaluations. As a case study, we focus on the popular Political Compass Test (PCT). In a systematic review, we find that most prior work using the PCT forces models to comply with the PCT's multiple-choice format. We show that models give substantively different answers when not forced; that answers change depending on how models are forced; and that answers lack paraphrase robustness. Then, we demonstrate that models give different answers yet again in a more realistic open-ended answer setting. We distill these findings into recommendations and open challenges in evaluating values and opinions in LLMs.

Title: Why Transformers Need Adam: A Hessian Perspective

• Authors: Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, Zhi-Quan Luo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16788
• Pdf URL: https://arxiv.org/pdf/2402.16788
• Copy Paste: [[2402.16788]] Why Transformers Need Adam: A Hessian Perspective(https://arxiv.org/abs/2402.16788)
• Keywords: transformer
• Abstract: SGD performs worse than Adam by a significant margin on Transformers, but the reason remains unclear. In this work, we provide an explanation of SGD's failure on Transformers through the lens of Hessian: (i) Transformers are ``heterogeneous'': the Hessian spectrum across parameter blocks vary dramatically, a phenomenon we call ``block heterogeneity"; (ii) Heterogeneity hampers SGD: SGD performs badly on problems with block heterogeneity. To validate that heterogeneity hampers SGD, we check various Transformers, CNNs, MLPs, and quadratic problems, and find that SGD works well on problems without block heterogeneity but performs badly when the heterogeneity exists. Our initial theoretical analysis indicates that SGD fails because it applies one single learning rate for all blocks, which cannot handle the heterogeneity among blocks. The failure could be rescued if we could assign different learning rates across blocks, as designed in Adam.

Title: Multi-Human Mesh Recovery with Transformers

• Authors: Zeyu Wang, Zhenzhen Weng, Serena Yeung-Levy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16806
• Pdf URL: https://arxiv.org/pdf/2402.16806
• Copy Paste: [[2402.16806]] Multi-Human Mesh Recovery with Transformers(https://arxiv.org/abs/2402.16806)
• Keywords: transformer
• Abstract: Conventional approaches to human mesh recovery predominantly employ a region-based strategy. This involves initially cropping out a human-centered region as a preprocessing step, with subsequent modeling focused on this zoomed-in image. While effective for single figures, this pipeline poses challenges when dealing with images featuring multiple individuals, as different people are processed separately, often leading to inaccuracies in relative positioning. Despite the advantages of adopting a whole-image-based approach to address this limitation, early efforts in this direction have fallen short in performance compared to recent region-based methods. In this work, we advocate for this under-explored area of modeling all people at once, emphasizing its potential for improved accuracy in multi-person scenarios through considering all individuals simultaneously and leveraging the overall context and interactions. We introduce a new model with a streamlined transformer-based design, featuring three critical design choices: multi-scale feature incorporation, focused attention mechanisms, and relative joint supervision. Our proposed model demonstrates a significant performance improvement, surpassing state-of-the-art region-based and whole-image-based methods on various benchmarks involving multiple individuals.

Title: OncoGPT: A Medical Conversational Model Tailored with Oncology Domain Expertise on a Large Language Model Meta-AI (LLaMA)

• Authors: Fujian Jia, Xin Liu, Lixi Deng, Jiwen Gu, Chunchao Pu, Tunan Bai, Mengjiang Huang, Yuanzhi Lu, Kang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16810
• Pdf URL: https://arxiv.org/pdf/2402.16810
• Copy Paste: [[2402.16810]] OncoGPT: A Medical Conversational Model Tailored with Oncology Domain Expertise on a Large Language Model Meta-AI (LLaMA)(https://arxiv.org/abs/2402.16810)
• Keywords: large language model
• Abstract: In the past year, there has been a growing trend in applying Large Language Models (LLMs) to the field of medicine, particularly with the advent of advanced language models such as ChatGPT developed by OpenAI. However, there is limited research on LLMs specifically addressing oncology-related queries. The primary aim of this research was to develop a specialized language model that demonstrates improved accuracy in providing advice related to oncology. We performed an extensive data collection of online question-answer interactions centered around oncology, sourced from reputable doctor-patient platforms. Following data cleaning and anonymization, a dataset comprising over 180K+ oncology-related conversations was established. The conversations were categorized and meticulously reviewed by field specialists and clinicians to ensure precision. Employing the LLaMA model and other selected open-source datasets, we conducted iterative fine-tuning to enhance the model's proficiency in basic medical conversation and specialized oncology knowledge. We observed a substantial enhancement in the model's understanding of genuine patient inquiries and its reliability in offering oncology-related advice through the utilization of real online question-answer interactions in the fine-tuning process. We release database and models to the research community (https://github.com/OncoGPT1).

Title: Investigating the Effectiveness of HyperTuning via Gisting

• Authors: Jason Phang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16817
• Pdf URL: https://arxiv.org/pdf/2402.16817
• Copy Paste: [[2402.16817]] Investigating the Effectiveness of HyperTuning via Gisting(https://arxiv.org/abs/2402.16817)
• Keywords: transformer
• Abstract: Gisting (Mu et al., 2023) is a simple method for training models to compress information into fewer token representations using a modified attention mask, and can serve as an economical approach to training Transformer-based hypernetworks. We introduce HyperLlama, a set of Gisting-based hypernetworks built on Llama-2 models that generates task-specific soft prefixes based on few-shot inputs. In experiments across P3, Super-NaturalInstructions and Symbol Tuning datasets, we show that HyperLlama models can effectively compress information from few-shot examples into soft prefixes. However, they still underperform multi-task fine-tuned language models with full attention over few-shot in-context examples. We also show that HyperLlama-generated soft prefixes can serve as better initializations for further prefix tuning. Overall, Gisting-based hypernetworks are economical and easy to implement, but have mixed empirical performance.

Title: Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

• Authors: Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, Roberta Raileanu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16822
• Pdf URL: https://arxiv.org/pdf/2402.16822
• Copy Paste: [[2402.16822]] Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts(https://arxiv.org/abs/2402.16822)
• Keywords: security, robust, large language model
• Abstract: As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to user inputs is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem, and uses open-ended search to generate prompts that are both effective and diverse. It can uncover a model's vulnerabilities across a broad range of domains including, in this paper, safety, question answering, and cybersecurity. We also demonstrate that fine-tuning on synthetic data generated by Rainbow Teaming improves the safety of state-of-the-art LLMs without hurting their general capabilities and helpfulness, paving the path to open-ended self-improvement.

Title: Weighted Monte Carlo augmented spherical Fourier-Bessel convolutional layers for 3D abdominal organ segmentation

• Authors: Wenzhao Zhao, Steffen Albert, Barbara D. Wichtmann, Angelika Maurer, Ulrike Attenberger, Frank G. Zöllner, Jürgen Hesser
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16825
• Pdf URL: https://arxiv.org/pdf/2402.16825
• Copy Paste: [[2402.16825]] Weighted Monte Carlo augmented spherical Fourier-Bessel convolutional layers for 3D abdominal organ segmentation(https://arxiv.org/abs/2402.16825)
• Keywords: extraction, segmentation
• Abstract: Filter-decomposition-based group equivariant convolutional neural networks show promising stability and data efficiency for 3D image feature extraction. However, the existing filter-decomposition-based 3D group equivariant neural networks rely on parameter-sharing designs and are mostly limited to rotation transform groups, where the chosen spherical harmonic filter bases consider only angular orthogonality. These limitations hamper its application to deep neural network architectures for medical image segmentation. To address these issues, this paper describes a non-parameter-sharing affine group equivariant neural network for 3D medical image segmentation based on an adaptive aggregation of Monte Carlo augmented spherical Fourier Bessel filter bases. The efficiency and flexibility of the adopted non-parameter strategy enable for the first time an efficient implementation of 3D affine group equivariant convolutional neural networks for volumetric data. The introduced spherical Bessel Fourier filter basis combines both angular and radial orthogonality for better feature extraction. The 3D image segmentation experiments on two abdominal image sets, BTCV and the NIH Pancreas datasets, show that the proposed methods excel the state-of-the-art 3D neural networks with high training stability and data efficiency. The code will be available at https://github.com/ZhaoWenzhao/WVMS.

Title: A Survey on Data Selection for Language Models

• Authors: Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, William Yang Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16827
• Pdf URL: https://arxiv.org/pdf/2402.16827
• Copy Paste: [[2402.16827]] A Survey on Data Selection for Language Models(https://arxiv.org/abs/2402.16827)
• Keywords: large language model
• Abstract: A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may not be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount of training required. Data selection methods aim to determine which candidate data points to include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection research. Consequently, knowledge of effective data selection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies. To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related research areas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing an entry point for new and established researchers. Additionally, throughout this review we draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research.

Title: Training Neural Networks from Scratch with Parallel Low-Rank Adapters

• Authors: Minyoung Huh, Brian Cheung, Jeremy Bernstein, Phillip Isola, Pulkit Agrawal
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16828
• Pdf URL: https://arxiv.org/pdf/2402.16828
• Copy Paste: [[2402.16828]] Training Neural Networks from Scratch with Parallel Low-Rank Adapters(https://arxiv.org/abs/2402.16828)
• Keywords: transformer
• Abstract: The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced the cost of model finetuning, its application in model pre-training remains largely unexplored. This paper explores extending LoRA to model pre-training, identifying the inherent constraints and limitations of standard LoRA in this context. We introduce LoRA-the-Explorer (LTE), a novel bi-level optimization algorithm designed to enable parallel training of multiple low-rank heads across computing nodes, thereby reducing the need for frequent synchronization. Our approach includes extensive experimentation on vision transformers using various vision datasets, demonstrating that LTE is competitive with standard pre-training.

Title: Mysterious Projections: Multimodal LLMs Gain Domain-Specific Visual Capabilities Without Richer Cross-Modal Projections

• Authors: Gaurav Verma, Minje Choi, Kartik Sharma, Jamelle Watson-Daniels, Sejoon Oh, Srijan Kumar
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16832
• Pdf URL: https://arxiv.org/pdf/2402.16832
• Copy Paste: [[2402.16832]] Mysterious Projections: Multimodal LLMs Gain Domain-Specific Visual Capabilities Without Richer Cross-Modal Projections(https://arxiv.org/abs/2402.16832)
• Keywords: interpretability, large language model
• Abstract: Multimodal large language models (MLLMs) like LLaVA and GPT-4(V) enable general-purpose conversations about images with the language modality. As off-the-shelf MLLMs may have limited capabilities on images from domains like dermatology and agriculture, they must be fine-tuned to unlock domain-specific applications. The prevalent architecture of current open-source MLLMs comprises two major modules: an image-language (cross-modal) projection network and a large language model. It is desirable to understand the roles of these two modules in modeling domain-specific visual attributes to inform the design of future models and streamline the interpretability efforts on the current models. To this end, via experiments on 4 datasets and under 2 fine-tuning settings, we find that as the MLLM is fine-tuned, it indeed gains domain-specific visual capabilities, but the updates do not lead to the projection extracting relevant domain-specific visual attributes. Our results indicate that the domain-specific visual attributes are modeled by the LLM, even when only the projection is fine-tuned. Through this study, we offer a potential reinterpretation of the role of cross-modal projections in MLLM architectures. Projection webpage: https://claws-lab.github.io/projection-in-MLLMs/

Title: Eight Methods to Evaluate Robust Unlearning in LLMs

• Authors: Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, Dylan Hadfield-Menell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16835
• Pdf URL: https://arxiv.org/pdf/2402.16835
• Copy Paste: [[2402.16835]] Eight Methods to Evaluate Robust Unlearning in LLMs(https://arxiv.org/abs/2402.16835)
• Keywords: robust, large language model
• Abstract: Machine unlearning can be useful for removing harmful capabilities and memorized text from large language models (LLMs), but there are not yet standardized methods for rigorously evaluating it. In this paper, we first survey techniques and limitations of existing unlearning evaluations. Second, we apply a comprehensive set of tests for the robustness and competitiveness of unlearning in the "Who's Harry Potter" (WHP) model from Eldan and Russinovich (2023). While WHP's unlearning generalizes well when evaluated with the "Familiarity" metric from Eldan and Russinovich, we find i) higher-than-baseline amounts of knowledge can reliably be extracted, ii) WHP performs on par with the original model on Harry Potter Q&A tasks, iii) it represents latent knowledge comparably to the original model, and iv) there is collateral unlearning in related domains. Overall, our results highlight the importance of comprehensive unlearning evaluation that avoids ad-hoc metrics.

Title: Do Large Language Models Latently Perform Multi-Hop Reasoning?

• Authors: Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, Sebastian Riedel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16837
• Pdf URL: https://arxiv.org/pdf/2402.16837
• Copy Paste: [[2402.16837]] Do Large Language Models Latently Perform Multi-Hop Reasoning?(https://arxiv.org/abs/2402.16837)
• Keywords: large language model
• Abstract: We study whether Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of 'Superstition' is". We look for evidence of a latent reasoning pathway where an LLM (1) latently identifies "the singer of 'Superstition'" as Stevie Wonder, the bridge entity, and (2) uses its knowledge of Stevie Wonder's mother to complete the prompt. We analyze these two hops individually and consider their co-occurrence as indicative of latent multi-hop reasoning. For the first hop, we test if changing the prompt to indirectly mention the bridge entity instead of any other entity increases the LLM's internal recall of the bridge entity. For the second hop, we test if increasing this recall causes the LLM to better utilize what it knows about the bridge entity. We find strong evidence of latent multi-hop reasoning for the prompts of certain relation types, with the reasoning pathway used in more than 80% of the prompts. However, the utilization is highly contextual, varying across different types of prompts. Also, on average, the evidence for the second hop and the full multi-hop traversal is rather moderate and only substantial for the first hop. Moreover, we find a clear scaling trend with increasing model size for the first hop of reasoning but not for the second hop. Our experimental findings suggest potential challenges and opportunities for future development and applications of LLMs.

Title: MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT

• Authors: Omkar Thawakar, Ashmal Vayani, Salman Khan, Hisham Cholakal, Rao M. Anwer, Michael Felsberg, Tim Baldwin, Eric P. Xing, Fahad Shahbaz Khan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16840
• Pdf URL: https://arxiv.org/pdf/2402.16840
• Copy Paste: [[2402.16840]] MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT(https://arxiv.org/abs/2402.16840)
• Keywords: security, privacy, large language model
• Abstract: "Bigger the better" has been the predominant trend in recent Large Language Models (LLMs) development. However, LLMs do not suit well for scenarios that require on-device processing, energy efficiency, low memory footprint, and response efficiency. These requisites are crucial for privacy, security, and sustainable deployment. This paper explores the "less is more" paradigm by addressing the challenge of designing accurate yet efficient Small Language Models (SLMs) for resource constrained devices. Our primary contribution is the introduction of an accurate and fully transparent open-source 0.5 billion (0.5B) parameter SLM, named MobiLlama, catering to the specific needs of resource-constrained computing with an emphasis on enhanced performance with reduced resource demands. MobiLlama is a SLM design that initiates from a larger model and applies a careful parameter sharing scheme to reduce both the pre-training and the deployment cost. Our work strives to not only bridge the gap in open-source SLMs but also ensures full transparency, where complete training data pipeline, training code, model weights, and over 300 checkpoints along with evaluation codes is available at : https://github.com/mbzuai-oryx/MobiLlama.

Title: Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding

• Authors: Benjamin Bergner, Andrii Skliar, Amelie Royer, Tijmen Blankevoort, Yuki Asano, Babak Ehteshami Bejnordi
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16844
• Pdf URL: https://arxiv.org/pdf/2402.16844
• Copy Paste: [[2402.16844]] Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding(https://arxiv.org/abs/2402.16844)
• Keywords: large language model
• Abstract: Large language models (LLMs) have become ubiquitous in practice and are widely used for generation tasks such as translation, summarization and instruction following. However, their enormous size and reliance on autoregressive decoding increase deployment costs and complicate their use in latency-critical applications. In this work, we propose a hybrid approach that combines language models of different sizes to increase the efficiency of autoregressive decoding while maintaining high performance. Our method utilizes a pretrained frozen LLM that encodes all prompt tokens once in parallel, and uses the resulting representations to condition and guide a small language model (SLM), which then generates the response more efficiently. We investigate the combination of encoder-decoder LLMs with both encoder-decoder and decoder-only SLMs from different model families and only require fine-tuning of the SLM. Experiments with various benchmarks show substantial speedups of up to $4\times$, with minor performance penalties of $1-2\%$ for translation and summarization tasks compared to the LLM.

Title: GROUNDHOG: Grounding Large Language Models to Holistic Segmentation

• Authors: Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, Joyce Chai
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16846
• Pdf URL: https://arxiv.org/pdf/2402.16846
• Copy Paste: [[2402.16846]] GROUNDHOG: Grounding Large Language Models to Holistic Segmentation(https://arxiv.org/abs/2402.16846)
• Keywords: large language model, segmentation
• Abstract: Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens. This paradigm lacks pixel-level representations that are important for fine-grained visual understanding and diagnosis. In this work, we introduce GROUNDHOG, an MLLM developed by grounding Large Language Models to holistic segmentation. GROUNDHOG incorporates a masked feature extractor and converts extracted features into visual entity tokens for the MLLM backbone, which then connects groundable phrases to unified grounding masks by retrieving and merging the entity masks. To train GROUNDHOG, we carefully curated M3G2, a grounded visual instruction tuning dataset with Multi-Modal Multi-Grained Grounding, by harvesting a collection of segmentation-grounded datasets with rich annotations. Our experimental results show that GROUNDHOG achieves superior performance on various language grounding tasks without task-specific fine-tuning, and significantly reduces object hallucination. GROUNDHOG also demonstrates better grounding towards complex forms of visual input and provides easy-to-understand diagnosis in failure cases.

Title: InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learning

• Authors: Babak Ehteshami Bejnordi, Gaurav Kumar, Amelie Royer, Christos Louizos, Tijmen Blankevoort, Mohsen Ghafoorian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16848
• Pdf URL: https://arxiv.org/pdf/2402.16848
• Copy Paste: [[2402.16848]] InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learning(https://arxiv.org/abs/2402.16848)
• Keywords: transformer
• Abstract: Jointly learning multiple tasks with a unified model can improve accuracy and data efficiency, but it faces the challenge of task interference, where optimizing one task objective may inadvertently compromise the performance of another. A solution to mitigate this issue is to allocate task-specific parameters, free from interference, on top of shared features. However, manually designing such architectures is cumbersome, as practitioners need to balance between the overall performance across all tasks and the higher computational cost induced by the newly added parameters. In this work, we propose \textit{InterroGate}, a novel multi-task learning (MTL) architecture designed to mitigate task interference while optimizing inference computational efficiency. We employ a learnable gating mechanism to automatically balance the shared and task-specific representations while preserving the performance of all tasks. Crucially, the patterns of parameter sharing and specialization dynamically learned during training, become fixed at inference, resulting in a static, optimized MTL architecture. Through extensive empirical evaluations, we demonstrate SoTA results on three MTL benchmarks using convolutional as well as transformer-based backbones on CelebA, NYUD-v2, and PASCAL-Context.