2024-01-31

Title: Do deep neural networks utilize the weight space efficiently?

• Authors: Onur Can Koyun, Behçet Uğur Töreyin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16438
• Pdf URL: https://arxiv.org/pdf/2401.16438
• Copy Paste: [[2401.16438]] Do deep neural networks utilize the weight space efficiently?(https://arxiv.org/abs/2401.16438)
• Keywords: transformer
• Abstract: Deep learning models like Transformers and Convolutional Neural Networks (CNNs) have revolutionized various domains, but their parameter-intensive nature hampers deployment in resource-constrained settings. In this paper, we introduce a novel concept utilizes column space and row space of weight matrices, which allows for a substantial reduction in model parameters without compromising performance. Leveraging this paradigm, we achieve parameter-efficient deep learning models.. Our approach applies to both Bottleneck and Attention layers, effectively halving the parameters while incurring only minor performance degradation. Extensive experiments conducted on the ImageNet dataset with ViT and ResNet50 demonstrate the effectiveness of our method, showcasing competitive performance when compared to traditional models. This approach not only addresses the pressing demand for parameter efficient deep learning solutions but also holds great promise for practical deployment in real-world scenarios.

Title: Polynomial time auditing of statistical subgroup fairness for Gaussian data

• Authors: Daniel Hsu, Jizhou Huang, Brendan Juba
• Subjects: cs.LG, cs.CC, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.16439
• Pdf URL: https://arxiv.org/pdf/2401.16439
• Copy Paste: [[2401.16439]] Polynomial time auditing of statistical subgroup fairness for Gaussian data(https://arxiv.org/abs/2401.16439)
• Keywords: fair
• Abstract: We study the problem of auditing classifiers with the notion of statistical subgroup fairness. Kearns et al. (2018) has shown that the problem of auditing combinatorial subgroups fairness is as hard as agnostic learning. Essentially all work on remedying statistical measures of discrimination against subgroups assumes access to an oracle for this problem, despite the fact that no efficient algorithms are known for it. If we assume the data distribution is Gaussian, or even merely log-concave, then a recent line of work has discovered efficient agnostic learning algorithms for halfspaces. Unfortunately, the boosting-style reductions given by Kearns et al. required the agnostic learning algorithm to succeed on reweighted distributions that may not be log-concave, even if the original data distribution was. In this work, we give positive and negative results on auditing for the Gaussian distribution: On the positive side, we an alternative approach to leverage these advances in agnostic learning and thereby obtain the first polynomial-time approximation scheme (PTAS) for auditing nontrivial combinatorial subgroup fairness: we show how to audit statistical notions of fairness over homogeneous halfspace subgroups when the features are Gaussian. On the negative side, we find that under cryptographic assumptions, no polynomial-time algorithm can guarantee any nontrivial auditing, even under Gaussian feature distributions, for general halfspace subgroups.

Title: Context-Former: Stitching via Latent Conditioned Sequence Modeling

• Authors: Ziqi Zhang, Jingzehua Xu, Zifeng Zhuang, Jinxin Liu, Donglin wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16452
• Pdf URL: https://arxiv.org/pdf/2401.16452
• Copy Paste: [[2401.16452]] Context-Former: Stitching via Latent Conditioned Sequence Modeling(https://arxiv.org/abs/2401.16452)
• Keywords: transformer
• Abstract: Offline reinforcement learning (RL) algorithms can improve the decision making via stitching sub-optimal trajectories to obtain more optimal ones. This capability is a crucial factor in enabling RL to learn policies that are superior to the behavioral policy. On the other hand, Decision Transformer (DT) abstracts the decision-making as sequence modeling, showcasing competitive performance on offline RL benchmarks, however, recent studies demonstrate that DT lacks of stitching capability, thus exploit stitching capability for DT is vital to further improve its performance. In order to endow stitching capability to DT, we abstract trajectory stitching as expert matching and introduce our approach, ContextFormer, which integrates contextual information-based imitation learning (IL) and sequence modeling to stitch sub-optimal trajectory fragments by emulating the representations of a limited number of expert trajectories. To validate our claim, we conduct experiments from two perspectives: 1) We conduct extensive experiments on D4RL benchmarks under the settings of IL, and experimental results demonstrate ContextFormer can achieve competitive performance in multi-IL settings. 2) More importantly, we conduct a comparison of ContextFormer with diverse competitive DT variants using identical training datasets. The experimental results unveiled ContextFormer's superiority, as it outperformed all other variants, showcasing its remarkable performance.

Title: Hybrid Transformer and Spatial-Temporal Self-Supervised Learning for Long-term Traffic Prediction

• Authors: Wang Zhu, Doudou Zhang, Baichao Long, Jianli Xiao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16453
• Pdf URL: https://arxiv.org/pdf/2401.16453
• Copy Paste: [[2401.16453]] Hybrid Transformer and Spatial-Temporal Self-Supervised Learning for Long-term Traffic Prediction(https://arxiv.org/abs/2401.16453)
• Keywords: robust, transformer
• Abstract: Long-term traffic prediction has always been a challenging task due to its dynamic temporal dependencies and complex spatial dependencies. In this paper, we propose a model that combines hybrid Transformer and spatio-temporal self-supervised learning. The model enhances its robustness by applying adaptive data augmentation techniques at the sequence-level and graph-level of the traffic data. It utilizes Transformer to overcome the limitations of recurrent neural networks in capturing long-term sequences, and employs Chebyshev polynomial graph convolution to capture complex spatial dependencies. Furthermore, considering the impact of spatio-temporal heterogeneity on traffic speed, we design two self-supervised learning tasks to model the temporal and spatial heterogeneity, thereby improving the accuracy and generalization ability of the model. Experimental evaluations are conducted on two real-world datasets, PeMS04 and PeMS08, and the results are visualized and analyzed, demonstrating the superior performance of the proposed model.

Title: SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design

• Authors: Seokju Yun, Youngmin Ro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16456
• Pdf URL: https://arxiv.org/pdf/2401.16456
• Copy Paste: [[2401.16456]] SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design(https://arxiv.org/abs/2401.16456)
• Keywords: transformer, segmentation
• Abstract: Recently, efficient Vision Transformers have shown great performance with low latency on resource-constrained devices. Conventionally, they use 4x4 patch embeddings and a 4-stage structure at the macro level, while utilizing sophisticated attention with multi-head configuration at the micro level. This paper aims to address computational redundancy at all design levels in a memory-efficient manner. We discover that using larger-stride patchify stem not only reduces memory access costs but also achieves competitive performance by leveraging token representations with reduced spatial redundancy from the early stages. Furthermore, our preliminary analyses suggest that attention layers in the early stages can be substituted with convolutions, and several attention heads in the latter stages are computationally redundant. To handle this, we introduce a single-head attention module that inherently prevents head redundancy and simultaneously boosts accuracy by parallelly combining global and local information. Building upon our solutions, we introduce SHViT, a Single-Head Vision Transformer that obtains the state-of-the-art speed-accuracy tradeoff. For example, on ImageNet-1k, our SHViT-S4 is 3.3x, 8.1x, and 2.4x faster than MobileViTv2 x1.0 on GPU, CPU, and iPhone12 mobile device, respectively, while being 1.3% more accurate. For object detection and instance segmentation on MS COCO using Mask-RCNN head, our model achieves performance comparable to FastViT-SA12 while exhibiting 3.8x and 2.0x lower backbone latency on GPU and mobile device, respectively.

Title: Effective Controllable Bias Mitigation for Classification and Retrieval using Gate Adapters

• Authors: Shahed Masoudian, Cornelia Volaucnik, Markus Schedl, Shahed Masoudian
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.16457
• Pdf URL: https://arxiv.org/pdf/2401.16457
• Copy Paste: [[2401.16457]] Effective Controllable Bias Mitigation for Classification and Retrieval using Gate Adapters(https://arxiv.org/abs/2401.16457)
• Keywords: protect, fair, interpretability
• Abstract: Bias mitigation of Language Models has been the topic of many studies with a recent focus on learning separate modules like adapters for on-demand debiasing. Besides optimizing for a modularized debiased model, it is often critical in practice to control the degree of bias reduction at inference time, e.g., in order to tune for a desired performance-fairness trade-off in search results or to control the strength of debiasing in classification tasks. In this paper, we introduce Controllable Gate Adapter (ConGater), a novel modular gating mechanism with adjustable sensitivity parameters, which allows for a gradual transition from the biased state of the model to the fully debiased version at inference time. We demonstrate ConGater performance by (1) conducting adversarial debiasing experiments with three different models on three classification tasks with four protected attributes, and (2) reducing the bias of search results through fairness list-wise regularization to enable adjusting a trade-off between performance and fairness metrics. Our experiments on the classification tasks show that compared to baselines of the same caliber, ConGater can maintain higher task performance while containing less information regarding the attributes. Our results on the retrieval task show that the fully debiased ConGater can achieve the same fairness performance while maintaining more than twice as high task performance than recent strong baselines. Overall, besides strong performance ConGater enables the continuous transitioning between biased and debiased states of models, enhancing personalization of use and interpretability through controllability.

Title: Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors

• Authors: Shiyin Dong, Mingrui Zhu, Kun Cheng, Nannan Wang, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16459
• Pdf URL: https://arxiv.org/pdf/2401.16459
• Copy Paste: [[2401.16459]] Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors(https://arxiv.org/abs/2401.16459)
• Keywords: robust, diffusion, generative, segmentation
• Abstract: The remarkable prowess of diffusion models in image generation has spurred efforts to extend their application beyond generative tasks. However, a persistent challenge exists in lacking a unified approach to apply diffusion models to visual perception tasks with diverse semantic granularity requirements. Our purpose is to establish a unified visual perception framework, capitalizing on the potential synergies between generative and discriminative models. In this paper, we propose Vermouth, a simple yet effective framework comprising a pre-trained Stable Diffusion (SD) model containing rich generative priors, a unified head (U-head) capable of integrating hierarchical representations, and an adapted expert providing discriminative priors. Comprehensive investigations unveil potential characteristics of Vermouth, such as varying granularity of perception concealed in latent variables at distinct time steps and various U-net stages. We emphasize that there is no necessity for incorporating a heavyweight or intricate decoder to transform diffusion models into potent representation learners. Extensive comparative evaluations against tailored discriminative models showcase the efficacy of our approach on zero-shot sketch-based image retrieval (ZS-SBIR), few-shot classification, and open-vocabulary semantic segmentation tasks. The promising results demonstrate the potential of diffusion models as formidable learners, establishing their significance in furnishing informative and robust visual representations.

Title: DressCode: Autoregressively Sewing and Generating Garments from Text Guidance

• Authors: Kai He, Kaixin Yao, Qixuan Zhang, Jingyi Yu, Lingjie Liu, Lan Xu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.16465
• Pdf URL: https://arxiv.org/pdf/2401.16465
• Copy Paste: [[2401.16465]] DressCode: Autoregressively Sewing and Generating Garments from Text Guidance(https://arxiv.org/abs/2401.16465)
• Keywords: diffusion, large language model
• Abstract: Apparel's significant role in human appearance underscores the importance of garment digitalization for digital human creation. Recent advances in 3D content creation are pivotal for digital human creation. Nonetheless, garment generation from text guidance is still nascent. We introduce a text-driven 3D garment generation framework, DressCode, which aims to democratize design for novices and offer immense potential in fashion design, virtual try-on, and digital human creation. For our framework, we first introduce SewingGPT, a GPT-based architecture integrating cross-attention with text-conditioned embedding to generate sewing patterns with text guidance. We also tailored a pre-trained Stable Diffusion for high-quality, tile-based PBR texture generation. By leveraging a large language model, our framework generates CG-friendly garments through natural language interaction. Our method also facilitates pattern completion and texture editing, simplifying the process for designers by user-friendly interaction. With comprehensive evaluations and comparisons with other state-of-the-art methods, our method showcases the best quality and alignment with input prompts. User studies further validate our high-quality rendering results, highlighting its practical utility and potential in production settings.

Title: A Discriminative Bayesian Gaussian Process Latent Variable Model for High-Dimensional Data

• Authors: Navid Ziaei, Behzad Nazari, Ali Yousefi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16497
• Pdf URL: https://arxiv.org/pdf/2401.16497
• Copy Paste: [[2401.16497]] A Discriminative Bayesian Gaussian Process Latent Variable Model for High-Dimensional Data(https://arxiv.org/abs/2401.16497)
• Keywords: robust, generative
• Abstract: Extracting meaningful information from high-dimensional data poses a formidable modeling challenge, particularly when the data is obscured by noise or represented through different modalities. In this research, we propose a novel non-parametric modeling approach, leveraging the Gaussian Process (GP), to characterize high-dimensional data by mapping it to a latent low-dimensional manifold. This model, named the Latent Discriminative Generative Decoder (LDGD), utilizes both the data (or its features) and associated labels (such as category or stimulus) in the manifold discovery process. To infer the latent variables, we derive a Bayesian solution, allowing LDGD to effectively capture inherent uncertainties in the data while enhancing the model's predictive accuracy and robustness. We demonstrate the application of LDGD on both synthetic and benchmark datasets. Not only does LDGD infer the manifold accurately, but its prediction accuracy in anticipating labels surpasses state-of-the-art approaches. We have introduced inducing points to reduce the computational complexity of Gaussian Processes (GPs) for large datasets. This enhancement facilitates batch training, allowing for more efficient processing and scalability in handling extensive data collections. Additionally, we illustrate that LDGD achieves higher accuracy in predicting labels and operates effectively with a limited training dataset, underscoring its efficiency and effectiveness in scenarios where data availability is constrained. These attributes set the stage for the development of non-parametric modeling approaches in the analysis of high-dimensional data; especially in fields where data are both high-dimensional and complex.

Title: MT-HCCAR: Multi-Task Deep Learning with Hierarchical Classification and Attention-based Regression for Cloud Property Retrieval

• Authors: Xingyan Li, Andrew M. Sayer, Ian T. Carroll, Xin Huang, Jianwu Wang
• Subjects: cs.LG, cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.16520
• Pdf URL: https://arxiv.org/pdf/2401.16520
• Copy Paste: [[2401.16520]] MT-HCCAR: Multi-Task Deep Learning with Hierarchical Classification and Attention-based Regression for Cloud Property Retrieval(https://arxiv.org/abs/2401.16520)
• Keywords: robust
• Abstract: In the realm of Earth science, effective cloud property retrieval, encompassing cloud masking, cloud phase classification, and cloud optical thickness (COT) prediction, remains pivotal. Traditional methodologies necessitate distinct models for each sensor instrument due to their unique spectral characteristics. Recent strides in Earth Science research have embraced machine learning and deep learning techniques to extract features from satellite datasets' spectral observations. However, prevailing approaches lack novel architectures accounting for hierarchical relationships among retrieval tasks. Moreover, considering the spectral diversity among existing sensors, the development of models with robust generalization capabilities over different sensor datasets is imperative. Surprisingly, there is a dearth of methodologies addressing the selection of an optimal model for diverse datasets. In response, this paper introduces MT-HCCAR, an end-to-end deep learning model employing multi-task learning to simultaneously tackle cloud masking, cloud phase retrieval (classification tasks), and COT prediction (a regression task). The MT-HCCAR integrates a hierarchical classification network (HC) and a classification-assisted attention-based regression network (CAR), enhancing precision and robustness in cloud labeling and COT prediction. Additionally, a comprehensive model selection method rooted in K-fold cross-validation, one standard error rule, and two introduced performance scores is proposed to select the optimal model over three simulated satellite datasets OCI, VIIRS, and ABI. The experiments comparing MT-HCCAR with baseline methods, the ablation studies, and the model selection affirm the superiority and the generalization capabilities of MT-HCCAR.

Title: Validation, Robustness, and Accuracy of Perturbation-Based Sensitivity Analysis Methods for Time-Series Deep Learning Models

• Authors: Zhengguang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16521
• Pdf URL: https://arxiv.org/pdf/2401.16521
• Copy Paste: [[2401.16521]] Validation, Robustness, and Accuracy of Perturbation-Based Sensitivity Analysis Methods for Time-Series Deep Learning Models(https://arxiv.org/abs/2401.16521)
• Keywords: robust, interpretability, transformer
• Abstract: This work undertakes studies to evaluate Interpretability Methods for Time-Series Deep Learning. Sensitivity analysis assesses how input changes affect the output, constituting a key component of interpretation. Among the post-hoc interpretation methods such as back-propagation, perturbation, and approximation, my work will investigate perturbation-based sensitivity Analysis methods on modern Transformer models to benchmark their performances. Specifically, my work answers three research questions: 1) Do different sensitivity analysis (SA) methods yield comparable outputs and attribute importance rankings? 2) Using the same sensitivity analysis method, do different Deep Learning (DL) models impact the output of the sensitivity analysis? 3) How well do the results from sensitivity analysis methods align with the ground truth?

Title: Efficient Observation Time Window Segmentation for Administrative Data Machine Learning

• Authors: Musa Taib, Geoffrey G. Messier
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.16537
• Pdf URL: https://arxiv.org/pdf/2401.16537
• Copy Paste: [[2401.16537]] Efficient Observation Time Window Segmentation for Administrative Data Machine Learning(https://arxiv.org/abs/2401.16537)
• Keywords: segmentation
• Abstract: Utilizing administrative data to predict outcomes is an important application area of machine learning, particularly in healthcare. Most administrative data records are timestamped and the pattern of records over time is a key input for machine learning models. This paper explores how best to divide the observation window of a machine learning model into time segments or "bins". A computationally efficient process is presented that identifies which data features benefit most from smaller, higher resolution time segments. Results generated on healthcare and housing/homelessness administrative data demonstrate that optimizing the time bin size of these high priority features while using a single time bin for the other features achieves machine learning models that are simpler and quicker to train. This approach also achieves similar and sometimes better performance than more complex models that default to representing all data features with the same time resolution.

Title: GuReT: Distinguishing Guilt and Regret related Text

• Authors: Sabur Butt, Fazlourrahman Balouchzahi, Abdul Gafar Manuel Meque, Maaz Amjad, Hector G. Ceballos Cancino, Grigori Sidorov, Alexander Gelbukh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16541
• Pdf URL: https://arxiv.org/pdf/2401.16541
• Copy Paste: [[2401.16541]] GuReT: Distinguishing Guilt and Regret related Text(https://arxiv.org/abs/2401.16541)
• Keywords: transformer
• Abstract: The intricate relationship between human decision-making and emotions, particularly guilt and regret, has significant implications on behavior and well-being. Yet, these emotions subtle distinctions and interplay are often overlooked in computational models. This paper introduces a dataset tailored to dissect the relationship between guilt and regret and their unique textual markers, filling a notable gap in affective computing research. Our approach treats guilt and regret recognition as a binary classification task and employs three machine learning and six transformer-based deep learning techniques to benchmark the newly created dataset. The study further implements innovative reasoning methods like chain-of-thought and tree-of-thought to assess the models interpretive logic. The results indicate a clear performance edge for transformer-based models, achieving a 90.4% macro F1 score compared to the 85.3% scored by the best machine learning classifier, demonstrating their superior capability in distinguishing complex emotional states.

Title: Deep Learning for Multi-Label Learning: A Comprehensive Survey

• Authors: Adane Nega Tarekegn, Mohib Ullah, Faouzi Alaya Cheikh
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16549
• Pdf URL: https://arxiv.org/pdf/2401.16549
• Copy Paste: [[2401.16549]] Deep Learning for Multi-Label Learning: A Comprehensive Survey(https://arxiv.org/abs/2401.16549)
• Keywords: robust, transformer
• Abstract: Multi-label learning is a rapidly growing research area that aims to predict multiple labels from a single input data point. In the era of big data, tasks involving multi-label classification (MLC) or ranking present significant and intricate challenges, capturing considerable attention in diverse domains. Inherent difficulties in MLC include dealing with high-dimensional data, addressing label correlations, and handling partial labels, for which conventional methods prove ineffective. Recent years have witnessed a notable increase in adopting deep learning (DL) techniques to address these challenges more effectively in MLC. Notably, there is a burgeoning effort to harness the robust learning capabilities of DL for improved modelling of label dependencies and other challenges in MLC. However, it is noteworthy that comprehensive studies specifically dedicated to DL for multi-label learning are limited. Thus, this survey aims to thoroughly review recent progress in DL for multi-label learning, along with a summary of open research problems in MLC. The review consolidates existing research efforts in DL for MLC,including deep neural networks, transformers, autoencoders, and convolutional and recurrent architectures. Finally, the study presents a comparative analysis of the existing methods to provide insightful observations and stimulate future research directions in this domain.

Title: SelectLLM: Can LLMs Select Important Instructions to Annotate?

• Authors: Ritik Sachin Parkar, Jaehyung Kim, Jong Inn Park, Dongyeop Kang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16553
• Pdf URL: https://arxiv.org/pdf/2401.16553
• Copy Paste: [[2401.16553]] SelectLLM: Can LLMs Select Important Instructions to Annotate?(https://arxiv.org/abs/2401.16553)
• Keywords: large language model
• Abstract: Training large language models (LLMs) with a large and diverse instruction dataset aligns the models to comprehend and follow human instructions. Recent works have shown that using a small set of high-quality instructions can outperform using large yet more noisy ones. Because instructions are unlabeled and their responses are natural text, traditional active learning schemes with the model's confidence cannot be directly applied to the selection of unlabeled instructions. In this work, we propose a novel method for instruction selection, called SelectLLM, that leverages LLMs for the selection of high-quality instructions. Our high-level idea is to use LLMs to estimate the usefulness and impactfulness of each instruction without the corresponding labels (i.e., responses), via prompting. SelectLLM involves two steps: dividing the unlabelled instructions using a clustering algorithm (e.g., CoreSet) to multiple clusters, and then prompting LLMs to choose high-quality instructions within each cluster. SelectLLM showed comparable or slightly better performance on the popular instruction benchmarks, compared to the recent state-of-the-art selection methods. All code and data are publicly available (https://github.com/minnesotanlp/select-llm).

Title: IEEE BigData 2023 Keystroke Verification Challenge (KVC)

• Authors: Giuseppe Stragapede, Ruben Vera-Rodriguez, Ruben Tolosana, Aythami Morales, Ivan DeAndres-Tame, Naser Damer, Julian Fierrez, Javier-Ortega Garcia, Nahuel Gonzalez, Andrei Shadrikov, Dmitrii Gordin, Leon Schmitt, Daniel Wimmer, Christoph Grossmann, Joerdis Krieger, Florian Heinz, Ron Krestel, Christoffer Mayer, Simon Haberl, Helena Gschrey, Yosuke Yamagishi, Sanjay Saha, Sanka Rasnayaka, Sandareka Wickramanayake, Terence Sim, Weronika Gutfeter, Adam Baran, Mateusz Krzyszton, Przemyslaw Jaskola
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16559
• Pdf URL: https://arxiv.org/pdf/2401.16559
• Copy Paste: [[2401.16559]] IEEE BigData 2023 Keystroke Verification Challenge (KVC)(https://arxiv.org/abs/2401.16559)
• Keywords: biometric
• Abstract: This paper describes the results of the IEEE BigData 2023 Keystroke Verification Challenge (KVC), that considers the biometric verification performance of Keystroke Dynamics (KD), captured as tweet-long sequences of variable transcript text from over 185,000 subjects. The data are obtained from two of the largest public databases of KD up to date, the Aalto Desktop and Mobile Keystroke Databases, guaranteeing a minimum amount of data per subject, age and gender annotations, absence of corrupted data, and avoiding excessively unbalanced subject distributions with respect to the considered demographic attributes. Several neural architectures were proposed by the participants, leading to global Equal Error Rates (EERs) as low as 3.33% and 3.61% achieved by the best team respectively in the desktop and mobile scenario, outperforming the current state of the art biometric verification performance for KD. Hosted on CodaLab, the KVC will be made ongoing to represent a useful tool for the research community to compare different approaches under the same experimental conditions and to deepen the knowledge of the field.

Title: Autoencoder-Based Domain Learning for Semantic Communication with Conceptual Spaces

• Authors: Dylan Wheeler, Balasubramaniam Natarajan
• Subjects: cs.LG, cs.AI, cs.CL, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.16569
• Pdf URL: https://arxiv.org/pdf/2401.16569
• Copy Paste: [[2401.16569]] Autoencoder-Based Domain Learning for Semantic Communication with Conceptual Spaces(https://arxiv.org/abs/2401.16569)
• Keywords: robust
• Abstract: Communication with the goal of accurately conveying meaning, rather than accurately transmitting symbols, has become an area of growing interest. This paradigm, termed semantic communication, typically leverages modern developments in artificial intelligence and machine learning to improve the efficiency and robustness of communication systems. However, a standard model for capturing and quantifying the details of "meaning" is lacking, with many leading approaches to semantic communication adopting a black-box framework with little understanding of what exactly the model is learning. One solution is to utilize the conceptual spaces framework, which models meaning explicitly in a geometric manner. Though prior work studying semantic communication with conceptual spaces has shown promising results, these previous attempts involve hand-crafting a conceptual space model, severely limiting the scalability and practicality of the approach. In this work, we develop a framework for learning a domain of a conceptual space model using only the raw data with high-level property labels. In experiments using the MNIST and CelebA datasets, we show that the domains learned using the framework maintain semantic similarity relations and possess interpretable dimensions.

Title: Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Masking

• Authors: Ivana Beňová, Jana Košecká, Michal Gregor, Martin Tamajka, Marcel Veselý, Marián Šimko
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16575
• Pdf URL: https://arxiv.org/pdf/2401.16575
• Copy Paste: [[2401.16575]] Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Masking(https://arxiv.org/abs/2401.16575)
• Keywords: transformer
• Abstract: The dominant probing approaches rely on the zero-shot performance of image-text matching tasks to gain a finer-grained understanding of the representations learned by recent multimodal image-language transformer models. The evaluation is carried out on carefully curated datasets focusing on counting, relations, attributes, and others. This work introduces an alternative probing strategy called guided masking. The proposed approach ablates different modalities using masking and assesses the model's ability to predict the masked word with high accuracy. We focus on studying multimodal models that consider regions of interest (ROI) features obtained by object detectors as input tokens. We probe the understanding of verbs using guided masking on ViLBERT, LXMERT, UNITER, and VisualBERT and show that these models can predict the correct verb with high accuracy. This contrasts with previous conclusions drawn from image-text matching probing techniques that frequently fail in situations requiring verb understanding. The code for all experiments will be publicly available https://github.com/ivana-13/guided_masking.

Title: LLMs as On-demand Customizable Service

• Authors: Souvika Sarkar, Mohammad Fakhruddin Babar, Monowar Hasan, Shubhra Kanti Karmaker (Santu)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16577
• Pdf URL: https://arxiv.org/pdf/2401.16577
• Copy Paste: [[2401.16577]] LLMs as On-demand Customizable Service(https://arxiv.org/abs/2401.16577)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable language understanding and generation capabilities. However, training, deploying, and accessing these models pose notable challenges, including resource-intensive demands, extended training durations, and scalability issues. To address these issues, we introduce a concept of hierarchical, distributed LLM architecture that aims at enhancing the accessibility and deployability of LLMs across heterogeneous computing platforms, including general-purpose computers (e.g., laptops) and IoT-style devices (e.g., embedded systems). By introducing a "layered" approach, the proposed architecture enables on-demand accessibility to LLMs as a customizable service. This approach also ensures optimal trade-offs between the available computational resources and the user's application needs. We envision that the concept of hierarchical LLM will empower extensive, crowd-sourced user bases to harness the capabilities of LLMs, thereby fostering advancements in AI technology in general.

Title: Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports

• Authors: Qingqing Zhu, Xiuying Chen, Qiao Jin, Benjamin Hou, Tejas Sudharshan Mathai, Pritam Mukherjee, Xin Gao, Ronald M Summers, Zhiyong Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16578
• Pdf URL: https://arxiv.org/pdf/2401.16578
• Copy Paste: [[2401.16578]] Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports(https://arxiv.org/abs/2401.16578)
• Keywords: robust, large language model
• Abstract: In radiology, Artificial Intelligence (AI) has significantly advanced report generation, but automatic evaluation of these AI-produced reports remains challenging. Current metrics, such as Conventional Natural Language Generation (NLG) and Clinical Efficacy (CE), often fall short in capturing the semantic intricacies of clinical contexts or overemphasize clinical details, undermining report clarity. To overcome these issues, our proposed method synergizes the expertise of professional radiologists with Large Language Models (LLMs), like GPT-3.5 and GPT-4 1. Utilizing In-Context Instruction Learning (ICIL) and Chain of Thought (CoT) reasoning, our approach aligns LLM evaluations with radiologist standards, enabling detailed comparisons between human and AI generated reports. This is further enhanced by a Regression model that aggregates sentence evaluation scores. Experimental results show that our ''Detailed GPT-4 (5-shot)'' model achieves a 0.48 score, outperforming the METEOR metric by 0.19, while our ''Regressed GPT-4'' model shows even greater alignment with expert evaluations, exceeding the best existing metric by a 0.35 margin. Moreover, the robustness of our explanations has been validated through a thorough iterative strategy. We plan to publicly release annotations from radiology experts, setting a new standard for accuracy in future assessments. This underscores the potential of our approach in enhancing the quality assessment of AI-driven medical reports.

Title: Data-Oblivious ML Accelerators using Hardware Security Extensions

• Authors: Hossam ElAtali, John Z. Jekel, Lachlan J. Gunn, N. Asokan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16583
• Pdf URL: https://arxiv.org/pdf/2401.16583
• Copy Paste: [[2401.16583]] Data-Oblivious ML Accelerators using Hardware Security Extensions(https://arxiv.org/abs/2401.16583)
• Keywords: secure, security, protect, attack
• Abstract: Outsourced computation can put client data confidentiality at risk. Existing solutions are either inefficient or insufficiently secure: cryptographic techniques like fully-homomorphic encryption incur significant overheads, even with hardware assistance, while the complexity of hardware-assisted trusted execution environments has been exploited to leak secret data. Recent proposals such as BliMe and OISA show how dynamic information flow tracking (DIFT) enforced in hardware can protect client data efficiently. They are designed to protect CPU-only workloads. However, many outsourced computing applications, like machine learning, make extensive use of accelerators. We address this gap with Dolma, which applies DIFT to the Gemmini matrix multiplication accelerator, efficiently guaranteeing client data confidentiality, even in the presence of malicious/vulnerable software and side channel attacks on the server. We show that accelerators can allow DIFT logic optimizations that significantly reduce area overhead compared with general-purpose processor architectures. Dolma is integrated with the BliMe framework to achieve end-to-end security guarantees. We evaluate Dolma on an FPGA using a ResNet-50 DNN model and show that it incurs low overheads for large configurations ($4.4\%$, $16.7\%$, $16.5\%$ for performance, resource usage and power, respectively, with a 32x32 configuration).

Title: ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks

• Authors: Bolei Ma, Ercong Nie, Shuzhou Yuan, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16589
• Pdf URL: https://arxiv.org/pdf/2401.16589
• Copy Paste: [[2401.16589]] ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks(https://arxiv.org/abs/2401.16589)
• Keywords: large language model
• Abstract: Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding. However, most previous studies primarily focused on sentence-level classification tasks, and only a few considered token-level labeling tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. In this paper, we propose Token-Level Prompt Decomposition (ToPro), which facilitates the prompt-based method for token-level sequence labeling tasks. The ToPro method decomposes an input sentence into single tokens and applies one prompt template to each token. Our experiments on multilingual NER and POS tagging datasets demonstrate that ToPro-based fine-tuning outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer, especially for languages that are typologically different from the source language English. Our method also attains state-of-the-art performance when employed with the mT5 model. Besides, our exploratory study in multilingual large language models shows that ToPro performs much better than the current in-context learning method. Overall, the performance improvements show that ToPro could potentially serve as a novel and simple benchmarking method for sequence labeling tasks.

Title: LeftoverLocals: Listening to LLM Responses Through Leaked GPU Local Memory

• Authors: Tyler Sorensen, Heidy Khlaaf
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.16603
• Pdf URL: https://arxiv.org/pdf/2401.16603
• Copy Paste: [[2401.16603]] LeftoverLocals: Listening to LLM Responses Through Leaked GPU Local Memory(https://arxiv.org/abs/2401.16603)
• Keywords: security, attack
• Abstract: This paper describes LeftoverLocals: a vulnerability that allows data recovery from GPU memory created by another process on Apple, Qualcomm, and AMD GPUs. LeftoverLocals impacts the security posture of GPU applications, with particular significance to LLMs and ML models that run on impacted GPUs. By recovering local memory, an optimized GPU memory region, we built a PoC where an attacker can listen into another user's interactive LLM session (e.g., llama.cpp) across process or container boundaries.

Title: Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

• Authors: Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, Chuang Gan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16635
• Pdf URL: https://arxiv.org/pdf/2401.16635
• Copy Paste: [[2401.16635]] Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble(https://arxiv.org/abs/2401.16635)
• Keywords: large language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is a widely adopted approach for aligning large language models with human values. However, RLHF relies on a reward model that is trained with a limited amount of human preference data, which could lead to inaccurate predictions. As a result, RLHF may produce outputs that are misaligned with human values. To mitigate this issue, we contribute a reward ensemble method that allows the reward model to make more accurate predictions. As using an ensemble of large language model-based reward models can be computationally and resource-expensive, we explore efficient ensemble methods including linear-layer ensemble and LoRA-based ensemble. Empirically, we run Best-of-$n$ and Proximal Policy Optimization with our ensembled reward models, and verify that our ensemble methods help improve the alignment performance of RLHF outputs.

Title: Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs

• Authors: Stepan Tytarenko, Mohammad Ruhul Amin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16638
• Pdf URL: https://arxiv.org/pdf/2401.16638
• Copy Paste: [[2401.16638]] Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs(https://arxiv.org/abs/2401.16638)
• Keywords: transformer
• Abstract: Fine-tuning large pre-trained language models (LLMs) on particular datasets is a commonly employed strategy in Natural Language Processing (NLP) classification tasks. However, this approach usually results in a loss of models generalizability. In this paper, we present a framework that allows for maintaining generalizability, and enhances the performance on the downstream task by utilizing task-specific context attribution. We show that a linear transformation of the text representation from any transformer model using the task-specific concept operator results in a projection onto the latent concept space, referred to as context attribution in this paper. The specific concept operator is optimized during the supervised learning stage via novel loss functions. The proposed framework demonstrates that context attribution of the text representation for each task objective can improve the capacity of the discriminator function and thus achieve better performance for the classification task. Experimental results on three datasets, namely HateXplain, IMDB reviews, and Social Media Attributions, illustrate that the proposed model attains superior accuracy and generalizability. Specifically, for the non-fine-tuned BERT on the HateXplain dataset, we observe 8% improvement in accuracy and 10% improvement in F1-score. Whereas for the IMDB dataset, fine-tuned state-of-the-art XLNet is outperformed by 1% for both accuracy and F1-score. Furthermore, in an out-of-domain cross-dataset test, DistilBERT fine-tuned on the IMDB dataset in conjunction with the proposed model improves the F1-score on the HateXplain dataset by 7%. For the Social Media Attributions dataset of YouTube comments, we observe 5.2% increase in F1-metric. The proposed framework is implemented with PyTorch and provided open-source on GitHub.

Title: TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese

• Authors: Nicholas Kluge Corrêa, Sophia Falk, Shiza Fatimah, Aniket Sen, Nythamar de Oliveira
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16640
• Pdf URL: https://arxiv.org/pdf/2401.16640
• Copy Paste: [[2401.16640]] TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese(https://arxiv.org/abs/2401.16640)
• Keywords: large language model
• Abstract: Large language models (LLMs) have significantly advanced natural language processing, but their progress has yet to be equal across languages. While most LLMs are trained in high-resource languages like English, multilingual models generally underperform monolingual ones. Additionally, aspects of their multilingual foundation sometimes restrict the byproducts they produce, like computational demands and licensing regimes. In this study, we document the development of open-foundation models tailored for use in low-resource settings, their limitations, and their benefits. This is the TeenyTinyLlama pair: two compact models for Brazilian Portuguese text generation. We release them under the permissive Apache 2.0 license on GitHub and Hugging Face for community use and further development. See https://github.com/Nkluge-correa/TeenyTinyLlama

Title: Incoherent Probability Judgments in Large Language Models

• Authors: Jian-Qiao Zhu, Thomas L. Griffiths
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16646
• Pdf URL: https://arxiv.org/pdf/2401.16646
• Copy Paste: [[2401.16646]] Incoherent Probability Judgments in Large Language Models(https://arxiv.org/abs/2401.16646)
• Keywords: large language model
• Abstract: Autoregressive Large Language Models (LLMs) trained for next-word prediction have demonstrated remarkable proficiency at producing coherent text. But are they equally adept at forming coherent probability judgments? We use probabilistic identities and repeated judgments to assess the coherence of probability judgments made by LLMs. Our results show that the judgments produced by these models are often incoherent, displaying human-like systematic deviations from the rules of probability theory. Moreover, when prompted to judge the same event, the mean-variance relationship of probability judgments produced by LLMs shows an inverted-U-shaped like that seen in humans. We propose that these deviations from rationality can be explained by linking autoregressive LLMs to implicit Bayesian inference and drawing parallels with the Bayesian Sampler model of human probability judgments.

Title: Using Motion Forecasting for Behavior-Based Virtual Reality (VR) Authentication

• Authors: Mingjun Li, Natasha Kholgade Banerjee, Sean Banerjee
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16649
• Pdf URL: https://arxiv.org/pdf/2401.16649
• Copy Paste: [[2401.16649]] Using Motion Forecasting for Behavior-Based Virtual Reality (VR) Authentication(https://arxiv.org/abs/2401.16649)
• Keywords: biometric, transformer
• Abstract: Task-based behavioral biometric authentication of users interacting in virtual reality (VR) environments enables seamless continuous authentication by using only the motion trajectories of the person's body as a unique signature. Deep learning-based approaches for behavioral biometrics show high accuracy when using complete or near complete portions of the user trajectory, but show lower performance when using smaller segments from the start of the task. Thus, any systems designed with existing techniques are vulnerable while waiting for future segments of motion trajectories to become available. In this work, we present the first approach that predicts future user behavior using Transformer-based forecasting and using the forecasted trajectory to perform user authentication. Our work leverages the notion that given the current trajectory of a user in a task-based environment we can predict the future trajectory of the user as they are unlikely to dramatically shift their behavior since it would preclude the user from successfully completing their task goal. Using the publicly available 41-subject ball throwing dataset of Miller et al. we show improvement in user authentication when using forecasted data. When compared to no forecasting, our approach reduces the authentication equal error rate (EER) by an average of 23.85% and a maximum reduction of 36.14%.

Title: Gradient-Based Language Model Red Teaming

• Authors: Nevan Wichers, Carson Denison, Ahmad Beirami
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16656
• Pdf URL: https://arxiv.org/pdf/2401.16656
• Copy Paste: [[2401.16656]] Gradient-Based Language Model Red Teaming(https://arxiv.org/abs/2401.16656)
• Keywords: generative
• Abstract: Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach, and succeeds even when the LM has been fine-tuned to produce safer outputs.

Title: OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer

• Authors: Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, Jee-weon Jung, Shinji Watanabe
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.16658
• Pdf URL: https://arxiv.org/pdf/2401.16658
• Copy Paste: [[2401.16658]] OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer(https://arxiv.org/abs/2401.16658)
• Keywords: transformer
• Abstract: Recent studies have advocated for fully open foundation models to promote transparency and open science. As an initial step, the Open Whisper-style Speech Model (OWSM) reproduced OpenAI's Whisper using publicly available data and open-source toolkits. With the aim of reproducing Whisper, the previous OWSM v1 through v3 models were still based on Transformer, which might lead to inferior performance compared to other state-of-the-art speech encoders. In this work, we aim to improve the performance and efficiency of OWSM without extra training data. We present E-Branchformer based OWSM v3.1 models at two scales, i.e., 100M and 1B. The 1B model is the largest E-Branchformer based speech model that has been made publicly available. It outperforms the previous OWSM v3 in a vast majority of evaluation benchmarks, while demonstrating up to 25% faster inference speed. We publicly release the data preparation scripts, pre-trained models and training logs.

Title: Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection

• Authors: Liangqi Yuan, Dong-Jun Han, Su Wang, Devesh Upadhyay, Christopher G. Brinton
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.16685
• Pdf URL: https://arxiv.org/pdf/2401.16685
• Copy Paste: [[2401.16685]] Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection(https://arxiv.org/abs/2401.16685)
• Keywords: federate
• Abstract: Multimodal federated learning (FL) aims to enrich model training in FL settings where clients are collecting measurements across multiple modalities. However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings where: (i) the set of modalities collected by each client will be diverse, and (ii) communication limitations prevent clients from uploading all their locally trained modality models to the server. In this paper, we propose multimodal Federated learning with joint Modality and Client selection (mmFedMC), a new FL methodology that can tackle the above-mentioned challenges in multimodal settings. The joint selection algorithm incorporates two main components: (a) A modality selection methodology for each client, which weighs (i) the impact of the modality, gauged by Shapley value analysis, (ii) the modality model size as a gauge of communication overhead, against (iii) the frequency of modality model updates, denoted recency, to enhance generalizability. (b) A client selection strategy for the server based on the local loss of modality model at each client. Experiments on five real-world datasets demonstrate the ability of mmFedMC to achieve comparable accuracy to several baselines while reducing the communication overhead by over 20x. A demo video of our methodology is available at https://liangqiy.com/mmfedmc/.

Title: Revisiting Gradient Pruning: A Dual Realization for Defending against Gradient Attacks

• Authors: Lulu Xue, Shengshan Hu, Ruizhi Zhao, Leo Yu Zhang, Shengqing Hu, Lichao Sun, Dezhong Yao
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16687
• Pdf URL: https://arxiv.org/pdf/2401.16687
• Copy Paste: [[2401.16687]] Revisiting Gradient Pruning: A Dual Realization for Defending against Gradient Attacks(https://arxiv.org/abs/2401.16687)
• Keywords: privacy, protect, defense, attack
• Abstract: Collaborative learning (CL) is a distributed learning framework that aims to protect user privacy by allowing users to jointly train a model by sharing their gradient updates only. However, gradient inversion attacks (GIAs), which recover users' training data from shared gradients, impose severe privacy threats to CL. Existing defense methods adopt different techniques, e.g., differential privacy, cryptography, and perturbation defenses, to defend against the GIAs. Nevertheless, all current defense methods suffer from a poor trade-off between privacy, utility, and efficiency. To mitigate the weaknesses of existing solutions, we propose a novel defense method, Dual Gradient Pruning (DGP), based on gradient pruning, which can improve communication efficiency while preserving the utility and privacy of CL. Specifically, DGP slightly changes gradient pruning with a stronger privacy guarantee. And DGP can also significantly improve communication efficiency with a theoretical analysis of its convergence and generalization. Our extensive experiments show that DGP can effectively defend against the most powerful GIAs and reduce the communication cost without sacrificing the model's utility.

Title: Towards Precise 3D Human Pose Estimation with Multi-Perspective Spatial-Temporal Relational Transformers

• Authors: Jianbin Jiao, Xina Cheng, Weijie Chen, Xiaoting Yin, Hao Shi, Kailun Yang
• Subjects: cs.CV, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.16700
• Pdf URL: https://arxiv.org/pdf/2401.16700
• Copy Paste: [[2401.16700]] Towards Precise 3D Human Pose Estimation with Multi-Perspective Spatial-Temporal Relational Transformers(https://arxiv.org/abs/2401.16700)
• Keywords: transformer
• Abstract: 3D human pose estimation captures the human joint points in three-dimensional space while keeping the depth information and physical structure. That is essential for applications that require precise pose information, such as human-computer interaction, scene understanding, and rehabilitation training. Due to the challenges in data collection, mainstream datasets of 3D human pose estimation are primarily composed of multi-view video data collected in laboratory environments, which contains rich spatial-temporal correlation information besides the image frame content. Given the remarkable self-attention mechanism of transformers, capable of capturing the spatial-temporal correlation from multi-view video datasets, we propose a multi-stage framework for 3D sequence-to-sequence (seq2seq) human pose detection. Firstly, the spatial module represents the human pose feature by intra-image content, while the frame-image relation module extracts temporal relationships and 3D spatial positional relationship features between the multi-perspective images. Secondly, the self-attention mechanism is adopted to eliminate the interference from non-human body parts and reduce computing resources. Our method is evaluated on Human3.6M, a popular 3D human pose detection dataset. Experimental results demonstrate that our approach achieves state-of-the-art performance on this dataset.

Title: Multi-granularity Correspondence Learning from Long-term Noisy Videos

• Authors: Yijie Lin, Jie Zhang, Zhenyu Huang, Jia Liu, Zujie Wen, Xi Peng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16702
• Pdf URL: https://arxiv.org/pdf/2401.16702
• Copy Paste: [[2401.16702]] Multi-granularity Correspondence Learning from Long-term Noisy Videos(https://arxiv.org/abs/2401.16702)
• Keywords: robust, segmentation
• Abstract: Existing video-language studies mainly focus on learning short video clips, leaving long-term temporal dependencies rarely explored due to over-high computational cost of modeling long videos. To address this issue, one feasible solution is learning the correspondence between video clips and captions, which however inevitably encounters the multi-granularity noisy correspondence (MNC) problem. To be specific, MNC refers to the clip-caption misalignment (coarse-grained) and frame-word misalignment (fine-grained), hindering temporal learning and video understanding. In this paper, we propose NOise Robust Temporal Optimal traNsport (Norton) that addresses MNC in a unified optimal transport (OT) framework. In brief, Norton employs video-paragraph and clip-caption contrastive losses to capture long-term dependencies based on OT. To address coarse-grained misalignment in video-paragraph contrast, Norton filters out the irrelevant clips and captions through an alignable prompt bucket and realigns asynchronous clip-caption pairs based on transport distance. To address the fine-grained misalignment, Norton incorporates a soft-maximum operator to identify crucial words and key frames. Additionally, Norton exploits the potential faulty negative samples in clip-caption contrast by rectifying the alignment target with OT assignment to ensure precise temporal modeling. Extensive experiments on video retrieval, videoQA, and action segmentation verify the effectiveness of our method. Code is available at https://lin-yijie.github.io/projects/Norton.

Title: Optimal-Landmark-Guided Image Blending for Face Morphing Attacks

• Authors: Qiaoyun He, Zongyong Deng, Zuyuan He, Qijun Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16722
• Pdf URL: https://arxiv.org/pdf/2401.16722
• Copy Paste: [[2401.16722]] Optimal-Landmark-Guided Image Blending for Face Morphing Attacks(https://arxiv.org/abs/2401.16722)
• Keywords: attack
• Abstract: In this paper, we propose a novel approach for conducting face morphing attacks, which utilizes optimal-landmark-guided image blending. Current face morphing attacks can be categorized into landmark-based and generation-based approaches. Landmark-based methods use geometric transformations to warp facial regions according to averaged landmarks but often produce morphed images with poor visual quality. Generation-based methods, which employ generation models to blend multiple face images, can achieve better visual quality but are often unsuccessful in generating morphed images that can effectively evade state-of-the-art face recognition systems~(FRSs). Our proposed method overcomes the limitations of previous approaches by optimizing the morphing landmarks and using Graph Convolutional Networks (GCNs) to combine landmark and appearance features. We model facial landmarks as nodes in a bipartite graph that is fully connected and utilize GCNs to simulate their spatial and structural relationships. The aim is to capture variations in facial shape and enable accurate manipulation of facial appearance features during the warping process, resulting in morphed facial images that are highly realistic and visually faithful. Experiments on two public datasets prove that our method inherits the advantages of previous landmark-based and generation-based methods and generates morphed images with higher quality, posing a more significant threat to state-of-the-art FRSs.

Title: Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models

• Authors: Ming Shan Hee, Shivam Sharma, Rui Cao, Palash Nandi, Preslav Nakov, Tanmoy Chakraborty, Roy Ka-Wei Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16727
• Pdf URL: https://arxiv.org/pdf/2401.16727
• Copy Paste: [[2401.16727]] Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models(https://arxiv.org/abs/2401.16727)
• Keywords: large language model
• Abstract: In the evolving landscape of online communication, moderating hate speech (HS) presents an intricate challenge, compounded by the multimodal nature of digital content. This comprehensive survey delves into the recent strides in HS moderation, spotlighting the burgeoning role of large language models (LLMs) and large multimodal models (LMMs). Our exploration begins with a thorough analysis of current literature, revealing the nuanced interplay between textual, visual, and auditory elements in propagating HS. We uncover a notable trend towards integrating these modalities, primarily due to the complexity and subtlety with which HS is disseminated. A significant emphasis is placed on the advances facilitated by LLMs and LMMs, which have begun to redefine the boundaries of detection and moderation capabilities. We identify existing gaps in research, particularly in the context of underrepresented languages and cultures, and the need for solutions to handle low-resource settings. The survey concludes with a forward-looking perspective, outlining potential avenues for future research, including the exploration of novel AI methodologies, the ethical governance of AI in moderation, and the development of more nuanced, context-aware systems. This comprehensive overview aims to catalyze further research and foster a collaborative effort towards more sophisticated, responsible, and human-centric approaches to HS moderation in the digital era.\footnote{ \textcolor{red}{WARNING: This paper contains offensive examples.

Title: Widely Linear Matched Filter: A Lynchpin towards the Interpretability of Complex-valued CNNs

• Authors: Qingchen Wang, Zhe Li, Zdenka Babic, Wei Deng, Ljubiša Stanković, Danilo P. Mandic
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16729
• Pdf URL: https://arxiv.org/pdf/2401.16729
• Copy Paste: [[2401.16729]] Widely Linear Matched Filter: A Lynchpin towards the Interpretability of Complex-valued CNNs(https://arxiv.org/abs/2401.16729)
• Keywords: interpretability, explainability
• Abstract: A recent study on the interpretability of real-valued convolutional neural networks (CNNs) \cite{Stankovic_Mandic_2023CNN} has revealed a direct and physically meaningful link with the task of finding features in data through matched filters. However, applying this paradigm to illuminate the interpretability of complex-valued CNNs meets a formidable obstacle: the extension of matched filtering to a general class of noncircular complex-valued data, referred to here as the widely linear matched filter (WLMF), has been only implicit in the literature. To this end, to establish the interpretability of the operation of complex-valued CNNs, we introduce a general WLMF paradigm, provide its solution and undertake analysis of its performance. For rigor, our WLMF solution is derived without imposing any assumption on the probability density of noise. The theoretical advantages of the WLMF over its standard strictly linear counterpart (SLMF) are provided in terms of their output signal-to-noise-ratios (SNRs), with WLMF consistently exhibiting enhanced SNR. Moreover, the lower bound on the SNR gain of WLMF is derived, together with condition to attain this bound. This serves to revisit the convolution-activation-pooling chain in complex-valued CNNs through the lens of matched filtering, which reveals the potential of WLMFs to provide physical interpretability and enhance explainability of general complex-valued CNNs. Simulations demonstrate the agreement between the theoretical and numerical results.

Title: Towards Generating Informative Textual Description for Neurons in Language Models

• Authors: Shrayani Mondal, Rishabh Garodia, Arbaaz Qureshi, Taesung Lee, Youngja Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16731
• Pdf URL: https://arxiv.org/pdf/2401.16731
• Copy Paste: [[2401.16731]] Towards Generating Informative Textual Description for Neurons in Language Models(https://arxiv.org/abs/2401.16731)
• Keywords: explainability, transformer, generative
• Abstract: Recent developments in transformer-based language models have allowed them to capture a wide variety of world knowledge that can be adapted to downstream tasks with limited resources. However, what pieces of information are understood in these models is unclear, and neuron-level contributions in identifying them are largely unknown. Conventional approaches in neuron explainability either depend on a finite set of pre-defined descriptors or require manual annotations for training a secondary model that can then explain the neurons of the primary model. In this paper, we take BERT as an example and we try to remove these constraints and propose a novel and scalable framework that ties textual descriptions to neurons. We leverage the potential of generative language models to discover human-interpretable descriptors present in a dataset and use an unsupervised approach to explain neurons with these descriptors. Through various qualitative and quantitative analyses, we demonstrate the effectiveness of this framework in generating useful data-specific descriptors with little human involvement in identifying the neurons that encode these descriptors. In particular, our experiment shows that the proposed approach achieves 75% precision@2, and 50% recall@2

Title: Flash: A Hybrid Private Inference Protocol for Deep CNNs with High Accuracy and Low Latency on CPU

• Authors: Hyeri Roh, Jinsu Yeo, Yeongil Ko, Gu-Yeon Wei, David Brooks, Woo-Seok Choi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16732
• Pdf URL: https://arxiv.org/pdf/2401.16732
• Copy Paste: [[2401.16732]] Flash: A Hybrid Private Inference Protocol for Deep CNNs with High Accuracy and Low Latency on CPU(https://arxiv.org/abs/2401.16732)
• Keywords: secure
• Abstract: This paper presents Flash, an optimized private inference (PI) hybrid protocol utilizing both homomorphic encryption (HE) and secure two-party computation (2PC), which can reduce the end-to-end PI latency for deep CNN models less than 1 minute with CPU. To this end, first, Flash proposes a low-latency convolution algorithm built upon a fast slot rotation operation and a novel data encoding scheme, which results in 4-94x performance gain over the state-of-the-art. Second, to minimize the communication cost introduced by the standard nonlinear activation function ReLU, Flash replaces the entire ReLUs with the polynomial $x^2+x$ and trains deep CNN models with the new activation function. The trained models improve the inference accuracy for CIFAR-10/100 and TinyImageNet by 16% on average (up to 40% for ResNet-32) compared to prior art. Last, Flash proposes an efficient 2PC-based $x^2+x$ evaluation protocol that does not require any offline communication and that reduces the total communication cost to process the activation layer by 84-196x over the state-of-the-art. As a result, the end-to-end PI latency of Flash implemented on CPU is 0.02 minute for CIFAR-100 and 0.57 minute for TinyImageNet classification, while the total data communication is 0.07GB for CIFAR-100 and 0.22GB for TinyImageNet. Flash improves the state-of-the-art PI by 16-45x in latency and 84-196x in communication cost. Moreover, even for ImageNet, Flash can deliver the latency less than 1 minute on CPU with the total communication less than 1GB.

Title: Engineering A Large Language Model From Scratch

• Authors: Abiodun Finbarrs Oketunji
• Subjects: cs.CL, cs.CY, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.16736
• Pdf URL: https://arxiv.org/pdf/2401.16736
• Copy Paste: [[2401.16736]] Engineering A Large Language Model From Scratch(https://arxiv.org/abs/2401.16736)
• Keywords: robust, transformer, large language model
• Abstract: The proliferation of deep learning in natural language processing (NLP) has led to the development and release of innovative technologies capable of understanding and generating human language with remarkable proficiency. Atinuke, a Transformer-based neural network, optimises performance across various language tasks by utilising a unique configuration. The architecture interweaves layers for processing sequential data with attention mechanisms to draw meaningful affinities between inputs and outputs. Due to the configuration of its topology and hyperparameter tuning, it can emulate human-like language by extracting features and learning complex mappings. Atinuke is modular, extensible, and integrates seamlessly with existing machine learning pipelines. Advanced matrix operations like softmax, embeddings, and multi-head attention enable nuanced handling of textual, acoustic, and visual signals. By unifying modern deep learning techniques with software design principles and mathematical theory, the system achieves state-of-the-art results on natural language tasks whilst remaining interpretable and robust.

Title: MESA: Matching Everything by Segmenting Anything

• Authors: Yesheng Zhang, Xu Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16741
• Pdf URL: https://arxiv.org/pdf/2401.16741
• Copy Paste: [[2401.16741]] MESA: Matching Everything by Segmenting Anything(https://arxiv.org/abs/2401.16741)
• Keywords: segmentation
• Abstract: Feature matching is a crucial task in the field of computer vision, which involves finding correspondences between images. Previous studies achieve remarkable performance using learning-based feature comparison. However, the pervasive presence of matching redundancy between images gives rise to unnecessary and error-prone computations in these methods, imposing limitations on their accuracy. To address this issue, we propose MESA, a novel approach to establish precise area (or region) matches for efficient matching redundancy reduction. MESA first leverages the advanced image understanding capability of SAM, a state-of-the-art foundation model for image segmentation, to obtain image areas with implicit semantic. Then, a multi-relational graph is proposed to model the spatial structure of these areas and construct their scale hierarchy. Based on graphical models derived from the graph, the area matching is reformulated as an energy minimization task and effectively resolved. Extensive experiments demonstrate that MESA yields substantial precision improvement for multiple point matchers in indoor and outdoor downstream tasks, e.g. +13.61% for DKM in indoor pose estimation.

Title: MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models

• Authors: Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16745
• Pdf URL: https://arxiv.org/pdf/2401.16745
• Copy Paste: [[2401.16745]] MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models(https://arxiv.org/abs/2401.16745)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) are increasingly relied upon for complex multi-turn conversations across diverse real-world applications. However, existing benchmarks predominantly focus on single-turn evaluations, overlooking the models' capabilities in multi-turn interactions. To address this gap, we introduce MT-Eval, a comprehensive benchmark designed to evaluate multi-turn conversational abilities. By analyzing human-LLM conversations, we categorize interaction patterns into four types: recollection, expansion, refinement, and follow-up. We construct multi-turn queries for each category either by augmenting existing datasets or by creating new examples with GPT-4 to avoid data leakage. To study the factors impacting multi-turn abilities, we create single-turn versions of the 1170 multi-turn queries and compare performance. Our evaluation of 11 well-known LLMs shows that while closed-source models generally surpass open-source ones, certain open-source models exceed GPT-3.5-Turbo in specific tasks. We observe significant performance degradation in multi-turn settings compared to single-turn settings in most models, which is not correlated with the models' fundamental capabilities. Moreover, we identify the distance to relevant content and susceptibility to error propagation as the key factors influencing multi-turn performance. MT-Eval is released publicly to encourage future research towards more robust conversational models.

Title: MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images

• Authors: Xurui Li, Ziming Huang, Feng Xue, Yu Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16753
• Pdf URL: https://arxiv.org/pdf/2401.16753
• Copy Paste: [[2401.16753]] MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images(https://arxiv.org/abs/2401.16753)
• Keywords: segmentation
• Abstract: This paper studies zero-shot anomaly classification (AC) and segmentation (AS) in industrial vision. We reveal that the abundant normal and abnormal cues implicit in unlabeled test images can be exploited for anomaly determination, which is ignored by prior methods. Our key observation is that for the industrial product images, the normal image patches could find a relatively large number of similar patches in other unlabeled images, while the abnormal ones only have a few similar patches. We leverage such a discriminative characteristic to design a novel zero-shot AC/AS method by Mutual Scoring (MuSc) of the unlabeled images, which does not need any training or prompts. Specifically, we perform Local Neighborhood Aggregation with Multiple Degrees (LNAMD) to obtain the patch features that are capable of representing anomalies in varying sizes. Then we propose the Mutual Scoring Mechanism (MSM) to leverage the unlabeled test images to assign the anomaly score to each other. Furthermore, we present an optimization approach named Re-scoring with Constrained Image-level Neighborhood (RsCIN) for image-level anomaly classification to suppress the false positives caused by noises in normal images. The superior performance on the challenging MVTec AD and VisA datasets demonstrates the effectiveness of our approach. Compared with the state-of-the-art zero-shot approaches, MuSc achieves a $\textbf{21.1%}$ PRO absolute gain (from 72.7% to 93.8%) on MVTec AD, a $\textbf{19.4%}$ pixel-AP gain and a $\textbf{14.7%}$ pixel-AUROC gain on VisA. In addition, our zero-shot approach outperforms most of the few-shot approaches and is comparable to some one-class methods. Code is available at https://github.com/xrli-U/MuSc.

Title: Diffusion model for relational inference

• Authors: Shuhan Zheng, Ziqiang Li, Kantaro Fujiwara, Gouhei Tanaka
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16755
• Pdf URL: https://arxiv.org/pdf/2401.16755
• Copy Paste: [[2401.16755]] Diffusion model for relational inference(https://arxiv.org/abs/2401.16755)
• Keywords: diffusion
• Abstract: Dynamical behaviors of complex interacting systems, including brain activities, financial price movements, and physical collective phenomena, are associated with underlying interactions between the system's components. The issue of uncovering interaction relations in such systems using observable dynamics is called relational inference. In this study, we propose a Diffusion model for Relational Inference (DiffRI), inspired by a self-supervised method for probabilistic time series imputation. DiffRI learns to infer the probability of the presence of connections between components through conditional diffusion modeling. Experiments on both simulated and quasi-real datasets show that DiffRI is highly competent compared with other state-of-the-art models in discovering ground truth interactions in an unsupervised manner. Our code will be made public soon.

Title: SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget

• Authors: Kun Wang, Jiani Cao, Zimu Zhou, Zhenjiang Li
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.16757
• Pdf URL: https://arxiv.org/pdf/2401.16757
• Copy Paste: [[2401.16757]] SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget(https://arxiv.org/abs/2401.16757)
• Keywords: large language model
• Abstract: Executing deep neural networks (DNNs) on edge artificial intelligence (AI) devices enables various autonomous mobile computing applications. However, the memory budget of edge AI devices restricts the number and complexity of DNNs allowed in such applications. Existing solutions, such as model compression or cloud offloading, reduce the memory footprint of DNN inference at the cost of decreased model accuracy or autonomy. To avoid these drawbacks, we divide DNN into blocks and swap them in and out in order, such that large DNNs can execute within a small memory budget. Nevertheless, naive swapping on edge AI devices induces significant delays due to the redundant memory operations in the DNN development ecosystem for edge AI devices. To this end, we develop SwapNet, an efficient DNN block swapping middleware for edge AI devices. We systematically eliminate the unnecessary memory operations during block swapping while retaining compatible with the deep learning frameworks, GPU backends, and hardware architectures of edge AI devices. We further showcase the utility of SwapNet via a multi-DNN scheduling scheme. Evaluations on eleven DNN inference tasks in three applications demonstrate that SwapNet achieves almost the same latency as the case with sufficient memory even when DNNs demand 2.32x to 5.81x memory beyond the available budget. The design of SwapNet also provides novel and feasible insights for deploying large language models (LLMs) on edge AI devices in the future.

Title: Sandi: A System for Accountability and Applications in Direct Communication

• Authors: F. Betül Durak, Kim Laine, Simon Langowski, Radames Cruz Moreno
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16759
• Pdf URL: https://arxiv.org/pdf/2401.16759
• Copy Paste: [[2401.16759]] Sandi: A System for Accountability and Applications in Direct Communication(https://arxiv.org/abs/2401.16759)
• Keywords: security, privacy, protect
• Abstract: We construct a system, Sandi, to bring trust in online communication between parties that share little or no context. Sandi is based on a unique ``somewhat monotone'' privacy-preserving reputation system, with strong privacy and security properties. Registered senders request cryptographic tags from Sandi, which they attach to their messages. Message receivers do not need registered accounts, but they can use a sender's score to decide how much the sender should be trusted. If a receiver finds the message inappropriate, they can use the tag to report the sender to Sandi, thus decreasing the sender's score. The design of Sandi ensures compatibility with any communication system that allows for small binary data transmission. Sandi aims to benefit both senders and receivers. Senders benefit, as receivers are more likely to react to their messages with reputation scores attached. Receivers benefit, as they can make better choices in who to interact with based on indisputable evidence from prior receivers. Sandi does not require senders or receivers to maintain long-term secret keys. We provide a score integrity guarantee for the senders, a full communication privacy guarantee for the senders and receivers, a report privacy guarantee to protect reporting receivers, and an unlinkability guarantee to protect senders. Finally, we provide a game-theoretic analysis for the sender. We prove that, for any score function satisfying a list of properties, Sandi drives rational senders towards a strategy, which reduces the amount of inappropriate messages.

Title: Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization

• Authors: Henglei Lv, Jiayu Xiao, Liang Li, Qingming Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16762
• Pdf URL: https://arxiv.org/pdf/2401.16762
• Copy Paste: [[2401.16762]] Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization(https://arxiv.org/abs/2401.16762)
• Keywords: diffusion, generative
• Abstract: Diffusion-based text-to-image personalization have achieved great success in generating subjects specified by users among various contexts. Even though, existing finetuning-based methods still suffer from model overfitting, which greatly harms the generative diversity, especially when given subject images are few. To this end, we propose Pick-and-Draw, a training-free semantic guidance approach to boost identity consistency and generative diversity for personalization methods. Our approach consists of two components: appearance picking guidance and layout drawing guidance. As for the former, we construct an appearance palette with visual features from the reference image, where we pick local patterns for generating the specified subject with consistent identity. As for layout drawing, we outline the subject's contour by referring to a generative template from the vanilla diffusion model, and inherit the strong image prior to synthesize diverse contexts according to different text conditions. The proposed approach can be applied to any personalized diffusion models and requires as few as a single reference image. Qualitative and quantitative experiments show that Pick-and-Draw consistently improves identity consistency and generative diversity, pushing the trade-off between subject fidelity and image-text fidelity to a new Pareto frontier.

Title: BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion

• Authors: Yonghao Yu, Shunan Zhu, Huai Qin, Haorui Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16764
• Pdf URL: https://arxiv.org/pdf/2401.16764
• Copy Paste: [[2401.16764]] BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion(https://arxiv.org/abs/2401.16764)
• Keywords: diffusion
• Abstract: Witnessing the evolution of text-to-image diffusion models, significant strides have been made in text-to-3D generation. Currently, two primary paradigms dominate the field of text-to-3D: the feed-forward generation solutions, capable of swiftly producing 3D assets but often yielding coarse results, and the Score Distillation Sampling (SDS) based solutions, known for generating high-fidelity 3D assets albeit at a slower pace. The synergistic integration of these methods holds substantial promise for advancing 3D generation techniques. In this paper, we present BoostDream, a highly efficient plug-and-play 3D refining method designed to transform coarse 3D assets into high-quality. The BoostDream framework comprises three distinct processes: (1) We introduce 3D model distillation that fits differentiable representations from the 3D assets obtained through feed-forward generation. (2) A novel multi-view SDS loss is designed, which utilizes a multi-view aware 2D diffusion model to refine the 3D assets. (3) We propose to use prompt and multi-view consistent normal maps as guidance in refinement.Our extensive experiment is conducted on different differentiable 3D representations, revealing that BoostDream excels in generating high-quality 3D assets rapidly, overcoming the Janus problem compared to conventional SDS-based methods. This breakthrough signifies a substantial advancement in both the efficiency and quality of 3D generation processes.

Title: A Cross-Language Investigation into Jailbreak Attacks in Large Language Models

• Authors: Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, Yinxing Xue
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16765
• Pdf URL: https://arxiv.org/pdf/2401.16765
• Copy Paste: [[2401.16765]] A Cross-Language Investigation into Jailbreak Attacks in Large Language Models(https://arxiv.org/abs/2401.16765)
• Keywords: security, defense, attack, interpretability, large language model
• Abstract: Large Language Models (LLMs) have become increasingly popular for their advanced text generation capabilities across various domains. However, like any software, they face security challenges, including the risk of 'jailbreak' attacks that manipulate LLMs to produce prohibited content. A particularly underexplored area is the Multilingual Jailbreak attack, where malicious questions are translated into various languages to evade safety filters. Currently, there is a lack of comprehensive empirical studies addressing this specific threat. To address this research gap, we conducted an extensive empirical study on Multilingual Jailbreak attacks. We developed a novel semantic-preserving algorithm to create a multilingual jailbreak dataset and conducted an exhaustive evaluation on both widely-used open-source and commercial LLMs, including GPT-4 and LLaMa. Additionally, we performed interpretability analysis to uncover patterns in Multilingual Jailbreak attacks and implemented a fine-tuning mitigation method. Our findings reveal that our mitigation strategy significantly enhances model defense, reducing the attack success rate by 96.2%. This study provides valuable insights into understanding and mitigating Multilingual Jailbreak attacks.

Title: Detection and Recovery Against Deep Neural Network Fault Injection Attacks Based on Contrastive Learning

• Authors: Chenan Wang, Pu Zhao, Siyue Wang, Xue Lin
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16766
• Pdf URL: https://arxiv.org/pdf/2401.16766
• Copy Paste: [[2401.16766]] Detection and Recovery Against Deep Neural Network Fault Injection Attacks Based on Contrastive Learning(https://arxiv.org/abs/2401.16766)
• Keywords: attack
• Abstract: Deep Neural Network (DNN) models when implemented on executing devices as the inference engines are susceptible to Fault Injection Attacks (FIAs) that manipulate model parameters to disrupt inference execution with disastrous performance. This work introduces Contrastive Learning (CL) of visual representations i.e., a self-supervised learning approach into the deep learning training and inference pipeline to implement DNN inference engines with self-resilience under FIAs. Our proposed CL based FIA Detection and Recovery (CFDR) framework features (i) real-time detection with only a single batch of testing data and (ii) fast recovery effective even with only a small amount of unlabeled testing data. Evaluated with the CIFAR-10 dataset on multiple types of FIAs, our CFDR shows promising detection and recovery effectiveness.

Title: Extrinsicaly Rewarded Soft Q Imitation Learning with Discriminator

• Authors: Ryoma Furuyama, Daiki Kuyoshi, Satoshi Yamane
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16772
• Pdf URL: https://arxiv.org/pdf/2401.16772
• Copy Paste: [[2401.16772]] Extrinsicaly Rewarded Soft Q Imitation Learning with Discriminator(https://arxiv.org/abs/2401.16772)
• Keywords: robust, generative
• Abstract: Imitation learning is often used in addition to reinforcement learning in environments where reward design is difficult or where the reward is sparse, but it is difficult to be able to imitate well in unknown states from a small amount of expert data and sampling data. Supervised learning methods such as Behavioral Cloning do not require sampling data, but usually suffer from distribution shift. The methods based on reinforcement learning, such as inverse reinforcement learning and Generative Adversarial imitation learning (GAIL), can learn from only a few expert data. However, they often need to interact with the environment. Soft Q imitation learning (SQIL) addressed the problems, and it was shown that it could learn efficiently by combining Behavioral Cloning and soft Q-learning with constant rewards. In order to make this algorithm more robust to distribution shift, we propose more efficient and robust algorithm by adding to this method a reward function based on adversarial inverse reinforcement learning that rewards the agent for performing actions in status similar to the demo. We call this algorithm Discriminator Soft Q Imitation Learning (DSQIL). We evaluated it on MuJoCo environments.

Title: Graph Fairness Learning under Distribution Shifts

• Authors: Yibo Li, Xiao Wang, Yujie Xing, Shaohua Fan, Ruijia Wang, Yaoqi Liu, Chuan Shi
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.16784
• Pdf URL: https://arxiv.org/pdf/2401.16784
• Copy Paste: [[2401.16784]] Graph Fairness Learning under Distribution Shifts(https://arxiv.org/abs/2401.16784)
• Keywords: fair
• Abstract: Graph neural networks (GNNs) have achieved remarkable performance on graph-structured data. However, GNNs may inherit prejudice from the training data and make discriminatory predictions based on sensitive attributes, such as gender and race. Recently, there has been an increasing interest in ensuring fairness on GNNs, but all of them are under the assumption that the training and testing data are under the same distribution, i.e., training data and testing data are from the same graph. Will graph fairness performance decrease under distribution shifts? How does distribution shifts affect graph fairness learning? All these open questions are largely unexplored from a theoretical perspective. To answer these questions, we first theoretically identify the factors that determine bias on a graph. Subsequently, we explore the factors influencing fairness on testing graphs, with a noteworthy factor being the representation distances of certain groups between the training and testing graph. Motivated by our theoretical analysis, we propose our framework FatraGNN. Specifically, to guarantee fairness performance on unknown testing graphs, we propose a graph generator to produce numerous graphs with significant bias and under different distributions. Then we minimize the representation distances for each certain group between the training graph and generated graphs. This empowers our model to achieve high classification and fairness performance even on generated graphs with significant bias, thereby effectively handling unknown testing graphs. Experiments on real-world and semi-synthetic datasets demonstrate the effectiveness of our model in terms of both accuracy and fairness.

Title: Enhancing Efficiency and Robustness in Support Vector Regression with HawkEye Loss

• Authors: Mushir Akhtar, M. Tanveer, Mohd. Arshad
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16785
• Pdf URL: https://arxiv.org/pdf/2401.16785
• Copy Paste: [[2401.16785]] Enhancing Efficiency and Robustness in Support Vector Regression with HawkEye Loss(https://arxiv.org/abs/2401.16785)
• Keywords: robust
• Abstract: Support vector regression (SVR) has garnered significant popularity over the past two decades owing to its wide range of applications across various fields. Despite its versatility, SVR encounters challenges when confronted with outliers and noise, primarily due to the use of the $\varepsilon$-insensitive loss function. To address this limitation, SVR with bounded loss functions has emerged as an appealing alternative, offering enhanced generalization performance and robustness. Notably, recent developments focus on designing bounded loss functions with smooth characteristics, facilitating the adoption of gradient-based optimization algorithms. However, it's crucial to highlight that these bounded and smooth loss functions do not possess an insensitive zone. In this paper, we address the aforementioned constraints by introducing a novel symmetric loss function named the HawkEye loss function. It is worth noting that the HawkEye loss function stands out as the first loss function in SVR literature to be bounded, smooth, and simultaneously possess an insensitive zone. Leveraging this breakthrough, we integrate the HawkEye loss function into the least squares framework of SVR and yield a new fast and robust model termed HE-LSSVR. The optimization problem inherent to HE-LSSVR is addressed by harnessing the adaptive moment estimation (Adam) algorithm, known for its adaptive learning rate and efficacy in handling large-scale problems. To our knowledge, this is the first time Adam has been employed to solve an SVR problem. To empirically validate the proposed HE-LSSVR model, we evaluate it on UCI, synthetic, and time series datasets. The experimental outcomes unequivocally reveal the superiority of the HE-LSSVR model both in terms of its remarkable generalization performance and its efficiency in training time.

Title: Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate

• Authors: Steffi Chern, Ethan Chern, Graham Neubig, Pengfei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16788
• Pdf URL: https://arxiv.org/pdf/2401.16788
• Copy Paste: [[2401.16788]] Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate(https://arxiv.org/abs/2401.16788)
• Keywords: large language model
• Abstract: Despite the utility of Large Language Models (LLMs) across a wide range of tasks and scenarios, developing a method for reliably evaluating LLMs across varied contexts continues to be challenging. Modern evaluation approaches often use LLMs to assess responses generated by LLMs. However, the meta-evaluation conducted to assess the effectiveness of these LLMs as evaluators is typically constrained by the coverage of existing benchmarks or requires extensive human annotation. This underscores the urgency of methods for scalable meta-evaluation that can effectively, reliably, and efficiently evaluate the performance of LLMs as evaluators across diverse tasks and scenarios, particularly in potentially new, user-defined scenarios. To fill this gap, we propose ScaleEval, an agent-debate-assisted meta-evaluation framework that leverages the capabilities of multiple communicative LLM agents. This framework supports multi-round discussions to assist human annotators in discerning the most capable LLMs as evaluators, which significantly eases their workload in cases that used to require large-scale annotations during meta-evaluation. We release the code for our framework, which is publicly available at: \url{https://github.com/GAIR-NLP/scaleeval}.

Title: Learnable Prompt as Pseudo-Imputation: Reassessing the Necessity of Traditional EHR Data Imputation in Downstream Clinical Prediction

• Authors: Weibin Liao, Yinghao Zhu, Zixiang Wang, Xu Chu, Yasha Wang, Liantao Ma
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16796
• Pdf URL: https://arxiv.org/pdf/2401.16796
• Copy Paste: [[2401.16796]] Learnable Prompt as Pseudo-Imputation: Reassessing the Necessity of Traditional EHR Data Imputation in Downstream Clinical Prediction(https://arxiv.org/abs/2401.16796)
• Keywords: robust
• Abstract: Analyzing the health status of patients based on Electronic Health Records (EHR) is a fundamental research problem in medical informatics. The presence of extensive missing values in EHR makes it challenging for deep neural networks to directly model the patient's health status based on EHR. Existing deep learning training protocols require the use of statistical information or imputation models to reconstruct missing values; however, the protocols inject non-realistic data into downstream EHR analysis models, significantly limiting model performance. This paper introduces Learnable Prompt as Pseudo Imputation (PAI) as a new training protocol. PAI no longer introduces any imputed data but constructs a learnable prompt to model the implicit preferences of the downstream model for missing values, resulting in a significant performance improvement for all EHR analysis models. Additionally, our experiments show that PAI exhibits higher robustness in situations of data insufficiency and high missing rates. More importantly, in a real-world application involving cross-institutional data with zero-shot evaluation, PAI demonstrates stronger model generalization capabilities for non-overlapping features.

Title: Online Algorithm for Node Feature Forecasting in Temporal Graphs

• Authors: Aniq Ur Rahman, Justin P. Coon
• Subjects: cs.LG, cs.DM, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.16800
• Pdf URL: https://arxiv.org/pdf/2401.16800
• Copy Paste: [[2401.16800]] Online Algorithm for Node Feature Forecasting in Temporal Graphs(https://arxiv.org/abs/2401.16800)
• Keywords: robust
• Abstract: In this paper, we propose an online algorithm "mspace" for forecasting node features in temporal graphs, which adeptly captures spatial cross-correlation among different nodes as well as the temporal autocorrelation within a node. The algorithm can be used for both probabilistic and deterministic multi-step forecasting, making it applicable for estimation and generation tasks. Comparative evaluations against various baselines, including graph neural network (GNN) based models and classical Kalman filters, demonstrate that mspace performs at par with the state-of-the-art and even surpasses them on some datasets. Importantly, mspace demonstrates consistent robustness across datasets with varying training sizes, a notable advantage over GNN-based methods requiring abundant training samples to learn the spatiotemporal trends in the data effectively. Therefore, employing mspace is advantageous in scenarios where the training sample availability is limited. Additionally, we establish theoretical bounds on multi-step forecasting error of mspace and show that it scales as $O(q)$ for $q$-step forecast.

Title: An Embeddable Implicit IUVD Representation for Part-based 3D Human Surface Reconstruction

• Authors: Baoxing Li, Yong Deng, Yehui Yang, Xu Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16810
• Pdf URL: https://arxiv.org/pdf/2401.16810
• Copy Paste: [[2401.16810]] An Embeddable Implicit IUVD Representation for Part-based 3D Human Surface Reconstruction(https://arxiv.org/abs/2401.16810)
• Keywords: robust, extraction, generative
• Abstract: To reconstruct a 3D human surface from a single image, it is important to consider human pose, shape and clothing details simultaneously. In recent years, a combination of parametric body models (such as SMPL) that capture body pose and shape prior, and neural implicit functions that learn flexible clothing details, has been used to integrate the advantages of both approaches. However, the combined representation introduces additional computation, e.g. signed distance calculation, in 3D body feature extraction, which exacerbates the redundancy of the implicit query-and-infer process and fails to preserve the underlying body shape prior. To address these issues, we propose a novel IUVD-Feedback representation, which consists of an IUVD occupancy function and a feedback query algorithm. With this representation, the time-consuming signed distance calculation is replaced by a simple linear transformation in the IUVD space, leveraging the SMPL UV maps. Additionally, the redundant query points in the query-and-infer process are reduced through a feedback mechanism. This leads to more reasonable 3D body features and more effective query points, successfully preserving the parametric body prior. Moreover, the IUVD-Feedback representation can be embedded into any existing implicit human reconstruction pipelines without modifying the trained neural networks. Experiments on THuman2.0 dataset demonstrate that the proposed IUVD-Feedback representation improves result robustness and achieves three times faster acceleration in the query-and-infer process. Furthermore, this representation has the potential to be used in generative applications by leveraging its inherited semantic information from the parametric body model.

Title: H2O-Danube-1.8B Technical Report

• Authors: Philipp Singer, Pascal Pfeiffer, Yauhen Babakhin, Maximilian Jeblick, Nischay Dhankhar, Gabor Fodor, Sri Satish Ambati
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16818
• Pdf URL: https://arxiv.org/pdf/2401.16818
• Copy Paste: [[2401.16818]] H2O-Danube-1.8B Technical Report(https://arxiv.org/abs/2401.16818)
• Keywords: large language model
• Abstract: We present H2O-Danube-1.8B, a 1.8B language model trained on 1T tokens following the core principles of LLama 2 and Mistral. We leverage and refine various techniques for pre-training large language models. Although our model is trained on significantly fewer total tokens compared to reference models of similar size, it exhibits highly competitive metrics across a multitude of benchmarks. We additionally release a chat model trained with supervised fine-tuning followed by direct preference optimization. We make H2O-Danube-1.8B openly available under Apache 2.0 license further democratizing LLMs to a wider audience economically.

Title: Provably Robust Multi-bit Watermarking for AI-generated Text via Error Correction Code

• Authors: Wenjie Qu, Dong Yin, Zixin He, Wei Zou, Tianyang Tao, Jinyuan Jia, Jiaheng Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16820
• Pdf URL: https://arxiv.org/pdf/2401.16820
• Copy Paste: [[2401.16820]] Provably Robust Multi-bit Watermarking for AI-generated Text via Error Correction Code(https://arxiv.org/abs/2401.16820)
• Keywords: attack, robust, watermark, large language model
• Abstract: Large Language Models (LLMs) have been widely deployed for their remarkable capability to generate texts resembling human language. However, they could be misused by criminals to create deceptive content, such as fake news and phishing emails, which raises ethical concerns. Watermarking is a key technique to mitigate the misuse of LLMs, which embeds a watermark (e.g., a bit string) into a text generated by a LLM. Consequently, this enables the detection of texts generated by a LLM as well as the tracing of generated texts to a specific user. The major limitation of existing watermark techniques is that they cannot accurately or efficiently extract the watermark from a text, especially when the watermark is a long bit string. This key limitation impedes their deployment for real-world applications, e.g., tracing generated texts to a specific user. This work introduces a novel watermarking method for LLM-generated text grounded in \textbf{error-correction codes} to address this challenge. We provide strong theoretical analysis, demonstrating that under bounded adversarial word/token edits (insertion, deletion, and substitution), our method can correctly extract watermarks, offering a provable robustness guarantee. This breakthrough is also evidenced by our extensive experimental results. The experiments show that our method substantially outperforms existing baselines in both accuracy and robustness on benchmark datasets. For instance, when embedding a bit string of length 12 into a 200-token generated text, our approach attains an impressive match rate of $98.4\%$, surpassing the performance of Yoo et al. (state-of-the-art baseline) at $85.6\%$. When subjected to a copy-paste attack involving the injection of 50 tokens to generated texts with 200 words, our method maintains a substantial match rate of $90.8\%$, while the match rate of Yoo et al. diminishes to below $65\%$.

Title: EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain

• Authors: Wei Zhang, Miaoxin Cai, Tong Zhang, Yin Zhuang, Xuerui Mao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16822
• Pdf URL: https://arxiv.org/pdf/2401.16822
• Copy Paste: [[2401.16822]] EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain(https://arxiv.org/abs/2401.16822)
• Keywords: large language model
• Abstract: Multi-modal large language models (MLLMs) have demonstrated remarkable success in vision and visual-language tasks within the natural image domain. Owing to the significant diversities between the natural image and RS image hinder the development of MLLMs in the remote sensing (RS) domain. Currently, the unified and powerful MLLM capable of various RS visual tasks is still under-explored. To fill the gap, a pioneer MLLM called EarthGPT is proposed for universal RS image comprehension, which integrates various multi-sensor RS interpretation tasks uniformly. More importantly, a large-scale multi-sensor multi-modal RS instruction-following dataset named MMRS is carefully constructed, which comprises 1005.842k image-text pairs based on 34 existing diverse RS datasets and includes multi-sensor images such as optical, synthetic aperture radar (SAR), and infrared. The MMRS addresses the issue of MLLMs lacking RS expert knowledge and stimulates the development of MMLMs in the RS domain. Extensive experiments demonstrate the EarthGPT's superior performance in various RS visual interpretation tasks compared with the other specialist models and MLLMs, which proves the effectiveness of the proposed EarthGPT and provides a versatile paradigm for open-set reasoning tasks.

Title: Evaluating ML-Based Anomaly Detection Across Datasets of Varied Integrity: A Case Study

• Authors: Adrian Pekar, Richard Jozsa
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.16843
• Pdf URL: https://arxiv.org/pdf/2401.16843
• Copy Paste: [[2401.16843]] Evaluating ML-Based Anomaly Detection Across Datasets of Varied Integrity: A Case Study(https://arxiv.org/abs/2401.16843)
• Keywords: security, robust
• Abstract: Cybersecurity remains a critical challenge in the digital age, with network traffic flow anomaly detection being a key pivotal instrument in the fight against cyber threats. In this study, we address the prevalent issue of data integrity in network traffic datasets, which are instrumental in developing machine learning (ML) models for anomaly detection. We introduce two refined versions of the CICIDS-2017 dataset, NFS-2023-nTE and NFS-2023-TE, processed using NFStream to ensure methodologically sound flow expiration and labeling. Our research contrasts the performance of the Random Forest (RF) algorithm across the original CICIDS-2017, its refined counterparts WTMC-2021 and CRiSIS-2022, and our NFStream-generated datasets, in both binary and multi-class classification contexts. We observe that the RF model exhibits exceptional robustness, achieving consistent high-performance metrics irrespective of the underlying dataset quality, which prompts a critical discussion on the actual impact of data integrity on ML efficacy. Our study underscores the importance of continual refinement and methodological rigor in dataset generation for network security research. As the landscape of network threats evolves, so must the tools and techniques used to detect and analyze them.

Title: Repositioning the Subject within Image

• Authors: Yikai Wang, Chenjie Cao, Qiaole Dong, Yifan Li, Yanwei Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16861
• Pdf URL: https://arxiv.org/pdf/2401.16861
• Copy Paste: [[2401.16861]] Repositioning the Subject within Image(https://arxiv.org/abs/2401.16861)
• Keywords: diffusion, generative
• Abstract: Current image manipulation primarily centers on static manipulation, such as replacing specific regions within an image or altering its overall style. In this paper, we introduce an innovative dynamic manipulation task, subject repositioning. This task involves relocating a user-specified subject to a desired position while preserving the image's fidelity. Our research reveals that the fundamental sub-tasks of subject repositioning, which include filling the void left by the repositioned subject, reconstructing obscured portions of the subject and blending the subject to be consistent with surrounding areas, can be effectively reformulated as a unified, prompt-guided inpainting task. Consequently, we can employ a single diffusion generative model to address these sub-tasks using various task prompts learned through our proposed task inversion technique. Additionally, we integrate pre-processing and post-processing techniques to further enhance the quality of subject repositioning. These elements together form our SEgment-gEnerate-and-bLEnd (SEELE) framework. To assess SEELE's effectiveness in subject repositioning, we assemble a real-world subject repositioning dataset called ReS. Our results on ReS demonstrate the quality of repositioned image generation.

Title: State Value Generation with Prompt Learning and Self-Training for Low-Resource Dialogue State Tracking

• Authors: Ming Gu, Yan Yang, Chengcai Chen, Zhou Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16862
• Pdf URL: https://arxiv.org/pdf/2401.16862
• Copy Paste: [[2401.16862]] State Value Generation with Prompt Learning and Self-Training for Low-Resource Dialogue State Tracking(https://arxiv.org/abs/2401.16862)
• Keywords: extraction
• Abstract: Recently, low-resource dialogue state tracking (DST) has received increasing attention. First obtaining state values then based on values to generate slot types has made great progress in this task. However, obtaining state values is still an under-studied problem. Existing extraction-based approaches cannot capture values that require the understanding of context and are not generalizable either. To address these issues, we propose a novel State VAlue Generation based framework (SVAG), decomposing DST into state value generation and domain slot generation. Specifically, we propose to generate state values and use self-training to further improve state value generation. Moreover, we design an estimator aiming at detecting incomplete generation and incorrect generation for pseudo-labeled data selection during self-training. Experimental results on the MultiWOZ 2.1 dataset show that our method which has only less than 1 billion parameters achieves state-of-the-art performance under the data ratio settings of 5%, 10%, and 25% when limited to models under 100 billion parameters. Compared to models with more than 100 billion parameters, SVAG still reaches competitive results.

Title: Zero-shot Classification using Hyperdimensional Computing

• Authors: Samuele Ruffino, Geethan Karunaratne, Michael Hersche, Luca Benini, Abu Sebastian, Abbas Rahimi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16876
• Pdf URL: https://arxiv.org/pdf/2401.16876
• Copy Paste: [[2401.16876]] Zero-shot Classification using Hyperdimensional Computing(https://arxiv.org/abs/2401.16876)
• Keywords: extraction, generative
• Abstract: Classification based on Zero-shot Learning (ZSL) is the ability of a model to classify inputs into novel classes on which the model has not previously seen any training examples. Providing an auxiliary descriptor in the form of a set of attributes describing the new classes involved in the ZSL-based classification is one of the favored approaches to solving this challenging task. In this work, inspired by Hyperdimensional Computing (HDC), we propose the use of stationary binary codebooks of symbol-like distributed representations inside an attribute encoder to compactly represent a computationally simple end-to-end trainable model, which we name Hyperdimensional Computing Zero-shot Classifier~(HDC-ZSC). It consists of a trainable image encoder, an attribute encoder based on HDC, and a similarity kernel. We show that HDC-ZSC can be used to first perform zero-shot attribute extraction tasks and, can later be repurposed for Zero-shot Classification tasks with minimal architectural changes and minimal model retraining. HDC-ZSC achieves Pareto optimal results with a 63.8% top-1 classification accuracy on the CUB-200 dataset by having only 26.6 million trainable parameters. Compared to two other state-of-the-art non-generative approaches, HDC-ZSC achieves 4.3% and 9.9% better accuracy, while they require more than 1.85x and 1.72x parameters compared to HDC-ZSC, respectively.

Title: CAFCT: Contextual and Attentional Feature Fusions of Convolutional Neural Networks and Transformer for Liver Tumor Segmentation

• Authors: Ming Kang, Chee-Ming Ting, Fung Fung Ting, Raphaël Phan
• Subjects: cs.CV, eess.SP, stat.AP
• Abstract URL: https://arxiv.org/abs/2401.16886
• Pdf URL: https://arxiv.org/pdf/2401.16886
• Copy Paste: [[2401.16886]] CAFCT: Contextual and Attentional Feature Fusions of Convolutional Neural Networks and Transformer for Liver Tumor Segmentation(https://arxiv.org/abs/2401.16886)
• Keywords: transformer, segmentation
• Abstract: Medical image semantic segmentation techniques can help identify tumors automatically from computed tomography (CT) scans. In this paper, we propose a Contextual and Attentional feature Fusions enhanced Convolutional Neural Network (CNN) and Transformer hybrid network (CAFCT) model for liver tumor segmentation. In the proposed model, three other modules are introduced in the network architecture: Attentional Feature Fusion (AFF), Atrous Spatial Pyramid Pooling (ASPP) of DeepLabv3, and Attention Gates (AGs) to improve contextual information related to tumor boundaries for accurate segmentation. Experimental results show that the proposed CAFCT achieves a mean Intersection over Union (IoU) of 90.38% and Dice score of 86.78%, respectively, on the Liver Tumor Segmentation Benchmark (LiTS) dataset, outperforming pure CNN or Transformer methods, e.g., Attention U-Net, and PVTFormer.

Title: Quantum-Secure Hybrid Blockchain System for DID-based Verifiable Random Function with NTRU Linkable Ring Signature

• Authors: Bong Gon Kim, Dennis Wong, Yoon Seok Yang
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.16906
• Pdf URL: https://arxiv.org/pdf/2401.16906
• Copy Paste: [[2401.16906]] Quantum-Secure Hybrid Blockchain System for DID-based Verifiable Random Function with NTRU Linkable Ring Signature(https://arxiv.org/abs/2401.16906)
• Keywords: secure, security, privacy, robust
• Abstract: In this study, we present a secure smart contract-based Verifiable Random Function (VRF) model, addressing the shortcomings of existing systems. As quantum computing emerges, conventional public key cryptography faces potential vulnerabilities. To enhance our VRF's robustness, we employ post-quantum Ring-LWE encryption for generating pseudo-random sequences. Given the computational intensity of this approach and associated on-chain gas costs, we propose a hybrid architecture of VRF system where on-chain and off-chain can communicate in a scalable and secure way. To ensure the validity and integrity of the off-chain computations (e.g., Ring-LWE encryption), we employ a quantum-secure linkable ring signature scheme on NTRU lattice and also delegated key generation (DKG) with a secure key encapsulation mechanism (KEM). Our decentralized VRF employs multi-party computation (MPC) with blockchain-based decentralized identifiers (DID), ensuring the collective efforts of enhanced randomness and security. We show the security and privacy advantages of our proposed VRF model with the approximated estimation of overall temporal and spatial complexities. We also evaluate our VRF MPC model's entropy and outline its Solidity smart contract integration. This research also provides a method to produce and verify the VRF output's proof, optimal for scenarios necessitating randomness and validation. Lastly, using NIST SP800-22 test suite for randomness, we demonstrate the commendable result with a 97.73% overall pass rate on 11 standard tests and 0.5459 of average p-value for the total 176 tests.

Title: Bit-flipping Decoder Failure Rate Estimation for (v,w)-regular Codes

• Authors: Alessandro Annechini, Alessandro Barenghi, Gerardo Pelosi
• Subjects: cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2401.16919
• Pdf URL: https://arxiv.org/pdf/2401.16919
• Copy Paste: [[2401.16919]] Bit-flipping Decoder Failure Rate Estimation for (v,w)-regular Codes(https://arxiv.org/abs/2401.16919)
• Keywords: security
• Abstract: Providing closed form estimates of the decoding failure rate of iterative decoder for low- and moderate-density parity check codes has attracted significant interest in the research community over the years. This interest has raised recently due to the use of iterative decoders in post-quantum cryptosystems, where the desired decoding failure rates are impossible to estimate via Monte Carlo simulations. In this work, we propose a new technique to provide accurate estimates of the DFR of a two-iterations (parallel) bit flipping decoder, which is also employable for cryptographic purposes. In doing so, we successfully tackle the estimation of the bit flipping probabilities at the second decoder iteration, and provide a fitting estimate for the syndrome weight distribution at the first iteration. We numerically validate our results, providing comparisons of the modeled and simulated weight of the syndrome, incorrectly-guessed error bit distribution at the end of the first iteration, and two-iteration Decoding Failure Rates (DFR), both in the floor and waterfall regime for simulatable codes. Finally, we apply our method to estimate the DFR of LEDAcrypt parameters, showing improvements by factors larger than $2^{70}$ (for NIST category $1$) with respect to the previous estimation techniques. This allows for a $\approx 20$% shortening in public key and ciphertext sizes, at no security loss, making the smallest ciphertext for NIST category $1$ only $6$% larger than the one of BIKE. We note that the analyzed two-iterations decoder is applicable in BIKE, where swapping it with the current black-gray decoder (and adjusting the parameters) would provide strong IND-CCA$2$ guarantees.

Title: Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation

• Authors: Ruiping Liu, Jiaming Zhang, Kunyu Peng, Yufan Chen, Ke Cao, Junwei Zheng, M. Saquib Sarfraz, Kailun Yang, Rainer Stiefelhagen
• Subjects: cs.CV, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.16923
• Pdf URL: https://arxiv.org/pdf/2401.16923
• Copy Paste: [[2401.16923]] Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation(https://arxiv.org/abs/2401.16923)
• Keywords: robust, segmentation
• Abstract: Integrating information from multiple modalities enhances the robustness of scene perception systems in autonomous vehicles, providing a more comprehensive and reliable sensory framework. However, the modality incompleteness in multi-modal segmentation remains under-explored. In this work, we establish a task called Modality-Incomplete Scene Segmentation (MISS), which encompasses both system-level modality absence and sensor-level modality errors. To avoid the predominant modality reliance in multi-modal fusion, we introduce a Missing-aware Modal Switch (MMS) strategy to proactively manage missing modalities during training. Utilizing bit-level batch-wise sampling enhances the model's performance in both complete and incomplete testing scenarios. Furthermore, we introduce the Fourier Prompt Tuning (FPT) method to incorporate representative spectral information into a limited number of learnable prompts that maintain robustness against all MISS scenarios. Akin to fine-tuning effects but with fewer tunable parameters (1.1%). Extensive experiments prove the efficacy of our proposed approach, showcasing an improvement of 5.84% mIoU over the prior state-of-the-art parameter-efficient methods in modality missing. The source code will be publicly available at https://github.com/RuipingL/MISS.

Title: Segmentation and Characterization of Macerated Fibers and Vessels Using Deep Learning

• Authors: Saqib Qamar, Abu Imran Baba, Stéphane Verger, Magnus Andersson
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16937
• Pdf URL: https://arxiv.org/pdf/2401.16937
• Copy Paste: [[2401.16937]] Segmentation and Characterization of Macerated Fibers and Vessels Using Deep Learning(https://arxiv.org/abs/2401.16937)
• Keywords: robust, segmentation
• Abstract: Purpose: Wood comprises different cell types, such as fibers and vessels, defining its properties. Studying their shape, size, and arrangement in microscopic images is crucial for understanding wood samples. Typically, this involves macerating (soaking) samples in a solution to separate cells, then spreading them on slides for imaging with a microscope that covers a wide area, capturing thousands of cells. However, these cells often cluster and overlap in images, making the segmentation difficult and time-consuming using standard image-processing methods. Results: In this work, we develop an automatic deep learning segmentation approach that utilizes the one-stage YOLOv8 model for fast and accurate fiber and vessel segmentation and characterization in microscopy images. The model can analyze 32640 x 25920 pixels images and demonstrate effective cell detection and segmentation, achieving a mAP_0.5-0.95 of 78 %. To assess the model's robustness, we examined fibers from a genetically modified tree line known for longer fibers. The outcomes were comparable to previous manual measurements. Additionally, we created a user-friendly web application for image analysis and provided the code for use on Google Colab. Conclusion: By leveraging YOLOv8's advances, this work provides a deep learning solution to enable efficient quantification and analysis of wood cells suitable for practical applications.

Title: WGAN-AFL: Seed Generation Augmented Fuzzer with Wasserstein-GAN

• Authors: Liqun Yang, Chunan Li, Yongxin Qiu, Chaoren Wei, Jian Yang, Hongcheng Guo, Jinxin Ma, Zhoujun Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16947
• Pdf URL: https://arxiv.org/pdf/2401.16947
• Copy Paste: [[2401.16947]] WGAN-AFL: Seed Generation Augmented Fuzzer with Wasserstein-GAN(https://arxiv.org/abs/2401.16947)
• Keywords: security, defense, generative
• Abstract: The importance of addressing security vulnerabilities is indisputable, with software becoming crucial in sectors such as national defense and finance. Consequently, The security issues caused by software vulnerabilities cannot be ignored. Fuzz testing is an automated software testing technology that can detect vulnerabilities in the software. However, most previous fuzzers encounter challenges that fuzzing performance is sensitive to initial input seeds. In the absence of high-quality initial input seeds, fuzzers may expend significant resources on program path exploration, leading to a substantial decrease in the efficiency of vulnerability detection. To address this issue, we propose WGAN-AFL. By collecting high-quality testcases, we train a generative adversarial network (GAN) to learn their features, thereby obtaining high-quality initial input seeds. To overcome drawbacks like mode collapse and training instability inherent in GANs, we utilize the Wasserstein GAN (WGAN) architecture for training, further enhancing the quality of the generated seeds. Experimental results demonstrate that WGAN-AFL significantly outperforms the original AFL in terms of code coverage, new paths, and vulnerability discovery, demonstrating the effective enhancement of seed quality by WGAN-AFL.

Title: Two Heads Are Better Than One: Integrating Knowledge from Knowledge Graphs and Large Language Models for Entity Alignment

• Authors: Linyao Yang, Hongyang Chen, Xiao Wang, Jing Yang, Fei-Yue Wang, Han Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16960
• Pdf URL: https://arxiv.org/pdf/2401.16960
• Copy Paste: [[2401.16960]] Two Heads Are Better Than One: Integrating Knowledge from Knowledge Graphs and Large Language Models for Entity Alignment(https://arxiv.org/abs/2401.16960)
• Keywords: large language model
• Abstract: Entity alignment, which is a prerequisite for creating a more comprehensive Knowledge Graph (KG), involves pinpointing equivalent entities across disparate KGs. Contemporary methods for entity alignment have predominantly utilized knowledge embedding models to procure entity embeddings that encapsulate various similarities-structural, relational, and attributive. These embeddings are then integrated through attention-based information fusion mechanisms. Despite this progress, effectively harnessing multifaceted information remains challenging due to inherent heterogeneity. Moreover, while Large Language Models (LLMs) have exhibited exceptional performance across diverse downstream tasks by implicitly capturing entity semantics, this implicit knowledge has yet to be exploited for entity alignment. In this study, we propose a Large Language Model-enhanced Entity Alignment framework (LLMEA), integrating structural knowledge from KGs with semantic knowledge from LLMs to enhance entity alignment. Specifically, LLMEA identifies candidate alignments for a given entity by considering both embedding similarities between entities across KGs and edit distances to a virtual equivalent entity. It then engages an LLM iteratively, posing multiple multi-choice questions to draw upon the LLM's inference capability. The final prediction of the equivalent entity is derived from the LLM's output. Experiments conducted on three public datasets reveal that LLMEA surpasses leading baseline models. Additional ablation studies underscore the efficacy of our proposed framework.

Title: Deep 3D World Models for Multi-Image Super-Resolution Beyond Optical Flow

• Authors: Luca Savant Aira, Diego Valsesia, Andrea Bordone Molini, Giulia Fracastoro, Enrico Magli, Andrea Mirabile
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.16972
• Pdf URL: https://arxiv.org/pdf/2401.16972
• Copy Paste: [[2401.16972]] Deep 3D World Models for Multi-Image Super-Resolution Beyond Optical Flow(https://arxiv.org/abs/2401.16972)
• Keywords: transformer
• Abstract: Multi-image super-resolution (MISR) allows to increase the spatial resolution of a low-resolution (LR) acquisition by combining multiple images carrying complementary information in the form of sub-pixel offsets in the scene sampling, and can be significantly more effective than its single-image counterpart. Its main difficulty lies in accurately registering and fusing the multi-image information. Currently studied settings, such as burst photography, typically involve assumptions of small geometric disparity between the LR images and rely on optical flow for image registration. We study a MISR method that can increase the resolution of sets of images acquired with arbitrary, and potentially wildly different, camera positions and orientations, generalizing the currently studied MISR settings. Our proposed model, called EpiMISR, moves away from optical flow and explicitly uses the epipolar geometry of the acquisition process, together with transformer-based processing of radiance feature fields to substantially improve over state-of-the-art MISR methods in presence of large disparities in the LR images.

Title: Randomized Key Encapsulation/Consolidation

• Authors: Amir K. Khandani
• Subjects: cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2401.16993
• Pdf URL: https://arxiv.org/pdf/2401.16993
• Copy Paste: [[2401.16993]] Randomized Key Encapsulation/Consolidation(https://arxiv.org/abs/2401.16993)
• Keywords: security
• Abstract: This article bridges the gap between two topics used in sharing an encryption key: (i) Key Consolidation, i.e., extracting two identical strings of bits from two information sources with similarities (common randomness). (ii) Quantum-safe Key Encapsulation by incorporating randomness in Public/Private Key pairs. In the context of Key Consolidation, the proposed scheme adds to the complexity Eve faces in extracting useful data from leaked information. In this context, it is applied to the method proposed in [1] for establishing common randomness from round-trip travel times in a packet data network. The proposed method allows adapting the secrecy level to the amount of similarity in common randomness. It can even encapsulate a Quantum-safe encryption key in the extreme case that no common randomness is available. In the latter case, it is shown that the proposed scheme offers improvements with respect to the McEliece cryptosystem which currently forms the foundation for Quantum safe key encapsulation. [1] A. K. Khandani, "Looping for Encryption Key Generation Over the Internet: A New Frontier in Physical Layer Security," 2023 Biennial Symposium on Communications (BSC), Montreal, QC, Canada, 2023, pp. 59-64

Title: Finetuning Large Language Models for Vulnerability Detection

• Authors: Alexey Shestov, Anton Cheshkov, Rodion Levichev, Ravil Mussabayev, Pavel Zadorozhny, Evgeny Maslov, Chibirev Vadim, Egor Bulychev
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17010
• Pdf URL: https://arxiv.org/pdf/2401.17010
• Copy Paste: [[2401.17010]] Finetuning Large Language Models for Vulnerability Detection(https://arxiv.org/abs/2401.17010)
• Keywords: large language model
• Abstract: This paper presents the results of finetuning large language models (LLMs) for the task of detecting vulnerabilities in source code. We leverage WizardCoder, a recent improvement of the state-of-the-art LLM StarCoder, and adapt it for vulnerability detection through further finetuning. To accelerate training, we modify WizardCoder's training procedure, also we investigate optimal training regimes. For the imbalanced dataset with many more negative examples than positive, we also explore different techniques to improve classification performance. The finetuned WizardCoder model achieves improvement in ROC AUC and F1 measures on balanced and imbalanced vulnerability datasets over CodeBERT-like model, demonstrating the effectiveness of adapting pretrained LLMs for vulnerability detection in source code. The key contributions are finetuning the state-of-the-art code LLM, WizardCoder, increasing its training speed without the performance harm, optimizing the training procedure and regimes, handling class imbalance, and improving performance on difficult vulnerability detection datasets. This demonstrates the potential for transfer learning by finetuning large pretrained language models for specialized source code analysis tasks.

Title: Evaluation of Out-of-Distribution Detection Performance on Autonomous Driving Datasets

• Authors: Jens Henriksson, Christian Berger, Stig Ursing, Markus Borg
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17013
• Pdf URL: https://arxiv.org/pdf/2401.17013
• Copy Paste: [[2401.17013]] Evaluation of Out-of-Distribution Detection Performance on Autonomous Driving Datasets(https://arxiv.org/abs/2401.17013)
• Keywords: segmentation
• Abstract: Safety measures need to be systemically investigated to what extent they evaluate the intended performance of Deep Neural Networks (DNNs) for critical applications. Due to a lack of verification methods for high-dimensional DNNs, a trade-off is needed between accepted performance and handling of out-of-distribution (OOD) samples. This work evaluates rejecting outputs from semantic segmentation DNNs by applying a Mahalanobis distance (MD) based on the most probable class-conditional Gaussian distribution for the predicted class as an OOD score. The evaluation follows three DNNs trained on the Cityscapes dataset and tested on four automotive datasets and finds that classification risk can drastically be reduced at the cost of pixel coverage, even when applied on unseen datasets. The applicability of our findings will support legitimizing safety measures and motivate their usage when arguing for safe usage of DNNs in automotive perception.

Title: MF-MOS: A Motion-Focused Model for Moving Object Segmentation

• Authors: Jintao Cheng, Kang Zeng, Zhuoxu Huang, Xiaoyu Tang, Jin Wu, Chengxi Zhang, Xieyuanli Chen, Rui Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17023
• Pdf URL: https://arxiv.org/pdf/2401.17023
• Copy Paste: [[2401.17023]] MF-MOS: A Motion-Focused Model for Moving Object Segmentation(https://arxiv.org/abs/2401.17023)
• Keywords: segmentation
• Abstract: Moving object segmentation (MOS) provides a reliable solution for detecting traffic participants and thus is of great interest in the autonomous driving field. Dynamic capture is always critical in the MOS problem. Previous methods capture motion features from the range images directly. Differently, we argue that the residual maps provide greater potential for motion information, while range images contain rich semantic guidance. Based on this intuition, we propose MF-MOS, a novel motion-focused model with a dual-branch structure for LiDAR moving object segmentation. Novelly, we decouple the spatial-temporal information by capturing the motion from residual maps and generating semantic features from range images, which are used as movable object guidance for the motion branch. Our straightforward yet distinctive solution can make the most use of both range images and residual maps, thus greatly improving the performance of the LiDAR-based MOS task. Remarkably, our MF-MOS achieved a leading IoU of 76.7% on the MOS leaderboard of the SemanticKITTI dataset upon submission, demonstrating the current state-of-the-art performance. The implementation of our MF-MOS has been released at https://github.com/SCNU-RISLAB/MF-MOS.

Title: Robust Kernel Sparse Subspace Clustering

• Authors: Ivica Kopriva
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17035
• Pdf URL: https://arxiv.org/pdf/2401.17035
• Copy Paste: [[2401.17035]] Robust Kernel Sparse Subspace Clustering(https://arxiv.org/abs/2401.17035)
• Keywords: robust
• Abstract: Kernel methods are applied to many problems in pattern recognition, including subspace clustering (SC). That way, nonlinear problems in the input data space become linear in mapped high-dimensional feature space. Thereby, computationally tractable nonlinear algorithms are enabled through implicit mapping by the virtue of kernel trick. However, kernelization of linear algorithms is possible only if square of the Froebenious norm of the error term is used in related optimization problem. That, however, implies normal distribution of the error. That is not appropriate for non-Gaussian errors such as gross sparse corruptions that are modeled by -norm. Herein, to the best of our knowledge, we propose for the first time robust kernel sparse SC (RKSSC) algorithm for data with gross sparse corruptions. The concept, in principle, can be applied to other SC algorithms to achieve robustness to the presence of such type of corruption. We validated proposed approach on two well-known datasets with linear robust SSC algorithm as a baseline model. According to Wilcoxon test, clustering performance obtained by the RKSSC algorithm is statistically significantly better than corresponding performance obtained by the robust SSC algorithm. MATLAB code of proposed RKSSC algorithm is posted on https://github.com/ikopriva/RKSSC.

Title: Towards Assessing the Synthetic-to-Measured Adversarial Vulnerability of SAR ATR

• Authors: Bowen Peng, Bo Peng, Jingyuan Xia, Tianpeng Liu, Yongxiang Liu, Li Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17038
• Pdf URL: https://arxiv.org/pdf/2401.17038
• Copy Paste: [[2401.17038]] Towards Assessing the Synthetic-to-Measured Adversarial Vulnerability of SAR ATR(https://arxiv.org/abs/2401.17038)
• Keywords: attack
• Abstract: Recently, there has been increasing concern about the vulnerability of deep neural network (DNN)-based synthetic aperture radar (SAR) automatic target recognition (ATR) to adversarial attacks, where a DNN could be easily deceived by clean input with imperceptible but aggressive perturbations. This paper studies the synthetic-to-measured (S2M) transfer setting, where an attacker generates adversarial perturbation based solely on synthetic data and transfers it against victim models trained with measured data. Compared with the current measured-to-measured (M2M) transfer setting, our approach does not need direct access to the victim model or the measured SAR data. We also propose the transferability estimation attack (TEA) to uncover the adversarial risks in this more challenging and practical scenario. The TEA makes full use of the limited similarity between the synthetic and measured data pairs for blind estimation and optimization of S2M transferability, leading to feasible surrogate model enhancement without mastering the victim model and data. Comprehensive evaluations based on the publicly available synthetic and measured paired labeled experiment (SAMPLE) dataset demonstrate that the TEA outperforms state-of-the-art methods and can significantly enhance various attack algorithms in computer vision and remote sensing applications. Codes and data are available at https://github.com/scenarri/S2M-TEA.

Title: Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests

• Authors: Brielen Madureira, David Schlangen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17039
• Pdf URL: https://arxiv.org/pdf/2401.17039
• Copy Paste: [[2401.17039]] Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests(https://arxiv.org/abs/2401.17039)
• Keywords: transformer
• Abstract: Clarification requests are a mechanism to help solve communication problems, e.g. due to ambiguity or underspecification, in instruction-following interactions. Despite their importance, even skilful models struggle with producing or interpreting such repair acts. In this work, we test three hypotheses concerning the effects of action taking as an auxiliary task in modelling iCR policies. Contrary to initial expectations, we conclude that its contribution to learning an iCR policy is limited, but some information can still be extracted from prediction uncertainty. We present further evidence that even well-motivated, Transformer-based models fail to learn good policies for when to ask Instruction CRs (iCRs), while the task of determining what to ask about can be more successfully modelled. Considering the implications of these findings, we further discuss the shortcomings of the data-driven paradigm for learning meta-communication acts.

Title: Forecasting VIX using Bayesian Deep Learning

• Authors: Héctor J. Hortúa, Andrés Mora-Valencia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17042
• Pdf URL: https://arxiv.org/pdf/2401.17042
• Copy Paste: [[2401.17042]] Forecasting VIX using Bayesian Deep Learning(https://arxiv.org/abs/2401.17042)
• Keywords: transformer
• Abstract: Recently, deep learning techniques are gradually replacing traditional statistical and machine learning models as the first choice for price forecasting tasks. In this paper, we leverage probabilistic deep learning for inferring the volatility index VIX. We employ the probabilistic counterpart of WaveNet, Temporal Convolutional Network (TCN), and Transformers. We show that TCN outperforms all models with an RMSE around 0.189. In addition, it has been well known that modern neural networks provide inaccurate uncertainty estimates. For solving this problem, we use the standard deviation scaling to calibrate the networks. Furthermore, we found out that MNF with Gaussian prior outperforms Reparameterization Trick and Flipout models in terms of precision and uncertainty predictions. Finally, we claim that MNF with Cauchy and LogUniform prior distributions yield well calibrated TCN and WaveNet networks being the former that best infer the VIX values.

Title: CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models

• Authors: Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, Enhong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17043
• Pdf URL: https://arxiv.org/pdf/2401.17043
• Copy Paste: [[2401.17043]] CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models(https://arxiv.org/abs/2401.17043)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This method addresses common LLM limitations, including outdated information and the tendency to produce inaccurate "hallucinated" content. However, the evaluation of RAG systems is challenging, as existing benchmarks are limited in scope and diversity. Most of the current benchmarks predominantly assess question-answering applications, overlooking the broader spectrum of situations where RAG could prove advantageous. Moreover, they only evaluate the performance of the LLM component of the RAG pipeline in the experiments, and neglect the influence of the retrieval component and the external knowledge database. To address these issues, this paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios. Specifically, we have categorized the range of RAG applications into four distinct types-Create, Read, Update, and Delete (CRUD), each representing a unique use case. "Create" refers to scenarios requiring the generation of original, varied content. "Read" involves responding to intricate questions in knowledge-intensive situations. "Update" focuses on revising and rectifying inaccuracies or inconsistencies in pre-existing texts. "Delete" pertains to the task of summarizing extensive texts into more concise forms. For each of these CRUD categories, we have developed comprehensive datasets to evaluate the performance of RAG systems. We also analyze the effects of various components of the RAG system, such as the retriever, the context length, the knowledge base construction, and the LLM. Finally, we provide useful insights for optimizing the RAG technology for different scenarios.

Title: ViTree: Single-path Neural Tree for Step-wise Interpretable Fine-grained Visual Categorization

• Authors: Danning Lao, Qi Liu, Jiazi Bu, Junchi Yan, Wei Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.17050
• Pdf URL: https://arxiv.org/pdf/2401.17050
• Copy Paste: [[2401.17050]] ViTree: Single-path Neural Tree for Step-wise Interpretable Fine-grained Visual Categorization(https://arxiv.org/abs/2401.17050)
• Keywords: extraction, interpretability, transformer
• Abstract: As computer vision continues to advance and finds widespread applications across various domains, the need for interpretability in deep learning models becomes paramount. Existing methods often resort to post-hoc techniques or prototypes to explain the decision-making process, which can be indirect and lack intrinsic illustration. In this research, we introduce ViTree, a novel approach for fine-grained visual categorization that combines the popular vision transformer as a feature extraction backbone with neural decision trees. By traversing the tree paths, ViTree effectively selects patches from transformer-processed features to highlight informative local regions, thereby refining representations in a step-wise manner. Unlike previous tree-based models that rely on soft distributions or ensembles of paths, ViTree selects a single tree path, offering a clearer and simpler decision-making process. This patch and path selectivity enhances model interpretability of ViTree, enabling better insights into the model's inner workings. Remarkably, extensive experimentation validates that this streamlined approach surpasses various strong competitors and achieves state-of-the-art performance while maintaining exceptional interpretability which is proved by multi-perspective methods. Code can be found at https://github.com/SJTU-DeepVisionLab/ViTree.

Title: Making Parametric Anomaly Detection on Tabular Data Non-Parametric Again

• Authors: Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17052
• Pdf URL: https://arxiv.org/pdf/2401.17052
• Copy Paste: [[2401.17052]] Making Parametric Anomaly Detection on Tabular Data Non-Parametric Again(https://arxiv.org/abs/2401.17052)
• Keywords: transformer
• Abstract: Deep learning for tabular data has garnered increasing attention in recent years, yet employing deep models for structured data remains challenging. While these models excel with unstructured data, their efficacy with structured data has been limited. Recent research has introduced retrieval-augmented models to address this gap, demonstrating promising results in supervised tasks such as classification and regression. In this work, we investigate using retrieval-augmented models for anomaly detection on tabular data. We propose a reconstruction-based approach in which a transformer model learns to reconstruct masked features of \textit{normal} samples. We test the effectiveness of KNN-based and attention-based modules to select relevant samples to help in the reconstruction process of the target sample. Our experiments on a benchmark of 31 tabular datasets reveal that augmenting this reconstruction-based anomaly detection (AD) method with non-parametric relationships via retrieval modules may significantly boost performance.

Title: BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation

• Authors: Zhennan Wu, Yang Li, Han Yan, Taizhang Shang, Weixuan Sun, Senbo Wang, Ruikai Cui, Weizhe Liu, Hiroyuki Sato, Hongdong Li, Pan Ji
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.17053
• Pdf URL: https://arxiv.org/pdf/2401.17053
• Copy Paste: [[2401.17053]] BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation(https://arxiv.org/abs/2401.17053)
• Keywords: diffusion
• Abstract: We present BlockFusion, a diffusion-based model that generates 3D scenes as unit blocks and seamlessly incorporates new blocks to extend the scene. BlockFusion is trained using datasets of 3D blocks that are randomly cropped from complete 3D scene meshes. Through per-block fitting, all training blocks are converted into the hybrid neural fields: with a tri-plane containing the geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the signed distance values. A variational auto-encoder is employed to compress the tri-planes into the latent tri-plane space, on which the denoising diffusion process is performed. Diffusion applied to the latent representations allows for high-quality and diverse 3D scene generation. To expand a scene during generation, one needs only to append empty blocks to overlap with the current scene and extrapolate existing latent tri-planes to populate new blocks. The extrapolation is done by conditioning the generation process with the feature samples from the overlapping tri-planes during the denoising iterations. Latent tri-plane extrapolation produces semantically and geometrically meaningful transitions that harmoniously blend with the existing scene. A 2D layout conditioning mechanism is used to control the placement and arrangement of scene elements. Experimental results indicate that BlockFusion is capable of generating diverse, geometrically consistent and unbounded large 3D scenes with unprecedented high-quality shapes in both indoor and outdoor scenarios.

Title: Atlanta Scaled layouts from non-central panoramas

• Authors: Bruno Berenguel-Baeta, Jesus Bermudez-Cameo, Jose J. Guerrero
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2401.17058
• Pdf URL: https://arxiv.org/pdf/2401.17058
• Copy Paste: [[2401.17058]] Atlanta Scaled layouts from non-central panoramas(https://arxiv.org/abs/2401.17058)
• Keywords: extraction
• Abstract: In this work we present a novel approach for 3D layout recovery of indoor environments using a non-central acquisition system. From a non-central panorama, full and scaled 3D lines can be independently recovered by geometry reasoning without geometric nor scale assumptions. However, their sensitivity to noise and complex geometric modeling has led these panoramas being little investigated. Our new pipeline aims to extract the boundaries of the structural lines of an indoor environment with a neural network and exploit the properties of non-central projection systems in a new geometrical processing to recover an scaled 3D layout. The results of our experiments show that we improve state-of-the-art methods for layout reconstruction and line extraction in non-central projection systems. We completely solve the problem in Manhattan and Atlanta environments, handling occlusions and retrieving the metric scale of the room without extra measurements. As far as the authors knowledge goes, our approach is the first work using deep learning on non-central panoramas and recovering scaled layouts from single panoramas.

Title: OmniSCV: An Omnidirectional Synthetic Image Generator for Computer Vision

• Authors: Bruno Berenguel-Baeta, Jesus Bermudez-Cameo, Jose J. Guerrero
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17061
• Pdf URL: https://arxiv.org/pdf/2401.17061
• Copy Paste: [[2401.17061]] OmniSCV: An Omnidirectional Synthetic Image Generator for Computer Vision(https://arxiv.org/abs/2401.17061)
• Keywords: extraction
• Abstract: Omnidirectional and 360{\deg} images are becoming widespread in industry and in consumer society, causing omnidirectional computer vision to gain attention. Their wide field of view allows the gathering of a great amount of information about the environment from only an image. However, the distortion of these images requires the development of specific algorithms for their treatment and interpretation. Moreover, a high number of images is essential for the correct training of computer vision algorithms based on learning. In this paper, we present a tool for generating datasets of omnidirectional images with semantic and depth information. These images are synthesized from a set of captures that are acquired in a realistic virtual environment for Unreal Engine 4 through an interface plugin. We gather a variety of well-known projection models such as equirectangular and cylindrical panoramas, different fish-eye lenses, catadioptric systems, and empiric models. Furthermore, we include in our tool photorealistic non-central-projection systems as non-central panoramas and non-central catadioptric systems. As far as we know, this is the first reported tool for generating photorealistic non-central images in the literature. Moreover, since the omnidirectional images are made virtually, we provide pixel-wise information about semantics and depth as well as perfect knowledge of the calibration parameters of the cameras. This allows the creation of ground-truth information with pixel precision for training learning algorithms and testing 3D vision approaches. To validate the proposed tool, different computer vision algorithms are tested as line extractions from dioptric and catadioptric central images, 3D Layout recovery and SLAM using equirectangular panoramas, and 3D reconstruction from non-central panoramas.

Title: SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity

• Authors: Ansar Aynetdinov, Alan Akbik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17072
• Pdf URL: https://arxiv.org/pdf/2401.17072
• Copy Paste: [[2401.17072]] SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity(https://arxiv.org/abs/2401.17072)
• Keywords: large language model
• Abstract: Instruction-tuned Large Language Models (LLMs) have recently showcased remarkable advancements in their ability to generate fitting responses to natural language instructions. However, many current works rely on manual evaluation to judge the quality of generated responses. Since such manual evaluation is time-consuming, it does not easily scale to the evaluation of multiple models and model variants. In this short paper, we propose a straightforward but remarkably effective evaluation metric called SemScore, in which we directly compare model outputs to gold target responses using semantic textual similarity (STS). We conduct a comparative evaluation of the model outputs of 12 prominent instruction-tuned LLMs using 8 widely-used evaluation metrics for text generation. We find that our proposed SemScore metric outperforms all other, in many cases more complex, evaluation metrics in terms of correlation to human evaluation. These findings indicate the utility of our proposed metric for the evaluation of instruction-tuned LLMs.

Title: Active Generation Network of Human Skeleton for Action Recognition

• Authors: Long Liu, Xin Wang, Fangming Li, Jiayu Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17086
• Pdf URL: https://arxiv.org/pdf/2401.17086
• Copy Paste: [[2401.17086]] Active Generation Network of Human Skeleton for Action Recognition(https://arxiv.org/abs/2401.17086)
• Keywords: generative
• Abstract: Data generation is a data augmentation technique for enhancing the generalization ability for skeleton-based human action recognition. Most existing data generation methods face challenges to ensure the temporal consistency of the dynamic information for action. In addition, the data generated by these methods lack diversity when only a few training samples are available. To solve those problems, We propose a novel active generative network (AGN), which can adaptively learn various action categories by motion style transfer to generate new actions when the data for a particular action is only a single sample or few samples. The AGN consists of an action generation network and an uncertainty metric network. The former, with ST-GCN as the Backbone, can implicitly learn the morphological features of the target action while preserving the category features of the source action. The latter guides generating actions. Specifically, an action recognition model generates prediction vectors for each action, which is then scored using an uncertainty metric. Finally, UMN provides the uncertainty sampling basis for the generated actions.

Title: NNOSE: Nearest Neighbor Occupational Skill Extraction

• Authors: Mike Zhang, Rob van der Goot, Min-Yen Kan, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17092
• Pdf URL: https://arxiv.org/pdf/2401.17092
• Copy Paste: [[2401.17092]] NNOSE: Nearest Neighbor Occupational Skill Extraction(https://arxiv.org/abs/2401.17092)
• Keywords: extraction
• Abstract: The labor market is changing rapidly, prompting increased interest in the automatic extraction of occupational skills from text. With the advent of English benchmark job description datasets, there is a need for systems that handle their diversity well. We tackle the complexity in occupational skill datasets tasks -- combining and leveraging multiple datasets for skill extraction, to identify rarely observed skills within a dataset, and overcoming the scarcity of skills across datasets. In particular, we investigate the retrieval-augmentation of language models, employing an external datastore for retrieving similar skills in a dataset-unifying manner. Our proposed method, \textbf{N}earest \textbf{N}eighbor \textbf{O}ccupational \textbf{S}kill \textbf{E}xtraction (NNOSE) effectively leverages multiple datasets by retrieving neighboring skills from other datasets in the datastore. This improves skill extraction \emph{without} additional fine-tuning. Crucially, we observe a performance gain in predicting infrequent patterns, with substantial gains of up to 30\% span-F1 in cross-dataset settings.

Title: StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis

• Authors: Zecheng Tang, Chenfei Wu, Zekai Zhang, Mingheng Ni, Shengming Yin, Yu Liu, Zhengyuan Yang, Lijuan Wang, Zicheng Liu, Juntao Li, Nan Duan
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17093
• Pdf URL: https://arxiv.org/pdf/2401.17093
• Copy Paste: [[2401.17093]] StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis(https://arxiv.org/abs/2401.17093)
• Keywords: segmentation
• Abstract: To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model's ability to capture the true semantic representation of visual scenes. This paper posits that an alternative representation of images, vector graphics, can effectively surmount this limitation by enabling a more natural and semantically coherent segmentation of the image information. Thus, we introduce StrokeNUWA, a pioneering work exploring a better visual representation ''stroke tokens'' on vector graphics, which is inherently visual semantics rich, naturally compatible with LLMs, and highly compressed. Equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods across various metrics in the vector graphic generation task. Besides, StrokeNUWA achieves up to a 94x speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6.9%.

Title: MT-Ranker: Reference-free machine translation evaluation by inter-system ranking

• Authors: Ibraheem Muhammad Moosa, Rui Zhang, Wenpeng Yin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17099
• Pdf URL: https://arxiv.org/pdf/2401.17099
• Copy Paste: [[2401.17099]] MT-Ranker: Reference-free machine translation evaluation by inter-system ranking(https://arxiv.org/abs/2401.17099)
• Keywords: interpretability
• Abstract: Traditionally, Machine Translation (MT) Evaluation has been treated as a regression problem -- producing an absolute translation-quality score. This approach has two limitations: i) the scores lack interpretability, and human annotators struggle with giving consistent scores; ii) most scoring methods are based on (reference, translation) pairs, limiting their applicability in real-world scenarios where references are absent. In practice, we often care about whether a new MT system is better or worse than some competitors. In addition, reference-free MT evaluation is increasingly practical and necessary. Unfortunately, these two practical considerations have yet to be jointly explored. In this work, we formulate the reference-free MT evaluation into a pairwise ranking problem. Given the source sentence and a pair of translations, our system predicts which translation is better. In addition to proposing this new formulation, we further show that this new paradigm can demonstrate superior correlation with human judgments by merely using indirect supervision from natural language inference and weak supervision from our synthetic data. In the context of reference-free evaluation, MT-Ranker, trained without any human annotations, achieves state-of-the-art results on the WMT Shared Metrics Task benchmarks DARR20, MQM20, and MQM21. On a more challenging benchmark, ACES, which contains fine-grained evaluation criteria such as addition, omission, and mistranslation errors, MT-Ranker marks state-of-the-art against reference-free as well as reference-based baselines.

Title: Evaluation in Neural Style Transfer: A Review

• Authors: Eleftherios Ioannou, Steve Maddock
• Subjects: cs.CV, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2401.17109
• Pdf URL: https://arxiv.org/pdf/2401.17109
• Copy Paste: [[2401.17109]] Evaluation in Neural Style Transfer: A Review(https://arxiv.org/abs/2401.17109)
• Keywords: robust, fair
• Abstract: The field of Neural Style Transfer (NST) has witnessed remarkable progress in the past few years, with approaches being able to synthesize artistic and photorealistic images and videos of exceptional quality. To evaluate such results, a diverse landscape of evaluation methods and metrics is used, including authors' opinions based on side-by-side comparisons, human evaluation studies that quantify the subjective judgements of participants, and a multitude of quantitative computational metrics which objectively assess the different aspects of an algorithm's performance. However, there is no consensus regarding the most suitable and effective evaluation procedure that can guarantee the reliability of the results. In this review, we provide an in-depth analysis of existing evaluation techniques, identify the inconsistencies and limitations of current evaluation methods, and give recommendations for standardized evaluation practices. We believe that the development of a robust evaluation framework will not only enable more meaningful and fairer comparisons among NST methods but will also enhance the comprehension and interpretation of research findings in the field.

Title: Explainable data-driven modeling via mixture of experts: towards effective blending of grey and black-box models

• Authors: Jessica Leoni, Valentina Breschi, Simone Formentin, Mara Tanelli
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.17118
• Pdf URL: https://arxiv.org/pdf/2401.17118
• Copy Paste: [[2401.17118]] Explainable data-driven modeling via mixture of experts: towards effective blending of grey and black-box models(https://arxiv.org/abs/2401.17118)
• Keywords: interpretability
• Abstract: Traditional models grounded in first principles often struggle with accuracy as the system's complexity increases. Conversely, machine learning approaches, while powerful, face challenges in interpretability and in handling physical constraints. Efforts to combine these models often often stumble upon difficulties in finding a balance between accuracy and complexity. To address these issues, we propose a comprehensive framework based on a "mixture of experts" rationale. This approach enables the data-based fusion of diverse local models, leveraging the full potential of first-principle-based priors. Our solution allows independent training of experts, drawing on techniques from both machine learning and system identification, and it supports both collaborative and competitive learning paradigms. To enhance interpretability, we penalize abrupt variations in the expert's combination. Experimental results validate the effectiveness of our approach in producing an interpretable combination of models closely resembling the target phenomena.

Title: Unsupervised Discovery of Steerable Factors When Graph Deep Generative Models Are Entangled

• Authors: Shengchao Liu, Chengpeng Wang, Jiarui Lu, Weili Nie, Hanchen Wang, Zhuoxinran Li, Bolei Zhou, Jian Tang
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2401.17123
• Pdf URL: https://arxiv.org/pdf/2401.17123
• Copy Paste: [[2401.17123]] Unsupervised Discovery of Steerable Factors When Graph Deep Generative Models Are Entangled(https://arxiv.org/abs/2401.17123)
• Keywords: generative
• Abstract: Deep generative models (DGMs) have been widely developed for graph data. However, much less investigation has been carried out on understanding the latent space of such pretrained graph DGMs. These understandings possess the potential to provide constructive guidelines for crucial tasks, such as graph controllable generation. Thus in this work, we are interested in studying this problem and propose GraphCG, a method for the unsupervised discovery of steerable factors in the latent space of pretrained graph DGMs. We first examine the representation space of three pretrained graph DGMs with six disentanglement metrics, and we observe that the pretrained representation space is entangled. Motivated by this observation, GraphCG learns the steerable factors via maximizing the mutual information between semantic-rich directions, where the controlled graph moving along the same direction will share the same steerable factors. We quantitatively verify that GraphCG outperforms four competitive baselines on two graph DGMs pretrained on two molecule datasets. Additionally, we qualitatively illustrate seven steerable factors learned by GraphCG on five pretrained DGMs over five graph datasets, including two for molecules and three for point clouds.

Title: Spectral Co-Distillation for Personalized Federated Learning

• Authors: Zihan Chen, Howard H. Yang, Tony Q.S. Quek, Kai Fong Ernest Chong
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.17124
• Pdf URL: https://arxiv.org/pdf/2401.17124
• Copy Paste: [[2401.17124]] Spectral Co-Distillation for Personalized Federated Learning(https://arxiv.org/abs/2401.17124)
• Keywords: federate
• Abstract: Personalized federated learning (PFL) has been widely investigated to address the challenge of data heterogeneity, especially when a single generic model is inadequate in satisfying the diverse performance requirements of local clients simultaneously. Existing PFL methods are inherently based on the idea that the relations between the generic global and personalized local models are captured by the similarity of model weights. Such a similarity is primarily based on either partitioning the model architecture into generic versus personalized components, or modeling client relationships via model weights. To better capture similar (yet distinct) generic versus personalized model representations, we propose \textit{spectral distillation}, a novel distillation method based on model spectrum information. Building upon spectral distillation, we also introduce a co-distillation framework that establishes a two-way bridge between generic and personalized model training. Moreover, to utilize the local idle time in conventional PFL, we propose a wait-free local training protocol. Through extensive experiments on multiple datasets over diverse heterogeneous data settings, we demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, as well as our wait-free training protocol.

Title: Personalized Differential Privacy for Ridge Regression

• Authors: Krishna Acharya, Franziska Boenisch, Rakshit Naidu, Juba Ziani
• Subjects: cs.LG, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.17127
• Pdf URL: https://arxiv.org/pdf/2401.17127
• Copy Paste: [[2401.17127]] Personalized Differential Privacy for Ridge Regression(https://arxiv.org/abs/2401.17127)
• Keywords: privacy, protect
• Abstract: The increased application of machine learning (ML) in sensitive domains requires protecting the training data through privacy frameworks, such as differential privacy (DP). DP requires to specify a uniform privacy level $\varepsilon$ that expresses the maximum privacy loss that each data point in the entire dataset is willing to tolerate. Yet, in practice, different data points often have different privacy requirements. Having to set one uniform privacy level is usually too restrictive, often forcing a learner to guarantee the stringent privacy requirement, at a large cost to accuracy. To overcome this limitation, we introduce our novel Personalized-DP Output Perturbation method (PDP-OP) that enables to train Ridge regression models with individual per data point privacy levels. We provide rigorous privacy proofs for our PDP-OP as well as accuracy guarantees for the resulting model. This work is the first to provide such theoretical accuracy guarantees when it comes to personalized DP in machine learning, whereas previous work only provided empirical evaluations. We empirically evaluate PDP-OP on synthetic and real datasets and with diverse privacy distributions. We show that by enabling each data point to specify their own privacy requirement, we can significantly improve the privacy-accuracy trade-offs in DP. We also show that PDP-OP outperforms the personalized privacy techniques of Jorgensen et al. (2015).

Title: Systematically Assessing the Security Risks of AI/ML-enabled Connected Healthcare Systems

• Authors: Mohammed Elnawawy, Mohammadreza Hallajiyan, Gargi Mitra, Shahrear Iqbal, Karthik Pattabiraman
• Subjects: cs.CR, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17136
• Pdf URL: https://arxiv.org/pdf/2401.17136
• Copy Paste: [[2401.17136]] Systematically Assessing the Security Risks of AI/ML-enabled Connected Healthcare Systems(https://arxiv.org/abs/2401.17136)
• Keywords: security, attack
• Abstract: The adoption of machine-learning-enabled systems in the healthcare domain is on the rise. While the use of ML in healthcare has several benefits, it also expands the threat surface of medical systems. We show that the use of ML in medical systems, particularly connected systems that involve interfacing the ML engine with multiple peripheral devices, has security risks that might cause life-threatening damage to a patient's health in case of adversarial interventions. These new risks arise due to security vulnerabilities in the peripheral devices and communication channels. We present a case study where we demonstrate an attack on an ML-enabled blood glucose monitoring system by introducing adversarial data points during inference. We show that an adversary can achieve this by exploiting a known vulnerability in the Bluetooth communication channel connecting the glucose meter with the ML-enabled app. We further show that state-of-the-art risk assessment techniques are not adequate for identifying and assessing these new risks. Our study highlights the need for novel risk analysis methods for analyzing the security of AI-enabled connected health devices.

Title: Large Language Model Evaluation via Matrix Entropy

• Authors: Lai Wei, Zhiquan Tan, Chenghai Li, Jindong Wang, Weiran Huang
• Subjects: cs.LG, cs.AI, cs.CL, cs.IT
• Abstract URL: https://arxiv.org/abs/2401.17139
• Pdf URL: https://arxiv.org/pdf/2401.17139
• Copy Paste: [[2401.17139]] Large Language Model Evaluation via Matrix Entropy(https://arxiv.org/abs/2401.17139)
• Keywords: large language model
• Abstract: Large language models (LLMs) have revolutionized the field of natural language processing, extending their strong capabilities into multi-modal domains. Thus, it is vital to define proper and diversified metrics for the evaluation of LLMs. In this paper, we introduce matrix entropy, a novel metric rooted in information theory and geometry principles to quantify the data compression proficiency in LLMs. It reflects the model's ability to extract relevant information and eliminate unnecessary elements, thereby providing insight into the language model's intrinsic capability. Specifically, we demonstrate its applicability in both single-modal (language) and multi-modal settings. For language models, our findings reveal that the matrix entropy of representations follows a scaling law type reduction when the model scales up, serving as a complement to the traditional loss scaling law. For the multi-modal setting, we also propose an evaluation method based on matrix entropy for assessing alignment quality and we find that modern large multi-modal models exhibit great alignment performance.

Title: Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios

• Authors: Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, Qun Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17167
• Pdf URL: https://arxiv.org/pdf/2401.17167
• Copy Paste: [[2401.17167]] Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios(https://arxiv.org/abs/2401.17167)
• Keywords: large language model
• Abstract: The recent trend of using Large Language Models (LLMs) as intelligent agents in real-world applications underscores the necessity for comprehensive evaluations of their capabilities, particularly in complex scenarios involving planning, creating, and using tools. However, existing benchmarks typically focus on simple synthesized queries that do not reflect real-world complexity, thereby offering limited perspectives in evaluating tool utilization. To address this issue, we present UltraTool, a novel benchmark designed to improve and evaluate LLMs' ability in tool utilization within real-world scenarios. UltraTool focuses on the entire process of using tools - from planning and creating to applying them in complex tasks. It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving. A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage and simplifies the task solving by mapping out the intermediate steps. Thus, unlike previous work, it eliminates the restriction of pre-defined toolset during planning. Through extensive experiments on various LLMs, we offer novel insights into the evaluation of capabilities of LLMs in tool utilization, thereby contributing a fresh perspective to this rapidly evolving field. The benchmark is publicly available at https://github.com/JoeYing1019/UltraTool.

Title: Conditional and Modal Reasoning in Large Language Models

• Authors: Wesley H. Holliday, Matthew Mandelkern
• Subjects: cs.CL, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2401.17169
• Pdf URL: https://arxiv.org/pdf/2401.17169
• Copy Paste: [[2401.17169]] Conditional and Modal Reasoning in Large Language Models(https://arxiv.org/abs/2401.17169)
• Keywords: large language model
• Abstract: The reasoning abilities of large language models (LLMs) are the topic of a growing body of research in artificial intelligence and cognitive science. In this paper, we probe the extent to which a dozen LLMs are able to distinguish logically correct inferences from logically fallacious ones. We focus on inference patterns involving conditionals (e.g., 'If Ann has a queen, then Bob has a jack') and epistemic modals (e.g., 'Ann might have an ace', 'Bob must have a king'). These inference patterns have been of special interest to logicians, philosophers, and linguists, since they plausibly play a central role in human reasoning. Assessing LLMs on these inference patterns is thus highly relevant to the question of how much the reasoning abilities of LLMs match those of humans. Among the LLMs we tested, all but GPT-4 often make basic mistakes with conditionals. Moreover, even GPT-4 displays logically inconsistent judgments across inference patterns involving epistemic modals.

Title: GraphViz2Vec: A Structure-aware Feature Generation Model to Improve Classification in GNNs

• Authors: Shraban Kumar Chatterjee, Suman Kundu
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.17178
• Pdf URL: https://arxiv.org/pdf/2401.17178
• Copy Paste: [[2401.17178]] GraphViz2Vec: A Structure-aware Feature Generation Model to Improve Classification in GNNs(https://arxiv.org/abs/2401.17178)
• Keywords: extraction
• Abstract: GNNs are widely used to solve various tasks including node classification and link prediction. Most of the GNN architectures assume the initial embedding to be random or generated from popular distributions. These initial embeddings require multiple layers of transformation to converge into a meaningful latent representation. While number of layers allow accumulation of larger neighbourhood of a node it also introduce the problem of over-smoothing. In addition, GNNs are inept at representing structural information. For example, the output embedding of a node does not capture its triangles participation. In this paper, we presented a novel feature extraction methodology GraphViz2Vec that can capture the structural information of a node's local neighbourhood to create meaningful initial embeddings for a GNN model. These initial embeddings helps existing models achieve state-of-the-art results in various classification tasks. Further, these initial embeddings help the model to produce desired results with only two layers which in turn reduce the problem of over-smoothing. The initial encoding of a node is obtained from an image classification model trained on multiple energy diagrams of its local neighbourhood. These energy diagrams are generated with the induced sub-graph of the nodes traversed by multiple random walks. The generated encodings increase the performance of existing models on classification tasks (with a mean increase of $4.65\%$ and $2.58\%$ for the node and link classification tasks, respectively), with some models achieving state-of-the-art results.

Title: Transfer Learning for Text Diffusion Models

• Authors: Kehang Han, Kathleen Kenealy, Aditya Barua, Noah Fiedel, Noah Constant
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17181
• Pdf URL: https://arxiv.org/pdf/2401.17181
• Copy Paste: [[2401.17181]] Transfer Learning for Text Diffusion Models(https://arxiv.org/abs/2401.17181)
• Keywords: diffusion, large language model
• Abstract: In this report, we explore the potential for text diffusion to replace autoregressive (AR) decoding for the training and deployment of large language models (LLMs). We are particularly interested to see whether pretrained AR models can be transformed into text diffusion models through a lightweight adaptation procedure we call ``AR2Diff''. We begin by establishing a strong baseline setup for training text diffusion models. Comparing across multiple architectures and pretraining objectives, we find that training a decoder-only model with a prefix LM objective is best or near-best across several tasks. Building on this finding, we test various transfer learning setups for text diffusion models. On machine translation, we find that text diffusion underperforms the standard AR approach. However, on code synthesis and extractive QA, we find diffusion models trained from scratch outperform AR models in many cases. We also observe quality gains from AR2Diff -- adapting AR models to use diffusion decoding. These results are promising given that text diffusion is relatively underexplored and can be significantly faster than AR decoding for long text generation.

Title: Single Word Change is All You Need: Designing Attacks and Defenses for Text Classifiers

• Authors: Lei Xu, Sarah Alnegheimish, Laure Berti-Equille, Alfredo Cuesta-Infante, Kalyan Veeramachaneni
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17196
• Pdf URL: https://arxiv.org/pdf/2401.17196
• Copy Paste: [[2401.17196]] Single Word Change is All You Need: Designing Attacks and Defenses for Text Classifiers(https://arxiv.org/abs/2401.17196)
• Keywords: defense, attack, robust
• Abstract: In text classification, creating an adversarial example means subtly perturbing a few words in a sentence without changing its meaning, causing it to be misclassified by a classifier. A concerning observation is that a significant portion of adversarial examples generated by existing methods change only one word. This single-word perturbation vulnerability represents a significant weakness in classifiers, which malicious users can exploit to efficiently create a multitude of adversarial examples. This paper studies this problem and makes the following key contributions: (1) We introduce a novel metric \r{ho} to quantitatively assess a classifier's robustness against single-word perturbation. (2) We present the SP-Attack, designed to exploit the single-word perturbation vulnerability, achieving a higher attack success rate, better preserving sentence meaning, while reducing computation costs compared to state-of-the-art adversarial methods. (3) We propose SP-Defense, which aims to improve \r{ho} by applying data augmentation in learning. Experimental results on 4 datasets and BERT and distilBERT classifiers show that SP-Defense improves \r{ho} by 14.6% and 13.9% and decreases the attack success rate of SP-Attack by 30.4% and 21.2% on two classifiers respectively, and decreases the attack success rate of existing attack methods that involve multiple-word perturbations.

Title: NormEnsembleXAI: Unveiling the Strengths and Weaknesses of XAI Ensemble Techniques

• Authors: Weronika Hryniewska-Guzik, Bartosz Sawicki, Przemysław Biecek
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17200
• Pdf URL: https://arxiv.org/pdf/2401.17200
• Copy Paste: [[2401.17200]] NormEnsembleXAI: Unveiling the Strengths and Weaknesses of XAI Ensemble Techniques(https://arxiv.org/abs/2401.17200)
• Keywords: interpretability
• Abstract: This paper presents a comprehensive comparative analysis of explainable artificial intelligence (XAI) ensembling methods. Our research brings three significant contributions. Firstly, we introduce a novel ensembling method, NormEnsembleXAI, that leverages minimum, maximum, and average functions in conjunction with normalization techniques to enhance interpretability. Secondly, we offer insights into the strengths and weaknesses of XAI ensemble methods. Lastly, we provide a library, facilitating the practical implementation of XAI ensembling, thus promoting the adoption of transparent and interpretable deep learning models.

Title: Self-Supervised Representation Learning for Nerve Fiber Distribution Patterns in 3D-PLI

• Authors: Alexander Oberstrass, Sascha E. A. Muenzing, Meiqi Niu, Nicola Palomero-Gallagher, Christian Schiffer, Markus Axer, Katrin Amunts, Timo Dickscheid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17207
• Pdf URL: https://arxiv.org/pdf/2401.17207
• Copy Paste: [[2401.17207]] Self-Supervised Representation Learning for Nerve Fiber Distribution Patterns in 3D-PLI(https://arxiv.org/abs/2401.17207)
• Keywords: robust
• Abstract: A comprehensive understanding of the organizational principles in the human brain requires, among other factors, well-quantifiable descriptors of nerve fiber architecture. Three-dimensional polarized light imaging (3D-PLI) is a microscopic imaging technique that enables insights into the fine-grained organization of myelinated nerve fibers with high resolution. Descriptors characterizing the fiber architecture observed in 3D-PLI would enable downstream analysis tasks such as multimodal correlation studies, clustering, and mapping. However, best practices for observer-independent characterization of fiber architecture in 3D-PLI are not yet available. To this end, we propose the application of a fully data-driven approach to characterize nerve fiber architecture in 3D-PLI images using self-supervised representation learning. We introduce a 3D-Context Contrastive Learning (CL-3D) objective that utilizes the spatial neighborhood of texture examples across histological brain sections of a 3D reconstructed volume to sample positive pairs for contrastive learning. We combine this sampling strategy with specifically designed image augmentations to gain robustness to typical variations in 3D-PLI parameter maps. The approach is demonstrated for the 3D reconstructed occipital lobe of a vervet monkey brain. We show that extracted features are highly sensitive to different configurations of nerve fibers, yet robust to variations between consecutive brain sections arising from histological processing. We demonstrate their practical applicability for retrieving clusters of homogeneous fiber architecture and performing data mining for interactively selected templates of specific components of fiber architecture such as U-fibers.

Title: ContactGen: Contact-Guided Interactive 3D Human Generation for Partners

• Authors: Dongjun Gu, Jaehyeok Shim, Jaehoon Jang, Changwoo Kang, Kyungdon Joo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17212
• Pdf URL: https://arxiv.org/pdf/2401.17212
• Copy Paste: [[2401.17212]] ContactGen: Contact-Guided Interactive 3D Human Generation for Partners(https://arxiv.org/abs/2401.17212)
• Keywords: diffusion
• Abstract: Among various interactions between humans, such as eye contact and gestures, physical interactions by contact can act as an essential moment in understanding human behaviors. Inspired by this fact, given a 3D partner human with the desired interaction label, we introduce a new task of 3D human generation in terms of physical contact. Unlike previous works of interacting with static objects or scenes, a given partner human can have diverse poses and different contact regions according to the type of interaction. To handle this challenge, we propose a novel method of generating interactive 3D humans for a given partner human based on a guided diffusion framework. Specifically, we newly present a contact prediction module that adaptively estimates potential contact regions between two input humans according to the interaction label. Using the estimated potential contact regions as complementary guidances, we dynamically enforce ContactGen to generate interactive 3D humans for a given partner human within a guided diffusion model. We demonstrate ContactGen on the CHI3D dataset, where our method generates physically plausible and diverse poses compared to comparison methods.

Title: MouSi: Poly-Visual-Expert Vision-Language Models

• Authors: Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang, Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17221
• Pdf URL: https://arxiv.org/pdf/2401.17221
• Copy Paste: [[2401.17221]] MouSi: Poly-Visual-Expert Vision-Language Models(https://arxiv.org/abs/2401.17221)
• Keywords: segmentation
• Abstract: Current large vision-language models (VLMs) often encounter challenges such as insufficient capabilities of a single visual component and excessively long visual tokens. These issues can limit the model's effectiveness in accurately interpreting complex visual information and over-lengthy contextual information. Addressing these challenges is crucial for enhancing the performance and applicability of VLMs. This paper proposes the use of ensemble experts technique to synergizes the capabilities of individual visual encoders, including those skilled in image-text matching, OCR, image segmentation, etc. This technique introduces a fusion network to unify the processing of outputs from different visual experts, while bridging the gap between image encoders and pre-trained LLMs. In addition, we explore different positional encoding schemes to alleviate the waste of positional encoding caused by lengthy image feature sequences, effectively addressing the issue of position overflow and length limitations. For instance, in our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient and manageable 64 or even down to 1. Experimental results demonstrate that VLMs with multiple experts exhibit consistently superior performance over isolated visual encoders and mark a significant performance boost as more experts are integrated. We have open-sourced the training code used in this report. All of these resources can be found on our project website.

Title: ReAlnet: Achieving More Human Brain-Like Vision via Human Neural Representational Alignment

• Authors: Zitong Lu, Yile Wang, Julie D. Golomb
• Subjects: cs.CV, cs.LG, cs.NE, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2401.17231
• Pdf URL: https://arxiv.org/pdf/2401.17231
• Copy Paste: [[2401.17231]] ReAlnet: Achieving More Human Brain-Like Vision via Human Neural Representational Alignment(https://arxiv.org/abs/2401.17231)
• Keywords: robust
• Abstract: Despite the remarkable strides made in artificial intelligence, current object recognition models still lag behind in emulating the mechanism of visual information processing in human brains. Recent studies have highlighted the potential of using neural data to mimic brain processing; however, these often reply on invasive neural recordings from non-human subjects, leaving a critical gap in our understanding of human visual perception and the development of more human brain-like vision models. Addressing this gap, we present, for the first time, "Re(presentational)Al(ignment)net", a vision model aligned with human brain activity based on non-invasive EEG recordings, demonstrating a significantly higher similarity to human brain representations. Our innovative image-to-brain multi-layer encoding alignment framework not only optimizes multiple layers of the model, marking a substantial leap in neural alignment, but also enables the model to efficiently learn and mimic human brain's visual representational patterns across object categories and different neural data modalities. Furthermore, we discover that alignment with human brain representations improves the model's adversarial robustness. Our findings suggest that ReAlnet sets a new precedent in the field, bridging the gap between artificial and human vision, and paving the way for more brain-like artificial intelligence systems.

Title: LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

• Authors: Yuan Chiang, Chia-Hong Chou, Janosh Riebesell
• Subjects: cs.CL, cond-mat.mtrl-sci, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.17244
• Pdf URL: https://arxiv.org/pdf/2401.17244
• Copy Paste: [[2401.17244]] LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation(https://arxiv.org/abs/2401.17244)
• Keywords: large language model
• Abstract: Reducing hallucination of Large Language Models (LLMs) is imperative for use in the sciences where reproducibility is crucial. However, LLMs inherently lack long-term memory, making it a nontrivial, ad hoc, and inevitably biased task to fine-tune them on domain-specific literature and data. Here we introduce LLaMP, a multimodal retrieval-augmented generation (RAG) framework of multiple data-aware reasoning-and-acting (ReAct) agents that dynamically interact with computational and experimental data on Materials Project (MP). Without fine-tuning, LLaMP demonstrates an ability to comprehend and integrate various modalities of materials science concepts, fetch relevant data stores on the fly, process higher-order data (such as crystal structures and elastic tensors), and summarize multi-step procedures for solid-state synthesis. We show that LLaMP effectively corrects errors in GPT-3.5's intrinsic knowledge, reducing a 5.21% MAPE on frequently-documented bandgaps and a significant 1103.54% MAPE on formation energies -- errors that GPT-3.5 seems to derive from mixed data sources. Additionally, LLaMP substantially reduces the hallucinated volumetric strain in a diamond cubic silicon structure from 66.3% to 0. The proposed framework offers an intuitive and nearly hallucination-free approach to exploring materials informatics and establishes a pathway for knowledge distillation and fine-tuning other language models. We envision the framework as a valuable component for scientific hypotheses and a foundation for future autonomous laboratories where multiple LLM agents communicate and cooperate with robotics to drive material synthesis and chemical reactions without hard-coded human logic and intervention.

Title: Weak-to-Strong Jailbreaking on Large Language Models

• Authors: Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, William Yang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17256
• Pdf URL: https://arxiv.org/pdf/2401.17256
• Copy Paste: [[2401.17256]] Weak-to-Strong Jailbreaking on Large Language Models(https://arxiv.org/abs/2401.17256)
• Keywords: protect, defense, attack, large language model
• Abstract: Although significant efforts have been dedicated to aligning large language models (LLMs), red-teaming reports suggest that these carefully aligned LLMs could still be jailbroken through adversarial prompts, tuning, or decoding. Upon examining the jailbreaking vulnerability of aligned LLMs, we observe that the decoding distributions of jailbroken and aligned models differ only in the initial generations. This observation motivates us to propose the weak-to-strong jailbreaking attack, where adversaries can utilize smaller unsafe/aligned LLMs (e.g., 7B) to guide jailbreaking against significantly larger aligned LLMs (e.g., 70B). To jailbreak, one only needs to additionally decode two smaller LLMs once, which involves minimal computation and latency compared to decoding the larger LLMs. The efficacy of this attack is demonstrated through experiments conducted on five models from three different organizations. Our study reveals a previously unnoticed yet efficient way of jailbreaking, exposing an urgent safety issue that needs to be considered when aligning LLMs. As an initial attempt, we propose a defense strategy to protect against such attacks, but creating more advanced defenses remains challenging. The code for replicating the method is available at https://github.com/XuandongZhao/weak-to-strong

Title: You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation

• Authors: Mehdi Noroozi, Isma Hadji, Brais Martinez, Adrian Bulat, Georgios Tzimiropoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17258
• Pdf URL: https://arxiv.org/pdf/2401.17258
• Copy Paste: [[2401.17258]] You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation(https://arxiv.org/abs/2401.17258)
• Keywords: diffusion
• Abstract: In this paper, we introduce YONOS-SR, a novel stable diffusion-based approach for image super-resolution that yields state-of-the-art results using only a single DDIM step. We propose a novel scale distillation approach to train our SR model. Instead of directly training our SR model on the scale factor of interest, we start by training a teacher model on a smaller magnification scale, thereby making the SR problem simpler for the teacher. We then train a student model for a higher magnification scale, using the predictions of the teacher as a target during the training. This process is repeated iteratively until we reach the target scale factor of the final model. The rationale behind our scale distillation is that the teacher aids the student diffusion model training by i) providing a target adapted to the current noise level rather than using the same target coming from ground truth data for all noise levels and ii) providing an accurate target as the teacher has a simpler task to solve. We empirically show that the distilled model significantly outperforms the model trained for high scales directly, specifically with few steps during inference. Having a strong diffusion model that requires only one step allows us to freeze the U-Net and fine-tune the decoder on top of it. We show that the combination of spatially distilled U-Net and fine-tuned decoder outperforms state-of-the-art methods requiring 200 steps with only one single step.

Title: Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks

• Authors: Andy Zhou, Bo Li, Haohan Wang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17263
• Pdf URL: https://arxiv.org/pdf/2401.17263
• Copy Paste: [[2401.17263]] Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks(https://arxiv.org/abs/2401.17263)
• Keywords: defense, attack, robust
• Abstract: Despite advances in AI alignment, language models (LM) remain vulnerable to adversarial attacks or jailbreaking, in which adversaries modify input prompts to induce harmful behavior. While some defenses have been proposed, they focus on narrow threat models and fall short of a strong defense, which we posit should be effective, universal, and practical. To achieve this, we propose the first adversarial objective for defending LMs against jailbreaking attacks and an algorithm, robust prompt optimization (RPO), that uses gradient-based token optimization to enforce harmless outputs. This results in an easily accessible suffix that significantly improves robustness to both jailbreaks seen during optimization and unknown, held-out jailbreaks, reducing the attack success rate on Starling-7B from 84% to 8.66% across 20 jailbreaks. In addition, we find that RPO has a minor effect on normal LM use, is successful under adaptive attacks, and can transfer to black-box models, reducing the success rate of the strongest attack on GPT-4 from 92% to 6%.

Title: Weaver: Foundation Models for Creative Writing

• Authors: Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, Yibin Liu, Jialong Wu, Shengwei Ding, Long Li, Zhiwei Huang, Xinle Deng, Teng Yu, Gangan Ma, Han Xiao, Zixin Chen, Danjun Xiang, Yunxia Wang, Yuanyuan Zhu, Yi Xiao, Jing Wang, Yiru Wang, Siran Ding, Jiayang Huang, Jiayi Xu, Yilihamu Tayier, Zhenyu Hu, Yuan Gao, Chengfeng Zheng, Yueshu Ye, Yihang Li, Lei Wan, Xinyue Jiang, Yujie Wang, Siyu Cheng, Zhule Song, Xiangru Tang, Xiaohua Xu, Ningyu Zhang, Huajun Chen, Yuchen Eleanor Jiang, Wangchunshu Zhou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17268
• Pdf URL: https://arxiv.org/pdf/2401.17268
• Copy Paste: [[2401.17268]] Weaver: Foundation Models for Creative Writing(https://arxiv.org/abs/2401.17268)
• Keywords: large language model
• Abstract: This work introduces Weaver, our first family of large language models (LLMs) dedicated to content creation. Weaver is pre-trained on a carefully selected corpus that focuses on improving the writing capabilities of large language models. We then fine-tune Weaver for creative and professional writing purposes and align it to the preference of professional writers using a suit of novel methods for instruction data synthesis and LLM alignment, making it able to produce more human-like texts and follow more diverse instructions for content creation. The Weaver family consists of models of Weaver Mini (1.8B), Weaver Base (6B), Weaver Pro (14B), and Weaver Ultra (34B) sizes, suitable for different applications and can be dynamically dispatched by a routing agent according to query complexity to balance response quality and computation cost. Evaluation on a carefully curated benchmark for assessing the writing capabilities of LLMs shows Weaver models of all sizes outperform generalist LLMs several times larger than them. Notably, our most-capable Weaver Ultra model surpasses GPT-4, a state-of-the-art generalist LLM, on various writing scenarios, demonstrating the advantage of training specialized LLMs for writing purposes. Moreover, Weaver natively supports retrieval-augmented generation (RAG) and function calling (tool usage). We present various use cases of these abilities for improving AI-assisted writing systems, including integration of external knowledge bases, tools, or APIs, and providing personalized writing assistance. Furthermore, we discuss and summarize a guideline and best practices for pre-training and fine-tuning domain-specific LLMs.

Title: YOLO-World: Real-Time Open-Vocabulary Object Detection

• Authors: Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17270
• Pdf URL: https://arxiv.org/pdf/2401.17270
• Copy Paste: [[2401.17270]] YOLO-World: Real-Time Open-Vocabulary Object Detection(https://arxiv.org/abs/2401.17270)
• Keywords: segmentation
• Abstract: The You Only Look Once (YOLO) series of detectors have established themselves as efficient and practical tools. However, their reliance on predefined and trained object categories limits their applicability in open scenarios. Addressing this limitation, we introduce YOLO-World, an innovative approach that enhances YOLO with open-vocabulary detection capabilities through vision-language modeling and pre-training on large-scale datasets. Specifically, we propose a new Re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN) and region-text contrastive loss to facilitate the interaction between visual and linguistic information. Our method excels in detecting a wide range of objects in a zero-shot manner with high efficiency. On the challenging LVIS dataset, YOLO-World achieves 35.4 AP with 52.0 FPS on V100, which outperforms many state-of-the-art methods in terms of both accuracy and speed. Furthermore, the fine-tuned YOLO-World achieves remarkable performance on several downstream tasks, including object detection and open-vocabulary instance segmentation.