2024-01-30

Title: Accelerating Material Property Prediction using Generically Complete Isometry Invariants

• Authors: Jonathan Balasingham, Viktor Zamaraev, Vitaliy Kurlin
• Subjects: cs.LG, cs.CG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2401.15089
• Pdf URL: https://arxiv.org/pdf/2401.15089
• Copy Paste: [[2401.15089]] Accelerating Material Property Prediction using Generically Complete Isometry Invariants(https://arxiv.org/abs/2401.15089)
• Keywords: transformer
• Abstract: Material or crystal property prediction using machine learning has grown popular in recent years as it provides a computationally efficient replacement to classical simulation methods. A crucial first step for any of these algorithms is the representation used for a periodic crystal. While similar objects like molecules and proteins have a finite number of atoms and their representation can be built based upon a finite point cloud interpretation, periodic crystals are unbounded in size, making their representation more challenging. In the present work, we adapt the Pointwise Distance Distribution (PDD), a continuous and generically complete isometry invariant for periodic point sets, as a representation for our learning algorithm. While the PDD is effective in distinguishing periodic point sets up to isometry, there is no consideration for the composition of the underlying material. We develop a transformer model with a modified self-attention mechanism that can utilize the PDD and incorporate compositional information via a spatial encoding method. This model is tested on the crystals of the Materials Project and Jarvis-DFT databases and shown to produce accuracy on par with state-of-the-art methods while being several times faster in both training and prediction time.

Title: Hi-Core: Hierarchical Knowledge Transfer for Continual Reinforcement Learning

• Authors: Chaofan Pan, Xin Yang, Hao Wang, Wei Wei, Tianrui Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15098
• Pdf URL: https://arxiv.org/pdf/2401.15098
• Copy Paste: [[2401.15098]] Hi-Core: Hierarchical Knowledge Transfer for Continual Reinforcement Learning(https://arxiv.org/abs/2401.15098)
• Keywords: large language model
• Abstract: Continual reinforcement learning (CRL) empowers RL agents with the ability to learn from a sequence of tasks, preserving previous knowledge and leveraging it to facilitate future learning. However, existing methods often focus on transferring low-level knowledge across similar tasks, which neglects the hierarchical structure of human cognitive control, resulting in insufficient knowledge transfer across diverse tasks. To enhance high-level knowledge transfer, we propose a novel framework named Hi-Core (Hierarchical knowledge transfer for Continual reinforcement learning), which is structured in two layers: 1) the high-level policy formulation which utilizes the powerful reasoning ability of the Large Language Model (LLM) to set goals and 2) the low-level policy learning through RL which is oriented by high-level goals. Moreover, the knowledge base (policy library) is constructed to store policies that can be retrieved for hierarchical knowledge transfer. Experiments conducted in MiniGrid have demonstrated the effectiveness of Hi-Core in handling diverse CRL tasks, outperforming popular baselines.

Title: Towards Global Glacier Mapping with Deep Learning and Open Earth Observation Data

• Authors: Konstantin A. Maslov, Claudio Persello, Thomas Schellenberger, Alfred Stein
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15113
• Pdf URL: https://arxiv.org/pdf/2401.15113
• Copy Paste: [[2401.15113]] Towards Global Glacier Mapping with Deep Learning and Open Earth Observation Data(https://arxiv.org/abs/2401.15113)
• Keywords: transformer
• Abstract: Accurate global glacier mapping is critical for understanding climate change impacts. It is challenged by glacier diversity, difficult-to-classify debris and big data processing. Here we propose Glacier-VisionTransformer-U-Net (GlaViTU), a convolutional-transformer deep learning model, and five strategies for multitemporal global-scale glacier mapping using open satellite imagery. Assessing the spatial, temporal and cross-sensor generalisation shows that our best strategy achieves intersection over union >0.85 on previously unobserved images in most cases, which drops to >0.75 for debris-rich areas such as High-Mountain Asia and increases to >0.90 for regions dominated by clean ice. Additionally, adding synthetic aperture radar data, namely, backscatter and interferometric coherence, increases the accuracy in all regions where available. The calibrated confidence for glacier extents is reported making the predictions more reliable and interpretable. We also release a benchmark dataset that covers 9% of glaciers worldwide. Our results support efforts towards automated multitemporal and global glacier mapping.

Title: Interpreting Time Series Transformer Models and Sensitivity Analysis of Population Age Groups to COVID-19 Infections

• Authors: Md Khairul Islam, Tyler Valentine, Timothy Joowon Sue, Ayush Karmacharya, Luke Neil Benham, Zhengguang Wang, Kingsley Kim, Judy Fox
• Subjects: cs.LG, cs.AI, q-bio.PE
• Abstract URL: https://arxiv.org/abs/2401.15119
• Pdf URL: https://arxiv.org/pdf/2401.15119
• Copy Paste: [[2401.15119]] Interpreting Time Series Transformer Models and Sensitivity Analysis of Population Age Groups to COVID-19 Infections(https://arxiv.org/abs/2401.15119)
• Keywords: transformer
• Abstract: Interpreting deep learning time series models is crucial in understanding the model's behavior and learning patterns from raw data for real-time decision-making. However, the complexity inherent in transformer-based time series models poses challenges in explaining the impact of individual features on predictions. In this study, we leverage recent local interpretation methods to interpret state-of-the-art time series models. To use real-world datasets, we collected three years of daily case data for 3,142 US counties. Firstly, we compare six transformer-based models and choose the best prediction model for COVID-19 infection. Using 13 input features from the last two weeks, we can predict the cases for the next two weeks. Secondly, we present an innovative way to evaluate the prediction sensitivity to 8 population age groups over highly dynamic multivariate infection data. Thirdly, we compare our proposed perturbation-based interpretation method with related work, including a total of eight local interpretation methods. Finally, we apply our framework to traffic and electricity datasets, demonstrating that our approach is generic and can be applied to other time-series domains.

Title: Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection

• Authors: Chen Liu, Shibo He, Qihang Zhou, Shizhong Li, Wenchao Meng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15123
• Pdf URL: https://arxiv.org/pdf/2401.15123
• Copy Paste: [[2401.15123]] Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection(https://arxiv.org/abs/2401.15123)
• Keywords: extraction, large language model
• Abstract: Self-supervised methods have gained prominence in time series anomaly detection due to the scarcity of available annotations. Nevertheless, they typically demand extensive training data to acquire a generalizable representation map, which conflicts with scenarios of a few available samples, thereby limiting their performance. To overcome the limitation, we propose \textbf{AnomalyLLM}, a knowledge distillation-based time series anomaly detection approach where the student network is trained to mimic the features of the large language model (LLM)-based teacher network that is pretrained on large-scale datasets. During the testing phase, anomalies are detected when the discrepancy between the features of the teacher and student networks is large. To circumvent the student network from learning the teacher network's feature of anomalous samples, we devise two key strategies. 1) Prototypical signals are incorporated into the student network to consolidate the normal feature extraction. 2) We use synthetic anomalies to enlarge the representation gap between the two networks. AnomalyLLM demonstrates state-of-the-art performance on 15 datasets, improving accuracy by at least 14.5\% in the UCR dataset.

Title: Evaluation of LLM Chatbots for OSINT-based Cyberthreat Awareness

• Authors: Samaneh Shafee, Alysson Bessani, Pedro M. Ferreira
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15127
• Pdf URL: https://arxiv.org/pdf/2401.15127
• Copy Paste: [[2401.15127]] Evaluation of LLM Chatbots for OSINT-based Cyberthreat Awareness(https://arxiv.org/abs/2401.15127)
• Keywords: security, large language model
• Abstract: Knowledge sharing about emerging threats is crucial in the rapidly advancing field of cybersecurity and forms the foundation of Cyber Threat Intelligence. In this context, Large Language Models are becoming increasingly significant in the field of cybersecurity, presenting a wide range of opportunities. This study explores the capability of chatbots such as ChatGPT, GPT4all, Dolly,Stanford Alpaca, Alpaca-LoRA, and Falcon to identify cybersecurity-related text within Open Source Intelligence. We assess the capabilities of existing chatbot models for Natural Language Processing tasks. We consider binary classification and Named Entity Recognition as tasks. This study analyzes well-established data collected from Twitter, derived from previous research efforts. Regarding cybersecurity binary classification, Chatbot GPT-4 as a commercial model achieved an acceptable F1-score of 0.94, and the open-source GPT4all model achieved an F1-score of 0.90. However, concerning cybersecurity entity recognition, chatbot models have limitations and are less effective. This study demonstrates the capability of these chatbots only for specific tasks, such as cybersecurity binary classification, while highlighting the need for further refinement in other tasks, such as Named Entity Recognition tasks.

Title: Chaotic Encryption for 10-Gb Ethernet Optical Links

• Authors: Adrián Pérez-Resa, Miguel Garcia-Bosque, Carlos Sánchez-Azqueta, Santiago Celma
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.15138
• Pdf URL: https://arxiv.org/pdf/2401.15138
• Copy Paste: [[2401.15138]] Chaotic Encryption for 10-Gb Ethernet Optical Links(https://arxiv.org/abs/2401.15138)
• Keywords: security
• Abstract: In this paper, a new physical layer encryption method for optical 10-Gb Ethernet links is proposed. Necessary modifications to introduce encryption in Ethernet 10GBase-R standard have been considered. This security enhancement has consisted of a symmetric streaming encryption of the 64b/66b data flow at physical coding sublayer level thanks to two keystream generators based on a chaotic algorithm. The overall system has been implemented and tested in a field programmable gate array. Ethernet traffic has been encrypted, transmitted, and decrypted over a multimode optical link. Experimental results are analyzed concluding that it is possible to cipher traffic at this level and hide the complete Ethernet traffic pattern from any passive eavesdropper. In addition, no overhead is introduced during encryption, getting no losses in the total throughput.

Title: Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks

• Authors: Zackary Okun Dunivin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15170
• Pdf URL: https://arxiv.org/pdf/2401.15170
• Copy Paste: [[2401.15170]] Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks(https://arxiv.org/abs/2401.15170)
• Keywords: large language model
• Abstract: Qualitative coding, or content analysis, extracts meaning from text to discern quantitative patterns across a corpus of texts. Recently, advances in the interpretive abilities of large language models (LLMs) offer potential for automating the coding process (applying category labels to texts), thereby enabling human researchers to concentrate on more creative research aspects, while delegating these interpretive tasks to AI. Our case study comprises a set of socio-historical codes on dense, paragraph-long passages representative of a humanistic study. We show that GPT-4 is capable of human-equivalent interpretations, whereas GPT-3.5 is not. Compared to our human-derived gold standard, GPT-4 delivers excellent intercoder reliability (Cohen's $\kappa \geq 0.79$) for 3 of 9 codes, and substantial reliability ($\kappa \geq 0.6$) for 8 of 9 codes. In contrast, GPT-3.5 greatly underperforms for all codes ($mean(\kappa) = 0.34$; $max(\kappa) = 0.55$). Importantly, we find that coding fidelity improves considerably when the LLM is prompted to give rationale justifying its coding decisions (chain-of-thought reasoning). We present these and other findings along with a set of best practices for adapting traditional codebooks for LLMs. Our results indicate that for certain codebooks, state-of-the-art LLMs are already adept at large-scale content analysis. Furthermore, they suggest the next generation of models will likely render AI coding a viable option for a majority of codebooks.

Title: Kitchen Food Waste Image Segmentation and Classification for Compost Nutrients Estimation

• Authors: Raiyan Rahman, Mohsena Chowdhury, Yueyang Tang, Huayi Gao, George Yin, Guanghui Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15175
• Pdf URL: https://arxiv.org/pdf/2401.15175
• Copy Paste: [[2401.15175]] Kitchen Food Waste Image Segmentation and Classification for Compost Nutrients Estimation(https://arxiv.org/abs/2401.15175)
• Keywords: segmentation
• Abstract: The escalating global concern over extensive food wastage necessitates innovative solutions to foster a net-zero lifestyle and reduce emissions. The LILA home composter presents a convenient means of recycling kitchen scraps and daily food waste into nutrient-rich, high-quality compost. To capture the nutritional information of the produced compost, we have created and annotated a large high-resolution image dataset of kitchen food waste with segmentation masks of 19 nutrition-rich categories. Leveraging this dataset, we benchmarked four state-of-the-art semantic segmentation models on food waste segmentation, contributing to the assessment of compost quality of Nitrogen, Phosphorus, or Potassium. The experiments demonstrate promising results of using segmentation models to discern food waste produced in our daily lives. Based on the experiments, SegFormer, utilizing MIT-B5 backbone, yields the best performance with a mean Intersection over Union (mIoU) of 67.09. Class-based results are also provided to facilitate further analysis of different food waste classes.

Title: FedGT: Federated Node Classification with Scalable Graph Transformer

• Authors: Zaixi Zhang, Qingyong Hu, Yang Yu, Weibo Gao, Qi Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15203
• Pdf URL: https://arxiv.org/pdf/2401.15203
• Copy Paste: [[2401.15203]] FedGT: Federated Node Classification with Scalable Graph Transformer(https://arxiv.org/abs/2401.15203)
• Keywords: privacy, protect, robust, federate, transformer
• Abstract: Graphs are widely used to model relational data. As graphs are getting larger and larger in real-world scenarios, there is a trend to store and compute subgraphs in multiple local systems. For example, recently proposed \emph{subgraph federated learning} methods train Graph Neural Networks (GNNs) distributively on local subgraphs and aggregate GNN parameters with a central server. However, existing methods have the following limitations: (1) The links between local subgraphs are missing in subgraph federated learning. This could severely damage the performance of GNNs that follow message-passing paradigms to update node/edge features. (2) Most existing methods overlook the subgraph heterogeneity issue, brought by subgraphs being from different parts of the whole graph. To address the aforementioned challenges, we propose a scalable \textbf{Fed}erated \textbf{G}raph \textbf{T}ransformer (\textbf{FedGT}) in the paper. Firstly, we design a hybrid attention scheme to reduce the complexity of the Graph Transformer to linear while ensuring a global receptive field with theoretical bounds. Specifically, each node attends to the sampled local neighbors and a set of curated global nodes to learn both local and global information and be robust to missing links. The global nodes are dynamically updated during training with an online clustering algorithm to capture the data distribution of the corresponding local subgraph. Secondly, FedGT computes clients' similarity based on the aligned global nodes with optimal transport. The similarity is then used to perform weighted averaging for personalized aggregation, which well addresses the data heterogeneity problem. Moreover, local differential privacy is applied to further protect the privacy of clients. Finally, extensive experimental results on 6 datasets and 2 subgraph settings demonstrate the superiority of FedGT.

Title: LYT-Net: Lightweight YUV Transformer-based Network for Low-Light Image Enhancement

• Authors: A. Brateanu, R. Balmez, A. Avram, C. C. Orhei
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.15204
• Pdf URL: https://arxiv.org/pdf/2401.15204
• Copy Paste: [[2401.15204]] LYT-Net: Lightweight YUV Transformer-based Network for Low-Light Image Enhancement(https://arxiv.org/abs/2401.15204)
• Keywords: transformer
• Abstract: In recent years, deep learning-based solutions have proven successful in the domains of image enhancement. This paper introduces LYT-Net, or Lightweight YUV Transformer-based Network, as a novel approach for low-light image enhancement. The proposed architecture, distinct from conventional Retinex-based models, leverages the YUV color space's natural separation of luminance (Y) and chrominance (U and V) to simplify the intricate task of disentangling light and color information in images. By utilizing the strengths of transformers, known for their capability to capture long-range dependencies, LYT-Net ensures a comprehensive contextual understanding of the image while maintaining reduced model complexity. By employing a novel hybrid loss function, our proposed method achieves state-of-the-art results on low-light image enhancement datasets, all while being considerably more compact than its counterparts. The source code and pre-trained models are available at https://github.com/albrateanu/LYT-Net

Title: Transfer Learning for the Prediction of Entity Modifiers in Clinical Text: Application to Opioid Use Disorder Case Detection

• Authors: Abdullateef I. Almudaifer, Tobias O`Leary, Whitney Covington, JaMor Hairston, Zachary Deitch, Ankit Anand, Caleb M. Carroll, Estera Crisan, William Bradford, Lauren Walter, Eaton Ellen, Sue S. Feldman, John D. Osborne
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15222
• Pdf URL: https://arxiv.org/pdf/2401.15222
• Copy Paste: [[2401.15222]] Transfer Learning for the Prediction of Entity Modifiers in Clinical Text: Application to Opioid Use Disorder Case Detection(https://arxiv.org/abs/2401.15222)
• Keywords: transformer
• Abstract: Background: The semantics of entities extracted from a clinical text can be dramatically altered by modifiers, including entity negation, uncertainty, conditionality, severity, and subject. Existing models for determining modifiers of clinical entities involve regular expression or features weights that are trained independently for each modifier. Methods: We develop and evaluate a multi-task transformer architecture design where modifiers are learned and predicted jointly using the publicly available SemEval 2015 Task 14 corpus and a new Opioid Use Disorder (OUD) data set that contains modifiers shared with SemEval as well as novel modifiers specific for OUD. We evaluate the effectiveness of our multi-task learning approach versus previously published systems and assess the feasibility of transfer learning for clinical entity modifiers when only a portion of clinical modifiers are shared. Results: Our approach achieved state-of-the-art results on the ShARe corpus from SemEval 2015 Task 14, showing an increase of 1.1% on weighted accuracy, 1.7% on unweighted accuracy, and 10% on micro F1 scores. Conclusions: We show that learned weights from our shared model can be effectively transferred to a new partially matched data set, validating the use of transfer learning for clinical text modifiers

Title: Biological Valuation Map of Flanders: A Sentinel-2 Imagery Analysis

• Authors: Mingshi Li, Dusan Grujicic, Steven De Saeger, Stien Heremans, Ben Somers, Matthew B. Blaschko
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15223
• Pdf URL: https://arxiv.org/pdf/2401.15223
• Copy Paste: [[2401.15223]] Biological Valuation Map of Flanders: A Sentinel-2 Imagery Analysis(https://arxiv.org/abs/2401.15223)
• Keywords: segmentation
• Abstract: In recent years, machine learning has become crucial in remote sensing analysis, particularly in the domain of Land-use/Land-cover (LULC). The synergy of machine learning and satellite imagery analysis has demonstrated significant productivity in this field, as evidenced by several studies. A notable challenge within this area is the semantic segmentation mapping of land usage over extensive territories, where the accessibility of accurate land-use data and the reliability of ground truth land-use labels pose significant difficulties. For example, providing a detailed and accurate pixel-wise labeled dataset of the Flanders region, a first-level administrative division of Belgium, can be particularly insightful. Yet there is a notable lack of regulated, formalized datasets and workflows for such studies in many regions globally. This paper introduces a comprehensive approach to addressing these gaps. We present a densely labeled ground truth map of Flanders paired with Sentinel-2 satellite imagery. Our methodology includes a formalized dataset division and sampling method, utilizing the topographic map layout 'Kaartbladversnijdingen,' and a detailed semantic segmentation model training pipeline. Preliminary benchmarking results are also provided to demonstrate the efficacy of our approach.

Title: Deep Learning with Tabular Data: A Self-supervised Approach

• Authors: Tirth Kiranbhai Vyas
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15238
• Pdf URL: https://arxiv.org/pdf/2401.15238
• Copy Paste: [[2401.15238]] Deep Learning with Tabular Data: A Self-supervised Approach(https://arxiv.org/abs/2401.15238)
• Keywords: transformer
• Abstract: We have described a novel approach for training tabular data using the TabTransformer model with self-supervised learning. Traditional machine learning models for tabular data, such as GBDT are being widely used though our paper examines the effectiveness of the TabTransformer which is a Transformer based model optimised specifically for tabular data. The TabTransformer captures intricate relationships and dependencies among features in tabular data by leveraging the self-attention mechanism of Transformers. We have used a self-supervised learning approach in this study, where the TabTransformer learns from unlabelled data by creating surrogate supervised tasks, eliminating the need for the labelled data. The aim is to find the most effective TabTransformer model representation of categorical and numerical features. To address the challenges faced during the construction of various input settings into the Transformers. Furthermore, a comparative analysis is also been conducted to examine performance of the TabTransformer model against baseline models such as MLP and supervised TabTransformer. The research has presented with a novel approach by creating various variants of TabTransformer model namely, Binned-TT, Vanilla-MLP-TT, MLP- based-TT which has helped to increase the effective capturing of the underlying relationship between various features of the tabular dataset by constructing optimal inputs. And further we have employed a self-supervised learning approach in the form of a masking-based unsupervised setting for tabular data. The findings shed light on the best way to represent categorical and numerical features, emphasizing the TabTransormer performance when compared to established machine learning models and other self-supervised learning methods.

Title: MEA-Defender: A Robust Watermark against Model Extraction Attack

• Authors: Peizhuo Lv, Hualong Ma, Kai Chen, Jiachen Zhou, Shengzhi Zhang, Ruigang Liang, Shenchen Zhu, Pan Li, Yingjun Zhang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15239
• Pdf URL: https://arxiv.org/pdf/2401.15239
• Copy Paste: [[2401.15239]] MEA-Defender: A Robust Watermark against Model Extraction Attack(https://arxiv.org/abs/2401.15239)
• Keywords: protect, attack, robust, extraction, watermark
• Abstract: Recently, numerous highly-valuable Deep Neural Networks (DNNs) have been trained using deep learning algorithms. To protect the Intellectual Property (IP) of the original owners over such DNN models, backdoor-based watermarks have been extensively studied. However, most of such watermarks fail upon model extraction attack, which utilizes input samples to query the target model and obtains the corresponding outputs, thus training a substitute model using such input-output pairs. In this paper, we propose a novel watermark to protect IP of DNN models against model extraction, named MEA-Defender. In particular, we obtain the watermark by combining two samples from two source classes in the input domain and design a watermark loss function that makes the output domain of the watermark within that of the main task samples. Since both the input domain and the output domain of our watermark are indispensable parts of those of the main task samples, the watermark will be extracted into the stolen model along with the main task during model extraction. We conduct extensive experiments on four model extraction attacks, using five datasets and six models trained based on supervised learning and self-supervised learning algorithms. The experimental results demonstrate that MEA-Defender is highly robust against different model extraction attacks, and various watermark removal/detection approaches.

Title: Training Differentially Private Ad Prediction Models with Semi-Sensitive Features

• Authors: Lynn Chua, Qiliang Cui, Badih Ghazi, Charlie Harrison, Pritish Kamath, Walid Krichene, Ravi Kumar, Pasin Manurangsi, Krishna Giri Narra, Amer Sinha, Avinash Varadarajan, Chiyuan Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15246
• Pdf URL: https://arxiv.org/pdf/2401.15246
• Copy Paste: [[2401.15246]] Training Differentially Private Ad Prediction Models with Semi-Sensitive Features(https://arxiv.org/abs/2401.15246)
• Keywords: protect, attack
• Abstract: Motivated by problems arising in digital advertising, we introduce the task of training differentially private (DP) machine learning models with semi-sensitive features. In this setting, a subset of the features is known to the attacker (and thus need not be protected) while the remaining features as well as the label are unknown to the attacker and should be protected by the DP guarantee. This task interpolates between training the model with full DP (where the label and all features should be protected) or with label DP (where all the features are considered known, and only the label should be protected). We present a new algorithm for training DP models with semi-sensitive features. Through an empirical evaluation on real ads datasets, we demonstrate that our algorithm surpasses in utility the baselines of (i) DP stochastic gradient descent (DP-SGD) run on all features (known and unknown), and (ii) a label DP algorithm run only on the known features (while discarding the unknown ones).

Title: Better Representations via Adversarial Training in Pre-Training: A Theoretical Perspective

• Authors: Yue Xing, Xiaofeng Lin, Qifan Song, Yi Xu, Belinda Zeng, Guang Cheng
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.15248
• Pdf URL: https://arxiv.org/pdf/2401.15248
• Copy Paste: [[2401.15248]] Better Representations via Adversarial Training in Pre-Training: A Theoretical Perspective(https://arxiv.org/abs/2401.15248)
• Keywords: attack, robust, large language model
• Abstract: Pre-training is known to generate universal representations for downstream tasks in large-scale deep learning such as large language models. Existing literature, e.g., \cite{kim2020adversarial}, empirically observe that the downstream tasks can inherit the adversarial robustness of the pre-trained model. We provide theoretical justifications for this robustness inheritance phenomenon. Our theoretical results reveal that feature purification plays an important role in connecting the adversarial robustness of the pre-trained model and the downstream tasks in two-layer neural networks. Specifically, we show that (i) with adversarial training, each hidden node tends to pick only one (or a few) feature; (ii) without adversarial training, the hidden nodes can be vulnerable to attacks. This observation is valid for both supervised pre-training and contrastive learning. With purified nodes, it turns out that clean training is enough to achieve adversarial robustness in downstream tasks.

Title: Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes

• Authors: Diandian Guo, Deng-Ping Fan, Tongyu Lu, Christos Sakaridis, Luc Van Gool
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15261
• Pdf URL: https://arxiv.org/pdf/2401.15261
• Copy Paste: [[2401.15261]] Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes(https://arxiv.org/abs/2401.15261)
• Keywords: segmentation
• Abstract: The estimation of implicit cross-frame correspondences and the high computational cost have long been major challenges in video semantic segmentation (VSS) for driving scenes. Prior works utilize keyframes, feature propagation, or cross-frame attention to address these issues. By contrast, we are the first to harness vanishing point (VP) priors for more effective segmentation. Intuitively, objects near VPs (i.e., away from the vehicle) are less discernible. Moreover, they tend to move radially away from the VP over time in the usual case of a forward-facing camera, a straight road, and linear forward motion of the vehicle. Our novel, efficient network for VSS, named VPSeg, incorporates two modules that utilize exactly this pair of static and dynamic VP priors: sparse-to-dense feature mining (DenseVP) and VP-guided motion fusion (MotionVP). MotionVP employs VP-guided motion estimation to establish explicit correspondences across frames and help attend to the most relevant features from neighboring frames, while DenseVP enhances weak dynamic features in distant regions around VPs. These modules operate within a context-detail framework, which separates contextual features from high-resolution local features at different input resolutions to reduce computational costs. Contextual and local features are integrated through contextualized motion attention (CMA) for the final prediction. Extensive experiments on two popular driving segmentation benchmarks, Cityscapes and ACDC, demonstrate that VPSeg outperforms previous SOTA methods, with only modest computational overhead.

Title: SAM-based instance segmentation models for the automation of masonry crack detection

• Authors: Zehao Ye, Lucy Lovell, Asaad Faramarzi, Jelena Ninic
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15266
• Pdf URL: https://arxiv.org/pdf/2401.15266
• Copy Paste: [[2401.15266]] SAM-based instance segmentation models for the automation of masonry crack detection(https://arxiv.org/abs/2401.15266)
• Keywords: segmentation
• Abstract: Automating visual inspection for capturing defects based on civil structures appearance is crucial due to its currently labour-intensive and time-consuming nature. An important aspect of automated inspection is image acquisition, which is rapid and cost-effective considering the pervasive developments in both software and hardware computing in recent years. Previous studies largely focused on concrete and asphalt, with less attention to masonry cracks. The latter also lacks publicly available datasets. In this paper, we first present a corresponding data set for instance segmentation with 1,300 annotated images (640 pixels x 640 pixels), named as MCrack1300, covering bricks, broken bricks, and cracks. We then test several leading algorithms for benchmarking, including the latest large-scale model, the prompt-based Segment Anything Model (SAM). We fine-tune the encoder using Low-Rank Adaptation (LoRA) and proposed two novel methods for automation of SAM execution. The first method involves abandoning the prompt encoder and connecting the SAM encoder to other decoders, while the second method introduces a learnable self-generating prompter. In order to ensure the seamless integration of the two proposed methods with SAM encoder section, we redesign the feature extractor. Both proposed methods exceed state-of-the-art performance, surpassing the best benchmark by approximately 3% for all classes and around 6% for cracks specifically. Based on successful detection, we propose a method based on a monocular camera and the Hough Line Transform to automatically transform images into orthographic projection maps. By incorporating known real sizes of brick units, we accurately estimate crack dimensions, with the results differing by less than 10% from those obtained by laser scanning. Overall, we address important research gaps in automated masonry crack detection and size estimation.

Title: Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models

• Authors: Minbyul Jeong, Jiwoong Sohn, Mujeen Sung, Jaewoo Kang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.15269
• Pdf URL: https://arxiv.org/pdf/2401.15269
• Copy Paste: [[2401.15269]] Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models(https://arxiv.org/abs/2401.15269)
• Keywords: large language model
• Abstract: Recent proprietary large language models (LLMs), such as GPT-4, have achieved a milestone in tackling diverse challenges in the biomedical domain, ranging from multiple-choice questions to long-form generations. To address challenges that still cannot be handled with the encoded knowledge of LLMs, various retrieval-augmented generation (RAG) methods have been developed by searching documents from the knowledge corpus and appending them unconditionally or selectively to the input of LLMs for generation. However, when applying existing methods to different domain-specific problems, poor generalization becomes apparent, leading to fetching incorrect documents or making inaccurate judgments. In this paper, we introduce Self-BioRAG, a framework reliable for biomedical text that specializes in generating explanations, retrieving domain-specific documents, and self-reflecting generated responses. We utilize 84k filtered biomedical instruction sets to train Self-BioRAG that can assess its generated explanations with customized reflective tokens. Our work proves that domain-specific components, such as a retriever, domain-related document corpus, and instruction sets are necessary for adhering to domain-related instructions. Using three major medical question-answering benchmark datasets, experimental results of Self-BioRAG demonstrate significant performance gains by achieving a 7.2% absolute improvement on average over the state-of-the-art open-foundation model with a parameter size of 7B or less. Overall, we analyze that Self-BioRAG finds the clues in the question, retrieves relevant documents if needed, and understands how to answer with information from retrieved documents and encoded knowledge as a medical expert does. We release our data and code for training our framework components and model weights (7B and 13B) to enhance capabilities in biomedical and clinical domains.

Title: SimFair: Physics-Guided Fairness-Aware Learning with Simulation Models

• Authors: Zhihao Wang, Yiqun Xie, Zhili Li, Xiaowei Jia, Zhe Jiang, Aolin Jia, Shuo Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15270
• Pdf URL: https://arxiv.org/pdf/2401.15270
• Copy Paste: [[2401.15270]] SimFair: Physics-Guided Fairness-Aware Learning with Simulation Models(https://arxiv.org/abs/2401.15270)
• Keywords: fair
• Abstract: Fairness-awareness has emerged as an essential building block for the responsible use of artificial intelligence in real applications. In many cases, inequity in performance is due to the change in distribution over different regions. While techniques have been developed to improve the transferability of fairness, a solution to the problem is not always feasible with no samples from the new regions, which is a bottleneck for pure data-driven attempts. Fortunately, physics-based mechanistic models have been studied for many problems with major social impacts. We propose SimFair, a physics-guided fairness-aware learning framework, which bridges the data limitation by integrating physical-rule-based simulation and inverse modeling into the training design. Using temperature prediction as an example, we demonstrate the effectiveness of the proposed SimFair in fairness preservation.

Title: Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning

• Authors: Chenyu Zhang, Han Wang, Aritra Mitra, James Anderson
• Subjects: cs.LG, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2401.15273
• Pdf URL: https://arxiv.org/pdf/2401.15273
• Copy Paste: [[2401.15273]] Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning(https://arxiv.org/abs/2401.15273)
• Keywords: federate
• Abstract: Federated reinforcement learning (FRL) has emerged as a promising paradigm for reducing the sample complexity of reinforcement learning tasks by exploiting information from different agents. However, when each agent interacts with a potentially different environment, little to nothing is known theoretically about the non-asymptotic performance of FRL algorithms. The lack of such results can be attributed to various technical challenges and their intricate interplay: Markovian sampling, linear function approximation, multiple local updates to save communication, heterogeneity in the reward functions and transition kernels of the agents' MDPs, and continuous state-action spaces. Moreover, in the on-policy setting, the behavior policies vary with time, further complicating the analysis. In response, we introduce FedSARSA, a novel federated on-policy reinforcement learning scheme, equipped with linear function approximation, to address these challenges and provide a comprehensive finite-time error analysis. Notably, we establish that FedSARSA converges to a policy that is near-optimal for all agents, with the extent of near-optimality proportional to the level of heterogeneity. Furthermore, we prove that FedSARSA leverages agent collaboration to enable linear speedups as the number of agents increases, which holds for both fixed and adaptive step-size configurations.

Title: Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks

• Authors: Yuliang Cai, Mohammad Rostami
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15275
• Pdf URL: https://arxiv.org/pdf/2401.15275
• Copy Paste: [[2401.15275]] Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks(https://arxiv.org/abs/2401.15275)
• Keywords: transformer
• Abstract: Transformer neural networks are increasingly replacing prior architectures in a wide range of applications in different data modalities. The increasing size and computational demands of fine-tuning large pre-trained transformer neural networks pose significant challenges for the widespread adoption of these models for applications that demand on-edge computing. To tackle this challenge, continual learning (CL) emerges as a solution by facilitating the transfer of knowledge across tasks that arrive sequentially for an autonomously learning agent. However, current CL methods mainly focus on learning tasks that are exclusively vision-based or language-based. We propose a transformer-based CL framework focusing on learning tasks that involve both vision and language, known as Vision-and-Language (VaL) tasks. Due to the success of transformers in other modalities, our architecture has the potential to be used in multimodal learning settings. In our framework, we benefit from introducing extra parameters to a base transformer to specialize the network for each task. As a result, we enable dynamic model expansion to learn several tasks in a sequence. We also use knowledge distillation to benefit from relevant past experiences to learn the current task more efficiently. Our proposed method, Task Attentive Multimodal Continual Learning (TAM-CL), allows for the exchange of information between tasks while mitigating the problem of catastrophic forgetting. Notably, our approach is scalable, incurring minimal memory and time overhead. TAM-CL achieves state-of-the-art (SOTA) performance on challenging multimodal tasks

Title: GEM: Boost Simple Network for Glass Surface Segmentation via Segment Anything Model and Data Synthesis

• Authors: Jing Hao, Moyun Liu, Kuo Feng Hung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15282
• Pdf URL: https://arxiv.org/pdf/2401.15282
• Copy Paste: [[2401.15282]] GEM: Boost Simple Network for Glass Surface Segmentation via Segment Anything Model and Data Synthesis(https://arxiv.org/abs/2401.15282)
• Keywords: diffusion, segmentation
• Abstract: Detecting glass regions is a challenging task due to the ambiguity of their transparency and reflection properties. These transparent glasses share the visual appearance of both transmitted arbitrary background scenes and reflected objects, thus having no fixed patterns.Recent visual foundation models, which are trained on vast amounts of data, have manifested stunning performance in terms of image perception and image generation. To segment glass surfaces with higher accuracy, we make full use of two visual foundation models: Segment Anything (SAM) and Stable Diffusion.Specifically, we devise a simple glass surface segmentor named GEM, which only consists of a SAM backbone, a simple feature pyramid, a discerning query selection module, and a mask decoder. The discerning query selection can adaptively identify glass surface features, assigning them as initialized queries in the mask decoder. We also propose a Synthetic but photorealistic large-scale Glass Surface Detection dataset dubbed S-GSD via diffusion model with four different scales, which contain 1x, 5x, 10x, and 20x of the original real data size. This dataset is a feasible source for transfer learning. The scale of synthetic data has positive impacts on transfer learning, while the improvement will gradually saturate as the amount of data increases. Extensive experiments demonstrate that GEM achieves a new state-of-the-art on the GSD-S validation set (IoU +2.1%). Codes and datasets are available at: https://github.com/isbrycee/GEM-Glass-Segmentor.

Title: Ransomware threat mitigation through network traffic analysis and machine learning techniques

• Authors: Ali Mehrban, Shirin Karimi Geransayeh
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15285
• Pdf URL: https://arxiv.org/pdf/2401.15285
• Copy Paste: [[2401.15285]] Ransomware threat mitigation through network traffic analysis and machine learning techniques(https://arxiv.org/abs/2401.15285)
• Keywords: attack
• Abstract: In recent years, there has been a noticeable increase in cyberattacks using ransomware. Attackers use this malicious software to break into networks and harm computer systems. This has caused significant and lasting damage to various organizations, including government, private companies, and regular users. These attacks often lead to the loss or exposure of sensitive information, disruptions in normal operations, and persistent vulnerabilities. This paper focuses on a method for recognizing and identifying ransomware in computer networks. The approach relies on using machine learning algorithms and analyzing the patterns of network traffic. By collecting and studying this traffic, and then applying machine learning models, we can accurately identify and detect ransomware. The results of implementing this method show that machine learning algorithms can effectively pinpoint ransomware based on network traffic, achieving high levels of precision and accuracy.

Title: Applications of Tao General Difference in Discrete Domain

• Authors: Linmi Tao, Ruiyang Liu, Donglai Tao, Wu Xia, Feilong Ma, Yu Cheng, Jingmao Cui
• Subjects: cs.CV, cs.DM, math.NA
• Abstract URL: https://arxiv.org/abs/2401.15287
• Pdf URL: https://arxiv.org/pdf/2401.15287
• Copy Paste: [[2401.15287]] Applications of Tao General Difference in Discrete Domain(https://arxiv.org/abs/2401.15287)
• Keywords: robust
• Abstract: Numerical difference computation is one of the cores and indispensable in the modern digital era. Tao general difference (TGD) is a novel theory and approach to difference computation for discrete sequences and arrays in multidimensional space. Built on the solid theoretical foundation of the general difference in a finite interval, the TGD operators demonstrate exceptional signal processing capabilities in real-world applications. A novel smoothness property of a sequence is defined on the first- and second TGD. This property is used to denoise one-dimensional signals, where the noise is the non-smooth points in the sequence. Meanwhile, the center of the gradient in a finite interval can be accurately location via TGD calculation. This solves a traditional challenge in computer vision, which is the precise localization of image edges with noise robustness. Furthermore, the power of TGD operators extends to spatio-temporal edge detection in three-dimensional arrays, enabling the identification of kinetic edges in video data. These diverse applications highlight the properties of TGD in discrete domain and the significant promise of TGD for the computation across signal processing, image analysis, and video analytic.

Title: Where's the "up"?! A Comprehensive (bottom-up) Study on the Security of Arm Cortex-M Systems

• Authors: Xi Tan, Zheyuan Ma, Sandro Pinto, Le Guan, Ning Zhang, Jun Xu, Zhiqiang Lin, Hongxing Hu, Ziming Zhao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.15289
• Pdf URL: https://arxiv.org/pdf/2401.15289
• Copy Paste: [[2401.15289]] Where's the "up"?! A Comprehensive (bottom-up) Study on the Security of Arm Cortex-M Systems(https://arxiv.org/abs/2401.15289)
• Keywords: security, protect
• Abstract: Arm Cortex-M processors are the most widely used 32-bit microcontrollers among embedded and Internetof-Things devices. Despite the widespread usage, there has been little effort in summarizing their hardware security features, characterizing the limitations and vulnerabilities of their hardware and software stack, and systematizing the research on securing these systems. The goals and contributions of this paper are multi-fold. First, we analyze the hardware security limitations and issues of Cortex-M systems. Second, we conducted a deep study of the software stack designed for Cortex-M and revealed its limitations, which is accompanied by an empirical analysis of 1,797 real-world firmware from seven hardware vendors. Third, we categorize the reported bugs in Cortex-M software systems. Finally, we systematize the efforts that aim at securing Cortex-M systems and evaluate them in terms of the protections they offer, run-time performance, required hardware features, etc. Based on the insights, we develop a set of recommendations for the research community and MCU software developers.

Title: SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection

• Authors: Foozhan Ataiefard, Walid Ahmed, Habib Hajimolahoseini, Saina Asani, Farnoosh Javadi, Mohammad Hassanpour, Omar Mohamed Awad, Austin Wen, Kangling Liu, Yang Liu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15293
• Pdf URL: https://arxiv.org/pdf/2401.15293
• Copy Paste: [[2401.15293]] SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection(https://arxiv.org/abs/2401.15293)
• Keywords: transformer
• Abstract: Vision transformers are known to be more computationally and data-intensive than CNN models. These transformer models such as ViT, require all the input image tokens to learn the relationship among them. However, many of these tokens are not informative and may contain irrelevant information such as unrelated background or unimportant scenery. These tokens are overlooked by the multi-head self-attention (MHSA), resulting in many redundant and unnecessary computations in MHSA and the feed-forward network (FFN). In this work, we propose a method to optimize the amount of unnecessary interactions between unimportant tokens by separating and sending them through a different low-cost computational path. Our method does not add any parameters to the ViT model and aims to find the best trade-off between training throughput and achieving a 0% loss in the Top-1 accuracy of the final model. Our experimental results on training ViT-small from scratch show that SkipViT is capable of effectively dropping 55% of the tokens while gaining more than 13% training throughput and maintaining classification accuracy at the level of the baseline model on Huawei Ascend910A.

Title: Multi-Trigger Backdoor Attacks: More Triggers, More Threats

• Authors: Yige Li, Xingjun Ma, Jiabo He, Hanxun Huang, Yu-Gang Jiang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15295
• Pdf URL: https://arxiv.org/pdf/2401.15295
• Copy Paste: [[2401.15295]] Multi-Trigger Backdoor Attacks: More Triggers, More Threats(https://arxiv.org/abs/2401.15295)
• Keywords: security, defense, attack
• Abstract: Backdoor attacks have emerged as a primary threat to (pre-)training and deployment of deep neural networks (DNNs). While backdoor attacks have been extensively studied in a body of works, most of them were focused on single-trigger attacks that poison a dataset using a single type of trigger. Arguably, real-world backdoor attacks can be much more complex, e.g., the existence of multiple adversaries for the same dataset if it is of high value. In this work, we investigate the practical threat of backdoor attacks under the setting of \textbf{multi-trigger attacks} where multiple adversaries leverage different types of triggers to poison the same dataset. By proposing and investigating three types of multi-trigger attacks, including parallel, sequential, and hybrid attacks, we provide a set of important understandings of the coexisting, overwriting, and cross-activating effects between different triggers on the same dataset. Moreover, we show that single-trigger attacks tend to cause overly optimistic views of the security of current defense techniques, as all examined defense methods struggle to defend against multi-trigger attacks. Finally, we create a multi-trigger backdoor poisoning dataset to help future evaluation of backdoor attacks and defenses. Although our work is purely empirical, we hope it can help steer backdoor research toward more realistic settings.

Title: Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Finance

• Authors: Adrian Theuma, Ehsan Shareghi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15328
• Pdf URL: https://arxiv.org/pdf/2401.15328
• Copy Paste: [[2401.15328]] Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Finance(https://arxiv.org/abs/2401.15328)
• Keywords: large language model
• Abstract: Large language models (LLMs) have exhibited an array of reasoning capabilities but face challenges like error propagation and hallucination, particularly in specialised areas like finance, where data is heterogeneous, and precision is paramount. We explore the potential of language model augmentation with external tools to mitigate these limitations and offload certain reasoning steps to external tools that are more suited for the task, instead of solely depending on the LLM's inherent abilities. More concretely, using financial domain question-answering datasets, we apply supervised fine-tuning on a LLaMA-2 13B Chat model to act both as a 'task router' and 'task solver'. The 'task router' dynamically directs a question to either be answered internally by the LLM or externally via the right tool from the tool set. Our tool-equipped SFT model, Raven, demonstrates an improvement of 35.2% and 5.06% over the base model and SFT-only baselines, respectively, and is highly competitive with strong GPT-3.5 results. To the best of our knowledge, our work is the first that investigates tool augmentation of language models for the finance domain.

Title: Optimal Sparse Survival Trees

• Authors: Rui Zhang, Rui Xin, Margo Seltzer, Cynthia Rudin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15330
• Pdf URL: https://arxiv.org/pdf/2401.15330
• Copy Paste: [[2401.15330]] Optimal Sparse Survival Trees(https://arxiv.org/abs/2401.15330)
• Keywords: interpretability
• Abstract: Interpretability is crucial for doctors, hospitals, pharmaceutical companies and biotechnology corporations to analyze and make decisions for high stakes problems that involve human health. Tree-based methods have been widely adopted for \textit{survival analysis} due to their appealing interpretablility and their ability to capture complex relationships. However, most existing methods to produce survival trees rely on heuristic (or greedy) algorithms, which risk producing sub-optimal models. We present a dynamic-programming-with-bounds approach that finds provably-optimal sparse survival tree models, frequently in only a few seconds.

Title: L-AutoDA: Leveraging Large Language Models for Automated Decision-based Adversarial Attacks

• Authors: Ping Guo, Fei Liu, Xi Lin, Qingchuan Zhao, Qingfu Zhang
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15335
• Pdf URL: https://arxiv.org/pdf/2401.15335
• Copy Paste: [[2401.15335]] L-AutoDA: Leveraging Large Language Models for Automated Decision-based Adversarial Attacks(https://arxiv.org/abs/2401.15335)
• Keywords: security, attack, robust, generative, large language model
• Abstract: In the rapidly evolving field of machine learning, adversarial attacks present a significant challenge to model robustness and security. Decision-based attacks, which only require feedback on the decision of a model rather than detailed probabilities or scores, are particularly insidious and difficult to defend against. This work introduces L-AutoDA (Large Language Model-based Automated Decision-based Adversarial Attacks), a novel approach leveraging the generative capabilities of Large Language Models (LLMs) to automate the design of these attacks. By iteratively interacting with LLMs in an evolutionary framework, L-AutoDA automatically designs competitive attack algorithms efficiently without much human effort. We demonstrate the efficacy of L-AutoDA on CIFAR-10 dataset, showing significant improvements over baseline methods in both success rate and computational efficiency. Our findings underscore the potential of language models as tools for adversarial attack generation and highlight new avenues for the development of robust AI systems.

Title: A Comprehensive Survey of Compression Algorithms for Language Models

• Authors: Seungcheol Park, Jaehyeon Choi, Sojin Lee, U Kang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15347
• Pdf URL: https://arxiv.org/pdf/2401.15347
• Copy Paste: [[2401.15347]] A Comprehensive Survey of Compression Algorithms for Language Models(https://arxiv.org/abs/2401.15347)
• Keywords: large language model
• Abstract: How can we compress language models without sacrificing accuracy? The number of compression algorithms for language models is rapidly growing to benefit from remarkable advances of recent language models without side effects due to the gigantic size of language models, such as increased carbon emissions and expensive maintenance fees. While numerous compression algorithms have shown remarkable progress in compressing language models, it ironically becomes challenging to capture emerging trends and identify the fundamental concepts underlying them due to the excessive number of algorithms. In this paper, we survey and summarize diverse compression algorithms including pruning, quantization, knowledge distillation, low-rank approximation, parameter sharing, and efficient architecture design. We not only summarize the overall trend of diverse compression algorithms but also select representative algorithms and provide in-depth analyses of them. We discuss the value of each category of compression algorithms, and the desired properties of low-cost compression algorithms which have a significant impact due to the emergence of large language models. Finally, we introduce promising future research topics based on our survey results.

Title: Transformer-based Clipped Contrastive Quantization Learning for Unsupervised Image Retrieval

• Authors: Ayush Dubey, Shiv Ram Dubey, Satish Kumar Singh, Wei-Ta Chu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15362
• Pdf URL: https://arxiv.org/pdf/2401.15362
• Copy Paste: [[2401.15362]] Transformer-based Clipped Contrastive Quantization Learning for Unsupervised Image Retrieval(https://arxiv.org/abs/2401.15362)
• Keywords: transformer
• Abstract: Unsupervised image retrieval aims to learn the important visual characteristics without any given level to retrieve the similar images for a given query image. The Convolutional Neural Network (CNN)-based approaches have been extensively exploited with self-supervised contrastive learning for image hashing. However, the existing approaches suffer due to lack of effective utilization of global features by CNNs and biased-ness created by false negative pairs in the contrastive learning. In this paper, we propose a TransClippedCLR model by encoding the global context of an image using Transformer having local context through patch based processing, by generating the hash codes through product quantization and by avoiding the potential false negative pairs through clipped contrastive learning. The proposed model is tested with superior performance for unsupervised image retrieval on benchmark datasets, including CIFAR10, NUS-Wide and Flickr25K, as compared to the recent state-of-the-art deep models. The results using the proposed clipped contrastive learning are greatly improved on all datasets as compared to same backbone network with vanilla contrastive learning.

Title: Face to Cartoon Incremental Super-Resolution using Knowledge Distillation

• Authors: Trinetra Devkatte, Shiv Ram Dubey, Satish Kumar Singh, Abdenour Hadid
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.15366
• Pdf URL: https://arxiv.org/pdf/2401.15366
• Copy Paste: [[2401.15366]] Face to Cartoon Incremental Super-Resolution using Knowledge Distillation(https://arxiv.org/abs/2401.15366)
• Keywords: generative
• Abstract: Facial super-resolution/hallucination is an important area of research that seeks to enhance low-resolution facial images for a variety of applications. While Generative Adversarial Networks (GANs) have shown promise in this area, their ability to adapt to new, unseen data remains a challenge. This paper addresses this problem by proposing an incremental super-resolution using GANs with knowledge distillation (ISR-KD) for face to cartoon. Previous research in this area has not investigated incremental learning, which is critical for real-world applications where new data is continually being generated. The proposed ISR-KD aims to develop a novel unified framework for facial super-resolution that can handle different settings, including different types of faces such as cartoon face and various levels of detail. To achieve this, a GAN-based super-resolution network was pre-trained on the CelebA dataset and then incrementally trained on the iCartoonFace dataset, using knowledge distillation to retain performance on the CelebA test set while improving the performance on iCartoonFace test set. Our experiments demonstrate the effectiveness of knowledge distillation in incrementally adding capability to the model for cartoon face super-resolution while retaining the learned knowledge for facial hallucination tasks in GANs.

Title: Towards Event Extraction from Speech with Contextual Clues

• Authors: Jingqi Kang, Tongtong Wu, Jinming Zhao, Guitao Wang, Guilin Qi, Yuan-Fang Li, Gholamreza Haffari
• Subjects: cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2401.15385
• Pdf URL: https://arxiv.org/pdf/2401.15385
• Copy Paste: [[2401.15385]] Towards Event Extraction from Speech with Contextual Clues(https://arxiv.org/abs/2401.15385)
• Keywords: extraction
• Abstract: While text-based event extraction has been an active research area and has seen successful application in many domains, extracting semantic events from speech directly is an under-explored problem. In this paper, we introduce the Speech Event Extraction (SpeechEE) task and construct three synthetic training sets and one human-spoken test set. Compared to event extraction from text, SpeechEE poses greater challenges mainly due to complex speech signals that are continuous and have no word boundaries. Additionally, unlike perceptible sound events, semantic events are more subtle and require a deeper understanding. To tackle these challenges, we introduce a sequence-to-structure generation paradigm that can produce events from speech signals in an end-to-end manner, together with a conditioned generation method that utilizes speech recognition transcripts as the contextual clue. We further propose to represent events with a flat format to make outputs more natural language-like. Our experimental results show that our method brings significant improvements on all datasets, achieving a maximum F1 gain of 10.7%. The code and datasets are released on https://github.com/jodie-kang/SpeechEE.

Title: MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

• Authors: Yixuan Tang, Yi Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15391
• Pdf URL: https://arxiv.org/pdf/2401.15391
• Copy Paste: [[2401.15391]] MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries(https://arxiv.org/abs/2401.15391)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge, showing promising potential in mitigating LLM hallucinations and enhancing response quality, thereby facilitating the great adoption of LLMs in practice. However, we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. Furthermore, to our knowledge, no existing RAG benchmarking dataset focuses on multi-hop queries. In this paper, we develop a novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence. We detail the procedure of building the dataset, utilizing an English news article dataset as the underlying RAG knowledge base. We demonstrate the benchmarking utility of MultiHop-RAG in two experiments. The first experiment compares different embedding models for retrieving evidence for multi-hop queries. In the second experiment, we examine the capabilities of various state-of-the-art LLMs, including GPT-4, PaLM, and Llama2-70B, in reasoning and answering multi-hop queries given the evidence. Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries. We hope MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice. The MultiHop-RAG and implemented RAG system is publicly available at https://github.com/yixuantt/MultiHop-RAG/.

Title: Semantics of Multiword Expressions in Transformer-Based Models: A Survey

• Authors: Filip Miletić, Sabine Schulte im Walde
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15393
• Pdf URL: https://arxiv.org/pdf/2401.15393
• Copy Paste: [[2401.15393]] Semantics of Multiword Expressions in Transformer-Based Models: A Survey(https://arxiv.org/abs/2401.15393)
• Keywords: robust, transformer
• Abstract: Multiword expressions (MWEs) are composed of multiple words and exhibit variable degrees of compositionality. As such, their meanings are notoriously difficult to model, and it is unclear to what extent this issue affects transformer architectures. Addressing this gap, we provide the first in-depth survey of MWE processing with transformer models. We overall find that they capture MWE semantics inconsistently, as shown by reliance on surface patterns and memorized information. MWE meaning is also strongly localized, predominantly in early layers of the architecture. Representations benefit from specific linguistic properties, such as lower semantic idiosyncrasy and ambiguity of target expressions. Our findings overall question the ability of transformer models to robustly capture fine-grained semantics. Furthermore, we highlight the need for more directly comparable evaluation setups.

Title: A Survey on Data Augmentation in Large Model Era

• Authors: Yue Zhou, Chenlu Guo, Xu Wang, Yi Chang, Yuan Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15422
• Pdf URL: https://arxiv.org/pdf/2401.15422
• Copy Paste: [[2401.15422]] A Survey on Data Augmentation in Large Model Era(https://arxiv.org/abs/2401.15422)
• Keywords: diffusion
• Abstract: Large models, encompassing large language and diffusion models, have shown exceptional promise in approximating human-level intelligence, garnering significant interest from both academic and industrial spheres. However, the training of these large models necessitates vast quantities of high-quality data, and with continuous updates to these models, the existing reservoir of high-quality data may soon be depleted. This challenge has catalyzed a surge in research focused on data augmentation methods. Leveraging large models, these data augmentation techniques have outperformed traditional approaches. This paper offers an exhaustive review of large model-driven data augmentation methods, adopting a comprehensive perspective. We begin by establishing a classification of relevant studies into three main categories: image augmentation, text augmentation, and paired data augmentation. Following this, we delve into various data post-processing techniques pertinent to large model-based data augmentation. Our discussion then expands to encompass the array of applications for these data augmentation methods within natural language processing, computer vision, and audio signal processing. We proceed to evaluate the successes and limitations of large model-based data augmentation across different scenarios. Concluding our review, we highlight prospective challenges and avenues for future exploration in the field of data augmentation. Our objective is to furnish researchers with critical insights, ultimately contributing to the advancement of more sophisticated large models. We consistently maintain the related open-source materials at: https://github.com/MLGroup-JLU/LLM-data-aug-survey.

Title: Pre-training and Diagnosing Knowledge Base Completion Models

• Authors: Vid Kocijan, Myeongjun Erik Jang, Thomas Lukasiewicz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15439
• Pdf URL: https://arxiv.org/pdf/2401.15439
• Copy Paste: [[2401.15439]] Pre-training and Diagnosing Knowledge Base Completion Models(https://arxiv.org/abs/2401.15439)
• Keywords: robust
• Abstract: In this work, we introduce and analyze an approach to knowledge transfer from one collection of facts to another without the need for entity or relation matching. The method works for both canonicalized knowledge bases and uncanonicalized or open knowledge bases, i.e., knowledge bases where more than one copy of a real-world entity or relation may exist. The main contribution is a method that can make use of large-scale pre-training on facts, which were collected from unstructured text, to improve predictions on structured data from a specific domain. The introduced method is most impactful on small datasets such as ReVerb20k, where a 6% absolute increase of mean reciprocal rank and 65% relative decrease of mean rank over the previously best method was achieved, despite not relying on large pre-trained models like Bert. To understand the obtained pre-trained models better, we then introduce a novel dataset for the analysis of pre-trained models for Open Knowledge Base Completion, called Doge (Diagnostics of Open knowledge Graph Embeddings). It consists of 6 subsets and is designed to measure multiple properties of a pre-trained model: robustness against synonyms, ability to perform deductive reasoning, presence of gender stereotypes, consistency with reverse relations, and coverage of different areas of general knowledge. Using the introduced dataset, we show that the existing OKBC models lack consistency in the presence of synonyms and inverse relations and are unable to perform deductive reasoning. Moreover, their predictions often align with gender stereotypes, which persist even when presented with counterevidence. We additionally investigate the role of pre-trained word embeddings and demonstrate that avoiding biased word embeddings is not a sufficient measure to prevent biased behavior of OKBC models.

Title: Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation

• Authors: Yuxin Liang, Zhuoyang Song, Hao Wang, Jiaxing Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15449
• Pdf URL: https://arxiv.org/pdf/2401.15449
• Copy Paste: [[2401.15449]] Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation(https://arxiv.org/abs/2401.15449)
• Keywords: robust, large language model
• Abstract: We evaluate the ability of Large Language Models (LLMs) to discern and express their internal knowledge state, a key factor in countering factual hallucination and ensuring reliable application of LLMs. We observe a robust self-awareness of internal knowledge state in LLMs, evidenced by over 85% accuracy in knowledge probing. However, LLMs often fail to express their internal knowledge during generation, leading to factual hallucinations. We develop an automated hallucination annotation tool, Dreamcatcher, which merges knowledge probing and consistency checking methods to rank factual preference data. Using knowledge preference as reward, We propose a Reinforcement Learning from Knowledge Feedback (RLKF) training framework, leveraging reinforcement learning to enhance the factuality and honesty of LLMs. Our experiments across multiple models show that RLKF training effectively enhances the ability of models to utilize their internal knowledge state, boosting performance in a variety of knowledge-based and honesty-related tasks.

Title: A New Method for Vehicle Logo Recognition Based on Swin Transformer

• Authors: Yang Li, Doudou Zhang, Jianli Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15458
• Pdf URL: https://arxiv.org/pdf/2401.15458
• Copy Paste: [[2401.15458]] A New Method for Vehicle Logo Recognition Based on Swin Transformer(https://arxiv.org/abs/2401.15458)
• Keywords: transformer
• Abstract: Intelligent Transportation Systems (ITS) utilize sensors, cameras, and big data analysis to monitor real-time traffic conditions, aiming to improve traffic efficiency and safety. Accurate vehicle recognition is crucial in this process, and Vehicle Logo Recognition (VLR) stands as a key method. VLR enables effective management and monitoring by distinguishing vehicles on the road. Convolutional Neural Networks (CNNs) have made impressive strides in VLR research. However, achieving higher performance demands significant time and computational resources for training. Recently, the rise of Transformer models has brought new opportunities to VLR. Swin Transformer, with its efficient computation and global feature modeling capabilities, outperforms CNNs under challenging conditions. In this paper, we implement real-time VLR using Swin Transformer and fine-tune it for optimal performance. Extensive experiments conducted on three public vehicle logo datasets (HFUT-VL1, XMU, CTGU-VLD) demonstrate impressive top accuracy results of 99.28%, 100%, and 99.17%, respectively. Additionally, the use of a transfer learning strategy enables our method to be on par with state-of-the-art VLR methods. These findings affirm the superiority of our approach over existing methods. Future research can explore and optimize the application of the Swin Transformer in other vehicle vision recognition tasks to drive advancements in ITS.

Title: DataFrame QA: A Universal LLM Framework on DataFrame Question Answering Without Data Exposure

• Authors: Junyi Ye, Mengnan Du, Guiling Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15463
• Pdf URL: https://arxiv.org/pdf/2401.15463
• Copy Paste: [[2401.15463]] DataFrame QA: A Universal LLM Framework on DataFrame Question Answering Without Data Exposure(https://arxiv.org/abs/2401.15463)
• Keywords: secure, privacy, large language model
• Abstract: This paper introduces DataFrame question answering (QA), a novel task that utilizes large language models (LLMs) to generate Pandas queries for information retrieval and data analysis on dataframes, emphasizing safe and non-revealing data handling. Our method, which solely relies on dataframe column names, not only ensures data privacy but also significantly reduces the context window in the prompt, streamlining information processing and addressing major challenges in LLM-based data analysis. We propose DataFrame QA as a comprehensive framework that includes safe Pandas query generation and code execution. Various LLMs, notably GPT-4, are evaluated using the pass@1 metric on the renowned WikiSQL and our newly developed 'UCI-DataFrameQA', tailored for complex data analysis queries. Our findings indicate that GPT-4 achieves pass@1 rates of 86% on WikiSQL and 97% on UCI-DataFrameQA, underscoring its capability in securely retrieving and aggregating dataframe values and conducting sophisticated data analyses. This approach, deployable in a zero-shot manner without prior training or adjustments, proves to be highly adaptable and secure for diverse applications.

Title: Wind speed super-resolution and validation: from ERA5 to CERRA via diffusion models

• Authors: Fabio Merizzi, Andrea Asperti, Stefano Colamonaco
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15469
• Pdf URL: https://arxiv.org/pdf/2401.15469
• Copy Paste: [[2401.15469]] Wind speed super-resolution and validation: from ERA5 to CERRA via diffusion models(https://arxiv.org/abs/2401.15469)
• Keywords: diffusion
• Abstract: The Copernicus Regional Reanalysis for Europe, CERRA, is a high-resolution regional reanalysis dataset for the European domain. In recent years it has shown significant utility across various climate-related tasks, ranging from forecasting and climate change research to renewable energy prediction, resource management, air quality risk assessment, and the forecasting of rare events, among others. Unfortunately, the availability of CERRA is lagging two years behind the current date, due to constraints in acquiring the requisite external data and the intensive computational demands inherent in its generation. As a solution, this paper introduces a novel method using diffusion models to approximate CERRA downscaling in a data-driven manner, without additional informations. By leveraging the lower resolution ERA5 dataset, which provides boundary conditions for CERRA, we approach this as a super-resolution task. Focusing on wind speed around Italy, our model, trained on existing CERRA data, shows promising results, closely mirroring original CERRA data. Validation with in-situ observations further confirms the model's accuracy in approximating ground measurements.

Title: ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI

• Authors: Sarah E. Finch, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15471
• Pdf URL: https://arxiv.org/pdf/2401.15471
• Copy Paste: [[2401.15471]] ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI(https://arxiv.org/abs/2401.15471)
• Keywords: generative
• Abstract: Mastering commonsense understanding and reasoning is a pivotal skill essential for conducting engaging conversations. While there have been several attempts to create datasets that facilitate commonsense inferences in dialogue contexts, existing datasets tend to lack in-depth details, restate information already present in the conversation, and often fail to capture the multifaceted nature of commonsense reasoning. In response to these limitations, we compile a new synthetic dataset for commonsense reasoning in dialogue contexts using GPT, ConvoSense, that boasts greater contextual novelty, offers a higher volume of inferences per example, and substantially enriches the detail conveyed by the inferences. Our dataset contains over 500,000 inferences across 12,000 dialogues with 10 popular inference types, which empowers the training of generative commonsense models for dialogue that are superior in producing plausible inferences with high novelty when compared to models trained on the previous datasets. To the best of our knowledge, ConvoSense is the first of its kind to provide such a multitude of novel inferences at such a large scale.

Title: Temporal evolution in synthetic handwriting

• Authors: Cristina Carmona-Duarte, Miguel A. Ferrer, Antonio Parziale, Angelo Marcelli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15472
• Pdf URL: https://arxiv.org/pdf/2401.15472
• Copy Paste: [[2401.15472]] Temporal evolution in synthetic handwriting(https://arxiv.org/abs/2401.15472)
• Keywords: biometric
• Abstract: New methods for generating synthetic handwriting images for biometric applications have recently been developed. The temporal evolution of handwriting from childhood to adulthood is usually left unexplored in these works. This paper proposes a novel methodology for including temporal evolution in a handwriting synthesizer by means of simplifying the text trajectory plan and handwriting dynamics. This is achieved through a tailored version of the kinematic theory of rapid human movements and the neuromotor inspired handwriting synthesizer. The realism of the proposed method has been evaluated by comparing the temporal evolution of real and synthetic samples both quantitatively and subjectively. The quantitative test is based on a visual perception algorithm that compares the letter variability and the number of strokes in the real and synthetic handwriting produced at different ages. In the subjective test, 30 people are asked to evaluate the perceived realism of the evolution of the synthetic handwriting.

Title: iDeLog: Iterative Dual Spatial and Kinematic Extraction of Sigma-Lognormal Parameters

• Authors: Miguel A. Ferrer, Moises Diaz, Cristina Carmona-Duarte, Rejean Plamondon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15473
• Pdf URL: https://arxiv.org/pdf/2401.15473
• Copy Paste: [[2401.15473]] iDeLog: Iterative Dual Spatial and Kinematic Extraction of Sigma-Lognormal Parameters(https://arxiv.org/abs/2401.15473)
• Keywords: extraction
• Abstract: The Kinematic Theory of rapid movements and its associated Sigma-Lognormal model have been extensively used in a large variety of applications. While the physical and biological meaning of the model have been widely tested and validated for rapid movements, some shortcomings have been detected when it is used with continuous long and complex movements. To alleviate such drawbacks, and inspired by the motor equivalence theory and a conceivable visual feedback, this paper proposes a novel framework to extract the Sigma-Lognormal parameters, namely iDeLog. Specifically, iDeLog consists of two steps. The first one, influenced by the motor equivalence model, separately derives an initial action plan defined by a set of virtual points and angles from the trajectory and a sequence of lognormals from the velocity. In the second step, based on a hypothetical visual feedback compatible with an open-loop motor control, the virtual target points of the action plan are iteratively moved to improve the matching between the observed and reconstructed trajectory and velocity. During experiments conducted with handwritten signatures, iDeLog obtained promising results as compared to the previous development of the Sigma-Lognormal.

Title: To Burst or Not to Burst: Generating and Quantifying Improbable Text

• Authors: Kuleen Sasse, Samuel Barham, Efsun Sarioglu Kayi, Edward W. Staley
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15476
• Pdf URL: https://arxiv.org/pdf/2401.15476
• Copy Paste: [[2401.15476]] To Burst or Not to Burst: Generating and Quantifying Improbable Text(https://arxiv.org/abs/2401.15476)
• Keywords: large language model
• Abstract: While large language models (LLMs) are extremely capable at text generation, their outputs are still distinguishable from human-authored text. We explore this separation across many metrics over text, many sampling techniques, many types of text data, and across two popular LLMs, LLaMA and Vicuna. Along the way, we introduce a new metric, recoverability, to highlight differences between human and machine text; and we propose a new sampling technique, burst sampling, designed to close this gap. We find that LLaMA and Vicuna have distinct distributions under many of the metrics, and that this influences our results: Recoverability separates real from fake text better than any other metric when using LLaMA. When using Vicuna, burst sampling produces text which is distributionally closer to real text compared to other sampling techniques.

Title: Social Interpretable Reinforcement Learning

• Authors: Leonardo Lucio Custode, Giovanni Iacca
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2401.15480
• Pdf URL: https://arxiv.org/pdf/2401.15480
• Copy Paste: [[2401.15480]] Social Interpretable Reinforcement Learning(https://arxiv.org/abs/2401.15480)
• Keywords: interpretability
• Abstract: Reinforcement Learning (RL) bears the promise of being an enabling technology for many applications. However, since most of the literature in the field is currently focused on opaque models, the use of RL in high-stakes scenarios, where interpretability is crucial, is still limited. Recently, some approaches to interpretable RL, e.g., based on Decision Trees, have been proposed, but one of the main limitations of these techniques is their training cost. To overcome this limitation, we propose a new population-based method, called Social Interpretable RL (SIRL), inspired by social learning principles, to improve learning efficiency. Our method mimics a social learning process, where each agent in a group learns to solve a given task based both on its own individual experience as well as the experience acquired together with its peers. Our approach is divided into two phases. In the \emph{collaborative phase}, all the agents in the population interact with a shared instance of the environment, where each agent observes the state and independently proposes an action. Then, voting is performed to choose the action that will actually be performed in the environment. In the \emph{individual phase}, each agent refines its individual performance by interacting with its own instance of the environment. This mechanism makes the agents experience a larger number of episodes while simultaneously reducing the computational cost of the process. Our results on six well-known benchmarks show that SIRL reaches state-of-the-art performance w.r.t. the alternative interpretable methods from the literature.

Title: Unsupervised Solution Operator Learning for Mean-Field Games via Sampling-Invariant Parametrizations

• Authors: Han Huang, Rongjie Lai
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2401.15482
• Pdf URL: https://arxiv.org/pdf/2401.15482
• Copy Paste: [[2401.15482]] Unsupervised Solution Operator Learning for Mean-Field Games via Sampling-Invariant Parametrizations(https://arxiv.org/abs/2401.15482)
• Keywords: robust
• Abstract: Recent advances in deep learning has witnessed many innovative frameworks that solve high dimensional mean-field games (MFG) accurately and efficiently. These methods, however, are restricted to solving single-instance MFG and demands extensive computational time per instance, limiting practicality. To overcome this, we develop a novel framework to learn the MFG solution operator. Our model takes a MFG instances as input and output their solutions with one forward pass. To ensure the proposed parametrization is well-suited for operator learning, we introduce and prove the notion of sampling invariance for our model, establishing its convergence to a continuous operator in the sampling limit. Our method features two key advantages. First, it is discretization-free, making it particularly suitable for learning operators of high-dimensional MFGs. Secondly, it can be trained without the need for access to supervised labels, significantly reducing the computational overhead associated with creating training datasets in existing operator learning methods. We test our framework on synthetic and realistic datasets with varying complexity and dimensionality to substantiate its robustness.

Title: Baichuan2-Sum: Instruction Finetune Baichuan2-7B Model for Dialogue Summarization

• Authors: Jianfei Xiao, Yancan Chen, Yimin Ou, Hanyi Yu, Yiyong Xiao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15496
• Pdf URL: https://arxiv.org/pdf/2401.15496
• Copy Paste: [[2401.15496]] Baichuan2-Sum: Instruction Finetune Baichuan2-7B Model for Dialogue Summarization(https://arxiv.org/abs/2401.15496)
• Keywords: large language model
• Abstract: Large language models (LLMs) like Llama, Baichuan and Bloom models show remarkable ability with instruction fine-tuning in many natural language tasks. Nevertheless, for the dialogue summarization task, which aims to generate summaries for different roles in dialogue, most of the state-of-the-art methods conduct on small models (e.g Bart and Bert). Existing methods try to add task specified optimization on small models like adding global-local centrality score to models. In this paper, we propose an instruction fine-tuning model: Baichuan2-Sum, for role-oriented diaglouge summarization. By setting different instructions for different roles, the model can learn from the dialogue interactions and output the expected summaries. Furthermore, we applied NEFTune technique to add suitable noise during training to improve the results. The experiments demonstrate that the proposed model achieves the new state-of-the-art results on two public dialogue summarization datasets: CSDS and SAMSUM. We release our model and related codes to facilitate future studies on dialogue summarization task.

Title: Do We Need Language-Specific Fact-Checking Models? The Case of Chinese

• Authors: Caiqi Zhang, Zhijiang Guo, Andreas Vlachos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15498
• Pdf URL: https://arxiv.org/pdf/2401.15498
• Copy Paste: [[2401.15498]] Do We Need Language-Specific Fact-Checking Models? The Case of Chinese(https://arxiv.org/abs/2401.15498)
• Keywords: large language model
• Abstract: This paper investigates the potential benefits of language-specific fact-checking models, focusing on the case of Chinese. We demonstrate the limitations of methods such as translating Chinese claims and evidence into English or directly using multilingual large language models (e.g. GPT4), highlighting the need for language-specific systems. We further develop a state-of-the-art Chinese fact-checking system that, in contrast to previous approaches which treat evidence selection as a pairwise sentence classification task, considers the context of sentences. We also create an adversarial dataset to identify biases in our model, and while they are present as in English language datasets and models, they are often specific to the Chinese culture. Our study emphasizes the importance of language-specific fact-checking models to effectively combat misinformation.

Title: FloodLense: A Framework for ChatGPT-based Real-time Flood Detection

• Authors: Pranath Reddy Kumbam, Kshitij Maruti Vejre
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15501
• Pdf URL: https://arxiv.org/pdf/2401.15501
• Copy Paste: [[2401.15501]] FloodLense: A Framework for ChatGPT-based Real-time Flood Detection(https://arxiv.org/abs/2401.15501)
• Keywords: large language model
• Abstract: This study addresses the vital issue of real-time flood detection and management. It innovatively combines advanced deep learning models with Large language models (LLM), enhancing flood monitoring and response capabilities. This approach addresses the limitations of current methods by offering a more accurate, versatile, user-friendly and accessible solution. The integration of UNet, RDN, and ViT models with natural language processing significantly improves flood area detection in diverse environments, including using aerial and satellite imagery. The experimental evaluation demonstrates the models' efficacy in accurately identifying and mapping flood zones, showcasing the project's potential in transforming environmental monitoring and disaster management fields.

Title: Style-News: Incorporating Stylized News Generation and Adversarial Verification for Neural Fake News Detection

• Authors: Wei-Yao Wang, Yu-Chieh Chang, Wen-Chih Peng
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.15509
• Pdf URL: https://arxiv.org/pdf/2401.15509
• Copy Paste: [[2401.15509]] Style-News: Incorporating Stylized News Generation and Adversarial Verification for Neural Fake News Detection(https://arxiv.org/abs/2401.15509)
• Keywords: attack, generative
• Abstract: With the improvements in generative models, the issues of producing hallucinations in various domains (e.g., law, writing) have been brought to people's attention due to concerns about misinformation. In this paper, we focus on neural fake news, which refers to content generated by neural networks aiming to mimic the style of real news to deceive people. To prevent harmful disinformation spreading fallaciously from malicious social media (e.g., content farms), we propose a novel verification framework, Style-News, using publisher metadata to imply a publisher's template with the corresponding text types, political stance, and credibility. Based on threat modeling aspects, a style-aware neural news generator is introduced as an adversary for generating news content conditioning for a specific publisher, and style and source discriminators are trained to defend against this attack by identifying which publisher the style corresponds with, and discriminating whether the source of the given news is human-written or machine-generated. To evaluate the quality of the generated content, we integrate various dimensional metrics (language fluency, content preservation, and style adherence) and demonstrate that Style-News significantly outperforms the previous approaches by a margin of 0.35 for fluency, 15.24 for content, and 0.38 for style at most. Moreover, our discriminative model outperforms state-of-the-art baselines in terms of publisher prediction (up to 4.64%) and neural fake news detection (+6.94% $\sim$ 31.72%).

Title: A Thorough Study of State Leakage Mitigation in Quantum Computing with One-Time Pad

• Authors: Chuanqi Xu, Jamie Sikora, Jakub Szefer
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2401.15529
• Pdf URL: https://arxiv.org/pdf/2401.15529
• Copy Paste: [[2401.15529]] A Thorough Study of State Leakage Mitigation in Quantum Computing with One-Time Pad(https://arxiv.org/abs/2401.15529)
• Keywords: secure, security, privacy, protect
• Abstract: The ability for users to access quantum computers through the cloud has increased rapidly in recent years. Despite still being Noisy Intermediate-Scale Quantum (NISQ) machines, modern quantum computers are now being actively employed for research and by numerous startups. Quantum algorithms typically produce probabilistic results, necessitating repeated execution to produce the desired outcomes. In order for the execution to begin from the specified ground state each time and for the results of the prior execution not to interfere with the results of the subsequent execution, the reset mechanism must be performed between each iteration to effectively reset the qubits. However, due to noise and errors in quantum computers and specifically these reset mechanisms, a noisy reset operation may lead to systematic errors in the overall computation, as well as potential security and privacy vulnerabilities of information leakage. To counter this issue, we thoroughly examine the state leakage problem in quantum computing, and then propose a solution by employing the classical and quantum one-time pads before the reset mechanism to prevent the state leakage, which works by randomly applying simple gates for each execution of the circuit. In addition, this work explores conditions under which the classical one-time pad, which uses fewer resources, is sufficient to protect state leakage. Finally, we study the role of various errors in state leakage, by evaluating the degrees of leakage under different error levels of gate, measurement, and sampling errors. Our findings offer new perspectives on the design of reset mechanisms and secure quantum computing systems.

Title: An Information-Theoretic Analysis of In-Context Learning

• Authors: Hong Jun Jeon, Jason D. Lee, Qi Lei, Benjamin Van Roy
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2401.15530
• Pdf URL: https://arxiv.org/pdf/2401.15530
• Copy Paste: [[2401.15530]] An Information-Theoretic Analysis of In-Context Learning(https://arxiv.org/abs/2401.15530)
• Keywords: transformer
• Abstract: Previous theoretical results pertaining to meta-learning on sequences build on contrived assumptions and are somewhat convoluted. We introduce new information-theoretic tools that lead to an elegant and very general decomposition of error into three components: irreducible error, meta-learning error, and intra-task error. These tools unify analyses across many meta-learning challenges. To illustrate, we apply them to establish new results about in-context learning with transformers. Our theoretical results characterizes how error decays in both the number of training sequences and sequence lengths. Our results are very general; for example, they avoid contrived mixing time assumptions made by all prior results that establish decay of error with sequence length.

Title: BrepGen: A B-rep Generative Diffusion Model with Structured Latent Geometry

• Authors: Xiang Xu, Joseph G. Lambourne, Pradeep Kumar Jayaraman, Zhengqing Wang, Karl D.D. Willis, Yasutaka Furukawa
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15563
• Pdf URL: https://arxiv.org/pdf/2401.15563
• Copy Paste: [[2401.15563]] BrepGen: A B-rep Generative Diffusion Model with Structured Latent Geometry(https://arxiv.org/abs/2401.15563)
• Keywords: diffusion, transformer, generative
• Abstract: This paper presents BrepGen, a diffusion-based generative approach that directly outputs a Boundary representation (B-rep) Computer-Aided Design (CAD) model. BrepGen represents a B-rep model as a novel structured latent geometry in a hierarchical tree. With the root node representing a whole CAD solid, each element of a B-rep model (i.e., a face, an edge, or a vertex) progressively turns into a child-node from top to bottom. B-rep geometry information goes into the nodes as the global bounding box of each primitive along with a latent code describing the local geometric shape. The B-rep topology information is implicitly represented by node duplication. When two faces share an edge, the edge curve will appear twice in the tree, and a T-junction vertex with three incident edges appears six times in the tree with identical node features. Starting from the root and progressing to the leaf, BrepGen employs Transformer-based diffusion models to sequentially denoise node features while duplicated nodes are detected and merged, recovering the B-Rep topology information. Extensive experiments show that BrepGen sets a new milestone in CAD B-rep generation, surpassing existing methods on various benchmarks. Results on our newly collected furniture dataset further showcase its exceptional capability in generating complicated geometry. While previous methods were limited to generating simple prismatic shapes, BrepGen incorporates free-form and doubly-curved surfaces for the first time. Additional applications of BrepGen include CAD autocomplete and design interpolation. The code, pretrained models, and dataset will be released.

Title: Intriguing Equivalence Structures of the Embedding Space of Vision Transformers

• Authors: Shaeke Salman, Md Montasir Bin Shams, Xiuwen Liu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15568
• Pdf URL: https://arxiv.org/pdf/2401.15568
• Copy Paste: [[2401.15568]] Intriguing Equivalence Structures of the Embedding Space of Vision Transformers(https://arxiv.org/abs/2401.15568)
• Keywords: transformer
• Abstract: Pre-trained large foundation models play a central role in the recent surge of artificial intelligence, resulting in fine-tuned models with remarkable abilities when measured on benchmark datasets, standard exams, and applications. Due to their inherent complexity, these models are not well understood. While small adversarial inputs to such models are well known, the structures of the representation space are not well characterized despite their fundamental importance. In this paper, using the vision transformers as an example due to the continuous nature of their input space, we show via analyses and systematic experiments that the representation space consists of large piecewise linear subspaces where there exist very different inputs sharing the same representations, and at the same time, local normal spaces where there are visually indistinguishable inputs having very different representations. The empirical results are further verified using the local directional estimations of the Lipschitz constants of the underlying models. Consequently, the resulting representations change the results of downstream models, and such models are subject to overgeneralization and with limited semantically meaningful generalization capability.

Title: Efficient Tuning and Inference for Large Language Models on Textual Graphs

• Authors: Yun Zhu, Yaoke Wang, Haizhou Shi, Siliang Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15569
• Pdf URL: https://arxiv.org/pdf/2401.15569
• Copy Paste: [[2401.15569]] Efficient Tuning and Inference for Large Language Models on Textual Graphs(https://arxiv.org/abs/2401.15569)
• Keywords: large language model
• Abstract: Rich textual and topological information of textual graphs need to be modeled in real-world applications such as webpages, e-commerce, and academic articles. Practitioners have been long following the path of adopting a shallow text encoder and a subsequent graph neural network (GNN) to solve this problem. In light of recent advancements in large language models (LLMs), it is apparent that integrating LLMs for enhanced textual encoding can substantially improve the performance of textual graphs. Nevertheless, the efficiency of these methods poses a significant challenge. In this paper, we propose ENGINE, a parameter- and memory-efficient fine-tuning method for textual graphs with an LLM encoder. The key insight is to combine the LLMs and GNNs through a tunable side structure, which significantly reduces the training complexity without impairing the joint model's capacity. Extensive experiments on textual graphs demonstrate our method's effectiveness by achieving the best model performance, meanwhile having the lowest training cost compared to previous methods. Moreover, we introduce two variants with caching and dynamic early exit to further enhance training and inference speed. Specifically, caching accelerates ENGINE's training by 12x, and dynamic early exit achieves up to 5x faster inference with a negligible performance drop (at maximum 1.17% relevant drop across 7 datasets).

Title: ARCNet: An Asymmetric Residual Wavelet Column Correction Network for Infrared Image Destriping

• Authors: Shuai Yuan, Hanlin Qin, Xiang Yan, Naveed Akhtar, Shiqi Yang, Shuowen Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15578
• Pdf URL: https://arxiv.org/pdf/2401.15578
• Copy Paste: [[2401.15578]] ARCNet: An Asymmetric Residual Wavelet Column Correction Network for Infrared Image Destriping(https://arxiv.org/abs/2401.15578)
• Keywords: robust
• Abstract: Infrared image destriping seeks to restore high-quality content from degraded images. Recent works mainly address this task by leveraging prior knowledge to separate stripe noise from the degraded image. However, constructing a robust decoupling model for that purpose remains challenging, especially when significant similarities exist between the stripe noise and vertical background structure. Addressing that, we introduce Asymmetric Residual wavelet Column correction Network (ARCNet) for image destriping, aiming to consistently preserve spatially precise high-resolution representations. Our neural model leverages a novel downsampler, residual haar discrete wavelet transform (RHDWT), stripe directional prior knowledge and data-driven learning to induce a model with enriched feature representation of stripe noise and background. In our technique, the inverse wavelet transform is replaced by transposed convolution for feature upsampling, which can suppress noise crosstalk and encourage the network to focus on robust image reconstruction. After each sampling, a proposed column non-uniformity correction module (CNCM) is leveraged by our method to enhance column uniformity, spatial correlation, and global self-dependence between each layer component. CNCM can establish structural characteristics of stripe noise and utilize contextual information at long-range dependencies to distinguish stripes with varying intensities and distributions. Extensive experiments on synthetic data, real data, and infrared small target detection tasks show that the proposed method outperforms state-of-the-art single-image destriping methods both visually and quantitatively by a considerable margin. Our code will be made publicly available at \url{https://github.com/xdFai}.

Title: SCTransNet: Spatial-channel Cross Transformer Network for Infrared Small Target Detection

• Authors: Shuai Yuan, Hanlin Qin, Xiang Yan, Naveed AKhtar, Ajmal Mian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15583
• Pdf URL: https://arxiv.org/pdf/2401.15583
• Copy Paste: [[2401.15583]] SCTransNet: Spatial-channel Cross Transformer Network for Infrared Small Target Detection(https://arxiv.org/abs/2401.15583)
• Keywords: transformer
• Abstract: Infrared small target detection (IRSTD) has recently benefitted greatly from U-shaped neural models. However, largely overlooking effective global information modeling, existing techniques struggle when the target has high similarities with the background. We present a Spatial-channel Cross Transformer Network (SCTransNet) that leverages spatial-channel cross transformer blocks (SCTBs) on top of long-range skip connections to address the aforementioned challenge. In the proposed SCTBs, the outputs of all encoders are interacted with cross transformer to generate mixed features, which are redistributed to all decoders to effectively reinforce semantic differences between the target and clutter at full scales. Specifically, SCTB contains the following two key elements: (a) spatial-embedded single-head channel-cross attention (SSCA) for exchanging local spatial features and full-level global channel information to eliminate ambiguity among the encoders and facilitate high-level semantic associations of the images, and (b) a complementary feed-forward network (CFN) for enhancing the feature discriminability via a multi-scale strategy and cross-spatial-channel information interaction to promote beneficial information transfer. Our SCTransNet effectively encodes the semantic differences between targets and backgrounds to boost its internal representation for detecting small infrared targets accurately. Extensive experiments on three public datasets, NUDT-SIRST, NUAA-SIRST, and IRSTD-1k, demonstrate that the proposed SCTransNet outperforms existing IRSTD methods. Our code will be made public at https://github.com/xdFai.

Title: DGNN: Decoupled Graph Neural Networks with Structural Consistency between Attribute and Graph Embedding Representations

• Authors: Jinlu Wang, Jipeng Guo, Yanfeng Sun, Junbin Gao, Shaofan Wang, Yachao Yang, Baocai Yin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15584
• Pdf URL: https://arxiv.org/pdf/2401.15584
• Copy Paste: [[2401.15584]] DGNN: Decoupled Graph Neural Networks with Structural Consistency between Attribute and Graph Embedding Representations(https://arxiv.org/abs/2401.15584)
• Keywords: robust
• Abstract: Graph neural networks (GNNs) demonstrate a robust capability for representation learning on graphs with complex structures, showcasing superior performance in various applications. The majority of existing GNNs employ a graph convolution operation by using both attribute and structure information through coupled learning. In essence, GNNs, from an optimization perspective, seek to learn a consensus and compromise embedding representation that balances attribute and graph information, selectively exploring and retaining valid information. To obtain a more comprehensive embedding representation of nodes, a novel GNNs framework, dubbed Decoupled Graph Neural Networks (DGNN), is introduced. DGNN explores distinctive embedding representations from the attribute and graph spaces by decoupled terms. Considering that semantic graph, constructed from attribute feature space, consists of different node connection information and provides enhancement for the topological graph, both topological and semantic graphs are combined for the embedding representation learning. Further, structural consistency among attribute embedding and graph embeddings is promoted to effectively remove redundant information and establish soft connection. This involves promoting factor sharing for adjacency reconstruction matrices, facilitating the exploration of a consensus and high-level correlation. Finally, a more powerful and complete representation is achieved through the concatenation of these embeddings. Experimental results conducted on several graph benchmark datasets verify its superiority in node classification task.

Title: Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting

• Authors: Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15585
• Pdf URL: https://arxiv.org/pdf/2401.15585
• Copy Paste: [[2401.15585]] Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting(https://arxiv.org/abs/2401.15585)
• Keywords: fair, large language model
• Abstract: There exist both scalable tasks, like reading comprehension and fact-checking, where model performance improves with model size, and unscalable tasks, like arithmetic reasoning and symbolic reasoning, where model performance does not necessarily improve with model size. Large language models (LLMs) equipped with Chain-of-Thought (CoT) prompting are able to make accurate incremental predictions even on unscalable tasks. Unfortunately, despite their exceptional reasoning abilities, LLMs tend to internalize and reproduce discriminatory societal biases. Whether CoT can provide discriminatory or egalitarian rationalizations for the implicit information in unscalable tasks remains an open question. In this study, we examine the impact of LLMs' step-by-step predictions on gender bias in unscalable tasks. For this purpose, we construct a benchmark for an unscalable task where the LLM is given a list of words comprising feminine, masculine, and gendered occupational words, and is required to count the number of feminine and masculine words. In our CoT prompts, we require the LLM to explicitly indicate whether each word in the word list is a feminine or masculine before making the final predictions. With counting and handling the meaning of words, this benchmark has characteristics of both arithmetic reasoning and symbolic reasoning. Experimental results in English show that without step-by-step prediction, most LLMs make socially biased predictions, despite the task being as simple as counting words. Interestingly, CoT prompting reduces this unconscious social bias in LLMs and encourages fair predictions.

Title: Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization

• Authors: Yinbin Han, Meisam Razaviyayn, Renyuan Xu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.15604
• Pdf URL: https://arxiv.org/pdf/2401.15604
• Copy Paste: [[2401.15604]] Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization(https://arxiv.org/abs/2401.15604)
• Keywords: robust, diffusion
• Abstract: Diffusion models have emerged as a powerful tool rivaling GANs in generating high-quality samples with improved fidelity, flexibility, and robustness. A key component of these models is to learn the score function through score matching. Despite empirical success on various tasks, it remains unclear whether gradient-based algorithms can learn the score function with a provable accuracy. As a first step toward answering this question, this paper establishes a mathematical framework for analyzing score estimation using neural networks trained by gradient descent. Our analysis covers both the optimization and the generalization aspects of the learning procedure. In particular, we propose a parametric form to formulate the denoising score-matching problem as a regression with noisy labels. Compared to the standard supervised learning setup, the score-matching problem introduces distinct challenges, including unbounded input, vector-valued output, and an additional time variable, preventing existing techniques from being applied directly. In this paper, we show that with a properly designed neural network architecture, the score function can be accurately approximated by a reproducing kernel Hilbert space induced by neural tangent kernels. Furthermore, by applying an early-stopping rule for gradient descent and leveraging certain coupling arguments between neural network training and kernel regression, we establish the first generalization error (sample complexity) bounds for learning the score function despite the presence of noise in the observations. Our analysis is grounded in a novel parametric form of the neural network and an innovative connection between score matching and regression analysis, facilitating the application of advanced statistical and optimization techniques.

Title: Addressing Noise and Efficiency Issues in Graph-Based Machine Learning Models From the Perspective of Adversarial Attack

• Authors: Yongyu Wang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15615
• Pdf URL: https://arxiv.org/pdf/2401.15615
• Copy Paste: [[2401.15615]] Addressing Noise and Efficiency Issues in Graph-Based Machine Learning Models From the Perspective of Adversarial Attack(https://arxiv.org/abs/2401.15615)
• Keywords: attack, robust
• Abstract: Given that no existing graph construction method can generate a perfect graph for a given dataset, graph-based algorithms are invariably affected by the plethora of redundant and erroneous edges present within the constructed graphs. In this paper, we propose treating these noisy edges as adversarial attack and use a spectral adversarial robustness evaluation method to diminish the impact of noisy edges on the performance of graph algorithms. Our method identifies those points that are less vulnerable to noisy edges and leverages only these robust points to perform graph-based algorithms. Our experiments with spectral clustering, one of the most representative and widely utilized graph algorithms, reveal that our methodology not only substantially elevates the precision of the algorithm but also greatly accelerates its computational efficiency by leveraging only a select number of robust data points.

Title: Diffusion-based graph generative methods

• Authors: Hongyang Chen, Can Xu, Lingyu Zheng, Qiang Zhang, Xuemin Lin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15617
• Pdf URL: https://arxiv.org/pdf/2401.15617
• Copy Paste: [[2401.15617]] Diffusion-based graph generative methods(https://arxiv.org/abs/2401.15617)
• Keywords: diffusion, generative
• Abstract: Being the most cutting-edge generative methods, diffusion methods have shown great advances in wide generation tasks. Among them, graph generation attracts significant research attention for its broad application in real life. In our survey, we systematically and comprehensively review on diffusion-based graph generative methods. We first make a review on three mainstream paradigms of diffusion methods, which are denoising diffusion probabilistic models, score-based genrative models, and stochastic differential equations. Then we further categorize and introduce the latest applications of diffusion models on graphs. In the end, we point out some limitations of current studies and future directions of future explorations. The summary of existing methods metioned in this survey is in https://github.com/zhejiangzhuque/Diffusion-based-Graph-Generative-Methods.

Title: Generative AI-enabled Blockchain Networks: Fundamentals, Applications, and Case Study

• Authors: Cong T. Nguyen, Yinqiu Liu, Hongyang Du, Dinh Thai Hoang, Dusit Niyato, Diep N. Nguyen, Shiwen Mao
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15625
• Pdf URL: https://arxiv.org/pdf/2401.15625
• Copy Paste: [[2401.15625]] Generative AI-enabled Blockchain Networks: Fundamentals, Applications, and Case Study(https://arxiv.org/abs/2401.15625)
• Keywords: security, privacy, attack, diffusion, generative
• Abstract: Generative Artificial Intelligence (GAI) has recently emerged as a promising solution to address critical challenges of blockchain technology, including scalability, security, privacy, and interoperability. In this paper, we first introduce GAI techniques, outline their applications, and discuss existing solutions for integrating GAI into blockchains. Then, we discuss emerging solutions that demonstrate the effectiveness of GAI in addressing various challenges of blockchain, such as detecting unknown blockchain attacks and smart contract vulnerabilities, designing key secret sharing schemes, and enhancing privacy. Moreover, we present a case study to demonstrate that GAI, specifically the generative diffusion model, can be employed to optimize blockchain network performance metrics. Experimental results clearly show that, compared to a baseline traditional AI approach, the proposed generative diffusion model approach can converge faster, achieve higher rewards, and significantly improve the throughput and latency of the blockchain network. Additionally, we highlight future research directions for GAI in blockchain applications, including personalized GAI-enabled blockchains, GAI-blockchain synergy, and privacy and security considerations within blockchain ecosystems.

Title: TA&AT: Enhancing Task-Oriented Dialog with Turn-Level Auxiliary Tasks and Action-Tree Based Scheduled Sampling

• Authors: Longxiang Liu, Xiuxing Li, Yang Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15626
• Pdf URL: https://arxiv.org/pdf/2401.15626
• Copy Paste: [[2401.15626]] TA&AT: Enhancing Task-Oriented Dialog with Turn-Level Auxiliary Tasks and Action-Tree Based Scheduled Sampling(https://arxiv.org/abs/2401.15626)
• Keywords: robust
• Abstract: Task-oriented dialog systems have witnessed substantial progress due to conversational pre-training techniques. Yet, two significant challenges persist. First, most systems primarily utilize the latest turn's state label for the generator. This practice overlooks the comprehensive value of state labels in boosting the model's understanding for future generations. Second, an overreliance on generated policy often leads to error accumulation, resulting in suboptimal responses when adhering to incorrect actions. To combat these challenges, we propose turn-level multi-task objectives for the encoder. With the guidance of essential information from labeled intermediate states, we establish a more robust representation for both understanding and generation. For the decoder, we introduce an action tree-based scheduled sampling technique. Specifically, we model the hierarchical policy as trees and utilize the similarity between trees to sample negative policy based on scheduled sampling, hoping the model to generate invariant responses under perturbations. This method simulates potential pitfalls by sampling similar negative policy, bridging the gap between task-oriented dialog training and inference. Among methods without continual pre-training, our approach achieved state-of-the-art (SOTA) performance on the MultiWOZ dataset series and was also competitive with pre-trained SOTA methods.

Title: FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models

• Authors: Feihong He, Gang Li, Mengyuan Zhang, Leilei Yan, Lingyu Si, Fanzhang Li
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.15636
• Pdf URL: https://arxiv.org/pdf/2401.15636
• Copy Paste: [[2401.15636]] FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models(https://arxiv.org/abs/2401.15636)
• Keywords: diffusion, generative
• Abstract: The rapid development of generative diffusion models has significantly advanced the field of style transfer. However, most current style transfer methods based on diffusion models typically involve a slow iterative optimization process, e.g., model fine-tuning and textual inversion of style concept. In this paper, we introduce FreeStyle, an innovative style transfer method built upon a pre-trained large diffusion model, requiring no further optimization. Besides, our method enables style transfer only through a text description of the desired style, eliminating the necessity of style images. Specifically, we propose a dual-stream encoder and single-stream decoder architecture, replacing the conventional U-Net in diffusion models. In the dual-stream encoder, two distinct branches take the content image and style text prompt as inputs, achieving content and style decoupling. In the decoder, we further modulate features from the dual streams based on a given content image and the corresponding style text prompt for precise style transfer. Our experimental results demonstrate high-quality synthesis and fidelity of our method across various content images and style text prompts. The code and more results are available at our project website:https://freestylefreelunch.github.io/.

Title: Cyto R-CNN and CytoNuke Dataset: Towards reliable whole-cell segmentation in bright-field histological images

• Authors: Johannes Raufeisen, Kunpeng Xie, Fabian Hörst, Till Braunschweig, Jianning Li, Jens Kleesiek, Rainer Röhrig, Jan Egger, Bastian Leibe, Frank Hölzle, Alexander Hermans, Behrus Puladi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15638
• Pdf URL: https://arxiv.org/pdf/2401.15638
• Copy Paste: [[2401.15638]] Cyto R-CNN and CytoNuke Dataset: Towards reliable whole-cell segmentation in bright-field histological images(https://arxiv.org/abs/2401.15638)
• Keywords: segmentation
• Abstract: Background: Cell segmentation in bright-field histological slides is a crucial topic in medical image analysis. Having access to accurate segmentation allows researchers to examine the relationship between cellular morphology and clinical observations. Unfortunately, most segmentation methods known today are limited to nuclei and cannot segmentate the cytoplasm. Material & Methods: We present a new network architecture Cyto R-CNN that is able to accurately segment whole cells (with both the nucleus and the cytoplasm) in bright-field images. We also present a new dataset CytoNuke, consisting of multiple thousand manual annotations of head and neck squamous cell carcinoma cells. Utilizing this dataset, we compared the performance of Cyto R-CNN to other popular cell segmentation algorithms, including QuPath's built-in algorithm, StarDist and Cellpose. To evaluate segmentation performance, we calculated AP50, AP75 and measured 17 morphological and staining-related features for all detected cells. We compared these measurements to the gold standard of manual segmentation using the Kolmogorov-Smirnov test. Results: Cyto R-CNN achieved an AP50 of 58.65\% and an AP75 of 11.56\% in whole-cell segmentation, outperforming all other methods (QuPath $19.46/0.91\%$; StarDist $45.33/2.32\%$; Cellpose $31.85/5.61\%$). Cell features derived from Cyto R-CNN showed the best agreement to the gold standard ($\bar{D} = 0.15$) outperforming QuPath ($\bar{D} = 0.22$), StarDist ($\bar{D} = 0.25$) and Cellpose ($\bar{D} = 0.23$). Conclusion: Our newly proposed Cyto R-CNN architecture outperforms current algorithms in whole-cell segmentation while providing more reliable cell measurements than any other model. This could improve digital pathology workflows, potentially leading to improved diagnosis. Moreover, our published dataset can be used to develop further models in the future.

Title: Improving Data Augmentation for Robust Visual Question Answering with Effective Curriculum Learning

• Authors: Yuhang Zheng, Zhen Wang, Long Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15646
• Pdf URL: https://arxiv.org/pdf/2401.15646
• Copy Paste: [[2401.15646]] Improving Data Augmentation for Robust Visual Question Answering with Effective Curriculum Learning(https://arxiv.org/abs/2401.15646)
• Keywords: robust
• Abstract: Being widely used in learning unbiased visual question answering (VQA) models, Data Augmentation (DA) helps mitigate language biases by generating extra training samples beyond the original samples. While today's DA methods can generate robust samples, the augmented training set, significantly larger than the original dataset, often exhibits redundancy in terms of difficulty or content repetition, leading to inefficient model training and even compromising the model performance. To this end, we design an Effective Curriculum Learning strategy ECL to enhance DA-based VQA methods. Intuitively, ECL trains VQA models on relatively ``easy'' samples first, and then gradually changes to ``harder'' samples, and less-valuable samples are dynamically removed. Compared to training on the entire augmented dataset, our ECL strategy can further enhance VQA models' performance with fewer training samples. Extensive ablations have demonstrated the effectiveness of ECL on various methods.

Title: UP-CrackNet: Unsupervised Pixel-Wise Road Crack Detection via Adversarial Image Restoration

• Authors: Nachuan Ma, Rui Fan, Lihua Xie
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.15647
• Pdf URL: https://arxiv.org/pdf/2401.15647
• Copy Paste: [[2401.15647]] UP-CrackNet: Unsupervised Pixel-Wise Road Crack Detection via Adversarial Image Restoration(https://arxiv.org/abs/2401.15647)
• Keywords: generative, segmentation
• Abstract: Over the past decade, automated methods have been developed to detect cracks more efficiently, accurately, and objectively, with the ultimate goal of replacing conventional manual visual inspection techniques. Among these methods, semantic segmentation algorithms have demonstrated promising results in pixel-wise crack detection tasks. However, training such data-driven algorithms requires a large amount of human-annotated datasets with pixel-level annotations, which is a highly labor-intensive and time-consuming process. Moreover, supervised learning-based methods often struggle with poor generalization ability in unseen datasets. Therefore, we propose an unsupervised pixel-wise road crack detection network, known as UP-CrackNet. Our approach first generates multi-scale square masks and randomly selects them to corrupt undamaged road images by removing certain regions. Subsequently, a generative adversarial network is trained to restore the corrupted regions by leveraging the semantic context learned from surrounding uncorrupted regions. During the testing phase, an error map is generated by calculating the difference between the input and restored images, which allows for pixel-wise crack detection. Our comprehensive experimental results demonstrate that UP-CrackNet outperforms other general-purpose unsupervised anomaly detection algorithms, and exhibits comparable performance and superior generalizability when compared with state-of-the-art supervised crack segmentation algorithms. Our source code is publicly available at mias.group/UP-CrackNet.

Title: CPDM: Content-Preserving Diffusion Model for Underwater Image Enhancement

• Authors: Xiaowen Shi, Yuan-Gen Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15649
• Pdf URL: https://arxiv.org/pdf/2401.15649
• Copy Paste: [[2401.15649]] CPDM: Content-Preserving Diffusion Model for Underwater Image Enhancement(https://arxiv.org/abs/2401.15649)
• Keywords: diffusion
• Abstract: Underwater image enhancement (UIE) is challenging since image degradation in aquatic environments is complicated and changing over time. Existing mainstream methods rely on either physical-model or data-driven, suffering from performance bottlenecks due to changes in imaging conditions or training instability. In this article, we make the first attempt to adapt the diffusion model to the UIE task and propose a Content-Preserving Diffusion Model (CPDM) to address the above challenges. CPDM first leverages a diffusion model as its fundamental model for stable training and then designs a content-preserving framework to deal with changes in imaging conditions. Specifically, we construct a conditional input module by adopting both the raw image and the difference between the raw and noisy images as the input, which can enhance the model's adaptability by considering the changes involving the raw images in underwater environments. To preserve the essential content of the raw images, we construct a content compensation module for content-aware training by extracting low-level features from the raw images. Extensive experimental results validate the effectiveness of our CPDM, surpassing the state-of-the-art methods in terms of both subjective and objective metrics.

Title: Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach

• Authors: Shaofeng Zhang, Jinfa Huang, Qiang Zhou, Zhibin Wang, Fan Wang, Jiebo Luo, Junchi Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15652
• Pdf URL: https://arxiv.org/pdf/2401.15652
• Copy Paste: [[2401.15652]] Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach(https://arxiv.org/abs/2401.15652)
• Keywords: diffusion, generative
• Abstract: Image outpainting aims to generate the content of an input sub-image beyond its original boundaries. It is an important task in content generation yet remains an open problem for generative models. This paper pushes the technical frontier of image outpainting in two directions that have not been resolved in literature: 1) outpainting with arbitrary and continuous multiples (without restriction), and 2) outpainting in a single step (even for large expansion multiples). Moreover, we develop a method that does not depend on a pre-trained backbone network, which is in contrast commonly required by the previous SOTA outpainting methods. The arbitrary multiple outpainting is achieved by utilizing randomly cropped views from the same image during training to capture arbitrary relative positional information. Specifically, by feeding one view and positional embeddings as queries, we can reconstruct another view. At inference, we generate images with arbitrary expansion multiples by inputting an anchor image and its corresponding positional embeddings. The one-step outpainting ability here is particularly noteworthy in contrast to previous methods that need to be performed for $N$ times to obtain a final multiple which is $N$ times of its basic and fixed multiple. We evaluate the proposed approach (called PQDiff as we adopt a diffusion-based generator as our embodiment, under our proposed \textbf{P}ositional \textbf{Q}uery scheme) on public benchmarks, demonstrating its superior performance over state-of-the-art approaches. Specifically, PQDiff achieves state-of-the-art FID scores on the Scenery (\textbf{21.512}), Building Facades (\textbf{25.310}), and WikiArts (\textbf{36.212}) datasets. Furthermore, under the 2.25x, 5x and 11.7x outpainting settings, PQDiff only takes \textbf{40.6\%}, \textbf{20.3\%} and \textbf{10.2\%} of the time of the benchmark state-of-the-art (SOTA) method.

Title: LLsM: Generative Linguistic Steganography with Large Language Model

• Authors: Yihao Wang, Ruiqi Song, Ru Zhang, Jianyi Liu, Lingxiao Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15656
• Pdf URL: https://arxiv.org/pdf/2401.15656
• Copy Paste: [[2401.15656]] LLsM: Generative Linguistic Steganography with Large Language Model(https://arxiv.org/abs/2401.15656)
• Keywords: privacy, generative, large language model
• Abstract: Linguistic Steganography (LS) tasks aim to generate steganographic texts (stego) based on secret information. Only authorized recipients can perceive the existence of secret information in the texts and accurately extract it, thereby preserving privacy. However, the controllability of the stego generated by existing schemes is poor, and the generated stego is difficult to contain specific discourse characteristics such as style, genre, and theme. As a result, the stego are often easily detectable, compromising covert communication. To address these problems, this paper proposes a novel scheme named LLsM, a generative LS based on a Large Language Model (LLM). We fine-tuned the LLM LLaMA2 with a large-scale constructed dataset encompassing rich discourse characteristics, which enables the fine-tuned LLM to generate texts with specific discourse in a controllable manner. Then the discourse characteristics are used as guiding information and inputted into the fine-tuned LLM in the form of Prompt together with secret information. The candidate pool, derived from sampling and truncation, undergoes range encoding to ensure the stego imitate natural text distribution. Experiments demonstrate that LLsM performs superior to prevalent baselines regarding text quality, statistical analysis, discourse matching, and anti-steganalysis. In particular, LLsM's MAUVE surpasses that of some baselines by 70%-80%, and its anti-steganalysis performance is 30%-40% higher. Notably, we also present the long stego generated by LLsM, showing its potential superiority in long LS tasks.

Title: Data-Free Generalized Zero-Shot Learning

• Authors: Bowen Tang, Long Yan, Jing Zhang, Qian Yu, Lu Sheng, Dong Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15657
• Pdf URL: https://arxiv.org/pdf/2401.15657
• Copy Paste: [[2401.15657]] Data-Free Generalized Zero-Shot Learning(https://arxiv.org/abs/2401.15657)
• Keywords: privacy, data-free, generative
• Abstract: Deep learning models have the ability to extract rich knowledge from large-scale datasets. However, the sharing of data has become increasingly challenging due to concerns regarding data copyright and privacy. Consequently, this hampers the effective transfer of knowledge from existing data to novel downstream tasks and concepts. Zero-shot learning (ZSL) approaches aim to recognize new classes by transferring semantic knowledge learned from base classes. However, traditional generative ZSL methods often require access to real images from base classes and rely on manually annotated attributes, which presents challenges in terms of data restrictions and model scalability. To this end, this paper tackles a challenging and practical problem dubbed as data-free zero-shot learning (DFZSL), where only the CLIP-based base classes data pre-trained classifier is available for zero-shot classification. Specifically, we propose a generic framework for DFZSL, which consists of three main components. Firstly, to recover the virtual features of the base data, we model the CLIP features of base class images as samples from a von Mises-Fisher (vMF) distribution based on the pre-trained classifier. Secondly, we leverage the text features of CLIP as low-cost semantic information and propose a feature-language prompt tuning (FLPT) method to further align the virtual image features and textual features. Thirdly, we train a conditional generative model using the well-aligned virtual image features and corresponding semantic text features, enabling the generation of new classes features and achieve better zero-shot generalization. Our framework has been evaluated on five commonly used benchmarks for generalized ZSL, as well as 11 benchmarks for the base-to-new ZSL. The results demonstrate the superiority and effectiveness of our approach. Our code is available in https://github.com/ylong4/DFZSL

Title: Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes

• Authors: Weifeng Liu, Tianyi She, Jiawei Liu, Run Wang, Dongyu Yao, Ziyou Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15668
• Pdf URL: https://arxiv.org/pdf/2401.15668
• Copy Paste: [[2401.15668]] Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes(https://arxiv.org/abs/2401.15668)
• Keywords: security, robust
• Abstract: In recent years, DeepFake technology has achieved unprecedented success in high-quality video synthesis, whereas these methods also pose potential and severe security threats to humanity. DeepFake can be bifurcated into entertainment applications like face swapping and illicit uses such as lip-syncing fraud. However, lip-forgery videos, which neither change identity nor have discernible visual artifacts, present a formidable challenge to existing DeepFake detection methods. Our preliminary experiments have shown that the effectiveness of the existing methods often drastically decreases or even fails when tackling lip-syncing videos. In this paper, for the first time, we propose a novel approach dedicated to lip-forgery identification that exploits the inconsistency between lip movements and audio signals. We also mimic human natural cognition by capturing subtle biological links between lips and head regions to boost accuracy. To better illustrate the effectiveness and advances of our proposed method, we curate a high-quality LipSync dataset by employing the SOTA lip generator. We hope this high-quality and diverse dataset could be well served the further research on this challenging and interesting field. Experimental results show that our approach gives an average accuracy of more than 95.3% in spotting lip-syncing videos, significantly outperforming the baselines. Extensive experiments demonstrate the capability to tackle deepfakes and the robustness in surviving diverse input transformations. Our method achieves an accuracy of up to 90.2% in real-world scenarios (e.g., WeChat video call) and shows its powerful capabilities in real scenario deployment. To facilitate the progress of this research community, we release all resources at https://github.com/AaronComo/LipFD.

Title: YODA: Teacher-Student Progressive Learning for Language Models

• Authors: Jianqiao Lu, Wanjun Zhong, Yufei Wang, Zhijiang Guo, Qi Zhu, Wenyong Huang, Yanlin Wang, Fei Mi, Baojun Wang, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15670
• Pdf URL: https://arxiv.org/pdf/2401.15670
• Copy Paste: [[2401.15670]] YODA: Teacher-Student Progressive Learning for Language Models(https://arxiv.org/abs/2401.15670)
• Keywords: robust, large language model
• Abstract: Although large language models (LLMs) have demonstrated adeptness in a range of tasks, they still lag behind human learning efficiency. This disparity is often linked to the inherent human capacity to learn from basic examples, gradually generalize and handle more complex problems, and refine their skills with continuous feedback. Inspired by this, this paper introduces YODA, a novel teacher-student progressive learning framework that emulates the teacher-student education process to improve the efficacy of model fine-tuning. The framework operates on an interactive \textit{basic-generalized-harder} loop. The teacher agent provides tailored feedback on the student's answers, and systematically organizes the education process. This process unfolds by teaching the student basic examples, reinforcing understanding through generalized questions, and then enhancing learning by posing questions with progressively enhanced complexity. With the teacher's guidance, the student learns to iteratively refine its answer with feedback, and forms a robust and comprehensive understanding of the posed questions. The systematic procedural data, which reflects the progressive learning process of humans, is then utilized for model training. Taking math reasoning as a testbed, experiments show that training LLaMA2 with data from YODA improves SFT with significant performance gain (+17.01\% on GSM8K and +9.98\% on MATH). In addition, we find that training with curriculum learning further improves learning robustness.

Title: Detection of a facemask in real-time using deep learning methods: Prevention of Covid 19

• Authors: Gautam Siddharth Kashyap, Jatin Sohlot, Ayesha Siddiqui, Ramsha Siddiqui, Karan Malik, Samar Wazir, Alexander E. I. Brownlee
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15675
• Pdf URL: https://arxiv.org/pdf/2401.15675
• Copy Paste: [[2401.15675]] Detection of a facemask in real-time using deep learning methods: Prevention of Covid 19(https://arxiv.org/abs/2401.15675)
• Keywords: protect
• Abstract: A health crisis is raging all over the world with the rapid transmission of the novel-coronavirus disease (Covid-19). Out of the guidelines issued by the World Health Organisation (WHO) to protect us against Covid-19, wearing a facemask is the most effective. Many countries have necessitated the wearing of face masks, but monitoring a large number of people to ensure that they are wearing masks in a crowded place is a challenging task in itself. The novel-coronavirus disease (Covid-19) has already affected our day-to-day life as well as world trade movements. By the end of April 2021, the world has recorded 144,358,956 confirmed cases of novel-coronavirus disease (Covid-19) including 3,066,113 deaths according to the world health organization (WHO). These increasing numbers motivate automated techniques for the detection of a facemask in real-time scenarios for the prevention of Covid-19. We propose a technique using deep learning that works for single and multiple people in a frame recorded via webcam in still or in motion. We have also experimented with our approach in night light. The accuracy of our model is good compared to the other approaches in the literature; ranging from 74% for multiple people in a nightlight to 99% for a single person in daylight.

Title: Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance

• Authors: Qingcheng Zhao, Pengyu Long, Qixuan Zhang, Dafei Qin, Han Liang, Longwen Zhang, Yingliang Zhang, Jingyi Yu, Lan Xu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.15687
• Pdf URL: https://arxiv.org/pdf/2401.15687
• Copy Paste: [[2401.15687]] Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance(https://arxiv.org/abs/2401.15687)
• Keywords: diffusion
• Abstract: The synthesis of 3D facial animations from speech has garnered considerable attention. Due to the scarcity of high-quality 4D facial data and well-annotated abundant multi-modality labels, previous methods often suffer from limited realism and a lack of lexible conditioning. We address this challenge through a trilogy. We first introduce Generalized Neural Parametric Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial geometry and images to a highly generalized expression latent space, decoupling expressions and identities. Then, we utilize GNPFA to extract high-quality expressions and accurate head poses from a large array of videos. This presents the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial animation dataset with well-annotated emotional and style labels. Finally, we propose Media2Face, a diffusion model in GNPFA latent space for co-speech facial animation generation, accepting rich multi-modality guidances from audio, text, and image. Extensive experiments demonstrate that our model not only achieves high fidelity in facial animation synthesis but also broadens the scope of expressiveness and style adaptability in 3D facial animation.

Title: Divide and Conquer: Language Models can Plan and Self-Correct for Compositional Text-to-Image Generation

• Authors: Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, Zhenguo Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15688
• Pdf URL: https://arxiv.org/pdf/2401.15688
• Copy Paste: [[2401.15688]] Divide and Conquer: Language Models can Plan and Self-Correct for Compositional Text-to-Image Generation(https://arxiv.org/abs/2401.15688)
• Keywords: extraction, large language model
• Abstract: Despite significant advancements in text-to-image models for generating high-quality images, these methods still struggle to ensure the controllability of text prompts over images in the context of complex text prompts, especially when it comes to retaining object attributes and relationships. In this paper, we propose CompAgent, a training-free approach for compositional text-to-image generation, with a large language model (LLM) agent as its core. The fundamental idea underlying CompAgent is premised on a divide-and-conquer methodology. Given a complex text prompt containing multiple concepts including objects, attributes, and relationships, the LLM agent initially decomposes it, which entails the extraction of individual objects, their associated attributes, and the prediction of a coherent scene layout. These individual objects can then be independently conquered. Subsequently, the agent performs reasoning by analyzing the text, plans and employs the tools to compose these isolated objects. The verification and human feedback mechanism is finally incorporated into our agent to further correct the potential attribute errors and refine the generated images. Guided by the LLM agent, we propose a tuning-free multi-concept customization model and a layout-to-image generation model as the tools for concept composition, and a local image editing method as the tool to interact with the agent for verification. The scene layout controls the image generation process among these tools to prevent confusion among multiple objects. Extensive experiments demonstrate the superiority of our approach for compositional text-to-image generation: CompAgent achieves more than 10\% improvement on T2I-CompBench, a comprehensive benchmark for open-world compositional T2I generation. The extension to various related tasks also illustrates the flexibility of our CompAgent for potential applications.

Title: Phoneme-Based Proactive Anti-Eavesdropping with Controlled Recording Privilege

• Authors: Peng Huang, Yao Wei, Peng Cheng, Zhongjie Ba, Li Lu, Feng Lin, Yang Wang, Kui Ren
• Subjects: cs.CR, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.15704
• Pdf URL: https://arxiv.org/pdf/2401.15704
• Copy Paste: [[2401.15704]] Phoneme-Based Proactive Anti-Eavesdropping with Controlled Recording Privilege(https://arxiv.org/abs/2401.15704)
• Keywords: privacy, robust
• Abstract: The widespread smart devices raise people's concerns of being eavesdropped on. To enhance voice privacy, recent studies exploit the nonlinearity in microphone to jam audio recorders with inaudible ultrasound. However, existing solutions solely rely on energetic masking. Their simple-form noise leads to several problems, such as high energy requirements and being easily removed by speech enhancement techniques. Besides, most of these solutions do not support authorized recording, which restricts their usage scenarios. In this paper, we design an efficient yet robust system that can jam microphones while preserving authorized recording. Specifically, we propose a novel phoneme-based noise with the idea of informational masking, which can distract both machines and humans and is resistant to denoising techniques. Besides, we optimize the noise transmission strategy for broader coverage and implement a hardware prototype of our system. Experimental results show that our system can reduce the recognition accuracy of recordings to below 50\% under all tested speech recognition systems, which is much better than existing solutions.

Title: Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with Prototypical Embedding

• Authors: Jianxiang Lu, Cong Xie, Hui Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15708
• Pdf URL: https://arxiv.org/pdf/2401.15708
• Copy Paste: [[2401.15708]] Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with Prototypical Embedding(https://arxiv.org/abs/2401.15708)
• Keywords: diffusion
• Abstract: As large-scale text-to-image generation models have made remarkable progress in the field of text-to-image generation, many fine-tuning methods have been proposed. However, these models often struggle with novel objects, especially with one-shot scenarios. Our proposed method aims to address the challenges of generalizability and fidelity in an object-driven way, using only a single input image and the object-specific regions of interest. To improve generalizability and mitigate overfitting, in our paradigm, a prototypical embedding is initialized based on the object's appearance and its class, before fine-tuning the diffusion model. And during fine-tuning, we propose a class-characterizing regularization to preserve prior knowledge of object classes. To further improve fidelity, we introduce object-specific loss, which can also use to implant multiple objects. Overall, our proposed object-driven method for implanting new objects can integrate seamlessly with existing concepts as well as with high fidelity and generalization. Our method outperforms several existing works. The code will be released.

Title: Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddings

• Authors: Rohan Kapur, Logan Hallee, Arjun Patel, Bohdan Khomtchouk
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15713
• Pdf URL: https://arxiv.org/pdf/2401.15713
• Copy Paste: [[2401.15713]] Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddings(https://arxiv.org/abs/2401.15713)
• Keywords: transformer
• Abstract: The advancement of transformer neural networks has significantly elevated the capabilities of sentence similarity models, particularly in creating effective vector representations of natural language inputs. However, these models face notable challenges in domain-specific contexts, especially in highly specialized scientific sub-fields. Traditional methods often struggle in this regime, either overgeneralizing similarities within a niche or being overly sensitive to minor differences, resulting in inaccurate text classification and subpar vector representation. In an era where retrieval augmentation and search are increasingly crucial, precise and concise numerical representations are essential. In this paper, we target this issue by assembling niche datasets using co-citations as a similarity metric, focusing on biomedical domains. We employ two key strategies for fine-tuning state-of-the-art models: 1. Domain-specific Fine-Tuning, which tailors pretrained models to a single domain, and 2. Universal Applicability with Mixture of Experts (MoE), adapting pretrained models with enforced routing for multiple domains simultaneously. Our training approach emphasizes the use of abstracts for faster training, incorporating Multiple Negative Rankings loss for efficient contrastive learning. Notably, our MoE variants, equipped with $N$ experts, achieve the efficacy of $N$ individual models, heralding a new era of versatile, One-Size-Fits-All transformer networks for various tasks. This methodology marks significant advancements in scientific text classification metrics and holds promise for enhancing vector database search and compilation.

Title: SegmentAnyTree: A sensor and platform agnostic deep learning model for tree segmentation using laser scanning data

• Authors: Maciej Wielgosz, Stefano Puliti, Binbin Xiang, Konrad Schindler, Rasmus Astrup
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15739
• Pdf URL: https://arxiv.org/pdf/2401.15739
• Copy Paste: [[2401.15739]] SegmentAnyTree: A sensor and platform agnostic deep learning model for tree segmentation using laser scanning data(https://arxiv.org/abs/2401.15739)
• Keywords: segmentation
• Abstract: This research advances individual tree crown (ITC) segmentation in lidar data, using a deep learning model applicable to various laser scanning types: airborne (ULS), terrestrial (TLS), and mobile (MLS). It addresses the challenge of transferability across different data characteristics in 3D forest scene analysis. The study evaluates the model's performance based on platform (ULS, MLS) and data density, testing five scenarios with varying input data, including sparse versions, to gauge adaptability and canopy layer efficacy. The model, based on PointGroup architecture, is a 3D CNN with separate heads for semantic and instance segmentation, validated on diverse point cloud datasets. Results show point cloud sparsification enhances performance, aiding sparse data handling and improving detection in dense forests. The model performs well with >50 points per sq. m densities but less so at 10 points per sq. m due to higher omission rates. It outperforms existing methods (e.g., Point2Tree, TLS2trees) in detection, omission, commission rates, and F1 score, setting new benchmarks on LAUTx, Wytham Woods, and TreeLearn datasets. In conclusion, this study shows the feasibility of a sensor-agnostic model for diverse lidar data, surpassing sensor-specific approaches and setting new standards in tree segmentation, particularly in complex forests. This contributes to future ecological modeling and forest management advancements.

Title: SERNet-Former: Semantic Segmentation by Efficient Residual Network with Attention-Boosting Gates and Attention-Fusion Networks

• Authors: Serdar Erisen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15741
• Pdf URL: https://arxiv.org/pdf/2401.15741
• Copy Paste: [[2401.15741]] SERNet-Former: Semantic Segmentation by Efficient Residual Network with Attention-Boosting Gates and Attention-Fusion Networks(https://arxiv.org/abs/2401.15741)
• Keywords: segmentation
• Abstract: Improving the efficiency of state-of-the-art methods in semantic segmentation requires overcoming the increasing computational cost as well as issues such as fusing semantic information from global and local contexts. Based on the recent success and problems that convolutional neural networks (CNNs) encounter in semantic segmentation, this research proposes an encoder-decoder architecture with a unique efficient residual network. Attention-boosting gates (AbGs) and attention-boosting modules (AbMs) are deployed by aiming to fuse the feature-based semantic information with the global context of the efficient residual network in the encoder. Respectively, the decoder network is developed with the additional attention-fusion networks (AfNs) inspired by AbM. AfNs are designed to improve the efficiency in the one-to-one conversion of the semantic information by deploying additional convolution layers in the decoder part. Our network is tested on the challenging CamVid and Cityscapes datasets, and the proposed methods reveal significant improvements on the existing baselines, such as ResNet-50. To the best of our knowledge, the developed network, SERNet-Former, achieves state-of-the-art results (84.62 % mean IoU) on CamVid dataset and challenging results (87.35 % mean IoU) on Cityscapes validation dataset.

Title: An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion

• Authors: Sharib Ali, Yamid Espinel, Yueming Jin, Peng Liu, Bianca Güttner, Xukun Zhang, Lihua Zhang, Tom Dowrick, Matthew J. Clarkson, Shiting Xiao, Yifan Wu, Yijun Yang, Lei Zhu, Dai Sun, Lan Li, Micha Pfeiffer, Shahid Farid, Lena Maier-Hein, Emmanuel Buc, Adrien Bartoli
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15753
• Pdf URL: https://arxiv.org/pdf/2401.15753
• Copy Paste: [[2401.15753]] An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion(https://arxiv.org/abs/2401.15753)
• Keywords: segmentation
• Abstract: Augmented reality for laparoscopic liver resection is a visualisation mode that allows a surgeon to localise tumours and vessels embedded within the liver by projecting them on top of a laparoscopic image. Preoperative 3D models extracted from CT or MRI data are registered to the intraoperative laparoscopic images during this process. In terms of 3D-2D fusion, most of the algorithms make use of anatomical landmarks to guide registration. These landmarks include the liver's inferior ridge, the falciform ligament, and the occluding contours. They are usually marked by hand in both the laparoscopic image and the 3D model, which is time-consuming and may contain errors if done by a non-experienced user. Therefore, there is a need to automate this process so that augmented reality can be used effectively in the operating room. We present the Preoperative-to-Intraoperative Laparoscopic Fusion Challenge (P2ILF), held during the Medical Imaging and Computer Assisted Interventions (MICCAI 2022) conference, which investigates the possibilities of detecting these landmarks automatically and using them in registration. The challenge was divided into two tasks: 1) A 2D and 3D landmark detection task and 2) a 3D-2D registration task. The teams were provided with training data consisting of 167 laparoscopic images and 9 preoperative 3D models from 9 patients, with the corresponding 2D and 3D landmark annotations. A total of 6 teams from 4 countries participated, whose proposed methods were evaluated on 16 images and two preoperative 3D models from two patients. All the teams proposed deep learning-based methods for the 2D and 3D landmark segmentation tasks and differentiable rendering-based methods for the registration task. Based on the experimental outcomes, we propose three key hypotheses that determine current limitations and future directions for research in this domain.

Title: Integrating Differential Privacy and Contextual Integrity

• Authors: Sebastian Benthall, Rachel Cummings
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.15774
• Pdf URL: https://arxiv.org/pdf/2401.15774
• Copy Paste: [[2401.15774]] Integrating Differential Privacy and Contextual Integrity(https://arxiv.org/abs/2401.15774)
• Keywords: privacy
• Abstract: In this work, we propose the first framework for integrating Differential Privacy (DP) and Contextual Integrity (CI). DP is a property of an algorithm that injects statistical noise to obscure information about individuals represented within a database. CI defines privacy as information flow that is appropriate to social context. Analyzed together, these paradigms outline two dimensions on which to analyze privacy of information flows: descriptive and normative properties. We show that our new integrated framework provides benefits to both CI and DP that cannot be attained when each definition is considered in isolation: it enables contextually-guided tuning of the epsilon parameter in DP, and it enables CI to be applied to a broader set of information flows occurring in real-world systems, such as those involving PETs and machine learning. We conclude with a case study based on the use of DP in the U.S. Census Bureau.

Title: cantnlp@LT-EDI-2024: Automatic Detection of Anti-LGBTQ+ Hate Speech in Under-resourced Languages

• Authors: Sidney G.-J. Wong, Matthew Durward
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15777
• Pdf URL: https://arxiv.org/pdf/2401.15777
• Copy Paste: [[2401.15777]] cantnlp@LT-EDI-2024: Automatic Detection of Anti-LGBTQ+ Hate Speech in Under-resourced Languages(https://arxiv.org/abs/2401.15777)
• Keywords: transformer
• Abstract: This paper describes our homophobia/transphobia in social media comments detection system developed as part of the shared task at LT-EDI-2024. We took a transformer-based approach to develop our multiclass classification model for ten language conditions (English, Spanish, Gujarati, Hindi, Kannada, Malayalam, Marathi, Tamil, Tulu, and Telugu). We introduced synthetic and organic instances of script-switched language data during domain adaptation to mirror the linguistic realities of social media language as seen in the labelled training data. Our system ranked second for Gujarati and Telugu with varying levels of performance for other language conditions. The results suggest incorporating elements of paralinguistic behaviour such as script-switching may improve the performance of language detection systems especially in the cases of under-resourced languages conditions.

Title: Fine-Tuned Large Language Models for Symptom Recognition from Spanish Clinical Text

• Authors: Mai A. Shaaban, Abbas Akkasi, Adnan Khan, Majid Komeili, Mohammad Yaqub
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15780
• Pdf URL: https://arxiv.org/pdf/2401.15780
• Copy Paste: [[2401.15780]] Fine-Tuned Large Language Models for Symptom Recognition from Spanish Clinical Text(https://arxiv.org/abs/2401.15780)
• Keywords: extraction, large language model
• Abstract: The accurate recognition of symptoms in clinical reports is significantly important in the fields of healthcare and biomedical natural language processing. These entities serve as essential building blocks for clinical information extraction, enabling retrieval of critical medical insights from vast amounts of textual data. Furthermore, the ability to identify and categorize these entities is fundamental for developing advanced clinical decision support systems, aiding healthcare professionals in diagnosis and treatment planning. In this study, we participated in SympTEMIST, a shared task on the detection of symptoms, signs and findings in Spanish medical documents. We combine a set of large language models fine-tuned with the data released by the organizers.

Title: Prediction of Breast Cancer Recurrence Risk Using a Multi-Model Approach Integrating Whole Slide Imaging and Clinicopathologic Features

• Authors: Manu Goyal, Jonathan D. Marotti, Adrienne A. Workman, Elaine P. Kuhn, Graham M. Tooker, Seth K. Ramin, Mary D. Chamberlin, Roberta M. diFlorio-Alexander, Saeed Hassanpour
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15805
• Pdf URL: https://arxiv.org/pdf/2401.15805
• Copy Paste: [[2401.15805]] Prediction of Breast Cancer Recurrence Risk Using a Multi-Model Approach Integrating Whole Slide Imaging and Clinicopathologic Features(https://arxiv.org/abs/2401.15805)
• Keywords: extraction, transformer
• Abstract: Breast cancer is the most common malignancy affecting women worldwide and is notable for its morphologic and biologic diversity, with varying risks of recurrence following treatment. The Oncotype DX Breast Recurrence Score test is an important predictive and prognostic genomic assay for estrogen receptor-positive breast cancer that guides therapeutic strategies; however, such tests can be expensive, delay care, and are not widely available. The aim of this study was to develop a multi-model approach integrating the analysis of whole slide images and clinicopathologic data to predict their associated breast cancer recurrence risks and categorize these patients into two risk groups according to the predicted score: low and high risk. The proposed novel methodology uses convolutional neural networks for feature extraction and vision transformers for contextual aggregation, complemented by a logistic regression model that analyzes clinicopathologic data for classification into two risk categories. This method was trained and tested on 993 hematoxylin and eosin-stained whole-slide images of breast cancers with corresponding clinicopathological features that had prior Oncotype DX testing. The model's performance was evaluated using an internal test set of 198 patients from Dartmouth Health and an external test set of 418 patients from the University of Chicago. The multi-model approach achieved an AUC of 0.92 (95 percent CI: 0.88-0.96) on the internal set and an AUC of 0.85 (95 percent CI: 0.79-0.90) on the external cohort. These results suggest that with further validation, the proposed methodology could provide an alternative to assist clinicians in personalizing treatment for breast cancer patients and potentially improving their outcomes.

Title: Transparency Attacks: How Imperceptible Image Layers Can Fool AI Perception

• Authors: Forrest McKee, David Noever
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15817
• Pdf URL: https://arxiv.org/pdf/2401.15817
• Copy Paste: [[2401.15817]] Transparency Attacks: How Imperceptible Image Layers Can Fool AI Perception(https://arxiv.org/abs/2401.15817)
• Keywords: attack, steal, watermark, transformer
• Abstract: This paper investigates a novel algorithmic vulnerability when imperceptible image layers confound multiple vision models into arbitrary label assignments and captions. We explore image preprocessing methods to introduce stealth transparency, which triggers AI misinterpretation of what the human eye perceives. The research compiles a broad attack surface to investigate the consequences ranging from traditional watermarking, steganography, and background-foreground miscues. We demonstrate dataset poisoning using the attack to mislabel a collection of grayscale landscapes and logos using either a single attack layer or randomly selected poisoning classes. For example, a military tank to the human eye is a mislabeled bridge to object classifiers based on convolutional networks (YOLO, etc.) and vision transformers (ViT, GPT-Vision, etc.). A notable attack limitation stems from its dependency on the background (hidden) layer in grayscale as a rough match to the transparent foreground image that the human eye perceives. This dependency limits the practical success rate without manual tuning and exposes the hidden layers when placed on the opposite display theme (e.g., light background, light transparent foreground visible, works best against a light theme image viewer or browser). The stealth transparency confounds established vision systems, including evading facial recognition and surveillance, digital watermarking, content filtering, dataset curating, automotive and drone autonomy, forensic evidence tampering, and retail product misclassifying. This method stands in contrast to traditional adversarial attacks that typically focus on modifying pixel values in ways that are either slightly perceptible or entirely imperceptible for both humans and machines.

Title: The Spectre of Surveillance and Censorship in Future Internet Architectures

• Authors: Michael Wrana, Diogo Barradas, N. Asokan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.15828
• Pdf URL: https://arxiv.org/pdf/2401.15828
• Copy Paste: [[2401.15828]] The Spectre of Surveillance and Censorship in Future Internet Architectures(https://arxiv.org/abs/2401.15828)
• Keywords: security, privacy
• Abstract: Recent initiatives known as Future Internet Architectures (FIAs) seek to redesign the Internet to improve performance, scalability, and security. However, some governments perceive Internet access as a threat to their political standing and engage in widespread network surveillance and censorship. In this paper, we provide an in-depth analysis into the designs of prominent FIAs, to help understand of how FIAs impact surveillance and censorship abilities. Then, we survey the applicability of privacy-enhancing technologies to FIAs. We conclude by providing guidelines for future research into novel FIA-based privacy-enhancing technologies, and recommendations to guide the evaluation of these technologies.

Title: Emergent Explainability: Adding a causal chain to neural network inference

• Authors: Adam Perrett
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15840
• Pdf URL: https://arxiv.org/pdf/2401.15840
• Copy Paste: [[2401.15840]] Emergent Explainability: Adding a causal chain to neural network inference(https://arxiv.org/abs/2401.15840)
• Keywords: explainability
• Abstract: This position paper presents a theoretical framework for enhancing explainable artificial intelligence (xAI) through emergent communication (EmCom), focusing on creating a causal understanding of AI model outputs. We explore the novel integration of EmCom into AI systems, offering a paradigm shift from conventional associative relationships between inputs and outputs to a more nuanced, causal interpretation. The framework aims to revolutionize how AI processes are understood, making them more transparent and interpretable. While the initial application of this model is demonstrated on synthetic data, the implications of this research extend beyond these simple applications. This general approach has the potential to redefine interactions with AI across multiple domains, fostering trust and informed decision-making in healthcare and in various sectors where AI's decision-making processes are critical. The paper discusses the theoretical underpinnings of this approach, its potential broad applications, and its alignment with the growing need for responsible and transparent AI systems in an increasingly digital world.

Title: LCVO: An Efficient Pretraining-Free Framework for Visual Question Answering Grounding

• Authors: Yuhan Chen, Lumei Su, Lihua Chen, Zhiwei Lin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15842
• Pdf URL: https://arxiv.org/pdf/2401.15842
• Copy Paste: [[2401.15842]] LCVO: An Efficient Pretraining-Free Framework for Visual Question Answering Grounding(https://arxiv.org/abs/2401.15842)
• Keywords: robust, large language model
• Abstract: In this paper, the LCVO modular method is proposed for the Visual Question Answering (VQA) Grounding task in the vision-language multimodal domain. This approach relies on a frozen large language model (LLM) as intermediate mediator between the off-the-shelf VQA model and the off-the-shelf Open-Vocabulary Object Detection (OVD) model, where the LLM transforms and conveys textual information between the two modules based on a designed prompt. LCVO establish an integrated plug-and-play framework without the need for any pre-training process. This framework can be deployed for VQA Grounding tasks under low computational resources. The modularized model within the framework allows application with various state-of-the-art pre-trained models, exhibiting significant potential to be advance with the times. Experimental implementations were conducted under constrained computational and memory resources, evaluating the proposed method's performance on benchmark datasets including GQA, CLEVR, and VizWiz-VQA-Grounding. Comparative analyses with baseline methods demonstrate the robust competitiveness of LCVO.

Title: Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing

• Authors: Maofeng Tang, Andrei Cozma, Konstantinos Georgiou, Hairong Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15855
• Pdf URL: https://arxiv.org/pdf/2401.15855
• Copy Paste: [[2401.15855]] Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing(https://arxiv.org/abs/2401.15855)
• Keywords: generative
• Abstract: Remote sensing images present unique challenges to image analysis due to the extensive geographic coverage, hardware limitations, and misaligned multi-scale images. This paper revisits the classical multi-scale representation learning problem but under the general framework of self-supervised learning for remote sensing image understanding. We present Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE).During pre-training, Cross-Scale MAE employs scale augmentation techniques and enforces cross-scale consistency constraints through both contrastive and generative losses to ensure consistent and meaningful representations well-suited for a wide range of downstream tasks. Further, our implementation leverages the xFormers library to accelerate network pre-training on a single GPU while maintaining the quality of learned representations. Experimental evaluations demonstrate that Cross-Scale MAE exhibits superior performance compared to standard MAE and other state-of-the-art remote sensing MAE methods.

Title: Diffusion Facial Forgery Detection

• Authors: Harry Cheng, Yangyang Guo, Tianyi Wang, Liqiang Nie, Mohan Kankanhalli
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15859
• Pdf URL: https://arxiv.org/pdf/2401.15859
• Copy Paste: [[2401.15859]] Diffusion Facial Forgery Detection(https://arxiv.org/abs/2401.15859)
• Keywords: diffusion
• Abstract: Detecting diffusion-generated images has recently grown into an emerging research area. Existing diffusion-based datasets predominantly focus on general image generation. However, facial forgeries, which pose a more severe social risk, have remained less explored thus far. To address this gap, this paper introduces DiFF, a comprehensive dataset dedicated to face-focused diffusion-generated images. DiFF comprises over 500,000 images that are synthesized using thirteen distinct generation methods under four conditions. In particular, this dataset leverages 30,000 carefully collected textual and visual prompts, ensuring the synthesis of images with both high fidelity and semantic consistency. We conduct extensive experiments on the DiFF dataset via a human test and several representative forgery detection methods. The results demonstrate that the binary detection accuracy of both human observers and automated detectors often falls below 30%, shedding light on the challenges in detecting diffusion-generated facial forgeries. Furthermore, we propose an edge graph regularization approach to effectively enhance the generalization capability of existing detectors.

Title: DrBERT: Unveiling the Potential of Masked Language Modeling Decoder in BERT pretraining

• Authors: Wen Liang, Youzhi Liang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15861
• Pdf URL: https://arxiv.org/pdf/2401.15861
• Copy Paste: [[2401.15861]] DrBERT: Unveiling the Potential of Masked Language Modeling Decoder in BERT pretraining(https://arxiv.org/abs/2401.15861)
• Keywords: transformer
• Abstract: BERT (Bidirectional Encoder Representations from Transformers) has revolutionized the field of natural language processing through its exceptional performance on numerous tasks. Yet, the majority of researchers have mainly concentrated on enhancements related to the model structure, such as relative position embedding and more efficient attention mechanisms. Others have delved into pretraining tricks associated with Masked Language Modeling, including whole word masking. DeBERTa introduced an enhanced decoder adapted for BERT's encoder model for pretraining, proving to be highly effective. We argue that the design and research around enhanced masked language modeling decoders have been underappreciated. In this paper, we propose several designs of enhanced decoders and introduce DrBERT (Decoder-refined BERT), a novel method for modeling training. Typically, a pretrained BERT model is fine-tuned for specific Natural Language Understanding (NLU) tasks. In our approach, we utilize the original BERT model as the encoder, making only changes to the decoder without altering the encoder. This approach does not necessitate extensive modifications to the model's architecture and can be seamlessly integrated into existing fine-tuning pipelines and services, offering an efficient and effective enhancement strategy. Compared to other methods, while we also incur a moderate training cost for the decoder during the pretraining process, our approach does not introduce additional training costs during the fine-tuning phase. We test multiple enhanced decoder structures after pretraining and evaluate their performance on the GLUE benchmark. Our results demonstrate that DrBERT, having only undergone subtle refinements to the model structure during pretraining, significantly enhances model performance without escalating the inference time and serving budget.

Title: Importance-Aware Adaptive Dataset Distillation

• Authors: Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15863
• Pdf URL: https://arxiv.org/pdf/2401.15863
• Copy Paste: [[2401.15863]] Importance-Aware Adaptive Dataset Distillation(https://arxiv.org/abs/2401.15863)
• Keywords: privacy, robust
• Abstract: Herein, we propose a novel dataset distillation method for constructing small informative datasets that preserve the information of the large original datasets. The development of deep learning models is enabled by the availability of large-scale datasets. Despite unprecedented success, large-scale datasets considerably increase the storage and transmission costs, resulting in a cumbersome model training process. Moreover, using raw data for training raises privacy and copyright concerns. To address these issues, a new task named dataset distillation has been introduced, aiming to synthesize a compact dataset that retains the essential information from the large original dataset. State-of-the-art (SOTA) dataset distillation methods have been proposed by matching gradients or network parameters obtained during training on real and synthetic datasets. The contribution of different network parameters to the distillation process varies, and uniformly treating them leads to degraded distillation performance. Based on this observation, we propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance by automatically assigning importance weights to different network parameters during distillation, thereby synthesizing more robust distilled datasets. IADD demonstrates superior performance over other SOTA dataset distillation methods based on parameter matching on multiple benchmark datasets and outperforms them in terms of cross-architecture generalization. In addition, the analysis of self-adaptive weights demonstrates the effectiveness of IADD. Furthermore, the effectiveness of IADD is validated in a real-world medical application such as COVID-19 detection.

Title: Quantum Circuit Reconstruction from Power Side-Channel Attacks on Quantum Computer Controllers

• Authors: Ferhat Erata, Chuanqi Xu, Ruzica Piskac, Jakub Szefer
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.15869
• Pdf URL: https://arxiv.org/pdf/2401.15869
• Copy Paste: [[2401.15869]] Quantum Circuit Reconstruction from Power Side-Channel Attacks on Quantum Computer Controllers(https://arxiv.org/abs/2401.15869)
• Keywords: protect, attack, steal
• Abstract: The interest in quantum computing has grown rapidly in recent years, and with it grows the importance of securing quantum circuits. A novel type of threat to quantum circuits that dedicated attackers could launch are power trace attacks. To address this threat, this paper presents first formalization and demonstration of using power traces to unlock and steal quantum circuit secrets. With access to power traces, attackers can recover information about the control pulses sent to quantum computers. From the control pulses, the gate level description of the circuits, and eventually the secret algorithms can be reverse engineered. This work demonstrates how and what information could be recovered. This work uses algebraic reconstruction from power traces to realize two new types of single trace attacks: per-channel and total power attacks. The former attack relies on per-channel measurements to perform a brute-force attack to reconstruct the quantum circuits. The latter attack performs a single-trace attack using Mixed-Integer Linear Programming optimization. Through the use of algebraic reconstruction, this work demonstrates that quantum circuit secrets can be stolen with high accuracy. Evaluation on 32 real benchmark quantum circuits shows that our technique is highly effective at reconstructing quantum circuits. The findings not only show the veracity of the potential attacks, but also the need to develop new means to protect quantum circuits from power trace attacks. Throughout this work real control pulse information from real quantum computers is used to demonstrate potential attacks based on simulation of collection of power traces.

Title: Combining Satellite and Weather Data for Crop Type Mapping: An Inverse Modelling Approach

• Authors: Praveen Ravirathinam, Rahul Ghosh, Ankush Khandelwal, Xiaowei Jia, David Mulla, Vipin Kumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15875
• Pdf URL: https://arxiv.org/pdf/2401.15875
• Copy Paste: [[2401.15875]] Combining Satellite and Weather Data for Crop Type Mapping: An Inverse Modelling Approach(https://arxiv.org/abs/2401.15875)
• Keywords: segmentation
• Abstract: Accurate and timely crop mapping is essential for yield estimation, insurance claims, and conservation efforts. Over the years, many successful machine learning models for crop mapping have been developed that use just the multi-spectral imagery from satellites to predict crop type over the area of interest. However, these traditional methods do not account for the physical processes that govern crop growth. At a high level, crop growth can be envisioned as physical parameters, such as weather and soil type, acting upon the plant leading to crop growth which can be observed via satellites. In this paper, we propose Weather-based Spatio-Temporal segmentation network with ATTention (WSTATT), a deep learning model that leverages this understanding of crop growth by formulating it as an inverse model that combines weather (Daymet) and satellite imagery (Sentinel-2) to generate accurate crop maps. We show that our approach provides significant improvements over existing algorithms that solely rely on spectral imagery by comparing segmentation maps and F1 classification scores. Furthermore, effective use of attention in WSTATT architecture enables detection of crop types earlier in the season (up to 5 months in advance), which is very useful for improving food supply projections. We finally discuss the impact of weather by correlating our results with crop phenology to show that WSTATT is able to capture physical properties of crop growth.

Title: Decoding the MITRE Engenuity ATT&CK Enterprise Evaluation: An Analysis of EDR Performance in Real-World Environments

• Authors: Xiangmin Shen, Zhenyuan Li, Graham Burleigh, Lingzhi Wang, Yan Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.15878
• Pdf URL: https://arxiv.org/pdf/2401.15878
• Copy Paste: [[2401.15878]] Decoding the MITRE Engenuity ATT&CK Enterprise Evaluation: An Analysis of EDR Performance in Real-World Environments(https://arxiv.org/abs/2401.15878)
• Keywords: security, attack
• Abstract: Endpoint detection and response (EDR) systems have emerged as a critical component of enterprise security solutions, effectively combating endpoint threats like APT attacks with extended lifecycles. In light of the growing significance of endpoint detection and response (EDR) systems, many cybersecurity providers have developed their own proprietary EDR solutions. It's crucial for users to assess the capabilities of these detection engines to make informed decisions about which products to choose. This is especially urgent given the market's size, which is expected to reach around 3.7 billion dollars by 2023 and is still expanding. MITRE is a leading organization in cyber threat analysis. In 2018, MITRE started to conduct annual APT emulations that cover major EDR vendors worldwide. Indicators include telemetry, detection and blocking capability, etc. Nevertheless, the evaluation results published by MITRE don't contain any further interpretations or suggestions. In this paper, we thoroughly analyzed MITRE evaluation results to gain further insights into real-world EDR systems under test. Specifically, we designed a whole-graph analysis method, which utilizes additional control flow and data flow information to measure the performance of EDR systems. Besides, we analyze MITRE evaluation's results over multiple years from various aspects, including detection coverage, detection confidence, detection modifier, data source, compatibility, etc. Through the above studies, we have compiled a thorough summary of our findings and gained valuable insights from the evaluation results. We believe these summaries and insights can assist researchers, practitioners, and vendors in better understanding the strengths and limitations of mainstream EDR products.

Title: TransTroj: Transferable Backdoor Attacks to Pre-trained Models via Embedding Indistinguishability

• Authors: Hao Wang, Tao Xiang, Shangwei Guo, Jialing He, Hangcheng Liu, Tianwei Zhang
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15883
• Pdf URL: https://arxiv.org/pdf/2401.15883
• Copy Paste: [[2401.15883]] TransTroj: Transferable Backdoor Attacks to Pre-trained Models via Embedding Indistinguishability(https://arxiv.org/abs/2401.15883)
• Keywords: attack
• Abstract: Pre-trained models (PTMs) are extensively utilized in various downstream tasks. Adopting untrusted PTMs may suffer from backdoor attacks, where the adversary can compromise the downstream models by injecting backdoors into the PTM. However, existing backdoor attacks to PTMs can only achieve partially task-agnostic and the embedded backdoors are easily erased during the fine-tuning process. In this paper, we propose a novel transferable backdoor attack, TransTroj, to simultaneously meet functionality-preserving, durable, and task-agnostic. In particular, we first formalize transferable backdoor attacks as the indistinguishability problem between poisoned and clean samples in the embedding space. We decompose the embedding indistinguishability into pre- and post-indistinguishability, representing the similarity of the poisoned and reference embeddings before and after the attack. Then, we propose a two-stage optimization that separately optimizes triggers and victim PTMs to achieve embedding indistinguishability. We evaluate TransTroj on four PTMs and six downstream tasks. Experimental results show that TransTroj significantly outperforms SOTA task-agnostic backdoor attacks (18%$\sim$99%, 68% on average) and exhibits superior performance under various system settings. The code is available at https://github.com/haowang-cqu/TransTroj .

Title: Corrective Retrieval Augmented Generation

• Authors: Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, Zhen-Hua Ling
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15884
• Pdf URL: https://arxiv.org/pdf/2401.15884
• Copy Paste: [[2401.15884]] Corrective Retrieval Augmented Generation(https://arxiv.org/abs/2401.15884)
• Keywords: secure, robust, large language model
• Abstract: Large language models (LLMs) inevitably exhibit hallucinations since the accuracy of generated texts cannot be secured solely by the parametric knowledge they encapsulate. Although retrieval-augmented generation (RAG) is a practicable complement to LLMs, it relies heavily on the relevance of retrieved documents, raising concerns about how the model behaves if retrieval goes wrong. To this end, we propose the Corrective Retrieval Augmented Generation (CRAG) to improve the robustness of generation. Specifically, a lightweight retrieval evaluator is designed to assess the overall quality of retrieved documents for a query, returning a confidence degree based on which different knowledge retrieval actions can be triggered. Since retrieval from static and limited corpora can only return sub-optimal documents, large-scale web searches are utilized as an extension for augmenting the retrieval results. Besides, a decompose-then-recompose algorithm is designed for retrieved documents to selectively focus on key information and filter out irrelevant information in them. CRAG is plug-and-play and can be seamlessly coupled with various RAG-based approaches. Experiments on four datasets covering short- and long-form generation tasks show that CRAG can significantly improve the performance of RAG-based approaches.

Title: Grey Level Texture Features for Segmentation of Chromogenic Dye RNAscope From Breast Cancer Tissue

• Authors: Andrew Davidson (1), Arthur Morley-Bunker (2), George Wiggins (2), Logan Walker (2), Gavin Harris (3), Ramakrishnan Mukundan (1), kConFab Investigators (4 and 5) ((1) University of Canterbury, (2) University of Otago, (3) Canterbury Health Laboratories, (4) The University of Melbourne, (5) Peter MacCallum Cancer Center)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15886
• Pdf URL: https://arxiv.org/pdf/2401.15886
• Copy Paste: [[2401.15886]] Grey Level Texture Features for Segmentation of Chromogenic Dye RNAscope From Breast Cancer Tissue(https://arxiv.org/abs/2401.15886)
• Keywords: segmentation
• Abstract: Chromogenic RNAscope dye and haematoxylin staining of cancer tissue facilitates diagnosis of the cancer type and subsequent treatment, and fits well into existing pathology workflows. However, manual quantification of the RNAscope transcripts (dots), which signify gene expression, is prohibitively time consuming. In addition, there is a lack of verified supporting methods for quantification and analysis. This paper investigates the usefulness of gray level texture features for automatically segmenting and classifying the positions of RNAscope transcripts from breast cancer tissue. Feature analysis showed that a small set of gray level features, including Gray Level Dependence Matrix and Neighbouring Gray Tone Difference Matrix features, were well suited for the task. The automated method performed similarly to expert annotators at identifying the positions of RNAscope transcripts, with an F1-score of 0.571 compared to the expert inter-rater F1-score of 0.596. These results demonstrate the potential of gray level texture features for automated quantification of RNAscope in the pathology workflow.

Title: A Gated MLP Architecture for Learning Topological Dependencies in Spatio-Temporal Graphs

• Authors: Yun Young Choi, Minho Lee, Sun Woo Park, Seunghwan Lee, Joohwan Ko
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15894
• Pdf URL: https://arxiv.org/pdf/2401.15894
• Copy Paste: [[2401.15894]] A Gated MLP Architecture for Learning Topological Dependencies in Spatio-Temporal Graphs(https://arxiv.org/abs/2401.15894)
• Keywords: transformer
• Abstract: Graph Neural Networks (GNNs) and Transformer have been increasingly adopted to learn the complex vector representations of spatio-temporal graphs, capturing intricate spatio-temporal dependencies crucial for applications such as traffic datasets. Although many existing methods utilize multi-head attention mechanisms and message-passing neural networks (MPNNs) to capture both spatial and temporal relations, these approaches encode temporal and spatial relations independently, and reflect the graph's topological characteristics in a limited manner. In this work, we introduce the Cycle to Mixer (Cy2Mixer), a novel spatio-temporal GNN based on topological non-trivial invariants of spatio-temporal graphs with gated multi-layer perceptrons (gMLP). The Cy2Mixer is composed of three blocks based on MLPs: A message-passing block for encapsulating spatial information, a cycle message-passing block for enriching topological information through cyclic subgraphs, and a temporal block for capturing temporal properties. We bolster the effectiveness of Cy2Mixer with mathematical evidence emphasizing that our cycle message-passing block is capable of offering differentiated information to the deep learning model compared to the message-passing block. Furthermore, empirical evaluations substantiate the efficacy of the Cy2Mixer, demonstrating state-of-the-art performances across various traffic benchmark datasets.

Title: MV2MAE: Multi-View Video Masked Autoencoders

• Authors: Ketul Shah, Robert Crandall, Jie Xu, Peng Zhou, Marian George, Mayank Bansal, Rama Chellappa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15900
• Pdf URL: https://arxiv.org/pdf/2401.15900
• Copy Paste: [[2401.15900]] MV2MAE: Multi-View Video Masked Autoencoders(https://arxiv.org/abs/2401.15900)
• Keywords: robust
• Abstract: Videos captured from multiple viewpoints can help in perceiving the 3D structure of the world and benefit computer vision tasks such as action recognition, tracking, etc. In this paper, we present a method for self-supervised learning from synchronized multi-view videos. We use a cross-view reconstruction task to inject geometry information in the model. Our approach is based on the masked autoencoder (MAE) framework. In addition to the same-view decoder, we introduce a separate cross-view decoder which leverages cross-attention mechanism to reconstruct a target viewpoint video using a video from source viewpoint, to help representations robust to viewpoint changes. For videos, static regions can be reconstructed trivially which hinders learning meaningful representations. To tackle this, we introduce a motion-weighted reconstruction loss which improves temporal modeling. We report state-of-the-art results on the NTU-60, NTU-120 and ETRI datasets, as well as in the transfer learning setting on NUCLA, PKU-MMD-II and ROCOG-v2 datasets, demonstrating the robustness of our approach. Code will be made available.

Title: Toward the Identifiability of Comparative Deep Generative Models

• Authors: Romain Lopez, Jan-Christian Huetter, Ehsan Hajiramezanali, Jonathan Pritchard, Aviv Regev
• Subjects: cs.LG, q-bio.GN, stat.ME
• Abstract URL: https://arxiv.org/abs/2401.15903
• Pdf URL: https://arxiv.org/pdf/2401.15903
• Copy Paste: [[2401.15903]] Toward the Identifiability of Comparative Deep Generative Models(https://arxiv.org/abs/2401.15903)
• Keywords: generative
• Abstract: Deep Generative Models (DGMs) are versatile tools for learning data representations while adequately incorporating domain knowledge such as the specification of conditional probability distributions. Recently proposed DGMs tackle the important task of comparing data sets from different sources. One such example is the setting of contrastive analysis that focuses on describing patterns that are enriched in a target data set compared to a background data set. The practical deployment of those models often assumes that DGMs naturally infer interpretable and modular latent representations, which is known to be an issue in practice. Consequently, existing methods often rely on ad-hoc regularization schemes, although without any theoretical grounding. Here, we propose a theory of identifiability for comparative DGMs by extending recent advances in the field of non-linear independent component analysis. We show that, while these models lack identifiability across a general class of mixing functions, they surprisingly become identifiable when the mixing function is piece-wise affine (e.g., parameterized by a ReLU neural network). We also investigate the impact of model misspecification, and empirically show that previously proposed regularization techniques for fitting comparative DGMs help with identifiability when the number of latent variables is not known in advance. Finally, we introduce a novel methodology for fitting comparative DGMs that improves the treatment of multiple data sources via multi-objective optimization and that helps adjust the hyperparameter for the regularization in an interpretable manner, using constrained optimization. We empirically validate our theory and new methodology using simulated data as well as a recent data set of genetic perturbations in cells profiled via single-cell RNA sequencing.

Title: Mean Estimation with User-Level Privacy for Spatio-Temporal IoT Datasets

• Authors: V. Arvind Rameshwar, Anshoo Tandon, Prajjwal Gupta, Novoneel Chakraborty, Abhay Sharma
• Subjects: cs.CR, cs.IT, stat.AP
• Abstract URL: https://arxiv.org/abs/2401.15906
• Pdf URL: https://arxiv.org/pdf/2401.15906
• Copy Paste: [[2401.15906]] Mean Estimation with User-Level Privacy for Spatio-Temporal IoT Datasets(https://arxiv.org/abs/2401.15906)
• Keywords: privacy
• Abstract: This paper considers the problem of the private release of sample means of speed values from traffic datasets. Our key contribution is the development of user-level differentially private algorithms that incorporate carefully chosen parameter values to ensure low estimation errors on real-world datasets, while ensuring privacy. We test our algorithms on ITMS (Intelligent Traffic Management System) data from an Indian city, where the speeds of different buses are drawn in a potentially non-i.i.d. manner from an unknown distribution, and where the number of speed samples contributed by different buses is potentially different. We then apply our algorithms to a synthetic dataset, generated based on the ITMS data, having either a large number of users or a large number of samples per user. Here, we provide recommendations for the choices of parameters and algorithm subroutines that result in low estimation errors, while guaranteeing user-level privacy.

Title: Blockchain-enabled Trustworthy Federated Unlearning

• Authors: Yijing Lin, Zhipeng Gao, Hongyang Du, Jinke Ren, Zhiqiang Xie, Dusit Niyato
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.15917
• Pdf URL: https://arxiv.org/pdf/2401.15917
• Copy Paste: [[2401.15917]] Blockchain-enabled Trustworthy Federated Unlearning(https://arxiv.org/abs/2401.15917)
• Keywords: protect, federate
• Abstract: Federated unlearning is a promising paradigm for protecting the data ownership of distributed clients. It allows central servers to remove historical data effects within the machine learning model as well as address the "right to be forgotten" issue in federated learning. However, existing works require central servers to retain the historical model parameters from distributed clients, such that allows the central server to utilize these parameters for further training even, after the clients exit the training process. To address this issue, this paper proposes a new blockchain-enabled trustworthy federated unlearning framework. We first design a proof of federated unlearning protocol, which utilizes the Chameleon hash function to verify data removal and eliminate the data contributions stored in other clients' models. Then, an adaptive contribution-based retraining mechanism is developed to reduce the computational overhead and significantly improve the training efficiency. Extensive experiments demonstrate that the proposed framework can achieve a better data removal effect than the state-of-the-art frameworks, marking a significant stride towards trustworthy federated unlearning.

Title: E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models

• Authors: Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, Min Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15927
• Pdf URL: https://arxiv.org/pdf/2401.15927
• Copy Paste: [[2401.15927]] E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models(https://arxiv.org/abs/2401.15927)
• Keywords: large language model
• Abstract: With the accelerating development of Large Language Models (LLMs), many LLMs are beginning to be used in the Chinese K-12 education domain. The integration of LLMs and education is getting closer and closer, however, there is currently no benchmark for evaluating LLMs that focuses on the Chinese K-12 education domain. Therefore, there is an urgent need for a comprehensive natural language processing benchmark to accurately assess the capabilities of various LLMs in the Chinese K-12 education domain. To address this, we introduce the E-EVAL, the first comprehensive evaluation benchmark specifically designed for the Chinese K-12 education field. The E-EVAL consists of 4,351 multiple-choice questions at the primary, middle, and high school levels across a wide range of subjects, including Chinese, English, Politics, History, Ethics, Physics, Chemistry, Mathematics, and Geography. We conducted a comprehensive evaluation of E-EVAL on advanced LLMs, including both English-dominant and Chinese-dominant models. Findings show that Chinese-dominant models perform well compared to English-dominant models, with many scoring even above the GPT 4.0. However, almost all models perform poorly in complex subjects such as mathematics. We also found that most Chinese-dominant LLMs did not achieve higher scores at the primary school level compared to the middle school level. We observe that the mastery of higher-order knowledge by the model does not necessarily imply the mastery of lower-order knowledge as well. Additionally, the experimental results indicate that the Chain of Thought (CoT) technique is effective only for the challenging science subjects, while Few-shot prompting is more beneficial for liberal arts subjects. With E-EVAL, we aim to analyze the strengths and limitations of LLMs in educational applications, and to contribute to the progress and development of Chinese K-12 education and LLMs.

Title: HICH Image/Text (HICH-IT): Comprehensive Text and Image Datasets for Hypertensive Intracerebral Hemorrhage Research

• Authors: Jie Li, Yulong Xia, Tongxin Yang, Fenglin Cai, Miao Wei, Zhiwei Zhang, Li Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15934
• Pdf URL: https://arxiv.org/pdf/2401.15934
• Copy Paste: [[2401.15934]] HICH Image/Text (HICH-IT): Comprehensive Text and Image Datasets for Hypertensive Intracerebral Hemorrhage Research(https://arxiv.org/abs/2401.15934)
• Keywords: segmentation
• Abstract: In this paper, we introduce a new multimodal dataset in the medical field of hypertensive intracerebral hemorrhage(HICH), called as HICH-IT, which includes both textual information and head CT images. This dataset is designed to enhance the accuracy of artificial intelligence in the diagnosis and treatment of HICH. This dataset, built upon the foundation of standard text and image data, incorporates specific annotations within the text data, extracting key content from the text information, and categorizes the annotation content of imaging data into four types: brain midline, hematoma, left cerebral ventricle, and right cerebral ventricle. HICH-IT aims to be a foundational dataset for feature learning in image segmentation tasks and named entity recognition. To further understand the dataset, we have trained deep learning algorithms to observe the performance. The pretrained models have been released at both www.daip.club and github.com/Deep-AI-Application-DAIP. The dataset has been uploaded to https://github.com/CYBUS123456/HICH-IT-Datasets. Index Terms-HICH, Deep learning, Intraparenchymal hemorrhage, named entity recognition, novel dataset

Title: Self-Supervised Learning in Event Sequences: A Comparative Study and Hybrid Approach of Generative Modeling and Contrastive Learning

• Authors: Viktor Moskvoretskii, Dmitry Osin, Egor Shvetsov, Igor Udovichenko, Maxim Zhelnin, Andrey Dukhovny, Anna Zhimerikina, Albert Efimov, Evgeny Burnaev
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15935
• Pdf URL: https://arxiv.org/pdf/2401.15935
• Copy Paste: [[2401.15935]] Self-Supervised Learning in Event Sequences: A Comparative Study and Hybrid Approach of Generative Modeling and Contrastive Learning(https://arxiv.org/abs/2401.15935)
• Keywords: generative
• Abstract: This study investigates self-supervised learning techniques to obtain representations of Event Sequences. It is a key modality in various applications, including but not limited to banking, e-commerce, and healthcare. We perform a comprehensive study of generative and contrastive approaches in self-supervised learning, applying them both independently. We find that there is no single supreme method. Consequently, we explore the potential benefits of combining these approaches. To achieve this goal, we introduce a novel method that aligns generative and contrastive embeddings as distinct modalities, drawing inspiration from contemporary multimodal research. Generative and contrastive approaches are often treated as mutually exclusive, leaving a gap for their combined exploration. Our results demonstrate that this aligned model performs at least on par with, and mostly surpasses, existing methods and is more universal across a variety of tasks. Furthermore, we demonstrate that self-supervised methods consistently outperform the supervised approach on our datasets.

Title: AdvNF: Reducing Mode Collapse in Conditional Normalising Flows using Adversarial Learning

• Authors: Vikas Kanaujia, Mathias S. Scheurer, Vipul Arora
• Subjects: cs.LG, cond-mat.stat-mech, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2401.15948
• Pdf URL: https://arxiv.org/pdf/2401.15948
• Copy Paste: [[2401.15948]] AdvNF: Reducing Mode Collapse in Conditional Normalising Flows using Adversarial Learning(https://arxiv.org/abs/2401.15948)
• Keywords: generative
• Abstract: Deep generative models complement Markov-chain-Monte-Carlo methods for efficiently sampling from high-dimensional distributions. Among these methods, explicit generators, such as Normalising Flows (NFs), in combination with the Metropolis Hastings algorithm have been extensively applied to get unbiased samples from target distributions. We systematically study central problems in conditional NFs, such as high variance, mode collapse and data efficiency. We propose adversarial training for NFs to ameliorate these problems. Experiments are conducted with low-dimensional synthetic datasets and XY spin models in two spatial dimensions.

Title: Scalable Federated Unlearning via Isolated and Coded Sharding

• Authors: Yijing Lin, Zhipeng Gao, Hongyang Du, Dusit Niyato, Gui Gui, Shuguang Cui, Jinke Ren
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.15957
• Pdf URL: https://arxiv.org/pdf/2401.15957
• Copy Paste: [[2401.15957]] Scalable Federated Unlearning via Isolated and Coded Sharding(https://arxiv.org/abs/2401.15957)
• Keywords: attack, membership infer, federate
• Abstract: Federated unlearning has emerged as a promising paradigm to erase the client-level data effect without affecting the performance of collaborative learning models. However, the federated unlearning process often introduces extensive storage overhead and consumes substantial computational resources, thus hindering its implementation in practice. To address this issue, this paper proposes a scalable federated unlearning framework based on isolated sharding and coded computing. We first divide distributed clients into multiple isolated shards across stages to reduce the number of clients being affected. Then, to reduce the storage overhead of the central server, we develop a coded computing mechanism by compressing the model parameters across different shards. In addition, we provide the theoretical analysis of time efficiency and storage effectiveness for the isolated and coded sharding. Finally, extensive experiments on two typical learning tasks, i.e., classification and generation, demonstrate that our proposed framework can achieve better performance than three state-of-the-art frameworks in terms of accuracy, retraining time, storage overhead, and F1 scores for resisting membership inference attacks.

Title: Spatio-Temporal Attention Graph Neural Network for Remaining Useful Life Prediction

• Authors: Zhixin Huang, Yujiang He, Bernhard Sick
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15964
• Pdf URL: https://arxiv.org/pdf/2401.15964
• Copy Paste: [[2401.15964]] Spatio-Temporal Attention Graph Neural Network for Remaining Useful Life Prediction(https://arxiv.org/abs/2401.15964)
• Keywords: extraction, explainability
• Abstract: Remaining useful life prediction plays a crucial role in the health management of industrial systems. Given the increasing complexity of systems, data-driven predictive models have attracted significant research interest. Upon reviewing the existing literature, it appears that many studies either do not fully integrate both spatial and temporal features or employ only a single attention mechanism. Furthermore, there seems to be inconsistency in the choice of data normalization methods, particularly concerning operating conditions, which might influence predictive performance. To bridge these observations, this study presents the Spatio-Temporal Attention Graph Neural Network. Our model combines graph neural networks and temporal convolutional neural networks for spatial and temporal feature extraction, respectively. The cascade of these extractors, combined with multi-head attention mechanisms for both spatio-temporal dimensions, aims to improve predictive precision and refine model explainability. Comprehensive experiments were conducted on the C-MAPSS dataset to evaluate the impact of unified versus clustering normalization. The findings suggest that our model performs state-of-the-art results using only the unified normalization. Additionally, when dealing with datasets with multiple operating conditions, cluster normalization enhances the performance of our proposed model by up to 27%.

Title: Response Generation for Cognitive Behavioral Therapy with Large Language Models: Comparative Study with Socratic Questioning

• Authors: Kenta Izumi, Hiroki Tanaka, Kazuhiro Shidara, Hiroyoshi Adachi, Daisuke Kanayama, Takashi Kudo, Satoshi Nakamura
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15966
• Pdf URL: https://arxiv.org/pdf/2401.15966
• Copy Paste: [[2401.15966]] Response Generation for Cognitive Behavioral Therapy with Large Language Models: Comparative Study with Socratic Questioning(https://arxiv.org/abs/2401.15966)
• Keywords: transformer
• Abstract: Dialogue systems controlled by predefined or rule-based scenarios derived from counseling techniques, such as cognitive behavioral therapy (CBT), play an important role in mental health apps. Despite the need for responsible responses, it is conceivable that using the newly emerging LLMs to generate contextually relevant utterances will enhance these apps. In this study, we construct dialogue modules based on a CBT scenario focused on conventional Socratic questioning using two kinds of LLMs: a Transformer-based dialogue model further trained with a social media empathetic counseling dataset, provided by Osaka Prefecture (OsakaED), and GPT-4, a state-of-the art LLM created by OpenAI. By comparing systems that use LLM-generated responses with those that do not, we investigate the impact of generated responses on subjective evaluations such as mood change, cognitive change, and dialogue quality (e.g., empathy). As a result, no notable improvements are observed when using the OsakaED model. When using GPT-4, the amount of mood change, empathy, and other dialogue qualities improve significantly. Results suggest that GPT-4 possesses a high counseling ability. However, they also indicate that even when using a dialogue model trained with a human counseling dataset, it does not necessarily yield better outcomes compared to scenario-based dialogues. While presenting LLM-generated responses, including GPT-4, and having them interact directly with users in real-life mental health care services may raise ethical issues, it is still possible for human professionals to produce example responses or response templates using LLMs in advance in systems that use rules, scenarios, or example responses.

Title: HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inference

• Authors: Tianshi Xu, Meng Li, Runsheng Wang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15970
• Pdf URL: https://arxiv.org/pdf/2401.15970
• Copy Paste: [[2401.15970]] HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inference(https://arxiv.org/abs/2401.15970)
• Keywords: secure, security, privacy, protect, robust
• Abstract: Secure two-party computation with homomorphic encryption (HE) protects data privacy with a formal security guarantee but suffers from high communication overhead. While previous works, e.g., Cheetah, Iron, etc, have proposed efficient HE-based protocols for different neural network (NN) operations, they still assume high precision, e.g., fixed point 37 bit, for the NN operations and ignore NNs' native robustness against quantization error. In this paper, we propose HEQuant, which features low-precision-quantization-aware optimization for the HE-based protocols. We observe the benefit of a naive combination of quantization and HE quickly saturates as bit precision goes down. Hence, to further improve communication efficiency, we propose a series of optimizations, including an intra-coefficient packing algorithm and a quantization-aware tiling algorithm, to simultaneously reduce the number and precision of the transferred data. Compared with prior-art HE-based protocols, e.g., CrypTFlow2, Cheetah, Iron, etc, HEQuant achieves $3.5\sim 23.4\times$ communication reduction and $3.0\sim 9.3\times$ latency reduction. Meanwhile, when compared with prior-art network optimization frameworks, e.g., SENet, SNL, etc, HEQuant also achieves $3.1\sim 3.6\times$ communication reduction.

Title: StableIdentity: Inserting Anybody into Anywhere at First Sight

• Authors: Qinghe Wang, Xu Jia, Xiaomin Li, Taiqing Li, Liqian Ma, Yunzhi Zhuge, Huchuan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15975
• Pdf URL: https://arxiv.org/pdf/2401.15975
• Copy Paste: [[2401.15975]] StableIdentity: Inserting Anybody into Anywhere at First Sight(https://arxiv.org/abs/2401.15975)
• Keywords: diffusion
• Abstract: Recent advances in large pretrained text-to-image models have shown unprecedented capabilities for high-quality human-centric generation, however, customizing face identity is still an intractable problem. Existing methods cannot ensure stable identity preservation and flexible editability, even with several images for each subject during training. In this work, we propose StableIdentity, which allows identity-consistent recontextualization with just one face image. More specifically, we employ a face encoder with an identity prior to encode the input face, and then land the face representation into a space with an editable prior, which is constructed from celeb names. By incorporating identity prior and editability prior, the learned identity can be injected anywhere with various contexts. In addition, we design a masked two-phase diffusion loss to boost the pixel-level perception of the input face and maintain the diversity of generation. Extensive experiments demonstrate our method outperforms previous customization methods. In addition, the learned identity can be flexibly combined with the off-the-shelf modules such as ControlNet. Notably, to the best knowledge, we are the first to directly inject the identity learned from a single image into video/3D generation without finetuning. We believe that the proposed StableIdentity is an important step to unify image, video, and 3D customized generation models.

Title: Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling

• Authors: Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Da, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15977
• Pdf URL: https://arxiv.org/pdf/2401.15977
• Copy Paste: [[2401.15977]] Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling(https://arxiv.org/abs/2401.15977)
• Keywords: diffusion
• Abstract: We introduce Motion-I2V, a novel framework for consistent and controllable image-to-video generation (I2V). In contrast to previous methods that directly learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into two stages with explicit motion modeling. For the first stage, we propose a diffusion-based motion field predictor, which focuses on deducing the trajectories of the reference image's pixels. For the second stage, we propose motion-augmented temporal attention to enhance the limited 1-D temporal attention in video latent diffusion models. This module can effectively propagate reference image's feature to synthesized frames with the guidance of predicted trajectories from the first stage. Compared with existing methods, Motion-I2V can generate more consistent videos even at the presence of large motion and viewpoint variation. By training a sparse trajectory ControlNet for the first stage, Motion-I2V can support users to precisely control motion trajectories and motion regions with sparse trajectory and region annotations. This offers more controllability of the I2V process than solely relying on textual instructions. Additionally, Motion-I2V's second stage naturally supports zero-shot video-to-video translation. Both qualitative and quantitative comparisons demonstrate the advantages of Motion-I2V over prior approaches in consistent and controllable image-to-video generation.

Title: AccessLens: Auto-detecting Inaccessibility of Everyday Objects

• Authors: Nahyun Kwon, Qian Lu, Muhammad Hasham Qazi, Joanne Liu, Changhoon Oh, Shu Kong, Jeeeun Kim
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.15996
• Pdf URL: https://arxiv.org/pdf/2401.15996
• Copy Paste: [[2401.15996]] AccessLens: Auto-detecting Inaccessibility of Everyday Objects(https://arxiv.org/abs/2401.15996)
• Keywords: robust
• Abstract: In our increasingly diverse society, everyday physical interfaces often present barriers, impacting individuals across various contexts. This oversight, from small cabinet knobs to identical wall switches that can pose different contextual challenges, highlights an imperative need for solutions. Leveraging low-cost 3D-printed augmentations such as knob magnifiers and tactile labels seems promising, yet the process of discovering unrecognized barriers remains challenging because disability is context-dependent. We introduce AccessLens, an end-to-end system designed to identify inaccessible interfaces in daily objects, and recommend 3D-printable augmentations for accessibility enhancement. Our approach involves training a detector using the novel AccessDB dataset designed to automatically recognize 21 distinct Inaccessibility Classes (e.g., bar-small and round-rotate) within 6 common object categories (e.g., handle and knob). AccessMeta serves as a robust way to build a comprehensive dictionary linking these accessibility classes to open-source 3D augmentation designs. Experiments demonstrate our detector's performance in detecting inaccessible objects.

Title: LESSON: Multi-Label Adversarial False Data Injection Attack for Deep Learning Locational Detection

• Authors: Jiwei Tian, Chao Shen, Buhong Wang, Xiaofang Xia, Meng Zhang, Chenhao Lin, Qian Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16001
• Pdf URL: https://arxiv.org/pdf/2401.16001
• Copy Paste: [[2401.16001]] LESSON: Multi-Label Adversarial False Data Injection Attack for Deep Learning Locational Detection(https://arxiv.org/abs/2401.16001)
• Keywords: security, defense, attack
• Abstract: Deep learning methods can not only detect false data injection attacks (FDIA) but also locate attacks of FDIA. Although adversarial false data injection attacks (AFDIA) based on deep learning vulnerabilities have been studied in the field of single-label FDIA detection, the adversarial attack and defense against multi-label FDIA locational detection are still not involved. To bridge this gap, this paper first explores the multi-label adversarial example attacks against multi-label FDIA locational detectors and proposes a general multi-label adversarial attack framework, namely muLti-labEl adverSarial falSe data injectiON attack (LESSON). The proposed LESSON attack framework includes three key designs, namely Perturbing State Variables, Tailored Loss Function Design, and Change of Variables, which can help find suitable multi-label adversarial perturbations within the physical constraints to circumvent both Bad Data Detection (BDD) and Neural Attack Location (NAL). Four typical LESSON attacks based on the proposed framework and two dimensions of attack objectives are examined, and the experimental results demonstrate the effectiveness of the proposed attack framework, posing serious and pressing security concerns in smart grids.

Title: GPS: Graph Contrastive Learning via Multi-scale Augmented Views from Adversarial Pooling

• Authors: Wei Ju, Yiyang Gu, Zhengyang Mao, Ziyue Qiao, Yifang Qin, Xiao Luo, Hui Xiong, Ming Zhang
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.16011
• Pdf URL: https://arxiv.org/pdf/2401.16011
• Copy Paste: [[2401.16011]] GPS: Graph Contrastive Learning via Multi-scale Augmented Views from Adversarial Pooling(https://arxiv.org/abs/2401.16011)
• Keywords: robust
• Abstract: Self-supervised graph representation learning has recently shown considerable promise in a range of fields, including bioinformatics and social networks. A large number of graph contrastive learning approaches have shown promising performance for representation learning on graphs, which train models by maximizing agreement between original graphs and their augmented views (i.e., positive views). Unfortunately, these methods usually involve pre-defined augmentation strategies based on the knowledge of human experts. Moreover, these strategies may fail to generate challenging positive views to provide sufficient supervision signals. In this paper, we present a novel approach named Graph Pooling ContraSt (GPS) to address these issues. Motivated by the fact that graph pooling can adaptively coarsen the graph with the removal of redundancy, we rethink graph pooling and leverage it to automatically generate multi-scale positive views with varying emphasis on providing challenging positives and preserving semantics, i.e., strongly-augmented view and weakly-augmented view. Then, we incorporate both views into a joint contrastive learning framework with similarity learning and consistency learning, where our pooling module is adversarially trained with respect to the encoder for adversarial robustness. Experiments on twelve datasets on both graph classification and transfer learning tasks verify the superiority of the proposed method over its counterparts.

Title: Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in Vector-symbolic Architectures

• Authors: Michael Hersche, Francesco di Stefano, Thomas Hofmann, Abu Sebastian, Abbas Rahimi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16024
• Pdf URL: https://arxiv.org/pdf/2401.16024
• Copy Paste: [[2401.16024]] Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in Vector-symbolic Architectures(https://arxiv.org/abs/2401.16024)
• Keywords: large language model
• Abstract: Abstract reasoning is a cornerstone of human intelligence, and replicating it with artificial intelligence (AI) presents an ongoing challenge. This study focuses on efficiently solving Raven's progressive matrices (RPM), a visual test for assessing abstract reasoning abilities, by using distributed computation and operators provided by vector-symbolic architectures (VSA). Instead of hard-coding the rule formulations associated with RPMs, our approach can learn the VSA rule formulations (hence the name Learn-VRF) with just one pass through the training data. Yet, our approach, with compact parameters, remains transparent and interpretable. Learn-VRF yields accurate predictions on I-RAVEN's in-distribution data, and exhibits strong out-of-distribution capabilities concerning unseen attribute-rule pairs, significantly outperforming pure connectionist baselines including large language models. Our code is available at https://github.com/IBM/learn-vector-symbolic-architectures-rule-formulations.

Title: Second Order Kinematic Surface Fitting in Anatomical Structures

• Authors: Wilhelm Wimmer, Hervé Delingette
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.16035
• Pdf URL: https://arxiv.org/pdf/2401.16035
• Copy Paste: [[2401.16035]] Second Order Kinematic Surface Fitting in Anatomical Structures(https://arxiv.org/abs/2401.16035)
• Keywords: robust
• Abstract: Symmetry detection and morphological classification of anatomical structures play pivotal roles in medical image analysis. The application of kinematic surface fitting, a method for characterizing shapes through parametric stationary velocity fields, has shown promising results in computer vision and computer-aided design. However, existing research has predominantly focused on first order rotational velocity fields, which may not adequately capture the intricate curved and twisted nature of anatomical structures. To address this limitation, we propose an innovative approach utilizing a second order velocity field for kinematic surface fitting. This advancement accommodates higher rotational shape complexity and improves the accuracy of symmetry detection in anatomical structures. We introduce a robust fitting technique and validate its performance through testing on synthetic shapes and real anatomical structures. Our method not only enables the detection of curved rotational symmetries (core lines) but also facilitates morphological classification by deriving intrinsic shape parameters related to curvature and torsion. We illustrate the usefulness of our technique by categorizing the shape of human cochleae in terms of the intrinsic velocity field parameters. The results showcase the potential of our method as a valuable tool for medical image analysis, contributing to the assessment of complex anatomical shapes.

Title: Dynamic Prototype Adaptation with Distillation for Few-shot Point Cloud Segmentation

• Authors: Jie Liu, Wenzhe Yin, Haochen Wang, Yunlu CHen, Jan-Jakob Sonke, Efstratios Gavves
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16051
• Pdf URL: https://arxiv.org/pdf/2401.16051
• Copy Paste: [[2401.16051]] Dynamic Prototype Adaptation with Distillation for Few-shot Point Cloud Segmentation(https://arxiv.org/abs/2401.16051)
• Keywords: segmentation
• Abstract: Few-shot point cloud segmentation seeks to generate per-point masks for previously unseen categories, using only a minimal set of annotated point clouds as reference. Existing prototype-based methods rely on support prototypes to guide the segmentation of query point clouds, but they encounter challenges when significant object variations exist between the support prototypes and query features. In this work, we present dynamic prototype adaptation (DPA), which explicitly learns task-specific prototypes for each query point cloud to tackle the object variation problem. DPA achieves the adaptation through prototype rectification, aligning vanilla prototypes from support with the query feature distribution, and prototype-to-query attention, extracting task-specific context from query point clouds. Furthermore, we introduce a prototype distillation regularization term, enabling knowledge transfer between early-stage prototypes and their deeper counterparts during adaption. By iteratively applying these adaptations, we generate task-specific prototypes for accurate mask predictions on query point clouds. Extensive experiments on two popular benchmarks show that DPA surpasses state-of-the-art methods by a significant margin, e.g., 7.43\% and 6.39\% under the 2-way 1-shot setting on S3DIS and ScanNet, respectively. Code is available at https://github.com/jliu4ai/DPA.

Title: Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing

• Authors: Vilém Zouhar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16055
• Pdf URL: https://arxiv.org/pdf/2401.16055
• Copy Paste: [[2401.16055]] Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing(https://arxiv.org/abs/2401.16055)
• Keywords: attack, steal
• Abstract: In learning-based functionality stealing, the attacker is trying to build a local model based on the victim's outputs. The attacker has to make choices regarding the local model's architecture, optimization method and, specifically for NLP models, subword vocabulary, such as BPE. On the machine translation task, we explore (1) whether the choice of the vocabulary plays a role in model stealing scenarios and (2) if it is possible to extract the victim's vocabulary. We find that the vocabulary itself does not have a large effect on the local model's performance. Given gray-box model access, it is possible to collect the victim's vocabulary by collecting the outputs (detokenized subwords on the output). The results of the minimum effect of vocabulary choice are important more broadly for black-box knowledge distillation.

Title: Neuromorphic Valence and Arousal Estimation

• Authors: Lorenzo Berlincioni, Luca Cultrera, Federico Becattini, Alberto Del Bimbo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16058
• Pdf URL: https://arxiv.org/pdf/2401.16058
• Copy Paste: [[2401.16058]] Neuromorphic Valence and Arousal Estimation(https://arxiv.org/abs/2401.16058)
• Keywords: biometric
• Abstract: Recognizing faces and their underlying emotions is an important aspect of biometrics. In fact, estimating emotional states from faces has been tackled from several angles in the literature. In this paper, we follow the novel route of using neuromorphic data to predict valence and arousal values from faces. Due to the difficulty of gathering event-based annotated videos, we leverage an event camera simulator to create the neuromorphic counterpart of an existing RGB dataset. We demonstrate that not only training models on simulated data can still yield state-of-the-art results in valence-arousal estimation, but also that our trained models can be directly applied to real data without further training to address the downstream task of emotion recognition. In the paper we propose several alternative models to solve the task, both frame-based and video-based.

Title: Non-Fluent Synthetic Target-Language Data Improve Neural Machine Translation

• Authors: Víctor M. Sánchez-Cartagena, Miquel Esplà-Gomis, Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16086
• Pdf URL: https://arxiv.org/pdf/2401.16086
• Copy Paste: [[2401.16086]] Non-Fluent Synthetic Target-Language Data Improve Neural Machine Translation(https://arxiv.org/abs/2401.16086)
• Keywords: robust
• Abstract: When the amount of parallel sentences available to train a neural machine translation is scarce, a common practice is to generate new synthetic training samples from them. A number of approaches have been proposed to produce synthetic parallel sentences that are similar to those in the parallel data available. These approaches work under the assumption that non-fluent target-side synthetic training samples can be harmful and may deteriorate translation performance. Even so, in this paper we demonstrate that synthetic training samples with non-fluent target sentences can improve translation performance if they are used in a multilingual machine translation framework as if they were sentences in another language. We conducted experiments on ten low-resource and four high-resource translation tasks and found out that this simple approach consistently improves translation performance as compared to state-of-the-art methods for generating synthetic training samples similar to those found in corpora. Furthermore, this improvement is independent of the size of the original training corpus, the resulting systems are much more robust against domain shift and produce less hallucinations.

Title: Fairness in Algorithmic Recourse Through the Lens of Substantive Equality of Opportunity

• Authors: Andrew Bell, Joao Fonseca, Carlo Abrate, Francesco Bonchi, Julia Stoyanovich
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16088
• Pdf URL: https://arxiv.org/pdf/2401.16088
• Copy Paste: [[2401.16088]] Fairness in Algorithmic Recourse Through the Lens of Substantive Equality of Opportunity(https://arxiv.org/abs/2401.16088)
• Keywords: fair
• Abstract: Algorithmic recourse -- providing recommendations to those affected negatively by the outcome of an algorithmic system on how they can take action and change that outcome -- has gained attention as a means of giving persons agency in their interactions with artificial intelligence (AI) systems. Recent work has shown that even if an AI decision-making classifier is ``fair'' (according to some reasonable criteria), recourse itself may be unfair due to differences in the initial circumstances of individuals, compounding disparities for marginalized populations and requiring them to exert more effort than others. There is a need to define more methods and metrics for evaluating fairness in recourse that span a range of normative views of the world, and specifically those that take into account time. Time is a critical element in recourse because the longer it takes an individual to act, the more the setting may change due to model or data drift. This paper seeks to close this research gap by proposing two notions of fairness in recourse that are in normative alignment with substantive equality of opportunity, and that consider time. The first considers the (often repeated) effort individuals exert per successful recourse event, and the second considers time per successful recourse event. Building upon an agent-based framework for simulating recourse, this paper demonstrates how much effort is needed to overcome disparities in initial circumstances. We then proposes an intervention to improve the fairness of recourse by rewarding effort, and compare it to existing strategies.

Title: Federated unsupervised random forest for privacy-preserving patient stratification

• Authors: Bastian Pfeifer, Christel Sirocchi, Marcus D. Bloice, Markus Kreuzthaler, Martin Urschler
• Subjects: cs.LG, cs.AI, cs.CR, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2401.16094
• Pdf URL: https://arxiv.org/pdf/2401.16094
• Copy Paste: [[2401.16094]] Federated unsupervised random forest for privacy-preserving patient stratification(https://arxiv.org/abs/2401.16094)
• Keywords: privacy, federate, interpretability
• Abstract: In the realm of precision medicine, effective patient stratification and disease subtyping demand innovative methodologies tailored for multi-omics data. Clustering techniques applied to multi-omics data have become instrumental in identifying distinct subgroups of patients, enabling a finer-grained understanding of disease variability. This work establishes a powerful framework for advancing precision medicine through unsupervised random-forest-based clustering and federated computing. We introduce a novel multi-omics clustering approach utilizing unsupervised random-forests. The unsupervised nature of the random forest enables the determination of cluster-specific feature importance, unraveling key molecular contributors to distinct patient groups. Moreover, our methodology is designed for federated execution, a crucial aspect in the medical domain where privacy concerns are paramount. We have validated our approach on machine learning benchmark data sets as well as on cancer data from The Cancer Genome Atlas (TCGA). Our method is competitive with the state-of-the-art in terms of disease subtyping, but at the same time substantially improves the cluster interpretability. Experiments indicate that local clustering performance can be improved through federated computing.

Title: Flexible Parallel Neural Network Architecture Model for Early Prediction of Lithium Battery Life

• Authors: Lidang Jiang, Zhuoxiang Li, Changyan Hu, Qingsong Huang, Ge He
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2401.16102
• Pdf URL: https://arxiv.org/pdf/2401.16102
• Copy Paste: [[2401.16102]] Flexible Parallel Neural Network Architecture Model for Early Prediction of Lithium Battery Life(https://arxiv.org/abs/2401.16102)
• Keywords: interpretability
• Abstract: The early prediction of battery life (EPBL) is vital for enhancing the efficiency and extending the lifespan of lithium batteries. Traditional models with fixed architectures often encounter underfitting or overfitting issues due to the diverse data distributions in different EPBL tasks. An interpretable deep learning model of flexible parallel neural network (FPNN) is proposed, which includes an InceptionBlock, a 3D convolutional neural network (CNN), a 2D CNN, and a dual-stream network. The proposed model effectively extracts electrochemical features from video-like formatted data using the 3D CNN and achieves advanced multi-scale feature abstraction through the InceptionBlock. The FPNN can adaptively adjust the number of InceptionBlocks to flexibly handle tasks of varying complexity in EPBL. The test on the MIT dataset shows that the FPNN model achieves outstanding predictive accuracy in EPBL tasks, with MAPEs of 2.47%, 1.29%, 1.08%, and 0.88% when the input cyclic data volumes are 10, 20, 30, and 40, respectively. The interpretability of the FPNN is mainly reflected in its flexible unit structure and parameter selection: its diverse branching structure enables the model to capture features at different scales, thus allowing the machine to learn informative features. The approach presented herein provides an accurate, adaptable, and comprehensible solution for early life prediction of lithium batteries, opening new possibilities in the field of battery health monitoring.

Title: A 2D Sinogram-Based Approach to Defect Localization in Computed Tomography

• Authors: Yuzhong Zhou, Linda-Sophie Schneider, Fuxin Fan, Andreas Maier
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.16104
• Pdf URL: https://arxiv.org/pdf/2401.16104
• Copy Paste: [[2401.16104]] A 2D Sinogram-Based Approach to Defect Localization in Computed Tomography(https://arxiv.org/abs/2401.16104)
• Keywords: segmentation
• Abstract: The rise of deep learning has introduced a transformative era in the field of image processing, particularly in the context of computed tomography. Deep learning has made a significant contribution to the field of industrial Computed Tomography. However, many defect detection algorithms are applied directly to the reconstructed domain, often disregarding the raw sensor data. This paper shifts the focus to the use of sinograms. Within this framework, we present a comprehensive three-step deep learning algorithm, designed to identify and analyze defects within objects without resorting to image reconstruction. These three steps are defect segmentation, mask isolation, and defect analysis. We use a U-Net-based architecture for defect segmentation. Our method achieves the Intersection over Union of 92.02% on our simulated data, with an average position error of 1.3 pixels for defect detection on a 512-pixel-wide detector.

Title: Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosis

• Authors: Haochun Wang, Sendong Zhao, Zewen Qiang, Nuwa Xi, Bing Qin, Ting Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16107
• Pdf URL: https://arxiv.org/pdf/2401.16107
• Copy Paste: [[2401.16107]] Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosis(https://arxiv.org/abs/2401.16107)
• Keywords: large language model
• Abstract: Automatic diagnosis is a significant application of AI in healthcare, where diagnoses are generated based on the symptom description of patients. Previous works have approached this task directly by modeling the relationship between the normalized symptoms and all possible diseases. However, in the clinical diagnostic process, patients are initially consulted by a general practitioner and, if necessary, referred to specialists in specific domains for a more comprehensive evaluation. The final diagnosis often emerges from a collaborative consultation among medical specialist groups. Recently, large language models have shown impressive capabilities in natural language understanding. In this study, we adopt tuning-free LLM-based agents as medical practitioners and propose the Agent-derived Multi-Specialist Consultation (AMSC) framework to model the diagnosis process in the real world by adaptively fusing probability distributions of agents over potential diseases. Experimental results demonstrate the superiority of our approach compared with baselines. Notably, our approach requires significantly less parameter updating and training time, enhancing efficiency and practical utility. Furthermore, we delve into a novel perspective on the role of implicit symptoms within the context of automatic diagnosis.

Title: CIMIL-CRC: a clinically-informed multiple instance learning framework for patient-level colorectal cancer molecular subtypes classification from H\&E stained images

• Authors: Hadar Hezi, Matan Gelber, Alexander Balabanov, Yosef E. Maruvka, Moti Freiman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16131
• Pdf URL: https://arxiv.org/pdf/2401.16131
• Copy Paste: [[2401.16131]] CIMIL-CRC: a clinically-informed multiple instance learning framework for patient-level colorectal cancer molecular subtypes classification from H\&E stained images(https://arxiv.org/abs/2401.16131)
• Keywords: extraction
• Abstract: Treatment approaches for colorectal cancer (CRC) are highly dependent on the molecular subtype, as immunotherapy has shown efficacy in cases with microsatellite instability (MSI) but is ineffective for the microsatellite stable (MSS) subtype. There is promising potential in utilizing deep neural networks (DNNs) to automate the differentiation of CRC subtypes by analyzing Hematoxylin and Eosin (H\&E) stained whole-slide images (WSIs). Due to the extensive size of WSIs, Multiple Instance Learning (MIL) techniques are typically explored. However, existing MIL methods focus on identifying the most representative image patches for classification, which may result in the loss of critical information. Additionally, these methods often overlook clinically relevant information, like the tendency for MSI class tumors to predominantly occur on the proximal (right side) colon. We introduce `CIMIL-CRC', a DNN framework that: 1) solves the MSI/MSS MIL problem by efficiently combining a pre-trained feature extraction model with principal component analysis (PCA) to aggregate information from all patches, and 2) integrates clinical priors, particularly the tumor location within the colon, into the model to enhance patient-level classification accuracy. We assessed our CIMIL-CRC method using the average area under the curve (AUC) from a 5-fold cross-validation experimental setup for model development on the TCGA-CRC-DX cohort, contrasting it with a baseline patch-level classification, MIL-only approach, and Clinically-informed patch-level classification approach. Our CIMIL-CRC outperformed all methods (AUROC: $0.92\pm0.002$ (95\% CI 0.91-0.92), vs. $0.79\pm0.02$ (95\% CI 0.76-0.82), $0.86\pm0.01$ (95\% CI 0.85-0.88), and $0.87\pm0.01$ (95\% CI 0.86-0.88), respectively). The improvement was statistically significant.

Title: Spatial-Aware Latent Initialization for Controllable Image Generation

• Authors: Wenqiang Sun, Teng Li, Zehong Lin, Jun Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16157
• Pdf URL: https://arxiv.org/pdf/2401.16157
• Copy Paste: [[2401.16157]] Spatial-Aware Latent Initialization for Controllable Image Generation(https://arxiv.org/abs/2401.16157)
• Keywords: diffusion
• Abstract: Recently, text-to-image diffusion models have demonstrated impressive ability to generate high-quality images conditioned on the textual input. However, these models struggle to accurately adhere to textual instructions regarding spatial layout information. While previous research has primarily focused on aligning cross-attention maps with layout conditions, they overlook the impact of the initialization noise on the layout guidance. To achieve better layout control, we propose leveraging a spatial-aware initialization noise during the denoising process. Specifically, we find that the inverted reference image with finite inversion steps contains valuable spatial awareness regarding the object's position, resulting in similar layouts in the generated images. Based on this observation, we develop an open-vocabulary framework to customize a spatial-aware initialization noise for each layout condition. Without modifying other modules except the initialization noise, our approach can be seamlessly integrated as a plug-and-play module within other training-free layout guidance frameworks. We evaluate our approach quantitatively and qualitatively on the available Stable Diffusion model and COCO dataset. Equipped with the spatial-aware latent initialization, our method significantly improves the effectiveness of layout guidance while preserving high-quality content.

Title: Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception

• Authors: Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16158
• Pdf URL: https://arxiv.org/pdf/2401.16158
• Copy Paste: [[2401.16158]] Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception(https://arxiv.org/abs/2401.16158)
• Keywords: large language model
• Abstract: Mobile device agent based on Multimodal Large Language Models (MLLM) is becoming a popular application. In this paper, we introduce Mobile-Agent, an autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual perception tools to accurately identify and locate both the visual and textual elements within the app's front-end interface. Based on the perceived vision context, it then autonomously plans and decomposes the complex operation task, and navigates the mobile Apps through operations step by step. Different from previous solutions that rely on XML files of Apps or mobile system metadata, Mobile-Agent allows for greater adaptability across diverse mobile operating environments in a vision-centric way, thereby eliminating the necessity for system-specific customizations. To assess the performance of Mobile-Agent, we introduced Mobile-Eval, a benchmark for evaluating mobile device operations. Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent. The experimental results indicate that Mobile-Agent achieved remarkable accuracy and completion rates. Even with challenging instructions, such as multi-app operations, Mobile-Agent can still complete the requirements. Code and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.

Title: LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

• Authors: Shaoxiang Chen, Zequn Jie, Lin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16160
• Pdf URL: https://arxiv.org/pdf/2401.16160
• Copy Paste: [[2401.16160]] LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs(https://arxiv.org/abs/2401.16160)
• Keywords: transformer, large language model
• Abstract: Instruction finetuning on a variety of image-text instruction data is the key to obtaining a versatile Multimodal Large Language Model (MLLM), and different configurations of the instruction data can lead to finetuned models with different capabilities. However, we have discovered that data conflicts are inevitable when mixing instruction data from distinct domains, which can result in performance drops for tasks of a specific domain. To address this issue, we propose to apply a sparse mixture of LoRA experts for instruction finetuning MLLMs. Within the Transformer layers, we extend the popular Low-Rank Adaption (LoRA) method by creating a set of LoRA experts specifically for the MLP layer, and route each token to the top-1 expert based on a routing function, allowing adaptive choices for tokens from different domains. Since the LoRA experts are sparsely activated, the training and inference cost are kept roughly constant compared to the original LoRA method. By replacing the plain-LoRA finetuing of LLaVA-1.5, our final model is named LLaVA-MoLE. Extensive experiments proved that LLaVA-MoLE effectively mitigates the data conflict issue when mixing multiple distinct instruction datasets with various configurations, and achieves consistent performance gains over the strong plain-LoRA baselines. Most importantly, on the mixed datasets, LLaVA-MoLE can even outperform the plain-LoRA baseline trained with twice the samples.

Title: A Privacy-preserving key transmission protocol to distribute QRNG keys using zk-SNARKs

• Authors: David Soler (1), Carlos Dafonte (1), Manuel Fernández-Veiga (2), Ana Fernández Vilas (2), Francisco J. Nóvoa (1) ((1) CITIC, Universidade da Coruňa, A Coruňa, Spain, (2) atlanTTic, Universidade de Vigo, Vigo, Spain)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16170
• Pdf URL: https://arxiv.org/pdf/2401.16170
• Copy Paste: [[2401.16170]] A Privacy-preserving key transmission protocol to distribute QRNG keys using zk-SNARKs(https://arxiv.org/abs/2401.16170)
• Keywords: security, privacy
• Abstract: High-entropy random numbers are an essential part of cryptography, and Quantum Random Number Generators (QRNG) are an emergent technology that can provide high-quality keys for cryptographic algorithms but unfortunately are currently difficult to access. Existing Entropy-as-a-Service solutions require users to trust the central authority distributing the key material, which is not desirable in a high-privacy environment. In this paper, we present a novel key transmission protocol that allows users to obtain cryptographic material generated by a QRNG in such a way that the server is unable to identify which user is receiving each key. This is achieved with the inclusion of Zero Knowledge Succinct Non-interactive Arguments of Knowledge (zk-SNARK), a cryptographic primitive that allow users to prove knowledge of some value without needing to reveal it. The security analysis of the protocol proves that it satisfies the properties of Anonymity, Unforgeability and Confidentiality, as defined in this document. We also provide an implementation of the protocol demonstrating its functionality and performance, using NFC as the transmission channel for the QRNG key.

Title: A Survey on Structure-Preserving Graph Transformers

• Authors: Van Thuy Hoang, O-Joun Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16176
• Pdf URL: https://arxiv.org/pdf/2401.16176
• Copy Paste: [[2401.16176]] A Survey on Structure-Preserving Graph Transformers(https://arxiv.org/abs/2401.16176)
• Keywords: transformer
• Abstract: The transformer architecture has shown remarkable success in various domains, such as natural language processing and computer vision. When it comes to graph learning, transformers are required not only to capture the interactions between pairs of nodes but also to preserve graph structures connoting the underlying relations and proximity between them, showing the expressive power to capture different graph structures. Accordingly, various structure-preserving graph transformers have been proposed and widely used for various tasks, such as graph-level tasks in bioinformatics and chemoinformatics. However, strategies related to graph structure preservation have not been well organized and systematized in the literature. In this paper, we provide a comprehensive overview of structure-preserving graph transformers and generalize these methods from the perspective of their design objective. First, we divide strategies into four main groups: node feature modulation, context node sampling, graph rewriting, and transformer architecture improvements. We then further divide the strategies according to the coverage and goals of graph structure preservation. Furthermore, we also discuss challenges and future directions for graph transformer models to preserve the graph structure and understand the nature of graphs.

Title: LLaMandement: Large Language Models for Summarization of French Legislative Proposals

• Authors: Joseph Gesnouin, Yannis Tannier, Christophe Gomes Da Silva, Hatim Tapory, Camille Brier, Hugo Simon, Raphael Rozenberg, Hermann Woehrel, Mehdi El Yakaabi, Thomas Binder, Guillaume Marie, Emilie Caron, Mathile Nogueira, Thomas Fontas, Laure Puydebois, Marie Theophile, Stephane Morandi, Mael Petit, David Creissac, Pauline Ennouchy, Elise Valetoux, Celine Visade, Severine Balloux, Emmanuel Cortes, Pierre-Etienne Devineau, Ulrich Tan, Esther Mac Namara, Su Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16182
• Pdf URL: https://arxiv.org/pdf/2401.16182
• Copy Paste: [[2401.16182]] LLaMandement: Large Language Models for Summarization of French Legislative Proposals(https://arxiv.org/abs/2401.16182)
• Keywords: robust, large language model
• Abstract: This report introduces LLaMandement, a state-of-the-art Large Language Model, fine-tuned by the French government and designed to enhance the efficiency and efficacy of processing parliamentary sessions (including the production of bench memoranda and documents required for interministerial meetings) by generating neutral summaries of legislative proposals. Addressing the administrative challenges of manually processing a growing volume of legislative amendments, LLaMandement stands as a significant legal technological milestone, providing a solution that exceeds the scalability of traditional human efforts while matching the robustness of a specialized legal drafter. We release all our fine-tuned models and training data to the community.

Title: On the Semantics of LM Latent Space: A Vocabulary-defined Approach

• Authors: Jian Gu, Chunyang Chen, Aldeida Aleti
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16184
• Pdf URL: https://arxiv.org/pdf/2401.16184
• Copy Paste: [[2401.16184]] On the Semantics of LM Latent Space: A Vocabulary-defined Approach(https://arxiv.org/abs/2401.16184)
• Keywords: interpretability, transformer
• Abstract: In the realm of deep learning, understanding the latent space of language models (LMs) like transformers is crucial for refining their performance and interpretability. However, existing analyses often fall short in providing absolute and model-centric insights into LM semantics, and neglect essential aspects of LM adaption. In response, we introduce a pioneering method called vocabulary-defined semantics, which establishes a fixed reference frame within the LM latent space, ensuring absolute semantic analysis grounded in LM vocabulary. Our approach transcends prior relative analyses, leveraging LM vocabulary for model-centric insights. Furthermore, we propose a novel technique to compute logits, emphasizing differentiability and local isotropy, and introduce a neural clustering module for semantically calibrating data representations during LM adaptation. Through extensive experiments across diverse text understanding datasets, our approach surpasses state-of-the-art methods of retrieval-augmented generation and parameters-efficient finetuning, showcasing its efficacy and broad applicability. Our findings not only shed light on LM mechanics but also offer practical solutions for enhancing LM performance and interpretability.

Title: LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning

• Authors: Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Miaolei Shi, Yang Liu
• Subjects: cs.CR, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.16185
• Pdf URL: https://arxiv.org/pdf/2401.16185
• Copy Paste: [[2401.16185]] LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning(https://arxiv.org/abs/2401.16185)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated significant poten- tial for many downstream tasks, including those requiring human- level intelligence, such as vulnerability detection. However, recent attempts to use LLMs for vulnerability detection are still prelim- inary, as they lack an in-depth understanding of a subject LLM's vulnerability reasoning capability - whether it originates from the model itself or from external assistance, such as invoking tool sup- port and retrieving vulnerability knowledge. In this paper, we aim to decouple LLMs' vulnerability reason- ing capability from their other capabilities, including the ability to actively seek additional information (e.g., via function calling in SOTA models), adopt relevant vulnerability knowledge (e.g., via vector-based matching and retrieval), and follow instructions to out- put structured results. To this end, we propose a unified evaluation framework named LLM4Vuln, which separates LLMs' vulnerability reasoning from their other capabilities and evaluates how LLMs' vulnerability reasoning could be enhanced when combined with the enhancement of other capabilities. To demonstrate the effectiveness of LLM4Vuln, we have designed controlled experiments using 75 ground-truth smart contract vulnerabilities, which were extensively audited as high-risk on Code4rena from August to November 2023, and tested them in 4,950 different scenarios across three represen- tative LLMs (GPT-4, Mixtral, and Code Llama). Our results not only reveal ten findings regarding the varying effects of knowledge en- hancement, context supplementation, prompt schemes, and models but also enable us to identify 9 zero-day vulnerabilities in two pilot bug bounty programs with over 1,000 USD being awarded.

Title: Geospatial Disparities: A Case Study on Real Estate Prices in Paris

• Authors: Agathe Fernandes Machado, François Hu, Philipp Ratz, Ewen Gallic, Arthur Charpentier
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.16197
• Pdf URL: https://arxiv.org/pdf/2401.16197
• Copy Paste: [[2401.16197]] Geospatial Disparities: A Case Study on Real Estate Prices in Paris(https://arxiv.org/abs/2401.16197)
• Keywords: fair
• Abstract: Driven by an increasing prevalence of trackers, ever more IoT sensors, and the declining cost of computing power, geospatial information has come to play a pivotal role in contemporary predictive models. While enhancing prognostic performance, geospatial data also has the potential to perpetuate many historical socio-economic patterns, raising concerns about a resurgence of biases and exclusionary practices, with their disproportionate impacts on society. Addressing this, our paper emphasizes the crucial need to identify and rectify such biases and calibration errors in predictive models, particularly as algorithms become more intricate and less interpretable. The increasing granularity of geospatial information further introduces ethical concerns, as choosing different geographical scales may exacerbate disparities akin to redlining and exclusionary zoning. To address these issues, we propose a toolkit for identifying and mitigating biases arising from geospatial data. Extending classical fairness definitions, we incorporate an ordinal regression case with spatial attributes, deviating from the binary classification focus. This extension allows us to gauge disparities stemming from data aggregation levels and advocates for a less interfering correction approach. Illustrating our methodology using a Parisian real estate dataset, we showcase practical applications and scrutinize the implications of choosing geographical aggregation levels for fairness and calibration measures.

Title: Diffutoon: High-Resolution Editable Toon Shading via Diffusion Models

• Authors: Zhongjie Duan, Chengyu Wang, Cen Chen, Weining Qian, Jun Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16224
• Pdf URL: https://arxiv.org/pdf/2401.16224
• Copy Paste: [[2401.16224]] Diffutoon: High-Resolution Editable Toon Shading via Diffusion Models(https://arxiv.org/abs/2401.16224)
• Keywords: diffusion
• Abstract: Toon shading is a type of non-photorealistic rendering task of animation. Its primary purpose is to render objects with a flat and stylized appearance. As diffusion models have ascended to the forefront of image synthesis methodologies, this paper delves into an innovative form of toon shading based on diffusion models, aiming to directly render photorealistic videos into anime styles. In video stylization, extant methods encounter persistent challenges, notably in maintaining consistency and achieving high visual quality. In this paper, we model the toon shading problem as four subproblems: stylization, consistency enhancement, structure guidance, and colorization. To address the challenges in video stylization, we propose an effective toon shading approach called \textit{Diffutoon}. Diffutoon is capable of rendering remarkably detailed, high-resolution, and extended-duration videos in anime style. It can also edit the content according to prompts via an additional branch. The efficacy of Diffutoon is evaluated through quantitive metrics and human evaluation. Notably, Diffutoon surpasses both open-source and closed-source baseline approaches in our experiments. Our work is accompanied by the release of both the source code and example videos on Github (Project page: https://ecnu-cilab.github.io/DiffutoonProjectPage/).

Title: Cross-Database Liveness Detection: Insights from Comparative Biometric Analysis

• Authors: Oleksandr Kuznetsov, Dmytro Zakharov, Emanuele Frontoni, Andrea Maranesi, Serhii Bohucharskyi
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16232
• Pdf URL: https://arxiv.org/pdf/2401.16232
• Copy Paste: [[2401.16232]] Cross-Database Liveness Detection: Insights from Comparative Biometric Analysis(https://arxiv.org/abs/2401.16232)
• Keywords: security, biometric
• Abstract: In an era where biometric security serves as a keystone of modern identity verification systems, ensuring the authenticity of these biometric samples is paramount. Liveness detection, the capability to differentiate between genuine and spoofed biometric samples, stands at the forefront of this challenge. This research presents a comprehensive evaluation of liveness detection models, with a particular focus on their performance in cross-database scenarios, a test paradigm notorious for its complexity and real-world relevance. Our study commenced by meticulously assessing models on individual datasets, revealing the nuances in their performance metrics. Delving into metrics such as the Half Total Error Rate, False Acceptance Rate, and False Rejection Rate, we unearthed invaluable insights into the models' strengths and weaknesses. Crucially, our exploration of cross-database testing provided a unique perspective, highlighting the chasm between training on one dataset and deploying on another. Comparative analysis with extant methodologies, ranging from convolutional networks to more intricate strategies, enriched our understanding of the current landscape. The variance in performance, even among state-of-the-art models, underscored the inherent challenges in this domain. In essence, this paper serves as both a repository of findings and a clarion call for more nuanced, data-diverse, and adaptable approaches in biometric liveness detection. In the dynamic dance between authenticity and deception, our work offers a blueprint for navigating the evolving rhythms of biometric security.

Title: DAEDALUS: Defense Against Firmware ROP Exploits Using Stochastic Software Diversity

• Authors: Islam Obaidat, Meera Sridhar, Fatemeh Tavakoli
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16234
• Pdf URL: https://arxiv.org/pdf/2401.16234
• Copy Paste: [[2401.16234]] DAEDALUS: Defense Against Firmware ROP Exploits Using Stochastic Software Diversity(https://arxiv.org/abs/2401.16234)
• Keywords: protect, defense, attack
• Abstract: This paper presents DAEDALUS, a software diversity-based framework designed to resist ROP attacks on Linux-based IoT devices. DAEDALUS generates unique, semantically equivalent but syntactically different rewrites of IoT firmware, disrupting large-scale replication of ROP attacks. DAEDALUS employs STOKE, a stochastic optimizer for x86 binaries, as its core diversity engine but introduces significant extensions to address unique IoT firmware challenges. DAEDALUS's effectiveness is evaluated using DDoSim, a published botnet DDoS attack simulation testbed. Results demonstrate that DAEDALUS successfully neutralizes ROP payloads by diversifying critical basic blocks in the firmware, preventing attackers from compromising multiple devices for DDoS attacks via memory error vulnerabilities. The findings indicate that DAEDALUS not only mitigates the impact of ROP attacks on individual IoT devices through probabilistic protection but also thwarts large-scale ROP attacks across multiple devices.

Title: Towards Red Teaming in Multimodal and Multilingual Translation

• Authors: Christophe Ropers, David Dale, Prangthip Hansanti, Gabriel Mejia Gonzalez, Ivan Evtimov, Corinne Wong, Christophe Touret, Kristina Pereyra, Seohyun Sonia Kim, Cristian Canton Ferrer, Pierre Andrews, Marta R. Costa-jussà
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.16247
• Pdf URL: https://arxiv.org/pdf/2401.16247
• Copy Paste: [[2401.16247]] Towards Red Teaming in Multimodal and Multilingual Translation(https://arxiv.org/abs/2401.16247)
• Keywords: generative
• Abstract: Assessing performance in Natural Language Processing is becoming increasingly complex. One particular challenge is the potential for evaluation datasets to overlap with training data, either directly or indirectly, which can lead to skewed results and overestimation of model performance. As a consequence, human evaluation is gaining increasing interest as a means to assess the performance and reliability of models. One such method is the red teaming approach, which aims to generate edge cases where a model will produce critical errors. While this methodology is becoming standard practice for generative AI, its application to the realm of conditional AI remains largely unexplored. This paper presents the first study on human-based red teaming for Machine Translation (MT), marking a significant step towards understanding and improving the performance of translation models. We delve into both human-based red teaming and a study on automation, reporting lessons learned and providing recommendations for both translation models and red teaming drills. This pioneering work opens up new avenues for research and development in the field of MT.

Title: Cross-silo Federated Learning with Record-level Personalized Differential Privacy

• Authors: Junxu Liu, Jian Lou, Li Xiong, Jinfei Liu, Xiaofeng Meng
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16251
• Pdf URL: https://arxiv.org/pdf/2401.16251
• Copy Paste: [[2401.16251]] Cross-silo Federated Learning with Record-level Personalized Differential Privacy(https://arxiv.org/abs/2401.16251)
• Keywords: privacy, protect, federate
• Abstract: Federated learning enhanced by differential privacy has emerged as a popular approach to better safeguard the privacy of client-side data by protecting clients' contributions during the training process. Existing solutions typically assume a uniform privacy budget for all records and provide one-size-fits-all solutions that may not be adequate to meet each record's privacy requirement. In this paper, we explore the uncharted territory of cross-silo FL with record-level personalized differential privacy. We devise a novel framework named rPDP-FL, employing a two-stage hybrid sampling scheme with both client-level sampling and non-uniform record-level sampling to accommodate varying privacy requirements. A critical and non-trivial problem is to select the ideal per-record sampling probability q given the personalized privacy budget {\epsilon}. We introduce a versatile solution named Simulation-CurveFitting, allowing us to uncover a significant insight into the nonlinear correlation between q and {\epsilon} and derive an elegant mathematical model to tackle the problem. Our evaluation demonstrates that our solution can provide significant performance gains over the baselines that do not consider personalized privacy preservation.

Title: CO2: Efficient Distributed Training with Full Communication-Computation Overlap

• Authors: Weigao Sun, Zhen Qin, Weixuan Sun, Shidi Li, Dong Li, Xuyang Shen, Yu Qiao, Yiran Zhong
• Subjects: cs.CL, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.16265
• Pdf URL: https://arxiv.org/pdf/2401.16265
• Copy Paste: [[2401.16265]] CO2: Efficient Distributed Training with Full Communication-Computation Overlap(https://arxiv.org/abs/2401.16265)
• Keywords: large language model
• Abstract: The fundamental success of large language models hinges upon the efficacious implementation of large-scale distributed training techniques. Nevertheless, building a vast, high-performance cluster featuring high-speed communication interconnectivity is prohibitively costly, and accessible only to prominent entities. In this work, we aim to lower this barrier and democratize large-scale training with limited bandwidth clusters. We propose a new approach called CO2 that introduces local-updating and asynchronous communication to the distributed data-parallel training, thereby facilitating the full overlap of COmunication with COmputation. CO2 is able to attain a high scalability even on extensive multi-node clusters constrained by very limited communication bandwidth. We further propose the staleness gap penalty and outer momentum clipping techniques together with CO2 to bolster its convergence and training stability. Besides, CO2 exhibits seamless integration with well-established ZeRO-series optimizers which mitigate memory consumption of model states with large model training. We also provide a mathematical proof of convergence, accompanied by the establishment of a stringent upper bound. Furthermore, we validate our findings through an extensive set of practical experiments encompassing a wide range of tasks in the fields of computer vision and natural language processing. These experiments serve to demonstrate the capabilities of CO2 in terms of convergence, generalization, and scalability when deployed across configurations comprising up to 128 A100 GPUs. The outcomes emphasize the outstanding capacity of CO2 to hugely improve scalability, no matter on clusters with 800Gbps RDMA or 80Gbps TCP/IP inter-node connections.

Title: Cutup and Detect: Human Fall Detection on Cutup Untrimmed Videos Using a Large Foundational Video Understanding Model

• Authors: Till Grutschus, Ola Karrar, Emir Esenov, Ekta Vats
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16280
• Pdf URL: https://arxiv.org/pdf/2401.16280
• Copy Paste: [[2401.16280]] Cutup and Detect: Human Fall Detection on Cutup Untrimmed Videos Using a Large Foundational Video Understanding Model(https://arxiv.org/abs/2401.16280)
• Keywords: transformer
• Abstract: This work explores the performance of a large video understanding foundation model on the downstream task of human fall detection on untrimmed video and leverages a pretrained vision transformer for multi-class action detection, with classes: "Fall", "Lying" and "Other/Activities of daily living (ADL)". A method for temporal action localization that relies on a simple cutup of untrimmed videos is demonstrated. The methodology includes a preprocessing pipeline that converts datasets with timestamp action annotations into labeled datasets of short action clips. Simple and effective clip-sampling strategies are introduced. The effectiveness of the proposed method has been empirically evaluated on the publicly available High-Quality Fall Simulation Dataset (HQFSD). The experimental results validate the performance of the proposed pipeline. The results are promising for real-time application, and the falls are detected on video level with a state-of-the-art 0.96 F1 score on the HQFSD dataset under the given experimental settings. The source code will be made available on GitHub.

Title: Leveraging Positional Encoding for Robust Multi-Reference-Based Object 6D Pose Estimation

• Authors: Jaewoo Park, Jaeguk Kim, Nam Ik Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16284
• Pdf URL: https://arxiv.org/pdf/2401.16284
• Copy Paste: [[2401.16284]] Leveraging Positional Encoding for Robust Multi-Reference-Based Object 6D Pose Estimation(https://arxiv.org/abs/2401.16284)
• Keywords: robust
• Abstract: Accurately estimating the pose of an object is a crucial task in computer vision and robotics. There are two main deep learning approaches for this: geometric representation regression and iterative refinement. However, these methods have some limitations that reduce their effectiveness. In this paper, we analyze these limitations and propose new strategies to overcome them. To tackle the issue of blurry geometric representation, we use positional encoding with high-frequency components for the object's 3D coordinates. To address the local minimum problem in refinement methods, we introduce a normalized image plane-based multi-reference refinement strategy that's independent of intrinsic matrix constraints. Lastly, we utilize adaptive instance normalization and a simple occlusion augmentation method to help our model concentrate on the target object. Our experiments on Linemod, Linemod-Occlusion, and YCB-Video datasets demonstrate that our approach outperforms existing methods. We will soon release the code.

Title: Capturing Pertinent Symbolic Features for Enhanced Content-Based Misinformation Detection

• Authors: Flavio Merenda, José Manuel Gómez-Pérez
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16285
• Pdf URL: https://arxiv.org/pdf/2401.16285
• Copy Paste: [[2401.16285]] Capturing Pertinent Symbolic Features for Enhanced Content-Based Misinformation Detection(https://arxiv.org/abs/2401.16285)
• Keywords: robust
• Abstract: Preventing the spread of misinformation is challenging. The detection of misleading content presents a significant hurdle due to its extreme linguistic and domain variability. Content-based models have managed to identify deceptive language by learning representations from textual data such as social media posts and web articles. However, aggregating representative samples of this heterogeneous phenomenon and implementing effective real-world applications is still elusive. Based on analytical work on the language of misinformation, this paper analyzes the linguistic attributes that characterize this phenomenon and how representative of such features some of the most popular misinformation datasets are. We demonstrate that the appropriate use of pertinent symbolic knowledge in combination with neural language models is helpful in detecting misleading content. Our results achieve state-of-the-art performance in misinformation datasets across the board, showing that our approach offers a valid and robust alternative to multi-task transfer learning without requiring any additional training data. Furthermore, our results show evidence that structured knowledge can provide the extra boost required to address a complex and unpredictable real-world problem like misinformation detection, not only in terms of accuracy but also time efficiency and resource utilization.

Title: Breaking the Barrier: Selective Uncertainty-based Active Learning for Medical Image Segmentation

• Authors: Siteng Ma, Haochang Wu, Aonghus Lawlor, Ruihai Dong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16298
• Pdf URL: https://arxiv.org/pdf/2401.16298
• Copy Paste: [[2401.16298]] Breaking the Barrier: Selective Uncertainty-based Active Learning for Medical Image Segmentation(https://arxiv.org/abs/2401.16298)
• Keywords: segmentation
• Abstract: Active learning (AL) has found wide applications in medical image segmentation, aiming to alleviate the annotation workload and enhance performance. Conventional uncertainty-based AL methods, such as entropy and Bayesian, often rely on an aggregate of all pixel-level metrics. However, in imbalanced settings, these methods tend to neglect the significance of target regions, eg., lesions, and tumors. Moreover, uncertainty-based selection introduces redundancy. These factors lead to unsatisfactory performance, and in many cases, even underperform random sampling. To solve this problem, we introduce a novel approach called the Selective Uncertainty-based AL, avoiding the conventional practice of summing up the metrics of all pixels. Through a filtering process, our strategy prioritizes pixels within target areas and those near decision boundaries. This resolves the aforementioned disregard for target areas and redundancy. Our method showed substantial improvements across five different uncertainty-based methods and two distinct datasets, utilizing fewer labeled data to reach the supervised baseline and consistently achieving the highest overall performance. Our code is available at https://github.com/HelenMa9998/Selective\_Uncertainty\_AL.

Title: Quantum-safe Encryption: A New Method to Reduce Complexity and/or Improve Security Level

• Authors: Amir K. Khandani
• Subjects: cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2401.16302
• Pdf URL: https://arxiv.org/pdf/2401.16302
• Copy Paste: [[2401.16302]] Quantum-safe Encryption: A New Method to Reduce Complexity and/or Improve Security Level(https://arxiv.org/abs/2401.16302)
• Keywords: security, attack
• Abstract: This work presents some novel techniques to enhance an encryption scheme motivated by classical McEliece cryptosystem. Contributions include: (1) using masking matrices to hide sensitive data, (2) allowing both legitimate parties to incorporate randomness in the public key without sharing any additional public information, (3) using concatenation of a repetition code for error correction, permitting key recovery with a negligible decoding complexity, (4) making attacks more difficult by increasing the complexity in verifying a given key candidate has resulted in the actual key, (5) introducing memory in the error sequence such that: (i) error vector is composed of a random number of erroneous bits, (ii) errors can be all corrected when used in conjunction with concatenation of a repetition code of length 3. Proposed techniques allow generating significantly larger keys, at the same time, with a much lower complexity, as compared to known post-quantum key generation techniques relying on randomization.

Title: Regressing Transformers for Data-efficient Visual Place Recognition

• Authors: María Leyva-Vallina, Nicola Strisciuglio, Nicolai Petkov
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16304
• Pdf URL: https://arxiv.org/pdf/2401.16304
• Copy Paste: [[2401.16304]] Regressing Transformers for Data-efficient Visual Place Recognition(https://arxiv.org/abs/2401.16304)
• Keywords: transformer
• Abstract: Visual place recognition is a critical task in computer vision, especially for localization and navigation systems. Existing methods often rely on contrastive learning: image descriptors are trained to have small distance for similar images and larger distance for dissimilar ones in a latent space. However, this approach struggles to ensure accurate distance-based image similarity representation, particularly when training with binary pairwise labels, and complex re-ranking strategies are required. This work introduces a fresh perspective by framing place recognition as a regression problem, using camera field-of-view overlap as similarity ground truth for learning. By optimizing image descriptors to align directly with graded similarity labels, this approach enhances ranking capabilities without expensive re-ranking, offering data-efficient training and strong generalization across several benchmark datasets.

Title: Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets

• Authors: Nikita Moghe, Arnisa Fazla, Chantal Amrhein, Tom Kocmi, Mark Steedman, Alexandra Birch, Rico Sennrich, Liane Guillou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16313
• Pdf URL: https://arxiv.org/pdf/2401.16313
• Copy Paste: [[2401.16313]] Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets(https://arxiv.org/abs/2401.16313)
• Keywords: large language model
• Abstract: Recent machine translation (MT) metrics calibrate their effectiveness by correlating with human judgement but without any insights about their behaviour across different error types. Challenge sets are used to probe specific dimensions of metric behaviour but there are very few such datasets and they either focus on a limited number of phenomena or a limited number of language pairs. We introduce ACES, a contrastive challenge set spanning 146 language pairs, aimed at discovering whether metrics can identify 68 translation accuracy errors. These phenomena range from simple alterations at the word/character level to more complex errors based on discourse and real-world knowledge. We conduct a large-scale study by benchmarking ACES on 50 metrics submitted to the WMT 2022 and 2023 metrics shared tasks. We benchmark metric performance, assess their incremental performance over successive campaigns, and measure their sensitivity to a range of linguistic phenomena. We also investigate claims that Large Language Models (LLMs) are effective as MT evaluators by evaluating on ACES. Our results demonstrate that different metric families struggle with different phenomena and that LLM-based methods fail to demonstrate reliable performance. Our analyses indicate that most metrics ignore the source sentence, tend to prefer surface-level overlap and end up incorporating properties of base models which are not always beneficial. We expand ACES to include error span annotations, denoted as SPAN-ACES and we use this dataset to evaluate span-based error metrics showing these metrics also need considerable improvement. Finally, we provide a set of recommendations for building better MT metrics, including focusing on error labels instead of scores, ensembling, designing strategies to explicitly focus on the source sentence, focusing on semantic content and choosing the right base model for representations.

Title: Tradeoffs Between Alignment and Helpfulness in Language Models

• Authors: Yotam Wolf, Noam Wies, Dorin Shteyman, Binyamin Rothberg, Yoav Levine, Amnon Shashua
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16332
• Pdf URL: https://arxiv.org/pdf/2401.16332
• Copy Paste: [[2401.16332]] Tradeoffs Between Alignment and Helpfulness in Language Models(https://arxiv.org/abs/2401.16332)
• Keywords: attack
• Abstract: Language model alignment has become an important component of AI safety, allowing safe interactions between humans and language models, by enhancing desired behaviors and inhibiting undesired ones. It is often done by tuning the model or inserting preset aligning prompts. Recently, representation engineering, a method which alters the model's behavior via changing its representations post-training, was shown to be effective in aligning LLMs (Zou et al., 2023a). Representation engineering yields gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases, but was also shown to cause a decrease in the ability of the model to perform basic tasks. In this paper we study the tradeoff between the increase in alignment and decrease in helpfulness of the model. We propose a theoretical framework which provides bounds for these two quantities, and demonstrate their relevance empirically. Interestingly, we find that while the helpfulness generally decreases, it does so quadratically with the norm of the representation engineering vector, while the alignment increases linearly with it, indicating a regime in which it is efficient to use representation engineering. We validate our findings empirically, and chart the boundaries to the usefulness of representation engineering for alignment.

Title: FedFair^3: Unlocking Threefold Fairness in Federated Learning

• Authors: Simin Javaherian, Sanjeev Panta, Shelby Williams, Md Sirajul Islam, Li Chen
• Subjects: cs.LG, cs.AI, cs.CY, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.16350
• Pdf URL: https://arxiv.org/pdf/2401.16350
• Copy Paste: [[2401.16350]] FedFair^3: Unlocking Threefold Fairness in Federated Learning(https://arxiv.org/abs/2401.16350)
• Keywords: federate, fair
• Abstract: Federated Learning (FL) is an emerging paradigm in machine learning without exposing clients' raw data. In practical scenarios with numerous clients, encouraging fair and efficient client participation in federated learning is of utmost importance, which is also challenging given the heterogeneity in data distribution and device properties. Existing works have proposed different client-selection methods that consider fairness; however, they fail to select clients with high utilities while simultaneously achieving fair accuracy levels. In this paper, we propose a fair client-selection approach that unlocks threefold fairness in federated learning. In addition to having a fair client-selection strategy, we enforce an equitable number of rounds for client participation and ensure a fair accuracy distribution over the clients. The experimental results demonstrate that FedFair^3, in comparison to the state-of-the-art baselines, achieves 18.15% less accuracy variance on the IID data and 54.78% on the non-IID data, without decreasing the global accuracy. Furthermore, it shows 24.36% less wall-clock training time on average.

Title: Adversarial Training on Purification (AToP): Advancing Both Robustness and Generalization

• Authors: Guang Lin, Chao Li, Jianhai Zhang, Toshihisa Tanaka, Qibin Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16352
• Pdf URL: https://arxiv.org/pdf/2401.16352
• Copy Paste: [[2401.16352]] Adversarial Training on Purification (AToP): Advancing Both Robustness and Generalization(https://arxiv.org/abs/2401.16352)
• Keywords: defense, attack, robust
• Abstract: The deep neural networks are known to be vulnerable to well-designed adversarial attacks. The most successful defense technique based on adversarial training (AT) can achieve optimal robustness against particular attacks but cannot generalize well to unseen attacks. Another effective defense technique based on adversarial purification (AP) can enhance generalization but cannot achieve optimal robustness. Meanwhile, both methods share one common limitation on the degraded standard accuracy. To mitigate these issues, we propose a novel framework called Adversarial Training on Purification (AToP), which comprises two components: perturbation destruction by random transforms (RT) and purifier model fine-tuned (FT) by adversarial loss. RT is essential to avoid overlearning to known attacks resulting in the robustness generalization to unseen attacks and FT is essential for the improvement of robustness. To evaluate our method in an efficient and scalable way, we conduct extensive experiments on CIFAR-10, CIFAR-100, and ImageNette to demonstrate that our method achieves state-of-the-art results and exhibits generalization ability against unseen attacks.

Title: Empirical and Theoretical Analysis of Liquid Staking Protocols

• Authors: Krzysztof Gogol, Benjamin Kraner, Malte Schlosser, Tao Yan, Claudio Tessone, Burkhard Stiller
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16353
• Pdf URL: https://arxiv.org/pdf/2401.16353
• Copy Paste: [[2401.16353]] Empirical and Theoretical Analysis of Liquid Staking Protocols(https://arxiv.org/abs/2401.16353)
• Keywords: security
• Abstract: Liquid staking has become the largest category of decentralized finance protocols in terms of total value locked. However, few studies exist on its implementation designs or underlying risks. The liquid staking protocols allow for earning staking rewards without the disadvantage of locking the capital at the validators. Yet, they are seen by some as a threat to the Proof-of-Stake blockchain security. This paper is the first work that classifies liquid staking implementations. It analyzes the historical performance of major liquid staking tokens in comparison to the traditional staking for the largest Proof-of-Stake blockchains. Furthermore, the research investigates the impact of centralization, maximum extractable value and the migration of Ethereum from Proof-of-Work to Proof-of-Stake on the tokens' performance. Examining the tracking error of the liquid stacking providers to the staking rewards shows that they are persistent and cannot be explained by macro-variables of the currency, such as the variance or return.

Title: PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology

• Authors: Yuxuan Sun, Hao Wu, Chenglu Zhu, Sunyi Zheng, Qizi Chen, Kai Zhang, Yunlong Zhang, Xiaoxiao Lan, Mengyue Zheng, Jingxiong Li, Xinheng Lyu, Tao Lin, Lin Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16355
• Pdf URL: https://arxiv.org/pdf/2401.16355
• Copy Paste: [[2401.16355]] PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology(https://arxiv.org/abs/2401.16355)
• Keywords: robust
• Abstract: The emergence of large multimodal models has unlocked remarkable potential in AI, particularly in pathology. However, the lack of specialized, high-quality benchmark impeded their development and precise evaluation. To address this, we introduce PathMMU, the largest and highest-quality expert-validated pathology benchmark for LMMs. It comprises 33,573 multimodal multi-choice questions and 21,599 images from various sources, and an explanation for the correct answer accompanies each question. The construction of PathMMU capitalizes on the robust capabilities of GPT-4V, utilizing approximately 30,000 gathered image-caption pairs to generate Q\&As. Significantly, to maximize PathMMU's authority, we invite six pathologists to scrutinize each question under strict standards in PathMMU's validation and test sets, while simultaneously setting an expert-level performance benchmark for PathMMU. We conduct extensive evaluations, including zero-shot assessments of 14 open-sourced and three closed-sourced LMMs and their robustness to image corruption. We also fine-tune representative LMMs to assess their adaptability to PathMMU. The empirical findings indicate that advanced LMMs struggle with the challenging PathMMU benchmark, with the top-performing LMM, GPT-4V, achieving only a 51.7\% zero-shot performance, significantly lower than the 71.4\% demonstrated by human pathologists. After fine-tuning, even open-sourced LMMs can surpass GPT-4V with a performance of over 60\%, but still fall short of the expertise shown by pathologists. We hope that the PathMMU will offer valuable insights and foster the development of more specialized, next-generation LLMs for pathology.

Title: Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

• Authors: Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, Navdeep Jaitly
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16380
• Pdf URL: https://arxiv.org/pdf/2401.16380
• Copy Paste: [[2401.16380]] Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling(https://arxiv.org/abs/2401.16380)
• Keywords: large language model
• Abstract: Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training ($\textbf{WRAP}$) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as "like Wikipedia" or in "question-answer format" to jointly pre-train LLMs on real and synthetic rephrases. First, we show that using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training by $\sim3x$. At the same pre-training compute budget, it improves perplexity by more than 10% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data has higher utility than just real data because it (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher 'quality' than web-scraped data.

Title: Continual Learning with Pre-Trained Models: A Survey

• Authors: Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16386
• Pdf URL: https://arxiv.org/pdf/2401.16386
• Copy Paste: [[2401.16386]] Continual Learning with Pre-Trained Models: A Survey(https://arxiv.org/abs/2401.16386)
• Keywords: robust, fair
• Abstract: Nowadays, real-world applications often face streaming data, which requires the learning system to absorb new knowledge as data evolves. Continual Learning (CL) aims to achieve this goal and meanwhile overcome the catastrophic forgetting of former knowledge when learning new ones. Typical CL methods build the model from scratch to grow with incoming data. However, the advent of the pre-trained model (PTM) era has sparked immense research interest, particularly in leveraging PTMs' robust representational capabilities. This paper presents a comprehensive survey of the latest advancements in PTM-based CL. We categorize existing methodologies into three distinct groups, providing a comparative analysis of their similarities, differences, and respective advantages and disadvantages. Additionally, we offer an empirical study contrasting various state-of-the-art methods to highlight concerns regarding fairness in comparisons. The source code to reproduce these evaluations is available at: https://github.com/sun-hailong/LAMDA-PILOT

Title: Scaling Sparse Fine-Tuning to Large Language Models

• Authors: Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, Edoardo M. Ponti
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16405
• Pdf URL: https://arxiv.org/pdf/2401.16405
• Copy Paste: [[2401.16405]] Scaling Sparse Fine-Tuning to Large Language Models(https://arxiv.org/abs/2401.16405)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are difficult to fully fine-tune (e.g., with instructions or human feedback) due to their sheer number of parameters. A family of parameter-efficient sparse fine-tuning (SFT) methods have proven promising in terms of performance but their memory requirements increase proportionally to the size of the LLMs. In this work, we scale sparse fine-tuning to state-of-the-art LLMs like LLaMA 2 7B and 13B. At any given time, for a desired density level, we maintain an array of parameter indices and the deltas of these parameters relative to their pretrained values. We iterate among: (a) updating the active deltas, (b) pruning indices (based on the change of magnitude of their deltas) and (c) regrowth of indices. For regrowth, we explore two criteria based on either the accumulated gradients of a few candidate parameters or their approximate momenta estimated using the efficient SM3 optimizer. We experiment with instruction-tuning of LLMs on standard dataset mixtures, finding that SFT is often superior to popular parameter-efficient fine-tuning methods like LoRA (low-rank adaptation) in terms of performance and comparable in terms of run time. We additionally show that SFT is compatible with both quantization and efficient optimizers, to facilitate scaling to ever-larger model sizes. We release the code for SFT at https://github.com/AlanAnsell/peft and for the instruction-tuning experiments at https://github.com/ducdauge/sft-llm.

Title: Semi-parametric Expert Bayesian Network Learning with Gaussian Processes and Horseshoe Priors

• Authors: Yidou Weng, Finale Doshi-Velez
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.16419
• Pdf URL: https://arxiv.org/pdf/2401.16419
• Copy Paste: [[2401.16419]] Semi-parametric Expert Bayesian Network Learning with Gaussian Processes and Horseshoe Priors(https://arxiv.org/abs/2401.16419)
• Keywords: interpretability
• Abstract: This paper proposes a model learning Semi-parametric rela- tionships in an Expert Bayesian Network (SEBN) with linear parameter and structure constraints. We use Gaussian Pro- cesses and a Horseshoe prior to introduce minimal nonlin- ear components. To prioritize modifying the expert graph over adding new edges, we optimize differential Horseshoe scales. In real-world datasets with unknown truth, we gen- erate diverse graphs to accommodate user input, addressing identifiability issues and enhancing interpretability. Evalua- tion on synthetic and UCI Liver Disorders datasets, using metrics like structural Hamming Distance and test likelihood, demonstrates our models outperform state-of-the-art semi- parametric Bayesian Network model.

Title: Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation

• Authors: Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Di He, Jingjing Xu, Zhi Zhang, Hongxia Yang, Liwei Wang
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.16421
• Pdf URL: https://arxiv.org/pdf/2401.16421
• Copy Paste: [[2401.16421]] Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation(https://arxiv.org/abs/2401.16421)
• Keywords: segmentation
• Abstract: In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.

Title: Synchformer: Efficient Synchronization from Sparse Cues

• Authors: Vladimir Iashin, Weidi Xie, Esa Rahtu, Andrew Zisserman
• Subjects: cs.CV, cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.16423
• Pdf URL: https://arxiv.org/pdf/2401.16423
• Copy Paste: [[2401.16423]] Synchformer: Efficient Synchronization from Sparse Cues(https://arxiv.org/abs/2401.16423)
• Keywords: extraction, interpretability
• Abstract: Our objective is audio-visual synchronization with a focus on 'in-the-wild' videos, such as those on YouTube, where synchronization cues can be sparse. Our contributions include a novel audio-visual synchronization model, and training that decouples feature extraction from synchronization modelling through multi-modal segment-level contrastive pre-training. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale 'in-the-wild' dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability.

Title: Computer Vision for Primate Behavior Analysis in the Wild

• Authors: Richard Vogg, Timo Lüddecke, Jonathan Henrich, Sharmita Dey, Matthias Nuske, Valentin Hassler, Derek Murphy, Julia Fischer, Julia Ostner, Oliver Schülke, Peter M. Kappeler, Claudia Fichtel, Alexander Gail, Stefan Treue, Hansjörg Scherberger, Florentin Wörgötter, Alexander S. Ecker
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2401.16424
• Pdf URL: https://arxiv.org/pdf/2401.16424
• Copy Paste: [[2401.16424]] Computer Vision for Primate Behavior Analysis in the Wild(https://arxiv.org/abs/2401.16424)
• Keywords: fair
• Abstract: Advances in computer vision as well as increasingly widespread video-based behavioral monitoring have great potential for transforming how we study animal cognition and behavior. However, there is still a fairly large gap between the exciting prospects and what can actually be achieved in practice today, especially in videos from the wild. With this perspective paper, we want to contribute towards closing this gap, by guiding behavioral scientists in what can be expected from current methods and steering computer vision researchers towards problems that are relevant to advance research in animal behavior. We start with a survey of the state-of-the-art methods for computer vision problems that are directly relevant to the video-based study of animal behavior, including object detection, multi-individual tracking, (inter)action recognition and individual identification. We then review methods for effort-efficient learning, which is one of the biggest challenges from a practical perspective. Finally, we close with an outlook into the future of the emerging field of computer vision for animal behavior, where we argue that the field should move fast beyond the common frame-by-frame processing and treat video as a first-class citizen.