2024-01-24

Title: The Global Impact of AI-Artificial Intelligence: Recent Advances and Future Directions, A Review

• Authors: Chandregowda Pachegowda
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12223
• Pdf URL: https://arxiv.org/pdf/2401.12223
• Copy Paste: [[2401.12223]] The Global Impact of AI-Artificial Intelligence: Recent Advances and Future Directions, A Review(https://arxiv.org/abs/2401.12223)
• Keywords: security, privacy
• Abstract: Artificial intelligence (AI) is an emerging technology that has the potential to transform many aspects of society, including the economy, healthcare, and transportation. This article synthesizes recent research literature on the global impact of AI, exploring its potential benefits and risks. The article highlights the implications of AI, including its impact on economic, ethical, social, security & privacy, and job displacement aspects. It discusses the ethical concerns surrounding AI development, including issues of bias, security, and privacy violations. To ensure the responsible development and deployment of AI, collaboration between government, industry, and academia is essential. The article concludes by emphasizing the importance of public engagement and education to promote awareness and understanding of AI's impact on society at large.

Title: The Surprising Harmfulness of Benign Overfitting for Adversarial Robustness

• Authors: Yifan Hao, Tong Zhang
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.12236
• Pdf URL: https://arxiv.org/pdf/2401.12236
• Copy Paste: [[2401.12236]] The Surprising Harmfulness of Benign Overfitting for Adversarial Robustness(https://arxiv.org/abs/2401.12236)
• Keywords: attack, robust
• Abstract: Recent empirical and theoretical studies have established the generalization capabilities of large machine learning models that are trained to (approximately or exactly) fit noisy data. In this work, we prove a surprising result that even if the ground truth itself is robust to adversarial examples, and the benignly overfitted model is benign in terms of the ``standard'' out-of-sample risk objective, this benign overfitting process can be harmful when out-of-sample data are subject to adversarial manipulation. More specifically, our main results contain two parts: (i) the min-norm estimator in overparameterized linear model always leads to adversarial vulnerability in the ``benign overfitting'' setting; (ii) we verify an asymptotic trade-off result between the standard risk and the ``adversarial'' risk of every ridge regression estimator, implying that under suitable conditions these two items cannot both be small at the same time by any single choice of the ridge regularization parameter. Furthermore, under the lazy training regime, we demonstrate parallel results on two-layer neural tangent kernel (NTK) model, which align with empirical observations in deep neural networks. Our finding provides theoretical insights into the puzzling phenomenon observed in practice, where the true target function (e.g., human) is robust against adverasrial attack, while beginly overfitted neural networks lead to models that are not robust.

Title: Quantised Neural Network Accelerators for Low-Power IDS in Automotive Networks

• Authors: Shashwat Khandelwal, Anneliese Walsh, Shanker Shreejith
• Subjects: cs.CR, cs.AR, cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.12240
• Pdf URL: https://arxiv.org/pdf/2401.12240
• Copy Paste: [[2401.12240]] Quantised Neural Network Accelerators for Low-Power IDS in Automotive Networks(https://arxiv.org/abs/2401.12240)
• Keywords: attack
• Abstract: In this paper, we explore low-power custom quantised Multi-Layer Perceptrons (MLPs) as an Intrusion Detection System (IDS) for automotive controller area network (CAN). We utilise the FINN framework from AMD/Xilinx to quantise, train and generate hardware IP of our MLP to detect denial of service (DoS) and fuzzying attacks on CAN network, using ZCU104 (XCZU7EV) FPGA as our target ECU architecture with integrated IDS capabilities. Our approach achieves significant improvements in latency (0.12 ms per-message processing latency) and inference energy consumption (0.25 mJ per inference) while achieving similar classification performance as state-of-the-art approaches in the literature.

Title: BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models

• Authors: Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, Bo Li
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12242
• Pdf URL: https://arxiv.org/pdf/2401.12242
• Copy Paste: [[2401.12242]] BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models(https://arxiv.org/abs/2401.12242)
• Keywords: defense, attack, robust, large language model
• Abstract: Large language models (LLMs) are shown to benefit from chain-of-thought (COT) prompting, particularly when tackling tasks that require systematic reasoning processes. On the other hand, COT prompting also poses new vulnerabilities in the form of backdoor attacks, wherein the model will output unintended malicious content under specific backdoor-triggered conditions during inference. Traditional methods for launching backdoor attacks involve either contaminating the training dataset with backdoored instances or directly manipulating the model parameters during deployment. However, these approaches are not practical for commercial LLMs that typically operate via API access. In this paper, we propose BadChain, the first backdoor attack against LLMs employing COT prompting, which does not require access to the training dataset or model parameters and imposes low computational overhead. BadChain leverages the inherent reasoning capabilities of LLMs by inserting a backdoor reasoning step into the sequence of reasoning steps of the model output, thereby altering the final response when a backdoor trigger exists in the query prompt. Empirically, we show the effectiveness of BadChain for two COT strategies across four LLMs (Llama2, GPT-3.5, PaLM2, and GPT-4) and six complex benchmark tasks encompassing arithmetic, commonsense, and symbolic reasoning. Moreover, we show that LLMs endowed with stronger reasoning capabilities exhibit higher susceptibility to BadChain, exemplified by a high average attack success rate of 97.0% across the six benchmark tasks on GPT-4. Finally, we propose two defenses based on shuffling and demonstrate their overall ineffectiveness against BadChain. Therefore, BadChain remains a severe threat to LLMs, underscoring the urgency for the development of robust and effective future defenses.

Title: Large-scale Reinforcement Learning for Diffusion Models

• Authors: Yinan Zhang, Eric Tzeng, Yilun Du, Dmitry Kislyuk
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12244
• Pdf URL: https://arxiv.org/pdf/2401.12244
• Copy Paste: [[2401.12244]] Large-scale Reinforcement Learning for Diffusion Models(https://arxiv.org/abs/2401.12244)
• Keywords: fair, diffusion, generative
• Abstract: Text-to-image diffusion models are a class of deep generative models that have demonstrated an impressive capacity for high-quality image generation. However, these models are susceptible to implicit biases that arise from web-scale text-image training pairs and may inaccurately model aspects of images we care about. This can result in suboptimal samples, model bias, and images that do not align with human ethics and preferences. In this paper, we present an effective scalable algorithm to improve diffusion models using Reinforcement Learning (RL) across a diverse set of reward functions, such as human preference, compositionality, and fairness over millions of images. We illustrate how our approach substantially outperforms existing methods for aligning diffusion models with human preferences. We further illustrate how this substantially improves pretrained Stable Diffusion (SD) models, generating samples that are preferred by humans 80.3% of the time over those from the base SD model while simultaneously improving both the composition and diversity of generated samples.

Title: Orion-14B: Open-source Multilingual Large Language Models

• Authors: Du Chen, Yi Huang, Xiaopu Li, Yongqiang Li, Yongqiang Liu, Haihui Pan, Leichao Xu, Dacheng Zhang, Zhipeng Zhang, Kun Han
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12246
• Pdf URL: https://arxiv.org/pdf/2401.12246
• Copy Paste: [[2401.12246]] Orion-14B: Open-source Multilingual Large Language Models(https://arxiv.org/abs/2401.12246)
• Keywords: large language model
• Abstract: In this study, we introduce Orion-14B, a collection of multilingual large language models with 14 billion parameters. We utilize a data scheduling approach to train a foundational model on a diverse corpus of 2.5 trillion tokens, sourced from texts in English, Chinese, Japanese, Korean, and other languages. Additionally, we fine-tuned a series of models tailored for conversational applications and other specific use cases. Our evaluation results demonstrate that Orion-14B achieves state-of-the-art performance across a broad spectrum of tasks. We make the Orion-14B model family and its associated code publicly accessible https://github.com/OrionStarAI/Orion, aiming to inspire future research and practical applications in the field.

Title: Diffusion Representation for Asymmetric Kernels

• Authors: Alvaro Almeida Gomez, Antonio Silva Neto, Jorge zubelli
• Subjects: cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.12251
• Pdf URL: https://arxiv.org/pdf/2401.12251
• Copy Paste: [[2401.12251]] Diffusion Representation for Asymmetric Kernels(https://arxiv.org/abs/2401.12251)
• Keywords: diffusion
• Abstract: We extend the diffusion-map formalism to data sets that are induced by asymmetric kernels. Analytical convergence results of the resulting expansion are proved, and an algorithm is proposed to perform the dimensional reduction. In this work we study data sets in which its geometry structure is induced by an asymmetric kernel. We use a priori coordinate system to represent this geometry and, thus, be able to improve the computational complexity of reducing the dimensionality of data sets. A coordinate system connected to the tensor product of Fourier basis is used to represent the underlying geometric structure obtained by the diffusion-map, thus reducing the dimensionality of the data set and making use of the speedup provided by the two-dimensional Fast Fourier Transform algorithm (2-D FFT). We compare our results with those obtained by other eigenvalue expansions, and verify the efficiency of the algorithms with synthetic data, as well as with real data from applications including climate change studies.

Title: Instructional Fingerprinting of Large Language Models

• Authors: Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, Muhao Chen
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12255
• Pdf URL: https://arxiv.org/pdf/2401.12255
• Copy Paste: [[2401.12255]] Instructional Fingerprinting of Large Language Models(https://arxiv.org/abs/2401.12255)
• Keywords: protect, robust, large language model
• Abstract: The exorbitant cost of training Large language models (LLMs) from scratch makes it essential to fingerprint the models to protect intellectual property via ownership authentication and to ensure downstream users and developers comply with their license terms (e.g. restricting commercial use). In this study, we present a pilot study on LLM fingerprinting as a form of very lightweight instruction tuning. Model publisher specifies a confidential private key and implants it as an instruction backdoor that causes the LLM to generate specific text when the key is present. Results on 11 popularly-used LLMs showed that this approach is lightweight and does not affect the normal behavior of the model. It also prevents publisher overclaim, maintains robustness against fingerprint guessing and parameter-efficient training, and supports multi-stage fingerprinting akin to MIT License. Code is available in https://cnut1648.github.io/Model-Fingerprint/.

Title: Analyzing the Quality Attributes of AI Vision Models in Open Repositories Under Adversarial Attacks

• Authors: Zerui Wang, Yan Liu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12261
• Pdf URL: https://arxiv.org/pdf/2401.12261
• Copy Paste: [[2401.12261]] Analyzing the Quality Attributes of AI Vision Models in Open Repositories Under Adversarial Attacks(https://arxiv.org/abs/2401.12261)
• Keywords: attack, robust, explainability, transformer
• Abstract: As AI models rapidly evolve, they are frequently released to open repositories, such as HuggingFace. It is essential to perform quality assurance validation on these models before integrating them into the production development lifecycle. In addition to evaluating efficiency in terms of balanced accuracy and computing costs, adversarial attacks are potential threats to the robustness and explainability of AI models. Meanwhile, XAI applies algorithms that approximate inputs to outputs post-hoc to identify the contributing features. Adversarial perturbations may also degrade the utility of XAI explanations that require further investigation. In this paper, we present an integrated process designed for downstream evaluation tasks, including validating AI model accuracy, evaluating robustness with benchmark perturbations, comparing explanation utility, and assessing overhead. We demonstrate an evaluation scenario involving six computer vision models, which include CNN-based, Transformer-based, and hybrid architectures, three types of perturbations, and five XAI methods, resulting in ninety unique combinations. The process reveals the explanation utility among the XAI methods in terms of the identified key areas responding to the adversarial perturbation. The process produces aggregated results that illustrate multiple attributes of each AI model.

Title: Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction

• Authors: Md. Alamin Talukder, Md. Manowarul Islam, Md Ashraf Uddin, Khondokar Fida Hasan, Selina Sharmin, Salem A. Alyami, Mohammad Ali Moni
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12262
• Pdf URL: https://arxiv.org/pdf/2401.12262
• Copy Paste: [[2401.12262]] Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction(https://arxiv.org/abs/2401.12262)
• Keywords: security, protect, extraction
• Abstract: Cybersecurity has emerged as a critical global concern. Intrusion Detection Systems (IDS) play a critical role in protecting interconnected networks by detecting malicious actors and activities. Machine Learning (ML)-based behavior analysis within the IDS has considerable potential for detecting dynamic cyber threats, identifying abnormalities, and identifying malicious conduct within the network. However, as the number of data grows, dimension reduction becomes an increasingly difficult task when training ML models. Addressing this, our paper introduces a novel ML-based network intrusion detection model that uses Random Oversampling (RO) to address data imbalance and Stacking Feature Embedding based on clustering results, as well as Principal Component Analysis (PCA) for dimension reduction and is specifically designed for large and imbalanced datasets. This model's performance is carefully evaluated using three cutting-edge benchmark datasets: UNSW-NB15, CIC-IDS-2017, and CIC-IDS-2018. On the UNSW-NB15 dataset, our trials show that the RF and ET models achieve accuracy rates of 99.59% and 99.95%, respectively. Furthermore, using the CIC-IDS2017 dataset, DT, RF, and ET models reach 99.99% accuracy, while DT and RF models obtain 99.94% accuracy on CIC-IDS2018. These performance results continuously outperform the state-of-art, indicating significant progress in the field of network intrusion detection. This achievement demonstrates the efficacy of the suggested methodology, which can be used practically to accurately monitor and identify network traffic intrusions, thereby blocking possible threats.

Title: The Ethics of Interaction: Mitigating Security Threats in LLMs

• Authors: Ashutosh Kumar, Sagarika Singh, Shiv Vignesh Murty, Swathy Ragupathy
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12273
• Pdf URL: https://arxiv.org/pdf/2401.12273
• Copy Paste: [[2401.12273]] The Ethics of Interaction: Mitigating Security Threats in LLMs(https://arxiv.org/abs/2401.12273)
• Keywords: security, privacy, attack, robust
• Abstract: This paper comprehensively explores the ethical challenges arising from security threats to Language Learning Models (LLMs). These intricate digital repositories are increasingly integrated into our daily lives, making them prime targets for attacks that can compromise their training data and the confidentiality of their data sources. The paper delves into the nuanced ethical repercussions of such security threats on society and individual privacy. We scrutinize five major threats: prompt injection, jailbreaking, Personal Identifiable Information (PII) exposure, sexually explicit content, and hate based content, going beyond mere identification to assess their critical ethical consequences and the urgency they create for robust defensive strategies. The escalating reliance on LLMs underscores the crucial need for ensuring these systems operate within the bounds of ethical norms, particularly as their misuse can lead to significant societal and individual harm. We propose conceptualizing and developing an evaluative tool tailored for LLMs, which would serve a dual purpose, guiding developers and designers in preemptive fortification of backend systems and scrutinizing the ethical dimensions of LLM chatbot responses during the testing phase. By comparing LLM responses with those expected from humans in a moral context, we aim to discern the degree to which AI behaviors align with the ethical values held by a broader society. Ultimately, this paper not only underscores the ethical troubles presented by LLMs, it also highlights a path toward cultivating trust in these systems.

Title: GRATH: Gradual Self-Truthifying for Large Language Models

• Authors: Weixin Chen, Bo Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12292
• Pdf URL: https://arxiv.org/pdf/2401.12292
• Copy Paste: [[2401.12292]] GRATH: Gradual Self-Truthifying for Large Language Models(https://arxiv.org/abs/2401.12292)
• Keywords: large language model
• Abstract: Truthfulness is paramount for large language models (LLMs) as they are increasingly deployed in real-world applications. However, existing LLMs still struggle with generating truthful answers and content, as evidenced by their modest performance on benchmarks like TruthfulQA. To address this issue, we propose GRAdual self-truTHifying (GRATH), a novel post-processing method to enhance truthfulness of LLMs. GRATH utilizes out-of-domain question prompts to generate corresponding answers and adaptively optimizes the model via direct preference optimization (DPO). Note that during this process, GRATH learns truthfulness in a self-supervised manner without requiring annotated answers. In particular, GRATH first generates pairwise truthfulness training data by prompting the LLM itself, with each pair containing a question and its correct and incorrect answers. The model is then fine-tuned using DPO to learn from the difference between answer pairs. Subsequently, GRATH iteratively refines the truthfulness data and optimizes the model, leading to a gradual improvement in model truthfulness. Empirically, we evaluate GRATH using different 7B-LLMs and compare with LLMs with similar or even larger sizes on benchmark datasets. Our results show that GRATH effectively improves LLMs' truthfulness without compromising other core capabilities. Notably, GRATH achieves state-of-the-art performance on TruthfulQA, with MC1 accuracy as 54.71% and MC2 accuracy as 69.10%, which even surpass those on larger-scale models, such as Llama2-Chat-70B, by 23.62% and 24.18%, respectively.

Title: Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data

• Authors: Leonardo Castro-Gonzalez, Yi-Ling Chung, Hannak Rose Kirk, John Francis, Angus R. Williams, Pica Johansson, Jonathan Bright
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12295
• Pdf URL: https://arxiv.org/pdf/2401.12295
• Copy Paste: [[2401.12295]] Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data(https://arxiv.org/abs/2401.12295)
• Keywords: large language model
• Abstract: The field of machine learning has recently made significant progress in reducing the requirements for labelled training data when building new models. These `cheaper' learning techniques hold significant potential for the social sciences, where development of large labelled training datasets is often a significant practical impediment to the use of machine learning for analytical tasks. In this article we review three `cheap' techniques that have developed in recent years: weak supervision, transfer learning and prompt engineering. For the latter, we also review the particular case of zero-shot prompting of large language models. For each technique we provide a guide of how it works and demonstrate its application across six different realistic social science applications (two different tasks paired with three different dataset makeups). We show good performance for all techniques, and in particular we demonstrate how prompting of large language models can achieve high accuracy at very low cost. Our results are accompanied by a code repository to make it easy for others to duplicate our work and use it in their own research. Overall, our article is intended to stimulate further uptake of these techniques in the social sciences.

Title: Fine-tuning Large Language Models for Multigenerator, Multidomain, and Multilingual Machine-Generated Text Detection

• Authors: Feng Xiong, Thanet Markchom, Ziwei Zheng, Subin Jung, Varun Ojha, Huizhi Liang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12326
• Pdf URL: https://arxiv.org/pdf/2401.12326
• Copy Paste: [[2401.12326]] Fine-tuning Large Language Models for Multigenerator, Multidomain, and Multilingual Machine-Generated Text Detection(https://arxiv.org/abs/2401.12326)
• Keywords: extraction, transformer, large language model
• Abstract: SemEval-2024 Task 8 introduces the challenge of identifying machine-generated texts from diverse Large Language Models (LLMs) in various languages and domains. The task comprises three subtasks: binary classification in monolingual and multilingual (Subtask A), multi-class classification (Subtask B), and mixed text detection (Subtask C). This paper focuses on Subtask A & B. Each subtask is supported by three datasets for training, development, and testing. To tackle this task, two methods: 1) using traditional machine learning (ML) with natural language preprocessing (NLP) for feature extraction, and 2) fine-tuning LLMs for text classification. The results show that transformer models, particularly LoRA-RoBERTa, exceed traditional ML methods in effectiveness, with majority voting being particularly effective in multilingual contexts for identifying machine-generated texts.

Title: Subgraph Extraction-based Feedback-guided Iterative Scheduling for HLS

• Authors: Hanchen Ye, David Z. Pan, Chris Leary, Deming Chen, Xiaoqing Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12343
• Pdf URL: https://arxiv.org/pdf/2401.12343
• Copy Paste: [[2401.12343]] Subgraph Extraction-based Feedback-guided Iterative Scheduling for HLS(https://arxiv.org/abs/2401.12343)
• Keywords: extraction
• Abstract: This paper proposes ISDC, a novel feedback-guided iterative system of difference constraints (SDC) scheduling algorithm for high-level synthesis (HLS). ISDC leverages subgraph extraction-based low-level feedback from downstream tools like logic synthesizers to iteratively refine HLS scheduling. Technical innovations include: (1) An enhanced SDC formulation that effectively integrates low-level feedback into the linear-programming (LP) problem; (2) A fanout and window-based subgraph extraction mechanism driving the feedback cycle; (3) A no-human-in-loop ISDC flow compatible with a wide range of downstream tools and process design kits (PDKs). Evaluation shows that ISDC reduces register usage by 28.5% against an industrial-strength open-source HLS tool.

Title: OCT-SelfNet: A Self-Supervised Framework with Multi-Modal Datasets for Generalized and Robust Retinal Disease Detection

• Authors: Fatema-E Jannat, Sina Gholami, Minhaj Nur Alam, Hamed Tabkhi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12344
• Pdf URL: https://arxiv.org/pdf/2401.12344
• Copy Paste: [[2401.12344]] OCT-SelfNet: A Self-Supervised Framework with Multi-Modal Datasets for Generalized and Robust Retinal Disease Detection(https://arxiv.org/abs/2401.12344)
• Keywords: robust
• Abstract: Despite the revolutionary impact of AI and the development of locally trained algorithms, achieving widespread generalized learning from multi-modal data in medical AI remains a significant challenge. This gap hinders the practical deployment of scalable medical AI solutions. Addressing this challenge, our research contributes a self-supervised robust machine learning framework, OCT-SelfNet, for detecting eye diseases using optical coherence tomography (OCT) images. In this work, various data sets from various institutions are combined enabling a more comprehensive range of representation. Our method addresses the issue using a two-phase training approach that combines self-supervised pretraining and supervised fine-tuning with a mask autoencoder based on the SwinV2 backbone by providing a solution for real-world clinical deployment. Extensive experiments on three datasets with different encoder backbones, low data settings, unseen data settings, and the effect of augmentation show that our method outperforms the baseline model, Resnet-50 by consistently attaining AUC-ROC performance surpassing 77% across all tests, whereas the baseline model exceeds 54%. Moreover, in terms of the AUC-PR metric, our proposed method exceeded 42%, showcasing a substantial increase of at least 10% in performance compared to the baseline, which exceeded only 33%. This contributes to our understanding of our approach's potential and emphasizes its usefulness in clinical settings.

Title: Fuzzy quantitative attack tree analysis

• Authors: Thi Kim Nhung Dang, Milan Lopuhaä-Zwakenberg, Mariëlle Stoelinga
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.12346
• Pdf URL: https://arxiv.org/pdf/2401.12346
• Copy Paste: [[2401.12346]] Fuzzy quantitative attack tree analysis(https://arxiv.org/abs/2401.12346)
• Keywords: security, attack
• Abstract: Attack trees are important for security, as they help to identify weaknesses and vulnerabilities in a system. Quantitative attack tree analysis supports a number security metrics, which formulate important KPIs such as the shortest, most likely and cheapest attacks. A key bottleneck in quantitative analysis is that the values are usually not known exactly, due to insufficient data and/or lack of knowledge. Fuzzy logic is a prominent framework to handle such uncertain values, with applications in numerous domains. While several studies proposed fuzzy approaches to attack tree analysis, none of them provided a firm definition of fuzzy metric values or generic algorithms for computation of fuzzy metrics. In this work, we define a generic formulation for fuzzy metric values that applies to most quantitative metrics. The resulting metric value is a fuzzy number obtained by following Zadeh's extension principle, obtained when we equip the basis attack steps, i.e., the leaves of the attack trees, with fuzzy numbers. In addition, we prove a modular decomposition theorem that yields a bottom-up algorithm to efficiently calculate the top fuzzy metric value.

Title: Scaling Up Quantization-Aware Neural Architecture Search for Efficient Deep Learning on the Edge

• Authors: Yao Lu, Hiram Rayo Torres Rodriguez, Sebastian Vogel, Nick van de Waterlaat, Pavol Jancura
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12350
• Pdf URL: https://arxiv.org/pdf/2401.12350
• Copy Paste: [[2401.12350]] Scaling Up Quantization-Aware Neural Architecture Search for Efficient Deep Learning on the Edge(https://arxiv.org/abs/2401.12350)
• Keywords: segmentation
• Abstract: Neural Architecture Search (NAS) has become the de-facto approach for designing accurate and efficient networks for edge devices. Since models are typically quantized for edge deployment, recent work has investigated quantization-aware NAS (QA-NAS) to search for highly accurate and efficient quantized models. However, existing QA-NAS approaches, particularly few-bit mixed-precision (FB-MP) methods, do not scale to larger tasks. Consequently, QA-NAS has mostly been limited to low-scale tasks and tiny networks. In this work, we present an approach to enable QA-NAS (INT8 and FB-MP) on large-scale tasks by leveraging the block-wise formulation introduced by block-wise NAS. We demonstrate strong results for the semantic segmentation task on the Cityscapes dataset, finding FB-MP models 33% smaller and INT8 models 17.6% faster than DeepLabV3 (INT8) without compromising task performance.

Title: Efficient Collaborations through Weight-Driven Coalition Dynamics in Federated Learning Systems

• Authors: Mohammed El Hanjri, Hamza Reguieg, Adil Attiaoui, Amine Abouaomar, Abdellatif Kobbane, Mohamed El Kamili
• Subjects: cs.LG, cs.DC, cs.GT
• Abstract URL: https://arxiv.org/abs/2401.12356
• Pdf URL: https://arxiv.org/pdf/2401.12356
• Copy Paste: [[2401.12356]] Efficient Collaborations through Weight-Driven Coalition Dynamics in Federated Learning Systems(https://arxiv.org/abs/2401.12356)
• Keywords: federate
• Abstract: In the era of the Internet of Things (IoT), decentralized paradigms for machine learning are gaining prominence. In this paper, we introduce a federated learning model that capitalizes on the Euclidean distance between device model weights to assess their similarity and disparity. This is foundational for our system, directing the formation of coalitions among devices based on the closeness of their model weights. Furthermore, the concept of a barycenter, representing the average of model weights, helps in the aggregation of updates from multiple devices. We evaluate our approach using homogeneous and heterogeneous data distribution, comparing it against traditional federated learning averaging algorithm. Numerical results demonstrate its potential in offering structured, outperformed and communication-efficient model for IoT-based machine learning.

Title: A Security Risk Assessment Method for Distributed Ledger Technology (DLT) based Applications: Three Industry Case Studies

• Authors: Elena Baninemeh, Slinger Jansen, Katsiaryna Labunets
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.12358
• Pdf URL: https://arxiv.org/pdf/2401.12358
• Copy Paste: [[2401.12358]] A Security Risk Assessment Method for Distributed Ledger Technology (DLT) based Applications: Three Industry Case Studies(https://arxiv.org/abs/2401.12358)
• Keywords: security, attack
• Abstract: Distributed ledger technologies have gained significant attention and adoption in recent years. Despite various security features distributed ledger technology provides, they are vulnerable to different and new malicious attacks, such as selfish mining and Sybil attacks. While such vulnerabilities have been investigated, detecting and discovering appropriate countermeasures still need to be reported. Cybersecurity knowledge is limited and fragmented in this domain, while distributed ledger technology usage grows daily. Thus, research focusing on overcoming potential attacks on distributed ledgers is required. This study aims to raise awareness of the cybersecurity of distributed ledger technology by designing a security risk assessment method for distributed ledger technology applications. We have developed a database with possible security threats and known attacks on distributed ledger technologies to accompany the method, including sets of countermeasures. We employed a semi-systematic literature review combined with method engineering to develop a method that organizations can use to assess their cybersecurity risk for distributed ledger applications. The method has subsequently been evaluated in three case studies, which show that the method helps to effectively conduct security risk assessments for distributed ledger applications in these organizations.

Title: Enhancing In-context Learning via Linear Probe Calibration

• Authors: Momin Abbas, Yi Zhou, Parikshit Ram, Nathalie Baracaldo, Horst Samulowitz, Theodoros Salonidis, Tianyi Chen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12406
• Pdf URL: https://arxiv.org/pdf/2401.12406
• Copy Paste: [[2401.12406]] Enhancing In-context Learning via Linear Probe Calibration(https://arxiv.org/abs/2401.12406)
• Keywords: robust, transformer, generative
• Abstract: In-context learning (ICL) is a new paradigm for natural language processing that utilizes Generative Pre-trained Transformer (GPT)-like models. This approach uses prompts that include in-context demonstrations to generate the corresponding output for a new query input. However, applying ICL in real cases does not scale with the number of samples, and lacks robustness to different prompt templates and demonstration permutations. In this paper, we first show that GPT-like models using ICL result in unreliable predictions based on a new metric based on Shannon entropy. Then, to solve this problem, we propose a new technique called the Linear Probe Calibration (LinC), a method that calibrates the model's output probabilities, resulting in reliable predictions and improved performance, while requiring only minimal additional samples (as few as five labeled data samples). LinC significantly enhances the ICL test performance of GPT models on various benchmark datasets, with an average improvement of up to 21%, and up to a 50% improvement in some cases, and significantly boosts the performance of PEFT methods, especially in the low resource regime. Moreover, LinC achieves lower expected calibration error, and is highly robust to varying label proportions, prompt templates, and demonstration permutations. Our code is available at \url{https://github.com/mominabbass/LinC}.

Title: Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations

• Authors: Soyed Tuhin Ahmed, Kamal Danouchi, Guillaume Prenat, Lorena Anghel, Mehdi B. Tahoori
• Subjects: cs.LG, cs.AR, cs.ET
• Abstract URL: https://arxiv.org/abs/2401.12416
• Pdf URL: https://arxiv.org/pdf/2401.12416
• Copy Paste: [[2401.12416]] Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations(https://arxiv.org/abs/2401.12416)
• Keywords: robust
• Abstract: Bayesian Neural Networks (BayNNs) naturally provide uncertainty in their predictions, making them a suitable choice in safety-critical applications. Additionally, their realization using memristor-based in-memory computing (IMC) architectures enables them for resource-constrained edge applications. In addition to predictive uncertainty, however, the ability to be inherently robust to noise in computation is also essential to ensure functional safety. In particular, memristor-based IMCs are susceptible to various sources of non-idealities such as manufacturing and runtime variations, drift, and failure, which can significantly reduce inference accuracy. In this paper, we propose a method to inherently enhance the robustness and inference accuracy of BayNNs deployed in IMC architectures. To achieve this, we introduce a novel normalization layer combined with stochastic affine transformations. Empirical results in various benchmark datasets show a graceful degradation in inference accuracy, with an improvement of up to $58.11\%$.

Title: AdaEmbed: Semi-supervised Domain Adaptation in the Embedding Space

• Authors: Ali Mottaghi, Mohammad Abdullah Jamal, Serena Yeung, Omid Mohareri
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12421
• Pdf URL: https://arxiv.org/pdf/2401.12421
• Copy Paste: [[2401.12421]] AdaEmbed: Semi-supervised Domain Adaptation in the Embedding Space(https://arxiv.org/abs/2401.12421)
• Keywords: robust
• Abstract: Semi-supervised domain adaptation (SSDA) presents a critical hurdle in computer vision, especially given the frequent scarcity of labeled data in real-world settings. This scarcity often causes foundation models, trained on extensive datasets, to underperform when applied to new domains. AdaEmbed, our newly proposed methodology for SSDA, offers a promising solution to these challenges. Leveraging the potential of unlabeled data, AdaEmbed facilitates the transfer of knowledge from a labeled source domain to an unlabeled target domain by learning a shared embedding space. By generating accurate and uniform pseudo-labels based on the established embedding space, the model overcomes the limitations of conventional SSDA, thus enhancing performance significantly. Our method's effectiveness is validated through extensive experiments on benchmark datasets such as DomainNet, Office-Home, and VisDA-C, where AdaEmbed consistently outperforms all the baselines, setting a new state of the art for SSDA. With its straightforward implementation and high data efficiency, AdaEmbed stands out as a robust and pragmatic solution for real-world scenarios, where labeled data is scarce. To foster further research and application in this area, we are sharing the codebase of our unified framework for semi-supervised domain adaptation.

Title: InverseMatrixVT3D: An Efficient Projection Matrix-Based Approach for 3D Occupancy Prediction

• Authors: Zhenxing Ming, Julie Stephany Berrio, Mao Shan, Stewart Worrall
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2401.12422
• Pdf URL: https://arxiv.org/pdf/2401.12422
• Copy Paste: [[2401.12422]] InverseMatrixVT3D: An Efficient Projection Matrix-Based Approach for 3D Occupancy Prediction(https://arxiv.org/abs/2401.12422)
• Keywords: transformer
• Abstract: This paper introduces InverseMatrixVT3D, an efficient method for transforming multi-view image features into 3D feature volumes for 3D semantic occupancy prediction. Existing methods for constructing 3D volumes often rely on depth estimation, device-specific operators, or transformer queries, which hinders the widespread adoption of 3D occupancy models. In contrast, our approach leverages two projection matrices to store the static mapping relationships and matrix multiplications to efficiently generate global Bird's Eye View (BEV) features and local 3D feature volumes. Specifically, we achieve this by performing matrix multiplications between multi-view image feature maps and two sparse projection matrices. We introduce a sparse matrix handling technique for the projection matrices to optimise GPU memory usage. Moreover, a global-local attention fusion module is proposed to integrate the global BEV features with the local 3D feature volumes to obtain the final 3D volume. We also employ a multi-scale supervision mechanism to further enhance performance. Comprehensive experiments on the nuScenes dataset demonstrate the simplicity and effectiveness of our method. The code will be made available at:https://github.com/DanielMing123/InverseMatrixVT3D

Title: The Neglected Tails of Vision-Language Models

• Authors: Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, Shu Kong
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12425
• Pdf URL: https://arxiv.org/pdf/2401.12425
• Copy Paste: [[2401.12425]] The Neglected Tails of Vision-Language Models(https://arxiv.org/abs/2401.12425)
• Keywords: robust
• Abstract: Vision-language models (VLMs) excel in zero-shot recognition but exhibit drastically imbalanced performance across visual concepts. For example, CLIP, despite an impressive mean zero-shot accuracy on ImageNet (72.7%), yields $<$10% on ten concepts (e.g., gyromitra and night snake), presumably, because these concepts are under-represented in VLMs' imbalanced pretraining data. Yet, assessing this imbalance is challenging as it is non-trivial to calculate the frequency of specific concepts within VLMs' large-scale pretraining data. Our work makes the first attempt to measure the concept frequency by analyzing pretraining texts. We use off-the-shelf language models to help count relevant texts that contain synonyms of the given concepts and resolve linguistic ambiguity. We confirm that popular VLM datasets like LAION indeed exhibit long-tailed concept distributions, which strongly correlate with per-class accuracies. Further, contemporary multimodal systems, e.g., visual chatbots and text-to-image generators, also struggle with the rare concepts identified by our method. To mitigate VLMs' imbalanced performance in zero-shot recognition, we propose REtrieval-Augmented Learning REAL. First, instead of prompting VLMs using the original class names, REAL uses their most frequent synonyms found in VLMs' pretraining texts. This already outperforms human-engineered and LLM-generated prompts over nine benchmark datasets, likely because VLMs have seen more images associated with the frequently used synonyms. Second, REAL uses all the concept synonyms to retrieve a small, class-balanced set of pretraining data to train a robust classifier. REAL surpasses the recent retrieval-augmented solution REACT, using 400x less storage and 10,000x less training time!

Title: Wasserstein Differential Privacy

• Authors: Chengyi Yang, Jiayin Qi, Aimin Zhou
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.12436
• Pdf URL: https://arxiv.org/pdf/2401.12436
• Copy Paste: [[2401.12436]] Wasserstein Differential Privacy(https://arxiv.org/abs/2401.12436)
• Keywords: privacy
• Abstract: Differential privacy (DP) has achieved remarkable results in the field of privacy-preserving machine learning. However, existing DP frameworks do not satisfy all the conditions for becoming metrics, which prevents them from deriving better basic private properties and leads to exaggerated values on privacy budgets. We propose Wasserstein differential privacy (WDP), an alternative DP framework to measure the risk of privacy leakage, which satisfies the properties of symmetry and triangle inequality. We show and prove that WDP has 13 excellent properties, which can be theoretical supports for the better performance of WDP than other DP frameworks. In addition, we derive a general privacy accounting method called Wasserstein accountant, which enables WDP to be applied in stochastic gradient descent (SGD) scenarios containing sub-sampling. Experiments on basic mechanisms, compositions and deep learning show that the privacy budgets obtained by Wasserstein accountant are relatively stable and less influenced by order. Moreover, the overestimation on privacy budgets can be effectively alleviated. The code is available at https://github.com/Hifipsysta/WDP.

Title: MAST: Video Polyp Segmentation with a Mixture-Attention Siamese Transformer

• Authors: Geng Chen, Junqing Yang, Xiaozhou Pu, Ge-Peng Ji, Huan Xiong, Yongsheng Pan, Hengfei Cui, Yong Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12439
• Pdf URL: https://arxiv.org/pdf/2401.12439
• Copy Paste: [[2401.12439]] MAST: Video Polyp Segmentation with a Mixture-Attention Siamese Transformer(https://arxiv.org/abs/2401.12439)
• Keywords: transformer, segmentation
• Abstract: Accurate segmentation of polyps from colonoscopy videos is of great significance to polyp treatment and early prevention of colorectal cancer. However, it is challenging due to the difficulties associated with modelling long-range spatio-temporal relationships within a colonoscopy video. In this paper, we address this challenging task with a novel Mixture-Attention Siamese Transformer (MAST), which explicitly models the long-range spatio-temporal relationships with a mixture-attention mechanism for accurate polyp segmentation. Specifically, we first construct a Siamese transformer architecture to jointly encode paired video frames for their feature representations. We then design a mixture-attention module to exploit the intra-frame and inter-frame correlations, enhancing the features with rich spatio-temporal relationships. Finally, the enhanced features are fed to two parallel decoders for predicting the segmentation maps. To the best of our knowledge, our MAST is the first transformer model dedicated to video polyp segmentation. Extensive experiments on the large-scale SUN-SEG benchmark demonstrate the superior performance of MAST in comparison with the cutting-edge competitors. Our code is publicly available at https://github.com/Junqing-Yang/MAST.

Title: Patch2QL: Discover Cognate Defects in Open Source Software Supply Chain With Auto-generated Static Analysis Rules

• Authors: Fuwei Wang
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.12443
• Pdf URL: https://arxiv.org/pdf/2401.12443
• Copy Paste: [[2401.12443]] Patch2QL: Discover Cognate Defects in Open Source Software Supply Chain With Auto-generated Static Analysis Rules(https://arxiv.org/abs/2401.12443)
• Keywords: security
• Abstract: In the open source software (OSS) ecosystem, there exists a complex software supply chain, where developers upstream and downstream widely borrow and reuse code. This results in the widespread occurrence of recurring defects, missing fixes, and propagation issues. These are collectively referred to as cognate defects, and their scale and threats have not received extensive attention and systematic research. Software composition analysis and code clone detection methods are unable to cover the various variant issues in the supply chain scenario, while code static analysis, or static application security testing (SAST) techniques struggle to target specific defects. In this paper, we propose a novel technique for detecting cognate defects in OSS through the automatic generation of SAST rules. Specifically, it extracts key syntax and semantic information from pre- and post-patch versions of code through structural comparison and control flow to data flow analysis, and generates rules that matches these key elements. We have implemented a prototype tool called Patch2QL and applied it to fundamental OSS in C/C++. In experiments, we discovered 7 new vulnerabilities with medium to critical severity in the most popular upstream software, as well as numerous potential security issues. When analyzing downstream projects in the supply chain, we found a significant number of representative cognate defects, clarifying the threat posed by this issue. Additionally, compared to general-purpose SAST and signature-based mechanisms, the generated rules perform better at discover all variants of cognate defects.

Title: Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration

• Authors: Yifan Zhang, Siyu Ren, Junhui Hou, Jinjian Wu, Guangming Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12452
• Pdf URL: https://arxiv.org/pdf/2401.12452
• Copy Paste: [[2401.12452]] Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration(https://arxiv.org/abs/2401.12452)
• Keywords: segmentation
• Abstract: This paper introduces a novel self-supervised learning framework for enhancing 3D perception in autonomous driving scenes. Specifically, our approach, named NCLR, focuses on 2D-3D neural calibration, a novel pretext task that estimates the rigid transformation aligning camera and LiDAR coordinate systems. First, we propose the learnable transformation alignment to bridge the domain gap between image and point cloud data, converting features into a unified representation space for effective comparison and matching. Second, we identify the overlapping area between the image and point cloud with the fused features. Third, we establish dense 2D-3D correspondences to estimate the rigid transformation. The framework not only learns fine-grained matching from points to pixels but also achieves alignment of the image and point cloud at a holistic level, understanding their relative pose. We demonstrate NCLR's efficacy by applying the pre-trained backbone to downstream tasks, such as LiDAR-based 3D semantic segmentation, object detection, and panoptic segmentation. Comprehensive experiments on various datasets illustrate the superiority of NCLR over existing self-supervised methods. The results confirm that joint learning from different modalities significantly enhances the network's understanding abilities and effectiveness of learned representation. Code will be available at \url{https://github.com/Eaphan/NCLR}.

Title: Exploration and Improvement of Nerf-based 3D Scene Editing Techniques

• Authors: Shun Fang, Ming Cui, Xing Feng, Yanan Zhang
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.12456
• Pdf URL: https://arxiv.org/pdf/2401.12456
• Copy Paste: [[2401.12456]] Exploration and Improvement of Nerf-based 3D Scene Editing Techniques(https://arxiv.org/abs/2401.12456)
• Keywords: transformer
• Abstract: NeRF's high-quality scene synthesis capability was quickly accepted by scholars in the years after it was proposed, and significant progress has been made in 3D scene representation and synthesis. However, the high computational cost limits intuitive and efficient editing of scenes, making NeRF's development in the scene editing field facing many challenges. This paper reviews the preliminary explorations of scholars on NeRF in the scene or object editing field in recent years, mainly changing the shape and texture of scenes or objects in new synthesized scenes; through the combination of residual models such as GaN and Transformer with NeRF, the generalization ability of NeRF scene editing has been further expanded, including realizing real-time new perspective editing feedback, multimodal editing of text synthesized 3D scenes, 4D synthesis performance, and in-depth exploration in light and shadow editing, initially achieving optimization of indirect touch editing and detail representation in complex scenes. Currently, most NeRF editing methods focus on the touch points and materials of indirect points, but when dealing with more complex or larger 3D scenes, it is difficult to balance accuracy, breadth, efficiency, and quality. Overcoming these challenges may become the direction of future NeRF 3D scene editing technology.

Title: Fast Adversarial Training against Textual Adversarial Attacks

• Authors: Yichen Yang, Xin Liu, Kun He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12461
• Pdf URL: https://arxiv.org/pdf/2401.12461
• Copy Paste: [[2401.12461]] Fast Adversarial Training against Textual Adversarial Attacks(https://arxiv.org/abs/2401.12461)
• Keywords: defense, attack, robust
• Abstract: Many adversarial defense methods have been proposed to enhance the adversarial robustness of natural language processing models. However, most of them introduce additional pre-set linguistic knowledge and assume that the synonym candidates used by attackers are accessible, which is an ideal assumption. We delve into adversarial training in the embedding space and propose a Fast Adversarial Training (FAT) method to improve the model robustness in the synonym-unaware scenario from the perspective of single-step perturbation generation and perturbation initialization. Based on the observation that the adversarial perturbations crafted by single-step and multi-step gradient ascent are similar, FAT uses single-step gradient ascent to craft adversarial examples in the embedding space to expedite the training process. Based on the observation that the perturbations generated on the identical training sample in successive epochs are similar, FAT fully utilizes historical information when initializing the perturbation. Extensive experiments demonstrate that FAT significantly boosts the robustness of BERT models in the synonym-unaware scenario, and outperforms the defense baselines under various attacks with character-level and word-level modifications.

Title: Zero Shot Open-ended Video Inference

• Authors: Ee Yeo Keat, Zhang Hao, Alexander Matyasko, Basura Fernando
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12471
• Pdf URL: https://arxiv.org/pdf/2401.12471
• Copy Paste: [[2401.12471]] Zero Shot Open-ended Video Inference(https://arxiv.org/abs/2401.12471)
• Keywords: large language model
• Abstract: Zero-shot open-ended inference on untrimmed videos poses a significant challenge, especially when no annotated data is utilized to navigate the inference direction. In this work, we aim to address this underexplored domain by introducing an adaptable framework that efficiently combines both the frozen vision-language (VL) model and off-the-shelf large language model (LLM) for conducting zero-shot open-ended inference tasks without requiring any additional training or fine-tuning. Our comprehensive experiments span various video action datasets for goal inference and action recognition tasks. The results demonstrate the framework's superior performance in goal inference compared to conventional vision-language models in open-ended and close-ended scenarios. Notably, the proposed framework exhibits the capability to generalize effectively to action recognition tasks, underscoring its versatility and potential contributions to advancing the video-based zero-shot understanding.

Title: Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

• Authors: Keming Lu, Bowen Yu, Chang Zhou, Jingren Zhou
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12474
• Pdf URL: https://arxiv.org/pdf/2401.12474
• Copy Paste: [[2401.12474]] Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment(https://arxiv.org/abs/2401.12474)
• Keywords: large language model
• Abstract: Considerable efforts have been invested in augmenting the role-playing proficiency of open-source large language models (LLMs) by emulating proprietary counterparts. Nevertheless, we posit that LLMs inherently harbor role-play capabilities, owing to the extensive knowledge of characters and potential dialogues ingrained in their vast training corpora. Thus, in this study, we introduce Ditto, a self-alignment method for role-play. Ditto capitalizes on character knowledge, encouraging an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension. This method creates a role-play training set comprising 4,000 characters, surpassing the scale of currently available datasets by tenfold regarding the number of roles. Subsequently, we fine-tune the LLM using this self-generated dataset to augment its role-playing capabilities. Upon evaluating our meticulously constructed and reproducible role-play benchmark and the roleplay subset of MT-Bench, Ditto, in various parameter scales, consistently maintains a consistent role identity and provides accurate role-specific knowledge in multi-turn role-play conversations. Notably, it outperforms all open-source role-play baselines, showcasing performance levels comparable to advanced proprietary chatbots. Furthermore, we present the first comprehensive cross-supervision alignment experiment in the role-play domain, revealing that the intrinsic capabilities of LLMs confine the knowledge within role-play. Meanwhile, the role-play styles can be easily acquired with the guidance of smaller models. We open-source related resources at https://github.com/OFA-Sys/Ditto.

Title: TD^2-Net: Toward Denoising and Debiasing for Dynamic Scene Graph Generation

• Authors: Xin Lin, Chong Shi, Yibing Zhan, Zuopeng Yang, Yaqi Wu, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12479
• Pdf URL: https://arxiv.org/pdf/2401.12479
• Copy Paste: [[2401.12479]] TD^2-Net: Toward Denoising and Debiasing for Dynamic Scene Graph Generation(https://arxiv.org/abs/2401.12479)
• Keywords: robust, transformer
• Abstract: Dynamic scene graph generation (SGG) focuses on detecting objects in a video and determining their pairwise relationships. Existing dynamic SGG methods usually suffer from several issues, including 1) Contextual noise, as some frames might contain occluded and blurred objects. 2) Label bias, primarily due to the high imbalance between a few positive relationship samples and numerous negative ones. Additionally, the distribution of relationships exhibits a long-tailed pattern. To address the above problems, in this paper, we introduce a network named TD$^2$-Net that aims at denoising and debiasing for dynamic SGG. Specifically, we first propose a denoising spatio-temporal transformer module that enhances object representation with robust contextual information. This is achieved by designing a differentiable Top-K object selector that utilizes the gumbel-softmax sampling strategy to select the relevant neighborhood for each object. Second, we introduce an asymmetrical reweighting loss to relieve the issue of label bias. This loss function integrates asymmetry focusing factors and the volume of samples to adjust the weights assigned to individual samples. Systematic experimental results demonstrate the superiority of our proposed TD$^2$-Net over existing state-of-the-art approaches on Action Genome databases. In more detail, TD$^2$-Net outperforms the second-best competitors by 12.7 \% on mean-Recall@10 for predicate classification.

Title: Explore Synergistic Interaction Across Frames for Interactive Video Object Segmentation

• Authors: Kexin Li, Tao Jiang, Zongxin Yang, Yi Yang, Yueting Zhuang, Jun Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12480
• Pdf URL: https://arxiv.org/pdf/2401.12480
• Copy Paste: [[2401.12480]] Explore Synergistic Interaction Across Frames for Interactive Video Object Segmentation(https://arxiv.org/abs/2401.12480)
• Keywords: segmentation
• Abstract: Interactive Video Object Segmentation (iVOS) is a challenging task that requires real-time human-computer interaction. To improve the user experience, it is important to consider the user's input habits, segmentation quality, running time and memory consumption.However, existing methods compromise user experience with single input mode and slow running speed. Specifically, these methods only allow the user to interact with one single frame, which limits the expression of the user's intent.To overcome these limitations and better align with people's usage habits, we propose a framework that can accept multiple frames simultaneously and explore synergistic interaction across frames (SIAF). Concretely, we designed the Across-Frame Interaction Module that enables users to annotate different objects freely on multiple frames. The AFI module will migrate scribble information among multiple interactive frames and generate multi-frame masks. Additionally, we employ the id-queried mechanism to process multiple objects in batches. Furthermore, for a more efficient propagation and lightweight model, we design a truncated re-propagation strategy to replace the previous multi-round fusion module, which employs an across-round memory that stores important interaction information. Our SwinB-SIAF achieves new state-of-the-art performance on DAVIS 2017 (89.6%, J&F@60). Moreover, our R50-SIAF is more than 3 faster than the state-of-the-art competitor under challenging multi-object scenarios.

Title: Assessing and Understanding Creativity in Large Language Models

• Authors: Yunpu Zhao, Rui Zhang, Wenyi Li, Di Huang, Jiaming Guo, Shaohui Peng, Yifan Hao, Yuanbo Wen, Xing Hu, Zidong Du, Qi Guo, Ling Li, Yunji Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12491
• Pdf URL: https://arxiv.org/pdf/2401.12491
• Copy Paste: [[2401.12491]] Assessing and Understanding Creativity in Large Language Models(https://arxiv.org/abs/2401.12491)
• Keywords: large language model
• Abstract: In the field of natural language processing, the rapid development of large language model (LLM) has attracted more and more attention. LLMs have shown a high level of creativity in various tasks, but the methods for assessing such creativity are inadequate. The assessment of LLM creativity needs to consider differences from humans, requiring multi-dimensional measurement while balancing accuracy and efficiency. This paper aims to establish an efficient framework for assessing the level of creativity in LLMs. By adapting the modified Torrance Tests of Creative Thinking, the research evaluates the creative performance of various LLMs across 7 tasks, emphasizing 4 criteria including Fluency, Flexibility, Originality, and Elaboration. In this context, we develop a comprehensive dataset of 700 questions for testing and an LLM-based evaluation method. In addition, this study presents a novel analysis of LLMs' responses to diverse prompts and role-play situations. We found that the creativity of LLMs primarily falls short in originality, while excelling in elaboration. Besides, the use of prompts and the role-play settings of the model significantly influence creativity. Additionally, the experimental results also indicate that collaboration among multiple LLMs can enhance originality. Notably, our findings reveal a consensus between human evaluations and LLMs regarding the personality traits that influence creativity. The findings underscore the significant impact of LLM design on creativity and bridges artificial intelligence and human creativity, offering insights into LLMs' creativity and potential applications.

Title: Convolutional Initialization for Data-Efficient Vision Transformers

• Authors: Jianqiao Zheng, Xueqian Li, Simon Lucey
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12511
• Pdf URL: https://arxiv.org/pdf/2401.12511
• Copy Paste: [[2401.12511]] Convolutional Initialization for Data-Efficient Vision Transformers(https://arxiv.org/abs/2401.12511)
• Keywords: transformer
• Abstract: Training vision transformer networks on small datasets poses challenges. In contrast, convolutional neural networks (CNNs) can achieve state-of-the-art performance by leveraging their architectural inductive bias. In this paper, we investigate whether this inductive bias can be reinterpreted as an initialization bias within a vision transformer network. Our approach is motivated by the finding that random impulse filters can achieve almost comparable performance to learned filters in CNNs. We introduce a novel initialization strategy for transformer networks that can achieve comparable performance to CNNs on small datasets while preserving its architectural flexibility.

Title: Detecting and recognizing characters in Greek papyri with YOLOv8, DeiT and SimCLR

• Authors: Robert Turnbull, Evelyn Mannix
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12513
• Pdf URL: https://arxiv.org/pdf/2401.12513
• Copy Paste: [[2401.12513]] Detecting and recognizing characters in Greek papyri with YOLOv8, DeiT and SimCLR(https://arxiv.org/abs/2401.12513)
• Keywords: transformer
• Abstract: The capacity to isolate and recognize individual characters from facsimile images of papyrus manuscripts yields rich opportunities for digital analysis. For this reason the `ICDAR 2023 Competition on Detection and Recognition of Greek Letters on Papyri' was held as part of the 17th International Conference on Document Analysis and Recognition. This paper discusses our submission to the competition. We used an ensemble of YOLOv8 models to detect and classify individual characters and employed two different approaches for refining the character predictions, including a transformer based DeiT approach and a ResNet-50 model trained on a large corpus of unlabelled data using SimCLR, a self-supervised learning method. Our submission won the recognition challenge with a mAP of 42.2%, and was runner-up in the detection challenge with a mean average precision (mAP) of 51.4%. At the more relaxed intersection over union threshold of 0.5, we achieved the highest mean average precision and mean average recall results for both detection and classification. We ran our prediction pipeline on more than 4,500 images from the Oxyrhynchus Papyri to illustrate the utility of our approach, and we release the results publicly in multiple formats.

Title: DDMI: Domain-Agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations

• Authors: Dogyun Park, Sihyeon Kim, Sojin Lee, Hyunwoo J. Kim
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.12517
• Pdf URL: https://arxiv.org/pdf/2401.12517
• Copy Paste: [[2401.12517]] DDMI: Domain-Agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations(https://arxiv.org/abs/2401.12517)
• Keywords: diffusion, generative
• Abstract: Recent studies have introduced a new class of generative models for synthesizing implicit neural representations (INRs) that capture arbitrary continuous signals in various domains. These models opened the door for domain-agnostic generative models, but they often fail to achieve high-quality generation. We observed that the existing methods generate the weights of neural networks to parameterize INRs and evaluate the network with fixed positional embeddings (PEs). Arguably, this architecture limits the expressive power of generative models and results in low-quality INR generation. To address this limitation, we propose Domain-agnostic Latent Diffusion Model for INRs (DDMI) that generates adaptive positional embeddings instead of neural networks' weights. Specifically, we develop a Discrete-to-continuous space Variational AutoEncoder (D2C-VAE), which seamlessly connects discrete data and the continuous signal functions in the shared latent space. Additionally, we introduce a novel conditioning mechanism for evaluating INRs with the hierarchically decomposed PEs to further enhance expressive power. Extensive experiments across four modalities, e.g., 2D images, 3D shapes, Neural Radiance Fields, and videos, with seven benchmark datasets, demonstrate the versatility of DDMI and its superior performance compared to the existing INR generative models.

Title: Key Information Retrieval to Classify the Unstructured Data Content of Preferential Trade Agreements

• Authors: Jiahui Zhao, Ziyi Meng, Stepan Gordeev, Zijie Pan, Dongjin Song, Sandro Steinbach, Caiwen Ding
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12520
• Pdf URL: https://arxiv.org/pdf/2401.12520
• Copy Paste: [[2401.12520]] Key Information Retrieval to Classify the Unstructured Data Content of Preferential Trade Agreements(https://arxiv.org/abs/2401.12520)
• Keywords: transformer
• Abstract: With the rapid proliferation of textual data, predicting long texts has emerged as a significant challenge in the domain of natural language processing. Traditional text prediction methods encounter substantial difficulties when grappling with long texts, primarily due to the presence of redundant and irrelevant information, which impedes the model's capacity to capture pivotal insights from the text. To address this issue, we introduce a novel approach to long-text classification and prediction. Initially, we employ embedding techniques to condense the long texts, aiming to diminish the redundancy therein. Subsequently,the Bidirectional Encoder Representations from Transformers (BERT) embedding method is utilized for text classification training. Experimental outcomes indicate that our method realizes considerable performance enhancements in classifying long texts of Preferential Trade Agreements. Furthermore, the condensation of text through embedding methods not only augments prediction accuracy but also substantially reduces computational complexity. Overall, this paper presents a strategy for long-text prediction, offering a valuable reference for researchers and engineers in the natural language processing sphere.

Title: BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models

• Authors: Feng Lin, Hanling Yi, Hongbin Li, Yifan Yang, Xiaotian Yu, Guangming Lu, Rong Xiao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12522
• Pdf URL: https://arxiv.org/pdf/2401.12522
• Copy Paste: [[2401.12522]] BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models(https://arxiv.org/abs/2401.12522)
• Keywords: large language model
• Abstract: Large language models (LLMs) commonly employ autoregressive generation during inference, leading to high memory bandwidth demand and consequently extended latency. To mitigate this inefficiency, we present Bi-directional Tuning for lossless Acceleration (BiTA), an innovative method expediting LLMs via streamlined semi-autoregressive generation and draft verification. Inspired by the concept of prompt tuning, we enhance LLMs with a parameter-efficient design called bi-directional tuning for the capability in semi-autoregressive generation. Employing efficient tree-based decoding, the models perform draft candidate generation and verification in parallel, ensuring outputs identical to their autoregressive counterparts under greedy sampling. BiTA serves as a lightweight plug-in module, seamlessly boosting the inference efficiency of existing LLMs without requiring additional assistance models or incurring significant extra memory costs. Applying the proposed BiTA, LLaMA-2-70B-Chat achieves a 2.7$\times$ speedup on the MT-Bench benchmark. Extensive experiments confirm our method surpasses state-of-the-art acceleration techniques.

Title: DAFA: Distance-Aware Fair Adversarial Training

• Authors: Hyungyu Lee, Saehyung Lee, Hyemi Jang, Junsung Park, Ho Bae, Sungroh Yoon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12532
• Pdf URL: https://arxiv.org/pdf/2401.12532
• Copy Paste: [[2401.12532]] DAFA: Distance-Aware Fair Adversarial Training(https://arxiv.org/abs/2401.12532)
• Keywords: attack, robust, fair
• Abstract: The disparity in accuracy between classes in standard training is amplified during adversarial training, a phenomenon termed the robust fairness problem. Existing methodologies aimed to enhance robust fairness by sacrificing the model's performance on easier classes in order to improve its performance on harder ones. However, we observe that under adversarial attacks, the majority of the model's predictions for samples from the worst class are biased towards classes similar to the worst class, rather than towards the easy classes. Through theoretical and empirical analysis, we demonstrate that robust fairness deteriorates as the distance between classes decreases. Motivated by these insights, we introduce the Distance-Aware Fair Adversarial training (DAFA) methodology, which addresses robust fairness by taking into account the similarities between classes. Specifically, our method assigns distinct loss weights and adversarial margins to each class and adjusts them to encourage a trade-off in robustness among similar classes. Experimental results across various datasets demonstrate that our method not only maintains average robust accuracy but also significantly improves the worst robust accuracy, indicating a marked improvement in robust fairness compared to existing methods.

Title: Self-Supervised Vision Transformers Are Efficient Segmentation Learners for Imperfect Labels

• Authors: Seungho Lee, Seoungyoon Kang, Hyunjung Shim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12535
• Pdf URL: https://arxiv.org/pdf/2401.12535
• Copy Paste: [[2401.12535]] Self-Supervised Vision Transformers Are Efficient Segmentation Learners for Imperfect Labels(https://arxiv.org/abs/2401.12535)
• Keywords: robust, transformer, segmentation
• Abstract: This study demonstrates a cost-effective approach to semantic segmentation using self-supervised vision transformers (SSVT). By freezing the SSVT backbone and training a lightweight segmentation head, our approach effectively utilizes imperfect labels, thereby improving robustness to label imperfections. Empirical experiments show significant performance improvements over existing methods for various annotation types, including scribble, point-level, and image-level labels. The research highlights the effectiveness of self-supervised vision transformers in dealing with imperfect labels, providing a practical and efficient solution for semantic segmentation while reducing annotation costs. Through extensive experiments, we confirm that our method outperforms baseline models for all types of imperfect labels. Especially under the zero-shot vision-language-model-based label, our model exhibits 11.5\%p performance gain compared to the baseline.

Title: Multi-Party Private Set Intersection: A Circuit-Based Protocol with Jaccard Similarity for Secure and Efficient Anomaly Detection in Network Traffic

• Authors: Jiuheng Su, Zhili Chen, Xiaomin Yang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.12542
• Pdf URL: https://arxiv.org/pdf/2401.12542
• Copy Paste: [[2401.12542]] Multi-Party Private Set Intersection: A Circuit-Based Protocol with Jaccard Similarity for Secure and Efficient Anomaly Detection in Network Traffic(https://arxiv.org/abs/2401.12542)
• Keywords: secure, privacy
• Abstract: We present a new circuit-based protocol for multi-party private set intersection (PSI) that allows m parties to compute the intersection of their datasets without revealing any additional information about the items outside the intersection. Building upon the two-party Sort-Compare-Shuffle (SCS) protocol, we seamlessly extend it to a multi-party setting. Demonstrating its practicality through implementation, our protocol exhibits acceptable performance. Specifically, with 7 parties, each possessing a set size of 2^{12}, our protocol completes in just 19 seconds. Moreover, circuit-based protocols like ours have an advantage over using custom protocols to perform more complex computation. We substantiate this advantage by incorporating a module for calculating the Jaccard similarity metric of the private sets which can be used in the application domain of network traffic analysis for anomaly detection. This extension showcases the versatility of our protocol beyond set intersection computations, demonstrating its efficacy in preserving privacy while efficiently identifying abnormal patterns in network flow.

Title: Automated Fact-Checking of Climate Change Claims with Large Language Models

• Authors: Markus Leippold, Saeid Ashraf Vaghefi, Dominik Stammbach, Veruska Muccione, Julia Bingler, Jingwei Ni, Chiara Colesanti-Senni, Tobias Wekhof, Tobias Schimanski, Glen Gostlow, Tingyu Yu, Juerg Luterbacher, Christian Huggel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12566
• Pdf URL: https://arxiv.org/pdf/2401.12566
• Copy Paste: [[2401.12566]] Automated Fact-Checking of Climate Change Claims with Large Language Models(https://arxiv.org/abs/2401.12566)
• Keywords: robust, large language model
• Abstract: This paper presents Climinator, a novel AI-based tool designed to automate the fact-checking of climate change claims. Utilizing an array of Large Language Models (LLMs) informed by authoritative sources like the IPCC reports and peer-reviewed scientific literature, Climinator employs an innovative Mediator-Advocate framework. This design allows Climinator to effectively synthesize varying scientific perspectives, leading to robust, evidence-based evaluations. Our model demonstrates remarkable accuracy when testing claims collected from Climate Feedback and Skeptical Science. Notably, when integrating an advocate with a climate science denial perspective in our framework, Climinator's iterative debate process reliably converges towards scientific consensus, underscoring its adeptness at reconciling diverse viewpoints into science-based, factual conclusions. While our research is subject to certain limitations and necessitates careful interpretation, our approach holds significant potential. We hope to stimulate further research and encourage exploring its applicability in other contexts, including political fact-checking and legal domains.

Title: LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools

• Authors: Qianli Wang, Tatiana Anikina, Nils Feldhus, Josef van Genabith, Leonhard Hennig, Sebastian Möller
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12576
• Pdf URL: https://arxiv.org/pdf/2401.12576
• Copy Paste: [[2401.12576]] LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools(https://arxiv.org/abs/2401.12576)
• Keywords: interpretability, large language model
• Abstract: Interpretability tools that offer explanations in the form of a dialogue have demonstrated their efficacy in enhancing users' understanding, as one-off explanations may occasionally fall short in providing sufficient information to the user. Current solutions for dialogue-based explanations, however, require many dependencies and are not easily transferable to tasks they were not designed for. With LLMCheckup, we present an easily accessible tool that allows users to chat with any state-of-the-art large language model (LLM) about its behavior. We enable LLMs to generate all explanations by themselves and take care of intent recognition without fine-tuning, by connecting them with a broad spectrum of Explainable AI (XAI) tools, e.g. feature attributions, embedding-based similarity, and prompting strategies for counterfactual and rationale generation. LLM (self-)explanations are presented as an interactive dialogue that supports follow-up questions and generates suggestions. LLMCheckup provides tutorials for operations available in the system, catering to individuals with varying levels of expertise in XAI and supports multiple input modalities. We introduce a new parsing strategy called multi-prompt parsing substantially enhancing the parsing accuracy of LLMs. Finally, we showcase the tasks of fact checking and commonsense question answering.

Title: ToDA: Target-oriented Diffusion Attacker against Recommendation System

• Authors: Xiaohao Liu, Zhulin Tao, Ting Jiang, He Chang, Yunshan Ma, Xianglin Huang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.12578
• Pdf URL: https://arxiv.org/pdf/2401.12578
• Copy Paste: [[2401.12578]] ToDA: Target-oriented Diffusion Attacker against Recommendation System(https://arxiv.org/abs/2401.12578)
• Keywords: security, attack, diffusion, generative
• Abstract: Recommendation systems (RS) have become indispensable tools for web services to address information overload, thus enhancing user experiences and bolstering platforms' revenues. However, with their increasing ubiquity, security concerns have also emerged. As the public accessibility of RS, they are susceptible to specific malicious attacks where adversaries can manipulate user profiles, leading to biased recommendations. Recent research often integrates additional modules using generative models to craft these deceptive user profiles, ensuring them are imperceptible while causing the intended harm. Albeit their efficacy, these models face challenges of unstable training and the exploration-exploitation dilemma, which can lead to suboptimal results. In this paper, we pioneer to investigate the potential of diffusion models (DMs), for shilling attacks. Specifically, we propose a novel Target-oriented Diffusion Attack model (ToDA). It incorporates a pre-trained autoencoder that transforms user profiles into a high dimensional space, paired with a Latent Diffusion Attacker (LDA)-the core component of ToDA. LDA introduces noise into the profiles within this latent space, adeptly steering the approximation towards targeted items through cross-attention mechanisms. The global horizon, implemented by a bipartite graph, is involved in LDA and derived from the encoded user profile feature. This makes LDA possible to extend the generation outwards the on-processing user feature itself, and bridges the gap between diffused user features and target item features. Extensive experiments compared to several SOTA baselines demonstrate ToDA's effectiveness. Specific studies exploit the elaborative design of ToDA and underscore the potency of advanced generative models in such contexts.

Title: SLANG: New Concept Comprehension of Large Language Models

• Authors: Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, Xueqi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12585
• Pdf URL: https://arxiv.org/pdf/2401.12585
• Copy Paste: [[2401.12585]] SLANG: New Concept Comprehension of Large Language Models(https://arxiv.org/abs/2401.12585)
• Keywords: large language model
• Abstract: The dynamic nature of language, particularly evident in the realm of slang and memes on the Internet, poses serious challenges to the adaptability of large language models (LLMs). Traditionally anchored to static datasets, these models often struggle to keep up with the rapid linguistic evolution characteristic of online communities. This research addresses the critical need to bridge this gap, aiming to enhance LLMs' comprehension of evolving new concepts on the internet, without the high cost and impracticality of continual retraining. To address this issue, we propose a new benchmark $\textbf{SLANG}$ to assess LLMs' proficiency in comprehending emerging linguistic trends and a baseline approach $\textbf{FOCUS}$, which uses causal inference to enhance LLMs to understand new phrases and usage patterns. This approach involves scrutinizing real-world instances of linguistic shifts, serving as contextual beacons, to form more precise and contextually relevant connections between newly emerging expressions and their intended meanings. The empirical analysis shows that our causal inference-based approach outperforms the traditional models in terms of precision and relevance in the interpretation of Internet slang and memes.

Title: Interpreting Equivariant Representations

• Authors: Andreas Abildtrup Hansen, Anna Calissano, Aasa Feragen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.12588
• Pdf URL: https://arxiv.org/pdf/2401.12588
• Copy Paste: [[2401.12588]] Interpreting Equivariant Representations(https://arxiv.org/abs/2401.12588)
• Keywords: extraction
• Abstract: Latent representations are used extensively for downstream tasks, such as visualization, interpolation or feature extraction of deep learning models. Invariant and equivariant neural networks are powerful and well-established models for enforcing inductive biases. In this paper, we demonstrate that the inductive bias imposed on the by an equivariant model must also be taken into account when using latent representations. We show how not accounting for the inductive biases leads to decreased performance on downstream tasks, and vice versa, how accounting for inductive biases can be done effectively by using an invariant projection of the latent representations. We propose principles for how to choose such a projection, and show the impact of using these principles in two common examples: First, we study a permutation equivariant variational auto-encoder trained for molecule graph generation; here we show that invariant projections can be designed that incur no loss of information in the resulting invariant representation. Next, we study a rotation-equivariant representation used for image classification. Here, we illustrate how random invariant projections can be used to obtain an invariant representation with a high degree of retained information. In both cases, the analysis of invariant latent representations proves superior to their equivariant counterparts. Finally, we illustrate that the phenomena documented here for equivariant neural networks have counterparts in standard neural networks where invariance is encouraged via augmentation. Thus, while these ambiguities may be known by experienced developers of equivariant models, we make both the knowledge as well as effective tools to handle the ambiguities available to the broader community.

Title: SCORPION Cyber Range: Fully Customizable Cyberexercises, Gamification and Learning Analytics to Train Cybersecurity Competencies

• Authors: Pantaleone Nespoli, Mariano Albaladejo-González, José Antonio Pastor Valera, José A. Ruipérez-Valiente, Joaquin Garcia-Alfaro, Félix Gómez Mármol
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.12594
• Pdf URL: https://arxiv.org/pdf/2401.12594
• Copy Paste: [[2401.12594]] SCORPION Cyber Range: Fully Customizable Cyberexercises, Gamification and Learning Analytics to Train Cybersecurity Competencies(https://arxiv.org/abs/2401.12594)
• Keywords: security, protect, attack, biometric
• Abstract: It is undeniable that we are witnessing an unprecedented digital revolution. However, recent years have been characterized by the explosion of cyberattacks, making cybercrime one of the most profitable businesses on the planet. That is why training in cybersecurity is increasingly essential to protect the assets of cyberspace. One of the most vital tools to train cybersecurity competencies is the Cyber Range, a virtualized environment that simulates realistic networks. The paper at hand introduces SCORPION, a fully functional and virtualized Cyber Range, which manages the authoring and automated deployment of scenarios. In addition, SCORPION includes several elements to improve student motivation, such as a gamification system with medals, points, or rankings, among other elements. Such a gamification system includes an adaptive learning module that is able to adapt the cyberexercise based on the users' performance. Moreover, SCORPION leverages learning analytics that collects and processes telemetric and biometric user data, including heart rate through a smartwatch, which is available through a dashboard for instructors. Finally, we developed a case study where SCORPION obtained 82.10\% in usability and 4.57 out of 5 in usefulness from the viewpoint of a student and an instructor. The positive evaluation results are promising, indicating that SCORPION can become an effective, motivating, and advanced cybersecurity training tool to help fill current gaps in this context.

Title: UniHDA: Towards Universal Hybrid Domain Adaptation of Image Generators

• Authors: Hengjia Li, Yang Liu, Yuqi Lin, Zhanwei Zhang, Yibo Zhao, weihang Pan, Tu Zheng, Zheng Yang, Yuchun Jiang, Boxi Wu, Deng Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12596
• Pdf URL: https://arxiv.org/pdf/2401.12596
• Copy Paste: [[2401.12596]] UniHDA: Towards Universal Hybrid Domain Adaptation of Image Generators(https://arxiv.org/abs/2401.12596)
• Keywords: diffusion, generative
• Abstract: Generative domain adaptation has achieved remarkable progress, enabling us to adapt a pre-trained generator to a new target domain. However, existing methods simply adapt the generator to a single target domain and are limited to a single modality, either text-driven or image-driven. Moreover, they are prone to overfitting domain-specific attributes, which inevitably compromises cross-domain consistency. In this paper, we propose UniHDA, a unified and versatile framework for generative hybrid domain adaptation with multi-modal references from multiple domains. We use CLIP encoder to project multi-modal references into a unified embedding space and then linear interpolate the direction vectors from multiple target domains to achieve hybrid domain adaptation. To ensure the cross-domain consistency, we propose a novel cross-domain spatial structure (CSS) loss that maintains detailed spatial structure information between source and target generator. Experiments show that the adapted generator can synthesise realistic images with various attribute compositions. Additionally, our framework is versatile to multiple generators, \eg, StyleGAN2 and Diffusion Models.

Title: The twin peaks of learning neural networks

• Authors: Elizaveta Demyanenko, Christoph Feinauer, Enrico M. Malatesta, Luca Saglietti
• Subjects: cs.LG, cond-mat.dis-nn, math.PR, math.ST
• Abstract URL: https://arxiv.org/abs/2401.12610
• Pdf URL: https://arxiv.org/pdf/2401.12610
• Copy Paste: [[2401.12610]] The twin peaks of learning neural networks(https://arxiv.org/abs/2401.12610)
• Keywords: attack, robust
• Abstract: Recent works demonstrated the existence of a double-descent phenomenon for the generalization error of neural networks, where highly overparameterized models escape overfitting and achieve good test performance, at odds with the standard bias-variance trade-off described by statistical learning theory. In the present work, we explore a link between this phenomenon and the increase of complexity and sensitivity of the function represented by neural networks. In particular, we study the Boolean mean dimension (BMD), a metric developed in the context of Boolean function analysis. Focusing on a simple teacher-student setting for the random feature model, we derive a theoretical analysis based on the replica method that yields an interpretable expression for the BMD, in the high dimensional regime where the number of data points, the number of features, and the input size grow to infinity. We find that, as the degree of overparameterization of the network is increased, the BMD reaches an evident peak at the interpolation threshold, in correspondence with the generalization error peak, and then slowly approaches a low asymptotic value. The same phenomenology is then traced in numerical experiments with different model classes and training setups. Moreover, we find empirically that adversarially initialized models tend to show higher BMD values, and that models that are more robust to adversarial attacks exhibit a lower BMD.

Title: Prompt Smells: An Omen for Undesirable Generative AI Outputs

• Authors: Krishna Ronanki, Beatriz Cabrero-Daniel, Christian Berger
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.12611
• Pdf URL: https://arxiv.org/pdf/2401.12611
• Copy Paste: [[2401.12611]] Prompt Smells: An Omen for Undesirable Generative AI Outputs(https://arxiv.org/abs/2401.12611)
• Keywords: generative
• Abstract: Recent Generative Artificial Intelligence (GenAI) trends focus on various applications, including creating stories, illustrations, poems, articles, computer code, music compositions, and videos. Extrinsic hallucinations are a critical limitation of such GenAI, which can lead to significant challenges in achieving and maintaining the trustworthiness of GenAI. In this paper, we propose two new concepts that we believe will aid the research community in addressing limitations associated with the application of GenAI models. First, we propose a definition for the "desirability" of GenAI outputs and three factors which are observed to influence it. Second, drawing inspiration from Martin Fowler's code smells, we propose the concept of "prompt smells" and the adverse effects they are observed to have on the desirability of GenAI outputs. We expect our work will contribute to the ongoing conversation about the desirability of GenAI outputs and help advance the field in a meaningful way.

Title: A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments

• Authors: Zhengxuan Wu, Atticus Geiger, Jing Huang, Aryaman Arora, Thomas Icard, Christopher Potts, Noah D. Goodman
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12631
• Pdf URL: https://arxiv.org/pdf/2401.12631
• Copy Paste: [[2401.12631]] A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments(https://arxiv.org/abs/2401.12631)
• Keywords: interpretability
• Abstract: We respond to the recent paper by Makelov et al. (2023), which reviews subspace interchange intervention methods like distributed alignment search (DAS; Geiger et al. 2023) and claims that these methods potentially cause "interpretability illusions". We first review Makelov et al. (2023)'s technical notion of what an "interpretability illusion" is, and then we show that even intuitive and desirable explanations can qualify as illusions in this sense. As a result, their method of discovering "illusions" can reject explanations they consider "non-illusory". We then argue that the illusions Makelov et al. (2023) see in practice are artifacts of their training and evaluation paradigms. We close by emphasizing that, though we disagree with their core characterization, Makelov et al. (2023)'s examples and discussion have undoubtedly pushed the field of interpretability forward.

Title: ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation

• Authors: Shengze Li, Jianjian Cao, Peng Ye, Yuhan Ding, Chongjun Tu, Tao Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12665
• Pdf URL: https://arxiv.org/pdf/2401.12665
• Copy Paste: [[2401.12665]] ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation(https://arxiv.org/abs/2401.12665)
• Keywords: segmentation
• Abstract: Recently, foundational models such as CLIP and SAM have shown promising performance for the task of Zero-Shot Anomaly Segmentation (ZSAS). However, either CLIP-based or SAM-based ZSAS methods still suffer from non-negligible key drawbacks: 1) CLIP primarily focuses on global feature alignment across different inputs, leading to imprecise segmentation of local anomalous parts; 2) SAM tends to generate numerous redundant masks without proper prompt constraints, resulting in complex post-processing requirements. In this work, we innovatively propose a CLIP and SAM collaboration framework called ClipSAM for ZSAS. The insight behind ClipSAM is to employ CLIP's semantic understanding capability for anomaly localization and rough segmentation, which is further used as the prompt constraints for SAM to refine the anomaly segmentation results. In details, we introduce a crucial Unified Multi-scale Cross-modal Interaction (UMCI) module for interacting language with visual features at multiple scales of CLIP to reason anomaly positions. Then, we design a novel Multi-level Mask Refinement (MMR) module, which utilizes the positional information as multi-level prompts for SAM to acquire hierarchical levels of masks and merges them. Extensive experiments validate the effectiveness of our approach, achieving the optimal segmentation performance on the MVTec-AD and VisA datasets.

Title: Context Matters: Pushing the Boundaries of Open-Ended Answer Generation with Graph-Structured Knowledge Context

• Authors: Somnath Banerjee, Amruit Sahoo, Sayan Layek, Avik Dutta, Rima Hazra, Animesh Mukherjee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12671
• Pdf URL: https://arxiv.org/pdf/2401.12671
• Copy Paste: [[2401.12671]] Context Matters: Pushing the Boundaries of Open-Ended Answer Generation with Graph-Structured Knowledge Context(https://arxiv.org/abs/2401.12671)
• Keywords: robust, large language model
• Abstract: In the continuously advancing AI landscape, crafting context-rich and meaningful responses via Large Language Models (LLMs) is essential. Researchers are becoming more aware of the challenges that LLMs with fewer parameters encounter when trying to provide suitable answers to open-ended questions. To address these hurdles, the integration of cutting-edge strategies, augmentation of rich external domain knowledge to LLMs, offers significant improvements. This paper introduces a novel framework that combines graph-driven context retrieval in conjunction to knowledge graphs based enhancement, honing the proficiency of LLMs, especially in domain specific community question answering platforms like AskUbuntu, Unix, and ServerFault. We conduct experiments on various LLMs with different parameter sizes to evaluate their ability to ground knowledge and determine factual accuracy in answers to open-ended questions. Our methodology GraphContextGen consistently outperforms dominant text-based retrieval systems, demonstrating its robustness and adaptability to a larger number of use cases. This advancement highlights the importance of pairing context rich data retrieval with LLMs, offering a renewed approach to knowledge sourcing and generation in AI systems. We also show that, due to rich contextual data retrieval, the crucial entities, along with the generated answer, remain factually coherent with the gold answer.

Title: Non-Neighbors Also Matter to Kriging: A New Contrastive-Prototypical Learning

• Authors: Zhishuai Li, Yunhao Nie, Ziyue Li, Lei Bai, Yisheng Lv, Rui Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12681
• Pdf URL: https://arxiv.org/pdf/2401.12681
• Copy Paste: [[2401.12681]] Non-Neighbors Also Matter to Kriging: A New Contrastive-Prototypical Learning(https://arxiv.org/abs/2401.12681)
• Keywords: robust
• Abstract: Kriging aims at estimating the attributes of unsampled geo-locations from observations in the spatial vicinity or physical connections, which helps mitigate skewed monitoring caused by under-deployed sensors. Existing works assume that neighbors' information offers the basis for estimating the attributes of the unobserved target while ignoring non-neighbors. However, non-neighbors could also offer constructive information, and neighbors could also be misleading. To this end, we propose ``Contrastive-Prototypical'' self-supervised learning for Kriging (KCP) to refine valuable information from neighbors and recycle the one from non-neighbors. As a pre-trained paradigm, we conduct the Kriging task from a new perspective of representation: we aim to first learn robust and general representations and then recover attributes from representations. A neighboring contrastive module is designed that coarsely learns the representations by narrowing the representation distance between the target and its neighbors while pushing away the non-neighbors. In parallel, a prototypical module is introduced to identify similar representations via exchanged prediction, thus refining the misleading neighbors and recycling the useful non-neighbors from the neighboring contrast component. As a result, not all the neighbors and some of the non-neighbors will be used to infer the target. To encourage the two modules above to learn general and robust representations, we design an adaptive augmentation module that incorporates data-driven attribute augmentation and centrality-based topology augmentation over the spatiotemporal Kriging graph data. Extensive experiments on real-world datasets demonstrate the superior performance of KCP compared to its peers with 6% improvements and exceptional transferability and robustness. The code is available at https://github.com/bonaldli/KCP

Title: LLpowershap: Logistic Loss-based Automated Shapley Values Feature Selection Method

• Authors: Iqbal Madakkatel, Elina Hyppönen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12683
• Pdf URL: https://arxiv.org/pdf/2401.12683
• Copy Paste: [[2401.12683]] LLpowershap: Logistic Loss-based Automated Shapley Values Feature Selection Method(https://arxiv.org/abs/2401.12683)
• Keywords: robust, fair
• Abstract: Shapley values have been used extensively in machine learning, not only to explain black box machine learning models, but among other tasks, also to conduct model debugging, sensitivity and fairness analyses and to select important features for robust modelling and for further follow-up analyses. Shapley values satisfy certain axioms that promote fairness in distributing contributions of features toward prediction or reducing error, after accounting for non-linear relationships and interactions when complex machine learning models are employed. Recently, a number of feature selection methods utilising Shapley values have been introduced. Here, we present a novel feature selection method, LLpowershap, which makes use of loss-based Shapley values to identify informative features with minimal noise among the selected sets of features. Our simulation results show that LLpowershap not only identifies higher number of informative features but outputs fewer noise features compared to other state-of-the-art feature selection methods. Benchmarking results on four real-world datasets demonstrate higher or at par predictive performance of LLpowershap compared to other Shapley based wrapper methods, or filter methods.

Title: Generating Unsupervised Abstractive Explanations for Rumour Verification

• Authors: Iman Munire Bilal, Preslav Nakov, Rob Procter, Maria Liakata
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12713
• Pdf URL: https://arxiv.org/pdf/2401.12713
• Copy Paste: [[2401.12713]] Generating Unsupervised Abstractive Explanations for Rumour Verification(https://arxiv.org/abs/2401.12713)
• Keywords: explainability, large language model
• Abstract: The task of rumour verification in social media concerns assessing the veracity of a claim on the basis of conversation threads that result from it. While previous work has focused on predicting a veracity label, here we reformulate the task to generate model-centric, free-text explanations of a rumour's veracity. We follow an unsupervised approach by first utilising post-hoc explainability methods to score the most important posts within a thread and then we use these posts to generate informative explanatory summaries by employing template-guided summarisation. To evaluate the informativeness of the explanatory summaries, we exploit the few-shot learning capabilities of a large language model (LLM). Our experiments show that LLMs can have similar agreement to humans in evaluating summaries. Importantly, we show that explanatory abstractive summaries are more informative and better reflect the predicted rumour veracity than just using the highest ranking posts in the thread.

Title: A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressions

• Authors: Guilherme H. Resende, Luiz F. Nery, Fabrício Benevenuto, Savvas Zannettou, Flavio Figueiredo
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.12720
• Pdf URL: https://arxiv.org/pdf/2401.12720
• Copy Paste: [[2401.12720]] A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressions(https://arxiv.org/abs/2401.12720)
• Keywords: diffusion
• Abstract: Language is a dynamic aspect of our culture that changes when expressed in different technologies/communities. Online social networks have enabled the diffusion and evolution of different dialects, including African American English (AAE). However, this increased usage is not without barriers. One particular barrier is how sentiment (Vader, TextBlob, and Flair) and toxicity (Google's Perspective and the open-source Detoxify) methods present biases towards utterances with AAE expressions. Consider Google's Perspective to understand bias. Here, an utterance such as ``All n*ggers deserve to die respectfully. The police murder us.'' it reaches a higher toxicity than ``African-Americans deserve to die respectfully. The police murder us.''. This score difference likely arises because the tool cannot understand the re-appropriation of the term ``n*gger''. One explanation for this bias is that AI models are trained on limited datasets, and using such a term in training data is more likely to appear in a toxic utterance. While this may be plausible, the tool will make mistakes regardless. Here, we study bias on two Web-based (YouTube and Twitter) datasets and two spoken English datasets. Our analysis shows how most models present biases towards AAE in most settings. We isolate the impact of AAE expression usage via linguistic control features from the Linguistic Inquiry and Word Count (LIWC) software, grammatical control features extracted via Part-of-Speech (PoS) tagging from Natural Language Processing (NLP) models, and the semantic of utterances by comparing sentence embeddings from recent language models. We present consistent results on how a heavy usage of AAE expressions may cause the speaker to be considered substantially more toxic, even when speaking about nearly the same subject. Our study complements similar analyses focusing on small datasets and/or one method only.

Title: Falcon: Fair Active Learning using Multi-armed Bandits

• Authors: Ki Hyun Tae, Hantian Zhang, Jaeyoung Park, Kexin Rong, Steven Euijong Whang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12722
• Pdf URL: https://arxiv.org/pdf/2401.12722
• Copy Paste: [[2401.12722]] Falcon: Fair Active Learning using Multi-armed Bandits(https://arxiv.org/abs/2401.12722)
• Keywords: fair
• Abstract: Biased data can lead to unfair machine learning models, highlighting the importance of embedding fairness at the beginning of data analysis, particularly during dataset curation and labeling. In response, we propose Falcon, a scalable fair active learning framework. Falcon adopts a data-centric approach that improves machine learning model fairness via strategic sample selection. Given a user-specified group fairness measure, Falcon identifies samples from "target groups" (e.g., (attribute=female, label=positive)) that are the most informative for improving fairness. However, a challenge arises since these target groups are defined using ground truth labels that are not available during sample selection. To handle this, we propose a novel trial-and-error method, where we postpone using a sample if the predicted label is different from the expected one and falls outside the target group. We also observe the trade-off that selecting more informative samples results in higher likelihood of postponing due to undesired label prediction, and the optimal balance varies per dataset. We capture the trade-off between informativeness and postpone rate as policies and propose to automatically select the best policy using adversarial multi-armed bandit methods, given their computational efficiency and theoretical guarantees. Experiments show that Falcon significantly outperforms existing fair active learning approaches in terms of fairness and accuracy and is more efficient. In particular, only Falcon supports a proper trade-off between accuracy and fairness where its maximum fairness score is 1.8-4.5x higher than the second-best results.

Title: Shift-ConvNets: Small Convolutional Kernel with Large Kernel Effects

• Authors: Dachong Li, Li Li, Zhuangzhuang Chen, Jianqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12736
• Pdf URL: https://arxiv.org/pdf/2401.12736
• Copy Paste: [[2401.12736]] Shift-ConvNets: Small Convolutional Kernel with Large Kernel Effects(https://arxiv.org/abs/2401.12736)
• Keywords: transformer
• Abstract: Recent studies reveal that the remarkable performance of Vision transformers (ViTs) benefits from large receptive fields. For this reason, the large convolutional kernel design becomes an ideal solution to make Convolutional Neural Networks (CNNs) great again. However, the typical large convolutional kernels turn out to be hardware-unfriendly operators, resulting in discount compatibility of various hardware platforms. Thus, it is unwise to simply enlarge the convolutional kernel size. In this paper, we reveal that small convolutional kernels and convolution operations can achieve the closing effects of large kernel sizes. Then, we propose a shift-wise operator that ensures the CNNs capture long-range dependencies with the help of the sparse mechanism, while remaining hardware-friendly. Experimental results show that our shift-wise operator significantly improves the accuracy of a regular CNN while markedly reducing computational requirements. On the ImageNet-1k, our shift-wise enhanced CNN model outperforms the state-of-the-art models. Code & models at https://github.com/lidc54/shift-wiseConv.

Title: Correlation-Embedded Transformer Tracking: A Single-Branch Framework

• Authors: Fei Xie, Wankou Yang, Chunyu Wang, Lei Chu, Yue Cao, Chao Ma, Wenjun Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12743
• Pdf URL: https://arxiv.org/pdf/2401.12743
• Copy Paste: [[2401.12743]] Correlation-Embedded Transformer Tracking: A Single-Branch Framework(https://arxiv.org/abs/2401.12743)
• Keywords: robust, extraction, transformer
• Abstract: Developing robust and discriminative appearance models has been a long-standing research challenge in visual object tracking. In the prevalent Siamese-based paradigm, the features extracted by the Siamese-like networks are often insufficient to model the tracked targets and distractor objects, thereby hindering them from being robust and discriminative simultaneously. While most Siamese trackers focus on designing robust correlation operations, we propose a novel single-branch tracking framework inspired by the transformer. Unlike the Siamese-like feature extraction, our tracker deeply embeds cross-image feature correlation in multiple layers of the feature network. By extensively matching the features of the two images through multiple layers, it can suppress non-target features, resulting in target-aware feature extraction. The output features can be directly used for predicting target locations without additional correlation steps. Thus, we reformulate the two-branch Siamese tracking as a conceptually simple, fully transformer-based Single-Branch Tracking pipeline, dubbed SBT. After conducting an in-depth analysis of the SBT baseline, we summarize many effective design principles and propose an improved tracker dubbed SuperSBT. SuperSBT adopts a hierarchical architecture with a local modeling layer to enhance shallow-level features. A unified relation modeling is proposed to remove complex handcrafted layer pattern designs. SuperSBT is further improved by masked image modeling pre-training, integrating temporal modeling, and equipping with dedicated prediction heads. Thus, SuperSBT outperforms the SBT baseline by 4.7%,3.0%, and 4.5% AUC scores in LaSOT, TrackingNet, and GOT-10K. Notably, SuperSBT greatly raises the speed of SBT from 37 FPS to 81 FPS. Extensive experiments show that our method achieves superior results on eight VOT benchmarks.

Title: MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty

• Authors: Tim Brödermann, David Bruggemann, Christos Sakaridis, Kevin Ta, Odysseas Liagouris, Jason Corkill, Luc Van Gool
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12761
• Pdf URL: https://arxiv.org/pdf/2401.12761
• Copy Paste: [[2401.12761]] MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty(https://arxiv.org/abs/2401.12761)
• Keywords: robust, segmentation
• Abstract: Achieving level-5 driving automation in autonomous vehicles necessitates a robust semantic visual perception system capable of parsing data from different sensors across diverse conditions. However, existing semantic perception datasets often lack important non-camera modalities typically used in autonomous vehicles, or they do not exploit such modalities to aid and improve semantic annotations in challenging conditions. To address this, we introduce MUSES, the MUlti-SEnsor Semantic perception dataset for driving in adverse conditions under increased uncertainty. MUSES includes synchronized multimodal recordings with 2D panoptic annotations for 2500 images captured under diverse weather and illumination. The dataset integrates a frame camera, a lidar, a radar, an event camera, and an IMU/GNSS sensor. Our new two-stage panoptic annotation protocol captures both class-level and instance-level uncertainty in the ground truth and enables the novel task of uncertainty-aware panoptic segmentation we introduce, along with standard semantic and panoptic segmentation. MUSES proves both effective for training and challenging for evaluating models under diverse visual conditions, and it opens new avenues for research in multimodal and uncertainty-aware dense semantic perception. Our dataset and benchmark will be made publicly available.

Title: Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study

• Authors: W. Ronny Huang, Cyril Allauzen, Tongzhou Chen, Kilol Gupta, Ke Hu, James Qin, Yu Zhang, Yongqiang Wang, Shuo-Yiin Chang, Tara N. Sainath
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.12789
• Pdf URL: https://arxiv.org/pdf/2401.12789
• Copy Paste: [[2401.12789]] Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study(https://arxiv.org/abs/2401.12789)
• Keywords: large language model
• Abstract: In the era of large models, the autoregressive nature of decoding often results in latency serving as a significant bottleneck. We propose a non-autoregressive LM-fused ASR system that effectively leverages the parallelization capabilities of accelerator hardware. Our approach combines the Universal Speech Model (USM) and the PaLM 2 language model in per-segment scoring mode, achieving an average relative WER improvement across all languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our comprehensive ablation study analyzes key parameters such as LLM size, context length, vocabulary size, fusion methodology. For instance, we explore the impact of LLM size ranging from 128M to 340B parameters on ASR performance. This study provides valuable insights into the factors influencing the effectiveness of practical large-scale LM-fused speech recognition systems.

Title: Benchmarking LLMs via Uncertainty Quantification

• Authors: Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shuming Shi, Zhaopeng Tu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12794
• Pdf URL: https://arxiv.org/pdf/2401.12794
• Copy Paste: [[2401.12794]] Benchmarking LLMs via Uncertainty Quantification(https://arxiv.org/abs/2401.12794)
• Keywords: large language model
• Abstract: The proliferation of open-source Large Language Models (LLMs) from various institutions has highlighted the urgent need for comprehensive evaluation methods. However, current evaluation platforms, such as the widely recognized HuggingFace open LLM leaderboard, neglect a crucial aspect -- uncertainty, which is vital for thoroughly assessing LLMs. To bridge this gap, we introduce a new benchmarking approach for LLMs that integrates uncertainty quantification. Our examination involves eight LLMs (LLM series) spanning five representative natural language processing tasks. Additionally, we introduce an uncertainty-aware evaluation metric, UAcc, which takes into account both prediction accuracy and prediction uncertainty. Our findings reveal that: I) LLMs with higher accuracy may exhibit lower certainty; II) Larger-scale LLMs may display greater uncertainty compared to their smaller counterparts; and III) Instruction-finetuning tends to increase the uncertainty of LLMs. By taking uncertainty into account, our new UAcc metric can either amplify or diminish the relative improvement of one LLM over another and may even change the relative ranking of two LLMs. These results underscore the significance of incorporating uncertainty in the evaluation of LLMs.

Title: Dynamic Layer Tying for Parameter-Efficient Transformers

• Authors: Tamir David Hay, Lior Wolf
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12819
• Pdf URL: https://arxiv.org/pdf/2401.12819
• Copy Paste: [[2401.12819]] Dynamic Layer Tying for Parameter-Efficient Transformers(https://arxiv.org/abs/2401.12819)
• Keywords: transformer
• Abstract: In the pursuit of reducing the number of trainable parameters in deep transformer networks, we employ Reinforcement Learning to dynamically select layers during training and tie them together. Every few iterations, the RL agent is asked whether to train each layer $i$ independently or to copy the weights of a previous layer $j

Title: DatUS^2: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer

• Authors: Sonal Kumar, Arijit Sur, Rashmi Dutta Baruah
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12820
• Pdf URL: https://arxiv.org/pdf/2401.12820
• Copy Paste: [[2401.12820]] DatUS^2: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer(https://arxiv.org/abs/2401.12820)
• Keywords: transformer, segmentation
• Abstract: Successive proposals of several self-supervised training schemes continue to emerge, taking one step closer to developing a universal foundation model. In this process, the unsupervised downstream tasks are recognized as one of the evaluation methods to validate the quality of visual features learned with a self-supervised training scheme. However, unsupervised dense semantic segmentation has not been explored as a downstream task, which can utilize and evaluate the quality of semantic information introduced in patch-level feature representations during self-supervised training of a vision transformer. Therefore, this paper proposes a novel data-driven approach for unsupervised semantic segmentation (DatUS^2) as a downstream task. DatUS^2 generates semantically consistent and dense pseudo annotate segmentation masks for the unlabeled image dataset without using any visual-prior or synchronized data. We compare these pseudo-annotated segmentation masks with ground truth masks for evaluating recent self-supervised training schemes to learn shared semantic properties at the patch level and discriminative semantic properties at the segment level. Finally, we evaluate existing state-of-the-art self-supervised training schemes with our proposed downstream task, i.e., DatUS^2. Also, the best version of DatUS^2 outperforms the existing state-of-the-art method for the unsupervised dense semantic segmentation task with 15.02% MiOU and 21.47% Pixel accuracy on the SUIM dataset. It also achieves a competitive level of accuracy for a large-scale and complex dataset, i.e., the COCO dataset.

Title: MAPPING: Debiasing Graph Neural Networks for Fair Node Classification with Limited Sensitive Information Leakage

• Authors: Ying Song, Balaji Palanisamy
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.12824
• Pdf URL: https://arxiv.org/pdf/2401.12824
• Copy Paste: [[2401.12824]] MAPPING: Debiasing Graph Neural Networks for Fair Node Classification with Limited Sensitive Information Leakage(https://arxiv.org/abs/2401.12824)
• Keywords: privacy, attack, fair
• Abstract: Despite remarkable success in diverse web-based applications, Graph Neural Networks(GNNs) inherit and further exacerbate historical discrimination and social stereotypes, which critically hinder their deployments in high-stake domains such as online clinical diagnosis, financial crediting, etc. However, current fairness research that primarily craft on i.i.d data, cannot be trivially replicated to non-i.i.d. graph structures with topological dependence among samples. Existing fair graph learning typically favors pairwise constraints to achieve fairness but fails to cast off dimensional limitations and generalize them into multiple sensitive attributes; besides, most studies focus on in-processing techniques to enforce and calibrate fairness, constructing a model-agnostic debiasing GNN framework at the pre-processing stage to prevent downstream misuses and improve training reliability is still largely under-explored. Furthermore, previous work on GNNs tend to enhance either fairness or privacy individually but few probe into their interplays. In this paper, we propose a novel model-agnostic debiasing framework named MAPPING (\underline{M}asking \underline{A}nd \underline{P}runing and Message-\underline{P}assing train\underline{ING}) for fair node classification, in which we adopt the distance covariance($dCov$)-based fairness constraints to simultaneously reduce feature and topology biases in arbitrary dimensions, and combine them with adversarial debiasing to confine the risks of attribute inference attacks. Experiments on real-world datasets with different GNN variants demonstrate the effectiveness and flexibility of MAPPING. Our results show that MAPPING can achieve better trade-offs between utility and fairness, and mitigate privacy risks of sensitive information leakage.

Title: SGTR+: End-to-end Scene Graph Generation with Transformer

• Authors: Rongjie Li, Songyang Zhang, Xuming He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12835
• Pdf URL: https://arxiv.org/pdf/2401.12835
• Copy Paste: [[2401.12835]] SGTR+: End-to-end Scene Graph Generation with Transformer(https://arxiv.org/abs/2401.12835)
• Keywords: transformer
• Abstract: Scene Graph Generation (SGG) remains a challenging visual understanding task due to its compositional property. Most previous works adopt a bottom-up, two-stage or point-based, one-stage approach, which often suffers from high time complexity or suboptimal designs. In this work, we propose a novel SGG method to address the aforementioned issues, formulating the task as a bipartite graph construction problem. To address the issues above, we create a transformer-based end-to-end framework to generate the entity and entity-aware predicate proposal set, and infer directed edges to form relation triplets. Moreover, we design a graph assembling module to infer the connectivity of the bipartite scene graph based on our entity-aware structure, enabling us to generate the scene graph in an end-to-end manner. Based on bipartite graph assembling paradigm, we further propose a new technical design to address the efficacy of entity-aware modeling and optimization stability of graph assembling. Equipped with the enhanced entity-aware design, our method achieves optimal performance and time-complexity. Extensive experimental results show that our design is able to achieve the state-of-the-art or comparable performance on three challenging benchmarks, surpassing most of the existing approaches and enjoying higher efficiency in inference. Code is available: https://github.com/Scarecrow0/SGTR

Title: Iterated Relevance Matrix Analysis (IRMA) for the identification of class-discriminative subspaces

• Authors: Sofie Lövdal, Michael Biehl
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12842
• Pdf URL: https://arxiv.org/pdf/2401.12842
• Copy Paste: [[2401.12842]] Iterated Relevance Matrix Analysis (IRMA) for the identification of class-discriminative subspaces(https://arxiv.org/abs/2401.12842)
• Keywords: robust
• Abstract: We introduce and investigate the iterated application of Generalized Matrix Learning Vector Quantizaton for the analysis of feature relevances in classification problems, as well as for the construction of class-discriminative subspaces. The suggested Iterated Relevance Matrix Analysis (IRMA) identifies a linear subspace representing the classification specific information of the considered data sets using Generalized Matrix Learning Vector Quantization (GMLVQ). By iteratively determining a new discriminative subspace while projecting out all previously identified ones, a combined subspace carrying all class-specific information can be found. This facilitates a detailed analysis of feature relevances, and enables improved low-dimensional representations and visualizations of labeled data sets. Additionally, the IRMA-based class-discriminative subspace can be used for dimensionality reduction and the training of robust classifiers with potentially improved performance.

Title: Classification of grapevine varieties using UAV hyperspectral imaging

• Authors: Alfonso López, Carlos Javier Ogayar, Francisco Ramón Feito, Joaquim João Sousa
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12851
• Pdf URL: https://arxiv.org/pdf/2401.12851
• Copy Paste: [[2401.12851]] Classification of grapevine varieties using UAV hyperspectral imaging(https://arxiv.org/abs/2401.12851)
• Keywords: extraction
• Abstract: The classification of different grapevine varieties is a relevant phenotyping task in Precision Viticulture since it enables estimating the growth of vineyard rows dedicated to different varieties, among other applications concerning the wine industry. This task can be performed with destructive methods that require time-consuming tasks, including data collection and analysis in the laboratory. However, Unmanned Aerial Vehicles (UAV) provide a more efficient and less prohibitive approach to collecting hyperspectral data, despite acquiring noisier data. Therefore, the first task is the processing of these data to correct and downsample large amounts of data. In addition, the hyperspectral signatures of grape varieties are very similar. In this work, a Convolutional Neural Network (CNN) is proposed for classifying seventeen varieties of red and white grape variants. Rather than classifying single samples, these are processed together with their neighbourhood. Hence, the extraction of spatial and spectral features is addressed with 1) a spatial attention layer and 2) Inception blocks. The pipeline goes from processing to dataset elaboration, finishing with the training phase. The fitted model is evaluated in terms of response time, accuracy and data separability, and compared with other state-of-the-art CNNs for classifying hyperspectral data. Our network was proven to be much more lightweight with a reduced number of input bands, a lower number of trainable weights and therefore, reduced training time. Despite this, the evaluated metrics showed much better results for our network (~99% overall accuracy), in comparison with previous works barely achieving 81% OA.

Title: FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units

• Authors: Shaoheng Fang, Rui Ye, Wenhao Wang, Zuhong Liu, Yuxiao Wang, Yafei Wang, Siheng Chen, Yanfeng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12862
• Pdf URL: https://arxiv.org/pdf/2401.12862
• Copy Paste: [[2401.12862]] FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units(https://arxiv.org/abs/2401.12862)
• Keywords: robust, federate
• Abstract: Roadside unit (RSU) can significantly improve the safety and robustness of autonomous vehicles through Vehicle-to-Everything (V2X) communication. Currently, the usage of a single RSU mainly focuses on real-time inference and V2X collaboration, while neglecting the potential value of the high-quality data collected by RSU sensors. Integrating the vast amounts of data from numerous RSUs can provide a rich source of data for model training. However, the absence of ground truth annotations and the difficulty of transmitting enormous volumes of data are two inevitable barriers to fully exploiting this hidden value. In this paper, we introduce FedRSU, an innovative federated learning framework for self-supervised scene flow estimation. In FedRSU, we present a recurrent self-supervision training paradigm, where for each RSU, the scene flow prediction of points at every timestamp can be supervised by its subsequent future multi-modality observation. Another key component of FedRSU is federated learning, where multiple devices collaboratively train an ML model while keeping the training data local and private. With the power of the recurrent self-supervised learning paradigm, FL is able to leverage innumerable underutilized data from RSU. To verify the FedRSU framework, we construct a large-scale multi-modality dataset RSU-SF. The dataset consists of 17 RSU clients, covering various scenarios, modalities, and sensor settings. Based on RSU-SF, we show that FedRSU can greatly improve model performance in ITS and provide a comprehensive benchmark under diverse FL scenarios. To the best of our knowledge, we provide the first real-world LiDAR-camera multi-modal dataset and benchmark for the FL community.

Title: KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning

• Authors: Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, Godawari Sudhakar Rao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12863
• Pdf URL: https://arxiv.org/pdf/2401.12863
• Copy Paste: [[2401.12863]] KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning(https://arxiv.org/abs/2401.12863)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive performance in natural language processing tasks by leveraging chain of thought (CoT) that enables step-by-step thinking. Extending LLMs with multimodal capabilities is the recent interest, but incurs computational cost and requires substantial hardware resources. To address these challenges, we propose KAM-CoT a framework that integrates CoT reasoning, Knowledge Graphs (KGs), and multiple modalities for a comprehensive understanding of multimodal tasks. KAM-CoT adopts a two-stage training process with KG grounding to generate effective rationales and answers. By incorporating external knowledge from KGs during reasoning, the model gains a deeper contextual understanding reducing hallucinations and enhancing the quality of answers. This knowledge-augmented CoT reasoning empowers the model to handle questions requiring external context, providing more informed answers. Experimental findings show KAM-CoT outperforms the state-of-the-art methods. On the ScienceQA dataset, we achieve an average accuracy of 93.87%, surpassing GPT-3.5 (75.17%) by 18% and GPT-4 (83.99%) by 10%. Remarkably, KAM-CoT achieves these results with only 280M trainable parameters at a time, demonstrating its cost-efficiency and effectiveness.

Title: Unlocking the Potential: Multi-task Deep Learning for Spaceborne Quantitative Monitoring of Fugitive Methane Plumes

• Authors: Guoxin Si, Shiliang Fu, Wei Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12870
• Pdf URL: https://arxiv.org/pdf/2401.12870
• Copy Paste: [[2401.12870]] Unlocking the Potential: Multi-task Deep Learning for Spaceborne Quantitative Monitoring of Fugitive Methane Plumes(https://arxiv.org/abs/2401.12870)
• Keywords: segmentation
• Abstract: With the intensification of global warming, the monitoring of methane emission and detection of gas plumes from landfills have increasingly received attention. We decompose methane emission monitoring into three sub-tasks: methane concentration inversion, plume segmentation, and emission rate estimation. Conventional algorithms have limitations: methane concentration inversion usually uses the matched filter, which is sensitive to global spectrum distribution and contains a large amount of noises. There is limited research on plume segmentation, with many studies resorting to manual segmentation that is likely to be subjective. The estimation of methane emission rate often utilizes IME algorithm, which relies on obtaining meteorological measurement data. Using the WENT landfill site in Hong Kong and PRISMA hyperspectral satellite imagery, we propose a new deep learning-based framework for quantitative monitoring of methane emissions from remote sensing images based on physical simulation. We generate simulated methane plumes using large eddy simulation (LES) and different concentration maps of fugitive emission using the radiative transfer equation (RTE), while combining augmentation techniques to create a simulated PRISMA dataset. We train a U-Net network for methane concentration inversion, a Mask R-CNN network for methane plume segmentation, and a ResNet-50 network for methane emission rate estimation. All three deep networks achieve higher validation accuracy compared to conventional algorithms. We further respectively combine the first two sub-tasks and the last two sub-tasks to design the multi-task learning models - MTL-01 and MTL-02, both of which achieve higher accuracy than single-task models. Our research serves as a demonstration of applying multi-task deep learning to quantitative methane monitoring and can be extended to a broad range of methane monitoring tasks.

Title: From Understanding to Utilization: A Survey on Explainability for Large Language Models

• Authors: Haoyan Luo, Lucia Specia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12874
• Pdf URL: https://arxiv.org/pdf/2401.12874
• Copy Paste: [[2401.12874]] From Understanding to Utilization: A Survey on Explainability for Large Language Models(https://arxiv.org/abs/2401.12874)
• Keywords: interpretability, explainability, transformer, large language model
• Abstract: This survey paper delves into the burgeoning field of explainability for Large Language Models (LLMs), a critical yet challenging aspect of natural language processing. With LLMs playing a pivotal role in various applications, their "black-box" nature raises concerns about transparency and ethical use. This paper emphasizes the necessity for enhanced explainability in LLMs, addressing both the general public's trust and the technical community's need for a deeper understanding of these models. We concentrate on pre-trained Transformer-based LLMs, such as LLaMA, which present unique interpretability challenges due to their scale and complexity. Our review categorizes existing explainability methods and discusses their application in improving model transparency and reliability. We also discuss representative evaluation methods, highlighting their strengths and limitations. The goal of this survey is to bridge the gap between theoretical understanding and practical application, offering insights for future research and development in the field of LLM explainability.

Title: pyAKI - An Open Source Solution to Automated KDIGO classification

• Authors: Christian Porschen, Jan Ernsting, Paul Brauckmann, Raphael Weiss, Till Würdemann, Hendrik Booke, Wida Amini, Ludwig Maidowski, Benjamin Risse, Tim Hahn, Thilo von Groote
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.12930
• Pdf URL: https://arxiv.org/pdf/2401.12930
• Copy Paste: [[2401.12930]] pyAKI - An Open Source Solution to Automated KDIGO classification(https://arxiv.org/abs/2401.12930)
• Keywords: robust
• Abstract: Acute Kidney Injury (AKI) is a frequent complication in critically ill patients, affecting up to 50% of patients in the intensive care units. The lack of standardized and open-source tools for applying the Kidney Disease Improving Global Outcomes (KDIGO) criteria to time series data has a negative impact on workload and study quality. This project introduces pyAKI, an open-source pipeline addressing this gap by providing a comprehensive solution for consistent KDIGO criteria implementation. The pyAKI pipeline was developed and validated using a subset of the Medical Information Mart for Intensive Care (MIMIC)-IV database, a commonly used database in critical care research. We defined a standardized data model in order to ensure reproducibility. Validation against expert annotations demonstrated pyAKI's robust performance in implementing KDIGO criteria. Comparative analysis revealed its ability to surpass the quality of human labels. This work introduces pyAKI as an open-source solution for implementing the KDIGO criteria for AKI diagnosis using time series data with high accuracy and performance.

Title: Lumiere: A Space-Time Diffusion Model for Video Generation

• Authors: Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, Inbar Mosseri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12945
• Pdf URL: https://arxiv.org/pdf/2401.12945
• Copy Paste: [[2401.12945]] Lumiere: A Space-Time Diffusion Model for Video Generation(https://arxiv.org/abs/2401.12945)
• Keywords: diffusion
• Abstract: We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution -- an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. We demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

Title: Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion

• Authors: Dylan Zhang, Curt Tigges, Zory Zhang, Stella Biderman, Maxim Raginsky, Talia Ringer
• Subjects: cs.CL, cs.AI, cs.FL, cs.LO, cs.PL
• Abstract URL: https://arxiv.org/abs/2401.12947
• Pdf URL: https://arxiv.org/pdf/2401.12947
• Copy Paste: [[2401.12947]] Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion(https://arxiv.org/abs/2401.12947)
• Keywords: transformer, large language model
• Abstract: This paper investigates the ability of transformer-based models to learn structural recursion from examples. Recursion is a universal concept in both natural and formal languages. Structural recursion is central to the programming language and formal mathematics tasks where symbolic tools currently excel beyond neural models, such as inferring semantic relations between datatypes and emulating program behavior. We introduce a general framework that nicely connects the abstract concepts of structural recursion in the programming language domain to concrete sequence modeling problems and learned models' behavior. The framework includes a representation that captures the general \textit{syntax} of structural recursion, coupled with two different frameworks for understanding their \textit{semantics} -- one that is more natural from a programming languages perspective and one that helps bridge that perspective with a mechanistic understanding of the underlying transformer architecture. With our framework as a powerful conceptual tool, we identify different issues under various set-ups. The models trained to emulate recursive computations cannot fully capture the recursion yet instead fit short-cut algorithms and thus cannot solve certain edge cases that are under-represented in the training distribution. In addition, it is difficult for state-of-the-art large language models (LLMs) to mine recursive rules from in-context demonstrations. Meanwhile, these LLMs fail in interesting ways when emulating reduction (step-wise computation) of the recursive function.

Title: Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding

• Authors: Mirac Suzgun, Adam Tauman Kalai
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.12954
• Pdf URL: https://arxiv.org/pdf/2401.12954
• Copy Paste: [[2401.12954]] Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding(https://arxiv.org/abs/2401.12954)
• Keywords: robust
• Abstract: We introduce meta-prompting, an effective scaffolding technique designed to enhance the functionality of language models (LMs). This approach transforms a single LM into a multi-faceted conductor, adept at managing and integrating multiple independent LM queries. By employing high-level instructions, meta-prompting guides the LM to break down complex tasks into smaller, more manageable subtasks. These subtasks are then handled by distinct "expert" instances of the same LM, each operating under specific, tailored instructions. Central to this process is the LM itself, in its role as the conductor, which ensures seamless communication and effective integration of the outputs from these expert models. It additionally employs its inherent critical thinking and robust verification processes to refine and authenticate the end result. This collaborative prompting approach empowers a single LM to simultaneously act as a comprehensive orchestrator and a panel of diverse experts, significantly enhancing its performance across a wide array of tasks. The zero-shot, task-agnostic nature of meta-prompting greatly simplifies user interaction by obviating the need for detailed, task-specific instructions. Furthermore, our research demonstrates the seamless integration of external tools, such as a Python interpreter, into the meta-prompting framework, thereby broadening its applicability and utility. Through rigorous experimentation with GPT-4, we establish the superiority of meta-prompting over conventional scaffolding methods: When averaged across all tasks, including the Game of 24, Checkmate-in-One, and Python Programming Puzzles, meta-prompting, augmented with a Python interpreter functionality, surpasses standard prompting by 17.1%, expert (dynamic) prompting by 17.3%, and multipersona prompting by 15.2%.

Title: Raidar: geneRative AI Detection viA Rewriting

• Authors: Chengzhi Mao, Carl Vondrick, Hao Wang, Junfeng Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12970
• Pdf URL: https://arxiv.org/pdf/2401.12970
• Copy Paste: [[2401.12970]] Raidar: geneRative AI Detection viA Rewriting(https://arxiv.org/abs/2401.12970)
• Keywords: robust, generative, large language model
• Abstract: We find that large language models (LLMs) are more likely to modify human-written text than AI-generated text when tasked with rewriting. This tendency arises because LLMs often perceive AI-generated text as high-quality, leading to fewer modifications. We introduce a method to detect AI-generated content by prompting LLMs to rewrite text and calculating the editing distance of the output. We dubbed our geneRative AI Detection viA Rewriting method Raidar. Raidar significantly improves the F1 detection scores of existing AI content detection models -- both academic and commercial -- across various domains, including News, creative writing, student essays, code, Yelp reviews, and arXiv papers, with gains of up to 29 points. Operating solely on word symbols without high-dimensional features, our method is compatible with black box LLMs, and is inherently robust on new content. Our results illustrate the unique imprint of machine-generated text through the lens of the machines themselves.

Title: On the Efficacy of Text-Based Input Modalities for Action Anticipation

• Authors: Apoorva Beedu, Karan Samel, Irfan Essa
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.12972
• Pdf URL: https://arxiv.org/pdf/2401.12972
• Copy Paste: [[2401.12972]] On the Efficacy of Text-Based Input Modalities for Action Anticipation(https://arxiv.org/abs/2401.12972)
• Keywords: transformer
• Abstract: Although the task of anticipating future actions is highly uncertain, information from additional modalities help to narrow down plausible action choices. Each modality provides different environmental context for the model to learn from. While previous multi-modal methods leverage information from modalities such as video and audio, we primarily explore how text inputs for actions and objects can also enable more accurate action anticipation. Therefore, we propose a Multi-modal Anticipative Transformer (MAT), an attention-based video transformer architecture that jointly learns from multi-modal features and text captions. We train our model in two-stages, where the model first learns to predict actions in the video clip by aligning with captions, and during the second stage, we fine-tune the model to predict future actions. Compared to existing methods, MAT has the advantage of learning additional environmental context from two kinds of text inputs: action descriptions during the pre-training stage, and the text inputs for detected objects and actions during modality feature fusion. Through extensive experiments, we evaluate the effectiveness of the pre-training stage, and show that our model outperforms previous methods on all datasets. In addition, we examine the impact of object and action information obtained via text and perform extensive ablations. We evaluate the performance on on three datasets: EpicKitchens-100, EpicKitchens-55 and EGTEA GAZE+; and show that text descriptions do indeed aid in more effective action anticipation.

Title: In-Context Language Learning: Arhitectures and Algorithms

• Authors: Ekin Akyürek, Bailin Wang, Yoon Kim, Jacob Andreas
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.12973
• Pdf URL: https://arxiv.org/pdf/2401.12973
• Copy Paste: [[2401.12973]] In-Context Language Learning: Arhitectures and Algorithms(https://arxiv.org/abs/2401.12973)
• Keywords: transformer
• Abstract: Large-scale neural language models exhibit a remarkable capacity for in-context learning (ICL): they can infer novel functions from datasets provided as input. Most of our current understanding of when and how ICL arises comes from LMs trained on extremely simple learning problems like linear regression and associative recall. There remains a significant gap between these model problems and the "real" ICL exhibited by LMs trained on large text corpora, which involves not just retrieval and function approximation but free-form generation of language and other structured outputs. In this paper, we study ICL through the lens of a new family of model problems we term in context language learning (ICLL). In ICLL, LMs are presented with a set of strings from a formal language, and must generate additional strings from the same language. We focus on in-context learning of regular languages generated by random finite automata. We evaluate a diverse set of neural sequence models (including several RNNs, Transformers, and state-space model variants) on regular ICLL tasks, aiming to answer three questions: (1) Which model classes are empirically capable of ICLL? (2) What algorithmic solutions do successful models implement to perform ICLL? (3) What architectural changes can improve ICLL in less performant models? We first show that Transformers significantly outperform neural sequence models with recurrent or convolutional representations on ICLL tasks. Next, we provide evidence that their ability to do so relies on specialized "n-gram heads" (higher-order variants of induction heads) that compute input-conditional next-token distributions. Finally, we show that hard-wiring these heads into recurrent and convolutional models improves performance not just on ICLL, but natural language modeling -- improving the perplexity of 340M-parameter models by up to 1.14 points (6.7%) on the SlimPajama dataset.

Title: HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments

• Authors: Qinhong Zhou, Sunli Chen, Yisong Wang, Haozhe Xu, Weihua Du, Hongxin Zhang, Yilun Du, Joshua B. Tenenbaum, Chuang Gan
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12975
• Pdf URL: https://arxiv.org/pdf/2401.12975
• Copy Paste: [[2401.12975]] HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments(https://arxiv.org/abs/2401.12975)
• Keywords: large language model
• Abstract: Recent advances in high-fidelity virtual environments serve as one of the major driving forces for building intelligent embodied agents to perceive, reason and interact with the physical world. Typically, these environments remain unchanged unless agents interact with them. However, in real-world scenarios, agents might also face dynamically changing environments characterized by unexpected events and need to rapidly take action accordingly. To remedy this gap, we propose a new simulated embodied benchmark, called HAZARD, specifically designed to assess the decision-making abilities of embodied agents in dynamic situations. HAZARD consists of three unexpected disaster scenarios, including fire, flood, and wind, and specifically supports the utilization of large language models (LLMs) to assist common sense reasoning and decision-making. This benchmark enables us to evaluate autonomous agents' decision-making capabilities across various pipelines, including reinforcement learning (RL), rule-based, and search-based methods in dynamically changing environments. As a first step toward addressing this challenge using large language models, we further develop an LLM-based agent and perform an in-depth analysis of its promise and challenge of solving these challenging tasks. HAZARD is available at https://vis-www.cs.umass.edu/hazard/.

Title: Zero-Shot Learning for the Primitives of 3D Affordance in General Objects

• Authors: Hyeonwoo Kim, Sookwan Han, Patrick Kwon, Hanbyul Joo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12978
• Pdf URL: https://arxiv.org/pdf/2401.12978
• Copy Paste: [[2401.12978]] Zero-Shot Learning for the Primitives of 3D Affordance in General Objects(https://arxiv.org/abs/2401.12978)
• Keywords: diffusion
• Abstract: One of the major challenges in AI is teaching machines to precisely respond and utilize environmental functionalities, thereby achieving the affordance awareness that humans possess. Despite its importance, the field has been lagging in terms of learning, especially in 3D, as annotating affordance accompanies a laborious process due to the numerous variations of human-object interaction. The low availability of affordance data limits the learning in terms of generalization for object categories, and also simplifies the representation of affordance, capturing only a fraction of the affordance. To overcome these challenges, we propose a novel, self-supervised method to generate the 3D affordance examples given only a 3D object, without any manual annotations. The method starts by capturing the 3D object into images and creating 2D affordance images by inserting humans into the image via inpainting diffusion models, where we present the Adaptive Mask algorithm to enable human insertion without altering the original details of the object. The method consequently lifts inserted humans back to 3D to create 3D human-object pairs, where the depth ambiguity is resolved within a depth optimization framework that utilizes pre-generated human postures from multiple viewpoints. We also provide a novel affordance representation defined on relative orientations and proximity between dense human and object points, that can be easily aggregated from any 3D HOI datasets. The proposed representation serves as a primitive that can be manifested to conventional affordance representations via simple transformations, ranging from physically exerted affordances to nonphysical ones. We demonstrate the efficacy of our method and representation by generating the 3D affordance samples and deriving high-quality affordance examples from the representation, including contact, orientation, and spatial occupancies.

Title: GALA: Generating Animatable Layered Assets from a Single Scan

• Authors: Taeksoo Kim, Byungjun Kim, Shunsuke Saito, Hanbyul Joo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12979
• Pdf URL: https://arxiv.org/pdf/2401.12979
• Copy Paste: [[2401.12979]] GALA: Generating Animatable Layered Assets from a Single Scan(https://arxiv.org/abs/2401.12979)
• Keywords: diffusion, segmentation
• Abstract: We present GALA, a framework that takes as input a single-layer clothed 3D human mesh and decomposes it into complete multi-layered 3D assets. The outputs can then be combined with other assets to create novel clothed human avatars with any pose. Existing reconstruction approaches often treat clothed humans as a single-layer of geometry and overlook the inherent compositionality of humans with hairstyles, clothing, and accessories, thereby limiting the utility of the meshes for downstream applications. Decomposing a single-layer mesh into separate layers is a challenging task because it requires the synthesis of plausible geometry and texture for the severely occluded regions. Moreover, even with successful decomposition, meshes are not normalized in terms of poses and body shapes, failing coherent composition with novel identities and poses. To address these challenges, we propose to leverage the general knowledge of a pretrained 2D diffusion model as geometry and appearance prior for humans and other assets. We first separate the input mesh using the 3D surface segmentation extracted from multi-view 2D segmentations. Then we synthesize the missing geometry of different layers in both posed and canonical spaces using a novel pose-guided Score Distillation Sampling (SDS) loss. Once we complete inpainting high-fidelity 3D geometry, we also apply the same SDS loss to its texture to obtain the complete appearance including the initially occluded regions. Through a series of decomposition steps, we obtain multiple layers of 3D assets in a shared canonical space normalized in terms of poses and human shapes, hence supporting effortless composition to novel identities and reanimation with novel poses. Our experiments demonstrate the effectiveness of our approach for decomposition, canonicalization, and composition tasks compared to existing solutions.