2024-02-28

Title: BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation

• Authors: Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, Ping Luo
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16880
• Pdf URL: https://arxiv.org/pdf/2402.16880
• Copy Paste: [[2402.16880]] BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation(https://arxiv.org/abs/2402.16880)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) have demonstrated outstanding performance in various tasks, such as text summarization, text question-answering, and etc. While their performance is impressive, the computational footprint due to their vast number of parameters can be prohibitive. Existing solutions such as SparseGPT and Wanda attempt to alleviate this issue through weight pruning. However, their layer-wise approach results in significant perturbation to the model's output and requires meticulous hyperparameter tuning, such as the pruning rate, which can adversely affect overall model performance. To address this, this paper introduces a novel LLM pruning technique dubbed blockwise parameter-efficient sparsity allocation (BESA) by applying a blockwise reconstruction loss. In contrast to the typical layer-wise pruning techniques, BESA is characterized by two distinctive attributes: i) it targets the overall pruning error with respect to individual transformer blocks, and ii) it allocates layer-specific sparsity in a differentiable manner, both of which ensure reduced performance degradation after pruning. Our experiments show that BESA achieves state-of-the-art performance, efficiently pruning LLMs like LLaMA1, and LLaMA2 with 7B to 70B parameters on a single A100 GPU in just five hours. Code is available at \href{https://github.com/OpenGVLab/LLMPrune-BESA}{here}.

Title: Generative Models are Self-Watermarked: Declaring Model Authentication through Re-Generation

• Authors: Aditya Desu, Xuanli He, Qiongkai Xu, Wei Lu
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16889
• Pdf URL: https://arxiv.org/pdf/2402.16889
• Copy Paste: [[2402.16889]] Generative Models are Self-Watermarked: Declaring Model Authentication through Re-Generation(https://arxiv.org/abs/2402.16889)
• Keywords: protect, robust, watermark, generative
• Abstract: As machine- and AI-generated content proliferates, protecting the intellectual property of generative models has become imperative, yet verifying data ownership poses formidable challenges, particularly in cases of unauthorized reuse of generated data. The challenge of verifying data ownership is further amplified by using Machine Learning as a Service (MLaaS), which often functions as a black-box system. Our work is dedicated to detecting data reuse from even an individual sample. Traditionally, watermarking has been leveraged to detect AI-generated content. However, unlike watermarking techniques that embed additional information as triggers into models or generated content, potentially compromising output quality, our approach identifies latent fingerprints inherently present within the outputs through re-generation. We propose an explainable verification procedure that attributes data ownership through re-generation, and further amplifies these fingerprints in the generative models through iterative data re-generation. This methodology is theoretically grounded and demonstrates viability and robustness using recent advanced text and image generative models. Our methodology is significant as it goes beyond protecting the intellectual property of APIs and addresses important issues such as the spread of misinformation and academic misconduct. It provides a useful tool to ensure the integrity of sources and authorship, expanding its application in different scenarios where authenticity and ownership verification are essential.

Title: The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)

• Authors: Shenglai Zeng, Jiankun Zhang, Pengfei He, Yue Xing, Yiding Liu, Han Xu, Jie Ren, Shuaiqiang Wang, Dawei Yin, Yi Chang, Jiliang Tang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16893
• Pdf URL: https://arxiv.org/pdf/2402.16893
• Copy Paste: [[2402.16893]] The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)(https://arxiv.org/abs/2402.16893)
• Keywords: privacy, protect, attack, large language model
• Abstract: Retrieval-augmented generation (RAG) is a powerful technique to facilitate language model with proprietary and private data, where data privacy is a pivotal concern. Whereas extensive research has demonstrated the privacy risks of large language models (LLMs), the RAG technique could potentially reshape the inherent behaviors of LLM generation, posing new privacy issues that are currently under-explored. In this work, we conduct extensive empirical studies with novel attack methods, which demonstrate the vulnerability of RAG systems on leaking the private retrieval database. Despite the new risk brought by RAG on the retrieval data, we further reveal that RAG can mitigate the leakage of the LLMs' training data. Overall, we provide new insights in this paper for privacy protection of retrieval-augmented LLMs, which benefit both LLMs and RAG systems builders. Our code is available at https://github.com/phycholosogy/RAG-privacy.

Title: On Trojan Signatures in Large Language Models of Code

• Authors: Aftab Hussain, Md Rafiqul Islam Rabin, Mohammad Amin Alipour
• Subjects: cs.CR, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.16896
• Pdf URL: https://arxiv.org/pdf/2402.16896
• Copy Paste: [[2402.16896]] On Trojan Signatures in Large Language Models of Code(https://arxiv.org/abs/2402.16896)
• Keywords: large language model
• Abstract: Trojan signatures, as described by Fields et al. (2021), are noticeable differences in the distribution of the trojaned class parameters (weights) and the non-trojaned class parameters of the trojaned model, that can be used to detect the trojaned model. Fields et al. (2021) found trojan signatures in computer vision classification tasks with image models, such as, Resnet, WideResnet, Densenet, and VGG. In this paper, we investigate such signatures in the classifier layer parameters of large language models of source code. Our results suggest that trojan signatures could not generalize to LLMs of code. We found that trojaned code models are stubborn, even when the models were poisoned under more explicit settings (finetuned with pre-trained weights frozen). We analyzed nine trojaned models for two binary classification tasks: clone and defect detection. To the best of our knowledge, this is the first work to examine weight-based trojan signature revelation techniques for large-language models of code and furthermore to demonstrate that detecting trojans only from the weights in such models is a hard problem.

Title: PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA

• Authors: Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, Chuan Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16902
• Pdf URL: https://arxiv.org/pdf/2402.16902
• Copy Paste: [[2402.16902]] PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA(https://arxiv.org/abs/2402.16902)
• Keywords: large language model
• Abstract: With the rapid scaling of large language models (LLMs), serving numerous LoRAs concurrently has become increasingly impractical, leading to unaffordable costs and necessitating more parameter-efficient finetuning methods. In this work, we introduce Partially Rotation-enhanced Low-Rank Adaptation (PRoLoRA), an intra-layer sharing mechanism comprising four essential components: broadcast reduction, rotation enhancement, partially-sharing refinement, and rectified initialization strategy. As a superset of LoRA, PRoLoRA pertains its advantages, and effectively circumvent the drawbacks of peer parameter-sharing methods with superior model capacity, practical feasibility, and broad applicability. Empirical experiments demonstrate the remarkably higher parameter efficiency of PRoLoRA in both specific parameter budget and performance target scenarios, and its scalability to larger LLMs. Notably, with one time less trainable parameters, PRoLoRA still outperforms LoRA on multiple instruction tuning datasets. Subsequently, an ablation study is conducted to validate the necessity of individual components and highlight the superiority of PRoLoRA over three potential variants. Hopefully, the conspicuously higher parameter efficiency can establish PRoLoRA as a resource-friendly alternative to LoRA.

Title: Trustworthy Personalized Bayesian Federated Learning via Posterior Fine-Tune

• Authors: Mengen Luo, Chi Xu, Ercan Engin Kuruoglu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16911
• Pdf URL: https://arxiv.org/pdf/2402.16911
• Copy Paste: [[2402.16911]] Trustworthy Personalized Bayesian Federated Learning via Posterior Fine-Tune(https://arxiv.org/abs/2402.16911)
• Keywords: robust, federate, interpretability
• Abstract: Performance degradation owing to data heterogeneity and low output interpretability are the most significant challenges faced by federated learning in practical applications. Personalized federated learning diverges from traditional approaches, as it no longer seeks to train a single model, but instead tailors a unique personalized model for each client. However, previous work focused only on personalization from the perspective of neural network parameters and lack of robustness and interpretability. In this work, we establish a novel framework for personalized federated learning, incorporating Bayesian methodology which enhances the algorithm's ability to quantify uncertainty. Furthermore, we introduce normalizing flow to achieve personalization from the parameter posterior perspective and theoretically analyze the impact of normalizing flow on out-of-distribution (OOD) detection for Bayesian neural networks. Finally, we evaluated our approach on heterogeneous datasets, and the experimental results indicate that the new algorithm not only improves accuracy but also outperforms the baseline significantly in OOD detection due to the reliable output of the Bayesian approach.

Title: An Adversarial Robustness Benchmark for Enterprise Network Intrusion Detection

• Authors: João Vitorino, Miguel Silva, Eva Maia, Isabel Praça
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16912
• Pdf URL: https://arxiv.org/pdf/2402.16912
• Copy Paste: [[2402.16912]] An Adversarial Robustness Benchmark for Enterprise Network Intrusion Detection(https://arxiv.org/abs/2402.16912)
• Keywords: attack, robust
• Abstract: As cyber-attacks become more sophisticated, improving the robustness of Machine Learning (ML) models must be a priority for enterprises of all sizes. To reliably compare the robustness of different ML models for cyber-attack detection in enterprise computer networks, they must be evaluated in standardized conditions. This work presents a methodical adversarial robustness benchmark of multiple decision tree ensembles with constrained adversarial examples generated from standard datasets. The robustness of regularly and adversarially trained RF, XGB, LGBM, and EBM models was evaluated on the original CICIDS2017 dataset, a corrected version of it designated as NewCICIDS, and the HIKARI dataset, which contains more recent network traffic. NewCICIDS led to models with a better performance, especially XGB and EBM, but RF and LGBM were less robust against the more recent cyber-attacks of HIKARI. Overall, the robustness of the models to adversarial cyber-attack examples was improved without their generalization to regular traffic being affected, enabling a reliable detection of suspicious activity without costly increases of false alarms.

Title: DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers

• Authors: Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, Cho-Jui Hsieh
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.16914
• Pdf URL: https://arxiv.org/pdf/2402.16914
• Copy Paste: [[2402.16914]] DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers(https://arxiv.org/abs/2402.16914)
• Keywords: attack, large language model
• Abstract: The safety alignment of Large Language Models (LLMs) is vulnerable to both manual and automated jailbreak attacks, which adversarially trigger LLMs to output harmful content. However, current methods for jailbreaking LLMs, which nest entire harmful prompts, are not effective at concealing malicious intent and can be easily identified and rejected by well-aligned LLMs. This paper discovers that decomposing a malicious prompt into separated sub-prompts can effectively obscure its underlying malicious intent by presenting it in a fragmented, less detectable form, thereby addressing these limitations. We introduce an automatic prompt \textbf{D}ecomposition and \textbf{R}econstruction framework for jailbreak \textbf{Attack} (DrAttack). DrAttack includes three key components: (a) `Decomposition' of the original prompt into sub-prompts, (b) `Reconstruction' of these sub-prompts implicitly by in-context learning with semantically similar but harmless reassembling demo, and (c) a `Synonym Search' of sub-prompts, aiming to find sub-prompts' synonyms that maintain the original intent while jailbreaking LLMs. An extensive empirical study across multiple open-source and closed-source LLMs demonstrates that, with a significantly reduced number of queries, DrAttack obtains a substantial gain of success rate over prior SOTA prompt-only attackers. Notably, the success rate of 78.0\% on GPT-4 with merely 15 queries surpassed previous art by 33.1\%.

Title: More Than Routing: Joint GPS and Route Modeling for Refine Trajectory Representation Learning

• Authors: Zhipeng Ma, Zheyan Tu, Xinhai Chen, Yan Zhang, Deguo Xia, Guyue Zhou, Yilun Chen, Yu Zheng, Jiangtao Gong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16915
• Pdf URL: https://arxiv.org/pdf/2402.16915
• Copy Paste: [[2402.16915]] More Than Routing: Joint GPS and Route Modeling for Refine Trajectory Representation Learning(https://arxiv.org/abs/2402.16915)
• Keywords: transformer
• Abstract: Trajectory representation learning plays a pivotal role in supporting various downstream tasks. Traditional methods in order to filter the noise in GPS trajectories tend to focus on routing-based methods used to simplify the trajectories. However, this approach ignores the motion details contained in the GPS data, limiting the representation capability of trajectory representation learning. To fill this gap, we propose a novel representation learning framework that Joint GPS and Route Modelling based on self-supervised technology, namely JGRM. We consider GPS trajectory and route as the two modes of a single movement observation and fuse information through inter-modal information interaction. Specifically, we develop two encoders, each tailored to capture representations of route and GPS trajectories respectively. The representations from the two modalities are fed into a shared transformer for inter-modal information interaction. Eventually, we design three self-supervised tasks to train the model. We validate the effectiveness of the proposed method on two real datasets based on extensive experiments. The experimental results demonstrate that JGRM outperforms existing methods in both road segment representation and trajectory representation tasks. Our source code is available at Anonymous Github.

Title: m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers

• Authors: Ka Man Lo, Yiming Liang, Wenyu Du, Yuantao Fan, Zili Wang, Wenhao Huang, Lei Ma, Jie Fu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.16918
• Pdf URL: https://arxiv.org/pdf/2402.16918
• Copy Paste: [[2402.16918]] m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers(https://arxiv.org/abs/2402.16918)
• Keywords: robust, transformer
• Abstract: Modular neural architectures are gaining increasing attention due to their powerful capability for generalization and sample-efficient adaptation to new domains. However, training modular models, particularly in the early stages, poses challenges due to the optimization difficulties arising from their intrinsic sparse connectivity. Leveraging the knowledge from monolithic models, using techniques such as knowledge distillation, is likely to facilitate the training of modular models and enable them to integrate knowledge from multiple models pretrained on diverse sources. Nevertheless, conventional knowledge distillation approaches are not tailored to modular models and can fail when directly applied due to the unique architectures and the enormous number of parameters involved. Motivated by these challenges, we propose a general module-to-module knowledge distillation (m2mKD) method for transferring knowledge between modules. Our approach involves teacher modules split from a pretrained monolithic model, and student modules of a modular model. m2mKD separately combines these modules with a shared meta model and encourages the student module to mimic the behaviour of the teacher module. We evaluate the effectiveness of m2mKD on two distinct modular neural architectures: Neural Attentive Circuits (NACs) and Vision Mixture-of-Experts (V-MoE). By applying m2mKD to NACs, we achieve significant improvements in IID accuracy on Tiny-ImageNet (up to 5.6%) and OOD robustness on Tiny-ImageNet-R (up to 4.2%). On average, we observe a 1% gain in both ImageNet and ImageNet-R. The V-MoE-Base model trained using m2mKD also achieves 3.5% higher accuracy than end-to-end training on ImageNet. The experimental results demonstrate that our method offers a promising solution for connecting modular networks with pretrained monolithic models. Code is available at https://github.com/kamanphoebe/m2mKD.

Title: Personalized Federated Instruction Tuning via Neural Architecture Search

• Authors: Pengyu Zhang, Yingbo Zhou, Ming Hu, Junxian Feng, Jiawen Weng, Mingsong Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16919
• Pdf URL: https://arxiv.org/pdf/2402.16919
• Copy Paste: [[2402.16919]] Personalized Federated Instruction Tuning via Neural Architecture Search(https://arxiv.org/abs/2402.16919)
• Keywords: federate
• Abstract: Federated Instruction Tuning (FIT) has shown the ability to achieve collaborative model instruction tuning among massive data owners without sharing private data. However, it still faces two key challenges, i.e., data and resource heterogeneity. Due to the varying data distribution and preferences among data owners, FIT cannot adapt to the personalized data of individual owners. Moreover, clients with superior computational abilities are constrained since they need to maintain the same fine-tuning architecture as the weaker clients. To address these issues, we propose a novel Personalized Federated Instruction Tuning (PerFIT) framework based on architecture search. Specifically, PerFIT allows each client to search for a personalized architecture by expanding the trainable parameter space of the global model followed by pruning the parameters to the original state. This procedure allows personalized instruction fine-tuning within expanded parameter spaces, concurrently preserving the same number of trainable parameters. Furthermore, to release the abilities of heterogeneous computational resources and enhance the performance of personalization on local data, we exploit personalized parameter-wise aggregation. The evaluation with multiple LLMs non-IID scenarios demonstrates that compared to the state-of-the-art FIT methods, our approach can achieve up to a 23% decrease in perplexity.

Title: FedReview: A Review Mechanism for Rejecting Poisoned Updates in Federated Learning

• Authors: Tianhang Zheng, Baochun Li
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.16934
• Pdf URL: https://arxiv.org/pdf/2402.16934
• Copy Paste: [[2402.16934]] FedReview: A Review Mechanism for Rejecting Poisoned Updates in Federated Learning(https://arxiv.org/abs/2402.16934)
• Keywords: federate
• Abstract: Federated learning has recently emerged as a decentralized approach to learn a high-performance model without access to user data. Despite its effectiveness, federated learning gives malicious users opportunities to manipulate the model by uploading poisoned model updates to the server. In this paper, we propose a review mechanism called FedReview to identify and decline the potential poisoned updates in federated learning. Under our mechanism, the server randomly assigns a subset of clients as reviewers to evaluate the model updates on their training datasets in each round. The reviewers rank the model updates based on the evaluation results and count the number of the updates with relatively low quality as the estimated number of poisoned updates. Based on review reports, the server employs a majority voting mechanism to integrate the rankings and remove the potential poisoned updates in the model aggregation process. Extensive evaluation on multiple datasets demonstrate that FedReview can assist the server to learn a well-performed global model in an adversarial environment.

Title: WIPI: A New Web Threat for LLM-Driven Web Agents

• Authors: Fangzhou Wu, Shutong Wu, Yulong Cao, Chaowei Xiao
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16965
• Pdf URL: https://arxiv.org/pdf/2402.16965
• Copy Paste: [[2402.16965]] WIPI: A New Web Threat for LLM-Driven Web Agents(https://arxiv.org/abs/2402.16965)
• Keywords: secure, attack, robust, steal, large language model
• Abstract: With the fast development of large language models (LLMs), LLM-driven Web Agents (Web Agents for short) have obtained tons of attention due to their superior capability where LLMs serve as the core part of making decisions like the human brain equipped with multiple web tools to actively interact with external deployed websites. As uncountable Web Agents have been released and such LLM systems are experiencing rapid development and drawing closer to widespread deployment in our daily lives, an essential and pressing question arises: "Are these Web Agents secure?". In this paper, we introduce a novel threat, WIPI, that indirectly controls Web Agent to execute malicious instructions embedded in publicly accessible webpages. To launch a successful WIPI works in a black-box environment. This methodology focuses on the form and content of indirect instructions within external webpages, enhancing the efficiency and stealthiness of the attack. To evaluate the effectiveness of the proposed methodology, we conducted extensive experiments using 7 plugin-based ChatGPT Web Agents, 8 Web GPTs, and 3 different open-source Web Agents. The results reveal that our methodology achieves an average attack success rate (ASR) exceeding 90% even in pure black-box scenarios. Moreover, through an ablation study examining various user prefix instructions, we demonstrated that the WIPI exhibits strong robustness, maintaining high performance across diverse prefix instructions.

Title: A Survey of Large Language Models in Cybersecurity

• Authors: Gabriel de Jesus Coelho da Silva, Carlos Becker Westphall
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.16968
• Pdf URL: https://arxiv.org/pdf/2402.16968
• Copy Paste: [[2402.16968]] A Survey of Large Language Models in Cybersecurity(https://arxiv.org/abs/2402.16968)
• Keywords: security, large language model
• Abstract: Large Language Models (LLMs) have quickly risen to prominence due to their ability to perform at or close to the state-of-the-art in a variety of fields while handling natural language. An important field of research is the application of such models at the cybersecurity context. This survey aims to identify where in the field of cybersecurity LLMs have already been applied, the ways in which they are being used and their limitations in the field. Finally, suggestions are made on how to improve such limitations and what can be expected from these systems once these limitations are overcome.

Title: Synthesizing Tight Privacy and Accuracy Bounds via Weighted Model Counting

• Authors: Lisa Oakley, Steven Holtzen, Alina Oprea
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2402.16982
• Pdf URL: https://arxiv.org/pdf/2402.16982
• Copy Paste: [[2402.16982]] Synthesizing Tight Privacy and Accuracy Bounds via Weighted Model Counting(https://arxiv.org/abs/2402.16982)
• Keywords: privacy
• Abstract: Programmatically generating tight differential privacy (DP) bounds is a hard problem. Two core challenges are (1) finding expressive, compact, and efficient encodings of the distributions of DP algorithms, and (2) state space explosion stemming from the multiple quantifiers and relational properties of the DP definition. We address the first challenge by developing a method for tight privacy and accuracy bound synthesis using weighted model counting on binary decision diagrams, a state of the art technique from the artificial intelligence and automated reasoning communities for exactly computing probability distributions. We address the second challenge by developing a framework for leveraging inherent symmetries in DP algorithms. Our solution benefits from ongoing research in probabilistic programming languages, allowing us to succinctly and expressively represent different DP algorithms with approachable language syntax that can be used by non-experts. We provide a detailed case study of our solution on the binary randomized response algorithm. We also evaluate an implementation of our solution using the Dice probabilistic programming language for the randomized response and truncated geometric above threshold algorithms. We compare to prior work on exact DP verification using Markov chain probabilistic model checking. Very few existing works consider mechanized analysis of accuracy guarantees for DP algorithms. We additionally provide a detailed analysis using our technique for finding tight accuracy bounds for DP algorithms.

Title: GEM3D: GEnerative Medial Abstractions for 3D Shape Synthesis

• Authors: Dmitry Petrov, Pradyumn Goyal, Vikas Thamizharasan, Vladimir G. Kim, Matheus Gadelha, Melinos Averkiou, Siddhartha Chaudhuri, Evangelos Kalogerakis
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.16994
• Pdf URL: https://arxiv.org/pdf/2402.16994
• Copy Paste: [[2402.16994]] GEM3D: GEnerative Medial Abstractions for 3D Shape Synthesis(https://arxiv.org/abs/2402.16994)
• Keywords: diffusion, generative
• Abstract: We introduce GEM3D -- a new deep, topology-aware generative model of 3D shapes. The key ingredient of our method is a neural skeleton-based representation encoding information on both shape topology and geometry. Through a denoising diffusion probabilistic model, our method first generates skeleton-based representations following the Medial Axis Transform (MAT), then generates surfaces through a skeleton-driven neural implicit formulation. The neural implicit takes into account the topological and geometric information stored in the generated skeleton representations to yield surfaces that are more topologically and geometrically accurate compared to previous neural field formulations. We discuss applications of our method in shape synthesis and point cloud reconstruction tasks, and evaluate our method both qualitatively and quantitatively. We demonstrate significantly more faithful surface reconstruction and diverse shape generation results compared to the state-of-the-art, also involving challenging scenarios of reconstructing and synthesizing structurally complex, high-genus shape surfaces from Thingi10K and ShapeNet.

Title: Discovering Symmetry Group Structures via Implicit Orthogonality Bias

• Authors: Dongsung Huh
• Subjects: cs.LG, math.GR, math.RT
• Abstract URL: https://arxiv.org/abs/2402.17002
• Pdf URL: https://arxiv.org/pdf/2402.17002
• Copy Paste: [[2402.17002]] Discovering Symmetry Group Structures via Implicit Orthogonality Bias(https://arxiv.org/abs/2402.17002)
• Keywords: transformer
• Abstract: We introduce the HyperCube network, a novel approach for autonomously discovering symmetry group structures within data. The key innovation is a unique factorization architecture coupled with a novel regularizer that instills a powerful inductive bias towards learning orthogonal representations. This leverages a fundamental theorem of representation theory that all compact/finite groups can be represented by orthogonal matrices. HyperCube efficiently learns general group operations from partially observed data, successfully recovering complete operation tables. Remarkably, the learned factors correspond directly to exact matrix representations of the underlying group. Moreover, these factors capture the group's complete set of irreducible representations, forming the generalized Fourier basis for performing group convolutions. In extensive experiments with both group and non-group symbolic operations, HyperCube demonstrates a dramatic 100-1000x improvement in training speed and 2-10x greater sample efficiency compared to the Transformer baseline. These results suggest that our approach unlocks a new class of deep learning models capable of harnessing inherent symmetries within data, leading to significant improvements in performance and broader applicability.

Title: Benchmarking LLMs on the Semantic Overlap Summarization Task

• Authors: John Salvador, Naman Bansal, Mousumi Akter, Souvika Sarkar, Anupam Das, Shubhra Kanti Karmaker ("Santu")
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17008
• Pdf URL: https://arxiv.org/pdf/2402.17008
• Copy Paste: [[2402.17008]] Benchmarking LLMs on the Semantic Overlap Summarization Task(https://arxiv.org/abs/2402.17008)
• Keywords: large language model
• Abstract: Semantic Overlap Summarization (SOS) is a constrained multi-document summarization task, where the constraint is to capture the common/overlapping information between two alternative narratives. While recent advancements in Large Language Models (LLMs) have achieved superior performance in numerous summarization tasks, a benchmarking study of the SOS task using LLMs is yet to be performed. As LLMs' responses are sensitive to slight variations in prompt design, a major challenge in conducting such a benchmarking study is to systematically explore a variety of prompts before drawing a reliable conclusion. Fortunately, very recently, the TELeR taxonomy has been proposed which can be used to design and explore various prompts for LLMs. Using this TELeR taxonomy and 15 popular LLMs, this paper comprehensively evaluates LLMs on the SOS Task, assessing their ability to summarize overlapping information from multiple alternative narratives. For evaluation, we report well-established metrics like ROUGE, BERTscore, and SEM-F1$ on two different datasets of alternative narratives. We conclude the paper by analyzing the strengths and limitations of various LLMs in terms of their capabilities in capturing overlapping information The code and datasets used to conduct this study are available at https://anonymous.4open.science/r/llm_eval-E16D.

Title: Can Large Language Models Recall Reference Location Like Humans?

• Authors: Ye Wang, Xinrun Xu, Rui Xie, Wenxin Hu, Wei Ye
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17010
• Pdf URL: https://arxiv.org/pdf/2402.17010
• Copy Paste: [[2402.17010]] Can Large Language Models Recall Reference Location Like Humans?(https://arxiv.org/abs/2402.17010)
• Keywords: large language model
• Abstract: When completing knowledge-intensive tasks, humans sometimes need not just an answer but also a corresponding reference passage for auxiliary reading. Previous methods required obtaining pre-segmented article chunks through additional retrieval models. This paper explores leveraging the parameterized knowledge stored during the pre-training phase of large language models (LLMs) to independently recall reference passage from any starting position. We propose a two-stage framework that simulates the scenario of humans recalling easily forgotten references. Initially, the LLM is prompted to recall document title identifiers to obtain a coarse-grained document set. Then, based on the acquired coarse-grained document set, it recalls fine-grained passage. In the two-stage recall process, we use constrained decoding to ensure that content outside of the stored documents is not generated. To increase speed, we only recall a short prefix in the second stage, then locate its position to retrieve a complete passage. Experiments on KILT knowledge-sensitive tasks have verified that LLMs can independently recall reference passage location in various task forms, and the obtained reference significantly assist downstream tasks.

Title: DiffuCOMET: Contextual Commonsense Knowledge Diffusion

• Authors: Silin Gao, Mete Ismayilzada, Mengjie Zhao, Hiromi Wakaki, Yuki Mitsufuji, Antoine Bosselut
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17011
• Pdf URL: https://arxiv.org/pdf/2402.17011
• Copy Paste: [[2402.17011]] DiffuCOMET: Contextual Commonsense Knowledge Diffusion(https://arxiv.org/abs/2402.17011)
• Keywords: diffusion
• Abstract: Inferring contextually-relevant and diverse commonsense to understand narratives remains challenging for knowledge models. In this work, we develop a series of knowledge models, DiffuCOMET, that leverage diffusion to learn to reconstruct the implicit semantic connections between narrative contexts and relevant commonsense knowledge. Across multiple diffusion steps, our method progressively refines a representation of commonsense facts that is anchored to a narrative, producing contextually-relevant and diverse commonsense inferences for an input context. To evaluate DiffuCOMET, we introduce new metrics for commonsense inference that more closely measure knowledge diversity and contextual relevance. Our results on two different benchmarks, ComFact and WebNLG+, show that knowledge generated by DiffuCOMET achieves a better trade-off between commonsense diversity, contextual relevance and alignment to known gold references, compared to baseline knowledge models.

Title: Pandora's White-Box: Increased Training Data Leakage in Open LLMs

• Authors: Jeffrey G. Wang, Jason Wang, Marvin Li, Seth Neel
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17012
• Pdf URL: https://arxiv.org/pdf/2402.17012
• Copy Paste: [[2402.17012]] Pandora's White-Box: Increased Training Data Leakage in Open LLMs(https://arxiv.org/abs/2402.17012)
• Keywords: privacy, attack, membership infer, large language model
• Abstract: In this paper we undertake a systematic study of privacy attacks against open source Large Language Models (LLMs), where an adversary has access to either the model weights, gradients, or losses, and tries to exploit them to learn something about the underlying training data. Our headline results are the first membership inference attacks (MIAs) against pre-trained LLMs that are able to simultaneously achieve high TPRs and low FPRs, and a pipeline showing that over $50\%$ (!) of the fine-tuning dataset can be extracted from a fine-tuned LLM in natural settings. We consider varying degrees of access to the underlying model, customization of the language model, and resources available to the attacker. In the pre-trained setting, we propose three new white-box MIAs: an attack based on the gradient norm, a supervised neural network classifier, and a single step loss ratio attack. All outperform existing black-box baselines, and our supervised attack closes the gap between MIA attack success against LLMs and other types of models. In fine-tuning, we find that given access to the loss of the fine-tuned and base models, a fine-tuned loss ratio attack FLoRA is able to achieve near perfect MIA peformance. We then leverage these MIAs to extract fine-tuning data from fine-tuned language models. We find that the pipeline of generating from fine-tuned models prompted with a small snippet of the prefix of each training example, followed by using FLoRa to select the most likely training sample, succeeds the majority of the fine-tuning dataset after only $3$ epochs of fine-tuning. Taken together, these findings show that highly effective MIAs are available in almost all LLM training settings, and highlight that great care must be taken before LLMs are fine-tuned on highly sensitive data and then deployed.

Title: Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset

• Authors: Santosh T.Y.S.S, Nina Baumgartner, Matthias Stürmer, Matthias Grabmair, Joel Niklaus
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17013
• Pdf URL: https://arxiv.org/pdf/2402.17013
• Copy Paste: [[2402.17013]] Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset(https://arxiv.org/abs/2402.17013)
• Keywords: fair, explainability
• Abstract: The assessment of explainability in Legal Judgement Prediction (LJP) systems is of paramount importance in building trustworthy and transparent systems, particularly considering the reliance of these systems on factors that may lack legal relevance or involve sensitive attributes. This study delves into the realm of explainability and fairness in LJP models, utilizing Swiss Judgement Prediction (SJP), the only available multilingual LJP dataset. We curate a comprehensive collection of rationales that `support' and `oppose' judgement from legal experts for 108 cases in German, French, and Italian. By employing an occlusion-based explainability approach, we evaluate the explainability performance of state-of-the-art monolingual and multilingual BERT-based LJP models, as well as models developed with techniques such as data augmentation and cross-lingual transfer, which demonstrated prediction performance improvement. Notably, our findings reveal that improved prediction performance does not necessarily correspond to enhanced explainability performance, underscoring the significance of evaluating models from an explainability perspective. Additionally, we introduce a novel evaluation framework, Lower Court Insertion (LCI), which allows us to quantify the influence of lower court information on model predictions, exposing current models' biases.

Title: Z-AGI Labs at ClimateActivism 2024: Stance and Hate Event Detection on Social Media

• Authors: Nikhil Narayan, Mrutyunjay Biswal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17014
• Pdf URL: https://arxiv.org/pdf/2402.17014
• Copy Paste: [[2402.17014]] Z-AGI Labs at ClimateActivism 2024: Stance and Hate Event Detection on Social Media(https://arxiv.org/abs/2402.17014)
• Keywords: robust
• Abstract: In the digital realm, rich data serves as a crucial source of insights into the complexities of social, political, and economic landscapes. Addressing the growing need for high-quality information on events and the imperative to combat hate speech, this research led to the establishment of the Shared Task on Climate Activism Stance and Hate Event Detection at CASE 2024. Focused on climate activists contending with hate speech on social media, our study contributes to hate speech identification from tweets. Analyzing three sub-tasks - Hate Speech Detection (Sub-task A), Targets of Hate Speech Identification (Sub-task B), and Stance Detection (Sub-task C) - Team Z-AGI Labs evaluated various models, including LSTM, Xgboost, and LGBM based on Tf-Idf. Results unveiled intriguing variations, with Catboost excelling in Subtask-B (F1: 0.5604) and Subtask-C (F1: 0.7081), while LGBM emerged as the top-performing model for Subtask-A (F1: 0.8684). This research provides valuable insights into the suitability of classical machine learning models for climate hate speech and stance detection, aiding informed model selection for robust mechanisms.

Title: A Curious Case of Remarkable Resilience to Gradient Attacks via Fully Convolutional and Differentiable Front End with a Skip Connection

• Authors: Leonid Boytsov, Ameya Joshi, Filipe Condessa
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17018
• Pdf URL: https://arxiv.org/pdf/2402.17018
• Copy Paste: [[2402.17018]] A Curious Case of Remarkable Resilience to Gradient Attacks via Fully Convolutional and Differentiable Front End with a Skip Connection(https://arxiv.org/abs/2402.17018)
• Keywords: attack, robust
• Abstract: We tested front-end enhanced neural models where a frozen classifier was prepended by a differentiable and fully convolutional model with a skip connection. By training them using a small learning rate for about one epoch, we obtained models that retained the accuracy of the backbone classifier while being unusually resistant to gradient attacks including APGD and FAB-T attacks from the AutoAttack package, which we attributed to gradient masking. The gradient masking phenomenon is not new, but the degree of masking was quite remarkable for fully differentiable models that did not have gradient-shattering components such as JPEG compression or components that are expected to cause diminishing gradients. Though black box attacks can be partially effective against gradient masking, they are easily defeated by combining models into randomized ensembles. We estimate that such ensembles achieve near-SOTA AutoAttack accuracy on CIFAR10, CIFAR100, and ImageNet despite having virtually zero accuracy under adaptive attacks. Adversarial training of the backbone classifier can further increase resistance of the front-end enhanced model to gradient attacks. On CIFAR10, the respective randomized ensemble achieved 90.8$\pm 2.5$% (99% CI) accuracy under AutoAttack while having only 18.2$\pm 3.6$% accuracy under the adaptive attack. We do not establish SOTA in adversarial robustness. Instead, we make methodological contributions and further supports the thesis that adaptive attacks designed with the complete knowledge of model architecture are crucial in demonstrating model robustness and that even the so-called white-box gradient attacks can have limited applicability. Although gradient attacks can be complemented with black-box attack such as the SQUARE attack or the zero-order PGD, black-box attacks can be weak against randomized ensembles, e.g., when ensemble models mask gradients.

Title: Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling

• Authors: Hang Jiang, Xiajie Zhang, Robert Mahari, Daniel Kessler, Eric Ma, Tal August, Irene Li, Alex 'Sandy' Pentland, Yoon Kim, Jad Kabbara, Deb Roy
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.17019
• Pdf URL: https://arxiv.org/pdf/2402.17019
• Copy Paste: [[2402.17019]] Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling(https://arxiv.org/abs/2402.17019)
• Keywords: large language model
• Abstract: Making legal knowledge accessible to non-experts is crucial for enhancing general legal literacy and encouraging civic participation in democracy. However, legal documents are often challenging to understand for people without legal backgrounds. In this paper, we present a novel application of large language models (LLMs) in legal education to help non-experts learn intricate legal concepts through storytelling, an effective pedagogical tool in conveying complex and abstract concepts. We also introduce a new dataset LegalStories, which consists of 295 complex legal doctrines, each accompanied by a story and a set of multiple-choice questions generated by LLMs. To construct the dataset, we experiment with various LLMs to generate legal stories explaining these concepts. Furthermore, we use an expert-in-the-loop method to iteratively design multiple-choice questions. Then, we evaluate the effectiveness of storytelling with LLMs through an RCT experiment with legal novices on 10 samples from the dataset. We find that LLM-generated stories enhance comprehension of legal concepts and interest in law among non-native speakers compared to only definitions. Moreover, stories consistently help participants relate legal concepts to their lives. Finally, we find that learning with stories shows a higher retention rate for non-native speakers in the follow-up assessment. Our work has strong implications for using LLMs in promoting teaching and learning in the legal field and beyond.

Title: Deep Learning Algorithms Used in Intrusion Detection Systems -- A Review

• Authors: Richard Kimanzi, Peter Kimanga, Dedan Cherori, Patrick K. Gikunda
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.17020
• Pdf URL: https://arxiv.org/pdf/2402.17020
• Copy Paste: [[2402.17020]] Deep Learning Algorithms Used in Intrusion Detection Systems -- A Review(https://arxiv.org/abs/2402.17020)
• Keywords: security, attack, robust
• Abstract: The increase in network attacks has necessitated the development of robust and efficient intrusion detection systems (IDS) capable of identifying malicious activities in real-time. In the last five years, deep learning algorithms have emerged as powerful tools in this domain, offering enhanced detection capabilities compared to traditional methods. This review paper studies recent advancements in the application of deep learning techniques, including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Belief Networks (DBN), Deep Neural Networks (DNN), Long Short-Term Memory (LSTM), autoencoders (AE), Multi-Layer Perceptrons (MLP), Self-Normalizing Networks (SNN) and hybrid models, within network intrusion detection systems. we delve into the unique architectures, training models, and classification methodologies tailored for network traffic analysis and anomaly detection. Furthermore, we analyze the strengths and limitations of each deep learning approach in terms of detection accuracy, computational efficiency, scalability, and adaptability to evolving threats. Additionally, this paper highlights prominent datasets and benchmarking frameworks commonly utilized for evaluating the performance of deep learning-based IDS. This review will provide researchers and industry practitioners with valuable insights into the state-of-the-art deep learning algorithms for enhancing the security framework of network environments through intrusion detection.

Title: An Investigation into the Performances of the State-of-the-art Machine Learning Approaches for Various Cyber-attack Detection: A Survey

• Authors: Tosin Ige, Christopher Kiekintveld, Aritran Piplai
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17045
• Pdf URL: https://arxiv.org/pdf/2402.17045
• Copy Paste: [[2402.17045]] An Investigation into the Performances of the State-of-the-art Machine Learning Approaches for Various Cyber-attack Detection: A Survey(https://arxiv.org/abs/2402.17045)
• Keywords: secure, security, attack
• Abstract: To secure computers and information systems from attackers taking advantage of vulnerabilities in the system to commit cybercrime, several methods have been proposed for real-time detection of vulnerabilities to improve security around information systems. Of all the proposed methods, machine learning had been the most effective method in securing a system with capabilities ranging from early detection of software vulnerabilities to real-time detection of ongoing compromise in a system. As there are different types of cyberattacks, each of the existing state-of-the-art machine learning models depends on different algorithms for training which also impact their suitability for detection of a particular type of cyberattack. In this research, we analyzed each of the current state-of-theart machine learning models for different types of cyberattack detection from the past 10 years with a major emphasis on the most recent works for comparative study to identify the knowledge gap where work is still needed to be done with regard to detection of each category of cyberattack

Title: Taming the Tail in Class-Conditional GANs: Knowledge Sharing via Unconditional Training at Lower Resolutions

• Authors: Saeed Khorram, Mingqi Jiang, Mohamad Shahbazi, Mohamad H. Danesh, Li Fuxin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17065
• Pdf URL: https://arxiv.org/pdf/2402.17065
• Copy Paste: [[2402.17065]] Taming the Tail in Class-Conditional GANs: Knowledge Sharing via Unconditional Training at Lower Resolutions(https://arxiv.org/abs/2402.17065)
• Keywords: fair, generative
• Abstract: Despite the extensive research on training generative adversarial networks (GANs) with limited training data, learning to generate images from long-tailed training distributions remains fairly unexplored. In the presence of imbalanced multi-class training data, GANs tend to favor classes with more samples, leading to the generation of low-quality and less diverse samples in tail classes. In this study, we aim to improve the training of class-conditional GANs with long-tailed data. We propose a straightforward yet effective method for knowledge sharing, allowing tail classes to borrow from the rich information from classes with more abundant training data. More concretely, we propose modifications to existing class-conditional GAN architectures to ensure that the lower-resolution layers of the generator are trained entirely unconditionally while reserving class-conditional generation for the higher-resolution layers. Experiments on several long-tail benchmarks and GAN architectures demonstrate a significant improvement over existing methods in both the diversity and fidelity of the generated images. The code is available at https://github.com/khorrams/utlo.

Title: A Pioneering Study and An Innovative Information Theory-based Approach to Enhance The Transparency in Phishing Detection

• Authors: Van Nguyen, Tingmin Wu, Xingliang Yuan, Marthie Grobler, Surya Nepal, Carsten Rudolph
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.17092
• Pdf URL: https://arxiv.org/pdf/2402.17092
• Copy Paste: [[2402.17092]] A Pioneering Study and An Innovative Information Theory-based Approach to Enhance The Transparency in Phishing Detection(https://arxiv.org/abs/2402.17092)
• Keywords: defense, attack, explainability
• Abstract: Phishing attacks have become a serious and challenging issue for detection, explanation, and defense. Despite more than a decade of research on phishing, encompassing both technical and non-technical remedies, phishing continues to be a serious problem. Nowadays, AI-based phishing detection stands out as one of the most effective solutions for defending against phishing attacks by providing vulnerability (i.e., phishing or benign) predictions for the data. However, it lacks explainability in terms of providing comprehensive interpretations for the predictions, such as identifying the specific information that causes the data to be classified as phishing. To this end, we propose an innovative deep learning-based approach for email (the most common phishing way) phishing attack localization. Our method can not only predict the vulnerability of the email data but also automatically figure out and highlight the most important and phishing-relevant information (i.e., sentences) in each phishing email. The selected information indicates useful explanations for the vulnerability of the phishing email data. The rigorous experiments on seven real-world email datasets show the effectiveness and advancement of our proposed method in providing comprehensive explanations (by successfully figuring out the most important and phishing-relevant information in phishing emails) for the vulnerability of corresponding phishing data with higher performances from nearly (1% to 3%) and (1% to 4%) in two main Label-Accuracy and Cognitive-True-Positive measures, respectively, compared to the state-of-the-art potential baselines.

Title: In Defense and Revival of Bayesian Filtering for Thermal Infrared Object Tracking

• Authors: Peng Gao, Shi-Min Li, Feng Gao, Fei Wang, Ru-Yue Yuan, Hamido Fujita
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17098
• Pdf URL: https://arxiv.org/pdf/2402.17098
• Copy Paste: [[2402.17098]] In Defense and Revival of Bayesian Filtering for Thermal Infrared Object Tracking(https://arxiv.org/abs/2402.17098)
• Keywords: defense
• Abstract: Deep learning-based methods monopolize the latest research in the field of thermal infrared (TIR) object tracking. However, relying solely on deep learning models to obtain better tracking results requires carefully selecting feature information that is beneficial to representing the target object and designing a reasonable template update strategy, which undoubtedly increases the difficulty of model design. Thus, recent TIR tracking methods face many challenges in complex scenarios. This paper introduces a novel Deep Bayesian Filtering (DBF) method to enhance TIR tracking in these challenging situations. DBF is distinctive in its dual-model structure: the system and observation models. The system model leverages motion data to estimate the potential positions of the target object based on two-dimensional Brownian motion, thus generating a prior probability. Following this, the observation model comes into play upon capturing the TIR image. It serves as a classifier and employs infrared information to ascertain the likelihood of these estimated positions, creating a likelihood probability. According to the guidance of the two models, the position of the target object can be determined, and the template can be dynamically updated. Experimental analysis across several benchmark datasets reveals that DBF achieves competitive performance, surpassing most existing TIR tracking methods in complex scenarios.

Title: T-HITL Effectively Addresses Problematic Associations in Image Generation and Maintains Overall Visual Quality

• Authors: Susan Epstein, Li Chen, Alessandro Vecchiato, Ankit Jain
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17101
• Pdf URL: https://arxiv.org/pdf/2402.17101
• Copy Paste: [[2402.17101]] T-HITL Effectively Addresses Problematic Associations in Image Generation and Maintains Overall Visual Quality(https://arxiv.org/abs/2402.17101)
• Keywords: generative
• Abstract: Generative AI image models may inadvertently generate problematic representations of people. Past research has noted that millions of users engage daily across the world with these models and that the models, including through problematic representations of people, have the potential to compound and accelerate real-world discrimination and other harms (Bianchi et al, 2023). In this paper, we focus on addressing the generation of problematic associations between demographic groups and semantic concepts that may reflect and reinforce negative narratives embedded in social data. Building on sociological literature (Blumer, 1958) and mapping representations to model behaviors, we have developed a taxonomy to study problematic associations in image generation models. We explore the effectiveness of fine tuning at the model level as a method to address these associations, identifying a potential reduction in visual quality as a limitation of traditional fine tuning. We also propose a new methodology with twice-human-in-the-loop (T-HITL) that promises improvements in both reducing problematic associations and also maintaining visual quality. We demonstrate the effectiveness of T-HITL by providing evidence of three problematic associations addressed by T-HITL at the model level. Our contributions to scholarship are two-fold. By defining problematic associations in the context of machine learning models and generative AI, we introduce a conceptual and technical taxonomy for addressing some of these associations. Finally, we provide a method, T-HITL, that addresses these associations and simultaneously maintains visual quality of image model generations. This mitigation need not be a tradeoff, but rather an enhancement.

Title: Sinkhorn Distance Minimization for Knowledge Distillation

• Authors: Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, Houqiang Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17110
• Pdf URL: https://arxiv.org/pdf/2402.17110
• Copy Paste: [[2402.17110]] Sinkhorn Distance Minimization for Knowledge Distillation(https://arxiv.org/abs/2402.17110)
• Keywords: large language model
• Abstract: Knowledge distillation (KD) has been widely adopted to compress large language models (LLMs). Existing KD methods investigate various divergence measures including the Kullback-Leibler (KL), reverse Kullback-Leibler (RKL), and Jensen-Shannon (JS) divergences. However, due to limitations inherent in their assumptions and definitions, these measures fail to deliver effective supervision when few distribution overlap exists between the teacher and the student. In this paper, we show that the aforementioned KL, RKL, and JS divergences respectively suffer from issues of mode-averaging, mode-collapsing, and mode-underestimation, which deteriorates logits-based KD for diverse NLP tasks. We propose the Sinkhorn Knowledge Distillation (SinKD) that exploits the Sinkhorn distance to ensure a nuanced and precise assessment of the disparity between teacher and student distributions. Besides, profit by properties of the Sinkhorn metric, we can get rid of sample-wise KD that restricts the perception of divergence in each teacher-student sample pair. Instead, we propose a batch-wise reformulation to capture geometric intricacies of distributions across samples in the high-dimensional space. Comprehensive evaluation on GLUE and SuperGLUE, in terms of comparability, validity, and generalizability, highlights our superiority over state-of-the-art methods on all kinds of LLMs with encoder-only, encoder-decoder, and decoder-only architectures.

Title: Transparent Image Layer Diffusion using Latent Transparency

• Authors: Lvmin Zhang, Maneesh Agrawala
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.17113
• Pdf URL: https://arxiv.org/pdf/2402.17113
• Copy Paste: [[2402.17113]] Transparent Image Layer Diffusion using Latent Transparency(https://arxiv.org/abs/2402.17113)
• Keywords: diffusion
• Abstract: We present LayerDiffusion, an approach enabling large-scale pretrained latent diffusion models to generate transparent images. The method allows generation of single transparent images or of multiple transparent layers. The method learns a "latent transparency" that encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. It preserves the production-ready quality of the large diffusion model by regulating the added transparency as a latent offset with minimal changes to the original latent distribution of the pretrained model. In this way, any latent diffusion model can be converted into a transparent image generator by finetuning it with the adjusted latent space. We train the model with 1M transparent image layer pairs collected using a human-in-the-loop collection scheme. We show that latent transparency can be applied to different open source image generators, or be adapted to various conditional control systems to achieve applications like foreground/background-conditioned layer generation, joint layer generation, structural control of layer contents, etc. A user study finds that in most cases (97%) users prefer our natively generated transparent content over previous ad-hoc solutions such as generating and then matting. Users also report the quality of our generated transparent images is comparable to real commercial transparent assets like Adobe Stock.

Title: Creating Suspenseful Stories: Iterative Planning with Large Language Models

• Authors: Kaige Xie, Mark Riedl
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17119
• Pdf URL: https://arxiv.org/pdf/2402.17119
• Copy Paste: [[2402.17119]] Creating Suspenseful Stories: Iterative Planning with Large Language Models(https://arxiv.org/abs/2402.17119)
• Keywords: large language model
• Abstract: Automated story generation has been one of the long-standing challenges in NLP. Among all dimensions of stories, suspense is very common in human-written stories but relatively under-explored in AI-generated stories. While recent advances in large language models (LLMs) have greatly promoted language generation in general, state-of-the-art LLMs are still unreliable when it comes to suspenseful story generation. We propose a novel iterative-prompting-based planning method that is grounded in two theoretical foundations of story suspense from cognitive psychology and narratology. This theory-grounded method works in a fully zero-shot manner and does not rely on any supervised story corpora. To the best of our knowledge, this paper is the first attempt at suspenseful story generation with LLMs. Extensive human evaluations of the generated suspenseful stories demonstrate the effectiveness of our method.

Title: LCEN: A Novel Feature Selection Algorithm for Nonlinear, Interpretable Machine Learning Models

• Authors: Pedro Seber, Richard D. Braatz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17120
• Pdf URL: https://arxiv.org/pdf/2402.17120
• Copy Paste: [[2402.17120]] LCEN: A Novel Feature Selection Algorithm for Nonlinear, Interpretable Machine Learning Models(https://arxiv.org/abs/2402.17120)
• Keywords: robust, interpretability
• Abstract: Interpretable architectures can have advantages over black-box architectures, and interpretability is essential for the application of machine learning in critical settings, such as aviation or medicine. However, the simplest, most commonly used interpretable architectures (such as LASSO or EN) are limited to linear predictions and have poor feature selection capabilities. In this work, we introduce the LASSO-Clip-EN (LCEN) algorithm for the creation of nonlinear, interpretable machine learning models. LCEN is tested on a wide variety of artificial and empirical datasets, creating more accurate, sparser models than other commonly used architectures. These experiments reveal that LCEN is robust against many issues typically present in datasets and modeling, including noise, multicollinearity, data scarcity, and hyperparameter variance. LCEN is also able to rediscover multiple physical laws from empirical data and, for processes with no known physical laws, LCEN achieves better results than many other dense and sparse methods -- including using 10.8 times fewer features than dense methods and 8.1 times fewer features than EN on one dataset, and is comparable to an ANN on another dataset.

Title: OSCaR: Object State Captioning and State Change Representation

• Authors: Nguyen Nguyen, Jing Bi, Ali Vosoughi, Yapeng Tian, Pooyan Fazli, Chenliang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17128
• Pdf URL: https://arxiv.org/pdf/2402.17128
• Copy Paste: [[2402.17128]] OSCaR: Object State Captioning and State Change Representation(https://arxiv.org/abs/2402.17128)
• Keywords: large language model
• Abstract: The capability of intelligent models to extrapolate and comprehend changes in object states is a crucial yet demanding aspect of AI research, particularly through the lens of human interaction in real-world settings. This task involves describing complex visual environments, identifying active objects, and interpreting their changes as conveyed through language. Traditional methods, which isolate object captioning and state change detection, offer a limited view of dynamic environments. Moreover, relying on a small set of symbolic words to represent changes has restricted the expressiveness of language. To address these challenges, in this paper, we introduce the Object State Captioning and State Change Representation (OSCaR) dataset and benchmark. OSCaR consists of 14,084 annotated video segments with nearly 1,000 unique objects from various egocentric video collections. It sets a new testbed for evaluating multimodal large language models (MLLMs). Our experiments demonstrate that while MLLMs show some skill, they lack a full understanding of object state changes. The benchmark includes a fine-tuned model that, despite initial capabilities, requires significant improvements in accuracy and generalization ability for effective understanding of these changes. Our code and dataset are available at https://github.com/nguyennm1024/OSCaR.

Title: Predicting O-GlcNAcylation Sites in Mammalian Proteins with Transformers and RNNs Trained with a New Loss Function

• Authors: Pedro Seber
• Subjects: cs.LG, q-bio.MN
• Abstract URL: https://arxiv.org/abs/2402.17131
• Pdf URL: https://arxiv.org/pdf/2402.17131
• Copy Paste: [[2402.17131]] Predicting O-GlcNAcylation Sites in Mammalian Proteins with Transformers and RNNs Trained with a New Loss Function(https://arxiv.org/abs/2402.17131)
• Keywords: transformer
• Abstract: Glycosylation, a protein modification, has multiple essential functional and structural roles. O-GlcNAcylation, a subtype of glycosylation, has the potential to be an important target for therapeutics, but methods to reliably predict O-GlcNAcylation sites had not been available until 2023; a 2021 review correctly noted that published models were insufficient and failed to generalize. Moreover, many are no longer usable. In 2023, a considerably better RNN model with an F$_1$ score of 36.17% and an MCC of 34.57% on a large dataset was published. This article first sought to improve these metrics using transformer encoders. While transformers displayed high performance on this dataset, their performance was inferior to that of the previously published RNN. We then created a new loss function, which we call the weighted focal differentiable MCC, to improve the performance of classification models. RNN models trained with this new function display superior performance to models trained using the weighted cross-entropy loss; this new function can also be used to fine-tune trained models. A two-cell RNN trained with this loss achieves state-of-the-art performance in O-GlcNAcylation site prediction with an F$_1$ score of 38.82% and an MCC of 38.21% on that large dataset.

Title: SAM-DiffSR: Structure-Modulated Diffusion Model for Image Super-Resolution

• Authors: Chengcheng Wang, Zhiwei Hao, Yehui Tang, Jianyuan Guo, Yujie Yang, Kai Han, Yunhe Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17133
• Pdf URL: https://arxiv.org/pdf/2402.17133
• Copy Paste: [[2402.17133]] SAM-DiffSR: Structure-Modulated Diffusion Model for Image Super-Resolution(https://arxiv.org/abs/2402.17133)
• Keywords: diffusion, segmentation
• Abstract: Diffusion-based super-resolution (SR) models have recently garnered significant attention due to their potent restoration capabilities. But conventional diffusion models perform noise sampling from a single distribution, constraining their ability to handle real-world scenes and complex textures across semantic regions. With the success of segment anything model (SAM), generating sufficiently fine-grained region masks can enhance the detail recovery of diffusion-based SR model. However, directly integrating SAM into SR models will result in much higher computational cost. In this paper, we propose the SAM-DiffSR model, which can utilize the fine-grained structure information from SAM in the process of sampling noise to improve the image quality without additional computational cost during inference. In the process of training, we encode structural position information into the segmentation mask from SAM. Then the encoded mask is integrated into the forward diffusion process by modulating it to the sampled noise. This adjustment allows us to independently adapt the noise mean within each corresponding segmentation area. The diffusion model is trained to estimate this modulated noise. Crucially, our proposed framework does NOT change the reverse diffusion process and does NOT require SAM at inference. Experimental results demonstrate the effectiveness of our proposed method, showcasing superior performance in suppressing artifacts, and surpassing existing diffusion-based methods by 0.74 dB at the maximum in terms of PSNR on DIV2K dataset. The code and dataset are available at https://github.com/lose4578/SAM-DiffSR.

Title: Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings

• Authors: Kevin Frans, Seohong Park, Pieter Abbeel, Sergey Levine
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17135
• Pdf URL: https://arxiv.org/pdf/2402.17135
• Copy Paste: [[2402.17135]] Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings(https://arxiv.org/abs/2402.17135)
• Keywords: transformer
• Abstract: Can we pre-train a generalist agent from a large amount of unlabeled offline trajectories such that it can be immediately adapted to any new downstream tasks in a zero-shot manner? In this work, we present a functional reward encoding (FRE) as a general, scalable solution to this zero-shot RL problem. Our main idea is to learn functional representations of any arbitrary tasks by encoding their state-reward samples using a transformer-based variational auto-encoder. This functional encoding not only enables the pre-training of an agent from a wide diversity of general unsupervised reward functions, but also provides a way to solve any new downstream tasks in a zero-shot manner, given a small number of reward-annotated samples. We empirically show that FRE agents trained on diverse random unsupervised reward functions can generalize to solve novel tasks in a range of simulated robotic benchmarks, often outperforming previous zero-shot RL and offline RL methods. Code for this project is provided at: https://github.com/kvfrans/fre

Title: Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

• Authors: Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, Yu Shi
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.17152
• Pdf URL: https://arxiv.org/pdf/2402.17152
• Copy Paste: [[2402.17152]] Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations(https://arxiv.org/abs/2402.17152)
• Keywords: transformer, generative
• Abstract: Large-scale recommendation systems are characterized by their reliance on high cardinality, heterogeneous features and the need to handle tens of billions of user actions on a daily basis. Despite being trained on huge volume of data with thousands of features, most Deep Learning Recommendation Models (DLRMs) in industry fail to scale with compute. Inspired by success achieved by Transformers in language and vision domains, we revisit fundamental design choices in recommendation systems. We reformulate recommendation problems as sequential transduction tasks within a generative modeling framework (``Generative Recommenders''), and propose a new architecture, HSTU, designed for high cardinality, non-stationary streaming recommendation data. HSTU outperforms baselines over synthetic and public datasets by up to 65.8\% in NDCG, and is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192 length sequences. HSTU-based Generative Recommenders, with 1.5 trillion parameters, improve metrics in online A/B tests by 12.4\% and have been deployed on multiple surfaces of a large internet platform with billions of users. More importantly, the model quality of Generative Recommenders empirically scales as a power-law of training compute across three orders of magnitude, up to GPT-3/LLaMa-2 scale, which reduces carbon footprint needed for future model developments, and further paves the way for the first foundational models in recommendations.

Title: TaxDiff: Taxonomic-Guided Diffusion Model for Protein Sequence Generation

• Authors: Lin Zongying, Li Hao, Lv Liuzhenghao, Lin Bin, Zhang Junwu, Chen Calvin Yu-Chian, Yuan Li, Tian Yonghong
• Subjects: cs.LG, cs.AI, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2402.17156
• Pdf URL: https://arxiv.org/pdf/2402.17156
• Copy Paste: [[2402.17156]] TaxDiff: Taxonomic-Guided Diffusion Model for Protein Sequence Generation(https://arxiv.org/abs/2402.17156)
• Keywords: diffusion, transformer, generative
• Abstract: Designing protein sequences with specific biological functions and structural stability is crucial in biology and chemistry. Generative models already demonstrated their capabilities for reliable protein design. However, previous models are limited to the unconditional generation of protein sequences and lack the controllable generation ability that is vital to biological tasks. In this work, we propose TaxDiff, a taxonomic-guided diffusion model for controllable protein sequence generation that combines biological species information with the generative capabilities of diffusion models to generate structurally stable proteins within the sequence space. Specifically, taxonomic control information is inserted into each layer of the transformer block to achieve fine-grained control. The combination of global and local attention ensures the sequence consistency and structural foldability of taxonomic-specific proteins. Extensive experiments demonstrate that TaxDiff can consistently achieve better performance on multiple protein sequence generation benchmarks in both taxonomic-guided controllable generation and unconditional generation. Remarkably, the sequences generated by TaxDiff even surpass those produced by direct-structure-generation models in terms of confidence based on predicted structures and require only a quarter of the time of models based on the diffusion model. The code for generating proteins and training new versions of TaxDiff is available at:https://github.com/Linzy19/TaxDiff.

Title: Generative Learning for Forecasting the Dynamics of Complex Systems

• Authors: Han Gao, Sebastian Kaltenbach, Petros Koumoutsakos
• Subjects: cs.LG, physics.comp-ph, physics.flu-dyn, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.17157
• Pdf URL: https://arxiv.org/pdf/2402.17157
• Copy Paste: [[2402.17157]] Generative Learning for Forecasting the Dynamics of Complex Systems(https://arxiv.org/abs/2402.17157)
• Keywords: diffusion, generative
• Abstract: We introduce generative models for accelerating simulations of complex systems through learning and evolving their effective dynamics. In the proposed Generative Learning of Effective Dynamics (G-LED), instances of high dimensional data are down sampled to a lower dimensional manifold that is evolved through an auto-regressive attention mechanism. In turn, Bayesian diffusion models, that map this low-dimensional manifold onto its corresponding high-dimensional space, capture the statistics of the system dynamics. We demonstrate the capabilities and drawbacks of G-LED in simulations of several benchmark systems, including the Kuramoto-Sivashinsky (KS) equation, two-dimensional high Reynolds number flow over a backward-facing step, and simulations of three-dimensional turbulent channel flow. The results demonstrate that generative learning offers new frontiers for the accurate forecasting of the statistical properties of complex systems at a reduced computational cost.

Title: NocPlace: Nocturnal Visual Place Recognition Using Generative and Inherited Knowledge Transfer

• Authors: Bingxi Liu, Yiqun Wang, Huaqi Tao, Tingjun Huang, Fulin Tang, Yihong Wu, Jinqiang Cui, Hong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17159
• Pdf URL: https://arxiv.org/pdf/2402.17159
• Copy Paste: [[2402.17159]] NocPlace: Nocturnal Visual Place Recognition Using Generative and Inherited Knowledge Transfer(https://arxiv.org/abs/2402.17159)
• Keywords: generative
• Abstract: Visual Place Recognition (VPR) is crucial in computer vision, aiming to retrieve database images similar to a query image from an extensive collection of known images. However, like many vision-related tasks, learning-based VPR often experiences a decline in performance during nighttime due to the scarcity of nighttime images. Specifically, VPR needs to address the cross-domain problem of night-to-day rather than just the issue of a single nighttime domain. In response to these issues, we present NocPlace, which leverages a generated large-scale, multi-view, nighttime VPR dataset to embed resilience against dazzling lights and extreme darkness in the learned global descriptor. Firstly, we establish a day-night urban scene dataset called NightCities, capturing diverse nighttime scenarios and lighting variations across 60 cities globally. Following this, an unpaired image-to-image translation network is trained on this dataset. Using this trained translation network, we process an existing VPR dataset, thereby obtaining its nighttime version. The NocPlace is then fine-tuned using night-style images, the original labels, and descriptors inherited from the Daytime VPR model. Comprehensive experiments on various nighttime VPR test sets reveal that NocPlace considerably surpasses previous state-of-the-art methods.

Title: Few-shot adaptation for morphology-independent cell instance segmentation

• Authors: Ram J. Zaveri, Voke Brume, Gianfranco Doretto
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17165
• Pdf URL: https://arxiv.org/pdf/2402.17165
• Copy Paste: [[2402.17165]] Few-shot adaptation for morphology-independent cell instance segmentation(https://arxiv.org/abs/2402.17165)
• Keywords: segmentation
• Abstract: Microscopy data collections are becoming larger and more frequent. Accurate and precise quantitative analysis tools like cell instance segmentation are necessary to benefit from them. This is challenging due to the variability in the data, which requires retraining the segmentation model to maintain high accuracy on new collections. This is needed especially for segmenting cells with elongated and non-convex morphology like bacteria. We propose to reduce the amount of annotation and computing power needed for retraining the model by introducing a few-shot domain adaptation approach that requires annotating only one to five cells of the new data to process and that quickly adapts the model to maintain high accuracy. Our results show a significant boost in accuracy after adaptation to very challenging bacteria datasets.

Title: Deep Umbra: A Generative Approach for Sunlight Access Computation in Urban Spaces

• Authors: Kazi Shahrukh Omar, Gustavo Moreira, Daniel Hodczak, Maryam Hosseini, Nicola Colaninno, Marcos Lage, Fabio Miranda
• Subjects: cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.17169
• Pdf URL: https://arxiv.org/pdf/2402.17169
• Copy Paste: [[2402.17169]] Deep Umbra: A Generative Approach for Sunlight Access Computation in Urban Spaces(https://arxiv.org/abs/2402.17169)
• Keywords: generative
• Abstract: Sunlight and shadow play critical roles in how urban spaces are utilized, thrive, and grow. While access to sunlight is essential to the success of urban environments, shadows can provide shaded places to stay during the hot seasons, mitigate heat island effect, and increase pedestrian comfort levels. Properly quantifying sunlight access and shadows in large urban environments is key in tackling some of the important challenges facing cities today. In this paper, we propose Deep Umbra, a novel computational framework that enables the quantification of sunlight access and shadows at a global scale. Our framework is based on a conditional generative adversarial network that considers the physical form of cities to compute high-resolution spatial information of accumulated sunlight access for the different seasons of the year. We use data from seven different cities to train our model, and show, through an extensive set of experiments, its low overall RMSE (below 0.1) as well as its extensibility to cities that were not part of the training set. Additionally, we contribute a set of case studies and a comprehensive dataset with sunlight access information for more than 100 cities across six continents of the world. Deep Umbra is available at https://urbantk.org/shadows.

Title: LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment

• Authors: Yiming Ren, Xiao Han, Chengfeng Zhao, Jingya Wang, Lan Xu, Jingyi Yu, Yuexin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17171
• Pdf URL: https://arxiv.org/pdf/2402.17171
• Copy Paste: [[2402.17171]] LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment(https://arxiv.org/abs/2402.17171)
• Keywords: robust
• Abstract: For human-centric large-scale scenes, fine-grained modeling for 3D human global pose and shape is significant for scene understanding and can benefit many real-world applications. In this paper, we present LiveHPS, a novel single-LiDAR-based approach for scene-level human pose and shape estimation without any limitation of light conditions and wearable devices. In particular, we design a distillation mechanism to mitigate the distribution-varying effect of LiDAR point clouds and exploit the temporal-spatial geometric and dynamic information existing in consecutive frames to solve the occlusion and noise disturbance. LiveHPS, with its efficient configuration and high-quality output, is well-suited for real-world applications. Moreover, we propose a huge human motion dataset, named FreeMotion, which is collected in various scenarios with diverse human poses, shapes and translations. It consists of multi-modal and multi-view acquisition data from calibrated and synchronized LiDARs, cameras, and IMUs. Extensive experiments on our new dataset and other public datasets demonstrate the SOTA performance and robustness of our approach. We will release our code and dataset soon.

Title: Lane2Seq: Towards Unified Lane Detection via Sequence Generation

• Authors: Kunyang Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17172
• Pdf URL: https://arxiv.org/pdf/2402.17172
• Copy Paste: [[2402.17172]] Lane2Seq: Towards Unified Lane Detection via Sequence Generation(https://arxiv.org/abs/2402.17172)
• Keywords: transformer
• Abstract: In this paper, we present a novel sequence generation-based framework for lane detection, called Lane2Seq. It unifies various lane detection formats by casting lane detection as a sequence generation task. This is different from previous lane detection methods, which depend on well-designed task-specific head networks and corresponding loss functions. Lane2Seq only adopts a plain transformer-based encoder-decoder architecture with a simple cross-entropy loss. Additionally, we propose a new multi-format model tuning based on reinforcement learning to incorporate the task-specific knowledge into Lane2Seq. Experimental results demonstrate that such a simple sequence generation paradigm not only unifies lane detection but also achieves competitive performance on benchmarks. For example, Lane2Seq gets 97.95\% and 97.42\% F1 score on Tusimple and LLAMAS datasets, establishing a new state-of-the-art result for two benchmarks.

Title: DeepDRK: Deep Dependency Regularized Knockoff for Feature Selection

• Authors: Hongyu Shen, Yici Yan, Zhizhen Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17176
• Pdf URL: https://arxiv.org/pdf/2402.17176
• Copy Paste: [[2402.17176]] DeepDRK: Deep Dependency Regularized Knockoff for Feature Selection(https://arxiv.org/abs/2402.17176)
• Keywords: transformer, generative
• Abstract: Model-X knockoff, among various feature selection methods, received much attention recently due to its guarantee on false discovery rate (FDR) control. Subsequent to its introduction in parametric design, knockoff is advanced to handle arbitrary data distributions using deep learning-based generative modeling. However, we observed that current implementations of the deep Model-X knockoff framework exhibit limitations. Notably, the "swap property" that knockoffs necessitate frequently encounter challenges on sample level, leading to a diminished selection power. To overcome, we develop "Deep Dependency Regularized Knockoff (DeepDRK)", a distribution-free deep learning method that strikes a balance between FDR and power. In DeepDRK, a generative model grounded in a transformer architecture is introduced to better achieve the "swap property". Novel efficient regularization techniques are also proposed to reach higher power. Our model outperforms other benchmarks in synthetic, semi-synthetic, and real-world data, especially when sample size is small and data distribution is complex.

Title: Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models

• Authors: Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, Lifang He, Lichao Sun
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17177
• Pdf URL: https://arxiv.org/pdf/2402.17177
• Copy Paste: [[2402.17177]] Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models(https://arxiv.org/abs/2402.17177)
• Keywords: generative
• Abstract: Sora is a text-to-video generative AI model, released by OpenAI in February 2024. The model is trained to generate videos of realistic or imaginative scenes from text instructions and show potential in simulating the physical world. Based on public technical reports and reverse engineering, this paper presents a comprehensive review of the model's background, related technologies, applications, remaining challenges, and future directions of text-to-video AI models. We first trace Sora's development and investigate the underlying technologies used to build this "world simulator". Then, we describe in detail the applications and potential impact of Sora in multiple industries ranging from film-making and education to marketing. We discuss the main challenges and limitations that need to be addressed to widely deploy Sora, such as ensuring safe and unbiased video generation. Lastly, we discuss the future development of Sora and video generation models in general, and how advancements in the field could enable new ways of human-AI interaction, boosting productivity and creativity of video generation.

Title: Dual-Space Optimization: Improved Molecule Sequence Design by Latent Prompt Transformer

• Authors: Deqian Kong, Yuhao Huang, Jianwen Xie, Edouardo Honig, Ming Xu, Shuanghong Xue, Pei Lin, Sanping Zhou, Sheng Zhong, Nanning Zheng, Ying Nian Wu
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2402.17179
• Pdf URL: https://arxiv.org/pdf/2402.17179
• Copy Paste: [[2402.17179]] Dual-Space Optimization: Improved Molecule Sequence Design by Latent Prompt Transformer(https://arxiv.org/abs/2402.17179)
• Keywords: transformer, generative
• Abstract: Designing molecules with desirable properties, such as drug-likeliness and high binding affinities towards protein targets, is a challenging problem. In this paper, we propose the Dual-Space Optimization (DSO) method that integrates latent space sampling and data space selection to solve this problem. DSO iteratively updates a latent space generative model and a synthetic dataset in an optimization process that gradually shifts the generative model and the synthetic data towards regions of desired property values. Our generative model takes the form of a Latent Prompt Transformer (LPT) where the latent vector serves as the prompt of a causal transformer. Our extensive experiments demonstrate effectiveness of the proposed method, which sets new performance benchmarks across single-objective, multi-objective and constrained molecule design tasks.

Title: AI-Driven Anonymization: Protecting Personal Data Privacy While Leveraging Machine Learning

• Authors: Le Yang, Miao Tian, Duan Xin, Qishuo Cheng, Jiajian Zheng
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17191
• Pdf URL: https://arxiv.org/pdf/2402.17191
• Copy Paste: [[2402.17191]] AI-Driven Anonymization: Protecting Personal Data Privacy While Leveraging Machine Learning(https://arxiv.org/abs/2402.17191)
• Keywords: security, privacy, protect, attack
• Abstract: The development of artificial intelligence has significantly transformed people's lives. However, it has also posed a significant threat to privacy and security, with numerous instances of personal information being exposed online and reports of criminal attacks and theft. Consequently, the need to achieve intelligent protection of personal information through machine learning algorithms has become a paramount concern. Artificial intelligence leverages advanced algorithms and technologies to effectively encrypt and anonymize personal data, enabling valuable data analysis and utilization while safeguarding privacy. This paper focuses on personal data privacy protection and the promotion of anonymity as its core research objectives. It achieves personal data privacy protection and detection through the use of machine learning's differential privacy protection algorithm. The paper also addresses existing challenges in machine learning related to privacy and personal data protection, offers improvement suggestions, and analyzes factors impacting datasets to enable timely personal data privacy detection and protection.

Title: When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

• Authors: Biao Zhang, Zhongtao Liu, Colin Cherry, Orhan Firat
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17193
• Pdf URL: https://arxiv.org/pdf/2402.17193
• Copy Paste: [[2402.17193]] When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method(https://arxiv.org/abs/2402.17193)
• Keywords: large language model
• Abstract: While large language models (LLMs) often adopt finetuning to unlock their capabilities for downstream applications, our understanding on the inductive biases (especially the scaling properties) of different finetuning methods is still limited. To fill this gap, we conduct systematic experiments studying whether and how different scaling factors, including LLM model size, pretraining data size, new finetuning parameter size and finetuning data size, affect the finetuning performance. We consider two types of finetuning -- full-model tuning (FMT) and parameter efficient tuning (PET, including prompt tuning and LoRA), and explore their scaling behaviors in the data-limited regime where the LLM model size substantially outweighs the finetuning data size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and experiments on bilingual machine translation and multilingual summarization benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative joint scaling law between finetuning data size and each other scaling factor; 2) LLM finetuning benefits more from LLM model scaling than pretraining data scaling, and PET parameter scaling is generally ineffective; and 3) the optimal finetuning method is highly task- and finetuning data-dependent. We hope our findings could shed light on understanding, selecting and developing LLM finetuning methods.

Title: FedBRB: An Effective Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learning

• Authors: Ziyue Xu, Mingfeng Xu, Tianchi Liao, Zibin Zheng, Chuan Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17202
• Pdf URL: https://arxiv.org/pdf/2402.17202
• Copy Paste: [[2402.17202]] FedBRB: An Effective Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learning(https://arxiv.org/abs/2402.17202)
• Keywords: federate
• Abstract: Recently, the success of large models has demonstrated the importance of scaling up model size. This has spurred interest in exploring collaborative training of large-scale models from federated learning perspective. Due to computational constraints, many institutions struggle to train a large-scale model locally. Thus, training a larger global model using only smaller local models has become an important scenario (i.e., the \textbf{small-to-large scenario}). Although recent device-heterogeneity federated learning approaches have started to explore this area, they face limitations in fully covering the parameter space of the global model. In this paper, we propose a method called \textbf{FedBRB} (\underline{B}lock-wise \underline{R}olling and weighted \underline{B}roadcast) based on the block concept. FedBRB can uses small local models to train all blocks of the large global model, and broadcasts the trained parameters to the entire space for faster information interaction. Experiments demonstrate FedBRB yields substantial performance gains, achieving state-of-the-art results in this scenario. Moreover, FedBRB using only minimal local models can even surpass baselines using larger local models.

Title: Advancing Generative Model Evaluation: A Novel Algorithm for Realistic Image Synthesis and Comparison in OCR System

• Authors: Majid Memari, Khaled R. Ahmed, Shahram Rahimi, Noorbakhsh Amiri Golilarz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17204
• Pdf URL: https://arxiv.org/pdf/2402.17204
• Copy Paste: [[2402.17204]] Advancing Generative Model Evaluation: A Novel Algorithm for Realistic Image Synthesis and Comparison in OCR System(https://arxiv.org/abs/2402.17204)
• Keywords: generative
• Abstract: This research addresses a critical challenge in the field of generative models, particularly in the generation and evaluation of synthetic images. Given the inherent complexity of generative models and the absence of a standardized procedure for their comparison, our study introduces a pioneering algorithm to objectively assess the realism of synthetic images. This approach significantly enhances the evaluation methodology by refining the Fr\'echet Inception Distance (FID) score, allowing for a more precise and subjective assessment of image quality. Our algorithm is particularly tailored to address the challenges in generating and evaluating realistic images of Arabic handwritten digits, a task that has traditionally been near-impossible due to the subjective nature of realism in image generation. By providing a systematic and objective framework, our method not only enables the comparison of different generative models but also paves the way for improvements in their design and output. This breakthrough in evaluation and comparison is crucial for advancing the field of OCR, especially for scripts that present unique complexities, and sets a new standard in the generation and assessment of high-quality synthetic images.

Title: VCD: Knowledge Base Guided Visual Commonsense Discovery in Images

• Authors: Xiangqing Shen, Yurun Song, Siwei Wu, Rui Xia
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17213
• Pdf URL: https://arxiv.org/pdf/2402.17213
• Copy Paste: [[2402.17213]] VCD: Knowledge Base Guided Visual Commonsense Discovery in Images(https://arxiv.org/abs/2402.17213)
• Keywords: generative
• Abstract: Visual commonsense contains knowledge about object properties, relationships, and behaviors in visual data. Discovering visual commonsense can provide a more comprehensive and richer understanding of images, and enhance the reasoning and decision-making capabilities of computer vision systems. However, the visual commonsense defined in existing visual commonsense discovery studies is coarse-grained and incomplete. In this work, we draw inspiration from a commonsense knowledge base ConceptNet in natural language processing, and systematically define the types of visual commonsense. Based on this, we introduce a new task, Visual Commonsense Discovery (VCD), aiming to extract fine-grained commonsense of different types contained within different objects in the image. We accordingly construct a dataset (VCDD) from Visual Genome and ConceptNet for VCD, featuring over 100,000 images and 14 million object-commonsense pairs. We furthermore propose a generative model (VCDM) that integrates a vision-language model with instruction tuning to tackle VCD. Automatic and human evaluations demonstrate VCDM's proficiency in VCD, particularly outperforming GPT-4V in implicit commonsense discovery. The value of VCD is further demonstrated by its application to two downstream tasks, including visual commonsense evaluation and visual question answering. The data and code will be made available on GitHub.

Title: CharacterGen: Efficient 3D Character Generation from Single Images with Multi-View Pose Canonicalization

• Authors: Hao-Yang Peng, Jia-Peng Zhang, Meng-Hao Guo, Yan-Pei Cao, Shi-Min Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17214
• Pdf URL: https://arxiv.org/pdf/2402.17214
• Copy Paste: [[2402.17214]] CharacterGen: Efficient 3D Character Generation from Single Images with Multi-View Pose Canonicalization(https://arxiv.org/abs/2402.17214)
• Keywords: diffusion, transformer
• Abstract: In the field of digital content creation, generating high-quality 3D characters from single images is challenging, especially given the complexities of various body poses and the issues of self-occlusion and pose ambiguity. In this paper, we present CharacterGen, a framework developed to efficiently generate 3D characters. CharacterGen introduces a streamlined generation pipeline along with an image-conditioned multi-view diffusion model. This model effectively calibrates input poses to a canonical form while retaining key attributes of the input image, thereby addressing the challenges posed by diverse poses. A transformer-based, generalizable sparse-view reconstruction model is the other core component of our approach, facilitating the creation of detailed 3D models from multi-view images. We also adopt a texture-back-projection strategy to produce high-quality texture maps. Additionally, we have curated a dataset of anime characters, rendered in multiple poses and views, to train and evaluate our model. Our approach has been thoroughly evaluated through quantitative and qualitative experiments, showing its proficiency in generating 3D characters with high-quality shapes and textures, ready for downstream applications such as rigging and animation.

Title: Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning

• Authors: Zijian Guo, Weichao Zhou, Wenchao Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17217
• Pdf URL: https://arxiv.org/pdf/2402.17217
• Copy Paste: [[2402.17217]] Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning(https://arxiv.org/abs/2402.17217)
• Keywords: transformer
• Abstract: Offline safe reinforcement learning (RL) aims to train a constraint satisfaction policy from a fixed dataset. Current state-of-the-art approaches are based on supervised learning with a conditioned policy. However, these approaches fall short in real-world applications that involve complex tasks with rich temporal and logical structures. In this paper, we propose temporal logic Specification-conditioned Decision Transformer (SDT), a novel framework that harnesses the expressive power of signal temporal logic (STL) to specify complex temporal rules that an agent should follow and the sequential modeling capability of Decision Transformer (DT). Empirical evaluations on the DSRL benchmarks demonstrate the better capacity of SDT in learning safe and high-reward policies compared with existing approaches. In addition, SDT shows good alignment with respect to different desired degrees of satisfaction of the STL specification that it is conditioned on.

Title: Blockchain for Finance: A Survey

• Authors: Hanjie Wu, Qian Yao, Zhenguang Liu, Butian Huang, Yuan Zhuang, Huayun Tang, Erwu Liu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.17219
• Pdf URL: https://arxiv.org/pdf/2402.17219
• Copy Paste: [[2402.17219]] Blockchain for Finance: A Survey(https://arxiv.org/abs/2402.17219)
• Keywords: security
• Abstract: As an innovative technology for enhancing authenticity, security, and risk management, blockchain is being widely adopted in trade and finance systems. The unique capabilities of blockchain, such as immutability and transparency, enable new business models of distributed data storage, point-to-point transactions, and decentralized autonomous organizations. In this paper, we focus on blockchain-based securities trading, in which blockchain technology plays a vital role in financial services as it ultimately lifts trust and frees the need for third-party verification by using consensus-based verification. We investigate the 12 most popular blockchain platforms and elaborate on 6 platforms that are related to finance, seeking to provide a panorama of securities trading practices. Meanwhile, this survey provides a comprehensive summary of blockchain-based securities trading applications. We gather numerous practical applications of blockchain-based securities trading and categorize them into four distinct categories. For each category, we introduce a typical example and explain how blockchain contributes to solving the key problems faced by FinTech companies and researchers. Finally, we provide interesting observations ranging from mainstream blockchain-based financial institutions to security issues of decentralized finance applications, aiming to picture the current blockchain ecosystem in finance.

Title: Time-Restricted Double-Spending Attack on PoW-based Blockchains

• Authors: Yiming Jiang, Jiangfan Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.17223
• Pdf URL: https://arxiv.org/pdf/2402.17223
• Copy Paste: [[2402.17223]] Time-Restricted Double-Spending Attack on PoW-based Blockchains(https://arxiv.org/abs/2402.17223)
• Keywords: attack
• Abstract: Numerous blockchain applications are designed with tasks that naturally have finite durations, and hence, a double-spending attack (DSA) on such blockchain applications leans towards being conducted within a finite timeframe, specifically before the completion of their tasks. Furthermore, existing research suggests that practical attackers typically favor executing a DSA within a finite timeframe due to their limited computational resources. These observations serve as the impetus for this paper to investigate a time-restricted DSA (TR-DSA) model on Proof-of-Work based blockchains. In this TR-DSA model, an attacker only mines its branch within a finite timeframe, and the TR-DSA is considered unsuccessful if the attacker's branch fails to surpass the honest miners' branch when the honest miners' branch has grown by a specific number of blocks. First, we developed a general closed-form expression for the success probability of a TR-DSA. This developed probability not only can assist in evaluating the risk of a DSA on blockchain applications with timely tasks, but also can enable practical attackers with limited computational resources to assess the feasibility and expected reward of launching a TR-DSA. In addition, we provide rigorous proof that the success probability of a TR-DSA is no greater than that of a time-unrestricted DSA where the attacker indefinitely mines its branch. This result implies that blockchain applications with timely tasks are less vulnerable to DSAs than blockchain applications that provide attackers with an unlimited timeframe for their attacks. Furthermore, we show that the success probability of a TR-DSA is always smaller than one even though the attacker controls more than half of the hash rate in the network. This result alerts attackers that there is still a risk of failure in launching a TR-DSA even if they amass a majority of the hash rate in the network.

Title: Reasoning in Conversation: Solving Subjective Tasks through Dialogue Simulation for Large Language Models

• Authors: Xiaolong Wang, Yile Wang, Yuanchi Zhang, Fuwen Luo, Peng Li, Maosong Sun, Yang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17226
• Pdf URL: https://arxiv.org/pdf/2402.17226
• Copy Paste: [[2402.17226]] Reasoning in Conversation: Solving Subjective Tasks through Dialogue Simulation for Large Language Models(https://arxiv.org/abs/2402.17226)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable performance in objective tasks such as open-domain question answering and mathematical reasoning, which can often be solved through recalling learned factual knowledge or chain-of-thought style reasoning. However, we find that the performance of LLMs in subjective tasks is still unsatisfactory, such as metaphor recognition, dark humor detection, etc. Compared to objective tasks, subjective tasks focus more on interpretation or emotional response rather than a universally accepted reasoning pathway. Based on the characteristics of the tasks and the strong dialogue-generation capabilities of LLMs, we propose RiC (Reasoning in Conversation), a method that focuses on solving subjective tasks through dialogue simulation. The motivation of RiC is to mine useful contextual information by simulating dialogues instead of supplying chain-of-thought style rationales, thereby offering potential useful knowledge behind dialogues for giving the final answers. We evaluate both API-based and open-source LLMs including GPT-4, ChatGPT, and OpenChat across twelve tasks. Experimental results show that RiC can yield significant improvement compared with various baselines.

Title: Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology

• Authors: Wenhao Tang, Fengtao Zhou, Sheng Huang, Xiang Zhu, Yi Zhang, Bo Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17228
• Pdf URL: https://arxiv.org/pdf/2402.17228
• Copy Paste: [[2402.17228]] Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology(https://arxiv.org/abs/2402.17228)
• Keywords: transformer
• Abstract: Multiple instance learning (MIL) is the most widely used framework in computational pathology, encompassing sub-typing, diagnosis, prognosis, and more. However, the existing MIL paradigm typically requires an offline instance feature extractor, such as a pre-trained ResNet or a foundation model. This approach lacks the capability for feature fine-tuning within the specific downstream tasks, limiting its adaptability and performance. To address this issue, we propose a Re-embedded Regional Transformer (R$^2$T) for re-embedding the instance features online, which captures fine-grained local features and establishes connections across different regions. Unlike existing works that focus on pre-training powerful feature extractor or designing sophisticated instance aggregator, R$^2$T is tailored to re-embed instance features online. It serves as a portable module that can seamlessly integrate into mainstream MIL models. Extensive experimental results on common computational pathology tasks validate that: 1) feature re-embedding improves the performance of MIL models based on ResNet-50 features to the level of foundation model features, and further enhances the performance of foundation model features; 2) the R$^2$T can introduce more significant performance improvements to various MIL models; 3) R$^2$T-MIL, as an R$^2$T-enhanced AB-MIL, outperforms other latest methods by a large margin. The code is available at:~\href{https://github.com/DearCaat/RRT-MIL}{https://github.com/DearCaat/RRT-MIL}.

Title: Preserving Fairness Generalization in Deepfake Detection

• Authors: Li Lin, Xinan He, Yan Ju, Xin Wang, Feng Ding, Shu Hu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17229
• Pdf URL: https://arxiv.org/pdf/2402.17229
• Copy Paste: [[2402.17229]] Preserving Fairness Generalization in Deepfake Detection(https://arxiv.org/abs/2402.17229)
• Keywords: fair
• Abstract: Although effective deepfake detection models have been developed in recent years, recent studies have revealed that these models can result in unfair performance disparities among demographic groups, such as race and gender. This can lead to particular groups facing unfair targeting or exclusion from detection, potentially allowing misclassified deepfakes to manipulate public opinion and undermine trust in the model. The existing method for addressing this problem is providing a fair loss function. It shows good fairness performance for intra-domain evaluation but does not maintain fairness for cross-domain testing. This highlights the significance of fairness generalization in the fight against deepfakes. In this work, we propose the first method to address the fairness generalization problem in deepfake detection by simultaneously considering features, loss, and optimization aspects. Our method employs disentanglement learning to extract demographic and domain-agnostic forgery features, fusing them to encourage fair learning across a flattened loss landscape. Extensive experiments on prominent deepfake datasets demonstrate our method's effectiveness, surpassing state-of-the-art approaches in preserving fairness during cross-domain deepfake detection. The code is available at https://github.com/Purdue-M2/Fairness-Generalization

Title: Chain-of-Thought Prompting of Large Language Models for Discovering and Fixing Software Vulnerabilities

• Authors: Yu Nong, Mohammed Aldeen, Long Cheng, Hongxin Hu, Feng Chen, Haipeng Cai
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.17230
• Pdf URL: https://arxiv.org/pdf/2402.17230
• Copy Paste: [[2402.17230]] Chain-of-Thought Prompting of Large Language Models for Discovering and Fixing Software Vulnerabilities(https://arxiv.org/abs/2402.17230)
• Keywords: security, large language model
• Abstract: Security vulnerabilities are increasingly prevalent in modern software and they are widely consequential to our society. Various approaches to defending against these vulnerabilities have been proposed, among which those leveraging deep learning (DL) avoid major barriers with other techniques hence attracting more attention in recent years. However, DL-based approaches face critical challenges including the lack of sizable and quality-labeled task-specific datasets and their inability to generalize well to unseen, real-world scenarios. Lately, large language models (LLMs) have demonstrated impressive potential in various domains by overcoming those challenges, especially through chain-of-thought (CoT) prompting. In this paper, we explore how to leverage LLMs and CoT to address three key software vulnerability analysis tasks: identifying a given type of vulnerabilities, discovering vulnerabilities of any type, and patching detected vulnerabilities. We instantiate the general CoT methodology in the context of these tasks through VSP , our unified, vulnerability-semantics-guided prompting approach, and conduct extensive experiments assessing VSP versus five baselines for the three tasks against three LLMs and two datasets. Results show substantial superiority of our CoT-inspired prompting (553.3%, 36.5%, and 30.8% higher F1 accuracy for vulnerability identification, discovery, and patching, respectively, on CVE datasets) over the baselines. Through in-depth case studies analyzing VSP failures, we also reveal current gaps in LLM/CoT for challenging vulnerability cases, while proposing and validating respective improvements.

Title: MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning

• Authors: Debrup Das, Debopriyo Banerjee, Somak Aditya, Ashish Kulkarni
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17231
• Pdf URL: https://arxiv.org/pdf/2402.17231
• Copy Paste: [[2402.17231]] MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning(https://arxiv.org/abs/2402.17231)
• Keywords: large language model
• Abstract: Tool-augmented Large Language Models (TALM) are known to enhance the skillset of large language models (LLM), thereby, leading to their improved reasoning abilities across many tasks. While, TALMs have been successfully employed in different question-answering benchmarks, their efficacy on complex mathematical reasoning benchmarks, and the potential complimentary benefits offered by tools for knowledge retrieval and mathematical equation solving, are open research questions. In this work, we present MATHSENSEI, a tool-augmented large language model for mathematical reasoning. Augmented with tools for knowledge retrieval (Bing Web Search), program execution (Python), and symbolic equation solving (Wolfram-Alpha), we study the complimentary benefits of these tools through evaluations on mathematical reasoning datasets. We perform exhaustive ablations on MATH,a popular dataset for evaluating mathematical reasoning on diverse mathematical disciplines. We also conduct experiments involving well-known tool planners to study the impact of tool sequencing on the model performance. MATHSENSEI achieves 13.5% better accuracy over gpt-3.5-turbo with chain-of-thought on the MATH dataset. We further observe that TALMs are not as effective for simpler math word problems (in GSM-8k), and the benefit increases as the complexity and required knowledge increases (progressively over AQuA, MMLU-Math, and higher level complex questions in MATH). The code and data are available at https://github.com/Debrup-61/MathSensei.

Title: Hybrid Square Neural ODE Causal Modeling

• Authors: Bob Junyi Zou, Matthew E. Levine, Dessi P. Zaharieva, Ramesh Johari, Emily B. Fox
• Subjects: cs.LG, stat.AP, stat.ME
• Abstract URL: https://arxiv.org/abs/2402.17233
• Pdf URL: https://arxiv.org/pdf/2402.17233
• Copy Paste: [[2402.17233]] Hybrid Square Neural ODE Causal Modeling(https://arxiv.org/abs/2402.17233)
• Keywords: interpretability
• Abstract: Hybrid models combine mechanistic ODE-based dynamics with flexible and expressive neural network components. Such models have grown rapidly in popularity, especially in scientific domains where such ODE-based modeling offers important interpretability and validated causal grounding (e.g., for counterfactual reasoning). The incorporation of mechanistic models also provides inductive bias in standard blackbox modeling approaches, critical when learning from small datasets or partially observed, complex systems. Unfortunately, as hybrid models become more flexible, the causal grounding provided by the mechanistic model can quickly be lost. We address this problem by leveraging another common source of domain knowledge: ranking of treatment effects for a set of interventions, even if the precise treatment effect is unknown. We encode this information in a causal loss that we combine with the standard predictive loss to arrive at a hybrid loss that biases our learning towards causally valid hybrid models. We demonstrate our ability to achieve a win-win -- state-of-the-art predictive performance and causal validity -- in the challenging task of modeling glucose dynamics during exercise.

Title: HardTaint: Production-Run Dynamic Taint Analysis via Selective Hardware Tracing

• Authors: Yiyu Zhang, Tianyi Liu, Yueyang Wang, Yun Qi, Kai Ji, Jian Tang, Xiaoliang Wang, Xuandong Li, Zhiqiang Zuo
• Subjects: cs.CR, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.17241
• Pdf URL: https://arxiv.org/pdf/2402.17241
• Copy Paste: [[2402.17241]] HardTaint: Production-Run Dynamic Taint Analysis via Selective Hardware Tracing(https://arxiv.org/abs/2402.17241)
• Keywords: security, privacy
• Abstract: Dynamic taint analysis (DTA), as a fundamental analysis technique, is widely used in security, privacy, and diagnosis, etc. As DTA demands to collect and analyze massive taint data online, it suffers extremely high runtime overhead. Over the past decades, numerous attempts have been made to lower the overhead of DTA. Unfortunately, the reductions they achieved are marginal, causing DTA only applicable to the debugging/testing scenarios. In this paper, we propose and implement HardTaint, a system that can realize production-run dynamic taint tracking. HardTaint adopts a hybrid and systematic design which combines static analysis, selective hardware tracing and parallel graph processing techniques. The comprehensive evaluations demonstrate that HardTaint introduces only around 9% runtime overhead which is an order of magnitude lower than the state-of-the-arts, while without sacrificing any taint detection capability.

Title: Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation

• Authors: Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, Suhail Doshi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17245
• Pdf URL: https://arxiv.org/pdf/2402.17245
• Copy Paste: [[2402.17245]] Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation(https://arxiv.org/abs/2402.17245)
• Keywords: diffusion, generative
• Abstract: In this work, we share three insights for achieving state-of-the-art aesthetic quality in text-to-image generative models. We focus on three critical aspects for model improvement: enhancing color and contrast, improving generation across multiple aspect ratios, and improving human-centric fine details. First, we delve into the significance of the noise schedule in training a diffusion model, demonstrating its profound impact on realism and visual fidelity. Second, we address the challenge of accommodating various aspect ratios in image generation, emphasizing the importance of preparing a balanced bucketed dataset. Lastly, we investigate the crucial role of aligning model outputs with human preferences, ensuring that generated images resonate with human perceptual expectations. Through extensive analysis and experiments, Playground v2.5 demonstrates state-of-the-art performance in terms of aesthetic quality under various conditions and aspect ratios, outperforming both widely-used open-source models like SDXL and Playground v2, and closed-source commercial systems such as DALLE 3 and Midjourney v5.2. Our model is open-source, and we hope the development of Playground v2.5 provides valuable guidelines for researchers aiming to elevate the aesthetic quality of diffusion-based image generation models.

Title: Deep Learning-Based Speech and Vision Synthesis to Improve Phishing Attack Detection through a Multi-layer Adaptive Framework

• Authors: Tosin Ige, Christopher Kiekintveld, Aritran Piplai
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17249
• Pdf URL: https://arxiv.org/pdf/2402.17249
• Copy Paste: [[2402.17249]] Deep Learning-Based Speech and Vision Synthesis to Improve Phishing Attack Detection through a Multi-layer Adaptive Framework(https://arxiv.org/abs/2402.17249)
• Keywords: attack
• Abstract: The ever-evolving ways attacker continues to im prove their phishing techniques to bypass existing state-of-the-art phishing detection methods pose a mountain of challenges to researchers in both industry and academia research due to the inability of current approaches to detect complex phishing attack. Thus, current anti-phishing methods remain vulnerable to complex phishing because of the increasingly sophistication tactics adopted by attacker coupled with the rate at which new tactics are being developed to evade detection. In this research, we proposed an adaptable framework that combines Deep learning and Randon Forest to read images, synthesize speech from deep-fake videos, and natural language processing at various predictions layered to significantly increase the performance of machine learning models for phishing attack detection.

Title: Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection

• Authors: Pei Wang, Keqing He, Yejie Wang, Xiaoshuai Song, Yutao Mou, Jingang Wang, Yunsen Xian, Xunliang Cai, Weiran Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17256
• Pdf URL: https://arxiv.org/pdf/2402.17256
• Copy Paste: [[2402.17256]] Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection(https://arxiv.org/abs/2402.17256)
• Keywords: large language model
• Abstract: Out-of-domain (OOD) intent detection aims to examine whether the user's query falls outside the predefined domain of the system, which is crucial for the proper functioning of task-oriented dialogue (TOD) systems. Previous methods address it by fine-tuning discriminative models. Recently, some studies have been exploring the application of large language models (LLMs) represented by ChatGPT to various downstream tasks, but it is still unclear for their ability on OOD detection task.This paper conducts a comprehensive evaluation of LLMs under various experimental settings, and then outline the strengths and weaknesses of LLMs. We find that LLMs exhibit strong zero-shot and few-shot capabilities, but is still at a disadvantage compared to models fine-tuned with full resource. More deeply, through a series of additional analysis experiments, we discuss and summarize the challenges faced by LLMs and provide guidance for future work including injecting domain knowledge, strengthening knowledge transfer from IND(In-domain) to OOD, and understanding long instructions.

Title: RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences

• Authors: Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, Fei-Yue Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17257
• Pdf URL: https://arxiv.org/pdf/2402.17257
• Copy Paste: [[2402.17257]] RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences(https://arxiv.org/abs/2402.17257)
• Keywords: robust
• Abstract: Preference-based Reinforcement Learning (PbRL) avoids the need for reward engineering by harnessing human preferences as the reward signal. However, current PbRL algorithms over-reliance on high-quality feedback from domain experts, which results in a lack of robustness. In this paper, we present RIME, a robust PbRL algorithm for effective reward learning from noisy preferences. Our method incorporates a sample selection-based discriminator to dynamically filter denoised preferences for robust training. To mitigate the accumulated error caused by incorrect selection, we propose to warm start the reward model, which additionally bridges the performance gap during transition from pre-training to online training in PbRL. Our experiments on robotic manipulation and locomotion tasks demonstrate that RIME significantly enhances the robustness of the current state-of-the-art PbRL method. Ablation studies further demonstrate that the warm start is crucial for both robustness and feedback-efficiency in limited-feedback cases.

Title: Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue

• Authors: Zhenhong Zhou, Jiuyang Xiang, Haopeng Chen, Quan Liu, Zherui Li, Sen Su
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17262
• Pdf URL: https://arxiv.org/pdf/2402.17262
• Copy Paste: [[2402.17262]] Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue(https://arxiv.org/abs/2402.17262)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been demonstrated to generate illegal or unethical responses, particularly when subjected to "jailbreak." Research on jailbreak has highlighted the safety issues of LLMs. However, prior studies have predominantly focused on single-turn dialogue, ignoring the potential complexities and risks presented by multi-turn dialogue, a crucial mode through which humans derive information from LLMs. In this paper, we argue that humans could exploit multi-turn dialogue to induce LLMs into generating harmful information. LLMs may not intend to reject cautionary or borderline unsafe queries, even if each turn is closely served for one malicious purpose in a multi-turn dialogue. Therefore, by decomposing an unsafe query into several sub-queries for multi-turn dialogue, we induced LLMs to answer harmful sub-questions incrementally, culminating in an overall harmful response. Our experiments, conducted across a wide range of LLMs, indicate current inadequacies in the safety mechanisms of LLMs in multi-turn dialogue. Our findings expose vulnerabilities of LLMs in complex scenarios involving multi-turn dialogue, presenting new challenges for the safety of LLMs.

Title: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning

• Authors: Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, Jiahuan Pei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17263
• Pdf URL: https://arxiv.org/pdf/2402.17263
• Copy Paste: [[2402.17263]] Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning(https://arxiv.org/abs/2402.17263)
• Keywords: large language model
• Abstract: Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models' scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to full-parameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. The core idea is to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters. This can capture a significant degree of diversity among mini LoRAs, thus promoting better generalization ability. We conduct a theoretical analysis and empirical studies on various NLP tasks. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks, which demonstrates the effectiveness of MELoRA.

Title: Explicit Interaction for Fusion-Based Place Recognition

• Authors: Jingyi Xu, Junyi Ma, Qi Wu, Zijie Zhou, Yue Wang, Xieyuanli Chen, Ling Pei
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.17264
• Pdf URL: https://arxiv.org/pdf/2402.17264
• Copy Paste: [[2402.17264]] Explicit Interaction for Fusion-Based Place Recognition(https://arxiv.org/abs/2402.17264)
• Keywords: robust
• Abstract: Fusion-based place recognition is an emerging technique jointly utilizing multi-modal perception data, to recognize previously visited places in GPS-denied scenarios for robots and autonomous vehicles. Recent fusion-based place recognition methods combine multi-modal features in implicit manners. While achieving remarkable results, they do not explicitly consider what the individual modality affords in the fusion system. Therefore, the benefit of multi-modal feature fusion may not be fully explored. In this paper, we propose a novel fusion-based network, dubbed EINet, to achieve explicit interaction of the two modalities. EINet uses LiDAR ranges to supervise more robust vision features for long time spans, and simultaneously uses camera RGB data to improve the discrimination of LiDAR point clouds. In addition, we develop a new benchmark for the place recognition task based on the nuScenes dataset. To establish this benchmark for future research with comprehensive comparisons, we introduce both supervised and self-supervised training schemes alongside evaluation protocols. We conduct extensive experiments on the proposed benchmark, and the experimental results show that our EINet exhibits better recognition performance as well as solid generalization ability compared to the state-of-the-art fusion-based place recognition approaches. Our open-source code and benchmark are released at: https://github.com/BIT-XJY/EINet.

Title: One-Shot Structure-Aware Stylized Image Synthesis

• Authors: Hansam Cho, Jonghyun Lee, Seunggyu Chang, Yonghyun Jeong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17275
• Pdf URL: https://arxiv.org/pdf/2402.17275
• Copy Paste: [[2402.17275]] One-Shot Structure-Aware Stylized Image Synthesis(https://arxiv.org/abs/2402.17275)
• Keywords: robust, diffusion
• Abstract: While GAN-based models have been successful in image stylization tasks, they often struggle with structure preservation while stylizing a wide range of input images. Recently, diffusion models have been adopted for image stylization but still lack the capability to maintain the original quality of input images. Building on this, we propose OSASIS: a novel one-shot stylization method that is robust in structure preservation. We show that OSASIS is able to effectively disentangle the semantics from the structure of an image, allowing it to control the level of content and style implemented to a given input. We apply OSASIS to various experimental settings, including stylization with out-of-domain reference images and stylization with text-driven manipulation. Results show that OSASIS outperforms other stylization methods, especially for input images that were rarely encountered during training, providing a promising solution to stylization via diffusion models.

Title: Enhancing Hyperspectral Images via Diffusion Model and Group-Autoencoder Super-resolution Network

• Authors: Zhaoyang Wang, Dongyang Li, Mingyang Zhang, Hao Luo, Maoguo Gong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17285
• Pdf URL: https://arxiv.org/pdf/2402.17285
• Copy Paste: [[2402.17285]] Enhancing Hyperspectral Images via Diffusion Model and Group-Autoencoder Super-resolution Network(https://arxiv.org/abs/2402.17285)
• Keywords: diffusion, generative
• Abstract: Existing hyperspectral image (HSI) super-resolution (SR) methods struggle to effectively capture the complex spectral-spatial relationships and low-level details, while diffusion models represent a promising generative model known for their exceptional performance in modeling complex relations and learning high and low-level visual features. The direct application of diffusion models to HSI SR is hampered by challenges such as difficulties in model convergence and protracted inference time. In this work, we introduce a novel Group-Autoencoder (GAE) framework that synergistically combines with the diffusion model to construct a highly effective HSI SR model (DMGASR). Our proposed GAE framework encodes high-dimensional HSI data into low-dimensional latent space where the diffusion model works, thereby alleviating the difficulty of training the diffusion model while maintaining band correlation and considerably reducing inference time. Experimental results on both natural and remote sensing hyperspectral datasets demonstrate that the proposed method is superior to other state-of-the-art methods both visually and metrically.

Title: An Interpretable Evaluation of Entropy-based Novelty of Generative Models

• Authors: Jingwei Zhang, Cheuk Ting Li, Farzan Farnia
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.17287
• Pdf URL: https://arxiv.org/pdf/2402.17287
• Copy Paste: [[2402.17287]] An Interpretable Evaluation of Entropy-based Novelty of Generative Models(https://arxiv.org/abs/2402.17287)
• Keywords: generative
• Abstract: The massive developments of generative model frameworks and architectures require principled methods for the evaluation of a model's novelty compared to a reference dataset or baseline generative models. While the recent literature has extensively studied the evaluation of the quality, diversity, and generalizability of generative models, the assessment of a model's novelty compared to a baseline model has not been adequately studied in the machine learning community. In this work, we focus on the novelty assessment under multi-modal generative models and attempt to answer the following question: Given the samples of a generative model $\mathcal{G}$ and a reference dataset $\mathcal{S}$, how can we discover and count the modes expressed by $\mathcal{G}$ more frequently than in $\mathcal{S}$. We introduce a spectral approach to the described task and propose the Kernel-based Entropic Novelty (KEN) score to quantify the mode-based novelty of distribution $P_\mathcal{G}$ with respect to distribution $P_\mathcal{S}$. We analytically interpret the behavior of the KEN score under mixture distributions with sub-Gaussian components. Next, we develop a method based on Cholesky decomposition to compute the KEN score from observed samples. We support the KEN-based quantification of novelty by presenting several numerical results on synthetic and real image distributions. Our numerical results indicate the success of the proposed approach in detecting the novel modes and the comparison of state-of-the-art generative models.

Title: DivAvatar: Diverse 3D Avatar Generation with a Single Prompt

• Authors: Weijing Tao, Biwen Lei, Kunhao Liu, Shijian Lu, Miaomiao Cui, Xuansong Xie, Chunyan Miao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17292
• Pdf URL: https://arxiv.org/pdf/2402.17292
• Copy Paste: [[2402.17292]] DivAvatar: Diverse 3D Avatar Generation with a Single Prompt(https://arxiv.org/abs/2402.17292)
• Keywords: diffusion, generative
• Abstract: Text-to-Avatar generation has recently made significant strides due to advancements in diffusion models. However, most existing work remains constrained by limited diversity, producing avatars with subtle differences in appearance for a given text prompt. We design DivAvatar, a novel framework that generates diverse avatars, empowering 3D creatives with a multitude of distinct and richly varied 3D avatars from a single text prompt. Different from most existing work that exploits scene-specific 3D representations such as NeRF, DivAvatar finetunes a 3D generative model (i.e., EVA3D), allowing diverse avatar generation from simply noise sampling in inference time. DivAvatar has two key designs that help achieve generation diversity and visual quality. The first is a noise sampling technique during training phase which is critical in generating diverse appearances. The second is a semantic-aware zoom mechanism and a novel depth loss, the former producing appearances of high textual fidelity by separate fine-tuning of specific body parts and the latter improving geometry quality greatly by smoothing the generated mesh in the features space. Extensive experiments show that DivAvatar is highly versatile in generating avatars of diverse appearances.

Title: Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

• Authors: Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, Alice Oh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17302
• Pdf URL: https://arxiv.org/pdf/2402.17302
• Copy Paste: [[2402.17302]] Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese(https://arxiv.org/abs/2402.17302)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators. Our experiments show that the current best-performing LLM, GPT-4 Turbo, is capable of generating questions with adequate knowledge in Indonesian but not in Sundanese, highlighting the performance discrepancy between medium- and lower-resource languages. We also benchmark various LLMs on our generated datasets and find that they perform better on the LLM-generated datasets compared to those created by humans.

Title: Probing Multimodal Large Language Models for Global and Local Semantic Representation

• Authors: Mingxu Tao, Quzhe Huang, Kun Xu, Liwei Chen, Yansong Feng, Dongyan Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17304
• Pdf URL: https://arxiv.org/pdf/2402.17304
• Copy Paste: [[2402.17304]] Probing Multimodal Large Language Models for Global and Local Semantic Representation(https://arxiv.org/abs/2402.17304)
• Keywords: large language model
• Abstract: The success of large language models has inspired researchers to transfer their exceptional representing ability to other modalities. Several recent works leverage image-caption alignment datasets to train multimodal large language models (MLLMs), which achieve state-of-the-art performance on image-to-text tasks. However, there are very few studies exploring whether MLLMs truly understand the complete image information, i.e., global information, or if they can only capture some local object information. In this study, we find that the intermediate layers of models can encode more global semantic information, whose representation vectors perform better on visual-language entailment tasks, rather than the topmost layers. We further probe models for local semantic representation through object detection tasks. And we draw a conclusion that the topmost layers may excessively focus on local information, leading to a diminished ability to encode global information.

Title: SKT5SciSumm - A Hybrid Generative Approach for Multi-Document Scientific Summarization

• Authors: Huy Quoc To, Hung-Nghiep Tran, Andr'e Greiner-Petter, Felix Beierle, Akiko Aizawa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17311
• Pdf URL: https://arxiv.org/pdf/2402.17311
• Copy Paste: [[2402.17311]] SKT5SciSumm - A Hybrid Generative Approach for Multi-Document Scientific Summarization(https://arxiv.org/abs/2402.17311)
• Keywords: transformer, generative
• Abstract: Summarization for scientific text has shown significant benefits both for the research community and human society. Given the fact that the nature of scientific text is distinctive and the input of the multi-document summarization task is substantially long, the task requires sufficient embedding generation and text truncation without losing important information. To tackle these issues, in this paper, we propose SKT5SciSumm - a hybrid framework for multi-document scientific summarization (MDSS). We leverage the Sentence-Transformer version of Scientific Paper Embeddings using Citation-Informed Transformers (SPECTER) to encode and represent textual sentences, allowing for efficient extractive summarization using k-means clustering. We employ the T5 family of models to generate abstractive summaries using extracted sentences. SKT5SciSumm achieves state-of-the-art performance on the Multi-XScience dataset. Through extensive experiments and evaluation, we showcase the benefits of our model by using less complicated models to achieve remarkable results, thereby highlighting its potential in advancing the field of multi-document summarization for scientific text.

Title: Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillation

• Authors: Yaofo Chen, Shuaicheng Niu, Shoukai Xu, Hengjie Song, Yaowei Wang, Mingkui Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17316
• Pdf URL: https://arxiv.org/pdf/2402.17316
• Copy Paste: [[2402.17316]] Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillation(https://arxiv.org/abs/2402.17316)
• Keywords: robust
• Abstract: The conventional deep learning paradigm often involves training a deep model on a server and then deploying the model or its distilled ones to resource-limited edge devices. Usually, the models shall remain fixed once deployed (at least for some period) due to the potential high cost of model adaptation for both the server and edge sides. However, in many real-world scenarios, the test environments may change dynamically (known as distribution shifts), which often results in degraded performance. Thus, one has to adapt the edge models promptly to attain promising performance. Moreover, with the increasing data collected at the edge, this paradigm also fails to further adapt the cloud model for better performance. To address these, we encounter two primary challenges: 1) the edge model has limited computation power and may only support forward propagation; 2) the data transmission budget between cloud and edge devices is limited in latency-sensitive scenarios. In this paper, we establish a Cloud-Edge Elastic Model Adaptation (CEMA) paradigm in which the edge models only need to perform forward propagation and the edge models can be adapted online. In our CEMA, to reduce the communication burden, we devise two criteria to exclude unnecessary samples from uploading to the cloud, i.e., dynamic unreliable and low-informative sample exclusion. Based on the uploaded samples, we update and distribute the affine parameters of normalization layers by distilling from the stronger foundation model to the edge model with a sample replay strategy. Extensive experimental results on ImageNet-C and ImageNet-R verify the effectiveness of our CEMA.

Title: A Vanilla Multi-Task Framework for Dense Visual Prediction Solution to 1st VCL Challenge -- Multi-Task Robustness Track

• Authors: Zehui Chen, Qiuchen Wang, Zhenyu Li, Jiaming Liu, Shanghang Zhang, Feng Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17319
• Pdf URL: https://arxiv.org/pdf/2402.17319
• Copy Paste: [[2402.17319]] A Vanilla Multi-Task Framework for Dense Visual Prediction Solution to 1st VCL Challenge -- Multi-Task Robustness Track(https://arxiv.org/abs/2402.17319)
• Keywords: robust, segmentation
• Abstract: In this report, we present our solution to the multi-task robustness track of the 1st Visual Continual Learning (VCL) Challenge at ICCV 2023 Workshop. We propose a vanilla framework named UniNet that seamlessly combines various visual perception algorithms into a multi-task model. Specifically, we choose DETR3D, Mask2Former, and BinsFormer for 3D object detection, instance segmentation, and depth estimation tasks, respectively. The final submission is a single model with InternImage-L backbone, and achieves a 49.6 overall score (29.5 Det mAP, 80.3 mTPS, 46.4 Seg mAP, and 7.93 silog) on SHIFT validation set. Besides, we provide some interesting observations in our experiments which may facilitate the development of multi-task learning in dense visual prediction.

Title: SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection

• Authors: Junsu Kim, Hoseong Cho, Jihyeon Kim, Yihalem Yimolal Tiruneh, Seungryul Baek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17323
• Pdf URL: https://arxiv.org/pdf/2402.17323
• Copy Paste: [[2402.17323]] SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection(https://arxiv.org/abs/2402.17323)
• Keywords: diffusion, generative
• Abstract: In the field of class incremental learning (CIL), genera- tive replay has become increasingly prominent as a method to mitigate the catastrophic forgetting, alongside the con- tinuous improvements in generative models. However, its application in class incremental object detection (CIOD) has been significantly limited, primarily due to the com- plexities of scenes involving multiple labels. In this paper, we propose a novel approach called stable diffusion deep generative replay (SDDGR) for CIOD. Our method utilizes a diffusion-based generative model with pre-trained text- to-diffusion networks to generate realistic and diverse syn- thetic images. SDDGR incorporates an iterative refinement strategy to produce high-quality images encompassing old classes. Additionally, we adopt an L2 knowledge distilla- tion technique to improve the retention of prior knowledge in synthetic images. Furthermore, our approach includes pseudo-labeling for old objects within new task images, pre- venting misclassification as background elements. Exten- sive experiments on the COCO 2017 dataset demonstrate that SDDGR significantly outperforms existing algorithms, achieving a new state-of-the-art in various CIOD scenarios. The source code will be made available to the public.

Title: SocialCVAE: Predicting Pedestrian Trajectory via Interaction Conditioned Latents

• Authors: Wei Xiang, Haoteng Yin, He Wang, Xiaogang Jin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17339
• Pdf URL: https://arxiv.org/pdf/2402.17339
• Copy Paste: [[2402.17339]] SocialCVAE: Predicting Pedestrian Trajectory via Interaction Conditioned Latents(https://arxiv.org/abs/2402.17339)
• Keywords: explainability
• Abstract: Pedestrian trajectory prediction is the key technology in many applications for providing insights into human behavior and anticipating human future motions. Most existing empirical models are explicitly formulated by observed human behaviors using explicable mathematical terms with a deterministic nature, while recent work has focused on developing hybrid models combined with learning-based techniques for powerful expressiveness while maintaining explainability. However, the deterministic nature of the learned steering behaviors from the empirical models limits the models' practical performance. To address this issue, this work proposes the social conditional variational autoencoder (SocialCVAE) for predicting pedestrian trajectories, which employs a CVAE to explore behavioral uncertainty in human motion decisions. SocialCVAE learns socially reasonable motion randomness by utilizing a socially explainable interaction energy map as the CVAE's condition, which illustrates the future occupancy of each pedestrian's local neighborhood area. The energy map is generated using an energy-based interaction model, which anticipates the energy cost (i.e., repulsion intensity) of pedestrians' interactions with neighbors. Experimental results on two public benchmarks including 25 scenes demonstrate that SocialCVAE significantly improves prediction accuracy compared with the state-of-the-art methods, with up to 16.85% improvement in Average Displacement Error (ADE) and 69.18% improvement in Final Displacement Error (FDE).

Title: A Scalable Multi-Layered Blockchain Architecture for Enhanced EHR Sharing and Drug Supply Chain Management

• Authors: Reza Javan, Mehrzad Mohammadi, Mohammad Beheshti-Atashgah, Mohammad Reza Aref
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.17342
• Pdf URL: https://arxiv.org/pdf/2402.17342
• Copy Paste: [[2402.17342]] A Scalable Multi-Layered Blockchain Architecture for Enhanced EHR Sharing and Drug Supply Chain Management(https://arxiv.org/abs/2402.17342)
• Keywords: secure, security, privacy, robust
• Abstract: In recent years, the healthcare sector's shift to online platforms has spotlighted challenges concerning data security, privacy, and scalability. Blockchain technology, known for its decentralized, secure, and immutable nature, emerges as a viable solution for these pressing issues. This article presents an innovative Electronic Health Records (EHR) sharing and drug supply chain management framework tailored to address scalability, security, data integrity, traceability, and secure data sharing. The framework introduces five layers and transactions, prioritizing patient-centric healthcare by granting patients comprehensive access control over their health information. This access facilitates smoother processes, such as insurance claims, while maintaining robust security measures. Notably, our implementation of parallelism significantly bolsters scalability and transaction throughput while minimizing network traffic. Performance evaluations conducted through the Caliper benchmark indicate a slight increase in processor consumption during specific transactions, mitigated effectively by parallelization. RAM requirements remain largely stable. Additionally, our approach notably reduces network traffic while tripling transaction throughput. The framework ensures patient privacy, data integrity, access control, and interoperability, aligning with traditional healthcare systems. Moreover, it provides transparency and real-time drug supply monitoring, empowering decision-makers with actionable insights. As healthcare evolves, our framework sets a crucial precedent for innovative, scalable, and secure systems. Future enhancements could focus on scalability, real-world deployment, standardized data formats, reinforced security protocols, privacy preservation, and IoT integration to comply with regulations and meet evolving industry needs.

Title: Towards an Enforceable GDPR Specification

• Authors: François Hublet, Alexander Kvamme, Srđan Krstić
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.17350
• Pdf URL: https://arxiv.org/pdf/2402.17350
• Copy Paste: [[2402.17350]] Towards an Enforceable GDPR Specification(https://arxiv.org/abs/2402.17350)
• Keywords: privacy
• Abstract: While Privacy by Design (PbD) is prescribed by modern privacy regulations such as the EU's GDPR, achieving PbD in real software systems is a notoriously difficult task. One emerging technique to realize PbD is Runtime enforcement (RE), in which an enforcer, loaded with a specification of a system's privacy requirements, observes the actions performed by the system and instructs it to perform actions that will ensure compliance with these requirements at all times. To be able to use RE techniques for PbD, privacy regulations first need to be translated into an enforceable specification. In this paper, we report on our ongoing work in formalizing the GDPR. We first present a set of requirements and an iterative methodology for creating enforceable formal specifications of legal provisions. Then, we report on a preliminary case study in which we used our methodology to derive an enforceable specification of part of the GDPR. Our case study suggests that our methodology can be effectively used to develop accurate enforceable specifications.

Title: RECOST: External Knowledge Guided Data-efficient Instruction Tuning

• Authors: Qi Zhang, Yiming Zhang, Haobo Wang, Junbo Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17355
• Pdf URL: https://arxiv.org/pdf/2402.17355
• Copy Paste: [[2402.17355]] RECOST: External Knowledge Guided Data-efficient Instruction Tuning(https://arxiv.org/abs/2402.17355)
• Keywords: large language model
• Abstract: In the current landscape of large language models (LLMs), the process of instruction tuning serves as an essential step. Considering the high computing power overhead, data-efficient instruction tuning was proposed to reduce the training data size in this process, aiming at selecting high-quality instructional data. Nevertheless, we argue that most current data-efficient instruction-tuning methods are highly dependent on the quality of the original instruction-tuning dataset. When it comes to datasets synthesized by LLMs, a common scenario in this field, dirty samples will even be selected with a higher probability than other samples. To address these challenges, we utilized external knowledge (relevant examples or paragraphs) to evaluate those samples synthesized by LLMs with an in-context-based relative predictive entropy. Based on the new metric, we proposed a framework, dubbed as \textbf{RECOST}, which integrates external-knowledge-base re-ranking and diversity-consistent sampling into a single pipeline. Through extensive experiments on several synthetic datasets (Alpaca and Alpaca-gpt4), we demonstrate the effectiveness of our method and achieve even better results with only \textbf{1\%} of the full dataset.

Title: SoFA: Shielded On-the-fly Alignment via Priority Rule Following

• Authors: Xinyu Lu, Bowen Yu, Yaojie Lu, Hongyu Lin, Haiyang Yu, Le Sun, Xianpei Han, Yongbin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17358
• Pdf URL: https://arxiv.org/pdf/2402.17358
• Copy Paste: [[2402.17358]] SoFA: Shielded On-the-fly Alignment via Priority Rule Following(https://arxiv.org/abs/2402.17358)
• Keywords: robust, large language model
• Abstract: The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values. This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards. This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions. Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules. Therefore, we present PriorityDistill, a semi-automated approach for distilling priority following signals from LLM simulations to ensure robust rule integration and adherence. Our experiments show that this method not only effectively minimizes misalignments utilizing only one general rule but also adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately.

Title: CAPT: Category-level Articulation Estimation from a Single Point Cloud Using Transformer

• Authors: Lian Fu, Ryoichi Ishikawa, Yoshihiro Sato, Takeshi Oishi
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.17360
• Pdf URL: https://arxiv.org/pdf/2402.17360
• Copy Paste: [[2402.17360]] CAPT: Category-level Articulation Estimation from a Single Point Cloud Using Transformer(https://arxiv.org/abs/2402.17360)
• Keywords: robust, transformer
• Abstract: The ability to estimate joint parameters is essential for various applications in robotics and computer vision. In this paper, we propose CAPT: category-level articulation estimation from a point cloud using Transformer. CAPT uses an end-to-end transformer-based architecture for joint parameter and state estimation of articulated objects from a single point cloud. The proposed CAPT methods accurately estimate joint parameters and states for various articulated objects with high precision and robustness. The paper also introduces a motion loss approach, which improves articulation estimation performance by emphasizing the dynamic features of articulated objects. Additionally, the paper presents a double voting strategy to provide the framework with coarse-to-fine parameter estimation. Experimental results on several category datasets demonstrate that our methods outperform existing alternatives for articulation estimation. Our research provides a promising solution for applying Transformer-based architectures in articulated object analysis.

Title: An Efficient MLP-based Point-guided Segmentation Network for Ore Images with Ambiguous Boundary

• Authors: Guodong Sun, Yuting Peng, Le Cheng, Mengya Xu, An Wang, Bo Wu, Hongliang Ren, Yang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17370
• Pdf URL: https://arxiv.org/pdf/2402.17370
• Copy Paste: [[2402.17370]] An Efficient MLP-based Point-guided Segmentation Network for Ore Images with Ambiguous Boundary(https://arxiv.org/abs/2402.17370)
• Keywords: segmentation
• Abstract: The precise segmentation of ore images is critical to the successful execution of the beneficiation process. Due to the homogeneous appearance of the ores, which leads to low contrast and unclear boundaries, accurate segmentation becomes challenging, and recognition becomes problematic. This paper proposes a lightweight framework based on Multi-Layer Perceptron (MLP), which focuses on solving the problem of edge burring. Specifically, we introduce a lightweight backbone better suited for efficiently extracting low-level features. Besides, we design a feature pyramid network consisting of two MLP structures that balance local and global information thus enhancing detection accuracy. Furthermore, we propose a novel loss function that guides the prediction points to match the instance edge points to achieve clear object boundaries. We have conducted extensive experiments to validate the efficacy of our proposed method. Our approach achieves a remarkable processing speed of over 27 frames per second (FPS) with a model size of only 73 MB. Moreover, our method delivers a consistently high level of accuracy, with impressive performance scores of 60.4 and 48.9 in~$AP_{50}^{box}$ and~$AP_{50}^{mask}$ respectively, as compared to the currently available state-of-the-art techniques, when tested on the ore image dataset. The source code will be released at \url{https://github.com/MVME-HBUT/ORENEXT}.

Title: Coupled Laplacian Eigenmaps for Locally-Aware 3D Rigid Point Cloud Matching

• Authors: Matteo Bastico, Etienne Decencière, Laurent Corté, Yannick Tillier, David Ryckelynck
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17372
• Pdf URL: https://arxiv.org/pdf/2402.17372
• Copy Paste: [[2402.17372]] Coupled Laplacian Eigenmaps for Locally-Aware 3D Rigid Point Cloud Matching(https://arxiv.org/abs/2402.17372)
• Keywords: robust
• Abstract: Point cloud matching, a crucial technique in computer vision, medical and robotics fields, is primarily concerned with finding correspondences between pairs of point clouds or voxels. In some practical scenarios, emphasizing local differences is crucial for accurately identifying a correct match, thereby enhancing the overall robustness and reliability of the matching process. Commonly used shape descriptors have several limitations and often fail to provide meaningful local insights on the paired geometries. In this work, we propose a new technique, based on graph Laplacian eigenmaps, to match point clouds by taking into account fine local structures. To deal with the order and sign ambiguity of Laplacian eigenmaps, we introduce a new operator, called Coupled Laplacian, that allows to easily generate aligned eigenspaces for multiple rigidly-registered geometries. We show that the similarity between those aligned high-dimensional spaces provides a locally meaningful score to match shapes. We initially evaluate the performance of the proposed technique in a point-wise manner, specifically focusing on the task of object anomaly localization using the MVTec 3D-AD dataset. Additionally, we define a new medical task, called automatic Bone Side Estimation (BSE), which we address through a global similarity score derived from coupled eigenspaces. In order to test it, we propose a benchmark collecting bone surface structures from various public datasets. Our matching technique, based on Coupled Laplacian, outperforms other methods by reaching an impressive accuracy on both tasks. The code to reproduce our experiments is publicly available at https://github.com/matteo-bastico/CoupledLaplacian and in the Supplementary Code.

Title: Accelerating Diffusion Sampling with Optimized Time Steps

• Authors: Shuchen Xue, Zhaoqiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, Zhenguo Li
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17376
• Pdf URL: https://arxiv.org/pdf/2402.17376
• Copy Paste: [[2402.17376]] Accelerating Diffusion Sampling with Optimized Time Steps(https://arxiv.org/abs/2402.17376)
• Keywords: diffusion
• Abstract: Diffusion probabilistic models (DPMs) have shown remarkable performance in high-resolution image synthesis, but their sampling efficiency is still to be desired due to the typically large number of sampling steps. Recent advancements in high-order numerical ODE solvers for DPMs have enabled the generation of high-quality images with much fewer sampling steps. While this is a significant development, most sampling methods still employ uniform time steps, which is not optimal when using a small number of steps. To address this issue, we propose a general framework for designing an optimization problem that seeks more appropriate time steps for a specific numerical ODE solver for DPMs. This optimization problem aims to minimize the distance between the ground-truth solution to the ODE and an approximate solution corresponding to the numerical solver. It can be efficiently solved using the constrained trust region method, taking less than $15$ seconds. Our extensive experiments on both unconditional and conditional sampling using pixel- and latent-space DPMs demonstrate that, when combined with the state-of-the-art sampling method UniPC, our optimized time steps significantly improve image generation performance in terms of FID scores for datasets such as CIFAR-10 and ImageNet, compared to using uniform time steps.

Title: FairBelief - Assessing Harmful Beliefs in Language Models

• Authors: Mattia Setzu, Marta Marchiori Manerba, Pasquale Minervini, Debora Nozza
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17389
• Pdf URL: https://arxiv.org/pdf/2402.17389
• Copy Paste: [[2402.17389]] FairBelief - Assessing Harmful Beliefs in Language Models(https://arxiv.org/abs/2402.17389)
• Keywords: fair
• Abstract: Language Models (LMs) have been shown to inherit undesired biases that might hurt minorities and underrepresented groups if such systems were integrated into real-world applications without careful fairness auditing. This paper proposes FairBelief, an analytical approach to capture and assess beliefs, i.e., propositions that an LM may embed with different degrees of confidence and that covertly influence its predictions. With FairBelief, we leverage prompting to study the behavior of several state-of-the-art LMs across different previously neglected axes, such as model scale and likelihood, assessing predictions on a fairness dataset specifically designed to quantify LMs' outputs' hurtfulness. Finally, we conclude with an in-depth qualitative assessment of the beliefs emitted by the models. We apply FairBelief to English LMs, revealing that, although these architectures enable high performances on diverse natural language processing tasks, they show hurtful beliefs about specific genders. Interestingly, training procedure and dataset, model scale, and architecture induce beliefs of different degrees of hurtfulness.

Title: Robustness-Congruent Adversarial Training for Secure Machine Learning Model Updates

• Authors: Daniele Angioni, Luca Demetrio, Maura Pintor, Luca Oneto, Davide Anguita, Battista Biggio, Fabio Roli
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.17390
• Pdf URL: https://arxiv.org/pdf/2402.17390
• Copy Paste: [[2402.17390]] Robustness-Congruent Adversarial Training for Secure Machine Learning Model Updates(https://arxiv.org/abs/2402.17390)
• Keywords: secure, security, robust
• Abstract: Machine-learning models demand for periodic updates to improve their average accuracy, exploiting novel architectures and additional data. However, a newly-updated model may commit mistakes that the previous model did not make. Such misclassifications are referred to as negative flips, and experienced by users as a regression of performance. In this work, we show that this problem also affects robustness to adversarial examples, thereby hindering the development of secure model update practices. In particular, when updating a model to improve its adversarial robustness, some previously-ineffective adversarial examples may become misclassified, causing a regression in the perceived security of the system. We propose a novel technique, named robustness-congruent adversarial training, to address this issue. It amounts to fine-tuning a model with adversarial training, while constraining it to retain higher robustness on the adversarial examples that were correctly classified before the update. We show that our algorithm and, more generally, learning with non-regression constraints, provides a theoretically-grounded framework to train consistent estimators. Our experiments on robust models for computer vision confirm that (i) both accuracy and robustness, even if improved after model update, can be affected by negative flips, and (ii) our robustness-congruent adversarial training can mitigate the problem, outperforming competing baseline methods.

Title: Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategies

• Authors: Flavio Petruzzellis, Alberto Testolin, Alessandro Sperduti
• Subjects: cs.CL, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.17396
• Pdf URL: https://arxiv.org/pdf/2402.17396
• Copy Paste: [[2402.17396]] Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategies(https://arxiv.org/abs/2402.17396)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) have revolutionized the field of Natural Language Processing thanks to their ability to reuse knowledge acquired on massive text corpora on a wide variety of downstream tasks, with minimal (if any) tuning steps. At the same time, it has been repeatedly shown that LLMs lack systematic generalization, which allows to extrapolate the learned statistical regularities outside the training distribution. In this work, we offer a systematic benchmarking of GPT-4, one of the most advanced LLMs available, on three algorithmic tasks characterized by the possibility to control the problem difficulty with two parameters. We compare the performance of GPT-4 with that of its predecessor (GPT-3.5) and with a variant of the Transformer-Encoder architecture recently introduced to solve similar tasks, the Neural Data Router. We find that the deployment of advanced prompting techniques allows GPT-4 to reach superior accuracy on all tasks, demonstrating that state-of-the-art LLMs constitute a very strong baseline also in challenging tasks that require systematic generalization.

Title: Investigating Continual Pretraining in Large Language Models: Insights and Implications

• Authors: Çağatay Yıldız, Nishaanth Kanna Ravichandran, Prishruit Punia, Matthias Bethge, Beyza Ermis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17400
• Pdf URL: https://arxiv.org/pdf/2402.17400
• Copy Paste: [[2402.17400]] Investigating Continual Pretraining in Large Language Models: Insights and Implications(https://arxiv.org/abs/2402.17400)
• Keywords: large language model
• Abstract: This paper studies the evolving domain of Continual Learning (CL) in large language models (LLMs), with a focus on developing strategies for efficient and sustainable training. Our primary emphasis is on continual domain-adaptive pretraining, a process designed to equip LLMs with the ability to integrate new information from various domains while retaining previously learned knowledge and enhancing cross-domain knowledge transfer without relying on domain-specific identification. Unlike previous studies, which mostly concentrate on a limited selection of tasks or domains and primarily aim to address the issue of forgetting, our research evaluates the adaptability and capabilities of LLMs to changing data landscapes in practical scenarios. To this end, we introduce a new benchmark designed to measure the adaptability of LLMs to these evolving data environments, offering a comprehensive framework for evaluation. We examine the impact of model size on learning efficacy and forgetting, as well as how the progression and similarity of emerging domains affect the knowledge transfer within these models. Our findings uncover several key insights: (i) when the sequence of domains shows semantic similarity, continual pretraining enables LLMs to better specialize in the current domain compared to stand-alone fine-tuning, (ii) training across a diverse range of domains enhances both backward and forward knowledge transfer, and (iii) smaller models are particularly sensitive to continual pretraining, showing the most significant rates of both forgetting and learning. We posit that our research marks a shift towards establishing a more realistic benchmark for investigating CL in LLMs, and has the potential to play a key role in guiding the direction of future research in the field.

Title: LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning

• Authors: Shentong Mo, Yansen Wang, Xufang Luo, Dongsheng Li
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17406
• Pdf URL: https://arxiv.org/pdf/2402.17406
• Copy Paste: [[2402.17406]] LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning(https://arxiv.org/abs/2402.17406)
• Keywords: transformer
• Abstract: Visual Prompt Tuning (VPT) techniques have gained prominence for their capacity to adapt pre-trained Vision Transformers (ViTs) to downstream visual tasks using specialized learnable tokens termed as prompts. Contemporary VPT methodologies, especially when employed with self-supervised vision transformers, often default to the introduction of new learnable prompts or gated prompt tokens predominantly sourced from the model's previous block. A pivotal oversight in such approaches is their failure to harness the potential of long-range previous blocks as sources of prompts within each self-supervised ViT. To bridge this crucial gap, we introduce Long-term Spatial Prompt Tuning (LSPT) - a revolutionary approach to visual representation learning. Drawing inspiration from the intricacies of the human brain, LSPT ingeniously incorporates long-term gated prompts. This feature serves as temporal coding, curbing the risk of forgetting parameters acquired from earlier blocks. Further enhancing its prowess, LSPT brings into play patch tokens, serving as spatial coding. This is strategically designed to perpetually amass class-conscious features, thereby fortifying the model's prowess in distinguishing and identifying visual categories. To validate the efficacy of our proposed method, we engaged in rigorous experimentation across 5 FGVC and 19 VTAB-1K benchmarks. Our empirical findings underscore the superiority of LSPT, showcasing its ability to set new benchmarks in visual prompt tuning performance.

Title: DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Model

• Authors: Shyam Marjit, Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, Pin-Yu Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17412
• Pdf URL: https://arxiv.org/pdf/2402.17412
• Copy Paste: [[2402.17412]] DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Model(https://arxiv.org/abs/2402.17412)
• Keywords: diffusion, generative
• Abstract: In the realm of subject-driven text-to-image (T2I) generative models, recent developments like DreamBooth and BLIP-Diffusion have led to impressive results yet encounter limitations due to their intensive fine-tuning demands and substantial parameter requirements. While the low-rank adaptation (LoRA) module within DreamBooth offers a reduction in trainable parameters, it introduces a pronounced sensitivity to hyperparameters, leading to a compromise between parameter efficiency and the quality of T2I personalized image synthesis. Addressing these constraints, we introduce \textbf{\textit{DiffuseKronA}}, a novel Kronecker product-based adaptation module that not only significantly reduces the parameter count by 35\% and 99.947\% compared to LoRA-DreamBooth and the original DreamBooth, respectively, but also enhances the quality of image synthesis. Crucially, \textit{DiffuseKronA} mitigates the issue of hyperparameter sensitivity, delivering consistent high-quality generations across a wide range of hyperparameters, thereby diminishing the necessity for extensive fine-tuning. Furthermore, a more controllable decomposition makes \textit{DiffuseKronA} more interpretable and even can achieve up to a 50\% reduction with results comparable to LoRA-Dreambooth. Evaluated against diverse and complex input images and text prompts, \textit{DiffuseKronA} consistently outperforms existing models, producing diverse images of higher quality with improved fidelity and a more accurate color distribution of objects, all the while upholding exceptional parameter efficiency, thus presenting a substantial advancement in the field of T2I generative modeling. Our project page, consisting of links to the code, and pre-trained checkpoints, is available at \href{https://diffusekrona.github.io/}{https://diffusekrona.github.io/}.

Title: CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification

• Authors: Haoran Lai, Qingsong Yao, Zihang Jiang, Rongsheng Wang, Zhiyang He, Xiaodong Tao, S. Kevin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17417
• Pdf URL: https://arxiv.org/pdf/2402.17417
• Copy Paste: [[2402.17417]] CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification(https://arxiv.org/abs/2402.17417)
• Keywords: large language model
• Abstract: The advancement of Zero-Shot Learning in the medical domain has been driven forward by using pre-trained models on large-scale image-text pairs, focusing on image-text alignment. However, existing methods primarily rely on cosine similarity for alignment, which may not fully capture the complex relationship between medical images and reports. To address this gap, we introduce a novel approach called Cross-Attention Alignment for Radiology Zero-Shot Classification (CARZero). Our approach innovatively leverages cross-attention mechanisms to process image and report features, creating a Similarity Representation that more accurately reflects the intricate relationships in medical semantics. This representation is then linearly projected to form an image-text similarity matrix for cross-modality alignment. Additionally, recognizing the pivotal role of prompt selection in zero-shot learning, CARZero incorporates a Large Language Model-based prompt alignment strategy. This strategy standardizes diverse diagnostic expressions into a unified format for both training and inference phases, overcoming the challenges of manual prompt design. Our approach is simple yet effective, demonstrating state-of-the-art performance in zero-shot classification on five official chest radiograph diagnostic test sets, including remarkable results on datasets with long-tail distributions of rare diseases. This achievement is attributed to our new image-text alignment strategy, which effectively addresses the complex relationship between medical images and reports.

Title: ViTaL: An Advanced Framework for Automated Plant Disease Identification in Leaf Images Using Vision Transformers and Linear Projection For Feature Reduction

• Authors: Abhishek Sebastian, Annis Fathima A, Pragna R, Madhan Kumar S, Yaswanth Kannan G, Vinay Murali
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17424
• Pdf URL: https://arxiv.org/pdf/2402.17424
• Copy Paste: [[2402.17424]] ViTaL: An Advanced Framework for Automated Plant Disease Identification in Leaf Images Using Vision Transformers and Linear Projection For Feature Reduction(https://arxiv.org/abs/2402.17424)
• Keywords: security, robust, extraction, transformer
• Abstract: Our paper introduces a robust framework for the automated identification of diseases in plant leaf images. The framework incorporates several key stages to enhance disease recognition accuracy. In the pre-processing phase, a thumbnail resizing technique is employed to resize images, minimizing the loss of critical image details while ensuring computational efficiency. Normalization procedures are applied to standardize image data before feature extraction. Feature extraction is facilitated through a novel framework built upon Vision Transformers, a state-of-the-art approach in image analysis. Additionally, alternative versions of the framework with an added layer of linear projection and blockwise linear projections are explored. This comparative analysis allows for the evaluation of the impact of linear projection on feature extraction and overall model performance. To assess the effectiveness of the proposed framework, various Convolutional Neural Network (CNN) architectures are utilized, enabling a com- prehensive evaluation of linear projection's influence on key evaluation metrics. The findings demonstrate the efficacy of the proposed framework, with the top- performing model achieving a Hamming loss of 0.054. Furthermore, we propose a novel hardware design specifically tailored for scanning diseased leaves in an omnidirectional fashion. The hardware implementation utilizes a Raspberry Pi Compute Module to address low-memory configurations, ensuring practicality and affordability. This innovative hardware solution enhances the overall feasibility and accessibility of the proposed automated disease identification system. This research contributes to the field of agriculture by offering valuable insights and tools for the early detection and management of plant diseases, potentially leading to improved crop yields and enhanced food security.

Title: Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder

• Authors: Jiaqi Wang, Zhenxi Song, Zhengyu Ma, Xipeng Qiu, Min Zhang, Zhiguo Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17433
• Pdf URL: https://arxiv.org/pdf/2402.17433
• Copy Paste: [[2402.17433]] Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder(https://arxiv.org/abs/2402.17433)
• Keywords: large language model
• Abstract: Reconstructing natural language from non-invasive electroencephalography (EEG) holds great promise as a language decoding technology for brain-computer interfaces (BCIs). However, EEG-based language decoding is still in its nascent stages, facing several technical issues such as: 1) Absence of a hybrid strategy that can effectively integrate cross-modality (between EEG and text) self-learning with intra-modality self-reconstruction of EEG features or textual sequences; 2) Under-utilization of large language models (LLMs) to enhance EEG-based language decoding. To address above issues, we propose the Contrastive EEG-Text Masked Autoencoder (CET-MAE), a novel model that orchestrates compound self-supervised learning across and within EEG and text through a dedicated multi-stream encoder. Furthermore, we develop a framework called E2T-PTR (EEG-to-Text decoding using Pretrained Transferable Representations), which leverages pre-trained modules alongside the EEG stream from CET-MAE and further enables an LLM (specifically BART) to decode text from EEG sequences. Comprehensive experiments conducted on the popular text-evoked EEG database, ZuCo, demonstrate the superiority of E2T-PTR, which outperforms the state-of-the-art in ROUGE-1 F1 and BLEU-4 scores by 8.34% and 32.21%, respectively. These results indicate significant advancements in the field and underscores the proposed framework's potential to enable more powerful and widespread BCI applications.

Title: Principled Architecture-aware Scaling of Hyperparameters

• Authors: Wuyang Chen, Junru Wu, Zhangyang Wang, Boris Hanin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17440
• Pdf URL: https://arxiv.org/pdf/2402.17440
• Copy Paste: [[2402.17440]] Principled Architecture-aware Scaling of Hyperparameters(https://arxiv.org/abs/2402.17440)
• Keywords: fair
• Abstract: Training a high-quality deep neural network requires choosing suitable hyperparameters, which is a non-trivial and expensive process. Current works try to automatically optimize or design principles of hyperparameters, such that they can generalize to diverse unseen scenarios. However, most designs or optimization methods are agnostic to the choice of network structures, and thus largely ignore the impact of neural architectures on hyperparameters. In this work, we precisely characterize the dependence of initializations and maximal learning rates on the network architecture, which includes the network depth, width, convolutional kernel size, and connectivity patterns. By pursuing every parameter to be maximally updated with the same mean squared change in pre-activations, we can generalize our initialization and learning rates across MLPs (multi-layer perception) and CNNs (convolutional neural network) with sophisticated graph topologies. We verify our principles with comprehensive experiments. More importantly, our strategy further sheds light on advancing current benchmarks for architecture design. A fair comparison of AutoML algorithms requires accurate network rankings. However, we demonstrate that network rankings can be easily changed by better training networks in benchmarks with our architecture-aware learning rates and initialization.

Title: Deep Learning Based Named Entity Recognition Models for Recipes

• Authors: Mansi Goel, Ayush Agarwal, Shubham Agrawal, Janak Kapuriya, Akhil Vamshi Konam, Rishabh Gupta, Shrey Rastogi, Niharika, Ganesh Bagler
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.17447
• Pdf URL: https://arxiv.org/pdf/2402.17447
• Copy Paste: [[2402.17447]] Deep Learning Based Named Entity Recognition Models for Recipes(https://arxiv.org/abs/2402.17447)
• Keywords: extraction, transformer, large language model
• Abstract: Food touches our lives through various endeavors, including flavor, nourishment, health, and sustainability. Recipes are cultural capsules transmitted across generations via unstructured text. Automated protocols for recognizing named entities, the building blocks of recipe text, are of immense value for various applications ranging from information extraction to novel recipe generation. Named entity recognition is a technique for extracting information from unstructured or semi-structured data with known labels. Starting with manually-annotated data of 6,611 ingredient phrases, we created an augmented dataset of 26,445 phrases cumulatively. Simultaneously, we systematically cleaned and analyzed ingredient phrases from RecipeDB, the gold-standard recipe data repository, and annotated them using the Stanford NER. Based on the analysis, we sampled a subset of 88,526 phrases using a clustering-based approach while preserving the diversity to create the machine-annotated dataset. A thorough investigation of NER approaches on these three datasets involving statistical, fine-tuning of deep learning-based language models and few-shot prompting on large language models (LLMs) provides deep insights. We conclude that few-shot prompting on LLMs has abysmal performance, whereas the fine-tuned spaCy-transformer emerges as the best model with macro-F1 scores of 95.9%, 96.04%, and 95.71% for the manually-annotated, augmented, and machine-annotated datasets, respectively.

Title: DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning

• Authors: Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, Jun Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17453
• Pdf URL: https://arxiv.org/pdf/2402.17453
• Copy Paste: [[2402.17453]] DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning(https://arxiv.org/abs/2402.17453)
• Keywords: large language model
• Abstract: In this work, we investigate the potential of large language models (LLMs) based agents to automate data science tasks, with the goal of comprehending task requirements, then building and training the best-fit machine learning models. Despite their widespread success, existing LLM agents are hindered by generating unreasonable experiment plans within this scenario. To this end, we present DS-Agent, a novel automatic framework that harnesses LLM agent and case-based reasoning (CBR). In the development stage, DS-Agent follows the CBR framework to structure an automatic iteration pipeline, which can flexibly capitalize on the expert knowledge from Kaggle, and facilitate consistent performance improvement through the feedback mechanism. Moreover, DS-Agent implements a low-resource deployment stage with a simplified CBR paradigm to adapt past successful solutions from the development stage for direct code generation, significantly reducing the demand on foundational capabilities of LLMs. Empirically, DS-Agent with GPT-4 achieves an unprecedented 100% success rate in the development stage, while attaining 36% improvement on average one pass rate across alternative LLMs in the deployment stage. In both stages, DS-Agent achieves the best rank in performance, costing \$1.60 and \$0.13 per run with GPT-4, respectively.

Title: Why do Learning Rates Transfer? Reconciling Optimization and Scaling Limits for Deep Learning

• Authors: Lorenzo Noci, Alexandru Meterez, Thomas Hofmann, Antonio Orvieto
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17457
• Pdf URL: https://arxiv.org/pdf/2402.17457
• Copy Paste: [[2402.17457]] Why do Learning Rates Transfer? Reconciling Optimization and Scaling Limits for Deep Learning(https://arxiv.org/abs/2402.17457)
• Keywords: transformer
• Abstract: Recently, there has been growing evidence that if the width and depth of a neural network are scaled toward the so-called rich feature learning limit ($\mu$P and its depth extension), then some hyperparameters - such as the learning rate - exhibit transfer from small to very large models, thus reducing the cost of hyperparameter tuning. From an optimization perspective, this phenomenon is puzzling, as it implies that the loss landscape is remarkably consistent across very different model sizes. In this work, we find empirical evidence that learning rate transfer can be attributed to the fact that under $\mu$P and its depth extension, the largest eigenvalue of the training loss Hessian (i.e. the sharpness) is largely independent of the width and depth of the network for a sustained period of training time. On the other hand, we show that under the neural tangent kernel (NTK) regime, the sharpness exhibits very different dynamics at different scales, thus preventing learning rate transfer. But what causes these differences in the sharpness dynamics? Through a connection between the spectra of the Hessian and the NTK matrix, we argue that the cause lies in the presence (for $\mu$P) or progressive absence (for the NTK regime) of feature learning, which results in a different evolution of the NTK, and thus of the sharpness. We corroborate our claims with a substantial suite of experiments, covering a wide range of datasets and architectures: from ResNets and Vision Transformers trained on benchmark vision datasets to Transformers-based language models trained on WikiText

Title: Training-Free Long-Context Scaling of Large Language Models

• Authors: Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, Lingpeng Kong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17463
• Pdf URL: https://arxiv.org/pdf/2402.17463
• Copy Paste: [[2402.17463]] Training-Free Long-Context Scaling of Large Language Models(https://arxiv.org/abs/2402.17463)
• Keywords: large language model
• Abstract: The ability of Large Language Models (LLMs) to process and generate coherent text is markedly weakened when the number of input tokens exceeds their pretraining length. Given the expensive overhead of finetuning large-scale models with longer sequences, we propose Dual Chunk Attention (DCA), which enables Llama2 70B to support context windows of more than 100k tokens without continual training. By decomposing the attention computation for long sequences into chunk-based modules, DCA manages to effectively capture the relative positional information of tokens within the same chunk (Intra-Chunk) and across distinct chunks (Inter-Chunk), as well as integrates seamlessly with Flash Attention. In addition to its impressive extrapolation capability, DCA achieves performance on practical long-context tasks that is comparable to or even better than that of finetuned models. When compared with proprietary models, our training-free 70B model attains 94% of the performance of gpt-3.5-16k, indicating it is a viable open-source alternative. All code and data used in this work are released at \url{https://github.com/HKUNLP/ChunkLlama}.

Title: Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing

• Authors: Bi'an Du, Xiang Gao, Wei Hu, Renjie Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17464
• Pdf URL: https://arxiv.org/pdf/2402.17464
• Copy Paste: [[2402.17464]] Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing(https://arxiv.org/abs/2402.17464)
• Keywords: interpretability, generative
• Abstract: Generative 3D part assembly involves understanding part relationships and predicting their 6-DoF poses for assembling a realistic 3D shape. Prior work often focus on the geometry of individual parts, neglecting part-whole hierarchies of objects. Leveraging two key observations: 1) super-part poses provide strong hints about part poses, and 2) predicting super-part poses is easier due to fewer superparts, we propose a part-whole-hierarchy message passing network for efficient 3D part assembly. We first introduce super-parts by grouping geometrically similar parts without any semantic labels. Then we employ a part-whole hierarchical encoder, wherein a super-part encoder predicts latent super-part poses based on input parts. Subsequently, we transform the point cloud using the latent poses, feeding it to the part encoder for aggregating super-part information and reasoning about part relationships to predict all part poses. In training, only ground-truth part poses are required. During inference, the predicted latent poses of super-parts enhance interpretability. Experimental results on the PartNet dataset show that our method achieves state-of-the-art performance in part and connectivity accuracy and enables an interpretable hierarchical part assembly.

Title: Model X-ray:Detect Backdoored Models via Decision Boundary

• Authors: Yanghao Su, Jie Zhang, Ting Xu, Tianwei Zhang, Weiming Zhang, Nenghai Yu
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17465
• Pdf URL: https://arxiv.org/pdf/2402.17465
• Copy Paste: [[2402.17465]] Model X-ray:Detect Backdoored Models via Decision Boundary(https://arxiv.org/abs/2402.17465)
• Keywords: attack
• Abstract: Deep neural networks (DNNs) have revolutionized various industries, leading to the rise of Machine Learning as a Service (MLaaS). In this paradigm, well-trained models are typically deployed through APIs. However, DNNs are susceptible to backdoor attacks, which pose significant risks to their applications. This vulnerability necessitates a method for users to ascertain whether an API is compromised before usage. Although many backdoor detection methods have been developed, they often operate under the assumption that the defender has access to specific information such as details of the attack, soft predictions from the model API, and even the knowledge of the model parameters, limiting their practicality in MLaaS scenarios. To address it, in this paper, we begin by presenting an intriguing observation: the decision boundary of the backdoored model exhibits a greater degree of closeness than that of the clean model. Simultaneously, if only one single label is infected, a larger portion of the regions will be dominated by the attacked label. Building upon this observation, we propose Model X-ray, a novel backdoor detection approach for MLaaS through the analysis of decision boundaries. Model X-ray can not only identify whether the target API is infected by backdoor attacks but also determine the target attacked label under the all-to-one attack strategy. Importantly, it accomplishes this solely by the hard prediction of clean inputs, regardless of any assumptions about attacks and prior knowledge of the training details of the model. Extensive experiments demonstrated that Model X-ray can be effective for MLaaS across diverse backdoor attacks, datasets, and architectures.

Title: Fraud Detection with Binding Global and Local Relational Interaction

• Authors: Haolin Li, Shuyang Jiang, Lifeng Zhang, Siyuan Du, Guangnan Ye, Hongfeng Chai
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17472
• Pdf URL: https://arxiv.org/pdf/2402.17472
• Copy Paste: [[2402.17472]] Fraud Detection with Binding Global and Local Relational Interaction(https://arxiv.org/abs/2402.17472)
• Keywords: transformer
• Abstract: Graph Neural Network has been proved to be effective for fraud detection for its capability to encode node interaction and aggregate features in a holistic view. Recently, Transformer network with great sequence encoding ability, has also outperformed other GNN-based methods in literatures. However, both GNN-based and Transformer-based networks only encode one perspective of the whole graph, while GNN encodes global features and Transformer network encodes local ones. Furthermore, previous works ignored encoding global interaction features of the heterogeneous graph with separate networks, thus leading to suboptimal performance. In this work, we present a novel framework called Relation-Aware GNN with transFormer (RAGFormer) which simultaneously embeds local and global features into a target node. The simple yet effective network applies a modified GAGA module where each transformer layer is followed by a cross-relation aggregation layer, to encode local embeddings and node interactions across different relations. Apart from the Transformer-based network, we further introduce a Relation-Aware GNN module to learn global embeddings, which is later merged into the local embeddings by an attention fusion module and a skip connection. Extensive experiments on two popular public datasets and an industrial dataset demonstrate that RAGFormer achieves the state-of-the-art performance. Substantial analysis experiments validate the effectiveness of each submodule of RAGFormer and its high efficiency in utilizing small-scale data and low hyper-parameter sensitivity.

Title: Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articles

• Authors: Maram Hasanain, Fatema Ahmed, Firoj Alam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17478
• Pdf URL: https://arxiv.org/pdf/2402.17478
• Copy Paste: [[2402.17478]] Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articles(https://arxiv.org/abs/2402.17478)
• Keywords: large language model
• Abstract: The use of propaganda has spiked on mainstream and social media, aiming to manipulate or mislead users. While efforts to automatically detect propaganda techniques in textual, visual, or multimodal content have increased, most of them primarily focus on English content. The majority of the recent initiatives targeting medium to low-resource languages produced relatively small annotated datasets, with a skewed distribution, posing challenges for the development of sophisticated propaganda detection models. To address this challenge, we carefully develop the largest propaganda dataset to date, ArPro, comprised of 8K paragraphs from newspaper articles, labeled at the text span level following a taxonomy of 23 propagandistic techniques. Furthermore, our work offers the first attempt to understand the performance of large language models (LLMs), using GPT-4, for fine-grained propaganda detection from text. Results showed that GPT-4's performance degrades as the task moves from simply classifying a paragraph as propagandistic or not, to the fine-grained task of detecting propaganda techniques and their manifestation in text. Compared to models fine-tuned on the dataset for propaganda detection at different classification granularities, GPT-4 is still far behind. Finally, we evaluate GPT-4 on a dataset consisting of six other languages for span detection, and results suggest that the model struggles with the task across languages. Our dataset and resources will be released to the community.

Title: EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

• Authors: Linrui Tian, Qi Wang, Bang Zhang, Liefeng Bo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17485
• Pdf URL: https://arxiv.org/pdf/2402.17485
• Copy Paste: [[2402.17485]] EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions(https://arxiv.org/abs/2402.17485)
• Keywords: diffusion
• Abstract: In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.

Title: MGE: A Training-Free and Efficient Model Generation and Enhancement Scheme

• Authors: Xuan Wang, Zeshan Pang, Yuliang Lu, Xuehu Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17486
• Pdf URL: https://arxiv.org/pdf/2402.17486
• Copy Paste: [[2402.17486]] MGE: A Training-Free and Efficient Model Generation and Enhancement Scheme(https://arxiv.org/abs/2402.17486)
• Keywords: defense
• Abstract: To provide a foundation for the research of deep learning models, the construction of model pool is an essential step. This paper proposes a Training-Free and Efficient Model Generation and Enhancement Scheme (MGE). This scheme primarily considers two aspects during the model generation process: the distribution of model parameters and model performance. Experiments result shows that generated models are comparable to models obtained through normal training, and even superior in some cases. Moreover, the time consumed in generating models accounts for only 1\% of the time required for normal model training. More importantly, with the enhancement of Evolution-MGE, generated models exhibits competitive generalization ability in few-shot tasks. And the behavioral dissimilarity of generated models has the potential of adversarial defense.

Title: Complexity Assessment of Analog Security Primitives Using the Disentropy of Autocorrelation

• Authors: Paul Jimenez, Raphael Cardoso, Maurìcio Gomes de Queiroz, Mohab Abdalla, Cédric Marchand, Xavier Letartre, Fabio Pavanello
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.17488
• Pdf URL: https://arxiv.org/pdf/2402.17488
• Copy Paste: [[2402.17488]] Complexity Assessment of Analog Security Primitives Using the Disentropy of Autocorrelation(https://arxiv.org/abs/2402.17488)
• Keywords: security
• Abstract: The study of regularity in signals can be of great importance, typically in medicine to analyse electrocardiogram (ECG) or electromyography (EMG) signals, but also in climate studies, finance or security. In this work we focus on security primitives such as Physical Unclonable Functions (PUFs) or Pseudo-Random Number Generators (PRNGs). Such primitives must have a high level of complexity or entropy in their responses to guarantee enough security for their applications. There are several ways of assessing the complexity of their responses, especially in the binary domain. With the development of analog PUFs such as optical (photonic) PUFs, it would be useful to be able to assess their complexity in the analog domain when designing them, for example, before converting analog signals into binary. In this numerical study, we decided to explore the potential of the disentropy of autocorrelation as a measure of complexity for security primitives as PUFs or PRNGs with analog output or responses. We compare this metric to others used to assess regularities in analog signals such as Approximate Entropy (ApEn) and Fuzzy Entropy (FuzEn). We show that the disentropy of autocorrelation is able to differentiate between well-known PRNGs and non-optimised or bad PRNGs in the analog and binary domain with a better contrast than ApEn and FuzEn. Next, we show that the disentropy of autocorrelation is able to detect small patterns injected in PUFs responses and then we applied it to photonic PUFs simulations.

Title: Prescribing Large Language Models for Perioperative Care: What's The Right Dose for Pre-trained Models?

• Authors: Bing Xue, Charles Alba, Joanna Abraham, Thomas Kannampallil, Chenyang Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17493
• Pdf URL: https://arxiv.org/pdf/2402.17493
• Copy Paste: [[2402.17493]] Prescribing Large Language Models for Perioperative Care: What's The Right Dose for Pre-trained Models?(https://arxiv.org/abs/2402.17493)
• Keywords: large language model
• Abstract: Postoperative risk predictions can inform effective perioperative care management and planning. We aimed to assess whether clinical large language models (LLMs) can predict postoperative risks using clinical texts with various training strategies. The main cohort involved 84,875 records from Barnes Jewish Hospital (BJH) system between 2018 and 2021. Methods were replicated on Beth Israel Deaconess's MIMIC dataset. Both studies had mean duration of follow-up based on the length of postoperative ICU stay less than 7 days. For the BJH dataset, outcomes included 30-day mortality, pulmonary embolism (PE) and pneumonia. Three domain adaptation and finetuning strategies were implemented for BioGPT, ClinicalBERT and BioClinicalBERT: self-supervised objectives; incorporating labels with semi-supervised fine-tuning; and foundational modelling through multi-task learning. Model performance was compared using the area under the receiver operating characteristic curve (AUROC) and the area under the precision recall curve (AUPRC) for classification tasks, and mean squared error (MSE) and R2 for regression tasks. Pre-trained LLMs outperformed traditional word embeddings, with absolute maximal gains of 38.3% for AUROC and 14% for AUPRC. Adapting models further improved performance: (1) self-supervised finetuning by 3.2% for AUROC and 1.5% for AUPRC; (2) semi-supervised finetuning by 1.8% for AUROC and 2% for AUPRC, compared to self-supervised finetuning; (3) foundational modelling by 3.6% for AUROC and 2.6% for AUPRC, compared to self-supervised finetuning. Pre-trained clinical LLMs offer opportunities for postoperative risk predictions in unforeseen data, with peaks in foundational models indicating the potential of task-agnostic learning towards the generalizability of LLMs in perioperative care.

Title: REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering

• Authors: Yuhao Wang, Ruiyang Ren, Junyi Li, Wayne Xin Zhao, Jing Liu, Ji-Rong Wen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.17497
• Pdf URL: https://arxiv.org/pdf/2402.17497
• Copy Paste: [[2402.17497]] REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering(https://arxiv.org/abs/2402.17497)
• Keywords: large language model
• Abstract: Considering the limited internal parametric knowledge, retrieval-augmented generation (RAG) has been widely used to extend the knowledge scope of large language models (LLMs). Despite the extensive efforts on RAG research, in existing methods, LLMs cannot precisely assess the relevance of retrieved documents, thus likely leading to misleading or even incorrect utilization of external knowledge (i.e., retrieved documents). To address this issue, in this paper, we propose REAR, a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA). As the key motivation, we aim to enhance the self-awareness of source relevance for LLMs, so as to adaptively utilize external knowledge in RAG systems. Specially, we develop a new architecture for LLM based RAG system, by incorporating a specially designed rank head that precisely assesses the relevance of retrieved documents. Furthermore, we propose an improved training method based on bi-granularity relevance fusion and noise-resistant training. By combining the improvements in both architecture and training, our proposed REAR can better utilize external knowledge by effectively perceiving the relevance of retrieved documents. Experiments on four open-domain QA tasks show that REAR significantly outperforms previous a number of competitive RAG approaches. Our code and data can be accessed at https://github.com/RUCAIBox/REAR.

Title: Intensive Care as One Big Sequence Modeling Problem

• Authors: Vadim Liventsev, Tobias Fritz
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17501
• Pdf URL: https://arxiv.org/pdf/2402.17501
• Copy Paste: [[2402.17501]] Intensive Care as One Big Sequence Modeling Problem(https://arxiv.org/abs/2402.17501)
• Keywords: transformer, large language model
• Abstract: Reinforcement Learning in Healthcare is typically concerned with narrow self-contained tasks such as sepsis prediction or anesthesia control. However, previous research has demonstrated the potential of generalist models (the prime example being Large Language Models) to outperform task-specific approaches due to their capability for implicit transfer learning. To enable training of foundation models for Healthcare as well as leverage the capabilities of state of the art Transformer architectures, we propose the paradigm of Healthcare as Sequence Modeling, in which interaction between the patient and the healthcare provider is represented as an event stream and tasks like diagnosis and treatment selection are modeled as prediction of future events in the stream. To explore this paradigm experimentally we develop MIMIC-SEQ, a sequence modeling benchmark derived by translating heterogenous clinical records from MIMIC-IV dataset into a uniform event stream format, train a baseline model and explore its capabilities.

Title: FedLPPA: Learning Personalized Prompt and Aggregation for Federated Weakly-supervised Medical Image Segmentation

• Authors: Li Lin, Yixiang Liu, Jiewei Wu, Pujin Cheng, Zhiyuan Cai, Kenneth K. Y. Wong, Xiaoying Tang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.17502
• Pdf URL: https://arxiv.org/pdf/2402.17502
• Copy Paste: [[2402.17502]] FedLPPA: Learning Personalized Prompt and Aggregation for Federated Weakly-supervised Medical Image Segmentation(https://arxiv.org/abs/2402.17502)
• Keywords: privacy, federate, segmentation
• Abstract: Federated learning (FL) effectively mitigates the data silo challenge brought about by policies and privacy concerns, implicitly harnessing more data for deep model training. However, traditional centralized FL models grapple with diverse multi-center data, especially in the face of significant data heterogeneity, notably in medical contexts. In the realm of medical image segmentation, the growing imperative to curtail annotation costs has amplified the importance of weakly-supervised techniques which utilize sparse annotations such as points, scribbles, etc. A pragmatic FL paradigm shall accommodate diverse annotation formats across different sites, which research topic remains under-investigated. In such context, we propose a novel personalized FL framework with learnable prompt and aggregation (FedLPPA) to uniformly leverage heterogeneous weak supervision for medical image segmentation. In FedLPPA, a learnable universal knowledge prompt is maintained, complemented by multiple learnable personalized data distribution prompts and prompts representing the supervision sparsity. Integrated with sample features through a dual-attention mechanism, those prompts empower each local task decoder to adeptly adjust to both the local distribution and the supervision form. Concurrently, a dual-decoder strategy, predicated on prompt similarity, is introduced for enhancing the generation of pseudo-labels in weakly-supervised learning, alleviating overfitting and noise accumulation inherent to local data, while an adaptable aggregation method is employed to customize the task decoder on a parameter-wise basis. Extensive experiments on three distinct medical image segmentation tasks involving different modalities underscore the superiority of FedLPPA, with its efficacy closely parallels that of fully supervised centralized training. Our code and data will be available.

Title: Extreme Miscalibration and the Illusion of Adversarial Robustness

• Authors: Vyas Raina, Samson Tan, Volkan Cevher, Aditya Rawal, Sheng Zha, George Karypis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17509
• Pdf URL: https://arxiv.org/pdf/2402.17509
• Copy Paste: [[2402.17509]] Extreme Miscalibration and the Illusion of Adversarial Robustness(https://arxiv.org/abs/2402.17509)
• Keywords: attack, robust
• Abstract: Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify. Adversarial Training (AT) is often used to increase model robustness. However, we have discovered an intriguing phenomenon: deliberately or accidentally miscalibrating models masks gradients in a way that interferes with adversarial attack search methods, giving rise to an apparent increase in robustness. We show that this observed gain in robustness is an illusion of robustness (IOR), and demonstrate how an adversary can perform various forms of test-time temperature calibration to nullify the aforementioned interference and allow the adversarial attack to find adversarial examples. Hence, we urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations to ensure that any observed gains are genuine. Finally, we show how the temperature can be scaled during \textit{training} to improve genuine robustness.

Title: Latent Attention for Linear Time Transformers

• Authors: Rares Dolga, Marius Cobzarenco, David Barber
• Subjects: cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.17512
• Pdf URL: https://arxiv.org/pdf/2402.17512
• Copy Paste: [[2402.17512]] Latent Attention for Linear Time Transformers(https://arxiv.org/abs/2402.17512)
• Keywords: transformer
• Abstract: The time complexity of the standard attention mechanism in a transformer scales quadratically with the length of the sequence. We introduce a method to reduce this to linear scaling with time, based on defining attention via latent vectors. The method is readily usable as a drop-in replacement for the standard attention mechanism. Our "Latte Transformer" model can be implemented for both bidirectional and unidirectional tasks, with the causal version allowing a recurrent implementation which is memory and time-efficient during inference of language generation tasks. Whilst next token prediction scales linearly with the sequence length for a standard transformer, a Latte Transformer requires constant time to compute the next token. The empirical performance of our method is comparable to standard attention, yet allows scaling to context windows much larger than practical in standard attention.

Title: Robust Unsupervised Crowd Counting and Localization with Adaptive Resolution SAM

• Authors: Jia Wan, Qiangqiang Wu, Wei Lin, Antoni B. Chan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17514
• Pdf URL: https://arxiv.org/pdf/2402.17514
• Copy Paste: [[2402.17514]] Robust Unsupervised Crowd Counting and Localization with Adaptive Resolution SAM(https://arxiv.org/abs/2402.17514)
• Keywords: robust, segmentation
• Abstract: The existing crowd counting models require extensive training data, which is time-consuming to annotate. To tackle this issue, we propose a simple yet effective crowd counting method by utilizing the Segment-Everything-Everywhere Model (SEEM), an adaptation of the Segmentation Anything Model (SAM), to generate pseudo-labels for training crowd counting models. However, our initial investigation reveals that SEEM's performance in dense crowd scenes is limited, primarily due to the omission of many persons in high-density areas. To overcome this limitation, we propose an adaptive resolution SEEM to handle the scale variations, occlusions, and overlapping of people within crowd scenes. Alongside this, we introduce a robust localization method, based on Gaussian Mixture Models, for predicting the head positions in the predicted people masks. Given the mask and point pseudo-labels, we propose a robust loss function, which is designed to exclude uncertain regions based on SEEM's predictions, thereby enhancing the training process of the counting networks. Finally, we propose an iterative method for generating pseudo-labels. This method aims at improving the quality of the segmentation masks by identifying more tiny persons in high-density regions, which are often missed in the first pseudo-labeling stage. Overall, our proposed method achieves the best unsupervised performance in crowd counting, while also being comparable results to some supervised methods. This makes it a highly effective and versatile tool for crowd counting, especially in situations where labeled data is not available.

Title: QUCE: The Minimisation and Quantification of Path-Based Uncertainty for Generative Counterfactual Explanations

• Authors: Jamie Duell, Hsuan Fu, Monika Seisenberger, Xiuyi Fan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17516
• Pdf URL: https://arxiv.org/pdf/2402.17516
• Copy Paste: [[2402.17516]] QUCE: The Minimisation and Quantification of Path-Based Uncertainty for Generative Counterfactual Explanations(https://arxiv.org/abs/2402.17516)
• Keywords: interpretability, generative
• Abstract: Deep Neural Networks (DNNs) stand out as one of the most prominent approaches within the Machine Learning (ML) domain. The efficacy of DNNs has surged alongside recent increases in computational capacity, allowing these approaches to scale to significant complexities for addressing predictive challenges in big data. However, as the complexity of DNN models rises, interpretability diminishes. In response to this challenge, explainable models such as Adversarial Gradient Integration (AGI) leverage path-based gradients provided by DNNs to elucidate their decisions. Yet the performance of path-based explainers can be compromised when gradients exhibit irregularities during out-of-distribution path traversal. In this context, we introduce Quantified Uncertainty Counterfactual Explanations (QUCE), a method designed to mitigate out-of-distribution traversal by minimizing path uncertainty. QUCE not only quantifies uncertainty when presenting explanations but also generates more certain counterfactual examples. We showcase the performance of the QUCE method by comparing it with competing methods for both path-based explanations and generative counterfactual examples. The code repository for the QUCE method is available at: https://github.com/jamie-duell/QUCE.

Title: Label-Noise Robust Diffusion Models

• Authors: Byeonghu Na, Yeongmin Kim, HeeSun Bae, Jung Hyun Lee, Se Jung Kwon, Wanmo Kang, Il-Chul Moon
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17517
• Pdf URL: https://arxiv.org/pdf/2402.17517
• Copy Paste: [[2402.17517]] Label-Noise Robust Diffusion Models(https://arxiv.org/abs/2402.17517)
• Keywords: robust, diffusion, generative
• Abstract: Conditional diffusion models have shown remarkable performance in various generative tasks, but training them requires large-scale datasets that often contain noise in conditional inputs, a.k.a. noisy labels. This noise leads to condition mismatch and quality degradation of generated data. This paper proposes Transition-aware weighted Denoising Score Matching (TDSM) for training conditional diffusion models with noisy labels, which is the first study in the line of diffusion models. The TDSM objective contains a weighted sum of score networks, incorporating instance-wise and time-dependent label transition probabilities. We introduce a transition-aware weight estimator, which leverages a time-dependent noisy-label classifier distinctively customized to the diffusion process. Through experiments across various datasets and noisy label settings, TDSM improves the quality of generated samples aligned with given conditions. Furthermore, our method improves generation performance even on prevalent benchmark datasets, which implies the potential noisy labels and their risk of generative model learning. Finally, we show the improved performance of TDSM on top of conventional noisy label corrections, which empirically proving its contribution as a part of label-noise robust generative models. Our code is available at: https://github.com/byeonghu-na/tdsm.

Title: AVS-Net: Point Sampling with Adaptive Voxel Size for 3D Point Cloud Analysis

• Authors: Hongcheng Yang, Dingkang Liang, Dingyuan Zhang, Xingyu Jiang, Zhe Liu, Zhikang Zou, Yingying Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17521
• Pdf URL: https://arxiv.org/pdf/2402.17521
• Copy Paste: [[2402.17521]] AVS-Net: Point Sampling with Adaptive Voxel Size for 3D Point Cloud Analysis(https://arxiv.org/abs/2402.17521)
• Keywords: extraction
• Abstract: Efficient downsampling plays a crucial role in point cloud learning, particularly for large-scale 3D scenes. Existing downsampling methods either require a huge computational burden or sacrifice fine-grained geometric information. This paper presents an advanced sampler that achieves both high accuracy and efficiency. The proposed method utilizes voxel-based sampling as a foundation, but effectively addresses the challenges regarding voxel size determination and the preservation of critical geometric cues. Specifically, we propose a Voxel Adaptation Module that adaptively adjusts voxel sizes with the reference of point-based downsampling ratio. This ensures the sampling results exhibit a favorable distribution for comprehending various 3D objects or scenes. Additionally, we introduce a network compatible with arbitrary voxel sizes for sampling and feature extraction while maintaining high efficiency. Our method achieves state-of-the-art accuracy on the ShapeNetPart and ScanNet benchmarks with promising efficiency. Code will be available at https://github.com/yhc2021/AVS-Net.

Title: Diffusion Model-Based Image Editing: A Survey

• Authors: Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, Liangliang Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17525
• Pdf URL: https://arxiv.org/pdf/2402.17525
• Copy Paste: [[2402.17525]] Diffusion Model-Based Image Editing: A Survey(https://arxiv.org/abs/2402.17525)
• Keywords: diffusion
• Abstract: Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks, facilitating the synthesis of visual content in an unconditional or input-conditional manner. The core idea behind them is learning to reverse the process of gradually adding noise to images, allowing them to generate high-quality samples from a complex distribution. In this survey, we provide an exhaustive overview of existing methods using diffusion models for image editing, covering both theoretical and practical aspects in the field. We delve into a thorough analysis and categorization of these works from multiple perspectives, including learning strategies, user-input conditions, and the array of specific editing tasks that can be accomplished. In addition, we pay special attention to image inpainting and outpainting, and explore both earlier traditional context-driven and current multimodal conditional methods, offering a comprehensive analysis of their methodologies. To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval, featuring an innovative metric, LMM Score. Finally, we address current limitations and envision some potential directions for future research. The accompanying repository is released at https://github.com/SiatMMLab/Awesome-Diffusion-Model-Based-Image-Editing-Methods.

Title: Predict the Next Word:

• Authors: Evgenia Ilia, Wilker Aziz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17527
• Pdf URL: https://arxiv.org/pdf/2402.17527
• Copy Paste: [[2402.17527]] Predict the Next Word: (https://arxiv.org/abs/2402.17527)
• Keywords: robust, fair
• Abstract: Language models (LMs) are statistical models trained to assign probability to human-generated text. As such, it is reasonable to question whether they approximate linguistic variability exhibited by humans well. This form of statistical assessment is difficult to perform at the passage level, for it requires acceptability judgements (i.e., human evaluation) or a robust automated proxy (which is non-trivial). At the word level, however, given some context, samples from an LM can be assessed via exact matching against a prerecorded dataset of alternative single-word continuations of the available context. We exploit this fact and evaluate the LM's ability to reproduce variability that humans (in particular, a population of English speakers) exhibit in the 'next word prediction' task. This can be seen as assessing a form of calibration, which, in the context of text classification, Baan et al. (2022) termed calibration to human uncertainty. We assess GPT2, BLOOM and ChatGPT and find that they exhibit fairly low calibration to human uncertainty. We also verify the failure of expected calibration error (ECE) to reflect this, and as such, advise the community against relying on it in this setting.

Title: Black-box Adversarial Attacks Against Image Quality Assessment Models

• Authors: Yu Ran, Ao-Xiang Zhang, Mingjie Li, Weixuan Tang, Yuan-Gen Wang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.17533
• Pdf URL: https://arxiv.org/pdf/2402.17533
• Copy Paste: [[2402.17533]] Black-box Adversarial Attacks Against Image Quality Assessment Models(https://arxiv.org/abs/2402.17533)
• Keywords: attack
• Abstract: The goal of No-Reference Image Quality Assessment (NR-IQA) is to predict the perceptual quality of an image in line with its subjective evaluation. To put the NR-IQA models into practice, it is essential to study their potential loopholes for model refinement. This paper makes the first attempt to explore the black-box adversarial attacks on NR-IQA models. Specifically, we first formulate the attack problem as maximizing the deviation between the estimated quality scores of original and perturbed images, while restricting the perturbed image distortions for visual quality preservation. Under such formulation, we then design a Bi-directional loss function to mislead the estimated quality scores of adversarial examples towards an opposite direction with maximum deviation. On this basis, we finally develop an efficient and effective black-box attack method against NR-IQA models. Extensive experiments reveal that all the evaluated NR-IQA models are vulnerable to the proposed attack method. And the generated perturbations are not transferable, enabling them to serve the investigation of specialities of disparate IQA models.

Title: Scribble Hides Class: Promoting Scribble-Based Weakly-Supervised Semantic Segmentation with Its Class Label

• Authors: Xinliang Zhang, Lei Zhu, Hangzhou He, Lujia Jin, Yanye Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17555
• Pdf URL: https://arxiv.org/pdf/2402.17555
• Copy Paste: [[2402.17555]] Scribble Hides Class: Promoting Scribble-Based Weakly-Supervised Semantic Segmentation with Its Class Label(https://arxiv.org/abs/2402.17555)
• Keywords: robust, diffusion, segmentation
• Abstract: Scribble-based weakly-supervised semantic segmentation using sparse scribble supervision is gaining traction as it reduces annotation costs when compared to fully annotated alternatives. Existing methods primarily generate pseudo-labels by diffusing labeled pixels to unlabeled ones with local cues for supervision. However, this diffusion process fails to exploit global semantics and class-specific cues, which are important for semantic segmentation. In this study, we propose a class-driven scribble promotion network, which utilizes both scribble annotations and pseudo-labels informed by image-level classes and global semantics for supervision. Directly adopting pseudo-labels might misguide the segmentation model, thus we design a localization rectification module to correct foreground representations in the feature space. To further combine the advantages of both supervisions, we also introduce a distance entropy loss for uncertainty reduction, which adapts per-pixel confidence weights according to the reliable region determined by the scribble and pseudo-label's boundary. Experiments on the ScribbleSup dataset with different qualities of scribble annotations outperform all the previous methods, demonstrating the superiority and robustness of our method.The code is available at https://github.com/Zxl19990529/Class-driven-Scribble-Promotion-Network.

Title: An Empirical Study of the Generalization Ability of Lidar 3D Object Detectors to Unseen Domains

• Authors: George Eskandar, Chongzhe Zhang, Abhishek Kaushik, Karim Guirguis, Mohamed Sayed, Bin Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17562
• Pdf URL: https://arxiv.org/pdf/2402.17562
• Copy Paste: [[2402.17562]] An Empirical Study of the Generalization Ability of Lidar 3D Object Detectors to Unseen Domains(https://arxiv.org/abs/2402.17562)
• Keywords: robust, transformer
• Abstract: 3D Object Detectors (3D-OD) are crucial for understanding the environment in many robotic tasks, especially autonomous driving. Including 3D information via Lidar sensors improves accuracy greatly. However, such detectors perform poorly on domains they were not trained on, i.e. different locations, sensors, weather, etc., limiting their reliability in safety-critical applications. There exist methods to adapt 3D-ODs to these domains; however, these methods treat 3D-ODs as a black box, neglecting underlying architectural decisions and source-domain training strategies. Instead, we dive deep into the details of 3D-ODs, focusing our efforts on fundamental factors that influence robustness prior to domain adaptation. We systematically investigate four design choices (and the interplay between them) often overlooked in 3D-OD robustness and domain adaptation: architecture, voxel encoding, data augmentations, and anchor strategies. We assess their impact on the robustness of nine state-of-the-art 3D-ODs across six benchmarks encompassing three types of domain gaps - sensor type, weather, and location. Our main findings are: (1) transformer backbones with local point features are more robust than 3D CNNs, (2) test-time anchor size adjustment is crucial for adaptation across geographical locations, significantly boosting scores without retraining, (3) source-domain augmentations allow the model to generalize to low-resolution sensors, and (4) surprisingly, robustness to bad weather is improved when training directly on more clean weather data than on training with bad weather data. We outline our main conclusions and findings to provide practical guidance on developing more robust 3D-ODs.

Title: Structure-Guided Adversarial Training of Diffusion Models

• Authors: Ling Yang, Haotian Qian, Zhilong Zhang, Jingwei Liu, Bin Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17563
• Pdf URL: https://arxiv.org/pdf/2402.17563
• Copy Paste: [[2402.17563]] Structure-Guided Adversarial Training of Diffusion Models(https://arxiv.org/abs/2402.17563)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion models have demonstrated exceptional efficacy in various generative applications. While existing models focus on minimizing a weighted sum of denoising score matching losses for data distribution modeling, their training primarily emphasizes instance-level optimization, overlooking valuable structural information within each mini-batch, indicative of pair-wise relationships among samples. To address this limitation, we introduce Structure-guided Adversarial training of Diffusion Models (SADM). In this pioneering approach, we compel the model to learn manifold structures between samples in each training batch. To ensure the model captures authentic manifold structures in the data distribution, we advocate adversarial training of the diffusion generator against a novel structure discriminator in a minimax game, distinguishing real manifold structures from the generated ones. SADM substantially improves existing diffusion transformers (DiT) and outperforms existing methods in image generation and cross-domain fine-tuning tasks across 12 datasets, establishing a new state-of-the-art FID of 1.58 and 2.11 on ImageNet for class-conditional image generation at resolutions of 256x256 and 512x512, respectively.

Title: Unleashing the Potential of Large Language Models as Prompt Optimizers: An Analogical Analysis with Gradient-based Model Optimizers

• Authors: Xinyu Tang, Xiaolei Wang, Wayne Xin Zhao, Siyuan Lu, Yaliang Li, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17564
• Pdf URL: https://arxiv.org/pdf/2402.17564
• Copy Paste: [[2402.17564]] Unleashing the Potential of Large Language Models as Prompt Optimizers: An Analogical Analysis with Gradient-based Model Optimizers(https://arxiv.org/abs/2402.17564)
• Keywords: large language model
• Abstract: Automatic prompt optimization is an important approach to improving the performance of large language models (LLMs). Recent research demonstrates the potential of using LLMs as prompt optimizers, which can generate improved task prompts via iterative refinement. In this paper, we propose a novel perspective to investigate the design of LLM-based prompt optimizers, by drawing an analogy with gradient-based model optimizers. To connect these two approaches, we identify two pivotal factors in model parameter learning: update direction and update method. Focused on the two aspects, we borrow the theoretical framework and learning methods from gradient-based optimization to design improved strategies for LLM-based prompt optimizers. By systematically analyzing a rich set of improvement strategies, we further develop a capable Gradient-inspired LLM-based Prompt Optimizer called GPO. At each step, it first retrieves relevant prompts from the optimization trajectory as the update direction. Then, it utilizes the generation-based refinement strategy to perform the update, while controlling the edit distance through a cosine-based decay strategy. Extensive experiments demonstrate the effectiveness and efficiency of GPO. In particular, GPO brings an additional improvement of up to 56.8% on Big-Bench Hard and 55.3% on MMLU compared to baseline methods.

Title: Hyperdimensional computing: a fast, robust and interpretable paradigm for biological data

• Authors: Michiel Stock, Dimitri Boeckaerts, Pieter Dewulf, Steff Taelman, Maxime Van Haeverbeke, Wim Van Criekinge, Bernard De Baets
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2402.17572
• Pdf URL: https://arxiv.org/pdf/2402.17572
• Copy Paste: [[2402.17572]] Hyperdimensional computing: a fast, robust and interpretable paradigm for biological data(https://arxiv.org/abs/2402.17572)
• Keywords: robust, interpretability
• Abstract: Advances in bioinformatics are primarily due to new algorithms for processing diverse biological data sources. While sophisticated alignment algorithms have been pivotal in analyzing biological sequences, deep learning has substantially transformed bioinformatics, addressing sequence, structure, and functional analyses. However, these methods are incredibly data-hungry, compute-intensive and hard to interpret. Hyperdimensional computing (HDC) has recently emerged as an intriguing alternative. The key idea is that random vectors of high dimensionality can represent concepts such as sequence identity or phylogeny. These vectors can then be combined using simple operators for learning, reasoning or querying by exploiting the peculiar properties of high-dimensional spaces. Our work reviews and explores the potential of HDC for bioinformatics, emphasizing its efficiency, interpretability, and adeptness in handling multimodal and structured data. HDC holds a lot of potential for various omics data searching, biosignal analysis and health applications.

Title: Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation

• Authors: Xiaohan Lei, Min Wang, Wengang Zhou, Li Li, Houqiang Li
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.17587
• Pdf URL: https://arxiv.org/pdf/2402.17587
• Copy Paste: [[2402.17587]] Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation(https://arxiv.org/abs/2402.17587)
• Keywords: robust, segmentation
• Abstract: As a new embodied vision task, Instance ImageGoal Navigation (IIN) aims to navigate to a specified object depicted by a goal image in an unexplored environment. The main challenge of this task lies in identifying the target object from different viewpoints while rejecting similar distractors. Existing ImageGoal Navigation methods usually adopt the simple Exploration-Exploitation framework and ignore the identification of specific instance during navigation. In this work, we propose to imitate the human behaviour of ``getting closer to confirm" when distinguishing objects from a distance. Specifically, we design a new modular navigation framework named Instance-aware Exploration-Verification-Exploitation (IEVE) for instance-level image goal navigation. Our method allows for active switching among the exploration, verification, and exploitation actions, thereby facilitating the agent in making reasonable decisions under different situations. On the challenging HabitatMatterport 3D semantic (HM3D-SEM) dataset, our method surpasses previous state-of-the-art work, with a classical segmentation model (0.684 vs. 0.561 success) or a robust model (0.702 vs. 0.561 success). Our code will be made publicly available at https://github.com/XiaohanLei/IEVE.

Title: A Large-scale Evaluation of Pretraining Paradigms for the Detection of Defects in Electroluminescence Solar Cell Images

• Authors: David Torpey, Lawrence Pratt, Richard Klein
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17611
• Pdf URL: https://arxiv.org/pdf/2402.17611
• Copy Paste: [[2402.17611]] A Large-scale Evaluation of Pretraining Paradigms for the Detection of Defects in Electroluminescence Solar Cell Images(https://arxiv.org/abs/2402.17611)
• Keywords: segmentation
• Abstract: Pretraining has been shown to improve performance in many domains, including semantic segmentation, especially in domains with limited labelled data. In this work, we perform a large-scale evaluation and benchmarking of various pretraining methods for Solar Cell Defect Detection (SCDD) in electroluminescence images, a field with limited labelled datasets. We cover supervised training with semantic segmentation, semi-supervised learning, and two self-supervised techniques. We also experiment with both in-distribution and out-of-distribution (OOD) pretraining and observe how this affects downstream performance. The results suggest that supervised training on a large OOD dataset (COCO), self-supervised pretraining on a large OOD dataset (ImageNet), and semi-supervised pretraining (CCT) all yield statistically equivalent performance for mean Intersection over Union (mIoU). We achieve a new state-of-the-art for SCDD and demonstrate that certain pretraining schemes result in superior performance on underrepresented classes. Additionally, we provide a large-scale unlabelled EL image dataset of $22000$ images, and a $642$-image labelled semantic segmentation EL dataset, for further research in developing self- and semi-supervised training techniques in this domain.

Title: Adapt Before Comparison: A New Perspective on Cross-Domain Few-Shot Segmentation

• Authors: Jonas Herzog
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17614
• Pdf URL: https://arxiv.org/pdf/2402.17614
• Copy Paste: [[2402.17614]] Adapt Before Comparison: A New Perspective on Cross-Domain Few-Shot Segmentation(https://arxiv.org/abs/2402.17614)
• Keywords: segmentation
• Abstract: Few-shot segmentation performance declines substantially when facing images from a domain different than the training domain, effectively limiting real-world use cases. To alleviate this, recently cross-domain few-shot segmentation (CD-FSS) has emerged. Works that address this task mainly attempted to learn segmentation on a source domain in a manner that generalizes across domains. Surprisingly, we can outperform these approaches while eliminating the training stage and removing their main segmentation network. We show test-time task-adaption is the key for successful CD-FSS instead. Task-adaption is achieved by appending small networks to the feature pyramid of a conventionally classification-pretrained backbone. To avoid overfitting to the few labeled samples in supervised fine-tuning, consistency across augmented views of input images serves as guidance while learning the parameters of the attached layers. Despite our self-restriction not to use any images other than the few labeled samples at test time, we achieve new state-of-the-art performance in CD-FSS, evidencing the need to rethink approaches for the task.

Title: Masked Gamma-SSL: Learning Uncertainty Estimation via Masked Image Modeling

• Authors: David S. W. Williams, Matthew Gadd, Paul Newman, Daniele De Martini
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.17622
• Pdf URL: https://arxiv.org/pdf/2402.17622
• Copy Paste: [[2402.17622]] Masked Gamma-SSL: Learning Uncertainty Estimation via Masked Image Modeling(https://arxiv.org/abs/2402.17622)
• Keywords: robust, segmentation
• Abstract: This work proposes a semantic segmentation network that produces high-quality uncertainty estimates in a single forward pass. We exploit general representations from foundation models and unlabelled datasets through a Masked Image Modeling (MIM) approach, which is robust to augmentation hyper-parameters and simpler than previous techniques. For neural networks used in safety-critical applications, bias in the training data can lead to errors; therefore it is crucial to understand a network's limitations at run time and act accordingly. To this end, we test our proposed method on a number of test domains including the SAX Segmentation benchmark, which includes labelled test data from dense urban, rural and off-road driving domains. The proposed method consistently outperforms uncertainty estimation and Out-of-Distribution (OoD) techniques on this difficult benchmark.

Title: CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing

• Authors: Chufeng Xiao, Hongbo Fu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.17624
• Pdf URL: https://arxiv.org/pdf/2402.17624
• Copy Paste: [[2402.17624]] CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing(https://arxiv.org/abs/2402.17624)
• Keywords: extraction
• Abstract: Personalization techniques for large text-to-image (T2I) models allow users to incorporate new concepts from reference images. However, existing methods primarily rely on textual descriptions, leading to limited control over customized images and failing to support fine-grained and local editing (e.g., shape, pose, and details). In this paper, we identify sketches as an intuitive and versatile representation that can facilitate such control, e.g., contour lines capturing shape information and flow lines representing texture. This motivates us to explore a novel task of sketch concept extraction: given one or more sketch-image pairs, we aim to extract a special sketch concept that bridges the correspondence between the images and sketches, thus enabling sketch-based image synthesis and editing at a fine-grained level. To accomplish this, we introduce CustomSketching, a two-stage framework for extracting novel sketch concepts. Considering that an object can often be depicted by a contour for general shapes and additional strokes for internal details, we introduce a dual-sketch representation to reduce the inherent ambiguity in sketch depiction. We employ a shape loss and a regularization loss to balance fidelity and editability during optimization. Through extensive experiments, a user study, and several applications, we show our method is effective and superior to the adapted baselines.

Title: Securing OPEN-RAN Equipment Using Blockchain-Based Supply Chain Verification

• Authors: Ali Mehrban, Mostafa Jani
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.17632
• Pdf URL: https://arxiv.org/pdf/2402.17632
• Copy Paste: [[2402.17632]] Securing OPEN-RAN Equipment Using Blockchain-Based Supply Chain Verification(https://arxiv.org/abs/2402.17632)
• Keywords: secure, security, robust
• Abstract: The disaggregated and multi-vendor nature of OPEN-RAN networks introduces new supply chain security risks, making equipment authenticity and integrity crucial challenges. Robust solutions are needed to mitigate vulnerabilities in manufacturing and integration. This paper puts forth a novel blockchain-based approach to secure OPEN-RAN equipment through its lifecycle. By combining firmware authentication codes, a permissioned blockchain ledger, and equipment node validators, we architect a tamper-resistant ecosystem to track provenance. The outlined design, while conceptual, establishes a foundation and roadmap for future realization. Through careful implementation planning, development of core components like firmware signed hashes and smart contracts, and rigorous performance evaluation, this paper can evolve from concept to practice. There is a vivid potential to make OPEN-RAN supply chains corner to corner secure, igniting further research and real-world deployment.

Title: From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions

• Authors: Fabian Retkowski, Alexander Waibel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17633
• Pdf URL: https://arxiv.org/pdf/2402.17633
• Copy Paste: [[2402.17633]] From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions(https://arxiv.org/abs/2402.17633)
• Keywords: segmentation
• Abstract: Text segmentation is a fundamental task in natural language processing, where documents are split into contiguous sections. However, prior research in this area has been constrained by limited datasets, which are either small in scale, synthesized, or only contain well-structured documents. In this paper, we address these limitations by introducing a novel benchmark YTSeg focusing on spoken content that is inherently more unstructured and both topically and structurally diverse. As part of this work, we introduce an efficient hierarchical segmentation model MiniSeg, that outperforms state-of-the-art baselines. Lastly, we expand the notion of text segmentation to a more practical "smart chaptering" task that involves the segmentation of unstructured content, the generation of meaningful segment titles, and a potential real-time application of the models.

Title: Variational Learning is Effective for Large Deep Networks

• Authors: Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, Thomas Möllenhoff
• Subjects: cs.LG, cs.AI, cs.CL, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.17641
• Pdf URL: https://arxiv.org/pdf/2402.17641
• Copy Paste: [[2402.17641]] Variational Learning is Effective for Large Deep Networks(https://arxiv.org/abs/2402.17641)
• Keywords: large language model
• Abstract: We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks. We show that an optimizer called Improved Variational Online Newton (IVON) consistently matches or outperforms Adam for training large networks such as GPT-2 and ResNets from scratch. IVON's computational costs are nearly identical to Adam but its predictive uncertainty is better. We show several new use cases of IVON where we improve fine-tuning and model merging in Large Language Models, accurately predict generalization error, and faithfully estimate sensitivity to data. We find overwhelming evidence in support of effectiveness of variational learning.

Title: Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data

• Authors: Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, Yansong Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.17644
• Pdf URL: https://arxiv.org/pdf/2402.17644
• Copy Paste: [[2402.17644]] Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data(https://arxiv.org/abs/2402.17644)
• Keywords: large language model
• Abstract: Quantitative reasoning is a critical skill to analyze data, yet the assessment of such ability remains limited. To address this gap, we introduce the Quantitative Reasoning with Data (QRData) benchmark, aiming to evaluate Large Language Models' capability in statistical and causal reasoning with real-world data. The benchmark comprises a carefully constructed dataset of 411 questions accompanied by data sheets from textbooks, online learning materials, and academic papers. To compare models' quantitative reasoning abilities on data and text, we enrich the benchmark with an auxiliary set of 290 text-only questions, namely QRText. We evaluate natural language reasoning, program-based reasoning, and agent reasoning methods including Chain-of-Thought, Program-of-Thoughts, ReAct, and code interpreter assistants on diverse models. The strongest model GPT-4 achieves an accuracy of 58%, which has a large room for improvement. Among open-source models, Deepseek-coder-instruct, a code LLM pretrained on 2T tokens, gets the highest accuracy of 37%. Analysis reveals that models encounter difficulties in data analysis and causal reasoning, and struggle in using causal knowledge and provided data simultaneously. Code and data are in https://github.com/xxxiaol/QRData.

Title: Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs

• Authors: Tanise Ceron, Neele Falk, Ana Barić, Dmitry Nikolaev, Sebastian Padó
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.17649
• Pdf URL: https://arxiv.org/pdf/2402.17649
• Copy Paste: [[2402.17649]] Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs(https://arxiv.org/abs/2402.17649)
• Keywords: protect, robust, large language model
• Abstract: Due to the widespread use of large language models (LLMs) in ubiquitous systems, we need to understand whether they embed a specific worldview and what these views reflect. Recent studies report that, prompted with political questionnaires, LLMs show left-liberal leanings. However, it is as yet unclear whether these leanings are reliable (robust to prompt variations) and whether the leaning is consistent across policies and political leaning. We propose a series of tests which assess the reliability and consistency of LLMs' stances on political statements based on a dataset of voting-advice questionnaires collected from seven EU countries and annotated for policy domains. We study LLMs ranging in size from 7B to 70B parameters and find that their reliability increases with parameter count. Larger models show overall stronger alignment with left-leaning parties but differ among policy programs: They evince a (left-wing) positive stance towards environment protection, social welfare but also (right-wing) law and order, with no consistent preferences in foreign policy, migration, and economy.

Title: Mitigating Distributional Shift in Semantic Segmentation via Uncertainty Estimation from Unlabelled Data

• Authors: David S. W. Williams, Daniele De Martini, Matthew Gadd, Paul Newman
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.17653
• Pdf URL: https://arxiv.org/pdf/2402.17653
• Copy Paste: [[2402.17653]] Mitigating Distributional Shift in Semantic Segmentation via Uncertainty Estimation from Unlabelled Data(https://arxiv.org/abs/2402.17653)
• Keywords: segmentation
• Abstract: Knowing when a trained segmentation model is encountering data that is different to its training data is important. Understanding and mitigating the effects of this play an important part in their application from a performance and assurance perspective - this being a safety concern in applications such as autonomous vehicles (AVs). This work presents a segmentation network that can detect errors caused by challenging test domains without any additional annotation in a single forward pass. As annotation costs limit the diversity of labelled datasets, we use easy-to-obtain, uncurated and unlabelled data to learn to perform uncertainty estimation by selectively enforcing consistency over data augmentation. To this end, a novel segmentation benchmark based on the SAX Dataset is used, which includes labelled test data spanning three autonomous-driving domains, ranging in appearance from dense urban to off-road. The proposed method, named Gamma-SSL, consistently outperforms uncertainty estimation and Out-of-Distribution (OoD) techniques on this difficult benchmark - by up to 10.7% in area under the receiver operating characteristic (ROC) curve and 19.2% in area under the precision-recall (PR) curve in the most challenging of the three scenarios.

Title: SoK: Cryptocurrency Wallets -- A Security Review and Classification based on Authentication Factors

• Authors: Ivan Homoliak, Martin Perešíni
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.17659
• Pdf URL: https://arxiv.org/pdf/2402.17659
• Copy Paste: [[2402.17659]] SoK: Cryptocurrency Wallets -- A Security Review and Classification based on Authentication Factors(https://arxiv.org/abs/2402.17659)
• Keywords: security
• Abstract: In this work, we review existing cryptocurrency wallet solutions with regard to authentication methods and factors from the user's point of view. In particular, we distinguish between authentication factors that are verified against the blockchain and the ones verified locally (or against a centralized party). With this in mind, we define notions for $k-factor$ authentication against the blockchain and $k-factor$ authentication against the authentication factors. Based on these notions, we propose a classification of authentication schemes. We extend our classification to accommodate the threshold signatures and signing transactions by centralized parties (such as exchanges or co-signing services). Finally, we apply our classification to existing wallet solutions, which we compare based on various security and key-management features.

Title: SDF2Net: Shallow to Deep Feature Fusion Network for PolSAR Image Classification

• Authors: Mohammed Q. Alkhatib, M. Sami Zitouni, Mina Al-Saad, Nour Aburaed, Hussain Al-Ahmad
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.17672
• Pdf URL: https://arxiv.org/pdf/2402.17672
• Copy Paste: [[2402.17672]] SDF2Net: Shallow to Deep Feature Fusion Network for PolSAR Image Classification(https://arxiv.org/abs/2402.17672)
• Keywords: extraction
• Abstract: Polarimetric synthetic aperture radar (PolSAR) images encompass valuable information that can facilitate extensive land cover interpretation and generate diverse output products. Extracting meaningful features from PolSAR data poses challenges distinct from those encountered in optical imagery. Deep learning (DL) methods offer effective solutions for overcoming these challenges in PolSAR feature extraction. Convolutional neural networks (CNNs) play a crucial role in capturing PolSAR image characteristics by leveraging kernel capabilities to consider local information and the complex-valued nature of PolSAR data. In this study, a novel three-branch fusion of complex-valued CNN, named the Shallow to Deep Feature Fusion Network (SDF2Net), is proposed for PolSAR image classification. To validate the performance of the proposed method, classification results are compared against multiple state-of-the-art approaches using the airborne synthetic aperture radar (AIRSAR) datasets of Flevoland and San Francisco, as well as the ESAR Oberpfaffenhofen dataset. The results indicate that the proposed approach demonstrates improvements in overallaccuracy, with a 1.3% and 0.8% enhancement for the AIRSAR datasets and a 0.5% improvement for the ESAR dataset. Analyses conducted on the Flevoland data underscore the effectiveness of the SDF2Net model, revealing a promising overall accuracy of 96.01% even with only a 1% sampling ratio.

Title: RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations

• Authors: Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, Atticus Geiger
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17700
• Pdf URL: https://arxiv.org/pdf/2402.17700
• Copy Paste: [[2402.17700]] RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations(https://arxiv.org/abs/2402.17700)
• Keywords: interpretability
• Abstract: Individual neurons participate in the representation of multiple high-level concepts. To what extent can different interpretability methods successfully disentangle these roles? To help address this question, we introduce RAVEL (Resolving Attribute-Value Entanglements in Language Models), a dataset that enables tightly controlled, quantitative comparisons between a variety of existing interpretability methods. We use the resulting conceptual framework to define the new method of Multi-task Distributed Alignment Search (MDAS), which allows us to find distributed representations satisfying multiple causal criteria. With Llama2-7B as the target language model, MDAS achieves state-of-the-art results on RAVEL, demonstrating the importance of going beyond neuron-level analyses to identify features distributed across activations. We release our benchmark at https://github.com/explanare/ravel.

Title: Federated Learning for Estimating Heterogeneous Treatment Effects

• Authors: Disha Makhija, Joydeep Ghosh, Yejin Kim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17705
• Pdf URL: https://arxiv.org/pdf/2402.17705
• Copy Paste: [[2402.17705]] Federated Learning for Estimating Heterogeneous Treatment Effects(https://arxiv.org/abs/2402.17705)
• Keywords: federate, transformer
• Abstract: Machine learning methods for estimating heterogeneous treatment effects (HTE) facilitate large-scale personalized decision-making across various domains such as healthcare, policy making, education, and more. Current machine learning approaches for HTE require access to substantial amounts of data per treatment, and the high costs associated with interventions makes centrally collecting so much data for each intervention a formidable challenge. To overcome this obstacle, in this work, we propose a novel framework for collaborative learning of HTE estimators across institutions via Federated Learning. We show that even under a diversity of interventions and subject populations across clients, one can jointly learn a common feature representation, while concurrently and privately learning the specific predictive functions for outcomes under distinct interventions across institutions. Our framework and the associated algorithm are based on this insight, and leverage tabular transformers to map multiple input data to feature representations which are then used for outcome prediction via multi-task learning. We also propose a novel way of federated training of personalised transformers that can work with heterogeneous input feature spaces. Experimental results on real-world clinical trial data demonstrate the effectiveness of our method.

Title: Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers

• Authors: Yiwei Lu, Yaoliang Yu, Xinlin Li, Vahid Partovi Nia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17710
• Pdf URL: https://arxiv.org/pdf/2402.17710
• Copy Paste: [[2402.17710]] Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers(https://arxiv.org/abs/2402.17710)
• Keywords: transformer
• Abstract: In neural network binarization, BinaryConnect (BC) and its variants are considered the standard. These methods apply the sign function in their forward pass and their respective gradients are backpropagated to update the weights. However, the derivative of the sign function is zero whenever defined, which consequently freezes training. Therefore, implementations of BC (e.g., BNN) usually replace the derivative of sign in the backward computation with identity or other approximate gradient alternatives. Although such practice works well empirically, it is largely a heuristic or ''training trick.'' We aim at shedding some light on these training tricks from the optimization perspective. Building from existing theory on ProxConnect (PC, a generalization of BC), we (1) equip PC with different forward-backward quantizers and obtain ProxConnect++ (PC++) that includes existing binarization techniques as special cases; (2) derive a principled way to synthesize forward-backward quantizers with automatic theoretical guarantees; (3) illustrate our theory by proposing an enhanced binarization algorithm BNN++; (4) conduct image classification experiments on CNNs and vision transformers, and empirically verify that BNN++ generally achieves competitive results on binarizing these models.

Title: On Central Primitives for Quantum Cryptography with Classical Communication

• Authors: Kai-Min Chung, Eli Goldin, Matthew Gray
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.17715
• Pdf URL: https://arxiv.org/pdf/2402.17715
• Copy Paste: [[2402.17715]] On Central Primitives for Quantum Cryptography with Classical Communication(https://arxiv.org/abs/2402.17715)
• Keywords: security
• Abstract: Recent work has introduced the "Quantum-Computation Classical-Communication" (QCCC) (Chung et. al.) setting for cryptography. There has been some evidence that One Way Puzzles (OWPuzz) are the natural central cryptographic primitive for this setting (Khurana and Tomer). For a primitive to be considered central it should have several characteristics. It should be well behaved (which for this paper we will think of as having amplification, combiners, and universal constructions); it should be implied by a wide variety of other primitives; and it should be equivalent to some class of useful primitives. We present combiners, correctness and security amplification, and a universal construction for OWPuzz. Our proof of security amplification uses a new and cleaner version construction of EFI from OWPuzz (in comparison to the result of Khurana and Tomer) that generalizes to weak OWPuzz and is the most technically involved section of the paper. It was previously known that OWPuzz are implied by other primitives of interest including commitments, symmetric key encryption, one way state generators (OWSG), and therefore pseudorandom states (PRS). However we are able to rule out OWPuzz's equivalence to many of these primitives by showing a black box separation between general OWPuzz and a restricted class of OWPuzz (those with efficient verification, which we call EV-OWPuzz). We then show that EV-OWPuzz are also implied by most of these primitives, which separates them from OWPuzz as well. This separation also separates extending PRS from highly compressing PRS answering an open question of Ananth et. al.

Title: AmbigNLG: Addressing Task Ambiguity in Instruction for NLG

• Authors: Ayana Niwa, Hayate Iso
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17717
• Pdf URL: https://arxiv.org/pdf/2402.17717
• Copy Paste: [[2402.17717]] AmbigNLG: Addressing Task Ambiguity in Instruction for NLG(https://arxiv.org/abs/2402.17717)
• Keywords: large language model
• Abstract: In this study, we introduce AmbigNLG, a new task designed to tackle the challenge of task ambiguity in instructions for Natural Language Generation (NLG) tasks. Despite the impressive capabilities of Large Language Models (LLMs) in understanding and executing a wide range of tasks through natural language interaction, their performance is significantly hindered by the ambiguity present in real-world instructions. To address this, AmbigNLG seeks to identify and mitigate such ambiguities, aiming to refine instructions to match user expectations better. We introduce a dataset, AmbigSNI-NLG, consisting of 2,500 instances, and develop an ambiguity taxonomy for categorizing and annotating instruction ambiguities. Our approach demonstrates substantial improvements in text generation quality, highlighting the critical role of clear and specific instructions in enhancing LLM performance in NLG tasks.

Title: Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners

• Authors: Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, Qifeng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17723
• Pdf URL: https://arxiv.org/pdf/2402.17723
• Copy Paste: [[2402.17723]] Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners(https://arxiv.org/abs/2402.17723)
• Keywords: diffusion
• Abstract: Video and audio content creation serves as the core technique for the movie industry and professional users. Recently, existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation. We observe the powerful generation ability of off-the-shelf video or audio generation models. Thus, instead of training the giant models from scratch, we propose to bridge the existing strong models with a shared latent representation space. Specifically, we propose a multimodality latent aligner with the pre-trained ImageBind model. Our latent aligner shares a similar core as the classifier guidance that guides the diffusion denoising process during inference time. Through carefully designed optimization strategy and loss functions, we show the superior performance of our method on joint video-audio generation, visual-steered audio generation, and audio-steered visual generation tasks. The project website can be found at https://yzxing87.github.io/Seeing-and-Hearing/

Title: VRP-SAM: SAM with Visual Reference Prompt

• Authors: Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, Gang Zhang, Errui Ding, Jingdong Wang, Zechao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17726
• Pdf URL: https://arxiv.org/pdf/2402.17726
• Copy Paste: [[2402.17726]] VRP-SAM: SAM with Visual Reference Prompt(https://arxiv.org/abs/2402.17726)
• Keywords: segmentation
• Abstract: In this paper, we propose a novel Visual Reference Prompt (VRP) encoder that empowers the Segment Anything Model (SAM) to utilize annotated reference images as prompts for segmentation, creating the VRP-SAM model. In essence, VRP-SAM can utilize annotated reference images to comprehend specific objects and perform segmentation of specific objects in target image. It is note that the VRP encoder can support a variety of annotation formats for reference images, including \textbf{point}, \textbf{box}, \textbf{scribble}, and \textbf{mask}. VRP-SAM achieves a breakthrough within the SAM framework by extending its versatility and applicability while preserving SAM's inherent strengths, thus enhancing user-friendliness. To enhance the generalization ability of VRP-SAM, the VRP encoder adopts a meta-learning strategy. To validate the effectiveness of VRP-SAM, we conducted extensive empirical studies on the Pascal and COCO datasets. Remarkably, VRP-SAM achieved state-of-the-art performance in visual reference segmentation with minimal learnable parameters. Furthermore, VRP-SAM demonstrates strong generalization capabilities, allowing it to perform segmentation of unseen objects and enabling cross-domain segmentation.

Title: Towards Fairness-Aware Adversarial Learning

• Authors: Yanghao Zhang, Tianle Zhang, Ronghui Mu, Xiaowei Huang, Wenjie Ruan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17729
• Pdf URL: https://arxiv.org/pdf/2402.17729
• Copy Paste: [[2402.17729]] Towards Fairness-Aware Adversarial Learning(https://arxiv.org/abs/2402.17729)
• Keywords: robust, fair
• Abstract: Although adversarial training (AT) has proven effective in enhancing the model's robustness, the recently revealed issue of fairness in robustness has not been well addressed, i.e. the robust accuracy varies significantly among different categories. In this paper, instead of uniformly evaluating the model's average class performance, we delve into the issue of robust fairness, by considering the worst-case distribution across various classes. We propose a novel learning paradigm, named Fairness-Aware Adversarial Learning (FAAL). As a generalization of conventional AT, we re-define the problem of adversarial training as a min-max-max framework, to ensure both robustness and fairness of the trained model. Specifically, by taking advantage of distributional robust optimization, our method aims to find the worst distribution among different categories, and the solution is guaranteed to obtain the upper bound performance with high probability. In particular, FAAL can fine-tune an unfair robust model to be fair within only two epochs, without compromising the overall clean and robust accuracies. Extensive experiments on various image datasets validate the superior performance and efficiency of the proposed FAAL compared to other state-of-the-art methods.

Title: Tower: An Open Multilingual Large Language Model for Translation-Related Tasks

• Authors: Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G.C. de Souza, André F.T. Martins
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17733
• Pdf URL: https://arxiv.org/pdf/2402.17733
• Copy Paste: [[2402.17733]] Tower: An Open Multilingual Large Language Model for Translation-Related Tasks(https://arxiv.org/abs/2402.17733)
• Keywords: large language model
• Abstract: While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only when specializing on a single task. In this paper, we propose a recipe for tailoring LLMs to multiple tasks present in translation workflows. We perform continued pretraining on a multilingual mixture of monolingual and parallel data, creating TowerBase, followed by finetuning on instructions relevant for translation processes, creating TowerInstruct. Our final model surpasses open alternatives on several tasks relevant to translation workflows and is competitive with general-purpose closed LLMs. To facilitate future research, we release the Tower models, our specialization dataset, an evaluation framework for LLMs focusing on the translation ecosystem, and a collection of model generations, including ours, on our benchmark.

Title: Evaluating Very Long-Term Conversational Memory of LLM Agents

• Authors: Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17753
• Pdf URL: https://arxiv.org/pdf/2402.17753
• Copy Paste: [[2402.17753]] Evaluating Very Long-Term Conversational Memory of LLM Agents(https://arxiv.org/abs/2402.17753)
• Keywords: large language model
• Abstract: Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in long-context large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to generate high-quality, very long-term dialogues by leveraging LLM-based agent architectures and grounding their dialogues on personas and temporal event graphs. Moreover, we equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs. Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 300 turns and 9K tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation tasks. Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues. Employing strategies like long-context LLMs or RAG can offer improvements but these models still substantially lag behind human performance.

Title: Robustly Learning Single-Index Models via Alignment Sharpness

• Authors: Nikos Zarifis, Puqian Wang, Ilias Diakonikolas, Jelena Diakonikolas
• Subjects: cs.LG, cs.DS, math.OC, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.17756
• Pdf URL: https://arxiv.org/pdf/2402.17756
• Copy Paste: [[2402.17756]] Robustly Learning Single-Index Models via Alignment Sharpness(https://arxiv.org/abs/2402.17756)
• Keywords: robust
• Abstract: We study the problem of learning Single-Index Models under the $L_2^2$ loss in the agnostic model. We give an efficient learning algorithm, achieving a constant factor approximation to the optimal loss, that succeeds under a range of distributions (including log-concave distributions) and a broad class of monotone and Lipschitz link functions. This is the first efficient constant factor approximate agnostic learner, even for Gaussian data and for any nontrivial class of link functions. Prior work for the case of unknown link function either works in the realizable setting or does not attain constant factor approximation. The main technical ingredient enabling our algorithm and analysis is a novel notion of a local error bound in optimization that we term alignment sharpness and that may be of broader interest.

Title: ADL4D: Towards A Contextually Rich Dataset for 4D Activities of Daily Living

• Authors: Marsil Zakour, Partha Pratim Nath, Ludwig Lohmer, Emre Faik Gökçe, Martin Piccolrovazzi, Constantin Patsch, Yuankai Wu, Rahul Chaudhari, Eckehard Steinbach
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17758
• Pdf URL: https://arxiv.org/pdf/2402.17758
• Copy Paste: [[2402.17758]] ADL4D: Towards A Contextually Rich Dataset for 4D Activities of Daily Living(https://arxiv.org/abs/2402.17758)
• Keywords: segmentation
• Abstract: Hand-Object Interactions (HOIs) are conditioned on spatial and temporal contexts like surrounding objects, pre- vious actions, and future intents (for example, grasping and handover actions vary greatly based on objects proximity and trajectory obstruction). However, existing datasets for 4D HOI (3D HOI over time) are limited to one subject inter- acting with one object only. This restricts the generalization of learning-based HOI methods trained on those datasets. We introduce ADL4D, a dataset of up to two subjects inter- acting with different sets of objects performing Activities of Daily Living (ADL) like breakfast or lunch preparation ac- tivities. The transition between multiple objects to complete a certain task over time introduces a unique context lacking in existing datasets. Our dataset consists of 75 sequences with a total of 1.1M RGB-D frames, hand and object poses, and per-hand fine-grained action annotations. We develop an automatic system for multi-view multi-hand 3D pose an- notation capable of tracking hand poses over time. We inte- grate and test it against publicly available datasets. Finally, we evaluate our dataset on the tasks of Hand Mesh Recov- ery (HMR) and Hand Action Segmentation (HAS).

Title: Massive Activations in Large Language Models

• Authors: Mingjie Sun, Xinlei Chen, J. Zico Kolter, Zhuang Liu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17762
• Pdf URL: https://arxiv.org/pdf/2402.17762
• Copy Paste: [[2402.17762]] Massive Activations in Large Language Models(https://arxiv.org/abs/2402.17762)
• Keywords: transformer, large language model
• Abstract: We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers.

Title: The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

• Authors: Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17764
• Pdf URL: https://arxiv.org/pdf/2402.17764
• Copy Paste: [[2402.17764]] The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits(https://arxiv.org/abs/2402.17764)
• Keywords: transformer, large language model
• Abstract: Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption. More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for training new generations of LLMs that are both high-performance and cost-effective. Furthermore, it enables a new computation paradigm and opens the door for designing specific hardware optimized for 1-bit LLMs.

Title: ShapeLLM: Universal 3D Object Understanding for Embodied Interaction

• Authors: Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, Kaisheng Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17766
• Pdf URL: https://arxiv.org/pdf/2402.17766
• Copy Paste: [[2402.17766]] ShapeLLM: Universal 3D Object Understanding for Embodied Interaction(https://arxiv.org/abs/2402.17766)
• Keywords: large language model
• Abstract: This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages. ShapeLLM is built upon an improved 3D encoder by extending ReCon to ReCon++ that benefits from multi-view image distillation for enhanced geometry understanding. By utilizing ReCon++ as the 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructed instruction-following data and tested on our newly human-curated evaluation benchmark, 3D MM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3D geometry understanding and language-unified 3D interaction tasks, such as embodied visual grounding.