2024-02-14

Title: QACP: An Annotated Question Answering Dataset for Assisting Chinese Python Programming Learners

• Authors: Rui Xiao, Lu Han, Xiaoying Zhou, Jiong Wang, Na Zong, Pengyu Zhang
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.07913
• Pdf URL: https://arxiv.org/pdf/2402.07913
• Copy Paste: [[2402.07913]] QACP: An Annotated Question Answering Dataset for Assisting Chinese Python Programming Learners(https://arxiv.org/abs/2402.07913)
• Keywords: large language model
• Abstract: In online learning platforms, particularly in rapidly growing computer programming courses, addressing the thousands of students' learning queries requires considerable human cost. The creation of intelligent assistant large language models (LLMs) tailored for programming education necessitates distinct data support. However, in real application scenarios, the data resources for training such LLMs are relatively scarce. Therefore, to address the data scarcity in intelligent educational systems for programming, this paper proposes a new Chinese question-and-answer dataset for Python learners. To ensure the authenticity and reliability of the sources of the questions, we collected questions from actual student questions and categorized them according to various dimensions such as the type of questions and the type of learners. This annotation principle is designed to enhance the effectiveness and quality of online programming education, providing a solid data foundation for developing the programming teaching assists (TA). Furthermore, we conducted comprehensive evaluations of various LLMs proficient in processing and generating Chinese content, highlighting the potential limitations of general LLMs as intelligent teaching assistants in computer programming courses.

Title: Sentinels of the Stream: Unleashing Large Language Models for Dynamic Packet Classification in Software Defined Networks -- Position Paper

• Authors: Shariq Murtuza
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.07950
• Pdf URL: https://arxiv.org/pdf/2402.07950
• Copy Paste: [[2402.07950]] Sentinels of the Stream: Unleashing Large Language Models for Dynamic Packet Classification in Software Defined Networks -- Position Paper(https://arxiv.org/abs/2402.07950)
• Keywords: security, large language model
• Abstract: With the release of OpenAI's ChatGPT, the field of large language models (LLM) saw an increase of academic interest in GPT based chat assistants. In the next few months multiple accesible large language models were released that included Meta's LLama models and Mistral AI's Mistral and Mixtral MoE models. These models are available openly for a wide array of purposes with a wide spectrum of licenses. These LLMs have found their use in a different number of fields like code development, SQL generation etc. In this work we propose our plan to explore the applicability of large language model in the domain of network security. We plan to create Sentinel, a LLM, to analyse network packet contents and pass a judgment on it's threat level. This work is a preliminary report that will lay our plan for our future endeavors.

Title: Integrating MLSecOps in the Biotechnology Industry 5.0

• Authors: Naseela Pervez, Alexander J. Titus
• Subjects: cs.CR, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2402.07967
• Pdf URL: https://arxiv.org/pdf/2402.07967
• Copy Paste: [[2402.07967]] Integrating MLSecOps in the Biotechnology Industry 5.0(https://arxiv.org/abs/2402.07967)
• Keywords: secure, security, attack
• Abstract: Biotechnology Industry 5.0 is advancing with the integration of cutting-edge technologies like Machine Learning (ML), the Internet Of Things (IoT), and cloud computing. It is no surprise that an industry that utilizes data from customers and can alter their lives is a target of a variety of attacks. This chapter provides a perspective of how Machine Learning Security Operations (MLSecOps) can help secure the biotechnology Industry 5.0. The chapter provides an analysis of the threats in the biotechnology Industry 5.0 and how ML algorithms can help secure with industry best practices. This chapter explores the scope of MLSecOps in the biotechnology Industry 5.0, highlighting how crucial it is to comply with current regulatory frameworks. With biotechnology Industry 5.0 developing innovative solutions in healthcare, supply chain management, biomanufacturing, pharmaceuticals sectors, and more, the chapter also discusses the MLSecOps best practices that industry and enterprises should follow while also considering ethical responsibilities. Overall, the chapter provides a discussion of how to integrate MLSecOps into the design, deployment, and regulation of the processes in biotechnology Industry 5.0.

Title: NetInfoF Framework: Measuring and Exploiting Network Usable Information

• Authors: Meng-Chieh Lee, Haiyang Yu, Jian Zhang, Vassilis N. Ioannidis, Xiang Song, Soji Adeshina, Da Zheng, Christos Faloutsos
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.07999
• Pdf URL: https://arxiv.org/pdf/2402.07999
• Copy Paste: [[2402.07999]] NetInfoF Framework: Measuring and Exploiting Network Usable Information(https://arxiv.org/abs/2402.07999)
• Keywords: robust
• Abstract: Given a node-attributed graph, and a graph task (link prediction or node classification), can we tell if a graph neural network (GNN) will perform well? More specifically, do the graph structure and the node features carry enough usable information for the task? Our goals are (1) to develop a fast tool to measure how much information is in the graph structure and in the node features, and (2) to exploit the information to solve the task, if there is enough. We propose NetInfoF, a framework including NetInfoF_Probe and NetInfoF_Act, for the measurement and the exploitation of network usable information (NUI), respectively. Given a graph data, NetInfoF_Probe measures NUI without any model training, and NetInfoF_Act solves link prediction and node classification, while two modules share the same backbone. In summary, NetInfoF has following notable advantages: (a) General, handling both link prediction and node classification; (b) Principled, with theoretical guarantee and closed-form solution; (c) Effective, thanks to the proposed adjustment to node similarity; (d) Scalable, scaling linearly with the input size. In our carefully designed synthetic datasets, NetInfoF correctly identifies the ground truth of NUI and is the only method being robust to all graph scenarios. Applied on real-world datasets, NetInfoF wins in 11 out of 12 times on link prediction compared to general GNN baselines.

Title: Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs

• Authors: Víctor Gallego
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08005
• Pdf URL: https://arxiv.org/pdf/2402.08005
• Copy Paste: [[2402.08005]] Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs(https://arxiv.org/abs/2402.08005)
• Keywords: robust, large language model
• Abstract: In this paper, we introduce \emph{refined Direct Preference Optimization} (rDPO), a method for improving the behavioral alignment of Large Language Models (LLMs) without the need for human-annotated data. The method involves creating synthetic data using self-critique prompting by a teacher LLM and then utilising a generalized DPO loss function to distil to a student LLM. The loss function incorporates an additional external reward model to improve the quality of synthetic data, making rDPO robust to potential noise in the synthetic dataset. rDPO is shown to be effective in a diverse set of behavioural alignment tasks, such as improved safety, robustness against role-playing, and reduced sycophancy. Code to be released at https://github.com/vicgalle/refined-dpo.

Title: Enhancing Amharic-LLaMA: Integrating Task Specific and Generative Datasets

• Authors: Israel Abebe Azime, Mitiku Yohannes Fuge, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walelign Tewabe Sewunetie, Seid Muhie Yimam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08015
• Pdf URL: https://arxiv.org/pdf/2402.08015
• Copy Paste: [[2402.08015]] Enhancing Amharic-LLaMA: Integrating Task Specific and Generative Datasets(https://arxiv.org/abs/2402.08015)
• Keywords: generative, large language model
• Abstract: Large language models (LLMs) have received a lot of attention in natural language processing (NLP) research because of their exceptional performance in understanding and generating human languages. However, low-resource languages are left behind due to the unavailability of resources. In this work, we focus on enhancing the LLaMA-2-Amharic model by integrating task-specific and generative datasets to improve language model performance for Amharic. We compile an Amharic instruction fine-tuning dataset and fine-tuned LLaMA-2-Amharic model. The fine-tuned model shows promising results in different NLP tasks. We open-source our dataset creation pipeline, instruction datasets, trained models, and evaluation outputs to promote language-specific studies on these models.

Title: Lumos : Empowering Multimodal LLMs with Scene Text Recognition

• Authors: Ashish Shenoy, Yichao Lu, Srihari Jayakumar, Debojeet Chatterjee, Mohsen Moslehpour, Pierce Chuang, Abhay Harpale, Vikas Bhardwaj, Di Xu, Shicong Zhao, Longfang Zhao, Ankit Ramchandani, Xin Luna Dong, Anuj Kumar
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08017
• Pdf URL: https://arxiv.org/pdf/2402.08017
• Copy Paste: [[2402.08017]] Lumos : Empowering Multimodal LLMs with Scene Text Recognition(https://arxiv.org/abs/2402.08017)
• Keywords: large language model
• Abstract: We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.

Title: Nearest Neighbour Score Estimators for Diffusion Generative Models

• Authors: Matthew Niedoba, Dylan Green, Saeid Naderiparizi, Vasileios Lioutas, Jonathan Wilder Lavington, Xiaoxuan Liang, Yunpeng Liu, Ke Zhang, Setareh Dabiri, Adam Ścibior, Berend Zwartsenberg, Frank Wood
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.08018
• Pdf URL: https://arxiv.org/pdf/2402.08018
• Copy Paste: [[2402.08018]] Nearest Neighbour Score Estimators for Diffusion Generative Models(https://arxiv.org/abs/2402.08018)
• Keywords: diffusion, generative
• Abstract: Score function estimation is the cornerstone of both training and sampling from diffusion generative models. Despite this fact, the most commonly used estimators are either biased neural network approximations or high variance Monte Carlo estimators based on the conditional score. We introduce a novel nearest neighbour score function estimator which utilizes multiple samples from the training set to dramatically decrease estimator variance. We leverage our low variance estimator in two compelling applications. Training consistency models with our estimator, we report a significant increase in both convergence speed and sample quality. In diffusion models, we show that our estimator can replace a learned network for probability-flow ODE integration, opening promising new avenues of future research.

Title: UGMAE: A Unified Framework for Graph Masked Autoencoders

• Authors: Yijun Tian, Chuxu Zhang, Ziyi Kou, Zheyuan Liu, Xiangliang Zhang, Nitesh V. Chawla
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08023
• Pdf URL: https://arxiv.org/pdf/2402.08023
• Copy Paste: [[2402.08023]] UGMAE: A Unified Framework for Graph Masked Autoencoders(https://arxiv.org/abs/2402.08023)
• Keywords: generative
• Abstract: Generative self-supervised learning on graphs, particularly graph masked autoencoders, has emerged as a popular learning paradigm and demonstrated its efficacy in handling non-Euclidean data. However, several remaining issues limit the capability of existing methods: 1) the disregard of uneven node significance in masking, 2) the underutilization of holistic graph information, 3) the ignorance of semantic knowledge in the representation space due to the exclusive use of reconstruction loss in the output space, and 4) the unstable reconstructions caused by the large volume of masked contents. In light of this, we propose UGMAE, a unified framework for graph masked autoencoders to address these issues from the perspectives of adaptivity, integrity, complementarity, and consistency. Specifically, we first develop an adaptive feature mask generator to account for the unique significance of nodes and sample informative masks (adaptivity). We then design a ranking-based structure reconstruction objective joint with feature reconstruction to capture holistic graph information and emphasize the topological proximity between neighbors (integrity). After that, we present a bootstrapping-based similarity module to encode the high-level semantic knowledge in the representation space, complementary to the low-level reconstruction in the output space (complementarity). Finally, we build a consistency assurance module to provide reconstruction objectives with extra stabilized consistency targets (consistency). Extensive experiments demonstrate that UGMAE outperforms both contrastive and generative state-of-the-art baselines on several tasks across multiple datasets.

Title: Beyond the Mud: Datasets and Benchmarks for Computer Vision in Off-Road Racing

• Authors: Jacob Tyo, Motolani Olarinre, Youngseog Chung, Zachary C. Lipton
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08025
• Pdf URL: https://arxiv.org/pdf/2402.08025
• Copy Paste: [[2402.08025]] Beyond the Mud: Datasets and Benchmarks for Computer Vision in Off-Road Racing(https://arxiv.org/abs/2402.08025)
• Keywords: robust
• Abstract: Despite significant progress in optical character recognition (OCR) and computer vision systems, robustly recognizing text and identifying people in images taken in unconstrained \emph{in-the-wild} environments remain an ongoing challenge. However, such obstacles must be overcome in practical applications of vision systems, such as identifying racers in photos taken during off-road racing events. To this end, we introduce two new challenging real-world datasets - the off-road motorcycle Racer Number Dataset (RND) and the Muddy Racer re-iDentification Dataset (MUDD) - to highlight the shortcomings of current methods and drive advances in OCR and person re-identification (ReID) under extreme conditions. These two datasets feature over 6,300 images taken during off-road competitions which exhibit a variety of factors that undermine even modern vision systems, namely mud, complex poses, and motion blur. We establish benchmark performance on both datasets using state-of-the-art models. Off-the-shelf models transfer poorly, reaching only 15% end-to-end (E2E) F1 score on text spotting, and 33% rank-1 accuracy on ReID. Fine-tuning yields major improvements, bringing model performance to 53% F1 score for E2E text spotting and 79% rank-1 accuracy on ReID, but still falls short of good performance. Our analysis exposes open problems in real-world OCR and ReID that necessitate domain-targeted techniques. With these datasets and analysis of model limitations, we aim to foster innovations in handling real-world conditions like mud and complex poses to drive progress in robust computer vision. All data was sourced from PerformancePhoto.co, a website used by professional motorsports photographers, racers, and fans. The top-performing text spotting and ReID models are deployed on this platform to power real-time race photo search.

Title: Dumviri: Detecting Trackers and Mixed Trackers with a Breakage Detector

• Authors: He Shuang, Lianying Zhao, David Lie
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08031
• Pdf URL: https://arxiv.org/pdf/2402.08031
• Copy Paste: [[2402.08031]] Dumviri: Detecting Trackers and Mixed Trackers with a Breakage Detector(https://arxiv.org/abs/2402.08031)
• Keywords: privacy
• Abstract: Previous automatic tracker detection work lacks features to recognize web page breakage and often resort to manual analysis to assess the breakage caused by blocking trackers. We introduce Dumviri, which incorporates a breakage detector that can automatically detect web page breakage caused by erroneously blocking a resource that is needed by the page to function properly. This addition allows Dumviri to prevent functional resources from being misclassified as trackers and increases overall detection accuracy. We designed Dumviri to take differential features. We further find that these features are agnostic to analysis granularity and enable Dumviri to predict tracking resources at the request field granularity, allowing Dumviri to handle some mixed trackers. Evaluating Dumviri on 15K pages shows its ability to replicate the labels of human-generated filter lists with an accuracy of 97.44%. Through a manual analysis, we found that Dumviri identified previously unreported trackers and its breakage detector can identify rules that cause web page breakage in commonly used filter lists like EasyPrivacy. In the case of mixed trackers, Dumviri, being the first automated mixed tracker detector, achieves a 79.09% accuracy. We have confirmed 22 previously unreported unique trackers and 26 unique mixed trackers. We promptly reported these findings to privacy developers, and we will publish our filter lists in uBlock Origin's extended syntax.

Title: Multiple Random Masking Autoencoder Ensembles for Robust Multimodal Semi-supervised Learning

• Authors: Alexandru-Raul Todoran, Marius Leordeanu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08035
• Pdf URL: https://arxiv.org/pdf/2402.08035
• Copy Paste: [[2402.08035]] Multiple Random Masking Autoencoder Ensembles for Robust Multimodal Semi-supervised Learning(https://arxiv.org/abs/2402.08035)
• Keywords: robust
• Abstract: There is an increasing number of real-world problems in computer vision and machine learning requiring to take into consideration multiple interpretation layers (modalities or views) of the world and learn how they relate to each other. For example, in the case of Earth Observations from satellite data, it is important to be able to predict one observation layer (e.g. vegetation index) from other layers (e.g. water vapor, snow cover, temperature etc), in order to best understand how the Earth System functions and also be able to reliably predict information for one layer when the data is missing (e.g. due to measurement failure or error).

Title: Multi-Attribute Vision Transformers are Efficient and Robust Learners

• Authors: Hanan Gani, Nada Saadi, Noor Hussein, Karthik Nandakumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08070
• Pdf URL: https://arxiv.org/pdf/2402.08070
• Copy Paste: [[2402.08070]] Multi-Attribute Vision Transformers are Efficient and Robust Learners(https://arxiv.org/abs/2402.08070)
• Keywords: attack, robust, transformer
• Abstract: Since their inception, Vision Transformers (ViTs) have emerged as a compelling alternative to Convolutional Neural Networks (CNNs) across a wide spectrum of tasks. ViTs exhibit notable characteristics, including global attention, resilience against occlusions, and adaptability to distribution shifts. One underexplored aspect of ViTs is their potential for multi-attribute learning, referring to their ability to simultaneously grasp multiple attribute-related tasks. In this paper, we delve into the multi-attribute learning capability of ViTs, presenting a straightforward yet effective strategy for training various attributes through a single ViT network as distinct tasks. We assess the resilience of multi-attribute ViTs against adversarial attacks and compare their performance against ViTs designed for single attributes. Moreover, we further evaluate the robustness of multi-attribute ViTs against a recent transformer based attack called Patch-Fool. Our empirical findings on the CelebA dataset provide validation for our assertion.

Title: Grounding Data Science Code Generation with Input-Output Specifications

• Authors: Yeming Wen, Pengcheng Yin, Kensen Shi, Henryk Michalewski, Swarat Chaudhuri, Alex Polozov
• Subjects: cs.LG, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.08073
• Pdf URL: https://arxiv.org/pdf/2402.08073
• Copy Paste: [[2402.08073]] Grounding Data Science Code Generation with Input-Output Specifications(https://arxiv.org/abs/2402.08073)
• Keywords: large language model
• Abstract: Large language models (LLMs) have recently demonstrated a remarkable ability to generate code from natural language (NL) prompts. However, in the real world, NL is often too ambiguous to capture the true intent behind programming problems, requiring additional input-output (I/O) specifications. Unfortunately, LLMs can have difficulty aligning their outputs with both the NL prompt and the I/O specification. In this paper, we give a way to mitigate this issue in the context of data science programming, where tasks require explicit I/O specifications for clarity. Specifically, we propose GIFT4Code, a novel approach for the instruction fine-tuning of LLMs with respect to I/O specifications. Our method leverages synthetic data produced by the LLM itself and utilizes execution-derived feedback as a key learning signal. This feedback, in the form of program I/O specifications, is provided to the LLM to facilitate instruction fine-tuning. We evaluated our approach on two challenging data science benchmarks, Arcade and DS-1000. The results demonstrate a significant improvement in the LLM's ability to generate code that is not only executable but also accurately aligned with user specifications, substantially improving the quality of code generation for complex data science tasks.

Title: Large Language Models as Agents in Two-Player Games

• Authors: Yang Liu, Peng Sun, Hang Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08078
• Pdf URL: https://arxiv.org/pdf/2402.08078
• Copy Paste: [[2402.08078]] Large Language Models as Agents in Two-Player Games(https://arxiv.org/abs/2402.08078)
• Keywords: large language model
• Abstract: By formally defining the training processes of large language models (LLMs), which usually encompasses pre-training, supervised fine-tuning, and reinforcement learning with human feedback, within a single and unified machine learning paradigm, we can glean pivotal insights for advancing LLM technologies. This position paper delineates the parallels between the training methods of LLMs and the strategies employed for the development of agents in two-player games, as studied in game theory, reinforcement learning, and multi-agent systems. We propose a re-conceptualization of LLM learning processes in terms of agent learning in language-based games. This framework unveils innovative perspectives on the successes and challenges in LLM development, offering a fresh understanding of addressing alignment issues among other strategic considerations. Furthermore, our two-player game approach sheds light on novel data preparation and machine learning techniques for training LLMs.

Title: CycPUF: Cyclic Physical Unclonable Function

• Authors: Michael Dominguez, Amin Rezaei
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08084
• Pdf URL: https://arxiv.org/pdf/2402.08084
• Copy Paste: [[2402.08084]] CycPUF: Cyclic Physical Unclonable Function(https://arxiv.org/abs/2402.08084)
• Keywords: security, attack
• Abstract: Physical Unclonable Functions (PUFs) leverage manufacturing process imperfections that cause propagation delay discrepancies for the signals traveling along these paths. While PUFs can be used for device authentication and chip-specific key generation, strong PUFs have been shown to be vulnerable to machine learning modeling attacks. Although there is an impression that combinational circuits must be designed without any loops, cyclic combinational circuits have been shown to increase design security against hardware intellectual property theft. In this paper, we introduce feedback signals into traditional delay-based PUF designs such as arbiter PUF, ring oscillator PUF, and butterfly PUF to give them a wider range of possible output behaviors and thus an edge against modeling attacks. Based on our analysis, cyclic PUFs produce responses that can be binary, steady-state, oscillating, or pseudo-random under fixed challenges. The proposed cyclic PUFs are implemented in field programmable gate arrays, and their power and area overhead, in addition to functional metrics, are reported compared with their traditional counterparts. The security gain of the proposed cyclic PUFs is also shown against state-of-the-art attacks.

Title: Message Detouring: A Simple Yet Effective Cycle Representation for Expressive Graph Learning

• Authors: Ziquan Wei, Tingting Dan, Guorong Wu
• Subjects: cs.LG, cs.AI, cs.CG
• Abstract URL: https://arxiv.org/abs/2402.08085
• Pdf URL: https://arxiv.org/pdf/2402.08085
• Copy Paste: [[2402.08085]] Message Detouring: A Simple Yet Effective Cycle Representation for Expressive Graph Learning(https://arxiv.org/abs/2402.08085)
• Keywords: transformer
• Abstract: Graph learning is crucial in the fields of bioinformatics, social networks, and chemicals. Although high-order graphlets, such as cycles, are critical to achieving an informative graph representation for node classification, edge prediction, and graph recognition, modeling high-order topological characteristics poses significant computational challenges, restricting its widespread applications in machine learning. To address this limitation, we introduce the concept of \textit{message detouring} to hierarchically characterize cycle representation throughout the entire graph, which capitalizes on the contrast between the shortest and longest pathways within a range of local topologies associated with each graph node. The topological feature representations derived from our message detouring landscape demonstrate comparable expressive power to high-order \textit{Weisfeiler-Lehman} (WL) tests but much less computational demands. In addition to the integration with graph kernel and message passing neural networks, we present a novel message detouring neural network, which uses Transformer backbone to integrate cycle representations across nodes and edges. Aside from theoretical results, experimental results on expressiveness, graph classification, and node classification show message detouring can significantly outperform current counterpart approaches on various benchmark datasets.

Title: Text-centric Alignment for Multi-Modality Learning

• Authors: Yun-Da Tsai, Ting-Yu Yen, Pei-Fu Guo, Zhe-Yan Li, Shou-De Lin
• Subjects: cs.LG, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08086
• Pdf URL: https://arxiv.org/pdf/2402.08086
• Copy Paste: [[2402.08086]] Text-centric Alignment for Multi-Modality Learning(https://arxiv.org/abs/2402.08086)
• Keywords: robust, large language model
• Abstract: This research paper addresses the challenge of modality mismatch in multimodal learning, where the modalities available during inference differ from those available at training. We propose the Text-centric Alignment for Multi-Modality Learning (TAMML) approach, an innovative method that utilizes Large Language Models (LLMs) with in-context learning and foundation models to enhance the generalizability of multimodal systems under these conditions. By leveraging the unique properties of text as a unified semantic space, TAMML demonstrates significant improvements in handling unseen, diverse, and unpredictable modality combinations. TAMML not only adapts to varying modalities but also maintains robust performance, showcasing the potential of foundation models in overcoming the limitations of traditional fixed-modality frameworks in embedding representations. This study contributes to the field by offering a flexible, effective solution for real-world applications where modality availability is dynamic and uncertain.

Title: Learning Neural Contracting Dynamics: Extended Linearization and Global Guarantees

• Authors: Sean Jaffe, Alexander Davydov, Deniz Lapsekili, Ambuj singh, Francesco Bullo
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2402.08090
• Pdf URL: https://arxiv.org/pdf/2402.08090
• Copy Paste: [[2402.08090]] Learning Neural Contracting Dynamics: Extended Linearization and Global Guarantees(https://arxiv.org/abs/2402.08090)
• Keywords: robust
• Abstract: Global stability and robustness guarantees in learned dynamical systems are essential to ensure well-behavedness of the systems in the face of uncertainty. We present Extended Linearized Contracting Dynamics (ELCD), the first neural network-based dynamical system with global contractivity guarantees in arbitrary metrics. The key feature of ELCD is a parametrization of the extended linearization of the nonlinear vector field. In its most basic form, ELCD is guaranteed to be (i) globally exponentially stable, (ii) equilibrium contracting, and (iii) globally contracting with respect to some metric. To allow for contraction with respect to more general metrics in the data space, we train diffeomorphisms between the data space and a latent space and enforce contractivity in the latent space, which ensures global contractivity in the data space. We demonstrate the performance of ELCD on the $2$D, $4$D, and $8$D LASA datasets.

Title: BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data

• Authors: Mateusz Łajszczak, Guillermo Cámbara, Yang Li, Fatih Beyhan, Arent van Korlaar, Fan Yang, Arnaud Joly, Álvaro Martín-Cortinas, Ammar Abbas, Adam Michalski, Alexis Moinet, Sri Karlapati, Ewa Muszyńska, Haohan Guo, Bartosz Putrycz, Soledad López Gambino, Kayeon Yoo, Elena Sokolova, Thomas Drugman
• Subjects: cs.LG, cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.08093
• Pdf URL: https://arxiv.org/pdf/2402.08093
• Copy Paste: [[2402.08093]] BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data(https://arxiv.org/abs/2402.08093)
• Keywords: transformer, large language model
• Abstract: We introduce a text-to-speech (TTS) model called BASE TTS, which stands for $\textbf{B}$ig $\textbf{A}$daptive $\textbf{S}$treamable TTS with $\textbf{E}$mergent abilities. BASE TTS is the largest TTS model to-date, trained on 100K hours of public domain speech data, achieving a new state-of-the-art in speech naturalness. It deploys a 1-billion-parameter autoregressive Transformer that converts raw texts into discrete codes ("speechcodes") followed by a convolution-based decoder which converts these speechcodes into waveforms in an incremental, streamable manner. Further, our speechcodes are built using a novel speech tokenization technique that features speaker ID disentanglement and compression with byte-pair encoding. Echoing the widely-reported "emergent abilities" of large language models when trained on increasing volume of data, we show that BASE TTS variants built with 10K+ hours and 500M+ parameters begin to demonstrate natural prosody on textually complex sentences. We design and share a specialized dataset to measure these emergent abilities for text-to-speech. We showcase state-of-the-art naturalness of BASE TTS by evaluating against baselines that include publicly available large-scale text-to-speech systems: YourTTS, Bark and TortoiseTTS. Audio samples generated by the model can be heard at https://amazon-ltts-paper.com/.

Title: Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translation

• Authors: Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, Fabio Massimo Zanzotto
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08100
• Pdf URL: https://arxiv.org/pdf/2402.08100
• Copy Paste: [[2402.08100]] Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translation(https://arxiv.org/abs/2402.08100)
• Keywords: large language model
• Abstract: Understanding textual description to generate code seems to be an achieved capability of instruction-following Large Language Models (LLMs) in zero-shot scenario. However, there is a severe possibility that this translation ability may be influenced by having seen target textual descriptions and the related code. This effect is known as Data Contamination. In this study, we investigate the impact of Data Contamination on the performance of GPT-3.5 in the Text-to-SQL code-generating tasks. Hence, we introduce a novel method to detect Data Contamination in GPTs and examine GPT-3.5's Text-to-SQL performances using the known Spider Dataset and our new unfamiliar dataset Termite. Furthermore, we analyze GPT-3.5's efficacy on databases with modified information via an adversarial table disconnection (ATD) approach, complicating Text-to-SQL tasks by removing structural pieces of information from the database. Our results indicate a significant performance drop in GPT-3.5 on the unfamiliar Termite dataset, even with ATD modifications, highlighting the effect of Data Contamination on LLMs in Text-to-SQL translation tasks.

Title: Addressing cognitive bias in medical language models

• Authors: Samuel Schmidgall, Carl Harris, Ime Essien, Daniel Olshvang, Tawsifur Rahman, Ji Woong Kim, Rojin Ziaei, Jason Eshraghian, Peter Abadir, Rama Chellappa
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.08113
• Pdf URL: https://arxiv.org/pdf/2402.08113
• Copy Paste: [[2402.08113]] Addressing cognitive bias in medical language models(https://arxiv.org/abs/2402.08113)
• Keywords: large language model
• Abstract: The integration of large language models (LLMs) into the medical field has gained significant attention due to their promising accuracy in simulated clinical decision-making settings. However, clinical decision-making is more complex than simulations because physicians' decisions are shaped by many factors, including the presence of cognitive bias. However, the degree to which LLMs are susceptible to the same cognitive biases that affect human clinicians remains unexplored. Our hypothesis posits that when LLMs are confronted with clinical questions containing cognitive biases, they will yield significantly less accurate responses compared to the same questions presented without such biases.In this study, we developed BiasMedQA, a novel benchmark for evaluating cognitive biases in LLMs applied to medical tasks. Using BiasMedQA we evaluated six LLMs, namely GPT-4, Mixtral-8x70B, GPT-3.5, PaLM-2, Llama 2 70B-chat, and the medically specialized PMC Llama 13B. We tested these models on 1,273 questions from the US Medical Licensing Exam (USMLE) Steps 1, 2, and 3, modified to replicate common clinically-relevant cognitive biases. Our analysis revealed varying effects for biases on these LLMs, with GPT-4 standing out for its resilience to bias, in contrast to Llama 2 70B-chat and PMC Llama 13B, which were disproportionately affected by cognitive bias. Our findings highlight the critical need for bias mitigation in the development of medical LLMs, pointing towards safer and more reliable applications in healthcare.

Title: Active Preference Learning for Large Language Models

• Authors: William Muldrew, Peter Hayes, Mingtian Zhang, David Barber
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08114
• Pdf URL: https://arxiv.org/pdf/2402.08114
• Copy Paste: [[2402.08114]] Active Preference Learning for Large Language Models(https://arxiv.org/abs/2402.08114)
• Keywords: large language model
• Abstract: As large language models (LLMs) become more capable, fine-tuning techniques for aligning with human intent are increasingly important. A key consideration for aligning these models is how to most effectively use human resources, or model resources in the case where LLMs themselves are used as oracles. Reinforcement learning from Human or AI preferences (RLHF/RLAIF) is the most prominent example of such a technique, but is complex and often unstable. Direct Preference Optimization (DPO) has recently been proposed as a simpler and more stable alternative. In this work, we develop an active learning strategy for DPO to make better use of preference labels. We propose a practical acquisition function for prompt/completion pairs based on the predictive entropy of the language model and a measure of certainty of the implicit preference model optimized by DPO. We demonstrate how our approach improves both the rate of learning and final performance of fine-tuning on pairwise preference data.

Title: On the Resurgence of Recurrent Models for Long Sequences: Survey and Research Opportunities in the Transformer Era

• Authors: Matteo Tiezzi, Michele Casoni, Alessandro Betti, Tommaso Guidi, Marco Gori, Stefano Melacci
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08132
• Pdf URL: https://arxiv.org/pdf/2402.08132
• Copy Paste: [[2402.08132]] On the Resurgence of Recurrent Models for Long Sequences: Survey and Research Opportunities in the Transformer Era(https://arxiv.org/abs/2402.08132)
• Keywords: robust, transformer, large language model
• Abstract: A longstanding challenge for the Machine Learning community is the one of developing models that are capable of processing and learning from very long sequences of data. The outstanding results of Transformers-based networks (e.g., Large Language Models) promotes the idea of parallel attention as the key to succeed in such a challenge, obfuscating the role of classic sequential processing of Recurrent Models. However, in the last few years, researchers who were concerned by the quadratic complexity of self-attention have been proposing a novel wave of neural models, which gets the best from the two worlds, i.e., Transformers and Recurrent Nets. Meanwhile, Deep Space-State Models emerged as robust approaches to function approximation over time, thus opening a new perspective in learning from sequential data, followed by many people in the field and exploited to implement a special class of (linear) Recurrent Neural Networks. This survey is aimed at providing an overview of these trends framed under the unifying umbrella of Recurrence. Moreover, it emphasizes novel research opportunities that become prominent when abandoning the idea of processing long sequences whose length is known-in-advance for the more realistic setting of potentially infinite-length sequences, thus intersecting the field of lifelong-online learning from streamed data.

Title: CMA-R:Causal Mediation Analysis for Explaining Rumour Detection

• Authors: Lin Tian, Xiuzhen Zhang, Jey Han Lau
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08155
• Pdf URL: https://arxiv.org/pdf/2402.08155
• Copy Paste: [[2402.08155]] CMA-R:Causal Mediation Analysis for Explaining Rumour Detection(https://arxiv.org/abs/2402.08155)
• Keywords: interpretability
• Abstract: We apply causal mediation analysis to explain the decision-making process of neural models for rumour detection on Twitter. Interventions at the input and network level reveal the causal impacts of tweets and words in the model output. We find that our approach CMA-R -- Causal Mediation Analysis for Rumour detection -- identifies salient tweets that explain model predictions and show strong agreement with human judgements for critical tweets determining the truthfulness of stories. CMA-R can further highlight causally impactful words in the salient tweets, providing another layer of interpretability and transparency into these blackbox rumour detection systems. Code is available at: https://github.com/ltian678/cma-r.

Title: Group Decision-Making among Privacy-Aware Agents

• Authors: Marios Papachristou, M. Amin Rahimian
• Subjects: cs.LG, cs.AI, cs.CR, cs.MA, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.08156
• Pdf URL: https://arxiv.org/pdf/2402.08156
• Copy Paste: [[2402.08156]] Group Decision-Making among Privacy-Aware Agents(https://arxiv.org/abs/2402.08156)
• Keywords: security, privacy, protect
• Abstract: How can individuals exchange information to learn from each other despite their privacy needs and security concerns? For example, consider individuals deliberating a contentious topic and being concerned about divulging their private experiences. Preserving individual privacy and enabling efficient social learning are both important desiderata but seem fundamentally at odds with each other and very hard to reconcile. We do so by controlling information leakage using rigorous statistical guarantees that are based on differential privacy (DP). Our agents use log-linear rules to update their beliefs after communicating with their neighbors. Adding DP randomization noise to beliefs provides communicating agents with plausible deniability with regard to their private information and their network neighborhoods. We consider two learning environments one for distributed maximum-likelihood estimation given a finite number of private signals and another for online learning from an infinite, intermittent signal stream. Noisy information aggregation in the finite case leads to interesting tradeoffs between rejecting low-quality states and making sure all high-quality states are accepted in the algorithm output. Our results flesh out the nature of the trade-offs in both cases between the quality of the group decision outcomes, learning accuracy, communication cost, and the level of privacy protections that the agents are afforded.

Title: LLaGA: Large Language and Graph Assistant

• Authors: Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, Zhangyang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08170
• Pdf URL: https://arxiv.org/pdf/2402.08170
• Copy Paste: [[2402.08170]] LLaGA: Large Language and Graph Assistant(https://arxiv.org/abs/2402.08170)
• Keywords: interpretability, large language model
• Abstract: Graph Neural Networks (GNNs) have empowered the advance in graph-structured data analysis. Recently, the rise of Large Language Models (LLMs) like GPT-4 has heralded a new era in deep learning. However, their application to graph data poses distinct challenges due to the inherent difficulty of translating graph structures to language. To this end, we introduce the \textbf{L}arge \textbf{L}anguage \textbf{a}nd \textbf{G}raph \textbf{A}ssistant (\textbf{LLaGA}), an innovative model that effectively integrates LLM capabilities to handle the complexities of graph-structured data. LLaGA retains the general-purpose nature of LLMs while adapting graph data into a format compatible with LLM input. LLaGA achieves this by reorganizing graph nodes to structure-aware sequences and then mapping these into the token embedding space through a versatile projector. LLaGA excels in versatility, generalizability and interpretability, allowing it to perform consistently well across different datasets and tasks, extend its ability to unseen datasets or tasks, and provide explanations for graphs. Our extensive experiments across popular graph benchmarks show that LLaGA delivers outstanding performance across four datasets and three tasks using one single model, surpassing state-of-the-art graph models in both supervised and zero-shot scenarios. Our code is available at \url{https://github.com/ChenRunjin/LLaGA}

Title: Learning time-dependent PDE via graph neural networks and deep operator network for robust accuracy on irregular grids

• Authors: Sung Woong Cho, Jae Yong Lee, Hyung Ju Hwang
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2402.08187
• Pdf URL: https://arxiv.org/pdf/2402.08187
• Copy Paste: [[2402.08187]] Learning time-dependent PDE via graph neural networks and deep operator network for robust accuracy on irregular grids(https://arxiv.org/abs/2402.08187)
• Keywords: robust
• Abstract: Scientific computing using deep learning has seen significant advancements in recent years. There has been growing interest in models that learn the operator from the parameters of a partial differential equation (PDE) to the corresponding solutions. Deep Operator Network (DeepONet) and Fourier Neural operator, among other models, have been designed with structures suitable for handling functions as inputs and outputs, enabling real-time predictions as surrogate models for solution operators. There has also been significant progress in the research on surrogate models based on graph neural networks (GNNs), specifically targeting the dynamics in time-dependent PDEs. In this paper, we propose GraphDeepONet, an autoregressive model based on GNNs, to effectively adapt DeepONet, which is well-known for successful operator learning. GraphDeepONet exhibits robust accuracy in predicting solutions compared to existing GNN-based PDE solver models. It maintains consistent performance even on irregular grids, leveraging the advantages inherited from DeepONet and enabling predictions on arbitrary grids. Additionally, unlike traditional DeepONet and its variants, GraphDeepONet enables time extrapolation for time-dependent PDE solutions. We also provide theoretical analysis of the universal approximation capability of GraphDeepONet in approximating continuous operators across arbitrary time intervals.

Title: Optimized Information Flow for Transformer Tracking

• Authors: Janani Kugarajeevan, Thanikasalam Kokul, Amirthalingam Ramanan, Subha Fernando
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08195
• Pdf URL: https://arxiv.org/pdf/2402.08195
• Copy Paste: [[2402.08195]] Optimized Information Flow for Transformer Tracking(https://arxiv.org/abs/2402.08195)
• Keywords: transformer
• Abstract: One-stream Transformer trackers have shown outstanding performance in challenging benchmark datasets over the last three years, as they enable interaction between the target template and search region tokens to extract target-oriented features with mutual guidance. Previous approaches allow free bidirectional information flow between template and search tokens without investigating their influence on the tracker's discriminative capability. In this study, we conducted a detailed study on the information flow of the tokens and based on the findings, we propose a novel Optimized Information Flow Tracking (OIFTrack) framework to enhance the discriminative capability of the tracker. The proposed OIFTrack blocks the interaction from all search tokens to target template tokens in early encoder layers, as the large number of non-target tokens in the search region diminishes the importance of target-specific features. In the deeper encoder layers of the proposed tracker, search tokens are partitioned into target search tokens and non-target search tokens, allowing bidirectional flow from target search tokens to template tokens to capture the appearance changes of the target. In addition, since the proposed tracker incorporates dynamic background cues, distractor objects are successfully avoided by capturing the surrounding information of the target. The OIFTrack demonstrated outstanding performance in challenging benchmarks, particularly excelling in the one-shot tracking benchmark GOT-10k, achieving an average overlap of 74.6\%. The code, models, and results of this work are available at \url{https://github.com/JananiKugaa/OIFTrack}

Title: Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation

• Authors: AprilPyone MaungMaung, Huy H. Nguyen, Hitoshi Kiya, Isao Echizen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08200
• Pdf URL: https://arxiv.org/pdf/2402.08200
• Copy Paste: [[2402.08200]] Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation(https://arxiv.org/abs/2402.08200)
• Keywords: robust, diffusion
• Abstract: We propose a method for generating spurious features by leveraging large-scale text-to-image diffusion models. Although the previous work detects spurious features in a large-scale dataset like ImageNet and introduces Spurious ImageNet, we found that not all spurious images are spurious across different classifiers. Although spurious images help measure the reliance of a classifier, filtering many images from the Internet to find more spurious features is time-consuming. To this end, we utilize an existing approach of personalizing large-scale text-to-image diffusion models with available discovered spurious images and propose a new spurious feature similarity loss based on neural features of an adversarially robust model. Precisely, we fine-tune Stable Diffusion with several reference images from Spurious ImageNet with a modified objective incorporating the proposed spurious-feature similarity loss. Experiment results show that our method can generate spurious images that are consistently spurious across different classifiers. Moreover, the generated spurious images are visually similar to reference images from Spurious ImageNet.

Title: Confronting Discrimination in Classification: Smote Based on Marginalized Minorities in the Kernel Space for Imbalanced Data

• Authors: Lingyun Zhong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08202
• Pdf URL: https://arxiv.org/pdf/2402.08202
• Copy Paste: [[2402.08202]] Confronting Discrimination in Classification: Smote Based on Marginalized Minorities in the Kernel Space for Imbalanced Data(https://arxiv.org/abs/2402.08202)
• Keywords: robust
• Abstract: Financial fraud detection poses a typical challenge characterized by class imbalance, where instances of fraud are extremely rare but can lead to unpredictable economic losses if misidentified. Precisely classifying these critical minority samples represents a challenging task within the classification. The primary difficulty arises from mainstream classifiers, which often exhibit "implicit discrimination" against minority samples in evaluation metrics, which results in frequent misclassifications, and the key to the problem lies in the overlap of feature spaces between majority and minority samples. To address these challenges, oversampling is a feasible solution, yet current classical oversampling methods often lack the necessary caution in sample selection, exacerbating feature space overlap. In response, we propose a novel classification oversampling approach based on the decision boundary and sample proximity relationships. This method carefully considers the distance between critical samples and the decision hyperplane, as well as the density of surrounding samples, resulting in an adaptive oversampling strategy in the kernel space. Finally, we test the proposed method on a classic financial fraud dataset, and the results show that our proposed method provides an effective and robust solution that can improve the classification accuracy of minorities.

Title: Translating Images to Road Network:A Non-Autoregressive Sequence-to-Sequence Approach

• Authors: Jiachen Lu, Renyuan Peng, Xinyue Cai, Hang Xu, Hongyang Li, Feng Wen, Wei Zhang, Li Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08207
• Pdf URL: https://arxiv.org/pdf/2402.08207
• Copy Paste: [[2402.08207]] Translating Images to Road Network:A Non-Autoregressive Sequence-to-Sequence Approach(https://arxiv.org/abs/2402.08207)
• Keywords: extraction, transformer
• Abstract: The extraction of road network is essential for the generation of high-definition maps since it enables the precise localization of road landmarks and their interconnections. However, generating road network poses a significant challenge due to the conflicting underlying combination of Euclidean (e.g., road landmarks location) and non-Euclidean (e.g., road topological connectivity) structures. Existing methods struggle to merge the two types of data domains effectively, but few of them address it properly. Instead, our work establishes a unified representation of both types of data domain by projecting both Euclidean and non-Euclidean data into an integer series called RoadNet Sequence. Further than modeling an auto-regressive sequence-to-sequence Transformer model to understand RoadNet Sequence, we decouple the dependency of RoadNet Sequence into a mixture of auto-regressive and non-autoregressive dependency. Building on this, our proposed non-autoregressive sequence-to-sequence approach leverages non-autoregressive dependencies while fixing the gap towards auto-regressive dependencies, resulting in success on both efficiency and accuracy. Extensive experiments on nuScenes dataset demonstrate the superiority of RoadNet Sequence representation and the non-autoregressive approach compared to existing state-of-the-art alternatives. The code is open-source on https://github.com/fudan-zvg/RoadNetworkTRansformer.

Title: BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models

• Authors: Haotian Sun, Yuchen Zhuang, Wei Wei, Chao Zhang, Bo Dai
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08219
• Pdf URL: https://arxiv.org/pdf/2402.08219
• Copy Paste: [[2402.08219]] BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models(https://arxiv.org/abs/2402.08219)
• Keywords: privacy, large language model
• Abstract: Adapting state-of-the-art Large Language Models (LLMs) like GPT-4 and Gemini for specific tasks is challenging. Due to the opacity in their parameters, embeddings, and even output probabilities, existing fine-tuning adaptation methods are inapplicable. Consequently, adapting these black-box LLMs is only possible through their API services, raising concerns about transparency, privacy, and cost. To address these challenges, we introduce BBox-Adapter, a novel lightweight adapter for black-box LLMs. BBox-Adapter distinguishes target and source domain data by treating target data as positive and source data as negative. It employs a ranking-based Noise Contrastive Estimation (NCE) loss to promote the likelihood of target domain data while penalizing that of the source domain. Furthermore, it features an online adaptation mechanism, which incorporates real-time positive data sampling from ground-truth, human, or AI feedback, coupled with negative data from previous adaptations. Extensive experiments demonstrate BBox-Adapter's effectiveness and cost efficiency. It improves model performance by up to 6.77% across diverse tasks and domains, while reducing training and inference costs by 31.30x and 1.84x, respectively.

Title: Improving Black-box Robustness with In-Context Rewriting

• Authors: Kyle O'Brien, Nathan Ng, Isha Puri, Jorge Mendez, Hamid Palangi, Yoon Kim, Marzyeh Ghassemi, Thomas Hartvigsen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08225
• Pdf URL: https://arxiv.org/pdf/2402.08225
• Copy Paste: [[2402.08225]] Improving Black-box Robustness with In-Context Rewriting(https://arxiv.org/abs/2402.08225)
• Keywords: robust
• Abstract: Machine learning models often excel on in-distribution (ID) data but struggle with unseen out-of-distribution (OOD) inputs. Most techniques for improving OOD robustness are not applicable to settings where the model is effectively a black box, such as when the weights are frozen, retraining is costly, or the model is leveraged via an API. Test-time augmentation (TTA) is a simple post-hoc technique for improving robustness that sidesteps black-box constraints by aggregating predictions across multiple augmentations of the test input. TTA has seen limited use in NLP due to the challenge of generating effective natural language augmentations. In this work, we propose LLM-TTA, which uses LLM-generated augmentations as TTA's augmentation function. LLM-TTA outperforms conventional augmentation functions across sentiment, toxicity, and news classification tasks for BERT and T5 models, with BERT's OOD robustness improving by an average of 4.30 percentage points without regressing average ID performance. We explore selectively augmenting inputs based on prediction entropy to reduce the rate of expensive LLM augmentations, allowing us to maintain performance gains while reducing the average number of generated augmentations by 57.76%. LLM-TTA is agnostic to the task model architecture, does not require OOD labels, and is effective across low and high-resource settings. We share our data, models, and code for reproducibility.

Title: Privacy-Preserving Language Model Inference with Instance Obfuscation

• Authors: Yixiang Yao, Fei Wang, Srivatsan Ravi, Muhao Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08227
• Pdf URL: https://arxiv.org/pdf/2402.08227
• Copy Paste: [[2402.08227]] Privacy-Preserving Language Model Inference with Instance Obfuscation(https://arxiv.org/abs/2402.08227)
• Keywords: privacy, protect
• Abstract: Language Models as a Service (LMaaS) offers convenient access for developers and researchers to perform inference using pre-trained language models. Nonetheless, the input data and the inference results containing private information are exposed as plaintext during the service call, leading to privacy issues. Recent studies have started tackling the privacy issue by transforming input data into privacy-preserving representation from the user-end with the techniques such as noise addition and content perturbation, while the exploration of inference result protection, namely decision privacy, is still a blank page. In order to maintain the black-box manner of LMaaS, conducting data privacy protection, especially for the decision, is a challenging task because the process has to be seamless to the models and accompanied by limited communication and computation overhead. We thus propose Instance-Obfuscated Inference (IOI) method, which focuses on addressing the decision privacy issue of natural language understanding tasks in their complete life-cycle. Besides, we conduct comprehensive experiments to evaluate the performance as well as the privacy-protection strength of the proposed method on various benchmarking tasks.

Title: Investigating Out-of-Distribution Generalization of GNNs: An Architecture Perspective

• Authors: Kai Guo, Hongzhi Wen, Wei Jin, Yaming Guo, Jiliang Tang, Yi Chang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08228
• Pdf URL: https://arxiv.org/pdf/2402.08228
• Copy Paste: [[2402.08228]] Investigating Out-of-Distribution Generalization of GNNs: An Architecture Perspective(https://arxiv.org/abs/2402.08228)
• Keywords: robust
• Abstract: Graph neural networks (GNNs) have exhibited remarkable performance under the assumption that test data comes from the same distribution of training data. However, in real-world scenarios, this assumption may not always be valid. Consequently, there is a growing focus on exploring the Out-of-Distribution (OOD) problem in the context of graphs. Most existing efforts have primarily concentrated on improving graph OOD generalization from two \textbf{model-agnostic} perspectives: data-driven methods and strategy-based learning. However, there has been limited attention dedicated to investigating the impact of well-known \textbf{GNN model architectures} on graph OOD generalization, which is orthogonal to existing research. In this work, we provide the first comprehensive investigation of OOD generalization on graphs from an architecture perspective, by examining the common building blocks of modern GNNs. Through extensive experiments, we reveal that both the graph self-attention mechanism and the decoupled architecture contribute positively to graph OOD generalization. In contrast, we observe that the linear classification layer tends to compromise graph OOD generalization capability. Furthermore, we provide in-depth theoretical insights and discussions to underpin these discoveries. These insights have empowered us to develop a novel GNN backbone model, DGAT, designed to harness the robust properties of both graph self-attention mechanism and the decoupled architecture. Extensive experimental results demonstrate the effectiveness of our model under graph OOD, exhibiting substantial and consistent enhancements across various training strategies.

Title: APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks

• Authors: Barathi Subramanian, Rathinaraja Jeyaraj, Rakhmonov Akhrorjon Akhmadjon Ugli, Jeonghong Kim
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.08244
• Pdf URL: https://arxiv.org/pdf/2402.08244
• Copy Paste: [[2402.08244]] APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks(https://arxiv.org/abs/2402.08244)
• Keywords: robust, extraction
• Abstract: Activation function is a pivotal component of deep learning, facilitating the extraction of intricate data patterns. While classical activation functions like ReLU and its variants are extensively utilized, their static nature and simplicity, despite being advantageous, often limit their effectiveness in specialized tasks. The trainable activation functions also struggle sometimes to adapt to the unique characteristics of the data. Addressing these limitations, we introduce a novel trainable activation function, adaptive piecewise approximated activation linear unit (APALU), to enhance the learning performance of deep learning across a broad range of tasks. It presents a unique set of features that enable it to maintain stability and efficiency in the learning process while adapting to complex data representations. Experiments reveal significant improvements over widely used activation functions for different tasks. In image classification, APALU increases MobileNet and GoogleNet accuracy by 0.37% and 0.04%, respectively, on the CIFAR10 dataset. In anomaly detection, it improves the average area under the curve of One-CLASS Deep SVDD by 0.8% on the MNIST dataset, 1.81% and 1.11% improvements with DifferNet, and knowledge distillation, respectively, on the MVTech dataset. Notably, APALU achieves 100% accuracy on a sign language recognition task with a limited dataset. For regression tasks, APALU enhances the performance of deep neural networks and recurrent neural networks on different datasets. These improvements highlight the robustness and adaptability of APALU across diverse deep-learning applications.

Title: Object Detection in Thermal Images Using Deep Learning for Unmanned Aerial Vehicles

• Authors: Minh Dang Tu, Kieu Trang Le, Manh Duong Phung
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08251
• Pdf URL: https://arxiv.org/pdf/2402.08251
• Copy Paste: [[2402.08251]] Object Detection in Thermal Images Using Deep Learning for Unmanned Aerial Vehicles(https://arxiv.org/abs/2402.08251)
• Keywords: transformer
• Abstract: This work presents a neural network model capable of recognizing small and tiny objects in thermal images collected by unmanned aerial vehicles. Our model consists of three parts, the backbone, the neck, and the prediction head. The backbone is developed based on the structure of YOLOv5 combined with the use of a transformer encoder at the end. The neck includes a BI-FPN block combined with the use of a sliding window and a transformer to increase the information fed into the prediction head. The prediction head carries out the detection by evaluating feature maps with the Sigmoid function. The use of transformers with attention and sliding windows increases recognition accuracy while keeping the model at a reasonable number of parameters and computation requirements for embedded systems. Experiments conducted on public dataset VEDAI and our collected datasets show that our model has a higher accuracy than state-of-the-art methods such as ResNet, Faster RCNN, ComNet, ViT, YOLOv5, SMPNet, and DPNetV3. Experiments on the embedded computer Jetson AGX show that our model achieves a real-time computation speed with a stability rate of over 90%.

Title: A Survey of Table Reasoning with Large Language Models

• Authors: Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Qingfu Zhu, Wanxiang Che
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08259
• Pdf URL: https://arxiv.org/pdf/2402.08259
• Copy Paste: [[2402.08259]] A Survey of Table Reasoning with Large Language Models(https://arxiv.org/abs/2402.08259)
• Keywords: large language model
• Abstract: Table reasoning, which aims to generate the corresponding answer to the question following the user requirement according to the provided table, and optionally a text description of the table, effectively improving the efficiency of obtaining information. Recently, using Large Language Models (LLMs) has become the mainstream method for table reasoning, because it not only significantly reduces the annotation cost but also exceeds the performance of previous methods. However, existing research still lacks a summary of LLM-based table reasoning works. Due to the existing lack of research, questions about which techniques can improve table reasoning performance in the era of LLMs, why LLMs excel at table reasoning, and how to enhance table reasoning abilities in the future, remain largely unexplored. This gap significantly limits progress in research. To answer the above questions and advance table reasoning research with LLMs, we present this survey to analyze existing research, inspiring future work. In this paper, we analyze the mainstream techniques used to improve table reasoning performance in the LLM era, and the advantages of LLMs compared to pre-LLMs for solving table reasoning. We provide research directions from both the improvement of existing methods and the expansion of practical applications to inspire future research.

Title: A Dense Reward View on Aligning Text-to-Image Diffusion with Preference

• Authors: Shentao Yang, Tianqi Chen, Mingyuan Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08265
• Pdf URL: https://arxiv.org/pdf/2402.08265
• Copy Paste: [[2402.08265]] A Dense Reward View on Aligning Text-to-Image Diffusion with Preference(https://arxiv.org/abs/2402.08265)
• Keywords: diffusion
• Abstract: Aligning text-to-image diffusion model (T2I) with preference has been gaining increasing research attention. While prior works exist on directly optimizing T2I by preference data, these methods are developed under the bandit assumption of a latent reward on the entire diffusion reverse chain, while ignoring the sequential nature of the generation process. From literature, this may harm the efficacy and efficiency of alignment. In this paper, we take on a finer dense reward perspective and derive a tractable alignment objective that emphasizes the initial steps of the T2I reverse chain. In particular, we introduce temporal discounting into the DPO-style explicit-reward-free loss, to break the temporal symmetry therein and suit the T2I generation hierarchy. In experiments on single and multiple prompt generation, our method is competitive with strong relevant baselines, both quantitatively and qualitatively. Further studies are conducted to illustrate the insight of our approach.

Title: Improving Image Coding for Machines through Optimizing Encoder via Auxiliary Loss

• Authors: Kei Iino, Shunsuke Akamatsu, Hiroshi Watanabe, Shohei Enomoto, Akira Sakamoto, Takeharu Eda
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08267
• Pdf URL: https://arxiv.org/pdf/2402.08267
• Copy Paste: [[2402.08267]] Improving Image Coding for Machines through Optimizing Encoder via Auxiliary Loss(https://arxiv.org/abs/2402.08267)
• Keywords: segmentation
• Abstract: Image coding for machines (ICM) aims to compress images for machine analysis using recognition models rather than human vision. Hence, in ICM, it is important for the encoder to recognize and compress the information necessary for the machine recognition task. There are two main approaches in learned ICM; optimization of the compression model based on task loss, and Region of Interest (ROI) based bit allocation. These approaches provide the encoder with the recognition capability. However, optimization with task loss becomes difficult when the recognition model is deep, and ROI-based methods often involve extra overhead during evaluation. In this study, we propose a novel training method for learned ICM models that applies auxiliary loss to the encoder to improve its recognition capability and rate-distortion performance. Our method achieves Bjontegaard Delta rate improvements of 27.7% and 20.3% in object detection and semantic segmentation tasks, compared to the conventional training method.

Title: World Model on Million-Length Video And Language With RingAttention

• Authors: Hao Liu, Wilson Yan, Matei Zaharia, Pieter Abbeel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08268
• Pdf URL: https://arxiv.org/pdf/2402.08268
• Copy Paste: [[2402.08268]] World Model on Million-Length Video And Language With RingAttention(https://arxiv.org/abs/2402.08268)
• Keywords: transformer
• Abstract: Current language models fall short in understanding aspects of the world not easily described in words, and struggle with complex, long-form tasks. Video sequences offer valuable temporal information absent in language and static images, making them attractive for joint modeling with language. Such models could develop a understanding of both human textual knowledge and the physical world, enabling broader AI capabilities for assisting humans. However, learning from millions of tokens of video and language sequences poses challenges due to memory constraints, computational complexity, and limited datasets. To address these challenges, we curate a large dataset of diverse videos and books, utilize the RingAttention technique to scalably train on long sequences, and gradually increase context size from 4K to 1M tokens. This paper makes the following contributions: (a) Largest context size neural network: We train one of the largest context size transformers on long video and language sequences, setting new benchmarks in difficult retrieval tasks and long video understanding. (b) Solutions for overcoming vision-language training challenges, including using masked sequence packing for mixing different sequence lengths, loss weighting to balance language and vision, and model-generated QA dataset for long sequence chat. (c) A highly-optimized implementation with RingAttention, masked sequence packing, and other key features for training on millions-length multimodal sequences. (d) Fully open-sourced a family of 7B parameter models capable of processing long text documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M tokens. This work paves the way for training on massive datasets of long video and language to develop understanding of both human knowledge and the multimodal world, and broader capabilities.

Title: Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering

• Authors: Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Ash, Markus Leippold
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08277
• Pdf URL: https://arxiv.org/pdf/2402.08277
• Copy Paste: [[2402.08277]] Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering(https://arxiv.org/abs/2402.08277)
• Keywords: robust, large language model
• Abstract: Advances towards more faithful and traceable answers of Large Language Models (LLMs) are crucial for various research and practical endeavors. One avenue in reaching this goal is basing the answers on reliable sources. However, this Evidence-Based QA has proven to work insufficiently with LLMs in terms of citing the correct sources (source quality) and truthfully representing the information within sources (answer attributability). In this work, we systematically investigate how to robustly fine-tune LLMs for better source quality and answer attributability. Specifically, we introduce a data generation pipeline with automated data quality filters, which can synthesize diversified high-quality training and testing data at scale. We further introduce four test sets to benchmark the robustness of fine-tuned specialist models. Extensive evaluation shows that fine-tuning on synthetic data improves performance on both in- and out-of-distribution. %Evidence-Based QA cases. Furthermore, we show that data quality, which can be drastically improved by proposed quality filters, matters more than quantity in improving Evidence-Based QA.

Title: Learning semantic image quality for fetal ultrasound from noisy ranking annotation

• Authors: Manxi Lin, Jakob Ambsdorf, Emilie Pi Fogtmann Sejer, Zahra Bashir, Chun Kit Wong, Paraskevas Pegios, Alberto Raheli, Morten Bo Søndergaard Svendsen, Mads Nielsen, Martin Grønnebæk Tolsgaard, Anders Nymark Christensen, Aasa Feragen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08294
• Pdf URL: https://arxiv.org/pdf/2402.08294
• Copy Paste: [[2402.08294]] Learning semantic image quality for fetal ultrasound from noisy ranking annotation(https://arxiv.org/abs/2402.08294)
• Keywords: robust
• Abstract: We introduce the notion of semantic image quality for applications where image quality relies on semantic requirements. Working in fetal ultrasound, where ranking is challenging and annotations are noisy, we design a robust coarse-to-fine model that ranks images based on their semantic image quality and endow our predicted rankings with an uncertainty estimate. To annotate rankings on training data, we design an efficient ranking annotation scheme based on the merge sort algorithm. Finally, we compare our ranking algorithm to a number of state-of-the-art ranking algorithms on a challenging fetal ultrasound quality assessment task, showing the superior performance of our method on the majority of rank correlation metrics.

Title: Zero Trust Score-based Network-level Access Control in Enterprise Networks

• Authors: Leonard Bradatsch, Oleksandr Miroshkin, Natasa Trkulja, Frank Kargl
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08299
• Pdf URL: https://arxiv.org/pdf/2402.08299
• Copy Paste: [[2402.08299]] Zero Trust Score-based Network-level Access Control in Enterprise Networks(https://arxiv.org/abs/2402.08299)
• Keywords: security
• Abstract: Zero Trust security has recently gained attention in enterprise network security. One of its key ideas is making network-level access decisions based on trust scores. However, score-based access control in the enterprise domain still lacks essential elements in our understanding, and in this paper, we contribute with respect to three crucial aspects. First, we provide a comprehensive list of 29 trust attributes that can be used to calculate a trust score. By introducing a novel mathematical approach, we demonstrate how to quantify these attributes. Second, we describe a dynamic risk-based method to calculate the trust threshold the trust score must meet for permitted access. Third, we introduce a novel trust algorithm based on Subjective Logic that incorporates the first two contributions and offers fine-grained decision possibilities. We discuss how this algorithm shows a higher expressiveness compared to a lightweight additive trust algorithm. Performance-wise, a prototype of the Subjective Logic-based approach showed similar calculation times for making an access decision as the additive approach. In addition, the dynamic threshold calculation showed only 7% increased decision-making times compared to a static threshold.

Title: ChatCell: Facilitating Single-Cell Analysis with Natural Language

• Authors: Yin Fang, Kangwei Liu, Ningyu Zhang, Xinle Deng, Penghui Yang, Zhuo Chen, Xiangru Tang, Mark Gerstein, Xiaohui Fan, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.CE, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08303
• Pdf URL: https://arxiv.org/pdf/2402.08303
• Copy Paste: [[2402.08303]] ChatCell: Facilitating Single-Cell Analysis with Natural Language(https://arxiv.org/abs/2402.08303)
• Keywords: robust, large language model
• Abstract: As Large Language Models (LLMs) rapidly evolve, their influence in science is becoming increasingly prominent. The emerging capabilities of LLMs in task generalization and free-form dialogue can significantly advance fields like chemistry and biology. However, the field of single-cell biology, which forms the foundational building blocks of living organisms, still faces several challenges. High knowledge barriers and limited scalability in current methods restrict the full exploitation of LLMs in mastering single-cell data, impeding direct accessibility and rapid iteration. To this end, we introduce ChatCell, which signifies a paradigm shift by facilitating single-cell analysis with natural language. Leveraging vocabulary adaptation and unified sequence generation, ChatCell has acquired profound expertise in single-cell biology and the capability to accommodate a diverse range of analysis tasks. Extensive experiments further demonstrate ChatCell's robust performance and potential to deepen single-cell insights, paving the way for more accessible and intuitive exploration in this pivotal field. Our project homepage is available at https://zjunlp.github.io/project/ChatCell.

Title: Prompted Contextual Vectors for Spear-Phishing Detection

• Authors: Daniel Nahmias, Gal Engelberg, Dan Klein, Asaf Shabtai
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08309
• Pdf URL: https://arxiv.org/pdf/2402.08309
• Copy Paste: [[2402.08309]] Prompted Contextual Vectors for Spear-Phishing Detection(https://arxiv.org/abs/2402.08309)
• Keywords: security, attack, large language model
• Abstract: Spear-phishing attacks present a significant security challenge, with large language models (LLMs) escalating the threat by generating convincing emails and facilitating target reconnaissance. To address this, we propose a detection approach based on a novel document vectorization method that utilizes an ensemble of LLMs to create representation vectors. By prompting LLMs to reason and respond to human-crafted questions, we quantify the presence of common persuasion principles in the email's content, producing prompted contextual document vectors for a downstream supervised machine learning model. We evaluate our method using a unique dataset generated by a proprietary system that automates target reconnaissance and spear-phishing email creation. Our method achieves a 91% F1 score in identifying LLM-generated spear-phishing emails, with the training set comprising only traditional phishing and benign emails. Key contributions include an innovative document vectorization method utilizing LLM reasoning, a publicly available dataset of high-quality spear-phishing emails, and the demonstrated effectiveness of our method in detecting such emails. This methodology can be utilized for various document classification tasks, particularly in adversarial problem domains.

Title: One-to-many Reconstruction of 3D Geometry of cultural Artifacts using a synthetically trained Generative Model

• Authors: Thomas Pöllabauer, Julius Kühn, Jiayi Li, Arjan Kuijper
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08310
• Pdf URL: https://arxiv.org/pdf/2402.08310
• Copy Paste: [[2402.08310]] One-to-many Reconstruction of 3D Geometry of cultural Artifacts using a synthetically trained Generative Model(https://arxiv.org/abs/2402.08310)
• Keywords: generative
• Abstract: Estimating the 3D shape of an object using a single image is a difficult problem. Modern approaches achieve good results for general objects, based on real photographs, but worse results on less expressive representations such as historic sketches. Our automated approach generates a variety of detailed 3D representation from a single sketch, depicting a medieval statue, and can be guided by multi-modal inputs, such as text prompts. It relies solely on synthetic data for training, making it adoptable even in cases of only small numbers of training examples. Our solution allows domain experts such as a curators to interactively reconstruct potential appearances of lost artifacts.

Title: Approximating Families of Sharp Solutions to Fisher's Equation with Physics-Informed Neural Networks

• Authors: Franz M. Rohrhofer, Stefan Posch, Clemens Gößnitzer, Bernhard C. Geiger
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08313
• Pdf URL: https://arxiv.org/pdf/2402.08313
• Copy Paste: [[2402.08313]] Approximating Families of Sharp Solutions to Fisher's Equation with Physics-Informed Neural Networks(https://arxiv.org/abs/2402.08313)
• Keywords: diffusion
• Abstract: This paper employs physics-informed neural networks (PINNs) to solve Fisher's equation, a fundamental representation of a reaction-diffusion system with both simplicity and significance. The focus lies specifically in investigating Fisher's equation under conditions of large reaction rate coefficients, wherein solutions manifest as traveling waves, posing a challenge for numerical methods due to the occurring steepness of the wave front. To address optimization challenges associated with the standard PINN approach, a residual weighting scheme is introduced. This scheme is designed to enhance the tracking of propagating wave fronts by considering the reaction term in the reaction-diffusion equation. Furthermore, a specific network architecture is studied which is tailored for solutions in the form of traveling waves. Lastly, the capacity of PINNs to approximate an entire family of solutions is assessed by incorporating the reaction rate coefficient as an additional input to the network architecture. This modification enables the approximation of the solution across a broad and continuous range of reaction rate coefficients, thus solving a class of reaction-diffusion systems using a single PINN instance.

Title: Explicit References to Social Values in Fairy Tales: A Comparison between Three European Cultures

• Authors: Alba Morollon Diaz-Faes, Carla Sofia Ribeiro Murteira, Martin Ruskov
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.08318
• Pdf URL: https://arxiv.org/pdf/2402.08318
• Copy Paste: [[2402.08318]] Explicit References to Social Values in Fairy Tales: A Comparison between Three European Cultures(https://arxiv.org/abs/2402.08318)
• Keywords: fair
• Abstract: The study of social values in fairy tales opens the possibility to learn about the communication of values across space and time. We propose to study the communication of values in fairy tales from Portugal, Italy and Germany using a technique called word embedding with a compass to quantify vocabulary differences and commonalities. We study how these three national traditions of fairy tales differ in their explicit references to values. To do this, we specify a list of value-charged tokens, consider their word stems and analyse the distance between these in a bespoke pre-trained Word2Vec model. We triangulate and critically discuss the validity of the resulting hypotheses emerging from this quantitative model. Our claim is that this is a reusable and reproducible method for the study of the values explicitly referenced in historical corpora. Finally, our preliminary findings hint at a shared cultural understanding and the expression of values such as Benevolence, Conformity, and Universalism across European societies, suggesting the existence of a pan-European cultural memory.

Title: The Paradox of Motion: Evidence for Spurious Correlations in Skeleton-based Gait Recognition Models

• Authors: Andy Cătrună, Adrian Cosma, Emilian Rădoi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08320
• Pdf URL: https://arxiv.org/pdf/2402.08320
• Copy Paste: [[2402.08320]] The Paradox of Motion: Evidence for Spurious Correlations in Skeleton-based Gait Recognition Models(https://arxiv.org/abs/2402.08320)
• Keywords: biometric, transformer
• Abstract: Gait, an unobtrusive biometric, is valued for its capability to identify individuals at a distance, across external outfits and environmental conditions. This study challenges the prevailing assumption that vision-based gait recognition, in particular skeleton-based gait recognition, relies primarily on motion patterns, revealing a significant role of the implicit anthropometric information encoded in the walking sequence. We show through a comparative analysis that removing height information leads to notable performance degradation across three models and two benchmarks (CASIA-B and GREW). Furthermore, we propose a spatial transformer model processing individual poses, disregarding any temporal information, which achieves unreasonably good accuracy, emphasizing the bias towards appearance information and indicating spurious correlations in existing benchmarks. These findings underscore the need for a nuanced understanding of the interplay between motion and appearance in vision-based gait recognition, prompting a reevaluation of the methodological assumptions in this field. Our experiments indicate that "in-the-wild" datasets are less prone to spurious correlations, prompting the need for more diverse and large scale datasets for advancing the field.

Title: Exploration by Optimization with Hybrid Regularizers: Logarithmic Regret with Adversarial Robustness in Partial Monitoring

• Authors: Taira Tsuchiya, Shinji Ito, Junya Honda
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.08321
• Pdf URL: https://arxiv.org/pdf/2402.08321
• Copy Paste: [[2402.08321]] Exploration by Optimization with Hybrid Regularizers: Logarithmic Regret with Adversarial Robustness in Partial Monitoring(https://arxiv.org/abs/2402.08321)
• Keywords: robust
• Abstract: Partial monitoring is a generic framework of online decision-making problems with limited observations. To make decisions from such limited observations, it is necessary to find an appropriate distribution for exploration. Recently, a powerful approach for this purpose, exploration by optimization (ExO), was proposed, which achieves the optimal bounds in adversarial environments with follow-the-regularized-leader for a wide range of online decision-making problems. However, a naive application of ExO in stochastic environments significantly degrades regret bounds. To resolve this problem in locally observable games, we first establish a novel framework and analysis for ExO with a hybrid regularizer. This development allows us to significantly improve the existing regret bounds of best-of-both-worlds (BOBW) algorithms, which achieves nearly optimal bounds both in stochastic and adversarial environments. In particular, we derive a stochastic regret bound of $O(\sum_{a \neq a^*} k^2 m^2 \log T / \Delta_a)$, where $k$, $m$, and $T$ are the numbers of actions, observations and rounds, $a^*$ is an optimal action, and $\Delta_a$ is the suboptimality gap for action $a$. This bound is roughly $\Theta(k^2 \log T)$ times smaller than existing BOBW bounds. In addition, for globally observable games, we provide a new BOBW algorithm with the first $O(\log T)$ stochastic bound.

Title: zk-IoT: Securing the Internet of Things with Zero-Knowledge Proofs on Blockchain Platforms

• Authors: Gholamreza Ramezan, Ehsan Meamari
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08322
• Pdf URL: https://arxiv.org/pdf/2402.08322
• Copy Paste: [[2402.08322]] zk-IoT: Securing the Internet of Things with Zero-Knowledge Proofs on Blockchain Platforms(https://arxiv.org/abs/2402.08322)
• Keywords: secure, security
• Abstract: This paper introduces the zk-IoT framework, a novel approach to enhancing the security of Internet of Things (IoT) ecosystems through the use of Zero-Knowledge Proofs (ZKPs) on blockchain platforms. Our framework ensures the integrity of firmware execution and data processing in potentially compromised IoT devices. By leveraging the concept of ZKP, we establish a trust layer that facilitates secure, autonomous communication between IoT devices in environments where devices may not inherently trust each other. The framework comprises zk-Devices, which utilize functional commitment to generate proofs for executed programs, and service contracts for encoding interaction logic among devices. It also provides for IoT device automation using proof-carrying data (PCD) and a blockchain layer for transparent and verifiable data processing. We conduct experiments, the results of which show that proof generation, publication, and verification timings meet the practical requirements of IoT device communication, demonstrating the feasibility and efficiency of our solution. The zk-IoT framework represents a significant advancement in the realm of IoT security, paving the way for reliable and scalable IoT networks across various applications, such as smart city infrastructures, healthcare systems, and industrial automation.

Title: Scribble-based fast weak-supervision and interactive corrections for segmenting whole slide images

• Authors: Antoine Habis, Roy Rosman Nathanson, Vannary Meas-Yedid, Elsa D. Angelini, Jean-Christophe Olivo-Marin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08333
• Pdf URL: https://arxiv.org/pdf/2402.08333
• Copy Paste: [[2402.08333]] Scribble-based fast weak-supervision and interactive corrections for segmenting whole slide images(https://arxiv.org/abs/2402.08333)
• Keywords: segmentation
• Abstract: This paper proposes a dynamic interactive and weakly supervised segmentation method with minimal user interactions to address two major challenges in the segmentation of whole slide histopathology images. First, the lack of hand-annotated datasets to train algorithms. Second, the lack of interactive paradigms to enable a dialogue between the pathologist and the machine, which can be a major obstacle for use in clinical routine. We therefore propose a fast and user oriented method to bridge this gap by giving the pathologist control over the final result while limiting the number of interactions needed to achieve a good result (over 90\% on all our metrics with only 4 correction scribbles).

Title: Eliciting Big Five Personality Traits in Large Language Models: A Textual Analysis with Classifier-Driven Approach

• Authors: Airlie Hilliard, Cristian Munoz, Zekun Wu, Adriano Soares Koshiyama
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08341
• Pdf URL: https://arxiv.org/pdf/2402.08341
• Copy Paste: [[2402.08341]] Eliciting Big Five Personality Traits in Large Language Models: A Textual Analysis with Classifier-Driven Approach(https://arxiv.org/abs/2402.08341)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are increasingly being utilized by both candidates and employers in the recruitment context. However, with this comes numerous ethical concerns, particularly related to the lack of transparency in these "black-box" models. Although previous studies have sought to increase the transparency of these models by investigating the personality traits of LLMs, many of the previous studies have provided them with personality assessments to complete. On the other hand, this study seeks to obtain a better understanding of such models by examining their output variations based on different input prompts. Specifically, we use a novel elicitation approach using prompts derived from common interview questions, as well as prompts designed to elicit particular Big Five personality traits to examine whether the models were susceptible to trait-activation like humans are, to measure their personality based on the language used in their outputs. To do so, we repeatedly prompted multiple LMs with different parameter sizes, including Llama-2, Falcon, Mistral, Bloom, GPT, OPT, and XLNet (base and fine tuned versions) and examined their personality using classifiers trained on the myPersonality dataset. Our results reveal that, generally, all LLMs demonstrate high openness and low extraversion. However, whereas LMs with fewer parameters exhibit similar behaviour in personality traits, newer and LMs with more parameters exhibit a broader range of personality traits, with increased agreeableness, emotional stability, and openness. Furthermore, a greater number of parameters is positively associated with openness and conscientiousness. Moreover, fine-tuned models exhibit minor modulations in their personality traits, contingent on the dataset. Implications and directions for future research are discussed.

Title: Visually Dehallucinative Instruction Generation

• Authors: Sungguk Cha, Jusung Lee, Younghyun Lee, Cheoljong Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08348
• Pdf URL: https://arxiv.org/pdf/2402.08348
• Copy Paste: [[2402.08348]] Visually Dehallucinative Instruction Generation(https://arxiv.org/abs/2402.08348)
• Keywords: generative
• Abstract: In recent years, synthetic visual instructions by generative language model have demonstrated plausible text generation performance on the visual question-answering tasks. However, challenges persist in the hallucination of generative language models, i.e., the generated image-text data contains unintended contents. This paper presents a novel and scalable method for generating visually dehallucinative instructions, dubbed CAP2QA, that constrains the scope to only image contents. Our key contributions lie in introducing image-aligned instructive QA dataset CAP2QA-COCO and its scalable recipe. In our experiments, we compare synthetic visual instruction datasets that share the same source data by visual instruction tuning and conduct general visual recognition tasks. It shows that our proposed method significantly reduces visual hallucination while consistently improving visual recognition ability and expressiveness.

Title: Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks

• Authors: Jusung Lee, Sungguk Cha, Younghyun Lee, Cheoljong Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08360
• Pdf URL: https://arxiv.org/pdf/2402.08360
• Copy Paste: [[2402.08360]] Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks(https://arxiv.org/abs/2402.08360)
• Keywords: large language model
• Abstract: Having revolutionized natural language processing (NLP) applications, large language models (LLMs) are expanding into the realm of multimodal inputs. Owing to their ability to interpret images, multimodal LLMs (MLLMs) have been primarily used for vision-language tasks. Currently, MLLMs have not yet been extended for domain-specific visual tasks, which require a more explicit understanding of visual information. We developed a method to transform domain-specific visual and vision-language datasets into a unified question answering format called Visual Question Answering Instruction (VQA-IN), thereby extending MLLM to domain-specific tasks. The VQA-IN was applied to train multiple MLLM architectures using smaller versions of LLMs (sLLMs). The experimental results indicated that the proposed method achieved a high score metric on domainspecific visual tasks while also maintaining its performance on vision-language tasks in a multitask manner.

Title: Punctuation Restoration Improves Structure Understanding without Supervision

• Authors: Junghyun Min, Minho Lee, Woochul Lee, Yeonsoo Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08382
• Pdf URL: https://arxiv.org/pdf/2402.08382
• Copy Paste: [[2402.08382]] Punctuation Restoration Improves Structure Understanding without Supervision(https://arxiv.org/abs/2402.08382)
• Keywords: robust, extraction, large language model
• Abstract: Unsupervised learning objectives like language modeling and de-noising constitute a significant part in producing pre-trained models that perform various downstream applications from natural language understanding to conversational tasks. However, despite impressive conversational capabilities of recent large language model, their abilities to capture syntactic or semantic structure within text lag behind. We hypothesize that the mismatch between linguistic performance and competence in machines is attributable to insufficient transfer of linguistic structure knowledge to computational systems with currently popular pre-training objectives. We show that punctuation restoration transfers to improvements in in- and out-of-distribution performance on structure-related tasks like named entity recognition, open information extraction, chunking, and part-of-speech tagging. Punctuation restoration is an effective learning objective that can improve structure understanding and yield a more robust structure-aware representations of natural language.

Title: Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution

• Authors: Tailin Wu, Willie Neiswanger, Hongtao Zheng, Stefano Ermon, Jure Leskovec
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08383
• Pdf URL: https://arxiv.org/pdf/2402.08383
• Copy Paste: [[2402.08383]] Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution(https://arxiv.org/abs/2402.08383)
• Keywords: robust
• Abstract: Deep learning-based surrogate models have demonstrated remarkable advantages over classical solvers in terms of speed, often achieving speedups of 10 to 1000 times over traditional partial differential equation (PDE) solvers. However, a significant challenge hindering their widespread adoption in both scientific and industrial domains is the lack of understanding about their prediction uncertainties, particularly in scenarios that involve critical decision making. To address this limitation, we propose a method that integrates efficient and precise uncertainty quantification into a deep learning-based surrogate model. Our method, termed Latent Evolution of PDEs with Uncertainty Quantification (LE-PDE-UQ), endows deep learning-based surrogate models with robust and efficient uncertainty quantification capabilities for both forward and inverse problems. LE-PDE-UQ leverages latent vectors within a latent space to evolve both the system's state and its corresponding uncertainty estimation. The latent vectors are decoded to provide predictions for the system's state as well as estimates of its uncertainty. In extensive experiments, we demonstrate the accurate uncertainty quantification performance of our approach, surpassing that of strong baselines including deep ensembles, Bayesian neural network layers, and dropout. Our method excels at propagating uncertainty over extended auto-regressive rollouts, making it suitable for scenarios involving long-term predictions. Our code is available at: https://github.com/AI4Science-WestlakeU/le-pde-uq.

Title: Selective Learning: Towards Robust Calibration with Dynamic Regularization

• Authors: Zongbo Han, Yifeng Yang, Changqing Zhang, Linjun Zhang, Joey Tianyi Zhou, Qinghua Hu, Huaxiu Yao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08384
• Pdf URL: https://arxiv.org/pdf/2402.08384
• Copy Paste: [[2402.08384]] Selective Learning: Towards Robust Calibration with Dynamic Regularization(https://arxiv.org/abs/2402.08384)
• Keywords: robust
• Abstract: Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. This problem usually arises due to the overfitting problem, which is characterized by learning everything presented in the training set, resulting in overconfident predictions during testing. Existing methods typically address overfitting and mitigate the miscalibration by adding a maximum-entropy regularizer to the objective function. The objective can be understood as seeking a model that fits the ground-truth labels by increasing the confidence while also maximizing the entropy of predicted probabilities by decreasing the confidence. However, previous methods lack clear guidance on confidence adjustment, leading to conflicting objectives (increasing but also decreasing confidence). Therefore, we introduce a method called Dynamic Regularization (DReg), which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off. At a high level, DReg aims to obtain a more reliable model capable of acknowledging what it knows and does not know. Specifically, DReg effectively fits the labels for in-distribution samples (samples that should be learned) while applying regularization dynamically to samples beyond model capabilities (e.g., outliers), thereby obtaining a robust calibrated model especially on the samples beyond model capabilities. Both theoretical and empirical analyses sufficiently demonstrate the superiority of DReg compared with previous methods.

Title: Large Language Models as Minecraft Agents

• Authors: Chris Madge, Massimo Poesio
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08392
• Pdf URL: https://arxiv.org/pdf/2402.08392
• Copy Paste: [[2402.08392]] Large Language Models as Minecraft Agents(https://arxiv.org/abs/2402.08392)
• Keywords: large language model
• Abstract: In this work we examine the use of Large Language Models (LLMs) in the challenging setting of acting as a Minecraft agent. We apply and evaluate LLMs in the builder and architect settings, introduce clarification questions and examining the challenges and opportunities for improvement. In addition, we present a platform for online interaction with the agents and an evaluation against previous works.

Title: Adaptive Hierarchical Certification for Segmentation using Randomized Smoothing

• Authors: Alaa Anani, Tobias Lorenz, Bernt Schiele, Mario Fritz
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08400
• Pdf URL: https://arxiv.org/pdf/2402.08400
• Copy Paste: [[2402.08400]] Adaptive Hierarchical Certification for Segmentation using Randomized Smoothing(https://arxiv.org/abs/2402.08400)
• Keywords: segmentation
• Abstract: Common certification methods operate on a flat pre-defined set of fine-grained classes. In this paper, however, we propose a novel, more general, and practical setting, namely adaptive hierarchical certification for image semantic segmentation. In this setting, the certification can be within a multi-level hierarchical label space composed of fine to coarse levels. Unlike classic methods where the certification would abstain for unstable components, our approach adaptively relaxes the certification to a coarser level within the hierarchy. This relaxation lowers the abstain rate whilst providing more certified semantically meaningful information. We mathematically formulate the problem setup and introduce, for the first time, an adaptive hierarchical certification algorithm for image semantic segmentation, that certifies image pixels within a hierarchy and prove the correctness of its guarantees. Since certified accuracy does not take the loss of information into account when traversing into a coarser hierarchy level, we introduce a novel evaluation paradigm for adaptive hierarchical certification, namely the certified information gain metric, which is proportional to the class granularity level. Our evaluation experiments on real-world challenging datasets such as Cityscapes and ACDC demonstrate that our adaptive algorithm achieves a higher certified information gain and a lower abstain rate compared to the current state-of-the-art certification method, as well as other non-adaptive versions of it.

Title: Coding-Based Hybrid Post-Quantum Cryptosystem for Non-Uniform Information

• Authors: Saar Tarnopolsky, Alejandro Cohen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08407
• Pdf URL: https://arxiv.org/pdf/2402.08407
• Copy Paste: [[2402.08407]] Coding-Based Hybrid Post-Quantum Cryptosystem for Non-Uniform Information(https://arxiv.org/abs/2402.08407)
• Keywords: secure, security, attack
• Abstract: We introduce for non-uniform messages a novel hybrid universal network coding cryptosystem (NU-HUNCC) in the finite blocklength regime that provides Post-Quantum (PQ) security at high communication rates. Recently, hybrid cryptosystems offered PQ security by premixing the data using secure coding schemes and encrypting only a small portion of it, assuming the data is uniformly distributed. An assumption that is often challenging to enforce. Standard fixed-length lossless source coding and compression schemes guarantee a uniform output in normalized divergence. Yet, his is not sufficient to guarantee security. We consider an efficient almost uniform compression scheme in non-normalized variational distance for the proposed hybrid cryptosystem, that by utilizing uniform sub-linear shared seed, guarantees PQ security. Specifically, for the proposed PQ cryptosystem, first, we provide an end-to-end coding scheme, NU-HUNCC, for non-uniform messages. Second, we show that NU-HUNCC is information-theoretic individually secured (IS) against an eavesdropper with access to any subset of the links. Third, we introduce a modified security definition, individually semantically secure under a chosen ciphertext attack (ISS-CCA1), and show that against an all-observing eavesdropper, NU-HUNCC satisfies its conditions. Finally, we provide an analysis that shows the high communication rate of NU-HUNCC and the negligibility of the shared seed size.

Title: Transferring Ultrahigh-Field Representations for Intensity-Guided Brain Segmentation of Low-Field Magnetic Resonance Imaging

• Authors: Kwanseok Oh, Jieun Lee, Da-Woon Heo, Dinggang Shen, Heung-Il Suk
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08409
• Pdf URL: https://arxiv.org/pdf/2402.08409
• Copy Paste: [[2402.08409]] Transferring Ultrahigh-Field Representations for Intensity-Guided Brain Segmentation of Low-Field Magnetic Resonance Imaging(https://arxiv.org/abs/2402.08409)
• Keywords: segmentation
• Abstract: Ultrahigh-field (UHF) magnetic resonance imaging (MRI), i.e., 7T MRI, provides superior anatomical details of internal brain structures owing to its enhanced signal-to-noise ratio and susceptibility-induced contrast. However, the widespread use of 7T MRI is limited by its high cost and lower accessibility compared to low-field (LF) MRI. This study proposes a deep-learning framework that systematically fuses the input LF magnetic resonance feature representations with the inferred 7T-like feature representations for brain image segmentation tasks in a 7T-absent environment. Specifically, our adaptive fusion module aggregates 7T-like features derived from the LF image by a pre-trained network and then refines them to be effectively assimilable UHF guidance into LF image features. Using intensity-guided features obtained from such aggregation and assimilation, segmentation models can recognize subtle structural representations that are usually difficult to recognize when relying only on LF features. Beyond such advantages, this strategy can seamlessly be utilized by modulating the contrast of LF features in alignment with UHF guidance, even when employing arbitrary segmentation models. Exhaustive experiments demonstrated that the proposed method significantly outperformed all baseline models on both brain tissue and whole-brain segmentation tasks; further, it exhibited remarkable adaptability and scalability by successfully integrating diverse segmentation models and tasks. These improvements were not only quantifiable but also visible in the superlative visual quality of segmentation masks.

Title: Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning

• Authors: Gelei Deng, Yi Liu, Kailong Wang, Yuekang Li, Tianwei Zhang, Yang Liu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08416
• Pdf URL: https://arxiv.org/pdf/2402.08416
• Copy Paste: [[2402.08416]] Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning(https://arxiv.org/abs/2402.08416)
• Keywords: security, attack, large language model
• Abstract: Large Language Models~(LLMs) have gained immense popularity and are being increasingly applied in various domains. Consequently, ensuring the security of these models is of paramount importance. Jailbreak attacks, which manipulate LLMs to generate malicious content, are recognized as a significant vulnerability. While existing research has predominantly focused on direct jailbreak attacks on LLMs, there has been limited exploration of indirect methods. The integration of various plugins into LLMs, notably Retrieval Augmented Generation~(RAG), which enables LLMs to incorporate external knowledge bases into their response generation such as GPTs, introduces new avenues for indirect jailbreak attacks. To fill this gap, we investigate indirect jailbreak attacks on LLMs, particularly GPTs, introducing a novel attack vector named Retrieval Augmented Generation Poisoning. This method, Pandora, exploits the synergy between LLMs and RAG through prompt manipulation to generate unexpected responses. Pandora uses maliciously crafted content to influence the RAG process, effectively initiating jailbreak attacks. Our preliminary tests show that Pandora successfully conducts jailbreak attacks in four different scenarios, achieving higher success rates than direct attacks, with 64.3\% for GPT-3.5 and 34.8\% for GPT-4.

Title: Leveraging Self-Supervised Instance Contrastive Learning for Radar Object Detection

• Authors: Colin Decourt, Rufin VanRullen, Didier Salle, Thomas Oberlin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08427
• Pdf URL: https://arxiv.org/pdf/2402.08427
• Copy Paste: [[2402.08427]] Leveraging Self-Supervised Instance Contrastive Learning for Radar Object Detection(https://arxiv.org/abs/2402.08427)
• Keywords: robust
• Abstract: In recent years, driven by the need for safer and more autonomous transport systems, the automotive industry has shifted toward integrating a growing number of Advanced Driver Assistance Systems (ADAS). Among the array of sensors employed for object recognition tasks, radar sensors have emerged as a formidable contender due to their abilities in adverse weather conditions or low-light scenarios and their robustness in maintaining consistent performance across diverse environments. However, the small size of radar datasets and the complexity of the labelling of those data limit the performance of radar object detectors. Driven by the promising results of self-supervised learning in computer vision, this paper presents RiCL, an instance contrastive learning framework to pre-train radar object detectors. We propose to exploit the detection from the radar and the temporal information to pre-train the radar object detection model in a self-supervised way using contrastive learning. We aim to pre-train an object detector's backbone, head and neck to learn with fewer data. Experiments on the CARRADA and the RADDet datasets show the effectiveness of our approach in learning generic representations of objects in range-Doppler maps. Notably, our pre-training strategy allows us to use only 20% of the labelled data to reach a similar mAP@0.5 than a supervised approach using the whole training set.

Title: The current state of security -- Insights from the German software industry

• Authors: Timo Langstrof, Alex R. Sabau
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08436
• Pdf URL: https://arxiv.org/pdf/2402.08436
• Copy Paste: [[2402.08436]] The current state of security -- Insights from the German software industry(https://arxiv.org/abs/2402.08436)
• Keywords: secure, security
• Abstract: These days, software development and security go hand in hand. Numerous techniques and strategies are discussed in the literature that can be applied to guarantee the incorporation of security into the software development process. In this paper the main ideas of secure software development that have been discussed in the literature are outlined. Next, a dataset on implementation in practice is gathered through a qualitative interview research involving 20 companies. Trends and correlations in this dataset are found and contrasted with theoretical ideas from the literature. The results show that the organizations that were polled are placing an increasing focus on security. Although the techniques covered in the literature are being used in the real world, they are frequently not fully integrated into formal, standardized processes. The insights gained from our research lay the groundwork for future research, which can delve deeper into specific elements of these methods to enhance our understanding of their application in real-world scenarios.

Title: Camera Calibration through Geometric Constraints from Rotation and Projection Matrices

• Authors: Muhammad Waleed, Abdul Rauf, Murtaza Taj
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08437
• Pdf URL: https://arxiv.org/pdf/2402.08437
• Copy Paste: [[2402.08437]] Camera Calibration through Geometric Constraints from Rotation and Projection Matrices(https://arxiv.org/abs/2402.08437)
• Keywords: interpretability
• Abstract: The process of camera calibration involves estimating the intrinsic and extrinsic parameters, which are essential for accurately performing tasks such as 3D reconstruction, object tracking and augmented reality. In this work, we propose a novel constraints-based loss for measuring the intrinsic (focal length: $(f_x, f_y)$ and principal point: $(p_x, p_y)$) and extrinsic (baseline: ($b$), disparity: ($d$), translation: $(t_x, t_y, t_z)$, and rotation specifically pitch: $(\theta_p)$) camera parameters. Our novel constraints are based on geometric properties inherent in the camera model, including the anatomy of the projection matrix (vanishing points, image of world origin, axis planes) and the orthonormality of the rotation matrix. Thus we proposed a novel Unsupervised Geometric Constraint Loss (UGCL) via a multitask learning framework. Our methodology is a hybrid approach that employs the learning power of a neural network to estimate the desired parameters along with the underlying mathematical properties inherent in the camera projection matrix. This distinctive approach not only enhances the interpretability of the model but also facilitates a more informed learning process. Additionally, we introduce a new CVGL Camera Calibration dataset, featuring over 900 configurations of camera parameters, incorporating 63,600 image pairs that closely mirror real-world conditions. By training and testing on both synthetic and real-world datasets, our proposed approach demonstrates improvements across all parameters when compared to the state-of-the-art (SOTA) benchmarks. The code and the updated dataset can be found here: https://github.com/CVLABLUMS/CVGL-Camera-Calibration

Title: JeFaPaTo -- A joint toolbox for blinking analysis and facial features extraction

• Authors: Tim Büchner, Oliver Mothes, Orlando Guntinas-Lichius, Joachim Denzler
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08439
• Pdf URL: https://arxiv.org/pdf/2402.08439
• Copy Paste: [[2402.08439]] JeFaPaTo -- A joint toolbox for blinking analysis and facial features extraction(https://arxiv.org/abs/2402.08439)
• Keywords: extraction
• Abstract: Analyzing facial features and expressions is a complex task in computer vision. The human face is intricate, with significant shape, texture, and appearance variations. In medical contexts, facial structures that differ from the norm, such as those affected by paralysis, are particularly important to study and require precise analysis. One area of interest is the subtle movements involved in blinking, a process that is not yet fully understood and needs high-resolution, time-specific analysis for detailed understanding. However, a significant challenge is that many advanced computer vision techniques demand programming skills, making them less accessible to medical professionals who may not have these skills. The Jena Facial Palsy Toolbox (JeFaPaTo) has been developed to bridge this gap. It utilizes cutting-edge computer vision algorithms and offers a user-friendly interface for those without programming expertise. This toolbox is designed to make advanced facial analysis more accessible to medical experts, simplifying integration into their workflow. The state of the eye closure is of high interest to medical experts, e.g., in the context of facial palsy or Parkinson's disease. Due to facial nerve damage, the eye-closing process might be impaired and could lead to many undesirable side effects. Hence, more than a simple distinction between open and closed eyes is required for a detailed analysis. Factors such as duration, synchronicity, velocity, complete closure, the time between blinks, and frequency over time are highly relevant. Such detailed analysis could help medical experts better understand the blinking process, its deviations, and possible treatments for better eye care.

Title: Subgraphormer: Unifying Subgraph GNNs and Graph Transformers via Graph Products

• Authors: Guy Bar-Shalom, Beatrice Bevilacqua, Haggai Maron
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08450
• Pdf URL: https://arxiv.org/pdf/2402.08450
• Copy Paste: [[2402.08450]] Subgraphormer: Unifying Subgraph GNNs and Graph Transformers via Graph Products(https://arxiv.org/abs/2402.08450)
• Keywords: transformer
• Abstract: In the realm of Graph Neural Networks (GNNs), two exciting research directions have recently emerged: Subgraph GNNs and Graph Transformers. In this paper, we propose an architecture that integrates both approaches, dubbed Subgraphormer, which combines the enhanced expressive power, message-passing mechanisms, and aggregation schemes from Subgraph GNNs with attention and positional encodings, arguably the most important components in Graph Transformers. Our method is based on an intriguing new connection we reveal between Subgraph GNNs and product graphs, suggesting that Subgraph GNNs can be formulated as Message Passing Neural Networks (MPNNs) operating on a product of the graph with itself. We use this formulation to design our architecture: first, we devise an attention mechanism based on the connectivity of the product graph. Following this, we propose a novel and efficient positional encoding scheme for Subgraph GNNs, which we derive as a positional encoding for the product graph. Our experimental results demonstrate significant performance improvements over both Subgraph GNNs and Graph Transformers on a wide range of datasets.

Title: Lying Blindly: Bypassing ChatGPT's Safeguards to Generate Hard-to-Detect Disinformation Claims at Scale

• Authors: Freddy Heppell, Mehmet E. Bakir, Kalina Bontcheva
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08467
• Pdf URL: https://arxiv.org/pdf/2402.08467
• Copy Paste: [[2402.08467]] Lying Blindly: Bypassing ChatGPT's Safeguards to Generate Hard-to-Detect Disinformation Claims at Scale(https://arxiv.org/abs/2402.08467)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) become more proficient, their misuse in large-scale viral disinformation campaigns is a growing concern. This study explores the capability of ChatGPT to generate unconditioned claims about the war in Ukraine, an event beyond its knowledge cutoff, and evaluates whether such claims can be differentiated by human readers and automated tools from human-written ones. We compare war-related claims from ClaimReview, authored by IFCN-registered fact-checkers, and similar short-form content generated by ChatGPT. We demonstrate that ChatGPT can produce realistic, target-specific disinformation cheaply, fast, and at scale, and that these claims cannot be reliably distinguished by humans or existing automated tools.

Title: ROSpace: Intrusion Detection Dataset for a ROS2-Based Cyber-Physical System

• Authors: Tommaso Puccetti, Simone Nardi, Cosimo Cinquilli, Tommaso Zoppi, Andrea Ceccarelli
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08468
• Pdf URL: https://arxiv.org/pdf/2402.08468
• Copy Paste: [[2402.08468]] ROSpace: Intrusion Detection Dataset for a ROS2-Based Cyber-Physical System(https://arxiv.org/abs/2402.08468)
• Keywords: attack
• Abstract: Most of the intrusion detection datasets to research machine learning-based intrusion detection systems (IDSs) are devoted to cyber-only systems, and they typically collect data from one architectural layer. Additionally, often the attacks are generated in dedicated attack sessions, without reproducing the realistic alternation and overlap of normal and attack actions. We present a dataset for intrusion detection by performing penetration testing on an embedded cyber-physical system built over Robot Operating System 2 (ROS2). Features are monitored from three architectural layers: the Linux operating system, the network, and the ROS2 services. The dataset is structured as a time series and describes the expected behavior of the system and its response to ROS2-specific attacks: it repeatedly alternates periods of attack-free operation with periods when a specific attack is being performed. Noteworthy, this allows measuring the time to detect an attacker and the number of malicious activities performed before detection. Also, it allows training an intrusion detector to minimize both, by taking advantage of the numerous alternating periods of normal and attack operations.

Title: Intriguing Differences Between Zero-Shot and Systematic Evaluations of Vision-Language Transformer Models

• Authors: Shaeke Salman, Md Montasir Bin Shams, Xiuwen Liu, Lingjiong Zhu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08473
• Pdf URL: https://arxiv.org/pdf/2402.08473
• Copy Paste: [[2402.08473]] Intriguing Differences Between Zero-Shot and Systematic Evaluations of Vision-Language Transformer Models(https://arxiv.org/abs/2402.08473)
• Keywords: robust, transformer
• Abstract: Transformer-based models have dominated natural language processing and other areas in the last few years due to their superior (zero-shot) performance on benchmark datasets. However, these models are poorly understood due to their complexity and size. While probing-based methods are widely used to understand specific properties, the structures of the representation space are not systematically characterized; consequently, it is unclear how such models generalize and overgeneralize to new inputs beyond datasets. In this paper, based on a new gradient descent optimization method, we are able to explore the embedding space of a commonly used vision-language model. Using the Imagenette dataset, we show that while the model achieves over 99\% zero-shot classification performance, it fails systematic evaluations completely. Using a linear approximation, we provide a framework to explain the striking differences. We have also obtained similar results using a different model to support that our results are applicable to other transformer models with continuous inputs. We also propose a robust way to detect the modified images.

Title: P-Mamba: Marrying Perona Malik Diffusion with Mamba for Efficient Pediatric Echocardiographic Left Ventricular Segmentation

• Authors: Zi Ye, Tianxiang Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08506
• Pdf URL: https://arxiv.org/pdf/2402.08506
• Copy Paste: [[2402.08506]] P-Mamba: Marrying Perona Malik Diffusion with Mamba for Efficient Pediatric Echocardiographic Left Ventricular Segmentation(https://arxiv.org/abs/2402.08506)
• Keywords: diffusion, transformer, segmentation
• Abstract: In pediatric cardiology, the accurate and immediate assessment of cardiac function through echocardiography is important since it can determine whether urgent intervention is required in many emergencies. However, echocardiography is characterized by ambiguity and heavy background noise interference, bringing more difficulty to accurate segmentation. Present methods lack efficiency and are also prone to mistakenly segmenting some background noise areas as the left ventricular area due to noise disturbance. To relieve the two issues, we introduce P-Mamba for efficient pediatric echocardiographic left ventricular segmentation. Specifically, we turn to the recently proposed vision mamba layers in our vision mamba encoder branch to improve the computing and memory efficiency of our model while modeling global dependencies. In the other DWT-based PMD encoder branch, we devise DWT-based Perona-Malik Diffusion (PMD) Blocks that utilize PMD for noise suppression, while simultaneously preserving the local shape cues of the left ventricle. Leveraging the strengths of both the two encoder branches, P-Mamba achieves superior accuracy and efficiency to established models, such as vision transformers with quadratic and linear computational complexity. This innovative approach promises significant advancements in pediatric cardiac imaging and beyond.

Title: Fairness Auditing with Multi-Agent Collaboration

• Authors: Martijn de Vos, Akash Dhasade, Jade Garcia Bourrée, Anne-Marie Kermarrec, Erwan Le Merrer, Benoit Rottembourg, Gilles Tredan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08522
• Pdf URL: https://arxiv.org/pdf/2402.08522
• Copy Paste: [[2402.08522]] Fairness Auditing with Multi-Agent Collaboration(https://arxiv.org/abs/2402.08522)
• Keywords: fair
• Abstract: Existing work in fairness audits assumes that agents operate independently. In this paper, we consider the case of multiple agents auditing the same platform for different tasks. Agents have two levers: their collaboration strategy, with or without coordination beforehand, and their sampling method. We theoretically study their interplay when agents operate independently or collaborate. We prove that, surprisingly, coordination can sometimes be detrimental to audit accuracy, whereas uncoordinated collaboration generally yields good results. Experimentation on real-world datasets confirms this observation, as the audit accuracy of uncoordinated collaboration matches that of collaborative optimal sampling.

Title: Approximately Piecewise E(3) Equivariant Point Networks

• Authors: Matan Atzmon, Jiahui Huang, Francis Williams, Or Litany
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08529
• Pdf URL: https://arxiv.org/pdf/2402.08529
• Copy Paste: [[2402.08529]] Approximately Piecewise E(3) Equivariant Point Networks(https://arxiv.org/abs/2402.08529)
• Keywords: segmentation
• Abstract: Integrating a notion of symmetry into point cloud neural networks is a provably effective way to improve their generalization capability. Of particular interest are $E(3)$ equivariant point cloud networks where Euclidean transformations applied to the inputs are preserved in the outputs. Recent efforts aim to extend networks that are $E(3)$ equivariant, to accommodate inputs made of multiple parts, each of which exhibits local $E(3)$ symmetry. In practical settings, however, the partitioning into individually transforming regions is unknown a priori. Errors in the partition prediction would unavoidably map to errors in respecting the true input symmetry. Past works have proposed different ways to predict the partition, which may exhibit uncontrolled errors in their ability to maintain equivariance to the actual partition. To this end, we introduce APEN: a general framework for constructing approximate piecewise-$E(3)$ equivariant point networks. Our primary insight is that functions that are equivariant with respect to a finer partition will also maintain equivariance in relation to the true partition. Leveraging this observation, we propose a design where the equivariance approximation error at each layers can be bounded solely in terms of (i) uncertainty quantification of the partition prediction, and (ii) bounds on the probability of failing to suggest a proper subpartition of the ground truth one. We demonstrate the effectiveness of APEN using two data types exemplifying part-based symmetry: (i) real-world scans of room scenes containing multiple furniture-type objects; and, (ii) human motions, characterized by articulated parts exhibiting rigid movement. Our empirical results demonstrate the advantage of integrating piecewise $E(3)$ symmetry into network design, showing a distinct improvement in generalization compared to prior works for both classification and segmentation tasks.

Title: A Distributional Analogue to the Successor Representation

• Authors: Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Yunhao Tang, André Barreto, Will Dabney, Marc G. Bellemare, Mark Rowland
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.08530
• Pdf URL: https://arxiv.org/pdf/2402.08530
• Copy Paste: [[2402.08530]] A Distributional Analogue to the Successor Representation(https://arxiv.org/abs/2402.08530)
• Keywords: generative
• Abstract: This paper contributes a new approach for distributional reinforcement learning which elucidates a clean separation of transition structure and reward in the learning process. Analogous to how the successor representation (SR) describes the expected consequences of behaving according to a given policy, our distributional successor measure (SM) describes the distributional consequences of this behaviour. We formulate the distributional SM as a distribution over distributions and provide theory connecting it with distributional and model-based reinforcement learning. Moreover, we propose an algorithm that learns the distributional SM from data by minimizing a two-level maximum mean discrepancy. Key to our method are a number of algorithmic techniques that are independently valuable for learning generative models of state. As an illustration of the usefulness of the distributional SM, we show that it enables zero-shot risk-sensitive policy evaluation in a way that was not previously possible.

Title: Generative VS non-Generative Models in Engineering Shape Optimization

• Authors: Muhammad Usama, Zahid Masood, Shahroz Khan, Konstantinos Kostas, Panagiotis Kaklis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08540
• Pdf URL: https://arxiv.org/pdf/2402.08540
• Copy Paste: [[2402.08540]] Generative VS non-Generative Models in Engineering Shape Optimization(https://arxiv.org/abs/2402.08540)
• Keywords: robust, generative
• Abstract: In this work, we perform a systematic comparison of the effectiveness and efficiency of generative and non-generative models in constructing design spaces for novel and efficient design exploration and shape optimization. We apply these models in the case of airfoil/hydrofoil design and conduct the comparison on the resulting design spaces. A conventional Generative Adversarial Network (GAN) and a state-of-the-art generative model, the Performance-Augmented Diverse Generative Adversarial Network (PaDGAN), are juxtaposed with a linear non-generative model based on the coupling of the Karhunen-Lo\`eve Expansion and a physics-informed Shape Signature Vector (SSV-KLE). The comparison demonstrates that, with an appropriate shape encoding and a physics-augmented design space, non-generative models have the potential to cost-effectively generate high-performing valid designs with enhanced coverage of the design space. In this work, both approaches are applied to two large foil profile datasets comprising real-world and artificial designs generated through either a profile-generating parametric model or deep-learning approach. These datasets are further enriched with integral properties of their members' shapes as well as physics-informed parameters. Our results illustrate that the design spaces constructed by the non-generative model outperform the generative model in terms of design validity, generating robust latent spaces with none or significantly fewer invalid designs when compared to generative models. We aspire that these findings will aid the engineering design community in making informed decisions when constructing designs spaces for shape optimization, as we have show that under certain conditions computationally inexpensive approaches can closely match or even outperform state-of-the art generative models.

Title: Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases

• Authors: Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, Dacheng Tao
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08552
• Pdf URL: https://arxiv.org/pdf/2402.08552
• Copy Paste: [[2402.08552]] Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases(https://arxiv.org/abs/2402.08552)
• Keywords: diffusion, generative
• Abstract: Bridging the gap between diffusion models and human preferences is crucial for their integration into practical generative workflows. While optimizing downstream reward models has emerged as a promising alignment strategy, concerns arise regarding the risk of excessive optimization with learned reward models, which potentially compromises ground-truth performance. In this work, we confront the reward overoptimization problem in diffusion model alignment through the lenses of both inductive and primacy biases. We first identify the divergence of current methods from the temporal inductive bias inherent in the multi-step denoising process of diffusion models as a potential source of overoptimization. Then, we surprisingly discover that dormant neurons in our critic model act as a regularization against overoptimization, while active neurons reflect primacy bias in this setting. Motivated by these observations, we propose Temporal Diffusion Policy Optimization with critic active neuron Reset (TDPO-R), a policy gradient algorithm that exploits the temporal inductive bias of intermediate timesteps, along with a novel reset strategy that targets active neurons to counteract the primacy bias. Empirical results demonstrate the superior efficacy of our algorithms in mitigating reward overoptimization.

Title: Higher Layers Need More LoRA Experts

• Authors: Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, VS Subrahmanian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08562
• Pdf URL: https://arxiv.org/pdf/2402.08562
• Copy Paste: [[2402.08562]] Higher Layers Need More LoRA Experts(https://arxiv.org/abs/2402.08562)
• Keywords: transformer, large language model
• Abstract: Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA) offer training efficiency on Large Language Models, but their impact on model performance remains limited. Recent efforts integrate LoRA and Mixture-of-Experts (MoE) to improve the performance of PEFT methods. Despite promising results, research on improving the efficiency of LoRA with MoE is still in its early stages. Recent studies have shown that experts in the MoE architecture have different strengths and also exhibit some redundancy. Does this statement also apply to parameter-efficient MoE? In this paper, we introduce a novel parameter-efficient MoE method, \textit{\textbf{M}oE-L\textbf{o}RA with \textbf{L}ayer-wise Expert \textbf{A}llocation (MoLA)} for Transformer-based models, where each model layer has the flexibility to employ a varying number of LoRA experts. We investigate several architectures with varying layer-wise expert configurations. Experiments on six well-known NLP and commonsense QA benchmarks demonstrate that MoLA achieves equal or superior performance compared to all baselines. We find that allocating more LoRA experts to higher layers further enhances the effectiveness of models with a certain number of experts in total. With much fewer parameters, this allocation strategy outperforms the setting with the same number of experts in every layer. This work can be widely used as a plug-and-play parameter-efficient tuning approach for various applications. The code is available at https://github.com/GCYZSL/MoLA.

Title: Denoising Diffusion Restoration Tackles Forward and Inverse Problems for the Laplace Operator

• Authors: Amartya Mukherjee, Melissa M. Stadt, Lena Podina, Mohammad Kohandel, Jun Liu
• Subjects: cs.LG, cs.CV, math.AP
• Abstract URL: https://arxiv.org/abs/2402.08563
• Pdf URL: https://arxiv.org/pdf/2402.08563
• Copy Paste: [[2402.08563]] Denoising Diffusion Restoration Tackles Forward and Inverse Problems for the Laplace Operator(https://arxiv.org/abs/2402.08563)
• Keywords: diffusion, generative
• Abstract: Diffusion models have emerged as a promising class of generative models that map noisy inputs to realistic images. More recently, they have been employed to generate solutions to partial differential equations (PDEs). However, they still struggle with inverse problems in the Laplacian operator, for instance, the Poisson equation, because the eigenvalues that are large in magnitude amplify the measurement noise. This paper presents a novel approach for the inverse and forward solution of PDEs through the use of denoising diffusion restoration models (DDRM). DDRMs were used in linear inverse problems to restore original clean signals by exploiting the singular value decomposition (SVD) of the linear operator. Equivalently, we present an approach to restore the solution and the parameters in the Poisson equation by exploiting the eigenvalues and the eigenfunctions of the Laplacian operator. Our results show that using denoising diffusion restoration significantly improves the estimation of the solution and parameters. Our research, as a result, pioneers the integration of diffusion models with the principles of underlying physics to solve PDEs.

Title: Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast

• Authors: Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, Min Lin
• Subjects: cs.CL, cs.CR, cs.CV, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2402.08567
• Pdf URL: https://arxiv.org/pdf/2402.08567
• Copy Paste: [[2402.08567]] Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast(https://arxiv.org/abs/2402.08567)
• Keywords: defense, large language model
• Abstract: A multimodal large language model (MLLM) agent can receive instructions, capture images, retrieve histories from memory, and decide which tools to use. Nonetheless, red-teaming efforts have revealed that adversarial images/prompts can jailbreak an MLLM and cause unaligned behaviors. In this work, we report an even more severe safety issue in multi-agent environments, referred to as infectious jailbreak. It entails the adversary simply jailbreaking a single agent, and without any further intervention from the adversary, (almost) all agents will become infected exponentially fast and exhibit harmful behaviors. To validate the feasibility of infectious jailbreak, we simulate multi-agent environments containing up to one million LLaVA-1.5 agents, and employ randomized pair-wise chat as a proof-of-concept instantiation for multi-agent interaction. Our results show that feeding an (infectious) adversarial image into the memory of any randomly chosen agent is sufficient to achieve infectious jailbreak. Finally, we derive a simple principle for determining whether a defense mechanism can provably restrain the spread of infectious jailbreak, but how to design a practical defense that meets this principle remains an open question to investigate. Our project page is available at https://sail-sg.github.io/Agent-Smith/.

Title: Glass Segmentation with Multi Scales and Primary Prediction Guiding

• Authors: Zhiyu Xu, Qingliang Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08571
• Pdf URL: https://arxiv.org/pdf/2402.08571
• Copy Paste: [[2402.08571]] Glass Segmentation with Multi Scales and Primary Prediction Guiding(https://arxiv.org/abs/2402.08571)
• Keywords: segmentation
• Abstract: Glass-like objects can be seen everywhere in our daily life which are very hard for existing methods to segment them. The properties of transparencies pose great challenges of detecting them from the chaotic background and the vague separation boundaries further impede the acquisition of their exact contours. Moving machines which ignore glasses have great risks of crashing into transparent barriers or difficulties in analysing objects reflected in the mirror, thus it is of substantial significance to accurately locate glass-like objects and completely figure out their contours. In this paper, inspired by the scale integration strategy and the refinement method, we proposed a brand-new network, named as MGNet, which consists of a Fine-Rescaling and Merging module (FRM) to improve the ability to extract spatially relationship and a Primary Prediction Guiding module (PPG) to better mine the leftover semantics from the fused features. Moreover, we supervise the model with a novel loss function with the uncertainty-aware loss to produce high-confidence segmentation maps. Unlike the existing glass segmentation models that must be trained on different settings with respect to varied datasets, our model are trained under consistent settings and has achieved superior performance on three popular public datasets. Code is available at

Title: Two Tales of Single-Phase Contrastive Hebbian Learning

• Authors: Rasmus Kjær Høier, Christopher Zach
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.08573
• Pdf URL: https://arxiv.org/pdf/2402.08573
• Copy Paste: [[2402.08573]] Two Tales of Single-Phase Contrastive Hebbian Learning(https://arxiv.org/abs/2402.08573)
• Keywords: robust
• Abstract: The search for "biologically plausible" learning algorithms has converged on the idea of representing gradients as activity differences. However, most approaches require a high degree of synchronization (distinct phases during learning) and introduce substantial computational overhead, which raises doubts regarding their biological plausibility as well as their potential utility for neuromorphic computing. Furthermore, they commonly rely on applying infinitesimal perturbations (nudges) to output units, which is impractical in noisy environments. Recently it has been shown that by modelling artificial neurons as dyads with two oppositely nudged compartments, it is possible for a fully local learning algorithm named ``dual propagation'' to bridge the performance gap to backpropagation, without requiring separate learning phases or infinitesimal nudging. However, the algorithm has the drawback that its numerical stability relies on symmetric nudging, which may be restrictive in biological and analog implementations. In this work we first provide a solid foundation for the objective underlying the dual propagation method, which also reveals a surprising connection with adversarial robustness. Second, we demonstrate how dual propagation is related to a particular adjoint state method, which is stable regardless of asymmetric nudging.

Title: Test-Time Backdoor Attacks on Multimodal Large Language Models

• Authors: Dong Lu, Tianyu Pang, Chao Du, Qian Liu, Xianjun Yang, Min Lin
• Subjects: cs.CL, cs.CR, cs.CV, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.08577
• Pdf URL: https://arxiv.org/pdf/2402.08577
• Copy Paste: [[2402.08577]] Test-Time Backdoor Attacks on Multimodal Large Language Models(https://arxiv.org/abs/2402.08577)
• Keywords: attack, large language model
• Abstract: Backdoor attacks are commonly executed by contaminating training data, such that a trigger can activate predetermined harmful effects during the test phase. In this work, we present AnyDoor, a test-time backdoor attack against multimodal large language models (MLLMs), which involves injecting the backdoor into the textual modality using adversarial test images (sharing the same universal perturbation), without requiring access to or modification of the training data. AnyDoor employs similar techniques used in universal adversarial attacks, but distinguishes itself by its ability to decouple the timing of setup and activation of harmful effects. In our experiments, we validate the effectiveness of AnyDoor against popular MLLMs such as LLaVA-1.5, MiniGPT-4, InstructBLIP, and BLIP-2, as well as provide comprehensive ablation studies. Notably, because the backdoor is injected by a universal perturbation, AnyDoor can dynamically change its backdoor trigger prompts/harmful effects, exposing a new challenge for defending against backdoor attacks. Our project page is available at https://sail-sg.github.io/AnyDoor/.

Title: FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharing

• Authors: Yongzhe Jia, Xuyun Zhang, Amin Beheshti, Wanchun Dou
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.08578
• Pdf URL: https://arxiv.org/pdf/2402.08578
• Copy Paste: [[2402.08578]] FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharing(https://arxiv.org/abs/2402.08578)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) has emerged as a promising solution in Edge Computing (EC) environments to process the proliferation of data generated by edge devices. By collaboratively optimizing the global machine learning models on distributed edge devices, FL circumvents the need for transmitting raw data and enhances user privacy. Despite practical successes, FL still confronts significant challenges including constrained edge device resources, multiple tasks deployment, and data heterogeneity. However, existing studies focus on mitigating the FL training costs of each single task whereas neglecting the resource consumption across multiple tasks in heterogeneous FL scenarios. In this paper, we propose Heterogeneous Federated Learning with Local Parameter Sharing (FedLPS) to fill this gap. FedLPS leverages principles from transfer learning to facilitate the deployment of multiple tasks on a single device by dividing the local model into a shareable encoder and task-specific encoders. To further reduce resource consumption, a channel-wise model pruning algorithm that shrinks the footprint of local models while accounting for both data and system heterogeneity is employed in FedLPS. Additionally, a novel heterogeneous model aggregation algorithm is proposed to aggregate the heterogeneous predictors in FedLPS. We implemented the proposed FedLPS on a real FL platform and compared it with state-of-the-art (SOTA) FL frameworks. The experimental results on five popular datasets and two modern DNN models illustrate that the proposed FedLPS significantly outperforms the SOTA FL frameworks by up to 4.88% and reduces the computational resource consumption by 21.3%. Our code is available at:https://github.com/jyzgh/FedLPS.

Title: FESS Loss: Feature-Enhanced Spatial Segmentation Loss for Optimizing Medical Image Analysis

• Authors: Charulkumar Chodvadiya, Navyansh Mahla, Kinshuk Gaurav Singh, Kshitij Sharad Jadhav
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08582
• Pdf URL: https://arxiv.org/pdf/2402.08582
• Copy Paste: [[2402.08582]] FESS Loss: Feature-Enhanced Spatial Segmentation Loss for Optimizing Medical Image Analysis(https://arxiv.org/abs/2402.08582)
• Keywords: segmentation
• Abstract: Medical image segmentation is a critical process in the field of medical imaging, playing a pivotal role in diagnosis, treatment, and research. It involves partitioning of an image into multiple regions, representing distinct anatomical or pathological structures. Conventional methods often grapple with the challenge of balancing spatial precision and comprehensive feature representation due to their reliance on traditional loss functions. To overcome this, we propose Feature-Enhanced Spatial Segmentation Loss (FESS Loss), that integrates the benefits of contrastive learning (which extracts intricate features, particularly in the nuanced domain of medical imaging) with the spatial accuracy inherent in the Dice loss. The objective is to augment both spatial precision and feature-based representation in the segmentation of medical images. FESS Loss signifies a notable advancement, offering a more accurate and refined segmentation process, ultimately contributing to heightened precision in the analysis of medical images. Further, FESS loss demonstrates superior performance in limited annotated data availability scenarios often present in the medical domain.

Title: Faster Repeated Evasion Attacks in Tree Ensembles

• Authors: Lorenzo Cascioli, Laurens Devos, Ondřej Kuželka, Jesse Davis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08586
• Pdf URL: https://arxiv.org/pdf/2402.08586
• Copy Paste: [[2402.08586]] Faster Repeated Evasion Attacks in Tree Ensembles(https://arxiv.org/abs/2402.08586)
• Keywords: attack
• Abstract: Tree ensembles are one of the most widely used model classes. However, these models are susceptible to adversarial examples, i.e., slightly perturbed examples that elicit a misprediction. There has been significant research on designing approaches to construct such examples for tree ensembles. But this is a computationally challenging problem that often must be solved a large number of times (e.g., for all examples in a training set). This is compounded by the fact that current approaches attempt to find such examples from scratch. In contrast, we exploit the fact that multiple similar problems are being solved. Specifically, our approach exploits the insight that adversarial examples for tree ensembles tend to perturb a consistent but relatively small set of features. We show that we can quickly identify this set of features and use this knowledge to speedup constructing adversarial examples.

Title: Graph Feature Preprocessor: Real-time Extraction of Subgraph-based Features from Transaction Graphs

• Authors: Jovan Blanuša, Maximo Cravero Baraja, Andreea Anghel, Luc von Niederhäusern, Erik Altman, Haris Pozidis, Kubilay Atasu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08593
• Pdf URL: https://arxiv.org/pdf/2402.08593
• Copy Paste: [[2402.08593]] Graph Feature Preprocessor: Real-time Extraction of Subgraph-based Features from Transaction Graphs(https://arxiv.org/abs/2402.08593)
• Keywords: extraction
• Abstract: In this paper, we present "Graph Feature Preprocessor", a software library for detecting typical money laundering and fraud patterns in financial transaction graphs in real time. These patterns are used to produce a rich set of transaction features for downstream machine learning training and inference tasks such as money laundering detection. We show that our enriched transaction features dramatically improve the prediction accuracy of gradient-boosting-based machine learning models. Our library exploits multicore parallelism, maintains a dynamic in-memory graph, and efficiently mines subgraph patterns in the incoming transaction stream, which enables it to be operated in a streaming manner. We evaluate our library using highly-imbalanced synthetic anti-money laundering (AML) and real-life Ethereum phishing datasets. In these datasets, the proportion of illicit transactions is very small, which makes the learning process challenging. Our solution, which combines our Graph Feature Preprocessor and gradient-boosting-based machine learning models, is able to detect these illicit transactions with higher minority-class F1 scores than standard graph neural networks. In addition, the end-to-end throughput rate of our solution executed on a multicore CPU outperforms the graph neural network baselines executed on a powerful V100 GPU. Overall, the combination of high accuracy, a high throughput rate, and low latency of our solution demonstrates the practical value of our library in real-world applications. Graph Feature Preprocessor has been integrated into IBM mainframe software products, namely "IBM Cloud Pak for Data on Z" and "AI Toolkit for IBM Z and LinuxONE".

Title: Latent Inversion with Timestep-aware Sampling for Training-free Non-rigid Editing

• Authors: Yunji Jung, Seokju Lee, Tair Djanibekov, Hyunjung Shim, Jongchul Ye
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08601
• Pdf URL: https://arxiv.org/pdf/2402.08601
• Copy Paste: [[2402.08601]] Latent Inversion with Timestep-aware Sampling for Training-free Non-rigid Editing(https://arxiv.org/abs/2402.08601)
• Keywords: diffusion
• Abstract: Text-guided non-rigid editing involves complex edits for input images, such as changing motion or compositions within their surroundings. Since it requires manipulating the input structure, existing methods often struggle with preserving object identity and background, particularly when combined with Stable Diffusion. In this work, we propose a training-free approach for non-rigid editing with Stable Diffusion, aimed at improving the identity preservation quality without compromising editability. Our approach comprises three stages: text optimization, latent inversion, and timestep-aware text injection sampling. Inspired by the recent success of Imagic, we employ their text optimization for smooth editing. Then, we introduce latent inversion to preserve the input image's identity without additional model fine-tuning. To fully utilize the input reconstruction ability of latent inversion, we suggest timestep-aware text inject sampling. This effectively retains the structure of the input image by injecting the source text prompt in early sampling steps and then transitioning to the target prompt in subsequent sampling steps. This strategic approach seamlessly harmonizes with text optimization, facilitating complex non-rigid edits to the input without losing the original identity. We demonstrate the effectiveness of our method in terms of identity preservation, editability, and aesthetic quality through extensive experiments.

Title: A Cost-Sensitive Transformer Model for Prognostics Under Highly Imbalanced Industrial Data

• Authors: Ali Beikmohammadi, Mohammad Hosein Hamian, Neda Khoeyniha, Tony Lindgren, Olof Steinert, Sindri Magnússon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08611
• Pdf URL: https://arxiv.org/pdf/2402.08611
• Copy Paste: [[2402.08611]] A Cost-Sensitive Transformer Model for Prognostics Under Highly Imbalanced Industrial Data(https://arxiv.org/abs/2402.08611)
• Keywords: transformer
• Abstract: The rapid influx of data-driven models into the industrial sector has been facilitated by the proliferation of sensor technology, enabling the collection of vast quantities of data. However, leveraging these models for failure detection and prognosis poses significant challenges, including issues like missing values and class imbalances. Moreover, the cost sensitivity associated with industrial operations further complicates the application of conventional models in this context. This paper introduces a novel cost-sensitive transformer model developed as part of a systematic workflow, which also integrates a hybrid resampler and a regression-based imputer. After subjecting our approach to rigorous testing using the APS failure dataset from Scania trucks and the SECOM dataset, we observed a substantial enhancement in performance compared to state-of-the-art methods. Moreover, we conduct an ablation study to analyze the contributions of different components in our proposed method. Our findings highlight the potential of our method in addressing the unique challenges of failure prediction in industrial settings, thereby contributing to enhanced reliability and efficiency in industrial operations.

Title: CaPS: Collaborative and Private Synthetic Data Generation from Distributed Sources

• Authors: Sikha Pentyala, Mayana Pereira, Martine De Cock
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08614
• Pdf URL: https://arxiv.org/pdf/2402.08614
• Copy Paste: [[2402.08614]] CaPS: Collaborative and Private Synthetic Data Generation from Distributed Sources(https://arxiv.org/abs/2402.08614)
• Keywords: secure, privacy, generative
• Abstract: Data is the lifeblood of the modern world, forming a fundamental part of AI, decision-making, and research advances. With increase in interest in data, governments have taken important steps towards a regulated data world, drastically impacting data sharing and data usability and resulting in massive amounts of data confined within the walls of organizations. While synthetic data generation (SDG) is an appealing solution to break down these walls and enable data sharing, the main drawback of existing solutions is the assumption of a trusted aggregator for generative model training. Given that many data holders may not want to, or be legally allowed to, entrust a central entity with their raw data, we propose a framework for the collaborative and private generation of synthetic tabular data from distributed data holders. Our solution is general, applicable to any marginal-based SDG, and provides input privacy by replacing the trusted aggregator with secure multi-party computation (MPC) protocols and output privacy via differential privacy (DP). We demonstrate the applicability and scalability of our approach for the state-of-the-art select-measure-generate SDG algorithms MWEM+PGM and AIM.

Title: Knowledge Editing on Black-box Large Language Models

• Authors: Xiaoshuai Song, Zhengyang Wang, Keqing He, Guanting Dong, Jinxu Zhao, Weiran Xu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08631
• Pdf URL: https://arxiv.org/pdf/2402.08631
• Copy Paste: [[2402.08631]] Knowledge Editing on Black-box Large Language Models(https://arxiv.org/abs/2402.08631)
• Keywords: privacy, large language model
• Abstract: Knowledge editing (KE) aims to efficiently and precisely modify the behavior of large language models (LLMs) to update specific knowledge without negatively influencing other knowledge. Current research primarily focuses on white-box LLMs editing, overlooking an important scenario: black-box LLMs editing, where LLMs are accessed through interfaces and only textual output is available. To address the limitations of existing evaluations that are not inapplicable to black-box LLM editing and lack comprehensiveness, we propose a multi-perspective evaluation framework, incorporating the assessment of style retention for the first time. To tackle privacy leaks of editing data and style over-editing in current methods, we introduce a novel postEdit framework, resolving privacy concerns through downstream post-processing and maintaining textual style consistency via fine-grained editing to original responses. Experiments and analysis on two benchmarks demonstrate that postEdit outperforms all baselines and achieves strong generalization, especially with huge improvements on style retention (average $+20.82\%\uparrow$).

Title: SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 14 Languages

• Authors: Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Abinew Ali Ayele, Pavan Baswani, Meriem Beloucif, Chris Biemann, Sofia Bourhim, Christine De Kock, Genet Shanko Dekebo, Oumaima Hourrane, Gopichand Kanumolu, Lokesh Madasu, Samuel Rutunda, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Hailegnaw Getaneh Tilaye, Krishnapriya Vishnubhotla, Genta Winata, Seid Muhie Yimam, Saif M. Mohammad
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08638
• Pdf URL: https://arxiv.org/pdf/2402.08638
• Copy Paste: [[2402.08638]] SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 14 Languages(https://arxiv.org/abs/2402.08638)
• Keywords: large language model
• Abstract: Exploring and quantifying semantic relatedness is central to representing language. It holds significant implications across various NLP tasks, including offering insights into the capabilities and performance of Large Language Models (LLMs). While earlier NLP research primarily focused on semantic similarity, often within the English language context, we instead investigate the broader phenomenon of semantic relatedness. In this paper, we present SemRel, a new semantic relatedness dataset collection annotated by native speakers across 14 languages:Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia -- regions characterised by a relatively limited availability of NLP resources. Each instance in the SemRel datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. The scores are obtained using a comparative annotation framework. We describe the data collection and annotation processes, related challenges when building the datasets, and their impact and utility in NLP. We further report experiments for each language and across the different languages.

Title: Peeking Behind the Curtains of Residual Learning

• Authors: Tunhou Zhang, Feng Yan, Hai Li, Yiran Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08645
• Pdf URL: https://arxiv.org/pdf/2402.08645
• Copy Paste: [[2402.08645]] Peeking Behind the Curtains of Residual Learning(https://arxiv.org/abs/2402.08645)
• Keywords: transformer
• Abstract: The utilization of residual learning has become widespread in deep and scalable neural nets. However, the fundamental principles that contribute to the success of residual learning remain elusive, thus hindering effective training of plain nets with depth scalability. In this paper, we peek behind the curtains of residual learning by uncovering the "dissipating inputs" phenomenon that leads to convergence failure in plain neural nets: the input is gradually compromised through plain layers due to non-linearities, resulting in challenges of learning feature representations. We theoretically demonstrate how plain neural nets degenerate the input to random noise and emphasize the significance of a residual connection that maintains a better lower bound of surviving neurons as a solution. With our theoretical discoveries, we propose "The Plain Neural Net Hypothesis" (PNNH) that identifies the internal path across non-linear layers as the most critical part in residual learning, and establishes a paradigm to support the training of deep plain neural nets devoid of residual connections. We thoroughly evaluate PNNH-enabled CNN architectures and Transformers on popular vision benchmarks, showing on-par accuracy, up to 0.3% higher training throughput, and 2x better parameter efficiency compared to ResNets and vision Transformers.

Title: Generating Universal Adversarial Perturbations for Quantum Classifiers

• Authors: Gautham Anil, Vishnu Vinod, Apurva Narayan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08648
• Pdf URL: https://arxiv.org/pdf/2402.08648
• Copy Paste: [[2402.08648]] Generating Universal Adversarial Perturbations for Quantum Classifiers(https://arxiv.org/abs/2402.08648)
• Keywords: attack, generative
• Abstract: Quantum Machine Learning (QML) has emerged as a promising field of research, aiming to leverage the capabilities of quantum computing to enhance existing machine learning methodologies. Recent studies have revealed that, like their classical counterparts, QML models based on Parametrized Quantum Circuits (PQCs) are also vulnerable to adversarial attacks. Moreover, the existence of Universal Adversarial Perturbations (UAPs) in the quantum domain has been demonstrated theoretically in the context of quantum classifiers. In this work, we introduce QuGAP: a novel framework for generating UAPs for quantum classifiers. We conceptualize the notion of additive UAPs for PQC-based classifiers and theoretically demonstrate their existence. We then utilize generative models (QuGAP-A) to craft additive UAPs and experimentally show that quantum classifiers are susceptible to such attacks. Moreover, we formulate a new method for generating unitary UAPs (QuGAP-U) using quantum generative models and a novel loss function based on fidelity constraints. We evaluate the performance of the proposed framework and show that our method achieves state-of-the-art misclassification rates, while maintaining high fidelity between legitimate and adversarial samples.

Title: SAGMAN: Stability Analysis of Graph Neural Networks on the Manifolds

• Authors: Wuxinlin Cheng, Chenhui Deng, Ali Aghdaei, Zhiru Zhang, Zhuo Feng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08653
• Pdf URL: https://arxiv.org/pdf/2402.08653
• Copy Paste: [[2402.08653]] SAGMAN: Stability Analysis of Graph Neural Networks on the Manifolds(https://arxiv.org/abs/2402.08653)
• Keywords: attack
• Abstract: Modern graph neural networks (GNNs) can be sensitive to changes in the input graph structure and node features, potentially resulting in unpredictable behavior and degraded performance. In this work, we introduce a spectral framework known as SAGMAN for examining the stability of GNNs. This framework assesses the distance distortions that arise from the nonlinear mappings of GNNs between the input and output manifolds: when two nearby nodes on the input manifold are mapped (through a GNN model) to two distant ones on the output manifold, it implies a large distance distortion and thus a poor GNN stability. We propose a distance-preserving graph dimension reduction (GDR) approach that utilizes spectral graph embedding and probabilistic graphical models (PGMs) to create low-dimensional input/output graph-based manifolds for meaningful stability analysis. Our empirical evaluations show that SAGMAN effectively assesses the stability of each node when subjected to various edge or feature perturbations, offering a scalable approach for evaluating the stability of GNNs, extending to applications within recommendation systems. Furthermore, we illustrate its utility in downstream tasks, notably in enhancing GNN stability and facilitating adversarial targeted attacks.

Title: Learning Continuous 3D Words for Text-to-Image Generation

• Authors: Ta-Ying Cheng, Matheus Gadelha, Thibault Groueix, Matthew Fisher, Radomir Mech, Andrew Markham, Niki Trigoni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08654
• Pdf URL: https://arxiv.org/pdf/2402.08654
• Copy Paste: [[2402.08654]] Learning Continuous 3D Words for Text-to-Image Generation(https://arxiv.org/abs/2402.08654)
• Keywords: diffusion, generative
• Abstract: Current controls over diffusion models (e.g., through text or ControlNet) for image generation fall short in recognizing abstract, continuous attributes like illumination direction or non-rigid shape change. In this paper, we present an approach for allowing users of text-to-image models to have fine-grained control of several attributes in an image. We do this by engineering special sets of input tokens that can be transformed in a continuous manner -- we call them Continuous 3D Words. These attributes can, for example, be represented as sliders and applied jointly with text prompts for fine-grained control over image generation. Given only a single mesh and a rendering engine, we show that our approach can be adopted to provide continuous user control over several 3D-aware attributes, including time-of-day illumination, bird wing orientation, dollyzoom effect, and object poses. Our method is capable of conditioning image creation with multiple Continuous 3D Words and text descriptions simultaneously while adding no overhead to the generative process. Project Page: https://ttchengab.github.io/continuous_3d_words

Title: NeuroBench: An Open-Source Benchmark Framework for the Standardization of Methodology in Brainwave-based Authentication Research

• Authors: Avinash Kumar Chaurasia (a), Matin Fallahi (b), Thorsten Strufe (b), Philipp Terhörst (a), Patricia Arias Cabarcos (a and b) ((a) University of Paderborn, (b) KASTEL Security Research Labs - KIT)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08656
• Pdf URL: https://arxiv.org/pdf/2402.08656
• Copy Paste: [[2402.08656]] NeuroBench: An Open-Source Benchmark Framework for the Standardization of Methodology in Brainwave-based Authentication Research(https://arxiv.org/abs/2402.08656)
• Keywords: attack, robust, biometric, fair
• Abstract: Biometric systems based on brain activity have been proposed as an alternative to passwords or to complement current authentication techniques. By leveraging the unique brainwave patterns of individuals, these systems offer the possibility of creating authentication solutions that are resistant to theft, hands-free, accessible, and potentially even revocable. However, despite the growing stream of research in this area, faster advance is hindered by reproducibility problems. Issues such as the lack of standard reporting schemes for performance results and system configuration, or the absence of common evaluation benchmarks, make comparability and proper assessment of different biometric solutions challenging. Further, barriers are erected to future work when, as so often, source code is not published open access. To bridge this gap, we introduce NeuroBench, a flexible open source tool to benchmark brainwave-based authentication models. It incorporates nine diverse datasets, implements a comprehensive set of pre-processing parameters and machine learning algorithms, enables testing under two common adversary models (known vs unknown attacker), and allows researchers to generate full performance reports and visualizations. We use NeuroBench to investigate the shallow classifiers and deep learning-based approaches proposed in the literature, and to test robustness across multiple sessions. We observe a 37.6\% reduction in Equal Error Rate (EER) for unknown attacker scenarios (typically not tested in the literature), and we highlight the importance of session variability to brainwave authentication. All in all, our results demonstrate the viability and relevance of NeuroBench in streamlining fair comparisons of algorithms, thereby furthering the advancement of brainwave-based authentication through robust methodological practices.

Title: PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs

• Authors: Michael Dorkenwald, Nimrod Barazani, Cees G. M. Snoek, Yuki M. Asano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08657
• Pdf URL: https://arxiv.org/pdf/2402.08657
• Copy Paste: [[2402.08657]] PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs(https://arxiv.org/abs/2402.08657)
• Keywords: large language model
• Abstract: Vision-Language Models (VLMs), such as Flamingo and GPT-4V, have shown immense potential by integrating large language models with vision systems. Nevertheless, these models face challenges in the fundamental computer vision task of object localisation, due to their training on multimodal data containing mostly captions without explicit spatial grounding. While it is possible to construct custom, supervised training pipelines with bounding box annotations that integrate with VLMs, these result in specialized and hard-to-scale models. In this paper, we aim to explore the limits of caption-based VLMs and instead propose to tackle the challenge in a simpler manner by i) keeping the weights of a caption-based VLM frozen and ii) not using any supervised detection data. To this end, we introduce an input-agnostic Positional Insert (PIN), a learnable spatial prompt, containing a minimal set of parameters that are slid inside the frozen VLM, unlocking object localisation capabilities. Our PIN module is trained with a simple next-token prediction task on synthetic data without requiring the introduction of new output heads. Our experiments demonstrate strong zero-shot localisation performances on a variety of images, including Pascal VOC, COCO, LVIS, and diverse images like paintings or cartoons.

Title: Improving Generalization in Semantic Parsing by Increasing Natural Language Variation

• Authors: Irina Saparina, Mirella Lapata
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08666
• Pdf URL: https://arxiv.org/pdf/2402.08666
• Copy Paste: [[2402.08666]] Improving Generalization in Semantic Parsing by Increasing Natural Language Variation(https://arxiv.org/abs/2402.08666)
• Keywords: robust, large language model
• Abstract: Text-to-SQL semantic parsing has made significant progress in recent years, with various models demonstrating impressive performance on the challenging Spider benchmark. However, it has also been shown that these models often struggle to generalize even when faced with small perturbations of previously (accurately) parsed expressions. This is mainly due to the linguistic form of questions in Spider which are overly specific, unnatural, and display limited variation. In this work, we use data augmentation to enhance the robustness of text-to-SQL parsers against natural language variations. Existing approaches generate question reformulations either via models trained on Spider or only introduce local changes. In contrast, we leverage the capabilities of large language models to generate more realistic and diverse questions. Using only a few prompts, we achieve a two-fold increase in the number of questions in Spider. Training on this augmented dataset yields substantial improvements on a range of evaluation sets, including robustness benchmarks and out-of-domain data.

Title: Target Score Matching

• Authors: Valentin De Bortoli, Michael Hutchinson, Peter Wirnsberger, Arnaud Doucet
• Subjects: cs.LG, stat.CO, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.08667
• Pdf URL: https://arxiv.org/pdf/2402.08667
• Copy Paste: [[2402.08667]] Target Score Matching(https://arxiv.org/abs/2402.08667)
• Keywords: diffusion
• Abstract: Denoising Score Matching estimates the score of a noised version of a target distribution by minimizing a regression loss and is widely used to train the popular class of Denoising Diffusion Models. A well known limitation of Denoising Score Matching, however, is that it yields poor estimates of the score at low noise levels. This issue is particularly unfavourable for problems in the physical sciences and for Monte Carlo sampling tasks for which the score of the clean original target is known. Intuitively, estimating the score of a slightly noised version of the target should be a simple task in such cases. In this paper, we address this shortcoming and show that it is indeed possible to leverage knowledge of the target score. We present a Target Score Identity and corresponding Target Score Matching regression loss which allows us to obtain score estimates admitting favourable properties at low noise levels.

Title: Graph Mamba: Towards Learning on Graphs with State Space Models

• Authors: Ali Behrouz, Farnoosh Hashemi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08678
• Pdf URL: https://arxiv.org/pdf/2402.08678
• Copy Paste: [[2402.08678]] Graph Mamba: Towards Learning on Graphs with State Space Models(https://arxiv.org/abs/2402.08678)
• Keywords: transformer
• Abstract: Graph Neural Networks (GNNs) have shown promising potential in graph representation learning. The majority of GNNs define a local message-passing mechanism, propagating information over the graph by stacking multiple layers. These methods, however, are known to suffer from two major limitations: over-squashing and poor capturing of long-range dependencies. Recently, Graph Transformers (GTs) emerged as a powerful alternative to Message-Passing Neural Networks (MPNNs). GTs, however, have quadratic computational cost, lack inductive biases on graph structures, and rely on complex Positional/Structural Encodings (SE/PE). In this paper, we show that while Transformers, complex message-passing, and SE/PE are sufficient for good performance in practice, neither is necessary. Motivated by the recent success of State Space Models (SSMs), such as Mamba, we present Graph Mamba Networks (GMNs), a general framework for a new class of GNNs based on selective SSMs. We discuss and categorize the new challenges when adopting SSMs to graph-structured data, and present four required and one optional steps to design GMNs, where we choose (1) Neighborhood Tokenization, (2) Token Ordering, (3) Architecture of Bidirectional Selective SSM Encoder, (4) Local Encoding, and dispensable (5) PE and SE. We further provide theoretical justification for the power of GMNs. Experiments demonstrate that despite much less computational cost, GMNs attain an outstanding performance in long-range, small-scale, large-scale, and heterophilic benchmark datasets.

Title: COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability

• Authors: Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, Bin Hu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08679
• Pdf URL: https://arxiv.org/pdf/2402.08679
• Copy Paste: [[2402.08679]] COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability(https://arxiv.org/abs/2402.08679)
• Keywords: attack, steal, large language model
• Abstract: Jailbreaks on Large language models (LLMs) have recently received increasing attention. For a comprehensive assessment of LLM safety, it is essential to consider jailbreaks with diverse attributes, such as contextual coherence and sentiment/stylistic variations, and hence it is beneficial to study controllable jailbreaking, i.e. how to enforce control on LLM attacks. In this paper, we formally formulate the controllable attack generation problem, and build a novel connection between this problem and controllable text generation, a well-explored topic of natural language processing. Based on this connection, we adapt the Energy-based Constrained Decoding with Langevin Dynamics (COLD), a state-of-the-art, highly efficient algorithm in controllable text generation, and introduce the COLD-Attack framework which unifies and automates the search of adversarial LLM attacks under a variety of control requirements such as fluency, stealthiness, sentiment, and left-right-coherence. The controllability enabled by COLD-Attack leads to diverse new jailbreak scenarios which not only cover the standard setting of generating fluent suffix attacks, but also allow us to address new controllable attack settings such as revising a user query adversarially with minimal paraphrasing, and inserting stealthy attacks in context with left-right-coherence. Our extensive experiments on various LLMs (Llama-2, Mistral, Vicuna, Guanaco, GPT-3.5) show COLD-Attack's broad applicability, strong controllability, high success rate, and attack transferability. Our code is available at https://github.com/Yu-Fangxu/COLD-Attack.

Title: IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation

• Authors: Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, Filippos Kokkinos
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08682
• Pdf URL: https://arxiv.org/pdf/2402.08682
• Copy Paste: [[2402.08682]] IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation(https://arxiv.org/abs/2402.08682)
• Keywords: robust, diffusion
• Abstract: Most text-to-3D generators build upon off-the-shelf text-to-image models trained on billions of images. They use variants of Score Distillation Sampling (SDS), which is slow, somewhat unstable, and prone to artifacts. A mitigation is to fine-tune the 2D generator to be multi-view aware, which can help distillation or can be combined with reconstruction networks to output 3D objects directly. In this paper, we further explore the design space of text-to-3D models. We significantly improve multi-view generation by considering video instead of image generators. Combined with a 3D reconstruction algorithm which, by using Gaussian splatting, can optimize a robust image-based loss, we directly produce high-quality 3D outputs from the generated views. Our new method, IM-3D, reduces the number of evaluations of the 2D generator network 10-100x, resulting in a much more efficient pipeline, better quality, fewer geometric inconsistencies, and higher yield of usable 3D assets.