2024-02-12

Title: Todyformer: Towards Holistic Dynamic Graph Transformers with Structure-Aware Tokenization

• Authors: Mahdi Biparva, Raika Karimi, Faezeh Faez, Yingxue Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05944
• Pdf URL: https://arxiv.org/pdf/2402.05944
• Copy Paste: [[2402.05944]] Todyformer: Towards Holistic Dynamic Graph Transformers with Structure-Aware Tokenization(https://arxiv.org/abs/2402.05944)
• Keywords: transformer
• Abstract: Temporal Graph Neural Networks have garnered substantial attention for their capacity to model evolving structural and temporal patterns while exhibiting impressive performance. However, it is known that these architectures are encumbered by issues that constrain their performance, such as over-squashing and over-smoothing. Meanwhile, Transformers have demonstrated exceptional computational capacity to effectively address challenges related to long-range dependencies. Consequently, we introduce Todyformer-a novel Transformer-based neural network tailored for dynamic graphs. It unifies the local encoding capacity of Message-Passing Neural Networks (MPNNs) with the global encoding of Transformers through i) a novel patchifying paradigm for dynamic graphs to improve over-squashing, ii) a structure-aware parametric tokenization strategy leveraging MPNNs, iii) a Transformer with temporal positional-encoding to capture long-range dependencies, and iv) an encoding architecture that alternates between local and global contextualization, mitigating over-smoothing in MPNNs. Experimental evaluations on public benchmark datasets demonstrate that Todyformer consistently outperforms the state-of-the-art methods for downstream tasks. Furthermore, we illustrate the underlying aspects of the proposed model in effectively capturing extensive temporal dependencies in dynamic graphs.

Title: Separable Multi-Concept Erasure from Diffusion Models

• Authors: Mengnan Zhao, Lihe Zhang, Tianhang Zheng, Yuqiu Kong, Baocai Yin
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05947
• Pdf URL: https://arxiv.org/pdf/2402.05947
• Copy Paste: [[2402.05947]] Separable Multi-Concept Erasure from Diffusion Models(https://arxiv.org/abs/2402.05947)
• Keywords: diffusion, generative
• Abstract: Large-scale diffusion models, known for their impressive image generation capabilities, have raised concerns among researchers regarding social impacts, such as the imitation of copyrighted artistic styles. In response, existing approaches turn to machine unlearning techniques to eliminate unsafe concepts from pre-trained models. However, these methods compromise the generative performance and neglect the coupling among multi-concept erasures, as well as the concept restoration problem. To address these issues, we propose a Separable Multi-concept Eraser (SepME), which mainly includes two parts: the generation of concept-irrelevant representations and the weight decoupling. The former aims to avoid unlearning substantial information that is irrelevant to forgotten concepts. The latter separates optimizable model weights, making each weight increment correspond to a specific concept erasure without affecting generative performance on other concepts. Specifically, the weight increment for erasing a specified concept is formulated as a linear combination of solutions calculated based on other known undesirable concepts. Extensive experiments indicate the efficacy of our approach in eliminating concepts, preserving model performance, and offering flexibility in the erasure or recovery of various concepts.

Title: DE$^3$-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks

• Authors: Jianing He, Qi Zhang, Weiping Ding, Duoqian Miao, Jun Zhao, Liang Hu, Longbing Cao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.05948
• Pdf URL: https://arxiv.org/pdf/2402.05948
• Copy Paste: [[2402.05948]] DE$^3$-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks(https://arxiv.org/abs/2402.05948)
• Keywords: interpretability
• Abstract: Early exiting has demonstrated its effectiveness in accelerating the inference of pre-trained language models like BERT by dynamically adjusting the number of layers executed. However, most existing early exiting methods only consider local information from an individual test sample to determine their exiting indicators, failing to leverage the global information offered by sample population. This leads to suboptimal estimation of prediction correctness, resulting in erroneous exiting decisions. To bridge the gap, we explore the necessity of effectively combining both local and global information to ensure reliable early exiting during inference. Purposefully, we leverage prototypical networks to learn class prototypes and devise a distance metric between samples and class prototypes. This enables us to utilize global information for estimating the correctness of early predictions. On this basis, we propose a novel Distance-Enhanced Early Exiting framework for BERT (DE$^3$-BERT). DE$^3$-BERT implements a hybrid exiting strategy that supplements classic entropy-based local information with distance-based global information to enhance the estimation of prediction correctness for more reliable early exiting decisions. Extensive experiments on the GLUE benchmark demonstrate that DE$^3$-BERT consistently outperforms state-of-the-art models under different speed-up ratios with minimal storage or computational overhead, yielding a better trade-off between model performance and inference efficiency. Additionally, an in-depth analysis further validates the generality and interpretability of our method.

Title: Advancing Graph Representation Learning with Large Language Models: A Comprehensive Survey of Techniques

• Authors: Qiheng Mao, Zemin Liu, Chenghao Liu, Zhuo Li, Jianling Sun
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.05952
• Pdf URL: https://arxiv.org/pdf/2402.05952
• Copy Paste: [[2402.05952]] Advancing Graph Representation Learning with Large Language Models: A Comprehensive Survey of Techniques(https://arxiv.org/abs/2402.05952)
• Keywords: large language model
• Abstract: The integration of Large Language Models (LLMs) with Graph Representation Learning (GRL) marks a significant evolution in analyzing complex data structures. This collaboration harnesses the sophisticated linguistic capabilities of LLMs to improve the contextual understanding and adaptability of graph models, thereby broadening the scope and potential of GRL. Despite a growing body of research dedicated to integrating LLMs into the graph domain, a comprehensive review that deeply analyzes the core components and operations within these models is notably lacking. Our survey fills this gap by proposing a novel taxonomy that breaks down these models into primary components and operation techniques from a novel technical perspective. We further dissect recent literature into two primary components including knowledge extractors and organizers, and two operation techniques including integration and training stratigies, shedding light on effective model design and training strategies. Additionally, we identify and explore potential future research avenues in this nascent yet underexplored field, proposing paths for continued progress.

Title: A Hyper-Transformer model for Controllable Pareto Front Learning with Split Feasibility Constraints

• Authors: Tran Anh Tuan, Nguyen Viet Dung, Tran Ngoc Thang
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2402.05955
• Pdf URL: https://arxiv.org/pdf/2402.05955
• Copy Paste: [[2402.05955]] A Hyper-Transformer model for Controllable Pareto Front Learning with Split Feasibility Constraints(https://arxiv.org/abs/2402.05955)
• Keywords: transformer
• Abstract: Controllable Pareto front learning (CPFL) approximates the Pareto solution set and then locates a Pareto optimal solution with respect to a given reference vector. However, decision-maker objectives were limited to a constraint region in practice, so instead of training on the entire decision space, we only trained on the constraint region. Controllable Pareto front learning with Split Feasibility Constraints (SFC) is a way to find the best Pareto solutions to a split multi-objective optimization problem that meets certain constraints. In the previous study, CPFL used a Hypernetwork model comprising multi-layer perceptron (Hyper-MLP) blocks. With the substantial advancement of transformer architecture in deep learning, transformers can outperform other architectures in various tasks. Therefore, we have developed a hyper-transformer (Hyper-Trans) model for CPFL with SFC. We use the theory of universal approximation for the sequence-to-sequence function to show that the Hyper-Trans model makes MED errors smaller in computational experiments than the Hyper-MLP model.

Title: Pathformer: Multi-scale transformers with Adaptive Pathways for Time Series Forecasting

• Authors: Peng Chen, Yingying Zhang, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, Chenjuan Guo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05956
• Pdf URL: https://arxiv.org/pdf/2402.05956
• Copy Paste: [[2402.05956]] Pathformer: Multi-scale transformers with Adaptive Pathways for Time Series Forecasting(https://arxiv.org/abs/2402.05956)
• Keywords: transformer
• Abstract: Transformer-based models have achieved some success in time series forecasting. Existing methods mainly model time series from limited or fixed scales, making it challenging to capture different characteristics spanning various scales. In this paper, we propose multi-scale transformers with adaptive pathways (Pathformer). The proposed Transformer integrates both temporal resolution and temporal distance for multi-scale modeling. Multi-scale division divides the time series into different temporal resolutions using patches of various sizes. Based on the division of each scale, dual attention is performed over these patches to capture global correlations and local details as temporal dependencies. We further enrich the multi-scale transformer with adaptive pathways, which adaptively adjust the multi-scale modeling process based on the varying temporal dynamics in the input time series, improving the prediction accuracy and generalization of Pathformer. Extensive experiments on eleven real-world datasets demonstrate that Pathformer not only achieves state-of-the-art performance by surpassing all current models but also exhibits stronger generalization abilities under various transfer scenarios.

Title: Nature-Inspired Local Propagation

• Authors: Alessandro Betti, Marco Gori
• Subjects: cs.LG, cond-mat.dis-nn, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.05959
• Pdf URL: https://arxiv.org/pdf/2402.05959
• Copy Paste: [[2402.05959]] Nature-Inspired Local Propagation(https://arxiv.org/abs/2402.05959)
• Keywords: generative
• Abstract: The spectacular results achieved in machine learning, including the recent advances in generative AI, rely on large data collections. On the opposite, intelligent processes in nature arises without the need for such collections, but simply by online processing of the environmental information. In particular, natural learning processes rely on mechanisms where data representation and learning are intertwined in such a way to respect spatiotemporal locality. This paper shows that such a feature arises from a pre-algorithmic view of learning that is inspired by related studies in Theoretical Physics. We show that the algorithmic interpretation of the derived "laws of learning", which takes the structure of Hamiltonian equations, reduces to Backpropagation when the speed of propagation goes to infinity. This opens the doors to machine learning studies based on full on-line information processing that are based the replacement of Backpropagation with the proposed spatiotemporal local algorithm.

Title: EXGC: Bridging Efficiency and Explainability in Graph Condensation

• Authors: Junfeng Fang, Xinglin Li, Yongduo Sui, Yuan Gao, Guibin Zhang, Kun Wang, Xiang Wang, Xiangnan He
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05962
• Pdf URL: https://arxiv.org/pdf/2402.05962
• Copy Paste: [[2402.05962]] EXGC: Bridging Efficiency and Explainability in Graph Condensation(https://arxiv.org/abs/2402.05962)
• Keywords: explainability
• Abstract: Graph representation learning on vast datasets, like web data, has made significant strides. However, the associated computational and storage overheads raise concerns. In sight of this, Graph condensation (GCond) has been introduced to distill these large real datasets into a more concise yet information-rich synthetic graph. Despite acceleration efforts, existing GCond methods mainly grapple with efficiency, especially on expansive web data graphs. Hence, in this work, we pinpoint two major inefficiencies of current paradigms: (1) the concurrent updating of a vast parameter set, and (2) pronounced parameter redundancy. To counteract these two limitations correspondingly, we first (1) employ the Mean-Field variational approximation for convergence acceleration, and then (2) propose the objective of Gradient Information Bottleneck (GDIB) to prune redundancy. By incorporating the leading explanation techniques (e.g., GNNExplainer and GSAT) to instantiate the GDIB, our EXGC, the Efficient and eXplainable Graph Condensation method is proposed, which can markedly boost efficiency and inject explainability. Our extensive evaluations across eight datasets underscore EXGC's superiority and relevance. Code is available at https://github.com/MangoKiller/EXGC.

Title: A Survey on Transformer Compression

• Authors: Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, Dacheng Tao
• Subjects: cs.LG, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05964
• Pdf URL: https://arxiv.org/pdf/2402.05964
• Copy Paste: [[2402.05964]] A Survey on Transformer Compression(https://arxiv.org/abs/2402.05964)
• Keywords: transformer
• Abstract: Large models based on the Transformer architecture play increasingly vital roles in artificial intelligence, particularly within the realms of natural language processing (NLP) and computer vision (CV). Model compression methods reduce their memory and computational cost, which is a necessary step to implement the transformer models on practical devices. Given the unique architecture of transformer, featuring alternative attention and Feedforward Neural Network (FFN) modules, specific compression techniques are required. The efficiency of these compression methods is also paramount, as it is usually impractical to retrain large models on the entire training dataset.This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to transformer models. The compression methods are primarily categorized into pruning, quantization, knowledge distillation, and efficient architecture design. In each category, we discuss compression methods for both CV and NLP tasks, highlighting common underlying principles. At last, we delve into the relation between various compression methods, and discuss the further directions in this domain.

Title: The last Dance : Robust backdoor attack via diffusion models and bayesian approach

• Authors: Orson Mengara
• Subjects: cs.LG, cs.AI, cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.05967
• Pdf URL: https://arxiv.org/pdf/2402.05967
• Copy Paste: [[2402.05967]] The last Dance : Robust backdoor attack via diffusion models and bayesian approach(https://arxiv.org/abs/2402.05967)
• Keywords: attack, robust, diffusion, transformer, generative
• Abstract: Diffusion models are state-of-the-art deep learning generative models that are trained on the principle of learning forward and backward diffusion processes via the progressive addition of noise and denoising. In this paper, we seek to trick audio-based DNN models, such as those in the Hugging Face framework, for example, those that focus on audio, in particular transformer-based artificial intelligence models, which are powerful machine learning models that save time and deliver faster, more efficient results. We demonstrate the feasibility of backdoor attacks (called `BacKBayDiffMod`) on audio transformers derived from Hugging Face, a popular framework in the world of artificial intelligence (AI) research. The backdoor attack developed in this paper is based on poisoning the model's training data by incorporating backdoor diffusion sampling and a Bayesian approach to the distribution of poisoned data.

Title: Federated Learning Priorities Under the European Union Artificial Intelligence Act

• Authors: Herbert Woisetschläger, Alexander Erben, Bill Marino, Shiqiang Wang, Nicholas D. Lane, Ruben Mayer, Hans-Arno Jacobsen
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.05968
• Pdf URL: https://arxiv.org/pdf/2402.05968
• Copy Paste: [[2402.05968]] Federated Learning Priorities Under the European Union Artificial Intelligence Act(https://arxiv.org/abs/2402.05968)
• Keywords: secure, privacy, federate
• Abstract: The age of AI regulation is upon us, with the European Union Artificial Intelligence Act (AI Act) leading the way. Our key inquiry is how this will affect Federated Learning (FL), whose starting point of prioritizing data privacy while performing ML fundamentally differs from that of centralized learning. We believe the AI Act and future regulations could be the missing catalyst that pushes FL toward mainstream adoption. However, this can only occur if the FL community reprioritizes its research focus. In our position paper, we perform a first-of-its-kind interdisciplinary analysis (legal and ML) of the impact the AI Act may have on FL and make a series of observations supporting our primary position through quantitative and qualitative analysis. We explore data governance issues and the concern for privacy. We establish new challenges regarding performance and energy efficiency within lifecycle monitoring. Taken together, our analysis suggests there is a sizable opportunity for FL to become a crucial component of AI Act-compliant ML systems and for the new regulation to drive the adoption of FL techniques in general. Most noteworthy are the opportunities to defend against data bias and enhance private and secure computation

Title: Breaking Symmetry When Training Transformers

• Authors: Chunsheng Zuo, Michael Guerzhoy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.05969
• Pdf URL: https://arxiv.org/pdf/2402.05969
• Copy Paste: [[2402.05969]] Breaking Symmetry When Training Transformers(https://arxiv.org/abs/2402.05969)
• Keywords: transformer
• Abstract: As we show in this paper, the prediction for output token $n+1$ of Transformer architectures without one of the mechanisms of positional encodings and causal attention is invariant to permutations of input tokens $1, 2, ..., n-1$. Usually, both mechanisms are employed and the symmetry with respect to the input tokens is broken. Recently, it has been shown that one can train Transformers without positional encodings. This must be enabled by the causal attention mechanism. In this paper, we elaborate on the argument that the causal connection mechanism must be responsible for the fact that Transformers are able to model input sequences where the order is important. Vertical "slices" of Transformers are all encouraged to represent the same location $k$ in the input sequence. We hypothesize that residual connections contribute to this phenomenon, and demonstrate evidence for this.

Title: Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning and Levels-of-Experts

• Authors: Kun Wang, Hao Wu, Guibin Zhang, Junfeng Fang, Yuxuan Liang, Yuankai Wu, Roger Zimmermann, Yang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.05970
• Pdf URL: https://arxiv.org/pdf/2402.05970
• Copy Paste: [[2402.05970]] Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning and Levels-of-Experts(https://arxiv.org/abs/2402.05970)
• Keywords: interpretability
• Abstract: In this paper, we address the issue of modeling and estimating changes in the state of the spatio-temporal dynamical systems based on a sequence of observations like video frames. Traditional numerical simulation systems depend largely on the initial settings and correctness of the constructed partial differential equations (PDEs). Despite recent efforts yielding significant success in discovering data-driven PDEs with neural networks, the limitations posed by singular scenarios and the absence of local insights prevent them from performing effectively in a broader real-world context. To this end, this paper propose the universal expert module -- that is, optical flow estimation component, to capture the evolution laws of general physical processes in a data-driven fashion. To enhance local insight, we painstakingly design a finer-grained physical pipeline, since local characteristics may be influenced by various internal contextual information, which may contradict the macroscopic properties of the whole system. Further, we harness currently popular neural discrete learning to unveil the underlying important features in its latent space, this process better injects interpretability, which can help us obtain a powerful prior over these discrete random variables. We conduct extensive experiments and ablations to demonstrate that the proposed framework achieves large performance margins, compared with the existing SOTA baselines.

Title: Blockchain-enabled Clustered and Scalable Federated Learning (BCS-FL) Framework in UAV Networks

• Authors: Sana Hafeez, Lina Mohjazi, Muhammad Ali Imran, Yao Sun
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.05973
• Pdf URL: https://arxiv.org/pdf/2402.05973
• Copy Paste: [[2402.05973]] Blockchain-enabled Clustered and Scalable Federated Learning (BCS-FL) Framework in UAV Networks(https://arxiv.org/abs/2402.05973)
• Keywords: privacy, federate
• Abstract: Privacy, scalability, and reliability are significant challenges in unmanned aerial vehicle (UAV) networks as distributed systems, especially when employing machine learning (ML) technologies with substantial data exchange. Recently, the application of federated learning (FL) to UAV networks has improved collaboration, privacy, resilience, and adaptability, making it a promising framework for UAV applications. However, implementing FL for UAV networks introduces drawbacks such as communication overhead, synchronization issues, scalability limitations, and resource constraints. To address these challenges, this paper presents the Blockchain-enabled Clustered and Scalable Federated Learning (BCS-FL) framework for UAV networks. This improves the decentralization, coordination, scalability, and efficiency of FL in large-scale UAV networks. The framework partitions UAV networks into separate clusters, coordinated by cluster head UAVs (CHs), to establish a connected graph. Clustering enables efficient coordination of updates to the ML model. Additionally, hybrid inter-cluster and intra-cluster model aggregation schemes generate the global model after each training round, improving collaboration and knowledge sharing among clusters. The numerical findings illustrate the achievement of convergence while also emphasizing the trade-offs between the effectiveness of training and communication efficiency.

Title: Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy

• Authors: Seyedarmin Azizi, Mahdi Nazemi, Massoud Pedram
• Subjects: cs.CV, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.06004
• Pdf URL: https://arxiv.org/pdf/2402.06004
• Copy Paste: [[2402.06004]] Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy(https://arxiv.org/abs/2402.06004)
• Keywords: transformer
• Abstract: As Vision Transformers (ViTs) increasingly set new benchmarks in computer vision, their practical deployment on inference engines is often hindered by their significant memory bandwidth and (on-chip) memory footprint requirements. This paper addresses this memory limitation by introducing an activation-aware model compression methodology that uses selective low-rank weight tensor approximations of different layers to reduce the parameter count of ViTs. The key idea is to decompose the weight tensors into a sum of two parameter-efficient tensors while minimizing the error between the product of the input activations with the original weight tensor and the product of the input activations with the approximate tensor sum. This approximation is further refined by adopting an efficient layer-wise error compensation technique that uses the gradient of the layer's output loss. The combination of these techniques achieves excellent results while it avoids being trapped in a shallow local minimum early in the optimization process and strikes a good balance between the model compression and output accuracy. Notably, the presented method significantly reduces the parameter count of DeiT-B by 60% with less than 1% accuracy drop on the ImageNet dataset, overcoming the usual accuracy degradation seen in low-rank approximations. In addition to this, the presented compression technique can compress large DeiT/ViT models to have about the same model size as smaller DeiT/ViT variants while yielding up to 1.8% accuracy gain. These results highlight the efficacy of our approach, presenting a viable solution for embedding ViTs in memory-constrained environments without compromising their performance.

Title: Decision Theory-Guided Deep Reinforcement Learning for Fast Learning

• Authors: Zelin Wan, Jin-Hee Cho, Mu Zhu, Ahmed H. Anwar, Charles Kamhoua, Munindar P. Singh
• Subjects: cs.LG, cs.AI, cs.GT
• Abstract URL: https://arxiv.org/abs/2402.06023
• Pdf URL: https://arxiv.org/pdf/2402.06023
• Copy Paste: [[2402.06023]] Decision Theory-Guided Deep Reinforcement Learning for Fast Learning(https://arxiv.org/abs/2402.06023)
• Keywords: robust
• Abstract: This paper introduces a novel approach, Decision Theory-guided Deep Reinforcement Learning (DT-guided DRL), to address the inherent cold start problem in DRL. By integrating decision theory principles, DT-guided DRL enhances agents' initial performance and robustness in complex environments, enabling more efficient and reliable convergence during learning. Our investigation encompasses two primary problem contexts: the cart pole and maze navigation challenges. Experimental results demonstrate that the integration of decision theory not only facilitates effective initial guidance for DRL agents but also promotes a more structured and informed exploration strategy, particularly in environments characterized by large and intricate state spaces. The results of experiment demonstrate that DT-guided DRL can provide significantly higher rewards compared to regular DRL. Specifically, during the initial phase of training, the DT-guided DRL yields up to an 184% increase in accumulated reward. Moreover, even after reaching convergence, it maintains a superior performance, ending with up to 53% more reward than standard DRL in large maze problems. DT-guided DRL represents an advancement in mitigating a fundamental challenge of DRL by leveraging functions informed by human (designer) knowledge, setting a foundation for further research in this promising interdisciplinary domain.

Title: Game-theoretic Counterfactual Explanation for Graph Neural Networks

• Authors: Chirag Chhablani, Sarthak Jain, Akshay Channesh, Ian A. Kash, Sourav Medya
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06030
• Pdf URL: https://arxiv.org/pdf/2402.06030
• Copy Paste: [[2402.06030]] Game-theoretic Counterfactual Explanation for Graph Neural Networks(https://arxiv.org/abs/2402.06030)
• Keywords: robust, interpretability
• Abstract: Graph Neural Networks (GNNs) have been a powerful tool for node classification tasks in complex networks. However, their decision-making processes remain a black-box to users, making it challenging to understand the reasoning behind their predictions. Counterfactual explanations (CFE) have shown promise in enhancing the interpretability of machine learning models. Prior approaches to compute CFE for GNNS often are learning-based approaches that require training additional graphs. In this paper, we propose a semivalue-based, non-learning approach to generate CFE for node classification tasks, eliminating the need for any additional training. Our results reveals that computing Banzhaf values requires lower sample complexity in identifying the counterfactual explanations compared to other popular methods such as computing Shapley values. Our empirical evidence indicates computing Banzhaf values can achieve up to a fourfold speed up compared to Shapley values. We also design a thresholding method for computing Banzhaf values and show theoretical and empirical results on its robustness in noisy environments, making it superior to Shapley values. Furthermore, the thresholded Banzhaf values are shown to enhance efficiency without compromising the quality (i.e., fidelity) in the explanations in three popular graph datasets.

Title: A Prompt Response to the Demand for Automatic Gender-Neutral Translation

• Authors: Beatrice Savoldi, Andrea Piergentili, Dennis Fucci, Matteo Negri, Luisa Bentivogli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06041
• Pdf URL: https://arxiv.org/pdf/2402.06041
• Copy Paste: [[2402.06041]] A Prompt Response to the Demand for Automatic Gender-Neutral Translation(https://arxiv.org/abs/2402.06041)
• Keywords: large language model
• Abstract: Gender-neutral translation (GNT) that avoids biased and undue binary assumptions is a pivotal challenge for the creation of more inclusive translation technologies. Advancements for this task in Machine Translation (MT), however, are hindered by the lack of dedicated parallel data, which are necessary to adapt MT systems to satisfy neutral constraints. For such a scenario, large language models offer hitherto unforeseen possibilities, as they come with the distinct advantage of being versatile in various (sub)tasks when provided with explicit instructions. In this paper, we explore this potential to automate GNT by comparing MT with the popular GPT-4 model. Through extensive manual analyses, our study empirically reveals the inherent limitations of current MT systems in generating GNTs and provides valuable insights into the potential and challenges associated with prompting for neutrality.

Title: Scaling Artificial Intelligence for Digital Wargaming in Support of Decision-Making

• Authors: Scotty Black, Christian Darken
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06075
• Pdf URL: https://arxiv.org/pdf/2402.06075
• Copy Paste: [[2402.06075]] Scaling Artificial Intelligence for Digital Wargaming in Support of Decision-Making(https://arxiv.org/abs/2402.06075)
• Keywords: robust
• Abstract: In this unprecedented era of technology-driven transformation, it becomes more critical than ever that we aggressively invest in developing robust artificial intelligence (AI) for wargaming in support of decision-making. By advancing AI-enabled systems and pairing these with human judgment, we will be able to enhance all-domain awareness, improve the speed and quality of our decision cycles, offer recommendations for novel courses of action, and more rapidly counter our adversary's actions. It therefore becomes imperative that we accelerate the development of AI to help us better address the complexity of modern challenges and dilemmas that currently requires human intelligence and, if possible, attempt to surpass human intelligence--not to replace humans, but to augment and better inform human decision-making at machine speed. Although deep reinforcement learning continues to show promising results in intelligent agent behavior development for the long-horizon, complex tasks typically found in combat modeling and simulation, further research is needed to enable the scaling of AI to deal with these intricate and expansive state-spaces characteristic of wargaming for either concept development, education, or analysis. To help address this challenge, in our research, we are developing and implementing a hierarchical reinforcement learning framework that includes a multi-model approach and dimension-invariant observation abstractions.

Title: SubGen: Token Generation in Sublinear Time and Memory

• Authors: Amir Zandieh, Insu Han, Vahab Mirrokni, Amin Karbasi
• Subjects: cs.LG, cs.AI, cs.DS
• Abstract URL: https://arxiv.org/abs/2402.06082
• Pdf URL: https://arxiv.org/pdf/2402.06082
• Copy Paste: [[2402.06082]] SubGen: Token Generation in Sublinear Time and Memory(https://arxiv.org/abs/2402.06082)
• Keywords: large language model
• Abstract: Despite the significant success of large language models (LLMs), their extensive memory requirements pose challenges for deploying them in long-context token generation. The substantial memory footprint of LLM decoders arises from the necessity to store all previous tokens in the attention module, a requirement imposed by key-value (KV) caching. In this work, our focus is on developing an efficient compression technique for the KV cache. Empirical evidence indicates a significant clustering tendency within key embeddings in the attention module. Building on this key insight, we have devised a novel caching method with sublinear complexity, employing online clustering on key tokens and online $\ell_2$ sampling on values. The result is a provably accurate and efficient attention decoding algorithm, termed SubGen. Not only does this algorithm ensure a sublinear memory footprint and sublinear time complexity, but we also establish a tight error bound for our approach. Empirical evaluations on long-context question-answering tasks demonstrate that SubGen significantly outperforms existing and state-of-the-art KV cache compression methods in terms of performance and efficiency.

Title: Animated Stickers: Bringing Stickers to Life with Video Diffusion

• Authors: David Yan, Winnie Zhang, Luxin Zhang, Anmol Kalia, Dingkang Wang, Ankit Ramchandani, Miao Liu, Albert Pumarola, Edgar Schoenfeld, Elliot Blanchard, Krishna Narni, Yaqiao Luo, Lawrence Chen, Guan Pang, Ali Thabet, Peter Vajda, Amy Bearman, Licheng Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06088
• Pdf URL: https://arxiv.org/pdf/2402.06088
• Copy Paste: [[2402.06088]] Animated Stickers: Bringing Stickers to Life with Video Diffusion(https://arxiv.org/abs/2402.06088)
• Keywords: diffusion
• Abstract: We introduce animated stickers, a video diffusion model which generates an animation conditioned on a text prompt and static sticker image. Our model is built on top of the state-of-the-art Emu text-to-image model, with the addition of temporal layers to model motion. Due to the domain gap, i.e. differences in visual and motion style, a model which performed well on generating natural videos can no longer generate vivid videos when applied to stickers. To bridge this gap, we employ a two-stage finetuning pipeline: first with weakly in-domain data, followed by human-in-the-loop (HITL) strategy which we term ensemble-of-teachers. It distills the best qualities of multiple teachers into a smaller student model. We show that this strategy allows us to specifically target improvements to motion quality while maintaining the style from the static image. With inference optimizations, our model is able to generate an eight-frame video with high-quality, interesting, and relevant motion in under one second.

Title: Early Fusion of Features for Semantic Segmentation

• Authors: Anupam Gupta, Ashok Krishnamurthy, Lisa Singh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06091
• Pdf URL: https://arxiv.org/pdf/2402.06091
• Copy Paste: [[2402.06091]] Early Fusion of Features for Semantic Segmentation(https://arxiv.org/abs/2402.06091)
• Keywords: robust, segmentation
• Abstract: This paper introduces a novel segmentation framework that integrates a classifier network with a reverse HRNet architecture for efficient image segmentation. Our approach utilizes a ResNet-50 backbone, pretrained in a semi-supervised manner, to generate feature maps at various scales. These maps are then processed by a reverse HRNet, which is adapted to handle varying channel dimensions through 1x1 convolutions, to produce the final segmentation output. We strategically avoid fine-tuning the backbone network to minimize memory consumption during training. Our methodology is rigorously tested across several benchmark datasets including Mapillary Vistas, Cityscapes, CamVid, COCO, and PASCAL-VOC2012, employing metrics such as pixel accuracy and mean Intersection over Union (mIoU) to evaluate segmentation performance. The results demonstrate the effectiveness of our proposed model in achieving high segmentation accuracy, indicating its potential for various applications in image analysis. By leveraging the strengths of both the ResNet-50 and reverse HRNet within a unified framework, we present a robust solution to the challenges of image segmentation.

Title: CLIP-Loc: Multi-modal Landmark Association for Global Localization in Object-based Maps

• Authors: Shigemichi Matsuzaki, Takuma Sugino, Kazuhito Tanaka, Zijun Sha, Shintaro Nakaoka, Shintaro Yoshizawa, Kazuhiro Shintani
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.06092
• Pdf URL: https://arxiv.org/pdf/2402.06092
• Copy Paste: [[2402.06092]] CLIP-Loc: Multi-modal Landmark Association for Global Localization in Object-based Maps(https://arxiv.org/abs/2402.06092)
• Keywords: extraction
• Abstract: This paper describes a multi-modal data association method for global localization using object-based maps and camera images. In global localization, or relocalization, using object-based maps, existing methods typically resort to matching all possible combinations of detected objects and landmarks with the same object category, followed by inlier extraction using RANSAC or brute-force search. This approach becomes infeasible as the number of landmarks increases due to the exponential growth of correspondence candidates. In this paper, we propose labeling landmarks with natural language descriptions and extracting correspondences based on conceptual similarity with image observations using a Vision Language Model (VLM). By leveraging detailed text information, our approach efficiently extracts correspondences compared to methods using only object categories. Through experiments, we demonstrate that the proposed method enables more accurate global localization with fewer iterations compared to baseline methods, exhibiting its efficiency.

Title: Formal Verification of the Sumcheck Protocol

• Authors: Azucena Garvía Bosshard, Jonathan Bootle, Christoph Sprenger
• Subjects: cs.CR, cs.LO
• Abstract URL: https://arxiv.org/abs/2402.06093
• Pdf URL: https://arxiv.org/pdf/2402.06093
• Copy Paste: [[2402.06093]] Formal Verification of the Sumcheck Protocol(https://arxiv.org/abs/2402.06093)
• Keywords: security
• Abstract: The sumcheck protocol, introduced in 1992, is an interactive proof which is a key component of many probabilistic proof systems in computational complexity theory and cryptography, some of which have been deployed. However, none of these proof systems based on the sumcheck protocol enjoy a formally-verified security analysis. In this paper, we make progress in this direction by providing a formally verified security analysis of the sumcheck protocol using the interactive theorem prover Isabelle/HOL. We follow a general and modular approach. First, we give a general formalization of public-coin interactive proofs. We then define a generalized sumcheck protocol for which we axiomatize the underlying mathematical structure and we establish its soundness and completeness. Finally, we prove that these axioms hold for multivariate polynomials, the original setting of the sumcheck protocol. Our modular analysis facilitates formal verification of sumcheck instances based on different mathematical structures with little effort, by simply proving that these structures satisfy the axioms. Moreover, the analysis supports the development and formal verification of future cryptographic protocols using the sumcheck protocol as a building block.

Title: Rethinking Data Selection for Supervised Fine-Tuning

• Authors: Ming Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06094
• Pdf URL: https://arxiv.org/pdf/2402.06094
• Copy Paste: [[2402.06094]] Rethinking Data Selection for Supervised Fine-Tuning(https://arxiv.org/abs/2402.06094)
• Keywords: large language model
• Abstract: Although supervised finetuning (SFT) has emerged as an essential technique to align large language models with humans, it is considered superficial, with style learning being its nature. At the same time, recent works indicate the importance of data selection for SFT, showing that finetuning with high-quality and diverse subsets of the original dataset leads to superior downstream performance. In this work, we rethink the intuition behind data selection for SFT. Considering SFT is superficial, we propose that essential demonstrations for SFT should focus on reflecting human-like interactions instead of data quality or diversity. However, it is not straightforward to directly assess to what extent a demonstration reflects human styles. Towards an initial attempt in this direction, we find selecting instances with long responses is surprisingly more effective for SFT than utilizing full datasets or instances selected based on quality and diversity. We hypothesize that such a simple heuristic implicitly mimics a crucial aspect of human-style conversation: detailed responses are usually more helpful.

Title: CLR-Face: Conditional Latent Refinement for Blind Face Restoration Using Score-Based Diffusion Models

• Authors: Maitreya Suin, Rama Chellappa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06106
• Pdf URL: https://arxiv.org/pdf/2402.06106
• Copy Paste: [[2402.06106]] CLR-Face: Conditional Latent Refinement for Blind Face Restoration Using Score-Based Diffusion Models(https://arxiv.org/abs/2402.06106)
• Keywords: robust, diffusion, generative
• Abstract: Recent generative-prior-based methods have shown promising blind face restoration performance. They usually project the degraded images to the latent space and then decode high-quality faces either by single-stage latent optimization or directly from the encoding. Generating fine-grained facial details faithful to inputs remains a challenging problem. Most existing methods produce either overly smooth outputs or alter the identity as they attempt to balance between generation and reconstruction. This may be attributed to the typical trade-off between quality and resolution in the latent space. If the latent space is highly compressed, the decoded output is more robust to degradations but shows worse fidelity. On the other hand, a more flexible latent space can capture intricate facial details better, but is extremely difficult to optimize for highly degraded faces using existing techniques. To address these issues, we introduce a diffusion-based-prior inside a VQGAN architecture that focuses on learning the distribution over uncorrupted latent embeddings. With such knowledge, we iteratively recover the clean embedding conditioning on the degraded counterpart. Furthermore, to ensure the reverse diffusion trajectory does not deviate from the underlying identity, we train a separate Identity Recovery Network and use its output to constrain the reverse diffusion process. Specifically, using a learnable latent mask, we add gradients from a face-recognition network to a subset of latent features that correlates with the finer identity-related details in the pixel space, leaving the other features untouched. Disentanglement between perception and fidelity in the latent space allows us to achieve the best of both worlds. We perform extensive evaluations on multiple real and synthetic datasets to validate the superiority of our approach.

Title: AI enhanced data assimilation and uncertainty quantification applied to Geological Carbon Storage

• Authors: G. S. Seabra (1, 2), N. T. Mücke (3, 4), V. L. S. Silva (2, 5), D. Voskov (1, 6), F. Vossepoel (1) ((1) TU Delft, Netherlands, (2) Petrobras, Brazil, (3) Centrum Wiskunde & Informatica, Netherlands, (4) Utrecht University, Netherlands, (5) Imperial College London, United Kingdom, (6) Stanford University, USA)
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06110
• Pdf URL: https://arxiv.org/pdf/2402.06110
• Copy Paste: [[2402.06110]] AI enhanced data assimilation and uncertainty quantification applied to Geological Carbon Storage(https://arxiv.org/abs/2402.06110)
• Keywords: transformer
• Abstract: This study investigates the integration of machine learning (ML) and data assimilation (DA) techniques, focusing on implementing surrogate models for Geological Carbon Storage (GCS) projects while maintaining high fidelity physical results in posterior states. Initially, we evaluate the surrogate modeling capability of two distinct machine learning models, Fourier Neural Operators (FNOs) and Transformer UNet (T-UNet), in the context of CO$_2$ injection simulations within channelized reservoirs. We introduce the Surrogate-based hybrid ESMDA (SH-ESMDA), an adaptation of the traditional Ensemble Smoother with Multiple Data Assimilation (ESMDA). This method uses FNOs and T-UNet as surrogate models and has the potential to make the standard ESMDA process at least 50% faster or more, depending on the number of assimilation steps. Additionally, we introduce Surrogate-based Hybrid RML (SH-RML), a variational data assimilation approach that relies on the randomized maximum likelihood (RML) where both the FNO and the T-UNet enable the computation of gradients for the optimization of the objective function, and a high-fidelity model is employed for the computation of the posterior states. Our comparative analyses show that SH-RML offers better uncertainty quantification compared to conventional ESMDA for the case study.

Title: ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling

• Authors: Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, Li Erran Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06118
• Pdf URL: https://arxiv.org/pdf/2402.06118
• Copy Paste: [[2402.06118]] ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling(https://arxiv.org/abs/2402.06118)
• Keywords: large language model
• Abstract: By combining natural language understanding and the generation capabilities and breadth of knowledge of large language models with image perception, recent large vision language models (LVLMs) have shown unprecedented reasoning capabilities in the real world. However, the generated text often suffers from inaccurate grounding in the visual input, resulting in errors such as hallucinating nonexistent scene elements, missing significant parts of the scene, and inferring incorrect attributes and relationships between objects. To address these issues, we introduce a novel framework, ViGoR (Visual Grounding Through Fine-Grained Reward Modeling) that utilizes fine-grained reward modeling to significantly enhance the visual grounding of LVLMs over pre-trained baselines. This improvement is efficiently achieved using much cheaper human evaluations instead of full supervisions, as well as automated methods. We show the effectiveness of our approach through numerous metrics on several benchmarks. Additionally, we construct a comprehensive and challenging dataset specifically designed to validate the visual grounding capabilities of LVLMs. Finally, we plan to release our human annotation comprising approximately 16,000 images and generated text pairs with fine-grained evaluations to contribute to related research in the community.

Title: ContPhy: Continuum Physical Concept Learning and Reasoning from Videos

• Authors: Zhicheng Zheng, Xin Yan, Zhenfang Chen, Jingzhou Wang, Qin Zhi Eddie Lim, Joshua B. Tenenbaum, Chuang Gan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06119
• Pdf URL: https://arxiv.org/pdf/2402.06119
• Copy Paste: [[2402.06119]] ContPhy: Continuum Physical Concept Learning and Reasoning from Videos(https://arxiv.org/abs/2402.06119)
• Keywords: large language model
• Abstract: We introduce the Continuum Physical Dataset (ContPhy), a novel benchmark for assessing machine physical commonsense. ContPhy complements existing physical reasoning benchmarks by encompassing the inference of diverse physical properties, such as mass and density, across various scenarios and predicting corresponding dynamics. We evaluated a range of AI models and found that they still struggle to achieve satisfactory performance on ContPhy, which shows that the current AI models still lack physical commonsense for the continuum, especially soft-bodies, and illustrates the value of the proposed dataset. We also introduce an oracle model (ContPRO) that marries the particle-based physical dynamic models with the recent large language models, which enjoy the advantages of both models, precise dynamic predictions, and interpretable reasoning. ContPhy aims to spur progress in perception and reasoning within diverse physical settings, narrowing the divide between human and machine intelligence in understanding the physical world. Project page: https://physical-reasoning-project.github.io.

Title: Exploring Group and Symmetry Principles in Large Language Models

• Authors: Shima Imani, Hamid Palangi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06120
• Pdf URL: https://arxiv.org/pdf/2402.06120
• Copy Paste: [[2402.06120]] Exploring Group and Symmetry Principles in Large Language Models(https://arxiv.org/abs/2402.06120)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive performance across a wide range of applications; however, assessing their reasoning capabilities remains a significant challenge. In this paper, we introduce a framework grounded in group and symmetry principles, which have played a crucial role in fields such as physics and mathematics, and offer another way to evaluate their capabilities. While the proposed framework is general, to showcase the benefits of employing these properties, we focus on arithmetic reasoning and investigate the performance of these models on four group properties: closure, identity, inverse, and associativity. Our findings reveal that LLMs studied in this work struggle to preserve group properties across different test regimes. In the closure test, we observe biases towards specific outputs and an abrupt degradation in their performance from 100% to 0% after a specific sequence length. They also perform poorly in the identity test, which represents adding irrelevant information in the context, and show sensitivity when subjected to inverse test, which examines the robustness of the model with respect to negation. In addition, we demonstrate that breaking down problems into smaller steps helps LLMs in the associativity test that we have conducted. To support these tests we have developed a synthetic dataset which will be released.

Title: Iterated Denoising Energy Matching for Sampling from Boltzmann Densities

• Authors: Tara Akhound-Sadegh, Jarrid Rector-Brooks, Avishek Joey Bose, Sarthak Mittal, Pablo Lemos, Cheng-Hao Liu, Marcin Sendera, Siamak Ravanbakhsh, Gauthier Gidel, Yoshua Bengio, Nikolay Malkin, Alexander Tong
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.06121
• Pdf URL: https://arxiv.org/pdf/2402.06121
• Copy Paste: [[2402.06121]] Iterated Denoising Energy Matching for Sampling from Boltzmann Densities(https://arxiv.org/abs/2402.06121)
• Keywords: diffusion
• Abstract: Efficiently generating statistically independent samples from an unnormalized probability distribution, such as equilibrium samples of many-body systems, is a foundational problem in science. In this paper, we propose Iterated Denoising Energy Matching (iDEM), an iterative algorithm that uses a novel stochastic score matching objective leveraging solely the energy function and its gradient -- and no data samples -- to train a diffusion-based sampler. Specifically, iDEM alternates between (I) sampling regions of high model density from a diffusion-based sampler and (II) using these samples in our stochastic matching objective to further improve the sampler. iDEM is scalable to high dimensions as the inner matching objective, is simulation-free, and requires no MCMC samples. Moreover, by leveraging the fast mode mixing behavior of diffusion, iDEM smooths out the energy landscape enabling efficient exploration and learning of an amortized sampler. We evaluate iDEM on a suite of tasks ranging from standard synthetic energy functions to invariant $n$-body particle systems. We show that the proposed approach achieves state-of-the-art performance on all metrics and trains $2-5\times$ faster, which allows it to be the first method to train using energy on the challenging $55$-particle Lennard-Jones system.

Title: Language Model Sentence Completion with a Parser-Driven Rhetorical Control Method

• Authors: Joshua Zingale, Jugal Kalita
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06125
• Pdf URL: https://arxiv.org/pdf/2402.06125
• Copy Paste: [[2402.06125]] Language Model Sentence Completion with a Parser-Driven Rhetorical Control Method(https://arxiv.org/abs/2402.06125)
• Keywords: large language model
• Abstract: Controlled text generation (CTG) seeks to guide large language model (LLM) output to produce text that conforms to desired criteria. The current study presents a novel CTG algorithm that enforces adherence toward specific rhetorical relations in an LLM sentence-completion context by a parser-driven decoding scheme that requires no model fine-tuning. The method is validated both with automatic and human evaluation. The code is accessible on GitHub.

Title: Learn To be Efficient: Build Structured Sparsity in Large Language Models

• Authors: Haizhong Zheng, Xiaoyan Bai, Beidi Chen, Fan Lai, Atul Prakash
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06126
• Pdf URL: https://arxiv.org/pdf/2402.06126
• Copy Paste: [[2402.06126]] Learn To be Efficient: Build Structured Sparsity in Large Language Models(https://arxiv.org/abs/2402.06126)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable success with their billion-level parameters, yet they incur high inference overheads. The emergence of activation sparsity in LLMs provides a natural approach to reduce this cost by involving only parts of the parameters for inference. Existing methods only focus on utilizing this naturally formed activation sparsity, overlooking the potential for further amplifying this inherent sparsity. In this paper, we hypothesize that LLMs can learn to be efficient by achieving more structured activation sparsity.To achieve this, we introduce a novel algorithm, Learn-To-be-Efficient (LTE), designed to train efficiency-aware LLMs to learn to activate fewer neurons and achieve a better trade-off between sparsity and performance. Furthermore, unlike SOTA MoEfication methods, which mainly focus on ReLU-based models, LTE can also be applied to LLMs like GPT and LLaMA with soft activation functions. We evaluate LTE on four models and eleven datasets. The experiments show that LTE achieves a better trade-off between sparsity and task performance. For instance, LTE with LLaMA provides a 1.83x-2.59x FLOPs speed-up on language generation tasks, outperforming the state-of-the-art methods.

Title: TETRIS: Towards Exploring the Robustness of Interactive Segmentation

• Authors: Andrey Moskalenko, Vlad Shakhuro, Anna Vorontsova, Anton Konushin, Anton Antonov, Alexander Krapukhin, Denis Shepelev, Konstantin Soshin
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.06132
• Pdf URL: https://arxiv.org/pdf/2402.06132
• Copy Paste: [[2402.06132]] TETRIS: Towards Exploring the Robustness of Interactive Segmentation(https://arxiv.org/abs/2402.06132)
• Keywords: attack, robust, segmentation
• Abstract: Interactive segmentation methods rely on user inputs to iteratively update the selection mask. A click specifying the object of interest is arguably the most simple and intuitive interaction type, and thereby the most common choice for interactive segmentation. However, user clicking patterns in the interactive segmentation context remain unexplored. Accordingly, interactive segmentation evaluation strategies rely more on intuition and common sense rather than empirical studies (e.g., assuming that users tend to click in the center of the area with the largest error). In this work, we conduct a real user study to investigate real user clicking patterns. This study reveals that the intuitive assumption made in the common evaluation strategy may not hold. As a result, interactive segmentation models may show high scores in the standard benchmarks, but it does not imply that they would perform well in a real world scenario. To assess the applicability of interactive segmentation methods, we propose a novel evaluation strategy providing a more comprehensive analysis of a model's performance. To this end, we propose a methodology for finding extreme user inputs by a direct optimization in a white-box adversarial attack on the interactive segmentation model. Based on the performance with such adversarial user inputs, we assess the robustness of interactive segmentation models w.r.t click positions. Besides, we introduce a novel benchmark for measuring the robustness of interactive segmentation, and report the results of an extensive evaluation of dozens of models.

Title: Jointly Learning Representations for Map Entities via Heterogeneous Graph Contrastive Learning

• Authors: Jiawei Jiang, Yifan Yang, Jingyuan Wang, Junjie Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06135
• Pdf URL: https://arxiv.org/pdf/2402.06135
• Copy Paste: [[2402.06135]] Jointly Learning Representations for Map Entities via Heterogeneous Graph Contrastive Learning(https://arxiv.org/abs/2402.06135)
• Keywords: transformer
• Abstract: The electronic map plays a crucial role in geographic information systems, serving various urban managerial scenarios and daily life services. Developing effective Map Entity Representation Learning (MERL) methods is crucial to extracting embedding information from electronic maps and converting map entities into representation vectors for downstream applications. However, existing MERL methods typically focus on one specific category of map entities, such as POIs, road segments, or land parcels, which is insufficient for real-world diverse map-based applications and might lose latent structural and semantic information interacting between entities of different types. Moreover, using representations generated by separate models for different map entities can introduce inconsistencies. Motivated by this, we propose a novel method named HOME-GCL for learning representations of multiple categories of map entities. Our approach utilizes a heterogeneous map entity graph (HOME graph) that integrates both road segments and land parcels into a unified framework. A HOME encoder with parcel-segment joint feature encoding and heterogeneous graph transformer is then deliberately designed to convert segments and parcels into representation vectors. Moreover, we introduce two types of contrastive learning tasks, namely intra-entity and inter-entity tasks, to train the encoder in a self-supervised manner. Extensive experiments on three large-scale datasets covering road segment-based, land parcel-based, and trajectory-based tasks demonstrate the superiority of our approach. To the best of our knowledge, HOME-GCL is the first attempt to jointly learn representations for road segments and land parcels using a unified model.

Title: On the Privacy of Selection Mechanisms with Gaussian Noise

• Authors: Jonathan Lebensold, Doina Precup, Borja Balle
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06137
• Pdf URL: https://arxiv.org/pdf/2402.06137
• Copy Paste: [[2402.06137]] On the Privacy of Selection Mechanisms with Gaussian Noise(https://arxiv.org/abs/2402.06137)
• Keywords: privacy
• Abstract: Report Noisy Max and Above Threshold are two classical differentially private (DP) selection mechanisms. Their output is obtained by adding noise to a sequence of low-sensitivity queries and reporting the identity of the query whose (noisy) answer satisfies a certain condition. Pure DP guarantees for these mechanisms are easy to obtain when Laplace noise is added to the queries. On the other hand, when instantiated using Gaussian noise, standard analyses only yield approximate DP guarantees despite the fact that the outputs of these mechanisms lie in a discrete space. In this work, we revisit the analysis of Report Noisy Max and Above Threshold with Gaussian noise and show that, under the additional assumption that the underlying queries are bounded, it is possible to provide pure ex-ante DP bounds for Report Noisy Max and pure ex-post DP bounds for Above Threshold. The resulting bounds are tight and depend on closed-form expressions that can be numerically evaluated using standard methods. Empirically we find these lead to tighter privacy accounting in the high privacy, low data regime. Further, we propose a simple privacy filter for composing pure ex-post DP guarantees, and use it to derive a fully adaptive Gaussian Sparse Vector Technique mechanism. Finally, we provide experiments on mobility and energy consumption datasets demonstrating that our Sparse Vector Technique is practically competitive with previous approaches and requires less hyper-parameter tuning.

Title: HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splatting

• Authors: Zhenglin Zhou, Fan Ma, Hehe Fan, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06149
• Pdf URL: https://arxiv.org/pdf/2402.06149
• Copy Paste: [[2402.06149]] HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splatting(https://arxiv.org/abs/2402.06149)
• Keywords: diffusion
• Abstract: Creating digital avatars from textual prompts has long been a desirable yet challenging task. Despite the promising outcomes obtained through 2D diffusion priors in recent works, current methods face challenges in achieving high-quality and animated avatars effectively. In this paper, we present $\textbf{HeadStudio}$, a novel framework that utilizes 3D Gaussian splatting to generate realistic and animated avatars from text prompts. Our method drives 3D Gaussians semantically to create a flexible and achievable appearance through the intermediate FLAME representation. Specifically, we incorporate the FLAME into both 3D representation and score distillation: 1) FLAME-based 3D Gaussian splatting, driving 3D Gaussian points by rigging each point to a FLAME mesh. 2) FLAME-based score distillation sampling, utilizing FLAME-based fine-grained control signal to guide score distillation from the text prompt. Extensive experiments demonstrate the efficacy of HeadStudio in generating animatable avatars from textual prompts, exhibiting visually appealing appearances. The avatars are capable of rendering high-quality real-time ($\geq 40$ fps) novel views at a resolution of 1024. They can be smoothly controlled by real-world speech and video. We hope that HeadStudio can advance digital avatar creation and that the present method can widely be applied across various domains.

Title: Passwords Are Meant to Be Secret: A Practical Secure Password Entry Channel for Web Browsers

• Authors: Anuj Gautam, Tarun Kumar Yadav, Kent Seamons, Scott Ruoti
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06159
• Pdf URL: https://arxiv.org/pdf/2402.06159
• Copy Paste: [[2402.06159]] Passwords Are Meant to Be Secret: A Practical Secure Password Entry Channel for Web Browsers(https://arxiv.org/abs/2402.06159)
• Keywords: secure, security, steal
• Abstract: Password-based authentication faces various security and usability issues. Password managers help alleviate some of these issues by enabling users to manage their passwords effectively. However, malicious client-side scripts and browser extensions can steal passwords after they have been autofilled by the manager into the web page. In this paper, we explore what role the password manager can take in preventing the theft of autofilled credentials without requiring a change to user behavior. To this end, we identify a threat model for password exfiltration and then use this threat model to explore the design space for secure password entry implemented using a password manager. We identify five potential designs that address this issue, each with varying security and deployability tradeoffs. Our analysis shows the design that best balances security and usability is for the manager to autofill a fake password and then rely on the browser to replace the fake password with the actual password immediately before the web request is handed over to the operating system to be transmitted over the network. This removes the ability for malicious client-side scripts or browser extensions to access and exfiltrate the real password. We implement our design in the Firefox browser and conduct experiments, which show that it successfully thwarts malicious scripts and extensions on 97\% of the Alexa top 1000 websites, while also maintaining the capability to revert to default behavior on the remaining websites, avoiding functionality regressions. Most importantly, this design is transparent to users, requiring no change to user behavior.

Title: Pushing Boundaries: Mixup's Influence on Neural Collapse

• Authors: Quinn Fisher, Haoming Meng, Vardan Papyan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06171
• Pdf URL: https://arxiv.org/pdf/2402.06171
• Copy Paste: [[2402.06171]] Pushing Boundaries: Mixup's Influence on Neural Collapse(https://arxiv.org/abs/2402.06171)
• Keywords: robust
• Abstract: Mixup is a data augmentation strategy that employs convex combinations of training instances and their respective labels to augment the robustness and calibration of deep neural networks. Despite its widespread adoption, the nuanced mechanisms that underpin its success are not entirely understood. The observed phenomenon of Neural Collapse, where the last-layer activations and classifier of deep networks converge to a simplex equiangular tight frame (ETF), provides a compelling motivation to explore whether mixup induces alternative geometric configurations and whether those could explain its success. In this study, we delve into the last-layer activations of training data for deep networks subjected to mixup, aiming to uncover insights into its operational efficacy. Our investigation, spanning various architectures and dataset pairs, reveals that mixup's last-layer activations predominantly converge to a distinctive configuration different than one might expect. In this configuration, activations from mixed-up examples of identical classes align with the classifier, while those from different classes delineate channels along the decision boundary. Moreover, activations in earlier layers exhibit patterns, as if trained with manifold mixup. These findings are unexpected, as mixed-up features are not simple convex combinations of feature class means (as one might get, for example, by training mixup with the mean squared error loss). By analyzing this distinctive geometric configuration, we elucidate the mechanisms by which mixup enhances model calibration. To further validate our empirical observations, we conduct a theoretical analysis under the assumption of an unconstrained features model, utilizing the mixup loss. Through this, we characterize and derive the optimal last-layer features under the assumption that the classifier forms a simplex ETF.

Title: A self-supervised framework for learning whole slide representations

• Authors: Xinhai Hou, Cheng Jiang, Akhil Kondepudi, Yiwei Lyu, Asadur Zaman Chowdury, Honglak Lee, Todd C. Hollon
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06188
• Pdf URL: https://arxiv.org/pdf/2402.06188
• Copy Paste: [[2402.06188]] A self-supervised framework for learning whole slide representations(https://arxiv.org/abs/2402.06188)
• Keywords: transformer
• Abstract: Whole slide imaging is fundamental to biomedical microscopy and computational pathology. However, whole slide images (WSIs) present a complex computer vision challenge due to their gigapixel size, diverse histopathologic features, spatial heterogeneity, and limited/absent data annotations. These challenges highlight that supervised training alone can result in suboptimal whole slide representations. Self-supervised representation learning can achieve high-quality WSI visual feature learning for downstream diagnostic tasks, such as cancer diagnosis or molecular genetic prediction. Here, we present a general self-supervised whole slide learning (S3L) framework for gigapixel-scale self-supervision of WSIs. S3L combines data transformation strategies from transformer-based vision and language modeling into a single unified framework to generate paired views for self-supervision. S3L leverages the inherent regional heterogeneity, histologic feature variability, and information redundancy within WSIs to learn high-quality whole-slide representations. We benchmark S3L visual representations on two diagnostic tasks for two biomedical microscopy modalities. S3L significantly outperforms WSI baselines for cancer diagnosis and genetic mutation prediction. Additionally, S3L achieves good performance using both in-domain and out-of-distribution patch encoders, demonstrating good flexibility and generalizability.

Title: Masked LoGoNet: Fast and Accurate 3D Image Analysis for Medical Domain

• Authors: Amin Karimi Monsefi, Payam Karisani, Mengxi Zhou, Stacey Choi, Nathan Doble, Heng Ji, Srinivasan Parthasarathy, Rajiv Ramnath
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06190
• Pdf URL: https://arxiv.org/pdf/2402.06190
• Copy Paste: [[2402.06190]] Masked LoGoNet: Fast and Accurate 3D Image Analysis for Medical Domain(https://arxiv.org/abs/2402.06190)
• Keywords: extraction, transformer, segmentation
• Abstract: Standard modern machine-learning-based imaging methods have faced challenges in medical applications due to the high cost of dataset construction and, thereby, the limited labeled training data available. Additionally, upon deployment, these methods are usually used to process a large volume of data on a daily basis, imposing a high maintenance cost on medical facilities. In this paper, we introduce a new neural network architecture, termed LoGoNet, with a tailored self-supervised learning (SSL) method to mitigate such challenges. LoGoNet integrates a novel feature extractor within a U-shaped architecture, leveraging Large Kernel Attention (LKA) and a dual encoding strategy to capture both long-range and short-range feature dependencies adeptly. This is in contrast to existing methods that rely on increasing network capacity to enhance feature extraction. This combination of novel techniques in our model is especially beneficial in medical image segmentation, given the difficulty of learning intricate and often irregular body organ shapes, such as the spleen. Complementary, we propose a novel SSL method tailored for 3D images to compensate for the lack of large labeled datasets. The method combines masking and contrastive learning techniques within a multi-task learning framework and is compatible with both Vision Transformer (ViT) and CNN-based models. We demonstrate the efficacy of our methods in numerous tasks across two standard datasets (i.e., BTCV and MSD). Benchmark comparisons with eight state-of-the-art models highlight LoGoNet's superior performance in both inference time and accuracy.

Title: The Berkeley Single Cell Computational Microscopy (BSCCM) Dataset

• Authors: Henry Pinkard, Cherry Liu, Fanice Nyatigo, Daniel A. Fletcher, Laura Waller
• Subjects: cs.CV, cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2402.06191
• Pdf URL: https://arxiv.org/pdf/2402.06191
• Copy Paste: [[2402.06191]] The Berkeley Single Cell Computational Microscopy (BSCCM) Dataset(https://arxiv.org/abs/2402.06191)
• Keywords: robust
• Abstract: Computational microscopy, in which hardware and algorithms of an imaging system are jointly designed, shows promise for making imaging systems that cost less, perform more robustly, and collect new types of information. Often, the performance of computational imaging systems, especially those that incorporate machine learning, is sample-dependent. Thus, standardized datasets are an essential tool for comparing the performance of different approaches. Here, we introduce the Berkeley Single Cell Computational Microscopy (BSCCM) dataset, which contains over ~12,000,000 images of 400,000 of individual white blood cells. The dataset contains images captured with multiple illumination patterns on an LED array microscope and fluorescent measurements of the abundance of surface proteins that mark different cell types. We hope this dataset will provide a valuable resource for the development and testing of new algorithms in computational microscopy and computer vision with practical biomedical applications.

Title: Large Language Models: A Survey

• Authors: Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, Jianfeng Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06196
• Pdf URL: https://arxiv.org/pdf/2402.06196
• Copy Paste: [[2402.06196]] Large Language Models: A Survey(https://arxiv.org/abs/2402.06196)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.

Title: The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate

• Authors: Juhyun Oh, Eunsu Kim, Inha Cha, Alice Oh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06204
• Pdf URL: https://arxiv.org/pdf/2402.06204
• Copy Paste: [[2402.06204]] The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate(https://arxiv.org/abs/2402.06204)
• Keywords: generative, large language model
• Abstract: This paper explores the assumption that Large Language Models (LLMs) skilled in generation tasks are equally adept as evaluators. We assess the performance of three LLMs and one open-source LM in Question-Answering (QA) and evaluation tasks using the TriviaQA (Joshi et al., 2017) dataset. Results indicate a significant disparity, with LLMs exhibiting lower performance in evaluation tasks compared to generation tasks. Intriguingly, we discover instances of unfaithful evaluation where models accurately evaluate answers in areas where they lack competence, underscoring the need to examine the faithfulness and trustworthiness of LLMs as evaluators. This study contributes to the understanding of "the Generative AI Paradox" (West et al., 2023), highlighting a need to explore the correlation between generative excellence and evaluation proficiency, and the necessity to scrutinize the faithfulness aspect in model evaluations.

Title: Multi-source-free Domain Adaptation via Uncertainty-aware Adaptive Distillation

• Authors: Yaxuan Song, Jianan Fan, Dongnan Liu, Weidong Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06213
• Pdf URL: https://arxiv.org/pdf/2402.06213
• Copy Paste: [[2402.06213]] Multi-source-free Domain Adaptation via Uncertainty-aware Adaptive Distillation(https://arxiv.org/abs/2402.06213)
• Keywords: privacy
• Abstract: Source-free domain adaptation (SFDA) alleviates the domain discrepancy among data obtained from domains without accessing the data for the awareness of data privacy. However, existing conventional SFDA methods face inherent limitations in medical contexts, where medical data are typically collected from multiple institutions using various equipment. To address this problem, we propose a simple yet effective method, named Uncertainty-aware Adaptive Distillation (UAD) for the multi-source-free unsupervised domain adaptation (MSFDA) setting. UAD aims to perform well-calibrated knowledge distillation from (i) model level to deliver coordinated and reliable base model initialisation and (ii) instance level via model adaptation guided by high-quality pseudo-labels, thereby obtaining a high-performance target domain model. To verify its general applicability, we evaluate UAD on two image-based diagnosis benchmarks among two multi-centre datasets, where our method shows a significant performance gain compared with existing works. The code will be available soon.

Title: ResumeFlow: An LLM-facilitated Pipeline for Personalized Resume Generation and Refinement

• Authors: Saurabh Bhausaheb Zinjad, Amrita Bhattacharjee, Amey Bhilegaonkar, Huan Liu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.06221
• Pdf URL: https://arxiv.org/pdf/2402.06221
• Copy Paste: [[2402.06221]] ResumeFlow: An LLM-facilitated Pipeline for Personalized Resume Generation and Refinement(https://arxiv.org/abs/2402.06221)
• Keywords: extraction, large language model
• Abstract: Crafting the ideal, job-specific resume is a challenging task for many job applicants, especially for early-career applicants. While it is highly recommended that applicants tailor their resume to the specific role they are applying for, manually tailoring resumes to job descriptions and role-specific requirements is often (1) extremely time-consuming, and (2) prone to human errors. Furthermore, performing such a tailoring step at scale while applying to several roles may result in a lack of quality of the edited resumes. To tackle this problem, in this demo paper, we propose ResumeFlow: a Large Language Model (LLM) aided tool that enables an end user to simply provide their detailed resume and the desired job posting, and obtain a personalized resume specifically tailored to that specific job posting in the matter of a few seconds. Our proposed pipeline leverages the language understanding and information extraction capabilities of state-of-the-art LLMs such as OpenAI's GPT-4 and Google's Gemini, in order to (1) extract details from a job description, (2) extract role-specific details from the user-provided resume, and then (3) use these to refine and generate a role-specific resume for the user. Our easy-to-use tool leverages the user-chosen LLM in a completely off-the-shelf manner, thus requiring no fine-tuning. We demonstrate the effectiveness of our tool via a video demo and propose novel task-specific evaluation metrics to control for alignment and hallucination. Our tool is available at https://job-aligned-resume.streamlit.app.

Title: Revealing Multimodal Contrastive Representation Learning through Latent Partial Causal Models

• Authors: Yuhang Liu, Zhen Zhang, Dong Gong, Biwei Huang, Mingming Gong, Anton van den Hengel, Kun Zhang, Javen Qinfeng Shi
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.06223
• Pdf URL: https://arxiv.org/pdf/2402.06223
• Copy Paste: [[2402.06223]] Revealing Multimodal Contrastive Representation Learning through Latent Partial Causal Models(https://arxiv.org/abs/2402.06223)
• Keywords: robust
• Abstract: Multimodal contrastive representation learning methods have proven successful across a range of domains, partly due to their ability to generate meaningful shared representations of complex phenomena. To enhance the depth of analysis and understanding of these acquired representations, we introduce a unified causal model specifically designed for multimodal data. By examining this model, we show that multimodal contrastive representation learning excels at identifying latent coupled variables within the proposed unified model, up to linear or permutation transformations resulting from different assumptions. Our findings illuminate the potential of pre-trained multimodal models, eg, CLIP, in learning disentangled representations through a surprisingly simple yet highly effective tool: linear independent component analysis. Experiments demonstrate the robustness of our findings, even when the assumptions are violated, and validate the effectiveness of the proposed method in learning disentangled representations.

Title: Quantifying and Enhancing Multi-modal Robustness with Modality Preference

• Authors: Zequn Yang, Yake Wei, Ce Liang, Di Hu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.06244
• Pdf URL: https://arxiv.org/pdf/2402.06244
• Copy Paste: [[2402.06244]] Quantifying and Enhancing Multi-modal Robustness with Modality Preference(https://arxiv.org/abs/2402.06244)
• Keywords: attack, robust
• Abstract: Multi-modal models have shown a promising capability to effectively integrate information from various sources, yet meanwhile, they are found vulnerable to pervasive perturbations, such as uni-modal attacks and missing conditions. To counter these perturbations, robust multi-modal representations are highly expected, which are positioned well away from the discriminative multi-modal decision boundary. In this paper, different from conventional empirical studies, we focus on a commonly used joint multi-modal framework and theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness. This discovery can further explain the limitation of multi-modal robustness and the phenomenon that multi-modal models are often vulnerable to attacks on the specific modality. Moreover, our analysis reveals how the widespread issue, that the model has different preferences for modalities, limits the multi-modal robustness by influencing the essential components and could lead to attacks on the specific modality highly effective. Inspired by our theoretical finding, we introduce a training procedure called Certifiable Robust Multi-modal Training (CRMT), which can alleviate this influence from modality preference and explicitly regulate essential components to significantly improve robustness in a certifiable manner. Our method demonstrates substantial improvements in performance and robustness compared with existing methods. Furthermore, our training procedure can be easily extended to enhance other robust training strategies, highlighting its credibility and flexibility.

Title: Anomaly Unveiled: Securing Image Classification against Adversarial Patch Attacks

• Authors: Nandish Chattopadhyay, Amira Guesmi, Muhammad Shafique
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06249
• Pdf URL: https://arxiv.org/pdf/2402.06249
• Copy Paste: [[2402.06249]] Anomaly Unveiled: Securing Image Classification against Adversarial Patch Attacks(https://arxiv.org/abs/2402.06249)
• Keywords: defense, attack, robust
• Abstract: Adversarial patch attacks pose a significant threat to the practical deployment of deep learning systems. However, existing research primarily focuses on image pre-processing defenses, which often result in reduced classification accuracy for clean images and fail to effectively counter physically feasible attacks. In this paper, we investigate the behavior of adversarial patches as anomalies within the distribution of image information and leverage this insight to develop a robust defense strategy. Our proposed defense mechanism utilizes a clustering-based technique called DBSCAN to isolate anomalous image segments, which is carried out by a three-stage pipeline consisting of Segmenting, Isolating, and Blocking phases to identify and mitigate adversarial noise. Upon identifying adversarial components, we neutralize them by replacing them with the mean pixel value, surpassing alternative replacement options. Our model-agnostic defense mechanism is evaluated across multiple models and datasets, demonstrating its effectiveness in countering various adversarial patch attacks in image classification tasks. Our proposed approach significantly improves accuracy, increasing from 38.8\% without the defense to 67.1\% with the defense against LaVAN and GoogleAp attacks, surpassing prominent state-of-the-art methods such as LGS (53.86\%) and Jujutsu (60\%)

Title: Studious Bob Fight Back Against Jailbreaking via Prompt Adversarial Tuning

• Authors: Yichuan Mo, Yuji Wang, Zeming Wei, Yisen Wang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06255
• Pdf URL: https://arxiv.org/pdf/2402.06255
• Copy Paste: [[2402.06255]] Studious Bob Fight Back Against Jailbreaking via Prompt Adversarial Tuning(https://arxiv.org/abs/2402.06255)
• Keywords: security, protect, defense, attack, large language model
• Abstract: Although Large Language Models (LLMs) have achieved tremendous success in various applications, they are also susceptible to certain prompts that can induce them to bypass built-in safety measures and provide dangerous or illegal content, a phenomenon known as jailbreak. To protect LLMs from producing harmful information, various defense strategies are proposed, with most focusing on content filtering or adversarial training of models. In this paper, we propose an approach named Prompt Adversarial Tuning (PAT) to train a defense control mechanism, which is then embedded as a prefix to user prompts to implement our defense strategy. We design a training process similar to adversarial training to achieve our optimized goal, alternating between updating attack and defense controls. To our knowledge, we are the first to implement defense from the perspective of prompt tuning. Once employed, our method will hardly impact the operational efficiency of LLMs. Experiments show that our method is effective in both black-box and white-box settings, reducing the success rate of advanced attacks to nearly 0 while maintaining the benign answer rate of 80% to simple benign questions. Our work might potentially chart a new perspective for future explorations in LLM security.

Title: On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference

• Authors: Siyu Ren, Kenny Q. Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06262
• Pdf URL: https://arxiv.org/pdf/2402.06262
• Copy Paste: [[2402.06262]] On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference(https://arxiv.org/abs/2402.06262)
• Keywords: robust, generative, large language model
• Abstract: Despite the recent success associated with Large Language Models~(LLMs), they are notably cost-prohibitive to deploy in resource-constrained environments due to their excessive memory and computational demands. In addition to model parameters, the key-value cache is also stored in GPU memory, growing linearly with batch size and sequence length. As a remedy, recent works have proposed various eviction policies for maintaining the overhead of key-value cache under a given budget. This paper embarks on the efficacy of existing eviction policies in terms of \textit{importance score calculation} and \textit{eviction scope construction}. We identify the deficiency of prior policies in these two aspects and introduce RoCo, a \underline{r}\underline{o}bust \underline{c}ache \underline{o}mission policy based on temporal attention scores and robustness measures. Extensive experimentation spanning prefilling and auto-regressive decoding stages validates the superiority of RoCo. Finally, we release EasyKV, a versatile software package dedicated to user-friendly key-value constrained generative inference. Code available at \url{https://github.com/DRSY/EasyKV}.

Title: Evaluating Membership Inference Attacks and Defenses in Federated Learning

• Authors: Gongxi Zhu, Donghao Li, Hanlin Gu, Yuxing Han, Yuan Yao, Lixin Fan, Qiang Yang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06289
• Pdf URL: https://arxiv.org/pdf/2402.06289
• Copy Paste: [[2402.06289]] Evaluating Membership Inference Attacks and Defenses in Federated Learning(https://arxiv.org/abs/2402.06289)
• Keywords: privacy, defense, attack, membership infer, federate
• Abstract: Membership Inference Attacks (MIAs) pose a growing threat to privacy preservation in federated learning. The semi-honest attacker, e.g., the server, may determine whether a particular sample belongs to a target client according to the observed model information. This paper conducts an evaluation of existing MIAs and corresponding defense strategies. Our evaluation on MIAs reveals two important findings about the trend of MIAs. Firstly, combining model information from multiple communication rounds (Multi-temporal) enhances the overall effectiveness of MIAs compared to utilizing model information from a single epoch. Secondly, incorporating models from non-target clients (Multi-spatial) significantly improves the effectiveness of MIAs, particularly when the clients' data is homogeneous. This highlights the importance of considering the temporal and spatial model information in MIAs. Next, we assess the effectiveness via privacy-utility tradeoff for two type defense mechanisms against MIAs: Gradient Perturbation and Data Replacement. Our results demonstrate that Data Replacement mechanisms achieve a more optimal balance between preserving privacy and maintaining model utility. Therefore, we recommend the adoption of Data Replacement methods as a defense strategy against MIAs. Our code is available in https://github.com/Liar-Mask/FedMIA.

Title: Multimodal Interpretable Data-Driven Models for Early Prediction of Antimicrobial Multidrug Resistance Using Multivariate Time-Series

• Authors: Sergio Martínez-Agüero, Antonio G. Marques, Inmaculada Mora-Jiménez, Joaquín Alvárez-Rodríguez, Cristina Soguero-Ruiza
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2402.06295
• Pdf URL: https://arxiv.org/pdf/2402.06295
• Copy Paste: [[2402.06295]] Multimodal Interpretable Data-Driven Models for Early Prediction of Antimicrobial Multidrug Resistance Using Multivariate Time-Series(https://arxiv.org/abs/2402.06295)
• Keywords: interpretability
• Abstract: Electronic health records (EHR) is an inherently multimodal register of the patient's health status characterized by static data and multivariate time series (MTS). While MTS are a valuable tool for clinical prediction, their fusion with other data modalities can possibly result in more thorough insights and more accurate results. Deep neural networks (DNNs) have emerged as fundamental tools for identifying and defining underlying patterns in the healthcare domain. However, fundamental improvements in interpretability are needed for DNN models to be widely used in the clinical setting. In this study, we present an approach built on a collection of interpretable multimodal data-driven models that may anticipate and understand the emergence of antimicrobial multidrug resistance (AMR) germs in the intensive care unit (ICU) of the University Hospital of Fuenlabrada (Madrid, Spain). The profile and initial health status of the patient are modeled using static variables, while the evolution of the patient's health status during the ICU stay is modeled using several MTS, including mechanical ventilation and antibiotics intake. The multimodal DNNs models proposed in this paper include interpretable principles in addition to being effective at predicting AMR and providing an explainable prediction support system for AMR in the ICU. Furthermore, our proposed methodology based on multimodal models and interpretability schemes can be leveraged in additional clinical problems dealing with EHR data, broadening the impact and applicability of our results.

Title: Multisource Semisupervised Adversarial Domain Generalization Network for Cross-Scene Sea\textendash Land Clutter Classification

• Authors: Xiaoxuan Zhang, Quan Pan, Salvador García
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06315
• Pdf URL: https://arxiv.org/pdf/2402.06315
• Copy Paste: [[2402.06315]] Multisource Semisupervised Adversarial Domain Generalization Network for Cross-Scene Sea\textendash Land Clutter Classification(https://arxiv.org/abs/2402.06315)
• Keywords: generative
• Abstract: Deep learning (DL)-based sea\textendash land clutter classification for sky-wave over-the-horizon-radar (OTHR) has become a novel research topic. In engineering applications, real-time predictions of sea\textendash land clutter with existing distribution discrepancies are crucial. To solve this problem, this article proposes a novel Multisource Semisupervised Adversarial Domain Generalization Network (MSADGN) for cross-scene sea\textendash land clutter classification. MSADGN can extract domain-invariant and domain-specific features from one labeled source domain and multiple unlabeled source domains, and then generalize these features to an arbitrary unseen target domain for real-time prediction of sea\textendash land clutter. Specifically, MSADGN consists of three modules: domain-related pseudolabeling module, domain-invariant module, and domain-specific module. The first module introduces an improved pseudolabel method called domain-related pseudolabel, which is designed to generate reliable pseudolabels to fully exploit unlabeled source domains. The second module utilizes a generative adversarial network (GAN) with a multidiscriminator to extract domain-invariant features, to enhance the model's transferability in the target domain. The third module employs a parallel multiclassifier branch to extract domain-specific features, to enhance the model's discriminability in the target domain. The effectiveness of our method is validated in twelve domain generalizations (DG) scenarios. Meanwhile, we selected 10 state-of-the-art DG methods for comparison. The experimental results demonstrate the superiority of our method.

Title: TimEHR: Image-based Time Series Generation for Electronic Health Records

• Authors: Hojjat Karami, Mary-Anne Hartley, David Atienza, Anisoara Ionescu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06318
• Pdf URL: https://arxiv.org/pdf/2402.06318
• Copy Paste: [[2402.06318]] TimEHR: Image-based Time Series Generation for Electronic Health Records(https://arxiv.org/abs/2402.06318)
• Keywords: privacy, generative
• Abstract: Time series in Electronic Health Records (EHRs) present unique challenges for generative models, such as irregular sampling, missing values, and high dimensionality. In this paper, we propose a novel generative adversarial network (GAN) model, TimEHR, to generate time series data from EHRs. In particular, TimEHR treats time series as images and is based on two conditional GANs. The first GAN generates missingness patterns, and the second GAN generates time series values based on the missingness pattern. Experimental results on three real-world EHR datasets show that TimEHR outperforms state-of-the-art methods in terms of fidelity, utility, and privacy metrics.

Title: InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

• Authors: Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, Dahua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06332
• Pdf URL: https://arxiv.org/pdf/2402.06332
• Copy Paste: [[2402.06332]] InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning(https://arxiv.org/abs/2402.06332)
• Keywords: large language model
• Abstract: The math abilities of large language models can represent their abstract reasoning ability. In this paper, we introduce and open-source our math reasoning LLMs InternLM-Math which is continue pre-trained from InternLM2. We unify chain-of-thought reasoning, reward modeling, formal reasoning, data augmentation, and code interpreter in a unified seq2seq format and supervise our model to be a versatile math reasoner, verifier, prover, and augmenter. These abilities can be used to develop the next math LLMs or self-iteration. InternLM-Math obtains open-sourced state-of-the-art performance under the setting of in-context learning, supervised fine-tuning, and code-assisted reasoning in various informal and formal benchmarks including GSM8K, MATH, Hungary math exam, MathBench-ZH, and MiniF2F. Our pre-trained model achieves 30.3 on the MiniF2F test set without fine-tuning. We further explore how to use LEAN to solve math problems and study its performance under the setting of multi-task learning which shows the possibility of using LEAN as a unified platform for solving and proving in math. Our models, codes, and data are released at \url{https://github.com/InternLM/InternLM-Math}.

Title: RareBench: Can LLMs Serve as Rare Diseases Specialists?

• Authors: Xuanzhong Chen, Xiaohao Mao, Qihan Guo, Lun Wang, Shuyang Zhang, Ting Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06341
• Pdf URL: https://arxiv.org/pdf/2402.06341
• Copy Paste: [[2402.06341]] RareBench: Can LLMs Serve as Rare Diseases Specialists?(https://arxiv.org/abs/2402.06341)
• Keywords: large language model
• Abstract: Generalist Large Language Models (LLMs), such as GPT-4, have shown considerable promise in various domains, including medical diagnosis. Rare diseases, affecting approximately 300 million people worldwide, often have unsatisfactory clinical diagnosis rates primarily due to a lack of experienced physicians and the complexity of differentiating among many rare diseases. In this context, recent news such as "ChatGPT correctly diagnosed a 4-year-old's rare disease after 17 doctors failed" underscore LLMs' potential, yet underexplored, role in clinically diagnosing rare diseases. To bridge this research gap, we introduce RareBench, a pioneering benchmark designed to systematically evaluate the capabilities of LLMs on 4 critical dimensions within the realm of rare diseases. Meanwhile, we have compiled the largest open-source dataset on rare disease patients, establishing a benchmark for future studies in this domain. To facilitate differential diagnosis of rare diseases, we develop a dynamic few-shot prompt methodology, leveraging a comprehensive rare disease knowledge graph synthesized from multiple knowledge bases, significantly enhancing LLMs' diagnostic performance. Moreover, we present an exhaustive comparative study of GPT-4's diagnostic capabilities against those of specialist physicians. Our experimental findings underscore the promising potential of integrating LLMs into the clinical diagnostic process for rare diseases. This paves the way for exciting possibilities in future advancements in this field.

Title: Fairness of Exposure in Online Restless Multi-armed Bandits

• Authors: Archit Sood, Shweta Jain, Sujit Gujar
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.06348
• Pdf URL: https://arxiv.org/pdf/2402.06348
• Copy Paste: [[2402.06348]] Fairness of Exposure in Online Restless Multi-armed Bandits(https://arxiv.org/abs/2402.06348)
• Keywords: fair
• Abstract: Restless multi-armed bandits (RMABs) generalize the multi-armed bandits where each arm exhibits Markovian behavior and transitions according to their transition dynamics. Solutions to RMAB exist for both offline and online cases. However, they do not consider the distribution of pulls among the arms. Studies have shown that optimal policies lead to unfairness, where some arms are not exposed enough. Existing works in fairness in RMABs focus heavily on the offline case, which diminishes their application in real-world scenarios where the environment is largely unknown. In the online scenario, we propose the first fair RMAB framework, where each arm receives pulls in proportion to its merit. We define the merit of an arm as a function of its stationary reward distribution. We prove that our algorithm achieves sublinear fairness regret in the single pull case $O(\sqrt{T\ln T})$, with $T$ being the total number of episodes. Empirically, we show that our algorithm performs well in the multi-pull scenario as well.

Title: Towards actionability for open medical imaging datasets: lessons from community-contributed platforms for data management and stewardship

• Authors: Amelia Jiménez-Sánchez, Natalia-Rozalia Avlona, Dovile Juodelyte, Théo Sourget, Caroline Vang-Larsen, Hubert Dariusz Zając, Veronika Cheplygina
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06353
• Pdf URL: https://arxiv.org/pdf/2402.06353
• Copy Paste: [[2402.06353]] Towards actionability for open medical imaging datasets: lessons from community-contributed platforms for data management and stewardship(https://arxiv.org/abs/2402.06353)
• Keywords: robust, fair
• Abstract: Medical imaging datasets are fundamental to artificial intelligence (AI) in healthcare. The accuracy, robustness and fairness of diagnostic algorithms depend on the data (and its quality) on which the models are trained and evaluated. Medical imaging datasets have become increasingly available to the public, and are often hosted on Community-Contributed Platforms (CCP), including private companies like Kaggle or HuggingFace. While open data is important to enhance the redistribution of data's public value, we find that the current CCP governance model fails to uphold the quality needed and recommended practices for sharing, documenting, and evaluating datasets. In this paper we investigate medical imaging datasets on CCPs and how they are documented, shared, and maintained. We first highlight some differences between medical imaging and computer vision, particularly in the potentially harmful downstream effects due to poor adoption of recommended dataset management practices. We then analyze 20 (10 medical and 10 computer vision) popular datasets on CCPs and find vague licenses, lack of persistent identifiers and storage, duplicates and missing metadata, with differences between the platforms. We present "actionability" as a conceptual metric to reveal the data quality gap between characteristics of data on CCPs and the desired characteristics of data for AI in healthcare. Finally, we propose a commons-based stewardship model for documenting, sharing and maintaining datasets on CCPs and end with a discussion of limitations and open questions.

Title: The SpongeNet Attack: Sponge Weight Poisoning of Deep Neural Networks

• Authors: Jona te Lintelo, Stefanos Koffas, Stjepan Picek
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06357
• Pdf URL: https://arxiv.org/pdf/2402.06357
• Copy Paste: [[2402.06357]] The SpongeNet Attack: Sponge Weight Poisoning of Deep Neural Networks(https://arxiv.org/abs/2402.06357)
• Keywords: defense, attack, steal
• Abstract: Sponge attacks aim to increase the energy consumption and computation time of neural networks deployed on hardware accelerators. Existing sponge attacks can be performed during inference via sponge examples or during training via Sponge Poisoning. Sponge examples leverage perturbations added to the model's input to increase energy and latency, while Sponge Poisoning alters the objective function of a model to induce inference-time energy/latency effects. In this work, we propose a novel sponge attack called SpongeNet. SpongeNet is the first sponge attack that is performed directly on the parameters of a pre-trained model. Our experiments show that SpongeNet can successfully increase the energy consumption of vision models with fewer samples required than Sponge Poisoning. Our experiments indicate that poisoning defenses are ineffective if not adjusted specifically for the defense against Sponge Poisoning (i.e., they decrease batch normalization bias values). Our work shows that SpongeNet is more effective on StarGAN than the state-of-the-art. Additionally, SpongeNet is stealthier than the previous Sponge Poisoning attack as it does not require significant changes in the victim model's weights. Our experiments indicate that the SpongeNet attack can be performed even when an attacker has access to only 1% of the entire dataset and reach up to 11% energy increase.

Title: StruQ: Defending Against Prompt Injection with Structured Queries

• Authors: Sizhe Chen, Julien Piet, Chawin Sitawarin, David Wagner
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06363
• Pdf URL: https://arxiv.org/pdf/2402.06363
• Copy Paste: [[2402.06363]] StruQ: Defending Against Prompt Injection with Structured Queries(https://arxiv.org/abs/2402.06363)
• Keywords: secure, defense, attack, large language model
• Abstract: Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capabilities. However, as LLMs have improved, so have the attacks against them. Prompt injection attacks are an important threat: they trick the model to deviate from the original application's instructions and instead follow user directives. These attacks rely on the LLM's ability to follow instructions and inability to separate the prompts and user data. We introduce structured queries, a general approach to tackle this problem. Structured queries separate prompts and data into two channels. We implement a system that supports structured queries. This system is made of (1) a secure front-end that formats a prompt and user data into a special format, and (2) a specially trained LLM that can produce high-quality outputs from these inputs. The LLM is trained using a novel fine-tuning strategy: we convert a base (non-instruction-tuned) LLM to a structured instruction-tuned model that will only follow instructions in the prompt portion of a query. To do so, we augment standard instruction tuning datasets with examples that also include instructions in the data portion of the query, and fine-tune the model to ignore these. Our system significantly improves resistance to prompt injection attacks, with little or no impact on utility. Our code is released at https://github.com/Sizhe-Chen/PromptInjectionDefense.

Title: TEE4EHR: Transformer Event Encoder for Better Representation Learning in Electronic Health Records

• Authors: Hojjat Karami, David Atienza, Anisoara Ionescu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06367
• Pdf URL: https://arxiv.org/pdf/2402.06367
• Copy Paste: [[2402.06367]] TEE4EHR: Transformer Event Encoder for Better Representation Learning in Electronic Health Records(https://arxiv.org/abs/2402.06367)
• Keywords: transformer
• Abstract: Irregular sampling of time series in electronic health records (EHRs) is one of the main challenges for developing machine learning models. Additionally, the pattern of missing data in certain clinical variables is not at random but depends on the decisions of clinicians and the state of the patient. Point process is a mathematical framework for analyzing event sequence data that is consistent with irregular sampling patterns. Our model, TEE4EHR, is a transformer event encoder (TEE) with point process loss that encodes the pattern of laboratory tests in EHRs. The utility of our TEE has been investigated in a variety of benchmark event sequence datasets. Additionally, we conduct experiments on two real-world EHR databases to provide a more comprehensive evaluation of our model. Firstly, in a self-supervised learning approach, the TEE is jointly learned with an existing attention-based deep neural network which gives superior performance in negative log-likelihood and future event prediction. Besides, we propose an algorithm for aggregating attention weights that can reveal the interaction between the events. Secondly, we transfer and freeze the learned TEE to the downstream task for the outcome prediction, where it outperforms state-of-the-art models for handling irregularly sampled time series. Furthermore, our results demonstrate that our approach can improve representation learning in EHRs and can be useful for clinical prediction tasks.

Title: FD-Vision Mamba for Endoscopic Exposure Correction

• Authors: Zhuoran Zheng, Jun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06378
• Pdf URL: https://arxiv.org/pdf/2402.06378
• Copy Paste: [[2402.06378]] FD-Vision Mamba for Endoscopic Exposure Correction(https://arxiv.org/abs/2402.06378)
• Keywords: extraction
• Abstract: In endoscopic imaging, the recorded images are prone to exposure abnormalities, so maintaining high-quality images is important to assist healthcare professionals in performing decision-making. To overcome this issue, We design a frequency-domain based network, called FD-Vision Mamba (FDVM-Net), which achieves high-quality image exposure correction by reconstructing the frequency domain of endoscopic images. Specifically, inspired by the State Space Sequence Models (SSMs), we develop a C-SSM block that integrates the local feature extraction ability of the convolutional layer with the ability of the SSM to capture long-range dependencies. A two-path network is built using C-SSM as the basic function cell, and these two paths deal with the phase and amplitude information of the image, respectively. Finally, a degraded endoscopic image is reconstructed by FDVM-Net to obtain a high-quality clear image. Extensive experimental results demonstrate that our method achieves state-of-the-art results in terms of speed and accuracy, and it is noteworthy that our method can enhance endoscopic images of arbitrary resolution. The URL of the code is \url{https://github.com/zzr-idam/FDVM-Net}.

Title: Learning using privileged information for segmenting tumors on digital mammograms

• Authors: Ioannis N. Tzortzis, Konstantinos Makantasis, Ioannis Rallis, Nikolaos Bakalos, Anastasios Doulamis, Nikolaos Doulamis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06379
• Pdf URL: https://arxiv.org/pdf/2402.06379
• Copy Paste: [[2402.06379]] Learning using privileged information for segmenting tumors on digital mammograms(https://arxiv.org/abs/2402.06379)
• Keywords: robust, segmentation
• Abstract: Limited amount of data and data sharing restrictions, due to GDPR compliance, constitute two common factors leading to reduced availability and accessibility when referring to medical data. To tackle these issues, we introduce the technique of Learning Using Privileged Information. Aiming to substantiate the idea, we attempt to build a robust model that improves the segmentation quality of tumors on digital mammograms, by gaining privileged information knowledge during the training procedure. Towards this direction, a baseline model, called student, is trained on patches extracted from the original mammograms, while an auxiliary model with the same architecture, called teacher, is trained on the corresponding enhanced patches accessing, in this way, privileged information. We repeat the student training procedure by providing the assistance of the teacher model this time. According to the experimental results, it seems that the proposed methodology performs better in the most of the cases and it can achieve 10% higher F1 score in comparison with the baseline.

Title: ImplicitDeepfake: Plausible Face-Swapping through Implicit Deepfake Generation using NeRF and Gaussian Splatting

• Authors: Georgii Stanishevskii, Jakub Steczkiewicz, Tomasz Szczepanik, Sławomir Tadeja, Jacek Tabor, Przemysław Spurek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06390
• Pdf URL: https://arxiv.org/pdf/2402.06390
• Copy Paste: [[2402.06390]] ImplicitDeepfake: Plausible Face-Swapping through Implicit Deepfake Generation using NeRF and Gaussian Splatting(https://arxiv.org/abs/2402.06390)
• Keywords: generative
• Abstract: Numerous emerging deep-learning techniques have had a substantial impact on computer graphics. Among the most promising breakthroughs are the recent rise of Neural Radiance Fields (NeRFs) and Gaussian Splatting (GS). NeRFs encode the object's shape and color in neural network weights using a handful of images with known camera positions to generate novel views. In contrast, GS provides accelerated training and inference without a decrease in rendering quality by encoding the object's characteristics in a collection of Gaussian distributions. These two techniques have found many use cases in spatial computing and other domains. On the other hand, the emergence of deepfake methods has sparked considerable controversy. Such techniques can have a form of artificial intelligence-generated videos that closely mimic authentic footage. Using generative models, they can modify facial features, enabling the creation of altered identities or facial expressions that exhibit a remarkably realistic appearance to a real person. Despite these controversies, deepfake can offer a next-generation solution for avatar creation and gaming when of desirable quality. To that end, we show how to combine all these emerging technologies to obtain a more plausible outcome. Our ImplicitDeepfake1 uses the classical deepfake algorithm to modify all training images separately and then train NeRF and GS on modified faces. Such relatively simple strategies can produce plausible 3D deepfake-based avatars.

Title: Hierarchical Transformers are Efficient Meta-Reinforcement Learners

• Authors: Gresa Shala, André Biedenkapp, Josif Grabocka
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06402
• Pdf URL: https://arxiv.org/pdf/2402.06402
• Copy Paste: [[2402.06402]] Hierarchical Transformers are Efficient Meta-Reinforcement Learners(https://arxiv.org/abs/2402.06402)
• Keywords: robust, transformer
• Abstract: We introduce Hierarchical Transformers for Meta-Reinforcement Learning (HTrMRL), a powerful online meta-reinforcement learning approach. HTrMRL aims to address the challenge of enabling reinforcement learning agents to perform effectively in previously unseen tasks. We demonstrate how past episodes serve as a rich source of information, which our model effectively distills and applies to new contexts. Our learned algorithm is capable of outperforming the previous state-of-the-art and provides more efficient meta-training while significantly improving generalization capabilities. Experimental results, obtained across various simulated tasks of the Meta-World Benchmark, indicate a significant improvement in learning efficiency and adaptability compared to the state-of-the-art on a variety of tasks. Our approach not only enhances the agent's ability to generalize from limited data but also paves the way for more robust and versatile AI systems.

Title: Trust the Process: Zero-Knowledge Machine Learning to Enhance Trust in Generative AI Interactions

• Authors: Bianca-Mihaela Ganescu, Jonathan Passerat-Palmbach
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06414
• Pdf URL: https://arxiv.org/pdf/2402.06414
• Copy Paste: [[2402.06414]] Trust the Process: Zero-Knowledge Machine Learning to Enhance Trust in Generative AI Interactions(https://arxiv.org/abs/2402.06414)
• Keywords: privacy, protect, fair, transformer, generative
• Abstract: Generative AI, exemplified by models like transformers, has opened up new possibilities in various domains but also raised concerns about fairness, transparency and reliability, especially in fields like medicine and law. This paper emphasizes the urgency of ensuring fairness and quality in these domains through generative AI. It explores using cryptographic techniques, particularly Zero-Knowledge Proofs (ZKPs), to address concerns regarding performance fairness and accuracy while protecting model privacy. Applying ZKPs to Machine Learning models, known as ZKML (Zero-Knowledge Machine Learning), enables independent validation of AI-generated content without revealing sensitive model information, promoting transparency and trust. ZKML enhances AI fairness by providing cryptographic audit trails for model predictions and ensuring uniform performance across users. We introduce snarkGPT, a practical ZKML implementation for transformers, to empower users to verify output accuracy and quality while preserving model privacy. We present a series of empirical results studying snarkGPT's scalability and performance to assess the feasibility and challenges of adopting a ZKML-powered approach to capture quality and performance fairness problems in generative AI models.

Title: CurveFormer++: 3D Lane Detection by Curve Propagation with Temporal Curve Queries and Attention

• Authors: Yifeng Bai, Zhirong Chen, Pengpeng Liang, Erkang Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06423
• Pdf URL: https://arxiv.org/pdf/2402.06423
• Copy Paste: [[2402.06423]] CurveFormer++: 3D Lane Detection by Curve Propagation with Temporal Curve Queries and Attention(https://arxiv.org/abs/2402.06423)
• Keywords: transformer
• Abstract: In autonomous driving, 3D lane detection using monocular cameras is an important task for various downstream planning and control tasks. Recent CNN and Transformer approaches usually apply a two-stage scheme in the model design. The first stage transforms the image feature from a front image into a bird's-eye-view (BEV) representation. Subsequently, a sub-network processes the BEV feature map to generate the 3D detection results. However, these approaches heavily rely on a challenging image feature transformation module from a perspective view to a BEV representation. In our work, we present CurveFormer++, a single-stage Transformer-based method that does not require the image feature view transform module and directly infers 3D lane detection results from the perspective image features. Specifically, our approach models the 3D detection task as a curve propagation problem, where each lane is represented by a curve query with a dynamic and ordered anchor point set. By employing a Transformer decoder, the model can iteratively refine the 3D lane detection results. A curve cross-attention module is introduced in the Transformer decoder to calculate similarities between image features and curve queries of lanes. To handle varying lane lengths, we employ context sampling and anchor point restriction techniques to compute more relevant image features for a curve query. Furthermore, we apply a temporal fusion module that incorporates selected informative sparse curve queries and their corresponding anchor point sets to leverage historical lane information. In the experiments, we evaluate our approach for the 3D lane detection task on two publicly available real-world datasets. The results demonstrate that our method provides outstanding performance compared with both CNN and Transformer based methods. We also conduct ablation studies to analyze the impact of each component in our approach.

Title: Where is the Truth? The Risk of Getting Confounded in a Continual World

• Authors: Florian Peter Busch, Roshni Kamath, Rupert Mitchell, Wolfgang Stammer, Kristian Kersting, Martin Mundt
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.06434
• Pdf URL: https://arxiv.org/pdf/2402.06434
• Copy Paste: [[2402.06434]] Where is the Truth? The Risk of Getting Confounded in a Continual World(https://arxiv.org/abs/2402.06434)
• Keywords: robust
• Abstract: A dataset is confounded if it is most easily solved via a spurious correlation which fails to generalize to new data. We will show that, in a continual learning setting where confounders may vary in time across tasks, the resulting challenge far exceeds the standard forgetting problem normally considered. In particular, we derive mathematically the effect of such confounders on the space of valid joint solutions to sets of confounded tasks. Interestingly, our theory predicts that for many such continual datasets, spurious correlations are easily ignored when the tasks are trained on jointly, but it is far harder to avoid confounding when they are considered sequentially. We construct such a dataset and demonstrate empirically that standard continual learning methods fail to ignore confounders, while training jointly on all tasks is successful. Our continually confounded dataset, ConCon, is based on CLEVR images and demonstrates the need for continual learning methods with more robust behavior with respect to confounding.

Title: Improving 2D-3D Dense Correspondences with Diffusion Models for 6D Object Pose Estimation

• Authors: Peter Hönig, Stefan Thalhammer, Markus Vincze
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06436
• Pdf URL: https://arxiv.org/pdf/2402.06436
• Copy Paste: [[2402.06436]] Improving 2D-3D Dense Correspondences with Diffusion Models for 6D Object Pose Estimation(https://arxiv.org/abs/2402.06436)
• Keywords: diffusion
• Abstract: Estimating 2D-3D correspondences between RGB images and 3D space is a fundamental problem in 6D object pose estimation. Recent pose estimators use dense correspondence maps and Point-to-Point algorithms to estimate object poses. The accuracy of pose estimation depends heavily on the quality of the dense correspondence maps and their ability to withstand occlusion, clutter, and challenging material properties. Currently, dense correspondence maps are estimated using image-to-image translation models based on GANs, Autoencoders, or direct regression models. However, recent advancements in image-to-image translation have led to diffusion models being the superior choice when evaluated on benchmarking datasets. In this study, we compare image-to-image translation networks based on GANs and diffusion models for the downstream task of 6D object pose estimation. Our results demonstrate that the diffusion-based image-to-image translation model outperforms the GAN, revealing potential for further improvements in 6D object pose estimation models.

Title: A Method for Decrypting Data Infected with Rhysida Ransomware

• Authors: Giyoon Kim, Soojin Kang, Seungjun Baek, Kimoon Kim, Jongsung Kim
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06440
• Pdf URL: https://arxiv.org/pdf/2402.06440
• Copy Paste: [[2402.06440]] A Method for Decrypting Data Infected with Rhysida Ransomware(https://arxiv.org/abs/2402.06440)
• Keywords: secure, security, attack
• Abstract: Ransomware is malicious software that is a prominent global cybersecurity threat. Typically, ransomware encrypts data on a system, rendering the victim unable to decrypt it without the attacker's private key. Subsequently, victims often pay a substantial ransom to recover their data, yet some may still incur damage or loss. This study examines Rhysida ransomware, which caused significant damage in the second half of 2023, and proposes a decryption method. Rhysida ransomware employed a secure random number generator to generate the encryption key and subsequently encrypt the data. However, an implementation vulnerability existed that enabled us to regenerate the internal state of the random number generator at the time of infection. We successfully decrypted the data using the regenerated random number generator. To the best of our knowledge, this is the first successful decryption of Rhysida ransomware. We aspire for our work to contribute to mitigating the damage inflicted by the Rhysida ransomware.

Title: ControlUDA: Controllable Diffusion-assisted Unsupervised Domain Adaptation for Cross-Weather Semantic Segmentation

• Authors: Fengyi Shen, Li Zhou, Kagan Kucukaytekin, Ziyuan Liu, He Wang, Alois Knoll
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06446
• Pdf URL: https://arxiv.org/pdf/2402.06446
• Copy Paste: [[2402.06446]] ControlUDA: Controllable Diffusion-assisted Unsupervised Domain Adaptation for Cross-Weather Semantic Segmentation(https://arxiv.org/abs/2402.06446)
• Keywords: diffusion, segmentation
• Abstract: Data generation is recognized as a potent strategy for unsupervised domain adaptation (UDA) pertaining semantic segmentation in adverse weathers. Nevertheless, these adverse weather scenarios encompass multiple possibilities, and high-fidelity data synthesis with controllable weather is under-researched in previous UDA works. The recent strides in large-scale text-to-image diffusion models (DM) have ushered in a novel avenue for research, enabling the generation of realistic images conditioned on semantic labels. This capability proves instrumental for cross-domain data synthesis from source to target domain owing to their shared label space. Thus, source domain labels can be paired with those generated pseudo target data for training UDA. However, from the UDA perspective, there exists several challenges for DM training: (i) ground-truth labels from target domain are missing; (ii) the prompt generator may produce vague or noisy descriptions of images from adverse weathers; (iii) existing arts often struggle to well handle the complex scene structure and geometry of urban scenes when conditioned only on semantic labels. To tackle the above issues, we propose ControlUDA, a diffusion-assisted framework tailored for UDA segmentation under adverse weather conditions. It first leverages target prior from a pre-trained segmentor for tuning the DM, compensating the missing target domain labels; It also contains UDAControlNet, a condition-fused multi-scale and prompt-enhanced network targeted at high-fidelity data generation in adverse weathers. Training UDA with our generated data brings the model performances to a new milestone (72.0 mIoU) on the popular Cityscapes-to-ACDC benchmark for adverse weathers. Furthermore, ControlUDA helps to achieve good model generalizability on unseen data.

Title: V-STaR: Training Verifiers for Self-Taught Reasoners

• Authors: Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, Rishabh Agarwal
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06457
• Pdf URL: https://arxiv.org/pdf/2402.06457
• Copy Paste: [[2402.06457]] V-STaR: Training Verifiers for Self-Taught Reasoners(https://arxiv.org/abs/2402.06457)
• Keywords: large language model
• Abstract: Common self-improvement approaches for large language models (LLMs), such as STaR (Zelikman et al., 2022), iteratively fine-tune LLMs on self-generated solutions to improve their problem-solving ability. However, these approaches discard the large amounts of incorrect solutions generated during this process, potentially neglecting valuable information in such solutions. To address this shortcoming, we propose V-STaR that utilizes both the correct and incorrect solutions generated during the self-improvement process to train a verifier using DPO that judges correctness of model-generated solutions. This verifier is used at inference time to select one solution among many candidate solutions. Running V-STaR for multiple iterations results in progressively better reasoners and verifiers, delivering a 4% to 17% test accuracy improvement over existing self-improvement and verification approaches on common code generation and math reasoning benchmarks with LLaMA2 models.

Title: Sequential Flow Matching for Generative Modeling

• Authors: Jongmin Yoon, Juho Lee
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.06461
• Pdf URL: https://arxiv.org/pdf/2402.06461
• Copy Paste: [[2402.06461]] Sequential Flow Matching for Generative Modeling(https://arxiv.org/abs/2402.06461)
• Keywords: diffusion, generative
• Abstract: Straightening the probability flow of the continuous-time generative models, such as diffusion models or flow-based models, is the key to fast sampling through the numerical solvers, existing methods learn a linear path by directly generating the probability path the joint distribution between the noise and data distribution. One key reason for the slow sampling speed of the ODE-based solvers that simulate these generative models is the global truncation error of the ODE solver, caused by the high curvature of the ODE trajectory, which explodes the truncation error of the numerical solvers in the low-NFE regime. To address this challenge, We propose a novel method called SeqRF, a learning technique that straightens the probability flow to reduce the global truncation error and hence enable acceleration of sampling and improve the synthesis quality. In both theoretical and empirical studies, we first observe the straightening property of our SeqRF. Through empirical evaluations via SeqRF over flow-based generative models, We achieve surpassing results on CIFAR-10, CelebA-$64 \times 64$, and LSUN-Church datasets.

Title: On Differentially Private Subspace Estimation Without Distributional Assumptions

• Authors: Eliad Tsfadia
• Subjects: cs.LG, cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2402.06465
• Pdf URL: https://arxiv.org/pdf/2402.06465
• Copy Paste: [[2402.06465]] On Differentially Private Subspace Estimation Without Distributional Assumptions(https://arxiv.org/abs/2402.06465)
• Keywords: privacy
• Abstract: Private data analysis faces a significant challenge known as the curse of dimensionality, leading to increased costs. However, many datasets possess an inherent low-dimensional structure. For instance, during optimization via gradient descent, the gradients frequently reside near a low-dimensional subspace. If the low-dimensional structure could be privately identified using a small amount of points, we could avoid paying (in terms of privacy and accuracy) for the high ambient dimension. On the negative side, Dwork, Talwar, Thakurta, and Zhang (STOC 2014) proved that privately estimating subspaces, in general, requires an amount of points that depends on the dimension. But Singhal and Steinke (NeurIPS 2021) bypassed this limitation by considering points that are i.i.d. samples from a Gaussian distribution whose covariance matrix has a certain eigenvalue gap. Yet, it was still left unclear whether we could provide similar upper bounds without distributional assumptions and whether we could prove lower bounds that depend on similar eigenvalue gaps. In this work, we make progress in both directions. We formulate the problem of private subspace estimation under two different types of singular value gaps of the input data and prove new upper and lower bounds for both types. In particular, our results determine what type of gap is sufficient and necessary for estimating a subspace with an amount of points that is independent of the dimension.

Title: Large Language Models for Captioning and Retrieving Remote Sensing Images

• Authors: João Daniel Silva, João Magalhães, Devis Tuia, Bruno Martins
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06475
• Pdf URL: https://arxiv.org/pdf/2402.06475
• Copy Paste: [[2402.06475]] Large Language Models for Captioning and Retrieving Remote Sensing Images(https://arxiv.org/abs/2402.06475)
• Keywords: large language model
• Abstract: Image captioning and cross-modal retrieval are examples of tasks that involve the joint analysis of visual and linguistic information. In connection to remote sensing imagery, these tasks can help non-expert users in extracting relevant Earth observation information for a variety of applications. Still, despite some previous efforts, the development and application of vision and language models to the remote sensing domain have been hindered by the relatively small size of the available datasets and models used in previous studies. In this work, we propose RS-CapRet, a Vision and Language method for remote sensing tasks, in particular image captioning and text-image retrieval. We specifically propose to use a highly capable large decoder language model together with image encoders adapted to remote sensing imagery through contrastive language-image pre-training. To bridge together the image encoder and language decoder, we propose training simple linear layers with examples from combining different remote sensing image captioning datasets, keeping the other parameters frozen. RS-CapRet can then generate descriptions for remote sensing images and retrieve images from textual descriptions, achieving SOTA or competitive performance with existing methods. Qualitative results illustrate that RS-CapRet can effectively leverage the pre-trained large language model to describe remote sensing images, retrieve them based on different types of queries, and also show the ability to process interleaved sequences of images and text in a dialogue manner.

Title: Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings

• Authors: Yichen Jiang, Xiang Zhou, Mohit Bansal
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06492
• Pdf URL: https://arxiv.org/pdf/2402.06492
• Copy Paste: [[2402.06492]] Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings(https://arxiv.org/abs/2402.06492)
• Keywords: transformer
• Abstract: Transformers generalize to novel compositions of structures and entities after being trained on a complex dataset, but easily overfit on datasets of insufficient complexity. We observe that when the training set is sufficiently complex, the model encodes sentences that have a common syntactic structure using a systematic attention pattern. Inspired by this observation, we propose SQ-Transformer (Structurally Quantized) that explicitly encourages systematicity in the embeddings and attention layers, even with a training set of low complexity. At the embedding level, we introduce Structure-oriented Vector Quantization (SoVQ) to cluster word embeddings into several classes of structurally equivalent entities. At the attention level, we devise the Systematic Attention Layer (SAL) and an alternative, Systematically Regularized Layer (SRL) that operate on the quantized word embeddings so that sentences of the same structure are encoded with invariant or similar attention patterns. Empirically, we show that SQ-Transformer achieves stronger compositional generalization than the vanilla Transformer on multiple low-complexity semantic parsing and machine translation datasets. In our analysis, we show that SoVQ indeed learns a syntactically clustered embedding space and SAL/SRL induces generalizable attention patterns, which lead to improved systematicity.

Title: Deep Learning-Based Auto-Segmentation of Planning Target Volume for Total Marrow and Lymph Node Irradiation

• Authors: Ricardo Coimbra Brioso, Damiano Dei, Nicola Lambri, Daniele Loiacono, Pietro Mancosu, Marta Scorsetti
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06494
• Pdf URL: https://arxiv.org/pdf/2402.06494
• Copy Paste: [[2402.06494]] Deep Learning-Based Auto-Segmentation of Planning Target Volume for Total Marrow and Lymph Node Irradiation(https://arxiv.org/abs/2402.06494)
• Keywords: robust, segmentation
• Abstract: In order to optimize the radiotherapy delivery for cancer treatment, especially when dealing with complex treatments such as Total Marrow and Lymph Node Irradiation (TMLI), the accurate contouring of the Planning Target Volume (PTV) is crucial. Unfortunately, relying on manual contouring for such treatments is time-consuming and prone to errors. In this paper, we investigate the application of Deep Learning (DL) to automate the segmentation of the PTV in TMLI treatment, building upon previous work that introduced a solution to this problem based on a 2D U-Net model. We extend the previous research (i) by employing the nnU-Net framework to develop both 2D and 3D U-Net models and (ii) by evaluating the trained models on the PTV with the exclusion of bones, which consist mainly of lymp-nodes and represent the most challenging region of the target volume to segment. Our result show that the introduction of nnU-NET framework led to statistically significant improvement in the segmentation performance. In addition, the analysis on the PTV after the exclusion of bones showed that the models are quite robust also on the most challenging areas of the target volume. Overall, our study is a significant step forward in the application of DL in a complex radiotherapy treatment such as TMLI, offering a viable and scalable solution to increase the number of patients who can benefit from this treatment.

Title: Iris-SAM: Iris Segmentation Using a Foundational Model

• Authors: Parisa Farmanifard, Arun Ross
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06497
• Pdf URL: https://arxiv.org/pdf/2402.06497
• Copy Paste: [[2402.06497]] Iris-SAM: Iris Segmentation Using a Foundational Model(https://arxiv.org/abs/2402.06497)
• Keywords: biometric, segmentation
• Abstract: Iris segmentation is a critical component of an iris biometric system and it involves extracting the annular iris region from an ocular image. In this work, we develop a pixel-level iris segmentation model from a foundational model, viz., Segment Anything Model (SAM), that has been successfully used for segmenting arbitrary objects. The primary contribution of this work lies in the integration of different loss functions during the fine-tuning of SAM on ocular images. In particular, the importance of Focal Loss is borne out in the fine-tuning process since it strategically addresses the class imbalance problem (i.e., iris versus non-iris pixels). Experiments on ND-IRIS-0405, CASIA-Iris-Interval-v3, and IIT-Delhi-Iris datasets convey the efficacy of the trained model for the task of iris segmentation. For instance, on the ND-IRIS-0405 dataset, an average segmentation accuracy of 99.58% was achieved, compared to the best baseline performance of 89.75%.

Title: Scalable Interactive Machine Learning for Future Command and Control

• Authors: Anna Madison, Ellen Novoseller, Vinicius G. Goecks, Benjamin T. Files, Nicholas Waytowich, Alfred Yu, Vernon J. Lawhern, Steven Thurman, Christopher Kelshaw, Kaleb McDowell
• Subjects: cs.LG, cs.AI, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.06501
• Pdf URL: https://arxiv.org/pdf/2402.06501
• Copy Paste: [[2402.06501]] Scalable Interactive Machine Learning for Future Command and Control(https://arxiv.org/abs/2402.06501)
• Keywords: robust
• Abstract: Future warfare will require Command and Control (C2) personnel to make decisions at shrinking timescales in complex and potentially ill-defined situations. Given the need for robust decision-making processes and decision-support tools, integration of artificial and human intelligence holds the potential to revolutionize the C2 operations process to ensure adaptability and efficiency in rapidly changing operational environments. We propose to leverage recent promising breakthroughs in interactive machine learning, in which humans can cooperate with machine learning algorithms to guide machine learning algorithm behavior. This paper identifies several gaps in state-of-the-art science and technology that future work should address to extend these approaches to function in complex C2 contexts. In particular, we describe three research focus areas that together, aim to enable scalable interactive machine learning (SIML): 1) developing human-AI interaction algorithms to enable planning in complex, dynamic situations; 2) fostering resilient human-AI teams through optimizing roles, configurations, and trust; and 3) scaling algorithms and human-AI teams for flexibility across a range of potential contexts and situations.

Title: Classifying point clouds at the facade-level using geometric features and deep learning networks

• Authors: Yue Tan, Olaf Wysocki, Ludwig Hoegner, Uwe Stilla
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06506
• Pdf URL: https://arxiv.org/pdf/2402.06506
• Copy Paste: [[2402.06506]] Classifying point clouds at the facade-level using geometric features and deep learning networks(https://arxiv.org/abs/2402.06506)
• Keywords: segmentation
• Abstract: 3D building models with facade details are playing an important role in many applications now. Classifying point clouds at facade-level is key to create such digital replicas of the real world. However, few studies have focused on such detailed classification with deep neural networks. We propose a method fusing geometric features with deep learning networks for point cloud classification at facade-level. Our experiments conclude that such early-fused features improve deep learning methods' performance. This method can be applied for compensating deep learning networks' ability in capturing local geometric information and promoting the advancement of semantic segmentation.

Title: Multimodal Clinical Trial Outcome Prediction with Large Language Models

• Authors: Wenhao Zheng, Dongsheng Peng, Hongxia Xu, Hongtu Zhu, Tianfan Fu, Huaxiu Yao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06512
• Pdf URL: https://arxiv.org/pdf/2402.06512
• Copy Paste: [[2402.06512]] Multimodal Clinical Trial Outcome Prediction with Large Language Models(https://arxiv.org/abs/2402.06512)
• Keywords: large language model
• Abstract: The clinical trial is a pivotal and costly process, often spanning multiple years and requiring substantial financial resources. Therefore, the development of clinical trial outcome prediction models aims to exclude drugs likely to fail and holds the potential for significant cost savings. Recent data-driven attempts leverage deep learning methods to integrate multimodal data for predicting clinical trial outcomes. However, these approaches rely on manually designed modal-specific encoders, which limits both the extensibility to adapt new modalities and the ability to discern similar information patterns across different modalities. To address these issues, we propose a multimodal mixture-of-experts (LIFTED) approach for clinical trial outcome prediction. Specifically, LIFTED unifies different modality data by transforming them into natural language descriptions. Then, LIFTED constructs unified noise-resilient encoders to extract information from modal-specific language descriptions. Subsequently, a sparse Mixture-of-Experts framework is employed to further refine the representations, enabling LIFTED to identify similar information patterns across different modalities and extract more consistent representations from those patterns using the same expert model. Finally, a mixture-of-experts module is further employed to dynamically integrate different modality representations for prediction, which gives LIFTED the ability to automatically weigh different modalities and pay more attention to critical information. The experiments demonstrate that LIFTED significantly enhances performance in predicting clinical trial outcomes across all three phases compared to the best baseline, showcasing the effectiveness of our proposed key components.

Title: The Decisive Power of Indecision: Low-Variance Risk-Limiting Audits and Election Contestation via Marginal Mark Recording

• Authors: Benjamin Fuller, Rashmi Pai, Alexander Russell
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06515
• Pdf URL: https://arxiv.org/pdf/2402.06515
• Copy Paste: [[2402.06515]] The Decisive Power of Indecision: Low-Variance Risk-Limiting Audits and Election Contestation via Marginal Mark Recording(https://arxiv.org/abs/2402.06515)
• Keywords: security
• Abstract: Risk-limiting audits (RLAs) are the established techniques for verifying large elections. While they provide rigorous guarantees of correctness, widespread adoption has been impeded by both efficiency concerns and the fact they offer statistical, rather than absolute, conclusions. We define new families of audits that help to address these issues. Our new audits are enabled by revisiting the standard notion of a cast-vote record so that it can declare multiple possible mark interpretations rather than a single decision; this can reflect the presence of ambiguous marks, which appear regularly on hand-marked ballots. We show that this simple expedient can offer significant efficiency improvements with only minor changes to existing auditing infrastructure. We establish that these "Bayesian" comparison audits are indeed risk-limiting in the formal sense of (Fuller, Harrison, and Russell, 2022). We then define a new type of post-election audit we call a contested audit. These call for each candidate to provide a cast-vote record table advancing their own claim to victory. We prove that these audits offer remarkable sample efficiency: they guarantee negligible risk with only a constant number of ballot inspections. This is a first for an audit with provable soundness. These results are formulated in a game-based security model that specify quantitative soundness and completeness guarantees. Finally, we observe that these audits provide a direct means to handle contestation of election results affirmed by conventional RLAs.

Title: HoneyDOC: An Efficient Honeypot Architecture Enabling All-Round Design

• Authors: Wenjun Fan, Zhihui Du, Max Smith-Creasey, David Fernández
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.06516
• Pdf URL: https://arxiv.org/pdf/2402.06516
• Copy Paste: [[2402.06516]] HoneyDOC: An Efficient Honeypot Architecture Enabling All-Round Design(https://arxiv.org/abs/2402.06516)
• Keywords: attack, steal
• Abstract: Honeypots are designed to trap the attacker with the purpose of investigating its malicious behavior. Owing to the increasing variety and sophistication of cyber attacks, how to capture high-quality attack data has become a challenge in the context of honeypot area. All-round honeypots, which mean significant improvement in sensibility, countermeasure and stealth, are necessary to tackle the problem. In this paper, we propose a novel honeypot architecture termed HoneyDOC to support all-round honeypot design and implementation. Our HoneyDOC architecture clearly identifies three essential independent and collaborative modules, Decoy, Captor and Orchestrator. Based on the efficient architecture, a Software-Defined Networking (SDN) enabled honeypot system is designed, which supplies high programmability for technically sustaining the features for capturing high-quality data. A proof-of-concept system is implemented to validate its feasibility and effectiveness. The experimental results show the benefits by using the proposed architecture comparing to the previous honeypot solutions.

Title: Generative Adversarial Bayesian Optimization for Surrogate Objectives

• Authors: Michael S. Yao, Yimeng Zeng, Hamsa Bastani, Jacob Gardner, James C. Gee, Osbert Bastani
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06532
• Pdf URL: https://arxiv.org/pdf/2402.06532
• Copy Paste: [[2402.06532]] Generative Adversarial Bayesian Optimization for Surrogate Objectives(https://arxiv.org/abs/2402.06532)
• Keywords: generative
• Abstract: Offline model-based policy optimization seeks to optimize a learned surrogate objective function without querying the true oracle objective during optimization. However, inaccurate surrogate model predictions are frequently encountered along the optimization trajectory. To address this limitation, we propose generative adversarial Bayesian optimization (GABO) using adaptive source critic regularization, a task-agnostic framework for Bayesian optimization that employs a Lipschitz-bounded source critic model to constrain the optimization trajectory to regions where the surrogate function is reliable. We show that under certain assumptions for the continuous input space prior, our algorithm dynamically adjusts the strength of the source critic regularization. GABO outperforms existing baselines on a number of different offline optimization tasks across a variety of scientific domains. Our code is available at https://github.com/michael-s-yao/gabo

Title: Hybridnet for depth estimation and semantic segmentation

• Authors: Dalila Sánchez-Escobedo, Xiao Lin, Josep R. Casas, Montse Pardàs
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.06539
• Pdf URL: https://arxiv.org/pdf/2402.06539
• Copy Paste: [[2402.06539]] Hybridnet for depth estimation and semantic segmentation(https://arxiv.org/abs/2402.06539)
• Keywords: extraction, segmentation
• Abstract: Semantic segmentation and depth estimation are two important tasks in the area of image processing. Traditionally, these two tasks are addressed in an independent manner. However, for those applications where geometric and semantic information is required, such as robotics or autonomous navigation,depth or semantic segmentation alone are not sufficient. In this paper, depth estimation and semantic segmentation are addressed together from a single input image through a hybrid convolutional network. Different from the state of the art methods where features are extracted by a sole feature extraction network for both tasks, the proposed HybridNet improves the features extraction by separating the relevant features for one task from those which are relevant for both. Experimental results demonstrate that HybridNet results are comparable with the state of the art methods, as well as the single task methods that HybridNet is based on.

Title: Calibrating Long-form Generations from Large Language Models

• Authors: Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, Bhuwan Dhingra
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06544
• Pdf URL: https://arxiv.org/pdf/2402.06544
• Copy Paste: [[2402.06544]] Calibrating Long-form Generations from Large Language Models(https://arxiv.org/abs/2402.06544)
• Keywords: large language model
• Abstract: To enhance Large Language Models' (LLMs) reliability, calibration is essential -- the model's assessed confidence scores should align with the actual likelihood of its responses being correct. However, current confidence elicitation methods and calibration metrics typically rely on a binary true/false assessment of response correctness. This approach does not apply to long-form generation, where an answer can be partially correct. Addressing this gap, we introduce a unified calibration framework, in which both the correctness of the LLMs' responses and their associated confidence levels are treated as distributions across a range of scores. Within this framework, we develop three metrics to precisely evaluate LLM calibration and further propose two confidence elicitation methods based on self-consistency and self-evaluation. Our experiments, which include long-form QA and summarization tasks, demonstrate that larger models don't necessarily guarantee better calibration, that calibration performance is found to be metric-dependent, and that self-consistency methods excel in factoid datasets. We also find that calibration can be enhanced through techniques such as fine-tuning, integrating relevant source documents, scaling the temperature, and combining self-consistency with self-evaluation. Lastly, we showcase a practical application of our system: selecting and cascading open-source models and ChatGPT to optimize correctness given a limited API budget. This research not only challenges existing notions of LLM calibration but also offers practical methodologies for improving trustworthiness in long-form generation.

Title: Bryndza at ClimateActivism 2024: Stance, Target and Hate Event Detection via Retrieval-Augmented GPT-4 and LLaMA

• Authors: Marek Šuppa, Daniel Skala, Daniela Jašš, Samuel Sučík, Andrej Švec, Peter Hraška
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06549
• Pdf URL: https://arxiv.org/pdf/2402.06549
• Copy Paste: [[2402.06549]] Bryndza at ClimateActivism 2024: Stance, Target and Hate Event Detection via Retrieval-Augmented GPT-4 and LLaMA(https://arxiv.org/abs/2402.06549)
• Keywords: large language model
• Abstract: This study details our approach for the CASE 2024 Shared Task on Climate Activism Stance and Hate Event Detection, focusing on Hate Speech Detection, Hate Speech Target Identification, and Stance Detection as classification challenges. We explored the capability of Large Language Models (LLMs), particularly GPT-4, in zero- or few-shot settings enhanced by retrieval augmentation and re-ranking for Tweet classification. Our goal was to determine if LLMs could match or surpass traditional methods in this context. We conducted an ablation study with LLaMA for comparison, and our results indicate that our models significantly outperformed the baselines, securing second place in the Target Detection task. The code for our submission is available at https://github.com/NaiveNeuron/bryndza-case-2024

Title: Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following

• Authors: Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung-Wei Ke, Ayush Jain, Jeff Schneider, Katerina Fragkiadaki
• Subjects: cs.LG, cs.AI, cs.CL, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.06559
• Pdf URL: https://arxiv.org/pdf/2402.06559
• Copy Paste: [[2402.06559]] Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following(https://arxiv.org/abs/2402.06559)
• Keywords: diffusion
• Abstract: Diffusion models excel at modeling complex and multimodal trajectory distributions for decision-making and control. Reward-gradient guided denoising has been recently proposed to generate trajectories that maximize both a differentiable reward function and the likelihood under the data distribution captured by a diffusion model. Reward-gradient guided denoising requires a differentiable reward function fitted to both clean and noised samples, limiting its applicability as a general trajectory optimizer. In this paper, we propose DiffusionES, a method that combines gradient-free optimization with trajectory denoising to optimize black-box non-differentiable objectives while staying in the data manifold. Diffusion-ES samples trajectories during evolutionary search from a diffusion model and scores them using a black-box reward function. It mutates high-scoring trajectories using a truncated diffusion process that applies a small number of noising and denoising steps, allowing for much more efficient exploration of the solution space. We show that DiffusionES achieves state-of-the-art performance on nuPlan, an established closed-loop planning benchmark for autonomous driving. Diffusion-ES outperforms existing sampling-based planners, reactive deterministic or diffusion-based policies, and reward-gradient guidance. Additionally, we show that unlike prior guidance methods, our method can optimize non-differentiable language-shaped reward functions generated by few-shot LLM prompting. When guided by a human teacher that issues instructions to follow, our method can generate novel, highly complex behaviors, such as aggressive lane weaving, which are not present in the training data. This allows us to solve the hardest nuPlan scenarios which are beyond the capabilities of existing trajectory optimization methods and driving policies.

Title: Video Annotator: A framework for efficiently building video classifiers using vision-language models and active learning

• Authors: Amir Ziai, Aneesh Vartakavi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06560
• Pdf URL: https://arxiv.org/pdf/2402.06560
• Copy Paste: [[2402.06560]] Video Annotator: A framework for efficiently building video classifiers using vision-language models and active learning(https://arxiv.org/abs/2402.06560)
• Keywords: robust
• Abstract: High-quality and consistent annotations are fundamental to the successful development of robust machine learning models. Traditional data annotation methods are resource-intensive and inefficient, often leading to a reliance on third-party annotators who are not the domain experts. Hard samples, which are usually the most informative for model training, tend to be difficult to label accurately and consistently without business context. These can arise unpredictably during the annotation process, requiring a variable number of iterations and rounds of feedback, leading to unforeseen expenses and time commitments to guarantee quality. We posit that more direct involvement of domain experts, using a human-in-the-loop system, can resolve many of these practical challenges. We propose a novel framework we call Video Annotator (VA) for annotating, managing, and iterating on video classification datasets. Our approach offers a new paradigm for an end-user-centered model development process, enhancing the efficiency, usability, and effectiveness of video classifiers. Uniquely, VA allows for a continuous annotation process, seamlessly integrating data collection and model training. We leverage the zero-shot capabilities of vision-language foundation models combined with active learning techniques, and demonstrate that VA enables the efficient creation of high-quality models. VA achieves a median 6.8 point improvement in Average Precision relative to the most competitive baseline across a wide-ranging assortment of tasks. We release a dataset with 153k labels across 56 video understanding tasks annotated by three professional video editors using VA, and also release code to replicate our experiments at: this http URL

Title: Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Control

• Authors: Zheng Xiong, Risto Vuorio, Jacob Beck, Matthieu Zimmer, Kun Shao, Shimon Whiteson
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.06570
• Pdf URL: https://arxiv.org/pdf/2402.06570
• Copy Paste: [[2402.06570]] Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Control(https://arxiv.org/abs/2402.06570)
• Keywords: transformer
• Abstract: Learning a universal policy across different robot morphologies can significantly improve learning efficiency and enable zero-shot generalization to unseen morphologies. However, learning a highly performant universal policy requires sophisticated architectures like transformers (TF) that have larger memory and computational cost than simpler multi-layer perceptrons (MLP). To achieve both good performance like TF and high efficiency like MLP at inference time, we propose HyperDistill, which consists of: (1) A morphology-conditioned hypernetwork (HN) that generates robot-wise MLP policies, and (2) A policy distillation approach that is essential for successful training. We show that on UNIMAL, a benchmark with hundreds of diverse morphologies, HyperDistill performs as well as a universal TF teacher policy on both training and unseen test robots, but reduces model size by 6-14 times, and computational cost by 67-160 times in different environments. Our analysis attributes the efficiency advantage of HyperDistill at inference time to knowledge decoupling, i.e., the ability to decouple inter-task and intra-task knowledge, a general principle that could also be applied to improve inference efficiency in other domains.

Title: More than the Sum of Its Parts: Ensembling Backbone Networks for Few-Shot Segmentation

• Authors: Nico Catalano, Alessandro Maranelli, Agnese Chiatti, Matteo Matteucci
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.06581
• Pdf URL: https://arxiv.org/pdf/2402.06581
• Copy Paste: [[2402.06581]] More than the Sum of Its Parts: Ensembling Backbone Networks for Few-Shot Segmentation(https://arxiv.org/abs/2402.06581)
• Keywords: robust, extraction, segmentation
• Abstract: Semantic segmentation is a key prerequisite to robust image understanding for applications in \acrlong{ai} and Robotics. \acrlong{fss}, in particular, concerns the extension and optimization of traditional segmentation methods in challenging conditions where limited training examples are available. A predominant approach in \acrlong{fss} is to rely on a single backbone for visual feature extraction. Choosing which backbone to leverage is a deciding factor contributing to the overall performance. In this work, we interrogate on whether fusing features from different backbones can improve the ability of \acrlong{fss} models to capture richer visual features. To tackle this question, we propose and compare two ensembling techniques-Independent Voting and Feature Fusion. Among the available \acrlong{fss} methods, we implement the proposed ensembling techniques on PANet. The module dedicated to predicting segmentation masks from the backbone embeddings in PANet avoids trainable parameters, creating a controlled `in vitro' setting for isolating the impact of different ensembling strategies. Leveraging the complementary strengths of different backbones, our approach outperforms the original single-backbone PANet across standard benchmarks even in challenging one-shot learning scenarios. Specifically, it achieved a performance improvement of +7.37\% on PASCAL-5\textsuperscript{i} and of +10.68\% on COCO-20\textsuperscript{i} in the top-performing scenario where three backbones are combined. These results, together with the qualitative inspection of the predicted subject masks, suggest that relying on multiple backbones in PANet leads to a more comprehensive feature representation, thus expediting the successful application of \acrlong{fss} methods in challenging, data-scarce environments.

Title: G-SciEdBERT: A Contextualized LLM for Science Assessment Tasks in German

• Authors: Ehsan Latif, Gyeong-Geon Lee, Knut Neuman, Tamara Kastorff, Xiaoming Zhai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06584
• Pdf URL: https://arxiv.org/pdf/2402.06584
• Copy Paste: [[2402.06584]] G-SciEdBERT: A Contextualized LLM for Science Assessment Tasks in German(https://arxiv.org/abs/2402.06584)
• Keywords: large language model
• Abstract: The advancement of natural language processing has paved the way for automated scoring systems in various languages, such as German (e.g., German BERT [G-BERT]). Automatically scoring written responses to science questions in German is a complex task and challenging for standard G-BERT as they lack contextual knowledge in the science domain and may be unaligned with student writing styles. This paper developed a contextualized German Science Education BERT (G-SciEdBERT), an innovative large language model tailored for scoring German-written responses to science tasks. Using G-BERT, we pre-trained G-SciEdBERT on a corpus of 50K German written science responses with 5M tokens to the Programme for International Student Assessment (PISA) 2015. We fine-tuned G-SciEdBERT on 59 assessment items and examined the scoring accuracy. We then compared its performance with G-BERT. Our findings reveal a substantial improvement in scoring accuracy with G-SciEdBERT, demonstrating a 10% increase of quadratic weighted kappa compared to G-BERT (mean accuracy difference = 0.096, SD = 0.024). These insights underline the significance of specialized language models like G-SciEdBERT, which is trained to enhance the accuracy of automated scoring, offering a substantial contribution to the field of AI in education.

Title: On the Out-Of-Distribution Generalization of Multimodal Large Language Models

• Authors: Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, Peng Cui
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06599
• Pdf URL: https://arxiv.org/pdf/2402.06599
• Copy Paste: [[2402.06599]] On the Out-Of-Distribution Generalization of Multimodal Large Language Models(https://arxiv.org/abs/2402.06599)
• Keywords: robust, extraction, large language model
• Abstract: We investigate the generalization boundaries of current Multimodal Large Language Models (MLLMs) via comprehensive evaluation under out-of-distribution scenarios and domain-specific tasks. We evaluate their zero-shot generalization across synthetic images, real-world distributional shifts, and specialized datasets like medical and molecular imagery. Empirical results indicate that MLLMs struggle with generalization beyond common training domains, limiting their direct application without adaptation. To understand the cause of unreliable performance, we analyze three hypotheses: semantic misinterpretation, visual feature extraction insufficiency, and mapping deficiency. Results identify mapping deficiency as the primary hurdle. To address this problem, we show that in-context learning (ICL) can significantly enhance MLLMs' generalization, opening new avenues for overcoming generalization barriers. We further explore the robustness of ICL under distribution shifts and show its vulnerability to domain shifts, label shifts, and spurious correlation shifts between in-context examples and test data.

Title: RQP-SGD: Differential Private Machine Learning through Noisy SGD and Randomized Quantization

• Authors: Ce Feng, Parv Venkitasubramaniam
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.06606
• Pdf URL: https://arxiv.org/pdf/2402.06606
• Copy Paste: [[2402.06606]] RQP-SGD: Differential Private Machine Learning through Noisy SGD and Randomized Quantization(https://arxiv.org/abs/2402.06606)
• Keywords: secure, privacy
• Abstract: The rise of IoT devices has prompted the demand for deploying machine learning at-the-edge with real-time, efficient, and secure data processing. In this context, implementing machine learning (ML) models with real-valued weight parameters can prove to be impractical particularly for large models, and there is a need to train models with quantized discrete weights. At the same time, these low-dimensional models also need to preserve privacy of the underlying dataset. In this work, we present RQP-SGD, a new approach for privacy-preserving quantization to train machine learning models for low-memory ML-at-the-edge. This approach combines differentially private stochastic gradient descent (DP-SGD) with randomized quantization, providing a measurable privacy guarantee in machine learning. In particular, we study the utility convergence of implementing RQP-SGD on ML tasks with convex objectives and quantization constraints and demonstrate its efficacy over deterministic quantization. Through experiments conducted on two datasets, we show the practical effectiveness of RQP-SGD.

Title: Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning

• Authors: Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.06619
• Pdf URL: https://arxiv.org/pdf/2402.06619
• Copy Paste: [[2402.06619]] Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning(https://arxiv.org/abs/2402.06619)
• Keywords: large language model
• Abstract: Datasets are foundational to many breakthroughs in modern artificial intelligence. Many recent achievements in the space of natural language processing (NLP) can be attributed to the finetuning of pre-trained models on a diverse set of tasks that enables a large language model (LLM) to respond to instructions. Instruction fine-tuning (IFT) requires specifically constructed and annotated datasets. However, existing datasets are almost all in the English language. In this work, our primary goal is to bridge the language gap by building a human-curated instruction-following dataset spanning 65 languages. We worked with fluent speakers of languages from around the world to collect natural instances of instructions and completions. Furthermore, we create the most extensive multilingual collection to date, comprising 513 million instances through templating and translating existing datasets across 114 languages. In total, we contribute four key resources: we develop and open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection, and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case study in participatory research, involving collaborators from 119 countries. We see this as a valuable framework for future research collaborations that aim to bridge gaps in resources.

Title: Understanding the Effects of Iterative Prompting on Truthfulness

• Authors: Satyapriya Krishna, Chirag Agarwal, Himabindu Lakkaraju
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.06625
• Pdf URL: https://arxiv.org/pdf/2402.06625
• Copy Paste: [[2402.06625]] Understanding the Effects of Iterative Prompting on Truthfulness(https://arxiv.org/abs/2402.06625)
• Keywords: large language model
• Abstract: The development of Large Language Models (LLMs) has notably transformed numerous sectors, offering impressive text generation capabilities. Yet, the reliability and truthfulness of these models remain pressing concerns. To this end, we investigate iterative prompting, a strategy hypothesized to refine LLM responses, assessing its impact on LLM truthfulness, an area which has not been thoroughly explored. Our extensive experiments delve into the intricacies of iterative prompting variants, examining their influence on the accuracy and calibration of model responses. Our findings reveal that naive prompting methods significantly undermine truthfulness, leading to exacerbated calibration errors. In response to these challenges, we introduce several prompting variants designed to address the identified issues. These variants demonstrate marked improvements over existing baselines, signaling a promising direction for future research. Our work provides a nuanced understanding of iterative prompting and introduces novel approaches to enhance the truthfulness of LLMs, thereby contributing to the development of more accurate and trustworthy AI systems.