2024-02-16

Title: Fourier Circuits in Neural Networks: Unlocking the Potential of Large Language Models in Mathematical Reasoning and Modular Arithmetic

• Authors: Jiuxiang Gu, Chenyang Li, Yingyu Liang, Zhenmei Shi, Zhao Song, Tianyi Zhou
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.09469
• Pdf URL: https://arxiv.org/pdf/2402.09469
• Copy Paste: [[2402.09469]] Fourier Circuits in Neural Networks: Unlocking the Potential of Large Language Models in Mathematical Reasoning and Modular Arithmetic(https://arxiv.org/abs/2402.09469)
• Keywords: transformer
• Abstract: In the evolving landscape of machine learning, a pivotal challenge lies in deciphering the internal representations harnessed by neural networks and Transformers. Building on recent progress toward comprehending how networks execute distinct target functions, our study embarks on an exploration of the underlying reasons behind networks adopting specific computational strategies. We direct our focus to the complex algebraic learning task of modular addition involving $k$ inputs. Our research presents a thorough analytical characterization of the features learned by stylized one-hidden layer neural networks and one-layer Transformers in addressing this task. A cornerstone of our theoretical framework is the elucidation of how the principle of margin maximization shapes the features adopted by one-hidden layer neural networks. Let $p$ denote the modulus, $D_p$ denote the dataset of modular arithmetic with $k$ inputs and $m$ denote the network width. We demonstrate that a neuron count of $ m \geq 2^{2k-2} \cdot (p-1) $, these networks attain a maximum $ L_{2,k+1} $-margin on the dataset $ D_p $. Furthermore, we establish that each hidden-layer neuron aligns with a specific Fourier spectrum, integral to solving modular addition problems. By correlating our findings with the empirical observations of similar studies, we contribute to a deeper comprehension of the intrinsic computational mechanisms of neural networks. Furthermore, we observe similar computational mechanisms in the attention matrix of the Transformer. This research stands as a significant stride in unraveling their operation complexities, particularly in the realm of complex algebraic tasks.

Title: Rolling Diffusion Models

• Authors: David Ruhe, Jonathan Heek, Tim Salimans, Emiel Hoogeboom
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.09470
• Pdf URL: https://arxiv.org/pdf/2402.09470
• Copy Paste: [[2402.09470]] Rolling Diffusion Models(https://arxiv.org/abs/2402.09470)
• Keywords: diffusion
• Abstract: Diffusion models have recently been increasingly applied to temporal data such as video, fluid mechanics simulations, or climate data. These methods generally treat subsequent frames equally regarding the amount of noise in the diffusion process. This paper explores Rolling Diffusion: a new approach that uses a sliding window denoising process. It ensures that the diffusion process progressively corrupts through time by assigning more noise to frames that appear later in a sequence, reflecting greater uncertainty about the future as the generation process unfolds. Empirically, we show that when the temporal dynamics are complex, Rolling Diffusion is superior to standard diffusion. In particular, this result is demonstrated in a video prediction task using the Kinetics-600 video dataset and in a chaotic fluid dynamics forecasting experiment.

Title: PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining

• Authors: Mishaal Kazmi, Hadrien Lautraite, Alireza Akbari, Mauricio Soroco, Qiaoyue Tang, Tao Wang, Sébastien Gambs, Mathias Lécuyer
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09477
• Pdf URL: https://arxiv.org/pdf/2402.09477
• Copy Paste: [[2402.09477]] PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining(https://arxiv.org/abs/2402.09477)
• Keywords: privacy, attack, membership infer
• Abstract: We introduce a privacy auditing scheme for ML models that relies on membership inference attacks using generated data as "non-members". This scheme, which we call PANORAMIA, quantifies the privacy leakage for large-scale ML models without control of the training process or model re-training and only requires access to a subset of the training data. To demonstrate its applicability, we evaluate our auditing scheme across multiple ML domains, ranging from image and tabular data classification to large-scale language models.

Title: Data Reconstruction Attacks and Defenses: A Systematic Evaluation

• Authors: Sheng Liu, Zihan Wang, Qi Lei
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09478
• Pdf URL: https://arxiv.org/pdf/2402.09478
• Copy Paste: [[2402.09478]] Data Reconstruction Attacks and Defenses: A Systematic Evaluation(https://arxiv.org/abs/2402.09478)
• Keywords: defense, attack, federate
• Abstract: Reconstruction attacks and defenses are essential in understanding the data leakage problem in machine learning. However, prior work has centered around empirical observations of gradient inversion attacks, lacks theoretical groundings, and was unable to disentangle the usefulness of defending methods versus the computational limitation of attacking methods. In this work, we propose a strong reconstruction attack in the setting of federated learning. The attack reconstructs intermediate features and nicely integrates with and outperforms most of the previous methods. On this stronger attack, we thoroughly investigate both theoretically and empirically the effect of the most common defense methods. Our findings suggest that among various defense mechanisms, such as gradient clipping, dropout, additive noise, local aggregation, etc., gradient pruning emerges as the most effective strategy to defend against state-of-the-art attacks.

Title: Cryptoanalysis of a key exchange protocol based on a congruence-simple semiring action

• Authors: Otero Sanchez Alvaro, Lopez Ramos Juan Antonio
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.09480
• Pdf URL: https://arxiv.org/pdf/2402.09480
• Copy Paste: [[2402.09480]] Cryptoanalysis of a key exchange protocol based on a congruence-simple semiring action(https://arxiv.org/abs/2402.09480)
• Keywords: secure, attack
• Abstract: We show that a previously introduced key exchange based on a congruence-simple semiring action is not secure by providing an attack that reveals the shared key from the distributed public information for any of such semirings

Title: Instruction Tuning for Secure Code Generation

• Authors: Jingxuan He, Mark Vero, Gabriela Krasnopolska, Martin Vechev
• Subjects: cs.CR, cs.AI, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.09497
• Pdf URL: https://arxiv.org/pdf/2402.09497
• Copy Paste: [[2402.09497]] Instruction Tuning for Secure Code Generation(https://arxiv.org/abs/2402.09497)
• Keywords: secure, security
• Abstract: Modern language models (LMs) have gained widespread acceptance in everyday and professional contexts, particularly in programming. An essential procedure enabling this adoption is instruction tuning, which substantially enhances LMs' practical utility by training them to follow user instructions and human preferences. However, existing instruction tuning schemes overlook a crucial aspect: the security of generated code. As a result, even the state-of-the-art instruction-tuned LMs frequently produce unsafe code, posing significant security risks. In this work, we introduce SafeCoder to address this gap. SafeCoder performs security-centric fine-tuning using a diverse and high-quality dataset that we collected using an automated pipeline. We integrate the security fine-tuning with standard instruction tuning, to facilitate a joint optimization of both security and utility. Despite its simplicity, we show that SafeCoder is effective across a variety of popular LMs and datasets. It is able to drastically improve security (by about 30%), while preserving utility.

Title: The Manifold Density Function: An Intrinsic Method for the Validation of Manifold Learning

• Authors: Benjamin Holmgren, Eli Quist, Jordan Schupbach, Brittany Terese Fasy, Bastian Rieck
• Subjects: cs.LG, math.AT
• Abstract URL: https://arxiv.org/abs/2402.09529
• Pdf URL: https://arxiv.org/pdf/2402.09529
• Copy Paste: [[2402.09529]] The Manifold Density Function: An Intrinsic Method for the Validation of Manifold Learning(https://arxiv.org/abs/2402.09529)
• Keywords: robust
• Abstract: We introduce the manifold density function, which is an intrinsic method to validate manifold learning techniques. Our approach adapts and extends Ripley's $K$-function, and categorizes in an unsupervised setting the extent to which an output of a manifold learning algorithm captures the structure of a latent manifold. Our manifold density function generalizes to broad classes of Riemannian manifolds. In particular, we extend the manifold density function to general two-manifolds using the Gauss-Bonnet theorem, and demonstrate that the manifold density function for hypersurfaces is well approximated using the first Laplacian eigenvalue. We prove desirable convergence and robustness properties.

Title: Reducing Texture Bias of Deep Neural Networks via Edge Enhancing Diffusion

• Authors: Edgar Heinert, Matthias Rottmann, Kira Maag, Karsten Kahl
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09530
• Pdf URL: https://arxiv.org/pdf/2402.09530
• Copy Paste: [[2402.09530]] Reducing Texture Bias of Deep Neural Networks via Edge Enhancing Diffusion(https://arxiv.org/abs/2402.09530)
• Keywords: robust, diffusion, transformer, segmentation
• Abstract: Convolutional neural networks (CNNs) for image processing tend to focus on localized texture patterns, commonly referred to as texture bias. While most of the previous works in the literature focus on the task of image classification, we go beyond this and study the texture bias of CNNs in semantic segmentation. In this work, we propose to train CNNs on pre-processed images with less texture to reduce the texture bias. Therein, the challenge is to suppress image texture while preserving shape information. To this end, we utilize edge enhancing diffusion (EED), an anisotropic image diffusion method initially introduced for image compression, to create texture reduced duplicates of existing datasets. Extensive numerical studies are performed with both CNNs and vision transformer models trained on original data and EED-processed data from the Cityscapes dataset and the CARLA driving simulator. We observe strong texture-dependence of CNNs and moderate texture-dependence of transformers. Training CNNs on EED-processed images enables the models to become completely ignorant with respect to texture, demonstrating resilience with respect to texture re-introduction to any degree. Additionally we analyze the performance reduction in depth on a level of connected components in the semantic segmentation and study the influence of EED pre-processing on domain generalization as well as adversarial robustness.

Title: Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks?

• Authors: Andrew Lowy, Zhuohang Li, Jing Liu, Toshiaki Koike-Akino, Kieran Parsons, Ye Wang
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09540
• Pdf URL: https://arxiv.org/pdf/2402.09540
• Copy Paste: [[2402.09540]] Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks?(https://arxiv.org/abs/2402.09540)
• Keywords: privacy, attack, membership infer
• Abstract: For small privacy parameter $\epsilon$, $\epsilon$-differential privacy (DP) provides a strong worst-case guarantee that no membership inference attack (MIA) can succeed at determining whether a person's data was used to train a machine learning model. The guarantee of DP is worst-case because: a) it holds even if the attacker already knows the records of all but one person in the data set; and b) it holds uniformly over all data sets. In practical applications, such a worst-case guarantee may be overkill: practical attackers may lack exact knowledge of (nearly all of) the private data, and our data set might be easier to defend, in some sense, than the worst-case data set. Such considerations have motivated the industrial deployment of DP models with large privacy parameter (e.g. $\epsilon \geq 7$), and it has been observed empirically that DP with large $\epsilon$ can successfully defend against state-of-the-art MIAs. Existing DP theory cannot explain these empirical findings: e.g., the theoretical privacy guarantees of $\epsilon \geq 7$ are essentially vacuous. In this paper, we aim to close this gap between theory and practice and understand why a large DP parameter can prevent practical MIAs. To tackle this problem, we propose a new privacy notion called practical membership privacy (PMP). PMP models a practical attacker's uncertainty about the contents of the private data. The PMP parameter has a natural interpretation in terms of the success rate of a practical MIA on a given data set. We quantitatively analyze the PMP parameter of two fundamental DP mechanisms: the exponential mechanism and Gaussian mechanism. Our analysis reveals that a large DP parameter often translates into a much smaller PMP parameter, which guarantees strong privacy against practical MIAs. Using our findings, we offer principled guidance for practitioners in choosing the DP parameter.

Title: A 3D Memristor Architecture for In-Memory Computing Demonstrated with SHA3

• Authors: Muayad J. Aljafar, Rasika Joshi, John M. Acken
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2402.09545
• Pdf URL: https://arxiv.org/pdf/2402.09545
• Copy Paste: [[2402.09545]] A 3D Memristor Architecture for In-Memory Computing Demonstrated with SHA3(https://arxiv.org/abs/2402.09545)
• Keywords: security
• Abstract: Security is a growing problem that needs hardware support. Memristors provide an alternative technology for hardware-supported security implementation. This paper presents a specific technique that utilizes the benefits of hybrid CMOS-memristors technology demonstrated with SHA3 over implementations that use only memristor technology. In the proposed technique, SHA3 is implemented in a set of perpendicular crossbar arrays structured to facilitate logic implementation and circular bit rotation (Rho operation), which is perhaps the most complex operation in SHA3 when carried out in memristor arrays. The Rho operation itself is implemented with CMOS multiplexers (MUXs). The proposed accelerator is standby power-free and circumvents the memory access bottleneck in conventional computers. In addition, our design obscures the intermediate values from the I/O interface and outperforms the state-of-the-art memristor-based designs in terms of size and energy. Demonstrating the memristor implementation of SHA3 provides an impetus for utilizing memristors in information security applications.

Title: Automated Plaque Detection and Agatston Score Estimation on Non-Contrast CT Scans: A Multicenter Study

• Authors: Andrew M. Nguyen, Jianfei Liu, Tejas Sudharshan Mathai, Peter C. Grayson, Ronald M. Summers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09569
• Pdf URL: https://arxiv.org/pdf/2402.09569
• Copy Paste: [[2402.09569]] Automated Plaque Detection and Agatston Score Estimation on Non-Contrast CT Scans: A Multicenter Study(https://arxiv.org/abs/2402.09569)
• Keywords: segmentation
• Abstract: Coronary artery calcification (CAC) is a strong and independent predictor of cardiovascular disease (CVD). However, manual assessment of CAC often requires radiological expertise, time, and invasive imaging techniques. The purpose of this multicenter study is to validate an automated cardiac plaque detection model using a 3D multiclass nnU-Net for gated and non-gated non-contrast chest CT volumes. CT scans were performed at three tertiary care hospitals and collected as three datasets, respectively. Heart, aorta, and lung segmentations were determined using TotalSegmentator, while plaques in the coronary arteries and heart valves were manually labeled for 801 volumes. In this work we demonstrate how the nnU-Net semantic segmentation pipeline may be adapted to detect plaques in the coronary arteries and valves. With a linear correction, nnU-Net deep learning methods may also accurately estimate Agatston scores on chest non-contrast CT scans. Compared to manual Agatson scoring, automated Agatston scoring indicated a slope of the linear regression of 0.841 with an intercept of +16 HU (R2 = 0.97). These results are an improvement over previous work assessing automated Agatston score computation in non-gated CT scans.

Title: Changes by Butterflies: Farsighted Forecasting with Group Reservoir Transformer

• Authors: Md Kowsher, Jia Xu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09573
• Pdf URL: https://arxiv.org/pdf/2402.09573
• Copy Paste: [[2402.09573]] Changes by Butterflies: Farsighted Forecasting with Group Reservoir Transformer(https://arxiv.org/abs/2402.09573)
• Keywords: robust, transformer
• Abstract: In Chaos, a minor divergence between two initial conditions exhibits exponential amplification over time, leading to far-away outcomes, known as the butterfly effect. Thus, the distant future is full of uncertainty and hard to forecast. We introduce Group Reservoir Transformer to predict long-term events more accurately and robustly by overcoming two challenges in Chaos: (1) the extensive historical sequences and (2) the sensitivity to initial conditions. A reservoir is attached to a Transformer to efficiently handle arbitrarily long historical lengths, with an extension of a group of reservoirs to reduce the uncertainty due to the initialization variations. Our architecture consistently outperforms state-of-the-art DNN models in multivariate time series, including NLinear, Pyformer, Informer, Autoformer, and the baseline Transformer, with an error reduction of up to -89.43\% in various fields such as ETTh, ETTm, and air quality, demonstrating that an ensemble of butterfly learning, the prediction can be improved to a more adequate and certain one, despite of the traveling time to the unknown future.

Title: Complexity Reduction in Machine Learning-Based Wireless Positioning: Minimum Description Features

• Authors: Myeung Suk Oh, Anindya Bijoy Das, Taejoon Kim, David J. Love, Christopher G. Brinton
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.09580
• Pdf URL: https://arxiv.org/pdf/2402.09580
• Copy Paste: [[2402.09580]] Complexity Reduction in Machine Learning-Based Wireless Positioning: Minimum Description Features(https://arxiv.org/abs/2402.09580)
• Keywords: robust
• Abstract: A recent line of research has been investigating deep learning approaches to wireless positioning (WP). Although these WP algorithms have demonstrated high accuracy and robust performance against diverse channel conditions, they also have a major drawback: they require processing high-dimensional features, which can be prohibitive for mobile applications. In this work, we design a positioning neural network (P-NN) that substantially reduces the complexity of deep learning-based WP through carefully crafted minimum description features. Our feature selection is based on maximum power measurements and their temporal locations to convey information needed to conduct WP. We also develop a novel methodology for adaptively selecting the size of feature space, which optimizes over balancing the expected amount of useful information and classification capability, quantified using information-theoretic measures on the signal bin selection. Numerical results show that P-NN achieves a significant advantage in performance-complexity tradeoff over deep learning baselines that leverage the full power delay profile (PDP).

Title: Combatting deepfakes: Policies to address national security threats and rights violations

• Authors: Andrea Miotti, Akash Wasil
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.09581
• Pdf URL: https://arxiv.org/pdf/2402.09581
• Copy Paste: [[2402.09581]] Combatting deepfakes: Policies to address national security threats and rights violations(https://arxiv.org/abs/2402.09581)
• Keywords: security
• Abstract: This paper provides policy recommendations to address threats from deepfakes. First, we provide background information about deepfakes and review the harms they pose. We describe how deepfakes are currently used to proliferate sexual abuse material, commit fraud, manipulate voter behavior, and pose threats to national security. Second, we review previous legislative proposals designed to address deepfakes. Third, we present a comprehensive policy proposal that focuses on addressing multiple parts of the deepfake supply chain. The deepfake supply chain begins with a small number of model developers, model providers, and compute providers, and it expands to include billions of potential deepfake creators. We describe this supply chain in greater detail and describe how entities at each step of the supply chain ought to take reasonable measures to prevent the creation and proliferation of deepfakes. Finally, we address potential counterpoints of our proposal. Overall, deepfakes will present increasingly severe threats to global security and individual liberties. To address these threats, we call on policymakers to enact legislation that addresses multiple parts of the deepfake supply chain.

Title: DeepATLAS: One-Shot Localization for Biomedical Data

• Authors: Peter D. Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09587
• Pdf URL: https://arxiv.org/pdf/2402.09587
• Copy Paste: [[2402.09587]] DeepATLAS: One-Shot Localization for Biomedical Data(https://arxiv.org/abs/2402.09587)
• Keywords: segmentation
• Abstract: This paper introduces the DeepATLAS foundational model for localization tasks in the domain of high-dimensional biomedical data. Upon convergence of the proposed self-supervised objective, a pretrained model maps an input to an anatomically-consistent embedding from which any point or set of points (e.g., boxes or segmentations) may be identified in a one-shot or few-shot approach. As a representative benchmark, a DeepATLAS model pretrained on a comprehensive cohort of 51,000+ unlabeled 3D computed tomography exams yields high one-shot segmentation performance on over 50 anatomic structures across four different external test sets, either matching or exceeding the performance of a standard supervised learning model. Further improvements in accuracy can be achieved by adding a small amount of labeled data using either a semisupervised or more conventional fine-tuning strategy.

Title: Low-Rank Graph Contrastive Learning for Node Classification

• Authors: Yancheng Wang, Yingzhen Yang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.09600
• Pdf URL: https://arxiv.org/pdf/2402.09600
• Copy Paste: [[2402.09600]] Low-Rank Graph Contrastive Learning for Node Classification(https://arxiv.org/abs/2402.09600)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have been widely used to learn node representations and with outstanding performance on various tasks such as node classification. However, noise, which inevitably exists in real-world graph data, would considerably degrade the performance of GNNs revealed by recent studies. In this work, we propose a novel and robust GNN encoder, Low-Rank Graph Contrastive Learning (LR-GCL). Our method performs transductive node classification in two steps. First, a low-rank GCL encoder named LR-GCL is trained by prototypical contrastive learning with low-rank regularization. Next, using the features produced by LR-GCL, a linear transductive classification algorithm is used to classify the unlabeled nodes in the graph. Our LR-GCL is inspired by the low frequency property of the graph data and its labels, and it is also theoretically motivated by our sharp generalization bound for transductive learning. To the best of our knowledge, our theoretical result is among the first to theoretically demonstrate the advantage of low-rank learning in graph contrastive learning supported by strong empirical performance. Extensive experiments on public benchmarks demonstrate the superior performance of LR-GCL and the robustness of the learned node representations. The code of LR-GCL is available at \url{https://anonymous.4open.science/r/Low-Rank_Graph_Contrastive_Learning-64A6/}.

Title: Medical Image Segmentation with InTEnt: Integrated Entropy Weighting for Single Image Test-Time Adaptation

• Authors: Haoyu Dong, Nicholas Konz, Hanxue Gu, Maciej A. Mazurowski
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09604
• Pdf URL: https://arxiv.org/pdf/2402.09604
• Copy Paste: [[2402.09604]] Medical Image Segmentation with InTEnt: Integrated Entropy Weighting for Single Image Test-Time Adaptation(https://arxiv.org/abs/2402.09604)
• Keywords: segmentation
• Abstract: Test-time adaptation (TTA) refers to adapting a trained model to a new domain during testing. Existing TTA techniques rely on having multiple test images from the same domain, yet this may be impractical in real-world applications such as medical imaging, where data acquisition is expensive and imaging conditions vary frequently. Here, we approach such a task, of adapting a medical image segmentation model with only a single unlabeled test image. Most TTA approaches, which directly minimize the entropy of predictions, fail to improve performance significantly in this setting, in which we also observe the choice of batch normalization (BN) layer statistics to be a highly important yet unstable factor due to only having a single test domain example. To overcome this, we propose to instead \textit{integrate} over predictions made with various estimates of target domain statistics between the training and test statistics, weighted based on their entropy statistics.

Title: Towards Privacy-Aware Sign Language Translation at Scale

• Authors: Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgöz, Jean Maillard
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09611
• Pdf URL: https://arxiv.org/pdf/2402.09611
• Copy Paste: [[2402.09611]] Towards Privacy-Aware Sign Language Translation at Scale(https://arxiv.org/abs/2402.09611)
• Keywords: privacy, biometric
• Abstract: A major impediment to the advancement of sign language translation (SLT) is data scarcity. Much of the sign language data currently available on the web cannot be used for training supervised models due to the lack of aligned captions. Furthermore, scaling SLT using large-scale web-scraped datasets bears privacy risks due to the presence of biometric information, which the responsible development of SLT technologies should account for. In this work, we propose a two-stage framework for privacy-aware SLT at scale that addresses both of these issues. We introduce SSVP-SLT, which leverages self-supervised video pretraining on anonymized and unannotated videos, followed by supervised SLT finetuning on a curated parallel dataset. SSVP-SLT achieves state-of-the-art finetuned and zero-shot gloss-free SLT performance on the How2Sign dataset, outperforming the strongest respective baselines by over 3 BLEU-4. Based on controlled experiments, we further discuss the advantages and limitations of self-supervised pretraining and anonymization via facial obfuscation for SLT.

Title: Probabilistic Reasoning in Generative Large Language Models

• Authors: Aliakbar Nafar, Kristen Brent Venable, Parisa Kordjamshidi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09614
• Pdf URL: https://arxiv.org/pdf/2402.09614
• Copy Paste: [[2402.09614]] Probabilistic Reasoning in Generative Large Language Models(https://arxiv.org/abs/2402.09614)
• Keywords: generative, large language model
• Abstract: This paper considers the challenges that Large Language Models (LLMs) face when reasoning over text that includes information involving uncertainty explicitly quantified via probability values. This type of reasoning is relevant to a variety of contexts ranging from everyday conversations to medical decision-making. Despite improvements in the mathematical reasoning capabilities of LLMs, they still exhibit significant difficulties when it comes to probabilistic reasoning. To deal with this problem, we first introduce the Bayesian Linguistic Inference Dataset (BLInD), a new dataset specifically designed to test the probabilistic reasoning capabilities of LLMs. We then leverage this new dataset to thoroughly illustrate the specific limitations of LLMs for tasks involving probabilistic reasoning and present several strategies that map the problem to different formal representations, including Python code, probabilistic inference algorithms, and probabilistic logical programming. We conclude by providing an evaluation of our methods on BLInD and on an adaptation of a causal reasoning question-answering dataset, which further shows their practical effectiveness.

Title: API Pack: A Massive Multilingual Dataset for API Call Generation

• Authors: Zhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen, Rameswar Panda
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09615
• Pdf URL: https://arxiv.org/pdf/2402.09615
• Copy Paste: [[2402.09615]] API Pack: A Massive Multilingual Dataset for API Call Generation(https://arxiv.org/abs/2402.09615)
• Keywords: large language model
• Abstract: We introduce API Pack, a multilingual dataset featuring over one million instruction-API call pairs aimed at advancing large language models' API call generation capabilities. Through experiments, we demonstrate API Pack's efficacy in enhancing models for this specialized task while maintaining their overall proficiency at general coding. Fine-tuning CodeLlama-13B on just 20,000 Python instances yields over 10% and 5% higher accuracy than GPT-3.5 and GPT-4 respectively in generating unseen API calls. Scaling to 100k examples improves generalization to new APIs not seen during training. In addition, cross-lingual API call generation is achieved without needing extensive data per language. The dataset, fine-tuned models, and overall code base are publicly available at https://github.com/anonymous_url.

Title: Schnorr Approval-Based Secure and Privacy-Preserving IoV Data Aggregation

• Authors: Rui Liu, Jianping Pan
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.09621
• Pdf URL: https://arxiv.org/pdf/2402.09621
• Copy Paste: [[2402.09621]] Schnorr Approval-Based Secure and Privacy-Preserving IoV Data Aggregation(https://arxiv.org/abs/2402.09621)
• Keywords: secure, privacy, attack
• Abstract: Secure and privacy-preserving data aggregation in the Internet of Vehicles (IoV) continues to be a focal point of interest in both the industry and academia. Aiming at tackling the challenges and solving the remaining limitations of existing works, this paper introduces a novel Schnorr approval-based IoV data aggregation framework based on a two-layered architecture. In this framework, a server can aggregate the IoV data from clusters without inferring the raw data, real identity and trajectories of vehicles. Notably, we avoid incorporating the widely-accepted techniques such as homomorphic encryption and digital pseudonym to avoid introducing high computation cost to vehicles. We propose a novel concept, data approval, based on the Schnorr signature scheme. With the approval, the fake data injection attack carried out by a cluster head can be defended against. The separation of liability is achieved as well. The evaluation shows that the framework is secure and lightweight for vehicles in terms of the computation and communication costs.

Title: Smart Information Exchange for Unsupervised Federated Learning via Reinforcement Learning

• Authors: Seohyun Lee, Anindya Bijoy Das, Satyavrat Wagle, Christopher G. Brinton
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09629
• Pdf URL: https://arxiv.org/pdf/2402.09629
• Copy Paste: [[2402.09629]] Smart Information Exchange for Unsupervised Federated Learning via Reinforcement Learning(https://arxiv.org/abs/2402.09629)
• Keywords: robust, federate
• Abstract: One of the main challenges of decentralized machine learning paradigms such as Federated Learning (FL) is the presence of local non-i.i.d. datasets. Device-to-device transfers (D2D) between distributed devices has been shown to be an effective tool for dealing with this problem and robust to stragglers. In an unsupervised case, however, it is not obvious how data exchanges should take place due to the absence of labels. In this paper, we propose an approach to create an optimal graph for data transfer using Reinforcement Learning. The goal is to form links that will provide the most benefit considering the environment's constraints and improve convergence speed in an unsupervised FL environment. Numerical analysis shows the advantages in terms of convergence speed and straggler resilience of the proposed method to different available FL schemes and benchmark datasets.

Title: Answer is All You Need: Instruction-following Text Embedding via Answering the Question

• Authors: Letian Peng, Yuwei Zhang, Zilong Wang, Jayanth Srinivasa, Gaowen Liu, Zihan Wang, Jingbo Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09642
• Pdf URL: https://arxiv.org/pdf/2402.09642
• Copy Paste: [[2402.09642]] Answer is All You Need: Instruction-following Text Embedding via Answering the Question(https://arxiv.org/abs/2402.09642)
• Keywords: robust, interpretability, large language model
• Abstract: This work aims to build a text embedder that can capture characteristics of texts specified by user instructions. Despite its tremendous potential to deploy user-oriented embeddings, none of previous approaches provides a concrete solution for it. This paper offers a new viewpoint, which treats the instruction as a question about the input text and encodes the expected answers to obtain the representation accordingly. Intuitively, texts with the same (implicit) semantics would share similar answers following the instruction, thus leading to more similar embeddings. Specifically, we propose InBedder that instantiates this embed-via-answering idea by only fine-tuning language models on abstractive question answering tasks. InBedder demonstrates significantly improved instruction-following capabilities according to our proposed instruction awareness tests and instruction robustness tests, when applied to both large language models (LLMs) (e.g., llama-2-7b) and smaller encoder-based LMs (e.g., roberta-large). Additionally, our qualitative analysis of clustering outcomes, achieved by applying different instructions to the same corpus, demonstrates a high degree of interpretability.

Title: Characterizing the Modification Space of Signature IDS Rules

• Authors: Ryan Guide, Eric Pauley, Yohan Beugin, Ryan Sheatsley, Patrick McDaniel
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.09644
• Pdf URL: https://arxiv.org/pdf/2402.09644
• Copy Paste: [[2402.09644]] Characterizing the Modification Space of Signature IDS Rules(https://arxiv.org/abs/2402.09644)
• Keywords: security
• Abstract: Signature-based Intrusion Detection Systems (SIDSs) are traditionally used to detect malicious activity in networks. A notable example of such a system is Snort, which compares network traffic against a series of rules that match known exploits. Current SIDS rules are designed to minimize the amount of legitimate traffic flagged incorrectly, reducing the burden on network administrators. However, different use cases than the traditional one--such as researchers studying trends or analyzing modified versions of known exploits--may require SIDSs to be less constrained in their operation. In this paper, we demonstrate that applying modifications to real-world SIDS rules allow for relaxing some constraints and characterizing the performance space of modified rules. We develop an iterative approach for exploring the space of modifications to SIDS rules. By taking the modifications that expand the ROC curve of performance and altering them further, we show how to modify rules in a directed manner. Using traffic collected and identified as benign or malicious from a cloud telescope, we find that the removal of a single component from SIDS rules has the largest impact on the performance space. Effectively modifying SIDS rules to reduce constraints can enable a broader range of detection for various objectives, from increased security to research purposes.

Title: Hand Shape and Gesture Recognition using Multiscale Template Matching, Background Subtraction and Binary Image Analysis

• Authors: Ketan Suhaas Saichandran
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09663
• Pdf URL: https://arxiv.org/pdf/2402.09663
• Copy Paste: [[2402.09663]] Hand Shape and Gesture Recognition using Multiscale Template Matching, Background Subtraction and Binary Image Analysis(https://arxiv.org/abs/2402.09663)
• Keywords: extraction
• Abstract: This paper presents a hand shape classification approach employing multiscale template matching. The integration of background subtraction is utilized to derive a binary image of the hand object, enabling the extraction of key features such as centroid and bounding box. The methodology, while simple, demonstrates effectiveness in basic hand shape classification tasks, laying the foundation for potential applications in straightforward human-computer interaction scenarios. Experimental results highlight the system's capability in controlled environments.

Title: EntailE: Introducing Textual Entailment in Commonsense Knowledge Graph Completion

• Authors: Ying Su, Tianqing Fang, Huiru Xiao, Weiqi Wang, Yangqiu Song, Tong Zhang, Lei Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09666
• Pdf URL: https://arxiv.org/pdf/2402.09666
• Copy Paste: [[2402.09666]] EntailE: Introducing Textual Entailment in Commonsense Knowledge Graph Completion(https://arxiv.org/abs/2402.09666)
• Keywords: transformer
• Abstract: Commonsense knowledge graph completion is a new challenge for commonsense knowledge graph construction and application. In contrast to factual knowledge graphs such as Freebase and YAGO, commonsense knowledge graphs (CSKGs; e.g., ConceptNet) utilize free-form text to represent named entities, short phrases, and events as their nodes. Such a loose structure results in large and sparse CSKGs, which makes the semantic understanding of these nodes more critical for learning rich commonsense knowledge graph embedding. While current methods leverage semantic similarities to increase the graph density, the semantic plausibility of the nodes and their relations are under-explored. Previous works adopt conceptual abstraction to improve the consistency of modeling (event) plausibility, but they are not scalable enough and still suffer from data sparsity. In this paper, we propose to adopt textual entailment to find implicit entailment relations between CSKG nodes, to effectively densify the subgraph connecting nodes within the same conceptual class, which indicates a similar level of plausibility. Each node in CSKG finds its top entailed nodes using a finetuned transformer over natural language inference (NLI) tasks, which sufficiently capture textual entailment signals. The entailment relation between these nodes are further utilized to: 1) build new connections between source triplets and entailed nodes to densify the sparse CSKGs; 2) enrich the generalization ability of node representations by comparing the node embeddings with a contrastive loss. Experiments on two standard CSKGs demonstrate that our proposed framework EntailE can improve the performance of CSKG completion tasks under both transductive and inductive settings.

Title: How to Train Data-Efficient LLMs

• Authors: Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley, Derek Zhiyuan Cheng
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09668
• Pdf URL: https://arxiv.org/pdf/2402.09668
• Copy Paste: [[2402.09668]] How to Train Data-Efficient LLMs(https://arxiv.org/abs/2402.09668)
• Keywords: large language model
• Abstract: The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.

Title: Exploiting Alpha Transparency In Language And Vision-Based AI Systems

• Authors: David Noever, Forrest McKee
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09671
• Pdf URL: https://arxiv.org/pdf/2402.09671
• Copy Paste: [[2402.09671]] Exploiting Alpha Transparency In Language And Vision-Based AI Systems(https://arxiv.org/abs/2402.09671)
• Keywords: security, attack
• Abstract: This investigation reveals a novel exploit derived from PNG image file formats, specifically their alpha transparency layer, and its potential to fool multiple AI vision systems. Our method uses this alpha layer as a clandestine channel invisible to human observers but fully actionable by AI image processors. The scope tested for the vulnerability spans representative vision systems from Apple, Microsoft, Google, Salesforce, Nvidia, and Facebook, highlighting the attack's potential breadth. This vulnerability challenges the security protocols of existing and fielded vision systems, from medical imaging to autonomous driving technologies. Our experiments demonstrate that the affected systems, which rely on convolutional neural networks or the latest multimodal language models, cannot quickly mitigate these vulnerabilities through simple patches or updates. Instead, they require retraining and architectural changes, indicating a persistent hole in multimodal technologies without some future adversarial hardening against such vision-language exploits.

Title: PAL: Proxy-Guided Black-Box Attack on Large Language Models

• Authors: Chawin Sitawarin, Norman Mu, David Wagner, Alexandre Araujo
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09674
• Pdf URL: https://arxiv.org/pdf/2402.09674
• Copy Paste: [[2402.09674]] PAL: Proxy-Guided Black-Box Attack on Large Language Models(https://arxiv.org/abs/2402.09674)
• Keywords: security, attack, large language model
• Abstract: Large Language Models (LLMs) have surged in popularity in recent months, but they have demonstrated concerning capabilities to generate harmful content when manipulated. While techniques like safety fine-tuning aim to minimize harmful use, recent works have shown that LLMs remain vulnerable to attacks that elicit toxic responses. In this work, we introduce the Proxy-Guided Attack on LLMs (PAL), the first optimization-based attack on LLMs in a black-box query-only setting. In particular, it relies on a surrogate model to guide the optimization and a sophisticated loss designed for real-world LLM APIs. Our attack achieves 84% attack success rate (ASR) on GPT-3.5-Turbo and 48% on Llama-2-7B, compared to 4% for the current state of the art. We also propose GCG++, an improvement to the GCG attack that reaches 94% ASR on white-box Llama-2-7B, and the Random-Search Attack on LLMs (RAL), a strong but simple baseline for query-based attacks. We believe the techniques proposed in this work will enable more comprehensive safety testing of LLMs and, in the long term, the development of better security guardrails. The code can be found at https://github.com/chawins/pal.

Title: Prompt-based Personalized Federated Learning for Medical Visual Question Answering

• Authors: He Zhu, Ren Togo, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09677
• Pdf URL: https://arxiv.org/pdf/2402.09677
• Copy Paste: [[2402.09677]] Prompt-based Personalized Federated Learning for Medical Visual Question Answering(https://arxiv.org/abs/2402.09677)
• Keywords: privacy, federate, transformer
• Abstract: We present a novel prompt-based personalized federated learning (pFL) method to address data heterogeneity and privacy concerns in traditional medical visual question answering (VQA) methods. Specifically, we regard medical datasets from different organs as clients and use pFL to train personalized transformer-based VQA models for each client. To address the high computational complexity of client-to-client communication in previous pFL methods, we propose a succinct information sharing system by introducing prompts that are small learnable parameters. In addition, the proposed method introduces a reliability parameter to prevent the negative effects of low performance and irrelevant clients. Finally, extensive evaluations on various heterogeneous medical datasets attest to the effectiveness of our proposed method.

Title: Seed Optimization with Frozen Generator for Superior Zero-shot Low-light Enhancement

• Authors: Yuxuan Gu, Yi Jin, Ben Wang, Zhixiang Wei, Xiaoxiao Ma, Pengyang Ling, Haoxuan Wang, Huaian Chen, Enhong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09694
• Pdf URL: https://arxiv.org/pdf/2402.09694
• Copy Paste: [[2402.09694]] Seed Optimization with Frozen Generator for Superior Zero-shot Low-light Enhancement(https://arxiv.org/abs/2402.09694)
• Keywords: generative
• Abstract: In this work, we observe that the generators, which are pre-trained on massive natural images, inherently hold the promising potential for superior low-light image enhancement against varying scenarios.Specifically, we embed a pre-trained generator to Retinex model to produce reflectance maps with enhanced detail and vividness, thereby recovering features degraded by low-light conditions.Taking one step further, we introduce a novel optimization strategy, which backpropagates the gradients to the input seeds rather than the parameters of the low-light enhancement model, thus intactly retaining the generative knowledge learned from natural images and achieving faster convergence speed. Benefiting from the pre-trained knowledge and seed-optimization strategy, the low-light enhancement model can significantly regularize the realness and fidelity of the enhanced result, thus rapidly generating high-quality images without training on any low-light dataset. Extensive experiments on various benchmarks demonstrate the superiority of the proposed method over numerous state-of-the-art methods qualitatively and quantitatively.

Title: Reward Poisoning Attack Against Offline Reinforcement Learning

• Authors: Yinglun Xu, Rohan Gumaste, Gagandeep Singh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09695
• Pdf URL: https://arxiv.org/pdf/2402.09695
• Copy Paste: [[2402.09695]] Reward Poisoning Attack Against Offline Reinforcement Learning(https://arxiv.org/abs/2402.09695)
• Keywords: attack
• Abstract: We study the problem of reward poisoning attacks against general offline reinforcement learning with deep neural networks for function approximation. We consider a black-box threat model where the attacker is completely oblivious to the learning algorithm and its budget is limited by constraining both the amount of corruption at each data point, and the total perturbation. We propose an attack strategy called `policy contrast attack'. The high-level idea is to make some low-performing policies appear as high-performing while making high-performing policies appear as low-performing. To the best of our knowledge, we propose the first black-box reward poisoning attack in the general offline RL setting. We provide theoretical insights on the attack design and empirically show that our attack is efficient against current state-of-the-art offline RL algorithms in different kinds of learning datasets.

Title: HOACS: Homomorphic Obfuscation Assisted Concealing of Secrets to Thwart Trojan Attacks in COTS Processor

• Authors: Tanvir Hossain, Matthew Showers, Mahmudul Hasan, Tamzidul Hoque
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.09701
• Pdf URL: https://arxiv.org/pdf/2402.09701
• Copy Paste: [[2402.09701]] HOACS: Homomorphic Obfuscation Assisted Concealing of Secrets to Thwart Trojan Attacks in COTS Processor(https://arxiv.org/abs/2402.09701)
• Keywords: security, protect, attack
• Abstract: Commercial-off-the-shelf (COTS) components are often preferred over custom Integrated Circuits (ICs) to achieve reduced system development time and cost, easy adoption of new technologies, and replaceability. Unfortunately, the integration of COTS components introduces serious security concerns. None of the entities in the COTS IC supply chain are trusted from a consumer's perspective, leading to a ''zero trust'' threat model. Any of these entities could introduce hidden malicious circuits or hardware Trojans within the component, allowing an attacker in the field to extract secret information (e.g., cryptographic keys) or cause a functional failure. Existing solutions to counter hardware Trojans are inapplicable in such a zero-trust scenario as they assume either the design house or the foundry to be trusted and consider the design to be available for either analysis or modification. In this work, we have proposed a software-oriented countermeasure to ensure the confidentiality of secret assets against hardware Trojans that can be seamlessly integrated in existing COTS microprocessors. The proposed solution does not require any supply chain entity to be trusted and does not require analysis or modification of the IC design. To protect secret assets in an untrusted microprocessor, the proposed method leverages the concept of residue number coding (RNC) to transform the software functions operating on the asset to be fully homomorphic. We have implemented the proposed solution to protect the secret key within the Advanced Encryption Standard (AES) program and presented a detailed security analysis. We also have developed a plugin for the LLVM compiler toolchain that automatically integrates the solution in AES. Finally, we compare the execution time overhead of the operations in the RNC-based technique with comparable homomorphic solutions and demonstrate significant improvement.

Title: Preserving Data Privacy for ML-driven Applications in Open Radio Access Networks

• Authors: Pranshav Gajjar, Azuka Chiejina, Vijay K. Shah
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09710
• Pdf URL: https://arxiv.org/pdf/2402.09710
• Copy Paste: [[2402.09710]] Preserving Data Privacy for ML-driven Applications in Open Radio Access Networks(https://arxiv.org/abs/2402.09710)
• Keywords: privacy, transformer
• Abstract: Deep learning offers a promising solution to improve spectrum access techniques by utilizing data-driven approaches to manage and share limited spectrum resources for emerging applications. For several of these applications, the sensitive wireless data (such as spectrograms) are stored in a shared database or multistakeholder cloud environment and are therefore prone to privacy leaks. This paper aims to address such privacy concerns by examining the representative case study of shared database scenarios in 5G Open Radio Access Network (O-RAN) networks where we have a shared database within the near-real-time (near-RT) RAN intelligent controller. We focus on securing the data that can be used by machine learning (ML) models for spectrum sharing and interference mitigation applications without compromising the model and network performances. The underlying idea is to leverage a (i) Shuffling-based learnable encryption technique to encrypt the data, following which, (ii) employ a custom Vision transformer (ViT) as the trained ML model that is capable of performing accurate inferences on such encrypted data. The paper offers a thorough analysis and comparisons with analogous convolutional neural networks (CNN) as well as deeper architectures (such as ResNet-50) as baselines. Our experiments showcase that the proposed approach significantly outperforms the baseline CNN with an improvement of 24.5% and 23.9% for the percent accuracy and F1-Score respectively when operated on encrypted data. Though deeper ResNet-50 architecture is obtained as a slightly more accurate model, with an increase of 4.4%, the proposed approach boasts a reduction of parameters by 99.32%, and thus, offers a much-improved prediction time by nearly 60%.

Title: Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement

• Authors: Tao Yang, Cuiling Lan, Yan Lu, Nanning zheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09712
• Pdf URL: https://arxiv.org/pdf/2402.09712
• Copy Paste: [[2402.09712]] Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement(https://arxiv.org/abs/2402.09712)
• Keywords: diffusion
• Abstract: Disentangled representation learning strives to extract the intrinsic factors within observed data. Factorizing these representations in an unsupervised manner is notably challenging and usually requires tailored loss functions or specific structural designs. In this paper, we introduce a new perspective and framework, demonstrating that diffusion models with cross-attention can serve as a powerful inductive bias to facilitate the learning of disentangled representations. We propose to encode an image to a set of concept tokens and treat them as the condition of the latent diffusion for image reconstruction, where cross-attention over the concept tokens is used to bridge the interaction between the encoder and diffusion. Without any additional regularization, this framework achieves superior disentanglement performance on the benchmark datasets, surpassing all previous methods with intricate designs. We have conducted comprehensive ablation studies and visualization analysis, shedding light on the functioning of this model. This is the first work to reveal the potent disentanglement capability of diffusion models with cross-attention, requiring no complex designs. We anticipate that our findings will inspire more investigation on exploring diffusion for disentangled representation learning towards more sophisticated data analysis and understanding.

Title: Visually Dehallucinative Instruction Generation: Know What You Don't Know

• Authors: Sungguk Cha, Jusung Lee, Younghyun Lee, Cheoljong Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09717
• Pdf URL: https://arxiv.org/pdf/2402.09717
• Copy Paste: [[2402.09717]] Visually Dehallucinative Instruction Generation: Know What You Don't Know(https://arxiv.org/abs/2402.09717)
• Keywords: generative
• Abstract: "When did the emperor Napoleon invented iPhone?" Such hallucination-inducing question is well known challenge in generative language modeling. In this study, we present an innovative concept of visual hallucination, referred to as "I Know (IK)" hallucination, to address scenarios where "I Don't Know" is the desired response. To effectively tackle this issue, we propose the VQAv2-IDK benchmark, the subset of VQAv2 comprising unanswerable image-question pairs as determined by human annotators. Stepping further, we present the visually dehallucinative instruction generation method for IK hallucination and introduce the IDK-Instructions visual instruction database. Our experiments show that current methods struggle with IK hallucination. Yet, our approach effectively reduces these hallucinations, proving its versatility across different frameworks and datasets.

Title: Region Feature Descriptor Adapted to High Affine Transformations

• Authors: Shaojie Zhang, Yinghui Wang, Peixuan Liu, Jinlong Yang, Tao Yan, Liangyi Huang, Mingfeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09724
• Pdf URL: https://arxiv.org/pdf/2402.09724
• Copy Paste: [[2402.09724]] Region Feature Descriptor Adapted to High Affine Transformations(https://arxiv.org/abs/2402.09724)
• Keywords: robust
• Abstract: To address the issue of feature descriptors being ineffective in representing grayscale feature information when images undergo high affine transformations, leading to a rapid decline in feature matching accuracy, this paper proposes a region feature descriptor based on simulating affine transformations using classification. The proposed method initially categorizes images with different affine degrees to simulate affine transformations and generate a new set of images. Subsequently, it calculates neighborhood information for feature points on this new image set. Finally, the descriptor is generated by combining the grayscale histogram of the maximum stable extremal region to which the feature point belongs and the normalized position relative to the grayscale centroid of the feature point's region. Experimental results, comparing feature matching metrics under affine transformation scenarios, demonstrate that the proposed descriptor exhibits higher precision and robustness compared to existing classical descriptors. Additionally, it shows robustness when integrated with other descriptors.

Title: Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularization

• Authors: Xinran Chen, Sufeng Duan, Gongshen Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09725
• Pdf URL: https://arxiv.org/pdf/2402.09725
• Copy Paste: [[2402.09725]] Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularization(https://arxiv.org/abs/2402.09725)
• Keywords: transformer
• Abstract: Being one of the IR-NAT (Iterative-refinemennt-based NAT) frameworks, the Conditional Masked Language Model (CMLM) adopts the mask-predict paradigm to re-predict the masked low-confidence tokens. However, CMLM suffers from the data distribution discrepancy between training and inference, where the observed tokens are generated differently in the two cases. In this paper, we address this problem with the training approaches of error exposure and consistency regularization (EECR). We construct the mixed sequences based on model prediction during training, and propose to optimize over the masked tokens under imperfect observation conditions. We also design a consistency learning method to constrain the data distribution for the masked tokens under different observing situations to narrow down the gap between training and inference. The experiments on five translation benchmarks obtains an average improvement of 0.68 and 0.40 BLEU scores compared to the base models, respectively, and our CMLMC-EECR achieves the best performance with a comparable translation quality with the Transformer. The experiments results demonstrate the effectiveness of our method.

Title: A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts

• Authors: Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, Ian Fischer
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.09727
• Pdf URL: https://arxiv.org/pdf/2402.09727
• Copy Paste: [[2402.09727]] A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts(https://arxiv.org/abs/2402.09727)
• Keywords: robust, large language model
• Abstract: Current Large Language Models (LLMs) are not only limited to some maximum context length, but also are not able to robustly consume long inputs. To address these limitations, we propose ReadAgent, an LLM agent system that increases effective context length up to 20x in our experiments. Inspired by how humans interactively read long documents, we implement ReadAgent as a simple prompting system that uses the advanced language capabilities of LLMs to (1) decide what content to store together in a memory episode, (2) compress those memory episodes into short episodic memories called gist memories, and (3) take actions to look up passages in the original text if ReadAgent needs to remind itself of relevant details to complete a task. We evaluate ReadAgent against baselines using retrieval methods, using the original long contexts, and using the gist memories. These evaluations are performed on three long-document reading comprehension tasks: QuALITY, NarrativeQA, and QMSum. ReadAgent outperforms the baselines on all three tasks while extending the effective context window by 3-20x.

Title: AbuseGPT: Abuse of Generative AI ChatBots to Create Smishing Campaigns

• Authors: Ashfak Md Shibli, Mir Mehedi A. Pritom, Maanak Gupta
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09728
• Pdf URL: https://arxiv.org/pdf/2402.09728
• Copy Paste: [[2402.09728]] AbuseGPT: Abuse of Generative AI ChatBots to Create Smishing Campaigns(https://arxiv.org/abs/2402.09728)
• Keywords: security, protect, attack, generative, large language model
• Abstract: SMS phishing, also known as "smishing", is a growing threat that tricks users into disclosing private information or clicking into URLs with malicious content through fraudulent mobile text messages. In recent past, we have also observed a rapid advancement of conversational generative AI chatbot services (e.g., OpenAI's ChatGPT, Google's BARD), which are powered by pre-trained large language models (LLMs). These AI chatbots certainly have a lot of utilities but it is not systematically understood how they can play a role in creating threats and attacks. In this paper, we propose AbuseGPT method to show how the existing generative AI-based chatbot services can be exploited by attackers in real world to create smishing texts and eventually lead to craftier smishing campaigns. To the best of our knowledge, there is no pre-existing work that evidently shows the impacts of these generative text-based models on creating SMS phishing. Thus, we believe this study is the first of its kind to shed light on this emerging cybersecurity threat. We have found strong empirical evidences to show that attackers can exploit ethical standards in the existing generative AI-based chatbot services by crafting prompt injection attacks to create newer smishing campaigns. We also discuss some future research directions and guidelines to protect the abuse of generative AI-based services and safeguard users from smishing attacks.

Title: Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States

• Authors: Hanyu Duan, Yi Yang, Kar Yan Tam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09733
• Pdf URL: https://arxiv.org/pdf/2402.09733
• Copy Paste: [[2402.09733]] Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States(https://arxiv.org/abs/2402.09733)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) can make up answers that are not real, and this is known as hallucination. This research aims to see if, how, and to what extent LLMs are aware of hallucination. More specifically, we check whether and how an LLM reacts differently in its hidden states when it answers a question right versus when it hallucinates. To do this, we introduce an experimental framework which allows examining LLM's hidden states in different hallucination situations. Building upon this framework, we conduct a series of experiments with language models in the LLaMA family (Touvron et al., 2023). Our empirical findings suggest that LLMs react differently when processing a genuine response versus a fabricated one. We then apply various model interpretation techniques to help understand and explain the findings better. Moreover, informed by the empirical observations, we show great potential of using the guidance derived from LLM's hidden representation space to mitigate hallucination. We believe this work provides insights into how LLMs produce hallucinated answers and how to make them occur less often.

Title: DFORM: Diffeomorphic vector field alignment for assessing dynamics across learned models

• Authors: Ruiqi Chen, Giacomo Vedovati, Todd Braver, ShiNung Ching
• Subjects: cs.LG, eess.SY, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2402.09735
• Pdf URL: https://arxiv.org/pdf/2402.09735
• Copy Paste: [[2402.09735]] DFORM: Diffeomorphic vector field alignment for assessing dynamics across learned models(https://arxiv.org/abs/2402.09735)
• Keywords: generative
• Abstract: Dynamical system models such as Recurrent Neural Networks (RNNs) have become increasingly popular as hypothesis-generating tools in scientific research. Evaluating the dynamics in such networks is key to understanding their learned generative mechanisms. However, comparison of learned dynamics across models is challenging due to their inherent nonlinearity and because a priori there is no enforced equivalence of their coordinate systems. Here, we propose the DFORM (Diffeomorphic vector field alignment for comparing dynamics across learned models) framework. DFORM learns a nonlinear coordinate transformation which provides a continuous, maximally one-to-one mapping between the trajectories of learned models, thus approximating a diffeomorphism between them. The mismatch between DFORM-transformed vector fields defines the orbital similarity between two models, thus providing a generalization of the concepts of smooth orbital and topological equivalence. As an example, we apply DFORM to models trained on a canonical neuroscience task, showing that learned dynamics may be functionally similar, despite overt differences in attractor landscapes.

Title: AI Hospital: Interactive Evaluation and Collaboration of LLMs as Intern Doctors for Clinical Diagnosis

• Authors: Zhihao Fan, Jialong Tang, Wei Chen, Siyuan Wang, Zhongyu Wei, Jun Xi, Fei Huang, Jingren Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09742
• Pdf URL: https://arxiv.org/pdf/2402.09742
• Copy Paste: [[2402.09742]] AI Hospital: Interactive Evaluation and Collaboration of LLMs as Intern Doctors for Clinical Diagnosis(https://arxiv.org/abs/2402.09742)
• Keywords: large language model
• Abstract: The incorporation of Large Language Models (LLMs) in healthcare marks a significant advancement. However, the application has predominantly been limited to discriminative and question-answering tasks, which does not fully leverage their interactive potential. To address this limitation, our paper presents AI Hospital, a framework designed to build a real-time interactive diagnosis environment. To simulate the procedure, we collect high-quality medical records to create patient, examiner, and medical director agents. AI Hospital is then utilized for the interactive evaluation and collaboration of LLMs. Initially, we create a Multi-View Medical Evaluation (MVME) benchmark where various LLMs serve as intern doctors for interactive diagnosis. Subsequently, to improve diagnostic accuracy, we introduce a collaborative mechanism that involves iterative discussions and a dispute resolution process under the supervision of the medical director. In our experiments, we validate the reliability of AI Hospital. The results not only explore the feasibility of apply LLMs in clinical consultation but also confirm the effectiveness of the dispute resolution focused collaboration method.

Title: Model Compression and Efficient Inference for Large Language Models: A Survey

• Authors: Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, Xiaofei He
• Subjects: cs.CL, cs.AI, cs.LG, cs.PF
• Abstract URL: https://arxiv.org/abs/2402.09748
• Pdf URL: https://arxiv.org/pdf/2402.09748
• Copy Paste: [[2402.09748]] Model Compression and Efficient Inference for Large Language Models: A Survey(https://arxiv.org/abs/2402.09748)
• Keywords: transformer, large language model
• Abstract: Transformer based large language models have achieved tremendous success. However, the significant memory and computational costs incurred during the inference process make it challenging to deploy large models on resource-constrained devices. In this paper, we investigate compression and efficient inference methods for large language models from an algorithmic perspective. Regarding taxonomy, similar to smaller models, compression and acceleration algorithms for large language models can still be categorized into quantization, pruning, distillation, compact architecture design, dynamic networks. However, Large language models have two prominent characteristics compared to smaller models: (1) Most of compression algorithms require finetuning or even retraining the model after compression. The most notable aspect of large models is the very high cost associated with model finetuning or training. Therefore, many algorithms for large models, such as quantization and pruning, start to explore tuning-free algorithms. (2) Large models emphasize versatility and generalization rather than performance on a single task. Hence, many algorithms, such as knowledge distillation, focus on how to preserving their versatility and generalization after compression. Since these two characteristics were not very pronounced in early large models, we further distinguish large language models into medium models and ``real'' large models. Additionally, we also provide an introduction to some mature frameworks for efficient inference of large models, which can support basic compression or acceleration algorithms, greatly facilitating model deployment for users.

Title: Efficient Language Adaptive Pre-training: Extending State-of-the-Art Large Language Models for Polish

• Authors: Szymon Ruciński
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09759
• Pdf URL: https://arxiv.org/pdf/2402.09759
• Copy Paste: [[2402.09759]] Efficient Language Adaptive Pre-training: Extending State-of-the-Art Large Language Models for Polish(https://arxiv.org/abs/2402.09759)
• Keywords: large language model
• Abstract: This study explores the potential of fine-tuning foundational English Large Language Models (LLMs) for generating Polish text. The first step involves Language Adaptive Pre-training (LAPT) on a high-quality dataset of 3.11 GB, consisting of 276 million Polish tokens. The LAPT is followed by additional fine-tuning aimed at solving nine KLEJ challenges. Our trained model Curie-7B-v1 not only generates Polish text with the lowest perplexity of 3.02 among decoder-based Polish models but also closely rivals the performance of the best Polish encoder-decoder models with a less than 2% gap on 8 out of 9 tasks. Curie-7B-v1 used approximately 2-3% of a typical dataset size to learn Polish. The LAPT was completed in less than five days using a consumer GPU, highlighting the method's efficiency. The proficiency of the model in Polish was significantly enhanced, demonstrating the viability of this approach for adding new languages to existing LLMs by training just 1.2% of its parameters. To contribute to the community's collaborative progress, the model has been released as open-source.

Title: NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models

• Authors: Shengrui Li, Xueting Han, Jing Bai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09773
• Pdf URL: https://arxiv.org/pdf/2402.09773
• Copy Paste: [[2402.09773]] NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models(https://arxiv.org/abs/2402.09773)
• Keywords: large language model
• Abstract: The considerable size of Large Language Models (LLMs) presents notable deployment challenges, particularly on resource-constrained hardware. Structured pruning, offers an effective means to compress LLMs, thereby reducing storage costs and enhancing inference speed for more efficient utilization. In this work, we study data-efficient and resource-efficient structure pruning methods to obtain smaller yet still powerful models. Knowledge Distillation is well-suited for pruning, as the intact model can serve as an excellent teacher for pruned students. However, it becomes challenging in the context of LLMs due to memory constraints. To address this, we propose an efficient progressive Numerous-teacher pruning method (NutePrune). NutePrune mitigates excessive memory costs by loading only one intact model and integrating it with various masks and LoRA modules, enabling it to seamlessly switch between teacher and student roles. This approach allows us to leverage numerous teachers with varying capacities to progressively guide the pruned model, enhancing overall performance. Extensive experiments across various tasks demonstrate the effectiveness of NutePrune. In LLaMA-7B zero-shot experiments, NutePrune retains 97.17% of the performance of the original model at 20% sparsity and 95.07% at 25% sparsity.

Title: A Comprehensive Review on Computer Vision Analysis of Aerial Data

• Authors: Vivek Tetarwal, Sandeep Kumar
• Subjects: cs.CV, cs.IT
• Abstract URL: https://arxiv.org/abs/2402.09781
• Pdf URL: https://arxiv.org/pdf/2402.09781
• Copy Paste: [[2402.09781]] A Comprehensive Review on Computer Vision Analysis of Aerial Data(https://arxiv.org/abs/2402.09781)
• Keywords: segmentation
• Abstract: With the emergence of new technologies in the field of airborne platforms and imaging sensors, aerial data analysis is becoming very popular, capitalizing on its advantages over land data. This paper presents a comprehensive review of the computer vision tasks within the domain of aerial data analysis. While addressing fundamental aspects such as object detection and tracking, the primary focus is on pivotal tasks like change detection, object segmentation, and scene-level analysis. The paper provides the comparison of various hyper parameters employed across diverse architectures and tasks. A substantial section is dedicated to an in-depth discussion on libraries, their categorization, and their relevance to different domain expertise. The paper encompasses aerial datasets, the architectural nuances adopted, and the evaluation metrics associated with all the tasks in aerial data analysis. Applications of computer vision tasks in aerial data across different domains are explored, with case studies providing further insights. The paper thoroughly examines the challenges inherent in aerial data analysis, offering practical solutions. Additionally, unresolved issues of significance are identified, paving the way for future research directions in the field of aerial data analysis.

Title: Examining Pathological Bias in a Generative Adversarial Network Discriminator: A Case Study on a StyleGAN3 Model

• Authors: Alvin Grissom II, Ryan F. Lei, Jeova Farias Sales Rocha Neto, Bailey Lin, Ryan Trotter
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09786
• Pdf URL: https://arxiv.org/pdf/2402.09786
• Copy Paste: [[2402.09786]] Examining Pathological Bias in a Generative Adversarial Network Discriminator: A Case Study on a StyleGAN3 Model(https://arxiv.org/abs/2402.09786)
• Keywords: generative
• Abstract: Generative adversarial networks generate photorealistic faces that are often indistinguishable by humans from real faces. We find that the discriminator in the pre-trained StyleGAN3 model, a popular GAN network, systematically stratifies scores by both image- and face-level qualities and that this disproportionately affects images across gender, race, and other categories. We examine the discriminator's bias for color and luminance across axes perceived race and gender; we then examine axes common in research on stereotyping in social psychology.

Title: An advanced data fabric architecture leveraging homomorphic encryption and federated learning

• Authors: Sakib Anwar Rieyan, Md. Raisul Kabir News, A.B.M. Muntasir Rahman, Sadia Afrin Khan, Sultan Tasneem Jawad Zaarif, Md. Golam Rabiul Alam, Mohammad Mehedi Hassan, Michele Ianni, Giancarlo Fortino
• Subjects: cs.CR, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2402.09795
• Pdf URL: https://arxiv.org/pdf/2402.09795
• Copy Paste: [[2402.09795]] An advanced data fabric architecture leveraging homomorphic encryption and federated learning(https://arxiv.org/abs/2402.09795)
• Keywords: secure, security, privacy, federate
• Abstract: Data fabric is an automated and AI-driven data fusion approach to accomplish data management unification without moving data to a centralized location for solving complex data problems. In a Federated learning architecture, the global model is trained based on the learned parameters of several local models that eliminate the necessity of moving data to a centralized repository for machine learning. This paper introduces a secure approach for medical image analysis using federated learning and partially homomorphic encryption within a distributed data fabric architecture. With this method, multiple parties can collaborate in training a machine-learning model without exchanging raw data but using the learned or fused features. The approach complies with laws and regulations such as HIPAA and GDPR, ensuring the privacy and security of the data. The study demonstrates the method's effectiveness through a case study on pituitary tumor classification, achieving a significant level of accuracy. However, the primary focus of the study is on the development and evaluation of federated learning and partially homomorphic encryption as tools for secure medical image analysis. The results highlight the potential of these techniques to be applied to other privacy-sensitive domains and contribute to the growing body of research on secure and privacy-preserving machine learning.

Title: EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models

• Authors: Shangyu Xing, Fei Zhao, Zhen Wu, Tuo An, Weihao Chen, Chunhui Li, Jianbing Zhang, Xinyu Dai
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09801
• Pdf URL: https://arxiv.org/pdf/2402.09801
• Copy Paste: [[2402.09801]] EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models(https://arxiv.org/abs/2402.09801)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) have attracted increasing attention in the past few years, but they may still generate descriptions that include objects not present in the corresponding images, a phenomenon known as object hallucination. To eliminate hallucinations, existing methods manually annotate paired responses with and without hallucinations, and then employ various alignment algorithms to improve the alignment capability between images and text. However, they not only demand considerable computation resources during the finetuning stage but also require expensive human annotation to construct paired data needed by the alignment algorithms. To address these issues, we borrow the idea of unlearning and propose an efficient fine-grained unlearning framework (EFUF), which can eliminate hallucinations without the need for paired data. Extensive experiments show that our method consistently reduces hallucinations while preserving the generation quality with modest computational overhead. Our code and datasets will be publicly available.

Title: DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization

• Authors: Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, Seunggyu Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09812
• Pdf URL: https://arxiv.org/pdf/2402.09812
• Copy Paste: [[2402.09812]] DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization(https://arxiv.org/abs/2402.09812)
• Keywords: diffusion
• Abstract: The objective of text-to-image (T2I) personalization is to customize a diffusion model to a user-provided reference concept, generating diverse images of the concept aligned with the target prompts. Conventional methods representing the reference concepts using unique text embeddings often fail to accurately mimic the appearance of the reference. To address this, one solution may be explicitly conditioning the reference images into the target denoising process, known as key-value replacement. However, prior works are constrained to local editing since they disrupt the structure path of the pre-trained T2I model. To overcome this, we propose a novel plug-in method, called DreamMatcher, which reformulates T2I personalization as semantic matching. Specifically, DreamMatcher replaces the target values with reference values aligned by semantic matching, while leaving the structure path unchanged to preserve the versatile capability of pre-trained T2I models for generating diverse structures. We also introduce a semantic-consistent masking strategy to isolate the personalized concept from irrelevant regions introduced by the target prompts. Compatible with existing T2I models, DreamMatcher shows significant improvements in complex scenarios. Intensive analyses demonstrate the effectiveness of our approach.

Title: Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment

• Authors: Angelos Zavras, Dimitrios Michail, Begüm Demir, Ioannis Papoutsis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09816
• Pdf URL: https://arxiv.org/pdf/2402.09816
• Copy Paste: [[2402.09816]] Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment(https://arxiv.org/abs/2402.09816)
• Keywords: robust
• Abstract: Deep Learning (DL) is undergoing a paradigm shift with the emergence of foundation models, aptly named by their crucial, yet incomplete nature. In this work, we focus on Contrastive Language-Image Pre-training (CLIP), an open-vocabulary foundation model, which achieves high accuracy across many image classification tasks and is often competitive with a fully supervised baseline without being explicitly trained. Nevertheless, there are still domains where zero-shot CLIP performance is far from optimal, such as Remote Sensing (RS) and medical imagery. These domains do not only exhibit fundamentally different distributions compared to natural images, but also commonly rely on complementary modalities, beyond RGB, to derive meaningful insights. To this end, we propose a methodology for the purpose of aligning distinct RS imagery modalities with the visual and textual modalities of CLIP. Our two-stage procedure, comprises of robust fine-tuning CLIP in order to deal with the distribution shift, accompanied by the cross-modal alignment of a RS modality encoder, in an effort to extend the zero-shot capabilities of CLIP. We ultimately demonstrate our method on the tasks of RS imagery classification and cross-modal retrieval. We empirically show that both robust fine-tuning and cross-modal alignment translate to significant performance gains, across several RS benchmark datasets. Notably, these enhancements are achieved without the reliance on textual descriptions, without introducing any task-specific parameters, without training from scratch and without catastrophic forgetting.

Title: Enhancing Cybersecurity Resilience in Finance with Deep Learning for Advanced Threat Detection

• Authors: Yulu Gong, Mengran Zhu, Shuning Huo, Yafei Xiang, Hanyi Yu
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09820
• Pdf URL: https://arxiv.org/pdf/2402.09820
• Copy Paste: [[2402.09820]] Enhancing Cybersecurity Resilience in Finance with Deep Learning for Advanced Threat Detection(https://arxiv.org/abs/2402.09820)
• Keywords: security, protect, defense, attack
• Abstract: In the age of the Internet, people's lives are increasingly dependent on today's network technology. However, network technology is a double-edged sword, bringing convenience to people but also posing many security challenges. Maintaining network security and protecting the legitimate interests of users is at the heart of network construction. Threat detection is an important part of a complete and effective defense system. In the field of network information security, the technical update of network attack and network protection is spiraling. How to effectively detect unknown threats is one of the concerns of network protection. Currently, network threat detection is usually based on rules and traditional machine learning methods, which create artificial rules or extract common spatiotemporal features, which cannot be applied to large-scale data applications, and the emergence of unknown threats causes the detection accuracy of the original model to decline. With this in mind, this paper uses deep learning for advanced threat detection to improve cybersecurity resilienc e in the financial industry. Many network security researchers have shifted their focus to exceptio n-based intrusion detection techniques. The detection technology mainly uses statistical machine learning methods - collecting normal program and network behavior data, extracting multidimensional features, and training decision machine learning models on this basis (commonly used include naive Bayes, decision trees, support vector machines, random forests, etc.). In the detection phase, program code or network behavior that deviates from the normal value beyond the tolerance is considered malicious code or network attack behavior.

Title: Utilizing GANs for Fraud Detection: Model Training with Synthetic Transaction Data

• Authors: Mengran Zhu, Yulu Gong, Yafei Xiang, Hanyi Yu, Shuning Huo
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2402.09830
• Pdf URL: https://arxiv.org/pdf/2402.09830
• Copy Paste: [[2402.09830]] Utilizing GANs for Fraud Detection: Model Training with Synthetic Transaction Data(https://arxiv.org/abs/2402.09830)
• Keywords: security, generative
• Abstract: Anomaly detection is a critical challenge across various research domains, aiming to identify instances that deviate from normal data distributions. This paper explores the application of Generative Adversarial Networks (GANs) in fraud detection, comparing their advantages with traditional methods. GANs, a type of Artificial Neural Network (ANN), have shown promise in modeling complex data distributions, making them effective tools for anomaly detection. The paper systematically describes the principles of GANs and their derivative models, emphasizing their application in fraud detection across different datasets. And by building a collection of adversarial verification graphs, we will effectively prevent fraud caused by bots or automated systems and ensure that the users in the transaction are real. The objective of the experiment is to design and implement a fake face verification code and fraud detection system based on Generative Adversarial network (GANs) algorithm to enhance the security of the transaction process.The study demonstrates the potential of GANs in enhancing transaction security through deep learning techniques.

Title: All in One and One for All: A Simple yet Effective Method towards Cross-domain Graph Pretraining

• Authors: Haihong Zhao, Aochuan Chen, Xiangguo Sun, Hong Cheng, Jia Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09834
• Pdf URL: https://arxiv.org/pdf/2402.09834
• Copy Paste: [[2402.09834]] All in One and One for All: A Simple yet Effective Method towards Cross-domain Graph Pretraining(https://arxiv.org/abs/2402.09834)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have revolutionized the fields of computer vision (CV) and natural language processing (NLP). One of the most notable advancements of LLMs is that a single model is trained on vast and diverse datasets spanning multiple domains -- a paradigm we term `All in One'. This methodology empowers LLMs with super generalization capabilities, facilitating an encompassing comprehension of varied data distributions. Leveraging these capabilities, a single LLM demonstrates remarkable versatility across a variety of domains -- a paradigm we term `One for All'. However, applying this idea to the graph field remains a formidable challenge, with cross-domain pretraining often resulting in negative transfer. This issue is particularly important in few-shot learning scenarios, where the paucity of training data necessitates the incorporation of external knowledge sources. In response to this challenge, we propose a novel approach called Graph COordinators for PrEtraining (GCOPE), that harnesses the underlying commonalities across diverse graph datasets to enhance few-shot learning. Our novel methodology involves a unification framework that amalgamates disparate graph datasets during the pretraining phase to distill and transfer meaningful knowledge to target tasks. Extensive experiments across multiple graph datasets demonstrate the superior efficacy of our approach. By successfully leveraging the synergistic potential of multiple graph datasets for pretraining, our work stands as a pioneering contribution to the realm of graph foundational model.

Title: LAPDoc: Layout-Aware Prompting for Documents

• Authors: Marcel Lamott, Yves-Noel Weweler, Adrian Ulges, Faisal Shafait, Dirk Krechel, Darko Obradovic
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09841
• Pdf URL: https://arxiv.org/pdf/2402.09841
• Copy Paste: [[2402.09841]] LAPDoc: Layout-Aware Prompting for Documents(https://arxiv.org/abs/2402.09841)
• Keywords: transformer, large language model
• Abstract: Recent advances in training large language models (LLMs) using massive amounts of solely textual data lead to strong generalization across many domains and tasks, including document-specific tasks. Opposed to that there is a trend to train multi-modal transformer architectures tailored for document understanding that are designed specifically to fuse textual inputs with the corresponding document layout. This involves a separate fine-tuning step for which additional training data is required. At present, no document transformers with comparable generalization to LLMs are available That raises the question which type of model is to be preferred for document understanding tasks. In this paper we investigate the possibility to use purely text-based LLMs for document-specific tasks by using layout enrichment. We explore drop-in modifications and rule-based methods to enrich purely textual LLM prompts with layout information. In our experiments we investigate the effects on the commercial ChatGPT model and the open-source LLM Solar. We demonstrate that using our approach both LLMs show improved performance on various standard document benchmarks. In addition, we study the impact of noisy OCR and layout errors, as well as the limitations of LLMs when it comes to utilizing document layout. Our results indicate that layout enrichment can improve the performance of purely text-based LLMs for document understanding by up to 15% compared to just using plain document text. In conclusion, this approach should be considered for the best model choice between text-based LLM or multi-modal document transformers.

Title: JustSTART: How to Find an RSA Authentication Bypass on Xilinx UltraScale(+) with Fuzzing

• Authors: Maik Ender, Felix Hahn, Marc Fyrbiak, Amir Moradi, Christof Paar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.09845
• Pdf URL: https://arxiv.org/pdf/2402.09845
• Copy Paste: [[2402.09845]] JustSTART: How to Find an RSA Authentication Bypass on Xilinx UltraScale(+) with Fuzzing(https://arxiv.org/abs/2402.09845)
• Keywords: secure, security, attack
• Abstract: Fuzzing is a well-established technique in the software domain to uncover bugs and vulnerabilities. Yet, applications of fuzzing for security vulnerabilities in hardware systems are scarce, as principal reasons are requirements for design information access (HDL source code). Moreover, observation of internal hardware state during runtime is typically an ineffective information source, as its documentation is often not publicly available. In addition, such observation during runtime is also inefficient due to bandwidth-limited analysis interfaces (JTAG, and minimal introspection of internal modules). In this work, we investigate fuzzing for 7-Series and UltraScale(+) FPGA configuration engines, the control plane governing the (secure) bitstream configuration within the FPGA. Our goal is to examine the effectiveness of fuzzing to analyze and document the opaque inner workings of FPGA configuration engines, with a primary emphasis on identifying security vulnerabilities. Using only the publicly available chip and dispersed documentation, we first design and implement ConFuzz, an advanced FPGA configuration engine fuzzing and rapid prototyping framework. Based on our detailed understanding of the bitstream file format, we then systematically define 3 novel key fuzzing strategies for Xilinx configuration engines. Moreover, our strategies are executed through mutational structure-aware fuzzers and incorporate various novel custom-tailored, FPGA-specific optimizations. Our evaluation reveals previously undocumented behavior within the configuration engine, including critical findings such as system crashes leading to unresponsive states of the FPGA. In addition, our investigations not only lead to the rediscovery of the starbleed attack but also uncover JustSTART (CVE-2023-20570), capable of circumventing RSA authentication for Xilinx UltraScale(+). Note that we also discuss countermeasures.

Title: Beyond Kalman Filters: Deep Learning-Based Filters for Improved Object Tracking

• Authors: Momir Adžemović, Predrag Tadić, Andrija Petrović, Mladen Nikolić
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09865
• Pdf URL: https://arxiv.org/pdf/2402.09865
• Copy Paste: [[2402.09865]] Beyond Kalman Filters: Deep Learning-Based Filters for Improved Object Tracking(https://arxiv.org/abs/2402.09865)
• Keywords: robust
• Abstract: Traditional tracking-by-detection systems typically employ Kalman filters (KF) for state estimation. However, the KF requires domain-specific design choices and it is ill-suited to handling non-linear motion patterns. To address these limitations, we propose two innovative data-driven filtering methods. Our first method employs a Bayesian filter with a trainable motion model to predict an object's future location and combines its predictions with observations gained from an object detector to enhance bounding box prediction accuracy. Moreover, it dispenses with most domain-specific design choices characteristic of the KF. The second method, an end-to-end trainable filter, goes a step further by learning to correct detector errors, further minimizing the need for domain expertise. Additionally, we introduce a range of motion model architectures based on Recurrent Neural Networks, Neural Ordinary Differential Equations, and Conditional Neural Processes, that are combined with the proposed filtering methods. Our extensive evaluation across multiple datasets demonstrates that our proposed filters outperform the traditional KF in object tracking, especially in the case of non-linear motion patterns -- the use case our filters are best suited to. We also conduct noise robustness analysis of our filters with convincing positive results. We further propose a new cost function for associating observations with tracks. Our tracker, which incorporates this new association cost with our proposed filters, outperforms the conventional SORT method and other motion-based trackers in multi-object tracking according to multiple metrics on motion-rich DanceTrack and SportsMOT datasets.

Title: Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community

• Authors: Arman Isajanyan, Artur Shatveryan, David Kocharyan, Zhangyang Wang, Humphrey Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09872
• Pdf URL: https://arxiv.org/pdf/2402.09872
• Copy Paste: [[2402.09872]] Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community(https://arxiv.org/abs/2402.09872)
• Keywords: generative
• Abstract: Social reward as a form of community recognition provides a strong source of motivation for users of online platforms to engage and contribute with content. The recent progress of text-conditioned image synthesis has ushered in a collaborative era where AI empowers users to craft original visual artworks seeking community validation. Nevertheless, assessing these models in the context of collective community preference introduces distinct challenges. Existing evaluation methods predominantly center on limited size user studies guided by image quality and prompt alignment. This work pioneers a paradigm shift, unveiling Social Reward - an innovative reward modeling framework that leverages implicit feedback from social network users engaged in creative editing of generated images. We embark on an extensive journey of dataset curation and refinement, drawing from Picsart: an online visual creation and editing platform, yielding a first million-user-scale dataset of implicit human preferences for user-generated visual art named Picsart Image-Social. Our analysis exposes the shortcomings of current metrics in modeling community creative preference of text-to-image models' outputs, compelling us to introduce a novel predictive model explicitly tailored to address these limitations. Rigorous quantitative experiments and user study show that our Social Reward model aligns better with social popularity than existing metrics. Furthermore, we utilize Social Reward to fine-tune text-to-image models, yielding images that are more favored by not only Social Reward, but also other established metrics. These findings highlight the relevance and effectiveness of Social Reward in assessing community appreciation for AI-generated artworks, establishing a closer alignment with users' creative goals: creating popular visual art. Codes can be accessed at https://github.com/Picsart-AI-Research/Social-Reward

Title: Camouflage is all you need: Evaluating and Enhancing Language Model Robustness Against Camouflage Adversarial Attacks

• Authors: Álvaro Huertas-García, Alejandro Martín, Javier Huertas-Tato, David Camacho
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09874
• Pdf URL: https://arxiv.org/pdf/2402.09874
• Copy Paste: [[2402.09874]] Camouflage is all you need: Evaluating and Enhancing Language Model Robustness Against Camouflage Adversarial Attacks(https://arxiv.org/abs/2402.09874)
• Keywords: attack, robust, transformer
• Abstract: Adversarial attacks represent a substantial challenge in Natural Language Processing (NLP). This study undertakes a systematic exploration of this challenge in two distinct phases: vulnerability evaluation and resilience enhancement of Transformer-based models under adversarial attacks. In the evaluation phase, we assess the susceptibility of three Transformer configurations, encoder-decoder, encoder-only, and decoder-only setups, to adversarial attacks of escalating complexity across datasets containing offensive language and misinformation. Encoder-only models manifest a 14% and 21% performance drop in offensive language detection and misinformation detection tasks, respectively. Decoder-only models register a 16% decrease in both tasks, while encoder-decoder models exhibit a maximum performance drop of 14% and 26% in the respective tasks. The resilience-enhancement phase employs adversarial training, integrating pre-camouflaged and dynamically altered data. This approach effectively reduces the performance drop in encoder-only models to an average of 5% in offensive language detection and 2% in misinformation detection tasks. Decoder-only models, occasionally exceeding original performance, limit the performance drop to 7% and 2% in the respective tasks. Although not surpassing the original performance, Encoder-decoder models can reduce the drop to an average of 6% and 2% respectively. Results suggest a trade-off between performance and robustness, with some models maintaining similar performance while gaining robustness. Our study and adversarial training techniques have been incorporated into an open-source tool for generating camouflaged datasets. However, methodology effectiveness depends on the specific camouflage technique and data encountered, emphasizing the need for continued exploration.

Title: Explaining Kernel Clustering via Decision Trees

• Authors: Maximilian Fleissner, Leena Chennuru Vankadara, Debarghya Ghoshdastidar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09881
• Pdf URL: https://arxiv.org/pdf/2402.09881
• Copy Paste: [[2402.09881]] Explaining Kernel Clustering via Decision Trees(https://arxiv.org/abs/2402.09881)
• Keywords: interpretability
• Abstract: Despite the growing popularity of explainable and interpretable machine learning, there is still surprisingly limited work on inherently interpretable clustering methods. Recently, there has been a surge of interest in explaining the classic k-means algorithm, leading to efficient algorithms that approximate k-means clusters using axis-aligned decision trees. However, interpretable variants of k-means have limited applicability in practice, where more flexible clustering methods are often needed to obtain useful partitions of the data. In this work, we investigate interpretable kernel clustering, and propose algorithms that construct decision trees to approximate the partitions induced by kernel k-means, a nonlinear extension of k-means. We further build on previous work on explainable k-means and demonstrate how a suitable choice of features allows preserving interpretability without sacrificing approximation guarantees on the interpretable model.

Title: Lester: rotoscope animation through video object segmentation and tracking

• Authors: Ruben Tous
• Subjects: cs.CV, cs.AI, cs.GR, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.09883
• Pdf URL: https://arxiv.org/pdf/2402.09883
• Copy Paste: [[2402.09883]] Lester: rotoscope animation through video object segmentation and tracking(https://arxiv.org/abs/2402.09883)
• Keywords: diffusion, segmentation
• Abstract: This article introduces Lester, a novel method to automatically synthetise retro-style 2D animations from videos. The method approaches the challenge mainly as an object segmentation and tracking problem. Video frames are processed with the Segment Anything Model (SAM) and the resulting masks are tracked through subsequent frames with DeAOT, a method of hierarchical propagation for semi-supervised video object segmentation. The geometry of the masks' contours is simplified with the Douglas-Peucker algorithm. Finally, facial traits, pixelation and a basic shadow effect can be optionally added. The results show that the method exhibits an excellent temporal consistency and can correctly process videos with different poses and appearances, dynamic shots, partial shots and diverse backgrounds. The proposed method provides a more simple and deterministic approach than diffusion models based video-to-video translation pipelines, which suffer from temporal consistency problems and do not cope well with pixelated and schematic outputs. The method is also much most practical than techniques based on 3D human pose estimation, which require custom handcrafted 3D models and are very limited with respect to the type of scenes they can process.

Title: COVIDHealth: A Benchmark Twitter Dataset and Machine Learning based Web Application for Classifying COVID-19 Discussions

• Authors: Mahathir Mohammad Bishal, Md. Rakibul Hassan Chowdory, Anik Das, Muhammad Ashad Kabir
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09897
• Pdf URL: https://arxiv.org/pdf/2402.09897
• Copy Paste: [[2402.09897]] COVIDHealth: A Benchmark Twitter Dataset and Machine Learning based Web Application for Classifying COVID-19 Discussions(https://arxiv.org/abs/2402.09897)
• Keywords: extraction
• Abstract: The COVID-19 pandemic has had adverse effects on both physical and mental health. During this pandemic, numerous studies have focused on gaining insights into health-related perspectives from social media. In this study, our primary objective is to develop a machine learning-based web application for automatically classifying COVID-19-related discussions on social media. To achieve this, we label COVID-19-related Twitter data, provide benchmark classification results, and develop a web application. We collected data using the Twitter API and labeled a total of 6,667 tweets into five different classes: health risks, prevention, symptoms, transmission, and treatment. We extracted features using various feature extraction methods and applied them to seven different traditional machine learning algorithms, including Decision Tree, Random Forest, Stochastic Gradient Descent, Adaboost, K-Nearest Neighbour, Logistic Regression, and Linear SVC. Additionally, we used four deep learning algorithms: LSTM, CNN, RNN, and BERT, for classification. Overall, we achieved a maximum F1 score of 90.43% with the CNN algorithm in deep learning. The Linear SVC algorithm exhibited the highest F1 score at 86.13%, surpassing other traditional machine learning approaches. Our study not only contributes to the field of health-related data analysis but also provides a valuable resource in the form of a web-based tool for efficient data classification, which can aid in addressing public health challenges and increasing awareness during pandemics. We made the dataset and application publicly available, which can be downloaded from this link https://github.com/Bishal16/COVID19-Health-Related-Data-Classification-Website.

Title: Revisiting Recurrent Reinforcement Learning with Memory Monoids

• Authors: Steven Morad, Chris Lu, Ryan Kortvelesy, Stephan Liwicki, Jakob Foerster, Amanda Prorok
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09900
• Pdf URL: https://arxiv.org/pdf/2402.09900
• Copy Paste: [[2402.09900]] Revisiting Recurrent Reinforcement Learning with Memory Monoids(https://arxiv.org/abs/2402.09900)
• Keywords: transformer
• Abstract: In RL, memory models such as RNNs and transformers address Partially Observable Markov Decision Processes (POMDPs) by mapping trajectories to latent Markov states. Neither model scales particularly well to long sequences, especially compared to an emerging class of memory models sometimes called linear recurrent models. We discover that the recurrent update of these models is a monoid, leading us to formally define a novel memory monoid framework. We revisit the traditional approach to batching in recurrent RL, highlighting both theoretical and empirical deficiencies. Leveraging the properties of memory monoids, we propose a new batching method that improves sample efficiency, increases the return, and simplifies the implementation of recurrent loss functions in RL.

Title: Generative Representational Instruction Tuning

• Authors: Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09906
• Pdf URL: https://arxiv.org/pdf/2402.09906
• Copy Paste: [[2402.09906]] Generative Representational Instruction Tuning(https://arxiv.org/abs/2402.09906)
• Keywords: generative, large language model
• Abstract: All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8x7B outperforms all open generative language models that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.

Title: Enhancing Large Language Models with Pseudo- and Multisource- Knowledge Graphs for Open-ended Question Answering

• Authors: Jiaxiang Liu, Tong Zhou, Yubo Chen, Kang Liu, Jun Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09911
• Pdf URL: https://arxiv.org/pdf/2402.09911
• Copy Paste: [[2402.09911]] Enhancing Large Language Models with Pseudo- and Multisource- Knowledge Graphs for Open-ended Question Answering(https://arxiv.org/abs/2402.09911)
• Keywords: large language model
• Abstract: Mitigating the hallucinations of Large Language Models (LLMs) and enhancing them is a crucial task. Although some existing methods employ model self-enhancement techniques, they fall short of effectively addressing unknown factual hallucinations. Using Knowledge Graph (KG) enhancement approaches fails to address the generalization across different KG sources and the enhancement of open-ended answer questions simultaneously. To tackle these limitations, there is a framework that combines Pseudo-Graph Generation and Atomic Knowledge Verification proposed. The enhancement of LLM using KG in an open-ended question-answering setting is implemented by leveraging the Pseudo-Graph Generation. Atomic Knowledge Verification utilizes atomic-level knowledge querying and verification to achieve generalizability under different KG sources. Compared to the baseline, this approach yields a minimum improvement of 11.5 in the ROUGE-L score for open-ended questions. For precise questions, we observe a minimum accuracy improvement of 7.5. Moreover, there is also demonstration that this framework exhibits generalizability across different KG sources. In summary, our results pave the way for enhancing LLMs by incorporating Pseudo- and Multisource-KGs, particularly in the context of open-ended questions.

Title: A Dataset of Open-Domain Question Answering with Multiple-Span Answers

• Authors: Zhiyi Luo, Yingying Zhang, Shuyun Luo, Ying Zhao, Wentao Lyu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.09923
• Pdf URL: https://arxiv.org/pdf/2402.09923
• Copy Paste: [[2402.09923]] A Dataset of Open-Domain Question Answering with Multiple-Span Answers(https://arxiv.org/abs/2402.09923)
• Keywords: extraction
• Abstract: Multi-span answer extraction, also known as the task of multi-span question answering (MSQA), is critical for real-world applications, as it requires extracting multiple pieces of information from a text to answer complex questions. Despite the active studies and rapid progress in English MSQA research, there is a notable lack of publicly available MSQA benchmark in Chinese. Previous efforts for constructing MSQA datasets predominantly emphasized entity-centric contextualization, resulting in a bias towards collecting factoid questions and potentially overlooking questions requiring more detailed descriptive responses. To overcome these limitations, we present CLEAN, a comprehensive Chinese multi-span question answering dataset that involves a wide range of open-domain subjects with a substantial number of instances requiring descriptive answers. Additionally, we provide established models from relevant literature as baselines for CLEAN. Experimental results and analysis show the characteristics and challenge of the newly proposed CLEAN dataset for the community. Our dataset, CLEAN, will be publicly released at zhiyiluo.site/misc/clean_v1.0_ sample.json.

Title: FedLion: Faster Adaptive Federated Optimization with Fewer Communication

• Authors: Zhiwei Tang, Tsung-Hui Chang
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.09941
• Pdf URL: https://arxiv.org/pdf/2402.09941
• Copy Paste: [[2402.09941]] FedLion: Faster Adaptive Federated Optimization with Fewer Communication(https://arxiv.org/abs/2402.09941)
• Keywords: federate
• Abstract: In Federated Learning (FL), a framework to train machine learning models across distributed data, well-known algorithms like FedAvg tend to have slow convergence rates, resulting in high communication costs during training. To address this challenge, we introduce FedLion, an adaptive federated optimization algorithm that seamlessly incorporates key elements from the recently proposed centralized adaptive algorithm, Lion (Chen et al. 2o23), into the FL framework. Through comprehensive evaluations on two widely adopted FL benchmarks, we demonstrate that FedLion outperforms previous state-of-the-art adaptive algorithms, including FAFED (Wu et al. 2023) and FedDA. Moreover, thanks to the use of signed gradients in local training, FedLion substantially reduces data transmission requirements during uplink communication when compared to existing adaptive algorithms, further reducing communication costs. Last but not least, this work also includes a novel theoretical analysis, showcasing that FedLion attains faster convergence rate than established FL algorithms like FedAvg.

Title: Loopy-SLAM: Dense Neural SLAM with Loop Closures

• Authors: Lorenzo Liso, Erik Sandström, Vladimir Yugay, Luc Van Gool, Martin R. Oswald
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09944
• Pdf URL: https://arxiv.org/pdf/2402.09944
• Copy Paste: [[2402.09944]] Loopy-SLAM: Dense Neural SLAM with Loop Closures(https://arxiv.org/abs/2402.09944)
• Keywords: robust
• Abstract: Neural RGBD SLAM techniques have shown promise in dense Simultaneous Localization And Mapping (SLAM), yet face challenges such as error accumulation during camera tracking resulting in distorted maps. In response, we introduce Loopy-SLAM that globally optimizes poses and the dense 3D model. We use frame-to-model tracking using a data-driven point-based submap generation method and trigger loop closures online by performing global place recognition. Robust pose graph optimization is used to rigidly align the local submaps. As our representation is point based, map corrections can be performed efficiently without the need to store the entire history of input frames used for mapping as typically required by methods employing a grid based mapping structure. Evaluation on the synthetic Replica and real-world TUM-RGBD and ScanNet datasets demonstrate competitive or superior performance in tracking, mapping, and rendering accuracy when compared to existing dense neural RGBD SLAM methods. Project page: notchla.github.io/Loopy-SLAM.

Title: Multi-Word Tokenization for Sequence Compression

• Authors: Leonidas Gee, Leonardo Rigutini, Marco Ernandes, Andrea Zugarini
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09949
• Pdf URL: https://arxiv.org/pdf/2402.09949
• Copy Paste: [[2402.09949]] Multi-Word Tokenization for Sequence Compression(https://arxiv.org/abs/2402.09949)
• Keywords: robust, large language model
• Abstract: Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this pa005 per, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length and budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation.

Title: Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of In-Context Learning for Persona-based Dialogue Generation

• Authors: Jiashu Pu, Yajing Wan, Yuru Zhang, Jing Chen, Ling Cheng, Qian Shao, Yongzhu Chang, Tangjie Lv, Rongsheng Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09954
• Pdf URL: https://arxiv.org/pdf/2402.09954
• Copy Paste: [[2402.09954]] Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of In-Context Learning for Persona-based Dialogue Generation(https://arxiv.org/abs/2402.09954)
• Keywords: large language model
• Abstract: Previous in-context learning (ICL) research has focused on tasks such as classification, machine translation, text2table, etc., while studies on whether ICL can improve human-like dialogue generation are scarce. Our work fills this gap by systematically investigating the ICL capabilities of large language models (LLMs) in persona-based dialogue generation, conducting extensive experiments on high-quality real human Chinese dialogue datasets. From experimental results, we draw three conclusions: 1) adjusting prompt instructions is the most direct, effective, and economical way to improve generation quality; 2) randomly retrieving demonstrations (demos) achieves the best results, possibly due to the greater diversity and the amount of effective information; counter-intuitively, retrieving demos with a context identical to the query performs the worst; 3) even when we destroy the multi-turn associations and single-turn semantics in the demos, increasing the number of demos still improves dialogue performance, proving that LLMs can learn from corrupted dialogue demos. Previous explanations of the ICL mechanism, such as $n$-gram induction head, cannot fully account for this phenomenon.

Title: On Designing Features for Condition Monitoring of Rotating Machines

• Authors: Seetaram Maurya, Nishchal K. Verma
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09957
• Pdf URL: https://arxiv.org/pdf/2402.09957
• Copy Paste: [[2402.09957]] On Designing Features for Condition Monitoring of Rotating Machines(https://arxiv.org/abs/2402.09957)
• Keywords: extraction
• Abstract: Various methods for designing input features have been proposed for fault recognition in rotating machines using one-dimensional raw sensor data. The available methods are complex, rely on empirical approaches, and may differ depending on the condition monitoring data used. Therefore, this article proposes a novel algorithm to design input features that unifies the feature extraction process for different time-series sensor data. This new insight for designing/extracting input features is obtained through the lens of histogram theory. The proposed algorithm extracts discriminative input features, which are suitable for a simple classifier to deep neural network-based classifiers. The designed input features are given as input to the classifier with end-to-end training in a single framework for machine conditions recognition. The proposed scheme has been validated through three real-time datasets: a) acoustic dataset, b) CWRU vibration dataset, and c) IMS vibration dataset. The real-time results and comparative study show the effectiveness of the proposed scheme for the prediction of the machine's health states.

Title: ViGEO: an Assessment of Vision GNNs in Earth Observation

• Authors: Luca Colomba, Paolo Garza
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09962
• Pdf URL: https://arxiv.org/pdf/2402.09962
• Copy Paste: [[2402.09962]] ViGEO: an Assessment of Vision GNNs in Earth Observation(https://arxiv.org/abs/2402.09962)
• Keywords: transformer
• Abstract: Satellite missions and Earth Observation (EO) systems represent fundamental assets for environmental monitoring and the timely identification of catastrophic events, long-term monitoring of both natural resources and human-made assets, such as vegetation, water bodies, forests as well as buildings. Different EO missions enables the collection of information on several spectral bandwidths, such as MODIS, Sentinel-1 and Sentinel-2. Thus, given the recent advances of machine learning, computer vision and the availability of labeled data, researchers demonstrated the feasibility and the precision of land-use monitoring systems and remote sensing image classification through the use of deep neural networks. Such systems may help domain experts and governments in constant environmental monitoring, enabling timely intervention in case of catastrophic events (e.g., forest wildfire in a remote area). Despite the recent advances in the field of computer vision, many works limit their analysis on Convolutional Neural Networks (CNNs) and, more recently, to vision transformers (ViTs). Given the recent successes of Graph Neural Networks (GNNs) on non-graph data, such as time-series and images, we investigate the performances of a recent Vision GNN architecture (ViG) applied to the task of land cover classification. The experimental results show that ViG achieves state-of-the-art performances in multiclass and multilabel classification contexts, surpassing both ViT and ResNet on large-scale benchmarks.

Title: Why are Sensitive Functions Hard for Transformers?

• Authors: Michael Hahn, Mark Rofin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09963
• Pdf URL: https://arxiv.org/pdf/2402.09963
• Copy Paste: [[2402.09963]] Why are Sensitive Functions Hard for Transformers?(https://arxiv.org/abs/2402.09963)
• Keywords: transformer
• Abstract: Empirical studies have identified a range of learnability biases and limitations of transformers, such as a persistent difficulty in learning to compute simple formal languages such as PARITY, and a bias towards low-degree functions. However, theoretical understanding remains limited, with existing expressiveness theory either overpredicting or underpredicting realistic learning abilities. We prove that, under the transformer architecture, the loss landscape is constrained by the input-space sensitivity: Transformers whose output is sensitive to many parts of the input string inhabit isolated points in parameter space, leading to a low-sensitivity bias in generalization. We show theoretically and empirically that this theory unifies a broad array of empirical observations about the learning abilities and biases of transformers, such as their generalization bias towards low sensitivity and low degree, and difficulty in length generalization for PARITY. This shows that understanding transformers' inductive biases requires studying not just their in-principle expressivity, but also their loss landscape.

Title: Textual Localization: Decomposing Multi-concept Images for Subject-Driven Text-to-Image Generation

• Authors: Junjie Shentu, Matthew Watson, Noura Al Moubayed
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.09966
• Pdf URL: https://arxiv.org/pdf/2402.09966
• Copy Paste: [[2402.09966]] Textual Localization: Decomposing Multi-concept Images for Subject-Driven Text-to-Image Generation(https://arxiv.org/abs/2402.09966)
• Keywords: diffusion
• Abstract: Subject-driven text-to-image diffusion models empower users to tailor the model to new concepts absent in the pre-training dataset using a few sample images. However, prevalent subject-driven models primarily rely on single-concept input images, facing challenges in specifying the target concept when dealing with multi-concept input images. To this end, we introduce a textual localized text-to-image model (Texual Localization) to handle multi-concept input images. During fine-tuning, our method incorporates a novel cross-attention guidance to decompose multiple concepts, establishing distinct connections between the visual representation of the target concept and the identifier token in the text prompt. Experimental results reveal that our method outperforms or performs comparably to the baseline models in terms of image fidelity and image-text alignment on multi-concept input images. In comparison to Custom Diffusion, our method with hard guidance achieves CLIP-I scores that are 7.04%, 8.13% higher and CLIP-T scores that are 2.22%, 5.85% higher in single-concept and multi-concept generation, respectively. Notably, our method generates cross-attention maps consistent with the target concept in the generated images, a capability absent in existing models.

Title: Case Study: Testing Model Capabilities in Some Reasoning Tasks

• Authors: Min Zhang, Sato Takumi, Jack Zhang, Jun Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09967
• Pdf URL: https://arxiv.org/pdf/2402.09967
• Copy Paste: [[2402.09967]] Case Study: Testing Model Capabilities in Some Reasoning Tasks(https://arxiv.org/abs/2402.09967)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) excel in generating personalized content and facilitating interactive dialogues, showcasing their remarkable aptitude for a myriad of applications. However, their capabilities in reasoning and providing explainable outputs, especially within the context of reasoning abilities, remain areas for improvement. In this study, we delve into the reasoning abilities of LLMs, highlighting the current challenges and limitations that hinder their effectiveness in complex reasoning scenarios.

Title: Accelerating Parallel Sampling of Diffusion Models

• Authors: Zhiwei Tang, Jiasheng Tang, Hao Luo, Fan Wang, Tsung-Hui Chang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.09970
• Pdf URL: https://arxiv.org/pdf/2402.09970
• Copy Paste: [[2402.09970]] Accelerating Parallel Sampling of Diffusion Models(https://arxiv.org/abs/2402.09970)
• Keywords: diffusion, generative
• Abstract: Diffusion models have emerged as state-of-the-art generative models for image generation. However, sampling from diffusion models is usually time-consuming due to the inherent autoregressive nature of their sampling process. In this work, we propose a novel approach that accelerates the sampling of diffusion models by parallelizing the autoregressive process. Specifically, we reformulate the sampling process as solving a system of triangular nonlinear equations through fixed-point iteration. With this innovative formulation, we explore several systematic techniques to further reduce the iteration steps required by the solving process. Applying these techniques, we introduce ParaTAA, a universal and training-free parallel sampling algorithm that can leverage extra computational and memory resources to increase the sampling speed. Our experiments demonstrate that ParaTAA can decrease the inference steps required by common sequential sampling algorithms such as DDIM and DDPM by a factor of 4~14 times. Notably, when applying ParaTAA with 100 steps DDIM for Stable Diffusion, a widely-used text-to-image diffusion model, it can produce the same images as the sequential sampling in only 7 inference steps.

Title: TSTEM: A Cognitive Platform for Collecting Cyber Threat Intelligence in the Wild

• Authors: Prasasthy Balasubramanian, Sadaf Nazari, Danial Khosh Kholgh, Alireza Mahmoodi, Justin Seby, Panos Kostakos
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.09973
• Pdf URL: https://arxiv.org/pdf/2402.09973
• Copy Paste: [[2402.09973]] TSTEM: A Cognitive Platform for Collecting Cyber Threat Intelligence in the Wild(https://arxiv.org/abs/2402.09973)
• Keywords: attack, extraction
• Abstract: The extraction of cyber threat intelligence (CTI) from open sources is a rapidly expanding defensive strategy that enhances the resilience of both Information Technology (IT) and Operational Technology (OT) environments against large-scale cyber-attacks. While previous research has focused on improving individual components of the extraction process, the community lacks open-source platforms for deploying streaming CTI data pipelines in the wild. To address this gap, the study describes the implementation of an efficient and well-performing platform capable of processing compute-intensive data pipelines based on the cloud computing paradigm for real-time detection, collecting, and sharing CTI from different online sources. We developed a prototype platform (TSTEM), a containerized microservice architecture that uses Tweepy, Scrapy, Terraform, ELK, Kafka, and MLOps to autonomously search, extract, and index IOCs in the wild. Moreover, the provisioning, monitoring, and management of the TSTEM platform are achieved through infrastructure as a code (IaC). Custom focus crawlers collect web content, which is then processed by a first-level classifier to identify potential indicators of compromise (IOCs). If deemed relevant, the content advances to a second level of extraction for further examination. Throughout this process, state-of-the-art NLP models are utilized for classification and entity extraction, enhancing the overall IOC extraction methodology. Our experimental results indicate that these models exhibit high accuracy (exceeding 98%) in the classification and extraction tasks, achieving this performance within a time frame of less than a minute. The effectiveness of our system can be attributed to a finely-tuned IOC extraction method that operates at multiple stages, ensuring precise identification of relevant information with low false positives.

Title: LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition

• Authors: Jinyuan Li, Han Li, Di Sun, Jiahao Wang, Wenkun Zhang, Zan Wang, Gang Pan
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.09989
• Pdf URL: https://arxiv.org/pdf/2402.09989
• Copy Paste: [[2402.09989]] LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition(https://arxiv.org/abs/2402.09989)
• Keywords: large language model
• Abstract: Grounded Multimodal Named Entity Recognition (GMNER) is a nascent multimodal task that aims to identify named entities, entity types and their corresponding visual regions. GMNER task exhibits two challenging properties: 1) The weak correlation between image-text pairs in social media results in a significant portion of named entities being ungroundable. 2) There exists a distinction between coarse-grained referring expressions commonly used in similar tasks (e.g., phrase localization, referring expression comprehension) and fine-grained named entities. In this paper, we propose RiVEG, a unified framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models (LLMs) as a connecting bridge. This reformulation brings two benefits: 1) It maintains the optimal MNER performance and eliminates the need for employing object detection methods to pre-extract regional features, thereby naturally addressing two major limitations of existing GMNER methods. 2) The introduction of entity expansion expression and Visual Entailment (VE) Module unifies Visual Grounding (VG) and Entity Grounding (EG). It enables RiVEG to effortlessly inherit the Visual Entailment and Visual Grounding capabilities of any current or prospective multimodal pretraining models. Extensive experiments demonstrate that RiVEG outperforms state-of-the-art methods on the existing GMNER dataset and achieves absolute leads of 10.65%, 6.21%, and 8.83% in all three subtasks.

Title: TIAViz: A Browser-based Visualization Tool for Computational Pathology Models

• Authors: Mark Eastwood, John Pocock, Mostafa Jahanifar, Adam Shephard, Skiros Habib, Ethar Alzaid, Abdullah Alsalemi, Jan Lukas Robertus, Nasir Rajpoot, Shan Raza, Fayyaz Minhas
• Subjects: cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.09990
• Pdf URL: https://arxiv.org/pdf/2402.09990
• Copy Paste: [[2402.09990]] TIAViz: A Browser-based Visualization Tool for Computational Pathology Models(https://arxiv.org/abs/2402.09990)
• Keywords: segmentation
• Abstract: Digital pathology has gained significant traction in modern healthcare systems. This shift from optical microscopes to digital imagery brings with it the potential for improved diagnosis, efficiency, and the integration of AI tools into the pathologists workflow. A critical aspect of this is visualization. Throughout the development of a machine learning (ML) model in digital pathology, it is crucial to have flexible, openly available tools to visualize models, from their outputs and predictions to the underlying annotations and images used to train or test a model. We introduce TIAViz, a Python-based visualization tool built into TIAToolbox which allows flexible, interactive, fully zoomable overlay of a wide variety of information onto whole slide images, including graphs, heatmaps, segmentations, annotations and other WSIs. The UI is browser-based, allowing use either locally, on a remote machine, or on a server to provide publicly available demos. This tool is open source and is made available at: https://github.com/TissueImageAnalytics/tiatoolbox and via pip installation (pip install tiatoolbox) and conda as part of TIAToolbox.

Title: Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts

• Authors: Tobias Enders, James Harrison, Maximilian Schiffer
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.09992
• Pdf URL: https://arxiv.org/pdf/2402.09992
• Copy Paste: [[2402.09992]] Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts(https://arxiv.org/abs/2402.09992)
• Keywords: robust
• Abstract: We study the robustness of deep reinforcement learning algorithms against distribution shifts within contextual multi-stage stochastic combinatorial optimization problems from the operations research domain. In this context, risk-sensitive algorithms promise to learn robust policies. While this field is of general interest to the reinforcement learning community, most studies up-to-date focus on theoretical results rather than real-world performance. With this work, we aim to bridge this gap by formally deriving a novel risk-sensitive deep reinforcement learning algorithm while providing numerical evidence for its efficacy. Specifically, we introduce discrete Soft Actor-Critic for the entropic risk measure by deriving a version of the Bellman equation for the respective Q-values. We establish a corresponding policy improvement result and infer a practical algorithm. We introduce an environment that represents typical contextual multi-stage stochastic combinatorial optimization problems and perform numerical experiments to empirically validate our algorithm's robustness against realistic distribution shifts, without compromising performance on the training distribution. We show that our algorithm is superior to risk-neutral Soft Actor-Critic as well as to two benchmark approaches for robust deep reinforcement learning. Thereby, we provide the first structured analysis on the robustness of reinforcement learning under distribution shifts in the realm of contextual multi-stage stochastic combinatorial optimization problems.

Title: Privacy Attacks in Decentralized Learning

• Authors: Abdellah El Mrini, Edwige Cyffers, Aurélien Bellet
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.10001
• Pdf URL: https://arxiv.org/pdf/2402.10001
• Copy Paste: [[2402.10001]] Privacy Attacks in Decentralized Learning(https://arxiv.org/abs/2402.10001)
• Keywords: privacy, attack
• Abstract: Decentralized Gradient Descent (D-GD) allows a set of users to perform collaborative learning without sharing their data by iteratively averaging local model updates with their neighbors in a network graph. The absence of direct communication between non-neighbor nodes might lead to the belief that users cannot infer precise information about the data of others. In this work, we demonstrate the opposite, by proposing the first attack against D-GD that enables a user (or set of users) to reconstruct the private data of other users outside their immediate neighborhood. Our approach is based on a reconstruction attack against the gossip averaging protocol, which we then extend to handle the additional challenges raised by D-GD. We validate the effectiveness of our attack on real graphs and datasets, showing that the number of users compromised by a single or a handful of attackers is often surprisingly large. We empirically investigate some of the factors that affect the performance of the attack, namely the graph topology, the number of attackers, and their position in the graph.

Title: MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding

• Authors: Hai-Tao Yu, Mofei Song
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.10002
• Pdf URL: https://arxiv.org/pdf/2402.10002
• Copy Paste: [[2402.10002]] MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding(https://arxiv.org/abs/2402.10002)
• Keywords: segmentation
• Abstract: In perception, multiple sensory information is integrated to map visual information from 2D views onto 3D objects, which is beneficial for understanding in 3D environments. But in terms of a single 2D view rendered from different angles, only limited partial information can be provided.The richness and value of Multi-view 2D information can provide superior self-supervised signals for 3D objects. In this paper, we propose a novel self-supervised point cloud representation learning method, MM-Point, which is driven by intra-modal and inter-modal similarity objectives. The core of MM-Point lies in the Multi-modal interaction and transmission between 3D objects and multiple 2D views at the same time. In order to more effectively simultaneously perform the consistent cross-modal objective of 2D multi-view information based on contrastive learning, we further propose Multi-MLP and Multi-level Augmentation strategies. Through carefully designed transformation strategies, we further learn Multi-level invariance in 2D Multi-views. MM-Point demonstrates state-of-the-art (SOTA) performance in various downstream tasks. For instance, it achieves a peak accuracy of 92.4% on the synthetic dataset ModelNet40, and a top accuracy of 87.8% on the real-world dataset ScanObjectNN, comparable to fully supervised methods. Additionally, we demonstrate its effectiveness in tasks such as few-shot classification, 3D part segmentation and 3D semantic segmentation.

Title: SAWEC: Sensing-Assisted Wireless Edge Computing

• Authors: Khandaker Foysal Haque, Francesca Meneghello, Md. Ebtidaul Karim, Francesco Restuccia
• Subjects: cs.CV, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.10021
• Pdf URL: https://arxiv.org/pdf/2402.10021
• Copy Paste: [[2402.10021]] SAWEC: Sensing-Assisted Wireless Edge Computing(https://arxiv.org/abs/2402.10021)
• Keywords: segmentation
• Abstract: Emerging mobile virtual reality (VR) systems will require to continuously perform complex computer vision tasks on ultra-high-resolution video frames through the execution of deep neural networks (DNNs)-based algorithms. Since state-of-the-art DNNs require computational power that is excessive for mobile devices, techniques based on wireless edge computing (WEC) have been recently proposed. However, existing WEC methods require the transmission and processing of a high amount of video data which may ultimately saturate the wireless link. In this paper, we propose a novel Sensing-Assisted Wireless Edge Computing (SAWEC) paradigm to address this issue. SAWEC leverages knowledge about the physical environment to reduce the end-to-end latency and overall computational burden by transmitting to the edge server only the relevant data for the delivery of the service. Our intuition is that the transmission of the portion of the video frames where there are no changes with respect to previous frames can be avoided. Specifically, we leverage wireless sensing techniques to estimate the location of objects in the environment and obtain insights about the environment dynamics. Hence, only the part of the frames where any environmental change is detected is transmitted and processed. We evaluated SAWEC by using a 10K 360$^{\circ}$ camera with a Wi-Fi 6 sensing system operating at 160 MHz and performing localization and tracking. We perform experiments in an anechoic chamber and a hall room with two human subjects in six different setups. Experimental results show that SAWEC reduces the channel occupation, and end-to-end latency by 93.81%, and 96.19% respectively while improving the instance segmentation performance by 46.98% with respect to state-of-the-art WEC approaches. For reproducibility purposes, we pledge to share our whole dataset and code repository.

Title: Self-Augmented In-Context Learning for Unsupervised Word Translation

• Authors: Yaoyiran Li, Anna Korhonen, Ivan Vulić
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10024
• Pdf URL: https://arxiv.org/pdf/2402.10024
• Copy Paste: [[2402.10024]] Self-Augmented In-Context Learning for Unsupervised Word Translation(https://arxiv.org/abs/2402.10024)
• Keywords: large language model
• Abstract: Recent work has shown that, while large language models (LLMs) demonstrate strong word translation or bilingual lexicon induction (BLI) capabilities in few-shot setups, they still cannot match the performance of 'traditional' mapping-based approaches in the unsupervised scenario where no seed translation pairs are available, especially for lower-resource languages. To address this challenge with LLMs, we propose self-augmented in-context learning (SAIL) for unsupervised BLI: starting from a zero-shot prompt, SAIL iteratively induces a set of high-confidence word translation pairs for in-context learning (ICL) from an LLM, which it then reapplies to the same LLM in the ICL fashion. Our method shows substantial gains over zero-shot prompting of LLMs on two established BLI benchmarks spanning a wide range of language pairs, also outperforming mapping-based baselines across the board. In addition to achieving state-of-the-art unsupervised BLI performance, we also conduct comprehensive analyses on SAIL and discuss its limitations.

Title: Diffusion Models Meet Contextual Bandits with Large Action Spaces

• Authors: Imad Aouali
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.10028
• Pdf URL: https://arxiv.org/pdf/2402.10028
• Copy Paste: [[2402.10028]] Diffusion Models Meet Contextual Bandits with Large Action Spaces(https://arxiv.org/abs/2402.10028)
• Keywords: diffusion
• Abstract: Efficient exploration is a key challenge in contextual bandits due to the large size of their action space, where uninformed exploration can result in computational and statistical inefficiencies. Fortunately, the rewards of actions are often correlated and this can be leveraged to explore them efficiently. In this work, we capture such correlations using pre-trained diffusion models; upon which we design diffusion Thompson sampling (dTS). Both theoretical and algorithmic foundations are developed for dTS, and empirical evaluation also shows its favorable performance.

Title: Systematic Literature Review of EM-SCA Attacks on Encryption

• Authors: Muhammad Rusyaidi Zunaidi, Asanka Sayakkara, Mark Scanlon
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.10030
• Pdf URL: https://arxiv.org/pdf/2402.10030
• Copy Paste: [[2402.10030]] Systematic Literature Review of EM-SCA Attacks on Encryption(https://arxiv.org/abs/2402.10030)
• Keywords: security, attack
• Abstract: Cryptography is vital for data security, but cryptographic algorithms can still be vulnerable to side-channel attacks (SCAs), physical assaults exploiting power consumption and EM radiation. SCAs pose a significant threat to cryptographic integrity, compromising device keys. While literature on SCAs focuses on real-world devices, the rise of sophisticated devices necessitates fresh approaches. Electromagnetic side-channel analysis (EM-SCA) gathers information by monitoring EM radiation, capable of retrieving encryption keys and detecting malicious activity. This study evaluates EM-SCA's impact on encryption across scenarios and explores its role in digital forensics and law enforcement. Addressing encryption susceptibility to EM-SCA can empower forensic investigators in overcoming encryption challenges, maintaining their crucial role in law enforcement. Additionally, the paper defines EM-SCA's current state in attacking encryption, highlighting vulnerable and resistant encryption algorithms and devices, and promising EM-SCA approaches. This study offers a comprehensive analysis of EM-SCA in law enforcement and digital forensics, suggesting avenues for further research.

Title: Investigation of Federated Learning Algorithms for Retinal Optical Coherence Tomography Image Classification with Statistical Heterogeneity

• Authors: Sanskar Amgain, Prashant Shrestha, Sophia Bano, Ignacio del Valle Torres, Michael Cunniffe, Victor Hernandez, Phil Beales, Binod Bhattarai
• Subjects: cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.10035
• Pdf URL: https://arxiv.org/pdf/2402.10035
• Copy Paste: [[2402.10035]] Investigation of Federated Learning Algorithms for Retinal Optical Coherence Tomography Image Classification with Statistical Heterogeneity(https://arxiv.org/abs/2402.10035)
• Keywords: privacy, robust, federate
• Abstract: Purpose: We apply federated learning to train an OCT image classifier simulating a realistic scenario with multiple clients and statistical heterogeneous data distribution where data in the clients lack samples of some categories entirely. Methods: We investigate the effectiveness of FedAvg and FedProx to train an OCT image classification model in a decentralized fashion, addressing privacy concerns associated with centralizing data. We partitioned a publicly available OCT dataset across multiple clients under IID and Non-IID settings and conducted local training on the subsets for each client. We evaluated two federated learning methods, FedAvg and FedProx for these settings. Results: Our experiments on the dataset suggest that under IID settings, both methods perform on par with training on a central data pool. However, the performance of both algorithms declines as we increase the statistical heterogeneity across the client data, while FedProx consistently performs better than FedAvg in the increased heterogeneity settings. Conclusion: Despite the effectiveness of federated learning in the utilization of private data across multiple medical institutions, the large number of clients and heterogeneous distribution of labels deteriorate the performance of both algorithms. Notably, FedProx appears to be more robust to the increased heterogeneity.

Title: RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models

• Authors: Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, Prathap Ramachandra
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10038
• Pdf URL: https://arxiv.org/pdf/2402.10038
• Copy Paste: [[2402.10038]] RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models(https://arxiv.org/abs/2402.10038)
• Keywords: large language model
• Abstract: Reinforcement learning from human feedback (RLHF) has been extensively employed to align large language models with user intent. However, proximal policy optimization (PPO) based RLHF is occasionally unstable requiring significant hyperparameter finetuning, and computationally expensive to maximize the estimated reward during alignment. Recently, direct preference optimization (DPO) is proposed to address those challenges. However, DPO relies on contrastive responses generated from human annotator and alternative LLM, instead of the policy model, limiting the effectiveness of the RLHF. In this paper, we addresses both challenges by systematically combining rejection sampling (RS) and DPO. Our proposed method, RS-DPO, initiates with the development of a supervised fine-tuned policy model (SFT). A varied set of k responses per prompt are sampled directly from the SFT model. RS-DPO identifies pairs of contrastive samples based on their reward distribution. Finally, we apply DPO with the contrastive samples to align the model to human preference. Our experiments indicate that our proposed method effectively fine-tunes LLMs with limited resource environments, leading to improved alignment with user intent. Furthermore, it outperforms existing methods, including RS, PPO, and DPO.

Title: Feature Accentuation: Revealing 'What' Features Respond to in Natural Images

• Authors: Chris Hamblin, Thomas Fel, Srijani Saha, Talia Konkle, George Alvarez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10039
• Pdf URL: https://arxiv.org/pdf/2402.10039
• Copy Paste: [[2402.10039]] Feature Accentuation: Revealing 'What' Features Respond to in Natural Images(https://arxiv.org/abs/2402.10039)
• Keywords: interpretability
• Abstract: Efforts to decode neural network vision models necessitate a comprehensive grasp of both the spatial and semantic facets governing feature responses within images. Most research has primarily centered around attribution methods, which provide explanations in the form of heatmaps, showing where the model directs its attention for a given feature. However, grasping 'where' alone falls short, as numerous studies have highlighted the limitations of those methods and the necessity to understand 'what' the model has recognized at the focal point of its attention. In parallel, 'Feature visualization' offers another avenue for interpreting neural network features. This approach synthesizes an optimal image through gradient ascent, providing clearer insights into 'what' features respond to. However, feature visualizations only provide one global explanation per feature; they do not explain why features activate for particular images. In this work, we introduce a new method to the interpretability tool-kit, 'feature accentuation', which is capable of conveying both where and what in arbitrary input images induces a feature's response. At its core, feature accentuation is image-seeded (rather than noise-seeded) feature visualization. We find a particular combination of parameterization, augmentation, and regularization yields naturalistic visualizations that resemble the seed image and target feature simultaneously. Furthermore, we validate these accentuations are processed along a natural circuit by the model. We make our precise implementation of feature accentuation available to the community as the Faccent library, an extension of Lucent.

Title: On the Domain Generalizability of RF Fingerprints Through Multifractal Dimension Representation

• Authors: Benjamin Johnson, Bechir Hamdaoui
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.10044
• Pdf URL: https://arxiv.org/pdf/2402.10044
• Copy Paste: [[2402.10044]] On the Domain Generalizability of RF Fingerprints Through Multifractal Dimension Representation(https://arxiv.org/abs/2402.10044)
• Keywords: secure, robust
• Abstract: RF data-driven device fingerprinting through the use of deep learning has recently surfaced as a possible method for enabling secure device identification and authentication. Traditional approaches are commonly susceptible to the domain adaptation problem where a model trained on data collected under one domain performs badly when tested on data collected under a different domain. Some examples of a domain change include varying the location or environment of the device and varying the time or day of the data collection. In this work, we propose using multifractal analysis and the variance fractal dimension trajectory (VFDT) as a data representation input to the deep neural network to extract device fingerprints that are domain generalizable. We analyze the effectiveness of the proposed VFDT representation in detecting device-specific signatures from hardware-impaired IQ (in-phase and quadrature) signals, and we evaluate its robustness in real-world settings, using an experimental testbed of 30 WiFi-enabled Pycom devices. Our experimental results show that the proposed VFDT representation improves the scalability, robustness and generalizability of the deep learning models significantly compared to when using IQ data samples.

Title: Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination

• Authors: Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, Ivan Vulić
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10052
• Pdf URL: https://arxiv.org/pdf/2402.10052
• Copy Paste: [[2402.10052]] Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination(https://arxiv.org/abs/2402.10052)
• Keywords: privacy, large language model
• Abstract: While displaying impressive generation capabilities across many tasks, Large Language Models (LLMs) still struggle with crucial issues of privacy violation and unwanted exposure of sensitive data. This raises an essential question: how should we prevent such undesired behavior of LLMs while maintaining their strong generation and natural language understanding (NLU) capabilities? In this work, we introduce a novel approach termed deliberate imagination in the context of LLM unlearning. Instead of trying to forget memorized data, we employ a self-distillation framework, guiding LLMs to deliberately imagine alternative scenarios. As demonstrated in a wide range of experiments, the proposed method not only effectively unlearns targeted text but also preserves the LLMs' capabilities in open-ended generation tasks as well as in NLU tasks. Our results demonstrate the usefulness of this approach across different models and sizes, and also with parameter-efficient fine-tuning, offering a novel pathway to addressing the challenges with private and sensitive data in LLM applications.

Title: Towards Safer Large Language Models through Machine Unlearning

• Authors: Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, Meng Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10058
• Pdf URL: https://arxiv.org/pdf/2402.10058
• Copy Paste: [[2402.10058]] Towards Safer Large Language Models through Machine Unlearning(https://arxiv.org/abs/2402.10058)
• Keywords: robust, large language model
• Abstract: The rapid advancement of Large Language Models (LLMs) has demonstrated their vast potential across various domains, attributed to their extensive pretraining knowledge and exceptional generalizability. However, LLMs often encounter challenges in generating harmful content when faced with problematic prompts. To address this problem, existing work attempted to implement a gradient ascent based approach to prevent LLMs from producing harmful output. While these methods can be effective, they frequently impact the model utility in responding to normal prompts. To address this gap, we introduce Selective Knowledge negation Unlearning (SKU), a novel unlearning framework for LLMs, designed to eliminate harmful knowledge while preserving utility on normal prompts. Specifically, SKU is consisted of two stages: harmful knowledge acquisition stage and knowledge negation stage. The first stage aims to identify and acquire harmful knowledge within the model, whereas the second is dedicated to remove this knowledge. SKU selectively isolates and removes harmful knowledge in model parameters, ensuring the model's performance remains robust on normal prompts. Our experiments conducted across various LLM architectures demonstrate that SKU identifies a good balance point between removing harmful information and preserving utility.

Title: How Much Does Each Datapoint Leak Your Privacy? Quantifying the Per-datum Membership Leakage

• Authors: Achraf Azize, Debabrota Basu
• Subjects: cs.LG, cs.CR, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.10065
• Pdf URL: https://arxiv.org/pdf/2402.10065
• Copy Paste: [[2402.10065]] How Much Does Each Datapoint Leak Your Privacy? Quantifying the Per-datum Membership Leakage(https://arxiv.org/abs/2402.10065)
• Keywords: privacy, attack, membership infer
• Abstract: We study the per-datum Membership Inference Attacks (MIAs), where an attacker aims to infer whether a fixed target datum has been included in the input dataset of an algorithm and thus, violates privacy. First, we define the membership leakage of a datum as the advantage of the optimal adversary targeting to identify it. Then, we quantify the per-datum membership leakage for the empirical mean, and show that it depends on the Mahalanobis distance between the target datum and the data-generating distribution. We further assess the effect of two privacy defences, i.e. adding Gaussian noise and sub-sampling. We quantify exactly how both of them decrease the per-datum membership leakage. Our analysis builds on a novel proof technique that combines an Edgeworth expansion of the likelihood ratio test and a Lindeberg-Feller central limit theorem. Our analysis connects the existing likelihood ratio and scalar product attacks, and also justifies different canary selection strategies used in the privacy auditing literature. Finally, our experiments demonstrate the impacts of the leakage score, the sub-sampling ratio and the noise scale on the per-datum membership leakage as indicated by the theory.

Title: NYCTALE: Neuro-Evidence Transformer for Adaptive and Personalized Lung Nodule Invasiveness Prediction

• Authors: Sadaf Khademi, Anastasia Oikonomou, Konstantinos N. Plataniotis, Arash Mohammadi
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.10066
• Pdf URL: https://arxiv.org/pdf/2402.10066
• Copy Paste: [[2402.10066]] NYCTALE: Neuro-Evidence Transformer for Adaptive and Personalized Lung Nodule Invasiveness Prediction(https://arxiv.org/abs/2402.10066)
• Keywords: transformer
• Abstract: Drawing inspiration from the primate brain's intriguing evidence accumulation process, and guided by models from cognitive psychology and neuroscience, the paper introduces the NYCTALE framework, a neuro-inspired and evidence accumulation-based Transformer architecture. The proposed neuro-inspired NYCTALE offers a novel pathway in the domain of Personalized Medicine (PM) for lung cancer diagnosis. In nature, Nyctales are small owls known for their nocturnal behavior, hunting primarily during the darkness of night. The NYCTALE operates in a similarly vigilant manner, i.e., processing data in an evidence-based fashion and making predictions dynamically/adaptively. Distinct from conventional Computed Tomography (CT)-based Deep Learning (DL) models, the NYCTALE performs predictions only when sufficient amount of evidence is accumulated. In other words, instead of processing all or a pre-defined subset of CT slices, for each person, slices are provided one at a time. The NYCTALE framework then computes an evidence vector associated with contribution of each new CT image. A decision is made once the total accumulated evidence surpasses a specific threshold. Preliminary experimental analyses conducted using a challenging in-house dataset comprising 114 subjects. The results are noteworthy, suggesting that NYCTALE outperforms the benchmark accuracy even with approximately 60% less training data on this demanding and small dataset.

Title: Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligence

• Authors: Weixiang Zhao, Zhuojun Li, Shilong Wang, Yang Wang, Yulin Hu, Yanyan Zhao, Chen Wei, Bing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10073
• Pdf URL: https://arxiv.org/pdf/2402.10073
• Copy Paste: [[2402.10073]] Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligence(https://arxiv.org/abs/2402.10073)
• Keywords: large language model
• Abstract: Emotional Intelligence (EI), consisting of emotion perception, emotion cognition and emotion expression, plays the critical roles in improving user interaction experience for the current large language model (LLM) based conversational general AI assistants. Previous works mainly focus on raising the emotion perception ability of them via naive fine-tuning on EI-related classification or regression tasks. However, this leads to the incomplete enhancement of EI and catastrophic forgetting of the general intelligence (GI). To this end, we first introduce \textsc{EiBench}, a large-scale collection of EI-related tasks in the text-to-text formation with task instructions that covers all three aspects of EI, which lays a solid foundation for the comprehensive EI enhancement of LLMs. Then a novel \underline{\textbf{Mo}}dular \underline{\textbf{E}}motional \underline{\textbf{I}}ntelligence enhancement method (\textbf{MoEI}), consisting of Modular Parameter Expansion and intra-inter modulation, is proposed to comprehensively enhance the EI of LLMs without compromise their GI. Extensive experiments on two representative LLM-based assistants, Flan-T5 and LLaMA-2-Chat, demonstrate the effectiveness of MoEI to improving EI while maintain GI.

Title: QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference

• Authors: Taesu Kim, Jongho Lee, Daehyun Ahn, Sarang Kim, Jiwoong Choi, Minkyu Kim, Hyungjun Kim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10076
• Pdf URL: https://arxiv.org/pdf/2402.10076
• Copy Paste: [[2402.10076]] QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference(https://arxiv.org/abs/2402.10076)
• Keywords: large language model
• Abstract: We introduce QUICK, a group of novel optimized CUDA kernels for the efficient inference of quantized Large Language Models (LLMs). QUICK addresses the shared memory bank-conflict problem of state-of-the-art mixed precision matrix multiplication kernels. Our method interleaves the quantized weight matrices of LLMs offline to skip the shared memory write-back after the dequantization. We demonstrate up to 1.91x speedup over existing kernels of AutoAWQ on larger batches and up to 1.94x throughput gain on representative LLM models on various NVIDIA GPU devices.

Title: Review of the Learning-based Camera and Lidar Simulation Methods for Autonomous Driving Systems

• Authors: Hamed Haghighi, Xiaomeng Wang, Hao Jing, Mehrdad Dianati
• Subjects: cs.CV, cs.GR, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.10079
• Pdf URL: https://arxiv.org/pdf/2402.10079
• Copy Paste: [[2402.10079]] Review of the Learning-based Camera and Lidar Simulation Methods for Autonomous Driving Systems(https://arxiv.org/abs/2402.10079)
• Keywords: generative
• Abstract: Perception sensors, particularly camera and Lidar, are key elements of Autonomous Driving Systems (ADS) that enable them to comprehend their surroundings for informed driving and control decisions. Therefore, developing realistic camera and Lidar simulation methods, also known as camera and Lidar models, is of paramount importance to effectively conduct simulation-based testing for ADS. Moreover, the rise of deep learning-based perception models has propelled the prevalence of perception sensor models as valuable tools for synthesising diverse training datasets. The traditional sensor simulation methods rely on computationally expensive physics-based algorithms, specifically in complex systems such as ADS. Hence, the current potential resides in learning-based models, driven by the success of deep generative models in synthesising high-dimensional data. This paper reviews the current state-of-the-art in learning-based sensor simulation methods and validation approaches, focusing on two main types of perception sensors: cameras and Lidars. This review covers two categories of learning-based approaches, namely raw-data-based and object-based models. Raw-data-based methods are explained concerning the employed learning strategy, while object-based models are categorised based on the type of error considered. Finally, the paper illustrates commonly used validation techniques for evaluating perception sensor models and highlights the existing research gaps in the area.

Title: FedRDF: A Robust and Dynamic Aggregation Function against Poisoning Attacks in Federated Learning

• Authors: Enrique Mármol Campos, Aurora González Vidal, José Luis Hernández Ramos, Antonio Skarmeta
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.10082
• Pdf URL: https://arxiv.org/pdf/2402.10082
• Copy Paste: [[2402.10082]] FedRDF: A Robust and Dynamic Aggregation Function against Poisoning Attacks in Federated Learning(https://arxiv.org/abs/2402.10082)
• Keywords: security, privacy, attack, robust, federate
• Abstract: Federated Learning (FL) represents a promising approach to typical privacy concerns associated with centralized Machine Learning (ML) deployments. Despite its well-known advantages, FL is vulnerable to security attacks such as Byzantine behaviors and poisoning attacks, which can significantly degrade model performance and hinder convergence. The effectiveness of existing approaches to mitigate complex attacks, such as median, trimmed mean, or Krum aggregation functions, has been only partially demonstrated in the case of specific attacks. Our study introduces a novel robust aggregation mechanism utilizing the Fourier Transform (FT), which is able to effectively handling sophisticated attacks without prior knowledge of the number of attackers. Employing this data technique, weights generated by FL clients are projected into the frequency domain to ascertain their density function, selecting the one exhibiting the highest frequency. Consequently, malicious clients' weights are excluded. Our proposed approach was tested against various model poisoning attacks, demonstrating superior performance over state-of-the-art aggregation methods.

Title: PICS: Pipeline for Image Captioning and Search

• Authors: Grant Rosario, David Noever
• Subjects: cs.CV, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10090
• Pdf URL: https://arxiv.org/pdf/2402.10090
• Copy Paste: [[2402.10090]] PICS: Pipeline for Image Captioning and Search(https://arxiv.org/abs/2402.10090)
• Keywords: large language model
• Abstract: The growing volume of digital images necessitates advanced systems for efficient categorization and retrieval, presenting a significant challenge in database management and information retrieval. This paper introduces PICS (Pipeline for Image Captioning and Search), a novel approach designed to address the complexities inherent in organizing large-scale image repositories. PICS leverages the advancements in Large Language Models (LLMs) to automate the process of image captioning, offering a solution that transcends traditional manual annotation methods. The approach is rooted in the understanding that meaningful, AI-generated captions can significantly enhance the searchability and accessibility of images in large databases. By integrating sentiment analysis into the pipeline, PICS further enriches the metadata, enabling nuanced searches that extend beyond basic descriptors. This methodology not only simplifies the task of managing vast image collections but also sets a new precedent for accuracy and efficiency in image retrieval. The significance of PICS lies in its potential to transform image database systems, harnessing the power of machine learning and natural language processing to meet the demands of modern digital asset management.

Title: Classification Diffusion Models

• Authors: Shahar Yadin, Noam Elata, Tomer Michaeli
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10095
• Pdf URL: https://arxiv.org/pdf/2402.10095
• Copy Paste: [[2402.10095]] Classification Diffusion Models(https://arxiv.org/abs/2402.10095)
• Keywords: diffusion, generative
• Abstract: A prominent family of methods for learning data distributions relies on density ratio estimation (DRE), where a model is trained to $\textit{classify}$ between data samples and samples from some reference distribution. These techniques are successful in simple low-dimensional settings but fail to achieve good results on complex high-dimensional data, like images. A different family of methods for learning distributions is that of denoising diffusion models (DDMs), in which a model is trained to $\textit{denoise}$ data samples. These approaches achieve state-of-the-art results in image, video, and audio generation. In this work, we present $\textit{Classification Diffusion Models}$ (CDMs), a generative technique that adopts the denoising-based formalism of DDMs while making use of a classifier that predicts the amount of noise added to a clean signal, similarly to DRE methods. Our approach is based on the observation that an MSE-optimal denoiser for white Gaussian noise can be expressed in terms of the gradient of a cross-entropy-optimal classifier for predicting the noise level. As we illustrate, CDM achieves better denoising results compared to DDM, and leads to at least comparable FID in image generation. CDM is also capable of highly efficient one-step exact likelihood estimation, achieving state-of-the-art results among methods that use a single step. Code is available on the project's webpage in https://shaharYadin.github.io/CDM/ .

Title: Adaptive Federated Learning in Heterogeneous Wireless Networks with Independent Sampling

• Authors: Jiaxiang Geng, Yanzhao Hou, Xiaofeng Tao, Juncheng Wang, Bing Luo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10097
• Pdf URL: https://arxiv.org/pdf/2402.10097
• Copy Paste: [[2402.10097]] Adaptive Federated Learning in Heterogeneous Wireless Networks with Independent Sampling(https://arxiv.org/abs/2402.10097)
• Keywords: federate
• Abstract: Federated Learning (FL) algorithms commonly sample a random subset of clients to address the straggler issue and improve communication efficiency. While recent works have proposed various client sampling methods, they have limitations in joint system and data heterogeneity design, which may not align with practical heterogeneous wireless networks. In this work, we advocate a new independent client sampling strategy to minimize the wall-clock training time of FL, while considering data heterogeneity and system heterogeneity in both communication and computation. We first derive a new convergence bound for non-convex loss functions with independent client sampling and then propose an adaptive bandwidth allocation scheme. Furthermore, we propose an efficient independent client sampling algorithm based on the upper bounds on the convergence rounds and the expected per-round training time, to minimize the wall-clock time of FL, while considering both the data and system heterogeneity. Experimental results under practical wireless network settings with real-world prototype demonstrate that the proposed independent sampling scheme substantially outperforms the current best sampling schemes under various training models and datasets.

Title: Parameter-tuning-free data entry error unlearning with adaptive selective synaptic dampening

• Authors: Stefan Schoepf, Jack Foster, Alexandra Brintrup
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10098
• Pdf URL: https://arxiv.org/pdf/2402.10098
• Copy Paste: [[2402.10098]] Parameter-tuning-free data entry error unlearning with adaptive selective synaptic dampening(https://arxiv.org/abs/2402.10098)
• Keywords: transformer
• Abstract: Data entry constitutes a fundamental component of the machine learning pipeline, yet it frequently results in the introduction of labelling errors. When a model has been trained on a dataset containing such errors its performance is reduced. This leads to the challenge of efficiently unlearning the influence of the erroneous data to improve the model performance without needing to completely retrain the model. While model editing methods exist for cases in which the correct label for a wrong entry is known, we focus on the case of data entry errors where we do not know the correct labels for the erroneous data. Our contribution is twofold. First, we introduce an extension to the selective synaptic dampening unlearning method that removes the need for parameter tuning, making unlearning accessible to practitioners. We demonstrate the performance of this extension, adaptive selective synaptic dampening (ASSD), on various ResNet18 and Vision Transformer unlearning tasks. Second, we demonstrate the performance of ASSD in a supply chain delay prediction problem with labelling errors using real-world data where we randomly introduce various levels of labelling errors. The application of this approach is particularly compelling in industrial settings, such as supply chain management, where a significant portion of data entry occurs manually through Excel sheets, rendering it error-prone. ASSD shows strong performance on general unlearning benchmarks and on the error correction problem where it outperforms fine-tuning for error correction.

Title: Any-Shift Prompting for Generalization over Distributions

• Authors: Zehao Xiao, Jiayi Shen, Mohammad Mahdi Derakhshani, Shengcai Liao, Cees G. M. Snoek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10099
• Pdf URL: https://arxiv.org/pdf/2402.10099
• Copy Paste: [[2402.10099]] Any-Shift Prompting for Generalization over Distributions(https://arxiv.org/abs/2402.10099)
• Keywords: transformer
• Abstract: Image-language models with prompt learning have shown remarkable advances in numerous downstream vision tasks. Nevertheless, conventional prompt learning methods overfit their training distribution and lose the generalization ability on test distributions. To improve generalization across various distribution shifts, we propose any-shift prompting: a general probabilistic inference framework that considers the relationship between training and test distributions during prompt learning. We explicitly connect training and test distributions in the latent space by constructing training and test prompts in a hierarchical architecture. Within this framework, the test prompt exploits the distribution relationships to guide the generalization of the CLIP image-language model from training to any test distribution. To effectively encode the distribution information and their relationships, we further introduce a transformer inference network with a pseudo-shift training mechanism. The network generates the tailored test prompt with both training and test information in a feedforward pass, avoiding extra training costs at test time. Extensive experiments on twenty-three datasets demonstrate the effectiveness of any-shift prompting on the generalization over various distribution shifts.

Title: Quantized Embedding Vectors for Controllable Diffusion Language Models

• Authors: Cheng Kang, Xinye Chen, Yong Hu, Daniel Novak
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10107
• Pdf URL: https://arxiv.org/pdf/2402.10107
• Copy Paste: [[2402.10107]] Quantized Embedding Vectors for Controllable Diffusion Language Models(https://arxiv.org/abs/2402.10107)
• Keywords: diffusion
• Abstract: Improving the controllability, portability, and inference speed of diffusion language models (DLMs) is a key challenge in natural language generation. While recent research has shown significant success in complex text generation with language models, the memory and computational power are still very demanding and fall short of expectations, which naturally results in low portability and instability for the models. To mitigate these issues, numerous well-established methods were proposed for neural network quantization. To further enhance their portability of independent deployment as well as improve their stability evaluated by language perplexity, we propose a novel approach called the Quantized Embedding Controllable Diffusion Language Model (QE-CDLM). QE-CDLM builds upon the recent successful controllable DLMs by remodeling the task-specific embedding space via quantization. This leads to a gradient-based controller for the generation tasks, and more stable intermediate latent variables are obtained, which naturally brings in an accelerated convergence as well as better controllability. Additionally, the adaption fine-tuning method is employed to reduce tunable weights. Experimental results on five challenging fine-grained control tasks demonstrate that QE-CDLM compares favorably to existing methods in terms of quality and feasibility, achieving better perplexity and lightweight fine-tuning.

Title: Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning

• Authors: Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, Tianyi Zhou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10110
• Pdf URL: https://arxiv.org/pdf/2402.10110
• Copy Paste: [[2402.10110]] Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning(https://arxiv.org/abs/2402.10110)
• Keywords: large language model
• Abstract: Instruction tuning is critical to large language models (LLMs) for achieving better instruction following and task adaptation capabilities but its success heavily relies on the training data quality. Many recent methods focus on improving the data quality but often overlook the compatibility of the data with the student model being finetuned. This paper introduces Selective Reflection-Tuning, a novel paradigm that synergizes a teacher LLM's reflection and introspection for improving existing data quality with the data selection capability of the student LLM, to automatically refine existing instruction-tuning data. This teacher-student collaboration produces high-quality and student-compatible instruction-response pairs, resulting in sample-efficient instruction tuning and LLMs of superior performance. Selective Reflection-Tuning is a data augmentation and synthesis that generally improves LLM finetuning and self-improvement without collecting brand-new data. We apply our method to Alpaca and WizardLM data and achieve much stronger and top-tier 7B and 13B LLMs. Our codes, models, and data will be released at https://github.com/tianyi-lab/Reflection_Tuning.

Title: Is Continual Learning Ready for Real-world Challenges?

• Authors: Theodora Kontogianni, Yuanwen Yue, Siyu Tang, Konrad Schindler
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10130
• Pdf URL: https://arxiv.org/pdf/2402.10130
• Copy Paste: [[2402.10130]] Is Continual Learning Ready for Real-world Challenges?(https://arxiv.org/abs/2402.10130)
• Keywords: segmentation
• Abstract: Despite continual learning's long and well-established academic history, its application in real-world scenarios remains rather limited. This paper contends that this gap is attributable to a misalignment between the actual challenges of continual learning and the evaluation protocols in use, rendering proposed solutions ineffective for addressing the complexities of real-world setups. We validate our hypothesis and assess progress to date, using a new 3D semantic segmentation benchmark, OCL-3DSS. We investigate various continual learning schemes from the literature by utilizing more realistic protocols that necessitate online and continual learning for dynamic, real-world scenarios (eg., in robotics and 3D vision applications). The outcomes are sobering: all considered methods perform poorly, significantly deviating from the upper bound of joint offline training. This raises questions about the applicability of existing methods in realistic settings. Our paper aims to initiate a paradigm shift, advocating for the adoption of continual learning methods through new experimental protocols that better emulate real-world conditions to facilitate breakthroughs in the field.

Title: Benchmarking federated strategies in Peer-to-Peer Federated learning for biomedical data

• Authors: Jose L. Salmeron, Irina Arévalo, Antonio Ruiz-Celma
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.10135
• Pdf URL: https://arxiv.org/pdf/2402.10135
• Copy Paste: [[2402.10135]] Benchmarking federated strategies in Peer-to-Peer Federated learning for biomedical data(https://arxiv.org/abs/2402.10135)
• Keywords: privacy, protect, robust, federate
• Abstract: The increasing requirements for data protection and privacy has attracted a huge research interest on distributed artificial intelligence and specifically on federated learning, an emerging machine learning approach that allows the construction of a model between several participants who hold their own private data. In the initial proposal of federated learning the architecture was centralised and the aggregation was done with federated averaging, meaning that a central server will orchestrate the federation using the most straightforward averaging strategy. This research is focused on testing different federated strategies in a peer-to-peer environment. The authors propose various aggregation strategies for federated learning, including weighted averaging aggregation, using different factors and strategies based on participant contribution. The strategies are tested with varying data sizes to identify the most robust ones. This research tests the strategies with several biomedical datasets and the results of the experiments show that the accuracy-based weighted average outperforms the classical federated averaging method.

Title: TOAD: Task-Oriented Automatic Dialogs with Diverse Response Styles

• Authors: Yinhong Liu, Yimai Fang, David Vandyke, Nigel Collier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10137
• Pdf URL: https://arxiv.org/pdf/2402.10137
• Copy Paste: [[2402.10137]] TOAD: Task-Oriented Automatic Dialogs with Diverse Response Styles(https://arxiv.org/abs/2402.10137)
• Keywords: large language model
• Abstract: In light of recent advances in large language models~(LLMs), the expectations for the next generation of virtual assistants include enhanced naturalness and adaptability across diverse usage scenarios. However, the creation of high-quality annotated data for Task-Oriented Dialog~(TOD) is recognized to be slow and costly. To address these challenges, we introduce Task-Oriented Automatic Dialogs~(TOAD), a novel and scalable TOD dataset along with its automatic generation pipeline. The TOAD dataset simulates realistic app context interaction and provide a variety of system response style options. Two aspects of system response styles are considered, verbosity level and users' expression mirroring. We benchmark TOAD on two response generation tasks and the results show that modeling more verbose or responses without user expression mirroring is more challenging.

Title: Transaction Capacity, Security and Latency in Blockchains

• Authors: Mustafa Doger, Sennur Ulukus
• Subjects: cs.CR, cs.DC, cs.DM, cs.IT
• Abstract URL: https://arxiv.org/abs/2402.10138
• Pdf URL: https://arxiv.org/pdf/2402.10138
• Copy Paste: [[2402.10138]] Transaction Capacity, Security and Latency in Blockchains(https://arxiv.org/abs/2402.10138)
• Keywords: secure, security, attack
• Abstract: We analyze how secure a block is after the block becomes k-deep, i.e., security-latency, for Nakamoto consensus under an exponential network delay model. We give parameter regimes for which transactions are safe when sufficiently deep in the chain. We compare our results for Nakamoto consensus under bounded network delay models and obtain analogous bounds for safety violation threshold. Next, modeling the blockchain system as a batch service queue with exponential network delay, we connect the security-latency analysis to sustainable transaction rate of the queue system. As our model assumes exponential network delay, batch service queue models give a meaningful trade-off between transaction capacity, security and latency. As adversary can attack the queue service to hamper the service process, we consider two different attacks for adversary. In an extreme scenario, we modify the selfish-mining attack for this purpose and consider its effect on the sustainable transaction rate of the queue.

Title: A chaotic maps-based privacy-preserving distributed deep learning for incomplete and Non-IID datasets

• Authors: Irina Arévalo, Jose L. Salmeron
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.10145
• Pdf URL: https://arxiv.org/pdf/2402.10145
• Copy Paste: [[2402.10145]] A chaotic maps-based privacy-preserving distributed deep learning for incomplete and Non-IID datasets(https://arxiv.org/abs/2402.10145)
• Keywords: secure, privacy, federate
• Abstract: Federated Learning is a machine learning approach that enables the training of a deep learning model among several participants with sensitive data that wish to share their own knowledge without compromising the privacy of their data. In this research, the authors employ a secured Federated Learning method with an additional layer of privacy and proposes a method for addressing the non-IID challenge. Moreover, differential privacy is compared with chaotic-based encryption as layer of privacy. The experimental approach assesses the performance of the federated deep learning model with differential privacy using both IID and non-IID data. In each experiment, the Federated Learning process improves the average performance metrics of the deep neural network, even in the case of non-IID data.

Title: $f$-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning

• Authors: Yiwei Lu, Guojun Zhang, Sun Sun, Hongyu Guo, Yaoliang Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10150
• Pdf URL: https://arxiv.org/pdf/2402.10150
• Copy Paste: [[2402.10150]] $f$-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning(https://arxiv.org/abs/2402.10150)
• Keywords: interpretability
• Abstract: In self-supervised contrastive learning, a widely-adopted objective function is InfoNCE, which uses the heuristic cosine similarity for the representation comparison, and is closely related to maximizing the Kullback-Leibler (KL)-based mutual information. In this paper, we aim at answering two intriguing questions: (1) Can we go beyond the KL-based objective? (2) Besides the popular cosine similarity, can we design a better similarity function? We provide answers to both questions by generalizing the KL-based mutual information to the $f$-Mutual Information in Contrastive Learning ($f$-MICL) using the $f$-divergences. To answer the first question, we provide a wide range of $f$-MICL objectives which share the nice properties of InfoNCE (e.g., alignment and uniformity), and meanwhile result in similar or even superior performance. For the second question, assuming that the joint feature distribution is proportional to the Gaussian kernel, we derive an $f$-Gaussian similarity with better interpretability and empirical performance. Finally, we identify close relationships between the $f$-MICL objective and several popular InfoNCE-based objectives. Using benchmark tasks from both vision and natural language, we empirically evaluate $f$-MICL with different $f$-divergences on various architectures (SimCLR, MoCo, and MoCo v3) and datasets. We observe that $f$-MICL generally outperforms the benchmarks and the best-performing $f$-divergence is task and dataset dependent.

Title: Knowledge-Infused LLM-Powered Conversational Health Agent: A Case Study for Diabetes Patients

• Authors: Mahyar Abbasian, Zhongqi Yang, Elahe Khatibi, Pengfei Zhang, Nitish Nagesh, Iman Azimi, Ramesh Jain, Amir M. Rahmani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10153
• Pdf URL: https://arxiv.org/pdf/2402.10153
• Copy Paste: [[2402.10153]] Knowledge-Infused LLM-Powered Conversational Health Agent: A Case Study for Diabetes Patients(https://arxiv.org/abs/2402.10153)
• Keywords: large language model
• Abstract: Effective diabetes management is crucial for maintaining health in diabetic patients. Large Language Models (LLMs) have opened new avenues for diabetes management, facilitating their efficacy. However, current LLM-based approaches are limited by their dependence on general sources and lack of integration with domain-specific knowledge, leading to inaccurate responses. In this paper, we propose a knowledge-infused LLM-powered conversational health agent (CHA) for diabetic patients. We customize and leverage the open-source openCHA framework, enhancing our CHA with external knowledge and analytical capabilities. This integration involves two key components: 1) incorporating the American Diabetes Association dietary guidelines and the Nutritionix information and 2) deploying analytical tools that enable nutritional intake calculation and comparison with the guidelines. We compare the proposed CHA with GPT4. Our evaluation includes 100 diabetes-related questions on daily meal choices and assessing the potential risks associated with the suggested diet. Our findings show that the proposed agent demonstrates superior performance in generating responses to manage essential nutrients.

Title: Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence

• Authors: Yinhong Liu, Yixuan Su, Ehsan Shareghi, Nigel Collier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10175
• Pdf URL: https://arxiv.org/pdf/2402.10175
• Copy Paste: [[2402.10175]] Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence(https://arxiv.org/abs/2402.10175)
• Keywords: large language model
• Abstract: Recent large language models (LLMs) have shown remarkable performance in aligning generated text with user intentions across various tasks. When it comes to long-form text generation, there has been a growing interest in generation from a discourse coherence perspective. However, existing lexical or semantic metrics such as BLEU, ROUGE, BertScore cannot effectively capture the discourse coherence. The development of discourse-specific automatic evaluation methods for assessing the output of LLMs warrants greater focus and exploration. In this paper, we present a novel automatic metric designed to quantify the discourse divergence between two long-form articles. Extensive experiments on three datasets from representative domains demonstrate that our metric aligns more closely with human preferences and GPT-4 coherence evaluation, outperforming existing evaluation methods.

Title: OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset

• Authors: Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, Igor Gitman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10176
• Pdf URL: https://arxiv.org/pdf/2402.10176
• Copy Paste: [[2402.10176]] OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset(https://arxiv.org/abs/2402.10176)
• Keywords: large language model
• Abstract: Recent work has shown the immense potential of synthetically generated datasets for training large language models (LLMs), especially for acquiring targeted skills. Current large-scale math instruction tuning datasets such as MetaMathQA (Yu et al., 2024) and MAmmoTH (Yue et al., 2024) are constructed using outputs from closed-source LLMs with commercially restrictive licenses. A key reason limiting the use of open-source LLMs in these data generation pipelines has been the wide gap between the mathematical skills of the best closed-source LLMs, such as GPT-4, and the best open-source LLMs. Building on the recent progress in open-source LLMs, our proposed prompting novelty, and some brute-force scaling, we construct OpenMathInstruct-1, a math instruction tuning dataset with 1.8M problem-solution pairs. The dataset is constructed by synthesizing code-interpreter solutions for GSM8K and MATH, two popular math reasoning benchmarks, using the recently released and permissively licensed Mixtral model. Our best model, OpenMath-CodeLlama-70B, trained on a subset of OpenMathInstruct-1, achieves a score of 84.6% on GSM8K and 50.7% on MATH, which is competitive with the best gpt-distilled models. We release our code, models, and the OpenMathInstruct-1 dataset under a commercially permissive license.

Title: TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generation

• Authors: Yaoxiang Wang, Zhiyong Wu, Junfeng Yao, Jinsong Su
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10178
• Pdf URL: https://arxiv.org/pdf/2402.10178
• Copy Paste: [[2402.10178]] TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generation(https://arxiv.org/abs/2402.10178)
• Keywords: large language model
• Abstract: The emergence of Large Language Models (LLMs) like ChatGPT has inspired the development of LLM-based agents capable of addressing complex, real-world tasks. However, these agents often struggle during task execution due to methodological constraints, such as error propagation and limited adaptability. To address this issue, we propose a multi-agent framework based on dynamic Task Decomposition and Agent Generation (TDAG). This framework dynamically decomposes complex tasks into smaller subtasks and assigns each to a specifically generated subagent, thereby enhancing adaptability in diverse and unpredictable real-world tasks. Simultaneously, existing benchmarks often lack the granularity needed to evaluate incremental progress in complex, multi-step tasks. In response, we introduce ItineraryBench in the context of travel planning, featuring interconnected, progressively complex tasks with a fine-grained evaluation system. ItineraryBench is designed to assess agents' abilities in memory, planning, and tool usage across tasks of varying complexity. Our experimental results reveal that TDAG significantly outperforms established baselines, showcasing its superior adaptability and context awareness in complex task scenarios.

Title: Rethinking Information Structures in RLHF: Reward Generalization from a Graph Theory Perspective

• Authors: Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Han Yang, Josef Dai, Xuehai Pan, Yaodong Yang
• Subjects: cs.LG, cs.AI, cs.CL, cs.DM
• Abstract URL: https://arxiv.org/abs/2402.10184
• Pdf URL: https://arxiv.org/pdf/2402.10184
• Copy Paste: [[2402.10184]] Rethinking Information Structures in RLHF: Reward Generalization from a Graph Theory Perspective(https://arxiv.org/abs/2402.10184)
• Keywords: large language model
• Abstract: There is a trilemma in reinforcement learning from human feedback (RLHF): the incompatibility between highly diverse contexts, low labeling cost, and reliable alignment performance. Here we aim to mitigate such incompatibility through the design of dataset information structures during reward modeling. Specifically, we first reexamine the RLHF process and propose a theoretical framework portraying it as an autoencoding process over text distributions. Our framework formalizes the RLHF objective of ensuring distributional consistency between human preference and large language model (LLM) behavior. Building on this framework, we then systematically investigate the performance impact of information structure in the reward modeling stage of RLHF. To further understand reward generalization in the reward modeling stage, we introduce a new method based on random graph theory that models generalization in the semantic space. A key insight of our analysis is the superiority of the tree-based information structure in reward modeling, compared to chain-based baselines adopted by conventional RLHF methods. We derive that under highly complex contexts with limited data, the tree-based reward model (RM) induces up to $\Theta(\log n/\log\log n)$ times less variance than chain-based RM where $n$ is the dataset size. To validate our theoretical contribution, we demonstrate that on three different NLP tasks, the tree-based RM achieves 65% win rate on average against chain-based baselines. Looking forward, we hope our framework can serve as a step towards understanding goal misgeneralization.

Title: Uncertainty Decomposition and Quantification for In-Context Learning of Large Language Models

• Authors: Chen Ling, Xujiang Zhao, Wei Cheng, Yanchi Liu, Yiyou Sun, Xuchao Zhang, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, Haifeng Chen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10189
• Pdf URL: https://arxiv.org/pdf/2402.10189
• Copy Paste: [[2402.10189]] Uncertainty Decomposition and Quantification for In-Context Learning of Large Language Models(https://arxiv.org/abs/2402.10189)
• Keywords: large language model
• Abstract: In-context learning has emerged as a groundbreaking ability of Large Language Models (LLMs) and revolutionized various fields by providing a few task-relevant demonstrations in the prompt. However, trustworthy issues with LLM's response, such as hallucination, have also been actively discussed. Existing works have been devoted to quantifying the uncertainty in LLM's response, but they often overlook the complex nature of LLMs and the uniqueness of in-context learning. In this work, we delve into the predictive uncertainty of LLMs associated with in-context learning, highlighting that such uncertainties may stem from both the provided demonstrations (aleatoric uncertainty) and ambiguities tied to the model's configurations (epistemic uncertainty). We propose a novel formulation and corresponding estimation method to quantify both types of uncertainties. The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. Extensive experiments are conducted to demonstrate the effectiveness of the decomposition. The code and data are available at: \url{https://github.com/lingchen0331/UQ_ICL}.

Title: FedAnchor: Enhancing Federated Semi-Supervised Learning with Label Contrastive Loss for Unlabeled Clients

• Authors: Xinchi Qiu, Yan Gao, Lorenzo Sani, Heng Pan, Wanru Zhao, Pedro P. B. Gusmao, Mina Alibeigi, Alex Iacob, Nicholas D. Lane
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10191
• Pdf URL: https://arxiv.org/pdf/2402.10191
• Copy Paste: [[2402.10191]] FedAnchor: Enhancing Federated Semi-Supervised Learning with Label Contrastive Loss for Unlabeled Clients(https://arxiv.org/abs/2402.10191)
• Keywords: federate
• Abstract: Federated learning (FL) is a distributed learning paradigm that facilitates collaborative training of a shared global model across devices while keeping data localized. The deployment of FL in numerous real-world applications faces delays, primarily due to the prevalent reliance on supervised tasks. Generating detailed labels at edge devices, if feasible, is demanding, given resource constraints and the imperative for continuous data updates. In addressing these challenges, solutions such as federated semi-supervised learning (FSSL), which relies on unlabeled clients' data and a limited amount of labeled data on the server, become pivotal. In this paper, we propose FedAnchor, an innovative FSSL method that introduces a unique double-head structure, called anchor head, paired with the classification head trained exclusively on labeled anchor data on the server. The anchor head is empowered with a newly designed label contrastive loss based on the cosine similarity metric. Our approach mitigates the confirmation bias and overfitting issues associated with pseudo-labeling techniques based on high-confidence model prediction samples. Extensive experiments on CIFAR10/100 and SVHN datasets demonstrate that our method outperforms the state-of-the-art method by a significant margin in terms of convergence rate and model accuracy.

Title: Multi-Excitation Projective Simulation with a Many-Body Physics Inspired Inductive Bias

• Authors: Philip A. LeMaitre, Marius Krumm, Hans J. Briegel
• Subjects: cs.LG, cs.AI, cs.DM, quant-ph
• Abstract URL: https://arxiv.org/abs/2402.10192
• Pdf URL: https://arxiv.org/pdf/2402.10192
• Copy Paste: [[2402.10192]] Multi-Excitation Projective Simulation with a Many-Body Physics Inspired Inductive Bias(https://arxiv.org/abs/2402.10192)
• Keywords: interpretability
• Abstract: With the impressive progress of deep learning, applications relying on machine learning are increasingly being integrated into daily life. However, most deep learning models have an opaque, oracle-like nature making it difficult to interpret and understand their decisions. This problem led to the development of the field known as eXplainable Artificial Intelligence (XAI). One method in this field known as Projective Simulation (PS) models a chain-of-thought as a random walk of a particle on a graph with vertices that have concepts attached to them. While this description has various benefits, including the possibility of quantization, it cannot be naturally used to model thoughts that combine several concepts simultaneously. To overcome this limitation, we introduce Multi-Excitation Projective Simulation (mePS), a generalization that considers a chain-of-thought to be a random walk of several particles on a hypergraph. A definition for a dynamic hypergraph is put forward to describe the agent's training history along with applications to AI and hypergraph visualization. An inductive bias inspired by the remarkably successful few-body interaction models used in quantum many-body physics is formalized for our classical mePS framework and employed to tackle the exponential complexity associated with naive implementations of hypergraphs. We prove that our inductive bias reduces the complexity from exponential to polynomial, with the exponent representing the cutoff on how many particles can interact. We numerically apply our method to two toy environments and a more complex scenario modelling the diagnosis of a broken computer. These environments demonstrate the resource savings provided by an appropriate choice of inductive bias, as well as showcasing aspects of interpretability. A quantum model for mePS is also briefly outlined and some future directions for it are discussed.

Title: BitDelta: Your Fine-Tune May Only Be Worth One Bit

• Authors: James Liu, Guangxuan Xiao, Kai Li, Jason D. Lee, Song Han, Tri Dao, Tianle Cai
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10193
• Pdf URL: https://arxiv.org/pdf/2402.10193
• Copy Paste: [[2402.10193]] BitDelta: Your Fine-Tune May Only Be Worth One Bit(https://arxiv.org/abs/2402.10193)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are typically trained in two phases: pre-training on large internet-scale datasets, and fine-tuning for downstream tasks. Given the higher computational demand of pre-training, it's intuitive to assume that fine-tuning adds less new information to the model, and is thus more compressible. We explore this assumption by decomposing the weights of fine-tuned models into their pre-trained components and an additional delta. We introduce a simple method, BitDelta, which successfully quantizes this delta down to 1 bit without compromising performance. This interesting finding not only highlights the potential redundancy of information added during fine-tuning, but also has significant implications for the multi-tenant serving and multi-tenant storage of fine-tuned models. By enabling the use of a single high-precision base model accompanied by multiple 1-bit deltas, BitDelta dramatically reduces GPU memory requirements by more than 10x, which can also be translated to enhanced generation latency in multi-tenant settings. We validate BitDelta through experiments across Llama-2 and Mistral model families, and on models up to 70B parameters, showcasing minimal performance degradation over all tested settings.

Title: A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents

• Authors: Lingbo Mo, Zeyi Liao, Boyuan Zheng, Yu Su, Chaowei Xiao, Huan Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10196
• Pdf URL: https://arxiv.org/pdf/2402.10196
• Copy Paste: [[2402.10196]] A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents(https://arxiv.org/abs/2402.10196)
• Keywords: attack, large language model
• Abstract: Language agents powered by large language models (LLMs) have seen exploding development. Their capability of using language as a vehicle for thought and communication lends an incredible level of flexibility and versatility. People have quickly capitalized on this capability to connect LLMs to a wide range of external components and environments: databases, tools, the Internet, robotic embodiment, etc. Many believe an unprecedentedly powerful automation technology is emerging. However, new automation technologies come with new safety risks, especially for intricate systems like language agents. There is a surprisingly large gap between the speed and scale of their development and deployment and our understanding of their safety risks. Are we building a house of cards? In this position paper, we present the first systematic effort in mapping adversarial attacks against language agents. We first present a unified conceptual framework for agents with three major components: Perception, Brain, and Action. Under this framework, we present a comprehensive discussion and propose 12 potential attack scenarios against different components of an agent, covering different attack strategies (e.g., input manipulation, adversarial demonstrations, jailbreaking, backdoors). We also draw connections to successful attack strategies previously applied to LLMs. We emphasize the urgency to gain a thorough understanding of language agent risks before their widespread deployment.

Title: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention

• Authors: Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov, Aladin Virmaux, Giuseppe Paolo, Themis Palpanas, Ievgen Redko
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.10198
• Pdf URL: https://arxiv.org/pdf/2402.10198
• Copy Paste: [[2402.10198]] Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention(https://arxiv.org/abs/2402.10198)
• Keywords: transformer
• Abstract: Transformer-based architectures achieved breakthrough performance in natural language processing and computer vision, yet they remain inferior to simpler linear baselines in multivariate long-term forecasting. To better understand this phenomenon, we start by studying a toy linear forecasting problem for which we show that transformers are incapable of converging to their true solution despite their high expressive power. We further identify the attention of transformers as being responsible for this low generalization capacity. Building upon this insight, we propose a shallow lightweight transformer model that successfully escapes bad local minima when optimized with sharpness-aware optimization. We empirically demonstrate that this result extends to all commonly used real-world multivariate time series datasets. In particular, SAMformer surpasses the current state-of-the-art model TSMixer by 14.33% on average, while having ~4 times fewer parameters. The code is available at https://github.com/romilbert/samformer.

Title: Chain-of-Thought Reasoning Without Prompting

• Authors: Xuezhi Wang, Denny Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10200
• Pdf URL: https://arxiv.org/pdf/2402.10200
• Copy Paste: [[2402.10200]] Chain-of-Thought Reasoning Without Prompting(https://arxiv.org/abs/2402.10200)
• Keywords: large language model
• Abstract: In enhancing the reasoning capabilities of large language models (LLMs), prior research primarily focuses on specific prompting techniques such as few-shot or zero-shot chain-of-thought (CoT) prompting. These methods, while effective, often involve manually intensive prompt engineering. Our study takes a novel approach by asking: Can LLMs reason effectively without prompting? Our findings reveal that, intriguingly, CoT reasoning paths can be elicited from pre-trained LLMs by simply altering the \textit{decoding} process. Rather than conventional greedy decoding, we investigate the top-$k$ alternative tokens, uncovering that CoT paths are frequently inherent in these sequences. This approach not only bypasses the confounders of prompting but also allows us to assess the LLMs' \textit{intrinsic} reasoning abilities. Moreover, we observe that the presence of a CoT in the decoding path correlates with a higher confidence in the model's decoded answer. This confidence metric effectively differentiates between CoT and non-CoT paths. Extensive empirical studies on various reasoning benchmarks show that the proposed CoT-decoding substantially outperforms the standard greedy decoding.

Title: Bridging Associative Memory and Probabilistic Modeling

• Authors: Rylan Schaeffer, Nika Zahedi, Mikail Khona, Dhruv Pai, Sang Truong, Yilun Du, Mitchell Ostrow, Sarthak Chandra, Andres Carranza, Ila Rani Fiete, Andrey Gromov, Sanmi Koyejo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10202
• Pdf URL: https://arxiv.org/pdf/2402.10202
• Copy Paste: [[2402.10202]] Bridging Associative Memory and Probabilistic Modeling(https://arxiv.org/abs/2402.10202)
• Keywords: transformer
• Abstract: Associative memory and probabilistic modeling are two fundamental topics in artificial intelligence. The first studies recurrent neural networks designed to denoise, complete and retrieve data, whereas the second studies learning and sampling from probability distributions. Based on the observation that associative memory's energy functions can be seen as probabilistic modeling's negative log likelihoods, we build a bridge between the two that enables useful flow of ideas in both directions. We showcase four examples: First, we propose new energy-based models that flexibly adapt their energy functions to new in-context datasets, an approach we term \textit{in-context learning of energy functions}. Second, we propose two new associative memory models: one that dynamically creates new memories as necessitated by the training data using Bayesian nonparametrics, and another that explicitly computes proportional memory assignments using the evidence lower bound. Third, using tools from associative memory, we analytically and numerically characterize the memory capacity of Gaussian kernel density estimators, a widespread tool in probababilistic modeling. Fourth, we study a widespread implementation choice in transformers -- normalization followed by self attention -- to show it performs clustering on the hypersphere. Altogether, this work urges further exchange of useful ideas between these two continents of artificial intelligence.

Title: Ising on the Graph: Task-specific Graph Subsampling via the Ising Model

• Authors: Maria Bånkestad, Jennifer Andersson, Sebastian Mair, Jens Sjölund
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10206
• Pdf URL: https://arxiv.org/pdf/2402.10206
• Copy Paste: [[2402.10206]] Ising on the Graph: Task-specific Graph Subsampling via the Ising Model(https://arxiv.org/abs/2402.10206)
• Keywords: segmentation
• Abstract: Reducing a graph while preserving its overall structure is an important problem with many applications. Typically, the reduction approaches either remove edges (sparsification) or merge nodes (coarsening) in an unsupervised way with no specific downstream task in mind. In this paper, we present an approach for subsampling graph structures using an Ising model defined on either the nodes or edges and learning the external magnetic field of the Ising model using a graph neural network. Our approach is task-specific as it can learn how to reduce a graph for a specific downstream task in an end-to-end fashion. The utilized loss function of the task does not even have to be differentiable. We showcase the versatility of our approach on three distinct applications: image segmentation, 3D shape sparsification, and sparse approximate matrix inverse determination.

Title: Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment

• Authors: Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, Jianshu Chen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10207
• Pdf URL: https://arxiv.org/pdf/2402.10207
• Copy Paste: [[2402.10207]] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment(https://arxiv.org/abs/2402.10207)
• Keywords: diffusion, large language model
• Abstract: We consider the problem of multi-objective alignment of foundation models with human preferences, which is a critical step towards helpful and harmless AI systems. However, it is generally costly and unstable to fine-tune large foundation models using reinforcement learning (RL), and the multi-dimensionality, heterogeneity, and conflicting nature of human preferences further complicate the alignment process. In this paper, we introduce Rewards-in-Context (RiC), which conditions the response of a foundation model on multiple rewards in its prompt context and applies supervised fine-tuning for alignment. The salient features of RiC are simplicity and adaptivity, as it only requires supervised fine-tuning of a single foundation model and supports dynamic adjustment for user preferences during inference time. Inspired by the analytical solution of an abstracted convex optimization problem, our dynamic inference-time adjustment method approaches the Pareto-optimal solution for multiple objectives. Empirical evidence demonstrates the efficacy of our method in aligning both Large Language Models (LLMs) and diffusion models to accommodate diverse rewards with only around $10\%$ GPU hours compared with multi-objective RL baseline.

Title: Recovering the Pre-Fine-Tuning Weights of Generative Models

• Authors: Eliahu Horwitz, Jonathan Kahana, Yedid Hoshen
• Subjects: cs.LG, cs.CL, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10208
• Pdf URL: https://arxiv.org/pdf/2402.10208
• Copy Paste: [[2402.10208]] Recovering the Pre-Fine-Tuning Weights of Generative Models(https://arxiv.org/abs/2402.10208)
• Keywords: attack, diffusion, generative
• Abstract: The dominant paradigm in generative modeling consists of two steps: i) pre-training on a large-scale but unsafe dataset, ii) aligning the pre-trained model with human values via fine-tuning. This practice is considered safe, as no current method can recover the unsafe, pre-fine-tuning model weights. In this paper, we demonstrate that this assumption is often false. Concretely, we present Spectral DeTuning, a method that can recover the weights of the pre-fine-tuning model using a few low-rank (LoRA) fine-tuned models. In contrast to previous attacks that attempt to recover pre-fine-tuning capabilities, our method aims to recover the exact pre-fine-tuning weights. Our approach exploits this new vulnerability against large-scale models such as a personalized Stable Diffusion and an aligned Mistral.

Title: Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation

• Authors: Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, Quanquan Gu
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.10210
• Pdf URL: https://arxiv.org/pdf/2402.10210
• Copy Paste: [[2402.10210]] Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation(https://arxiv.org/abs/2402.10210)
• Keywords: diffusion, generative, large language model
• Abstract: Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI), especially when compared with the remarkable progress made in fine-tuning Large Language Models (LLMs). While cutting-edge diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised fine-tuning, their performance inevitably plateaus after seeing a certain volume of data. Recently, reinforcement learning (RL) has been employed to fine-tune diffusion models with human preference data, but it requires at least two images ("winner" and "loser" images) for each text prompt. In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion), where the diffusion model engages in competition with its earlier versions, facilitating an iterative self-improvement process. Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment. Our experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms the existing supervised fine-tuning method in aspects of human preference alignment and visual appeal right from its first iteration. By the second iteration, it exceeds the performance of RLHF-based methods across all metrics, achieving these results with less data.

Title: Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling

• Authors: Raunaq Bhirangi, Chenyu Wang, Venkatesh Pattabiraman, Carmel Majidi, Abhinav Gupta, Tess Hellebrekers, Lerrel Pinto
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.10211
• Pdf URL: https://arxiv.org/pdf/2402.10211
• Copy Paste: [[2402.10211]] Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling(https://arxiv.org/abs/2402.10211)
• Keywords: transformer
• Abstract: Reasoning from sequences of raw sensory data is a ubiquitous problem across fields ranging from medical devices to robotics. These problems often involve using long sequences of raw sensor data (e.g. magnetometers, piezoresistors) to predict sequences of desirable physical quantities (e.g. force, inertial measurements). While classical approaches are powerful for locally-linear prediction problems, they often fall short when using real-world sensors. These sensors are typically non-linear, are affected by extraneous variables (e.g. vibration), and exhibit data-dependent drift. For many problems, the prediction task is exacerbated by small labeled datasets since obtaining ground-truth labels requires expensive equipment. In this work, we present Hierarchical State-Space Models (HiSS), a conceptually simple, new technique for continuous sequential prediction. HiSS stacks structured state-space models on top of each other to create a temporal hierarchy. Across six real-world sensor datasets, from tactile-based state prediction to accelerometer-based inertial measurement, HiSS outperforms state-of-the-art sequence models such as causal Transformers, LSTMs, S4, and Mamba by at least 23% on MSE. Our experiments further indicate that HiSS demonstrates efficient scaling to smaller datasets and is compatible with existing data-filtering techniques. Code, datasets and videos can be found on https://hiss-csp.github.io.