2024-03-06

Title: On the Convergence of Federated Learning Algorithms without Data Similarity

• Authors: Ali Beikmohammadi, Sarit Khirirat, Sindri Magnússon
• Subjects: cs.LG, cs.GT
• Abstract URL: https://arxiv.org/abs/2403.02347
• Pdf URL: https://arxiv.org/pdf/2403.02347
• Copy Paste: [[2403.02347]] On the Convergence of Federated Learning Algorithms without Data Similarity(https://arxiv.org/abs/2403.02347)
• Keywords: federate
• Abstract: Data similarity assumptions have traditionally been relied upon to understand the convergence behaviors of federated learning methods. Unfortunately, this approach often demands fine-tuning step sizes based on the level of data similarity. When data similarity is low, these small step sizes result in an unacceptably slow convergence speed for federated methods. In this paper, we present a novel and unified framework for analyzing the convergence of federated learning algorithms without the need for data similarity conditions. Our analysis centers on an inequality that captures the influence of step sizes on algorithmic convergence performance. By applying our theorems to well-known federated algorithms, we derive precise expressions for three widely used step size schedules: fixed, diminishing, and step-decay step sizes, which are independent of data similarity conditions. Finally, we conduct comprehensive evaluations of the performance of these federated learning algorithms, employing the proposed step size strategies to train deep neural network models on benchmark datasets under varying data similarity conditions. Our findings demonstrate significant improvements in convergence speed and overall performance, marking a substantial advancement in federated learning research.

Title: Towards Optimal Customized Architecture for Heterogeneous Federated Learning with Contrastive Cloud-Edge Model Decoupling

• Authors: Xingyan Chen, Tian Du, Mu Wang, Tiancheng Gu, Yu Zhao, Gang Kou, Changqiao Xu, Dapeng Oliver Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02360
• Pdf URL: https://arxiv.org/pdf/2403.02360
• Copy Paste: [[2403.02360]] Towards Optimal Customized Architecture for Heterogeneous Federated Learning with Contrastive Cloud-Edge Model Decoupling(https://arxiv.org/abs/2403.02360)
• Keywords: federate
• Abstract: Federated learning, as a promising distributed learning paradigm, enables collaborative training of a global model across multiple network edge clients without the need for central data collecting. However, the heterogeneity of edge data distribution drags the model towards the local minima, which can be distant from the global optimum. Such heterogeneity often leads to slow convergence and substantial communication overhead. To address these issues, we propose a novel federated learning framework called FedCMD, a model decoupling tailored to the Cloud-edge supported federated learning that separates deep neural networks into a body for capturing shared representations in Cloud and a personalized head for migrating data heterogeneity. Our motivation is that, by the deep investigation of the performance of selecting different neural network layers as the personalized head, we found rigidly assigning the last layer as the personalized head in current studies is not always optimal. Instead, it is necessary to dynamically select the personalized layer that maximizes the training performance by taking the representation difference between neighbor layers into account. To find the optimal personalized layer, we utilize the low-dimensional representation of each layer to contrast feature distribution transfer and introduce a Wasserstein-based layer selection method, aimed at identifying the best-match layer for personalization. Additionally, a weighted global aggregation algorithm is proposed based on the selected personalized layer for the practical application of FedCMD. Extensive experiments on ten benchmarks demonstrate the efficiency and superior performance of our solution compared with nine state-of-the-art solutions. All code and results are available at https://github.com/elegy112138/FedCMD.

Title: Addressing Long-Tail Noisy Label Learning Problems: a Two-Stage Solution with Label Refurbishment Considering Label Rarity

• Authors: Ying-Hsuan Wu, Jun-Wei Hsieh, Li Xin, Shin-You Teng, Yi-Kuan Hsieh, Ming-Ching Chang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02363
• Pdf URL: https://arxiv.org/pdf/2403.02363
• Copy Paste: [[2403.02363]] Addressing Long-Tail Noisy Label Learning Problems: a Two-Stage Solution with Label Refurbishment Considering Label Rarity(https://arxiv.org/abs/2403.02363)
• Keywords: robust
• Abstract: Real-world datasets commonly exhibit noisy labels and class imbalance, such as long-tailed distributions. While previous research addresses this issue by differentiating noisy and clean samples, reliance on information from predictions based on noisy long-tailed data introduces potential errors. To overcome the limitations of prior works, we introduce an effective two-stage approach by combining soft-label refurbishing with multi-expert ensemble learning. In the first stage of robust soft label refurbishing, we acquire unbiased features through contrastive learning, making preliminary predictions using a classifier trained with a carefully designed BAlanced Noise-tolerant Cross-entropy (BANC) loss. In the second stage, our label refurbishment method is applied to obtain soft labels for multi-expert ensemble learning, providing a principled solution to the long-tail noisy label problem. Experiments conducted across multiple benchmarks validate the superiority of our approach, Label Refurbishment considering Label Rarity (LR^2), achieving remarkable accuracies of 94.19% and 77.05% on simulated noisy CIFAR-10 and CIFAR-100 long-tail datasets, as well as 77.74% and 81.40% on real-noise long-tail datasets, Food-101N and Animal-10N, surpassing existing state-of-the-art methods.

Title: Human Evaluation of English--Irish Transformer-Based NMT

• Authors: Séamus Lankford, Haithem Afli, Andy Way
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02366
• Pdf URL: https://arxiv.org/pdf/2403.02366
• Copy Paste: [[2403.02366]] Human Evaluation of English--Irish Transformer-Based NMT(https://arxiv.org/abs/2403.02366)
• Keywords: transformer
• Abstract: In this study, a human evaluation is carried out on how hyperparameter settings impact the quality of Transformer-based Neural Machine Translation (NMT) for the low-resourced English--Irish pair. SentencePiece models using both Byte Pair Encoding (BPE) and unigram approaches were appraised. Variations in model architectures included modifying the number of layers, evaluating the optimal number of heads for attention and testing various regularisation techniques. The greatest performance improvement was recorded for a Transformer-optimized model with a 16k BPE subword model. Compared with a baseline Recurrent Neural Network (RNN) model, a Transformer-optimized model demonstrated a BLEU score improvement of 7.8 points. When benchmarked against Google Translate, our translation engines demonstrated significant improvements. Furthermore, a quantitative fine-grained manual evaluation was conducted which compared the performance of machine translation systems. Using the Multidimensional Quality Metrics (MQM) error taxonomy, a human evaluation of the error types generated by an RNN-based system and a Transformer-based system was explored. Our findings show the best-performing Transformer system significantly reduces both accuracy and fluency errors when compared with an RNN-based model.

Title: adaptNMT: an open-source, language-agnostic development environment for Neural Machine Translation

• Authors: Séamus Lankford, Haithem Afli, Andy Way
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02367
• Pdf URL: https://arxiv.org/pdf/2403.02367
• Copy Paste: [[2403.02367]] adaptNMT: an open-source, language-agnostic development environment for Neural Machine Translation(https://arxiv.org/abs/2403.02367)
• Keywords: transformer, segmentation
• Abstract: adaptNMT streamlines all processes involved in the development and deployment of RNN and Transformer neural translation models. As an open-source application, it is designed for both technical and non-technical users who work in the field of machine translation. Built upon the widely-adopted OpenNMT ecosystem, the application is particularly useful for new entrants to the field since the setup of the development environment and creation of train, validation and test splits is greatly simplified. Graphing, embedded within the application, illustrates the progress of model training, and SentencePiece is used for creating subword segmentation models. Hyperparameter customization is facilitated through an intuitive user interface, and a single-click model development approach has been implemented. Models developed by adaptNMT can be evaluated using a range of metrics, and deployed as a translation service within the application. To support eco-friendly research in the NLP space, a green report also flags the power consumption and kgCO$_{2}$ emissions generated during model development. The application is freely available.

Title: adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds

• Authors: Séamus Lankford, Haithem Afli, Andy Way
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02370
• Pdf URL: https://arxiv.org/pdf/2403.02370
• Copy Paste: [[2403.02370]] adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds(https://arxiv.org/abs/2403.02370)
• Keywords: large language model
• Abstract: The advent of Multilingual Language Models (MLLMs) and Large Language Models has spawned innovation in many areas of natural language processing. Despite the exciting potential of this technology, its impact on developing high-quality Machine Translation (MT) outputs for low-resource languages remains relatively under-explored. Furthermore, an open-source application, dedicated to both fine-tuning MLLMs and managing the complete MT workflow for low-resources languages, remains unavailable. We aim to address these imbalances through the development of adaptMLLM, which streamlines all processes involved in the fine-tuning of MLLMs for MT. This open-source application is tailored for developers, translators, and users who are engaged in MT. An intuitive interface allows for easy customisation of hyperparameters, and the application offers a range of metrics for model evaluation and the capability to deploy models as a translation service directly within the application. As a multilingual tool, we used adaptMLLM to fine-tune models for two low-resource language pairs: English to Irish (EN$\leftrightarrow$GA) and English to Marathi (EN$\leftrightarrow$MR). Compared with baselines from the LoResMT2021 Shared Task, the adaptMLLM system demonstrated significant improvements. In the EN$\rightarrow$GA direction, an improvement of 5.2 BLEU points was observed and an increase of 40.5 BLEU points was recorded in the GA$\rightarrow$EN direction. Significant improvements in the translation performance of the EN$\leftrightarrow$MR pair were also observed notably in the MR$\rightarrow$EN direction with an increase of 21.3 BLEU points. Finally, a fine-grained human evaluation of the MLLM output on the EN$\rightarrow$GA pair was conducted using the Multidimensional Quality Metrics and Scalar Quality Metrics error taxonomies. The application and models are freely available.

Title: OTClean: Data Cleaning for Conditional Independence Violations using Optimal Transport

• Authors: Alireza Pirhadi, Mohammad Hossein Moslemi, Alexander Cloninger, Mostafa Milani, Babak Salimi
• Subjects: cs.LG, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2403.02372
• Pdf URL: https://arxiv.org/pdf/2403.02372
• Copy Paste: [[2403.02372]] OTClean: Data Cleaning for Conditional Independence Violations using Optimal Transport(https://arxiv.org/abs/2403.02372)
• Keywords: fair
• Abstract: Ensuring Conditional Independence (CI) constraints is pivotal for the development of fair and trustworthy machine learning models. In this paper, we introduce \sys, a framework that harnesses optimal transport theory for data repair under CI constraints. Optimal transport theory provides a rigorous framework for measuring the discrepancy between probability distributions, thereby ensuring control over data utility. We formulate the data repair problem concerning CIs as a Quadratically Constrained Linear Program (QCLP) and propose an alternating method for its solution. However, this approach faces scalability issues due to the computational cost associated with computing optimal transport distances, such as the Wasserstein distance. To overcome these scalability challenges, we reframe our problem as a regularized optimization problem, enabling us to develop an iterative algorithm inspired by Sinkhorn's matrix scaling algorithm, which efficiently addresses high-dimensional and large-scale data. Through extensive experiments, we demonstrate the efficacy and efficiency of our proposed methods, showcasing their practical utility in real-world data cleaning and preprocessing tasks. Furthermore, we provide comparisons with traditional approaches, highlighting the superiority of our techniques in terms of preserving data utility while ensuring adherence to the desired CI constraints.

Title: NiNformer: A Network in Network Transformer with Token Mixing Generated Gating Function

• Authors: Abdullah Nazhat Abdullah, Tarkan Aydin
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02411
• Pdf URL: https://arxiv.org/pdf/2403.02411
• Copy Paste: [[2403.02411]] NiNformer: A Network in Network Transformer with Token Mixing Generated Gating Function(https://arxiv.org/abs/2403.02411)
• Keywords: transformer, segmentation
• Abstract: The Attention mechanism is the main component of the Transformer architecture, and since its introduction, it has led to significant advancements in Deep Learning that span many domains and multiple tasks. The Attention Mechanism was utilized in Computer Vision as the Vision Transformer ViT, and its usage has expanded into many tasks in the vision domain, such as classification, segmentation, object detection, and image generation. While this mechanism is very expressive and capable, it comes with the drawback of being computationally expensive and requiring datasets of considerable size for effective optimization. To address these shortcomings, many designs have been proposed in the literature to reduce the computational burden and alleviate the data size requirements. Examples of such attempts in the vision domain are the MLP-Mixer, the Conv-Mixer, the Perciver-IO, and many more. This paper introduces a new computational block as an alternative to the standard ViT block that reduces the compute burdens by replacing the normal Attention layers with a Network in Network structure that enhances the static approach of the MLP Mixer with a dynamic system of learning an element-wise gating function by a token mixing process. Extensive experimentation shows that the proposed design provides better performance than the baseline architectures on multiple datasets applied in the image classification task of the vision domain.

Title: Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems

• Authors: Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, James Zou
• Subjects: cs.LG, cs.AI, cs.CL, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.02419
• Pdf URL: https://arxiv.org/pdf/2403.02419
• Copy Paste: [[2403.02419]] Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems(https://arxiv.org/abs/2403.02419)
• Keywords: large language model
• Abstract: Many recent state-of-the-art results in language tasks were achieved using compound systems that perform multiple Large Language Model (LLM) calls and aggregate their responses. However, there is little understanding of how the number of LLM calls -- e.g., when asking the LLM to answer each question multiple times and taking a consensus -- affects such a compound system's performance. In this paper, we initiate the study of scaling laws of compound inference systems. We analyze, theoretically and empirically, how the number of LLM calls affects the performance of one-layer Voting Inference Systems -- one of the simplest compound systems, which aggregates LLM responses via majority voting. We find empirically that across multiple language tasks, surprisingly, Voting Inference Systems' performance first increases but then decreases as a function of the number of LLM calls. Our theoretical results suggest that this non-monotonicity is due to the diversity of query difficulties within a task: more LLM calls lead to higher performance on "easy" queries, but lower performance on "hard" queries, and non-monotone behavior emerges when a task contains both types of queries. This insight then allows us to compute, from a small number of samples, the number of LLM calls that maximizes system performance, and define a scaling law of Voting Inference Systems. Experiments show that our scaling law can predict the performance of Voting Inference Systems and find the optimal number of LLM calls to make.

Title: Towards efficient deep autoencoders for multivariate time series anomaly detection

• Authors: Marcin Pietroń, Dominik Żurek, Kamil Faber, Roberto Corizzo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02429
• Pdf URL: https://arxiv.org/pdf/2403.02429
• Copy Paste: [[2403.02429]] Towards efficient deep autoencoders for multivariate time series anomaly detection(https://arxiv.org/abs/2403.02429)
• Keywords: robust
• Abstract: Multivariate time series anomaly detection is a crucial problem in many industrial and research applications. Timely detection of anomalies allows, for instance, to prevent defects in manufacturing processes and failures in cyberphysical systems. Deep learning methods are preferred among others for their accuracy and robustness for the analysis of complex multivariate data. However, a key aspect is being able to extract predictions in a timely manner, to accommodate real-time requirements in different applications. In the case of deep learning models, model reduction is extremely important to achieve optimal results in real-time systems with limited time and memory constraints. In this paper, we address this issue by proposing a novel compression method for deep autoencoders that involves three key factors. First, pruning reduces the number of weights, while preventing catastrophic drops in accuracy by means of a fast search process that identifies high sparsity levels. Second, linear and non-linear quantization reduces model complexity by reducing the number of bits for every single weight. The combined contribution of these three aspects allow the model size to be reduced, by removing a subset of the weights (pruning), and decreasing their bit-width (quantization). As a result, the compressed model is faster and easier to adopt in highly constrained hardware environments. Experiments performed on popular multivariate anomaly detection benchmarks, show that our method is capable of achieving significant model compression ratio (between 80% and 95%) without a significant reduction in the anomaly detection performance.

Title: How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider Transformer Models

• Authors: Xin Lu, Yanyan Zhao, Bing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02436
• Pdf URL: https://arxiv.org/pdf/2403.02436
• Copy Paste: [[2403.02436]] How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider Transformer Models(https://arxiv.org/abs/2403.02436)
• Keywords: transformer
• Abstract: Pre-trained language models have been proven to possess strong base capabilities, which not only excel in in-distribution language modeling but also show powerful abilities in out-of-distribution language modeling, transfer learning and few-shot learning. Unlike existing work focusing on the influence of scale on base capabilities, our work examines the influence of architecture on those. Specifically, our concern is: How does architecture influence the base capabilities of pre-trained language models? In this work, we attempt to explain and reverse the decline in base capabilities caused by the architecture of FFN-Wider Transformers, seeking to provide some insights. Through analysis, we found the contribution ratio of Multi-Head Attention (a combination function) to pre-trained language modeling is a key factor affecting base capabilities. FFN-Wider Transformers reduce the contribution ratio of this combination function, leading to a decline in base capabilities. We confirmed this by experiments and proposed Combination Enhancement Architecture (CEA) to address the decline in base capabilities of such models. Significantly, we extended our explanation and CEA to Mixture of Experts (MoE) architecture Transformers, which also alleviated their decline in base capabilities to some extent, proving our work can offer useful guidance for architecture analysis, architecture improvement and architecture design.

Title: SoK: Challenges and Opportunities in Federated Unlearning

• Authors: Hyejun Jeong, Shiqing Ma, Amir Houmansadr
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.02437
• Pdf URL: https://arxiv.org/pdf/2403.02437
• Copy Paste: [[2403.02437]] SoK: Challenges and Opportunities in Federated Unlearning(https://arxiv.org/abs/2403.02437)
• Keywords: privacy, federate
• Abstract: Federated learning (FL), introduced in 2017, facilitates collaborative learning between non-trusting parties with no need for the parties to explicitly share their data among themselves. This allows training models on user data while respecting privacy regulations such as GDPR and CPRA. However, emerging privacy requirements may mandate model owners to be able to \emph{forget} some learned data, e.g., when requested by data owners or law enforcement. This has given birth to an active field of research called \emph{machine unlearning}. In the context of FL, many techniques developed for unlearning in centralized settings are not trivially applicable! This is due to the unique differences between centralized and distributed learning, in particular, interactivity, stochasticity, heterogeneity, and limited accessibility in FL. In response, a recent line of work has focused on developing unlearning mechanisms tailored to FL. This SoK paper aims to take a deep look at the \emph{federated unlearning} literature, with the goal of identifying research trends and challenges in this emerging field. By carefully categorizing papers published on FL unlearning (since 2020), we aim to pinpoint the unique complexities of federated unlearning, highlighting limitations on directly applying centralized unlearning methods. We compare existing federated unlearning methods regarding influence removal and performance recovery, compare their threat models and assumptions, and discuss their implications and limitations. For instance, we analyze the experimental setup of FL unlearning studies from various perspectives, including data heterogeneity and its simulation, the datasets used for demonstration, and evaluation metrics. Our work aims to offer insights and suggestions for future research on federated unlearning.

Title: Anatomically Constrained Tractography of the Fetal Brain

• Authors: Camilo Calixto, Camilo Jaimes, Matheus D. Soldatelli, Simon K. Warfield, Ali Gholipour, Davood Karimi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02444
• Pdf URL: https://arxiv.org/pdf/2403.02444
• Copy Paste: [[2403.02444]] Anatomically Constrained Tractography of the Fetal Brain(https://arxiv.org/abs/2403.02444)
• Keywords: diffusion, segmentation
• Abstract: Diffusion-weighted Magnetic Resonance Imaging (dMRI) is increasingly used to study the fetal brain in utero. An important computation enabled by dMRI is streamline tractography, which has unique applications such as tract-specific analysis of the brain white matter and structural connectivity assessment. However, due to the low fetal dMRI data quality and the challenging nature of tractography, existing methods tend to produce highly inaccurate results. They generate many false streamlines while failing to reconstruct streamlines that constitute the major white matter tracts. In this paper, we advocate for anatomically constrained tractography based on an accurate segmentation of the fetal brain tissue directly in the dMRI space. We develop a deep learning method to compute the segmentation automatically. Experiments on independent test data show that this method can accurately segment the fetal brain tissue and drastically improve tractography results. It enables the reconstruction of highly curved tracts such as optic radiations. Importantly, our method infers the tissue segmentation and streamline propagation direction from a diffusion tensor fit to the dMRI data, making it applicable to routine fetal dMRI scans. The proposed method can lead to significant improvements in the accuracy and reproducibility of quantitative assessment of the fetal brain with dMRI.

Title: Free Proxies Unmasked: A Vulnerability and Longitudinal Analysis of Free Proxy Services

• Authors: Naif Mehanna (1, 2 and 3), Walter Rudametkin (4, 5 and 6), Pierre Laperdrix (2, 1 and 3), Antoine Vastel (7) ((1) University of Lille, (2) CNRS, (3) Inria Lille, (4) University of Rennes, (5) IRISA, (6) IUF, (7) Datadome)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02445
• Pdf URL: https://arxiv.org/pdf/2403.02445
• Copy Paste: [[2403.02445]] Free Proxies Unmasked: A Vulnerability and Longitudinal Analysis of Free Proxy Services(https://arxiv.org/abs/2403.02445)
• Keywords: security, privacy
• Abstract: Free-proxies have been widespread since the early days of the Web, helping users bypass geo-blocked content and conceal their IP addresses. Various proxy providers promise faster Internet or increased privacy while advertising their lists comprised of hundreds of readily available free proxies. However, while paid proxy services advertise the support of encrypted connections and high stability, free proxies often lack such guarantees, making them prone to malicious activities such as eavesdropping or modifying content. Furthermore, there is a market that encourages exploiting devices to install proxies. In this paper, we present a 30-month longitudinal study analyzing the stability, security, and potential manipulation of free web proxies that we collected from 11 providers. Our collection resulted in over 640,600 proxies, that we cumulatively tested daily. We find that only 34.5% of proxies were active at least once during our tests, showcasing the general instability of free proxies. Geographically, a majority of proxies originate from the US and China. Leveraging the Shodan search engine, we identified 4,452 distinct vulnerabilities on the proxies' IP addresses, including 1,755 vulnerabilities that allow unauthorized remote code execution and 2,036 that enable privilege escalation on the host device. Through the software analysis on the proxies' IP addresses, we find that 42,206 of them appear to run on MikroTik routers. Worryingly, we also discovered 16,923 proxies that manipulate content, indicating potential malicious intent by proxy owners. Ultimately, our research reveals that the use of free web proxies poses significant risks to users' privacy and security. The instability, vulnerabilities, and potential for malicious actions uncovered in our analysis lead us to strongly caution users against relying on free proxies.

Title: On Latency Predictors for Neural Architecture Search

• Authors: Yash Akhauri, Mohamed S. Abdelfattah
• Subjects: cs.LG, cs.AR, cs.CV, cs.PF
• Abstract URL: https://arxiv.org/abs/2403.02446
• Pdf URL: https://arxiv.org/pdf/2403.02446
• Copy Paste: [[2403.02446]] On Latency Predictors for Neural Architecture Search(https://arxiv.org/abs/2403.02446)
• Keywords: robust
• Abstract: Efficient deployment of neural networks (NN) requires the co-optimization of accuracy and latency. For example, hardware-aware neural architecture search has been used to automatically find NN architectures that satisfy a latency constraint on a specific hardware device. Central to these search algorithms is a prediction model that is designed to provide a hardware latency estimate for a candidate NN architecture. Recent research has shown that the sample efficiency of these predictive models can be greatly improved through pre-training on some \textit{training} devices with many samples, and then transferring the predictor on the \textit{test} (target) device. Transfer learning and meta-learning methods have been used for this, but often exhibit significant performance variability. Additionally, the evaluation of existing latency predictors has been largely done on hand-crafted training/test device sets, making it difficult to ascertain design features that compose a robust and general latency predictor. To address these issues, we introduce a comprehensive suite of latency prediction tasks obtained in a principled way through automated partitioning of hardware device sets. We then design a general latency predictor to comprehensively study (1) the predictor architecture, (2) NN sample selection methods, (3) hardware device representations, and (4) NN operation encoding schemes. Building on conclusions from our study, we present an end-to-end latency predictor training strategy that outperforms existing methods on 11 out of 12 difficult latency prediction tasks, improving latency prediction by 22.5\% on average, and up to to 87.6\% on the hardest tasks. Focusing on latency prediction, our HW-Aware NAS reports a $5.8\times$ speedup in wall-clock time. Our code is available on \href{https://github.com/abdelfattah-lab/nasflat_latency}{https://github.com/abdelfattah-lab/nasflat\_latency}.

Title: Cybersecurity competence of older adult users of mobile devices

• Authors: Simon Vrhovec, Igor Bernik, Damjan Fujs, Damjan Vavpotič
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02459
• Pdf URL: https://arxiv.org/pdf/2403.02459
• Copy Paste: [[2403.02459]] Cybersecurity competence of older adult users of mobile devices(https://arxiv.org/abs/2403.02459)
• Keywords: security
• Abstract: This work reports on a cross-sectional study on device proficiency, support availability and cybersecurity competence of older adult users of smartphones and/or tablets. Results indicate that cybersecurity competence is associated with both device proficiency and support availability although the variance explained is relatively low. There were no differences in cybersecurity competence between users and non-users of either mobile devices. Users of both smartphones and tablets had significantly higher device proficiency than non-users. Users of tablets had significantly higher support availability than non-users while there were no significant differences between users and non-users of smartphones.

Title: Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review

• Authors: Iryna Hartsock, Ghulam Rasool
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02469
• Pdf URL: https://arxiv.org/pdf/2403.02469
• Copy Paste: [[2403.02469]] Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review(https://arxiv.org/abs/2403.02469)
• Keywords: privacy
• Abstract: Medical vision-language models (VLMs) combine computer vision and natural language processing to analyze visual and textual medical data. Our paper reviews recent advancements in developing VLMs specialized for healthcare, focusing on models designed for medical report generation and visual question answering. We provide background on natural language processing and computer vision, explaining how techniques from both fields are integrated into VLMs to enable learning from multimodal data. Key areas we address include the exploration of medical vision-language datasets, in-depth analyses of architectures and pre-training strategies employed in recent noteworthy medical VLMs, and comprehensive discussion on evaluation metrics for assessing VLMs' performance in medical report generation and visual question answering. We also highlight current challenges and propose future directions, including enhancing clinical validity and addressing patient privacy concerns. Overall, our review summarizes recent progress in developing VLMs to harness multimodal medical data for improved healthcare applications.

Title: OffLanDat: A Community Based Implicit Offensive Language Dataset Generated by Large Language Model Through Prompt Engineering

• Authors: Amit Das, Mostafa Rahgouy, Dongji Feng, Zheng Zhang, Tathagata Bhattacharya, Nilanjana Raychawdhary, Mary Sandage, Lauramarie Pope, Gerry Dozier, Cheryl Seals
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02472
• Pdf URL: https://arxiv.org/pdf/2403.02472
• Copy Paste: [[2403.02472]] OffLanDat: A Community Based Implicit Offensive Language Dataset Generated by Large Language Model Through Prompt Engineering(https://arxiv.org/abs/2403.02472)
• Keywords: large language model
• Abstract: The widespread presence of offensive languages on social media has resulted in adverse effects on societal well-being. As a result, it has become very important to address this issue with high priority. Offensive languages exist in both explicit and implicit forms, with the latter being more challenging to detect. Current research in this domain encounters several challenges. Firstly, the existing datasets primarily rely on the collection of texts containing explicit offensive keywords, making it challenging to capture implicitly offensive contents that are devoid of these keywords. Secondly, usual methodologies tend to focus solely on textual analysis, neglecting the valuable insights that community information can provide. In this research paper, we introduce a novel dataset OffLanDat, a community based implicit offensive language dataset generated by ChatGPT containing data for 38 different target groups. Despite limitations in generating offensive texts using ChatGPT due to ethical constraints, we present a prompt-based approach that effectively generates implicit offensive languages. To ensure data quality, we evaluate our data with human. Additionally, we employ a prompt-based Zero-Shot method with ChatGPT and compare the detection results between human annotation and ChatGPT annotation. We utilize existing state-of-the-art models to see how effective they are in detecting such languages. We will make our code and dataset public for other researchers.

Title: When do Convolutional Neural Networks Stop Learning?

• Authors: Sahan Ahmad, Gabriel Trahan, Aminul Islam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02473
• Pdf URL: https://arxiv.org/pdf/2403.02473
• Copy Paste: [[2403.02473]] When do Convolutional Neural Networks Stop Learning?(https://arxiv.org/abs/2403.02473)
• Keywords: segmentation
• Abstract: Convolutional Neural Networks (CNNs) have demonstrated outstanding performance in computer vision tasks such as image classification, detection, segmentation, and medical image analysis. In general, an arbitrary number of epochs is used to train such neural networks. In a single epoch, the entire training data -- divided by batch size -- are fed to the network. In practice, validation error with training loss is used to estimate the neural network's generalization, which indicates the optimal learning capacity of the network. Current practice is to stop training when the training loss decreases and the gap between training and validation error increases (i.e., the generalization gap) to avoid overfitting. However, this is a trial-and-error-based approach which raises a critical question: Is it possible to estimate when neural networks stop learning based on training data? This research work introduces a hypothesis that analyzes the data variation across all the layers of a CNN variant to anticipate its near-optimal learning capacity. In the training phase, we use our hypothesis to anticipate the near-optimal learning capacity of a CNN variant without using any validation data. Our hypothesis can be deployed as a plug-and-play to any existing CNN variant without introducing additional trainable parameters to the network. We test our hypothesis on six different CNN variants and three different general image datasets (CIFAR10, CIFAR100, and SVHN). The result based on these CNN variants and datasets shows that our hypothesis saves 58.49\% of computational time (on average) in training. We further conduct our hypothesis on ten medical image datasets and compared with the MedMNIST-V2 benchmark. Based on our experimental result, we save $\approx$ 44.1\% of computational time without losing accuracy against the MedMNIST-V2 benchmark.

Title: Enhancing LLM Safety via Constrained Direct Preference Optimization

• Authors: Zixuan Liu, Xiaolin Sun, Zizhan Zheng
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02475
• Pdf URL: https://arxiv.org/pdf/2403.02475
• Copy Paste: [[2403.02475]] Enhancing LLM Safety via Constrained Direct Preference Optimization(https://arxiv.org/abs/2403.02475)
• Keywords: large language model
• Abstract: The rapidly increasing capabilities of large language models (LLMs) raise an urgent need to align AI systems with diverse human preferences to simultaneously enhance their usefulness and safety, despite the often conflicting nature of these goals. To address this important problem, a promising approach is to enforce a safety constraint at the fine-tuning stage through a constrained Reinforcement Learning from Human Feedback (RLHF) framework. This approach, however, is computationally expensive and often unstable. In this work, we introduce Constrained DPO (C-DPO), a novel extension of the recently proposed Direct Preference Optimization (DPO) approach for fine-tuning LLMs that is both efficient and lightweight. By integrating dual gradient descent and DPO, our method identifies a nearly optimal trade-off between helpfulness and harmlessness without using reinforcement learning. Empirically, our approach provides a safety guarantee to LLMs that is missing in DPO while achieving significantly higher rewards under the same safety constraint compared to a recently proposed safe RLHF approach. Warning: This paper contains example data that may be offensive or harmful.

Title: A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation

• Authors: Aritra Mitra
• Subjects: cs.LG, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2403.02476
• Pdf URL: https://arxiv.org/pdf/2403.02476
• Copy Paste: [[2403.02476]] A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation(https://arxiv.org/abs/2403.02476)
• Keywords: fair
• Abstract: We study the finite-time convergence of TD learning with linear function approximation under Markovian sampling. Existing proofs for this setting either assume a projection step in the algorithm to simplify the analysis, or require a fairly intricate argument to ensure stability of the iterates. We ask: \textit{Is it possible to retain the simplicity of a projection-based analysis without actually performing a projection step in the algorithm?} Our main contribution is to show this is possible via a novel two-step argument. In the first step, we use induction to prove that under a standard choice of a constant step-size $\alpha$, the iterates generated by TD learning remain uniformly bounded in expectation. In the second step, we establish a recursion that mimics the steady-state dynamics of TD learning up to a bounded perturbation on the order of $O(\alpha^2)$ that captures the effect of Markovian sampling. Combining these pieces leads to an overall approach that considerably simplifies existing proofs. We conjecture that our inductive proof technique will find applications in the analyses of more complex stochastic approximation algorithms, and conclude by providing some examples of such applications.

Title: Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents

• Authors: Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, Bill Yuchen Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02502
• Pdf URL: https://arxiv.org/pdf/2403.02502
• Copy Paste: [[2403.02502]] Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents(https://arxiv.org/abs/2403.02502)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have become integral components in various autonomous agent systems. In this study, we present an exploration-based trajectory optimization approach, referred to as ETO. This learning method is designed to enhance the performance of open LLM agents. Contrary to previous studies that exclusively train on successful expert trajectories, our method allows agents to learn from their exploration failures. This leads to improved performance through an iterative optimization framework. During the exploration phase, the agent interacts with the environment while completing given tasks, gathering failure trajectories to create contrastive trajectory pairs. In the subsequent training phase, the agent utilizes these trajectory preference pairs to update its policy using contrastive learning methods like DPO. This iterative cycle of exploration and training fosters continued improvement in the agents. Our experiments on three complex tasks demonstrate that ETO consistently surpasses baseline performance by a large margin. Furthermore, an examination of task-solving efficiency and potential in scenarios lacking expert trajectory underscores the effectiveness of our approach.

Title: Differentially Private Representation Learning via Image Captioning

• Authors: Tom Sander, Yaodong Yu, Maziar Sanjabi, Alain Durmus, Yi Ma, Kamalika Chaudhuri, Chuan Guo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02506
• Pdf URL: https://arxiv.org/pdf/2403.02506
• Copy Paste: [[2403.02506]] Differentially Private Representation Learning via Image Captioning(https://arxiv.org/abs/2403.02506)
• Keywords: privacy
• Abstract: Differentially private (DP) machine learning is considered the gold-standard solution for training a model from sensitive data while still preserving privacy. However, a major barrier to achieving this ideal is its sub-optimal privacy-accuracy trade-off, which is particularly visible in DP representation learning. Specifically, it has been shown that under modest privacy budgets, most models learn representations that are not significantly better than hand-crafted features. In this work, we show that effective DP representation learning can be done via image captioning and scaling up to internet-scale multimodal datasets. Through a series of engineering tricks, we successfully train a DP image captioner (DP-Cap) on a 233M subset of LAION-2B from scratch using a reasonable amount of computation, and obtaining unprecedented high-quality image features that can be used in a variety of downstream vision and vision-language tasks. For example, under a privacy budget of $\varepsilon=8$, a linear classifier trained on top of learned DP-Cap features attains 65.8% accuracy on ImageNet-1K, considerably improving the previous SOTA of 56.5%. Our work challenges the prevailing sentiment that high-utility DP representation learning cannot be achieved by training from scratch.

Title: SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models

• Authors: Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, Kamalika Das
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02509
• Pdf URL: https://arxiv.org/pdf/2403.02509
• Copy Paste: [[2403.02509]] SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models(https://arxiv.org/abs/2403.02509)
• Keywords: large language model
• Abstract: In recent years, large language models (LLMs) have become increasingly prevalent, offering remarkable text generation capabilities. However, a pressing challenge is their tendency to make confidently wrong predictions, highlighting the critical need for uncertainty quantification (UQ) in LLMs. While previous works have mainly focused on addressing aleatoric uncertainty, the full spectrum of uncertainties, including epistemic, remains inadequately explored. Motivated by this gap, we introduce a novel UQ method, sampling with perturbation for UQ (SPUQ), designed to tackle both aleatoric and epistemic uncertainties. The method entails generating a set of perturbations for LLM inputs, sampling outputs for each perturbation, and incorporating an aggregation module that generalizes the sampling uncertainty approach for text generation tasks. Through extensive experiments on various datasets, we investigated different perturbation and aggregation techniques. Our findings show a substantial improvement in model uncertainty calibration, with a reduction in Expected Calibration Error (ECE) by 50\% on average. Our findings suggest that our proposed UQ method offers promising steps toward enhancing the reliability and trustworthiness of LLMs.

Title: Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF

• Authors: Chen Zheng, Ke Sun, Hang Wu, Chenguang Xi, Xun Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02513
• Pdf URL: https://arxiv.org/pdf/2403.02513
• Copy Paste: [[2403.02513]] Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF(https://arxiv.org/abs/2403.02513)
• Keywords: large language model
• Abstract: In recent advancements in Conversational Large Language Models (LLMs), a concerning trend has emerged, showing that many new base LLMs experience a knowledge reduction in their foundational capabilities following Supervised Fine-Tuning (SFT). This process often leads to issues such as forgetting or a decrease in the base model's abilities. Moreover, fine-tuned models struggle to align with user preferences, inadvertently increasing the generation of toxic outputs when specifically prompted. To overcome these challenges, we adopted an innovative approach by completely bypassing SFT and directly implementing Harmless Reinforcement Learning from Human Feedback (RLHF). Our method not only preserves the base model's general capabilities but also significantly enhances its conversational abilities, while notably reducing the generation of toxic outputs. Our approach holds significant implications for fields that demand a nuanced understanding and generation of responses, such as customer service. We applied this methodology to Mistral, the most popular base model, thereby creating Mistral-Plus. Our validation across 11 general tasks demonstrates that Mistral-Plus outperforms similarly sized open-source base models and their corresponding instruct versions. Importantly, the conversational abilities of Mistral-Plus were significantly improved, indicating a substantial advancement over traditional SFT models in both safety and user preference alignment.

Title: Wukong: Towards a Scaling Law for Large-Scale Recommendation

• Authors: Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, Guna Lakshminarayanan, Ellie Dingqiao Wen, Jongsoo Park, Maxim Naumov, Wenlin Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02545
• Pdf URL: https://arxiv.org/pdf/2403.02545
• Copy Paste: [[2403.02545]] Wukong: Towards a Scaling Law for Large-Scale Recommendation(https://arxiv.org/abs/2403.02545)
• Keywords: large language model
• Abstract: Scaling laws play an instrumental role in the sustainable improvement in model quality. Unfortunately, recommendation models to date do not exhibit such laws similar to those observed in the domain of large language models, due to the inefficiencies of their upscaling mechanisms. This limitation poses significant challenges in adapting these models to increasingly more complex real-world datasets. In this paper, we propose an effective network architecture based purely on stacked factorization machines, and a synergistic upscaling strategy, collectively dubbed Wukong, to establish a scaling law in the domain of recommendation. Wukong's unique design makes it possible to capture diverse, any-order of interactions simply through taller and wider layers. We conducted extensive evaluations on six public datasets, and our results demonstrate that Wukong consistently outperforms state-of-the-art models quality-wise. Further, we assessed Wukong's scalability on an internal, large-scale dataset. The results show that Wukong retains its superiority in quality over state-of-the-art models, while holding the scaling law across two orders of magnitude in model complexity, extending beyond 100 Gflop or equivalently up to GPT-3/LLaMa-2 scale of total training compute, where prior arts fall short.

Title: Catch'em all: Classification of Rare, Prominent, and Novel Malware Families

• Authors: Maksim E. Eren, Ryan Barron, Manish Bhattarai, Selma Wanna, Nicholas Solovyev, Kim Rasmussen, Boian S. Alexandrov, Charles Nicholas
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02546
• Pdf URL: https://arxiv.org/pdf/2403.02546
• Copy Paste: [[2403.02546]] Catch'em all: Classification of Rare, Prominent, and Novel Malware Families(https://arxiv.org/abs/2403.02546)
• Keywords: security, extraction
• Abstract: National security is threatened by malware, which remains one of the most dangerous and costly cyber threats. As of last year, researchers reported 1.3 billion known malware specimens, motivating the use of data-driven machine learning (ML) methods for analysis. However, shortcomings in existing ML approaches hinder their mass adoption. These challenges include detection of novel malware and the ability to perform malware classification in the face of class imbalance: a situation where malware families are not equally represented in the data. Our work addresses these shortcomings with MalwareDNA: an advanced dimensionality reduction and feature extraction framework. We demonstrate stable task performance under class imbalance for the following tasks: malware family classification and novel malware detection with a trade-off in increased abstention or reject-option rate.

Title: Updating the Minimum Information about CLinical Artificial Intelligence (MI-CLAIM) checklist for generative modeling research

• Authors: Brenda Y. Miao, Irene Y. Chen, Christopher YK Williams, Jaysón Davidson, Augusto Garcia-Agundez, Harry Sun, Travis Zack, Atul J. Butte, Madhumita Sushil
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02558
• Pdf URL: https://arxiv.org/pdf/2403.02558
• Copy Paste: [[2403.02558]] Updating the Minimum Information about CLinical Artificial Intelligence (MI-CLAIM) checklist for generative modeling research(https://arxiv.org/abs/2403.02558)
• Keywords: interpretability, diffusion, generative, large language model
• Abstract: Recent advances in generative models, including large language models (LLMs), vision language models (VLMs), and diffusion models, have accelerated the field of natural language and image processing in medicine and marked a significant paradigm shift in how biomedical models can be developed and deployed. While these models are highly adaptable to new tasks, scaling and evaluating their usage presents new challenges not addressed in previous frameworks. In particular, the ability of these models to produce useful outputs with little to no specialized training data ("zero-" or "few-shot" approaches), as well as the open-ended nature of their outputs, necessitate the development of updated guidelines in using and evaluating these models. In response to gaps in standards and best practices for the development of clinical AI tools identified by US Executive Order 141103 and several emerging national networks for clinical AI evaluation, we begin to formalize some of these guidelines by building on the "Minimum information about clinical artificial intelligence modeling" (MI-CLAIM) checklist. The MI-CLAIM checklist, originally developed in 2020, provided a set of six steps with guidelines on the minimum information necessary to encourage transparent, reproducible research for artificial intelligence (AI) in medicine. Here, we propose modifications to the original checklist that highlight differences in training, evaluation, interpretability, and reproducibility of generative models compared to traditional AI models for clinical research. This updated checklist also seeks to clarify cohort selection reporting and adds additional items on alignment with ethical standards.

Title: Semantic Human Mesh Reconstruction with Textures

• Authors: Xiaoyu Zhan, Jianxin Yang, Yuanqi Li, Jie Guo, Yanwen Guo, Wenping Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02561
• Pdf URL: https://arxiv.org/pdf/2403.02561
• Copy Paste: [[2403.02561]] Semantic Human Mesh Reconstruction with Textures(https://arxiv.org/abs/2403.02561)
• Keywords: robust, diffusion
• Abstract: The field of 3D detailed human mesh reconstruction has made significant progress in recent years. However, current methods still face challenges when used in industrial applications due to unstable results, low-quality meshes, and a lack of UV unwrapping and skinning weights. In this paper, we present SHERT, a novel pipeline that can reconstruct semantic human meshes with textures and high-precision details. SHERT applies semantic- and normal-based sampling between the detailed surface (eg mesh and SDF) and the corresponding SMPL-X model to obtain a partially sampled semantic mesh and then generates the complete semantic mesh by our specifically designed self-supervised completion and refinement networks. Using the complete semantic mesh as a basis, we employ a texture diffusion model to create human textures that are driven by both images and texts. Our reconstructed meshes have stable UV unwrapping, high-quality triangle meshes, and consistent semantic information. The given SMPL-X model provides semantic information and shape priors, allowing SHERT to perform well even with incorrect and incomplete inputs. The semantic information also makes it easy to substitute and animate different body parts such as the face, body, and hands. Quantitative and qualitative experiments demonstrate that SHERT is capable of producing high-fidelity and robust semantic meshes that outperform state-of-the-art methods.

Title: Eliciting Better Multilingual Structured Reasoning from LLMs through Code

• Authors: Bryan Li, Tamer Alkhouli, Daniele Bonadiman, Nikolaos Pappas, Saab Mansour
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02567
• Pdf URL: https://arxiv.org/pdf/2403.02567
• Copy Paste: [[2403.02567]] Eliciting Better Multilingual Structured Reasoning from LLMs through Code(https://arxiv.org/abs/2403.02567)
• Keywords: large language model
• Abstract: Development of large language models (LLM) have shown progress on reasoning, though studies have been limited to English or simple reasoning tasks. We thus introduce a multilingual structured reasoning and explanation dataset, termed xSTREET, that covers four tasks across six languages. xSTREET exposes a gap in base LLM performance between English and non-English reasoning tasks. We then propose two methods to remedy this gap, building on the insight that LLMs trained on code are better reasoners. First, at training time, we augment a code dataset with multi-lingual comments using machine translation while keeping program code as-is. Second, at inference time, we bridge the gap between training and inference by employing a prompt structure that incorporates step-by-step code primitives to derive new facts and find a solution. Our methods show improved multilingual performance on xSTREET, most notably on the scientific commonsense reasoning subtask. Furthermore, the models show no regression on non-reasoning tasks, thus showing our techniques maintain general-purpose abilities.

Title: DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-training

• Authors: Zihao Wang, Rui Zhu, Dongruo Zhou, Zhikun Zhang, John Mitchell, Haixu Tang, XiaoFeng Wang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02571
• Pdf URL: https://arxiv.org/pdf/2403.02571
• Copy Paste: [[2403.02571]] DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-training(https://arxiv.org/abs/2403.02571)
• Keywords: privacy, robust
• Abstract: Recent developments have underscored the critical role of \textit{differential privacy} (DP) in safeguarding individual data for training machine learning models. However, integrating DP oftentimes incurs significant model performance degradation due to the perturbation introduced into the training process, presenting a formidable challenge in the {differentially private machine learning} (DPML) field. To this end, several mitigative efforts have been proposed, typically revolving around formulating new DPML algorithms or relaxing DP definitions to harmonize with distinct contexts. In spite of these initiatives, the diminishment induced by DP on models, particularly large-scale models, remains substantial and thus, necessitates an innovative solution that adeptly circumnavigates the consequential impairment of model utility. In response, we introduce DPAdapter, a pioneering technique designed to amplify the model performance of DPML algorithms by enhancing parameter robustness. The fundamental intuition behind this strategy is that models with robust parameters are inherently more resistant to the noise introduced by DP, thereby retaining better performance despite the perturbations. DPAdapter modifies and enhances the sharpness-aware minimization (SAM) technique, utilizing a two-batch strategy to provide a more accurate perturbation estimate and an efficient gradient descent, thereby improving parameter robustness against noise. Notably, DPAdapter can act as a plug-and-play component and be combined with existing DPML algorithms to further improve their performance. Our experiments show that DPAdapter vastly enhances state-of-the-art DPML algorithms, increasing average accuracy from 72.92\% to 77.09\% with a privacy budget of $\epsilon=4$.

Title: Learning-augmented Online Minimization of Age of Information and Transmission Costs

• Authors: Zhongdong Liu, Keyuan Zhang, Bin Li, Yin Sun, Y. Thomas Hou, Bo Ji
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02573
• Pdf URL: https://arxiv.org/pdf/2403.02573
• Copy Paste: [[2403.02573]] Learning-augmented Online Minimization of Age of Information and Transmission Costs(https://arxiv.org/abs/2403.02573)
• Keywords: robust
• Abstract: We consider a discrete-time system where a resource-constrained source (e.g., a small sensor) transmits its time-sensitive data to a destination over a time-varying wireless channel. Each transmission incurs a fixed transmission cost (e.g., energy cost), and no transmission results in a staleness cost represented by the Age-of-Information. The source must balance the tradeoff between transmission and staleness costs. To address this challenge, we develop a robust online algorithm to minimize the sum of transmission and staleness costs, ensuring a worst-case performance guarantee. While online algorithms are robust, they are usually overly conservative and may have a poor average performance in typical scenarios. In contrast, by leveraging historical data and prediction models, machine learning (ML) algorithms perform well in average cases. However, they typically lack worst-case performance guarantees. To achieve the best of both worlds, we design a learning-augmented online algorithm that exhibits two desired properties: (i) consistency: closely approximating the optimal offline algorithm when the ML prediction is accurate and trusted; (ii) robustness: ensuring worst-case performance guarantee even ML predictions are inaccurate. Finally, we perform extensive simulations to show that our online algorithm performs well empirically and that our learning-augmented algorithm achieves both consistency and robustness.

Title: Improving Event Definition Following For Zero-Shot Event Detection

• Authors: Zefan Cai, Po-Nien Kung, Ashima Suvarna, Mingyu Derek Ma, Hritik Bansal, Baobao Chang, P. Jeffrey Brantingham, Wei Wang, Nanyun Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02586
• Pdf URL: https://arxiv.org/pdf/2403.02586
• Copy Paste: [[2403.02586]] Improving Event Definition Following For Zero-Shot Event Detection(https://arxiv.org/abs/2403.02586)
• Keywords: extraction, large language model
• Abstract: Existing approaches on zero-shot event detection usually train models on datasets annotated with known event types, and prompt them with unseen event definitions. These approaches yield sporadic successes, yet generally fall short of expectations. In this work, we aim to improve zero-shot event detection by training models to better follow event definitions. We hypothesize that a diverse set of event types and definitions are the key for models to learn to follow event definitions while existing event extraction datasets focus on annotating many high-quality examples for a few event types. To verify our hypothesis, we construct an automatically generated Diverse Event Definition (DivED) dataset and conduct comparative studies. Our experiments reveal that a large number of event types (200) and diverse event definitions can significantly boost event extraction performance; on the other hand, the performance does not scale with over ten examples per event type. Beyond scaling, we incorporate event ontology information and hard-negative samples during training, further boosting the performance. Based on these findings, we fine-tuned a LLaMA-2-7B model on our DivED dataset, yielding performance that surpasses SOTA large language models like GPT-3.5 across three open benchmarks on zero-shot event detection.

Title: A Unified Framework for Microscopy Defocus Deblur with Multi-Pyramid Transformer and Contrastive Learning

• Authors: Yuelin Zhang, Pengyu Zheng, Wanquan Yan, Chengyu Fang, Shing Shin Cheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02611
• Pdf URL: https://arxiv.org/pdf/2403.02611
• Copy Paste: [[2403.02611]] A Unified Framework for Microscopy Defocus Deblur with Multi-Pyramid Transformer and Contrastive Learning(https://arxiv.org/abs/2403.02611)
• Keywords: transformer
• Abstract: Defocus blur is a persistent problem in microscope imaging that poses harm to pathology interpretation and medical intervention in cell microscopy and microscope surgery. To address this problem, a unified framework including multi-pyramid transformer (MPT) and extended frequency contrastive regularization (EFCR) is proposed to tackle two outstanding challenges in microscopy deblur: longer attention span and feature deficiency. The MPT employs an explicit pyramid structure at each network stage that integrates the cross-scale window attention (CSWA), the intra-scale channel attention (ISCA), and the feature-enhancing feed-forward network (FEFN) to capture long-range cross-scale spatial interaction and global channel context. The EFCR addresses the feature deficiency problem by exploring latent deblur signals from different frequency bands. It also enables deblur knowledge transfer to learn cross-domain information from extra data, improving deblur performance for labeled and unlabeled data. Extensive experiments and downstream task validation show the framework achieves state-of-the-art performance across multiple datasets. Project page: https://github.com/PieceZhang/MPT-CataBlur.

Title: Exploring the Limitations of Large Language Models in Compositional Relation Reasoning

• Authors: Jinman Zhao, Xueyan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02615
• Pdf URL: https://arxiv.org/pdf/2403.02615
• Copy Paste: [[2403.02615]] Exploring the Limitations of Large Language Models in Compositional Relation Reasoning(https://arxiv.org/abs/2403.02615)
• Keywords: robust, large language model
• Abstract: We present a comprehensive evaluation of large language models(LLMs)' ability to reason about composition relations through a benchmark encompassing 1,500 test cases in English, designed to cover six distinct types of composition relations: Positional, Comparative, Personal, Mathematical, Identity, and Other. Acknowledging the significance of multilingual capabilities, we expanded our assessment to include translations of these cases into Chinese, Japanese, French, and Korean. Our Multilingual Composition Relation (MCR) benchmark aims at investigating the robustness and adaptability of LLMs in handling composition relation reasoning across diverse linguistic contexts.

Title: Unsupervised Spatio-Temporal State Estimation for Fine-grained Adaptive Anomaly Diagnosis of Industrial Cyber-physical Systems

• Authors: Haili Sun, Yan Huang, Lansheng Han, Cai Fu, Chunjie Zhou
• Subjects: cs.LG, cs.AI, cs.CR, cs.NI, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.02616
• Pdf URL: https://arxiv.org/pdf/2403.02616
• Copy Paste: [[2403.02616]] Unsupervised Spatio-Temporal State Estimation for Fine-grained Adaptive Anomaly Diagnosis of Industrial Cyber-physical Systems(https://arxiv.org/abs/2403.02616)
• Keywords: attack, robust, transformer
• Abstract: Accurate detection and diagnosis of abnormal behaviors such as network attacks from multivariate time series (MTS) are crucial for ensuring the stable and effective operation of industrial cyber-physical systems (CPS). However, existing researches pay little attention to the logical dependencies among system working states, and have difficulties in explaining the evolution mechanisms of abnormal signals. To reveal the spatio-temporal association relationships and evolution mechanisms of the working states of industrial CPS, this paper proposes a fine-grained adaptive anomaly diagnosis method (i.e. MAD-Transformer) to identify and diagnose anomalies in MTS. MAD-Transformer first constructs a temporal state matrix to characterize and estimate the change patterns of the system states in the temporal dimension. Then, to better locate the anomalies, a spatial state matrix is also constructed to capture the inter-sensor state correlation relationships within the system. Subsequently, based on these two types of state matrices, a three-branch structure of series-temporal-spatial attention module is designed to simultaneously capture the series, temporal, and space dependencies among MTS. Afterwards, three associated alignment loss functions and a reconstruction loss are constructed to jointly optimize the model. Finally, anomalies are determined and diagnosed by comparing the residual matrices with the original matrices. We conducted comparative experiments on five publicly datasets spanning three application domains (service monitoring, spatial and earth exploration, and water treatment), along with a petroleum refining simulation dataset collected by ourselves. The results demonstrate that MAD-Transformer can adaptively detect fine-grained anomalies with short duration, and outperforms the state-of-the-art baselines in terms of noise robustness and localization performance.

Title: Training Machine Learning models at the Edge: A Survey

• Authors: Aymen Rayane Khouas, Mohamed Reda Bouadjenek, Hakim Hacid, Sunil Aryal
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.02619
• Pdf URL: https://arxiv.org/pdf/2403.02619
• Copy Paste: [[2403.02619]] Training Machine Learning models at the Edge: A Survey(https://arxiv.org/abs/2403.02619)
• Keywords: federate
• Abstract: Edge Computing (EC) has gained significant traction in recent years, promising enhanced efficiency by integrating Artificial Intelligence (AI) capabilities at the edge. While the focus has primarily been on the deployment and inference of Machine Learning (ML) models at the edge, the training aspect remains less explored. This survey delves into Edge Learning (EL), specifically the optimization of ML model training at the edge. The objective is to comprehensively explore diverse approaches and methodologies in EL, synthesize existing knowledge, identify challenges, and highlight future trends. Utilizing Scopus' advanced search, relevant literature on EL was identified, revealing a concentration of research efforts in distributed learning methods, particularly Federated Learning (FL). This survey further provides a guideline for comparing techniques used to optimize ML for edge learning, along with an exploration of different frameworks, libraries, and simulation tools available for EL. In doing so, the paper contributes to a holistic understanding of the current landscape and future directions in the intersection of edge computing and machine learning, paving the way for informed comparisons between optimization methods and techniques designed for edge learning.

Title: Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use

• Authors: Imad Eddine Toubal, Aditya Avinash, Neil Gordon Alldrin, Jan Dlabal, Wenlei Zhou, Enming Luo, Otilia Stretcu, Hao Xiong, Chun-Ta Lu, Howard Zhou, Ranjay Krishna, Ariel Fuxman, Tom Duerig
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02626
• Pdf URL: https://arxiv.org/pdf/2403.02626
• Copy Paste: [[2403.02626]] Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use(https://arxiv.org/abs/2403.02626)
• Keywords: large language model
• Abstract: From content moderation to wildlife conservation, the number of applications that require models to recognize nuanced or subjective visual concepts is growing. Traditionally, developing classifiers for such concepts requires substantial manual effort measured in hours, days, or even months to identify and annotate data needed for training. Even with recently proposed Agile Modeling techniques, which enable rapid bootstrapping of image classifiers, users are still required to spend 30 minutes or more of monotonous, repetitive data labeling just to train a single classifier. Drawing on Fiske's Cognitive Miser theory, we propose a new framework that alleviates manual effort by replacing human labeling with natural language interactions, reducing the total effort required to define a concept by an order of magnitude: from labeling 2,000 images to only 100 plus some natural language interactions. Our framework leverages recent advances in foundation models, both large language models and vision-language models, to carve out the concept space through conversation and by automatically labeling training data points. Most importantly, our framework eliminates the need for crowd-sourced annotations. Moreover, our framework ultimately produces lightweight classification models that are deployable in cost-sensitive scenarios. Across 15 subjective concepts and across 2 public image classification datasets, our trained models outperform traditional Agile Modeling as well as state-of-the-art zero-shot classification models like ALIGN, CLIP, CuPL, and large visual question-answering models like PaLI-X.

Title: Interactive Continual Learning: Fast and Slow Thinking

• Authors: Biqing Qi, Xingquan Chen, Junqi Gao, Jianxing Liu, Ligang Wu, Bowen Zhou
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02628
• Pdf URL: https://arxiv.org/pdf/2403.02628
• Copy Paste: [[2403.02628]] Interactive Continual Learning: Fast and Slow Thinking(https://arxiv.org/abs/2403.02628)
• Keywords: large language model
• Abstract: Advanced life forms, sustained by the synergistic interaction of neural cognitive mechanisms, continually acquire and transfer knowledge throughout their lifespan. In contrast, contemporary machine learning paradigms exhibit limitations in emulating the facets of continual learning (CL). Nonetheless, the emergence of large language models (LLMs) presents promising avenues for realizing CL via interactions with these models. Drawing on Complementary Learning System theory, this paper presents a novel Interactive Continual Learning (ICL) framework, enabled by collaborative interactions among models of various sizes. Specifically, we assign the ViT model as System1 and multimodal LLM as System2. To enable the memory module to deduce tasks from class information and enhance Set2Set retrieval, we propose the Class-Knowledge-Task Multi-Head Attention (CKT-MHA). Additionally, to improve memory retrieval in System1 through enhanced geometric representation, we introduce the CL-vMF mechanism, based on the von Mises-Fisher (vMF) distribution. Meanwhile, we introduce the von Mises-Fisher Outlier Detection and Interaction (vMF-ODI) strategy to identify hard examples, thus enhancing collaboration between System1 and System2 for complex reasoning realization. Comprehensive evaluation of our proposed ICL demonstrates significant resistance to forgetting and superior performance relative to existing methods.

Title: FedHCDR: Federated Cross-Domain Recommendation with Hypergraph Signal Decoupling

• Authors: Hongyu Zhang, Dongyi Zheng, Lin Zhong, Xu Yang, Jiyuan Feng, Yunqing Feng, Qing Liao
• Subjects: cs.LG, cs.IR, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.02630
• Pdf URL: https://arxiv.org/pdf/2403.02630
• Copy Paste: [[2403.02630]] FedHCDR: Federated Cross-Domain Recommendation with Hypergraph Signal Decoupling(https://arxiv.org/abs/2403.02630)
• Keywords: protect, federate
• Abstract: In recent years, Cross-Domain Recommendation (CDR) has drawn significant attention, which utilizes user data from multiple domains to enhance the recommendation performance. However, current CDR methods require sharing user data across domains, thereby violating the General Data Protection Regulation (GDPR). Consequently, numerous approaches have been proposed for Federated Cross-Domain Recommendation (FedCDR). Nevertheless, the data heterogeneity across different domains inevitably influences the overall performance of federated learning. In this study, we propose FedHCDR, a novel Federated Cross-Domain Recommendation framework with Hypergraph signal decoupling. Specifically, to address the data heterogeneity across domains, we introduce an approach called hypergraph signal decoupling (HSD) to decouple the user features into domain-exclusive and domain-shared features. The approach employs high-pass and low-pass hypergraph filters to decouple domain-exclusive and domain-shared user representations, which are trained by the local-global bi-directional transfer algorithm. In addition, a hypergraph contrastive learning (HCL) module is devised to enhance the learning of domain-shared user relationship information by perturbing the user hypergraph. Extensive experiments conducted on three real-world scenarios demonstrate that FedHCDR outperforms existing baselines significantly.

Title: BSDP: Brain-inspired Streaming Dual-level Perturbations for Online Open World Object Detection

• Authors: Yu Chen, Liyan Ma, Liping Jing, Jian Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02637
• Pdf URL: https://arxiv.org/pdf/2403.02637
• Copy Paste: [[2403.02637]] BSDP: Brain-inspired Streaming Dual-level Perturbations for Online Open World Object Detection(https://arxiv.org/abs/2403.02637)
• Keywords: robust
• Abstract: Humans can easily distinguish the known and unknown categories and can recognize the unknown object by learning it once instead of repeating it many times without forgetting the learned object. Hence, we aim to make deep learning models simulate the way people learn. We refer to such a learning manner as OnLine Open World Object Detection(OLOWOD). Existing OWOD approaches pay more attention to the identification of unknown categories, while the incremental learning part is also very important. Besides, some neuroscience research shows that specific noises allow the brain to form new connections and neural pathways which may improve learning speed and efficiency. In this paper, we take the dual-level information of old samples as perturbations on new samples to make the model good at learning new knowledge without forgetting the old knowledge. Therefore, we propose a simple plug-and-play method, called Brain-inspired Streaming Dual-level Perturbations(BSDP), to solve the OLOWOD problem. Specifically, (1) we first calculate the prototypes of previous categories and use the distance between samples and the prototypes as the sample selecting strategy to choose old samples for replay; (2) then take the prototypes as the streaming feature-level perturbations of new samples, so as to improve the plasticity of the model through revisiting the old knowledge; (3) and also use the distribution of the features of the old category samples to generate adversarial data in the form of streams as the data-level perturbations to enhance the robustness of the model to new categories. We empirically evaluate BSDP on PASCAL VOC and MS-COCO, and the excellent results demonstrate the promising performance of our proposed method and learning manner.

Title: FinReport: Explainable Stock Earnings Forecasting via News Factor Analyzing Model

• Authors: Xiangyu Li, Xinjie Shen, Yawen Zeng, Xiaofen Xing, Jin Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02647
• Pdf URL: https://arxiv.org/pdf/2403.02647
• Copy Paste: [[2403.02647]] FinReport: Explainable Stock Earnings Forecasting via News Factor Analyzing Model(https://arxiv.org/abs/2403.02647)
• Keywords: explainability, large language model
• Abstract: The task of stock earnings forecasting has received considerable attention due to the demand investors in real-world scenarios. However, compared with financial institutions, it is not easy for ordinary investors to mine factors and analyze news. On the other hand, although large language models in the financial field can serve users in the form of dialogue robots, it still requires users to have financial knowledge to ask reasonable questions. To serve the user experience, we aim to build an automatic system, FinReport, for ordinary investors to collect information, analyze it, and generate reports after summarizing. Specifically, our FinReport is based on financial news announcements and a multi-factor model to ensure the professionalism of the report. The FinReport consists of three modules: news factorization module, return forecasting module, risk assessment module. The news factorization module involves understanding news information and combining it with stock factors, the return forecasting module aim to analysis the impact of news on market sentiment, and the risk assessment module is adopted to control investment risk. Extensive experiments on real-world datasets have well verified the effectiveness and explainability of our proposed FinReport. Our codes and datasets are available at https://github.com/frinkleko/FinReport.

Title: Few-shot Learner Parameterization by Diffusion Time-steps

• Authors: Zhongqi Yue, Pan Zhou, Richang Hong, Hanwang Zhang, Qianru Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02649
• Pdf URL: https://arxiv.org/pdf/2403.02649
• Copy Paste: [[2403.02649]] Few-shot Learner Parameterization by Diffusion Time-steps(https://arxiv.org/abs/2403.02649)
• Keywords: diffusion
• Abstract: Even when using large multi-modal foundation models, few-shot learning is still challenging -- if there is no proper inductive bias, it is nearly impossible to keep the nuanced class attributes while removing the visually prominent attributes that spuriously correlate with class labels. To this end, we find an inductive bias that the time-steps of a Diffusion Model (DM) can isolate the nuanced class attributes, i.e., as the forward diffusion adds noise to an image at each time-step, nuanced attributes are usually lost at an earlier time-step than the spurious attributes that are visually prominent. Building on this, we propose Time-step Few-shot (TiF) learner. We train class-specific low-rank adapters for a text-conditioned DM to make up for the lost attributes, such that images can be accurately reconstructed from their noisy ones given a prompt. Hence, at a small time-step, the adapter and prompt are essentially a parameterization of only the nuanced class attributes. For a test image, we can use the parameterization to only extract the nuanced class attributes for classification. TiF learner significantly outperforms OpenCLIP and its adapters on a variety of fine-grained and customized few-shot learning tasks. Codes are in https://github.com/yue-zhongqi/tif.

Title: Revisiting Meta-evaluation for Grammatical Error Correction

• Authors: Masamune Kobayashi, Masato Mita, Mamoru Komachi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02674
• Pdf URL: https://arxiv.org/pdf/2403.02674
• Copy Paste: [[2403.02674]] Revisiting Meta-evaluation for Grammatical Error Correction(https://arxiv.org/abs/2403.02674)
• Keywords: large language model
• Abstract: Metrics are the foundation for automatic evaluation in grammatical error correction (GEC), with their evaluation of the metrics (meta-evaluation) relying on their correlation with human judgments. However, conventional meta-evaluations in English GEC encounter several challenges including biases caused by inconsistencies in evaluation granularity, and an outdated setup using classical systems. These problems can lead to misinterpretation of metrics and potentially hinder the applicability of GEC techniques. To address these issues, this paper proposes SEEDA, a new dataset for GEC meta-evaluation. SEEDA consists of corrections with human ratings along two different granularities: edit-based and sentence-based, covering 12 state-of-the-art systems including large language models (LLMs), and two human corrections with different focuses. The results of improved correlations by aligning the granularity in the sentence-level meta-evaluation, suggest that edit-based metrics may have been underestimated in existing studies. Furthermore, correlations of most metrics decrease when changing from classical to neural systems, indicating that traditional metrics are relatively poor at evaluating fluently corrected sentences with many edits.

Title: A Dual-Level Cancelable Framework for Palmprint Verification and Hack-Proof Data Storage

• Authors: Ziyuan Yang, Ming Kang, Andrew Beng Jin Teoh, Chengrui Gao, Wen Chen, Bob Zhang, Yi Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02680
• Pdf URL: https://arxiv.org/pdf/2403.02680
• Copy Paste: [[2403.02680]] A Dual-Level Cancelable Framework for Palmprint Verification and Hack-Proof Data Storage(https://arxiv.org/abs/2403.02680)
• Keywords: security, privacy, protect
• Abstract: In recent years, palmprints have been widely used for individual verification. The rich privacy information in palmprint data necessitates its protection to ensure security and privacy without sacrificing system performance. Existing systems often use cancelable technologies to protect templates, but these technologies ignore the potential risk of data leakage. Upon breaching the system and gaining access to the stored database, a hacker could easily manipulate the stored templates, compromising the security of the verification system. To address this issue, we propose a dual-level cancelable palmprint verification framework in this paper. Specifically, the raw template is initially encrypted using a competition hashing network with a first-level token, facilitating the end-to-end generation of cancelable templates. Different from previous works, the protected template undergoes further encryption to differentiate the second-level protected template from the first-level one. The system specifically creates a negative database (NDB) with the second-level token for dual-level protection during the enrollment stage. Reversing the NDB is NP-hard and a fine-grained algorithm for NDB generation is introduced to manage the noise and specified bits. During the verification stage, we propose an NDB matching algorithm based on matrix operation to accelerate the matching process of previous NDB methods caused by dictionary-based matching rules. This approach circumvents the need to store templates identical to those utilized for verification, reducing the risk of potential data leakage. Extensive experiments conducted on public palmprint datasets have confirmed the effectiveness and generality of the proposed framework. Upon acceptance of the paper, the code will be accessible at https://github.com/Deep-Imaging-Group/NPR.

Title: Time Weaver: A Conditional Time Series Generation Model

• Authors: Sai Shankar Narasimhan, Shubhankar Agarwal, Oguzhan Akcin, Sujay Sanghavi, Sandeep Chinchali
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.02682
• Pdf URL: https://arxiv.org/pdf/2403.02682
• Copy Paste: [[2403.02682]] Time Weaver: A Conditional Time Series Generation Model(https://arxiv.org/abs/2403.02682)
• Keywords: diffusion, generative
• Abstract: Imagine generating a city's electricity demand pattern based on weather, the presence of an electric vehicle, and location, which could be used for capacity planning during a winter freeze. Such real-world time series are often enriched with paired heterogeneous contextual metadata (weather, location, etc.). Current approaches to time series generation often ignore this paired metadata, and its heterogeneity poses several practical challenges in adapting existing conditional generation approaches from the image, audio, and video domains to the time series domain. To address this gap, we introduce Time Weaver, a novel diffusion-based model that leverages the heterogeneous metadata in the form of categorical, continuous, and even time-variant variables to significantly improve time series generation. Additionally, we show that naive extensions of standard evaluation metrics from the image to the time series domain are insufficient. These metrics do not penalize conditional generation approaches for their poor specificity in reproducing the metadata-specific features in the generated time series. Thus, we innovate a novel evaluation metric that accurately captures the specificity of conditional generation and the realism of the generated time series. We show that Time Weaver outperforms state-of-the-art benchmarks, such as Generative Adversarial Networks (GANs), by up to 27% in downstream classification tasks on real-world energy, medical, air quality, and traffic data sets.

Title: Learning to Defer to a Population: A Meta-Learning Approach

• Authors: Dharmesh Tailor, Aditya Patra, Rajeev Verma, Putra Manggala, Eric Nalisnick
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.02683
• Pdf URL: https://arxiv.org/pdf/2403.02683
• Copy Paste: [[2403.02683]] Learning to Defer to a Population: A Meta-Learning Approach(https://arxiv.org/abs/2403.02683)
• Keywords: robust
• Abstract: The learning to defer (L2D) framework allows autonomous systems to be safe and robust by allocating difficult decisions to a human expert. All existing work on L2D assumes that each expert is well-identified, and if any expert were to change, the system should be re-trained. In this work, we alleviate this constraint, formulating an L2D system that can cope with never-before-seen experts at test-time. We accomplish this by using meta-learning, considering both optimization- and model-based variants. Given a small context set to characterize the currently available expert, our framework can quickly adapt its deferral policy. For the model-based approach, we employ an attention mechanism that is able to look for points in the context set that are similar to a given test point, leading to an even more precise assessment of the expert's abilities. In the experiments, we validate our methods on image recognition, traffic sign detection, and skin lesion diagnosis benchmarks.

Title: Deep Common Feature Mining for Efficient Video Semantic Segmentation

• Authors: Yaoyan Zheng, Hongyu Yang, Di Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02689
• Pdf URL: https://arxiv.org/pdf/2403.02689
• Copy Paste: [[2403.02689]] Deep Common Feature Mining for Efficient Video Semantic Segmentation(https://arxiv.org/abs/2403.02689)
• Keywords: robust, segmentation
• Abstract: Recent advancements in video semantic segmentation have made substantial progress by exploiting temporal correlations. Nevertheless, persistent challenges, including redundant computation and the reliability of the feature propagation process, underscore the need for further innovation. In response, we present Deep Common Feature Mining (DCFM), a novel approach strategically designed to address these challenges by leveraging the concept of feature sharing. DCFM explicitly decomposes features into two complementary components. The common representation extracted from a key-frame furnishes essential high-level information to neighboring non-key frames, allowing for direct re-utilization without feature propagation. Simultaneously, the independent feature, derived from each video frame, captures rapidly changing information, providing frame-specific clues crucial for segmentation. To achieve such decomposition, we employ a symmetric training strategy tailored for sparsely annotated data, empowering the backbone to learn a robust high-level representation enriched with common information. Additionally, we incorporate a self-supervised loss function to reinforce intra-class feature similarity and enhance temporal consistency. Experimental evaluations on the VSPW and Cityscapes datasets demonstrate the effectiveness of our method, showing a superior balance between accuracy and efficiency.

Title: InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

• Authors: Qiusi Zhan, Zhixiang Liang, Zifan Ying, Daniel Kang
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02691
• Pdf URL: https://arxiv.org/pdf/2403.02691
• Copy Paste: [[2403.02691]] InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents(https://arxiv.org/abs/2403.02691)
• Keywords: attack, large language model
• Abstract: Recent work has embodied LLMs as agents, allowing them to access tools, perform actions, and interact with external content (e.g., emails or websites). However, external content introduces the risk of indirect prompt injection (IPI) attacks, where malicious instructions are embedded within the content processed by LLMs, aiming to manipulate these agents into executing detrimental actions against users. Given the potentially severe consequences of such attacks, establishing benchmarks to assess and mitigate these risks is imperative. In this work, we introduce InjecAgent, a benchmark designed to assess the vulnerability of tool-integrated LLM agents to IPI attacks. InjecAgent comprises 1,054 test cases covering 17 different user tools and 62 attacker tools. We categorize attack intentions into two primary types: direct harm to users and exfiltration of private data. We evaluate 30 different LLM agents and show that agents are vulnerable to IPI attacks, with ReAct-prompted GPT-4 vulnerable to attacks 24% of the time. Further investigation into an enhanced setting, where the attacker instructions are reinforced with a hacking prompt, shows additional increases in success rates, nearly doubling the attack success rate on the ReAct-prompted GPT-4. Our findings raise questions about the widespread deployment of LLM Agents. Our benchmark is available at https://github.com/uiuc-kang-lab/InjecAgent.

Title: Privacy-Aware Semantic Cache for Large Language Models

• Authors: Waris Gill (1), Mohamed Elidrisi (2), Pallavi Kalapatapu (2), Ali Anwar (3), Muhammad Ali Gulzar (1) ((1) Virginia Tech, USA, (2) Cisco, USA (3) University of Minnesota, Minneapolis, USA)
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.02694
• Pdf URL: https://arxiv.org/pdf/2403.02694
• Copy Paste: [[2403.02694]] Privacy-Aware Semantic Cache for Large Language Models(https://arxiv.org/abs/2403.02694)
• Keywords: privacy, federate, large language model
• Abstract: Large Language Models (LLMs) like ChatGPT, Google Bard, Claude, and Llama 2 have revolutionized natural language processing and search engine dynamics. However, these models incur exceptionally high computational costs. For instance, GPT-3 consists of 175 billion parameters and inference on these models also demands billions of floating-point operations. Caching is a natural solution to reduce LLM inference costs on repeated queries. However, existing caching methods are incapable of finding semantic similarities among LLM queries, leading to unacceptable false hit-and-miss rates. This paper introduces MeanCache, a semantic cache for LLMs that identifies semantically similar queries to determine cache hit or miss. Using MeanCache, the response to a user's semantically similar query can be retrieved from a local cache rather than re-querying the LLM, thus reducing costs, service provider load, and environmental impact. MeanCache leverages Federated Learning (FL) to collaboratively train a query similarity model in a distributed manner across numerous users without violating privacy. By placing a local cache in each user's device and using FL, MeanCache reduces the latency and costs and enhances model performance, resulting in lower cache false hit rates. Our experiments, benchmarked against the GPTCache, reveal that MeanCache attains an approximately 17% higher F-score and a 20% increase in precision during semantic cache hit-and-miss decisions. Furthermore, MeanCache reduces the storage requirement by 83% and accelerates semantic cache hit-and-miss decisions by 11%, while still surpassing GPTCache.

Title: Controllable Prompt Tuning For Balancing Group Distributional Robustness

• Authors: Hoang Phan, Andrew Gordon Wilson, Qi Lei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02695
• Pdf URL: https://arxiv.org/pdf/2403.02695
• Copy Paste: [[2403.02695]] Controllable Prompt Tuning For Balancing Group Distributional Robustness(https://arxiv.org/abs/2403.02695)
• Keywords: robust, transformer
• Abstract: Models trained on data composed of different groups or domains can suffer from severe performance degradation under distribution shifts. While recent methods have largely focused on optimizing the worst-group objective, this often comes at the expense of good performance on other groups. To address this problem, we introduce an optimization scheme to achieve good performance across groups and find a good solution for all without severely sacrificing performance on any of them. However, directly applying such optimization involves updating the parameters of the entire network, making it both computationally expensive and challenging. Thus, we introduce Controllable Prompt Tuning (CPT), which couples our approach with prompt-tuning techniques. On spurious correlation benchmarks, our procedures achieve state-of-the-art results across both transformer and non-transformer architectures, as well as unimodal and multimodal data, while requiring only 0.4% tunable parameters.

Title: Causal Walk: Debiasing Multi-Hop Fact Verification with Front-Door Adjustment

• Authors: Congzhi Zhang, Linhai Zhang, Deyu Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02698
• Pdf URL: https://arxiv.org/pdf/2403.02698
• Copy Paste: [[2403.02698]] Causal Walk: Debiasing Multi-Hop Fact Verification with Front-Door Adjustment(https://arxiv.org/abs/2403.02698)
• Keywords: large language model
• Abstract: Conventional multi-hop fact verification models are prone to rely on spurious correlations from the annotation artifacts, leading to an obvious performance decline on unbiased datasets. Among the various debiasing works, the causal inference-based methods become popular by performing theoretically guaranteed debiasing such as casual intervention or counterfactual reasoning. However, existing causal inference-based debiasing methods, which mainly formulate fact verification as a single-hop reasoning task to tackle shallow bias patterns, cannot deal with the complicated bias patterns hidden in multiple hops of evidence. To address the challenge, we propose Causal Walk, a novel method for debiasing multi-hop fact verification from a causal perspective with front-door adjustment. Specifically, in the structural causal model, the reasoning path between the treatment (the input claim-evidence graph) and the outcome (the veracity label) is introduced as the mediator to block the confounder. With the front-door adjustment, the causal effect between the treatment and the outcome is decomposed into the causal effect between the treatment and the mediator, which is estimated by applying the idea of random walk, and the causal effect between the mediator and the outcome, which is estimated with normalized weighted geometric mean approximation. To investigate the effectiveness of the proposed method, an adversarial multi-hop fact verification dataset and a symmetric multi-hop fact verification dataset are proposed with the help of the large language model. Experimental results show that Causal Walk outperforms some previous debiasing methods on both existing datasets and the newly constructed datasets. Code and data will be released at https://github.com/zcccccz/CausalWalk.

Title: FastOcc: Accelerating 3D Occupancy Prediction by Fusing the 2D Bird's-Eye View and Perspective View

• Authors: Jiawei Hou, Xiaoyan Li, Wenhao Guan, Gang Zhang, Di Feng, Yuheng Du, Xiangyang Xue, Jian Pu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.02710
• Pdf URL: https://arxiv.org/pdf/2403.02710
• Copy Paste: [[2403.02710]] FastOcc: Accelerating 3D Occupancy Prediction by Fusing the 2D Bird's-Eye View and Perspective View(https://arxiv.org/abs/2403.02710)
• Keywords: segmentation
• Abstract: In autonomous driving, 3D occupancy prediction outputs voxel-wise status and semantic labels for more comprehensive understandings of 3D scenes compared with traditional perception tasks, such as 3D object detection and bird's-eye view (BEV) semantic segmentation. Recent researchers have extensively explored various aspects of this task, including view transformation techniques, ground-truth label generation, and elaborate network design, aiming to achieve superior performance. However, the inference speed, crucial for running on an autonomous vehicle, is neglected. To this end, a new method, dubbed FastOcc, is proposed. By carefully analyzing the network effect and latency from four parts, including the input image resolution, image backbone, view transformation, and occupancy prediction head, it is found that the occupancy prediction head holds considerable potential for accelerating the model while keeping its accuracy. Targeted at improving this component, the time-consuming 3D convolution network is replaced with a novel residual-like architecture, where features are mainly digested by a lightweight 2D BEV convolution network and compensated by integrating the 3D voxel features interpolated from the original image features. Experiments on the Occ3D-nuScenes benchmark demonstrate that our FastOcc achieves state-of-the-art results with a fast inference speed.

Title: Android in the Zoo: Chain-of-Action-Thought for GUI Agents

• Authors: Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, Duyu Tang
• Subjects: cs.CL, cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02713
• Pdf URL: https://arxiv.org/pdf/2403.02713
• Copy Paste: [[2403.02713]] Android in the Zoo: Chain-of-Action-Thought for GUI Agents(https://arxiv.org/abs/2403.02713)
• Keywords: large language model
• Abstract: Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone, which completes a task triggered by natural language through predicting a sequence of actions of API. Even though the task highly relies on past actions and visual observations, existing studies typical consider little semantic information carried out by intermediate screenshots and screen operations. To address this, this work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the current screen, and more importantly the action thinking of what actions should be performed and the outcomes led by the chosen action. We demonstrate that, in a zero-shot setting upon an off-the-shell LLM, CoAT significantly improves the goal progress compared to standard context modeling. To further facilitate the research in this line, we construct a benchmark Android-In-The-Zoo (AitZ), which contains 18,643 screen-action pairs together with chain-of-action-thought annotations. Experiments show that fine-tuning a 200M model on our AitZ dataset achieves on par performance with CogAgent-Chat-18B.

Title: Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

• Authors: Sang T. Truong, Duc Q. Nguyen, Toan Nguyen, Dong D. Le, Nhi N. Truong, Tho Quan, Sanmi Koyejo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02715
• Pdf URL: https://arxiv.org/pdf/2403.02715
• Copy Paste: [[2403.02715]] Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models(https://arxiv.org/abs/2403.02715)
• Keywords: generative, large language model
• Abstract: Recent advancements in large language models (LLMs) have underscored their importance in the evolution of artificial intelligence. However, despite extensive pretraining on multilingual datasets, available open-sourced LLMs exhibit limited effectiveness in processing Vietnamese. The challenge is exacerbated by the absence of systematic benchmark datasets and metrics tailored for Vietnamese LLM evaluation. To mitigate these issues, we have finetuned LLMs specifically for Vietnamese and developed a comprehensive evaluation framework encompassing 10 common tasks and 31 metrics. Our evaluation results reveal that the fine-tuned LLMs exhibit enhanced comprehension and generative capabilities in Vietnamese. Moreover, our analysis indicates that models with more parameters can introduce more biases and uncalibrated outputs and the key factor influencing LLM performance is the quality of the training or fine-tuning datasets. These insights underscore the significance of meticulous fine-tuning with high-quality datasets in enhancing LLM performance.

Title: DP-CRE: Continual Relation Extraction via Decoupled Contrastive Learning and Memory Structure Preservation

• Authors: Mengyi Huang, Meng Xiao, Ludi Wang, Yi Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02718
• Pdf URL: https://arxiv.org/pdf/2403.02718
• Copy Paste: [[2403.02718]] DP-CRE: Continual Relation Extraction via Decoupled Contrastive Learning and Memory Structure Preservation(https://arxiv.org/abs/2403.02718)
• Keywords: extraction
• Abstract: Continuous Relation Extraction (CRE) aims to incrementally learn relation knowledge from a non-stationary stream of data. Since the introduction of new relational tasks can overshadow previously learned information, catastrophic forgetting becomes a significant challenge in this domain. Current replay-based training paradigms prioritize all data uniformly and train memory samples through multiple rounds, which would result in overfitting old tasks and pronounced bias towards new tasks because of the imbalances of the replay set. To handle the problem, we introduce the DecouPled CRE (DP-CRE) framework that decouples the process of prior information preservation and new knowledge acquisition. This framework examines alterations in the embedding space as new relation classes emerge, distinctly managing the preservation and acquisition of knowledge. Extensive experiments show that DP-CRE significantly outperforms other CRE baselines across two datasets.

Title: HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?

• Authors: Sijie Ji, Xinzhe Zheng, Chenshu Wu
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.02727
• Pdf URL: https://arxiv.org/pdf/2403.02727
• Copy Paste: [[2403.02727]] HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?(https://arxiv.org/abs/2403.02727)
• Keywords: large language model
• Abstract: There is an ongoing debate regarding the potential of Large Language Models (LLMs) as foundational models seamlessly integrated with Cyber-Physical Systems (CPS) for interpreting the physical world. In this paper, we carry out a case study to answer the following question: Are LLMs capable of zero-shot human activity recognition (HAR). Our study, HARGPT, presents an affirmative answer by demonstrating that LLMs can comprehend raw IMU data and perform HAR tasks in a zero-shot manner, with only appropriate prompts. HARGPT inputs raw IMU data into LLMs and utilizes the role-play and think step-by-step strategies for prompting. We benchmark HARGPT on GPT4 using two public datasets of different inter-class similarities and compare various baselines both based on traditional machine learning and state-of-the-art deep classification models. Remarkably, LLMs successfully recognize human activities from raw IMU data and consistently outperform all the baselines on both datasets. Our findings indicate that by effective prompting, LLMs can interpret raw IMU data based on their knowledge base, possessing a promising potential to analyze raw sensor data of the physical world effectively.

Title: A Two-Stage Training Method for Modeling Constrained Systems With Neural Networks

• Authors: C. Coelho, M. Fernanda P. Costa, L.L. Ferrás
• Subjects: cs.LG, cs.CE, math.OC
• Abstract URL: https://arxiv.org/abs/2403.02730
• Pdf URL: https://arxiv.org/pdf/2403.02730
• Copy Paste: [[2403.02730]] A Two-Stage Training Method for Modeling Constrained Systems With Neural Networks(https://arxiv.org/abs/2403.02730)
• Keywords: explainability
• Abstract: Real-world systems are often formulated as constrained optimization problems. Techniques to incorporate constraints into Neural Networks (NN), such as Neural Ordinary Differential Equations (Neural ODEs), have been used. However, these introduce hyperparameters that require manual tuning through trial and error, raising doubts about the successful incorporation of constraints into the generated model. This paper describes in detail the two-stage training method for Neural ODEs, a simple, effective, and penalty parameter-free approach to model constrained systems. In this approach the constrained optimization problem is rewritten as two unconstrained sub-problems that are solved in two stages. The first stage aims at finding feasible NN parameters by minimizing a measure of constraints violation. The second stage aims to find the optimal NN parameters by minimizing the loss function while keeping inside the feasible region. We experimentally demonstrate that our method produces models that satisfy the constraints and also improves their predictive performance. Thus, ensuring compliance with critical system properties and also contributing to reducing data quantity requirements. Furthermore, we show that the proposed method improves the convergence to an optimal solution and improves the explainability of Neural ODE models. Our proposed two-stage training method can be used with any NN architectures.

Title: Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment

• Authors: Congzhi Zhang, Linhai Zhang, Deyu Zhou, Guoqiang Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02738
• Pdf URL: https://arxiv.org/pdf/2403.02738
• Copy Paste: [[2403.02738]] Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment(https://arxiv.org/abs/2403.02738)
• Keywords: large language model
• Abstract: Despite the significant achievements of existing prompting methods such as in-context learning and chain-of-thought for large language models (LLMs), they still face challenges of various biases. Traditional debiasing methods primarily focus on the model training stage, including data augmentation-based and reweight-based approaches, with the limitations of addressing the complex biases of LLMs. To address such limitations, the causal relationship behind the prompting methods is uncovered using a structural causal model, and a novel causal prompting method based on front-door adjustment is proposed to effectively mitigate the bias of LLMs. In specific, causal intervention is implemented by designing the prompts without accessing the parameters and logits of LLMs.The chain-of-thoughts generated by LLMs are employed as the mediator variable and the causal effect between the input prompt and the output answers is calculated through front-door adjustment to mitigate model biases. Moreover, to obtain the representation of the samples precisely and estimate the causal effect more accurately, contrastive learning is used to fine-tune the encoder of the samples by aligning the space of the encoder with the LLM. Experimental results show that the proposed causal prompting approach achieves excellent performance on 3 natural language processing datasets on both open-source and closed-source LLMs.

Title: Towards Training A Chinese Large Language Model for Anesthesiology

• Authors: Zhonghai Wang, Jie Jiang, Yibing Zhan, Bohao Zhou, Yanhong Li, Chong Zhang, Liang Ding, Hua Jin, Jun Peng, Xu Lin, Weifeng Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02742
• Pdf URL: https://arxiv.org/pdf/2403.02742
• Copy Paste: [[2403.02742]] Towards Training A Chinese Large Language Model for Anesthesiology(https://arxiv.org/abs/2403.02742)
• Keywords: large language model
• Abstract: Medical large language models (LLMs) have gained popularity recently due to their significant practical utility. However, most existing research focuses on general medicine, and there is a need for in-depth study of LLMs in specific fields like anesthesiology. To fill the gap, we introduce Hypnos, a Chinese Anesthesia model built upon existing LLMs, e.g., Llama. Hypnos' contributions have three aspects: 1) The data, such as utilizing Self-Instruct, acquired from current LLMs likely includes inaccuracies. Hypnos implements a cross-filtering strategy to improve the data quality. This strategy involves using one LLM to assess the quality of the generated data from another LLM and filtering out the data with low quality. 2) Hypnos employs a general-to-specific training strategy that starts by fine-tuning LLMs using the general medicine data and subsequently improving the fine-tuned LLMs using data specifically from Anesthesiology. The general medical data supplement the medical expertise in Anesthesiology and enhance the effectiveness of Hypnos' generation. 3) We introduce a standardized benchmark for evaluating medical LLM in Anesthesiology. Our benchmark includes both publicly available instances from the Internet and privately obtained cases from the Hospital. Hypnos outperforms other medical LLMs in anesthesiology in metrics, GPT-4, and human evaluation on the benchmark dataset.

Title: Self-adaptive Traffic Anomaly Detection System for IoT Smart Home Environments

• Authors: Naoto Watanabe (1), Taku Yamazaki (1), Takumi Miyoshi (1), Ryo Yamamoto (2), Masataka Nakahara (3), Norihiro Okui (3), Ayumu Kubota (3) ((1) Shibaura Institute of Technology, (2) The University of Electro-Communications, (3) KDDI Research, Inc.)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02744
• Pdf URL: https://arxiv.org/pdf/2403.02744
• Copy Paste: [[2403.02744]] Self-adaptive Traffic Anomaly Detection System for IoT Smart Home Environments(https://arxiv.org/abs/2403.02744)
• Keywords: security, protect, attack
• Abstract: With the growth of internet of things (IoT) devices, cyberattacks, such as distributed denial of service, that exploit vulnerable devices infected with malware have increased. Therefore, vendors and users must keep their device firmware updated to eliminate vulnerabilities and quickly handle unknown cyberattacks. However, it is difficult for both vendors and users to continually keep the devices safe because vendors must provide updates quickly and the users must continuously manage the conditions of all deployed devices. Therefore, to ensure security, it is necessary for a system to adapt autonomously to changes in cyberattacks. In addition, it is important to consider network-side security that detects and filters anomalous traffic at the gateway to comprehensively protect those devices. This paper proposes a self-adaptive anomaly detection system for IoT traffic, including unknown attacks. The proposed system comprises a honeypot server and a gateway. The honeypot server continuously captures traffic and adaptively generates an anomaly detection model using real-time captured traffic. Thereafter, the gateway uses the generated model to detect anomalous traffic. Thus, the proposed system can adapt to unknown attacks to reflect pattern changes in anomalous traffic based on real-time captured traffic. Three experiments were conducted to evaluate the proposed system: a virtual experiment using pre-captured traffic from various regions across the world, a demonstration experiment using real-time captured traffic, and a virtual experiment using a public dataset containing the traffic generated by malware. The experimental results indicate that a system adaptable in real time to evolving cyberattacks is a novel approach for ensuring the comprehensive security of IoT devices against both known and unknown attacks.

Title: Learning without Exact Guidance: Updating Large-scale High-resolution Land Cover Maps from Low-resolution Historical Labels

• Authors: Zhuohong Li, Wei He, Jiepan Li, Fangxiao Lu, Hongyan Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02746
• Pdf URL: https://arxiv.org/pdf/2403.02746
• Copy Paste: [[2403.02746]] Learning without Exact Guidance: Updating Large-scale High-resolution Land Cover Maps from Low-resolution Historical Labels(https://arxiv.org/abs/2403.02746)
• Keywords: transformer, segmentation
• Abstract: Large-scale high-resolution (HR) land-cover mapping is a vital task to survey the Earth's surface and resolve many challenges facing humanity. However, it is still a non-trivial task hindered by complex ground details, various landforms, and the scarcity of accurate training labels over a wide-span geographic area. In this paper, we propose an efficient, weakly supervised framework (Paraformer), a.k.a. Low-to-High Network (L2HNet) V2, to guide large-scale HR land-cover mapping with easy-access historical land-cover data of low resolution (LR). Specifically, existing land-cover mapping approaches reveal the dominance of CNNs in preserving local ground details but still suffer from insufficient global modeling in various landforms. Therefore, we design a parallel CNN-Transformer feature extractor in Paraformer, consisting of a downsampling-free CNN branch and a Transformer branch, to jointly capture local and global contextual information. Besides, facing the spatial mismatch of training data, a pseudo-label-assisted training (PLAT) module is adopted to reasonably refine LR labels for weakly supervised semantic segmentation of HR images. Experiments on two large-scale datasets demonstrate the superiority of Paraformer over other state-of-the-art methods for automatically updating HR land-cover maps from LR historical labels.

Title: Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models

• Authors: Rui Wang, Fei Mi, Yi Chen, Boyang Xue, Hongru Wang, Qi Zhu, Kam-Fai Wong, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02756
• Pdf URL: https://arxiv.org/pdf/2403.02756
• Copy Paste: [[2403.02756]] Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models(https://arxiv.org/abs/2403.02756)
• Keywords: robust, large language model
• Abstract: The growing interest in Large Language Models (LLMs) for specialized applications has revealed a significant challenge: when tailored to specific domains, LLMs tend to experience catastrophic forgetting, compromising their general capabilities and leading to a suboptimal user experience. Additionally, crafting a versatile model for multiple domains simultaneously often results in a decline in overall performance due to confusion between domains. In response to these issues, we present the RolE Prompting Guided Multi-Domain Adaptation (REGA) strategy. This novel approach effectively manages multi-domain LLM adaptation through three key components: 1) Self-Distillation constructs and replays general-domain exemplars to alleviate catastrophic forgetting. 2) Role Prompting assigns a central prompt to the general domain and a unique role prompt to each specific domain to minimize inter-domain confusion during training. 3) Role Integration reuses and integrates a small portion of domain-specific data to the general-domain data, which are trained under the guidance of the central prompt. The central prompt is used for a streamlined inference process, removing the necessity to switch prompts for different domains. Empirical results demonstrate that REGA effectively alleviates catastrophic forgetting and inter-domain confusion. This leads to improved domain-specific performance compared to standard fine-tuned models, while still preserving robust general capabilities.

Title: Data Collaboration Analysis Over Matrix Manifolds

• Authors: Keiyu Nosaka, Akiko Yoshise
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2403.02780
• Pdf URL: https://arxiv.org/pdf/2403.02780
• Copy Paste: [[2403.02780]] Data Collaboration Analysis Over Matrix Manifolds(https://arxiv.org/abs/2403.02780)
• Keywords: secure, privacy, protect, robust
• Abstract: The effectiveness of machine learning (ML) algorithms is deeply intertwined with the quality and diversity of their training datasets. Improved datasets, marked by superior quality, enhance the predictive accuracy and broaden the applicability of models across varied scenarios. Researchers often integrate data from multiple sources to mitigate biases and limitations of single-source datasets. However, this extensive data amalgamation raises significant ethical concerns, particularly regarding user privacy and the risk of unauthorized data disclosure. Various global legislative frameworks have been established to address these privacy issues. While crucial for safeguarding privacy, these regulations can complicate the practical deployment of ML technologies. Privacy-Preserving Machine Learning (PPML) addresses this challenge by safeguarding sensitive information, from health records to geolocation data, while enabling the secure use of this data in developing robust ML models. Within this realm, the Non-Readily Identifiable Data Collaboration (NRI-DC) framework emerges as an innovative approach, potentially resolving the 'data island' issue among institutions through non-iterative communication and robust privacy protections. However, in its current state, the NRI-DC framework faces model performance instability due to theoretical unsteadiness in creating collaboration functions. This study establishes a rigorous theoretical foundation for these collaboration functions and introduces new formulations through optimization problems on matrix manifolds and efficient solutions. Empirical analyses demonstrate that the proposed approach, particularly the formulation over orthogonal matrix manifolds, significantly enhances performance, maintaining consistency and efficiency without compromising communication efficiency or privacy protections.

Title: DDF: A Novel Dual-Domain Image Fusion Strategy for Remote Sensing Image Semantic Segmentation with Unsupervised Domain Adaptation

• Authors: Lingyan Ran, Lushuang Wang, Tao Zhuo, Yinghui Xing
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02784
• Pdf URL: https://arxiv.org/pdf/2403.02784
• Copy Paste: [[2403.02784]] DDF: A Novel Dual-Domain Image Fusion Strategy for Remote Sensing Image Semantic Segmentation with Unsupervised Domain Adaptation(https://arxiv.org/abs/2403.02784)
• Keywords: segmentation
• Abstract: Semantic segmentation of remote sensing images is a challenging and hot issue due to the large amount of unlabeled data. Unsupervised domain adaptation (UDA) has proven to be advantageous in incorporating unclassified information from the target domain. However, independently fine-tuning UDA models on the source and target domains has a limited effect on the outcome. This paper proposes a hybrid training strategy as well as a novel dual-domain image fusion strategy that effectively utilizes the original image, transformation image, and intermediate domain information. Moreover, to enhance the precision of pseudo-labels, we present a pseudo-label region-specific weight strategy. The efficacy of our approach is substantiated by extensive benchmark experiments and ablation studies conducted on the ISPRS Vaihingen and Potsdam datasets.

Title: Semi-Supervised Graph Representation Learning with Human-centric Explanation for Predicting Fatty Liver Disease

• Authors: So Yeon Kim, Sehee Wang, Eun Kyung Choe
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02786
• Pdf URL: https://arxiv.org/pdf/2403.02786
• Copy Paste: [[2403.02786]] Semi-Supervised Graph Representation Learning with Human-centric Explanation for Predicting Fatty Liver Disease(https://arxiv.org/abs/2403.02786)
• Keywords: interpretability
• Abstract: Addressing the challenge of limited labeled data in clinical settings, particularly in the prediction of fatty liver disease, this study explores the potential of graph representation learning within a semi-supervised learning framework. Leveraging graph neural networks (GNNs), our approach constructs a subject similarity graph to identify risk patterns from health checkup data. The effectiveness of various GNN approaches in this context is demonstrated, even with minimal labeled samples. Central to our methodology is the inclusion of human-centric explanations through explainable GNNs, providing personalized feature importance scores for enhanced interpretability and clinical relevance, thereby underscoring the potential of our approach in advancing healthcare practices with a keen focus on graph representation learning and human-centric explanation.

Title: DPPA: Pruning Method for Large Language Model to Model Merging

• Authors: Yaochen Zhu, Rui Xia, Jiajun Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02799
• Pdf URL: https://arxiv.org/pdf/2403.02799
• Copy Paste: [[2403.02799]] DPPA: Pruning Method for Large Language Model to Model Merging(https://arxiv.org/abs/2403.02799)
• Keywords: large language model
• Abstract: Model merging is to combine fine-tuned models derived from multiple domains, with the intent of enhancing the model's proficiency across various domains. The principal concern is the resolution of parameter conflicts. A substantial amount of existing research remedy this issue during the merging stage, with the latest study focusing on resolving this issue throughout the pruning stage. The DARE approach has exhibited promising outcomes when applied to a simplistic fine-tuned model. However, the efficacy of this method tends to wane when employed on complex fine-tuned models that show a significant parameter bias relative to the baseline model. In this paper, we introduce a dual-stage method termed Dynamic Pruning Partition Amplification (DPPA), devised to tackle the challenge of merging complex fine-tuned models. Initially, we introduce Dynamically Pruning (DP), an improved approach based on magnitude pruning, which aim is to enhance performance at higher pruning rates. Subsequently, we propose Dynamically Partition Amplification (DPA), a rescaling strategy, is designed to dynamically amplify parameter partitions in relation to their significance levels. The experimental results show that our method maintains a mere 20% of domain-specific parameters and yet delivers a performance comparable to other methodologies that preserve up to 90% of parameters. Furthermore, our method displays outstanding performance post-pruning, leading to a significant improvement of nearly 20% performance in model merging. We make our code on Github.

Title: Towards Robust Federated Learning via Logits Calibration on Non-IID Data

• Authors: Yu Qiao, Apurba Adhikary, Chaoning Zhang, Choong Seon Hong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02803
• Pdf URL: https://arxiv.org/pdf/2403.02803
• Copy Paste: [[2403.02803]] Towards Robust Federated Learning via Logits Calibration on Non-IID Data(https://arxiv.org/abs/2403.02803)
• Keywords: privacy, attack, robust, federate
• Abstract: Federated learning (FL) is a privacy-preserving distributed management framework based on collaborative model training of distributed devices in edge networks. However, recent studies have shown that FL is vulnerable to adversarial examples (AEs), leading to a significant drop in its performance. Meanwhile, the non-independent and identically distributed (non-IID) challenge of data distribution between edge devices can further degrade the performance of models. Consequently, both AEs and non-IID pose challenges to deploying robust learning models at the edge. In this work, we adopt the adversarial training (AT) framework to improve the robustness of FL models against adversarial example (AE) attacks, which can be termed as federated adversarial training (FAT). Moreover, we address the non-IID challenge by implementing a simple yet effective logits calibration strategy under the FAT framework, which can enhance the robustness of models when subjected to adversarial attacks. Specifically, we employ a direct strategy to adjust the logits output by assigning higher weights to classes with small samples during training. This approach effectively tackles the class imbalance in the training data, with the goal of mitigating biases between local and global models. Experimental results on three dataset benchmarks, MNIST, Fashion-MNIST, and CIFAR-10 show that our strategy achieves competitive results in natural and robust accuracy compared to several baselines.

Title: Dynamic Gaussian Graph Operator: Learning parametric partial differential equations in arbitrary discrete mechanics problems

• Authors: Chu Wang, Jinhong Wu, Yanzhi Wang, Zhijian Zha, Qi Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02810
• Pdf URL: https://arxiv.org/pdf/2403.02810
• Copy Paste: [[2403.02810]] Dynamic Gaussian Graph Operator: Learning parametric partial differential equations in arbitrary discrete mechanics problems(https://arxiv.org/abs/2403.02810)
• Keywords: robust
• Abstract: Deep learning methods have access to be employed for solving physical systems governed by parametric partial differential equations (PDEs) due to massive scientific data. It has been refined to operator learning that focuses on learning non-linear mapping between infinite-dimensional function spaces, offering interface from observations to solutions. However, state-of-the-art neural operators are limited to constant and uniform discretization, thereby leading to deficiency in generalization on arbitrary discretization schemes for computational domain. In this work, we propose a novel operator learning algorithm, referred to as Dynamic Gaussian Graph Operator (DGGO) that expands neural operators to learning parametric PDEs in arbitrary discrete mechanics problems. The Dynamic Gaussian Graph (DGG) kernel learns to map the observation vectors defined in general Euclidean space to metric vectors defined in high-dimensional uniform metric space. The DGG integral kernel is parameterized by Gaussian kernel weighted Riemann sum approximating and using dynamic message passing graph to depict the interrelation within the integral term. Fourier Neural Operator is selected to localize the metric vectors on spatial and frequency domains. Metric vectors are regarded as located on latent uniform domain, wherein spatial and spectral transformation offer highly regular constraints on solution space. The efficiency and robustness of DGGO are validated by applying it to solve numerical arbitrary discrete mechanics problems in comparison with mainstream neural operators. Ablation experiments are implemented to demonstrate the effectiveness of spatial transformation in the DGG kernel. The proposed method is utilized to forecast stress field of hyper-elastic material with geometrically variable void as engineering application.

Title: InjectTST: A Transformer Method of Injecting Global Information into Independent Channels for Long Time Series Forecasting

• Authors: Ce Chi, Xing Wang, Kexin Yang, Zhiyan Song, Di Jin, Lin Zhu, Chao Deng, Junlan Feng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02814
• Pdf URL: https://arxiv.org/pdf/2403.02814
• Copy Paste: [[2403.02814]] InjectTST: A Transformer Method of Injecting Global Information into Independent Channels for Long Time Series Forecasting(https://arxiv.org/abs/2403.02814)
• Keywords: robust, transformer
• Abstract: Transformer has become one of the most popular architectures for multivariate time series (MTS) forecasting. Recent Transformer-based MTS models generally prefer channel-independent structures with the observation that channel independence can alleviate noise and distribution drift issues, leading to more robustness. Nevertheless, it is essential to note that channel dependency remains an inherent characteristic of MTS, carrying valuable information. Designing a model that incorporates merits of both channel-independent and channel-mixing structures is a key to further improvement of MTS forecasting, which poses a challenging conundrum. To address the problem, an injection method for global information into channel-independent Transformer, InjectTST, is proposed in this paper. Instead of designing a channel-mixing model directly, we retain the channel-independent backbone and gradually inject global information into individual channels in a selective way. A channel identifier, a global mixing module and a self-contextual attention module are devised in InjectTST. The channel identifier can help Transformer distinguish channels for better representation. The global mixing module produces cross-channel global information. Through the self-contextual attention module, the independent channels can selectively concentrate on useful global information without robustness degradation, and channel mixing is achieved implicitly. Experiments indicate that InjectTST can achieve stable improvement compared with state-of-the-art models.

Title: Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications

• Authors: Stav Cohen, Ron Bitton, Ben Nassi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02817
• Pdf URL: https://arxiv.org/pdf/2403.02817
• Copy Paste: [[2403.02817]] Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications(https://arxiv.org/abs/2403.02817)
• Keywords: attack, membership infer, generative
• Abstract: In the past year, numerous companies have incorporated Generative AI (GenAI) capabilities into new and existing applications, forming interconnected Generative AI (GenAI) ecosystems consisting of semi/fully autonomous agents powered by GenAI services. While ongoing research highlighted risks associated with the GenAI layer of agents (e.g., dialog poisoning, membership inference, prompt leaking, jailbreaking), a critical question emerges: Can attackers develop malware to exploit the GenAI component of an agent and launch cyber-attacks on the entire GenAI ecosystem? This paper introduces Morris II, the first worm designed to target GenAI ecosystems through the use of adversarial self-replicating prompts. The study demonstrates that attackers can insert such prompts into inputs that, when processed by GenAI models, prompt the model to replicate the input as output (replication), engaging in malicious activities (payload). Additionally, these inputs compel the agent to deliver them (propagate) to new agents by exploiting the connectivity within the GenAI ecosystem. We demonstrate the application of Morris II against GenAIpowered email assistants in two use cases (spamming and exfiltrating personal data), under two settings (black-box and white-box accesses), using two types of input data (text and images). The worm is tested against three different GenAI models (Gemini Pro, ChatGPT 4.0, and LLaVA), and various factors (e.g., propagation rate, replication, malicious activity) influencing the performance of the worm are evaluated.

Title: An Adaptive Hydropower Management Approach for Downstream Ecosystem Preservation

• Authors: C. Coelho, M. Jing, M. Fernanda P. Costa, L.L. Ferrás
• Subjects: cs.LG, cs.CE, math.OC
• Abstract URL: https://arxiv.org/abs/2403.02821
• Pdf URL: https://arxiv.org/pdf/2403.02821
• Copy Paste: [[2403.02821]] An Adaptive Hydropower Management Approach for Downstream Ecosystem Preservation(https://arxiv.org/abs/2403.02821)
• Keywords: protect
• Abstract: Hydropower plants play a pivotal role in advancing clean and sustainable energy production, contributing significantly to the global transition towards renewable energy sources. However, hydropower plants are currently perceived both positively as sources of renewable energy and negatively as disruptors of ecosystems. In this work, we highlight the overlooked potential of using hydropower plant as protectors of ecosystems by using adaptive ecological discharges. To advocate for this perspective, we propose using a neural network to predict the minimum ecological discharge value at each desired time. Additionally, we present a novel framework that seamlessly integrates it into hydropower management software, taking advantage of the well-established approach of using traditional constrained optimisation algorithms. This novel approach not only protects the ecosystems from climate change but also contributes to potentially increase the electricity production.

Title: Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generation

• Authors: Weijie Li, Litong Gong, Yiran Zhu, Fanda Fan, Biao Wang, Tiezheng Ge, Bo Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02827
• Pdf URL: https://arxiv.org/pdf/2403.02827
• Copy Paste: [[2403.02827]] Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generation(https://arxiv.org/abs/2403.02827)
• Keywords: diffusion
• Abstract: Image-to-video (I2V) generation tasks always suffer from keeping high fidelity in the open domains. Traditional image animation techniques primarily focus on specific domains such as faces or human poses, making them difficult to generalize to open domains. Several recent I2V frameworks based on diffusion models can generate dynamic content for open domain images but fail to maintain fidelity. We found that two main factors of low fidelity are the loss of image details and the noise prediction biases during the denoising process. To this end, we propose an effective method that can be applied to mainstream video diffusion models. This method achieves high fidelity based on supplementing more precise image information and noise rectification. Specifically, given a specified image, our method first adds noise to the input image latent to keep more details, then denoises the noisy latent with proper rectification to alleviate the noise prediction biases. Our method is tuning-free and plug-and-play. The experimental results demonstrate the effectiveness of our approach in improving the fidelity of generated videos. For more image-to-video generated results, please refer to the project website: https://noise-rectification.github.io.

Title: An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Models are Task-specific Classifiers

• Authors: Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, Tiejun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02839
• Pdf URL: https://arxiv.org/pdf/2403.02839
• Copy Paste: [[2403.02839]] An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Models are Task-specific Classifiers(https://arxiv.org/abs/2403.02839)
• Keywords: fair, large language model
• Abstract: Recently, there has been a growing trend of utilizing Large Language Model (LLM) to evaluate the quality of other LLMs. Many studies have employed proprietary close-source models, especially GPT4, as the evaluator. Alternatively, other works have fine-tuned judge models based on open-source LLMs as the evaluator. In this study, we conduct an empirical study of different judge models on their evaluation capability. Our findings indicate that although the fine-tuned judge models achieve high accuracy on in-domain test sets, even surpassing GPT4, they are inherently task-specific classifiers, and their generalizability and fairness severely underperform GPT4.

Title: FLGuard: Byzantine-Robust Federated Learning via Ensemble of Contrastive Models

• Authors: Younghan Lee, Yungi Cho, Woorim Han, Ho Bae, Yunheung Paek
• Subjects: cs.LG, cs.AI, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.02846
• Pdf URL: https://arxiv.org/pdf/2403.02846
• Copy Paste: [[2403.02846]] FLGuard: Byzantine-Robust Federated Learning via Ensemble of Contrastive Models(https://arxiv.org/abs/2403.02846)
• Keywords: defense, attack, robust, federate
• Abstract: Federated Learning (FL) thrives in training a global model with numerous clients by only sharing the parameters of their local models trained with their private training datasets. Therefore, without revealing the private dataset, the clients can obtain a deep learning (DL) model with high performance. However, recent research proposed poisoning attacks that cause a catastrophic loss in the accuracy of the global model when adversaries, posed as benign clients, are present in a group of clients. Therefore, recent studies suggested byzantine-robust FL methods that allow the server to train an accurate global model even with the adversaries present in the system. However, many existing methods require the knowledge of the number of malicious clients or the auxiliary (clean) dataset or the effectiveness reportedly decreased hugely when the private dataset was non-independently and identically distributed (non-IID). In this work, we propose FLGuard, a novel byzantine-robust FL method that detects malicious clients and discards malicious local updates by utilizing the contrastive learning technique, which showed a tremendous improvement as a self-supervised learning method. With contrastive models, we design FLGuard as an ensemble scheme to maximize the defensive capability. We evaluate FLGuard extensively under various poisoning attacks and compare the accuracy of the global model with existing byzantine-robust FL methods. FLGuard outperforms the state-of-the-art defense methods in most cases and shows drastic improvement, especially in non-IID settings. https://github.com/201younghanlee/FLGuard

Title: Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

• Authors: Philipp J. Rösch, Norbert Oswald, Michaela Geierhos, Jindřich Libovický
• Subjects: cs.CV, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.02875
• Pdf URL: https://arxiv.org/pdf/2403.02875
• Copy Paste: [[2403.02875]] Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples(https://arxiv.org/abs/2403.02875)
• Keywords: generative
• Abstract: Current multimodal models leveraging contrastive learning often face limitations in developing fine-grained conceptual understanding. This is due to random negative samples during pretraining, causing almost exclusively very dissimilar concepts to be compared in the loss function. Consequently, the models struggle with fine-grained semantic differences. To address this problem, we introduce a novel pretraining method incorporating synthetic hard negative text examples. The hard negatives permute terms corresponding to visual concepts, leading to a more fine-grained visual and textual concept alignment. Further, we introduce InpaintCOCO, a new challenging dataset for assessing the fine-grained alignment of colors, objects, and sizes in vision-language models. We created the dataset using generative inpainting from COCO images by changing the visual concepts so that the images no longer match their original captions. Our results show significant improvements in fine-grained concept understanding across a wide range of vision-language datasets, including our InpaintCOCO dataset.

Title: ActiveAD: Planning-Oriented Active Learning for End-to-End Autonomous Driving

• Authors: Han Lu, Xiaosong Jia, Yichen Xie, Wenlong Liao, Xiaokang Yang, Junchi Yan
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.02877
• Pdf URL: https://arxiv.org/pdf/2403.02877
• Copy Paste: [[2403.02877]] ActiveAD: Planning-Oriented Active Learning for End-to-End Autonomous Driving(https://arxiv.org/abs/2403.02877)
• Keywords: segmentation
• Abstract: End-to-end differentiable learning for autonomous driving (AD) has recently become a prominent paradigm. One main bottleneck lies in its voracious appetite for high-quality labeled data e.g. 3D bounding boxes and semantic segmentation, which are notoriously expensive to manually annotate. The difficulty is further pronounced due to the prominent fact that the behaviors within samples in AD often suffer from long tailed distribution. In other words, a large part of collected data can be trivial (e.g. simply driving forward in a straight road) and only a few cases are safety-critical. In this paper, we explore a practically important yet under-explored problem about how to achieve sample and label efficiency for end-to-end AD. Specifically, we design a planning-oriented active learning method which progressively annotates part of collected raw data according to the proposed diversity and usefulness criteria for planning routes. Empirically, we show that our planning-oriented approach could outperform general active learning methods by a large margin. Notably, our method achieves comparable performance with state-of-the-art end-to-end AD methods - by using only 30% nuScenes data. We hope our work could inspire future works to explore end-to-end AD from a data-centric perspective in addition to methodology efforts.

Title: Zero-LED: Zero-Reference Lighting Estimation Diffusion Model for Low-Light Image Enhancement

• Authors: Jinhong He, Minglong Xue, Zhipu Liu, Chengyun Song, Senming Zhong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02879
• Pdf URL: https://arxiv.org/pdf/2403.02879
• Copy Paste: [[2403.02879]] Zero-LED: Zero-Reference Lighting Estimation Diffusion Model for Low-Light Image Enhancement(https://arxiv.org/abs/2403.02879)
• Keywords: diffusion
• Abstract: Diffusion model-based low-light image enhancement methods rely heavily on paired training data, leading to limited extensive application. Meanwhile, existing unsupervised methods lack effective bridging capabilities for unknown degradation. To address these limitations, we propose a novel zero-reference lighting estimation diffusion model for low-light image enhancement called Zero-LED. It utilizes the stable convergence ability of diffusion models to bridge the gap between low-light domains and real normal-light domains and successfully alleviates the dependence on pairwise training data via zero-reference learning. Specifically, we first design the initial optimization network to preprocess the input image and implement bidirectional constraints between the diffusion model and the initial optimization network through multiple objective functions. Subsequently, the degradation factors of the real-world scene are optimized iteratively to achieve effective light enhancement. In addition, we explore a frequency-domain based and semantically guided appearance reconstruction module that encourages feature alignment of the recovered image at a fine-grained level and satisfies subjective expectations. Finally, extensive experiments demonstrate the superiority of our approach to other state-of-the-art methods and more significant generalization capabilities. We will open the source code upon acceptance of the paper.

Title: MathScale: Scaling Instruction Tuning for Mathematical Reasoning

• Authors: Zhengyang Tang, Xingxing Zhang, Benyou Wan, Furu Wei
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02884
• Pdf URL: https://arxiv.org/pdf/2403.02884
• Copy Paste: [[2403.02884]] MathScale: Scaling Instruction Tuning for Mathematical Reasoning(https://arxiv.org/abs/2403.02884)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in problem-solving. However, their proficiency in solving mathematical problems remains inadequate. We propose MathScale, a simple and scalable method to create high-quality mathematical reasoning data using frontier LLMs (e.g., {\tt GPT-3.5}). Inspired by the cognitive mechanism in human mathematical learning, it first extracts topics and knowledge points from seed math questions and then build a concept graph, which is subsequently used to generate new math questions. MathScale exhibits effective scalability along the size axis of the math dataset that we generate. As a result, we create a mathematical reasoning dataset (MathScaleQA) containing two million math question-answer pairs. To evaluate mathematical reasoning abilities of LLMs comprehensively, we construct {\sc MwpBench}, a benchmark of Math Word Problems, which is a collection of ten datasets (including GSM8K and MATH) covering K-12, college, and competition level math problems. We apply MathScaleQA to fine-tune open-source LLMs (e.g., LLaMA-2 and Mistral), resulting in significantly improved capabilities in mathematical reasoning. Evaluated on {\sc MwpBench}, MathScale-7B achieves state-of-the-art performance across all datasets, surpassing its best peers of equivalent size by 42.9\% in micro average accuracy and 43.7\% in macro average accuracy, respectively.

Title: Enhancing the Rate-Distortion-Perception Flexibility of Learned Image Codecs with Conditional Diffusion Decoders

• Authors: Daniele Mari, Simone Milani
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.02887
• Pdf URL: https://arxiv.org/pdf/2403.02887
• Copy Paste: [[2403.02887]] Enhancing the Rate-Distortion-Perception Flexibility of Learned Image Codecs with Conditional Diffusion Decoders(https://arxiv.org/abs/2403.02887)
• Keywords: diffusion, generative
• Abstract: Learned image compression codecs have recently achieved impressive compression performances surpassing the most efficient image coding architectures. However, most approaches are trained to minimize rate and distortion which often leads to unsatisfactory visual results at low bitrates since perceptual metrics are not taken into account. In this paper, we show that conditional diffusion models can lead to promising results in the generative compression task when used as a decoder, and that, given a compressed representation, they allow creating new tradeoff points between distortion and perception at the decoder side based on the sampling method.

Title: In Search of Truth: An Interrogation Approach to Hallucination Detection

• Authors: Yakir Yehuda, Itzik Malkiel, Oren Barkan, Jonathan Weill, Royi Ronen, Noam Koenigstein
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02889
• Pdf URL: https://arxiv.org/pdf/2403.02889
• Copy Paste: [[2403.02889]] In Search of Truth: An Interrogation Approach to Hallucination Detection(https://arxiv.org/abs/2403.02889)
• Keywords: large language model
• Abstract: Despite the many advances of Large Language Models (LLMs) and their unprecedented rapid evolution, their impact and integration into every facet of our daily lives is limited due to various reasons. One critical factor hindering their widespread adoption is the occurrence of hallucinations, where LLMs invent answers that sound realistic, yet drift away from factual truth. In this paper, we present a novel method for detecting hallucinations in large language models, which tackles a critical issue in the adoption of these models in various real-world scenarios. Through extensive evaluations across multiple datasets and LLMs, including Llama-2, we study the hallucination levels of various recent LLMs and demonstrate the effectiveness of our method to automatically detect them. Notably, we observe up to 62% hallucinations for Llama-2 in a specific experiment, where our method achieves a Balanced Accuracy (B-ACC) of 87%, all without relying on external knowledge.

Title: Enhancing Long-Term Person Re-Identification Using Global, Local Body Part, and Head Streams

• Authors: Duy Tran Thanh, Yeejin Lee, Byeongkeun Kang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02892
• Pdf URL: https://arxiv.org/pdf/2403.02892
• Copy Paste: [[2403.02892]] Enhancing Long-Term Person Re-Identification Using Global, Local Body Part, and Head Streams(https://arxiv.org/abs/2403.02892)
• Keywords: segmentation
• Abstract: This work addresses the task of long-term person re-identification. Typically, person re-identification assumes that people do not change their clothes, which limits its applications to short-term scenarios. To overcome this limitation, we investigate long-term person re-identification, which considers both clothes-changing and clothes-consistent scenarios. In this paper, we propose a novel framework that effectively learns and utilizes both global and local information. The proposed framework consists of three streams: global, local body part, and head streams. The global and head streams encode identity-relevant information from an entire image and a cropped image of the head region, respectively. Both streams encode the most distinct, less distinct, and average features using the combinations of adversarial erasing, max pooling, and average pooling. The local body part stream extracts identity-related information for each body part, allowing it to be compared with the same body part from another image. Since body part annotations are not available in re-identification datasets, pseudo-labels are generated using clustering. These labels are then utilized to train a body part segmentation head in the local body part stream. The proposed framework is trained by backpropagating the weighted summation of the identity classification loss, the pair-based loss, and the pseudo body part segmentation loss. To demonstrate the effectiveness of the proposed method, we conducted experiments on three publicly available datasets (Celeb-reID, PRCC, and VC-Clothes). The experimental results demonstrate that the proposed method outperforms the previous state-of-the-art method.

Title: ImgTrojan: Jailbreaking Vision-Language Models with ONE Image

• Authors: Xijia Tao, Shuai Zhong, Lei Li, Qi Liu, Lingpeng Kong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02910
• Pdf URL: https://arxiv.org/pdf/2403.02910
• Copy Paste: [[2403.02910]] ImgTrojan: Jailbreaking Vision-Language Models with ONE Image(https://arxiv.org/abs/2403.02910)
• Keywords: attack, steal, large language model
• Abstract: There has been an increasing interest in the alignment of large language models (LLMs) with human values. However, the safety issues of their integration with a vision module, or vision language models (VLMs), remain relatively underexplored. In this paper, we propose a novel jailbreaking attack against VLMs, aiming to bypass their safety barrier when a user inputs harmful instructions. A scenario where our poisoned (image, text) data pairs are included in the training data is assumed. By replacing the original textual captions with malicious jailbreak prompts, our method can perform jailbreak attacks with the poisoned images. Moreover, we analyze the effect of poison ratios and positions of trainable parameters on our attack's success rate. For evaluation, we design two metrics to quantify the success rate and the stealthiness of our attack. Together with a list of curated harmful instructions, a benchmark for measuring attack efficacy is provided. We demonstrate the efficacy of our attack by comparing it with baseline methods.

Title: Cross-Domain Image Conversion by CycleDM

• Authors: Sho Shimotsumagari, Shumpei Takezaki, Daichi Haraguchi, Seiichi Uchida
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02919
• Pdf URL: https://arxiv.org/pdf/2403.02919
• Copy Paste: [[2403.02919]] Cross-Domain Image Conversion by CycleDM(https://arxiv.org/abs/2403.02919)
• Keywords: diffusion
• Abstract: The purpose of this paper is to enable the conversion between machine-printed character images (i.e., font images) and handwritten character images through machine learning. For this purpose, we propose a novel unpaired image-to-image domain conversion method, CycleDM, which incorporates the concept of CycleGAN into the diffusion model. Specifically, CycleDM has two internal conversion models that bridge the denoising processes of two image domains. These conversion models are efficiently trained without explicit correspondence between the domains. By applying machine-printed and handwritten character images to the two modalities, CycleDM realizes the conversion between them. Our experiments for evaluating the converted images quantitatively and qualitatively found that ours performs better than other comparable approaches.

Title: TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax

• Authors: Tobias Christian Nauen, Sebastian Palacio, Andreas Dengel
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02920
• Pdf URL: https://arxiv.org/pdf/2403.02920
• Copy Paste: [[2403.02920]] TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax(https://arxiv.org/abs/2403.02920)
• Keywords: transformer
• Abstract: The quadratic complexity of the attention mechanism represents one of the biggest hurdles for processing long sequences using Transformers. Current methods, relying on sparse representations or stateful recurrence, sacrifice token-to-token interactions, which ultimately leads to compromises in performance. This paper introduces TaylorShift, a novel reformulation of the Taylor softmax that enables computing full token-to-token interactions in linear time and space. We analytically determine the crossover points where employing TaylorShift becomes more efficient than traditional attention, aligning closely with empirical measurements. Specifically, our findings demonstrate that TaylorShift enhances memory efficiency for sequences as short as 800 tokens and accelerates inference for inputs of approximately 1700 tokens and beyond. For shorter sequences, TaylorShift scales comparably with the vanilla attention. Furthermore, a classification benchmark across five tasks involving long sequences reveals no degradation in accuracy when employing Transformers equipped with TaylorShift. For reproducibility, we provide access to our code under https://github.com/tobna/TaylorShift.

Title: From Spectra to Biophysical Insights: End-to-End Learning with a Biased Radiative Transfer Model

• Authors: Yihang She, Clement Atzberger, Andrew Blake, Srinivasan Keshav
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02922
• Pdf URL: https://arxiv.org/pdf/2403.02922
• Copy Paste: [[2403.02922]] From Spectra to Biophysical Insights: End-to-End Learning with a Biased Radiative Transfer Model(https://arxiv.org/abs/2403.02922)
• Keywords: interpretability
• Abstract: Advances in machine learning have boosted the use of Earth observation data for climate change research. Yet, the interpretability of machine-learned representations remains a challenge, particularly in understanding forests' biophysical reactions to climate change. Traditional methods in remote sensing that invert radiative transfer models (RTMs) to retrieve biophysical variables from spectral data often fail to account for biases inherent in the RTM, especially for complex forests. We propose to integrate RTMs into an auto-encoder architecture, creating an end-to-end learning approach. Our method not only corrects biases in RTMs but also outperforms traditional techniques for variable retrieval like neural network regression. Furthermore, our framework has potential generally for inverting biased physical models. The code is available on https://github.com/yihshe/ai-refined-rtm.git.

Title: RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules

• Authors: Miaomiao Li, Jiaqi Zhu, Yang Wang, Yi Yang, Yilin Li, Hongan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02932
• Pdf URL: https://arxiv.org/pdf/2403.02932
• Copy Paste: [[2403.02932]] RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules(https://arxiv.org/abs/2403.02932)
• Keywords: robust
• Abstract: Weakly supervised text classification (WSTC), also called zero-shot or dataless text classification, has attracted increasing attention due to its applicability in classifying a mass of texts within the dynamic and open Web environment, since it requires only a limited set of seed words (label names) for each category instead of labeled data. With the help of recently popular prompting Pre-trained Language Models (PLMs), many studies leveraged manually crafted and/or automatically identified verbalizers to estimate the likelihood of categories, but they failed to differentiate the effects of these category-indicative words, let alone capture their correlations and realize adaptive adjustments according to the unlabeled corpus. In this paper, in order to let the PLM effectively understand each category, we at first propose a novel form of rule-based knowledge using logical expressions to characterize the meanings of categories. Then, we develop a prompting PLM-based approach named RulePrompt for the WSTC task, consisting of a rule mining module and a rule-enhanced pseudo label generation module, plus a self-supervised fine-tuning module to make the PLM align with this task. Within this framework, the inaccurate pseudo labels assigned to texts and the imprecise logical rules associated with categories mutually enhance each other in an alternative manner. That establishes a self-iterative closed loop of knowledge (rule) acquisition and utilization, with seed words serving as the starting point. Extensive experiments validate the effectiveness and robustness of our approach, which markedly outperforms state-of-the-art weakly supervised methods. What is more, our approach yields interpretable category rules, proving its advantage in disambiguating easily-confused categories.

Title: Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity

• Authors: Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, Jaeho Lee
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02944
• Pdf URL: https://arxiv.org/pdf/2403.02944
• Copy Paste: [[2403.02944]] Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity(https://arxiv.org/abs/2403.02944)
• Keywords: generative
• Abstract: Recent advances in text-guided image compression have shown great potential to enhance the perceptual quality of reconstructed images. These methods, however, tend to have significantly degraded pixel-wise fidelity, limiting their practicality. To fill this gap, we develop a new text-guided image compression algorithm that achieves both high perceptual and pixel-wise fidelity. In particular, we propose a compression framework that leverages text information mainly by text-adaptive encoding and training with joint image-text loss. By doing so, we avoid decoding based on text-guided generative models -- known for high generative diversity -- and effectively utilize the semantic information of text at a global level. Experimental results on various datasets show that our method can achieve high pixel-level and perceptual quality, with either human- or machine-generated captions. In particular, our method outperforms all baselines in terms of LPIPS, with some room for even more improvements when we use more carefully generated captions.

Title: Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation

• Authors: Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, Sun Yang, Chi Harold Liu, Rui Zhao, Ziyue Li, Hangyu Mao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02951
• Pdf URL: https://arxiv.org/pdf/2403.02951
• Copy Paste: [[2403.02951]] Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation(https://arxiv.org/abs/2403.02951)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to-SQL task, significantly outperforming traditional methods. Nevertheless, as a nascent research field, there is still no consensus on the optimal prompt templates and design frameworks. Additionally, existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to-SQL process, which hinders the assessment of LLMs' cognitive capabilities and the optimization of LLM-based solutions.To address the aforementioned issues, we firstly construct a new dataset designed to mitigate the risk of overfitting in LLMs. Then we formulate five evaluation tasks to comprehensively assess the performance of diverse methods across various LLMs throughout the Text-to-SQL process.Our study highlights the performance disparities among LLMs and proposes optimal in-context learning solutions tailored to each task. These findings offer valuable insights for enhancing the development of LLM-based Text-to-SQL systems.

Title: XAI-Based Detection of Adversarial Attacks on Deepfake Detectors

• Authors: Ben Pinhasov, Raz Lapid, Rony Ohayon, Moshe Sipper, Yehudit Aperstein
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02955
• Pdf URL: https://arxiv.org/pdf/2403.02955
• Copy Paste: [[2403.02955]] XAI-Based Detection of Adversarial Attacks on Deepfake Detectors(https://arxiv.org/abs/2403.02955)
• Keywords: attack, interpretability
• Abstract: We introduce a novel methodology for identifying adversarial attacks on deepfake detectors using eXplainable Artificial Intelligence (XAI). In an era characterized by digital advancement, deepfakes have emerged as a potent tool, creating a demand for efficient detection systems. However, these systems are frequently targeted by adversarial attacks that inhibit their performance. We address this gap, developing a defensible deepfake detector by leveraging the power of XAI. The proposed methodology uses XAI to generate interpretability maps for a given method, providing explicit visualizations of decision-making factors within the AI models. We subsequently employ a pretrained feature extractor that processes both the input image and its corresponding XAI image. The feature embeddings extracted from this process are then used for training a simple yet effective classifier. Our approach contributes not only to the detection of deepfakes but also enhances the understanding of possible adversarial attacks, pinpointing potential vulnerabilities. Furthermore, this approach does not change the performance of the deepfake detector. The paper demonstrates promising results suggesting a potential pathway for future deepfake detection mechanisms. We believe this study will serve as a valuable contribution to the community, sparking much-needed discourse on safeguarding deepfake detectors.

Title: On the Asymptotic Mean Square Error Optimality of Diffusion Probabilistic Models

• Authors: Benedikt Fesl, Benedikt Böck, Florian Strasser, Michael Baur, Michael Joham, Wolfgang Utschick
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.02957
• Pdf URL: https://arxiv.org/pdf/2403.02957
• Copy Paste: [[2403.02957]] On the Asymptotic Mean Square Error Optimality of Diffusion Probabilistic Models(https://arxiv.org/abs/2403.02957)
• Keywords: diffusion
• Abstract: Diffusion probabilistic models (DPMs) have recently shown great potential for denoising tasks. Despite their practical utility, there is a notable gap in their theoretical understanding. This paper contributes novel theoretical insights by rigorously proving the asymptotic convergence of a specific DPM denoising strategy to the mean square error (MSE)-optimal conditional mean estimator (CME) over a large number of diffusion steps. The studied DPM-based denoiser shares the training procedure of DPMs but distinguishes itself by forwarding only the conditional mean during the reverse inference process after training. We highlight the unique perspective that DPMs are composed of an asymptotically optimal denoiser while simultaneously inheriting a powerful generator by switching re-sampling in the reverse process on and off. The theoretical findings are validated by numerical results.

Title: SimuCourt: Building Judicial Decision-Making Agents with Real-world Judgement Documents

• Authors: Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, Jun Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02959
• Pdf URL: https://arxiv.org/pdf/2403.02959
• Copy Paste: [[2403.02959]] SimuCourt: Building Judicial Decision-Making Agents with Real-world Judgement Documents(https://arxiv.org/abs/2403.02959)
• Keywords: large language model
• Abstract: With the development of deep learning, natural language processing technology has effectively improved the efficiency of various aspects of the traditional judicial industry. However, most current efforts focus solely on individual judicial stage, overlooking cross-stage collaboration. As the autonomous agents powered by large language models are becoming increasingly smart and able to make complex decisions in real-world settings, offering new insights for judicial intelligence. In this paper, (1) we introduce SimuCourt, a judicial benchmark that encompasses 420 judgment documents from real-world, spanning the three most common types of judicial cases, and a novel task Judicial Decision-Making to evaluate the judicial analysis and decision-making power of agents. To support this task, we construct a large-scale judicial knowledge base, JudicialKB, with multiple legal knowledge. (2) we propose a novel multi-agent framework, AgentsCourt. Our framework follows the real-world classic court trial process, consisting of court debate simulation, legal information retrieval and judgement refinement to simulate the decision-making of judge. (3) we perform extensive experiments, the results demonstrate that, our framework outperforms the existing advanced methods in various aspects, especially in generating legal grounds, where our model achieves significant improvements of 8.6% and 9.1% F1 score in the first and second instance settings, respectively.

Title: ChatGPT and biometrics: an assessment of face recognition, gender detection, and age estimation capabilities

• Authors: Ahmad Hassanpour, Yasamin Kowsari, Hatef Otroshi Shahreza, Bian Yang, Sebastien Marcel
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02965
• Pdf URL: https://arxiv.org/pdf/2403.02965
• Copy Paste: [[2403.02965]] ChatGPT and biometrics: an assessment of face recognition, gender detection, and age estimation capabilities(https://arxiv.org/abs/2403.02965)
• Keywords: biometric, large language model
• Abstract: This paper explores the application of large language models (LLMs), like ChatGPT, for biometric tasks. We specifically examine the capabilities of ChatGPT in performing biometric-related tasks, with an emphasis on face recognition, gender detection, and age estimation. Since biometrics are considered as sensitive information, ChatGPT avoids answering direct prompts, and thus we crafted a prompting strategy to bypass its safeguard and evaluate the capabilities for biometrics tasks. Our study reveals that ChatGPT recognizes facial identities and differentiates between two facial images with considerable accuracy. Additionally, experimental results demonstrate remarkable performance in gender detection and reasonable accuracy for the age estimation tasks. Our findings shed light on the promising potentials in the application of LLMs and foundation models for biometrics.

Title: Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering

• Authors: Sungho Ko, Hyunjin Cho, Hyungjoo Chae, Jinyoung Yeo, Dongha Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02966
• Pdf URL: https://arxiv.org/pdf/2403.02966
• Copy Paste: [[2403.02966]] Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering(https://arxiv.org/abs/2403.02966)
• Keywords: large language model
• Abstract: Recent studies have investigated utilizing Knowledge Graphs (KGs) to enhance Quesetion Answering (QA) performance of Large Language Models (LLMs), yet structured KG verbalization remains challengin. Existing methods, such as triple-form or free-form textual conversion of triple-form facts, encounter several issues. These include reduced evidence density due to duplicated entities or relationships, and reduced evidence clarity due to an inability to emphasize crucial evidence. To address these issues, we propose EFSum, an Evidence-focused Fact Summarization framework for enhanced QA with knowledge-augmented LLMs. We optimize an open-source LLM as a fact summarizer through distillation and preference alignment. Our extensive experiments show that EFSum improves LLM's zero-shot QA performance, and it is possible to ensure both the helpfulness and faithfulness of the summary.

Title: Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

• Authors: Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan He, Jin-Peng Lan, Bin Luo, Xuansong Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02969
• Pdf URL: https://arxiv.org/pdf/2403.02969
• Copy Paste: [[2403.02969]] Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception(https://arxiv.org/abs/2403.02969)
• Keywords: large language model, segmentation
• Abstract: Multimodal Large Language Model (MLLMs) leverages Large Language Models as a cognitive framework for diverse visual-language tasks. Recent efforts have been made to equip MLLMs with visual perceiving and grounding capabilities. However, there still remains a gap in providing fine-grained pixel-level perceptions and extending interactions beyond text-specific inputs. In this work, we propose {\bf{AnyRef}}, a general MLLM model that can generate pixel-wise object perceptions and natural language descriptions from multi-modality references, such as texts, boxes, images, or audio. This innovation empowers users with greater flexibility to engage with the model beyond textual and regional prompts, without modality-specific designs. Through our proposed refocusing mechanism, the generated grounding output is guided to better focus on the referenced object, implicitly incorporating additional pixel-level supervision. This simple modification utilizes attention scores generated during the inference of LLM, eliminating the need for extra computations while exhibiting performance enhancements in both grounding masks and referring expressions. With only publicly available training data, our model achieves state-of-the-art results across multiple benchmarks, including diverse modality referring segmentation and region-level referring expression generation.

Title: Federated Learning Under Attack: Exposing Vulnerabilities through Data Poisoning Attacks in Computer Networks

• Authors: Ehsan Nowroozi, Imran Haider, Rahim Taheri, Mauro Conti
• Subjects: cs.CR, cs.AI, cs.CY, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.02983
• Pdf URL: https://arxiv.org/pdf/2403.02983
• Copy Paste: [[2403.02983]] Federated Learning Under Attack: Exposing Vulnerabilities through Data Poisoning Attacks in Computer Networks(https://arxiv.org/abs/2403.02983)
• Keywords: attack, federate
• Abstract: Federated Learning (FL) is a machine learning (ML) approach that enables multiple decentralized devices or edge servers to collaboratively train a shared model without exchanging raw data. During the training and sharing of model updates between clients and servers, data and models are susceptible to different data-poisoning attacks. In this study, our motivation is to explore the severity of data poisoning attacks in the computer network domain because they are easy to implement but difficult to detect. We considered two types of data-poisoning attacks, label flipping (LF) and feature poisoning (FP), and applied them with a novel approach. In LF, we randomly flipped the labels of benign data and trained the model on the manipulated data. For FP, we randomly manipulated the highly contributing features determined using the Random Forest algorithm. The datasets used in this experiment were CIC and UNSW related to computer networks. We generated adversarial samples using the two attacks mentioned above, which were applied to a small percentage of datasets. Subsequently, we trained and tested the accuracy of the model on adversarial datasets. We recorded the results for both benign and manipulated datasets and observed significant differences between the accuracy of the models on different datasets. From the experimental results, it is evident that the LF attack failed, whereas the FP attack showed effective results, which proved its significance in fooling a server. With a 1% LF attack on the CIC, the accuracy was approximately 0.0428 and the ASR was 0.9564; hence, the attack is easily detectable, while with a 1% FP attack, the accuracy and ASR were both approximately 0.9600, hence, FP attacks are difficult to detect. We repeated the experiment with different poisoning percentages.

Title: Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges

• Authors: Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, Shafiq Joty
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02990
• Pdf URL: https://arxiv.org/pdf/2403.02990
• Copy Paste: [[2403.02990]] Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges(https://arxiv.org/abs/2403.02990)
• Keywords: large language model
• Abstract: In the rapidly evolving field of machine learning (ML), data augmentation (DA) has emerged as a pivotal technique for enhancing model performance by diversifying training examples without the need for additional data collection. This survey explores the transformative impact of Large Language Models (LLMs) on DA, particularly addressing the unique challenges and opportunities they present in the context of natural language processing (NLP) and beyond. From a data perspective and a learning perspective, we examine various strategies that utilize Large Language Models for data augmentation, including a novel exploration of learning paradigms where LLM-generated data is used for further training. Additionally, this paper delineates the primary challenges faced in this domain, ranging from controllable data augmentation to multi modal data augmentation. This survey highlights the paradigm shift introduced by LLMs in DA, aims to serve as a foundational guide for researchers and practitioners in this field.

Title: MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer

• Authors: Jianjian Cao, Peng Ye, Shengze Li, Chong Yu, Yansong Tang, Jiwen Lu, Tao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02991
• Pdf URL: https://arxiv.org/pdf/2403.02991
• Copy Paste: [[2403.02991]] MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer(https://arxiv.org/abs/2403.02991)
• Keywords: transformer
• Abstract: Vision-Language Transformers (VLTs) have shown great success recently, but are meanwhile accompanied by heavy computation costs, where a major reason can be attributed to the large number of visual and language tokens. Existing token pruning research for compressing VLTs mainly follows a single-modality-based scheme yet ignores the critical role of aligning different modalities for guiding the token pruning process, causing the important tokens for one modality to be falsely pruned in another modality branch. Meanwhile, existing VLT pruning works also lack the flexibility to dynamically compress each layer based on different input samples. To this end, we propose a novel framework named Multimodal Alignment-Guided Dynamic Token Pruning (MADTP) for accelerating various VLTs. Specifically, we first introduce a well-designed Multi-modality Alignment Guidance (MAG) module that can align features of the same semantic concept from different modalities, to ensure the pruned tokens are less important for all modalities. We further design a novel Dynamic Token Pruning (DTP) module, which can adaptively adjust the token compression ratio in each layer based on different input instances. Extensive experiments on various benchmarks demonstrate that MADTP significantly reduces the computational complexity of kinds of multimodal models while preserving competitive performance. Notably, when applied to the BLIP model in the NLVR2 dataset, MADTP can reduce the GFLOPs by 80% with less than 4% performance degradation.

Title: Mitigating Label Flipping Attacks in Malicious URL Detectors Using Ensemble Trees

• Authors: Ehsan Nowroozi, Nada Jadalla, Samaneh Ghelichkhani, Alireza Jolfaei
• Subjects: cs.CR, cs.AI, cs.CY, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.02995
• Pdf URL: https://arxiv.org/pdf/2403.02995
• Copy Paste: [[2403.02995]] Mitigating Label Flipping Attacks in Malicious URL Detectors Using Ensemble Trees(https://arxiv.org/abs/2403.02995)
• Keywords: security, defense, attack
• Abstract: Malicious URLs provide adversarial opportunities across various industries, including transportation, healthcare, energy, and banking which could be detrimental to business operations. Consequently, the detection of these URLs is of crucial importance; however, current Machine Learning (ML) models are susceptible to backdoor attacks. These attacks involve manipulating a small percentage of training data labels, such as Label Flipping (LF), which changes benign labels to malicious ones and vice versa. This manipulation results in misclassification and leads to incorrect model behavior. Therefore, integrating defense mechanisms into the architecture of ML models becomes an imperative consideration to fortify against potential attacks. The focus of this study is on backdoor attacks in the context of URL detection using ensemble trees. By illuminating the motivations behind such attacks, highlighting the roles of attackers, and emphasizing the critical importance of effective defense strategies, this paper contributes to the ongoing efforts to fortify ML models against adversarial threats within the ML domain in network security. We propose an innovative alarm system that detects the presence of poisoned labels and a defense mechanism designed to uncover the original class labels with the aim of mitigating backdoor attacks on ensemble tree classifiers. We conducted a case study using the Alexa and Phishing Site URL datasets and showed that LF attacks can be addressed using our proposed defense mechanism. Our experimental results prove that the LF attack achieved an Attack Success Rate (ASR) between 50-65% within 2-5%, and the innovative defense method successfully detected poisoned labels with an accuracy of up to 100%.

Title: Towards Calibrated Deep Clustering Network

• Authors: Yuheng Jia, Jianhong Cheng, Hui Liu, Junhui Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02998
• Pdf URL: https://arxiv.org/pdf/2403.02998
• Copy Paste: [[2403.02998]] Towards Calibrated Deep Clustering Network(https://arxiv.org/abs/2403.02998)
• Keywords: robust
• Abstract: Deep clustering has exhibited remarkable performance; however, the overconfidence problem, i.e., the estimated confidence for a sample belonging to a particular cluster greatly exceeds its actual prediction accuracy, has been overlooked in prior research. To tackle this critical issue, we pioneer the development of a calibrated deep clustering framework. Specifically, we propose a novel dual-head deep clustering pipeline that can effectively calibrate the estimated confidence and the actual accuracy. The calibration head adjusts the overconfident predictions of the clustering head using regularization methods, generating prediction confidence and pseudo-labels that match the model learning status. This calibration process also guides the clustering head in dynamically selecting reliable high-confidence samples for training. Additionally, we introduce an effective network initialization strategy that enhances both training speed and network robustness. Extensive experiments demonstrate the proposed calibrated deep clustering framework not only surpasses state-of-the-art deep clustering methods by approximately 10 times in terms of expected calibration error but also significantly outperforms them in terms of clustering accuracy.

Title: Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models

• Authors: Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03003
• Pdf URL: https://arxiv.org/pdf/2403.03003
• Copy Paste: [[2403.03003]] Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models(https://arxiv.org/abs/2403.03003)
• Keywords: large language model
• Abstract: Despite remarkable progress, existing multimodal large language models (MLLMs) are still inferior in granular visual recognition. Contrary to previous works, we study this problem from the perspective of image resolution, and reveal that a combination of low- and high-resolution visual features can effectively mitigate this shortcoming. Based on this observation, we propose a novel and efficient method for MLLMs, termed Mixture-of-Resolution Adaptation (MRA). In particular, MRA adopts two visual pathways for images with different resolutions, where high-resolution visual information is embedded into the low-resolution pathway via the novel mixture-of-resolution adapters (MR-Adapters). This design also greatly reduces the input sequence length of MLLMs. To validate MRA, we apply it to a recent MLLM called LLaVA, and term the new model LLaVA-HR. We conduct extensive experiments on 11 vision-language (VL) tasks, which show that LLaVA-HR outperforms existing MLLMs on 8 VL tasks, e.g., +9.4% on TextVQA. More importantly, both training and inference of LLaVA-HR remain efficient with MRA, e.g., 20 training hours and 3$\times$ inference speed than LLaVA-1.5. Source codes are released at: https://github.com/luogen1996/LLaVA-HR.

Title: CRISPR: Ensemble Model

• Authors: Mohammad Rostami, Amin Ghariyazi, Hamed Dashti, Mohammad Hossein Rohban, Hamid R. Rabiee
• Subjects: cs.LG, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2403.03018
• Pdf URL: https://arxiv.org/pdf/2403.03018
• Copy Paste: [[2403.03018]] CRISPR: Ensemble Model(https://arxiv.org/abs/2403.03018)
• Keywords: robust
• Abstract: Clustered Regularly Interspaced Short Palindromic Repeats (CRISPR) is a gene editing technology that has revolutionized the fields of biology and medicine. However, one of the challenges of using CRISPR is predicting the on-target efficacy and off-target sensitivity of single-guide RNAs (sgRNAs). This is because most existing methods are trained on separate datasets with different genes and cells, which limits their generalizability. In this paper, we propose a novel ensemble learning method for sgRNA design that is accurate and generalizable. Our method combines the predictions of multiple machine learning models to produce a single, more robust prediction. This approach allows us to learn from a wider range of data, which improves the generalizability of our model. We evaluated our method on a benchmark dataset of sgRNA designs and found that it outperformed existing methods in terms of both accuracy and generalizability. Our results suggest that our method can be used to design sgRNAs with high sensitivity and specificity, even for new genes or cells. This could have important implications for the clinical use of CRISPR, as it would allow researchers to design more effective and safer treatments for a variety of diseases.

Title: Socratic Reasoning Improves Positive Text Rewriting

• Authors: Anmol Goel, Nico Daheim, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03029
• Pdf URL: https://arxiv.org/pdf/2403.03029
• Copy Paste: [[2403.03029]] Socratic Reasoning Improves Positive Text Rewriting(https://arxiv.org/abs/2403.03029)
• Keywords: large language model
• Abstract: Reframing a negative into a positive thought is at the crux of several cognitive approaches to mental health and psychotherapy that could be made more accessible by large language model-based solutions. Such reframing is typically non-trivial and requires multiple rationalization steps to uncover the underlying issue of a negative thought and transform it to be more positive. However, this rationalization process is currently neglected by both datasets and models which reframe thoughts in one step. In this work, we address this gap by augmenting open-source datasets for positive text rewriting with synthetically-generated Socratic rationales using a novel framework called \textsc{SocraticReframe}. \textsc{SocraticReframe} uses a sequence of question-answer pairs to rationalize the thought rewriting process. We show that such Socratic rationales significantly improve positive text rewriting for different open-source LLMs according to both automatic and human evaluations guided by criteria from psychotherapy research.

Title: Learning to Use Tools via Cooperative and Interactive Agents

• Authors: Zhengliang Shi, Shen Gao, Xiuyi Chen, Lingyong Yan, Haibo Shi, Dawei Yin, Zhumin Chen, Pengjie Ren, Suzan Verberne, Zhaochun Ren
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03031
• Pdf URL: https://arxiv.org/pdf/2403.03031
• Copy Paste: [[2403.03031]] Learning to Use Tools via Cooperative and Interactive Agents(https://arxiv.org/abs/2403.03031)
• Keywords: large language model
• Abstract: Tool learning empowers large language models (LLMs) as agents to use external tools to extend their capability. Existing methods employ one single LLM-based agent to iteratively select and execute tools, thereafter incorporating the result into the next action prediction. However, they still suffer from potential performance degradation when addressing complex tasks due to: (1) the limitation of the inherent capability of a single LLM to perform diverse actions, and (2) the struggle to adaptively correct mistakes when the task fails. To mitigate these problems, we propose the ConAgents, a Cooperative and interactive Agents framework, which modularizes the workflow of tool learning into Grounding, Execution, and Observing agents. We also introduce an iterative calibration (IterCali) method, enabling the agents to adapt themselves based on the feedback from the tool environment. Experiments conducted on three datasets demonstrate the superiority of our ConAgents (e.g., 6 point improvement over the SOTA baseline). We further provide fine-granularity analysis for the efficiency and consistency of our framework.

Title: CrackNex: a Few-shot Low-light Crack Segmentation Model Based on Retinex Theory for UAV Inspections

• Authors: Zhen Yao, Jiawei Xu, Shuhang Hou, Mooi Choo Chuah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03063
• Pdf URL: https://arxiv.org/pdf/2403.03063
• Copy Paste: [[2403.03063]] CrackNex: a Few-shot Low-light Crack Segmentation Model Based on Retinex Theory for UAV Inspections(https://arxiv.org/abs/2403.03063)
• Keywords: segmentation
• Abstract: Routine visual inspections of concrete structures are imperative for upholding the safety and integrity of critical infrastructure. Such visual inspections sometimes happen under low-light conditions, e.g., checking for bridge health. Crack segmentation under such conditions is challenging due to the poor contrast between cracks and their surroundings. However, most deep learning methods are designed for well-illuminated crack images and hence their performance drops dramatically in low-light scenes. In addition, conventional approaches require many annotated low-light crack images which is time-consuming. In this paper, we address these challenges by proposing CrackNex, a framework that utilizes reflectance information based on Retinex Theory to help the model learn a unified illumination-invariant representation. Furthermore, we utilize few-shot segmentation to solve the inefficient training data problem. In CrackNex, both a support prototype and a reflectance prototype are extracted from the support set. Then, a prototype fusion module is designed to integrate the features from both prototypes. CrackNex outperforms the SOTA methods on multiple datasets. Additionally, we present the first benchmark dataset, LCSD, for low-light crack segmentation. LCSD consists of 102 well-illuminated crack images and 41 low-light crack images. The dataset and code are available at https://github.com/zy1296/CrackNex.

Title: MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Grounding

• Authors: Chun-Peng Chang, Shaoxiang Wang, Alain Pagani, Didier Stricker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03077
• Pdf URL: https://arxiv.org/pdf/2403.03077
• Copy Paste: [[2403.03077]] MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Grounding(https://arxiv.org/abs/2403.03077)
• Keywords: explainability, transformer
• Abstract: 3D visual grounding involves matching natural language descriptions with their corresponding objects in 3D spaces. Existing methods often face challenges with accuracy in object recognition and struggle in interpreting complex linguistic queries, particularly with descriptions that involve multiple anchors or are view-dependent. In response, we present the MiKASA (Multi-Key-Anchor Scene-Aware) Transformer. Our novel end-to-end trained model integrates a self-attention-based scene-aware object encoder and an original multi-key-anchor technique, enhancing object recognition accuracy and the understanding of spatial relationships. Furthermore, MiKASA improves the explainability of decision-making, facilitating error diagnosis. Our model achieves the highest overall accuracy in the Referit3D challenge for both the Sr3D and Nr3D datasets, particularly excelling by a large margin in categories that require viewpoint-dependent descriptions. The source code and additional resources for this project are available on GitHub: https://github.com/birdy666/MiKASA-3DVG

Title: Recall-Oriented Continual Learning with Generative Adversarial Meta-Model

• Authors: Haneol Kang, Dong-Wan Choi
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03082
• Pdf URL: https://arxiv.org/pdf/2403.03082
• Copy Paste: [[2403.03082]] Recall-Oriented Continual Learning with Generative Adversarial Meta-Model(https://arxiv.org/abs/2403.03082)
• Keywords: generative
• Abstract: The stability-plasticity dilemma is a major challenge in continual learning, as it involves balancing the conflicting objectives of maintaining performance on previous tasks while learning new tasks. In this paper, we propose the recall-oriented continual learning framework to address this challenge. Inspired by the human brain's ability to separate the mechanisms responsible for stability and plasticity, our framework consists of a two-level architecture where an inference network effectively acquires new knowledge and a generative network recalls past knowledge when necessary. In particular, to maximize the stability of past knowledge, we investigate the complexity of knowledge depending on different representations, and thereby introducing generative adversarial meta-model (GAMM) that incrementally learns task-specific parameters instead of input data samples of the task. Through our experiments, we show that our framework not only effectively learns new knowledge without any disruption but also achieves high stability of previous knowledge in both task-aware and task-agnostic learning scenarios. Our code is available at: https://github.com/bigdata-inha/recall-oriented-cl-framework.

Title: KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents

• Authors: Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Ningyu Zhang, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2403.03101
• Pdf URL: https://arxiv.org/pdf/2403.03101
• Copy Paste: [[2403.03101]] KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents(https://arxiv.org/abs/2403.03101)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated great potential in complex reasoning tasks, yet they fall short when tackling more sophisticated challenges, especially when interacting with environments through generating executable actions. This inadequacy primarily stems from the lack of built-in action knowledge in language agents, which fails to effectively guide the planning trajectories during task solving and results in planning hallucination. To address this issue, we introduce KnowAgent, a novel approach designed to enhance the planning capabilities of LLMs by incorporating explicit action knowledge. Specifically, KnowAgent employs an action knowledge base and a knowledgeable self-learning strategy to constrain the action path during planning, enabling more reasonable trajectory synthesis, and thereby enhancing the planning performance of language agents. Experimental results on HotpotQA and ALFWorld based on various backbone models demonstrate that KnowAgent can achieve comparable or superior performance to existing baselines. Further analysis indicates the effectiveness of KnowAgent in terms of planning hallucinations mitigation. Code is available in https://github.com/zjunlp/KnowAgent.

Title: "In Dialogues We Learn": Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning

• Authors: Chuanqi Cheng, Quan Tu, Wei Wu, Shuo Shang, Cunli Mao, Zhengtao Yu, Rui Yan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.03102
• Pdf URL: https://arxiv.org/pdf/2403.03102
• Copy Paste: [[2403.03102]] "In Dialogues We Learn": Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning(https://arxiv.org/abs/2403.03102)
• Keywords: large language model
• Abstract: Personalized dialogue systems have gained significant attention in recent years for their ability to generate responses in alignment with different personas. However, most existing approaches rely on pre-defined personal profiles, which are not only time-consuming and labor-intensive to create but also lack flexibility. We propose In-Dialogue Learning (IDL), a fine-tuning framework that enhances the ability of pre-trained large language models to leverage dialogue history to characterize persona for completing personalized dialogue generation tasks without pre-defined profiles. Our experiments on three datasets demonstrate that IDL brings substantial improvements, with BLEU and ROUGE scores increasing by up to 200% and 247%, respectively. Additionally, the results of human evaluations further validate the efficacy of our proposed method.

Title: Improved LiDAR Odometry and Mapping using Deep Semantic Segmentation and Novel Outliers Detection

• Authors: Mohamed Afifi, Mohamed ElHelw
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.03111
• Pdf URL: https://arxiv.org/pdf/2403.03111
• Copy Paste: [[2403.03111]] Improved LiDAR Odometry and Mapping using Deep Semantic Segmentation and Novel Outliers Detection(https://arxiv.org/abs/2403.03111)
• Keywords: robust, segmentation
• Abstract: Perception is a key element for enabling intelligent autonomous navigation. Understanding the semantics of the surrounding environment and accurate vehicle pose estimation are essential capabilities for autonomous vehicles, including self-driving cars and mobile robots that perform complex tasks. Fast moving platforms like self-driving cars impose a hard challenge for localization and mapping algorithms. In this work, we propose a novel framework for real-time LiDAR odometry and mapping based on LOAM architecture for fast moving platforms. Our framework utilizes semantic information produced by a deep learning model to improve point-to-line and point-to-plane matching between LiDAR scans and build a semantic map of the environment, leading to more accurate motion estimation using LiDAR data. We observe that including semantic information in the matching process introduces a new type of outlier matches to the process, where matching occur between different objects of the same semantic class. To this end, we propose a novel algorithm that explicitly identifies and discards potential outliers in the matching process. In our experiments, we study the effect of improving the matching process on the robustness of LiDAR odometry against high speed motion. Our experimental evaluations on KITTI dataset demonstrate that utilizing semantic information and rejecting outliers significantly enhance the robustness of LiDAR odometry and mapping when there are large gaps between scan acquisition poses, which is typical for fast moving platforms.

Title: Motion-Corrected Moving Average: Including Post-Hoc Temporal Information for Improved Video Segmentation

• Authors: Robert Mendel (1), Tobias Rueckert (1), Dirk Wilhelm (2), Daniel Rueckert (3, 4), Christoph Palm (1, 5) ((1) Regensburg Medical Image Computing (ReMIC), Ostbayerische Technische Hochschule Regensburg (OTH Regensburg), Regensburg, Germany, (2) Department of Surgery, Faculty of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany, (3) Artificial Intelligence in Healthcare and Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany, (4) Department of Computing, Imperial College London, London, UK, (5) Regensburg Center of Health Sciences and Technology (RCHST), OTH Regensburg, Regensburg, Germany)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03120
• Pdf URL: https://arxiv.org/pdf/2403.03120
• Copy Paste: [[2403.03120]] Motion-Corrected Moving Average: Including Post-Hoc Temporal Information for Improved Video Segmentation(https://arxiv.org/abs/2403.03120)
• Keywords: segmentation
• Abstract: Real-time computational speed and a high degree of precision are requirements for computer-assisted interventions. Applying a segmentation network to a medical video processing task can introduce significant inter-frame prediction noise. Existing approaches can reduce inconsistencies by including temporal information but often impose requirements on the architecture or dataset. This paper proposes a method to include temporal information in any segmentation model and, thus, a technique to improve video segmentation performance without alterations during training or additional labeling. With Motion-Corrected Moving Average, we refine the exponential moving average between the current and previous predictions. Using optical flow to estimate the movement between consecutive frames, we can shift the prior term in the moving-average calculation to align with the geometry of the current frame. The optical flow calculation does not require the output of the model and can therefore be performed in parallel, leading to no significant runtime penalty for our approach. We evaluate our approach on two publicly available segmentation datasets and two proprietary endoscopic datasets and show improvements over a baseline approach.

Title: Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution

• Authors: Flor Miriam Plaza-del-Arco, Amanda Cercas Curry, Alba Curry, Gavin Abercrombie, Dirk Hovy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03121
• Pdf URL: https://arxiv.org/pdf/2403.03121
• Copy Paste: [[2403.03121]] Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution(https://arxiv.org/abs/2403.03121)
• Keywords: large language model
• Abstract: Large language models (LLMs) reflect societal norms and biases, especially about gender. While societal biases and stereotypes have been extensively researched in various NLP applications, there is a surprising gap for emotion analysis. However, emotion and gender are closely linked in societal discourse. E.g., women are often thought of as more empathetic, while men's anger is more socially accepted. To fill this gap, we present the first comprehensive study of gendered emotion attribution in five state-of-the-art LLMs (open- and closed-source). We investigate whether emotions are gendered, and whether these variations are based on societal stereotypes. We prompt the models to adopt a gendered persona and attribute emotions to an event like 'When I had a serious argument with a dear person'. We then analyze the emotions generated by the models in relation to the gender-event pairs. We find that all models consistently exhibit gendered emotions, influenced by gender stereotypes. These findings are in line with established research in psychology and gender studies. Our study sheds light on the complex societal interplay between language, gender, and emotion. The reproduction of emotion stereotypes in LLMs allows us to use those models to study the topic in detail, but raises questions about the predictive use of those same LLMs for emotion applications.

Title: NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors

• Authors: Yannan He, Garvita Tiwari, Tolga Birdal, Jan Eric Lenssen, Gerard Pons-Moll
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03122
• Pdf URL: https://arxiv.org/pdf/2403.03122
• Copy Paste: [[2403.03122]] NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors(https://arxiv.org/abs/2403.03122)
• Keywords: generative
• Abstract: Faithfully modeling the space of articulations is a crucial task that allows recovery and generation of realistic poses, and remains a notorious challenge. To this end, we introduce Neural Riemannian Distance Fields (NRDFs), data-driven priors modeling the space of plausible articulations, represented as the zero-level-set of a neural field in a high-dimensional product-quaternion space. To train NRDFs only on positive examples, we introduce a new sampling algorithm, ensuring that the geodesic distances follow a desired distribution, yielding a principled distance field learning paradigm. We then devise a projection algorithm to map any random pose onto the level-set by an adaptive-step Riemannian optimizer, adhering to the product manifold of joint rotations at all times. NRDFs can compute the Riemannian gradient via backpropagation and by mathematical analogy, are related to Riemannian flow matching, a recent generative model. We conduct a comprehensive evaluation of NRDF against other pose priors in various downstream tasks, i.e., pose generation, image-based pose estimation, and solving inverse kinematics, highlighting NRDF's superior performance. Besides humans, NRDF's versatility extends to hand and animal poses, as it can effectively represent any articulation.

Title: CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following

• Authors: Kaiyan Zhang, Jianyu Wang, Ermo Hua, Biqing Qi, Ning Ding, Bowen Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03129
• Pdf URL: https://arxiv.org/pdf/2403.03129
• Copy Paste: [[2403.03129]] CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following(https://arxiv.org/abs/2403.03129)
• Keywords: secure, privacy
• Abstract: With the advancement of language models (LMs), their exposure to private data is increasingly inevitable, and their deployment (especially for smaller ones) on personal devices, such as PCs and smartphones, has become a prevailing trend. In contexts laden with user information, enabling models to both safeguard user privacy and execute commands efficiently emerges as an essential research imperative. In this paper, we propose CoGenesis, a collaborative generation framework integrating large (hosted on cloud infrastructure) and small models (deployed on local devices) to address privacy concerns logically. Initially, we design a pipeline to create personalized writing instruction datasets enriched with extensive context details as the testbed of this research issue. Subsequently, we introduce two variants of CoGenesis based on sketch and logits respectively. Our experimental findings, based on our synthesized dataset and two additional open-source datasets, indicate that: 1) Large-scale models perform well when provided with user context but struggle in the absence of such context. 2) While specialized smaller models fine-tuned on the synthetic dataset show promise, they still lag behind their larger counterparts. 3) Our CoGenesis framework, utilizing mixed-scale models, showcases competitive performance, providing a feasible solution to privacy issues.

Title: Simplicity in Complexity

• Authors: Kevin Shen, Surabhi S Nath, Aenne Brielmann, Peter Dayan
• Subjects: cs.CV, cs.AI, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2403.03134
• Pdf URL: https://arxiv.org/pdf/2403.03134
• Copy Paste: [[2403.03134]] Simplicity in Complexity(https://arxiv.org/abs/2403.03134)
• Keywords: segmentation
• Abstract: The complexity of visual stimuli plays an important role in many cognitive phenomena, including attention, engagement, memorability, time perception and aesthetic evaluation. Despite its importance, complexity is poorly understood and ironically, previous models of image complexity have been quite \textit{complex}. There have been many attempts to find handcrafted features that explain complexity, but these features are usually dataset specific, and hence fail to generalise. On the other hand, more recent work has employed deep neural networks to predict complexity, but these models remain difficult to interpret, and do not guide a theoretical understanding of the problem. Here we propose to model complexity using segment-based representations of images. We use state-of-the-art segmentation models, SAM and FC-CLIP, to quantify the number of segments at multiple granularities, and the number of classes in an image respectively. We find that complexity is well-explained by a simple linear model with these two features across six diverse image-sets of naturalistic scene and art images. This suggests that the complexity of images can be surprisingly simple.

Title: Language Guided Exploration for RL Agents in Text Environments

• Authors: Hitesh Golchha, Sahil Yerawar, Dhruvesh Patel, Soham Dan, Keerthiram Murugesan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03141
• Pdf URL: https://arxiv.org/pdf/2403.03141
• Copy Paste: [[2403.03141]] Language Guided Exploration for RL Agents in Text Environments(https://arxiv.org/abs/2403.03141)
• Keywords: transformer, large language model
• Abstract: Real-world sequential decision making is characterized by sparse rewards and large decision spaces, posing significant difficulty for experiential learning systems like $\textit{tabula rasa}$ reinforcement learning (RL) agents. Large Language Models (LLMs), with a wealth of world knowledge, can help RL agents learn quickly and adapt to distribution shifts. In this work, we introduce Language Guided Exploration (LGE) framework, which uses a pre-trained language model (called GUIDE ) to provide decision-level guidance to an RL agent (called EXPLORER). We observe that on ScienceWorld (Wang et al.,2022), a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.

Title: Robust Federated Learning Mitigates Client-side Training Data Distribution Inference Attacks

• Authors: Yichang Xu, Ming Yin, Minghong Fang, Neil Zhenqiang Gong
• Subjects: cs.CR, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.03149
• Pdf URL: https://arxiv.org/pdf/2403.03149
• Copy Paste: [[2403.03149]] Robust Federated Learning Mitigates Client-side Training Data Distribution Inference Attacks(https://arxiv.org/abs/2403.03149)
• Keywords: secure, protect, defense, attack, robust, federate
• Abstract: Recent studies have revealed that federated learning (FL), once considered secure due to clients not sharing their private data with the server, is vulnerable to attacks such as client-side training data distribution inference, where a malicious client can recreate the victim's data. While various countermeasures exist, they are not practical, often assuming server access to some training data or knowledge of label distribution before the attack. In this work, we bridge the gap by proposing InferGuard, a novel Byzantine-robust aggregation rule aimed at defending against client-side training data distribution inference attacks. In our proposed InferGuard, the server first calculates the coordinate-wise median of all the model updates it receives. A client's model update is considered malicious if it significantly deviates from the computed median update. We conduct a thorough evaluation of our proposed InferGuard on five benchmark datasets and perform a comparison with ten baseline methods. The results of our experiments indicate that our defense mechanism is highly effective in protecting against client-side training data distribution inference attacks, even against strong adaptive attacks. Furthermore, our method substantially outperforms the baseline methods in various practical FL scenarios.

Title: Design2Code: How Far Are We From Automating Front-End Engineering?

• Authors: Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, Diyi Yang
• Subjects: cs.CL, cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.03163
• Pdf URL: https://arxiv.org/pdf/2403.03163
• Copy Paste: [[2403.03163]] Design2Code: How Far Are We From Automating Front-End Engineering?(https://arxiv.org/abs/2403.03163)
• Keywords: generative
• Abstract: Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development, in which multimodal LLMs might directly convert visual designs into code implementations. In this work, we formalize this as a Design2Code task and conduct comprehensive benchmarking. Specifically, we manually curate a benchmark of 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations. We develop a suite of multimodal prompting methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We further finetune an open-source Design2Code-18B model that successfully matches the performance of Gemini Pro Vision. Both human evaluation and automatic metrics show that GPT-4V performs the best on this task compared to other models. Moreover, annotators think GPT-4V generated webpages can replace the original reference webpages in 49% of cases in terms of visual appearance and content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages are considered better than the original reference webpages. Our fine-grained break-down metrics indicate that open-source models mostly lag in recalling visual elements from the input webpages and in generating correct layout designs, while aspects like text content and coloring can be drastically improved with proper finetuning.

Title: PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset

• Authors: Arda Uzunoğlu, Abdalfatah Rashid Safa, Gözde Gül Şahin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03167
• Pdf URL: https://arxiv.org/pdf/2403.03167
• Copy Paste: [[2403.03167]] PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset(https://arxiv.org/abs/2403.03167)
• Keywords: generative, large language model
• Abstract: Recently, there has been growing interest within the community regarding whether large language models are capable of planning or executing plans. However, most prior studies use LLMs to generate high-level plans for simplified scenarios lacking linguistic complexity and domain diversity, limiting analysis of their planning abilities. These setups constrain evaluation methods (e.g., predefined action space), architectural choices (e.g., only generative models), and overlook the linguistic nuances essential for realistic analysis. To tackle this, we present PARADISE, an abductive reasoning task using Q\&A format on practical procedural text sourced from wikiHow. It involves warning and tip inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing fine-tuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios. Despite advancements, all models fall short of human performance. Notably, our analysis uncovers intriguing insights, such as variations in model behavior with dropped keywords, struggles of BERT-family and GPT-4 with physical and abstract goals, and the proposed tasks offering valuable prior knowledge for other unseen procedural tasks. The PARADISE dataset and associated resources are publicly available for further research exploration with https://github.com/GGLAB-KU/paradise.

Title: Solving the bongard-logo problem by modeling a probabilistic model

• Authors: Ruizhuo Song, Beiming Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03173
• Pdf URL: https://arxiv.org/pdf/2403.03173
• Copy Paste: [[2403.03173]] Solving the bongard-logo problem by modeling a probabilistic model(https://arxiv.org/abs/2403.03173)
• Keywords: transformer
• Abstract: Abstract reasoning problems challenge the perceptual and cognitive abilities of AI algorithms, demanding deeper pattern discernment and inductive reasoning beyond explicit image features. This study introduces PMoC, a tailored probability model for the Bongard-Logo problem, achieving high reasoning accuracy by constructing independent probability models. Additionally, we present Pose-Transformer, an enhanced Transformer-Encoder designed for complex abstract reasoning tasks, including Bongard-Logo, RAVEN, I-RAVEN, and PGM. Pose-Transformer incorporates positional information learning, inspired by capsule networks' pose matrices, enhancing its focus on local positional relationships in image data processing. When integrated with PMoC, it further improves reasoning accuracy. Our approach effectively addresses reasoning difficulties associated with abstract entities' positional changes, outperforming previous models on the OIG, D3$\times$3 subsets of RAVEN, and PGM databases. This research contributes to advancing AI's capabilities in abstract reasoning and cognitive pattern recognition.

Title: Behavior Generation with Latent Actions

• Authors: Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, Lerrel Pinto
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.03181
• Pdf URL: https://arxiv.org/pdf/2403.03181
• Copy Paste: [[2403.03181]] Behavior Generation with Latent Actions(https://arxiv.org/abs/2403.03181)
• Keywords: diffusion, transformer, generative
• Abstract: Generative modeling of complex behaviors from labeled datasets has been a longstanding problem in decision making. Unlike language or image generation, decision making requires modeling actions - continuous-valued vectors that are multimodal in their distribution, potentially drawn from uncurated sources, where generation errors can compound in sequential prediction. A recent class of models called Behavior Transformers (BeT) addresses this by discretizing actions using k-means clustering to capture different modes. However, k-means struggles to scale for high-dimensional action spaces or long sequences, and lacks gradient information, and thus BeT suffers in modeling long-range actions. In this work, we present Vector-Quantized Behavior Transformer (VQ-BeT), a versatile model for behavior generation that handles multimodal action prediction, conditional generation, and partial observations. VQ-BeT augments BeT by tokenizing continuous actions with a hierarchical vector quantization module. Across seven environments including simulated manipulation, autonomous driving, and robotics, VQ-BeT improves on state-of-the-art models such as BeT and Diffusion Policies. Importantly, we demonstrate VQ-BeT's improved ability to capture behavior modes while accelerating inference speed 5x over Diffusion Policies. Videos and code can be found https://sjlee.cc/vq-bet

Title: How Well Can Transformers Emulate In-context Newton's Method?

• Authors: Angeliki Giannou, Liu Yang, Tianhao Wang, Dimitris Papailiopoulos, Jason D. Lee
• Subjects: cs.LG, cs.AI, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.03183
• Pdf URL: https://arxiv.org/pdf/2403.03183
• Copy Paste: [[2403.03183]] How Well Can Transformers Emulate In-context Newton's Method?(https://arxiv.org/abs/2403.03183)
• Keywords: transformer
• Abstract: Transformer-based models have demonstrated remarkable in-context learning capabilities, prompting extensive research into its underlying mechanisms. Recent studies have suggested that Transformers can implement first-order optimization algorithms for in-context learning and even second order ones for the case of linear regression. In this work, we study whether Transformers can perform higher order optimization methods, beyond the case of linear regression. We establish that linear attention Transformers with ReLU layers can approximate second order optimization algorithms for the task of logistic regression and achieve $\epsilon$ error with only a logarithmic to the error more layers. As a by-product we demonstrate the ability of even linear attention-only Transformers in implementing a single step of Newton's iteration for matrix inversion with merely two layers. These results suggest the ability of the Transformer architecture to implement complex algorithms, beyond gradient descent.

Title: Triple-CFN: Restructuring Conceptual Spaces for Enhancing Abstract Reasoning process

• Authors: Ruizhuo Song, Beiming Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03190
• Pdf URL: https://arxiv.org/pdf/2403.03190
• Copy Paste: [[2403.03190]] Triple-CFN: Restructuring Conceptual Spaces for Enhancing Abstract Reasoning process(https://arxiv.org/abs/2403.03190)
• Keywords: interpretability
• Abstract: Abstract reasoning problems pose significant challenges to artificial intelligence algorithms, demanding cognitive capabilities beyond those required for perception tasks. This study introduces the Triple-CFN approach to tackle the Bongard-Logo problem, achieving notable reasoning accuracy by implicitly reorganizing the concept space of conflicting instances. Additionally, the Triple-CFN paradigm proves effective for the RPM problem with necessary modifications, yielding competitive results. To further enhance performance on the RPM issue, we develop the Meta Triple-CFN network, which explicitly structures the problem space while maintaining interpretability on progressive patterns. The success of Meta Triple-CFN is attributed to its paradigm of modeling the conceptual space, equivalent to normalizing reasoning information. Based on this ideology, we introduce the Re-space layer, enhancing the performance of both Meta Triple-CFN and Triple-CFN. This paper aims to contribute to advancements in machine intelligence by exploring innovative network designs for addressing abstract reasoning problems, paving the way for further breakthroughs in this domain.

Title: MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets

• Authors: Hossein Aboutalebi, Hwanjun Song, Yusheng Xie, Arshit Gupta, Justin Sun, Hang Su, Igor Shalyminov, Nikolaos Pappas, Siffi Singh, Saab Mansour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03194
• Pdf URL: https://arxiv.org/pdf/2403.03194
• Copy Paste: [[2403.03194]] MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets(https://arxiv.org/abs/2403.03194)
• Keywords: privacy, diffusion
• Abstract: Development of multimodal interactive systems is hindered by the lack of rich, multimodal (text, images) conversational data, which is needed in large quantities for LLMs. Previous approaches augment textual dialogues with retrieved images, posing privacy, diversity, and quality constraints. In this work, we introduce \textbf{M}ultimodal \textbf{A}ugmented \textbf{G}enerative \textbf{I}mages \textbf{D}ialogues (MAGID), a framework to augment text-only dialogues with diverse and high-quality images. Subsequently, a diffusion model is applied to craft corresponding images, ensuring alignment with the identified text. Finally, MAGID incorporates an innovative feedback loop between an image description generation module (textual LLM) and image quality modules (addressing aesthetics, image-text matching, and safety), that work in tandem to generate high-quality and multi-modal dialogues. We compare MAGID to other SOTA baselines on three dialogue datasets, using automated and human evaluation. Our results show that MAGID is comparable to or better than baselines, with significant improvements in human evaluation, especially against retrieval baselines where the image database is small.

Title: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

• Authors: Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, Robin Rombach
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03206
• Pdf URL: https://arxiv.org/pdf/2403.03206
• Copy Paste: [[2403.03206]] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis(https://arxiv.org/abs/2403.03206)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is not yet decisively established as standard practice. In this work, we improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. Through a large-scale study, we demonstrate the superior performance of this approach compared to established diffusion formulations for high-resolution text-to-image synthesis. Additionally, we present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension, typography, and human preference ratings. We demonstrate that this architecture follows predictable scaling trends and correlates lower validation loss to improved text-to-image synthesis as measured by various metrics and human evaluations. Our largest models outperform state-of-the-art models, and we will make our experimental data, code, and model weights publicly available.

Title: Self-supervised 3D Patient Modeling with Multi-modal Attentive Fusion

• Authors: Meng Zheng, Benjamin Planche, Xuan Gong, Fan Yang, Terrence Chen, Ziyan Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03217
• Pdf URL: https://arxiv.org/pdf/2403.03217
• Copy Paste: [[2403.03217]] Self-supervised 3D Patient Modeling with Multi-modal Attentive Fusion(https://arxiv.org/abs/2403.03217)
• Keywords: robust
• Abstract: 3D patient body modeling is critical to the success of automated patient positioning for smart medical scanning and operating rooms. Existing CNN-based end-to-end patient modeling solutions typically require a) customized network designs demanding large amount of relevant training data, covering extensive realistic clinical scenarios (e.g., patient covered by sheets), which leads to suboptimal generalizability in practical deployment, b) expensive 3D human model annotations, i.e., requiring huge amount of manual effort, resulting in systems that scale poorly. To address these issues, we propose a generic modularized 3D patient modeling method consists of (a) a multi-modal keypoint detection module with attentive fusion for 2D patient joint localization, to learn complementary cross-modality patient body information, leading to improved keypoint localization robustness and generalizability in a wide variety of imaging (e.g., CT, MRI etc.) and clinical scenarios (e.g., heavy occlusions); and (b) a self-supervised 3D mesh regression module which does not require expensive 3D mesh parameter annotations to train, bringing immediate cost benefits for clinical deployment. We demonstrate the efficacy of the proposed method by extensive patient positioning experiments on both public and clinical data. Our evaluation results achieve superior patient positioning performance across various imaging modalities in real clinical scenarios.

Title: The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

• Authors: Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Ariel Herbert-Voss, Cort B. Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Liu, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, et al. (1 additional author not shown)
• Subjects: cs.LG, cs.AI, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.03218
• Pdf URL: https://arxiv.org/pdf/2403.03218
• Copy Paste: [[2403.03218]] The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning(https://arxiv.org/abs/2403.03218)
• Keywords: security, large language model
• Abstract: The White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. To measure these risks of malicious use, government institutions and major AI labs are developing evaluations for hazardous capabilities in LLMs. However, current evaluations are private, preventing further research into mitigating risk. Furthermore, they focus on only a few, highly specific pathways for malicious use. To fill these gaps, we publicly release the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of 4,157 multiple-choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP was developed by a consortium of academics and technical consultants, and was stringently filtered to eliminate sensitive information prior to public release. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. To guide progress on unlearning, we develop CUT, a state-of-the-art unlearning method based on controlling model representations. CUT reduces model performance on WMDP while maintaining general capabilities in areas such as biology and computer science, suggesting that unlearning may be a concrete path towards reducing malicious use from LLMs. We release our benchmark and code publicly at https://wmdp.ai

Title: FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimation

• Authors: Chris Rockwell, Nilesh Kulkarni, Linyi Jin, Jeong Joon Park, Justin Johnson, David F. Fouhey
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03221
• Pdf URL: https://arxiv.org/pdf/2403.03221
• Copy Paste: [[2403.03221]] FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimation(https://arxiv.org/abs/2403.03221)
• Keywords: robust, transformer
• Abstract: Estimating relative camera poses between images has been a central problem in computer vision. Methods that find correspondences and solve for the fundamental matrix offer high precision in most cases. Conversely, methods predicting pose directly using neural networks are more robust to limited overlap and can infer absolute translation scale, but at the expense of reduced precision. We show how to combine the best of both methods; our approach yields results that are both precise and robust, while also accurately inferring translation scales. At the heart of our model lies a Transformer that (1) learns to balance between solved and learned pose estimations, and (2) provides a prior to guide a solver. A comprehensive analysis supports our design choices and demonstrates that our method adapts flexibly to various feature extractors and correspondence estimators, showing state-of-the-art performance in 6DoF pose estimation on Matterport3D, InteriorNet, StreetLearn, and Map-free Relocalization.