2024-07-29

Title: Physics-guided machine learning predicts the planet-scale performance of solar farms with sparse, heterogeneous, public data

• Authors: Jabir Bin Jahangir, Muhammad Ashraful Alam
• Subjects: cs.LG, physics.app-ph, physics.data-an
• Abstract URL: https://arxiv.org/abs/2407.18284
• Pdf URL: https://arxiv.org/pdf/2407.18284
• Copy Paste: [[2407.18284]] Physics-guided machine learning predicts the planet-scale performance of solar farms with sparse, heterogeneous, public data(https://arxiv.org/abs/2407.18284)
• Keywords: privacy
• Abstract: The photovoltaics (PV) technology landscape is evolving rapidly. To predict the potential and scalability of emerging PV technologies, a global understanding of these systems' performance is essential. Traditionally, experimental and computational studies at large national research facilities have focused on PV performance in specific regional climates. However, synthesizing these regional studies to understand the worldwide performance potential has proven difficult. Given the expense of obtaining experimental data, the challenge of coordinating experiments at national labs across a politically-divided world, and the data-privacy concerns of large commercial operators, however, a fundamentally different, data-efficient approach is desired. Here, we present a physics-guided machine learning (PGML) scheme to demonstrate that: (a) The world can be divided into a few PV-specific climate zones, called PVZones, illustrating that the relevant meteorological conditions are shared across continents; (b) by exploiting the climatic similarities, high-quality monthly energy yield data from as few as five locations can accurately predict yearly energy yield potential with high spatial resolution and a root mean square error of less than 8 kWhm$^{2}$, and (c) even with noisy, heterogeneous public PV performance data, the global energy yield can be predicted with less than 6% relative error compared to physics-based simulations provided that the dataset is representative. This PGML scheme is agnostic to PV technology and farm topology, making it adaptable to new PV technologies or farm configurations. The results encourage physics-guided, data-driven collaboration among national policymakers and research organizations to build efficient decision support systems for accelerated PV qualification and deployment across the world.

Title: Leveraging Foundation Models via Knowledge Distillation in Multi-Object Tracking: Distilling DINOv2 Features to FairMOT

• Authors: Niels G. Faber, Seyed Sahand Mohammadi Ziabari, Fatemeh Karimi Najadasl
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18288
• Pdf URL: https://arxiv.org/pdf/2407.18288
• Copy Paste: [[2407.18288]] Leveraging Foundation Models via Knowledge Distillation in Multi-Object Tracking: Distilling DINOv2 Features to FairMOT(https://arxiv.org/abs/2407.18288)
• Keywords: fair
• Abstract: Multiple Object Tracking (MOT) is a computer vision task that has been employed in a variety of sectors. Some common limitations in MOT are varying object appearances, occlusions, or crowded scenes. To address these challenges, machine learning methods have been extensively deployed, leveraging large datasets, sophisticated models, and substantial computational resources. Due to practical limitations, access to the above is not always an option. However, with the recent release of foundation models by prominent AI companies, pretrained models have been trained on vast datasets and resources using state-of-the-art methods. This work tries to leverage one such foundation model, called DINOv2, through using knowledge distillation. The proposed method uses a teacher-student architecture, where DINOv2 is the teacher and the FairMOT backbone HRNetv2 W18 is the student. The results imply that although the proposed method shows improvements in certain scenarios, it does not consistently outperform the original FairMOT model. These findings highlight the potential and limitations of applying foundation models in knowledge

Title: MARINE: A Computer Vision Model for Detecting Rare Predator-Prey Interactions in Animal Videos

• Authors: Zsófia Katona, Seyed Sahand Mohammadi Ziabari, Fatemeh Karimi Najadasl
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18289
• Pdf URL: https://arxiv.org/pdf/2407.18289
• Copy Paste: [[2407.18289]] MARINE: A Computer Vision Model for Detecting Rare Predator-Prey Interactions in Animal Videos(https://arxiv.org/abs/2407.18289)
• Keywords: attack, extraction, transformer
• Abstract: Encounters between predator and prey play an essential role in ecosystems, but their rarity makes them difficult to detect in video recordings. Although advances in action recognition (AR) and temporal action detection (AD), especially transformer-based models and vision foundation models, have achieved high performance on human action datasets, animal videos remain relatively under-researched. This thesis addresses this gap by proposing the model MARINE, which utilizes motion-based frame selection designed for fast animal actions and DINOv2 feature extraction with a trainable classification head for action recognition. MARINE outperforms VideoMAE in identifying predator attacks in videos of fish, both on a small and specific coral reef dataset (81.53\% against 52.64\% accuracy), and on a subset of the more extensive Animal Kingdom dataset (94.86\% against 83.14\% accuracy). In a multi-label setting on a representative sample of Animal Kingdom, MARINE achieves 23.79\% mAP, positioning it mid-field among existing benchmarks. Furthermore, in an AD task on the coral reef dataset, MARINE achieves 80.78\% AP (against VideoMAE's 34.89\%) although at a lowered t-IoU threshold of 25\%. Therefore, despite room for improvement, MARINE offers an effective starter framework to apply to AR and AD tasks on animal recordings and thus contribute to the study of natural ecosystems.

Title: The Need for Guardrails with Large Language Models in Medical Safety-Critical Settings: An Artificial Intelligence Application in the Pharmacovigilance Ecosystem

• Authors: Joe B Hakim, Jeffery L Painter, Darmendra Ramcharran, Vijay Kara, Greg Powell, Paulina Sobczak, Chiho Sato, Andrew Bate, Andrew Beam
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18322
• Pdf URL: https://arxiv.org/pdf/2407.18322
• Copy Paste: [[2407.18322]] The Need for Guardrails with Large Language Models in Medical Safety-Critical Settings: An Artificial Intelligence Application in the Pharmacovigilance Ecosystem(https://arxiv.org/abs/2407.18322)
• Keywords: large language model
• Abstract: Large language models (LLMs) are useful tools with the capacity for performing specific types of knowledge work at an effective scale. However, LLM deployments in high-risk and safety-critical domains pose unique challenges, notably the issue of ``hallucination,'' where LLMs can generate fabricated information. This is particularly concerning in settings such as drug safety, where inaccuracies could lead to patient harm. To mitigate these risks, we have developed and demonstrated a proof of concept suite of guardrails specifically designed to mitigate certain types of hallucinations and errors for drug safety, and potentially applicable to other medical safety-critical contexts. These guardrails include mechanisms to detect anomalous documents to prevent the ingestion of inappropriate data, identify incorrect drug names or adverse event terms, and convey uncertainty in generated content. We integrated these guardrails with an LLM fine-tuned for a text-to-text task, which involves converting both structured and unstructured data within adverse event reports into natural language. This method was applied to translate individual case safety reports, demonstrating effective application in a pharmacovigilance processing task. Our guardrail framework offers a set of tools with broad applicability across various domains, ensuring LLMs can be safely used in high-risk situations by eliminating the occurrence of key errors, including the generation of incorrect pharmacovigilance-related terms, thus adhering to stringent regulatory and quality standards in medical safety-critical environments.

Title: AMA-LSTM: Pioneering Robust and Fair Financial Audio Analysis for Stock Volatility Prediction

• Authors: Shengkun Wang, Taoran Ji, Jianfeng He, Mariam Almutairi, Dan Wang, Linhan Wang, Min Zhang, Chang-Tien Lu
• Subjects: cs.LG, cs.CL, eess.AS, q-fin.CP, q-fin.ST
• Abstract URL: https://arxiv.org/abs/2407.18324
• Pdf URL: https://arxiv.org/pdf/2407.18324
• Copy Paste: [[2407.18324]] AMA-LSTM: Pioneering Robust and Fair Financial Audio Analysis for Stock Volatility Prediction(https://arxiv.org/abs/2407.18324)
• Keywords: robust, fair
• Abstract: Stock volatility prediction is an important task in the financial industry. Recent advancements in multimodal methodologies, which integrate both textual and auditory data, have demonstrated significant improvements in this domain, such as earnings calls (Earnings calls are public available and often involve the management team of a public company and interested parties to discuss the company's earnings). However, these multimodal methods have faced two drawbacks. First, they often fail to yield reliable models and overfit the data due to their absorption of stochastic information from the stock market. Moreover, using multimodal models to predict stock volatility suffers from gender bias and lacks an efficient way to eliminate such bias. To address these aforementioned problems, we use adversarial training to generate perturbations that simulate the inherent stochasticity and bias, by creating areas resistant to random information around the input space to improve model robustness and fairness. Our comprehensive experiments on two real-world financial audio datasets reveal that this method exceeds the performance of current state-of-the-art solution. This confirms the value of adversarial training in reducing stochasticity and bias for stock volatility prediction tasks.

Title: Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring

• Authors: Xuansheng Wu, Padmaja Pravin Saraf, Gyeong-Geon Lee, Ehsan Latif, Ninghao Liu, Xiaoming Zhai
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.18328
• Pdf URL: https://arxiv.org/pdf/2407.18328
• Copy Paste: [[2407.18328]] Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring(https://arxiv.org/abs/2407.18328)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated strong potential in performing automatic scoring for constructed response assessments. While constructed responses graded by humans are usually based on given grading rubrics, the methods by which LLMs assign scores remain largely unclear. It is also uncertain how closely AI's scoring process mirrors that of humans, or if it adheres to the same grading criteria. To address this gap, this paper uncovers the grading rubrics that LLMs used to score students' written responses to science tasks and their alignment with human scores. We also examine whether enhancing the alignments can improve scoring accuracy. Specifically, we prompt LLMs to generate analytic rubrics that they use to assign scores and study the alignment gap with human grading rubrics. Based on a series of experiments with various configurations of LLM settings, we reveal a notable alignment gap between human and LLM graders. While LLMs can adapt quickly to scoring tasks, they often resort to shortcuts, bypassing deeper logical reasoning expected in human grading. We found that incorporating high-quality analytical rubrics designed to reflect human grading logic can mitigate this gap and enhance LLMs' scoring accuracy. These results caution against the simplistic application of LLMs in science education and highlight the importance of aligning LLM outputs with human expectations to ensure efficient and accurate automatic scoring.

Title: Introducing {\delta}-XAI: a novel sensitivity-based method for local AI explanations

• Authors: Alessandro De Carlo, Enea Parimbelli, Nicola Melillo, Giovanna Nicora
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18343
• Pdf URL: https://arxiv.org/pdf/2407.18343
• Copy Paste: [[2407.18343]] Introducing {\delta}-XAI: a novel sensitivity-based method for local AI explanations(https://arxiv.org/abs/2407.18343)
• Keywords: robust, interpretability
• Abstract: Explainable Artificial Intelligence (XAI) is central to the debate on integrating Artificial Intelligence (AI) and Machine Learning (ML) algorithms into clinical practice. High-performing AI/ML models, such as ensemble learners and deep neural networks, often lack interpretability, hampering clinicians' trust in their predictions. To address this, XAI techniques are being developed to describe AI/ML predictions in human-understandable terms. One promising direction is the adaptation of sensitivity analysis (SA) and global sensitivity analysis (GSA), which inherently rank model inputs by their impact on predictions. Here, we introduce a novel delta-XAI method that provides local explanations of ML model predictions by extending the delta index, a GSA metric. The delta-XAI index assesses the impact of each feature's value on the predicted output for individual instances in both regression and classification problems. We formalize the delta-XAI index and provide code for its implementation. The delta-XAI method was evaluated on simulated scenarios using linear regression models, with Shapley values serving as a benchmark. Results showed that the delta-XAI index is generally consistent with Shapley values, with notable discrepancies in models with highly impactful or extreme feature values. The delta-XAI index demonstrated higher sensitivity in detecting dominant features and handling extreme feature values. Qualitatively, the delta-XAI provides intuitive explanations by leveraging probability density functions, making feature rankings clearer and more explainable for practitioners. Overall, the delta-XAI method appears promising for robustly obtaining local explanations of ML model predictions. Further investigations in real-world clinical settings will be conducted to evaluate its impact on AI-assisted clinical workflows.

Title: Privacy-Preserving Model-Distributed Inference at the Edge

• Authors: Fatemeh Jafarian Dehkordi, Yasaman Keshtkarjahromi, Hulya Seferoglu
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18353
• Pdf URL: https://arxiv.org/pdf/2407.18353
• Copy Paste: [[2407.18353]] Privacy-Preserving Model-Distributed Inference at the Edge(https://arxiv.org/abs/2407.18353)
• Keywords: privacy
• Abstract: This paper focuses on designing a privacy-preserving Machine Learning (ML) inference protocol for a hierarchical setup, where clients own/generate data, model owners (cloud servers) have a pre-trained ML model, and edge servers perform ML inference on clients' data using the cloud server's ML model. Our goal is to speed up ML inference while providing privacy to both data and the ML model. Our approach (i) uses model-distributed inference (model parallelization) at the edge servers and (ii) reduces the amount of communication to/from the cloud server. Our privacy-preserving hierarchical model-distributed inference, privateMDI design uses additive secret sharing and linearly homomorphic encryption to handle linear calculations in the ML inference, and garbled circuit and a novel three-party oblivious transfer are used to handle non-linear functions. privateMDI consists of offline and online phases. We designed these phases in a way that most of the data exchange is done in the offline phase while the communication overhead of the online phase is reduced. In particular, there is no communication to/from the cloud server in the online phase, and the amount of communication between the client and edge servers is minimized. The experimental results demonstrate that privateMDI significantly reduces the ML inference time as compared to the baselines.

Title: Generative AI like ChatGPT in Blockchain Federated Learning: use cases, opportunities and future

• Authors: Sai Puppala, Ismail Hossain, Md Jahangir Alam, Sajedul Talukder, Jannatul Ferdaus, Mahedi Hasan, Sameera Pisupati, Shanmukh Mathukumilli
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2407.18358
• Pdf URL: https://arxiv.org/pdf/2407.18358
• Copy Paste: [[2407.18358]] Generative AI like ChatGPT in Blockchain Federated Learning: use cases, opportunities and future(https://arxiv.org/abs/2407.18358)
• Keywords: privacy, robust, federate, generative
• Abstract: Federated learning has become a significant approach for training machine learning models using decentralized data without necessitating the sharing of this data. Recently, the incorporation of generative artificial intelligence (AI) methods has provided new possibilities for improving privacy, augmenting data, and customizing models. This research explores potential integrations of generative AI in federated learning, revealing various opportunities to enhance privacy, data efficiency, and model performance. It particularly emphasizes the importance of generative models like generative adversarial networks (GANs) and variational autoencoders (VAEs) in creating synthetic data that replicates the distribution of real data. Generating synthetic data helps federated learning address challenges related to limited data availability and supports robust model development. Additionally, we examine various applications of generative AI in federated learning that enable more personalized solutions.

Title: FADAS: Towards Federated Adaptive Asynchronous Optimization

• Authors: Yujia Wang, Shiqiang Wang, Songtao Lu, Jinghui Chen
• Subjects: cs.LG, cs.AI, cs.DC, math.OC
• Abstract URL: https://arxiv.org/abs/2407.18365
• Pdf URL: https://arxiv.org/pdf/2407.18365
• Copy Paste: [[2407.18365]] FADAS: Towards Federated Adaptive Asynchronous Optimization(https://arxiv.org/abs/2407.18365)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) has emerged as a widely adopted training paradigm for privacy-preserving machine learning. While the SGD-based FL algorithms have demonstrated considerable success in the past, there is a growing trend towards adopting adaptive federated optimization methods, particularly for training large-scale models. However, the conventional synchronous aggregation design poses a significant challenge to the practical deployment of those adaptive federated optimization methods, particularly in the presence of straggler clients. To fill this research gap, this paper introduces federated adaptive asynchronous optimization, named FADAS, a novel method that incorporates asynchronous updates into adaptive federated optimization with provable guarantees. To further enhance the efficiency and resilience of our proposed method in scenarios with significant asynchronous delays, we also extend FADAS with a delay-adaptive learning adjustment strategy. We rigorously establish the convergence rate of the proposed algorithms and empirical results demonstrate the superior performance of FADAS over other asynchronous FL baselines.

Title: Robust Claim Verification Through Fact Detection

• Authors: Nazanin Jafari, James Allan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18367
• Pdf URL: https://arxiv.org/pdf/2407.18367
• Copy Paste: [[2407.18367]] Robust Claim Verification Through Fact Detection(https://arxiv.org/abs/2407.18367)
• Keywords: robust, extraction, explainability, large language model
• Abstract: Claim verification can be a challenging task. In this paper, we present a method to enhance the robustness and reasoning capabilities of automated claim verification through the extraction of short facts from evidence. Our novel approach, FactDetect, leverages Large Language Models (LLMs) to generate concise factual statements from evidence and label these facts based on their semantic relevance to the claim and evidence. The generated facts are then combined with the claim and evidence. To train a lightweight supervised model, we incorporate a fact-detection task into the claim verification process as a multitasking approach to improve both performance and explainability. We also show that augmenting FactDetect in the claim verification prompt enhances performance in zero-shot claim verification using LLMs. Our method demonstrates competitive results in the supervised claim verification model by 15% on the F1 score when evaluated for challenging scientific claim verification datasets. We also demonstrate that FactDetect can be augmented with claim and evidence for zero-shot prompting (AugFactDetect) in LLMs for verdict prediction. We show that AugFactDetect outperforms the baseline with statistical significance on three challenging scientific claim verification datasets with an average of 17.3% performance gain compared to the best performing baselines.

Title: Physics Informed Kolmogorov-Arnold Neural Networks for Dynamical Analysis via Efficent-KAN and WAV-KAN

• Authors: Subhajit Patra, Sonali Panda, Bikram Keshari Parida, Mahima Arya, Kurt Jacobs, Denys I. Bondar, Abhijit Sen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18373
• Pdf URL: https://arxiv.org/pdf/2407.18373
• Copy Paste: [[2407.18373]] Physics Informed Kolmogorov-Arnold Neural Networks for Dynamical Analysis via Efficent-KAN and WAV-KAN(https://arxiv.org/abs/2407.18373)
• Keywords: data-free
• Abstract: Physics-informed neural networks have proven to be a powerful tool for solving differential equations, leveraging the principles of physics to inform the learning process. However, traditional deep neural networks often face challenges in achieving high accuracy without incurring significant computational costs. In this work, we implement the Physics-Informed Kolmogorov-Arnold Neural Networks (PIKAN) through efficient-KAN and WAV-KAN, which utilize the Kolmogorov-Arnold representation theorem. PIKAN demonstrates superior performance compared to conventional deep neural networks, achieving the same level of accuracy with fewer layers and reduced computational overhead. We explore both B-spline and wavelet-based implementations of PIKAN and benchmark their performance across various ordinary and partial differential equations using unsupervised (data-free) and supervised (data-driven) techniques. For certain differential equations, the data-free approach suffices to find accurate solutions, while in more complex scenarios, the data-driven method enhances the PIKAN's ability to converge to the correct solution. We validate our results against numerical solutions and achieve $99 \%$ accuracy in most scenarios.

Title: Effect of Data Degradation on Motion Re-Identification

• Authors: Vivek Nair, Mark Roman Miller, Rui Wang, Brandon Huang, Christian Rack, Marc Erich Latoschik, James F. O'Brien
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.18378
• Pdf URL: https://arxiv.org/pdf/2407.18378
• Copy Paste: [[2407.18378]] Effect of Data Degradation on Motion Re-Identification(https://arxiv.org/abs/2407.18378)
• Keywords: privacy, defense, attack
• Abstract: The use of virtual and augmented reality devices is increasing, but these sensor-rich devices pose risks to privacy. The ability to track a user's motion and infer the identity or characteristics of the user poses a privacy risk that has received significant attention. Existing deep-network-based defenses against this risk, however, require significant amounts of training data and have not yet been shown to generalize beyond specific applications. In this work, we study the effect of signal degradation on identifiability, specifically through added noise, reduced framerate, reduced precision, and reduced dimensionality of the data. Our experiment shows that state-of-the-art identification attacks still achieve near-perfect accuracy for each of these degradations. This negative result demonstrates the difficulty of anonymizing this motion data and gives some justification to the existing data- and compute-intensive deep-network based methods.

Title: Gaussian Process Kolmogorov-Arnold Networks

• Authors: Andrew Siyuan Chen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.18397
• Pdf URL: https://arxiv.org/pdf/2407.18397
• Copy Paste: [[2407.18397]] Gaussian Process Kolmogorov-Arnold Networks(https://arxiv.org/abs/2407.18397)
• Keywords: robust
• Abstract: In this paper, we introduce a probabilistic extension to Kolmogorov Arnold Networks (KANs) by incorporating Gaussian Process (GP) as non-linear neurons, which we refer to as GP-KAN. A fully analytical approach to handling the output distribution of one GP as an input to another GP is achieved by considering the function inner product of a GP function sample with the input distribution. These GP neurons exhibit robust non-linear modelling capabilities while using few parameters and can be easily and fully integrated in a feed-forward network structure. They provide inherent uncertainty estimates to the model prediction and can be trained directly on the log-likelihood objective function, without needing variational lower bounds or approximations. In the context of MNIST classification, a model based on GP-KAN of 80 thousand parameters achieved 98.5% prediction accuracy, compared to current state-of-the-art models with 1.5 million parameters.

Title: Large Language Model Integrated Healthcare Cyber-Physical Systems Architecture

• Authors: Malithi Wanniarachchi Kankanamge, Syed Mhamudul Hasan, Abdur R. Shahid, Ning Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18407
• Pdf URL: https://arxiv.org/pdf/2407.18407
• Copy Paste: [[2407.18407]] Large Language Model Integrated Healthcare Cyber-Physical Systems Architecture(https://arxiv.org/abs/2407.18407)
• Keywords: large language model
• Abstract: Cyber-physical systems have become an essential part of the modern healthcare industry. The healthcare cyber-physical systems (HCPS) combine physical and cyber components to improve the healthcare industry. While HCPS has many advantages, it also has some drawbacks, such as a lengthy data entry process, a lack of real-time processing, and limited real-time patient visualization. To overcome these issues, this paper represents an innovative approach to integrating large language model (LLM) to enhance the efficiency of the healthcare system. By incorporating LLM at various layers, HCPS can leverage advanced AI capabilities to improve patient outcomes, advance data processing, and enhance decision-making.

Title: Adversarial Robust Decision Transformer: Enhancing Robustness of RvS via Minimax Returns-to-go

• Authors: Xiaohang Tang, Afonso Marques, Parameswaran Kamalaruban, Ilija Bogunovic
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18414
• Pdf URL: https://arxiv.org/pdf/2407.18414
• Copy Paste: [[2407.18414]] Adversarial Robust Decision Transformer: Enhancing Robustness of RvS via Minimax Returns-to-go(https://arxiv.org/abs/2407.18414)
• Keywords: robust, transformer
• Abstract: Decision Transformer (DT), as one of the representative Reinforcement Learning via Supervised Learning (RvS) methods, has achieved strong performance in offline learning tasks by leveraging the powerful Transformer architecture for sequential decision-making. However, in adversarial environments, these methods can be non-robust, since the return is dependent on the strategies of both the decision-maker and adversary. Training a probabilistic model conditioned on observed return to predict action can fail to generalize, as the trajectories that achieve a return in the dataset might have done so due to a weak and suboptimal behavior adversary. To address this, we propose a worst-case-aware RvS algorithm, the Adversarial Robust Decision Transformer (ARDT), which learns and conditions the policy on in-sample minimax returns-to-go. ARDT aligns the target return with the worst-case return learned through minimax expectile regression, thereby enhancing robustness against powerful test-time adversaries. In experiments conducted on sequential games with full data coverage, ARDT can generate a maximin (Nash Equilibrium) strategy, the solution with the largest adversarial robustness. In large-scale sequential games and continuous adversarial RL environments with partial data coverage, ARDT demonstrates significantly superior robustness to powerful test-time adversaries and attains higher worst-case returns compared to contemporary DT methods.

Title: The Art of Refusal: A Survey of Abstention in Large Language Models

• Authors: Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, Lucy Lu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18418
• Pdf URL: https://arxiv.org/pdf/2407.18418
• Copy Paste: [[2407.18418]] The Art of Refusal: A Survey of Abstention in Large Language Models(https://arxiv.org/abs/2407.18418)
• Keywords: large language model
• Abstract: Abstention, the refusal of large language models (LLMs) to provide an answer, is increasingly recognized for its potential to mitigate hallucinations and enhance safety in building LLM systems. In this survey, we introduce a framework to examine abstention behavior from three perspectives: the query, the model, and human values. We review the literature on abstention methods (categorized based on the development stages of LLMs), benchmarks, and evaluation metrics, and discuss the merits and limitations of prior work. We further identify and motivate areas for future research, such as encouraging the study of abstention as a meta-capability across tasks and customizing abstention abilities based on context. In doing so, we aim to broaden the scope and impact of abstention methodologies in AI systems.

Title: HDL-GPT: High-Quality HDL is All You Need

• Authors: Bhuvnesh Kumar, Saurav Nanda, Ganapathy Parthasarathy, Pawan Patil, Austin Tsai, Parivesh Choudhary
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18423
• Pdf URL: https://arxiv.org/pdf/2407.18423
• Copy Paste: [[2407.18423]] HDL-GPT: High-Quality HDL is All You Need(https://arxiv.org/abs/2407.18423)
• Keywords: transformer, generative
• Abstract: This paper presents Hardware Description Language Generative Pre-trained Transformers (HDL-GPT), a novel approach that leverages the vast repository of open-source High Definition Language (HDL) codes to train superior quality large code models. The core premise of this paper is the hypothesis that high-quality HDL is all you need to create models with exceptional performance and broad zero-shot generalization abilities. The paper elucidates the methods employed for the curation and augmentation of large corpora from open-source HDL code, transforming highly variable quality data into high-quality data through careful prompting and context maintenance. We demonstrate that the careful selection, filtering, and augmentation of data across HDLs can yield powerful models that surpass current state-of-the-art models. We also explore the impact of different fine-tuning methods on the quality of results. We describe experimental results across a range of fine-tuned SOTA LLMs, substantiating our claims. We demonstrate improvements of 50% to 200% over SOTA HDL models on current benchmarks in tasks ranging from HDL circuit explanations, code generation, formal and simulation testbench creation, triaging bugs, and fixing them. HDL-GPT opens new avenues for the development of advanced model training techniques for circuit design tasks.

Title: Investigating the Privacy Risk of Using Robot Vacuum Cleaners in Smart Environments

• Authors: Benjamin Ulsmaag, Jia-Chun Lin, Ming-Chang Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18433
• Pdf URL: https://arxiv.org/pdf/2407.18433
• Copy Paste: [[2407.18433]] Investigating the Privacy Risk of Using Robot Vacuum Cleaners in Smart Environments(https://arxiv.org/abs/2407.18433)
• Keywords: secure, privacy
• Abstract: Robot vacuum cleaners have become increasingly popular and are widely used in various smart environments. To improve user convenience, manufacturers also introduced smartphone applications that enable users to customize cleaning settings or access information about their robot vacuum cleaners. While this integration enhances the interaction between users and their robot vacuum cleaners, it results in potential privacy concerns because users' personal information may be exposed. To address these concerns, end-to-end encryption is implemented between the application, cloud service, and robot vacuum cleaners to secure the exchanged information. Nevertheless, network header metadata remains unencrypted and it is still vulnerable to network eavesdropping. In this paper, we investigate the potential risk of private information exposure through such metadata. A popular robot vacuum cleaner was deployed in a real smart environment where passive network eavesdropping was conducted during several selected cleaning events. Our extensive analysis, based on Association Rule Learning, demonstrates that it is feasible to identify certain events using only the captured Internet traffic metadata, thereby potentially exposing private user information and raising privacy concerns.

Title: A Model for Combinatorial Dictionary Learning and Inference

• Authors: Avrim Blum, Kavya Ravichandran
• Subjects: cs.LG, cs.DS
• Abstract URL: https://arxiv.org/abs/2407.18436
• Pdf URL: https://arxiv.org/pdf/2407.18436
• Copy Paste: [[2407.18436]] A Model for Combinatorial Dictionary Learning and Inference(https://arxiv.org/abs/2407.18436)
• Keywords: robust
• Abstract: We are often interested in decomposing complex, structured data into simple components that explain the data. The linear version of this problem is well-studied as dictionary learning and factor analysis. In this work, we propose a combinatorial model in which to study this question, motivated by the way objects occlude each other in a scene to form an image. First, we identify a property we call "well-structuredness" of a set of low-dimensional components which ensures that no two components in the set are too similar. We show how well-structuredness is sufficient for learning the set of latent components comprising a set of sample instances. We then consider the problem: given a set of components and an instance generated from some unknown subset of them, identify which parts of the instance arise from which components. We consider two variants: (1) determine the minimal number of components required to explain the instance; (2) determine the correct explanation for as many locations as possible. For the latter goal, we also devise a version that is robust to adversarial corruptions, with just a slightly stronger assumption on the components. Finally, we show that the learning problem is computationally infeasible in the absence of any assumptions.

Title: Mixed Non-linear Quantization for Vision Transformers

• Authors: Gihwan Kim, Jemin Lee, Sihyeong Park, Yongin Kwon, Hyungshin Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18437
• Pdf URL: https://arxiv.org/pdf/2407.18437
• Copy Paste: [[2407.18437]] Mixed Non-linear Quantization for Vision Transformers(https://arxiv.org/abs/2407.18437)
• Keywords: transformer
• Abstract: The majority of quantization methods have been proposed to reduce the model size of Vision Transformers, yet most of them have overlooked the quantization of non-linear operations. Only a few works have addressed quantization for non-linear operations, but they applied a single quantization method across all non-linear operations. We believe that this can be further improved by employing a different quantization method for each non-linear operation. Therefore, to assign the most error-minimizing quantization method from the known methods to each non-linear layer, we propose a mixed non-linear quantization that considers layer-wise quantization sensitivity measured by SQNR difference metric. The results show that our method outperforms I-BERT, FQ-ViT, and I-ViT in both 8-bit and 6-bit settings for ViT, DeiT, and Swin models by an average of 0.6%p and 19.6%p, respectively. Our method outperforms I-BERT and I-ViT by 0.6%p and 20.8%p, respectively, when training time is limited. We plan to release our code at this https URL.

Title: Impact of Recurrent Neural Networks and Deep Learning Frameworks on Real-time Lightweight Time Series Anomaly Detection

• Authors: Ming-Chang Lee, Jia-Chun Lin, Sokratis Katsikas
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18439
• Pdf URL: https://arxiv.org/pdf/2407.18439
• Copy Paste: [[2407.18439]] Impact of Recurrent Neural Networks and Deep Learning Frameworks on Real-time Lightweight Time Series Anomaly Detection(https://arxiv.org/abs/2407.18439)
• Keywords: security
• Abstract: Real-time lightweight time series anomaly detection has become increasingly crucial in cybersecurity and many other domains. Its ability to adapt to unforeseen pattern changes and swiftly identify anomalies enables prompt responses and critical decision-making. While several such anomaly detection approaches have been introduced in recent years, they primarily utilize a single type of recurrent neural networks (RNNs) and have been implemented in only one deep learning framework. It is unclear how the use of different types of RNNs available in various deep learning frameworks affects the performance of these anomaly detection approaches due to the absence of comprehensive evaluations. Arbitrarily choosing a RNN variant and a deep learning framework to implement an anomaly detection approach may not reflect its true performance and could potentially mislead users into favoring one approach over another. In this paper, we aim to study the influence of various types of RNNs available in popular deep learning frameworks on real-time lightweight time series anomaly detection. We reviewed several state-of-the-art approaches and implemented a representative anomaly detection approach using well-known RNN variants supported by three widely recognized deep learning frameworks. A comprehensive evaluation is then conducted to analyze the performance of each implementation across real-world, open-source time series datasets. The evaluation results provide valuable guidance for selecting the appropriate RNN variant and deep learning framework for real-time, lightweight time series anomaly detection.

Title: HybridDepth: Robust Depth Fusion for Mobile AR by Leveraging Depth from Focus and Single-Image Priors

• Authors: Ashkan Ganj, Hang Su, Tian Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18443
• Pdf URL: https://arxiv.org/pdf/2407.18443
• Copy Paste: [[2407.18443]] HybridDepth: Robust Depth Fusion for Mobile AR by Leveraging Depth from Focus and Single-Image Priors(https://arxiv.org/abs/2407.18443)
• Keywords: robust
• Abstract: We propose HYBRIDDEPTH, a robust depth estimation pipeline that addresses the unique challenges of depth estimation for mobile AR, such as scale ambiguity, hardware heterogeneity, and generalizability. HYBRIDDEPTH leverages the camera features available on mobile devices. It effectively combines the scale accuracy inherent in Depth from Focus (DFF) methods with the generalization capabilities enabled by strong single-image depth priors. By utilizing the focal planes of a mobile camera, our approach accurately captures depth values from focused pixels and applies these values to compute scale and shift parameters for transforming relative depths into metric depths. We test our pipeline as an end-to-end system, with a newly developed mobile client to capture focal stacks, which are then sent to a GPU-powered server for depth estimation. Through comprehensive quantitative and qualitative analyses, we demonstrate that HYBRIDDEPTH not only outperforms state-of-the-art (SOTA) models in common datasets (DDFF12, NYU Depth v2) and a real-world AR dataset ARKitScenes but also demonstrates strong zero-shot generalization. For example, HYBRIDDEPTH trained on NYU Depth v2 achieves comparable performance on the DDFF12 to existing models trained on DDFF12. it also outperforms all the SOTA models in zero-shot performance on the ARKitScenes dataset. Additionally, we conduct a qualitative comparison between our model and the ARCore framework, demonstrating that our models output depth maps are significantly more accurate in terms of structural details and metric accuracy. The source code of this project is available at github.

Title: Capturing the security expert knowledge in feature selection for web application attack detection

• Authors: Amanda Riverol, Gustavo Betarte, Rodrigo Martínez, Álvaro Pardo
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18445
• Pdf URL: https://arxiv.org/pdf/2407.18445
• Copy Paste: [[2407.18445]] Capturing the security expert knowledge in feature selection for web application attack detection(https://arxiv.org/abs/2407.18445)
• Keywords: security, protect, attack
• Abstract: This article puts forward the use of mutual information values to replicate the expertise of security professionals in selecting features for detecting web attacks. The goal is to enhance the effectiveness of web application firewalls (WAFs). Web applications are frequently vulnerable to various security threats, making WAFs essential for their protection. WAFs analyze HTTP traffic using rule-based approaches to identify known attack patterns and to detect and block potential malicious requests. However, a major challenge is the occurrence of false positives, which can lead to blocking legitimate traffic and impact the normal functioning of the application. The problem is addressed as an approach that combines supervised learning for feature selection with a semi-supervised learning scenario for training a One-Class SVM model. The experimental findings show that the model trained with features selected by the proposed algorithm outperformed the expert-based selection approach in terms of performance. Additionally, the results obtained by the traditional rule-based WAF ModSecurity, configured with a vanilla set of OWASP CRS rules, were also improved.

Title: Textile Anomaly Detection: Evaluation of the State-of-the-Art for Automated Quality Inspection of Carpet

• Authors: Briony Forsberg, Dr Henry Williams, Prof Bruce MacDonald, Tracy Chen, Dr Kirstine Hulse
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18450
• Pdf URL: https://arxiv.org/pdf/2407.18450
• Copy Paste: [[2407.18450]] Textile Anomaly Detection: Evaluation of the State-of-the-Art for Automated Quality Inspection of Carpet(https://arxiv.org/abs/2407.18450)
• Keywords: robust, generative
• Abstract: In this study, state-of-the-art unsupervised detection models were evaluated for the purpose of automated anomaly inspection of wool carpets. A custom dataset of four unique types of carpet textures was created to thoroughly test the models and their robustness in detecting subtle anomalies in complex textures. Due to the requirements of an inline inspection system in a manufacturing use case, the metrics of importance in this study were accuracy in detecting anomalous areas, the number of false detections, and the inference times of each model for real-time performance. Of the evaluated models, the student-teacher network based methods were found on average to yield the highest detection accuracy and lowest false detection rates. When trained on a multi-class dataset the models were found to yield comparable if not better results than single-class training. Finally, in terms of detection speed, with exception to the generative model, all other evaluated models were found to have comparable inference times on a GPU, with an average of 0.16s per image. On a CPU, most of these models typically produced results between 1.5 to 2 times the respective GPU inference times.

Title: Fairness Definitions in Language Models Explained

• Authors: Thang Viet Doan, Zhibo Chu, Zichong Wang, Wenbin Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18454
• Pdf URL: https://arxiv.org/pdf/2407.18454
• Copy Paste: [[2407.18454]] Fairness Definitions in Language Models Explained(https://arxiv.org/abs/2407.18454)
• Keywords: fair
• Abstract: Language Models (LMs) have demonstrated exceptional performance across various Natural Language Processing (NLP) tasks. Despite these advancements, LMs can inherit and amplify societal biases related to sensitive attributes such as gender and race, limiting their adoption in real-world applications. Therefore, fairness has been extensively explored in LMs, leading to the proposal of various fairness notions. However, the lack of clear agreement on which fairness definition to apply in specific contexts (\textit{e.g.,} medium-sized LMs versus large-sized LMs) and the complexity of understanding the distinctions between these definitions can create confusion and impede further progress. To this end, this paper proposes a systematic survey that clarifies the definitions of fairness as they apply to LMs. Specifically, we begin with a brief introduction to LMs and fairness in LMs, followed by a comprehensive, up-to-date overview of existing fairness notions in LMs and the introduction of a novel taxonomy that categorizes these concepts based on their foundational principles and operational distinctions. We further illustrate each definition through experiments, showcasing their practical implications and outcomes. Finally, we discuss current research challenges and open questions, aiming to foster innovative ideas and advance the field. The implementation and additional resources are publicly available at this https URL.

Title: MistralBSM: Leveraging Mistral-7B for Vehicular Networks Misbehavior Detection

• Authors: Wissal Hamhoum, Soumaya Cherkaoui
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.18462
• Pdf URL: https://arxiv.org/pdf/2407.18462
• Copy Paste: [[2407.18462]] MistralBSM: Leveraging Mistral-7B for Vehicular Networks Misbehavior Detection(https://arxiv.org/abs/2407.18462)
• Keywords: security, attack, large language model
• Abstract: Vehicular networks are exposed to various threats resulting from malicious attacks. These threats compromise the security and reliability of communications among road users, thereby jeopardizing road and traffic safety. One of the main vectors of these attacks within vehicular networks is misbehaving vehicles. To address this challenge, we propose deploying a pretrained Large Language Model (LLM)-empowered Misbehavior Detection System (MDS) within an edge-cloud detection framework. Specifically, we fine-tune Mistral-7B, a state-of-the-art LLM, as the edge component to enable real-time detection, whereas a larger LLM deployed in the cloud can conduct a more comprehensive analysis. Our experiments conducted on the extended VeReMi dataset demonstrate Mistral-7B's superior performance, achieving 98\% accuracy compared to other LLMs such as LLAMA2-7B and RoBERTa. Additionally, we investigate the impact of window size on computational costs to optimize deployment efficiency. Leveraging LLMs in MDS shows interesting results in improving the detection of vehicle misbehavior, consequently strengthening vehicular network security to ensure the safety of road users.

Title: Machine Unlearning using a Multi-GAN based Model

• Authors: Amartya Hatua, Trung T. Nguyen, Andrew H. Sung
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18467
• Pdf URL: https://arxiv.org/pdf/2407.18467
• Copy Paste: [[2407.18467]] Machine Unlearning using a Multi-GAN based Model(https://arxiv.org/abs/2407.18467)
• Keywords: attack, membership infer, generative
• Abstract: This article presents a new machine unlearning approach that utilizes multiple Generative Adversarial Network (GAN) based models. The proposed method comprises two phases: i) data reorganization in which synthetic data using the GAN model is introduced with inverted class labels of the forget datasets, and ii) fine-tuning the pre-trained model. The GAN models consist of two pairs of generators and discriminators. The generator discriminator pairs generate synthetic data for the retain and forget datasets. Then, a pre-trained model is utilized to get the class labels of the synthetic datasets. The class labels of synthetic and original forget datasets are inverted. Finally, all combined datasets are used to fine-tune the pre-trained model to get the unlearned model. We have performed the experiments on the CIFAR-10 dataset and tested the unlearned models using Membership Inference Attacks (MIA). The inverted class labels procedure and synthetically generated data help to acquire valuable information that enables the model to outperform state-of-the-art models and other standard unlearning classifiers.

Title: Diffusion-Driven Semantic Communication for Generative Models with Bandwidth Constraints

• Authors: Lei Guo, Wei Chen, Yuxuan Sun, Bo Ai, Nikolaos Pappas, Tony Quek
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18468
• Pdf URL: https://arxiv.org/pdf/2407.18468
• Copy Paste: [[2407.18468]] Diffusion-Driven Semantic Communication for Generative Models with Bandwidth Constraints(https://arxiv.org/abs/2407.18468)
• Keywords: diffusion, generative
• Abstract: Diffusion models have been extensively utilized in AI-generated content (AIGC) in recent years, thanks to the superior generation capabilities. Combining with semantic communications, diffusion models are used for tasks such as denoising, data reconstruction, and content generation. However, existing diffusion-based generative models do not consider the stringent bandwidth limitation, which limits its application in wireless communication. This paper introduces a diffusion-driven semantic communication framework with advanced VAE-based compression for bandwidth-constrained generative model. Our designed architecture utilizes the diffusion model, where the signal transmission process through the wireless channel acts as the forward process in diffusion. To reduce bandwidth requirements, we incorporate a downsampling module and a paired upsampling module based on a variational auto-encoder with reparameterization at the receiver to ensure that the recovered features conform to the Gaussian distribution. Furthermore, we derive the loss function for our proposed system and evaluate its performance through comprehensive experiments. Our experimental results demonstrate significant improvements in pixel-level metrics such as peak signal to noise ratio (PSNR) and semantic metrics like learned perceptual image patch similarity (LPIPS). These enhancements are more profound regarding the compression rates and SNR compared to deep joint source-channel coding (DJSCC).

Title: Constructing the CORD-19 Vaccine Dataset

• Authors: Manisha Singh, Divy Sharma, Alonso Ma, Bridget Tyree, Margaret Mitchell
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18471
• Pdf URL: https://arxiv.org/pdf/2407.18471
• Copy Paste: [[2407.18471]] Constructing the CORD-19 Vaccine Dataset(https://arxiv.org/abs/2407.18471)
• Keywords: extraction
• Abstract: We introduce new dataset 'CORD-19-Vaccination' to cater to scientists specifically looking into COVID-19 vaccine-related research. This dataset is extracted from CORD-19 dataset [Wang et al., 2020] and augmented with new columns for language detail, author demography, keywords, and topic per paper. Facebook's fastText model is used to identify languages [Joulin et al., 2016]. To establish author demography (author affiliation, lab/institution location, and lab/institution country columns) we processed the JSON file for each paper and then further enhanced using Google's search API to determine country values. 'Yake' was used to extract keywords from the title, abstract, and body of each paper and the LDA (Latent Dirichlet Allocation) algorithm was used to add topic information [Campos et al., 2020, 2018a,b]. To evaluate the dataset, we demonstrate a question-answering task like the one used in the CORD-19 Kaggle challenge [Goldbloom et al., 2022]. For further evaluation, sequential sentence classification was performed on each paper's abstract using the model from Dernoncourt et al. [2016]. We partially hand annotated the training dataset and used a pre-trained BERT-PubMed layer. 'CORD- 19-Vaccination' contains 30k research papers and can be immensely valuable for NLP research such as text mining, information extraction, and question answering, specific to the domain of COVID-19 vaccine research.

Title: A Role-specific Guided Large Language Model for Ophthalmic Consultation Based on Stylistic Differentiation

• Authors: Laiyi Fu, Binbin Fan, Hongkai Du, Yanxiang Feng, Chunhua Li, Huping Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18483
• Pdf URL: https://arxiv.org/pdf/2407.18483
• Copy Paste: [[2407.18483]] A Role-specific Guided Large Language Model for Ophthalmic Consultation Based on Stylistic Differentiation(https://arxiv.org/abs/2407.18483)
• Keywords: large language model
• Abstract: Ophthalmology consultations are crucial for diagnosing, treating, and preventing eye diseases. However, the growing demand for consultations exceeds the availability of ophthalmologists. By leveraging large pre-trained language models, we can design effective dialogues for specific scenarios, aiding in consultations. Traditional fine-tuning strategies for question-answering tasks are impractical due to increasing model size and often ignoring patient-doctor role function during consultations. In this paper, we propose EyeDoctor, an ophthalmic medical questioning large language model that enhances accuracy through doctor-patient role perception guided and an augmented knowledge base with external disease information. Experimental results show EyeDoctor achieves higher question-answering precision in ophthalmology consultations. Notably, EyeDoctor demonstrated a 7.25% improvement in Rouge-1 scores and a 10.16% improvement in F1 scores on multi-round datasets compared to second best model ChatGPT, highlighting the importance of doctor-patient role differentiation and dynamic knowledge base expansion for intelligent medical consultations. EyeDoc also serves as a free available web based service and souce code is available at this https URL.

Title: SMPISD-MTPNet: Scene Semantic Prior-Assisted Infrared Ship Detection Using Multi-Task Perception Networks

• Authors: Chen Hu, Xiaogang Dong, Yian Huang Lele Wang, Liang Xu, Tian Pu, Zhenming Peng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18487
• Pdf URL: https://arxiv.org/pdf/2407.18487
• Copy Paste: [[2407.18487]] SMPISD-MTPNet: Scene Semantic Prior-Assisted Infrared Ship Detection Using Multi-Task Perception Networks(https://arxiv.org/abs/2407.18487)
• Keywords: robust, extraction, segmentation
• Abstract: Infrared ship detection (IRSD) has received increasing attention in recent years due to the robustness of infrared images to adverse weather. However, a large number of false alarms may occur in complex scenes. To address these challenges, we propose the Scene Semantic Prior-Assisted Multi-Task Perception Network (SMPISD-MTPNet), which includes three stages: scene semantic extraction, deep feature extraction, and prediction. In the scene semantic extraction stage, we employ a Scene Semantic Extractor (SSE) to guide the network by the features extracted based on expert knowledge. In the deep feature extraction stage, a backbone network is employed to extract deep features. These features are subsequently integrated by a fusion network, enhancing the detection capabilities across targets of varying sizes. In the prediction stage, we utilize the Multi-Task Perception Module, which includes the Gradient-based Module and the Scene Segmentation Module, enabling precise detection of small and dim targets within complex scenes. For the training process, we introduce the Soft Fine-tuning training strategy to suppress the distortion caused by data augmentation. Besides, due to the lack of a publicly available dataset labelled for scenes, we introduce the Infrared Ship Dataset with Scene Segmentation (IRSDSS). Finally, we evaluate the network and compare it with state-of-the-art (SOTA) methods, indicating that SMPISD-MTPNet outperforms existing approaches. The source code and dataset for this research can be accessed at this https URL.

Title: Towards More Accurate Prediction of Human Empathy and Emotion in Text and Multi-turn Conversations by Combining Advanced NLP, Transformers-based Networks, and Linguistic Methodologies

• Authors: Manisha Singh, Divy Sharma, Alonso Ma, Nora Goldfine
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18496
• Pdf URL: https://arxiv.org/pdf/2407.18496
• Copy Paste: [[2407.18496]] Towards More Accurate Prediction of Human Empathy and Emotion in Text and Multi-turn Conversations by Combining Advanced NLP, Transformers-based Networks, and Linguistic Methodologies(https://arxiv.org/abs/2407.18496)
• Keywords: transformer
• Abstract: Based on the WASSA 2022 Shared Task on Empathy Detection and Emotion Classification, we predict the level of empathic concern and personal distress displayed in essays. For the first stage of this project we implemented a Feed-Forward Neural Network using sentence-level embeddings as features. We experimented with four different embedding models for generating the inputs to the neural network. The subsequent stage builds upon the previous work and we have implemented three types of revisions. The first revision focuses on the enhancements to the model architecture and the training approach. The second revision focuses on handling class imbalance using stratified data sampling. The third revision focuses on leveraging lexical resources, where we apply four different resources to enrich the features associated with the dataset. During the final stage of this project, we have created the final end-to-end system for the primary task using an ensemble of models to revise primary task performance. Additionally, as part of the final stage, these approaches have been adapted to the WASSA 2023 Shared Task on Empathy Emotion and Personality Detection in Interactions, in which the empathic concern, emotion polarity, and emotion intensity in dyadic text conversations are predicted.

Title: Answerability Fields: Answerable Location Estimation via Diffusion Models

• Authors: Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, Koya Sakamoto, Motoaki Kawanabe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18497
• Pdf URL: https://arxiv.org/pdf/2407.18497
• Copy Paste: [[2407.18497]] Answerability Fields: Answerable Location Estimation via Diffusion Models(https://arxiv.org/abs/2407.18497)
• Keywords: diffusion
• Abstract: In an era characterized by advancements in artificial intelligence and robotics, enabling machines to interact with and understand their environment is a critical research endeavor. In this paper, we propose Answerability Fields, a novel approach to predicting answerability within complex indoor environments. Leveraging a 3D question answering dataset, we construct a comprehensive Answerability Fields dataset, encompassing diverse scenes and questions from ScanNet. Using a diffusion model, we successfully infer and evaluate these Answerability Fields, demonstrating the importance of objects and their locations in answering questions within a scene. Our results showcase the efficacy of Answerability Fields in guiding scene-understanding tasks, laying the foundation for their application in enhancing interactions between intelligent agents and their environments.

Title: A Reliable Common-Sense Reasoning Socialbot Built Using LLMs and Goal-Directed ASP

• Authors: Yankai Zeng, Abhiramon Rajashekharan, Kinjal Basu, Huaduo Wang, Joaquín Arias, Gopal Gupta
• Subjects: cs.CL, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2407.18498
• Pdf URL: https://arxiv.org/pdf/2407.18498
• Copy Paste: [[2407.18498]] A Reliable Common-Sense Reasoning Socialbot Built Using LLMs and Goal-Directed ASP(https://arxiv.org/abs/2407.18498)
• Keywords: large language model
• Abstract: The development of large language models (LLMs), such as GPT, has enabled the construction of several socialbots, like ChatGPT, that are receiving a lot of attention for their ability to simulate a human conversation. However, the conversation is not guided by a goal and is hard to control. In addition, because LLMs rely more on pattern recognition than deductive reasoning, they can give confusing answers and have difficulty integrating multiple topics into a cohesive response. These limitations often lead the LLM to deviate from the main topic to keep the conversation interesting. We propose AutoCompanion, a socialbot that uses an LLM model to translate natural language into predicates (and vice versa) and employs commonsense reasoning based on Answer Set Programming (ASP) to hold a social conversation with a human. In particular, we rely on s(CASP), a goal-directed implementation of ASP as the backend. This paper presents the framework design and how an LLM is used to parse user messages and generate a response from the s(CASP) engine output. To validate our proposal, we describe (real) conversations in which the chatbot's goal is to keep the user entertained by talking about movies and books, and s(CASP) ensures (i) correctness of answers, (ii) coherence (and precision) during the conversation, which it dynamically regulates to achieve its specific purpose, and (iii) no deviation from the main topic.

Title: Revisit Event Generation Model: Self-Supervised Learning of Event-to-Video Reconstruction with Implicit Neural Representations

• Authors: Zipeng Wang, Yunfan Lu, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18500
• Pdf URL: https://arxiv.org/pdf/2407.18500
• Copy Paste: [[2407.18500]] Revisit Event Generation Model: Self-Supervised Learning of Event-to-Video Reconstruction with Implicit Neural Representations(https://arxiv.org/abs/2407.18500)
• Keywords: interpretability
• Abstract: Reconstructing intensity frames from event data while maintaining high temporal resolution and dynamic range is crucial for bridging the gap between event-based and frame-based computer vision. Previous approaches have depended on supervised learning on synthetic data, which lacks interpretability and risk over-fitting to the setting of the event simulator. Recently, self-supervised learning (SSL) based methods, which primarily utilize per-frame optical flow to estimate intensity via photometric constancy, has been actively investigated. However, they are vulnerable to errors in the case of inaccurate optical flow. This paper proposes a novel SSL event-to-video reconstruction approach, dubbed EvINR, which eliminates the need for labeled data or optical flow estimation. Our core idea is to reconstruct intensity frames by directly addressing the event generation model, essentially a partial differential equation (PDE) that describes how events are generated based on the time-varying brightness signals. Specifically, we utilize an implicit neural representation (INR), which takes in spatiotemporal coordinate $(x, y, t)$ and predicts intensity values, to represent the solution of the event generation equation. The INR, parameterized as a fully-connected Multi-layer Perceptron (MLP), can be optimized with its temporal derivatives supervised by events. To make EvINR feasible for online requisites, we propose several acceleration techniques that substantially expedite the training process. Comprehensive experiments demonstrate that our EvINR surpasses previous SSL methods by 38% w.r.t. Mean Squared Error (MSE) and is comparable or superior to SoTA supervised methods. Project page: this https URL.

Title: Homomorphic Encryption-Enabled Federated Learning for Privacy-Preserving Intrusion Detection in Resource-Constrained IoV Networks

• Authors: Bui Duc Manh, Chi-Hieu Nguyen, Dinh Thai Hoang, Diep N. Nguyen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.18503
• Pdf URL: https://arxiv.org/pdf/2407.18503
• Copy Paste: [[2407.18503]] Homomorphic Encryption-Enabled Federated Learning for Privacy-Preserving Intrusion Detection in Resource-Constrained IoV Networks(https://arxiv.org/abs/2407.18503)
• Keywords: secure, privacy, federate
• Abstract: This paper aims to propose a novel framework to address the data privacy issue for Federated Learning (FL)-based Intrusion Detection Systems (IDSs) in Internet-of-Vehicles(IoVs) with limited computational resources. In particular, in conventional FL systems, it is usually assumed that the computing nodes have sufficient computational resources to process the training tasks. However, in practical IoV systems, vehicles usually have limited computational resources to process intensive training tasks, compromising the effectiveness of deploying FL in IDSs. While offloading data from vehicles to the cloud can mitigate this issue, it introduces significant privacy concerns for vehicle users (VUs). To resolve this issue, we first propose a highly-effective framework using homomorphic encryption to secure data that requires offloading to a centralized server for processing. Furthermore, we develop an effective training algorithm tailored to handle the challenges of FL-based systems with encrypted data. This algorithm allows the centralized server to directly compute on quantum-secure encrypted ciphertexts without needing decryption. This approach not only safeguards data privacy during the offloading process from VUs to the centralized server but also enhances the efficiency of utilizing FL for IDSs in IoV systems. Our simulation results show that our proposed approach can achieve a performance that is as close to that of the solution without encryption, with a gap of less than 0.8%.

Title: TCGPN: Temporal-Correlation Graph Pre-trained Network for Stock Forecasting

• Authors: Wenbo Yan, Ying Tan
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.18519
• Pdf URL: https://arxiv.org/pdf/2407.18519
• Copy Paste: [[2407.18519]] TCGPN: Temporal-Correlation Graph Pre-trained Network for Stock Forecasting(https://arxiv.org/abs/2407.18519)
• Keywords: robust
• Abstract: Recently, the incorporation of both temporal features and the correlation across time series has become an effective approach in time series prediction. Spatio-Temporal Graph Neural Networks (STGNNs) demonstrate good performance on many Temporal-correlation Forecasting Problem. However, when applied to tasks lacking periodicity, such as stock data prediction, the effectiveness and robustness of STGNNs are found to be unsatisfactory. And STGNNs are limited by memory savings so that cannot handle problems with a large number of nodes. In this paper, we propose a novel approach called the Temporal-Correlation Graph Pre-trained Network (TCGPN) to address these limitations. TCGPN utilize Temporal-correlation fusion encoder to get a mixed representation and pre-training method with carefully designed temporal and correlation pre-training tasks. Entire structure is independent of the number and order of nodes, so better results can be obtained through various data enhancements. And memory consumption during training can be significantly reduced through multiple sampling. Experiments are conducted on real stock market data sets CSI300 and CSI500 that exhibit minimal periodicity. We fine-tune a simple MLP in downstream tasks and achieve state-of-the-art results, validating the capability to capture more robust temporal correlation patterns.

Title: DTFormer: A Transformer-Based Method for Discrete-Time Dynamic Graph Representation Learning

• Authors: Xi Chen, Yun Xiong, Siwei Zhang, Jiawei Zhang, Yao Zhang, Shiyang Zhou, Xixi Wu, Mingyang Zhang, Tengfei Liu, Weiqiang Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18523
• Pdf URL: https://arxiv.org/pdf/2407.18523
• Copy Paste: [[2407.18523]] DTFormer: A Transformer-Based Method for Discrete-Time Dynamic Graph Representation Learning(https://arxiv.org/abs/2407.18523)
• Keywords: transformer
• Abstract: Discrete-Time Dynamic Graphs (DTDGs), which are prevalent in real-world implementations and notable for their ease of data acquisition, have garnered considerable attention from both academic researchers and industry practitioners. The representation learning of DTDGs has been extensively applied to model the dynamics of temporally changing entities and their evolving connections. Currently, DTDG representation learning predominantly relies on GNN+RNN architectures, which manifest the inherent limitations of both Graph Neural Networks (GNNs) and Recurrent Neural Networks (RNNs). GNNs suffer from the over-smoothing issue as the models architecture goes deeper, while RNNs struggle to capture long-term dependencies effectively. GNN+RNN architectures also grapple with scaling to large graph sizes and long sequences. Additionally, these methods often compute node representations separately and focus solely on individual node characteristics, thereby overlooking the behavior intersections between the two nodes whose link is being predicted, such as instances where the two nodes appear together in the same context or share common neighbors. This paper introduces a novel representation learning method DTFormer for DTDGs, pivoting from the traditional GNN+RNN framework to a Transformer-based architecture. Our approach exploits the attention mechanism to concurrently process topological information within the graph at each timestamp and temporal dynamics of graphs along the timestamps, circumventing the aforementioned fundamental weakness of both GNNs and RNNs. Moreover, we enhance the model's expressive capability by incorporating the intersection relationships among nodes and integrating a multi-patching module. Extensive experiments conducted on six public dynamic graph benchmark datasets confirm our model's efficacy, achieving the SOTA performance.

Title: Is larger always better? Evaluating and prompting large language models for non-generative medical tasks

• Authors: Yinghao Zhu, Junyi Gao, Zixiang Wang, Weibin Liao, Xiaochen Zheng, Lifang Liang, Yasha Wang, Chengwei Pan, Ewen M. Harrison, Liantao Ma
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18525
• Pdf URL: https://arxiv.org/pdf/2407.18525
• Copy Paste: [[2407.18525]] Is larger always better? Evaluating and prompting large language models for non-generative medical tasks(https://arxiv.org/abs/2407.18525)
• Keywords: robust, generative, large language model
• Abstract: The use of Large Language Models (LLMs) in medicine is growing, but their ability to handle both structured Electronic Health Record (EHR) data and unstructured clinical notes is not well-studied. This study benchmarks various models, including GPT-based LLMs, BERT-based models, and traditional clinical predictive models, for non-generative medical tasks utilizing renowned datasets. We assessed 14 language models (9 GPT-based and 5 BERT-based) and 7 traditional predictive models using the MIMIC dataset (ICU patient records) and the TJH dataset (early COVID-19 EHR data), focusing on tasks such as mortality and readmission prediction, disease hierarchy reconstruction, and biomedical sentence matching, comparing both zero-shot and finetuned performance. Results indicated that LLMs exhibited robust zero-shot predictive capabilities on structured EHR data when using well-designed prompting strategies, frequently surpassing traditional models. However, for unstructured medical texts, LLMs did not outperform finetuned BERT models, which excelled in both supervised and unsupervised tasks. Consequently, while LLMs are effective for zero-shot learning on structured data, finetuned BERT models are more suitable for unstructured texts, underscoring the importance of selecting models based on specific task requirements and data characteristics to optimize the application of NLP technology in healthcare.

Title: Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformers

• Authors: Longkun Zou, Wanru Zhu, Ke Chen, Lihua Guo, Kailing Guo, Kui Jia, Yaowei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18534
• Pdf URL: https://arxiv.org/pdf/2407.18534
• Copy Paste: [[2407.18534]] Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformers(https://arxiv.org/abs/2407.18534)
• Keywords: transformer
• Abstract: Semantic pattern of an object point cloud is determined by its topological configuration of local geometries. Learning discriminative representations can be challenging due to large shape variations of point sets in local regions and incomplete surface in a global perspective, which can be made even more severe in the context of unsupervised domain adaptation (UDA). In specific, traditional 3D networks mainly focus on local geometric details and ignore the topological structure between local geometries, which greatly limits their cross-domain generalization. Recently, the transformer-based models have achieved impressive performance gain in a range of image-based tasks, benefiting from its strong generalization capability and scalability stemming from capturing long range correlation across local patches. Inspired by such successes of visual transformers, we propose a novel Relational Priors Distillation (RPD) method to extract relational priors from the well-trained transformers on massive images, which can significantly empower cross-domain representations with consistent topological priors of objects. To this end, we establish a parameter-frozen pre-trained transformer module shared between 2D teacher and 3D student models, complemented by an online knowledge distillation strategy for semantically regularizing the 3D student model. Furthermore, we introduce a novel self-supervised task centered on reconstructing masked point cloud patches using corresponding masked multi-view image features, thereby empowering the model with incorporating 3D geometric information. Experiments on the PointDA-10 and the Sim-to-Real datasets verify that the proposed method consistently achieves the state-of-the-art performance of UDA for point cloud classification. The source code of this work is available at this https URL.

Title: Towards a Multidimensional Evaluation Framework for Empathetic Conversational Systems

• Authors: Aravind Sesagiri Raamkumar, Siyuan Brandon Loh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18538
• Pdf URL: https://arxiv.org/pdf/2407.18538
• Copy Paste: [[2407.18538]] Towards a Multidimensional Evaluation Framework for Empathetic Conversational Systems(https://arxiv.org/abs/2407.18538)
• Keywords: large language model
• Abstract: Empathetic Conversational Systems (ECS) are built to respond empathetically to the user's emotions and sentiments, regardless of the application domain. Current ECS studies evaluation approaches are restricted to offline evaluation experiments primarily for gold standard comparison & benchmarking, and user evaluation studies for collecting human ratings on specific constructs. These methods are inadequate in measuring the actual quality of empathy in conversations. In this paper, we propose a multidimensional empathy evaluation framework with three new methods for measuring empathy at (i) structural level using three empathy-related dimensions, (ii) behavioral level using empathy behavioral types, and (iii) overall level using an empathy lexicon, thereby fortifying the evaluation process. Experiments were conducted with the state-of-the-art ECS models and large language models (LLMs) to show the framework's usefulness.

Title: A Universal Prompting Strategy for Extracting Process Model Information from Natural Language Text using Large Language Models

• Authors: Julian Neuberger, Lars Ackermann, Han van der Aa, Stefan Jablonski
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18540
• Pdf URL: https://arxiv.org/pdf/2407.18540
• Copy Paste: [[2407.18540]] A Universal Prompting Strategy for Extracting Process Model Information from Natural Language Text using Large Language Models(https://arxiv.org/abs/2407.18540)
• Keywords: extraction, generative, large language model
• Abstract: Over the past decade, extensive research efforts have been dedicated to the extraction of information from textual process descriptions. Despite the remarkable progress witnessed in natural language processing (NLP), information extraction within the Business Process Management domain remains predominantly reliant on rule-based systems and machine learning methodologies. Data scarcity has so far prevented the successful application of deep learning techniques. However, the rapid progress in generative large language models (LLMs) makes it possible to solve many NLP tasks with very high quality without the need for extensive data. Therefore, we systematically investigate the potential of LLMs for extracting information from textual process descriptions, targeting the detection of process elements such as activities and actors, and relations between them. Using a heuristic algorithm, we demonstrate the suitability of the extracted information for process model generation. Based on a novel prompting strategy, we show that LLMs are able to outperform state-of-the-art machine learning approaches with absolute performance improvements of up to 8\% $F_1$ score across three different datasets. We evaluate our prompting strategy on eight different LLMs, showing it is universally applicable, while also analyzing the impact of certain prompt parts on extraction quality. The number of example texts, the specificity of definitions, and the rigour of format instructions are identified as key for improving the accuracy of extracted information. Our code, prompts, and data are publicly available.

Title: Utilising Explainable Techniques for Quality Prediction in a Complex Textiles Manufacturing Use Case

• Authors: Briony Forsberg, Dr Henry Williams, Prof Bruce MacDonald, Tracy Chen, Dr Reza Hamzeh, Dr Kirstine Hulse
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18544
• Pdf URL: https://arxiv.org/pdf/2407.18544
• Copy Paste: [[2407.18544]] Utilising Explainable Techniques for Quality Prediction in a Complex Textiles Manufacturing Use Case(https://arxiv.org/abs/2407.18544)
• Keywords: explainability
• Abstract: This paper develops an approach to classify instances of product failure in a complex textiles manufacturing dataset using explainable techniques. The dataset used in this study was obtained from a New Zealand manufacturer of woollen carpets and rugs. In investigating the trade-off between accuracy and explainability, three different tree-based classification algorithms were evaluated: a Decision Tree and two ensemble methods, Random Forest and XGBoost. Additionally, three feature selection methods were also evaluated: the SelectKBest method, using chi-squared as the scoring function, the Pearson Correlation Coefficient, and the Boruta algorithm. Not surprisingly, the ensemble methods typically produced better results than the Decision Tree model. The Random Forest model yielded the best results overall when combined with the Boruta feature selection technique. Finally, a tree ensemble explaining technique was used to extract rule lists to capture necessary and sufficient conditions for classification by a trained model that could be easily interpreted by a human. Notably, several features that were in the extracted rule lists were statistical features and calculated features that were added to the original dataset. This demonstrates the influence that bringing in additional information during the data preprocessing stages can have on the ultimate model performance.

Title: Skin Cancer Detection utilizing Deep Learning: Classification of Skin Lesion Images using a Vision Transformer

• Authors: Carolin Flosdorf, Justin Engelker, Igor Keller, Nicolas Mohr
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18554
• Pdf URL: https://arxiv.org/pdf/2407.18554
• Copy Paste: [[2407.18554]] Skin Cancer Detection utilizing Deep Learning: Classification of Skin Lesion Images using a Vision Transformer(https://arxiv.org/abs/2407.18554)
• Keywords: transformer
• Abstract: Skin cancer detection still represents a major challenge in healthcare. Common detection methods can be lengthy and require human assistance which falls short in many countries. Previous research demonstrates how convolutional neural networks (CNNs) can help effectively through both automation and an accuracy that is comparable to the human level. However, despite the progress in previous decades, the precision is still limited, leading to substantial misclassifications that have a serious impact on people's health. Hence, we employ a Vision Transformer (ViT) that has been developed in recent years based on the idea of a self-attention mechanism, specifically two configurations of a pre-trained ViT. We generally find superior metrics for classifying skin lesions after comparing them to base models such as decision tree classifier and k-nearest neighbor (KNN) classifier, as well as to CNNs and less complex ViTs. In particular, we attach greater importance to the performance of melanoma, which is the most lethal type of skin cancer. The ViT-L32 model achieves an accuracy of 91.57% and a melanoma recall of 58.54%, while ViT-L16 achieves an accuracy of 92.79% and a melanoma recall of 56.10%. This offers a potential tool for faster and more accurate diagnoses and an overall improvement for the healthcare sector.

Title: Look Globally and Reason: Two-stage Path Reasoning over Sparse Knowledge Graphs

• Authors: Saiping Guan, Jiyao Wei, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18556
• Pdf URL: https://arxiv.org/pdf/2407.18556
• Copy Paste: [[2407.18556]] Look Globally and Reason: Two-stage Path Reasoning over Sparse Knowledge Graphs(https://arxiv.org/abs/2407.18556)
• Keywords: explainability
• Abstract: Sparse Knowledge Graphs (KGs), frequently encountered in real-world applications, contain fewer facts in the form of (head entity, relation, tail entity) compared to more populated KGs. The sparse KG completion task, which reasons answers for given queries in the form of (head entity, relation, ?) for sparse KGs, is particularly challenging due to the necessity of reasoning missing facts based on limited facts. Path-based models, known for excellent explainability, are often employed for this task. However, existing path-based models typically rely on external models to fill in missing facts and subsequently perform path reasoning. This approach introduces unexplainable factors or necessitates meticulous rule design. In light of this, this paper proposes an alternative approach by looking inward instead of seeking external assistance. We introduce a two-stage path reasoning model called LoGRe (Look Globally and Reason) over sparse KGs. LoGRe constructs a relation-path reasoning schema by globally analyzing the training data to alleviate the sparseness problem. Based on this schema, LoGRe then aggregates paths to reason out answers. Experimental results on five benchmark sparse KG datasets demonstrate the effectiveness of the proposed LoGRe model.

Title: VSSD: Vision Mamba with Non-Casual State Space Duality

• Authors: Yuheng Shi, Minjing Dong, Mingjia Li, Chang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18559
• Pdf URL: https://arxiv.org/pdf/2407.18559
• Copy Paste: [[2407.18559]] VSSD: Vision Mamba with Non-Casual State Space Duality(https://arxiv.org/abs/2407.18559)
• Keywords: robust, transformer, segmentation
• Abstract: Vision transformers have significantly advanced the field of computer vision, offering robust modeling capabilities and global receptive field. However, their high computational demands limit their applicability in processing long sequences. To tackle this issue, State Space Models (SSMs) have gained prominence in vision tasks as they offer linear computational complexity. Recently, State Space Duality (SSD), an improved variant of SSMs, was introduced in Mamba2 to enhance model performance and efficiency. However, the inherent causal nature of SSD/SSMs restricts their applications in non-causal vision tasks. To address this limitation, we introduce Visual State Space Duality (VSSD) model, which has a non-causal format of SSD. Specifically, we propose to discard the magnitude of interactions between the hidden state and tokens while preserving their relative weights, which relieves the dependencies of token contribution on previous tokens. Together with the involvement of multi-scan strategies, we show that the scanning results can be integrated to achieve non-causality, which not only improves the performance of SSD in vision tasks but also enhances its efficiency. We conduct extensive experiments on various benchmarks including image classification, detection, and segmentation, where VSSD surpasses existing state-of-the-art SSM-based models. Code and weights are available at \url{this https URL}.

Title: Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation

• Authors: Chaoyi Ai, Yong Jiang, Shen Huang, Pengjun Xie, Kewei Tu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18562
• Pdf URL: https://arxiv.org/pdf/2407.18562
• Copy Paste: [[2407.18562]] Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation(https://arxiv.org/abs/2407.18562)
• Keywords: robust, transformer
• Abstract: Named entity recognition (NER) models often struggle with noisy inputs, such as those with spelling mistakes or errors generated by Optical Character Recognition processes, and learning a robust NER model is challenging. Existing robust NER models utilize both noisy text and its corresponding gold text for training, which is infeasible in many real-world applications in which gold text is not available. In this paper, we consider a more realistic setting in which only noisy text and its NER labels are available. We propose to retrieve relevant text of the noisy text from a knowledge corpus and use it to enhance the representation of the original noisy input. We design three retrieval methods: sparse retrieval based on lexicon similarity, dense retrieval based on semantic similarity, and self-retrieval based on task-specific text. After retrieving relevant text, we concatenate the retrieved text with the original noisy text and encode them with a transformer network, utilizing self-attention to enhance the contextual token representations of the noisy text using the retrieved text. We further employ a multi-view training framework that improves robust NER without retrieving text during inference. Experiments show that our retrieval-augmented model achieves significant improvements in various noisy NER settings.

Title: Unveiling Privacy Vulnerabilities: Investigating the Role of Structure in Graph Data

• Authors: Hanyang Yuan, Jiarong Xu, Cong Wang, Ziqi Yang, Chunping Wang, Keting Yin, Yang Yang
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2407.18564
• Pdf URL: https://arxiv.org/pdf/2407.18564
• Copy Paste: [[2407.18564]] Unveiling Privacy Vulnerabilities: Investigating the Role of Structure in Graph Data(https://arxiv.org/abs/2407.18564)
• Keywords: privacy, protect, attack
• Abstract: The public sharing of user information opens the door for adversaries to infer private data, leading to privacy breaches and facilitating malicious activities. While numerous studies have concentrated on privacy leakage via public user attributes, the threats associated with the exposure of user relationships, particularly through network structure, are often neglected. This study aims to fill this critical gap by advancing the understanding and protection against privacy risks emanating from network structure, moving beyond direct connections with neighbors to include the broader implications of indirect network structural patterns. To achieve this, we first investigate the problem of Graph Privacy Leakage via Structure (GPS), and introduce a novel measure, the Generalized Homophily Ratio, to quantify the various mechanisms contributing to privacy breach risks in GPS. Based on this insight, we develop a novel graph private attribute inference attack, which acts as a pivotal tool for evaluating the potential for privacy leakage through network structures under worst-case scenarios. To protect users' private data from such vulnerabilities, we propose a graph data publishing method incorporating a learnable graph sampling technique, effectively transforming the original graph into a privacy-preserving version. Extensive experiments demonstrate that our attack model poses a significant threat to user privacy, and our graph data publishing method successfully achieves the optimal privacy-utility trade-off compared to baselines.

Title: Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation

• Authors: Jingjun Yi, Qi Bi, Hao Zheng, Haolan Zhan, Wei Ji, Yawen Huang, Yuexiang Li, Yefeng Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18568
• Pdf URL: https://arxiv.org/pdf/2407.18568
• Copy Paste: [[2407.18568]] Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation(https://arxiv.org/abs/2407.18568)
• Keywords: extraction, segmentation
• Abstract: The rapid development of Vision Foundation Model (VFM) brings inherent out-domain generalization for a variety of down-stream tasks. Among them, domain generalized semantic segmentation (DGSS) holds unique challenges as the cross-domain images share common pixel-wise content information but vary greatly in terms of the style. In this paper, we present a novel Spectral-dEcomposed Token (SET) learning framework to advance the frontier. Delving into further than existing fine-tuning token & frozen backbone paradigm, the proposed SET especially focuses on the way learning style-invariant features from these learnable tokens. Particularly, the frozen VFM features are first decomposed into the phase and amplitude components in the frequency space, which mainly contain the information of content and style, respectively, and then separately processed by learnable tokens for task-specific information extraction. After the decomposition, style variation primarily impacts the token-based feature enhancement within the amplitude branch. To address this issue, we further develop an attention optimization method to bridge the gap between style-affected representation and static tokens during inference. Extensive cross-domain experiments show its state-of-the-art performance.

Title: Learning to Enhance Aperture Phasor Field for Non-Line-of-Sight Imaging

• Authors: In Cho, Hyunbo Shim, Seon Joo Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18574
• Pdf URL: https://arxiv.org/pdf/2407.18574
• Copy Paste: [[2407.18574]] Learning to Enhance Aperture Phasor Field for Non-Line-of-Sight Imaging(https://arxiv.org/abs/2407.18574)
• Keywords: robust
• Abstract: This paper aims to facilitate more practical NLOS imaging by reducing the number of samplings and scan areas. To this end, we introduce a phasor-based enhancement network that is capable of predicting clean and full measurements from noisy partial observations. We leverage a denoising autoencoder scheme to acquire rich and noise-robust representations in the measurement space. Through this pipeline, our enhancement network is trained to accurately reconstruct complete measurements from their corrupted and partial counterparts. However, we observe that the \naive application of denoising often yields degraded and over-smoothed results, caused by unnecessary and spurious frequency signals present in measurements. To address this issue, we introduce a phasor-based pipeline designed to limit the spectrum of our network to the frequency range of interests, where the majority of informative signals are detected. The phasor wavefronts at the aperture, which are band-limited signals, are employed as inputs and outputs of the network, guiding our network to learn from the frequency range of interests and discard unnecessary information. The experimental results in more practical acquisition scenarios demonstrate that we can look around the corners with $16\times$ or $64\times$ fewer samplings and $4\times$ smaller apertures. Our code is available at \url{this https URL}.

Title: HICEScore: A Hierarchical Metric for Image Captioning Evaluation

• Authors: Zequn Zeng, Jianqiao Sun, Hao Zhang, Tiansheng Wen, Yudi Su, Yan Xie, Zhengjue Wang, Bo Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18589
• Pdf URL: https://arxiv.org/pdf/2407.18589
• Copy Paste: [[2407.18589]] HICEScore: A Hierarchical Metric for Image Captioning Evaluation(https://arxiv.org/abs/2407.18589)
• Keywords: large language model
• Abstract: Image captioning evaluation metrics can be divided into two categories, reference-based metrics and reference-free metrics. However, reference-based approaches may struggle to evaluate descriptive captions with abundant visual details produced by advanced multimodal large language models, due to their heavy reliance on limited human-annotated references. In contrast, previous reference-free metrics have been proven effective via CLIP cross-modality similarity. Nonetheless, CLIP-based metrics, constrained by their solution of global image-text compatibility, often have a deficiency in detecting local textual hallucinations and are insensitive to small visual objects. Besides, their single-scale designs are unable to provide an interpretable evaluation process such as pinpointing the position of caption mistakes and identifying visual regions that have not been described. To move forward, we propose a novel reference-free metric for image captioning evaluation, dubbed Hierarchical Image Captioning Evaluation Score (HICE-S). By detecting local visual regions and textual phrases, HICE-S builds an interpretable hierarchical scoring mechanism, breaking through the barriers of the single-scale structure of existing reference-free metrics. Comprehensive experiments indicate that our proposed metric achieves the SOTA performance on several benchmarks, outperforming existing reference-free metrics like CLIP-S and PAC-S, and reference-based metrics like METEOR and CIDEr. Moreover, several case studies reveal that the assessment process of HICE-S on detailed captions closely resembles interpretable human judgments.Our code is available at this https URL.

Title: LinguaLinker: Audio-Driven Portraits Animation with Implicit Facial Control Enhancement

• Authors: Rui Zhang, Yixiao Fang, Zhengnan Lu, Pei Cheng, Zebiao Huang, Bin Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18595
• Pdf URL: https://arxiv.org/pdf/2407.18595
• Copy Paste: [[2407.18595]] LinguaLinker: Audio-Driven Portraits Animation with Implicit Facial Control Enhancement(https://arxiv.org/abs/2407.18595)
• Keywords: diffusion
• Abstract: This study delves into the intricacies of synchronizing facial dynamics with multilingual audio inputs, focusing on the creation of visually compelling, time-synchronized animations through diffusion-based techniques. Diverging from traditional parametric models for facial animation, our approach, termed LinguaLinker, adopts a holistic diffusion-based framework that integrates audio-driven visual synthesis to enhance the synergy between auditory stimuli and visual responses. We process audio features separately and derive the corresponding control gates, which implicitly govern the movements in the mouth, eyes, and head, irrespective of the portrait's origin. The advanced audio-driven visual synthesis mechanism provides nuanced control but keeps the compatibility of output video and input audio, allowing for a more tailored and effective portrayal of distinct personas across different languages. The significant improvements in the fidelity of animated portraits, the accuracy of lip-syncing, and the appropriate motion variations achieved by our method render it a versatile tool for animating any portrait in any language.

Title: Denoising L\'evy Probabilistic Models

• Authors: Dario Shariatian, Umut Simsekli, Alain Durmus
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.18609
• Pdf URL: https://arxiv.org/pdf/2407.18609
• Copy Paste: [[2407.18609]] Denoising L\'evy Probabilistic Models(https://arxiv.org/abs/2407.18609)
• Keywords: diffusion, generative
• Abstract: Investigating noise distribution beyond Gaussian in diffusion generative models is an open problem. The Gaussian case has seen success experimentally and theoretically, fitting a unified SDE framework for score-based and denoising formulations. Recent studies suggest heavy-tailed noise distributions can address mode collapse and manage datasets with class imbalance, heavy tails, or outliers. Yoon et al. (NeurIPS 2023) introduced the Lévy-Ito model (LIM), extending the SDE framework to heavy-tailed SDEs with $\alpha$-stable noise. Despite its theoretical elegance and performance gains, LIM's complex mathematics may limit its accessibility and broader adoption. This study takes a simpler approach by extending the denoising diffusion probabilistic model (DDPM) with $\alpha$-stable noise, creating the denoising Lévy probabilistic model (DLPM). Using elementary proof techniques, we show DLPM reduces to running vanilla DDPM with minimal changes, allowing the use of existing implementations with minimal changes. DLPM and LIM have different training algorithms and, unlike the Gaussian case, they admit different backward processes and sampling algorithms. Our experiments demonstrate that DLPM achieves better coverage of data distribution tail, improved generation of unbalanced datasets, and faster computation times with fewer backward steps.

Title: Dilated Strip Attention Network for Image Restoration

• Authors: Fangwei Hao, Jiesheng Wu, Ji Du, Yinjie Wang, Jing Xu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.18613
• Pdf URL: https://arxiv.org/pdf/2407.18613
• Copy Paste: [[2407.18613]] Dilated Strip Attention Network for Image Restoration(https://arxiv.org/abs/2407.18613)
• Keywords: robust, transformer
• Abstract: Image restoration is a long-standing task that seeks to recover the latent sharp image from its deteriorated counterpart. Due to the robust capacity of self-attention to capture long-range dependencies, transformer-based methods or some attention-based convolutional neural networks have demonstrated promising results on many image restoration tasks in recent years. However, existing attention modules encounters limited receptive fields or abundant parameters. In order to integrate contextual information more effectively and efficiently, in this paper, we propose a dilated strip attention network (DSAN) for image restoration. Specifically, to gather more contextual information for each pixel from its neighboring pixels in the same row or column, a dilated strip attention (DSA) mechanism is elaborately proposed. By employing the DSA operation horizontally and vertically, each location can harvest the contextual information from a much wider region. In addition, we utilize multi-scale receptive fields across different feature groups in DSA to improve representation learning. Extensive experiments show that our DSAN outperforms state-of-the-art algorithms on several image restoration tasks.

Title: Every Part Matters: Integrity Verification of Scientific Figures Based on Multimodal Large Language Models

• Authors: Xiang Shi, Jiawei Liu, Yinpeng Liu, Qikai Cheng, Wei Lu
• Subjects: cs.CL, cs.AI, cs.CV, cs.DL, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.18626
• Pdf URL: https://arxiv.org/pdf/2407.18626
• Copy Paste: [[2407.18626]] Every Part Matters: Integrity Verification of Scientific Figures Based on Multimodal Large Language Models(https://arxiv.org/abs/2407.18626)
• Keywords: large language model
• Abstract: This paper tackles a key issue in the interpretation of scientific figures: the fine-grained alignment of text and figures. It advances beyond prior research that primarily dealt with straightforward, data-driven visualizations such as bar and pie charts and only offered a basic understanding of diagrams through captioning and classification. We introduce a novel task, Figure Integrity Verification, designed to evaluate the precision of technologies in aligning textual knowledge with visual elements in scientific figures. To support this, we develop a semi-automated method for constructing a large-scale dataset, Figure-seg, specifically designed for this task. Additionally, we propose an innovative framework, Every Part Matters (EPM), which leverages Multimodal Large Language Models (MLLMs) to not only incrementally improve the alignment and verification of text-figure integrity but also enhance integrity through analogical reasoning. Our comprehensive experiments show that these innovations substantially improve upon existing methods, allowing for more precise and thorough analysis of complex scientific figures. This progress not only enhances our understanding of multimodal technologies but also stimulates further research and practical applications across fields requiring the accurate interpretation of complex visual data.

Title: Robust VAEs via Generating Process of Noise Augmented Data

• Authors: Hiroo Irobe, Wataru Aoki, Kimihiro Yamazaki, Yuhui Zhang, Takumi Nakagawa, Hiroki Waida, Yuichiro Wada, Takafumi Kanamori
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18632
• Pdf URL: https://arxiv.org/pdf/2407.18632
• Copy Paste: [[2407.18632]] Robust VAEs via Generating Process of Noise Augmented Data(https://arxiv.org/abs/2407.18632)
• Keywords: defense, attack, robust, generative
• Abstract: Advancing defensive mechanisms against adversarial attacks in generative models is a critical research topic in machine learning. Our study focuses on a specific type of generative models - Variational Auto-Encoders (VAEs). Contrary to common beliefs and existing literature which suggest that noise injection towards training data can make models more robust, our preliminary experiments revealed that naive usage of noise augmentation technique did not substantially improve VAE robustness. In fact, it even degraded the quality of learned representations, making VAEs more susceptible to adversarial perturbations. This paper introduces a novel framework that enhances robustness by regularizing the latent space divergence between original and noise-augmented data. Through incorporating a paired probabilistic prior into the standard variational lower bound, our method significantly boosts defense against adversarial attacks. Our empirical evaluations demonstrate that this approach, termed Robust Augmented Variational Auto-ENcoder (RAVEN), yields superior performance in resisting adversarial inputs on widely-recognized benchmark datasets.

Title: Vulnerability Detection in Ethereum Smart Contracts via Machine Learning: A Qualitative Analysis

• Authors: Dalila Ressi, Alvise Spanò, Lorenzo Benetollo, Carla Piazza, Michele Bugliesi, Sabina Rossi
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18639
• Pdf URL: https://arxiv.org/pdf/2407.18639
• Copy Paste: [[2407.18639]] Vulnerability Detection in Ethereum Smart Contracts via Machine Learning: A Qualitative Analysis(https://arxiv.org/abs/2407.18639)
• Keywords: secure, security
• Abstract: Smart contracts are central to a myriad of critical blockchain applications, from financial transactions to supply chain management. However, their adoption is hindered by security vulnerabilities that can result in significant financial losses. Most vulnerability detection tools and methods available nowadays leverage either static analysis methods or machine learning. Unfortunately, as valuable as they are, both approaches suffer from limitations that make them only partially effective. In this survey, we analyze the state of the art in machine-learning vulnerability detection for Ethereum smart contracts, by categorizing existing tools and methodologies, evaluating them, and highlighting their limitations. Our critical assessment unveils issues such as restricted vulnerability coverage and dataset construction flaws, providing us with new metrics to overcome the difficulties that restrain a sound comparison of existing solutions. Driven by our findings, we discuss best practices to enhance the accuracy, scope, and efficiency of vulnerability detection in smart contracts. Our guidelines address the known flaws while at the same time opening new avenues for research and development. By shedding light on current challenges and offering novel directions for improvement, we contribute to the advancement of secure smart contract development and blockchain technology as a whole.

Title: Auto DragGAN: Editing the Generative Image Manifold in an Autoregressive Manner

• Authors: Pengxiang Cai, Zhiwei Liu, Guibo Zhu, Yunfang Niu, Jinqiao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18656
• Pdf URL: https://arxiv.org/pdf/2407.18656
• Copy Paste: [[2407.18656]] Auto DragGAN: Editing the Generative Image Manifold in an Autoregressive Manner(https://arxiv.org/abs/2407.18656)
• Keywords: transformer, generative
• Abstract: Pixel-level fine-grained image editing remains an open challenge. Previous works fail to achieve an ideal trade-off between control granularity and inference speed. They either fail to achieve pixel-level fine-grained control, or their inference speed requires optimization. To address this, this paper for the first time employs a regression-based network to learn the variation patterns of StyleGAN latent codes during the image dragging process. This method enables pixel-level precision in dragging editing with little time cost. Users can specify handle points and their corresponding target points on any GAN-generated images, and our method will move each handle point to its corresponding target point. Through experimental analysis, we discover that a short movement distance from handle points to target points yields a high-fidelity edited image, as the model only needs to predict the movement of a small portion of pixels. To achieve this, we decompose the entire movement process into multiple sub-processes. Specifically, we develop a transformer encoder-decoder based network named 'Latent Predictor' to predict the latent code motion trajectories from handle points to target points in an autoregressive manner. Moreover, to enhance the prediction stability, we introduce a component named 'Latent Regularizer', aimed at constraining the latent code motion within the distribution of natural images. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) inference speed and image editing performance at the pixel-level granularity.

Title: Adversarial Robustification via Text-to-Image Diffusion Models

• Authors: Daewon Choi, Jongheon Jeong, Huiwon Jang, Jinwoo Shin
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18658
• Pdf URL: https://arxiv.org/pdf/2407.18658
• Copy Paste: [[2407.18658]] Adversarial Robustification via Text-to-Image Diffusion Models(https://arxiv.org/abs/2407.18658)
• Keywords: attack, robust, diffusion, data-free
• Abstract: Adversarial robustness has been conventionally believed as a challenging property to encode for neural networks, requiring plenty of training data. In the recent paradigm of adopting off-the-shelf models, however, access to their training data is often infeasible or not practical, while most of such models are not originally trained concerning adversarial robustness. In this paper, we develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data. Our intuition is to view recent text-to-image diffusion models as "adaptable" denoisers that can be optimized to specify target tasks. Based on this, we propose: (a) to initiate a denoise-and-classify pipeline that offers provable guarantees against adversarial attacks, and (b) to leverage a few synthetic reference images generated from the text-to-image model that enables novel adaptation schemes. Our experiments show that our data-free scheme applied to the pre-trained CLIP could improve the (provable) adversarial robustness of its diverse zero-shot classification derivatives (while maintaining their accuracy), significantly surpassing prior approaches that utilize the full training data. Not only for CLIP, we also demonstrate that our framework is easily applicable for robustifying other visual classifiers efficiently.

Title: Local Binary Pattern(LBP) Optimization for Feature Extraction

• Authors: Zeinab Sedaghatjoo, Hossein Hosseinzadeh, Bahram Sadeghi Bigham
• Subjects: cs.CV, math.NA
• Abstract URL: https://arxiv.org/abs/2407.18665
• Pdf URL: https://arxiv.org/pdf/2407.18665
• Copy Paste: [[2407.18665]] Local Binary Pattern(LBP) Optimization for Feature Extraction(https://arxiv.org/abs/2407.18665)
• Keywords: extraction, segmentation
• Abstract: The rapid growth of image data has led to the development of advanced image processing and computer vision techniques, which are crucial in various applications such as image classification, image segmentation, and pattern recognition. Texture is an important feature that has been widely used in many image processing tasks. Therefore, analyzing and understanding texture plays a pivotal role in image analysis and understanding.Local binary pattern (LBP) is a powerful operator that describes the local texture features of images. This paper provides a novel mathematical representation of the LBP by separating the operator into three matrices, two of which are always fixed and do not depend on the input data. These fixed matrices are analyzed in depth, and a new algorithm is proposed to optimize them for improved classification performance. The optimization process is based on the singular value decomposition (SVD) algorithm. As a result, the authors present optimal LBPs that effectively describe the texture of human face images. Several experiment results presented in this paper convincingly verify the efficiency and superiority of the optimized LBPs for face detection and facial expression recognition tasks.

Title: A Labeled Ophthalmic Ultrasound Dataset with Medical Report Generation Based on Cross-modal Deep Learning

• Authors: Jing Wang, Junyan Fan, Meng Zhou, Yanzhu Zhang, Mingyu Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18667
• Pdf URL: https://arxiv.org/pdf/2407.18667
• Copy Paste: [[2407.18667]] A Labeled Ophthalmic Ultrasound Dataset with Medical Report Generation Based on Cross-modal Deep Learning(https://arxiv.org/abs/2407.18667)
• Keywords: privacy, protect
• Abstract: Ultrasound imaging reveals eye morphology and aids in diagnosing and treating eye diseases. However, interpreting diagnostic reports requires specialized physicians. We present a labeled ophthalmic dataset for the precise analysis and the automated exploration of medical images along with their associated reports. It collects three modal data, including the ultrasound images, blood flow information and examination reports from 2,417 patients at an ophthalmology hospital in Shenyang, China, during the year 2018, in which the patient information is de-identified for privacy protection. To the best of our knowledge, it is the only ophthalmic dataset that contains the three modal information simultaneously. It incrementally consists of 4,858 images with the corresponding free-text reports, which describe 15 typical imaging findings of intraocular diseases and the corresponding anatomical locations. Each image shows three kinds of blood flow indices at three specific arteries, i.e., nine parameter values to describe the spectral characteristics of blood flow distribution. The reports were written by ophthalmologists during the clinical care. The proposed dataset is applied to generate medical report based on the cross-modal deep learning model. The experimental results demonstrate that our dataset is suitable for training supervised models concerning cross-modal medical data.

Title: A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attention

• Authors: João D. Nunes, Diana Montezuma, Domingos Oliveira, Tania Pereira, Jaime S. Cardoso
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18673
• Pdf URL: https://arxiv.org/pdf/2407.18673
• Copy Paste: [[2407.18673]] A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attention(https://arxiv.org/abs/2407.18673)
• Keywords: extraction, segmentation
• Abstract: Manually annotating nuclei from the gigapixel Hematoxylin and Eosin (H&E)-stained Whole Slide Images (WSIs) is a laborious and costly task, meaning automated algorithms for cell nuclei instance segmentation and classification could alleviate the workload of pathologists and clinical researchers and at the same time facilitate the automatic extraction of clinically interpretable features. But due to high intra- and inter-class variability of nuclei morphological and chromatic features, as well as H&E-stains susceptibility to artefacts, state-of-the-art algorithms cannot correctly detect and classify instances with the necessary performance. In this work, we hypothesise context and attention inductive biases in artificial neural networks (ANNs) could increase the generalization of algorithms for cell nuclei instance segmentation and classification. We conduct a thorough survey on context and attention methods for cell nuclei instance segmentation and classification from H&E-stained microscopy imaging, while providing a comprehensive discussion of the challenges being tackled with context and attention. Besides, we illustrate some limitations of current approaches and present ideas for future research. As a case study, we extend both a general instance segmentation and classification method (Mask-RCNN) and a tailored cell nuclei instance segmentation and classification model (HoVer-Net) with context- and attention-based mechanisms, and do a comparative analysis on a multi-centre colon nuclei identification and counting dataset. Although pathologists rely on context at multiple levels while paying attention to specific Regions of Interest (RoIs) when analysing and annotating WSIs, our findings suggest translating that domain knowledge into algorithm design is no trivial task, but to fully exploit these mechanisms, the scientific understanding of these methods should be addressed.

Title: Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift

• Authors: Seongho Son, William Bankes, Sayak Ray Chowdhury, Brooks Paige, Ilija Bogunovic
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18676
• Pdf URL: https://arxiv.org/pdf/2407.18676
• Copy Paste: [[2407.18676]] Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift(https://arxiv.org/abs/2407.18676)
• Keywords: robust, large language model
• Abstract: Reinforcement learning from human feedback (RLHF) aligns Large Language Models (LLMs) with human preferences. However, these preferences can often change over time due to external factors (e.g. environment change and societal influence). Consequently, what was wrong then might be right now. Current preference optimization algorithms do not account for temporal preference drift in their modeling, which can lead to severe misalignment. To address this limitation, we use a Dynamic Bradley-Terry model that models preferences via time-dependent reward functions, and propose Non-Stationary Direct Preference Optimisation (NS-DPO). By introducing a discount parameter in the loss function, NS-DPO applies exponential weighting, which proportionally focuses learning on more time-relevant datapoints. We theoretically analyse the convergence of NS-DPO in the offline setting, providing upper bounds on the estimation error caused by non-stationary preferences. Finally, we demonstrate the effectiveness of NS-DPO1 for fine-tuning LLMs in scenarios with drifting preferences. By simulating preference drift using renowned reward models and modifying popular LLM datasets accordingly, we show that NS-DPO fine-tuned LLMs remain robust under non-stationarity, significantly outperforming baseline algorithms that ignore temporal preference changes, without sacrificing performance in stationary cases.

Title: VeriCHERI: Exhaustive Formal Security Verification of CHERI at the RTL

• Authors: Anna Lena Duque Antón, Johannes Müller, Philipp Schmitz, Tobias Jauch, Alex Wezel, Lucas Deutschmann, Mohammad Rahmani Fadiheh, Dominik Stoffel, Wolfgang Kunz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.18679
• Pdf URL: https://arxiv.org/pdf/2407.18679
• Copy Paste: [[2407.18679]] VeriCHERI: Exhaustive Formal Security Verification of CHERI at the RTL(https://arxiv.org/abs/2407.18679)
• Keywords: security, protect, attack
• Abstract: Protecting data in memory from attackers continues to be a concern in computing systems. CHERI is a promising approach to achieve such protection, by providing and enforcing fine-grained memory protection directly in the hardware. Creating trust for the entire system stack, however, requires a gap-free verification of CHERI's hardware-based protection mechanisms. Existing verification methods for CHERI target the abstract ISA model rather than the underlying hardware implementation. Fully ensuring the CHERI security guarantees for a concrete RTL implementation is a challenge in previous flows and demands high manual efforts. This paper presents VeriCHERI, a novel approach to security verification. It is conceptionally different from previous works in that it does not require any ISA specification. Instead of checking compliance with a golden ISA model, we check against well-established global security objectives of confidentiality and integrity. Fully covering these objectives, VeriCHERI uses as few as four unbounded properties to exhaustively prove or disprove any vulnerability. We demonstrate the effectiveness and scalability of VeriCHERI on a RISC-V based processor implementing a CHERI variant.

Title: Graph Neural Networks for Virtual Sensing in Complex Systems: Addressing Heterogeneous Temporal Dynamics

• Authors: Mengjie Zhao, Cees Taal, Stephan Baggerohr, Olga Fink
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2407.18691
• Pdf URL: https://arxiv.org/pdf/2407.18691
• Copy Paste: [[2407.18691]] Graph Neural Networks for Virtual Sensing in Complex Systems: Addressing Heterogeneous Temporal Dynamics(https://arxiv.org/abs/2407.18691)
• Keywords: robust
• Abstract: Real-time condition monitoring is crucial for the reliable and efficient operation of complex systems. However, relying solely on physical sensors can be limited due to their cost, placement constraints, or inability to directly measure certain critical parameters. Virtual sensing addresses these limitations by leveraging readily available sensor data and system knowledge to estimate inaccessible parameters or infer system states. The increasing complexity of industrial systems necessitates deployments of sensors with diverse modalities to provide a comprehensive understanding of system states. These sensors capture data at varying frequencies to monitor both rapid and slowly varying system dynamics, as well as local and global state evolutions of the systems. This leads to heterogeneous temporal dynamics, which, particularly under varying operational end environmental conditions, pose a significant challenge for accurate virtual sensing. To address this, we propose a Heterogeneous Temporal Graph Neural Network (HTGNN) framework. HTGNN explicitly models signals from diverse sensors and integrates operating conditions into the model architecture. We evaluate HTGNN using two newly released datasets: a bearing dataset with diverse load conditions for bearing load prediction and a year-long simulated dataset for predicting bridge live loads. Our results demonstrate that HTGNN significantly outperforms established baseline methods in both tasks, particularly under highly varying operating conditions. These results highlight HTGNN's potential as a robust and accurate virtual sensing approach for complex systems, paving the way for improved monitoring, predictive maintenance, and enhanced system performance.

Title: Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation

• Authors: Esteban Garces Arias, Julian Rodemann, Meimingwei Li, Christian Heumann, Matthias Aßenmacher
• Subjects: cs.CL, cs.LG, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.18698
• Pdf URL: https://arxiv.org/pdf/2407.18698
• Copy Paste: [[2407.18698]] Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation(https://arxiv.org/abs/2407.18698)
• Keywords: large language model
• Abstract: Decoding from the output distributions of large language models to produce high-quality text is a complex challenge in language modeling. Various approaches, such as beam search, sampling with temperature, $k-$sampling, nucleus $p-$sampling, typical decoding, contrastive decoding, and contrastive search, have been proposed to address this problem, aiming to improve coherence, diversity, as well as resemblance to human-generated text. In this study, we introduce adaptive contrastive search, a novel decoding strategy extending contrastive search by incorporating an adaptive degeneration penalty, guided by the estimated uncertainty of the model at each generation step. This strategy is designed to enhance both the creativity and diversity of the language modeling process while at the same time producing coherent and high-quality generated text output. Our findings indicate performance enhancement in both aspects, across different model architectures and datasets, underscoring the effectiveness of our method in text generation tasks. Our code base, datasets, and models are publicly available.

Title: BCTR: Bidirectional Conditioning Transformer for Scene Graph Generation

• Authors: Peng Hao, Xiaobing Wang, Yingying Jiang, Hanchao Jia, Xiaoshuai Hao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18715
• Pdf URL: https://arxiv.org/pdf/2407.18715
• Copy Paste: [[2407.18715]] BCTR: Bidirectional Conditioning Transformer for Scene Graph Generation(https://arxiv.org/abs/2407.18715)
• Keywords: transformer
• Abstract: Scene Graph Generation (SGG) remains a challenging task due to its compositional property. Previous approaches improve prediction efficiency by learning in an end-to-end manner. However, these methods exhibit limited performance as they assume unidirectional conditioning between entities and predicates, leading to insufficient information interaction. To address this limitation, we propose a novel bidirectional conditioning factorization for SGG, introducing efficient interaction between entities and predicates. Specifically, we develop an end-to-end scene graph generation model, Bidirectional Conditioning Transformer (BCTR), to implement our factorization. BCTR consists of two key modules. First, the Bidirectional Conditioning Generator (BCG) facilitates multi-stage interactive feature augmentation between entities and predicates, enabling mutual benefits between the two predictions. Second, Random Feature Alignment (RFA) regularizes the feature space by distilling multi-modal knowledge from pre-trained models, enhancing BCTR's ability on tailed categories without relying on statistical priors. We conduct a series of experiments on Visual Genome and Open Image V6, demonstrating that BCTR achieves state-of-the-art performance on both benchmarks. The code will be available upon acceptance of the paper.

Title: ChatSchema: A pipeline of extracting structured information with Large Multimodal Models based on schema

• Authors: Fei Wang, Yuewen Zheng, Qin Li, Jingyi Wu, Pengfei Li, Luxia Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18716
• Pdf URL: https://arxiv.org/pdf/2407.18716
• Copy Paste: [[2407.18716]] ChatSchema: A pipeline of extracting structured information with Large Multimodal Models based on schema(https://arxiv.org/abs/2407.18716)
• Keywords: extraction
• Abstract: Objective: This study introduces ChatSchema, an effective method for extracting and structuring information from unstructured data in medical paper reports using a combination of Large Multimodal Models (LMMs) and Optical Character Recognition (OCR) based on the schema. By integrating predefined schema, we intend to enable LMMs to directly extract and standardize information according to the schema specifications, facilitating further data entry. Method: Our approach involves a two-stage process, including classification and extraction for categorizing report scenarios and structuring information. We established and annotated a dataset to verify the effectiveness of ChatSchema, and evaluated key extraction using precision, recall, F1-score, and accuracy metrics. Based on key extraction, we further assessed value extraction. We conducted ablation studies on two LMMs to illustrate the improvement of structured information extraction with different input modals and methods. Result: We analyzed 100 medical reports from Peking University First Hospital and established a ground truth dataset with 2,945 key-value pairs. We evaluated ChatSchema using GPT-4o and Gemini 1.5 Pro and found a higher overall performance of GPT-4o. The results are as follows: For the result of key extraction, key-precision was 98.6%, key-recall was 98.5%, key-F1-score was 98.6%. For the result of value extraction based on correct key extraction, the overall accuracy was 97.2%, precision was 95.8%, recall was 95.8%, and F1-score was 95.8%. An ablation study demonstrated that ChatSchema achieved significantly higher overall accuracy and overall F1-score of key-value extraction, compared to the Baseline, with increases of 26.9% overall accuracy and 27.4% overall F1-score, respectively.

Title: LLASP: Fine-tuning Large Language Models for Answer Set Programming

• Authors: Erica Coppolillo, Francesco Calimeri, Giuseppe Manco, Simona Perri, Francesco Ricca
• Subjects: cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2407.18723
• Pdf URL: https://arxiv.org/pdf/2407.18723
• Copy Paste: [[2407.18723]] LLASP: Fine-tuning Large Language Models for Answer Set Programming(https://arxiv.org/abs/2407.18723)
• Keywords: large language model
• Abstract: Recently, Large Language Models (LLMs) have showcased their potential in various natural language processing tasks, including code generation. However, while significant progress has been made in adapting LLMs to generate code for several imperative programming languages and tasks, there remains a notable gap in their application to declarative formalisms, such as Answer Set Programming (ASP). In this paper, we move a step towards exploring the capabilities of LLMs for ASP code generation. First, we perform a systematic evaluation of several state-of-the-art LLMs. Despite their power in terms of number of parameters, training data and computational resources, empirical results demonstrate inadequate performances in generating correct ASP programs. Therefore, we propose LLASP, a fine-tuned lightweight model specifically trained to encode fundamental ASP program patterns. To this aim, we create an ad-hoc dataset covering a wide variety of fundamental problem specifications that can be encoded in ASP. Our experiments demonstrate that the quality of ASP programs generated by LLASP is remarkable. This holds true not only when compared to the non-fine-tuned counterpart but also when compared to the majority of eager LLM candidates, particularly from a semantic perspective. All the code and data used to perform the experiments are publicly available at https://anonymous.4open.science/r/LLASP-D86C/.

Title: AutoRDF2GML: Facilitating RDF Integration in Graph Machine Learning

• Authors: Michael Färber, David Lamprecht, Yuni Susanti
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.18735
• Pdf URL: https://arxiv.org/pdf/2407.18735
• Copy Paste: [[2407.18735]] AutoRDF2GML: Facilitating RDF Integration in Graph Machine Learning(https://arxiv.org/abs/2407.18735)
• Keywords: extraction
• Abstract: In this paper, we introduce AutoRDF2GML, a framework designed to convert RDF data into data representations tailored for graph machine learning tasks. AutoRDF2GML enables, for the first time, the creation of both content-based features -- i.e., features based on RDF datatype properties -- and topology-based features -- i.e., features based on RDF object properties. Characterized by automated feature extraction, AutoRDF2GML makes it possible even for users less familiar with RDF and SPARQL to generate data representations ready for graph machine learning tasks, such as link prediction, node classification, and graph classification. Furthermore, we present four new benchmark datasets for graph machine learning, created from large RDF knowledge graphs using our framework. These datasets serve as valuable resources for evaluating graph machine learning approaches, such as graph neural networks. Overall, our framework effectively bridges the gap between the Graph Machine Learning and Semantic Web communities, paving the way for RDF-based machine learning applications.

Title: Towards Generalized Offensive Language Identification

• Authors: Alphaeus Dmonte, Tejas Arya, Tharindu Ranasinghe, Marcos Zampieri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18738
• Pdf URL: https://arxiv.org/pdf/2407.18738
• Copy Paste: [[2407.18738]] Towards Generalized Offensive Language Identification(https://arxiv.org/abs/2407.18738)
• Keywords: robust, large language model
• Abstract: The prevalence of offensive content on the internet, encompassing hate speech and cyberbullying, is a pervasive issue worldwide. Consequently, it has garnered significant attention from the machine learning (ML) and natural language processing (NLP) communities. As a result, numerous systems have been developed to automatically identify potentially harmful content and mitigate its impact. These systems can follow two approaches; (1) Use publicly available models and application endpoints, including prompting large language models (LLMs) (2) Annotate datasets and train ML models on them. However, both approaches lack an understanding of how generalizable they are. Furthermore, the applicability of these systems is often questioned in off-domain and practical environments. This paper empirically evaluates the generalizability of offensive language detection models and datasets across a novel generalized benchmark. We answer three research questions on generalizability. Our findings will be useful in creating robust real-world offensive language detection systems.

Title: Towards Effective and Efficient Continual Pre-training of Large Language Models

• Authors: Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Wayne Xin Zhao, Zhicheng Dou, Jiaxin Mao, Yankai Lin, Ruihua Song, Jun Xu, Xu Chen, Rui Yan, Zhewei Wei, Di Hu, Wenbing Huang, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18743
• Pdf URL: https://arxiv.org/pdf/2407.18743
• Copy Paste: [[2407.18743]] Towards Effective and Efficient Continual Pre-training of Large Language Models(https://arxiv.org/abs/2407.18743)
• Keywords: large language model
• Abstract: Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. To make the CPT approach more traceable, this paper presents a technical report for continually pre-training Llama-3 (8B), which significantly enhances the Chinese language ability and scientific reasoning ability of the backbone model. To enhance the new abilities while retaining the original abilities, we design specific data mixture and curriculum strategies by utilizing existing datasets and synthesizing high-quality datasets. Specifically, we synthesize multidisciplinary scientific question and answer (QA) pairs based on related web pages, and subsequently incorporate these synthetic data to improve the scientific reasoning ability of Llama-3. We refer to the model after CPT as Llama-3-SynE (Synthetic data Enhanced Llama-3). We also present the tuning experiments with a relatively small model -- TinyLlama, and employ the derived findings to train the backbone model. Extensive experiments on a number of evaluation benchmarks show that our approach can largely improve the performance of the backbone models, including both the general abilities (+8.81 on C-Eval and +6.31 on CMMLU) and the scientific reasoning abilities (+12.00 on MATH and +4.13 on SciEval), without hurting the original capacities. Our model, data, and codes are available at this https URL.

Title: FairAIED: Navigating Fairness, Bias, and Ethics in Educational AI Applications

• Authors: Sribala Vidyadhari Chinta, Zichong Wang, Zhipeng Yin, Nhat Hoang, Matthew Gonzalez, Tai Le Quy, Wenbin Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18745
• Pdf URL: https://arxiv.org/pdf/2407.18745
• Copy Paste: [[2407.18745]] FairAIED: Navigating Fairness, Bias, and Ethics in Educational AI Applications(https://arxiv.org/abs/2407.18745)
• Keywords: fair
• Abstract: The integration of Artificial Intelligence (AI) into education has transformative potential, providing tailored learning experiences and creative instructional approaches. However, the inherent biases in AI algorithms hinder this improvement by unintentionally perpetuating prejudice against specific demographics, especially in human-centered applications like education. This survey delves deeply into the developing topic of algorithmic fairness in educational contexts, providing a comprehensive evaluation of the diverse literature on fairness, bias, and ethics in AI-driven educational applications. It identifies the common forms of biases, such as data-related, algorithmic, and user-interaction, that fundamentally undermine the accomplishment of fairness in AI teaching aids. By outlining existing techniques for mitigating these biases, ranging from varied data gathering to algorithmic fairness interventions, the survey emphasizes the critical role of ethical considerations and legal frameworks in shaping a more equitable educational environment. Furthermore, it guides readers through the complexities of fairness measurements, methods, and datasets, shedding light on the way to bias reduction. Despite these gains, this survey highlights long-standing issues, such as achieving a balance between fairness and accuracy, as well as the need for diverse datasets. Overcoming these challenges and ensuring the ethical and fair use of AI's promise in education call for a collaborative, interdisciplinary approach.

Title: FLUE: Federated Learning with Un-Encrypted model weights

• Authors: Elie Atallah
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18750
• Pdf URL: https://arxiv.org/pdf/2407.18750
• Copy Paste: [[2407.18750]] FLUE: Federated Learning with Un-Encrypted model weights(https://arxiv.org/abs/2407.18750)
• Keywords: privacy, federate
• Abstract: Federated Learning enables diverse devices to collaboratively train a shared model while keeping training data locally stored, avoiding the need for centralized cloud storage. Despite existing privacy measures, concerns arise from potential reverse engineering of gradients, even with added noise, revealing private data. To address this, recent research emphasizes using encrypted model parameters during training. This paper introduces a novel federated learning algorithm, leveraging coded local gradients without encryption, exchanging coded proxies for model parameters, and injecting surplus noise for enhanced privacy. Two algorithm variants are presented, showcasing convergence and learning rates adaptable to coding schemes and raw data characteristics. Two encryption-free implementations with fixed and random coding matrices are provided, demonstrating promising simulation results from both federated optimization and machine learning perspectives.

Title: Knowledge Graph Structure as Prompt: Improving Small Language Models Capabilities for Knowledge-based Causal Discovery

• Authors: Yuni Susanti, Michael Färber
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18752
• Pdf URL: https://arxiv.org/pdf/2407.18752
• Copy Paste: [[2407.18752]] Knowledge Graph Structure as Prompt: Improving Small Language Models Capabilities for Knowledge-based Causal Discovery(https://arxiv.org/abs/2407.18752)
• Keywords: large language model
• Abstract: Causal discovery aims to estimate causal structures among variables based on observational data. Large Language Models (LLMs) offer a fresh perspective to tackle the causal discovery problem by reasoning on the metadata associated with variables rather than their actual data values, an approach referred to as knowledge-based causal discovery. In this paper, we investigate the capabilities of Small Language Models (SLMs, defined as LLMs with fewer than 1 billion parameters) with prompt-based learning for knowledge-based causal discovery. Specifically, we present KG Structure as Prompt, a novel approach for integrating structural information from a knowledge graph, such as common neighbor nodes and metapaths, into prompt-based learning to enhance the capabilities of SLMs. Experimental results on three types of biomedical and open-domain datasets under few-shot settings demonstrate the effectiveness of our approach, surpassing most baselines and even conventional fine-tuning approaches trained on full datasets. Our findings further highlight the strong capabilities of SLMs: in combination with knowledge graphs and prompt-based learning, SLMs demonstrate the potential to surpass LLMs with larger number of parameters. Our code and datasets are available on GitHub.

Title: Java-Class-Hijack: Software Supply Chain Attack for Java based on Maven Dependency Resolution and Java Classloading

• Authors: Federico Bono, Frank Reyes, Aman Sharma, Benoit Baudry, Martin Monperrus
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.18760
• Pdf URL: https://arxiv.org/pdf/2407.18760
• Copy Paste: [[2407.18760]] Java-Class-Hijack: Software Supply Chain Attack for Java based on Maven Dependency Resolution and Java Classloading(https://arxiv.org/abs/2407.18760)
• Keywords: security, attack
• Abstract: We introduce Java-Class-Hijack, a novel software supply chain attack that enables an attacker to inject malicious code by crafting a class that shadows a legitimate class that is in the dependency tree. We describe the attack, provide a proof-of-concept demonstrating its feasibility, and replicate it in the German Corona-Warn-App server application. The proof-of-concept illustrates how a transitive dependency deep within the dependency tree can hijack a class from a direct dependency and entirely alter its behavior, posing a significant security risk to Java applications. The replication on the Corona-Warn-App demonstrates how compromising a small JSON validation library could result in a complete database takeover.

Title: The power of Prompts: Evaluating and Mitigating Gender Bias in MT with LLMs

• Authors: Aleix Sant, Carlos Escolano, Audrey Mash, Francesca De Luca Fornaciari, Maite Melero
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18786
• Pdf URL: https://arxiv.org/pdf/2407.18786
• Copy Paste: [[2407.18786]] The power of Prompts: Evaluating and Mitigating Gender Bias in MT with LLMs(https://arxiv.org/abs/2407.18786)
• Keywords: large language model
• Abstract: This paper studies gender bias in machine translation through the lens of Large Language Models (LLMs). Four widely-used test sets are employed to benchmark various base LLMs, comparing their translation quality and gender bias against state-of-the-art Neural Machine Translation (NMT) models for English to Catalan (En $\rightarrow$ Ca) and English to Spanish (En $\rightarrow$ Es) translation directions. Our findings reveal pervasive gender bias across all models, with base LLMs exhibiting a higher degree of bias compared to NMT models. To combat this bias, we explore prompting engineering techniques applied to an instruction-tuned LLM. We identify a prompt structure that significantly reduces gender bias by up to 12% on the WinoMT evaluation dataset compared to more straightforward prompts. These results significantly reduce the gender bias accuracy gap between LLMs and traditional NMT systems.

Title: Granularity is crucial when applying differential privacy to text: An investigation for neural machine translation

• Authors: Doan Nam Long Vu, Timour Igamberdiev, Ivan Habernal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18789
• Pdf URL: https://arxiv.org/pdf/2407.18789
• Copy Paste: [[2407.18789]] Granularity is crucial when applying differential privacy to text: An investigation for neural machine translation(https://arxiv.org/abs/2407.18789)
• Keywords: privacy, protect, attack, membership infer
• Abstract: Applying differential privacy (DP) by means of the DP-SGD algorithm to protect individual data points during training is becoming increasingly popular in NLP. However, the choice of granularity at which DP is applied is often neglected. For example, neural machine translation (NMT) typically operates on the sentence-level granularity. From the perspective of DP, this setup assumes that each sentence belongs to a single person and any two sentences in the training dataset are independent. This assumption is however violated in many real-world NMT datasets, e.g. those including dialogues. For proper application of DP we thus must shift from sentences to entire documents. In this paper, we investigate NMT at both the sentence and document levels, analyzing the privacy/utility trade-off for both scenarios, and evaluating the risks of not using the appropriate privacy granularity in terms of leaking personally identifiable information (PII). Our findings indicate that the document-level NMT system is more resistant to membership inference attacks, emphasizing the significance of using the appropriate granularity when working with DP.

Title: Robust Learning in Bayesian Parallel Branching Graph Neural Networks: The Narrow Width Limit

• Authors: Zechen Zhang, Haim Sompolinsky
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18807
• Pdf URL: https://arxiv.org/pdf/2407.18807
• Copy Paste: [[2407.18807]] Robust Learning in Bayesian Parallel Branching Graph Neural Networks: The Narrow Width Limit(https://arxiv.org/abs/2407.18807)
• Keywords: robust
• Abstract: The infinite width limit of random neural networks is known to result in Neural Networks as Gaussian Process (NNGP) (Lee et al. [2018]), characterized by task-independent kernels. It is widely accepted that larger network widths contribute to improved generalization (Park et al. [2019]). However, this work challenges this notion by investigating the narrow width limit of the Bayesian Parallel Branching Graph Neural Network (BPB-GNN), an architecture that resembles residual networks. We demonstrate that when the width of a BPB-GNN is significantly smaller compared to the number of training examples, each branch exhibits more robust learning due to a symmetry breaking of branches in kernel renormalization. Surprisingly, the performance of a BPB-GNN in the narrow width limit is generally superior or comparable to that achieved in the wide width limit in bias-limited scenarios. Furthermore, the readout norms of each branch in the narrow width limit are mostly independent of the architectural hyperparameters but generally reflective of the nature of the data. Our results characterize a newly defined narrow-width regime for parallel branching networks in general.

Title: Deep Companion Learning: Enhancing Generalization Through Historical Consistency

• Authors: Ruizhao Zhu, Venkatesh Saligrama
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18821
• Pdf URL: https://arxiv.org/pdf/2407.18821
• Copy Paste: [[2407.18821]] Deep Companion Learning: Enhancing Generalization Through Historical Consistency(https://arxiv.org/abs/2407.18821)
• Keywords: transformer
• Abstract: We propose Deep Companion Learning (DCL), a novel training method for Deep Neural Networks (DNNs) that enhances generalization by penalizing inconsistent model predictions compared to its historical performance. To achieve this, we train a deep-companion model (DCM), by using previous versions of the model to provide forecasts on new inputs. This companion model deciphers a meaningful latent semantic structure within the data, thereby providing targeted supervision that encourages the primary model to address the scenarios it finds most challenging. We validate our approach through both theoretical analysis and extensive experimentation, including ablation studies, on a variety of benchmark datasets (CIFAR-100, Tiny-ImageNet, ImageNet-1K) using diverse architectural models (ShuffleNetV2, ResNet, Vision Transformer, etc.), demonstrating state-of-the-art performance.

Title: Accurate and Scalable Detection and Investigation of Cyber Persistence Threats

• Authors: Qi Liu, Muhammad Shoaib, Mati Ur Rehman, Kaibin Bao, Veit Hagenmeyer, Wajih Ul Hassan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.18832
• Pdf URL: https://arxiv.org/pdf/2407.18832
• Copy Paste: [[2407.18832]] Accurate and Scalable Detection and Investigation of Cyber Persistence Threats(https://arxiv.org/abs/2407.18832)
• Keywords: attack, steal
• Abstract: In Advanced Persistent Threat (APT) attacks, achieving stealthy persistence within target systems is often crucial for an attacker's success. This persistence allows adversaries to maintain prolonged access, often evading detection mechanisms. Recognizing its pivotal role in the APT lifecycle, this paper introduces Cyber Persistence Detector (CPD), a novel system dedicated to detecting cyber persistence through provenance analytics. CPD is founded on the insight that persistent operations typically manifest in two phases: the "persistence setup" and the subsequent "persistence execution". By causally relating these phases, we enhance our ability to detect persistent threats. First, CPD discerns setups signaling an impending persistent threat and then traces processes linked to remote connections to identify persistence execution activities. A key feature of our system is the introduction of pseudo-dependency edges (pseudo-edges), which effectively connect these disjoint phases using data provenance analysis, and expert-guided edges, which enable faster tracing and reduced log size. These edges empower us to detect persistence threats accurately and efficiently. Moreover, we propose a novel alert triage algorithm that further reduces false positives associated with persistence threats. Evaluations conducted on well-known datasets demonstrate that our system reduces the average false positive rate by 93% compared to state-of-the-art methods.

Title: Scalable Group Choreography via Variational Phase Manifold Learning

• Authors: Nhat Le, Khoa Do, Xuan Bui, Tuong Do, Erman Tjiputra, Quang D.Tran, Anh Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18839
• Pdf URL: https://arxiv.org/pdf/2407.18839
• Copy Paste: [[2407.18839]] Scalable Group Choreography via Variational Phase Manifold Learning(https://arxiv.org/abs/2407.18839)
• Keywords: generative
• Abstract: Generating group dance motion from the music is a challenging task with several industrial applications. Although several methods have been proposed to tackle this problem, most of them prioritize optimizing the fidelity in dancing movement, constrained by predetermined dancer counts in datasets. This limitation impedes adaptability to real-world applications. Our study addresses the scalability problem in group choreography while preserving naturalness and synchronization. In particular, we propose a phase-based variational generative model for group dance generation on learning a generative manifold. Our method achieves high-fidelity group dance motion and enables the generation with an unlimited number of dancers while consuming only a minimal and constant amount of memory. The intensive experiments on two public datasets show that our proposed method outperforms recent state-of-the-art approaches by a large margin and is scalable to a great number of dancers beyond the training data.

Title: The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning

• Authors: Andrew Patterson, Samuel Neumann, Raksha Kumaraswamy, Martha White, Adam White
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18840
• Pdf URL: https://arxiv.org/pdf/2407.18840
• Copy Paste: [[2407.18840]] The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning(https://arxiv.org/abs/2407.18840)
• Keywords: robust
• Abstract: This paper introduces a new empirical methodology, the Cross-environment Hyperparameter Setting Benchmark, that compares RL algorithms across environments using a single hyperparameter setting, encouraging algorithmic development which is insensitive to hyperparameters. We demonstrate that this benchmark is robust to statistical noise and obtains qualitatively similar results across repeated applications, even when using few samples. This robustness makes the benchmark computationally cheap to apply, allowing statistically sound insights at low cost. We demonstrate two example instantiations of the CHS, on a set of six small control environments (SC-CHS) and on the entire DM Control suite of 28 environments (DMC-CHS). Finally, to illustrate the applicability of the CHS to modern RL algorithms on challenging environments, we conduct a novel empirical study of an open question in the continuous control literature. We show, with high confidence, that there is no meaningful difference in performance between Ornstein-Uhlenbeck noise and uncorrelated Gaussian noise for exploration with the DDPG algorithm on the DMC-CHS.

Title: QT-TDM: Planning with Transformer Dynamics Model and Autoregressive Q-Learning

• Authors: Mostafa Kotb, Cornelius Weber, Muhammad Burhan Hafez, Stefan Wermter
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18841
• Pdf URL: https://arxiv.org/pdf/2407.18841
• Copy Paste: [[2407.18841]] QT-TDM: Planning with Transformer Dynamics Model and Autoregressive Q-Learning(https://arxiv.org/abs/2407.18841)
• Keywords: robust, transformer
• Abstract: Inspired by the success of the Transformer architecture in natural language processing and computer vision, we investigate the use of Transformers in Reinforcement Learning (RL), specifically in modeling the environment's dynamics using Transformer Dynamics Models (TDMs). We evaluate the capabilities of TDMs for continuous control in real-time planning scenarios with Model Predictive Control (MPC). While Transformers excel in long-horizon prediction, their tokenization mechanism and autoregressive nature lead to costly planning over long horizons, especially as the environment's dimensionality increases. To alleviate this issue, we use a TDM for short-term planning, and learn an autoregressive discrete Q-function using a separate Q-Transformer (QT) model to estimate a long-term return beyond the short-horizon planning. Our proposed method, QT-TDM, integrates the robust predictive capabilities of Transformers as dynamics models with the efficacy of a model-free Q-Transformer to mitigate the computational burden associated with real-time planning. Experiments in diverse state-based continuous control tasks show that QT-TDM is superior in performance and sample efficiency compared to existing Transformer-based RL models while achieving fast and computationally efficient inference.

Title: Enhancing material property prediction with ensemble deep graph convolutional networks

• Authors: Chowdhury Mohammad Abid Rahman, Ghadendra Bhandari, Nasser M Nasrabadi, Aldo H. Romero, Prashnna K. Gyawali
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18847
• Pdf URL: https://arxiv.org/pdf/2407.18847
• Copy Paste: [[2407.18847]] Enhancing material property prediction with ensemble deep graph convolutional networks(https://arxiv.org/abs/2407.18847)
• Keywords: robust
• Abstract: Machine learning (ML) models have emerged as powerful tools for accelerating materials discovery and design by enabling accurate predictions of properties from compositional and structural data. These capabilities are vital for developing advanced technologies across fields such as energy, electronics, and biomedicine, potentially reducing the time and resources needed for new material exploration and promoting rapid innovation cycles. Recent efforts have focused on employing advanced ML algorithms, including deep learning - based graph neural network, for property prediction. Additionally, ensemble models have proven to enhance the generalizability and robustness of ML and DL. However, the use of such ensemble strategies in deep graph networks for material property prediction remains underexplored. Our research provides an in-depth evaluation of ensemble strategies in deep learning - based graph neural network, specifically targeting material property prediction tasks. By testing the Crystal Graph Convolutional Neural Network (CGCNN) and its multitask version, MT-CGCNN, we demonstrated that ensemble techniques, especially prediction averaging, substantially improve precision beyond traditional metrics for key properties like formation energy per atom ($\Delta E^{f}$), band gap ($E_{g}$) and density ($\rho$) in 33,990 stable inorganic materials. These findings support the broader application of ensemble methods to enhance predictive accuracy in the field.

Title: Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment

• Authors: Yuze Zheng, Zixuan Li, Xiangxian Li, Jinxing Liu, Yuqing Wang, Xiangxu Meng, Lei Meng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18854
• Pdf URL: https://arxiv.org/pdf/2407.18854
• Copy Paste: [[2407.18854]] Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment(https://arxiv.org/abs/2407.18854)
• Keywords: extraction, diffusion
• Abstract: Image classification models often demonstrate unstable performance in real-world applications due to variations in image information, driven by differing visual perspectives of subject objects and lighting discrepancies. To mitigate these challenges, existing studies commonly incorporate additional modal information matching the visual data to regularize the model's learning process, enabling the extraction of high-quality visual features from complex image regions. Specifically, in the realm of multimodal learning, cross-modal alignment is recognized as an effective strategy, harmonizing different modal information by learning a domain-consistent latent feature space for visual and semantic features. However, this approach may face limitations due to the heterogeneity between multimodal information, such as differences in feature distribution and structure. To address this issue, we introduce a Multimodal Alignment and Reconstruction Network (MARNet), designed to enhance the model's resistance to visual noise. Importantly, MARNet includes a cross-modal diffusion reconstruction module for smoothly and stably blending information across different domains. Experiments conducted on two benchmark datasets, Vireo-Food172 and Ingredient-101, demonstrate that MARNet effectively improves the quality of image information extracted by the model. It is a plug-and-play framework that can be rapidly integrated into various image classification frameworks, boosting model performance.

Title: HADES: Detecting Active Directory Attacks via Whole Network Provenance Analytics

• Authors: Qi Liu, Kaibin Bao, Wajih Ul Hassan, Veit Hagenmeyer
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.18858
• Pdf URL: https://arxiv.org/pdf/2407.18858
• Copy Paste: [[2407.18858]] HADES: Detecting Active Directory Attacks via Whole Network Provenance Analytics(https://arxiv.org/abs/2407.18858)
• Keywords: attack, steal
• Abstract: Due to its crucial role in identity and access management in modern enterprise networks, Active Directory (AD) is a top target of Advanced Persistence Threat (APT) actors. Conventional intrusion detection systems (IDS) excel at identifying malicious behaviors caused by malware, but often fail to detect stealthy attacks launched by APT actors. Recent advance in provenance-based IDS (PIDS) shows promises by exposing malicious system activities in causal attack graphs. However, existing approaches are restricted to intra-machine tracing, and unable to reveal the scope of attackers' traversal inside a network. We propose HADES, the first PIDS capable of performing accurate causality-based cross-machine tracing by leveraging a novel concept called logon session based execution partitioning to overcome several challenges in cross-machine tracing. We design HADES as an efficient on-demand tracing system, which performs whole-network tracing only when it first identifies an authentication anomaly signifying an ongoing AD attack, for which we introduce a novel lightweight authentication anomaly detection model rooted in our extensive analysis of AD attacks. To triage attack alerts, we present a new algorithm integrating two key insights we identified in AD attacks. Our evaluations show that HADES outperforms both popular open source detection systems and a prominent commercial AD attack detector.

Title: Generative Adversarial Networks for Imputing Sparse Learning Performance

• Authors: Liang Zhang, Mohammed Yeasin, Jionghao Lin, Felix Havugimana, Xiangen Hu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18875
• Pdf URL: https://arxiv.org/pdf/2407.18875
• Copy Paste: [[2407.18875]] Generative Adversarial Networks for Imputing Sparse Learning Performance(https://arxiv.org/abs/2407.18875)
• Keywords: generative
• Abstract: Learning performance data, such as correct or incorrect responses to questions in Intelligent Tutoring Systems (ITSs) is crucial for tracking and assessing the learners' progress and mastery of knowledge. However, the issue of data sparsity, characterized by unexplored questions and missing attempts, hampers accurate assessment and the provision of tailored, personalized instruction within ITSs. This paper proposes using the Generative Adversarial Imputation Networks (GAIN) framework to impute sparse learning performance data, reconstructed into a three-dimensional (3D) tensor representation across the dimensions of learners, questions and attempts. Our customized GAIN-based method computational process imputes sparse data in a 3D tensor space, significantly enhanced by convolutional neural networks for its input and output layers. This adaptation also includes the use of a least squares loss function for optimization and aligns the shapes of the input and output with the dimensions of the questions-attempts matrices along the learners' dimension. Through extensive experiments on six datasets from various ITSs, including AutoTutor, ASSISTments and MATHia, we demonstrate that the GAIN approach generally outperforms existing methods such as tensor factorization and other generative adversarial network (GAN) based approaches in terms of imputation accuracy. This finding enhances comprehensive learning data modeling and analytics in AI-based education.

Title: Small Molecule Optimization with Large Language Models

• Authors: Philipp Guevorguian, Menua Bedrosian, Tigran Fahradyan, Gayane Chilingaryan, Hrant Khachatrian, Armen Aghajanyan
• Subjects: cs.LG, cs.NE, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2407.18897
• Pdf URL: https://arxiv.org/pdf/2407.18897
• Copy Paste: [[2407.18897]] Small Molecule Optimization with Large Language Models(https://arxiv.org/abs/2407.18897)
• Keywords: generative, large language model
• Abstract: Recent advancements in large language models have opened new possibilities for generative molecular drug design. We present Chemlactica and Chemma, two language models fine-tuned on a novel corpus of 110M molecules with computed properties, totaling 40B tokens. These models demonstrate strong performance in generating molecules with specified properties and predicting new molecular characteristics from limited samples. We introduce a novel optimization algorithm that leverages our language models to optimize molecules for arbitrary properties given limited access to a black box oracle. Our approach combines ideas from genetic algorithms, rejection sampling, and prompt optimization. It achieves state-of-the-art performance on multiple molecular optimization benchmarks, including an 8% improvement on Practical Molecular Optimization compared to previous methods. We publicly release the training corpus, the language models and the optimization algorithm.

Title: Learn from the Learnt: Source-Free Active Domain Adaptation via Contrastive Sampling and Visual Persistence

• Authors: Mengyao Lyu, Tianxiang Hao, Xinhao Xu, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.18899
• Pdf URL: https://arxiv.org/pdf/2407.18899
• Copy Paste: [[2407.18899]] Learn from the Learnt: Source-Free Active Domain Adaptation via Contrastive Sampling and Visual Persistence(https://arxiv.org/abs/2407.18899)
• Keywords: data-free
• Abstract: Domain Adaptation (DA) facilitates knowledge transfer from a source domain to a related target domain. This paper investigates a practical DA paradigm, namely Source data-Free Active Domain Adaptation (SFADA), where source data becomes inaccessible during adaptation, and a minimum amount of annotation budget is available in the target domain. Without referencing the source data, new challenges emerge in identifying the most informative target samples for labeling, establishing cross-domain alignment during adaptation, and ensuring continuous performance improvements through the iterative query-and-adaptation process. In response, we present learn from the learnt (LFTL), a novel paradigm for SFADA to leverage the learnt knowledge from the source pretrained model and actively iterated models without extra overhead. We propose Contrastive Active Sampling to learn from the hypotheses of the preceding model, thereby querying target samples that are both informative to the current model and persistently challenging throughout active learning. During adaptation, we learn from features of actively selected anchors obtained from previous intermediate models, so that the Visual Persistence-guided Adaptation can facilitate feature distribution alignment and active sample exploitation. Extensive experiments on three widely-used benchmarks show that our LFTL achieves state-of-the-art performance, superior computational efficiency and continuous improvements as the annotation budget increases. Our code is available at this https URL.

Title: SHIC: Shape-Image Correspondences with no Keypoint Supervision

• Authors: Aleksandar Shtedritski, Christian Rupprecht, Andrea Vedaldi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.18907
• Pdf URL: https://arxiv.org/pdf/2407.18907
• Copy Paste: [[2407.18907]] SHIC: Shape-Image Correspondences with no Keypoint Supervision(https://arxiv.org/abs/2407.18907)
• Keywords: diffusion
• Abstract: Canonical surface mapping generalizes keypoint detection by assigning each pixel of an object to a corresponding point in a 3D template. Popularised by DensePose for the analysis of humans, authors have since attempted to apply the concept to more categories, but with limited success due to the high cost of manual supervision. In this work, we introduce SHIC, a method to learn canonical maps without manual supervision which achieves better results than supervised methods for most categories. Our idea is to leverage foundation computer vision models such as DINO and Stable Diffusion that are open-ended and thus possess excellent priors over natural categories. SHIC reduces the problem of estimating image-to-template correspondences to predicting image-to-image correspondences using features from the foundation models. The reduction works by matching images of the object to non-photorealistic renders of the template, which emulates the process of collecting manual annotations for this task. These correspondences are then used to supervise high-quality canonical maps for any object of interest. We also show that image generators can further improve the realism of the template views, which provide an additional source of supervision for the model.

Title: SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments

• Authors: Shu Ishida, João F. Henriques
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.18913
• Pdf URL: https://arxiv.org/pdf/2407.18913
• Copy Paste: [[2407.18913]] SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments(https://arxiv.org/abs/2407.18913)
• Keywords: robust
• Abstract: This work compares ways of extending Reinforcement Learning algorithms to Partially Observed Markov Decision Processes (POMDPs) with options. One view of options is as temporally extended action, which can be realized as a memory that allows the agent to retain historical information beyond the policy's context window. While option assignment could be handled using heuristics and hand-crafted objectives, learning temporally consistent options and associated sub-policies without explicit supervision is a challenge. Two algorithms, PPOEM and SOAP, are proposed and studied in depth to address this problem. PPOEM applies the forward-backward algorithm (for Hidden Markov Models) to optimize the expected returns for an option-augmented policy. However, this learning approach is unstable during on-policy rollouts. It is also unsuited for learning causal policies without the knowledge of future trajectories, since option assignments are optimized for offline sequences where the entire episode is available. As an alternative approach, SOAP evaluates the policy gradient for an optimal option assignment. It extends the concept of the generalized advantage estimation (GAE) to propagate option advantages through time, which is an analytical equivalent to performing temporal back-propagation of option policy gradients. This option policy is only conditional on the history of the agent, not future actions. Evaluated against competing baselines, SOAP exhibited the most robust performance, correctly discovering options for POMDP corridor environments, as well as on standard benchmarks including Atari and MuJoCo, outperforming PPOEM, as well as LSTM and Option-Critic baselines. The open-sourced code is available at this https URL.