2024-08-07

Title: On Feasibility of Intent Obfuscating Attacks

• Authors: Zhaobin Li, Patrick Shafto
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02674
• Pdf URL: https://arxiv.org/pdf/2408.02674
• Copy Paste: [[2408.02674]] On Feasibility of Intent Obfuscating Attacks(https://arxiv.org/abs/2408.02674)
• Keywords: defense, attack
• Abstract: Intent obfuscation is a common tactic in adversarial situations, enabling the attacker to both manipulate the target system and avoid culpability. Surprisingly, it has rarely been implemented in adversarial attacks on machine learning systems. We are the first to propose incorporating intent obfuscation in generating adversarial examples for object detectors: by perturbing another non-overlapping object to disrupt the target object, the attacker hides their intended target. We conduct a randomized experiment on 5 prominent detectors -- YOLOv3, SSD, RetinaNet, Faster R-CNN, and Cascade R-CNN -- using both targeted and untargeted attacks and achieve success on all models and attacks. We analyze the success factors characterizing intent obfuscating attacks, including target object confidence and perturb object sizes. We then demonstrate that the attacker can exploit these success factors to increase success rates for all models and attacks. Finally, we discuss known defenses and legal repercussions.

Title: On Biases in a UK Biobank-based Retinal Image Classification Model

• Authors: Anissa Alloula, Rima Mustafa, Daniel R McGowan, Bartłomiej W. Papież
• Subjects: cs.LG, cs.AI, cs.CV, cs.CY, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.02676
• Pdf URL: https://arxiv.org/pdf/2408.02676
• Copy Paste: [[2408.02676]] On Biases in a UK Biobank-based Retinal Image Classification Model(https://arxiv.org/abs/2408.02676)
• Keywords: fair
• Abstract: Recent work has uncovered alarming disparities in the performance of machine learning models in healthcare. In this study, we explore whether such disparities are present in the UK Biobank fundus retinal images by training and evaluating a disease classification model on these images. We assess possible disparities across various population groups and find substantial differences despite strong overall performance of the model. In particular, we discover unfair performance for certain assessment centres, which is surprising given the rigorous data standardisation protocol. We compare how these differences emerge and apply a range of existing bias mitigation methods to each one. A key insight is that each disparity has unique properties and responds differently to the mitigation methods. We also find that these methods are largely unable to enhance fairness, highlighting the need for better bias mitigation methods tailored to the specific type of bias.

Title: Patient-centered data science: an integrative framework for evaluating and predicting clinical outcomes in the digital health era

• Authors: Mohsen Amoei, Dan Poenaru
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.02677
• Pdf URL: https://arxiv.org/pdf/2408.02677
• Copy Paste: [[2408.02677]] Patient-centered data science: an integrative framework for evaluating and predicting clinical outcomes in the digital health era(https://arxiv.org/abs/2408.02677)
• Keywords: large language model
• Abstract: This study proposes a novel, integrative framework for patient-centered data science in the digital health era. We developed a multidimensional model that combines traditional clinical data with patient-reported outcomes, social determinants of health, and multi-omic data to create comprehensive digital patient representations. Our framework employs a multi-agent artificial intelligence approach, utilizing various machine learning techniques including large language models, to analyze complex, longitudinal datasets. The model aims to optimize multiple patient outcomes simultaneously while addressing biases and ensuring generalizability. We demonstrate how this framework can be implemented to create a learning healthcare system that continuously refines strategies for optimal patient care. This approach has the potential to significantly improve the translation of digital health innovations into real-world clinical benefits, addressing current limitations in AI-driven healthcare models.

Title: Improving Machine Learning Based Sepsis Diagnosis Using Heart Rate Variability

• Authors: Sai Balaji, Christopher Sun, Anaiy Somalwar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02683
• Pdf URL: https://arxiv.org/pdf/2408.02683
• Copy Paste: [[2408.02683]] Improving Machine Learning Based Sepsis Diagnosis Using Heart Rate Variability(https://arxiv.org/abs/2408.02683)
• Keywords: interpretability
• Abstract: The early and accurate diagnosis of sepsis is critical for enhancing patient outcomes. This study aims to use heart rate variability (HRV) features to develop an effective predictive model for sepsis detection. Critical HRV features are identified through feature engineering methods, including statistical bootstrapping and the Boruta algorithm, after which XGBoost and Random Forest classifiers are trained with differential hyperparameter settings. In addition, ensemble models are constructed to pool the prediction probabilities of high-recall and high-precision classifiers and improve model performance. Finally, a neural network model is trained on the HRV features, achieving an F1 score of 0.805, a precision of 0.851, and a recall of 0.763. The best-performing machine learning model is compared to this neural network through an interpretability analysis, where Local Interpretable Model-agnostic Explanations are implemented to determine decision-making criterion based on numerical ranges and thresholds for specific features. This study not only highlights the efficacy of HRV in automated sepsis diagnosis but also increases the transparency of black box outputs, maximizing clinical applicability.

Title: A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications

• Authors: Valerio Guarrasi, Fatih Aksu, Camillo Maria Caruso, Francesco Di Feola, Aurora Rofena, Filippo Ruffini, Paolo Soda
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02686
• Pdf URL: https://arxiv.org/pdf/2408.02686
• Copy Paste: [[2408.02686]] A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications(https://arxiv.org/abs/2408.02686)
• Keywords: robust
• Abstract: Deep learning has revolutionized biomedical research by providing sophisticated methods to handle complex, high-dimensional data. Multimodal deep learning (MDL) further enhances this capability by integrating diverse data types such as imaging, textual data, and genetic information, leading to more robust and accurate predictive models. In MDL, differently from early and late fusion methods, intermediate fusion stands out for its ability to effectively combine modality-specific features during the learning process. This systematic review aims to comprehensively analyze and formalize current intermediate fusion methods in biomedical applications. We investigate the techniques employed, the challenges faced, and potential future directions for advancing intermediate fusion methods. Additionally, we introduce a structured notation to enhance the understanding and application of these methods beyond the biomedical domain. Our findings are intended to support researchers, healthcare professionals, and the broader deep learning community in developing more sophisticated and insightful multimodal models. Through this review, we aim to provide a foundational framework for future research and practical applications in the dynamic field of MDL.

Title: KAN based Autoencoders for Factor Models

• Authors: Tianqi Wang, Shubham Singh
• Subjects: cs.LG, cs.AI, q-fin.CP
• Abstract URL: https://arxiv.org/abs/2408.02694
• Pdf URL: https://arxiv.org/pdf/2408.02694
• Copy Paste: [[2408.02694]] KAN based Autoencoders for Factor Models(https://arxiv.org/abs/2408.02694)
• Keywords: interpretability
• Abstract: Inspired by recent advances in Kolmogorov-Arnold Networks (KANs), we introduce a novel approach to latent factor conditional asset pricing models. While previous machine learning applications in asset pricing have predominantly used Multilayer Perceptrons with ReLU activation functions to model latent factor exposures, our method introduces a KAN-based autoencoder which surpasses MLP models in both accuracy and interpretability. Our model offers enhanced flexibility in approximating exposures as nonlinear functions of asset characteristics, while simultaneously providing users with an intuitive framework for interpreting latent factors. Empirical backtesting demonstrates our model's superior ability to explain cross-sectional risk exposures. Moreover, long-short portfolios constructed using our model's predictions achieve higher Sharpe ratios, highlighting its practical value in investment management.

Title: Bayesian Kolmogorov Arnold Networks (Bayesian_KANs): A Probabilistic Approach to Enhance Accuracy and Interpretability

• Authors: Masoud Muhammed Hassan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02706
• Pdf URL: https://arxiv.org/pdf/2408.02706
• Copy Paste: [[2408.02706]] Bayesian Kolmogorov Arnold Networks (Bayesian_KANs): A Probabilistic Approach to Enhance Accuracy and Interpretability(https://arxiv.org/abs/2408.02706)
• Keywords: interpretability
• Abstract: Because of its strong predictive skills, deep learning has emerged as an essential tool in many industries, including healthcare. Traditional deep learning models, on the other hand, frequently lack interpretability and omit to take prediction uncertainty into account two crucial components of clinical decision making. In order to produce explainable and uncertainty aware predictions, this study presents a novel framework called Bayesian Kolmogorov Arnold Networks (BKANs), which combines the expressive capacity of Kolmogorov Arnold Networks with Bayesian inference. We employ BKANs on two medical datasets, which are widely used benchmarks for assessing machine learning models in medical diagnostics: the Pima Indians Diabetes dataset and the Cleveland Heart Disease dataset. Our method provides useful insights into prediction confidence and decision boundaries and outperforms traditional deep learning models in terms of prediction accuracy. Moreover, BKANs' capacity to represent aleatoric and epistemic uncertainty guarantees doctors receive more solid and trustworthy decision support. Our Bayesian strategy improves the interpretability of the model and considerably minimises overfitting, which is important for tiny and imbalanced medical datasets, according to experimental results. We present possible expansions to further use BKANs in more complicated multimodal datasets and address the significance of these discoveries for future research in building reliable AI systems for healthcare. This work paves the way for a new paradigm in deep learning model deployment in vital sectors where transparency and reliability are crucial.

Title: SnapE -- Training Snapshot Ensembles of Link Prediction Models

• Authors: Ali Shaban, Heiko Paulheim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02707
• Pdf URL: https://arxiv.org/pdf/2408.02707
• Copy Paste: [[2408.02707]] SnapE -- Training Snapshot Ensembles of Link Prediction Models(https://arxiv.org/abs/2408.02707)
• Keywords: robust
• Abstract: Snapshot ensembles have been widely used in various fields of prediction. They allow for training an ensemble of prediction models at the cost of training a single one. They are known to yield more robust predictions by creating a set of diverse base models. In this paper, we introduce an approach to transfer the idea of snapshot ensembles to link prediction models in knowledge graphs. Moreover, since link prediction in knowledge graphs is a setup without explicit negative examples, we propose a novel training loop that iteratively creates negative examples using previous snapshot models. An evaluation with four base models across four datasets shows that this approach constantly outperforms the single model approach, while keeping the training time constant.

Title: RCDM: Enabling Robustness for Conditional Diffusion Model

• Authors: Weifeng Xu, Xiang Zhu, Xiaoyong Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02710
• Pdf URL: https://arxiv.org/pdf/2408.02710
• Copy Paste: [[2408.02710]] RCDM: Enabling Robustness for Conditional Diffusion Model(https://arxiv.org/abs/2408.02710)
• Keywords: robust, diffusion
• Abstract: The conditional diffusion model (CDM) enhances the standard diffusion model by providing more control, improving the quality and relevance of the outputs, and making the model adaptable to a wider range of complex tasks. However, inaccurate conditional inputs in the inverse process of CDM can easily lead to generating fixed errors in the neural network, which diminishes the adaptability of a well-trained model. The existing methods like data augmentation, adversarial training, robust optimization can improve the robustness, while they often face challenges such as high computational complexity, limited applicability to unknown perturbations, and increased training difficulty. In this paper, we propose a lightweight solution, the Robust Conditional Diffusion Model (RCDM), based on control theory to dynamically reduce the impact of noise and significantly enhance the model's robustness. RCDM leverages the collaborative interaction between two neural networks, along with optimal control strategies derived from control theory, to optimize the weights of two networks during the sampling process. Unlike conventional techniques, RCDM establishes a mathematical relationship between fixed errors and the weights of the two neural networks without incurring additional computational overhead. Extensive experiments were conducted on MNIST and CIFAR-10 datasets, and the results demonstrate the effectiveness and adaptability of our proposed model.

Title: Privacy-Safe Iris Presentation Attack Detection

• Authors: Mahsa Mitcheff, Patrick Tinsley, Adam Czajka
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.02750
• Pdf URL: https://arxiv.org/pdf/2408.02750
• Copy Paste: [[2408.02750]] Privacy-Safe Iris Presentation Attack Detection(https://arxiv.org/abs/2408.02750)
• Keywords: privacy, attack, generative
• Abstract: This paper proposes a framework for a privacy-safe iris presentation attack detection (PAD) method, designed solely with synthetically-generated, identity-leakage-free iris images. Once trained, the method is evaluated in a classical way using state-of-the-art iris PAD benchmarks. We designed two generative models for the synthesis of ISO/IEC 19794-6-compliant iris images. The first model synthesizes bona fide-looking samples. To avoid ``identity leakage,'' the generated samples that accidentally matched those used in the model's training were excluded. The second model synthesizes images of irises with textured contact lenses and is conditioned by a given contact lens brand to have better control over textured contact lens appearance when forming the training set. Our experiments demonstrate that models trained solely on synthetic data achieve a lower but still reasonable performance when compared to solutions trained with iris images collected from human subjects. This is the first-of-its-kind attempt to use solely synthetic data to train a fully-functional iris PAD solution, and despite the performance gap between regular and the proposed methods, this study demonstrates that with the increasing fidelity of generative models, creating such privacy-safe iris PAD methods may be possible. The source codes and generative models trained for this work are offered along with the paper.

Title: A Novel Hybrid Approach for Tornado Prediction in the United States: Kalman-Convolutional BiLSTM with Multi-Head Attention

• Authors: Jiawei Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02751
• Pdf URL: https://arxiv.org/pdf/2408.02751
• Copy Paste: [[2408.02751]] A Novel Hybrid Approach for Tornado Prediction in the United States: Kalman-Convolutional BiLSTM with Multi-Head Attention(https://arxiv.org/abs/2408.02751)
• Keywords: robust, large language model
• Abstract: Tornadoes are among the most intense atmospheric vortex phenomena and pose significant challenges for detection and forecasting. Conventional methods, which heavily depend on ground-based observations and radar data, are limited by issues such as decreased accuracy over greater distances and a high rate of false positives. To address these challenges, this study utilizes the Seamless Hybrid Scan Reflectivity (SHSR) dataset from the Multi-Radar Multi-Sensor (MRMS) system, which integrates data from multiple radar sources to enhance accuracy. A novel hybrid model, the Kalman-Convolutional BiLSTM with Multi-Head Attention, is introduced to improve dynamic state estimation and capture both spatial and temporal dependencies within the data. This model demonstrates superior performance in precision, recall, F1-Score, and accuracy compared to methods such as K-Nearest Neighbors (KNN) and LightGBM. The results highlight the considerable potential of advanced machine learning techniques to improve tornado prediction and reduce false alarm rates. Future research will focus on expanding datasets, exploring innovative model architectures, and incorporating large language models (LLMs) to provide deeper insights. This research introduces a novel model for tornado prediction, offering a robust framework for enhancing forecasting accuracy and public safety.

Title: Diffusion Models as Data Mining Tools

• Authors: Ioannis Siglidis, Aleksander Holynski, Alexei A. Efros, Mathieu Aubry, Shiry Ginosar
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02752
• Pdf URL: https://arxiv.org/pdf/2408.02752
• Copy Paste: [[2408.02752]] Diffusion Models as Data Mining Tools(https://arxiv.org/abs/2408.02752)
• Keywords: diffusion, generative
• Abstract: This paper demonstrates how to use generative models trained for image synthesis as tools for visual data mining. Our insight is that since contemporary generative models learn an accurate representation of their training data, we can use them to summarize the data by mining for visual patterns. Concretely, we show that after finetuning conditional diffusion models to synthesize images from a specific dataset, we can use these models to define a typicality measure on that dataset. This measure assesses how typical visual elements are for different data labels, such as geographic location, time stamps, semantic labels, or even the presence of a disease. This analysis-by-synthesis approach to data mining has two key advantages. First, it scales much better than traditional correspondence-based approaches since it does not require explicitly comparing all pairs of visual elements. Second, while most previous works on visual data mining focus on a single dataset, our approach works on diverse datasets in terms of content and scale, including a historical car dataset, a historical face dataset, a large worldwide street-view dataset, and an even larger scene dataset. Furthermore, our approach allows for translating visual elements across class labels and analyzing consistent changes.

Title: Dimensionality Reduction and Nearest Neighbors for Improving Out-of-Distribution Detection in Medical Image Segmentation

• Authors: McKell Woodland, Nihil Patel, Austin Castelo, Mais Al Taie, Mohamed Eltaher, Joshua P. Yung, Tucker J. Netherton, Tiffany L. Calderone, Jessica I. Sanchez, Darrel W. Cleere, Ahmed Elsaiey, Nakul Gupta, David Victor, Laura Beretta, Ankit B. Patel Kristy K. Brock
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02761
• Pdf URL: https://arxiv.org/pdf/2408.02761
• Copy Paste: [[2408.02761]] Dimensionality Reduction and Nearest Neighbors for Improving Out-of-Distribution Detection in Medical Image Segmentation(https://arxiv.org/abs/2408.02761)
• Keywords: segmentation
• Abstract: Clinically deployed deep learning-based segmentation models are known to fail on data outside of their training distributions. While clinicians review the segmentations, these models tend to perform well in most instances, which could exacerbate automation bias. Therefore, detecting out-of-distribution images at inference is critical to warn the clinicians that the model likely failed. This work applied the Mahalanobis distance (MD) post hoc to the bottleneck features of four Swin UNETR and nnU-net models that segmented the liver on T1-weighted magnetic resonance imaging and computed tomography. By reducing the dimensions of the bottleneck features with either principal component analysis or uniform manifold approximation and projection, images the models failed on were detected with high performance and minimal computational load. In addition, this work explored a non-parametric alternative to the MD, a k-th nearest neighbors distance (KNN). KNN drastically improved scalability and performance over MD when both were applied to raw and average-pooled bottleneck features.

Title: ConDL: Detector-Free Dense Image Matching

• Authors: Monika Kwiatkowski, Simon Matern, Olaf Hellwich
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02766
• Pdf URL: https://arxiv.org/pdf/2408.02766
• Copy Paste: [[2408.02766]] ConDL: Detector-Free Dense Image Matching(https://arxiv.org/abs/2408.02766)
• Keywords: robust
• Abstract: In this work, we introduce a deep-learning framework designed for estimating dense image correspondences. Our fully convolutional model generates dense feature maps for images, where each pixel is associated with a descriptor that can be matched across multiple images. Unlike previous methods, our model is trained on synthetic data that includes significant distortions, such as perspective changes, illumination variations, shadows, and specular highlights. Utilizing contrastive learning, our feature maps achieve greater invariance to these distortions, enabling robust matching. Notably, our method eliminates the need for a keypoint detector, setting it apart from many existing image-matching techniques.

Title: Refined Infrared Small Target Detection Scheme with Single-Point Supervision

• Authors: Jinmiao Zhao, Zelin Shi, Chuang Yu, Yunpeng Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02773
• Pdf URL: https://arxiv.org/pdf/2408.02773
• Copy Paste: [[2408.02773]] Refined Infrared Small Target Detection Scheme with Single-Point Supervision(https://arxiv.org/abs/2408.02773)
• Keywords: segmentation
• Abstract: Recently, infrared small target detection with single-point supervision has attracted extensive attention. However, the detection accuracy of existing methods has difficulty meeting actual needs. Therefore, we propose an innovative refined infrared small target detection scheme with single-point supervision, which has excellent segmentation accuracy and detection rate. Specifically, we introduce label evolution with single point supervision (LESPS) framework and explore the performance of various excellent infrared small target detection networks based on this framework. Meanwhile, to improve the comprehensive performance, we construct a complete post-processing strategy. On the one hand, to improve the segmentation accuracy, we use a combination of test-time augmentation (TTA) and conditional random field (CRF) for post-processing. On the other hand, to improve the detection rate, we introduce an adjustable sensitivity (AS) strategy for post-processing, which fully considers the advantages of multiple detection results and reasonably adds some areas with low confidence to the fine segmentation image in the form of centroid points. In addition, to further improve the performance and explore the characteristics of this task, on the one hand, we construct and find that a multi-stage loss is helpful for fine-grained detection. On the other hand, we find that a reasonable sliding window cropping strategy for test samples has better performance for actual multi-size samples. Extensive experimental results show that the proposed scheme achieves state-of-the-art (SOTA) performance. Notably, the proposed scheme won the third place in the "ICPR 2024 Resource-Limited Infrared Small Target Detection Challenge Track 1: Weakly Supervised Infrared Small Target Detection".

Title: LR-Net: A Lightweight and Robust Network for Infrared Small Target Detection

• Authors: Chuang Yu, Yunpeng Liu, Jinmiao Zhao, Zelin Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02780
• Pdf URL: https://arxiv.org/pdf/2408.02780
• Copy Paste: [[2408.02780]] LR-Net: A Lightweight and Robust Network for Infrared Small Target Detection(https://arxiv.org/abs/2408.02780)
• Keywords: robust, extraction
• Abstract: Limited by equipment limitations and the lack of target intrinsic features, existing infrared small target detection methods have difficulty meeting actual comprehensive performance requirements. Therefore, we propose an innovative lightweight and robust network (LR-Net), which abandons the complex structure and achieves an effective balance between detection accuracy and resource consumption. Specifically, to ensure the lightweight and robustness, on the one hand, we construct a lightweight feature extraction attention (LFEA) module, which can fully extract target features and strengthen information interaction across channels. On the other hand, we construct a simple refined feature transfer (RFT) module. Compared with direct cross-layer connections, the RFT module can improve the network's feature refinement extraction capability with little resource consumption. Meanwhile, to solve the problem of small target loss in high-level feature maps, on the one hand, we propose a low-level feature distribution (LFD) strategy to use low-level features to supplement the information of high-level features. On the other hand, we introduce an efficient simplified bilinear interpolation attention module (SBAM) to promote the guidance constraints of low-level features on high-level features and the fusion of the two. In addition, We abandon the traditional resizing method and adopt a new training and inference cropping strategy, which is more robust to datasets with multi-scale samples. Extensive experimental results show that our LR-Net achieves state-of-the-art (SOTA) performance. Notably, on the basis of the proposed LR-Net, we achieve 3rd place in the "ICPR 2024 Resource-Limited Infrared Small Target Detection Challenge Track 2: Lightweight Infrared Small Target Detection".

Title: LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory

• Authors: Jillian Ross, Yoon Kim, Andrew W. Lo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02784
• Pdf URL: https://arxiv.org/pdf/2408.02784
• Copy Paste: [[2408.02784]] LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory(https://arxiv.org/abs/2408.02784)
• Keywords: large language model
• Abstract: Humans are not homo economicus (i.e., rational economic beings). As humans, we exhibit systematic behavioral biases such as loss aversion, anchoring, framing, etc., which lead us to make suboptimal economic decisions. Insofar as such biases may be embedded in text data on which large language models (LLMs) are trained, to what extent are LLMs prone to the same behavioral biases? Understanding these biases in LLMs is crucial for deploying LLMs to support human decision-making. We propose utility theory-a paradigm at the core of modern economic theory-as an approach to evaluate the economic biases of LLMs. Utility theory enables the quantification and comparison of economic behavior against benchmarks such as perfect rationality or human behavior. To demonstrate our approach, we quantify and compare the economic behavior of a variety of open- and closed-source LLMs. We find that the economic behavior of current LLMs is neither entirely human-like nor entirely economicus-like. We also find that most current LLMs struggle to maintain consistent economic behavior across settings. Finally, we illustrate how our approach can measure the effect of interventions such as prompting on economic biases.

Title: Segmentation Style Discovery: Application to Skin Lesion Images

• Authors: Kumar Abhishek, Jeremy Kawahara, Ghassan Hamarneh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02787
• Pdf URL: https://arxiv.org/pdf/2408.02787
• Copy Paste: [[2408.02787]] Segmentation Style Discovery: Application to Skin Lesion Images(https://arxiv.org/abs/2408.02787)
• Keywords: segmentation
• Abstract: Variability in medical image segmentation, arising from annotator preferences, expertise, and their choice of tools, has been well documented. While the majority of multi-annotator segmentation approaches focus on modeling annotator-specific preferences, they require annotator-segmentation correspondence. In this work, we introduce the problem of segmentation style discovery, and propose StyleSeg, a segmentation method that learns plausible, diverse, and semantically consistent segmentation styles from a corpus of image-mask pairs without any knowledge of annotator correspondence. StyleSeg consistently outperforms competing methods on four publicly available skin lesion segmentation (SLS) datasets. We also curate ISIC-MultiAnnot, the largest multi-annotator SLS dataset with annotator correspondence, and our results show a strong alignment, using our newly proposed measure AS2, between the predicted styles and annotator preferences. The code and the dataset are available at this https URL.

Title: Gaussian Mixture based Evidential Learning for Stereo Matching

• Authors: Weide Liu, Xingxing Wang, Lu Wang, Jun Cheng, Fayao Liu, Xulei Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02796
• Pdf URL: https://arxiv.org/pdf/2408.02796
• Copy Paste: [[2408.02796]] Gaussian Mixture based Evidential Learning for Stereo Matching(https://arxiv.org/abs/2408.02796)
• Keywords: robust
• Abstract: In this paper, we introduce a novel Gaussian mixture based evidential learning solution for robust stereo matching. Diverging from previous evidential deep learning approaches that rely on a single Gaussian distribution, our framework posits that individual image data adheres to a mixture-of-Gaussian distribution in stereo matching. This assumption yields more precise pixel-level predictions and more accurately mirrors the real-world image distribution. By further employing the inverse-Gamma distribution as an intermediary prior for each mixture component, our probabilistic model achieves improved depth estimation compared to its counterpart with the single Gaussian and effectively captures the model uncertainty, which enables a strong cross-domain generation ability. We evaluated our method for stereo matching by training the model using the Scene Flow dataset and testing it on KITTI 2015 and Middlebury 2014. The experiment results consistently show that our method brings improvements over the baseline methods in a trustworthy manner. Notably, our approach achieved new state-of-the-art results on both the in-domain validated data and the cross-domain datasets, demonstrating its effectiveness and robustness in stereo matching tasks.

Title: Examining Gender and Power on Wikipedia Through Face and Politeness

• Authors: Adil Soubki, Shyne Choi, Owen Rambow
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02798
• Pdf URL: https://arxiv.org/pdf/2408.02798
• Copy Paste: [[2408.02798]] Examining Gender and Power on Wikipedia Through Face and Politeness(https://arxiv.org/abs/2408.02798)
• Keywords: robust
• Abstract: We propose a framework for analyzing discourse by combining two interdependent concepts from sociolinguistic theory: face acts and politeness. While politeness has robust existing tools and data, face acts are less resourced. We introduce a new corpus created by annotating Wikipedia talk pages with face acts and we use this to train a face act tagger. We then employ our framework to study how face and politeness interact with gender and power in discussions between Wikipedia editors. Among other findings, we observe that female Wikipedians are not only more polite, which is consistent with prior studies, but that this difference corresponds with significantly more language directed at humbling aspects of their own face. Interestingly, the distinction nearly vanishes once limiting to editors with administrative power.

Title: Deciphering Air Travel Disruptions: A Machine Learning Approach

• Authors: Aravinda Jatavallabha, Jacob Gerlach, Aadithya Naresh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02802
• Pdf URL: https://arxiv.org/pdf/2408.02802
• Copy Paste: [[2408.02802]] Deciphering Air Travel Disruptions: A Machine Learning Approach(https://arxiv.org/abs/2408.02802)
• Keywords: security
• Abstract: This research investigates flight delay trends by examining factors such as departure time, airline, and airport. It employs regression machine learning methods to predict the contributions of various sources to delays. Time-series models, including LSTM, Hybrid LSTM, and Bi-LSTM, are compared with baseline regression models such as Multiple Regression, Decision Tree Regression, Random Forest Regression, and Neural Network. Despite considerable errors in the baseline models, the study aims to identify influential features in delay prediction, potentially informing flight planning strategies. Unlike previous work, this research focuses on regression tasks and explores the use of time-series models for predicting flight delays. It offers insights into aviation operations by independently analyzing each delay component (e.g., security, weather).

Title: Mitigating Malicious Attacks in Federated Learning via Confidence-aware Defense

• Authors: Qilei Li, Ahmed M. Abdelmoniem
• Subjects: cs.LG, cs.CR, cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.02813
• Pdf URL: https://arxiv.org/pdf/2408.02813
• Copy Paste: [[2408.02813]] Mitigating Malicious Attacks in Federated Learning via Confidence-aware Defense(https://arxiv.org/abs/2408.02813)
• Keywords: defense, attack, robust, federate
• Abstract: Federated Learning (FL) is an emerging distributed machine learning paradigm that allows multiple clients to collaboratively train a global model without sharing private local data. However, FL systems are vulnerable to attacks from malicious clients, who can degrade the global model performance through data poisoning and model poisoning. Existing defense methods typically focus on a single type of attack, such as Byzantine attacks or backdoor attacks, and are often ineffective against potential data poisoning attacks like label flipping and label shuffling. Additionally, these methods often lack accuracy and robustness in detecting and handling malicious updates. To address these issues, we propose a novel method based on model confidence scores, which evaluates the uncertainty of client model updates to detect and defend against malicious clients. Our approach is comprehensively effective for both model poisoning and data poisoning attacks and is capable of accurately identifying and mitigating potential malicious updates from being aggregated. Experimental results demonstrate that our method significantly improves the robustness of FL systems against various types of attacks, also achieving higher model accuracy and stability across various scenarios.

Title: Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Services

• Authors: Shaopeng Fu, Xuexue Sun, Ke Qing, Tianhang Zheng, Di Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.02814
• Pdf URL: https://arxiv.org/pdf/2408.02814
• Copy Paste: [[2408.02814]] Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Services(https://arxiv.org/abs/2408.02814)
• Keywords: security, privacy, attack
• Abstract: Though pre-trained encoders can be easily accessed online to build downstream machine learning (ML) services quickly, various attacks have been designed to compromise the security and privacy of these encoders. While most attacks target encoders on the upstream side, it remains unknown how an encoder could be threatened when deployed in a downstream ML service. This paper unveils a new vulnerability: the Pre-trained Encoder Inference (PEI) attack, which posts privacy threats toward encoders hidden behind downstream ML services. By only providing API accesses to a targeted downstream service and a set of candidate encoders, the PEI attack can infer which encoder is secretly used by the targeted service based on candidate ones. We evaluate the attack performance of PEI against real-world encoders on three downstream tasks: image classification, text classification, and text-to-image generation. Experiments show that the PEI attack succeeds in revealing the hidden encoder in most cases and seldom makes mistakes even when the hidden encoder is not in the candidate set. We also conducted a case study on one of the most recent vision-language models, LLaVA, to illustrate that the PEI attack is useful in assisting other ML attacks such as adversarial attacks. The code is available at this https URL.

Title: Wave-RVFL: A Randomized Neural Network Based on Wave Loss Function

• Authors: M. Sajid, A. Quadir, M. Tanveer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02824
• Pdf URL: https://arxiv.org/pdf/2408.02824
• Copy Paste: [[2408.02824]] Wave-RVFL: A Randomized Neural Network Based on Wave Loss Function(https://arxiv.org/abs/2408.02824)
• Keywords: robust
• Abstract: The random vector functional link (RVFL) network is well-regarded for its strong generalization capabilities in the field of machine learning. However, its inherent dependencies on the square loss function make it susceptible to noise and outliers. Furthermore, the calculation of RVFL's unknown parameters necessitates matrix inversion of the entire training sample, which constrains its scalability. To address these challenges, we propose the Wave-RVFL, an RVFL model incorporating the wave loss function. We formulate and solve the proposed optimization problem of the Wave-RVFL using the adaptive moment estimation (Adam) algorithm in a way that successfully eliminates the requirement for matrix inversion and significantly enhances scalability. The Wave-RVFL exhibits robustness against noise and outliers by preventing over-penalization of deviations, thereby maintaining a balanced approach to managing noise and outliers. The proposed Wave-RVFL model is evaluated on multiple UCI datasets, both with and without the addition of noise and outliers, across various domains and sizes. Empirical results affirm the superior performance and robustness of the Wave-RVFL compared to baseline models, establishing it as a highly effective and scalable classification solution.

Title: Efficient ECC-based authentication scheme for fog-based IoT environment

• Authors: Mohamed Ali Shaaban, Almohammady S. Alsharkawy, Mohammad T. AbouKreisha, Mohammed Abdel Razek
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.02826
• Pdf URL: https://arxiv.org/pdf/2408.02826
• Copy Paste: [[2408.02826]] Efficient ECC-based authentication scheme for fog-based IoT environment(https://arxiv.org/abs/2408.02826)
• Keywords: secure, security, attack
• Abstract: The rapid growth of cloud computing and Internet of Things (IoT) applications faces several threats, such as latency, security, network failure, and performance. These issues are solved with the development of fog computing, which brings storage and computation closer to IoT-devices. However, there are several challenges faced by security designers, engineers, and researchers to secure this environment. To ensure the confidentiality of data that passes between the connected devices, digital signature protocols have been applied to the authentication of identities and messages. However, in the traditional method, a user's private key is directly stored on IoTs, so the private key may be disclosed under various malicious attacks. Furthermore, these methods require a lot of energy, which drains the resources of IoT-devices. A signature scheme based on the elliptic curve digital signature algorithm (ECDSA) is proposed in this paper to improve the security of the private key and the time taken for key-pair generation. ECDSA security is based on the intractability of the Elliptic Curve Discrete Logarithm Problem (ECDLP), which allows one to use much smaller groups. Smaller group sizes directly translate into shorter signatures, which is a crucial feature in settings where communication bandwidth is limited, or data transfer consumes a large amount of energy. The efficiency and effectiveness of ECDSA in the IoT environment are validated by experimental evaluation and comparison analysis. The results indicate that, in comparison to the two-party ECDSA and RSA, the proposed ECDSA decreases computation time by 65% and 87%, respectively. Additionally, as compared to two-party ECDSA and RSA, respectively, it reduces energy consumption by 77% and 82%.

Title: DaCapo: a modular deep learning framework for scalable 3D image segmentation

• Authors: William Patton, Jeff L. Rhoades, Marwan Zouinkhi, David G. Ackerman, Caroline Malin-Mayor, Diane Adjavon, Larissa Heinrich, Davis Bennett, Yurii Zubov, CellMap Project Team, Aubrey V. Weigel, Jan Funke
• Subjects: cs.CV, cs.LG, eess.IV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2408.02834
• Pdf URL: https://arxiv.org/pdf/2408.02834
• Copy Paste: [[2408.02834]] DaCapo: a modular deep learning framework for scalable 3D image segmentation(https://arxiv.org/abs/2408.02834)
• Keywords: segmentation
• Abstract: DaCapo is a specialized deep learning library tailored to expedite the training and application of existing machine learning approaches on large, near-isotropic image data. In this correspondence, we introduce DaCapo's unique features optimized for this specific domain, highlighting its modular structure, efficient experiment management tools, and scalable deployment capabilities. We discuss its potential to improve access to large-scale, isotropic image segmentation and invite the community to explore and contribute to this open-source initiative.

Title: GAReT: Cross-view Video Geolocalization with Adapters and Auto-Regressive Transformers

• Authors: Manu S Pillai, Mamshad Nayeem Rizve, Mubarak Shah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02840
• Pdf URL: https://arxiv.org/pdf/2408.02840
• Copy Paste: [[2408.02840]] GAReT: Cross-view Video Geolocalization with Adapters and Auto-Regressive Transformers(https://arxiv.org/abs/2408.02840)
• Keywords: extraction, transformer
• Abstract: Cross-view video geo-localization (CVGL) aims to derive GPS trajectories from street-view videos by aligning them with aerial-view images. Despite their promising performance, current CVGL methods face significant challenges. These methods use camera and odometry data, typically absent in real-world scenarios. They utilize multiple adjacent frames and various encoders for feature extraction, resulting in high computational costs. Moreover, these approaches independently predict each street-view frame's location, resulting in temporally inconsistent GPS trajectories. To address these challenges, in this work, we propose GAReT, a fully transformer-based method for CVGL that does not require camera and odometry data. We introduce GeoAdapter, a transformer-adapter module designed to efficiently aggregate image-level representations and adapt them for video inputs. Specifically, we train a transformer encoder on video frames and aerial images, then freeze the encoder to optimize the GeoAdapter module to obtain video-level representation. To address temporally inconsistent trajectories, we introduce TransRetriever, an encoder-decoder transformer model that predicts GPS locations of street-view frames by encoding top-k nearest neighbor predictions per frame and auto-regressively decoding the best neighbor based on the previous frame's predictions. Our method's effectiveness is validated through extensive experiments, demonstrating state-of-the-art performance on benchmark datasets. Our code is available at this https URL.

Title: Heterogeneous graph attention network improves cancer multiomics integration

• Authors: Sina Tabakhi, Charlotte Vandermeulen, Ian Sudbery, Haiping Lu
• Subjects: cs.LG, cs.MA, q-bio.BM, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2408.02845
• Pdf URL: https://arxiv.org/pdf/2408.02845
• Copy Paste: [[2408.02845]] Heterogeneous graph attention network improves cancer multiomics integration(https://arxiv.org/abs/2408.02845)
• Keywords: interpretability
• Abstract: The increase in high-dimensional multiomics data demands advanced integration models to capture the complexity of human diseases. Graph-based deep learning integration models, despite their promise, struggle with small patient cohorts and high-dimensional features, often applying independent feature selection without modeling relationships among omics. Furthermore, conventional graph-based omics models focus on homogeneous graphs, lacking multiple types of nodes and edges to capture diverse structures. We introduce a Heterogeneous Graph ATtention network for omics integration (HeteroGATomics) to improve cancer diagnosis. HeteroGATomics performs joint feature selection through a multi-agent system, creating dedicated networks of feature and patient similarity for each omic modality. These networks are then combined into one heterogeneous graph for learning holistic omic-specific representations and integrating predictions across modalities. Experiments on three cancer multiomics datasets demonstrate HeteroGATomics' superior performance in cancer diagnosis. Moreover, HeteroGATomics enhances interpretability by identifying important biomarkers contributing to the diagnosis outcomes.

Title: Less Is More: A Mixed-Methods Study on Security-Sensitive API Calls in Java for Better Dependency Selection

• Authors: Imranur Rahman, Ranidya Paramitha, Henrik Plate, Dominik Wermke, Laurie Williams
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.02846
• Pdf URL: https://arxiv.org/pdf/2408.02846
• Copy Paste: [[2408.02846]] Less Is More: A Mixed-Methods Study on Security-Sensitive API Calls in Java for Better Dependency Selection(https://arxiv.org/abs/2408.02846)
• Keywords: security, attack
• Abstract: Security sensitive APIs provide access to security-sensitive resources, e.g., the filesystem or network resources. Including such API calls -- directly or through dependencies -- increases the application's attack surface. An example of such a phenomenon is Log4Shell, which rendered many applications vulnerable due to network-related capabilities (JNDI lookup) in log4j package. Before the Log4Shell incident, alternate logging libraries to log4j were available that do not make JNDI lookup calls. The impact of such an incident would be minimal if information about network-related API calls by logging libraries were available to the developers. And so the lack of visibility into the calls to these security sensitive APIs by functionally similar open-source packages makes it difficult for developers to use them as a dependency selection criterion. The goal of this study is to aid developers in selecting their dependency by understanding security sensitive APIs in their dependency through call graph analysis. We conducted a mixed-methods study with 45 Java packages and defined a list of 219 security sensitive APIs. We then used call graph analysis to analyze the prevalence of these APIs in our selected package versions, with and without their dependencies. Finally, we conducted a survey with open-source developers (110 respondents) showing the comparison of functionally similar packages w.r.t. Security sensitive API calls to understand the usefulness of this API information in the dependency selection process. The number of Security sensitive API calls of functionally similar packages can vary from 0 to 368 in one API category and 0 to 429 in total. Our survey results show that 73% developers agree that information about the number and type of security-sensitive API calls of functionally similar packages would have been useful in their dependency selection.

Title: A Framework for Fine-Tuning LLMs using Heterogeneous Feedback

• Authors: Ryan Aponte (1), Ryan A. Rossi (2), Shunan Guo (2), Franck Dernoncourt (2), Tong Yu (2), Xiang Chen (2), Subrata Mitra (2), Nedim Lipka (2) ((1) Carnegie Mellon University, (2) Adobe Research)
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02861
• Pdf URL: https://arxiv.org/pdf/2408.02861
• Copy Paste: [[2408.02861]] A Framework for Fine-Tuning LLMs using Heterogeneous Feedback(https://arxiv.org/abs/2408.02861)
• Keywords: large language model
• Abstract: Large language models (LLMs) have been applied to a wide range of tasks, including text summarization, web navigation, and chatbots. They have benefitted from supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) following an unsupervised pretraining. These datasets can be difficult to collect, limited in scope, and vary in sample quality. Additionally, datasets can vary extensively in supervision format, from numerical to binary as well as multi-dimensional with many different values. We present a framework for fine-tuning LLMs using heterogeneous feedback, which has two main components. First, we combine the heterogeneous feedback data into a single supervision format, compatible with methods like SFT and RLHF. Next, given this unified feedback dataset, we extract a high-quality and diverse subset to obtain performance increases potentially exceeding the full dataset. We conduct extensive experiments to understand the effectiveness of these techniques for incorporating heterogeneous feedback, and demonstrate improvements from using a high-quality and diverse subset of the data. We find that our framework is able to improve models in multiple areas simultaneously, such as in instruction following and bias reduction.

Title: Back-Projection Diffusion: Solving the Wideband Inverse Scattering Problem with Diffusion Models

• Authors: Borong Zhang, Martín Guerra, Qin Li, Leonardo Zepeda-Núñez
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2408.02866
• Pdf URL: https://arxiv.org/pdf/2408.02866
• Copy Paste: [[2408.02866]] Back-Projection Diffusion: Solving the Wideband Inverse Scattering Problem with Diffusion Models(https://arxiv.org/abs/2408.02866)
• Keywords: diffusion
• Abstract: We present \textit{Wideband back-projection diffusion}, an end-to-end probabilistic framework for approximating the posterior distribution induced by the inverse scattering map from wideband scattering data. This framework leverages conditional diffusion models coupled with the underlying physics of wave-propagation and symmetries in the problem, to produce highly accurate reconstructions. The framework introduces a factorization of the score function into a physics-based latent representation inspired by the filtered back-propagation formula and a conditional score function conditioned on this latent representation. These two steps are also constrained to obey symmetries in the formulation while being amenable to compression by imposing the rank structure found in the filtered back-projection formula. As a result, empirically, our framework is able to provide sharp reconstructions effortlessly, even recovering sub-Nyquist features in the multiple-scattering regime. It has low-sample and computational complexity, its number of parameters scales sub-linearly with the target resolution, and it has stable training dynamics.

Title: Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learning

• Authors: Dmitri Iourovitski, Sanat Sharma, Rakshak Talwar
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02871
• Pdf URL: https://arxiv.org/pdf/2408.02871
• Copy Paste: [[2408.02871]] Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learning(https://arxiv.org/abs/2408.02871)
• Keywords: security, large language model
• Abstract: As content generated by Large Language Model (LLM) has grown exponentially, the ability to accurately identify and fingerprint such text has become increasingly crucial. In this work, we introduce a novel black-box approach for fingerprinting LLMs, achieving an impressive 72% accuracy in identifying the correct family of models (Such as Llama, Mistral, Gemma, etc) among a lineup of LLMs. We present an evolutionary strategy that leverages the capabilities of one LLM to discover the most salient features for identifying other LLMs. Our method employs a unique "Hide and Seek" algorithm, where an Auditor LLM generates discriminative prompts, and a Detective LLM analyzes the responses to fingerprint the target models. This approach not only demonstrates the feasibility of LLM-driven model identification but also reveals insights into the semantic manifolds of different LLM families. By iteratively refining prompts through in-context learning, our system uncovers subtle distinctions between model outputs, providing a powerful tool for LLM analysis and verification. This research opens new avenues for understanding LLM behavior and has significant implications for model attribution, security, and the broader field of AI transparency.

Title: Body of Her: A Preliminary Study on End-to-End Humanoid Agent

• Authors: Tenglong Ao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02879
• Pdf URL: https://arxiv.org/pdf/2408.02879
• Copy Paste: [[2408.02879]] Body of Her: A Preliminary Study on End-to-End Humanoid Agent(https://arxiv.org/abs/2408.02879)
• Keywords: large language model
• Abstract: Interactive virtual humanoid agent is a crucial interface with the physical world. A relatively complete humanoid agent first needs to have face and body, then possess both verbal and non-verbal (such as eye contact, facial expression, lip motion, gesture, and manipulation) abilities, and finally, it is capable of real-time duplex communication, e.g., the ability to actively interrupt conversations. Most prior systems typically only consider a subset of these elements, leaving a gap from realistic humanoid agent. In this work, we propose a real-time, duplex, interactive end-to-end network capable of modeling realistic agent behaviors, including speech, full-body movements for talking, responding, idling, and manipulation. This system is a multimodal model integrating audio and visual inputs, extended from a pre-trained large language model (LLM). We collect approximately 200,000 hours of audio, around 130,000 hours of video data, and about 20,000 alignment samples to build the model. The final model demonstrates capabilities that are difficult to achieve in previous systems, such as generalized object manipulation. This work performs a preliminary exploration of the end-to-end approach in this field, aiming to inspire further research towards scaling up.

Title: Diverse Generation while Maintaining Semantic Coordination: A Diffusion-Based Data Augmentation Method for Object Detection

• Authors: Sen Nie, Zhuo Wang, Xinxin Wang, Kun He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02891
• Pdf URL: https://arxiv.org/pdf/2408.02891
• Copy Paste: [[2408.02891]] Diverse Generation while Maintaining Semantic Coordination: A Diffusion-Based Data Augmentation Method for Object Detection(https://arxiv.org/abs/2408.02891)
• Keywords: diffusion
• Abstract: Recent studies emphasize the crucial role of data augmentation in enhancing the performance of object detection models. However,existing methodologies often struggle to effectively harmonize dataset diversity with semantic this http URL bridge this gap, we introduce an innovative augmentation technique leveraging pre-trained conditional diffusion models to mediate this balance. Our approach encompasses the development of a Category Affinity Matrix, meticulously designed to enhance dataset diversity, and a Surrounding Region Alignment strategy, which ensures the preservation of semantic coordination in the augmented images. Extensive experimental evaluations confirm the efficacy of our method in enriching dataset diversity while seamlessly maintaining semantic coordination. Our method yields substantial average improvements of +1.4AP, +0.9AP, and +3.4AP over existing alternatives on three distinct object detection models, respectively.

Title: SETN: Stock Embedding Enhanced with Textual and Network Information

• Authors: Takehiro Takayanagi, Hiroki Sakaji, Kiyoshi Izumi
• Subjects: cs.CL, cs.CE
• Abstract URL: https://arxiv.org/abs/2408.02899
• Pdf URL: https://arxiv.org/pdf/2408.02899
• Copy Paste: [[2408.02899]] SETN: Stock Embedding Enhanced with Textual and Network Information(https://arxiv.org/abs/2408.02899)
• Keywords: extraction, transformer
• Abstract: Stock embedding is a method for vector representation of stocks. There is a growing demand for vector representations of stock, i.e., stock embedding, in wealth management sectors, and the method has been applied to various tasks such as stock price prediction, portfolio optimization, and similar fund identifications. Stock embeddings have the advantage of enabling the quantification of relative relationships between stocks, and they can extract useful information from unstructured data such as text and network data. In this study, we propose stock embedding enhanced with textual and network information (SETN) using a domain-adaptive pre-trained transformer-based model to embed textual information and a graph neural network model to grasp network information. We evaluate the performance of our proposed model on related company information extraction tasks. We also demonstrate that stock embeddings obtained from the proposed model perform better in creating thematic funds than those obtained from baseline methods, providing a promising pathway for various applications in the wealth management industry.

Title: MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

• Authors: Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, Yuyin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02900
• Pdf URL: https://arxiv.org/pdf/2408.02900
• Copy Paste: [[2408.02900]] MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine(https://arxiv.org/abs/2408.02900)
• Keywords: large language model, segmentation
• Abstract: This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities, with multigranular annotations for more than 65 diseases. These enriched annotations encompass both global textual information, such as disease/lesion type, modality, region-specific descriptions, and inter-regional relationships, as well as detailed local annotations for regions of interest (ROIs), including bounding boxes, segmentation masks. Unlike existing approach which is limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and texual annotations (in the form of image-ROI-description triplets) without the need for any paired text descriptions. Specifically, data from over 90 different sources have been collected, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions. We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmented generation with the identified ROIs as guidance, resulting in multigranular texual descriptions. Compared to existing datasets, MedTrinity-25M provides the most enriched annotations, supporting a comprehensive range of multimodal tasks such as captioning and report generation, as well as vision-centric tasks like classification and segmentation. Pretraining on MedTrinity-25M, our model achieves state-of-the-art performance on VQA-RAD and PathVQA, surpassing both multimodal large language models and other representative SoTA approaches. This dataset can also be utilized to support large-scale pre-training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain.

Title: Enabling Intelligent Traffic Systems: A Deep Learning Method for Accurate Arabic License Plate Recognition

• Authors: M. A. Sayedelahl
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02904
• Pdf URL: https://arxiv.org/pdf/2408.02904
• Copy Paste: [[2408.02904]] Enabling Intelligent Traffic Systems: A Deep Learning Method for Accurate Arabic License Plate Recognition(https://arxiv.org/abs/2408.02904)
• Keywords: robust
• Abstract: This paper introduces a novel two-stage framework for accurate Egyptian Vehicle License Plate Recognition (EVLPR). The first stage employs image processing techniques to reliably localize license plates, while the second stage utilizes a custom-designed deep learning model for robust Arabic character recognition. The proposed system achieves a remarkable 99.3% accuracy on a diverse dataset, surpassing existing approaches. Its potential applications extend to intelligent traffic management, including traffic violation detection and parking optimization. Future research will focus on enhancing the system's capabilities through architectural refinements, expanded datasets, and addressing system dependencies.

Title: Leveraging Inter-Chunk Interactions for Enhanced Retrieval in Large Language Model-Based Question Answering

• Authors: Tiezheng Guo, Chen Wang, Yanyi Liu, Jiawei Tang, Pan Li, Sai Xu, Qingwen Yang, Xianlin Gao, Zhi Li, Yingyou Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02907
• Pdf URL: https://arxiv.org/pdf/2408.02907
• Copy Paste: [[2408.02907]] Leveraging Inter-Chunk Interactions for Enhanced Retrieval in Large Language Model-Based Question Answering(https://arxiv.org/abs/2408.02907)
• Keywords: large language model
• Abstract: Retrieving external knowledge and prompting large language models with relevant information is an effective paradigm to enhance the performance of question-answering tasks. Previous research typically handles paragraphs from external documents in isolation, resulting in a lack of context and ambiguous references, particularly in multi-document and complex tasks. To overcome these challenges, we propose a new retrieval framework IIER, that leverages Inter-chunk Interactions to Enhance Retrieval. This framework captures the internal connections between document chunks by considering three types of interactions: structural, keyword, and semantic. We then construct a unified Chunk-Interaction Graph to represent all external documents comprehensively. Additionally, we design a graph-based evidence chain retriever that utilizes previous paths and chunk interactions to guide the retrieval process. It identifies multiple seed nodes based on the target question and iteratively searches for relevant chunks to gather supporting evidence. This retrieval process refines the context and reasoning chain, aiding the large language model in reasoning and answer generation. Extensive experiments demonstrate that IIER outperforms strong baselines across four datasets, highlighting its effectiveness in improving retrieval and reasoning capabilities.

Title: Interoperability and Explicable AI-based Zero-Day Attacks Detection Process in Smart Community

• Authors: Mohammad Sayduzzaman, Jarin Tasnim Tamanna, Dipanjali Kundu, Tawhidur Rahman
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.02921
• Pdf URL: https://arxiv.org/pdf/2408.02921
• Copy Paste: [[2408.02921]] Interoperability and Explicable AI-based Zero-Day Attacks Detection Process in Smart Community(https://arxiv.org/abs/2408.02921)
• Keywords: security, protect, attack
• Abstract: Systems, technologies, protocols, and infrastructures all face interoperability challenges. It is among the most crucial parameters to give real-world effectiveness. Organizations that achieve interoperability will be able to identify, prevent, and provide appropriate protection on an international scale, which can be relied upon. This paper aims to explain how future technologies such as 6G mobile communication, Internet of Everything (IoE), Artificial Intelligence (AI), and Smart Contract embedded WPA3 protocol-based WiFi-8 can work together to prevent known attack vectors and provide protection against zero-day attacks, thus offering intelligent solutions for smart cities. The phrase zero-day refers to an attack that occurs on the day zero of the vulnerability's disclosure to the public or vendor. Existing systems require an extra layer of security. In the security world, interoperability enables disparate security solutions and systems to collaborate seamlessly. AI improves cybersecurity by enabling improved capabilities for detecting, responding, and preventing zero-day attacks. When interoperability and Explainable Artificial Intelligence (XAI) are integrated into cybersecurity, they form a strong protection against zero-day assaults. Additionally, we evaluate a couple of parameters based on the accuracy and time required for efficiently analyzing attack patterns and anomalies.

Title: Pose Magic: Efficient and Temporally Consistent Human Pose Estimation with a Hybrid Mamba-GCN Network

• Authors: Xinyi Zhang, Qiqi Bao, Qinpeng Cui, Wenming Yang, Qingmin Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02922
• Pdf URL: https://arxiv.org/pdf/2408.02922
• Copy Paste: [[2408.02922]] Pose Magic: Efficient and Temporally Consistent Human Pose Estimation with a Hybrid Mamba-GCN Network(https://arxiv.org/abs/2408.02922)
• Keywords: transformer
• Abstract: Current state-of-the-art (SOTA) methods in 3D Human Pose Estimation (HPE) are primarily based on Transformers. However, existing Transformer-based 3D HPE backbones often encounter a trade-off between accuracy and computational efficiency. To resolve the above dilemma, in this work, leveraging recent advances in state space models, we utilize Mamba for high-quality and efficient long-range modeling. Nonetheless, Mamba still faces challenges in precisely exploiting the local dependencies between joints. To address these issues, we propose a new attention-free hybrid spatiotemporal architecture named Hybrid Mamba-GCN (Pose Magic). This architecture introduces local enhancement with GCN by capturing relationships between neighboring joints, thus producing new representations to complement Mamba's outputs. By adaptively fusing representations from Mamba and GCN, Pose Magic demonstrates superior capability in learning the underlying 3D structure. To meet the requirements of real-time inference, we also provide a fully causal version. Extensive experiments show that Pose Magic achieves new SOTA results ($\downarrow 0.9 mm$) while saving $74.1\%$ FLOPs. In addition, Pose Magic exhibits optimal motion consistency and the ability to generalize to unseen sequence lengths.

Title: Intermediate direct preference optimization

• Authors: Atsushi Kojima
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02923
• Pdf URL: https://arxiv.org/pdf/2408.02923
• Copy Paste: [[2408.02923]] Intermediate direct preference optimization(https://arxiv.org/abs/2408.02923)
• Keywords: large language model
• Abstract: We propose the intermediate direct preference optimization (DPO) method to calculate the DPO loss at selected intermediate layers as an auxiliary loss for finetuning large language models (LLMs). The conventional DPO method fine-tunes a supervised fine-tuning (SFT) model by calculating the DPO loss using logits from the final layer. In our intermediate DPO approach, DPO losses are calculated using the logits from K-selected intermediate layers and averaged to obtain the intermediate DPO loss. For training the intermediate DPO model, the final loss is obtained by calculating the weighted sum of the DPO and intermediate DPO losses. During inference, the intermediate DPO model decodes using the final layer logits similarly to the conventional DPO model. In experiments using the ultrafeedback dataset, the performance of the intermediate DPO model was evaluated using GPT-4. As a result, the intermediate DPO model trained using the intermediate DPO loss calculated at the 22nd layer of a 32-layer SFT model achieved win rates of 52.5% and 67.5% against the conventional DPO and SFT models, respectively, demonstrating the effectiveness of the proposed method. Furthermore, we report the relationships among the position of the selected intermediate layers, the number of layers, and performance.

Title: Evaluation of Segment Anything Model 2: The Role of SAM2 in the Underwater Environment

• Authors: Shijie Lian, Hua Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02924
• Pdf URL: https://arxiv.org/pdf/2408.02924
• Copy Paste: [[2408.02924]] Evaluation of Segment Anything Model 2: The Role of SAM2 in the Underwater Environment(https://arxiv.org/abs/2408.02924)
• Keywords: segmentation
• Abstract: With breakthroughs in large-scale modeling, the Segment Anything Model (SAM) and its extensions have been attempted for applications in various underwater visualization tasks in marine sciences, and have had a significant impact on the academic community. Recently, Meta has further developed the Segment Anything Model 2 (SAM2), which significantly improves running speed and segmentation accuracy compared to its predecessor. This report aims to explore the potential of SAM2 in marine science by evaluating it on the underwater instance segmentation benchmark datasets UIIS and USIS10K. The experiments show that the performance of SAM2 is extremely dependent on the type of user-provided prompts. When using the ground truth bounding box as prompt, SAM2 performed excellently in the underwater instance segmentation domain. However, when running in automatic mode, SAM2's ability with point prompts to sense and segment underwater instances is significantly degraded. It is hoped that this paper will inspire researchers to further explore the SAM model family in the underwater domain. The results and evaluation codes in this paper are available at this https URL.

Title: HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection

• Authors: Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, Hao Wang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.02927
• Pdf URL: https://arxiv.org/pdf/2408.02927
• Copy Paste: [[2408.02927]] HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection(https://arxiv.org/abs/2408.02927)
• Keywords: privacy, protect
• Abstract: Data serves as the fundamental foundation for advancing deep learning, particularly tabular data presented in a structured format, which is highly conducive to modeling. However, even in the era of LLM, obtaining tabular data from sensitive domains remains a challenge due to privacy or copyright concerns. Hence, exploring how to effectively use models like LLMs to generate realistic and privacy-preserving synthetic tabular data is urgent. In this paper, we take a step forward to explore LLMs for tabular data synthesis and privacy protection, by introducing a new framework HARMONIC for tabular data generation and evaluation. In the tabular data generation of our framework, unlike previous small-scale LLM-based methods that rely on continued pre-training, we explore the larger-scale LLMs with fine-tuning to generate tabular data and enhance privacy. Based on idea of the k-nearest neighbors algorithm, an instruction fine-tuning dataset is constructed to inspire LLMs to discover inter-row relationships. Then, with fine-tuning, LLMs are trained to remember the format and connections of the data rather than the data itself, which reduces the risk of privacy leakage. In the evaluation part of our framework, we develop specific privacy risk metrics DLT for LLM synthetic data generation, as well as performance evaluation metrics LLE for downstream LLM tasks. Our experiments find that this tabular data generation framework achieves equivalent performance to existing methods with better privacy, which also demonstrates our evaluation framework for the effectiveness of synthetic data and privacy risks in LLM scenarios.

Title: Segmenting Small Stroke Lesions with Novel Labeling Strategies

• Authors: Liang Shang, Zhengyang Lou, Andrew L. Alexander, Vivek Prabhakaran, William A. Sethares, Veena A. Nair, Nagesh Adluru
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02929
• Pdf URL: https://arxiv.org/pdf/2408.02929
• Copy Paste: [[2408.02929]] Segmenting Small Stroke Lesions with Novel Labeling Strategies(https://arxiv.org/abs/2408.02929)
• Keywords: segmentation
• Abstract: Deep neural networks have demonstrated exceptional efficacy in stroke lesion segmentation. However, the delineation of small lesions, critical for stroke diagnosis, remains a challenge. In this study, we propose two straightforward yet powerful approaches that can be seamlessly integrated into a variety of networks: Multi-Size Labeling (MSL) and Distance-Based Labeling (DBL), with the aim of enhancing the segmentation accuracy of small lesions. MSL divides lesion masks into various categories based on lesion volume while DBL emphasizes the lesion boundaries. Experimental evaluations on the Anatomical Tracings of Lesions After Stroke (ATLAS) v2.0 dataset showcase that an ensemble of MSL and DBL achieves consistently better or equal performance on recall (3.6% and 3.7%), F1 (2.4% and 1.5%), and Dice scores (1.3% and 0.0%) compared to the top-1 winner of the 2022 MICCAI ATLAS Challenge on both the subset only containing small lesions and the entire dataset, respectively. Notably, on the mini-lesion subset, a single MSL model surpasses the previous best ensemble strategy, with enhancements of 1.0% and 0.3% on F1 and Dice scores, respectively. Our code is available at: this https URL.

Title: Scaling Laws for Data Poisoning in LLMs

• Authors: Dillon Bowen, Brendan Murphy, Will Cai, David Khachaturov, Adam Gleave, Kellin Pelrine
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02946
• Pdf URL: https://arxiv.org/pdf/2408.02946
• Copy Paste: [[2408.02946]] Scaling Laws for Data Poisoning in LLMs(https://arxiv.org/abs/2408.02946)
• Keywords: robust
• Abstract: Recent work shows that LLMs are vulnerable to data poisoning, in which they are trained on partially corrupted or harmful data. Poisoned data is hard to detect, breaks guardrails, and leads to undesirable and harmful behavior. Given the intense efforts by leading labs to train and deploy increasingly larger and more capable LLMs, it is critical to ask if the risk of data poisoning will be naturally mitigated by scale, or if it is an increasing threat. We consider three threat models by which data poisoning can occur: malicious fine-tuning, imperfect data curation, and intentional data contamination. Our experiments evaluate the effects of data poisoning on 23 frontier LLMs ranging from 1.5-72 billion parameters on three datasets which speak to each of our threat models. We find that larger LLMs are increasingly vulnerable, learning harmful behavior -- including sleeper agent behavior -- significantly more quickly than smaller LLMs with even minimal data poisoning. These results underscore the need for robust safeguards against data poisoning in larger LLMs.

Title: Kolmogorov-Arnold PointNet: Deep learning for prediction of fluid fields on irregular geometries

• Authors: Ali Kashefi
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2408.02950
• Pdf URL: https://arxiv.org/pdf/2408.02950
• Copy Paste: [[2408.02950]] Kolmogorov-Arnold PointNet: Deep learning for prediction of fluid fields on irregular geometries(https://arxiv.org/abs/2408.02950)
• Keywords: segmentation
• Abstract: We present Kolmogorov-Arnold PointNet (KA-PointNet) as a novel supervised deep learning framework for the prediction of incompressible steady-state fluid flow fields in irregular domains, where the predicted fields are a function of the geometry of the domains. In KA-PointNet, we implement shared Kolmogorov-Arnold Networks (KANs) in the segmentation branch of the PointNet architecture. We utilize Jacobi polynomials to construct shared KANs. As a benchmark test case, we consider incompressible laminar steady-state flow over a cylinder, where the geometry of its cross-section varies over the data set. We investigate the performance of Jacobi polynomials with different degrees as well as special cases of Jacobi polynomials such as Legendre polynomials, Chebyshev polynomials of the first and second kinds, and Gegenbauer polynomials, in terms of the computational cost of training and accuracy of prediction of the test set. Additionally, we compare the performance of PointNet with shared KANs (i.e., KA-PointNet) and PointNet with shared Multilayer Perceptrons (MLPs). It is observed that when the number of trainable parameters is approximately equal, PointNet with shared KANs (i.e., KA-PointNet) outperforms PointNet with shared MLPs.

Title: WWW: Where, Which and Whatever Enhancing Interpretability in Multimodal Deepfake Detection

• Authors: Juho Jung, Sangyoun Lee, Jooeon Kang, Yunjin Na
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02954
• Pdf URL: https://arxiv.org/pdf/2408.02954
• Copy Paste: [[2408.02954]] WWW: Where, Which and Whatever Enhancing Interpretability in Multimodal Deepfake Detection(https://arxiv.org/abs/2408.02954)
• Keywords: attack, robust, interpretability
• Abstract: All current benchmarks for multimodal deepfake detection manipulate entire frames using various generation techniques, resulting in oversaturated detection accuracies exceeding 94% at the video-level classification. However, these benchmarks struggle to detect dynamic deepfake attacks with challenging frame-by-frame alterations presented in real-world scenarios. To address this limitation, we introduce FakeMix, a novel clip-level evaluation benchmark aimed at identifying manipulated segments within both video and audio, providing insight into the origins of deepfakes. Furthermore, we propose novel evaluation metrics, Temporal Accuracy (TA) and Frame-wise Discrimination Metric (FDM), to assess the robustness of deepfake detection models. Evaluating state-of-the-art models against diverse deepfake benchmarks, particularly FakeMix, demonstrates the effectiveness of our approach comprehensively. Specifically, while achieving an Average Precision (AP) of 94.2% at the video-level, the evaluation of the existing models at the clip-level using the proposed metrics, TA and FDM, yielded sharp declines in accuracy to 53.1%, and 52.1%, respectively.

Title: Online Temporal Action Localization with Memory-Augmented Transformer

• Authors: Youngkil Song, Dongkeun Kim, Minsu Cho, Suha Kwak
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02957
• Pdf URL: https://arxiv.org/pdf/2408.02957
• Copy Paste: [[2408.02957]] Online Temporal Action Localization with Memory-Augmented Transformer(https://arxiv.org/abs/2408.02957)
• Keywords: transformer
• Abstract: Online temporal action localization (On-TAL) is the task of identifying multiple action instances given a streaming video. Since existing methods take as input only a video segment of fixed size per iteration, they are limited in considering long-term context and require tuning the segment size carefully. To overcome these limitations, we propose memory-augmented transformer (MATR). MATR utilizes the memory queue that selectively preserves the past segment features, allowing to leverage long-term context for inference. We also propose a novel action localization method that observes the current input segment to predict the end time of the ongoing action and accesses the memory queue to estimate the start time of the action. Our method outperformed existing methods on two datasets, THUMOS14 and MUSES, surpassing not only TAL methods in the online setting but also some offline TAL methods.

Title: Accuracy and Consistency of LLMs in the Registered Dietitian Exam: The Impact of Prompt Engineering and Knowledge Retrieval

• Authors: Iman Azimi, Mohan Qi, Li Wang, Amir M. Rahmani, Youlin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02964
• Pdf URL: https://arxiv.org/pdf/2408.02964
• Copy Paste: [[2408.02964]] Accuracy and Consistency of LLMs in the Registered Dietitian Exam: The Impact of Prompt Engineering and Knowledge Retrieval(https://arxiv.org/abs/2408.02964)
• Keywords: large language model
• Abstract: Large language models (LLMs) are fundamentally transforming human-facing applications in the health and well-being domains: boosting patient engagement, accelerating clinical decision-making, and facilitating medical education. Although state-of-the-art LLMs have shown superior performance in several conversational applications, evaluations within nutrition and diet applications are still insufficient. In this paper, we propose to employ the Registered Dietitian (RD) exam to conduct a standard and comprehensive evaluation of state-of-the-art LLMs, GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro, assessing both accuracy and consistency in nutrition queries. Our evaluation includes 1050 RD exam questions encompassing several nutrition topics and proficiency levels. In addition, for the first time, we examine the impact of Zero-Shot (ZS), Chain of Thought (CoT), Chain of Thought with Self Consistency (CoT-SC), and Retrieval Augmented Prompting (RAP) on both accuracy and consistency of the responses. Our findings revealed that while these LLMs obtained acceptable overall performance, their results varied considerably with different prompts and question domains. GPT-4o with CoT-SC prompting outperformed the other approaches, whereas Gemini 1.5 Pro with ZS recorded the highest consistency. For GPT-4o and Claude 3.5, CoT improved the accuracy, and CoT-SC improved both accuracy and consistency. RAP was particularly effective for GPT-4o to answer Expert level questions. Consequently, choosing the appropriate LLM and prompting technique, tailored to the proficiency level and specific domain, can mitigate errors and potential risks in diet and nutrition chatbots.

Title: Data-Driven Stochastic Closure Modeling via Conditional Diffusion Model and Neural Operator

• Authors: Xinghao Dong, Chuanqi Chen, Jin-Long Wu
• Subjects: cs.LG, math.DS, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2408.02965
• Pdf URL: https://arxiv.org/pdf/2408.02965
• Copy Paste: [[2408.02965]] Data-Driven Stochastic Closure Modeling via Conditional Diffusion Model and Neural Operator(https://arxiv.org/abs/2408.02965)
• Keywords: diffusion, generative
• Abstract: Closure models are widely used in simulating complex multiscale dynamical systems such as turbulence and the earth system, for which direct numerical simulation that resolves all scales is often too expensive. For those systems without a clear scale separation, deterministic and local closure models often lack enough generalization capability, which limits their performance in many real-world applications. In this work, we propose a data-driven modeling framework for constructing stochastic and non-local closure models via conditional diffusion model and neural operator. Specifically, the Fourier neural operator is incorporated into a score-based diffusion model, which serves as a data-driven stochastic closure model for complex dynamical systems governed by partial differential equations (PDEs). We also demonstrate how accelerated sampling methods can improve the efficiency of the data-driven stochastic closure model. The results show that the proposed methodology provides a systematic approach via generative machine learning techniques to construct data-driven stochastic closure models for multiscale dynamical systems with continuous spatiotemporal fields.

Title: EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantization

• Authors: Zhaopeng Feng, Zijie Meng, Zuozhu Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02970
• Pdf URL: https://arxiv.org/pdf/2408.02970
• Copy Paste: [[2408.02970]] EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantization(https://arxiv.org/abs/2408.02970)
• Keywords: large language model
• Abstract: Large language models (LLMs) have attracted considerable attention in various fields for their cost-effective solutions to diverse challenges, especially with advancements in instruction tuning and quantization. E-commerce, with its complex tasks and extensive product-user interactions, presents a promising application area for LLMs. However, the domain-specific concepts and knowledge inherent in e-commerce pose significant challenges for adapting general LLMs. To address this issue, we developed EC-Guide \href{this https URL}, a comprehensive e-commerce guide for instruction tuning and quantization of LLMs. We also heuristically integrated Chain-of-Thought (CoT) during inference to enhance arithmetic performance. Our approach achieved the 2nd place in Track 2 and 5th place in Track 5 at the Amazon KDD Cup'24 \href{this https URL}. Additionally, our solution is model-agnostic, enabling effective scalability across larger systems.

Title: Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Models

• Authors: Haonan Zheng, Wen Jiang, Xinyang Deng, Wenrui Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02980
• Pdf URL: https://arxiv.org/pdf/2408.02980
• Copy Paste: [[2408.02980]] Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Models(https://arxiv.org/abs/2408.02980)
• Keywords: security, attack, robust
• Abstract: Recent studies on AI security have highlighted the vulnerability of Vision-Language Pre-training (VLP) models to subtle yet intentionally designed perturbations in images and texts. Investigating multimodal systems' robustness via adversarial attacks is crucial in this field. Most multimodal attacks are sample-specific, generating a unique perturbation for each sample to construct adversarial samples. To the best of our knowledge, it is the first work through multimodal decision boundaries to explore the creation of a universal, sample-agnostic perturbation that applies to any image. Initially, we explore strategies to move sample points beyond the decision boundaries of linear classifiers, refining the algorithm to ensure successful attacks under the top $k$ accuracy metric. Based on this foundation, in visual-language tasks, we treat visual and textual modalities as reciprocal sample points and decision hyperplanes, guiding image embeddings to traverse text-constructed decision boundaries, and vice versa. This iterative process consistently refines a universal perturbation, ultimately identifying a singular direction within the input space which is exploitable to impair the retrieval performance of VLP models. The proposed algorithms support the creation of global perturbations or adversarial patches. Comprehensive experiments validate the effectiveness of our method, showcasing its data, task, and model transferability across various VLP models and datasets. Code: this https URL

Title: Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond

• Authors: Jichuan Zhang, Yali Li, Xin Liu, Shengjin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02983
• Pdf URL: https://arxiv.org/pdf/2408.02983
• Copy Paste: [[2408.02983]] Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond(https://arxiv.org/abs/2408.02983)
• Keywords: diffusion
• Abstract: Non-exemplar class-incremental learning (NECIL) is to resist catastrophic forgetting without saving old class samples. Prior methodologies generally employ simple rules to generate features for replaying, suffering from large distribution gap between replayed features and real ones. To address the aforementioned issue, we propose a simple, yet effective \textbf{Diff}usion-based \textbf{F}eature \textbf{R}eplay (\textbf{DiffFR}) method for NECIL. First, to alleviate the limited representational capacity caused by fixing the feature extractor, we employ Siamese-based self-supervised learning for initial generalizable features. Second, we devise diffusion models to generate class-representative features highly similar to real features, which provides an effective way for exemplar-free knowledge memorization. Third, we introduce prototype calibration to direct the diffusion model's focus towards learning the distribution shapes of features, rather than the entire distribution. Extensive experiments on public datasets demonstrate significant performance gains of our DiffFR, outperforming the state-of-the-art NECIL methods by 3.0\% in average. The code will be made publicly available soon.

Title: A Differential Smoothness-based Compact-Dynamic Graph Convolutional Network for Spatiotemporal Signal Recovery

• Authors: Pengcheng Gao, Zicheng Gao, Ye Yuan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02987
• Pdf URL: https://arxiv.org/pdf/2408.02987
• Copy Paste: [[2408.02987]] A Differential Smoothness-based Compact-Dynamic Graph Convolutional Network for Spatiotemporal Signal Recovery(https://arxiv.org/abs/2408.02987)
• Keywords: security
• Abstract: High quality spatiotemporal signal is vitally important for real application scenarios like energy management, traffic planning and cyber security. Due to the uncontrollable factors like abrupt sensors breakdown or communication fault, the spatiotemporal signal collected by sensors is always incomplete. A dynamic graph convolutional network (DGCN) is effective for processing spatiotemporal signal recovery. However, it adopts a static GCN and a sequence neural network to explore the spatial and temporal patterns, separately. Such a separated two-step processing is loose spatiotemporal, thereby failing to capture the complex inner spatiotemporal correlation. To address this issue, this paper proposes a Compact-Dynamic Graph Convolutional Network (CDGCN) for spatiotemporal signal recovery with the following two-fold ideas: a) leveraging the tensor M-product to build a unified tensor graph convolution framework, which considers both spatial and temporal patterns simultaneously; and b) constructing a differential smoothness-based objective function to reduce the noise interference in spatiotemporal signal, thereby further improve the recovery accuracy. Experiments on real-world spatiotemporal datasets demonstrate that the proposed CDGCN significantly outperforms the state-of-the-art models in terms of recovery accuracy.

Title: Federated Learning Architectures: A Performance Evaluation with Crop Yield Prediction Application

• Authors: Anwesha Mukherjee, Rajkumar Buyya
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02998
• Pdf URL: https://arxiv.org/pdf/2408.02998
• Copy Paste: [[2408.02998]] Federated Learning Architectures: A Performance Evaluation with Crop Yield Prediction Application(https://arxiv.org/abs/2408.02998)
• Keywords: federate
• Abstract: Federated learning has become an emerging technology for data analysis for IoT applications. This paper implements centralized and decentralized federated learning frameworks for crop yield prediction based on Long Short-Term Memory Network. For centralized federated learning, multiple clients and one server is considered, where the clients exchange their model updates with the server that works as the aggregator to build the global model. For the decentralized framework, a collaborative network is formed among the devices either using ring topology or using mesh topology. In this network, each device receives model updates from the neighbour devices, and performs aggregation to build the upgraded model. The performance of the centralized and decentralized federated learning frameworks are evaluated in terms of prediction accuracy, precision, recall, F1-Score, and training time. The experimental results present that $\geq$97% and $>$97.5% prediction accuracy are achieved using the centralized and decentralized federated learning-based frameworks respectively. The results also show that the using centralized federated learning the response time can be reduced by $\sim$75% than the cloud-only framework. Finally, the future research directions of the use of federated learning in crop yield prediction are explored in this paper.

Title: Multitask and Multimodal Neural Tuning for Large Models

• Authors: Hao Sun, Yu Song, Jihong Hu, Yen-Wei Chen, Lanfen Lin
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.03001
• Pdf URL: https://arxiv.org/pdf/2408.03001
• Copy Paste: [[2408.03001]] Multitask and Multimodal Neural Tuning for Large Models(https://arxiv.org/abs/2408.03001)
• Keywords: segmentation
• Abstract: In recent years, large-scale multimodal models have demonstrated impressive capabilities across various domains. However, enabling these models to effectively perform multiple multimodal tasks simultaneously remains a significant challenge. To address this, we introduce a novel tuning method called neural tuning, designed to handle diverse multimodal tasks concurrently, including reasoning segmentation, referring segmentation, image captioning, and text-to-image generation. Neural tuning emulates sparse distributed representation in human brain, where only specific subsets of neurons are activated for each task. Additionally, we present a new benchmark, MMUD, where each sample is annotated with multiple task labels. By applying neural tuning to pretrained large models on the MMUD benchmark, we achieve simultaneous task handling in a streamlined and efficient manner. All models, code, and datasets will be publicly available after publication, facilitating further research and development in this field.

Title: Fact Finder -- Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs

• Authors: Daniel Steinigen, Roman Teucher, Timm Heine Ruland, Max Rudat, Nicolas Flores-Herr, Peter Fischer, Nikola Milosevic, Christopher Schymura, Angelo Ziletti
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.03010
• Pdf URL: https://arxiv.org/pdf/2408.03010
• Copy Paste: [[2408.03010]] Fact Finder -- Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs(https://arxiv.org/abs/2408.03010)
• Keywords: large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have showcased their proficiency in answering natural language queries. However, their effectiveness is hindered by limited domain-specific knowledge, raising concerns about the reliability of their responses. We introduce a hybrid system that augments LLMs with domain-specific knowledge graphs (KGs), thereby aiming to enhance factual correctness using a KG-based retrieval approach. We focus on a medical KG to demonstrate our methodology, which includes (1) pre-processing, (2) Cypher query generation, (3) Cypher query processing, (4) KG retrieval, and (5) LLM-enhanced response generation. We evaluate our system on a curated dataset of 69 samples, achieving a precision of 78\% in retrieving correct KG nodes. Our findings indicate that the hybrid system surpasses a standalone LLM in accuracy and completeness, as verified by an LLM-as-a-Judge evaluation method. This positions the system as a promising tool for applications that demand factual correctness and completeness, such as target identification -- a critical process in pinpointing biological entities for disease treatment or crop enhancement. Moreover, its intuitive search interface and ability to provide accurate responses within seconds make it well-suited for time-sensitive, precision-focused research contexts. We publish the source code together with the dataset and the prompt templates used.

Title: L3iTC at the FinLLM Challenge Task: Quantization for Financial Text Classification & Summarization

• Authors: Elvys Linhares Pontes, Carlos-Emiliano González-Gallardo, Mohamed Benjannet, Caryn Qu, Antoine Doucet
• Subjects: cs.CL, cs.CE
• Abstract URL: https://arxiv.org/abs/2408.03033
• Pdf URL: https://arxiv.org/pdf/2408.03033
• Copy Paste: [[2408.03033]] L3iTC at the FinLLM Challenge Task: Quantization for Financial Text Classification & Summarization(https://arxiv.org/abs/2408.03033)
• Keywords: secure, large language model
• Abstract: This article details our participation (L3iTC) in the FinLLM Challenge Task 2024, focusing on two key areas: Task 1, financial text classification, and Task 2, financial text summarization. To address these challenges, we fine-tuned several large language models (LLMs) to optimize performance for each task. Specifically, we used 4-bit quantization and LoRA to determine which layers of the LLMs should be trained at a lower precision. This approach not only accelerated the fine-tuning process on the training data provided by the organizers but also enabled us to run the models on low GPU memory. Our fine-tuned models achieved third place for the financial classification task with an F1-score of 0.7543 and secured sixth place in the financial summarization task on the official test datasets.

Title: Targeted Visual Prompting for Medical Visual Question Answering

• Authors: Sergio Tascon-Morales, Pablo Márquez-Neila, Raphael Sznitman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03043
• Pdf URL: https://arxiv.org/pdf/2408.03043
• Copy Paste: [[2408.03043]] Targeted Visual Prompting for Medical Visual Question Answering(https://arxiv.org/abs/2408.03043)
• Keywords: large language model
• Abstract: With growing interest in recent years, medical visual question answering (Med-VQA) has rapidly evolved, with multimodal large language models (MLLMs) emerging as an alternative to classical model architectures. Specifically, their ability to add visual information to the input of pre-trained LLMs brings new capabilities for image interpretation. However, simple visual errors cast doubt on the actual visual understanding abilities of these models. To address this, region-based questions have been proposed as a means to assess and enhance actual visual understanding through compositional evaluation. To combine these two perspectives, this paper introduces targeted visual prompting to equip MLLMs with region-based questioning capabilities. By presenting the model with both the isolated region and the region in its context in a customized visual prompt, we show the effectiveness of our method across multiple datasets while comparing it to several baseline models. Our code and data are available at this https URL.

Title: Comb, Prune, Distill: Towards Unified Pruning for Vision Model Compression

• Authors: Jonas Schmitt, Ruiping Liu, Junwei Zheng, Jiaming Zhang, Rainer Stiefelhagen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03046
• Pdf URL: https://arxiv.org/pdf/2408.03046
• Copy Paste: [[2408.03046]] Comb, Prune, Distill: Towards Unified Pruning for Vision Model Compression(https://arxiv.org/abs/2408.03046)
• Keywords: transformer, segmentation
• Abstract: Lightweight and effective models are essential for devices with limited resources, such as intelligent vehicles. Structured pruning offers a promising approach to model compression and efficiency enhancement. However, existing methods often tie pruning techniques to specific model architectures or vision tasks. To address this limitation, we propose a novel unified pruning framework Comb, Prune, Distill (CPD), which addresses both model-agnostic and task-agnostic concerns simultaneously. Our framework employs a combing step to resolve hierarchical layer-wise dependency issues, enabling architecture independence. Additionally, the pruning pipeline adaptively remove parameters based on the importance scoring metrics regardless of vision tasks. To support the model in retaining its learned information, we introduce knowledge distillation during the pruning step. Extensive experiments demonstrate the generalizability of our framework, encompassing both convolutional neural network (CNN) and transformer models, as well as image classification and segmentation tasks. In image classification we achieve a speedup of up to x4.3 with a accuracy loss of 1.8% and in semantic segmentation up to x1.89 with a 5.1% loss in mIoU.

Title: MGFs: Masked Gaussian Fields for Meshing Building based on Multi-View Images

• Authors: Tengfei Wang, Zongqian Zhan, Rui Xia, Linxia Ji, Xin Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.03060
• Pdf URL: https://arxiv.org/pdf/2408.03060
• Copy Paste: [[2408.03060]] MGFs: Masked Gaussian Fields for Meshing Building based on Multi-View Images(https://arxiv.org/abs/2408.03060)
• Keywords: extraction
• Abstract: Over the last few decades, image-based building surface reconstruction has garnered substantial research interest and has been applied across various fields, such as heritage preservation, architectural planning, etc. Compared to the traditional photogrammetric and NeRF-based solutions, recently, Gaussian fields-based methods have exhibited significant potential in generating surface meshes due to their time-efficient training and detailed 3D information preservation. However, most gaussian fields-based methods are trained with all image pixels, encompassing building and nonbuilding areas, which results in a significant noise for building meshes and degeneration in time efficiency. This paper proposes a novel framework, Masked Gaussian Fields (MGFs), designed to generate accurate surface reconstruction for building in a time-efficient way. The framework first applies EfficientSAM and COLMAP to generate multi-level masks of building and the corresponding masked point clouds. Subsequently, the masked gaussian fields are trained by integrating two innovative losses: a multi-level perceptual masked loss focused on constructing building regions and a boundary loss aimed at enhancing the details of the boundaries between different masks. Finally, we improve the tetrahedral surface mesh extraction method based on the masked gaussian spheres. Comprehensive experiments on UAV images demonstrate that, compared to the traditional method and several NeRF-based and Gaussian-based SOTA solutions, our approach significantly improves both the accuracy and efficiency of building surface reconstruction. Notably, as a byproduct, there is an additional gain in the novel view synthesis of building.

Title: Towards an Analysis of Discourse and Interactional Pragmatic Reasoning Capabilities of Large Language Models

• Authors: Amelie Robrecht, Judith Sieker, Clara Lachenmaier, Sina Zarieß, Stefan Kopp
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03074
• Pdf URL: https://arxiv.org/pdf/2408.03074
• Copy Paste: [[2408.03074]] Towards an Analysis of Discourse and Interactional Pragmatic Reasoning Capabilities of Large Language Models(https://arxiv.org/abs/2408.03074)
• Keywords: large language model
• Abstract: In this work, we want to give an overview on which pragmatic abilities have been tested in LLMs so far and how these tests have been carried out. To do this, we first discuss the scope of the field of pragmatics and suggest a subdivision into discourse pragmatics and interactional pragmatics. We give a non-exhaustive overview of the phenomena of those two subdomains and the methods traditionally used to analyze them. We subsequently consider the resulting heterogeneous set of phenomena and methods as a starting point for our survey of work on discourse pragmatics and interactional pragmatics in the context of LLMs.

Title: BodySLAM: A Generalized Monocular Visual SLAM Framework for Surgical Applications

• Authors: G. Manni (1 and 2), C. Lauretti (2), F. Prata (3), R. Papalia (3), L. Zollo (2), P. Soda (1) ((1) Research Unit of Computer Systems and Bioinformatics Department of Engineering Università Campus Bio-Medico di Roma, (2) Unit of Advanced Robotics and Human-Centred Technologies Department of Engineering Università Campus Bio-Medico di Roma, (3) Department of Urology Fondazione Policlinico Universitario Campus Bio-Medico)
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.03078
• Pdf URL: https://arxiv.org/pdf/2408.03078
• Copy Paste: [[2408.03078]] BodySLAM: A Generalized Monocular Visual SLAM Framework for Surgical Applications(https://arxiv.org/abs/2408.03078)
• Keywords: robust
• Abstract: Endoscopic surgery relies on two-dimensional views, posing challenges for surgeons in depth perception and instrument manipulation. While Simultaneous Localization and Mapping (SLAM) has emerged as a promising solution to address these limitations, its implementation in endoscopic procedures presents significant challenges due to hardware limitations, such as the use of a monocular camera and the absence of odometry sensors. This study presents a robust deep learning-based SLAM approach that combines state-of-the-art and newly developed models. It consists of three main parts: the Monocular Pose Estimation Module that introduces a novel unsupervised method based on the CycleGAN architecture, the Monocular Depth Estimation Module that leverages the novel Zoe architecture, and the 3D Reconstruction Module which uses information from the previous models to create a coherent surgical map. The performance of the procedure was rigorously evaluated using three publicly available datasets (Hamlyn, EndoSLAM, and SCARED) and benchmarked against two state-of-the-art methods, EndoSFMLearner and EndoDepth. The integration of Zoe in the MDEM demonstrated superior performance compared to state-of-the-art depth estimation algorithms in endoscopy, whereas the novel approach in the MPEM exhibited competitive performance and the lowest inference time. The results showcase the robustness of our approach in laparoscopy, gastroscopy, and colonoscopy, three different scenarios in endoscopic surgery. The proposed SLAM approach has the potential to improve the accuracy and efficiency of endoscopic procedures by providing surgeons with enhanced depth perception and 3D reconstruction capabilities.

Title: Enhancing Complex Causality Extraction via Improved Subtask Interaction and Knowledge Fusion

• Authors: Jinglong Gao, Chen Lu, Xiao Ding, Zhongyang Li, Ting Liu, Bing Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03079
• Pdf URL: https://arxiv.org/pdf/2408.03079
• Copy Paste: [[2408.03079]] Enhancing Complex Causality Extraction via Improved Subtask Interaction and Knowledge Fusion(https://arxiv.org/abs/2408.03079)
• Keywords: extraction
• Abstract: Event Causality Extraction (ECE) aims at extracting causal event pairs from texts. Despite ChatGPT's recent success, fine-tuning small models remains the best approach for the ECE task. However, existing fine-tuning based ECE methods cannot address all three key challenges in ECE simultaneously: 1) Complex Causality Extraction, where multiple causal-effect pairs occur within a single sentence; 2) Subtask~ Interaction, which involves modeling the mutual dependence between the two subtasks of ECE, i.e., extracting events and identifying the causal relationship between extracted events; and 3) Knowledge Fusion, which requires effectively fusing the knowledge in two modalities, i.e., the expressive pretrained language models and the structured knowledge graphs. In this paper, we propose a unified ECE framework (UniCE to address all three issues in ECE simultaneously. Specifically, we design a subtask interaction mechanism to enable mutual interaction between the two ECE subtasks. Besides, we design a knowledge fusion mechanism to fuse knowledge in the two modalities. Furthermore, we employ separate decoders for each subtask to facilitate complex causality extraction. Experiments on three benchmark datasets demonstrate that our method achieves state-of-the-art performance and outperforms ChatGPT with a margin of at least 30% F1-score. More importantly, our model can also be used to effectively improve the ECE performance of ChatGPT via in-context learning.

Title: Research on Autonomous Driving Decision-making Strategies based Deep Reinforcement Learning

• Authors: Zixiang Wang, Hao Yan, Changsong Wei, Junyu Wang, Shi Bo, Minheng Xiao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03084
• Pdf URL: https://arxiv.org/pdf/2408.03084
• Copy Paste: [[2408.03084]] Research on Autonomous Driving Decision-making Strategies based Deep Reinforcement Learning(https://arxiv.org/abs/2408.03084)
• Keywords: robust
• Abstract: The behavior decision-making subsystem is a key component of the autonomous driving system, which reflects the decision-making ability of the vehicle and the driver, and is an important symbol of the high-level intelligence of the vehicle. However, the existing rule-based decision-making schemes are limited by the prior knowledge of designers, and it is difficult to cope with complex and changeable traffic scenarios. In this work, an advanced deep reinforcement learning model is adopted, which can autonomously learn and optimize driving strategies in a complex and changeable traffic environment by modeling the driving decision-making process as a reinforcement learning problem. Specifically, we used Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) for comparative experiments. DQN guides the agent to choose the best action by approximating the state-action value function, while PPO improves the decision-making quality by optimizing the policy function. We also introduce improvements in the design of the reward function to promote the robustness and adaptability of the model in real-world driving situations. Experimental results show that the decision-making strategy based on deep reinforcement learning has better performance than the traditional rule-based method in a variety of driving tasks.

Title: Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement

• Authors: Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03092
• Pdf URL: https://arxiv.org/pdf/2408.03092
• Copy Paste: [[2408.03092]] Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement(https://arxiv.org/abs/2408.03092)
• Keywords: large language model
• Abstract: Merging Large Language Models (LLMs) aims to amalgamate multiple homologous LLMs into one with all the capabilities. Ideally, any LLMs sharing the same backbone should be mergeable, irrespective of whether they are Fine-Tuned (FT) with minor parameter changes or Pre-Trained (PT) with substantial parameter shifts. However, existing methods often manually assign the model importance, rendering them feasible only for LLMs with similar parameter alterations, such as multiple FT LLMs. The diverse parameter changed ranges between FT and PT LLMs pose challenges for current solutions in empirically determining the optimal combination. In this paper, we make a pioneering effort to broaden the applicability of merging techniques from FT to PT LLMs. We initially examine the efficacy of current methods in merging FT and PT LLMs, discovering that they struggle to deal with PT LLMs. Subsequently, we introduce an approach based on WeIght DisENtanglement (WIDEN) to effectively extend the merging scope, which first disentangles model weights into magnitude and direction components, and then performs adaptive fusion by considering their respective contributions. In the experiments, we merge Qwen1.5-Chat (an FT LLM with instruction-following skills) with Sailor (a PT LLM with multilingual abilities) across 7B and 14B model scales. Results reveal that: (1) existing solutions usually fail when merging Sailor, either losing both abilities or only retaining instruction-following skills; (2) WIDEN successfully injects the multilingual abilities of Sailor into Qwen1.5-Chat and make it proficient in Southeast Asian languages, achieving enhancements in the fundamental capabilities. In light of previous research, we also merge multiple 13B FT LLMs and observe that WIDEN achieves a balanced amalgamation of instruction following, mathematical reasoning, and code generation skills.

Title: Learning Provably Robust Policies in Uncertain Parametric Environments

• Authors: Yannik Schnitzer, Alessandro Abate, David Parker
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.03093
• Pdf URL: https://arxiv.org/pdf/2408.03093
• Copy Paste: [[2408.03093]] Learning Provably Robust Policies in Uncertain Parametric Environments(https://arxiv.org/abs/2408.03093)
• Keywords: robust
• Abstract: We present a data-driven approach for learning MDP policies that are robust across stochastic environments whose transition probabilities are defined by parameters with an unknown distribution. We produce probably approximately correct (PAC) guarantees for the performance of these learned policies in a new, unseen environment over the unknown distribution. Our approach is based on finite samples of the MDP environments, for each of which we build an approximation of the model as an interval MDP, by exploring a set of generated trajectories. We use the built approximations to synthesise a single policy that performs well (meets given requirements) across the sampled environments, and furthermore bound its risk (of not meeting the given requirements) when deployed in an unseen environment. Our procedure offers a trade-off between the guaranteed performance of the learned policy and the risk of not meeting the guarantee in an unseen environment. Our approach exploits knowledge of the environment's state space and graph structure, and we show how additional knowledge of its parametric structure can be leveraged to optimize learning and to obtain tighter guarantees from less samples. We evaluate our approach on a diverse range of established benchmarks, demonstrating that we can generate highly performing and robust policies, along with guarantees that tightly quantify their performance and the associated risk.

Title: 500xCompressor: Generalized Prompt Compression for Large Language Models

• Authors: Zongqian Li, Yixuan Su, Nigel Collier
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03094
• Pdf URL: https://arxiv.org/pdf/2408.03094
• Copy Paste: [[2408.03094]] 500xCompressor: Generalized Prompt Compression for Large Language Models(https://arxiv.org/abs/2408.03094)
• Keywords: large language model
• Abstract: Prompt compression is crucial for enhancing inference speed, reducing costs, and improving user experience. However, current methods face challenges such as low compression ratios and potential data leakage during evaluation. To address these issues, we propose 500xCompressor, a method that compresses extensive natural language contexts into a minimum of one single special token. The 500xCompressor introduces approximately 0.3% additional parameters and achieves compression ratios ranging from 6x to 480x. It is designed to compress any text, answer various types of questions, and could be utilized by the original large language model (LLM) without requiring fine-tuning. Initially, 500xCompressor was pretrained on the Arxiv Corpus, followed by fine-tuning on the ArxivQA dataset, and subsequently evaluated on strictly unseen and classical question answering (QA) datasets. The results demonstrate that the LLM retained 62.26-72.89% of its capabilities compared to using non-compressed prompts. This study also shows that not all the compressed tokens are equally utilized and that K V values have significant advantages over embeddings in preserving information at high compression ratios. The highly compressive nature of natural language prompts, even for fine-grained complex information, suggests promising potential for future applications and further research into developing a new LLM language.

Title: Topic Modeling with Fine-tuning LLMs and Bag of Sentences

• Authors: Johannes Schneider
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03099
• Pdf URL: https://arxiv.org/pdf/2408.03099
• Copy Paste: [[2408.03099]] Topic Modeling with Fine-tuning LLMs and Bag of Sentences(https://arxiv.org/abs/2408.03099)
• Keywords: large language model
• Abstract: Large language models (LLM)'s are increasingly used for topic modeling outperforming classical topic models such as LDA. Commonly, pre-trained LLM encoders such as BERT are used out-of-the-box despite the fact that fine-tuning is known to improve LLMs considerably. The challenge lies in obtaining a suitable (labeled) dataset for fine-tuning. In this paper, we use the recent idea to use bag of sentences as the elementary unit in computing topics. In turn, we derive an approach FT-Topic to perform unsupervised fine-tuning relying primarily on two steps for constructing a training dataset in an automatic fashion. First, a heuristic method to identifies pairs of sentence groups that are either assumed to be of the same or different topics. Second, we remove sentence pairs that are likely labeled incorrectly. The dataset is then used to fine-tune an encoder LLM, which can be leveraged by any topic modeling approach using embeddings. However, in this work, we demonstrate its effectiveness by deriving a novel state-of-the-art topic modeling method called SenClu, which achieves fast inference through an expectation-maximization algorithm and hard assignments of sentence groups to a single topic, while giving users the possibility to encode prior knowledge on the topic-document distribution. Code is at \url{this https URL}

Title: Evaluating the Translation Performance of Large Language Models Based on Euas-20

• Authors: Yan Huang, Wei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03119
• Pdf URL: https://arxiv.org/pdf/2408.03119
• Copy Paste: [[2408.03119]] Evaluating the Translation Performance of Large Language Models Based on Euas-20(https://arxiv.org/abs/2408.03119)
• Keywords: large language model
• Abstract: In recent years, with the rapid development of deep learning technology, large language models (LLMs) such as BERT and GPT have achieved breakthrough results in natural language processing tasks. Machine translation (MT), as one of the core tasks of natural language processing, has also benefited from the development of large language models and achieved a qualitative leap. Despite the significant progress in translation performance achieved by large language models, machine translation still faces many challenges. Therefore, in this paper, we construct the dataset Euas-20 to evaluate the performance of large language models on translation tasks, the translation ability on different languages, and the effect of pre-training data on the translation ability of LLMs for researchers and developers.

Title: COMMENTATOR: A Code-mixed Multilingual Text Annotation Framework

• Authors: Rajvee Sheth, Shubh Nisar, Heenaben Prajapati, Himanshu Beniwal, Mayank Singh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03125
• Pdf URL: https://arxiv.org/pdf/2408.03125
• Copy Paste: [[2408.03125]] COMMENTATOR: A Code-mixed Multilingual Text Annotation Framework(https://arxiv.org/abs/2408.03125)
• Keywords: robust
• Abstract: As the NLP community increasingly addresses challenges associated with multilingualism, robust annotation tools are essential to handle multilingual datasets efficiently. In this paper, we introduce a code-mixed multilingual text annotation framework, COMMENTATOR, specifically designed for annotating code-mixed text. The tool demonstrates its effectiveness in token-level and sentence-level language annotation tasks for Hinglish text. We perform robust qualitative human-based evaluations to showcase COMMENTATOR led to 5x faster annotations than the best baseline. Our code is publicly available at \url{this https URL}. The demonstration video is available at \url{this https URL}.

Title: Lisbon Computational Linguists at SemEval-2024 Task 2: Using A Mistral 7B Model and Data Augmentation

• Authors: Artur Guimarães, Bruno Martins, João Magalhães
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03127
• Pdf URL: https://arxiv.org/pdf/2408.03127
• Copy Paste: [[2408.03127]] Lisbon Computational Linguists at SemEval-2024 Task 2: Using A Mistral 7B Model and Data Augmentation(https://arxiv.org/abs/2408.03127)
• Keywords: large language model
• Abstract: This paper describes our approach to the SemEval-2024 safe biomedical Natural Language Inference for Clinical Trials (NLI4CT) task, which concerns classifying statements about Clinical Trial Reports (CTRs). We explored the capabilities of Mistral-7B, a generalist open-source Large Language Model (LLM). We developed a prompt for the NLI4CT task, and fine-tuned a quantized version of the model using an augmented version of the training dataset. The experimental results show that this approach can produce notable results in terms of the macro F1-score, while having limitations in terms of faithfulness and consistency. All the developed code is publicly available on a GitHub repository

Title: Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations

• Authors: Leo Donisch, Sigurd Schacht, Carsten Lanquillon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03130
• Pdf URL: https://arxiv.org/pdf/2408.03130
• Copy Paste: [[2408.03130]] Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations(https://arxiv.org/abs/2408.03130)
• Keywords: large language model
• Abstract: Large language models are ubiquitous in natural language processing because they can adapt to new tasks without retraining. However, their sheer scale and complexity present unique challenges and opportunities, prompting researchers and practitioners to explore novel model training, optimization, and deployment methods. This literature review focuses on various techniques for reducing resource requirements and compressing large language models, including quantization, pruning, knowledge distillation, and architectural optimizations. The primary objective is to explore each method in-depth and highlight its unique challenges and practical applications. The discussed methods are categorized into a taxonomy that presents an overview of the optimization landscape and helps navigate it to understand the research trajectory better.

Title: Conditioning LLMs with Emotion in Neural Machine Translation

• Authors: Charles Brazier, Jean-Luc Rouas
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03150
• Pdf URL: https://arxiv.org/pdf/2408.03150
• Copy Paste: [[2408.03150]] Conditioning LLMs with Emotion in Neural Machine Translation(https://arxiv.org/abs/2408.03150)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown remarkable performance in Natural Language Processing tasks, including Machine Translation (MT). In this work, we propose a novel MT pipeline that integrates emotion information extracted from a Speech Emotion Recognition (SER) model into LLMs to enhance translation quality. We first fine-tune five existing LLMs on the Libri-trans dataset and select the most performant model. Subsequently, we augment LLM prompts with different dimensional emotions and train the selected LLM under these different configurations. Our experiments reveal that integrating emotion information, especially arousal, into LLM prompts leads to notable improvements in translation quality.

Title: Optimizing Disease Prediction with Artificial Intelligence Driven Feature Selection and Attention Networks

• Authors: D. Dhinakaran, S. Edwin Raja, M. Thiyagarajan, J. Jeno Jasmine, P. Raghavan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03151
• Pdf URL: https://arxiv.org/pdf/2408.03151
• Copy Paste: [[2408.03151]] Optimizing Disease Prediction with Artificial Intelligence Driven Feature Selection and Attention Networks(https://arxiv.org/abs/2408.03151)
• Keywords: robust
• Abstract: The rapid integration of machine learning methodologies in healthcare has ignited innovative strategies for disease prediction, particularly with the vast repositories of Electronic Health Records (EHR) data. This article delves into the realm of multi-disease prediction, presenting a comprehensive study that introduces a pioneering ensemble feature selection model. This model, designed to optimize learning systems, combines statistical, deep, and optimally selected features through the innovative Stabilized Energy Valley Optimization with Enhanced Bounds (SEV-EB) algorithm. The objective is to achieve unparalleled accuracy and stability in predicting various disorders. This work proposes an advanced ensemble model that synergistically integrates statistical, deep, and optimally selected features. This combination aims to enhance the predictive power of the model by capturing diverse aspects of the health data. At the heart of the proposed model lies the SEV-EB algorithm, a novel approach to optimal feature selection. The algorithm introduces enhanced bounds and stabilization techniques, contributing to the robustness and accuracy of the overall prediction model. To further elevate the predictive capabilities, an HSC-AttentionNet is introduced. This network architecture combines deep temporal convolution capabilities with LSTM, allowing the model to capture both short-term patterns and long-term dependencies in health data. Rigorous evaluations showcase the remarkable performance of the proposed model. Achieving a 95% accuracy and 94% F1-score in predicting various disorders, the model surpasses traditional methods, signifying a significant advancement in disease prediction accuracy. The implications of this research extend beyond the confines of academia.

Title: Iterative CT Reconstruction via Latent Variable Optimization of Shallow Diffusion Models

• Authors: Sho Ozaki, Shizuo Kaji, Toshikazu Imae, Kanabu Nawa, Hideomi Yamashita, Keiichi Nakagawa
• Subjects: cs.CV, cs.LG, physics.med-ph
• Abstract URL: https://arxiv.org/abs/2408.03156
• Pdf URL: https://arxiv.org/pdf/2408.03156
• Copy Paste: [[2408.03156]] Iterative CT Reconstruction via Latent Variable Optimization of Shallow Diffusion Models(https://arxiv.org/abs/2408.03156)
• Keywords: diffusion, generative
• Abstract: Image generative AI has garnered significant attention in recent years. In particular, the diffusion model, a core component of recent generative AI, produces high-quality images with rich diversity. In this study, we propose a novel CT reconstruction method by combining the denoising diffusion probabilistic model with iterative CT reconstruction. In sharp contrast to previous studies, we optimize the fidelity loss of CT reconstruction with respect to the latent variable of the diffusion model, instead of the image and model parameters. To suppress anatomical structure changes produced by the diffusion model, we shallow the diffusion and reverse processes, and fix a set of added noises in the reverse process to make it deterministic during inference. We demonstrate the effectiveness of the proposed method through sparse view CT reconstruction of 1/10 view projection data. Despite the simplicity of the implementation, the proposed method shows the capability of reconstructing high-quality images while preserving the patient's anatomical structure, and outperforms existing methods including iterative reconstruction, iterative reconstruction with total variation, and the diffusion model alone in terms of quantitative indices such as SSIM and PSNR. We also explore further sparse view CT using 1/20 view projection data with the same trained diffusion model. As the number of iterations increases, image quality improvement comparable to that of 1/10 sparse view CT reconstruction is achieved. In principle, the proposed method can be widely applied not only to CT but also to other imaging modalities such as MRI, PET, and SPECT.

Title: User-in-the-loop Evaluation of Multimodal LLMs for Activity Assistance

• Authors: Mrinal Verghese, Brian Chen, Hamid Eghbalzadeh, Tushar Nagarajan, Ruta Desai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03160
• Pdf URL: https://arxiv.org/pdf/2408.03160
• Copy Paste: [[2408.03160]] User-in-the-loop Evaluation of Multimodal LLMs for Activity Assistance(https://arxiv.org/abs/2408.03160)
• Keywords: large language model
• Abstract: Our research investigates the capability of modern multimodal reasoning models, powered by Large Language Models (LLMs), to facilitate vision-powered assistants for multi-step daily activities. Such assistants must be able to 1) encode relevant visual history from the assistant's sensors, e.g., camera, 2) forecast future actions for accomplishing the activity, and 3) replan based on the user in the loop. To evaluate the first two capabilities, grounding visual history and forecasting in short and long horizons, we conduct benchmarking of two prominent classes of multimodal LLM approaches -- Socratic Models and Vision Conditioned Language Models (VCLMs) on video-based action anticipation tasks using offline datasets. These offline benchmarks, however, do not allow us to close the loop with the user, which is essential to evaluate the replanning capabilities and measure successful activity completion in assistive scenarios. To that end, we conduct a first-of-its-kind user study, with 18 participants performing 3 different multi-step cooking activities while wearing an egocentric observation device called Aria and following assistance from multimodal LLMs. We find that the Socratic approach outperforms VCLMs in both offline and online settings. We further highlight how grounding long visual history, common in activity assistance, remains challenging in current models, especially for VCLMs, and demonstrate that offline metrics do not indicate online performance.

Title: Dilated Convolution with Learnable Spacings makes visual models more aligned with humans: a Grad-CAM study

• Authors: Rabih Chamas, Ismail Khalfaoui-Hassani, Timothee Masquelier
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03164
• Pdf URL: https://arxiv.org/pdf/2408.03164
• Copy Paste: [[2408.03164]] Dilated Convolution with Learnable Spacings makes visual models more aligned with humans: a Grad-CAM study(https://arxiv.org/abs/2408.03164)
• Keywords: interpretability
• Abstract: Dilated Convolution with Learnable Spacing (DCLS) is a recent advanced convolution method that allows enlarging the receptive fields (RF) without increasing the number of parameters, like the dilated convolution, yet without imposing a regular grid. DCLS has been shown to outperform the standard and dilated convolutions on several computer vision benchmarks. Here, we show that, in addition, DCLS increases the models' interpretability, defined as the alignment with human visual strategies. To quantify it, we use the Spearman correlation between the models' GradCAM heatmaps and the ClickMe dataset heatmaps, which reflect human visual attention. We took eight reference models - ResNet50, ConvNeXt (T, S and B), CAFormer, ConvFormer, and FastViT (sa 24 and 36) - and drop-in replaced the standard convolution layers with DCLS ones. This improved the interpretability score in seven of them. Moreover, we observed that Grad-CAM generated random heatmaps for two models in our study: CAFormer and ConvFormer models, leading to low interpretability scores. We addressed this issue by introducing Threshold-Grad-CAM, a modification built on top of Grad-CAM that enhanced interpretability across nearly all models. The code and checkpoints to reproduce this study are available at: this https URL.

Title: Leveraging Parameter Efficient Training Methods for Low Resource Text Classification: A Case Study in Marathi

• Authors: Pranita Deshmukh, Nikita Kulkarni, Sanhita Kulkarni, Kareena Manghani, Raviraj Joshi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03172
• Pdf URL: https://arxiv.org/pdf/2408.03172
• Copy Paste: [[2408.03172]] Leveraging Parameter Efficient Training Methods for Low Resource Text Classification: A Case Study in Marathi(https://arxiv.org/abs/2408.03172)
• Keywords: transformer, large language model
• Abstract: With the surge in digital content in low-resource languages, there is an escalating demand for advanced Natural Language Processing (NLP) techniques tailored to these languages. BERT (Bidirectional Encoder Representations from Transformers), serving as the foundational framework for numerous NLP architectures and language models, is increasingly employed for the development of low-resource NLP models. Parameter Efficient Fine-Tuning (PEFT) is a method for fine-tuning Large Language Models (LLMs) and reducing the training parameters to some extent to decrease the computational costs needed for training the model and achieve results comparable to a fully fine-tuned model. In this work, we present a study of PEFT methods for the Indic low-resource language Marathi. We conduct a comprehensive analysis of PEFT methods applied to various monolingual and multilingual Marathi BERT models. These approaches are evaluated on prominent text classification datasets like MahaSent, MahaHate, and MahaNews. The incorporation of PEFT techniques is demonstrated to significantly expedite the training speed of the models, addressing a critical aspect of model development and deployment. In this study, we explore Low-Rank Adaptation of Large Language Models (LoRA) and adapter methods for low-resource text classification. We show that these methods are competitive with full fine-tuning and can be used without loss in accuracy. This study contributes valuable insights into the effectiveness of Marathi BERT models, offering a foundation for the continued advancement of NLP capabilities in Marathi and similar Indic languages.

Title: An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion

• Authors: Xingguang Yan, Han-Hung Lee, Ziyu Wan, Angel X. Chang
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03178
• Pdf URL: https://arxiv.org/pdf/2408.03178
• Copy Paste: [[2408.03178]] An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion(https://arxiv.org/abs/2408.03178)
• Keywords: diffusion, transformer, generative
• Abstract: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

Title: MaskAnyone Toolkit: Offering Strategies for Minimizing Privacy Risks and Maximizing Utility in Audio-Visual Data Archiving

• Authors: Babajide Alamu Owoyele, Martin Schilling, Rohan Sawahn, Niklas Kaemer, Pavel Zherebenkov, Bhuvanesh Verma, Wim Pouw, Gerard de Melo
• Subjects: cs.CR, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.03185
• Pdf URL: https://arxiv.org/pdf/2408.03185
• Copy Paste: [[2408.03185]] MaskAnyone Toolkit: Offering Strategies for Minimizing Privacy Risks and Maximizing Utility in Audio-Visual Data Archiving(https://arxiv.org/abs/2408.03185)
• Keywords: privacy
• Abstract: This paper introduces MaskAnyone, a novel toolkit designed to navigate some privacy and ethical concerns of sharing audio-visual data in research. MaskAnyone offers a scalable, user-friendly solution for de-identifying individuals in video and audio content through face-swapping and voice alteration, supporting multi-person masking and real-time bulk processing. By integrating this tool within research practices, we aim to enhance data reproducibility and utility in social science research. Our approach draws on Design Science Research, proposing that MaskAnyone can facilitate safer data sharing and potentially reduce the storage of fully identifiable data. We discuss the development and capabilities of MaskAnyone, explore its integration into ethical research practices, and consider the broader implications of audio-visual data masking, including issues of consent and the risk of misuse. The paper concludes with a preliminary evaluation framework for assessing the effectiveness and ethical integration of masking tools in such research settings.

Title: Personalizing Federated Instrument Segmentation with Visual Trait Priors in Robotic Surgery

• Authors: Jialang Xu, Jiacheng Wang, Lequan Yu, Danail Stoyanov, Yueming Jin, Evangelos B. Mazomenos
• Subjects: cs.CV, cs.AI, cs.RO, physics.med-ph
• Abstract URL: https://arxiv.org/abs/2408.03208
• Pdf URL: https://arxiv.org/pdf/2408.03208
• Copy Paste: [[2408.03208]] Personalizing Federated Instrument Segmentation with Visual Trait Priors in Robotic Surgery(https://arxiv.org/abs/2408.03208)
• Keywords: privacy, federate, segmentation
• Abstract: Personalized federated learning (PFL) for surgical instrument segmentation (SIS) is a promising approach. It enables multiple clinical sites to collaboratively train a series of models in privacy, with each model tailored to the individual distribution of each site. Existing PFL methods rarely consider the personalization of multi-headed self-attention, and do not account for appearance diversity and instrument shape similarity, both inherent in surgical scenes. We thus propose PFedSIS, a novel PFL method with visual trait priors for SIS, incorporating global-personalized disentanglement (GPD), appearance-regulation personalized enhancement (APE), and shape-similarity global enhancement (SGE), to boost SIS performance in each site. GPD represents the first attempt at head-wise assignment for multi-headed self-attention personalization. To preserve the unique appearance representation of each site and gradually leverage the inter-site difference, APE introduces appearance regulation and provides customized layer-wise aggregation solutions via hypernetworks for each site's personalized parameters. The mutual shape information of instruments is maintained and shared via SGE, which enhances the cross-style shape consistency on the image level and computes the shape-similarity contribution of each site on the prediction level for updating the global parameters. PFedSIS outperforms state-of-the-art methods with +1.51% Dice, +2.11% IoU, -2.79 ASSD, -15.55 HD95 performance gains. The corresponding code and models will be released at this https URL.

Title: IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts

• Authors: Ciara Rowles, Shimon Vainer, Dante De Nigris, Slava Elizarov, Konstantin Kutsy, Simon Donné
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03209
• Pdf URL: https://arxiv.org/pdf/2408.03209
• Copy Paste: [[2408.03209]] IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts(https://arxiv.org/abs/2408.03209)
• Keywords: extraction, diffusion, generative
• Abstract: Diffusion models continuously push the boundary of state-of-the-art image generation, but the process is hard to control with any nuance: practice proves that textual prompts are inadequate for accurately describing image style or fine structural details (such as faces). ControlNet and IPAdapter address this shortcoming by conditioning the generative process on imagery instead, but each individual instance is limited to modeling a single conditional posterior: for practical use-cases, where multiple different posteriors are desired within the same workflow, training and using multiple adapters is cumbersome. We propose IPAdapter-Instruct, which combines natural-image conditioning with ``Instruct'' prompts to swap between interpretations for the same conditioning image: style transfer, object extraction, both, or something else still? IPAdapterInstruct efficiently learns multiple tasks with minimal loss in quality compared to dedicated per-task models.

Title: FedBAT: Communication-Efficient Federated Learning via Learnable Binarization

• Authors: Shiwei Li, Wenchao Xu, Haozhao Wang, Xing Tang, Yining Qi, Shijie Xu, Weihong Luo, Yuhua Li, Xiuqiang He, Ruixuan Li
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.03215
• Pdf URL: https://arxiv.org/pdf/2408.03215
• Copy Paste: [[2408.03215]] FedBAT: Communication-Efficient Federated Learning via Learnable Binarization(https://arxiv.org/abs/2408.03215)
• Keywords: privacy, federate
• Abstract: Federated learning is a promising distributed machine learning paradigm that can effectively exploit large-scale data without exposing users' privacy. However, it may incur significant communication overhead, thereby potentially impairing the training efficiency. To address this challenge, numerous studies suggest binarizing the model updates. Nonetheless, traditional methods usually binarize model updates in a post-training manner, resulting in significant approximation errors and consequent degradation in model accuracy. To this end, we propose Federated Binarization-Aware Training (FedBAT), a novel framework that directly learns binary model updates during the local training process, thus inherently reducing the approximation errors. FedBAT incorporates an innovative binarization operator, along with meticulously designed derivatives to facilitate efficient learning. In addition, we establish theoretical guarantees regarding the convergence of FedBAT. Extensive experiments are conducted on four popular datasets. The results show that FedBAT significantly accelerates the convergence and exceeds the accuracy of baselines by up to 9\%, even surpassing that of FedAvg in some cases.

Title: Learning to Learn without Forgetting using Attention

• Authors: Anna Vettoruzzo, Joaquin Vanschoren, Mohamed-Rafik Bouguelia, Thorsteinn Rögnvaldsson
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03219
• Pdf URL: https://arxiv.org/pdf/2408.03219
• Copy Paste: [[2408.03219]] Learning to Learn without Forgetting using Attention(https://arxiv.org/abs/2408.03219)
• Keywords: transformer
• Abstract: Continual learning (CL) refers to the ability to continually learn over time by accommodating new knowledge while retaining previously learned experience. While this concept is inherent in human learning, current machine learning methods are highly prone to overwrite previously learned patterns and thus forget past experience. Instead, model parameters should be updated selectively and carefully, avoiding unnecessary forgetting while optimally leveraging previously learned patterns to accelerate future learning. Since hand-crafting effective update mechanisms is difficult, we propose meta-learning a transformer-based optimizer to enhance CL. This meta-learned optimizer uses attention to learn the complex relationships between model parameters across a stream of tasks, and is designed to generate effective weight updates for the current task while preventing catastrophic forgetting on previously encountered tasks. Evaluations on benchmark datasets like SplitMNIST, RotatedMNIST, and SplitCIFAR-100 affirm the efficacy of the proposed approach in terms of both forward and backward transfer, even on small sets of labeled data, highlighting the advantages of integrating a meta-learned optimizer within the continual learning framework.

Title: Masked Random Noise for Communication Efficient Federaetd Learning

• Authors: Shiwei Li, Yingyi Cheng, Haozhao Wang, Xing Tang, Shijie Xu, Weihong Luo, Yuhua Li, Dugang Liu, Xiuqiang He, and Ruixuan Li
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.03220
• Pdf URL: https://arxiv.org/pdf/2408.03220
• Copy Paste: [[2408.03220]] Masked Random Noise for Communication Efficient Federaetd Learning(https://arxiv.org/abs/2408.03220)
• Keywords: privacy, federate
• Abstract: Federated learning is a promising distributed training paradigm that effectively safeguards data privacy. However, it may involve significant communication costs, which hinders training efficiency. In this paper, we aim to enhance communication efficiency from a new perspective. Specifically, we request the distributed clients to find optimal model updates relative to global model parameters within predefined random noise. For this purpose, we propose Federated Masked Random Noise (FedMRN), a novel framework that enables clients to learn a 1-bit mask for each model parameter and apply masked random noise (i.e., the Hadamard product of random noise and masks) to represent model updates. To make FedMRN feasible, we propose an advanced mask training strategy, called progressive stochastic masking (PSM). After local training, each client only need to transmit local masks and a random seed to the server. Additionally, we provide theoretical guarantees for the convergence of FedMRN under both strongly convex and non-convex assumptions. Extensive experiments are conducted on four popular datasets. The results show that FedMRN exhibits superior convergence speed and test accuracy compared to relevant baselines, while attaining a similar level of accuracy as FedAvg.

Title: Line-based 6-DoF Object Pose Estimation and Tracking With an Event Camera

• Authors: Zibin Liu, Banglei Guan, Yang Shang, Qifeng Yu, Laurent Kneip
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03225
• Pdf URL: https://arxiv.org/pdf/2408.03225
• Copy Paste: [[2408.03225]] Line-based 6-DoF Object Pose Estimation and Tracking With an Event Camera(https://arxiv.org/abs/2408.03225)
• Keywords: robust
• Abstract: Pose estimation and tracking of objects is a fundamental application in 3D vision. Event cameras possess remarkable attributes such as high dynamic range, low latency, and resilience against motion blur, which enables them to address challenging high dynamic range scenes or high-speed motion. These features make event cameras an ideal complement over standard cameras for object pose estimation. In this work, we propose a line-based robust pose estimation and tracking method for planar or non-planar objects using an event camera. Firstly, we extract object lines directly from events, then provide an initial pose using a globally-optimal Branch-and-Bound approach, where 2D-3D line correspondences are not known in advance. Subsequently, we utilize event-line matching to establish correspondences between 2D events and 3D models. Furthermore, object poses are refined and continuously tracked by minimizing event-line distances. Events are assigned different weights based on these distances, employing robust estimation algorithms. To evaluate the precision of the proposed methods in object pose estimation and tracking, we have devised and established an event-based moving object dataset. Compared against state-of-the-art methods, the robustness and accuracy of our methods have been validated both on synthetic experiments and the proposed dataset. The source code is available at this https URL.

Title: Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons

• Authors: Yifei Wang, Yuheng Chen, Wanting Wen, Yu Sheng, Linjing Li, Daniel Dajun Zeng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03247
• Pdf URL: https://arxiv.org/pdf/2408.03247
• Copy Paste: [[2408.03247]] Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons(https://arxiv.org/abs/2408.03247)
• Keywords: large language model
• Abstract: In this paper, we investigate whether Large Language Models (LLMs) actively recall or retrieve their internal repositories of factual knowledge when faced with reasoning tasks. Through an analysis of LLMs' internal factual recall at each reasoning step via Knowledge Neurons, we reveal that LLMs fail to harness the critical factual associations under certain circumstances. Instead, they tend to opt for alternative, shortcut-like pathways to answer reasoning questions. By manually manipulating the recall process of parametric knowledge in LLMs, we demonstrate that enhancing this recall process directly improves reasoning performance whereas suppressing it leads to notable degradation. Furthermore, we assess the effect of Chain-of-Thought (CoT) prompting, a powerful technique for addressing complex reasoning tasks. Our findings indicate that CoT can intensify the recall of factual knowledge by encouraging LLMs to engage in orderly and reliable reasoning. Furthermore, we explored how contextual conflicts affect the retrieval of facts during the reasoning process to gain a comprehensive understanding of the factual recall behaviors of LLMs. Code and data will be available soon.

Title: Synthesizing Text-to-SQL Data from Weak and Strong LLMs

• Authors: Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang, Junyang Lin, Chang Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03256
• Pdf URL: https://arxiv.org/pdf/2408.03256
• Copy Paste: [[2408.03256]] Synthesizing Text-to-SQL Data from Weak and Strong LLMs(https://arxiv.org/abs/2408.03256)
• Keywords: large language model
• Abstract: The capability gap between open-source and closed-source large language models (LLMs) remains a challenge in text-to-SQL tasks. In this paper, we introduce a synthetic data approach that combines data produced by larger, more powerful models (strong models) with error information data generated by smaller, not well-aligned models (weak models). The method not only enhances the domain generalization of text-to-SQL models but also explores the potential of error data supervision through preference learning. Furthermore, we employ the synthetic data approach for instruction tuning on open-source LLMs, resulting SENSE, a specialized text-to-SQL model. The effectiveness of SENSE is demonstrated through state-of-the-art results on the SPIDER and BIRD benchmarks, bridging the performance gap between open-source models and methods prompted by closed-source models.

Title: StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation

• Authors: Boxi Cao, Mengjie Ren, Hongyu Lin, Xianpei Han, Feng Zhang, Junfeng Zhan, Le Sun
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03281
• Pdf URL: https://arxiv.org/pdf/2408.03281
• Copy Paste: [[2408.03281]] StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation(https://arxiv.org/abs/2408.03281)
• Keywords: robust, large language model
• Abstract: Evaluation is the baton for the development of large language models. Current evaluations typically employ a single-item assessment paradigm for each atomic test objective, which struggles to discern whether a model genuinely possesses the required capabilities or merely memorizes/guesses the answers to specific questions. To this end, we propose a novel evaluation framework referred to as StructEval. Starting from an atomic test objective, StructEval deepens and broadens the evaluation by conducting a structured assessment across multiple cognitive levels and critical concepts, and therefore offers a comprehensive, robust and consistent evaluation for LLMs. Experiments on three widely-used benchmarks demonstrate that StructEval serves as a reliable tool for resisting the risk of data contamination and reducing the interference of potential biases, thereby providing more reliable and consistent conclusions regarding model capabilities. Our framework also sheds light on the design of future principled and trustworthy LLM evaluation protocols.

Title: AMES: Asymmetric and Memory-Efficient Similarity Estimation for Instance-level Retrieval

• Authors: Pavel Suma, Giorgos Kordopatis-Zilos, Ahmet Iscen, Giorgos Tolias
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03282
• Pdf URL: https://arxiv.org/pdf/2408.03282
• Copy Paste: [[2408.03282]] AMES: Asymmetric and Memory-Efficient Similarity Estimation for Instance-level Retrieval(https://arxiv.org/abs/2408.03282)
• Keywords: transformer
• Abstract: This work investigates the problem of instance-level image retrieval re-ranking with the constraint of memory efficiency, ultimately aiming to limit memory usage to 1KB per image. Departing from the prevalent focus on performance enhancements, this work prioritizes the crucial trade-off between performance and memory requirements. The proposed model uses a transformer-based architecture designed to estimate image-to-image similarity by capturing interactions within and across images based on their local descriptors. A distinctive property of the model is the capability for asymmetric similarity estimation. Database images are represented with a smaller number of descriptors compared to query images, enabling performance improvements without increasing memory consumption. To ensure adaptability across different applications, a universal model is introduced that adjusts to a varying number of local descriptors during the testing phase. Results on standard benchmarks demonstrate the superiority of our approach over both hand-crafted and learned models. In particular, compared with current state-of-the-art methods that overlook their memory footprint, our approach not only attains superior performance but does so with a significantly reduced memory footprint. The code and pretrained models are publicly available at: this https URL

Title: ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer

• Authors: Jiazhi Guan, Zhiliang Xu, Hang Zhou, Kaisiyuan Wang, Shengyi He, Zhanwang Zhang, Borong Liang, Haocheng Feng, Errui Ding, Jingtuo Liu, Jingdong Wang, Youjian Zhao, Ziwei Liu
• Subjects: cs.CV, cs.GR, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.03284
• Pdf URL: https://arxiv.org/pdf/2408.03284
• Copy Paste: [[2408.03284]] ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer(https://arxiv.org/abs/2408.03284)
• Keywords: transformer
• Abstract: Lip-syncing videos with given audio is the foundation for various applications including the creation of virtual presenters or performers. While recent studies explore high-fidelity lip-sync with different techniques, their task-orientated models either require long-term videos for clip-specific training or retain visible artifacts. In this paper, we propose a unified and effective framework ReSyncer, that synchronizes generalized audio-visual facial information. The key design is revisiting and rewiring the Style-based generator to efficiently adopt 3D facial dynamics predicted by a principled style-injected Transformer. By simply re-configuring the information insertion mechanisms within the noise and style space, our framework fuses motion and appearance with unified training. Extensive experiments demonstrate that ReSyncer not only produces high-fidelity lip-synced videos according to audio, but also supports multiple appealing properties that are suitable for creating virtual presenters and performers, including fast personalized fine-tuning, video-driven lip-syncing, the transfer of speaking styles, and even face swapping. Resources can be found at this https URL.

Title: Biomedical SAM 2: Segment Anything in Biomedical Images and Videos

• Authors: Zhiling Yan, Weixiang Sun, Rong Zhou, Zhengqing Yuan, Kai Zhang, Yiwei Li, Tianming Liu, Quanzheng Li, Xiang Li, Lifang He, Lichao Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03286
• Pdf URL: https://arxiv.org/pdf/2408.03286
• Copy Paste: [[2408.03286]] Biomedical SAM 2: Segment Anything in Biomedical Images and Videos(https://arxiv.org/abs/2408.03286)
• Keywords: segmentation
• Abstract: Medical image segmentation and video object segmentation are essential for diagnosing and analyzing diseases by identifying and measuring biological structures. Recent advances in natural domain have been driven by foundation models like the Segment Anything Model 2 (SAM 2). To explore the performance of SAM 2 in biomedical applications, we designed two evaluation pipelines for single-frame image segmentation and multi-frame video segmentation with varied prompt designs, revealing SAM 2's limitations in medical contexts. Consequently, we developed BioSAM 2, an enhanced foundation model optimized for biomedical data based on SAM 2. Our experiments show that BioSAM 2 not only surpasses the performance of existing state-of-the-art foundation models but also matches or even exceeds specialist models, demonstrating its efficacy and potential in the medical domain.

Title: Malicious Internet Entity Detection Using Local Graph Inference

• Authors: Simon Mandlik, Tomas Pevny, Vaclav Smidl, Lukas Bajer
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.03287
• Pdf URL: https://arxiv.org/pdf/2408.03287
• Copy Paste: [[2408.03287]] Malicious Internet Entity Detection Using Local Graph Inference(https://arxiv.org/abs/2408.03287)
• Keywords: security
• Abstract: Detection of malicious behavior in a large network is a challenging problem for machine learning in computer security, since it requires a model with high expressive power and scalable inference. Existing solutions struggle to achieve this feat -- current cybersec-tailored approaches are still limited in expressivity, and methods successful in other domains do not scale well for large volumes of data, rendering frequent retraining impossible. This work proposes a new perspective for learning from graph data that is modeling network entity interactions as a large heterogeneous graph. High expressivity of the method is achieved with neural network architecture HMILnet that naturally models this type of data and provides theoretical guarantees. The scalability is achieved by pursuing local graph inference, i.e., classifying individual vertices and their neighborhood as independent samples. Our experiments exhibit improvement over the state-of-the-art Probabilistic Threat Propagation (PTP) algorithm, show a further threefold accuracy improvement when additional data is used, which is not possible with the PTP algorithm, and demonstrate the generalization capabilities of the method to new, previously unseen entities.

Title: DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers

• Authors: Lianwei Yang, Haisong Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03291
• Pdf URL: https://arxiv.org/pdf/2408.03291
• Copy Paste: [[2408.03291]] DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers(https://arxiv.org/abs/2408.03291)
• Keywords: transformer
• Abstract: Vision transformers (ViTs) have garnered significant attention for their performance in vision tasks; however, the high computational cost and significant latency issues have hinder widespread adoption. Post-training quantization (PTQ), a promising method for model compression, still faces accuracy degradation challenges with ViTs. There are two reasons for this: the existing quantization paradigm does not fit the power-law distribution of post-Softmax activations well, and accuracy inevitably decreases after reparameterizing post-LayerNorm activations. We propose a Distribution-Friendly and Outlier-Aware Post-training Quantization method for Vision Transformers, named DopQ-ViT. DopQ-ViT analyzes the inefficiencies of current quantizers and introduces a distribution-friendly Tan Quantizer called TanQ. TanQ focuses more on values near 1, more accurately preserving the power-law distribution of post-Softmax activations, and achieves favorable results. Moreover, when reparameterizing post-LayerNorm activations from channel-wise to layer-wise quantization, the accuracy degradation is mainly due to the significant impact of outliers in the scaling factors. Therefore, DopQ-ViT proposes a method to Search for the Optimal Scaling Factor, denoted as SOSF, which compensates for the influence of outliers and preserves the performance of the quantization model. DopQ-ViT has undergone extensive validation and demonstrates significant performance improvements in quantization models, particularly in low-bit settings.

Title: Left of Fab: Securing Design and Collaboration in the Semiconductor Value Chain

• Authors: John C. Hoag
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.03295
• Pdf URL: https://arxiv.org/pdf/2408.03295
• Copy Paste: [[2408.03295]] Left of Fab: Securing Design and Collaboration in the Semiconductor Value Chain(https://arxiv.org/abs/2408.03295)
• Keywords: secure, security
• Abstract: The purpose of this paper is to fill a gap in the general understanding -- and academic scrutiny -- of current and emerging workflows for designing and fabricating integrated circuits. The approach is to compare the IC design workflow with that for printed circuit boards, then to discern a classification for threats. The need to define and secure workflows is amplified by both U.S. investment in the semiconductor manufacturing and market forces affecting GPU production for AI applications. The origin of this knowledge gap can be the proprietary nature of solution spaces, but it can be the lack of demand for teaching and learning for engineers and technicians in this domain. This paper presents a framework for understanding the security of design workflows in a vendor- and tool-agnostic way.

Title: KaPO: Knowledge-aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Models

• Authors: Ruizhe Zhang, Yongxin Xu, Yuzhen Xiao, Runchuan Zhu, Xinke Jiang, Xu Chu, Junfeng Zhao, Yasha Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03297
• Pdf URL: https://arxiv.org/pdf/2408.03297
• Copy Paste: [[2408.03297]] KaPO: Knowledge-aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Models(https://arxiv.org/abs/2408.03297)
• Keywords: robust, large language model
• Abstract: By integrating external knowledge, Retrieval-Augmented Generation (RAG) has become an effective strategy for mitigating the hallucination problems that large language models (LLMs) encounter when dealing with knowledge-intensive tasks. However, in the process of integrating external non-parametric supporting evidence with internal parametric knowledge, inevitable knowledge conflicts may arise, leading to confusion in the model's responses. To enhance the knowledge selection of LLMs in various contexts, some research has focused on refining their behavior patterns through instruction-tuning. Nonetheless, due to the absence of explicit negative signals and comparative objectives, models fine-tuned in this manner may still exhibit undesirable behaviors in the intricate and realistic retrieval scenarios. To this end, we propose a Knowledge-aware Preference Optimization, dubbed KaPO, aimed at achieving controllable knowledge selection in real retrieval scenarios. Concretely, we explore and simulate error types across diverse context combinations and learn how to avoid these negative signals through preference optimization methods. Simultaneously, by adjusting the balance between response length and the proportion of preference data representing different behavior patterns, we enhance the adherence capabilities and noise robustness of LLMs in a balanced manner. Experimental results show that KaPO outperforms previous methods for handling knowledge conflicts by over 37%, while also exhibiting robust generalization across various out-of-distribution datasets.

Title: TextIM: Part-aware Interactive Motion Synthesis from Text

• Authors: Siyuan Fan, Bo Du, Xiantao Cai, Bo Peng, Longling Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03302
• Pdf URL: https://arxiv.org/pdf/2408.03302
• Copy Paste: [[2408.03302]] TextIM: Part-aware Interactive Motion Synthesis from Text(https://arxiv.org/abs/2408.03302)
• Keywords: diffusion, large language model
• Abstract: In this work, we propose TextIM, a novel framework for synthesizing TEXT-driven human Interactive Motions, with a focus on the precise alignment of part-level semantics. Existing methods often overlook the critical roles of interactive body parts and fail to adequately capture and align part-level semantics, resulting in inaccuracies and even erroneous movement outcomes. To address these issues, TextIM utilizes a decoupled conditional diffusion framework to enhance the detailed alignment between interactive movements and corresponding semantic intents from textual descriptions. Our approach leverages large language models, functioning as a human brain, to identify interacting human body parts and to comprehend interaction semantics to generate complicated and subtle interactive motion. Guided by the refined movements of the interacting parts, TextIM further extends these movements into a coherent whole-body motion. We design a spatial coherence module to complement the entire body movements while maintaining consistency and harmony across body parts using a part graph convolutional network. For training and evaluation, we carefully selected and re-labeled interactive motions from HUMANML3D to develop a specialized dataset. Experimental results demonstrate that TextIM produces semantically accurate human interactive motions, significantly enhancing the realism and applicability of synthesized interactive motions in diverse scenarios, even including interactions with deformable and dynamically changing objects.

Title: Fusing Forces: Deep-Human-Guided Refinement of Segmentation Masks

• Authors: Rafael Sterzinger, Christian Stippel, Robert Sablatnig
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03304
• Pdf URL: https://arxiv.org/pdf/2408.03304
• Copy Paste: [[2408.03304]] Fusing Forces: Deep-Human-Guided Refinement of Segmentation Masks(https://arxiv.org/abs/2408.03304)
• Keywords: segmentation
• Abstract: Etruscan mirrors constitute a significant category in Etruscan art, characterized by elaborate figurative illustrations featured on their backside. A laborious and costly aspect of their analysis and documentation is the task of manually tracing these illustrations. In previous work, a methodology has been proposed to automate this process, involving photometric-stereo scanning in combination with deep neural networks. While achieving quantitative performance akin to an expert annotator, some results still lack qualitative precision and, thus, require annotators for inspection and potential correction, maintaining resource intensity. In response, we propose a deep neural network trained to interactively refine existing annotations based on human guidance. Our human-in-the-loop approach streamlines annotation, achieving equal quality with up to 75% less manual input required. Moreover, during the refinement process, the relative improvement of our methodology over pure manual labeling reaches peak values of up to 26%, attaining drastically better quality quicker. By being tailored to the complex task of segmenting intricate lines, specifically distinguishing it from previous methods, our approach offers drastic improvements in efficacy, transferable to a broad spectrum of applications beyond Etruscan mirrors.

Title: MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation

• Authors: Xiaofeng Mao, Zhengkai Jiang, Qilin Wang, Chencan Fu, Jiangning Zhang, Jiafu Wu, Yabiao Wang, Chengjie Wang, Wei Li, Mingmin Chi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.03312
• Pdf URL: https://arxiv.org/pdf/2408.03312
• Copy Paste: [[2408.03312]] MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation(https://arxiv.org/abs/2408.03312)
• Keywords: diffusion, transformer
• Abstract: Recent advancements in the field of Diffusion Transformers have substantially improved the generation of high-quality 2D images, 3D videos, and 3D shapes. However, the effectiveness of the Transformer architecture in the domain of co-speech gesture generation remains relatively unexplored, as prior methodologies have predominantly employed the Convolutional Neural Network (CNNs) or simple a few transformer layers. In an attempt to bridge this research gap, we introduce a novel Masked Diffusion Transformer for co-speech gesture generation, referred to as MDT-A2G, which directly implements the denoising process on gesture sequences. To enhance the contextual reasoning capability of temporally aligned speech-driven gestures, we incorporate a novel Masked Diffusion Transformer. This model employs a mask modeling scheme specifically designed to strengthen temporal relation learning among sequence gestures, thereby expediting the learning process and leading to coherent and realistic motions. Apart from audio, Our MDT-A2G model also integrates multi-modal information, encompassing text, emotion, and identity. Furthermore, we propose an efficient inference strategy that diminishes the denoising computation by leveraging previously calculated results, thereby achieving a speedup with negligible performance degradation. Experimental results demonstrate that MDT-A2G excels in gesture generation, boasting a learning speed that is over 6$\times$ faster than traditional diffusion transformers and an inference speed that is 5.7$\times$ than the standard diffusion model.