2024-10-29

Title: Adaptive Real-Time Multi-Loss Function Optimization Using Dynamic Memory Fusion Framework: A Case Study on Breast Cancer Segmentation

• Authors: Amin Golnari, Mostafa Diba
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19745
• Pdf URL: https://arxiv.org/pdf/2410.19745
• Copy Paste: [[2410.19745]] Adaptive Real-Time Multi-Loss Function Optimization Using Dynamic Memory Fusion Framework: A Case Study on Breast Cancer Segmentation(https://arxiv.org/abs/2410.19745)
• Keywords: segmentation
• Abstract: Deep learning has proven to be a highly effective tool for a wide range of applications, significantly when leveraging the power of multi-loss functions to optimize performance on multiple criteria simultaneously. However, optimal selection and weighting loss functions in deep learning tasks can significantly influence model performance, yet manual tuning of these functions is often inefficient and inflexible. We propose a novel framework called dynamic memory fusion for adaptive multi-loss function penalizing in real-time to address this. This framework leverages historical loss values data to dynamically adjust the weighting of multiple loss functions throughout the training process. Additionally, this framework integrates an auxiliary loss function to enhance model performance in the early stages. To further research horizons, we introduce the class-balanced dice loss function, designed to address class imbalance by prioritizing underrepresented classes. Experiments on breast ultrasound datasets demonstrate that the framework improves segmentation performance across various metrics. These results demonstrate the effectiveness of our proposed framework in ensuring that the model dynamically adjusts its focus to prioritize the most relevant criteria, leading to improved performance in evolving environments. The source code for our proposed methodology is publicly available on GitHub.

Title: C^2DA: Contrastive and Context-aware Domain Adaptive Semantic Segmentation

• Authors: Md. Al-Masrur Khan, Zheng Chen, Lantao Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19748
• Pdf URL: https://arxiv.org/pdf/2410.19748
• Copy Paste: [[2410.19748]] C^2DA: Contrastive and Context-aware Domain Adaptive Semantic Segmentation(https://arxiv.org/abs/2410.19748)
• Keywords: robust, segmentation
• Abstract: Unsupervised domain adaptive semantic segmentation (UDA-SS) aims to train a model on the source domain data (e.g., synthetic) and adapt the model to predict target domain data (e.g., real-world) without accessing target annotation data. Most existing UDA-SS methods only focus on inter-domain knowledge to mitigate the data-shift problem. However, learning the inherent structure of the images and exploring the intrinsic pixel distribution of both domains are ignored, which prevents the UDA-SS methods from producing satisfactory performance like supervised learning. Moreover, incorporating contextual knowledge is also often overlooked. Considering these issues, in this work, we propose a UDA-SS framework that learns both intra-domain and context-aware knowledge. To learn the intra-domain knowledge, we incorporate contrastive loss in both domains, which pulls pixels of similar classes together and pushes the rest away, facilitating intra-image-pixel-wise correlations. To learn context-aware knowledge, we modify the mixing technique by leveraging contextual dependency among the classes. Moreover, we adapt the Mask Image Modeling (MIM) technique to properly use context clues for robust visual recognition, using limited information about the masked images. Comprehensive experiments validate that our proposed method improves the state-of-the-art UDA-SS methods by a margin of 0.51% mIoU and 0.54% mIoU in the adaptation of GTA-V->Cityscapes and Synthia->Cityscapes, respectively. We open-source our C2DA code. Code link: this http URL

Title: A SAM based Tool for Semi-Automatic Food Annotation

• Authors: Lubnaa Abdur Rahman, Ioannis Papathanail, Lorenzo Brigato, Stavroula Mougiakakou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19756
• Pdf URL: https://arxiv.org/pdf/2410.19756
• Copy Paste: [[2410.19756]] A SAM based Tool for Semi-Automatic Food Annotation(https://arxiv.org/abs/2410.19756)
• Keywords: segmentation
• Abstract: The advancement of artificial intelligence (AI) in food and nutrition research is hindered by a critical bottleneck: the lack of annotated food data. Despite the rise of highly efficient AI models designed for tasks such as food segmentation and classification, their practical application might necessitate proficiency in AI and machine learning principles, which can act as a challenge for non-AI experts in the field of nutritional sciences. Alternatively, it highlights the need to translate AI models into user-friendly tools that are accessible to all. To address this, we present a demo of a semi-automatic food image annotation tool leveraging the Segment Anything Model (SAM). The tool enables prompt-based food segmentation via user interactions, promoting user engagement and allowing them to further categorise food items within meal images and specify weight/volume if necessary. Additionally, we release a fine-tuned version of SAM's mask decoder, dubbed MealSAM, with the ViT-B backbone tailored specifically for food image segmentation. Our objective is not only to contribute to the field by encouraging participation, collaboration, and the gathering of more annotated food data but also to make AI technology available for a broader audience by translating AI into practical tools.

Title: Movie Trailer Genre Classification Using Multimodal Pretrained Features

• Authors: Serkan Sulun, Paula Viana, Matthew E. P. Davies
• Subjects: cs.CV, cs.AI, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.19760
• Pdf URL: https://arxiv.org/pdf/2410.19760
• Copy Paste: [[2410.19760]] Movie Trailer Genre Classification Using Multimodal Pretrained Features(https://arxiv.org/abs/2410.19760)
• Keywords: transformer
• Abstract: We introduce a novel method for movie genre classification, capitalizing on a diverse set of readily accessible pretrained models. These models extract high-level features related to visual scenery, objects, characters, text, speech, music, and audio effects. To intelligently fuse these pretrained features, we train small classifier models with low time and memory requirements. Employing the transformer model, our approach utilizes all video and audio frames of movie trailers without performing any temporal pooling, efficiently exploiting the correspondence between all elements, as opposed to the fixed and low number of frames typically used by traditional methods. Our approach fuses features originating from different tasks and modalities, with different dimensionalities, different temporal lengths, and complex dependencies as opposed to current approaches. Our method outperforms state-of-the-art movie genre classification models in terms of precision, recall, and mean average precision (mAP). To foster future research, we make the pretrained features for the entire MovieNet dataset, along with our genre classification code and the trained models, publicly available.

Title: Reliable, Routable, and Reproducible: Collection of Pedestrian Pathways at Statewide Scale

• Authors: Yuxiang Zhang, Bill Howe, Anat Caspi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19762
• Pdf URL: https://arxiv.org/pdf/2410.19762
• Copy Paste: [[2410.19762]] Reliable, Routable, and Reproducible: Collection of Pedestrian Pathways at Statewide Scale(https://arxiv.org/abs/2410.19762)
• Keywords: robust, segmentation
• Abstract: While advances in mobility technology including autonomous vehicles and multi-modal navigation systems can improve mobility equity for people with disabilities, these technologies depend crucially on accurate, standardized, and complete pedestrian path networks. Ad hoc collection efforts lead to a data record that is sparse, unreliable, and non-interoperable. This paper presents a sociotechnical methodology to collect, manage, serve, and maintain pedestrian path data at a statewide scale. Combining the automation afforded by computer-vision approaches applied to aerial imagery and existing road network data with the quality control afforded by interactive tools, we aim to produce routable pedestrian pathways for the entire State of Washington within approximately two years. We extract paths, crossings, and curb ramps at scale from aerial imagery, integrating multi-input segmentation methods with road topology data to ensure connected, routable networks. We then organize the predictions into project regions selected for their value to the public interest, where each project region is divided into intersection-scale tasks. These tasks are assigned and tracked through an interactive tool that manages concurrency, progress, feedback, and data management. We demonstrate that our automated systems outperform state-of-the-art methods in producing routable pathway networks, which then significantly reduces the time required for human vetting. Our results demonstrate the feasibility of yielding accurate, robust pedestrian pathway networks at the scale of an entire state. This paper intends to inform procedures for national-scale ADA compliance by providing pedestrian equity, safety, and accessibility, and improving urban environments for all users.

Title: A New Perspective to Boost Performance Fairness for Medical Federated Learning

• Authors: Yunlu Yan, Lei Zhu, Yuexiang Li, Xinxing Xu, Rick Siow Mong Goh, Yong Liu, Salman Khan, Chun-Mei Feng
• Subjects: cs.LG, cs.CR, cs.CY, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.19765
• Pdf URL: https://arxiv.org/pdf/2410.19765
• Copy Paste: [[2410.19765]] A New Perspective to Boost Performance Fairness for Medical Federated Learning(https://arxiv.org/abs/2410.19765)
• Keywords: federate, fair, segmentation
• Abstract: Improving the fairness of federated learning (FL) benefits healthy and sustainable collaboration, especially for medical applications. However, existing fair FL methods ignore the specific characteristics of medical FL applications, i.e., domain shift among the datasets from different hospitals. In this work, we propose Fed-LWR to improve performance fairness from the perspective of feature shift, a key issue influencing the performance of medical FL systems caused by domain shift. Specifically, we dynamically perceive the bias of the global model across all hospitals by estimating the layer-wise difference in feature representations between local and global models. To minimize global divergence, we assign higher weights to hospitals with larger differences. The estimated client weights help us to re-aggregate the local models per layer to obtain a fairer global model. We evaluate our method on two widely used federated medical image segmentation benchmarks. The results demonstrate that our method achieves better and fairer performance compared with several state-of-the-art fair FL methods.

Title: Data-Driven Uncertainty-Aware Forecasting of Sea Ice Conditions in the Gulf of Ob Based on Satellite Radar Imagery

• Authors: Stefan Maria Ailuro, Anna Nedorubova, Timofey Grigoryev, Evgeny Burnaev, Vladimir Vanovskiy
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19782
• Pdf URL: https://arxiv.org/pdf/2410.19782
• Copy Paste: [[2410.19782]] Data-Driven Uncertainty-Aware Forecasting of Sea Ice Conditions in the Gulf of Ob Based on Satellite Radar Imagery(https://arxiv.org/abs/2410.19782)
• Keywords: robust
• Abstract: The increase in Arctic marine activity due to rapid warming and significant sea ice loss necessitates highly reliable, short-term sea ice forecasts to ensure maritime safety and operational efficiency. In this work, we present a novel data-driven approach for sea ice condition forecasting in the Gulf of Ob, leveraging sequences of radar images from Sentinel-1, weather observations, and GLORYS forecasts. Our approach integrates advanced video prediction models, originally developed for vision tasks, with domain-specific data preprocessing and augmentation techniques tailored to the unique challenges of Arctic sea ice dynamics. Central to our methodology is the use of uncertainty quantification to assess the reliability of predictions, ensuring robust decision-making in safety-critical applications. Furthermore, we propose a confidence-based model mixture mechanism that enhances forecast accuracy and model robustness, crucial for reliable operations in volatile Arctic environments. Our results demonstrate substantial improvements over baseline approaches, underscoring the importance of uncertainty quantification and specialized data handling for effective and safe operations and reliable forecasting.

Title: How to Backdoor Consistency Models?

• Authors: Chengen Wang, Murat Kantarcioglu
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19785
• Pdf URL: https://arxiv.org/pdf/2410.19785
• Copy Paste: [[2410.19785]] How to Backdoor Consistency Models?(https://arxiv.org/abs/2410.19785)
• Keywords: attack, robust, diffusion
• Abstract: Consistency models are a new class of models that generate images by directly mapping noise to data, allowing for one-step generation and significantly accelerating the sampling process. However, their robustness against adversarial attacks has not yet been thoroughly investigated. In this work, we conduct the first study on the vulnerability of consistency models to backdoor attacks. While previous research has explored backdoor attacks on diffusion models, these studies have primarily focused on conventional diffusion models, employing a customized backdoor training process and objective, whereas consistency models have distinct training processes and objectives. Our proposed framework demonstrates the vulnerability of consistency models to backdoor attacks. During image generation, poisoned consistency models produce images with a Fréchet Inception Distance (FID) comparable to that of a clean model when sampling from Gaussian noise. However, once the trigger is activated, they generate backdoor target images. We explore various trigger and target configurations to evaluate the vulnerability of consistency models, including the use of random noise as a trigger. This type of trigger is less conspicuous and aligns well with the sampling process of consistency models. Across all configurations, our framework successfully compromises the consistency models while maintaining high utility and specificity.

Title: DiffGAN: A Test Generation Approach for Differential Testing of Deep Neural Networks

• Authors: Zohreh Aghababaeyan, Manel Abdellatif, Lionel Briand, Ramesh S
• Subjects: cs.CV, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.19794
• Pdf URL: https://arxiv.org/pdf/2410.19794
• Copy Paste: [[2410.19794]] DiffGAN: A Test Generation Approach for Differential Testing of Deep Neural Networks(https://arxiv.org/abs/2410.19794)
• Keywords: generative
• Abstract: Deep Neural Networks (DNNs) are increasingly deployed across applications. However, ensuring their reliability remains a challenge, and in many situations, alternative models with similar functionality and accuracy are available. Traditional accuracy-based evaluations often fail to capture behavioral differences between models, especially with limited test datasets, making it difficult to select or combine models effectively. Differential testing addresses this by generating test inputs that expose discrepancies in DNN model behavior. However, existing approaches face significant limitations: many rely on model internals or are constrained by available seed inputs. To address these challenges, we propose DiffGAN, a black-box test image generation approach for differential testing of DNN models. DiffGAN leverages a Generative Adversarial Network (GAN) and the Non-dominated Sorting Genetic Algorithm II to generate diverse and valid triggering inputs that reveal behavioral discrepancies between models. DiffGAN employs two custom fitness functions, focusing on diversity and divergence, to guide the exploration of the GAN input space and identify discrepancies between models' outputs. By strategically searching this space, DiffGAN generates inputs with specific features that trigger differences in model behavior. DiffGAN is black-box, making it applicable in more situations. We evaluate DiffGAN on eight DNN model pairs trained on widely used image datasets. Our results show DiffGAN significantly outperforms a SOTA baseline, generating four times more triggering inputs, with greater diversity and validity, within the same budget. Additionally, the generated inputs improve the accuracy of a machine learning-based model selection mechanism, which selects the best-performing model based on input characteristics and can serve as a smart output voting mechanism when using alternative models.

Title: Feature Clipping for Uncertainty Calibration

• Authors: Linwei Tao, Minjing Dong, Chang Xu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19796
• Pdf URL: https://arxiv.org/pdf/2410.19796
• Copy Paste: [[2410.19796]] Feature Clipping for Uncertainty Calibration(https://arxiv.org/abs/2410.19796)
• Keywords: transformer
• Abstract: Deep neural networks (DNNs) have achieved significant success across various tasks, but ensuring reliable uncertainty estimates, known as model calibration, is crucial for their safe and effective deployment. Modern DNNs often suffer from overconfidence, leading to miscalibration. We propose a novel post-hoc calibration method called feature clipping (FC) to address this issue. FC involves clipping feature values to a specified threshold, effectively increasing entropy in high calibration error samples while maintaining the information in low calibration error samples. This process reduces the overconfidence in predictions, improving the overall calibration of the model. Our extensive experiments on datasets such as CIFAR-10, CIFAR-100, and ImageNet, and models including CNNs and transformers, demonstrate that FC consistently enhances calibration performance. Additionally, we provide a theoretical analysis that validates the effectiveness of our method. As the first calibration technique based on feature modification, feature clipping offers a novel approach to improving model calibration, showing significant improvements over both post-hoc and train-time calibration methods and pioneering a new avenue for feature-based model calibration.

Title: Stable Diffusion with Continuous-time Neural Network

• Authors: Andras Horvath
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19798
• Pdf URL: https://arxiv.org/pdf/2410.19798
• Copy Paste: [[2410.19798]] Stable Diffusion with Continuous-time Neural Network(https://arxiv.org/abs/2410.19798)
• Keywords: diffusion, transformer
• Abstract: Stable diffusion models have ushered in a new era of advancements in image generation, currently reigning as the state-of-the-art approach, exhibiting unparalleled performance. The process of diffusion, accompanied by denoising through iterative convolutional or transformer network steps, stands at the core of their implementation. Neural networks operating in continuous time naturally embrace the concept of diffusion, this way they could enable more accurate and energy efficient implementation. Within the confines of this paper, my focus delves into an exploration and demonstration of the potential of celllular neural networks in image generation. I will demonstrate their superiority in performance, showcasing their adeptness in producing higher quality images and achieving quicker training times in comparison to their discrete-time counterparts on the commonly cited MNIST dataset.

Title: Stochastic Flow Matching for Resolving Small-Scale Physics

• Authors: Stathi Fotiadis, Noah Brenowitz, Tomas Geffner, Yair Cohen, Michael Pritchard, Arash Vahdat, Morteza Mardani
• Subjects: cs.CV, physics.ao-ph, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.19814
• Pdf URL: https://arxiv.org/pdf/2410.19814
• Copy Paste: [[2410.19814]] Stochastic Flow Matching for Resolving Small-Scale Physics(https://arxiv.org/abs/2410.19814)
• Keywords: diffusion
• Abstract: Conditioning diffusion and flow models have proven effective for super-resolving small-scale details in natural this http URL, in physical sciences such as weather, super-resolving small-scale details poses significant challenges due to: (i) misalignment between input and output distributions (i.e., solutions to distinct partial differential equations (PDEs) follow different trajectories), (ii) multi-scale dynamics, deterministic dynamics at large scales vs. stochastic at small scales, and (iii) limited data, increasing the risk of overfitting. To address these challenges, we propose encoding the inputs to a latent base distribution that is closer to the target distribution, followed by flow matching to generate small-scale physics. The encoder captures the deterministic components, while flow matching adds stochastic small-scale details. To account for uncertainty in the deterministic part, we inject noise into the encoder output using an adaptive noise scaling mechanism, which is dynamically adjusted based on maximum-likelihood estimates of the encoder predictions. We conduct extensive experiments on both the real-world CWA weather dataset and the PDE-based Kolmogorov dataset, with the CWA task involving super-resolving the weather variables for the region of Taiwan from 25 km to 2 km scales. Our results show that the proposed stochastic flow matching (SFM) framework significantly outperforms existing methods such as conditional diffusion and flows.

Title: Explainable AI in Handwriting Detection for Dyslexia Using Transfer Learning

• Authors: Mahmoud Robaa, Mazen Balat, Rewaa Awaad, Esraa Omar, Salah A. Aly
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19821
• Pdf URL: https://arxiv.org/pdf/2410.19821
• Copy Paste: [[2410.19821]] Explainable AI in Handwriting Detection for Dyslexia Using Transfer Learning(https://arxiv.org/abs/2410.19821)
• Keywords: interpretability, transformer
• Abstract: Dyslexia is one of the most common learning disorders, often characterized by distinct features in handwriting. Early detection is essential for effective intervention. In this paper, we propose an explainable AI (XAI) framework for dyslexia detection through handwriting analysis, utilizing transfer learning and transformer-based models. Our approach surpasses state-of-the-art methods, achieving a test accuracy of 0.9958, while ensuring model interpretability through Grad-CAM visualizations that highlight the critical handwriting features influencing model decisions. The main contributions of this work include the integration of XAI for enhanced interpretability, adaptation to diverse languages and writing systems, and demonstration of the method's global applicability. This framework not only improves diagnostic accuracy but also fosters trust and understanding among educators, clinicians, and parents, supporting earlier diagnoses and the development of personalized educational strategies.

Title: Flame quality monitoring of flare stack based on deep visual features

• Authors: Xing Mu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19823
• Pdf URL: https://arxiv.org/pdf/2410.19823
• Copy Paste: [[2410.19823]] Flame quality monitoring of flare stack based on deep visual features(https://arxiv.org/abs/2410.19823)
• Keywords: protect, segmentation
• Abstract: Flare stacks play an important role in the treatment of waste gas and waste materials in petroleum fossil energy plants. Monitoring the efficiency of flame combustion is of great significance for environmental protection. The traditional method of monitoring with sensors is not only expensive, but also easily damaged in harsh combustion environments. In this paper, we propose to monitor the quality of flames using only visual features, including the area ratio of flame to smoke, RGB information of flames, angle of flames and other features. Comprehensive use of image segmentation, target detection, target tracking, principal component analysis, GPT-4 and other methods or tools to complete this task. In the end, real-time monitoring of the picture can be achieved, and when the combustion efficiency is low, measures such as adjusting the ratio of air and waste can be taken in time. As far as we know, the method of this paper is relatively innovative and has industrial production value.

Title: Automating Video Thumbnails Selection and Generation with Multimodal and Multistage Analysis

• Authors: Elia Fantini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19825
• Pdf URL: https://arxiv.org/pdf/2410.19825
• Copy Paste: [[2410.19825]] Automating Video Thumbnails Selection and Generation with Multimodal and Multistage Analysis(https://arxiv.org/abs/2410.19825)
• Keywords: diffusion, transformer, large language model, segmentation
• Abstract: This thesis presents an innovative approach to automate video thumbnail selection for traditional broadcast content. Our methodology establishes stringent criteria for diverse, representative, and aesthetically pleasing thumbnails, considering factors like logo placement space, incorporation of vertical aspect ratios, and accurate recognition of facial identities and emotions. We introduce a sophisticated multistage pipeline that can select candidate frames or generate novel images by blending video elements or using diffusion models. The pipeline incorporates state-of-the-art models for various tasks, including downsampling, redundancy reduction, automated cropping, face recognition, closed-eye and emotion detection, shot scale and aesthetic prediction, segmentation, matting, and harmonization. It also leverages large language models and visual transformers for semantic consistency. A GUI tool facilitates rapid navigation of the pipeline's output. To evaluate our method, we conducted comprehensive experiments. In a study of 69 videos, 53.6% of our proposed sets included thumbnails chosen by professional designers, with 73.9% containing similar images. A survey of 82 participants showed a 45.77% preference for our method, compared to 37.99% for manually chosen thumbnails and 16.36% for an alternative method. Professional designers reported a 3.57-fold increase in valid candidates compared to the alternative method, confirming that our approach meets established criteria. In conclusion, our findings affirm that the proposed method accelerates thumbnail creation while maintaining high-quality standards and fostering greater user engagement.

Title: GNNRL-Smoothing: A Prior-Free Reinforcement Learning Model for Mesh Smoothing

• Authors: Zhichao Wang, Xinhai Chen, Chunye Gong, Bo Yang, Liang Deng, Yufei Sun, Yufei Pang, Jie Liu
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19834
• Pdf URL: https://arxiv.org/pdf/2410.19834
• Copy Paste: [[2410.19834]] GNNRL-Smoothing: A Prior-Free Reinforcement Learning Model for Mesh Smoothing(https://arxiv.org/abs/2410.19834)
• Keywords: robust
• Abstract: Mesh smoothing methods can enhance mesh quality by eliminating distorted elements, leading to improved convergence in simulations. To balance the efficiency and robustness of traditional mesh smoothing process, previous approaches have employed supervised learning and reinforcement learning to train intelligent smoothing models. However, these methods heavily rely on labeled dataset or prior knowledge to guide the models' learning. Furthermore, their limited capacity to enhance mesh connectivity often restricts the effectiveness of smoothing. In this paper, we first systematically analyze the learning mechanisms of recent intelligent smoothing methods and propose a prior-free reinforcement learning model for intelligent mesh smoothing. Our proposed model integrates graph neural networks with reinforcement learning to implement an intelligent node smoothing agent and introduces, for the first time, a mesh connectivity improvement agent. We formalize mesh optimization as a Markov Decision Process and successfully train both agents using Twin Delayed Deep Deterministic Policy Gradient and Double Dueling Deep Q-Network in the absence of any prior data or knowledge. We verified the proposed model on both 2D and 3D meshes. Experimental results demonstrate that our model achieves feature-preserving smoothing on complex 3D surface meshes. It also achieves state-of-the-art results among intelligent smoothing methods on 2D meshes and is 7.16 times faster than traditional optimization-based smoothing methods. Moreover, the connectivity improvement agent can effectively enhance the quality distribution of the mesh.

Title: Upsampling DINOv2 features for unsupervised vision tasks and weakly supervised materials segmentation

• Authors: Ronan Docherty, Antonis Vamvakeros, Samuel J. Cooper
• Subjects: cs.CV, cond-mat.mtrl-sci, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.19836
• Pdf URL: https://arxiv.org/pdf/2410.19836
• Copy Paste: [[2410.19836]] Upsampling DINOv2 features for unsupervised vision tasks and weakly supervised materials segmentation(https://arxiv.org/abs/2410.19836)
• Keywords: transformer, segmentation
• Abstract: The features of self-supervised vision transformers (ViTs) contain strong semantic and positional information relevant to downstream tasks like object localization and segmentation. Recent works combine these features with traditional methods like clustering, graph partitioning or region correlations to achieve impressive baselines without finetuning or training additional networks. We leverage upsampled features from ViT networks (e.g DINOv2) in two workflows: in a clustering based approach for object localization and segmentation, and paired with standard classifiers in weakly supervised materials segmentation. Both show strong performance on benchmarks, especially in weakly supervised segmentation where the ViT features capture complex relationships inaccessible to classical approaches. We expect the flexibility and generalizability of these features will both speed up and strengthen materials characterization, from segmentation to property-prediction.

Title: Scene-Segmentation-Based Exposure Compensation for Tone Mapping of High Dynamic Range Scenes

• Authors: Yuma Kinoshita, Hitoshi Kiya
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.19839
• Pdf URL: https://arxiv.org/pdf/2410.19839
• Copy Paste: [[2410.19839]] Scene-Segmentation-Based Exposure Compensation for Tone Mapping of High Dynamic Range Scenes(https://arxiv.org/abs/2410.19839)
• Keywords: segmentation
• Abstract: We propose a novel scene-segmentation-based exposure compensation method for multi-exposure image fusion (MEF) based tone mapping. The aim of MEF-based tone mapping is to display high dynamic range (HDR) images on devices with limited dynamic range. To achieve this, this method generates a stack of differently exposed images from an input HDR image and fuses them into a single image. Our approach addresses the limitations of MEF-based tone mapping with existing segmentation-based exposure compensation, which often result in visually unappealing outcomes due to inappropriate exposure value selection. The proposed exposure compensation method first segments the input HDR image into subregions based on luminance values of pixels. It then determines exposure values for multi-exposure images to maximize contrast between regions while preserving relative luminance relationships. This approach contrasts with conventional methods that may invert luminance relationships or compromise contrast between regions. Additionally, we present an improved technique for calculating fusion weights to better reflect the effects of exposure compensation in the final fused image. In a simulation experiment to evaluate the quality of tone-mapped images, the MEF-based tone mapping with the proposed method outperforms three typical tone mapping methods including conventional MEF-based one, in terms of the tone mapped image quality index (TMQI).

Title: Enhancing Trust and Safety in Digital Payments: An LLM-Powered Approach

• Authors: Devendra Dahiphale, Naveen Madiraju, Justin Lin, Rutvik Karve, Monu Agrawal, Anant Modwal, Ramanan Balakrishnan, Shanay Shah, Govind Kaushal, Priya Mandawat, Prakash Hariramani, Arif Merchant (Google, Inc)
• Subjects: cs.CR, cs.AI, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19845
• Pdf URL: https://arxiv.org/pdf/2410.19845
• Copy Paste: [[2410.19845]] Enhancing Trust and Safety in Digital Payments: An LLM-Powered Approach(https://arxiv.org/abs/2410.19845)
• Keywords: secure, robust, large language model
• Abstract: Digital payment systems have revolutionized financial transactions, offering unparalleled convenience and accessibility to users worldwide. However, the increasing popularity of these platforms has also attracted malicious actors seeking to exploit their vulnerabilities for financial gain. To address this challenge, robust and adaptable scam detection mechanisms are crucial for maintaining the trust and safety of digital payment ecosystems. This paper presents a comprehensive approach to scam detection, focusing on the Unified Payments Interface (UPI) in India, Google Pay (GPay) as a specific use case. The approach leverages Large Language Models (LLMs) to enhance scam classification accuracy and designs a digital assistant to aid human reviewers in identifying and mitigating fraudulent activities. The results demonstrate the potential of LLMs in augmenting existing machine learning models and improving the efficiency, accuracy, quality, and consistency of scam reviews, ultimately contributing to a safer and more secure digital payment landscape. Our evaluation of the Gemini Ultra model on curated transaction data showed a 93.33% accuracy in scam classification. Furthermore, the model demonstrated 89% accuracy in generating reasoning for these classifications. A promising fact, the model identified 32% new accurate reasons for suspected scams that human reviewers had not included in the review notes.

Title: YOLO11 and Vision Transformers based 3D Pose Estimation of Immature Green Fruits in Commercial Apple Orchards for Robotic Thinning

• Authors: Ranjan Sapkota, Manoj Karkee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19846
• Pdf URL: https://arxiv.org/pdf/2410.19846
• Copy Paste: [[2410.19846]] YOLO11 and Vision Transformers based 3D Pose Estimation of Immature Green Fruits in Commercial Apple Orchards for Robotic Thinning(https://arxiv.org/abs/2410.19846)
• Keywords: robust, transformer
• Abstract: In this study, a robust method for 3D pose estimation of immature green apples (fruitlets) in commercial orchards was developed, utilizing the YOLO11 object detection and pose estimation algorithm alongside Vision Transformers (ViT) for depth estimation (Dense Prediction Transformer (DPT) and Depth Anything V2). For object detection and pose estimation, performance comparisons of YOLO11 (YOLO11n, YOLO11s, YOLO11m, YOLO11l and YOLO11x) and YOLOv8 (YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l and YOLOv8x) were made under identical hyperparameter settings among the all configurations. It was observed that YOLO11n surpassed all configurations of YOLO11 and YOLOv8 in terms of box precision and pose precision, achieving scores of 0.91 and 0.915, respectively. Conversely, YOLOv8n exhibited the highest box and pose recall scores of 0.905 and 0.925, respectively. Regarding the mean average precision at 50\% intersection over union (mAP@50), YOLO11s led all configurations with a box mAP@50 score of 0.94, while YOLOv8n achieved the highest pose mAP@50 score of 0.96. In terms of image processing speed, YOLO11n outperformed all configurations with an impressive inference speed of 2.7 ms, significantly faster than the quickest YOLOv8 configuration, YOLOv8n, which processed images in 7.8 ms. Subsequent integration of ViTs for the green fruit's pose depth estimation revealed that Depth Anything V2 outperformed Dense Prediction Transformer in 3D pose length validation, achieving the lowest Root Mean Square Error (RMSE) of 1.52 and Mean Absolute Error (MAE) of 1.28, demonstrating exceptional precision in estimating immature green fruit lengths. Integration of YOLO11 and Depth Anything Model provides a promising solution to 3D pose estimation of immature green fruits for robotic thinning applications.

Title: AEPL: Automated and Editable Prompt Learning for Brain Tumor Segmentation

• Authors: Yongheng Sun, Mingxia Liu, Chunfeng Lian
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19847
• Pdf URL: https://arxiv.org/pdf/2410.19847
• Copy Paste: [[2410.19847]] AEPL: Automated and Editable Prompt Learning for Brain Tumor Segmentation(https://arxiv.org/abs/2410.19847)
• Keywords: segmentation
• Abstract: Brain tumor segmentation is crucial for accurate diagnosisand treatment planning, but the small size and irregular shapeof tumors pose significant challenges. Existing methods of-ten fail to effectively incorporate medical domain knowledgesuch as tumor grade, which correlates with tumor aggres-siveness and morphology, providing critical insights for moreaccurate detection of tumor subregions during this http URL propose an Automated and Editable Prompt Learning(AEPL) framework that integrates tumor grade into the seg-mentation process by combining multi-task learning andprompt learning with automatic and editable prompt gen-eration. Specifically, AEPL employs an encoder to extractimage features for both tumor-grade prediction and segmen-tation mask generation. The predicted tumor grades serveas auto-generated prompts, guiding the decoder to produceprecise segmentation masks. This eliminates the need formanual prompts while allowing clinicians to manually editthe auto-generated prompts to fine-tune the segmentation,enhancing both flexibility and precision. The proposed AEPLachieves state-of-the-art performance on the BraTS 2018dataset, demonstrating its effectiveness and clinical this http URL source code can be accessed online.

Title: Benchmarking Large Language Models for Image Classification of Marine Mammals

• Authors: Yijiashun Qi, Shuzhang Cai, Zunduo Zhao, Jiaming Li, Yanbin Lin, Zhiqiang Wang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.19848
• Pdf URL: https://arxiv.org/pdf/2410.19848
• Copy Paste: [[2410.19848]] Benchmarking Large Language Models for Image Classification of Marine Mammals(https://arxiv.org/abs/2410.19848)
• Keywords: large language model
• Abstract: As Artificial Intelligence (AI) has developed rapidly over the past few decades, the new generation of AI, Large Language Models (LLMs) trained on massive datasets, has achieved ground-breaking performance in many applications. Further progress has been made in multimodal LLMs, with many datasets created to evaluate LLMs with vision abilities. However, none of those datasets focuses solely on marine mammals, which are indispensable for ecological equilibrium. In this work, we build a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level. Moreover, we evaluate several approaches for classifying these marine mammals: (1) machine learning (ML) algorithms using embeddings provided by neural networks, (2) influential pre-trained neural networks, (3) zero-shot models: CLIP and LLMs, and (4) a novel LLM-based multi-agent system (MAS). The results demonstrate the strengths of traditional models and LLMs in different aspects, and the MAS can further improve the classification performance. The dataset is available on GitHub: this https URL.

Title: Deep Learning and Machine Learning -- Python Data Structures and Mathematics Fundamental: From Theory to Practice

• Authors: Silin Chen, Ziqian Bi, Junyu Liu, Benji Peng, Sen Zhang, Xuanhe Pan, Jiawei Xu, Jinlang Wang, Keyu Chen, Caitlyn Heqi Yin, Pohsun Feng, Yizhu Wen, Tianyang Wang, Ming Li, Jintao Ren, Qian Niu, Ming Liu
• Subjects: cs.LG, cs.DS, cs.PL
• Abstract URL: https://arxiv.org/abs/2410.19849
• Pdf URL: https://arxiv.org/pdf/2410.19849
• Copy Paste: [[2410.19849]] Deep Learning and Machine Learning -- Python Data Structures and Mathematics Fundamental: From Theory to Practice(https://arxiv.org/abs/2410.19849)
• Keywords: large language model
• Abstract: This book provides a comprehensive introduction to the foundational concepts of machine learning (ML) and deep learning (DL). It bridges the gap between theoretical mathematics and practical application, focusing on Python as the primary programming language for implementing key algorithms and data structures. The book covers a wide range of topics, including basic and advanced Python programming, fundamental mathematical operations, matrix operations, linear algebra, and optimization techniques crucial for training ML and DL models. Advanced subjects like neural networks, optimization algorithms, and frequency domain methods are also explored, along with real-world applications of large language models (LLMs) and artificial intelligence (AI) in big data management. Designed for both beginners and advanced learners, the book emphasizes the critical role of mathematical principles in developing scalable AI solutions. Practical examples and Python code are provided throughout, ensuring readers gain hands-on experience in applying theoretical knowledge to solve complex problems in ML, DL, and big data analytics.

Title: Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts

• Authors: Sheryl Paul, Jyotirmoy V. Deshmukh
• Subjects: cs.LG, cs.AI, cs.GT, cs.NE
• Abstract URL: https://arxiv.org/abs/2410.19852
• Pdf URL: https://arxiv.org/pdf/2410.19852
• Copy Paste: [[2410.19852]] Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts(https://arxiv.org/abs/2410.19852)
• Keywords: robust, fair
• Abstract: Reinforcement learning (RL) has been successfully applied to solve the problem of finding obstacle-free paths for autonomous agents operating in stochastic and uncertain environments. However, when the underlying stochastic dynamics of the environment experiences drastic distribution shifts, the optimal policy obtained in the trained environment may be sub-optimal or may entirely fail in helping find goal-reaching paths for the agent. Approaches like domain randomization and robust RL can provide robust policies, but typically assume minor (bounded) distribution shifts. For substantial distribution shifts, retraining (either with a warm-start policy or from scratch) is an alternative approach. In this paper, we develop a novel approach called {\em Evolutionary Robust Policy Optimization} (ERPO), an adaptive re-training algorithm inspired by evolutionary game theory (EGT). ERPO learns an optimal policy for the shifted environment iteratively using a temperature parameter that controls the trade off between exploration and adherence to the old optimal policy. The policy update itself is an instantiation of the replicator dynamics used in EGT. We show that under fairly common sparsity assumptions on rewards in such environments, ERPO converges to the optimal policy in the shifted environment. We empirically demonstrate that for path finding tasks in a number of environments, ERPO outperforms several popular RL and deep RL algorithms (PPO, A3C, DQN) in many scenarios and popular environments. This includes scenarios where the RL algorithms are allowed to train from scratch in the new environment, when they are retrained on the new environment, or when they are used in conjunction with domain randomization. ERPO shows faster policy adaptation, higher average rewards, and reduced computational costs in policy adaptation.

Title: Enhancing Deep Learning based RMT Data Inversion using Gaussian Random Field

• Authors: Koustav Ghosal, Arun Singh, Samir Malakar, Shalivahan Srivastava, Deepak Gupta
• Subjects: cs.LG, cs.CE, eess.SP, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2410.19858
• Pdf URL: https://arxiv.org/pdf/2410.19858
• Copy Paste: [[2410.19858]] Enhancing Deep Learning based RMT Data Inversion using Gaussian Random Field(https://arxiv.org/abs/2410.19858)
• Keywords: robust
• Abstract: Deep learning (DL) methods have emerged as a powerful tool for the inversion of geophysical data. When applied to field data, these models often struggle without additional fine-tuning of the network. This is because they are built on the assumption that the statistical patterns in the training and test datasets are the same. To address this, we propose a DL-based inversion scheme for Radio Magnetotelluric data where the subsurface resistivity models are generated using Gaussian Random Fields (GRF). The network's generalization ability was tested with an out-of-distribution (OOD) dataset comprising a homogeneous background and various rectangular-shaped anomalous bodies. After end-to-end training with the GRF dataset, the pre-trained network successfully identified anomalies in the OOD dataset. Synthetic experiments confirmed that the GRF dataset enhances generalization compared to a homogeneous background OOD dataset. The network accurately recovered structures in a checkerboard resistivity model, and demonstrated robustness to noise, outperforming traditional gradient-based methods. Finally, the developed scheme is tested using exemplary field data from a waste site near Roorkee, India. The proposed scheme enhances generalization in a data-driven supervised learning framework, suggesting a promising direction for OOD generalization in DL methods.

Title: Real-Time Weapon Detection Using YOLOv8 for Enhanced Safety

• Authors: Ayush Thakur, Akshat Shrivastav, Rohan Sharma, Triyank Kumar, Kabir Puri
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19862
• Pdf URL: https://arxiv.org/pdf/2410.19862
• Copy Paste: [[2410.19862]] Real-Time Weapon Detection Using YOLOv8 for Enhanced Safety(https://arxiv.org/abs/2410.19862)
• Keywords: security, protect, robust
• Abstract: This research paper presents the development of an AI model utilizing YOLOv8 for real-time weapon detection, aimed at enhancing safety in public spaces such as schools, airports, and public transportation systems. As incidents of violence continue to rise globally, there is an urgent need for effective surveillance technologies that can quickly identify potential threats. Our approach focuses on leveraging advanced deep learning techniques to create a highly accurate and efficient system capable of detecting weapons in real-time video streams. The model was trained on a comprehensive dataset containing thousands of images depicting various types of firearms and edged weapons, ensuring a robust learning process. We evaluated the model's performance using key metrics such as precision, recall, F1-score, and mean Average Precision (mAP) across multiple Intersection over Union (IoU) thresholds, revealing a significant capability to differentiate between weapon and non-weapon classes with minimal error. Furthermore, we assessed the system's operational efficiency, demonstrating that it can process frames at high speeds suitable for real-time applications. The findings indicate that our YOLOv8-based weapon detection model not only contributes to the existing body of knowledge in computer vision but also addresses critical societal needs for improved safety measures in vulnerable environments. By harnessing the power of artificial intelligence, this research lays the groundwork for developing practical solutions that can be deployed in security settings, ultimately enhancing the protective capabilities of law enforcement and public safety agencies.

Title: Breaking the Illusion: Real-world Challenges for Adversarial Patches in Object Detection

• Authors: Jakob Shack, Katarina Petrovic, Olga Saukh
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19863
• Pdf URL: https://arxiv.org/pdf/2410.19863
• Copy Paste: [[2410.19863]] Breaking the Illusion: Real-world Challenges for Adversarial Patches in Object Detection(https://arxiv.org/abs/2410.19863)
• Keywords: defense, attack, robust
• Abstract: Adversarial attacks pose a significant threat to the robustness and reliability of machine learning systems, particularly in computer vision applications. This study investigates the performance of adversarial patches for the YOLO object detection network in the physical world. Two attacks were tested: a patch designed to be placed anywhere within the scene - global patch, and another patch intended to partially overlap with specific object targeted for removal from detection - local patch. Various factors such as patch size, position, rotation, brightness, and hue were analyzed to understand their impact on the effectiveness of the adversarial patches. The results reveal a notable dependency on these parameters, highlighting the challenges in maintaining attack efficacy in real-world conditions. Learning to align digitally applied transformation parameters with those measured in the real world still results in up to a 64\% discrepancy in patch performance. These findings underscore the importance of understanding environmental influences on adversarial attacks, which can inform the development of more robust defenses for practical machine learning applications.

Title: Comparing YOLO11 and YOLOv8 for instance segmentation of occluded and non-occluded immature green fruits in complex orchard environment

• Authors: Ranjan Sapkota, Manoj Karkee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19869
• Pdf URL: https://arxiv.org/pdf/2410.19869
• Copy Paste: [[2410.19869]] Comparing YOLO11 and YOLOv8 for instance segmentation of occluded and non-occluded immature green fruits in complex orchard environment(https://arxiv.org/abs/2410.19869)
• Keywords: segmentation
• Abstract: This study conducted a comprehensive performance evaluation on YOLO11 and YOLOv8, the latest in the "You Only Look Once" (YOLO) series, focusing on their instance segmentation capabilities for immature green apples in orchard environments. YOLO11n-seg achieved the highest mask precision across all categories with a notable score of 0.831, highlighting its effectiveness in fruit detection. YOLO11m-seg and YOLO11l-seg excelled in non-occluded and occluded fruitlet segmentation with scores of 0.851 and 0.829, respectively. Additionally, YOLO11x-seg led in mask recall for all categories, achieving a score of 0.815, with YOLO11m-seg performing best for non-occluded immature green fruitlets at 0.858 and YOLOv8x-seg leading the occluded category with 0.800. In terms of mean average precision at a 50\% intersection over union (mAP@50), YOLO11m-seg consistently outperformed, registering the highest scores for both box and mask segmentation, at 0.876 and 0.860 for the "All" class and 0.908 and 0.909 for non-occluded immature fruitlets, respectively. YOLO11l-seg and YOLOv8l-seg shared the top box mAP@50 for occluded immature fruitlets at 0.847, while YOLO11m-seg achieved the highest mask mAP@50 of 0.810. Despite the advancements in YOLO11, YOLOv8n surpassed its counterparts in image processing speed, with an impressive inference speed of 3.3 milliseconds, compared to the fastest YOLO11 series model at 4.8 milliseconds, underscoring its suitability for real-time agricultural applications related to complex green fruit environments.

Title: Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey

• Authors: Kun Shi, Shibo He, Zhenyu Shi, Anjun Chen, Zehui Xiong, Jiming Chen, Jun Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19872
• Pdf URL: https://arxiv.org/pdf/2410.19872
• Copy Paste: [[2410.19872]] Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey(https://arxiv.org/abs/2410.19872)
• Keywords: robust
• Abstract: Multi-modal fusion is imperative to the implementation of reliable object detection and tracking in complex environments. Exploiting the synergy of heterogeneous modal information endows perception systems the ability to achieve more comprehensive, robust, and accurate performance. As a nucleus concern in wireless-vision collaboration, radar-camera fusion has prompted prospective research directions owing to its extensive applicability, complementarity, and compatibility. Nonetheless, there still lacks a systematic survey specifically focusing on deep fusion of radar and camera for object detection and tracking. To fill this void, we embark on an endeavor to comprehensively review radar-camera fusion in a holistic way. First, we elaborate on the fundamental principles, methodologies, and applications of radar-camera fusion perception. Next, we delve into the key techniques concerning sensor calibration, modal representation, data alignment, and fusion operation. Furthermore, we provide a detailed taxonomy covering the research topics related to object detection and tracking in the context of radar and camera this http URL, we discuss the emerging perspectives in the field of radar-camera fusion perception and highlight the potential areas for future research.

Title: Paved or unpaved? A Deep Learning derived Road Surface Global Dataset from Mapillary Street-View Imagery

• Authors: Sukanya Randhawa, Eren Aygun, Guntaj Randhawa, Benjamin Herfort, Sven Lautenbach, Alexander Zipf
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19874
• Pdf URL: https://arxiv.org/pdf/2410.19874
• Copy Paste: [[2410.19874]] Paved or unpaved? A Deep Learning derived Road Surface Global Dataset from Mapillary Street-View Imagery(https://arxiv.org/abs/2410.19874)
• Keywords: transformer, segmentation
• Abstract: We have released an open dataset with global coverage on road surface characteristics (paved or unpaved) derived utilising 105 million images from the world's largest crowdsourcing-based street view platform, Mapillary, leveraging state-of-the-art geospatial AI methods. We propose a hybrid deep learning approach which combines SWIN-Transformer based road surface prediction and CLIP-and-DL segmentation based thresholding for filtering of bad quality images. The road surface prediction results have been matched and integrated with OpenStreetMap (OSM) road geometries. This study provides global data insights derived from maps and statistics about spatial distribution of Mapillary coverage and road pavedness on a continent and countries scale, with rural and urban this http URL dataset expands the availability of global road surface information by over 3 million kilometers, now representing approximately 36% of the total length of the global road this http URL regions showed moderate to high paved road coverage (60-80%), but significant gaps were noted in specific areas of Africa and Asia. Urban areas tend to have near-complete paved coverage, while rural regions display more variability. Model validation against OSM surface data achieved strong performance, with F1 scores for paved roads between 91-97% across this http URL forward the work of Mapillary and their contributors and enrichment of OSM road attributes, our work provides valuable insights for applications in urban planning, disaster routing, logistics optimisation and addresses various Sustainable Development Goals (SDGS): especially SDGs 1 (No poverty), 3 (Good health and well-being), 8 (Decent work and economic growth), 9 (Industry, Innovation and Infrastructure), 11 (Sustainable cities and communities), 12 (Responsible consumption and production), and 13 (Climate action).

Title: A Survey of AI-Generated Video Evaluation

• Authors: Xiao Liu, Xinhao Xiang, Zizhong Li, Yongheng Wang, Zhuoheng Li, Zhuosheng Liu, Weidi Zhang, Weiqi Ye, Jiawei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19884
• Pdf URL: https://arxiv.org/pdf/2410.19884
• Copy Paste: [[2410.19884]] A Survey of AI-Generated Video Evaluation(https://arxiv.org/abs/2410.19884)
• Keywords: robust
• Abstract: The growing capabilities of AI in generating video content have brought forward significant challenges in effectively evaluating these videos. Unlike static images or text, video content involves complex spatial and temporal dynamics which may require a more comprehensive and systematic evaluation of its contents in aspects like video presentation quality, semantic information delivery, alignment with human intentions, and the virtual-reality consistency with our physical world. This survey identifies the emerging field of AI-Generated Video Evaluation (AIGVE), highlighting the importance of assessing how well AI-generated videos align with human perception and meet specific instructions. We provide a structured analysis of existing methodologies that could be potentially used to evaluate AI-generated videos. By outlining the strengths and gaps in current approaches, we advocate for the development of more robust and nuanced evaluation frameworks that can handle the complexities of video content, which include not only the conventional metric-based evaluations, but also the current human-involved evaluations, and the future model-centered evaluations. This survey aims to establish a foundational knowledge base for both researchers from academia and practitioners from the industry, facilitating the future advancement of evaluation methods for AI-generated video content.

Title: Topology-aware Mamba for Crack Segmentation in Structures

• Authors: Xin Zuo, Yu Sheng, Jifeng Shen, Yongwei Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19894
• Pdf URL: https://arxiv.org/pdf/2410.19894
• Copy Paste: [[2410.19894]] Topology-aware Mamba for Crack Segmentation in Structures(https://arxiv.org/abs/2410.19894)
• Keywords: extraction, transformer, segmentation
• Abstract: CrackMamba, a Mamba-based model, is designed for efficient and accurate crack segmentation for monitoring the structural health of infrastructure. Traditional Convolutional Neural Network (CNN) models struggle with limited receptive fields, and while Vision Transformers (ViT) improve segmentation accuracy, they are computationally intensive. CrackMamba addresses these challenges by utilizing the VMambaV2 with pre-trained ImageNet-1k weights as the encoder and a newly designed decoder for better performance. To handle the random and complex nature of crack development, a Snake Scan module is proposed to reshape crack feature sequences, enhancing feature extraction. Additionally, the three-branch Snake Conv VSS (SCVSS) block is proposed to target cracks more effectively. Experiments show that CrackMamba achieves state-of-the-art (SOTA) performance on the CrackSeg9k and SewerCrack datasets, and demonstrates competitive performance on the retinal vessel segmentation dataset CHASE\underline{~}DB1, highlighting its generalization capability. The code is publicly available at: {this https URL.}

Title: A Review of Deep Learning Approaches for Non-Invasive Cognitive Impairment Detection

• Authors: Muath Alsuhaibani, Ali Pourramezan Fard, Jian Sun, Farida Far Poor, Peter S. Pressman, Mohammad H. Mahoor
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19898
• Pdf URL: https://arxiv.org/pdf/2410.19898
• Copy Paste: [[2410.19898]] A Review of Deep Learning Approaches for Non-Invasive Cognitive Impairment Detection(https://arxiv.org/abs/2410.19898)
• Keywords: explainability
• Abstract: This review paper explores recent advances in deep learning approaches for non-invasive cognitive impairment detection. We examine various non-invasive indicators of cognitive decline, including speech and language, facial, and motoric mobility. The paper provides an overview of relevant datasets, feature-extracting techniques, and deep-learning architectures applied to this domain. We have analyzed the performance of different methods across modalities and observed that speech and language-based methods generally achieved the highest detection performance. Studies combining acoustic and linguistic features tended to outperform those using a single modality. Facial analysis methods showed promise for visual modalities but were less extensively studied. Most papers focused on binary classification (impaired vs. non-impaired), with fewer addressing multi-class or regression tasks. Transfer learning and pre-trained language models emerged as popular and effective techniques, especially for linguistic analysis. Despite significant progress, several challenges remain, including data standardization and accessibility, model explainability, longitudinal analysis limitations, and clinical adaptation. Lastly, we propose future research directions, such as investigating language-agnostic speech analysis methods, developing multi-modal diagnostic systems, and addressing ethical considerations in AI-assisted healthcare. By synthesizing current trends and identifying key obstacles, this review aims to guide further development of deep learning-based cognitive impairment detection systems to improve early diagnosis and ultimately patient outcomes.

Title: Collaborative Inference over Wireless Channels with Feature Differential Privacy

• Authors: Mohamed Seif, Yuqi Nie, Andrea J. Goldsmith, H. Vincent Poor
• Subjects: cs.CR, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19917
• Pdf URL: https://arxiv.org/pdf/2410.19917
• Copy Paste: [[2410.19917]] Collaborative Inference over Wireless Channels with Feature Differential Privacy(https://arxiv.org/abs/2410.19917)
• Keywords: secure, privacy, extraction
• Abstract: Collaborative inference among multiple wireless edge devices has the potential to significantly enhance Artificial Intelligence (AI) applications, particularly for sensing and computer vision. This approach typically involves a three-stage process: a) data acquisition through sensing, b) feature extraction, and c) feature encoding for transmission. However, transmitting the extracted features poses a significant privacy risk, as sensitive personal data can be exposed during the process. To address this challenge, we propose a novel privacy-preserving collaborative inference mechanism, wherein each edge device in the network secures the privacy of extracted features before transmitting them to a central server for inference. Our approach is designed to achieve two primary objectives: 1) reducing communication overhead and 2) ensuring strict privacy guarantees during feature transmission, while maintaining effective inference performance. Additionally, we introduce an over-the-air pooling scheme specifically designed for classification tasks, which provides formal guarantees on the privacy of transmitted features and establishes a lower bound on classification accuracy.

Title: Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting

• Authors: Mohamed Salim Aissi, Clement Romac, Thomas Carta, Sylvain Lamprier, Pierre-Yves Oudeyer, Olivier Sigaud, Laure Soulier, Nicolas Thome
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19920
• Pdf URL: https://arxiv.org/pdf/2410.19920
• Copy Paste: [[2410.19920]] Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting(https://arxiv.org/abs/2410.19920)
• Keywords: robust, large language model
• Abstract: Reinforcement learning (RL) is a promising approach for aligning large language models (LLMs) knowledge with sequential decision-making tasks. However, few studies have thoroughly investigated the impact on LLM agents capabilities of fine-tuning them with RL in a specific environment. In this paper, we propose a novel framework to analyze the sensitivity of LLMs to prompt formulations following RL training in a textual environment. Our findings reveal that the performance of LLMs degrades when faced with prompt formulations different from those used during the RL training phase. Besides, we analyze the source of this sensitivity by examining the model's internal representations and salient tokens. Finally, we propose to use a contrastive loss to mitigate this sensitivity and improve the robustness and generalization capabilities of LLMs.

Title: Improving Multimodal Large Language Models Using Continual Learning

• Authors: Shikhar Srivastava, Md Yousuf Harun, Robik Shrestha, Christopher Kanan
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19925
• Pdf URL: https://arxiv.org/pdf/2410.19925
• Copy Paste: [[2410.19925]] Improving Multimodal Large Language Models Using Continual Learning(https://arxiv.org/abs/2410.19925)
• Keywords: robust, generative, large language model
• Abstract: Generative large language models (LLMs) exhibit impressive capabilities, which can be further augmented by integrating a pre-trained vision model into the original LLM to create a multimodal LLM (MLLM). However, this integration often significantly decreases performance on natural language understanding and generation tasks, compared to the original LLM. This study investigates this issue using the LLaVA MLLM, treating the integration as a continual learning problem. We evaluate five continual learning methods to mitigate forgetting and identify a technique that enhances visual understanding while minimizing linguistic performance loss. Our approach reduces linguistic performance degradation by up to 15\% over the LLaVA recipe, while maintaining high multimodal accuracy. We also demonstrate the robustness of our method through continual learning on a sequence of vision-language tasks, effectively preserving linguistic skills while acquiring new multimodal capabilities.

Title: Provable optimal transport with transformers: The essence of depth and prompt engineering

• Authors: Hadi Daneshmand
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.19931
• Pdf URL: https://arxiv.org/pdf/2410.19931
• Copy Paste: [[2410.19931]] Provable optimal transport with transformers: The essence of depth and prompt engineering(https://arxiv.org/abs/2410.19931)
• Keywords: transformer, generative
• Abstract: Can we establish provable performance guarantees for transformers? Establishing such theoretical guarantees is a milestone in developing trustworthy generative AI. In this paper, we take a step toward addressing this question by focusing on optimal transport, a fundamental problem at the intersection of combinatorial and continuous optimization. Leveraging the computational power of attention layers, we prove that a transformer with fixed parameters can effectively solve the optimal transport problem in Wasserstein-2 with entropic regularization for an arbitrary number of points. Consequently, the transformer can sort lists of arbitrary sizes up to an approximation factor. Our results rely on an engineered prompt that enables the transformer to implement gradient descent with adaptive stepsizes on the dual optimal transport. Combining the convergence analysis of gradient descent with Sinkhorn dynamics, we establish an explicit approximation bound for optimal transport with transformers, which improves as depth increases. Our findings provide novel insights into the essence of prompt engineering and depth for solving optimal transport. In particular, prompt engineering boosts the algorithmic expressivity of transformers, allowing them implement an optimization method. With increasing depth, transformers can simulate several iterations of gradient descent.

Title: Tracking and triangulating firefly flashes in field recordings

• Authors: Raphael Sarfati
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19932
• Pdf URL: https://arxiv.org/pdf/2410.19932
• Copy Paste: [[2410.19932]] Tracking and triangulating firefly flashes in field recordings(https://arxiv.org/abs/2410.19932)
• Keywords: robust
• Abstract: Identifying firefly flashes from other bright features in nature images is complicated. I provide a training dataset and trained neural networks for reliable flash classification. The training set consists of thousands of cropped images (patches) extracted by manual labeling from video recordings of fireflies in their natural habitat. The trained network appears as considerably more reliable to differentiate flashes from other sources of light compared to traditional methods relying solely on intensity thresholding. This robust tracking enables a new calibration-free method for the 3D reconstruction of flash occurrences from stereoscopic 360-degree videos, which I also present here.

Title: Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization

• Authors: Xiyue Peng, Hengquan Guo, Jiawei Zhang, Dongqing Zou, Ziyu Shao, Honghao Wei, Xin Liu
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.19933
• Pdf URL: https://arxiv.org/pdf/2410.19933
• Copy Paste: [[2410.19933]] Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization(https://arxiv.org/abs/2410.19933)
• Keywords: large language model
• Abstract: Balancing helpfulness and safety (harmlessness) is a critical challenge in aligning large language models (LLMs). Current approaches often decouple these two objectives, training separate preference models for helpfulness and safety, while framing safety as a constraint within a constrained Markov Decision Process (CMDP) framework. However, these methods can lead to ``safety interference'', where average-based safety constraints compromise the safety of some prompts in favor of others. To address this issue, we propose \textbf{Rectified Policy Optimization (RePO)}, which replaces the average safety constraint with stricter (per prompt) safety constraints. At the core of RePO is a policy update mechanism driven by rectified policy gradients, which penalizes the strict safety violation of every prompt, thereby enhancing safety across nearly all prompts. Our experiments on Alpaca-7B demonstrate that RePO improves the safety alignment and reduces the safety interference compared to baseline methods. Code is available at this https URL.

Title: RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Eviction

• Authors: Tanqiu Jiang, Zian Wang, Jiacheng Liang, Changjiang Li, Yuhui Wang, Ting Wang
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.19937
• Pdf URL: https://arxiv.org/pdf/2410.19937
• Copy Paste: [[2410.19937]] RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Eviction(https://arxiv.org/abs/2410.19937)
• Keywords: defense, attack, robust, large language model
• Abstract: Jailbreak attacks circumvent LLMs' built-in safeguards by concealing harmful queries within jailbreak prompts. While existing defenses primarily focus on mitigating the effects of jailbreak prompts, they often prove inadequate as jailbreak prompts can take arbitrary, adaptive forms. This paper presents RobustKV, a novel defense that adopts a fundamentally different approach by selectively removing critical tokens of harmful queries from key-value (KV) caches. Intuitively, for a jailbreak prompt to be effective, its tokens must achieve sufficient `importance' (as measured by attention scores), which inevitably lowers the importance of tokens in the concealed harmful query. Thus, by strategically evicting the KVs of the lowest-ranked tokens, RobustKV diminishes the presence of the harmful query in the KV cache, thus preventing the LLM from generating malicious responses. Extensive evaluation using benchmark datasets and models demonstrates that RobustKV effectively counters state-of-the-art jailbreak attacks while maintaining the LLM's general performance on benign queries. Moreover, RobustKV creates an intriguing evasiveness dilemma for adversaries, forcing them to balance between evading RobustKV and bypassing the LLM's built-in safeguards. This trade-off contributes to RobustKV's robustness against adaptive attacks. (warning: this paper contains potentially harmful content generated by LLMs.)

Title: Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training

• Authors: Kristjan Greenewald, Yuancheng Yu, Hao Wang, Kai Xu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.19941
• Pdf URL: https://arxiv.org/pdf/2410.19941
• Copy Paste: [[2410.19941]] Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training(https://arxiv.org/abs/2410.19941)
• Keywords: privacy, generative
• Abstract: Training generative models with differential privacy (DP) typically involves injecting noise into gradient updates or adapting the discriminator's training procedure. As a result, such approaches often struggle with hyper-parameter tuning and convergence. We consider the slicing privacy mechanism that injects noise into random low-dimensional projections of the private data, and provide strong privacy guarantees for it. These noisy projections are used for training generative models. To enable optimizing generative models using this DP approach, we introduce the smoothed-sliced $f$-divergence and show it enjoys statistical consistency. Moreover, we present a kernel-based estimator for this divergence, circumventing the need for adversarial training. Extensive numerical experiments demonstrate that our approach can generate synthetic data of higher quality compared with baselines. Beyond performance improvement, our method, by sidestepping the need for noisy gradients, offers data scientists the flexibility to adjust generator architecture and hyper-parameters, run the optimization over any number of epochs, and even restart the optimization process -- all without incurring additional privacy costs.

Title: A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT

• Authors: Nagarajan Ganapathy, Podakanti Satyajith Chary, Teja Venkata Ramana Kumar Pithani, Pavan Kavati, Arun Kumar S
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19944
• Pdf URL: https://arxiv.org/pdf/2410.19944
• Copy Paste: [[2410.19944]] A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT(https://arxiv.org/abs/2410.19944)
• Keywords: transformer
• Abstract: This Paper presents an advanced approach for fine-tuning BiomedCLIP PubMedBERT, a multimodal model, to classify abnormalities in Video Capsule Endoscopy (VCE) frames, aiming to enhance diagnostic efficiency in gastrointestinal healthcare. By integrating the PubMedBERT language model with a Vision Transformer (ViT) to process endoscopic images, our method categorizes images into ten specific classes: angioectasia, bleeding, erosion, erythema, foreign body, lymphangiectasia, polyp, ulcer, worms, and normal. Our workflow incorporates image preprocessing and fine-tunes the BiomedCLIP model to generate high-quality embeddings for both visual and textual inputs, aligning them through similarity scoring for classification. Performance metrics, including classification, accuracy, recall, and F1 score, indicate the models strong ability to accurately identify abnormalities in endoscopic frames, showing promise for practical use in clinical diagnostics.

Title: Turn-by-Turn Indoor Navigation for the Visually Impaired

• Authors: Santosh Srinivasaiah, Sai Kumar Nekkanti, Rohith Reddy Nedhunuri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19954
• Pdf URL: https://arxiv.org/pdf/2410.19954
• Copy Paste: [[2410.19954]] Turn-by-Turn Indoor Navigation for the Visually Impaired(https://arxiv.org/abs/2410.19954)
• Keywords: large language model
• Abstract: Navigating indoor environments presents significant challenges for visually impaired individuals due to complex layouts and the absence of GPS signals. This paper introduces a novel system that provides turn-by-turn navigation inside buildings using only a smartphone equipped with a camera, leveraging multimodal models, deep learning algorithms, and large language models (LLMs). The smartphone's camera captures real-time images of the surroundings, which are then sent to a nearby Raspberry Pi capable of running on-device LLM models, multimodal models, and deep learning algorithms to detect and recognize architectural features, signage, and obstacles. The interpreted visual data is then translated into natural language instructions by an LLM running on the Raspberry Pi, which is sent back to the user, offering intuitive and context-aware guidance via audio prompts. This solution requires minimal workload on the user's device, preventing it from being overloaded and offering compatibility with all types of devices, including those incapable of running AI models. This approach enables the client to not only run advanced models but also ensure that the training data and other information do not leave the building. Preliminary evaluations demonstrate the system's effectiveness in accurately guiding users through complex indoor spaces, highlighting its potential for widespread application

Title: DualMAR: Medical-Augmented Representation from Dual-Expertise Perspectives

• Authors: Pengfei Hu, Chang Lu, Fei Wang, Yue Ning
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.19955
• Pdf URL: https://arxiv.org/pdf/2410.19955
• Copy Paste: [[2410.19955]] DualMAR: Medical-Augmented Representation from Dual-Expertise Perspectives(https://arxiv.org/abs/2410.19955)
• Keywords: large language model
• Abstract: Electronic Health Records (EHR) has revolutionized healthcare data management and prediction in the field of AI and machine learning. Accurate predictions of diagnosis and medications significantly mitigate health risks and provide guidance for preventive care. However, EHR driven models often have limited scope on understanding medical-domain knowledge and mostly rely on simple-and-sole ontologies. In addition, due to the missing features and incomplete disease coverage of EHR, most studies only focus on basic analysis on conditions and medication. We propose DualMAR, a framework that enhances EHR prediction tasks through both individual observation data and public knowledge bases. First, we construct a bi-hierarchical Diagnosis Knowledge Graph (KG) using verified public clinical ontologies and augment this KG via Large Language Models (LLMs); Second, we design a new proxy-task learning on lab results in EHR for pretraining, which further enhance KG representation and patient embeddings. By retrieving radial and angular coordinates upon polar space, DualMAR enables accurate predictions based on rich hierarchical and semantic embeddings from KG. Experiments also demonstrate that DualMAR outperforms state-of-the-art models, validating its effectiveness in EHR prediction and KG integration in medical domains.

Title: Understanding Adam Requires Better Rotation Dependent Assumptions

• Authors: Lucas Maes, Tianyue H. Zhang, Alexia Jolicoeur-Martineau, Ioannis Mitliagkas, Damien Scieur, Simon Lacoste-Julien, Charles Guille-Escuret
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19964
• Pdf URL: https://arxiv.org/pdf/2410.19964
• Copy Paste: [[2410.19964]] Understanding Adam Requires Better Rotation Dependent Assumptions(https://arxiv.org/abs/2410.19964)
• Keywords: transformer
• Abstract: Despite its widespread adoption, Adam's advantage over Stochastic Gradient Descent (SGD) lacks a comprehensive theoretical explanation. This paper investigates Adam's sensitivity to rotations of the parameter space. We demonstrate that Adam's performance in training transformers degrades under random rotations of the parameter space, indicating a crucial sensitivity to the choice of basis. This reveals that conventional rotation-invariant assumptions are insufficient to capture Adam's advantages theoretically. To better understand the rotation-dependent properties that benefit Adam, we also identify structured rotations that preserve or even enhance its empirical performance. We then examine the rotation-dependent assumptions in the literature, evaluating their adequacy in explaining Adam's behavior across various rotation types. This work highlights the need for new, rotation-dependent theoretical frameworks to fully understand Adam's empirical success in modern machine learning tasks.

Title: OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery

• Authors: Philipe Dias, Aristeidis Tsaris, Jordan Bowman, Abhishek Potnis, Jacob Arndt, H. Lexie Yang, Dalton Lunga
• Subjects: cs.CV, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.19965
• Pdf URL: https://arxiv.org/pdf/2410.19965
• Copy Paste: [[2410.19965]] OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery(https://arxiv.org/abs/2410.19965)
• Keywords: transformer, segmentation
• Abstract: While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.

Title: Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements

• Authors: Silvia Terragni, Hoang Cuong, Joachim Daiber, Pallavi Gudipati, Pablo N. Mendes
• Subjects: cs.LG, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.19974
• Pdf URL: https://arxiv.org/pdf/2410.19974
• Copy Paste: [[2410.19974]] Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements(https://arxiv.org/abs/2410.19974)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated potential as effective search relevance evaluators. However, there is a lack of comprehensive guidance on which models consistently perform optimally across various contexts or within specific use cases. In this paper, we assess several LLMs and Multimodal Language Models (MLLMs) in terms of their alignment with human judgments across multiple multimodal search scenarios. Our analysis investigates the trade-offs between cost and accuracy, highlighting that model performance varies significantly depending on the context. Interestingly, in smaller models, the inclusion of a visual component may hinder performance rather than enhance it. These findings highlight the complexities involved in selecting the most appropriate model for practical applications.

Title: Residual Random Neural Networks

• Authors: M. Andrecut
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.19987
• Pdf URL: https://arxiv.org/pdf/2410.19987
• Copy Paste: [[2410.19987]] Residual Random Neural Networks(https://arxiv.org/abs/2410.19987)
• Keywords: protect
• Abstract: The single-layer feedforward neural network with random weights is a recurring motif in the neural networks literature. The advantage of these networks is their simplified training, which reduces to solving a ridge-regression problem. However, a general assumption is that these networks require a large number of hidden neurons relative to the dimensionality of the data samples, in order to achieve good classification accuracy. Contrary to this assumption, here we show that one can obtain good classification results even if the number of hidden neurons has the same order of magnitude as the dimensionality of the data samples, if this dimensionality is reasonably high. We also develop an efficient iterative residual training method for such random neural networks, which significantly improves their classification accuracy. Moreover, we also describe an encryption (obfuscation) method which can be used to protect both the data and the neural network model.

Title: A-MFST: Adaptive Multi-Flow Sparse Tracker for Real-Time Tissue Tracking Under Occlusion

• Authors: Yuxin Chen, Zijian Wu, Adam Schmidt, Septimiu E. Salcudean
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.19996
• Pdf URL: https://arxiv.org/pdf/2410.19996
• Copy Paste: [[2410.19996]] A-MFST: Adaptive Multi-Flow Sparse Tracker for Real-Time Tissue Tracking Under Occlusion(https://arxiv.org/abs/2410.19996)
• Keywords: robust
• Abstract: Purpose: Tissue tracking is critical for downstream tasks in robot-assisted surgery. The Sparse Efficient Neural Depth and Deformation (SENDD) model has previously demonstrated accurate and real-time sparse point tracking, but struggled with occlusion handling. This work extends SENDD to enhance occlusion detection and tracking consistency while maintaining real-time performance. Methods: We use the Segment Anything Model2 (SAM2) to detect and mask occlusions by surgical tools, and we develop and integrate into SENDD an Adaptive Multi-Flow Sparse Tracker (A-MFST) with forward-backward consistency metrics, to enhance occlusion and uncertainty estimation. A-MFST is an unsupervised variant of the Multi-Flow Dense Tracker (MFT). Results: We evaluate our approach on the STIR dataset and demonstrate a significant improvement in tracking accuracy under occlusion, reducing average tracking errors by 12 percent in Mean Endpoint Error (MEE) and showing a 6 percent improvement in the averaged accuracy over thresholds of 4, 8, 16, 32, and 64 pixels. The incorporation of forward-backward consistency further improves the selection of optimal tracking paths, reducing drift and enhancing robustness. Notably, these improvements were achieved without compromising the model's real-time capabilities. Conclusions: Using A-MFST and SAM2, we enhance SENDD's ability to track tissue in real time under instrument and tissue occlusions.

Title: Lightweight, Secure and Stateful Serverless Computing with PSL

• Authors: Alexander Thomas, Shubham Mishra, Kaiyuan Chen, John Kubiatowicz
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.20004
• Pdf URL: https://arxiv.org/pdf/2410.20004
• Copy Paste: [[2410.20004]] Lightweight, Secure and Stateful Serverless Computing with PSL(https://arxiv.org/abs/2410.20004)
• Keywords: secure
• Abstract: We present PSL, a lightweight, secure and stateful Function-as-a-Serivce (FaaS) framework for Trusted Execution Environments (TEEs). The framework provides rich programming language support on heterogeneous TEE hardware for statically compiled binaries and/or WebAssembly (WASM) bytecodes, with a familiar Key-Value Store (KVS) interface to secure, performant, network-embedded storage. It achieves near-native execution speeds by utilizing the dynamic memory mapping capabilities of Intel SGX2 to create an in-enclave WASM runtime with Just-In-Time (JIT) compilation. PSL is designed to efficiently operate within an asynchronous environment with a distributed tamper-proof confidential storage system, assuming minority failures. The system exchanges eventually-consistent state updates across nodes while utilizing release-consistent locking mechanisms to enhance transactional capabilities. The execution of PSL is up to 3.7x faster than the state-of-the-art SGX WASM runtime. PSL reaches 95k ops/s with YCSB 100% read workload and 89k ops/s with 50% read/write workload. We demonstrate the scalability and adaptivity of PSL through a case study of secure and distributed training of deep neural networks.

Title: Unsupervised Machine Learning for Detecting and Locating Human-Made Objects in 3D Point Cloud

• Authors: Hong Zhao, Huyunting Huang, Tonglin Zhang, Baijian Yang, Jin Wei-Kocsis, Songlin Fei
• Subjects: cs.CV, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.20006
• Pdf URL: https://arxiv.org/pdf/2410.20006
• Copy Paste: [[2410.20006]] Unsupervised Machine Learning for Detecting and Locating Human-Made Objects in 3D Point Cloud(https://arxiv.org/abs/2410.20006)
• Keywords: extraction
• Abstract: A 3D point cloud is an unstructured, sparse, and irregular dataset, typically collected by airborne LiDAR systems over a geological region. Laser pulses emitted from these systems reflect off objects both on and above the ground, resulting in a dataset containing the longitude, latitude, and elevation of each point, as well as information about the corresponding laser pulse strengths. A widely studied research problem, addressed in many previous works, is ground filtering, which involves partitioning the points into ground and non-ground subsets. This research introduces a novel task: detecting and identifying human-made objects amidst natural tree structures. This task is performed on the subset of non-ground points derived from the ground filtering stage. Marked Point Fields (MPFs) are used as models well-suited to these tasks. The proposed methodology consists of three stages: ground filtering, local information extraction (LIE), and clustering. In the ground filtering stage, a statistical method called One-Sided Regression (OSR) is introduced, addressing the limitations of prior ground filtering methods on uneven terrains. In the LIE stage, unsupervised learning methods are lacking. To mitigate this, a kernel-based method for the Hessian matrix of the MPF is developed. In the clustering stage, the Gaussian Mixture Model (GMM) is applied to the results of the LIE stage to partition the non-ground points into trees and human-made objects. The underlying assumption is that LiDAR points from trees exhibit a three-dimensional distribution, while those from human-made objects follow a two-dimensional distribution. The Hessian matrix of the MPF effectively captures this distinction. Experimental results demonstrate that the proposed ground filtering method outperforms previous techniques, and the LIE method successfully distinguishes between points representing trees and human-made objects.

Title: Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models

• Authors: Zheng Zhao, Yftah Ziser, Shay B. Cohen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20008
• Pdf URL: https://arxiv.org/pdf/2410.20008
• Copy Paste: [[2410.20008]] Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models(https://arxiv.org/abs/2410.20008)
• Keywords: large language model
• Abstract: Fine-tuning pre-trained large language models (LLMs) on a diverse array of tasks has become a common approach for building models that can solve various natural language processing (NLP) tasks. However, where and to what extent these models retain task-specific knowledge remains largely unexplored. This study investigates the task-specific information encoded in pre-trained LLMs and the effects of instruction tuning on their representations across a diverse set of over 60 NLP tasks. We use a set of matrix analysis tools to examine the differences between the way pre-trained and instruction-tuned LLMs store task-specific information. Our findings reveal that while some tasks are already encoded within the pre-trained LLMs, others greatly benefit from instruction tuning. Additionally, we pinpointed the layers in which the model transitions from high-level general representations to more task-oriented representations. This finding extends our understanding of the governing mechanisms of LLMs and facilitates future research in the fields of parameter-efficient transfer learning and multi-task learning.

Title: Vulnerability of LLMs to Vertically Aligned Text Manipulations

• Authors: Zhecheng Li, Yiwei Wang, Bryan Hooi, Yujun Cai, Zhen Xiong, Nanyun Peng, Kai-wei Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20016
• Pdf URL: https://arxiv.org/pdf/2410.20016
• Copy Paste: [[2410.20016]] Vulnerability of LLMs to Vertically Aligned Text Manipulations(https://arxiv.org/abs/2410.20016)
• Keywords: large language model
• Abstract: Text classification involves categorizing a given text, such as determining its sentiment or identifying harmful content. With the advancement of large language models (LLMs), these models have become highly effective at performing text classification tasks. However, they still show vulnerabilities to variations in text formatting. Recent research demonstrates that modifying input formats, such as vertically aligning words for encoder-based models, can substantially lower accuracy in text classification tasks. While easily understood by humans, these inputs can significantly mislead models, posing a potential risk of bypassing detection in real-world scenarios involving harmful or sensitive information. With the expanding application of LLMs, a crucial question arises: Do decoder-based LLMs exhibit similar vulnerabilities to vertically formatted text input? In this paper, we investigate the impact of vertical text input on the performance of various LLMs across multiple text classification datasets and analyze the underlying causes. Our findings are as follows: (i) Vertical text input significantly degrades the accuracy of LLMs in text classification tasks. (ii) Chain of Thought (CoT) reasoning does not help LLMs recognize vertical input or mitigate its vulnerability, but few-shot learning with careful analysis does. (iii) We explore the underlying cause of the vulnerability by analyzing the inherent issues in tokenization and attention matrices.

Title: Off-Policy Selection for Initiating Human-Centric Experimental Design

• Authors: Ge Gao, Xi Yang, Qitong Gao, Song Ju, Miroslav Pajic, Min Chi
• Subjects: cs.LG, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.20017
• Pdf URL: https://arxiv.org/pdf/2410.20017
• Copy Paste: [[2410.20017]] Off-Policy Selection for Initiating Human-Centric Experimental Design(https://arxiv.org/abs/2410.20017)
• Keywords: segmentation
• Abstract: In human-centric tasks such as healthcare and education, the heterogeneity among patients and students necessitates personalized treatments and instructional interventions. While reinforcement learning (RL) has been utilized in those tasks, off-policy selection (OPS) is pivotal to close the loop by offline evaluating and selecting policies without online interactions, yet current OPS methods often overlook the heterogeneity among participants. Our work is centered on resolving a pivotal challenge in human-centric systems (HCSs): how to select a policy to deploy when a new participant joining the cohort, without having access to any prior offline data collected over the participant? We introduce First-Glance Off-Policy Selection (FPS), a novel approach that systematically addresses participant heterogeneity through sub-group segmentation and tailored OPS criteria to each sub-group. By grouping individuals with similar traits, FPS facilitates personalized policy selection aligned with unique characteristics of each participant or group of participants. FPS is evaluated via two important but challenging applications, intelligent tutoring systems and a healthcare application for sepsis treatment and intervention. FPS presents significant advancement in enhancing learning outcomes of students and in-hospital care outcomes.

Title: Attacks against Abstractive Text Summarization Models through Lead Bias and Influence Functions

• Authors: Poojitha Thota, Shirin Nilizadeh
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.20019
• Pdf URL: https://arxiv.org/pdf/2410.20019
• Copy Paste: [[2410.20019]] Attacks against Abstractive Text Summarization Models through Lead Bias and Influence Functions(https://arxiv.org/abs/2410.20019)
• Keywords: attack, robust, large language model
• Abstract: Large Language Models have introduced novel opportunities for text comprehension and generation. Yet, they are vulnerable to adversarial perturbations and data poisoning attacks, particularly in tasks like text classification and translation. However, the adversarial robustness of abstractive text summarization models remains less explored. In this work, we unveil a novel approach by exploiting the inherent lead bias in summarization models, to perform adversarial perturbations. Furthermore, we introduce an innovative application of influence functions, to execute data poisoning, which compromises the model's integrity. This approach not only shows a skew in the models behavior to produce desired outcomes but also shows a new behavioral change, where models under attack tend to generate extractive summaries rather than abstractive summaries.

Title: Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization

• Authors: Zhecheng Li, Yiwei Wang, Bryan Hooi, Yujun Cai, Naifan Cheung, Nanyun Peng, Kai-wei Chang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20021
• Pdf URL: https://arxiv.org/pdf/2410.20021
• Copy Paste: [[2410.20021]] Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization(https://arxiv.org/abs/2410.20021)
• Keywords: large language model
• Abstract: Cross-lingual summarization (CLS) aims to generate a summary for the source text in a different target language. Currently, instruction-tuned large language models (LLMs) excel at various English tasks. However, unlike languages such as English, Chinese or Spanish, for those relatively low-resource languages with limited usage or data, recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings. This raises the question: Are LLMs capable of handling cross-lingual summarization tasks for low-resource languages? To resolve this question, we fully explore the potential of large language models on cross-lingual summarization task for low-resource languages through our four-step zero-shot method: Summarization, Improvement, Translation and Refinement (SITR) with correspondingly designed prompts. We test our proposed method with multiple LLMs on two well-known cross-lingual summarization datasets with various low-resource target languages. The results show that: i) GPT-3.5 and GPT-4 significantly and consistently outperform other baselines when using our zero-shot SITR methods. ii) By employing our proposed method, we unlock the potential of LLMs, enabling them to effectively handle cross-lingual summarization tasks for relatively low-resource languages.

Title: Dynamic layer selection in decoder-only transformers

• Authors: Theodore Glavas, Joud Chataoui, Florence Regol, Wassim Jabbour, Antonios Valkanas, Boris N. Oreshkin, Mark Coates
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20022
• Pdf URL: https://arxiv.org/pdf/2410.20022
• Copy Paste: [[2410.20022]] Dynamic layer selection in decoder-only transformers(https://arxiv.org/abs/2410.20022)
• Keywords: robust, transformer, large language model
• Abstract: The vast size of Large Language Models (LLMs) has prompted a search to optimize inference. One effective approach is dynamic inference, which adapts the architecture to the sample-at-hand to reduce the overall computational cost. We empirically examine two common dynamic inference methods for natural language generation (NLG): layer skipping and early exiting. We find that a pre-trained decoder-only model is significantly more robust to layer removal via layer skipping, as opposed to early exit. We demonstrate the difficulty of using hidden state information to adapt computation on a per-token basis for layer skipping. Finally, we show that dynamic computation allocation on a per-sequence basis holds promise for significant efficiency gains by constructing an oracle controller. Remarkably, we find that there exists an allocation which achieves equal performance to the full model using only 23.3% of its layers on average.

Title: Beyond Fine-Tuning: Effective Strategies for Mitigating Hallucinations in Large Language Models for Data Analytics

• Authors: Mikhail Rumiantsau, Aliaksei Vertsel, Ilya Hrytsuk, Isaiah Ballah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20024
• Pdf URL: https://arxiv.org/pdf/2410.20024
• Copy Paste: [[2410.20024]] Beyond Fine-Tuning: Effective Strategies for Mitigating Hallucinations in Large Language Models for Data Analytics(https://arxiv.org/abs/2410.20024)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have become increasingly important in natural language processing, enabling advanced data analytics through natural language queries. However, these models often generate "hallucinations"-inaccurate or fabricated information-that can undermine their reliability in critical data-driven decision-making. Addressing the challenge of hallucinations is essential to improve the accuracy and trustworthiness of LLMs in processing natural language queries. This research focuses on mitigating hallucinations in LLMs, specifically within the context of data analytics. We introduce and evaluate four targeted strategies: Structured Output Generation, Strict Rules Enforcement, System Prompt Enhancements, and Semantic Layer Integration. Our findings show that these methods are more effective than traditional fine-tuning approaches in reducing hallucinations, offering a more reliable framework for deploying LLMs in natural language queries for data analytics. This research demonstrates the potential of these strategies to enhance the accuracy of LLM-driven data queries, ensuring more dependable results in data-driven environments.

Title: Towards Robust Algorithms for Surgical Phase Recognition via Digital Twin-based Scene Representation

• Authors: Hao Ding, Yuqian Zhang, Hongchao Shu, Xu Lian, Ji Woong Kim, Axel Krieger, Mathias Unberath
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20026
• Pdf URL: https://arxiv.org/pdf/2410.20026
• Copy Paste: [[2410.20026]] Towards Robust Algorithms for Surgical Phase Recognition via Digital Twin-based Scene Representation(https://arxiv.org/abs/2410.20026)
• Keywords: robust, extraction, interpretability
• Abstract: Purpose: Surgical phase recognition (SPR) is an integral component of surgical data science, enabling high-level surgical analysis. End-to-end trained neural networks that predict surgical phase directly from videos have shown excellent performance on benchmarks. However, these models struggle with robustness due to non-causal associations in the training set, resulting in poor generalizability. Our goal is to improve model robustness to variations in the surgical videos by leveraging the digital twin (DT) paradigm -- an intermediary layer to separate high-level analysis (SPR) from low-level processing (geometric understanding). This approach takes advantage of the recent vision foundation models that ensure reliable low-level scene understanding to craft DT-based scene representations that support various high-level tasks. Methods: We present a DT-based framework for SPR from videos. The framework employs vision foundation models to extract representations. We embed the representation in place of raw video inputs in the state-of-the-art Surgformer model. The framework is trained on the Cholec80 dataset and evaluated on out-of-distribution (OOD) and corrupted test samples. Results: Contrary to the vulnerability of the baseline model, our framework demonstrates strong robustness on both OOD and corrupted samples, with a video-level accuracy of 51.1 on the challenging CRCD dataset, 96.0 on an internal robotics training dataset, and 64.4 on a highly corrupted Cholec80 test set. Conclusion: Our findings lend support to the thesis that DT-based scene representations are effective in enhancing model robustness. Future work will seek to improve the feature informativeness, automate feature extraction, and incorporate interpretability for a more comprehensive framework.

Title: SCube: Instant Large-Scale Scene Reconstruction using VoxSplats

• Authors: Xuanchi Ren, Yifan Lu, Hanxue Liang, Zhangjie Wu, Huan Ling, Mike Chen, Sanja Fidler, Francis Williams, Jiahui Huang
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.20030
• Pdf URL: https://arxiv.org/pdf/2410.20030
• Copy Paste: [[2410.20030]] SCube: Instant Large-Scale Scene Reconstruction using VoxSplats(https://arxiv.org/abs/2410.20030)
• Keywords: diffusion
• Abstract: We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images. Our method encodes reconstructed scenes using a novel representation VoxSplat, which is a set of 3D Gaussians supported on a high-resolution sparse-voxel scaffold. To reconstruct a VoxSplat from images, we employ a hierarchical voxel latent diffusion model conditioned on the input images followed by a feedforward appearance prediction model. The diffusion model generates high-resolution grids progressively in a coarse-to-fine manner, and the appearance network predicts a set of Gaussians within each voxel. From as few as 3 non-overlapping input images, SCube can generate millions of Gaussians with a 1024^3 voxel grid spanning hundreds of meters in 20 seconds. Past works tackling scene reconstruction from images either rely on per-scene optimization and fail to reconstruct the scene away from input views (thus requiring dense view coverage as input) or leverage geometric priors based on low-resolution models, which produce blurry results. In contrast, SCube leverages high-resolution sparse networks and produces sharp outputs from few views. We show the superiority of SCube compared to prior art using the Waymo self-driving dataset on 3D reconstruction and demonstrate its applications, such as LiDAR simulation and text-to-scene generation.

Title: Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors

• Authors: Wenqiang Chen, Jiaxuan Cheng, Leyao Wang, Wei Zhao, Wojciech Matusik
• Subjects: cs.LG, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.20034
• Pdf URL: https://arxiv.org/pdf/2410.20034
• Copy Paste: [[2410.20034]] Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors(https://arxiv.org/abs/2410.20034)
• Keywords: privacy, large language model
• Abstract: Visual Question-Answering, a technology that generates textual responses from an image and natural language question, has progressed significantly. Notably, it can aid in tracking and inquiring about daily activities, crucial in healthcare monitoring, especially for elderly patients or those with memory disabilities. However, video poses privacy concerns and has a limited field of view. This paper presents Sensor2Text, a model proficient in tracking daily activities and engaging in conversations using wearable sensors. The approach outlined here tackles several challenges, including low information density in wearable sensor data, insufficiency of single wearable sensors in human activities recognition, and model's limited capacity for Question-Answering and interactive conversations. To resolve these obstacles, transfer learning and student-teacher networks are utilized to leverage knowledge from visual-language models. Additionally, an encoder-decoder neural network model is devised to jointly process language and sensor data for conversational purposes. Furthermore, Large Language Models are also utilized to enable interactive capabilities. The model showcases the ability to identify human activities and engage in Q\&A dialogues using various wearable sensor modalities. It performs comparably to or better than existing visual-language models in both captioning and conversational tasks. To our knowledge, this represents the first model capable of conversing about wearable sensor data, offering an innovative approach to daily activity tracking that addresses privacy and field-of-view limitations associated with current vision-based solutions.

Title: Training the Untrainable: Introducing Inductive Bias via Representational Alignment

• Authors: Vighnesh Subramaniam, David Mayo, Colin Conwell, Tomaso Poggio, Boris Katz, Brian Cheung, Andrei Barbu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20035
• Pdf URL: https://arxiv.org/pdf/2410.20035
• Copy Paste: [[2410.20035]] Training the Untrainable: Introducing Inductive Bias via Representational Alignment(https://arxiv.org/abs/2410.20035)
• Keywords: transformer
• Abstract: We demonstrate that architectures which traditionally are considered to be ill-suited for a task can be trained using inductive biases from another architecture. Networks are considered untrainable when they overfit, underfit, or converge to poor results even when tuning their hyperparameters. For example, plain fully connected networks overfit on object recognition while deep convolutional networks without residual connections underfit. The traditional answer is to change the architecture to impose some inductive bias, although what that bias is remains unknown. We introduce guidance, where a guide network guides a target network using a neural distance function. The target is optimized to perform well and to match its internal representations, layer-by-layer, to those of the guide; the guide is unchanged. If the guide is trained, this transfers over part of the architectural prior and knowledge of the guide to the target. If the guide is untrained, this transfers over only part of the architectural prior of the guide. In this manner, we can investigate what kinds of priors different architectures place on untrainable networks such as fully connected networks. We demonstrate that this method overcomes the immediate overfitting of fully connected networks on vision tasks, makes plain CNNs competitive to ResNets, closes much of the gap between plain vanilla RNNs and Transformers, and can even help Transformers learn tasks which RNNs can perform more easily. We also discover evidence that better initializations of fully connected networks likely exist to avoid overfitting. Our method provides a mathematical tool to investigate priors and architectures, and in the long term, may demystify the dark art of architecture creation, even perhaps turning architectures into a continuous optimizable parameter of the network.

Title: Annotation Efficiency: Identifying Hard Samples via Blocked Sparse Linear Bandits

• Authors: Adit Jain, Soumyabrata Pal, Sunav Choudhary, Ramasuri Narayanam, Vikram Krishnamurthy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20041
• Pdf URL: https://arxiv.org/pdf/2410.20041
• Copy Paste: [[2410.20041]] Annotation Efficiency: Identifying Hard Samples via Blocked Sparse Linear Bandits(https://arxiv.org/abs/2410.20041)
• Keywords: robust
• Abstract: This paper considers the problem of annotating datapoints using an expert with only a few annotation rounds in a label-scarce setting. We propose soliciting reliable feedback on difficulty in annotating a datapoint from the expert in addition to ground truth label. Existing literature in active learning or coreset selection turns out to be less relevant to our setting since they presume the existence of a reliable trained model, which is absent in the label-scarce regime. However, the literature on coreset selection emphasizes the presence of difficult data points in the training set to perform supervised learning in downstream tasks (Mindermann et al., 2022). Therefore, for a given fixed annotation budget of $\mathsf{T}$ rounds, we model the sequential decision-making problem of which (difficult) datapoints to choose for annotation in a sparse linear bandits framework with the constraint that no arm can be pulled more than once (blocking constraint). With mild assumptions on the datapoints, our (computationally efficient) Explore-Then-Commit algorithm BSLB achieves a regret guarantee of $\widetilde{\mathsf{O}}(k^{\frac{1}{3}} \mathsf{T}^{\frac{2}{3}} +k^{-\frac{1}{2}} \beta_k + k^{-\frac{1}{12}} \beta_k^{\frac{1}{2}}\mathsf{T}^{\frac{5}{6}})$ where the unknown parameter vector has tail magnitude $\beta_k$ at sparsity level $k$. To this end, we show offline statistical guarantees of Lasso estimator with mild Restricted Eigenvalue (RE) condition that is also robust to sparsity. Finally, we propose a meta-algorithm C-BSLB that does not need knowledge of the optimal sparsity parameters at a no-regret cost. We demonstrate the efficacy of our BSLB algorithm for annotation in the label-scarce setting for an image classification task on the PASCAL-VOC dataset, where we use real-world annotation difficulty scores.

Title: Evaluating Neural Networks for Early Maritime Threat Detection

• Authors: Dhanush Tella, Chandra Teja Tiriveedhi, Naphtali Rishe, Dan E. Tamir, Jonathan I. Tamir
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20054
• Pdf URL: https://arxiv.org/pdf/2410.20054
• Copy Paste: [[2410.20054]] Evaluating Neural Networks for Early Maritime Threat Detection(https://arxiv.org/abs/2410.20054)
• Keywords: robust
• Abstract: We consider the task of classifying trajectories of boat activities as a proxy for assessing maritime threats. Previous approaches have considered entropy-based metrics for clustering boat activity into three broad categories: random walk, following, and chasing. Here, we comprehensively assess the accuracy of neural network-based approaches as alternatives to entropy-based clustering. We train four neural network models and compare them to shallow learning using synthetic data. We also investigate the accuracy of models as time steps increase and with and without rotated data. To improve test-time robustness, we normalize trajectories and perform rotation-based data augmentation. Our results show that deep networks can achieve a test-set accuracy of up to 100% on a full trajectory, with graceful degradation as the number of time steps decreases, outperforming entropy-based clustering.

Title: 3D Distance-color-coded Assessment of PCI Stent Apposition via Deep-learning-based Three-dimensional Multi-object Segmentation

• Authors: Xiaoyang Qin, Hao Huang, Shuaichen Lin, Xinhao Zeng, Kaizhi Cao, Renxiong Wu, Yuming Huang, Junqing Yang, Yong Liu, Gang Li, Guangming Ni
• Subjects: cs.CV, physics.optics
• Abstract URL: https://arxiv.org/abs/2410.20055
• Pdf URL: https://arxiv.org/pdf/2410.20055
• Copy Paste: [[2410.20055]] 3D Distance-color-coded Assessment of PCI Stent Apposition via Deep-learning-based Three-dimensional Multi-object Segmentation(https://arxiv.org/abs/2410.20055)
• Keywords: segmentation
• Abstract: Coronary artery disease poses a significant global health challenge, often necessitating percutaneous coronary intervention (PCI) with stent implantation. Assessing stent apposition holds pivotal importance in averting and identifying PCI complications that lead to in-stent restenosis. Here we proposed a novel three-dimensional (3D) distance-color-coded assessment (DccA)for PCI stent apposition via deep-learning-based 3D multi-object segmentation in intravascular optical coherence tomography (IV-OCT). Our proposed 3D DccA accurately segments 3D vessel lumens and stents in IV-OCT images, using a spatial matching network and dual-layer training with style transfer. It quantifies and maps stent-lumen distances into a 3D color space, facilitating 3D visual assessment of PCI stent apposition. Achieving over 95% segmentation precision, our proposed DccA enhances clinical evaluation of PCI stent deployment and supports personalized treatment planning.

Title: Deep Concept Identification for Generative Design

• Authors: Ryo Tsumoto, Kentaro Yaji, Yutaka Nomaguchi, Kikuo Fujita
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20061
• Pdf URL: https://arxiv.org/pdf/2410.20061
• Copy Paste: [[2410.20061]] Deep Concept Identification for Generative Design(https://arxiv.org/abs/2410.20061)
• Keywords: generative
• Abstract: A generative design based on topology optimization provides diverse alternatives as entities in a computational model with a high design degree. However, as the diversity of the generated alternatives increases, the cognitive burden on designers to select the most appropriate alternatives also increases. Whereas the concept identification approach, which finds various categories of entities, is an effective means to structure alternatives, evaluation of their similarities is challenging due to shape diversity. To address this challenge, this study proposes a concept identification framework for generative design using deep learning (DL) techniques. One of the key abilities of DL is the automatic learning of different representations of a specific task. Deep concept identification finds various categories that provide insights into the mapping relationships between geometric properties and structural performance through representation learning using DL. The proposed framework generates diverse alternatives using a generative design technique, clusters the alternatives into several categories using a DL technique, and arranges these categories for design practice using a classification model. This study demonstrates its fundamental capabilities by implementing variational deep embedding, a generative and clustering model based on the DL paradigm, and logistic regression as a classification model. A simplified design problem of a two-dimensional bridge structure is applied as a case study to validate the proposed framework. Although designers are required to determine the viewing aspect level by setting the number of concepts, this implementation presents the identified concepts and their relationships in the form of a decision tree based on a specified level.

Title: CGKN: A Deep Learning Framework for Modeling Complex Dynamical Systems and Efficient Data Assimilation

• Authors: Chuanqi Chen, Nan Chen, Yinling Zhang, Jin-Long Wu
• Subjects: cs.LG, math.DS
• Abstract URL: https://arxiv.org/abs/2410.20072
• Pdf URL: https://arxiv.org/pdf/2410.20072
• Copy Paste: [[2410.20072]] CGKN: A Deep Learning Framework for Modeling Complex Dynamical Systems and Efficient Data Assimilation(https://arxiv.org/abs/2410.20072)
• Keywords: robust
• Abstract: Deep learning is widely used to predict complex dynamical systems in many scientific and engineering areas. However, the black-box nature of these deep learning models presents significant challenges for carrying out simultaneous data assimilation (DA), which is a crucial technique for state estimation, model identification, and reconstructing missing data. Integrating ensemble-based DA methods with nonlinear deep learning models is computationally expensive and may suffer from large sampling errors. To address these challenges, we introduce a deep learning framework designed to simultaneously provide accurate forecasts and efficient DA. It is named Conditional Gaussian Koopman Network (CGKN), which transforms general nonlinear systems into nonlinear neural differential equations with conditional Gaussian structures. CGKN aims to retain essential nonlinear components while applying systematic and minimal simplifications to facilitate the development of analytic formulae for nonlinear DA. This allows for seamless integration of DA performance into the deep learning training process, eliminating the need for empirical tuning as required in ensemble methods. CGKN compensates for structural simplifications by lifting the dimension of the system, which is motivated by Koopman theory. Nevertheless, CGKN exploits special nonlinear dynamics within the lifted space. This enables the model to capture extreme events and strong non-Gaussian features in joint and marginal distributions with appropriate uncertainty quantification. We demonstrate the effectiveness of CGKN for both prediction and DA on three strongly nonlinear and non-Gaussian turbulent systems: the projected stochastic Burgers--Sivashinsky equation, the Lorenz 96 system, and the El Niño-Southern Oscillation. The results justify the robustness and computational efficiency of CGKN.

Title: SFTrack: A Robust Scale and Motion Adaptive Algorithm for Tracking Small and Fast Moving Objects

• Authors: InPyo Song, Jangwon Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20079
• Pdf URL: https://arxiv.org/pdf/2410.20079
• Copy Paste: [[2410.20079]] SFTrack: A Robust Scale and Motion Adaptive Algorithm for Tracking Small and Fast Moving Objects(https://arxiv.org/abs/2410.20079)
• Keywords: robust
• Abstract: This paper addresses the problem of multi-object tracking in Unmanned Aerial Vehicle (UAV) footage. It plays a critical role in various UAV applications, including traffic monitoring systems and real-time suspect tracking by the police. However, this task is highly challenging due to the fast motion of UAVs, as well as the small size of target objects in the videos caused by the high-altitude and wide angle views of drones. In this study, we thus introduce a simple yet more effective method compared to previous work to overcome these challenges. Our approach involves a new tracking strategy, which initiates the tracking of target objects from low-confidence detections commonly encountered in UAV application scenarios. Additionally, we propose revisiting traditional appearance-based matching algorithms to improve the association of low-confidence detections. To evaluate the effectiveness of our method, we conducted benchmark evaluations on two UAV-specific datasets (VisDrone2019, UAVDT) and one general object tracking dataset (MOT17). The results demonstrate that our approach surpasses current state-of-the art methodologies, highlighting its robustness and adaptability in diverse tracking environments. Furthermore, we have improved the annotation of the UAVDT dataset by rectifying several errors and addressing omissions found in the original annotations. We will provide this refined version of the dataset to facilitate better benchmarking in the field.

Title: emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography

• Authors: Viswanath Sivakumar, Jeffrey Seely, Alan Du, Sean R Bittner, Adam Berenzweig, Anuoluwapo Bolarinwa, Alexandre Gramfort, Michael I Mandel
• Subjects: cs.LG, cs.HC, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.20081
• Pdf URL: https://arxiv.org/pdf/2410.20081
• Copy Paste: [[2410.20081]] emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography(https://arxiv.org/abs/2410.20081)
• Keywords: generative
• Abstract: Surface electromyography (sEMG) non-invasively measures signals generated by muscle activity with sufficient sensitivity to detect individual spinal neurons and richness to identify dozens of gestures and their nuances. Wearable wrist-based sEMG sensors have the potential to offer low friction, subtle, information rich, always available human-computer inputs. To this end, we introduce emg2qwerty, a large-scale dataset of non-invasive electromyographic signals recorded at the wrists while touch typing on a QWERTY keyboard, together with ground-truth annotations and reproducible baselines. With 1,135 sessions spanning 108 users and 346 hours of recording, this is the largest such public dataset to date. These data demonstrate non-trivial, but well defined hierarchical relationships both in terms of the generative process, from neurons to muscles and muscle combinations, as well as in terms of domain shift across users and user sessions. Applying standard modeling techniques from the closely related field of Automatic Speech Recognition (ASR), we show strong baseline performance on predicting key-presses using sEMG signals alone. We believe the richness of this task and dataset will facilitate progress in several problems of interest to both the machine learning and neuroscientific communities. Dataset and code can be accessed at this https URL.

Title: Generative Adversarial Patches for Physical Attacks on Cross-Modal Pedestrian Re-Identification

• Authors: Yue Su, Hao Li, Maoguo Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20097
• Pdf URL: https://arxiv.org/pdf/2410.20097
• Copy Paste: [[2410.20097]] Generative Adversarial Patches for Physical Attacks on Cross-Modal Pedestrian Re-Identification(https://arxiv.org/abs/2410.20097)
• Keywords: security, attack, transformer, generative
• Abstract: Visible-infrared pedestrian Re-identification (VI-ReID) aims to match pedestrian images captured by infrared cameras and visible cameras. However, VI-ReID, like other traditional cross-modal image matching tasks, poses significant challenges due to its human-centered nature. This is evidenced by the shortcomings of existing methods, which struggle to extract common features across modalities, while losing valuable information when bridging the gap between them in the implicit feature space, potentially compromising security. To address this vulnerability, this paper introduces the first physical adversarial attack against VI-ReID models. Our method, termed Edge-Attack, specifically tests the models' ability to leverage deep-level implicit features by focusing on edge information, the most salient explicit feature differentiating individuals across modalities. Edge-Attack utilizes a novel two-step approach. First, a multi-level edge feature extractor is trained in a self-supervised manner to capture discriminative edge representations for each individual. Second, a generative model based on Vision Transformer Generative Adversarial Networks (ViTGAN) is employed to generate adversarial patches conditioned on the extracted edge features. By applying these patches to pedestrian clothing, we create realistic, physically-realizable adversarial samples. This black-box, self-supervised approach ensures the generalizability of our attack against various VI-ReID models. Extensive experiments on SYSU-MM01 and RegDB datasets, including real-world deployments, demonstrate the effectiveness of Edge- Attack in significantly degrading the performance of state-of-the-art VI-ReID methods.

Title: Self-Normalized Resets for Plasticity in Continual Learning

• Authors: Vivek F. Farias, Adam D. Jozefiak
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20098
• Pdf URL: https://arxiv.org/pdf/2410.20098
• Copy Paste: [[2410.20098]] Self-Normalized Resets for Plasticity in Continual Learning(https://arxiv.org/abs/2410.20098)
• Keywords: robust
• Abstract: Plasticity Loss is an increasingly important phenomenon that refers to the empirical observation that as a neural network is continually trained on a sequence of changing tasks, its ability to adapt to a new task diminishes over time. We introduce Self-Normalized Resets (SNR), a simple adaptive algorithm that mitigates plasticity loss by resetting a neuron's weights when evidence suggests its firing rate has effectively dropped to zero. Across a battery of continual learning problems and network architectures, we demonstrate that SNR consistently attains superior performance compared to its competitor algorithms. We also demonstrate that SNR is robust to its sole hyperparameter, its rejection percentile threshold, while competitor algorithms show significant sensitivity. SNR's threshold-based reset mechanism is motivated by a simple hypothesis test that we derive. Seen through the lens of this hypothesis test, competing reset proposals yield suboptimal error rates in correctly detecting inactive neurons, potentially explaining our experimental observations. We also conduct a theoretical investigation of the optimization landscape for the problem of learning a single ReLU. We show that even when initialized adversarially, an idealized version of SNR learns the target ReLU, while regularization-based approaches can fail to learn.

Title: Anatomical 3D Style Transfer Enabling Efficient Federated Learning with Extremely Low Communication Costs

• Authors: Yuto Shibata, Yasunori Kudo, Yohei Sugawara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20102
• Pdf URL: https://arxiv.org/pdf/2410.20102
• Copy Paste: [[2410.20102]] Anatomical 3D Style Transfer Enabling Efficient Federated Learning with Extremely Low Communication Costs(https://arxiv.org/abs/2410.20102)
• Keywords: privacy, federate, segmentation
• Abstract: In this study, we propose a novel federated learning (FL) approach that utilizes 3D style transfer for the multi-organ segmentation task. The multi-organ dataset, obtained by integrating multiple datasets, has high scalability and can improve generalization performance as the data volume increases. However, the heterogeneity of data owing to different clients with diverse imaging conditions and target organs can lead to severe overfitting of local models. To align models that overfit to different local datasets, existing methods require frequent communication with the central server, resulting in higher communication costs and risk of privacy leakage. To achieve an efficient and safe FL, we propose an Anatomical 3D Frequency Domain Generalization (A3DFDG) method for FL. A3DFDG utilizes structural information of human organs and clusters the 3D styles based on the location of organs. By mixing styles based on these clusters, it preserves the anatomical information and leads models to learn intra-organ diversity, while aligning the optimization of each local model. Experiments indicate that our method can maintain its accuracy even in cases where the communication cost is highly limited (=1.25% of the original cost) while achieving a significant difference compared to baselines, with a higher global dice similarity coefficient score of 4.3%. Despite its simplicity and minimal computational overhead, these results demonstrate that our method has high practicality in real-world scenarios where low communication costs and a simple pipeline are required. The code used in this project will be publicly available.

Title: FedSSP: Federated Graph Learning with Spectral Knowledge and Personalized Preference

• Authors: Zihan Tan, Guancheng Wan, Wenke Huang, Mang Ye
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.20105
• Pdf URL: https://arxiv.org/pdf/2410.20105
• Copy Paste: [[2410.20105]] FedSSP: Federated Graph Learning with Spectral Knowledge and Personalized Preference(https://arxiv.org/abs/2410.20105)
• Keywords: privacy, federate
• Abstract: Personalized Federated Graph Learning (pFGL) facilitates the decentralized training of Graph Neural Networks (GNNs) without compromising privacy while accommodating personalized requirements for non-IID participants. In cross-domain scenarios, structural heterogeneity poses significant challenges for pFGL. Nevertheless, previous pFGL methods incorrectly share non-generic knowledge globally and fail to tailor personalized solutions locally under domain structural shift. We innovatively reveal that the spectral nature of graphs can well reflect inherent domain structural shifts. Correspondingly, our method overcomes it by sharing generic spectral knowledge. Moreover, we indicate the biased message-passing schemes for graph structures and propose the personalized preference module. Combining both strategies, we propose our pFGL framework FedSSP which Shares generic Spectral knowledge while satisfying graph Preferences. Furthermore, We perform extensive experiments on cross-dataset and cross-domain settings to demonstrate the superiority of our framework. The code is available at this https URL.

Title: GiVE: Guiding Visual Encoder to Perceive Overlooked Information

• Authors: Junjie Li, Jianghong Ma, Xiaofeng Zhang, Yuhang Li, Jianyang Shi
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.20109
• Pdf URL: https://arxiv.org/pdf/2410.20109
• Copy Paste: [[2410.20109]] GiVE: Guiding Visual Encoder to Perceive Overlooked Information(https://arxiv.org/abs/2410.20109)
• Keywords: large language model
• Abstract: Multimodal Large Language Models have advanced AI in applications like text-to-video generation and visual question answering. These models rely on visual encoders to convert non-text data into vectors, but current encoders either lack semantic alignment or overlook non-salient objects. We propose the Guiding Visual Encoder to Perceive Overlooked Information (GiVE) approach. GiVE enhances visual representation with an Attention-Guided Adapter (AG-Adapter) module and an Object-focused Visual Semantic Learning module. These incorporate three novel loss terms: Object-focused Image-Text Contrast (OITC) loss, Object-focused Image-Image Contrast (OIIC) loss, and Object-focused Image Discrimination (OID) loss, improving object consideration, retrieval accuracy, and comprehensiveness. Our contributions include dynamic visual focus adjustment, novel loss functions to enhance object retrieval, and the Multi-Object Instruction (MOInst) dataset. Experiments show our approach achieves state-of-the-art performance.

Title: GeoFUSE: A High-Efficiency Surrogate Model for Seawater Intrusion Prediction and Uncertainty Reduction

• Authors: Su Jiang, Chuyang Liu, Dipankar Dwivedi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20118
• Pdf URL: https://arxiv.org/pdf/2410.20118
• Copy Paste: [[2410.20118]] GeoFUSE: A High-Efficiency Surrogate Model for Seawater Intrusion Prediction and Uncertainty Reduction(https://arxiv.org/abs/2410.20118)
• Keywords: robust
• Abstract: Seawater intrusion into coastal aquifers poses a significant threat to groundwater resources, especially with rising sea levels due to climate change. Accurate modeling and uncertainty quantification of this process are crucial but are often hindered by the high computational costs of traditional numerical simulations. In this work, we develop GeoFUSE, a novel deep-learning-based surrogate framework that integrates the U-Net Fourier Neural Operator (U-FNO) with Principal Component Analysis (PCA) and Ensemble Smoother with Multiple Data Assimilation (ESMDA). GeoFUSE enables fast and efficient simulation of seawater intrusion while significantly reducing uncertainty in model predictions. We apply GeoFUSE to a 2D cross-section of the Beaver Creek tidal stream-floodplain system in Washington State. Using 1,500 geological realizations, we train the U-FNO surrogate model to approximate salinity distribution and accumulation. The U-FNO model successfully reduces the computational time from hours (using PFLOTRAN simulations) to seconds, achieving a speedup of approximately 360,000 times while maintaining high accuracy. By integrating measurement data from monitoring wells, the framework significantly reduces geological uncertainty and improves the predictive accuracy of the salinity distribution over a 20-year period. Our results demonstrate that GeoFUSE improves computational efficiency and provides a robust tool for real-time uncertainty quantification and decision making in groundwater management. Future work will extend GeoFUSE to 3D models and incorporate additional factors such as sea-level rise and extreme weather events, making it applicable to a broader range of coastal and subsurface flow systems.

Title: Semantic Feature Decomposition based Semantic Communication System of Images with Large-scale Visual Generation Models

• Authors: Senran Fan, Zhicheng Bao, Chen Dong, Haotai Liang, Xiaodong Xu, Ping Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20126
• Pdf URL: https://arxiv.org/pdf/2410.20126
• Copy Paste: [[2410.20126]] Semantic Feature Decomposition based Semantic Communication System of Images with Large-scale Visual Generation Models(https://arxiv.org/abs/2410.20126)
• Keywords: interpretability
• Abstract: The end-to-end image communication system has been widely studied in the academic community. The escalating demands on image communication systems in terms of data volume, environmental complexity, and task precision require enhanced communication efficiency, anti-noise ability and semantic fidelity. Therefore, we proposed a novel paradigm based on Semantic Feature Decomposition (SeFD) for the integration of semantic communication and large-scale visual generation models to achieve high-performance, highly interpretable and controllable image communication. According to this paradigm, a Texture-Color based Semantic Communication system of Images TCSCI is proposed. TCSCI decomposing the images into their natural language description (text), texture and color semantic features at the transmitter. During the transmission, features are transmitted over the wireless channel, and at the receiver, a large-scale visual generation model is utilized to restore the image through received features. TCSCI can achieve extremely compressed, highly noise-resistant, and visually similar image semantic communication, while ensuring the interpretability and editability of the transmission process. The experiments demonstrate that the TCSCI outperforms traditional image communication systems and existing semantic communication systems under extreme compression with good anti-noise performance and interpretability.

Title: CodePurify: Defend Backdoor Attacks on Neural Code Models via Entropy-based Purification

• Authors: Fangwen Mu, Junjie Wang, Zhuohao Yu, Lin Shi, Song Wang, Mingyang Li, Qing Wang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20136
• Pdf URL: https://arxiv.org/pdf/2410.20136
• Copy Paste: [[2410.20136]] CodePurify: Defend Backdoor Attacks on Neural Code Models via Entropy-based Purification(https://arxiv.org/abs/2410.20136)
• Keywords: defense, attack, robust
• Abstract: Neural code models have found widespread success in tasks pertaining to code intelligence, yet they are vulnerable to backdoor attacks, where an adversary can manipulate the victim model's behavior by inserting triggers into the source code. Recent studies indicate that advanced backdoor attacks can achieve nearly 100% attack success rates on many software engineering tasks. However, effective defense techniques against such attacks remain insufficiently explored. In this study, we propose CodePurify, a novel defense against backdoor attacks on code models through entropy-based purification. Entropy-based purification involves the process of precisely detecting and eliminating the possible triggers in the source code while preserving its semantic information. Within this process, CodePurify first develops a confidence-driven entropy-based measurement to determine whether a code snippet is poisoned and, if so, locates the triggers. Subsequently, it purifies the code by substituting the triggers with benign tokens using a masked language model. We extensively evaluate CodePurify against four advanced backdoor attacks across three representative tasks and two popular code models. The results show that CodePurify significantly outperforms four commonly used defense baselines, improving average defense performance by at least 40%, 40%, and 12% across the three tasks, respectively. These findings highlight the potential of CodePurify to serve as a robust defense against backdoor attacks on neural code models.

Title: FedMABA: Towards Fair Federated Learning through Multi-Armed Bandits Allocation

• Authors: Zhichao Wang, Lin Wang, Yongxin Guo, Ying-Jun Angela Zhang, Xiaoying Tang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.20141
• Pdf URL: https://arxiv.org/pdf/2410.20141
• Copy Paste: [[2410.20141]] FedMABA: Towards Fair Federated Learning through Multi-Armed Bandits Allocation(https://arxiv.org/abs/2410.20141)
• Keywords: privacy, federate, fair
• Abstract: The increasing concern for data privacy has driven the rapid development of federated learning (FL), a privacy-preserving collaborative paradigm. However, the statistical heterogeneity among clients in FL results in inconsistent performance of the server model across various clients. Server model may show favoritism towards certain clients while performing poorly for others, heightening the challenge of fairness. In this paper, we reconsider the inconsistency in client performance distribution and introduce the concept of adversarial multi-armed bandit to optimize the proposed objective with explicit constraints on performance disparities. Practically, we propose a novel multi-armed bandit-based allocation FL algorithm (FedMABA) to mitigate performance unfairness among diverse clients with different data distributions. Extensive experiments, in different Non-I.I.D. scenarios, demonstrate the exceptional performance of FedMABA in enhancing fairness.

Title: Mask-based Membership Inference Attacks for Retrieval-Augmented Generation

• Authors: Mingrui Liu, Sixiao Zhang, Cheng Long
• Subjects: cs.CR, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.20142
• Pdf URL: https://arxiv.org/pdf/2410.20142
• Copy Paste: [[2410.20142]] Mask-based Membership Inference Attacks for Retrieval-Augmented Generation(https://arxiv.org/abs/2410.20142)
• Keywords: protect, attack, membership infer, explainability, large language model
• Abstract: Retrieval-Augmented Generation (RAG) has been an effective approach to mitigate hallucinations in large language models (LLMs) by incorporating up-to-date and domain-specific knowledge. Recently, there has been a trend of storing up-to-date or copyrighted data in RAG knowledge databases instead of using it for LLM training. This practice has raised concerns about Membership Inference Attacks (MIAs), which aim to detect if a specific target document is stored in the RAG system's knowledge database so as to protect the rights of data producers. While research has focused on enhancing the trustworthiness of RAG systems, existing MIAs for RAG systems remain largely insufficient. Previous work either relies solely on the RAG system's judgment or is easily influenced by other documents or the LLM's internal knowledge, which is unreliable and lacks explainability. To address these limitations, we propose a Mask-Based Membership Inference Attacks (MBA) framework. Our framework first employs a masking algorithm that effectively masks a certain number of words in the target document. The masked text is then used to prompt the RAG system, and the RAG system is required to predict the mask values. If the target document appears in the knowledge database, the masked text will retrieve the complete target document as context, allowing for accurate mask prediction. Finally, we adopt a simple yet effective threshold-based method to infer the membership of target document by analyzing the accuracy of mask prediction. Our mask-based approach is more document-specific, making the RAG system's generation less susceptible to distractions from other documents or the LLM's internal knowledge. Extensive experiments demonstrate the effectiveness of our approach compared to existing baseline models.

Title: GFlowNet Fine-tuning for Diverse Correct Solutions in Mathematical Reasoning Tasks

• Authors: Ryoichi Takase, Masaya Tsunokake, Yuta Tsuchiya, Shota Inuzuka
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20147
• Pdf URL: https://arxiv.org/pdf/2410.20147
• Copy Paste: [[2410.20147]] GFlowNet Fine-tuning for Diverse Correct Solutions in Mathematical Reasoning Tasks(https://arxiv.org/abs/2410.20147)
• Keywords: generative, large language model
• Abstract: Mathematical reasoning problems are among the most challenging, as they typically require an understanding of fundamental laws to solve. The laws are universal, but the derivation of the final answer changes depending on how a problem is approached. When training large language models (LLMs), learning the capability of generating such multiple solutions is essential to accelerate their use in mathematical education. To this end, we train LLMs using generative flow network (GFlowNet). Different from reward-maximizing reinforcement learning (RL), GFlowNet fine-tuning seeks to find diverse solutions by training the LLM whose distribution is proportional to a reward function. In numerical experiments, we evaluate GFlowNet fine-tuning and reward-maximizing RL in terms of accuracy and diversity. The results show that GFlowNet fine-tuning derives correct final answers from diverse intermediate reasoning steps, indicating the improvement of the capability of alternative solution generation.

Title: Detection-Guided Deep Learning-Based Model with Spatial Regularization for Lung Nodule Segmentation

• Authors: Jiasen Zhang, Mingrui Yang, Weihong Guo, Brian A. Xavier, Michael Bolen, Xiaojuan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20154
• Pdf URL: https://arxiv.org/pdf/2410.20154
• Copy Paste: [[2410.20154]] Detection-Guided Deep Learning-Based Model with Spatial Regularization for Lung Nodule Segmentation(https://arxiv.org/abs/2410.20154)
• Keywords: segmentation
• Abstract: Lung cancer ranks as one of the leading causes of cancer diagnosis and is the foremost cause of cancer-related mortality worldwide. The early detection of lung nodules plays a pivotal role in improving outcomes for patients, as it enables timely and effective treatment interventions. The segmentation of lung nodules plays a critical role in aiding physicians in distinguishing between malignant and benign lesions. However, this task remains challenging due to the substantial variation in the shapes and sizes of lung nodules, and their frequent proximity to lung tissues, which complicates clear delineation. In this study, we introduce a novel model for segmenting lung nodules in computed tomography (CT) images, leveraging a deep learning framework that integrates segmentation and classification processes. This model is distinguished by its use of feature combination blocks, which facilitate the sharing of information between the segmentation and classification components. Additionally, we employ the classification outcomes as priors to refine the size estimation of the predicted nodules, integrating these with a spatial regularization technique to enhance precision. Furthermore, recognizing the challenges posed by limited training datasets, we have developed an optimal transfer learning strategy that freezes certain layers to further improve performance. The results show that our proposed model can capture the target nodules more accurately compared to other commonly used models. By applying transfer learning, the performance can be further improved, achieving a sensitivity score of 0.885 and a Dice score of 0.814.

Title: Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models

• Authors: Liulei Li, Wenguan Wang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20155
• Pdf URL: https://arxiv.org/pdf/2410.20155
• Copy Paste: [[2410.20155]] Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models(https://arxiv.org/abs/2410.20155)
• Keywords: diffusion, generative
• Abstract: Prevalent human-object interaction (HOI) detection approaches typically leverage large-scale visual-linguistic models to help recognize events involving humans and objects. Though promising, models trained via contrastive learning on text-image pairs often neglect mid/low-level visual cues and struggle at compositional reasoning. In response, we introduce DIFFUSIONHOI, a new HOI detector shedding light on text-to-image diffusion models. Unlike the aforementioned models, diffusion models excel in discerning mid/low-level visual concepts as generative models, and possess strong compositionality to handle novel concepts expressed in text inputs. Considering diffusion models usually emphasize instance objects, we first devise an inversion-based strategy to learn the expression of relation patterns between humans and objects in embedding space. These learned relation embeddings then serve as textual prompts, to steer diffusion models generate images that depict specific interactions, and extract HOI-relevant cues from images without heavy fine-tuning. Benefited from above, DIFFUSIONHOI achieves SOTA performance on three datasets under both regular and zero-shot setups.

Title: Your Image is Secretly the Last Frame of a Pseudo Video

• Authors: Wenlong Chen, Wenlin Chen, Lapo Rastrelli, Yingzhen Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20158
• Pdf URL: https://arxiv.org/pdf/2410.20158
• Copy Paste: [[2410.20158]] Your Image is Secretly the Last Frame of a Pseudo Video(https://arxiv.org/abs/2410.20158)
• Keywords: diffusion, generative
• Abstract: Diffusion models, which can be viewed as a special case of hierarchical variational autoencoders (HVAEs), have shown profound success in generating photo-realistic images. In contrast, standard HVAEs often produce images of inferior quality compared to diffusion models. In this paper, we hypothesize that the success of diffusion models can be partly attributed to the additional self-supervision information for their intermediate latent states provided by corrupted images, which along with the original image form a pseudo video. Based on this hypothesis, we explore the possibility of improving other types of generative models with such pseudo videos. Specifically, we first extend a given image generative model to their video generative model counterpart, and then train the video generative model on pseudo videos constructed by applying data augmentation to the original images. Furthermore, we analyze the potential issues of first-order Markov data augmentation methods, which are typically used in diffusion models, and propose to use more expressive data augmentation to construct more useful information in pseudo videos. Our empirical results on the CIFAR10 and CelebA datasets demonstrate that improved image generation quality can be achieved with additional self-supervised information from pseudo videos.

Title: Causal Abstraction in Model Interpretability: A Compact Survey

• Authors: Yihao Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20161
• Pdf URL: https://arxiv.org/pdf/2410.20161
• Copy Paste: [[2410.20161]] Causal Abstraction in Model Interpretability: A Compact Survey(https://arxiv.org/abs/2410.20161)
• Keywords: interpretability
• Abstract: The pursuit of interpretable artificial intelligence has led to significant advancements in the development of methods that aim to explain the decision-making processes of complex models, such as deep learning systems. Among these methods, causal abstraction stands out as a theoretical framework that provides a principled approach to understanding and explaining the causal mechanisms underlying model behavior. This survey paper delves into the realm of causal abstraction, examining its theoretical foundations, practical applications, and implications for the field of model interpretability.

Title: Prompt Diffusion Robustifies Any-Modality Prompt Learning

• Authors: Yingjun Du, Gaowen Liu, Yuzhang Shang, Yuguang Yao, Ramana Kompella, Cees G. M. Snoek
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20164
• Pdf URL: https://arxiv.org/pdf/2410.20164
• Copy Paste: [[2410.20164]] Prompt Diffusion Robustifies Any-Modality Prompt Learning(https://arxiv.org/abs/2410.20164)
• Keywords: robust, diffusion, generative
• Abstract: Foundation models enable prompt-based classifiers for zero-shot and few-shot learning. Nonetheless, the conventional method of employing fixed prompts suffers from distributional shifts that negatively impact generalizability to unseen samples. This paper introduces prompt diffusion, which uses a diffusion model to gradually refine the prompts to obtain a customized prompt for each sample. Specifically, we first optimize a collection of prompts to obtain over-fitted prompts per sample. Then, we propose a prompt diffusion model within the prompt space, enabling the training of a generative transition process from a random prompt to its overfitted prompt. As we cannot access the label of a test image during inference, our model gradually generates customized prompts solely from random prompts using our trained, prompt diffusion. Our prompt diffusion is generic, flexible, and modality-agnostic, making it a simple plug-and-play module seamlessly embedded into existing prompt learning methods for textual, visual, or multi-modal prompt learning. Our diffusion model uses a fast ODE-based sampling strategy to optimize test sample prompts in just five steps, offering a good trade-off between performance improvement and computational efficiency. For all prompt learning methods tested, adding prompt diffusion yields more robust results for base-to-new generalization, cross-dataset generalization, and domain generalization in classification tasks tested over 15 diverse datasets.

Title: Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model

• Authors: Peng Huang, Bowen Guo, Shuyu Liang, Junhu Fu, Yuanyuan Wang, Yi Guo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20165
• Pdf URL: https://arxiv.org/pdf/2410.20165
• Copy Paste: [[2410.20165]] Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model(https://arxiv.org/abs/2410.20165)
• Keywords: diffusion
• Abstract: Text-To-Image (TTI) generation is significant for controlled and diverse image generation with broad potential applications. Although current medical TTI methods have made some progress in report-to-Chest-Xray (CXR) generation, their generation performance may be limited due to the intrinsic characteristics of medical data. In this paper, we propose a novel disease-knowledge enhanced Diffusion-based TTI learning framework, named Diff-CXR, for medical report-to-CXR generation. First, to minimize the negative impacts of noisy data on generation, we devise a Latent Noise Filtering Strategy that gradually learns the general patterns of anomalies and removes them in the latent space. Then, an Adaptive Vision-Aware Textual Learning Strategy is designed to learn concise and important report embeddings in a domain-specific Vision-Language Model, providing textual guidance for Chest-Xray generation. Finally, by incorporating the general disease knowledge into the pretrained TTI model via a delicate control adapter, a disease-knowledge enhanced diffusion model is introduced to achieve realistic and precise report-to-CXR generation. Experimentally, our Diff-CXR outperforms previous SOTA medical TTI methods by 33.4\% / 8.0\% and 23.8\% / 56.4\% in the FID and mAUC score on MIMIC-CXR and IU-Xray, with the lowest computational complexity at 29.641 GFLOPs. Downstream experiments on three thorax disease classification benchmarks and one CXR-report generation benchmark demonstrate that Diff-CXR is effective in improving classical CXR analysis methods. Notably, models trained on the combination of 1\% real data and synthetic data can achieve a competitive mAUC score compared to models trained on all data, presenting promising clinical applications.

Title: Infectious Disease Forecasting in India using LLM's and Deep Learning

• Authors: Chaitya Shah, Kashish Gandhi, Javal Shah, Kreena Shah, Nilesh Patil, Kiran Bhowmick
• Subjects: cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2410.20168
• Pdf URL: https://arxiv.org/pdf/2410.20168
• Copy Paste: [[2410.20168]] Infectious Disease Forecasting in India using LLM's and Deep Learning(https://arxiv.org/abs/2410.20168)
• Keywords: robust
• Abstract: Many uncontrollable disease outbreaks of the past exposed several vulnerabilities in the healthcare systems worldwide. While advancements in technology assisted in the rapid creation of the vaccinations, there needs to be a pressing focus on the prevention and prediction of such massive outbreaks. Early detection and intervention of an outbreak can drastically reduce its impact on public health while also making the healthcare system more resilient. The complexity of disease transmission dynamics, influence of various directly and indirectly related factors and limitations of traditional approaches are the main bottlenecks in taking preventive actions. Specifically, this paper implements deep learning algorithms and LLM's to predict the severity of infectious disease outbreaks. Utilizing the historic data of several diseases that have spread in India and the climatic data spanning the past decade, the insights from our research aim to assist in creating a robust predictive system for any outbreaks in the future.

Title: Alternatives of Unsupervised Representations of Variables on the Latent Space

• Authors: Alex Glushkovsky
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20172
• Pdf URL: https://arxiv.org/pdf/2410.20172
• Copy Paste: [[2410.20172]] Alternatives of Unsupervised Representations of Variables on the Latent Space(https://arxiv.org/abs/2410.20172)
• Keywords: interpretability
• Abstract: The article addresses the application of unsupervised machine learning to represent variables on the 2D latent space by applying a variational autoencoder (beta-VAE). Representation of variables on low dimensional spaces allows for data visualization, disentanglement of variables based on underlying characteristics, finding of meaningful patterns and outliers, and supports interpretability. Five distinct methods have been introduced to represent variables on the latent space: (1) straightforward transposed, (2) univariate metadata of variables, such as variable statistics, empirical probability density and cumulative distribution functions, (3) adjacency matrices of different metrics, such as correlations, R2 values, Jaccard index, cosine similarity, and mutual information, (4) gradient mappings followed by spot cross product calculation, and (5) combined. Twenty-eight approaches of variable representations by beta-VAE have been considered. The pairwise spot cross product addresses relationships of gradients of two variables along latent space axes, such as orthogonal, confounded positive, confounded negative, and everything in between. The article addresses generalized representations of variables that cover both features and labels. Dealing with categorical variables, reinforced entanglement has been introduced to represent one-hot encoded categories. The article includes three examples: (1) synthetic data with known dependencies, (2) famous MNIST example of handwritten numbers, and (3) real-world multivariate time series of Canadian financial market interest rates. As a result, unsupervised representations of interest rates on the latent space correctly disentangled rates based on their type, such as bonds, T-bills, GICs, or conventional mortgages, positioned bonds and T-bills along a single curve, and ordered rates by their terms along that curve.

Title: A Stack-Propagation Framework for Low-Resource Personalized Dialogue Generation

• Authors: Haoyu Song, Wei-Nan Zhang, Kaiyan Zhang, Ting Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20174
• Pdf URL: https://arxiv.org/pdf/2410.20174
• Copy Paste: [[2410.20174]] A Stack-Propagation Framework for Low-Resource Personalized Dialogue Generation(https://arxiv.org/abs/2410.20174)
• Keywords: transformer
• Abstract: With the resurgent interest in building open-domain dialogue systems, the dialogue generation task has attracted increasing attention over the past few years. This task is usually formulated as a conditional generation problem, which aims to generate a natural and meaningful response given dialogue contexts and specific constraints, such as persona. And maintaining a consistent persona is essential for the dialogue systems to gain trust from the users. Although tremendous advancements have been brought, traditional persona-based dialogue models are typically trained by leveraging a large number of persona-dense dialogue examples. Yet, such persona-dense training data are expensive to obtain, leading to a limited scale. This work presents a novel approach to learning from limited training examples by regarding consistency understanding as a regularization of response generation. To this end, we propose a novel stack-propagation framework for learning a generation and understanding this http URL, the framework stacks a Transformer encoder and two Transformer decoders, where the first decoder models response generation and the second serves as a regularizer and jointly models response generation and consistency understanding. The proposed framework can benefit from the stacked encoder and decoders to learn from much smaller personalized dialogue data while maintaining competitive performance. Under different low-resource settings, subjective and objective evaluations prove that the stack-propagation framework outperforms strong baselines in response quality and persona consistency and largely overcomes the shortcomings of traditional models that rely heavily on the persona-dense dialogue data.

Title: Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning

• Authors: Yuting Tang, Xin-Qiang Cai, Jing-Cheng Pang, Qiyu Wu, Yao-Xiang Ding, Masashi Sugiyama
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20176
• Pdf URL: https://arxiv.org/pdf/2410.20176
• Copy Paste: [[2410.20176]] Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning(https://arxiv.org/abs/2410.20176)
• Keywords: transformer
• Abstract: Reinforcement Learning (RL) empowers agents to acquire various skills by learning from reward signals. Unfortunately, designing high-quality instance-level rewards often demands significant effort. An emerging alternative, RL with delayed reward, focuses on learning from rewards presented periodically, which can be obtained from human evaluators assessing the agent's performance over sequences of behaviors. However, traditional methods in this domain assume the existence of underlying Markovian rewards and that the observed delayed reward is simply the sum of instance-level rewards, both of which often do not align well with real-world scenarios. In this paper, we introduce the problem of RL from Composite Delayed Reward (RLCoDe), which generalizes traditional RL from delayed rewards by eliminating the strong assumption. We suggest that the delayed reward may arise from a more complex structure reflecting the overall contribution of the sequence. To address this problem, we present a framework for modeling composite delayed rewards, using a weighted sum of non-Markovian components to capture the different contributions of individual steps. Building on this framework, we propose Composite Delayed Reward Transformer (CoDeTr), which incorporates a specialized in-sequence attention mechanism to effectively model these contributions. We conduct experiments on challenging locomotion tasks where the agent receives delayed rewards computed from composite functions of observable step rewards. The experimental results indicate that CoDeTr consistently outperforms baseline methods across evaluated metrics. Additionally, we demonstrate that it effectively identifies the most significant time steps within the sequence and accurately predicts rewards that closely reflect the environment feedback.

Title: Copyright-Aware Incentive Scheme for Generative Art Models Using Hierarchical Reinforcement Learning

• Authors: Zhuan Shi, Yifei Song, Xiaoli Tang, Lingjuan Lyu, Boi Faltings
• Subjects: cs.LG, cs.GT
• Abstract URL: https://arxiv.org/abs/2410.20180
• Pdf URL: https://arxiv.org/pdf/2410.20180
• Copy Paste: [[2410.20180]] Copyright-Aware Incentive Scheme for Generative Art Models Using Hierarchical Reinforcement Learning(https://arxiv.org/abs/2410.20180)
• Keywords: protect, diffusion, generative
• Abstract: Generative art using Diffusion models has achieved remarkable performance in image generation and text-to-image tasks. However, the increasing demand for training data in generative art raises significant concerns about copyright infringement, as models can produce images highly similar to copyrighted works. Existing solutions attempt to mitigate this by perturbing Diffusion models to reduce the likelihood of generating such images, but this often compromises model performance. Another approach focuses on economically compensating data holders for their contributions, yet it fails to address copyright loss adequately. Our approach begin with the introduction of a novel copyright metric grounded in copyright law and court precedents on infringement. We then employ the TRAK method to estimate the contribution of data holders. To accommodate the continuous data collection process, we divide the training into multiple rounds. Finally, We designed a hierarchical budget allocation method based on reinforcement learning to determine the budget for each round and the remuneration of the data holder based on the data holder's contribution and copyright loss in each round. Extensive experiments across three datasets show that our method outperforms all eight benchmarks, demonstrating its effectiveness in optimizing budget distribution in a copyright-aware manner. To the best of our knowledge, this is the first technical work that introduces to incentive contributors and protect their copyrights by compensating them.

Title: Chemical Language Model Linker: blending text and molecules with modular adapters

• Authors: Yifan Deng, Spencer S. Ericksen, Anthony Gitter
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.20182
• Pdf URL: https://arxiv.org/pdf/2410.20182
• Copy Paste: [[2410.20182]] Chemical Language Model Linker: blending text and molecules with modular adapters(https://arxiv.org/abs/2410.20182)
• Keywords: generative, large language model
• Abstract: The development of large language models and multi-modal models has enabled the appealing idea of generating novel molecules from text descriptions. Generative modeling would shift the paradigm from relying on large-scale chemical screening to find molecules with desired properties to directly generating those molecules. However, multi-modal models combining text and molecules are often trained from scratch, without leveraging existing high-quality pretrained models. That approach consumes more computational resources and prohibits model scaling. In contrast, we propose a lightweight adapter-based strategy named Chemical Language Model Linker (ChemLML). ChemLML blends the two single domain models and obtains conditional molecular generation from text descriptions while still operating in the specialized embedding spaces of the molecular domain. ChemLML can tailor diverse pretrained text models for molecule generation by training relatively few adapter parameters. We find that the choice of molecular representation used within ChemLML, SMILES versus SELFIES, has a strong influence on conditional molecular generation performance. SMILES is often preferable despite not guaranteeing valid molecules. We raise issues in using the large PubChem dataset of molecules and their associated descriptions for evaluating molecule generation and provide a filtered version of the dataset as a generation test set. To demonstrate how ChemLML could be used in practice, we generate candidate protein inhibitors and use docking to assess their quality.

Title: Uncertainty-Penalized Direct Preference Optimization

• Authors: Sam Houliston, Alizée Pace, Alexander Immer, Gunnar Rätsch
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.20187
• Pdf URL: https://arxiv.org/pdf/2410.20187
• Copy Paste: [[2410.20187]] Uncertainty-Penalized Direct Preference Optimization(https://arxiv.org/abs/2410.20187)
• Keywords: large language model
• Abstract: Aligning Large Language Models (LLMs) to human preferences in content, style, and presentation is challenging, in part because preferences are varied, context-dependent, and sometimes inherently ambiguous. While successful, Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are prone to the issue of proxy reward overoptimization. Analysis of the DPO loss reveals a critical need for regularization for mislabeled or ambiguous preference pairs to avoid reward hacking. In this work, we develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes, inspired by offline reinforcement learning. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. Evaluation of the methods is performed with GPT2 Medium on the Anthropic-HH dataset using a model ensemble to obtain uncertainty estimates, and shows improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.

Title: Transferable Adversarial Attacks on SAM and Its Downstream Models

• Authors: Song Xia, Wenhan Yang, Yi Yu, Xun Lin, Henghui Ding, Lingyu Duan, Xudong Jiang
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20197
• Pdf URL: https://arxiv.org/pdf/2410.20197
• Copy Paste: [[2410.20197]] Transferable Adversarial Attacks on SAM and Its Downstream Models(https://arxiv.org/abs/2410.20197)
• Keywords: attack, robust
• Abstract: The utilization of large foundational models has a dilemma: while fine-tuning downstream tasks from them holds promise for making use of the well-generalized knowledge in practical applications, their open accessibility also poses threats of adverse usage. This paper, for the first time, explores the feasibility of adversarial attacking various downstream models fine-tuned from the segment anything model (SAM), by solely utilizing the information from the open-sourced SAM. In contrast to prevailing transfer-based adversarial attacks, we demonstrate the existence of adversarial dangers even without accessing the downstream task and dataset to train a similar surrogate model. To enhance the effectiveness of the adversarial attack towards models fine-tuned on unknown datasets, we propose a universal meta-initialization (UMI) algorithm to extract the intrinsic vulnerability inherent in the foundation model, which is then utilized as the prior knowledge to guide the generation of adversarial perturbations. Moreover, by formulating the gradient difference in the attacking process between the open-sourced SAM and its fine-tuned downstream models, we theoretically demonstrate that a deviation occurs in the adversarial update direction by directly maximizing the distance of encoded feature embeddings in the open-sourced SAM. Consequently, we propose a gradient robust loss that simulates the associated uncertainty with gradient-based noise augmentation to enhance the robustness of generated adversarial examples (AEs) towards this deviation, thus improving the transferability. Extensive experiments demonstrate the effectiveness of the proposed universal meta-initialized and gradient robust adversarial attack (UMI-GRAT) toward SAMs and their downstream models. Code is available at this https URL.

Title: Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs

• Authors: Houman Mehrafarin, Arash Eshghi, Ioannis Konstas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20200
• Pdf URL: https://arxiv.org/pdf/2410.20200
• Copy Paste: [[2410.20200]] Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs(https://arxiv.org/abs/2410.20200)
• Keywords: large language model
• Abstract: Evaluating Large Language Models (LLMs) on reasoning benchmarks demonstrates their ability to solve compositional questions. However, little is known of whether these models engage in genuine logical reasoning or simply rely on implicit cues to generate answers. In this paper, we investigate the transitive reasoning capabilities of two distinct LLM architectures, LLaMA 2 and Flan-T5, by manipulating facts within two compositional datasets: QASC and Bamboogle. We controlled for potential cues that might influence the models' performance, including (a) word/phrase overlaps across sections of test input; (b) models' inherent knowledge during pre-training or fine-tuning; and (c) Named Entities. Our findings reveal that while both models leverage (a), Flan-T5 shows more resilience to experiments (b and c), having less variance than LLaMA 2. This suggests that models may develop an understanding of transitivity through fine-tuning on knowingly relevant datasets, a hypothesis we leave to future work.

Title: An Efficient Watermarking Method for Latent Diffusion Models via Low-Rank Adaptation

• Authors: Dongdong Lin, Yue Li, Benedetta Tondi, Bin Li, Mauro Barni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20202
• Pdf URL: https://arxiv.org/pdf/2410.20202
• Copy Paste: [[2410.20202]] An Efficient Watermarking Method for Latent Diffusion Models via Low-Rank Adaptation(https://arxiv.org/abs/2410.20202)
• Keywords: watermark, diffusion, generative
• Abstract: The rapid proliferation of deep neural networks (DNNs) is driving a surge in model watermarking technologies, as the trained deep models themselves serve as intellectual properties. The core of existing model watermarking techniques involves modifying or tuning the models' weights. However, with the emergence of increasingly complex models, ensuring the efficiency of watermarking process is essential to manage the growing computational demands. Prioritizing efficiency not only optimizes resource utilization, making the watermarking process more applicable, but also minimizes potential impacts on model performance. In this letter, we propose an efficient watermarking method for latent diffusion models (LDMs) which is based on Low-Rank Adaptation (LoRA). We specifically choose to add trainable low-rank matrices to the existing weight matrices of the models to embed watermark, while keeping the original weights frozen. Moreover, we also propose a dynamic loss weight tuning algorithm to balance the generative task with the watermark embedding task, ensuring that the model can be watermarked with a limited impact on the quality of the generated images. Experimental results show that the proposed method ensures fast watermark embedding and maintains a very low bit error rate of the watermark, a high-quality of the generated image, and a zero false negative rate (FNR) for verification.

Title: Generative AI in Health Economics and Outcomes Research: A Taxonomy of Key Definitions and Emerging Applications, an ISPOR Working Group Report

• Authors: Rachael Fleurence, Xiaoyan Wang, Jiang Bian, Mitchell K. Higashi, Turgay Ayer, Hua Xu, Dalia Dawoud, Jagpreet Chhatwal
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.20204
• Pdf URL: https://arxiv.org/pdf/2410.20204
• Copy Paste: [[2410.20204]] Generative AI in Health Economics and Outcomes Research: A Taxonomy of Key Definitions and Emerging Applications, an ISPOR Working Group Report(https://arxiv.org/abs/2410.20204)
• Keywords: interpretability, generative
• Abstract: Objective: This article offers a taxonomy of generative artificial intelligence (AI) for health economics and outcomes research (HEOR), explores its emerging applications, and outlines methods to enhance the accuracy and reliability of AI-generated outputs. Methods: The review defines foundational generative AI concepts and highlights current HEOR applications, including systematic literature reviews, health economic modeling, real-world evidence generation, and dossier development. Approaches such as prompt engineering (zero-shot, few-shot, chain-of-thought, persona pattern prompting), retrieval-augmented generation, model fine-tuning, and the use of domain-specific models are introduced to improve AI accuracy and reliability. Results: Generative AI shows significant potential in HEOR, enhancing efficiency, productivity, and offering novel solutions to complex challenges. Foundation models are promising in automating complex tasks, though challenges remain in scientific reliability, bias, interpretability, and workflow integration. The article discusses strategies to improve the accuracy of these AI tools. Conclusion: Generative AI could transform HEOR by increasing efficiency and accuracy across various applications. However, its full potential can only be realized by building HEOR expertise and addressing the limitations of current AI technologies. As AI evolves, ongoing research and innovation will shape its future role in the field.

Title: Looking Beyond The Top-1: Transformers Determine Top Tokens In Order

• Authors: Daria Lioubashevski, Tomer Schlank, Gabriel Stanovsky, Ariel Goldstein
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20210
• Pdf URL: https://arxiv.org/pdf/2410.20210
• Copy Paste: [[2410.20210]] Looking Beyond The Top-1: Transformers Determine Top Tokens In Order(https://arxiv.org/abs/2410.20210)
• Keywords: transformer
• Abstract: Understanding the inner workings of Transformers is crucial for achieving more accurate and efficient predictions. In this work, we analyze the computation performed by Transformers in the layers after the top-1 prediction has become fixed, which has been previously referred to as the "saturation event". We expand the concept of saturation events for top-k tokens, demonstrating that similar saturation events occur across language, vision, and speech models. We find that these saturation events happen in order of the corresponding tokens' ranking, i.e., the model first decides on the top ranking token, then the second highest ranking token, and so on. This phenomenon seems intrinsic to the Transformer architecture, occurring across different architectural variants (decoder-only, encoder-only, and to a lesser extent full-Transformer), and even in untrained Transformers. We propose an underlying mechanism of task transition for this sequential saturation, where task k corresponds to predicting the k-th most probable token, and the saturation events are in fact discrete transitions between the tasks. In support of this we show that it is possible to predict the current task from hidden layer embedding. Furthermore, using an intervention method we demonstrate that we can cause the model to switch from one task to the next. Finally, leveraging our findings, we introduce a novel token-level early-exit strategy, which surpasses existing methods in balancing performance and efficiency.

Title: DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learning

• Authors: Xinyu Tang, Xiaolei Wang, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20215
• Pdf URL: https://arxiv.org/pdf/2410.20215
• Copy Paste: [[2410.20215]] DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learning(https://arxiv.org/abs/2410.20215)
• Keywords: large language model
• Abstract: Zero-shot in-context learning (ZS-ICL) aims to conduct in-context learning (ICL) without using human-annotated demonstrations. Most ZS-ICL methods use large language models (LLMs) to generate (input, label) pairs as pseudo-demonstrations and leverage historical pseudo-demonstrations to help solve the current problem. They assume that problems are from the same task and traverse them in a random order. However, in real-world scenarios, problems usually come from diverse tasks, and only a few belong to the same task. The random traversing order may generate unreliable pseudo-demonstrations and lead to error accumulation. To address this problem, we reformulate ZS-ICL as a planning problem and propose a Demonstration-aware Monte Carlo Tree Search (MCTS) approach (DAWN-ICL), which leverages MCTS to strategically plan the problem-solving trajectories for ZS-ICL. In addition, to achieve effective and efficient Q value estimation, we propose a novel demonstration-aware Q-value function and use it to enhance the selection phase and accelerate the expansion and simulation phases in MCTS. Extensive experiments demonstrate the effectiveness and efficiency of DAWN-ICL on in-domain and cross-domain scenarios, and it even outperforms ICL using human-annotated labels. The code is available at this https URL.

Title: Generative linguistics contribution to artificial intelligence: Where this contribution lies?

• Authors: Mohammed Q. Shormani (Ibb University, University of Cyprus)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20221
• Pdf URL: https://arxiv.org/pdf/2410.20221
• Copy Paste: [[2410.20221]] Generative linguistics contribution to artificial intelligence: Where this contribution lies?(https://arxiv.org/abs/2410.20221)
• Keywords: generative, large language model
• Abstract: This article aims to characterize Generative linguistics (GL) contribution to artificial intelligence (AI), alluding to the debate among linguists and AI scientists on whether linguistics belongs to humanities or science. In this article, I will try not to be biased as a linguist, studying the phenomenon from an independent scientific perspective. The article walks the researcher/reader through the scientific theorems and rationales involved in AI which belong from GL, specifically the Chomsky School. It, thus, provides good evidence from syntax, semantics, language faculty, Universal Grammar, computational system of human language, language acquisition, human brain, programming languages (e.g. Python), Large Language Models, and unbiased AI scientists that this contribution is huge, and that this contribution cannot be denied. It concludes that however the huge GL contribution to AI, there are still points of divergence including the nature and type of language input."

Title: CAVE: Classifying Abnormalities in Video Capsule Endoscopy

• Authors: Ishita Harish, Saurav Mishra, Neha Bhadoria, Rithik Kumar, Madhav Arora, Syed Rameem Zahra, Ankur Gupta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20231
• Pdf URL: https://arxiv.org/pdf/2410.20231
• Copy Paste: [[2410.20231]] CAVE: Classifying Abnormalities in Video Capsule Endoscopy(https://arxiv.org/abs/2410.20231)
• Keywords: robust, extraction
• Abstract: In this study, we explore an ensemble-based approach to improve classification accuracy in complex image datasets. Utilizing a Convolutional Block Attention Module (CBAM) alongside a Deep Neural Network (DNN) we leverage the unique feature-extraction capabilities of each model to enhance the overall accuracy. Additional models, such as Random Forest, XGBoost, Support Vector Machine (SVM), and K-Nearest Neighbors (KNN), are introduced to further diversify the predictive power of our ensemble. By leveraging these methods, the proposed approach provides robust feature discrimination and improved classification results. Experimental evaluations demonstrate that the ensemble achieves higher accuracy and robustness across challenging and imbalanced classes, showing significant promise for broader applications in computer vision tasks.

Title: SAFE setup for generative molecular design

• Authors: Yassir El Mesbahi, Emmanuel Noutahi
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.20232
• Pdf URL: https://arxiv.org/pdf/2410.20232
• Copy Paste: [[2410.20232]] SAFE setup for generative molecular design(https://arxiv.org/abs/2410.20232)
• Keywords: robust, generative
• Abstract: SMILES-based molecular generative models have been pivotal in drug design but face challenges in fragment-constrained tasks. To address this, the Sequential Attachment-based Fragment Embedding (SAFE) representation was recently introduced as an alternative that streamlines those tasks. In this study, we investigate the optimal setups for training SAFE generative models, focusing on dataset size, data augmentation through randomization, model architecture, and bond disconnection algorithms. We found that larger, more diverse datasets improve performance, with the LLaMA architecture using Rotary Positional Embedding proving most robust. SAFE-based models also consistently outperform SMILES-based approaches in scaffold decoration and linker design, particularly with BRICS decomposition yielding the best results. These insights highlight key factors that significantly impact the efficacy of SAFE-based generative models.

Title: Enhancing CNN Classification with Lamarckian Memetic Algorithms and Local Search

• Authors: Akhilbaran Ghosh, Rama Sai Adithya Kalidindi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20234
• Pdf URL: https://arxiv.org/pdf/2410.20234
• Copy Paste: [[2410.20234]] Enhancing CNN Classification with Lamarckian Memetic Algorithms and Local Search(https://arxiv.org/abs/2410.20234)
• Keywords: robust
• Abstract: Optimization is critical for optimal performance in deep neural networks (DNNs). Traditional gradient-based methods often face challenges like local minima entrapment. This paper explores population-based metaheuristic optimization algorithms for image classification networks. We propose a novel approach integrating a two-stage training technique with population-based optimization algorithms incorporating local search capabilities. Our experiments demonstrate that the proposed method outperforms state-of-the-art gradient-based techniques, such as ADAM, in accuracy and computational efficiency, particularly with high computational complexity and numerous trainable parameters. The results suggest that our approach offers a robust alternative to traditional methods for weight optimization in convolutional neural networks (CNNs). Future work will explore integrating adaptive mechanisms for parameter tuning and applying the proposed method to other types of neural networks and real-time applications.

Title: A Survey of Large Language Models for Arabic Language and its Dialects

• Authors: Malak Mashaabi, Shahad Al-Khalifa, Hend Al-Khalifa
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20238
• Pdf URL: https://arxiv.org/pdf/2410.20238
• Copy Paste: [[2410.20238]] A Survey of Large Language Models for Arabic Language and its Dialects(https://arxiv.org/abs/2410.20238)
• Keywords: large language model
• Abstract: This survey offers a comprehensive overview of Large Language Models (LLMs) designed for Arabic language and its dialects. It covers key architectures, including encoder-only, decoder-only, and encoder-decoder models, along with the datasets used for pre-training, spanning Classical Arabic, Modern Standard Arabic, and Dialectal Arabic. The study also explores monolingual, bilingual, and multilingual LLMs, analyzing their architectures and performance across downstream tasks, such as sentiment analysis, named entity recognition, and question answering. Furthermore, it assesses the openness of Arabic LLMs based on factors, such as source code availability, training data, model weights, and documentation. The survey highlights the need for more diverse dialectal datasets and attributes the importance of openness for research reproducibility and transparency. It concludes by identifying key challenges and opportunities for future research and stressing the need for more inclusive and representative models.

Title: SmartX Intelligent Sec: A Security Framework Based on Machine Learning and eBPF/XDP

• Authors: Talaya Farasat, JongWon Kim, Joachim Posegga
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.20244
• Pdf URL: https://arxiv.org/pdf/2410.20244
• Copy Paste: [[2410.20244]] SmartX Intelligent Sec: A Security Framework Based on Machine Learning and eBPF/XDP(https://arxiv.org/abs/2410.20244)
• Keywords: security
• Abstract: Information and Communication Technologies (ICT) infrastructures are becoming increasingly complex day by day, facing numerous challenges to support the latest networking paradigms. Security is undeniably a critical component for the effective functioning of these advanced ICT infrastructures. By considering the current network security challenges, we propose SmartX Intelligent Sec, an innovative intelligent security framework. SmartX Intelligent Sec leverages a combination of the lightweight extended Berkeley Packet Filter/eXpress Data Path (eBPF/XDP) for efficient network packet capturing and filtering malicious network traffic, and a Bidirectional Long Short-Term Memory (BiLSTM) classifier for network threat detection. Our real-time prototype demonstrates that SmartX Intelligent Sec offers comprehensive automation features, enabling continuous network packet capturing, effective network threat detection, and efficient filtering of malicious network traffic. This framework ensures enhanced security and operational efficiency for modern ICT infrastructures.

Title: Model Equality Testing: Which Model Is This API Serving?

• Authors: Irena Gao, Percy Liang, Carlos Guestrin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20247
• Pdf URL: https://arxiv.org/pdf/2410.20247
• Copy Paste: [[2410.20247]] Model Equality Testing: Which Model Is This API Serving?(https://arxiv.org/abs/2410.20247)
• Keywords: watermark, large language model
• Abstract: Users often interact with large language models through black-box inference APIs, both for closed- and open-weight models (e.g., Llama models are popularly accessed via Amazon Bedrock and Azure AI Studio). In order to cut costs or add functionality, API providers may quantize, watermark, or finetune the underlying model, changing the output distribution -- often without notifying users. We formalize detecting such distortions as Model Equality Testing, a two-sample testing problem, where the user collects samples from the API and a reference distribution and conducts a statistical test to see if the two distributions are the same. We find that tests based on the Maximum Mean Discrepancy between distributions are powerful for this task: a test built on a simple string kernel achieves a median of 77.4% power against a range of distortions, using an average of just 10 samples per prompt. We then apply this test to commercial inference APIs for four Llama models, finding that 11 out of 31 endpoints serve different distributions than reference weights released by Meta.

Title: Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning

• Authors: Sullam Jeoung, Goeric Huybrechts, Bhavana Ganesh, Aram Galstyan, Sravan Bodapati
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20252
• Pdf URL: https://arxiv.org/pdf/2410.20252
• Copy Paste: [[2410.20252]] Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning(https://arxiv.org/abs/2410.20252)
• Keywords: large language model
• Abstract: Understanding long-form video content presents significant challenges due to its temporal complexity and the substantial computational resources required. In this work, we propose an agent-based approach to enhance both the efficiency and effectiveness of long-form video understanding by utilizing large language models (LLMs) and their tool-harnessing ability. A key aspect of our method is query-adaptive frame sampling, which leverages the reasoning capabilities of LLMs to process only the most relevant frames in real-time, and addresses an important limitation of existing methods which typically involve sampling redundant or irrelevant frames. To enhance the reasoning abilities of our video-understanding agent, we leverage the self-reflective capabilities of LLMs to provide verbal reinforcement to the agent, which leads to improved performance while minimizing the number of frames accessed. We evaluate our method across several video understanding benchmarks and demonstrate that not only it enhances state-of-the-art performance but also improves efficiency by reducing the number of frames sampled.

Title: Equivariant Blurring Diffusion for Hierarchical Molecular Conformer Generation

• Authors: Jiwoong Park, Yang Shen
• Subjects: cs.LG, cs.AI, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.20255
• Pdf URL: https://arxiv.org/pdf/2410.20255
• Copy Paste: [[2410.20255]] Equivariant Blurring Diffusion for Hierarchical Molecular Conformer Generation(https://arxiv.org/abs/2410.20255)
• Keywords: diffusion, generative
• Abstract: How can diffusion models process 3D geometries in a coarse-to-fine manner, akin to our multiscale view of the world? In this paper, we address the question by focusing on a fundamental biochemical problem of generating 3D molecular conformers conditioned on molecular graphs in a multiscale manner. Our approach consists of two hierarchical stages: i) generation of coarse-grained fragment-level 3D structure from the molecular graph, and ii) generation of fine atomic details from the coarse-grained approximated structure while allowing the latter to be adjusted simultaneously. For the challenging second stage, which demands preserving coarse-grained information while ensuring SE(3) equivariance, we introduce a novel generative model termed Equivariant Blurring Diffusion (EBD), which defines a forward process that moves towards the fragment-level coarse-grained structure by blurring the fine atomic details of conformers, and a reverse process that performs the opposite operation using equivariant networks. We demonstrate the effectiveness of EBD by geometric and chemical comparison to state-of-the-art denoising diffusion models on a benchmark of drug-like molecules. Ablation studies draw insights on the design of EBD by thoroughly analyzing its architecture, which includes the design of the loss function and the data corruption process. Codes are released at this https URL .

Title: FL-DABE-BC: A Privacy-Enhanced, Decentralized Authentication, and Secure Communication for Federated Learning Framework with Decentralized Attribute-Based Encryption and Blockchain for IoT Scenarios

• Authors: Sathwik Narkedimilli, Amballa Venkata Sriram, Satvik Raghav
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.20259
• Pdf URL: https://arxiv.org/pdf/2410.20259
• Copy Paste: [[2410.20259]] FL-DABE-BC: A Privacy-Enhanced, Decentralized Authentication, and Secure Communication for Federated Learning Framework with Decentralized Attribute-Based Encryption and Blockchain for IoT Scenarios(https://arxiv.org/abs/2410.20259)
• Keywords: secure, security, privacy, robust, federate
• Abstract: This study proposes an advanced Federated Learning (FL) framework designed to enhance data privacy and security in IoT environments by integrating Decentralized Attribute-Based Encryption (DABE), Homomorphic Encryption (HE), Secure Multi-Party Computation (SMPC), and Blockchain technology. Unlike traditional FL, our framework enables secure, decentralized authentication and encryption directly on IoT devices using DABE, allowing sensitive data to remain locally encrypted. Homomorphic Encryption permits computations on encrypted data, and SMPC ensures privacy in collaborative computations, while Blockchain technology provides transparent, immutable record-keeping for all transactions and model updates. Local model weights are encrypted and transmitted to fog layers for aggregation using HE and SMPC, then iteratively refined by the central server using differential privacy to safeguard against data leakage. This secure, privacy-preserving FL framework delivers a robust solution for efficient model training and real-time analytics across distributed IoT devices, offering significant advancements in secure decentralized learning for IoT applications.

Title: You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Models

• Authors: Eric Slyman, Anirudh Kanneganti, Sanghyun Hong, Stefan Lee
• Subjects: cs.CV, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20265
• Pdf URL: https://arxiv.org/pdf/2410.20265
• Copy Paste: [[2410.20265]] You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Models(https://arxiv.org/abs/2410.20265)
• Keywords: fair
• Abstract: We study the impact of a standard practice in compressing foundation vision-language models - quantization - on the models' ability to produce socially-fair outputs. In contrast to prior findings with unimodal models that compression consistently amplifies social biases, our extensive evaluation of four quantization settings across three datasets and three CLIP variants yields a surprising result: while individual models demonstrate bias, we find no consistent change in bias magnitude or direction across a population of compressed models due to quantization.

Title: Library Learning Doesn't: The Curious Case of the Single-Use "Library"

• Authors: Ian Berlot-Attwell, Frank Rudzicz, Xujie Si
• Subjects: cs.LG, cs.CL, cs.SC
• Abstract URL: https://arxiv.org/abs/2410.20274
• Pdf URL: https://arxiv.org/pdf/2410.20274
• Copy Paste: [[2410.20274]] Library Learning Doesn't: The Curious Case of the Single-Use "Library"(https://arxiv.org/abs/2410.20274)
• Keywords: large language model
• Abstract: Advances in Large Language Models (LLMs) have spurred a wave of LLM library learning systems for mathematical reasoning. These systems aim to learn a reusable library of tools, such as formal Isabelle lemmas or Python programs that are tailored to a family of tasks. Many of these systems are inspired by the human structuring of knowledge into reusable and extendable concepts, but do current methods actually learn reusable libraries of tools? We study two library learning systems for mathematics which both reported increased accuracy: LEGO-Prover and TroVE. We find that function reuse is extremely infrequent on miniF2F and MATH. Our followup ablation experiments suggest that, rather than reuse, self-correction and self-consistency are the primary drivers of the observed performance gains. Our code and data are available at this https URL

Title: MarDini: Masked Autoregressive Diffusion for Video Generation at Scale

• Authors: Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xie, Xiao Han, Juan C. Pérez, Ding Liu, Kumara Kahatapitiya, Menglin Jia, Jui-Chieh Wu, Sen He, Tao Xiang, Jürgen Schmidhuber, Juan-Manuel Pérez-Rúa
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20280
• Pdf URL: https://arxiv.org/pdf/2410.20280
• Copy Paste: [[2410.20280]] MarDini: Masked Autoregressive Diffusion for Video Generation at Scale(https://arxiv.org/abs/2410.20280)
• Keywords: diffusion
• Abstract: We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses on spatial generation in an asymmetric network design: i) a MAR-based planning model containing most of the parameters generates planning signals for each masked frame using low-resolution input; ii) a lightweight generation model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini's MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the computational resources to the low-resolution planning model, making computationally expensive but important spatio-temporal attention feasible at scale. MarDini sets a new state-of-the-art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models.

Title: Proactive Fraud Defense: Machine Learning's Evolving Role in Protecting Against Online Fraud

• Authors: Md Kamrul Hasan Chy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20281
• Pdf URL: https://arxiv.org/pdf/2410.20281
• Copy Paste: [[2410.20281]] Proactive Fraud Defense: Machine Learning's Evolving Role in Protecting Against Online Fraud(https://arxiv.org/abs/2410.20281)
• Keywords: protect, defense
• Abstract: As online fraud becomes more sophisticated and pervasive, traditional fraud detection methods are struggling to keep pace with the evolving tactics employed by fraudsters. This paper explores the transformative role of machine learning in addressing these challenges by offering more advanced, scalable, and adaptable solutions for fraud detection and prevention. By analyzing key models such as Random Forest, Neural Networks, and Gradient Boosting, this paper highlights the strengths of machine learning in processing vast datasets, identifying intricate fraud patterns, and providing real-time predictions that enable a proactive approach to fraud prevention. Unlike rule-based systems that react after fraud has occurred, machine learning models continuously learn from new data, adapting to emerging fraud schemes and reducing false positives, which ultimately minimizes financial losses. This research emphasizes the potential of machine learning to revolutionize fraud detection frameworks by making them more dynamic, efficient, and capable of handling the growing complexity of fraud across various industries. Future developments in machine learning, including deep learning and hybrid models, are expected to further enhance the predictive accuracy and applicability of these systems, ensuring that organizations remain resilient in the face of new and emerging fraud tactics.

Title: Classification under strategic adversary manipulation using pessimistic bilevel optimisation

• Authors: David Benfield, Stefano Coniglio, Martin Kunc, Phan Tu Vuong, Alain Zemkoho
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2410.20284
• Pdf URL: https://arxiv.org/pdf/2410.20284
• Copy Paste: [[2410.20284]] Classification under strategic adversary manipulation using pessimistic bilevel optimisation(https://arxiv.org/abs/2410.20284)
• Keywords: security, attack
• Abstract: Adversarial machine learning concerns situations in which learners face attacks from active adversaries. Such scenarios arise in applications such as spam email filtering, malware detection and fake-image generation, where security methods must be actively updated to keep up with the ever improving generation of malicious this http URL model these interactions between the learner and the adversary as a game and formulate the problem as a pessimistic bilevel optimisation problem with the learner taking the role of the leader. The adversary, modelled as a stochastic data generator, takes the role of the follower, generating data in response to the classifier. While existing models rely on the assumption that the adversary will choose the least costly solution leading to a convex lower-level problem with a unique solution, we present a novel model and solution method which do not make such assumptions. We compare these to the existing approach and see significant improvements in performance suggesting that relaxing these assumptions leads to a more realistic model.

Title: AI-Driven Cyber Threat Intelligence Automation

• Authors: Shrit Shah, Fatemeh Khoda Parast
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.20287
• Pdf URL: https://arxiv.org/pdf/2410.20287
• Copy Paste: [[2410.20287]] AI-Driven Cyber Threat Intelligence Automation(https://arxiv.org/abs/2410.20287)
• Keywords: security, large language model
• Abstract: This study introduces an innovative approach to automating Cyber Threat Intelligence (CTI) processes in industrial environments by leveraging Microsoft's AI-powered security technologies. Historically, CTI has heavily relied on manual methods for collecting, analyzing, and interpreting data from various sources such as threat feeds. This study introduces an innovative approach to automating CTI processes in industrial environments by leveraging Microsoft's AI-powered security technologies. Historically, CTI has heavily relied on manual methods for collecting, analyzing, and interpreting data from various sources such as threat feeds, security logs, and dark web forums -- a process prone to inefficiencies, especially when rapid information dissemination is critical. By employing the capabilities of GPT-4o and advanced one-shot fine-tuning techniques for large language models, our research delivers a novel CTI automation solution. The outcome of the proposed architecture is a reduction in manual effort while maintaining precision in generating final CTI reports. This research highlights the transformative potential of AI-driven technologies to enhance both the speed and accuracy of CTI and reduce expert demands, offering a vital advantage in today's dynamic threat landscape.

Title: Fast Best-of-N Decoding via Speculative Rejection

• Authors: Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, Andrea Zanette
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20290
• Pdf URL: https://arxiv.org/pdf/2410.20290
• Copy Paste: [[2410.20290]] Fast Best-of-N Decoding via Speculative Rejection(https://arxiv.org/abs/2410.20290)
• Keywords: large language model
• Abstract: The safe and effective deployment of Large Language Models (LLMs) involves a critical step called alignment, which ensures that the model's responses are in accordance with human preferences. Prevalent alignment techniques, such as DPO, PPO and their variants, align LLMs by changing the pre-trained model weights during a phase called post-training. While predominant, these post-training methods add substantial complexity before LLMs can be deployed. Inference-time alignment methods avoid the complex post-training step and instead bias the generation towards responses that are aligned with human preferences. The best-known inference-time alignment method, called Best-of-N, is as effective as the state-of-the-art post-training procedures. Unfortunately, Best-of-N requires vastly more resources at inference time than standard decoding strategies, which makes it computationally not viable. In this work, we introduce Speculative Rejection, a computationally-viable inference-time alignment algorithm. It generates high-scoring responses according to a given reward model, like Best-of-N does, while being between 16 to 32 times more computationally efficient.

Title: DeCaf: A Causal Decoupling Framework for OOD Generalization on Node Classification

• Authors: Xiaoxue Han, Huzefa Rangwala, Yue Ning
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20295
• Pdf URL: https://arxiv.org/pdf/2410.20295
• Copy Paste: [[2410.20295]] DeCaf: A Causal Decoupling Framework for OOD Generalization on Node Classification(https://arxiv.org/abs/2410.20295)
• Keywords: security
• Abstract: Graph Neural Networks (GNNs) are susceptible to distribution shifts, creating vulnerability and security issues in critical domains. There is a pressing need to enhance the generalizability of GNNs on out-of-distribution (OOD) test data. Existing methods that target learning an invariant (feature, structure)-label mapping often depend on oversimplified assumptions about the data generation process, which do not adequately reflect the actual dynamics of distribution shifts in graphs. In this paper, we introduce a more realistic graph data generation model using Structural Causal Models (SCMs), allowing us to redefine distribution shifts by pinpointing their origins within the generation process. Building on this, we propose a casual decoupling framework, DeCaf, that independently learns unbiased feature-label and structure-label mappings. We provide a detailed theoretical framework that shows how our approach can effectively mitigate the impact of various distribution shifts. We evaluate DeCaf across both real-world and synthetic datasets that demonstrate different patterns of shifts, confirming its efficacy in enhancing the generalizability of GNNs.

Title: Fine-Tuning and Evaluating Open-Source Large Language Models for the Army Domain

• Authors: Daniel C. Ruiz, John Sell
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2410.20297
• Pdf URL: https://arxiv.org/pdf/2410.20297
• Copy Paste: [[2410.20297]] Fine-Tuning and Evaluating Open-Source Large Language Models for the Army Domain(https://arxiv.org/abs/2410.20297)
• Keywords: large language model
• Abstract: In recent years, the widespread adoption of Large Language Models (LLMs) has sparked interest in their potential for application within the military domain. However, the current generation of LLMs demonstrate sub-optimal performance on Army use cases, due to the prevalence of domain-specific vocabulary and jargon. In order to fully leverage LLMs in-domain, many organizations have turned to fine-tuning to circumvent the prohibitive costs involved in training new LLMs from scratch. In light of this trend, we explore the viability of adapting open-source LLMs for usage in the Army domain in order to address their existing lack of domain-specificity. Our investigations have resulted in the creation of three distinct generations of TRACLM, a family of LLMs fine-tuned by The Research and Analysis Center (TRAC), Army Futures Command (AFC). Through continuous refinement of our training pipeline, each successive iteration of TRACLM displayed improved capabilities when applied to Army tasks and use cases. Furthermore, throughout our fine-tuning experiments, we recognized the need for an evaluation framework that objectively quantifies the Army domain-specific knowledge of LLMs. To address this, we developed MilBench, an extensible software framework that efficiently evaluates the Army knowledge of a given LLM using tasks derived from doctrine and assessments. We share preliminary results, models, methods, and recommendations on the creation of TRACLM and MilBench. Our work significantly informs the development of LLM technology across the DoD and augments senior leader decisions with respect to artificial intelligence integration.

Title: Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data

• Authors: Xinhong Xie, Tao Li, Quanyan Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20298
• Pdf URL: https://arxiv.org/pdf/2410.20298
• Copy Paste: [[2410.20298]] Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data(https://arxiv.org/abs/2410.20298)
• Keywords: large language model
• Abstract: Text detoxification, a variant of style transfer tasks, finds useful applications in online social media. This work presents a fine-tuning method that only uses non-parallel data to turn large language models (LLM) into a detoxification rewritter. We model the fine-tuning process as a Stackelberg game between an LLM (leader) and a toxicity screener (follower), which is a binary style classifier (toxic or non-toxic). The LLM aims to align its preference according to the screener and generate paraphases passing the screening. The primary challenge of non-parallel data fine-tuning is incomplete preference. In the case of unsuccessful paraphrases, the classifier cannot establish a preference between the input and paraphrase, as they belong to the same toxic style. Hence, preference-alignment fine-tuning methods, such as direct preference optimization (DPO), no longer apply. To address the challenge of incomplete preference, we propose Stackelberg response optimization (SRO), adapted from DPO, to enable the LLM to learn from the follower's response. The gist is that SRO decreases the likelihood of generating the paraphrase if it fails the follower's screening while performing DPO on the pair of the toxic input and its paraphrase when the latter passes the screening. Experiments indicate that the SRO-fine-tunned LLM achieves satisfying performance comparable to state-of-the-art models regarding style accuracy, content similarity, and fluency. The overall detoxification performance surpasses other computing methods and matches the human reference. Additional empirical evidence suggests that SRO is sensitive to the screener's feedback, and a slight perturbation leads to a significant performance drop. We release the code and LLM models at \url{this https URL}.

Title: Predicting Mortality and Functional Status Scores of Traumatic Brain Injury Patients using Supervised Machine Learning

• Authors: Lucas Steinmetz, Shivam Maheshwari, Garik Kazanjian, Abigail Loyson, Tyler Alexander, Venkat Margapuri, C. Nataraj
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20300
• Pdf URL: https://arxiv.org/pdf/2410.20300
• Copy Paste: [[2410.20300]] Predicting Mortality and Functional Status Scores of Traumatic Brain Injury Patients using Supervised Machine Learning(https://arxiv.org/abs/2410.20300)
• Keywords: interpretability
• Abstract: Traumatic brain injury (TBI) presents a significant public health challenge, often resulting in mortality or lasting disability. Predicting outcomes such as mortality and Functional Status Scale (FSS) scores can enhance treatment strategies and inform clinical decision-making. This study applies supervised machine learning (ML) methods to predict mortality and FSS scores using a real-world dataset of 300 pediatric TBI patients from the University of Colorado School of Medicine. The dataset captures clinical features, including demographics, injury mechanisms, and hospitalization outcomes. Eighteen ML models were evaluated for mortality prediction, and thirteen models were assessed for FSS score prediction. Performance was measured using accuracy, ROC AUC, F1-score, and mean squared error. Logistic regression and Extra Trees models achieved high precision in mortality prediction, while linear regression demonstrated the best FSS score prediction. Feature selection reduced 103 clinical variables to the most relevant, enhancing model efficiency and interpretability. This research highlights the role of ML models in identifying high-risk patients and supporting personalized interventions, demonstrating the potential of data-driven analytics to improve TBI care and integrate into clinical workflows.

Title: Sequential Large Language Model-Based Hyper-Parameter Optimization

• Authors: Kanan Mahammadli
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20302
• Pdf URL: https://arxiv.org/pdf/2410.20302
• Copy Paste: [[2410.20302]] Sequential Large Language Model-Based Hyper-Parameter Optimization(https://arxiv.org/abs/2410.20302)
• Keywords: robust, large language model, segmentation
• Abstract: This study introduces SLLMBO, an innovative framework that leverages Large Language Models (LLMs) for hyperparameter optimization (HPO), incorporating dynamic search space adaptability, enhanced parameter landscape exploitation, and a hybrid, novel LLM-Tree-structured Parzen Estimator (LLM-TPE) sampler. By addressing limitations in recent fully LLM-based methods and traditional Bayesian Optimization (BO), SLLMBO achieves more robust optimization. This comprehensive benchmarking evaluates multiple LLMs, including GPT-3.5-turbo, GPT-4o, Claude-Sonnet-3.5, and Gemini-1.5-flash, extending prior work beyond GPT-3.5 and GPT-4 and establishing SLLMBO as the first framework to benchmark a diverse set of LLMs for HPO. By integrating LLMs' established strengths in parameter initialization with the exploitation abilities demonstrated in this study, alongside TPE's exploration capabilities, the LLM-TPE sampler achieves a balanced exploration-exploitation trade-off, reduces API costs, and mitigates premature early stoppings for more effective parameter searches. Across 14 tabular tasks in classification and regression, the LLM-TPE sampler outperformed fully LLM-based methods and achieved superior results over BO methods in 9 tasks. Testing early stopping in budget-constrained scenarios further demonstrated competitive performance, indicating that LLM-based methods generally benefit from extended iterations for optimal results. This work lays the foundation for future research exploring open-source LLMs, reproducibility of LLM results in HPO, and benchmarking SLLMBO on complex datasets, such as image classification, segmentation, and machine translation.

Title: Deep Learning, Machine Learning -- Digital Signal and Image Processing: From Theory to Application

• Authors: Weiche Hsieh, Ziqian Bi, Junyu Liu, Benji Peng, Sen Zhang, Xuanhe Pan, Jiawei Xu, Jinlang Wang, Keyu Chen, Caitlyn Heqi Yin, Pohsun Feng, Yizhu Wen, Tianyang Wang, Ming Li, Jintao Ren, Qian Niu, Silin Chen, Ming Liu
• Subjects: cs.CV, cs.GR, eess.IV, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.20304
• Pdf URL: https://arxiv.org/pdf/2410.20304
• Copy Paste: [[2410.20304]] Deep Learning, Machine Learning -- Digital Signal and Image Processing: From Theory to Application(https://arxiv.org/abs/2410.20304)
• Keywords: robust, extraction
• Abstract: Digital Signal Processing (DSP) and Digital Image Processing (DIP) with Machine Learning (ML) and Deep Learning (DL) are popular research areas in Computer Vision and related fields. We highlight transformative applications in image enhancement, filtering techniques, and pattern recognition. By integrating frameworks like the Discrete Fourier Transform (DFT), Z-Transform, and Fourier Transform methods, we enable robust data manipulation and feature extraction essential for AI-driven tasks. Using Python, we implement algorithms that optimize real-time data processing, forming a foundation for scalable, high-performance solutions in computer vision. This work illustrates the potential of ML and DL to advance DSP and DIP methodologies, contributing to artificial intelligence, automated feature extraction, and applications across diverse domains.

Title: Wavelet-based Mamba with Fourier Adjustment for Low-light Image Enhancement

• Authors: Junhao Tan, Songwen Pei, Wei Qin, Bo Fu, Ximing Li, Libo Huang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.20314
• Pdf URL: https://arxiv.org/pdf/2410.20314
• Copy Paste: [[2410.20314]] Wavelet-based Mamba with Fourier Adjustment for Low-light Image Enhancement(https://arxiv.org/abs/2410.20314)
• Keywords: diffusion, transformer
• Abstract: Frequency information (e.g., Discrete Wavelet Transform and Fast Fourier Transform) has been widely applied to solve the issue of Low-Light Image Enhancement (LLIE). However, existing frequency-based models primarily operate in the simple wavelet or Fourier space of images, which lacks utilization of valid global and local information in each space. We found that wavelet frequency information is more sensitive to global brightness due to its low-frequency component while Fourier frequency information is more sensitive to local details due to its phase component. In order to achieve superior preliminary brightness enhancement by optimally integrating spatial channel information with low-frequency components in the wavelet transform, we introduce channel-wise Mamba, which compensates for the long-range dependencies of CNNs and has lower complexity compared to Diffusion and Transformer models. So in this work, we propose a novel Wavelet-based Mamba with Fourier Adjustment model called WalMaFa, consisting of a Wavelet-based Mamba Block (WMB) and a Fast Fourier Adjustment Block (FFAB). We employ an Encoder-Latent-Decoder structure to accomplish the end-to-end transformation. Specifically, WMB is adopted in the Encoder and Decoder to enhance global brightness while FFAB is adopted in the Latent to fine-tune local texture details and alleviate ambiguity. Extensive experiments demonstrate that our proposed WalMaFa achieves state-of-the-art performance with fewer computational resources and faster speed. Code is now available at: this https URL.

Title: Deep Learning Based Dense Retrieval: A Comparative Study

• Authors: Ming Zhong, Zhizhi Wu, Nanako Honda
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20315
• Pdf URL: https://arxiv.org/pdf/2410.20315
• Copy Paste: [[2410.20315]] Deep Learning Based Dense Retrieval: A Comparative Study(https://arxiv.org/abs/2410.20315)
• Keywords: defense, robust
• Abstract: Dense retrievers have achieved state-of-the-art performance in various information retrieval tasks, but their robustness against tokenizer poisoning remains underexplored. In this work, we assess the vulnerability of dense retrieval systems to poisoned tokenizers by evaluating models such as BERT, Dense Passage Retrieval (DPR), Contriever, SimCSE, and ANCE. We find that supervised models like BERT and DPR experience significant performance degradation when tokenizers are compromised, while unsupervised models like ANCE show greater resilience. Our experiments reveal that even small perturbations can severely impact retrieval accuracy, highlighting the need for robust defenses in critical applications.

Title: ProtSCAPE: Mapping the landscape of protein conformations in molecular dynamics

• Authors: Siddharth Viswanath, Dhananjay Bhaskar, David R. Johnson, Joao Felipe Rocha, Egbert Castro, Jackson D. Grady, Alex T. Grigas, Michael A. Perlmutter, Corey S. O'Hern, Smita Krishnaswamy
• Subjects: cs.LG, physics.chem-ph, q-bio.BM, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.20317
• Pdf URL: https://arxiv.org/pdf/2410.20317
• Copy Paste: [[2410.20317]] ProtSCAPE: Mapping the landscape of protein conformations in molecular dynamics(https://arxiv.org/abs/2410.20317)
• Keywords: transformer
• Abstract: Understanding the dynamic nature of protein structures is essential for comprehending their biological functions. While significant progress has been made in predicting static folded structures, modeling protein motions on microsecond to millisecond scales remains challenging. To address these challenges, we introduce a novel deep learning architecture, Protein Transformer with Scattering, Attention, and Positional Embedding (ProtSCAPE), which leverages the geometric scattering transform alongside transformer-based attention mechanisms to capture protein dynamics from molecular dynamics (MD) simulations. ProtSCAPE utilizes the multi-scale nature of the geometric scattering transform to extract features from protein structures conceptualized as graphs and integrates these features with dual attention structures that focus on residues and amino acid signals, generating latent representations of protein trajectories. Furthermore, ProtSCAPE incorporates a regression head to enforce temporally coherent latent representations.

Title: Few-shot Open Relation Extraction with Gaussian Prototype and Adaptive Margin

• Authors: Tianlin Guo, Lingling Zhang, Jiaxin Wang, Yuokuo Lei, Yifei Li, Haofen Wang, Jun Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20320
• Pdf URL: https://arxiv.org/pdf/2410.20320
• Copy Paste: [[2410.20320]] Few-shot Open Relation Extraction with Gaussian Prototype and Adaptive Margin(https://arxiv.org/abs/2410.20320)
• Keywords: robust, extraction
• Abstract: Few-shot relation extraction with none-of-the-above (FsRE with NOTA) aims at predicting labels in few-shot scenarios with unknown classes. FsRE with NOTA is more challenging than the conventional few-shot relation extraction task, since the boundaries of unknown classes are complex and difficult to learn. Meta-learning based methods, especially prototype-based methods, are the mainstream solutions to this task. They obtain the classification boundary by learning the sample distribution of each class. However, their performance is limited because few-shot overfitting and NOTA boundary confusion lead to misclassification between known and unknown classes. To this end, we propose a novel framework based on Gaussian prototype and adaptive margin named GPAM for FsRE with NOTA, which includes three modules, semi-factual representation, GMM-prototype metric learning and decision boundary learning. The first two modules obtain better representations to solve the few-shot problem through debiased information enhancement and Gaussian space distance measurement. The third module learns more accurate classification boundaries and prototypes through adaptive margin and negative sampling. In the training procedure of GPAM, we use contrastive learning loss to comprehensively consider the effects of range and margin on the classification of known and unknown classes to ensure the model's stability and robustness. Sufficient experiments and ablations on the FewRel dataset show that GPAM surpasses previous prototype methods and achieves state-of-the-art performance.

Title: A New Non-Binary Response Generation Scheme from Physical Unclonable Functions

• Authors: Yonghong Bai, Zhiyuan Yan
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.20324
• Pdf URL: https://arxiv.org/pdf/2410.20324
• Copy Paste: [[2410.20324]] A New Non-Binary Response Generation Scheme from Physical Unclonable Functions(https://arxiv.org/abs/2410.20324)
• Keywords: extraction
• Abstract: Physical Unclonable Functions (PUFs) are widely used in key generation, with each PUF cell typically producing one bit of data. To enable the extraction of longer keys, a new non-binary response generation scheme based on the one-probability of PUF bits is proposed. Instead of using PUF bits directly as keys, non-binary responses are first derived by comparing the one-frequency of PUF bits with thresholds that evenly divide the area under the probability density function of the one-probability distribution and then converted to binary keys. To simplify the calculation of these thresholds, a re-scaling process is proposed and the beta distribution is used to model the one-probability distribution. Our FPGA implementation results demonstrate a significant increase in effective key length as opposed to previous works. Finally, we estimate the error rates and biases of the generated keys, and confirm the feasibility of the proposed key generation scheme.

Title: Embedded Nonlocal Operator Regression (ENOR): Quantifying model error in learning nonlocal operators

• Authors: Yiming Fan, Habib Najm, Yue Yu, Stewart Silling, Marta D'Elia
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2410.20331
• Pdf URL: https://arxiv.org/pdf/2410.20331
• Copy Paste: [[2410.20331]] Embedded Nonlocal Operator Regression (ENOR): Quantifying model error in learning nonlocal operators(https://arxiv.org/abs/2410.20331)
• Keywords: robust
• Abstract: Nonlocal, integral operators have become an efficient surrogate for bottom-up homogenization, due to their ability to represent long-range dependence and multiscale effects. However, the nonlocal homogenized model has unavoidable discrepancy from the microscale model. Such errors accumulate and propagate in long-term simulations, making the resultant prediction unreliable. To develop a robust and reliable bottom-up homogenization framework, we propose a new framework, which we coin Embedded Nonlocal Operator Regression (ENOR), to learn a nonlocal homogenized surrogate model and its structural model error. This framework provides discrepancy-adaptive uncertainty quantification for homogenized material response predictions in long-term simulations. The method is built on Nonlocal Operator Regression (NOR), an optimization-based nonlocal kernel learning approach, together with an embedded model error term in the trainable kernel. Then, Bayesian inference is employed to infer the model error term parameters together with the kernel parameters. To make the problem computationally feasible, we use a multilevel delayed acceptance Markov chain Monte Carlo (MLDA-MCMC) method, enabling efficient Bayesian model calibration and model error estimation. We apply this technique to predict long-term wave propagation in a heterogeneous one-dimensional bar, and compare its performance with additive noise models. Owing to its ability to capture model error, the learned ENOR achieves improved estimation of posterior predictive uncertainty.

Title: Improving Speech-based Emotion Recognition with Contextual Utterance Analysis and LLMs

• Authors: Enshi Zhang, Christian Poellabauer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20334
• Pdf URL: https://arxiv.org/pdf/2410.20334
• Copy Paste: [[2410.20334]] Improving Speech-based Emotion Recognition with Contextual Utterance Analysis and LLMs(https://arxiv.org/abs/2410.20334)
• Keywords: large language model
• Abstract: Speech Emotion Recognition (SER) focuses on identifying emotional states from spoken language. The 2024 IEEE SLT-GenSEC Challenge on Post Automatic Speech Recognition (ASR) Emotion Recognition tasks participants to explore the capabilities of large language models (LLMs) for emotion recognition using only text data. We propose a novel approach that first refines all available transcriptions to ensure data reliability. We then segment each complete conversation into smaller dialogues and use these dialogues as context to predict the emotion of the target utterance within the dialogue. Finally, we investigated different context lengths and prompting techniques to improve prediction accuracy. Our best submission exceeded the baseline by 20% in unweighted accuracy, achieving the best performance in the challenge. All our experiments' codes, prediction results, and log files are publicly available.

Title: Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation

• Authors: Maohao Shen, Shun Zhang, Jilong Wu, Zhiping Xiu, Ehab AlBadawy, Yiting Lu, Mike Seltzer, Qing He
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.20336
• Pdf URL: https://arxiv.org/pdf/2410.20336
• Copy Paste: [[2410.20336]] Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation(https://arxiv.org/abs/2410.20336)
• Keywords: large language model
• Abstract: Large language models (LLMs) have revolutionized natural language processing (NLP) with impressive performance across various text-based tasks. However, the extension of text-dominant LLMs to with speech generation tasks remains under-explored. In this work, we introduce a text-to-speech (TTS) system powered by a fine-tuned Llama model, named TTS-Llama, that achieves state-of-the-art speech synthesis performance. Building on TTS-Llama, we further propose MoLE-Llama, a text-and-speech multimodal LLM developed through purely late-fusion parameter-efficient fine-tuning (PEFT) and a mixture-of-expert architecture. Extensive empirical results demonstrate MoLE-Llama's competitive performance on both text-only question-answering (QA) and TTS tasks, mitigating catastrophic forgetting issue in either modality. Finally, we further explore MoLE-Llama in text-in-speech-out QA tasks, demonstrating its great potential as a multimodal dialog system capable of speech generation.

Title: Maintaining Informative Coherence: Migrating Hallucinations in Large Language Models via Absorbing Markov Chains

• Authors: Jiemin Wu, Songning Lai, Ruiqiang Xiao, Tianlang Xue, Jiayu Yang, Yutao Yue
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20340
• Pdf URL: https://arxiv.org/pdf/2410.20340
• Copy Paste: [[2410.20340]] Maintaining Informative Coherence: Migrating Hallucinations in Large Language Models via Absorbing Markov Chains(https://arxiv.org/abs/2410.20340)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are powerful tools for text generation, translation, and summarization, but they often suffer from hallucinations-instances where they fail to maintain the fidelity and coherence of contextual information during decoding, sometimes overlooking critical details due to their sampling strategies and inherent biases from training data and fine-tuning discrepancies. These hallucinations can propagate through the web, affecting the trustworthiness of information disseminated online. To address this issue, we propose a novel decoding strategy that leverages absorbing Markov chains to quantify the significance of contextual information and measure the extent of information loss during generation. By considering all possible paths from the first to the last token, our approach enhances the reliability of model outputs without requiring additional training or external data. Evaluations on datasets including TruthfulQA, FACTOR, and HaluEval highlight the superior performance of our method in mitigating hallucinations, underscoring the necessity of ensuring accurate information flow in web-based applications.

Title: Historical Test-time Prompt Tuning for Vision Foundation Models

• Authors: Jingyi Zhang, Jiaxing Huang, Xiaoqin Zhang, Ling Shao, Shijian Lu
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20346
• Pdf URL: https://arxiv.org/pdf/2410.20346
• Copy Paste: [[2410.20346]] Historical Test-time Prompt Tuning for Vision Foundation Models(https://arxiv.org/abs/2410.20346)
• Keywords: robust, segmentation
• Abstract: Test-time prompt tuning, which learns prompts online with unlabelled test samples during the inference stage, has demonstrated great potential by learning effective prompts on-the-fly without requiring any task-specific annotations. However, its performance often degrades clearly along the tuning process when the prompts are continuously updated with the test data flow, and the degradation becomes more severe when the domain of test samples changes continuously. We propose HisTPT, a Historical Test-time Prompt Tuning technique that memorizes the useful knowledge of the learnt test samples and enables robust test-time prompt tuning with the memorized knowledge. HisTPT introduces three types of knowledge banks, namely, local knowledge bank, hard-sample knowledge bank, and global knowledge bank, each of which works with different mechanisms for effective knowledge memorization and test-time prompt optimization. In addition, HisTPT features an adaptive knowledge retrieval mechanism that regularizes the prediction of each test sample by adaptively retrieving the memorized knowledge. Extensive experiments show that HisTPT achieves superior prompt tuning performance consistently while handling different visual recognition tasks (e.g., image classification, semantic segmentation, and object detection) and test samples from continuously changing domains.

Title: UTSRMorph: A Unified Transformer and Superresolution Network for Unsupervised Medical Image Registration

• Authors: Runshi Zhang, Hao Mo, Junchen Wang, Bimeng Jie, Yang He, Nenghao Jin, Liang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20348
• Pdf URL: https://arxiv.org/pdf/2410.20348
• Copy Paste: [[2410.20348]] UTSRMorph: A Unified Transformer and Superresolution Network for Unsupervised Medical Image Registration(https://arxiv.org/abs/2410.20348)
• Keywords: transformer
• Abstract: Complicated image registration is a key issue in medical image analysis, and deep learning-based methods have achieved better results than traditional methods. The methods include ConvNet-based and Transformer-based methods. Although ConvNets can effectively utilize local information to reduce redundancy via small neighborhood convolution, the limited receptive field results in the inability to capture global dependencies. Transformers can establish long-distance dependencies via a self-attention mechanism; however, the intense calculation of the relationships among all tokens leads to high redundancy. We propose a novel unsupervised image registration method named the unified Transformer and superresolution (UTSRMorph) network, which can enhance feature representation learning in the encoder and generate detailed displacement fields in the decoder to overcome these problems. We first propose a fusion attention block to integrate the advantages of ConvNets and Transformers, which inserts a ConvNet-based channel attention module into a multihead self-attention module. The overlapping attention block, a novel cross-attention method, uses overlapping windows to obtain abundant correlations with match information of a pair of images. Then, the blocks are flexibly stacked into a new powerful encoder. The decoder generation process of a high-resolution deformation displacement field from low-resolution features is considered as a superresolution process. Specifically, the superresolution module was employed to replace interpolation upsampling, which can overcome feature degradation. UTSRMorph was compared to state-of-the-art registration methods in the 3D brain MR (OASIS, IXI) and MR-CT datasets. The qualitative and quantitative results indicate that UTSRMorph achieves relatively better performance. The code and datasets are publicly available at this https URL.

Title: Idempotent Unsupervised Representation Learning for Skeleton-Based Action Recognition

• Authors: Lilang Lin, Lehong Wu, Jiahang Zhang, Jiaying Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20349
• Pdf URL: https://arxiv.org/pdf/2410.20349
• Copy Paste: [[2410.20349]] Idempotent Unsupervised Representation Learning for Skeleton-Based Action Recognition(https://arxiv.org/abs/2410.20349)
• Keywords: generative
• Abstract: Generative models, as a powerful technique for generation, also gradually become a critical tool for recognition tasks. However, in skeleton-based action recognition, the features obtained from existing pre-trained generative methods contain redundant information unrelated to recognition, which contradicts the nature of the skeleton's spatially sparse and temporally consistent properties, leading to undesirable performance. To address this challenge, we make efforts to bridge the gap in theory and methodology and propose a novel skeleton-based idempotent generative model (IGM) for unsupervised representation learning. More specifically, we first theoretically demonstrate the equivalence between generative models and maximum entropy coding, which demonstrates a potential route that makes the features of generative models more compact by introducing contrastive learning. To this end, we introduce the idempotency constraint to form a stronger consistency regularization in the feature space, to push the features only to maintain the critical information of motion semantics for the recognition task. Our extensive experiments on benchmark datasets, NTU RGB+D and PKUMMD, demonstrate the effectiveness of our proposed method. On the NTU 60 xsub dataset, we observe a performance improvement from 84.6$\%$ to 86.2$\%$. Furthermore, in zero-shot adaptation scenarios, our model demonstrates significant efficacy by achieving promising results in cases that were previously unrecognizable. Our project is available at \url{this https URL}.

Title: Leveraging Auxiliary Task Relevance for Enhanced Industrial Fault Diagnosis through Curriculum Meta-learning

• Authors: Jinze Wang, Tiehua Zhang, Boon Xian Chai, Adriano Di Pietro, Dimitrios Georgakopoulos, Jiong Jin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20351
• Pdf URL: https://arxiv.org/pdf/2410.20351
• Copy Paste: [[2410.20351]] Leveraging Auxiliary Task Relevance for Enhanced Industrial Fault Diagnosis through Curriculum Meta-learning(https://arxiv.org/abs/2410.20351)
• Keywords: robust
• Abstract: The accurate diagnosis of machine breakdowns is crucial for maintaining operational safety in smart manufacturing. Despite the promise shown by deep learning in automating fault identification, the scarcity of labeled training data, particularly for equipment failure instances, poses a significant challenge. This limitation hampers the development of robust classification models. Existing methods like model-agnostic meta-learning (MAML) do not adequately address variable working conditions, affecting knowledge transfer. To address these challenges, a Related Task Aware Curriculum Meta-learning (RT-ACM) enhanced fault diagnosis framework is proposed in this paper, inspired by human cognitive learning processes. RT-ACM improves training by considering the relevance of auxiliary working conditions, adhering to the principle of ``paying more attention to more relevant knowledge", and focusing on ``easier first, harder later" curriculum sampling. This approach aids the meta-learner in achieving a superior convergence state. Extensive experiments on two real-world datasets demonstrate the superiority of RT-ACM framework.

Title: FoldMark: Protecting Protein Generative Models with Watermarking

• Authors: Zaixi Zhang, Ruofan Jin, Kaidi Fu, Le Cong, Marinka Zitnik, Mengdi Wang
• Subjects: cs.CR, cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.20354
• Pdf URL: https://arxiv.org/pdf/2410.20354
• Copy Paste: [[2410.20354]] FoldMark: Protecting Protein Generative Models with Watermarking(https://arxiv.org/abs/2410.20354)
• Keywords: security, protect, attack, robust, watermark, generative
• Abstract: Protein structure is key to understanding protein function and is essential for progress in bioengineering, drug discovery, and molecular biology. Recently, with the incorporation of generative AI, the power and accuracy of computational protein structure prediction/design have been improved significantly. However, ethical concerns such as copyright protection and harmful content generation (biosecurity) pose challenges to the wide implementation of protein generative models. Here, we investigate whether it is possible to embed watermarks into protein generative models and their outputs for copyright authentication and the tracking of generated structures. As a proof of concept, we propose a two-stage method FoldMark as a generalized watermarking strategy for protein generative models. FoldMark first pretrain watermark encoder and decoder, which can minorly adjust protein structures to embed user-specific information and faithfully recover the information from the encoded structure. In the second step, protein generative models are fine-tuned with watermark Low-Rank Adaptation (LoRA) modules to preserve generation quality while learning to generate watermarked structures with high recovery rates. Extensive experiments are conducted on open-source protein structure prediction models (e.g., ESMFold and MultiFlow) and de novo structure design models (e.g., FrameDiff and FoldFlow) and we demonstrate that our method is effective across all these generative models. Meanwhile, our watermarking framework only exerts a negligible impact on the original protein structure quality and is robust under potential post-processing and adaptive attacks.

Title: RopeTP: Global Human Motion Recovery via Integrating Robust Pose Estimation with Diffusion Trajectory Prior

• Authors: Mingjiang Liang, Yongkang Cheng, Hualin Liang, Shaoli Huang, Wei Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20358
• Pdf URL: https://arxiv.org/pdf/2410.20358
• Copy Paste: [[2410.20358]] RopeTP: Global Human Motion Recovery via Integrating Robust Pose Estimation with Diffusion Trajectory Prior(https://arxiv.org/abs/2410.20358)
• Keywords: robust, diffusion
• Abstract: We present RopeTP, a novel framework that combines Robust pose estimation with a diffusion Trajectory Prior to reconstruct global human motion from videos. At the heart of RopeTP is a hierarchical attention mechanism that significantly improves context awareness, which is essential for accurately inferring the posture of occluded body parts. This is achieved by exploiting the relationships with visible anatomical structures, enhancing the accuracy of local pose estimations. The improved robustness of these local estimations allows for the reconstruction of precise and stable global trajectories. Additionally, RopeTP incorporates a diffusion trajectory model that predicts realistic human motion from local pose sequences. This model ensures that the generated trajectories are not only consistent with observed local actions but also unfold naturally over time, thereby improving the realism and stability of 3D human motion reconstruction. Extensive experimental validation shows that RopeTP surpasses current methods on two benchmark datasets, particularly excelling in scenarios with occlusions. It also outperforms methods that rely on SLAM for initial camera estimates and extensive optimization, delivering more accurate and realistic trajectories.

Title: Rethinking Data Synthesis: A Teacher Model Training Recipe with Interpretation

• Authors: Yifang Chen, David Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20362
• Pdf URL: https://arxiv.org/pdf/2410.20362
• Copy Paste: [[2410.20362]] Rethinking Data Synthesis: A Teacher Model Training Recipe with Interpretation(https://arxiv.org/abs/2410.20362)
• Keywords: large language model
• Abstract: Recent advances in large language model (LLM) training have highlighted the need for diverse, high-quality instruction data. Recently, many works are exploring synthetic data generation using LLMs. However, they primarily focus on prompt engineering with standard supervised instruction-finetuned models, which contains a fundamental limitation: these models are optimized for general question-answering/problem-solving rather than data generation. We propose a paradigm shift named \textbf{NOMAD} by investigating how to specifically train models for data generation, demonstrating that this task differs significantly from training a classical LM. We identify two key factors: no-prompt-masked training and proper training set size selection. Our method, NOMAD, shows substantial improvements over baselines, achieving >4\% gains in TriviaQA and >2\% in GSM8K with limited training data. Finally, we offer new insights by interpreting synthetic data through the lenses of "relevance" and "novelty".

Title: FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion

• Authors: Zhenheng Tang, Yonggang Zhang, Peijie Dong, Yiu-ming Cheung, Amelie Chi Zhou, Bo Han, Xiaowen Chu
• Subjects: cs.LG, cs.AI, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2410.20380
• Pdf URL: https://arxiv.org/pdf/2410.20380
• Copy Paste: [[2410.20380]] FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion(https://arxiv.org/abs/2410.20380)
• Keywords: federate
• Abstract: One-shot Federated Learning (OFL) significantly reduces communication costs in FL by aggregating trained models only once. However, the performance of advanced OFL methods is far behind the normal FL. In this work, we provide a causal view to find that this performance drop of OFL methods comes from the isolation problem, which means that local isolatedly trained models in OFL may easily fit to spurious correlations due to the data heterogeneity. From the causal perspective, we observe that the spurious fitting can be alleviated by augmenting intermediate features from other clients. Built upon our observation, we propose a novel learning approach to endow OFL with superb performance and low communication and storage costs, termed as FuseFL. Specifically, FuseFL decomposes neural networks into several blocks, and progressively trains and fuses each block following a bottom-up manner for feature augmentation, introducing no additional communication costs. Comprehensive experiments demonstrate that FuseFL outperforms existing OFL and ensemble FL by a significant margin. We conduct comprehensive experiments to show that FuseFL supports high scalability of clients, heterogeneous model training, and low memory costs. Our work is the first attempt using causality to analyze and alleviate data heterogeneity of OFL.

Title: Multiple kernel concept factorization algorithm based on global fusion

• Authors: Fei Li, Liang Du, Chaohong Ren
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20383
• Pdf URL: https://arxiv.org/pdf/2410.20383
• Copy Paste: [[2410.20383]] Multiple kernel concept factorization algorithm based on global fusion(https://arxiv.org/abs/2410.20383)
• Keywords: robust
• Abstract: Non-negative Matrix Factorization(NMF) algorithm can only be used to find low rank approximation of original non-negative data while Concept Factorization(CF) algorithm extends matrix factorization to single non-linear kernel space, improving learning ability and adaptability of matrix factorization. In unsupervised environment, to design or select proper kernel function for specific dataset, a new algorithm called Globalized Multiple Kernel CF(GMKCF)was proposed. Multiple candidate kernel functions were input in the same time and learned in the CF framework based on global linear fusion, obtaining a clustering result with high quality and stability and solving the problem of kernel function selection that the CF faced. The convergence of the proposed algorithm was verified by solving the model with alternate iteration. The experimental results on several real databases show that the proposed algorithm outperforms comparison algorithms in data clustering, such as Kernel K-Means(KKM), Spectral Clustering(SC), Kernel CF(KCF), Co-regularized multi-view spectral clustering(Coreg), and Robust Multiple KKM(RMKKM).

Title: Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patterns

• Authors: Ronghui Li, Hongwen Zhang, Yachao Zhang, Yuxiang Zhang, Youliang Zhang, Jie Guo, Yan Zhang, Xiu Li, Yebin Liu
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.20389
• Pdf URL: https://arxiv.org/pdf/2410.20389
• Copy Paste: [[2410.20389]] Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patterns(https://arxiv.org/abs/2410.20389)
• Keywords: diffusion
• Abstract: We propose Lodge++, a choreography framework to generate high-quality, ultra-long, and vivid dances given the music and desired genre. To handle the challenges in computational efficiency, the learning of complex and vivid global choreography patterns, and the physical quality of local dance movements, Lodge++ adopts a two-stage strategy to produce dances from coarse to fine. In the first stage, a global choreography network is designed to generate coarse-grained dance primitives that capture complex global choreography patterns. In the second stage, guided by these dance primitives, a primitive-based dance diffusion model is proposed to further generate high-quality, long-sequence dances in parallel, faithfully adhering to the complex choreography patterns. Additionally, to improve the physical plausibility, Lodge++ employs a penetration guidance module to resolve character self-penetration, a foot refinement module to optimize foot-ground contact, and a multi-genre discriminator to maintain genre consistency throughout the dance. Lodge++ is validated by extensive experiments, which show that our method can rapidly generate ultra-long dances suitable for various dance genres, ensuring well-organized global choreography patterns and high-quality local motion.

Title: Depth Attention for Robust RGB Tracking

• Authors: Yu Liu, Arif Mahmood, Muhammad Haris Khan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.20395
• Pdf URL: https://arxiv.org/pdf/2410.20395
• Copy Paste: [[2410.20395]] Depth Attention for Robust RGB Tracking(https://arxiv.org/abs/2410.20395)
• Keywords: robust
• Abstract: RGB video object tracking is a fundamental task in computer vision. Its effectiveness can be improved using depth information, particularly for handling motion-blurred target. However, depth information is often missing in commonly used tracking benchmarks. In this work, we propose a new framework that leverages monocular depth estimation to counter the challenges of tracking targets that are out of view or affected by motion blur in RGB video sequences. Specifically, our work introduces following contributions. To the best of our knowledge, we are the first to propose a depth attention mechanism and to formulate a simple framework that allows seamlessly integration of depth information with state of the art tracking algorithms, without RGB-D cameras, elevating accuracy and robustness. We provide extensive experiments on six challenging tracking benchmarks. Our results demonstrate that our approach provides consistent gains over several strong baselines and achieves new SOTA performance. We believe that our method will open up new possibilities for more sophisticated VOT solutions in real-world scenarios. Our code and models are publicly released: this https URL.

Title: Prototypical Extreme Multi-label Classification with a Dynamic Margin Loss

• Authors: Kunal Dahiya, Diego Ortego, David Jiménez
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.20401
• Pdf URL: https://arxiv.org/pdf/2410.20401
• Copy Paste: [[2410.20401]] Prototypical Extreme Multi-label Classification with a Dynamic Margin Loss(https://arxiv.org/abs/2410.20401)
• Keywords: transformer
• Abstract: Extreme Multi-label Classification (XMC) methods predict relevant labels for a given query in an extremely large label space. Recent works in XMC address this problem using deep encoders that project text descriptions to an embedding space suitable for recovering the closest labels. However, learning deep models can be computationally expensive in large output spaces, resulting in a trade-off between high performing brute-force approaches and efficient solutions. In this paper, we propose PRIME, a XMC method that employs a novel prototypical contrastive learning technique to reconcile efficiency and performance surpassing brute-force approaches. We frame XMC as a data-to-prototype prediction task where label prototypes aggregate information from related queries. More precisely, we use a shallow transformer encoder that we coin as Label Prototype Network, which enriches label representations by aggregating text-based embeddings, label centroids and learnable free vectors. We jointly train a deep encoder and the Label Prototype Network using an adaptive triplet loss objective that better adapts to the high granularity and ambiguity of extreme label spaces. PRIME achieves state-of-the-art results in several public benchmarks of different sizes and domains, while keeping the model efficient.

Title: Deep Learning-Driven Microstructure Characterization and Vickers Hardness Prediction of Mg-Gd Alloys

• Authors: Lu Wang, Hongchan Chen, Bing Wang, Qian Li, Qun Luo, Yuexing Han
• Subjects: cs.LG, cond-mat.mtrl-sci, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20402
• Pdf URL: https://arxiv.org/pdf/2410.20402
• Copy Paste: [[2410.20402]] Deep Learning-Driven Microstructure Characterization and Vickers Hardness Prediction of Mg-Gd Alloys(https://arxiv.org/abs/2410.20402)
• Keywords: transformer
• Abstract: In the field of materials science, exploring the relationship between composition, microstructure, and properties has long been a critical research focus. The mechanical performance of solid-solution Mg-Gd alloys is significantly influenced by Gd content, dendritic structures, and the presence of secondary phases. To better analyze and predict the impact of these factors, this study proposes a multimodal fusion learning framework based on image processing and deep learning techniques. This framework integrates both elemental composition and microstructural features to accurately predict the Vickers hardness of solid-solution Mg-Gd alloys. Initially, deep learning methods were employed to extract microstructural information from a variety of solid-solution Mg-Gd alloy images obtained from literature and experiments. This provided precise grain size and secondary phase microstructural features for performance prediction tasks. Subsequently, these quantitative analysis results were combined with Gd content information to construct a performance prediction dataset. Finally, a regression model based on the Transformer architecture was used to predict the Vickers hardness of Mg-Gd alloys. The experimental results indicate that the Transformer model performs best in terms of prediction accuracy, achieving an R^2 value of 0.9. Additionally, SHAP analysis identified critical values for four key features affecting the Vickers hardness of Mg-Gd alloys, providing valuable guidance for alloy design. These findings not only enhance the understanding of alloy performance but also offer theoretical support for future material design and optimization.

Title: Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Models

• Authors: Zhengmian Hu, Heng Huang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20418
• Pdf URL: https://arxiv.org/pdf/2410.20418
• Copy Paste: [[2410.20418]] Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Models(https://arxiv.org/abs/2410.20418)
• Keywords: watermark, large language model
• Abstract: Large language models are probabilistic models, and the process of generating content is essentially sampling from the output distribution of the language model. Existing watermarking techniques inject watermarks into the generated content without altering the output quality. On the other hand, existing acceleration techniques, specifically speculative sampling, leverage a draft model to speed up the sampling process while preserving the output distribution. However, there is no known method to simultaneously accelerate the sampling process and inject watermarks into the generated content. In this paper, we investigate this direction and find that the integration of watermarking and acceleration is non-trivial. We prove a no-go theorem, which states that it is impossible to simultaneously maintain the highest watermark strength and the highest sampling efficiency. Furthermore, we propose two methods that maintain either the sampling efficiency or the watermark strength, but not both. Our work provides a rigorous theoretical foundation for understanding the inherent trade-off between watermark strength and sampling efficiency in accelerating the generation of watermarked tokens for large language models. We also conduct numerical experiments to validate our theoretical findings and demonstrate the effectiveness of the proposed methods.

Title: YourSkatingCoach: A Figure Skating Video Benchmark for Fine-Grained Element Analysis

• Authors: Wei-Yi Chen, Yu-An Su, Wei-Hsin Yeh, Lun-Wei Ku
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20427
• Pdf URL: https://arxiv.org/pdf/2410.20427
• Copy Paste: [[2410.20427]] YourSkatingCoach: A Figure Skating Video Benchmark for Fine-Grained Element Analysis(https://arxiv.org/abs/2410.20427)
• Keywords: transformer
• Abstract: Combining sports and machine learning involves leveraging ML algorithms and techniques to extract insight from sports-related data such as player statistics, game footage, and other relevant information. However, datasets related to figure skating in the literature focus primarily on element classification and are currently unavailable or exhibit only limited access, which greatly raise the entry barrier to developing visual sports technology for it. Moreover, when using such data to help athletes improve their skills, we find they are very coarse-grained: they work for learning what an element is, but they are poorly suited to learning whether the element is good or bad. Here we propose air time detection, a novel motion analysis task, the goal of which is to accurately detect the duration of the air time of a jump. We present YourSkatingCoach, a large, novel figure skating dataset which contains 454 videos of jump elements, the detected skater skeletons in each video, along with the gold labels of the start and ending frames of each jump, together as a video benchmark for figure skating. In addition, although this type of task is often viewed as classification, we cast it as a sequential labeling problem and propose a Transformer-based model to calculate the duration. Experimental results show that the proposed model yields a favorable results for a strong baseline. To further verify the generalizability of the fine-grained labels, we apply the same process to other sports as cross-sports tasks but for coarse-grained task action classification. Here we fine-tune the classification to demonstrate that figure skating, as it contains the essential body movements, constitutes a strong foundation for adaptation to other sports.

Title: MedGo: A Chinese Medical Large Language Model

• Authors: Haitao Zhang, Bo An
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20428
• Pdf URL: https://arxiv.org/pdf/2410.20428
• Copy Paste: [[2410.20428]] MedGo: A Chinese Medical Large Language Model(https://arxiv.org/abs/2410.20428)
• Keywords: generative, large language model
• Abstract: Large models are a hot research topic in the field of artificial intelligence. Leveraging their generative capabilities has the potential to enhance the level and quality of medical services. In response to the limitations of current large language models, which often struggle with accuracy and have narrow capabilities in medical applications, this paper presents a Chinese medical large language model, MedGo. MedGo was trained using a combination of high quality unsupervised medical data, supervised data, and preference alignment data, aimed at enhancing both its versatility and precision in medical tasks. The model was evaluated through the public CBLUE benchmark and a manually constructed dataset ClinicalQA. The results demonstrate that MedGo achieved promising performance across various Chinese medical information processing tasks, achieved the first place in the CBLUE evaluation. Additionally, on our constructed dataset ClinicalQA, MedGo outperformed its base model Qwen2, highlighting its potential to improve both automated medical question answering and clinical decision support. These experimental results demonstrate that MedGo possesses strong information processing capabilities in the medical field. At present, we have successfully deployed MedGo at Shanghai East Hospital.

Title: Integrating uncertainty quantification into randomized smoothing based robustness guarantees

• Authors: Sina Däubener, Kira Maag, David Krueger, Asja Fischer
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.20432
• Pdf URL: https://arxiv.org/pdf/2410.20432
• Copy Paste: [[2410.20432]] Integrating uncertainty quantification into randomized smoothing based robustness guarantees(https://arxiv.org/abs/2410.20432)
• Keywords: attack, robust
• Abstract: Deep neural networks have proven to be extremely powerful, however, they are also vulnerable to adversarial attacks which can cause hazardous incorrect predictions in safety-critical applications. Certified robustness via randomized smoothing gives a probabilistic guarantee that the smoothed classifier's predictions will not change within an $\ell_2$-ball around a given input. On the other hand (uncertainty) score-based rejection is a technique often applied in practice to defend models against adversarial attacks. In this work, we fuse these two approaches by integrating a classifier that abstains from predicting when uncertainty is high into the certified robustness framework. This allows us to derive two novel robustness guarantees for uncertainty aware classifiers, namely (i) the radius of an $\ell_2$-ball around the input in which the same label is predicted and uncertainty remains low and (ii) the $\ell_2$-radius of a ball in which the predictions will either not change or be uncertain. While the former provides robustness guarantees with respect to attacks aiming at increased uncertainty, the latter informs about the amount of input perturbation necessary to lead the uncertainty aware model into a wrong prediction. Notably, this is on CIFAR10 up to 20.93% larger than for models not allowing for uncertainty based rejection. We demonstrate, that the novel framework allows for a systematic robustness evaluation of different network architectures and uncertainty measures and to identify desired properties of uncertainty quantification techniques. Moreover, we show that leveraging uncertainty in a smoothed classifier helps out-of-distribution detection.

Title: CoralSCOP-LAT: Labeling and Analyzing Tool for Coral Reef Images with Dense Mask

• Authors: Yuk-Kwan Wong, Ziqiang Zheng, Mingzhe Zhang, David Suggett, Sai-Kit Yeung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20436
• Pdf URL: https://arxiv.org/pdf/2410.20436
• Copy Paste: [[2410.20436]] CoralSCOP-LAT: Labeling and Analyzing Tool for Coral Reef Images with Dense Mask(https://arxiv.org/abs/2410.20436)
• Keywords: robust, segmentation
• Abstract: Images of coral reefs provide invaluable information, which is essentially critical for surveying and monitoring the coral reef ecosystems. Robust and precise identification of coral reef regions within surveying imagery is paramount for assessing coral coverage, spatial distribution, and other statistical analyses. However, existing coral reef analytical approaches mainly focus on sparse points sampled from the whole imagery, which are highly subject to the sampling density and cannot accurately express the coral ambulance. Meanwhile, the analysis is both time-consuming and labor-intensive, and it is also limited to coral biologists. In this work, we propose CoralSCOP-LAT, an automatic and semi-automatic coral reef labeling and analysis tool, specially designed to segment coral reef regions (dense pixel masks) in coral reef images, significantly promoting analysis proficiency and accuracy. CoralSCOP-LAT leverages the advanced coral reef foundation model to accurately delineate coral regions, supporting dense coral reef analysis and reducing the dependency on manual annotation. The proposed CoralSCOP-LAT surpasses the existing tools by a large margin from analysis efficiency, accuracy, and flexibility. We perform comprehensive evaluations from various perspectives and the comparison demonstrates that CoralSCOP-LAT not only accelerates the coral reef analysis but also improves accuracy in coral segmentation and analysis. Our CoralSCOP-LAT, as the first dense coral reef analysis tool in the market, facilitates repeated large-scale coral reef monitoring analysis, contributing to more informed conservation efforts and sustainable management of coral reef ecosystems. Our tool will be available at this https URL.

Title: TEAFormers: TEnsor-Augmented Transformers for Multi-Dimensional Time Series Forecasting

• Authors: Linghang Kong, Elynn Chen, Yuzhou Chen, Yuefeng Han
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.20439
• Pdf URL: https://arxiv.org/pdf/2410.20439
• Copy Paste: [[2410.20439]] TEAFormers: TEnsor-Augmented Transformers for Multi-Dimensional Time Series Forecasting(https://arxiv.org/abs/2410.20439)
• Keywords: transformer
• Abstract: Multi-dimensional time series data, such as matrix and tensor-variate time series, are increasingly prevalent in fields such as economics, finance, and climate science. Traditional Transformer models, though adept with sequential data, do not effectively preserve these multi-dimensional structures, as their internal operations in effect flatten multi-dimensional observations into vectors, thereby losing critical multi-dimensional relationships and patterns. To address this, we introduce the Tensor-Augmented Transformer (TEAFormer), a novel method that incorporates tensor expansion and compression within the Transformer framework to maintain and leverage the inherent multi-dimensional structures, thus reducing computational costs and improving prediction accuracy. The core feature of the TEAFormer, the Tensor-Augmentation (TEA) module, utilizes tensor expansion to enhance multi-view feature learning and tensor compression for efficient information aggregation and reduced computational load. The TEA module is not just a specific model architecture but a versatile component that is highly compatible with the attention mechanism and the encoder-decoder structure of Transformers, making it adaptable to existing Transformer architectures. Our comprehensive experiments, which integrate the TEA module into three popular time series Transformer models across three real-world benchmarks, show significant performance enhancements, highlighting the potential of TEAFormers for cutting-edge time series forecasting.

Title: TrajAgent: An Agent Framework for Unified Trajectory Modelling

• Authors: Yuwei Du, Jie Feng, Jie Zhao, Yong Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20445
• Pdf URL: https://arxiv.org/pdf/2410.20445
• Copy Paste: [[2410.20445]] TrajAgent: An Agent Framework for Unified Trajectory Modelling(https://arxiv.org/abs/2410.20445)
• Keywords: large language model
• Abstract: Trajectory modeling, which includes research on trajectory data pattern mining and future prediction, has widespread applications in areas such as life services, urban transportation, and public administration. Numerous methods have been proposed to address specific problems within trajectory modelling. However, due to the heterogeneity of data and the diversity of trajectory tasks, achieving unified trajectory modelling remains an important yet challenging task. In this paper, we propose TrajAgent, a large language model-based agentic framework, to unify various trajectory modelling tasks. In TrajAgent, we first develop UniEnv, an execution environment with a unified data and model interface, to support the execution and training of various models. Building on UniEnv, we introduce TAgent, an agentic workflow designed for automatic trajectory modelling across various trajectory tasks. Specifically, we design AutOpt, a systematic optimization module within TAgent, to further improve the performance of the integrated model. With diverse trajectory tasks input in natural language, TrajAgent automatically generates competitive results via training and executing appropriate models. Extensive experiments on four tasks using four real-world datasets demonstrate the effectiveness of TrajAgent in unified trajectory modelling, achieving an average performance improvement of 15.43% over baseline methods.

Title: Unlocking Comics: The AI4VA Dataset for Visual Understanding

• Authors: Peter Grönquist, Deblina Bhattacharjee, Bahar Aydemir, Baran Ozaydin, Tong Zhang, Mathieu Salzmann, Sabine Süsstrunk
• Subjects: cs.CV, cs.ET
• Abstract URL: https://arxiv.org/abs/2410.20459
• Pdf URL: https://arxiv.org/pdf/2410.20459
• Copy Paste: [[2410.20459]] Unlocking Comics: The AI4VA Dataset for Visual Understanding(https://arxiv.org/abs/2410.20459)
• Keywords: segmentation
• Abstract: In the evolving landscape of deep learning, there is a pressing need for more comprehensive datasets capable of training models across multiple modalities. Concurrently, in digital humanities, there is a growing demand to leverage technology for diverse media adaptation and creation, yet limited by sparse datasets due to copyright and stylistic constraints. Addressing this gap, our paper presents a novel dataset comprising Franco-Belgian comics from the 1950s annotated for tasks including depth estimation, semantic segmentation, saliency detection, and character identification. It consists of two distinct and consistent styles and incorporates object concepts and labels taken from natural images. By including such diverse information across styles, this dataset not only holds promise for computational creativity but also offers avenues for the digitization of art and storytelling innovation. This dataset is a crucial component of the AI4VA Workshop Challenges~\url{this https URL}, where we specifically explore depth and saliency. Dataset details at \url{this https URL}.

Title: Hamiltonian Score Matching and Generative Flows

• Authors: Peter Holderrieth, Yilun Xu, Tommi Jaakkola
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.20470
• Pdf URL: https://arxiv.org/pdf/2410.20470
• Copy Paste: [[2410.20470]] Hamiltonian Score Matching and Generative Flows(https://arxiv.org/abs/2410.20470)
• Keywords: diffusion, generative
• Abstract: Classical Hamiltonian mechanics has been widely used in machine learning in the form of Hamiltonian Monte Carlo for applications with predetermined force fields. In this work, we explore the potential of deliberately designing force fields for Hamiltonian ODEs, introducing Hamiltonian velocity predictors (HVPs) as a tool for score matching and generative models. We present two innovations constructed with HVPs: Hamiltonian Score Matching (HSM), which estimates score functions by augmenting data via Hamiltonian trajectories, and Hamiltonian Generative Flows (HGFs), a novel generative model that encompasses diffusion models and flow matching as HGFs with zero force fields. We showcase the extended design space of force fields by introducing Oscillation HGFs, a generative model inspired by harmonic oscillators. Our experiments validate our theoretical insights about HSM as a novel score matching metric and demonstrate that HGFs rival leading generative modeling techniques.

Title: GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation

• Authors: Phillip Y. Lee, Taehoon Yoon, Minhyuk Sung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20474
• Pdf URL: https://arxiv.org/pdf/2410.20474
• Copy Paste: [[2410.20474]] GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation(https://arxiv.org/abs/2410.20474)
• Keywords: robust, diffusion, transformer
• Abstract: We introduce a novel training-free spatial grounding technique for text-to-image generation using Diffusion Transformers (DiT). Spatial grounding with bounding boxes has gained attention for its simplicity and versatility, allowing for enhanced user control in image generation. However, prior training-free approaches often rely on updating the noisy image during the reverse diffusion process via backpropagation from custom loss functions, which frequently struggle to provide precise control over individual bounding boxes. In this work, we leverage the flexibility of the Transformer architecture, demonstrating that DiT can generate noisy patches corresponding to each bounding box, fully encoding the target object and allowing for fine-grained control over each region. Our approach builds on an intriguing property of DiT, which we refer to as semantic sharing. Due to semantic sharing, when a smaller patch is jointly denoised alongside a generatable-size image, the two become "semantic clones". Each patch is denoised in its own branch of the generation process and then transplanted into the corresponding region of the original noisy image at each timestep, resulting in robust spatial grounding for each bounding box. In our experiments on the HRS and DrawBench benchmarks, we achieve state-of-the-art performance compared to previous training-free spatial grounding approaches.

Title: What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration

• Authors: Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, Wanxiang Che
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20482
• Pdf URL: https://arxiv.org/pdf/2410.20482
• Copy Paste: [[2410.20482]] What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration(https://arxiv.org/abs/2410.20482)
• Keywords: large language model
• Abstract: Recently, rapid advancements in Multi-Modal In-Context Learning (MM-ICL) have achieved notable success, which is capable of achieving superior performance across various tasks without requiring additional parameter tuning. However, the underlying rules for the effectiveness of MM-ICL remain under-explored. To fill this gap, this work aims to investigate the research question: "What factors affect the performance of MM-ICL?'' To this end, we investigate extensive experiments on the three core steps of MM-ICL including demonstration retrieval, demonstration ordering, and prompt construction using 6 vision large language models and 20 strategies. Our findings highlight (1) the necessity of a multi-modal retriever for demonstration retrieval, (2) the importance of intra-demonstration ordering over inter-demonstration ordering, and (3) the enhancement of task comprehension through introductory instructions in prompts. We hope this study can serve as a foundational guide for optimizing MM-ICL strategies in future research.

Title: Improving Decision Sparsity

• Authors: Yiyang Sun, Tong Wang, Cynthia Rudin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20483
• Pdf URL: https://arxiv.org/pdf/2410.20483
• Copy Paste: [[2410.20483]] Improving Decision Sparsity(https://arxiv.org/abs/2410.20483)
• Keywords: interpretability
• Abstract: Sparsity is a central aspect of interpretability in machine learning. Typically, sparsity is measured in terms of the size of a model globally, such as the number of variables it uses. However, this notion of sparsity is not particularly relevant for decision-making; someone subjected to a decision does not care about variables that do not contribute to the decision. In this work, we dramatically expand a notion of decision sparsity called the Sparse Explanation Value(SEV) so that its explanations are more meaningful. SEV considers movement along a hypercube towards a reference point. By allowing flexibility in that reference and by considering how distances along the hypercube translate to distances in feature space, we can derive sparser and more meaningful explanations for various types of function classes. We present cluster-based SEV and its variant tree-based SEV, introduce a method that improves credibility of explanations, and propose algorithms that optimize decision sparsity in machine learning models.

Title: FIRP: Faster LLM inference via future intermediate representation prediction

• Authors: Pengfei Wu, Jiahao Liu, Zhuocheng Gong, Qifan Wang, Jinpeng Li, Jingang Wang, Xunliang Cai, Dongyan Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20488
• Pdf URL: https://arxiv.org/pdf/2410.20488
• Copy Paste: [[2410.20488]] FIRP: Faster LLM inference via future intermediate representation prediction(https://arxiv.org/abs/2410.20488)
• Keywords: large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have shown remarkable performance across a wide range of tasks. Despite this, the auto-regressive nature of LLM decoding, which generates only a single token per forward propagation, fails to fully exploit the parallel computational power of GPUs, leading to considerable latency. To address this, we introduce a novel speculative decoding method named FIRP which generates multiple tokens instead of one at each decoding step. We achieve this by predicting the intermediate hidden states of future tokens (tokens have not been decoded yet) and then using these pseudo hidden states to decode future tokens, specifically, these pseudo hidden states are predicted with simple linear transformation in intermediate layers of LLMs. Once predicted, they participate in the computation of all the following layers, thereby assimilating richer semantic information. As the layers go deeper, the semantic gap between pseudo and real hidden states is narrowed and it becomes feasible to decode future tokens with high accuracy. To validate the effectiveness of FIRP, we conduct extensive experiments, showing a speedup ratio of 1.9x-3x in several models and datasets, analytical experiments also prove our motivations.

Title: $\textit{Who Speaks Matters}$: Analysing the Influence of the Speaker's Ethnicity on Hate Classification

• Authors: Ananya Malik, Kartik Sharma, Lynnette Hui Xian Ng, Shaily Bhatt
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20490
• Pdf URL: https://arxiv.org/pdf/2410.20490
• Copy Paste: [[2410.20490]] $\textit{Who Speaks Matters}$: Analysing the Influence of the Speaker's Ethnicity on Hate Classification(https://arxiv.org/abs/2410.20490)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) offer a lucrative promise for scalable content moderation, including hate speech detection. However, they are also known to be brittle and biased against marginalised communities and dialects. This requires their applications to high-stakes tasks like hate speech detection to be critically scrutinized. In this work, we investigate the robustness of hate speech classification using LLMs, particularly when explicit and implicit markers of the speaker's ethnicity are injected into the input. For the explicit markers, we inject a phrase that mentions the speaker's identity. For the implicit markers, we inject dialectal features. By analysing how frequently model outputs flip in the presence of these markers, we reveal varying degrees of brittleness across 4 popular LLMs and 5 ethnicities. We find that the presence of implicit dialect markers in inputs causes model outputs to flip more than the presence of explicit markers. Further, the percentage of flips varies across ethnicities. Finally, we find that larger models are more robust. Our findings indicate the need for exercising caution in deploying LLMs for high-stakes tasks like hate speech detection.

Title: MatViX: Multimodal Information Extraction from Visually Rich Articles

• Authors: Ghazal Khalighinejad, Sharon Scott, Ollie Liu, Kelly L. Anderson, Rickard Stureborg, Aman Tyagi, Bhuwan Dhingra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20494
• Pdf URL: https://arxiv.org/pdf/2410.20494
• Copy Paste: [[2410.20494]] MatViX: Multimodal Information Extraction from Visually Rich Articles(https://arxiv.org/abs/2410.20494)
• Keywords: extraction
• Abstract: Multimodal information extraction (MIE) is crucial for scientific literature, where valuable data is often spread across text, figures, and tables. In materials science, extracting structured information from research articles can accelerate the discovery of new materials. However, the multimodal nature and complex interconnections of scientific content present challenges for traditional text-based methods. We introduce \textsc{MatViX}, a benchmark consisting of $324$ full-length research articles and $1,688$ complex structured JSON files, carefully curated by domain experts. These JSON files are extracted from text, tables, and figures in full-length documents, providing a comprehensive challenge for MIE. We introduce an evaluation method to assess the accuracy of curve similarity and the alignment of hierarchical structures. Additionally, we benchmark vision-language models (VLMs) in a zero-shot manner, capable of processing long contexts and multimodal inputs, and show that using a specialized model (DePlot) can improve performance in extracting curves. Our results demonstrate significant room for improvement in current models. Our dataset and evaluation code are available\footnote{\url{this https URL}}.

Title: ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation

• Authors: Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, Furu Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20502
• Pdf URL: https://arxiv.org/pdf/2410.20502
• Copy Paste: [[2410.20502]] ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation(https://arxiv.org/abs/2410.20502)
• Keywords: diffusion, transformer
• Abstract: Text-to-video models have recently undergone rapid and substantial advancements. Nevertheless, due to limitations in data and computational resources, achieving efficient generation of long videos with rich motion dynamics remains a significant challenge. To generate high-quality, dynamic, and temporally consistent long videos, this paper presents ARLON, a novel framework that boosts diffusion Transformers with autoregressive models for long video generation, by integrating the coarse spatial and long-range temporal information provided by the AR model to guide the DiT model. Specifically, ARLON incorporates several key innovations: 1) A latent Vector Quantized Variational Autoencoder (VQ-VAE) compresses the input latent space of the DiT model into compact visual tokens, bridging the AR and DiT models and balancing the learning complexity and information density; 2) An adaptive norm-based semantic injection module integrates the coarse discrete visual units from the AR model into the DiT model, ensuring effective guidance during video generation; 3) To enhance the tolerance capability of noise introduced from the AR inference, the DiT model is trained with coarser visual latent tokens incorporated with an uncertainty sampling module. Experimental results demonstrate that ARLON significantly outperforms the baseline OpenSora-V1.2 on eight out of eleven metrics selected from VBench, with notable improvements in dynamic degree and aesthetic quality, while delivering competitive results on the remaining three and simultaneously accelerating the generation process. In addition, ARLON achieves state-of-the-art performance in long video generation. Detailed analyses of the improvements in inference efficiency are presented, alongside a practical application that demonstrates the generation of long videos using progressive text prompts. See demos of ARLON at \url{this http URL}.

Title: Is Moral Self-correction An Innate Capability of Large Language Models? A Mechanistic Analysis to Self-correction

• Authors: Zimo Qi, Guangliang Liu, Kristen Marie Johnson, Lu Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20513
• Pdf URL: https://arxiv.org/pdf/2410.20513
• Copy Paste: [[2410.20513]] Is Moral Self-correction An Innate Capability of Large Language Models? A Mechanistic Analysis to Self-correction(https://arxiv.org/abs/2410.20513)
• Keywords: robust, large language model
• Abstract: Though intensive attentions to the self-correction capability of Large Language Models (LLMs), the underlying mechanism of this capability is still under-explored. In this paper, we aim to answer two fundamental questions for moral self-correction: (1) how different components in self-correction, such as Chain-of-Thought (CoT) reasoning, external feedback, and instructional prompts, interact to enable moral self-correction; and (2) is the self-correction one of LLMs' innate capabilities? To answer the first question, we examine how different self-correction components interact to intervene the embedded morality within hidden states, therefore contributing to different performance. For the second question, we (i) evaluate the robustness of moral self-correction by introducing natural language interventions of weak evidence into prompts; (ii) propose a validation framework, self-distinguish, that requires effective self-correction to enable LLMs to distinguish between desirable and undesirable outputs. Our experimental results indicate that there is no universally optimal self-correction method for the tasks considered, although external feedback and CoT can contribute to additional performance gains. However, our mechanistic analysis reveals negative interactions among instructional prompts, CoT, and external feedback, suggesting a conflict between internal knowledge and external feedback. The self-distinguish experiments demonstrate that while LLMs can self-correct their responses, they are unable to reliably distinguish between desired and undesired outputs. With our empirical evidence, we can conclude that moral self-correction is not an innate capability of LLMs acquired during pretraining.

Title: A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing

• Authors: Julia Balla, Siddharth Mishra-Sharma, Carolina Cuesta-Lazaro, Tommi Jaakkola, Tess Smidt
• Subjects: cs.LG, astro-ph.IM
• Abstract URL: https://arxiv.org/abs/2410.20516
• Pdf URL: https://arxiv.org/pdf/2410.20516
• Copy Paste: [[2410.20516]] A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing(https://arxiv.org/abs/2410.20516)
• Keywords: extraction
• Abstract: Efficiently processing structured point cloud data while preserving multiscale information is a key challenge across domains, from graphics to atomistic modeling. Using a curated dataset of simulated galaxy positions and properties, represented as point clouds, we benchmark the ability of graph neural networks to simultaneously capture local clustering environments and long-range correlations. Given the homogeneous and isotropic nature of the Universe, the data exhibits a high degree of symmetry. We therefore focus on evaluating the performance of Euclidean symmetry-preserving ($E(3)$-equivariant) graph neural networks, showing that they can outperform non-equivariant counterparts and domain-specific information extraction techniques in downstream performance as well as simulation-efficiency. However, we find that current architectures fail to capture information from long-range correlations as effectively as domain-specific baselines, motivating future work on architectures better suited for extracting long-range information.

Title: Fractal and Turbulent Feature Extraction and NFT Label Generation for Pollock Style Migration Paintings Based on VGG19

• Authors: Yiquan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20519
• Pdf URL: https://arxiv.org/pdf/2410.20519
• Copy Paste: [[2410.20519]] Fractal and Turbulent Feature Extraction and NFT Label Generation for Pollock Style Migration Paintings Based on VGG19(https://arxiv.org/abs/2410.20519)
• Keywords: protect, extraction
• Abstract: This paper puts forth an innovative approach that fuses deep learning, fractal analysis, and turbulence feature extraction techniques to create abstract artworks in the style of Pollock. The content and style characteristics of the image are extracted by the MindSpore deep learning framework and a pre-trained VGG19 model. An optimisation process is then employed to The method generates high-quality Pollock-style images by combining content loss, style loss and full variance loss to achieve accurate style migration. Furthermore, this paper implements a fractal dimension calculation method based on the difference box-counting method, which effectively estimates the fractal dimension of an image through edge extraction and fractal analysis. The method is based on a two-dimensional discrete wavelet transform using a Haar wavelet to decompose the image in order to extract different frequency information. This is followed by the combination of multiple features to generate unique non-homogeneous token (NFT) labels for the authentication and protection of digital artwork. The experimental results demonstrate that the generated artworks exhibit The method demonstrates significant diversity and complexity in terms of fractal dimensions and turbulence features, while the generated NFT tags ensure the uniqueness and tamperability of each digital collection. The present method organically combines computer vision, digital signal processing and blockchain technology to provide a new solution for the creation and authentication of digital artworks.

Title: Props for Machine-Learning Security

• Authors: Ari Juels, Farinaz Koushanfar
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20522
• Pdf URL: https://arxiv.org/pdf/2410.20522
• Copy Paste: [[2410.20522]] Props for Machine-Learning Security(https://arxiv.org/abs/2410.20522)
• Keywords: secure, security, privacy, protect
• Abstract: We propose protected pipelines or props for short, a new approach for authenticated, privacy-preserving access to deep-web data for machine learning (ML). By permitting secure use of vast sources of deep-web data, props address the systemic bottleneck of limited high-quality training data in ML development. Props also enable privacy-preserving and trustworthy forms of inference, allowing for safe use of sensitive data in ML applications. Props are practically realizable today by leveraging privacy-preserving oracle systems initially developed for blockchain applications.

Title: Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders

• Authors: Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, Xipeng Qiu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20526
• Pdf URL: https://arxiv.org/pdf/2410.20526
• Copy Paste: [[2410.20526]] Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders(https://arxiv.org/abs/2410.20526)
• Keywords: interpretability
• Abstract: Sparse Autoencoders (SAEs) have emerged as a powerful unsupervised method for extracting sparse representations from language models, yet scalable training remains a significant challenge. We introduce a suite of 256 SAEs, trained on each layer and sublayer of the Llama-3.1-8B-Base model, with 32K and 128K features. Modifications to a state-of-the-art SAE variant, Top-K SAEs, are evaluated across multiple dimensions. In particular, we assess the generalizability of SAEs trained on base models to longer contexts and fine-tuned models. Additionally, we analyze the geometry of learned SAE latents, confirming that \emph{feature splitting} enables the discovery of new features. The Llama Scope SAE checkpoints are publicly available at~\url{this https URL}, alongside our scalable training, interpretation, and visualization tools at \url{this https URL}. These contributions aim to advance the open-source Sparse Autoencoder ecosystem and support mechanistic interpretability research by reducing the need for redundant SAE training.

Title: PaPaGei: Open Foundation Models for Optical Physiological Signals

• Authors: Arvind Pillai, Dimitris Spathis, Fahim Kawsar, Mohammad Malekzadeh
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.20542
• Pdf URL: https://arxiv.org/pdf/2410.20542
• Copy Paste: [[2410.20542]] PaPaGei: Open Foundation Models for Optical Physiological Signals(https://arxiv.org/abs/2410.20542)
• Keywords: robust
• Abstract: Photoplethysmography (PPG) is the most widely used non-invasive technique for monitoring biosignals and cardiovascular health, with applications in both clinical settings and consumer health through wearable devices. Current machine learning models trained on PPG signals are mostly task-specific and lack generalizability. Previous works often used single-device datasets, did not explore out-of-domain generalization, or did not release their models, hindering reproducibility and further research. We introduce PaPaGei, the first open foundation model for PPG signals. PaPaGei is pre-trained on more than 57,000 hours of 20 million unlabeled segments of PPG signals using publicly available datasets exclusively. We evaluate against popular time-series foundation models and other benchmarks on 20 tasks of 10 diverse datasets spanning cardiovascular health, sleep disorders, pregnancy monitoring, and wellbeing assessment. Our architecture incorporates novel representation learning approaches that leverage differences in PPG signal morphology across individuals, enabling it to capture richer representations than traditional contrastive learning methods. Across 20 tasks, PaPaGei improves classification and regression performance by an average of 6.3% and 2.9%, respectively, compared to other competitive time-series foundation models in at least 14 tasks. PaPaGei is more data- and parameter-efficient than other foundation models or methods, as it outperforms 70x larger models. Beyond accuracy, we also investigate robustness against different skin tones, establishing a benchmark for bias evaluations of future models. Notably, PaPaGei can be used out of the box as both a feature extractor and an encoder for other multimodal models, opening up new opportunities for multimodal health monitoring

Title: Privacy-Enhanced Adaptive Authentication: User Profiling with Privacy Guarantees

• Authors: Yaser Baseri, Abdelhakim Senhaji Hafid, Dimitrios Makrakis
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.20555
• Pdf URL: https://arxiv.org/pdf/2410.20555
• Copy Paste: [[2410.20555]] Privacy-Enhanced Adaptive Authentication: User Profiling with Privacy Guarantees(https://arxiv.org/abs/2410.20555)
• Keywords: security, privacy, protect, attack, robust
• Abstract: User profiling is a critical component of adaptive risk-based authentication, yet it raises significant privacy concerns, particularly when handling sensitive data. Profiling involves collecting and aggregating various user features, potentially creating quasi-identifiers that can reveal identities and compromise privacy. Even anonymized profiling methods remain vulnerable to re-identification attacks through these quasi-identifiers. This paper introduces a novel privacy-enhanced adaptive authentication protocol that leverages Oblivious Pseudorandom Functions (OPRF), anonymous tokens, and Differential Privacy (DP) to provide robust privacy guarantees. Our proposed approach dynamically adjusts authentication requirements based on real-time risk assessments, enhancing security while safeguarding user privacy. By integrating privacy considerations into the core of adaptive risk-based adaptive authentication, this approach addresses a gap often overlooked in traditional models. Advanced cryptographic techniques ensure confidentiality, integrity, and unlinkability of user data, while differential privacy mechanisms minimize the impact of individual data points on overall analysis. Formal security and privacy proofs demonstrate the protocol's resilience against various threats and its ability to provide strong privacy guarantees. Additionally, a comprehensive performance evaluation reveals that the computational and communication overheads are manageable, making the protocol practical for real-world deployment. By adhering to data protection regulations such as GDPR and CCPA, our protocol not only enhances security but also fosters user trust and compliance with legal standards.

Title: Unsupervised Panoptic Interpretation of Latent Spaces in GANs Using Space-Filling Vector Quantization

• Authors: Mohammad Hassan Vali, Tom Bäckström
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20573
• Pdf URL: https://arxiv.org/pdf/2410.20573
• Copy Paste: [[2410.20573]] Unsupervised Panoptic Interpretation of Latent Spaces in GANs Using Space-Filling Vector Quantization(https://arxiv.org/abs/2410.20573)
• Keywords: generative
• Abstract: Generative adversarial networks (GANs) learn a latent space whose samples can be mapped to real-world images. Such latent spaces are difficult to interpret. Some earlier supervised methods aim to create an interpretable latent space or discover interpretable directions that require exploiting data labels or annotated synthesized samples for training. However, we propose using a modification of vector quantization called space-filling vector quantization (SFVQ), which quantizes the data on a piece-wise linear curve. SFVQ can capture the underlying morphological structure of the latent space and thus make it interpretable. We apply this technique to model the latent space of pretrained StyleGAN2 and BigGAN networks on various datasets. Our experiments show that the SFVQ curve yields a general interpretable model of the latent space that determines which part of the latent space corresponds to what specific generative factors. Furthermore, we demonstrate that each line of SFVQ's curve can potentially refer to an interpretable direction for applying intelligible image transformations. We also showed that the points located on an SFVQ line can be used for controllable data augmentation.

Title: Encrypted system identification as-a-service via reliable encrypted matrix inversion

• Authors: Janis Adamek, Philipp Binfet, Nils Schlüter, Moritz Schulze Darup
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.20575
• Pdf URL: https://arxiv.org/pdf/2410.20575
• Copy Paste: [[2410.20575]] Encrypted system identification as-a-service via reliable encrypted matrix inversion(https://arxiv.org/abs/2410.20575)
• Keywords: privacy
• Abstract: Encrypted computation opens up promising avenues across a plethora of application domains, including machine learning, health-care, finance, and control. Arithmetic homomorphic encryption, in particular, is a natural fit for cloud-based computational services. However, computations are essentially limited to polynomial circuits, while comparisons, transcendental functions, and iterative algorithms are notoriously hard to realize. Against this background, the paper presents an encrypted system identification service enabled by a reliable encrypted solution to least squares problems. More precisely, we devise an iterative algorithm for matrix inversion and present reliable initializations as well as certificates for the achieved accuracy without compromising the privacy of provided I/O-data. The effectiveness of the approach is illustrated with three popular identification tasks.

Title: Generator Matching: Generative modeling with arbitrary Markov processes

• Authors: Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, Yaron Lipman
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20587
• Pdf URL: https://arxiv.org/pdf/2410.20587
• Copy Paste: [[2410.20587]] Generator Matching: Generative modeling with arbitrary Markov processes(https://arxiv.org/abs/2410.20587)
• Keywords: diffusion, generative
• Abstract: We introduce generator matching, a modality-agnostic framework for generative modeling using arbitrary Markov processes. Generators characterize the infinitesimal evolution of a Markov process, which we leverage for generative modeling in a similar vein to flow matching: we construct conditional generators which generate single data points, then learn to approximate the marginal generator which generates the full data distribution. We show that generator matching unifies various generative modeling methods, including diffusion models, flow matching and discrete diffusion models. Furthermore, it provides the foundation to expand the design space to new and unexplored Markov processes such as jump processes. Finally, generator matching enables the construction of superpositions of Markov generative processes and enables the construction of multimodal models in a rigorous manner. We empirically validate our method on protein and image structure generation, showing that superposition with a jump process improves image generation.

Title: A Framework for Real-Time Volcano-Seismic Event Recognition Based on Multi-Station Seismograms and Semantic Segmentation Models

• Authors: Camilo Espinosa-Curilem, Millaray Curilem, Daniel Basualto
• Subjects: cs.CV, cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.20595
• Pdf URL: https://arxiv.org/pdf/2410.20595
• Copy Paste: [[2410.20595]] A Framework for Real-Time Volcano-Seismic Event Recognition Based on Multi-Station Seismograms and Semantic Segmentation Models(https://arxiv.org/abs/2410.20595)
• Keywords: robust, segmentation
• Abstract: In volcano monitoring, effective recognition of seismic events is essential for understanding volcanic activity and raising timely warning alerts. Traditional methods rely on manual analysis, which can be subjective and labor-intensive. Furthermore, current automatic approaches often tackle detection and classification separately, mostly rely on single station information and generally require tailored preprocessing and representations to perform predictions. These limitations often hinder their application to real-time monitoring and utilization across different volcano conditions. This study introduces a novel approach that utilizes Semantic Segmentation models to automate seismic event recognition by applying a straight forward transformation of multi-channel 1D signals into 2D representations, enabling their use as images. Our framework employs a data-driven, end-to-end design that integrates multi-station seismic data with minimal preprocessing, performing both detection and classification simultaneously for five seismic event classes. We evaluated four state-of-the-art segmentation models (UNet, UNet++, DeepLabV3+ and SwinUNet) on approximately 25.000 seismic events recorded at four different Chilean volcanoes: Nevados del Chillán Volcanic Complex, Laguna del Maule, Villarrica and Puyehue-Cordón Caulle. Among these models, the UNet architecture was identified as the most effective model, achieving mean F1 and Intersection over Union (IoU) scores of up to 0.91 and 0.88, respectively, and demonstrating superior noise robustness and model flexibility to unseen volcano datasets.

Title: Zero-Trust Network Access (ZTNA)

• Authors: Vasilios Mavroudis
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2410.20611
• Pdf URL: https://arxiv.org/pdf/2410.20611
• Copy Paste: [[2410.20611]] Zero-Trust Network Access (ZTNA)(https://arxiv.org/abs/2410.20611)
• Keywords: security
• Abstract: Zero-Trust Network Access (ZTNA) marks a significant shift in network security by adopting a "never trust, always verify" approach. This work provides an in-depth analysis of ZTNA, offering a comprehensive framework for understanding its principles, architectures, and applications. We discuss its role in securing modern, complex network environments, which include cloud platforms, Internet of Things (IoT) devices, and hybrid enterprise networks. Our objective is to create a key resource for researchers and practitioners by reviewing critical methodologies, analyzing current implementations, and highlighting open challenges and research directions.

Title: LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization

• Authors: Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S. Dhillon, Cho-Jui Hsieh, Sanjiv Kumar
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20625
• Pdf URL: https://arxiv.org/pdf/2410.20625
• Copy Paste: [[2410.20625]] LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization(https://arxiv.org/abs/2410.20625)
• Keywords: robust
• Abstract: Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the actual updates to the weights depends on how the two LoRA factors are scaled or rotated. This deficiency leads to inefficient learning and sub-optimal solutions in practice. This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization, which can achieve transformation invariance and remain computationally efficient. We provide theoretical analysis to demonstrate the benefit of our method and conduct experiments on various LLM tasks with different models including Gemma 2B, 7B, and mT5-XXL. The results demonstrate consistent improvements against existing optimizers. For example, replacing Adam with LoRA-RITE during LoRA fine-tuning of Gemma-2B yielded 4.6\% accuracy gain on Super-Natural Instructions and 3.5\% accuracy gain across other four LLM benchmarks (HellaSwag, ArcChallenge, GSM8K, OpenBookQA).

Title: TabDiff: a Multi-Modal Diffusion Model for Tabular Data Generation

• Authors: Juntong Shi, Minkai Xu, Harper Hua, Hengrui Zhang, Stefano Ermon, Jure Leskovec
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20626
• Pdf URL: https://arxiv.org/pdf/2410.20626
• Copy Paste: [[2410.20626]] TabDiff: a Multi-Modal Diffusion Model for Tabular Data Generation(https://arxiv.org/abs/2410.20626)
• Keywords: privacy, protect, diffusion, transformer, generative
• Abstract: Synthesizing high-quality tabular data is an important topic in many data science tasks, ranging from dataset augmentation to privacy protection. However, developing expressive generative models for tabular data is challenging due to its inherent heterogeneous data types, complex inter-correlations, and intricate column-wise distributions. In this paper, we introduce TabDiff, a joint diffusion framework that models all multi-modal distributions of tabular data in one model. Our key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data, where we propose feature-wise learnable diffusion processes to counter the high disparity of different feature distributions. TabDiff is parameterized by a transformer handling different input types, and the entire framework can be efficiently optimized in an end-to-end fashion. We further introduce a multi-modal stochastic sampler to automatically correct the accumulated decoding error during sampling, and propose classifier-free guidance for conditional missing column value imputation. Comprehensive experiments on seven datasets demonstrate that TabDiff achieves superior average performance over existing competitive baselines across all eight metrics, with up to $22.5\%$ improvement over the state-of-the-art model on pair-wise column correlation estimations. Code is available at this https URL.

Title: PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection

• Authors: Tianhao Zhang, Zhixiang Chen, Lyudmila S. Mihaylova
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20631
• Pdf URL: https://arxiv.org/pdf/2410.20631
• Copy Paste: [[2410.20631]] PViT: Prior-augmented Vision Transformer for Out-of-distribution Detection(https://arxiv.org/abs/2410.20631)
• Keywords: robust, transformer
• Abstract: Vision Transformers (ViTs) have achieved remarkable success over various vision tasks, yet their robustness against data distribution shifts and inherent inductive biases remain underexplored. To enhance the robustness of ViT models for image Out-of-Distribution (OOD) detection, we introduce a novel and generic framework named Prior-augmented Vision Transformer (PViT). PViT identifies OOD samples by quantifying the divergence between the predicted class logits and the prior logits obtained from pre-trained models. Unlike existing state-of-the-art OOD detection methods, PViT shapes the decision boundary between ID and OOD by utilizing the proposed prior guide confidence, without requiring additional data modeling, generation methods, or structural modifications. Extensive experiments on the large-scale ImageNet benchmark demonstrate that PViT significantly outperforms existing state-of-the-art OOD detection methods. Additionally, through comprehensive analyses, ablation studies, and discussions, we show how PViT can strategically address specific challenges in managing large vision models, paving the way for new advancements in OOD detection.

Title: Language Models And A Second Opinion Use Case: The Pocket Professional

• Authors: David Noever
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20636
• Pdf URL: https://arxiv.org/pdf/2410.20636
• Copy Paste: [[2410.20636]] Language Models And A Second Opinion Use Case: The Pocket Professional(https://arxiv.org/abs/2410.20636)
• Keywords: large language model
• Abstract: This research tests the role of Large Language Models (LLMs) as formal second opinion tools in professional decision-making, particularly focusing on complex medical cases where even experienced physicians seek peer consultation. The work analyzed 183 challenging medical cases from Medscape over a 20-month period, testing multiple LLMs' performance against crowd-sourced physician responses. A key finding was the high overall score possible in the latest foundational models (>80% accuracy compared to consensus opinion), which exceeds most human metrics reported on the same clinical cases (450 pages of patient profiles, test results). The study rates the LLMs' performance disparity between straightforward cases (>81% accuracy) and complex scenarios (43% accuracy), particularly in these cases generating substantial debate among human physicians. The research demonstrates that LLMs may be valuable as generators of comprehensive differential diagnoses rather than as primary diagnostic tools, potentially helping to counter cognitive biases in clinical decision-making, reduce cognitive loads, and thus remove some sources of medical error. The inclusion of a second comparative legal dataset (Supreme Court cases, N=21) provides added empirical context to the AI use to foster second opinions, though these legal challenges proved considerably easier for LLMs to analyze. In addition to the original contributions of empirical evidence for LLM accuracy, the research aggregated a novel benchmark for others to score highly contested question and answer reliability between both LLMs and disagreeing human practitioners. These results suggest that the optimal deployment of LLMs in professional settings may differ substantially from current approaches that emphasize automation of routine tasks.

Title: Visualizing attention zones in machine reading comprehension models

• Authors: Yiming Cui, Wei-Nan Zhang, Ting Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20652
• Pdf URL: https://arxiv.org/pdf/2410.20652
• Copy Paste: [[2410.20652]] Visualizing attention zones in machine reading comprehension models(https://arxiv.org/abs/2410.20652)
• Keywords: explainability
• Abstract: The attention mechanism plays an important role in the machine reading comprehension (MRC) model. Here, we describe a pipeline for building an MRC model with a pretrained language model and visualizing the effect of each attention zone in different layers, which can indicate the explainability of the model. With the presented protocol and accompanying code, researchers can easily visualize the relevance of each attention zone in the MRC model. This approach can be generalized to other pretrained language models.

Title: Video to Video Generative Adversarial Network for Few-shot Learning Based on Policy Gradient

• Authors: Yintai Ma, Diego Klabjan, Jean Utke
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20657
• Pdf URL: https://arxiv.org/pdf/2410.20657
• Copy Paste: [[2410.20657]] Video to Video Generative Adversarial Network for Few-shot Learning Based on Policy Gradient(https://arxiv.org/abs/2410.20657)
• Keywords: generative
• Abstract: The development of sophisticated models for video-to-video synthesis has been facilitated by recent advances in deep reinforcement learning and generative adversarial networks (GANs). In this paper, we propose RL-V2V-GAN, a new deep neural network approach based on reinforcement learning for unsupervised conditional video-to-video synthesis. While preserving the unique style of the source video domain, our approach aims to learn a mapping from a source video domain to a target video domain. We train the model using policy gradient and employ ConvLSTM layers to capture the spatial and temporal information by designing a fine-grained GAN architecture and incorporating spatio-temporal adversarial goals. The adversarial losses aid in content translation while preserving style. Unlike traditional video-to-video synthesis methods requiring paired inputs, our proposed approach is more general because it does not require paired inputs. Thus, when dealing with limited videos in the target domain, i.e., few-shot learning, it is particularly effective. Our experiments show that RL-V2V-GAN can produce temporally coherent video results. These results highlight the potential of our approach for further advances in video-to-video synthesis.

Title: TurboHopp: Accelerated Molecule Scaffold Hopping with Consistency Models

• Authors: Kiwoong Yoo, Owen Oertell, Junhyun Lee, Sanghoon Lee, Jaewoo Kang
• Subjects: cs.LG, cs.AI, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.20660
• Pdf URL: https://arxiv.org/pdf/2410.20660
• Copy Paste: [[2410.20660]] TurboHopp: Accelerated Molecule Scaffold Hopping with Consistency Models(https://arxiv.org/abs/2410.20660)
• Keywords: diffusion, generative
• Abstract: Navigating the vast chemical space of druggable compounds is a formidable challenge in drug discovery, where generative models are increasingly employed to identify viable candidates. Conditional 3D structure-based drug design (3D-SBDD) models, which take into account complex three-dimensional interactions and molecular geometries, are particularly promising. Scaffold hopping is an efficient strategy that facilitates the identification of similar active compounds by strategically modifying the core structure of molecules, effectively narrowing the wide chemical space and enhancing the discovery of drug-like products. However, the practical application of 3D-SBDD generative models is hampered by their slow processing speeds. To address this bottleneck, we introduce TurboHopp, an accelerated pocket-conditioned 3D scaffold hopping model that merges the strategic effectiveness of traditional scaffold hopping with rapid generation capabilities of consistency models. This synergy not only enhances efficiency but also significantly boosts generation speeds, achieving up to 30 times faster inference speed as well as superior generation quality compared to existing diffusion-based models, establishing TurboHopp as a powerful tool in drug discovery. Supported by faster inference speed, we further optimize our model, using Reinforcement Learning for Consistency Models (RLCM), to output desirable molecules. We demonstrate the broad applicability of TurboHopp across multiple drug discovery scenarios, underscoring its potential in diverse molecular settings.

Title: Embedding with Large Language Models for Classification of HIPAA Safeguard Compliance Rules

• Authors: Md Abdur Rahman, Md Abdul Barek, ABM Kamrul Islam Riad, Md Mostafizur Rahman, Md Bajlur Rashid, Smita Ambedkar, Md Raihan Miaa, Fan Wu, Alfredo Cuzzocrea, Sheikh Iqbal Ahamed
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20664
• Pdf URL: https://arxiv.org/pdf/2410.20664
• Copy Paste: [[2410.20664]] Embedding with Large Language Models for Classification of HIPAA Safeguard Compliance Rules(https://arxiv.org/abs/2410.20664)
• Keywords: secure, security, privacy, protect, transformer, large language model
• Abstract: Although software developers of mHealth apps are responsible for protecting patient data and adhering to strict privacy and security requirements, many of them lack awareness of HIPAA regulations and struggle to distinguish between HIPAA rules categories. Therefore, providing guidance of HIPAA rules patterns classification is essential for developing secured applications for Google Play Store. In this work, we identified the limitations of traditional Word2Vec embeddings in processing code patterns. To address this, we adopt multilingual BERT (Bidirectional Encoder Representations from Transformers) which offers contextualized embeddings to the attributes of dataset to overcome the issues. Therefore, we applied this BERT to our dataset for embedding code patterns and then uses these embedded code to various machine learning approaches. Our results demonstrate that the models significantly enhances classification performance, with Logistic Regression achieving a remarkable accuracy of 99.95\%. Additionally, we obtained high accuracy from Support Vector Machine (99.79\%), Random Forest (99.73\%), and Naive Bayes (95.93\%), outperforming existing approaches. This work underscores the effectiveness and showcases its potential for secure application development.

Title: Segmenting Watermarked Texts From Language Models

• Authors: Xingchi Li, Guanxun Li, Xianyang Zhang
• Subjects: cs.LG, cs.MM, cs.NE, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.20670
• Pdf URL: https://arxiv.org/pdf/2410.20670
• Copy Paste: [[2410.20670]] Segmenting Watermarked Texts From Language Models(https://arxiv.org/abs/2410.20670)
• Keywords: watermark
• Abstract: Watermarking is a technique that involves embedding nearly unnoticeable statistical signals within generated content to help trace its source. This work focuses on a scenario where an untrusted third-party user sends prompts to a trusted language model (LLM) provider, who then generates a text from their LLM with a watermark. This setup makes it possible for a detector to later identify the source of the text if the user publishes it. The user can modify the generated text by substitutions, insertions, or deletions. Our objective is to develop a statistical method to detect if a published text is LLM-generated from the perspective of a detector. We further propose a methodology to segment the published text into watermarked and non-watermarked sub-strings. The proposed approach is built upon randomization tests and change point detection techniques. We demonstrate that our method ensures Type I and Type II error control and can accurately identify watermarked sub-strings by finding the corresponding change point locations. To validate our technique, we apply it to texts generated by several language models with prompts extracted from Google's C4 dataset and obtain encouraging numerical results. We release all code publicly at this https URL.

Title: Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

• Authors: Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, Tal Schuster
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20672
• Pdf URL: https://arxiv.org/pdf/2410.20672
• Copy Paste: [[2410.20672]] Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA(https://arxiv.org/abs/2410.20672)
• Keywords: fair, transformer, large language model
• Abstract: Large language models (LLMs) are expensive to deploy. Parameter sharing offers a possible path towards reducing their size and cost, but its effectiveness in modern LLMs remains fairly limited. In this work, we revisit "layer tying" as form of parameter sharing in Transformers, and introduce novel methods for converting existing LLMs into smaller "Recursive Transformers" that share parameters across layers, with minimal loss of performance. Here, our Recursive Transformers are efficiently initialized from standard pretrained Transformers, but only use a single block of unique layers that is then repeated multiple times in a loop. We further improve performance by introducing Relaxed Recursive Transformers that add flexibility to the layer tying constraint via depth-wise low-rank adaptation (LoRA) modules, yet still preserve the compactness of the overall model. We show that our recursive models (e.g., recursive Gemma 1B) outperform both similar-sized vanilla pretrained models (such as TinyLlama 1.1B and Pythia 1B) and knowledge distillation baselines -- and can even recover most of the performance of the original "full-size" model (e.g., Gemma 2B with no shared parameters). Finally, we propose Continuous Depth-wise Batching, a promising new inference paradigm enabled by the Recursive Transformer when paired with early exiting. In a theoretical analysis, we show that this has the potential to lead to significant (2-3x) gains in inference throughput.

Title: Reprogramming Pretrained Target-Specific Diffusion Models for Dual-Target Drug Design

• Authors: Xiangxin Zhou, Jiaqi Guan, Yijia Zhang, Xingang Peng, Liang Wang, Jianzhu Ma
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.20688
• Pdf URL: https://arxiv.org/pdf/2410.20688
• Copy Paste: [[2410.20688]] Reprogramming Pretrained Target-Specific Diffusion Models for Dual-Target Drug Design(https://arxiv.org/abs/2410.20688)
• Keywords: diffusion, generative
• Abstract: Dual-target therapeutic strategies have become a compelling approach and attracted significant attention due to various benefits, such as their potential in overcoming drug resistance in cancer therapy. Considering the tremendous success that deep generative models have achieved in structure-based drug design in recent years, we formulate dual-target drug design as a generative task and curate a novel dataset of potential target pairs based on synergistic drug combinations. We propose to design dual-target drugs with diffusion models that are trained on single-target protein-ligand complex pairs. Specifically, we align two pockets in 3D space with protein-ligand binding priors and build two complex graphs with shared ligand nodes for SE(3)-equivariant composed message passing, based on which we derive a composed drift in both 3D and categorical probability space in the generative process. Our algorithm can well transfer the knowledge gained in single-target pretraining to dual-target scenarios in a zero-shot manner. We also repurpose linker design methods as strong baselines for this task. Extensive experiments demonstrate the effectiveness of our method compared with various baselines.

Title: Combining Domain-Specific Models and LLMs for Automated Disease Phenotyping from Survey Data

• Authors: Gal Beeri, Benoit Chamot, Elena Latchem, Shruthi Venkatesh, Sarah Whalan, Van Zyl Kruger, David Martino
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20695
• Pdf URL: https://arxiv.org/pdf/2410.20695
• Copy Paste: [[2410.20695]] Combining Domain-Specific Models and LLMs for Automated Disease Phenotyping from Survey Data(https://arxiv.org/abs/2410.20695)
• Keywords: large language model
• Abstract: This exploratory pilot study investigated the potential of combining a domain-specific model, BERN2, with large language models (LLMs) to enhance automated disease phenotyping from research survey data. Motivated by the need for efficient and accurate methods to harmonize the growing volume of survey data with standardized disease ontologies, we employed BERN2, a biomedical named entity recognition and normalization model, to extract disease information from the ORIGINS birth cohort survey data. After rigorously evaluating BERN2's performance against a manually curated ground truth dataset, we integrated various LLMs using prompt engineering, Retrieval-Augmented Generation (RAG), and Instructional Fine-Tuning (IFT) to refine the model's outputs. BERN2 demonstrated high performance in extracting and normalizing disease mentions, and the integration of LLMs, particularly with Few Shot Inference and RAG orchestration, further improved accuracy. This approach, especially when incorporating structured examples, logical reasoning prompts, and detailed context, offers a promising avenue for developing tools to enable efficient cohort profiling and data harmonization across large, heterogeneous research datasets.

Title: DisasterQA: A Benchmark for Assessing the performance of LLMs in Disaster Response

• Authors: Rajat Rawat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20707
• Pdf URL: https://arxiv.org/pdf/2410.20707
• Copy Paste: [[2410.20707]] DisasterQA: A Benchmark for Assessing the performance of LLMs in Disaster Response(https://arxiv.org/abs/2410.20707)
• Keywords: large language model
• Abstract: Disasters can result in the deaths of many, making quick response times vital. Large Language Models (LLMs) have emerged as valuable in the field. LLMs can be used to process vast amounts of textual information quickly providing situational context during a disaster. However, the question remains whether LLMs should be used for advice and decision making in a disaster. To evaluate the capabilities of LLMs in disaster response knowledge, we introduce a benchmark: DisasterQA created from six online sources. The benchmark covers a wide range of disaster response topics. We evaluated five LLMs each with four different prompting methods on our benchmark, measuring both accuracy and confidence levels through Logprobs. The results indicate that LLMs require improvement on disaster response knowledge. We hope that this benchmark pushes forth further development of LLMs in disaster response, ultimately enabling these models to work alongside. emergency managers in disasters.

Title: Relation-based Counterfactual Data Augmentation and Contrastive Learning for Robustifying Natural Language Inference Models

• Authors: Heerin Yang, Sseung-won Hwang, Jungmin So
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20710
• Pdf URL: https://arxiv.org/pdf/2410.20710
• Copy Paste: [[2410.20710]] Relation-based Counterfactual Data Augmentation and Contrastive Learning for Robustifying Natural Language Inference Models(https://arxiv.org/abs/2410.20710)
• Keywords: robust
• Abstract: Although pre-trained language models show good performance on various natural language processing tasks, they often rely on non-causal features and patterns to determine the outcome. For natural language inference tasks, previous results have shown that even a model trained on a large number of data fails to perform well on counterfactually revised data, indicating that the model is not robustly learning the semantics of the classes. In this paper, we propose a method in which we use token-based and sentence-based augmentation methods to generate counterfactual sentence pairs that belong to each class, and apply contrastive learning to help the model learn the difference between sentence pairs of different classes with similar contexts. Evaluation results with counterfactually-revised dataset and general NLI datasets show that the proposed method can improve the performance and robustness of the NLI model.

Title: Face-MLLM: A Large Face Perception Model

• Authors: Haomiao Sun, Mingjie He, Tianheng Lian, Hu Han, Shiguang Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20717
• Pdf URL: https://arxiv.org/pdf/2410.20717
• Copy Paste: [[2410.20717]] Face-MLLM: A Large Face Perception Model(https://arxiv.org/abs/2410.20717)
• Keywords: large language model
• Abstract: Although multimodal large language models (MLLMs) have achieved promising results on a wide range of vision-language tasks, their ability to perceive and understand human faces is rarely explored. In this work, we comprehensively evaluate existing MLLMs on face perception tasks. The quantitative results reveal that existing MLLMs struggle to handle these tasks. The primary reason is the lack of image-text datasets that contain fine-grained descriptions of human faces. To tackle this problem, we design a practical pipeline for constructing datasets, upon which we further build a novel multimodal large face perception model, namely Face-MLLM. Specifically, we re-annotate LAION-Face dataset with more detailed face captions and facial attribute labels. Besides, we re-formulate traditional face datasets using the question-answer style, which is fit for MLLMs. Together with these enriched datasets, we develop a novel three-stage MLLM training method. In the first two stages, our model learns visual-text alignment and basic visual question answering capability, respectively. In the third stage, our model learns to handle multiple specialized face perception tasks. Experimental results show that our model surpasses previous MLLMs on five famous face perception tasks. Besides, on our newly introduced zero-shot facial attribute analysis task, our Face-MLLM also presents superior performance.

Title: Interpretable Image Classification with Adaptive Prototype-based Vision Transformers

• Authors: Chiyu Ma, Jon Donnelly, Wenjun Liu, Soroush Vosoughi, Cynthia Rudin, Chaofan Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20722
• Pdf URL: https://arxiv.org/pdf/2410.20722
• Copy Paste: [[2410.20722]] Interpretable Image Classification with Adaptive Prototype-based Vision Transformers(https://arxiv.org/abs/2410.20722)
• Keywords: transformer
• Abstract: We present ProtoViT, a method for interpretable image classification combining deep learning and case-based reasoning. This method classifies an image by comparing it to a set of learned prototypes, providing explanations of the form ``this looks like that.'' In our model, a prototype consists of \textit{parts}, which can deform over irregular geometries to create a better comparison between images. Unlike existing models that rely on Convolutional Neural Network (CNN) backbones and spatially rigid prototypes, our model integrates Vision Transformer (ViT) backbones into prototype based models, while offering spatially deformed prototypes that not only accommodate geometric variations of objects but also provide coherent and clear prototypical feature representations with an adaptive number of prototypical parts. Our experiments show that our model can generally achieve higher performance than the existing prototype based models. Our comprehensive analyses ensure that the prototypes are consistent and the interpretations are faithful.

Title: CompGS: Unleashing 2D Compositionality for Compositional Text-to-3D via Dynamically Optimizing 3D Gaussians

• Authors: Chongjian Ge, Chenfeng Xu, Yuanfeng Ji, Chensheng Peng, Masayoshi Tomizuka, Ping Luo, Mingyu Ding, Varun Jampani, Wei Zhan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20723
• Pdf URL: https://arxiv.org/pdf/2410.20723
• Copy Paste: [[2410.20723]] CompGS: Unleashing 2D Compositionality for Compositional Text-to-3D via Dynamically Optimizing 3D Gaussians(https://arxiv.org/abs/2410.20723)
• Keywords: generative
• Abstract: Recent breakthroughs in text-guided image generation have significantly advanced the field of 3D generation. While generating a single high-quality 3D object is now feasible, generating multiple objects with reasonable interactions within a 3D space, a.k.a. compositional 3D generation, presents substantial challenges. This paper introduces CompGS, a novel generative framework that employs 3D Gaussian Splatting (GS) for efficient, compositional text-to-3D content generation. To achieve this goal, two core designs are proposed: (1) 3D Gaussians Initialization with 2D compositionality: We transfer the well-established 2D compositionality to initialize the Gaussian parameters on an entity-by-entity basis, ensuring both consistent 3D priors for each entity and reasonable interactions among multiple entities; (2) Dynamic Optimization: We propose a dynamic strategy to optimize 3D Gaussians using Score Distillation Sampling (SDS) loss. CompGS first automatically decomposes 3D Gaussians into distinct entity parts, enabling optimization at both the entity and composition levels. Additionally, CompGS optimizes across objects of varying scales by dynamically adjusting the spatial parameters of each entity, enhancing the generation of fine-grained details, particularly in smaller entities. Qualitative comparisons and quantitative evaluations on T3Bench demonstrate the effectiveness of CompGS in generating compositional 3D objects with superior image quality and semantic alignment over existing methods. CompGS can also be easily extended to controllable 3D editing, facilitating scene generation. We hope CompGS will provide new insights to the compositional 3D generation. Project page: this https URL.

Title: Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation

• Authors: Mufei Li, Siqi Miao, Pan Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20724
• Pdf URL: https://arxiv.org/pdf/2410.20724
• Copy Paste: [[2410.20724]] Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation(https://arxiv.org/abs/2410.20724)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) demonstrate strong reasoning abilities but face limitations such as hallucinations and outdated knowledge. Knowledge Graph (KG)-based Retrieval-Augmented Generation (RAG) addresses these issues by grounding LLM outputs in structured external knowledge from KGs. However, current KG-based RAG frameworks still struggle to optimize the trade-off between retrieval effectiveness and efficiency in identifying a suitable amount of relevant graph information for the LLM to digest. We introduce SubgraphRAG, extending the KG-based RAG framework that retrieves subgraphs and leverages LLMs for reasoning and answer prediction. Our approach innovatively integrates a lightweight multilayer perceptron with a parallel triple-scoring mechanism for efficient and flexible subgraph retrieval while encoding directional structural distances to enhance retrieval effectiveness. The size of retrieved subgraphs can be flexibly adjusted to match the query's need and the downstream LLM's capabilities. This design strikes a balance between model complexity and reasoning power, enabling scalable and generalizable retrieval processes. Notably, based on our retrieved subgraphs, smaller LLMs like Llama3.1-8B-Instruct deliver competitive results with explainable reasoning, while larger models like GPT-4o achieve state-of-the-art accuracy compared with previous baselines -- all without fine-tuning. Extensive evaluations on the WebQSP and CWQ benchmarks highlight SubgraphRAG's strengths in efficiency, accuracy, and reliability by reducing hallucinations and improving response grounding.

Title: Faster WIND: Accelerating Iterative Best-of-$N$ Distillation for LLM Alignment

• Authors: Tong Yang, Jincheng Mei, Hanjun Dai, Zixin Wen, Shicong Cen, Dale Schuurmans, Yuejie Chi, Bo Dai
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.20727
• Pdf URL: https://arxiv.org/pdf/2410.20727
• Copy Paste: [[2410.20727]] Faster WIND: Accelerating Iterative Best-of-$N$ Distillation for LLM Alignment(https://arxiv.org/abs/2410.20727)
• Keywords: large language model
• Abstract: Recent advances in aligning large language models with human preferences have corroborated the growing importance of best-of-N distillation (BOND). However, the iterative BOND algorithm is prohibitively expensive in practice due to the sample and computation inefficiency. This paper addresses the problem by revealing a unified game-theoretic connection between iterative BOND and self-play alignment, which unifies seemingly disparate algorithmic paradigms. Based on the connection, we establish a novel framework, WIN rate Dominance (WIND), with a series of efficient algorithms for regularized win rate dominance optimization that approximates iterative BOND in the parameter space. We provides provable sample efficiency guarantee for one of the WIND variant with the square loss objective. The experimental results confirm that our algorithm not only accelerates the computation, but also achieves superior sample efficiency compared to existing methods.

Title: Comparative Simulation of Phishing Attacks on a Critical Information Infrastructure Organization: An Empirical Study

• Authors: Patsita Sirawongphatsara, Phisit Pornpongtechavanich, Nattapong Phanthuna, Therdpong Daengsi
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.20728
• Pdf URL: https://arxiv.org/pdf/2410.20728
• Copy Paste: [[2410.20728]] Comparative Simulation of Phishing Attacks on a Critical Information Infrastructure Organization: An Empirical Study(https://arxiv.org/abs/2410.20728)
• Keywords: security, attack
• Abstract: Nowadays, cybersecurity is crucial. Therefore, cybersecurity awareness should be a concern for businesses, particularly critical infrastructure organizations. The results of this study, using simulated phishing attacks, indicate that in the first attempt, workers of a Thai railway firm received a phony email purporting to inform recipients of a special deal from a reputable retailer of IT equipment. The findings showed that 10.9% of the 735 workers fell for the scam. This demonstrates a good level of awareness regarding cyber dangers. The workers who were duped by the initial attack received awareness training. Next, a second attempt was carried out. This time, the strategy was for the workers to change their passwords through an email notification from the fake IT staff. According to the findings, 1.4% of the workers fell victim to both attacks (different email content), and a further 8.0% of the workers who did not fall victim to the first attack were deceived. Furthermore, after the statistical analysis, it was confirmed that there is a difference in the relationship between the workers and the two phishing attack simulations using different content. As a result, this study has demonstrated that different types of content can affect levels of awareness.

Title: Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models

• Authors: Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, Haodong Wang, Zhengyang Wang, Wenju Xu, Jingfeng Yang, Qingyu Yin, Xian Li, Priyanka Nigam, Yi Xu, Kai Chen, Qiang Yang, Meng Jiang, Bing Yin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20745
• Pdf URL: https://arxiv.org/pdf/2410.20745
• Copy Paste: [[2410.20745]] Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models(https://arxiv.org/abs/2410.20745)
• Keywords: large language model
• Abstract: Online shopping is a complex multi-task, few-shot learning problem with a wide and evolving range of entities, relations, and tasks. However, existing models and benchmarks are commonly tailored to specific tasks, falling short of capturing the full complexity of online shopping. Large Language Models (LLMs), with their multi-task and few-shot learning abilities, have the potential to profoundly transform online shopping by alleviating task-specific engineering efforts and by providing users with interactive conversations. Despite the potential, LLMs face unique challenges in online shopping, such as domain-specific concepts, implicit knowledge, and heterogeneous user behaviors. Motivated by the potential and challenges, we propose Shopping MMLU, a diverse multi-task online shopping benchmark derived from real-world Amazon data. Shopping MMLU consists of 57 tasks covering 4 major shopping skills: concept understanding, knowledge reasoning, user behavior alignment, and multi-linguality, and can thus comprehensively evaluate the abilities of LLMs as general shop assistants. With Shopping MMLU, we benchmark over 20 existing LLMs and uncover valuable insights about practices and prospects of building versatile LLM-based shop assistants. Shopping MMLU can be publicly accessed at this https URL. In addition, with Shopping MMLU, we host a competition in KDD Cup 2024 with over 500 participating teams. The winning solutions and the associated workshop can be accessed at our website this https URL.

Title: ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agents

• Authors: Xinnong Zhang, Jiayu Lin, Libo Sun, Weihong Qi, Yihang Yang, Yue Chen, Hanjia Lyu, Xinyi Mou, Siming Chen, Jiebo Luo, Xuanjing Huang, Shiping Tang, Zhongyu Wei
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.20746
• Pdf URL: https://arxiv.org/pdf/2410.20746
• Copy Paste: [[2410.20746]] ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agents(https://arxiv.org/abs/2410.20746)
• Keywords: robust, large language model
• Abstract: The massive population election simulation aims to model the preferences of specific groups in particular election scenarios. It has garnered significant attention for its potential to forecast real-world social trends. Traditional agent-based modeling (ABM) methods are constrained by their ability to incorporate complex individual background information and provide interactive prediction results. In this paper, we introduce ElectionSim, an innovative election simulation framework based on large language models, designed to support accurate voter simulations and customized distributions, together with an interactive platform to dialogue with simulated voters. We present a million-level voter pool sampled from social media platforms to support accurate individual simulation. We also introduce PPE, a poll-based presidential election benchmark to assess the performance of our framework under the U.S. presidential election scenario. Through extensive experiments and analyses, we demonstrate the effectiveness and robustness of our framework in U.S. presidential election simulations.

Title: Matryoshka: Learning to Drive Black-Box LLMs with LLMs

• Authors: Changhao Li, Yuchen Zhuang, Rushi Qiang, Haotian Sun, Hanjun Dai, Chao Zhang, Bo Dai
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20749
• Pdf URL: https://arxiv.org/pdf/2410.20749
• Copy Paste: [[2410.20749]] Matryoshka: Learning to Drive Black-Box LLMs with LLMs(https://arxiv.org/abs/2410.20749)
• Keywords: generative, large language model
• Abstract: Despite the impressive generative abilities of black-box large language models (LLMs), their inherent opacity hinders further advancements in capabilities such as reasoning, planning, and personalization. Existing works aim to enhance LLM capabilities via domain-specific adaptation or in-context learning, which require additional training on accessible model parameters, an infeasible option for black-box LLMs. To address this challenge, we introduce Matryoshika, a lightweight white-box LLM controller that guides a large-scale black-box LLM generator by decomposing complex tasks into a series of intermediate outputs. Specifically, we consider the black-box LLM as an environment, with Matryoshika serving as a policy to provide intermediate guidance through prompts for driving the black-box LLM. Matryoshika is trained to pivot the outputs of the black-box LLM aligning with preferences during iterative interaction, which enables controllable multi-turn generation and self-improvement in optimizing intermediate guidance. Empirical evaluations on three diverse tasks demonstrate that Matryoshika effectively enhances the capabilities of black-box LLMs in complex, long-horizon tasks, including reasoning, planning, and personalization. By leveraging this pioneering controller-generator framework to mitigate dependence on model parameters, Matryoshika provides a transparent and practical solution for improving black-box LLMs through controllable multi-turn generation using white-box LLMs.

Title: Bidirectional Recurrence for Cardiac Motion Tracking with Gaussian Process Latent Coding

• Authors: Jiewen Yang, Yiqun Lin, Bin Pu, Xiaomeng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20752
• Pdf URL: https://arxiv.org/pdf/2410.20752
• Copy Paste: [[2410.20752]] Bidirectional Recurrence for Cardiac Motion Tracking with Gaussian Process Latent Coding(https://arxiv.org/abs/2410.20752)
• Keywords: robust
• Abstract: Quantitative analysis of cardiac motion is crucial for assessing cardiac function. This analysis typically uses imaging modalities such as MRI and Echocardiograms that capture detailed image sequences throughout the heartbeat cycle. Previous methods predominantly focused on the analysis of image pairs lacking consideration of the motion dynamics and spatial variability. Consequently, these methods often overlook the long-term relationships and regional motion characteristic of cardiac. To overcome these limitations, we introduce the \textbf{GPTrack}, a novel unsupervised framework crafted to fully explore the temporal and spatial dynamics of cardiac motion. The GPTrack enhances motion tracking by employing the sequential Gaussian Process in the latent space and encoding statistics by spatial information at each time stamp, which robustly promotes temporal consistency and spatial variability of cardiac dynamics. Also, we innovatively aggregate sequential information in a bidirectional recursive manner, mimicking the behavior of diffeomorphic registration to better capture consistent long-term relationships of motions across cardiac regions such as the ventricles and atria. Our GPTrack significantly improves the precision of motion tracking in both 3D and 4D medical images while maintaining computational efficiency. The code is available at: this https URL

Title: Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings

• Authors: Milad Khademi Nori, Il-Min Kim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20768
• Pdf URL: https://arxiv.org/pdf/2410.20768
• Copy Paste: [[2410.20768]] Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings(https://arxiv.org/abs/2410.20768)
• Keywords: generative
• Abstract: In class-incremental learning (class-IL), models must classify all previously seen classes at test time without task-IDs, leading to task confusion. Despite being a key challenge, task confusion lacks a theoretical understanding. We present a novel mathematical framework for class-IL and prove the Infeasibility Theorem, showing optimal class-IL is impossible with discriminative modeling due to task confusion. However, we establish the Feasibility Theorem, demonstrating that generative modeling can achieve optimal class-IL by overcoming task confusion. We then assess popular class-IL strategies, including regularization, bias-correction, replay, and generative classifier, using our framework. Our analysis suggests that adopting generative modeling, either for generative replay or direct classification (generative classifier), is essential for optimal class-IL.

Title: Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting

• Authors: Bong Gyun Kang, Dongjun Lee, HyunGi Kim, DoHyun Chung
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.20772
• Pdf URL: https://arxiv.org/pdf/2410.20772
• Copy Paste: [[2410.20772]] Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting(https://arxiv.org/abs/2410.20772)
• Keywords: transformer
• Abstract: Sequence modeling faces challenges in capturing long-range dependencies across diverse tasks. Recent linear and transformer-based forecasters have shown superior performance in time series forecasting. However, they are constrained by their inherent inability to effectively address long-range dependencies in time series data, primarily due to using fixed-size inputs for prediction. Furthermore, they typically sacrifice essential temporal correlation among consecutive training samples by shuffling them into mini-batches. To overcome these limitations, we introduce a fast and effective Spectral Attention mechanism, which preserves temporal correlations among samples and facilitates the handling of long-range information while maintaining the base model structure. Spectral Attention preserves long-period trends through a low-pass filter and facilitates gradient to flow between samples. Spectral Attention can be seamlessly integrated into most sequence models, allowing models with fixed-sized look-back windows to capture long-range dependencies over thousands of steps. Through extensive experiments on 11 real-world time series datasets using 7 recent forecasting models, we consistently demonstrate the efficacy of our Spectral Attention mechanism, achieving state-of-the-art results.

Title: Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation

• Authors: Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, Kyomin Jung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20774
• Pdf URL: https://arxiv.org/pdf/2410.20774
• Copy Paste: [[2410.20774]] Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation(https://arxiv.org/abs/2410.20774)
• Keywords: robust, large language model
• Abstract: In line with the principle of honesty, there has been a growing effort to train large language models (LLMs) to generate outputs containing epistemic markers. However, evaluation in the presence of epistemic markers has been largely overlooked, raising a critical question: Could the use of epistemic markers in LLM-generated outputs lead to unintended negative consequences? To address this, we present EMBER, a benchmark designed to assess the robustness of LLM-judges to epistemic markers in both single and pairwise evaluation settings. Our findings, based on evaluations using EMBER, reveal that all tested LLM-judges, including GPT-4o, show a notable lack of robustness in the presence of epistemic markers. Specifically, we observe a negative bias toward epistemic markers, with a stronger bias against markers expressing uncertainty. This suggests that LLM-judges are influenced by the presence of these markers and do not focus solely on the correctness of the content.

Title: KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation

• Authors: Rambod Azimi, Rishav Rishav, Marek Teichmann, Samira Ebrahimi Kahou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20777
• Pdf URL: https://arxiv.org/pdf/2410.20777
• Copy Paste: [[2410.20777]] KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation(https://arxiv.org/abs/2410.20777)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across various downstream tasks. However, the high computational and memory requirements of LLMs are a major bottleneck. To address this, parameter-efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) have been proposed to reduce computational costs while ensuring minimal loss in performance. Additionally, knowledge distillation (KD) has been a popular choice for obtaining compact student models from teacher models. In this work, we present KD-LoRA, a novel fine-tuning method that combines LoRA with KD. Our results demonstrate that KD-LoRA achieves performance comparable to full fine-tuning (FFT) and LoRA while significantly reducing resource requirements. Specifically, KD-LoRA retains 98% of LoRA's performance on the GLUE benchmark, while being 40% more compact. Additionally, KD-LoRA reduces GPU memory usage by 30% compared to LoRA, while decreasing inference time by 30% compared to both FFT and LoRA. We evaluate KD-LoRA across three encoder-only models: BERT, RoBERTa, and DeBERTaV3. Code is available at this https URL.

Title: Graph-based Uncertainty Metrics for Long-form Language Model Outputs

• Authors: Mingjian Jiang, Yangjun Ruan, Prasanna Sattigeri, Salim Roukos, Tatsunori Hashimoto
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20783
• Pdf URL: https://arxiv.org/pdf/2410.20783
• Copy Paste: [[2410.20783]] Graph-based Uncertainty Metrics for Long-form Language Model Outputs(https://arxiv.org/abs/2410.20783)
• Keywords: large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have significantly improved text generation capabilities, but these systems are still known to hallucinate, and granular uncertainty estimation for long-form LLM generations remains challenging. In this work, we propose Graph Uncertainty -- which represents the relationship between LLM generations and claims within them as a bipartite graph and estimates the claim-level uncertainty with a family of graph centrality metrics. Under this view, existing uncertainty estimation methods based on the concept of self-consistency can be viewed as using degree centrality as an uncertainty measure, and we show that more sophisticated alternatives such as closeness centrality provide consistent gains at claim-level uncertainty estimation. Moreover, we present uncertainty-aware decoding techniques that leverage both the graph structure and uncertainty estimates to improve the factuality of LLM generations by preserving only the most reliable claims. Compared to existing methods, our graph-based uncertainty metrics lead to an average of 6.8% relative gains on AUPRC across various long-form generation settings, and our end-to-end system provides consistent 2-4% gains in factuality over existing decoding techniques while significantly improving the informativeness of generated responses.

Title: SCULPT: Systematic Tuning of Long Prompts

• Authors: Shanu Kumar, Akhila Yesantarao Venkata, Shubhanshu Khandelwal, Bishal Santra, Parag Agrawal, Manish Gupta
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20788
• Pdf URL: https://arxiv.org/pdf/2410.20788
• Copy Paste: [[2410.20788]] SCULPT: Systematic Tuning of Long Prompts(https://arxiv.org/abs/2410.20788)
• Keywords: robust, large language model
• Abstract: As large language models become increasingly central to solving complex tasks, the challenge of optimizing long, unstructured prompts has become critical. Existing optimization techniques often struggle to effectively handle such prompts, leading to suboptimal performance. We introduce SCULPT (Systematic Tuning of Long Prompts), a novel framework that systematically refines long prompts by structuring them hierarchically and applying an iterative actor-critic mechanism. To enhance robustness and generalizability, SCULPT utilizes two complementary feedback mechanisms: Preliminary Assessment, which assesses the prompt's structure before execution, and Error Assessment, which diagnoses and addresses errors post-execution. By aggregating feedback from these mechanisms, SCULPT avoids overfitting and ensures consistent improvements in performance. Our experimental results demonstrate significant accuracy gains and enhanced robustness, particularly in handling erroneous and misaligned prompts. SCULPT consistently outperforms existing approaches, establishing itself as a scalable solution for optimizing long prompts across diverse and real-world tasks.

Title: SparseTem: Boosting the Efficiency of CNN-Based Video Encoders by Exploiting Temporal Continuity

• Authors: Kunyun Wang, Jieru Zhao, Shuo Yang, Wenchao Ding, Minyi Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20790
• Pdf URL: https://arxiv.org/pdf/2410.20790
• Copy Paste: [[2410.20790]] SparseTem: Boosting the Efficiency of CNN-Based Video Encoders by Exploiting Temporal Continuity(https://arxiv.org/abs/2410.20790)
• Keywords: extraction, transformer
• Abstract: Deep learning models have become pivotal in the field of video processing and is increasingly critical in practical applications such as autonomous driving and object detection. Although Vision Transformers (ViTs) have demonstrated their power, Convolutional Neural Networks (CNNs) remain a highly efficient and high-performance choice for feature extraction and encoding. However, the intensive computational demands of convolution operations hinder its broader adoption as a video encoder. Given the inherent temporal continuity in video frames, changes between consecutive frames are minimal, allowing for the skipping of redundant computations. This technique, which we term as Diff Computation, presents two primary challenges. First, Diff Computation requires to cache intermediate feature maps to ensure the correctness of non-linear computations, leading to significant memory consumption. Second, the imbalance of sparsity among layers, introduced by Diff Computation, incurs accuracy degradation. To address these issues, we propose a memory-efficient scheduling method to eliminate memory overhead and an online adjustment mechanism to minimize accuracy degradation. We integrate these techniques into our framework, SparseTem, to seamlessly support various CNN-based video encoders. SparseTem achieves speedup of 1.79x for EfficientDet and 4.72x for CRNN, with minimal accuracy drop and no additional memory overhead. Extensive experimental results demonstrate that SparseTem sets a new state-of-the-art by effectively utilizing temporal continuity to accelerate CNN-based video encoders.

Title: Deep Learning for Medical Text Processing: BERT Model Fine-Tuning and Comparative Study

• Authors: Jiacheng Hu, Yiru Cang, Guiran Liu, Meiqi Wang, Weijie He, Runyuan Bao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20792
• Pdf URL: https://arxiv.org/pdf/2410.20792
• Copy Paste: [[2410.20792]] Deep Learning for Medical Text Processing: BERT Model Fine-Tuning and Comparative Study(https://arxiv.org/abs/2410.20792)
• Keywords: transformer
• Abstract: This paper proposes a medical literature summary generation method based on the BERT model to address the challenges brought by the current explosion of medical information. By fine-tuning and optimizing the BERT model, we develop an efficient summary generation system that can quickly extract key information from medical literature and generate coherent, accurate summaries. In the experiment, we compared various models, including Seq-Seq, Attention, Transformer, and BERT, and demonstrated that the improved BERT model offers significant advantages in the Rouge and Recall metrics. Furthermore, the results of this study highlight the potential of knowledge distillation techniques to further enhance model performance. The system has demonstrated strong versatility and efficiency in practical applications, offering a reliable tool for the rapid screening and analysis of medical literature.

Title: Rephrasing natural text data with different languages and quality levels for Large Language Model pre-training

• Authors: Michael Pieler, Marco Bellagente, Hannah Teufel, Duy Phung, Nathan Cooper, Jonathan Tow, Paulo Rocha, Reshinth Adithyan, Zaid Alyafeai, Nikhil Pinnaparaju, Maksym Zhuravinskyi, Carlos Riquelme
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20796
• Pdf URL: https://arxiv.org/pdf/2410.20796
• Copy Paste: [[2410.20796]] Rephrasing natural text data with different languages and quality levels for Large Language Model pre-training(https://arxiv.org/abs/2410.20796)
• Keywords: large language model
• Abstract: Recently published work on rephrasing natural text data for pre-training LLMs has shown promising results when combining the original dataset with the synthetically rephrased data. We build upon previous work by replicating existing results on C4 and extending them with our optimized rephrasing pipeline to the English, German, Italian, and Spanish Oscar subsets of CulturaX. Our pipeline leads to increased performance on standard evaluation benchmarks in both the mono- and multilingual setup. In addition, we provide a detailed study of our pipeline, investigating the choice of the base dataset and LLM for the rephrasing, as well as the relationship between the model size and the performance after pre-training. By exploring data with different perceived quality levels, we show that gains decrease with higher quality. Furthermore, we find the difference in performance between model families to be bigger than between different model sizes. This highlights the necessity for detailed tests before choosing an LLM to rephrase large amounts of data. Moreover, we investigate the effect of pre-training with synthetic data on supervised fine-tuning. Here, we find increasing but inconclusive results that highly depend on the used benchmark. These results (again) highlight the need for better benchmarking setups. In summary, we show that rephrasing multilingual and low-quality data is a very promising direction to extend LLM pre-training data.

Title: Transformer-Based Tooth Alignment Prediction With Occlusion And Collision Constraints

• Authors: ZhenXing Dong, JiaZhou Chen, YangHui Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20806
• Pdf URL: https://arxiv.org/pdf/2410.20806
• Copy Paste: [[2410.20806]] Transformer-Based Tooth Alignment Prediction With Occlusion And Collision Constraints(https://arxiv.org/abs/2410.20806)
• Keywords: transformer
• Abstract: The planning of digital orthodontic treatment requires providing tooth alignment, which not only consumes a lot of time and labor to determine manually but also relays clinical experiences heavily. In this work, we proposed a lightweight tooth alignment neural network based on Swin-transformer. We first re-organized 3D point clouds based on virtual arch lines and converted them into order-sorted multi-channel textures, which improves the accuracy and efficiency simultaneously. We then designed two new occlusal loss functions that quantitatively evaluate the occlusal relationship between the upper and lower jaws. They are important clinical constraints, first introduced to the best of our knowledge, and lead to cutting-edge prediction accuracy. To train our network, we collected a large digital orthodontic dataset that has 591 clinical cases, including various complex clinical cases. This dataset will benefit the community after its release since there is no open dataset so far. Furthermore, we also proposed two new orthodontic dataset augmentation methods considering tooth spatial distribution and occlusion. We evaluated our method with this dataset and extensive experiments, including comparisons with STAT methods and ablation studies, and demonstrate the high prediction accuracy of our method.

Title: zGAN: An Outlier-focused Generative Adversarial Network For Realistic Synthetic Data Generation

• Authors: Azizjon Azimi, Bonu Boboeva, Ilyas Varshavskiy, Shuhrat Khalilbekov, Akhlitdin Nizamitdinov, Najima Noyoftova, Sergey Shulgin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20808
• Pdf URL: https://arxiv.org/pdf/2410.20808
• Copy Paste: [[2410.20808]] zGAN: An Outlier-focused Generative Adversarial Network For Realistic Synthetic Data Generation(https://arxiv.org/abs/2410.20808)
• Keywords: generative
• Abstract: The phenomenon of "black swans" has posed a fundamental challenge to performance of classical machine learning models. Perceived rise in frequency of outlier conditions, especially in post-pandemic environment, has necessitated exploration of synthetic data as a complement real data in model training. This article provides a general overview and experimental investigation of the zGAN model architecture developed for the purpose of generating synthetic tabular data with outlier characteristics. The model is put to test in binary classification environments and shows promising results on not only synthetic data generation, but also on uplift capabilities vis-à-vis model performance. A distinctive feature of zGAN is its enhanced correlation capability between features in the generated data, replicating correlations of features in real training data. Furthermore, crucial is the ability of zGAN to generate outliers based on covariance of real data or synthetically generated covariances. This approach to outlier generation enables modeling of complex economic events and augmentation of outliers for tasks such as training predictive models and detecting, processing or removing outliers. Experiments and comparative analyses as part of this study were conducted on both private (credit risk in financial services) and public datasets.

Title: Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation

• Authors: Jaechang Kim, Jinmin Goh, Inseok Hwang, Jaewoong Cho, Jungseul Ok
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20811
• Pdf URL: https://arxiv.org/pdf/2410.20811
• Copy Paste: [[2410.20811]] Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation(https://arxiv.org/abs/2410.20811)
• Keywords: explainability, large language model
• Abstract: Deep learning-based expert models have reached superhuman performance in decision-making domains such as chess and Go. However, it is under-explored to explain or comment on given decisions although it is important for human education and model explainability. The outputs of expert models are accurate, but yet difficult to interpret for humans. On the other hand, large language models (LLMs) produce fluent commentary but are prone to hallucinations due to their limited decision-making capabilities. To bridge this gap between expert models and LLMs, we focus on chess commentary as a representative case of explaining complex decision-making processes through language and address both the generation and evaluation of commentary. We introduce Concept-guided Chess Commentary generation (CCC) for producing commentary and GPT-based Chess Commentary Evaluation (GCC-Eval) for assessing it. CCC integrates the decision-making strengths of expert models with the linguistic fluency of LLMs through prioritized, concept-based explanations. GCC-Eval leverages expert knowledge to evaluate chess commentary based on informativeness and linguistic quality. Experimental results, validated by both human judges and GCC-Eval, demonstrate that CCC generates commentary that is accurate, informative, and fluent.

Title: NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates

• Authors: Hexuan Deng, Wenxiang Jiao, Xuebo Liu, Min Zhang, Zhaopeng Tu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20814
• Pdf URL: https://arxiv.org/pdf/2410.20814
• Copy Paste: [[2410.20814]] NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates(https://arxiv.org/abs/2410.20814)
• Keywords: large language model
• Abstract: Despite their remarkable abilities in various tasks, large language models (LLMs) still struggle with real-time information (e.g., new facts and terms) due to the knowledge cutoff in their development process. However, existing benchmarks focus on outdated content and limited fields, facing difficulties in real-time updating and leaving new terms unexplored. To address this problem, we propose an adaptive benchmark, NewTerm, for real-time evaluation of new terms. We design a highly automated construction method to ensure high-quality benchmark construction with minimal human effort, allowing flexible updates for real-time information. Empirical results on various LLMs demonstrate over 20% performance reduction caused by new terms. Additionally, while updates to the knowledge cutoff of LLMs can cover some of the new terms, they are unable to generalize to more distant new terms. We also analyze which types of terms are more challenging and why LLMs struggle with new terms, paving the way for future research. Finally, we construct NewTerm 2022 and 2023 to evaluate the new terms updated each year and will continue updating annually. The benchmark and codes can be found at this https URL.

Title: Novel Object Synthesis via Adaptive Text-Image Harmony

• Authors: Zeren Xiong, Zedong Zhang, Zikun Chen, Shuo Chen, Xiang Li, Gan Sun, Jian Yang, Jun Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20823
• Pdf URL: https://arxiv.org/pdf/2410.20823
• Copy Paste: [[2410.20823]] Novel Object Synthesis via Adaptive Text-Image Harmony(https://arxiv.org/abs/2410.20823)
• Keywords: diffusion
• Abstract: In this paper, we study an object synthesis task that combines an object text with an object image to create a new object image. However, most diffusion models struggle with this task, \textit{i.e.}, often generating an object that predominantly reflects either the text or the image due to an imbalance between their inputs. To address this issue, we propose a simple yet effective method called Adaptive Text-Image Harmony (ATIH) to generate novel and surprising objects. First, we introduce a scale factor and an injection step to balance text and image features in cross-attention and to preserve image information in self-attention during the text-image inversion diffusion process, respectively. Second, to better integrate object text and image, we design a balanced loss function with a noise parameter, ensuring both optimal editability and fidelity of the object image. Third, to adaptively adjust these parameters, we present a novel similarity score function that not only maximizes the similarities between the generated object image and the input text/image but also balances these similarities to harmonize text and image integration. Extensive experiments demonstrate the effectiveness of our approach, showcasing remarkable object creations such as colobus-glass jar. Project page: this https URL.

Title: FreqMark: Invisible Image Watermarking via Frequency Based Optimization in Latent Space

• Authors: Yiyang Guo, Ruizhe Li, Mude Hui, Hanzhong Guo, Chen Zhang, Chuangjian Cai, Le Wan, Shangfei Wang
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20824
• Pdf URL: https://arxiv.org/pdf/2410.20824
• Copy Paste: [[2410.20824]] FreqMark: Invisible Image Watermarking via Frequency Based Optimization in Latent Space(https://arxiv.org/abs/2410.20824)
• Keywords: protect, attack, robust, watermark
• Abstract: Invisible watermarking is essential for safeguarding digital content, enabling copyright protection and content authentication. However, existing watermarking methods fall short in robustness against regeneration attacks. In this paper, we propose a novel method called FreqMark that involves unconstrained optimization of the image latent frequency space obtained after VAE encoding. Specifically, FreqMark embeds the watermark by optimizing the latent frequency space of the images and then extracts the watermark through a pre-trained image encoder. This optimization allows a flexible trade-off between image quality with watermark robustness and effectively resists regeneration attacks. Experimental results demonstrate that FreqMark offers significant advantages in image quality and robustness, permits flexible selection of the encoding bit number, and achieves a bit accuracy exceeding 90% when encoding a 48-bit hidden message under various attack scenarios.

Title: ADLM -- stega: A Universal Adaptive Token Selection Algorithm for Improving Steganographic Text Quality via Information Entropy

• Authors: Zezheng Qin, Congcong Sun, Taiyi He, Yuke He, Azizol Abdullah, Normalia Samian, Nuur Alifah Roslan
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20825
• Pdf URL: https://arxiv.org/pdf/2410.20825
• Copy Paste: [[2410.20825]] ADLM -- stega: A Universal Adaptive Token Selection Algorithm for Improving Steganographic Text Quality via Information Entropy(https://arxiv.org/abs/2410.20825)
• Keywords: security, privacy, protect, generative
• Abstract: In the context of widespread global information sharing, information security and privacy protection have become focal points. Steganographic systems enhance information security by embedding confidential information into public carriers; however, existing generative text steganography methods face challenges in handling the long-tail distribution of candidate word pools, which impacts the imperceptibility of steganographic information. This paper proposes a quality control theory for steganographic text generation based on information entropy constraints, exploring the relationship between the imperceptibility of steganographic texts and information entropy. By controlling the information entropy of the candidate word pool within a specific range, we optimize the imperceptibility of the steganographic text. We establish upper and lower bounds for information entropy and introduce an adaptive truncation method to balance semantic coherence and lexical diversity. Experimental results demonstrate that reasonably controlling the candidate pool size and information entropy thresholds significantly enhances the quality and detection resistance of steganographic texts, showcasing broad application potential in the field of natural language processing.

Title: LLMs are Biased Evaluators But Not Biased for Retrieval Augmented Generation

• Authors: Yen-Shan Chen, Jing Jin, Peng-Ting Kuo, Chao-Wei Huang, Yun-Nung Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20833
• Pdf URL: https://arxiv.org/pdf/2410.20833
• Copy Paste: [[2410.20833]] LLMs are Biased Evaluators But Not Biased for Retrieval Augmented Generation(https://arxiv.org/abs/2410.20833)
• Keywords: robust, extraction, large language model
• Abstract: Recent studies have demonstrated that large language models (LLMs) exhibit significant biases in evaluation tasks, particularly in preferentially rating and favoring self-generated content. However, the extent to which this bias manifests in fact-oriented tasks, especially within retrieval-augmented generation (RAG) frameworks-where keyword extraction and factual accuracy take precedence over stylistic elements-remains unclear. Our study addresses this knowledge gap by simulating two critical phases of the RAG framework. In the first phase, we access the suitability of human-authored versus model-generated passages, emulating the pointwise reranking process. The second phase involves conducting pairwise reading comprehension tests to simulate the generation process. Contrary to previous findings indicating a self-preference in rating tasks, our results reveal no significant self-preference effect in RAG frameworks. Instead, we observe that factual accuracy significantly influences LLMs' output, even in the absence of prior knowledge. Our research contributes to the ongoing discourse on LLM biases and their implications for RAG-based system, offering insights that may inform the development of more robust and unbiased LLM systems.

Title: A Simple Yet Effective Corpus Construction Framework for Indonesian Grammatical Error Correction

• Authors: Nankai Lin, Meiyu Zeng, Wentao Huang, Shengyi Jiang, Lixian Xiao, Aimin Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20838
• Pdf URL: https://arxiv.org/pdf/2410.20838
• Copy Paste: [[2410.20838]] A Simple Yet Effective Corpus Construction Framework for Indonesian Grammatical Error Correction(https://arxiv.org/abs/2410.20838)
• Keywords: large language model
• Abstract: Currently, the majority of research in grammatical error correction (GEC) is concentrated on universal languages, such as English and Chinese. Many low-resource languages lack accessible evaluation corpora. How to efficiently construct high-quality evaluation corpora for GEC in low-resource languages has become a significant challenge. To fill these gaps, in this paper, we present a framework for constructing GEC corpora. Specifically, we focus on Indonesian as our research language and construct an evaluation corpus for Indonesian GEC using the proposed framework, addressing the limitations of existing evaluation corpora in Indonesian. Furthermore, we investigate the feasibility of utilizing existing large language models (LLMs), such as GPT-3.5-Turbo and GPT-4, to streamline corpus annotation efforts in GEC tasks. The results demonstrate significant potential for enhancing the performance of LLMs in low-resource language settings. Our code and corpus can be obtained from this https URL.

Title: ByteNet: Rethinking Multimedia File Fragment Classification through Visual Perspectives

• Authors: Wenyang Liu, Kejun Wu, Tianyi Liu, Yi Wang, Kim-Hui Yap, Lap-Pui Chau
• Subjects: cs.CV, cs.CR, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.20855
• Pdf URL: https://arxiv.org/pdf/2410.20855
• Copy Paste: [[2410.20855]] ByteNet: Rethinking Multimedia File Fragment Classification through Visual Perspectives(https://arxiv.org/abs/2410.20855)
• Keywords: robust, extraction, transformer
• Abstract: Multimedia file fragment classification (MFFC) aims to identify file fragment types, e.g., image/video, audio, and text without system metadata. It is of vital importance in multimedia storage and communication. Existing MFFC methods typically treat fragments as 1D byte sequences and emphasize the relations between separate bytes (interbytes) for classification. However, the more informative relations inside bytes (intrabytes) are overlooked and seldom investigated. By looking inside bytes, the bit-level details of file fragments can be accessed, enabling a more accurate classification. Motivated by this, we first propose Byte2Image, a novel visual representation model that incorporates previously overlooked intrabyte information into file fragments and reinterprets these fragments as 2D grayscale images. This model involves a sliding byte window to reveal the intrabyte information and a rowwise stacking of intrabyte ngrams for embedding fragments into a 2D space. Thus, complex interbyte and intrabyte correlations can be mined simultaneously using powerful vision networks. Additionally, we propose an end-to-end dual-branch network ByteNet to enhance robust correlation mining and feature representation. ByteNet makes full use of the raw 1D byte sequence and the converted 2D image through a shallow byte branch feature extraction (BBFE) and a deep image branch feature extraction (IBFE) network. In particular, the BBFE, composed of a single fully-connected layer, adaptively recognizes the co-occurrence of several some specific bytes within the raw byte sequence, while the IBFE, built on a vision Transformer, effectively mines the complex interbyte and intrabyte correlations from the converted image. Experiments on the two representative benchmarks, including 14 cases, validate that our proposed method outperforms state-of-the-art approaches on different cases by up to 12.2%.

Title: Fakeium: A Dynamic Execution Environment for JavaScript Program Analysis

• Authors: José Miguel Moreno, Narseo Vallina-Rodriguez, Juan Tapiador
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.20862
• Pdf URL: https://arxiv.org/pdf/2410.20862
• Copy Paste: [[2410.20862]] Fakeium: A Dynamic Execution Environment for JavaScript Program Analysis(https://arxiv.org/abs/2410.20862)
• Keywords: security, attack, steal
• Abstract: The JavaScript programming language, which began as a simple scripting language for the Web, has become ubiquitous, spanning desktop, mobile, and server applications. This increase in usage has made JavaScript an attractive target for nefarious actors, resulting in the proliferation of malicious browser extensions that steal user information and supply chain attacks that target the official this http URL package registry. To combat these threats, researchers have developed specialized tools and frameworks for analyzing the behavior of JavaScript programs to detect malicious patterns. Static analysis tools typically struggle with the highly dynamic nature of the language and fail to process obfuscated sources, while dynamic analysis pipelines take several minutes to run and require more resources per program, making them unfeasible for large-scale analyses. In this paper, we present Fakeium, a novel, open source, and lightweight execution environment designed for efficient, large-scale dynamic analysis of JavaScript programs. Built on top of the popular V8 engine, Fakeium complements traditional static analysis by providing additional API calls and string literals that would otherwise go unnoticed without the need for resource-intensive instrumented browsers or synthetic user input. Besides its negligible execution overhead, our tool is highly customizable and supports hooks for advanced analysis scenarios such as network traffic emulation. Fakeium's flexibility and ability to detect hidden API calls, especially in obfuscated sources, highlights its potential as a valuable tool for security analysts to detect malicious behavior.

Title: Reward Modeling with Weak Supervision for Language Models

• Authors: Ben Hauptvogel, Malte Ostendorff, Georg Rehm, Sebastian Möller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20869
• Pdf URL: https://arxiv.org/pdf/2410.20869
• Copy Paste: [[2410.20869]] Reward Modeling with Weak Supervision for Language Models(https://arxiv.org/abs/2410.20869)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have led to their increased application across various tasks, with reinforcement learning from human feedback (RLHF) being a crucial part of their training to align responses with user intentions. In the RLHF process, a reward model is trained using responses preferences determined by human labelers or AI systems, which then refines the LLM through reinforcement learning. This work introduces weak supervision as a strategy to extend RLHF datasets and enhance reward model performance. Weak supervision employs noisy or imprecise data labeling, reducing reliance on expensive manually labeled data. By analyzing RLHF datasets to identify heuristics that correlate with response preference, we wrote simple labeling functions and then calibrated a label model to weakly annotate unlabeled data. Our evaluation show that while weak supervision significantly benefits smaller datasets by improving reward model performance, its effectiveness decreases with larger, originally labeled datasets. Additionally, using an LLM to generate and then weakly label responses offers a promising method for extending preference data.

Title: AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline

• Authors: Dongkyu Kim, Byoungwook Kim, Donggeon Han, Matouš Eibich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20878
• Pdf URL: https://arxiv.org/pdf/2410.20878
• Copy Paste: [[2410.20878]] AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline(https://arxiv.org/abs/2410.20878)
• Keywords: large language model
• Abstract: Using LLMs (Large Language Models) in conjunction with external documents has made RAG (Retrieval-Augmented Generation) an essential technology. Numerous techniques and modules for RAG are being researched, but their performance can vary across different datasets. Finding RAG modules that perform well on specific datasets is challenging. In this paper, we propose the AutoRAG framework, which automatically identifies suitable RAG modules for a given dataset. AutoRAG explores and approximates the optimal combination of RAG modules for the dataset. Additionally, we share the results of optimizing a dataset using AutoRAG. All experimental results and data are publicly available and can be accessed through our GitHub repository this https URL .

Title: CODES: Benchmarking Coupled ODE Surrogates

• Authors: Robin Janssen, Immanuel Sulzer, Tobias Buck
• Subjects: cs.LG, astro-ph.IM, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2410.20886
• Pdf URL: https://arxiv.org/pdf/2410.20886
• Copy Paste: [[2410.20886]] CODES: Benchmarking Coupled ODE Surrogates(https://arxiv.org/abs/2410.20886)
• Keywords: fair
• Abstract: We introduce CODES, a benchmark for comprehensive evaluation of surrogate architectures for coupled ODE systems. Besides standard metrics like mean squared error (MSE) and inference time, CODES provides insights into surrogate behaviour across multiple dimensions like interpolation, extrapolation, sparse data, uncertainty quantification and gradient correlation. The benchmark emphasizes usability through features such as integrated parallel training, a web-based configuration generator, and pre-implemented baseline models and datasets. Extensive documentation ensures sustainability and provides the foundation for collaborative improvement. By offering a fair and multi-faceted comparison, CODES helps researchers select the most suitable surrogate for their specific dataset and application while deepening our understanding of surrogate learning behaviour.

Title: Generative Example-Based Explanations: Bridging the Gap between Generative Modeling and Explainability

• Authors: Philipp Vaeth, Alexander M. Fruehwald, Benjamin Paassen, Magda Gregorova
• Subjects: cs.LG, cs.CY, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.20890
• Pdf URL: https://arxiv.org/pdf/2410.20890
• Copy Paste: [[2410.20890]] Generative Example-Based Explanations: Bridging the Gap between Generative Modeling and Explainability(https://arxiv.org/abs/2410.20890)
• Keywords: explainability, generative
• Abstract: Recently, several methods have leveraged deep generative modeling to produce example-based explanations of decision algorithms for high-dimensional input data. Despite promising results, a disconnect exists between these methods and the classical explainability literature, which focuses on lower-dimensional data with semantically meaningful features. This conceptual and communication gap leads to misunderstandings and misalignments in goals and expectations. In this paper, we bridge this gap by proposing a novel probabilistic framework for local example-based explanations. Our framework integrates the critical characteristics of classical local explanation desiderata while being amenable to high-dimensional data and their modeling through deep generative models. Our aim is to facilitate communication, foster rigor and transparency, and improve the quality of peer discussion and research progress.

Title: Evaluating the Robustness of LiDAR Point Cloud Tracking Against Adversarial Attack

• Authors: Shengjing Tian, Yinan Han, Xiantong Zhao, Bin Liu, Xiuping Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20893
• Pdf URL: https://arxiv.org/pdf/2410.20893
• Copy Paste: [[2410.20893]] Evaluating the Robustness of LiDAR Point Cloud Tracking Against Adversarial Attack(https://arxiv.org/abs/2410.20893)
• Keywords: attack, robust, transformer
• Abstract: In this study, we delve into the robustness of neural network-based LiDAR point cloud tracking models under adversarial attacks, a critical aspect often overlooked in favor of performance enhancement. These models, despite incorporating advanced architectures like Transformer or Bird's Eye View (BEV), tend to neglect robustness in the face of challenges such as adversarial attacks, domain shifts, or data corruption. We instead focus on the robustness of the tracking models under the threat of adversarial attacks. We begin by establishing a unified framework for conducting adversarial attacks within the context of 3D object tracking, which allows us to thoroughly investigate both white-box and black-box attack strategies. For white-box attacks, we tailor specific loss functions to accommodate various tracking paradigms and extend existing methods such as FGSM, C\&W, and PGD to the point cloud domain. In addressing black-box attack scenarios, we introduce a novel transfer-based approach, the Target-aware Perturbation Generation (TAPG) algorithm, with the dual objectives of achieving high attack performance and maintaining low perceptibility. This method employs a heuristic strategy to enforce sparse attack constraints and utilizes random sub-vector factorization to bolster transferability. Our experimental findings reveal a significant vulnerability in advanced tracking methods when subjected to both black-box and white-box attacks, underscoring the necessity for incorporating robustness against adversarial attacks into the design of LiDAR point cloud tracking models. Notably, compared to existing methods, the TAPG also strikes an optimal balance between the effectiveness of the attack and the concealment of the perturbations.

Title: Diff-Instruct*: Towards Human-Preferred One-step Text-to-image Generative Models

• Authors: Weijian Luo, Colin Zhang, Debing Zhang, Zhengyang Geng
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.20898
• Pdf URL: https://arxiv.org/pdf/2410.20898
• Copy Paste: [[2410.20898]] Diff-Instruct*: Towards Human-Preferred One-step Text-to-image Generative Models(https://arxiv.org/abs/2410.20898)
• Keywords: diffusion, data-free, generative
• Abstract: In this paper, we introduce the Diff-Instruct*(DI*), a data-free approach for building one-step text-to-image generative models that align with human preference while maintaining the ability to generate highly realistic images. We frame human preference alignment as online reinforcement learning using human feedback (RLHF), where the goal is to maximize the reward function while regularizing the generator distribution to remain close to a reference diffusion process. Unlike traditional RLHF approaches, which rely on the KL divergence for regularization, we introduce a novel score-based divergence regularization, which leads to significantly better performances. Although the direct calculation of this divergence remains intractable, we demonstrate that we can efficiently compute its \emph{gradient} by deriving an equivalent yet tractable loss function. Remarkably, with Stable Diffusion V1.5 as the reference diffusion model, DI* outperforms \emph{all} previously leading models by a large margin. When using the 0.6B PixelArt-$\alpha$ model as the reference diffusion, DI* achieves a new record Aesthetic Score of 6.30 and an Image Reward of 1.31 with only a single generation step, almost doubling the scores of the rest of the models with similar sizes. It also achieves an HPSv2 score of 28.70, establishing a new state-of-the-art benchmark. We also observe that DI* can improve the layout and enrich the colors of generated images.

Title: Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks

• Authors: Dario Pasquini, Evgenios M. Kornaropoulos, Giuseppe Ateniese
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20911
• Pdf URL: https://arxiv.org/pdf/2410.20911
• Copy Paste: [[2410.20911]] Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks(https://arxiv.org/abs/2410.20911)
• Keywords: defense, attack, large language model
• Abstract: Large language models (LLMs) are increasingly being harnessed to automate cyberattacks, making sophisticated exploits more accessible and scalable. In response, we propose a new defense strategy tailored to counter LLM-driven cyberattacks. We introduce Mantis, a defensive framework that exploits LLMs' susceptibility to adversarial inputs to undermine malicious operations. Upon detecting an automated cyberattack, Mantis plants carefully crafted inputs into system responses, leading the attacker's LLM to disrupt their own operations (passive defense) or even compromise the attacker's machine (active defense). By deploying purposefully vulnerable decoy services to attract the attacker and using dynamic prompt injections for the attacker's LLM, Mantis can autonomously hack back the attacker. In our experiments, Mantis consistently achieved over 95% effectiveness against automated LLM-driven attacks. To foster further research and collaboration, Mantis is available as an open-source tool: this https URL

Title: Constrained Optimal Fuel Consumption of HEV:Considering the Observational Perturbation

• Authors: Shuchang Yan, Haoran Sun
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.20913
• Pdf URL: https://arxiv.org/pdf/2410.20913
• Copy Paste: [[2410.20913]] Constrained Optimal Fuel Consumption of HEV:Considering the Observational Perturbation(https://arxiv.org/abs/2410.20913)
• Keywords: robust
• Abstract: We assume accurate observation of battery state of charge (SOC) and precise speed curves when addressing the constrained optimal fuel consumption (COFC) problem via constrained reinforcement learning (CRL). However, in practice, SOC measurements are often distorted by noise or confidentiality protocols, and actual reference speeds may deviate from expectations. We aim to minimize fuel consumption while maintaining SOC balance under observational perturbations in SOC and speed. This work first worldwide uses seven training approaches to solve the COFC problem under five types of perturbations, including one based on a uniform distribution, one designed to maximize rewards, one aimed at maximizing costs, and one along with its improved version that seeks to decrease reward on Toyota Hybrid Systems (THS) under New European Driving Cycle (NEDC) condition. The result verifies that the six can successfully solve the COFC problem under observational perturbations, and we further compare the robustness and safety of these training approaches and analyze their impact on optimal fuel consumption.

Title: Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning

• Authors: Aosong Feng, Rex Ying, Leandros Tassiulas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20926
• Pdf URL: https://arxiv.org/pdf/2410.20926
• Copy Paste: [[2410.20926]] Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning(https://arxiv.org/abs/2410.20926)
• Keywords: transformer
• Abstract: As the demand for processing extended textual data grows, the ability to handle long-range dependencies and maintain computational efficiency is more critical than ever. One of the key issues for long-sequence modeling using attention-based model is the mismatch between the limited-range modeling power of full attention and the long-range token dependency in the input sequence. In this work, we propose to scale up the attention receptive field by tensorizing long input sequences into compact tensor representations followed by attention on each transformed dimension. The resulting Tensorized Attention can be adopted as efficient transformer backbones to extend input context length with improved memory and time efficiency. We show that the proposed attention tensorization encodes token dependencies as a multi-hop attention process, and is equivalent to Kronecker decomposition of full attention. Extensive experiments show that tensorized attention can be used to adapt pretrained LLMs with improved efficiency. Notably, Llama-8B with tensorization is trained under 32,768 context length and can steadily extrapolate to 128k length during inference with $11\times$ speedup, compared to full attention with FlashAttention-2.

Title: Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency

• Authors: Zenan Li, Yifan Wu, Zhaoyu Li, Xinming Wei, Xian Zhang, Fan Yang, Xiaoxing Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20936
• Pdf URL: https://arxiv.org/pdf/2410.20936
• Copy Paste: [[2410.20936]] Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency(https://arxiv.org/abs/2410.20936)
• Keywords: large language model
• Abstract: Autoformalization, the task of automatically translating natural language descriptions into a formal language, poses a significant challenge across various domains, especially in mathematics. Recent advancements in large language models (LLMs) have unveiled their promising capabilities to formalize even competition-level math problems. However, we observe a considerable discrepancy between pass@1 and pass@k accuracies in LLM-generated formalizations. To address this gap, we introduce a novel framework that scores and selects the best result from k autoformalization candidates based on two complementary self-consistency methods: symbolic equivalence and semantic consistency. Elaborately, symbolic equivalence identifies the logical homogeneity among autoformalization candidates using automated theorem provers, and semantic consistency evaluates the preservation of the original meaning by informalizing the candidates and computing the similarity between the embeddings of the original and informalized texts. Our extensive experiments on the MATH and miniF2F datasets demonstrate that our approach significantly enhances autoformalization accuracy, achieving up to 0.22-1.35x relative improvements across various LLMs and baseline methods.

Title: Attacking Misinformation Detection Using Adversarial Examples Generated by Language Models

• Authors: Piotr Przybyła
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.20940
• Pdf URL: https://arxiv.org/pdf/2410.20940
• Copy Paste: [[2410.20940]] Attacking Misinformation Detection Using Adversarial Examples Generated by Language Models(https://arxiv.org/abs/2410.20940)
• Keywords: attack, robust, large language model
• Abstract: We investigate the challenge of generating adversarial examples to test the robustness of text classification algorithms detecting low-credibility content, including propaganda, false claims, rumours and hyperpartisan news. We focus on simulation of content moderation by setting realistic limits on the number of queries an attacker is allowed to attempt. Within our solution (TREPAT), initial rephrasings are generated by large language models with prompts inspired by meaning-preserving NLP tasks, e.g. text simplification and style transfer. Subsequently, these modifications are decomposed into small changes, applied through beam search procedure until the victim classifier changes its decision. The evaluation confirms the superiority of our approach in the constrained scenario, especially in case of long input text (news articles), where exhaustive search is not feasible.

Title: Instruction-Tuned LLMs Succeed in Document-Level MT Without Fine-Tuning -- But BLEU Turns a Blind Eye

• Authors: Yirong Sun, Dawei Zhu, Yanjun Chen, Erjia Xiao, Xinghao Chen, Xiaoyu Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20941
• Pdf URL: https://arxiv.org/pdf/2410.20941
• Copy Paste: [[2410.20941]] Instruction-Tuned LLMs Succeed in Document-Level MT Without Fine-Tuning -- But BLEU Turns a Blind Eye(https://arxiv.org/abs/2410.20941)
• Keywords: large language model
• Abstract: Large language models (LLMs) have excelled in various NLP tasks, including machine translation (MT), yet most studies focus on sentence-level translation. This work investigates the inherent capability of instruction-tuned LLMs for document-level translation (docMT). Unlike prior approaches that require specialized techniques, we evaluate LLMs by directly prompting them to translate entire documents in a single pass. Our results show that this method improves translation quality compared to translating sentences separately, even without document-level fine-tuning. However, this advantage is not reflected in BLEU scores, which often favor sentence-based translations. We propose using the LLM-as-a-judge paradigm for evaluation, where GPT-4 is used to assess document coherence, accuracy, and fluency in a more nuanced way than n-gram-based metrics. Overall, our work demonstrates that instruction-tuned LLMs can effectively leverage document context for translation. However, we caution against using BLEU scores for evaluating docMT, as they often provide misleading outcomes, failing to capture the quality of document-level translation. Code and data are available at this https URL

Title: IndraEye: Infrared Electro-Optical UAV-based Perception Dataset for Robust Downstream Tasks

• Authors: Manjunath D, Prajwal Gurunath, Sumanth Udupa, Aditya Gandhamal, Shrikar Madhu, Aniruddh Sikdar, Suresh Sundaram
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20953
• Pdf URL: https://arxiv.org/pdf/2410.20953
• Copy Paste: [[2410.20953]] IndraEye: Infrared Electro-Optical UAV-based Perception Dataset for Robust Downstream Tasks(https://arxiv.org/abs/2410.20953)
• Keywords: robust, segmentation
• Abstract: Deep neural networks (DNNs) have shown exceptional performance when trained on well-illuminated images captured by Electro-Optical (EO) cameras, which provide rich texture details. However, in critical applications like aerial perception, it is essential for DNNs to maintain consistent reliability across all conditions, including low-light scenarios where EO cameras often struggle to capture sufficient detail. Additionally, UAV-based aerial object detection faces significant challenges due to scale variability from varying altitudes and slant angles, adding another layer of complexity. Existing methods typically address only illumination changes or style variations as domain shifts, but in aerial perception, correlation shifts also impact DNN performance. In this paper, we introduce the IndraEye dataset, a multi-sensor (EO-IR) dataset designed for various tasks. It includes 5,612 images with 145,666 instances, encompassing multiple viewing angles, altitudes, seven backgrounds, and different times of the day across the Indian subcontinent. The dataset opens up several research opportunities, such as multimodal learning, domain adaptation for object detection and segmentation, and exploration of sensor-specific strengths and weaknesses. IndraEye aims to advance the field by supporting the development of more robust and accurate aerial perception systems, particularly in challenging conditions. IndraEye dataset is benchmarked with object detection and semantic segmentation tasks. Dataset and source codes are available at this https URL.

Title: CovFUZZ: Coverage-based fuzzer for 4G&5G protocols

• Authors: Ilja Siroš, Dave Singelée, Bart Preneel
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.20958
• Pdf URL: https://arxiv.org/pdf/2410.20958
• Copy Paste: [[2410.20958]] CovFUZZ: Coverage-based fuzzer for 4G&5G protocols(https://arxiv.org/abs/2410.20958)
• Keywords: security
• Abstract: 4G and 5G represent the current cellular communication standards utilized daily by billions of users for various applications. Consequently, ensuring the security of 4G and 5G network implementations is critically important. This paper introduces an automated fuzzing framework designed to test the security of 4G and 5G attach procedure implementations. Our framework provides a comprehensive solution for uplink and downlink fuzzing in 4G, as well as downlink fuzzing in 5G, while supporting fuzzing on all layers except the physical layer. To guide the fuzzing process, we introduce a novel algorithm that assigns probabilities to packet fields and adjusts these probabilities based on coverage information from the device-under-test (DUT). For cases where coverage information from the DUT is unavailable, we propose a novel methodology to estimate it. When evaluating our framework, we first run the random fuzzing experiments, where the mutation probabilities are fixed throughout the fuzzing, and give an insight into how those probabilities should be chosen to optimize the Random fuzzer to achieve the best coverage. Next, we evaluate the efficiency of the proposed coverage-based algorithms by fuzzing open-source 4G stack (srsRAN) instances and show that the fuzzer guided by our algorithm outperforms the optimized Random fuzzer in terms of DUT's code coverage. In addition, we run fuzzing tests on 12 commercial off-the-shelf (COTS) devices. In total, we discovered vulnerabilities in 10 COTS devices and all of the srsRAN 4G instances.

Title: DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learning

• Authors: Xun Guo, Shan Zhang, Yongxin He, Ting Zhang, Wanquan Feng, Haibin Huang, Chongyang Ma
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20964
• Pdf URL: https://arxiv.org/pdf/2410.20964
• Copy Paste: [[2410.20964]] DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learning(https://arxiv.org/abs/2410.20964)
• Keywords: large language model
• Abstract: Current techniques for detecting AI-generated text are largely confined to manual feature crafting and supervised binary classification paradigms. These methodologies typically lead to performance bottlenecks and unsatisfactory generalizability. Consequently, these methods are often inapplicable for out-of-distribution (OOD) data and newly emerged large language models (LLMs). In this paper, we revisit the task of AI-generated text detection. We argue that the key to accomplishing this task lies in distinguishing writing styles of different authors, rather than simply classifying the text into human-written or AI-generated text. To this end, we propose DeTeCtive, a multi-task auxiliary, multi-level contrastive learning framework. DeTeCtive is designed to facilitate the learning of distinct writing styles, combined with a dense information retrieval pipeline for AI-generated text detection. Our method is compatible with a range of text encoders. Extensive experiments demonstrate that our method enhances the ability of various text encoders in detecting AI-generated text across multiple benchmarks and achieves state-of-the-art results. Notably, in OOD zero-shot evaluation, our method outperforms existing approaches by a large margin. Moreover, we find our method boasts a Training-Free Incremental Adaptation (TFIA) capability towards OOD data, further enhancing its efficacy in OOD detection scenarios. We will open-source our code and models in hopes that our work will spark new thoughts in the field of AI-generated text detection, ensuring safe application of LLMs and enhancing compliance. Our code is available at this https URL.

Title: Simultaneous Unlearning of Multiple Protected User Attributes From Variational Autoencoder Recommenders Using Adversarial Training

• Authors: Gustavo Escobedo, Christian Ganhör, Stefan Brandl, Mirjam Augstein, Markus Schedl
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.20965
• Pdf URL: https://arxiv.org/pdf/2410.20965
• Copy Paste: [[2410.20965]] Simultaneous Unlearning of Multiple Protected User Attributes From Variational Autoencoder Recommenders Using Adversarial Training(https://arxiv.org/abs/2410.20965)
• Keywords: privacy, protect, fair
• Abstract: In widely used neural network-based collaborative filtering models, users' history logs are encoded into latent embeddings that represent the users' preferences. In this setting, the models are capable of mapping users' protected attributes (e.g., gender or ethnicity) from these user embeddings even without explicit access to them, resulting in models that may treat specific demographic user groups unfairly and raise privacy issues. While prior work has approached the removal of a single protected attribute of a user at a time, multiple attributes might come into play in real-world scenarios. In the work at hand, we present AdvXMultVAE which aims to unlearn multiple protected attributes (exemplified by gender and age) simultaneously to improve fairness across demographic user groups. For this purpose, we couple a variational autoencoder (VAE) architecture with adversarial training (AdvMultVAE) to support simultaneous removal of the users' protected attributes with continuous and/or categorical values. Our experiments on two datasets, LFM-2b-100k and Ml-1m, from the music and movie domains, respectively, show that our approach can yield better results than its singular removal counterparts (based on AdvMultVAE) in effectively mitigating demographic biases whilst improving the anonymity of latent embeddings.

Title: BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks

• Authors: Yunhan Zhao, Xiang Zheng, Lin Luo, Yige Li, Xingjun Ma, Yu-Gang Jiang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.20971
• Pdf URL: https://arxiv.org/pdf/2410.20971
• Copy Paste: [[2410.20971]] BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks(https://arxiv.org/abs/2410.20971)
• Keywords: defense, attack, robust
• Abstract: Despite their superb multimodal capabilities, Vision-Language Models (VLMs) have been shown to be vulnerable to jailbreak attacks, which are inference-time attacks that induce the model to output harmful responses with tricky prompts. It is thus essential to defend VLMs against potential jailbreaks for their trustworthy deployment in real-world applications. In this work, we focus on black-box defense for VLMs against jailbreak attacks. Existing black-box defense methods are either unimodal or bimodal. Unimodal methods enhance either the vision or language module of the VLM, while bimodal methods robustify the model through text-image representation realignment. However, these methods suffer from two limitations: 1) they fail to fully exploit the cross-modal information, or 2) they degrade the model performance on benign inputs. To address these limitations, we propose a novel blue-team method BlueSuffix that defends the black-box target VLM against jailbreak attacks without compromising its performance. BlueSuffix includes three key components: 1) a visual purifier against jailbreak images, 2) a textual purifier against jailbreak texts, and 3) a blue-team suffix generator fine-tuned via reinforcement learning for enhancing cross-modal robustness. We empirically show on three VLMs (LLaVA, MiniGPT-4, and Gemini) and two safety benchmarks (MM-SafetyBench and RedTeam-2K) that BlueSuffix outperforms the baseline defenses by a significant margin. Our BlueSuffix opens up a promising direction for defending VLMs against jailbreak attacks.

Title: Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!

• Authors: Arash Marioriyad, Mohammadali Banayeeanzade, Reza Abbasi, Mohammad Hossein Rohban, Mahdieh Soleymani Baghshah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20972
• Pdf URL: https://arxiv.org/pdf/2410.20972
• Copy Paste: [[2410.20972]] Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!(https://arxiv.org/abs/2410.20972)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models, such as Stable Diffusion and DALL-E, are capable of generating high-quality, diverse, and realistic images from textual prompts. However, they sometimes struggle to accurately depict specific entities described in prompts, a limitation known as the entity missing problem in compositional generation. While prior studies suggested that adjusting cross-attention maps during the denoising process could alleviate this problem, they did not systematically investigate which objective functions could best address it. This study examines three potential causes of the entity-missing problem, focusing on cross-attention dynamics: (1) insufficient attention intensity for certain entities, (2) overly broad attention spread, and (3) excessive overlap between attention maps of different entities. We found that reducing overlap in attention maps between entities can effectively minimize the rate of entity missing. Specifically, we hypothesize that tokens related to specific entities compete for attention on certain image regions during the denoising process, which can lead to divided attention across tokens and prevent accurate representation of each entity. To address this issue, we introduced four loss functions, Intersection over Union (IoU), center-of-mass (CoM) distance, Kullback-Leibler (KL) divergence, and clustering compactness (CC) to regulate attention overlap during denoising steps without the need for retraining. Experimental results across a wide variety of benchmarks reveal that these proposed training-free methods significantly improve compositional accuracy, outperforming previous approaches in visual question answering (VQA), captioning scores, CLIP similarity, and human evaluations. Notably, these methods improved human evaluation scores by 9% over the best baseline, demonstrating substantial improvements in compositional alignment.

Title: MovieCharacter: A Tuning-Free Framework for Controllable Character Video Synthesis

• Authors: Di Qiu, Zheng Chen, Rui Wang, Mingyuan Fan, Changqian Yu, Junshi Huan, Xiang Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.20974
• Pdf URL: https://arxiv.org/pdf/2410.20974
• Copy Paste: [[2410.20974]] MovieCharacter: A Tuning-Free Framework for Controllable Character Video Synthesis(https://arxiv.org/abs/2410.20974)
• Keywords: segmentation
• Abstract: Recent advancements in character video synthesis still depend on extensive fine-tuning or complex 3D modeling processes, which can restrict accessibility and hinder real-time applicability. To address these challenges, we propose a simple yet effective tuning-free framework for character video synthesis, named MovieCharacter, designed to streamline the synthesis process while ensuring high-quality outcomes. Our framework decomposes the synthesis task into distinct, manageable modules: character segmentation and tracking, video object removal, character motion imitation, and video composition. This modular design not only facilitates flexible customization but also ensures that each component operates collaboratively to effectively meet user needs. By leveraging existing open-source models and integrating well-established techniques, MovieCharacter achieves impressive synthesis results without necessitating substantial resources or proprietary datasets. Experimental results demonstrate that our framework enhances the efficiency, accessibility, and adaptability of character video synthesis, paving the way for broader creative and interactive applications.

Title: EEG-Driven 3D Object Reconstruction with Color Consistency and Diffusion Prior

• Authors: Xin Xiang, Wenhui Zhou, Guojun Dai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.20981
• Pdf URL: https://arxiv.org/pdf/2410.20981
• Copy Paste: [[2410.20981]] EEG-Driven 3D Object Reconstruction with Color Consistency and Diffusion Prior(https://arxiv.org/abs/2410.20981)
• Keywords: diffusion
• Abstract: EEG-based visual perception reconstruction has become a current research hotspot. Neuroscientific studies have shown that humans can perceive various types of visual information, such as color, shape, and texture, when observing objects. However, existing technical methods often face issues such as inconsistencies in texture, shape, and color between the visual stimulus images and the reconstructed images. In this paper, we propose a method for reconstructing 3D objects with color consistency based on EEG signals. The method adopts a two-stage strategy: in the first stage, we train an implicit neural EEG encoder with the capability of perceiving 3D objects, enabling it to capture regional semantic features; in the second stage, based on the latent EEG codes obtained in the first stage, we integrate a diffusion model, neural style loss, and NeRF to implicitly decode the 3D objects. Finally, through experimental validation, we demonstrate that our method can reconstruct 3D objects with color consistency using EEG.

Title: Push-Forward Signed Distance Functions enable interpretable and robust continuous shape quantification

• Authors: Roua Rouatbi, Juan Esteban Suarez, Ivo F. Sbalzarini
• Subjects: cs.CV, cs.CG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.21004
• Pdf URL: https://arxiv.org/pdf/2410.21004
• Copy Paste: [[2410.21004]] Push-Forward Signed Distance Functions enable interpretable and robust continuous shape quantification(https://arxiv.org/abs/2410.21004)
• Keywords: robust
• Abstract: We introduce the Push-Forward Signed Distance Morphometric (PF-SDM), a novel method for shape quantification in biomedical imaging that is continuous, interpretable, and invariant to shape-preserving transformations. PF-SDM effectively captures the geometric properties of shapes, including their topological skeletons and radial symmetries. This results in a robust and interpretable shape descriptor that generalizes to capture temporal shape dynamics. Importantly, PF-SDM avoids certain issues of previous geometric morphometrics, like Elliptical Fourier Analysis and Generalized Procrustes Analysis, such as coefficient correlations and landmark choices. We present the PF-SDM theory, provide a practically computable algorithm, and benchmark it on synthetic data.

Title: FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrieval

• Authors: Jinlin Wang, Suyuchen Wang, Ziwen Xia, Sirui Hong, Yun Zhu, Bang Liu, Chenglin Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21012
• Pdf URL: https://arxiv.org/pdf/2410.21012
• Copy Paste: [[2410.21012]] FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrieval(https://arxiv.org/abs/2410.21012)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are proficient at retrieving single facts from extended contexts, yet they struggle with tasks requiring the simultaneous retrieval of multiple facts, especially during generation. This paper identifies a novel "lost-in-the-middle" phenomenon, where LLMs progressively lose track of critical information throughout the generation process, resulting in incomplete or inaccurate retrieval. To address this challenge, we introduce Find All Crucial Texts (FACT), an iterative retrieval method that refines context through successive rounds of rewriting. This approach enables models to capture essential facts incrementally, which are often overlooked in single-pass retrieval. Experiments demonstrate that FACT substantially enhances multi-fact retrieval performance across various tasks, though improvements are less notable in general-purpose QA scenarios. Our findings shed light on the limitations of LLMs in multi-fact retrieval and underscore the need for more resilient long-context retrieval strategies.

Title: Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers

• Authors: Akhilesh Kakolu Ramarao, Kevin Tang, Dinah Baer-Henney
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21013
• Pdf URL: https://arxiv.org/pdf/2410.21013
• Copy Paste: [[2410.21013]] Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers(https://arxiv.org/abs/2410.21013)
• Keywords: transformer
• Abstract: The present paper evaluates the learning behaviour of a transformer-based neural network with regard to an irregular inflectional paradigm. We apply the paradigm cell filling problem to irregular patterns. We approach this problem using the morphological reinflection task and model it as a character sequence-to-sequence learning problem. The test case under investigation are irregular verbs in Spanish. Besides many regular verbs in Spanish L-shaped verbs the first person singular indicative stem irregularly matches the subjunctive paradigm, while other indicative forms remain unaltered. We examine the role of frequency during learning and compare models under differing input frequency conditions. We train the model on a corpus of Spanish with a realistic distribution of regular and irregular verbs to compare it with models trained on input with augmented distributions of (ir)regular words. We explore how the neural models learn this L-shaped pattern using post-hoc analyses. Our experiments show that, across frequency conditions, the models are surprisingly capable of learning the irregular pattern. Furthermore, our post-hoc analyses reveal the possible sources of errors. All code and data are available at \url{this https URL} under MIT license.

Title: Informed Deep Abstaining Classifier: Investigating noise-robust training for diagnostic decision support systems

• Authors: Helen Schneider, Sebastian Nowak, Aditya Parikh, Yannik C. Layer, Maike Theis, Wolfgang Block, Alois M. Sprinkart, Ulrike Attenberger, Rafet Sifa
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21014
• Pdf URL: https://arxiv.org/pdf/2410.21014
• Copy Paste: [[2410.21014]] Informed Deep Abstaining Classifier: Investigating noise-robust training for diagnostic decision support systems(https://arxiv.org/abs/2410.21014)
• Keywords: robust, transformer
• Abstract: Image-based diagnostic decision support systems (DDSS) utilizing deep learning have the potential to optimize clinical workflows. However, developing DDSS requires extensive datasets with expert annotations and is therefore costly. Leveraging report contents from radiological data bases with Natural Language Processing to annotate the corresponding image data promises to replace labor-intensive manual annotation. As mining "real world" databases can introduce label noise, noise-robust training losses are of great interest. However, current noise-robust losses do not consider noise estimations that can for example be derived based on the performance of the automatic label generator used. In this study, we expand the noise-robust Deep Abstaining Classifier (DAC) loss to an Informed Deep Abstaining Classifier (IDAC) loss by incorporating noise level estimations during training. Our findings demonstrate that IDAC enhances the noise robustness compared to DAC and several state-of-the-art loss functions. The results are obtained on various simulated noise levels using a public chest X-ray data set. These findings are reproduced on an in-house noisy data set, where labels were extracted from the clinical systems of the University Hospital Bonn by a text-based transformer. The IDAC can therefore be a valuable tool for researchers, companies or clinics aiming to develop accurate and reliable DDSS from routine clinical data.

Title: Graph Based Traffic Analysis and Delay Prediction

• Authors: Gabriele Borg, Charlie Abela
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21028
• Pdf URL: https://arxiv.org/pdf/2410.21028
• Copy Paste: [[2410.21028]] Graph Based Traffic Analysis and Delay Prediction(https://arxiv.org/abs/2410.21028)
• Keywords: diffusion
• Abstract: This research is focused on traffic congestion in the small island of Malta which is the most densely populated country in the EU with about 1,672 inhabitants per square kilometre (4,331 inhabitants/sq mi). Furthermore, Malta has a rapid vehicle growth. Based on our research, the number of vehicles increased by around 11,000 in a little more than 6 months, which shows how important it is to have an accurate and comprehensive means of collecting data to tackle the issue of fluctuating traffic in Malta. In this paper, we first present the newly built comprehensive traffic dataset, called MalTra. This dataset includes realistic trips made by members of the public across the island over a period of 200 days. We then describe the methodology we adopted to generate syntactic data to complete our data set as much as possible. In our research, we consider both MalTra and the Q-Traffic dataset, which has been used in several other research studies. The statistical ARIMA model and two graph neural networks, the spatial temporal graph convolutional network (STGCN) and the diffusion convolutional recurrent network (DCRNN) were used to analyse and compare the results with existing research. From the evaluation, we found that the DCRNN model outperforms the STGCN with the former resulting in MAE of 3.98 (6.65 in the case of the latter) and a RMSE of 7.78 (against 12.73 of the latter).

Title: Beyond Autoregression: Fast LLMs via Self-Distillation Through Time

• Authors: Justin Deschenaux, Caglar Gulcehre
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21035
• Pdf URL: https://arxiv.org/pdf/2410.21035
• Copy Paste: [[2410.21035]] Beyond Autoregression: Fast LLMs via Self-Distillation Through Time(https://arxiv.org/abs/2410.21035)
• Keywords: diffusion, large language model
• Abstract: Autoregressive (AR) Large Language Models (LLMs) have demonstrated significant success across numerous tasks. However, the AR modeling paradigm presents certain limitations; for instance, contemporary autoregressive LLMs are trained to generate one token at a time, which can result in noticeable latency. Recent advances have indicated that search and repeated sampling can enhance performance in various applications, such as theorem proving, code generation, and alignment, by utilizing greater computational resources during inference. In this study, we demonstrate that diffusion language models are capable of generating at least 32 tokens simultaneously, while exceeding the performance of AR models in text quality and on the LAMBADA natural language understanding benchmark. This outcome is achieved through a novel distillation method for discrete diffusion models, which reduces the number of inference steps by a factor of 32-64. Practically, our models, even without caching, can generate tokens at a rate that is up to 8 times faster than AR models employing KV caching, and we anticipate further improvements with the inclusion of caching. Moreover, we demonstrate the efficacy of our approach for diffusion language models with up to 860M parameters.

Title: Sorting Out the Bad Seeds: Automatic Classification of Cryptocurrency Abuse Reports

• Authors: Gibran Gomez, Kevin van Liebergen, Davide Sanvito, Giuseppe Siracusano, Roberto Gonzalez, Juan Caballero
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21041
• Pdf URL: https://arxiv.org/pdf/2410.21041
• Copy Paste: [[2410.21041]] Sorting Out the Bad Seeds: Automatic Classification of Cryptocurrency Abuse Reports(https://arxiv.org/abs/2410.21041)
• Keywords: large language model
• Abstract: Abuse reporting services collect reports about abuse victims have suffered. Accurate classification of the submitted reports is fundamental to analyzing the prevalence and financial impact of different abuse types (e.g., sextortion, investment, romance). Current classification approaches are problematic because they require the reporter to select the abuse type from a list, assuming the reporter has the necessary experience for the classification, which we show is frequently not the case, or require manual classification by analysts, which does not scale. To address these issues, this paper presents a novel approach to classify cryptocurrency abuse reports automatically. We first build a taxonomy of 19 frequently reported abuse types. Given as input the textual description written by the reporter, our classifier leverages a large language model (LLM) to interpret the text and assign it an abuse type in our taxonomy. We collect 290K cryptocurrency abuse reports from two popular reporting services: BitcoinAbuse and BBB's ScamTracker. We build ground truth datasets for 20K of those reports and use them to evaluate three designs for our LLM-based classifier and four LLMs, as well as a supervised ML classifier used as a baseline. Our LLM-based classifier achieves a precision of 0.92, a recall of 0.87, and an F1 score of 0.89, compared to an F1 score of 0.55 for the baseline. We demonstrate our classifier in two applications: providing financial loss statistics for fine-grained abuse types and generating tagged addresses for cryptocurrency analysis platforms.

Title: Disentangled and Self-Explainable Node Representation Learning

• Authors: Simone Piaggesi, André Panisson, Megha Khosla
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.21043
• Pdf URL: https://arxiv.org/pdf/2410.21043
• Copy Paste: [[2410.21043]] Disentangled and Self-Explainable Node Representation Learning(https://arxiv.org/abs/2410.21043)
• Keywords: interpretability
• Abstract: Node representations, or embeddings, are low-dimensional vectors that capture node properties, typically learned through unsupervised structural similarity objectives or supervised tasks. While recent efforts have focused on explaining graph model decisions, the interpretability of unsupervised node embeddings remains underexplored. To bridge this gap, we introduce DiSeNE (Disentangled and Self-Explainable Node Embedding), a framework that generates self-explainable embeddings in an unsupervised manner. Our method employs disentangled representation learning to produce dimension-wise interpretable embeddings, where each dimension is aligned with distinct topological structure of the graph. We formalize novel desiderata for disentangled and interpretable embeddings, which drive our new objective functions, optimizing simultaneously for both interpretability and disentanglement. Additionally, we propose several new metrics to evaluate representation quality and human interpretability. Extensive experiments across multiple benchmark datasets demonstrate the effectiveness of our approach.

Title: Computable Lipschitz Bounds for Deep Neural Networks

• Authors: Moreno Pintore, Bruno Després
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.21053
• Pdf URL: https://arxiv.org/pdf/2410.21053
• Copy Paste: [[2410.21053]] Computable Lipschitz Bounds for Deep Neural Networks(https://arxiv.org/abs/2410.21053)
• Keywords: robust
• Abstract: Deriving sharp and computable upper bounds of the Lipschitz constant of deep neural networks is crucial to formally guarantee the robustness of neural-network based models. We analyse three existing upper bounds written for the $l^2$ norm. We highlight the importance of working with the $l^1$ and $l^\infty$ norms and we propose two novel bounds for both feed-forward fully-connected neural networks and convolutional neural networks. We treat the technical difficulties related to convolutional neural networks with two different methods, called explicit and implicit. Several numerical tests empirically confirm the theoretical results, help to quantify the relationship between the presented bounds and establish the better accuracy of the new bounds. Four numerical tests are studied: two where the output is derived from an analytical closed form are proposed; another one with random matrices; and the last one for convolutional neural networks trained on the MNIST dataset. We observe that one of our bound is optimal in the sense that it is exact for the first test with the simplest analytical form and it is better than other bounds for the other tests.

Title: CTINEXUS: Leveraging Optimized LLM In-Context Learning for Constructing Cybersecurity Knowledge Graphs Under Data Scarcity

• Authors: Yutong Cheng, Osama Bajaber, Saimon Amanuel Tsegai, Dawn Song, Peng Gao
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21060
• Pdf URL: https://arxiv.org/pdf/2410.21060
• Copy Paste: [[2410.21060]] CTINEXUS: Leveraging Optimized LLM In-Context Learning for Constructing Cybersecurity Knowledge Graphs Under Data Scarcity(https://arxiv.org/abs/2410.21060)
• Keywords: security, extraction, large language model
• Abstract: Textual descriptions in cyber threat intelligence (CTI) reports, such as security articles and news, are rich sources of knowledge about cyber threats, crucial for organizations to stay informed about the rapidly evolving threat landscape. However, current CTI extraction methods lack flexibility and generalizability, often resulting in inaccurate and incomplete knowledge extraction. Syntax parsing relies on fixed rules and dictionaries, while model fine-tuning requires large annotated datasets, making both paradigms challenging to adapt to new threats and ontologies. To bridge the gap, we propose CTINexus, a novel framework leveraging optimized in-context learning (ICL) of large language models (LLMs) for data-efficient CTI knowledge extraction and high-quality cybersecurity knowledge graph (CSKG) construction. Unlike existing methods, CTINexus requires neither extensive data nor parameter tuning and can adapt to various ontologies with minimal annotated examples. This is achieved through (1) a carefully designed automatic prompt construction strategy with optimal demonstration retrieval for extracting a wide range of cybersecurity entities and relations; (2) a hierarchical entity alignment technique that canonicalizes the extracted knowledge and removes redundancy; (3) an ICL-enhanced long-distance relation prediction technique to further complete the CKSG with missing links. Our extensive evaluations using 150 real-world CTI reports collected from 10 platforms demonstrate that CTINexus significantly outperforms existing methods in constructing accurate and complete CSKGs, highlighting its potential to transform CTI analysis with an efficient and adaptable solution for the dynamic threat landscape.

Title: Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework

• Authors: Vladimir Arkhipkin, Viacheslav Vasilev, Andrei Filatov, Igor Pavlov, Julia Agafonova, Nikolai Gerasimenko, Anna Averchenkova, Evelina Mironova, Anton Bukashkin, Konstantin Kulikov, Andrey Kuznetsov, Denis Dimitrov
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.21061
• Pdf URL: https://arxiv.org/pdf/2410.21061
• Copy Paste: [[2410.21061]] Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework(https://arxiv.org/abs/2410.21061)
• Keywords: diffusion, generative
• Abstract: Text-to-image (T2I) diffusion models are popular for introducing image manipulation methods, such as editing, image fusion, inpainting, etc. At the same time, image-to-video (I2V) and text-to-video (T2V) models are also built on top of T2I models. We present Kandinsky 3, a novel T2I model based on latent diffusion, achieving a high level of quality and photorealism. The key feature of the new architecture is the simplicity and efficiency of its adaptation for many types of generation tasks. We extend the base T2I model for various applications and create a multifunctional generation system that includes text-guided inpainting/outpainting, image fusion, text-image fusion, image variations generation, I2V and T2V generation. We also present a distilled version of the T2I model, evaluating inference in 4 steps of the reverse process without reducing image quality and 3 times faster than the base model. We deployed a user-friendly demo system in which all the features can be tested in the public domain. Additionally, we released the source code and checkpoints for the Kandinsky 3 and extended models. Human evaluations show that Kandinsky 3 demonstrates one of the highest quality scores among open source generation systems.

Title: CRAT: A Multi-Agent Framework for Causality-Enhanced Reflective and Retrieval-Augmented Translation with Large Language Models

• Authors: Meiqi Chen, Fandong Meng, Yingxue Zhang, Yan Zhang, Jie Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21067
• Pdf URL: https://arxiv.org/pdf/2410.21067
• Copy Paste: [[2410.21067]] CRAT: A Multi-Agent Framework for Causality-Enhanced Reflective and Retrieval-Augmented Translation with Large Language Models(https://arxiv.org/abs/2410.21067)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While Retrieval-Augmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. In this paper, we propose CRAT, a novel multi-agent translation framework that leverages RAG and causality-enhanced self-reflection to address these challenges. This framework consists of several specialized agents: the Unknown Terms Identification agent detects unknown terms within the context, the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, the Causality-enhanced Judge agent validates the accuracy of the information, and the Translator agent incorporates the refined information into the final output. This automated process allows for more precise and consistent handling of key terms during translation. Our results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary.

Title: Federated Time Series Generation on Feature and Temporally Misaligned Data

• Authors: Chenrui Fan, Zhi Wen Soi, Aditya Shankar, Abele Mălan, Lydia Y. Chen
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.21072
• Pdf URL: https://arxiv.org/pdf/2410.21072
• Copy Paste: [[2410.21072]] Federated Time Series Generation on Feature and Temporally Misaligned Data(https://arxiv.org/abs/2410.21072)
• Keywords: federate, diffusion
• Abstract: Distributed time series data presents a challenge for federated learning, as clients often possess different feature sets and have misaligned time steps. Existing federated time series models are limited by the assumption of perfect temporal or feature alignment across clients. In this paper, we propose FedTDD, a novel federated time series diffusion model that jointly learns a synthesizer across clients. At the core of FedTDD is a novel data distillation and aggregation framework that reconciles the differences between clients by imputing the misaligned timesteps and features. In contrast to traditional federated learning, FedTDD learns the correlation across clients' time series through the exchange of local synthetic outputs instead of model parameters. A coordinator iteratively improves a global distiller network by leveraging shared knowledge from clients through the exchange of synthetic data. As the distiller becomes more refined over time, it subsequently enhances the quality of the clients' local feature estimates, allowing each client to then improve its local imputations for missing data using the latest, more accurate distiller. Experimental results on five datasets demonstrate FedTDD's effectiveness compared to centralized training, and the effectiveness of sharing synthetic outputs to transfer knowledge of local time series. Notably, FedTDD achieves 79.4% and 62.8% improvement over local training in Context-FID and Correlational scores.

Title: Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring

• Authors: Honglin Mu, Han He, Yuxin Zhou, Yunlong Feng, Yang Xu, Libo Qin, Xiaoming Shi, Zeming Liu, Xudong Han, Qi Shi, Qingfu Zhu, Wanxiang Che
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21083
• Pdf URL: https://arxiv.org/pdf/2410.21083
• Copy Paste: [[2410.21083]] Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring(https://arxiv.org/abs/2410.21083)
• Keywords: security, defense, attack, robust, steal, large language model
• Abstract: Large language model (LLM) safety is a critical issue, with numerous studies employing red team testing to enhance model security. Among these, jailbreak methods explore potential vulnerabilities by crafting malicious prompts that induce model outputs contrary to safety alignments. Existing black-box jailbreak methods often rely on model feedback, repeatedly submitting queries with detectable malicious instructions during the attack search process. Although these approaches are effective, the attacks may be intercepted by content moderators during the search process. We propose an improved transfer attack method that guides malicious prompt construction by locally training a mirror model of the target black-box model through benign data distillation. This method offers enhanced stealth, as it does not involve submitting identifiable malicious instructions to the target model during the search phase. Our approach achieved a maximum attack success rate of 92%, or a balanced value of 80% with an average of 1.5 detectable jailbreak queries per sample against GPT-3.5 Turbo on a subset of AdvBench. These results underscore the need for more robust defense mechanisms.

Title: KA$^2$ER: Knowledge Adaptive Amalgamation of ExpeRts for Medical Images Segmentation

• Authors: Shangde Gao, Yichao Fu, Ke Liu, Hongxia Xu, Jian Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21085
• Pdf URL: https://arxiv.org/pdf/2410.21085
• Copy Paste: [[2410.21085]] KA$^2$ER: Knowledge Adaptive Amalgamation of ExpeRts for Medical Images Segmentation(https://arxiv.org/abs/2410.21085)
• Keywords: segmentation
• Abstract: Recently, many foundation models for medical image analysis such as MedSAM, SwinUNETR have been released and proven to be useful in multiple tasks. However, considering the inherent heterogeneity and inhomogeneity of real-world medical data, directly applying these models to specific medical image segmentation tasks often leads to negative domain shift effects, which can severely weaken the model's segmentation capabilities. To this end, we propose an adaptive amalgamation knowledge framework that aims to train a versatile foundation model to handle the joint goals of multiple expert models, each specialized for a distinct task. Specifically, we first train an nnUNet-based expert model for each task, and reuse the pre-trained SwinUNTER as the target foundation model. Then, the input data for all challenging tasks are encoded in the foundation model and the expert models, respectively, and their backbone features are jointly projected into the adaptive amalgamation layer. Within the hidden layer, the hierarchical attention mechanisms are designed to achieve adaptive merging of the target model to the hidden layer feature knowledge of all experts, which significantly reduces the domain shift arising from the inter-task differences. Finally, the gold amalgamated features and the prompt features are fed into the mask decoder to obtain the segmentation results. Extensive experiments conducted in these challenging tasks demonstrate the effectiveness and adaptability of our foundation model for real-world medical image segmentation.

Title: Shallow Diffuse: Robust and Invisible Watermarking through Low-Dimensional Subspaces in Diffusion Models

• Authors: Wenda Li, Huijie Zhang, Qing Qu
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21088
• Pdf URL: https://arxiv.org/pdf/2410.21088
• Copy Paste: [[2410.21088]] Shallow Diffuse: Robust and Invisible Watermarking through Low-Dimensional Subspaces in Diffusion Models(https://arxiv.org/abs/2410.21088)
• Keywords: robust, watermark, diffusion
• Abstract: The widespread use of AI-generated content from diffusion models has raised significant concerns regarding misinformation and copyright infringement. Watermarking is a crucial technique for identifying these AI-generated images and preventing their misuse. In this paper, we introduce Shallow Diffuse, a new watermarking technique that embeds robust and invisible watermarks into diffusion model outputs. Unlike existing approaches that integrate watermarking throughout the entire diffusion sampling process, Shallow Diffuse decouples these steps by leveraging the presence of a low-dimensional subspace in the image generation process. This method ensures that a substantial portion of the watermark lies in the null space of this subspace, effectively separating it from the image generation process. Our theoretical and empirical analyses show that this decoupling strategy greatly enhances the consistency of data generation and the detectability of the watermark. Extensive experiments further validate that our Shallow Diffuse outperforms existing watermarking methods in terms of robustness and consistency. The codes will be released at this https URL.

Title: Tree-Wasserstein Distance for High Dimensional Data with a Latent Feature Hierarchy

• Authors: Ya-Wei Eileen Lin, Ronald R. Coifman, Gal Mishne, Ronen Talmon
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.21107
• Pdf URL: https://arxiv.org/pdf/2410.21107
• Copy Paste: [[2410.21107]] Tree-Wasserstein Distance for High Dimensional Data with a Latent Feature Hierarchy(https://arxiv.org/abs/2410.21107)
• Keywords: diffusion
• Abstract: Finding meaningful distances between high-dimensional data samples is an important scientific task. To this end, we propose a new tree-Wasserstein distance (TWD) for high-dimensional data with two key aspects. First, our TWD is specifically designed for data with a latent feature hierarchy, i.e., the features lie in a hierarchical space, in contrast to the usual focus on embedding samples in hyperbolic space. Second, while the conventional use of TWD is to speed up the computation of the Wasserstein distance, we use its inherent tree as a means to learn the latent feature hierarchy. The key idea of our method is to embed the features into a multi-scale hyperbolic space using diffusion geometry and then present a new tree decoding method by establishing analogies between the hyperbolic embedding and trees. We show that our TWD computed based on data observations provably recovers the TWD defined with the latent feature hierarchy and that its computation is efficient and scalable. We showcase the usefulness of the proposed TWD in applications to word-document and single-cell RNA-sequencing datasets, demonstrating its advantages over existing TWDs and methods based on pre-trained models.

Title: LiGAR: LiDAR-Guided Hierarchical Transformer for Multi-Modal Group Activity Recognition

• Authors: Naga Venkata Sai Raviteja Chappa, Khoa Luu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21108
• Pdf URL: https://arxiv.org/pdf/2410.21108
• Copy Paste: [[2410.21108]] LiGAR: LiDAR-Guided Hierarchical Transformer for Multi-Modal Group Activity Recognition(https://arxiv.org/abs/2410.21108)
• Keywords: robust, transformer
• Abstract: Group Activity Recognition (GAR) remains challenging in computer vision due to the complex nature of multi-agent interactions. This paper introduces LiGAR, a LIDAR-Guided Hierarchical Transformer for Multi-Modal Group Activity Recognition. LiGAR leverages LiDAR data as a structural backbone to guide the processing of visual and textual information, enabling robust handling of occlusions and complex spatial arrangements. Our framework incorporates a Multi-Scale LIDAR Transformer, Cross-Modal Guided Attention, and an Adaptive Fusion Module to integrate multi-modal data at different semantic levels effectively. LiGAR's hierarchical architecture captures group activities at various granularities, from individual actions to scene-level dynamics. Extensive experiments on the JRDB-PAR, Volleyball, and NBA datasets demonstrate LiGAR's superior performance, achieving state-of-the-art results with improvements of up to 10.6% in F1-score on JRDB-PAR and 5.9% in Mean Per Class Accuracy on the NBA dataset. Notably, LiGAR maintains high performance even when LiDAR data is unavailable during inference, showcasing its adaptability. Our ablation studies highlight the significant contributions of each component and the effectiveness of our multi-modal, multi-scale approach in advancing the field of group activity recognition.

Title: LAMA: Stable Dual-Domain Deep Reconstruction For Sparse-View CT

• Authors: Chi Ding, Qingchao Zhang, Ge Wang, Xiaojing Ye, Yunmei Chen
• Subjects: cs.CV, cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2410.21111
• Pdf URL: https://arxiv.org/pdf/2410.21111
• Copy Paste: [[2410.21111]] LAMA: Stable Dual-Domain Deep Reconstruction For Sparse-View CT(https://arxiv.org/abs/2410.21111)
• Keywords: interpretability
• Abstract: Inverse problems arise in many applications, especially tomographic imaging. We develop a Learned Alternating Minimization Algorithm (LAMA) to solve such problems via two-block optimization by synergizing data-driven and classical techniques with proven convergence. LAMA is naturally induced by a variational model with learnable regularizers in both data and image domains, parameterized as composite functions of neural networks trained with domain-specific data. We allow these regularizers to be nonconvex and nonsmooth to extract features from data effectively. We minimize the overall objective function using Nesterov's smoothing technique and residual learning architecture. It is demonstrated that LAMA reduces network complexity, improves memory efficiency, and enhances reconstruction accuracy, stability, and interpretability. Extensive experiments show that LAMA significantly outperforms state-of-the-art methods on popular benchmark datasets for Computed Tomography.

Title: FusedInf: Efficient Swapping of DNN Models for On-Demand Serverless Inference Services on the Edge

• Authors: Sifat Ut Taki, Arthi Padmanabhan, Spyridon Mastorakis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21120
• Pdf URL: https://arxiv.org/pdf/2410.21120
• Copy Paste: [[2410.21120]] FusedInf: Efficient Swapping of DNN Models for On-Demand Serverless Inference Services on the Edge(https://arxiv.org/abs/2410.21120)
• Keywords: robust
• Abstract: Edge AI computing boxes are a new class of computing devices that are aimed to revolutionize the AI industry. These compact and robust hardware units bring the power of AI processing directly to the source of data--on the edge of the network. On the other hand, on-demand serverless inference services are becoming more and more popular as they minimize the infrastructural cost associated with hosting and running DNN models for small to medium-sized businesses. However, these computing devices are still constrained in terms of resource availability. As such, the service providers need to load and unload models efficiently in order to meet the growing demand. In this paper, we introduce FusedInf to efficiently swap DNN models for on-demand serverless inference services on the edge. FusedInf combines multiple models into a single Direct Acyclic Graph (DAG) to efficiently load the models into the GPU memory and make execution faster. Our evaluation of popular DNN models showed that creating a single DAG can make the execution of the models up to 14\% faster while reducing the memory requirement by up to 17\%. The prototype implementation is available at this https URL.

Title: Extrapolating Prospective Glaucoma Fundus Images through Diffusion Model in Irregular Longitudinal Sequences

• Authors: Zhihao Zhao, Junjie Yang, Shahrooz Faghihroohi, Yinzheng Zhao, Daniel Zapp, Kai Huang, Nassir Navab, M.Ali Nasseri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21130
• Pdf URL: https://arxiv.org/pdf/2410.21130
• Copy Paste: [[2410.21130]] Extrapolating Prospective Glaucoma Fundus Images through Diffusion Model in Irregular Longitudinal Sequences(https://arxiv.org/abs/2410.21130)
• Keywords: diffusion
• Abstract: The utilization of longitudinal datasets for glaucoma progression prediction offers a compelling approach to support early therapeutic interventions. Predominant methodologies in this domain have primarily focused on the direct prediction of glaucoma stage labels from longitudinal datasets. However, such methods may not adequately encapsulate the nuanced developmental trajectory of the disease. To enhance the diagnostic acumen of medical practitioners, we propose a novel diffusion-based model to predict prospective images by extrapolating from existing longitudinal fundus images of patients. The methodology delineated in this study distinctively leverages sequences of images as inputs. Subsequently, a time-aligned mask is employed to select a specific year for image generation. During the training phase, the time-aligned mask resolves the issue of irregular temporal intervals in longitudinal image sequence sampling. Additionally, we utilize a strategy of randomly masking a frame in the sequence to establish the ground truth. This methodology aids the network in continuously acquiring knowledge regarding the internal relationships among the sequences throughout the learning phase. Moreover, the introduction of textual labels is instrumental in categorizing images generated within the sequence. The empirical findings from the conducted experiments indicate that our proposed model not only effectively generates longitudinal data but also significantly improves the precision of downstream classification tasks.

Title: uOttawa at LegalLens-2024: Transformer-based Classification Experiments

• Authors: Nima Meghdadi, Diana Inkpen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21139
• Pdf URL: https://arxiv.org/pdf/2410.21139
• Copy Paste: [[2410.21139]] uOttawa at LegalLens-2024: Transformer-based Classification Experiments(https://arxiv.org/abs/2410.21139)
• Keywords: transformer
• Abstract: This paper presents the methods used for LegalLens-2024 shared task, which focused on detecting legal violations within unstructured textual data and associating these violations with potentially affected individuals. The shared task included two subtasks: A) Legal Named Entity Recognition (L-NER) and B) Legal Natural Language Inference (L-NLI). For subtask A, we utilized the spaCy library, while for subtask B, we employed a combined model incorporating RoBERTa and CNN. Our results were 86.3% in the L-NER subtask and 88.25% in the L-NLI subtask. Overall, our paper demonstrates the effectiveness of transformer models in addressing complex tasks in the legal domain. The source code for our implementation is publicly available at this https URL

Title: LLM-initialized Differentiable Causal Discovery

• Authors: Shiv Kampani, David Hidary, Constantijn van der Poel, Martin Ganahl, Brenda Miao
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.21141
• Pdf URL: https://arxiv.org/pdf/2410.21141
• Copy Paste: [[2410.21141]] LLM-initialized Differentiable Causal Discovery(https://arxiv.org/abs/2410.21141)
• Keywords: interpretability, large language model
• Abstract: The discovery of causal relationships between random variables is an important yet challenging problem that has applications across many scientific domains. Differentiable causal discovery (DCD) methods are effective in uncovering causal relationships from observational data; however, these approaches often suffer from limited interpretability and face challenges in incorporating domain-specific prior knowledge. In contrast, Large Language Models (LLMs)-based causal discovery approaches have recently been shown capable of providing useful priors for causal discovery but struggle with formal causal reasoning. In this paper, we propose LLM-DCD, which uses an LLM to initialize the optimization of the maximum likelihood objective function of DCD approaches, thereby incorporating strong priors into the discovery method. To achieve this initialization, we design our objective function to depend on an explicitly defined adjacency matrix of the causal graph as its only variational parameter. Directly optimizing the explicitly defined adjacency matrix provides a more interpretable approach to causal discovery. Additionally, we demonstrate higher accuracy on key benchmarking datasets of our approach compared to state-of-the-art alternatives, and provide empirical evidence that the quality of the initialization directly impacts the quality of the final output of our DCD approach. LLM-DCD opens up new opportunities for traditional causal discovery methods like DCD to benefit from future improvements in the causal reasoning capabilities of LLMs.

Title: Enhancing Learned Image Compression via Cross Window-based Attention

• Authors: Priyanka Mudgal, Feng Liu
• Subjects: cs.CV, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.21144
• Pdf URL: https://arxiv.org/pdf/2410.21144
• Copy Paste: [[2410.21144]] Enhancing Learned Image Compression via Cross Window-based Attention(https://arxiv.org/abs/2410.21144)
• Keywords: transformer
• Abstract: In recent years, learned image compression methods have demonstrated superior rate-distortion performance compared to traditional image compression methods. Recent methods utilize convolutional neural networks (CNN), variational autoencoders (VAE), invertible neural networks (INN), and transformers. Despite their significant contributions, a main drawback of these models is their poor performance in capturing local redundancy. Therefore, to leverage global features along with local redundancy, we propose a CNN-based solution integrated with a feature encoding module. The feature encoding module encodes important features before feeding them to the CNN and then utilizes cross-scale window-based attention, which further captures local redundancy. Cross-scale window-based attention is inspired by the attention mechanism in transformers and effectively enlarges the receptive field. Both the feature encoding module and the cross-scale window-based attention module in our architecture are flexible and can be incorporated into any other network architecture. We evaluate our method on the Kodak and CLIC datasets and demonstrate that our approach is effective and on par with state-of-the-art methods.

Title: Palisade -- Prompt Injection Detection Framework

• Authors: Sahasra Kokkula, Somanathan R, Nandavardhan R, Aashishkumar, G Divya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21146
• Pdf URL: https://arxiv.org/pdf/2410.21146
• Copy Paste: [[2410.21146]] Palisade -- Prompt Injection Detection Framework(https://arxiv.org/abs/2410.21146)
• Keywords: secure, attack, large language model
• Abstract: The advent of Large Language Models LLMs marks a milestone in Artificial Intelligence, altering how machines comprehend and generate human language. However, LLMs are vulnerable to malicious prompt injection attacks, where crafted inputs manipulate the models behavior in unintended ways, compromising system integrity and causing incorrect outcomes. Conventional detection methods rely on static, rule-based approaches, which often fail against sophisticated threats like abnormal token sequences and alias substitutions, leading to limited adaptability and higher rates of false positives and false this http URL paper proposes a novel NLP based approach for prompt injection detection, emphasizing accuracy and optimization through a layered input screening process. In this framework, prompts are filtered through three distinct layers rule-based, ML classifier, and companion LLM before reaching the target model, thereby minimizing the risk of malicious this http URL show the ML classifier achieves the highest accuracy among individual layers, yet the multi-layer framework enhances overall detection accuracy by reducing false negatives. Although this increases false positives, it minimizes the risk of overlooking genuine injected prompts, thus prioritizing this http URL multi-layered detection approach highlights LLM vulnerabilities and provides a comprehensive framework for future research, promoting secure interactions between humans and AI systems.

Title: Synthetica: Large Scale Synthetic Data for Robot Perception

• Authors: Ritvik Singh, Jingzhou Liu, Karl Van Wyk, Yu-Wei Chao, Jean-Francois Lafleche, Florian Shkurti, Nathan Ratliff, Ankur Handa
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.21153
• Pdf URL: https://arxiv.org/pdf/2410.21153
• Copy Paste: [[2410.21153]] Synthetica: Large Scale Synthetic Data for Robot Perception(https://arxiv.org/abs/2410.21153)
• Keywords: robust, transformer
• Abstract: Vision-based object detectors are a crucial basis for robotics applications as they provide valuable information about object localisation in the environment. These need to ensure high reliability in different lighting conditions, occlusions, and visual artifacts, all while running in real-time. Collecting and annotating real-world data for these networks is prohibitively time consuming and costly, especially for custom assets, such as industrial objects, making it untenable for generalization to in-the-wild scenarios. To this end, we present Synthetica, a method for large-scale synthetic data generation for training robust state estimators. This paper focuses on the task of object detection, an important problem which can serve as the front-end for most state estimation problems, such as pose estimation. Leveraging data from a photorealistic ray-tracing renderer, we scale up data generation, generating 2.7 million images, to train highly accurate real-time detection transformers. We present a collection of rendering randomization and training-time data augmentation techniques conducive to robust sim-to-real performance for vision tasks. We demonstrate state-of-the-art performance on the task of object detection while having detectors that run at 50-100Hz which is 9 times faster than the prior SOTA. We further demonstrate the usefulness of our training methodology for robotics applications by showcasing a pipeline for use in the real world with custom objects for which there do not exist prior datasets. Our work highlights the importance of scaling synthetic data generation for robust sim-to-real transfer while achieving the fastest real-time inference speeds. Videos and supplementary information can be found at this URL: this https URL.

Title: Trajectory Flow Matching with Applications to Clinical Time Series Modeling

• Authors: Xi Zhang, Yuan Pu, Yuki Kawamura, Andrew Loza, Yoshua Bengio, Dennis L. Shung, Alexander Tong
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.21154
• Pdf URL: https://arxiv.org/pdf/2410.21154
• Copy Paste: [[2410.21154]] Trajectory Flow Matching with Applications to Clinical Time Series Modeling(https://arxiv.org/abs/2410.21154)
• Keywords: diffusion, generative
• Abstract: Modeling stochastic and irregularly sampled time series is a challenging problem found in a wide range of applications, especially in medicine. Neural stochastic differential equations (Neural SDEs) are an attractive modeling technique for this problem, which parameterize the drift and diffusion terms of an SDE with neural networks. However, current algorithms for training Neural SDEs require backpropagation through the SDE dynamics, greatly limiting their scalability and stability. To address this, we propose Trajectory Flow Matching (TFM), which trains a Neural SDE in a simulation-free manner, bypassing backpropagation through the dynamics. TFM leverages the flow matching technique from generative modeling to model time series. In this work we first establish necessary conditions for TFM to learn time series data. Next, we present a reparameterization trick which improves training stability. Finally, we adapt TFM to the clinical time series setting, demonstrating improved performance on three clinical time series datasets both in terms of absolute performance and uncertainty prediction.

Title: SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents

• Authors: Qi Zhang, Zhijia Chen, Huitong Pan, Cornelia Caragea, Longin Jan Latecki, Eduard Dragut
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21155
• Pdf URL: https://arxiv.org/pdf/2410.21155
• Copy Paste: [[2410.21155]] SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents(https://arxiv.org/abs/2410.21155)
• Keywords: extraction
• Abstract: Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data (entities and relations). Several datasets have been proposed for training and validating SciIE models. However, due to the high complexity and cost of annotating scientific texts, those datasets restrict their annotations to specific parts of paper, such as abstracts, resulting in the loss of diverse entity mentions and relations in context. In this paper, we release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations. To capture the intricate use and interactions among entities in full texts, our dataset contains a fine-grained tag set for relations. Additionally, we provide an out-of-distribution test set to offer a more realistic evaluation. We conduct comprehensive experiments, including state-of-the-art supervised models and our proposed LLM-based baselines, and highlight the challenges presented by our dataset, encouraging the development of innovative models to further the field of SciIE.

Title: M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation

• Authors: Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, Ge Zhang, Zekun Wang, Guoan Zhang, Bangyu Xiang, Wenbo Su, Bo Zheng
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.21157
• Pdf URL: https://arxiv.org/pdf/2410.21157
• Copy Paste: [[2410.21157]] M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation(https://arxiv.org/abs/2410.21157)
• Keywords: large language model
• Abstract: Repository-level code completion has drawn great attention in software engineering, and several benchmark datasets have been introduced. However, existing repository-level code completion benchmarks usually focus on a limited number of languages (<5), which cannot evaluate the general code intelligence abilities across different languages for existing code Large Language Models (LLMs). Besides, the existing benchmarks usually report overall average scores of different languages, where the fine-grained abilities in different completion scenarios are ignored. Therefore, to facilitate the research of code LLMs in multilingual scenarios, we propose a massively multilingual repository-level code completion benchmark covering 18 programming languages (called M2RC-EVAL), and two types of fine-grained annotations (i.e., bucket-level and semantic-level) on different completion scenarios are provided, where we obtain these annotations based on the parsed abstract syntax tree. Moreover, we also curate a massively multilingual instruction corpora M2RC- INSTRUCT dataset to improve the repository-level code completion abilities of existing code LLMs. Comprehensive experimental results demonstrate the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.

Title: Resilience in Knowledge Graph Embeddings

• Authors: Arnab Sharma, N'Dah Jean Kouagou, Axel-Cyrille Ngonga Ngomo
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.21163
• Pdf URL: https://arxiv.org/pdf/2410.21163
• Copy Paste: [[2410.21163]] Resilience in Knowledge Graph Embeddings(https://arxiv.org/abs/2410.21163)
• Keywords: attack, robust
• Abstract: In recent years, knowledge graphs have gained interest and witnessed widespread applications in various domains, such as information retrieval, question-answering, recommendation systems, amongst others. Large-scale knowledge graphs to this end have demonstrated their utility in effectively representing structured knowledge. To further facilitate the application of machine learning techniques, knowledge graph embedding (KGE) models have been developed. Such models can transform entities and relationships within knowledge graphs into vectors. However, these embedding models often face challenges related to noise, missing information, distribution shift, adversarial attacks, etc. This can lead to sub-optimal embeddings and incorrect inferences, thereby negatively impacting downstream applications. While the existing literature has focused so far on adversarial attacks on KGE models, the challenges related to the other critical aspects remain unexplored. In this paper, we, first of all, give a unified definition of resilience, encompassing several factors such as generalisation, performance consistency, distribution adaption, and robustness. After formalizing these concepts for machine learning in general, we define them in the context of knowledge graphs. To find the gap in the existing works on resilience in the context of knowledge graphs, we perform a systematic survey, taking into account all these aspects mentioned previously. Our survey results show that most of the existing works focus on a specific aspect of resilience, namely robustness. After categorizing such works based on their respective aspects of resilience, we discuss the challenges and future research directions.

Title: Deep Learning-Based Fatigue Cracks Detection in Bridge Girders using Feature Pyramid Networks

• Authors: Jiawei Zhang, Jun Li, Reachsak Ly, Yunyi Liu, Jiangpeng Shu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21175
• Pdf URL: https://arxiv.org/pdf/2410.21175
• Copy Paste: [[2410.21175]] Deep Learning-Based Fatigue Cracks Detection in Bridge Girders using Feature Pyramid Networks(https://arxiv.org/abs/2410.21175)
• Keywords: segmentation
• Abstract: For structural health monitoring, continuous and automatic crack detection has been a challenging problem. This study is conducted to propose a framework of automatic crack segmentation from high-resolution images containing crack information about steel box girders of bridges. Considering the multi-scale feature of cracks, convolutional neural network architecture of Feature Pyramid Networks (FPN) for crack detection is proposed. As for input, 120 raw images are processed via two approaches (shrinking the size of images and splitting images into sub-images). Then, models with the proposed structure of FPN for crack detection are developed. The result shows all developed models can automatically detect the cracks at the raw images. By shrinking the images, the computation efficiency is improved without decreasing accuracy. Because of the separable characteristic of crack, models using the splitting method provide more accurate crack segmentations than models using the resizing method. Therefore, for high-resolution images, the FPN structure coupled with the splitting method is an promising solution for the crack segmentation and detection.

Title: Privacy-Preserving for Images in Satellite Communications: A Comprehensive Review of Chaos-Based Encryption

• Authors: Farrukh Bin Rashid, Windhya Rankothge, Somayeh Sadeghi, Hesamodin Mohammadian, Ali Ghorbani
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.21177
• Pdf URL: https://arxiv.org/pdf/2410.21177
• Copy Paste: [[2410.21177]] Privacy-Preserving for Images in Satellite Communications: A Comprehensive Review of Chaos-Based Encryption(https://arxiv.org/abs/2410.21177)
• Keywords: security, privacy, protect
• Abstract: In an era where global connectivity has become critical, satellite communication is essential for businesses, governments, and individuals. Widely used services with satellite communication such as climate change monitoring, military surveillance and real-time event broadcasting, involve data in the form of images rather text. Therefore, securing image transmission in satellite communication using efficient and effective encryption approaches, has gained a significant attention from academia as well as the industry. In this paper, we specifically focus on chaos based image encryption as one of the key privacy-preserving techniques for satellite communication. While there are several privacy enhancing techniques for protecting image data but chaos based encryption has distinct advantages such as high flexibility, high security, less computational overheads, less computing power and ease of implementation. First, we present a solid background about satellite communication and image encryption in satellite communication, covering theoretical aspects of chaotic systems and their practical usage for image encryption. Next we present a comprehensive literature review on all state-of-the-art studies specifically for chaos based satellite image encryption, with a detailed analysis of the evaluation process, including evaluation parameters and conditions. Finally, we discuss about existing challenges and open research problems for chaos based satellite image encryption.

Title: Unharmful Backdoor-based Client-side Watermarking in Federated Learning

• Authors: Kaijing Luo, Ka-Ho Chow
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.21179
• Pdf URL: https://arxiv.org/pdf/2410.21179
• Copy Paste: [[2410.21179]] Unharmful Backdoor-based Client-side Watermarking in Federated Learning(https://arxiv.org/abs/2410.21179)
• Keywords: protect, federate, watermark
• Abstract: Protecting intellectual property (IP) in federated learning (FL) is increasingly important as clients contribute proprietary data to collaboratively train models. Model watermarking, particularly through backdoor-based methods, has emerged as a popular approach for verifying ownership and contributions in deep neural networks trained via FL. By manipulating their datasets, clients can embed a secret pattern, resulting in non-intuitive predictions that serve as proof of participation, useful for claiming incentives or IP co-ownership. However, this technique faces practical challenges: client watermarks can collide, leading to ambiguous ownership claims, and malicious clients may exploit watermarks to inject harmful backdoors, jeopardizing model integrity. To address these issues, we propose Sanitizer, a server-side method that ensures client-embedded backdoors cannot be triggered on natural queries in harmful ways. It identifies subnets within client-submitted models, extracts backdoors throughout the FL process, and confines them to harmless, client-specific input subspaces. This approach not only enhances Sanitizer's efficiency but also resolves conflicts when clients use similar triggers with different target labels. Our empirical results demonstrate that Sanitizer achieves near-perfect success in verifying client contributions while mitigating the risks of malicious watermark use. Additionally, it reduces GPU memory consumption by 85% and cuts processing time by at least 5 times compared to the baseline.

Title: On Homomorphic Encryption Based Strategies for Class Imbalance in Federated Learning

• Authors: Arpit Guleria, J. Harshan, Ranjitha Prasad, B. N. Bharath
• Subjects: cs.CR, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21192
• Pdf URL: https://arxiv.org/pdf/2410.21192
• Copy Paste: [[2410.21192]] On Homomorphic Encryption Based Strategies for Class Imbalance in Federated Learning(https://arxiv.org/abs/2410.21192)
• Keywords: privacy, federate
• Abstract: Class imbalance in training datasets can lead to bias and poor generalization in machine learning models. While pre-processing of training datasets can efficiently address both these issues in centralized learning environments, it is challenging to detect and address these issues in a distributed learning environment such as federated learning. In this paper, we propose FLICKER, a privacy preserving framework to address issues related to global class imbalance in federated learning. At the heart of our contribution lies the popular CKKS homomorphic encryption scheme, which is used by the clients to privately share their data attributes, and subsequently balance their datasets before implementing the FL scheme. Extensive experimental results show that our proposed method significantly improves the FL accuracy numbers when used along with popular datasets and relevant baselines.

Title: Belief in the Machine: Investigating Epistemological Blind Spots of Language Models

• Authors: Mirac Suzgun, Tayfun Gur, Federico Bianchi, Daniel E. Ho, Thomas Icard, Dan Jurafsky, James Zou
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.21195
• Pdf URL: https://arxiv.org/pdf/2410.21195
• Copy Paste: [[2410.21195]] Belief in the Machine: Investigating Epistemological Blind Spots of Language Models(https://arxiv.org/abs/2410.21195)
• Keywords: robust
• Abstract: As language models (LMs) become integral to fields like healthcare, law, and journalism, their ability to differentiate between fact, belief, and knowledge is essential for reliable decision-making. Failure to grasp these distinctions can lead to significant consequences in areas such as medical diagnosis, legal judgments, and dissemination of fake news. Despite this, current literature has largely focused on more complex issues such as theory of mind, overlooking more fundamental epistemic challenges. This study systematically evaluates the epistemic reasoning capabilities of modern LMs, including GPT-4, Claude-3, and Llama-3, using a new dataset, KaBLE, consisting of 13,000 questions across 13 tasks. Our results reveal key limitations. First, while LMs achieve 86% accuracy on factual scenarios, their performance drops significantly with false scenarios, particularly in belief-related tasks. Second, LMs struggle with recognizing and affirming personal beliefs, especially when those beliefs contradict factual data, which raises concerns for applications in healthcare and counseling, where engaging with a person's beliefs is critical. Third, we identify a salient bias in how LMs process first-person versus third-person beliefs, performing better on third-person tasks (80.7%) compared to first-person tasks (54.4%). Fourth, LMs lack a robust understanding of the factive nature of knowledge, namely, that knowledge inherently requires truth. Fifth, LMs rely on linguistic cues for fact-checking and sometimes bypass the deeper reasoning. These findings highlight significant concerns about current LMs' ability to reason about truth, belief, and knowledge while emphasizing the need for advancements in these areas before broad deployment in critical sectors.

Title: BongLLaMA: LLaMA for Bangla Language

• Authors: Abdullah Khan Zehady, Safi Al Mamun, Naymul Islam, Santu Karmaker
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21200
• Pdf URL: https://arxiv.org/pdf/2410.21200
• Copy Paste: [[2410.21200]] BongLLaMA: LLaMA for Bangla Language(https://arxiv.org/abs/2410.21200)
• Keywords: large language model
• Abstract: Bangla (or "Bengali") is a language spoken by approximately 240 million native speakers and around 300 million people worldwide. Despite being the 5th largest spoken language in the world, Bangla is still a "low-resource" language, and existing pretrained language models often struggle to perform well on Bangla Language Processing (BLP) tasks. This work addresses this gap by introducing BongLLaMA (i.e., Bangla-LLaMA), an open-source large language model fine-tuned exclusively on large Bangla corpora and instruction-tuning datasets. We present our methodology, data augmentation techniques, fine-tuning details, and comprehensive benchmarking results showcasing the utility of BongLLaMA on BLP tasks. We believe BongLLaMA will serve as the new standard baseline for Bangla Language Models and, thus, facilitate future benchmarking studies focused on this widely-spoken yet "low-resource" language. All BongLLaMA models are available for public use at this https URL.

Title: SeriesGAN: Time Series Generation via Adversarial and Autoregressive Learning

• Authors: MohammadReza EskandariNasab, Shah Muhammad Hamdi, Soukaina Filali Boubrahimi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21203
• Pdf URL: https://arxiv.org/pdf/2410.21203
• Copy Paste: [[2410.21203]] SeriesGAN: Time Series Generation via Adversarial and Autoregressive Learning(https://arxiv.org/abs/2410.21203)
• Keywords: generative
• Abstract: Current Generative Adversarial Network (GAN)-based approaches for time series generation face challenges such as suboptimal convergence, information loss in embedding spaces, and instability. To overcome these challenges, we introduce an advanced framework that integrates the advantages of an autoencoder-generated embedding space with the adversarial training dynamics of GANs. This method employs two discriminators: one to specifically guide the generator and another to refine both the autoencoder's and generator's output. Additionally, our framework incorporates a novel autoencoder-based loss function and supervision from a teacher-forcing supervisor network, which captures the stepwise conditional distributions of the data. The generator operates within the latent space, while the two discriminators work on latent and feature spaces separately, providing crucial feedback to both the generator and the autoencoder. By leveraging this dual-discriminator approach, we minimize information loss in the embedding space. Through joint training, our framework excels at generating high-fidelity time series data, consistently outperforming existing state-of-the-art benchmarks both qualitatively and quantitatively across a range of real and synthetic multivariate time series datasets.

Title: Exploring contextual modeling with linear complexity for point cloud segmentation

• Authors: Yong Xien Chng, Xuchong Qiu, Yizeng Han, Yifan Pu, Jiewei Cao, Gao Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21211
• Pdf URL: https://arxiv.org/pdf/2410.21211
• Copy Paste: [[2410.21211]] Exploring contextual modeling with linear complexity for point cloud segmentation(https://arxiv.org/abs/2410.21211)
• Keywords: robust, transformer, segmentation
• Abstract: Point cloud segmentation is an important topic in 3D understanding that has traditionally has been tackled using either the CNN or Transformer. Recently, Mamba has emerged as a promising alternative, offering efficient long-range contextual modeling capabilities without the quadratic complexity associated with Transformer's attention mechanisms. However, despite Mamba's potential, early efforts have all failed to achieve better performance than the best CNN-based and Transformer-based methods. In this work, we address this challenge by identifying the key components of an effective and efficient point cloud segmentation architecture. Specifically, we show that: 1) Spatial locality and robust contextual understanding are critical for strong performance, and 2) Mamba features linear computational complexity, offering superior data and inference efficiency compared to Transformers, while still being capable of delivering strong contextual understanding. Additionally, we further enhance the standard Mamba specifically for point cloud segmentation by identifying its two key shortcomings. First, the enforced causality in the original Mamba is unsuitable for processing point clouds that have no such dependencies. Second, its unidirectional scanning strategy imposes a directional bias, hampering its ability to capture the full context of unordered point clouds in a single pass. To address these issues, we carefully remove the causal convolutions and introduce a novel Strided Bidirectional SSM to enhance the model's capability to capture spatial relationships. Our efforts culminate in the development of a novel architecture named MEEPO, which effectively integrates the strengths of CNN and Mamba. MEEPO surpasses the previous state-of-the-art method, PTv3, by up to +0.8 mIoU on multiple key benchmark datasets, while being 42.1% faster and 5.53x more memory efficient.

Title: Reconstructing dynamics from sparse observations with no training on target system

• Authors: Zheng-Meng Zhai, Jun-Yin Huang, Benjamin D. Stern, Ying-Cheng Lai
• Subjects: cs.LG, nlin.CD, physics.data-an
• Abstract URL: https://arxiv.org/abs/2410.21222
• Pdf URL: https://arxiv.org/pdf/2410.21222
• Copy Paste: [[2410.21222]] Reconstructing dynamics from sparse observations with no training on target system(https://arxiv.org/abs/2410.21222)
• Keywords: transformer
• Abstract: In applications, an anticipated situation is where the system of interest has never been encountered before and sparse observations can be made only once. Can the dynamics be faithfully reconstructed from the limited observations without any training data? This problem defies any known traditional methods of nonlinear time-series analysis as well as existing machine-learning methods that typically require extensive data from the target system for training. We address this challenge by developing a hybrid transformer and reservoir-computing machine-learning scheme. The key idea is that, for a complex and nonlinear target system, the training of the transformer can be conducted not using any data from the target system, but with essentially unlimited synthetic data from known chaotic systems. The trained transformer is then tested with the sparse data from the target system. The output of the transformer is further fed into a reservoir computer for predicting the long-term dynamics or the attractor of the target system. The power of the proposed hybrid machine-learning framework is demonstrated using a large number of prototypical nonlinear dynamical systems, with high reconstruction accuracy even when the available data is only 20% of that required to faithfully represent the dynamical behavior of the underlying system. The framework provides a paradigm of reconstructing complex and nonlinear dynamics in the extreme situation where training data does not exist and the observations are random and sparse.

Title: LoRA vs Full Fine-tuning: An Illusion of Equivalence

• Authors: Reece Shuttleworth, Jacob Andreas, Antonio Torralba, Pratyusha Sharma
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21228
• Pdf URL: https://arxiv.org/pdf/2410.21228
• Copy Paste: [[2410.21228]] LoRA vs Full Fine-tuning: An Illusion of Equivalence(https://arxiv.org/abs/2410.21228)
• Keywords: robust, large language model
• Abstract: Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to match the performance of fully fine-tuned models on various tasks with an extreme reduction in the number of trainable parameters. Even in settings where both methods learn similarly accurate models, \emph{are their learned solutions really equivalent?} We study how different fine-tuning methods change pre-trained models by analyzing the model's weight matrices through the lens of their spectral properties. We find that full fine-tuning and LoRA yield weight matrices whose singular value decompositions exhibit very different structure; moreover, the fine-tuned models themselves show distinct generalization behaviors when tested outside the adaptation task's distribution. More specifically, we first show that the weight matrices trained with LoRA have new, high-ranking singular vectors, which we call \emph{intruder dimensions}. Intruder dimensions do not appear during full fine-tuning. Second, we show that LoRA models with intruder dimensions, despite achieving similar performance to full fine-tuning on the target task, become worse models of the pre-training distribution and adapt less robustly to multiple tasks sequentially. Higher-rank, rank-stabilized LoRA models closely mirror full fine-tuning, even when performing on par with lower-rank LoRA models on the same tasks. These results suggest that models updated with LoRA and full fine-tuning access different parts of parameter space, even when they perform equally on the fine-tuned distribution. We conclude by examining why intruder dimensions appear in LoRA fine-tuned models, why they are undesirable, and how their effects can be minimized.

Title: $\texttt{skwdro}$: a library for Wasserstein distributionally robust machine learning

• Authors: Florian Vincent, Waïss Azizian, Franck Iutzeler, Jérôme Malick
• Subjects: cs.LG, cs.MS, math.OC
• Abstract URL: https://arxiv.org/abs/2410.21231
• Pdf URL: https://arxiv.org/pdf/2410.21231
• Copy Paste: [[2410.21231]] $\texttt{skwdro}$: a library for Wasserstein distributionally robust machine learning(https://arxiv.org/abs/2410.21231)
• Keywords: robust
• Abstract: We present skwdro, a Python library for training robust machine learning models. The library is based on distributionally robust optimization using optimal transport distances. For ease of use, it features both scikit-learn compatible estimators for popular objectives, as well as a wrapper for PyTorch modules, enabling researchers and practitioners to use it in a wide range of models with minimal code changes. Its implementation relies on an entropic smoothing of the original robust objective in order to ensure maximal model flexibility. The library is available at this https URL

Title: Flaming-hot Initiation with Regular Execution Sampling for Large Language Models

• Authors: Weizhe Chen, Zhicheng Zhang, Guanlin Liu, Renjie Zheng, Wenlei Shi, Chen Dun, Zheng Wu, Xing Jin, Lin Yan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21236
• Pdf URL: https://arxiv.org/pdf/2410.21236
• Copy Paste: [[2410.21236]] Flaming-hot Initiation with Regular Execution Sampling for Large Language Models(https://arxiv.org/abs/2410.21236)
• Keywords: large language model
• Abstract: Since the release of ChatGPT, large language models (LLMs) have demonstrated remarkable capabilities across various domains. A key challenge in developing these general capabilities is efficiently sourcing diverse, high-quality data. This becomes especially critical in reasoning-related tasks with sandbox checkers, such as math or code, where the goal is to generate correct solutions to specific problems with higher probability. In this work, we introduce Flaming-hot Initiation with Regular Execution (FIRE) sampling, a simple yet highly effective method to efficiently find good responses. Our empirical findings show that FIRE sampling enhances inference-time generation quality and also benefits training in the alignment stage. Furthermore, we explore how FIRE sampling improves performance by promoting diversity and analyze the impact of employing FIRE at different positions within a response.

Title: LongReward: Improving Long-context Large Language Models with AI Feedback

• Authors: Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao, Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong, Ling Feng, Juanzi Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21252
• Pdf URL: https://arxiv.org/pdf/2410.21252
• Copy Paste: [[2410.21252]] LongReward: Improving Long-context Large Language Models with AI Feedback(https://arxiv.org/abs/2410.21252)
• Keywords: large language model
• Abstract: Though significant advancements have been achieved in developing long-context large language models (LLMs), the compromised quality of LLM-synthesized data for supervised fine-tuning (SFT) often affects the long-context performance of SFT models and leads to inherent limitations. In principle, reinforcement learning (RL) with appropriate reward signals can further enhance models' capacities. However, how to obtain reliable rewards in long-context scenarios remains unexplored. To this end, we propose LongReward, a novel method that utilizes an off-the-shelf LLM to provide rewards for long-context model responses from four human-valued dimensions: helpfulness, logicality, faithfulness, and completeness, each with a carefully designed assessment pipeline. By combining LongReward and offline RL algorithm DPO, we are able to effectively improve long-context SFT models. Our experiments indicate that LongReward not only significantly improves models' long-context performance but also enhances their ability to follow short instructions. We also find that long-context DPO with LongReward and conventional short-context DPO can be used together without hurting either one's performance.

Title: LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior

• Authors: Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, Abhinav Shrivastava
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21264
• Pdf URL: https://arxiv.org/pdf/2410.21264
• Copy Paste: [[2410.21264]] LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior(https://arxiv.org/abs/2410.21264)
• Keywords: transformer, generative, large language model
• Abstract: We present LARP, a novel video tokenizer designed to overcome limitations in current video tokenization methods for autoregressive (AR) generative models. Unlike traditional patchwise tokenizers that directly encode local visual patches into discrete tokens, LARP introduces a holistic tokenization scheme that gathers information from the visual content using a set of learned holistic queries. This design allows LARP to capture more global and semantic representations, rather than being limited to local patch-level information. Furthermore, it offers flexibility by supporting an arbitrary number of discrete tokens, enabling adaptive and efficient tokenization based on the specific requirements of the task. To align the discrete token space with downstream AR generation tasks, LARP integrates a lightweight AR transformer as a training-time prior model that predicts the next token on its discrete latent space. By incorporating the prior model during training, LARP learns a latent space that is not only optimized for video reconstruction but is also structured in a way that is more conducive to autoregressive generation. Moreover, this process defines a sequential order for the discrete tokens, progressively pushing them toward an optimal configuration during training, ensuring smoother and more accurate AR generation at inference time. Comprehensive experiments demonstrate LARP's strong performance, achieving state-of-the-art FVD on the UCF101 class-conditional video generation benchmark. LARP enhances the compatibility of AR models with videos and opens up the potential to build unified high-fidelity multimodal large language models (MLLMs).

Title: Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics

• Authors: Yaniv Nikankin, Anja Reusch, Aaron Mueller, Yonatan Belinkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21272
• Pdf URL: https://arxiv.org/pdf/2410.21272
• Copy Paste: [[2410.21272]] Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics(https://arxiv.org/abs/2410.21272)
• Keywords: robust, large language model
• Abstract: Do large language models (LLMs) solve reasoning tasks by learning robust generalizable algorithms, or do they memorize training data? To investigate this question, we use arithmetic reasoning as a representative task. Using causal analysis, we identify a subset of the model (a circuit) that explains most of the model's behavior for basic arithmetic logic and examine its functionality. By zooming in on the level of individual circuit neurons, we discover a sparse set of important neurons that implement simple heuristics. Each heuristic identifies a numerical input pattern and outputs corresponding answers. We hypothesize that the combination of these heuristic neurons is the mechanism used to produce correct arithmetic answers. To test this, we categorize each neuron into several heuristic types-such as neurons that activate when an operand falls within a certain range-and find that the unordered combination of these heuristic types is the mechanism that explains most of the model's accuracy on arithmetic prompts. Finally, we demonstrate that this mechanism appears as the main source of arithmetic accuracy early in training. Overall, our experimental results across several LLMs show that LLMs perform arithmetic using neither robust algorithms nor memorization; rather, they rely on a "bag of heuristics".

Title: On Inductive Biases That Enable Generalization of Diffusion Transformers

• Authors: Jie An, De Wang, Pengsheng Guo, Jiebo Luo, Alexander Schwing
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21273
• Pdf URL: https://arxiv.org/pdf/2410.21273
• Copy Paste: [[2410.21273]] On Inductive Biases That Enable Generalization of Diffusion Transformers(https://arxiv.org/abs/2410.21273)
• Keywords: diffusion, transformer
• Abstract: Recent work studying the generalization of diffusion models with UNet-based denoisers reveals inductive biases that can be expressed via geometry-adaptive harmonic bases. However, in practice, more recent denoising networks are often based on transformers, e.g., the diffusion transformer (DiT). This raises the question: do transformer-based denoising networks exhibit inductive biases that can also be expressed via geometry-adaptive harmonic bases? To our surprise, we find that this is not the case. This discrepancy motivates our search for the inductive bias that can lead to good generalization in DiT models. Investigating the pivotal attention modules of a DiT, we find that locality of attention maps are closely associated with generalization. To verify this finding, we modify the generalization of a DiT by restricting its attention windows. We inject local attention windows to a DiT and observe an improvement in generalization. Furthermore, we empirically find that both the placement and the effective attention size of these local attention windows are crucial factors. Experimental results on the CelebA, ImageNet, and LSUN datasets show that strengthening the inductive bias of a DiT can improve both generalization and generation quality when less training data is available. Source code will be released publicly upon paper publication. Project page: this http URL.

Title: Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Context

• Authors: Manuel Benavent-Lledo, David Mulero-Pérez, David Ortiz-Perez, Jose Garcia-Rodriguez, Antonis Argyros
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21275
• Pdf URL: https://arxiv.org/pdf/2410.21275
• Copy Paste: [[2410.21275]] Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Context(https://arxiv.org/abs/2410.21275)
• Keywords: transformer
• Abstract: The sequential execution of actions and their hierarchical structure consisting of different levels of abstraction, provide features that remain unexplored in the task of action recognition. In this study, we present a novel approach to improve action recognition by exploiting the hierarchical organization of actions and by incorporating contextualized textual information, including location and prior actions to reflect the sequential context. To achieve this goal, we introduce a novel transformer architecture tailored for action recognition that utilizes both visual and textual features. Visual features are obtained from RGB and optical flow data, while text embeddings represent contextual information. Furthermore, we define a joint loss function to simultaneously train the model for both coarse and fine-grained action recognition, thereby exploiting the hierarchical nature of actions. To demonstrate the effectiveness of our method, we extend the Toyota Smarthome Untrimmed (TSU) dataset to introduce action hierarchies, introducing the Hierarchical TSU dataset. We also conduct an ablation study to assess the impact of different methods for integrating contextual and hierarchical data on action recognition performance. Results show that the proposed approach outperforms pre-trained SOTA methods when trained with the same hyperparameters. Moreover, they also show a 17.12% improvement in top-1 accuracy over the equivalent fine-grained RGB version when using ground-truth contextual information, and a 5.33% improvement when contextual information is obtained from actual predictions.