2024-11-12

Title: A Comprehensive Survey of Time Series Forecasting: Architectural Diversity and Open Challenges

• Authors: Jongseon Kim (1 and 3), Hyungjoon Kim (1 and 4), HyunGi Kim (2), Dongjun Lee (1), Sungroh Yoon (1 and 2) ((1) Interdisciplinary Program in Artificial Intelligence, Seoul National University, (2) Department of Electrical and Computer Engineering, Seoul National University, (3) R&D Department, LG Chem, (4) R&D Department, Samsung SDI)
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05793
• Pdf URL: https://arxiv.org/pdf/2411.05793
• Copy Paste: [[2411.05793]] A Comprehensive Survey of Time Series Forecasting: Architectural Diversity and Open Challenges(https://arxiv.org/abs/2411.05793)
• Keywords: extraction, diffusion, transformer
• Abstract: Time series forecasting is a critical task that provides key information for decision-making across various fields. Recently, various fundamental deep learning architectures such as MLPs, CNNs, RNNs, and GNNs have been developed and applied to solve time series forecasting problems. However, the structural limitations caused by the inductive biases of each deep learning architecture constrained their performance. Transformer models, which excel at handling long-term dependencies, have become significant architectural components for time series forecasting. However, recent research has shown that alternatives such as simple linear layers can outperform Transformers. These findings have opened up new possibilities for using diverse architectures. In this context of exploration into various models, the architectural modeling of time series forecasting has now entered a renaissance. This survey not only provides a historical context for time series forecasting but also offers comprehensive and timely analysis of the movement toward architectural diversification. By comparing and re-examining various deep learning models, we uncover new perspectives and presents the latest trends in time series forecasting, including the emergence of hybrid models, diffusion models, Mamba models, and foundation models. By focusing on the inherent characteristics of time series data, we also address open challenges that have gained attention in time series forecasting, such as channel dependency, distribution shift, causality, and feature extraction. This survey explores vital elements that can enhance forecasting performance through diverse approaches. These contributions lead to lowering the entry barriers for newcomers to the field of time series forecasting, while also offering seasoned researchers broad perspectives, new opportunities, and deep insights.

Title: Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives

• Authors: Vincent Hanke, Tom Blanchard, Franziska Boenisch, Iyiola Emmanuel Olatunji, Michael Backes, Adam Dziedzic
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05818
• Pdf URL: https://arxiv.org/pdf/2411.05818
• Copy Paste: [[2411.05818]] Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives(https://arxiv.org/abs/2411.05818)
• Keywords: privacy, protect, large language model
• Abstract: While open Large Language Models (LLMs) have made significant progress, they still fall short of matching the performance of their closed, proprietary counterparts, making the latter attractive even for the use on highly private data. Recently, various new methods have been proposed to adapt closed LLMs to private data without leaking private information to third parties and/or the LLM provider. In this work, we analyze the privacy protection and performance of the four most recent methods for private adaptation of closed LLMs. By examining their threat models and thoroughly comparing their performance under different privacy levels according to differential privacy (DP), various LLM architectures, and multiple datasets for classification and generation tasks, we find that: (1) all the methods leak query data, i.e., the (potentially sensitive) user data that is queried at inference time, to the LLM provider, (2) three out of four methods also leak large fractions of private training data to the LLM provider while the method that protects private data requires a local open LLM, (3) all the methods exhibit lower performance compared to three private gradient-based adaptation methods for local open LLMs, and (4) the private adaptation methods for closed LLMs incur higher monetary training and query costs than running the alternative methods on local open LLMs. This yields the conclusion that, to achieve truly privacy-preserving LLM adaptations that yield high performance and more privacy at lower costs, taking into account current methods and models, one should use open LLMs.

Title: Hierarchical Sentiment Analysis Framework for Hate Speech Detection: Implementing Binary and Multiclass Classification Strategy

• Authors: Faria Naznin, Md Touhidur Rahman, Shahran Rahman Alve
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05819
• Pdf URL: https://arxiv.org/pdf/2411.05819
• Copy Paste: [[2411.05819]] Hierarchical Sentiment Analysis Framework for Hate Speech Detection: Implementing Binary and Multiclass Classification Strategy(https://arxiv.org/abs/2411.05819)
• Keywords: transformer
• Abstract: A significant challenge in automating hate speech detection on social media is distinguishing hate speech from regular and offensive language. These identify an essential category of content that web filters seek to remove. Only automated methods can manage this volume of daily data. To solve this problem, the community of Natural Language Processing is currently investigating different ways of hate speech detection. In addition to those, previous approaches (e.g., Convolutional Neural Networks, multi-channel BERT models, and lexical detection) have always achieved low precision without carefully treating other related tasks like sentiment analysis and emotion classification. They still like to group all messages with specific words in them as hate speech simply because those terms often appear alongside hateful rhetoric. In this research, our paper presented the hate speech text classification system model drawn upon deep learning and machine learning. In this paper, we propose a new multitask model integrated with shared emotional representations to detect hate speech across the English language. The Transformer-based model we used from Hugging Face and sentiment analysis helped us prevent false positives. Conclusion. We conclude that utilizing sentiment analysis and a Transformer-based trained model considerably improves hate speech detection across multiple datasets.

Title: SPACE: SPAtial-aware Consistency rEgularization for anomaly detection in Industrial applications

• Authors: Daehwan Kim, Hyungmin Kim, Daun Jeong, Sungho Suh, Hansang Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05822
• Pdf URL: https://arxiv.org/pdf/2411.05822
• Copy Paste: [[2411.05822]] SPACE: SPAtial-aware Consistency rEgularization for anomaly detection in Industrial applications(https://arxiv.org/abs/2411.05822)
• Keywords: protect, robust
• Abstract: In this paper, we propose SPACE, a novel anomaly detection methodology that integrates a Feature Encoder (FE) into the structure of the Student-Teacher method. The proposed method has two key elements: Spatial Consistency regularization Loss (SCL) and Feature converter Module (FM). SCL prevents overfitting in student models by avoiding excessive imitation of the teacher model. Simultaneously, it facilitates the expansion of normal data features by steering clear of abnormal areas generated through data augmentation. This dual functionality ensures a robust boundary between normal and abnormal data. The FM prevents the learning of ambiguous information from the FE. This protects the learned features and enables more effective detection of structural and logical anomalies. Through these elements, SPACE is available to minimize the influence of the FE while integrating various data this http URL this study, we evaluated the proposed method on the MVTec LOCO, MVTec AD, and VisA datasets. Experimental results, through qualitative evaluation, demonstrate the superiority of detection and efficiency of each module compared to state-of-the-art methods.

Title: FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Models

• Authors: Zhanwei Zhang, Shizhao Sun, Wenxiao Wang, Deng Cai, Jiang Bian
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.05823
• Pdf URL: https://arxiv.org/pdf/2411.05823
• Copy Paste: [[2411.05823]] FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Models(https://arxiv.org/abs/2411.05823)
• Keywords: large language model
• Abstract: Recently, there is a growing interest in creating computer-aided design (CAD) models based on user intent, known as controllable CAD generation. Existing work offers limited controllability and needs separate models for different types of control, reducing efficiency and practicality. To achieve controllable generation across all CAD construction hierarchies, such as sketch-extrusion, extrusion, sketch, face, loop and curve, we propose FlexCAD, a unified model by fine-tuning large language models (LLMs). First, to enhance comprehension by LLMs, we represent a CAD model as a structured text by abstracting each hierarchy as a sequence of text tokens. Second, to address various controllable generation tasks in a unified model, we introduce a hierarchy-aware masking strategy. Specifically, during training, we mask a hierarchy-aware field in the CAD text with a mask token. This field, composed of a sequence of tokens, can be set flexibly to represent various hierarchies. Subsequently, we ask LLMs to predict this masked field. During inference, the user intent is converted into a CAD text with a mask token replacing the part the user wants to modify, which is then fed into FlexCAD to generate new CAD models. Comprehensive experiments on public dataset demonstrate the effectiveness of FlexCAD in both generation quality and controllability. Code will be available at this https URL.

Title: From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing

• Authors: Xintian Sun, Benji Peng, Charles Zhang, Fei Jin, Qian Niu, Junyu Liu, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Ming Liu, Yichao Zhang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05826
• Pdf URL: https://arxiv.org/pdf/2411.05826
• Copy Paste: [[2411.05826]] From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing(https://arxiv.org/abs/2411.05826)
• Keywords: transformer
• Abstract: Remote sensing has evolved from simple image acquisition to complex systems capable of integrating and processing visual and textual data. This review examines the development and application of multi-modal language models (MLLMs) in remote sensing, focusing on their ability to interpret and describe satellite imagery using natural language. We cover the technical underpinnings of MLLMs, including dual-encoder architectures, Transformer models, self-supervised and contrastive learning, and cross-modal integration. The unique challenges of remote sensing data--varying spatial resolutions, spectral richness, and temporal changes--are analyzed for their impact on MLLM performance. Key applications such as scene description, object detection, change detection, text-to-image retrieval, image-to-text generation, and visual question answering are discussed to demonstrate their relevance in environmental monitoring, urban planning, and disaster response. We review significant datasets and resources supporting the training and evaluation of these models. Challenges related to computational demands, scalability, data quality, and domain adaptation are highlighted. We conclude by proposing future research directions and technological advancements to further enhance MLLM utility in remote sensing.

Title: A Theory of Stabilization by Skull Carving

• Authors: Mathieu Lamarre, Patrick Anderson, Étienne Danvoye
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05827
• Pdf URL: https://arxiv.org/pdf/2411.05827
• Copy Paste: [[2411.05827]] A Theory of Stabilization by Skull Carving(https://arxiv.org/abs/2411.05827)
• Keywords: robust
• Abstract: Accurate stabilization of facial motion is essential for applications in photoreal avatar construction for 3D games, virtual reality, movies, and training data collection. For the latter, stabilization must work automatically for the general population with people of varying morphology. Distinguishing rigid skull motion from facial expressions is critical since misalignment between skull motion and facial expressions can lead to animation models that are hard to control and can not fit natural motion. Existing methods struggle to work with sparse sets of very different expressions, such as when combining multiple units from the Facial Action Coding System (FACS). Certain approaches are not robust enough, some depend on motion data to find stable points, while others make one-for-all invalid physiological assumptions. In this paper, we leverage recent advances in neural signed distance fields and differentiable isosurface meshing to compute skull stabilization rigid transforms directly on unstructured triangle meshes or point clouds, significantly enhancing accuracy and robustness. We introduce the concept of a stable hull as the surface of the boolean intersection of stabilized scans, analogous to the visual hull in shape-from-silhouette and the photo hull from space carving. This hull resembles a skull overlaid with minimal soft tissue thickness, upper teeth are automatically included. Our skull carving algorithm simultaneously optimizes the stable hull shape and rigid transforms to get accurate stabilization of complex expressions for large diverse sets of people, outperforming existing methods.

Title: Prion-ViT: Prions-Inspired Vision Transformers for Temperature prediction with Specklegrams

• Authors: Abhishek Sebastian, Pragna R
• Subjects: cs.CV, eess.SP, physics.optics
• Abstract URL: https://arxiv.org/abs/2411.05836
• Pdf URL: https://arxiv.org/pdf/2411.05836
• Copy Paste: [[2411.05836]] Prion-ViT: Prions-Inspired Vision Transformers for Temperature prediction with Specklegrams(https://arxiv.org/abs/2411.05836)
• Keywords: robust, transformer
• Abstract: Fiber Specklegram Sensors (FSS) are widely used in environmental monitoring due to their high sensitivity to temperature fluctuations, yet the complex, nonlinear nature of specklegram data poses significant challenges for conventional predictive models. This study introduces a novel Prion-Vision Transformer (Prion-ViT) model, inspired by biological prion memory mechanisms, to enhance long-term dependency modeling for accurate temperature prediction using FSS data. By leveraging a persistent memory state, the Prion-ViT effectively retains and propagates essential features across multiple layers, thereby improving prediction accuracy and reducing mean absolute error (MAE) to "0.52 Degree Celsius" outperforming traditional models like ResNet, Inception Net V2, and existing transformer-based architectures. The study addresses the specific challenges of applying Vision Transformers (ViTs) to FSS data and demonstrates that the prion-inspired memory mechanism offers a robust solution for capturing complex optical interference patterns in specklegrams. These findings establish Prion-ViT as a promising advancement for real-time industrial temperature monitoring applications, with potential applicability to other optical sensing domains.

Title: StegaVision: Enhancing Steganography with Attention Mechanism

• Authors: Abhinav Kumar, Pratham Singla, Aayan Yadav
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05838
• Pdf URL: https://arxiv.org/pdf/2411.05838
• Copy Paste: [[2411.05838]] StegaVision: Enhancing Steganography with Attention Mechanism(https://arxiv.org/abs/2411.05838)
• Keywords: security
• Abstract: Image steganography is the technique of embedding secret information within images. The development of deep learning has led to significant advances in this field. However, existing methods often struggle to balance image quality, embedding capacity, and security. This paper proposes a novel approach to image steganography by enhancing an encoder-decoder architecture with attention mechanisms, specifically focusing on channel and spatial attention modules. We systematically investigate five configurations: (1) channel attention, (2) spatial attention, (3) sequential channel followed by spatial attention, (4) spatial attention followed by channel attention and (5) parallel channel and spatial attention. Our experiments show that adding attention mechanisms improves the ability to embed hidden information while maintaining the visual quality of the images. The increase in the PSNR and SSIM scores shows that using a parallel combination of channel and spatial attention improves image quality and hiding capacity simultaneously. This is in contrast to previous works where there is a tradeoff between them. This study shows that attention mechanisms in image steganography lead to better hiding of secret information. Our code is available at this https URL.

Title: FLEXtime: Filterbank learning for explaining time series

• Authors: Thea Brüsch, Kristoffer K. Wickstrøm, Mikkel N. Schmidt, Robert Jenssen, Tommy S. Alstrøm
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05841
• Pdf URL: https://arxiv.org/pdf/2411.05841
• Copy Paste: [[2411.05841]] FLEXtime: Filterbank learning for explaining time series(https://arxiv.org/abs/2411.05841)
• Keywords: explainability
• Abstract: State-of-the-art methods for explaining predictions based on time series are built on learning an instance-wise saliency mask for each time step. However, for many types of time series, the salient information is found in the frequency domain. Adopting existing methods to the frequency domain involves naively zeroing out frequency content in the signals, which goes against established signal processing theory. Therefore, we propose a new method entitled FLEXtime, that uses a filterbank to split the time series into frequency bands and learns the optimal combinations of these bands. FLEXtime avoids the drawbacks of zeroing out frequency bins and is more stable and easier to train compared to the naive method. Our extensive evaluation shows that FLEXtime on average outperforms state-of-the-art explainability methods across a range of datasets. FLEXtime fills an important gap in the time series explainability literature and can provide a valuable tool for a wide range of time series like EEG and audio.

Title: Reducing catastrophic forgetting of incremental learning in the absence of rehearsal memory with task-specific token

• Authors: Young Jo Choi, Min Kyoon Yoo, Yu Rang Park
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05846
• Pdf URL: https://arxiv.org/pdf/2411.05846
• Copy Paste: [[2411.05846]] Reducing catastrophic forgetting of incremental learning in the absence of rehearsal memory with task-specific token(https://arxiv.org/abs/2411.05846)
• Keywords: security, privacy, transformer
• Abstract: Deep learning models generally display catastrophic forgetting when learning new data continuously. Many incremental learning approaches address this problem by reusing data from previous tasks while learning new tasks. However, the direct access to past data generates privacy and security concerns. To address these issues, we present a novel method that preserves previous knowledge without storing previous data. This method is inspired by the architecture of a vision transformer and employs a unique token capable of encapsulating the compressed knowledge of each task. This approach generates task-specific embeddings by directing attention differently based on the task associated with the data, thereby effectively mimicking the impact of having multiple models through tokens. Our method incorporates a distillation process that ensures efficient interactions even after multiple additional learning steps, thereby optimizing the model against forgetting. We measured the performance of our model in terms of accuracy and backward transfer using a benchmark dataset for different task-incremental learning scenarios. Our results demonstrate the superiority of our approach, which achieved the highest accuracy and lowest backward transfer among the compared methods. In addition to presenting a new model, our approach lays the foundation for various extensions within the spectrum of vision-transformer architectures.

Title: Multivariate Data Augmentation for Predictive Maintenance using Diffusion

• Authors: Andrew Thompson, Alexander Sommers, Alicia Russell-Gilbert, Logan Cummins, Sudip Mittal, Shahram Rahimi, Maria Seale, Joseph Jaboure, Thomas Arnold, Joshua Church
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05848
• Pdf URL: https://arxiv.org/pdf/2411.05848
• Copy Paste: [[2411.05848]] Multivariate Data Augmentation for Predictive Maintenance using Diffusion(https://arxiv.org/abs/2411.05848)
• Keywords: diffusion
• Abstract: Predictive maintenance has been used to optimize system repairs in the industrial, medical, and financial domains. This technique relies on the consistent ability to detect and predict anomalies in critical systems. AI models have been trained to detect system faults, improving predictive maintenance efficiency. Typically there is a lack of fault data to train these models, due to organizations working to keep fault occurrences and down time to a minimum. For newly installed systems, no fault data exists since they have yet to fail. By using diffusion models for synthetic data generation, the complex training datasets for these predictive models can be supplemented with high level synthetic fault data to improve their performance in anomaly detection. By learning the relationship between healthy and faulty data in similar systems, a diffusion model can attempt to apply that relationship to healthy data of a newly installed system that has no fault data. The diffusion model would then be able to generate useful fault data for the new system, and enable predictive models to be trained for predictive maintenance. The following paper demonstrates a system for generating useful, multivariate synthetic data for predictive maintenance, and how it can be applied to systems that have yet to fail.

Title: Saliency Assisted Quantization for Neural Networks

• Authors: Elmira Mousa Rezabeyk, Salar Beigzad, Yasin Hamzavi, Mohsen Bagheritabar, Seyedeh Sogol Mirikhoozani
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05858
• Pdf URL: https://arxiv.org/pdf/2411.05858
• Copy Paste: [[2411.05858]] Saliency Assisted Quantization for Neural Networks(https://arxiv.org/abs/2411.05858)
• Keywords: interpretability
• Abstract: Deep learning methods have established a significant place in image classification. While prior research has focused on enhancing final outcomes, the opaque nature of the decision-making process in these models remains a concern for experts. Additionally, the deployment of these methods can be problematic in resource-limited environments. This paper tackles the inherent black-box nature of these models by providing real-time explanations during the training phase, compelling the model to concentrate on the most distinctive and crucial aspects of the input. Furthermore, we employ established quantization techniques to address resource constraints. To assess the effectiveness of our approach, we explore how quantization influences the interpretability and accuracy of Convolutional Neural Networks through a comparative analysis of saliency maps from standard and quantized models. Quantization is implemented during the training phase using the Parameterized Clipping Activation method, with a focus on the MNIST and FashionMNIST benchmark datasets. We evaluated three bit-width configurations (2-bit, 4-bit, and mixed 4/2-bit) to explore the trade-off between efficiency and interpretability, with each configuration designed to highlight varying impacts on saliency map clarity and model accuracy. The results indicate that while quantization is crucial for implementing models on resource-limited devices, it necessitates a trade-off between accuracy and interpretability. Lower bit-widths result in more pronounced reductions in both metrics, highlighting the necessity of meticulous quantization parameter selection in applications where model transparency is paramount. The study underscores the importance of achieving a balance between efficiency and interpretability in the deployment of neural networks.

Title: Enhancing Financial Fraud Detection with Human-in-the-Loop Feedback and Feedback Propagation

• Authors: Prashank Kadam
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2411.05859
• Pdf URL: https://arxiv.org/pdf/2411.05859
• Copy Paste: [[2411.05859]] Enhancing Financial Fraud Detection with Human-in-the-Loop Feedback and Feedback Propagation(https://arxiv.org/abs/2411.05859)
• Keywords: robust, interpretability
• Abstract: Human-in-the-loop (HITL) feedback mechanisms can significantly enhance machine learning models, particularly in financial fraud detection, where fraud patterns change rapidly, and fraudulent nodes are sparse. Even small amounts of feedback from Subject Matter Experts (SMEs) can notably boost model performance. This paper examines the impact of HITL feedback on both traditional and advanced techniques using proprietary and publicly available datasets. Our results show that HITL feedback improves model accuracy, with graph-based techniques benefiting the most. We also introduce a novel feedback propagation method that extends feedback across the dataset, further enhancing detection accuracy. By leveraging human expertise, this approach addresses challenges related to evolving fraud patterns, data sparsity, and model interpretability, ultimately improving model robustness and streamlining the annotation process.

Title: Conditional Diffusion Model for Longitudinal Medical Image Generation

• Authors: Duy-Phuong Dao, Hyung-Jeong Yang, Jahae Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05860
• Pdf URL: https://arxiv.org/pdf/2411.05860
• Copy Paste: [[2411.05860]] Conditional Diffusion Model for Longitudinal Medical Image Generation(https://arxiv.org/abs/2411.05860)
• Keywords: diffusion
• Abstract: Alzheimers disease progresses slowly and involves complex interaction between various biological factors. Longitudinal medical imaging data can capture this progression over time. However, longitudinal data frequently encounter issues such as missing data due to patient dropouts, irregular follow-up intervals, and varying lengths of observation periods. To address these issues, we designed a diffusion-based model for 3D longitudinal medical imaging generation using single magnetic resonance imaging (MRI). This involves the injection of a conditioning MRI and time-visit encoding to the model, enabling control in change between source and target images. The experimental results indicate that the proposed method generates higher-quality images compared to other competing methods.

Title: Dialectal Coverage And Generalization in Arabic Speech Recognition

• Authors: Amirbek Djanibekov, Hawau Olamide Toyin, Raghad Alshalan, Abdullah Alitr, Hanan Aldarmaki
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.05872
• Pdf URL: https://arxiv.org/pdf/2411.05872
• Copy Paste: [[2411.05872]] Dialectal Coverage And Generalization in Arabic Speech Recognition(https://arxiv.org/abs/2411.05872)
• Keywords: robust
• Abstract: Developing robust automatic speech recognition (ASR) systems for Arabic, a language characterized by its rich dialectal diversity and often considered a low-resource language in speech technology, demands effective strategies to manage its complexity. This study explores three critical factors influencing ASR performance: the role of dialectal coverage in pre-training, the effectiveness of dialect-specific fine-tuning compared to a multi-dialectal approach, and the ability to generalize to unseen dialects. Through extensive experiments across different dialect combinations, our findings offer key insights towards advancing the development of ASR systems for pluricentric languages like Arabic.

Title: Interplay between Federated Learning and Explainable Artificial Intelligence: a Scoping Review

• Authors: Luis M. Lopez-Ramos, Florian Leiser, Aditya Rastogi, Steven Hicks, Inga Strümke, Vince I. Madai, Tobias Budig, Ali Sunyaev, Adam Hilbert
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05874
• Pdf URL: https://arxiv.org/pdf/2411.05874
• Copy Paste: [[2411.05874]] Interplay between Federated Learning and Explainable Artificial Intelligence: a Scoping Review(https://arxiv.org/abs/2411.05874)
• Keywords: privacy, federate, interpretability
• Abstract: The joint implementation of Federated learning (FL) and Explainable artificial intelligence (XAI) will allow training models from distributed data and explaining their inner workings while preserving important aspects of privacy. Towards establishing the benefits and tensions associated with their interplay, this scoping review maps those publications that jointly deal with FL and XAI, focusing on publications where an interplay between FL and model interpretability or post-hoc explanations was found. In total, 37 studies met our criteria, with more papers focusing on explanation methods (mainly feature relevance) than on interpretability (mainly algorithmic transparency). Most works used simulated horizontal FL setups involving 10 or fewer data centers. Only one study explicitly and quantitatively analyzed the influence of FL on model explanations, revealing a significant research gap. Aggregation of interpretability metrics across FL nodes created generalized global insights at the expense of node-specific patterns being diluted. 8 papers addressed the benefits of incorporating explanation methods as a component of the FL algorithm. Studies using established FL libraries or following reporting guidelines are a minority. More quantitative research and structured, transparent practices are needed to fully understand their mutual impact and under which conditions it happens.

Title: Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

• Authors: Zhuotong Chen, Fang Liu, Jennifer Zhu, Wanyu Du, Yanjun Qi
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05875
• Pdf URL: https://arxiv.org/pdf/2411.05875
• Copy Paste: [[2411.05875]] Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization(https://arxiv.org/abs/2411.05875)
• Keywords: large language model
• Abstract: Direct Preference Optimization (DPO) and its variants have become the de facto standards for aligning large language models (LLMs) with human preferences or specific goals. However, DPO requires high-quality preference data and suffers from unstable preference optimization. In this work, we aim to improve the preference optimization pipeline by taking a closer look at preference data generation and training regularization techniques. For preference data generation, we demonstrate that existing scoring-based reward models produce unsatisfactory preference data and perform poorly on out-of-distribution tasks. This significantly impacts the LLM alignment performance when using these data for preference tuning. To ensure high-quality preference data generation, we propose an iterative pairwise ranking mechanism that derives preference ranking of completions using pairwise comparison signals. For training regularization, we observe that preference optimization tends to achieve better convergence when the LLM predicted likelihood of preferred samples gets slightly reduced. However, the widely used supervised next-word prediction regularization strictly prevents any likelihood reduction of preferred samples. This observation motivates our design of a budget-controlled regularization formulation. Empirically we show that combining the two designs leads to aligned models that surpass existing SOTA across two popular benchmarks.

Title: Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

• Authors: Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, Hao Cheng
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.05877
• Pdf URL: https://arxiv.org/pdf/2411.05877
• Copy Paste: [[2411.05877]] Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass(https://arxiv.org/abs/2411.05877)
• Keywords: generative, large language model
• Abstract: Large language models (LMs) are typically adapted to improve performance on new contexts (\eg text prompts that define new tasks or domains) through fine-tuning or prompting. However, there is an accuracy compute tradeoff -- fine-tuning incurs significant training cost and prompting increases inference overhead. We introduce $GenerativeAdapter$, an effective and efficient adaptation method that directly maps new contexts to low-rank LM adapters, thereby significantly reducing inference overhead with no need for finetuning. The adapter generator is trained via self-supervised learning, and can be used to adapt a single frozen LM for any new task simply by mapping the associated task or domain context to a new adapter. We apply $GenerativeAdapter$ to two pretrained LMs (Mistral-7B-Instruct and Llama2-7B-Chat) and evaluate the adapted models in three adaption scenarios: knowledge acquisition from documents, learning from demonstrations, and personalization for users. In StreamingQA, our approach is effective in injecting knowledge into the LM's parameters, achieving a 63.5% improvement in F1 score over the model with supervised fine-tuning (from $19.5$ to $31.5$) for contexts as long as 32K tokens. In the MetaICL in-context learning evaluation, our method achieves an average accuracy of $44.9$ across 26 tasks, outperforming the base model. On MSC, our method proves to be highly competitive in memorizing user information from conversations with a 4x reduction in computation and memory costs compared to prompting with full conversation history. Together, these results suggest that $GenerativeAdapter$ should allow for general adaption to a wide range of different contexts.

Title: Joint-Optimized Unsupervised Adversarial Domain Adaptation in Remote Sensing Segmentation with Prompted Foundation Model

• Authors: Shuchang Lyu, Qi Zhaoa, Guangliang Cheng, Yiwei He, Zheng Zhou, Guangbiao Wang, Zhenwei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05878
• Pdf URL: https://arxiv.org/pdf/2411.05878
• Copy Paste: [[2411.05878]] Joint-Optimized Unsupervised Adversarial Domain Adaptation in Remote Sensing Segmentation with Prompted Foundation Model(https://arxiv.org/abs/2411.05878)
• Keywords: robust, interpretability, segmentation
• Abstract: Unsupervised Domain Adaptation for Remote Sensing Semantic Segmentation (UDA-RSSeg) addresses the challenge of adapting a model trained on source domain data to target domain samples, thereby minimizing the need for annotated data across diverse remote sensing scenes. This task presents two principal challenges: (1) severe inconsistencies in feature representation across different remote sensing domains, and (2) a domain gap that emerges due to the representation bias of source domain patterns when translating features to predictive logits. To tackle these issues, we propose a joint-optimized adversarial network incorporating the "Segment Anything Model (SAM) (SAM-JOANet)" for UDA-RSSeg. Our approach integrates SAM to leverage its robust generalized representation capabilities, thereby alleviating feature inconsistencies. We introduce a finetuning decoder designed to convert SAM-Encoder features into predictive logits. Additionally, a feature-level adversarial-based prompted segmentor is employed to generate class-agnostic maps, which guide the finetuning decoder's feature representations. The network is optimized end-to-end, combining the prompted segmentor and the finetuning decoder. Extensive evaluations on benchmark datasets, including ISPRS (Potsdam/Vaihingen) and CITY-OSM (Paris/Chicago), demonstrate the effectiveness of our method. The results, supported by visualization and analysis, confirm the method's interpretability and robustness. The code of this paper is available at this https URL.

Title: Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors

• Authors: Yuanyuan Liu, Lin Wei, Kejun Liu, Yibing Zhan, Zijing Chen, Zhe Chen, Shiguang Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05879
• Pdf URL: https://arxiv.org/pdf/2411.05879
• Copy Paste: [[2411.05879]] Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors(https://arxiv.org/abs/2411.05879)
• Keywords: robust, transformer
• Abstract: Emotion Recognition (ER) is the process of identifying human emotions from given data. Currently, the field heavily relies on facial expression recognition (FER) because facial expressions contain rich emotional cues. However, it is important to note that facial expressions may not always precisely reflect genuine emotions and FER-based results may yield misleading ER. To understand and bridge this gap between FER and ER, we introduce eye behaviors as an important emotional cues for the creation of a new Eye-behavior-aided Multimodal Emotion Recognition (EMER) dataset. Different from existing multimodal ER datasets, the EMER dataset employs a stimulus material-induced spontaneous emotion generation method to integrate non-invasive eye behavior data, like eye movements and eye fixation maps, with facial videos, aiming to obtain natural and accurate human emotions. Notably, for the first time, we provide annotations for both ER and FER in the EMER, enabling a comprehensive analysis to better illustrate the gap between both tasks. Furthermore, we specifically design a new EMERT architecture to concurrently enhance performance in both ER and FER by efficiently identifying and bridging the emotion gap between the this http URL, our EMERT employs modality-adversarial feature decoupling and multi-task Transformer to augment the modeling of eye behaviors, thus providing an effective complement to facial expressions. In the experiment, we introduce seven multimodal benchmark protocols for a variety of comprehensive evaluations of the EMER dataset. The results show that the EMERT outperforms other state-of-the-art multimodal methods by a great margin, revealing the importance of modeling eye behaviors for robust ER. To sum up, we provide a comprehensive analysis of the importance of eye behaviors in ER, advancing the study on addressing the gap between FER and ER for more robust ER performance.

Title: Towards Equitable ASD Diagnostics: A Comparative Study of Machine and Deep Learning Models Using Behavioral and Facial Data

• Authors: Mohammed Aledhari, Mohamed Rahouti, Ali Alfatemi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05880
• Pdf URL: https://arxiv.org/pdf/2411.05880
• Copy Paste: [[2411.05880]] Towards Equitable ASD Diagnostics: A Comparative Study of Machine and Deep Learning Models Using Behavioral and Facial Data(https://arxiv.org/abs/2411.05880)
• Keywords: robust, explainability
• Abstract: Autism Spectrum Disorder (ASD) is often underdiagnosed in females due to gender-specific symptom differences overlooked by conventional diagnostics. This study evaluates machine learning models, particularly Random Forest and convolutional neural networks, for enhancing ASD diagnosis through structured data and facial image analysis. Random Forest achieved 100% validation accuracy across datasets, highlighting its ability to manage complex relationships and reduce false negatives, which is crucial for early intervention and addressing gender biases. In image-based analysis, MobileNet outperformed the baseline CNN, achieving 87% accuracy, though a 30% validation loss suggests possible overfitting, requiring further optimization for robustness in clinical settings. Future work will emphasize hyperparameter tuning, regularization, and transfer learning. Integrating behavioral data with facial analysis could improve diagnosis for underdiagnosed groups. These findings suggest Random Forest's high accuracy and balanced precision-recall metrics could enhance clinical workflows. MobileNet's lightweight structure also shows promise for resource-limited environments, enabling accessible ASD screening. Addressing model explainability and clinician trust will be vital.

Title: When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization

• Authors: Jacob Nielsen, Lukas Galke, Peter Schneider-Kamp
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05882
• Pdf URL: https://arxiv.org/pdf/2411.05882
• Copy Paste: [[2411.05882]] When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization(https://arxiv.org/abs/2411.05882)
• Keywords: transformer
• Abstract: Contemporary machine learning models, such as language models, are powerful, but come with immense resource requirements both at training and inference time. It has been shown that decoder-only language models can be trained to a competitive state with ternary weights (1.58 bits per weight), facilitating efficient inference. Here, we start our exploration with non-transformer model architectures, investigating 1.58-bit training for multi-layer perceptrons and graph neural networks. Then, we explore 1.58-bit training in other transformer-based language models, namely encoder-only and encoder-decoder models. Our results show that in all of these settings, 1.58-bit training is on par with or sometimes even better than the standard 32/16-bit models.

Title: Predictive Digital Twin for Condition Monitoring Using Thermal Imaging

• Authors: Daniel Menges, Florian Stadtmann, Henrik Jordheim, Adil Rasheed
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05887
• Pdf URL: https://arxiv.org/pdf/2411.05887
• Copy Paste: [[2411.05887]] Predictive Digital Twin for Condition Monitoring Using Thermal Imaging(https://arxiv.org/abs/2411.05887)
• Keywords: robust
• Abstract: This paper explores the development and practical application of a predictive digital twin specifically designed for condition monitoring, using advanced mathematical models and thermal imaging techniques. Our work presents a comprehensive approach to integrating Proper Orthogonal Decomposition (POD), Robust Principal Component Analysis (RPCA), and Dynamic Mode Decomposition (DMD) to establish a robust predictive digital twin framework. We employ these methods in a real-time experimental setup involving a heated plate monitored through thermal imaging. This system effectively demonstrates the digital twin's capabilities in real-time predictions, condition monitoring, and anomaly detection. Additionally, we introduce the use of a human-machine interface that includes virtual reality, enhancing user interaction and system understanding. The primary contributions of our research lie in the demonstration of these advanced techniques in a tangible setup, showcasing the potential of digital twins to transform industry practices by enabling more proactive and strategic asset management.

Title: Sdn Intrusion Detection Using Machine Learning Method

• Authors: Muhammad Zawad Mahmud, Shahran Rahman Alve, Samiha Islam, Mohammad Monirujjaman Khan
• Subjects: cs.CR, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.05888
• Pdf URL: https://arxiv.org/pdf/2411.05888
• Copy Paste: [[2411.05888]] Sdn Intrusion Detection Using Machine Learning Method(https://arxiv.org/abs/2411.05888)
• Keywords: secure, security
• Abstract: Software-defined network (SDN) is a new approach that allows network control to become directly programmable, and the underlying infrastructure can be abstracted from applications and network services. Control plane). When it comes to security, the centralization that this demands is ripe for a variety of cyber threats that are not typically seen in other network architectures. The authors in this research developed a novel machine-learning method to capture infections in networks. We applied the classifier to the UNSW-NB 15 intrusion detection benchmark and trained a model with this data. Random Forest and Decision Tree are classifiers used to assess with Gradient Boosting and AdaBoost. Out of these best-performing models was Gradient Boosting with an accuracy, recall, and F1 score of 99.87%,100%, and 99.85%, respectively, which makes it reliable in the detection of intrusions for SDN networks. The second best-performing classifier was also a Random Forest with 99.38% of accuracy, followed by Ada Boost and Decision Tree. The research shows that the reason that Gradient Boosting is so effective in this task is that it combines weak learners and creates a strong ensemble model that can predict if traffic belongs to a normal or malicious one with high accuracy. This paper indicates that the GBDT-IDS model is able to improve network security significantly and has better features in terms of both real-time detection accuracy and low false positive rates. In future work, we will integrate this model into live SDN space to observe its application and scalability. This research serves as an initial base on which one can make further strides forward to enhance security in SDN using ML techniques and have more secure, resilient networks.

Title: A Comparative Analysis of Machine Learning Models for DDoS Detection in IoT Networks

• Authors: Sushil Shakya, Robert Abbas
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05890
• Pdf URL: https://arxiv.org/pdf/2411.05890
• Copy Paste: [[2411.05890]] A Comparative Analysis of Machine Learning Models for DDoS Detection in IoT Networks(https://arxiv.org/abs/2411.05890)
• Keywords: security, attack, robust
• Abstract: This paper presents the detection of DDoS attacks in IoT networks using machine learning models. Their rapid growth has made them highly susceptible to various forms of cyberattacks, many of whose security procedures are implemented in an irregular manner. It evaluates the efficacy of different machine learning models, such as XGBoost, K-Nearest Neighbours, Stochastic Gradient Descent, and Naïve Bayes, in detecting DDoS attacks from normal network traffic. Each model has been explained on several performance metrics, such as accuracy, precision, recall, and F1-score to understand the suitability of each model in real-time detection and response against DDoS threats. This comparative analysis will, therefore, enumerate the unique strengths and weaknesses of each model with respect to the IoT environments that are dynamic and hence moving in nature. The effectiveness of these models is analyzed, showing how machine learning can greatly enhance IoT security frameworks, offering adaptive, efficient, and reliable DDoS detection capabilities. These findings have shown the potential of machine learning in addressing the pressing need for robust IoT security solutions that can mitigate modern cyber threats and assure network integrity.

Title: Identifying and Decomposing Compound Ingredients in Meal Plans Using Large Language Models

• Authors: Leon Kopitar, Leon Bedrac, Larissa J Strath, Jiang Bian, Gregor Stiglic
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.05892
• Pdf URL: https://arxiv.org/pdf/2411.05892
• Copy Paste: [[2411.05892]] Identifying and Decomposing Compound Ingredients in Meal Plans Using Large Language Models(https://arxiv.org/abs/2411.05892)
• Keywords: large language model
• Abstract: This study explores the effectiveness of Large Language Models in meal planning, focusing on their ability to identify and decompose compound ingredients. We evaluated three models-GPT-4o, Llama-3 (70b), and Mixtral (8x7b)-to assess their proficiency in recognizing and breaking down complex ingredient combinations. Preliminary results indicate that while Llama-3 (70b) and GPT-4o excels in accurate decomposition, all models encounter difficulties with identifying essential elements like seasonings and oils. Despite strong overall performance, variations in accuracy and completeness were observed across models. These findings underscore LLMs' potential to enhance personalized nutrition but highlight the need for further refinement in ingredient decomposition. Future research should address these limitations to improve nutritional recommendations and health outcomes.

Title: SSSD: Simply-Scalable Speculative Decoding

• Authors: Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Lorenz K. Müller, Lukas Cavigelli
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05894
• Pdf URL: https://arxiv.org/pdf/2411.05894
• Copy Paste: [[2411.05894]] SSSD: Simply-Scalable Speculative Decoding(https://arxiv.org/abs/2411.05894)
• Keywords: large language model
• Abstract: Over the past year, Speculative Decoding has gained popularity as a technique for accelerating Large Language Model inference. While several methods have been introduced, most struggle to deliver satisfactory performance at batch sizes typical for data centers ($\geq 8$) and often involve significant deployment complexities. In this work, we offer a theoretical explanation of how Speculative Decoding can be effectively utilized with larger batch sizes. We also introduce a method that integrates seamlessly into existing systems without additional training or the complexity of deploying a small LLM. In a continuous batching setting, we achieve a 4x increase in throughput without any latency impact for short context generation, and a 1.7-2x improvement in both latency and throughput for longer contexts.

Title: One Small and One Large for Document-level Event Argument Extraction

• Authors: Jiaren Peng, Hongda Sun, Wenzhong Yang, Fuyuan Wei, Liang He, Liejun Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05895
• Pdf URL: https://arxiv.org/pdf/2411.05895
• Copy Paste: [[2411.05895]] One Small and One Large for Document-level Event Argument Extraction(https://arxiv.org/abs/2411.05895)
• Keywords: extraction, generative, large language model
• Abstract: Document-level Event Argument Extraction (EAE) faces two challenges due to increased input length: 1) difficulty in distinguishing semantic boundaries between events, and 2) interference from redundant information. To address these issues, we propose two methods. The first method introduces the Co and Structure Event Argument Extraction model (CsEAE) based on Small Language Models (SLMs). CsEAE includes a co-occurrences-aware module, which integrates information about all events present in the current input through context labeling and co-occurrences event prompts extraction. Additionally, CsEAE includes a structure-aware module that reduces interference from redundant information by establishing structural relationships between the sentence containing the trigger and other sentences in the document. The second method introduces new prompts to transform the extraction task into a generative task suitable for Large Language Models (LLMs), addressing gaps in EAE performance using LLMs under Supervised Fine-Tuning (SFT) conditions. We also fine-tuned multiple datasets to develop an LLM that performs better across most datasets. Finally, we applied insights from CsEAE to LLMs, achieving further performance improvements. This suggests that reliable insights validated on SLMs are also applicable to LLMs. We tested our models on the Rams, WikiEvents, and MLEE datasets. The CsEAE model achieved improvements of 2.1\%, 2.3\%, and 3.2\% in the Arg-C F1 metric compared to the baseline, PAIE~\cite{PAIE}. For LLMs, we demonstrated that their performance on document-level datasets is comparable to that of SLMs~\footnote{All code is available at this https URL}.

Title: Humans Continue to Outperform Large Language Models in Complex Clinical Decision-Making: A Study with Medical Calculators

• Authors: Nicholas Wan, Qiao Jin, Joey Chan, Guangzhi Xiong, Serina Applebaum, Aidan Gilson, Reid McMurry, R. Andrew Taylor, Aidong Zhang, Qingyu Chen, Zhiyong Lu
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.05897
• Pdf URL: https://arxiv.org/pdf/2411.05897
• Copy Paste: [[2411.05897]] Humans Continue to Outperform Large Language Models in Complex Clinical Decision-Making: A Study with Medical Calculators(https://arxiv.org/abs/2411.05897)
• Keywords: large language model
• Abstract: Although large language models (LLMs) have been assessed for general medical knowledge using medical licensing exams, their ability to effectively support clinical decision-making tasks, such as selecting and using medical calculators, remains uncertain. Here, we evaluate the capability of both medical trainees and LLMs to recommend medical calculators in response to various multiple-choice clinical scenarios such as risk stratification, prognosis, and disease diagnosis. We assessed eight LLMs, including open-source, proprietary, and domain-specific models, with 1,009 question-answer pairs across 35 clinical calculators and measured human performance on a subset of 100 questions. While the highest-performing LLM, GPT-4o, provided an answer accuracy of 74.3% (CI: 71.5-76.9%), human annotators, on average, outperformed LLMs with an accuracy of 79.5% (CI: 73.5-85.0%). With error analysis showing that the highest-performing LLMs continue to make mistakes in comprehension (56.6%) and calculator knowledge (8.1%), our findings emphasize that humans continue to surpass LLMs on complex clinical tasks such as calculator recommendation.

Title: Enhancing Cardiovascular Disease Prediction through Multi-Modal Self-Supervised Learning

• Authors: Francesco Girlanda, Olga Demler, Bjoern Menze, Neda Davoudi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05900
• Pdf URL: https://arxiv.org/pdf/2411.05900
• Copy Paste: [[2411.05900]] Enhancing Cardiovascular Disease Prediction through Multi-Modal Self-Supervised Learning(https://arxiv.org/abs/2411.05900)
• Keywords: robust
• Abstract: Accurate prediction of cardiovascular diseases remains imperative for early diagnosis and intervention, necessitating robust and precise predictive models. Recently, there has been a growing interest in multi-modal learning for uncovering novel insights not available through uni-modal datasets alone. By combining cardiac magnetic resonance images, electrocardiogram signals, and available medical information, our approach enables the capture of holistic status about individuals' cardiovascular health by leveraging shared information across modalities. Integrating information from multiple modalities and benefiting from self-supervised learning techniques, our model provides a comprehensive framework for enhancing cardiovascular disease prediction with limited annotated datasets. We employ a masked autoencoder to pre-train the electrocardiogram ECG encoder, enabling it to extract relevant features from raw electrocardiogram data, and an image encoder to extract relevant features from cardiac magnetic resonance images. Subsequently, we utilize a multi-modal contrastive learning objective to transfer knowledge from expensive and complex modality, cardiac magnetic resonance image, to cheap and simple modalities such as electrocardiograms and medical information. Finally, we fine-tuned the pre-trained encoders on specific predictive tasks, such as myocardial infarction. Our proposed method enhanced the image information by leveraging different available modalities and outperformed the supervised approach by 7.6% in balanced accuracy.

Title: ViT Enhanced Privacy-Preserving Secure Medical Data Sharing and Classification

• Authors: Al Amin, Kamrul Hasan, Sharif Ullah, M. Shamim Hossain
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05901
• Pdf URL: https://arxiv.org/pdf/2411.05901
• Copy Paste: [[2411.05901]] ViT Enhanced Privacy-Preserving Secure Medical Data Sharing and Classification(https://arxiv.org/abs/2411.05901)
• Keywords: secure, security, privacy, attack, robust, transformer
• Abstract: Privacy-preserving and secure data sharing are critical for medical image analysis while maintaining accuracy and minimizing computational overhead are also crucial. Applying existing deep neural networks (DNNs) to encrypted medical data is not always easy and often compromises performance and security. To address these limitations, this research introduces a secure framework consisting of a learnable encryption method based on the block-pixel operation to encrypt the data and subsequently integrate it with the Vision Transformer (ViT). The proposed framework ensures data privacy and security by creating unique scrambling patterns per key, providing robust performance against leading bit attacks and minimum difference attacks.

Title: Autoregressive Models in Vision: A Survey

• Authors: Jing Xiong, Gongye Liu, Lun Huang, Chengyue Wu, Taiqiang Wu, Yao Mu, Yuan Yao, Hui Shen, Zhongwei Wan, Jinfa Huang, Chaofan Tao, Shen Yan, Huaxiu Yao, Lingpeng Kong, Hongxia Yang, Mi Zhang, Guillermo Sapiro, Jiebo Luo, Ping Luo, Ngai Wong
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05902
• Pdf URL: https://arxiv.org/pdf/2411.05902
• Copy Paste: [[2411.05902]] Autoregressive Models in Vision: A Survey(https://arxiv.org/abs/2411.05902)
• Keywords: generative
• Abstract: Autoregressive modeling has been a huge success in the field of natural language processing (NLP). Recently, autoregressive models have emerged as a significant area of focus in computer vision, where they excel in producing high-quality visual content. Autoregressive models in NLP typically operate on subword tokens. However, the representation strategy in computer vision can vary in different levels, \textit{i.e.}, pixel-level, token-level, or scale-level, reflecting the diverse and hierarchical nature of visual data compared to the sequential structure of language. This survey comprehensively examines the literature on autoregressive models applied to vision. To improve readability for researchers from diverse research backgrounds, we start with preliminary sequence representation and modeling in vision. Next, we divide the fundamental frameworks of visual autoregressive models into three general sub-categories, including pixel-based, token-based, and scale-based models based on the strategy of representation. We then explore the interconnections between autoregressive models and other generative models. Furthermore, we present a multi-faceted categorization of autoregressive models in computer vision, including image generation, video generation, 3D generation, and multi-modal generation. We also elaborate on their applications in diverse domains, including emerging domains such as embodied AI and 3D medical AI, with about 250 related references. Finally, we highlight the current challenges to autoregressive models in vision with suggestions about potential research directions. We have also set up a Github repository to organize the papers included in this survey at: \url{this https URL}.

Title: DNAMite: Interpretable Calibrated Survival Analysis with Discretized Additive Models

• Authors: Mike Van Ness, Billy Block, Madeleine Udell
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.05923
• Pdf URL: https://arxiv.org/pdf/2411.05923
• Copy Paste: [[2411.05923]] DNAMite: Interpretable Calibrated Survival Analysis with Discretized Additive Models(https://arxiv.org/abs/2411.05923)
• Keywords: interpretability
• Abstract: Survival analysis is a classic problem in statistics with important applications in healthcare. Most machine learning models for survival analysis are black-box models, limiting their use in healthcare settings where interpretability is paramount. More recently, glass-box machine learning models have been introduced for survival analysis, with both strong predictive performance and interpretability. Still, several gaps remain, as no prior glass-box survival model can produce calibrated shape functions with enough flexibility to capture the complex patterns often found in real data. To fill this gap, we introduce a new glass-box machine learning model for survival analysis called DNAMite. DNAMite uses feature discretization and kernel smoothing in its embedding module, making it possible to learn shape functions with a flexible balance of smoothness and jaggedness. Further, DNAMite produces calibrated shape functions that can be directly interpreted as contributions to the cumulative incidence function. Our experiments show that DNAMite generates shape functions closer to true shape functions on synthetic data, while making predictions with comparable predictive performance and better calibration than previous glass-box and black-box models.

Title: Reducing Distraction in Long-Context Language Models by Focused Learning

• Authors: Zijun Wu, Bingyuan Liu, Ran Yan, Lei Chen, Thomas Delteil
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05928
• Pdf URL: https://arxiv.org/pdf/2411.05928
• Copy Paste: [[2411.05928]] Reducing Distraction in Long-Context Language Models by Focused Learning(https://arxiv.org/abs/2411.05928)
• Keywords: large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have significantly enhanced their capacity to process long contexts. However, effectively utilizing this long context remains a challenge due to the issue of distraction, where irrelevant information dominates lengthy contexts, causing LLMs to lose focus on the most relevant segments. To address this, we propose a novel training method that enhances LLMs' ability to discern relevant information through a unique combination of retrieval-based data augmentation and contrastive learning. Specifically, during fine-tuning with long contexts, we employ a retriever to extract the most relevant segments, serving as augmented inputs. We then introduce an auxiliary contrastive learning objective to explicitly ensure that outputs from the original context and the retrieved sub-context are closely aligned. Extensive experiments on long single-document and multi-document QA benchmarks demonstrate the effectiveness of our proposed method.

Title: BERTrend: Neural Topic Modeling for Emerging Trends Detection

• Authors: Allaa Boutaleb, Jerome Picault, Guillaume Grosjean
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.05930
• Pdf URL: https://arxiv.org/pdf/2411.05930
• Copy Paste: [[2411.05930]] BERTrend: Neural Topic Modeling for Emerging Trends Detection(https://arxiv.org/abs/2411.05930)
• Keywords: interpretability, large language model
• Abstract: Detecting and tracking emerging trends and weak signals in large, evolving text corpora is vital for applications such as monitoring scientific literature, managing brand reputation, surveilling critical infrastructure and more generally to any kind of text-based event detection. Existing solutions often fail to capture the nuanced context or dynamically track evolving patterns over time. BERTrend, a novel method, addresses these limitations using neural topic modeling in an online setting. It introduces a new metric to quantify topic popularity over time by considering both the number of documents and update frequency. This metric classifies topics as noise, weak, or strong signals, flagging emerging, rapidly growing topics for further investigation. Experimentation on two large real-world datasets demonstrates BERTrend's ability to accurately detect and track meaningful weak signals while filtering out noise, offering a comprehensive solution for monitoring emerging trends in large-scale, evolving text corpora. The method can also be used for retrospective analysis of past events. In addition, the use of Large Language Models together with BERTrend offers efficient means for the interpretability of trends of events.

Title: GCI-ViTAL: Gradual Confidence Improvement with Vision Transformers for Active Learning on Label Noise

• Authors: Moseli Mots'oehli, kyungim Baek
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05939
• Pdf URL: https://arxiv.org/pdf/2411.05939
• Copy Paste: [[2411.05939]] GCI-ViTAL: Gradual Confidence Improvement with Vision Transformers for Active Learning on Label Noise(https://arxiv.org/abs/2411.05939)
• Keywords: robust, transformer
• Abstract: Active learning aims to train accurate classifiers while minimizing labeling costs by strategically selecting informative samples for annotation. This study focuses on image classification tasks, comparing AL methods on CIFAR10, CIFAR100, Food101, and the Chest X-ray datasets under varying label noise rates. We investigate the impact of model architecture by comparing Convolutional Neural Networks (CNNs) and Vision Transformer (ViT)-based models. Additionally, we propose a novel deep active learning algorithm, GCI-ViTAL, designed to be robust to label noise. GCI-ViTAL utilizes prediction entropy and the Frobenius norm of last-layer attention vectors compared to class-centric clean set attention vectors. Our method identifies samples that are both uncertain and semantically divergent from typical images in their assigned class. This allows GCI-ViTAL to select informative data points even in the presence of label noise while flagging potentially mislabeled candidates. Label smoothing is applied to train a model that is not overly confident about potentially noisy labels. We evaluate GCI-ViTAL under varying levels of symmetric label noise and compare it to five other AL strategies. Our results demonstrate that using ViTs leads to significant performance improvements over CNNs across all AL strategies, particularly in noisy label settings. We also find that using the semantic information of images as label grounding helps in training a more robust model under label noise. Notably, we do not perform extensive hyperparameter tuning, providing an out-of-the-box comparison that addresses the common challenge practitioners face in selecting models and active learning strategies without an exhaustive literature review on training and fine-tuning vision models on real-world application data.

Title: NeKo: Toward Post Recognition Generative Correction Large Language Models with Task-Oriented Experts

• Authors: Yen-Ting Lin, Chao-Han Huck Yang, Zhehuai Chen, Piotr Zelasko, Xuesong Yang, Zih-Ching Chen, Krishna C Puvvada, Szu-Wei Fu, Ke Hu, Jun Wei Chiu, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang
• Subjects: cs.CL, cs.AI, cs.LG, cs.MA, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.05945
• Pdf URL: https://arxiv.org/pdf/2411.05945
• Copy Paste: [[2411.05945]] NeKo: Toward Post Recognition Generative Correction Large Language Models with Task-Oriented Experts(https://arxiv.org/abs/2411.05945)
• Keywords: generative, large language model
• Abstract: Construction of a general-purpose post-recognition error corrector poses a crucial question: how can we most effectively train a model on a large mixture of domain datasets? The answer would lie in learning dataset-specific features and digesting their knowledge in a single model. Previous methods achieve this by having separate correction language models, resulting in a significant increase in parameters. In this work, we present Mixture-of-Experts as a solution, highlighting that MoEs are much more than a scalability tool. We propose a Multi-Task Correction MoE, where we train the experts to become an ``expert'' of speech-to-text, language-to-text and vision-to-text datasets by learning to route each dataset's tokens to its mapped expert. Experiments on the Open ASR Leaderboard show that we explore a new state-of-the-art performance by achieving an average relative $5.0$% WER reduction and substantial improvements in BLEU scores for speech and translation tasks. On zero-shot evaluation, NeKo outperforms GPT-3.5 and Claude-Opus with $15.5$% to $27.6$% relative WER reduction in the Hyporadise benchmark. NeKo performs competitively on grammar and post-OCR correction as a multi-task model.

Title: Ideal Pseudorandom Codes

• Authors: Omar Alrabiah, Prabhanjan Ananth, Miranda Christ, Yevgeniy Dodis, Sam Gunn
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.05947
• Pdf URL: https://arxiv.org/pdf/2411.05947
• Copy Paste: [[2411.05947]] Ideal Pseudorandom Codes(https://arxiv.org/abs/2411.05947)
• Keywords: secure, security, robust, watermark, generative
• Abstract: Pseudorandom codes are error-correcting codes with the property that no efficient adversary can distinguish encodings from uniformly random strings. They were recently introduced by Christ and Gunn [CRYPTO 2024] for the purpose of watermarking the outputs of randomized algorithms, such as generative AI models. Several constructions of pseudorandom codes have since been proposed, but none of them are robust to error channels that depend on previously seen codewords. This stronger kind of robustness is referred to as adaptive robustness, and it is important for meaningful applications to watermarking. In this work, we show the following. - Adaptive robustness: We show that the pseudorandom codes of Christ and Gunn are adaptively robust, resolving a conjecture posed by Cohen, Hoover, and Schoenbach [S&P 2025]. - Ideal security: We define an ideal pseudorandom code as one which is indistinguishable from the ideal functionality, capturing both the pseudorandomness and robustness properties in one simple definition. We show that any adaptively robust pseudorandom code for single-bit messages can be bootstrapped to build an ideal pseudorandom code with linear information rate, under no additional assumptions. - CCA security: In the setting where the encoding key is made public, we define a CCA-secure pseudorandom code in analogy with CCA-secure encryption. We show that any adaptively robust public-key pseudorandom code for single-bit messages can be used to build a CCA-secure pseudorandom code with linear information rate, in the random oracle model. These results immediately imply stronger robustness guarantees for generative AI watermarking schemes, such as the practical quality-preserving image watermarks of Gunn, Zhao, and Song (2024).

Title: Utilisation of Vision Systems and Digital Twin for Maintaining Cleanliness in Public Spaces

• Authors: Mateusz Wasala, Krzysztof Blachut, Hubert Szolc, Marcin Kowalczyk, Michal Danilowicz, Tomasz Kryjak
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.05964
• Pdf URL: https://arxiv.org/pdf/2411.05964
• Copy Paste: [[2411.05964]] Utilisation of Vision Systems and Digital Twin for Maintaining Cleanliness in Public Spaces(https://arxiv.org/abs/2411.05964)
• Keywords: segmentation
• Abstract: Nowadays, the increasing demand for maintaining high cleanliness standards in public spaces results in the search for innovative solutions. The deployment of CCTV systems equipped with modern cameras and software enables not only real-time monitoring of the cleanliness status but also automatic detection of impurities and optimisation of cleaning schedules. The Digital Twin technology allows for the creation of a virtual model of the space, facilitating the simulation, training, and testing of cleanliness management strategies before implementation in the real world. In this paper, we present the utilisation of advanced vision surveillance systems and the Digital Twin technology in cleanliness management, using a railway station as an example. The Digital Twin was created based on an actual 3D model in the Nvidia Omniverse Isaac Sim simulator. A litter detector, bin occupancy level detector, stain segmentation, and a human detector (including the cleaning crew) along with their movement analysis were implemented. A preliminary assessment was conducted, and potential modifications for further enhancement and future development of the system were identified.

Title: The Empirical Impact of Data Sanitization on Language Models

• Authors: Anwesan Pal, Radhika Bhargava, Kyle Hinsz, Jacques Esterhuizen, Sudipta Bhattacharya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05978
• Pdf URL: https://arxiv.org/pdf/2411.05978
• Copy Paste: [[2411.05978]] The Empirical Impact of Data Sanitization on Language Models(https://arxiv.org/abs/2411.05978)
• Keywords: privacy, large language model
• Abstract: Data sanitization in the context of language modeling involves identifying sensitive content, such as personally identifiable information (PII), and redacting them from a dataset corpus. It is a common practice used in natural language processing (NLP) to maintain privacy. Nevertheless, the impact of data sanitization on the language understanding capability of a language model remains less studied. This paper empirically analyzes the effects of data sanitization across several benchmark language-modeling tasks including comprehension question answering (Q&A), entailment, sentiment analysis, and text classification. Our experiments cover a wide spectrum comprising finetuning small-scale language models, to prompting large language models (LLMs), on both original and sanitized datasets, and comparing their performance across the tasks. Interestingly, our results suggest that for some tasks such as sentiment analysis or entailment, the impact of redaction is quite low, typically around 1-5%, while for tasks such as comprehension Q&A there is a big drop of >25% in performance observed in redacted queries as compared to the original. For tasks that have a higher impact, we perform a deeper dive to inspect the presence of task-critical entities. Finally, we investigate correlation between performance and number of redacted entities, and also suggest a strategy to repair an already redacted dataset by means of content-based subsampling. Additional details are available at this https URL.

Title: FactLens: Benchmarking Fine-Grained Fact Verification

• Authors: Kushan Mitra, Dan Zhang, Sajjadur Rahman, Estevam Hruschka
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05980
• Pdf URL: https://arxiv.org/pdf/2411.05980
• Copy Paste: [[2411.05980]] FactLens: Benchmarking Fine-Grained Fact Verification(https://arxiv.org/abs/2411.05980)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown impressive capability in language generation and understanding, but their tendency to hallucinate and produce factually incorrect information remains a key limitation. To verify LLM-generated contents and claims from other sources, traditional verification approaches often rely on holistic models that assign a single factuality label to complex claims, potentially obscuring nuanced errors. In this paper, we advocate for a shift toward fine-grained verification, where complex claims are broken down into smaller sub-claims for individual verification, allowing for more precise identification of inaccuracies, improved transparency, and reduced ambiguity in evidence retrieval. However, generating sub-claims poses challenges, such as maintaining context and ensuring semantic equivalence with respect to the original claim. We introduce FactLens, a benchmark for evaluating fine-grained fact verification, with metrics and automated evaluators of sub-claim quality. The benchmark data is manually curated to ensure high-quality ground truth. Our results show alignment between automated FactLens evaluators and human judgments, and we discuss the impact of sub-claim characteristics on the overall verification performance.

Title: Unmasking the Shadows: Pinpoint the Implementations of Anti-Dynamic Analysis Techniques in Malware Using LLM

• Authors: Haizhou Wang, Nanqing Luo, Peng LIu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05982
• Pdf URL: https://arxiv.org/pdf/2411.05982
• Copy Paste: [[2411.05982]] Unmasking the Shadows: Pinpoint the Implementations of Anti-Dynamic Analysis Techniques in Malware Using LLM(https://arxiv.org/abs/2411.05982)
• Keywords: large language model
• Abstract: Sandboxes and other dynamic analysis processes are prevalent in malware detection systems nowadays to enhance the capability of detecting 0-day malware. Therefore, techniques of anti-dynamic analysis (TADA) are prevalent in modern malware samples, and sandboxes can suffer from false negatives and analysis failures when analyzing the samples with TADAs. In such cases, human reverse engineers will get involved in conducting dynamic analysis manually (i.e., debugging, patching), which in turn also gets obstructed by TADAs. In this work, we propose a Large Language Model (LLM) based workflow that can pinpoint the location of the TADA implementation in the code, to help reverse engineers place breakpoints used in debugging. Our evaluation shows that we successfully identified the locations of 87.80% known TADA implementations adopted from public repositories. In addition, we successfully pinpoint the locations of TADAs in 4 well-known malware samples that are documented in online malware analysis blogs.

Title: Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings

• Authors: Miguel Moura Ramos, Tomás Almeida, Daniel Vareta, Filipe Azevedo, Sweta Agrawal, Patrick Fernandes, André F. T. Martins
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05986
• Pdf URL: https://arxiv.org/pdf/2411.05986
• Copy Paste: [[2411.05986]] Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings(https://arxiv.org/abs/2411.05986)
• Keywords: robust
• Abstract: Reinforcement learning (RL) has been proven to be an effective and robust method for training neural machine translation systems, especially when paired with powerful reward models that accurately assess translation quality. However, most research has focused on RL methods that use sentence-level feedback, which leads to inefficient learning signals due to the reward sparsity problem -- the model receives a single score for the entire sentence. To address this, we introduce a novel approach that leverages fine-grained token-level reward mechanisms with RL methods. We use xCOMET, a state-of-the-art quality estimation system as our token-level reward model. xCOMET provides detailed feedback by predicting fine-grained error spans and their severity given source-translation pairs. We conduct experiments on small and large translation datasets to compare the impact of sentence-level versus fine-grained reward signals on translation quality. Our results show that training with token-level rewards improves translation quality across language pairs over baselines according to automatic and human evaluation. Furthermore, token-level reward optimization also improves training stability, evidenced by a steady increase in mean rewards over training epochs.

Title: GUIDEQ: Framework for Guided Questioning for progressive informational collection and classification

• Authors: Priya Mishra, Suraj Racha, Kaustubh Ponkshe, Adit Akarsh, Ganesh Ramakrishnan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05991
• Pdf URL: https://arxiv.org/pdf/2411.05991
• Copy Paste: [[2411.05991]] GUIDEQ: Framework for Guided Questioning for progressive informational collection and classification(https://arxiv.org/abs/2411.05991)
• Keywords: explainability
• Abstract: Question Answering (QA) is an important part of tasks like text classification through information gathering. These are finding increasing use in sectors like healthcare, customer support, legal services, etc., to collect and classify responses into actionable categories. LLMs, although can support QA systems, they face a significant challenge of insufficient or missing information for classification. Although LLMs excel in reasoning, the models rely on their parametric knowledge to answer. However, questioning the user requires domain-specific information aiding to collect accurate information. Our work, GUIDEQ, presents a novel framework for asking guided questions to further progress a partial information. We leverage the explainability derived from the classifier model for along with LLMs for asking guided questions to further enhance the information. This further information helps in more accurate classification of a text. GUIDEQ derives the most significant key-words representative of a label using occlusions. We develop GUIDEQ's prompting strategy for guided questions based on the top-3 classifier label outputs and the significant words, to seek specific and relevant information, and classify in a targeted manner. Through our experimental results, we demonstrate that GUIDEQ outperforms other LLM-based baselines, yielding improved F1-Score through the accurate collection of relevant further information. We perform various analytical studies and also report better question quality compared to our method.

Title: A Modular Conditional Diffusion Framework for Image Reconstruction

• Authors: Magauiya Zhussip, Iaroslav Koshelev, Stamatis Lefkimmiatis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05993
• Pdf URL: https://arxiv.org/pdf/2411.05993
• Copy Paste: [[2411.05993]] A Modular Conditional Diffusion Framework for Image Reconstruction(https://arxiv.org/abs/2411.05993)
• Keywords: diffusion, generative
• Abstract: Diffusion Probabilistic Models (DPMs) have been recently utilized to deal with various blind image restoration (IR) tasks, where they have demonstrated outstanding performance in terms of perceptual quality. However, the task-specific nature of existing solutions and the excessive computational costs related to their training, make such models impractical and challenging to use for different IR tasks than those that were initially trained for. This hinders their wider adoption, especially by those who lack access to powerful computational resources and vast amount of training data. In this work we aim to address the above issues and enable the successful adoption of DPMs in practical IR-related applications. Towards this goal, we propose a modular diffusion probabilistic IR framework (DP-IR), which allows us to combine the performance benefits of existing pre-trained state-of-the-art IR networks and generative DPMs, while it requires only the additional training of a relatively small module (0.7M params) related to the particular IR task of interest. Moreover, the architecture of the proposed framework allows for a sampling strategy that leads to at least four times reduction of neural function evaluations without suffering any performance loss, while it can also be combined with existing acceleration techniques such as DDIM. We evaluate our model on four benchmarks for the tasks of burst JDD-SR, dynamic scene deblurring, and super-resolution. Our method outperforms existing approaches in terms of perceptual quality while it retains a competitive performance with respect to fidelity metrics.

Title: The Dark Patterns of Personalized Persuasion in Large Language Models: Exposing Persuasive Linguistic Features for Big Five Personality Traits in LLMs Responses

• Authors: Wiktoria Mieleszczenko-Kowszewicz, Dawid Płudowski, Filip Kołodziejczyk, Jakub Świstak, Julian Sienkiewicz, Przemysław Biecek
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06008
• Pdf URL: https://arxiv.org/pdf/2411.06008
• Copy Paste: [[2411.06008]] The Dark Patterns of Personalized Persuasion in Large Language Models: Exposing Persuasive Linguistic Features for Big Five Personality Traits in LLMs Responses(https://arxiv.org/abs/2411.06008)
• Keywords: large language model
• Abstract: This study explores how the Large Language Models (LLMs) adjust linguistic features to create personalized persuasive outputs. While research showed that LLMs personalize outputs, a gap remains in understanding the linguistic features of their persuasive capabilities. We identified 13 linguistic features crucial for influencing personalities across different levels of the Big Five model of personality. We analyzed how prompts with personality trait information influenced the output of 19 LLMs across five model families. The findings show that models use more anxiety-related words for neuroticism, increase achievement-related words for conscientiousness, and employ fewer cognitive processes words for openness to experience. Some model families excel at adapting language for openness to experience, others for conscientiousness, while only one model adapts language for neuroticism. Our findings show how LLMs tailor responses based on personality cues in prompts, indicating their potential to create persuasive content affecting the mind and well-being of the recipients.

Title: A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualization

• Authors: Haoxin Liu, Chenghao Liu, B. Aditya Prakash
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06018
• Pdf URL: https://arxiv.org/pdf/2411.06018
• Copy Paste: [[2411.06018]] A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualization(https://arxiv.org/abs/2411.06018)
• Keywords: large language model
• Abstract: Large language models (LLMs), with demonstrated reasoning abilities across multiple domains, are largely underexplored for time-series reasoning (TsR), which is ubiquitous in the real world. In this work, we propose TimerBed, the first comprehensive testbed for evaluating LLMs' TsR performance. Specifically, TimerBed includes stratified reasoning patterns with real-world tasks, comprehensive combinations of LLMs and reasoning strategies, and various supervised models as comparison anchors. We perform extensive experiments with TimerBed, test multiple current beliefs, and verify the initial failures of LLMs in TsR, evidenced by the ineffectiveness of zero shot (ZST) and performance degradation of few shot in-context learning (ICL). Further, we identify one possible root cause: the numerical modeling of data. To address this, we propose a prompt-based solution VL-Time, using visualization-modeled data and language-guided reasoning. Experimental results demonstrate that Vl-Time enables multimodal LLMs to be non-trivial ZST and powerful ICL reasoners for time series, achieving about 140% average performance improvement and 99% average token costs reduction.

Title: Dynamic Textual Prompt For Rehearsal-free Lifelong Person Re-identification

• Authors: Hongyu Chen, Bingliang Jiao, Wenxuan Wang, Peng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06023
• Pdf URL: https://arxiv.org/pdf/2411.06023
• Copy Paste: [[2411.06023]] Dynamic Textual Prompt For Rehearsal-free Lifelong Person Re-identification(https://arxiv.org/abs/2411.06023)
• Keywords: privacy
• Abstract: Lifelong person re-identification attempts to recognize people across cameras and integrate new knowledge from continuous data streams. Key challenges involve addressing catastrophic forgetting caused by parameter updating and domain shift, and maintaining performance in seen and unseen domains. Many previous works rely on data memories to retain prior samples. However, the amount of retained data increases linearly with the number of training domains, leading to continually increasing memory consumption. Additionally, these methods may suffer significant performance degradation when data preservation is prohibited due to privacy concerns. To address these limitations, we propose using textual descriptions as guidance to encourage the ReID model to learn cross-domain invariant features without retaining samples. The key insight is that natural language can describe pedestrian instances with an invariant style, suggesting a shared textual space for any pedestrian images. By leveraging this shared textual space as an anchor, we can prompt the ReID model to embed images from various domains into a unified semantic space, thereby alleviating catastrophic forgetting caused by domain shifts. To achieve this, we introduce a task-driven dynamic textual prompt framework in this paper. This model features a dynamic prompt fusion module, which adaptively constructs and fuses two different textual prompts as anchors. This effectively guides the ReID model to embed images into a unified semantic space. Additionally, we design a text-visual feature alignment module to learn a more precise mapping between fine-grained visual and textual features. We also developed a learnable knowledge distillation module that allows our model to dynamically balance retaining existing knowledge with acquiring new knowledge. Extensive experiments demonstrate that our method outperforms SOTAs under various settings.

Title: LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Output

• Authors: Elise Karinshak, Amanda Hu, Kewen Kong, Vishwanatha Rao, Jingren Wang, Jindong Wang, Yi Zeng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06032
• Pdf URL: https://arxiv.org/pdf/2411.06032
• Copy Paste: [[2411.06032]] LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Output(https://arxiv.org/abs/2411.06032)
• Keywords: generative
• Abstract: Immense effort has been dedicated to minimizing the presence of harmful or biased generative content and better aligning AI output to human intention; however, research investigating the cultural values of LLMs is still in very early stages. Cultural values underpin how societies operate, providing profound insights into the norms, priorities, and decision making of their members. In recognition of this need for further research, we draw upon cultural psychology theory and the empirically-validated GLOBE framework to propose the LLM-GLOBE benchmark for evaluating the cultural value systems of LLMs, and we then leverage the benchmark to compare the values of Chinese and US LLMs. Our methodology includes a novel "LLMs-as-a-Jury" pipeline which automates the evaluation of open-ended content to enable large-scale analysis at a conceptual level. Results clarify similarities and differences that exist between Eastern and Western cultural value systems and suggest that open-generation tasks represent a more promising direction for evaluation of cultural values. We interpret the implications of this research for subsequent model development, evaluation, and deployment efforts as they relate to LLMs, AI cultural alignment more broadly, and the influence of AI cultural value systems on human-AI collaboration outcomes.

Title: CGLearn: Consistent Gradient-Based Learning for Out-of-Distribution Generalization

• Authors: Jawad Chowdhury, Gabriel Terejanu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06040
• Pdf URL: https://arxiv.org/pdf/2411.06040
• Copy Paste: [[2411.06040]] CGLearn: Consistent Gradient-Based Learning for Out-of-Distribution Generalization(https://arxiv.org/abs/2411.06040)
• Keywords: robust
• Abstract: Improving generalization and achieving highly predictive, robust machine learning models necessitates learning the underlying causal structure of the variables of interest. A prominent and effective method for this is learning invariant predictors across multiple environments. In this work, we introduce a simple yet powerful approach, CGLearn, which relies on the agreement of gradients across various environments. This agreement serves as a powerful indication of reliable features, while disagreement suggests less reliability due to potential differences in underlying causal mechanisms. Our proposed method demonstrates superior performance compared to state-of-the-art methods in both linear and nonlinear settings across various regression and classification tasks. CGLearn shows robust applicability even in the absence of separate environments by exploiting invariance across different subsamples of observational data. Comprehensive experiments on both synthetic and real-world datasets highlight its effectiveness in diverse scenarios. Our findings underscore the importance of leveraging gradient agreement for learning causal invariance, providing a significant step forward in the field of robust machine learning. The source code of the linear and nonlinear implementation of CGLearn is open-source and available at: this https URL.

Title: Personalized Hierarchical Split Federated Learning in Wireless Networks

• Authors: Md-Ferdous Pervej, Andreas F. Molisch
• Subjects: cs.LG, cs.NI, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.06042
• Pdf URL: https://arxiv.org/pdf/2411.06042
• Copy Paste: [[2411.06042]] Personalized Hierarchical Split Federated Learning in Wireless Networks(https://arxiv.org/abs/2411.06042)
• Keywords: federate
• Abstract: Extreme resource constraints make large-scale machine learning (ML) with distributed clients challenging in wireless networks. On the one hand, large-scale ML requires massive information exchange between clients and server(s). On the other hand, these clients have limited battery and computation powers that are often dedicated to operational computations. Split federated learning (SFL) is emerging as a potential solution to mitigate these challenges, by splitting the ML model into client-side and server-side model blocks, where only the client-side block is trained on the client device. However, practical applications require personalized models that are suitable for the client's personal task. Motivated by this, we propose a personalized hierarchical split federated learning (PHSFL) algorithm that is specially designed to achieve better personalization performance. More specially, owing to the fact that regardless of the severity of the statistical data distributions across the clients, many of the features have similar attributes, we only train the body part of the federated learning (FL) model while keeping the (randomly initialized) classifier frozen during the training phase. We first perform extensive theoretical analysis to understand the impact of model splitting and hierarchical model aggregations on the global model. Once the global model is trained, we fine-tune each client classifier to obtain the personalized models. Our empirical findings suggest that while the globally trained model with the untrained classifier performs quite similarly to other existing solutions, the fine-tuned models show significantly improved personalized performance.

Title: Learning Mixtures of Experts with EM

• Authors: Quentin Fruytier, Aryan Mokhtari, Sujay Sanghavi
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06056
• Pdf URL: https://arxiv.org/pdf/2411.06056
• Copy Paste: [[2411.06056]] Learning Mixtures of Experts with EM(https://arxiv.org/abs/2411.06056)
• Keywords: large language model
• Abstract: Mixtures of Experts (MoE) are Machine Learning models that involve partitioning the input space, with a separate "expert" model trained on each partition. Recently, MoE have become popular as components in today's large language models as a means to reduce training and inference costs. There, the partitioning function and the experts are both learnt jointly via gradient descent on the log-likelihood. In this paper we focus on studying the efficiency of the Expectation Maximization (EM) algorithm for the training of MoE models. We first rigorously analyze EM for the cases of linear or logistic experts, where we show that EM is equivalent to Mirror Descent with unit step size and a Kullback-Leibler Divergence regularizer. This perspective allows us to derive new convergence results and identify conditions for local linear convergence based on the signal-to-noise ratio (SNR). Experiments on synthetic and (small-scale) real-world data show that EM outperforms the gradient descent algorithm both in terms of convergence rate and the achieved accuracy.

Title: AI-Driven Stylization of 3D Environments

• Authors: Yuanbo Chen, Yixiao Kang, Yukun Song, Cyrus Vachha, Sining Huang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.06067
• Pdf URL: https://arxiv.org/pdf/2411.06067
• Copy Paste: [[2411.06067]] AI-Driven Stylization of 3D Environments(https://arxiv.org/abs/2411.06067)
• Keywords: generative
• Abstract: In this system, we discuss methods to stylize a scene of 3D primitive objects into a higher fidelity 3D scene using novel 3D representations like NeRFs and 3D Gaussian Splatting. Our approach leverages existing image stylization systems and image-to-3D generative models to create a pipeline that iteratively stylizes and composites 3D objects into scenes. We show our results on adding generated objects into a scene and discuss limitations.

Title: Aquila: A Hierarchically Aligned Visual-Language Model for Enhanced Remote Sensing Image Comprehension

• Authors: Kaixuan Lu, Ruiqian Zhang, Xiao Huang, Yuxing Xie
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06074
• Pdf URL: https://arxiv.org/pdf/2411.06074
• Copy Paste: [[2411.06074]] Aquila: A Hierarchically Aligned Visual-Language Model for Enhanced Remote Sensing Image Comprehension(https://arxiv.org/abs/2411.06074)
• Keywords: large language model
• Abstract: Recently, large vision language models (VLMs) have made significant strides in visual language capabilities through visual instruction tuning, showing great promise in the field of remote sensing image interpretation. However, existing remote sensing vision language models (RSVLMs) often fall short in capturing the complex characteristics of remote sensing scenes, as they typically rely on low resolution, single scale visual features and simplistic methods to map visual features to language features. In this paper, we present Aquila, an advanced visual language foundation model designed to enable richer visual feature representation and more precise visual-language feature alignment for remote sensing images. Our approach introduces a learnable Hierarchical Spatial Feature Integration (SFI) module that supports high resolution image inputs and aggregates multi scale visual features, allowing for the detailed representation of complex visual information. Additionally, the SFI module is repeatedly integrated into the layers of the large language model (LLM) to achieve deep visual language feature alignment, without compromising the model's performance in natural language processing tasks. These innovations, capturing detailed visual effects through higher resolution and multi scale input, and enhancing feature alignment significantly improve the model's ability to learn from image text data. We validate the effectiveness of Aquila through extensive quantitative experiments and qualitative analyses, demonstrating its superior performance.

Title: A Survey on Kolmogorov-Arnold Network

• Authors: Shriyank Somvanshi, Syed Aaqib Javed, Md Monzurul Islam, Diwas Pandit, Subasish Das
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06078
• Pdf URL: https://arxiv.org/pdf/2411.06078
• Copy Paste: [[2411.06078]] A Survey on Kolmogorov-Arnold Network(https://arxiv.org/abs/2411.06078)
• Keywords: interpretability, transformer
• Abstract: This systematic review explores the theoretical foundations, evolution, applications, and future potential of Kolmogorov-Arnold Networks (KAN), a neural network model inspired by the Kolmogorov-Arnold representation theorem. KANs distinguish themselves from traditional neural networks by using learnable, spline-parameterized functions instead of fixed activation functions, allowing for flexible and interpretable representations of high-dimensional functions. This review details KAN's architectural strengths, including adaptive edge-based activation functions that improve parameter efficiency and scalability in applications such as time series forecasting, computational biomedicine, and graph learning. Key advancements, including Temporal-KAN, FastKAN, and Partial Differential Equation (PDE) KAN, illustrate KAN's growing applicability in dynamic environments, enhancing interpretability, computational efficiency, and adaptability for complex function approximation tasks. Additionally, this paper discusses KAN's integration with other architectures, such as convolutional, recurrent, and transformer-based models, showcasing its versatility in complementing established neural networks for tasks requiring hybrid approaches. Despite its strengths, KAN faces computational challenges in high-dimensional and noisy data settings, motivating ongoing research into optimization strategies, regularization techniques, and hybrid models. This paper highlights KAN's role in modern neural architectures and outlines future directions to improve its computational efficiency, interpretability, and scalability in data-intensive applications.

Title: Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques

• Authors: Jahid Hasan
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06084
• Pdf URL: https://arxiv.org/pdf/2411.06084
• Copy Paste: [[2411.06084]] Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques(https://arxiv.org/abs/2411.06084)
• Keywords: large language model
• Abstract: This paper presents a comprehensive analysis of quantization techniques for optimizing Large Language Models (LLMs), specifically focusing on Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). Through empirical evaluation across models ranging from 10M to 1B parameters, we demonstrate that quantization can achieve up to 68% reduction in model size while maintaining performance within 6% of full-precision baselines when utilizing our proposed scaling factor {\gamma}. Our experiments show that INT8 quantization delivers a 40% reduction in computational cost and power consumption, while INT4 quantization further improves these metrics by 60%. We introduce a novel theoretical framework for mixed-precision quantization, deriving optimal bit allocation strategies based on layer sensitivity and weight variance. Hardware efficiency evaluations on edge devices reveal that our quantization approach enables up to 2.4x throughput improvement for INT8 and 3x for INT4, with 60% power reduction compared to full-precision models.

Title: Concept Bottleneck Language Models For protein design

• Authors: Aya Abdelsalam Ismail, Tuomas Oikarinen, Amy Wang, Julius Adebayo, Samuel Stanton, Taylor Joren, Joseph Kleinhenz, Allen Goodman, Héctor Corrada Bravo, Kyunghyun Cho, Nathan C. Frey
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06090
• Pdf URL: https://arxiv.org/pdf/2411.06090
• Copy Paste: [[2411.06090]] Concept Bottleneck Language Models For protein design(https://arxiv.org/abs/2411.06090)
• Keywords: interpretability, generative
• Abstract: We introduce Concept Bottleneck Protein Language Models (CB-pLM), a generative masked language model with a layer where each neuron corresponds to an interpretable concept. Our architecture offers three key benefits: i) Control: We can intervene on concept values to precisely control the properties of generated proteins, achieving a 3 times larger change in desired concept values compared to baselines. ii) Interpretability: A linear mapping between concept values and predicted tokens allows transparent analysis of the model's decision-making process. iii) Debugging: This transparency facilitates easy debugging of trained models. Our models achieve pre-training perplexity and downstream task performance comparable to traditional masked protein language models, demonstrating that interpretability does not compromise performance. While adaptable to any language model, we focus on masked protein language models due to their importance in drug discovery and the ability to validate our model's capabilities through real-world experiments and expert knowledge. We scale our CB-pLM from 24 million to 3 billion parameters, making them the largest Concept Bottleneck Models trained and the first capable of generative language modeling.

Title: Pattern Integration and Enhancement Vision Transformer for Self-Supervised Learning in Remote Sensing

• Authors: Kaixuan Lu, Ruiqian Zhang, Xiao Huang, Yuxing Xie, Xiaogang Ning, Hanchao Zhang, Mengke Yuan, Pan Zhang, Tao Wang, Tongkui Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06091
• Pdf URL: https://arxiv.org/pdf/2411.06091
• Copy Paste: [[2411.06091]] Pattern Integration and Enhancement Vision Transformer for Self-Supervised Learning in Remote Sensing(https://arxiv.org/abs/2411.06091)
• Keywords: robust, transformer, segmentation
• Abstract: Recent self-supervised learning (SSL) methods have demonstrated impressive results in learning visual representations from unlabeled remote sensing images. However, most remote sensing images predominantly consist of scenographic scenes containing multiple ground objects without explicit foreground targets, which limits the performance of existing SSL methods that focus on foreground targets. This raises the question: Is there a method that can automatically aggregate similar objects within scenographic remote sensing images, thereby enabling models to differentiate knowledge embedded in various geospatial patterns for improved feature representation? In this work, we present the Pattern Integration and Enhancement Vision Transformer (PIEViT), a novel self-supervised learning framework designed specifically for remote sensing imagery. PIEViT utilizes a teacher-student architecture to address both image-level and patch-level tasks. It employs the Geospatial Pattern Cohesion (GPC) module to explore the natural clustering of patches, enhancing the differentiation of individual features. The Feature Integration Projection (FIP) module further refines masked token reconstruction using geospatially clustered patches. We validated PIEViT across multiple downstream tasks, including object detection, semantic segmentation, and change detection. Experiments demonstrated that PIEViT enhances the representation of internal patch features, providing significant improvements over existing self-supervised baselines. It achieves excellent results in object detection, land cover classification, and change detection, underscoring its robustness, generalization, and transferability for remote sensing image interpretation tasks.

Title: Detecting Reference Errors in Scientific Literature with Large Language Models

• Authors: Tianmai M. Zhang, Neil F. Abernethy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06101
• Pdf URL: https://arxiv.org/pdf/2411.06101
• Copy Paste: [[2411.06101]] Detecting Reference Errors in Scientific Literature with Large Language Models(https://arxiv.org/abs/2411.06101)
• Keywords: large language model
• Abstract: Reference errors, such as citation and quotation errors, are common in scientific papers. Such errors can result in the propagation of inaccurate information, but are difficult and time-consuming to detect, posing a significant challenge to scientific publishing. To support automatic detection of reference errors, this work evaluated the ability of large language models in OpenAI's GPT family to detect quotation errors. Specifically, we prepared an expert-annotated, general-domain dataset of statement-reference pairs from journal articles. Large language models were evaluated in different settings with varying amounts of reference information provided by retrieval augmentation. Our results showed that large language models are able to detect erroneous citations with limited context and without fine-tuning. This study contributes to the growing literature that seeks to utilize artificial intelligence to assist in the writing, reviewing, and publishing of scientific papers. Potential avenues for further improvements in this task are also discussed.

Title: Scalable, Tokenization-Free Diffusion Model Architectures with Efficient Initial Convolution and Fixed-Size Reusable Structures for On-Device Image Generation

• Authors: Sanchar Palit, Sathya Veera Reddy Dendi, Mallikarjuna Talluri, Raj Narayana Gadde
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06119
• Pdf URL: https://arxiv.org/pdf/2411.06119
• Copy Paste: [[2411.06119]] Scalable, Tokenization-Free Diffusion Model Architectures with Efficient Initial Convolution and Fixed-Size Reusable Structures for On-Device Image Generation(https://arxiv.org/abs/2411.06119)
• Keywords: diffusion, transformer
• Abstract: Vision Transformers and U-Net architectures have been widely adopted in the implementation of Diffusion Models. However, each architecture presents specific challenges while realizing them on-device. Vision Transformers require positional embedding to maintain correspondence between the tokens processed by the transformer, although they offer the advantage of using fixed-size, reusable repetitive blocks following tokenization. The U-Net architecture lacks these attributes, as it utilizes variable-sized intermediate blocks for down-convolution and up-convolution in the noise estimation backbone for the diffusion process. To address these issues, we propose an architecture that utilizes a fixed-size, reusable transformer block as a core structure, making it more suitable for hardware implementation. Our architecture is characterized by low complexity, token-free design, absence of positional embeddings, uniformity, and scalability, making it highly suitable for deployment on mobile and resource-constrained devices. The proposed model exhibit competitive and consistent performance across both unconditional and conditional image generation tasks. The model achieved a state-of-the-art FID score of 1.6 on unconditional image generation with the CelebA.

Title: A Sharded Blockchain-Based Secure Federated Learning Framework for LEO Satellite Networks

• Authors: Wenbo Wu, Cheng Tan, Kangcheng Yang, Zhishu Shen, Qiushi Zheng, Jiong Jin
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.06137
• Pdf URL: https://arxiv.org/pdf/2411.06137
• Copy Paste: [[2411.06137]] A Sharded Blockchain-Based Secure Federated Learning Framework for LEO Satellite Networks(https://arxiv.org/abs/2411.06137)
• Keywords: secure, security, attack, robust, federate
• Abstract: Low Earth Orbit (LEO) satellite networks are increasingly essential for space-based artificial intelligence (AI) applications. However, as commercial use expands, LEO satellite networks face heightened cyberattack risks, especially through satellite-to-satellite communication links, which are more vulnerable than ground-based connections. As the number of operational satellites continues to grow, addressing these security challenges becomes increasingly critical. Traditional approaches, which focus on sending models to ground stations for validation, often overlook the limited communication windows available to LEO satellites, leaving critical security risks unaddressed. To tackle these challenges, we propose a sharded blockchain-based federated learning framework for LEO networks, called SBFL-LEO. This framework improves the reliability of inter-satellite communications using blockchain technology and assigns specific roles to each satellite. Miner satellites leverage cosine similarity (CS) and Density-Based Spatial Clustering of Applications with Noise (DBSCAN) to identify malicious models and monitor each other to detect inaccurate aggregated models. Security analysis and experimental results demonstrate that our approach outperforms baseline methods in both model accuracy and energy efficiency, significantly enhancing system robustness against attacks.

Title: A Critical Analysis of Foundations, Challenges and Directions for Zero Trust Security in Cloud Environments

• Authors: Ganiyu Oladimeji
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2411.06139
• Pdf URL: https://arxiv.org/pdf/2411.06139
• Copy Paste: [[2411.06139]] A Critical Analysis of Foundations, Challenges and Directions for Zero Trust Security in Cloud Environments(https://arxiv.org/abs/2411.06139)
• Keywords: security, privacy, segmentation
• Abstract: This review discusses the theoretical frameworks and application prospects of Zero Trust Security (ZTS) in cloud computing context. This is because, as organisations move more of their applications and data to the cloud, the old borders-based security model that many implemented are inadequate, therefore a model that has a trust no one, verify everything approach is required. This paper analyzes the core principles of ZTS, including micro-segmentation, least privileged access, and continuous monitoring, while critically examining four major controversies: scalability issues, Economics, Integration issues with existing systems, and Compliance to legal requirements. In this paper, having reviewed the existing literature in the field and various implementation cases, the main barriers to implementing zero trust security were outlined, including the dimensions of decreased performance in large-scale production and the need for major upfront investments that can be difficult for small companies to meet effectively. This research shows that there is no clear correlation between security effectiveness and operational efficiency: while organisations experience up to 40% decrease of security incidents after implementation, they note first negative impacts on performance. This study also shows that to support ZTS there is a need to address the context as the economics and operations of ZTS differ in strengths depending on the size of the organizations and the infrastructures. Some of these are: performance enhancement and optimizations, economic optimization, architectural blend, and privacy-preserving technologies. This review enriches the existing literature on cloud security by presenting both the theoretical framework of ZTS and the observed issues, and provides suggestions useful for future research and practice in the construction of the cloud security architecture.

Title: Aquila-plus: Prompt-Driven Visual-Language Models for Pixel-Level Remote Sensing Image Understanding

• Authors: Kaixuan Lu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06142
• Pdf URL: https://arxiv.org/pdf/2411.06142
• Copy Paste: [[2411.06142]] Aquila-plus: Prompt-Driven Visual-Language Models for Pixel-Level Remote Sensing Image Understanding(https://arxiv.org/abs/2411.06142)
• Keywords: large language model
• Abstract: The recent development of vision language models (VLMs) has led to significant advances in visual-language integration through visual instruction tuning, and they have rapidly evolved in the field of remote sensing image understanding, demonstrating their powerful capabilities. However, existing RSVLMs mainly focus on image-level or frame-level understanding, making it difficult to achieve fine-grained pixel-level visual-language alignment. Additionally, the lack of mask-based instructional data limits their further development. In this paper, we propose a mask-text instruction tuning method called Aquila-plus, which extends the capabilities of RSVLMs to achieve pixel-level visual understanding by incorporating fine-grained mask regions into language instructions. To achieve this, we first meticulously constructed a mask region-text dataset containing 100K samples, and then designed a visual-language model by injecting pixel-level representations into a large language model (LLM). Specifically, Aquila-plus uses a convolutional CLIP as the visual encoder and employs a mask-aware visual extractor to extract precise visual mask features from high-resolution inputs. Experimental results demonstrate that Aquila-plus outperforms existing methods in various region understanding tasks, showcasing its novel capabilities in pixel-level instruction tuning.

Title: Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust

• Authors: Vera Pavlova, Mohammed Makhlouf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06151
• Pdf URL: https://arxiv.org/pdf/2411.06151
• Copy Paste: [[2411.06151]] Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust(https://arxiv.org/abs/2411.06151)
• Keywords: large language model
• Abstract: The widespread use of large language models (LLMs) has dramatically improved many applications of Natural Language Processing (NLP), including Information Retrieval (IR). However, domains that are not driven by commercial interest often lag behind in benefiting from AI-powered solutions. One such area is religious and heritage corpora. Alongside similar domains, Islamic literature holds significant cultural value and is regularly utilized by scholars and the general public. Navigating this extensive amount of text is challenging, and there is currently no unified resource that allows for easy searching of this data using advanced AI tools. This work focuses on the development of a multilingual non-profit IR system for the Islamic domain. This process brings a few major challenges, such as preparing multilingual domain-specific corpora when data is limited in certain languages, deploying a model on resource-constrained devices, and enabling fast search on a limited budget. By employing methods like continued pre-training for domain adaptation and language reduction to decrease model size, a lightweight multilingual retrieval model was prepared, demonstrating superior performance compared to larger models pre-trained on general domain data. Furthermore, evaluating the proposed architecture that utilizes Rust Language capabilities shows the possibility of implementing efficient semantic search in a low-resource setting.

Title: From References to Insights: Collaborative Knowledge Minigraph Agents for Automating Scholarly Literature Review

• Authors: Zhi Zhang, Yan Liu, Sheng-hua Zhong, Gong Chen, Yu Yang, Jiannong Cao
• Subjects: cs.CL, cs.CE
• Abstract URL: https://arxiv.org/abs/2411.06159
• Pdf URL: https://arxiv.org/pdf/2411.06159
• Copy Paste: [[2411.06159]] From References to Insights: Collaborative Knowledge Minigraph Agents for Automating Scholarly Literature Review(https://arxiv.org/abs/2411.06159)
• Keywords: large language model
• Abstract: Literature reviews play a crucial role in scientific research for understanding the current state of research, identifying gaps, and guiding future studies on specific topics. However, the process of conducting a comprehensive literature review is yet time-consuming. This paper proposes a novel framework, collaborative knowledge minigraph agents (CKMAs), to automate scholarly literature reviews. A novel prompt-based algorithm, the knowledge minigraph construction agent (KMCA), is designed to identify relationships between information pieces from academic literature and automatically constructs knowledge minigraphs. By leveraging the capabilities of large language models on constructed knowledge minigraphs, the multiple path summarization agent (MPSA) efficiently organizes information pieces and relationships from different viewpoints to generate literature review paragraphs. We evaluate CKMAs on three benchmark datasets. Experimental results demonstrate that the proposed techniques generate informative, complete, consistent, and insightful summaries for different research problems, promoting the use of LLMs in more professional fields.

Title: SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models

• Authors: Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, Jinqiao Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06171
• Pdf URL: https://arxiv.org/pdf/2411.06171
• Copy Paste: [[2411.06171]] SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models(https://arxiv.org/abs/2411.06171)
• Keywords: large language model
• Abstract: Continual learning (CL) is crucial for language models to dynamically adapt to the evolving real-world demands. To mitigate the catastrophic forgetting problem in CL, data replay has been proven a simple and effective strategy, and the subsequent data-replay-based distillation can further enhance the performance. However, existing methods fail to fully exploit the knowledge embedded in models from previous tasks, resulting in the need for a relatively large number of replay samples to achieve good results. In this work, we first explore and emphasize the importance of attention weights in knowledge retention, and then propose a SElective attEntion-guided Knowledge Retention method (SEEKR) for data-efficient replay-based continual learning of large language models (LLMs). Specifically, SEEKR performs attention distillation on the selected attention heads for finer-grained knowledge retention, where the proposed forgettability-based and task-sensitivity-based measures are used to identify the most valuable attention heads. Experimental results on two continual learning benchmarks for LLMs demonstrate the superiority of SEEKR over the existing methods on both performance and efficiency. Explicitly, SEEKR achieves comparable or even better performance with only 1/10 of the replayed data used by other methods, and reduces the proportion of replayed data to 1%.

Title: IDU-Detector: A Synergistic Framework for Robust Masquerader Attack Detection

• Authors: Zilin Huang, Xiulai Li, Xinyi Cao, Ke Chen, Longjuan Wang, Logan Bo-Yee Liu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.06172
• Pdf URL: https://arxiv.org/pdf/2411.06172
• Copy Paste: [[2411.06172]] IDU-Detector: A Synergistic Framework for Robust Masquerader Attack Detection(https://arxiv.org/abs/2411.06172)
• Keywords: security, privacy, attack, robust
• Abstract: In the digital age, users store personal data in corporate databases, making data security central to enterprise management. Given the extensive attack surface, assets face challenges like weak authentication, vulnerabilities, and malware. Attackers may exploit vulnerabilities to gain unauthorized access, masquerading as legitimate users. Such attacks can lead to privacy breaches, business disruption, financial losses, and reputational damage. Complex attack vectors blur lines between insider and external threats. To address this, we introduce the IDU-Detector, integrating Intrusion Detection Systems (IDS) with User and Entity Behavior Analytics (UEBA). This integration monitors unauthorized access, bridges system gaps, ensures continuous monitoring, and enhances threat identification. Existing insider threat datasets lack depth and coverage of diverse attack vectors. This hinders detection technologies from addressing complex attack surfaces. We propose new, diverse datasets covering more attack scenarios, enhancing detection technologies. Testing our framework, the IDU-Detector achieved average accuracies of 98.96% and 99.12%. These results show effectiveness in detecting attacks, improving security and response speed, and providing higher asset safety assurance.

Title: State Chrono Representation for Enhancing Generalization in Reinforcement Learning

• Authors: Jianda Chen, Wen Zheng Terence Ng, Zichen Chen, Sinno Jialin Pan, Tianwei Zhang
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.06174
• Pdf URL: https://arxiv.org/pdf/2411.06174
• Copy Paste: [[2411.06174]] State Chrono Representation for Enhancing Generalization in Reinforcement Learning(https://arxiv.org/abs/2411.06174)
• Keywords: robust
• Abstract: In reinforcement learning with image-based inputs, it is crucial to establish a robust and generalizable state representation. Recent advancements in metric learning, such as deep bisimulation metric approaches, have shown promising results in learning structured low-dimensional representation space from pixel observations, where the distance between states is measured based on task-relevant features. However, these approaches face challenges in demanding generalization tasks and scenarios with non-informative rewards. This is because they fail to capture sufficient long-term information in the learned representations. To address these challenges, we propose a novel State Chrono Representation (SCR) approach. SCR augments state metric-based representations by incorporating extensive temporal information into the update step of bisimulation metric learning. It learns state distances within a temporal framework that considers both future dynamics and cumulative rewards over current and long-term future states. Our learning strategy effectively incorporates future behavioral information into the representation space without introducing a significant number of additional parameters for modeling dynamics. Extensive experiments conducted in DeepMind Control and Meta-World environments demonstrate that SCR achieves better performance comparing to other recent metric-based methods in demanding generalization tasks. The codes of SCR are available in this https URL.

Title: BM-PAW: A Profitable Mining Attack in the PoW-based Blockchain System

• Authors: Junjie Hu, Xunzhi Chen, Huan Yan, Na Ruan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.06187
• Pdf URL: https://arxiv.org/pdf/2411.06187
• Copy Paste: [[2411.06187]] BM-PAW: A Profitable Mining Attack in the PoW-based Blockchain System(https://arxiv.org/abs/2411.06187)
• Keywords: security, attack
• Abstract: Mining attacks enable an adversary to procure a disproportionately large portion of mining rewards by deviating from honest mining practices within the PoW-based blockchain system. In this paper, we demonstrate that the security vulnerabilities of PoW-based blockchain extend beyond what these mining attacks initially reveal. We introduce a novel mining strategy, named BM-PAW, which yields superior rewards for both the attacker and the targeted pool compared to the state-of-the-art mining attack: PAW. Our analysis reveals that BM-PAW attackers are incentivized to offer appropriate bribe money to other targets, as they comply with the attacker's directives upon receiving payment. We find the BM-PAW attacker can circumvent the "miner's dilemma" through equilibrium analysis in a two-pool BM-PAW game scenario, wherein the outcome is determined by the attacker's mining power. We finally propose practical countermeasures to mitigate these novel pool attacks.

Title: Text2CAD: Text to 3D CAD Generation via Technical Drawings

• Authors: Mohsen Yavartanoo, Sangmin Hong, Reyhaneh Neshatavar, Kyoung Mu Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06206
• Pdf URL: https://arxiv.org/pdf/2411.06206
• Copy Paste: [[2411.06206]] Text2CAD: Text to 3D CAD Generation via Technical Drawings(https://arxiv.org/abs/2411.06206)
• Keywords: diffusion
• Abstract: The generation of industrial Computer-Aided Design (CAD) models from user requests and specifications is crucial to enhancing efficiency in modern manufacturing. Traditional methods of CAD generation rely heavily on manual inputs and struggle with complex or non-standard designs, making them less suited for dynamic industrial needs. To overcome these challenges, we introduce Text2CAD, a novel framework that employs stable diffusion models tailored to automate the generation process and efficiently bridge the gap between user specifications in text and functional CAD models. This approach directly translates the user's textural descriptions into detailed isometric images, which are then precisely converted into orthographic views, e.g., top, front, and side, providing sufficient information to reconstruct 3D CAD models. This process not only streamlines the creation of CAD models from textual descriptions but also ensures that the resulting models uphold physical and dimensional consistency essential for practical engineering applications. Our experimental results show that Text2CAD effectively generates technical drawings that are accurately translated into high-quality 3D CAD models, showing substantial potential to revolutionize CAD automation in response to user demands.

Title: Exploring Knowledge Boundaries in Large Language Models for Retrieval Judgment

• Authors: Zhen Zhang, Xinyu Wang, Yong Jiang, Zhuo Chen, Feiteng Mu, Mengting Hu, Pengjun Xie, Fei Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06207
• Pdf URL: https://arxiv.org/pdf/2411.06207
• Copy Paste: [[2411.06207]] Exploring Knowledge Boundaries in Large Language Models for Retrieval Judgment(https://arxiv.org/abs/2411.06207)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are increasingly recognized for their practical applications. However, these models often encounter challenges in dynamically changing knowledge, as well as in managing unknown static knowledge. Retrieval-Augmented Generation (RAG) tackles this challenge and has shown a significant impact on LLMs. Actually, we find that the impact of RAG on the question answering capabilities of LLMs can be categorized into three groups: beneficial, neutral, and harmful. By minimizing retrieval requests that yield neutral or harmful results, we can effectively reduce both time and computational costs, while also improving the overall performance of LLMs. This insight motivates us to differentiate between types of questions using certain metrics as indicators, to decrease the retrieval ratio without compromising performance. In our work, we propose a method that is able to identify different types of questions from this view by training a Knowledge Boundary Model (KBM). Experiments conducted on 11 English and Chinese datasets illustrate that the KBM effectively delineates the knowledge boundary, significantly decreasing the proportion of retrievals required for optimal end-to-end performance. Specifically, we evaluate the effectiveness of KBM in three complex scenarios: dynamic knowledge, long-tail static knowledge, and multi-hop problems, as well as its functionality as an external LLM plug-in.

Title: IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization

• Authors: Xinghua Zhang, Haiyang Yu, Cheng Fu, Fei Huang, Yongbin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06208
• Pdf URL: https://arxiv.org/pdf/2411.06208
• Copy Paste: [[2411.06208]] IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization(https://arxiv.org/abs/2411.06208)
• Keywords: large language model
• Abstract: In the realm of large language models (LLMs), the ability of models to accurately follow instructions is paramount as more agents and applications leverage LLMs for construction, where the complexity of instructions are rapidly increasing. However, on the one hand, there is only a certain amount of complex instruction evaluation data; on the other hand, there are no dedicated algorithms to improve the ability to follow complex instructions. To this end, this paper introduces TRACE, a benchmark for improving and evaluating the complex instructionfollowing ability, which consists of 120K training data and 1K evaluation data. Furthermore, we propose IOPO (Input-Output Preference Optimization) alignment method which takes both input and output preference pairs into consideration, where LLMs not only rapidly align with response preferences but also meticulously explore the instruction preferences. Extensive experiments on both in-domain and outof-domain datasets confirm the effectiveness of IOPO, showing 8.15%, 2.18% improvements on in-domain data and 6.29%, 3.13% on outof-domain data compared to SFT and DPO respectively.

Title: Multistage non-deterministic classification using secondary concept graphs and graph convolutional networks for high-level feature extraction

• Authors: Masoud Kargar, Nasim Jelodari, Alireza Assadzadeh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06212
• Pdf URL: https://arxiv.org/pdf/2411.06212
• Copy Paste: [[2411.06212]] Multistage non-deterministic classification using secondary concept graphs and graph convolutional networks for high-level feature extraction(https://arxiv.org/abs/2411.06212)
• Keywords: extraction
• Abstract: Graphs, comprising nodes and edges, visually depict relationships and structures, posing challenges in extracting high-level features due to their intricate connections. Multiple connections introduce complexities in discovering patterns, where node weights may affect some features more than others. In domains with diverse topics, graph representations illustrate interrelations among features. Pattern discovery within graphs is recognized as NP-hard. Graph Convolutional Networks (GCNs) are a prominent deep learning approach for acquiring meaningful representations by leveraging node connectivity and characteristics. Despite achievements, predicting and assigning 9 deterministic classes often involves errors. To address this challenge, we present a multi-stage non-deterministic classification method based on a secondary conceptual graph and graph convolutional networks, which includes distinct steps: 1) leveraging GCN for the extraction and generation of 12 high-level features: 2) employing incomplete, non-deterministic models for feature extraction, conducted before reaching a definitive prediction: and 3) formulating definitive forecasts grounded in conceptual (logical) graphs. The empirical findings indicate that our proposed approach outperforms contemporary methods in classification tasks. Across three datasets Cora, Citeseer, and PubMed the achieved accuracies are 96%, 93%, and 95%, respectively. Code is available at this https URL.

Title: Incorporating Human Explanations for Robust Hate Speech Detection

• Authors: Jennifer L. Chen, Faisal Ladhak, Daniel Li, Noémie Elhadad
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06213
• Pdf URL: https://arxiv.org/pdf/2411.06213
• Copy Paste: [[2411.06213]] Incorporating Human Explanations for Robust Hate Speech Detection(https://arxiv.org/abs/2411.06213)
• Keywords: robust, transformer
• Abstract: Given the black-box nature and complexity of large transformer language models (LM), concerns about generalizability and robustness present ethical implications for domains such as hate speech (HS) detection. Using the content rich Social Bias Frames dataset, containing human-annotated stereotypes, intent, and targeted groups, we develop a three stage analysis to evaluate if LMs faithfully assess hate speech. First, we observe the need for modeling contextually grounded stereotype intents to capture implicit semantic meaning. Next, we design a new task, Stereotype Intent Entailment (SIE), which encourages a model to contextually understand stereotype presence. Finally, through ablation tests and user studies, we find a SIE objective improves content understanding, but challenges remain in modeling implicit intent.

Title: Early Prediction of Natural Gas Pipeline Leaks Using the MKTCN Model

• Authors: Xuguang Li, Zhonglin Zuo, Zheng Dong, Yang Yang
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.06214
• Pdf URL: https://arxiv.org/pdf/2411.06214
• Copy Paste: [[2411.06214]] Early Prediction of Natural Gas Pipeline Leaks Using the MKTCN Model(https://arxiv.org/abs/2411.06214)
• Keywords: robust
• Abstract: Natural gas pipeline leaks pose severe risks, leading to substantial economic losses and potential hazards to human safety. In this study, we develop an accurate model for the early prediction of pipeline leaks. To the best of our knowledge, unlike previous anomaly detection, this is the first application to use internal pipeline data for early prediction of leaks. The modeling process addresses two main challenges: long-term dependencies and sample imbalance. First, we introduce a dilated convolution-based prediction model to capture long-term dependencies, as dilated convolution expands the model's receptive field without added computational cost. Second, to mitigate sample imbalance, we propose the MKTCN model, which incorporates the Kolmogorov-Arnold Network as the fully connected layer in a dilated convolution model, enhancing network generalization. Finally, we validate the MKTCN model through extensive experiments on two real-world datasets. Results demonstrate that MKTCN outperforms in generalization and classification, particularly under severe data imbalance, and effectively predicts leaks up to 5000 seconds in advance. Overall, the MKTCN model represents a significant advancement in early pipeline leak prediction, providing robust generalization and improved modeling of the long-term dependencies inherent in multi-dimensional time-series data.

Title: Smart-LLaMA: Two-Stage Post-Training of Large Language Models for Smart Contract Vulnerability Detection and Explanation

• Authors: Lei Yu, Shiqi Chen, Hang Yuan, Peng Wang, Zhirong Huang, Jingyuan Zhang, Chenjie Shen, Fengjun Zhang, Li Yang, Jiajia Ma
• Subjects: cs.CR, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.06221
• Pdf URL: https://arxiv.org/pdf/2411.06221
• Copy Paste: [[2411.06221]] Smart-LLaMA: Two-Stage Post-Training of Large Language Models for Smart Contract Vulnerability Detection and Explanation(https://arxiv.org/abs/2411.06221)
• Keywords: security, large language model
• Abstract: With the rapid development of blockchain technology, smart contract security has become a critical challenge. Existing smart contract vulnerability detection methods face three main issues: (1) Insufficient quality of datasets, lacking detailed explanations and precise vulnerability locations. (2) Limited adaptability of large language models (LLMs) to the smart contract domain, as most LLMs are pre-trained on general text data but minimal smart contract-specific data. (3) Lack of high-quality explanations for detected vulnerabilities, as existing methods focus solely on detection without clear explanations. These limitations hinder detection performance and make it harder for developers to understand and fix vulnerabilities quickly, potentially leading to severe financial losses. To address these problems, we propose Smart-LLaMA, an advanced detection method based on the LLaMA language model. First, we construct a comprehensive dataset covering four vulnerability types with labels, detailed explanations, and precise vulnerability locations. Second, we introduce Smart Contract-Specific Continual Pre-Training, using raw smart contract data to enable the LLM to learn smart contract syntax and semantics, enhancing their domain adaptability. Furthermore, we propose Explanation-Guided Fine-Tuning, which fine-tunes the LLM using paired vulnerable code and explanations, enabling both vulnerability detection and reasoned explanations. We evaluate explanation quality through LLM and human evaluation, focusing on Correctness, Completeness, and Conciseness. Experimental results show that Smart-LLaMA outperforms state-of-the-art baselines, with average improvements of 6.49% in F1 score and 3.78% in accuracy, while providing reliable explanations.

Title: Crowd3D++: Robust Monocular Crowd Reconstruction with Upright Space

• Authors: Jing Huang, Hao Wen, Tianyi Zhou, Haozhe Lin, Yu-Kun Lai, Kun Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06232
• Pdf URL: https://arxiv.org/pdf/2411.06232
• Copy Paste: [[2411.06232]] Crowd3D++: Robust Monocular Crowd Reconstruction with Upright Space(https://arxiv.org/abs/2411.06232)
• Keywords: robust
• Abstract: This paper aims to reconstruct hundreds of people's 3D poses, shapes, and locations from a single image with unknown camera parameters. Due to the small and highly varying 2D human scales, depth ambiguity, and perspective distortion, no existing methods can achieve globally consistent reconstruction and accurate reprojection. To address these challenges, we first propose Crowd3D, which leverages a new concept, Human-scene Virtual Interaction Point (HVIP), to convert the complex 3D human localization into 2D-pixel localization with robust camera and ground estimation to achieve globally consistent reconstruction. To achieve stable generalization on different camera FoVs without test-time optimization, we propose an extended version, Crowd3D++, which eliminates the influence of camera parameters and the cropping operation by the proposed canonical upright space and ground-aware normalization transform. In the defined upright space, Crowd3D++ also designs an HVIPNet to regress 2D HVIP and infer the depths. Besides, we contribute two benchmark datasets, LargeCrowd and SyntheticCrowd, for evaluating crowd reconstruction in large scenes. The source code and data will be made publicly available after acceptance.

Title: Zero-Shot NAS via the Suppression of Local Entropy Decrease

• Authors: Ning Wu, Han Huang, Yueting Xu, Zhifeng Hao
• Subjects: cs.LG, cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2411.06236
• Pdf URL: https://arxiv.org/pdf/2411.06236
• Copy Paste: [[2411.06236]] Zero-Shot NAS via the Suppression of Local Entropy Decrease(https://arxiv.org/abs/2411.06236)
• Keywords: data-free
• Abstract: Architecture performance evaluation is the most time-consuming part of neural architecture search (NAS). Zero-Shot NAS accelerates the evaluation by utilizing zero-cost proxies instead of training. Though effective, existing zero-cost proxies require invoking backpropagations or running networks on input data, making it difficult to further accelerate the computation of proxies. To alleviate this issue, architecture topologies are used to evaluate the performance of networks in this study. We prove that particular architectural topologies decrease the local entropy of feature maps, which degrades specific features to a bias, thereby reducing network performance. Based on this proof, architectural topologies are utilized to quantify the suppression of local entropy decrease (SED) as a data-free and running-free proxy. Experimental results show that SED outperforms most state-of-the-art proxies in terms of architecture selection on five benchmarks, with computation time reduced by three orders of magnitude. We further compare the SED-based NAS with state-of-the-art proxies. SED-based NAS selects the architecture with higher accuracy and fewer parameters in only one second. The theoretical analyses of local entropy and experimental results demonstrate that the suppression of local entropy decrease facilitates selecting optimal architectures in Zero-Shot NAS.

Title: Web Scale Graph Mining for Cyber Threat Intelligence

• Authors: Scott Freitas, Amir Gharib
• Subjects: cs.CR, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2411.06239
• Pdf URL: https://arxiv.org/pdf/2411.06239
• Copy Paste: [[2411.06239]] Web Scale Graph Mining for Cyber Threat Intelligence(https://arxiv.org/abs/2411.06239)
• Keywords: security, attack
• Abstract: Defending against today's increasingly sophisticated and large-scale cyberattacks demands accurate, real-time threat intelligence. Traditional approaches struggle to scale, integrate diverse telemetry, and adapt to a constantly evolving security landscape. We introduce Threat Intelligence Tracking via Adaptive Networks (TITAN), an industry-scale graph mining framework that generates cyber threat intelligence at unprecedented speed and scale. TITAN introduces a suite of innovations specifically designed to address the complexities of the modern security landscape, including: (1) a dynamic threat intelligence graph that maps the intricate relationships between millions of entities, incidents, and organizations; (2) real-time update mechanisms that automatically decay and prune outdated intel; (3) integration of security domain knowledge to bootstrap initial reputation scores; and (4) reputation propagation algorithms that uncover hidden threat actor infrastructure. Integrated into Microsoft Unified Security Operations Platform (USOP), which is deployed across hundreds of thousands of organizations worldwide, TITAN's threat intelligence powers key detection and disruption capabilities. With an impressive average macro-F1 score of 0.89 and a precision-recall AUC of 0.94, TITAN identifies millions of high-risk entities each week, enabling a 6x increase in non-file threat intelligence. Since its deployment, TITAN has increased the product's incident disruption rate by a remarkable 21%, while reducing the time to disrupt by a factor of 1.9x, and maintaining 99% precision, as confirmed by customer feedback and thorough manual evaluation by security experts--ultimately saving customers from costly security breaches.

Title: Robust Detection of LLM-Generated Text: A Comparative Analysis

• Authors: Yongye Su, Yuqing Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06248
• Pdf URL: https://arxiv.org/pdf/2411.06248
• Copy Paste: [[2411.06248]] Robust Detection of LLM-Generated Text: A Comparative Analysis(https://arxiv.org/abs/2411.06248)
• Keywords: protect, attack, robust, large language model
• Abstract: The ability of large language models to generate complex texts allows them to be widely integrated into many aspects of life, and their output can quickly fill all network resources. As the impact of LLMs grows, it becomes increasingly important to develop powerful detectors for the generated text. This detector is essential to prevent the potential misuse of these technologies and to protect areas such as social media from the negative effects of false content generated by LLMS. The main goal of LLM-generated text detection is to determine whether text is generated by an LLM, which is a basic binary classification task. In our work, we mainly use three different classification methods based on open source datasets: traditional machine learning techniques such as logistic regression, k-means clustering, Gaussian Naive Bayes, support vector machines, and methods based on converters such as BERT, and finally algorithms that use LLMs to detect LLM-generated text. We focus on model generalization, potential adversarial attacks, and accuracy of model evaluation. Finally, the possible research direction in the future is proposed, and the current experimental results are summarized.

Title: Federated Split Learning for Human Activity Recognition with Differential Privacy

• Authors: Josue Ndeko, Shaba Shaon, Aubrey Beal, Avimanyu Sahoo, Dinh C. Nguyen
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.06263
• Pdf URL: https://arxiv.org/pdf/2411.06263
• Copy Paste: [[2411.06263]] Federated Split Learning for Human Activity Recognition with Differential Privacy(https://arxiv.org/abs/2411.06263)
• Keywords: privacy, federate
• Abstract: This paper proposes a novel intelligent human activity recognition (HAR) framework based on a new design of Federated Split Learning (FSL) with Differential Privacy (DP) over edge networks. Our FSL-DP framework leverages both accelerometer and gyroscope data, achieving significant improvements in HAR accuracy. The evaluation includes a detailed comparison between traditional Federated Learning (FL) and our FSL framework, showing that the FSL framework outperforms FL models in both accuracy and loss metrics. Additionally, we examine the privacy-performance trade-off under different data settings in the DP mechanism, highlighting the balance between privacy guarantees and model accuracy. The results also indicate that our FSL framework achieves faster communication times per training round compared to traditional FL, further emphasizing its efficiency and effectiveness. This work provides valuable insight and a novel framework which was tested on a real-life dataset.

Title: Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models

• Authors: Xiaojun Wu, Junxi Liu, Huanyi Su, Zhouchi Lin, Yiyan Qi, Chengjin Xu, Jiajun Su, Jiajie Zhong, Fuwei Wang, Saizhuo Wang, Fengrui Hua, Jia Li, Jian Guo
• Subjects: cs.CL, cs.CE
• Abstract URL: https://arxiv.org/abs/2411.06272
• Pdf URL: https://arxiv.org/pdf/2411.06272
• Copy Paste: [[2411.06272]] Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models(https://arxiv.org/abs/2411.06272)
• Keywords: large language model
• Abstract: As large language models become increasingly prevalent in the financial sector, there is a pressing need for a standardized method to comprehensively assess their performance. However, existing finance benchmarks often suffer from limited language and task coverage, as well as challenges such as low-quality datasets and inadequate adaptability for LLM evaluation. To address these limitations, we propose "Golden Touchstone", the first comprehensive bilingual benchmark for financial LLMs, which incorporates representative datasets from both Chinese and English across eight core financial NLP tasks. Developed from extensive open source data collection and industry-specific demands, this benchmark includes a variety of financial tasks aimed at thoroughly assessing models' language understanding and generation capabilities. Through comparative analysis of major models on the benchmark, such as GPT-4o Llama3, FinGPT and FinMA, we reveal their strengths and limitations in processing complex financial information. Additionally, we open-sourced Touchstone-GPT, a financial LLM trained through continual pre-training and financial instruction tuning, which demonstrates strong performance on the bilingual benchmark but still has limitations in specific this http URL research not only provides the financial large language models with a practical evaluation tool but also guides the development and optimization of future research. The source code for Golden Touchstone and model weight of Touchstone-GPT have been made publicly available at \url{this https URL}, contributing to the ongoing evolution of FinLLMs and fostering further research in this critical area.

Title: SPIKANs: Separable Physics-Informed Kolmogorov-Arnold Networks

• Authors: Bruno Jacob, Amanda A. Howard, Panos Stinis
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2411.06286
• Pdf URL: https://arxiv.org/pdf/2411.06286
• Copy Paste: [[2411.06286]] SPIKANs: Separable Physics-Informed Kolmogorov-Arnold Networks(https://arxiv.org/abs/2411.06286)
• Keywords: interpretability
• Abstract: Physics-Informed Neural Networks (PINNs) have emerged as a promising method for solving partial differential equations (PDEs) in scientific computing. While PINNs typically use multilayer perceptrons (MLPs) as their underlying architecture, recent advancements have explored alternative neural network structures. One such innovation is the Kolmogorov-Arnold Network (KAN), which has demonstrated benefits over traditional MLPs, including faster neural scaling and better interpretability. The application of KANs to physics-informed learning has led to the development of Physics-Informed KANs (PIKANs), enabling the use of KANs to solve PDEs. However, despite their advantages, KANs often suffer from slower training speeds, particularly in higher-dimensional problems where the number of collocation points grows exponentially with the dimensionality of the system. To address this challenge, we introduce Separable Physics-Informed Kolmogorov-Arnold Networks (SPIKANs). This novel architecture applies the principle of separation of variables to PIKANs, decomposing the problem such that each dimension is handled by an individual KAN. This approach drastically reduces the computational complexity of training without sacrificing accuracy, facilitating their application to higher-dimensional PDEs. Through a series of benchmark problems, we demonstrate the effectiveness of SPIKANs, showcasing their superior scalability and performance compared to PIKANs and highlighting their potential for solving complex, high-dimensional PDEs in scientific computing.

Title: Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models

• Authors: Arshia Hemmat, Adam Davies, Tom A. Lamb, Jianhao Yuan, Philip Torr, Ashkan Khakzar, Francesco Pinto
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06287
• Pdf URL: https://arxiv.org/pdf/2411.06287
• Copy Paste: [[2411.06287]] Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models(https://arxiv.org/abs/2411.06287)
• Keywords: robust
• Abstract: Despite the importance of shape perception in human vision, early neural image classifiers relied less on shape information for object recognition than other (often spurious) features. While recent research suggests that current large Vision-Language Models (VLMs) exhibit more reliance on shape, we find them to still be seriously limited in this regard. To quantify such limitations, we introduce IllusionBench, a dataset that challenges current cutting-edge VLMs to decipher shape information when the shape is represented by an arrangement of visual elements in a scene. Our extensive evaluations reveal that, while these shapes are easily detectable by human annotators, current VLMs struggle to recognize them, indicating important avenues for future work in developing more robust visual perception systems. The full dataset and codebase are available at: \url{this https URL}

Title: TinyML NLP Approach for Semantic Wireless Sentiment Classification

• Authors: Ahmed Y. Radwan, Mohammad Shehab, Mohamed-Slim Alouini
• Subjects: cs.LG, cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2411.06291
• Pdf URL: https://arxiv.org/pdf/2411.06291
• Copy Paste: [[2411.06291]] TinyML NLP Approach for Semantic Wireless Sentiment Classification(https://arxiv.org/abs/2411.06291)
• Keywords: privacy, federate
• Abstract: Natural Language Processing (NLP) operations, such as semantic sentiment analysis and text synthesis, may often impair users' privacy and demand significant on device computational resources. Centralized learning (CL) on the edge offers an alternative energy-efficient approach, yet requires the collection of raw information, which affects the user's privacy. While Federated learning (FL) preserves privacy, it requires high computational energy on board tiny user devices. We introduce split learning (SL) as an energy-efficient alternative, privacy-preserving tiny machine learning (TinyML) scheme and compare it to FL and CL in the presence of Rayleigh fading and additive noise. Our results show that SL reduces processing power and CO2 emissions while maintaining high accuracy, whereas FL offers a balanced compromise between efficiency and privacy. Hence, this study provides insights into deploying energy-efficient, privacy-preserving NLP models on edge devices.

Title: Adaptive Aspect Ratios with Patch-Mixup-ViT-based Vehicle ReID

• Authors: Mei Qiu, Lauren Ann Christopher, Stanley Chien, Lingxi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06297
• Pdf URL: https://arxiv.org/pdf/2411.06297
• Copy Paste: [[2411.06297]] Adaptive Aspect Ratios with Patch-Mixup-ViT-based Vehicle ReID(https://arxiv.org/abs/2411.06297)
• Keywords: robust, transformer
• Abstract: Vision Transformers (ViTs) have shown exceptional performance in vehicle re-identification (ReID) tasks. However, non-square aspect ratios of image or video inputs can negatively impact re-identification accuracy. To address this challenge, we propose a novel, human perception driven, and general ViT-based ReID framework that fuses models trained on various aspect ratios. Our key contributions are threefold: (i) We analyze the impact of aspect ratios on performance using the VeRi-776 and VehicleID datasets, providing guidance for input settings based on the distribution of original image aspect ratios. (ii) We introduce patch-wise mixup strategy during ViT patchification (guided by spatial attention scores) and implement uneven stride for better alignment with object aspect ratios. (iii) We propose a dynamic feature fusion ReID network to enhance model robustness. Our method outperforms state-of-the-art transformer-based approaches on both datasets, with only a minimal increase in inference time per image.

Title: NeuReg: Domain-invariant 3D Image Registration on Human and Mouse Brains

• Authors: Taha Razzaq, Asim Iqbal
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.06315
• Pdf URL: https://arxiv.org/pdf/2411.06315
• Copy Paste: [[2411.06315]] NeuReg: Domain-invariant 3D Image Registration on Human and Mouse Brains(https://arxiv.org/abs/2411.06315)
• Keywords: transformer
• Abstract: Medical brain imaging relies heavily on image registration to accurately curate structural boundaries of brain features for various healthcare applications. Deep learning models have shown remarkable performance in image registration in recent years. Still, they often struggle to handle the diversity of 3D brain volumes, challenged by their structural and contrastive variations and their imaging domains. In this work, we present NeuReg, a Neuro-inspired 3D image registration architecture with the feature of domain invariance. NeuReg generates domain-agnostic representations of imaging features and incorporates a shifting window-based Swin Transformer block as the encoder. This enables our model to capture the variations across brain imaging modalities and species. We demonstrate a new benchmark in multi-domain publicly available datasets comprising human and mouse 3D brain volumes. Extensive experiments reveal that our model (NeuReg) outperforms the existing baseline deep learning-based image registration models and provides a high-performance boost on cross-domain datasets, where models are trained on 'source-only' domain and tested on completely 'unseen' target domains. Our work establishes a new state-of-the-art for domain-agnostic 3D brain image registration, underpinned by Neuro-inspired Transformer-based architecture.

Title: Prompts Matter: Comparing ML/GAI Approaches for Generating Inductive Qualitative Coding Results

• Authors: John Chen, Alexandros Lotsos, Lexie Zhao, Grace Wang, Uri Wilensky, Bruce Sherin, Michael Horn
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.06316
• Pdf URL: https://arxiv.org/pdf/2411.06316
• Copy Paste: [[2411.06316]] Prompts Matter: Comparing ML/GAI Approaches for Generating Inductive Qualitative Coding Results(https://arxiv.org/abs/2411.06316)
• Keywords: generative
• Abstract: Inductive qualitative methods have been a mainstay of education research for decades, yet it takes much time and effort to conduct rigorously. Recent advances in artificial intelligence, particularly with generative AI (GAI), have led to initial success in generating inductive coding results. Like human coders, GAI tools rely on instructions to work, and how to instruct it may matter. To understand how ML/GAI approaches could contribute to qualitative coding processes, this study applied two known and two theory-informed novel approaches to an online community dataset and evaluated the resulting coding results. Our findings show significant discrepancies between ML/GAI approaches and demonstrate the advantage of our approaches, which introduce human coding processes into GAI prompts.

Title: Harpocrates: Oblivious Privacy in a Statically Typed World

• Authors: Sinan Pehlivanoglu, Malte Schwarzkopf
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.06317
• Pdf URL: https://arxiv.org/pdf/2411.06317
• Copy Paste: [[2411.06317]] Harpocrates: Oblivious Privacy in a Statically Typed World(https://arxiv.org/abs/2411.06317)
• Keywords: privacy, protect
• Abstract: In this paper, we introduce Harpocrates, a compiler plugin and a framework pair for Scala that binds the privacy policies to the data during data creation in form of oblivious membranes. Harpocrates eliminates raw data for a policy protected type from the application, ensuring it can only exist in protected form and centralizes the policy checking to the policy declaration site, making the privacy logic easy to maintain and verify. Instead of approaching privacy from an information flow verification perspective, Harpocrates allow the data to flow freely throughout the application, inside the policy membranes but enforces the policies when the data is tried to be accessed, mutated, declassified or passed through the application boundary. The centralization of the policies allow the maintainers to change the enforced logic simply by updating a single function while keeping the rest of the application oblivious to the change. Especially in a setting where the data definition is shared by multiple applications, the publisher can update the policies without requiring the dependent applications to make any changes beyond updating the dependency version.

Title: SEM-Net: Efficient Pixel Modelling for image inpainting with Spatially Enhanced SSM

• Authors: Shuang Chen, Haozheng Zhang, Amir Atapour-Abarghouei, Hubert P. H. Shum
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06318
• Pdf URL: https://arxiv.org/pdf/2411.06318
• Copy Paste: [[2411.06318]] SEM-Net: Efficient Pixel Modelling for image inpainting with Spatially Enhanced SSM(https://arxiv.org/abs/2411.06318)
• Keywords: transformer
• Abstract: Image inpainting aims to repair a partially damaged image based on the information from known regions of the images. \revise{Achieving semantically plausible inpainting results is particularly challenging because it requires the reconstructed regions to exhibit similar patterns to the semanticly consistent regions}. This requires a model with a strong capacity to capture long-range dependencies. Existing models struggle in this regard due to the slow growth of receptive field for Convolutional Neural Networks (CNNs) based methods and patch-level interactions in Transformer-based methods, which are ineffective for capturing long-range dependencies. Motivated by this, we propose SEM-Net, a novel visual State Space model (SSM) vision network, modelling corrupted images at the pixel level while capturing long-range dependencies (LRDs) in state space, achieving a linear computational complexity. To address the inherent lack of spatial awareness in SSM, we introduce the Snake Mamba Block (SMB) and Spatially-Enhanced Feedforward Network. These innovations enable SEM-Net to outperform state-of-the-art inpainting methods on two distinct datasets, showing significant improvements in capturing LRDs and enhancement in spatial consistency. Additionally, SEM-Net achieves state-of-the-art performance on motion deblurring, demonstrating its generalizability. Our source code will be released in this https URL.

Title: CRTRE: Causal Rule Generation with Target Trial Emulation Framework

• Authors: Junda Wang, Weijian Li, Han Wang, Hanjia Lyu, Caroline P. Thirukumaran, Addisu Mesfin, Hong Yu, Jiebo Luo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06338
• Pdf URL: https://arxiv.org/pdf/2411.06338
• Copy Paste: [[2411.06338]] CRTRE: Causal Rule Generation with Target Trial Emulation Framework(https://arxiv.org/abs/2411.06338)
• Keywords: interpretability
• Abstract: Causal inference and model interpretability are gaining increasing attention, particularly in the biomedical domain. Despite recent advance, decorrelating features in nonlinear environments with human-interpretable representations remains underexplored. In this study, we introduce a novel method called causal rule generation with target trial emulation framework (CRTRE), which applies randomize trial design principles to estimate the causal effect of association rules. We then incorporate such association rules for the downstream applications such as prediction of disease onsets. Extensive experiments on six healthcare datasets, including synthetic data, real-world disease collections, and MIMIC-III/IV, demonstrate the model's superior performance. Specifically, our method achieved a $\beta$ error of 0.907, outperforming DWR (1.024) and SVM (1.141). On real-world datasets, our model achieved accuracies of 0.789, 0.920, and 0.300 for Esophageal Cancer, Heart Disease, and Cauda Equina Syndrome prediction task, respectively, consistently surpassing baseline models. On the ICD code prediction tasks, it achieved AUC Macro scores of 92.8 on MIMIC-III and 96.7 on MIMIC-IV, outperforming the state-of-the-art models KEPT and MSMN. Expert evaluations further validate the model's effectiveness, causality, and interpretability.

Title: CityGuessr: City-Level Video Geo-Localization on a Global Scale

• Authors: Parth Parag Kulkarni, Gaurav Kumar Nayak, Mubarak Shah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06344
• Pdf URL: https://arxiv.org/pdf/2411.06344
• Copy Paste: [[2411.06344]] CityGuessr: City-Level Video Geo-Localization on a Global Scale(https://arxiv.org/abs/2411.06344)
• Keywords: transformer
• Abstract: Video geolocalization is a crucial problem in current times. Given just a video, ascertaining where it was captured from can have a plethora of advantages. The problem of worldwide geolocalization has been tackled before, but only using the image modality. Its video counterpart remains relatively unexplored. Meanwhile, video geolocalization has also garnered some attention in the recent past, but the existing methods are all restricted to specific regions. This motivates us to explore the problem of video geolocalization at a global scale. Hence, we propose a novel problem of worldwide video geolocalization with the objective of hierarchically predicting the correct city, state/province, country, and continent, given a video. However, no large scale video datasets that have extensive worldwide coverage exist, to train models for solving this problem. To this end, we introduce a new dataset, CityGuessr68k comprising of 68,269 videos from 166 cities all over the world. We also propose a novel baseline approach to this problem, by designing a transformer-based architecture comprising of an elegant Self-Cross Attention module for incorporating scenes as well as a TextLabel Alignment strategy for distilling knowledge from textlabels in feature space. To further enhance our location prediction, we also utilize soft-scene labels. Finally we demonstrate the performance of our method on our new dataset as well as Mapillary(MSLS). Our code and datasets are available at: this https URL

Title: AMAZE: Accelerated MiMC Hardware Architecture for Zero-Knowledge Applications on the Edge

• Authors: Anees Ahmed, Nojan Sheybani, Davi Moreno, Nges Brian Njungle, Tengkai Gong, Michel Kinsy, Farinaz Koushanfar
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2411.06350
• Pdf URL: https://arxiv.org/pdf/2411.06350
• Copy Paste: [[2411.06350]] AMAZE: Accelerated MiMC Hardware Architecture for Zero-Knowledge Applications on the Edge(https://arxiv.org/abs/2411.06350)
• Keywords: security, privacy
• Abstract: Collision-resistant, cryptographic hash (CRH) functions have long been an integral part of providing security and privacy in modern systems. Certain constructions of zero-knowledge proof (ZKP) protocols aim to utilize CRH functions to perform cryptographic hashing. Standard CRH functions, such as SHA2, are inefficient when employed in the ZKP domain, thus calling for ZK-friendly hashes, which are CRH functions built with ZKP efficiency in mind. The most mature ZK-friendly hash, MiMC, presents a block cipher and hash function with a simple algebraic structure that is well-suited, due to its achieved security and low complexity, for ZKP applications. Although ZK-friendly hashes have improved the performance of ZKP generation in software, the underlying computation of ZKPs, including CRH functions, must be optimized on hardware to enable practical applications. The challenge we address in this work is determining how to efficiently incorporate ZK-friendly hash functions, such as MiMC, into hardware accelerators, thus enabling more practical applications. In this work, we introduce AMAZE, a highly hardware-optimized open-source framework for computing the MiMC block cipher and hash function. Our solution has been primarily directed at resource-constrained edge devices; consequently, we provide several implementations of MiMC with varying power, resource, and latency profiles. Our extensive evaluations show that the AMAZE-powered implementation of MiMC outperforms standard CPU implementations by more than 13$\times$. In all settings, AMAZE enables efficient ZK-friendly hashing on resource-constrained devices. Finally, we highlight AMAZE's underlying open-source arithmetic backend as part of our end-to-end design, thus allowing developers to utilize the AMAZE framework for custom ZKP applications.

Title: Client Contribution Normalization for Enhanced Federated Learning

• Authors: Mayank Kumar Kundalwal, Anurag Saraswat, Ishan Mishra, Deepak Mishra
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06352
• Pdf URL: https://arxiv.org/pdf/2411.06352
• Copy Paste: [[2411.06352]] Client Contribution Normalization for Enhanced Federated Learning(https://arxiv.org/abs/2411.06352)
• Keywords: privacy, robust, federate
• Abstract: Mobile devices, including smartphones and laptops, generate decentralized and heterogeneous data, presenting significant challenges for traditional centralized machine learning models due to substantial communication costs and privacy risks. Federated Learning (FL) offers a promising alternative by enabling collaborative training of a global model across decentralized devices without data sharing. However, FL faces challenges due to statistical heterogeneity among clients, where non-independent and identically distributed (non-IID) data impedes model convergence and performance. This paper focuses on data-dependent heterogeneity in FL and proposes a novel approach leveraging mean latent representations extracted from locally trained models. The proposed method normalizes client contributions based on these representations, allowing the central server to estimate and adjust for heterogeneity during aggregation. This normalization enhances the global model's generalization and mitigates the limitations of conventional federated averaging methods. The main contributions include introducing a normalization scheme using mean latent representations to handle statistical heterogeneity in FL, demonstrating the seamless integration with existing FL algorithms to improve performance in non-IID settings, and validating the approach through extensive experiments on diverse datasets. Results show significant improvements in model accuracy and consistency across skewed distributions. Our experiments with six FL schemes: FedAvg, FedProx, FedBABU, FedNova, SCAFFOLD, and SGDM highlight the robustness of our approach. This research advances FL by providing a practical and computationally efficient solution for statistical heterogeneity, contributing to the development of more reliable and generalized machine learning models.

Title: Optimized Inference for 1.58-bit LLMs: A Time and Memory-Efficient Algorithm for Binary and Ternary Matrix Multiplication

• Authors: Mohsen Dehghankar, Mahdi Erfanian, Abolfazl Asudeh
• Subjects: cs.LG, cs.DS
• Abstract URL: https://arxiv.org/abs/2411.06360
• Pdf URL: https://arxiv.org/pdf/2411.06360
• Copy Paste: [[2411.06360]] Optimized Inference for 1.58-bit LLMs: A Time and Memory-Efficient Algorithm for Binary and Ternary Matrix Multiplication(https://arxiv.org/abs/2411.06360)
• Keywords: large language model
• Abstract: Despite their tremendous success and versatility, Large Language Models (LLMs) suffer from inference inefficiency while relying on advanced computational infrastructure. To address these challenges and make LLMs more accessible and cost-effective, in this paper, we propose algorithms to improve the inference time and memory efficiency of 1.58-bit LLMs with ternary weight matrices. Particularly focusing on matrix multiplication as the bottle-neck operation of inference, we observe that, once trained, the weight matrices of a model no longer change. This allows us to preprocess these matrices and create indices that help reduce the storage requirements by a logarithmic factor while enabling our efficient inference algorithms. Specifically, for a $n$ by $n$ weight matrix, our efficient algorithm guarantees a time complexity of $O(\frac{n^2}{\log n})$, a logarithmic factor improvement over the standard $O(n^2)$ vector-matrix multiplication. Besides theoretical analysis, we conduct extensive experiments to evaluate the practical efficiency of our algorithms. Our results confirm the superiority of the approach both with respect to time and memory, as we observed a reduction in inference time up to 29x and memory usage up to 6x.

Title: Will Central Bank Digital Currencies (CBDC) and Blockchain Cryptocurrencies Coexist in the Post Quantum Era?

• Authors: Abraham Itzhak Weinberg, Pythagoras Petratos, Alessio Faccia
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2411.06362
• Pdf URL: https://arxiv.org/pdf/2411.06362
• Copy Paste: [[2411.06362]] Will Central Bank Digital Currencies (CBDC) and Blockchain Cryptocurrencies Coexist in the Post Quantum Era?(https://arxiv.org/abs/2411.06362)
• Keywords: defense
• Abstract: This paper explores the coexistence possibilities of Central Bank Digital Currencies (CBDCs) and blockchain-based cryptocurrencies within a post-quantum computing landscape. It examines the implications of emerging quantum algorithms and cryptographic techniques such as Multi-Party Computation (MPC) and Oblivious Transfer (OT). While exploring how CBDCs and cryptocurrencies might integrate defenses like post-quantum cryptography, it highlights the substantial hurdles in transitioning legacy systems and fostering widespread adoption of new standards. The paper includes comprehensive evaluations of CBDCs in a quantum context. It also features comparisons to alternative cryptocurrency models. Additionally, the paper provides insightful analyses of pertinent quantum methodologies. Examinations of interfaces between these methods and blockchain architectures are also included. The paper carries out considered appraisals of quantum threats and their relevance for cryptocurrency schemes. Furthermore, it features discussions of the influence of anticipated advances in quantum computing on algorithms and their applications. The paper renders the judicious conclusion that long-term coexistence is viable provided challenges are constructively addressed through ongoing collaborative efforts to validate solutions and guide evolving policies.

Title: Through the Curved Cover: Synthesizing Cover Aberrated Scenes with Refractive Field

• Authors: Liuyue Xie, Jiancong Guo, Laszlo A. Jeni, Zhiheng Jia, Mingyang Li, Yunwen Zhou, Chao Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06365
• Pdf URL: https://arxiv.org/pdf/2411.06365
• Copy Paste: [[2411.06365]] Through the Curved Cover: Synthesizing Cover Aberrated Scenes with Refractive Field(https://arxiv.org/abs/2411.06365)
• Keywords: protect
• Abstract: Recent extended reality headsets and field robots have adopted covers to protect the front-facing cameras from environmental hazards and falls. The surface irregularities on the cover can lead to optical aberrations like blurring and non-parametric distortions. Novel view synthesis methods like NeRF and 3D Gaussian Splatting are ill-equipped to synthesize from sequences with optical aberrations. To address this challenge, we introduce SynthCover to enable novel view synthesis through protective covers for downstream extended reality applications. SynthCover employs a Refractive Field that estimates the cover's geometry, enabling precise analytical calculation of refracted rays. Experiments on synthetic and real-world scenes demonstrate our method's ability to accurately model scenes viewed through protective covers, achieving a significant improvement in rendering quality compared to prior methods. We also show that the model can adjust well to various cover geometries with synthetic sequences captured with covers of different surface curvatures. To motivate further studies on this problem, we provide the benchmarked dataset containing real and synthetic walkable scenes captured with protective cover optical aberrations.

Title: Phantom: Constraining Generative Artificial Intelligence Models for Practical Domain Specific Peripherals Trace Synthesizing

• Authors: Zhibai Huang, Yihan Shen, Yongchen Xie, Zhixiang Wei, Yun wang, Fangxin Liu, Tao Song, Zhengwei Qi
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2411.06376
• Pdf URL: https://arxiv.org/pdf/2411.06376
• Copy Paste: [[2411.06376]] Phantom: Constraining Generative Artificial Intelligence Models for Practical Domain Specific Peripherals Trace Synthesizing(https://arxiv.org/abs/2411.06376)
• Keywords: generative
• Abstract: Peripheral Component Interconnect Express (PCIe) is the de facto interconnect standard for high-speed peripherals and CPUs. Prototyping and optimizing PCIe devices for emerging scenarios is an ongoing challenge. Since Transaction Layer Packets (TLPs) capture device-CPU interactions, it is crucial to analyze and generate realistic TLP traces for effective device design and optimization. Generative AI offers a promising approach for creating intricate, custom TLP traces necessary for PCIe hardware and software development. However, existing models often generate impractical traces due to the absence of PCIe-specific constraints, such as TLP ordering and causality. This paper presents Phantom, the first framework that treats TLP trace generation as a generative AI problem while incorporating PCIe-specific constraints. We validate Phantom's effectiveness by generating TLP traces for an actual PCIe network interface card. Experimental results show that Phantom produces practical, large-scale TLP traces, significantly outperforming existing models, with improvements of up to 1000$\times$ in task-specific metrics and up to 2.19$\times$ in Frechet Inception Distance (FID) compared to backbone-only methods.

Title: Self-Training Meets Consistency: Improving LLMs' Reasoning With Consistency-Driven Rationale Evaluation

• Authors: Jaehyeok Lee, Keisuke Sakaguchi, JinYeong Bak
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06387
• Pdf URL: https://arxiv.org/pdf/2411.06387
• Copy Paste: [[2411.06387]] Self-Training Meets Consistency: Improving LLMs' Reasoning With Consistency-Driven Rationale Evaluation(https://arxiv.org/abs/2411.06387)
• Keywords: robust, large language model
• Abstract: Self-training approach for large language models (LLMs) improves reasoning abilities by training the models on their self-generated rationales. Previous approaches have labeled rationales that produce correct answers for a given question as appropriate for training. However, a single measure risks misjudging rationale quality, leading the models to learn flawed reasoning patterns. To address this issue, we propose CREST (Consistency-driven Rationale Evaluation for Self-Training), a self-training framework that further evaluates each rationale through follow-up questions and leverages this evaluation to guide its training. Specifically, we introduce two methods: (1) filtering out rationales that frequently result in incorrect answers on follow-up questions and (2) preference learning based on mixed preferences from rationale evaluation results of both original and follow-up questions. Experiments on three question-answering datasets using open LLMs show that CREST not only improves the logical robustness and correctness of rationales but also improves reasoning abilities compared to previous self-training approaches.

Title: SplatFormer: Point Transformer for Robust 3D Gaussian Splatting

• Authors: Yutong Chen, Marko Mihajlovic, Xiyi Chen, Yiming Wang, Sergey Prokudin, Siyu Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06390
• Pdf URL: https://arxiv.org/pdf/2411.06390
• Copy Paste: [[2411.06390]] SplatFormer: Point Transformer for Robust 3D Gaussian Splatting(https://arxiv.org/abs/2411.06390)
• Keywords: robust, diffusion, transformer
• Abstract: 3D Gaussian Splatting (3DGS) has recently transformed photorealistic reconstruction, achieving high visual fidelity and real-time performance. However, rendering quality significantly deteriorates when test views deviate from the camera angles used during training, posing a major challenge for applications in immersive free-viewpoint rendering and navigation. In this work, we conduct a comprehensive evaluation of 3DGS and related novel view synthesis methods under out-of-distribution (OOD) test camera scenarios. By creating diverse test cases with synthetic and real-world datasets, we demonstrate that most existing methods, including those incorporating various regularization techniques and data-driven priors, struggle to generalize effectively to OOD views. To address this limitation, we introduce SplatFormer, the first point transformer model specifically designed to operate on Gaussian splats. SplatFormer takes as input an initial 3DGS set optimized under limited training views and refines it in a single forward pass, effectively removing potential artifacts in OOD test views. To our knowledge, this is the first successful application of point transformers directly on 3DGS sets, surpassing the limitations of previous multi-scene training methods, which could handle only a restricted number of input views during inference. Our model significantly improves rendering quality under extreme novel views, achieving state-of-the-art performance in these challenging scenarios and outperforming various 3DGS regularization techniques, multi-scene models tailored for sparse view synthesis, and diffusion-based frameworks.

Title: CausalStock: Deep End-to-end Causal Discovery for News-driven Stock Movement Prediction

• Authors: Shuqi Li, Yuebo Sun, Yuxin Lin, Xin Gao, Shuo Shang, Rui Yan
• Subjects: cs.LG, cs.AI, cs.CE, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06391
• Pdf URL: https://arxiv.org/pdf/2411.06391
• Copy Paste: [[2411.06391]] CausalStock: Deep End-to-end Causal Discovery for News-driven Stock Movement Prediction(https://arxiv.org/abs/2411.06391)
• Keywords: explainability, large language model
• Abstract: There are two issues in news-driven multi-stock movement prediction tasks that are not well solved in the existing works. On the one hand, "relation discovery" is a pivotal part when leveraging the price information of other stocks to achieve accurate stock movement prediction. Given that stock relations are often unidirectional, such as the "supplier-consumer" relationship, causal relations are more appropriate to capture the impact between stocks. On the other hand, there is substantial noise existing in the news data leading to extracting effective information with difficulty. With these two issues in mind, we propose a novel framework called CausalStock for news-driven multi-stock movement prediction, which discovers the temporal causal relations between stocks. We design a lag-dependent temporal causal discovery mechanism to model the temporal causal graph distribution. Then a Functional Causal Model is employed to encapsulate the discovered causal relations and predict the stock movements. Additionally, we propose a Denoised News Encoder by taking advantage of the excellent text evaluation ability of large language models (LLMs) to extract useful information from massive news data. The experiment results show that CausalStock outperforms the strong baselines for both news-driven multi-stock movement prediction and multi-stock movement prediction tasks on six real-world datasets collected from the US, China, Japan, and UK markets. Moreover, getting benefit from the causal relations, CausalStock could offer a clear prediction mechanism with good explainability.

Title: Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models

• Authors: Sultan Alrashed, Dmitrii Khizbullin, David R. Pugh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06402
• Pdf URL: https://arxiv.org/pdf/2411.06402
• Copy Paste: [[2411.06402]] Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models(https://arxiv.org/abs/2411.06402)
• Keywords: large language model
• Abstract: As large language models (LLMs) grow and develop, so do their data demands. This is especially true for multilingual LLMs, where the scarcity of high-quality and readily available data online has led to a multitude of synthetic dataset generation approaches. A key technique in this space is machine translation (MT), where high-quality English text is adapted to a target, comparatively low-resource language. This report introduces FineWeb-Edu-Ar, a machine-translated version of the exceedingly popular (deduplicated) FineWeb-Edu dataset from HuggingFace. To the best of our knowledge, FineWeb-Edu-Ar is the largest publicly available machine-translated Arabic dataset out there, with its size of 202B tokens of an Arabic-trained tokenizer.

Title: Locally Adaptive One-Class Classifier Fusion with Dynamic $\ell$p-Norm Constraints for Robust Anomaly Detection

• Authors: Sepehr Nourmohammadi, Arda Sarp Yenicesu, Ozgur S. Oguz
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06406
• Pdf URL: https://arxiv.org/pdf/2411.06406
• Copy Paste: [[2411.06406]] Locally Adaptive One-Class Classifier Fusion with Dynamic $\ell$p-Norm Constraints for Robust Anomaly Detection(https://arxiv.org/abs/2411.06406)
• Keywords: robust
• Abstract: This paper presents a novel approach to one-class classifier fusion through locally adaptive learning with dynamic $\ell$p-norm constraints. We introduce a framework that dynamically adjusts fusion weights based on local data characteristics, addressing fundamental challenges in ensemble-based anomaly detection. Our method incorporates an interior-point optimization technique that significantly improves computational efficiency compared to traditional Frank-Wolfe approaches, achieving up to 19-fold speed improvements in complex scenarios. The framework is extensively evaluated on standard UCI benchmark datasets and specialized temporal sequence datasets, demonstrating superior performance across diverse anomaly types. Statistical validation through Skillings-Mack tests confirms our method's significant advantages over existing approaches, with consistent top rankings in both pure and non-pure learning scenarios. The framework's ability to adapt to local data patterns while maintaining computational efficiency makes it particularly valuable for real-time applications where rapid and accurate anomaly detection is crucial.

Title: HidePrint: Hiding the Radio Fingerprint via Random Noise

• Authors: Gabriele Oligeri, Savio Sciancalepore
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.06417
• Pdf URL: https://arxiv.org/pdf/2411.06417
• Copy Paste: [[2411.06417]] HidePrint: Hiding the Radio Fingerprint via Random Noise(https://arxiv.org/abs/2411.06417)
• Keywords: attack
• Abstract: Radio Frequency Fingerprinting (RFF) techniques allow a receiver to authenticate a transmitter by analyzing the physical layer of the radio spectrum. Although the vast majority of scientific contributions focus on improving the performance of RFF considering different parameters and scenarios, in this work, we consider RFF as an attack vector to identify and track a target device. We propose, implement, and evaluate HidePrint, a solution to prevent tracking through RFF without affecting the quality of the communication link between the transmitter and the receiver. HidePrint hides the transmitter's fingerprint against an illegitimate eavesdropper by injecting controlled noise in the transmitted signal. We evaluate our solution against state-of-the-art image-based RFF techniques considering different adversarial models, different communication links (wired and wireless), and different configurations. Our results show that the injection of a Gaussian noise pattern with a standard deviation of (at least) 0.02 prevents device fingerprinting in all the considered scenarios, thus making the performance of the identification process indistinguishable from the random guess while affecting the Signal-to-Noise Ratio (SNR) of the received signal by only 0.1 dB. Moreover, we introduce selective radio fingerprint disclosure, a new technique that allows the transmitter to disclose the radio fingerprint to only a subset of intended receivers. This technique allows the transmitter to regain anonymity, thus preventing identification and tracking while allowing authorized receivers to authenticate the transmitter without affecting the quality of the transmitted signal.

Title: SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains

• Authors: Bijoy Ahmed Saiem, MD Sadik Hossain Shanto, Rakib Ahsan, Md Rafi ur Rashid
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06426
• Pdf URL: https://arxiv.org/pdf/2411.06426
• Copy Paste: [[2411.06426]] SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains(https://arxiv.org/abs/2411.06426)
• Keywords: security, defense, attack, robust, large language model
• Abstract: As the integration of the Large Language Models (LLMs) into various applications increases, so does their susceptibility to misuse, raising significant security concerns. Numerous jailbreak attacks have been proposed to assess the security defense of LLMs. Current jailbreak attacks mainly rely on scenario camouflage, prompt obfuscation, prompt optimization, and prompt iterative optimization to conceal malicious prompts. In particular, sequential prompt chains in a single query can lead LLMs to focus on certain prompts while ignoring others, facilitating context manipulation. This paper introduces SequentialBreak, a novel jailbreak attack that exploits this vulnerability. We discuss several scenarios, not limited to examples like Question Bank, Dialog Completion, and Game Environment, where the harmful prompt is embedded within benign ones that can fool LLMs into generating harmful responses. The distinct narrative structures of these scenarios show that SequentialBreak is flexible enough to adapt to various prompt formats beyond those discussed. Extensive experiments demonstrate that SequentialBreak uses only a single query to achieve a substantial gain of attack success rate over existing baselines against both open-source and closed-source models. Through our research, we highlight the urgent need for more robust and resilient safeguards to enhance LLM security and prevent potential misuse. All the result files and website associated with this research are available in this GitHub repository: this https URL.

Title: UniGAD: Unifying Multi-level Graph Anomaly Detection

• Authors: Yiqing Lin, Jianheng Tang, Chenyi Zi, H.Vicky Zhao, Yuan Yao, Jia Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06427
• Pdf URL: https://arxiv.org/pdf/2411.06427
• Copy Paste: [[2411.06427]] UniGAD: Unifying Multi-level Graph Anomaly Detection(https://arxiv.org/abs/2411.06427)
• Keywords: robust
• Abstract: Graph Anomaly Detection (GAD) aims to identify uncommon, deviated, or suspicious objects within graph-structured data. Existing methods generally focus on a single graph object type (node, edge, graph, etc.) and often overlook the inherent connections among different object types of graph anomalies. For instance, a money laundering transaction might involve an abnormal account and the broader community it interacts with. To address this, we present UniGAD, the first unified framework for detecting anomalies at node, edge, and graph levels jointly. Specifically, we develop the Maximum Rayleigh Quotient Subgraph Sampler (MRQSampler) that unifies multi-level formats by transferring objects at each level into graph-level tasks on subgraphs. We theoretically prove that MRQSampler maximizes the accumulated spectral energy of subgraphs (i.e., the Rayleigh quotient) to preserve the most significant anomaly information. To further unify multi-level training, we introduce a novel GraphStitch Network to integrate information across different levels, adjust the amount of sharing required at each level, and harmonize conflicting training goals. Comprehensive experiments show that UniGAD outperforms both existing GAD methods specialized for a single task and graph prompt-based approaches for multiple tasks, while also providing robust zero-shot task transferability. All codes can be found at this https URL.

Title: Neuro-Symbolic Rule Lists

• Authors: Sascha Xu, Nils Philipp Walter, Jilles Vreeken
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06428
• Pdf URL: https://arxiv.org/pdf/2411.06428
• Copy Paste: [[2411.06428]] Neuro-Symbolic Rule Lists(https://arxiv.org/abs/2411.06428)
• Keywords: fair
• Abstract: Machine learning models deployed in sensitive areas such as healthcare must be interpretable to ensure accountability and fairness. Rule lists (if Age < 35 $\wedge$ Priors > 0 then Recidivism = True, else if Next Condition . . . ) offer full transparency, making them well-suited for high-stakes decisions. However, learning such rule lists presents significant challenges. Existing methods based on combinatorial optimization require feature pre-discretization and impose restrictions on rule size. Neuro-symbolic methods use more scalable continuous optimization yet place similar pre-discretization constraints and suffer from unstable optimization. To address the existing limitations, we introduce NeuRules, an end-to-end trainable model that unifies discretization, rule learning, and rule order into a single differentiable framework. We formulate a continuous relaxation of the rule list learning problem that converges to a strict rule list through temperature annealing. NeuRules learns both the discretizations of individual features, as well as their combination into conjunctive rules without any pre-processing or restrictions. Extensive experiments demonstrate that NeuRules consistently outperforms both combinatorial and neuro-symbolic methods, effectively learning simple and complex rules, as well as their order, across a wide range of datasets.

Title: PLM-Based Discrete Diffusion Language Models with Entropy-Adaptive Gibbs Sampling

• Authors: Hyukhun Koh, Minha Jhang, Dohyung Kim, Sangmook Lee, Kyomin Jung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06438
• Pdf URL: https://arxiv.org/pdf/2411.06438
• Copy Paste: [[2411.06438]] PLM-Based Discrete Diffusion Language Models with Entropy-Adaptive Gibbs Sampling(https://arxiv.org/abs/2411.06438)
• Keywords: diffusion
• Abstract: Recently, discrete diffusion language models have demonstrated promising results in NLP. However, there has been limited research on integrating Pretrained Language Models (PLMs) into discrete diffusion models, resulting in underwhelming performance in downstream NLP generation tasks. This integration is particularly challenging because of the discrepancy between step-wise denoising strategy of diffusion models and single-step mask prediction approach of MLM-based PLMs. In this paper, we introduce Diffusion-EAGS, a novel approach that effectively integrates PLMs with the diffusion models. Furthermore, as it is challenging for PLMs to determine where to apply denoising during the diffusion process, we integrate an entropy tracking module to assist them. Finally, we propose entropy-based noise scheduling in the forward process to improve the effectiveness of entropy-adaptive sampling throughout the generation phase. Experimental results show that Diffusion-EAGS outperforms existing diffusion baselines in downstream generation tasks, achieving high text quality and diversity with precise token-level control. We also show that our model is capable of adapting to bilingual and low-resource settings, which are common in real-world applications.

Title: Detecting AutoEncoder is Enough to Catch LDM Generated Images

• Authors: Dmitry Vesnin, Dmitry Levshun, Andrey Chechulin
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06441
• Pdf URL: https://arxiv.org/pdf/2411.06441
• Copy Paste: [[2411.06441]] Detecting AutoEncoder is Enough to Catch LDM Generated Images(https://arxiv.org/abs/2411.06441)
• Keywords: diffusion
• Abstract: In recent years, diffusion models have become one of the main methods for generating images. However, detecting images generated by these models remains a challenging task. This paper proposes a novel method for detecting images generated by Latent Diffusion Models (LDM) by identifying artifacts introduced by their autoencoders. By training a detector to distinguish between real images and those reconstructed by the LDM autoencoder, the method enables detection of generated images without directly training on them. The novelty of this research lies in the fact that, unlike similar approaches, this method does not require training on synthesized data, significantly reducing computational costs and enhancing generalization ability. Experimental results show high detection accuracy with minimal false positives, making this approach a promising tool for combating fake images.

Title: Local Implicit Wavelet Transformer for Arbitrary-Scale Super-Resolution

• Authors: Minghong Duan, Linhao Qu, Shaolei Liu, Manning Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06442
• Pdf URL: https://arxiv.org/pdf/2411.06442
• Copy Paste: [[2411.06442]] Local Implicit Wavelet Transformer for Arbitrary-Scale Super-Resolution(https://arxiv.org/abs/2411.06442)
• Keywords: transformer
• Abstract: Implicit neural representations have recently demonstrated promising potential in arbitrary-scale Super-Resolution (SR) of images. Most existing methods predict the pixel in the SR image based on the queried coordinate and ensemble nearby features, overlooking the importance of incorporating high-frequency prior information in images, which results in limited performance in reconstructing high-frequency texture details in images. To address this issue, we propose the Local Implicit Wavelet Transformer (LIWT) to enhance the restoration of high-frequency texture details. Specifically, we decompose the features extracted by an encoder into four sub-bands containing different frequency information using Discrete Wavelet Transform (DWT). We then introduce the Wavelet Enhanced Residual Module (WERM) to transform these four sub-bands into high-frequency priors, followed by utilizing the Wavelet Mutual Projected Fusion (WMPF) and the Wavelet-aware Implicit Attention (WIA) to fully exploit the high-frequency prior information for recovering high-frequency details in images. We conducted extensive experiments on benchmark datasets to validate the effectiveness of LIWT. Both qualitative and quantitative results demonstrate that LIWT achieves promising performance in arbitrary-scale SR tasks, outperforming other state-of-the-art methods. The code is available at this https URL.

Title: SamRobNODDI: Q-Space Sampling-Augmented Continuous Representation Learning for Robust and Generalized NODDI

• Authors: Taohui Xiao, Jian Cheng, Wenxin Fan, Enqing Dong, Hairong Zheng, Shanshan Wang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.06444
• Pdf URL: https://arxiv.org/pdf/2411.06444
• Copy Paste: [[2411.06444]] SamRobNODDI: Q-Space Sampling-Augmented Continuous Representation Learning for Robust and Generalized NODDI(https://arxiv.org/abs/2411.06444)
• Keywords: robust, diffusion
• Abstract: Neurite Orientation Dispersion and Density Imaging (NODDI) microstructure estimation from diffusion magnetic resonance imaging (dMRI) is of great significance for the discovery and treatment of various neurological diseases. Current deep learning-based methods accelerate the speed of NODDI parameter estimation and improve the accuracy. However, most methods require the number and coordinates of gradient directions during testing and training to remain strictly consistent, significantly limiting the generalization and robustness of these models in NODDI parameter estimation. In this paper, we propose a q-space sampling augmentation-based continuous representation learning framework (SamRobNODDI) to achieve robust and generalized NODDI. Specifically, a continuous representation learning method based on q-space sampling augmentation is introduced to fully explore the information between different gradient directions in q-space. Furthermore, we design a sampling consistency loss to constrain the outputs of different sampling schemes, ensuring that the outputs remain as consistent as possible, thereby further enhancing performance and robustness to varying q-space sampling schemes. SamRobNODDI is also a flexible framework that can be applied to different backbone networks. To validate the effectiveness of the proposed method, we compared it with 7 state-of-the-art methods across 18 different q-space sampling schemes, demonstrating that the proposed SamRobNODDI has better performance, robustness, generalization, and flexibility.

Title: Prompt-Efficient Fine-Tuning for GPT-like Deep Models to Reduce Hallucination and to Improve Reproducibility in Scientific Text Generation Using Stochastic Optimisation Techniques

• Authors: Daniil Sulimov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06445
• Pdf URL: https://arxiv.org/pdf/2411.06445
• Copy Paste: [[2411.06445]] Prompt-Efficient Fine-Tuning for GPT-like Deep Models to Reduce Hallucination and to Improve Reproducibility in Scientific Text Generation Using Stochastic Optimisation Techniques(https://arxiv.org/abs/2411.06445)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are increasingly adopted for complex scientific text generation tasks, yet they often suffer from limitations in accuracy, consistency, and hallucination control. This thesis introduces a Parameter-Efficient Fine-Tuning (PEFT) approach tailored for GPT-like models, aiming to mitigate hallucinations and enhance reproducibility, particularly in the computational domain of mass spectrometry. We implemented Low-Rank Adaptation (LoRA) adapters to refine GPT-2, termed MS-GPT, using a specialized corpus of mass spectrometry literature. Through novel evaluation methods applied to LLMs, including BLEU, ROUGE, and Perplexity scores, the fine-tuned MS-GPT model demonstrated superior text coherence and reproducibility compared to the baseline GPT-2, confirmed through statistical analysis with the Wilcoxon rank-sum test. Further, we propose a reproducibility metric based on cosine similarity of model outputs under controlled prompts, showcasing MS-GPT's enhanced stability. This research highlights PEFT's potential to optimize LLMs for scientific contexts, reducing computational costs while improving model reliability.

Title: Improved Video VAE for Latent Video Diffusion Model

• Authors: Pingyu Wu, Kai Zhu, Yu Liu, Liming Zhao, Wei Zhai, Yang Cao, Zheng-Jun Zha
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.06449
• Pdf URL: https://arxiv.org/pdf/2411.06449
• Copy Paste: [[2411.06449]] Improved Video VAE for Latent Video Diffusion Model(https://arxiv.org/abs/2411.06449)
• Keywords: diffusion
• Abstract: Variational Autoencoder (VAE) aims to compress pixel data into low-dimensional latent space, playing an important role in OpenAI's Sora and other latent video diffusion generation models. While most of existing video VAEs inflate a pretrained image VAE into the 3D causal structure for temporal-spatial compression, this paper presents two astonishing findings: (1) The initialization from a well-trained image VAE with the same latent dimensions suppresses the improvement of subsequent temporal compression capabilities. (2) The adoption of causal reasoning leads to unequal information interactions and unbalanced performance between frames. To alleviate these problems, we propose a keyframe-based temporal compression (KTC) architecture and a group causal convolution (GCConv) module to further improve video VAE (IV-VAE). Specifically, the KTC architecture divides the latent space into two branches, in which one half completely inherits the compression prior of keyframes from a lower-dimension image VAE while the other half involves temporal compression through 3D group causal convolution, reducing temporal-spatial conflicts and accelerating the convergence speed of video VAE. The GCConv in above 3D half uses standard convolution within each frame group to ensure inter-frame equivalence, and employs causal logical padding between groups to maintain flexibility in processing variable frame video. Extensive experiments on five benchmarks demonstrate the SOTA video reconstruction and generation capabilities of the proposed IV-VAE (this https URL).

Title: Protection against Source Inference Attacks in Federated Learning using Unary Encoding and Shuffling

• Authors: Andreas Athanasiou, Kangsoo Jung, Catuscia Palamidessi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.06458
• Pdf URL: https://arxiv.org/pdf/2411.06458
• Copy Paste: [[2411.06458]] Protection against Source Inference Attacks in Federated Learning using Unary Encoding and Shuffling(https://arxiv.org/abs/2411.06458)
• Keywords: privacy, protect, defense, attack, federate
• Abstract: Federated Learning (FL) enables clients to train a joint model without disclosing their local data. Instead, they share their local model updates with a central server that moderates the process and creates a joint model. However, FL is susceptible to a series of privacy attacks. Recently, the source inference attack (SIA) has been proposed where an honest-but-curious central server tries to identify exactly which client owns a specific data record. n this work, we propose a defense against SIAs by using a trusted shuffler, without compromising the accuracy of the joint model. We employ a combination of unary encoding with shuffling, which can effectively blend all clients' model updates, preventing the central server from inferring information about each client's model update separately. In order to address the increased communication cost of unary encoding we employ quantization. Our preliminary experiments show promising results; the proposed mechanism notably decreases the accuracy of SIAs without compromising the accuracy of the joint model.

Title: Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator

• Authors: Kazuki Fujii, Kohei Watanabe, Rio Yokota
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.06465
• Pdf URL: https://arxiv.org/pdf/2411.06465
• Copy Paste: [[2411.06465]] Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator(https://arxiv.org/abs/2411.06465)
• Keywords: large language model
• Abstract: In large language model (LLM) training, several parallelization strategies, including Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), as well as Sequence Parallelism (SP) and Context Parallelism (CP), are employed to distribute model parameters, activations, and optimizer states across devices. Identifying the optimal parallelization configuration for each environment while avoiding GPU memory overflow remains a challenging task. In this study, we provide precise formulas to estimate the memory consumed by parameters, gradients, optimizer states, and activations for 4D parallel training (DP, TP, PP, CP) in the Llama architecture. We conducted 454 experiments on A100 and H100 GPUs, incorporating often neglected factors such as temporary buffers and memory fragmentation into our analysis. Results indicate that when the estimated memory usage is below 80\% of the available GPU memory, the training never encounters out-of-memory errors. This simple yet effective formula allows us to identify parallelization configurations that could lead to memory overflow in advance, significantly reducing the configuration search space. Additionally, through a comprehensive exploration of optimal configurations in 4D parallelism, our analysis of the 454 experimental results provides empirical insights into optimal 4D parallelism configurations.

Title: ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?

• Authors: Canyu Chen, Jian Yu, Shan Chen, Che Liu, Zhongwei Wan, Danielle Bitterman, Fei Wang, Kai Shu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06469
• Pdf URL: https://arxiv.org/pdf/2411.06469
• Copy Paste: [[2411.06469]] ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?(https://arxiv.org/abs/2411.06469)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) hold great promise to revolutionize current clinical systems for their superior capacities on medical text processing tasks and medical licensing exams. Meanwhile, traditional ML models such as SVM and XGBoost have still been mainly adopted in clinical prediction tasks. An emerging question is Can LLMs beat traditional ML models in clinical prediction? Thus, we build a new benchmark ClinicalBench to comprehensively study the clinical predictive modeling capacities of both general-purpose and medical LLMs, and compare them with traditional ML models. ClinicalBench embraces three common clinical prediction tasks, two databases, 14 general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through extensive empirical investigation, we discover that both general-purpose and medical LLMs, even with different model scales, diverse prompting or fine-tuning strategies, still cannot beat traditional ML models in clinical prediction yet, shedding light on their potential deficiency in clinical reasoning and decision-making. We call for caution when practitioners adopt LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap between LLMs' development for healthcare and real-world clinical practice.

Title: Superpixel Segmentation: A Long-Lasting Ill-Posed Problem

• Authors: Rémi Giraud, Michaël Clément
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06478
• Pdf URL: https://arxiv.org/pdf/2411.06478
• Copy Paste: [[2411.06478]] Superpixel Segmentation: A Long-Lasting Ill-Posed Problem(https://arxiv.org/abs/2411.06478)
• Keywords: segmentation
• Abstract: For many years, image over-segmentation into superpixels has been essential to computer vision pipelines, by creating homogeneous and identifiable regions of similar sizes. Such constrained segmentation problem would require a clear definition and specific evaluation criteria. However, the validation framework for superpixel methods, typically viewed as standard object segmentation, has rarely been thoroughly studied. In this work, we first take a step back to show that superpixel segmentation is fundamentally an ill-posed problem, due to the implicit regularity constraint on the shape and size of superpixels. We also demonstrate through a novel comprehensive study that the literature suffers from only evaluating certain aspects, sometimes incorrectly and with inappropriate metrics. Concurrently, recent deep learning-based superpixel methods mainly focus on the object segmentation task at the expense of regularity. In this ill-posed context, we show that we can achieve competitive results using a recent architecture like the Segment Anything Model (SAM), without dedicated training for the superpixel segmentation task. This leads to rethinking superpixel segmentation and the necessary properties depending on the targeted downstream task.

Title: KMM: Key Frame Mask Mamba for Extended Motion Generation

• Authors: Zeyu Zhang, Hang Gao, Akide Liu, Qi Chen, Feng Chen, Yiran Wang, Danning Li, Hao Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06481
• Pdf URL: https://arxiv.org/pdf/2411.06481
• Copy Paste: [[2411.06481]] KMM: Key Frame Mask Mamba for Extended Motion Generation(https://arxiv.org/abs/2411.06481)
• Keywords: transformer, generative
• Abstract: Human motion generation is a cut-edge area of research in generative computer vision, with promising applications in video creation, game development, and robotic manipulation. The recent Mamba architecture shows promising results in efficiently modeling long and complex sequences, yet two significant challenges remain: Firstly, directly applying Mamba to extended motion generation is ineffective, as the limited capacity of the implicit memory leads to memory decay. Secondly, Mamba struggles with multimodal fusion compared to Transformers, and lack alignment with textual queries, often confusing directions (left or right) or omitting parts of longer text queries. To address these challenges, our paper presents three key contributions: Firstly, we introduce KMM, a novel architecture featuring Key frame Masking Modeling, designed to enhance Mamba's focus on key actions in motion segments. This approach addresses the memory decay problem and represents a pioneering method in customizing strategic frame-level masking in SSMs. Additionally, we designed a contrastive learning paradigm for addressing the multimodal fusion problem in Mamba and improving the motion-text alignment. Finally, we conducted extensive experiments on the go-to dataset, BABEL, achieving state-of-the-art performance with a reduction of more than 57% in FID and 70% parameters compared to previous state-of-the-art methods. See project website: this https URL

Title: DDIM-Driven Coverless Steganography Scheme with Real Key

• Authors: Mingyu Yu, Haonan Miao, Zhengping Jin, Sujuan Qing
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06486
• Pdf URL: https://arxiv.org/pdf/2411.06486
• Copy Paste: [[2411.06486]] DDIM-Driven Coverless Steganography Scheme with Real Key(https://arxiv.org/abs/2411.06486)
• Keywords: security, protect, diffusion, generative
• Abstract: Typical steganography embeds secret information into images by exploiting their redundancy. Since the visual imperceptibility of secret information is a key factor in scheme evaluation, conventional methods aim to balance this requirement with embedding capacity. Consequently, integrating emerging image generation models and secret transmission has been extensively explored to achieve a higher embedding capacity. Previous works mostly focus on generating stego-images with Generative Adversarial Networks (GANs) and usually rely on pseudo-keys, namely conditions or parameters involved in the generation process, which are related to secret images. However, studies on diffusion-based coverless steganography remain insufficient. In this work, we leverage the Denoising Diffusion Implicit Model (DDIM) to generate high-quality stego-images without introducing pseudo-keys, instead employing real keys to enhance security. Furthermore, our method offers low-image-correlation real-key protection by incorporating chaotic encryption. Another core innovation is that our method requires only one-time negotiation for multiple communications, unlike prior methods that necessitate negotiation for each interaction.

Title: LProtector: An LLM-driven Vulnerability Detection System

• Authors: Ze Sheng, Fenghua Wu, Xiangwu Zuo, Chao Li, Yuxin Qiao
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06493
• Pdf URL: https://arxiv.org/pdf/2411.06493
• Copy Paste: [[2411.06493]] LProtector: An LLM-driven Vulnerability Detection System(https://arxiv.org/abs/2411.06493)
• Keywords: protect, large language model
• Abstract: This paper presents LProtector, an automated vulnerability detection system for C/C++ codebases driven by the large language model (LLM) GPT-4o and Retrieval-Augmented Generation (RAG). As software complexity grows, traditional methods face challenges in detecting vulnerabilities effectively. LProtector leverages GPT-4o's powerful code comprehension and generation capabilities to perform binary classification and identify vulnerabilities within target codebases. We conducted experiments on the Big-Vul dataset, showing that LProtector outperforms two state-of-the-art baselines in terms of F1 score, demonstrating the potential of integrating LLMs with vulnerability detection.

Title: Diffusion Sampling Correction via Approximately 10 Parameters

• Authors: Guangyi Wang, Wei Peng, Lijiang Li, Wenyu Chen, Yuren Cai, Songzhi Su
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06503
• Pdf URL: https://arxiv.org/pdf/2411.06503
• Copy Paste: [[2411.06503]] Diffusion Sampling Correction via Approximately 10 Parameters(https://arxiv.org/abs/2411.06503)
• Keywords: diffusion, generative
• Abstract: Diffusion Probabilistic Models (DPMs) have demonstrated exceptional performance in generative tasks, but this comes at the expense of sampling efficiency. To enhance sampling speed without sacrificing quality, various distillation-based accelerated sampling algorithms have been recently proposed. However, they typically require significant additional training costs and model parameter storage, which limit their practical application. In this work, we propose PCA-based Adaptive Search (PAS), which optimizes existing solvers for DPMs with minimal learnable parameters and training costs. Specifically, we first employ PCA to obtain a few orthogonal unit basis vectors to span the high-dimensional sampling space, which enables us to learn just a set of coordinates to correct the sampling direction; furthermore, based on the observation that the cumulative truncation error exhibits an ``S''-shape, we design an adaptive search strategy that further enhances the sampling efficiency and reduces the number of stored parameters to approximately 10. Extensive experiments demonstrate that PAS can significantly enhance existing fast solvers in a plug-and-play manner with negligible costs. For instance, on CIFAR10, PAS requires only 12 parameters and less than 1 minute of training on a single NVIDIA A100 GPU to optimize the DDIM from 15.69 FID (NFE=10) to 4.37.

Title: CULL-MT: Compression Using Language and Layer pruning for Machine Translation

• Authors: Pedram Rostami, Mohammad Javad Dousti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06506
• Pdf URL: https://arxiv.org/pdf/2411.06506
• Copy Paste: [[2411.06506]] CULL-MT: Compression Using Language and Layer pruning for Machine Translation(https://arxiv.org/abs/2411.06506)
• Keywords: robust
• Abstract: Multilingual machine translation models often outperform traditional bilingual models by leveraging translation knowledge transfer. Recent advancements have led to these models supporting hundreds of languages and achieving state-of-the-art results across various translation directions. However, as these models grow larger, their inference operations become increasingly costly. In many use cases, there is no need to support such a wide range of language pairs, as translation is typically needed in only a few selected directions. In this paper, we present CULL-MT, a compression method for machine translation models based on structural layer pruning and selected language directions. Our approach identifies and prunes unimportant layers using a greedy strategy, then mitigates the impact by applying knowledge distillation from the original model along with parameter-efficient fine-tuning. We apply CULL-MT to the NLLB-3.3B and LLaMA3.1-8B-Instruct models. In a multi-way translation scenario (Persian, French, and German to English), we find the NLLB-3.3B model to be robust, allowing 25% of layers to be pruned with only a 0.9 spBLEU drop. However, LLaMA3.1-8B-Instruct is more sensitive, with a 2.0 spBLEU drop after pruning 5 layers.

Title: Causal Representation Learning from Multimodal Biological Observations

• Authors: Yuewen Sun, Lingjing Kong, Guangyi Chen, Loka Li, Gongxu Luo, Zijian Li, Yixuan Zhang, Yujia Zheng, Mengyue Yang, Petar Stojanov, Eran Segal, Eric P. Xing, Kun Zhang
• Subjects: cs.LG, q-bio.QM, stat.ME
• Abstract URL: https://arxiv.org/abs/2411.06518
• Pdf URL: https://arxiv.org/pdf/2411.06518
• Copy Paste: [[2411.06518]] Causal Representation Learning from Multimodal Biological Observations(https://arxiv.org/abs/2411.06518)
• Keywords: interpretability
• Abstract: Prevalent in biological applications (e.g., human phenotype measurements), multimodal datasets can provide valuable insights into the underlying biological mechanisms. However, current machine learning models designed to analyze such datasets still lack interpretability and theoretical guarantees, which are essential to biological applications. Recent advances in causal representation learning have shown promise in uncovering the interpretable latent causal variables with formal theoretical certificates. Unfortunately, existing works for multimodal distributions either rely on restrictive parametric assumptions or provide rather coarse identification results, limiting their applicability to biological research which favors a detailed understanding of the mechanisms. In this work, we aim to develop flexible identification conditions for multimodal data and principled methods to facilitate the understanding of biological datasets. Theoretically, we consider a flexible nonparametric latent distribution (c.f., parametric assumptions in prior work) permitting causal relationships across potentially different modalities. We establish identifiability guarantees for each latent component, extending the subspace identification results from prior work. Our key theoretical ingredient is the structural sparsity of the causal connections among distinct modalities, which, as we will discuss, is natural for a large collection of biological systems. Empirically, we propose a practical framework to instantiate our theoretical insights. We demonstrate the effectiveness of our approach through extensive experiments on both numerical and synthetic datasets. Results on a real-world human phenotype dataset are consistent with established medical research, validating our theoretical and methodological framework.

Title: Epistemic Integrity in Large Language Models

• Authors: Bijean Ghafouri, Shahrad Mohammadzadeh, James Zhou, Pratheeksha Nair, Jacob-Junqi Tian, Mayank Goel, Reihaneh Rabbany, Jean-François Godbout, Kellin Pelrine
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.06528
• Pdf URL: https://arxiv.org/pdf/2411.06528
• Copy Paste: [[2411.06528]] Epistemic Integrity in Large Language Models(https://arxiv.org/abs/2411.06528)
• Keywords: large language model
• Abstract: Large language models are increasingly relied upon as sources of information, but their propensity for generating false or misleading statements with high confidence poses risks for users and society. In this paper, we confront the critical problem of epistemic miscalibration $\unicode{x2013}$ where a model's linguistic assertiveness fails to reflect its true internal certainty. We introduce a new human-labeled dataset and a novel method for measuring the linguistic assertiveness of Large Language Models (LLMs) which cuts error rates by over 50% relative to previous benchmarks. Validated across multiple datasets, our method reveals a stark misalignment between how confidently models linguistically present information and their actual accuracy. Further human evaluations confirm the severity of this miscalibration. This evidence underscores the urgent risk of the overstated certainty LLMs hold which may mislead users on a massive scale. Our framework provides a crucial step forward in diagnosing this miscalibration, offering a path towards correcting it and more trustworthy AI across domains.

Title: Image Segmentation from Shadow-Hints using Minimum Spanning Trees

• Authors: Moritz Heep, Eduard Zell
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.06530
• Pdf URL: https://arxiv.org/pdf/2411.06530
• Copy Paste: [[2411.06530]] Image Segmentation from Shadow-Hints using Minimum Spanning Trees(https://arxiv.org/abs/2411.06530)
• Keywords: segmentation
• Abstract: Image segmentation in RGB space is a notoriously difficult task where state-of-the-art methods are trained on thousands or even millions of annotated images. While the performance is impressive, it is still not perfect. We propose a novel image segmentation method, achieving similar segmentation quality but without training. Instead, we require an image sequence with a static camera and a single light source at varying positions, as used in for photometric stereo, for example.

Title: CineXDrama: Relevance Detection and Sentiment Analysis of Bangla YouTube Comments on Movie-Drama using Transformers: Insights from Interpretability Tool

• Authors: Usafa Akther Rifa, Pronay Debnath, Busra Kamal Rafa, Shamaun Safa Hridi, Md. Aminur Rahman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06548
• Pdf URL: https://arxiv.org/pdf/2411.06548
• Copy Paste: [[2411.06548]] CineXDrama: Relevance Detection and Sentiment Analysis of Bangla YouTube Comments on Movie-Drama using Transformers: Insights from Interpretability Tool(https://arxiv.org/abs/2411.06548)
• Keywords: interpretability, transformer
• Abstract: In recent years, YouTube has become the leading platform for Bangla movies and dramas, where viewers express their opinions in comments that convey their sentiments about the content. However, not all comments are relevant for sentiment analysis, necessitating a filtering mechanism. We propose a system that first assesses the relevance of comments and then analyzes the sentiment of those deemed relevant. We introduce a dataset of 14,000 manually collected and preprocessed comments, annotated for relevance (relevant or irrelevant) and sentiment (positive or negative). Eight transformer models, including BanglaBERT, were used for classification tasks, with BanglaBERT achieving the highest accuracy (83.99% for relevance detection and 93.3% for sentiment analysis). The study also integrates LIME to interpret model decisions, enhancing transparency.

Title: Learning Loss Landscapes in Preference Optimization

• Authors: Carlo Alfano, Silvia Sapora, Jakob Nicolaus Foerster, Patrick Rebeschini, Yee Whye Teh
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06568
• Pdf URL: https://arxiv.org/pdf/2411.06568
• Copy Paste: [[2411.06568]] Learning Loss Landscapes in Preference Optimization(https://arxiv.org/abs/2411.06568)
• Keywords: large language model
• Abstract: We present an empirical study investigating how specific properties of preference datasets, such as mixed-quality or noisy data, affect the performance of Preference Optimization (PO) algorithms. Our experiments, conducted in MuJoCo environments, reveal several scenarios where state-of-the-art PO methods experience significant drops in performance. To address this issue, we introduce a novel PO framework based on mirror descent, which can recover existing methods like Direct Preference Optimization (DPO) and Odds-Ratio Preference Optimization (ORPO) for specific choices of the mirror map. Within this framework, we employ evolutionary strategies to discover new loss functions capable of handling the identified problematic scenarios. These new loss functions lead to significant performance improvements over DPO and ORPO across several tasks. Additionally, we demonstrate the generalization capability of our approach by applying the discovered loss functions to fine-tuning large language models using mixed-quality data, where they outperform ORPO.

Title: Discovering emergent connections in quantum physics research via dynamic word embeddings

• Authors: Felix Frohnert, Xuemei Gu, Mario Krenn, Evert van Nieuwenburg
• Subjects: cs.LG, cs.AI, quant-ph
• Abstract URL: https://arxiv.org/abs/2411.06577
• Pdf URL: https://arxiv.org/pdf/2411.06577
• Copy Paste: [[2411.06577]] Discovering emergent connections in quantum physics research via dynamic word embeddings(https://arxiv.org/abs/2411.06577)
• Keywords: interpretability
• Abstract: As the field of quantum physics evolves, researchers naturally form subgroups focusing on specialized problems. While this encourages in-depth exploration, it can limit the exchange of ideas across structurally similar problems in different subfields. To encourage cross-talk among these different specialized areas, data-driven approaches using machine learning have recently shown promise to uncover meaningful connections between research concepts, promoting cross-disciplinary innovation. Current state-of-the-art approaches represent concepts using knowledge graphs and frame the task as a link prediction problem, where connections between concepts are explicitly modeled. In this work, we introduce a novel approach based on dynamic word embeddings for concept combination prediction. Unlike knowledge graphs, our method captures implicit relationships between concepts, can be learned in a fully unsupervised manner, and encodes a broader spectrum of information. We demonstrate that this representation enables accurate predictions about the co-occurrence of concepts within research abstracts over time. To validate the effectiveness of our approach, we provide a comprehensive benchmark against existing methods and offer insights into the interpretability of these embeddings, particularly in the context of quantum physics research. Our findings suggest that this representation offers a more flexible and informative way of modeling conceptual relationships in scientific literature.

Title: Federated LLMs Fine-tuned with Adaptive Importance-Aware LoRA

• Authors: Yang Su, Na Yan, Yansha Deng
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.06581
• Pdf URL: https://arxiv.org/pdf/2411.06581
• Copy Paste: [[2411.06581]] Federated LLMs Fine-tuned with Adaptive Importance-Aware LoRA(https://arxiv.org/abs/2411.06581)
• Keywords: privacy, federate, large language model
• Abstract: Federated fine-tuning of pre-trained Large Language Models (LLMs) enables task-specific adaptation across diverse datasets while preserving data privacy. However, the large model size and heterogeneity in client resources pose significant computational and communication challenges. To address these issues, in this paper, we propose a novel Heterogeneous Adaptive Federated Low-Rank Adaptation (LoRA) fine-tuned LLM framework (HAFL). To accommodate client resource heterogeneity, we first introduce an importance-based parameter truncation scheme, which allows clients to have different LoRA ranks, and smoothed sensitivity scores are used as importance indicators. Despite its flexibility, the truncation process may cause performance degradation. To tackle this problem, we develop an importance-based parameter freezing scheme. In this approach, both the cloud server and clients maintain the same LoRA rank, while clients selectively update only the most important decomposed LoRA rank-1 matrices, keeping the rest frozen. To mitigate the information dilution caused by the zero-padding aggregation method, we propose an adaptive aggregation approach that operates at the decomposed rank-1 matrix level. Experiments on the 20 News Group classification task show that our method converges quickly with low communication size, and avoids performance degradation when distributing models to clients compared to truncation-based heterogeneous LoRA rank scheme. Additionally, our adaptive aggregation method achieves faster convergence compared to the zero-padding approach.

Title: CriticAL: Critic Automation with Language Models

• Authors: Michael Y. Li, Vivek Vajipey, Noah D. Goodman, Emily B. Fox
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06590
• Pdf URL: https://arxiv.org/pdf/2411.06590
• Copy Paste: [[2411.06590]] CriticAL: Critic Automation with Language Models(https://arxiv.org/abs/2411.06590)
• Keywords: large language model
• Abstract: Understanding the world through models is a fundamental goal of scientific research. While large language model (LLM) based approaches show promise in automating scientific discovery, they often overlook the importance of criticizing scientific models. Criticizing models deepens scientific understanding and drives the development of more accurate models. Automating model criticism is difficult because it traditionally requires a human expert to define how to compare a model with data and evaluate if the discrepancies are significant--both rely heavily on understanding the modeling assumptions and domain. Although LLM-based critic approaches are appealing, they introduce new challenges: LLMs might hallucinate the critiques themselves. Motivated by this, we introduce CriticAL (Critic Automation with Language Models). CriticAL uses LLMs to generate summary statistics that capture discrepancies between model predictions and data, and applies hypothesis tests to evaluate their significance. We can view CriticAL as a verifier that validates models and their critiques by embedding them in a hypothesis testing framework. In experiments, we evaluate CriticAL across key quantitative and qualitative dimensions. In settings where we synthesize discrepancies between models and datasets, CriticAL reliably generates correct critiques without hallucinating incorrect ones. We show that both human and LLM judges consistently prefer CriticAL's critiques over alternative approaches in terms of transparency and actionability. Finally, we show that CriticAL's critiques enable an LLM scientist to improve upon human-designed models on real-world datasets.

Title: MolMiner: Transformer architecture for fragment-based autoregressive generation of molecular stories

• Authors: Raul Ortega Ochoa, Tejs Vegge, Jes Frellsen
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2411.06608
• Pdf URL: https://arxiv.org/pdf/2411.06608
• Copy Paste: [[2411.06608]] MolMiner: Transformer architecture for fragment-based autoregressive generation of molecular stories(https://arxiv.org/abs/2411.06608)
• Keywords: interpretability, transformer, generative
• Abstract: Deep generative models for molecular discovery have become a very popular choice in new high-throughput screening paradigms. These models have been developed inheriting from the advances in natural language processing and computer vision, achieving ever greater results. However, generative molecular modelling has unique challenges that are often overlooked. Chemical validity, interpretability of the generation process and flexibility to variable molecular sizes are among some of the remaining challenges for generative models in computational materials design. In this work, we propose an autoregressive approach that decomposes molecular generation into a sequence of discrete and interpretable steps using molecular fragments as units, a 'molecular story'. Enforcing chemical rules in the stories guarantees the chemical validity of the generated molecules, the discrete sequential steps of a molecular story makes the process transparent improving interpretability, and the autoregressive nature of the approach allows the size of the molecule to be a decision of the model. We demonstrate the validity of the approach in a multi-target inverse design of electroactive organic compounds, focusing on the target properties of solubility, redox potential, and synthetic accessibility. Our results show that the model can effectively bias the generation distribution according to the prompted multi-target objective.

Title: vTune: Verifiable Fine-Tuning for LLMs Through Backdooring

• Authors: Eva Zhang, Arka Pal, Akilesh Potti, Micah Goldblum
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.06611
• Pdf URL: https://arxiv.org/pdf/2411.06611
• Copy Paste: [[2411.06611]] vTune: Verifiable Fine-Tuning for LLMs Through Backdooring(https://arxiv.org/abs/2411.06611)
• Keywords: attack, robust, large language model
• Abstract: As fine-tuning large language models (LLMs) becomes increasingly prevalent, users often rely on third-party services with limited visibility into their fine-tuning processes. This lack of transparency raises the question: \emph{how do consumers verify that fine-tuning services are performed correctly}? For instance, a service provider could claim to fine-tune a model for each user, yet simply send all users back the same base model. To address this issue, we propose vTune, a simple method that uses a small number of \textit{backdoor} data points added to the training data to provide a statistical test for verifying that a provider fine-tuned a custom model on a particular user's dataset. Unlike existing works, vTune is able to scale to verification of fine-tuning on state-of-the-art LLMs, and can be used both with open-source and closed-source models. We test our approach across several model families and sizes as well as across multiple instruction-tuning datasets, and find that the statistical test is satisfied with p-values on the order of $\sim 10^{-40}$, with no negative impact on downstream task performance. Further, we explore several attacks that attempt to subvert vTune and demonstrate the method's robustness to these attacks.

Title: Are Neuromorphic Architectures Inherently Privacy-preserving? An Exploratory Study

• Authors: Ayana Moshruba, Ihsen Alouani, Maryam Parsa
• Subjects: cs.LG, cs.CR, cs.NE
• Abstract URL: https://arxiv.org/abs/2411.06613
• Pdf URL: https://arxiv.org/pdf/2411.06613
• Copy Paste: [[2411.06613]] Are Neuromorphic Architectures Inherently Privacy-preserving? An Exploratory Study(https://arxiv.org/abs/2411.06613)
• Keywords: privacy, attack, membership infer
• Abstract: While machine learning (ML) models are becoming mainstream, especially in sensitive application areas, the risk of data leakage has become a growing concern. Attacks like membership inference (MIA) have shown that trained models can reveal sensitive data, jeopardizing confidentiality. While traditional Artificial Neural Networks (ANNs) dominate ML applications, neuromorphic architectures, specifically Spiking Neural Networks (SNNs), are emerging as promising alternatives due to their low power consumption and event-driven processing, akin to biological neurons. Privacy in ANNs is well-studied; however, little work has explored the privacy-preserving properties of SNNs. This paper examines whether SNNs inherently offer better privacy. Using MIAs, we assess the privacy resilience of SNNs versus ANNs across diverse datasets. We analyze the impact of learning algorithms (surrogate gradient and evolutionary), frameworks (snnTorch, TENNLab, LAVA), and parameters on SNN privacy. Our findings show that SNNs consistently outperform ANNs in privacy preservation, with evolutionary algorithms offering additional resilience. For instance, on CIFAR-10, SNNs achieve an AUC of 0.59, significantly lower than ANNs' 0.82, and on CIFAR-100, SNNs maintain an AUC of 0.58 compared to ANNs' 0.88. Additionally, we explore the privacy-utility trade-off with Differentially Private Stochastic Gradient Descent (DPSGD), finding that SNNs sustain less accuracy loss than ANNs under similar privacy constraints.

Title: Using Diffusion Models as Generative Replay in Continual Federated Learning -- What will Happen?

• Authors: Yongsheng Mei, Liangqi Yuan, Dong-Jun Han, Kevin S. Chan, Christopher G. Brinton, Tian Lan
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.06618
• Pdf URL: https://arxiv.org/pdf/2411.06618
• Copy Paste: [[2411.06618]] Using Diffusion Models as Generative Replay in Continual Federated Learning -- What will Happen?(https://arxiv.org/abs/2411.06618)
• Keywords: federate, diffusion, generative
• Abstract: Federated learning (FL) has become a cornerstone in decentralized learning, where, in many scenarios, the incoming data distribution will change dynamically over time, introducing continuous learning (CL) problems. This continual federated learning (CFL) task presents unique challenges, particularly regarding catastrophic forgetting and non-IID input data. Existing solutions include using a replay buffer to store historical data or leveraging generative adversarial networks. Nevertheless, motivated by recent advancements in the diffusion model for generative tasks, this paper introduces DCFL, a novel framework tailored to address the challenges of CFL in dynamic distributed learning environments. Our approach harnesses the power of the conditional diffusion model to generate synthetic historical data at each local device during communication, effectively mitigating latent shifts in dynamic data distribution inputs. We provide the convergence bound for the proposed CFL framework and demonstrate its promising performance across multiple datasets, showcasing its effectiveness in tackling the complexities of CFL tasks.

Title: Few-shot Semantic Learning for Robust Multi-Biome 3D Semantic Mapping in Off-Road Environments

• Authors: Deegan Atha, Xianmei Lei, Shehryar Khattak, Anna Sabel, Elle Miller, Aurelio Noca, Grace Lim, Jeffrey Edlund, Curtis Padgett, Patrick Spieler
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.06632
• Pdf URL: https://arxiv.org/pdf/2411.06632
• Copy Paste: [[2411.06632]] Few-shot Semantic Learning for Robust Multi-Biome 3D Semantic Mapping in Off-Road Environments(https://arxiv.org/abs/2411.06632)
• Keywords: robust, transformer, segmentation
• Abstract: Off-road environments pose significant perception challenges for high-speed autonomous navigation due to unstructured terrain, degraded sensing conditions, and domain-shifts among biomes. Learning semantic information across these conditions and biomes can be challenging when a large amount of ground truth data is required. In this work, we propose an approach that leverages a pre-trained Vision Transformer (ViT) with fine-tuning on a small (<500 images), sparse and coarsely labeled (<30% pixels) multi-biome dataset to predict 2D semantic segmentation classes. These classes are fused over time via a novel range-based metric and aggregated into a 3D semantic voxel map. We demonstrate zero-shot out-of-biome 2D semantic segmentation on the Yamaha (52.9 mIoU) and Rellis (55.5 mIoU) datasets along with few-shot coarse sparse labeling with existing data for improved segmentation performance on Yamaha (66.6 mIoU) and Rellis (67.2 mIoU). We further illustrate the feasibility of using a voxel map with a range-based semantic fusion approach to handle common off-road hazards like pop-up hazards, overhangs, and water features.

Title: Mixed Effects Deep Learning Autoencoder for interpretable analysis of single cell RNA Sequencing data

• Authors: Aixa X. Andrade, Son Nguyen, Albert Montillo
• Subjects: cs.LG, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2411.06635
• Pdf URL: https://arxiv.org/pdf/2411.06635
• Copy Paste: [[2411.06635]] Mixed Effects Deep Learning Autoencoder for interpretable analysis of single cell RNA Sequencing data(https://arxiv.org/abs/2411.06635)
• Keywords: robust, interpretability
• Abstract: Single-cell RNA sequencing (scRNA-seq) data are often confounded due to technical or biological batch effects. Existing deep learning models aim to mitigate these effects but may inadvertently discard batch-specific information. We propose a Mixed Effects Deep Learning (MEDL) Autoencoder framework that separately models batch-invariant (fixed effects) and batch-specific (random effects) components. By decoupling fixed effects representing biological states from random effects capturing batch-specific variations, MEDL integrates both types of information into predictive models, minimizing information loss. This approach improves interpretability enabling 2D visualizations that show how the same cell would appear across different batches, facilitating exploration of batch-specific variations. We applied MEDL to three datasets: Healthy Heart, Autism Spectrum Disorder (ASDc), and Acute Myeloid Leukemia (AML). In Healthy Heart, MEDL managed 147 batches, assessing its capacity to handle high batch numbers. In ASDc, MEDL captured donor heterogeneity between autistic and healthy individuals, while in AML, it distinguished heterogeneity in a complex setting with variable cell-type presence and malignant cells in diseased donors. These applications demonstrate MEDL's potential to capture fixed and random effects, improve visualization, and enhance predictive accuracy, offering a robust framework for cellular heterogeneity analysis across diverse datasets.

Title: Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data

• Authors: Alex Havrilla, Wenjing Liao
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06646
• Pdf URL: https://arxiv.org/pdf/2411.06646
• Copy Paste: [[2411.06646]] Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data(https://arxiv.org/abs/2411.06646)
• Keywords: transformer, large language model
• Abstract: When training deep neural networks, a model's generalization error is often observed to follow a power scaling law dependent both on the model size and the data size. Perhaps the best known example of such scaling laws are for transformer-based large language models, where networks with billions of parameters are trained on trillions of tokens of text. Yet, despite sustained widespread interest, a rigorous understanding of why transformer scaling laws exist is still missing. To answer this question, we establish novel statistical estimation and mathematical approximation theories for transformers when the input data are concentrated on a low-dimensional manifold. Our theory predicts a power law between the generalization error and both the training data size and the network size for transformers, where the power depends on the intrinsic dimension $d$ of the training data. Notably, the constructed model architecture is shallow, requiring only logarithmic depth in $d$. By leveraging low-dimensional data structures under a manifold hypothesis, we are able to explain transformer scaling laws in a way which respects the data geometry. Moreover, we test our theory with empirical observation by training LLMs on natural language datasets. We find the observed empirical data scaling laws closely agree with our theoretical predictions. Taken together, these results rigorously show the intrinsic dimension of data to be a crucial quantity affecting transformer scaling laws in both theory and practice.

Title: Machine learning enabled velocity model building with uncertainty quantification

• Authors: Rafael Orozco, Huseyin Tuna Erdinc, Yunlin Zeng, Mathias Louboutin, Felix J. Herrmann
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06651
• Pdf URL: https://arxiv.org/pdf/2411.06651
• Copy Paste: [[2411.06651]] Machine learning enabled velocity model building with uncertainty quantification(https://arxiv.org/abs/2411.06651)
• Keywords: diffusion, generative
• Abstract: Accurately characterizing migration velocity models is crucial for a wide range of geophysical applications, from hydrocarbon exploration to monitoring of CO2 sequestration projects. Traditional velocity model building methods such as Full-Waveform Inversion (FWI) are powerful but often struggle with the inherent complexities of the inverse problem, including noise, limited bandwidth, receiver aperture and computational constraints. To address these challenges, we propose a scalable methodology that integrates generative modeling, in the form of Diffusion networks, with physics-informed summary statistics, making it suitable for complicated imaging problems including field datasets. By defining these summary statistics in terms of subsurface-offset image volumes for poor initial velocity models, our approach allows for computationally efficient generation of Bayesian posterior samples for migration velocity models that offer a useful assessment of uncertainty. To validate our approach, we introduce a battery of tests that measure the quality of the inferred velocity models, as well as the quality of the inferred uncertainties. With modern synthetic datasets, we reconfirm gains from using subsurface-image gathers as the conditioning observable. For complex velocity model building involving salt, we propose a new iterative workflow that refines amortized posterior approximations with salt flooding and demonstrate how the uncertainty in the velocity model can be propagated to the final product reverse time migrated images. Finally, we present a proof of concept on field datasets to show that our method can scale to industry-sized problems.

Title: LFSamba: Marry SAM with Mamba for Light Field Salient Object Detection

• Authors: Zhengyi Liu, Longzhen Wang, Xianyong Fang, Zhengzheng Tu, Linbo Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06652
• Pdf URL: https://arxiv.org/pdf/2411.06652
• Copy Paste: [[2411.06652]] LFSamba: Marry SAM with Mamba for Light Field Salient Object Detection(https://arxiv.org/abs/2411.06652)
• Keywords: extraction
• Abstract: A light field camera can reconstruct 3D scenes using captured multi-focus images that contain rich spatial geometric information, enhancing applications in stereoscopic photography, virtual reality, and robotic vision. In this work, a state-of-the-art salient object detection model for multi-focus light field images, called LFSamba, is introduced to emphasize four main insights: (a) Efficient feature extraction, where SAM is used to extract modality-aware discriminative features; (b) Inter-slice relation modeling, leveraging Mamba to capture long-range dependencies across multiple focal slices, thus extracting implicit depth cues; (c) Inter-modal relation modeling, utilizing Mamba to integrate all-focus and multi-focus images, enabling mutual enhancement; (d) Weakly supervised learning capability, developing a scribble annotation dataset from an existing pixel-level mask dataset, establishing the first scribble-supervised baseline for light field salient object this http URL://github.com/liuzywen/LFScribble

Title: Explore the Reasoning Capability of LLMs in the Chess Testbed

• Authors: Shu Wang, Lei Ji, Renxi Wang, Wenxiao Zhao, Haokun Liu, Yifan Hou, Ying Nian Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06655
• Pdf URL: https://arxiv.org/pdf/2411.06655
• Copy Paste: [[2411.06655]] Explore the Reasoning Capability of LLMs in the Chess Testbed(https://arxiv.org/abs/2411.06655)
• Keywords: large language model
• Abstract: Reasoning is a central capability of human intelligence. In recent years, with the advent of large-scale datasets, pretrained large language models have emerged with new capabilities, including reasoning. However, these models still struggle with long-term, complex reasoning tasks, such as playing chess. Based on the observation that expert chess players employ a dual approach combining long-term strategic play with short-term tactical play along with language explanation, we propose improving the reasoning capability of large language models in chess by integrating annotated strategy and tactic. Specifically, we collect a dataset named MATE, which consists of 1 million chess positions with candidate moves annotated by chess experts for strategy and tactics. We finetune the LLaMA-3-8B model and compare it against state-of-the-art commercial language models in the task of selecting better chess moves. Our experiments show that our models perform better than GPT, Claude, and Gemini models. We find that language explanations can enhance the reasoning capability of large language models.

Title: Renaissance: Investigating the Pretraining of Vision-Language Encoders

• Authors: Clayton Fields, Casey Kennington
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06657
• Pdf URL: https://arxiv.org/pdf/2411.06657
• Copy Paste: [[2411.06657]] Renaissance: Investigating the Pretraining of Vision-Language Encoders(https://arxiv.org/abs/2411.06657)
• Keywords: transformer
• Abstract: In the past several years there has been an explosion of available models for vision-language tasks. Unfortunately, the literature still leaves open a number of questions related to best practices in designing and training such models. In this paper we seek to answer several questions related to the pretraining of vision-language encoders through meta-analysis. In our first set of experiments, we show that we can save significant compute at no cost to downstream performance, by freezing large parts of vision-language models during pretraining. In our second set of experiments we examine the effect of basing a VL transformer on a vision model versus a text model. Additionally, we introduce a VL modeling platform called Renaissance that we use to conduct all of the experiments. This program offers a great deal of flexibility in creating, training and evaluating transformer encoders for VL modeling. The source code for Renaissance can be found at this https URL.

Title: Learning from Different Samples: A Source-free Framework for Semi-supervised Domain Adaptation

• Authors: Xinyang Huang, Chuang Zhu, Bowen Zhang, Shanghang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06665
• Pdf URL: https://arxiv.org/pdf/2411.06665
• Copy Paste: [[2411.06665]] Learning from Different Samples: A Source-free Framework for Semi-supervised Domain Adaptation(https://arxiv.org/abs/2411.06665)
• Keywords: robust
• Abstract: Semi-supervised domain adaptation (SSDA) has been widely studied due to its ability to utilize a few labeled target data to improve the generalization ability of the model. However, existing methods only consider designing certain strategies for target samples to adapt, ignoring the exploration of customized learning for different target samples. When the model encounters complex target distribution, existing methods will perform limited due to the inability to clearly and comprehensively learn the knowledge of multiple types of target samples. To fill this gap, this paper focuses on designing a framework to use different strategies for comprehensively mining different target samples. We propose a novel source-free framework (SOUF) to achieve semi-supervised fine-tuning of the source pre-trained model on the target domain. Different from existing SSDA methods, SOUF decouples SSDA from the perspectives of different target samples, specifically designing robust learning techniques for unlabeled, reliably labeled, and noisy pseudo-labeled target samples. For unlabeled target samples, probability-based weighted contrastive learning (PWC) helps the model learn more discriminative feature representations. To mine the latent knowledge of labeled target samples, reliability-based mixup contrastive learning (RMC) learns complex knowledge from the constructed reliable sample set. Finally, predictive regularization learning (PR) further mitigates the misleading effect of noisy pseudo-labeled samples on the model. Extensive experiments on benchmark datasets demonstrate the superiority of our framework over state-of-the-art methods.

Title: WDMoE: Wireless Distributed Mixture of Experts for Large Language Models

• Authors: Nan Xue, Yaping Sun, Zhiyong Chen, Meixia Tao, Xiaodong Xu, Liang Qian, Shuguang Cui, Wenjun Zhang, Ping Zhang
• Subjects: cs.LG, cs.AI, cs.DC, cs.IT
• Abstract URL: https://arxiv.org/abs/2411.06681
• Pdf URL: https://arxiv.org/pdf/2411.06681
• Copy Paste: [[2411.06681]] WDMoE: Wireless Distributed Mixture of Experts for Large Language Models(https://arxiv.org/abs/2411.06681)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but the role of wireless networks in supporting LLMs has not been thoroughly explored. In this paper, we propose a wireless distributed Mixture of Experts (WDMoE) architecture to enable collaborative deployment of LLMs across edge servers at the base station (BS) and mobile devices in wireless networks. Specifically, we decompose the MoE layer in LLMs by placing the gating network and the preceding neural network layer at BS, while distributing the expert networks among the devices. This deployment leverages the parallel inference capabilities of expert networks on mobile devices, effectively utilizing the limited computing and caching resources of these devices. Accordingly, we develop a performance metric for WDMoE-based LLMs, which accounts for both model capability and latency. To minimize the latency while maintaining accuracy, we jointly optimize expert selection and bandwidth allocation based on the performance metric. Moreover, we build a hardware testbed using NVIDIA Jetson kits to validate the effectiveness of WDMoE. Both theoretical simulations and practical hardware experiments demonstrate that the proposed method can significantly reduce the latency without compromising LLM performance.

Title: SeedEdit: Align Image Re-Generation to Image Editing

• Authors: Yichun Shi, Peng Wang, Weilin Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06686
• Pdf URL: https://arxiv.org/pdf/2411.06686
• Copy Paste: [[2411.06686]] SeedEdit: Align Image Re-Generation to Image Editing(https://arxiv.org/abs/2411.06686)
• Keywords: diffusion
• Abstract: We introduce SeedEdit, a diffusion model that is able to revise a given image with any text prompt. In our perspective, the key to such a task is to obtain an optimal balance between maintaining the original image, i.e. image reconstruction, and generating a new image, i.e. image re-generation. To this end, we start from a weak generator (text-to-image model) that creates diverse pairs between such two directions and gradually align it into a strong image editor that well balances between the two tasks. SeedEdit can achieve more diverse and stable editing capability over prior image editing methods, enabling sequential revision over images generated by diffusion models.

Title: Layout Control and Semantic Guidance with Attention Loss Backward for T2I Diffusion Model

• Authors: Guandong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06692
• Pdf URL: https://arxiv.org/pdf/2411.06692
• Copy Paste: [[2411.06692]] Layout Control and Semantic Guidance with Attention Loss Backward for T2I Diffusion Model(https://arxiv.org/abs/2411.06692)
• Keywords: diffusion
• Abstract: Controllable image generation has always been one of the core demands in image generation, aiming to create images that are both creative and logical while satisfying additional specified conditions. In the post-AIGC era, controllable generation relies on diffusion models and is accomplished by maintaining certain components or introducing inference interferences. This paper addresses key challenges in controllable generation: 1. mismatched object attributes during generation and poor prompt-following effects; 2. inadequate completion of controllable layouts. We propose a train-free method based on attention loss backward, cleverly controlling the cross attention map. By utilizing external conditions such as prompts that can reasonably map onto the attention map, we can control image generation without any training or fine-tuning. This method addresses issues like attribute mismatch and poor prompt-following while introducing explicit layout constraints for controllable image generation. Our approach has achieved excellent practical applications in production, and we hope it can serve as an inspiring technical report in this field.

Title: Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noise

• Authors: Shuyao Li, Sushrut Karmalkar, Ilias Diakonikolas, Jelena Diakonikolas
• Subjects: cs.LG, cs.DS, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06697
• Pdf URL: https://arxiv.org/pdf/2411.06697
• Copy Paste: [[2411.06697]] Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noise(https://arxiv.org/abs/2411.06697)
• Keywords: robust
• Abstract: We study the problem of learning a single neuron with respect to the $L_2^2$-loss in the presence of adversarial distribution shifts, where the labels can be arbitrary, and the goal is to find a ``best-fit'' function. More precisely, given training samples from a reference distribution $\mathcal{p}_0$, the goal is to approximate the vector $\mathbf{w}^*$ which minimizes the squared loss with respect to the worst-case distribution that is close in $\chi^2$-divergence to $\mathcal{p}_{0}$. We design a computationally efficient algorithm that recovers a vector $ \hat{\mathbf{w}}$ satisfying $\mathbb{E}_{\mathcal{p}^*} (\sigma(\hat{\mathbf{w}} \cdot \mathbf{x}) - y)^2 \leq C \, \mathbb{E}_{\mathcal{p}^*} (\sigma(\mathbf{w}^* \cdot \mathbf{x}) - y)^2 + \epsilon$, where $C>1$ is a dimension-independent constant and $(\mathbf{w}^*, \mathcal{p}^*)$ is the witness attaining the min-max risk $\min_{\mathbf{w}~:~\|\mathbf{w}\| \leq W} \max_{\mathcal{p}} \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{p}} (\sigma(\mathbf{w} \cdot \mathbf{x}) - y)^2 - \nu \chi^2(\mathcal{p}, \mathcal{p}_0)$. Our algorithm follows a primal-dual framework and is designed by directly bounding the risk with respect to the original, nonconvex $L_2^2$ loss. From an optimization standpoint, our work opens new avenues for the design of primal-dual algorithms under structured nonconvexity.

Title: Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs

• Authors: Jia Syuen Lim, Yadan Luo, Zhi Chen, Tianqi Wei, Scott Chapman, Zi Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06702
• Pdf URL: https://arxiv.org/pdf/2411.06702
• Copy Paste: [[2411.06702]] Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs(https://arxiv.org/abs/2411.06702)
• Keywords: segmentation
• Abstract: In the Detection and Multi-Object Tracking of Sweet Peppers Challenge, we present Track Any Peppers (TAP) - a weakly supervised ensemble technique for sweet peppers tracking. TAP leverages the zero-shot detection capabilities of vision-language foundation models like Grounding DINO to automatically generate pseudo-labels for sweet peppers in video sequences with minimal human intervention. These pseudo-labels, refined when necessary, are used to train a YOLOv8 segmentation network. To enhance detection accuracy under challenging conditions, we incorporate pre-processing techniques such as relighting adjustments and apply depth-based filtering during post-inference. For object tracking, we integrate the Matching by Segment Anything (MASA) adapter with the BoT-SORT algorithm. Our approach achieves a HOTA score of 80.4%, MOTA of 66.1%, Recall of 74.0%, and Precision of 90.7%, demonstrating effective tracking of sweet peppers without extensive manual effort. This work highlights the potential of foundation models for efficient and accurate object detection and tracking in agricultural settings.

Title: United Domain Cognition Network for Salient Object Detection in Optical Remote Sensing Images

• Authors: Yanguang Sun, Jian Yang, Lei Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06703
• Pdf URL: https://arxiv.org/pdf/2411.06703
• Copy Paste: [[2411.06703]] United Domain Cognition Network for Salient Object Detection in Optical Remote Sensing Images(https://arxiv.org/abs/2411.06703)
• Keywords: transformer
• Abstract: Recently, deep learning-based salient object detection (SOD) in optical remote sensing images (ORSIs) have achieved significant breakthroughs. We observe that existing ORSIs-SOD methods consistently center around optimizing pixel features in the spatial domain, progressively distinguishing between backgrounds and objects. However, pixel information represents local attributes, which are often correlated with their surrounding context. Even with strategies expanding the local region, spatial features remain biased towards local characteristics, lacking the ability of global perception. To address this problem, we introduce the Fourier transform that generate global frequency features and achieve an image-size receptive field. To be specific, we propose a novel United Domain Cognition Network (UDCNet) to jointly explore the global-local information in the frequency and spatial domains. Technically, we first design a frequency-spatial domain transformer block that mutually amalgamates the complementary local spatial and global frequency features to strength the capability of initial input features. Furthermore, a dense semantic excavation module is constructed to capture higher-level semantic for guiding the positioning of remote sensing objects. Finally, we devise a dual-branch joint optimization decoder that applies the saliency and edge branches to generate high-quality representations for predicting salient objects. Experimental results demonstrate the superiority of the proposed UDCNet method over 24 state-of-the-art models, through extensive quantitative and qualitative comparisons in three widely-used ORSIs-SOD datasets. The source code is available at: \href{this https URL}{\color{blue} this https URL}.

Title: GTA-Net: An IoT-Integrated 3D Human Pose Estimation System for Real-Time Adolescent Sports Posture Correction

• Authors: Shizhe Yuan, Li Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06725
• Pdf URL: https://arxiv.org/pdf/2411.06725
• Copy Paste: [[2411.06725]] GTA-Net: An IoT-Integrated 3D Human Pose Estimation System for Real-Time Adolescent Sports Posture Correction(https://arxiv.org/abs/2411.06725)
• Keywords: robust
• Abstract: With the advancement of artificial intelligence, 3D human pose estimation-based systems for sports training and posture correction have gained significant attention in adolescent sports. However, existing methods face challenges in handling complex movements, providing real-time feedback, and accommodating diverse postures, particularly with occlusions, rapid movements, and the resource constraints of Internet of Things (IoT) devices, making it difficult to balance accuracy and real-time performance. To address these issues, we propose GTA-Net, an intelligent system for posture correction and real-time feedback in adolescent sports, integrated within an IoT-enabled environment. This model enhances pose estimation in dynamic scenes by incorporating Graph Convolutional Networks (GCN), Temporal Convolutional Networks (TCN), and Hierarchical Attention mechanisms, achieving real-time correction through IoT devices. Experimental results show GTA-Net's superior performance on Human3.6M, HumanEva-I, and MPI-INF-3DHP datasets, with Mean Per Joint Position Error (MPJPE) values of 32.2mm, 15.0mm, and 48.0mm, respectively, significantly outperforming existing methods. The model also demonstrates strong robustness in complex scenarios, maintaining high accuracy even with occlusions and rapid movements. This system enhances real-time posture correction and offers broad applications in intelligent sports and health management.

Title: Can KAN Work? Exploring the Potential of Kolmogorov-Arnold Networks in Computer Vision

• Authors: Yueyang Cang, Yu hang liu, Li Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06727
• Pdf URL: https://arxiv.org/pdf/2411.06727
• Copy Paste: [[2411.06727]] Can KAN Work? Exploring the Potential of Kolmogorov-Arnold Networks in Computer Vision(https://arxiv.org/abs/2411.06727)
• Keywords: robust, segmentation
• Abstract: Kolmogorov-Arnold Networks(KANs), as a theoretically efficient neural network architecture, have garnered attention for their potential in capturing complex patterns. However, their application in computer vision remains relatively unexplored. This study first analyzes the potential of KAN in computer vision tasks, evaluating the performance of KAN and its convolutional variants in image classification and semantic segmentation. The focus is placed on examining their characteristics across varying data scales and noise levels. Results indicate that while KAN exhibits stronger fitting capabilities, it is highly sensitive to noise, limiting its robustness. To address this challenge, we propose a smoothness regularization method and introduce a Segment Deactivation technique. Both approaches enhance KAN's stability and generalization, demonstrating its potential in handling complex visual data tasks.

Title: Reverse Prompt Engineering

• Authors: Hanqing Li, Diego Klabjan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06729
• Pdf URL: https://arxiv.org/pdf/2411.06729
• Copy Paste: [[2411.06729]] Reverse Prompt Engineering(https://arxiv.org/abs/2411.06729)
• Keywords: large language model
• Abstract: This paper explores a new black-box, zero-shot language model inversion problem and proposes an innovative framework for prompt reconstruction using only text outputs from a language model. Leveraging a large language model alongside an optimization algorithm, the proposed method effectively recovers prompts with minimal resources. Experimental results on several datasets derived from public sources indicate that the proposed approach achieves high-quality prompt recovery and generates prompts more similar to the originals than current state-of-the-art methods. Additionally, the use-case study demonstrates the method's strong potential for generating high-quality text data.

Title: Dockformer: A transformer-based molecular docking paradigm for large-scale virtual screening

• Authors: Zhangfan Yang, Junkai Ji, Shan He, Jianqiang Li, Ruibin Bai, Zexuan Zhu, Yew Soon Ong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06740
• Pdf URL: https://arxiv.org/pdf/2411.06740
• Copy Paste: [[2411.06740]] Dockformer: A transformer-based molecular docking paradigm for large-scale virtual screening(https://arxiv.org/abs/2411.06740)
• Keywords: robust, transformer
• Abstract: Molecular docking enables virtual screening of compound libraries to identify potential ligands that target proteins of interest, a crucial step in drug development; however, as the size of the compound library increases, the computational complexity of traditional docking models increases. Deep learning algorithms can provide data-driven research and development models to increase the speed of the docking process. Unfortunately, few models can achieve superior screening performance compared to that of traditional models. Therefore, a novel deep learning-based docking approach named Dockformer is introduced in this study. Dockformer leverages multimodal information to capture the geometric topology and structural knowledge of molecules and can directly generate binding conformations with the corresponding confidence measures in an end-to-end manner. The experimental results show that Dockformer achieves success rates of 90.53\% and 82.71\% on the PDBbind core set and PoseBusters benchmarks, respectively, and more than a 100-fold increase in the inference process speed, outperforming almost all state-of-the-art docking methods. In addition, the ability of Dockformer to identify the main protease inhibitors of coronaviruses is demonstrated in a real-world virtual screening scenario. Considering its high docking accuracy and screening efficiency, Dockformer can be regarded as a powerful and robust tool in the field of drug design.

Title: Research on an intelligent fault diagnosis method for nuclear power plants based on ETCN-SSA combined algorithm

• Authors: Jiayan Fang, Siwei Li, Yichun Wu
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.06765
• Pdf URL: https://arxiv.org/pdf/2411.06765
• Copy Paste: [[2411.06765]] Research on an intelligent fault diagnosis method for nuclear power plants based on ETCN-SSA combined algorithm(https://arxiv.org/abs/2411.06765)
• Keywords: extraction
• Abstract: Utilizing fault diagnosis methods is crucial for nuclear power professionals to achieve efficient and accurate fault diagnosis for nuclear power plants (NPPs). The performance of traditional methods is limited by their dependence on complex feature extraction and skilled expert knowledge, which can be time-consuming and subjective. This paper proposes a novel intelligent fault diagnosis method for NPPs that combines enhanced temporal convolutional network (ETCN) with sparrow search algorithm (SSA). ETCN utilizes temporal convolutional network (TCN), self-attention (SA) mechanism and residual block for enhancing performance. ETCN excels at extracting local features and capturing time series information, while SSA adaptively optimizes its hyperparameters for superior performance. The proposed method's performance is experimentally verified on a CPR1000 simulation dataset. Compared to other advanced intelligent fault diagnosis methods, the proposed one demonstrates superior performance across all evaluation metrics. This makes it a promising tool for NPP intelligent fault diagnosis, ultimately enhancing operational reliability.

Title: PDC & DM-SFT: A Road for LLM SQL Bug-Fix Enhancing

• Authors: Yiwen Duan, Yonghong Yu, Xiaoming Zhao, Yichang Wu, Wenbo Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06767
• Pdf URL: https://arxiv.org/pdf/2411.06767
• Copy Paste: [[2411.06767]] PDC & DM-SFT: A Road for LLM SQL Bug-Fix Enhancing(https://arxiv.org/abs/2411.06767)
• Keywords: large language model
• Abstract: Code Large Language Models (Code LLMs), such as Code llama and DeepSeek-Coder, have demonstrated exceptional performance in the code generation tasks. However, most existing models focus on the abilities of generating correct code, but often struggle with bug repair. We introduce a suit of methods to enhance LLM's SQL bug-fixing abilities. The methods are mainly consisted of two parts: A Progressive Dataset Construction (PDC) from scratch and Dynamic Mask Supervised Fine-tuning (DM-SFT). PDC proposes two data expansion methods from the perspectives of breadth first and depth first respectively. DM-SFT introduces an efficient bug-fixing supervised learning approach, which effectively reduce the total training steps and mitigate the "disorientation" in SQL code bug-fixing training. In our evaluation, the code LLM models trained with two methods have exceeds all current best performing model which size is much larger.

Title: Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis

• Authors: Zhijie Chen, Qiaobo Li, Arindam Banerjee
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06770
• Pdf URL: https://arxiv.org/pdf/2411.06770
• Copy Paste: [[2411.06770]] Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis(https://arxiv.org/abs/2411.06770)
• Keywords: federate
• Abstract: Combining gradient compression methods (e.g., CountSketch, quantization) and adaptive optimizers (e.g., Adam, AMSGrad) is a desirable goal in federated learning (FL), with potential benefits on both fewer communication rounds and less per-round communication. In spite of the preliminary empirical success of sketched adaptive methods, existing convergence analyses show the communication cost to have a linear dependence on the ambient dimension, i.e., number of parameters, which is prohibitively high for modern deep learning models. In this work, we introduce specific sketched adaptive federated learning (SAFL) algorithms and, as our main contribution, provide theoretical convergence analyses in different FL settings with guarantees on communication cost depending only logarithmically (instead of linearly) on the ambient dimension. Unlike existing analyses, we show that the entry-wise sketching noise existent in the preconditioners and the first moments of SAFL can be implicitly addressed by leveraging the recently-popularized anisotropic curvatures in deep learning losses, e.g., fast decaying loss Hessian eigen-values. In the i.i.d. client setting of FL, we show that SAFL achieves asymptotic $O(1/\sqrt{T})$ convergence, and converges faster in the initial epochs. In the non-i.i.d. client setting, where non-adaptive methods lack convergence guarantees, we show that SACFL (SAFL with clipping) algorithms can provably converge in spite of the additional heavy-tailed noise. Our theoretical claims are supported by empirical studies on vision and language tasks, and in both fine-tuning and training-from-scratch regimes. Surprisingly, as a by-product of our analysis, the proposed SAFL methods are competitive with the state-of-the-art communication-efficient federated learning algorithms based on error feedback.

Title: Model Partition and Resource Allocation for Split Learning in Vehicular Edge Networks

• Authors: Lu Yu, Zheng Chang, Yunjian Jia, Geyong Min
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.06773
• Pdf URL: https://arxiv.org/pdf/2411.06773
• Copy Paste: [[2411.06773]] Model Partition and Resource Allocation for Split Learning in Vehicular Edge Networks(https://arxiv.org/abs/2411.06773)
• Keywords: privacy, protect, federate
• Abstract: The integration of autonomous driving technologies with vehicular networks presents significant challenges in privacy preservation, communication efficiency, and resource allocation. This paper proposes a novel U-shaped split federated learning (U-SFL) framework to address these challenges on the way of realizing in vehicular edge networks. U-SFL is able to enhance privacy protection by keeping both raw data and labels on the vehicular user (VU) side while enabling parallel processing across multiple vehicles. To optimize communication efficiency, we introduce a semantic-aware auto-encoder (SAE) that significantly reduces the dimensionality of transmitted data while preserving essential semantic information. Furthermore, we develop a deep reinforcement learning (DRL) based algorithm to solve the NP-hard problem of dynamic resource allocation and split point selection. Our comprehensive evaluation demonstrates that U-SFL achieves comparable classification performance to traditional split learning (SL) while substantially reducing data transmission volume and communication latency. The proposed DRL-based optimization algorithm shows good convergence in balancing latency, energy consumption, and learning performance.

Title: HSTrack: Bootstrap End-to-End Multi-Camera 3D Multi-object Tracking with Hybrid Supervision

• Authors: Shubo Lin, Yutong Kou, Bing Li, Weiming Hu, Jin Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06780
• Pdf URL: https://arxiv.org/pdf/2411.06780
• Copy Paste: [[2411.06780]] HSTrack: Bootstrap End-to-End Multi-Camera 3D Multi-object Tracking with Hybrid Supervision(https://arxiv.org/abs/2411.06780)
• Keywords: transformer
• Abstract: In camera-based 3D multi-object tracking (MOT), the prevailing methods follow the tracking-by-query-propagation paradigm, which employs track queries to manage the lifecycle of identity-consistent tracklets while object queries handle the detection of new-born tracklets. However, this intertwined paradigm leads the inter-temporal tracking task and the single-frame detection task utilize the same model parameters, complicating training optimization. Drawing inspiration from studies on the roles of attention components in transformer-based decoders, we identify that the dispersing effect of self-attention necessitates object queries to match with new-born tracklets. This matching strategy diverges from the detection pre-training phase, where object queries align with all ground-truth targets, resulting in insufficient supervision signals. To address these issues, we present HSTrack, a novel plug-and-play method designed to co-facilitate multi-task learning for detection and tracking. HSTrack constructs a parallel weight-share decoder devoid of self-attention layers, circumventing competition between different types of queries. Considering the characteristics of cross-attention layer and distinct query types, our parallel decoder adopt one-to-one and one-to-many label assignment strategies for track queries and object queries, respectively. Leveraging the shared architecture, HSTrack further improve trackers for spatio-temporal modeling and quality candidates generation. Extensive experiments demonstrate that HSTrack consistently delivers improvements when integrated with various query-based 3D MOT trackers. For example, HSTrack improves the state-of-the-art PF-Track method by $+2.3\%$ AMOTA and $+1.7\%$ mAP on the nuScenes dataset.

Title: White-Box Diffusion Transformer for single-cell RNA-seq generation

• Authors: Zhuorui Cui, Shengze Dong, Ding Liu
• Subjects: cs.LG, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2411.06785
• Pdf URL: https://arxiv.org/pdf/2411.06785
• Copy Paste: [[2411.06785]] White-Box Diffusion Transformer for single-cell RNA-seq generation(https://arxiv.org/abs/2411.06785)
• Keywords: interpretability, diffusion, transformer, generative
• Abstract: As a powerful tool for characterizing cellular subpopulations and cellular heterogeneity, single cell RNA sequencing (scRNA-seq) technology offers advantages of high throughput and multidimensional analysis. However, the process of data acquisition is often constrained by high cost and limited sample availability. To overcome these limitations, we propose a hybrid model based on Diffusion model and White-Box transformer that aims to generate synthetic and biologically plausible scRNA-seq data. Diffusion model progressively introduce noise into the data and then recover the original data through a denoising process, a forward and reverse process that is particularly suitable for generating complex data distributions. White-Box transformer is a deep learning architecture that emphasizes mathematical interpretability. By minimizing the encoding rate of the data and maximizing the sparsity of the representation, it not only reduces the computational burden, but also provides clear insight into underlying structure. Our White-Box Diffusion Transformer combines the generative capabilities of Diffusion model with the mathematical interpretability of White-Box transformer. Through experiments using six different single-cell RNA-Seq datasets, we visualize both generated and real data using t-SNE dimensionality reduction technique, as well as quantify similarity between generated and real data using various metrics to demonstrate comparable performance of White-Box Diffusion Transformer and Diffusion Transformer in generating scRNA-seq data alongside significant improvements in training efficiency and resource utilization. Our code is available at this https URL

Title: ScaleKD: Strong Vision Transformers Could Be Excellent Teachers

• Authors: Jiawei Fan, Chao Li, Xiaolong Liu, Anbang Yao
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06786
• Pdf URL: https://arxiv.org/pdf/2411.06786
• Copy Paste: [[2411.06786]] ScaleKD: Strong Vision Transformers Could Be Excellent Teachers(https://arxiv.org/abs/2411.06786)
• Keywords: transformer
• Abstract: In this paper, we question if well pre-trained vision transformer (ViT) models could be used as teachers that exhibit scalable properties to advance cross architecture knowledge distillation (KD) research, in the context of using large-scale datasets for evaluation. To make this possible, our analysis underlines the importance of seeking effective strategies to align (1) feature computing paradigm differences, (2) model scale differences, and (3) knowledge density differences. By combining three coupled components namely cross attention projector, dual-view feature mimicking and teacher parameter perception tailored to address the above problems, we present a simple and effective KD method, called ScaleKD. Our method can train student backbones that span across a variety of convolutional neural network (CNN), multi-layer perceptron (MLP), and ViT architectures on image classification datasets, achieving state-of-the-art distillation performance. For instance, taking a well pre-trained Swin-L as the teacher model, our method gets 75.15%|82.03%|84.16%|78.63%|81.96%|83.93%|83.80%|85.53% top-1 accuracies for MobileNet-V1|ResNet-50|ConvNeXt-T|Mixer-S/16|Mixer-B/16|ViT-S/16|Swin-T|ViT-B/16 models trained on ImageNet-1K dataset from scratch, showing 3.05%|3.39%|2.02%|4.61%|5.52%|4.03%|2.62%|3.73% absolute gains to the individually trained counterparts. Intriguingly, when scaling up the size of teacher models or their pre-training datasets, our method showcases the desired scalable properties, bringing increasingly larger gains to student models. The student backbones trained by our method transfer well on downstream MS-COCO and ADE20K datasets. More importantly, our method could be used as a more efficient alternative to the time-intensive pre-training paradigm for any target student model if a strong pre-trained ViT is available, reducing the amount of viewed training samples up to 195x.

Title: AssistRAG: Boosting the Potential of Large Language Models with an Intelligent Information Assistant

• Authors: Yujia Zhou, Zheng Liu, Zhicheng Dou
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.06805
• Pdf URL: https://arxiv.org/pdf/2411.06805
• Copy Paste: [[2411.06805]] AssistRAG: Boosting the Potential of Large Language Models with an Intelligent Information Assistant(https://arxiv.org/abs/2411.06805)
• Keywords: large language model
• Abstract: The emergence of Large Language Models (LLMs) has significantly advanced natural language processing, but these models often generate factually incorrect information, known as "hallucination". Initial retrieval-augmented generation (RAG) methods like the "Retrieve-Read" framework was inadequate for complex reasoning tasks. Subsequent prompt-based RAG strategies and Supervised Fine-Tuning (SFT) methods improved performance but required frequent retraining and risked altering foundational LLM capabilities. To cope with these challenges, we propose Assistant-based Retrieval-Augmented Generation (AssistRAG), integrating an intelligent information assistant within LLMs. This assistant manages memory and knowledge through tool usage, action execution, memory building, and plan specification. Using a two-phase training approach, Curriculum Assistant Learning and Reinforced Preference Optimization. AssistRAG enhances information retrieval and decision-making. Experiments show AssistRAG significantly outperforms benchmarks, especially benefiting less advanced LLMs, by providing superior reasoning capabilities and accurate responses.

Title: Streetwise Agents: Empowering Offline RL Policies to Outsmart Exogenous Stochastic Disturbances in RTC

• Authors: Aditya Soni, Mayukh Das, Anjaly Parayil, Supriyo Ghosh, Shivam Shandilya, Ching-An Cheng, Vishak Gopal, Sami Khairy, Gabriel Mittag, Yasaman Hosseinkashi, Chetan Bansal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06815
• Pdf URL: https://arxiv.org/pdf/2411.06815
• Copy Paste: [[2411.06815]] Streetwise Agents: Empowering Offline RL Policies to Outsmart Exogenous Stochastic Disturbances in RTC(https://arxiv.org/abs/2411.06815)
• Keywords: robust
• Abstract: The difficulty of exploring and training online on real production systems limits the scope of real-time online data/feedback-driven decision making. The most feasible approach is to adopt offline reinforcement learning from limited trajectory samples. However, after deployment, such policies fail due to exogenous factors that temporarily or permanently disturb/alter the transition distribution of the assumed decision process structure induced by offline samples. This results in critical policy failures and generalization errors in sensitive domains like Real-Time Communication (RTC). We solve this crucial problem of identifying robust actions in presence of domain shifts due to unseen exogenous stochastic factors in the wild. As it is impossible to learn generalized offline policies within the support of offline data that are robust to these unseen exogenous disturbances, we propose a novel post-deployment shaping of policies (Streetwise), conditioned on real-time characterization of out-of-distribution sub-spaces. This leads to robust actions in bandwidth estimation (BWE) of network bottlenecks in RTC and in standard benchmarks. Our extensive experimental results on BWE and other standard offline RL benchmark environments demonstrate a significant improvement ($\approx$ 18% on some scenarios) in final returns wrt. end-user metrics over state-of-the-art baselines.

Title: Large Language Model in Medical Informatics: Direct Classification and Enhanced Text Representations for Automatic ICD Coding

• Authors: Zeyd Boukhers, AmeerAli Khan, Qusai Ramadan, Cong Yang
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.06823
• Pdf URL: https://arxiv.org/pdf/2411.06823
• Copy Paste: [[2411.06823]] Large Language Model in Medical Informatics: Direct Classification and Enhanced Text Representations for Automatic ICD Coding(https://arxiv.org/abs/2411.06823)
• Keywords: large language model
• Abstract: Addressing the complexity of accurately classifying International Classification of Diseases (ICD) codes from medical discharge summaries is challenging due to the intricate nature of medical documentation. This paper explores the use of Large Language Models (LLM), specifically the LLAMA architecture, to enhance ICD code classification through two methodologies: direct application as a classifier and as a generator of enriched text representations within a Multi-Filter Residual Convolutional Neural Network (MultiResCNN) framework. We evaluate these methods by comparing them against state-of-the-art approaches, revealing LLAMA's potential to significantly improve classification outcomes by providing deep contextual insights into medical texts.

Title: HarmLevelBench: Evaluating Harm-Level Compliance and the Impact of Quantization on Model Alignment

• Authors: Yannis Belkhiter, Giulio Zizzo, Sergio Maffeis
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.06835
• Pdf URL: https://arxiv.org/pdf/2411.06835
• Copy Paste: [[2411.06835]] HarmLevelBench: Evaluating Harm-Level Compliance and the Impact of Quantization on Model Alignment(https://arxiv.org/abs/2411.06835)
• Keywords: security, attack, robust, transformer
• Abstract: With the introduction of the transformers architecture, LLMs have revolutionized the NLP field with ever more powerful models. Nevertheless, their development came up with several challenges. The exponential growth in computational power and reasoning capabilities of language models has heightened concerns about their security. As models become more powerful, ensuring their safety has become a crucial focus in research. This paper aims to address gaps in the current literature on jailbreaking techniques and the evaluation of LLM vulnerabilities. Our contributions include the creation of a novel dataset designed to assess the harmfulness of model outputs across multiple harm levels, as well as a focus on fine-grained harm-level analysis. Using this framework, we provide a comprehensive benchmark of state-of-the-art jailbreaking attacks, specifically targeting the Vicuna 13B v1.5 model. Additionally, we examine how quantization techniques, such as AWQ and GPTQ, influence the alignment and robustness of models, revealing trade-offs between enhanced robustness with regards to transfer attacks and potential increases in vulnerability on direct ones. This study aims to demonstrate the influence of harmful input queries on the complexity of jailbreaking techniques, as well as to deepen our understanding of LLM vulnerabilities and improve methods for assessing model robustness when confronted with harmful content, particularly in the context of compression strategies.

Title: Spatially Constrained Transformer with Efficient Global Relation Modelling for Spatio-Temporal Prediction

• Authors: Ashutosh Sao, Simon Gottschalk
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06836
• Pdf URL: https://arxiv.org/pdf/2411.06836
• Copy Paste: [[2411.06836]] Spatially Constrained Transformer with Efficient Global Relation Modelling for Spatio-Temporal Prediction(https://arxiv.org/abs/2411.06836)
• Keywords: transformer
• Abstract: Accurate spatio-temporal prediction is crucial for the sustainable development of smart cities. However, current approaches often struggle to capture important spatio-temporal relationships, particularly overlooking global relations among distant city regions. Most existing techniques predominantly rely on Convolutional Neural Networks (CNNs) to capture global relations. However, CNNs exhibit neighbourhood bias, making them insufficient for capturing distant relations. To address this limitation, we propose ST-SampleNet, a novel transformer-based architecture that combines CNNs with self-attention mechanisms to capture both local and global relations effectively. Moreover, as the number of regions increases, the quadratic complexity of self-attention becomes a challenge. To tackle this issue, we introduce a lightweight region sampling strategy that prunes non-essential regions and enhances the efficiency of our approach. Furthermore, we introduce a spatially constrained position embedding that incorporates spatial neighbourhood information into the self-attention mechanism, aiding in semantic interpretation and improving the performance of ST-SampleNet. Our experimental evaluation on three real-world datasets demonstrates the effectiveness of ST-SampleNet. Additionally, our efficient variant achieves a 40% reduction in computational costs with only a marginal compromise in performance, approximately 1%.

Title: Persuasion with Large Language Models: a Survey

• Authors: Alexander Rogiers, Sander Noels, Maarten Buyl, Tijl De Bie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06837
• Pdf URL: https://arxiv.org/pdf/2411.06837
• Copy Paste: [[2411.06837]] Persuasion with Large Language Models: a Survey(https://arxiv.org/abs/2411.06837)
• Keywords: privacy, large language model
• Abstract: The rapid rise of Large Language Models (LLMs) has created new disruptive possibilities for persuasive communication, by enabling fully-automated personalized and interactive content generation at an unprecedented scale. In this paper, we survey the research field of LLM-based persuasion that has emerged as a result. We begin by exploring the different modes in which LLM Systems are used to influence human attitudes and behaviors. In areas such as politics, marketing, public health, e-commerce, and charitable giving, such LLM Systems have already achieved human-level or even super-human persuasiveness. We identify key factors influencing their effectiveness, such as the manner of personalization and whether the content is labelled as AI-generated. We also summarize the experimental designs that have been used to evaluate progress. Our survey suggests that the current and future potential of LLM-based persuasion poses profound ethical and societal risks, including the spread of misinformation, the magnification of biases, and the invasion of privacy. These risks underscore the urgent need for ethical guidelines and updated regulatory frameworks to avoid the widespread deployment of irresponsible and harmful LLM Systems.

Title: LLM-Neo: Parameter Efficient Knowledge Distillation for Large Language Models

• Authors: Runming Yang, Taiqiang Wu, Jiahao Wang, Pengfei Hu, Ngai Wong, Yujiu Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06839
• Pdf URL: https://arxiv.org/pdf/2411.06839
• Copy Paste: [[2411.06839]] LLM-Neo: Parameter Efficient Knowledge Distillation for Large Language Models(https://arxiv.org/abs/2411.06839)
• Keywords: robust, large language model
• Abstract: In this paper, we propose a novel LLM-Neo framework that efficiently transfers knowledge from a large language model (LLM) teacher to a compact student. Initially, we revisit the knowledge distillation (KD) and low-rank adaption (LoRA), and argue that they share the same paradigm. Inspired by this observation, we explore the strategy that combines LoRA and KD to enhance the efficiency of knowledge transfer. We first summarize some guidelines for this design and further develop the LLM-Neo. Experimental results on compressing Llama 2 and Llama 3 show that LLM-Neo outperforms various baselines. Further analysis demonstrates the robustness of the proposed LLM-Neo on variants of LoRA. The trained models have been available at \href{this https URL}{this repository}.

Title: Generative Feature Training of Thin 2-Layer Networks

• Authors: Johannes Hertrich, Sebastian Neumayer
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06848
• Pdf URL: https://arxiv.org/pdf/2411.06848
• Copy Paste: [[2411.06848]] Generative Feature Training of Thin 2-Layer Networks(https://arxiv.org/abs/2411.06848)
• Keywords: generative
• Abstract: We consider the approximation of functions by 2-layer neural networks with a small number of hidden weights based on the squared loss and small datasets. Due to the highly non-convex energy landscape, gradient-based training often suffers from local minima. As a remedy, we initialize the hidden weights with samples from a learned proposal distribution, which we parameterize as a deep generative model. To train this model, we exploit the fact that with fixed hidden weights, the optimal output weights solve a linear equation. After learning the generative model, we refine the sampled weights with a gradient-based post-processing in the latent space. Here, we also include a regularization scheme to counteract potential noise. Finally, we demonstrate the effectiveness of our approach by numerical examples.

Title: 1-800-SHARED-TASKS @ NLU of Devanagari Script Languages: Detection of Language, Hate Speech, and Targets using LLMs

• Authors: Jebish Purbey, Siddartha Pullakhandam, Kanwal Mehreen, Muhammad Arham, Drishti Sharma, Ashay Srivastava, Ram Mohan Rao Kadiyala
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06850
• Pdf URL: https://arxiv.org/pdf/2411.06850
• Copy Paste: [[2411.06850]] 1-800-SHARED-TASKS @ NLU of Devanagari Script Languages: Detection of Language, Hate Speech, and Targets using LLMs(https://arxiv.org/abs/2411.06850)
• Keywords: transformer, large language model
• Abstract: This paper presents a detailed system description of our entry for the CHiPSAL 2025 shared task, focusing on language detection, hate speech identification, and target detection in Devanagari script languages. We experimented with a combination of large language models and their ensembles, including MuRIL, IndicBERT, and Gemma-2, and leveraged unique techniques like focal loss to address challenges in the natural understanding of Devanagari languages, such as multilingual processing and class imbalance. Our approach achieved competitive results across all tasks: F1 of 0.9980, 0.7652, and 0.6804 for Sub-tasks A, B, and C respectively. This work provides insights into the effectiveness of transformer models in tasks with domain-specific and linguistic challenges, as well as areas for potential improvement in future iterations.

Title: Fast and Efficient Transformer-based Method for Bird's Eye View Instance Prediction

• Authors: Miguel Antunes-García, Luis M. Bergasa, Santiago Montiel-Marín, Rafael Barea, Fabio Sánchez-García, Ángel Llamazares
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06851
• Pdf URL: https://arxiv.org/pdf/2411.06851
• Copy Paste: [[2411.06851]] Fast and Efficient Transformer-based Method for Bird's Eye View Instance Prediction(https://arxiv.org/abs/2411.06851)
• Keywords: transformer, segmentation
• Abstract: Accurate object detection and prediction are critical to ensure the safety and efficiency of self-driving architectures. Predicting object trajectories and occupancy enables autonomous vehicles to anticipate movements and make decisions with future information, increasing their adaptability and reducing the risk of accidents. Current State-Of-The-Art (SOTA) approaches often isolate the detection, tracking, and prediction stages, which can lead to significant prediction errors due to accumulated inaccuracies between stages. Recent advances have improved the feature representation of multi-camera perception systems through Bird's-Eye View (BEV) transformations, boosting the development of end-to-end systems capable of predicting environmental elements directly from vehicle sensor data. These systems, however, often suffer from high processing times and number of parameters, creating challenges for real-world deployment. To address these issues, this paper introduces a novel BEV instance prediction architecture based on a simplified paradigm that relies only on instance segmentation and flow prediction. The proposed system prioritizes speed, aiming at reduced parameter counts and inference times compared to existing SOTA architectures, thanks to the incorporation of an efficient transformer-based architecture. Furthermore, the implementation of the proposed architecture is optimized for performance improvements in PyTorch version 2.1. Code and trained models are available at this https URL

Title: Evaluating Large Language Models on Financial Report Summarization: An Empirical Study

• Authors: Xinqi Yang, Scott Zang, Yong Ren, Dingjie Peng, Zheng Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06852
• Pdf URL: https://arxiv.org/pdf/2411.06852
• Copy Paste: [[2411.06852]] Evaluating Large Language Models on Financial Report Summarization: An Empirical Study(https://arxiv.org/abs/2411.06852)
• Keywords: robust, large language model
• Abstract: In recent years, Large Language Models (LLMs) have demonstrated remarkable versatility across various applications, including natural language understanding, domain-specific knowledge tasks, etc. However, applying LLMs to complex, high-stakes domains like finance requires rigorous evaluation to ensure reliability, accuracy, and compliance with industry standards. To address this need, we conduct a comprehensive and comparative study on three state-of-the-art LLMs, GLM-4, Mistral-NeMo, and LLaMA3.1, focusing on their effectiveness in generating automated financial reports. Our primary motivation is to explore how these models can be harnessed within finance, a field demanding precision, contextual relevance, and robustness against erroneous or misleading information. By examining each model's capabilities, we aim to provide an insightful assessment of their strengths and limitations. Our paper offers benchmarks for financial report analysis, encompassing proposed metrics such as ROUGE-1, BERT Score, and LLM Score. We introduce an innovative evaluation framework that integrates both quantitative metrics (e.g., precision, recall) and qualitative analyses (e.g., contextual fit, consistency) to provide a holistic view of each model's output quality. Additionally, we make our financial dataset publicly available, inviting researchers and practitioners to leverage, scrutinize, and enhance our findings through broader community engagement and collaborative improvement. Our dataset is available on huggingface.

Title: A Unified Multi-Task Learning Architecture for Hate Detection Leveraging User-Based Information

• Authors: Prashant Kapil, Asif Ekbal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06855
• Pdf URL: https://arxiv.org/pdf/2411.06855
• Copy Paste: [[2411.06855]] A Unified Multi-Task Learning Architecture for Hate Detection Leveraging User-Based Information(https://arxiv.org/abs/2411.06855)
• Keywords: transformer
• Abstract: Hate speech, offensive language, aggression, racism, sexism, and other abusive language are common phenomena in social media. There is a need for Artificial Intelligence(AI)based intervention which can filter hate content at scale. Most existing hate speech detection solutions have utilized the features by treating each post as an isolated input instance for the classification. This paper addresses this issue by introducing a unique model that improves hate speech identification for the English language by utilising intra-user and inter-user-based information. The experiment is conducted over single-task learning (STL) and multi-task learning (MTL) paradigms that use deep neural networks, such as convolutional neural networks (CNN), gated recurrent unit (GRU), bidirectional encoder representations from the transformer (BERT), and A Lite BERT (ALBERT). We use three benchmark datasets and conclude that combining certain user features with textual features gives significant improvements in macro-F1 and weighted-F1.

Title: Scientific machine learning in ecological systems: A study on the predator-prey dynamics

• Authors: Ranabir Devgupta, Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06858
• Pdf URL: https://arxiv.org/pdf/2411.06858
• Copy Paste: [[2411.06858]] Scientific machine learning in ecological systems: A study on the predator-prey dynamics(https://arxiv.org/abs/2411.06858)
• Keywords: robust
• Abstract: In this study, we apply two pillars of Scientific Machine Learning: Neural Ordinary Differential Equations (Neural ODEs) and Universal Differential Equations (UDEs) to the Lotka Volterra Predator Prey Model, a fundamental ecological model describing the dynamic interactions between predator and prey populations. The Lotka-Volterra model is critical for understanding ecological dynamics, population control, and species interactions, as it is represented by a system of differential equations. In this work, we aim to uncover the underlying differential equations without prior knowledge of the system, relying solely on training data and neural networks. Using robust modeling in the Julia programming language, we demonstrate that both Neural ODEs and UDEs can be effectively utilized for prediction and forecasting of the Lotka-Volterra system. More importantly, we introduce the forecasting breakdown point: the time at which forecasting fails for both Neural ODEs and UDEs. We observe how UDEs outperform Neural ODEs by effectively recovering the underlying dynamics and achieving accurate forecasting with significantly less training data. Additionally, we introduce Gaussian noise of varying magnitudes (from mild to high) to simulate real-world data perturbations and show that UDEs exhibit superior robustness, effectively recovering the underlying dynamics even in the presence of noisy data, while Neural ODEs struggle with high levels of noise. Through extensive hyperparameter optimization, we offer insights into neural network architectures, activation functions, and optimizers that yield the best results. This study opens the door to applying Scientific Machine Learning frameworks for forecasting tasks across a wide range of ecological and scientific domains.

Title: Enhancing Phishing Detection through Feature Importance Analysis and Explainable AI: A Comparative Study of CatBoost, XGBoost, and EBM Models

• Authors: Abdullah Fajar, Setiadi Yazid, Indra Budi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06860
• Pdf URL: https://arxiv.org/pdf/2411.06860
• Copy Paste: [[2411.06860]] Enhancing Phishing Detection through Feature Importance Analysis and Explainable AI: A Comparative Study of CatBoost, XGBoost, and EBM Models(https://arxiv.org/abs/2411.06860)
• Keywords: security, defense, attack, robust, interpretability
• Abstract: Phishing attacks remain a persistent threat to online security, demanding robust detection methods. This study investigates the use of machine learning to identify phishing URLs, emphasizing the crucial role of feature selection and model interpretability for improved performance. Employing Recursive Feature Elimination, the research pinpointed key features like "length_url," "time_domain_activation" and "Page_rank" as strong indicators of phishing attempts. The study evaluated various algorithms, including CatBoost, XGBoost, and Explainable Boosting Machine, assessing their robustness and scalability. XGBoost emerged as highly efficient in terms of runtime, making it well-suited for large datasets. CatBoost, on the other hand, demonstrated resilience by maintaining high accuracy even with reduced features. To enhance transparency and trustworthiness, Explainable AI techniques, such as SHAP, were employed to provide insights into feature importance. The study's findings highlight that effective feature selection and model interpretability can significantly bolster phishing detection systems, paving the way for more efficient and adaptable defenses against evolving cyber threats

Title: Computable Model-Independent Bounds for Adversarial Quantum Machine Learning

• Authors: Bacui Li, Tansu Alpcan, Chandra Thapa, Udaya Parampalli
• Subjects: cs.LG, cs.AI, cs.ET, quant-ph
• Abstract URL: https://arxiv.org/abs/2411.06863
• Pdf URL: https://arxiv.org/pdf/2411.06863
• Copy Paste: [[2411.06863]] Computable Model-Independent Bounds for Adversarial Quantum Machine Learning(https://arxiv.org/abs/2411.06863)
• Keywords: attack, robust
• Abstract: By leveraging the principles of quantum mechanics, QML opens doors to novel approaches in machine learning and offers potential speedup. However, machine learning models are well-documented to be vulnerable to malicious manipulations, and this susceptibility extends to the models of QML. This situation necessitates a thorough understanding of QML's resilience against adversarial attacks, particularly in an era where quantum computing capabilities are expanding. In this regard, this paper examines model-independent bounds on adversarial performance for QML. To the best of our knowledge, we introduce the first computation of an approximate lower bound for adversarial error when evaluating model resilience against sophisticated quantum-based adversarial attacks. Experimental results are compared to the computed bound, demonstrating the potential of QML models to achieve high robustness. In the best case, the experimental error is only 10% above the estimated bound, offering evidence of the inherent robustness of quantum models. This work not only advances our theoretical understanding of quantum model resilience but also provides a precise reference bound for the future development of robust QML algorithms.

Title: Veri-Car: Towards Open-world Vehicle Information Retrieval

• Authors: Andrés Muñoz, Nancy Thomas, Annita Vapsi, Daciel Borrajo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06864
• Pdf URL: https://arxiv.org/pdf/2411.06864
• Copy Paste: [[2411.06864]] Veri-Car: Towards Open-world Vehicle Information Retrieval(https://arxiv.org/abs/2411.06864)
• Keywords: robust
• Abstract: Many industrial and service sectors require tools to extract vehicle characteristics from images. This is a complex task not only by the variety of noise, and large number of classes, but also by the constant introduction of new vehicle models to the market. In this paper, we present Veri-Car, an information retrieval integrated approach designed to help on this task. It leverages supervised learning techniques to accurately identify the make, type, model, year, color, and license plate of cars. The approach also addresses the challenge of handling open-world problems, where new car models and variations frequently emerge, by employing a sophisticated combination of pre-trained models, and a hierarchical multi-similarity loss. Veri-Car demonstrates robust performance, achieving high precision and accuracy in classifying both seen and unseen data. Additionally, it integrates an ensemble license plate detection, and an OCR model to extract license plate numbers with impressive accuracy.

Title: Subgraph Retrieval Enhanced by Graph-Text Alignment for Commonsense Question Answering

• Authors: Boci Peng, Yongchao Liu, Xiaohe Bo, Sheng Tian, Baokun Wang, Chuntao Hong, Yan Zhang
• Subjects: cs.LG, cs.AI, cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2411.06866
• Pdf URL: https://arxiv.org/pdf/2411.06866
• Copy Paste: [[2411.06866]] Subgraph Retrieval Enhanced by Graph-Text Alignment for Commonsense Question Answering(https://arxiv.org/abs/2411.06866)
• Keywords: robust
• Abstract: Commonsense question answering is a crucial task that requires machines to employ reasoning according to commonsense. Previous studies predominantly employ an extracting-and-modeling paradigm to harness the information in KG, which first extracts relevant subgraphs based on pre-defined rules and then proceeds to design various strategies aiming to improve the representations and fusion of the extracted structural knowledge. Despite their effectiveness, there are still two challenges. On one hand, subgraphs extracted by rule-based methods may have the potential to overlook critical nodes and result in uncontrollable subgraph size. On the other hand, the misalignment between graph and text modalities undermines the effectiveness of knowledge fusion, ultimately impacting the task performance. To deal with the problems above, we propose a novel framework: \textbf{S}ubgraph R\textbf{E}trieval Enhanced by Gra\textbf{P}h-\textbf{T}ext \textbf{A}lignment, named \textbf{SEPTA}. Firstly, we transform the knowledge graph into a database of subgraph vectors and propose a BFS-style subgraph sampling strategy to avoid information loss, leveraging the analogy between BFS and the message-passing mechanism. In addition, we propose a bidirectional contrastive learning approach for graph-text alignment, which effectively enhances both subgraph retrieval and knowledge fusion. Finally, all the retrieved information is combined for reasoning in the prediction module. Extensive experiments on five datasets demonstrate the effectiveness and robustness of our framework.

Title: CapeLLM: Support-Free Category-Agnostic Pose Estimation with Multimodal Large Language Models

• Authors: Junho Kim, Hyungjin Chung, Byung-Hoon Kim
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06869
• Pdf URL: https://arxiv.org/pdf/2411.06869
• Copy Paste: [[2411.06869]] CapeLLM: Support-Free Category-Agnostic Pose Estimation with Multimodal Large Language Models(https://arxiv.org/abs/2411.06869)
• Keywords: robust, large language model
• Abstract: Category-agnostic pose estimation (CAPE) has traditionally relied on support images with annotated keypoints, a process that is often cumbersome and may fail to fully capture the necessary correspondences across diverse object categories. Recent efforts have begun exploring the use of text-based queries, where the need for support keypoints is eliminated. However, the optimal use of textual descriptions for keypoints remains an underexplored area. In this work, we introduce CapeLLM, a novel approach that leverages a text-based multimodal large language model (MLLM) for CAPE. Our method only employs query image and detailed text descriptions as an input to estimate category-agnostic keypoints. We conduct extensive experiments to systematically explore the design space of LLM-based CAPE, investigating factors such as choosing the optimal description for keypoints, neural network architectures, and training strategies. Thanks to the advanced reasoning capabilities of the pre-trained MLLM, CapeLLM demonstrates superior generalization and robust performance. Our approach sets a new state-of-the-art on the MP-100 benchmark in the challenging 1-shot setting, marking a significant advancement in the field of category-agnostic pose estimation.

Title: Multi-Modal interpretable automatic video captioning

• Authors: Antoine Hanna-Asaad, Decky Aspandi, Titus Zaharia
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06872
• Pdf URL: https://arxiv.org/pdf/2411.06872
• Copy Paste: [[2411.06872]] Multi-Modal interpretable automatic video captioning(https://arxiv.org/abs/2411.06872)
• Keywords: interpretability
• Abstract: Video captioning aims to describe video contents using natural language format that involves understanding and interpreting scenes, actions and events that occurs simultaneously on the view. Current approaches have mainly concentrated on visual cues, often neglecting the rich information available from other important modality of audio information, including their inter-dependencies. In this work, we introduce a novel video captioning method trained with multi-modal contrastive loss that emphasizes both multi-modal integration and interpretability. Our approach is designed to capture the dependency between these modalities, resulting in more accurate, thus pertinent captions. Furthermore, we highlight the importance of interpretability, employing multiple attention mechanisms that provide explanation into the model's decision-making process. Our experimental results demonstrate that our proposed method performs favorably against the state-of the-art models on commonly used benchmark datasets of MSR-VTT and VATEX.

Title: GraphRPM: Risk Pattern Mining on Industrial Large Attributed Graphs

• Authors: Sheng Tian, Xintan Zeng, Yifei Hu, Baokun Wang, Yongchao Liu, Yue Jin, Changhua Meng, Chuntao Hong, Tianyi Zhang, Weiqiang Wang
• Subjects: cs.LG, cs.AI, cs.DC, cs.SI
• Abstract URL: https://arxiv.org/abs/2411.06878
• Pdf URL: https://arxiv.org/pdf/2411.06878
• Copy Paste: [[2411.06878]] GraphRPM: Risk Pattern Mining on Industrial Large Attributed Graphs(https://arxiv.org/abs/2411.06878)
• Keywords: extraction, interpretability
• Abstract: Graph-based patterns are extensively employed and favored by practitioners within industrial companies due to their capacity to represent the behavioral attributes and topological relationships among users, thereby offering enhanced interpretability in comparison to black-box models commonly utilized for classification and recognition tasks. For instance, within the scenario of transaction risk management, a graph pattern that is characteristic of a particular risk category can be readily employed to discern transactions fraught with risk, delineate networks of criminal activity, or investigate the methodologies employed by fraudsters. Nonetheless, graph data in industrial settings is often characterized by its massive scale, encompassing data sets with millions or even billions of nodes, making the manual extraction of graph patterns not only labor-intensive but also necessitating specialized knowledge in particular domains of risk. Moreover, existing methodologies for mining graph patterns encounter significant obstacles when tasked with analyzing large-scale attributed graphs. In this work, we introduce GraphRPM, an industry-purpose parallel and distributed risk pattern mining framework on large attributed graphs. The framework incorporates a novel edge-involved graph isomorphism network alongside optimized operations for parallel graph computation, which collectively contribute to a considerable reduction in computational complexity and resource expenditure. Moreover, the intelligent filtration of efficacious risky graph patterns is facilitated by the proposed evaluation metrics. Comprehensive experimental evaluations conducted on real-world datasets of varying sizes substantiate the capability of GraphRPM to adeptly address the challenges inherent in mining patterns from large-scale industrial attributed graphs, thereby underscoring its substantial value for industrial deployment.

Title: WassFFed: Wasserstein Fair Federated Learning

• Authors: Zhongxuan Han, Li Zhang, Chaochao Chen, Xiaolin Zheng, Fei Zheng, Yuyuan Li, Jianwei Yin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06881
• Pdf URL: https://arxiv.org/pdf/2411.06881
• Copy Paste: [[2411.06881]] WassFFed: Wasserstein Fair Federated Learning(https://arxiv.org/abs/2411.06881)
• Keywords: federate, fair
• Abstract: Federated Learning (FL) employs a training approach to address scenarios where users' data cannot be shared across clients. Achieving fairness in FL is imperative since training data in FL is inherently geographically distributed among diverse user groups. Existing research on fairness predominantly assumes access to the entire training data, making direct transfer to FL challenging. However, the limited existing research on fairness in FL does not effectively address two key challenges, i.e., (CH1) Current methods fail to deal with the inconsistency between fair optimization results obtained with surrogate functions and fair classification results. (CH2) Directly aggregating local fair models does not always yield a globally fair model due to non Identical and Independent data Distributions (non-IID) among clients. To address these challenges, we propose a Wasserstein Fair Federated Learning framework, namely WassFFed. To tackle CH1, we ensure that the outputs of local models, rather than the loss calculated with surrogate functions or classification results with a threshold, remain independent of various user groups. To resolve CH2, we employ a Wasserstein barycenter calculation of all local models' outputs for each user group, bringing local model outputs closer to the global output distribution to ensure consistency between the global model and local models. We conduct extensive experiments on three real-world datasets, demonstrating that WassFFed outperforms existing approaches in striking a balance between accuracy and fairness.

Title: SPARTAN: A Sparse Transformer Learning Local Causation

• Authors: Anson Lei, Ingmar Posner, Bernhard Schölkopf
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.06890
• Pdf URL: https://arxiv.org/pdf/2411.06890
• Copy Paste: [[2411.06890]] SPARTAN: A Sparse Transformer Learning Local Causation(https://arxiv.org/abs/2411.06890)
• Keywords: robust, transformer
• Abstract: Causal structures play a central role in world models that flexibly adapt to changes in the environment. While recent works motivate the benefits of discovering local causal graphs for dynamics modelling, in this work we demonstrate that accurately capturing these relationships in complex settings remains challenging for the current state-of-the-art. To remedy this shortcoming, we postulate that sparsity is a critical ingredient for the discovery of such local causal structures. To this end we present the SPARse TrANsformer World model (SPARTAN), a Transformer-based world model that learns local causal structures between entities in a scene. By applying sparsity regularisation on the attention pattern between object-factored tokens, SPARTAN identifies sparse local causal models that accurately predict future object states. Furthermore, we extend our model to capture sparse interventions with unknown targets on the dynamics of the environment. This results in a highly interpretable world model that can efficiently adapt to changes. Empirically, we evaluate SPARTAN against the current state-of-the-art in object-centric world models on observation-based environments and demonstrate that our model can learn accurate local causal graphs and achieve significantly improved few-shot adaptation to changes in the dynamics of the environment as well as robustness against removing irrelevant distractors.

Title: Multi-scale Frequency Enhancement Network for Blind Image Deblurring

• Authors: Yawen Xiang, Heng Zhou, Chengyang Li, Zhongbo Li, Yongqiang Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06893
• Pdf URL: https://arxiv.org/pdf/2411.06893
• Copy Paste: [[2411.06893]] Multi-scale Frequency Enhancement Network for Blind Image Deblurring(https://arxiv.org/abs/2411.06893)
• Keywords: robust, extraction
• Abstract: Image deblurring is an essential image preprocessing technique, aiming to recover clear and detailed images form blurry ones. However, existing algorithms often fail to effectively integrate multi-scale feature extraction with frequency enhancement, limiting their ability to reconstruct fine textures. Additionally, non-uniform blur in images also restricts the effectiveness of image restoration. To address these issues, we propose a multi-scale frequency enhancement network (MFENet) for blind image deblurring. To capture the multi-scale spatial and channel information of blurred images, we introduce a multi-scale feature extraction module (MS-FE) based on depthwise separable convolutions, which provides rich target features for deblurring. We propose a frequency enhanced blur perception module (FEBP) that employs wavelet transforms to extract high-frequency details and utilizes multi-strip pooling to perceive non-uniform blur, combining multi-scale information with frequency enhancement to improve the restoration of image texture details. Experimental results on the GoPro and HIDE datasets demonstrate that the proposed method achieves superior deblurring performance in both visual quality and objective evaluation metrics. Furthermore, in downstream object detection tasks, the proposed blind image deblurring algorithm significantly improves detection accuracy, further validating its effectiveness androbustness in the field of image deblurring.

Title: LongSafetyBench: Long-Context LLMs Struggle with Safety Issues

• Authors: Mianqiu Huang, Xiaoran Liu, Shaojun Zhou, Mozhi Zhang, Chenkun Tan, Pengyu Wang, Qipeng Guo, Zhe Xu, Linyang Li, Zhikai Lei, Linlin Li, Qun Liu, Yaqian Zhou, Xipeng Qiu, Xuanjing Huang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06899
• Pdf URL: https://arxiv.org/pdf/2411.06899
• Copy Paste: [[2411.06899]] LongSafetyBench: Long-Context LLMs Struggle with Safety Issues(https://arxiv.org/abs/2411.06899)
• Keywords: large language model
• Abstract: With the development of large language models (LLMs), the sequence length of these models continues to increase, drawing significant attention to long-context language models. However, the evaluation of these models has been primarily limited to their capabilities, with a lack of research focusing on their safety. Existing work, such as ManyShotJailbreak, has to some extent demonstrated that long-context language models can exhibit safety concerns. However, the methods used are limited and lack comprehensiveness. In response, we introduce \textbf{LongSafetyBench}, the first benchmark designed to objectively and comprehensively evaluate the safety of long-context models. LongSafetyBench consists of 10 task categories, with an average length of 41,889 words. After testing eight long-context language models on LongSafetyBench, we found that existing models generally exhibit insufficient safety capabilities. The proportion of safe responses from most mainstream long-context LLMs is below 50\%. Moreover, models' safety performance in long-context scenarios does not always align with that in short-context scenarios. Further investigation revealed that long-context models tend to overlook harmful content within lengthy texts. We also proposed a simple yet effective solution, allowing open-source models to achieve performance comparable to that of top-tier closed-source models. We believe that LongSafetyBench can serve as a valuable benchmark for evaluating the safety capabilities of long-context language models. We hope that our work will encourage the broader community to pay attention to the safety of long-context models and contribute to the development of solutions to improve the safety of long-context LLMs.

Title: EVQAScore: Efficient Video Question Answering Data Evaluation

• Authors: Hao Liang, Zirong Chen, Wentao Zhang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06908
• Pdf URL: https://arxiv.org/pdf/2411.06908
• Copy Paste: [[2411.06908]] EVQAScore: Efficient Video Question Answering Data Evaluation(https://arxiv.org/abs/2411.06908)
• Keywords: robust, extraction, large language model
• Abstract: Video question-answering (QA) is a core task in video understanding. Evaluating the quality of video QA and video caption data quality for training video large language models (VideoLLMs) is an essential challenge. Although various methods have been proposed for assessing video caption quality, there remains a lack of dedicated evaluation methods for Video QA. To address this gap, we introduce EVQAScore, a reference-free method that leverages keyword extraction to assess both video caption and video QA data quality. Additionally, we incorporate frame sampling and rescaling techniques to enhance the efficiency and robustness of our evaluation, this enables our score to evaluate the quality of extremely long videos. Our approach achieves state-of-the-art (SOTA) performance (32.8 for Kendall correlation and 42.3 for Spearman correlation, 4.7 and 5.9 higher than the previous method PAC-S++) on the VATEX-EVAL benchmark for video caption evaluation. Furthermore, by using EVQAScore for data selection, we achieved SOTA results with only 12.5\% of the original data volume, outperforming the previous SOTA method PAC-S and 100\% of data.

Title: Gaussian Process Emulators for Few-Shot Segmentation in Cardiac MRI

• Authors: Bruno Viti, Franz Thaler, Kathrin Lisa Kapper, Martin Urschler, Martin Holler, Elias Karabelas
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.06911
• Pdf URL: https://arxiv.org/pdf/2411.06911
• Copy Paste: [[2411.06911]] Gaussian Process Emulators for Few-Shot Segmentation in Cardiac MRI(https://arxiv.org/abs/2411.06911)
• Keywords: segmentation
• Abstract: Segmentation of cardiac magnetic resonance images (MRI) is crucial for the analysis and assessment of cardiac function, helping to diagnose and treat various cardiovascular diseases. Most recent techniques rely on deep learning and usually require an extensive amount of labeled data. To overcome this problem, few-shot learning has the capability of reducing data dependency on labeled data. In this work, we introduce a new method that merges few-shot learning with a U-Net architecture and Gaussian Process Emulators (GPEs), enhancing data integration from a support set for improved performance. GPEs are trained to learn the relation between the support images and the corresponding masks in latent space, facilitating the segmentation of unseen query images given only a small labeled support set at inference. We test our model with the M&Ms-2 public dataset to assess its ability to segment the heart in cardiac magnetic resonance imaging from different orientations, and compare it with state-of-the-art unsupervised and few-shot methods. Our architecture shows higher DICE coefficients compared to these methods, especially in the more challenging setups where the size of the support set is considerably small.

Title: Cancer-Answer: Empowering Cancer Care with Advanced Large Language Models

• Authors: Aniket Deroy, Subhankar Maity
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.06946
• Pdf URL: https://arxiv.org/pdf/2411.06946
• Copy Paste: [[2411.06946]] Cancer-Answer: Empowering Cancer Care with Advanced Large Language Models(https://arxiv.org/abs/2411.06946)
• Keywords: large language model
• Abstract: Gastrointestinal (GI) tract cancers account for a substantial portion of the global cancer burden, where early diagnosis is critical for improved management and patient outcomes. The complex aetiologies and overlapping symptoms across GI cancers often delay diagnosis, leading to suboptimal treatment strategies. Cancer-related queries are crucial for timely diagnosis, treatment, and patient education, as access to accurate, comprehensive information can significantly influence outcomes. However, the complexity of cancer as a disease, combined with the vast amount of available data, makes it difficult for clinicians and patients to quickly find precise answers. To address these challenges, we leverage large language models (LLMs) such as GPT-3.5 Turbo to generate accurate, contextually relevant responses to cancer-related queries. Pre-trained with medical data, these models provide timely, actionable insights that support informed decision-making in cancer diagnosis and care, ultimately improving patient outcomes. We calculate two metrics: A1 (which represents the fraction of entities present in the model-generated answer compared to the gold standard) and A2 (which represents the linguistic correctness and meaningfulness of the model-generated answer with respect to the gold standard), achieving maximum values of 0.546 and 0.881, respectively.

Title: Sniff AI: Is My 'Spicy' Your 'Spicy'? Exploring LLM's Perceptual Alignment with Human Smell Experiences

• Authors: Shu Zhong, Zetao Zhou, Christopher Dawes, Giada Brianz, Marianna Obrist
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.06950
• Pdf URL: https://arxiv.org/pdf/2411.06950
• Copy Paste: [[2411.06950]] Sniff AI: Is My 'Spicy' Your 'Spicy'? Exploring LLM's Perceptual Alignment with Human Smell Experiences(https://arxiv.org/abs/2411.06950)
• Keywords: large language model
• Abstract: Aligning AI with human intent is important, yet perceptual alignment-how AI interprets what we see, hear, or smell-remains underexplored. This work focuses on olfaction, human smell experiences. We conducted a user study with 40 participants to investigate how well AI can interpret human descriptions of scents. Participants performed "sniff and describe" interactive tasks, with our designed AI system attempting to guess what scent the participants were experiencing based on their descriptions. These tasks evaluated the Large Language Model's (LLMs) contextual understanding and representation of scent relationships within its internal states - high-dimensional embedding space. Both quantitative and qualitative methods were used to evaluate the AI system's performance. Results indicated limited perceptual alignment, with biases towards certain scents, like lemon and peppermint, and continued failing to identify others, like rosemary. We discuss these findings in light of human-AI alignment advancements, highlighting the limitations and opportunities for enhancing HCI systems with multisensory experience integration.

Title: ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis

• Authors: Zanlin Ni, Yulin Wang, Renping Zhou, Yizeng Han, Jiayi Guo, Zhiyuan Liu, Yuan Yao, Gao Huang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.06959
• Pdf URL: https://arxiv.org/pdf/2411.06959
• Copy Paste: [[2411.06959]] ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis(https://arxiv.org/abs/2411.06959)
• Keywords: transformer
• Abstract: Recently, token-based generation have demonstrated their effectiveness in image synthesis. As a representative example, non-autoregressive Transformers (NATs) can generate decent-quality images in a few steps. NATs perform generation in a progressive manner, where the latent tokens of a resulting image are incrementally revealed. At each step, the unrevealed image regions are padded with mask tokens and inferred by NAT. In this paper, we delve into the mechanisms behind the effectiveness of NATs and uncover two important patterns that naturally emerge from NATs: Spatially (within a step), although mask and visible tokens are processed uniformly by NATs, the interactions between them are highly asymmetric. In specific, mask tokens mainly gather information for decoding, while visible tokens tend to primarily provide information, and their deep representations can be built only upon themselves. Temporally (across steps), the interactions between adjacent generation steps mostly concentrate on updating the representations of a few critical tokens, while the computation for the majority of tokens is generally repetitive. Driven by these findings, we propose EfficientNAT (ENAT), a NAT model that explicitly encourages these critical interactions inherent in NATs. At the spatial level, we disentangle the computations of visible and mask tokens by encoding visible tokens independently, while decoding mask tokens conditioned on the fully encoded visible tokens. At the temporal level, we prioritize the computation of the critical tokens at each step, while maximally reusing previously computed token representations to supplement necessary information. ENAT improves the performance of NATs notably with significantly reduced computational cost. Experiments on ImageNet-256, ImageNet-512 and MS-COCO validate the effectiveness of ENAT. Code is available at this https URL.

Title: Robust Fine-tuning of Zero-shot Models via Variance Reduction

• Authors: Beier Zhu, Jiequan Cui, Hanwang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06966
• Pdf URL: https://arxiv.org/pdf/2411.06966
• Copy Paste: [[2411.06966]] Robust Fine-tuning of Zero-shot Models via Variance Reduction(https://arxiv.org/abs/2411.06966)
• Keywords: robust
• Abstract: When fine-tuning zero-shot models like CLIP, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD). Recently, ensemble-based models (ESM) have been shown to offer significant robustness improvement, while preserving high ID accuracy. However, our study finds that ESMs do not solve the ID-OOD trade-offs: they achieve peak performance for ID and OOD accuracy at different mixing coefficients. When optimized for OOD accuracy, the ensemble model exhibits a noticeable decline in ID accuracy, and vice versa. In contrast, we propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs. Specifically, we construct a Zero-Shot Failure (ZSF) set containing training samples incorrectly predicted by the zero-shot model. For each test sample, we calculate its distance to the ZSF set and assign a higher weight to the fine-tuned model in the ensemble if the distance is small. We term our method Variance Reduction Fine-tuning (VRF), as it effectively reduces the variance in ensemble predictions, thereby decreasing residual error. On ImageNet and five derived distribution shifts, our VRF further improves the OOD accuracy by 1.5 - 2.0 pp over the ensemble baselines while maintaining or increasing ID accuracy. VRF achieves similar large robustness gains (0.9 - 3.1 pp) on other distribution shifts benchmarks. Codes are available in this https URL.

Title: MapSAM: Adapting Segment Anything Model for Automated Feature Detection in Historical Maps

• Authors: Xue Xia, Daiwei Zhang, Wenxuan Song, Wei Huang, Lorenz Hurni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06971
• Pdf URL: https://arxiv.org/pdf/2411.06971
• Copy Paste: [[2411.06971]] MapSAM: Adapting Segment Anything Model for Automated Feature Detection in Historical Maps(https://arxiv.org/abs/2411.06971)
• Keywords: segmentation
• Abstract: Automated feature detection in historical maps can significantly accelerate the reconstruction of the geospatial past. However, this process is often constrained by the time-consuming task of manually digitizing sufficient high-quality training data. The emergence of visual foundation models, such as the Segment Anything Model (SAM), offers a promising solution due to their remarkable generalization capabilities and rapid adaptation to new data distributions. Despite this, directly applying SAM in a zero-shot manner to historical map segmentation poses significant challenges, including poor recognition of certain geospatial features and a reliance on input prompts, which limits its ability to be fully automated. To address these challenges, we introduce MapSAM, a parameter-efficient fine-tuning strategy that adapts SAM into a prompt-free and versatile solution for various downstream historical map segmentation tasks. Specifically, we employ Weight-Decomposed Low-Rank Adaptation (DoRA) to integrate domain-specific knowledge into the image encoder. Additionally, we develop an automatic prompt generation process, eliminating the need for manual input. We further enhance the positional prompt in SAM, transforming it into a higher-level positional-semantic prompt, and modify the cross-attention mechanism in the mask decoder with masked attention for more effective feature aggregation. The proposed MapSAM framework demonstrates promising performance across two distinct historical map segmentation tasks: one focused on linear features and the other on areal features. Experimental results show that it adapts well to various features, even when fine-tuned with extremely limited data (e.g. 10 shots).

Title: SIESEF-FusionNet: Spatial Inter-correlation Enhancement and Spatially-Embedded Feature Fusion Network for LiDAR Point Cloud Semantic Segmentation

• Authors: Jiale Chen, Fei Xia, Jianliang Mao, Haoping Wang, Chuanlin Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.06991
• Pdf URL: https://arxiv.org/pdf/2411.06991
• Copy Paste: [[2411.06991]] SIESEF-FusionNet: Spatial Inter-correlation Enhancement and Spatially-Embedded Feature Fusion Network for LiDAR Point Cloud Semantic Segmentation(https://arxiv.org/abs/2411.06991)
• Keywords: segmentation
• Abstract: The ambiguity at the boundaries of different semantic classes in point cloud semantic segmentation often leads to incorrect decisions in intelligent perception systems, such as autonomous driving. Hence, accurate delineation of the boundaries is crucial for improving safety in autonomous driving. A novel spatial inter-correlation enhancement and spatially-embedded feature fusion network (SIESEF-FusionNet) is proposed in this paper, enhancing spatial inter-correlation by combining inverse distance weighting and angular compensation to extract more beneficial spatial information without causing redundancy. Meanwhile, a new spatial adaptive pooling module is also designed, embedding enhanced spatial information into semantic features for strengthening the context-awareness of semantic features. Experimental results demonstrate that 83.7% mIoU and 97.8% OA are achieved by SIESEF-FusionNet on the Toronto3D dataset, with performance superior to other baseline methods. A value of 61.1% mIoU is reached on the semanticKITTI dataset, where a marked improvement in segmentation performance is observed. In addition, the effectiveness and plug-and-play capability of the proposed modules are further verified through ablation studies.

Title: Hierarchical Conditional Tabular GAN for Multi-Tabular Synthetic Data Generation

• Authors: Wilhelm Ågren, Victorio Úbeda Sosa
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2411.07009
• Pdf URL: https://arxiv.org/pdf/2411.07009
• Copy Paste: [[2411.07009]] Hierarchical Conditional Tabular GAN for Multi-Tabular Synthetic Data Generation(https://arxiv.org/abs/2411.07009)
• Keywords: privacy, fair
• Abstract: The generation of synthetic data is a state-of-the-art approach to leverage when access to real data is limited or privacy regulations limit the usability of sensitive data. A fair amount of research has been conducted on synthetic data generation for single-tabular datasets, but only a limited amount of research has been conducted on multi-tabular datasets with complex table relationships. In this paper we propose the algorithm HCTGAN to synthesize multi-tabular data from complex multi-tabular datasets. We compare our results to the probabilistic model HMA1. Our findings show that our proposed algorithm can more efficiently sample large amounts of synthetic data for deep and complex multi-tabular datasets, whilst achieving adequate data quality and always guaranteeing referential integrity. We conclude that the HCTGAN algorithm is suitable for generating large amounts of synthetic data efficiently for deep multi-tabular datasets with complex relationships. We additionally suggest that the HMA1 model should be used on smaller datasets when emphasis is on data quality.

Title: A neural-network based anomaly detection system and a safety protocol to protect vehicular network

• Authors: Marco Franceschini
• Subjects: cs.LG, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.07013
• Pdf URL: https://arxiv.org/pdf/2411.07013
• Copy Paste: [[2411.07013]] A neural-network based anomaly detection system and a safety protocol to protect vehicular network(https://arxiv.org/abs/2411.07013)
• Keywords: secure, protect, defense
• Abstract: This thesis addresses the use of Cooperative Intelligent Transport Systems (CITS) to improve road safety and efficiency by enabling vehicle-to-vehicle communication, highlighting the importance of secure and accurate data exchange. To ensure safety, the thesis proposes a Machine Learning-based Misbehavior Detection System (MDS) using Long Short-Term Memory (LSTM) networks to detect and mitigate incorrect or misleading messages within vehicular networks. Trained offline on the VeReMi dataset, the detection model is tested in real-time within a platooning scenario, demonstrating that it can prevent nearly all accidents caused by misbehavior by triggering a defense protocol that dissolves the platoon if anomalies are detected. The results show that while the system can accurately detect general misbehavior, it struggles to label specific types due to varying traffic conditions, implying the difficulty of creating a universally adaptive protocol. However, the thesis suggests that with more data and further refinement, this MDS could be implemented in real-world CITS, enhancing driving safety by mitigating risks from misbehavior in cooperative driving networks.

Title: ProP: Efficient Backdoor Detection via Propagation Perturbation for Overparametrized Models

• Authors: Tao Ren, Qiongxiu Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.07036
• Pdf URL: https://arxiv.org/pdf/2411.07036
• Copy Paste: [[2411.07036]] ProP: Efficient Backdoor Detection via Propagation Perturbation for Overparametrized Models(https://arxiv.org/abs/2411.07036)
• Keywords: security, attack, robust
• Abstract: Backdoor attacks pose significant challenges to the security of machine learning models, particularly for overparameterized models like deep neural networks. In this paper, we propose ProP (Propagation Perturbation), a novel and scalable backdoor detection method that leverages statistical output distributions to identify backdoored models and their target classes without relying on exhausive optimization strategies. ProP introduces a new metric, the benign score, to quantify output distributions and effectively distinguish between benign and backdoored models. Unlike existing approaches, ProP operates with minimal assumptions, requiring no prior knowledge of triggers or malicious samples, making it highly applicable to real-world scenarios. Extensive experimental validation across multiple popular backdoor attacks demonstrates that ProP achieves high detection accuracy and computational efficiency, outperforming existing methods. These results highlight ProP's potential as a robust and practical solution for backdoor detection.

Title: LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios

• Authors: Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Xiangju Lu, Junmin Zhu, Wei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07037
• Pdf URL: https://arxiv.org/pdf/2411.07037
• Copy Paste: [[2411.07037]] LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios(https://arxiv.org/abs/2411.07037)
• Keywords: robust, large language model
• Abstract: As Large Language Models (LLMs) continue to advance in natural language processing (NLP), their ability to stably follow instructions in long-context inputs has become crucial for real-world applications. While existing benchmarks assess various LLM capabilities, they rarely focus on instruction-following in long-context scenarios or stability on different inputs. In response, we introduce the Long-context Instruction-Following Benchmark (LIFBench), a scalable dataset designed to evaluate LLMs' instruction-following capabilities and stability across long contexts. LIFBench comprises three long-context scenarios and eleven diverse tasks, supported by 2,766 instructions generated through an automated expansion method across three dimensions: length, expression, and variables. For evaluation, we propose LIFEval, a rubric-based assessment framework that provides precise, automated scoring of complex LLM responses without relying on LLM-assisted evaluations or human judgments. This approach facilitates a comprehensive analysis of model performance and stability across various perspectives. We conduct extensive experiments on 20 notable LLMs across six length intervals, analyzing their instruction-following capabilities and stability. Our work contributes LIFBench and LIFEval as robust tools for assessing LLM performance in complex, long-context settings, providing insights that can inform future LLM development.

Title: On Active Privacy Auditing in Supervised Fine-tuning for White-Box Language Models

• Authors: Qian Sun, Hanpeng Wu, Xi Sheryl Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07070
• Pdf URL: https://arxiv.org/pdf/2411.07070
• Copy Paste: [[2411.07070]] On Active Privacy Auditing in Supervised Fine-tuning for White-Box Language Models(https://arxiv.org/abs/2411.07070)
• Keywords: privacy, attack, membership infer
• Abstract: The pretraining and fine-tuning approach has become the leading technique for various NLP applications. However, recent studies reveal that fine-tuning data, due to their sensitive nature, domain-specific characteristics, and identifiability, pose significant privacy concerns. To help develop more privacy-resilient fine-tuning models, we introduce a novel active privacy auditing framework, dubbed Parsing, designed to identify and quantify privacy leakage risks during the supervised fine-tuning (SFT) of language models (LMs). The framework leverages improved white-box membership inference attacks (MIAs) as the core technology, utilizing novel learning objectives and a two-stage pipeline to monitor the privacy of the LMs' fine-tuning process, maximizing the exposure of privacy risks. Additionally, we have improved the effectiveness of MIAs on large LMs including GPT-2, Llama2, and certain variants of them. Our research aims to provide the SFT community of LMs with a reliable, ready-to-use privacy auditing tool, and to offer valuable insights into safeguarding privacy during the fine-tuning process. Experimental results confirm the framework's efficiency across various models and tasks, emphasizing notable privacy concerns in the fine-tuning process. Project code available for this https URL.

Title: Universal Response and Emergence of Induction in LLMs

• Authors: Niclas Luick
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07071
• Pdf URL: https://arxiv.org/pdf/2411.07071
• Copy Paste: [[2411.07071]] Universal Response and Emergence of Induction in LLMs(https://arxiv.org/abs/2411.07071)
• Keywords: robust
• Abstract: While induction is considered a key mechanism for in-context learning in LLMs, understanding its precise circuit decomposition beyond toy models remains elusive. Here, we study the emergence of induction behavior within LLMs by probing their response to weak single-token perturbations of the residual stream. We find that LLMs exhibit a robust, universal regime in which their response remains scale-invariant under changes in perturbation strength, thereby allowing us to quantify the build-up of token correlations throughout the model. By applying our method, we observe signatures of induction behavior within the residual stream of Gemma-2-2B, Llama-3.2-3B, and GPT-2-XL. Across all models, we find that these induction signatures gradually emerge within intermediate layers and identify the relevant model sections composing this behavior. Our results provide insights into the collective interplay of components within LLMs and serve as a benchmark for large-scale circuit analysis.

Title: Increasing Rosacea Awareness Among Population Using Deep Learning and Statistical Approaches

• Authors: Chengyu Yang, Chengjun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07074
• Pdf URL: https://arxiv.org/pdf/2411.07074
• Copy Paste: [[2411.07074]] Increasing Rosacea Awareness Among Population Using Deep Learning and Statistical Approaches(https://arxiv.org/abs/2411.07074)
• Keywords: explainability
• Abstract: Approximately 16 million Americans suffer from rosacea according to the National Rosacea Society. To increase rosacea awareness, automatic rosacea detection methods using deep learning and explainable statistical approaches are presented in this paper. The deep learning method applies the ResNet-18 for rosacea detection, and the statistical approaches utilize the means of the two classes, namely, the rosacea class vs. the normal class, and the principal component analysis to extract features from the facial images for automatic rosacea detection. The contributions of the proposed methods are three-fold. First, the proposed methods are able to automatically distinguish patients who are suffering from rosacea from people who are clean of this disease. Second, the statistical approaches address the explainability issue that allows doctors and patients to understand and trust the results. And finally, the proposed methods will not only help increase rosacea awareness in the general population but also help remind the patients who suffer from this disease of possible early treatment since rosacea is more treatable at its early stages. The code and data are available at this https URL.

Title: Transformer verbatim in-context retrieval across time and scale

• Authors: Kristijan Armeni, Marko Pranjić, Senja Pollak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07075
• Pdf URL: https://arxiv.org/pdf/2411.07075
• Copy Paste: [[2411.07075]] Transformer verbatim in-context retrieval across time and scale(https://arxiv.org/abs/2411.07075)
• Keywords: transformer
• Abstract: To predict upcoming text, language models must in some cases retrieve in-context information verbatim. In this report, we investigated how the ability of language models to retrieve arbitrary in-context nouns developed during training (across time) and as language models trained on the same dataset increase in size (across scale). We then asked whether learning of in-context retrieval correlates with learning of more challenging zero-shot benchmarks. Furthermore, inspired by semantic effects in human short-term memory, we evaluated the retrieval with respect to a major semantic component of target nouns, namely whether they denote a concrete or abstract entity, as rated by humans. We show that verbatim in-context retrieval developed in a sudden transition early in the training process, after about 1% of the training tokens. This was observed across model sizes (from 14M and up to 12B parameters), and the transition occurred slightly later for the two smallest models. We further found that the development of verbatim in-context retrieval is positively correlated with the learning of zero-shot benchmarks. Around the transition point, all models showed the advantage of retrieving concrete nouns as opposed to abstract nouns. In all but two smallest models, the advantage dissipated away toward the end of training.

Title: StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification

• Authors: Yichen He, Yuan Lin, Jianchao Wu, Hanchong Zhang, Yuchen Zhang, Ruicheng Le
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07076
• Pdf URL: https://arxiv.org/pdf/2411.07076
• Copy Paste: [[2411.07076]] StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification(https://arxiv.org/abs/2411.07076)
• Keywords: large language model
• Abstract: Existing large vision-language models (LVLMs) are largely limited to processing short, seconds-long videos and struggle with generating coherent descriptions for extended video spanning minutes or more. Long video description introduces new challenges, such as plot-level consistency across descriptions. To address these, we figure out audio-visual character identification, matching character names to each dialogue, as a key factor. We propose StoryTeller, a system for generating dense descriptions of long videos, incorporating both low-level visual concepts and high-level plot information. StoryTeller uses a multimodal large language model that integrates visual, audio, and text modalities to perform audio-visual character identification on minute-long video clips. The results are then fed into a LVLM to enhance consistency of video description. We validate our approach on movie description tasks and introduce MovieStory101, a dataset with dense descriptions for three-minute movie clips. To evaluate long video descriptions, we create MovieQA, a large set of multiple-choice questions for the MovieStory101 test set. We assess descriptions by inputting them into GPT-4 to answer these questions, using accuracy as an automatic evaluation metric. Experiments show that StoryTeller outperforms all open and closed-source baselines on MovieQA, achieving 9.5% higher accuracy than the strongest baseline, Gemini-1.5-pro, and demonstrating a +15.56% advantage in human side-by-side evaluations. Additionally, incorporating audio-visual character identification from StoryTeller improves the performance of all video description models, with Gemini-1.5-pro and GPT-4o showing relative improvement of 5.5% and 13.0%, respectively, in accuracy on MovieQA.

Title: Differentially-Private Collaborative Online Personalized Mean Estimation

• Authors: Yauhen Yakimenka, Chung-Wei Weng, Hsuan-Yin Lin, Eirik Rosnes, Jörg Kliewer
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2411.07094
• Pdf URL: https://arxiv.org/pdf/2411.07094
• Copy Paste: [[2411.07094]] Differentially-Private Collaborative Online Personalized Mean Estimation(https://arxiv.org/abs/2411.07094)
• Keywords: privacy
• Abstract: We consider the problem of collaborative personalized mean estimation under a privacy constraint in an environment of several agents continuously receiving data according to arbitrary unknown agent-specific distributions. In particular, we provide a method based on hypothesis testing coupled with differential privacy and data variance estimation. Two privacy mechanisms and two data variance estimation schemes are proposed, and we provide a theoretical convergence analysis of the proposed algorithm for any bounded unknown distributions on the agents' data, showing that collaboration provides faster convergence than a fully local approach where agents do not share data. Moreover, we provide analytical performance curves for the case with an oracle class estimator, i.e., the class structure of the agents, where agents receiving data from distributions with the same mean are considered to be in the same class, is known. The theoretical faster-than-local convergence guarantee is backed up by extensive numerical results showing that for a considered scenario the proposed approach indeed converges much faster than a fully local approach, and performs comparably to ideal performance where all data is public. This illustrates the benefit of private collaboration in an online setting.

Title: Extreme Rotation Estimation in the Wild

• Authors: Hana Bezalel, Dotan Ankri, Ruojin Cai, Hadar Averbuch-Elor
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07096
• Pdf URL: https://arxiv.org/pdf/2411.07096
• Copy Paste: [[2411.07096]] Extreme Rotation Estimation in the Wild(https://arxiv.org/abs/2411.07096)
• Keywords: transformer
• Abstract: We present a technique and benchmark dataset for estimating the relative 3D orientation between a pair of Internet images captured in an extreme setting, where the images have limited or non-overlapping field of views. Prior work targeting extreme rotation estimation assume constrained 3D environments and emulate perspective images by cropping regions from panoramic views. However, real images captured in the wild are highly diverse, exhibiting variation in both appearance and camera intrinsics. In this work, we propose a Transformer-based method for estimating relative rotations in extreme real-world settings, and contribute the ExtremeLandmarkPairs dataset, assembled from scene-level Internet photo collections. Our evaluation demonstrates that our approach succeeds in estimating the relative rotations in a wide variety of extremeview Internet image pairs, outperforming various baselines, including dedicated rotation estimation techniques and contemporary 3D reconstruction methods.

Title: Arctique: An artificial histopathological dataset unifying realism and controllability for uncertainty quantification

• Authors: Jannik Franzen, Claudia Winklmayr, Vanessa E. Guarino, Christoph Karg, Xiaoyan Yu, Nora Koreuber, Jan P. Albrecht, Philip Bischoff, Dagmar Kainmueller
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07097
• Pdf URL: https://arxiv.org/pdf/2411.07097
• Copy Paste: [[2411.07097]] Arctique: An artificial histopathological dataset unifying realism and controllability for uncertainty quantification(https://arxiv.org/abs/2411.07097)
• Keywords: segmentation
• Abstract: Uncertainty Quantification (UQ) is crucial for reliable image segmentation. Yet, while the field sees continual development of novel methods, a lack of agreed-upon benchmarks limits their systematic comparison and evaluation: Current UQ methods are typically tested either on overly simplistic toy datasets or on complex real-world datasets that do not allow to discern true uncertainty. To unify both controllability and complexity, we introduce Arctique, a procedurally generated dataset modeled after histopathological colon images. We chose histopathological images for two reasons: 1) their complexity in terms of intricate object structures and highly variable appearance, which yields challenging segmentation problems, and 2) their broad prevalence for medical diagnosis and respective relevance of high-quality UQ. To generate Arctique, we established a Blender-based framework for 3D scene creation with intrinsic noise manipulation. Arctique contains 50,000 rendered images with precise masks as well as noisy label simulations. We show that by independently controlling the uncertainty in both images and labels, we can effectively study the performance of several commonly used UQ methods. Hence, Arctique serves as a critical resource for benchmarking and advancing UQ techniques and other methodologies in complex, multi-object environments, bridging the gap between realism and controllability. All code is publicly available, allowing re-creation and controlled manipulations of our shipped images as well as creation and rendering of new scenes.

Title: Training Neural Networks as Recognizers of Formal Languages

• Authors: Alexandra Butoi, Ghazal Khalighinejad, Anej Svete, Josef Valvoda, Ryan Cotterell, Brian DuSell
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07107
• Pdf URL: https://arxiv.org/pdf/2411.07107
• Copy Paste: [[2411.07107]] Training Neural Networks as Recognizers of Formal Languages(https://arxiv.org/abs/2411.07107)
• Keywords: transformer
• Abstract: Characterizing the computational power of neural network architectures in terms of formal language theory remains a crucial line of research, as it describes lower and upper bounds on the reasoning capabilities of modern AI. However, when empirically testing these bounds, existing work often leaves a discrepancy between experiments and the formal claims they are meant to support. The problem is that formal language theory pertains specifically to recognizers: machines that receive a string as input and classify whether it belongs to a language. On the other hand, it is common to instead use proxy tasks that are similar in only an informal sense, such as language modeling or sequence-to-sequence transduction. We correct this mismatch by training and evaluating neural networks directly as binary classifiers of strings, using a general method that can be applied to a wide variety of languages. As part of this, we extend an algorithm recently proposed by Snæbjarnarson et al. (2024) to do length-controlled sampling of strings from regular languages, with much better asymptotic time complexity than previous methods. We provide results on a variety of languages across the Chomsky hierarchy for three neural architectures: a simple RNN, an LSTM, and a causally-masked transformer. We find that the RNN and LSTM often outperform the transformer, and that auxiliary training objectives such as language modeling can help, although no single objective uniformly improves performance across languages and architectures. Our contributions will facilitate theoretically sound empirical testing of language recognition claims in future work. We have released our datasets as a benchmark called FLaRe (Formal Language Recognition), along with our code.

Title: Building a Taiwanese Mandarin Spoken Language Model: A First Attempt

• Authors: Chih-Kai Yang, Yu-Kuan Fu, Chen-An Li, Yi-Cheng Lin, Yu-Xiang Lin, Wei-Chih Chen, Ho Lam Chung, Chun-Yi Kuan, Wei-Ping Huang, Ke-Han Lu, Tzu-Quan Lin, Hsiu-Hsuan Wang, En-Pei Hu, Chan-Jan Hsu, Liang-Hsuan Tseng, I-Hsiang Chiu, Ulin Sanga, Xuanjun Chen, Po-chun Hsu, Shu-wen Yang, Hung-yi Lee
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.07111
• Pdf URL: https://arxiv.org/pdf/2411.07111
• Copy Paste: [[2411.07111]] Building a Taiwanese Mandarin Spoken Language Model: A First Attempt(https://arxiv.org/abs/2411.07111)
• Keywords: transformer, large language model
• Abstract: This technical report presents our initial attempt to build a spoken large language model (LLM) for Taiwanese Mandarin, specifically tailored to enable real-time, speech-to-speech interaction in multi-turn conversations. Our end-to-end model incorporates a decoder-only transformer architecture and aims to achieve seamless interaction while preserving the conversational flow, including full-duplex capabilities allowing simultaneous speaking and listening. The paper also details the training process, including data preparation with synthesized dialogues and adjustments for real-time interaction. We also developed a platform to evaluate conversational fluency and response coherence in multi-turn dialogues. We hope the release of the report can contribute to the future development of spoken LLMs in Taiwanese Mandarin.

Title: TinyML Security: Exploring Vulnerabilities in Resource-Constrained Machine Learning Systems

• Authors: Jacob Huckelberry, Yuke Zhang, Allison Sansone, James Mickens, Peter A. Beerel, Vijay Janapa Reddi
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07114
• Pdf URL: https://arxiv.org/pdf/2411.07114
• Copy Paste: [[2411.07114]] TinyML Security: Exploring Vulnerabilities in Resource-Constrained Machine Learning Systems(https://arxiv.org/abs/2411.07114)
• Keywords: secure, security, protect, defense, attack, robust
• Abstract: Tiny Machine Learning (TinyML) systems, which enable machine learning inference on highly resource-constrained devices, are transforming edge computing but encounter unique security challenges. These devices, restricted by RAM and CPU capabilities two to three orders of magnitude smaller than conventional systems, make traditional software and hardware security solutions impractical. The physical accessibility of these devices exacerbates their susceptibility to side-channel attacks and information leakage. Additionally, TinyML models pose security risks, with weights potentially encoding sensitive data and query interfaces that can be exploited. This paper offers the first thorough survey of TinyML security threats. We present a device taxonomy that differentiates between IoT, EdgeML, and TinyML, highlighting vulnerabilities unique to TinyML. We list various attack vectors, assess their threat levels using the Common Vulnerability Scoring System, and evaluate both existing and possible defenses. Our analysis identifies where traditional security measures are adequate and where solutions tailored to TinyML are essential. Our results underscore the pressing need for specialized security solutions in TinyML to ensure robust and secure edge computing applications. We aim to inform the research community and inspire innovative approaches to protecting this rapidly evolving and critical field.

Title: ConvMixFormer- A Resource-efficient Convolution Mixer for Transformer-based Dynamic Hand Gesture Recognition

• Authors: Mallika Garg, Debashis Ghosh, Pyari Mohan Pradhan
• Subjects: cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07118
• Pdf URL: https://arxiv.org/pdf/2411.07118
• Copy Paste: [[2411.07118]] ConvMixFormer- A Resource-efficient Convolution Mixer for Transformer-based Dynamic Hand Gesture Recognition(https://arxiv.org/abs/2411.07118)
• Keywords: transformer
• Abstract: Transformer models have demonstrated remarkable success in many domains such as natural language processing (NLP) and computer vision. With the growing interest in transformer-based architectures, they are now utilized for gesture recognition. So, we also explore and devise a novel ConvMixFormer architecture for dynamic hand gestures. The transformers use quadratic scaling of the attention features with the sequential data, due to which these models are computationally complex and heavy. We have considered this drawback of the transformer and designed a resource-efficient model that replaces the self-attention in the transformer with the simple convolutional layer-based token mixer. The computational cost and the parameters used for the convolution-based mixer are comparatively less than the quadratic self-attention. Convolution-mixer helps the model capture the local spatial features that self-attention struggles to capture due to their sequential processing nature. Further, an efficient gate mechanism is employed instead of a conventional feed-forward network in the transformer to help the model control the flow of features within different stages of the proposed model. This design uses fewer learnable parameters which is nearly half the vanilla transformer that helps in fast and efficient training. The proposed method is evaluated on NVidia Dynamic Hand Gesture and Briareo datasets and our model has achieved state-of-the-art results on single and multimodal inputs. We have also shown the parameter efficiency of the proposed ConvMixFormer model compared to other methods. The source code is available at this https URL.

Title: Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

• Authors: Yanchen Wang, Adam Turnbull, Tiange Xiang, Yunlong Xu, Sa Zhou, Adnan Masoud, Shekoofeh Azizi, Feng Vankee Lin, Ehsan Adeli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07121
• Pdf URL: https://arxiv.org/pdf/2411.07121
• Copy Paste: [[2411.07121]] Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models(https://arxiv.org/abs/2411.07121)
• Keywords: generative
• Abstract: Neural decoding, the process of understanding how brain activity corresponds to different stimuli, has been a primary objective in cognitive sciences. Over the past three decades, advancements in functional Magnetic Resonance Imaging and machine learning have greatly improved our ability to map visual stimuli to brain activity, especially in the visual cortex. Concurrently, research has expanded into decoding more complex processes like language and memory across the whole brain, utilizing techniques to handle greater variability and improve signal accuracy. We argue that "seeing" involves more than just mapping visual stimuli onto the visual cortex; it engages the entire brain, as various emotions and cognitive states can emerge from observing different scenes. In this paper, we develop algorithms to enhance our understanding of visual processes by incorporating whole-brain activation maps while individuals are exposed to visual stimuli. We utilize large-scale fMRI encoders and Image generative models pre-trained on large public datasets, which are then fine-tuned through Image-fMRI contrastive learning. Our models hence can decode visual experience across the entire cerebral cortex, surpassing the traditional confines of the visual cortex. We first compare our method with state-of-the-art approaches to decoding visual processing and show improved predictive semantic accuracy by 43%. A network ablation analysis suggests that beyond the visual cortex, the default mode network contributes most to decoding stimuli, in line with the proposed role of this network in sense-making and semantic processing. Additionally, we implemented zero-shot imagination decoding on an extra validation dataset, achieving a p-value of 0.0206 for mapping the reconstructed images and ground-truth text stimuli, which substantiates the model's capability to capture semantic meanings across various scenarios.

Title: SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMs

• Authors: Ruben Härle, Felix Friedrich, Manuel Brack, Björn Deiseroth, Patrick Schramowski, Kristian Kersting
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07122
• Pdf URL: https://arxiv.org/pdf/2411.07122
• Copy Paste: [[2411.07122]] SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMs(https://arxiv.org/abs/2411.07122)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in generating human-like text, but their output may not be aligned with the user or even produce harmful content. This paper presents a novel approach to detect and steer concepts such as toxicity before generation. We introduce the Sparse Conditioned Autoencoder (SCAR), a single trained module that extends the otherwise untouched LLM. SCAR ensures full steerability, towards and away from concepts (e.g., toxic content), without compromising the quality of the model's text generation on standard evaluation benchmarks. We demonstrate the effective application of our approach through a variety of concepts, including toxicity, safety, and writing style alignment. As such, this work establishes a robust framework for controlling LLM generations, ensuring their ethical and safe deployment in real-world applications.

Title: Fast and Robust Contextual Node Representation Learning over Dynamic Graphs

• Authors: Xingzhi Guo, Silong Wang, Baojian Zhou, Yanghua Xiao, Steven Skiena
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07123
• Pdf URL: https://arxiv.org/pdf/2411.07123
• Copy Paste: [[2411.07123]] Fast and Robust Contextual Node Representation Learning over Dynamic Graphs(https://arxiv.org/abs/2411.07123)
• Keywords: robust
• Abstract: Real-world graphs grow rapidly with edge and vertex insertions over time, motivating the problem of efficiently maintaining robust node representation over evolving graphs. Recent efficient GNNs are designed to decouple recursive message passing from the learning process, and favor Personalized PageRank (PPR) as the underlying feature propagation mechanism. However, most PPR-based GNNs are designed for static graphs, and efficient PPR maintenance remains as an open problem. Further, there is surprisingly little theoretical justification for the choice of PPR, despite its impressive empirical performance. In this paper, we are inspired by the recent PPR formulation as an explicit $\ell_1$-regularized optimization problem and propose a unified dynamic graph learning framework based on sparse node-wise attention. We also present a set of desired properties to justify the choice of PPR in STOA GNNs, and serves as the guideline for future node attention designs. Meanwhile, we take advantage of the PPR-equivalent optimization formulation and employ the proximal gradient method (ISTA) to improve the efficiency of PPR-based GNNs upto 6 times. Finally, we instantiate a simple-yet-effective model (\textsc{GoPPE}) with robust positional encodings by maximizing PPR previously used as attention. The model performs comparably to or better than the STOA baselines and greatly outperforms when the initial node attributes are noisy during graph evolution, demonstrating the effectiveness and robustness of \textsc{GoPPE}.

Title: Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Models

• Authors: NVIDIA: Yuval Atzmon, Maciej Bala, Yogesh Balaji, Tiffany Cai, Yin Cui, Jiaojiao Fan, Yunhao Ge, Siddharth Gururani, Jacob Huffman, Ronald Isaac, Pooya Jannaty, Tero Karras, Grace Lam, J. P. Lewis, Aaron Licata, Yen-Chen Lin, Ming-Yu Liu, Qianli Ma, Arun Mallya, Ashlee Martino-Tarr, Doug Mendez, Seungjun Nah, Chris Pruett, Fitsum Reda, Jiaming Song, Ting-Chun Wang, Fangyin Wei, Xiaohui Zeng, Yu Zeng, Qinsheng Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07126
• Pdf URL: https://arxiv.org/pdf/2411.07126
• Copy Paste: [[2411.07126]] Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Models(https://arxiv.org/abs/2411.07126)
• Keywords: diffusion
• Abstract: We introduce Edify Image, a family of diffusion models capable of generating photorealistic image content with pixel-perfect accuracy. Edify Image utilizes cascaded pixel-space diffusion models trained using a novel Laplacian diffusion process, in which image signals at different frequency bands are attenuated at varying rates. Edify Image supports a wide range of applications, including text-to-image synthesis, 4K upsampling, ControlNets, 360 HDR panorama generation, and finetuning for image customization.

Title: Benchmarking LLMs' Judgments with No Gold Standard

• Authors: Shengwei Xu, Yuxuan Lu, Grant Schoenebeck, Yuqing Kong
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07127
• Pdf URL: https://arxiv.org/pdf/2411.07127
• Copy Paste: [[2411.07127]] Benchmarking LLMs' Judgments with No Gold Standard(https://arxiv.org/abs/2411.07127)
• Keywords: robust, generative, large language model
• Abstract: We introduce the GEM (Generative Estimator for Mutual Information), an evaluation metric for assessing language generation by Large Language Models (LLMs), particularly in generating informative judgments, without the need for a gold standard reference. GEM broadens the scenarios where we can benchmark LLM generation performance-from traditional ones, like machine translation and summarization, where gold standard references are readily available, to subjective tasks without clear gold standards, such as academic peer review. GEM uses a generative model to estimate mutual information between candidate and reference responses, without requiring the reference to be a gold standard. In experiments on a human-annotated dataset, GEM demonstrates competitive correlations with human scores compared to the state-of-the-art GPT-4o Examiner, and outperforms all other baselines. Additionally, GEM is more robust against strategic manipulations, such as rephrasing or elongation, which can artificially inflate scores under a GPT-4o Examiner. We also present GRE-bench (Generating Review Evaluation Benchmark) which evaluates LLMs based on how well they can generate high-quality peer reviews for academic research papers. Because GRE-bench is based upon GEM, it inherits its robustness properties. Additionally, GRE-bench circumvents data contamination problems (or data leakage) by using the continuous influx of new open-access research papers and peer reviews each year. We show GRE-bench results of various popular LLMs on their peer review capabilities using the ICLR2023 dataset.

Title: ZT-RIC:A Zero Trust RIC Framework for ensuring data Privacy and Confidentiality in Open RAN

• Authors: Diana Lin, Samarth Bhargav, Azuka Chiejina, Mohamed I. Ibrahem, Vijay K. Shah
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.07128
• Pdf URL: https://arxiv.org/pdf/2411.07128
• Copy Paste: [[2411.07128]] ZT-RIC:A Zero Trust RIC Framework for ensuring data Privacy and Confidentiality in Open RAN(https://arxiv.org/abs/2411.07128)
• Keywords: privacy
• Abstract: The advancement of 5G and NextG networks through Open Radio Access Network (O-RAN) architecture enables a shift toward virtualized, modular, and disaggregated configurations. A core component of O-RAN is the RAN Intelligent Controller (RIC), which manages RAN using machine learning-driven xApps that access sensitive data from RAN and User Equipment (UE), stored in the near Real-Time RIC (Near-RT RIC) database. This shared, open environment increases the risk of unauthorized data exposure. To address these concerns, this paper proposes a zero-trust RIC (ZT-RIC) framework that preserves data privacy across the RIC platform, including the RIC database, xApps, and E2 interface. ZT-RIC employs Inner Product Functional Encryption (IPFE) to encrypt RAN/UE data at the base station, preventing leaks through the E2 interface and shared database. Additionally, ZT-RIC enables xApps to perform inference on encrypted data without exposing sensitive information. For evaluation, a state-of-the-art InterClass xApp, which detects jamming signals using RAN key performance metrics (KPMs), is implemented. Testing on an LTE/5G O-RAN testbed shows that ZT-RIC preserves data confidentiality while achieving 97.9% accuracy in jamming detection and meeting sub-second latency requirements, with a round-trip time (RTT) of 0.527 seconds.

Title: Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis

• Authors: Taihang Hu, Linxuan Li, Joost van de Weijer, Hongcheng Gao, Fahad Shahbaz Khan, Jian Yang, Ming-Ming Cheng, Kai Wang, Yaxing Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07132
• Pdf URL: https://arxiv.org/pdf/2411.07132
• Copy Paste: [[2411.07132]] Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis(https://arxiv.org/abs/2411.07132)
• Keywords: large language model
• Abstract: Although text-to-image (T2I) models exhibit remarkable generation capabilities, they frequently fail to accurately bind semantically related objects or attributes in the input prompts; a challenge termed semantic binding. Previous approaches either involve intensive fine-tuning of the entire T2I model or require users or large language models to specify generation layouts, adding complexity. In this paper, we define semantic binding as the task of associating a given object with its attribute, termed attribute binding, or linking it to other related sub-objects, referred to as object binding. We introduce a novel method called Token Merging (ToMe), which enhances semantic binding by aggregating relevant tokens into a single composite token. This ensures that the object, its attributes and sub-objects all share the same cross-attention map. Additionally, to address potential confusion among main objects with complex textual prompts, we propose end token substitution as a complementary strategy. To further refine our approach in the initial stages of T2I generation, where layouts are determined, we incorporate two auxiliary losses, an entropy loss and a semantic binding loss, to iteratively update the composite token to improve the generation integrity. We conducted extensive experiments to validate the effectiveness of ToMe, comparing it against various existing methods on the T2I-CompBench and our proposed GPT-4o object binding benchmark. Our method is particularly effective in complex scenarios that involve multiple objects and attributes, which previous methods often fail to address. The code will be publicly available at \url{this https URL}.

Title: Edify 3D: Scalable High-Quality 3D Asset Generation

• Authors: NVIDIA: Maciej Bala, Yin Cui, Yifan Ding, Yunhao Ge, Zekun Hao, Jon Hasselgren, Jacob Huffman, Jingyi Jin, J.P. Lewis, Zhaoshuo Li, Chen-Hsuan Lin, Yen-Chen Lin, Tsung-Yi Lin, Ming-Yu Liu, Alice Luo, Qianli Ma, Jacob Munkberg, Stella Shi, Fangyin Wei, Donglai Xiang, Jiashu Xu, Xiaohui Zeng, Qinsheng Zhang
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.07135
• Pdf URL: https://arxiv.org/pdf/2411.07135
• Copy Paste: [[2411.07135]] Edify 3D: Scalable High-Quality 3D Asset Generation(https://arxiv.org/abs/2411.07135)
• Keywords: diffusion
• Abstract: We introduce Edify 3D, an advanced solution designed for high-quality 3D asset generation. Our method first synthesizes RGB and surface normal images of the described object at multiple viewpoints using a diffusion model. The multi-view observations are then used to reconstruct the shape, texture, and PBR materials of the object. Our method can generate high-quality 3D assets with detailed geometry, clean shape topologies, high-resolution textures, and materials within 2 minutes of runtime.

Title: Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models

• Authors: Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Hui Huang, Weixun Wang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, Xuepeng Liu, Dekai Sun, Wenbo Su, Bo Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07140
• Pdf URL: https://arxiv.org/pdf/2411.07140
• Copy Paste: [[2411.07140]] Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models(https://arxiv.org/abs/2411.07140)
• Keywords: large language model
• Abstract: New LLM evaluation benchmarks are important to align with the rapid development of Large Language Models (LLMs). In this work, we present Chinese SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions, and Chinese SimpleQA mainly has five properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6 major topics with 99 diverse subtopics. Second, we conduct a comprehensive quality control process to achieve high-quality questions and answers, where the reference answers are static and cannot be changed over time. Third, following SimpleQA, the questions and answers are very short, and the grading process is easy-to-evaluate based on OpenAI API. Based on Chinese SimpleQA, we perform a comprehensive evaluation on the factuality abilities of existing LLMs. Finally, we hope that Chinese SimpleQA could guide the developers to better understand the Chinese factuality abilities of their models and facilitate the growth of foundation models.

Title: Variational Graph Contrastive Learning

• Authors: Shifeng Xie, Jhony H. Giraldo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07150
• Pdf URL: https://arxiv.org/pdf/2411.07150
• Copy Paste: [[2411.07150]] Variational Graph Contrastive Learning(https://arxiv.org/abs/2411.07150)
• Keywords: robust
• Abstract: Graph representation learning (GRL) is a fundamental task in machine learning, aiming to encode high-dimensional graph-structured data into low-dimensional vectors. Self-supervised learning (SSL) methods are widely used in GRL because they can avoid expensive human annotation. In this work, we propose a novel Subgraph Gaussian Embedding Contrast (SGEC) method. Our approach introduces a subgraph Gaussian embedding module, which adaptively maps subgraphs to a structured Gaussian space, ensuring the preservation of graph characteristics while controlling the distribution of generated subgraphs. We employ optimal transport distances, including Wasserstein and Gromov-Wasserstein distances, to effectively measure the similarity between subgraphs, enhancing the robustness of the contrastive learning process. Extensive experiments across multiple benchmarks demonstrate that SGEC outperforms or presents competitive performance against state-of-the-art approaches. Our findings provide insights into the design of SSL methods for GRL, emphasizing the importance of the distribution of the generated contrastive pairs.

Title: Cascaded Dual Vision Transformer for Accurate Facial Landmark Detection

• Authors: Ziqiang Dang, Jianfang Li, Lin Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07167
• Pdf URL: https://arxiv.org/pdf/2411.07167
• Copy Paste: [[2411.07167]] Cascaded Dual Vision Transformer for Accurate Facial Landmark Detection(https://arxiv.org/abs/2411.07167)
• Keywords: transformer
• Abstract: Facial landmark detection is a fundamental problem in computer vision for many downstream applications. This paper introduces a new facial landmark detector based on vision transformers, which consists of two unique designs: Dual Vision Transformer (D-ViT) and Long Skip Connections (LSC). Based on the observation that the channel dimension of feature maps essentially represents the linear bases of the heatmap space, we propose learning the interconnections between these linear bases to model the inherent geometric relations among landmarks via Channel-split ViT. We integrate such channel-split ViT into the standard vision transformer (i.e., spatial-split ViT), forming our Dual Vision Transformer to constitute the prediction blocks. We also suggest using long skip connections to deliver low-level image features to all prediction blocks, thereby preventing useful information from being discarded by intermediate supervision. Extensive experiments are conducted to evaluate the performance of our proposal on the widely used benchmarks, i.e., WFLW, COFW, and 300W, demonstrating that our model outperforms the previous SOTAs across all three benchmarks.

Title: Continual Memorization of Factoids in Large Language Models

• Authors: Howard Chen, Jiayi Geng, Adithya Bhaskar, Dan Friedman, Danqi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07175
• Pdf URL: https://arxiv.org/pdf/2411.07175
• Copy Paste: [[2411.07175]] Continual Memorization of Factoids in Large Language Models(https://arxiv.org/abs/2411.07175)
• Keywords: protect, large language model
• Abstract: Large language models can absorb a massive amount of knowledge through pretraining, but pretraining is inefficient for acquiring long-tailed or specialized facts. Therefore, fine-tuning on specialized or new knowledge that reflects changes in the world has become popular, though it risks disrupting the model's original capabilities. We study this fragility in the context of continual memorization, where the model is trained on a small set of long-tail factoids (factual associations) and must retain these factoids after multiple stages of subsequent training on other datasets. Through extensive experiments, we show that LLMs suffer from forgetting across a wide range of subsequent tasks, and simple replay techniques do not fully prevent forgetting, especially when the factoid datasets are trained in the later stages. We posit that there are two ways to alleviate forgetting: 1) protect the memorization process as the model learns the factoids, or 2) reduce interference from training in later stages. With this insight, we develop an effective mitigation strategy: REMIX (Random and Generic Data Mixing). REMIX prevents forgetting by mixing generic data sampled from pretraining corpora or even randomly generated word sequences during each stage, despite being unrelated to the memorized factoids in the first stage. REMIX can recover performance from severe forgetting, often outperforming replay-based methods that have access to the factoids from the first stage. We then analyze how REMIX alters the learning process and find that successful forgetting prevention is associated with a pattern: the model stores factoids in earlier layers than usual and diversifies the set of layers that store these factoids. The efficacy of REMIX invites further investigation into the underlying dynamics of memorization and forgetting, opening exciting possibilities for future research.

Title: More Expressive Attention with Negative Weights

• Authors: Ang Lv, Ruobing Xie, Shuaipeng Li, Jiayi Liao, Xingwu Sun, Zhanhui Kang, Rui Yan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07176
• Pdf URL: https://arxiv.org/pdf/2411.07176
• Copy Paste: [[2411.07176]] More Expressive Attention with Negative Weights(https://arxiv.org/abs/2411.07176)
• Keywords: robust, diffusion, transformer
• Abstract: We propose a novel attention mechanism, named Cog Attention, that enables attention weights to be negative for enhanced expressiveness, which stems from two key factors: (1) Cog Attention can shift the token deletion and copying function from a static OV matrix to dynamic QK inner products, with the OV matrix now focusing more on refinement or modification. The attention head can simultaneously delete, copy, or retain tokens by assigning them negative, positive, or minimal attention weights, respectively. As a result, a single attention head becomes more flexible and expressive. (2) Cog Attention improves the model's robustness against representational collapse, which can occur when earlier tokens are over-squashed into later positions, leading to homogeneous representations. Negative weights reduce effective information paths from earlier to later tokens, helping to mitigate this issue. We develop Transformer-like models which use Cog Attention as attention modules, including decoder-only models for language modeling and U-ViT diffusion models for image generation. Experiments show that models using Cog Attention exhibit superior performance compared to those employing traditional softmax attention modules. Our approach suggests a promising research direction for rethinking and breaking the entrenched constraints of traditional softmax attention, such as the requirement for non-negative weights.

Title: Revisiting Ensembling in One-Shot Federated Learning

• Authors: Youssef Allouah, Akash Dhasade, Rachid Guerraoui, Nirupam Gupta, Anne-Marie Kermarrec, Rafael Pinot, Rafael Pires, Rishi Sharma
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.07182
• Pdf URL: https://arxiv.org/pdf/2411.07182
• Copy Paste: [[2411.07182]] Revisiting Ensembling in One-Shot Federated Learning(https://arxiv.org/abs/2411.07182)
• Keywords: federate
• Abstract: Federated learning (FL) is an appealing approach to training machine learning models without sharing raw data. However, standard FL algorithms are iterative and thus induce a significant communication cost. One-shot federated learning (OFL) trades the iterative exchange of models between clients and the server with a single round of communication, thereby saving substantially on communication costs. Not surprisingly, OFL exhibits a performance gap in terms of accuracy with respect to FL, especially under high data heterogeneity. We introduce FENS, a novel federated ensembling scheme that approaches the accuracy of FL with the communication efficiency of OFL. Learning in FENS proceeds in two phases: first, clients train models locally and send them to the server, similar to OFL; second, clients collaboratively train a lightweight prediction aggregator model using FL. We showcase the effectiveness of FENS through exhaustive experiments spanning several datasets and heterogeneity levels. In the particular case of heterogeneously distributed CIFAR-10 dataset, FENS achieves up to a 26.9% higher accuracy over state-of-the-art (SOTA) OFL, being only 3.1% lower than FL. At the same time, FENS incurs at most 4.3x more communication than OFL, whereas FL is at least 10.9x more communication-intensive than FENS.

Title: SAMPart3D: Segment Any Part in 3D Objects

• Authors: Yunhan Yang, Yukun Huang, Yuan-Chen Guo, Liangjun Lu, Xiaoyang Wu, Edmund Y. Lam, Yan-Pei Cao, Xihui Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07184
• Pdf URL: https://arxiv.org/pdf/2411.07184
• Copy Paste: [[2411.07184]] SAMPart3D: Segment Any Part in 3D Objects(https://arxiv.org/abs/2411.07184)
• Keywords: extraction, segmentation
• Abstract: 3D part segmentation is a crucial and challenging task in 3D perception, playing a vital role in applications such as robotics, 3D generation, and 3D editing. Recent methods harness the powerful Vision Language Models (VLMs) for 2D-to-3D knowledge distillation, achieving zero-shot 3D part segmentation. However, these methods are limited by their reliance on text prompts, which restricts the scalability to large-scale unlabeled datasets and the flexibility in handling part ambiguities. In this work, we introduce SAMPart3D, a scalable zero-shot 3D part segmentation framework that segments any 3D object into semantic parts at multiple granularities, without requiring predefined part label sets as text prompts. For scalability, we use text-agnostic vision foundation models to distill a 3D feature extraction backbone, allowing scaling to large unlabeled 3D datasets to learn rich 3D priors. For flexibility, we distill scale-conditioned part-aware 3D features for 3D part segmentation at multiple granularities. Once the segmented parts are obtained from the scale-conditioned part-aware 3D features, we use VLMs to assign semantic labels to each part based on the multi-view renderings. Compared to previous methods, our SAMPart3D can scale to the recent large-scale 3D object dataset Objaverse and handle complex, non-ordinary objects. Additionally, we contribute a new 3D part segmentation benchmark to address the lack of diversity and complexity of objects and parts in existing benchmarks. Experiments show that our SAMPart3D significantly outperforms existing zero-shot 3D part segmentation methods, and can facilitate various applications such as part-level editing and interactive segmentation.

Title: The Super Weight in Large Language Models

• Authors: Mengxia Yu, De Wang, Qi Shan, Colorado Reed, Alvin Wan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07191
• Pdf URL: https://arxiv.org/pdf/2411.07191
• Copy Paste: [[2411.07191]] The Super Weight in Large Language Models(https://arxiv.org/abs/2411.07191)
• Keywords: data-free, large language model
• Abstract: Recent works have shown a surprising result: a small fraction of Large Language Model (LLM) parameter outliers are disproportionately important to the quality of the model. LLMs contain billions of parameters, so these small fractions, such as 0.01%, translate to hundreds of thousands of parameters. In this work, we present an even more surprising finding: Pruning as few as a single parameter can destroy an LLM's ability to generate text -- increasing perplexity by 3 orders of magnitude and reducing zero-shot accuracy to guessing. We propose a data-free method for identifying such parameters, termed super weights, using a single forward pass through the model. We additionally find that these super weights induce correspondingly rare and large activation outliers, termed super activations. When preserved with high precision, super activations can improve simple round-to-nearest quantization to become competitive with state-of-the-art methods. For weight quantization, we similarly find that by preserving the super weight and clipping other weight outliers, round-to-nearest quantization can scale to much larger block sizes than previously considered. To facilitate further research into super weights, we provide an index of super weight coordinates for common, openly available LLMs.

Title: OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision

• Authors: Cong Wei, Zheyang Xiong, Weiming Ren, Xinrun Du, Ge Zhang, Wenhu Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07199
• Pdf URL: https://arxiv.org/pdf/2411.07199
• Copy Paste: [[2411.07199]] OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision(https://arxiv.org/abs/2411.07199)
• Keywords: diffusion
• Abstract: Instruction-guided image editing methods have demonstrated significant potential by training diffusion models on automatically synthesized or manually annotated image editing pairs. However, these methods remain far from practical, real-life applications. We identify three primary challenges contributing to this gap. Firstly, existing models have limited editing skills due to the biased synthesis process. Secondly, these methods are trained with datasets with a high volume of noise and artifacts. This is due to the application of simple filtering methods like CLIP-score. Thirdly, all these datasets are restricted to a single low resolution and fixed aspect ratio, limiting the versatility to handle real-world use cases. In this paper, we present \omniedit, which is an omnipotent editor to handle seven different image editing tasks with any aspect ratio seamlessly. Our contribution is in four folds: (1) \omniedit is trained by utilizing the supervision from seven different specialist models to ensure task coverage. (2) we utilize importance sampling based on the scores provided by large multimodal models (like GPT-4o) instead of CLIP-score to improve the data quality. (3) we propose a new editing architecture called EditNet to greatly boost the editing success rate, (4) we provide images with different aspect ratios to ensure that our model can handle any image in the wild. We have curated a test set containing images of different aspect ratios, accompanied by diverse instructions to cover different tasks. Both automatic evaluation and human evaluations demonstrate that \omniedit can significantly outperform all the existing models. Our code, dataset and model will be available at \url{this https URL}

Title: DLCR: A Generative Data Expansion Framework via Diffusion for Clothes-Changing Person Re-ID

• Authors: Nyle Siddiqui, Florinel Alin Croitoru, Gaurav Kumar Nayak, Radu Tudor Ionescu, Mubarak Shah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07205
• Pdf URL: https://arxiv.org/pdf/2411.07205
• Copy Paste: [[2411.07205]] DLCR: A Generative Data Expansion Framework via Diffusion for Clothes-Changing Person Re-ID(https://arxiv.org/abs/2411.07205)
• Keywords: diffusion, generative, large language model
• Abstract: With the recent exhibited strength of generative diffusion models, an open research question is \textit{if images generated by these models can be used to learn better visual representations}. While this generative data expansion may suffice for easier visual tasks, we explore its efficacy on a more difficult discriminative task: clothes-changing person re-identification (CC-ReID). CC-ReID aims to match people appearing in non-overlapping cameras, even when they change their clothes across cameras. Not only are current CC-ReID models constrained by the limited diversity of clothing in current CC-ReID datasets, but generating additional data that retains important personal features for accurate identification is a current challenge. To address this issue we propose DLCR, a novel data expansion framework that leverages pre-trained diffusion and large language models (LLMs) to accurately generate diverse images of individuals in varied attire. We generate additional data for five benchmark CC-ReID datasets (PRCC, CCVID, LaST, VC-Clothes, and LTCC) and \textbf{increase their clothing diversity by \boldmath{$10$}x, totaling over \boldmath{$2.1$}M images generated}. DLCR employs diffusion-based text-guided inpainting, conditioned on clothing prompts constructed using LLMs, to generate synthetic data that only modifies a subject's clothes while preserving their personally identifiable features. With this massive increase in data, we introduce two novel strategies - progressive learning and test-time prediction refinement - that respectively reduce training time and further boosts CC-ReID performance. On the PRCC dataset, we obtain a large top-1 accuracy improvement of $11.3\%$ by training CAL, a previous state of the art (SOTA) method, with DLCR-generated data. We publicly release our code and generated data for each dataset here: \url{this https URL}.

Title: Comparing Bottom-Up and Top-Down Steering Approaches on In-Context Learning Tasks

• Authors: Madeline Brumley, Joe Kwon, David Krueger, Dmitrii Krasheninnikov, Usman Anwar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07213
• Pdf URL: https://arxiv.org/pdf/2411.07213
• Copy Paste: [[2411.07213]] Comparing Bottom-Up and Top-Down Steering Approaches on In-Context Learning Tasks(https://arxiv.org/abs/2411.07213)
• Keywords: robust, interpretability, large language model
• Abstract: A key objective of interpretability research on large language models (LLMs) is to develop methods for robustly steering models toward desired behaviors. To this end, two distinct approaches to interpretability -- ``bottom-up" and ``top-down" -- have been presented, but there has been little quantitative comparison between them. We present a case study comparing the effectiveness of representative vector steering methods from each branch: function vectors (FV; arXiv:2310.15213), as a bottom-up method, and in-context vectors (ICV; arXiv:2311.06668) as a top-down method. While both aim to capture compact representations of broad in-context learning tasks, we find they are effective only on specific types of tasks: ICVs outperform FVs in behavioral shifting, whereas FVs excel in tasks requiring more precision. We discuss the implications for future evaluations of steering methods and for further research into top-down and bottom-up steering given these findings.

Title: Feature Selection Based on Wasserstein Distance

• Authors: Fuwei Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07217
• Pdf URL: https://arxiv.org/pdf/2411.07217
• Copy Paste: [[2411.07217]] Feature Selection Based on Wasserstein Distance(https://arxiv.org/abs/2411.07217)
• Keywords: robust
• Abstract: In this paper, we present a novel feature selection method based on the Wasserstein distance. Feature selection plays a critical role in reducing the dimensionality of input data, thereby improving machine learning efficiency and generalization performance. Unlike traditional feature selection approaches that rely on criteria such as correlation or KL divergence, our method leverages the Wasserstein distance to measure the similarity between distributions of selected features and original features. This approach inherently accounts for similarities between classes, making it robust in scenarios involving noisy labels. Experimental results demonstrate that our method outperforms traditional approaches, particularly in challenging settings involving noisy labeled data.

Title: TreeCoders: Trees of Transformers

• Authors: Pierre Colonna D'Istria, Abdulrahman Altahhan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07218
• Pdf URL: https://arxiv.org/pdf/2411.07218
• Copy Paste: [[2411.07218]] TreeCoders: Trees of Transformers(https://arxiv.org/abs/2411.07218)
• Keywords: transformer
• Abstract: In this paper, we introduce TreeCoders, a novel family of transformer trees. We moved away from traditional linear transformers to complete k-ary trees. Transformer blocks serve as nodes, and generic classifiers learn to select the best child and route the sequence of tokens to a specific leaf. The selectors, moved outside the transformer blocks, allow for the use of a variety of architecture without further modifications. Furthermore, our proposed architecture supports sparse node activation due to the logarithmic complexity of a tree search. We validate our idea by testing a series of decoder-only tree transformers, achieving competitive results across a diverse range of language datasets. Our study demonstrates that the proposed tree transformer model outperforms a size-equivalent linear transformer model 76\% of the time over a wide range of tree architectures. Furthermore, our proposed model naturally lends itself to distributed implementation.

Title: TempCharBERT: Keystroke Dynamics for Continuous Access Control Based on Pre-trained Language Models

• Authors: Matheus Simão, Fabiano Prado, Omar Abdul Wahab, Anderson Avila
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07224
• Pdf URL: https://arxiv.org/pdf/2411.07224
• Copy Paste: [[2411.07224]] TempCharBERT: Keystroke Dynamics for Continuous Access Control Based on Pre-trained Language Models(https://arxiv.org/abs/2411.07224)
• Keywords: privacy, attack, federate
• Abstract: With the widespread of digital environments, reliable authentication and continuous access control has become crucial. It can minimize cyber attacks and prevent frauds, specially those associated with identity theft. A particular interest lies on keystroke dynamics (KD), which refers to the task of recognizing individuals' identity based on their unique typing style. In this work, we propose the use of pre-trained language models (PLMs) to recognize such patterns. Although PLMs have shown high performance on multiple NLP benchmarks, the use of these models on specific tasks requires customization. BERT and RoBERTa, for instance, rely on subword tokenization, and they cannot be directly applied to KD, which requires temporal-character information to recognize users. Recent character-aware PLMs are able to process both subwords and character-level information and can be an alternative solution. Notwithstanding, they are still not suitable to be directly fine-tuned for KD as they are not optimized to account for user's temporal typing information (e.g., hold time and flight time). To overcome this limitation, we propose TempCharBERT, an architecture that incorporates temporal-character information in the embedding layer of CharBERT. This allows modeling keystroke dynamics for the purpose of user identification and authentication. Our results show a significant improvement with this customization. We also showed the feasibility of training TempCharBERT on a federated learning settings in order to foster data privacy.

Title: Learning from Limited and Imperfect Data

• Authors: Harsh Rangwani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.07229
• Pdf URL: https://arxiv.org/pdf/2411.07229
• Copy Paste: [[2411.07229]] Learning from Limited and Imperfect Data(https://arxiv.org/abs/2411.07229)
• Keywords: robust, generative
• Abstract: The datasets used for Deep Neural Network training (e.g., ImageNet, MSCOCO, etc.) are often manually balanced across categories (classes) to facilitate learning of all the categories. This curation process is often expensive and requires throwing away precious annotated data to balance the frequency across classes. This is because the distribution of data in the world (e.g., internet, etc.) significantly differs from the well-curated datasets and is often over-populated with samples from common categories. The algorithms designed for well-curated datasets perform suboptimally when used to learn from imperfect datasets with long-tailed imbalances and distribution shifts. For deep models to be widely used, getting away with the costly curation process by developing robust algorithms that can learn from real-world data distribution is necessary. Toward this goal, we develop practical algorithms for Deep Neural Networks that can learn from limited and imperfect data present in the real world. These works are divided into four segments, each covering a scenario of learning from limited or imperfect data. The first part of the works focuses on Learning Generative Models for Long-Tail Data, where we mitigate the mode-collapse for tail (minority) classes and enable diverse aesthetic image generations as head (majority) classes. In the second part, we enable effective generalization on tail classes through Inductive Regularization schemes, which allow tail classes to generalize as the head classes without enforcing explicit generation of images. In the third part, we develop algorithms for Optimizing Relevant Metrics compared to the average accuracy for learning from long-tailed data with limited annotation (semi-supervised), followed by the fourth part, which focuses on the effective domain adaptation of the model to various domains with zero to very few labeled samples.

Title: Watermark Anything with Localized Messages

• Authors: Tom Sander, Pierre Fernandez, Alain Durmus, Teddy Furon, Matthijs Douze
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.07231
• Pdf URL: https://arxiv.org/pdf/2411.07231
• Copy Paste: [[2411.07231]] Watermark Anything with Localized Messages(https://arxiv.org/abs/2411.07231)
• Keywords: robust, watermark
• Abstract: Image watermarking methods are not tailored to handle small watermarked areas. This restricts applications in real-world scenarios where parts of the image may come from different sources or have been edited. We introduce a deep-learning model for localized image watermarking, dubbed the Watermark Anything Model (WAM). The WAM embedder imperceptibly modifies the input image, while the extractor segments the received image into watermarked and non-watermarked areas and recovers one or several hidden messages from the areas found to be watermarked. The models are jointly trained at low resolution and without perceptual constraints, then post-trained for imperceptibility and multiple watermarks. Experiments show that WAM is competitive with state-of-the art methods in terms of imperceptibility and robustness, especially against inpainting and splicing, even on high-resolution images. Moreover, it offers new capabilities: WAM can locate watermarked areas in spliced images and extract distinct 32-bit messages with less than 1 bit error from multiple small regions - no larger than 10% of the image surface - even for small $256\times 256$ images.

Title: Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Models

• Authors: Yoad Tewel, Rinon Gal, Dvir Samuel Yuval Atzmon, Lior Wolf, Gal Chechik
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07232
• Pdf URL: https://arxiv.org/pdf/2411.07232
• Copy Paste: [[2411.07232]] Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Models(https://arxiv.org/abs/2411.07232)
• Keywords: diffusion
• Abstract: Adding Object into images based on text instructions is a challenging task in semantic image editing, requiring a balance between preserving the original scene and seamlessly integrating the new object in a fitting location. Despite extensive efforts, existing models often struggle with this balance, particularly with finding a natural location for adding an object in complex scenes. We introduce Add-it, a training-free approach that extends diffusion models' attention mechanisms to incorporate information from three key sources: the scene image, the text prompt, and the generated image itself. Our weighted extended-attention mechanism maintains structural consistency and fine details while ensuring natural object placement. Without task-specific fine-tuning, Add-it achieves state-of-the-art results on both real and generated image insertion benchmarks, including our newly constructed "Additing Affordance Benchmark" for evaluating object placement plausibility, outperforming supervised methods. Human evaluations show that Add-it is preferred in over 80% of cases, and it also demonstrates improvements in various automated metrics.

Title: Score-based generative diffusion with "active" correlated noise sources

• Authors: Alexandra Lamtyugina, Agnish Kumar Behera, Aditya Nandy, Carlos Floyd, Suriyanarayanan Vaikuntanathan
• Subjects: cs.LG, cond-mat.dis-nn
• Abstract URL: https://arxiv.org/abs/2411.07233
• Pdf URL: https://arxiv.org/pdf/2411.07233
• Copy Paste: [[2411.07233]] Score-based generative diffusion with "active" correlated noise sources(https://arxiv.org/abs/2411.07233)
• Keywords: robust, diffusion, generative
• Abstract: Diffusion models exhibit robust generative properties by approximating the underlying distribution of a dataset and synthesizing data by sampling from the approximated distribution. In this work, we explore how the generative performance may be be modulated if noise sources with temporal correlations -- akin to those used in the field of active matter -- are used for the destruction of the data in the forward process. Our numerical and analytical experiments suggest that the corresponding reverse process may exhibit improved generative properties.

Title: OpenThaiGPT 1.5: A Thai-Centric Open Source Large Language Model

• Authors: Sumeth Yuenyong, Kobkrit Viriyayudhakorn, Apivadee Piyatumrong, Jillaphat Jaroenkantasima
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.07238
• Pdf URL: https://arxiv.org/pdf/2411.07238
• Copy Paste: [[2411.07238]] OpenThaiGPT 1.5: A Thai-Centric Open Source Large Language Model(https://arxiv.org/abs/2411.07238)
• Keywords: large language model
• Abstract: OpenThaiGPT 1.5 is an advanced Thai language chat model based on Qwen v2.5, finetuned on over 2,000,000 Thai instruction pairs. This report provides an engineering perspective on the model's development, capabilities, and performance. We discuss the model's architecture, training process, and key features, including multi-turn conversation support, Retrieval Augmented Generation (RAG) compatibility, and tool-calling functionality. Benchmark results demonstrate OpenThaiGPT 1.5's state-of-the-art performance on various Thai language tasks, outperforming other open-source Thai language models. We also address practical considerations such as GPU memory requirements and deployment strategies.

Title: DeepONet as a Multi-Operator Extrapolation Model: Distributed Pretraining with Physics-Informed Fine-Tuning

• Authors: Zecheng Zhang, Christian Moya, Lu Lu, Guang Lin, Hayden Schaeffer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.07239
• Pdf URL: https://arxiv.org/pdf/2411.07239
• Copy Paste: [[2411.07239]] DeepONet as a Multi-Operator Extrapolation Model: Distributed Pretraining with Physics-Informed Fine-Tuning(https://arxiv.org/abs/2411.07239)
• Keywords: robust
• Abstract: We propose a novel fine-tuning method to achieve multi-operator learning through training a distributed neural operator with diverse function data and then zero-shot fine-tuning the neural network using physics-informed losses for downstream tasks. Operator learning effectively approximates solution operators for PDEs and various PDE-related problems, yet it often struggles to generalize to new tasks. To address this, we investigate fine-tuning a pretrained model, while carefully selecting an initialization that enables rapid adaptation to new tasks with minimal data. Our approach combines distributed learning to integrate data from various operators in pre-training, while physics-informed methods enable zero-shot fine-tuning, minimizing the reliance on downstream data. We investigate standard fine-tuning and Low-Rank Adaptation fine-tuning, applying both to train complex nonlinear target operators that are difficult to learn only using random initialization. Through comprehensive numerical examples, we demonstrate the advantages of our approach, showcasing significant improvements in accuracy. Our findings provide a robust framework for advancing multi-operator learning and highlight the potential of transfer learning techniques in this domain.

Title: UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts

• Authors: Bo Yang, Qingping Yang, Runtao Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.07240
• Pdf URL: https://arxiv.org/pdf/2411.07240
• Copy Paste: [[2411.07240]] UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts(https://arxiv.org/abs/2411.07240)
• Keywords: robust, large language model
• Abstract: The evaluation of mathematical reasoning capabilities is essential for advancing Artificial General Intelligence (AGI). While Large Language Models (LLMs) have shown impressive performance in solving mathematical problems, existing benchmarks such as GSM8K and MATH present limitations, including narrow problem definitions with specific numbers and reliance on predetermined rules that hinder accurate assessments of reasoning and adaptability. This paper introduces the UTMath Benchmark, which robustly evaluates the models through extensive unit tests. It consists of 1,053 problems across 9 mathematical domains, with over 68 test cases per this http URL propose an innovative evaluation framework inspired by unit testing in software development, focusing on both accuracy and reliability of results. Furthermore, we introduce the Reasoning-to-Coding of Thoughts (RCoT) approach, which encourages LLMs to perform explicit reasoning before generating code, leading to generating more advanced solution and improved performance. Furthermore, we are releasing not only the UTMath benchmark but also the UTMath-Train training dataset (more than 70k samples), to support the community in further exploring mathematical reasoning.